-
Bug
-
Resolution: Fixed
-
High
-
9.2.2, 9.2.3, 9.2.4
-
6
-
Severity 2 - Major
-
56
-
Issue Summary
In environments with a high volume of pages containing inline tasks, the flushTaskIndexQueue Processing can lead to an out-of-memory (OOM) situation. This occurs during the traversal of a page and its descendants, when the code instantiates a large number of objects in memory before they are processed. When the number of pages is sufficiently large, the resulting memory pressure can cause an OOM situation.
Steps to Reproduce
- Create a large page tree containing a high volume of inline tasks, around 50,000+ or more inline tasks spread across the entire page hierarchy.
- Modify the page permissions for a parent page with a large number of child pages containing inline tasks.
- When page permissions are modified for a parent page with a large number of child pages containing inline tasks, a REINDEX_INLINE_TASKS_FROM_PAGE_INCLUDING_CHILDREN entry is added to the JOURNALENTRY table, which is picked by the flushTaskIndexQueue job for processing.
- For parent pages with a large number of child pages, this triggers the loading of every inline task from all child pages into memory for processing, potentially leading to memory pressure or an OOM situation.
Expected Results
The flushTaskIndexQueue job successfully processes journal entries for task indexing, ensuring that the task indexes remain up to date.
Actual Results
The environment or node enters an Out of Memory (OOM) state, making it unusable.
Diagnosis 1
Capturing Thread dumps will show Caesium thread running through:
... at com.atlassian.confluence.plugins.tasklist.report.searchindex.InlineTaskSearchDocumentFactory.buildDocument(Lcom/atlassian/confluence/core/ContentEntityObject;Lcom/atlassian/confluence/plugins/tasklist/ao/AOInlineTask;)Lcom/atlassian/confluence/search/v2/AtlassianDocument; (InlineTaskSearchDocumentFactory.java:97) at com.atlassian.confluence.plugins.tasklist.report.searchindex.indexaction.AddSearchDocumentForInlineTaskAction.<init>(Lcom/atlassian/confluence/plugins/tasklist/report/searchindex/InlineTaskSearchDocumentFactory;Lcom/atlassian/confluence/core/ContentEntityObject;Lcom/atlassian/confluence/plugins/tasklist/ao/AOInlineTask;)V (AddSearchDocumentForInlineTaskAction.java:30) at com.atlassian.confluence.plugins.tasklist.report.searchindex.indexmanagement.indexqueue.IndexQueueProcessorImpl.lambda$createActionsForAddingTasksFromPage$7(Lcom/atlassian/confluence/core/ContentEntityObject;Lcom/atlassian/confluence/plugins/tasklist/ao/AOInlineTask;)Lcom/atlassian/confluence/plugins/tasklist/report/searchindex/indexaction/AddSearchDocumentForInlineTaskAction; (IndexQueueProcessorImpl.java:345) at com.atlassian.confluence.plugins.tasklist.report.searchindex.indexmanagement.indexqueue.IndexQueueProcessorImpl$$Lambda+0x00007f42afd968e0.apply(Ljava/lang/Object;)Ljava/lang/Object; (Unknown Source) at java.util.stream.ReferencePipeline$3$1.accept(Ljava/lang/Object;)V (ReferencePipeline.java:197) at java.util.Spliterators$ArraySpliterator.forEachRemaining(Ljava/util/function/Consumer;)V (Spliterators.java:1024) at java.util.stream.AbstractPipeline.copyInto(Ljava/util/stream/Sink;Ljava/util/Spliterator;)V (AbstractPipeline.java:509) at java.util.stream.AbstractPipeline.wrapAndCopyInto(Ljava/util/stream/Sink;Ljava/util/Spliterator;)Ljava/util/stream/Sink; (AbstractPipeline.java:499) at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Ljava/util/stream/PipelineHelper;Ljava/util/Spliterator;)Ljava/lang/Object; (ReduceOps.java:921) at java.util.stream.AbstractPipeline.evaluate(Ljava/util/stream/TerminalOp;)Ljava/lang/Object; (AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.collect(Ljava/util/stream/Collector;)Ljava/lang/Object; (ReferencePipeline.java:682) at com.atlassian.confluence.plugins.tasklist.report.searchindex.indexmanagement.indexqueue.IndexQueueProcessorImpl.createActionsForAddingTasksFromPage(Lcom/atlassian/confluence/core/ContentEntityObject;)Ljava/util/Collection; (IndexQueueProcessorImpl.java:346) at com.atlassian.confluence.plugins.tasklist.report.searchindex.indexmanagement.indexqueue.IndexQueueProcessorImpl.createActionForAddingTasksFromPageAndAllItsDescendants(Ljava/lang/Long;)Ljava/util/Collection; (IndexQueueProcessorImpl.java:290) at com.atlassian.confluence.plugins.tasklist.report.searchindex.indexmanagement.indexqueue.IndexQueueProcessorImpl.createActionForAddingInlineTasksToTheSearchIndex(Lcom/atlassian/confluence/plugins/tasklist/report/searchindex/indexmanagement/task/TaskLevel;Ljava/lang/Long;)Ljava/util/Collection; (IndexQueueProcessorImpl.java:270) at com.atlassian.confluence.plugins.tasklist.report.searchindex.indexmanagement.indexqueue.IndexQueueProcessorImpl.lambda$getActionsForParticularTaskType$4(Ljava/util/List;Lcom/atlassian/confluence/plugins/tasklist/report/searchindex/indexmanagement/task/InlineTaskIndexTaskType;Ljava/lang/Long;)V (IndexQueueProcessorImpl.java:201) at com.atlassian.confluence.plugins.tasklist.report.searchindex.indexmanagement.indexqueue.IndexQueueProcessorImpl$$Lambda+0x00007f42afd961b0.accept(Ljava/lang/Object;)V (Unknown Source) at java.lang.Iterable.forEach(Ljava/util/function/Consumer;)V (Iterable.java:75) at com.atlassian.confluence.plugins.tasklist.report.searchindex.indexmanagement.indexqueue.IndexQueueProcessorImpl.getActionsForParticularTaskType(Lcom/atlassian/confluence/plugins/tasklist/report/searchindex/indexmanagement/task/InlineTaskIndexTaskType;Ljava/util/List;)Ljava/util/List; (IndexQueueProcessorImpl.java:201) at com.atlassian.confluence.plugins.tasklist.report.searchindex.indexmanagement.indexqueue.IndexQueueProcessorImpl.lambda$getSearchIndexActionsForAllIndexingLevels$3(Ljava/util/List;Ljava/lang/String;Ljava/util/List;)V (IndexQueueProcessorImpl.java:187) at com.atlassian.confluence.plugins.tasklist.report.searchindex.indexmanagement.indexqueue.IndexQueueProcessorImpl$$Lambda+0x00007f42afd956e0.accept(Ljava/lang/Object;Ljava/lang/Object;)V (Unknown Source) at java.util.HashMap.forEach(Ljava/util/function/BiConsumer;)V (HashMap.java:1429) at com.atlassian.confluence.plugins.tasklist.report.searchindex.indexmanagement.indexqueue.IndexQueueProcessorImpl.getSearchIndexActionsForAllIndexingLevels(Ljava/util/List;)Ljava/util/List; (IndexQueueProcessorImpl.java:185) at com.atlassian.confluence.plugins.tasklist.report.searchindex.indexmanagement.indexqueue.IndexQueueProcessorImpl.lambda$processJournalEntries$2(ZLjava/util/concurrent/atomic/AtomicInteger;Ljava/lang/Iterable;)Lcom/atlassian/confluence/api/service/journal/EntryProcessorResult; (IndexQueueProcessorImpl.java:157) ...
Diagnosis 2
In another variant of the issue, capturing Thread dumps will show Caesium thread running through:
... at com.atlassian.confluence.plugins.tasklist.report.searchindex.InlineTaskSearchDocumentFactory.getPageAncestors(Lcom/atlassian/confluence/pages/Page;)Ljava/util/Collection; (InlineTaskSearchDocumentFactory.java:155) at com.atlassian.confluence.plugins.tasklist.report.searchindex.InlineTaskSearchDocumentFactory.addAncestors(Lcom/atlassian/confluence/search/v2/AtlassianDocument;Lcom/atlassian/confluence/core/ContentEntityObject;)V (InlineTaskSearchDocumentFactory.java:132) at com.atlassian.confluence.plugins.tasklist.report.searchindex.InlineTaskSearchDocumentFactory.buildDocument(Lcom/atlassian/confluence/core/ContentEntityObject;Lcom/atlassian/confluence/plugins/tasklist/ao/AOInlineTask;)Lcom/atlassian/confluence/search/v2/AtlassianDocument; (InlineTaskSearchDocumentFactory.java:93) at com.atlassian.confluence.plugins.tasklist.report.searchindex.indexaction.AddSearchDocumentForInlineTaskAction.<init>(Lcom/atlassian/confluence/plugins/tasklist/report/searchindex/InlineTaskSearchDocumentFactory;Lcom/atlassian/confluence/core/ContentEntityObject;Lcom/atlassian/confluence/plugins/tasklist/ao/AOInlineTask;)V (AddSearchDocumentForInlineTaskAction.java:30) at com.atlassian.confluence.plugins.tasklist.report.searchindex.indexmanagement.indexqueue.IndexQueueProcessorImpl.createActionForAddingSingleTask(Ljava/lang/Long;)Ljava/util/Collection; (IndexQueueProcessorImpl.java:325) at com.atlassian.confluence.plugins.tasklist.report.searchindex.indexmanagement.indexqueue.IndexQueueProcessorImpl.createActionForAddingInlineTasksToTheSearchIndex(Lcom/atlassian/confluence/plugins/tasklist/report/searchindex/indexmanagement/task/TaskLevel;Ljava/lang/Long;)Ljava/util/Collection; (IndexQueueProcessorImpl.java:271) at com.atlassian.confluence.plugins.tasklist.report.searchindex.indexmanagement.indexqueue.IndexQueueProcessorImpl.lambda$getActionsForParticularTaskType$4(Ljava/util/List;Lcom/atlassian/confluence/plugins/tasklist/report/searchindex/indexmanagement/task/InlineTaskIndexTaskType;Ljava/lang/Long;)V (IndexQueueProcessorImpl.java:206) at com.atlassian.confluence.plugins.tasklist.report.searchindex.indexmanagement.indexqueue.IndexQueueProcessorImpl$$Lambda$4304+0x0000000303d84250.accept(Ljava/lang/Object;)V () at java.lang.Iterable.forEach(Ljava/util/function/Consumer;)V (Iterable.java:75) at com.atlassian.confluence.plugins.tasklist.report.searchindex.indexmanagement.indexqueue.IndexQueueProcessorImpl.getActionsForParticularTaskType(Lcom/atlassian/confluence/plugins/tasklist/report/searchindex/indexmanagement/task/InlineTaskIndexTaskType;Ljava/util/List;)Ljava/util/List; (IndexQueueProcessorImpl.java:206) at com.atlassian.confluence.plugins.tasklist.report.searchindex.indexmanagement.indexqueue.IndexQueueProcessorImpl.lambda$getSearchIndexActionsForAllIndexingLevels$3(Ljava/util/List;Ljava/lang/String;Ljava/util/List;)V (IndexQueueProcessorImpl.java:188) at com.atlassian.confluence.plugins.tasklist.report.searchindex.indexmanagement.indexqueue.IndexQueueProcessorImpl$$Lambda$4301+0x0000000302d47560.accept(Ljava/lang/Object;Ljava/lang/Object;)V () at java.util.HashMap.forEach(Ljava/util/function/BiConsumer;)V (HashMap.java:1421) at com.atlassian.confluence.plugins.tasklist.report.searchindex.indexmanagement.indexqueue.IndexQueueProcessorImpl.getSearchIndexActionsForAllIndexingLevels(Ljava/util/List;)Ljava/util/List; (IndexQueueProcessorImpl.java:185) ...
Workaround
- Disable the flushTaskIndexQueue Scheduled Job - Log in to Confluence Base URL > General Configuration > Scheduled Jobs and search for "scheduledjob.desc.flushTaskIndexQueue" and disable the job.
- Enable the dark feature "confluence.task-report.use-database-for-reports" in your Confluence instance. This will configure the Task Report macro to use the database for retrieving inline tasks instead of relying on the Task Index.
