High CPU usage with several write threads_pools in queue

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Opensearch 2.8

Describe the issue:
CPU is constantly going 100% when I analysed I found that there are several events queued in the write thread_pool, Upon further investigation on

GET _node/hot_threads

::: {ip-10-220-48-104}{zc74roPORfeMjGpeX26hmg}{YnlszwN4QM-Y8Uz0_8EixQ}{10.220.48.104}{10.220.48.104:9300}{dimr}{shard_indexing_pressure_enabled=true}
   Hot threads at 2023-08-24T03:43:58.963Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
   
   99.1% (495.5ms out of 500ms) cpu usage by thread 'opensearch[ip-10-220-48-104][write][T#1]'
     10/10 snapshots sharing following 30 elements
       app//org.opensearch.index.mapper.ParseContext$Document.getFields(ParseContext.java:146)
       app//org.opensearch.index.mapper.FlatObjectFieldMapper.parseValueAddFields(FlatObjectFieldMapper.java:631)
       app//org.opensearch.index.mapper.FlatObjectFieldMapper.parseCreateField(FlatObjectFieldMapper.java:561)
       app//org.opensearch.index.mapper.FieldMapper.parse(FieldMapper.java:270)
       app//org.opensearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:522)
       app//org.opensearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:540)
       app//org.opensearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:442)
       app//org.opensearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:414)
       app//org.opensearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:519)
       app//org.opensearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:540)
       app//org.opensearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:442)
       app//org.opensearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:414)
       app//org.opensearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:136)
       app//org.opensearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:91)
       app//org.opensearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:253)
       app//org.opensearch.index.shard.IndexShard.prepareIndex(IndexShard.java:1029)
       app//org.opensearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:986)
       app//org.opensearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:909)
       app//org.opensearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:621)
       app//org.opensearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:467)
       app//org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
       app//org.opensearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:531)
       app//org.opensearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:412)
       app//org.opensearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:124)
       app//org.opensearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:223)
       app//org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:806)
       app//org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
       java.base@17.0.7/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
       java.base@17.0.7/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
       java.base@17.0.7/java.lang.Thread.run(Thread.java:833)

I recently added a new mapping for the indices, am I doing something wrong. Reverting the flat_object mapping change fixed the CPU.
What could have caused the parser to use 100% of CPU ?

{
  "filebeat-custom": {
    "order": 100,
    "index_patterns": [
      "filebeat-*"
    ],
    "settings": {},
    "mappings": {
      "properties": {
        "board": {
          "properties": {
            "request": {
              "type": "object",
              "enabled": false
            },
            "variables": {
              "type": "flat_object"
            },
            "user": {
              "ignore_above": 1024,
              "type": "keyword"
            }
          }
        }
      }
    },
    "aliases": {}
  }
}

Configuration:

Relevant Logs or Screenshots:

2

Thank you @srikar , the flat_object is a new field type and is not yet battle tested, the issue you are highlighting is quite concerning, may I ask you please to open the issue at [1] so it could be addressed?

In addition, could you please share the shape of the data you are ingesting into
flat_object field (having live sample would be the best but may not be feasible because of the privacy concerns if any).

Thank you.

[1] Issues · opensearch-project/OpenSearch · GitHub

3

Hi @srikar

I can only second @reta, it will help us a lot if you can share more details about your data that goes into the flat_object field.

Also opening a ticket (as suggested) will help a lot.

Regards,
Lukáš

4

We have an issue created here: [BUG] High CPU usage with flat_object field type · Issue #12030 · opensearch-project/OpenSearch · GitHub