Multi index issue with shards

Hi everyone,

I have a problem with my dev tenant…

I have 2 wazuh manager separate (on different network), they have both one instance of filebeat installed with wazuh-template.json (cf. picture 1) as pattern, on wazuh-template.json, i have add order 0 on pattern_1 et order 1 on pattern_2, I also have try with both order 0.

I have custom the name of the index like on the wazuh documentation (I also have custom the manifest.yml, there is not the * on the real file, I miss this one, but it’s just a demo for this topic), all work fine until 00:00, when the index rotate to the next day, one of my index is perfectly created (with 3 shards on one elasticsearch node) but the other have an issue, one shard is created (number 0) and another 0 who is a replication (cf. picture 3)

I have the same problem with the security-audit log…

I don’t have the problem on the first day with 2 indexs…

Picture 1 - I can’t show you a picture, so there is the content

{
  "order": 0,
  "index_patterns": ["pattern-1-*"],
  "settings": {
    "index.refresh_interval": "5s",
    "index.number_of_shards": "3",
    "index.number_of_replicas": "0",
    "index.auto_expand_replicas": "0-1",
    "index.mapping.total_fields.limit": 10000,
    "index.query.default_field": [
      "GeoLocation.city_name",
      "GeoLocation.continent_code",
      "GeoLocation.country_code2",
etc......

Picture 2 - I can’t show you a picture, so there is the content

module_version: 0.1

var:
  - name: paths
    default:
      - /var/ossec/logs/alerts/alerts.json
  - name: index_prefix
    default: pattern-1-

input: config/alerts.yml

ingest_pipeline: ingest/pipeline.json

Picture 3 - I can’t show you a picture, so there is the content

index                    shard prirep state      docs   store ip       node
pattern1-alerts-2021.09.08 0     p      STARTED     726 729.4kb 10.0.2.2 elasticsearch
pattern1-alerts-2021.09.08 0     r      UNASSIGNED                       
pattern1-alerts-2021.09.07 2     p      STARTED    3395   3.2mb 10.0.2.2 elasticsearch
pattern1-alerts-2021.09.07 1     p      STARTED    3326   3.2mb 10.0.2.2 elasticsearch
pattern1-alerts-2021.09.07 0     p      STARTED    3368   3.4mb 10.0.2.2 elasticsearch
pattern2-alerts-2021.09.08 2     p      STARTED     369 624.9kb 10.0.2.2 elasticsearch
pattern2-alerts-2021.09.08 1     p      STARTED     352 600.7kb 10.0.2.2 elasticsearch
pattern2-alerts-2021.09.08 0     p      STARTED     402 675.3kb 10.0.2.2 elasticsearch
pattern2-alerts-2021.09.07 2     p      STARTED     984   1.5mb 10.0.2.2 elasticsearch
pattern2-alerts-2021.09.07 1     p      STARTED     916   1.4mb 10.0.2.2 elasticsearch
pattern2-alerts-2021.09.07 0     p      STARTED    1012   1.4mb 10.0.2.2 elasticsearch

I have filtred some informations (but the index name are the same on my elasticsearch)

Sorry for my bad english…

Thank you,
Simon

You have two different settings for replicas. Choose one and see if it fixes the issue. The logs may also have some info when the index is created as to what it is doing.

I forget to specify it, but here you can see my other replica…

Thanks @tony I will try your solution

index                                              shard prirep state      docs   store ip       node
security-auditlog-2021.09.08                       0     p      STARTED     100 165.3kb 10.0.2.2 elasticsearch
security-auditlog-2021.09.08                       0     r      UNASSIGNED                       
.kibana_-41519569_pattern1user_1                     0     p      STARTED       2   8.9kb 10.0.2.2 elasticsearch
pattern2-alerts-2021.09.08                           2     p      STARTED     791   1.2mb 10.0.2.2 elasticsearch
pattern2-alerts-2021.09.08                           1     p      STARTED     786   1.2mb 10.0.2.2 elasticsearch
pattern2-alerts-2021.09.08                           0     p      STARTED     845   1.3mb 10.0.2.2 elasticsearch
.kibana_294056114_pattern2tenant_1                   0     p      STARTED       3  19.5kb 10.0.2.2 elasticsearch
.kibana_-1286903282_pattern1tenant_1                 0     p      STARTED       5  43.4kb 10.0.2.2 elasticsearch
.opendistro-ism-managed-index-history-2021.09.08-1 0     p      STARTED                 10.0.2.2 elasticsearch
.opendistro-ism-managed-index-history-2021.09.08-1 0     r      UNASSIGNED                       
.kibana_-902287808_simonv_1                        0     p      STARTED       2  15.3kb 10.0.2.2 elasticsearch
pattern1-alerts-2021.09.07                           2     p      STARTED    3395   3.2mb 10.0.2.2 elasticsearch
pattern1-alerts-2021.09.07                           1     p      STARTED    3326   3.2mb 10.0.2.2 elasticsearch
pattern1-alerts-2021.09.07                           0     p      STARTED    3368   3.4mb 10.0.2.2 elasticsearch
pattern2-alerts-2021.09.07                           2     p      STARTED     984   1.5mb 10.0.2.2 elasticsearch
pattern2-alerts-2021.09.07                           1     p      STARTED     916   1.4mb 10.0.2.2 elasticsearch
pattern2-alerts-2021.09.07                           0     p      STARTED    1012   1.4mb 10.0.2.2 elasticsearch
.kibana_1                                          0     p      STARTED     183 189.1kb 10.0.2.2 elasticsearch
.opendistro-ism-config                             0     p      STARTED                 10.0.2.2 elasticsearch
.opendistro-ism-config                             0     r      UNASSIGNED                       
.opendistro-job-scheduler-lock                     0     p      STARTED       4  30.9kb 10.0.2.2 elasticsearch
.opendistro-job-scheduler-lock                     0     r      UNASSIGNED                       
.opendistro-reports-instances                      0     p      STARTED       1   5.9kb 10.0.2.2 elasticsearch
.opendistro-reports-definitions                    0     p      STARTED       1   4.8kb 10.0.2.2 elasticsearch
.opendistro_security                               0     p      STARTED       9  74.2kb 10.0.2.2 elasticsearch

Thank you,
Simon

@simonverbois You also haven’t mentioned the size of the cluster ( x nodes) as this can affect your replicas as well.

It’s a mono node elasticsearch, I don’t understand why he created these replication…

You solution work,I have remove :

"index.auto_expand_replicas": "0-1",

Now my index is green, with one exception, he create these replica…

index                                              shard prirep state      docs   store ip       node
.opendistro_security                               0     p      STARTED       9  74.2kb 10.0.2.2 elasticsearch
.kibana_-41519569_pattern1user_1                     0     p      STARTED       2   8.9kb 10.0.2.2 elasticsearch
.opendistro-ism-managed-index-history-2021.09.08-1 0     p      STARTED                 10.0.2.2 elasticsearch
.opendistro-ism-managed-index-history-2021.09.08-1 0     r      UNASSIGNED                       
pattern1-alerts-2021.09.07                           2     p      STARTED    3395   3.2mb 10.0.2.2 elasticsearch
pattern1-alerts-2021.09.07                           1     p      STARTED    3326   3.2mb 10.0.2.2 elasticsearch
pattern1-alerts-2021.09.07                           0     p      STARTED    3368   3.4mb 10.0.2.2 elasticsearch
.opendistro-reports-definitions                      0     p      STARTED       1   4.8kb 10.0.2.2 elasticsearch
.kibana_294056114_pattern2tenant_1                     0     p      STARTED       3  19.5kb 10.0.2.2 elasticsearch
pattern2-alerts-2021.09.07                           2     p      STARTED     984   1.5mb 10.0.2.2 elasticsearch
pattern2-alerts-2021.09.07                           1     p      STARTED     916   1.4mb 10.0.2.2 elasticsearch
pattern2-alerts-2021.09.07                           0     p      STARTED    1012   1.4mb 10.0.2.2 elasticsearch
pattern1-alerts-2021.09.08                           2     p      STARTED       6  41.4kb 10.0.2.2 elasticsearch
pattern1-alerts-2021.09.08                           2     r      UNASSIGNED                       
pattern1-alerts-2021.09.08                           1     p      STARTED      11  49.5kb 10.0.2.2 elasticsearch
pattern1-alerts-2021.09.08                           1     r      UNASSIGNED                       
pattern1-alerts-2021.09.08                           0     p      STARTED       4  23.7kb 10.0.2.2 elasticsearch
pattern1-alerts-2021.09.08                           0     r      UNASSIGNED                       
.kibana_1                                            0     p      STARTED     198 187.6kb 10.0.2.2 elasticsearch
.kibana_-1286903282_pattern1tenant_1                   0     p      STARTED       5  43.4kb 10.0.2.2 elasticsearch 
.opendistro-reports-instances                        0     p      STARTED       1   5.9kb 10.0.2.2 elasticsearch
.kibana_-902287808_simonv_1                          0     p      STARTED       2  15.3kb 10.0.2.2 elasticsearch
pattern2-alerts-2021.09.08                           2     p      STARTED     791   1.2mb 10.0.2.2 elasticsearch
pattern2-alerts-2021.09.08                           1     p      STARTED     786   1.2mb 10.0.2.2 elasticsearch
pattern2-alerts-2021.09.08                           0     p      STARTED     845   1.3mb 10.0.2.2 elasticsearch

I don’t have specify anywhere to elasticsearch a second node…

I have this settings in elasticsearch.yml one of them can be the reason of this trouble ?

node.name: elasticsearch

cluster.initial_master_nodes: elasticsearch

cluster.routing.allocation.disk.threshold_enabled: false

node.max_local_storage_nodes: 3

I would change node.max_local_storage_nodes to 1.