CCS-query to same cluster is not working

OpenSearch
1

Versions: 2.11.1.

Describe the issue:

After upgrading from OpenSearch 2.4.1 to 2.11.1, CCS search queries to the same cluster stopped working. Remote clusters respond correctly, but the local one gives errors.

Configuration:

opensearch.yml:

path:
  logs: /var/log
cluster:
  name: cluster-1
  routing:
    allocation:
      disk:
        watermark:
          low: 85%
          high: 90%
          flood_stage: 95%
        threshold_enabled: true
      node_initial_primaries_recoveries: 20
  max_shards_per_node: 1000
indices:
  query:
    bool:
      max_clause_count: 4096
  breaker:
    total:
      limit: 90%
      use_real_memory: false
    request:
      limit: 60%
  fielddata:
    cache:
      size: 10%
network:
  host: 0.0.0.0
plugins:
  security:
    ssl:
      http:
        enabled: true
        pemkey_filepath: esnode-key.pem
        pemcert_filepath: esnode.pem
        enabled_protocols:
          - TLSv1.3
          - TLSv1.2
          - TLSv1.1
        pemtrustedcas_filepath: cas.pem
      transport:
        enabled: true
        pemkey_filepath: esnode-key.pem
        pemcert_filepath: esnode.pem
        enabled_protocols:
          - TLSv1.3
          - TLSv1.2
          - TLSv1.1
        truststore_filepath: cas.jks
        pemtrustedcas_filepath: cas.pem
        enforce_hostname_verification: false
    audit:
      type: log4j
      config:
        log4j:
          level: TRACE
          logger_name: sgaudit
    authcz:
      admin_dn: [...]
    restapi:
      roles_enabled:
        - all_access
        - security_rest_api_access
    nodes_dn: [...]
    ssl_cert_reload_enabled: true
    allow_unsafe_democertificates: false
    allow_default_init_securityindex: true
    enable_snapshot_restore_privilege: true
    check_snapshot_restore_write_privileges: true
bootstrap:
  memory_lock: true
discovery:
  type: single-node
compatibility:
  override_main_response_version: true
transport:
  port: 9300
http:
  port: 9200
node:
  attr:
    speed: hot
  name: cluster-1-hot-1
  roles:
    - data
    - master
    - ingest
    - remote_cluster_client
  max_local_storage_nodes: 1

security/config.yml:

config:
  dynamic:
    http:
      anonymous_auth_enabled: false
      xff:
        enabled: false
    authc:
      basic_internal_auth_domain:
        description: "Authenticate via HTTP Basic against internal users database"
        http_enabled: true
        transport_enabled: true
        order: 1
        http_authenticator:
          type: basic
          challenge: true

Relevant Logs or Screenshots:

Error in opensearch-plain.log:

[2024-02-21T09:28:25,837][ERROR][org.opensearch.security.filter.SecurityFilter] No user found for indices:data/read/search from 192.168.0.1:58732 REST via transport {_opendistro_security_initial_action_class_header=SearchRequest, _opendistro_security_origin_header=REST, _opendistro_security_remotecn=cluster-1}

192.168.0.1 - address of cluster-1

2

did you also try version 2.12? it may have been fixed

3

Unfortunately, I can’t update the version right now and want to know possible workarounds or fix errors in my configuration.

4

Since 2.9 we have Adds a check to skip serialization-deserialization if request is for same node by DarshitChanpura · Pull Request #2765 · opensearch-project/security · GitHub.

Solution is:

  1. Add to config
plugins.security.unsupported.passive_intertransport_auth_initially: true

  1. Create user with role opendistro_security_default_transport_backendrole and indices:data/read/search

  2. Optionally add

logger.custom_security_filter.name = org.opensearch.security.filter
logger.custom_security_filter.level = error

to log4j config (or in log will be many warnings like ‘Transport auth in passive mode and no user found. Injecting default user’)

UPD One more solution:
add dynamic.disable_intertransport_auth: true to security config.yml