Rolling upgrade - opendistro_security 0.X to 1.X

Hey folks,

Just checking if there is a supported rolling upgrade path from opendistro-security plugin 0.X to 1.X ?

Some similar queries posted on the forums here, but I haven’t seen a definitive answer here or in the documentation -

For my specific query, we’re not running the full Opendistro suite of plugins, but are using the Opendistro Security plugin to complement elasticsearch-oss and provide LDAP auth functionality .

Current version: elasticsearch-oss v.6.8.1, opendistro-security 0.10.0.0-1
Target version: elasticsearch-oss v 7.10.2 , opendistro-security 1.13.1.0-1

Running a 5 node cluster - attempted a rolling upgrade, starting with a single node first . After upgrade (elasicsearch & security plugin) , the upgraded node rejoins the cluster OK, and Opendistro security is initialized on the node OK (using the existing .opendistro_security index)

LDAP auth seems to continue to work OK (tested by making requests to the upgraded node directly) , but native authentication using the internal users database does not work on the upgraded node (it still works fine on the non upgraded nodes). Local auth fails (order 1) and falls back to LDAP (order 2), but the internal users don’t exist in the LDAP directory of course.

Reluctant to proceed further as applications interacting with the cluster use the internal users DB . Disabling the plugin temporarily is not an option either, as it’s required for TLS communication between nodes.

I expect a full cluster shutdown followed by an upgrade on each node, and recreation of the .opendistro_security index will result in a successful upgrade - but I’d like to avoid a shutdown if possible.

Any pointers appreciated!

Cheers

1 Like

Hi @lechuck
I managed to migrate however it was rather a messy process, I will describe below, hoping it helps.

  1. Install ES6 OSS + odfe security plugin on the nodes (probably current state)

  2. Disable shard allocation.

    PUT _cluster/settings
    {
    “persistent”: {
    “cluster.routing.allocation.enable”: “primaries”
    }
    }

  3. perform synced flush
    POST _flush/synced

    OPTIONAL: Temporarily stop the tasks associated with active machine learning jobs and datafeeds.
    POST _ml/set_upgrade_mode?enabled=true

  4. Download securityadmin.sh tool from new version of odfe plugin (1.4) and backup config.
    ./tools/securityadmin.sh -backup “./sgadmin7” -icl -key “…/config/kirk-key.pem” -cert “…/config/kirk.pem” -cacert “…/config/root-ca.pem” -nhnv

  5. Verify the exported files:
    ./tools/securityadmin.sh -vc 6 -cd “./sgadmin7” -icl -key “…/config/kirk-key.pem” -cert “…/config/kirk.pem” -cacert “…/config/root-ca.pem” -nhnv
    (fix any errors- if any)

  6. Perform offline migration to new versions of odfe config using below command:

./securityadmin.sh -mo "./securityadmin_config" -icl -key ".../config/kirk-key.pem" -cert ".../config/kirk.pem" -cacert ".../config/root-ca.pem" -nhnv -h <host>

This should produce new files and mark old as backup.

6.1 In my case whitelist.yml doesn’t get created and nodes_dn.yml is empty, add following:

whitelist.yml:
— (<- this is - - -)
_meta:
type: “whitelist”
config_version: 2

nodes_dn.yml

_meta:
type: “nodesdn”
config_version: 2

audit.yml (I had to update audit.yml file as below to avoid future issues)

_meta:
  type: "audit"
  config_version: 2

config:
  # enable/disable audit logging
  enabled: true

  audit:
    # Enable/disable REST API auditing
    enable_rest: true

    # Categories to exclude from REST API auditing
    disabled_rest_categories:
      - AUTHENTICATED
      - GRANTED_PRIVILEGES

    # Enable/disable Transport API auditing
    enable_transport: true

    # Categories to exclude from Transport API auditing
    disabled_transport_categories:
      - AUTHENTICATED
      - GRANTED_PRIVILEGES

    # Users to be excluded from auditing. Wildcard patterns are supported. Eg:
    # ignore_users: ["test-user", "employee-*"]
    ignore_users:
      - kibanaserver

    # Requests to be excluded from auditing. Wildcard patterns are supported. Eg:
    # ignore_requests: ["indices:data/read/*", "SearchRequest"]
    ignore_requests: []

    # Log individual operations in a bulk request
    resolve_bulk_requests: false

    # Include the body of the request (if available) for both REST and the transport layer
    log_request_body: true

    # Logs all indices affected by a request. Resolves aliases and wildcards/date patterns
    resolve_indices: true

    # Exclude sensitive headers from being included in the logs. Eg: Authorization
    exclude_sensitive_headers: true

  compliance:
    # enable/disable compliance
    enabled: true

    # Log updates to internal security changes
    internal_config: true

    # Log external config files for the node
    external_config: false

    # Log only metadata of the document for read events
    read_metadata_only: true

    # Map of indexes and fields to monitor for read events. Wildcard patterns are supported for both index names and fields. Eg:
    # read_watched_fields: {
    #   "twitter": ["message"]
    #   "logs-*": ["id", "attr*"]
    # }
    read_watched_fields: {}

    # List of users to ignore for read events. Wildcard patterns are supported. Eg:
    # read_ignore_users: ["test-user", "employee-*"]
    read_ignore_users:
      - kibanaserver

    # Log only metadata of the document for write events
    write_metadata_only: true

    # Log only diffs for document updates
    write_log_diffs: false

    # List of indices to watch for write events. Wildcard patterns are supported
    # write_watched_indices: ["twitter", "logs-*"]
    write_watched_indices: []

    # List of users to ignore for write events. Wildcard patterns are supported. Eg:
    # write_ignore_users: ["test-user", "employee-*"]
    write_ignore_users:
      - kibanaserver

roles.yml file contains system roles which can not be present as they are now built it. remove them, these are:

  • manage_snapshots
  • logstash
  • kibana_user
  • own_index
  • all_access
  • kibana_server
  • kibanaserver
  • readall

if you didn’t have any custom roles defined previously you should only have:

---
_meta:
  type: "roles"
  config_version: 2
  1. stop the old ES and start the new ES with new plugin installed one by one, make sure to point the path.data in elasticsearch.yml to the old directory to ensure data persistence, if not using shared volumes.

  2. once all nodes are upgrades and connected to cluster, delete the old index .opendistro_security with below command:
    ./securityadmin.sh -icl -key ".../config/kirk-key.pem" -cert ".../config/kirk.pem" -cacert ".../config/root-ca.pem" -nhnv -h <host> -dci

  3. Enable shard allocation:

    PUT _cluster/settings
    {
    “persistent”: {
    “cluster.routing.allocation.enable”: null
    }
    }

  4. Upload the new config with below command:
    ./securityadmin.sh -cd "./securityadmin_config" -icl -key ".../config/kirk-key.pem" -cert ".../config/kirk.pem" -cacert ".../config/root-ca.pem" -nhnv -h <host>

You should be Done with success

I’m a bit late, but thank you very much for the detailed response! Much appreciated.