Opendistro Upgrade Reports 1.11 - 1.12 (or previous)

General Feedback
1

I have started planning my 1.11 to 1.12 upgrade. I would like to know how other Opendistro operators are handling the upgrades, and how comfortable operators are doing upgrades.

  1. What plugins / components are you using? Are you using the multi-tenancy? Other features?
  2. What method of installation are you utilising docker? Rpm? Source? Linux/Windows?
  3. How many nodes in the cluster?
  4. How long did you plan the upgrade before upgrading?
  5. How long did it take for you to do the upgrade?
  6. Are you using any tools when doing the upgrade? Shell scripts? Check lists? Ansible?
  7. Have you done any OpenDistros upgrades before how many/which upgrades?
  8. Did you have any planned or unplanned downtime related to the upgrade?
  9. Do you have a devel, test and production environment? How beneficial were they for the upgrade?
  10. Any unexpected hiccups while upgrading?
  11. Any aftermath clean up or issues?
  12. Any bugs in the new version that is affecting your installation? Do you have any GitHub links to any issues that affected your installation?
  13. Any comments about your own operations/processes?
  14. Overall how comfortable are you feeling with ELK/Opendistro?

I’ll answer my own question to get this rolling for my 1.10 - 1.11 upgrade

  1. Kibana, Logstash and opendistro_security with multitenancy.
  2. CentOS-7 Rpm
  3. 10 elastic nodes
  4. <2 weeks
  5. Less than an hour per environment
  6. Checklists + ansible + home brewed python application.
  7. 1.8-1.9, 1.9-1.10, and 1.10-1.11
  8. Kibana was down for ~ 5minutes while restarting. We are running redundant Logstash.
  9. Yes, 2x devel, 1x test 1x production.
  10. Not really. Elasticsearch startup time where not consistent, probably because of different amount of indices/cores per environemnt.
  11. Forgot to reenable a cronjob.
  12. No.
  13. Admins does not want to read the upgrade procedure they have written themselves and does task out of order, no big deal… …I look forward to the day when robots replace admins.
  14. Pretty comfortable, probably a bit cocky I have not lost any data in production… …yet.
2

I’m busy upgrading from think 1.9 to 1.12, my issues so far is mainly with the reporting module.

  1. JWT-auth not working
  2. CSV export for reporting not working but pdf is working
  3. Reporting doesn’t support server.basePath: /kibana
    server.rewriteBasePath: true
  4. I made a helm-chart to deploy a multi tenanted cluster and run it on 3xEC2. t3.large instances, performing very well, aggrigating about 5-10 mil entries a day and doesn’t seem too affected.

Hope it helps