I was wondering if anyone has attempted to load data into an OpenSearch cluster from Spark or other frameworks provided by the
elasticsearch-hadoop libraries but using the AWS message signing to leverage IAM authentication? All of the clients we work with that use OpenSearch in AWS are loading data from Apache Spark and this is currently the only thing stopping them from using IAM auth in place of basic/key based auth.
I found a blog post stating that OpenSearch will at some point release their own version of
elasticsearch-hadoop at which point hopefully the ability to add dynamic headers to each request will be possible, however it seems Elasticsearch are not interested in adding this functionality since there’s been an open Github issue for over 6 years: Append dynamic custom headers for http requests · Issue #626 · elastic/elasticsearch-hadoop · GitHub
I was wondering if anyone had found a nice way to achieve this via the existing Elasticsearch libraries? It seems to me it would be a little bit tricky since there are several levels of classes you’d need to go down to switch out the existing static header functionality which is provided in
I was hoping someone had found some nice workaround for this or has any advice.
Is it common to load data using a special service account which uses basic auth and then do all other client requests from applications using IAM, where it’s much easier to add a http callback to the regular ES Java client? Is it even possible to mix IAM and basic auth on a single cluster?
Thanks in advance,