About python client library and elasticsearch DSL support

Open Source Elasticsearch and Kibana
1

Hi all

I am using elasticsearch oss but now shifting towards open distro for elasticsearch. I have installed and configure kibana. but i am confuse about ingestion data to elasticsearch using python client libraries.
Can we use Python Elasticsearch Client or elasticsearch dsl library with open distro version.?
If not then what would be the alternation solution for writing python scripts with open distro version?

Thank you in advance…

1 Like
2

Hi

I did not do any big implementation in Python but I did with the official PERL Elasticsearch Client. It worked for me (both ODFE 1.10 and 1.12) without an issue . I assume it will work for any client as long as you don’t use any X-PACK features.

1 Like
3

@igorid70 What if we are using basic features from security plugin of open distro specially default settings?

4

I really don’t know - you need to try and see how it works. I have different roles and users in Opendistro security plug-in and the user which connects and activates Elasticsearch APIs is a user which was defined with the permissions set similar to built-in user admin and as I said, it worked smoothly. When I tried to activate the same process which connects and deletes old indices based on age/size criteria with user which does not have the respective permissions, the connect method immediately failed with HTTP403, so, it really works as expected. My gut feeling - it will work for you but of course you need to test it.

5

I’m not aware of any problems with the client libraries on Open Distro. From the perspective of your client library, it’s sending HTTP requests with Query DSL payloads to the server.

If you’re running DSL queries that are specific to security (e.g. updating role, permissions, or something similar) then yes, those would have a different set of DSL queries. You can read more about this in our docs. But for normal ingest this isn’t an issue.

As @igorid70 mentions, you’ll have to try it out, but I doubt you’ll have issues. If you do, just come back here and perhaps we can help you.

6

Hi @searchymcsearchface
Open distro for elasticsearch has its own security plugin. so due to that we have some default settings for this plugin included in elasticsearch.yml… To establish successful connection with elasticsearch server we need to provide basic authentication credentials(admin:admin). If we are using Python Elasticsearch Client for establishing connection with elasticsearch server(open distro version)will not work(Protocol not found error). So do we need to disable security plugin for using python library?

7

@igorid70 @searchymcsearchface
Yes. We can use python elasticsearch client library with open distro Elasticsearch by disabling security plugins. I have tried that and worked for me.

8

Oh on! Don’t disable security. It’s really important - elasticsearch-py will definitely work with security.

I’m not a python master, but this works and should get you started.

import ssl
from elasticsearch import Elasticsearch
from elasticsearch.connection import create_ssl_context

def main():
   open_distro_ssl_context = create_ssl_context()
   # next two lines are if you're running localhost with a self-signed cert (aka docker)
   open_distro_ssl_context.check_hostname = False
   open_distro_ssl_context.verify_mode = ssl.CERT_NONE

   es = Elasticsearch(
        scheme="https",
        hosts=[ { 'port': 9200, 'host': 'localhost' } ],
        ssl_context=open_distro_ssl_context,
        http_auth=("username", "password") # replace with your username then password
   )
   foo = es.get(index="myindex", id="mydoc")
   print(foo)


if __name__ == '__main__':
    main()
1 Like
9

Hi @searchymcsearchface (kyle :blush:)

Thank you very much for your efforts.I could not make it work.
I am using RPM package of ODFE. I am running my scripts from remote machine. I tried as per your suggestion.
elasticsearch.exceptions.ConnectionError: ConnectionError(<urllib3.connection.HTTPSConnection object at 0x7f3d0734c198>: Failed to establish a new connection: [Errno -2] Name or service not known) caused by: NewConnectionError(<urllib3.connection.HTTPSConnection object at 0x7f3d0734c198>: Failed to establish a new connection: [Errno -2] Name or service not known)
Trying to resolve this error. Firewall is not running.

1 Like
10

Resolved that issue with few minor changes.

Thank you !!

1 Like
11

Hi @pranali4796,

Can you please let me know the fix you did.

I am too trying to connect to ODFE and I m getting following error.

:elasticsearch.exceptions.UnsupportedProductError: The client noticed that the server is not a supported distribution of Elasticsearch"

using the same code provided by @searchymcsearchface

import ssl
from elasticsearch import Elasticsearch
from elasticsearch.connection import create_ssl_context

def main():
   host='myhost'
   open_distro_ssl_context = create_ssl_context()
   open_distro_ssl_context.check_hostname = False
   open_distro_ssl_context.verify_mode = ssl.CERT_NONE

   es = Elasticsearch(
        scheme="https",
        hosts=[ { 'port': 9200, 'host': host } ],
        ssl_context=open_distro_ssl_context,
        http_auth=('user', 'password'),
        verify_certs=False
   )
   foo = es.get(index="indexname", id="doc")
   print(foo)


if __name__ == '__main__':
    main()

Appreciate any help on the

12

Hey @Naushad79. I would point you this this blog post:

Essentially, Elastic has placed code checking logic into the client libraries to block the current version of the clients from connection to OpenSearch and OSS Elasticsearch (hence Open Distro). You can use a past version that lacks the checking logic (7.13.x) or use the OpenSearch client (under development, should be production ready within a few weeks).