Opensearch connector is not working with apache flink

OpenSearch
1

This is the Apache flink SQL query to create the dynamic table in the flink

    report_type_os_sink_sql = """
        CREATE TABLE report_type_sink (
                          `id` INTEGER,
                          `uuid` STRING,
                          `value` STRING,
                          `label` STRING,
                          `es_index_name` STRING,
                          `sequence_number` INTEGER,
                          `created_at` TIMESTAMP,
                          `updated_at` TIMESTAMP,
                          PRIMARY KEY(id) NOT ENFORCED
                      ) WITH (
                            'connector' = 'opensearch',
                            'hosts' = 'https://search-dev-000-reports-gizgrvlkkvi34pijqesj7fe5ju.us-east-1.es.amazonaws.com:9200',
                            'index' = 'report_type_sink_flink',
                            'username' = '<username-here>',
                            'password' = '<password-here>'
                      );
        """

I am using this jar file : flink-sql-connector-opensearch-1.1.0-1.18.jar

But while running it, Neither I am getting any error nor it is able to connect.
Even when I am giving wrong hosts/username/password then also it is not throwing any error.

Why doing the sink via postgres-cdc connector from postgres to postgres for the same table, it is working fine, but from postgres to opensearch this opensearch connector is not working properly, can anyone help me out here. What wrong here in the above configuration. I am not able to solve it.

Thanks

2

It is hard to say what is happening without seeing the error cause, you may try to add allow-insecure option [1], as I can tell, you are connecting to dev cluster over HTTPS.

[1] Opensearch | Apache Flink

3

I have already tried with allow-insecure option, but still it is not able to sink. The main problem here is It is not throwing any error, so I am not able to figure out what is the problem.

Even if I am giving hosts option as : http:abcd:4565, then also it is not throwing any error, but doing nothing.

But whenever I am giving the wrong options for example : instead of hosts, if I give host, then it is throwing error that host is not supported, by this at least it is clear that it knows what option to provide so jar file is not the problem.

4

The way I would suggest to troubleshoot it is to try the SQL client cli [1], it should report any issues promptly to help you find the cause.

[1] SQL Client | Apache Flink

5

Thanks @reta I tried with SQS client cli. There also it was not throwing any error, but on the flink CLI, I got to know what the error was, it was a parsing related error for the date data type.

At least I am able to see the errors.

Thanks a lot one again.

1 Like