Creating TypeMapping from inputStream not working

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

Describe the issue:

Using the Java client I want my app to be able create and configure the index. I want to load an explicit mapping from file. The simple problem is, I don’t seem to be able to create the TypeMapping object properly - it always looks empty and my index is created with dynamic mapping.

I’ve tried creating the transport with and without a provided mapper, I’ve even tried adding additional modules, which I think are unnecessary.

    ObjectMapper mapper = JsonMapper.builder()
      .addModule(new Jdk8Module())
      .addModule(new JavaTimeModule())
      .build();

    // Create Transport
    transport = ApacheHttpClient5TransportBuilder.builder(host)
      .setHttpClientConfigCallback(httpClientBuilder -> {
        final PoolingAsyncClientConnectionManager connectionManager = PoolingAsyncClientConnectionManagerBuilder
            .create()
            .build();

        if (credentialsProvider != null) {
          httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider);
        }

        return httpClientBuilder.setConnectionManager(connectionManager);
      })
      .setMapper(new JacksonJsonpMapper(mapper))
      .build();

    client = new OpenSearchClient(transport);

I then try to create the index with a TypeMapping, parsing my mapping.json from a valid inputStream:

      JsonpMapper mapper = client._transport().jsonpMapper();
      JsonProvider provider = mapper.jsonProvider();
      JsonParser parser = provider.createParser(getMapping());
      TypeMapping mapping = new TypeMapping.Builder().withJson(parser, mapper).build();

     
      CreateIndexRequest createIndexRequest = new CreateIndexRequest.Builder().index(site.getIdentifier())
          .mappings(mapping).build();
      CreateIndexResponse createIndexResponse = client.indices().create(createIndexRequest);

Examing the mapping in the debugger shows it has no properties. I tested with a small example mappings.json from the docs with no luck. I get no errors; adding a deliberate error into mapping.json made no difference

Creating the index using curl -X PUT /my-index -d @mapping.json works

Configuration

  "version" : {
    "distribution" : "opensearch",
    "number" : "2.18.0",
    "build_type" : "tar",
    "build_hash" : "99a9a81da366173b0c2b963b26ea92e15ef34547",
    "build_date" : "2024-10-31T19:08:39.157471098Z",
    "build_snapshot" : false,
    "lucene_version" : "9.12.0",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },

java 11, OSGI application

opensearch-java 2.26.0 - embedded in osgi bundle

parson 1.1.7 - embedded in osgi bundle

jackson 2.17.3

FYI I’ve tried this but still no luck:

TypeMapping mapping = new TypeMapping.Builder().withJson(getMapping()).build();

@jperrin, could you share an example of your mapping.json?

@pablo hi, thanks for your interest. As mentioned, I’ve tested with a small example mapping.json from the docs and that does nothing either. For completeness, this is the mapping I’m using, which works fine if I submit it using curl.

{
    "mappings" : {
        "_meta": {
            "version": "5000"
        },
        "properties" : {
            "doc_type": { "type" : "keyword" },
            "uid": { "type" : "keyword" },

            "id": { "type" : "keyword" },
            "path": { "type" : "keyword" },
            "type": { "type" : "keyword" },
            "version": { "type" : "long" },
            "alternate_version": { "type" : "long" },

            "origin": { "type" : "keyword" },
            "original_identifier": { "type" : "keyword" },
            "subjects": { "type" : "keyword" },
            "template": { "type" : "keyword" },
            "flavor": { "type" : "keyword" },
            "stationary": { "type" : "keyword" },
            "series": { "type" : "keyword" },

            "page_xml": { "type" : "text", "index" : false },
            "page_header_xml": { "type" : "text", "index" : false },
            "preview_xml": { "type" : "text", "index" : false },

            "owned_by": { "type" : "keyword" },
            "owned_by_name": { "type" : "keyword" },
            "created": { "type" : "date" },
            "created_by": { "type" : "keyword" },
            "created_by_name": { "type" : "keyword" },
            "modified": { "type" : "date" },
            "modified_by": { "type" : "keyword" },
            "modified_by_name": { "type" : "keyword" },
            "published_from": { "type" : "date" },
            "published_to": { "type" : "date" },
            "published_by": { "type" : "keyword" },
            "published_by_name": { "type" : "keyword" },

            "locked_by": { "type" : "keyword" },
            "locked_by_name": { "type" : "keyword" },

            "description": { "type" : "text", "index": false },
            "coverage": { "type" : "text", "index": false },
            "rights": { "type" : "text", "index": false },
            "title": { "type" : "text", "index": false },

            "pagelet_type": { "type" : "keyword" },
            "pagelet_contents": { "type" : "text", "index": false },
            "pagelet_properties": { "type" : "keyword" },
            "pagelet_property_value": { "type" : "keyword" },

            "path_prefix": { "type" : "keyword" },
            "text": { "type" : "text", "index": false },
            "text_fuzzy": { "type" : "text" },
            "fulltext": { "type" : "text", "index": false },
            "fulltext_fuzzy": { "type" : "text" },

            "content_source": { "type" : "text", "index": false },
            "content_external_representation": { "type" : "text", "index": false },
            "content_mimetype": { "type" : "text", "index": false },
            "content_filename": { "type" : "text", "index": false }
        },
        "dynamic_templates" : [
        {
            "allowdeny_allow_template": {
                "match": "allowdeny_allow_*",
                "match_mapping_type": "string",
                "mapping": {
                    "type": "text",
                    "index": false
                }
            }
        },
        {
            "description_template": {
                "match": "description_*",
                "match_mapping_type": "string",
                "mapping": {
                    "type": "text",
                    "index": false
                }
            }
        },
        {
            "coverage_template": {
                "match": "coverage_*",
                "match_mapping_type": "string",
                "mapping": {
                    "type": "text",
                    "index": false
                }
            }
        },
        {
            "rights_template": {
                "match": "rights_*",
                "match_mapping_type": "string",
                "mapping": {
                    "type": "text",
                    "index": false
                }
            }
        },
        {
            "title_template": {
                "match": "title_*",
                "match_mapping_type": "string",
                "mapping": {
                    "type": "text",
                    "index": false
                }
            }
        },
        {
            "pagelet_type_composer": {
                "match": "pagelet_type_composer_*",
                "match_mapping_type": "string",
                "mapping": {
                    "type": "text",
                    "index": false
                }
            }
        },
        {
            "pagelet_type_position": {
                "match": "pagelet_type_position_*",
                "match_mapping_type": "string",
                "mapping": {
                    "type": "text",
                    "index": false
                }
            }
        },
        {
            "pagelet_type_composer_position": {
                "match": "pagelet_type_composer_*_position_*",
                "match_mapping_type": "string",
                "mapping": {
                    "type": "text",
                    "index": false
                }
            }
        },
        {
            "pagelet_contents_language": {
                "match": "pagelet_contents_language_*",
                "match_mapping_type": "string",
                "mapping": {
                    "type": "text",
                    "index": false
                }
            }
        },
        {
            "pagelet_contents_composer": {
                "match": "pagelet_contents_composer_*",
                "match_mapping_type": "string",
                "mapping": {
                    "type": "text",
                    "index": false
                }
            }
        },
        {
            "pagelet_contents_position": {
                "match": "pagelet_contents_position_*",
                "match_mapping_type": "string",
                "mapping": {
                    "type": "text",
                    "index": false
                }
            }
        },
        {
            "pagelet_xml": {
                "match": "pagelet_xml_*",
                "match_mapping_type": "string",
                "mapping": {
                    "type": "text",
                    "index": false
                }
            }
        },
        {
            "content_mimetype": {
                "match": "content_mimetype_*",
                "match_mapping_type": "string",
                "mapping": {
                    "type": "text",
                    "index": false
                }
            }
        },
        {
            "content_filename": {
                "match": "content_filename_*",
                "match_mapping_type": "string",
                "mapping": {
                    "type": "text",
                    "index": false
                }
            }
        },
        {
            "content_created": {
                "match": "content_created_*",
                "match_mapping_type": "date",
                "mapping": {
                    "type": "date",
                    "index": false
                }
            }
        },
        {
            "content_created_by": {
                "match": "content_created_by_*",
                "match_mapping_type": "string",
                "mapping": {
                    "type": "text",
                    "index": false
                }
            }
        },
        {
            "content_xml": {
                "match": "content_xml_*",
                "match_mapping_type": "string",
                "mapping": {
                    "type": "text",
                    "index": false
                }
            }
        },
        {
            "text": {
                "match": "text_*",
                "match_mapping_type": "string",
                "mapping": {
                    "type": "text"
                }
            }
        },
        {
            "fulltext": {
                "match": "fulltext_*",
                "match_mapping_type": "string",
                "mapping": {
                    "type": "text"
                }
            }
        }
        ]
    }
}

This part is adding mappings. However, in your JSON file you’re wrapping with mappings. As a result you will get mappings.mappings, which is incorrect.

Like running your curl with the following.

curl -X PUT /my-index/mappings -d @mapping.json

Try removing mappings wrapper.

{

        "properties" : {
            "doc_type": { "type" : "keyword" },
            "uid": { "type" : "keyword" },

            "id": { "type" : "keyword" },
            "path": { "type" : "keyword" },
            "type": { "type" : "keyword" },
            "version": { "type" : "long" },
            "alternate_version": { "type" : "long" },

            "origin": { "type" : "keyword" },
            "original_identifier": { "type" : "keyword" },
            "subjects": { "type" : "keyword" },
            "template": { "type" : "keyword" },
            "flavor": { "type" : "keyword" },
            "stationary": { "type" : "keyword" },
            "series": { "type" : "keyword" },

            "page_xml": { "type" : "text", "index" : false },
            "page_header_xml": { "type" : "text", "index" : false },
            "preview_xml": { "type" : "text", "index" : false },

            "owned_by": { "type" : "keyword" },
            "owned_by_name": { "type" : "keyword" },
            "created": { "type" : "date" },
            "created_by": { "type" : "keyword" },
            "created_by_name": { "type" : "keyword" },
            "modified": { "type" : "date" },
            "modified_by": { "type" : "keyword" },
            "modified_by_name": { "type" : "keyword" },
            "published_from": { "type" : "date" },
            "published_to": { "type" : "date" },
            "published_by": { "type" : "keyword" },
            "published_by_name": { "type" : "keyword" },

            "locked_by": { "type" : "keyword" },
            "locked_by_name": { "type" : "keyword" },

            "description": { "type" : "text", "index": false },
            "coverage": { "type" : "text", "index": false },
            "rights": { "type" : "text", "index": false },
            "title": { "type" : "text", "index": false },

            "pagelet_type": { "type" : "keyword" },
            "pagelet_contents": { "type" : "text", "index": false },
            "pagelet_properties": { "type" : "keyword" },
            "pagelet_property_value": { "type" : "keyword" },

            "path_prefix": { "type" : "keyword" },
            "text": { "type" : "text", "index": false },
            "text_fuzzy": { "type" : "text" },
            "fulltext": { "type" : "text", "index": false },
            "fulltext_fuzzy": { "type" : "text" },

            "content_source": { "type" : "text", "index": false },
            "content_external_representation": { "type" : "text", "index": false },
            "content_mimetype": { "type" : "text", "index": false },
            "content_filename": { "type" : "text", "index": false }
        },
        "dynamic_templates" : [
        {
            "allowdeny_allow_template": {
                "match": "allowdeny_allow_*",
                "match_mapping_type": "string",
                "mapping": {
                    "type": "text",
                    "index": false
                }
            }
        },
        {
            "description_template": {
                "match": "description_*",
                "match_mapping_type": "string",
                "mapping": {
                    "type": "text",
                    "index": false
                }
            }
        },
        {
            "coverage_template": {
                "match": "coverage_*",
                "match_mapping_type": "string",
                "mapping": {
                    "type": "text",
                    "index": false
                }
            }
        },
        {
            "rights_template": {
                "match": "rights_*",
                "match_mapping_type": "string",
                "mapping": {
                    "type": "text",
                    "index": false
                }
            }
        },
        {
            "title_template": {
                "match": "title_*",
                "match_mapping_type": "string",
                "mapping": {
                    "type": "text",
                    "index": false
                }
            }
        },
        {
            "pagelet_type_composer": {
                "match": "pagelet_type_composer_*",
                "match_mapping_type": "string",
                "mapping": {
                    "type": "text",
                    "index": false
                }
            }
        },
        {
            "pagelet_type_position": {
                "match": "pagelet_type_position_*",
                "match_mapping_type": "string",
                "mapping": {
                    "type": "text",
                    "index": false
                }
            }
        },
        {
            "pagelet_type_composer_position": {
                "match": "pagelet_type_composer_*_position_*",
                "match_mapping_type": "string",
                "mapping": {
                    "type": "text",
                    "index": false
                }
            }
        },
        {
            "pagelet_contents_language": {
                "match": "pagelet_contents_language_*",
                "match_mapping_type": "string",
                "mapping": {
                    "type": "text",
                    "index": false
                }
            }
        },
        {
            "pagelet_contents_composer": {
                "match": "pagelet_contents_composer_*",
                "match_mapping_type": "string",
                "mapping": {
                    "type": "text",
                    "index": false
                }
            }
        },
        {
            "pagelet_contents_position": {
                "match": "pagelet_contents_position_*",
                "match_mapping_type": "string",
                "mapping": {
                    "type": "text",
                    "index": false
                }
            }
        },
        {
            "pagelet_xml": {
                "match": "pagelet_xml_*",
                "match_mapping_type": "string",
                "mapping": {
                    "type": "text",
                    "index": false
                }
            }
        },
        {
            "content_mimetype": {
                "match": "content_mimetype_*",
                "match_mapping_type": "string",
                "mapping": {
                    "type": "text",
                    "index": false
                }
            }
        },
        {
            "content_filename": {
                "match": "content_filename_*",
                "match_mapping_type": "string",
                "mapping": {
                    "type": "text",
                    "index": false
                }
            }
        },
        {
            "content_created": {
                "match": "content_created_*",
                "match_mapping_type": "date",
                "mapping": {
                    "type": "date",
                    "index": false
                }
            }
        },
        {
            "content_created_by": {
                "match": "content_created_by_*",
                "match_mapping_type": "string",
                "mapping": {
                    "type": "text",
                    "index": false
                }
            }
        },
        {
            "content_xml": {
                "match": "content_xml_*",
                "match_mapping_type": "string",
                "mapping": {
                    "type": "text",
                    "index": false
                }
            }
        },
        {
            "text": {
                "match": "text_*",
                "match_mapping_type": "string",
                "mapping": {
                    "type": "text"
                }
            }
        },
        {
            "fulltext": {
                "match": "fulltext_*",
                "match_mapping_type": "string",
                "mapping": {
                    "type": "text"
                }
            }
        }
        ]
    }

Thanks Pablo, that makes sense and fixes the problem.

1 Like