# Not all Hindi matches are being returned when using custom analyzer

**URL:** <https://forum.opensearch.org/t/not-all-hindi-matches-are-being-returned-when-using-custom-analyzer/22096>\
**Category:** OpenSearch\
**Tags:** troubleshoot, index-management\
**Created:** [October 25, 2024, 7:33am UTC](https://forum.opensearch.org/t/not-all-hindi-matches-are-being-returned-when-using-custom-analyzer/22096 "2024-10-25T07:33:21Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![aakashbashyal21](https://avatars.discourse-cdn.com/v4/letter/a/7bcc69/32.png) [@aakashbashyal21](https://forum.opensearch.org/u/aakashbashyal21)\
**Post date:** [October 25, 2024, 7:33am UTC](https://forum.opensearch.org/t/not-all-hindi-matches-are-being-returned-when-using-custom-analyzer/22096/1 "2024-10-25T07:33:21Z")

</div>

I am working on a `FastAPI` project that uses `OpenSearch` with a custom index to support both Hindi and English text. I created a custom `analyzer` for Hindi, but I am encountering an issue where search results seem incomplete.

Index Creation:

I created an index with a custom analyzer for the `chapter_title` field like this:

```auto
def create_index():
    """Create an OpenSearch index with language-specific analyzers for Hindi and English."""
    if not opensearch_client.indices.exists(INDEX_NAME):
        opensearch_client.indices.create(
            INDEX_NAME,
            body={
                "settings": {
                    "analysis": {
                        "analyzer": {
                            "custom_chapter_title_analyzer": {
                                "type": "custom",
                                "tokenizer": "standard",
                                "filter": [
                                    "hindi_normalization",
                                    "indic_normalization",
                                ],
                            }
                        }
                    }
                },
                "mappings": {
                    "properties": {
                        "book_title": {"type": "text", "analyzer": "english"},
                        "chapter_title": {
                            "type": "text",
                            "analyzer": "custom_chapter_title_analyzer",
                        },
                    }
                },
            },
        )

```

Search Implementation:

For searching, I am using a multi\_match query on both book\_title and chapter\_title fields:

```auto
@app.get("/search/")
def search_books(q: str):
    
    print('\nSearch query:')
    print(q)

    query = {
        'size': 5,
        "query": {
            "multi_match": {
                "query": q,
                "fields": ["book_title", "chapter_title"]
            }
        }
    }
    response = opensearch_client.search_book(query)
    print('\nSearch results:')
    print(response)
    return response

```

Issue:

When searching for the term “संगठन”, I only get results for one of the following `chapter_title` values:

1. संगठनः अनूठा और क्रांतिकारी
2. संगठन और धर्म

I only get a match for “संगठन और धर्म” but not for “संगठनः अनूठा और क्रांतिकारी”.

I suspect this might be due to the : character (the symbol “ः”) being treated differently during tokenization, but I’m unsure how to resolve this.

Like wise:

In search when i have: `केंद्रः`

I get this result as first result:

1. ध्यान-केंद्रः
2. जीवन का गुह्यतम केंद्रः मृत्यु

But when i have only the search query as: `केंद्र`

I didn’t get above result, i get only matching value of `केंद्र`

Question:

How can I adjust my custom analyzer to ensure that search queries return all results containing the Hindi term “संगठन”, regardless of the presence of punctuation or other special characters like “ः”?

Is this even possible?
