Fuzzy Search

Procedure to integrate Elastic Search for Fuzzy search with any of the eGov modules.

Creating the index as per the requirement

1. The elastic index must be used for fuzzy search to improve performance, and when PII fields are involved, which cannot be accessed directly in the DB, to apply fuzzy logic.

2. When PII data are involved, those fields shouldn’t be returned by Elasticsearch; they can only be used to apply filters. To enable this, that particular field should be indexed but not stored.

Example:

{
  "mappings": {
    "general": {
      "_source": {
        "excludes": [
          "name"
        ]
      },
      "properties": {
        "id": {
          "type": "text"
        },
        "dob": {
          "type": "date"
        },
        "name": {
          "type": "text"
        }
      }
    }
  }
}

In the example above, when data is posted, all fields are stored, but the index never returns excluded source fields when searched.

Querying an index

curl --location --request GET 'localhost:9200/_search?pretty' \

--header 'Content-Type: application/json' \
--data-raw '{
    "query": {
        "fuzzy": {
            "name": {
                "value": "ki"
            }
        }
    }
}'

This is a simple request; additional parameters can be used to enhance the fuzzy search.

curl -X GET "localhost:9200/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "query": {
    "fuzzy": {
      "name": {
        "value": "ki",
        "fuzziness": "AUTO",
        "max_expansions": 50,
        "prefix_length": 0,
        "transpositions": true
      }
    }
  }
}
'

Elastic Integration with code

1. Add the Elasticsearch host from the eGov cluster.

2. Divide the search params between the ones that will be sent to Elastic and others to DB (search to Elastic should be made only if the fuzzy fields are present in the search request)

3. The result of the Elasticsearch query should be only the required primary ID of the entity (property-id, trade license-id, etc.).

4. Then the resulting IDs should be used in the standard search as-is.

Please refer to the following commit for the integration

Application Properties

Properties need to be added for integration

#Elastic search properties

elasticsearch.host=http://localhost:9200/
elasticsearch.search.endpoint=/_search
property.es.index=property-services
pt.search.name.fuziness=2
pt.search.doorno.fuziness=2
pt.search.oldpropertyid.fuziness=2
pt.fuzzy.searh.is.wildcard=true

The indices need to be reindexed for fuzzy search to work if the index is modified, particularly when protected fields are indexed but not returned in search results.

Reindexing for PT: Refer here.