Indexer

Overview

In DIGIT, the Indexer Service is used to store data in Elastic by listening to messages from Kafka topics.

It needs a configuration file (in YAML) to understand:

• Which Kafka topic to listen to

• What data to extract from the Kafka message

• Where and how to insert/update this data in the database

Steps

1

Add YAML Config in Config Repo

• Go to your GitHub config repo.

• Use the right branch based on your environment from where your indexer service is reading data. Example - dev, uat, prod, master.

• Add a new file named like this for the reference given in the link. There are two indexer files.

  • egov-indexer/pgr-services.yml

  • Reference link for the pgr-service.yml indexer config

  • egov-indexer/pgr-inbox-indexer.yml

  • Reference link for the pgr-inbox-indexer.yml indexer config

Sample Indexer Configuration (For CMS)

ServiceMaps:
  serviceName: Public Grievance Redressal - rainmaker
  version: 1.0.0
  mappings:
    - topic: save-pgr-request
      configKey: INDEX
      indexes:
        - name: inbox-pgr-services
          type: general
          id: $.service.serviceRequestId
          jsonPath: $
          timeStampField: $.service.auditDetails.createdTime
          customJsonMapping:
            indexMapping: {"Data":{"service":{},"tenantId": "","auditDetails": {},"currentProcessInstance": {}}}
            fieldMapping:
              - inJsonPath: $.service
                outJsonPath: $.Data.service
              - inJsonPath: $.service.tenantId
                outJsonPath: $.Data.tenantId
              - inJsonPath: $.service.auditDetails
                outJsonPath: $.Data.auditDetails
              - inJsonPath: $.service.processInstance
                outJsonPath: $.Data.currentProcessInstance

    - topic: update-pgr-request
      configKey: INDEX
      indexes:
        - name: inbox-pgr-services
          type: general
          id: $.service.serviceRequestId
          jsonPath: $
          timeStampField: $.service.auditDetails.createdTime
          customJsonMapping:
            indexMapping: {"Data":{"service":{},"tenantId": "","auditDetails": {},"currentProcessInstance": {}}}
            fieldMapping:
              - inJsonPath: $.service
                outJsonPath: $.Data.service
              - inJsonPath: $.service.tenantId
                outJsonPath: $.Data.tenantId
              - inJsonPath: $.service.auditDetails
                outJsonPath: $.Data.auditDetails
              - inJsonPath: $.service.processInstance
                outJsonPath: $.Data.currentProcessInstance

    - topic: pgr-services-inboxIndex
      configKey: LEGACYINDEX
      indexes:
        - name: inbox-pgr-services-legacy
          type: general
          isBulk: true
          id: $.service.serviceRequestId
          jsonPath: $.ServiceWrappers.*
          timeStampField: $.service.auditDetails.createdTime
          customJsonMapping:
            indexMapping: {"Data":{"service":{},"tenantId": "","auditDetails": {},"currentProcessInstance": {}}}
            fieldMapping:
              - inJsonPath: $.service
                outJsonPath: $.Data.service
              - inJsonPath: $.service.tenantId
                outJsonPath: $.Data.tenantId
              - inJsonPath: $.service.auditDetails
                outJsonPath: $.Data.auditDetails
              - inJsonPath: $.service.ProcessInstances[0]
                outJsonPath: $.Data.currentProcessInstance

    - topic: pgr-service-reindex
      configKey: REINDEX
      indexes:
        - name: pgrreindex-v2
          type: general
          id: $.Data.service.serviceRequestId
          isBulk: true
          jsonPath: $.hits
          timeStampField: $.Data.service.auditDetails.lastModifiedTime

2

Copy and paste both the indexer configs

• Copy and paste both the indexer configs into the created file.

• Restart the indexer service to read the updated indexer file.

• This file tells the indexer how to save CMS complaints in Elastic Search.

How to read the details:

Term	What it means
moduleName	Logical name (here, PGR)
topic	Kafka topic (like save-pgr-request) that triggers DB action
basePath	Root path in JSON (e.g.,$.or $.service) from which data is picked
jsonPath	Specific path in the request JSON to map to an elastic reference mapping column

3

Restart the indexer service

• After saving the config, run the following command to reload the updated config:

kubectl rollout restart deployment <persister-deployment-name> -n <namespace>

• Or through the Jenkins

Why restart? The service won't read your new config file until it's restarted.

4

Validate the configuration

Once the service is restarted:

• Submit a complaint.

• Check:

  • A message is published on Kafka (e.g., save-pgr-request)

  • The data is stored in the Elastic (check the Elastic Indexer)

  • Complaint Indexing in Elasticsearch (CMS)

    When a complaint is created in CCRS, the data is indexed into Elasticsearch for fast search and reporting. The index is named inbox-pgr-services.

    The mapping between complaint creation and the index is defined in the Indexer configuration, which listens to Kafka topics save-pgr-request & update-pgr-request and pushes data into the index.

    Indexes are usually created during indexer service onboarding using predefined mappings via configs. You can find index and topic mappings in the configs folder under the Indexer module.

• If nothing happens:

  • Check that the topic name matches in both the application and the indexer YAML

  • Look for logs in the indexer pod for errors

---

Reference

• Indexer Config Samples on GitHub

Summary

• The indexer config ensures compliant data is saved automatically when it is created or updated to the Elastic Index.

• It works behind the scenes, triggered by Kafka.

• You configure it using a simple YAML file to define what to save and where.

• Restart the service after adding or editing the config.