Indexer

Overview

In DIGIT, the Indexer Service is used to store data in Elastic by listening to messages from Kafka topics.

It needs a configuration file (in YAML) to understand:

• Which Kafka topic to listen to

• What data to extract from the Kafka message

• Where and how to insert/update this data in the database

Steps

1

Add YAML Config in Config Repo

• Go to your GitHub config repo.

• Use the right branch based on your environment from where your indexer service is reading data. Example - dev, uat, prod, master.

• Add a new file named like this for the reference given in the link. There are two indexer files.

  • egov-indexer/pgr-services.yml

  • Reference link for the pgr-service.yml indexer config

  • egov-indexer/pgr-inbox-indexer.yml

  • Reference link for the pgr-inbox-indexer.yml indexer config

Sample Indexer Configuration (For CCRS)

ServiceMaps:
  serviceName: Public Grievance Redressal - rainmaker
  version: 1.0.0
  mappings:
    - topic: save-pgr-request
      configKey: INDEX
      indexes:
        - name: inbox-pgr-services
          type: general
          id: $.service.serviceRequestId
          jsonPath: $
          timeStampField: $.service.auditDetails.createdTime
          customJsonMapping:
            indexMapping: {"Data":{"service":{},"tenantId": "","auditDetails": {},"currentProcessInstance": {}}}
            fieldMapping:
              - inJsonPath: $.service
                outJsonPath: $.Data.service
              - inJsonPath: $.service.tenantId
                outJsonPath: $.Data.tenantId
              - inJsonPath: $.service.auditDetails
                outJsonPath: $.Data.auditDetails
              - inJsonPath: $.service.processInstance
                outJsonPath: $.Data.currentProcessInstance

    - topic: update-pgr-request
      configKey: INDEX
      indexes:
        - name: inbox-pgr-services
          type: general
          id: $.service.serviceRequestId
          jsonPath: $
          timeStampField: $.service.auditDetails.createdTime
          customJsonMapping:
            indexMapping: {"Data":{"service":{},"tenantId": "","auditDetails": {},"currentProcessInstance": {}}}
            fieldMapping:
              - inJsonPath: $.service
                outJsonPath: $.Data.service
              - inJsonPath: $.service.tenantId
                outJsonPath: $.Data.tenantId
              - inJsonPath: $.service.auditDetails
                outJsonPath: $.Data.auditDetails
              - inJsonPath: $.service.processInstance
                outJsonPath: $.Data.currentProcessInstance

    - topic: pgr-services-inboxIndex
      configKey: LEGACYINDEX
      indexes:
        - name: inbox-pgr-services-legacy
          type: general
          isBulk: true
          id: $.service.serviceRequestId
          jsonPath: $.ServiceWrappers.*
          timeStampField: $.service.auditDetails.createdTime
          customJsonMapping:
            indexMapping: {"Data":{"service":{},"tenantId": "","auditDetails": {},"currentProcessInstance": {}}}
            fieldMapping:
              - inJsonPath: $.service
                outJsonPath: $.Data.service
              - inJsonPath: $.service.tenantId
                outJsonPath: $.Data.tenantId
              - inJsonPath: $.service.auditDetails
                outJsonPath: $.Data.auditDetails
              - inJsonPath: $.service.ProcessInstances[0]
                outJsonPath: $.Data.currentProcessInstance

    - topic: pgr-service-reindex
      configKey: REINDEX
      indexes:
        - name: pgrreindex-v2
          type: general
          id: $.Data.service.serviceRequestId
          isBulk: true
          jsonPath: $.hits
          timeStampField: $.Data.service.auditDetails.lastModifiedTime

2

Copy and paste both the indexer configs

• Copy and paste both the indexer configs into the created file.

• Restart the indexer service to read the updated indexer file.

• This file tells the indexer how to save CCRS complaints in Elastic Search.

How to read the details:

Term	What it means
moduleName	Logical name (here, PGR)
topic	Kafka topic (like save-pgr-request) that triggers DB action
basePath	Root path in JSON (e.g.,$.or $.service) from which data is picked
jsonPath	Specific path in the request JSON to map to an elastic reference mapping column

3

Restart the indexer service

• After saving the config, run the below command to reload the updated config:

kubectl rollout restart deployment <persister-deployment-name> -n <namespace>

• Or through the Jenkins

Why restart? The service won't read your new config file until it's restarted.

4

Validate the configuration

Once the service is restarted:

• Submit a complaint .

• Check:

  • A message is published on Kafka (e.g., save-pgr-request)

  • The data is stored in the Elastic (check the Elastic Indexer)

  • Complaint Indexing in Elasticsearch (CCRS)

    When a complaint is created in CCRS, the data is indexed into Elasticsearch for fast search and reporting. The index is named inbox-pgr-services .

    The mapping between complaint creation and the index is defined in the Indexer configuration, which listens to Kafka topics save-pgr-request & update-pgr-request and pushes data into the index.

    Indexes are usually created during indexer service onboarding using predefined mappings via configs . You can find index and topic mappings in configs folder under the Indexer module.

• If nothing happens:

  • Check that the topic name matches in both the application and the indexer YAML

  • Look for logs in the indexer pod for errors

---

Reference

• Indexer Config Samples on GitHub

Summary

• The indexer config ensures compliant data is saved automatically when it is created or updated to the Elastic Index.

• It works behind the scenes, triggered by Kafka.

• You configure it using a simple YAML file to define what to save and where.

• Restart the service after adding or editing the config.