Reindexing

We use re-indexing to transfer all data to the appropriate indexer. This process involves two steps:

  1. Run the connector from the playground.

  2. Call the legacy indexer service, which internally queries the plain search service to retrieve data and send it to the appropriate indexer.

Pre-requisites

  • Access to kubectl of the environment targeted

  • Plain search APIs in the respective services

NalJalSeva uses 3 main indexes for re-indexing:

  • Water-services

  • e-challan-services

  • dss-collection_v2

Re-indexing Steps

  • ws-services re-indexing - Kafka Connector Curl to be run from playground pod

curl --location --request POST 'http://kafka-connect.mgramseva:8083/connectors/' \
--header 'Cache-Control: no-cache' \
--header 'Content-Type: application/json' \
--header 'Postman-Token: 419e68ba-ffb9-4da9-86e1-7ad5a4c8d0b9' \
--data-raw '{
    "name": "water-services-enriched-es-sink",
    "config": {
        "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
        "type.name": "general",
        "tasks.max": "1",
        "max.retries": "15",
        "key.ignore": "false",
        "retry.backoff.ms": "5000",
        "max.buffered.records": "25",
        "value.converter": "org.apache.kafka.connect.json.JsonConverter",
        "errors.log.enable": "true",
        "key.converter": "org.apache.kafka.connect.storage.StringConverter",
        "read.timeout.ms": "100000",
        "topics": "water-services-enriched",
        "batch.size": "25",
        "max.in.flight.requests": "2",
        "schema.ignore": "true",
        "behavior.on.malformed.documents": "warn",
        "flush.timeout.ms": "3600000",
        "errors.deadletterqueue.topic.name": "water-services-enriched-failed",
        "errors.tolerance": "all",
        "value.converter.schemas.enable": "false",
        "name": "water-services-enriched-es-sink",
        "connection.url": "http://elasticsearch-data-v1.mgramseva:9200",
        "linger.ms": "1000",
        "transforms": "TopicNameRouter",
        "transforms.TopicNameRouter.type": "org.apache.kafka.connect.transforms.RegexRouter",
        "transforms.TopicNameRouter.regex": "water-services-enriched*",
        "transforms.TopicNameRouter.replacement": "water-services-enriched"
    }
}'

Plain Search Call

EChallan-Reindexing

Kafka Connector Call to be run from Playground pod:

Legacy Index call from postman:

  • Dss collection v2 re-indexing - Kafka Connector call to be run from playground pod

payment re-indexing run from postman call

PreviousPropertyNextRollout Dashboard

Last updated

Was this helpful?