1 of 12

Availability

Infrastructure

How to check if Infra is working as expected?
How to monitor and setup alerts? Other debugging tools?
Solutions to common problems and next steps

Backbone services

Database

DB monitoring, alerting and debugging guidelines

Kafka

Kafka Connect

Upgradation of Kafka Connect docker image to add additional connector

Overview

This page provides the steps to follow for upgrading Kafka Connect.

Steps

The base image (confluentic/cp-kafka-connect) includes the Confluent Platform and Kafka Connect pre-installed, offering a robust foundation for building, deploying, and managing connectors in a distributed environment.
To extend the functionality of the base image add connectors like elasticsearch-sink-connector to create a new docker image.
Download the elasticsearch-sink-connector jar files on your local machine using the link .

Run the below command to build the docker image.

Run the below command to rename the docker image.

Push the image to the dockerhub using the below command.

Replace the image tag in kafka-connect helm chart values.yaml and redploy the kafka-connect.

Elastic search

Elastic Search Rolling Upgrade

Overview

This page provides comprehensive documentation and instructions for implementing a rolling upgrade strategy for your Elasticsearch cluster.

Steps

Note: During the rolling upgrade, it is anticipated that there will be some downtime. Additionally, ensure to take an elasticdump of the Elasticsearch data using the script provided below in the playground pod.

Copy the below script and save it as es-dump.sh. Replace the elasticsearch URL and the indices names in the script.

Run the below commands in the terminal.

Now, run the below command inside the playground pod.

Rolling upgrade from v6.6.2 to v7.17.15

Steps

List the elasticsearch pods and enter into any of the elasticsearch pod shells.

Disable shard allocation: You can avoid racing the clock by of replicas before shutting down . Stop non-essential indexing and perform a synced flush: While you can continue indexing during the upgrade, shard recovery is much faster if you temporarily stop non-essential indexing and perform a . Run the below curls inside elasticsearch data pod.

Scale down the replica count of elasticsearch master and data from 3 to 0.

Edit the Statefulset of elasticsearch master by replacing the docker image removing deprecated environment variables and adding compatible environment variables. Replace the elasticsearch image tag from 6.6.2 to 7.17.15. The below code provides the depraced environment variables and compatible environment variables.

Edit elasticsearch-master values.yaml file

Edit the Statefulset of elasticsearch data by replacing the docker image removing deprecated environment variables and adding compatible environment variables. Replace the elasticsearch image tag from 6.6.2 to 7.17.15.

Edit elasticsearch-data values.yaml file.

After making the changes, scale up the statefulsets of elasticsearch data and master.

After all pods are in running state, re-enable shard allocation and check cluster health.

You have successfully upgraded the elasticsearch cluster from v6.6.2 to v7.17.15 :)

ReIndexing the Indices:

After successfully upgrading the elasticsearch, reindex the indices present in elasticsearch using below script which are created in v6.6.2 or earlier.

Copy the below script and save it as es-reindex.sh. Replace the elasticsearch URL in the script.

Run the below commands in the terminal.

Now, run the below command inside the playground pod.

NOTE: Make Sure to delete jaeger indices as mapping is not supported in v8.11.3 and the indices which are created before v7.17.15 by reindexing. If the indices which are created in v6.6.2 or earlier are present then the upgradation from v7.17.15 to v8.11.3 may fail.

Rolling upgrade from v7.17.15 to v8.11.3 & security is disabled

Steps

Scale down the replica count of elasticsearch master and data from 3 to 0.

Edit the Statefulset of elasticsearch master by replacing the docker image removing deprecated environment variables and adding compatible environment variables. Replace the elasticsearch image tag from 7.17.15 to 8.11.3. The below code provides the compatible environment variables and if you are following a rolling upgrade then there are no deprecated environment variables from v7.17.15 to v8.11.3.

Edit the Statefulset of elasticsearch data by replacing the docker image removing deprecated environment variables and adding compatible environment variables. Replace the elasticsearch image tag from 7.17.15 to 8.11.3.

After making the changes, scale up the statefulsets of elasticsearch data and master.

After all pods are in running state, re-enable shard allocation and check cluster health.

You have successfully upgraded the elasticsearch cluster from v7.17.15 to v8.11.3 👏

ElasticSearch Direct Upgrade

Overview

Unlike rolling upgrades, direct upgrades involve migrating from an older version to a newer one in a single coordinated operation.

This comprehensive guide outlines the step-by-step process for deploying an Elasticsearch 8.11.3 cluster with enhanced security features. The document not only covers the initial deployment of the cluster but also includes instructions for seamlessly migrating data from an existing Elasticsearch cluster to the new one, allowing for a direct upgrade.

Steps

Clone the DIGIT-DevOps repo and checkout to the branch digit-lts-go.

If you want to make any changes to the elasticsearch cluster like namespaces etc. You'll find the helm chart for elastic search in the path provided below. In the below chart, security is enabled for elasticsearch. If you want to disable the security, please set the environment variable xpack.security.enabled as false in the helm chart statefulset template.

Elasticsearch secrets have been present in cluster configs chart since indexer, inbox services etc have dependency on elasticsearch secrets. Below is the template.

In cluster-configs values.yaml, add the namespaces in which you want to deploy the elasticsearch secrets.

Add the elasticsearch password in the env-secrets.yaml file, if not it will automatically creates a random password which will be updated everytime you deploy the elasticsearch.

Deploy the Elastic Search Cluster using the below commands.

Check the pods status using the below command.

Once all pods are running, execute the below commands inside the playground pod to dump data from the old elasticsearch cluster and restore it to the new elasticsearch cluster.

Using the above script, you can take the data dump from the old cluster and restore it in the new elasticsearch in a single command.
After restoring the data successfully in the new elasticsearch cluster, check the cluster health and document count using the below command.

Now the deployment and restoring the data are completed successfully. It's time to change the es_url and indexer_url in egov-config present under cluster-configs of the environment file. The same can be updated directly using the below command.

Restart all the pods which have a dependency on elasticsearchwith cluster-configs to pick a new elasticsearch_url.

Core services

Monitoring how-to
Debugging
Fixing/escalating

DIGIT apps

Monitor, debug, fix

DSS dashboard

ElasticSearch Direct Upgrade

Overview

Unlike rolling upgrades, direct upgrades involve migrating from an older version to a newer one in a single coordinated operation.

Steps

Clone the DIGIT-DevOps repo and checkout to the branch digit-lts-go.

If you want to make any changes to the elasticsearch cluster like namespaces etc. You'll find the helm chart for elastic search in the path provided below. In the below chart, security is enabled for elasticsearch. If you want to disable the security, please set the environment variable xpack.security.enabled as false in the helm chart statefulset template.

Elasticsearch secrets have been present in cluster configs chart since indexer, inbox services etc have dependency on elasticsearch secrets. Below is the template.

In cluster-configs values.yaml, add the namespaces in which you want to deploy the elasticsearch secrets.

Add the elasticsearch password in the env-secrets.yaml file, if not it will automatically creates a random password which will be updated everytime you deploy the elasticsearch.

Deploy the Elastic Search Cluster using the below commands.

Check the pods status using the below command.

Once all pods are running, execute the below commands inside the playground pod to dump data from the old elasticsearch cluster and restore it to the new elasticsearch cluster.

Using the above script, you can take the data dump from the old cluster and restore it in the new elasticsearch in a single command.
After restoring the data successfully in the new elasticsearch cluster, check the cluster health and document count using the below command.

Now the deployment and restoring the data are completed successfully. It's time to change the es_url and indexer_url in egov-config present under cluster-configs of the environment file. The same can be updated directly using the below command.

Restart all the pods which have a dependency on elasticsearchwith cluster-configs to pick a new elasticsearch_url.

#!/bin/bash
# Elasticsearch cluster information
ELASTICSEARCH_OLD_URL="<old_elasticsearch_url>"    # eg:- elasticsearch-data-v1.es-cluster:9200
ELASTICSEARCH_NEW_URL="<new_elasticsearch_url>"    # eg:- elasticsearch-data.es-cluster:9200

# Authentication credentials
USERNAME="elastic"
PASSWORD="<es_pwd>"

DUMP_ENABLE=true
RESTORE_ENABLE=true

# Disable SSL/TLS validation
export NODE_TLS_REJECT_UNAUTHORIZED=0

# Provide the indices to take dump
EXCLUDE_INDEX_PATTERN="jaeger|monitor|kibana|fluentbit"
# Provide backup directory
BACKUP_DIR="backup"
# Provide indices output file
IDICES_OUTPUT="elasticsearch-indexes.txt"

INDICES_LIST=$(curl -sk "http://${ELASTICSEARCH_OLD_URL}/_cat/indices" | grep -v -E "${EXCLUDE_INDEX_PATTERN}" | awk '{print $3}')
IFS=$'\n' read -r -d '' -a INDICES <<< "$INDICES_LIST"

printf "%s\n" "${INDICES[@]}" > $IDICES_OUTPUT

if [ "$DUMP_ENABLE" = true ]; then
    # Create backup directory if it doesn't exist
    mkdir -p "$BACKUP_DIR"

    # Loop through each index and perform export
    for INDEX in "${INDICES[@]}"; do
        OUTPUT_FILE="${BACKUP_DIR}/${INDEX}_mapping_backup.json"

        # Build the elasticdump command
        ELASTICDUMP_CMD="elasticdump \
            --input=http://${ELASTICSEARCH_OLD_URL}/${INDEX} \
            --output=${OUTPUT_FILE} \
            --type=mapping"

        # Execute the elasticdump command
        $ELASTICDUMP_CMD

        # Check if the elasticdump command was successful
        if [ $? -eq 0 ]; then
            echo "Backup of index ${INDEX} mapping completed successfully."
        else
            echo "Error backing up index ${INDEX}."
        fi
    done

    for INDEX in "${INDICES[@]}"; do
        OUTPUT_FILE="${BACKUP_DIR}/${INDEX}_data_backup.json"

        # Build the elasticdump command
        ELASTICDUMP_CMD="elasticdump \
            --input=http://${ELASTICSEARCH_OLD_URL}/${INDEX} \
            --output=${OUTPUT_FILE} \
            --type=data
            --limit 10000"

        # Execute the elasticdump command
        $ELASTICDUMP_CMD

        # Check if the elasticdump command was successful
        if [ $? -eq 0 ]; then
            echo "Backup of index ${INDEX} completed successfully."
        else
            echo "Error backing up index ${INDEX}."
        fi
    done
fi

if [ "$RESTORE_ENABLE" = true ]; then
    for INDEX in "${INDICES[@]}"; do
        OUTPUT_FILE="${BACKUP_DIR}/${INDEX}_mapping_backup.json"
        
        # Process the mapping file to remove unsupported parameters

        PROCESSED_FILE="${BACKUP_DIR}/${INDEX}_mapping_processed.json"

        jq 'del(.mappings._default_, .mappings._meta, .mappings.dynamic_templates, .mappings.dynamic, .mappings.general) | .mappings = .mappings["_doc"]' "${INPUT_FILE}" > "${PROCESSED_FILE}"

        # Print the contents of the processed file for debugging

        echo "Contents of ${PROCESSED_FILE}:"

        cat "${PROCESSED_FILE}"

        # Build the elasticdump command
        ELASTICDUMP_CMD="elasticdump \
            --input=${PROCESSED_FILE} \
            --output=https://${USERNAME}:${PASSWORD}@${ELASTICSEARCH_NEW_URL}/${INDEX} \
            --type=mapping"

        # Execute the elasticdump command
        $ELASTICDUMP_CMD

        # Check if the elasticdump command was successful
        if [ $? -eq 0 ]; then
            echo "Restoring of index ${INDEX} mapping completed successfully."
        else
            echo "Error Restoring index ${INDEX}."
        fi
    done

    for INDEX in "${INDICES[@]}"; do
        OUTPUT_FILE="${BACKUP_DIR}/${INDEX}_data_backup.json"

        # Build the elasticdump command
        ELASTICDUMP_CMD="elasticdump \
            --input=${OUTPUT_FILE} \
            --output=https://${USERNAME}:${PASSWORD}@${ELASTICSEARCH_NEW_URL}/${INDEX} \
            --type=data
            --limit 10000"

        # Execute the elasticdump command
        $ELASTICDUMP_CMD

        # Check if the elasticdump command was successful
        if [ $? -eq 0 ]; then
            echo "Restoring of index ${INDEX} data completed successfully."
        else
            echo "Error Restoring index ${INDEX}."
        fi
    done
fi

#!/bin/bash
# Elasticsearch cluster information
ELASTICSEARCH_OLD_URL="<old_elasticsearch_url>"    # eg:- elasticsearch-data-v1.es-cluster:9200
ELASTICSEARCH_NEW_URL="<new_elasticsearch_url>"    # eg:- elasticsearch-data.es-cluster:9200

# Authentication credentials
USERNAME="elastic"
PASSWORD="<es_pwd>"

DUMP_ENABLE=true
RESTORE_ENABLE=true

# Disable SSL/TLS validation
export NODE_TLS_REJECT_UNAUTHORIZED=0

# Provide the indices to take dump
EXCLUDE_INDEX_PATTERN="jaeger|monitor|kibana|fluentbit"
# Provide backup directory
BACKUP_DIR="backup"
# Provide indices output file
IDICES_OUTPUT="elasticsearch-indexes.txt"

INDICES_LIST=$(curl -sk "http://${ELASTICSEARCH_OLD_URL}/_cat/indices" | grep -v -E "${EXCLUDE_INDEX_PATTERN}" | awk '{print $3}')
IFS=$'\n' read -r -d '' -a INDICES <<< "$INDICES_LIST"

printf "%s\n" "${INDICES[@]}" > $IDICES_OUTPUT

if [ "$DUMP_ENABLE" = true ]; then
    # Create backup directory if it doesn't exist
    mkdir -p "$BACKUP_DIR"

    # Loop through each index and perform export
    for INDEX in "${INDICES[@]}"; do
        OUTPUT_FILE="${BACKUP_DIR}/${INDEX}_mapping_backup.json"

        # Build the elasticdump command
        ELASTICDUMP_CMD="elasticdump \
            --input=http://${ELASTICSEARCH_OLD_URL}/${INDEX} \
            --output=${OUTPUT_FILE} \
            --type=mapping"

        # Execute the elasticdump command
        $ELASTICDUMP_CMD

        # Check if the elasticdump command was successful
        if [ $? -eq 0 ]; then
            echo "Backup of index ${INDEX} mapping completed successfully."
        else
            echo "Error backing up index ${INDEX}."
        fi
    done

    for INDEX in "${INDICES[@]}"; do
        OUTPUT_FILE="${BACKUP_DIR}/${INDEX}_data_backup.json"

        # Build the elasticdump command
        ELASTICDUMP_CMD="elasticdump \
            --input=http://${ELASTICSEARCH_OLD_URL}/${INDEX} \
            --output=${OUTPUT_FILE} \
            --type=data
            --limit 10000"

        # Execute the elasticdump command
        $ELASTICDUMP_CMD

        # Check if the elasticdump command was successful
        if [ $? -eq 0 ]; then
            echo "Backup of index ${INDEX} completed successfully."
        else
            echo "Error backing up index ${INDEX}."
        fi
    done
fi

if [ "$RESTORE_ENABLE" = true ]; then
    for INDEX in "${INDICES[@]}"; do
        OUTPUT_FILE="${BACKUP_DIR}/${INDEX}_mapping_backup.json"
        
        # Process the mapping file to remove unsupported parameters

        PROCESSED_FILE="${BACKUP_DIR}/${INDEX}_mapping_processed.json"

        jq 'del(.mappings._default_, .mappings._meta, .mappings.dynamic_templates, .mappings.dynamic, .mappings.general) | .mappings = .mappings["_doc"]' "${INPUT_FILE}" > "${PROCESSED_FILE}"

        # Print the contents of the processed file for debugging

        echo "Contents of ${PROCESSED_FILE}:"

        cat "${PROCESSED_FILE}"

        # Build the elasticdump command
        ELASTICDUMP_CMD="elasticdump \
            --input=${PROCESSED_FILE} \
            --output=https://${USERNAME}:${PASSWORD}@${ELASTICSEARCH_NEW_URL}/${INDEX} \
            --type=mapping"

        # Execute the elasticdump command
        $ELASTICDUMP_CMD

        # Check if the elasticdump command was successful
        if [ $? -eq 0 ]; then
            echo "Restoring of index ${INDEX} mapping completed successfully."
        else
            echo "Error Restoring index ${INDEX}."
        fi
    done

    for INDEX in "${INDICES[@]}"; do
        OUTPUT_FILE="${BACKUP_DIR}/${INDEX}_data_backup.json"

        # Build the elasticdump command
        ELASTICDUMP_CMD="elasticdump \
            --input=${OUTPUT_FILE} \
            --output=https://${USERNAME}:${PASSWORD}@${ELASTICSEARCH_NEW_URL}/${INDEX} \
            --type=data
            --limit 10000"

        # Execute the elasticdump command
        $ELASTICDUMP_CMD

        # Check if the elasticdump command was successful
        if [ $? -eq 0 ]; then
            echo "Restoring of index ${INDEX} data completed successfully."
        else
            echo "Error Restoring index ${INDEX}."
        fi
    done
fi

Elastic Search Rolling Upgrade

Overview

This page provides comprehensive documentation and instructions for implementing a rolling upgrade strategy for your Elasticsearch cluster.

Steps

Copy the below script and save it as es-dump.sh. Replace the elasticsearch URL and the indices names in the script.

Run the below commands in the terminal.

Now, run the below command inside the playground pod.

Rolling upgrade from v6.6.2 to v7.17.15

Steps

List the elasticsearch pods and enter into any of the elasticsearch pod shells.

Disable shard allocation: You can avoid racing the clock by of replicas before shutting down . Stop non-essential indexing and perform a synced flush: While you can continue indexing during the upgrade, shard recovery is much faster if you temporarily stop non-essential indexing and perform a . Run the below curls inside elasticsearch data pod.

Scale down the replica count of elasticsearch master and data from 3 to 0.

Edit the Statefulset of elasticsearch master by replacing the docker image removing deprecated environment variables and adding compatible environment variables. Replace the elasticsearch image tag from 6.6.2 to 7.17.15. The below code provides the depraced environment variables and compatible environment variables.

Edit elasticsearch-master values.yaml file

Edit the Statefulset of elasticsearch data by replacing the docker image removing deprecated environment variables and adding compatible environment variables. Replace the elasticsearch image tag from 6.6.2 to 7.17.15.

Edit elasticsearch-data values.yaml file.

After making the changes, scale up the statefulsets of elasticsearch data and master.

After all pods are in running state, re-enable shard allocation and check cluster health.

You have successfully upgraded the elasticsearch cluster from v6.6.2 to v7.17.15 :)

ReIndexing the Indices:

After successfully upgrading the elasticsearch, reindex the indices present in elasticsearch using below script which are created in v6.6.2 or earlier.

Copy the below script and save it as es-reindex.sh. Replace the elasticsearch URL in the script.

Run the below commands in the terminal.

Now, run the below command inside the playground pod.

Rolling upgrade from v7.17.15 to v8.11.3 & security is disabled

Steps

Scale down the replica count of elasticsearch master and data from 3 to 0.

Edit the Statefulset of elasticsearch master by replacing the docker image removing deprecated environment variables and adding compatible environment variables. Replace the elasticsearch image tag from 7.17.15 to 8.11.3. The below code provides the compatible environment variables and if you are following a rolling upgrade then there are no deprecated environment variables from v7.17.15 to v8.11.3.

Edit the Statefulset of elasticsearch data by replacing the docker image removing deprecated environment variables and adding compatible environment variables. Replace the elasticsearch image tag from 7.17.15 to 8.11.3.

After making the changes, scale up the statefulsets of elasticsearch data and master.

After all pods are in running state, re-enable shard allocation and check cluster health.

You have successfully upgraded the elasticsearch cluster from v7.17.15 to v8.11.3 👏