Skip to content
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Backup & Recovery

All critical DIGIT state lives in three places: PostgreSQL (transactional data and audit), ElasticSearch (indexed search and analytics data), and the DIGIT-DevOps Git repository (all configuration and infrastructure code). A complete backup and recovery strategy must cover all three, as well as the Kafka topic offsets and persistent volume snapshots.

PostgreSQL Backup On AWS

For AWS deployments using RDS, automated backups and snapshots are the primary mechanism:

MECHANISM
CONFIGURATION

Automated snapshots

Enable in RDS console: Backup retention period = 7 days (minimum for Prod). RDS takes a daily snapshot and stores transaction logs for point-in-time recovery.

Manual snapshots

Take a manual snapshot before every major deployment or schema migration. These are retained until explicitly deleted.

Point-in-time recovery

Restore to any second within the retention window. Use when a data corruption event must be rolled back precisely.

Cross-region backup

Enable RDS automated backup replication to a second region for DR. Adds storage cost, but is essential for state data centres with no DR region.

Manual backup of a PostgreSQL database (non-managed/SDC deployments):

# Take a pg_dump backup of the DIGIT database
pg_dump -h <db-host> -U <db-user> -d digit_db -F c -f /backup/digit_db_$(date +%Y%m%d).dump

# Restore from dump file
pg_restore -h <db-host> -U <db-user> -d digit_db -F c /backup/digit_db_20250201.dump

# List available backups
ls -lh /backup/digit_db_*.dump

BACKUP SCHEDULE: Prod: daily automated snapshots + manual snapshot before every deployment. UAT: weekly snapshots. Dev: on-demand only. Store snapshots in a different availability zone or region from the primary database.

PostgreSQL High Availability

For production environments, configure PostgreSQL with a standby replica to reduce recovery time in the event of a primary failure:

HA PATTERN
DESCRIPTION

Managed HA (cloud)

Use RDS Multi-AZ on AWS (or equivalent on GCP/Azure). The managed service handles synchronous replication, automatic failover, and DNS update — zero manual intervention required. Failover completes in 60–120 seconds.

Self-hosted HA (SDC)

Deploy a PostgreSQL primary + standby using streaming replication (postgresql.conf: wal_level = replica, max_wal_senders = 3). Use Patroni or repmgr for automatic failover management on bare-metal/VM deployments.

Read replica

Add one or more read replicas for analytics and reporting workloads. Configure DSS and Indexer services to use the replica connection string.

Connection pooling

Deploy PgBouncer (transaction mode) in front of PostgreSQL to reduce connection overhead. Kubernetes services have many short-lived connections that can exhaust max_connections without pooling.

Elastic Search Backup & Index Management

ElasticSearch data is derived from PostgreSQL (via the Indexer service) and can be reconstructed by re-running the indexer. However, taking regular snapshots protects against slow or incomplete re-indexing after an ES cluster failure.

Use ES-Curator to delete old indices and prevent disk exhaustion automatically. Configure retention policies aligned with your data governance requirements (typically 90 days for audit data):

Persistent Volume Snapshots

ENVIRONMENT
SNAPSHOT APPROACH

AWS (EBS)

Use AWS Data Lifecycle Manager (DLM) to schedule automated EBS snapshots for all PVCs (Kafka, ZooKeeper, ElasticSearch data volumes). Snapshot daily. Retain 7 days on Prod.

GCP

Enable Cloud Storage scheduled backups for Persistent Disks via gcloud scheduler. Or use Velero (open-source Kubernetes backup tool) for application-consistent volume snapshots.

Azure

Azure Managed Disk snapshots via Azure Backup. Configure via Azure portal or Terraform.

SDC / On-prem

Use iSCSI volume snapshots provided by the SDC storage team. Request daily snapshots and verify recovery procedures with the SDC team at least once per quarter.

All envs

Velero (https://velero.io) provides cloud-agnostic Kubernetes backup including PVC snapshots and Kubernetes object backup. Recommended for standardising backup across cloud and SDC environments.

Disaster Recovery Plan

DIGIT recommends defining Recovery Time Objective (RTO) and Recovery Point Objective (RPO) targets before go-live. The table below shows typical targets and the mechanisms required to achieve them:

SCENARIO
RTO TARGET
RPO TARGET
MECHANISMS REQUIRED

Pod/container failure

< 2 min

0 (stateless)

Kubernetes liveness/readiness probes + pod restart policy. Zero data loss — stateless services restart automatically.

Node failure

< 5 min

0

Multi-replica deployments + pod anti-affinity to spread replicas across nodes. Kubernetes reschedules pods to surviving nodes automatically.

PostgreSQL primary failure

< 3 min

< 30 sec

RDS Multi-AZ (cloud) or Patroni (SDC). Automatic failover to standby replica. DNS update is the only delay.

Kubernetes cluster failure

< 30 min

< 24 hours

Rebuild cluster from Terraform IaC (DIGIT-DevOps repo). Restore DB from latest RDS snapshot. Re-apply Helm charts. Data loss = time since last backup.

Data corruption

< 2 hours

< 24 hours

PostgreSQL point-in-time recovery (PITR) to last known good state. Trigger ES re-indexing after DB restore.

Full region failure

< 4 hours

< 24 hours

Cross-region DB backup replication + re-provision infrastructure in DR region via Terraform. Update DNS to DR endpoint.

Recovery Runbook (POD & Service Failure)

STEP
ACTION

1. Detect

Alert fires (KubePodCrashLooping or EndpointDown). Check Grafana for the scope of impact.

2. Isolate

kubectl get pods -n egov | grep -v Running — identify failing pods and their namespaces.

3. Diagnose

kubectl describe pod <pod-name> -n egov (check Events section). kubectl logs <pod-name> -n egov --previous (last crash logs).

4. Quick fix

kubectl rollout restart deployment/<name> -n egov — forces a rolling restart. Often resolves transient errors.

5. Rollback

If recent deployment caused the failure: helm rollback <release-name> -n egov to previous revision.

6. Escalate

If rollback does not resolve: raise incident ticket, notify SPOC, activate incident management process.

7. Root cause

After service is restored, conduct blameless post-mortem. Update runbook with new pattern. Add alert rule if gap identified.

SRE Rituals & Standard Operating Procedures

DIGIT recommends SRE (Site Reliability Engineering) principles for running production deployments. The following regular rituals help prevent incidents and maintain operational confidence:

RITUAL
CADENCE
ACTIVITIES

Infrastructure review

Weekly

Review Grafana node and PVC dashboards. Check Kafka consumer lag trends. Confirm all backup jobs completed successfully. Review open alerts.

Capacity review

Monthly

Project storage growth (ES, Kafka, PVC). Compare actual transaction volume to provisioned capacity. Plan additional worker nodes or PVC expansions 4 weeks ahead.

Security review

Monthly

Review the Kubernetes RBAC access list. Rotate API keys and service account tokens. Check the SSL certificate expiry countdown on the Blackbox Exporter dashboard.

DR drill

Quarterly

Simulate a node failure in UAT. Verify automatic pod rescheduling. Test DB failover. Validate backup restoration from snapshot. Update RTO/RPO estimates.

Post-mortem

Per incident

Blameless review of every P1/P2 incident. Document timeline, root cause, and corrective actions. Publish summary to all stakeholders within 48 hours.

On-call handover

Per shift

Share current alert status, open incidents, pending deployments, and any known instability with incoming on-call engineer.

Team Responsibilities

TEAM
RESPONSIBILITY DOMAIN
SPOC ROLE

State IT / Cloud team

Infra, network, OS licensing, firewall, compute, disk, IOPS, SSL, DNS, data backups, snapshots, capacity monitoring

Infrastructure SPOC

State program team

Overall DIGIT implementation, application rollouts, capacity building, synchronising between teams

Program Manager

Implementation partner

DIGIT configuration, customisation, application performance monitoring, log scrutiny, DB query analysis, distributed tracing

Tech Lead

Operations team

DIGIT deployments, CI/CD, change management, traffic monitoring and alerting, log monitoring, DB backups, application uptime

DevOps Lead

All four teams must be represented in the incident management process. Use a shared ticketing system (e.g. Jira) to log and track all operational issues. Conduct regular sync meetings and ensure monitoring dashboards are visible to all teams.

Reference Reads

TOPIC

DIGIT Infra Overview

Monitoring (Prometheus/Grafana)

Alerting Setup

Logging (ECK / Loki)

Tracing (Jaeger)

Deployment using Helm

Pre/Post Deployment Checklist

Multi-tenancy Setup

PostgreSQL HA

Reliability & DR

Performance Guide

Incident Management

SRE Rituals

Last updated

Was this helpful?