Backup & Recovery
All critical DIGIT state lives in three places: PostgreSQL (transactional data and audit), ElasticSearch (indexed search and analytics data), and the DIGIT-DevOps Git repository (all configuration and infrastructure code). A complete backup and recovery strategy must cover all three, as well as the Kafka topic offsets and persistent volume snapshots.
PostgreSQL Backup On AWS
For AWS deployments using RDS, automated backups and snapshots are the primary mechanism:
Automated snapshots
Enable in RDS console: Backup retention period = 7 days (minimum for Prod). RDS takes a daily snapshot and stores transaction logs for point-in-time recovery.
Manual snapshots
Take a manual snapshot before every major deployment or schema migration. These are retained until explicitly deleted.
Point-in-time recovery
Restore to any second within the retention window. Use when a data corruption event must be rolled back precisely.
Cross-region backup
Enable RDS automated backup replication to a second region for DR. Adds storage cost, but is essential for state data centres with no DR region.
Manual backup of a PostgreSQL database (non-managed/SDC deployments):
# Take a pg_dump backup of the DIGIT database
pg_dump -h <db-host> -U <db-user> -d digit_db -F c -f /backup/digit_db_$(date +%Y%m%d).dump
# Restore from dump file
pg_restore -h <db-host> -U <db-user> -d digit_db -F c /backup/digit_db_20250201.dump
# List available backups
ls -lh /backup/digit_db_*.dumpPostgreSQL High Availability
For production environments, configure PostgreSQL with a standby replica to reduce recovery time in the event of a primary failure:
Managed HA (cloud)
Use RDS Multi-AZ on AWS (or equivalent on GCP/Azure). The managed service handles synchronous replication, automatic failover, and DNS update — zero manual intervention required. Failover completes in 60–120 seconds.
Self-hosted HA (SDC)
Deploy a PostgreSQL primary + standby using streaming replication (postgresql.conf: wal_level = replica, max_wal_senders = 3). Use Patroni or repmgr for automatic failover management on bare-metal/VM deployments.
Read replica
Add one or more read replicas for analytics and reporting workloads. Configure DSS and Indexer services to use the replica connection string.
Connection pooling
Deploy PgBouncer (transaction mode) in front of PostgreSQL to reduce connection overhead. Kubernetes services have many short-lived connections that can exhaust max_connections without pooling.
Elastic Search Backup & Index Management
ElasticSearch data is derived from PostgreSQL (via the Indexer service) and can be reconstructed by re-running the indexer. However, taking regular snapshots protects against slow or incomplete re-indexing after an ES cluster failure.
Use ES-Curator to delete old indices and prevent disk exhaustion automatically. Configure retention policies aligned with your data governance requirements (typically 90 days for audit data):
Persistent Volume Snapshots
AWS (EBS)
Use AWS Data Lifecycle Manager (DLM) to schedule automated EBS snapshots for all PVCs (Kafka, ZooKeeper, ElasticSearch data volumes). Snapshot daily. Retain 7 days on Prod.
GCP
Enable Cloud Storage scheduled backups for Persistent Disks via gcloud scheduler. Or use Velero (open-source Kubernetes backup tool) for application-consistent volume snapshots.
Azure
Azure Managed Disk snapshots via Azure Backup. Configure via Azure portal or Terraform.
SDC / On-prem
Use iSCSI volume snapshots provided by the SDC storage team. Request daily snapshots and verify recovery procedures with the SDC team at least once per quarter.
All envs
Velero (https://velero.io) provides cloud-agnostic Kubernetes backup including PVC snapshots and Kubernetes object backup. Recommended for standardising backup across cloud and SDC environments.
Disaster Recovery Plan
DIGIT recommends defining Recovery Time Objective (RTO) and Recovery Point Objective (RPO) targets before go-live. The table below shows typical targets and the mechanisms required to achieve them:
Pod/container failure
< 2 min
0 (stateless)
Kubernetes liveness/readiness probes + pod restart policy. Zero data loss — stateless services restart automatically.
Node failure
< 5 min
0
Multi-replica deployments + pod anti-affinity to spread replicas across nodes. Kubernetes reschedules pods to surviving nodes automatically.
PostgreSQL primary failure
< 3 min
< 30 sec
RDS Multi-AZ (cloud) or Patroni (SDC). Automatic failover to standby replica. DNS update is the only delay.
Kubernetes cluster failure
< 30 min
< 24 hours
Rebuild cluster from Terraform IaC (DIGIT-DevOps repo). Restore DB from latest RDS snapshot. Re-apply Helm charts. Data loss = time since last backup.
Data corruption
< 2 hours
< 24 hours
PostgreSQL point-in-time recovery (PITR) to last known good state. Trigger ES re-indexing after DB restore.
Full region failure
< 4 hours
< 24 hours
Cross-region DB backup replication + re-provision infrastructure in DR region via Terraform. Update DNS to DR endpoint.
Recovery Runbook (POD & Service Failure)
1. Detect
Alert fires (KubePodCrashLooping or EndpointDown). Check Grafana for the scope of impact.
2. Isolate
kubectl get pods -n egov | grep -v Running — identify failing pods and their namespaces.
3. Diagnose
kubectl describe pod <pod-name> -n egov (check Events section). kubectl logs <pod-name> -n egov --previous (last crash logs).
4. Quick fix
kubectl rollout restart deployment/<name> -n egov — forces a rolling restart. Often resolves transient errors.
5. Rollback
If recent deployment caused the failure: helm rollback <release-name> -n egov to previous revision.
6. Escalate
If rollback does not resolve: raise incident ticket, notify SPOC, activate incident management process.
7. Root cause
After service is restored, conduct blameless post-mortem. Update runbook with new pattern. Add alert rule if gap identified.
SRE Rituals & Standard Operating Procedures
DIGIT recommends SRE (Site Reliability Engineering) principles for running production deployments. The following regular rituals help prevent incidents and maintain operational confidence:
Infrastructure review
Weekly
Review Grafana node and PVC dashboards. Check Kafka consumer lag trends. Confirm all backup jobs completed successfully. Review open alerts.
Capacity review
Monthly
Project storage growth (ES, Kafka, PVC). Compare actual transaction volume to provisioned capacity. Plan additional worker nodes or PVC expansions 4 weeks ahead.
Security review
Monthly
Review the Kubernetes RBAC access list. Rotate API keys and service account tokens. Check the SSL certificate expiry countdown on the Blackbox Exporter dashboard.
DR drill
Quarterly
Simulate a node failure in UAT. Verify automatic pod rescheduling. Test DB failover. Validate backup restoration from snapshot. Update RTO/RPO estimates.
Post-mortem
Per incident
Blameless review of every P1/P2 incident. Document timeline, root cause, and corrective actions. Publish summary to all stakeholders within 48 hours.
On-call handover
Per shift
Share current alert status, open incidents, pending deployments, and any known instability with incoming on-call engineer.
Team Responsibilities
State IT / Cloud team
Infra, network, OS licensing, firewall, compute, disk, IOPS, SSL, DNS, data backups, snapshots, capacity monitoring
Infrastructure SPOC
State program team
Overall DIGIT implementation, application rollouts, capacity building, synchronising between teams
Program Manager
Implementation partner
DIGIT configuration, customisation, application performance monitoring, log scrutiny, DB query analysis, distributed tracing
Tech Lead
Operations team
DIGIT deployments, CI/CD, change management, traffic monitoring and alerting, log monitoring, DB backups, application uptime
DevOps Lead
All four teams must be represented in the incident management process. Use a shared ticketing system (e.g. Jira) to log and track all operational issues. Conduct regular sync meetings and ensure monitoring dashboards are visible to all teams.
Reference Reads
DIGIT Infra Overview
Monitoring (Prometheus/Grafana)
Alerting Setup
Logging (ECK / Loki)
Tracing (Jaeger)
Deployment using Helm
Pre/Post Deployment Checklist
Multi-tenancy Setup
PostgreSQL HA
Reliability & DR
Performance Guide
Incident Management
SRE Rituals
Last updated
Was this helpful?