> For the complete documentation index, see [llms.txt](https://docs.digit.org/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.digit.org/platform/deploy/operations-guide/backup-and-recovery.md).

# Backup & Recovery

All critical DIGIT state lives in three places: PostgreSQL (transactional data and audit), ElasticSearch (indexed search and analytics data), and the DIGIT-DevOps Git repository (all configuration and infrastructure code). A complete backup and recovery strategy must cover all three, as well as the Kafka topic offsets and persistent volume snapshots.

## PostgreSQL Backup On AWS

For AWS deployments using RDS, automated backups and snapshots are the primary mechanism:

<table><thead><tr><th width="191.2578125">MECHANISM</th><th>CONFIGURATION</th></tr></thead><tbody><tr><td>Automated snapshots</td><td>Enable in RDS console: Backup retention period = 7 days (minimum for Prod). RDS takes a daily snapshot and stores transaction logs for point-in-time recovery.</td></tr><tr><td>Manual snapshots</td><td>Take a manual snapshot before every major deployment or schema migration. These are retained until explicitly deleted.</td></tr><tr><td>Point-in-time recovery</td><td>Restore to any second within the retention window. Use when a data corruption event must be rolled back precisely.</td></tr><tr><td>Cross-region backup</td><td>Enable RDS automated backup replication to a second region for DR. Adds storage cost, but is essential for state data centres with no DR region.</td></tr></tbody></table>

Manual backup of a PostgreSQL database (non-managed/SDC deployments):

```
# Take a pg_dump backup of the DIGIT database
pg_dump -h <db-host> -U <db-user> -d digit_db -F c -f /backup/digit_db_$(date +%Y%m%d).dump

# Restore from dump file
pg_restore -h <db-host> -U <db-user> -d digit_db -F c /backup/digit_db_20250201.dump

# List available backups
ls -lh /backup/digit_db_*.dump
```

{% hint style="info" %}
**BACKUP SCHEDULE:** Prod: daily automated snapshots + manual snapshot before every deployment. UAT: weekly snapshots. Dev: on-demand only. Store snapshots in a different availability zone or region from the primary database.<br>
{% endhint %}

#### PostgreSQL High Availability

For production environments, configure PostgreSQL with a standby replica to reduce recovery time in the event of a primary failure:

<table><thead><tr><th width="200.9921875">HA PATTERN</th><th>DESCRIPTION</th></tr></thead><tbody><tr><td>Managed HA (cloud)</td><td>Use RDS Multi-AZ on AWS (or equivalent on GCP/Azure). The managed service handles synchronous replication, automatic failover, and DNS update — zero manual intervention required. Failover completes in 60–120 seconds.</td></tr><tr><td>Self-hosted HA (SDC)</td><td>Deploy a PostgreSQL primary + standby using streaming replication (postgresql.conf: wal_level = replica, max_wal_senders = 3). Use Patroni or repmgr for automatic failover management on bare-metal/VM deployments.</td></tr><tr><td>Read replica</td><td>Add one or more read replicas for analytics and reporting workloads. Configure DSS and Indexer services to use the replica connection string.</td></tr><tr><td>Connection pooling</td><td>Deploy PgBouncer (transaction mode) in front of PostgreSQL to reduce connection overhead. Kubernetes services have many short-lived connections that can exhaust max_connections without pooling.</td></tr></tbody></table>

## Elastic Search Backup & Index Management

ElasticSearch data is derived from PostgreSQL (via the Indexer service) and can be reconstructed by re-running the indexer. However, taking regular snapshots protects against slow or incomplete re-indexing after an ES cluster failure.

```
# Register an S3 snapshot repository
PUT _snapshot/digit_backup_repo
{
  "type": "s3",
  "settings": {
    "bucket": "<your-s3-bucket>",
    "region": "<aws-region>",
    "base_path": "es-snapshots/digit"
  }
}

# Take a snapshot
PUT _snapshot/digit_backup_repo/snapshot_$(date +%Y%m%d)
{
  "indices": "*",
  "include_global_state": false
}

# Restore from snapshot
POST _snapshot/digit_backup_repo/<snapshot-name>/_restore
```

Use ES-Curator to delete old indices and prevent disk exhaustion automatically. Configure retention policies aligned with your data governance requirements (typically 90 days for audit data):

```
# ES-Curator action file (actions.yaml)
actions:
  1:
    action: delete_indices
    description: Delete indices older than 90 days
    filters:
      - filtertype: age
        source: name
        direction: older
        unit: days
        unit_count: 90
        timestring: '%Y.%m.%d'
```

## Persistent Volume Snapshots

<table><thead><tr><th width="170.8828125">ENVIRONMENT</th><th>SNAPSHOT APPROACH</th></tr></thead><tbody><tr><td>AWS (EBS)</td><td>Use AWS Data Lifecycle Manager (DLM) to schedule automated EBS snapshots for all PVCs (Kafka, ZooKeeper, ElasticSearch data volumes). Snapshot daily. Retain 7 days on Prod.</td></tr><tr><td>GCP</td><td>Enable Cloud Storage scheduled backups for Persistent Disks via gcloud scheduler. Or use Velero (open-source Kubernetes backup tool) for application-consistent volume snapshots.</td></tr><tr><td>Azure</td><td>Azure Managed Disk snapshots via Azure Backup. Configure via Azure portal or Terraform.</td></tr><tr><td>SDC / On-prem</td><td>Use iSCSI volume snapshots provided by the SDC storage team. Request daily snapshots and verify recovery procedures with the SDC team at least once per quarter.</td></tr><tr><td>All envs</td><td>Velero (https://velero.io) provides cloud-agnostic Kubernetes backup including PVC snapshots and Kubernetes object backup. Recommended for standardising backup across cloud and SDC environments.</td></tr></tbody></table>

## Disaster Recovery Plan

DIGIT recommends defining Recovery Time Objective (RTO) and Recovery Point Objective (RPO) targets before go-live. The table below shows typical targets and the mechanisms required to achieve them:

<table><thead><tr><th width="129.09765625">SCENARIO</th><th width="123.09765625">RTO TARGET</th><th width="126.2578125">RPO TARGET</th><th>MECHANISMS REQUIRED</th></tr></thead><tbody><tr><td>Pod/container failure</td><td>&#x3C; 2 min</td><td>0 (stateless)</td><td>Kubernetes liveness/readiness probes + pod restart policy. Zero data loss — stateless services restart automatically.</td></tr><tr><td>Node failure</td><td>&#x3C; 5 min</td><td>0</td><td>Multi-replica deployments + pod anti-affinity to spread replicas across nodes. Kubernetes reschedules pods to surviving nodes automatically.</td></tr><tr><td>PostgreSQL primary failure</td><td>&#x3C; 3 min</td><td>&#x3C; 30 sec</td><td>RDS Multi-AZ (cloud) or Patroni (SDC). Automatic failover to standby replica. DNS update is the only delay.</td></tr><tr><td>Kubernetes cluster failure</td><td>&#x3C; 30 min</td><td>&#x3C; 24 hours</td><td>Rebuild cluster from Terraform IaC (DIGIT-DevOps repo). Restore DB from latest RDS snapshot. Re-apply Helm charts. Data loss = time since last backup.</td></tr><tr><td>Data corruption</td><td>&#x3C; 2 hours</td><td>&#x3C; 24 hours</td><td>PostgreSQL point-in-time recovery (PITR) to last known good state. Trigger ES re-indexing after DB restore.</td></tr><tr><td>Full region failure</td><td>&#x3C; 4 hours</td><td>&#x3C; 24 hours</td><td>Cross-region DB backup replication + re-provision infrastructure in DR region via Terraform. Update DNS to DR endpoint.</td></tr></tbody></table>

## Recovery Runbook (POD & Service Failure)

<table><thead><tr><th width="168.33984375">STEP</th><th>ACTION</th></tr></thead><tbody><tr><td>1. Detect</td><td>Alert fires (KubePodCrashLooping or EndpointDown). Check Grafana for the scope of impact.</td></tr><tr><td>2. Isolate</td><td>kubectl get pods -n egov | grep -v Running   —  identify failing pods and their namespaces.</td></tr><tr><td>3. Diagnose</td><td>kubectl describe pod &#x3C;pod-name> -n egov (check Events section). kubectl logs &#x3C;pod-name> -n egov --previous (last crash logs).</td></tr><tr><td>4. Quick fix</td><td>kubectl rollout restart deployment/&#x3C;name> -n egov   —  forces a rolling restart. Often resolves transient errors.</td></tr><tr><td>5. Rollback</td><td>If recent deployment caused the failure: helm rollback &#x3C;release-name> -n egov   to previous revision.</td></tr><tr><td>6. Escalate</td><td>If rollback does not resolve: raise incident ticket, notify SPOC, activate incident management process.</td></tr><tr><td>7. Root cause</td><td>After service is restored, conduct blameless post-mortem. Update runbook with new pattern. Add alert rule if gap identified.</td></tr></tbody></table>

## SRE Rituals & Standard Operating Procedures

DIGIT recommends SRE (Site Reliability Engineering) principles for running production deployments. The following regular rituals help prevent incidents and maintain operational confidence:

<table><thead><tr><th width="143.15625">RITUAL</th><th width="122.7109375">CADENCE</th><th>ACTIVITIES</th></tr></thead><tbody><tr><td>Infrastructure review</td><td>Weekly</td><td>Review Grafana node and PVC dashboards. Check Kafka consumer lag trends. Confirm all backup jobs completed successfully. Review open alerts.</td></tr><tr><td>Capacity review</td><td>Monthly</td><td>Project storage growth (ES, Kafka, PVC). Compare actual transaction volume to provisioned capacity. Plan additional worker nodes or PVC expansions 4 weeks ahead.</td></tr><tr><td>Security review</td><td>Monthly</td><td>Review the Kubernetes RBAC access list. Rotate API keys and service account tokens. Check the SSL certificate expiry countdown on the Blackbox Exporter dashboard.</td></tr><tr><td>DR drill</td><td>Quarterly</td><td>Simulate a node failure in UAT. Verify automatic pod rescheduling. Test DB failover. Validate backup restoration from snapshot. Update RTO/RPO estimates.</td></tr><tr><td>Post-mortem</td><td>Per incident</td><td>Blameless review of every P1/P2 incident. Document timeline, root cause, and corrective actions. Publish summary to all stakeholders within 48 hours.</td></tr><tr><td>On-call handover</td><td>Per shift</td><td>Share current alert status, open incidents, pending deployments, and any known instability with incoming on-call engineer.</td></tr></tbody></table>

## Team Responsibilities&#x20;

<table><thead><tr><th width="188.4921875">TEAM</th><th width="378">RESPONSIBILITY DOMAIN</th><th>SPOC ROLE</th></tr></thead><tbody><tr><td>State IT / Cloud team</td><td>Infra, network, OS licensing, firewall, compute, disk, IOPS, SSL, DNS, data backups, snapshots, capacity monitoring</td><td>Infrastructure SPOC</td></tr><tr><td>State program team</td><td>Overall DIGIT implementation, application rollouts, capacity building, synchronising between teams</td><td>Program Manager</td></tr><tr><td>Implementation partner</td><td>DIGIT configuration, customisation, application performance monitoring, log scrutiny, DB query analysis, distributed tracing</td><td>Tech Lead</td></tr><tr><td>Operations team</td><td>DIGIT deployments, CI/CD, change management, traffic monitoring and alerting, log monitoring, DB backups, application uptime</td><td>DevOps Lead</td></tr></tbody></table>

All four teams must be represented in the incident management process. Use a shared ticketing system (e.g. Jira) to log and track all operational issues. Conduct regular sync meetings and ensure monitoring dashboards are visible to all teams.

## Reference Reads

| TOPIC                           |
| ------------------------------- |
| DIGIT Infra Overview            |
| Monitoring (Prometheus/Grafana) |
| Alerting Setup                  |
| Logging (ECK / Loki)            |
| Tracing (Jaeger)                |
| Deployment using Helm           |
| Pre/Post Deployment Checklist   |
| Multi-tenancy Setup             |
| PostgreSQL HA                   |
| Reliability & DR                |
| Performance Guide               |
| Incident Management             |
| SRE Rituals                     |


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.digit.org/platform/deploy/operations-guide/backup-and-recovery.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
