> For the complete documentation index, see [llms.txt](https://docs.digit.org/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.digit.org/platform/deploy/operations-guide/scaling.md).

# Scaling

## Scaling

DIGIT's microservices architecture is designed for horizontal scaling. Each service runs as a Kubernetes Deployment and can be independently scaled without affecting other services. The platform supports both manual scaling via replica count changes in Helm values and automatic scaling via Horizontal Pod Autoscaler (HPA).

## Horizontal POD Autoscaling (HPA)

Kubernetes HPA automatically adjusts the number of pod replicas based on observed CPU or memory utilisation. Configure HPA in the service's Helm values or as a standalone Kubernetes resource:

```
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: egov-workflow-v2
  namespace: egov
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: egov-workflow-v2
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70   # Scale out when CPU > 70%
```

Monitor HPA status with:

```
kubectl get hpa -n egov
kubectl describe hpa egov-workflow-v2 -n egov
```

## Manual Scaling Via Helm Values

For services with predictable load patterns, set replica counts directly in the environment values file and redeploy:

```
# In deploy-as-code/helm/environments/<env>.yaml
egov-workflow-v2:
  replicas: 3        # Increase before known high-load events
  resources:
    requests:
      cpu: 200m
      memory: 512Mi
    limits:
      cpu: 500m
      memory: 1Gi
```

<table><thead><tr><th width="202.98046875">SERVICE</th><th>RECOMMENDED PROD REPLICAS</th></tr></thead><tbody><tr><td>API Gateway (Zuul)</td><td>3 minimum. Front door for all traffic — keep HA. Scale first when seeing gateway timeouts.</td></tr><tr><td>egov-workflow-v2</td><td>2–3. Scales well horizontally. CPU-bound on complex workflow evaluations.</td></tr><tr><td>egov-persister</td><td>2–3. Kafka consumer — each replica handles a partition. Keep replicas ≤ Kafka partitions.</td></tr><tr><td>egov-indexer</td><td>2. Kafka consumer writing to ElasticSearch. Keep at 2 for HA.</td></tr><tr><td>egov-user</td><td>2–3. In the critical path of every API request (auth check). Keep replicas ≥ 2.</td></tr><tr><td>egov-notification-sms</td><td>1–2. IO-bound on SMS gateway. Scale if delivery latency increases.</td></tr><tr><td>PDF-service</td><td>2. CPU-intensive PDF generation. Scale for bulk certificate issuance periods.</td></tr></tbody></table>

## Resource Requests & Limits

Set resource requests and limits for every service. Requests determine scheduling; limits prevent runaway memory consumption. Kubernetes evicts pods that exceed memory limits and CPU-throttles pods that exceed CPU limits.

<table><thead><tr><th width="139.1015625">SETTING</th><th width="296.05859375">RECOMMENDATION</th><th>RISK IF MISSING</th></tr></thead><tbody><tr><td>requests.cpu</td><td>50–200m for most DIGIT services. Higher for Workflow and Persister.</td><td>Pod may be scheduled on an overloaded node, degrading performance.</td></tr><tr><td>requests.memory</td><td>256–512Mi for Spring Boot services. 1–2Gi for ElasticSearch/Kafka.</td><td>Scheduler cannot make optimal placement decisions.</td></tr><tr><td>limits.cpu</td><td>2–3x the request value. CPU is compressible — throttling is safe.</td><td>Without limits, a noisy neighbour can starve other pods on the same node.</td></tr><tr><td>limits.memory</td><td>1.5–2x the request value. Closely match observed peak consumption.</td><td>Without limits, a memory leak causes an OOMKill of the pod (restart). With limits too tight, legitimate spikes cause OOMKills.</td></tr></tbody></table>

Identifying Performance Bottlenecks

<table><thead><tr><th width="186.66015625">SYMPTOM</th><th width="316.73828125">LIKELY CAUSE</th><th>INVESTIGATION TOOL</th></tr></thead><tbody><tr><td>High API response latency</td><td>Slow database query, Kafka consumer lag, overloaded Persister, or insufficient Workflow replicas</td><td>Jaeger tracing — find the slowest span in the request chain</td></tr><tr><td>Pod OOMKilled frequently</td><td>Memory limit too low, memory leak in service, or JVM heap not tuned</td><td>kubectl describe pod &#x3C;pod> | grep -i oom; Grafana Pods dashboard</td></tr><tr><td>Kafka consumer lag growing</td><td>Persister or Indexer replicas insufficient, Kafka partition count too low, or downstream write bottleneck (DB/ES)</td><td>prometheus-kafka-exporter in Grafana; kubectl logs -n egov &#x3C;persister-pod></td></tr><tr><td>ElasticSearch index errors</td><td>Disk full, too many shards, or JVM heap exhaustion on ES nodes</td><td>GET /_cluster/health?pretty; ES-Curator to clear old indices</td></tr><tr><td>Node CPU throttling</td><td>Too many CPU-heavy services co-located on one node, or requests set too low</td><td>Grafana Nodes Overview; kubectl top nodes</td></tr><tr><td>DB connection pool exhausted</td><td>Service holding connections open too long, max_connections too low</td><td>Grafana or pg_stat_activity; tune pg pool size in service config</td></tr></tbody></table>

CAPACITY PLANNING GUIDELINES

<table><thead><tr><th width="160.9296875">GUIDELINE</th><th>DETAIL</th></tr></thead><tbody><tr><td>Watch PVC fill rate</td><td>ElasticSearch and Kafka produce the most data. Alert at 70% and expand PVC before reaching 85%. ES-Curator should auto-delete old indices before this threshold is reached.</td></tr><tr><td>Node autoscaling</td><td>On cloud deployments (EKS/GKE/AKS), enable cluster autoscaler. On SDC, manually provision additional worker VMs as 70% average node utilisation is approached.</td></tr><tr><td>Kafka partitions</td><td>Kafka consumer parallelism is bounded by partition count. If Persister consumer lag grows and replicas are already at max, increase partition count (requires Kafka topic reconfiguration).</td></tr><tr><td>DB read replicas</td><td>Add a PostgreSQL read replica for analytics and reporting queries. Route read-heavy services (DSS, Indexer) to the replica to reduce load on the primary.</td></tr><tr><td>Peak load testing</td><td>Run load tests against UAT before major citizen-facing campaigns (tax payment deadlines, enrollment drives). Use k6 or JMeter targeting the API Gateway endpoint.</td></tr><tr><td>SRE team rituals</td><td>Conduct weekly infrastructure review (node headroom, PVC fill rate, consumer lag). Monthly capacity review against transaction growth projections.</td></tr></tbody></table>


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.digit.org/platform/deploy/operations-guide/scaling.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
