> For the complete documentation index, see [llms.txt](https://docs.digit.org/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.digit.org/platform/deploy/operations-guide/monitoring.md).

# Monitoring

DIGIT's observability stack is built on three open-source pillars: Prometheus for metrics, Loki for log aggregation, and Jaeger for distributed tracing. All three are deployed via Helmfile and visualised through Grafana. Together, they give operations teams full visibility into infrastructure health, service performance, and request flows.

## Monitoring Stack Overview

<table><thead><tr><th width="119.140625">TOOL</th><th width="418.1484375">ROLE</th><th>ACCESS</th></tr></thead><tbody><tr><td>Prometheus</td><td>Time-series metrics database. Scrapes metrics from Kubernetes nodes, pods, services, and backbone components using a pull model. Stores data with timestamps and labels.</td><td>Internal; exposed via Grafana data source</td></tr><tr><td>Grafana</td><td>Visualisation layer. Connects to Prometheus and Loki. Pre-built dashboards for Kubernetes, Nginx Ingress, pods, nodes, PVCs, Redis, and Loki logs.</td><td>&#x3C;domain>/monitoring  —  e.g. https://unified-dev.digit.org/monitoring/</td></tr><tr><td>Loki</td><td>Log aggregation system inspired by Prometheus. Collects, stores, and indexes logs from all pods using label-based indexing. Queried using LogQL via Grafana.</td><td>Via Grafana Loki data source</td></tr><tr><td>Alertmanager</td><td>Receives alerts from Prometheus. Groups, deduplicates, and routes them to notification channels (Slack, email, PagerDuty). Manages silences and inhibition rules.</td><td>kubectl port-forward port 9093 or via Grafana</td></tr><tr><td>Jaeger</td><td>Distributed tracing. Tracks API requests across microservices. Provides end-to-end latency analysis and dependency mapping for debugging performance issues.</td><td>&#x3C;domain>/tracing</td></tr><tr><td>Blackbox Exporter</td><td>Probes HTTP endpoints to verify availability. Tracks SSL expiry, DNS lookup time, probe duration, and HTTP status codes from outside the cluster.</td><td>Via Grafana Blackbox Exporter dashboard</td></tr></tbody></table>

## Deploying The Monitoring Stack

All monitoring tools are deployed together from the DIGIT-2.9LTS-monitoring branch using Helmfile:

```
git checkout DIGIT-2.9LTS-monitoring
cd deploy-as-code/charts/monitoring

# Preview manifests
helmfile -e <env> -f monitoring-helmfile.yaml template

# Review changes
helmfile -e <env> -f monitoring-helmfile.yaml diff

# Deploy all monitoring tools
helmfile -e <env> -f monitoring-helmfile.yaml apply
```

After deployment, retrieve Grafana admin credentials:

```
kubectl get secret grafana -n monitoring -o json | jq -r '.data | map_values(@base64d)'
```

Enable nginx-ingress serviceMonitor so Prometheus can scrape ingress metrics. Add the following to your env.yaml, then redeploy nginx-ingress:

```
nginx-ingress:
  controller:
    metrics:
      enabled: true
      serviceMonitor:
        enabled: true
    service:
      prometheusRule:
        enabled: true

```

## Grafana Dashboards

DIGIT ships with nine pre-built Grafana dashboards covering all layers of the stack. Dashboard JSON files are stored in the configs repository under monitoring-dashboards/ and auto-loaded via a Grafana init container (git-sync).

<table><thead><tr><th width="168.35546875">DASHBOARD</th><th>KEY METRICS &#x26; PURPOSE</th></tr></thead><tbody><tr><td>Blackbox Exporter</td><td>Endpoint availability (UP/DOWN), HTTP response codes, SSL certificate expiry countdown, probe duration, DNS lookup time. Use to detect outages before users report them.</td></tr><tr><td>Kubernetes Overview</td><td>Cluster-wide CPU and memory utilisation, requested vs limit allocations, active node/namespace/pod counts, service and PVC object counts. Capacity planning and baseline health.</td></tr><tr><td>Namespaces Overview</td><td>Per-namespace CPU and memory usage as a percentage of cluster capacity, pod counts, configmap and secret counts. Identifies namespaces consuming disproportionate resources.</td></tr><tr><td>Nodes Overview</td><td>Per-node CPU, RAM, and pod count with uptime. Use to detect overloaded nodes or nodes approaching eviction thresholds.</td></tr><tr><td>Pods Overview</td><td>Per-pod CPU/memory requests vs limits with real-time gauges. Container-level breakdown inside each pod. Identifies over-provisioned or OOMKilling containers.</td></tr><tr><td>Nginx Ingress</td><td>Total HTTP request volume, success rate, HTTP status code distribution (2xx/3xx/4xx/5xx), latency percentiles, active connections. Monitors API gateway health.</td></tr><tr><td>PV / PVC Dashboard</td><td>Persistent Volume Claim utilisation, available storage, and bound/pending/lost status. Fires alerts when PVCs approach capacity.</td></tr><tr><td>Loki Dashboard</td><td>Log stream frequency over time, error/fatal log filtering by namespace and pod, query performance. Incident analysis and log-based SLIs.</td></tr><tr><td>Redis Monitoring</td><td>Uptime, client connections, memory utilisation %, commands per second, cache hit/miss rate, network I/O, expiring vs non-expiring key counts.</td></tr></tbody></table>

To add a custom panel to an existing dashboard: log in to Grafana, click Add Panel, configure PromQL queries, save the dashboard, export the JSON, then commit the updated JSON file to the configs repo monitoring-dashboards/ folder.

Log Aggregation With ECK & LOKI

<table><thead><tr><th width="139.2578125">APPROACH</th><th>COMPONENTS &#x26; NOTES</th></tr></thead><tbody><tr><td>ECK Stack</td><td>Elastic Cloud on Kubernetes (ECK) Operator manages Elasticsearch (3-node cluster, 100 Gi persistent storage per node), Kibana (single instance), and Filebeat (DaemonSet — one pod per node, collects /var/log/containers logs). Deploy via: helm install elastic-operator elastic/eck-operator -n monitoring</td></tr><tr><td>Loki Stack</td><td>Lighter-weight alternative to ECK. Label-based log indexing (no full-text indexing of all fields). Queried using LogQL in Grafana. Lower storage cost than Elasticsearch. Recommended for new deployments on DIGIT 2.9 LTS.</td></tr><tr><td>Log retention</td><td>Set retention_period: 168h (7 days) in Loki config. For Elasticsearch, use ES-Curator to automatically delete indices older than a configurable threshold to prevent disk exhaustion.</td></tr></tbody></table>

Access Kibana logs after ECK deployment:

```
# Get Kibana password
kubectl get secret elasticsearch-logging-es-elastic-user -ojson \
  | jq '.data.elastic' -r | base64 -d

# Port-forward Kibana
kubectl port-forward svc/kibana-logging-kb-http 5601:5601 -n monitoring
# Open http://localhost:5601 with user: elastic
```

## Distributed Tracing With JAEGER

Jaeger is used to track requests across DIGIT microservices. When a citizen submits an application, the request traverses the API Gateway, User Service, the relevant domain service, Persister, and notification services. Jaeger records the latency at each hop, enabling rapid pinpointing of bottlenecks.

<table><thead><tr><th width="152.55078125">FEATURE</th><th>DETAIL</th></tr></thead><tbody><tr><td>Access URL</td><td>&#x3C;domain>/tracing  —  e.g. https://qa.digit.org/tracing/search</td></tr><tr><td>Use for</td><td>Identifying which microservice is slow, diagnosing cascading failures, measuring end-to-end API latency per service type</td></tr><tr><td>Core services</td><td>Use tracing to monitor all core service API call chains — especially Workflow Engine, Persister, and User Service which are in the critical path of every transaction</td></tr><tr><td>Setup</td><td>Deploy via Helmfile (jaeger chart). Configure each Spring Boot service to emit OpenTelemetry spans to the Jaeger collector endpoint.</td></tr></tbody></table>

## What To Monitor Per Infrastructure Layer

<table><thead><tr><th width="139.765625">LAYER</th><th width="394.3671875">KEY METRICS TO WATCH</th><th>TOOLS</th></tr></thead><tbody><tr><td>Infrastructure</td><td>Node CPU/RAM utilisation, disk IOPS, network throughput, PVC fill rate, node count vs pod scheduling pressure</td><td>Grafana Nodes/PVC dashboards, Prometheus node-exporter</td></tr><tr><td>Backbone Services</td><td>Kafka consumer lag, ElasticSearch cluster health (green/yellow/red), ZooKeeper quorum, Redis memory %, PostgreSQL connections and replication lag</td><td>Grafana, prometheus-kafka-exporter, ES health API</td></tr><tr><td>Core Services</td><td>API response time per service, error rate (5xx %) per endpoint, Kafka topic publish/consume rate, Persister queue depth</td><td>Jaeger tracing, Grafana Nginx Ingress, Prometheus</td></tr><tr><td>DIGIT Apps</td><td>Citizen/employee portal availability (Blackbox Exporter), application-specific KPIs in DSS dashboard, workflow completion rates</td><td>Blackbox Exporter, DSS dashboard, Grafana</td></tr></tbody></table>


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.digit.org/platform/deploy/operations-guide/monitoring.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
