> For the complete documentation index, see [llms.txt](https://docs.digit.org/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.digit.org/platform/deploy/operations-guide/alerting.md).

# Alerting

Alertmanager is the notification hub of the DIGIT observability stack. It receives firing alerts from Prometheus, applies grouping and deduplication rules, suppresses noise, and routes notifications to the right channels. It is deployed as part of the kube-prometheus-stack via Helmfile.

## Accessing Alert Manager

Alertmanager does not have a public ingress by default. Use port-forwarding to access the web UI:

```
kubectl port-forward svc/kube-prometheus-stack-alertmanager -n monitoring 9093:9093
# Open http://localhost:9093 in a browser
```

The Alertmanager web UI shows all currently firing alerts, their labels, severity, and routing destination. You can also create silences directly in the UI for planned maintenance windows.

## Configuring Alert Rules & Channels

Alertmanager configuration is stored as a Kubernetes secret. Add the alertmanager config to your \<env>-secrets.yaml (encrypted with SOPS) before deploying. A typical configuration routes all alerts to a Slack channel:

```
# In <env>-secrets.yaml (encrypt with SOPS before committing)
alertmanager:
  config:
    global:
      resolve_timeout: 5m
    route:
      group_by: ['alertname', 'namespace']
      group_wait: 30s           # Wait before sending first alert in a group
      group_interval: 5m        # Wait before re-notifying after new alert in group
      repeat_interval: 12h      # Re-notify if alert is still firing after this interval
      receiver: 'slack-critical'
      routes:
        - match:
            severity: warning
          receiver: 'slack-warnings'
    receivers:
      - name: 'slack-critical'
        slack_configs:
          - api_url: '<your-slack-webhook-url>'
            channel: '#digit-alerts-critical'
            send_resolved: true
      - name: 'slack-warnings'
        slack_configs:
          - api_url: '<your-slack-webhook-url>'
            channel: '#digit-alerts-warnings'
            send_resolved: true
```

## Supported Notification Channels

<table><thead><tr><th width="168.5859375">CHANNEL</th><th>CONFIGURATION KEY &#x26; USE CASE</th></tr></thead><tbody><tr><td>Slack</td><td>slack_configs with api_url (webhook URL) and channel. Most common for DevOps teams. Supports message templating with alert labels and annotations.</td></tr><tr><td>Email</td><td>email_configs with to, from, smarthost, auth_username, auth_password. Use for stakeholder notifications and escalation to management.</td></tr><tr><td>PagerDuty</td><td>pagerduty_configs with service_key. Use for critical P1 alerts requiring on-call paging with escalation policies.</td></tr><tr><td>Webhook</td><td>webhook_configs with url. Send alert payloads to any HTTP endpoint — integrate with ticketing systems (Jira), custom dashboards, or runbook automations.</td></tr><tr><td>OpsGenie</td><td>opsgenie_configs with api_key and api_url. Alternative to PagerDuty for on-call management.</td></tr></tbody></table>

## Key Alert Rules To Configure

The kube-prometheus-stack ships with default Kubernetes alerting rules. The following are the most critical rules to verify are enabled and tuned for DIGIT production environments:

<table><thead><tr><th width="147.9375">ALERT NAME</th><th width="115.015625">SEVERITY</th><th width="210.33203125">FIRES WHEN</th><th>ACTION</th></tr></thead><tbody><tr><td>KubePodCrashLooping</td><td>Critical</td><td>A pod restarts more than 5 times in 1 hour</td><td>Check logs: kubectl logs &#x3C;pod> -n egov --previous</td></tr><tr><td>KubeNodeNotReady</td><td>Critical</td><td>A node has not been Ready for > 15 minutes</td><td>Inspect node: kubectl describe node &#x3C;node-name></td></tr><tr><td>KubePVCFull</td><td>Critical</td><td>PVC is > 85% full</td><td>Expand PVC or clean up ES indices with ES-Curator</td></tr><tr><td>KubeMemoryOvercommit</td><td>Warning</td><td>Pod memory requests exceed node capacity</td><td>Review resource limits in Helm values and reschedule pods</td></tr><tr><td>KubeDeploymentReplicasFail</td><td>Warning</td><td>Deployment has fewer Ready replicas than desired</td><td>Check pod events: kubectl get events -n egov</td></tr><tr><td>TargetDown</td><td>Critical</td><td>A Prometheus scrape target is unreachable</td><td>Verify service endpoints and serviceMonitor config</td></tr><tr><td>EndpointDown (Blackbox)</td><td>Critical</td><td>An HTTP probe fails (citizen/employee portal)</td><td>Check Nginx Ingress, pod health, and DNS resolution</td></tr><tr><td>SSLCertExpiry</td><td>Warning</td><td>SSL certificate expires in &#x3C; 30 days</td><td>Trigger Cert-Manager renewal or upload new certificate</td></tr><tr><td>KafkaConsumerLag</td><td>Warning</td><td>Kafka consumer group lag > configured threshold</td><td>Scale Persister replicas or check Kafka broker health</td></tr><tr><td>ElasticClusterHealth</td><td>Critical</td><td>Elasticsearch cluster status is Red</td><td>Check unassigned shards: GET /_cluster/health?pretty</td></tr><tr><td>RedisMemoryHigh</td><td>Warning</td><td>Redis memory usage > 80%</td><td>Review TTL policies, increase memory limit, or scale Redis</td></tr><tr><td>PostgresConnections</td><td>Warning</td><td>Active connections > 80% of max_connections</td><td>Check for connection leaks, tune max_connections in postgres config</td></tr></tbody></table>

## Silences & Inhibition Rules

<table><thead><tr><th width="148.640625">MECHANISM</th><th>WHEN TO USE</th></tr></thead><tbody><tr><td>Silence</td><td>Mute alerts during a planned maintenance window. Create via the Alertmanager web UI at http://localhost:9093 or via the API. Set an expiry time so silences do not persist indefinitely.</td></tr><tr><td>Inhibition rule</td><td>Suppress lower-severity alerts when a higher-severity alert is already firing. Example: suppress pod-level warnings when the entire node is reporting as NotReady. Configure in the alertmanager.config in secrets.yaml.</td></tr><tr><td>Repeat interval</td><td>Set repeat_interval to prevent notification storms for persistent issues. Default recommendation: 12h for critical, 24h for warnings.</td></tr></tbody></table>

{% hint style="info" %}
**ALERT FATIGUE:** Keep the number of active alert rules focused on actionable conditions. Alerts that fire and resolve without human action erode team responsiveness. Review and tune thresholds after the first month of production operation.
{% endhint %}


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.digit.org/platform/deploy/operations-guide/alerting.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
