> For the complete documentation index, see [llms.txt](https://docs.digit.org/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.digit.org/health/v2.0/deploy/installation/production-setup/install-monitoring-stack/1.-configure.md).

# 1. Configure

## Overview

Configure the monitoring stack via Helmfile: Grafana, Loki, Prometheus, and alerting.

## Setup Helmfile environment

Define environments, namespace, and environment-specific values files.

**File:** `config-as-code/helm/charts/monitoring/monitoring-helmfile.yaml`

```yaml
environments:
  demo:
    values:
      - namespace: monitoring
      - ../../../environments/egov-demo.yaml
      - ../../../environments/egov-demo-secrets.yaml
```

{% hint style="info" %}

* `demo` is the environment name.
* `monitoring` is the namespace.
* Update file paths when you add environments.
  {% endhint %}

## Configure Grafana

1. **Create GitHub OAuth App.** Note the **Client ID** and **Client Secret**.
   * **Homepage URL:** `https://<your_domain_name>`
   * **Authorization Callback URL:** `https://<your_domain_name>/monitoring/login/github`
2. **Update Grafana OAuth secrets.**

   **File:** `config-as-code/environments/egov-demo-secrets.yaml`

   ```yaml
   cluster-configs:
     secrets:
       grafana:
         clientID: <oauth-client-id>
         clientSecret: <oauth-client-secret>
   ```
3. **Restrict access by GitHub org/team.**

   **File:** `config-as-code/environments/egov-demo.yaml`

   ```yaml
   grafana:
     github:
       allowed_organizations: ["<organization>"]
       role_attribute_path: contains(groups[*], '@<organization>/<team>') && 'Viewer'
   ```

{% hint style="info" %}
Valid roles: `None`, `Viewer`, `Editor`, `Admin`, `GrafanaAdmin`.\
See [Grafana GitHub OAuth documentation](https://grafana.com/docs/grafana/latest/setup-grafana/configure-security/configure-authentication/github/#configure-role-mapping) for advanced mappings.
{% endhint %}

## Configure Loki (log storage)

Choose **one** storage option.

### Option A: Local filesystem storage

**File:** `config-as-code/environments/egov-demo.yaml`

```yaml
loki:
  persistence:
    enabled: true
    accessModes: [ReadWriteOnce]
    size: 15Gi
  serviceAccount:
    annotations: {}
  additionalConfigs:
    schema_config:
      configs:
        - from: 2020-10-24
          store: boltdb-shipper
          object_store: filesystem
          schema: v11
          index:
            prefix: index_
            period: 24h
    storage_config:
      boltdb_shipper:
        active_index_directory: /data/loki/index
        cache_location: /data/loki/index_cache
        shared_store: filesystem
        cache_ttl: 24h
      filesystem:
        directory: /data/loki/chunks
    compactor:
      working_directory: /data/loki/boltdb-shipper-compactor
      shared_store: filesystem
      retention_enabled: true
      compaction_interval: 72h
    table_manager:
      retention_deletes_enabled: true
      retention_period: 72h
```

### Option B: AWS S3 storage

{% hint style="warning" %}
When using OIDC, use the **`sub` claim** (not `aud`) in IAM trust policies.
{% endhint %}

1. **Create an IAM role with S3 access** (policy example):

   ```json
   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Sid": "AccessToLokiBucket",
         "Effect": "Allow",
         "Action": ["s3:PutObject", "s3:GetObject", "s3:DeleteObject", "s3:ListBucket"],
         "Resource": ["arn:aws:s3:::<s3-bucket>", "arn:aws:s3:::<s3-bucket>/*"]
       }
     ]
   }
   ```
2. **Update Loki config** (example for EKS IRSA):

   **File:** `config-as-code/environments/egov-demo.yaml`

   ```yaml
   loki:
     persistence:
       enabled: true
       accessModes: [ReadWriteOnce]
       size: 10Gi
     serviceAccount:
       annotations:
         eks.amazonaws.com/role-arn: <s3-role-arn>
     additionalConfigs:
       schema_config:
         configs:
           - from: 2020-10-24
             store: boltdb-shipper
             object_store: s3
             schema: v11
             index:
               prefix: index_
               period: 24h
       storage_config:
         boltdb_shipper:
           active_index_directory: /data/loki/index
           cache_location: /data/loki/index_cache
           shared_store: s3
           cache_ttl: 24h
         aws:
           s3: s3://<region>/<s3-bucket>
       compactor:
         working_directory: /data/loki/boltdb-shipper-compactor
         shared_store: s3
         retention_enabled: true
         compaction_interval: 168h
       table_manager:
         retention_deletes_enabled: true
         retention_period: 168h
   ```

### Option C: Azure Blob storage

**File:** `config-as-code/environments/egov-demo.yaml`

```yaml
loki:
  persistence:
    enabled: true
    accessModes: [ReadWriteOnce]
    size: 15Gi
  serviceAccount:
    annotations: {}
  additionalConfigs:
    schema_config:
      configs:
        - from: 2020-10-24
          store: boltdb-shipper
          object_store: azure
          schema: v11
          index:
            prefix: index_
            period: 24h
    storage_config:
      azure:
        account_name: <storage-account-name>
        account_key: <storage-account-key>
        container_name: <container-name>
        # use_managed_identity: <true|false>
        # user_assigned_id: <user-assigned-identity-id>
        request_timeout: 0
        # Use this for private clouds (example shown).
        # endpoint_suffix: <endpoint-suffix>
      boltdb_shipper:
        active_index_directory: /data/loki/index
        cache_location: /data/loki/index_cache
        shared_store: azure
        cache_ttl: 24h
    compactor:
      working_directory: /data/loki/boltdb-shipper-compactor
      shared_store: azure
      retention_enabled: true
      compaction_interval: 72h
    table_manager:
      retention_deletes_enabled: true
      retention_period: 72h
```

{% hint style="info" %}
See the [Loki configuration docs](https://grafana.com/docs/loki/latest/configure/) for all supported options.
{% endhint %}

## Configure Prometheus

**File:** `config-as-code/environments/egov-demo.yaml`

```yaml
prometheus:
  prometheusSpec:
    retention: 7d
    storageSpec:
      volumeClaimTemplate:
        spec:
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 30Gi
    alertmanager:
      enabled: false
    externalLabels:
      cluster: <cluster-name>
    additionalScrapeConfigs:
      - job_name: 'nginx-ingress-metrics'
        static_configs:
          - targets: ['ingress-nginx-controller-metrics.egov:10254']
      - job_name: 'redis-exporter'
        static_configs:
          - targets: ['prometheus-redis-exporter.backbone:9121']
      - job_name: 'blackbox'
        metrics_path: /probe
        params:
          module: [http_2xx]
        static_configs:
          - targets:
              - <url-to-monitor>
        relabel_configs:
          - source_labels: [__address__]
            target_label: __param_target
          - source_labels: [__param_target]
            target_label: instance
          - target_label: __address__
            replacement: blackbox-prometheus-blackbox-exporter:9115
      - job_name: 'blackbox_exporter'
        static_configs:
          - targets: ['blackbox-prometheus-blackbox-exporter:9115']
```

#### Additional scrape targets

* NGINX Ingress
* Redis exporter
* Blackbox monitoring (URLs, APIs)

Update targets to match your cluster and namespaces.

## Configure alerts

### Enable Alertmanager

**File:** `config-as-code/environments/egov-demo.yaml`

```yaml
prometheus:
  prometheusSpec:
    alertmanager:
      enabled: true
```

{% hint style="info" %}
This enables the Alertmanager shipped with the Prometheus Operator stack.
{% endhint %}

### Slack alerts

**File:** `config-as-code/environments/egov-demo-secrets.yaml`

* Create a [Slack Incoming Webhook](https://api.slack.com/messaging/webhooks).
* Update `slack_api_url` and the receiver `channel`.

Alerts will be sent for `warning` and `critical` severities.

```yaml
cluster-configs:
  secrets:
    alertmanager:
      config:
        global:
          slack_api_url: <slack-webhook-url>
          resolve_timeout: 5m
        route:
          group_by: ['alertname']
          group_wait: 30s
          group_interval: 5m
          repeat_interval: 10m
          routes:
            - receiver: slack-notification
              match_re:
                severity: "warning|critical"
              continue: true
        receivers:
          - name: slack-notification
            slack_configs:
              - channel: '<slack-channel>'
                send_resolved: true
                username: 'Alertmanager'
                title: |
                  [{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .CommonLabels.alertname }}
                text: |-
                  {{ range .Alerts -}}
                  {{- "\n" -}}
                  *Alert:* {{ .Annotations.summary }}
                  {{ if .Labels.severity }}*Severity:* `{{ .Labels.severity }}`{{ end }}
                  *Cluster:* {{ .Labels.cluster }}
                  *Details:*
                  {{ .Annotations.description }}
                  {{ end }}
                color: |-
                  {{ if eq .Status "firing" -}}
                    {{ if eq .CommonLabels.severity "warning" -}}
                      warning
                    {{- else if eq .CommonLabels.severity "critical" -}}
                      danger
                    {{- else -}}
                      #439FE0
                    {{- end -}}
                  {{ else -}}
                    good
                  {{- end }}
```

### Email alerts

**File:** `config-as-code/environments/egov-demo-secrets.yaml`

Configure:

* SMTP server (for example, Gmail)
* Sender and receiver email IDs
* SMTP auth token

```yaml
cluster-configs:
  secrets:
    alertmanager:
      config:
        global:
          resolve_timeout: 5m
        route:
          group_by: ['alertname']
          group_wait: 30s
          group_interval: 5m
          repeat_interval: 10m
          routes:
            - receiver: email-notification
              match_re:
                severity: "warning|critical"
              continue: true
        receivers:
          - name: email-notification
            email_configs:
              - to: '<recipient-email-address>'
                from: '<sender-email-address>'
                smarthost: 'smtp.gmail.com:587'
                auth_username: '<sender-email-address>'
                auth_password: '<auth-token>'
                send_resolved: true
                headers:
                  subject: |
                    [{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .CommonLabels.cluster }} - {{ .CommonLabels.alertname }}
                html: |
                  <html>
                  <head>
                  <title>Alert!</title>
                  </head>
                  <body>
                  {{ range .Alerts.Firing }}
                  <ul>
                  <li><b>Alert Name:</b> {{ .Labels.alertname }}</li>
                  <li><b>Severity:</b> {{ if eq .Labels.severity "critical" }}<b style="color:red;">CRITICAL</b>{{ else if eq .Labels.severity "warning" }}<b style="color:orange;">WARNING</b>{{ else }}<b>{{ .Labels.severity | toUpper }}</b>{{ end }}</li>
                  <li><b>Summary:</b> {{ .Annotations.summary }}</li>
                  <li><b>Cluster:</b> {{ .Labels.cluster }}</li>
                  <li><b>Details:</b>
                    <p style="margin-left: 20px;">{{ .Annotations.description | replace "\n" "<br>" }}</p>
                  </li>
                  </ul><br>
                  {{ end }}
                  </body></html>
```

{% hint style="info" %}
Gmail SMTP setup: follow [this guide](https://support.google.com/a/answer/176600?hl=en).
{% endhint %}


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.digit.org/health/v2.0/deploy/installation/production-setup/install-monitoring-stack/1.-configure.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.