1. Configure

Overview

Configure the monitoring stack via Helmfile: Grafana, Loki, Prometheus, and alerting.

Setup Helmfile environment

Define environments, namespace, and environment-specific values files.

File: config-as-code/helm/charts/monitoring/monitoring-helmfile.yaml

environments:
  demo:
    values:
      - namespace: monitoring
      - ../../../environments/egov-demo.yaml
      - ../../../environments/egov-demo-secrets.yaml

• demo is the environment name.

• monitoring is the namespace.

• Update file paths when you add environments.

Configure Grafana

1. Create GitHub OAuth App. Note the Client ID and Client Secret.

   • Homepage URL: https://<your_domain_name>

   • Authorization Callback URL: https://<your_domain_name>/monitoring/login/github

2. Update Grafana OAuth secrets.

   File: config-as-code/environments/egov-demo-secrets.yaml

   cluster-configs:
     secrets:
       grafana:
         clientID: <oauth-client-id>
         clientSecret: <oauth-client-secret>

3. Restrict access by GitHub org/team.

   File: config-as-code/environments/egov-demo.yaml

   grafana:
     github:
       allowed_organizations: ["<organization>"]
       role_attribute_path: contains(groups[*], '@<organization>/<team>') && 'Viewer'

Valid roles: None, Viewer, Editor, Admin, GrafanaAdmin. See Grafana GitHub OAuth documentation for advanced mappings.

Configure Loki (log storage)

Choose one storage option.

Option A: Local filesystem storage

File: config-as-code/environments/egov-demo.yaml

loki:
  persistence:
    enabled: true
    accessModes: [ReadWriteOnce]
    size: 15Gi
  serviceAccount:
    annotations: {}
  additionalConfigs:
    schema_config:
      configs:
        - from: 2020-10-24
          store: boltdb-shipper
          object_store: filesystem
          schema: v11
          index:
            prefix: index_
            period: 24h
    storage_config:
      boltdb_shipper:
        active_index_directory: /data/loki/index
        cache_location: /data/loki/index_cache
        shared_store: filesystem
        cache_ttl: 24h
      filesystem:
        directory: /data/loki/chunks
    compactor:
      working_directory: /data/loki/boltdb-shipper-compactor
      shared_store: filesystem
      retention_enabled: true
      compaction_interval: 72h
    table_manager:
      retention_deletes_enabled: true
      retention_period: 72h

Option B: AWS S3 storage

When using OIDC, use the sub claim (not aud) in IAM trust policies.

1. Create an IAM role with S3 access (policy example):

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Sid": "AccessToLokiBucket",
         "Effect": "Allow",
         "Action": ["s3:PutObject", "s3:GetObject", "s3:DeleteObject", "s3:ListBucket"],
         "Resource": ["arn:aws:s3:::<s3-bucket>", "arn:aws:s3:::<s3-bucket>/*"]
       }
     ]
   }

2. Update Loki config (example for EKS IRSA):

   File: config-as-code/environments/egov-demo.yaml

   loki:
     persistence:
       enabled: true
       accessModes: [ReadWriteOnce]
       size: 10Gi
     serviceAccount:
       annotations:
         eks.amazonaws.com/role-arn: <s3-role-arn>
     additionalConfigs:
       schema_config:
         configs:
           - from: 2020-10-24
             store: boltdb-shipper
             object_store: s3
             schema: v11
             index:
               prefix: index_
               period: 24h
       storage_config:
         boltdb_shipper:
           active_index_directory: /data/loki/index
           cache_location: /data/loki/index_cache
           shared_store: s3
           cache_ttl: 24h
         aws:
           s3: s3://<region>/<s3-bucket>
       compactor:
         working_directory: /data/loki/boltdb-shipper-compactor
         shared_store: s3
         retention_enabled: true
         compaction_interval: 168h
       table_manager:
         retention_deletes_enabled: true
         retention_period: 168h

Option C: Azure Blob storage

File: config-as-code/environments/egov-demo.yaml

loki:
  persistence:
    enabled: true
    accessModes: [ReadWriteOnce]
    size: 15Gi
  serviceAccount:
    annotations: {}
  additionalConfigs:
    schema_config:
      configs:
        - from: 2020-10-24
          store: boltdb-shipper
          object_store: azure
          schema: v11
          index:
            prefix: index_
            period: 24h
    storage_config:
      azure:
        account_name: <storage-account-name>
        account_key: <storage-account-key>
        container_name: <container-name>
        # use_managed_identity: <true|false>
        # user_assigned_id: <user-assigned-identity-id>
        request_timeout: 0
        # Use this for private clouds (example shown).
        # endpoint_suffix: <endpoint-suffix>
      boltdb_shipper:
        active_index_directory: /data/loki/index
        cache_location: /data/loki/index_cache
        shared_store: azure
        cache_ttl: 24h
    compactor:
      working_directory: /data/loki/boltdb-shipper-compactor
      shared_store: azure
      retention_enabled: true
      compaction_interval: 72h
    table_manager:
      retention_deletes_enabled: true
      retention_period: 72h

See the Loki configuration docs for all supported options.

Configure Prometheus

File: config-as-code/environments/egov-demo.yaml

prometheus:
  prometheusSpec:
    retention: 7d
    storageSpec:
      volumeClaimTemplate:
        spec:
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 30Gi
    alertmanager:
      enabled: false
    externalLabels:
      cluster: <cluster-name>
    additionalScrapeConfigs:
      - job_name: 'nginx-ingress-metrics'
        static_configs:
          - targets: ['ingress-nginx-controller-metrics.egov:10254']
      - job_name: 'redis-exporter'
        static_configs:
          - targets: ['prometheus-redis-exporter.backbone:9121']
      - job_name: 'blackbox'
        metrics_path: /probe
        params:
          module: [http_2xx]
        static_configs:
          - targets:
              - <url-to-monitor>
        relabel_configs:
          - source_labels: [__address__]
            target_label: __param_target
          - source_labels: [__param_target]
            target_label: instance
          - target_label: __address__
            replacement: blackbox-prometheus-blackbox-exporter:9115
      - job_name: 'blackbox_exporter'
        static_configs:
          - targets: ['blackbox-prometheus-blackbox-exporter:9115']

Additional scrape targets

• NGINX Ingress

• Redis exporter

• Blackbox monitoring (URLs, APIs)

Update targets to match your cluster and namespaces.

Configure alerts

Enable Alertmanager

File: config-as-code/environments/egov-demo.yaml

prometheus:
  prometheusSpec:
    alertmanager:
      enabled: true

This enables the Alertmanager shipped with the Prometheus Operator stack.

Slack alerts

File: config-as-code/environments/egov-demo-secrets.yaml

• Create a Slack Incoming Webhook.

• Update slack_api_url and the receiver channel.

Alerts will be sent for warning and critical severities.

cluster-configs:
  secrets:
    alertmanager:
      config:
        global:
          slack_api_url: <slack-webhook-url>
          resolve_timeout: 5m
        route:
          group_by: ['alertname']
          group_wait: 30s
          group_interval: 5m
          repeat_interval: 10m
          routes:
            - receiver: slack-notification
              match_re:
                severity: "warning|critical"
              continue: true
        receivers:
          - name: slack-notification
            slack_configs:
              - channel: '<slack-channel>'
                send_resolved: true
                username: 'Alertmanager'
                title: |
                  [{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .CommonLabels.alertname }}
                text: |-
                  {{ range .Alerts -}}
                  {{- "\n" -}}
                  *Alert:* {{ .Annotations.summary }}
                  {{ if .Labels.severity }}*Severity:* `{{ .Labels.severity }}`{{ end }}
                  *Cluster:* {{ .Labels.cluster }}
                  *Details:*
                  {{ .Annotations.description }}
                  {{ end }}
                color: |-
                  {{ if eq .Status "firing" -}}
                    {{ if eq .CommonLabels.severity "warning" -}}
                      warning
                    {{- else if eq .CommonLabels.severity "critical" -}}
                      danger
                    {{- else -}}
                      #439FE0
                    {{- end -}}
                  {{ else -}}
                    good
                  {{- end }}

Email alerts

File: config-as-code/environments/egov-demo-secrets.yaml

Configure:

• SMTP server (for example, Gmail)

• Sender and receiver email IDs

• SMTP auth token

cluster-configs:
  secrets:
    alertmanager:
      config:
        global:
          resolve_timeout: 5m
        route:
          group_by: ['alertname']
          group_wait: 30s
          group_interval: 5m
          repeat_interval: 10m
          routes:
            - receiver: email-notification
              match_re:
                severity: "warning|critical"
              continue: true
        receivers:
          - name: email-notification
            email_configs:
              - to: '<recipient-email-address>'
                from: '<sender-email-address>'
                smarthost: 'smtp.gmail.com:587'
                auth_username: '<sender-email-address>'
                auth_password: '<auth-token>'
                send_resolved: true
                headers:
                  subject: |
                    [{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .CommonLabels.cluster }} - {{ .CommonLabels.alertname }}
                html: |
                  <html>
                  <head>
                  <title>Alert!</title>
                  </head>
                  <body>
                  {{ range .Alerts.Firing }}
                  <ul>
                  <li><b>Alert Name:</b> {{ .Labels.alertname }}</li>
                  <li><b>Severity:</b> {{ if eq .Labels.severity "critical" }}<b style="color:red;">CRITICAL</b>{{ else if eq .Labels.severity "warning" }}<b style="color:orange;">WARNING</b>{{ else }}<b>{{ .Labels.severity | toUpper }}</b>{{ end }}</li>
                  <li><b>Summary:</b> {{ .Annotations.summary }}</li>
                  <li><b>Cluster:</b> {{ .Labels.cluster }}</li>
                  <li><b>Details:</b>
                    <p style="margin-left: 20px;">{{ .Annotations.description | replace "\n" "<br>" }}</p>
                  </li>
                  </ul><br>
                  {{ end }}
                  </body></html>

Gmail SMTP setup: follow this guide.