> For the complete documentation index, see [llms.txt](https://docs.digit.org/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.digit.org/complaints-management/deploy/operations-handbook.md).

# Operations Handbook

DIGIT Complaint Management — Operations & Support Handbook

## **Overview**

**Audience:** the team that operates a deployed DIGIT Complaint Management instance — the IT or infrastructure staff inside the department, council or ministry that owns the server.

**Purpose:** to help you find out *what* broke without waiting for us, and to make it easy to hand us a report we can act on straight away.

## Start Here 

Check the table below to find the page that will help with the issues you are facing.

<table><thead><tr><th width="400.04296875">You are</th><th>Go to</th></tr></thead><tbody><tr><td><strong>On the support desk as L1 support personnel</strong>, someone has just reported a problem</td><td><a href="/complaints-management/deploy/operations-handbook/l1-first-response.md"><strong>L1 - first response</strong> </a>— the checklist</td></tr><tr><td><strong>Operating &#x26; maintaining the deployment</strong>, and a ticket has reached you from L1</td><td><a href="/complaints-management/deploy/operations-handbook/l2-diagnosis.md"><strong>L2 - diagnosis</strong></a> — the diagnosis runbook</td></tr><tr><td><strong>Looking for something already seen before</strong></td><td><a href="/complaints-management/deploy/operations-handbook/known-issues.md"><strong>Known issues</strong></a></td></tr><tr><td><strong>About to report something to L3/L4</strong></td><td><a href="/complaints-management/deploy/operations-handbook/incident-report.md"><strong>Incident Report</strong></a></td></tr><tr><td><strong>Mid-incident and want one page to refer to</strong></td><td><a href="/complaints-management/deploy/operations-handbook/cheat-sheet.md"><strong>Cheatsheet</strong></a></td></tr><tr><td><strong>Setting up monitoring</strong> so you hear about problems first</td><td><a href="/complaints-management/deploy/operations-handbook/setup-alerts.md"><strong>Alerts setup</strong> </a>and <a href="/complaints-management/deploy/operations-handbook/alert-channels.md"><strong>Alert channels</strong></a></td></tr><tr><td><strong>Looking at a Grafana dashboard</strong> and unsure what it means</td><td><a href="/complaints-management/deploy/operations-handbook/the-grafana-dashboards.md"><strong>Dashboards</strong></a></td></tr><tr><td><strong>Looking something up</strong></td><td><a href="/complaints-management/deploy/operations-handbook/reference.md"><strong>Reference</strong></a></td></tr></tbody></table>

This handbook assumes two support tiers — a first line (L1) and a second line (L2). In case this does not apply to your team, read the labels as "the checklist" and "the deep dive".

## Who Does What: L1 vs L2&#x20;

A simple rule decides almost every case:

{% hint style="info" %} <mark style="color:$primary;">**If there is a screen for it, it is L1. If it needs server access, a log file, a database, or a config file, it is L2.**</mark>
{% endhint %}

L1 is not only incidents. A large part of day-to-day operations is *requests* to add new master data or edit existing values — "add this clerk", "we have a new department", "this complaint type is missing" — and all of that is done through the product admin screens. These issues belong to the first line of support. Refer to the table below for details:

<table><thead><tr><th width="355.7109375">Work</th><th width="120.1796875">Tier</th><th>Why</th></tr></thead><tbody><tr><td>Creating and deactivating <strong>users / employees</strong> (HRMS screens)</td><td><strong>L1</strong></td><td>It has a UI. No server access, no configuration files</td></tr><tr><td>Adding or editing <strong>master data</strong> — departments, designations, complaint types, boundaries, localisation strings (Configurator / DIGIT Studio)</td><td><strong>L1</strong></td><td>Same. These are supported product screens</td></tr><tr><td>Bulk loading master data from the <strong>onboarding spreadsheets</strong></td><td><strong>L1</strong></td><td>Driven from the wizard; validation errors come back on screen</td></tr><tr><td>The <strong>first-response checklist</strong> — is it up, did something crash, what does the log say</td><td><strong>L1</strong></td><td>All of it is read-only, in a browser</td></tr><tr><td>Resolving anything on the <a href="/complaints-management/deploy/operations-handbook/known-issues.md"><strong>known-issues</strong> list</a></td><td><strong>L1</strong></td><td>The resolutions there are deliberately UI-only</td></tr><tr><td>Restarting a service, reading container logs, <code>docker</code>/<code>psql</code> on the box</td><td><strong>L2</strong></td><td>Server access</td></tr><tr><td>Anything about <strong>why</strong> — diagnosis, correlating services, tracing a request</td><td><strong>L2</strong></td><td>Complexity, and it needs the full stack in view</td></tr><tr><td><strong>Configuration</strong> the UI does not expose — env vars, nginx, Kong routes, compose files</td><td><strong>L2</strong></td><td>Not a screen; changes survive only through a redeploy</td></tr><tr><td><strong>Capacity</strong> — raising a memory limit, disk filling, log rotation</td><td><strong>L2</strong></td><td>Host-level, see <a href="/complaints-management/deploy/operations-handbook/l2-diagnosis.md">L2-diagnosis</a></td></tr><tr><td>Deploying, upgrading, restoring a backup</td><td><strong>L2</strong>, usually with us</td><td>Changes the deployment itself</td></tr></tbody></table>

**Two important takeaways:**

1. **Give L1 real access.** Give L1 support personnel an admin login for the Configurator and HRMS.&#x20;
2. **"Has a UI" is about the task, not the person.** If something needs a database edit because the screen simply can't do it, that's L2 work — and worth escalating to L3/L4, since it likely means a screen is missing.

Where exactly your team draws the line is still yours to set. This is the split the handbook assumes when it says "L1" and "L2".

### How a Problem Moves Through the System

```
   Report  ─▶  L1 first response  ─▶  known issue?  ─── yes ──▶  resolve, log it
                                            │
                                            no
                                            ▼
                                     Part A ─▶  L2 diagnosis  ─▶  config / restart fix
                                                      │
                                                 product defect
                                                 or deployment change
                                                      ▼
                                            Part A+B+C ─▶  L3/L4
```

The report template is **one document filled in three stages** — L1 completes Part A, L2 adds Part B, and Part C is added when it comes to us. Nothing is re-typed at a handover.

Where the boundary between the tiers sits — what L1 is allowed to do, who may restart a service, who carries the phone — is your team's call. This handbook provides a recommendation as a starting point.

### Four Key URLs

Replace `<your-domain>` with your deployment's domain throughout this handbook.

<table><thead><tr><th width="167.55859375">What</th><th width="312.17578125">URL</th><th>Use it for</th></tr></thead><tbody><tr><td><strong>Health dashboard</strong> (Gatus)</td><td><code>https://&#x3C;your-domain>/status/</code></td><td><em>Are all services up?</em> — up to 57 live checks in 12 groups, red/green, refreshed every 30s</td></tr><tr><td><strong>Grafana</strong></td><td><code>https://&#x3C;your-domain>/grafana/</code></td><td>Detailed metrics, logs, traces, alerts</td></tr><tr><td><strong>Logs</strong></td><td><code>https://&#x3C;your-domain>/grafana/d/digit-loki-logs/</code></td><td>The actual error text</td></tr><tr><td><strong>Service metrics</strong></td><td><code>https://&#x3C;your-domain>/grafana/d/digit-jvm/</code></td><td>Memory, CPU, restarts, OOM</td></tr></tbody></table>

Those four are all first-response needs. Grafana carries **nine dashboards** in total — including the database, the API gateway, the message broker and the supervisor dashboard's own timings. What each one shows and which panels to read is [**dashboards**.](/complaints-management/deploy/operations-handbook/the-grafana-dashboards.md)

{% hint style="info" %}
**Access note — who gives you a Grafana login.** Grafana asks for a login, and a fresh deployment has exactly **one** account: `admin`. That account belongs to your **system administrator** — its password is generated on the first deploy and stored in this deployment's OpenBao. Self-registration is disabled, so **the administrator creates a Grafana account for each L1 and L2 person** who needs one. Ask them for *their own* account rather than for the admin password; see [reference and Credentials](/complaints-management/deploy/operations-handbook/reference.md) for what to ask for.

**Ask for the Editor role.** Grafana hands new accounts the **Viewer** role by default, and a Viewer cannot open **Explore** — which [Step 4](/complaints-management/deploy/operations-handbook/l1-first-response.md#what-does-the-log-say) of the first response and most of L2's work depend on. **Editor** is the right role for the service desk. It still cannot change passwords, add users or edit datasources, and nothing in this handbook writes to the system — Grafana only displays data.

Anonymous access is off unless the deployment explicitly sets `grafana_anonymous_enabled: true`, and even then it grants **Viewer**, never Admin. If your deployment does have anonymous access on and is reachable from the public internet, raise it with us: a Viewer can run arbitrary Loki queries, and the logs contain live session tokens. See [alerts-setup > Before you start.](/complaints-management/deploy/operations-handbook/setup-alerts.md#before-you-start)

**Credentials.** The health dashboard needs no login; Grafana does, so the service desk needs accounts in advance — see above. For everything else — the Novu notification dashboard, the SMS/WhatsApp provider console, or server access — **ask your system administrator**. Never send credentials in a ticket or a chat message, including to us.
{% endhint %}

## Monitoring Scope&#x20;

**Not every deployment collects everything.** Monitoring costs memory and disk, so a deployment picks one of three **observability levels**. The levels are cumulative, and the default is the full stack:

<table><thead><tr><th width="177.58984375">Level</th><th width="379.16015625">You have</th><th>You do not have</th></tr></thead><tbody><tr><td><strong>metrics</strong></td><td>Grafana, health dashboard, host / database / broker / gateway metrics, and every dashboard except the two below</td><td><strong>Logs (Loki)</strong> and <strong>Traces (Tempo)</strong></td></tr><tr><td><strong>logs</strong></td><td>the above, plus searchable logs</td><td><strong>Traces (Tempo)</strong></td></tr><tr><td><strong>traces</strong> <em>(the default)</em></td><td>everything in this handbook</td><td>—</td></tr></tbody></table>

**Why this matters more than it sounds.** Reading the logs is [Step 4 of the first-response checklist](/complaints-management/deploy/operations-handbook/l1-first-response.md#what-does-the-log-say) and most of the second line's method. On a **metrics**-level deployment, there is no Loki to read them from — the log dashboard is simply not there. That is a deployment decision, not a fault, and nobody should spend an incident discovering it.

{% hint style="info" %}
**Find out which level your deployment runs and write it on your cheat sheet now** — it saves confusion during an actual incident. Changing levels requires a redeploy, so if you want more monitoring, just ask us or whoever is handling your deployment.
{% endhint %}

Changing the level is a redeploy, not a switch on the box. If you are on **metrics** and want logs, that is a reasonable thing to ask us for — say so, and say why.

### Good Incident Report Guidelines

Five facts, all available from a browser in a few minutes, let us start working immediately:

<table><thead><tr><th width="62.05078125">#</th><th width="375.109375">Question</th><th>Where you get it</th></tr></thead><tbody><tr><td>1</td><td><strong>What</strong> is broken (URL, screen, action, API)</td><td>your own observation</td></tr><tr><td>2</td><td><strong>When</strong> it started, and whether it's still happening</td><td>Health dashboard / Grafana time picker</td></tr><tr><td>3</td><td><strong>Who/how many</strong> are affected (one user, one office, everyone)</td><td>your own observation</td></tr><tr><td>4</td><td><strong>Which service</strong> is unhealthy or restarting</td><td>Health dashboard + Grafana</td></tr><tr><td>5</td><td><strong>The error text</strong> from that service's logs</td><td>Grafana → Logs (Loki)</td></tr></tbody></table>

If you can only get some of these, send what you have — a partial report sent quickly beats a full report sent late.

### Early Reporting Counts

Observability data on these deployments is deliberately short-lived, so it costs no disk. The trade-off is that **a problem reported late may no longer have any evidence behind it**.

| Data                                | Where      | Kept for                               | What that means for you                                          |
| ----------------------------------- | ---------- | -------------------------------------- | ---------------------------------------------------------------- |
| **Traces** (per-request timing)     | Tempo      | **24 hours**                           | Slowness reports older than a day can't be diagnosed from traces |
| **Logs**                            | Loki       | **72 hours (3 days)**                  | The error text is gone after 3 days                              |
| **Metrics** (memory, CPU, restarts) | Prometheus | **15 days**                            | Trends and restart history survive two weeks                     |
| Health check history                | Gatus      | **survives restarts** — SQLite on disk | Roughly the last 100 results and 50 events per endpoint          |

If a problem happened over a weekend, the logs may already have rolled off by Monday. Alter this depending on the deployment needs.

### Recommended Severity Levels

Use the recommended severity labels (given in the table below) in the subject line of your report; Response times and who is on call have to be defined per deployment.

<table><thead><tr><th width="161.390625">Severity</th><th width="300.66015625">Definition</th><th>Examples</th></tr></thead><tbody><tr><td><strong>S1 — Critical</strong></td><td>Citizens or staff cannot use the system at all</td><td>Site down, login broken for everyone, no complaint can be filed</td></tr><tr><td><strong>S2 — Major</strong></td><td>A core function is broken for many users, with no workaround</td><td>Complaints can be filed but not assigned; notifications not going out; dashboard empty for all supervisors</td></tr><tr><td><strong>S3 — Minor</strong></td><td>Degraded or broken for some users, workaround exists</td><td>One office's boundary missing; a report is slow; one screen mis-labelled</td></tr><tr><td><strong>S4 — Question / request</strong></td><td>Not broken; a change, a doubt, a new tenant, a data load</td><td>"How do I add a department?"</td></tr></tbody></table>

Keeping S1 for genuine full-stop outages is what keeps the S1 path fast for everyone.

**A quick check before escalating to S1:** confirm the problem isn't local to one machine — try a different network (mobile hotspot), a different browser in private mode, and a different user account. Client-side DNS and stale logins produce symptoms that look identical to an outage, and ruling them out takes a minute.

## Escalate To L3/L4

1. Fill in the template in the [**incident report** ](/complaints-management/deploy/operations-handbook/incident-report.md)— Parts A, B and C.
2. Attach the evidence it asks for (log excerpts, screenshots, trace IDs, the queries and time ranges you used).
3. Post it in the agreed support channel; for S1, also use the phone/escalation path.
4. **If you can, capture the evidence before restarting or redeploying.** A restart often clears the symptom and the logs together, which leaves us without much to work from. If you do need to restart to get users moving — that is usually the right call — just note in the report that you did, and when.

Details L3/L4 will ask for if they're missing, so they're worth including up front: the **exact time** (with timezone), the **tenant/city code**, the **complaint or user ID** involved, and whether it is **reproducible on demand**.

## Document Map

<table><thead><tr><th width="201.671875">Document</th><th width="136.24609375">For</th><th>Read it when</th></tr></thead><tbody><tr><td>Operations Handbook</td><td>everyone</td><td>Start here. Routing, severity, URLs, escalation</td></tr><tr><td><a href="/complaints-management/deploy/operations-handbook/cheat-sheet.md"><strong>Cheatsheet</strong></a></td><td>everyone</td><td>Mid-incident. One printable page</td></tr><tr><td><a href="/complaints-management/deploy/operations-handbook/l1-first-response.md"><strong>L1-first-response</strong></a></td><td>L1</td><td>A problem has just been reported. Checklist, what to capture, when to hand over</td></tr><tr><td><a href="/complaints-management/deploy/operations-handbook/l2-diagnosis.md"><strong>L2-diagnosis</strong></a></td><td>L2</td><td>A ticket has reached you. Layer model, deeper queries, host, server commands</td></tr><tr><td><a href="/complaints-management/deploy/operations-handbook/known-issues.md"><strong>Known-issues</strong></a></td><td>L1 + L2</td><td>Before diagnosing anything. Symptom → known resolution</td></tr><tr><td><a href="/complaints-management/deploy/operations-handbook/incident-report.md"><strong>Incident-report</strong></a></td><td>L1 + L2</td><td>At every handover. The three-part template and evidence checklist</td></tr><tr><td><a href="/complaints-management/deploy/operations-handbook/setup-alerts.md"><strong>Alerts-setup</strong></a></td><td>L2</td><td>Making the system tell you first. Rules, thresholds, provisioning</td></tr><tr><td><a href="/complaints-management/deploy/operations-handbook/alert-channels.md"><strong>Alert-channels</strong></a></td><td>L2</td><td>Where alerts land — Slack, Chat, Teams, email, WhatsApp, SMS</td></tr><tr><td><a href="/complaints-management/deploy/operations-handbook/the-grafana-dashboards.md"><strong>Dashboards</strong></a></td><td>L1 + L2</td><td>At a desk, with Grafana open. What each of the nine dashboards shows and which panels to read</td></tr><tr><td><a href="/complaints-management/deploy/operations-handbook/reference.md"><strong>Reference</strong></a></td><td>everyone</td><td>Lookup: service catalogue, coverage, retention, queries, glossary</td></tr></tbody></table>

## Out Of Scope&#x20;

This handbook is about **things going wrong**. The routine admin work in the table above is L1s, but it is documented elsewhere because it is not incident response:

* **Onboarding a city, boundaries, departments or employees** — see [`../migration/operator-runbook.md`](https://github.com/egovernments/Citizen-Complaint-Resolution-System/blob/master/docs/migration/operator-runbook.md) and the XLSX onboarding flow.
* **Changing application configuration** (complaint types, SLAs, branding) — that is the Configurator. Doing it is L1 work; it is just not an incident.
* **Deploying or upgrading** the stack — that is `local-setup/ansible/`, and normally ours.

If you're not sure whether your problem is an incident or a configuration question, file it as **S4,** and we will reclassify it.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.digit.org/complaints-management/deploy/operations-handbook.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.