> For the complete documentation index, see [llms.txt](https://docs.digit.org/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.digit.org/complaints-management/deploy/operations-handbook/the-grafana-dashboards.md).

# The Grafana Dashboards

What each dashboard shows, which panels to read, and what a reading means.

This page was split out of [reference ](/complaints-management/deploy/operations-handbook/reference.md)when the deployment grew from four dashboards to nine. It is the page to have open at a desk; reference.md is the lookup.

## Terms On Dashboard

Grafana is one website showing several collections of data. A few terms recur, and knowing them makes every dashboard readable:

<table><thead><tr><th width="126.89453125">Term</th><th>What it means</th></tr></thead><tbody><tr><td><strong>Dashboard</strong></td><td>One page of charts about one subject. Pick it from <strong>Dashboards</strong> in the left menu, or open it by URL</td></tr><tr><td><strong>Panel</strong></td><td>One box on that page — a single chart, number or table. Each panel answers one question</td></tr><tr><td><strong>Row</strong></td><td>A collapsible heading holding a group of panels. <strong>A collapsed row hides its panels entirely</strong> — click the heading to expand. Several dashboards below ship with most rows collapsed</td></tr><tr><td><strong>Time range</strong></td><td><strong>Top right of every dashboard.</strong> Everything on the page describes the window you choose. <strong>Getting this right matters more than anything else</strong>: if the problem happened at 09:15 and your range is "Last 5 minutes", every panel will look perfectly healthy</td></tr><tr><td><strong>Datasource</strong></td><td>Where a panel's numbers come from. Three exist: <strong>Prometheus</strong> (measurements over time), <strong>Loki</strong> (log messages), <strong>Tempo</strong> (request timings)</td></tr><tr><td><strong>Variable</strong></td><td>A dropdown across the top that filters the whole page — by service, tenant, consumer group. Leave it at its default until you have a reason not to</td></tr><tr><td><strong>Explore</strong></td><td>A left-menu item: a blank page where you pick a datasource and run one query. Used when this handbook gives you a query to paste</td></tr><tr><td><strong>Refresh</strong></td><td>Dashboards do not update by themselves unless set to. Use the refresh icon beside the time range if you are watching something live</td></tr></tbody></table>

**Panel shapes**, because the same data looks different depending on the shape:

<table><thead><tr><th width="157.40625">Shape</th><th width="167.77734375">Looks like</th><th>Read it as</th></tr></thead><tbody><tr><td><strong>Stat</strong></td><td>One big number</td><td>A total or current value for the whole time range. Fast to read, no detail</td></tr><tr><td><strong>Timeseries / graph</strong></td><td>A line graph, time along the bottom</td><td>How something changed. One coloured line per service, named in the legend</td></tr><tr><td><strong>Table</strong></td><td>Rows and columns</td><td>One row per service, several values side by side. Best for comparing</td></tr><tr><td><strong>Logs</strong></td><td>Scrolling text lines with timestamps</td><td>The actual messages the software wrote</td></tr><tr><td><strong>Heatmap</strong></td><td>Coloured grid</td><td>Density over time. Rare here; used on one imported dashboard</td></tr></tbody></table>

**Jargon that appears in the panel names themselves:**

* **JVM** — the Java runtime. Most services here are Java programs, each in its own JVM.
* **Heap** — the memory a Java service is allowed to use for its work, with a fixed ceiling. Normal behaviour is for heap use to rise and fall repeatedly.
* **OOM ("out of memory")** — the service needed more than its heap ceiling allowed and **crashed**. Not a slowdown; a stop.
* **GC ("garbage collection")** — the JVM periodically clears memory it no longer needs. Routine and constant. It only matters when a service spends so much time doing it that it has little left for serving requests — then it *looks* frozen while technically alive.
* **Thread** — one unit of work happening inside a service. Many run at once.
* **Trace** — the record of one request as it travelled through several services, with the time each step took.
* **Lag** — how far behind a message consumer is. Growing lag means a stuck pipeline.
* **p95 / p99** — "95% of requests were faster than this". A better measure of what users feel than an average, which one very slow request can drag around.

## Nine Dashboards & Who Reads It

| Dashboard                                                                                 | Answers                                                 | Read it when                                          | Tier                           |
| ----------------------------------------------------------------------------------------- | ------------------------------------------------------- | ----------------------------------------------------- | ------------------------------ |
| [**DIGIT — Logs (Loki)**](#digit-logs-loki-grafana-d-digit-loki-logs)                     | *What is the actual error?*                             | Always. The most valuable page on the system          | L1 + L2                        |
| [**DIGIT JVM Services**](#digit-jvm-services-grafana-d-digit-jvm)                         | *Did something crash, restart or run out of memory?*    | Every incident                                        | L1 reads 2 panels; L2 the rest |
| [**Node Exporter Full**](#node-exporter-full-grafana-d-node-exporter-full)                | *Is the server itself running out of something?*        | Anything slow or widespread; **any disk question**    | L2                             |
| [**PostgreSQL Database**](#postgresql-database-grafana-d-postgres-database)               | *Is the database the bottleneck?*                       | "Everything is slow but nothing is down"              | L2                             |
| [**Kong API Gateway**](#kong-api-gateway-grafana-d-kong-gateway)                          | *Is it the gateway, or the service behind it?*          | 502 / 504, or one API failing                         | L2                             |
| [**Redpanda (Kafka) Broker**](#redpanda-kafka-broker-grafana-d-redpanda-broker)           | *Is the message broker healthy, and does it have disk?* | Notifications or indexing stalled                     | L2                             |
| [**DIGIT Kafka Consumer Lag**](#digit-kafka-consumer-lag-grafana-d-kafka-consumer-lag)    | *Which pipeline is stuck, and by how much?*             | Complaint filed but not in the inbox; no notification | L2                             |
| [**DIGIT — Traces (Tempo)**](#digit-traces-tempo-grafana-d-digit-tempo-traces)            | *Why was this slow, and which step took the time?*      | A slowness report, within 24h                         | L2 (L1 copies a Trace ID)      |
| [**DIGIT — CMS Analytics Queries**](#digit-cms-analytics-queries-grafana-d-pgr-analytics) | *Why is the supervisor dashboard slow?*                 | Reports of a slow or empty dashboard                  | L2                             |

{% hint style="info" %}
**Not every deployment has all nine.** Which ones carry data depends on this deployment's **observability level** and on which optional services it runs — see [If a dashboard is empty.](#if-a-dashboard-is-empty)
{% endhint %}

## Dashboards Written For DIGIT

These five were written for this system, in the same terms this handbook uses. Every panel on them is meant to be read.

### DIGIT JVM Services — `/grafana/d/digit-jvm/`

**What it is:** the memory and health of the Java services, one line or row per service.

**The question it answers:** *did something crash, restart, or run out of memory — and when?*

**Who uses it:** L1 reads two panels from it (see [l1-first-response](/complaints-management/deploy/operations-handbook/l1-first-response.md)). L2 uses the rest.

<table><thead><tr><th width="166.1796875">Panel</th><th width="130.21484375">Shape</th><th>What it shows and what the reading means</th></tr></thead><tbody><tr><td><strong>OOM events (current range)</strong></td><td>Stat</td><td>A count of out-of-memory crashes in your time range. <strong><code>0</code> is the healthy answer.</strong> Above <code>0</code>, <strong>check the panel below before believing it</strong> — this panel's query does not exclude Grafana and Loki, which log your own search text back into the stream, so searching for "OutOfMemoryError" inflates it. A reading in the hundreds with every incident line naming <code>grafana</code> or <code>loki</code> means zero real crashes</td></tr><tr><td><strong>Incidents — OOM / heap-space errors (last range)</strong></td><td>Logs</td><td>The actual crash messages behind the number above, naming the service. If the stat panel is above zero, copy your evidence from here</td></tr><tr><td><strong>Right-sizing snapshot — heap profile per service (heap, MB)</strong></td><td>Table</td><td>One row per service: memory in use now, its peak in the last hour, its ceiling. The <strong>headroom</strong> column is the useful one — the percentage of its allowance still free. Low headroom means the service is close to crashing. A capacity judgement, so L2's to act on</td></tr><tr><td><strong>Heap used (MB) — by service</strong></td><td>Timeseries</td><td>Memory in use over time. A healthy line <strong>rises and falls repeatedly</strong> — that is normal. A line that <strong>drops to zero and climbs again from the bottom</strong> is a service that crashed and restarted at that moment</td></tr><tr><td><strong>Heap committed vs used (MB) — by service</strong></td><td>Timeseries</td><td>Two lines per service: memory reserved versus actually used. When "used" sits against "committed" for a long time, the service is under real pressure</td></tr><tr><td><strong>JVM CPU — recent utilization (ratio)</strong></td><td>Timeseries</td><td>Processor per service. <strong><code>1.0</code> is one whole CPU core.</strong> <code>0.05</code> is idle chatter. High with no users on the system means stuck in a loop</td></tr><tr><td><strong>GC pause time (s/s)</strong></td><td>Timeseries</td><td>Seconds per second spent on garbage collection. <code>0.05</code> is routine. <strong>Sustained above about <code>0.2</code></strong> means a fifth of its life is spent tidying memory rather than working — users experience this as hanging</td></tr><tr><td><strong>Live threads</strong></td><td>Timeseries</td><td>How many pieces of work are in flight. Steady is fine. <strong>Climbing and never coming back down</strong> means work is piling up, usually because something it depends on is slow or dead</td></tr><tr><td><strong>Loaded classes</strong></td><td>Timeseries</td><td>How much program code is loaded. Rarely useful in an incident; it flattens shortly after start-up</td></tr></tbody></table>

{% hint style="info" %}
**Coverage.** This dashboard covers the **15 Java services carrying the OpenTelemetry agent** only — see [reference](/complaints-management/deploy/operations-handbook/reference.md) for the list and for what covers the rest.
{% endhint %}

### DIGIT — Logs (Loki) — `/grafana/d/digit-loki-logs/`

**What it is:** the messages the software writes as it runs — the closest thing to the system explaining what went wrong, in its own words.

**The question it answers:** *what is the actual error?*

**Who uses it:** everyone. Unlike the JVM dashboard, it covers **every container**, not just the Java ones.

**Three controls sit across the top.** You do not need to write a query — set these and read:

<figure><img src="https://github.com/egovernments/Citizen-Complaint-Resolution-System/raw/master/docs/operations/images/30-logs-controls.png" alt=""><figcaption></figcaption></figure>

<table><thead><tr><th width="162.88671875">Control</th><th>What to do with it</th></tr></thead><tbody><tr><td><strong>Service</strong></td><td>Which service's messages to show. Leave it as <code>.+</code> to see all of them</td></tr><tr><td><strong>Level</strong></td><td>How serious a message is. Set it to <strong><code>ERROR</code></strong> to hide routine chatter. <code>WARN</code> is "worth noticing", <code>INFO</code> is normal commentary</td></tr><tr><td><strong>Search regex</strong></td><td>A free-text filter. Paste a complaint number or user ID here to follow one case across every service that touched it</td></tr></tbody></table>

<table><thead><tr><th width="151.265625">Panel</th><th width="119.046875">Shape</th><th>What it shows and what the reading means</th></tr></thead><tbody><tr><td><strong>Log lines (current range)</strong></td><td>Stat</td><td>How many messages were written in your range. Context only — a large number is not itself a problem</td></tr><tr><td><strong>Errors / Exceptions (current range)</strong></td><td>Stat</td><td>How many were failures. <strong>The "how bad is it" number.</strong> Compare against a quiet period to judge whether it is unusual</td></tr><tr><td><strong>Log volume by service (rate / sec)</strong></td><td>Timeseries</td><td>How talkative each service is. <strong>Both directions are signals</strong>: a spike means something started failing repeatedly; a line <strong>dropping to silence</strong> means a service stopped running</td></tr><tr><td><strong>Logs</strong></td><td>Logs</td><td>The messages themselves, newest at the top. <strong>Scroll to the oldest one in your range and read that first</strong> — the earliest error is the cause, the rest are consequences</td></tr></tbody></table>

### DIGIT — Traces (Tempo) — `/grafana/d/digit-tempo-traces/`

**What it is:** timings for individual requests. A **trace** follows one click — one complaint submission, say — through every service it touched, and shows how long each step took.

**The question it answers:** *why was this slow, and which step took the time?*

**Who uses it:** mainly L2. L1's involvement is usually to copy a **Trace ID** into the ticket, which is enormously useful and takes seconds.

<table><thead><tr><th width="158.8046875">Panel</th><th width="130.6640625">Shape</th><th>What it shows and what the reading means</th></tr></thead><tbody><tr><td><strong>Services with traces (last 30m)</strong></td><td>Stat</td><td>How many services are currently reporting timings. A sanity check that trace collection works at all</td></tr><tr><td><strong>Recent traces</strong></td><td>Table</td><td>The most recent requests. Click any row to open its breakdown</td></tr><tr><td><strong>Slow traces — duration > 500ms (last range)</strong></td><td>Table</td><td>Requests that took longer than half a second. <strong>The panel to use for a "the system is slow" report.</strong> Click a row to see which service and which database call consumed the time</td></tr><tr><td><strong>Error traces (status = error)</strong></td><td>Table</td><td>Requests that failed outright rather than merely being slow</td></tr><tr><td><strong>Trace by ID</strong></td><td>Trace view</td><td>Paste a Trace ID to see one request as a waterfall — each service a bar, its width the time spent</td></tr></tbody></table>

{% hint style="info" %}
**Traces are kept for 24 hours only.** A slowness report raised on Monday about Friday cannot be answered from traces. Copy the Trace ID into the ticket the same day.
{% endhint %}

### DIGIT Kafka Consumer Lag — `/grafana/d/kafka-consumer-lag/`

**What it is:** how far behind each background pipeline is.

**The words first.** Complaints, notifications and search indexing do not happen inside the click that triggers them. The click writes a **message** onto a queue, and a separate service — a **consumer** — picks it up afterwards and does the work. Consumers are organised into **consumer groups**, one per job. **Lag** is the number of messages written but not yet picked up.

**The question it answers:** *a complaint was filed — why hasn't it appeared / notified / been indexed?*

**Why it matters:** this is the failure that looks like nothing is wrong. Every service is green, the complaint saved successfully, and it simply never shows up in the inbox. Lag is the only signal that says so.

**Who uses it:** L2.

<table><thead><tr><th width="163.77734375">Panel</th><th width="119.19921875">Shape</th><th>What it shows and what the reading means</th></tr></thead><tbody><tr><td><strong>Total lag by consumer group</strong></td><td>Timeseries</td><td>Messages waiting, per pipeline, over time. <strong>The shape is what matters, not the number.</strong> Lag that rises and falls is a pipeline keeping up with bursts. Lag that <strong>climbs and never comes back down</strong> is a stuck consumer — that is the finding</td></tr><tr><td><strong>Current lag by group</strong></td><td>Stat</td><td>The same figure right now, one number per group. Use it to say "<code>egov-indexer</code> is 40,000 messages behind" in a report</td></tr><tr><td><strong>Lag by topic and partition</strong></td><td>Table</td><td>Which specific queue is backed up. A single partition lagging while the others are fine usually means one poisonous message the consumer keeps failing on</td></tr><tr><td><strong>Consumers per group</strong></td><td>Timeseries</td><td>How many workers are attached to each group. <strong>Dropping to <code>0</code> means nothing is processing that queue at all</strong> — the consumer died or lost its connection to the broker</td></tr></tbody></table>

<figure><img src="https://github.com/egovernments/Citizen-Complaint-Resolution-System/raw/master/docs/operations/images/50-consumers-per-group.png" alt=""><figcaption></figcaption></figure>

*One line per pipeline, each sitting at a steady number. The two narrow dips toward the bottom are a consumer briefly detaching and rejoining — normal. A line that **reaches zero and stays there** is the failure this panel exists to catch.*

There is a **group** dropdown at the top to focus on one pipeline.

{% hint style="info" %}
**This dashboard measures lag at the broker**, by comparing the newest message on each queue against the last one each group confirmed. That covers **every** consumer group, including services with no Java instrumentation. There is also a *client-side* lag metric (`kafka_consumer_records_lag_max`) reported by the instrumented Java services only; it agrees with this dashboard where both exist, but it goes blank when a consumer dies — which is exactly when you need it. **Prefer this dashboard.**
{% endhint %}

The usual suspects when lag climbs: `egov-indexer` (complaint not in the inbox), `egov-persister` (complaint appears to save but never persists), `novu-bridge` (no notification sent).

### DIGIT — CMS Analytics Queries — `/grafana/d/pgr-analytics/`

**What it is:** how long the supervisor dashboard's own database queries take.

**The words first.** The dashboard supervisors look at is built from a set of measurements — complaints by status, by ward, resolution times. Each one is a **KPI**, and each is a separate database query identified by a **`kpi_id`**. **Grain** is the time bucket a KPI is rolled up into — by day, by week, by month; coarser grains scan more rows.

**The question it answers:** *the supervisor dashboard is slow — which measurement is slow, and for which city?*

**Who uses it:** L2, when a report says the dashboard is slow or times out.

Three dropdowns filter the page: **tenant**, **kpi\_id** and **grain**.

<table><thead><tr><th width="176.34375">Panel</th><th width="138.42578125">Shape</th><th>What it shows and what the reading means</th></tr></thead><tbody><tr><td><strong>Query duration percentiles</strong></td><td>Timeseries</td><td>The p50 / p95 / p99 of all analytics queries. <strong>Watch p95</strong> — if p50 is flat and p95 climbs, most users are fine and a minority wait a long time, which is what generates complaints</td></tr><tr><td><strong>p95 duration by KPI</strong></td><td>Timeseries</td><td>The same split per measurement. This is the panel that names the culprit</td></tr><tr><td><strong>Query rate by KPI</strong></td><td>Timeseries</td><td>How often each is run. A slow query that runs rarely matters less than a middling one that runs constantly</td></tr><tr><td><strong>Rows scanned per second</strong></td><td>Timeseries</td><td>How much data is being read. A rise here with no rise in query rate means the dataset grew — the query did not get worse, the city did</td></tr><tr><td><strong>Duration by planner grain</strong></td><td>Timeseries</td><td>Duration split by time bucket. Confirms whether the slowness is specific to one grain, which points at the query plan rather than the data</td></tr><tr><td><strong>Slowest KPIs by mean duration</strong></td><td>Table</td><td>Top 5, ranked. Start here and read the panels above for context</td></tr><tr><td><strong>Queries executed (selected range)</strong></td><td>Stat</td><td>Volume over the window — context for everything else</td></tr><tr><td><strong>Rows scanned (selected range)</strong></td><td>Stat</td><td>Total data read over the window</td></tr></tbody></table>

{% hint style="info" %}
**A blank page here is normal on a quiet deployment.** These metrics are only produced when somebody actually opens the supervisor dashboard. Widen the time range to 24 hours before concluding anything is wrong.
{% endhint %}

## Dashboards Imported From Community

The next four were **not** written for DIGIT. They are standard dashboards for Linux hosts, Postgres, Redpanda and Kong, published by those projects and imported as they are.

That has one consequence worth stating plainly:

{% hint style="info" %}
**Most of the panels on these pages are not for you.** They are written for someone tuning that specific piece of software, and they use its internal vocabulary. Reading all of them is not a goal. Each section below names **the few panels worth opening during an incident** and says what the rest is. Ignoring the remainder is the correct use of these pages, not a shortcut.
{% endhint %}

They also behave slightly differently: **most of their panels are inside collapsed rows**, so the page looks short until you expand one.

### Node Exporter Full — `/grafana/d/node-exporter-full/`

**What it is:** the health of the machine everything runs on — processor, memory, and above all **disk space**.

**The question it answers:** *is the server itself running out of something?*

**Who uses it:** L2.

**The readings that matter** (the rest of this dashboard is network, hardware and kernel detail you will not need):

<table><thead><tr><th width="150.0859375">Reading</th><th width="182.91796875">Concerning when</th><th>Why it matters</th></tr></thead><tbody><tr><td><strong>Disk Space Used</strong></td><td>Above <strong>85%</strong> (warning), above <strong>95%</strong> (critical)</td><td><strong>The most common cause of a complete outage.</strong> Postgres refuses writes when the disk fills, so complaints stop saving. Check this first, every time</td></tr><tr><td><strong>RAM Used</strong></td><td>Above 90%</td><td>The operating system begins killing containers to reclaim memory, usually the largest one, without warning</td></tr><tr><td><strong>CPU Busy</strong></td><td>Above 90% sustained</td><td>Everything slows; health checks time out and services get killed for being unresponsive</td></tr><tr><td><strong>Swap</strong></td><td>Any sustained use</td><td>The machine has run out of real memory and is using disk instead. Everything becomes dramatically slower. <strong>Note:</strong> many of these boxes have no swap configured at all, in which case this panel is permanently empty — that is not a fault</td></tr><tr><td><strong>Load average</strong></td><td>Higher than the CPU core count, sustained</td><td>More work queued than the machine can get through</td></tr></tbody></table>

{% hint style="info" %}
**If this dashboard is empty, that is not an incident** — it has two possible causes, and both require L2 support. Refer to [If a dashboard is empty](#if-a-dashboard-is-empty).
{% endhint %}

### PostgreSQL Database — `/grafana/d/postgres-database/`

**What it is:** the database's own view of itself — sessions, cache, locks, transactions.

**The question it answers:** *is the database the reason everything is slow?*

**Why it matters:** the health dashboard only proves Postgres accepts a connection. It stays green while the database is deadlocked, out of connections, or reading every query from disk. This page is where "everything is slow, but nothing is down" gets explained.

**Who uses it:** L2.

**The page has around 35 panels in four rows.** Five of them answer almost every question:

<table><thead><tr><th width="157.53125">Panel</th><th width="166.82421875">Row</th><th>What the reading means</th></tr></thead><tbody><tr><td><strong>Active sessions</strong></td><td>Database Stats</td><td>Queries running right now. Compare against <strong>Max Connections</strong> in the <em>Settings</em> row — approaching it means new work will start being refused. A sustained climb with no matching rise in traffic means queries are piling up rather than finishing</td></tr><tr><td><strong>Cache Hit Rate</strong></td><td>Database Stats</td><td>The share of reads served from memory rather than disk. <strong>Healthy is above ~99%</strong> — a well-fed database rarely touches disk. A drop is the clearest single sign the database has become the bottleneck</td></tr><tr><td><strong>Conflicts/Deadlocks</strong></td><td>Database Stats</td><td>Transactions killed because they blocked each other. <strong>Should be flat at zero.</strong> Anything non-zero is a real finding and belongs in a report — it names a code-level problem, not a capacity one</td></tr><tr><td><strong>Lock tables</strong></td><td>Database Stats</td><td>Locks held. A rising, non-draining count means something is holding a transaction open and everything else is queuing behind it</td></tr><tr><td><strong>Temp File (Bytes)</strong></td><td>Database Stats</td><td>Data spilled to disk because a query would not fit in memory. Sustained non-zero values mean expensive queries — often the analytics ones</td></tr></tbody></table>

**Also useful, occasionally:** **Idle sessions** (many idle connections that never close is a leak in a service, not a database fault), and **Checkpoint Stats** (if checkpoints are constant, write volume is high).

**Everything else on the page** — the *Settings* row, bgwriter buffers, WAL sizing, page-cost tuning — is for tuning a database, not for diagnosing an incident. Leave it alone.

{% hint style="info" %}
This dashboard reads the **database itself** (`postgres-db`), not the pooler in front of it. The two are separate on purpose; see the note in [reference](/complaints-management/deploy/operations-handbook/reference.md).
{% endhint %}

### Redpanda (Kafka) Broker — `/grafana/d/redpanda-broker/`

**What it is:** the health of the message broker that carries complaints, notifications and indexing between services.

**The question it answers:** *is the broker itself healthy, and does it have disk?*

**Who uses it:** L2. For "which pipeline is stuck", use [DIGIT Kafka Consumer Lag ](#digit-kafka-consumer-lag-grafana-d-kafka-consumer-lag)instead — this page is about the broker; that one is about the consumers.

**The page is large and mostly internal.** Four things are worth reading:

<table><thead><tr><th width="176.61328125">Panel</th><th width="168.76953125">Where</th><th>What the reading means</th></tr></thead><tbody><tr><td><strong>Nodes Up</strong></td><td>Top of page</td><td>How many brokers are running. On a single-server deployment this is <code>1</code>. <strong><code>0</code> is a total pipeline outage</strong> — nothing is being carried at all</td></tr><tr><td><strong>Disk storage bytes free</strong></td><td><em>storage</em> row (collapsed)</td><td>Space left for messages. <strong>The broker stops accepting writes when it fills</strong>, which stops complaints being indexed and notifications being sent. This is a slow-moving number: check it during any disk investigation</td></tr><tr><td><strong>Status of low storage space alert</strong></td><td><em>storage</em> row (collapsed)</td><td>The broker's own verdict on the line above. <strong><code>0</code> is OK, <code>1</code> is low space, <code>2</code> is degraded.</strong> The single fastest read on this page</td></tr><tr><td><strong>Throughput of Kafka produce/consume requests</strong></td><td>Top of page</td><td>Messages moving in and out. A drop to flat zero while the system is in use means the pipeline has stopped</td></tr></tbody></table>

The produce/consume **latency p95/p99** graphs at the top are worth a glance if the broker is suspected of being slow rather than stuck.

**Everything else** — the *raft*, *scheduler*, *io\_queue*, *memory* and *other* rows — is Redpanda's internal engine instrumentation, written for the Redpanda project's own engineers. If you find yourself reading it during an incident, the answer you want is almost certainly on the consumer-lag dashboard instead.

### Kong API Gateway — `/grafana/d/kong-gateway/`

**What it is:** the traffic director every API request passes through, measured at the door.

**The question it answers:** *is the gateway the problem, or the service behind it?*

**Why it matters:** this is the fastest answer to a 502, a 504, or "one screen is slow". Kong measures both halves of every request separately, so it can tell you which half is at fault without opening a single log.

**Who uses it:** L2.

**The page is six collapsed rows.** Two of them matter:

**Row&#x20;*****Latencies*****&#x20;— the important one.** It carries two different measurements, and the distinction is the whole value of this dashboard:

<table><thead><tr><th width="189.37109375">Panel</th><th width="216.6796875">What it measures</th><th>What it means when it rises</th></tr></thead><tbody><tr><td><strong>Kong Proxy Latency across all services</strong></td><td>Time spent <strong>inside Kong itself</strong> — routing, authentication, plugins</td><td>The gateway is the bottleneck. Rare, and a genuine finding. Normally a small number — single-digit to low-tens of milliseconds</td></tr><tr><td><strong>Upstream time across all services</strong></td><td>Time Kong spent <strong>waiting for the service behind it</strong></td><td>The service is the bottleneck, not the gateway. This is the usual answer — and it tells you to go and look at that service instead</td></tr><tr><td><em>…per Service</em> / <em>…per Route</em></td><td>The same two splits, broken down</td><td>Names the specific API that is slow. Use these once the totals tell you which half to look at</td></tr></tbody></table>

**What the pair looks like in practice.** Both panels below cover the same six hours on the same deployment — note the vertical scales, which are the whole point:

<figure><img src="https://github.com/egovernments/Citizen-Complaint-Resolution-System/raw/master/docs/operations/images/40-kong-proxy-latency.png" alt=""><figcaption></figcaption></figure>

<figure><img src="https://github.com/egovernments/Citizen-Complaint-Resolution-System/raw/master/docs/operations/images/41-kong-upstream-time.png" alt=""><figcaption></figcaption></figure>

The gateway's own work stays in the **tens of milliseconds**. The services behind it spike to **five seconds**. Same requests, same window, two orders of magnitude apart — so a user reporting "that screen is slow" is describing the second graph, and there is nothing to fix in the gateway.

{% hint style="info" %}
**Read these two together.** Upstream time high and proxy latency flat means "the gateway is fine, the service behind it is slow" — which redirects the whole investigation. Both flat while users report slowness means the delay is not in the API layer at all: look at the browser, the network, or the front-end.
{% endhint %}

**Row&#x20;*****Request rate*****.** **RPS per route/service by status code** is the panel to open for a 502/504 report — it shows failing responses broken down by which API produced them, so you learn *which* service is returning errors and how many.

**The other four rows** — Bandwidth, Caching, Upstream health, Nginx connections — are for capacity planning and Kong tuning. Not incident material.

The **service**, **route** and **upstream** dropdowns at the top filter the page once you know what you are looking for.

## If Dashboard Is Empty

An empty dashboard is usually a deployment fact, not a fault. Work through these in order.

**1. Check the time range first.** The single most common cause. If your range is "Last 5 minutes" and the deployment is quiet, most panels legitimately have nothing to draw. Widen to 24 hours before concluding anything.

**2. Is the data source running at all on this deployment?** Deployments choose how much observability to run, and the choice removes whole components:

<table><thead><tr><th width="188.61328125">If this deployment runs</th><th width="340.1328125">You have</th><th>You do not have</th></tr></thead><tbody><tr><td>the <strong>metrics</strong> level</td><td>Grafana, Prometheus, host and database metrics — every dashboard on this page except the two below</td><td><strong>Logs (Loki)</strong> and <strong>Traces (Tempo)</strong></td></tr><tr><td>the <strong>logs</strong> level</td><td>the above, plus <strong>DIGIT — Logs (Loki)</strong></td><td><strong>Traces (Tempo)</strong></td></tr><tr><td>the <strong>traces</strong> level (the default)</td><td>everything on this page</td><td>—</td></tr></tbody></table>

If **DIGIT — Logs (Loki)** is unavailable rather than merely empty, that is almost certainly this and not a fault. Ask L2 which level this deployment runs — and note it on your [cheat sheet](/complaints-management/deploy/operations-handbook/cheat-sheet.md), because the whole of [Step 4 of the first-response checklist ](/complaints-management/deploy/operations-handbook/l1-first-response.md#what-does-the-log-say)depends on it. See - [How much monitoring this deployment runs](/complaints-management/deploy/operations-handbook.md#monitoring-scope).

**3. Is the feature it measures deployed at all?** Some dashboards measure optional work:

<table><thead><tr><th width="272.671875">Dashboard</th><th>Blank when</th></tr></thead><tbody><tr><td><strong>DIGIT Kafka Consumer Lag</strong></td><td>it works on any deployment — but the <em>groups</em> listed depend on which consumers run</td></tr><tr><td><strong>DIGIT — PGR Analytics Queries</strong></td><td>nobody has opened the supervisor dashboard in your time range</td></tr><tr><td><strong>Node Exporter Full</strong></td><td><code>node-exporter</code> is missing or not being scraped — see below</td></tr></tbody></table>

**4. `Node Exporter Full` specifically has two causes, and they need different fixes.** L2 tells them apart in a minute: run `up` in **Explore → Prometheus**. If `job="node"` is missing, see [L2-diagnosis - The host itself ](/complaints-management/deploy/operations-handbook/l2-diagnosis.md#the-host-itself-cpu-ram-disk)— one cause is a config reload with no downtime, the other is a redeploy.

**5. If a panel is empty but its neighbours have data**, the metric behind it genuinely has no value right now. On a quiet deployment, that is common and healthy — zero errors, zero deadlocks, zero lag. An empty error panel is good news.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.digit.org/complaints-management/deploy/operations-handbook/the-grafana-dashboards.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.