> For the complete documentation index, see [llms.txt](https://docs.digit.org/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.digit.org/health/v2.1/design/architecture/low-level-design/performance-and-api-benchmarking.md).

# Performance & API Benchmarking

## Overview

**How to read the performance & benchmarking report?**

There is one report per service. Each reports how many concurrent requests a single pod of that service absorbed before it began rejecting them, and how response time behaved up to that point. This section contains definitions and background that apply to every service, so the service reports include only their own measurements.

<table><thead><tr><th width="194.69140625">Term</th><th>Meaning</th></tr></thead><tbody><tr><td>Concurrent requests (users)</td><td>How many requests were in flight at the same instant. Every request is sent by its own thread and all threads are released together, so this is a genuine simultaneous load rather than a steady arrival rate.</td></tr><tr><td>Clean level</td><td>A concurrency level at which every single request succeeded.</td></tr><tr><td>Failing level</td><td>The first level at which any request failed. Testing stopped there; levels above it were not run.</td></tr><tr><td>Breakpoint</td><td>The boundary between the two — reported as a bracket, for example clean to 900 and failing at 1,000.</td></tr></tbody></table>

## Metric Definitions

<table><thead><tr><th width="144.87109375">Metric</th><th>What it means</th></tr></thead><tbody><tr><td>p50 (median)</td><td>Half the requests were faster than this. Put another way, the slowest of the fastest half.</td></tr><tr><td>p90</td><td>90% were faster; the slowest 10% took longer.</td></tr><tr><td>p95</td><td>95% were faster; the slowest 5% took longer. The usual tail-latency measure.</td></tr><tr><td>p99</td><td>99% were faster; only the slowest 1% took longer.</td></tr><tr><td>Error %</td><td>Requests that did not return success, as a share of the run. Includes dropped connections as well as HTTP error codes.</td></tr><tr><td>Throughput</td><td>Requests completed per second, for that endpoint alone, measured from the first request start to the last response end.</td></tr></tbody></table>

A working example: If 100 requests are sent at once and p95 is 10.1 seconds, that means 95 of the 100 finished in under 10.1 seconds and the slowest 5 took longer. If p50 for the same run is 7.0 seconds, half finished in under 7 seconds. The gap between p50 and p99 is the spread: a wide gap means a few requests waited much longer than most.

Latency and throughput in every service report are computed from successful requests only. A failed request still has a response time, so counting failures would make a failing level look faster and busier than a clean one. Error % is over all requests.

### Throughput at Each Percentile

Each service report shows throughput at p50, p90, p95 and p99 as well as for the run overall. This is a real measurement, not a rescaled overall figure. Because every thread is released together, the Nth-slowest request is also the Nth to finish, so the rate at p50 is simply the number of requests completed by that point divided by the time they took.

Read the shape of the column rather than any single value. A rate that climbs and then flattens means the service delivered steadily through the run — the pipeline fills at the start, then work comes out at a constant rate. A rate that falls away at p99 means the last few requests stalled: the service handled most of the load briskly and then left a small group waiting. A single overall throughput figure hides that difference entirely, which is why the breakdown is given.

## Test Method

<table><thead><tr><th width="185.18359375">Item</th><th>Value</th></tr></thead><tbody><tr><td>Load generator</td><td>Threaded HTTP client, one request per thread, all threads released together</td></tr><tr><td>Connection to the service</td><td>kubectl port-forward direct to the pod, bypassing ingress and gateway</td></tr><tr><td>Authentication</td><td>One pre-fetched auth token shared by every thread; no per-thread login call</td></tr><tr><td>Stopping rule</td><td>Escalate until the first level that produces errors, then narrow the gap to that level by testing midpoints</td></tr><tr><td>Endpoints</td><td>Measured one at a time; only one endpoint and one concurrency level under load at any moment</td></tr><tr><td>Requests within a level</td><td>All concurrent — every thread released together, one request each</td></tr><tr><td>Samples measured</td><td>Each service's own endpoint only</td></tr></tbody></table>

Gateway-routed runs are not reported: they measure the shared gateway and login path, not the service.

## Read Failure

Two kinds of failure appear in these reports, and they carry different weight.

<table><thead><tr><th width="180.82421875">Failure class</th><th width="269.66015625">What it means</th><th>How much to trust it</th></tr></thead><tbody><tr><td>HTTP response (for example 400)</td><td>The service received the request, processed it, and returned an error. The response body is quoted in the report.</td><td>Firm. Only the service can produce an HTTP response, so the failure is definitely the service.</td></tr><tr><td>Connection-level (RemoteDisconnected, IncompleteRead)</td><td>No response came back at all. The connection was closed before the service replied.</td><td>Indicative. The sample log cannot tell whether the pod refused the connection or the port-forward tunnel used to reach it dropped. These levels need re-confirming from inside the cluster.</td></tr></tbody></table>

## Dataset Size - What It Means

Each service report states how many rows the tested search matched, and how many rows the table held for the tenant. This matters because the two extremes measure very different things.

<table><thead><tr><th width="217.26953125">What the filter matched</th><th>What the numbers describe</th></tr></thead><tbody><tr><td>One row, found by id</td><td>An indexed point lookup. The figures show how fast the service can fetch a known record under load. They do not show what a broad search costs.</td></tr><tr><td>Thousands of rows, capped at 50 returned</td><td>A real search: the database filters a large set and the service serialises the first page. The figures include that scan and filter cost.</td></tr><tr><td>Zero rows</td><td>The query executed but matched nothing. The figures show request handling overhead only, with no result set to build.</td></tr></tbody></table>

Row counts were read from each service's own search API after testing finished, so they include any rows the create tests inserted.

## Glossary

<table><thead><tr><th width="134.56640625">Term</th><th>Meaning</th></tr></thead><tbody><tr><td>Pod</td><td>One running copy of a service. Kubernetes runs services as pods.</td></tr><tr><td>Replica</td><td>How many pods a service runs. Testing here connects to a single pod, so the figures describe one replica, not the total across all of them.</td></tr><tr><td>CPU limit</td><td>The maximum processor share a pod may use, in millicores. 300m is roughly a third of one CPU core.</td></tr><tr><td>Memory limit</td><td>The maximum memory a pod may use before Kubernetes restarts it.</td></tr><tr><td>JVM heap</td><td>The memory the Java application itself is allowed to allocate, set below the pod memory limit.</td></tr><tr><td>Tomcat max threads</td><td>How many requests the service can work on simultaneously. Requests beyond this number wait in a queue.</td></tr><tr><td>Connection pool (Hikari)</td><td>How many database connections the service may hold open at once. Requests needing the database queue for one of these, and give up if they wait too long.</td></tr><tr><td>Kafka</td><td>A message queue. A service can accept work by putting a message on the queue and replying immediately, then doing the real work afterwards.</td></tr><tr><td>Synchronous endpoint</td><td>Does the work and returns the result in the same request. The caller waits for everything.</td></tr><tr><td>Asynchronous endpoint</td><td>Validates the request, puts it on Kafka, and replies 202 Accepted straight away. The database write happens afterwards. Figures for these endpoints measure how fast work is accepted, not how fast it is stored.</td></tr><tr><td>202 Accepted</td><td>A success response meaning the request was valid and queued, not that it is finished.</td></tr><tr><td>HTTP 400</td><td>The service rejected the request. Under load this usually means it could not obtain a resource, such as a database connection, in time.</td></tr><tr><td>Port-forward</td><td>A direct tunnel from the test machine to a single pod, used here so the numbers describe the service rather than the shared gateway in front of it.</td></tr><tr><td>Restart</td><td>Kubernetes stopping and starting a pod, usually because its health check failed. Reported per service with the time it happened.</td></tr></tbody></table>

### Service Reports

* [Beneficiary ID Generation](https://docs.digit.org/health/v2.1/design/architecture/low-level-design/performance-and-api-benchmarking/beneficiary-id-generation-performance-testing-and-benchmarking-report.md)
* [Facility](https://docs.digit.org/health/v2.1/design/architecture/low-level-design/performance-and-api-benchmarking/facility-service-performance-testing-and-benchmarking-report.md)
* [Household](https://docs.digit.org/health/v2.1/design/architecture/low-level-design/performance-and-api-benchmarking/household-service-performance-testing-and-benchmarking-report.md)
* [HRMS](https://docs.digit.org/health/v2.1/design/architecture/low-level-design/performance-and-api-benchmarking/hrms-service-performance-testing-and-benchmarking-report.md)
* [Individual](https://docs.digit.org/health/v2.1/design/architecture/low-level-design/performance-and-api-benchmarking/individual-service-performance-testing-and-benchmarking-report.md)
* [PGR](https://docs.digit.org/health/v2.1/design/architecture/low-level-design/performance-and-api-benchmarking/complaints-service-performance-testing-and-benchmarking-report.md)
* [Product](https://docs.digit.org/health/v2.1/design/architecture/low-level-design/performance-and-api-benchmarking/product-service-performance-testing-and-benchmarking-report.md)
* [Project](https://docs.digit.org/health/v2.1/design/architecture/low-level-design/performance-and-api-benchmarking/project-service-performance-testing-and-benchmarking-report.md)
* [Referral Management](https://docs.digit.org/health/v2.1/design/architecture/low-level-design/performance-and-api-benchmarking/referral-management-service-performance-testing-and-benchmarking-report.md)
* [Stock](https://docs.digit.org/health/v2.1/design/architecture/low-level-design/performance-and-api-benchmarking/stock-service-performance-testing-and-benchmarking-report.md)
* [Worker Registry](https://docs.digit.org/health/v2.1/design/architecture/low-level-design/performance-and-api-benchmarking/worker-registry-service-performance-testing-and-benchmarking-report.md)

{% hint style="info" %}

### Source Formatting Note

Some scalability tables in the source label their percentile column `p95 (ms)`, while the values include an `s` suffix, and the benchmark text states that latency is in seconds. The original labels and values are retained here; no measurements have been recalculated.
{% endhint %}


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.digit.org/health/v2.1/design/architecture/low-level-design/performance-and-api-benchmarking.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
