> For the complete documentation index, see [llms.txt](https://docs.digit.org/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.digit.org/health/v2.1/design/architecture/low-level-design/services/health-services/console-services/project-factory-campaign-manager/project-factory/project-factory-enhancements-and-benchmarking.md).

# Project Factory - Enhancements & Benchmarking

## Overview

Project-factory is the system that sets up a full health campaign — think of a large-scale vaccination or health-worker deployment program — from a single Excel file upload. In one go, it needs to:

* Create the entire project hierarchy — one entry for every geographic area ("boundary"), which can be up to **\~21,000 areas** for a national campaign.
* Register all the physical **facilities** (clinics, centres, etc.) involved.
* Create **user accounts** for every field worker across four different backend systems.
* **Map** all staff, facilities, and supplies to the right projects — over **100,000 mapping records** for a large campaign.

**In plain terms:** imagine onboarding 20,000 locations, 10,000 facilities, and 20,000 employees — plus linking each employee, facility, and supply item to the right project — all from one spreadsheet upload. That's what this system automates.

{% hint style="info" %}

#### The two problems this release fixes

Before this release, running a large campaign (20,793 areas / 10,000 facilities / 20,000 users) had two serious problems:

1. **It was painfully slow.** The system processed everything one item at a time — one HTTP request per project, per user, per mapping — like a single checkout line serving thousands of customers. A full campaign took **over six and a half hours**.
2. **It failed unpredictably, even though nothing was actually wrong.** Campaigns would show hard failures like `PROJECT_CONFIRMATION_FAILED`. Thousands of user accounts would get permanently marked as failed. Newly added facilities would get stuck showing "pending" forever and never recover.

   **What was really happening:** in every single one of these cases, the record the system was "looking for" *did* exist — it just hadn't finished being saved yet by the time the system checked for it. It's like calling a warehouse two seconds after placing an order and being told "we don't have that item," when really it just hasn't been shelved yet. The system was treating this **temporary delay** as a **permanent failure**.

#### Why both problems had to be fixed together

Making the system faster (problem #1) actually makes problem #2 *worse* — the faster things move, the more likely the system is to check for something before it's finished saving. So this release fixes both at once:

* **Speed improvements** (bulk operations, parallel processing, smarter batching) make campaigns fast.
* **Consistency improvements** (waiting/confirming before checking, "defer instead of fail" logic) make sure that speed doesn't cause new false failures.
  {% endhint %}

***

### Enhancements List

<table><thead><tr><th width="72.17578125">#</th><th>Change</th><th>Type</th><th>Plain-Language Impact</th><th>Measured Impact</th></tr></thead><tbody><tr><td>1</td><td><strong>Bulk project creation</strong> — create many projects in one request instead of one at a time</td><td>Speed improvement</td><td>Setting up all 20,793 project areas is now ~5x faster</td><td>~60 min → <strong>5–13 min</strong></td></tr><tr><td>2</td><td><strong>Bulk mapping creation</strong> — link staff/facilities/resources to projects in batches, with double-checking</td><td>Speed improvement</td><td>Linking everything to the right project no longer takes hours</td><td>5 h 54 m → <strong>7.6 min</strong> (~37× faster per record)</td></tr><tr><td>3</td><td><strong>Project "parent confirm" fix</strong> — wait long enough, and smartly, before checking if a project was saved</td><td>Bug fix</td><td>Stops campaigns from being wrongly declared "failed"</td><td>Eliminates recurring <code>PROJECT_CONFIRMATION_FAILED</code> errors</td></tr><tr><td>4</td><td><strong>Worker-creation "defer guard"</strong> — don't permanently fail a user account just because a check was too early</td><td>Bug fix</td><td>Field workers no longer lose their accounts due to timing issues</td><td>1,008 false permanent failures per run → <strong>0</strong></td></tr><tr><td>5</td><td><strong>Facility persistence gate</strong> — wait for facility data to finish saving before checking on it</td><td>Bug fix</td><td>New facilities no longer get stuck "pending" forever</td><td>Fixes facilities silently stuck in limbo</td></tr><tr><td>6</td><td><strong>Infrastructure tuning</strong> — more parallel processing lanes, bigger batches, more server capacity</td><td>Behind-the-scenes tuning</td><td>This is what makes all the above improvements actually deliver the 24-minute result</td><td>Enables the full 16× speed-up</td></tr></tbody></table>

***

### Architecture - How a Campaign Flows Through the System

**In plain terms:** think of this as an assembly line with several stations. Here's what happens, in order, when someone uploads a campaign spreadsheet:

1. **Upload** — An administrator uploads the filled-in Excel template.
2. **Ingestion** — The system reads and checks every row of the spreadsheet, then saves each row to its own database table. This saving step happens **in the background**, not instantly — a detail that matters a lot for the fixes below.
3. **Parallel creation** — Three separate "assembly lines" now run **at the same time** instead of one after another:
   * Creating projects (one per geographic area)
   * Creating facilities
   * Creating user accounts for field workers
4. **Mapping** — Once creation is done, the system links every staff member, facility, and supply item to its correct project. This is called "mapping," and it repeats in cycles until everything is linked.
5. **Credentials** — Finally, login credentials are generated for all the new user accounts, and the campaign is marked **"created"** — done.

#### The three guarantees that now hold everywhere

For a non-technical reader, the following three rules are the heart of this release — every fix below is really just an application of one of them:

* **"Look only after it's actually there."** Every time one step in the assembly line needs to read something another step just created, it now *waits and confirms* that the data has actually finished saving — instead of checking immediately and assuming "not found" means "doesn't exist."
* **"Never create the same thing twice."** Every creation step first checks whether the record already exists before creating a new one. This means if a campaign is retried (say, after a network hiccup), it **picks up where it left off** instead of creating duplicates.
* **"Running things in parallel doesn't risk mixing things up."** Even though many things now happen simultaneously for speed, the system guarantees each record is handled by exactly one process, so nothing gets duplicated or double-counted — this was specifically tested and verified with real large-scale data.

#### System block diagram

<div align="left"><figure><img src="https://2077406040-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FI0KFbSBTySIQ7TCOKnJF%2Fuploads%2F1c3qWAjNTX1fuCxhMbyL%2Fblock_diagram_architecture.png?alt=media&amp;token=11090f72-45dd-4c1c-8a18-56365ff845ec" alt=""><figcaption></figcaption></figure></div>

***

### Enhancements in Detail

#### Bulk Project Creation

**The problem before:** Each of the 20,793 project areas was created with its own separate request, one after another — like a bank teller serving one customer at a time with 20,000 people in line. This alone took about an hour, and the longer it ran, the more chances there were for something to glitch along the way.

**The fix:** The system now sends projects in **batches of 100** rather than one at a time, with up to **5 batches running simultaneously**. A small pause between batches keeps the backend from being overwhelmed. Crucially, the very first project of a batch is double-checked ("confirmed") before its "child" projects underneath it are created, ensuring the hierarchy is built correctly.

There's also a safeguard: before creating anything, the system first checks what projects already exist for this campaign, so re-running a campaign never creates duplicates.

**Result:** This phase dropped from about an hour to **5–13 minutes**. If anything ever goes wrong with this new approach, there's an instant "off switch" (a configuration setting) that reverts to the old, slower-but-proven one-at-a-time method.

#### Sequence — bulk project creation with paced waves + parent confirmation

<figure><img src="https://2077406040-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FI0KFbSBTySIQ7TCOKnJF%2Fuploads%2FfkkXgXNt7gamYQzRMymK%2Fseq_bulk_project_confirm.png?alt=media&amp;token=5e5a697a-cd21-4e3c-8198-49aa08fe2a37" alt=""><figcaption></figcaption></figure>

#### Bulk Mapping Creation (Linking Staff, Facilities & Resources to Projects)

**The problem before:** Every single link between a staff member/facility/resource and its project — **113,172 of them** on a large campaign — was created one at a time. This one phase alone took almost **six hours**.

**The fix:** Mappings are now sent in **batches** to a bulk creation service. Because this bulk service works in the background (it says "got it, I'll get to it" rather than "done" immediately), the system now **double-checks** afterwards that each mapping was actually completed, and records the confirmed result. If a mapping can't be confirmed yet, it's simply tried again in the next cycle — nothing is ever marked as done unless it's actually verified. **No record is ever guessed complete.**

#### Sequence — bulk mapping with confirm-by-search + reconciler

<figure><img src="https://2077406040-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FI0KFbSBTySIQ7TCOKnJF%2Fuploads%2FjzRl1iw08I89zsQIeLcY%2Fseq_bulk_mapping_confirm.png?alt=media&amp;token=efa4aff1-7f9a-4dfa-93ee-4298363a67cf" alt=""><figcaption></figcaption></figure>

Additional safety nets added:

* **Duplicate detection** — if two spreadsheet rows accidentally try to create the same mapping, the system now flags this clearly instead of silently ignoring it.
* **Isolated failures** — if one batch fails, it doesn't affect any other batch.
* **Safe retries on check failures** — if the "did this work?" check itself fails, the record is simply retried, not marked as failed.
* **A safety switch for staff mappings specifically** — testing found that on some backend versions, staff mappings silently didn't work through the new bulk method (it looked successful but created nothing). So staff mappings default to the older, proven one-at-a-time method unless a specific setting confirms the new method is safe to use.

**Why batch size mattered so much:** double-checking a batch takes roughly the same amount of time (8–15 seconds) whether the batch has 30 records or 300. So making batches **10x bigger** (30 → 300 records) let that fixed "checking cost" cover 10x more work at once — like inspecting a truck full of boxes once, rather than inspecting each box's own truck. This single change was responsible for a large part of the overall speed-up.

#### Project "Parent Confirm" Fix — Stopping False Failures

**What people were seeing:** Large campaigns would suddenly fail with an error like `PROJECT_CONFIRMATION_FAILED` — and confusingly, if you just tried again, it would work fine, because the project was actually there all along.

**What was really happening:** When a project is created, its ID comes back right away, but the actual record takes a little longer to finish saving in the background. The system used to only wait a **fixed 6 seconds** before double-checking a project existed. Under heavy load (creating 20,000 projects at once), that 6-second wait wasn't nearly long enough — the system would give up too early and wrongly declare the whole campaign dead.

**The fix, two-pronged:**

1. Slow down the *creation* itself slightly (small pauses between batches) so the backend has time to keep up.
2. Make the *waiting/checking* period configurable and longer — **up to about 30 seconds** — but design it to move on immediately the moment the project is confirmed, so healthy campaigns aren't slowed down at all (they typically confirm within 2–4 seconds).

**Result:** This specific failure has **not happened once** since the fix was deployed.

#### Worker-Creation "Defer Guard" — Turning False Failures into Safe Retries

**What people were seeing:** After a fast batch of new user accounts was created, hundreds or even thousands of those accounts would get **permanently** marked as failed with an error like `INDIVIDUAL_NOT_FOUND` — even though those very accounts and their underlying records were completely fine and existed in the database.

**What was really happening:** A separate lookup system (which indexes records for searching, a bit like updating a library's card catalogue after new books arrive) takes a little time to "catch up" after new records are created. Under heavy load, that catch-up time occasionally spiked to as long as **32 seconds** — longer than the system's old 20-second wait. When the wait ran out, the system used to just go ahead anyway and try to create the worker record — which then permanently failed because the lookup wasn't ready yet.

**The fix, two layers:**

1. **Wait longer, but smartly.** The waiting window was extended to **50 seconds** (based on the actual worst-case delay observed, plus a safety margin) — but it still exits immediately the moment things are confirmed ready, so well-behaved campaigns aren't slowed down at all.
2. **The actual "guard":** if a few records *still* aren't ready even after the longer wait, the system no longer tries to force them through (which would permanently fail them). Instead, those specific records are marked **"deferred — try again later"** and simply skipped for now. A routine retry of the campaign picks them up automatically once they're ready — by then, the lookup has long since caught up.

#### Sequence — user creation: bulk with per-user fallback + consistency gate + defer guard

<figure><img src="https://2077406040-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FI0KFbSBTySIQ7TCOKnJF%2Fuploads%2FUgtiJgu3FuapvvI2D645%2Fseq_user_creation_defer_guard.png?alt=media&amp;token=6ca733c3-f17b-4798-8601-cd320065b2de" alt=""><figcaption></figcaption></figure>

**A bonus safety net (HRMS fallback):** the system always tries to create user accounts in one big efficient batch first. But if that batch gets rejected for any reason, it automatically falls back to creating those same accounts **one by one** instead of giving up. In one real test run, two-thirds of the batch attempts failed and fell back this way — and **all 20,000 users were still successfully created** in the end. Nothing was lost.

**Measured effect:** the number of accounts falsely marked as failed dropped from **1,008 in one run to zero** in the next two runs — even while the system was running *faster* than before (\~1,500 new users per minute).

#### Facility Persistence Gate — the Last Piece of the Puzzle

**What people were seeing:** On a spreadsheet that mixed existing facilities (already had a code) with brand-new facilities, the existing ones completed fine — but the new ones would get stuck showing something like `6/1006 completed, 0 failed, 1000 pending` and just... never move past that.

**What was really happening:** New facility rows are saved to the database in the background, the same as everything else in this release. But the part of the system responsible for creating facilities was checking the database **immediately**, before those new rows had finished saving. Seeing nothing to do yet, it simply gave up and never checked again — leaving those rows stuck forever. It looked like "old facilities work, new ones are broken," but that was a coincidence: existing facilities (with a code already) don't need this creation step at all, so they were never affected by the bug in the first place.

**The fix:** This step now works exactly like the other steps already described — it **waits and confirms** that the expected number of new facility rows has actually finished saving before it starts checking for them. Campaigns that were already stuck like this simply need to be retried once, and they'll complete correctly — the data was there all along, just waiting to be picked up.

***

### Configuration Reference (For Technical Teams)

*The section below is intended for engineers deploying or tuning the system. Non-technical readers can safely skip to Section 6 (Benchmarking).*

Each value below was tuned against measurements on the test environment (unified-UAT) and can be changed through configuration — no code changes needed.

#### Environment (project-factory)

```yaml
# ── users ─────────────────────────────────────────────────────────────
USER_KAFKA_CREATE_BATCH_SIZE: "150"            # users per Kafka batch; 150 balances HRMS call duration vs per-batch overheads
USER_INDIVIDUAL_CONSISTENCY_MAX_POLL_ATTEMPTS: "25"   # ×2s = 50s ceiling; sized from the MEASURED max indexing lag of 32s; early-exit
USER_INDIVIDUAL_CONSISTENCY_POLL_INTERVAL_MS: "2000"
USER_INDIVIDUAL_SEARCH_BATCH_SIZE: "150"       # one searchability query per batch
USER_WORKER_CREATE_BATCH_LAG: "2"              # in-process path only: workers for batch N created after batch N+2's individuals

# ── mapping ───────────────────────────────────────────────────────────
MAPPING_KAFKA_BATCH_SIZE: "300"                # THE amortization lever: 30 → 300 ≈ 13× per-partition throughput
MAPPING_BULK_CREATE_CHUNK_SIZE: "150"          # 2 bulk calls per batch, one shared confirm cycle
STAFF_MAPPING_BULK: "true"                     # enable ONLY after verifying the project-service build creates staff via bulk

# ── projects ──────────────────────────────────────────────────────────
PROJECT_BULK_CREATE_CHUNK_SIZE: "100"          # projects per array _create call; 0 = legacy per-boundary rollback
PROJECT_BULK_CREATE_CONCURRENCY: "5"
PROJECT_CREATE_PACE_DELAY_MS: "100"            # pause between waves so the persister drains
PROJECT_CONFIRM_RETRIES: "15"                  # ×2000ms ≈ 30s parent-confirm ceiling, early-exit
PROJECT_CONFIRM_POLL_INTERVAL_MS: "2000"
```

#### Kafka Partitions

**In plain terms:** a "partition" is like a checkout lane — one lane can only be worked by one cashier at a time, no matter how many cashiers are on staff. Adding more lanes lets more work happen simultaneously. These changes are permanent (they survive restarts and deployments) and are always safe to increase.

| Topic (what it carries)     | Lanes (Partitions) | Carries                                |
| --------------------------- | ------------------ | -------------------------------------- |
| `hcm-user-create-batch`     | 4                  | user-creation batches (150 users each) |
| `hcm-mapping-batch`         | 4                  | mapping batches (300 rows each)        |
| `hcm-facility-create-batch` | 3                  | facility-creation batches              |

#### Service Resources

<table><thead><tr><th width="136.69140625">Service</th><th width="168.82421875">Setting</th><th>Why (Plain Language)</th></tr></thead><tbody><tr><td>egov-user</td><td>4 replicas, CPU limit <strong>4000m</strong></td><td>This service handles password security, which is deliberately slow and CPU-intensive <strong>on purpose</strong>, for security reasons. It just needed more computing power to keep up with bursts of new accounts. <strong>Security was never weakened for speed</strong> — the password-hashing strength stayed exactly the same.</td></tr><tr><td>health-hrms</td><td>4 replicas</td><td>Handles the full chain of creating a user's account, personal record, and employee record.</td></tr><tr><td>worker-registry</td><td>2 replicas</td><td>Handles field-worker record validation.</td></tr><tr><td>Redis (shared cache)</td><td>memory limit <strong>1Gi</strong></td><td>The cache used to run out of memory and crash repeatedly under load, which made the "false failure" problems described above even worse. Giving it more memory fixed the crashing.</td></tr></tbody></table>

***

### Benchmarking Results

Measured on a realistic test environment with a **national-scale campaign**: 20,793 geographic areas / 10,000 facilities / 20,000 users. Here's how the final, best-performing run broke down:

<table><thead><tr><th width="337.95703125">Phase</th><th width="177.37890625">Duration</th><th>Rate</th></tr></thead><tbody><tr><td>Reading &#x26; validating the spreadsheet</td><td>~2.5 min</td><td>—</td></tr><tr><td>Creating projects, facilities &#x26; users (in parallel)</td><td><strong>16.3 min</strong></td><td>~1,500 new users/min</td></tr><tr><td>Linking staff/facilities/resources (50,793 links)</td><td><strong>7.6 min</strong></td><td>~6,700/min (larger campaigns sustain 8,000–9,000/min)</td></tr><tr><td>Generating login credentials</td><td>0.6 min</td><td>—</td></tr><tr><td><strong>Total, start to finish</strong></td><td><strong>24.4 min</strong> (core processing: ~21.8 min)</td><td></td></tr></tbody></table>

#### The improvement over time

As the team tuned the system over the course of a week, using the **same test environment** each time, total campaign time dropped like this:

**6 h 41 m → 1 h 24 m → 43 m → 41 m → 34 m → 24.4 m**

That's roughly a **16× improvement** from where it started. And importantly, the **last two runs completed perfectly** — zero failed user accounts, zero stuck workers, and zero mapping failures. Speed and correctness were achieved together, not traded off against each other.

> **A note on how "time" is measured:** there are two honest ways to measure this. "Click-to-created" is what a user actually experiences, including the \~2.5 minutes it takes to read the spreadsheet. "Processing time" measures only the actual data-creation work. When you see two slightly different numbers for what looks like the same run, this is almost always the reason.

***

## Upgrade & Deployment Notes (For Technical Teams)

1. **Order matters:** database updates (new columns for tracking mapping retries and errors) must be applied **before** the new software version is rolled out — the system needs those columns to exist for both old and new campaigns.
2. Apply the new **configuration values** (Section 5.1) as part of the deployment — the update rolls out automatically. Double-check the settings actually took effect on a fresh server after deploying.
3. Apply the **Kafka partition increases** (Section 5.2) once per environment. Always safe to increase; just confirm there's no backlog of unprocessed messages before making the change.
4. This release only **adds** new information to existing messages — it doesn't change or remove anything other systems rely on, so it's safe alongside anything else reading those messages.
5. **After upgrading, re-run any campaign that has stuck or deferred records** — the retry is safe to run and will automatically sweep up and finish anything left in limbo, in a single pass.
6. **Instant rollback options, no need to redeploy anything:** if any of the new bulk features cause issues, they can each be individually switched back off through configuration — projects, mappings, and staff mappings each have their own independent "off switch" back to the old, proven behaviour.

***

## Glossary (Quick Reference)

<table><thead><tr><th width="198.93359375">Term</th><th>Plain-Language Meaning</th></tr></thead><tbody><tr><td>Campaign</td><td>A full health program rollout — projects, facilities, workers, and supply links, all set up together.</td></tr><tr><td>Boundary</td><td>A geographic area (e.g., a district or village) that gets its own "project" record.</td></tr><tr><td>Mapping</td><td>Linking a staff member, facility, or supply item to the project it belongs to.</td></tr><tr><td>Bulk creation</td><td>Creating many records in one request instead of one at a time — much faster.</td></tr><tr><td>Confirm-by-search</td><td>Double-checking that something was actually saved, instead of just trusting an initial "OK" response.</td></tr><tr><td>Persistence / persisted</td><td>The point at which data has actually finished being saved to the database (as opposed to just being "in progress").</td></tr><tr><td>Idempotent</td><td>Safe to repeat — running the same action twice doesn't create duplicates or cause harm.</td></tr><tr><td>Defer / deferred</td><td>Instead of permanently failing something due to a timing issue, it's set aside to be automatically retried later.</td></tr><tr><td>Partition (Kafka)</td><td>A "lane" of parallel processing — more lanes mean more work can happen at the same time.</td></tr></tbody></table>


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.digit.org/health/v2.1/design/architecture/low-level-design/services/health-services/console-services/project-factory-campaign-manager/project-factory/project-factory-enhancements-and-benchmarking.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
