> For the complete documentation index, see [llms.txt](https://docs.digit.org/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.digit.org/complaints-management/deploy/configure/notifications/migrate-notifications.md).

# Migrate Notifications

Migrating notifications from 2.12 to 2.20

## Overview

Read this before deploying 2.20 over a 2.12 deployment. First-time setup is in setup-guide.md.

### What changes

| Area                        | 2.12                                                                                                                                                                                 | 2.20                                                                                                                                                                                                                                                            |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Configuration masters       | `RAINMAKER-PGR.NotificationRouting` / `.NotificationTemplate` / `.NotificationProviderTemplate` / `.NotificationChannel`, keyed `(businessService, action, toState)`, bare audiences | `NOTIFICATIONS.EventCatalogue` / `.Routing` / `.Template` / `.ProviderTemplate` / `.Channel` at the state tenant, keyed on `eventName`, audiences `ACTOR:` / `ROLE:` / `EVENT_RECIPIENTS` / `A\|B`                                                              |
| What pgr-services publishes | Pre-rendered envelopes, one per recipient × channel, behind `pgr.notification.config.driven`                                                                                         | One thin event per workflow transition on `complaints.domain.events`; novu-bridge routes, resolves and renders. No flag                                                                                                                                         |
| Login OTP SMS               | `egov-notification-sms` (or `otp-publisher`)                                                                                                                                         | Citizen sign-in moved to the Identity BFF, whose codes do not go through notifications yet (§4). Where a deployment still uses DIGIT's older OTP service, novu-bridge consumes `egov.core.notification.sms`: same channel policy, provider and log as other SMS |
| Providers                   | Novu dashboard / deploy-time env                                                                                                                                                     | Configurator → Notifications → Providers (credentials stored in Novu)                                                                                                                                                                                           |
| Channel on/off              | `novu_bridge_channels_enabled`                                                                                                                                                       | Channel rows (Configurator → Notifications → Channels). The env list is a fallback for a tenant with no rows; the upgrade turns it into rows (step 3)                                                                                                           |
| Images                      | pgr-services and novu-bridge pinned separately                                                                                                                                       | pgr-services, pgr-services-db, novu-bridge and novu-bridge-db share one tag, and digit-user-preferences-service with its -db image follows it (step 1)                                                                                                          |

### Before you upgrade

* [x] Choose **one build** for pgr-services, pgr-services-db, novu-bridge and novu-bridge-db (step 1).
* [x] Note which channels each tenant uses today (`NOVU_BRIDGE_CHANNELS_ENABLED` on the running `novu-bridge` container) and which provider sends them. After the upgrade, they must still be on.
* [x] Remove the flags and properties listed under Removed settings from your host\_vars, Compose overlays and Helm values. Move any bridge setting you added to `/opt/digit/.env` by hand into host\_vars (step 5).
* [x] If the deployment still uses DIGIT's older OTP service (`enable_otp_services: true`), confirm `enable_novu: true` (the deploy refuses it without).
* [x] If the Novu admin's password was ever changed by hand (in the Novu dashboard), pin it before upgrading: see Per-tenant notification accounts.
* [x] Kubernetes: keep `deploymentStrategy.type: Recreate` on `pgr-services` and upgrade novu-bridge first (step 2).
* [x] Book a quiet period per tenant for moving its notification configuration (step 3): the deploy does not move it.

### 1. Take all four images from one build

pgr-services publishes thin events that only a novu-bridge from the same build can resolve, and each application needs the Flyway migrations its `-db` image carries: 2.20's bridge writes columns that `novu-bridge-db` adds (`V20260916120000__dispatch_log_delivery_receipts.sql`, `V20260921130000__add_source_path.sql`). A new bridge on the old schema fails every dispatch-log write. `build/build-config.yml` builds all four from the same commit in every run, so pin them with **one** tag:

| Tier                      | Setting                                                                                | Default                                             |
| ------------------------- | -------------------------------------------------------------------------------------- | --------------------------------------------------- |
| Compose (Ansible)         | `notification_stack_tag` in host\_vars → `NOTIFICATION_STACK_TAG` in `/opt/digit/.env` | `nightly-develop`                                   |
| Helm                      | `global.notificationStackTag` in `devops/deploy-as-code/charts/environments/env.yaml`  | empty → the charts' own defaults, `nightly-develop` |
| `enable-notifications.sh` | `NOTIFICATION_STACK_TAG`                                                               | `nightly-develop`                                   |

`digit-user-preferences-service` and `digit-user-preferences-service-db` follow the same tag. Every 2.12 tag of that image is the retired Go service, which 2.20's compose and Helm definitions cannot start, so a deployment that pins the stack to an immutable tag must not leave them on a rolling default of their own. The tag therefore has to exist for all six.

`nightly-develop` is the rolling tag the develop nightly publishes for all six (`egovio/<image>:nightly-develop`, plus an immutable `develop-<sha8>`; see `build/NIGHTLY-BUILDS.md`). It is the default only until the first release after this stack merges pins an immutable `develop-<sha8>` in its place (Release step). It moves **per image**: when one image's build fails, or two runs overlap, it points at different commits for different images. **On a live deployment, pin an immutable tag that exists for all six** — a release tag, or a `develop-<sha8>` you have checked on Docker Hub for `pgr-services`, `pgr-services-db`, `novu-bridge`, `novu-bridge-db`, `digit-user-preferences-service` and `digit-user-preferences-service-db`. The Ansible deploy warns when any image it runs resolves to a rolling tag; Helm pulls a rolling tag `Always` and an immutable one `IfNotPresent`.

A per-image override (`pgr_services_image`, `pgr_services_db_image`, `novu_bridge_image`, `novu_bridge_db_image`, `digit_user_preferences_service_image`, `digit_user_preferences_service_db_image`; Helm `<chart>.image.tag` / `<chart>.initContainers.dbMigration.image.tag`) wins over the shared tag for that one image, so pin all six to the same build or none. The deploy warns when host\_vars pin only some of the images it runs: `pgr_services_image` + `pgr_services_db_image` on every box (they run with notifications off too), all six when `enable_novu` is on.

### 2. New novu-bridge before new pgr-services

A 2.12 bridge cannot read a thin event: it dead-letters it to `novu-bridge.dlq`, and nothing replays the DLQ, so every transition published in that window is a notification lost. The new bridge accepts both the old pre-rendered envelopes and thin events, so upgrading it first is safe.

* **Docker Compose**: the deploy recreates `novu-bridge` (and runs its migrator) on its own before the full `up -d`, whenever a bridge is already running. A failed `up -d novu-bridge` (image pull, `novu-bridge-migration`) **stops the deploy** there, before pgr-services and with the old OTP senders untouched (`BRIDGE-UP-FAILED`); so does a `novu-bridge` container that is not the image and config hash compose resolves now (`BRIDGE-NOT-CURRENT`), because the old bridge is still running at that point and must not be taken for the new one. Doing it by hand: `docker compose <files> up -d novu-bridge` first, then `up -d pgr-services`.
* **Kubernetes**: `digit-helmfile.yaml` applies `common-services` (novu-bridge, released with `wait: true`) before `urban` (pgr-services), so a plain `helmfile -e env sync` waits for the new bridge to be Ready before touching pgr-services; a bridge that never becomes Ready stops the sync with the old pair still running. Syncing charts one at a time, do it in this order:

  ```bash
  cd devops/deploy-as-code
  helmfile -f charts/common-services/common-services-helmfile.yaml -e env -l name=novu-bridge sync
  kubectl -n egov rollout status deploy/novu-bridge
  helmfile -f charts/urban/urban-helmfile.yaml -e env -l name=pgr-services sync
  ```

**Never run an old and a new pgr-services together.** A 2.12 replica (pre-rendered envelopes) and a 2.20 replica (thin events) each notify the citizen, under two different transaction ids, so the bridge's refusal to re-send an id it has already `SENT` or `DELIVERED` does not catch it. Compose stops the old container before starting the new one. On Kubernetes `devops/deploy-as-code/charts/urban/pgr-services/values.yaml` sets `deploymentStrategy.type: Recreate`; keep it and do not add a `rollingUpdate` block (the API rejects it with `Recreate`).

### 3. Copy each tenant's configuration

**The deploy upgrades software only.** `./deploy.sh <tenant>` (or `--tags notifications`) creates the notification schemas and access-control rows, gives a tenant with **no** channel rows rows that say what `NOVU_BRIDGE_CHANNELS_ENABLED` says (below), and — only for a **fresh install**: no notification configuration in either namespace **and no complaint ever filed** at the tenant or its cities — writes the shipped defaults straight into `NOTIFICATIONS.*`. It never copies a tenant's 2.12 rows and never adds a default row to a configured tenant. A 2.12 tenant keeps being served from its legacy masters, unchanged; the deploy says so (`notif-seed — ACTION: this tenant's notification configuration is not migrated`).

**A tenant that ran 2.12's hard-coded notifications gets no defaults from the deploy.** With `pgr_notification_config_driven: false` (every tenant but Bomet) the wording came from `novu-bridge-endpoint`, not MDMS, so such a tenant has no notification rows in either namespace — the same as a fresh install. The deploy tells them apart by the complaints filed at the tenant, counted in the database: with any (or when they cannot be counted) it writes nothing, reports `NOTIFICATIONS-STATE: … state=none` and `notif-seed — ACTION: this tenant has no notification configuration`, and **the tenant sends no complaint notifications** (`SKIPPED / NB_NO_ROUTING`; codes from DIGIT's older OTP service, where a deployment still uses it, are not affected) until you review and install the defaults:

```bash
python3 migrate-notifications.py plan  --tenant mycity --adopt-defaults
python3 migrate-notifications.py apply --tenant mycity --adopt-defaults --yes
```

The plan calls it `none` and lists every row it would write. `notifications_adopt_defaults` in host\_vars makes the deploy seed the defaults anyway — only for roots whose complaints are demo data (the `full-dump.sql` complaints under `pg`): `true` for every root the deploy seeds (`localhost-full.yml.example` sets it), or a list such as `[pg]` for those roots only. It never touches a tenant that has configuration.

**Every state root with complaints is decided, not just `state_root`** ([#1943](https://github.com/egovernments/Citizen-Complaint-Resolution-System/issues/1943)). novu-bridge resolves a complaint's configuration at the complaint's own state root (`pg.citya` → `pg`), so the deploy seeds `state_root` plus every root that has a complaint (`split_part(tenantid, '.', 1)` of `eg_pgr_service_v2`), each with its own complaint count and its own login (`notif_seed_user` at that root). A box whose `state_root` is `ke` but which still carries the dump's `pg` complaints therefore gets `pg` decided and reported too. The deploy prints one line per root (`notif-seed — result per state root`): `fresh`, `notifications`, `legacy` or `none`, each `legacy` / `none` root with its own `ACTION` and command. **Only `state_root` can fail the deploy.** At any other root — typically the dump's `pg` demo complaints on a `ke` box — a refused login (`WARNING: could not log in at a complaint root`), a write still refused with 403, or MDMS that cannot be read (`WARNING: a complaint root other than state_root could not be seeded`) is a warning that names the exit code and the command that finishes the root; that root's configuration is left as it was. A root that fails does not stop the others. Roots matching `notifications_seed_exclude` (default `(?i)^(PW_|pwt)`, the test suite's junk tenants) are listed and skipped. To see the roots yourself:

```bash
sudo docker exec docker-postgres psql -U egov -d egov -c \
  "select split_part(tenantid, '.', 1) as state_root, count(*) from eg_pgr_service_v2 group by 1"
```

`migrate-notifications.py --all` still takes its roots from `--roots` (else `STATE_ROOT`, else the bridge's OTP default tenant): pass `--roots ke,pg` on such a box, or `--tenant pg`. Until it is migrated, the Configure and Channels screens show it read-only (*"This tenant has not been migrated yet — shown read-only"*) and refuse a raw first `NOTIFICATIONS.Routing` / `NOTIFICATIONS.Channel` row (`namespace-switch`): that row alone would switch the tenant over.

**The legacy masters stay writable through the API.** The Configurator shows the four `RAINMAKER-PGR.Notification*` masters read-only, but that only hides its own edit screens: the MDMS role-actions that let `MDMS_ADMIN` / `ACCOUNT_ADMIN` write them are unchanged (a 2.12 tenant that has not been migrated is still configured through them by anything that writes MDMS directly). Once a tenant is migrated novu-bridge no longer reads them, so **an edit made to them after migration has no effect** — change a migrated tenant in Notifications → Configure (`NOTIFICATIONS.*`).

Moving a tenant is a separate step, per tenant, with `local-setup/scripts/migrate-notifications.py` (the deploy stages it as `/opt/digit/notification-seed/migrate-notifications.py`). **It is one-way.** novu-bridge serves a tenant from `NOTIFICATIONS.*` from its first `NOTIFICATIONS.Routing` row on (active or not — MDMS has no delete), and no command or setting moves it back. The legacy rows are never modified.

```bash
cd /opt/digit/notification-seed                 # or local-setup/scripts in a checkout
export DIGIT_URL=http://127.0.0.1:18000         # Kong; DIGIT_USERNAME / DIGIT_PASSWORD default ADMIN / eGov@123
export DIGIT_LOGIN_TENANT=mycity                # the root you migrate (see "Who logs in")
python3 migrate-notifications.py plan  --all --report plan.json
python3 migrate-notifications.py apply --all --only defaults --yes --report apply.json
python3 migrate-notifications.py apply --tenant mycity --yes --report apply-mycity.json
```

#### Who logs in

An admin with no DIGIT password — a self-serve workspace's founder signs in through Keycloak only — authenticates with a token instead: `export DIGIT_ACCESS_TOKEN=<that admin's DIGIT access token>` (where to find it: setup-guide §8.6), and `DIGIT_USERNAME` / `DIGIT_PASSWORD` are then ignored. The token is checked with egov-user's own introspection (`/user/_details`) before anything else — `REFUSED: the supplied DIGIT_ACCESS_TOKEN was refused (HTTP 401)` when it has expired, or `(HTTP 200 without a user)` — and never printed. Keep `DIGIT_LOGIN_TENANT` (or `--login-tenant`) at that admin's root: it still decides who may create a provider, below.

Log in **at the root you migrate** (`DIGIT_LOGIN_TENANT=<root>`, default: the first root): its MDMS rows are written with that admin's roles, and novu-bridge previews a tenant's events (`_resolve`, `_dry-run` without `send`) for an admin of the event tenant's own state root. That is what each deploy `ACTION` line prints. One exception: **creating a provider** (`--create-provider` / `--create-smscountry-provider`). Providers are shared by the whole deployment, so novu-bridge lets only an admin of a state that **owns** them create one — `state_root` (it reads it from `NOVU_BRIDGE_CORE_SMS_DEFAULT_TENANT`) plus any root listed in `novu_bridge_provider_admin_tenants`. Logged in anywhere else the creation is refused with `403 NB_TENANT_NOT_ALLOWED`; the script then stops with `REFUSED: provider creation refused by novu-bridge …` and exit `4` **before any tenant is read or written** (the plan already prints the same as a `WARNING` when it can read the bridge's settings from the `novu-bridge` container). For such a root:

1. create the provider as an admin of `state_root` — Configurator → Notifications → Providers, or `migrate-notifications.py apply --tenant <state_root> --create-provider …` logged in there (only when `state_root` is already `migrated`; otherwise that apply migrates it too);
2. pin it at the root, logged in at the root: `DIGIT_LOGIN_TENANT=pg python3 migrate-notifications.py apply --tenant pg --provider SMS=<identifier> --yes`.

Or list the root in `novu_bridge_provider_admin_tenants` and redeploy, so its admins manage providers themselves. A preview refused for the login (`403 NB_TENANT_NOT_ALLOWED` / `NB_ADMIN_ROLE_REQUIRED` from `_resolve`) is an `ACTION` that makes the tenant `WARN`, never a silent skip: re-run logged in at that tenant.

#### Plan (read-only)

`--all` takes every state tenant listed in `tenant.tenants` at the root(s): `--roots a,b`, else `STATE_ROOT`, else the running bridge's `NOVU_BRIDGE_CORE_SMS_DEFAULT_TENANT` (the deploy sets it to `state_root`). City codes are skipped (configuration lives at the state tenant), and so is test junk (`--exclude`, default `(?i)^(PW_|pwt)`). `--tenant X` (repeatable) names tenants instead.

<table><thead><tr><th width="150.8984375">Category</th><th>Meaning</th><th>apply does</th></tr></thead><tbody><tr><td><code>none</code></td><td>no configuration in either namespace</td><td>nothing, unless <code>--adopt-defaults</code> (then the shipped defaults)</td></tr><tr><td><code>defaults</code></td><td>legacy rows equal the shipped defaults, after conversion</td><td>copies the tenant's rows</td></tr><tr><td><code>customised</code></td><td>legacy rows differ; the plan lists rows only in the tenant, rows only in the defaults (<strong>not</strong> added) and every changed field</td><td>copies the tenant's rows, exactly</td></tr><tr><td><code>migrated</code></td><td>already served from <code>NOTIFICATIONS.*</code></td><td>adds missing catalogue rows, channel rows that are still only in the legacy master (when <code>NOTIFICATIONS.Channel</code> is empty, so nothing changes), and provider pins; nothing else</td></tr><tr><td><code>partial</code></td><td><code>NOTIFICATIONS.*</code> rows but no Routing, a copy that did not finish (configuration rows, or channel rows — a legacy channel code missing from a non-empty <code>NOTIFICATIONS.Channel</code> is <strong>off</strong> until it is copied), no event catalogue, or an interrupted default seed</td><td>finishes the copy with the tenant's own rows, channel codes one by one (<code>--adopt-defaults</code> for an interrupted default seed)</td></tr></tbody></table>

For each tenant, the plan also shows the rows `apply` would create, after conversion (`--show-rows full` prints them whole); the event catalogue, generated from the tenant's **live** PGR workflow in egov-workflow-v2 (the repository template, with a warning, when that cannot answer); channel policy now and after; the provider decision per channel; and warnings: routing for events the live workflow cannot produce, routed audiences with no template or none in the default locale, WhatsApp routes without an approved provider template, and rows the conversion drops (`AUTO_ESCALATE` / `SYSTEM` — the bridge drops them today too).

#### Review

Read every `customised` and `partial` tenant. The copy is the tenant's own rows through the conversion the bridge already applies to them (`notifications_convert.py` mirrors `LegacyMasterAdapter`), so what it sends today is what it sends after. Inactive rows stay inactive. Default rows it never had are not added.

The shipped default **wording** changed in 2.20: no `EGOVS` / `ई-गव फाउंडेशन` signature, no `DIGIT:` subject prefix, the assignee addressed as "Dear {emp\_name}" instead of "Shri …", and no `{ao_designation} - {ulb}` line (it shipped as literal `{ulb}` on a tenant without districts). A 2.12 tenant still on the 2.12 default wording therefore plans as `customised`, its only differences being `body` / `subject` against the new defaults. That is expected: `apply` copies the tenant's own rows, old wording included — nothing is rewritten. Edit the templates on **Configure** afterwards if you want the new wording.

**India (DLT):** a tenant that gets the new default wording — a fresh install, a self-serve workspace, `--adopt-defaults`, or templates edited to it — sends text that no longer matches SMS templates registered against the 2.12 wording. DLT matches on content and drops an unregistered message after accepting it, so re-register those templates first (setup-guide §8.4).

#### Apply

* `apply` needs `--yes`; without it, it prints the plan and exits `4`.
* With `--all` it needs `--only <categories>`. `customised` and `partial` tenants are applied only when listed there or named with `--tenant`: do `--only defaults` first, then the customised tenants one at a time.
* Write order: event catalogue, templates, provider templates, channel rows, **routing last**. Routing is held back when any catalogue, template or provider-template write fails; the tenant then stays on its legacy masters. Re-run to finish.
* **Apply in a quiet period.** The bridge rejects an event whose name is missing from a non-empty event catalogue — whichever namespace serves the tenant — dead-letters it, and caches the catalogue for 60 s. MDMS creates one row per call, so a tenant's first catalogue appears row by row (well under a second), and an event arriving in that window can be rejected (`NB_EVENT_NOT_IN_CATALOGUE`) for up to 60 s. `apply` prints the window; check Logs for `REJECTED` rows in it. A catalogue left incomplete fails the tenant (`EVENT CATALOGUE INCOMPLETE`): re-run at once. The catalogue lists PGR's workflow transitions; a thin event with any other name is rejected for a tenant that has one.
* Before and after writing, `apply` resolves one synthetic thin event per catalogued event and locale through `POST /novu-bridge/novu-adapter/v1/dispatch/_resolve` (nothing is sent, no ledger row is written), with made-up citizen and assignee contacts and the tenant's real role pools, and compares recipient, channel, locale and text. Any difference is listed and makes the tenant `WARN`. When the bridge cannot answer, the preview is skipped, and the output says so; when it refuses this login, that is an `ACTION` and the tenant ends `WARN` (Who logs in).
* It then re-reads every master and reports counts and any row that is missing or reads back differently.
* Re-running skips rows already there and reports them as present. A failed tenant does not stop the others.

Exit codes: `0` ok · `1` warnings · `2` a tenant failed · `3` a write was refused with 403 (restart `egov-accesscontrol`; a tenant that never had the deploy's notification step also lacks its access-control rows — run `NOTIF_TENANT=<tenant> NOTIF_SEED_PHASE=access python3 seed-notifications.py` first) · `4` refused to start (including provider creation refused for this login — Who logs in).

#### Providers

Novu integrations belong to the whole deployment; `NOTIFICATIONS.Channel.provider` pins one per channel per tenant. A pin outranks the channel's `gateway` and `NOVU_BRIDGE_SMS_PROVIDER`. For each channel that is on:

<table><thead><tr><th width="359.921875">Situation</th><th>apply</th></tr></thead><tbody><tr><td>exactly one active integration of that channel (Novu's in-app inbox ignored)</td><td>pins it</td></tr><tr><td>several</td><td>lists them and leaves the channel unpinned (Novu keeps using its primary); choose with <code>--provider SMS=&#x3C;identifier></code></td></tr><tr><td>none</td><td>operator action — the channel cannot deliver; the tenant ends <code>WARN</code></td></tr><tr><td>SMS on the direct SMSCountry route (<code>gateway: smscountry</code>, or <code>NOVU_BRIDGE_SMS_PROVIDER=smscountry</code>)</td><td>kept as it is: a pin would move SMS onto a Novu integration. Move it on purpose with <code>--create-smscountry-provider</code> or <code>--provider SMS=…</code></td></tr><tr><td>on through <code>NOVU_BRIDGE_CHANNELS_ENABLED</code>, no channel row</td><td>no pin (nowhere to store one)</td></tr></tbody></table>

`--create-smscountry-provider` (or `--create-provider <type>`, for `twilio-sms`, `twilio-whatsapp`, `smtp`, `smscountry`, `ozeki`, `jasmin`) creates a catalogue provider through `POST /novu-bridge/novu-adapter/v1/providers` and pins it for the tenants in the run. It needs a login at a state that owns the providers (Who logs in). Credentials are read only from `--credentials-file`, a JSON object keyed by type, with the catalogue's field keys (the Novu provider's credential keys):

```json
{"smscountry": {"name": "SMSCountry", "credentials": {"user": "…", "password": "…", "from": "<sender id>"}}}
```

An SMSCountry, Ozeki or Jasmin provider is one of **DIGIT's providers mounted into the Novu worker** (providers.md). A deploy of this release mounts them; redeploy **before** moving a tenant onto one if the box was deployed earlier. While the `novu-worker` container does not preload them the script refuses to create them (exit `4`). It also refuses (exit `4`) when it **cannot check** the worker — no docker where it runs, a worker container by another name (`--worker-container <name>`), a remote box reached with `--digit-url`, a Kubernetes install, or `--worker-container ''`. Check the worker yourself (Compose: `docker inspect novu-worker` shows `NODE_OPTIONS` with `digit-novu-providers/register.js`; Kubernetes: `global.novuWorkerDigitProviders` / the novu chart's `worker.digitProviders.enabled` is `true`), then re-run with `--assume-worker-providers`; the script says it did not check. The direct SMSCountry route needs no Novu worker at all.

The file must be mode `0600` or stricter, or it is refused. Values are never printed or written to the report. The identifier is derived from the name, so a re-run finds the provider rather than creating a second one. An `"identifier"` you set yourself must start with the type (`smscountry-…`, `twilio-sms-…`): novu-bridge reads the type back from it and refuses any other (`400 NB_INVALID_PROVIDER`), so the script refuses it at plan time (exit `4`) instead. Integrations created before the catalogue are listed as working but not rotatable from the Configurator; re-create them from the catalogue when convenient.

#### The report

`--report <file>` writes JSON: `mode`, `roots`, `excluded`, `bridgeEnv`, `providers` (`integrations` as the bridge lists them, `preCatalog`, `created` — credential key names only), and per tenant `category`, `reasons`, `counts`, `defaultsDiff` (`onlyInTenant`, `onlyInDefaults`, `changed` with the default and tenant values), `planned` (full rows per master), `catalogue`, `channels` (`now`, `after`), `providers` (the decision per channel), `warnings`, `operatorActions`, `notes`. After `apply`: `apply.write` (created / present / failed per master), `apply.catalogue` (the write window, `COMPLETE` or `PARTIAL`), `apply.verify`, `apply.mismatches`, `apply.preview` (`differences`, and both previews) and `result` — `OK`, `WARN`, `FAILED` or `SKIPPED`.

#### Verify

Run `plan --all` again: every tenant you applied is `migrated` with nothing to create, and `/config/source` (below) reports `NOTIFICATIONS.Routing` for it. Then send one real complaint transition per migrated tenant and check Logs.

#### Access-control actions

The deploy's access phase gives the Notifications screens and the provider, `/config/source` and `/dispatch/_resolve` endpoints their 17 access-control actions (`NOTIF_ACTIONS` in `local-setup/scripts/seed-notifications.py`) and grants them to the roles the tenant has. On a fresh or `full-dump.sql` install, they are actions 4623–4639. The seed matches each one **by URL**:

* An action that already has the URL is reused, whatever its id, so a re-run creates nothing and restarts nothing. A deactivated one stays deactivated: its grants are written, but requests to it stay refused, and the seed says so (`ACL-ACTION-INACTIVE`). The deploy goes on, unless its own data phase needs that action, in which case it stops with a 403 (exit 3) naming it;
* before creating one, the seed looks its URL up in MDMS exactly, so an action it already has (say, one an operator switched off) is found and never created a second time;
* a missing one is created at its usual id when that id is free;
* when the usual id is already another endpoint's at the tenant (a 2.12 tenant whose own actions table outgrew the dump), it is created at the next free id above 9000000 and the deploy prints `ACL-ID-COLLISION: action 4630 is <that endpoint> at <state>, not /mdms-v2/v2/_create/NOTIFICATIONS.Routing — creating it as action 9000008`. This is not a failure; the role-actions go to the new id. No seed that creates actions by a fixed ID uses IDs above 9000000 (the platform baseline uses 900000–901017). If MDMS answers "duplicate" for an ID the seed thought was free, the seed looks that ID up: it is used if it holds the same URL, and otherwise the next ID is tried.

egov-accesscontrol only sees new or changed role-actions after a restart. The deploy restarts it whenever the seed prints `ACL-CHANGED`; by hand, see setup-guide.md §8.2.

The queries below use `<state>` for the state tenant the seed ran at (`ke`, `mz`, `pg`, ...). Replace it before running them; a query left with another deployment's tenant returns no rows and looks like a clean result.

**Lines that fail the step at that root:**

* `ACL-ID-TAKEN: no free action id for <url> at <state> after 5 attempts` (exit 2). Every id tried was held by another URL that the seed's read did not show. Nothing was granted for that URL. Look at what holds the IDs the line before it names:

  ```sql
  select uniqueidentifier, isactive, data->>'url' from eg_mdms_data
  where tenantid = '<state>' and schemacode = 'ACCESSCONTROL-ACTIONS-TEST.actions-test'
    and data->>'id' = '<id>';
  ```

  Then create the notification action yourself (an `ACCESSCONTROL-ACTIONS-TEST.actions-test` row with that `url` at an unused id above 9000000) and re-run `--tags notifications`: the seed finds it by URL and grants it.
* `ACL-ACTION-INACTIVE` followed by a 403 in the data phase (exit 3): the data phase needs that action, and it is switched off. Reactivate it to let the deploy finish.

**Lines to review** (the deploy shows them under `notif-seed — WARNING: access-control rows to review`):

* `ACL-STALE-GRANT: <role> is granted action 4635 at <state>, which is <another endpoint>, not <notification url>`. That ID holds another endpoint, and the grant has exactly the shape this seed writes. A 2.20 build older than [#2603](https://github.com/egovernments/Citizen-Complaint-Resolution-System/pull/2603) wrote such grants: it created the 17 actions at 4623–4639 regardless, counted "already present" where an id was another endpoint's, and granted `MDMS_ADMIN` / `ACCOUNT_ADMIN` / `SUPERUSER` (plus `CSR` / `GRO` / `PGR_LME` on 4625 and 4638) to that endpoint. The step then failed with a 403 on `RAINMAKER-PGR.NotificationChannel`. The seed never removes a grant, because the tenant may have made the same one itself. If the role should not reach that endpoint, deactivate the grant (`isActive: false` through `/mdms-v2/v2/_update/ACCESSCONTROL-ROLEACTIONS.roleactions`), then restart egov-accesscontrol. To see every grant on those IDs with the URL it opens:

  ```sql
  select r.data->>'rolecode' as role, r.data->>'actionid' as action, a.data->>'url' as url,
         r.isactive, to_timestamp(r.createdtime / 1000) as created
  from eg_mdms_data r
  left join eg_mdms_data a on a.tenantid = r.tenantid
    and a.schemacode = 'ACCESSCONTROL-ACTIONS-TEST.actions-test' and a.data->>'id' = r.data->>'actionid'
  where r.tenantid = '<state>' and r.schemacode = 'ACCESSCONTROL-ROLEACTIONS.roleactions'
    and r.data->>'actionid' in (select generate_series(4623, 4639)::text)
  order by r.createdtime desc;
  ```
* `ACL-BASELINE-ID-RETURNED: action 901001 at <state> is /localization/messages/cache-bust again`. One of the 17 actions sat on an ID the platform baseline defines for another endpoint. The baseline is `backend/pgr-services/src/main/resources/onboarding/platform-baseline-v1.json`, which digit-mcp's tenant bootstrap applies at `state_root` on every deploy. It skips an id that is already taken and grants its roles to whatever holds it. The first build of this change put a tenant's actions on 901001–901017 that way. The seed moves the action above 9000000, keeps every role that reached it, and hands the id back in an order that never widens access: it deactivates the grants on that id that the baseline does not make, rewrites the row to the baseline's own action, and makes the baseline's grants on it active.
* `ACL-BASELINE-ID-KEPT`: the same, but a step was refused or failed (it needs `/mdms-v2/v2/_update/ACCESSCONTROL-ACTIONS-TEST.actions-test` and `.../ACCESSCONTROL-ROLEACTIONS.roleactions`, which `MDMS_ADMIN` holds). The notification action already works from its new ID. Re-run `--tags notifications` as a user that holds them, or make the same changes by hand in that order.

#### Channel rows: what happens to an existing tenant

Channel rows switch delivery on and off the moment they exist: a tenant with **any** active channel row (in `NOTIFICATIONS.Channel`, else `RAINMAKER-PGR.NotificationChannel`) is decided by those rows alone — a channel without a row is off — and only a tenant with none follows `NOVU_BRIDGE_CHANNELS_ENABLED`. So the seed does not write the committed all-off rows; it reads the allowlist the **running** `novu-bridge` container has and decides per tenant:

| The tenant today                                           | The seed writes                                                                                                                                              | Effect                                                                                                |
| ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------- |
| No channel rows; allowlist e.g. `SMS`                      | One row per channel, `enabled` = listed (SMS on, EMAIL/WHATSAPP off), no `gateway` — in the legacy master for a 2.12 tenant, else in `NOTIFICATIONS.Channel` | None: the rows say what the env said. `NOVU_BRIDGE_SMS_PROVIDER` still picks the SMS transport        |
| No channel rows; empty allowlist (a new deployment)        | All three rows, off                                                                                                                                          | None: nothing was enabled. The out-of-box default                                                     |
| Has channel rows                                           | Nothing                                                                                                                                                      | None. A channel on the allowlist without a row is reported: it has been off since the tenant got rows |
| Only inactive rows, one of them for an allowlisted channel | Nothing (`mode=conflict`)                                                                                                                                    | None; decide that channel on the Channels screen                                                      |
| Seed run by hand without `NOTIF_CHANNELS_ALLOWLIST`        | Nothing (`mode=unknown`)                                                                                                                                     | None; the tenant stays on the env allowlist                                                           |

The seed prints its decision — `CHANNEL-POLICY: tenant=… mode=… allowlist=… target=…` and one line per channel — and the deploy shows it under `notif-seed — result`. A run that dies halfway is finished by the next one with the same rule. The migration copies a 2.12 tenant's channel rows into `NOTIFICATIONS.Channel` as they are (plus provider pins), **per channel code**: from its first row there, `NOTIFICATIONS.Channel` alone decides and a channel without a row is off, so a re-run of `apply` copies any code an interrupted one left out (the tenant plans as `partial`, and the plan shows e.g. `EMAIL off → ON`). A code already there is never rewritten. The bridge caches channel policy for 60 s, so a change takes up to a minute to apply.

A tenant created by `default-data-handler` (a new tenant, or a Helm bootstrap) starts with the committed all-off rows whatever the allowlist says; the Helm tier runs no seed and sets no `NOVU_BRIDGE_CHANNELS_ENABLED`, so switch channels on in the Configurator there.

#### Where is a tenant?

There is no setting that chooses old or new — the data does. Check with any of:

* `migrate-notifications.py plan --tenant mycity`: the category.
* **Configurator → Notifications → Configure**: no banner = on `NOTIFICATIONS.*`.
* `GET /novu-bridge/novu-adapter/v1/config/source?tenantId=mycity` (employee token): one entry per master with the schema read, row count, `legacy` and `stale` flags.
* The `source_path` column on `nb_dispatch_log` (and the **Produced by** column / `sourcePath` filter on Logs): `PRERENDERED` = a finished envelope from a producer, `RESOLVED` = a thin event resolved by the bridge. After the cutover, complaint rows should read `RESOLVED`.

### 4. DIGIT's older OTP service sends through novu-bridge

Citizen sign-in in 2.20 goes through the Identity BFF: the deploy refuses `enable_keycloak: false`, and Kong answers 404 for the older OTP endpoints (`/user-otp/v1/_send` and the rest) unless `identity_legacy_user_endpoints` reopens them. The BFF sends its codes with its own sender (`http` or `log`), not through notifications; sending them through novu-bridge (`messages/_send`) is a follow-up. This section is about **DIGIT's older OTP service** (`egov-otp` + `user-otp`), where a deployment still uses it: its SMS moves from the retired senders to novu-bridge.

* `egov-notification-sms`, `otp-publisher` and `novu-bridge-endpoint` are gone from Compose. `up -d` does not stop containers of services it no longer knows, and a leftover `egov-notification-sms` still consumes `egov.core.notification.sms` next to novu-bridge — every OTP sent twice, through two providers. The deploy removes the three containers when `enable_novu` is on (only a container this deployment's compose created), **once novu-bridge has taken over `egov.core.notification.sms`**: it is healthy, and every partition of the topic either has an offset committed by the `novu-bridge` consumer group (it resumes from there) or is owned by a live member of it (`docker exec digit-redpanda rpk group describe novu-bridge`). "Running" is not enough: on a 2.12 box whose OTPs went through `egov-notification-sms` the group has never committed an offset on that topic, and the bridge's first subscription starts at the **latest** offset (below), so every OTP published while it boots would be skipped — and lost if the old sender were already gone. On an upgrade, the deploy recreates novu-bridge first (a failed `up -d novu-bridge` stops the deploy there, old senders untouched), waits up to 5 minutes for the handoff, and removes them before the main `up -d`; otherwise (a first enable, or no handoff yet) it waits up to 10 minutes after the main `up -d`. Without a handoff, they are **kept**, with `WARNING: the retired OTP senders were kept`: until then, an OTP can arrive twice, never not at all. Redpanda that cannot be asked (`rpk` fails) six times in a row, about a minute, ends either waiting early with the same outcome: the old senders stay. `enable-notifications.sh` does the same (step 1 and step 2). By hand: start the new bridge, wait until `rpk group describe novu-bridge` lists every partition of `egov.core.notification.sms` with a member or a committed offset, then `docker rm -f egov-notification-sms otp-publisher novu-bridge-endpoint`. On Helm `egov-notification-sms` is `installed: false` in `devops/deploy-as-code/charts/core-services/coreservices-helmfile.yaml` (kept for a one-release rollback).
* `egov-otp` + `user-otp` still generate and validate OTPs and publish the SMS to `egov.core.notification.sms`; novu-bridge translates it (`eventType CORE_SMS`, ledger event `CORE.SMS.OTP`) and delivers it through the tenant's SMS channel.
* The bridge's first subscription to `egov.core.notification.sms` starts at the **latest** offset, not the beginning, so it never replays old OTPs; the old senders stay until it is subscribed (above), so none published meanwhile is lost. An OTP whose `expiryTime` has passed is dropped with an INFO log (no phone, no text) — no ledger row, no DLQ message.
* An OTP's national number now gets the country code of the OTP's tenant (`common-masters.MobileNumberValidation`), which outranks `NOVU_BRIDGE_CORE_SMS_COUNTRY_CODE`; the latter applies only at a tenant with no rule. A tenant whose seeded rule differs from the deployment code — `pg`'s `+91` on a Kenyan box — sends its OTPs with the rule's code: fix the rule, not the setting (setup-guide §7).
* The tenant an OTP is checked against is `NOVU_BRIDGE_CORE_SMS_DEFAULT_TENANT` when the message carries none: Compose rewrites it to `state_root` right after it copies the compose file, before the bridge-first recreate or any other container starts, so `novu-bridge` never runs with `pg` on a box whose state root is not `pg` (it would file every tenant-less OTP under `pg` and drop it as `SKIPPED / NB_NO_PROVIDER`, with the old sender already gone). The deploy fails at the end if the running bridge has anything other than `state_root`. Helm reads `state-level-tenant-id` from the `egov-config` ConfigMap. **SMS must be switched on, with a provider, at that tenant** or every OTP is `SKIPPED / NB_NO_PROVIDER` and phone login through this service stops.

### 5. Bridge settings now come from host\_vars

`./deploy.sh` regenerates `/opt/digit/.env` from `templates/digit.env.j2` on every run. These are now rendered from host\_vars, so a value you added to `.env` by hand must move there or it is lost on the next deploy:

<table><thead><tr><th width="364.53515625">host_vars</th><th>Env</th></tr></thead><tbody><tr><td><code>novu_bridge_receipts_secret</code></td><td><code>NOVU_BRIDGE_RECEIPTS_SECRET</code></td></tr><tr><td><code>novu_bridge_preference_enabled</code> / <code>novu_bridge_preference_fail_open</code></td><td><code>NOVU_BRIDGE_PREFERENCE_ENABLED</code> / <code>_FAIL_OPEN</code></td></tr><tr><td><code>novu_bridge_core_sms_country_code</code></td><td><code>NOVU_BRIDGE_CORE_SMS_COUNTRY_CODE</code></td></tr><tr><td><code>notification_stack_tag</code>, <code>pgr_services_db_image</code>, <code>novu_bridge_db_image</code>, <code>digit_user_preferences_service_db_image</code></td><td><code>NOTIFICATION_STACK_TAG</code>, <code>PGR_SERVICES_DB_IMAGE</code>, <code>NOVU_BRIDGE_DB_IMAGE</code>, <code>DIGIT_USER_PREFERENCES_SERVICE_DB_IMAGE</code></td></tr></tbody></table>

### Rollback

* **Redeploy the 2.12 checkout, not just the 2.12 image tags.** The 2.12 pgr-services needs `PGR_NOTIFICATION_CONFIG_DRIVEN=true` (and the `MDMS_HOST` / legacy topic settings) to send anything, and 2.20's compose files no longer set them; with 2.20's files, a 2.12 pgr-services sends nothing. Roll pgr-services and novu-bridge back together, and the retired containers come back with the 2.12 compose files.
* The database needs no down-migration: 2.20's bridge migrations only add columns with defaults, which the 2.12 bridge ignores.
* **Configurator edits made after a tenant's migration are lost.** They were written to `NOTIFICATIONS.*` only; 2.12 reads `RAINMAKER-PGR.*`, which still holds the configuration as it was when the tenant was migrated. (Under 2.20 there is no way back: migration is one-way per tenant.)
* The 2.12 bridge ignores channel rows and goes back to `NOVU_BRIDGE_CHANNELS_ENABLED`.
* Thin events dead-lettered while the versions were mixed cannot be replayed by a 2.12 bridge.

### Removed settings

Delete these wherever you set them; nothing reads them in 2.20.

<table><thead><tr><th width="227.08203125">Where</th><th>Removed</th></tr></thead><tbody><tr><td>host_vars</td><td><code>pgr_notification_config_driven</code>, <code>build_otp_publisher</code>, <code>otp_publisher_image</code></td></tr><tr><td>pgr-services env / properties</td><td><code>PGR_NOTIFICATION_CONFIG_DRIVEN</code> (<code>pgr.notification.config.driven</code>), <code>pgr.notification.default.locale</code>, <code>pgr.notification.rolepool.page.size</code>, <code>pgr.notification.rolepool.max.pages</code>, <code>complaints.domain.events.enabled</code>, <code>complaints.domain.events.default.locale</code>, <code>notification.sms.enabled</code>, <code>egov.user.event.notification.enabled</code>, <code>kafka.topics.notification.sms</code>, <code>egov.usr.events.*</code>, <code>egov.pgr.events.*</code>, <code>egov.ui.app.host.map</code></td></tr><tr><td>novu-bridge env / properties</td><td><code>NOVU_BRIDGE_KAFKA_INPUT_TOPIC</code> (now <code>NOVU_BRIDGE_KAFKA_INPUT_TOPICS</code>, comma-separated), <code>NOVU_BRIDGE_KAFKA_RETRY_TOPIC</code> (no retry topic), <code>NOVU_BRIDGE_CHANNEL</code>, <code>MDMS_HOST</code> / <code>MDMS_SEARCH_PATH</code> (now <code>NOVU_BRIDGE_MDMS_HOST</code> / <code>_SEARCH_PATH</code>), <code>NOVU_BRIDGE_CONFIG_HOST</code> / <code>_CONFIG_RESOLVE_PATH</code> / <code>_CONFIG_SEARCH_PATH</code> (Helm; env.yaml <code>novu-bridge.config-resolve-path</code> / <code>config-search-path</code>)</td></tr><tr><td>build</td><td><code>novu-bridge-endpoint</code> and <code>otp-publisher</code> images (no longer built)</td></tr></tbody></table>

Changed defaults worth checking: `NOVU_BRIDGE_CHANNEL_POLICY_SCHEMA` is `NOTIFICATIONS.Channel` (legacy fallback automatic — leave it unset); `NOVU_BRIDGE_KAFKA_INPUT_TOPICS` adds `notifications.events`; `NOVU_BRIDGE_PROXY_ALLOWED_ROLES` adds `MDMS_ADMIN`; the new `NOVU_BRIDGE_PROXY_ADMIN_ROLES` gates provider create/rotate/delete.

### Per-tenant notification accounts

New in this release (tenant-accounts.md): each signup workspace gets its own Novu organisation at tenant creation. On a box with `enable_novu` it is on by default (Helm: off unless `tenant-accounts.enabled: true`); the upgrade deploy then generates its secrets into OpenBao and moves the Novu admin off its old password (dashboard login included: read the new one with `bao kv get -field=novu_admin_password <secrets_path>`). Nothing moves by itself: every existing tenant, migrated or not, keeps the shared account until you list it in `novu_tenant_accounts_backfill` (then add its providers in its own organisation). To stay exactly as before, set `novu_tenant_accounts: false`.

**A Novu admin with a custom password: pin it before upgrading.** The deploy can move the admin only off a password it knows (`novu_admin_password_legacy`, default the published `Digit@12345`), and it signs in with the result before novu-bridge gets it. If your admin's password was changed by hand, put the current one in host\_vars as `novu_admin_password_legacy` (the deploy moves it to a generated one) or as `novu_admin_password` (kept as it is), plus `novu_admin_email` if the admin is another account. Unpinned, Novu refuses the sign-in and the deploy switches per-tenant accounts **off for that deploy** and continues on the shared account, printing `novu tenant accounts — ACTION: per-tenant notification accounts are OFF on this box for this deploy` with what to pin; the next deploy checks again. It stops instead when host\_vars sets `novu_tenant_accounts: true` or when workspaces already have their own organisation (tenant-accounts.md).

### After the upgrade

* [x] `docker ps` (or `kubectl get pods -o wide`) shows pgr-services and novu-bridge on the tag you chose, and the `novu-bridge-migration` / `pgr-services-migration` containers (init containers on Helm) exited 0 from the same tag.
* [x] No `egov-notification-sms`, `otp-publisher` or `novu-bridge-endpoint` container is running.
* [x] `./deploy.sh <tenant>` (or `--tags notifications`) ran with `failed=0` and no `notif-seed — WARNING` task.
* [x] `migrate-notifications.py plan --all` leaves no tenant `defaults`, `customised` or `partial` that you meant to migrate, and every `apply` ended `OK` (or its `WARN` lines are understood).
* [x] The `CHANNEL-POLICY` lines show every channel you use today as `ON` or as an existing enabled row.
* [x] `/config/source` shows no `legacy: true` for any tenant you expect to edit.
* [x] Each channel you use is on with a provider selected (**Notifications → Channels**).
* [x] **Validate** on Configure reports zero errors.
* [x] One real complaint transition produced `RESOLVED` rows on Logs and arrived on a handset.
* [x] Where DIGIT's older OTP service is still used: a phone login through it works and left a `CORE.SMS.OTP` row.
* [x] `novu-bridge.dlq` has no new messages (`rpk topic consume novu-bridge.dlq -o start`; see kafka-events.md).


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.digit.org/complaints-management/deploy/configure/notifications/migrate-notifications.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
