> For the complete documentation index, see [llms.txt](https://docs.digit.org/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.digit.org/complaints-management/release-notes/deployment-changelog.md).

# Deployment Changelog

CMS v2.20 Deployment Changelog (v2.12 → v2.20)

## Overview

This page covers what an operator must change to move a CMS deployment from v2.12 to v2.20. It lists only what affects a deployment.&#x20;

**The six things that matter most**

1. **Sign-in is now Keycloak only.** Set the identity secrets and an SMTP server before deploying (section 1).
2. **Existing staff must be linked before they can sign in.** Staff on city tenants cannot be linked (section 4).
3. **Run a full deploy, not `--tags`.** Keycloak 24 → 26.7.3 is a one-way migration.
4. **Notifications need a one-time migration per tenant**, or tenants with complaints stop getting notifications.
5. **Automatic escalation is off after the upgrade.** Turning it back on is one switch for the whole deployment (`PGR_ESCALATION_ENABLED`, section 2).
6. **Back up every database first.** There is no tested rollback.

### 1. Settings to change

All in `local-setup/ansible/inventory/host_vars/<tenant>.yml`. Compare with `_example.yml`.

**What stops the deploy**

The deploy stops only if `enable_keycloak` is `false`, or if `auth_provider`, `citizen_auth_provider` or `employee_auth_provider` is set to `keycloak`. The other retired keys below are ignored or only print a warning.

**Keep or remove**

<table><thead><tr><th width="241.1484375">Setting</th><th width="153.96875">Action</th><th>Why</th></tr></thead><tbody><tr><td><code>enable_keycloak</code></td><td>Remove</td><td>Keycloak is always on. Setting it to <code>false</code> stops the deploy. On master, removing it also stops Prometheus scraping Keycloak (section 7)</td></tr><tr><td><code>auth_provider</code>, <code>citizen_auth_provider</code>, <code>employee_auth_provider</code></td><td>Remove</td><td>Replaced by the Identity BFF; the value <code>keycloak</code> stops the deploy</td></tr><tr><td><code>enable_digit_ui_v2</code>, <code>nginx_features.digit_ui_v2</code>, <code>digit_ui_v2_repo</code>, <code>digit_ui_v2_branch</code></td><td>Remove</td><td>The old <code>/citizen/</code> app is retired (#2290); these only print a warning</td></tr><tr><td><code>keycloak_client_id</code>, <code>login_tenant_allowlist</code>, <code>token_exchange_system_username</code>, <code>token_exchange_system_user_type</code></td><td>Remove</td><td>No longer used</td></tr><tr><td><code>pgr_notification_config_driven</code>, <code>novu_bridge_channel</code></td><td>Remove</td><td>Notifications are configured per tenant in the Configurator</td></tr><tr><td><code>build_otp_publisher</code>, <code>otp_publisher_image</code>, <code>enable_overpass</code>, <code>build_default_data_handler</code></td><td>Remove</td><td>Services removed</td></tr><tr><td><code>pgr_services_image</code>, <code>configurator_image</code> and every other per-image pin</td><td>Remove</td><td>A pin wins over <code>--image-tag</code>, so a 2.12 pin keeps the old build running against the 2.20 migrations and notification stack. The deploy only warns about mismatched pins</td></tr><tr><td><code>token_exchange_system_password</code></td><td>Keep until <code>identity_digit_admin_password</code> is in OpenBao</td><td>The deploy falls back to it for the BFF admin password</td></tr></tbody></table>

**Secrets** (OpenBao)

`bootstrap_secrets` is read only on a tenant's first deploy. On an existing box, add these with `bao kv patch`. Passwords must not contain `$`.

| Secret                                                           | Required                                | Notes                                                                                                                                                                                                                        |
| ---------------------------------------------------------------- | --------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `identity_digit_admin_password`                                  | Yes                                     | DIGIT admin the BFF uses                                                                                                                                                                                                     |
| `keycloak_admin_password`                                        | Yes                                     | If Keycloak 24 already runs, it must match its current admin password, or realm setup fails after the one-way migration                                                                                                      |
| `keycloak_db_password`                                           | Yes (not enforced yet, #2618)           | If it is unset, Keycloak's database keeps the compose default password `keycloak`. If it is set, the deploy changes the Keycloak database user's password to match, so it is safe to add on a box where Keycloak already ran |
| `identity_smtp_password`                                         | Yes, unless `identity_smtp_auth: false` |                                                                                                                                                                                                                              |
| `keycloak_google_client_secret`, `keycloak_github_client_secret` | Only with Google or GitHub sign-in      | A `keycloak_google_client_id` or `keycloak_github_client_id` without its secret fails the deploy                                                                                                                             |

**Add** (needed for sign-in to work)

| Setting                                                                                                                      | Default                                          | What to put                                                                                                                                                                                                                                                                                                                                                                                                                   |
| ---------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `identity_smtp_host`, `identity_smtp_from`, `identity_smtp_user`                                                             | none                                             | Mail server for sign-in and password-reset emails. The deploy checks these only after Keycloak has migrated (#2618)                                                                                                                                                                                                                                                                                                           |
| `identity_smtp_port`, `identity_smtp_from_display_name`, `identity_smtp_auth`, `identity_smtp_ssl`, `identity_smtp_starttls` | `587`, `DIGIT Identity`, `true`, `false`, `true` | Change only if your server differs                                                                                                                                                                                                                                                                                                                                                                                            |
| `identity_citizen_signin_methods`                                                                                            | `[]` (citizens cannot sign in)                   | Usually `[phone_otp]`. That needs an OTP channel: `identity_citizen_otp_sender: http` **and** `IDENTITY_OTP_SENDER_URL`, which host\_vars does not set yet, so add it with a compose overlay (guide, citizen OTP). With `http` set but no URL, phone sign-in is hidden and the deploy does not warn. On test boxes only, `identity_dev_fixed_otp: true` works instead. `enable_otp_services` is not a channel for these codes |

**Review** (these have defaults and do not stop the deploy)

| Setting                                   | Default                                                                                                                                                    | What to check                                                                          |
| ----------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- |
| `identity_signup_methods`                 | `[magic_link, google, github]`                                                                                                                             | Who can create a new workspace. An empty list falls back to the default. See section 2 |
| `digit_ui_default_tenant_slug`            | empty                                                                                                                                                      | Tenant that old `/digit-ui/` links redirect to; empty means they return 404            |
| `notification_stack_tag` (or `image_tag`) | unset: the compose default, `master-b4adf0d` on master (`nightly-develop` on develop). The deploy's rolling-tag warning names `nightly-develop` either way | One release tag for pgr-services, novu-bridge and user-preferences                     |

**Add** (only if you use them)

<table><thead><tr><th width="312.59765625">Setting</th><th>When</th></tr></thead><tbody><tr><td><code>keycloak_google_client_id</code> or <code>keycloak_github_client_id</code> (+ secret above), and <code>google</code> / <code>github</code> in <code>identity_signin_methods</code> / <code>identity_signup_methods</code></td><td>Google or GitHub sign-in or sign-up. Register the redirect URI <code>https://&#x3C;domain>/auth/realms/digit/broker/google/endpoint</code> (or <code>.../broker/github/endpoint</code>) with the provider (the realm is now <code>digit</code>)</td></tr><tr><td><code>configurator_posthog_key</code>, <code>configurator_posthog_host</code>, <code>configurator_sentry_dsn</code></td><td>You track Configurator usage or errors</td></tr><tr><td><code>identity_configurator_roles</code></td><td>Roles other than the admin defaults need the Configurator</td></tr><tr><td><code>kong_cors_origins</code></td><td>Browsers call the APIs from a domain other than your own. The list replaces the default, so include <code>https://&#x3C;domain></code> too</td></tr></tbody></table>

**HRMS default password (#2725):** the HRMS env var `EGOV_HRMS_DEFAULT_PASSWORD` is renamed `EGOV_HRMS_EMPLOYEE_DEFAULT_PASSWORD` in the Ansible `.env` and compose files (Helm already used the new name). On `hrms_dev_mode: true` boxes, the OpenBao value `egov_hrms_default_password` now reaches HRMS; replace the `CHANGE_ME` placeholder, or employee creation without a password fails.

### 2. Defaults that now behave differently

| What                                         | v2.12                                                                                            | v2.20                                                                                                                                                                                                                                                        | To get the old behaviour                                                                                                                                           |
| -------------------------------------------- | ------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Self-serve workspace sign-up                 | n/a                                                                                              | Magic link, Google and GitHub are listed by default, but none works until `identity_magic_link_enabled: true` or the Google / GitHub client IDs are set. A 2.12 host that still has a Google client ID and secret lets any Google account create a workspace | To turn self-sign-up off, set `identity_signup_methods: [magic_link]` and keep `identity_magic_link_enabled: false`. An empty list turns all three methods back on |
| Citizen Google sign-in                       | Available                                                                                        | Removed; citizens use phone and code                                                                                                                                                                                                                         | None                                                                                                                                                               |
| Automatic escalation                         | On                                                                                               | Off for the whole deployment: `PGR_ESCALATION_ENABLED` is hard-coded to `'false'` in `docker-compose.egov-digit.yaml`. There is no per-tenant switch                                                                                                         | After the escalation migration (section 4), set `PGR_ESCALATION_ENABLED: 'true'` for pgr-services in a `docker-compose.<tenant>.yml` overlay                       |
| Fixed test OTP 123456                        | On                                                                                               | Off                                                                                                                                                                                                                                                          | `identity_dev_fixed_otp: true` (test servers only)                                                                                                                 |
| Old login pages and endpoints                | Open                                                                                             | Closed                                                                                                                                                                                                                                                       | `identity_legacy_user_endpoints: true` (temporary rollback only)                                                                                                   |
| App links without a tenant (`/digit-ui/...`) | Work                                                                                             | 404                                                                                                                                                                                                                                                          | Set `digit_ui_default_tenant_slug`                                                                                                                                 |
| `/citizen`                                   | Old citizen app                                                                                  | 404; the app is at `/<tenant>/digit-ui/citizen`                                                                                                                                                                                                              | None                                                                                                                                                               |
| Container ports                              | Open on all interfaces                                                                           | Localhost only, behind a host firewall                                                                                                                                                                                                                       | `manage_host_firewall: false` if you run your own firewall                                                                                                         |
| Browser CORS                                 | Any origin                                                                                       | Your domain only                                                                                                                                                                                                                                             | List every allowed origin, including `https://<domain>`, in `kong_cors_origins`; the list replaces the default                                                     |
| `/status/` board                             | On (in `_example.yml`), no password                                                              | On, always behind HTTP basic auth; the password is generated into OpenBao (`bao kv get -field=status_basic_auth_password <secrets_path>`, for example `kv/digit/<tenant>`)                                                                                   | None; `nginx_features.status: false` turns the board off                                                                                                           |
| Fast-path database seed                      | Refused without `db_fast_path_ack_data_wipe: true`, and `_example.yml` shipped the ack as `true` | `_example.yml` ships the ack as `false`, and a Postgres storage guard checks the box first (#2101)                                                                                                                                                           | A host\_vars copied from 2.12 still has `true` and gets no new protection: set it to `false` on any box with data                                                  |
| Employee password reset                      | SMS                                                                                              | Email only                                                                                                                                                                                                                                                   | None; add staff email addresses                                                                                                                                    |
| HRMS dev-mode password (#2725)               | `eGov@123`                                                                                       | OpenBao `egov_hrms_default_password`                                                                                                                                                                                                                         | Set that secret                                                                                                                                                    |

### 3. Services added and removed

<table><thead><tr><th width="169.1328125">Change</th><th width="209.671875">Service</th><th>What you need to do</th></tr></thead><tbody><tr><td>Upgraded or added, always on</td><td>Keycloak (24 → 26.7.3) + Identity BFF (new, sign-in)</td><td>Plan resources for Keycloak, its database, the BFF and Redis; set the secrets and SMTP settings</td></tr><tr><td>Added, always on</td><td>Host firewall, status board, container metrics</td><td>Nothing, unless you manage your own firewall</td></tr><tr><td>Removed, <strong>not</strong> cleaned up</td><td>token-exchange-svc</td><td>Keeps running as an orphan. Remove it by hand after the cut-over (guide §3, §6)</td></tr><tr><td>Removed, cleaned up only sometimes</td><td>otp-publisher, novu-bridge-endpoint, egov-notification-sms</td><td>The deploy removes them only when <code>enable_novu</code> is on and novu-bridge has taken over the OTP topic, and never on macOS. Remove them by hand only after the deploy reports that novu-bridge has taken over. With <code>enable_novu</code> off nothing is sent twice; removing them during a deferred handoff loses OTPs</td></tr><tr><td>Removed</td><td>digit-ui-v2 (<code>/citizen</code>)</td><td>The deploy deletes <code>/opt/digit-ui-v2</code> and <code>/var/www/citizen</code> (not on macOS, or with <code>nginx_preserve_vhost</code>). Helm: <code>helm uninstall digit-ui-v2 -n egov</code></td></tr><tr><td>Rewritten</td><td>user-preferences (Go → Java)</td><td>Do not reuse the v2.12 image tag. Existing data is adopted automatically (#2703)</td></tr><tr><td>Optional</td><td>WhatsApp chatbot, alerting (Sherlog, master only), AI diagnosis (OpenSRE), read-only MCP</td><td>Off by default; see section 6</td></tr></tbody></table>

### 4. Manual steps

pgr-services, novu-bridge, user-preferences and chatbot migrations run automatically when the new images start. These steps do not.

<table><thead><tr><th width="153.8203125">Step</th><th width="187.77734375">When</th><th>How</th></tr></thead><tbody><tr><td>Rebuild db-migrations</td><td>Every existing Compose box</td><td>The deploy reuses the old db-migrations image, so the 2.20 MDMS migrations (including the workspace route grants) never run and admins get <code>AccessDeniedException</code> after picking a workspace. Rebuild and run it after the deploy (guide §4). Kubernetes: apply <code>V20261022000000__workspace_route_grants.sql</code> by hand</td></tr><tr><td>Link existing staff</td><td>Every existing box</td><td>A staff member can sign in only with a Keycloak user for their email, membership in the root's Organization and a binding to their DIGIT account. Link the first admin through the BFF internal API (guide §5.1), then the rest from Configurator › People › Employees. Staff whose home tenant is a city tenant cannot be linked; re-create them at the root</td></tr><tr><td>Staff email addresses</td><td>Employees without an email</td><td>Add them before upgrading; sign-in and password reset need email</td></tr><tr><td>Escalation migration</td><td>Server is on v2.12 (not v2.12.1)</td><td>Step 6a of the v2.12 → v2.12.1 migration guide, then <code>docs/operations/data-migration/pgr-escalation-self-loop.md</code> per state tenant</td></tr><tr><td>Notification migration</td><td>Every tenant that already has complaints</td><td><code>migrate-notifications.py plan</code>, then <code>apply --adopt-defaults</code> (notifications migration). Without it, those tenants send no complaint notifications</td></tr><tr><td>Remove token-exchange-svc</td><td>After staff sign-in works</td><td><code>docker rm -f token-exchange-svc</code></td></tr></tbody></table>

### 5. Upgrade steps

1. Back up every Postgres database.
2. If the server is on v2.12, upgrade to v2.12.1 first (including the escalation migration).
3. Update host\_vars and OpenBao secrets as in section 1.
4. Move any hand-edited `.env` values into host\_vars; the notification services read host\_vars only.
5. Run a **full** deploy with one tag: `./deploy.sh <tenant> --image-tag=<tag>`. Never use `--tags` for this upgrade: `configure-identity-keycloak.sh` runs only in a full deploy and sets the clients, redirect URIs, social-provider visibility and SMTP. Use the release tag `master-<promotion sha7>` (set when the final promotion merges) for every image, `identity-keycloak` and `identity-bff` included: the `master-b4adf0d` defaults predate the sign-in fixes in the release notes (#2700, #2704, #2714, #2724). The full deploy also adds `https://<domain>/identity/v1/callback/*` to the digit-ui clients' redirect URIs, which emailed email-change links need to return to the workspace (#2731); swapping only the images leaves them on the default workspace. On Helm, the Keycloak configure job adds it on `helm upgrade`.
6. Rebuild and run db-migrations (section 4).
7. Link the first admin, then the other staff (section 4).
8. Run the notification migration for each tenant.
9. Remove token-exchange-svc.
10. If you used automatic escalation, turn it back on: `PGR_ESCALATION_ENABLED: 'true'` in a compose overlay (applies to every tenant).

**Kubernetes / Helm differences**

* Set `identity.enabled: true` and deploy `identity-helmfile.yaml` (`docs/setup/deployment/helm-identity.md`).
* Set the SMTP host and from address (marked REQUIRED in `env.yaml`), create the `identity-secrets` Secret and the Keycloak database.
* Set `IDENTITY_OTP_SENDER` and `IDENTITY_OTP_SENDER_URL` for citizen phone sign-in.
* Set `ingress.defaultTenantSlug` and `global.notificationStackTag`.
* Set `NOVU_BRIDGE_PREFERENCE_ENABLED="false"` unless you collect citizen consent, or most messages are skipped.
* Run `helm uninstall digit-ui-v2 -n egov`.

**Check after upgrading**

* Health checks and the `/status/` board are green. Unhealthy observability containers (Tempo, Loki, Prometheus and others) now only warn (#2703); any other unhealthy container still fails the gate and skips the Keycloak realm setup.
* An admin, an employee and a citizen can each sign in at `/<tenant>/digit-ui/...`.
* A complaint can be filed, assigned, escalated, resolved and reopened, with a notification at each step.
* `tests/integration-tests`: `npm run test:lifecycle`.

**Rollback.** Not tested, and the Keycloak migration is one-way. Restore the backup and redeploy v2.12.1 images.

### 6. Optional new features

All off by default. Turn on only what you need.

| Feature                                                            | Turn on with                                                                                                                                                                                                        |
| ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 30-day trial for self-serve workspaces                             | `identity_workspace_trial_days: 30` (needs Novu email for reminders)                                                                                                                                                |
| Self-serve sign-up by email link                                   | `identity_magic_link_enabled: true` (`magic_link` is already in the default `identity_signup_methods`). Rate limit: `identity_magic_link_request_limit` (3) per `identity_magic_link_request_window_seconds` (1800) |
| Google sign-in or sign-up                                          | See section 1                                                                                                                                                                                                       |
| Staff phone join (an employee's phone signs in to the same person) | `identity_staff_phone_join: true`. Off until #2631 is done                                                                                                                                                          |
| WhatsApp complaints                                                | `enable_chatbot: true` + Twilio settings                                                                                                                                                                            |
| Alerting with automatic investigation (master only)                | `enable_sherlog: true` + a model API key in OpenBao                                                                                                                                                                 |
| AI incident diagnosis                                              | `enable_opensre: true` + an Anthropic API key in OpenBao                                                                                                                                                            |
| Public read-only automation server                                 | `enable_mcp_readonly: true` and `nginx_features.mcp_readonly: true`                                                                                                                                                 |
| Choose several boundaries at once in Geography                     | `configurator_geography_multiple_selections: true`                                                                                                                                                                  |
| Home-page banners                                                  | `citizen_banner_enabled`, `configurator_banner_enabled`                                                                                                                                                             |

Self-serve workspaces are upgraded to the latest platform seed in the background (`pgr_onboarding_baseline_upgrade_enabled: true`, the default): v12 at the beta, v16 on master and develop. Deploy-seeded roots such as `ke` are not upgraded this way.

### 7. Known deployment issues

| Issue                                                                                                                                     | Impact                           | Status                                                                            |
| ----------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- | --------------------------------------------------------------------------------- |
| master pins images to `master-b4adf0d`; develop still uses `nightly-develop`                                                              | Neither is a release tag         | The release tag is `master-<promotion sha7>`, set when the final promotion merges |
| On master before the pin PR, Keycloak metrics aren't scraped unless `enable_keycloak: true` is still set                                  | No sign-in metrics in monitoring | Fix in the master image-pin PR (the scrape target becomes unconditional)          |
| The migration guide (§2.1, §2.2) still says to keep `enable_keycloak: true`; this changelog and the deploy's own message say to delete it | Conflicting advice               | Guide to be updated                                                               |
| The identity SMTP check runs only after Keycloak has migrated                                                                             | A bad SMTP setting is found late | Open (#2618)                                                                      |
| Setup can show a boundary "usually temporary" error, then fail on Try again, although setup succeeded                                     | Confusing onboarding             | Open (#2709)                                                                      |
| No tested rollback; the Keycloak and database migrations cannot be undone                                                                 | Restore from backup only         | Open                                                                              |
| `_example.yml` comments still say images default to `nightly-develop`                                                                     | Misleading on master             | Docs fix pending                                                                  |


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.digit.org/complaints-management/release-notes/deployment-changelog.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
