> For the complete documentation index, see [llms.txt](https://docs.digit.org/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.digit.org/health/design/architecture/low-level-design/services/platform-services/facial-recognition.md).

# Facial Recognition

Offline facial verification for HCM field workers

## Overview

Offline-first facial recognition for HCM field workers. Provides identity verification at login, periodic re-verification during the workday, co-worker (non-mobile user) enrollment/verification on a supervisor's device, and a PIN fallback — all functional without connectivity, with audit events and enrollment data syncing to the server when online.

#### At a glance

* **Used by:** HCM field workers, supervisors, and non-mobile co-workers.
* **Works offline:** profiles and verification events remain available without a network.
* **Protects:** sign-in, periodic attendance checks, and co-worker verification.
* **Fallback:** workers can use a backup PIN after configured failed attempts.
* **Syncs:** profiles go to the worker registry. Audit events use HCM sync.

{% hint style="info" %}
In short: enrol once, verify offline, and sync securely when online.
{% endhint %}

## Dependencies

* **MobileFaceNet and ML Kit** create face profiles and perform liveness checks.
* **MDMS** defines match thresholds, prompt timing, and attempt limits.
* **Worker registry** stores profiles for recovery and co-worker verification.
* **Attendance service and HCM sync** receive audit events when the device reconnects.

## Business Flow

#### Enrol a worker

1. The worker captures five face angles.
2. The app checks liveness through a blink or head-turn challenge.
3. The app rejects duplicate profiles and saves the approved profile locally.
4. The worker receives a backup PIN once. The app stores only its salted hash.
5. The profile and enrollment event sync when connectivity returns.

Supervisors can enrol non-mobile co-workers from **Non-mobile users**.

#### Verify a worker

1. After HCM authentication, the app captures and compares the worker's face.
2. It compares only with the signed-in worker's profile.
3. A match opens the app and records a success event.
4. Failed attempts lead to backup PIN verification, then HCM password verification.

The app also schedules periodic attendance prompts. Team and district supervisors are exempt. Co-workers verify after the system user verifies.

## Configuration

MDMS loads tenant-scoped settings at app start and login.

<table><thead><tr><th>Setting</th><th width="176.2734375">Default</th><th>Purpose</th></tr></thead><tbody><tr><td><code>START_HOUR</code> / <code>END_HOUR</code></td><td><code>8</code> / <code>18</code></td><td>Daily verification window</td></tr><tr><td><code>PROMPT_COUNT</code></td><td><code>3</code></td><td>Prompts each day</td></tr><tr><td><code>MIN_GAP_MINUTES</code></td><td><code>180</code></td><td>Minimum time between prompts</td></tr><tr><td><code>COUNTDOWN_DURATION_MINUTES</code></td><td><code>5</code></td><td>Time allowed to respond</td></tr><tr><td><code>MAX_FACE_ATTEMPTS</code></td><td><code>3</code></td><td>Attempts before PIN fallback</td></tr><tr><td><code>FACE_MATCH_THRESHOLD</code></td><td><code>0.70</code></td><td>Minimum face-match score</td></tr></tbody></table>

## Failure & Privacy Handling

* **No connection:** verification continues with the locally stored profile.
* **Missing profile:** the app restores it from the registry or starts enrollment.
* **Failed upload:** updates remain queued until the device reconnects.
* **Expired prompt:** the app records a `MISSED` event.
* **Local protection:** audit events use an encrypted database and secure storage.

The app stores one-way face embeddings, not reconstructable face images. A face profile can unlock only its matching signed-in worker. Logout and user switches clear local face data.

<details>

<summary>Technical reference</summary>

### 1. Architecture and implementation details

The system is split into three layers:

* **App layer** (`health_campaign_field_worker_app`) — owns policy: when to gate, when to prompt, what to log, how to sync.
* **Package layer** (`digit_face_verification`) — self-contained ML + UI.
* **Shared layer** — data models and sync infrastructure used across the app.

#### App layer

<table><thead><tr><th width="160.46484375">Category</th><th>Components</th></tr></thead><tbody><tr><td>Pages</td><td><code>FaceGatePage</code>, <code>PinFallbackPage</code>, <code>FaceAuthHistoryPage</code>, <code>NonMobileUserListPage</code>, <code>NonMobileFaceEnrollPage</code></td></tr><tr><td>Widgets</td><td><code>ReVerificationListener</code>, reverification popup/sheet, countdown banner, <code>FaceVerificationDialog</code></td></tr><tr><td>Blocs</td><td><code>FaceGateBloc</code>, <code>ReVerificationBloc</code></td></tr><tr><td>Services</td><td><code>ReVerificationScheduler</code>, <code>FaceAuthEventLogger</code>, <code>WorkerRegistryService</code>, <code>NotificationService</code>, <code>FaceDataCleanupService</code></td></tr></tbody></table>

#### Package layer

<table><thead><tr><th width="153.55859375">Category</th><th>Components</th></tr></thead><tbody><tr><td>Views</td><td><code>FaceEnrollmentView</code>, <code>FaceVerificationView</code>, <code>FaceCaptureView</code>, <code>LivenessChallengeView</code>, <code>CustomPinPad</code></td></tr><tr><td>Blocs</td><td><code>FaceEnrollmentBloc</code>, <code>FaceVerificationBloc</code>, <code>LivenessBloc</code></td></tr><tr><td>Data</td><td><code>FaceModelService</code> (TFLite), <code>LivenessDetectionService</code>, <code>FaceEmbeddingRepository</code> (Isar), <code>PinService</code></td></tr><tr><td>Assets</td><td><code>mobilefacenet.tflite</code>, audio cues (<code>add.wav</code>, <code>buzzer.wav</code>)</td></tr></tbody></table>

#### Shared layer

* `digit_data_model` — `FaceAuthEventModel`, oplog
* `transit_post` — `FaceAuthEvent` local/remote/oplog repositories
* `sync_service` — `SyncService`, `SyncLock`

### 2. ML Pipeline

<table><thead><tr><th width="217.2265625">Item</th><th>Value</th></tr></thead><tbody><tr><td>Model</td><td>MobileFaceNet (TFLite), bundled at <code>packages/digit_face_verification/assets/models/mobilefacenet.tflite</code></td></tr><tr><td>Input</td><td>112 × 112 × 3 cropped face (JPEG-encoded crop retained for audit)</td></tr><tr><td>Output</td><td>192-dimension embedding vector</td></tr><tr><td>Similarity metric</td><td>Cosine similarity (<code>utils/distance_metrics.dart</code>)</td></tr><tr><td>Match threshold</td><td>Default <strong>0.70</strong>; overridable from MDMS (<code>FACE_MATCH_THRESHOLD</code>, currently 0.75 on demo)</td></tr><tr><td>Duplicate-enrollment threshold</td><td><strong>0.75</strong> — a new enrollment whose embedding matches an existing profile above this is rejected (<code>FaceEnrollmentBloc.duplicateThreshold</code>)</td></tr></tbody></table>

Face detection/landmarks come from ML Kit via the camera stream in `FaceCaptureView`; lighting quality is assessed (`lighting_assessment.dart`) before capture.

#### Liveness detection

`LivenessDetectionService` requires active challenges before an enrollment is accepted:

* **Blink** — eye-open probability crossing 0.7 → 0.3 → 0.7
* **Head turn** — left/right

Verification flows use the same service via `LivenessChallengeView`.

### 3. Enrollment

#### Entry points

* **System user (distributor)** — `FaceGatePage` routes to `FaceEnrollmentView` on first login after authentication (`HomePage._checkFaceEnrollment` → `FaceGateRoute`).
* **Co-worker (non-mobile user)** — sidebar → *Non mobile users* → `NonMobileFaceEnrollPage`, performed on the supervisor/distributor device.

#### Flow (`FaceEnrollmentView`)

1. Intro screen (steps + logout escape hatch).
2. Capture **5 angles**: straight, left, right, up, down.
3. **Liveness challenge** pass.
4. Duplicate check against all locally stored profiles (≥ 0.75 → rejected with `duplicateDetected`; device cap reached → `maxUsersReached`).
5. A **backup PIN** is generated and shown once. Only `SHA-256(salt:pin)` + a 16-byte random salt are stored (`PinService`) — the PIN itself is never persisted.
6. Profile persisted locally and pushed to the **worker registry**. GPS is captured for the enrollment audit event.
7. `FaceAuthEventLogger.logEnrollment` writes an `ENROLLMENT` audit event into the sync pipeline

#### Sequence diagram for Enrollment

<figure><img src="https://2077406040-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FI0KFbSBTySIQ7TCOKnJF%2Fuploads%2FU7gzyLY3NtKKJQcl8zYy%2Fenrollment-sequence.png?alt=media&amp;token=0afc9b14-c25d-4154-8f06-c9ba115bb0e4" alt=""><figcaption></figcaption></figure>

### 4. Verification Flows

#### 4.1 Login gate (`FaceGatePage` + `FaceGateBloc`)

After HCM login, the gate matches the camera face **only against the logged-in user's own embedding** (never all profiles — a co-worker's face or PIN must not unlock another user's session).

**Outcomes:**

* Face match ≥ threshold → proceed (`LOGIN / FACE_SUCCESS` event).
* Max attempts (MDMS `MAX_FACE_ATTEMPTS`, default 3) → PIN fallback (`PIN_FALLBACK`) → HCM-password fallback (`HCM_FALLBACK`).
* If the local embedding is missing (fresh install), the bloc attempts `WorkerRegistryService.syncEnrollmentFromRegistry` to restore the enrollment from the server; fails open to enrollment when offline.

#### Login gate sequence diagram

<figure><img src="https://2077406040-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FI0KFbSBTySIQ7TCOKnJF%2Fuploads%2FAfkNLyWFe8qG8WbpVtvY%2Flogin-gate-sequence.png?alt=media&amp;token=ca546577-cf61-4505-b4df-b82e8cafefa2" alt=""><figcaption></figcaption></figure>

#### 4.2 Periodic re-verification

Wired in `AuthenticatedPageWrapper` (`lib/pages/authenticated.dart`):

* **`ReVerificationScheduler`** generates `PROMPT_COUNT` random prompt times inside `START_HOUR`–`END_HOUR`, at least `MIN_GAP_MINUTES` apart. The day's schedule is **persisted in SharedPreferences** and reused for the rest of the day (a same-day MDMS timing change only takes effect after a config-change restart or the next day).
* On trigger: countdown banner (`_ReVerificationCountdownBanner`) + bottom sheet (*Verify Now / Remind Me Later*) with a `COUNTDOWN_DURATION_MINUTES` timer, and an "Attempt X of Y" AppBar counter. Expiry logs `MISSED`.
* Triggers are deferred (`markPending`) while no enrollment exists, on the FaceGate/enrollment routes, or when the app is backgrounded — a local **notification** fires instead; tapping it (even cold-start) re-dispatches the trigger (`_checkNotificationLaunch`, pending-tap drain).
* **Supervisors are exempt** (`TEAM_SUPERVISOR`, `DISTRICT_SUPERVISOR` roles → `face_reverify_skip`).
* Attendance-register `additionalDetails.startHour/endHour` override the MDMS window when an active register exists for today.
* App resume calls `scheduler.checkNow()` to catch missed windows.

#### Periodic re-verification sequence Diagram

> <img src="https://2077406040-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FI0KFbSBTySIQ7TCOKnJF%2Fuploads%2FpMLrKgNxANM2FAIc1KdD%2Freverification-sequence.png?alt=media&amp;token=3b4feecd-caa7-4547-8564-4fb7b9d385b2" alt="" data-size="original">

#### 4.3 Co-worker verification

When the system user verifies, pending co-workers on today's attendance register are prompted next (`coWorkerPendingNotifier`, "System user must verify their face first before co-workers can proceed"). Co-worker embeddings are **prefetched from the worker registry** after login (`_prefetchCoWorkerEmbeddings`) so verification works offline later. Events are logged with the co-worker's `individualId` and the subject name in `additionalFields`.

#### 4.3 Co-worker verification

When the system user verifies, pending co-workers on today's attendance register are prompted next (`coWorkerPendingNotifier`, "System user must verify their face first before co-workers can proceed"). Co-worker embeddings are **prefetched from the worker registry** after login (`_prefetchCoWorkerEmbeddings`) so verification works offline later. Events are logged with the co-worker's `individualId` and the subject name in `additionalFields`.<br>

### 5. Server Configuration (MDMS, tenant-scoped)

#### 5.1 Configuration

Fetched **live** (never cached) by `MdmsRepository.searchFaceAuthConfig` each time the authenticated wrapper mounts (login / app start):

| Key                          | Default (client) | Purpose                         |
| ---------------------------- | ---------------- | ------------------------------- |
| `START_HOUR` / `END_HOUR`    | 8 / 18           | Re-verification window          |
| `PROMPT_COUNT`               | 3                | Prompts per day                 |
| `MIN_GAP_MINUTES`            | 180              | Minimum spacing between prompts |
| `COUNTDOWN_DURATION_MINUTES` | 5                | Response window per prompt      |
| `MAX_FACE_ATTEMPTS`          | 3                | Attempts before PIN fallback    |
| `FACE_MATCH_THRESHOLD`       | 0.70             | Cosine-similarity cutoff        |

#### 5.2 Service registry (`HCM-SERVICE-REGISTRY.serviceRegistry`)

Required entry (present on demo under `AttendanceService`):

```json
{
  "actions": [
    { "path": "/health-attendance/face-auth/v1/_create",      "action": "create",     "entityName": "FaceAuthEvent" },
    { "path": "/health-attendance/face-auth/v1/bulk/_create", "action": "bulkCreate", "entityName": "FaceAuthEvent" },
    { "path": "/health-attendance/face-auth/v1/_search",      "action": "search",     "entityName": "FaceAuthEvent" }
  ]
}
```

`entityName` must be exactly `FaceAuthEvent` (matched by camelCase against `DataModelType.faceAuthEvent`). The app refreshes its cached registry when the MDMS **module-version** changes; without that bump a device keeps its first-install registry.

#### 5.3 Access control (`ACCESSCONTROL-ACTIONS-TEST` / role-actions)

The DIGIT gateway 401s any URI not mapped to the caller's roles. The worker-registry endpoints must be mapped for enrolling roles (e.g. `DISTRIBUTOR`):

* `/worker/v1/_search` — present on demo (action id 4124).
* `/worker/v1/bulk/_update` — **must also be mapped**; without it the embedding push fails with HTTP 401 and stays queued (known gap on demo as of 2026-07-20).

***

### 6. Local Data Model & Storage

<table><thead><tr><th width="243.05078125">Store</th><th>Contents</th></tr></thead><tbody><tr><td><strong>Isar</strong> (<code>FaceEmbeddingRepository</code>)</td><td>Per-individual profile: primary embedding, per-angle embeddings (flattened, <code>angleCount</code> × 192), model version, <code>isSystemUser</code>, <code>enrolledBy</code>, enrollment profile (PIN hash + salt, <code>enrolledAt</code>, <code>supervisorApprovalStatus</code>, <code>twinFlagged</code>)</td></tr><tr><td><strong>Encrypted SQLCipher DB</strong> (<code>LocalSqlDataStore</code>)</td><td><code>FaceAuthEvent</code> rows + oplog entries for sync</td></tr><tr><td><strong>FlutterSecureStorage</strong></td><td>Session/user data, <code>isFaceEnrollmentComplete</code>, <code>userIndividualId</code>, DB encryption key, <code>SyncLock</code></td></tr><tr><td><strong>SharedPreferences</strong></td><td>Day schedule + completed prompt indices, <code>face_reverify_skip</code>, pending worker-registry queues (<code>face_registry_sync_pending</code>, <code>face_registry_sync_pending_ids</code>), notification bookkeeping</td></tr></tbody></table>

#### `FaceAuthEventModel` (audit event) — field reference

| Field                                         | Description                                                                                            |
| --------------------------------------------- | ------------------------------------------------------------------------------------------------------ |
| `clientReferenceId`                           | UUID for the event                                                                                     |
| `individualId`                                | Subject of the event                                                                                   |
| `eventType`                                   | One of `LOGIN`, `CHECK_IN`, `RE_VERIFY`, `ENROLLMENT`                                                  |
| `outcome`                                     | One of `FACE_SUCCESS`, `FACE_REJECTED`, `PIN_FALLBACK`, `HCM_FALLBACK`, `MISSED`                       |
| `confidence`                                  | Match confidence score                                                                                 |
| `latitude` / `longitude` / `locationAccuracy` | GPS at time of event                                                                                   |
| `timestamp`                                   | Event time                                                                                             |
| `failedAttemptCount`                          | Number of failed attempts leading to this event                                                        |
| `fallbackReason`                              | Reason fallback was triggered                                                                          |
| `popupTime` / `responseTime` / `responseType` | Re-verification prompt timing; `responseType` is one of `FACE`, `PIN`, `DISMISS`, `TIMEOUT`, `ATTEMPT` |
| `anomalyFlags`                                | Flags raised during the event                                                                          |
| `faceImage`                                   | Optional base64 112×112 crop for review                                                                |
| `projectId` / `tenantId`                      | Scoping identifiers                                                                                    |
| Audit details                                 | Includes `createdBy` (system-user UUID — required for oplog pickup)                                    |
| `additionalFields`                            | App-only fields: `isSync`, `deviceId`, `boundaryCode`, `syncTimestamp`                                 |

***

### 7. Sync Architecture

#### 7.1 FaceAuthEvent audit events (oplog → sync service)

Standard HCM offline-first path: local create writes the row + an oplog entry; `SyncService.performSync` (manual **Sync Data** button, and the background service) bulk-uploads via `FaceAuthEventRemoteRepository.bulkCreate` → `POST /health-attendance/face-auth/v1/bulk/_create` (HTTP 202 = accepted by the async persister).

**Wired in:**

* `Constants.getLocalRepositories / getRemoteRepositories` (background isolate)
* `NetworkManagerProviderWrapper` (UI providers)
* `attemptSyncUp` (`utils.dart`) — manual sync repo lists
* `EntityMapper` (`sync_service_mapper.dart`) — sync count / response mapping

> ⚠️ A pending oplog type with **no** matching remote repository throws `Remote repository is not passed to sync service` and blocks the whole sync — this is what a missing/stale service-registry entry looks like.

#### 7.2 Worker registry (enrollment data)

Direct calls (not oplog-based), handled by `WorkerRegistryService`:

* **Push** after enrollment: `_search` worker by `individualId`, then `POST /worker/v1/bulk/_update` with face fields in `additionalDetails.fields` (embedding, angle embeddings, model version, PIN hash+salt, enrolledBy, approval status…).
* **Pull**: `syncEnrollmentFromRegistry` restores a profile from the server (login-gate restore; co-worker prefetch).
* **Offline queues**: a failed push leaves the ID queued —
  * `face_registry_sync_pending` (self-enrollment, `FaceGatePage`)
  * `face_registry_sync_pending_ids` (co-workers)
  * Drained by `_retryPendingWorkerRegistrySync` in `AuthenticatedPageWrapper` on mount and on every offline→online transition.
  * `updateWorkerWithFaceEnrollment` returns `true` for success *or* terminal states (dequeue), `false` only for retryable network failures.

> #### Data sync sequence Diagram

<figure><img src="https://2077406040-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FI0KFbSBTySIQ7TCOKnJF%2Fuploads%2FBiCexFc1uOP6g392sKZx%2Fsync-sequence.png?alt=media&amp;token=0f6a800a-dd31-4a2f-bcbf-6ee8ef5f377e" alt=""><figcaption></figcaption></figure>

### 8. Localisation

All UI codes are `FACE_AUTH_*` (153 codes) under module **`hcm-common`** for every locale (`en_DEMO`, `pt_DEMO`, `fr_DEMO` on demo). `hcm-common` is in `Constants.initialLocalizationModules`, so codes are cached at boot for the selected locale.

The package's `FaceVerificationLocalization.translate` resolves from its own delegate rows first, then falls back to the **app-level `AppLocalizations` resolver** via the static `appTranslate` hook (set in `getAppLocalizationDelegates`) — this keeps package screens localised even when the package delegate's one-shot row snapshot is stale.

***

### 9. Security & Privacy

* Raw face images are **not** stored as a gallery; only the 112×112 crop optionally rides along on audit events (`faceImage`) for review.
* Embeddings are one-way features; the model never reconstructs faces.
* PIN: salted SHA-256 only; shown once at enrollment.
* Event rows live in the SQLCipher-encrypted DB; keys in secure storage.
* Login gate matches strictly against the logged-in user's profile — co-worker embeddings on the same device cannot unlock a session.
* SSL pinning enforced by `DioClient`; bad certificates always rejected.
* `FaceDataCleanupService` handles local face-data removal (logout / user-switch hygiene).

***

***

</details>


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.digit.org/health/design/architecture/low-level-design/services/platform-services/facial-recognition.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.