> For the complete documentation index, see [llms.txt](https://docs.digit.org/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.digit.org/health/v2.0/design/architecture/low-level-design/services/console-services/project-factory-campaign-manager/manage-data-apis/end-to-end-process-flow.md).
# End-to-End Process Flow
The system supports an Excel-based workflow for various health campaign types (e.g., boundary, user, facility). It provides a dynamic, config-driven approach for **template generation**, **data validation**, and **data processing**.
### **🔁 Overall Flow:**
**Generate Template → Fill Data → Upload & Validate → Process Data**
### **1. 📄 Template Generation Flow (generateFlowClasses)**
The template generation process creates structured Excel templates with localized headers, metadata, and placeholder content based on the campaign type.
#### **A. Configuration – generationTemplateConfigs.ts**
Defines the sheet structure and metadata for each campaign type:
```json
boundary: {
sheets: [
{
sheetName: "HCM_README_SHEETNAME",
schemaName: "target-readme",
lockWholeSheet: true
}
]
}
```
#### **B. Generate Classes – e.g., boundary-generateClass.ts**
**Purpose**: Dynamically generate campaign-specific Excel templates.
**Key Method**: TemplateClass.generate()
**Steps:**
1. Fetch campaign details from DB
2. Load boundary hierarchy and relationships
3. Localize boundary data and headers
4. Generate readme and metadata sheets
5. Create SheetMap with structured data
6. Return data to generate Excel
#### **C. Template Generation Process – sheetManageUtils.ts**
```
export async function generateResource(responseToSend: any, templateConfig: any) {
// 1. Get localization maps
// 2. Load generate class dynamically
// 3. Generate SheetMap using TemplateClass.generate()
// 4. Create Excel file with localized structure
// 5. Upload to file store
// 6. Send status/event to Kafka
}
```
### **2. 🧩 Data Processing Flow (processFlowClasses)**
Handles the uploaded Excel file, validates it, and processes the data to create or update entities in the system.
#### **A. Configuration – processTemplateConfigs.ts**
Defines how to process uploaded Excel files:
```
boundary: {
sheets: [
{
sheetName: "HCM_README_SHEETNAME",
lockWholeSheet: true
}
],
enrichmentFunction: "enrichTargetProcessConfig"
}
```
#### **B. Process Classes – e.g., boundary-processClass.ts**
**Purpose**: Validate and persist uploaded data.
**Key Method**: TemplateClass.process()
**Steps:**
1. Validate resource details (file, campaign)
2. Extract data from uploaded sheets
3. Enrich data using boundary hierarchy
4. Create/update boundary records
5. Generate mapping and project data
6. Process in topological (dependency-resolved) order
#### **C. Validation Classes – e.g., boundaryValidation-processClass.ts**
**Purpose**: Perform schema-based and business-rule validations.
**Key Method**: TemplateClass.process()
**Steps:**
1. Validate against schema (column names, types)
2. Check required vs optional columns
3. Validate logical integrity (parent-child)
4. Mark errors in Excel (cell comments, color)
5. Return annotated Excel for user corrections
#### **D. Data Processing Utility – sheetManageUtils.ts**
```
export async function processResource(ResourceDetails: any, templateConfig: any) {
// 1. Download Excel file from file store
// 2. Extract locale and sheet structure
// 3. Load localization maps
// 4. Dynamically load process/validation class
// 5. Perform validation and/or processing
// 6. Annotate sheet with validation results
// 7. Upload processed file
// 8. Send status/event to Kafka
}
```
### **3. 🔄 Complete End-to-End Flow**
#### **✅ Step 1: Template Generation**
* **Trigger**: User requests a template for a specific campaign type (e.g., boundary, user, facility)
* **Process**:
1. Load configuration from generationTemplateConfigs
2. Fetch campaign data and related hierarchy
3. Dynamically import \*-generateClass.ts
4. Call TemplateClass.generate() to generate data
5. Generate Excel file with formatting, validation, and localization
6. Upload to file store
* **Response**: Returns fileStoreId and generation status
#### **✅ Step 2: Data Upload & Processing**
* **Upload**: User uploads the filled Excel template
**🔍 Validation Phase (Optional)**
* Load validation class (e.g., boundaryValidation-processClass)
* Validate against schema and business rules
* Annotate errors directly in Excel file
* Return Excel for correction
**⚙️ Processing Phase**
* Load processing config and class (e.g., boundary-processClass)
* Call TemplateClass.process() to:
* Extract and validate data
* Enrich using hierarchy
* Persist via Kafka (batch insert/update)
* **Response**: Returns processed Excel and summary
### **4. 🧠 Key Features**
#### **🔧 Dynamic Class Loading**
const className = \`${type}-generateClass\`; // or \`${type}-processClass\`
const classFilePath = path.join(\_\_dirname, '..', 'generateFlowClasses', \`${className}.ts\`);
const { TemplateClass } = await import(classFilePath);
* Plug-and-play support for new types (boundary, user, etc.)
* Reduces code duplication
#### **🌐 Localization Support**
* Detects locale from Excel metadata
* Supports multilingual sheet names and headers
* Error messages and field names localized dynamically
#### **⚡ Kafka Integration**
* Asynchronous, scalable data ingestion
* Sends status, logs, and errors to Kafka
* Enables reliable batch processing
#### **✅ Validation Framework**
* Schema-driven validation
* Business rule enforcement
* Marks errors with cell comments and styles
* Supports required vs optional fields
#### **🏗️ Hierarchical Data Processing**
* Handles parent-child boundary relationships
* Uses topological sorting to maintain dependency order
* Supports complex graph-based boundary structures
### **5. 📦 Supported Sheet types**
| **Type** | **Description** |
| -------------- | --------------------------------- |
| boundary | Geographic boundary and hierarchy |
| user | User onboarding and management |
| facility | Healthcare facility setup |
| userCredential | Authentication data for users |
Each type has:
* A **generate class** for template creation
* A **process class** for data ingestion
* A **validation class** for pre-checking data
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:
```
GET https://docs.digit.org/health/v2.0/design/architecture/low-level-design/services/console-services/project-factory-campaign-manager/manage-data-apis/end-to-end-process-flow.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.