End-to-End Process Flow

The system supports an Excel-based workflow for various health campaign types (e.g., boundary, user, facility). It provides a dynamic, config-driven approach for template generation, data validation, and data processing.

🔁 Overall Flow:

Generate Template → Fill Data → Upload & Validate → Process Data

1. 📄 Template Generation Flow (generateFlowClasses)

The template generation process creates structured Excel templates with localized headers, metadata, and placeholder content based on the campaign type.

A. Configuration – generationTemplateConfigs.ts

Defines the sheet structure and metadata for each campaign type:

boundary: {
sheets: [
{
sheetName: "HCM_README_SHEETNAME",
schemaName: "target-readme",
lockWholeSheet: true
}
]
}

B. Generate Classes – e.g., boundary-generateClass.ts

Purpose: Dynamically generate campaign-specific Excel templates.

Key Method: TemplateClass.generate()

Steps:

1. Fetch campaign details from DB

2. Load boundary hierarchy and relationships

3. Localize boundary data and headers

4. Generate readme and metadata sheets

5. Create SheetMap with structured data

6. Return data to generate Excel

C. Template Generation Process – sheetManageUtils.ts

export async function generateResource(responseToSend: any, templateConfig: any) {
// 1. Get localization maps
// 2. Load generate class dynamically
// 3. Generate SheetMap using TemplateClass.generate()
// 4. Create Excel file with localized structure
// 5. Upload to file store
// 6. Send status/event to Kafka
}

2. 🧩 Data Processing Flow (processFlowClasses)

Handles the uploaded Excel file, validates it, and processes the data to create or update entities in the system.

A. Configuration – processTemplateConfigs.ts

Defines how to process uploaded Excel files:

boundary: {
sheets: [
{
sheetName: "HCM_README_SHEETNAME",
lockWholeSheet: true
}
],
enrichmentFunction: "enrichTargetProcessConfig"
}

B. Process Classes – e.g., boundary-processClass.ts

Purpose: Validate and persist uploaded data.

Key Method: TemplateClass.process()

Steps:

1. Validate resource details (file, campaign)

2. Extract data from uploaded sheets

3. Enrich data using boundary hierarchy

4. Create/update boundary records

5. Generate mapping and project data

6. Process in topological (dependency-resolved) order

C. Validation Classes – e.g., boundaryValidation-processClass.ts

Purpose: Perform schema-based and business-rule validations.

Key Method: TemplateClass.process()

Steps:

1. Validate against schema (column names, types)

2. Check required vs optional columns

3. Validate logical integrity (parent-child)

4. Mark errors in Excel (cell comments, color)

5. Return annotated Excel for user corrections

D. Data Processing Utility – sheetManageUtils.ts

export async function processResource(ResourceDetails: any, templateConfig: any) {
// 1. Download Excel file from file store
// 2. Extract locale and sheet structure
// 3. Load localization maps
// 4. Dynamically load process/validation class
// 5. Perform validation and/or processing
// 6. Annotate sheet with validation results
// 7. Upload processed file
// 8. Send status/event to Kafka
}

3. 🔄 Complete End-to-End Flow

✅ Step 1: Template Generation

• Trigger: User requests a template for a specific campaign type (e.g., boundary, user, facility)

• Process:

  1. Load configuration from generationTemplateConfigs

  2. Fetch campaign data and related hierarchy

  3. Dynamically import *-generateClass.ts

  4. Call TemplateClass.generate() to generate data

  5. Generate Excel file with formatting, validation, and localization

  6. Upload to file store

• Response: Returns fileStoreId and generation status

✅ Step 2: Data Upload & Processing

• Upload: User uploads the filled Excel template

🔍 Validation Phase (Optional)

• Load validation class (e.g., boundaryValidation-processClass)

• Validate against schema and business rules

• Annotate errors directly in Excel file

• Return Excel for correction

⚙️ Processing Phase

• Load processing config and class (e.g., boundary-processClass)

• Call TemplateClass.process() to:

  • Extract and validate data

  • Enrich using hierarchy

  • Persist via Kafka (batch insert/update)

• Response: Returns processed Excel and summary

4. 🧠 Key Features

🔧 Dynamic Class Loading

const className = `${type}-generateClass`; // or `${type}-processClass`

const classFilePath = path.join(__dirname, '..', 'generateFlowClasses', `${className}.ts`);

const { TemplateClass } = await import(classFilePath);

• Plug-and-play support for new types (boundary, user, etc.)

• Reduces code duplication

🌐 Localization Support

• Detects locale from Excel metadata

• Supports multilingual sheet names and headers

• Error messages and field names localized dynamically

⚡ Kafka Integration

• Asynchronous, scalable data ingestion

• Sends status, logs, and errors to Kafka

• Enables reliable batch processing

✅ Validation Framework

• Schema-driven validation

• Business rule enforcement

• Marks errors with cell comments and styles

• Supports required vs optional fields

🏗️ Hierarchical Data Processing

• Handles parent-child boundary relationships

• Uses topological sorting to maintain dependency order

• Supports complex graph-based boundary structures

5. 📦 Supported Sheet types

Type	Description
boundary	Geographic boundary and hierarchy
user	User onboarding and management
facility	Healthcare facility setup
userCredential	Authentication data for users

Each type has:

• A generate class for template creation

• A process class for data ingestion

• A validation class for pre-checking data