Excel Ingestion Configuration

Advanced Configurations

Overview

The Excel Ingestion service is a core component of the HCM (Health Campaign Management) platform. Its primary responsibility is to:

• Generate Excel templates with validations and localisation

• Ingest uploaded Excel files

• Validate, transform, and process bulk data

• Interact asynchronously with downstream services

The service supports ingestion for campaigns, boundaries, facilities, individuals, users, and related domain data.

Dependencies

The Excel Ingestion service depends on the following external services and infrastructure components:

• PostgreSQL – persistent storage and migration tracking

• Kafka – event-based asynchronous processing

• MDMS – master data lookup

• File Store – upload and download of Excel files

• Boundary, Facility, Individual, Campaign services – domain validations

• LocalisationLocalization service – multilingual support

Configuration is managed using Helm charts and deployed on Kubernetes via a values.yaml file.

Steps

Helm Configuration

The Excel Ingestion service depends on Kafka and Postgres. You need to set environment variables in the values.yaml file or Helm templates.

DIGIT-DevOps/deploy-as-code/helm/charts/health-services/excel-ingestion/values.yaml at unified-env-lts · egovernments/DIGIT-DevOpsGitHub

Excel Ingestion Helm

Common Labels

labels:
  app: "excel-ingestion"
  group: "health"

• Used for Kubernetes resource identification and grouping.

Namespace Configuration

namespace: health

• Deploys the service in the health namespace.

• Database schema and DB URL selection depend on this value.

Ingress Configuration

ingress:
  namespace: egov
  enabled: true
  zuul: true
  context: "excel-ingestion"

• Exposes the service through Zuul API Gateway

• Base context path: /excel-ingestion

Init Containers – Database Migration

initContainers:
  dbMigration:
    enabled: true
    schemaTable: "excel_ingestion_flyway_history"
    image:
      repository: "excel-ingestion-db"

• Executes Flyway database migrations before the application starts

• Tracks migration history in excel_ingestion_flyway_history

Application Container Configuration

image:
  repository: "excel-ingestion"
replicas: "1"

• Runs a single replica by default

• Can be scaled based on ingestion load

Health Checks

healthChecks:
  enabled: true
  livenessProbePath: "/excel-ingestion/health"
  readinessProbePath: "/excel-ingestion/health"

• Ensures Kubernetes can detect unhealthy pods

• Uses Spring Boot actuator-style health endpoint

Application Runtime Configuration

appType: "java-spring"
tracing-enabled: true
memory_limits: "3072Mi"
cpu_limits: "500m"
heap: "-Xmx2900m -Xms256m"
java-args: ""

• Java Spring Boot application

• Tracing enabled for distributed request tracking

• Heap size tuned to avoid OOM during large Excel processing

Debug Configuration

java-enable-debug: false
java-debug-port: 5005

• Enables remote debugging when required

• Disabled by default in production

Server Configuration

- name: SERVER_PORT
  value: "8080"
- name: SERVER_SERVLET_CONTEXT_PATH
  value: "/excel-ingestion"

• Application runs on port 8080

• Base context path: /excel-ingestion

Localisation Configuration

- name: DEFAULT_LOCALE
  value: "en_IN"

• Default locale used for:

  • Template generation

  • Validation messages

  • Error responses

External Service Host Configuration

All service hosts are resolved from the egov-service-host ConfigMap.

Environment Variable	ConfigMap Key
EGOV_BOUNDARY_HOST	boundary-service
EGOV_FILESTORE_HOST	egov-filestore
EGOV_CAMPAIGN_HOST	project-factory
EGOV_HEALTH_INDIVIDUAL_HOST	health-individual
EGOV_FACILITY_HOST	facility
EGOV_LOCALIZATION_HOST	egov-localization
EGOV_MDMS_HOST	egov-mdms-service

Service Endpoints

Purpose	Endpoint
Campaign Search	project-factory/v1/data/campaign/_search
Campaign Bulk Decrypt	project-factory/v1/crypto/_bulkDecrypt
Individual Search	health-individual/v1/_search
Hierarchy Search	boundary-service/boundary-hierarchy-definition/_search
Boundary Relationship Search	boundary-service/boundary-relationships/_search
Facility Search	facility/v1/_search
File Upload	filestore/v1/files
File URL Fetch	filestore/v1/files/url
Localization Search	localization/messages/v1/_search
MDMS Search	egov-mdms-service/v2/_search

Database Configuration

Database connection values are dynamically selected based on the namespace.

Database Connection Details

Parameter	Source	Description
SPRING_DATASOURCE_URL	health-db-url or db-url	Namespace-based DB URL
DB_HOST	egov-config	Database host
DB_PORT	5432	PostgreSQL default port
DB_NAME	egov-config	Database name
DB_SCHEMA	Namespace or public	Schema selection
DB_USER	db secret	Database username
DB_PASSWORD	db secret	Database password

JPA / Hibernate Configuration

- name: SPRING_JPA_HIBERNATE_DDL_AUTO
  value: "update"
- name: SPRING_JPA_PROPERTIES_HIBERNATE_DIALECT
  value: "org.hibernate.dialect.PostgreSQLDialect"

• Automatically updates schema

• Uses PostgreSQL dialect

Flyway Migration Configuration

Parameter	Source
FLYWAY_USER	db secret
FLYWAY_PASSWORD	db secret
FLYWAY_LOCATIONS	egov-config
SCHEMA_TABLE	excel_ingestion_flyway_history

Kafka Producer Configuration

- name: SPRING_KAFKA_PRODUCER_KEY_SERIALIZER
  value: org.apache.kafka.common.serialization.StringSerializer
- name: SPRING_KAFKA_PRODUCER_VALUE_SERIALIZER
  value: org.springframework.kafka.support.serializer.JsonSerializer

• Enables event publishing for async processing

Redis Health Check Configuration

- name: MANAGEMENT_HEALTH_REDIS_ENABLED
  value: "false"

• Redis health check is disabled as Redis is not mandatory for Excel Ingestion

Notes

• Secrets vs ConfigMaps

  • Sensitive data (DB credentials, Flyway credentials) is stored in Kubernetes Secrets

  • Non-sensitive values are stored in ConfigMaps

• Namespace-Aware Behaviour

  • Database URL and schema automatically change based on the namespace

  • Ensures isolation between environments

• Performance Considerations

  • Heap and memory limits are tuned for large Excel uploads

  • Increase replicas if the concurrent ingestion volume increases