> For the complete documentation index, see [llms.txt](https://docs.digit.org/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.digit.org/platform/deploy/operations-guide/manage-deployment.md).

# Manage Deployment

DIGIT uses Helm as its primary packaging and deployment tool, with all environment configuration stored declaratively in the DIGIT-DevOps Git repository. This GitOps approach means every deployment is version-controlled, reproducible, and auditable. Services are deployed via either Jenkins automation pipelines or direct Helm/Helmfile commands.

## Deployment Approach

<table><thead><tr><th width="169.80078125">LAYER</th><th>TOOL / APPROACH</th></tr></thead><tbody><tr><td>Packaging</td><td>Helm charts. Every DIGIT service is packaged as a Helm chart. Charts consist of templates (YAML manifests) and a values.yaml file containing configurable defaults.</td></tr><tr><td>Environment config</td><td>Environment-specific values are stored in deploy-as-code/helm/environments/&#x3C;env>.yaml inside DIGIT-DevOps. Secrets are encrypted with SOPS before committing.</td></tr><tr><td>Orchestration</td><td>Helmfile. Manages multiple Helm releases declaratively. Supports diff, apply, sync, and destroy commands. Uses helm-diff and helm-secrets plugins.</td></tr><tr><td>CI/CD</td><td>Jenkins (self-hosted) or managed DevOps (AWS CodeDeploy, Azure DevOps, NIC DevOps). Jenkins pipelines trigger deployments from the DIGIT-DevOps repo.</td></tr><tr><td>Source of truth</td><td>Git (DIGIT-DevOps repository). No live environment changes without a corresponding Git commit. Drift is prevented by re-applying the Git state.</td></tr></tbody></table>

## Helmfile Deployment Workflow

Clone the DIGIT-DevOps repository and switch to the LTS branch before running any deploy commands:

```
git clone https://github.com/egovernments/DIGIT-DevOps.git
cd DIGIT-DevOps
git checkout DIGIT-2.9LTS(Helmfile)
cd deploy-as-code/charts/monitoring
```

Install the required Helm plugins before the first deployment:

```
# Package manager for Kubernetes
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

# Declarative Helm release manager
wget https://github.com/helmfile/helmfile/releases/latest/download/helmfile_linux_amd64 \
  -O /usr/local/bin/helmfile && chmod +x /usr/local/bin/helmfile

# Shows diff between current and proposed state
helm plugin install https://github.com/databus23/helm-diff

# Decrypts SOPS-encrypted secrets automatically during deploy
helm plugin install https://github.com/jkroepke/helm-secrets

# Verify plugins
helm plugin list
```

Standard deployment sequence for any environment:

<table><thead><tr><th width="129.46875">STEP</th><th width="218.05859375">COMMAND</th><th>PURPOSE</th></tr></thead><tbody><tr><td>1. Preview</td><td>helmfile -e &#x3C;env> -f helmfile.yaml template</td><td>Renders Kubernetes manifests without applying. Review before committing.</td></tr><tr><td>2. Diff</td><td>helmfile -e &#x3C;env> -f helmfile.yaml diff</td><td>Shows exactly what will change in the cluster vs current state.</td></tr><tr><td>3. Apply</td><td>helmfile -e &#x3C;env> -f helmfile.yaml apply</td><td>Applies changes. Prompts for confirmation when using --interactive flag.</td></tr><tr><td>4. Sync</td><td>helmfile -e &#x3C;env> -f helmfile.yaml sync</td><td>Forces full sync. Use when diff reports drift between Git state and cluster.</td></tr><tr><td>5. Destroy</td><td>helmfile --interactive destroy</td><td>Removes all releases defined in the helmfile. Requests confirmation before deleting.</td></tr></tbody></table>

## Jenkins CI/CD Pipeline

Jenkins provides a GUI-driven deployment pipeline as an alternative to direct Helmfile commands. It is suited for teams that require approval gates, audit logs, and role-based deployment access.

<table><thead><tr><th width="216.23046875">COMPONENT</th><th>DETAIL</th></tr></thead><tbody><tr><td>Jenkins Master VM</td><td>4 vCore, 8 GB RAM. Manages job scheduling, user authentication, and build history.</td></tr><tr><td>Jenkins Slave VM</td><td>4 vCore, 8 GB RAM. Executes deployment jobs. Communicates with the Kubernetes cluster via kubeconfig.</td></tr><tr><td>Deployment jobs</td><td>Environment-specific jobs (e.g. deploy-to-dev, deploy-to-uat, deploy-to-prod) defined in seed-deployment-jobs in jenkins/values.yaml.</td></tr><tr><td>Build trigger</td><td>Paste a Docker image ID into the build job and click Build. Jenkins runs the Helm deployer against the target environment.</td></tr><tr><td>Service deployment</td><td>go run main.go deploy -e &#x3C;env> -c '&#x3C;service-list>' — Golang deployer script used to deploy one or more named services to a specific environment.</td></tr></tbody></table>

To deploy specific services using the Go deployer (alternative to full Helmfile sync):

```
# Deploy prometheus-operator, grafana, and kafka-exporter to dev
go run main.go deploy -e dev -c 'prometheus-operator,grafana,prometheus-kafka-exporter'

# Redeploy nginx-ingress after enabling serviceMonitor
go run main.go deploy -e qa -c 'nginx-ingress'
```

## Rollback & Version Management

Helm maintains a revision history for every release. If a deployment introduces a regression, roll back to the previous stable revision without redeploying from scratch.

```
# List all releases in a namespace
helm list -n egov

# View revision history for a specific release
helm history <release-name> -n egov

# Roll back to the previous revision
helm rollback <release-name> -n egov

# Roll back to a specific revision number
helm rollback <release-name> <revision-number> -n egov
```

{% hint style="info" %}
**ROLLBACK NOTE:** Helmfile's improved rollback capability is one of the primary reasons DIGIT recommends Helmfile over standalone Helm for production deployments. Always run helmfile diff before applying to confirm the scope of changes.
{% endhint %}

## Pre & Post Deployment Checklist

<table><thead><tr><th width="151.375">WHEN</th><th>CHECK</th></tr></thead><tbody><tr><td>Pre-deploy</td><td>Confirm environment config (env.yaml) is committed and SOPS secrets are encrypted</td></tr><tr><td>Pre-deploy</td><td>Run helmfile diff and review all changes — no surprises in apply</td></tr><tr><td>Pre-deploy</td><td>Verify all pods in the target namespace are Running or Completed (no pending/crash-loop pods)</td></tr><tr><td>Pre-deploy</td><td>Confirm database backup has been taken before any schema-changing deployment</td></tr><tr><td>Pre-deploy</td><td>Notify stakeholders of planned maintenance window for Prod deployments</td></tr><tr><td>Post-deploy</td><td>Verify citizen portal is available at &#x3C;hostname>/digit-ui/citizen</td></tr><tr><td>Post-deploy</td><td>Verify employee portal is available at &#x3C;hostname>/digit-ui/employee</td></tr><tr><td>Post-deploy</td><td>Check pod status: kubectl get pods -n egov — all should be Running</td></tr><tr><td>Post-deploy</td><td>Run a smoke test API call against the API gateway to confirm routing</td></tr><tr><td>Post-deploy</td><td>Check Grafana dashboards for any spike in error rates or latency following deployment</td></tr><tr><td>Post-deploy</td><td>Confirm Alertmanager has not fired any critical alerts within 10 minutes of deployment</td></tr></tbody></table>

## GitOps & Environment Configuration

All environment configuration lives in DIGIT-DevOps as code. Configuration changes follow the same Git workflow as application changes: branch, commit, review, merge, deploy.

<table><thead><tr><th width="234.015625">CONFIG FILE</th><th>PURPOSE</th></tr></thead><tbody><tr><td>environments/&#x3C;env>.yaml</td><td>Environment-specific service values: db-host, ingress domain, service toggles, replica counts, resource limits.</td></tr><tr><td>environments/&#x3C;env>-secrets.yaml</td><td>Encrypted secrets: DB passwords, Alertmanager credentials, Git private key, payment gateway keys. Encrypted with SOPS before commit.</td></tr><tr><td>charts/monitoring/monitoring-helmfile.yaml</td><td>Declares all monitoring stack releases (Prometheus, Grafana, Loki, Alertmanager) and their value overrides.</td></tr><tr><td>charts/backbone-services/</td><td>Kafka, ZooKeeper, PostgreSQL, ElasticSearch, Redis Helm charts and their values.</td></tr></tbody></table>

To update Loki to use S3 storage on AWS, modify the environment YAML and reapply:

```
# In deploy-as-code/helm/environments/<env>.yaml
loki:
  persistence:
    enabled: true
  serviceAccount:
    annotations:
      eks.amazonaws.com/role-arn: <your-s3-role-arn>
  additionalConfigs:
    storage_config:
      aws:
        s3: s3://<region>/<bucket-name>
    compactor:
      retention_enabled: true
      compaction_interval: 168h   # 7 days
    table_manager:
      retention_deletes_enabled: true
      retention_period: 168h
```


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.digit.org/platform/deploy/operations-guide/manage-deployment.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
