Overview

The image below illustrates the multiple components deployed. These include the EKS, Worker Nodes, Postgres DB, EBS Volumes, and Load Balancer.

Steps

1. Clone the DIGIT-DevOps repository:

git clone https://github.com/egovernments/DIGIT-DevOps.git

1. Navigate to the cloned repository and checkout the release-1.28-Kubernetes branch:

cd DIGIT-DevOps 
git checkout release-1.28-kubernetes

1. Check if the correct credentials are configured using the command below. Refer to the attached doc to setup AWS Account on the local machine.

aws configure list

Make sure that the above command reflects the set AWS credentials. Proceed once the details are confirmed. (If the credentials are not set follow Step 2 Setup AWS account )

1. Choose either method below to generate SSH key pairs

   • a. Use an online website (not recommended in a production setup. To be only used for demo setups): https://8gwifi.org/sshfunctions.jsp

   • b. Use openssl:

openssl genpkey -algorithm RSA -out private_key.pem openssl rsa -pubout -in private_key.pem -out public_key.pem

1. Add the public key to your GitHub account.

2. Open input.yaml file in vscode. Use the below code to open it in VS code:

   code infra-as-code/terraform/sample-aws/input.yaml

   If the command does not work, open the file in VS code manually. Once the file is open, fill in the inputs. (If you are not using vscode, open it in any editor of your choice).

3. Fill in the inputs as per the regex mentioned in the comments.

4. Go to infra-as-code/terraform/sample-aws and run init.go script to enrich different files based on input.yaml.

cd infra-as-code/terraform/sample-aws 
go run ../scripts/init.go

Terraform Execution: Infrastructure Resources Provisioning

Once we are complete declaring the resources, begin with deploying all resources.

Run the terraform scripts to provision infra required to Deploy DIGIT on AWS.

1. CD (change directory) to the following directory and run the below commands to create the remote state.

### Create the remote-state first, remember that the state name should be unique
### You may need to use sudo incase you don't have requisite permission

cd remote-state

terraform init

terraform plan

terraform apply

Once the remote state is created, it is time to provision DIGIT infra. Run the below commands:

### Once the remote state is created, you can create the DIGIT Infra

cd ..

terraform init

terraform plan

terraform apply

Important:

• DB password is asked for in the application stage. Remember the password you have provided. It should be at least 8 characters long. Otherwise, RDS provisioning will fail.

• The output of the apply command will be displayed on the console. Store this in a file somewhere. Values from this file will be used in the next step of deployment.

2. Use this link to get the kubeconfig from EKS for the cluster. The region code is the default region provided in the availability zones in variables.tf. For example - ap-south-1. EKS cluster name also should've been filled in variables.tf.

aws sts get-caller-identity

# Run the below command and give the respective region-code and the cluster name
aws eks --region <region-code> update-kubeconfig --name <cluster_name>

3. Verify that you can connect to the cluster by running the following command

kubectl config use-context <cluster_name>

kubectl get nodes

NAME                                             STATUS AGE   VERSION               OS-Image           
ip-192-168-xx-1.ap-south-1.compute.internal   Ready  45d   v1.15.10-eks-bac369   Amazon Linux 2   
ip-192-168-xx-2.ap-south-1.compute.internal   Ready  45d   v1.15.10-eks-bac369   Amazon Linux 2   
ip-192-168-xx-3.ap-south-1.compute.internal   Ready  45d   v1.15.10-eks-bac369   Amazon Linux 2   
ip-192-168-xx-4.ap-south-1.compute.internal   Ready  45d   v1.15.10-eks-bac369   Amazon Linux 2 

At this point, your basic infra has been provisioned.

Note: Refer to the DIGIT deployment documentation to deploy DIGIT services.

Destroying Infra

To destroy the previously created infrastructure with Terraform, run the command below:

1. ELB is not deployed via Terraform. ELB was created at deployment time by the setup of Kubernetes Ingress. This has to be deleted manually by deleting the ingress service.

   • kubectl delete deployment nginx-ingress-controller -n <namespace>

   • kubectl delete svc nginx-ingress-controller -n <namespace>

     Note: Namespace can be either egov or jenkins.

2. Delete S3 buckets manually from the AWS console and verify if ELB got deleted.

   • In case of if ELB is not deleted, you need to delete ELB from the AWS console.

3. Run terraform destroy.

Topics Covered:

Basics

DIGIT can be deployed on a public cloud like AWS, Azure or a private cloud.

Pre-reads

• Learn the basics of Kubernetes:

• Learn the commands

Choose Your Cloud

Choose your cloud and follow the instructions to set up a Kubernetes cluster before deploying.

This guide takes you through a step by step process for installing DIGIT Core. Installation consists of provisioning and configuring the hardware and network, installing the backbone services like PostgresDB, ElasticSearch, Apache Kakfa etc and DIGIT core services i.e. API Gateway, User Service etc. We also install a sample product built on top of the DIGIT core called Public Grievance Redressal (PGR). DIGIT Core can be installed on public cloud like AWS, Azure or on Private Cloud. Installing DIGIT on personal computers is not recommended.

Follow the steps outlined in the below below sections to install DIGIT.

Topics covered:

Overview

Follow the details below to set up your AWS account before you proceed with the DIGIT deployment.

Get AWS Access

Once the command line access is configured, everything is set to proceed with the terraform to provision the DIGIT Infra-as-code.

Topics covered:

Overview

is an AWS service for deploying, managing, and scaling distributed and containerized workloads. With EKS, you can easily provision a cluster on AWS using , which automates the process. Then, deploy the DIGIT services configuration using .

Pre-reads

• Know about EKS:

• Know what is terraform:

Installation Steps

Topics covered:

• ​

Pre-requisites

1. ​ with admin access to provision infrastructure. You will need a paid subscription to the AWS.

2. Install (any version) on the local machine - it helps interact with the Kubernetes cluster.

3. Install - this helps package the services, configurations, environments, secrets, etc into . Verify that the installed version of helm is equal to 3.0 or higher.

1. Install - to run different versions of Terraform on the machine. supports multiple terraform versions on the same machine and you can toggle between the desired versions.

5. Run tfswitch and it will show a list of terraform versions. Scroll down and select version (0.14.10) for the (IaC) to provision cloud resources as code. This provides the desired resource graph and helps destroy the cluster in one go.​

If you have any questions please write to us.

Make sure to use the appropriate discussion category and labels to address the issues better.

Discussion Board

Annexures:

• s

Topics covered:

Overview

The Azure Kubernetes Service (AKS) is one of the Azure services for deploying, managing and scaling any distributed and containerized workloads, here we can provision the AKS cluster on Azure from the ground up using (infra-as-code) and then deploy the DIGIT platform services as config-as-code using .

Pre-reads

• Know about AKS: ​

• Know what is terraform:

Pre-requisites​

1. Azure subscription: If you don't have an Azure subscription, create a before you begin.

2. Install Azure

3. Configure Terraform: Follow the directions in the article,

4. Azure service principal: Follow the directions in the Create the service principal section in the article, . Take note of the values for the appId, displayName, password, and tenant.

5. Install on your local machine which helps you interact with the Kubernetes cluster.

6. Install that helps you package the services along with the configurations, environments, secrets, etc into

Topics covered:

Overview

Running Kubernetes on-premise gives a cloud-native experience on SDC when it comes to deploying DIGIT.

Whether States have their own on-premise data centre or have decided to forego the various managed cloud solutions, there are a few things one should know when getting started with on-premise K8s.

One should be familiar with Kubernetes and the consists of the Kube-apiserver, Kube-scheduler, Kube-controller-manager and an ETCD datastore. For managed cloud solutions like or , it also includes the cloud-controller-manager. This is the component that connects the cluster to external cloud services to provide networking, storage, authentication, and other support features.

To successfully deploy a bespoke Kubernetes cluster and achieve a cloud-like experience on SDC, one needs to replicate all the same features you get with a managed solution. At a high level, this means that we probably want to:

• Automate the deployment process

• Choose a networking solution

• Choose a right storage solution

• Handle security and authentication

The subsequent sections look at each of these challenges individually, and provide enough of a context required to help in getting started.

Automating The Deployment Process

Using a tool like Ansible can make deploying Kubernetes clusters on-premise trivial.

When deciding to manage your own Kubernetes clusters, we need to set up a few proofs-of-concept (PoC) clusters to learn how everything works, perform performance and conformance tests, and try out different configuration options.

After this phase, automating the deployment process is an important if not necessary step to ensure consistency across any clusters you build. For this, you have a few options, but the most popular are:

If you already using Ansible, Kubespray is a great option, otherwise, we recommend writing automation around Kubeadm using your preferred playbook tool after using it a few times. This will also increase your confidence and knowledge of Kubernetes.

Installation Steps

Topics covered:

Overview

manages your hosted Kubernetes environment. AKS allows you to deploy and manage containerized applications without container orchestration expertise. AKS also enables you to do many common maintenance operations without taking your app offline. These operations include provisioning, upgrading, and scaling resources on demand.

Installation Steps

Topics covered:

Overview

Terraform is an open-source infrastructure as code () software tool that allows DevOps engineers to programmatically provision the physical resources an application requires to run.

Infrastructure as code is an IT practice that manages an application's underlying IT infrastructure through programming. This approach to resource allocation allows developers to logically manage, monitor and provision resources -- as opposed to requiring that an operations team manually configure each required resource.

Terraform users define and by using a JSON-like configuration language called HCL (HashiCorp Configuration Language). HCL's simple syntax makes it easy for DevOps teams to provision and re-provision infrastructure across multiple clouds and on-premises data centres.

Cloud Resources Required

Before we provision the cloud resources, we need to understand and be sure about what resources need to be provisioned by Terraform to deploy DIGIT. The following picture shows the various key components. (AKS, Node Pools, Postgres DB, Volumes, Load Balancer)

Understand Terraform Script

• Here we have already written the terraform script that one can reuse/leverage that provisions the production-grade DIGIT Infra and can be customized with the user-specific configuration.

Deployment Steps

Save the file and exit the editor.

Terraform Execution: Infrastructure Resources Provisioning

Once you have finished declaring the resources, you can deploy all resources.

1. terraform init: command is used to initialize a working directory containing Terraform configuration files.

2. terraform plan: command creates an execution plan, which lets you preview the changes that Terraform plans to make to your infrastructure.

3. terraform apply: command executes the actions proposed in a Terraform plan to create or update infrastructure.

After the complete creation, you can see resources in your Azure account.

Now we know what the terraform script does, the resources graph that it provisions and what custom values should be given with respect to your environment. The next step is to begin to run the terraform scripts to provision infra required to Deploy DIGIT on Azure.

1. Use the CD command to move into the following directory run the following commands 1-by-1 and watch the output closely.

Test Kubernetes Cluster

The Kubernetes tools can be used to verify the newly created cluster.

1. Once Terraform Apply execution is complete, it generates the Kubernetes configuration file or you can get it from Terraform state.

2. Use the below command to get kubeconfig. It will automatically store your kubeconfig in .kube folder.

3. Verify the health of the cluster.

Topics covered:

• Hardware pre-requisites

• Software pre-requisites

Overview

Check the hardware and software pre-requisites for deployment on SDC.

Hardware Pre-requisites

Software Pre-requisites

1. Kubernetes nodes

   1. Ubuntu 18.04

   2. SSH

   3. Privileged user

   4. Python

2. Bastion machine

   1. Ansible

   2. Git

   3. Python

Topics covered:

• A brief introduction to Kubespray

• Pre-requisites for deployment using Kubespray

• Deployment steps

Overview

Kubespray is a composition of Ansible playbooks, inventory, provisioning tools, and domain knowledge for generic OS/Kubernetes cluster configuration management tasks. Kubespray provides:

• a highly available cluster

• composable attributes

• support for most popular Linux distributions

• continuous-integration tests

Pre-requisites

Before we can get started, we need a few prerequisites to be in place. This is what we are going to need:

1. A host with Ansible installed. Click here to learn more about Ansible. Find the Ansible installation details here.

2. You should also set up an SSH key pair to authenticate to the Kubernetes nodes without using a password. This permits Ansible to perform optimally.

3. Few servers/hosts/VMs to serve as our targets to deploy Kubernetes. I am using Ubuntu 18.04, and my servers each have 4GB RAM and 2vCPUs. This is fine for my testing purposes, which I use to try out new things using Kubernetes. You need to be able to SSH into each of these nodes as root using the SSH key pair I mentioned above.

Deployment Steps

The above will do the following:

• Create a new Linux User Account for use with Kubernetes on each node

• Install Kubernetes and containers on each node

• Configure the Master node

• Join the Worker nodes to the new cluster

Install Python

Ansible needs Python to be installed on all the machines.

apt-get update && apt-get install python3-pip -y

Disable Swap

sudo swapoff -a
sudo sed -i '/ swap /d' /etc/fstab

Setup SSH using key-based authentication

All the machines should be in the same network with Ubuntu or Centos installed.

ssh key should be generated from the Bastion machine and must be copied to all the servers part of your inventory.

• Generate the ssh key ssh-keygen -t rsa

• Copy over the public key to all nodes.

  Copy

  ssh-copy-id root@<node-ip-address>

Setup Ansible Controller Machine Setup Kubespray

• Clone the official repository

git clone https://github.com/kubernetes-incubator/kubespray.git

cd kubespray

• Install dependencies from requirements.txt

sudo pip install -r requirements.txt

• Create Inventory

cp -rfp inventory/sample inventory/mycluster

where mycluster is the custom configuration name. Replace with whatever name you would like to assign to the current cluster.

Create inventory using an inventory generator.

declare -a IPS=(10.67.53.158 10.67.53.159 10.67.53.160 10.67.53.161 10.67.53.162 10.67.53.163 10.67.53.164)
CONFIG_FILE=inventory/mycluster/hosts.yaml python3 contrib/inventory_builder/inventory.py ${IPS[@]}

Once it runs, you can see an inventory file that looks like the below:

all:
  hosts:
    node1:
      ansible_host: 10.67.53.158
      ip: 10.67.53.158
      access_ip: 10.67.53.158
    node2:
      ansible_host: 10.67.53.159
      ip: 10.67.53.159
      access_ip: 10.67.53.159
    node3:
      ansible_host: 10.67.53.160
      ip: 10.67.53.160
      access_ip: 10.67.53.160
    node4:
      ansible_host: 10.67.53.161
      ip: 10.67.53.161
      access_ip: 10.67.53.161
    node5:
      ansible_host: 10.67.53.162
      ip: 10.67.53.162
      access_ip: 10.67.53.162
    node6:
      ansible_host: 10.67.53.163
      ip: 10.67.53.163
      access_ip: 10.67.53.163
   node7:
      ansible_host: 10.67.53.164
      ip: 10.67.53.164
      access_ip: 10.67.53.164    
  children:
    kube-master:
      hosts:
        node1:
        node2:
        node3:
    kube-node:
      hosts:
        node1:
        node2:
        node3:
        node4:
        node5:
        node6:
        node7:
    etcd:
      hosts:
        node1:
        node2:
        node3:
        node4:
    k8s-cluster:
      children:
        kube-master:
        kube-node:
    calico-rr:
      hosts: {}

• Review and change parameters under inventory/mycluster/group_vars

vim inventory/mycluster/group_vars/all/all.yml
...
## External LB example config
## apiserver_loadbalancer_domain_name: "elb.some.domain"
apiserver_loadbalancer_domain_name: "10.211.55.101"
loadbalancer_apiserver:
  address: 10.211.55.101
  port: 443

Deploy Kubespray with Ansible Playbook - run the playbook as Ubuntu

• The option --become is required for example writing SSL keys in /etc/, installing packages and interacting with various system daemons.

• Note: Without --become - the playbook will fail to run!

ansible-playbook -i inventory/mycluster/hosts.yaml  --become --become-user=ubuntu cluster.yml

Kubernetes cluster will be created with three masters and four nodes using the above process.

Kube config will be generated in a .Kubefolder. The cluster can be accessible via kubeconfig.

HA-Proxy

• Install haproxy package in a haproxy machine that will be allocated for proxy

sudo apt-get install haproxy -y

• IPs need to be whitelisted as per the requirements in the config.

sudo vim /etc/haproxy/haproxy.cfg

global
	log /dev/log	local0
	log /dev/log	local1 notice
	chroot /var/lib/haproxy
	stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
	stats timeout 30s
	user haproxy
	group haproxy
	daemon

	# Default SSL material locations
       # ca-base /etc/ssl/certs
      #	crt-base /etc/ssl/private

	# Default ciphers to use on SSL-enabled listening sockets.
	# For more information, see ciphers(1SSL). This list is from:
	#  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
	# An alternative list with additional directives can be obtained from
	#  https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy
	#ssl-default-bind-ciphers 
	#ssl-default-bind-options no-sslv3

defaults
	log	global
	mode	http
	option	httplog
	option	dontlognull
            timeout connect 5000
            timeout check   5000
            timeout client  30000
             timeout server  60000

frontend http-in
    bind *:80
    mode tcp
    default_backend http-servers
    http-request redirect scheme https unless { ssl_fc }

frontend https-in
    bind *:443
    mode tcp
    default_backend https-servers

frontend kube-in
    bind *:8383
    mode tcp
    timeout client 3h
    #Jenkins_CD eGov_ACT eGov_Spectra 
    #acl network_allowed src 35.244.58.192 106.51.69.20 180.151.198.122 103.122.14.159 35.154.77.83 35.154.203.141 125.16.100.118 10.67.53.252  52.71.194.45 132.154.83.214 27.6.189.204 10.67.53.120
    #tcp-request connection reject if !network_allowed
    default_backend kube-servers

frontend kube2-in
    bind *:6363
    mode tcp
    timeout client 3h
    #Jenkins_CD eGov_ACT eGov_Spectra
    acl network_allowed src 35.244.58.192 106.51.69.20 180.151.198.122 103.122.14.159 35.154.77.83 35.154.203.141 125.16.100.118 10.67.53.252  52.71.194.45 132.154.83.214 27.6.189.204 10.67.53.120
    tcp-request connection reject if !network_allowed
    default_backend kube-servers


backend http-servers
        mode tcp
        balance roundrobin
        server srv4 10.67.53.161:32080 send-proxy
        server srv5 10.67.53.162:32080 send-proxy
        server srv6 10.67.53.163:32080 send-proxy

backend https-servers
        mode tcp
        balance roundrobin
        server srv4 10.67.53.161:32443 send-proxy
        server srv5 10.67.53.162:32443 send-proxy
        server srv6 10.67.53.163:32443 send-proxy

backend kube-servers
        mode tcp     
        option log-health-checks
        timeout server 3h
        server master1 10.67.53.158:6443 check check-ssl verify none inter 10000
        server master2 10.67.53.159:6443 check check-ssl verify none inter 10000
        server master3 10.67.53.160:6443 check check-ssl verify none inter 10003
        balance roundrobin  

Volumes

Iscsi volumes will be provided by the SDC team as per the requisition and the same can be used for statefulsets.

sudo iscsiadm -m discovery -t sendtargets -p 10.67.49.8:3260

Note: Please refer to the DIGIT deployment documentation to deploy DIGIT services.

Topics covered:

• Pre-requisites for setting up CI/CD on SDC

• Preparing the nodes

• CI/CD Build Job Pipeline Setup

Overview

Kubespray is a composition of Ansible playbooks, inventory, provisioning tools, and domain knowledge for generic OS/Kubernetes cluster configuration management tasks. Kubespray provides:

• a highly available cluster

• composable attributes

• support for most popular Linux distributions

• continuous-integration tests

Pre-requisites

1. GitHub Organization account

2. Fork the repos below to your GitHub Organization account

   • https://github.com/egovernments/DIGIT-DevOps

   • https://github.com/egovernments/CIOps

3. Go lang (version 1.13.X)

4. SOPS

5. GitHub user

6. Docker Hub account

7. Install kubectl on your local machine to interact with the Kubernetes cluster.

8. Install Helm to help package the services along with the configurations, environment, secrets, etc into Kubernetes manifests.

Hardware

• One Bastion machine to run Kubespray

• HA-PROXY machine which acts as a load balancer with Public IP. (CPU: 2Core , Memory: 4Gb)

• one machine which acts as a master node. (CPU: 2Core , Memory: 4Gb)

• one machine which acts as a worker node. (CPU: 8Core , Memory: 16Gb)

• ISCSI volumes for persistence volume. (number of quantity: 2 )

  • kaniko-cache-claim:- 10Gb

  • Jenkins home:- 100Gb

Software

1. Kubernetes nodes

   1. Ubuntu 18.04

   2. SSH

   3. Privileged user

   4. Python

2. Bastion machine

   1. Ansible

   2. git

   3. Python

Preparing The Nodes

Run and follow instructions on all nodes.

Install Python

Ansible needs Python to be installed on all the machines.

apt-get update && apt-get install python3-pip -y

Disable Swap

sudo swapoff -a
sudo sed -i '/ swap /d' /etc/fstab

Setup SSH using key-based authentication

All the machines should be in the same network with ubuntu or centos installed.

ssh key should be generated from the Bastion machine and must be copied to all the servers part of your inventory.

• Generate the ssh key ssh-keygen -t rsa

• Copy over the public key to all nodes.

ssh-copy-id root@<node-ip-address>

Setup Ansible Controller machine Setup kubespray

• Clone the official repository

git clone https://github.com/kubernetes-incubator/kubespray.git

cd kubespray

• Install dependencies from requirements.txt

sudo pip install -r requirements.txt

• Create Inventory

cp -rfp inventory/sample inventory/mycluster

where mycluster is the custom configuration name. Replace with whatever name you would like to assign to the current cluster.

Create inventory using an inventory generator.

declare -a IPS=(10.67.53.158 10.67.53.159 )
CONFIG_FILE=inventory/mycluster/hosts.yaml python3 contrib/inventory_builder/inventory.py ${IPS[@]}

Once it runs, you can see an inventory file that looks like the below:

all:
  hosts:
    node1:
      ansible_host: 10.67.53.158
      ip: 10.67.53.158
      access_ip: 10.67.53.158
    node2:
      ansible_host: 10.67.53.159
      ip: 10.67.53.159
      access_ip: 10.67.53.159  
  children:
    kube-master:
      hosts:
        node1:
    kube-node:
      hosts:
        node1:
        node2:
    etcd:
      hosts:
        node1:
    k8s-cluster:
      children:
        kube-master:
        kube-node:
    calico-rr:
      hosts: {}

• Review and change parameters under inventory/mycluster/group_vars

vim inventory/mycluster/group_vars/all/all.yml
...
## External LB example config
## apiserver_loadbalancer_domain_name: "elb.some.domain"
apiserver_loadbalancer_domain_name: "10.211.55.101"
loadbalancer_apiserver:
  address: 10.211.55.101
  port: 443

• Deploy Kubespray with Ansible Playbook - run the playbook as Ubuntu

  • The option --become is required, for example writing SSL keys in /etc/, installing packages and interacting with various system daemons.

  • Note: Without --become - the playbook will fail to run!

ansible-playbook -i inventory/mycluster/hosts.yaml  --become --become-user=ubuntu cluster.yml

Kubernetes cluster will be created with three masters and four nodes with the above process.

Kube config will be generated in a .Kubefolder. The cluster can be accessible via kubeconfig.

HA-Proxy

• Install haproxy package in a haproxy machine that will be allocated for proxy

sudo apt-get install haproxy -y

• IPs need to be whitelisted as per the requirements in the config.

sudo vim /etc/haproxy/haproxy.cfg

global
	log /dev/log	local0
	log /dev/log	local1 notice
	chroot /var/lib/haproxy
	stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
	stats timeout 30s
	user haproxy
	group haproxy
	daemon

	# Default SSL material locations
       # ca-base /etc/ssl/certs
      #	crt-base /etc/ssl/private

	# Default ciphers to use on SSL-enabled listening sockets.
	# For more information, see ciphers(1SSL). This list is from:
	#  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
	# An alternative list with additional directives can be obtained from
	#  https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy
	#ssl-default-bind-ciphers 
	#ssl-default-bind-options no-sslv3

defaults
	log	global
	mode	http
	option	httplog
	option	dontlognull
            timeout connect 5000
            timeout check   5000
            timeout client  30000
             timeout server  60000

frontend http-in
    bind *:80
    mode tcp
    default_backend http-servers
    http-request redirect scheme https unless { ssl_fc }

frontend https-in
    bind *:443
    mode tcp
    default_backend https-servers

frontend kube-in
    bind *:8383
    mode tcp
    timeout client 3h
    #Jenkins_CD eGov_ACT eGov_Spectra 
    #acl network_allowed src 35.244.58.192 106.51.69.20 180.151.198.122 103.122.14.159 35.154.77.83 35.154.203.141 125.16.100.118 10.67.53.252  52.71.194.45 132.154.83.214 27.6.189.204 10.67.53.120
    #tcp-request connection reject if !network_allowed
    default_backend kube-servers

frontend kube2-in
    bind *:6363
    mode tcp
    timeout client 3h
    #Jenkins_CD eGov_ACT eGov_Spectra
    acl network_allowed src 35.244.58.192 106.51.69.20 180.151.198.122 103.122.14.159 35.154.77.83 35.154.203.141 125.16.100.118 10.67.53.252  52.71.194.45 132.154.83.214 27.6.189.204 10.67.53.120
    tcp-request connection reject if !network_allowed
    default_backend kube-servers


backend http-servers
        mode tcp
        balance roundrobin
        server srv4 10.67.53.159:32080 send-proxy

backend https-servers
        mode tcp
        balance roundrobin
        server srv4 10.67.53.159:32443 send-proxy

backend kube-servers
        mode tcp     
        option log-health-checks
        timeout server 3h
        server master1 10.67.53.158:6443 check check-ssl verify none inter 10000
        balance roundrobin  

Volumes

Iscsi volumes will be provided by the SDC team as per the requisition and the same can be used for statefulsets.

sudo iscsiadm -m discovery -t sendtargets -p 10.67.49.8:3260

CI/CD Build Job Pipeline Setup

Refer to the doc here.

Topics covered:

• Pre-requisites for CI/CD setup

• CI/CD cluster setup details

• Custom variables and configuration details

• Run terraform scripts

• Jenkins deployment steps

Overview

Terraform helps you build a graph of all your resources and parallelizes the creation or modification of any non-dependent resources. Thus, Terraform builds infrastructure as efficiently as possible while providing the operators with clear insight into the dependencies on the infrastructure.

Pre-requisites

1. GitHub Organization account

2. Fork the repos below to your GitHub Organization account

   • https://github.com/egovernments/DIGIT-DevOps

   • https://github.com/egovernments/CIOps

3. AWS KMS

4. Go lang (version 1.13.X)

5. SOPS

6. GitHub user

7. Docker Hub account

8. ​AWS account with admin access to provision EKS Service. Try subscribing to a free AWS account to learn the basics. There is a limit on what is offered as free. This demo requires a commercial subscription to the EKS service. The cost for a one or two days trial might range between Rs 500-1000. (Note: Post the demo, for the internal folks, eGov will provide a 2-3 hrs time-bound access to eGov's AWS account based on the request and the available number of slots per day).

9. Install kubectl on your local machine to interact with the Kubernetes cluster.

10. Install Helm to help package the services along with the configurations, environment, secrets, etc into a Kubernetes manifests.

11. Install terraform version (0.14.10) for the Infra-as-code (IaC) to provision cloud resources as code and with desired resource graph. It also helps destroy the cluster in one go.

12. ​Install AWS CLI on your local machine so that you can use AWS CLI commands to provision and manage the cloud resources on your account.

13. Install AWS IAM Authenticator to help authenticate your connection from your local machine and deploy DIGIT services.

14. Use the AWS IAM User credentials provided for the Terraform (Infra-as-code) to connect to the AWS account and provision the cloud resources.

    ​

    • You will receive a Secret Access Key and Access Key ID. Save the keys.

    • Open the terminal and run the command given below. The AWS CLI is already installed and the credentials are saved. (Provide the credentials, leave the region and output format blank).

    Copy

    aws configure --profile cicd-infra-account 
    
    AWS Access Key ID []:<Your access key>
    AWS Secret Access Key []:<Your secret key>
    Default region name []: ap-south-1
    Default output format []: text

    • The above creates the following file on your machine as /Users/.aws/credentials.

    Copy

    [cicd-infra-account] 
    aws_access_key_id=*********** 
    aws_secret_access_key=****************************

Setup Details

Before we provision the cloud resources, we need to understand and be sure about what resources need to be provisioned by terraform to deploy CI/CD.

The following is the resource graph that we are going to provision using terraform in a standard way so that every time and for every environment, the infra is the same.

• EKS Control Plane (Kubernetes master)

• Work node group (VMs with the estimated number of vCPUs, Memory)

• EBS Volumes (Persistent volumes)

• VPCs (Private networks)

• Users to access, deploy and read-only

Terraform Script Resource Graph

• Ideally, one would write the terraform script from scratch using this doc.

• Here we have already written the terraform script that provisions the production-grade DIGIT Infra and can be customized with the specified configuration.

• Clone the DIGIT-DevOps GitHub repo. The terraform script to provision the EKS cluster is available in this repo. The structure of the files is given below.

git clone --branch release https://github.com/egovernments/DIGIT-DevOps.git
cd DIGIT-DevOps/infra-as-code/terraform


└── modules
    ├── kubernetes
    │   └── aws
    │       ├── eks-cluster
    │       │   ├── main.tf
    │       │   ├── outputs.tf
    │       │   └── variables.tf
    │       ├── network
    │       │   ├── main.tf
    │       │   ├── outputs.tf
    │       │   └── variables.tf
    │       └── workers
    │           ├── main.tf
    │           ├── outputs.tf
    │           └── variables.tf
    └── storage
        └── aws
            ├── main.tf
            ├── outputs.tf
            └── variables.tf

Here, you will find the main.tf under each of the modules that have the provisioning definition for resources like EKS cluster, storage, etc. All these are modularized and react as per the customized options provided.

Example:

1. VPC Resources -

   • VPC

   • Subnets

   • Internet Gateway

   • Route Table

2. EKS Cluster Resources -

   • IAM Role to allow EKS service to manage other AWS services

   • EC2 Security Group to allow networking traffic with the EKS cluster

   • EKS Cluster

3. EKS Worker Nodes Resources -

   • IAM role allowing Kubernetes actions to access other AWS services

   • EC2 Security Group to allow networking traffic

   • Data source to fetch the latest EKS worker AMI

   • AutoScaling launch configuration to configure worker instances

   • AutoScaling group to launch worker instances

4. Storage Module -

   • Configuration in this directory creates EBS volume and attaches it together.

The following main.tf with create s3 bucket to store all the states of the execution to keep track.

//DIGIT-DevOps/Infra-as-code/terraform/egov-cicd/remote-state

[**main.tf**](https://github.com/egovernments/DIGIT-DevOps/blob/release/infra-as-code/terraform/egov-cicd/remote-state/main.tf)\*\*\*\*

provider "aws" {
  region = "ap-south-1"
}

#This is a bucket name that you can name as you wish
resource "aws_s3_bucket" "terraform_state" {
  bucket = "try-cicd-workshop-yourname" 

  versioning {
    enabled = true
  }

  lifecycle {
    prevent_destroy = true
  }
}

#This is a bucket name that you can name as you wish
resource "aws_dynamodb_table" "terraform_state_lock" {
  name           = "try-cicd-workshop-yourname"
  read_capacity  = 1
  write_capacity = 1
  hash_key       = "LockID"

  attribute {
    name = "LockID"
    type = "S"
  }
}

The following main.tf contains the detailed resource definitions that need to be provisioned.

Dir: DIGIT-DevOps/Infra-as-code/terraform/egov-cicd

main.tf

terraform {
  backend "s3" {
    bucket = "try-cicd-workshop-yourname"
    key = "terraform"
    region = "ap-south-1"
  }
}

module "network" {
  source             = "../modules/kubernetes/aws/network"
  vpc_cidr_block     = "${var.vpc_cidr_block}"
  cluster_name       = "${var.cluster_name}"
  availability_zones = "${var.network_availability_zones}"
}


data "aws_eks_cluster" "cluster" {
  name = "${module.eks.cluster_id}"
}

data "aws_eks_cluster_auth" "cluster" {
  name = "${module.eks.cluster_id}"
}
  
provider "kubernetes" {
  host                   = "${data.aws_eks_cluster.cluster.endpoint}"
  cluster_ca_certificate = "${base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)}"
  token                  = "${data.aws_eks_cluster_auth.cluster.token}"
  #load_config_file       = false
}
  
module "eks" {
  source          = "terraform-aws-modules/eks/aws"
  version         = "17.24.0"
  cluster_name    = "${var.cluster_name}"
  vpc_id          = "${module.network.vpc_id}"
  cluster_version = "${var.kubernetes_version}"
  subnets         = "${concat(module.network.private_subnets, module.network.public_subnets)}"

  worker_groups = [
    {
      name                          = "spot"
      subnets                       = "${concat(slice(module.network.private_subnets, 0, length(var.availability_zones)))}"
      override_instance_types       = "${var.override_instance_types}"
      kubelet_extra_args            = "--node-labels=node.kubernetes.io/lifecycle=spot"
      additional_security_group_ids = ["${module.network.worker_nodes_sg_id}"]
      asg_max_size                  = "${var.number_of_worker_nodes}"
      asg_desired_capacity          = "${var.number_of_worker_nodes}"
      spot_allocation_strategy      = "capacity-optimized"
      spot_instance_pools           = null
    }
  ]
  tags = "${
    tomap({
      "kubernetes.io/cluster/${var.cluster_name}" = "owned",
      "KubernetesCluster" = "${var.cluster_name}"
    })
  }"
 
}

module "jenkins" {

  source = "../modules/storage/aws"
  storage_count = 1
  environment = "${var.cluster_name}"
  disk_prefix = "jenkins-home"
  availability_zones = "${var.availability_zones}"
  storage_sku = "gp2"
  disk_size_gb = "50"
  
}

Custom Variables/Configurations

Define your configurations in variables.tf and provide the environment-specific cloud requirements. The same terraform template can be used to customize the configurations.

├── egov-cicd
│   ├── main.tf 
│   ├── outputs.tf
│   ├── providers.tf
│   ├── remote-state
│   │   └── main.tf
│   └── variables.tf

Following are the values that you need to mention in the following files. The blank ones will prompt for inputs during execution.

variables.tf

#
# Variables Configuration
#

variable "cluster_name" {
  default = "<Desired Cluster name>"  #eg: my-digit-cicd
}

variable "vpc_cidr_block" {
  default = "192.168.0.0/16"
}

variable "network_availability_zones" {
  default = ["ap-south-1b", "ap-south-1a"]
}

variable "availability_zones" {
  default = ["ap-south-1b"]
}

variable "kubernetes_version" {
  default = "1.18"
}

variable "instance_type" {
  default = "t3a.xlarge"
}

variable "override_instance_types" {
  default = ["t3.xlarge", "r5ad.xlarge", "r5a.xlarge", "t3a.xlarge"]
}

variable "number_of_worker_nodes" {
  default = "1"
}

variable "spot_max_price" {
  default = "0.0538"
}

variable "ssh_key_name" {
  default = "egov-cicd"
}

Run Terraform Scripts

We have covered what the terraform script does, the resources graph that it provisions and what custom values should be given with respect to the selected environment.

Now, run the terraform scripts to provision the infra required to Deploy DIGIT on AWS.

Use the 'cd' command to change to the following directory and run the following commands. Check the output.

cd DIGIT-DevOps/infra-as-code/terraform/egov-cicd/remote-state
terraform init
terraform plan
terraform apply


cd DIGIT-DevOps/infra-as-code/terraform/egov-cicd
terraform init
terraform plan
terraform apply

After successful execution, the following resources get created and can be verified by the command "terraform output".

• s3 bucket: to store terraform state

• Network: VPC, security groups

• IAM users auth: using keybase to create admin, deployer, the user

Use the URL https://keybase.io/ to create your own PGP key. This creates both public and private keys on the machine, upload the public key into the keybase account that you have just created, give a name to it and ensure that you mention that in your terraform. This allows you to encrypt sensitive information.

• Example: Create a user keybase. This is "egovterraform" in the case of eGov. Upload the public key here - https://keybase.io/egovterraform/pgp_keys.asc

• Use this portal to Decrypt the secret key. To decrypt the PGP message, upload the PGP Message, PGP Private Key and Passphrase.

• EKS cluster: with master(s) & worker node(s).

• Storage(s): for es-master, es-data-v1, es-master-infra, es-data-infra-v1, zookeeper, kafka, kafka-infra.

• Use this link to get the kubeconfig from EKS to fetch the kubeconfig file. This enables you to connect to the cluster from your local machine and deploy DIGIT services to the cluster.

aws sts get-caller-identity

# Run the below command and give the respective region-code and the cluster name
aws eks --region <region-code> update-kubeconfig --name <cluster_name>

Finally, verify that you are able to connect to the cluster by running the command below:

export KUBECONFIG=<path-to-your-kube_config>
kubectl config use-context <your cluster name>

kubectl get nodes

NAME                                             STATUS AGE   VERSION               OS-Image           
ip-192-168-xx-1.ap-south-1.compute.internal   Ready  45d   v1.18.10-eks-bac369   Amazon Linux 2   

Whola! All set and now you can Deploy Jenkins

Jenkins Deployment

Post infra setup (Kubernetes Cluster), we start with deploying the Jenkins and kaniko-cache-warmer.

Pre-requisites

• Sub domain to expose CI/CD URL

• GitHub Oauth App (this provides you with the clientId, clientSecret)

  • Under Authorization callback URL enter the below url ie (Replace <domain_name> with your domain) https://<domain_name>/securityRealm/finishLogin

• Github User

• Generate a new ssh key for the above user (this provides the ssh public and private keys)

• Add the earlier created ssh public key to GitHub user account

• Add ssh private key to the gitReadSshPrivateKey

• With previously created GitHub users generate a personal read-only access token

• Docker hub account details (username and password)

• SSL certificate for the sub-domain

Prepare an <ci.yaml> master config file and <ci-secrets.yaml>. Name this file as desired. It has the following configurations:

• credentials, secrets (you need to encrypt using sops and create a ci-secret.yaml separately)

• Add subdomain name in ci.yaml

• Check and add your project specific ci-secrets.yaml details (like github Oauth app clientId, clientSecret, gitReadSshPrivateKey, gitReadAccessToken, dockerConfigJson, dockerUsername and dockerPassword)

• To create a Jenkins namespace mark this flag true

• Add your environment-specific kubconfigs under kubConfigs like https://github.com/egovernments/DIGIT-DevOps/blob/release/config-as-code/environments/ci-demo-secrets.yaml#L50

• KubeConfig environment name and deploymentJobs name from ci.yaml should be the same

• Update the CIOps and DIGIT-DevOps repo ssh url with the forked repo's ssh url.

• Make sure earlier created github users have read-only access to the forked DIGIT-DevOps and CIOps repos.

• SSL certificate for the sub-domain.

Prepare CIOps Configs

• Update the DOCKER_NAMESPACE with your docker hub organization name.

• Update the repo name "egovio" with your docker hub organization name in buildPipeline.groovy

• Update the cache repo name https://github.com/egovernments/CIOps/blob/master/vars/buildPipeline.groovy#L166https://github.com/egovernments/CIOps/blob/master/vars/buildPipeline.groovy#L185

• Remove the below env:

  • https://github.com/egovernments/CIOps/blob/master/vars/buildPipeline.groovy#L35-L54

  • https://github.com/egovernments/CIOps/blob/master/vars/buildPipeline.groovy#L156-L159

  • https://github.com/egovernments/CIOps/blob/master/vars/buildPipeline.groovy#L176-L179

  • https://github.com/egovernments/CIOps/blob/master/vars/buildPipeline.groovy#L33-L34

• Remove the below volumeMounts:

  • https://github.com/egovernments/CIOps/blob/master/vars/buildPipeline.groovy#L60-L61

  • https://github.com/egovernments/CIOps/blob/master/vars/deployer.groovy#L20-L21

• Remove the below volume:

  • https://github.com/egovernments/CIOps/blob/master/vars/buildPipeline.groovy#L80-L87

  • https://github.com/egovernments/CIOps/blob/master/vars/deployer.groovy#L32-L39

cd DIGIT-DevOps/deploy-as-code/deployer

export KUBECONFIG=<path-to-your-kube_config>
kubectl config use-context <your cluster name>
go run main.go deploy -c -e ci 'jenkins,kaniko-cache-warmer,nginx-ingress,cert-manager'

Jenkins is launched. You can access the same through your sub-domain configured in ci.yaml.

Basics

DIGIT is an open-source, customizable platform that lends itself to extensibility. New modules can be built on top of the platform to suit new use-cases or existing modules can be modified or replaced. To enable this, in addition to deploying DIGIT, a CI/CD pipeline should be set up. CD/CI pipelines enable the end user to automate & simplify the build/deploy process.

DIGIT comes with configurable "CI as code", "Deploy as code" etc.. which can be utilized to set up the pipelines and deploy new modules. More on that in the steps below.

Note: Changing the DIGIT code has implications for upgrades. That is, you may not be able to upgrade to the latest version of DIGIT depending on the changes that have been made. New modules are generally not a problem for upgrades.

Pre-reads

• Find out more on kubernetes manifests: https://www.youtube.com/watch?v=ohSUtEfDefc

• Learn how to manage env values, secrets of any service deployed in kubernetes https://www.youtube.com/watch?v=OW244LxB4oI

• Explore how to port forward to a pod running inside k8s cluster and work locally https://www.youtube.com/watch?v=TT3nd5n5Yus

• Find the SOPs to secure your keys/creds: https://www.youtube.com/watch?v=DWzJ87KbwxA

Deployment Strategies

• Setup CD/CI pipeline – This enables you to change code for individual modules/ add new modules (ILMS), build individual modules and deploy them into your environment.

• Develop on top of DIGIT

  1. Design Guide - Reference docs

  2. Create a new module. Refer to the steps in the development guide.

  3. Build the new module (part of the development guide) using Jenkins → Reference docs.

  4. Deploy the new module into your DIGIT environment.

Topics covered:

• More about the concept of cicd-as-service

• Integrating services to CI/CD

• Continuous integration

• Continuous deployment

Overview

Since there are many DIGIT services and the development code is part of various git repos, one needs to understand the concept of cicd-as-service which is open-sourced. This page guides you through the process of creating a CI/CD pipeline.

Integrating Services To CI/CD

The initial steps for integrating any new service/app to the CI/CD are discussed below.

Once the desired service is ready for integration: decide the service name, type of service, and if DB migration is required or not. While you commit the source code of the service to the git repository, the following file should be added with the relevant details which are mentioned below:

Build-config.yml – It is present under the build directory in the repository

https://raw.githubusercontent.com/egovernments/DIGIT-OSS/master/build/build-config.yml

This file contains the below details used for creating the automated Jenkins pipeline job for the newly created service.

config:
 -   name: < Name of the job, foo/bar would create job named bar inside folder foo >
     build:
     - work-dir: < Working directory of the app to be built >
       dockerfile: < Path to the dockerfile, optional, assumes dockerfile in working directory if not provided >
       image-name: < Docker image name should be similar to your helm chart name >

While integrating a new service/app, the above content needs to be added to the build-config.yml file of that app repository. For example: to onboard a new service called egov-test, the build-config.yml should be added as mentioned below.

config:  
- name: builds/DIGIT-OSS/core-services/egov-test     
  build:     
  - work-dir: egov-test      
    dockerfile: build/maven/Dockerfile       
    image-name: egov-test

If a job requires multiple images to be created (DB Migration) then it should be added as below,

config:   
- name: builds/DIGIT-OSS/core-services/egov-test     
  build:     
  - work-dir: egov-test       
    dockerfile: build/maven/Dockerfile       
    image-name: egov-test     
  - work-dir: egov-test/src/main/resources/db       
    dockerfile: egov-test/src/main/resources/db/Dockerfile       
    image-name: egov-test-db

Note - If a new repository is created then the build-config.yml is created under the build folder and the config values are added to it.

The git repository URL is then added to the Job Builder parameters

When the Jenkins Job => job builder is executed, the CI Pipeline gets created automatically based on the above details in build-config.yml. Eg: egov-test job is created in the builds/DIGIT-OSS/core-services folder in Jenkins since the “build-config is edited under core-services” And it should be the “master” branch. Once the pipeline job is created, it can be executed for any feature branch with build parameters - specifying the branch to be built (master or feature branch).

As a result of the pipeline execution, the respective app/service docker image is built and pushed to the Docker repository.

On repo provide read-only access to GitHub users (created while ci/cd deployment)

Continuous Integration (CI)

The Jenkins CI pipeline is configured and managed 'as code'.

• Job Builder – Job Builder is a Generic Jenkins job which creates the Jenkins pipeline automatically which is then used to build the application, create the docker image of it and push the image to the Docker repository. The Job Builder job requires the git repository URL as a parameter. It clones the respective git repository and reads the build/build-config.yml file for each git repository and uses it to create the service build job.

Check and ‌add your repo ssh URL in ci.yaml​‌‌

If the git repository ssh URL is available, build the Job-Builder Job.

If the git repository URL is not available, check and add the same team.

Continuous Deployment (CD)

The services are deployed and managed on a Kubernetes cluster in cloud platforms like AWS, Azure, GCP, OpenStack, etc. Here, we use helm charts to manage and generate the Kubernetes manifest files and use them for further deployment to the respective Kubernetes cluster. Each service is created as charts which have the below-mentioned files.

billing-service/   # Directory – name of the service/app
    |
    |-- Chart.yaml  # A YAML file containing information about the chart
    |-- LICENSE     # OPTIONAL: A plain text file containing the license for the chart
    |-- README.md   # OPTIONAL: A human-readable README file
    |-- values.yaml # The default configuration values for this chart
    |-- templates/  # A directory of templates that, when combined with values, will generate valid Kubernetes manifest files.
            |
            | -- deployment.yaml
            | -- service.yaml 
            | -- ingress.yaml 

Note: The steps below are only for the introduction and implementation of new services.

• To deploy a new service, you need to create a new helm chart for it( refer to the above example). The chart should be created under the charts/helm directory in the DIGIT-DevOps repository.

• If you are going to introduce a new module with the help of multiple services, we suggest you create a new Directory with your module name.

Example.:-

charts/helm/<new_module_name>/<new_service_chart_name-1>
                   |-------- /<new_service_chart_name-2>
                   |-------- /<new_service_chart_name-3>

• You can refer to the existing helm chart structure here

This chart can also be modified further based on user requirements.

The deployment of manifests to the Kubernetes cluster is made very simple and easy. There are Jenkins Jobs for each state and are environment-specific. We need to provide the image name or the service name for the respective Jenkins deployment job.

‌The deployment Jenkins job internally performs the following operations:

• Reads the image name or the service name given and finds the chart that is specific to it.

• Generates the Kubernetes manifests files from the chart using the helm template engine.

• Execute the deployment manifest with the specified docker image(s) to the Kubernetes cluster.

Topics covered:

Overview

This page details the steps to deploy the core platform services and reference applications.

The steps here can be used to deploy:

• DIGIT core platform services

• Public Grievance & Redressal module

• Trade Licence module

• Property Tax module

• Water & Sewerage module etc.

Pre-requisites

1. DIGIT uses (required v1.13.3) automated scripts to deploy the builds onto Kubernetes - or or

2. All DIGIT services are packaged using helm charts

3. is a CLI to connect to the Kubernetes cluster from your machine

4. Install for making API calls

5. IDE Code for better code/configuration editing capabilities

6. to run digit bootstrap scripts

Run Deployer

Once all the deployments configs are ready, run the command given below. Input the necessary details as prompted on the screen and the interactive installer will take care of the rest.

All done, wait and watch for 10 min. The DIGIT setup is complete, and the application will run on the URL.

Note:

• If you do not have your domain yet, you can edit the host file entries and map the nginx-ingress-service load balancer id like below

  • When you find it, add the following lines to the host file, save and close it.

  • aws-load-balancer-id digit.try.com

You can now test the DIGIT application status in the command prompt/terminal using the command below.

Overview

This documentation provides a detailed explanation of how to create a new helm chart using common templates and deploy it using helmfile.

Pre-requisites

• helm

Steps

• Clone the DIGIT-DevOps repository using the below command and checkout to the DIGIT-2.9LTS branch.

• Navigate to the common_chart_template chart

• Edit the chart.yaml with your service_name and update the dependency chart path.

• Now, edit the values.yaml file to override the values present in the common chart and the values need to be provided for your service. If your service doesn't depend on db then you can disable the DB migration by setting the value enable as false. Also if you are using your docker account you need to update the docker container in the below values.yaml file.

• After making these changes you need to provide this chart configuration in helmfile.yaml file

Overview

This guide walks you through the steps required to set up DIGIT using helmfile.

Pre-requisites

• git

• Kubernetes Cluster

Helmfile is a declarative spec for deploying helm charts. It lets you…

• Keep a directory of chart value files and maintain changes in version control.

• Apply CI/CD to configuration changes.

• Periodically sync to avoid skew in environments.

To avoid upgrades for each iteration of helm, the helmfile executable delegates to helm - as a result, must be installed.

Why Helmfile

• Standardisation of Helm templates (Override specific parameters such as namespace)

• To improve the Utilisation of Helm capabilities (Rollback)

• Easy to add any open-source helm chart to your DIGIT stack

Installation

• run as a container

• Archlinux: install via pacman -S helmfile

Running as a container

You can also use a shim to make calling the binary easier:

Helm File - commands

init

The helmfile init sub-command checks the dependencies required for helmfile operation, such as helm, helm diff plugin, helm secrets plugin, helm helm-git plugin, helm s3 plugin. When it does not exist or the version is too low, it can be installed automatically.

sync

The helmfile sync sub-command syncs your cluster state as described in your helmfile. The default helmfile is helmfile.yaml, but any YAML file can be passed by specifying a --file path/to/your/yaml/file flag.

apply

The helmfile apply sub-command begins by executing diff. If diff finds that there are any changes, sync is executed. Adding --interactive instructs Helm File to request your confirmation before sync.

destroy

The helmfile destroys sub-commands uninstalls and purges all the releases defined in the manifests. helmfile --interactive destroy instructs Helm File to request your confirmation before actually deleting releases.

Deploying DIGIT Using Helmfile

• Update domain name in env.yaml

• Update db password , flywaypassword, loginusername, loginpassword and git-sync private key in env-secrets.yaml

Note: Make sure the db_password and flywaypassword are same

• Run the below command to install DIGIT successfully.

Deploying DIGIT Using Managed Database

This guide outlines a deployment strategy for running containerized applications on Kubernetes, focusing on seamless database integration. It's suitable for teams looking to simplify their database setup using in-cluster PostgreSQL or externally managed database services.

Transitioning to Managed Database Services

By updating the Kubernetes deployment configuration, teams can easily switch from an in-cluster PostgreSQL database to a managed service. This move enhances scalability and reliability while reducing the operational overhead of database management.

Key Benefits:

• Scalability and Reliability: Managed services offer superior scalability and reliability compared to in-cluster databases.

• Reduced Operational Overhead: Outsourcing database management allows teams to concentrate on application development.

Integration Steps

• To integrate a managed PostgreSQL service, modify the following parameters in the

  deploy-as-code/charts/environments/env.yaml configuration file:

  • db-host: Update with the database service host address.

  • db-name: Update with the specific database name.

  • db-url: Update with the complete database connection URL.

  • domain: Update domain name with your domain name

• Update db password, db username, flyway username, flyway password, login username, login password and git-sync private key in env-secrets.yaml

• Run the below command to install DIGIT successfully.

Post Deployment

Please hit the below URL to login into the employee dashboard with SUPERUSER access

Log in with the user credentials which you have provided in the below file path

Destroying The Deployed DIGIT Using Helmfile

Tested Environment

This deployment approach has been thoroughly tested on an Amazon Web Services Elastic Kubernetes Service (AWS EKS) Cluster with Kubernetes version 1.28.

Topics covered:

Prepare New Release Chart For Existing Products

This section of the document walks you through the details of how to prepare a new release chart for existing products.

Pre-requisites

1. Git

2. ​ IDE Code for better code visualization/editing capabilities

Steps To Prepare Release Charts

Clone the following where we have all the release charts for you to refer.

Create a new release version of the below products.

Select your product, copy the previous release version file, and rename it with your new version.

The above code ensures the dependancy_chart-digit-v2.6.yaml with your new release version is copied and renamed.

Note: replace <your_release_version> with your new release version.​

Navigate to the release file on your local machine. Open the file using Visualstudio or any other file editor.

• Update the release version "v2.6" with your new release version.

• Update the modules(core, business, utilities, m_pgr, m_property-tax,..etc) service images with new release service images.

Add new modules

1. name - add your module name with "m_demo" ideal format ie. "m" means module and "demo" would be your module name

2. dependencies - add your module dependencies (name of other modules)

3. services - add your module-specific new service images

Prepare New Release Chart For New Product

This section of the document walks you through the details of how to prepare a new release chart for new products.

Pre-requisites

1. Git

2. GitHub Organization Account

Steps To Prepare Release Chart

When you have a new product to introduce, you can follow the below steps to create the release chart for a new product.

eGov partners can follow the below steps:

2. Clone the forked DIGIT-DevOps repo to your local machine

   • git clone --branch release https://github.com/<your_organization_account_name>/DIGIT-DevOps.git

   • Note: replace this <your_organization_account_name> with your github organization account name.

3. Navigate to the product-release-charts folder and create a new folder with your product name. cd DIGIT-DevOps/config-as-code/product-release-charts mkdir <new_product_name>Note: replace <new_product_name> with your new product name.

4. Create a new release chart file in the above-created product folder.touch dependancy_chart-<new_product_name>-<release_version>.yaml1. Open your release chart file dependancy_chart-<new_product_name>-<release_version>.yaml and start preparing as mentioned in the below release template.

eGov users can follow the below steps:

1. Clone the forked DIGIT-DevOps repo to your local machine

   • git clone --branch release https://github.com/egovernments/DIGIT-DevOps.git

   • Navigate to the product-release-charts folder and create a new folder with your product name. cd DIGIT-DevOps/config-as-code/product-release-charts mkdir <new_product_name>Note: replace <new_product_name> with your new product name

2. Create a new release chart file in the above-created product folder.touch dependancy_chart-<new_product_name>-<release_version>.yaml1. Open your release chart file dependancy_chart-<new_product_name>-<release_version>.yaml and start preparing as mentioned in the below release template.

Topics covered:

Overview

It's important to prepare a global deployment configuration yaml file that contains all necessary user-specific custom values like URL, gateways, persistent storage ids, DB details etc.

Pre-reads

• Know the basics of Kubernetes:

• Know the commands

• Know kubernetes manifests:

• Know how to manage env values, secrets of any service deployed in kubernetes

• Know how to port forward to a pod running inside k8s cluster and work locally

• Know sops to secure your keys/creds:

Post-Kubernetes Cluster setup, the deployment consists of 2 stages. As part of this sample exercise, we can deploy PGR and show the required configurations. The deployment steps are similar for all other modules except that the prerequisites differ depending on required features like SMS Gateway, Payment Gateway, etc.

Steps To Prepare Deployment Configuration File

Navigate to the following file in your local machine from the previously cloned DevOps git repository.

$ git clone -b digit-lts-go https://github.com/egovernments/DIGIT-DevOps

Step 2: After cloning the repo CD to the DIGIT-DevOps folder, type the "code" command to open the visual editor and open all the files from the DIGIT-DevOps repo.

• Replace the following as per the applicable values -

  • SMS gateway to receive OTP, transaction mobile notification, etc.

  • MDMS, Config repo URL, here is where you provide master data, tenants and various user/role access details.

  • GMap key for the location service

  • Payment gateway, in case you use PT, TL, etc

Step 4: Update your credentials and sensitive data in the secret file as per your details.

3. SOPS expects an encryption key to encrypt/decrypt a specified plain text and keep the details secured. The following are the options to generate the encryption key -

Step 5: Important: Fork the following repos that contain the master data and default configs which you would customize as per your specific implementation later. Like (Master Data, ULB, Tenant details, Users, etc) to your respective GitHub organization account.

3. New GitHub users should be enabled to access the earlier forked repos.

Step 6: Update the deployment configs for the below as per your specification:

1. Number of replicas/scale of each service (depending on whether dev or prod load).

2. Update the SMS Gateway, Email Gateway, and Payment Gateway details for the notification and payment gateway services, etc.

3. Update the config, MDMS GitHub repos wherever marked

4. Update GMap key (In case you are using Google Map services in your PGR, PT, TL, etc)

6. URL/DNS on which the DIGIT will be exposed.

7. SSL certificate for the above URL.

8. Any specific endpoint configs (Internal/external).