Pre-requisites

• Make sure you have your Azure account with the necessary credentials.

• Install Golang - Use these links to install- Linux or Windows or Mac

• ​All DIGIT services are packaged using helm charts, Install helm using the link Installing Helm

• ​kubectl is a CLI to connect to the Kubernetes cluster on your machine

• Install CURL for making API calls

• ​Install VisualStudio IDE Code for better code visualization/editing capabilities

• ​Install Postman to run digit bootstrap scripts

• Install Terraform to provide infrastructure on Azure

• Install Azure CLI and git

Infra Setup

• Clone the DIGIT-DevOps Repo and check out to the Azure branch

git clone https://github.com/egovernments/DIGIT-DevOps.git
cd DIGIT-DevOps
git checkout azure-install
code

• Change to the remote state in the sample-azure directory

cd infra-as-code/terraform/sample-azure/remote-state

• Login to Azure using the below command in the terminal

az login

• Update the variables in variables.tf file

• Run the below commands to create resource-group, storage-account and container

terraform init
terraform plan
terraform apply

• Copy the storage account name and change to the sample-azure directory

cd ..

• Open main.tf file and update the below placeholder details

```
terraform {
  backend "azurerm" {
      resource_group_name  = "<resource_group>"
      storage_account_name = "<storage_account>"
      container_name       = "<container>"
      key                  = "terraform.tfstate"
  }
}
```

• Create client-id and client-secret with necessary permissions

az ad sp create-for-rbac --name <sp_name> \
             --role owner \
             --scopes /subscriptions/<subscription_id>

• Open variables.tf file - update the variables and run the below commands

terraform init
terraform plan
terraform apply

• Note the db_name and server_name

• Fetch the kubeconfig using the below command. This will automatically store your kubeconfig in .kube folder

az aks get-credentials --resource-group <resource_group_name> --name <cluster_name>

• Check the kubeconfig and pods by running the below commands

kubectl config get-contexts
kubectl config use-context <cluster_name>
kubectl get pods -A

Deployment

• Change to the environments directory and open egov-demo.yaml

cd ../../..
cd config-as-code/environments

• Update the below configurations in egov-demo.yaml

global:
   domain: <domain_name> ## Add your Domain Name "Eg: site.mydomain.com" Do not use the dummy domain
   setup: fullsetup

cluster-configs:
    configmaps:
        egov-config:
            namespace: [ egov, monitoring ]
            data:
                db-host: <db_host_name> ## Add db-host name eg: egov-demo.database.azure.com
                db-name: <db_name> ## Add db-name
                db-url: jdbc:postgresql://<db_host_name>/<db_name> ## example: jdbc:postgresql://egov-demo.postgres.database.azure.com:5432/egov_demo
                domain: <domain_name> ## Add your Domain Name    
                egov-services-fqdn-name: https://<domain_name>/ ## Add your Domain Name
    

• Open the egov-demo-secrets.yaml file and update db details and private key

cluster-configs:
    secrets:
        db:          # update the postgres db credentials
            username: <db_username>
            password: <db_pwd> # must be more than 8 characters
            flywayUsername: <db_username>
            flywayPassword: <db_pwd> 
        git-sync:
            ssh: |
                -----BEGIN RSA PRIVATE KEY-----
                MIIEogIBAAKCAQEAg5idfPBCic+oyvNH4pkRm7OAO6bLDJT2sFtNHkXmVN3OGLUZ
                NBnXUEJS8Gkdal1JOhWSZBv6YBpOXX7m/sI3B3klxj5sLayyj9p21Yrc+Jcadsam
                XZWvl8nI1VZDBgmddnnWSHcYP+3kD6ChxykoVrbJKKi0PGNDYEKOLHvbQ/Qy5x6M
                w73xSlvF+80A3f7JhcssW/aZOIscTcNB8dAi84csjLcGIKQLKKB9omFbnd9Jh5V4
                TipjkYWhxpYo3bRGL3MfwYjzq/dGHT0I72XoeD8TT5TqYATV05KSwYPWOfDoRSp6
                LX3gyWFlibzwUkblL0rQqqEYXeXMpvUM3HadnQIDAQABAoIBACRz4Bw9yZC3L0CY
                x27ji9cfkAP2HgTsNrF/eQtLvZQApRh/Ae5Gwjf/R05FL9rI4IHwe86zWVXJs69+
                eapUTj4JtwcFP54fWo8yqvxYLQHHiZMhT/BYiH15beJ6tLI1c6Lf+RW1t8fts+EI
                VAgBRKVQmMRkhxi7Pmypwwxbes+FrZKK28CkkE6oyTXNco1Fpw/Txn93bscHgf3+
                3bHjKJw6y3e+Zgg73oLdLRSUUgrgWRlb0ShlrGrgu1hXANFGLoRGPNas4AcbqzRZ
                34a1ddroiXpcZdY+XHkLn1SupWyS0lj//EvGG0DjILjWYH1mLVsas/PB/l83T/O+
                UyQ8OdkCgYEAx23W6NndOq6tPnGMdP1NUxXGy7WwahSszNy7PWzBmfYP5yByncWi
                lBNT2ahl26qiPt98EuHEotm2lVN+yF+8sVWZJeGYNp3A5CP/plkRfPeEmKP0sqyK
                4BXgLca440/EsJnkwUOV26OOMnRu72wzJ2aUZO7l7bkyVZk7kwaIHN8CgYEAqOzg
                5F1kuAd8qwodrquVc8P+iO8uJtBMZqjjwLVw/DyfA4Eq6kdLCWQMByRojXiFYg/1
                X5xNiwLs/0qzEfh3ruBFahUNC2dzlEJwochDfXaKpCUDkxH7fKUhbMmij3s/C07Y
                js9OGdzxU3X/cGOSXONdHWDjuItoQkyANgcDmQMCgYA2uOYSqM1yr8Gr875lz6er
                F7uf5DAPO7Ma16qtNS1+kK1Wb2nj7voohZEplXK1rwGsHOjPyZGKWhEmsm2Ej/iX
                9HP8mAWLXwgx0crxm1kYIFcLB1o6uOu2h2onRXMwNJA1IVVKzr/NL/jx0U8rdVYo
                BpbLh14iOAIeyNg3BMDOowKBgEK/lv2lia7OBozvKltioWNlBqbFG89qb0YBZj03
                dLW2nn6cA4EfOp8zUS1hTY7ZGJtvAt4MvPc46LzXn3pyW5hWNhd8yfK/pgPnXOoQ
                X9qrhIzns0nhySWvek2qPvnDEV4+gYOslofRren0rkKSlbrufFSnfFPnggLwh5jR
                nLJRAoGAcQFfoWwwP5cpPM9g4WgaYENbV0BQchwwqti0TQWXrTUsgHCdHj+mrTv4
                F0R/hKrVuk1WrWDK/nkL94gTytLsjS5wF84Na+QZKcVxEUqRhndHZomPX3iVRLkV
                MvayNKpGzZEs+Qd3WyJq4y19vWwhCFQ802Pa5IAOz+tPWNi/6v4=
                -----END RSA PRIVATE KEY-----

Generate SSH key pairs (Use either method (a) or method (b)) to update the private key.

a. Using the online website (not recommended for production setup. To be only used for demo setups): https://8gwifi.org/sshfunctions.jsp

b. Using OpenSSL :

openssl genpkey -algorithm RSA -out private_key.pem openssl rsa -pubout -in private_key.pem -out public_key.pem

Add the public key to your GitHub account (reference: https://www.youtube.com/watch?v=9C7_jBn9XJ0&ab_channel=AOSNote )

• Change to the deployer directory

cd ../..
cd deploy-as-code/deployer
go run standalone_installer.go

• Run the below command to deploy nginx-ingress

kubectl apply -f ../../config-as-code/helm/charts/backbone-services/azure-nginx/ingress.yaml

• Check the pods once all services are deployed successfully

kubectl get pods -A

• Run the below command to get the load balancer id

kubectl get svc -A | grep ingress

• Copy the load balancer id and add it to your domain provider against your domain name.

Pre-requisites

Following are the pre-requisites and installation steps for setting up DIGIT on AWS:

1. Install :

   • For Linux: Follow the to install Golang on Linux.

   • For Windows: Download the installer using the and follow the installation instructions.

   • For Mac: Download the installer using the and follow the installation instructions.

2. Install - DIGIT services are packaged with Helm Charts

3. Install - CLI to connect to the Kubernetes cluster on your machine

4. Install - for making API calls

5. Install - for better code visualization/editing capabilities

6. Install - to run digit bootstrap scripts

7. Install - to provide infrastructure on AWS

8. Install and

Once you have installed all these pre-requisites, you are ready to set up DIGIT and its services.

Deployment

To provision infrastructure and set up DIGIT, follow the steps below:

1. Clone the DIGIT-DevOps repository:
2. Navigate to the cloned repository and checkout the release-1.28-kubernetes branch:

   cd DIGIT-DevOps git checkout release-1.28-kubernetes

3. Check if correct credentials are configured using the command:

   aws configure list
4. b. Using openssl :

   openssl genpkey -algorithm RSA -out private_key.pem openssl rsa -pubout -in private_key.pem -out public_key.pem

6. Open input.yaml file in vscode. You can use the below code to directly open it in VS code:

   code infra-as-code/terraform/sample-aws/input.yaml

   If the command does not work you can manually go and open the file in VS code. Once the file is open, fill the inputs. (In case you are not using vscode, you can open it any editor of your choice)

7. Fill in the inputs as per the regex mentioned in the comments.

8. Open egov-demo-secret.yaml and add DB password (line number 5), flywayPassword (line number 7) and private key.

   code config-as-code/environments/egov-demo-secrets.yaml

   Make sure the DB password and flywayPassword are same. Private key has to be added inside git-sync key against ssh key (line number 37).

9. Go to infra-as-code/terraform/sample-aws and run init.go script to enrich different files based on input.yaml.

   cd infra-as-code/terraform/sample-aws go run ../scripts/init.go

10. Navigate to the remote-state folder and run terraform to create a S3 bucket and DynamoDB.

    cd remote-state terraform init terraform plan terraform apply

11. Navigate back to sample-aws folder and run terraform to provision infrastructure for DIGIT.

    cd .. terraform init terraform plan terraform apply

    (Add the same DB password which you have added in egov-demo-secret.yaml when prompted after running terraform apply)

12. Execute the following command to generate a kubeConfig file and update the volumeIds, DB URL, and other relevant details in the egov-demo.yaml file.

    terraform output -json | go run ../scripts/envYAMLUpdater.go

13. Run the export KUBECONFIG command shown on terminal. (Note: The exact command to run will be printed on terminal. It will be something like this: export KUBECONFIG=<LOCAL_KUBECONFIGPATH> )

14. Run the digit-installer.go script to install DIGIT using the following command:

    cd ../../../deploy-as-code/deployer go run digit_installer.go

15. Once the deployment is done get the CNAME of the nginx-ingress-controller:

    kubectl get svc nginx-ingress-controller -n egov -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'

Seed Data Setup

Follow the steps below to set up seed data:

2. Port-forward user pod using the following command -

   kubectl port-forward <egov_user_pod> 8080:8080 -n egov

3. Hit super_user_creation cURL. This will create a super user with username as GRO and password as eGov@4321.

4. Open the accessToken_generation cURL. The credentials have already been populated. Change "{{YOUR_DOMAIN_NAME}}" placeholder to the domain name defined in input.yaml file while provisioning and hit the cURL.

5. In the response, you will get "access_token" field. Highlight this value, right click on it and set it as global "token "value.

6. Execute rainmaker common, rainmaker locality, rainmaker PGR localization and PGR workflow cURLs by changing "{{YOUR_DOMAIN_NAME}}" placeholder to the domain name defined in input.yaml file to setup localization and workflow seed data.

Destroying The Cluster

Follow the steps below to destroy the cluster once the demo is done:

1. Delete the nginx-ingress-controller service in the egov namespace using the below command and navigate to the infra-as-code/terraform/sample-aws directory: kubectl delete svc nginx-ingress-controller -n egov cd ../../infra-as-code/terraform/sample-aws terraform destroy Run the Terraform destroy command to delete the cluster.

2. To destroy the remote state bucket, first set the lifecycle value to false in the main.tf file in the remote-state folder:

   lifecycle { prevent_destroy = false }

3. After making this change, go to the AWS console and empty the S3 bucket associated with the remote state.

4. Once the bucket is emptied, you can proceed to destroy the remote state bucket using the Terraform destroy command.

Topics covered:

Overview

The is one of the AWS services for deploying, managing, and scaling any distributed and containerized workloads. Here we can provision the EKS cluster on AWS from the ground up and using an automated way (infra-as-code) using and then deploy the DIGIT services config-as-code using .

Pre-reads

• Know about EKS:

• Know what is terraform:

Installation Steps

Topics covered:

Overview

The is one of the AWS services for deploying, managing and scaling any distributed and containerized workloads. Here we can provision the EKS cluster on AWS from the ground up using (infra-as-code) and then deploy the DIGIT platform services as config-as-code using .

Pre-reads

• Know about EKS: ​

• Know what is terraform: ​

Pre-requisites

1. ​ with admin access to provision EKS Service. You can always subscribe to a free AWS account to learn the basics and try, but there is a limit to . For this demo, you need a commercial subscription to the . If you want to try for a day or two, it might cost you about Rs 500 - 1000.

   Note: Post the Demo (for eGov internal folks only) - request for the AWS access for 4 hrs. Time-bound access to eGov's training AWS account is available upon request and the available number of slots per day).

2. Install (any version) on the local machine - it helps in interaction with the Kubernetes cluster.

3. Install - this helps package the services along with the configurations, environments, secrets, etc into .

4. Install to run different versions of terraform on the machine. supports multiple terraform versions on the same machine and you can toggle between the desired versions.

Basics

DIGIT is an open-source, customizable platform that lends itself to extensibility. New modules can be built on top of the platform to suit new use-cases or existing modules can be modified or replaced. To enable this, in addition to deploying DIGIT, a CD/CI pipeline should be set up. CD/CI pipelines enable the end user to automate & simplify the build/deploy process.

DIGIT comes with configurable "CI as code", "Deploy as code" etc.. which can be utilized to set up the pipelines and deploy new modules. More on that in the steps below.

Note: Changing the DIGIT code has implications for upgrades. That is, you may not be able to upgrade to the latest version of DIGIT depending on the changes that have been made. New modules are generally not a problem for upgrades.

• – This enables you to change code for individual modules/ add new modules (ILMS), build individual modules and deploy them into your environment.

• Develop on top of DIGIT

  1. Design Guide -

  2. Create a new module. Follow the

  3. Build the new module (part of the development guide) →

  4. Deploy the new module into your DIGIT environment

DIGIT Core consists of 25+ services that use Postgres, Apache Kafka, Elastic Search etc. It is recommended to deploy DIGIT on a server-grade machine. Installing DIGIT on laptops is not recommended as the free memory requirements might not be met.

Follow the resources listed below to Install DIGIT.

Reference links:

Topics covered:

Overview

Follow the details below to set up your AWS account before you proceed with the DIGIT deployment.

Pre-requisites

• on your local machine so that you can use AWS CLI commands to provision and manage the cloud resources on your account.

• Install - it helps you authenticate your connection from your local machine so that you can deploy DIGIT services.

Get AWS Access

Topics Covered:

Basics

The would have helped you to get your hands dirty and build the Kubernetes cluster on a local/single VM instance - which you can consider for either local development or to understand the details involved in infra and deployment.

However, DIGIT is a platform and at the same time . Depending on the scale and performance running DIGIT on production requires advanced capabilities like HA, DRS, autoscaling, resiliency, etc. All these capabilities are supported by commercial clouds like AWS, Google, Azure, VMware, OpenStack, etc.. and also the private clouds like NIC and a few SDCs implemented clouds. These cloud providers provide the Kubernetes-as-a-managed-service that makes the entire infra setup and management seamless and automated, like infra-as-code, and config-as-code.

Pre-reads

• Know the basics of Kubernetes:

• Know the commands

• Know kubernetes manifests:

• Know how to manage env values, secrets of any service deployed in kubernetes

• Know how to port forward to a pod running inside k8s cluster and work locally

• Know sops to secure your keys/creds:

Pre-requisites

• Unlike quickstart, full installation requires state/user-specific configurations ready before proceeding with the deployment.

• You need to have a fully qualified DNS (URL) - (Should not be a dummy)

• Persistent storage depends on the cloud you are using for Kafka, ES, etc.

• Either a standalone or a hosted PostGres DB above v11.x

• GeoLocation provider configs (Google Location API), SMS Gateway, Payment Gateway, etc.

• The newly created user must have access to the MDMS and config forked repo.

1. Choose Your Cloud

Choose your cloud and follow the instructions to set up a Kubernetes cluster before moving on to deployment.

2. Deploy DIGIT

Post-infra setup (Kubernetes Cluster), the deployment involves 2 stages and 2 modes. Check out the stages first and then the modes. As part of a sample exercise, we will deploy the PGR module. However, deployment steps are similar. The pre-requisites have to be configured accordingly.

2 Stages For Deployment

• each service global, local env variables

• Number of replicas/scale of individual services (Depending on whether dev or prod)

• MDMS, config repos (Master data, ULB, tenant details, users, etc)

• sms g/w, email g/w, payment g/w

• GMap key (In case you are using Google Map services in your PGR, PT, TL, etc)

• S3 Bucket for Filestore

• URL/DNS on which the DIGIT will be exposed

• SSL certificate for the above URL

• End-points configs (Internal/external)

Stage 2: Run the digit_setup deployment script and simply answer the questions that it asks.

All done, wait and watch for 10 min, you'll have the DIGIT setup completed and the application will be running on the given URL.

2 Modes Of Deployment

Essentially, DIGIT deployment means that we need to generate Kubernetes manifests for each individual service. We use the tool called the helm, which is an easy, effective and customizable packaging and deployment solution. So depending on where and in which env you initiate the deployment there are 2 modes that you can deploy.

1. From local machine - whatever we are trying in this sample exercise so far.

3. Post-Deployment Steps

Post-deployment - the application is now accessible from the configured domain.

To try out PGR employee login - Create a sample tenant, city, and user to log in and assign an LME employee role using the seed script.

• Seed the sample data

• Import the following postman collection into the postman and run it. This collection has the seed data that enable sample test users and localisation data.

4. Assess DIGIT Deployment

By now we have successfully completed the DIGIT setup on the cloud. Use the URL that you mentioned in your env.yaml Eg: https://mysetup.digit.org and create a grievance to ensure the PGR module deployed is working fine. Refer to the product documentation below for the steps.

Credentials:

1. Citizen: You can use your default mobile number (9999999999) to sign in using the default Mobile OTP 123456.

2. Employee: Username: GRO and password: eGov@4321

Post grievance creation and assignment of the same to LME, capture the screenshot of the same and share it to ensure your setup is working fine.

5. Destroy The Cluster

Post validating the PGR functionality share the API response of the following request to assess the correctness of successful DIGIT PGR Deployment.

Finally, clean up the DIGIT setup if you wish, using the following command. This will delete the entire cluster and other cloud resources that were provisioned for the DIGIT setup.

To destroy previously-created infrastructure with Terraform, run the command below:

1. ELB is not deployed via Terraform. ELB has created at deployment time by the setup of Kubernetes Ingress. This has to be deleted manually by deleting the ingress service.

   1. kubectl delete deployment nginx-ingress-controller -n <namespace>

   2. kubectl delete svc nginx-ingress-controller -n <namespace>

      Note: Namespace can be one of egov or jenkins.

2. Delete S3 buckets manually from the AWS console and also verify if ELB got deleted.
3. Run terraform destroy.

Conclusion

All done, we have successfully created infra on the cloud, deployed DIGIT, bootstrapped DIGIT, performed a transaction on PGR and finally destroyed the cluster.

Topics covered:

Overview

It's important to prepare a global deployment configuration yaml file that contains all necessary user-specific custom values like URL, gateways, persistent storage ids, DB details etc.

Pre-reads

• Know the basics of Kubernetes:

• Know the commands

• Know kubernetes manifests:

• Know how to manage env values, secrets of any service deployed in kubernetes

• Know how to port forward to a pod running inside k8s cluster and work locally

• Know sops to secure your keys/creds:

Post-Kubernetes Cluster setup, the deployment has got 2 stages. As part of this sample exercise, we can deploy PGR and show what are the various configurations required, however deployment steps are similar for all other modules too, just that the prerequisites differ depending on the feature like SMS Gateway, Payment Gateway, etc

Steps To Prepare Deployment Configuration File

Navigate to the following file in your local machine from the previously cloned DevOps git repository.

• root@ip:/# git clone -b release https://github.com/egovernments/DIGIT-DevOps

Step 2: After cloning the repo CD into the folder DIGIT-DevOps and type the "code ." command that will open the visual editor and opens all the files from the repo DIGIT-DevOps

• Here you need to replace the following as per your values

  • SMS gateway to receive OTP, transaction mobile notification, etc.

  • MDMS, Config repo URL, here is where you provide master data, tenants and various user/role access details.

  • GMap key for the location service

  • Payment gateway, in case you use PT, TL, etc

Step 4: Update your credentials and sensitive data in the secret file as per your details.

3. SOPS expects an encryption key to use it to encrypt/decrypt a specified plain text and keep the details secured, there are a couple of options which you can use to generate the encryption key

Step 5: Important: Fork the following repos that contain the master data and default configs which you would customize as per your specific implementation later. Like (Master Data, ULB, Tenant details, Users, etc) to your respective GitHub organization account.

3. New github users should be enabled to access the earlier forked repos

Step 6: Update the deployment configs for the below as per your specification:

1. Number of replicas/scale of each individual service (Depending on whether dev or prod load)

2. You must update sms gateway, email gateway, and payment gateway details for the notification and payment gateway services, etc.

3. Update the config, MDMS github repos wherever marked

4. Update GMap key (In case you are using Google Map services in your PGR, PT, TL, etc)

6. URL/DNS on which the DIGIT will be exposed.

7. SSL certificate for the above URL.

8. Any specific endpoints configs (Internal/external)

Topics covered:

Overview

This page details the steps to deploy the core platform services and reference applications.

The steps here can be used to deploy:

• DIGIT core platform services

• Public Grievance & Redressal module

• Trade Licence module

• Property Tax module

• Water & Sewerage module etc.

Pre-requisites

1. DIGIT uses (required v1.13.3) automated scripts to deploy the builds onto Kubernetes - or or

2. All DIGIT services are packaged using helm charts

3. is a CLI to connect to the Kubernetes cluster from your machine

4. Install for making API calls

5. IDE Code for better code/configuration editing capabilities

6. to run digit bootstrap scripts

Run Deployer

Once all the deployments configs are ready, run the command given below. Input the necessary details as prompted on the screen and the interactive installer will take care of the rest.

All done, wait and watch for 10 min. The DIGIT setup is complete, and the application will run on the URL.

Note:

• If you do not have your domain yet, you can edit the host file entries and map the nginx-ingress-service load balancer id like below

  • When you find it, add the following lines to the host file, save and close it.

  • aws-load-balancer-id digit.try.com

You can now test the DIGIT application status in the command prompt/terminal using the command below.

Note: Initially pgr-services would be in crashloopbackoff state, but after performing the post-deployment steps the pgr-services will start running.

Adding Security Group ID

After deploying your environment config into the EC2 cluster, we have to add the security group ID of the instance to RDS.

Follow the steps below:

1. Go to the AWS console and search for EC2.

2. Click on Instances and select the Instance ID which you created.

3. Scroll down and go to Security and copy the Security Group ID. It starts with sg-xxxxxxxxxxxxxxxxx.

4. In the search bar, search for RDS and then go to Databases. Choose the db you had created.

5. Scroll down for Security Group rules and click any one sg. It redirects to another tab.

6. Scroll down and click on Edit inbound rules. Click on Add rule.

7. Change the Type to Postgresql and paste the copied sg-xxxxxxxxxxxxxxxxx beside the custom and click on save rules.

If you have any questions please write to us.

Make sure to use the appropriate discussion category and labels to address the issues better.

Discussion Board

Overview

The page here provides the checklist for setting up the production environment on DIGIT.

Checklist

• Deploy Observability Stack

  • Monitoring - Prometheus, Grafana

  • Logging - ES, Kibana

  • Log Rotation (configurable, default is 7 days) - ES-curator

    • ES default

  • Tracing - Jaeger (Application properties - Tracing to be enabled)

• Autoscaling/Scaling

• Backup

  • Log backup (7 days)

  • ES Data (S3)

• Troubleshooting

  • Kafka - Lag, Topics are not being consumed

  • ES - Indexes are mismatching, status red, misconfiguration

  • DB Connection outage

  • DB-as-a-pod (Disk issue)

Topics covered:

• Pre-reads for deploying on Azure

• Pre-requisites for deploying on Azure

Overview

The Azure Kubernetes Service (AKS) is one of the Azure services for deploying, managing and scaling any distributed and containerized workloads, here we can provision the AKS cluster on Azure from the ground up using terraform (infra-as-code) and then deploy the DIGIT platform services as config-as-code using Helm.

Pre-reads

• Know about AKS: https://www.youtube.com/watch?v=i5aALhhXDwc&ab_channel=DevOpsCoach​

• Know what is terraform: https://youtu.be/h970ZBgKINg

Pre-requisites​

1. Azure subscription: If you don't have an Azure subscription, create a free account before you begin.

2. Install Azure CLI

3. Configure Terraform: Follow the directions in the article, Terraform and configure access to Azure

4. Azure service principal: Follow the directions in the Create the service principal section in the article, Create an Azure service principal with Azure CLI. Take note of the values for the appId, displayName, password, and tenant.

5. Install kubectl on your local machine which helps you interact with the Kubernetes cluster.

6. Install Helm that helps you package the services along with the configurations, environments, secrets, etc into Kubernetes manifests.

Annexures:

• Kubernetes Fundamentals

• AKS Fundamentals

• Manually Provision AKS

Topics covered:

• Pre-requisites for setting up CI/CD on SDC

• Preparing the nodes

• CI/CD Build Job Pipeline Setup

Overview

Kubespray is a composition of Ansible playbooks, inventory, provisioning tools, and domain knowledge for generic OS/Kubernetes cluster configuration management tasks. Kubespray provides:

• a highly available cluster

• composable attributes

• support for most popular Linux distributions

• continuous-integration tests

Pre-requisites

1. GitHub Organization account

2. Fork the repos below to your GitHub Organization account

   • https://github.com/egovernments/DIGIT-DevOps

   • https://github.com/egovernments/CIOps

3. Go lang (version 1.13.X)

4. SOPS

5. GitHub user

6. Docker Hub account

7. Install kubectl on your local machine to interact with the Kubernetes cluster.

8. Install Helm to help package the services along with the configurations, environment, secrets, etc into Kubernetes manifests.

Hardware

• One Bastion machine to run Kubespray

• HA-PROXY machine which acts as a load balancer with Public IP. (CPU: 2Core , Memory: 4Gb)

• one machine which acts as a master node. (CPU: 2Core , Memory: 4Gb)

• one machine which acts as a worker node. (CPU: 8Core , Memory: 16Gb)

• ISCSI volumes for persistence volume. (number of quantity: 2 )

  • kaniko-cache-claim:- 10Gb

  • Jenkins home:- 100Gb

Software

1. Kubernetes nodes

   1. Ubuntu 18.04

   2. SSH

   3. Privileged user

   4. Python

2. Bastion machine

   1. Ansible

   2. git

   3. Python

Preparing The Nodes

Run and follow instructions on all nodes.

Install Python

Ansible needs Python to be installed on all the machines.

apt-get update && apt-get install python3-pip -y

Disable Swap

sudo swapoff -a
sudo sed -i '/ swap /d' /etc/fstab

Setup SSH using key-based authentication

All the machines should be in the same network with ubuntu or centos installed.

ssh key should be generated from the Bastion machine and must be copied to all the servers part of your inventory.

• Generate the ssh key ssh-keygen -t rsa

• Copy over the public key to all nodes.

ssh-copy-id root@<node-ip-address>

Setup Ansible Controller machine Setup kubespray

• Clone the official repository

git clone https://github.com/kubernetes-incubator/kubespray.git

cd kubespray

• Install dependencies from requirements.txt

sudo pip install -r requirements.txt

• Create Inventory

cp -rfp inventory/sample inventory/mycluster

where mycluster is the custom configuration name. Replace with whatever name you would like to assign to the current cluster.

Create inventory using an inventory generator.

declare -a IPS=(10.67.53.158 10.67.53.159 )
CONFIG_FILE=inventory/mycluster/hosts.yaml python3 contrib/inventory_builder/inventory.py ${IPS[@]}

Once it runs, you can see an inventory file that looks like the below:

all:
  hosts:
    node1:
      ansible_host: 10.67.53.158
      ip: 10.67.53.158
      access_ip: 10.67.53.158
    node2:
      ansible_host: 10.67.53.159
      ip: 10.67.53.159
      access_ip: 10.67.53.159  
  children:
    kube-master:
      hosts:
        node1:
    kube-node:
      hosts:
        node1:
        node2:
    etcd:
      hosts:
        node1:
    k8s-cluster:
      children:
        kube-master:
        kube-node:
    calico-rr:
      hosts: {}

• Review and change parameters under inventory/mycluster/group_vars

vim inventory/mycluster/group_vars/all/all.yml
...
## External LB example config
## apiserver_loadbalancer_domain_name: "elb.some.domain"
apiserver_loadbalancer_domain_name: "10.211.55.101"
loadbalancer_apiserver:
  address: 10.211.55.101
  port: 443

• Deploy Kubespray with Ansible Playbook - run the playbook as Ubuntu

  • The option --become is required, for example writing SSL keys in /etc/, installing packages and interacting with various system daemons.

  • Note: Without --become - the playbook will fail to run!

ansible-playbook -i inventory/mycluster/hosts.yaml  --become --become-user=ubuntu cluster.yml

Kubernetes cluster will be created with three masters and four nodes with the above process.

Kube config will be generated in a .Kubefolder. The cluster can be accessible via kubeconfig.

HA-Proxy

• Install haproxy package in a haproxy machine that will be allocated for proxy

sudo apt-get install haproxy -y

• IPs need to be whitelisted as per the requirements in the config.

sudo vim /etc/haproxy/haproxy.cfg

global
	log /dev/log	local0
	log /dev/log	local1 notice
	chroot /var/lib/haproxy
	stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
	stats timeout 30s
	user haproxy
	group haproxy
	daemon

	# Default SSL material locations
       # ca-base /etc/ssl/certs
      #	crt-base /etc/ssl/private

	# Default ciphers to use on SSL-enabled listening sockets.
	# For more information, see ciphers(1SSL). This list is from:
	#  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
	# An alternative list with additional directives can be obtained from
	#  https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy
	#ssl-default-bind-ciphers 
	#ssl-default-bind-options no-sslv3

defaults
	log	global
	mode	http
	option	httplog
	option	dontlognull
            timeout connect 5000
            timeout check   5000
            timeout client  30000
             timeout server  60000

frontend http-in
    bind *:80
    mode tcp
    default_backend http-servers
    http-request redirect scheme https unless { ssl_fc }

frontend https-in
    bind *:443
    mode tcp
    default_backend https-servers

frontend kube-in
    bind *:8383
    mode tcp
    timeout client 3h
    #Jenkins_CD eGov_ACT eGov_Spectra 
    #acl network_allowed src 35.244.58.192 106.51.69.20 180.151.198.122 103.122.14.159 35.154.77.83 35.154.203.141 125.16.100.118 10.67.53.252  52.71.194.45 132.154.83.214 27.6.189.204 10.67.53.120
    #tcp-request connection reject if !network_allowed
    default_backend kube-servers

frontend kube2-in
    bind *:6363
    mode tcp
    timeout client 3h
    #Jenkins_CD eGov_ACT eGov_Spectra
    acl network_allowed src 35.244.58.192 106.51.69.20 180.151.198.122 103.122.14.159 35.154.77.83 35.154.203.141 125.16.100.118 10.67.53.252  52.71.194.45 132.154.83.214 27.6.189.204 10.67.53.120
    tcp-request connection reject if !network_allowed
    default_backend kube-servers


backend http-servers
        mode tcp
        balance roundrobin
        server srv4 10.67.53.159:32080 send-proxy

backend https-servers
        mode tcp
        balance roundrobin
        server srv4 10.67.53.159:32443 send-proxy

backend kube-servers
        mode tcp     
        option log-health-checks
        timeout server 3h
        server master1 10.67.53.158:6443 check check-ssl verify none inter 10000
        balance roundrobin  

Volumes

Iscsi volumes will be provided by the SDC team as per the requisition and the same can be used for statefulsets.

sudo iscsiadm -m discovery -t sendtargets -p 10.67.49.8:3260

CI/CD Build Job Pipeline Setup

Refer to the doc here.

Topics covered:

• A brief introduction to Kubespray

• Pre-requisites for deployment using Kubespray

• Deployment steps

Overview

Kubespray is a composition of Ansible playbooks, inventory, provisioning tools, and domain knowledge for generic OS/Kubernetes cluster configuration management tasks. Kubespray provides:

• a highly available cluster

• composable attributes

• support for most popular Linux distributions

• continuous-integration tests

Pre-requisites

Before we can get started, we need a few prerequisites to be in place. This is what we are going to need:

1. A host with Ansible installed. Click here to learn more about Ansible. Find the Ansible installation details here.

2. You should also set up an SSH key pair to authenticate to the Kubernetes nodes without using a password. This permits Ansible to perform optimally.

3. Few servers/hosts/VMs to serve as our targets to deploy Kubernetes. I am using Ubuntu 18.04, and my servers each have 4GB RAM and 2vCPUs. This is fine for my testing purposes, which I use to try out new things using Kubernetes. You need to be able to SSH into each of these nodes as root using the SSH key pair I mentioned above.

Deployment Steps

The above will do the following:

• Create a new Linux User Account for use with Kubernetes on each node

• Install Kubernetes and containers on each node

• Configure the Master node

• Join the Worker nodes to the new cluster

Install Python

Ansible needs Python to be installed on all the machines.

apt-get update && apt-get install python3-pip -y

Disable Swap

sudo swapoff -a
sudo sed -i '/ swap /d' /etc/fstab

Setup SSH using key-based authentication

All the machines should be in the same network with Ubuntu or Centos installed.

ssh key should be generated from the Bastion machine and must be copied to all the servers part of your inventory.

• Generate the ssh key ssh-keygen -t rsa

• Copy over the public key to all nodes.

  Copy

  ssh-copy-id root@<node-ip-address>

Setup Ansible Controller Machine Setup Kubespray

• Clone the official repository

git clone https://github.com/kubernetes-incubator/kubespray.git

cd kubespray

• Install dependencies from requirements.txt

sudo pip install -r requirements.txt

• Create Inventory

cp -rfp inventory/sample inventory/mycluster

where mycluster is the custom configuration name. Replace with whatever name you would like to assign to the current cluster.

Create inventory using an inventory generator.

declare -a IPS=(10.67.53.158 10.67.53.159 10.67.53.160 10.67.53.161 10.67.53.162 10.67.53.163 10.67.53.164)
CONFIG_FILE=inventory/mycluster/hosts.yaml python3 contrib/inventory_builder/inventory.py ${IPS[@]}

Once it runs, you can see an inventory file that looks like the below:

all:
  hosts:
    node1:
      ansible_host: 10.67.53.158
      ip: 10.67.53.158
      access_ip: 10.67.53.158
    node2:
      ansible_host: 10.67.53.159
      ip: 10.67.53.159
      access_ip: 10.67.53.159
    node3:
      ansible_host: 10.67.53.160
      ip: 10.67.53.160
      access_ip: 10.67.53.160
    node4:
      ansible_host: 10.67.53.161
      ip: 10.67.53.161
      access_ip: 10.67.53.161
    node5:
      ansible_host: 10.67.53.162
      ip: 10.67.53.162
      access_ip: 10.67.53.162
    node6:
      ansible_host: 10.67.53.163
      ip: 10.67.53.163
      access_ip: 10.67.53.163
   node7:
      ansible_host: 10.67.53.164
      ip: 10.67.53.164
      access_ip: 10.67.53.164    
  children:
    kube-master:
      hosts:
        node1:
        node2:
        node3:
    kube-node:
      hosts:
        node1:
        node2:
        node3:
        node4:
        node5:
        node6:
        node7:
    etcd:
      hosts:
        node1:
        node2:
        node3:
        node4:
    k8s-cluster:
      children:
        kube-master:
        kube-node:
    calico-rr:
      hosts: {}

• Review and change parameters under inventory/mycluster/group_vars

vim inventory/mycluster/group_vars/all/all.yml
...
## External LB example config
## apiserver_loadbalancer_domain_name: "elb.some.domain"
apiserver_loadbalancer_domain_name: "10.211.55.101"
loadbalancer_apiserver:
  address: 10.211.55.101
  port: 443

Deploy Kubespray with Ansible Playbook - run the playbook as Ubuntu

• The option --become is required for example writing SSL keys in /etc/, installing packages and interacting with various system daemons.

• Note: Without --become - the playbook will fail to run!

ansible-playbook -i inventory/mycluster/hosts.yaml  --become --become-user=ubuntu cluster.yml

Kubernetes cluster will be created with three masters and four nodes using the above process.

Kube config will be generated in a .Kubefolder. The cluster can be accessible via kubeconfig.

HA-Proxy

• Install haproxy package in a haproxy machine that will be allocated for proxy

sudo apt-get install haproxy -y

• IPs need to be whitelisted as per the requirements in the config.

sudo vim /etc/haproxy/haproxy.cfg

global
	log /dev/log	local0
	log /dev/log	local1 notice
	chroot /var/lib/haproxy
	stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
	stats timeout 30s
	user haproxy
	group haproxy
	daemon

	# Default SSL material locations
       # ca-base /etc/ssl/certs
      #	crt-base /etc/ssl/private

	# Default ciphers to use on SSL-enabled listening sockets.
	# For more information, see ciphers(1SSL). This list is from:
	#  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
	# An alternative list with additional directives can be obtained from
	#  https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy
	#ssl-default-bind-ciphers 
	#ssl-default-bind-options no-sslv3

defaults
	log	global
	mode	http
	option	httplog
	option	dontlognull
            timeout connect 5000
            timeout check   5000
            timeout client  30000
             timeout server  60000

frontend http-in
    bind *:80
    mode tcp
    default_backend http-servers
    http-request redirect scheme https unless { ssl_fc }

frontend https-in
    bind *:443
    mode tcp
    default_backend https-servers

frontend kube-in
    bind *:8383
    mode tcp
    timeout client 3h
    #Jenkins_CD eGov_ACT eGov_Spectra 
    #acl network_allowed src 35.244.58.192 106.51.69.20 180.151.198.122 103.122.14.159 35.154.77.83 35.154.203.141 125.16.100.118 10.67.53.252  52.71.194.45 132.154.83.214 27.6.189.204 10.67.53.120
    #tcp-request connection reject if !network_allowed
    default_backend kube-servers

frontend kube2-in
    bind *:6363
    mode tcp
    timeout client 3h
    #Jenkins_CD eGov_ACT eGov_Spectra
    acl network_allowed src 35.244.58.192 106.51.69.20 180.151.198.122 103.122.14.159 35.154.77.83 35.154.203.141 125.16.100.118 10.67.53.252  52.71.194.45 132.154.83.214 27.6.189.204 10.67.53.120
    tcp-request connection reject if !network_allowed
    default_backend kube-servers


backend http-servers
        mode tcp
        balance roundrobin
        server srv4 10.67.53.161:32080 send-proxy
        server srv5 10.67.53.162:32080 send-proxy
        server srv6 10.67.53.163:32080 send-proxy

backend https-servers
        mode tcp
        balance roundrobin
        server srv4 10.67.53.161:32443 send-proxy
        server srv5 10.67.53.162:32443 send-proxy
        server srv6 10.67.53.163:32443 send-proxy

backend kube-servers
        mode tcp     
        option log-health-checks
        timeout server 3h
        server master1 10.67.53.158:6443 check check-ssl verify none inter 10000
        server master2 10.67.53.159:6443 check check-ssl verify none inter 10000
        server master3 10.67.53.160:6443 check check-ssl verify none inter 10003
        balance roundrobin  

Volumes

Iscsi volumes will be provided by the SDC team as per the requisition and the same can be used for statefulsets.

sudo iscsiadm -m discovery -t sendtargets -p 10.67.49.8:3260

Topics covered:

• Hardware pre-requisites

• Software pre-requisites

Overview

Check the hardware and software pre-requisites for deployment on SDC.

Hardware Pre-requisites

Software Pre-requisites

1. Kubernetes nodes

   1. Ubuntu 18.04

   2. SSH

   3. Privileged user

   4. Python

2. Bastion machine

   1. Ansible

   2. Git

   3. Python

Topics covered:

• Pre-requisites for CI/CD setup

• CI/CD cluster setup details

• Custom variables and configuration details

• Run terraform scripts

• Jenkins deployment steps

Overview

Terraform helps you build a graph of all your resources and parallelizes the creation or modification of any non-dependent resources. Thus, Terraform builds infrastructure as efficiently as possible while providing the operators with clear insight into the dependencies on the infrastructure.

Pre-requisites

1. GitHub Organization account

2. Fork the repos below to your GitHub Organization account

   • https://github.com/egovernments/DIGIT-DevOps

   • https://github.com/egovernments/CIOps

3. AWS KMS

4. Go lang (version 1.13.X)

5. SOPS

6. GitHub user

7. Docker Hub account

8. ​AWS account with admin access to provision EKS Service. Try subscribing to a free AWS account to learn the basics. There is a limit on what is offered as free. This demo requires a commercial subscription to the EKS service. The cost for a one or two days trial might range between Rs 500-1000. (Note: Post the demo, for the internal folks, eGov will provide a 2-3 hrs time-bound access to eGov's AWS account based on the request and the available number of slots per day).

9. Install kubectl on your local machine to interact with the Kubernetes cluster.

10. Install Helm to help package the services along with the configurations, environment, secrets, etc into a Kubernetes manifests.

11. Install terraform version (0.14.10) for the Infra-as-code (IaC) to provision cloud resources as code and with desired resource graph. It also helps destroy the cluster in one go.

12. ​Install AWS CLI on your local machine so that you can use AWS CLI commands to provision and manage the cloud resources on your account.

13. Install AWS IAM Authenticator to help authenticate your connection from your local machine and deploy DIGIT services.

14. Use the AWS IAM User credentials provided for the Terraform (Infra-as-code) to connect to the AWS account and provision the cloud resources.

    ​

    • You will receive a Secret Access Key and Access Key ID. Save the keys.

    • Open the terminal and run the command given below. The AWS CLI is already installed and the credentials are saved. (Provide the credentials, leave the region and output format blank).

    Copy

    aws configure --profile cicd-infra-account 
    
    AWS Access Key ID []:<Your access key>
    AWS Secret Access Key []:<Your secret key>
    Default region name []: ap-south-1
    Default output format []: text

    • The above creates the following file on your machine as /Users/.aws/credentials.

    Copy

    [cicd-infra-account] 
    aws_access_key_id=*********** 
    aws_secret_access_key=****************************

Setup Details

Before we provision the cloud resources, we need to understand and be sure about what resources need to be provisioned by terraform to deploy CI/CD.

The following is the resource graph that we are going to provision using terraform in a standard way so that every time and for every environment, the infra is the same.

• EKS Control Plane (Kubernetes master)

• Work node group (VMs with the estimated number of vCPUs, Memory)

• EBS Volumes (Persistent volumes)

• VPCs (Private networks)

• Users to access, deploy and read-only

Terraform Script Resource Graph

• Ideally, one would write the terraform script from scratch using this doc.

• Here we have already written the terraform script that provisions the production-grade DIGIT Infra and can be customized with the specified configuration.

• Clone the DIGIT-DevOps GitHub repo. The terraform script to provision the EKS cluster is available in this repo. The structure of the files is given below.

git clone --branch release https://github.com/egovernments/DIGIT-DevOps.git
cd DIGIT-DevOps/infra-as-code/terraform


└── modules
    ├── kubernetes
    │   └── aws
    │       ├── eks-cluster
    │       │   ├── main.tf
    │       │   ├── outputs.tf
    │       │   └── variables.tf
    │       ├── network
    │       │   ├── main.tf
    │       │   ├── outputs.tf
    │       │   └── variables.tf
    │       └── workers
    │           ├── main.tf
    │           ├── outputs.tf
    │           └── variables.tf
    └── storage
        └── aws
            ├── main.tf
            ├── outputs.tf
            └── variables.tf

Here, you will find the main.tf under each of the modules that have the provisioning definition for resources like EKS cluster, storage, etc. All these are modularized and react as per the customized options provided.

Example:

1. VPC Resources -

   • VPC

   • Subnets

   • Internet Gateway

   • Route Table

2. EKS Cluster Resources -

   • IAM Role to allow EKS service to manage other AWS services

   • EC2 Security Group to allow networking traffic with the EKS cluster

   • EKS Cluster

3. EKS Worker Nodes Resources -

   • IAM role allowing Kubernetes actions to access other AWS services

   • EC2 Security Group to allow networking traffic

   • Data source to fetch the latest EKS worker AMI

   • AutoScaling launch configuration to configure worker instances

   • AutoScaling group to launch worker instances

4. Storage Module -

   • Configuration in this directory creates EBS volume and attaches it together.

The following main.tf with create s3 bucket to store all the states of the execution to keep track.

//DIGIT-DevOps/Infra-as-code/terraform/egov-cicd/remote-state

[**main.tf**](https://github.com/egovernments/DIGIT-DevOps/blob/release/infra-as-code/terraform/egov-cicd/remote-state/main.tf)\*\*\*\*

provider "aws" {
  region = "ap-south-1"
}

#This is a bucket name that you can name as you wish
resource "aws_s3_bucket" "terraform_state" {
  bucket = "try-cicd-workshop-yourname" 

  versioning {
    enabled = true
  }

  lifecycle {
    prevent_destroy = true
  }
}

#This is a bucket name that you can name as you wish
resource "aws_dynamodb_table" "terraform_state_lock" {
  name           = "try-cicd-workshop-yourname"
  read_capacity  = 1
  write_capacity = 1
  hash_key       = "LockID"

  attribute {
    name = "LockID"
    type = "S"
  }
}

The following main.tf contains the detailed resource definitions that need to be provisioned.

Dir: DIGIT-DevOps/Infra-as-code/terraform/egov-cicd

main.tf

terraform {
  backend "s3" {
    bucket = "try-cicd-workshop-yourname"
    key = "terraform"
    region = "ap-south-1"
  }
}

module "network" {
  source             = "../modules/kubernetes/aws/network"
  vpc_cidr_block     = "${var.vpc_cidr_block}"
  cluster_name       = "${var.cluster_name}"
  availability_zones = "${var.network_availability_zones}"
}


data "aws_eks_cluster" "cluster" {
  name = "${module.eks.cluster_id}"
}

data "aws_eks_cluster_auth" "cluster" {
  name = "${module.eks.cluster_id}"
}
  
provider "kubernetes" {
  host                   = "${data.aws_eks_cluster.cluster.endpoint}"
  cluster_ca_certificate = "${base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)}"
  token                  = "${data.aws_eks_cluster_auth.cluster.token}"
  #load_config_file       = false
}
  
module "eks" {
  source          = "terraform-aws-modules/eks/aws"
  version         = "17.24.0"
  cluster_name    = "${var.cluster_name}"
  vpc_id          = "${module.network.vpc_id}"
  cluster_version = "${var.kubernetes_version}"
  subnets         = "${concat(module.network.private_subnets, module.network.public_subnets)}"

  worker_groups = [
    {
      name                          = "spot"
      subnets                       = "${concat(slice(module.network.private_subnets, 0, length(var.availability_zones)))}"
      override_instance_types       = "${var.override_instance_types}"
      kubelet_extra_args            = "--node-labels=node.kubernetes.io/lifecycle=spot"
      additional_security_group_ids = ["${module.network.worker_nodes_sg_id}"]
      asg_max_size                  = "${var.number_of_worker_nodes}"
      asg_desired_capacity          = "${var.number_of_worker_nodes}"
      spot_allocation_strategy      = "capacity-optimized"
      spot_instance_pools           = null
    }
  ]
  tags = "${
    tomap({
      "kubernetes.io/cluster/${var.cluster_name}" = "owned",
      "KubernetesCluster" = "${var.cluster_name}"
    })
  }"
 
}

module "jenkins" {

  source = "../modules/storage/aws"
  storage_count = 1
  environment = "${var.cluster_name}"
  disk_prefix = "jenkins-home"
  availability_zones = "${var.availability_zones}"
  storage_sku = "gp2"
  disk_size_gb = "50"
  
}

Custom Variables/Configurations

Define your configurations in variables.tf and provide the environment-specific cloud requirements. The same terraform template can be used to customize the configurations.

├── egov-cicd
│   ├── main.tf 
│   ├── outputs.tf
│   ├── providers.tf
│   ├── remote-state
│   │   └── main.tf
│   └── variables.tf

Following are the values that you need to mention in the following files. The blank ones will prompt for inputs during execution.

variables.tf

#
# Variables Configuration
#

variable "cluster_name" {
  default = "<Desired Cluster name>"  #eg: my-digit-cicd
}

variable "vpc_cidr_block" {
  default = "192.168.0.0/16"
}

variable "network_availability_zones" {
  default = ["ap-south-1b", "ap-south-1a"]
}

variable "availability_zones" {
  default = ["ap-south-1b"]
}

variable "kubernetes_version" {
  default = "1.18"
}

variable "instance_type" {
  default = "t3a.xlarge"
}

variable "override_instance_types" {
  default = ["t3.xlarge", "r5ad.xlarge", "r5a.xlarge", "t3a.xlarge"]
}

variable "number_of_worker_nodes" {
  default = "1"
}

variable "spot_max_price" {
  default = "0.0538"
}

variable "ssh_key_name" {
  default = "egov-cicd"
}

Run Terraform Scripts

We have covered what the terraform script does, the resources graph that it provisions and what custom values should be given with respect to the selected environment.

Now, run the terraform scripts to provision the infra required to Deploy DIGIT on AWS.

Use the 'cd' command to change to the following directory and run the following commands. Check the output.

cd DIGIT-DevOps/infra-as-code/terraform/egov-cicd/remote-state
terraform init
terraform plan
terraform apply


cd DIGIT-DevOps/infra-as-code/terraform/egov-cicd
terraform init
terraform plan
terraform apply

After successful execution, the following resources get created and can be verified by the command "terraform output".

• s3 bucket: to store terraform state

• Network: VPC, security groups

• IAM users auth: using keybase to create admin, deployer, the user

Use the URL https://keybase.io/ to create your own PGP key. This creates both public and private keys on the machine, upload the public key into the keybase account that you have just created, give a name to it and ensure that you mention that in your terraform. This allows you to encrypt sensitive information.

• Example: Create a user keybase. This is "egovterraform" in the case of eGov. Upload the public key here - https://keybase.io/egovterraform/pgp_keys.asc

• Use this portal to Decrypt the secret key. To decrypt the PGP message, upload the PGP Message, PGP Private Key and Passphrase.

• EKS cluster: with master(s) & worker node(s).

• Storage(s): for es-master, es-data-v1, es-master-infra, es-data-infra-v1, zookeeper, kafka, kafka-infra.

• Use this link to get the kubeconfig from EKS to fetch the kubeconfig file. This enables you to connect to the cluster from your local machine and deploy DIGIT services to the cluster.

aws sts get-caller-identity

# Run the below command and give the respective region-code and the cluster name
aws eks --region <region-code> update-kubeconfig --name <cluster_name>

Finally, verify that you are able to connect to the cluster by running the command below:

export KUBECONFIG=<path-to-your-kube_config>
kubectl config use-context <your cluster name>

kubectl get nodes

NAME                                             STATUS AGE   VERSION               OS-Image           
ip-192-168-xx-1.ap-south-1.compute.internal   Ready  45d   v1.18.10-eks-bac369   Amazon Linux 2   

Whola! All set and now you can Deploy Jenkins

Jenkins Deployment

Post infra setup (Kubernetes Cluster), we start with deploying the Jenkins and kaniko-cache-warmer.

Pre-requisites

• Sub domain to expose CI/CD URL

• GitHub Oauth App (this provides you with the clientId, clientSecret)

  • Under Authorization callback URL enter the below url ie (Replace <domain_name> with your domain) https://<domain_name>/securityRealm/finishLogin

• Github User

• Generate a new ssh key for the above user (this provides the ssh public and private keys)

• Add the earlier created ssh public key to GitHub user account

• Add ssh private key to the gitReadSshPrivateKey

• With previously created GitHub users generate a personal read-only access token

• Docker hub account details (username and password)

• SSL certificate for the sub-domain

Prepare an <ci.yaml> master config file and <ci-secrets.yaml>. Name this file as desired. It has the following configurations:

• credentials, secrets (you need to encrypt using sops and create a ci-secret.yaml separately)

• Add subdomain name in ci.yaml

• Check and add your project specific ci-secrets.yaml details (like github Oauth app clientId, clientSecret, gitReadSshPrivateKey, gitReadAccessToken, dockerConfigJson, dockerUsername and dockerPassword)

• To create a Jenkins namespace mark this flag true

• Add your environment-specific kubconfigs under kubConfigs like https://github.com/egovernments/DIGIT-DevOps/blob/release/config-as-code/environments/ci-demo-secrets.yaml#L50

• KubeConfig environment name and deploymentJobs name from ci.yaml should be the same

• Update the CIOps and DIGIT-DevOps repo ssh url with the forked repo's ssh url.

• Make sure earlier created github users have read-only access to the forked DIGIT-DevOps and CIOps repos.

• SSL certificate for the sub-domain.

Prepare CIOps Configs

• Update the DOCKER_NAMESPACE with your docker hub organization name.

• Update the repo name "egovio" with your docker hub organization name in buildPipeline.groovy

• Update the cache repo name https://github.com/egovernments/CIOps/blob/master/vars/buildPipeline.groovy#L166https://github.com/egovernments/CIOps/blob/master/vars/buildPipeline.groovy#L185

• Remove the below env:

  • https://github.com/egovernments/CIOps/blob/master/vars/buildPipeline.groovy#L35-L54

  • https://github.com/egovernments/CIOps/blob/master/vars/buildPipeline.groovy#L156-L159

  • https://github.com/egovernments/CIOps/blob/master/vars/buildPipeline.groovy#L176-L179

  • https://github.com/egovernments/CIOps/blob/master/vars/buildPipeline.groovy#L33-L34

• Remove the below volumeMounts:

  • https://github.com/egovernments/CIOps/blob/master/vars/buildPipeline.groovy#L60-L61

  • https://github.com/egovernments/CIOps/blob/master/vars/deployer.groovy#L20-L21

• Remove the below volume:

  • https://github.com/egovernments/CIOps/blob/master/vars/buildPipeline.groovy#L80-L87

  • https://github.com/egovernments/CIOps/blob/master/vars/deployer.groovy#L32-L39

cd DIGIT-DevOps/deploy-as-code/deployer

export KUBECONFIG=<path-to-your-kube_config>
kubectl config use-context <your cluster name>
go run main.go deploy -c -e ci 'jenkins,kaniko-cache-warmer,nginx-ingress,cert-manager'

Jenkins is launched. You can access the same through your sub-domain configured in ci.yaml.

On this page:

• Security overview

• Infra security

Introduction

Security is always a difficult subject to approach either by lack of experience; or by the fact you should know when the level of security is right for what you have to secure.

Security is a major concern when it comes to government systems and infra. As an architect, we can consider that working with technically educated people (engineers, experts) and tools (systems, frameworks, IDE) should prevent key VAPT issues.

However, it’s quite difficult to avoid, a certain infatuation from different categories of people to try to hack the systems.

Infra Security

1. Update to the latest version

There aren’t only bug fixes in each release but also new security measures to require advantage of them, we recommend working with the newest stable version.

Updates and support could also be harder than the new features offered in releases, so plan your updates a minimum of once a quarter. Significantly simplified updates can utilize the providers of managed Kubernetes solutions.

2. Enable role-based access control (RBAC)

Use RBAC (Role-Based Access Control) to regulate who can access and what rights they need. Usually, RBAC is enabled by default in Kubernetes version 1.6 and later (or later for a few providers), but if you’ve got been updated since then and didn’t change the configuration, you ought to double-check your settings.

However, enabling RBAC isn’t enough — it still must be used effectively. within the general case, the rights to the whole cluster (cluster-wide) should be avoided, giving preference to rights in certain namespaces. Avoid giving someone cluster administrator privileges even for debugging — it’s much safer to grant rights only necessary and from time to time.

If the appliance requires access to the Kubernetes API, create separate service accounts. and provides them with the minimum set of rights required for every use case. This approach is far better than giving an excessive amount of privilege to the default account within the namespace.

3. Use namespaces to set security boundaries

Creating separate namespaces is vital because of the first level of component isolation. it’s much easier to regulate security settings — for instance, network policies — when different types of workloads are deployed in separate namespaces.

To get in-depth knowledge of Kubernetes, enrol for a live demo on the Kubernetes course.

4. Separate sensitive workloads

A good practice to limit the potential consequences of compromise is to run workloads with sensitive data on a fanatical set of machines. This approach reduces the risk of a less secure application accessing the application with sensitive data running in the same container executable environment or on the same host.

For example, a Kubelet of a compromised node usually has access to the contents of secrets only if they are mounted on pods that are scheduled to be executed on the same node. If important secrets are often found on multiple cluster nodes, the attacker will have more opportunities to urge them.

Separation can be done using node pools (in the cloud or on-premises), as well as Kubernetes controlling mechanisms, such as namespaces, taints, tolerations, and others.

5. Protect access to cloud service metadata

Sensitive metadata — for instance, Kubelet administrative credentials, are often stolen or used with malicious intent to escalate privileges during a cluster. For example, a recent find within Shopify’s bug bounty showed in detail how a user could exceed authority by receiving metadata from a cloud provider using specially generated data for one of the microservices.

The GKE metadata concealment function changes the mechanism for deploying the cluster in such how that avoids such a drag. And we recommend using it until a permanent solution is implemented.

6. Create and define cluster network policies

Network Policies — allow you to control access to the network in and out of containerized applications. To use them, you must have a network provider with support for such a resource. For managed Kubernetes solution providers such as Google Kubernetes Engine (GKE), support will need to be enabled.

Once everything is ready, start with simple default network policies — for example, blocking (by default) traffic from other namespaces.

7. Set the Pod Security Policy for the cluster

Pod Security Policy sets the default values ​​used to start workloads in the cluster. Consider defining a policy and enabling the Pod Security Policy admission controller: the instructions for these steps vary depending on the cloud provider or deployment model used.

In the beginning, you might want to disable the NET_RAW capability in containers to protect yourself from certain types of spoofing attacks.

8. Work on node security

To improve host security, you can follow these steps:

• Ensure that the host is securely and correctly configured. One way is CIS Benchmarks; Many products have an auto checker that automatically checks the system for compliance with these standards.

• Monitor the network availability of important ports. Ensure that the network is blocking access to the ports used by Kubelet, including 10250 and 10255. Consider restricting access to the Kubernetes API server — with the exception of trusted networks. In clusters that did not require authentication and authorization in the Kubelet API, attackers used to access to such ports to launch cryptocurrency miners.

• Minimize administrative access to Kubernetes hosts Access to cluster nodes should in principle be limited: for debugging and solving other problems, as a rule, you can do without direct access to the node.

9. Enable Audit Logging

Make sure that audit logs are enabled and that you are monitoring for the occurrence of unusual or unwanted API calls in them, especially in the context of any authorization failures — such entries will have a message with the “Forbidden” status. Authorization failures can mean that an attacker is trying to take advantage of the credentials obtained.

Managed solution providers (including GKE) provide access to this data in their interfaces and can help you set up notifications in case of authorization failures.

Conclusion

Follow these guidelines for a more secure Kubernetes cluster. Remember that even after the cluster is configured securely, you need to ensure security in other aspects of the configuration and operation of containers. To improve the security of the technology stack, study the tools that provide a central system for managing deployed containers, constantly monitoring and protecting containers and cloud-native applications.

Topics covered:

• Steps to Bootstrap DIGIT

• Assess deployment

• Destroy the cluster

Overview

Post-deployment, the application can now be accessed from the configured domain. This page provides the bootstrapping steps.

Bootstrapping Steps

To try out employee login, let us create a sample tenant, city, and user to log in and assign the LME employee role through the seed script.

1. Perform the kubectl port-forwarding of the egov-user service running from the Kubernetes cluster to your localhost. This provides access to egov-user service and allows users to interact with the API directly.

kubectl port-forward svc/egov-user 8080:8080 -n egov
Output:
Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080

2. Seed the sample data

• Ensure the Postman is installed to run the following seed data API. if not, Install postman on your local machine.

• Import the following Postman collection into the Postman and run it. This contains the seed data that enables sample test users and localisation data.

  DIGIT Bootstrap

1. Execute the below commands to test your local machine's Kubernetes operations through kubectl.

#get the pods running in the cluster
kubectl get pods -n egov

Output:

NAME                                                    READY   STATUS    RESTARTS   AGE
citizen-79cf89659c-8glf4                                1/1     Running   0          14d
egov-accesscontrol-78b78dddb9-tfbkf                     1/1     Running   0          21d
egov-enc-service-574cd7b5b5-hmhh4                       1/1     Running   0          36d
egov-idgen-84954b565b-xsnqj                             1/1     Running   0          45d
egov-indexer-5f5fbb6f4b-58rtm                           1/1     Running   0          43d
egov-localization-6cc5977bb9-gm7f9                      1/1     Running   0          27d
egov-mdms-service-65d6d65d8c-t85d9                      1/1     Running   0          21d
egov-user-6676968d76-8n6t6                              1/1     Running   0          29d
egov-workflow-v2-5cdb96bcf5-dcgmf                       1/1     Running   0          36d
employee-749464fbfb-tptlh                               1/1     Running   0          14d
nginx-ingress-controller-b9678869c-mkslb                1/1     Running   0          49d
pgr-services-b9f4ffdbf-5h5kd                            1/1     Running   0          38d
zuul-788bf8cd8b-9nxfl                                   1/1     Running   0          41d


#Delete the pods so that it gets restarted automatically

kubectl delete pods zuul-788bf8cd8b-9nxfl egov-workflow-v2-5cdb96bcf5-dcgmf pgr-services-b9f4ffdbf-5h5kd -n egov

Output:

pod "zuul-788bf8cd8b-9nxfl" deleted
pod "egov-workflow-v2-5cdb96bcf5-dcgmf" deleted
pod "pgr-services-b9f4ffdbf-5h5kd" deleted

You have successfully completed the DIGIT Infra, deployment setup and installed a DIGIT - PGR module.

Use the below link in the browser -

digit.try.com/employee

Use the below credentials to login into the complaint section

• Username: GRO

• Password: eGov@4321

• City: CITYA

DIGIT Deployment Assessment

By now we have successfully completed the DIGIT setup on the cloud, use the URL you mentioned in your env.yaml.

Eg: https://mysetup.digit.org and create a grievance to ensure the PGR module deployed is working fine. Refer to the below product documentation for the steps.

Credentials:

1. Citizen: You can use your default mobile number (9999999999) to sign in using the default Mobile OTP 123456.

2. Employee: Username: GRO and password: eGov@4321

Post grievance creation and assignment of the same to LME, capture the screenshot of the same and share it to ensure your setup is working fine. Post validation, the PGR functionality shares the API response of the following request to assess the correctness of successful DIGIT PGR Deployment.

Destroy Cluster

Follow the steps below to clean up the DIGIT setup, if required. This will delete the entire cluster and other cloud resources that were provisioned for the DIGIT Setup.

Run the command below to destroy previously-created infrastructure using Terraform -

1. ELB is not deployed via Terraform. ELB has created at deployment time by the setup of Kubernetes Ingress. This has to be deleted manually by deleting the ingress service.

   • kubectl delete deployment nginx-ingress-controller -n <namespace>

   • kubectl delete svc nginx-ingress-controller -n <namespace>

   Note: Namespace can be either provided by egov or Jenkins.

2. Delete S3 buckets manually from the AWS console and also verify if the ELB got deleted.

   • In case the ELB is not deleted, you need to delete ELB from the AWS console.

3. Run terraform destroy.

cd DIGIT-DevOps/infra-as-code/terraform/my-digit-eks
terraform destroy

Conclusion

All done, we have successfully created infra on the cloud, deployed DIGIT, bootstrapped DIGIT, performed a transaction on PGR and finally destroyed the cluster.

Topics covered:

• Introduction to SDC

• Automating the deployment process

• Installation steps

Overview

Running Kubernetes on-premise gives a cloud-native experience on SDC when it comes to deploying DIGIT.

Whether States have their own on-premise data centre or have decided to forego the various managed cloud solutions, there are a few things one should know when getting started with on-premise K8s.

One should be familiar with Kubernetes and the control plane consists of the Kube-apiserver, Kube-scheduler, Kube-controller-manager and an ETCD datastore. For managed cloud solutions like Google’s Kubernetes Engine (GKE) or Azure’s Kubernetes Service (AKS), it also includes the cloud-controller-manager. This is the component that connects the cluster to external cloud services to provide networking, storage, authentication, and other support features.

To successfully deploy a bespoke Kubernetes cluster and achieve a cloud-like experience on SDC, one needs to replicate all the same features you get with a managed solution. At a high level, this means that we probably want to:

• Automate the deployment process

• Choose a networking solution

• Choose a right storage solution

• Handle security and authentication

The subsequent sections look at each of these challenges individually, and provide enough of a context required to help in getting started.

Automating The Deployment Process

Using a tool like Ansible can make deploying Kubernetes clusters on-premise trivial.

When deciding to manage your own Kubernetes clusters, we need to set up a few proofs-of-concept (PoC) clusters to learn how everything works, perform performance and conformance tests, and try out different configuration options.

After this phase, automating the deployment process is an important if not necessary step to ensure consistency across any clusters you build. For this, you have a few options, but the most popular are:

• kubeadm: a low-level tool that helps you bootstrap a minimum viable Kubernetes cluster that conforms to best practices

• kubespray: an Ansible playbook that helps deploy production-ready clusters

If you already using Ansible, Kubespray is a great option, otherwise, we recommend writing automation around Kubeadm using your preferred playbook tool after using it a few times. This will also increase your confidence and knowledge of Kubernetes.

Installation Steps

Topics covered:

• More about the concept of cicd-as-service

• Integrating services to CI/CD

• Continuous integration

• Continuous deployment

Overview

Since there are many DIGIT services and the development code is part of various git repos, one needs to understand the concept of cicd-as-service which is open-sourced. This page guides you through the process of creating a CI/CD pipeline.

Integrating Services To CI/CD

The initial steps for integrating any new service/app to the CI/CD are discussed below.

Once the desired service is ready for integration: decide the service name, type of service, and if DB migration is required or not. While you commit the source code of the service to the git repository, the following file should be added with the relevant details which are mentioned below:

Build-config.yml – It is present under the build directory in the repository

https://raw.githubusercontent.com/egovernments/DIGIT-OSS/master/build/build-config.yml

This file contains the below details used for creating the automated Jenkins pipeline job for the newly created service.

config:
 -   name: < Name of the job, foo/bar would create job named bar inside folder foo >
     build:
     - work-dir: < Working directory of the app to be built >
       dockerfile: < Path to the dockerfile, optional, assumes dockerfile in working directory if not provided >
       image-name: < Docker image name should be similar to your helm chart name >

While integrating a new service/app, the above content needs to be added to the build-config.yml file of that app repository. For example: to onboard a new service called egov-test, the build-config.yml should be added as mentioned below.

config:  
- name: builds/DIGIT-OSS/core-services/egov-test     
  build:     
  - work-dir: egov-test      
    dockerfile: build/maven/Dockerfile       
    image-name: egov-test

If a job requires multiple images to be created (DB Migration) then it should be added as below,

config:   
- name: builds/DIGIT-OSS/core-services/egov-test     
  build:     
  - work-dir: egov-test       
    dockerfile: build/maven/Dockerfile       
    image-name: egov-test     
  - work-dir: egov-test/src/main/resources/db       
    dockerfile: egov-test/src/main/resources/db/Dockerfile       
    image-name: egov-test-db

Note - If a new repository is created then the build-config.yml is created under the build folder and the config values are added to it.

The git repository URL is then added to the Job Builder parameters

When the Jenkins Job => job builder is executed, the CI Pipeline gets created automatically based on the above details in build-config.yml. Eg: egov-test job is created in the builds/DIGIT-OSS/core-services folder in Jenkins since the “build-config is edited under core-services” And it should be the “master” branch. Once the pipeline job is created, it can be executed for any feature branch with build parameters - specifying the branch to be built (master or feature branch).

As a result of the pipeline execution, the respective app/service docker image is built and pushed to the Docker repository.

On repo provide read-only access to GitHub users (created while ci/cd deployment)

Continuous Integration (CI)

The Jenkins CI pipeline is configured and managed 'as code'.

• Job Builder – Job Builder is a Generic Jenkins job which creates the Jenkins pipeline automatically which is then used to build the application, create the docker image of it and push the image to the Docker repository. The Job Builder job requires the git repository URL as a parameter. It clones the respective git repository and reads the build/build-config.yml file for each git repository and uses it to create the service build job.

Check and ‌add your repo ssh URL in ci.yaml​‌‌

If the git repository ssh URL is available, build the Job-Builder Job.

If the git repository URL is not available, check and add the same team.

Continuous Deployment (CD)

The services are deployed and managed on a Kubernetes cluster in cloud platforms like AWS, Azure, GCP, OpenStack, etc. Here, we use helm charts to manage and generate the Kubernetes manifest files and use them for further deployment to the respective Kubernetes cluster. Each service is created as charts which have the below-mentioned files.

billing-service/   # Directory – name of the service/app
    |
    |-- Chart.yaml  # A YAML file containing information about the chart
    |-- LICENSE     # OPTIONAL: A plain text file containing the license for the chart
    |-- README.md   # OPTIONAL: A human-readable README file
    |-- values.yaml # The default configuration values for this chart
    |-- templates/  # A directory of templates that, when combined with values, will generate valid Kubernetes manifest files.
            |
            | -- deployment.yaml
            | -- service.yaml 
            | -- ingress.yaml 

Note: The steps below are only for the introduction and implementation of new services.

• To deploy a new service, you need to create a new helm chart for it( refer to the above example). The chart should be created under the charts/helm directory in the DIGIT-DevOps repository.

• If you are going to introduce a new module with the help of multiple services, we suggest you create a new Directory with your module name.

Example.:-

charts/helm/<new_module_name>/<new_service_chart_name-1>
                   |-------- /<new_service_chart_name-2>
                   |-------- /<new_service_chart_name-3>

• You can refer to the existing helm chart structure here

This chart can also be modified further based on user requirements.

The deployment of manifests to the Kubernetes cluster is made very simple and easy. There are Jenkins Jobs for each state and are environment-specific. We need to provide the image name or the service name for the respective Jenkins deployment job.

‌The deployment Jenkins job internally performs the following operations:

• Reads the image name or the service name given and finds the chart that is specific to it.

• Generates the Kubernetes manifests files from the chart using the helm template engine.

• Execute the deployment manifest with the specified docker image(s) to the Kubernetes cluster.

Topics covered:

• Prepare new release chart for existing products

• Pre-requisites for preparing new release charts for existing products

• Steps to prepare release chart

• Prepare new release chart for new products

• Pre-requisites for preparing release charts for new products

• Steps to prepare release chart

Prepare New Release Chart For Existing Products

This section of the document walks you through the details of how to prepare a new release chart for existing products.

Pre-requisites