Topics covered:

• Brief intro to AWS services

• Pre-reads for deployment

• Pre-requisites to check before deployment

Overview

The Amazon Elastic Kubernetes Service (EKS) is one of the AWS services for deploying, managing and scaling any distributed and containerized workloads. Here we can provision the EKS cluster on AWS from the ground up using terraform (infra-as-code) and then deploy the DIGIT platform services as config-as-code using Helm.

Pre-reads

• Know about EKS: https://www.youtube.com/watch?v=SsUnPWp5ilc​

• Know what is terraform: https://youtu.be/h970ZBgKINg​

Pre-requisites

1. ​AWS account with admin access to provision EKS Service. You can always subscribe to a free AWS account to learn the basics and try, but there is a limit to what is offered as free. For this demo, you need a commercial subscription to the EKS service. If you want to try for a day or two, it might cost you about Rs 500 - 1000.

   Note: Post the Demo (for eGov internal folks only) - request for the AWS access for 4 hrs. Time-bound access to eGov's training AWS account is available upon request and the available number of slots per day).

2. Install kubectl (any version) on the local machine - it helps in interaction with the Kubernetes cluster.

3. Install Helm - this helps package the services along with the configurations, environments, secrets, etc into Kubernetes manifests.

4. Install tfswitch to run different versions of terraform on the machine. tfswitch supports multiple terraform versions on the same machine and you can toggle between the desired versions.

// On Mac
brew install warrensbox/tap/tfswitch

// On Linux
curl -L https://raw.githubusercontent.com/warrensbox/terraform-switcher/release/install.sh | bash

5. Run tfswitch and it will show a list of terraform versions. Scroll down and select terraform version (0.14.10) for the Infra-as-code (IaC) to provision cloud resources as code. This provides the desired resource graph and also helps destroy the cluster in one go.​

Topics covered:

• Pre-requisites to setting up the AWS account

• Get access to the AWS account

Overview

Follow the details below to set up your AWS account before you proceed with the DIGIT deployment.

Pre-requisites

• Install AWS CLI on your local machine so that you can use AWS CLI commands to provision and manage the cloud resources on your account.

• Install AWS IAM Authenticator - it helps you authenticate your connection from your local machine so that you can deploy DIGIT services.

Get AWS Access

aws configure --profile digit-quickstart-poc 

AWS Access Key ID []:<Your access key>
AWS Secret Access Key []:<Your secret key>
Default region name []: ap-south-1
Default output format []: text

[digit-quickstart-poc] 
aws_access_key_id=*********** 
aws_secret_access_key=*************************

When you have the command line access configured, everything is set for you to proceed with the terraform to provision the DIGIT Infra-as-code.

Overview

The page here provides the checklist for setting up the production environment on DIGIT.

Checklist

• Deploy Observability Stack

  • Monitoring - Prometheus, Grafana

  • Logging - ES, Kibana

  • Log Rotation (configurable, default is 7 days) - ES-curator

    • ES default

  • Tracing - Jaeger (Application properties - Tracing to be enabled)

• Autoscaling/Scaling

• Backup

  • Log backup (7 days)

  • ES Data (S3)

• Troubleshooting

  • Kafka - Lag, Topics are not being consumed

  • ES - Indexes are mismatching, status red, misconfiguration

  • DB Connection outage

  • DB-as-a-pod (Disk issue)

Topics covered:

• AWS deployment basics

• Pre-reads before deployment

• Steps to install

Overview

The Amazon Elastic Kubernetes Service (EKS) is one of the AWS services for deploying, managing, and scaling any distributed and containerized workloads. Here we can provision the EKS cluster on AWS from the ground up and using an automated way (infra-as-code) using terraform and then deploy the DIGIT services config-as-code using Helm.

Pre-reads

• Know about EKS: https://www.youtube.com/watch?v=SsUnPWp5ilc

• Know what is terraform: https://youtu.be/h970ZBgKINg

Installation Steps

Reference links:

Topics covered:

Overview

Before we provision the cloud resources, we need to understand and be sure about what resources need to be provisioned by terraform to deploy DIGIT. The following picture shows the various key components. (EKS, Worker Nodes, PostGres DB, EBS Volumes, Load Balancer).

Considering the above deployment architecture, the following is the resource graph that we are going to provision using terraform in a standard way so that every time and for every env, it'll have the same infra.

• EKS Control Plane (Kubernetes Master)

• Work node group (VMs with the estimated number of vCPUs, Memory)

• EBS Volumes (Persistent Volumes)

• RDS (PostGres)

• VPCs (Private network)

• Users to access, deploy and read-only

Pre-requisites

3. (Optional) Create your own keybase key before you run the terraform
4. Fork the DIGIT-DevOps repository into your organization account using the GitHub web portal. Make sure to add the right users to the repository. Clone the forked DIGIT-DevOps repository (not the egovernments one). Navigate to the sample-aws directory which contains the sample AWS infra provisioning script.

Create Terraform backend to specify the location of the backend Terraform state file on S3 and the DynamoDB table used for the state file locking. This step is optional. S3 buckets have to be created outside of the Terraform script.

The remote state is simply storing that state file remotely, rather than on your local filesystem. In an enterprise project and/or if Terraform is used by a team, it is recommended to set up and use the remote state.

2. Network Infrastructure Setup

The terraform script once executed performs all of the below infrastructure setups.

Amazon EKS requires subnets must be in at least two different availability zones.

1. Create AWS VPC (Virtual Private Cloud).

2. Create two public and two private Subnets in different availability zones.

3. Create an Internet Gateway to provide internet access for services within VPC.

4. Create NAT Gateway in public subnets. It is used in private subnets to allow services to connect to the internet.

5. Create Routing Tables and associate subnets with them. Add required routing rules.

6. Create Security Groups and associate subnets with them. Add required routing rules.

7. EKS cluster setup

3. Custom Variables/Configurations

Navigate to the directory: DIGIT-DevOps/Infra-as-code/terraform/sample-aws. Configurations are defined in variables.tf and provide the environment-specific cloud requirements.

Following are the values that you need to replace in the following files. The blank ones will be prompted for inputs while execution.

1. cluster_name - provide your EKS cluster name here.

2. availability_zones - This is a comma-separated list. If you would like your infra to have multi-AZ setup, please provide multiple zones here. If you provide a single zone, all infra will be provisioned within that zone. For example:

3. bucket_name - if you've created a special S3 bucket to store Terraform state.

4. dbname - Any DB name of your choice. Note that this CANNOT have hyphens or other special characters. Underscore is permitted. Example: digit_test

All other variables are default and can be modified if the admin is knowledgeable about it.

5. In the providers.tf file in the same directory, modify the "profile" variable to point to the AWS profile that was created in Step 3.

Terraform Execution: Infrastructure Resources Provisioning

Once you have finished declaring the resources, you can deploy all resources.

Let's begin to run the terraform scripts to provision infra required to Deploy DIGIT on AWS.

1. First CD into the following directory and run the following command to create the remote state.

Once the remote state is created, you are ready to provision DIGIT infra. Please run the following commands:

Important:

• DB password will be asked for in the application stage. Please remember the password you have provided. It should be at least 8 characters long. Otherwise, RDS provisioning will fail.

• The output of the apply command will be displayed on the console. Store this in a file somewhere. Values from this file will be used in the next step of deployment.

3. Finally, verify that you are able to connect to the cluster by running the following command

At this point, your basic infra has been provisioned. Please move to the next step to install DIGIT.

Destroying Infra

To destroy previously-created infrastructure with Terraform, run the command below:

1. ELB is not deployed via Terraform. ELB has created at deployment time by the setup of Kubernetes Ingress. This has to be deleted manually by deleting the ingress service.

   • kubectl delete deployment nginx-ingress-controller -n <namespace>

   • kubectl delete svc nginx-ingress-controller -n <namespace>

     Note: Namespace can be one of egov or jenkins.

2. Delete S3 buckets manually from the AWS console and also verify if ELB got deleted.
3. Run terraform destroy.

Topics covered:

Overview

It's important to prepare a global deployment configuration yaml file that contains all necessary user-specific custom values like URL, gateways, persistent storage ids, DB details etc.

Pre-reads

• Know the basics of Kubernetes:

• Know the commands

• Know kubernetes manifests:

• Know how to manage env values, secrets of any service deployed in kubernetes

• Know how to port forward to a pod running inside k8s cluster and work locally

• Know sops to secure your keys/creds:

Post-Kubernetes Cluster setup, the deployment has got 2 stages. As part of this sample exercise, we can deploy PGR and show what are the various configurations required, however deployment steps are similar for all other modules too, just that the prerequisites differ depending on the feature like SMS Gateway, Payment Gateway, etc

Steps To Prepare Deployment Configuration File

Navigate to the following file in your local machine from the previously cloned DevOps git repository.

• root@ip:/# git clone -b release https://github.com/egovernments/DIGIT-DevOps

Step 2: After cloning the repo CD into the folder DIGIT-DevOps and type the "code ." command that will open the visual editor and opens all the files from the repo DIGIT-DevOps

• Here you need to replace the following as per your values

  • SMS gateway to receive OTP, transaction mobile notification, etc.

  • MDMS, Config repo URL, here is where you provide master data, tenants and various user/role access details.

  • GMap key for the location service

  • Payment gateway, in case you use PT, TL, etc

Step 4: Update your credentials and sensitive data in the secret file as per your details.

3. SOPS expects an encryption key to use it to encrypt/decrypt a specified plain text and keep the details secured, there are a couple of options which you can use to generate the encryption key

Step 5: Important: Fork the following repos that contain the master data and default configs which you would customize as per your specific implementation later. Like (Master Data, ULB, Tenant details, Users, etc) to your respective GitHub organization account.

3. New github users should be enabled to access the earlier forked repos

Step 6: Update the deployment configs for the below as per your specification:

1. Number of replicas/scale of each individual service (Depending on whether dev or prod load)

2. You must update sms gateway, email gateway, and payment gateway details for the notification and payment gateway services, etc.

3. Update the config, MDMS github repos wherever marked

4. Update GMap key (In case you are using Google Map services in your PGR, PT, TL, etc)

6. URL/DNS on which the DIGIT will be exposed.

7. SSL certificate for the above URL.

8. Any specific endpoints configs (Internal/external)

If you have any questions please write to us.

Make sure to use the appropriate discussion category and labels to address the issues better.

Topics Covered:

Basics

The would have helped you to get your hands dirty and build the Kubernetes cluster on a local/single VM instance - which you can consider for either local development or to understand the details involved in infra and deployment.

However, DIGIT is a platform and at the same time . Depending on the scale and performance running DIGIT on production requires advanced capabilities like HA, DRS, autoscaling, resiliency, etc. All these capabilities are supported by commercial clouds like AWS, Google, Azure, VMware, OpenStack, etc.. and also the private clouds like NIC and a few SDCs implemented clouds. These cloud providers provide the Kubernetes-as-a-managed-service that makes the entire infra setup and management seamless and automated, like infra-as-code, and config-as-code.

Pre-reads

• Know the basics of Kubernetes:

• Know the commands

• Know kubernetes manifests:

• Know how to manage env values, secrets of any service deployed in kubernetes

• Know how to port forward to a pod running inside k8s cluster and work locally

• Know sops to secure your keys/creds:

Pre-requisites

• Unlike quickstart, full installation requires state/user-specific configurations ready before proceeding with the deployment.

• You need to have a fully qualified DNS (URL) - (Should not be a dummy)

• Persistent storage depends on the cloud you are using for Kafka, ES, etc.

• Either a standalone or a hosted PostGres DB above v11.x

• GeoLocation provider configs (Google Location API), SMS Gateway, Payment Gateway, etc.

• The newly created user must have access to the MDMS and config forked repo.

1. Choose Your Cloud

Choose your cloud and follow the instructions to set up a Kubernetes cluster before moving on to deployment.

2. Deploy DIGIT

Post-infra setup (Kubernetes Cluster), the deployment involves 2 stages and 2 modes. Check out the stages first and then the modes. As part of a sample exercise, we will deploy the PGR module. However, deployment steps are similar. The pre-requisites have to be configured accordingly.

2 Stages For Deployment

• each service global, local env variables

• Number of replicas/scale of individual services (Depending on whether dev or prod)

• MDMS, config repos (Master data, ULB, tenant details, users, etc)

• sms g/w, email g/w, payment g/w

• GMap key (In case you are using Google Map services in your PGR, PT, TL, etc)

• S3 Bucket for Filestore

• URL/DNS on which the DIGIT will be exposed

• SSL certificate for the above URL

• End-points configs (Internal/external)

Stage 2: Run the digit_setup deployment script and simply answer the questions that it asks.

All done, wait and watch for 10 min, you'll have the DIGIT setup completed and the application will be running on the given URL.

2 Modes Of Deployment

Essentially, DIGIT deployment means that we need to generate Kubernetes manifests for each individual service. We use the tool called the helm, which is an easy, effective and customizable packaging and deployment solution. So depending on where and in which env you initiate the deployment there are 2 modes that you can deploy.

1. From local machine - whatever we are trying in this sample exercise so far.

3. Post-Deployment Steps

Post-deployment - the application is now accessible from the configured domain.

To try out PGR employee login - Create a sample tenant, city, and user to log in and assign an LME employee role using the seed script.

• Seed the sample data

• Import the following postman collection into the postman and run it. This collection has the seed data that enable sample test users and localisation data.

4. Assess DIGIT Deployment

By now we have successfully completed the DIGIT setup on the cloud. Use the URL that you mentioned in your env.yaml Eg: https://mysetup.digit.org and create a grievance to ensure the PGR module deployed is working fine. Refer to the product documentation below for the steps.

Credentials:

1. Citizen: You can use your default mobile number (9999999999) to sign in using the default Mobile OTP 123456.

2. Employee: Username: GRO and password: eGov@4321

Post grievance creation and assignment of the same to LME, capture the screenshot of the same and share it to ensure your setup is working fine.

5. Destroy The Cluster

Post validating the PGR functionality share the API response of the following request to assess the correctness of successful DIGIT PGR Deployment.

Finally, clean up the DIGIT setup if you wish, using the following command. This will delete the entire cluster and other cloud resources that were provisioned for the DIGIT setup.

To destroy previously-created infrastructure with Terraform, run the command below:

1. ELB is not deployed via Terraform. ELB has created at deployment time by the setup of Kubernetes Ingress. This has to be deleted manually by deleting the ingress service.

   1. kubectl delete deployment nginx-ingress-controller -n <namespace>

   2. kubectl delete svc nginx-ingress-controller -n <namespace>

      Note: Namespace can be one of egov or jenkins.

2. Delete S3 buckets manually from the AWS console and also verify if ELB got deleted.
3. Run terraform destroy.

Conclusion

All done, we have successfully created infra on the cloud, deployed DIGIT, bootstrapped DIGIT, performed a transaction on PGR and finally destroyed the cluster.

Topics covered:

Overview

This page details the steps to deploy the core platform services and reference applications.

The steps here can be used to deploy:

• DIGIT core platform services

• Public Grievance & Redressal module

• Trade Licence module

• Property Tax module

• Water & Sewerage module etc.

Pre-requisites

1. DIGIT uses (required v1.13.3) automated scripts to deploy the builds onto Kubernetes - or or

2. All DIGIT services are packaged using helm charts

3. is a CLI to connect to the Kubernetes cluster from your machine

4. Install for making API calls

5. IDE Code for better code/configuration editing capabilities

6. to run digit bootstrap scripts

Run Deployer

Once all the deployments configs are ready, run the command given below. Input the necessary details as prompted on the screen and the interactive installer will take care of the rest.

All done, wait and watch for 10 min. The DIGIT setup is complete, and the application will run on the URL.

Note:

• If you do not have your domain yet, you can edit the host file entries and map the nginx-ingress-service load balancer id like below

  • When you find it, add the following lines to the host file, save and close it.

  • aws-load-balancer-id digit.try.com

You can now test the DIGIT application status in the command prompt/terminal using the command below.

Note: Initially pgr-services would be in crashloopbackoff state, but after performing the post-deployment steps the pgr-services will start running.

Adding Security Group ID

After deploying your environment config into the EC2 cluster, we have to add the security group ID of the instance to RDS.

Follow the steps below:

1. Go to the AWS console and search for EC2.

2. Click on Instances and select the Instance ID which you created.

3. Scroll down and go to Security and copy the Security Group ID. It starts with sg-xxxxxxxxxxxxxxxxx.

4. In the search bar, search for RDS and then go to Databases. Choose the db you had created.

5. Scroll down for Security Group rules and click any one sg. It redirects to another tab.

6. Scroll down and click on Edit inbound rules. Click on Add rule.

7. Change the Type to Postgresql and paste the copied sg-xxxxxxxxxxxxxxxxx beside the custom and click on save rules.

Annexures:

• s

Topics covered:

• Hardware pre-requisites

• Software pre-requisites

Overview

Check the hardware and software pre-requisites for deployment on SDC.

Hardware Pre-requisites

Software Pre-requisites

1. Kubernetes nodes

   1. Ubuntu 18.04

   2. SSH

   3. Privileged user

   4. Python

2. Bastion machine

   1. Ansible

   2. Git

   3. Python

Topics covered:

• A brief introduction to Kubespray

• Pre-requisites for deployment using Kubespray

• Deployment steps

Overview

Kubespray is a composition of Ansible playbooks, inventory, provisioning tools, and domain knowledge for generic OS/Kubernetes cluster configuration management tasks. Kubespray provides:

• a highly available cluster

• composable attributes

• support for most popular Linux distributions

• continuous-integration tests

Pre-requisites

Before we can get started, we need a few prerequisites to be in place. This is what we are going to need:

1. A host with Ansible installed. Click here to learn more about Ansible. Find the Ansible installation details here.

2. You should also set up an SSH key pair to authenticate to the Kubernetes nodes without using a password. This permits Ansible to perform optimally.

3. Few servers/hosts/VMs to serve as our targets to deploy Kubernetes. I am using Ubuntu 18.04, and my servers each have 4GB RAM and 2vCPUs. This is fine for my testing purposes, which I use to try out new things using Kubernetes. You need to be able to SSH into each of these nodes as root using the SSH key pair I mentioned above.

Deployment Steps

The above will do the following:

• Create a new Linux User Account for use with Kubernetes on each node

• Install Kubernetes and containers on each node

• Configure the Master node

• Join the Worker nodes to the new cluster

Install Python

Ansible needs Python to be installed on all the machines.

apt-get update && apt-get install python3-pip -y

Disable Swap

sudo swapoff -a
sudo sed -i '/ swap /d' /etc/fstab

Setup SSH using key-based authentication

All the machines should be in the same network with Ubuntu or Centos installed.

ssh key should be generated from the Bastion machine and must be copied to all the servers part of your inventory.

• Generate the ssh key ssh-keygen -t rsa

• Copy over the public key to all nodes.

  Copy

  ssh-copy-id root@<node-ip-address>

Setup Ansible Controller Machine Setup Kubespray

• Clone the official repository

git clone https://github.com/kubernetes-incubator/kubespray.git

cd kubespray

• Install dependencies from requirements.txt

sudo pip install -r requirements.txt

• Create Inventory

cp -rfp inventory/sample inventory/mycluster

where mycluster is the custom configuration name. Replace with whatever name you would like to assign to the current cluster.

Create inventory using an inventory generator.

declare -a IPS=(10.67.53.158 10.67.53.159 10.67.53.160 10.67.53.161 10.67.53.162 10.67.53.163 10.67.53.164)
CONFIG_FILE=inventory/mycluster/hosts.yaml python3 contrib/inventory_builder/inventory.py ${IPS[@]}

Once it runs, you can see an inventory file that looks like the below:

all:
  hosts:
    node1:
      ansible_host: 10.67.53.158
      ip: 10.67.53.158
      access_ip: 10.67.53.158
    node2:
      ansible_host: 10.67.53.159
      ip: 10.67.53.159
      access_ip: 10.67.53.159
    node3:
      ansible_host: 10.67.53.160
      ip: 10.67.53.160
      access_ip: 10.67.53.160
    node4:
      ansible_host: 10.67.53.161
      ip: 10.67.53.161
      access_ip: 10.67.53.161
    node5:
      ansible_host: 10.67.53.162
      ip: 10.67.53.162
      access_ip: 10.67.53.162
    node6:
      ansible_host: 10.67.53.163
      ip: 10.67.53.163
      access_ip: 10.67.53.163
   node7:
      ansible_host: 10.67.53.164
      ip: 10.67.53.164
      access_ip: 10.67.53.164    
  children:
    kube-master:
      hosts:
        node1:
        node2:
        node3:
    kube-node:
      hosts:
        node1:
        node2:
        node3:
        node4:
        node5:
        node6:
        node7:
    etcd:
      hosts:
        node1:
        node2:
        node3:
        node4:
    k8s-cluster:
      children:
        kube-master:
        kube-node:
    calico-rr:
      hosts: {}

• Review and change parameters under inventory/mycluster/group_vars

vim inventory/mycluster/group_vars/all/all.yml
...
## External LB example config
## apiserver_loadbalancer_domain_name: "elb.some.domain"
apiserver_loadbalancer_domain_name: "10.211.55.101"
loadbalancer_apiserver:
  address: 10.211.55.101
  port: 443

Deploy Kubespray with Ansible Playbook - run the playbook as Ubuntu

• The option --become is required for example writing SSL keys in /etc/, installing packages and interacting with various system daemons.

• Note: Without --become - the playbook will fail to run!

ansible-playbook -i inventory/mycluster/hosts.yaml  --become --become-user=ubuntu cluster.yml

Kubernetes cluster will be created with three masters and four nodes using the above process.

Kube config will be generated in a .Kubefolder. The cluster can be accessible via kubeconfig.

HA-Proxy

• Install haproxy package in a haproxy machine that will be allocated for proxy

sudo apt-get install haproxy -y

• IPs need to be whitelisted as per the requirements in the config.

sudo vim /etc/haproxy/haproxy.cfg

global
	log /dev/log	local0
	log /dev/log	local1 notice
	chroot /var/lib/haproxy
	stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
	stats timeout 30s
	user haproxy
	group haproxy
	daemon

	# Default SSL material locations
       # ca-base /etc/ssl/certs
      #	crt-base /etc/ssl/private

	# Default ciphers to use on SSL-enabled listening sockets.
	# For more information, see ciphers(1SSL). This list is from:
	#  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
	# An alternative list with additional directives can be obtained from
	#  https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy
	#ssl-default-bind-ciphers 
	#ssl-default-bind-options no-sslv3

defaults
	log	global
	mode	http
	option	httplog
	option	dontlognull
            timeout connect 5000
            timeout check   5000
            timeout client  30000
             timeout server  60000

frontend http-in
    bind *:80
    mode tcp
    default_backend http-servers
    http-request redirect scheme https unless { ssl_fc }

frontend https-in
    bind *:443
    mode tcp
    default_backend https-servers

frontend kube-in
    bind *:8383
    mode tcp
    timeout client 3h
    #Jenkins_CD eGov_ACT eGov_Spectra 
    #acl network_allowed src 35.244.58.192 106.51.69.20 180.151.198.122 103.122.14.159 35.154.77.83 35.154.203.141 125.16.100.118 10.67.53.252  52.71.194.45 132.154.83.214 27.6.189.204 10.67.53.120
    #tcp-request connection reject if !network_allowed
    default_backend kube-servers

frontend kube2-in
    bind *:6363
    mode tcp
    timeout client 3h
    #Jenkins_CD eGov_ACT eGov_Spectra
    acl network_allowed src 35.244.58.192 106.51.69.20 180.151.198.122 103.122.14.159 35.154.77.83 35.154.203.141 125.16.100.118 10.67.53.252  52.71.194.45 132.154.83.214 27.6.189.204 10.67.53.120
    tcp-request connection reject if !network_allowed
    default_backend kube-servers


backend http-servers
        mode tcp
        balance roundrobin
        server srv4 10.67.53.161:32080 send-proxy
        server srv5 10.67.53.162:32080 send-proxy
        server srv6 10.67.53.163:32080 send-proxy

backend https-servers
        mode tcp
        balance roundrobin
        server srv4 10.67.53.161:32443 send-proxy
        server srv5 10.67.53.162:32443 send-proxy
        server srv6 10.67.53.163:32443 send-proxy

backend kube-servers
        mode tcp     
        option log-health-checks
        timeout server 3h
        server master1 10.67.53.158:6443 check check-ssl verify none inter 10000
        server master2 10.67.53.159:6443 check check-ssl verify none inter 10000
        server master3 10.67.53.160:6443 check check-ssl verify none inter 10003
        balance roundrobin  

Volumes

Iscsi volumes will be provided by the SDC team as per the requisition and the same can be used for statefulsets.

sudo iscsiadm -m discovery -t sendtargets -p 10.67.49.8:3260

Topics covered:

• Pre-reads for deploying on Azure

• Pre-requisites for deploying on Azure

Overview

The Azure Kubernetes Service (AKS) is one of the Azure services for deploying, managing and scaling any distributed and containerized workloads, here we can provision the AKS cluster on Azure from the ground up using terraform (infra-as-code) and then deploy the DIGIT platform services as config-as-code using Helm.

Pre-reads

• Know about AKS: https://www.youtube.com/watch?v=i5aALhhXDwc&ab_channel=DevOpsCoach​

• Know what is terraform: https://youtu.be/h970ZBgKINg

Pre-requisites​

1. Azure subscription: If you don't have an Azure subscription, create a free account before you begin.

2. Install Azure CLI

3. Configure Terraform: Follow the directions in the article, Terraform and configure access to Azure

4. Azure service principal: Follow the directions in the Create the service principal section in the article, Create an Azure service principal with Azure CLI. Take note of the values for the appId, displayName, password, and tenant.

5. Install kubectl on your local machine which helps you interact with the Kubernetes cluster.

6. Install Helm that helps you package the services along with the configurations, environments, secrets, etc into Kubernetes manifests.

Topics covered:

• Pre-requisites for setting up CI/CD on SDC

• Preparing the nodes

• CI/CD Build Job Pipeline Setup

Overview

Kubespray is a composition of Ansible playbooks, inventory, provisioning tools, and domain knowledge for generic OS/Kubernetes cluster configuration management tasks. Kubespray provides:

• a highly available cluster

• composable attributes

• support for most popular Linux distributions

• continuous-integration tests

Pre-requisites

1. GitHub Organization account

2. Fork the repos below to your GitHub Organization account

   • https://github.com/egovernments/DIGIT-DevOps

   • https://github.com/egovernments/CIOps

3. Go lang (version 1.13.X)

4. SOPS

5. GitHub user

6. Docker Hub account

7. Install kubectl on your local machine to interact with the Kubernetes cluster.

8. Install Helm to help package the services along with the configurations, environment, secrets, etc into Kubernetes manifests.

Hardware

• One Bastion machine to run Kubespray

• HA-PROXY machine which acts as a load balancer with Public IP. (CPU: 2Core , Memory: 4Gb)

• one machine which acts as a master node. (CPU: 2Core , Memory: 4Gb)

• one machine which acts as a worker node. (CPU: 8Core , Memory: 16Gb)

• ISCSI volumes for persistence volume. (number of quantity: 2 )

  • kaniko-cache-claim:- 10Gb

  • Jenkins home:- 100Gb

Software

1. Kubernetes nodes

   1. Ubuntu 18.04

   2. SSH

   3. Privileged user

   4. Python

2. Bastion machine

   1. Ansible

   2. git

   3. Python

Preparing The Nodes

Run and follow instructions on all nodes.

Install Python

Ansible needs Python to be installed on all the machines.

apt-get update && apt-get install python3-pip -y

Disable Swap

sudo swapoff -a
sudo sed -i '/ swap /d' /etc/fstab

Setup SSH using key-based authentication

All the machines should be in the same network with ubuntu or centos installed.

ssh key should be generated from the Bastion machine and must be copied to all the servers part of your inventory.

• Generate the ssh key ssh-keygen -t rsa

• Copy over the public key to all nodes.

ssh-copy-id root@<node-ip-address>

Setup Ansible Controller machine Setup kubespray

• Clone the official repository

git clone https://github.com/kubernetes-incubator/kubespray.git

cd kubespray

• Install dependencies from requirements.txt

sudo pip install -r requirements.txt

• Create Inventory

cp -rfp inventory/sample inventory/mycluster

where mycluster is the custom configuration name. Replace with whatever name you would like to assign to the current cluster.

Create inventory using an inventory generator.

declare -a IPS=(10.67.53.158 10.67.53.159 )
CONFIG_FILE=inventory/mycluster/hosts.yaml python3 contrib/inventory_builder/inventory.py ${IPS[@]}

Once it runs, you can see an inventory file that looks like the below:

all:
  hosts:
    node1:
      ansible_host: 10.67.53.158
      ip: 10.67.53.158
      access_ip: 10.67.53.158
    node2:
      ansible_host: 10.67.53.159
      ip: 10.67.53.159
      access_ip: 10.67.53.159  
  children:
    kube-master:
      hosts:
        node1:
    kube-node:
      hosts:
        node1:
        node2:
    etcd:
      hosts:
        node1:
    k8s-cluster:
      children:
        kube-master:
        kube-node:
    calico-rr:
      hosts: {}

• Review and change parameters under inventory/mycluster/group_vars

vim inventory/mycluster/group_vars/all/all.yml
...
## External LB example config
## apiserver_loadbalancer_domain_name: "elb.some.domain"
apiserver_loadbalancer_domain_name: "10.211.55.101"
loadbalancer_apiserver:
  address: 10.211.55.101
  port: 443

• Deploy Kubespray with Ansible Playbook - run the playbook as Ubuntu

  • The option --become is required, for example writing SSL keys in /etc/, installing packages and interacting with various system daemons.

  • Note: Without --become - the playbook will fail to run!

ansible-playbook -i inventory/mycluster/hosts.yaml  --become --become-user=ubuntu cluster.yml

Kubernetes cluster will be created with three masters and four nodes with the above process.

Kube config will be generated in a .Kubefolder. The cluster can be accessible via kubeconfig.

HA-Proxy

• Install haproxy package in a haproxy machine that will be allocated for proxy

sudo apt-get install haproxy -y

• IPs need to be whitelisted as per the requirements in the config.

sudo vim /etc/haproxy/haproxy.cfg

global
	log /dev/log	local0
	log /dev/log	local1 notice
	chroot /var/lib/haproxy
	stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
	stats timeout 30s
	user haproxy
	group haproxy
	daemon

	# Default SSL material locations
       # ca-base /etc/ssl/certs
      #	crt-base /etc/ssl/private

	# Default ciphers to use on SSL-enabled listening sockets.
	# For more information, see ciphers(1SSL). This list is from:
	#  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
	# An alternative list with additional directives can be obtained from
	#  https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy
	#ssl-default-bind-ciphers 
	#ssl-default-bind-options no-sslv3

defaults
	log	global
	mode	http
	option	httplog
	option	dontlognull
            timeout connect 5000
            timeout check   5000
            timeout client  30000
             timeout server  60000

frontend http-in
    bind *:80
    mode tcp
    default_backend http-servers
    http-request redirect scheme https unless { ssl_fc }

frontend https-in
    bind *:443
    mode tcp
    default_backend https-servers

frontend kube-in
    bind *:8383
    mode tcp
    timeout client 3h
    #Jenkins_CD eGov_ACT eGov_Spectra 
    #acl network_allowed src 35.244.58.192 106.51.69.20 180.151.198.122 103.122.14.159 35.154.77.83 35.154.203.141 125.16.100.118 10.67.53.252  52.71.194.45 132.154.83.214 27.6.189.204 10.67.53.120
    #tcp-request connection reject if !network_allowed
    default_backend kube-servers

frontend kube2-in
    bind *:6363
    mode tcp
    timeout client 3h
    #Jenkins_CD eGov_ACT eGov_Spectra
    acl network_allowed src 35.244.58.192 106.51.69.20 180.151.198.122 103.122.14.159 35.154.77.83 35.154.203.141 125.16.100.118 10.67.53.252  52.71.194.45 132.154.83.214 27.6.189.204 10.67.53.120
    tcp-request connection reject if !network_allowed
    default_backend kube-servers


backend http-servers
        mode tcp
        balance roundrobin
        server srv4 10.67.53.159:32080 send-proxy

backend https-servers
        mode tcp
        balance roundrobin
        server srv4 10.67.53.159:32443 send-proxy

backend kube-servers
        mode tcp     
        option log-health-checks
        timeout server 3h
        server master1 10.67.53.158:6443 check check-ssl verify none inter 10000
        balance roundrobin  

Volumes

Iscsi volumes will be provided by the SDC team as per the requisition and the same can be used for statefulsets.

sudo iscsiadm -m discovery -t sendtargets -p 10.67.49.8:3260

CI/CD Build Job Pipeline Setup

Refer to the doc here.

On this page:

• Security overview

• Infra security

Introduction

Security is always a difficult subject to approach either by lack of experience; or by the fact you should know when the level of security is right for what you have to secure.

Security is a major concern when it comes to government systems and infra. As an architect, we can consider that working with technically educated people (engineers, experts) and tools (systems, frameworks, IDE) should prevent key VAPT issues.

However, it’s quite difficult to avoid, a certain infatuation from different categories of people to try to hack the systems.

Infra Security

1. Update to the latest version

There aren’t only bug fixes in each release but also new security measures to require advantage of them, we recommend working with the newest stable version.

Updates and support could also be harder than the new features offered in releases, so plan your updates a minimum of once a quarter. Significantly simplified updates can utilize the providers of managed Kubernetes solutions.

2. Enable role-based access control (RBAC)

Use RBAC (Role-Based Access Control) to regulate who can access and what rights they need. Usually, RBAC is enabled by default in Kubernetes version 1.6 and later (or later for a few providers), but if you’ve got been updated since then and didn’t change the configuration, you ought to double-check your settings.

However, enabling RBAC isn’t enough — it still must be used effectively. within the general case, the rights to the whole cluster (cluster-wide) should be avoided, giving preference to rights in certain namespaces. Avoid giving someone cluster administrator privileges even for debugging — it’s much safer to grant rights only necessary and from time to time.

If the appliance requires access to the Kubernetes API, create separate service accounts. and provides them with the minimum set of rights required for every use case. This approach is far better than giving an excessive amount of privilege to the default account within the namespace.

3. Use namespaces to set security boundaries

Creating separate namespaces is vital because of the first level of component isolation. it’s much easier to regulate security settings — for instance, network policies — when different types of workloads are deployed in separate namespaces.

To get in-depth knowledge of Kubernetes, enrol for a live demo on the Kubernetes course.

4. Separate sensitive workloads

A good practice to limit the potential consequences of compromise is to run workloads with sensitive data on a fanatical set of machines. This approach reduces the risk of a less secure application accessing the application with sensitive data running in the same container executable environment or on the same host.

For example, a Kubelet of a compromised node usually has access to the contents of secrets only if they are mounted on pods that are scheduled to be executed on the same node. If important secrets are often found on multiple cluster nodes, the attacker will have more opportunities to urge them.

Separation can be done using node pools (in the cloud or on-premises), as well as Kubernetes controlling mechanisms, such as namespaces, taints, tolerations, and others.

5. Protect access to cloud service metadata

Sensitive metadata — for instance, Kubelet administrative credentials, are often stolen or used with malicious intent to escalate privileges during a cluster. For example, a recent find within Shopify’s bug bounty showed in detail how a user could exceed authority by receiving metadata from a cloud provider using specially generated data for one of the microservices.

The GKE metadata concealment function changes the mechanism for deploying the cluster in such how that avoids such a drag. And we recommend using it until a permanent solution is implemented.

6. Create and define cluster network policies

Network Policies — allow you to control access to the network in and out of containerized applications. To use them, you must have a network provider with support for such a resource. For managed Kubernetes solution providers such as Google Kubernetes Engine (GKE), support will need to be enabled.

Once everything is ready, start with simple default network policies — for example, blocking (by default) traffic from other namespaces.

7. Set the Pod Security Policy for the cluster

Pod Security Policy sets the default values ​​used to start workloads in the cluster. Consider defining a policy and enabling the Pod Security Policy admission controller: the instructions for these steps vary depending on the cloud provider or deployment model used.

In the beginning, you might want to disable the NET_RAW capability in containers to protect yourself from certain types of spoofing attacks.

8. Work on node security

To improve host security, you can follow these steps:

• Ensure that the host is securely and correctly configured. One way is CIS Benchmarks; Many products have an auto checker that automatically checks the system for compliance with these standards.

• Monitor the network availability of important ports. Ensure that the network is blocking access to the ports used by Kubelet, including 10250 and 10255. Consider restricting access to the Kubernetes API server — with the exception of trusted networks. In clusters that did not require authentication and authorization in the Kubelet API, attackers used to access to such ports to launch cryptocurrency miners.

• Minimize administrative access to Kubernetes hosts Access to cluster nodes should in principle be limited: for debugging and solving other problems, as a rule, you can do without direct access to the node.

9. Enable Audit Logging

Make sure that audit logs are enabled and that you are monitoring for the occurrence of unusual or unwanted API calls in them, especially in the context of any authorization failures — such entries will have a message with the “Forbidden” status. Authorization failures can mean that an attacker is trying to take advantage of the credentials obtained.

Managed solution providers (including GKE) provide access to this data in their interfaces and can help you set up notifications in case of authorization failures.

Conclusion

Follow these guidelines for a more secure Kubernetes cluster. Remember that even after the cluster is configured securely, you need to ensure security in other aspects of the configuration and operation of containers. To improve the security of the technology stack, study the tools that provide a central system for managing deployed containers, constantly monitoring and protecting containers and cloud-native applications.

Topics covered:

• Introduction to SDC

• Automating the deployment process

• Installation steps

Overview

Running Kubernetes on-premise gives a cloud-native experience on SDC when it comes to deploying DIGIT.

Whether States have their own on-premise data centre or have decided to forego the various managed cloud solutions, there are a few things one should know when getting started with on-premise K8s.

One should be familiar with Kubernetes and the control plane consists of the Kube-apiserver, Kube-scheduler, Kube-controller-manager and an ETCD datastore. For managed cloud solutions like Google’s Kubernetes Engine (GKE) or Azure’s Kubernetes Service (AKS), it also includes the cloud-controller-manager. This is the component that connects the cluster to external cloud services to provide networking, storage, authentication, and other support features.

To successfully deploy a bespoke Kubernetes cluster and achieve a cloud-like experience on SDC, one needs to replicate all the same features you get with a managed solution. At a high level, this means that we probably want to:

• Automate the deployment process

• Choose a networking solution

• Choose a right storage solution

• Handle security and authentication

The subsequent sections look at each of these challenges individually, and provide enough of a context required to help in getting started.

Automating The Deployment Process

Using a tool like Ansible can make deploying Kubernetes clusters on-premise trivial.

When deciding to manage your own Kubernetes clusters, we need to set up a few proofs-of-concept (PoC) clusters to learn how everything works, perform performance and conformance tests, and try out different configuration options.

After this phase, automating the deployment process is an important if not necessary step to ensure consistency across any clusters you build. For this, you have a few options, but the most popular are:

• kubeadm: a low-level tool that helps you bootstrap a minimum viable Kubernetes cluster that conforms to best practices

• kubespray: an Ansible playbook that helps deploy production-ready clusters

If you already using Ansible, Kubespray is a great option, otherwise, we recommend writing automation around Kubeadm using your preferred playbook tool after using it a few times. This will also increase your confidence and knowledge of Kubernetes.

Installation Steps

Topics covered:

• Steps to Bootstrap DIGIT

• Assess deployment

• Destroy the cluster

Overview

Post-deployment, the application can now be accessed from the configured domain. This page provides the bootstrapping steps.

Bootstrapping Steps

To try out employee login, let us create a sample tenant, city, and user to log in and assign the LME employee role through the seed script.

1. Perform the kubectl port-forwarding of the egov-user service running from the Kubernetes cluster to your localhost. This provides access to egov-user service and allows users to interact with the API directly.

kubectl port-forward svc/egov-user 8080:8080 -n egov
Output:
Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080

2. Seed the sample data

• Ensure the Postman is installed to run the following seed data API. if not, Install postman on your local machine.

• Import the following Postman collection into the Postman and run it. This contains the seed data that enables sample test users and localisation data.

  DIGIT Bootstrap

1. Execute the below commands to test your local machine's Kubernetes operations through kubectl.

#get the pods running in the cluster
kubectl get pods -n egov

Output:

NAME                                                    READY   STATUS    RESTARTS   AGE
citizen-79cf89659c-8glf4                                1/1     Running   0          14d
egov-accesscontrol-78b78dddb9-tfbkf                     1/1     Running   0          21d
egov-enc-service-574cd7b5b5-hmhh4                       1/1     Running   0          36d
egov-idgen-84954b565b-xsnqj                             1/1     Running   0          45d
egov-indexer-5f5fbb6f4b-58rtm                           1/1     Running   0          43d
egov-localization-6cc5977bb9-gm7f9                      1/1     Running   0          27d
egov-mdms-service-65d6d65d8c-t85d9                      1/1     Running   0          21d
egov-user-6676968d76-8n6t6                              1/1     Running   0          29d
egov-workflow-v2-5cdb96bcf5-dcgmf                       1/1     Running   0          36d
employee-749464fbfb-tptlh                               1/1     Running   0          14d
nginx-ingress-controller-b9678869c-mkslb                1/1     Running   0          49d
pgr-services-b9f4ffdbf-5h5kd                            1/1     Running   0          38d
zuul-788bf8cd8b-9nxfl                                   1/1     Running   0          41d


#Delete the pods so that it gets restarted automatically

kubectl delete pods zuul-788bf8cd8b-9nxfl egov-workflow-v2-5cdb96bcf5-dcgmf pgr-services-b9f4ffdbf-5h5kd -n egov

Output:

pod "zuul-788bf8cd8b-9nxfl" deleted
pod "egov-workflow-v2-5cdb96bcf5-dcgmf" deleted
pod "pgr-services-b9f4ffdbf-5h5kd" deleted

You have successfully completed the DIGIT Infra, deployment setup and installed a DIGIT - PGR module.

Use the below link in the browser -

digit.try.com/employee

Use the below credentials to login into the complaint section

• Username: GRO

• Password: eGov@4321

• City: CITYA

DIGIT Deployment Assessment

By now we have successfully completed the DIGIT setup on the cloud, use the URL you mentioned in your env.yaml.

Eg: https://mysetup.digit.org and create a grievance to ensure the PGR module deployed is working fine. Refer to the below product documentation for the steps.

Credentials:

1. Citizen: You can use your default mobile number (9999999999) to sign in using the default Mobile OTP 123456.

2. Employee: Username: GRO and password: eGov@4321

Post grievance creation and assignment of the same to LME, capture the screenshot of the same and share it to ensure your setup is working fine. Post validation, the PGR functionality shares the API response of the following request to assess the correctness of successful DIGIT PGR Deployment.

Destroy Cluster

Follow the steps below to clean up the DIGIT setup, if required. This will delete the entire cluster and other cloud resources that were provisioned for the DIGIT Setup.

Run the command below to destroy previously-created infrastructure using Terraform -

1. ELB is not deployed via Terraform. ELB has created at deployment time by the setup of Kubernetes Ingress. This has to be deleted manually by deleting the ingress service.

   • kubectl delete deployment nginx-ingress-controller -n <namespace>

   • kubectl delete svc nginx-ingress-controller -n <namespace>

   Note: Namespace can be either provided by egov or Jenkins.

2. Delete S3 buckets manually from the AWS console and also verify if the ELB got deleted.

   • In case the ELB is not deleted, you need to delete ELB from the AWS console.

3. Run terraform destroy.

cd DIGIT-DevOps/infra-as-code/terraform/my-digit-eks
terraform destroy

Conclusion

All done, we have successfully created infra on the cloud, deployed DIGIT, bootstrapped DIGIT, performed a transaction on PGR and finally destroyed the cluster.

Topics covered:

• Brief introduction to terraform infrastructure as code tool

• Cloud resources required

• Understand terraform script

• Deployment steps

• Execute terraform: provision infrastructure resources

• Test Kubernetes cluster

Overview

Terraform is an open-source infrastructure as code (IaC) software tool that allows DevOps engineers to programmatically provision the physical resources an application requires to run.

Infrastructure as code is an IT practice that manages an application's underlying IT infrastructure through programming. This approach to resource allocation allows developers to logically manage, monitor and provision resources -- as opposed to requiring that an operations team manually configure each required resource.

Terraform users define and enforce infrastructure configurations by using a JSON-like configuration language called HCL (HashiCorp Configuration Language). HCL's simple syntax makes it easy for DevOps teams to provision and re-provision infrastructure across multiple clouds and on-premises data centres.

Cloud Resources Required

Before we provision the cloud resources, we need to understand and be sure about what resources need to be provisioned by Terraform to deploy DIGIT. The following picture shows the various key components. (AKS, Node Pools, Postgres DB, Volumes, Load Balancer)

Understand Terraform Script

• Ideally, one would write the terraform script from scratch using this doc.

• Here we have already written the terraform script that one can reuse/leverage that provisions the production-grade DIGIT Infra and can be customized with the user-specific configuration.

Deployment Steps

1. Clone the following DIGIT-DevOps where we have all the sample terraform scripts available for you to leverage.

git clone --branch release https://github.com/egovernments/DIGIT-DevOps.git
cd DIGIT-DevOps
git checkout azure-install
code .
cd infra-as-code/terraform/sample-azure

### You'll see the following file structure 

├── sample-azure
│       ├── main.tf
│       ├── outputs.tf
│       ├── providers.tf
│       ├── remote-state
│       │            └── main.tf
│       └── variables.tf
└── modules
     ├── db
     │    ├── aws
     │    |    ├── main.tf
     │    │    ├── outputs.tf
     │    |    └── variables.tf
     │    └── azure
     │         ├── main.tf
     │         └── variables.tf
     │── kubernetes
     │    └── azure
     │         │   	
     │         ├── main.tf
     │         ├── outputs.tf
     |         └── variables.tf
     └── storage
    	  └── azure
          ├── main.tf
     	  ├── outputs.tf
     	  └── variables.tf       

2. Change the main.tf according to your requirements.

provider "azurerm" {
  # whilst the `version` attribute is optional, we recommend pinning to a given version of the Provider
  subscription_id  = "${var.subscription_id}"
  tenant_id        = "${var.tenant_id}" 
  client_id        = "${var.client_id}"
  client_secret    = "${var.client_secret}"
  features {}
  skip_provider_registration = true
}

terraform {
  backend "azurerm" {
      resource_group_name  = "demo-azure-rg-terraform"
      storage_account_name = "tfstate7v0qn"
      container_name       = "demo-azure-container"
      key                  = "terraform.tfstate"
  }
}

resource "azurerm_virtual_network" "example" {
  name                = "${var.resource_group}-virtual-network"
  address_space       = ["10.0.0.0/16"]
  location            = "${var.location}"
  resource_group_name = "${var.resource_group}"
}

resource "azurerm_subnet" "aks" {
  name                 = "${var.resource_group}-aks-subnet"
  resource_group_name  = "${var.resource_group}"
  virtual_network_name = azurerm_virtual_network.example.name
  address_prefixes     = ["10.0.1.0/24"]
}

resource "azurerm_subnet" "postgres" {
  name                 = "${var.resource_group}-postgres-subnet"
  resource_group_name  = "${var.resource_group}"
  virtual_network_name = azurerm_virtual_network.example.name
  address_prefixes     = ["10.0.2.0/24"]
  service_endpoints    = ["Microsoft.Storage"]

  delegation {
    name = "fs"

    service_delegation {
      name = "Microsoft.DBforPostgreSQL/flexibleServers"

      actions = [
        "Microsoft.Network/virtualNetworks/subnets/join/action",
      ]
    }
  }
}

resource "azurerm_network_security_group" "aks_nsg" {
  name                = "aks-nsg"
  location            = "${var.location}"
  resource_group_name = "${var.resource_group}"
}

resource "azurerm_network_security_group" "db_nsg" {
  name                = "db-nsg"
  location            = "${var.location}"
  resource_group_name = "${var.resource_group}"
}

resource "azurerm_network_security_rule" "example" {
  name                        = "aks_rule1"
  priority                    = 100
  direction                   = "Inbound"
  access                      = "Allow"
  protocol                    = "Tcp"
  source_port_range           = "*"
  destination_port_range      = "80"
  source_address_prefix       = "Internet"
  destination_address_prefix  = "*"
  resource_group_name         = "${var.resource_group}"
  network_security_group_name = azurerm_network_security_group.aks_nsg.name
}

resource "azurerm_network_security_rule" "example2" {
  name                        = "aks_rule2"
  priority                    = 500
  direction                   = "Inbound"
  access                      = "Allow"
  protocol                    = "Tcp"
  source_port_range           = "*"
  destination_port_range      = "443"
  source_address_prefix       = "Internet"
  destination_address_prefix  = "*"
  resource_group_name         = "${var.resource_group}"
  network_security_group_name = azurerm_network_security_group.aks_nsg.name
}

module "kubernetes" {
  source = "../modules/kubernetes/azure"
  environment = "${var.environment}"
  name = "${var.environment}"
  location = "${var.location}"
  resource_group = "${var.resource_group}"
  client_id =  "${var.client_id}"
  client_secret = "${var.client_secret}"
  vm_size = "Standard_B4ms"
  ssh_public_key = "${var.environment}"
  node_count = 5
  network_security_group_id = azurerm_network_security_group.aks_nsg.id
  subnet_id = azurerm_subnet.aks.id

}

module "postgres-db" {
  source = "../modules/db/azure"
  resource_group = "${var.resource_group}"  
  location = "${var.location}"
  sku_tier = "B_Standard_B2ms"
  storage_mb = "65536"
  backup_retention_days = "7"
  administrator_login = "${var.db_user}"
  administrator_password = "${var.db_password}"
  db_version = "${var.db_version}" 
  subnet_id = azurerm_subnet.postgres.id
  network_security_group_id = azurerm_network_security_group.db_nsg.id
  virtual_network_id = azurerm_virtual_network.example.id
}

3. Declare the variables in variables.tf

variable "environment" {
    default = "demo-azure-terraform"
}
variable "resource_group" {
    default = "demo-azure-rg-terraform"
}

variable "location" {
    default = "South India"
}

variable "db_version" {
    default = "13"
}

variable "db_user" {
    default = "azurepostgres"
}

variable "db_password"{}

variable "subscription_id" {
    default = "8d8fadc3-5236-461c-a0ee-00836a75c4d1"
}

variable "tenant_id" {
    default = "d1127efd-6841-40e9-8c06-300f458a7c00"
}

variable "client_id" {
}

variable "client_secret" {
}

Save the file and exit the editor.

4. Create a Terraform output file (output.tf) and paste the following code into the file.

output "azurerm_postgresql_flexible_server" {
  value = module.postgres-db.azurerm_postgresql_flexible_server
}

output "postgresql_flexible_server_database_name" {
  value = module.postgres-db.postgresql_flexible_server_database_name
}

output "resource_group"{
  value = var.resource_group
}

output "db_user" {
  value = var.db_user
}

output "cluster_name" {
  value = var.environment
}

Terraform Execution: Infrastructure Resources Provisioning

Once you have finished declaring the resources, you can deploy all resources.

1. terraform init: command is used to initialize a working directory containing Terraform configuration files.

2. terraform plan: command creates an execution plan, which lets you preview the changes that Terraform plans to make to your infrastructure.

3. terraform apply: command executes the actions proposed in a Terraform plan to create or update infrastructure.

After the complete creation, you can see resources in your Azure account.

Now we know what the terraform script does, the resources graph that it provisions and what custom values should be given with respect to your environment. The next step is to begin to run the terraform scripts to provision infra required to Deploy DIGIT on Azure.

1. Use the CD command to move into the following directory run the following commands 1-by-1 and watch the output closely.

##### Create the DIGIT Infra #####

az login
##### using above command you can get subscription id and tenant id

az ad sp create-for-rbac --name <sp_name> \
             --role owner \
             --scopes /subscriptions/<subscription_id>
##### Using above command you are creating client-id and client-secret

cd DIGIT-DevOps/infra-as-code/terraform/sample-azure/remote-state

terraform init

terraform plan

terraform apply 

cd ..

terraform init

terraform plan

terraform apply

Test Kubernetes Cluster

The Kubernetes tools can be used to verify the newly created cluster.

1. Once Terraform Apply execution is complete, it generates the Kubernetes configuration file or you can get it from Terraform state.

2. Use the below command to get kubeconfig. It will automatically store your kubeconfig in .kube folder.

az aks get-credentials --resource-group <resource_group_name> --name <cluster_name>

3. Verify the health of the cluster.

kubectl get nodes -A

The details of the worker nodes should reflect the status as Ready for All.

Topics covered:

• Steps to install DIGIT on Azure

Overview

Azure Kubernetes Service (AKS) manages your hosted Kubernetes environment. AKS allows you to deploy and manage containerized applications without container orchestration expertise. AKS also enables you to do many common maintenance operations without taking your app offline. These operations include provisioning, upgrading, and scaling resources on demand.

Installation Steps

Topics covered:

• More about the concept of cicd-as-service

• Integrating services to CI/CD

• Continuous integration

• Continuous deployment

Overview

Since there are many DIGIT services and the development code is part of various git repos, one needs to understand the concept of cicd-as-service which is open-sourced. This page guides you through the process of creating a CI/CD pipeline.

Integrating Services To CI/CD

The initial steps for integrating any new service/app to the CI/CD are discussed below.

Once the desired service is ready for integration: decide the service name, type of service, and if DB migration is required or not. While you commit the source code of the service to the git repository, the following file should be added with the relevant details which are mentioned below:

Build-config.yml – It is present under the build directory in the repository

https://raw.githubusercontent.com/egovernments/DIGIT-OSS/master/build/build-config.yml

This file contains the below details used for creating the automated Jenkins pipeline job for the newly created service.

config:
 -   name: < Name of the job, foo/bar would create job named bar inside folder foo >
     build:
     - work-dir: < Working directory of the app to be built >
       dockerfile: < Path to the dockerfile, optional, assumes dockerfile in working directory if not provided >
       image-name: < Docker image name should be similar to your helm chart name >

While integrating a new service/app, the above content needs to be added to the build-config.yml file of that app repository. For example: to onboard a new service called egov-test, the build-config.yml should be added as mentioned below.

config:  
- name: builds/DIGIT-OSS/core-services/egov-test     
  build:     
  - work-dir: egov-test      
    dockerfile: build/maven/Dockerfile       
    image-name: egov-test

If a job requires multiple images to be created (DB Migration) then it should be added as below,

config:   
- name: builds/DIGIT-OSS/core-services/egov-test     
  build:     
  - work-dir: egov-test       
    dockerfile: build/maven/Dockerfile       
    image-name: egov-test     
  - work-dir: egov-test/src/main/resources/db       
    dockerfile: egov-test/src/main/resources/db/Dockerfile       
    image-name: egov-test-db

Note - If a new repository is created then the build-config.yml is created under the build folder and the config values are added to it.

The git repository URL is then added to the Job Builder parameters

When the Jenkins Job => job builder is executed, the CI Pipeline gets created automatically based on the above details in build-config.yml. Eg: egov-test job is created in the builds/DIGIT-OSS/core-services folder in Jenkins since the “build-config is edited under core-services” And it should be the “master” branch. Once the pipeline job is created, it can be executed for any feature branch with build parameters - specifying the branch to be built (master or feature branch).

As a result of the pipeline execution, the respective app/service docker image is built and pushed to the Docker repository.

On repo provide read-only access to GitHub users (created while ci/cd deployment)

Continuous Integration (CI)

The Jenkins CI pipeline is configured and managed 'as code'.

• Job Builder – Job Builder is a Generic Jenkins job which creates the Jenkins pipeline automatically which is then used to build the application, create the docker image of it and push the image to the Docker repository. The Job Builder job requires the git repository URL as a parameter. It clones the respective git repository and reads the build/build-config.yml file for each git repository and uses it to create the service build job.

Check and ‌add your repo ssh URL in ci.yaml​‌‌

If the git repository ssh URL is available, build the Job-Builder Job.

If the git repository URL is not available, check and add the same team.

Continuous Deployment (CD)

The services are deployed and managed on a Kubernetes cluster in cloud platforms like AWS, Azure, GCP, OpenStack, etc. Here, we use helm charts to manage and generate the Kubernetes manifest files and use them for further deployment to the respective Kubernetes cluster. Each service is created as charts which have the below-mentioned files.

billing-service/   # Directory – name of the service/app
    |
    |-- Chart.yaml  # A YAML file containing information about the chart
    |-- LICENSE     # OPTIONAL: A plain text file containing the license for the chart
    |-- README.md   # OPTIONAL: A human-readable README file
    |-- values.yaml # The default configuration values for this chart
    |-- templates/  # A directory of templates that, when combined with values, will generate valid Kubernetes manifest files.
            |
            | -- deployment.yaml
            | -- service.yaml 
            | -- ingress.yaml 

Note: The steps below are only for the introduction and implementation of new services.

• To deploy a new service, you need to create a new helm chart for it( refer to the above example). The chart should be created under the charts/helm directory in the DIGIT-DevOps repository.

• If you are going to introduce a new module with the help of multiple services, we suggest you create a new Directory with your module name.

Example.:-

charts/helm/<new_module_name>/<new_service_chart_name-1>
                   |-------- /<new_service_chart_name-2>
                   |-------- /<new_service_chart_name-3>

• You can refer to the existing helm chart structure here

This chart can also be modified further based on user requirements.

The deployment of manifests to the Kubernetes cluster is made very simple and easy. There are Jenkins Jobs for each state and are environment-specific. We need to provide the image name or the service name for the respective Jenkins deployment job.

‌The deployment Jenkins job internally performs the following operations:

• Reads the image name or the service name given and finds the chart that is specific to it.

• Generates the Kubernetes manifests files from the chart using the helm template engine.

• Execute the deployment manifest with the specified docker image(s) to the Kubernetes cluster.

Topics covered:

Overview

helps you build a graph of all your resources and parallelizes the creation or modification of any non-dependent resources. Thus, Terraform builds infrastructure as efficiently as possible while providing the operators with clear insight into the dependencies on the infrastructure.

Pre-requisites

2. the repos below to your GitHub Organization account

4. (version 1.13.X)

8. ​ with admin access to provision EKS Service. Try subscribing to a free AWS account to learn the basics. There is a limit on . This demo requires a commercial subscription to the EKS service. The cost for a one or two days trial might range between Rs 500-1000. (Note: Post the demo, for the internal folks, eGov will provide a 2-3 hrs time-bound access to eGov's AWS account based on the request and the available number of slots per day).

9. Install on your local machine to interact with the Kubernetes cluster.

10. Install to help package the services along with the configurations, environment, secrets, etc into a .

11. Install version (0.14.10) for the Infra-as-code (IaC) to provision cloud resources as code and with desired resource graph. It also helps destroy the cluster in one go.

12. ​ on your local machine so that you can use AWS CLI commands to provision and manage the cloud resources on your account.

13. Install to help authenticate your connection from your local machine and deploy DIGIT services.

14. Use the credentials provided for the Terraform () to connect to the AWS account and provision the cloud resources.

    ​

    • You will receive a Secret Access Key and Access Key ID. Save the keys.

    • Open the terminal and run the command given below. The AWS CLI is already installed and the credentials are saved. (Provide the credentials, leave the region and output format blank).

    • The above creates the following file on your machine as /Users/.aws/credentials.

Setup Details

Before we provision the cloud resources, we need to understand and be sure about what resources need to be provisioned by terraform to deploy CI/CD.

The following is the resource graph that we are going to provision using terraform in a standard way so that every time and for every environment, the infra is the same.

• EKS Control Plane (Kubernetes master)

• Work node group (VMs with the estimated number of vCPUs, Memory)

• EBS Volumes (Persistent volumes)

• VPCs (Private networks)

• Users to access, deploy and read-only

Terraform Script Resource Graph

• Here we have already written the terraform script that provisions the production-grade DIGIT Infra and can be customized with the specified configuration.

Here, you will find the main.tf under each of the modules that have the provisioning definition for resources like EKS cluster, storage, etc. All these are modularized and react as per the customized options provided.

Example:

1. VPC Resources -

   • VPC

   • Subnets

   • Internet Gateway

   • Route Table

2. EKS Cluster Resources -

   • IAM Role to allow EKS service to manage other AWS services

   • EC2 Security Group to allow networking traffic with the EKS cluster

   • EKS Cluster

3. EKS Worker Nodes Resources -

   • IAM role allowing Kubernetes actions to access other AWS services

   • EC2 Security Group to allow networking traffic

   • Data source to fetch the latest EKS worker AMI

   • AutoScaling launch configuration to configure worker instances

   • AutoScaling group to launch worker instances

4. Storage Module -

   • Configuration in this directory creates EBS volume and attaches it together.

The following main.tf with create s3 bucket to store all the states of the execution to keep track.

The following main.tf contains the detailed resource definitions that need to be provisioned.

Dir: DIGIT-DevOps/Infra-as-code/terraform/egov-cicd

Custom Variables/Configurations

Define your configurations in variables.tf and provide the environment-specific cloud requirements. The same terraform template can be used to customize the configurations.

Following are the values that you need to mention in the following files. The blank ones will prompt for inputs during execution.

Run Terraform Scripts

We have covered what the terraform script does, the resources graph that it provisions and what custom values should be given with respect to the selected environment.

Use the 'cd' command to change to the following directory and run the following commands. Check the output.

After successful execution, the following resources get created and can be verified by the command "terraform output".

• s3 bucket: to store terraform state

• Network: VPC, security groups

• IAM users auth: using keybase to create admin, deployer, the user

• EKS cluster: with master(s) & worker node(s).

• Storage(s): for es-master, es-data-v1, es-master-infra, es-data-infra-v1, zookeeper, kafka, kafka-infra.

Finally, verify that you are able to connect to the cluster by running the command below:

Whola! All set and now you can Deploy Jenkins

Jenkins Deployment

Post infra setup (Kubernetes Cluster), we start with deploying the Jenkins and kaniko-cache-warmer.

Pre-requisites

• Sub domain to expose CI/CD URL

• • Under Authorization callback URL enter the below url ie (Replace <domain_name> with your domain) https://<domain_name>/securityRealm/finishLogin

• SSL certificate for the sub-domain

• Make sure earlier created github users have read-only access to the forked DIGIT-DevOps and CIOps repos.

• SSL certificate for the sub-domain.

Prepare CIOps Configs

• Remove the below env:
• Remove the below volumeMounts:
• Remove the below volume:

Jenkins is launched. You can access the same through your sub-domain configured in ci.yaml.

Topics covered: