Topics covered:

• AWS deployment basics

• Pre-reads before deployment

• Steps to install

Overview

Amazon Elastic Kubernetes Service (EKS) is an AWS service for deploying, managing, and scaling distributed and containerized workloads. With EKS, you can easily provision a cluster on AWS using Terraform, which automates the process. Then, deploy the DIGIT services configuration using Helm.

Pre-reads

• Know about EKS: https://www.youtube.com/watch?v=SsUnPWp5ilc

• Know what is terraform: https://youtu.be/h970ZBgKINg

Installation Steps

If you have any questions please write to us.

Make sure to use the appropriate discussion category and labels to address the issues better.

Topics covered:

• ​

Pre-requisites

1. ​ with admin access to provision infrastructure. You will need a paid subscription to the AWS.

2. Install (any version) on the local machine - it helps interact with the Kubernetes cluster.

3. Install - this helps package the services, configurations, environments, secrets, etc into . Verify that the installed version of helm is equal to 3.0 or higher.

1. Install - to run different versions of Terraform on the machine. supports multiple terraform versions on the same machine and you can toggle between the desired versions.

5. Run tfswitch and it will show a list of terraform versions. Scroll down and select version (0.14.10) for the (IaC) to provision cloud resources as code. This provides the desired resource graph and helps destroy the cluster in one go.​

Topics covered:

Overview

manages your hosted Kubernetes environment. AKS allows you to deploy and manage containerized applications without container orchestration expertise. AKS also enables you to do many common maintenance operations without taking your app offline. These operations include provisioning, upgrading, and scaling resources on demand.

Installation Steps

Topics covered:

Overview

Follow the details below to set up your AWS account before you proceed with the DIGIT deployment.

Get AWS Access

Once the command line access is configured, everything is set to proceed with the terraform to provision the DIGIT Infra-as-code.

Topics covered:

• Pre-reads for deploying on Azure

• Pre-requisites for deploying on Azure

Overview

The Azure Kubernetes Service (AKS) is one of the Azure services for deploying, managing and scaling any distributed and containerized workloads, here we can provision the AKS cluster on Azure from the ground up using terraform (infra-as-code) and then deploy the DIGIT platform services as config-as-code using Helm.

Pre-reads

• Know about AKS: https://www.youtube.com/watch?v=i5aALhhXDwc&ab_channel=DevOpsCoach​

• Know what is terraform: https://youtu.be/h970ZBgKINg

Pre-requisites​

1. Azure subscription: If you don't have an Azure subscription, create a free account before you begin.

2. Install Azure CLI

3. Configure Terraform: Follow the directions in the article, Terraform and configure access to Azure

4. Azure service principal: Follow the directions in the Create the service principal section in the article, Create an Azure service principal with Azure CLI. Take note of the values for the appId, displayName, password, and tenant.

5. Install kubectl on your local machine which helps you interact with the Kubernetes cluster.

6. Install Helm that helps you package the services along with the configurations, environments, secrets, etc into Kubernetes manifests.

Annexures:

• Kubernetes Fundamentals

• AKS Fundamentals

• Manually Provision AKS

Topics covered:

• Introduction to SDC

• Automating the deployment process

• Installation steps

Overview

Running Kubernetes on-premise gives a cloud-native experience on SDC when it comes to deploying DIGIT.

Whether States have their own on-premise data centre or have decided to forego the various managed cloud solutions, there are a few things one should know when getting started with on-premise K8s.

One should be familiar with Kubernetes and the control plane consists of the Kube-apiserver, Kube-scheduler, Kube-controller-manager and an ETCD datastore. For managed cloud solutions like Google’s Kubernetes Engine (GKE) or Azure’s Kubernetes Service (AKS), it also includes the cloud-controller-manager. This is the component that connects the cluster to external cloud services to provide networking, storage, authentication, and other support features.

To successfully deploy a bespoke Kubernetes cluster and achieve a cloud-like experience on SDC, one needs to replicate all the same features you get with a managed solution. At a high level, this means that we probably want to:

• Automate the deployment process

• Choose a networking solution

• Choose a right storage solution

• Handle security and authentication

The subsequent sections look at each of these challenges individually, and provide enough of a context required to help in getting started.

Automating The Deployment Process

Using a tool like Ansible can make deploying Kubernetes clusters on-premise trivial.

When deciding to manage your own Kubernetes clusters, we need to set up a few proofs-of-concept (PoC) clusters to learn how everything works, perform performance and conformance tests, and try out different configuration options.

After this phase, automating the deployment process is an important if not necessary step to ensure consistency across any clusters you build. For this, you have a few options, but the most popular are:

• kubeadm: a low-level tool that helps you bootstrap a minimum viable Kubernetes cluster that conforms to best practices

• kubespray: an Ansible playbook that helps deploy production-ready clusters

If you already using Ansible, Kubespray is a great option, otherwise, we recommend writing automation around Kubeadm using your preferred playbook tool after using it a few times. This will also increase your confidence and knowledge of Kubernetes.

Installation Steps

Topics Covered:

• List of pre-reads for a better understanding

• Choose your cloud for setup

Basics

DIGIT can be deployed on a public cloud like AWS, Azure or a private cloud.

Pre-reads

• Learn the basics of Kubernetes: https://www.youtube.com/watch?v=PH-2FfFD2PU&t=3s

• Learn the basics of kubectl commands

Choose Your Cloud

Choose your cloud and follow the instructions to set up a Kubernetes cluster before deploying.

Overview

The image below illustrates the multiple components deployed. These include the EKS, Worker Nodes, Postgres DB, EBS Volumes, and Load Balancer.

Steps

1. Clone the DIGIT-DevOps repository:

git clone https://github.com/egovernments/DIGIT-DevOps.git

1. Navigate to the cloned repository and checkout the release-1.28-Kubernetes branch:

cd DIGIT-DevOps 
git checkout release-1.28-kubernetes

1. Check if the correct credentials are configured using the command below. Refer to the attached doc to setup AWS Account on the local machine.

aws configure list

Make sure that the above command reflects the set AWS credentials. Proceed once the details are confirmed. (If the credentials are not set follow Step 2 Setup AWS account )

1. Choose either method below to generate SSH key pairs

   • a. Use an online website (not recommended in a production setup. To be only used for demo setups): https://8gwifi.org/sshfunctions.jsp

   • b. Use openssl:

openssl genpkey -algorithm RSA -out private_key.pem openssl rsa -pubout -in private_key.pem -out public_key.pem

1. Add the public key to your GitHub account.

2. Open input.yaml file in vscode. Use the below code to open it in VS code:

   code infra-as-code/terraform/sample-aws/input.yaml

   If the command does not work, open the file in VS code manually. Once the file is open, fill in the inputs. (If you are not using vscode, open it in any editor of your choice).

3. Fill in the inputs as per the regex mentioned in the comments.

4. Go to infra-as-code/terraform/sample-aws and run init.go script to enrich different files based on input.yaml.

cd infra-as-code/terraform/sample-aws 
go run ../scripts/init.go

Terraform Execution: Infrastructure Resources Provisioning

Once we are complete declaring the resources, begin with deploying all resources.

Run the terraform scripts to provision infra required to Deploy DIGIT on AWS.

1. CD (change directory) to the following directory and run the below commands to create the remote state.

### Create the remote-state first, remember that the state name should be unique
### You may need to use sudo incase you don't have requisite permission

cd remote-state

terraform init

terraform plan

terraform apply

Once the remote state is created, it is time to provision DIGIT infra. Run the below commands:

### Once the remote state is created, you can create the DIGIT Infra

cd ..

terraform init

terraform plan

terraform apply

Important:

• DB password is asked for in the application stage. Remember the password you have provided. It should be at least 8 characters long. Otherwise, RDS provisioning will fail.

• The output of the apply command will be displayed on the console. Store this in a file somewhere. Values from this file will be used in the next step of deployment.

2. Use this link to get the kubeconfig from EKS for the cluster. The region code is the default region provided in the availability zones in variables.tf. For example - ap-south-1. EKS cluster name also should've been filled in variables.tf.

aws sts get-caller-identity

# Run the below command and give the respective region-code and the cluster name
aws eks --region <region-code> update-kubeconfig --name <cluster_name>

3. Verify that you can connect to the cluster by running the following command

kubectl config use-context <cluster_name>

kubectl get nodes

NAME                                             STATUS AGE   VERSION               OS-Image           
ip-192-168-xx-1.ap-south-1.compute.internal   Ready  45d   v1.15.10-eks-bac369   Amazon Linux 2   
ip-192-168-xx-2.ap-south-1.compute.internal   Ready  45d   v1.15.10-eks-bac369   Amazon Linux 2   
ip-192-168-xx-3.ap-south-1.compute.internal   Ready  45d   v1.15.10-eks-bac369   Amazon Linux 2   
ip-192-168-xx-4.ap-south-1.compute.internal   Ready  45d   v1.15.10-eks-bac369   Amazon Linux 2 

At this point, your basic infra has been provisioned.

Note: Refer to the DIGIT deployment documentation to deploy DIGIT services.

Destroying Infra

To destroy the previously created infrastructure with Terraform, run the command below:

1. ELB is not deployed via Terraform. ELB was created at deployment time by the setup of Kubernetes Ingress. This has to be deleted manually by deleting the ingress service.

   • kubectl delete deployment nginx-ingress-controller -n <namespace>

   • kubectl delete svc nginx-ingress-controller -n <namespace>

     Note: Namespace can be either egov or jenkins.

2. Delete S3 buckets manually from the AWS console and verify if ELB got deleted.

   • In case of if ELB is not deleted, you need to delete ELB from the AWS console.

3. Run terraform destroy.

Topics covered:

Overview

Kubespray is a composition of playbooks, , provisioning tools, and domain knowledge for generic OS/Kubernetes cluster configuration management tasks. Kubespray provides:

• a highly available cluster

• composable attributes

• support for most popular Linux distributions

• continuous-integration tests

Pre-requisites

Before we can get started, we need a few prerequisites to be in place. This is what we are going to need:

1. A host with Ansible installed. to learn more about Ansible. Find the Ansible installation details .

2. You should also set up an SSH key pair to authenticate to the Kubernetes nodes without using a password. This permits Ansible to perform optimally.

3. Few servers/hosts/VMs to serve as our targets to deploy Kubernetes. I am using Ubuntu 18.04, and my servers each have 4GB RAM and 2vCPUs. This is fine for my testing purposes, which I use to try out new things using Kubernetes. You need to be able to SSH into each of these nodes as root using the SSH key pair I mentioned above.

Deployment Steps

The above will do the following:

• Create a new Linux User Account for use with Kubernetes on each node

• Install Kubernetes and containers on each node

• Configure the Master node

• Join the Worker nodes to the new cluster

Install Python

Ansible needs Python to be installed on all the machines.

apt-get update && apt-get install python3-pip -y

Disable Swap

Setup SSH using key-based authentication

All the machines should be in the same network with Ubuntu or Centos installed.

ssh key should be generated from the Bastion machine and must be copied to all the servers part of your inventory.

• Generate the ssh key ssh-keygen -t rsa

• Copy over the public key to all nodes.

Setup Ansible Controller Machine Setup Kubespray

• Clone the official repository

• Install dependencies from requirements.txt

• Create Inventory

where mycluster is the custom configuration name. Replace with whatever name you would like to assign to the current cluster.

Create inventory using an inventory generator.

Once it runs, you can see an inventory file that looks like the below:

• Review and change parameters under inventory/mycluster/group_vars

Deploy Kubespray with Ansible Playbook - run the playbook as Ubuntu

• The option --become is required for example writing SSL keys in /etc/, installing packages and interacting with various system daemons.

• Note: Without --become - the playbook will fail to run!

Kubernetes cluster will be created with three masters and four nodes using the above process.

Kube config will be generated in a .Kubefolder. The cluster can be accessible via kubeconfig.

HA-Proxy

• Install haproxy package in a haproxy machine that will be allocated for proxy

sudo apt-get install haproxy -y

• IPs need to be whitelisted as per the requirements in the config.

sudo vim /etc/haproxy/haproxy.cfg

Volumes

Iscsi volumes will be provided by the SDC team as per the requisition and the same can be used for statefulsets.

Topics covered:

Overview

Terraform is an open-source infrastructure as code () software tool that allows DevOps engineers to programmatically provision the physical resources an application requires to run.

Infrastructure as code is an IT practice that manages an application's underlying IT infrastructure through programming. This approach to resource allocation allows developers to logically manage, monitor and provision resources -- as opposed to requiring that an operations team manually configure each required resource.

Terraform users define and by using a JSON-like configuration language called HCL (HashiCorp Configuration Language). HCL's simple syntax makes it easy for DevOps teams to provision and re-provision infrastructure across multiple clouds and on-premises data centres.

Cloud Resources Required

Before we provision the cloud resources, we need to understand and be sure about what resources need to be provisioned by Terraform to deploy DIGIT. The following picture shows the various key components. (AKS, Node Pools, Postgres DB, Volumes, Load Balancer)

Understand Terraform Script

• Here we have already written the terraform script that one can reuse/leverage that provisions the production-grade DIGIT Infra and can be customized with the user-specific configuration.

Deployment Steps

Save the file and exit the editor.

Terraform Execution: Infrastructure Resources Provisioning

Once you have finished declaring the resources, you can deploy all resources.

1. terraform init: command is used to initialize a working directory containing Terraform configuration files.

2. terraform plan: command creates an execution plan, which lets you preview the changes that Terraform plans to make to your infrastructure.

3. terraform apply: command executes the actions proposed in a Terraform plan to create or update infrastructure.

After the complete creation, you can see resources in your Azure account.

Now we know what the terraform script does, the resources graph that it provisions and what custom values should be given with respect to your environment. The next step is to begin to run the terraform scripts to provision infra required to Deploy DIGIT on Azure.

1. Use the CD command to move into the following directory run the following commands 1-by-1 and watch the output closely.

Test Kubernetes Cluster

The Kubernetes tools can be used to verify the newly created cluster.

1. Once Terraform Apply execution is complete, it generates the Kubernetes configuration file or you can get it from Terraform state.

2. Use the below command to get kubeconfig. It will automatically store your kubeconfig in .kube folder.

3. Verify the health of the cluster.

Topics covered:

Overview

Kubespray is a composition of playbooks, , provisioning tools, and domain knowledge for generic OS/Kubernetes cluster configuration management tasks. Kubespray provides:

• a highly available cluster

• composable attributes

• support for most popular Linux distributions

• continuous-integration tests

Pre-requisites

2. the repos below to your GitHub Organization account

3. (version 1.13.X)

7. Install on your local machine to interact with the Kubernetes cluster.

8. Install to help package the services along with the configurations, environment, secrets, etc into .

Hardware

• One Bastion machine to run Kubespray

• HA-PROXY machine which acts as a load balancer with Public IP. (CPU: 2Core , Memory: 4Gb)

• one machine which acts as a master node. (CPU: 2Core , Memory: 4Gb)

• one machine which acts as a worker node. (CPU: 8Core , Memory: 16Gb)

• ISCSI volumes for persistence volume. (number of quantity: 2 )

  • kaniko-cache-claim:- 10Gb

  • Jenkins home:- 100Gb

Software

1. Kubernetes nodes

   1. Ubuntu 18.04

   2. SSH

   3. Privileged user

   4. Python

2. Bastion machine

   1. Ansible

   2. git

   3. Python

Preparing The Nodes

Run and follow instructions on all nodes.

Install Python

Ansible needs Python to be installed on all the machines.

apt-get update && apt-get install python3-pip -y

Disable Swap

Setup SSH using key-based authentication

All the machines should be in the same network with ubuntu or centos installed.

ssh key should be generated from the Bastion machine and must be copied to all the servers part of your inventory.

• Generate the ssh key ssh-keygen -t rsa

• Copy over the public key to all nodes.

Setup Ansible Controller machine Setup kubespray

• Clone the official repository

• Install dependencies from requirements.txt

• Create Inventory

where mycluster is the custom configuration name. Replace with whatever name you would like to assign to the current cluster.

Create inventory using an inventory generator.

Once it runs, you can see an inventory file that looks like the below:

• Review and change parameters under inventory/mycluster/group_vars

• Deploy Kubespray with Ansible Playbook - run the playbook as Ubuntu

  • The option --become is required, for example writing SSL keys in /etc/, installing packages and interacting with various system daemons.

  • Note: Without --become - the playbook will fail to run!

Kubernetes cluster will be created with three masters and four nodes with the above process.

Kube config will be generated in a .Kubefolder. The cluster can be accessible via kubeconfig.

HA-Proxy

• Install haproxy package in a haproxy machine that will be allocated for proxy

sudo apt-get install haproxy -y

• IPs need to be whitelisted as per the requirements in the config.

sudo vim /etc/haproxy/haproxy.cfg

Volumes

Iscsi volumes will be provided by the SDC team as per the requisition and the same can be used for statefulsets.

CI/CD Build Job Pipeline Setup

Topics covered:

• Hardware pre-requisites

• Software pre-requisites

Overview

Check the hardware and software pre-requisites for deployment on SDC.

Hardware Pre-requisites

Software Pre-requisites

1. Kubernetes nodes

   1. Ubuntu 18.04

   2. SSH

   3. Privileged user

   4. Python

2. Bastion machine

   1. Ansible

   2. Git

   3. Python

Topics covered:

• Pre-requisites for CI/CD setup

• CI/CD cluster setup details

• Custom variables and configuration details

• Run terraform scripts

• Jenkins deployment steps

Overview

Terraform helps you build a graph of all your resources and parallelizes the creation or modification of any non-dependent resources. Thus, Terraform builds infrastructure as efficiently as possible while providing the operators with clear insight into the dependencies on the infrastructure.

Pre-requisites

1. GitHub Organization account

2. Fork the repos below to your GitHub Organization account

   • https://github.com/egovernments/DIGIT-DevOps

   • https://github.com/egovernments/CIOps

3. AWS KMS

4. Go lang (version 1.13.X)

5. SOPS

6. GitHub user

7. Docker Hub account

8. ​AWS account with admin access to provision EKS Service. Try subscribing to a free AWS account to learn the basics. There is a limit on what is offered as free. This demo requires a commercial subscription to the EKS service. The cost for a one or two days trial might range between Rs 500-1000. (Note: Post the demo, for the internal folks, eGov will provide a 2-3 hrs time-bound access to eGov's AWS account based on the request and the available number of slots per day).

9. Install kubectl on your local machine to interact with the Kubernetes cluster.

10. Install Helm to help package the services along with the configurations, environment, secrets, etc into a Kubernetes manifests.

11. Install terraform version (0.14.10) for the Infra-as-code (IaC) to provision cloud resources as code and with desired resource graph. It also helps destroy the cluster in one go.

12. ​Install AWS CLI on your local machine so that you can use AWS CLI commands to provision and manage the cloud resources on your account.

13. Install AWS IAM Authenticator to help authenticate your connection from your local machine and deploy DIGIT services.

14. Use the AWS IAM User credentials provided for the Terraform (Infra-as-code) to connect to the AWS account and provision the cloud resources.

    ​

    • You will receive a Secret Access Key and Access Key ID. Save the keys.

    • Open the terminal and run the command given below. The AWS CLI is already installed and the credentials are saved. (Provide the credentials, leave the region and output format blank).

    Copy

    aws configure --profile cicd-infra-account 
    
    AWS Access Key ID []:<Your access key>
    AWS Secret Access Key []:<Your secret key>
    Default region name []: ap-south-1
    Default output format []: text

    • The above creates the following file on your machine as /Users/.aws/credentials.

    Copy

    [cicd-infra-account] 
    aws_access_key_id=*********** 
    aws_secret_access_key=****************************

Setup Details

Before we provision the cloud resources, we need to understand and be sure about what resources need to be provisioned by terraform to deploy CI/CD.

The following is the resource graph that we are going to provision using terraform in a standard way so that every time and for every environment, the infra is the same.

• EKS Control Plane (Kubernetes master)

• Work node group (VMs with the estimated number of vCPUs, Memory)

• EBS Volumes (Persistent volumes)

• VPCs (Private networks)

• Users to access, deploy and read-only

Terraform Script Resource Graph

• Ideally, one would write the terraform script from scratch using this doc.

• Here we have already written the terraform script that provisions the production-grade DIGIT Infra and can be customized with the specified configuration.

• Clone the DIGIT-DevOps GitHub repo. The terraform script to provision the EKS cluster is available in this repo. The structure of the files is given below.

git clone --branch release https://github.com/egovernments/DIGIT-DevOps.git
cd DIGIT-DevOps/infra-as-code/terraform


└── modules
    ├── kubernetes
    │   └── aws
    │       ├── eks-cluster
    │       │   ├── main.tf
    │       │   ├── outputs.tf
    │       │   └── variables.tf
    │       ├── network
    │       │   ├── main.tf
    │       │   ├── outputs.tf
    │       │   └── variables.tf
    │       └── workers
    │           ├── main.tf
    │           ├── outputs.tf
    │           └── variables.tf
    └── storage
        └── aws
            ├── main.tf
            ├── outputs.tf
            └── variables.tf

Here, you will find the main.tf under each of the modules that have the provisioning definition for resources like EKS cluster, storage, etc. All these are modularized and react as per the customized options provided.

Example:

1. VPC Resources -

   • VPC

   • Subnets

   • Internet Gateway

   • Route Table

2. EKS Cluster Resources -

   • IAM Role to allow EKS service to manage other AWS services

   • EC2 Security Group to allow networking traffic with the EKS cluster

   • EKS Cluster

3. EKS Worker Nodes Resources -

   • IAM role allowing Kubernetes actions to access other AWS services

   • EC2 Security Group to allow networking traffic

   • Data source to fetch the latest EKS worker AMI

   • AutoScaling launch configuration to configure worker instances

   • AutoScaling group to launch worker instances

4. Storage Module -

   • Configuration in this directory creates EBS volume and attaches it together.

The following main.tf with create s3 bucket to store all the states of the execution to keep track.

//DIGIT-DevOps/Infra-as-code/terraform/egov-cicd/remote-state

[**main.tf**](https://github.com/egovernments/DIGIT-DevOps/blob/release/infra-as-code/terraform/egov-cicd/remote-state/main.tf)\*\*\*\*

provider "aws" {
  region = "ap-south-1"
}

#This is a bucket name that you can name as you wish
resource "aws_s3_bucket" "terraform_state" {
  bucket = "try-cicd-workshop-yourname" 

  versioning {
    enabled = true
  }

  lifecycle {
    prevent_destroy = true
  }
}

#This is a bucket name that you can name as you wish
resource "aws_dynamodb_table" "terraform_state_lock" {
  name           = "try-cicd-workshop-yourname"
  read_capacity  = 1
  write_capacity = 1
  hash_key       = "LockID"

  attribute {
    name = "LockID"
    type = "S"
  }
}

The following main.tf contains the detailed resource definitions that need to be provisioned.

Dir: DIGIT-DevOps/Infra-as-code/terraform/egov-cicd

main.tf

terraform {
  backend "s3" {
    bucket = "try-cicd-workshop-yourname"
    key = "terraform"
    region = "ap-south-1"
  }
}

module "network" {
  source             = "../modules/kubernetes/aws/network"
  vpc_cidr_block     = "${var.vpc_cidr_block}"
  cluster_name       = "${var.cluster_name}"
  availability_zones = "${var.network_availability_zones}"
}


data "aws_eks_cluster" "cluster" {
  name = "${module.eks.cluster_id}"
}

data "aws_eks_cluster_auth" "cluster" {
  name = "${module.eks.cluster_id}"
}
  
provider "kubernetes" {
  host                   = "${data.aws_eks_cluster.cluster.endpoint}"
  cluster_ca_certificate = "${base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)}"
  token                  = "${data.aws_eks_cluster_auth.cluster.token}"
  #load_config_file       = false
}
  
module "eks" {
  source          = "terraform-aws-modules/eks/aws"
  version         = "17.24.0"
  cluster_name    = "${var.cluster_name}"
  vpc_id          = "${module.network.vpc_id}"
  cluster_version = "${var.kubernetes_version}"
  subnets         = "${concat(module.network.private_subnets, module.network.public_subnets)}"

  worker_groups = [
    {
      name                          = "spot"
      subnets                       = "${concat(slice(module.network.private_subnets, 0, length(var.availability_zones)))}"
      override_instance_types       = "${var.override_instance_types}"
      kubelet_extra_args            = "--node-labels=node.kubernetes.io/lifecycle=spot"
      additional_security_group_ids = ["${module.network.worker_nodes_sg_id}"]
      asg_max_size                  = "${var.number_of_worker_nodes}"
      asg_desired_capacity          = "${var.number_of_worker_nodes}"
      spot_allocation_strategy      = "capacity-optimized"
      spot_instance_pools           = null
    }
  ]
  tags = "${
    tomap({
      "kubernetes.io/cluster/${var.cluster_name}" = "owned",
      "KubernetesCluster" = "${var.cluster_name}"
    })
  }"
 
}

module "jenkins" {

  source = "../modules/storage/aws"
  storage_count = 1
  environment = "${var.cluster_name}"
  disk_prefix = "jenkins-home"
  availability_zones = "${var.availability_zones}"
  storage_sku = "gp2"
  disk_size_gb = "50"
  
}

Custom Variables/Configurations

Define your configurations in variables.tf and provide the environment-specific cloud requirements. The same terraform template can be used to customize the configurations.

├── egov-cicd
│   ├── main.tf 
│   ├── outputs.tf
│   ├── providers.tf
│   ├── remote-state
│   │   └── main.tf
│   └── variables.tf

Following are the values that you need to mention in the following files. The blank ones will prompt for inputs during execution.

variables.tf

#
# Variables Configuration
#

variable "cluster_name" {
  default = "<Desired Cluster name>"  #eg: my-digit-cicd
}

variable "vpc_cidr_block" {
  default = "192.168.0.0/16"
}

variable "network_availability_zones" {
  default = ["ap-south-1b", "ap-south-1a"]
}

variable "availability_zones" {
  default = ["ap-south-1b"]
}

variable "kubernetes_version" {
  default = "1.18"
}

variable "instance_type" {
  default = "t3a.xlarge"
}

variable "override_instance_types" {
  default = ["t3.xlarge", "r5ad.xlarge", "r5a.xlarge", "t3a.xlarge"]
}

variable "number_of_worker_nodes" {
  default = "1"
}

variable "spot_max_price" {
  default = "0.0538"
}

variable "ssh_key_name" {
  default = "egov-cicd"
}

Run Terraform Scripts

We have covered what the terraform script does, the resources graph that it provisions and what custom values should be given with respect to the selected environment.

Now, run the terraform scripts to provision the infra required to Deploy DIGIT on AWS.

Use the 'cd' command to change to the following directory and run the following commands. Check the output.

cd DIGIT-DevOps/infra-as-code/terraform/egov-cicd/remote-state
terraform init
terraform plan
terraform apply


cd DIGIT-DevOps/infra-as-code/terraform/egov-cicd
terraform init
terraform plan
terraform apply

After successful execution, the following resources get created and can be verified by the command "terraform output".

• s3 bucket: to store terraform state

• Network: VPC, security groups

• IAM users auth: using keybase to create admin, deployer, the user

Use the URL https://keybase.io/ to create your own PGP key. This creates both public and private keys on the machine, upload the public key into the keybase account that you have just created, give a name to it and ensure that you mention that in your terraform. This allows you to encrypt sensitive information.

• Example: Create a user keybase. This is "egovterraform" in the case of eGov. Upload the public key here - https://keybase.io/egovterraform/pgp_keys.asc

• Use this portal to Decrypt the secret key. To decrypt the PGP message, upload the PGP Message, PGP Private Key and Passphrase.

• EKS cluster: with master(s) & worker node(s).

• Storage(s): for es-master, es-data-v1, es-master-infra, es-data-infra-v1, zookeeper, kafka, kafka-infra.

• Use this link to get the kubeconfig from EKS to fetch the kubeconfig file. This enables you to connect to the cluster from your local machine and deploy DIGIT services to the cluster.

aws sts get-caller-identity

# Run the below command and give the respective region-code and the cluster name
aws eks --region <region-code> update-kubeconfig --name <cluster_name>

Finally, verify that you are able to connect to the cluster by running the command below:

export KUBECONFIG=<path-to-your-kube_config>
kubectl config use-context <your cluster name>

kubectl get nodes

NAME                                             STATUS AGE   VERSION               OS-Image           
ip-192-168-xx-1.ap-south-1.compute.internal   Ready  45d   v1.18.10-eks-bac369   Amazon Linux 2   

Whola! All set and now you can Deploy Jenkins

Jenkins Deployment

Post infra setup (Kubernetes Cluster), we start with deploying the Jenkins and kaniko-cache-warmer.

Pre-requisites

• Sub domain to expose CI/CD URL

• GitHub Oauth App (this provides you with the clientId, clientSecret)

  • Under Authorization callback URL enter the below url ie (Replace <domain_name> with your domain) https://<domain_name>/securityRealm/finishLogin

• Github User

• Generate a new ssh key for the above user (this provides the ssh public and private keys)

• Add the earlier created ssh public key to GitHub user account

• Add ssh private key to the gitReadSshPrivateKey

• With previously created GitHub users generate a personal read-only access token

• Docker hub account details (username and password)

• SSL certificate for the sub-domain

Prepare an <ci.yaml> master config file and <ci-secrets.yaml>. Name this file as desired. It has the following configurations:

• credentials, secrets (you need to encrypt using sops and create a ci-secret.yaml separately)

• Add subdomain name in ci.yaml

• Check and add your project specific ci-secrets.yaml details (like github Oauth app clientId, clientSecret, gitReadSshPrivateKey, gitReadAccessToken, dockerConfigJson, dockerUsername and dockerPassword)

• To create a Jenkins namespace mark this flag true

• Add your environment-specific kubconfigs under kubConfigs like https://github.com/egovernments/DIGIT-DevOps/blob/release/config-as-code/environments/ci-demo-secrets.yaml#L50

• KubeConfig environment name and deploymentJobs name from ci.yaml should be the same

• Update the CIOps and DIGIT-DevOps repo ssh url with the forked repo's ssh url.

• Make sure earlier created github users have read-only access to the forked DIGIT-DevOps and CIOps repos.

• SSL certificate for the sub-domain.

Prepare CIOps Configs

• Update the DOCKER_NAMESPACE with your docker hub organization name.

• Update the repo name "egovio" with your docker hub organization name in buildPipeline.groovy

• Update the cache repo name https://github.com/egovernments/CIOps/blob/master/vars/buildPipeline.groovy#L166https://github.com/egovernments/CIOps/blob/master/vars/buildPipeline.groovy#L185

• Remove the below env:

  • https://github.com/egovernments/CIOps/blob/master/vars/buildPipeline.groovy#L35-L54

  • https://github.com/egovernments/CIOps/blob/master/vars/buildPipeline.groovy#L156-L159

  • https://github.com/egovernments/CIOps/blob/master/vars/buildPipeline.groovy#L176-L179

  • https://github.com/egovernments/CIOps/blob/master/vars/buildPipeline.groovy#L33-L34

• Remove the below volumeMounts:

  • https://github.com/egovernments/CIOps/blob/master/vars/buildPipeline.groovy#L60-L61

  • https://github.com/egovernments/CIOps/blob/master/vars/deployer.groovy#L20-L21

• Remove the below volume:

  • https://github.com/egovernments/CIOps/blob/master/vars/buildPipeline.groovy#L80-L87

  • https://github.com/egovernments/CIOps/blob/master/vars/deployer.groovy#L32-L39

cd DIGIT-DevOps/deploy-as-code/deployer

export KUBECONFIG=<path-to-your-kube_config>
kubectl config use-context <your cluster name>
go run main.go deploy -c -e ci 'jenkins,kaniko-cache-warmer,nginx-ingress,cert-manager'

Jenkins is launched. You can access the same through your sub-domain configured in ci.yaml.

Topics covered:

Overview

Since there are many DIGIT services and the development code is part of various git repos, one needs to understand the concept of cicd-as-service which is open-sourced. This page guides you through the process of creating a CI/CD pipeline.

Integrating Services To CI/CD

The initial steps for integrating any new service/app to the CI/CD are discussed below.

Once the desired service is ready for integration: decide the service name, type of service, and if DB migration is required or not. While you commit the source code of the service to the git repository, the following file should be added with the relevant details which are mentioned below:

Build-config.yml – It is present under the build directory in the repository

This file contains the below details used for creating the automated Jenkins pipeline job for the newly created service.

While integrating a new service/app, the above content needs to be added to the build-config.yml file of that app repository. For example: to onboard a new service called egov-test, the build-config.yml should be added as mentioned below.

If a job requires multiple images to be created (DB Migration) then it should be added as below,

Note - If a new repository is created then the build-config.yml is created under the build folder and the config values are added to it.

The git repository URL is then added to the Job Builder parameters

When the Jenkins Job => job builder is executed, the CI Pipeline gets created automatically based on the above details in build-config.yml. Eg: egov-test job is created in the builds/DIGIT-OSS/core-services folder in Jenkins since the “build-config is edited under core-services” And it should be the “master” branch. Once the pipeline job is created, it can be executed for any feature branch with build parameters - specifying the branch to be built (master or feature branch).

As a result of the pipeline execution, the respective app/service docker image is built and pushed to the Docker repository.

On repo provide read-only access to GitHub users (created while ci/cd deployment)

Continuous Integration (CI)

The Jenkins CI pipeline is configured and managed 'as code'.

If the git repository ssh URL is available, build the Job-Builder Job.

If the git repository URL is not available, check and add the same team.

Continuous Deployment (CD)

The services are deployed and managed on a Kubernetes cluster in cloud platforms like AWS, Azure, GCP, OpenStack, etc. Here, we use helm charts to manage and generate the Kubernetes manifest files and use them for further deployment to the respective Kubernetes cluster. Each service is created as charts which have the below-mentioned files.

Note: The steps below are only for the introduction and implementation of new services.

• If you are going to introduce a new module with the help of multiple services, we suggest you create a new Directory with your module name.

Example.:-

This chart can also be modified further based on user requirements.

The deployment of manifests to the Kubernetes cluster is made very simple and easy. There are Jenkins Jobs for each state and are environment-specific. We need to provide the image name or the service name for the respective Jenkins deployment job.

‌The deployment Jenkins job internally performs the following operations:

• Reads the image name or the service name given and finds the chart that is specific to it.

• Generates the Kubernetes manifests files from the chart using the helm template engine.

• Execute the deployment manifest with the specified docker image(s) to the Kubernetes cluster.