Introduction

iFIX/mGramSeva is a microservices-based distributed cloud-native application. The microservices streamline processes to meet outcomes at scale and speed. Each of the microservices is dockerized and deployed on Kubernetes infrastructure.

Pre-reads

It is essential to understand some of the key concepts, benefits and best practices of Kubernetes platform before we understand the deployment of the iFix/mGramSeva.

• Know the basics of Kubernetes: https://www.youtube.com/watch?v=PH-2FfFD2PU&t=3s​

• Know the basics of kubectl commands

• Know Kubernetes manifests: https://www.youtube.com/watch?v=ohSUtEfDefc​

• Know how to manage env values, secrets of any service deployed in Kubernetes https://www.youtube.com/watch?v=OW244LxB4oI​

• Know how to port forward to a pod running inside k8s cluster and work locally https://www.youtube.com/watch?v=TT3nd5n5Yus​

• Know sops to secure your keys/creds: https://www.youtube.com/watch?v=DWzJ87KbwxA​

1. Choose Installation Type

Choose the target infra type and follow the instructions to set up a Kubernetes cluster before moving on to the deployment.

2. Deployment

Before we begin the deployment, it is important to understand the deployment architecture that starts from the source code to the production-ready stage. Deploying and managing Kubernetes have emerged as a streamlined way to deploy containers in the cloud infrastructure. When running Kubernetes at scale, managing, operating, and scaling its infrastructure to maximize cluster utilization can be challenging. There are too many parameters the development team needs to manage and configure. This includes selecting the best instance type and size, determining when to scale up or down, and making sure all of the containers are scheduled and running on the best instances — and that is even before starting to think about cost resource optimization.

The simplest way to get started with the deployment process is to manage deployment configuration as code. Each service deployment configuration is defined as Helm charts and deployed into the Kubernetes cluster. We can collocate the deployment-as-code as source code, leveraging all the benefits of source control including change tracking and branching, and then package it. So below is the source code repo that contains all the deployment-as-code for iFIX.

1. mGramSeva Installation

2. iFIX-adapter Installation

3. iFIX Ref Dashboard Installation

3. Destroy the Setup

Clean up the cluster Setup using the command below, if required. This deletes the entire cluster and other cloud resources that were provisioned for the mGramSeva Infra Setup.

cd iFix-DevOps/infra-as-code/terraform/my-iFix-eksterraformdestroy​

Conclusion

All done, the infra on local, cloud, and deployment of iFIX into the Kubernetes cluster is completed successfully.

Quickstart Installation helps you jump-start with the mGramSeva basic installation with limited functionalities.‌

mGramSeva is a distributed microservice-based platform that comprises many services that are containerized. Depending upon the required features, the specific services can be run on any container-supported orchestration platform like docker-compose, Kubernetes, etc.‌

The Quickstart guide covers the installation steps for basic services to get the platform up. Before setting up mGramSeva, create a lightweight Kubernetes cluster called k3d on a local machine with specified H/W requirements. The H/W requirements are listed below to ensure before we proceed further.‌

1. Infra Setup

To provision a lightweight Kubernetes cluster, please follow the instructions below in context to your OS and install the k3d on your machine.‌

H/W or VM Size

• min 4 vCPUs (recommended 8)

• min 8GiB of RAM (recommended 16)

• min 30GiB of HDD (recommended 30+)

Tools

• Linux distribution running in a VM or bare metal

  • Ubuntu 18.04 or Debian 10 (VM or bare metal)

  • Install Docker​

  • ​Install kubectl on Linux

  • Open terminal and Install k3d on Linux using the below command

  Copy

  wget -q -O - https://raw.githubusercontent.com/rancher/k3d/main/install.sh | bash

• OSX or Mac

  • ​Docker Desktop local Kubernetes cluster enabled

  • ​Install kubectl on Mac

  • Install k3d on Mac, on terminal use Homebrew (Homebrew is available for MacOS) using the below command

  Copy

  brew install k3d

• Windows 10 or above

  • ​Docker Desktop for windows need to be installed

  • ​Install kubectl on Windows

  • ​Install Chocolatey package manager for windows

  • Install GitBash as an alternative command prompt that allows most of the Linux commands on windows.

  • Open gitbash and Install k3d on Windows using the below command

  Copy

  choco install k3d

Infra (Kubernetes Cluster) Creation

Once the above prerequisites are met, run the following tasks depending upon your OS.‌

• login/ssh into the machine, go to terminal/command prompt and run the following commands as an admin user.

• Create /Kube directory and change permission. To use this directory for persistent data mount. This means data from all container logs will be stored here.

 mkdir /kube
 chmod 777 /kube

 #if you are unable to create the /kube folder in the root
 # you can create it your user directory and provide the absolue path below

• Create a cluster with a single master node and 2 agents (Worker Nodes) and mount the pre-created directory (for data persistence).

k3d cluster create --k3s-server-arg "--no-deploy=traefik" --agents 2 -v "/kube:/kube@agent[0,1]" -v "/kube:/kube@server[0]" --port "80:80@loadbalancer"

• ‌When cluster creation is successful, get the kubeconfig file, that allows you to connect to the cluster at any time.

k3d kubeconfig get k3s-default > k3dconfig
kubectl config use-context k3d-k3s-default --kubeconfig=k3dconfig

• Verify the Cluster creation by running the following commands from your local machine where the kubectl is installed. It gives you the sample output as below

root@ip:/# kubectl cluster-info

OutPut:
Kubernetes control plane is running at https://0.0.0.0:33931
CoreDNS is running at https://0.0.0.0:33931/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://0.0.0.0:33931/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy



root@ip:/# k3d cluster list
NAME          SERVERS   AGENTS   LOADBALANCER
k3s-default   1/1       2/2      true

• You can verify the workers' nodes created by using the following command.

kubectl get nodes

Output:
NAME                       STATUS   ROLES                  AGE     VERSION
k3d-k3s-default-agent-0    Ready    <none>                 3d18h   v1.21.1+k3s1
k3d-k3s-default-agent-1    Ready    <none>                 3d18h   v1.21.1+k3s1
k3d-k3s-default-server-0   Ready    control-plane,master   3d18h   v1.21.1+k3s1


kubectl top nodes

Output:
W0625 07:56:24.588781   12810 top_node.go:119] Using json format to get metrics. Next release will switch to protocol-buffers, switch early by passing --use-protocol-buffers flag
NAME                       CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
k3d-k3s-default-agent-0    547m         6%     1505Mi          9%        
k3d-k3s-default-agent-1    40m          0%     2175Mi          13%       
k3d-k3s-default-server-0   59m          0%     2286Mi          14%

‌Once the above steps are completed successfully, your Cluster is now up and running ready to proceed with the DIGIT Deployment.‌

2. mGramSeva Setup

‌Now that we have the Infra setup to proceed with the DIGIT Deployment. Following are the tools that need to be installed on the machine before proceeding with the DIGIT Services deployment.‌

What we'll deploy in Quickstart:‌

• mGramSeva core platform services

Prerequisites

1. mGramSeva uses golang (required v1.13.3) automated scripts to deploy the builds on to Kubernetes - Linux or Windows or Mac​

3. ​kubectl is a CLI to connect to the kubernetes cluster from your machine

4. Install CURL for making API calls

5. ​Install Visualstudio IDE Code for better code/configuration editing capabilities

6. The mGramSeva services deployment configurations are in GitRepo which needs to install git and then git clone it to your local.

7. ​Install Postman to run some DIGIT bootstrap scripts

root@ip:/# git clone -b quickstart https://github.com/misdwss/iFix-DevOps

‌After cloning the repo CD into the folder iFix-DevOps and type the "code ." command that will open the visual editor and opens all the files from the repo iFix-DevOps

root@ip:/# cd iFix-DevOps
root@ip:DIGIT-DevOps# code .

Check the sample deployment config file that needs to be configured as per any specific values according to your needs. (For a quick start you can run as it is)

https://github.com/misdwss/iFix-DevOps/blob/quickstart/deploy-as-code/helm/environments/quickstart-config.yaml

Add the following entries in your host file /etc/hosts depending on your OS, instructions can be found here.

127.0.0.1 quickstart.local.digit

‌3. Deployment

‌Once the prerequisite setup is complete, go to the following repo, run the command and follow the instructions.

root@ip:# cd iFix-DevOps/deploy-as-code/egov-deployer

root@ip:# go run mgramseva_setup.go

#Be prepared for the following questions
1. Do you have the Kubernetes Setup?
2. Provide the path of the intented env kubeconfig file
3. Which version of the DIGIT that you want to install - Choose "Quickstart"
4. What mGremSeva Modules that you choose to install
5. All, done, Now do you want to preview the deployment manifests 
6. Are you good to proceed with the DIGIT Installation

All Done.

‌You can now test the DIGIT application status in the command prompt/terminal by using the below command.

curl -Is http://quickstart.local.digit/employee/login |  head -n 1HTTP/2 200

The Amazon Elastic Kubernetes Service (EKS) is one of the AWS services for deploying, managing, and scaling any distributed and containerized workloads, here we can provision the EKS cluster on AWS from ground up and using an automated way (infra-as-code) using terraform and then deploy the DIGIT-iFIX Services config-as-code using Helm.

Pre-reads

• Know about EKS: https://www.youtube.com/watch?v=SsUnPWp5ilc

• Know what is terraform: https://youtu.be/h970ZBgKINg

Pre-requisites

1. ​AWS account with the admin access to provision EKS Service, you can always subscribe to free AWS account to learn the basics and try, but there is a limit to what is offered as free, for this demo you need to have a commercial subscription to the EKS service.

2. Install kubectl on your local machine that helps you interact with the kubernetes cluster

3. Install Helm that helps you package the services along with the configurations, envs, secrets, etc into a kubernetes manifests

4. Install terraform version (0.14.10) for the Infra-as-code (IaC) to provision cloud resources as code and with desired resource graph and also it helps to destroy the cluster at one go.

5. ​Install AWS CLI on your local machine so that you can use aws cli commands to provision and manage the cloud resources on your account.

6. Install AWS IAM Authenticator that helps you authenticate your connection from your local machine so that you should be able to deploy DIGIT services.

7. Use the AWS IAM User credentials provided for the Terraform (Infra-as-code) to connect with your AWS account and provision the cloud resources.

   ​

   • You'll get a Secret Access Key and Access Key ID. Save them safely.

   • Open the terminal and run the following command. The AWS CLI is already installed and the credentials are saved. (Provide the credentials and you can leave the region and output format blank).

   Copy

   aws configure --profile mgramseva-infra-account 
   
   AWS Access Key ID []:<Your access key>
   AWS Secret Access Key []:<Your secret key>
   Default region name []: ap-south-1
   Default output format []: text

   The above will create the following file In your machine as /Users/.aws/credentials

   Copy

   [mgramseva-infra-account] 
   aws_access_key_id=*********** 
   aws_secret_access_key=****************************

Before we provision the cloud resources, we need to understand and be sure about what resources need to be provisioned by terraform to deploy DIGIT. The following picture shows the various key components. (EKS, Worker Nodes, Postgress DB, EBS Volumes, Load Balancer)

Considering the above deployment architecture, the following is the resource graph that we are going to provision using terraform in a standard way so that every time and for every environment, it'll have the same infra.

• EKS Control Plane (Kubernetes Master)

• Work node group (VMs with the estimated number of vCPUs, Memory)

• Node-pool's (mgramseva and ifix)

• EBS Volumes (persistent volumes)

• RDS (Postgresql)

• VPCs (private network)

• Users to access, deploy and read-only

Understand The Resource Graph In Terraform Script

• Ideally, one would write the terraform script from the scratch using this doc.

• Here we have already written the terraform script that provisions the production-grade DIGIT Infra and can be customized with the specified configuration.

• Let's clone the iFix-DevOps GitHub repo where the terraform script to provision EKS cluster is available and below is the structure of the files.

git clone https://github.com/misdwss/iFix-DevOps.git
cd iFix-DevOps/infra-as-code/terraform


└── modules
    ├── db
    │   └── aws
    │       ├── main.tf
    │       ├── outputs.tf
    │       └── variables.tf
    ├── kubernetes
    │   └── aws
    │       ├── eks-cluster
    │       │   ├── main.tf
    │       │   ├── outputs.tf
    │       │   └── variables.tf
    │       ├── network
    │       │   ├── main.tf
    │       │   ├── outputs.tf
    │       │   └── variables.tf
    │       └── workers
    │           ├── main.tf
    │           ├── outputs.tf
    │           └── variables.tf
    └── storage
        └── aws
            ├── main.tf
            ├── outputs.tf
            └── variables.tf

Example:

• VPC Resources:

  • VPC

  • Subnets

  • Internet Gateway

  • Route Table

• EKS Cluster Resources:

  • IAM Role to allow EKS service to manage other AWS services

  • EC2 Security Group to allow networking traffic with EKS cluster

  • EKS Cluster

• EKS Worker Nodes Resources:

  • IAM role allowing Kubernetes actions to access other AWS services

  • EC2 Security Group to allow networking traffic

  • Data source to fetch the latest EKS worker AMI

  • AutoScaling Launch Configuration to configure worker instances

  • AutoScaling Group to launch worker instances

• Database

  • Configuration in this directory creates a set of RDS resources including DB instance, DB subnet group, and DB parameter group.

• Storage Module

  • Configuration in this directory creates EBS volume and attaches it together.​

The following main.tf with create s3 bucket to store all the state of the execution to keep track.

iFix-DevOps/Infra-as-code/terraform/sample-eks/remote-state

main.tf​

provider "aws" {
  region = "ap-south-1"
}

#This is a bucket name that you can name as you wish
resource "aws_s3_bucket" "terraform_state" {
  bucket = "try-workshop-yourname" 

  versioning {
    enabled = true
  }

  lifecycle {
    prevent_destroy = true
  }
}

#This is a bucket name that you can name as you wish
resource "aws_dynamodb_table" "terraform_state_lock" {
  name           = "try-workshop-yourname"
  read_capacity  = 1
  write_capacity = 1
  hash_key       = "LockID"

  attribute {
    name = "LockID"
    type = "S"
  }
}

2. The following main.tf contains the detailed resource definitions that need to be provisioned, please have a look at it.

Dir: iFix-DevOps/Infra-as-code/terraform/sample-eks

main.tf

terraform {
  backend "s3" {
    bucket = "try-workshop-yourname"
    key = "terraform"
    region = "ap-south-1"
  }
}

module "network" {
  source             = "../modules/kubernetes/aws/network"
  vpc_cidr_block     = "${var.vpc_cidr_block}"
  cluster_name       = "${var.cluster_name}"
  availability_zones = "${var.network_availability_zones}"
}


module "iam_user_deployer" {
  source  = "terraform-aws-modules/iam/aws//modules/iam-user"

  name          = "${var.cluster_name}-kube-deployer"
  force_destroy = true  
  create_iam_user_login_profile = false
  create_iam_access_key         = true

  # User "egovterraform" has uploaded his public key here - https://keybase.io/egovterraform/pgp_keys.asc
  pgp_key = "${var.iam_keybase_user}"
}

module "iam_user_admin" {
  source  = "terraform-aws-modules/iam/aws//modules/iam-user"

  name          = "${var.cluster_name}-kube-admin"
  force_destroy = true  
  create_iam_user_login_profile = false
  create_iam_access_key         = true

  # User "egovterraform" has uploaded his public key here - https://keybase.io/egovterraform/pgp_keys.asc
  pgp_key = "${var.iam_keybase_user}"
}

module "iam_user_user" {
  source  = "terraform-aws-modules/iam/aws//modules/iam-user"

  name          = "${var.cluster_name}-kube-user"
  force_destroy = true  
  create_iam_user_login_profile = false
  create_iam_access_key         = true

  # User "test" has uploaded his public key here - https://keybase.io/test/pgp_keys.asc
  pgp_key = "${var.iam_keybase_user}"
}

data "aws_eks_cluster" "cluster" {
  name = "${module.eks.cluster_id}"
}

data "aws_eks_cluster_auth" "cluster" {
  name = "${module.eks.cluster_id}"
}
provider "kubernetes" {
  host                   = "${data.aws_eks_cluster.cluster.endpoint}"
  cluster_ca_certificate = "${base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)}"
  token                  = "${data.aws_eks_cluster_auth.cluster.token}"
  load_config_file       = false
  version                = "~> 1.11"
}

module "eks" {
  source          = "terraform-aws-modules/eks/aws"
  cluster_name    = "${var.cluster_name}"
  cluster_version = "${var.kubernetes_version}"
  subnets         = "${concat(module.network.private_subnets, module.network.public_subnets)}"

  tags = "${
    map(
      "kubernetes.io/cluster/${var.cluster_name}", "owned",
      "KubernetesCluster", "${var.cluster_name}"
    )
  }"

  vpc_id = "${module.network.vpc_id}"

  worker_groups_launch_template = [
    {
      name                    = "spot"
      subnets                 = "${concat(slice(module.network.private_subnets, 0, length(var.availability_zones)), slice(module.network.public_subnets, 0, length(var.availability_zones)))}"
      override_instance_types = "${var.override_instance_types}"
      asg_max_size            = 1
      asg_desired_capacity    = 1
      kubelet_extra_args      = "--node-labels=node.kubernetes.io/lifecycle=spot"
      spot_allocation_strategy= "capacity-optimized"
      spot_instance_pools     = null
    },
  ]
  
  map_users    = [
    {
      userarn  = "${module.iam_user_deployer.iam_user_arn}"
      username = "${module.iam_user_deployer.iam_user_name}"
      groups   = ["system:masters"]
    },
    {
      userarn  = "${module.iam_user_admin.iam_user_arn}"
      username = "${module.iam_user_admin.iam_user_name}"
      groups   = ["global-readonly", "digit-user"]
    },
    {
      userarn  = "${module.iam_user_user.iam_user_arn}"
      username = "${module.iam_user_user.iam_user_name}"
      groups   = ["global-readonly"]
    },    
  ]
}

module "db" {
  source                        = "../modules/db/aws"
  subnet_ids                    = "${module.network.private_subnets}"
  vpc_security_group_ids        = ["${module.network.rds_db_sg_id}"]
  availability_zone             = "${element(var.availability_zones, 0)}"
  instance_class                = "db.t3.medium"
  engine_version                = "11.5"
  storage_type                  = "gp2"
  storage_gb                    = "100"
  backup_retention_days         = "7"
  administrator_login           = "egovdev"
  administrator_login_password  = "${var.db_password}"
  db_name                       = "${var.cluster_name}-db"
  environment                   = "${var.cluster_name}"
}

module "es-master" {

  source = "../modules/storage/aws"
  storage_count = 3
  environment = "${var.cluster_name}"
  disk_prefix = "es-master"
  availability_zones = "${var.availability_zones}"
  storage_sku = "gp2"
  disk_size_gb = "2"
  
}
module "es-data-v1" {

  source = "../modules/storage/aws"
  storage_count = 3
  environment = "${var.cluster_name}"
  disk_prefix = "es-data-v1"
  availability_zones = "${var.availability_zones}"
  storage_sku = "gp2"
  disk_size_gb = "25"
  
}

module "zookeeper" {

  source = "../modules/storage/aws"
  storage_count = 3
  environment = "${var.cluster_name}"
  disk_prefix = "zookeeper"
  availability_zones = "${var.availability_zones}"
  storage_sku = "gp2"
  disk_size_gb = "2"
  
}

module "kafka" {

  source = "../modules/storage/aws"
  storage_count = 3
  environment = "${var.cluster_name}"
  disk_prefix = "kafka"
  availability_zones = "${var.availability_zones}"
  storage_sku = "gp2"
  disk_size_gb = "50"
  
}

module "zookeeper-ifix" {

  source = "../modules/storage/aws"
  storage_count = 3
  environment = "${var.cluster_name}"
  disk_prefix = "zookeeper-ifix"
  availability_zones = "${var.availability_zones}"
  storage_sku = "gp2"
  disk_size_gb = "2"
  
}

module "kafka-ifix" {

  source = "../modules/storage/aws"
  storage_count = 3
  environment = "${var.cluster_name}"
  disk_prefix = "kafka-ifix"
  availability_zones = "${var.availability_zones}"
  storage_sku = "gp2"
  disk_size_gb = "50"
  
}

module "node-group" {  
  for_each = toset(["ifix-dev", "mgramseva"])
  source = "../modules/node-pool/aws"

  cluster_name        = "${var.cluster_name}"
  node_group_name     = "${each.key}-ng"
  kubernetes_version  = "${var.kubernetes_version}"
  security_groups     =  ["${module.network.worker_nodes_sg_id}"]
  subnet              = "${concat(slice(module.network.private_subnets, 0, length(var.node_pool_zone)))}"
  node_group_max_size = 1
  node_group_desired_size = 1
}  

Custom Variables/Configurations

Define your configurations in variables.tf. Provide the environment-specific cloud requirements and use the same terraform template to customize the configurations.

├── sample-eks
│   ├── main.tf 
│   ├── outputs.tf
│   ├── providers.tf
│   ├── remote-state
│   │   └── main.tf
│   └── variables.tf

The values given below must be mentioned in the following files. The blank ones will be prompted for inputs while execution.

​variables.tf ​

## Add Cluster Name
variable "cluster_name" {
  default = "<Desired Cluster name>"  #eg: my-digit-eks
}

## Add vpc_cidr_block
variable "vpc_cidr_block" {
  default = "CIDR" 
}

# If you want prod grade N/W, you can define HA, DRS with multi zone
variable "network_availability_zones" {  
  default = ["ap-south-1b", "ap-south-1a"]
}

# Which zone, it matters
variable "availability_zones" {
  default = ["ap-south-1b"]
}

variable "node_pool_zone" {
 default = "ap-south-1a"  
}

variable "kubernetes_version" {
  default = "1.18"
}

# instance type for your worker nodes like r5a.large is 8 vCPU and 16GB RAM
variable "instance_type" {
  default = "r5a.large"
}

# spot instance configuration
variable "override_instance_types" {
  default = ["r5a.large", "r5ad.large", "r5d.large", "t3a.xlarge"] 
}

# number of machines as per estimate
variable "number_of_worker_nodes" {
  default = "1"
}

##Add ssh key in case you want to ssh to nodes
variable "ssh_key_name" {
  default = "ssh key name"
}

# terraform users ssh public key, you need to one for you, refer below to create yours
variable "iam_keybase_user" {
  default = "keybase:egovterraform"
}

# will be prompted to provide during the execution
variable "db_password" {}

​Use the URL https://keybase.io/ to create your own PGP key. This creates both public and private keys on your machine. Upload the public key into the keybase account that you have just created, give a name to it and ensure that you mention that in your terraform. This allows the encryption of all sensitive information.

• Example - the keybase user (in eGov case is "egovterraform") needs to be created and has to be uploaded the public key here - https://keybase.io/egovterraform/pgp_keys.asc​

• you can use this portal to decrypt your secret key. To decrypt PGP Message, upload the PGP Message, PGP Private Key and the Passphrase.

Run Terraform

Now that we know what the terraform script does, the resources graph that it provisions and what custom values should be given with respect to your env.

Let's begin to run the Terraform scripts to provision infra required to Deploy DIGIT on AWS.

1. First CD into the following directory and run the following command 1-by-1 and watch the output closely.

cd iFix-DevOps/infra-as-code/terraform/sample-eks/remote-state 
terraform init
terraform plan
terraform apply


cd iFix-DevOps/infra-as-code/terraform/sample-eks 
terraform init
terraform plan
terraform apply

Upon Successful execution following resources get created which can be verified by the command "terraform output"

• s3 bucket: to store terraform state.

• Network: VPC, security groups.

• IAM users auth: using the key base to create admin, deployer and the user. Use this URL https://keybase.io/ to create your own PGP key, this will create both public and private keys in your machine. Upload the public key into the keybase account that you have just created, give a name to it and ensure that you mention that in your terraform. This allows for encrypting all sensitive information.

  • Example: keybase user (in eGov case is "egovterraform") needs to be created and has to be uploaded the public key here - https://keybase.io/egovterraform/pgp_keys.asc​

  • you can use this portal to decrypt your secret key. To decrypt PGP Message, Upload the PGP Message, PGP Private Key and Passphrase.

• EKS cluster: with master(s) & worker node(s).

• Storage(s): for es-master, es-data-v1, es-master-infra, es-data-infra-v1, zookeeper, kafka, kafka-infra.

2. Use this link to get the kubeconfig from EKS to get the kubeconfig file and be able to connect to the cluster from your local machine so that you should be able to deploy DIGIT services to the cluster.

aws sts get-caller-identity

# Run the below command and give the respective region-code and the cluster name
aws eks --region <region-code> update-kubeconfig --name <cluster_name>

3. Verify that you are able to connect to the cluster by running the following command

kubectl config use-context <your cluster name>

kubectl get nodes

NAME                                             STATUS AGE   VERSION               OS-Image           
ip-192-168-xx-1.ap-south-1.compute.internal   Ready  45d   v1.15.10-eks-bac369   Amazon Linux 2   
ip-192-168-xx-2.ap-south-1.compute.internal   Ready  45d   v1.15.10-eks-bac369   Amazon Linux 2   
ip-192-168-xx-3.ap-south-1.compute.internal   Ready  45d   v1.15.10-eks-bac369   Amazon Linux 2   
ip-192-168-xx-4.ap-south-1.compute.internal   Ready  45d   v1.15.10-eks-bac369   Amazon Linux 2 

Whola! All set and now you can go with Deploy Product..

Introduction

iFIX/mGramSeva is a microservices-based distributed cloud-native application. Each of these context-specific microservices is dockerized and deployed on Kubernetes infrastructure.

Pre-reads

It is essential to understand some of the key concepts, benefits and best practices of the Kubernetes platform before we understand the deployment of the iFIX/mGramSeva.

• Know the basics of Kubernetes: https://www.youtube.com/watch?v=PH-2FfFD2PU&t=3s​

• Know the basics of kubectl commands

• Know Kubernetes manifests: https://www.youtube.com/watch?v=ohSUtEfDefc​

• Know how to manage environment values and secrets of any service deployed in Kubernetes https://www.youtube.com/watch?v=OW244LxB4oI​

• Know how to port forward to a pod running inside k8s cluster and work locally https://www.youtube.com/watch?v=TT3nd5n5Yus​

• Know sops to secure your keys/creds: https://www.youtube.com/watch?v=DWzJ87KbwxA​

1. Choose the Install Type

Choose the target infra type and follow the instructions to set up a Kubernetes cluster before moving on to the deployment.

2. Deployment

Before we begin the deployment, it is important to understand the deployment architecture that starts from the source code to the production-ready stage. Deploying and managing Kubernetes have emerged as a streamlined way to deploy containers in the cloud infrastructure. When running Kubernetes at scale, managing, operating, and scaling its infrastructure to maximize cluster utilization can be challenging. There are too many parameters the development team needs to manage and configure. This includes selecting the best instance type and size, determining when to scale up or down, and making sure all of the containers are scheduled and running on the best instances — and that is even before starting to think about cost resource optimization.

The simplest way to get started with the deployment process is to manage deployment configuration as code. Each service deployment configuration is defined as Helm charts and deployed into the Kubernetes cluster. We can collocate the deployment-as-code as source code, leveraging all the benefits of source control including change tracking and branching and then packaging it. The source code repo below contains the deployment-as-code details for iFIX.

1. mGramSeva Installation

2. iFIX-adapter Installation

3. iFIX Ref Dashboard Installation

3. Destroy the Setup

Use the command below to clean up the setup cluster. This deletes the entire cluster and other cloud resources that were provisioned for the mGramSeva Infra Setup.

cd iFix-DevOps/infra-as-code/terraform/my-iFix-eksterraformdestroy​

Conclusion

All done, the infra on local, cloud, and deployment of iFIX into the Kubernetes cluster is completed successfully.

The Azure Kubernetes Service (AKS) is one of the Azure services used for deploying, managing, and scaling any distributed and containerized workloads. Here we can provision the AKS cluster on Azure from the ground up and using an automated way (infra-as-code) using and then deploy the DIGIT-iFIX Services config-as-code using .

This quickstart assumes a basic understanding of Kubernetes concepts. For more information, see .

If you don't have an , create a before you begin.

Pre-requisites

• Use the Bash environment in .

• If you prefer, the Azure CLI to run CLI reference commands.

  • If you're using a local installation, sign in to the Azure CLI by using the command. To finish the authentication process, follow the steps displayed in your terminal. For additional sign-in options, see .

  • When you're prompted, install Azure CLI extensions on first use. For more information about extensions, see .

  • Run to find the version and dependent libraries that are installed. To upgrade to the latest version, run .

• This article requires version 2.0.64 or greater of the Azure CLI. If using Azure Cloud Shell, the latest version is already installed.

• The identity you are using to create your cluster has the appropriate minimum permissions. For more details on access and identity for AKS, see .

• Install on your local machine that helps you interact with the kubernetes cluster

• Install that helps you package the services along with the configurations, envs, secrets, etc into a

• Install version (0.14.10) for the Infra-as-code (IaC) to provision cloud resources as code and with desired resource graph and also it helps to destroy the cluster at one go.

Note: Run the commands as administrator if you plan to run the commands in this quickstart locally instead of in Azure Cloud Shell.

AKS Architecture

Before we provision the cloud resources, we need to understand and be sure about what resources need to be provisioned by terraform to deploy DIGIT. The following picture shows the various key components. (AKS, Worker Nodes, Postgres DB, Volumes, Load Balancer)

AKS Architecture For iFIX Setup

Considering the above deployment architecture, the following is the resource graph that we are going to provision using terraform in a standard way so that every time and for every env, it'll have the same infra.

• AKS Azure (Kubernetes Service Master)

• Work node group (VMs with the estimated number of vCPUs, Memory

• Volumes (persistent volumes)

• PostGres Database

• Virtual Network

• Users to access, deploy and read-only

Understand the Resource Graph In Terraform Script

• Here we have already written the terraform script that provisions the production-grade DIGIT Infra and can be customized with the specified configuration.

The following main.tf contains the detailed resource definitions that need to be provisioned, please have a look at it.

Dir: iFix-DevOps/Infra-as-code/terraform/aks-ifix-dev

Custom Variables/Configurations

You can define your configurations in variables.tf and provide the environment-specific cloud requirements so that using the same terraform template you can customize the configurations.

Following are the values that you need to mention in the following files, the blank ones will be prompted for inputs while execution.

​Run Terraform

Now that we know what the terraform script does, the resources graph that it provisions and what custom values should be given with respect to your env.

Let's begin to run the terraform scripts to provision infra required to Deploy DIGIT on AZ.

1. First CD into the following directory and run the following command 1-by-1 and watch the output closely.

Upon Successful execution following resources gets created which can be verified by the command "terraform output"

• Network: Virtual Network.

• AKS cluster: with nodepool(s), master(s) & worker node(s).

• Storage(s): for es-master, es-data-v1, es-master-infra, es-data-infra-v1, zookeeper, kafka, kafka-infra.

Connect To The Cluster

2. • Downloads credentials and configures the Kubernetes CLI to use them.

3. Finally, Verify that you are able to connect to the cluster by running the following command

ADMIN ONLY

Only the administrator can generate the API key and secret. The steps for doing this is as follows.

Keycloak console will be available at https://<host-name>/auth. The admin will log in using the username and password secret.

Step 1: Import Realm (If not already done)

1. Open the Keycloak console

2. Near the top-left corner in the realm drop down menu, select Add Realm.

3. Select the file.

4. After the realm gets created, select the ifix realm from the drop-down near the top-left corner.

Creating Clients

1. Remember to select the ifix realm from the Keycloak console before proceeding.

2. From the Clients section of Keycloak Admin Console, create a client.

3. Provide a unique username for the client.

4. Go to the client's settings

5. Change Access Type to confidential

6. Turn on Service Account Enabled

7. In the Valid Redirect URIs field provide the root URL of the iFIX Instance. (Not important for our purposes but need to set it because it is mandatory)

8. And Save these changes

9. In the Service Account Roles tab, assign the role "fiscal-event-producer"

10. In the Mappers tab, create a new mapper to associate the client with a tenantId

    1. Select Mapper Type to be "Hardcoded claim"

    2. In Token Claim Name, write "tenantId"

    3. In Claim value, write the under which the client is being created. (For example, "pb")

    4. Set Name same as Token Claim Name i.e. "tenantId"

    5. Select Claim Json Type to be "String"

Now you can get the credentials from the Credentials tab and configure them in the client's system.

Deploy mGramSeva

Post infra setup (Kubernetes Cluster), there starts the deployment process.

Pipeline as code is a practice of defining deployment pipelines through source code, such as Git. Pipeline as code is part of a larger “as code” movement that includes infrastructure as code. Teams can configure builds, tests, and deployment in code that is trackable and stored in a centralized source repository. Teams can use a declarative approach or a vendor-specific programming language, such as Jenkins and Groovy, but the premise remains the same.

A pipeline as code file specifies the stages, jobs, and actions for a pipeline to perform. Because the file is versioned, changes in pipeline code can be tested in branches with the corresponding application release.

The pipeline as code model of creating continuous integration pipelines is an industry best practice, but deployment pipelines used to be created very differently.

The deployment process has got 2 stages and 2 modes. We can see the modes first and then the stages.‌

The 2 Modes of Deployment

Essentially, mGramSeva deployment means that we need to generate Kubernetes manifests for each individual service. We use the tool called the helm, which is an easy, effective and customizable packaging and deployment solution. So depending on where and which env you initiate the deployment there are 2 modes that you can deploy.‌

1. From - whatever we are trying in this sample exercise so far.

2. Advanced: like Jenkins - Depending on how you want to set up your CI/CD and the expertise the steps will vary, however here you can find how we eGov has set up an exemplar CI/CD on Jenkins and the pipelines are created automatically without any manual intervention.

All content on this page by is licensed under a .

The 2 Stages

Stage 1 : Jenkins Setup

Post infra setup (Kubernetes Cluster), We start with deploying the Jenkins and kaniko-cache-warmer.

Pre-requisites

• Sub Domain to expose CI/CD URL

• GitHub

• With

• (username and password)

• SSL Certificate for the sub-domain

Prepare an <> master config file and <>, you can name this file as you wish which will have the following configurations.

• credentials, secrets (You need to encrypt using and create a ci-secret.yaml separately)

• Check and Update details (like github Oauth app clientId and clientSecret, GitHub user details gitReadSshPrivateKey and gitReadAccessToken etc..)

• To create Jenkins namespace mark this true

• Add your env's kubconfigs under kubConfigs like

• KubeConfig env's name and deploymentJobs name from ci.yaml should be the same

• Update the and repo name with your forked repo name and provide read-only access to github user to those repo's.

• SSL Certificate for the sub-domain

You have launched the Jenkins. You can access the same through your sub-domain which you configured in ci.yaml.

The Jenkins CI pipeline is configured and managed 'as code'.

​Example URL - https://<Jenkins_domain>​

Stage 2: Continuous Integration (CI)

Since there are many services and the development code is part of various git repos, you need to understand the concept of cicd-as-service which is open-sourced. This page also guides you through the process of creating a CI/CD pipeline.

As a developer - To integrate any new service/app to the CI/CD below is the starting point:

Once the desired service is ready for the integration: decide the service name, type of service, whether DB migration is required or not. While you commit the source code of the service to the git repository, the following file should be added with the relevant details which are mentioned below:

Build-config.yml –It is present under the build directory in each repository

This file contains the below details which are used for creating the automated Jenkins pipeline job for your newly created service.

While integrating a new service/app, the above content needs to be added in the build-config.yml file of that app repository. For example: If we are onboarding a new service called egov-test, then the build-config.yml should be added as mentioned below.

If a job requires multiple images to be created (DB Migration) then it should be added as below,

Note - If a new repository is created then the build-config.yml should be created under the build folder and then the config values are added to it.

The git repository URL is then added to the Job Builder parameters

When the Jenkins Job => job builder is executed the CI Pipeline gets created automatically based on the above details in build-config.yml. Eg: egov-test job will be created under the core-services folder in Jenkins because the “build-config was edited under core-services” And it should be the “master” branch only. Once the pipeline job is created, it can be executed for any feature branch with build parameters (Specifying which branch to be built – master or any feature branch).

As a result of the pipeline execution, the respective app/service docker image will be built and pushed to the Docker repository.

If git repository URL is available build the Job-Builder Job

If the git repository URL is not available ask the Devops team to add it.

Continuous Deployment (CD)‌

The services deployed and managed on a Kubernetes cluster in cloud platforms like AWS, Azure, GCP, OpenStack, etc. Here, we use helm charts to manage and generate the Kubernetes manifest files and use them for further deployment to the respective Kubernetes cluster. Each service is created as charts which will have the below-mentioned files in them.

To deploy a new service, we need to create the helm chart for it. The chart should be created under the charts/helm directory in iFix-DevOps repository.

We have an automatic helm chart generator utility that needs to be installed on the local machine, the utility prompts for user inputs about the newly developed service (app specifications) for creating the helm chart. The requested chart with the configuration values (created based on the inputs provided) will be created for the user.

‌ Name of the service? test-service Application Type? NA Kubernetes health checks to be enabled? Yes Flyway DB migration container necessary? No, Expose service to the internet? Yes, Route through API gateway [zuul] No Context path? hello‌

The generated chart will have the following files.

This chart can also be modified further based on user requirements.

The Deployment of manifests to the Kubernetes cluster is made very simple and easy. We have Jenkins Jobs for each state and are environment-specific. We need to provide the image name or the service name in the respective Jenkins deployment job.

Enter a caption for this image (optional)

Enter a caption for this image (optional)

‌The deployment Jenkins job internally performs the following operations,‌

• Reads the image name or the service name given and finds the chart that is specific to it.

• Generates the Kubernetes manifests files from the chart using the helm template engine.

• Execute the deployment manifest with the specified docker image(s) to the Kubernetes cluster.

The 2 Stages

Essentially, there are 2 stages that should allow you to use the full potential of DeploymentConfig and pipeline-as-code.

Stage 1: Clone the DevOps repo, choose your iFix product branch as mGramSeva‌

Prepare an <env.yaml> master config file, you can name this file as you wish which will have the following configurations, this env file needs to be in line with your cluster name.‌

• each service global, local env variables

• credentials, secrets (You need to encrypt using sops and create a <env>-secret.yaml separately)

• Number of replicas/scale of individual services (Depending on whether dev or prod)

• mdms, config repos (Master Data, ULB, Tenant details, Users, etc)

• sms g/w, email g/w, payment g/w

• GMap key (In case you are using Google Map services in your PGR, PT, TL, etc)

• S3 Bucket for Filestore

• URL/DNS on which the DIGIT will be exposed

• SSL Certificate for the above URL

• End-points configs (Internal/external)

Stage 2: Run the mGramSeva_setup deployment script and simply answer the questions that it asks

cd iFix-DevOps/deploy-as-code/egov-deployer

go run mgramseva_setup.go

#Be prepared for the following questions
1. Do you have the Kubernetes Setup?
2. Provide the path of the intented env kubeconfig file
3. Which version of the DIGIT that you want to install
4. What mGramSeva Modules that you choose to install (Choose PGR)
5. All, done, Now do you want to preview the deployment manifests 
6. Are you good to proceed with the DIGIT Installation

All Done.Essentially, DIGIT deployment means that we need to generate Kubernetes manifests for each individual service. We use the tool called helm, which is an easy, effective and customizable packaging and deployment solution. So depending on where and which env you initiate the deployment there are 2 modes that you can deploy.