Installation GuideDIGIT - Service Delivery and Governance PlatformLoading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
mGramSeva Quickstart, this is not for a production
Quickstart Installation helps you jump-start with the mGramSeva basic installation with limited functionalities.
mGramSeva is a distributed microservice-based platform that comprises many services that are containerized. Depending upon the required features, the specific services can be run on any container-supported orchestration platform like docker-compose, Kubernetes, etc.
The Quickstart guide covers the installation steps for basic services to get the platform up. Before setting up mGramSeva, create a lightweight Kubernetes cluster called k3d on a local machine with specified H/W requirements. The H/W requirements are listed below to ensure before we proceed further.
To provision a lightweight Kubernetes cluster, please follow the instructions below in context to your OS and install the k3d on your machine.
min 4 vCPUs (recommended 8)
min 8GiB of RAM (recommended 16)
min 30GiB of HDD (recommended 30+)
Linux distribution running in a VM or bare metal
Ubuntu 18.04 or Debian 10 (VM or bare metal)
Install Docker
Install kubectl on Linux
Open terminal and Install k3d on Linux using the below command
OSX or Mac
Docker Desktop local Kubernetes cluster enabled
Install kubectl on Mac
Install k3d on Mac, on terminal use Homebrew (Homebrew is available for MacOS) using the below command
Windows 10 or above
Docker Desktop for windows need to be installed
Install kubectl on Windows
Install Chocolatey package manager for windows
Install GitBash as an alternative command prompt that allows most of the Linux commands on windows.
Open gitbash and Install k3d on Windows using the below command
Once the above prerequisites are met, run the following tasks depending upon your OS.
login/ssh into the machine, go to terminal/command prompt and run the following commands as an admin user.
Create /Kube directory and change permission. To use this directory for persistent data mount. This means data from all container logs will be stored here.
Create a cluster with a single master node and 2 agents (Worker Nodes) and mount the pre-created directory (for data persistence).
When cluster creation is successful, get the kubeconfig file, that allows you to connect to the cluster at any time.
Verify the Cluster creation by running the following commands from your local machine where the kubectl is installed. It gives you the sample output as below
You can verify the workers' nodes created by using the following command.
Once the above steps are completed successfully, your Cluster is now up and running ready to proceed with the DIGIT Deployment.
Now that we have the Infra setup to proceed with the DIGIT Deployment. Following are the tools that need to be installed on the machine before proceeding with the DIGIT Services deployment.
What we'll deploy in Quickstart:
mGramSeva core platform services
kubectl is a CLI to connect to the kubernetes cluster from your machine
Install CURL for making API calls
Install Visualstudio IDE Code for better code/configuration editing capabilities
The mGramSeva services deployment configurations are in GitRepo which needs to install git and then git clone it to your local.
Install Postman to run some DIGIT bootstrap scripts
After cloning the repo CD into the folder iFix-DevOps and type the "code ." command that will open the visual editor and opens all the files from the repo iFix-DevOps
Check the sample deployment config file that needs to be configured as per any specific values according to your needs. (For a quick start you can run as it is)
Add the following entries in your host file /etc/hosts depending on your OS, instructions can be found here.
Once the prerequisite setup is complete, go to the following repo, run the command and follow the instructions.
You can now test the DIGIT application status in the command prompt/terminal by using the below command.
iFix Infra Setup & Deployment
The Azure Kubernetes Service (AKS) is one of the Azure services used for deploying, managing, and scaling any distributed and containerized workloads. Here we can provision the AKS cluster on Azure from the ground up and using an automated way (infra-as-code) using and then deploy the DIGIT-iFIX Services config-as-code using .
This quickstart assumes a basic understanding of Kubernetes concepts. For more information, see .
If you don't have an , create a before you begin.
Use the Bash environment in .
If you prefer, the Azure CLI to run CLI reference commands.
If you're using a local installation, sign in to the Azure CLI by using the command. To finish the authentication process, follow the steps displayed in your terminal. For additional sign-in options, see .
When you're prompted, install Azure CLI extensions on first use. For more information about extensions, see .
Run to find the version and dependent libraries that are installed. To upgrade to the latest version, run .
This article requires version 2.0.64 or greater of the Azure CLI. If using Azure Cloud Shell, the latest version is already installed.
The identity you are using to create your cluster has the appropriate minimum permissions. For more details on access and identity for AKS, see .
Install on your local machine that helps you interact with the kubernetes cluster
Install that helps you package the services along with the configurations, envs, secrets, etc into a
Install version (0.14.10) for the Infra-as-code (IaC) to provision cloud resources as code and with desired resource graph and also it helps to destroy the cluster at one go.
Note: Run the commands as administrator if you plan to run the commands in this quickstart locally instead of in Azure Cloud Shell.
Before we provision the cloud resources, we need to understand and be sure about what resources need to be provisioned by terraform to deploy DIGIT. The following picture shows the various key components. (AKS, Worker Nodes, Postgres DB, Volumes, Load Balancer)
Considering the above deployment architecture, the following is the resource graph that we are going to provision using terraform in a standard way so that every time and for every env, it'll have the same infra.
AKS Azure (Kubernetes Service Master)
Work node group (VMs with the estimated number of vCPUs, Memory
Volumes (persistent volumes)
PostGres Database
Virtual Network
Users to access, deploy and read-only
Here we have already written the terraform script that provisions the production-grade DIGIT Infra and can be customized with the specified configuration.
The following main.tf contains the detailed resource definitions that need to be provisioned, please have a look at it.
Dir: iFix-DevOps/Infra-as-code/terraform/aks-ifix-dev
You can define your configurations in variables.tf and provide the environment-specific cloud requirements so that using the same terraform template you can customize the configurations.
Following are the values that you need to mention in the following files, the blank ones will be prompted for inputs while execution.
Now that we know what the terraform script does, the resources graph that it provisions and what custom values should be given with respect to your env.
Let's begin to run the terraform scripts to provision infra required to Deploy DIGIT on AZ.
First CD into the following directory and run the following command 1-by-1 and watch the output closely.
Upon Successful execution following resources gets created which can be verified by the command "terraform output"
Network: Virtual Network.
AKS cluster: with nodepool(s), master(s) & worker node(s).
Storage(s): for es-master, es-data-v1, es-master-infra, es-data-infra-v1, zookeeper, kafka, kafka-infra.
Downloads credentials and configures the Kubernetes CLI to use them.
3. Finally, Verify that you are able to connect to the cluster by running the following command
Choose your infra type and provision the necessary infra before you actually deploy the services
iFIX/mGramSeva is a microservices-based distributed cloud-native application. Each of these context-specific microservices is dockerized and deployed on Kubernetes infrastructure.
It is essential to understand some of the key concepts, benefits and best practices of the Kubernetes platform before we understand the deployment of the iFIX/mGramSeva.
Know the basics of Kubernetes:
Know the commands
Know Kubernetes manifests:
Know how to manage environment values and secrets of any service deployed in Kubernetes
Know how to port forward to a pod running inside k8s cluster and work locally
Know sops to secure your keys/creds:
Choose the target infra type and follow the instructions to set up a Kubernetes cluster before moving on to the deployment.
Before we begin the deployment, it is important to understand the deployment architecture that starts from the source code to the production-ready stage. Deploying and managing Kubernetes have emerged as a streamlined way to deploy containers in the cloud infrastructure. When running Kubernetes at scale, managing, operating, and scaling its infrastructure to maximize cluster utilization can be challenging. There are too many parameters the development team needs to manage and configure. This includes selecting the best instance type and size, determining when to scale up or down, and making sure all of the containers are scheduled and running on the best instances — and that is even before starting to think about cost resource optimization.
The simplest way to get started with the deployment process is to manage deployment configuration as code. Each service deployment configuration is defined as Helm charts and deployed into the Kubernetes cluster. We can collocate the deployment-as-code as source code, leveraging all the benefits of source control including change tracking and branching and then packaging it. The source code repo below contains the deployment-as-code details for iFIX.
Use the command below to clean up the setup cluster. This deletes the entire cluster and other cloud resources that were provisioned for the mGramSeva Infra Setup.
All done, the infra on local, cloud, and deployment of iFIX into the Kubernetes cluster is completed successfully.
The is one of the AWS services for deploying, managing, and scaling any distributed and containerized workloads, here we can provision the EKS cluster on AWS from ground up and using an automated way (infra-as-code) using and then deploy the DIGIT-iFIX Services config-as-code using .
Know about EKS:
Know what is terraform:
with the admin access to provision EKS Service, you can always subscribe to free AWS account to learn the basics and try, but there is a limit to , for this demo you need to have a commercial subscription to the EKS service.
Install on your local machine that helps you interact with the kubernetes cluster
Install that helps you package the services along with the configurations, envs, secrets, etc into a
Install version (0.14.10) for the Infra-as-code (IaC) to provision cloud resources as code and with desired resource graph and also it helps to destroy the cluster at one go.
on your local machine so that you can use aws cli commands to provision and manage the cloud resources on your account.
Install that helps you authenticate your connection from your local machine so that you should be able to deploy DIGIT services.
Use the credentials provided for the Terraform () to connect with your AWS account and provision the cloud resources.
You'll get a Secret Access Key and Access Key ID. Save them safely.
Open the terminal and run the following command. The AWS CLI is already installed and the credentials are saved. (Provide the credentials and you can leave the region and output format blank).
The above will create the following file In your machine as /Users/.aws/credentials
Before we provision the cloud resources, we need to understand and be sure about what resources need to be provisioned by terraform to deploy DIGIT. The following picture shows the various key components. (EKS, Worker Nodes, Postgress DB, EBS Volumes, Load Balancer)
Considering the above deployment architecture, the following is the resource graph that we are going to provision using terraform in a standard way so that every time and for every environment, it'll have the same infra.
EKS Control Plane (Kubernetes Master)
Work node group (VMs with the estimated number of vCPUs, Memory)
Node-pool's (mgramseva and ifix)
EBS Volumes (persistent volumes)
RDS (Postgresql)
VPCs (private network)
Users to access, deploy and read-only
Here we have already written the terraform script that provisions the production-grade DIGIT Infra and can be customized with the specified configuration.
Example:
VPC Resources:
VPC
Subnets
Internet Gateway
Route Table
EKS Cluster Resources:
IAM Role to allow EKS service to manage other AWS services
EC2 Security Group to allow networking traffic with EKS cluster
EKS Cluster
EKS Worker Nodes Resources:
IAM role allowing Kubernetes actions to access other AWS services
EC2 Security Group to allow networking traffic
Data source to fetch the latest EKS worker AMI
AutoScaling Launch Configuration to configure worker instances
AutoScaling Group to launch worker instances
Database
Configuration in this directory creates a set of RDS resources including DB instance, DB subnet group, and DB parameter group.
Storage Module
Configuration in this directory creates EBS volume and attaches it together.
The following main.tf with create s3 bucket to store all the state of the execution to keep track.
iFix-DevOps/Infra-as-code/terraform/sample-eks/remote-state
2. The following main.tf contains the detailed resource definitions that need to be provisioned, please have a look at it.
Dir: iFix-DevOps/Infra-as-code/terraform/sample-eks
Define your configurations in variables.tf. Provide the environment-specific cloud requirements and use the same terraform template to customize the configurations.
The values given below must be mentioned in the following files. The blank ones will be prompted for inputs while execution.
Important: Create your own key base key before you run the terraform
Now that we know what the terraform script does, the resources graph that it provisions and what custom values should be given with respect to your env.
Let's begin to run the Terraform scripts to provision infra required to Deploy DIGIT on AWS.
First CD into the following directory and run the following command 1-by-1 and watch the output closely.
Upon Successful execution following resources get created which can be verified by the command "terraform output"
s3 bucket: to store terraform state.
Network: VPC, security groups.
EKS cluster: with master(s) & worker node(s).
Storage(s): for es-master, es-data-v1, es-master-infra, es-data-infra-v1, zookeeper, kafka, kafka-infra.
3. Verify that you are able to connect to the cluster by running the following command
Infrastructure Setup
iFIX/mGramSeva is a microservices-based distributed cloud-native application. The microservices streamline processes to meet outcomes at scale and speed. Each of the microservices is dockerized and deployed on Kubernetes infrastructure.
It is essential to understand some of the key concepts, benefits and best practices of Kubernetes platform before we understand the deployment of the iFix/mGramSeva.
Know the basics of Kubernetes:
Know the commands
Know Kubernetes manifests:
Know how to manage env values, secrets of any service deployed in Kubernetes
Know how to port forward to a pod running inside k8s cluster and work locally
Know sops to secure your keys/creds:
Choose the target infra type and follow the instructions to set up a Kubernetes cluster before moving on to the deployment.
Before we begin the deployment, it is important to understand the deployment architecture that starts from the source code to the production-ready stage. Deploying and managing Kubernetes have emerged as a streamlined way to deploy containers in the cloud infrastructure. When running Kubernetes at scale, managing, operating, and scaling its infrastructure to maximize cluster utilization can be challenging. There are too many parameters the development team needs to manage and configure. This includes selecting the best instance type and size, determining when to scale up or down, and making sure all of the containers are scheduled and running on the best instances — and that is even before starting to think about cost resource optimization.
The simplest way to get started with the deployment process is to manage deployment configuration as code. Each service deployment configuration is defined as Helm charts and deployed into the Kubernetes cluster. We can collocate the deployment-as-code as source code, leveraging all the benefits of source control including change tracking and branching, and then package it. So below is the source code repo that contains all the deployment-as-code for iFIX.
Clean up the cluster Setup using the command below, if required. This deletes the entire cluster and other cloud resources that were provisioned for the mGramSeva Infra Setup.
All done, the infra on local, cloud, and deployment of iFIX into the Kubernetes cluster is completed successfully.
All mGramSeva services are packaged using helm charts Installing Helm
All content on this page by eGov Foundation is licensed under a Creative Commons Attribution 4.0 International License.
Ideally, one would write the terraform script from the scratch using this .
Let's clone the GitHub repo where the terraform script to provision the AKS cluster is available and below is the structure of the files.
To manage a Kubernetes cluster, use the Kubernetes command-line client, . kubectl is already installed if you use Azure Cloud Shell.
Install kubectl locally using the command:
Configure kubectl to connect to your Kubernetes cluster using the command. The following command:
Uses ~/.kube/config, the default location for the . Specify a different location for your Kubernetes configuration file using --file.
Whola! All set and now you can go with .
All content on this page by is licensed under a .
All content on this page by is licensed under a .
Ideally, one would write the terraform script from the scratch using this .
Let's clone the GitHub repo where the terraform script to provision EKS cluster is available and below is the structure of the files.
Use the URL to . This creates both public and private keys on your machine. Upload the public key into the account that you have just created, give a name to it and ensure that you mention that in your terraform. This allows the encryption of all sensitive information.
Example - the keybase user (in eGov case is "egovterraform") needs to be created and has to be uploaded the public key here -
you can use this to decrypt your secret key. To decrypt PGP Message, upload the PGP Message, PGP Private Key and the Passphrase.
IAM users auth: using the key base to create admin, deployer and the user. Use this URL to , this will create both public and private keys in your machine. Upload the public key into the account that you have just created, give a name to it and ensure that you mention that in your terraform. This allows for encrypting all sensitive information.
Example: keybase user (in eGov case is "egovterraform") needs to be created and has to be uploaded the public key here -
you can use this to decrypt your secret key. To decrypt PGP Message, Upload the PGP Message, PGP Private Key and Passphrase.
2. Use this link to to get the kubeconfig file and be able to connect to the cluster from your local machine so that you should be able to deploy DIGIT services to the cluster.
Whola! All set and now you can go with ..
All content on this page by is licensed under a .
All content on this page by is licensed under a .
Essentially, there are 2 stages that should allow you to use the full potential of DeploymentConfig and pipeline-as-code.
Stage 1: Clone the DevOps repo, choose your iFix product branch as mGramSeva
Prepare an <env.yaml> master config file, you can name this file as you wish which will have the following configurations, this env file needs to be in line with your cluster name.
each service global, local env variables
credentials, secrets (You need to encrypt using sops and create a <env>-secret.yaml separately)
Number of replicas/scale of individual services (Depending on whether dev or prod)
mdms, config repos (Master Data, ULB, Tenant details, Users, etc)
sms g/w, email g/w, payment g/w
GMap key (In case you are using Google Map services in your PGR, PT, TL, etc)
S3 Bucket for Filestore
URL/DNS on which the DIGIT will be exposed
SSL Certificate for the above URL
End-points configs (Internal/external)
Stage 2: Run the mGramSeva_setup deployment script and simply answer the questions that it asks
Only the administrator can generate the API key and secret. The steps for doing this is as follows.
Keycloak console will be available at https://<host-name>/auth. The admin will log in using the username and password secret.
Open the Keycloak console
Near the top-left corner in the realm drop down menu, select Add Realm.
Select the ifix-realm.json file.
After the realm gets created, select the ifix realm from the drop-down near the top-left corner.
Remember to select the ifix realm from the Keycloak console before proceeding.
From the Clients section of Keycloak Admin Console, create a client.
Provide a unique username for the client.
Go to the client's settings
Change Access Type to confidential
Turn on Service Account Enabled
In the Valid Redirect URIs field provide the root URL of the iFIX Instance. (Not important for our purposes but need to set it because it is mandatory)
And Save these changes
In the Service Account Roles tab, assign the role "fiscal-event-producer"
In the Mappers tab, create a new mapper to associate the client with a tenantId
Select Mapper Type to be "Hardcoded claim"
In Token Claim Name, write "tenantId"
In Claim value, write the under which the client is being created. (For example, "pb")
Set Name same as Token Claim Name i.e. "tenantId"
Select Claim Json Type to be "String"
Now you can get the credentials from the Credentials tab and configure them in the client's system.
CI/CD setup
Post infra setup (Kubernetes Cluster), We start with deploying the Jenkins and kaniko-cache-warmer.
Sub Domain to expose CI/CD URL
GitHub Oauth App
Docker hub account details (username and password)
SSL Certificate for the sub-domain
Prepare an <ci.yaml> master config file and <ci-secrets.yaml>, you can name this file as you wish which will have the following configurations.
credentials, secrets (You need to encrypt using sops and create a ci-secret.yaml separately)
Check and Update ci-secrets.yaml details (like github Oauth app clientId and clientSecret, GitHub user details gitReadSshPrivateKey and gitReadAccessToken etc..)
To create Jenkins namespace mark this flag true
Add your env's kubconfigs under kubConfigs like https://github.com/misdwss/iFix-DevOps/blob/mgramseva/deploy-as-code/helm/environments/ci-secrets.yaml#L19
KubeConfig env's name and deploymentJobs name from ci.yaml should be the same
Update the CIOps and DIGIT-DevOps repo name with your forked repo name and provide read-only access to github user to those repo's.
SSL Certificate for the sub-domain
You have launched the Jenkins. You can access the same through your sub-domain which you configured in ci.yaml.
The Jenkins CI pipeline is configured and managed 'as code'.
Example URL - https://<Jenkins_domain>
Since there are many services and the development code is part of various git repos, you need to understand the concept of cicd-as-service which is open-sourced. This page also guides you through the process of creating a CI/CD pipeline.
As a developer - To integrate any new service/app to the CI/CD below is the starting point:
Once the desired service is ready for the integration: decide the service name, type of service, whether DB migration is required or not. While you commit the source code of the service to the git repository, the following file should be added with the relevant details which are mentioned below:
Build-config.yml –It is present under the build directory in each repository
This file contains the below details which are used for creating the automated Jenkins pipeline job for your newly created service.
While integrating a new service/app, the above content needs to be added in the build-config.yml file of that app repository. For example: If we are onboarding a new service called egov-test, then the build-config.yml should be added as mentioned below.
If a job requires multiple images to be created (DB Migration) then it should be added as below,
Note - If a new repository is created then the build-config.yml should be created under the build folder and then the config values are added to it.
The git repository URL is then added to the Job Builder parameters
When the Jenkins Job => job builder is executed the CI Pipeline gets created automatically based on the above details in build-config.yml. Eg: egov-test job will be created under the core-services folder in Jenkins because the “build-config was edited under core-services” And it should be the “master” branch only. Once the pipeline job is created, it can be executed for any feature branch with build parameters (Specifying which branch to be built – master or any feature branch).
As a result of the pipeline execution, the respective app/service docker image will be built and pushed to the Docker repository.
Job Builder – Job Builder is a Generic Jenkins job that creates the Jenkins pipeline automatically which are then used to build the application, create the docker image of it and push the image to the docker repository. The Job Builder job requires the git repository URL as a parameter. It clones the respective git repository and reads the build/build-config.yml file for each git repository and uses it to create the service build job.
Check git repository URL is available in ci.yaml
If git repository URL is available build the Job-Builder Job
If the git repository URL is not available ask the Devops team to add it.
The services deployed and managed on a Kubernetes cluster in cloud platforms like AWS, Azure, GCP, OpenStack, etc. Here, we use helm charts to manage and generate the Kubernetes manifest files and use them for further deployment to the respective Kubernetes cluster. Each service is created as charts which will have the below-mentioned files in them.
To deploy a new service, we need to create the helm chart for it. The chart should be created under the charts/helm directory in iFix-DevOps repository.
We have an automatic helm chart generator utility that needs to be installed on the local machine, the utility prompts for user inputs about the newly developed service (app specifications) for creating the helm chart. The requested chart with the configuration values (created based on the inputs provided) will be created for the user.
Name of the service? test-service Application Type? NA Kubernetes health checks to be enabled? Yes Flyway DB migration container necessary? No, Expose service to the internet? Yes, Route through API gateway [zuul] No Context path? hello
The generated chart will have the following files.
This chart can also be modified further based on user requirements.
The Deployment of manifests to the Kubernetes cluster is made very simple and easy. We have Jenkins Jobs for each state and are environment-specific. We need to provide the image name or the service name in the respective Jenkins deployment job.
Enter a caption for this image (optional)
Enter a caption for this image (optional)
The deployment Jenkins job internally performs the following operations,
Reads the image name or the service name given and finds the chart that is specific to it.
Generates the Kubernetes manifests files from the chart using the helm template engine.
Execute the deployment manifest with the specified docker image(s) to the Kubernetes cluster.
mGramSeva Deployment
Post infra setup (Kubernetes Cluster), there starts the deployment process.
Pipeline as code is a practice of defining deployment pipelines through source code, such as Git. Pipeline as code is part of a larger “as code” movement that includes infrastructure as code. Teams can configure builds, tests, and deployment in code that is trackable and stored in a centralized source repository. Teams can use a declarative YAML approach or a vendor-specific programming language, such as Jenkins and Groovy, but the premise remains the same.
A pipeline as code file specifies the stages, jobs, and actions for a pipeline to perform. Because the file is versioned, changes in pipeline code can be tested in branches with the corresponding application release.
The pipeline as code model of creating continuous integration pipelines is an industry best practice, but deployment pipelines used to be created very differently.
The deployment process has got 2 stages and 2 modes. We can see the modes first and then the stages.
Essentially, mGramSeva deployment means that we need to generate Kubernetes manifests for each individual service. We use the tool called the helm, which is an easy, effective and customizable packaging and deployment solution. So depending on where and which env you initiate the deployment there are 2 modes that you can deploy.
From local machine - whatever we are trying in this sample exercise so far.
Advanced: Setup CI/CD System like Jenkins - Depending on how you want to set up your CI/CD and the expertise the steps will vary, however here you can find how we eGov has set up an exemplar CI/CD on Jenkins and the pipelines are created automatically without any manual intervention.
All content on this page by eGov Foundation is licensed under a Creative Commons Attribution 4.0 International License.
All content on this page by eGov Foundation is licensed under a Creative Commons Attribution 4.0 International License.
All content on this page by eGov Foundation is licensed under a Creative Commons Attribution 4.0 International License.
All content on this page by eGov Foundation is licensed under a Creative Commons Attribution 4.0 International License.
