Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Find the pre-requisites for deploying DIGIT platform services on AWS
The Amazon Elastic Kubernetes Service (EKS) is one of the AWS services for deploying, managing and scaling any distributed and containerized workloads. Here we can provision the EKS cluster on AWS from the ground up using terraform (infra-as-code) and then deploy the DIGIT platform services as config-as-code using Helm.
Know about EKS: https://www.youtube.com/watch?v=SsUnPWp5ilc
Know what is terraform: https://youtu.be/h970ZBgKINg
AWS account with admin access to provision EKS Service. You can always subscribe to a free AWS account to learn the basics and try, but there is a limit to what is offered as free. For this demo, you need a commercial subscription to the EKS service. If you want to try for a day or two, it might cost you about Rs 500 - 1000.
Note: Post the Demo (for eGov internal folks only) - request for the AWS access for 4 hrs. Time-bound access to eGov's training AWS account is available upon request and the available number of slots per day).
Install kubectl (any version) on the local machine - it helps in interaction with the Kubernetes cluster.
Install Helm - this helps package the services along with the configurations, environments, secrets, etc into Kubernetes manifests.
Please refer to tfswitch documentation for different platforms. Terraform version 0.14.10 can be installed directly as well.
5. Run tfswitch and it will show a list of terraform versions. Scroll down and select terraform version (0.14.10) for the Infra-as-code (IaC) to provision cloud resources as code. This provides the desired resource graph and also helps destroy the cluster in one go.
Steps to setup the AWS account for deployment
Follow the details below to set up your AWS account before you proceed with the DIGIT deployment.
Install AWS CLI on your local machine so that you can use AWS CLI commands to provision and manage the cloud resources on your account.
Install AWS IAM Authenticator - it helps you authenticate your connection from your local machine so that you can deploy DIGIT services.
When you have the command line access configured, everything is set for you to proceed with the terraform to provision the DIGIT Infra-as-code.
Explore the basics of AWS Elastic Kubernetes Service
Steps to prepare the deployment configuration file
It's important to prepare a global deployment configuration yaml file that contains all necessary user-specific custom values like URL, gateways, persistent storage ids, DB details etc.
Know the basics of Kubernetes: https://www.youtube.com/watch?v=PH-2FfFD2PU&t=3s
Know the basics of kubectl commands
Know kubernetes manifests: https://www.youtube.com/watch?v=ohSUtEfDefc
Know how to manage env values, secrets of any service deployed in kubernetes https://www.youtube.com/watch?v=OW244LxB4oI
Know how to port forward to a pod running inside k8s cluster and work locally https://www.youtube.com/watch?v=TT3nd5n5Yus
Know sops to secure your keys/creds: https://www.youtube.com/watch?v=DWzJ87KbwxA
Post-Kubernetes Cluster setup, the deployment has got 2 stages. As part of this sample exercise, we can deploy PGR and show what are the various configurations required, however deployment steps are similar for all other modules too, just that the prerequisites differ depending on the feature like SMS Gateway, Payment Gateway, etc
Navigate to the following file in your local machine from the previously cloned DevOps git repository.
Step 1: Clone the following DIGIT-DevOps repo (If not already done as part of Infra setup), you may need to install git and then run git clone it to your machine.
root@ip:/# git clone -b release https://github.com/egovernments/DIGIT-DevOps
Step 2: After cloning the repo CD into the folder DIGIT-DevOps and type the "code ." command that will open the visual editor and opens all the files from the repo DIGIT-DevOps
Step 3: Update the deployment config file with your details, you can use the following template egov-demo-sample.
Here you need to replace the following as per your values
Important: Add your domain name here, which you want to use for accessing DIGIT. ( Do not use the dummy domain )
Important: As per your cloud provider uncomment the related backbone services (Kafka, Zk, elasticsearch, etc) and comment on others. As per your cloud provider, you have to add the volume_ids/diskURI/iqn and zone/diskName/targetPortal in egov-demo.yaml that you got as a terraform output or from sdc team (Kafka, Zk, elasticsearch, etc)
SMS gateway to receive OTP, transaction mobile notification, etc.
MDMS, Config repo URL, here is where you provide master data, tenants and various user/role access details.
GMap key for the location service
Payment gateway, in case you use PT, TL, etc
Step 4: Update your credentials and sensitive data in the secret file as per your details.
credentials, secrets (You need to encrypt using sops and create a <env>-secret.yaml separately)
SOPS expects an encryption key to use it to encrypt/decrypt a specified plain text and keep the details secured, there are a couple of options which you can use to generate the encryption key
Option 1: Generate PGP keys https://fedingo.com/how-to-generate-pgp-key-in-ubuntu
Option 2: Create AWS KMS keys when you want to use the AWS cloud provider.
Once you generate your encryption key, create a .sops.yaml configuration file under the /helm directory of the cloned repo to define which keys are used for which specific file. refer to the SOP doc for info.
Note: For demo purposes, you can use the egov-demo-secrets.yaml as it is without sops configuration, but make sure you update your specific details like Git SSH, URL etc. When you decide to push these configurations into any git or public space, please make sure you follow the sops configuration mentioned in this article sops to encrypt your secrets.
Step 5: Important: Fork the following repos that contain the master data and default configs which you would customize as per your specific implementation later. Like (Master Data, ULB, Tenant details, Users, etc) to your respective GitHub organization account.
Once you fork the repos into your GitHub organization account, Create a github user account, and generate ssh authentication key( generate new SSH key and add it to above user account.
New github users should be enabled to access the earlier forked repos
Add the ssh private key that you generated in the previous step to egov-demo-secrets.yaml under the git-sync section.
Modify the services git-Sync repo and branch with your fork repo and branch in egov-demo.yaml
Step 6: Update the deployment configs for the below as per your specification:
Number of replicas/scale of each individual service (Depending on whether dev or prod load)
You must update sms gateway, email gateway, and payment gateway details for the notification and payment gateway services, etc.
Update the config, MDMS github repos wherever marked
Update GMap key (In case you are using Google Map services in your PGR, PT, TL, etc)
Create one private s3 Bucket for Filestore and one public bucket for logos. Add the bucket details respectively and create an IAM user with the s3 bucket access. Add IAM user details to <env-secrets.yaml>.
URL/DNS on which the DIGIT will be exposed.
SSL certificate for the above URL.
Any specific endpoints configs (Internal/external)
Provision infra for DIGIT on AWS using Terraform
The Amazon Elastic Kubernetes Service (EKS) is one of the AWS services for deploying, managing, and scaling any distributed and containerized workloads. Here we can provision the EKS cluster on AWS from the ground up and using an automated way (infra-as-code) using terraform and then deploy the DIGIT services config-as-code using Helm.
Know about EKS: https://www.youtube.com/watch?v=SsUnPWp5ilc
Know what is terraform: https://youtu.be/h970ZBgKINg
There are multiple options available to deploy the solution to the cloud. Here we provide the steps to use terraform Infra-as-code.
Before we provision the cloud resources, we need to understand and be sure about what resources need to be provisioned by terraform to deploy DIGIT. The following picture shows the various key components. (EKS, Worker Nodes, PostGres DB, EBS Volumes, Load Balancer).
Considering the above deployment architecture, the following is the resource graph that we are going to provision using terraform in a standard way so that every time and for every env, it'll have the same infra.
EKS Control Plane (Kubernetes Master)
Work node group (VMs with the estimated number of vCPUs, Memory)
EBS Volumes (Persistent Volumes)
RDS (PostGres)
VPCs (Private network)
Users to access, deploy and read-only
(Optional) Create your own keybase key before you run the terraform
Use this URL https://keybase.io/ to create your own PGP key, this will create both public and private keys in your machine, upload the public key into the keybase account that you have just created, and give a name to it and ensure that you mention that in your terraform. This allows encrypting of all sensitive information.
Example user keybase user in eGov case is "egovterraform" needs to be created and has to upload the public key here - https://keybase.io/egovterraform/pgp_keys.asc
you can use this portal to Decrypt your secret key. To decrypt PGP Message, Upload the PGP Message, PGP Private Key and Passphrase.
Fork the DIGIT-DevOps repository into your organization account using the GitHub web portal. Make sure to add the right users to the repository. Clone the forked DIGIT-DevOps repository (not the egovernments one). Navigate to the sample-aws directory which contains the sample AWS infra provisioning script.
The sample-aws terraform script is provided as a helper/guide. An experienced DevOps can choose to modify or customize this as per the organization's infra needs.
Create Terraform backend to specify the location of the backend Terraform state file on S3 and the DynamoDB table used for the state file locking. This step is optional. S3 buckets have to be created outside of the Terraform script.
The remote state is simply storing that state file remotely, rather than on your local filesystem. In an enterprise project and/or if Terraform is used by a team, it is recommended to set up and use the remote state.
In the sample-aws/remote-state/main.tf file, specify the s3 bucket to store all the states of the execution to keep track.
The terraform script once executed performs all of the below infrastructure setups.
Amazon EKS requires subnets must be in at least two different availability zones.
Create AWS VPC (Virtual Private Cloud).
Create two public and two private Subnets in different availability zones.
Create an Internet Gateway to provide internet access for services within VPC.
Create NAT Gateway in public subnets. It is used in private subnets to allow services to connect to the internet.
Create Routing Tables and associate subnets with them. Add required routing rules.
Create Security Groups and associate subnets with them. Add required routing rules.
EKS cluster setup
The main.tf inside the sample-aws folder contains the detailed resource definitions that need to be provisioned, please have a look at it.
Navigate to the directory: DIGIT-DevOps/Infra-as-code/terraform/sample-aws. Configurations are defined in variables.tf and provide the environment-specific cloud requirements.
Following are the values that you need to replace in the following files. The blank ones will be prompted for inputs while execution.
cluster_name - provide your EKS cluster name here.
availability_zones - This is a comma-separated list. If you would like your infra to have multi-AZ setup, please provide multiple zones here. If you provide a single zone, all infra will be provisioned within that zone. For example:
3. bucket_name - if you've created a special S3 bucket to store Terraform state.
4. dbname - Any DB name of your choice. Note that this CANNOT have hyphens or other special characters. Underscore is permitted. Example: digit_test
All other variables are default and can be modified if the admin is knowledgeable about it.
5. In the providers.tf file in the same directory, modify the "profile" variable to point to the AWS profile that was created in Step 3.
Make sure your AWS session tokens are up to date in the /user/home/.aws/credentials file
Before running Terraform, make sure to clean up .terraform.lock.hcl, .terraform, terraform.tfstate files if you are starting from scratch.
Once you have finished declaring the resources, you can deploy all resources.
Let's begin to run the terraform scripts to provision infra required to Deploy DIGIT on AWS.
First CD into the following directory and run the following command to create the remote state.
Once the remote state is created, you are ready to provision DIGIT infra. Please run the following commands:
Important:
DB password will be asked for in the application stage. Please remember the password you have provided. It should be at least 8 characters long. Otherwise, RDS provisioning will fail.
The output of the apply command will be displayed on the console. Store this in a file somewhere. Values from this file will be used in the next step of deployment.
2. Use this link to get the kubeconfig from EKS to get the kubeconfig file for the cluster. The region code is the default region provided in the availability zones in variables.tf. Eg. ap-south-1. EKS cluster name also should've been filled in variables.tf.
3. Finally, verify that you are able to connect to the cluster by running the following command
At this point, your basic infra has been provisioned. Please move to the next step to install DIGIT.
To destroy previously-created infrastructure with Terraform, run the command below:
ELB is not deployed via Terraform. ELB has created at deployment time by the setup of Kubernetes Ingress. This has to be deleted manually by deleting the ingress service.
kubectl delete deployment nginx-ingress-controller -n <namespace>
kubectl delete svc nginx-ingress-controller -n <namespace>
Note: Namespace can be one of egov or jenkins.
Delete S3 buckets manually from the AWS console and also verify if ELB got deleted.
In case of if ELB is not deleted, you need to delete ELB from AWS console.
Run terraform destroy.
Sometimes all artefacts that are associated with a deployment cannot be deleted through Terraform. For example, RDS instances might have to be deleted manually. It is recommended to log in to the AWS management console and look through the infra to delete any remnants.
If you have any questions please write to us.
Make sure to use the appropriate discussion category and labels to address the issues better.
This page details the steps to deploy the core platform services and reference applications.
The steps here can be used to deploy:
DIGIT core platform services
Public Grievance & Redressal module
Trade Licence module
Property Tax module
Water & Sewerage module etc.
DIGIT uses (required v1.13.3) automated scripts to deploy the builds onto Kubernetes - or or
All DIGIT services are packaged using helm charts
is a CLI to connect to the Kubernetes cluster from your machine
Install for making API calls
IDE Code for better code/configuration editing capabilities
to run digit bootstrap scripts
Once all the deployments configs are ready, run the command given below. Input the necessary details as prompted on the screen and the interactive installer will take care of the rest.
All done, wait and watch for 10 min. The DIGIT setup is complete, and the application will run on the URL.
Note:
If you do not have your domain yet, you can edit the host file entries and map the nginx-ingress-service load balancer id like below
When you find it, add the following lines to the host file, save and close it.
aws-load-balancer-id digit.try.com
You can now test the DIGIT application status in the command prompt/terminal using the command below.
Note: Initially pgr-services would be in crashloopbackoff state, but after performing the post-deployment steps the pgr-services will start running.
After deploying your environment config into the EC2 cluster, we have to add the security group ID of the instance to RDS.
Follow the steps below:
Go to the AWS console and search for EC2.
Click on Instances and select the Instance ID which you created.
Scroll down and go to Security and copy the Security Group ID. It starts with sg-xxxxxxxxxxxxxxxxx.
In the search bar, search for RDS and then go to Databases. Choose the db you had created.
Scroll down for Security Group rules and click any one sg. It redirects to another tab.
Scroll down and click on Edit inbound rules. Click on Add rule.
Change the Type to Postgresql and paste the copied sg-xxxxxxxxxxxxxxxxx beside the custom and click on save rules.
Run the egov-deployer golang script from the .
If you have a GoDaddy account or similar and a DNS records edit access you can map the load balancer id to desired DNS. Create a record with the load balancer ID and domain.
Steps to bootstrap DIGIT
Post-deployment, the application can now be accessed from the configured domain. This page provides the bootstrapping steps.
To try out employee login, let us create a sample tenant, city, and user to log in and assign the LME employee role through the seed script.
Perform the kubectl port-forwarding of the egov-user service running from the Kubernetes cluster to your localhost. This provides access to egov-user service and allows users to interact with the API directly.
2. Seed the sample data
Ensure the Postman is installed to run the following seed data API. if not, Install postman on your local machine.
Import the following Postman collection into the Postman and run it. This contains the seed data that enables sample test users and localisation data.
Execute the below commands to test your local machine's Kubernetes operations through kubectl.
You have successfully completed the DIGIT Infra, deployment setup and installed a DIGIT - PGR module.
Use the below link in the browser -
Use the below credentials to login into the complaint section
Username: GRO
Password: eGov@4321
City: CITYA
By now we have successfully completed the DIGIT setup on the cloud, use the URL you mentioned in your env.yaml.
Eg: https://mysetup.digit.org and create a grievance to ensure the PGR module deployed is working fine. Refer to the below product documentation for the steps.
Credentials:
Citizen: You can use your default mobile number (9999999999) to sign in using the default Mobile OTP 123456.
Employee: Username: GRO and password: eGov@4321
Post grievance creation and assignment of the same to LME, capture the screenshot of the same and share it to ensure your setup is working fine. Post validation, the PGR functionality shares the API response of the following request to assess the correctness of successful DIGIT PGR Deployment.
Follow the steps below to clean up the DIGIT setup, if required. This will delete the entire cluster and other cloud resources that were provisioned for the DIGIT Setup.
Run the command below to destroy previously-created infrastructure using Terraform -
ELB is not deployed via Terraform. ELB has created at deployment time by the setup of Kubernetes Ingress. This has to be deleted manually by deleting the ingress service.
kubectl delete deployment nginx-ingress-controller -n <namespace>
kubectl delete svc nginx-ingress-controller -n <namespace>
Note: Namespace can be either provided by egov or Jenkins.
Delete S3 buckets manually from the AWS console and also verify if the ELB got deleted.
In case the ELB is not deleted, you need to delete ELB from the AWS console.
Run terraform destroy.
All done, we have successfully created infra on the cloud, deployed DIGIT, bootstrapped DIGIT, performed a transaction on PGR and finally destroyed the cluster.
