Overview

On this page, you will find the steps on how to create a database dump.

Steps

1. To create a database dump, execute the dump command (given below) in the playground pod.

   kubectl get pods -n playground

   kubectl exec <playground-pod-name> -it -n playground bash

2. Use the below command to take a backup.

   pg_dump -Fp --no-acl --no-owner --no-privileges -h <db-host> egov_db -U dbusername > backup.sql

   gzip backup.sql.gz backup.sql

3. Copy the zip file to your local machine using the below command.

   kubectl cp <playground-pod-name>:/backup.sql.gz backup.sql.gz -n playground

• GitHub is an open source tool which helps the developers to manage, store, track and control changes in their code. If we want to clone(copy) the data from GItHub we need to install Git.

• There are some alternatives for GitHub like GitLab, Bitbucket. But many developers prefer GitHub because it's more popular and they are used to the navigation. So we are using Git in DIGIT

• GitHub is used to create Individual projects.

Checking for Git:

To check whether Git is already installed in your systems, open in terminal.

• If you are in Mac, look for the command prompt application called "Terminal".

• If you are in Windows, open the windows command prompt or "Git Bash".

• Type the below command:

git version

Installing Git on Linux:

Ubuntu:

In Ubuntu using terminal we can directly install Git using terminal.

• Go to command prompt shell and run the following command to make sure everything is up-to-date.

sudo apt-get update

• After that run the following command to install Git.

sudo apt-get install git-all

• Once the command output has completed, verify the installation using

git version

Installing Git on Windows and Mac:

• Go to the following page to download the Git latest version: For Windows: https://gitforwindows.org/ For Mac: https://sourceforge.net/projects/git-osx-installer/files/git-2.23.0-intel-universal-mavericks.dmg/download?use_mirror=autoselect

• Once the installation is done, open the windows command prompt or Git Bash and type

git version

This guide provides a step-by-step guide to monitoring and operating the DIGIT Platform and services in production.

• You can fetch updates from or submit changes to the original repository with pull requests

• A fork often occurs when a developer becomes dissatisfied or disillusioned with the direction of a project and wants to detach their work from that of the original project.

While we are adopting the Microservices architecture, it also demands to have an efficient CI/CD tools like jenkins. Along the cloud-native application developement and deployment jenkins can also be run cloud-native.

Since all processes, including software build, test and deployment, are performed every two or four weeks, this is an ideal playground for automation tools like Jenkins: After the developer commits a code change to the repository, Jenkins will detect this change and will trigger the build and test process. So Let's setup Jenkins as a docker container. Step-by-step.

Installing Jenkins the Docker Way

Tools and Versions used

• VM or EC2 Instance or a Standalone on-premisis machin

• Docker 1.12.1

• Jenkins 2.32.2

• Job DSL Plugin 1.58

Prerequisites:

• Ubuntu or an Liniux Machine

• Free RAM for the a VM/Machine >~ 4 GB.

• Docker Host is available.

• Tested with 3 vCPU (2 vCPU might work as well).

Step 1: Install a Docker on the Host you provisioned for jenkins and Connect to the Host via SSH

If you are using an host already has docker installed, you can skip this step. Make sure that your host has enough memory.

We will run Jenkins in a Docker container in order to allow for maximum interoperability. This way, we always can use the latest Jenkins version without the need to control the java version used.

If you are new to Docker, you might want to read this blog post.

Installing Docker on Windows and Mac can be a real challenge, but possible: here we will see an efficient way by using linux machine.

Prerequisites of this step:

• I recommend to have direct access to the Internet: via Firewall, but without HTTP proxy.

• Administration rights on you computer.

Step 2: Download Jenkins Image

This extra download step is optional, since the Docker image will be downloaded automatically in step 3, if it is not already found on the system:

(dockerhost)$ sudo docker pull jenkins
Using default tag: latest
latest: Pulling from library/jenkins
Digest: sha256:8820149b54bfc5d05146b82150b5fdab583eef3e0499fb4ed630f77647a42942
Status: Image is up to date for jenkins:latest

The version of the downloaded Jenkins image can be checked with following command:

(dockerhost)$ sudo docker run -it --rm jenkins --version
2.19.3

We are using version 2.9.13 currently. If you want to make sure that you use the exact same version as I have used in this blog, you can use the imagename jenkins:2.19.3 in all docker commands instead of jenkins only.

Note: The content of the jenkins image can be reviewed on this link. There, we find that the image has an entrypoint /bin/tini -- /usr/local/bin/jenkins.sh, which we could override with the --entrypoint bash option, if we wanted to start a bash shell in the jenkins image. However, in Step 3, we will keep the entrypoint for now.

Step 3: Start Jenkins in interactive Terminal Mode

In this step, we will run Jenkins interactively (with -it switch instead of -d switch) to better see, what is happening. But first, we check that the port we will use is free:

(dockerhost)$ sudo docker ps
CONTAINER ID        IMAGE                    COMMAND                  CREATED             STATUS              PORTS                                            NAMES
0ec82b4ca2fd        google/cadvisor:latest   "/usr/bin/cadvisor -l"   2 days ago          Up 2 days           0.0.0.0:8080->8080/tcp                           cadvisor
...

Since we see that one of the standard ports of Jenkins (8080, 50000) is already occupied and I do not want to confuse the readers of this blog post by mapping the port to another host port, I just stop the cadvisor container for this „hello world“:

(dockerhost)$ sudo docker stop cadvisor
cadvisor

Jenkins will be in need of a persistent storage. For that, we create a new folder on the Docker host:

(dockerhost)$ mkdir jenkins_home; cd jenkins_home

Note: The content of the jenkins image can be reviewed on this link. There, we find that the image has an entrypoint /bin/tini -- /usr/local/bin/jenkins.sh, which we could override with the --entrypoint bash option, if we wanted to start a bash shell in the jenkins image.

We start the Jenkins container with the jenkins_home Docker host volume mapped to /var/jenkins_home:

(dockerhost)$ sudo docker run -it --rm --name jenkins -p8080:8080 -p50000:50000 -v`pwd`:/var/jenkins_home jenkins
Running from: /usr/share/jenkins/jenkins.war
webroot: EnvVars.masterEnvVars.get("JENKINS_HOME")
Nov 30, 2016 6:12:14 PM Main deleteWinstoneTempContents
WARNING: Failed to delete the temporary Winstone file /tmp/winstone/jenkins.war
Nov 30, 2016 6:12:14 PM org.eclipse.jetty.util.log.JavaUtilLog info
INFO: Logging initialized @347ms
Nov 30, 2016 6:12:14 PM winstone.Logger logInternal
INFO: Beginning extraction from war file
Nov 30, 2016 6:12:14 PM org.eclipse.jetty.util.log.JavaUtilLog warn
WARNING: Empty contextPath
Nov 30, 2016 6:12:14 PM org.eclipse.jetty.util.log.JavaUtilLog info
INFO: jetty-9.2.z-SNAPSHOT
Nov 30, 2016 6:12:16 PM org.eclipse.jetty.util.log.JavaUtilLog info
INFO: NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet
Jenkins home directory: /var/jenkins_home found at: EnvVars.masterEnvVars.get("JENKINS_HOME")
Nov 30, 2016 6:12:17 PM org.eclipse.jetty.util.log.JavaUtilLog info
INFO: Started w.@7674f035{/,file:/var/jenkins_home/war/,AVAILABLE}{/var/jenkins_home/war}
Nov 30, 2016 6:12:17 PM org.eclipse.jetty.util.log.JavaUtilLog info
INFO: Started ServerConnector@548d708a{HTTP/1.1}{0.0.0.0:8080}
Nov 30, 2016 6:12:17 PM org.eclipse.jetty.util.log.JavaUtilLog info
INFO: Started @3258ms
Nov 30, 2016 6:12:17 PM winstone.Logger logInternal
INFO: Winstone Servlet Engine v2.0 running: controlPort=disabled
Nov 30, 2016 6:12:17 PM jenkins.InitReactorRunner$1 onAttained
INFO: Started initialization
Nov 30, 2016 6:12:17 PM jenkins.InitReactorRunner$1 onAttained
INFO: Listed all plugins
Nov 30, 2016 6:12:19 PM jenkins.InitReactorRunner$1 onAttained
INFO: Prepared all plugins
Nov 30, 2016 6:12:19 PM jenkins.InitReactorRunner$1 onAttained
INFO: Started all plugins
Nov 30, 2016 6:12:19 PM jenkins.InitReactorRunner$1 onAttained
INFO: Augmented all extensions
Nov 30, 2016 6:12:20 PM jenkins.InitReactorRunner$1 onAttained
INFO: Loaded all jobs
Nov 30, 2016 6:12:20 PM hudson.model.AsyncPeriodicWork$1 run
INFO: Started Download metadata
Nov 30, 2016 6:12:20 PM hudson.model.AsyncPeriodicWork$1 run
INFO: Finished Download metadata. 97 ms
Nov 30, 2016 6:12:20 PM org.jenkinsci.main.modules.sshd.SSHD start
INFO: Started SSHD at port 44955
Nov 30, 2016 6:12:21 PM jenkins.util.groovy.GroovyHookScript execute
INFO: Executing /var/jenkins_home/init.groovy.d/tcp-slave-agent-port.groovy
Nov 30, 2016 6:12:22 PM jenkins.InitReactorRunner$1 onAttained
INFO: Completed initialization
Nov 30, 2016 6:12:22 PM org.springframework.context.support.AbstractApplicationContext prepareRefresh
INFO: Refreshing org.springframework.web.context.support.StaticWebApplicationContext@453fc3cf: display name [Root WebApplicationContext]; startup date [Wed Nov 30 18:12:22 UTC 2016]; root of context hierarchy
Nov 30, 2016 6:12:22 PM org.springframework.context.support.AbstractApplicationContext obtainFreshBeanFactory
INFO: Bean factory for application context [org.springframework.web.context.support.StaticWebApplicationContext@453fc3cf]: org.springframework.beans.factory.support.DefaultListableBeanFactory@79a53f4b
Nov 30, 2016 6:12:22 PM org.springframework.beans.factory.support.DefaultListableBeanFactory preInstantiateSingletons
INFO: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@79a53f4b: defining beans [authenticationManager]; root of factory hierarchy
Nov 30, 2016 6:12:22 PM org.springframework.context.support.AbstractApplicationContext prepareRefresh
INFO: Refreshing org.springframework.web.context.support.StaticWebApplicationContext@7ea44b7: display name [Root WebApplicationContext]; startup date [Wed Nov 30 18:12:22 UTC 2016]; root of context hierarchy
Nov 30, 2016 6:12:22 PM org.springframework.context.support.AbstractApplicationContext obtainFreshBeanFactory
INFO: Bean factory for application context [org.springframework.web.context.support.StaticWebApplicationContext@7ea44b7]: org.springframework.beans.factory.support.DefaultListableBeanFactory@12544046
Nov 30, 2016 6:12:22 PM org.springframework.beans.factory.support.DefaultListableBeanFactory preInstantiateSingletons
INFO: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@12544046: defining beans [filter,legacy]; root of factory hierarchy
Nov 30, 2016 6:12:22 PM jenkins.install.SetupWizard init
INFO:

*************************************************************
*************************************************************
*************************************************************

Jenkins initial setup is required. An admin user has been created and a password generated.
Please use the following password to proceed to installation:

0c4a8413a47943ac935a4902e3b8167e

This may also be found at: /var/jenkins_home/secrets/initialAdminPassword

*************************************************************
*************************************************************
*************************************************************

Nov 30, 2016 6:12:27 PM hudson.model.UpdateSite updateData
INFO: Obtained the latest update center data file for UpdateSource default
Nov 30, 2016 6:12:27 PM hudson.WebAppMain$3 run
INFO: Jenkins is fully up and running
--> setting agent port for jnlp
--> setting agent port for jnlp... done

Step 4: Open Jenkins in a Browser

Now we want to connect to the Jenkins portal. For that, open a browser and open the URL

<your_jenkins_host>:8080

In our case, Jenkins is running in a container and we have mapped the container-port 8080 to the local port 8080 of the Docker host. On the Docker host, we can open the URL.

localhost:8080

The Jenkins login screen will open:

The admin password can be retrieved from the startup log, we have seen above (0c4a8413a47943ac935a4902e3b8167e), or we can find it by typing

(dockerhost: .../jenkins_home)$ cat secrets/initialAdminPassword
0c4a8413a47943ac935a4902e3b8167e

on the mapped jenkins_home folder on the Docker host.

Step 5: Install Plugins

Let us install the suggested plugins:

This may take a while to finish:

Step 6: Create an Admin User and log in

Then we reach a page, where we can create an Admin user:

Let us do so and save and finish.

Note: After this step, I have deleted the Jenkins container and started a new container attached to the same Jenkins Home directory. After that, all configuration and plugins were still available and we can delete containers after usage without loosing relevant information.

I have had a dinner break at this point. Maybe this is the reason I got following message when clicking the „Start using Jenkins“ button?

What ever. After clicking „retry“, we reach the login page:

Create a New Job

In the nex, we will create our first Jenkins job. I plan to trigger the Maven and/or Gradle build of a Java executable file upon detection of a code change.

Step 2: Install the Job DSL Plugin

The Job DSL Plugin can be installed like any other Jenkins plugin:

Step 3: Create Job DSL Jenkins Project

We create a Job DSL Job like follows:

Step 4: Configure Job DSL Project

-> if you have got a Github account, fork this open source Java Hello World software (originally created by of LableOrg) that will allow you to see, what happens with your Jenkins job, if you check in changed code. Moreover the hello world software allows you to perform JUnit 4 tests, run PowerMockito Mock services, run JUnit 4 Integration tests and calculate the code coverage using the tool Cobertura.

-> insert:

job('Job-DSL-Hello-World-Job') {
    scm {
        git('git://github.com/<org>/java-maven-junit-helloworld')
    }
    triggers {
        scm('H/15 * * * *')
    }
    steps {
        maven('-e clean test')
    }
}

here, exchange the username oveits by your own Github username.

Step 5: Prepare Maven Usage

Goto Jenkins -> Manage Jenkins -> Global Tool Configuration (available for Jenkins >2.0)

-> choose Version (3.3.9 in my case)

-> Add a name („Maven 3.3.9“ in my case)

Since we have checked „Install automatically“ above, I expect that it will be installed automatically on first usage.

Step 6: Prepare Git Usage

As described in this StackOverflow Q&A, we need to add the Git username and email address, since Jenkins tries to tag and commit on the Git repo, which requires those configuration items to be set. For that, we perform:

-> scroll down to „Git plugin“

->

Step 7: Create Jenkins Job from Code

Step 7.1 Build Project

Step 7.2 (optional): Check Console Output

Step 7.3: Review automatically built Project

This is showing a build failure, since I had not performed Step 5 and 6 before. In your case, it should be showing a success (in blue). If you are experiencing problems here, check out the Appendices below.

-> scroll down to Source Code Management

-> Scroll down to Build Triggers

-> Scroll down to Build

-> verify that „Maven 3.3.9“ is chosen as defined in Step 5

-> enter „-e clean test“ as Maven Goal

See, what happens by clicking on:

-> Build History

-> #nnn

If everything went fine, we will see many downloads and a „BUILD SUCCESS“:

Appendix A: Solve Git Problem: „tell me who you are“

Symptoms: Git Error: status code 128

In a new installation of Jenkins, Git does not seem to work out of the box. You can see this by choosing the Jenkins project Job-DSL-Hello-World-Job on the dashboard, then click „build now“, if the build was not already automatically triggered. Then:

-> Build History

-> Last Build (link works only, if Jenkins is running on localhost:8080 and you have chosen the same job name)

There, we will see:

Caused by: hudson.plugins.git.GitException: Command "git tag -a -f -m Jenkins Build #1 jenkins-Job-DSL-Hello-World-Job-1" returned status code 128:
stdout: 
stderr: 
*** Please tell me who you are.

Run

  git config --global user.email "you@example.com"
  git config --global user.name "Your Name"

to set your account's default identity.
Omit --global to set the identity only in this repository.

fatal: empty ident name (for <jenkins@61915398735e.(none)>) not allowed

Resolution:

Step 1: Enter Git Username and Email

As described in this StackOverflow Q&A: we can resolve this issue by either suppressing the git tagging, or (I think this is better) by adding your username and email address to git:

-> scroll down to „Git plugin“

Step 2: Re-run „Build Now“ on the Project

To test the new configuration, we go to

-> the Job-DSL-Hello-World-Job and press

Now, we should see a BUILD SUCCESS like follows:

-> Build History

-> #nnn

If everything went fine, we will a „BUILD SUCCESS“:

Appendix B: Maven Error: Cannot run program „mvn“

Symptoms:

When running a Maven Goal, the following error may appear on the Console log:

FATAL: command execution failed
java.io.IOException: Cannot run program "mvn" (in directory "/var/jenkins_home/workspace/Job-DSL-Hello-World-Job"): error=2, No such file or directory
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
	at hudson.Proc$LocalProc.(Proc.java:245)
	at hudson.Proc$LocalProc.(Proc.java:214)
	at hudson.Launcher$LocalLauncher.launch(Launcher.java:846)
	at hudson.Launcher$ProcStarter.start(Launcher.java:384)
	at hudson.Launcher$ProcStarter.join(Launcher.java:395)
	at hudson.tasks.Maven.perform(Maven.java:367)
	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:779)
	at hudson.model.Build$BuildExecution.build(Build.java:205)
	at hudson.model.Build$BuildExecution.doRun(Build.java:162)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:534)
	at hudson.model.Run.execute(Run.java:1728)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:98)
	at hudson.model.Executor.run(Executor.java:404)
Caused by: java.io.IOException: error=2, No such file or directory
	at java.lang.UNIXProcess.forkAndExec(Native Method)
	at java.lang.UNIXProcess.(UNIXProcess.java:247)
	at java.lang.ProcessImpl.start(ProcessImpl.java:134)
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
	... 15 more
Build step 'Invoke top-level Maven targets' marked build as failure
Finished: FAILURE

Resolution:

Perform Step 5

and

For Test, you can test a manual: choose the correct Maven version, when configuring a Maven build step like in Step 7:

For our case, we need to correct the Job DSL like follows:

In the Script, we had defined the step:

    steps {
        maven('-e clean test')
    }

However, we need to define the Maven Installation like follows:

    steps {
        maven {
            mavenInstallation("Maven 3.3.9")
            goals('-e clean test')
        }
    }

Here, the mavenInstallation needs to specify the exact same name, as the one we have chosen in Step 5 above.

After correction, we will receive the correct Maven goal

Now, we can check the Maven configuration:

After scrolling down, we will see the correct Maven Version:

DONE

Appendix C: Updating Jenkins

Updating Jenkins (in my case: from 2.32.1 to 2.32.2) was as simple as following the steps below

Note: you might want to make a backup of your jenkins_home though. Just in case…

(dockerhost)$ cd <path_to_jenkins_home> # in my case: cd /user/jenkins_home/
(dockerhost)$ docker pull jenkins # to update the jenkins image
(dockerhost)$ docker rm jenkins # to make shure the container named jenkins is removed
(dockerhost:jenkins_home)$ sudo docker run -d --rm --name jenkins -p8080:8080 -p50000:50000 -v`pwd`:/var/jenkins_home jenkins

However, after that, some data was unreadable:

I have clicked

to resolve the issue (hopefully…). At least, after that, the warning was gone.

Appendix D: Job DSL Syntax

The reference for the Job DSL syntax can be found on the Job DSL Plugin API pages. As an example, the syntax of Maven within a Freestyle project can be found on this page found via the path

> freeStyleJob > steps > maven:

maven {

• // Allows direct manipulation of the generated XML.

  configure(Closure configureBlock)

• // Specifies the goals to execute including other command line options.

  goals(String goals)

• // Skip injecting build variables as properties into the Maven process.

  injectBuildVariables(boolean injectBuildVariables = true)

• // Set to use isolated local Maven repositories.

  localRepository(javaposse.jobdsl.dsl.helpers.LocalRepositoryLocation location)

• // Specifies the Maven installation for executing this step.

  mavenInstallation(String name)

• // Specifies the JVM options needed when launching Maven as an external process.

  mavenOpts(String mavenOpts)

• // Adds properties for the Maven build.

  properties(Map props)

• // Adds a property for the Maven build.

  property(String key, String value)

• // Specifies the managed global Maven settings to be used.

  providedGlobalSettings(String settingsIdOrName)

• // Specifies the managed Maven settings to be used.

  providedSettings(String settingsIdOrName)

• // Specifies the path to the root POM.

  rootPOM(String rootPOM)

}

A Maven example can be found on the same page:

job('example') {
    steps {
        maven('verify')
        maven('clean verify', 'module-a/pom.xml')
        maven {
            goals('clean')
            goals('verify')
            mavenOpts('-Xms256m')
            mavenOpts('-Xmx512m')
            localRepository(LocalRepositoryLocation.LOCAL_TO_WORKSPACE)
            properties(skipTests: true)
            mavenInstallation('Maven 3.1.1')
            providedSettings('central-mirror')
        }
    }
}

Summary

In this blog post, we have learned how to

1. Start and initialize Jenkins via Docker

2. Prepare the usage of Git and Maven

3. Install the Job DSL Plugin

4. Define a Jenkins Job via Groovy script

5. Create a Jenkins Job by a push of the „Build now“ button

6. Review and run the automatically created Jenkins job

We have seen that the usage of the Job DSL is no rocket science. The only topic, we had to take care, is, that Git and Maven need to be prepared for first usage on a Jenkins server.

Objective

The objective is to provide a clear guide for efficiently using DIGIT infrastructure on various platforms like SDC, NIC, or commercial clouds. This document outlines the infrastructure overview, operational guidelines, and recommendations, along with the segregation of duties (SoD). It helps to plan the procurement and build the necessary capabilities to deploy and implement DIGIT.

In a shared control scenario, the state program team must adhere to these guidelines and develop their own control implementation for the state's cloud infrastructure and collaborations with partners. This ensures standardized and smooth operational excellence in the overall system.

DIGIT Infrastructure Overview

DIGIT Platform is designed as a microservices architecture, using open-source technologies and containerized apps and services. DIGIT components/services are deployed as docker containers on a platform called Kubernetes, which provides flexibility for running cloud-native applications anywhere like physical or virtual infrastructure or hypervisor or HCI and so on. Kubernetes handles the work of scheduling containerized services onto a compute cluster and manages the workloads to ensure they run as intended. And it substantially simplifies the deployment and management of microservices.

Provisioning the Kubernetes cluster will vary across from commercial clouds to state data centres, especially in the absence of managed Kubernetes services like AWS, Azure, GCP and NIC. Kubernetes clusters can also be provisioned on state data centres with bare-metal, virtual machines, hypervisors, HCI, etc. However providing integrated networking, monitoring, logging, and alerting is critical for operating Kubernetes Clusters when it comes to State data centers. DIGIT Platform also offers add-ons to monitor Kubernetes cluster performance, logging, tracing, service monitoring and alerting, which the implementation team can take advantage.

Below are the useful links to understand Kubernetes:

• Understand containers, dockers

• What is Kubernetes and Architecture

• Requirements to setup Kubernetes cluster

• Provider Managed Cluster vs Self-provisioning Cluster

• DIGIT Deployment on Kubernetes

DIGIT On Kubernetes - High-Level Deployment Diagram

DIGIT Infra Specification On SDC or NIC Or Any Commercial Cloud

Operational Recommendations

DIGIT strongly recommends Site reliability engineering (SRE) principles as a key means to bridge development and operations gaps by applying a software engineering mindset to system and IT administration topics. In general, an SRE team is responsible for the availability, latency, performance, efficiency, change management, monitoring, emergency response, and capacity planning.

Monitoring Tools Recommendations

Commercial clouds like AWS, Azure and GCP offer sophisticated monitoring solutions across various infra levels like CloudWatch and StackDriver. In the absence of such managed services to monitor, we can look at various best practices and tools listed below which help in debugging and troubleshooting efficiently.

• Network monitoring, Diagnosis, Assessment, Identify the latency

• Systems

• Application monitoring

• Web application monitoring

• Alert management, Alert Types

• Distributed Tracing

• Telemetry

Key Standard Operating Procedures (SOPs)

• Segregation of duties and responsibilities.

• SME and SPOCs for L1.5 support along with the SLAs defined.

• Ticketing system to manage incidents, converge and collaborate on various operational issues.

• Monitoring dashboards at various levels like Infrastructure, networks and applications.

• Transparency of monitoring data and collaboration between teams.

• Periodic remote sync-up meetings, acceptance and attendance to the meeting.

• Ability to see stakeholders' availability of calendar time to schedule meetings.

• Periodic (weekly, monthly) summary reports of the various infra, operations incident categories.

• Communication channels and synchronization regularly and also upon critical issues, changes, upgrades, releases etc.

Segregation Of Duties

While DIGIT is deployed at state cloud infrastructure, it is essential to identify and distinguish the responsibilities between Infrastructure, Operations and Implementation partners. Identify these teams and assign SPOC, define responsibilities and ensure the Incident Management process is followed to visualize, track issues and manage dependencies between teams. Essentially these are monitored through dashboards and alerts are sent to the stakeholders proactively. eGov team can provide consultation and training on a need basis depending on any of the below categories.

State IT/Cloud team -Refers to state infra team for the Infra, network architecture, LAN network speed, internet speed, OS Licensing and upgrade, patch, compute, memory, disk, firewall, IOPS, security, access, SSL, DNS, data backups/recovery, snapshots, capacity monitoring dashboard.

• State program team - Refers to the owner for the whole DIGIT implementation, application rollouts, and capacity building. Responsible for identifying and synchronizing the operating mechanism between the below teams.

• Implementation partner - Refers to the DIGIT Implementation, application performance monitoring for errors, logs scrutiny, TPS on peak load, distributed tracing, DB queries analysis, etc.

• Operations team - this team could be an extension of the implementation team that is responsible for DIGIT deployments, configurations, CI/CD, change management, traffic monitoring and alerting, log monitoring and dashboard, application security, DB Backups, application uptime, etc.

Skills To Setup, Operate & Maintain DIGIT On SDC

Resource Requirement

DIGIT - Security Standards & Operational Recommendations

This section provides insights on security principles, security layers and the line of control that we focus on to prevent DIGIT security from the code, application, access, infra and operations. The target audience of this section are internal teams, partners, ecosystems and states to understand what security measures to be considered to secure DIGIT from an infrastructure and operations perspective .

DIGIT - Key Security Principles

Subscribe to the DIGIT applicable OWASP top 10 standard across various security layers.

1. Minimize attack surface area

2. Implement a strong identity foundation - Who accesses what and who does what.

3. Apply security at all possible layers

4. Automate security best practices

5. Separation of duties (SoD).

6. The principle of Least privilege (PoLP)

7. Templatized design - (Code, Images, Infra-as-code, Deploy-as-code, Conf-as-code, etc)

8. Align with MeiTY Standards to meet SDC Infra policies.

Security Layers & Line of Control

Application Layer

The presentation layer is likely to be the #1 attack vector for malicious individuals seeking to breach security defences like DDoS attacks, Malicious bots, Cross-Site Scripting (XSS) and SQL injection. Need to invest in web security testing with the powerful combination of tools, automation, process and speed that seamlessly integrates testing into software development, helping to eliminate vulnerabilities more effectively, deploy a web application firewall (WAF) that monitors and filters traffic to and from the application, blocking bad actors while safe traffic proceeds normally.

Key Security Measures

1. TLS-protocols/Encryption: Access control to secure authentication and authorization. All APIs that are exposed must have HTTPS certificates and encrypt all the communication between client and server with transport layer security (TLS).

2. Auth Tokens: An authorization framework that allows users to obtain admittance to a resource from the server. This is done using tokens in microservices security patterns: resource server, resource owner, authorization server, and client. These tokens are responsible for access to the resource before its expiry time. Also, Refresh Tokens that are responsible for requesting new access after the original token has expired.

3. Multi-factor Authentication: authorize users on the front end, which requires a username and password as well as another form of identity verification to offer users better protection by default as some aspects are harder to steal than others. For instance, using OTP for authentication takes microservice security to a whole new level.

4. Rate Limit/DDoS: denial-of-service attacks are the attempts to send an overwhelming number of service messages to cause application failure by concentrating on volumetric flooding of the network pipe. Such attacks can target the entire platform and network stack.

To prevent this:

• We should set a limit on how many requests in a given period can be sent to each API.

• If the number exceeds the limit, block access from a particular API, at least for some reasonable interval.

• Also, make sure to analyze the payload for threats.

• The incoming calls from a gateway API would also have to be rate-limited.

• Add filters to the router to drop packets from suspicious sources.

5. Cross-site scripting (XSS): scripts that are embedded in a webpage and executed on the client side, in a user’s browser, instead of on the server side. When applications take data from users and dynamically include it in webpages without validating the data properly, attackers can execute arbitrary commands and display arbitrary content in the user’s browser to gain access to account credentials.

How to prevent:

• Applications must validate data input to the web application from user browsers.

• All output from the web application to user browsers must be encoded.

• Users must have the option to disable client-side scripts.

6. Cross-Site Request Forgery (CSRF): is an attack whereby a malicious website will send a request to a web application that a user is already authenticated against from a different website. This way an attacker can access functionality in a target web application via the victim's already authenticated browser. Targets include web applications like social media, in-browser email clients, online banking and web interfaces for network devices. To prevent this CSRF tokens are appended to each request and associated to the user’s session. Such tokens should at a minimum be unique per user session, but can also be unique per request.

How to prevent:

• By including a challenge token with each request, the developer can ensure that the request is valid and not coming from a source other than the user.

8. SQL Injection (SQLi): allows attackers to control an application’s database – letting them access or delete data, change an application’s data-driven behaviour, and do other undesirable things – by tricking the application into sending unexpected SQL commands. SQL injections are among the most frequent threats to data security.

How to prevent:

• Using parameterized queries which specify placeholders for parameters so that the database will always treat them as data rather than part of an SQL command. Prepared statements and object-relational mappers (ORMs) make this easy for developers.

• Remediate SQLi vulnerabilities in legacy systems by escaping inputs before adding them to the query. Use this technique only where prepared statements or similar facilities are unavailable.

• Mitigate the impact of SQLi vulnerabilities by enforcing the least privilege on the database. Ensure that each application has its database credentials and that these credentials have the minimum rights the application needs.

Security In The Code

The primary causes of commonly exploited software vulnerabilities are consistent defects, bugs, and logic flaws in the code. Poor coding practices can create vulnerabilities in the system that can be exploited by cybercriminals.

What defines a security in the code:

1. White-box code analysis: As developers write code, the IDE needs to provide focused, real-time security feedback with white-box code analysis. It also helps developers remediate faster and learn on the job through positive reinforcement, remediation guidance, code examples, etc.

2. Static Code Analysis (SAST): A static analysis tool reviews program code, searching for application coding flaws, back doors or other malicious code that could give hackers access to critical data or customer information. However, most static analysis tools can only scan source code.

3: Vulnerability assessment: Vulnerability assessment for the third-party libraries/artefacts as part of CI and GitHub PR process. Test results are returned quickly and prioritized in a Fix-First Analysis that identifies both the most urgent flaws and the ones that can be fixed most quickly, allowing developers to optimize efforts and save additional resources.

4. Secure PII/Encrypt: Personally identifying information – to make sure that it is not being displayed as plain text. All the passwords and usernames must be masked during the storing in logs or records. However, adding extra encryption above TLS/HTTP won’t add protection for traffic travelling through the wire. It can only help a little bit at the point where TLS terminates, so it can protect sensitive data (such as passwords or credit card numbers) from accidental dumping into a request log. Extra encryption (RSA 2048+ or Blowfish) might help protect data against those attacks that aim at accessing the log data. But it will not help with those who try accessing the memory of the application servers or the main data storage.

5. Manual Penetration Testing: Some categories of vulnerabilities, such as authorization issues and business logic flaws, cannot be found with automated assessments and will always require a skilled penetration tester to identify them. Need to employ Manual Penetration Testing that uses proven practices to provide extensive and comprehensive security testing results for web, mobile, desktop, and back-end with detailed results, including attack simulations.

Libraries/Containers

Components, such as libraries, frameworks, container images, and other software modules, almost always run with full privileges. If a vulnerable component is exploited, such an attack can facilitate serious data loss or server takeover. Applications using components with known vulnerabilities may undermine application defences and enable a range of possible attacks and impacts.

Automating dependency checks for the libraries and container auditing, as well as using other container security processes as part of the CI periodically or as part of PRs can largely prevent these vulnerabilities. Subscribing to tools that comply with vulnerable library databases such as OSVDB, Node Security Project, CIS, National Vulnerability Database, and Docker Bench for Security can help identify and fix the vulnerabilities periodically. A private docker registry can help.

Data Security

Data Security involves putting in place specific controls, standard policies, and procedures to protect data from a range of issues, including:

• Enforced encryption: Encrypt, manage and secure data by safeguarding it in transit. Password-based, easy to use and very efficient.

• Unauthorized access: Blocking unauthorized access plays a central role in preventing data breaches. Implementing Strong Password Policy and MFA.

• Accidental loss: All data should be backed up. In the event of hardware or software failure, breach, or any other error to data; a backup allows it to continue with minimal interruption. Storing the files elsewhere can also quickly determine how much data was lost and/or corrupted.

• Destruction: Endpoint Detection and Response (EDR) – provides visibility and defensive measures on the endpoint itself, when attacks occur on endpoint devices this can eliminate gaining access systems and avoid destruction of the data.

Infra/Cloud

In microservices and the Cloud Native architectural approach, the explosion of ephemeral, containerized services that arise from scaling applications developed increases the complexity of delivery. Fortunately, Kubernetes was developed just for this purpose. It provides DevOps teams with an orchestration capability for managing the multitude of deployed services, with in-built automation, resilience, load balancing, and much more. It's perfect for the reliable delivery of Cloud Native applications. Below are some of the key areas to get more control to establish policies, procedures and safeguards through the implementation of a set of rules for compliance. These rules cover infra privacy, security, breach notification, enforcement, and an omnibus rule that deals with security compliance.

• Strong stance on authentication and authorization

• Role-Based Access Control (RBAC)

• Kubernetes infrastructure vulnerability scanning

• Hunting misplaced secrets

• Workload hardening from Pod Security to network policies

• Ingress Controllers for security best practices

• Constantly watch your Kubernetes deployments

• Find deviations from desired baselines

• Should alert or deny on policy violation

• Block/Whitelist (IP or DNS) connections before entering the workloads.

• Templatize the deployment/secrets configs and serve as config-as-code.

Network Security

Kubernetes brings new requirements for network security, because applications, that are designed to run on Kubernetes, are usually architected as microservices that rely on the network. They make API calls to each other. Steps must be taken to ensure proper security protocols are in place. The following are the key areas for implementing network security for a Kubernetes platform:

• Container Groups: Coupled communication between grouped containers, is achieved inside the Pod that contains one or more containers.

• Communication between Pods: Pods are the smallest unit of deployment in Kubernetes. A Pod can be scheduled on one of the many nodes in a cluster and has a unique IP address. Kubernetes places certain requirements on communication between Pods when the network has not been intentionally segmented. These requirements include:

• Containers should be able to communicate with other Pods without using network address translation (NAT).

• All the nodes in the cluster should be able to communicate with all the containers in the cluster.

• The IP address assigned to a container should be the same that is visible to other entities communicating with the container.

• Pods and Services: Since Pods are ephemeral in nature, an abstraction called a Service provides a long-lived virtual IP address that is tied to the service locator (e.g., a DNS name). Traffic destined for that service VIP is then redirected to one of the Pods and offers the service using that specific Pod’s IP address as the destination.

• Traffic Direction: Traffic is directed to Pods and services in the cluster via multiple mechanisms. The most common is via an ingress controller, which exposes one or more service VIPs to the external network. Other mechanisms include node ports and even publicly-addressed Pods.

Operational Security

It is a procedural security that manages risk and encourages to view of operations from the perspective of an adversary to protect sensitive information from falling into the wrong hands. Following are a few best practices to implement a robust, comprehensive operational security program:

• Implement precise change management processes: All changes should be logged and controlled so they can be monitored and audited.

• Restrict access to network devices using AAA authentication: a “need-to-know” is a rule of thumb regarding access and sharing of information.

• Least Privilege (PoLP): Give the minimum access necessary to perform their jobs.

• Implement dual control: Those who work on the tasks are not the same people in charge of security.

• Automate tasks: reduce the need for human intervention. Humans are the weakest link in any organization’s operational security initiatives because they make mistakes, overlook details, forget things, and bypass processes.

Incident response and disaster recovery planning: are always crucial components of a sound security posture, we must have a plan to identify risks, respond to them, and mitigate potential damages.

Overview

This page provides the steps and process to set up the central monitoring dashboard.

Pre-reads

https://github.com/kubecost/cost-analyzer-helm-chart#kubecost-helm-chart https://prometheus.io/docs/introduction/overview/ https://grafana.com/docs/

Pre-requisites

• DIGIT uses golang (required v1.13.3) automated scripts to deploy the builds onto Kubernetes - Linux or Windows or Mac.

• All DIGIT services are packaged using helm charts Installing Helm.

• kubectl is a CLI to connect to the Kubernetes cluster from your machine.

• Install VisualStudio IDE Code for better code/configuration editing capabilities.

• Git

• Cost-analyzer - cost-analyzer must be deployed on the client side.

• Prometheus-operator - Prometheus must be deployed on the client side.

• prometheus-kafka-exporter (A Prometheus exporter acts as a proxy between such an application and the Prometheus server) - prometheus-kafka-exporter must be deployed on the client side

Steps

Step 1: Install Prometheus Operator On Each Client Cluster

Step 2: Expose The Prometheus Operator

Expose the Prometheus Operator using nginx-ingress rule in each client cluster. This makes it easy to access Prometheus metrics in central-dashboard clusters

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    certmanager.k8s.io/cluster-issuer: letsencrypt-prod  //By using this annotation, a certificate will be issued
  labels:
    app: prometheus-operator
  name: prometheus-prometheus
  namespace: monitoring
spec:
  rules:
  - host: <host_name>                     //Replace with prometheus host
    http:
      paths:
      - backend:
          service:
            name: prometheus-prometheus
            port:
              number: 9090
        path: /
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - <host_name>                         //Replace with prometheus host
    secretName: <host_name>-tls-certs     //Replace with prometheus host

Note: Ensure you create the CANAME DNS record with the hostname and loadbalancer ID.

Step 3: Install Cost Analyzer On Each Client Cluster

Kubecost provides visibility into current and historical Kubernetes spending and resource allocation. These provide cost transparency in Kubernetes environments.

You can deploy the cost analyser using one of the below methods.

1. Deploy using go lang deployer

git clone https://github.com/egovernments/DIGIT-DevOps.git
cd DIGIT-DevOps/deploy-as-code/egov-deployer
go run main.go -e <environment_name> "cost-analyzer"

2. Deploy using Jenkin’s deployment job. Here we are using deploy-to-dev. Choose your environment-specific deployment job.

Step 4: Install Grafana On The Central Dashboard Cluster

Below Grafana configuration should be added to the environments file, and then Grafana should be deployed using one of the following methods.

Based on the number of client clusters, you have to add data sources. There should be one entry per client cluster as shown below.

grafana: 
  image:
    repository: grafana/grafana
    tag: 9.0.0

  initContainers:
    gitSync:
      enabled: true
      repo: "git@github.com:egovernments/configs"  // repo contain the dashboard json files 
      branch: "central-dashboard"                 
  ingress:
    hostName: central-dashboard.digit.org  //central-dashboard hostname
    context: ""
    additionalAnnotations: | 
  grafana.ini:
    server:
      root_url: "%(protocol)s://%(domain)s"
      serve_from_sub_path: true 

  env: |
    - name: GF_SERVER_DOMAIN
      value: {{ .Values.ingress.hostName | quote }}    
  datasources:                // Configure Grafana to use prometheus ddata sources    datasources.yaml:
      apiVersion: 1
      datasources:
      - name: DIGIT-Dev
        type: prometheus
        url: https://<Dev_hostname>  //Add exposed hostname for dev environment to access the dev prometheus metrics
        isDefault: false 
      - name: DIGIT-QA
        type: prometheus
        url: https://<QA_hostname>
        isDefault: false
      - name: DIGIT-UAT
        type: prometheus
        url: https://<UAT_hostname>
        isDefault: false
      - name: DIGIT-Staging
        type: prometheus
        url: https://<Staging_hostname>   
        isDefault: false

1. Deploy using go lang deployer

   Copy

   git clone https://github.com/egovernments/DIGIT-DevOps.git
   cd DIGIT-DevOps/deploy-as-code/egov-deployer
   go run main.go -e <environment_name> "grafana"

2. Deploy using Jenkin’s deployment job.

Step 4: DNS Mapping

To access the monitoring central dashboard with this https://central-dashboard.digit.org URL. Ensure you create the CANAME DNS record with the hostname and load balancer ID.

• To create a new repository, click on + icon and New repository

• Create your with repo with any name based on your code. Make it as public. Then anyone can able to see your code.

• If you want to add a README file, click on add a README file. It is helpful to understand how does the code present in repo will be helpful.

• Next click on create repository.

What is organization and why we are creating in GitHub:

An organization are shared accounts where businesses and open source projects can collaborate across many projects at once.There are three types accounts in GitHub

1. Personal accounts

2. Organization accounts

3. Enterprise accounts

• Here the main reason for creating organization account is, accounts can be shared among unlimited number of people and they can collaborate across many projects at once.

• Our organization name is eGovernments Foundation.

GitHub account creation:

• Go to

• After completing the process Your GitHub account will be created.

• Click on Sign Up. Create your account by using email and password. Then add Username.

• After completing the process Your GitHub account will be created.

Creating Organization in GitHub:

• After setting up the GitHub account, we have to create an organization. Here we can add the data or code in the form of repository. Creating a repository, we will see this topic next.

• Open Github and click on the "+" icon add top tight corner. You will see the option"new organization". click it.

• click on "create a free organization"and enter your organization name you want to create with email and then '"next"

• After Organization got created, you can see your organizations by clicking on "Accounts

Creating a branch inside repository:

• Go to the repository and click on new branch.

• Here I have created a branch named DIGIT

• After, go to that branch in the same repository.

Creating branch protection rule:

• Branch protection rule states that, how to manage the branch restrictions/permissions in GitHub.

NOTE : You must have admin access orelse you have to be a codeowner to make these changes for branch restrictions/permissions.

• Open https://github.com and choose any repository.Go to the main page. Click on settings.

• Click on branches

• If you click on the Edit rules you can able to see the rules which are applied for that branch.you should follow the rules when ever you are going to made any changes to that branch and pushing it.

• If you want to create new branch protection rule click on Add Rule.

The common restrictions we are following to merge branches are :

1.Requires pull request

2.Requires approvals from CODE OWNERS

• Only the CODE OWNERS can have access to merge and makes changes to these rules.

Overview

Role-based access control (RBAC) regulates access to a computer or network resources based on the roles of individual users within your organization.

RBAC authorization uses the rbac.authorization.k8s.io API group to drive authorization decisions, allowing you to configure policies through the Kubernetes API dynamically.

API objects

The RBAC API declares four Kubernetes objects: Role, ClusterRole, RoleBinding and ClusterRoleBinding. You can , or amend them, using tools such as kubectl, just like any other Kubernetes object.

Role & ClusterRole

An RBAC Role or ClusterRole contains rules that represent a set of permissions. Permissions are purely additive (there are no "deny" rules).

A Role always sets permissions within a particular ; when you create a Role, you have to specify the namespace it belongs in.

ClusterRole, by contrast, is a non-namespaced resource. The resources have different names (Role and ClusterRole) because a Kubernetes object always has to be either namespaced or not namespaced; it can't be both.

ClusterRoles have several uses. You can use a ClusterRole to:

1. define permissions on namespaced resources and be granted access within individual namespace(s)

2. define permissions on namespaced resources and be granted access across all namespaces

3. define permissions on cluster-scoped resources

If you want to define a role within a namespace, use a Role; if you want to define a role cluster-wide, use a ClusterRole.

A ClusterRole can be used to grant the same permissions as a Role. Because ClusterRoles are cluster-scoped, you can also use them to grant access to:

• non-resource endpoints (like /healthz)

• namespaced resources (like Pods), across all namespaces

  For example: you can use a ClusterRole to allow a particular user to run kubectl get pods --all-namespaces

RoleBinding & ClusterRoleBinding

A role binding grants the permissions defined in a role to a user or set of users. It holds a list of subjects (users, groups, or service accounts), and a reference to the role being granted. A RoleBinding grants permissions within a specific namespace whereas a ClusterRoleBinding grants that access cluster-wide.

A RoleBinding may reference any Role in the same namespace. Alternatively, a RoleBinding can reference a ClusterRole and bind that ClusterRole to the namespace of the RoleBinding. If you want to bind a ClusterRole to all the namespaces in your cluster, you use a ClusterRoleBinding.

A RoleBinding can also reference a ClusterRole to grant the permissions defined in that ClusterRole to resources inside the RoleBinding's namespace. This kind of reference lets you define a set of common roles across your cluster, and then reuse them within multiple namespaces.

For instance, even though the following RoleBinding refers to a ClusterRole, "dave" (the subject, case sensitive) will only be able to read Secrets in the "development" namespace, because the RoleBinding's namespace (in its metadata) is "development".

Define the RBAC config in the Environment file

You must add a namespace to a role section to grant access to a group of a namespace.

• In eGovernments Foundations we are having multiple number of Teams. we can create independent teams to manage repository permissions and mentions for groups of people.

• Only organization owners and maintainers can create team. Owners can also restrict creation permissions for all teams in an organization.

• First sign in to your organization github account.

• Once you sign in to your account and if you open view organization you can able to see the above page.

• Click on Teams. You will see the below image.

• Now, click on the New team

• Fill the details as shown in the below image:

• After creating team, you will able to see the below image.

• If you click on members.you can add members to your team by providing their github username or mail.

• Now, you have successfully created GitHub team.

Adding new SSH key to your GitHub account:

• Open Your "Command prompt" or "Terminal".

• Type below commands to generate SSH key

• Now a .ssh folder is created in your home directory. Go to that directory.

• copy the SSH key which we get after running the above commands.

• open GitHub and add this SSH key as shown below:

• open Settings and go to SSH and GPG keys

• Click on New SSH key and paste it. Click on Add SSH key.

• If you want check the private key, use

Overview

This page provides the step-by-step process for setting up the central-instance infra.

Pre-reads

• Know about EKS:

• Know node groups​

• Know about Taints and Tolerations

• Know about node-affinity

• Know what is terraform: ​

Pre-requisites

1. ​ with admin access to provision EKS Service, you can always subscribe to a free AWS account to learn the basics and try, but there is a limit to , for this demo you need to have a commercial subscription to the EKS service.

2. Install version (0.14.10) for the Infra-as-code (IaC) to provision cloud resources as code and with desired resource graph and also it helps to destroy the cluster in one go.

3. Install on your local machine which helps you interact with the Kubernetes cluster

4. Install that helps you package the services along with the configurations, environments, secrets, etc into a

5. ​ on your local machine so that you can use AWS CLI commands to provision and manage the cloud resources on your account.

6. Install which helps you authenticate your connection from your local machine so that you should be able to deploy DIGIT services.

7. ​Use the credentials provided for the Terraform () to connect to your AWS account and provision the cloud resources.

   • You'll get a Secret Access Key and Access Key ID. Save them safely.

   • Open the terminal and run the following command. The AWS CLI is already installed and the credentials are saved. (Provide the credentials and you can leave the region and output format blank).

     The above will create the following file In your machine as /Users/.aws/credentials

Before we provision the cloud resources, we need to understand and be sure about what resources need to be provisioned by Terraform to deploy DIGIT. The following picture shows the various key components. (EKS, Worker Nodes, Postgres DB, EBS Volumes, Load Balancer).

The following are the resources that we are going to provision using Terraform in a standard way so that every time and for every environment, it'll have the same infra.

• EKS Control Plane (Kubernetes Master)

• Work node group (VMs with the estimated number of vCPUs and memory)

• Node-Groups

• EBS Volumes (persistent volumes)

• RDS (Postgresql)

• VPCs (private network)

• Users to access, deploy and read only

Provisioning Central Instance Infra Using Terraform

Fork the DIGIT-DevOps repository into your organization account using the GitHub web portal. Make sure to add the right users to the repository. Clone the forked DIGIT-DevOps repository. Navigate to the sample-central-instance directory which contains the sample AWS infra provisioning script.

• People with admin or owner permissions can set up a CODEOWNERS file in a repository.

• The people you choose as code owners must have write permissions for the repository.

• When the code owner is a team, that team must be visible and it must have write permissions, even if all the individual members of the team already have write permissions directly, through organization membership, or through another team membership.

• For every branch there will be a CODEOWNER file. Only they can able to write the code and able to merge the pull requests.

Create CODEOWNER file:

• Go to any of your branch(DIGIT branch created previously) in a repository and click on new file and name it as CODEOWNERS

• Click on "Create a new branch for this commit and start a pull request" and click on propose new file

• Next click on Create pull request and then Merge pull request and confirm merge.

• Add the GitHub Id's of all the team or people whom you want to add.

• Kubernetes also known as K8s, is an open-source system for automating deployment, scaling, and management of containerized applications.kubectl, allows you to run commands against Kubernetes clusters.

• If you want to study about kubernetes in detail, open

Why kubectl is using in DIGIT?

There are some other tools like kubelet along with kubectl. kubectl is the command-line interface (CLI) tool for working with a Kubernetes cluster. Kubelet is the technology that applies, creates, updates, and destroys containers on a Kubernetes node.But the only difference is, using kubectl the developer can interacts with kubernetes cluster. So we are using kubectl in DIGIT.

To know kubernetes is installed:

To install or update kubectl:

In Windows:

• Download the kubectl . or if you have curl installed use this command:

• To download curl follow the page and proceed the download with curl

• Append or prepend the kubectl binary folder to your PATH environment variable. To perform this, complete the following steps:

• 1.Open the kubectl.exe folder in files and copy that folder.

• 2.Create a new folder in Local Disk(C:) with name Kube.

• 3. Paste the kubectl.exe folder there.

• 4. Open windows option Search for Advanced system settings

• 5. Click on Environmental variables and then System variables>Path>add and add your path name i.e c:\kube

• 6. Save it.

• Once you install kubectl, you can verify its version with the following command:

To install kubectl in linux:

To install kubectl in MacOs:

Pre-requisites:

• Install in your local machine.

• account.

Procedure:

• To move the existing docker images from one account to another account by changing tags.

• First, we have to login to the docker account in which the images are present.

• We need to pull the image from the docker container to local machine.

• Next, we have to change the tag name to our required docker container tag

• Now, we have our required images with tags in our local machine. We need to push these images from local machine to destination container. First, login to the destination account using the above docker login command and then push the image using below command.

• Once successfully pushed, if you check in your docker hub account the images will be present.

Terraform: Terraform is an open-source infrastructure as code software tool that enables you to safely and predictably create, change, and improve infrastructure.

what is Terraform is used for: Terraform is an IAC tool, used primarily by DevOps teams to automate various infrastructure tasks. The provisioning of cloud resources, for instance, is one of the main use cases of Terraform. It is a open-source provisioning tool written in the Go language and created by HashiCorp.

• To install Terraform, use the following link to download the zip file.

• As per our requirment we have to install a specific version which is 0.14.10.

• Install the unzip.

• Extract the downloaded file archive.

• Move the executable into a directory searched for executables.

• Run the below command to check whether the terraform is working.

Pre-requisites:

• IDE Code for better code/configuration editing capabilities

• Install v0.14.10.

• Install .

Customization

• Clone the DIGIT-DevOps repo

• Here we are using AWS cloud service provider to create terraform infra. So, we are choosing sample-aws module (Terraform module is a collection of standard configuration files in a dedicated directory).

• Open sample-aws in visual studio using the below command.

• In that sample-aws module we can find the below terraform templates

• main.tf will contain the main set of configuration for your module.

• outputs.tf will contain the output definitions for your module. Module outputs are made available to the configuration using the module, so they are often used to pass information about the parts of your infrastructure defined by the module to other parts of your configuration.

• providers.tf allow terraform to interact with cloud providers,SAAS providers. In this sample-aws our provider is aws.

• variables.tf will contain the variable definitions for your module. When your module is used by others, the variables will be configured as arguments in the module block. Since all Terraform values must be defined, any variables that are not given a default value will become required arguments. Variables with default values can also be provided as module arguments, overriding the default value.

• To setup the DIGIT infra we made changes in variables.tf. Open variables.tf in visual studio using the below code.

• Change the values in variables.tf which are specified to replace based on our requirements.For example: cluster_name, network_availability_zones, availability_zones, ssh_key_name, db_name, db_username.

• After customizing the values in variables.tf configure the aws credentials using the below commands.

• Provide AWS access key id,AWS secret access key,Default region and Default output format.

• Set aws_session _token using the below command.

• To make sure that aws credentials are configured use the below command.

• The output should be similar to the below image.

• After that run the below commands in the terminal one after another.

• terraform init is used to initialize your code to download the requirements mentioned in your code.

• terraform plan is used to review changes and choose whether to simply accept them or not.

• terraform apply is used to accept changes and apply them against real infrastructure.

• After successfully running these commands we are able to set up the infra in aws. We are able to see the config file which is used to deploy the environment.

• Want to destroy the terraform use the below command.

Once DIGIT is installed, check the health of the system to ensure it is ready for usage:

• Check to make sure all services are running. This can be done by fetching all k8s pods:

kubectl get pods

All pods should be in "running" state.

• Step 2

• Step 3

Overview

This documentation serves as a comprehensive guide for migrating an older version of Apache Kafka v2.3.0 to the latest version v3.6.0. The latest version of Kafka has introduced significant changes, particularly in the adoption of Kraft as a controller, rendering the previous dependence on Zookeeper unnecessary.

Steps

• The first step is to stop receiving the requests from nginx-ingress.

• Make sure all the Kafka data is consumed by the consumer and the offsets are set to zero.

• Next, take a backup of the old Kafka volume snapshots.

• Now, deploy the latest version of Kafka using the below commands.

git clone https://github.com/egovernments/DIGIT-DevOps.git
cd Digit-DevOps
git checkout unified-env
cd deploy-as-code/deployer
export KUBECONFIG=<path_to_your_kubeconfig>
kubectl config currrent context
go run main.go deploy -e <env_file_name> kafka-kraft 

• If you want to customize the values of the new Kafka helm chart according to your requirements i.e. storage_size, namespace etc, a path to the helm chart is provided below.

https://github.com/egovernments/DIGIT-DevOps/tree/unified-env/deploy-as-code/helm/charts/backbone-services/kafka-kraft

• Once the kafka-kraft helm chart is deployed and all the Kafka pods are running successfully, change the kafka-brokers value in egov-configmap as "release-name-kafka-controller-headless.kafka-kraft:9092".

https://github.com/egovernments/DIGIT-DevOps/blob/unified-env/deploy-as-code/helm/environments/egov-demo.yaml#L27

• After updating the kafka-broker value in configmap, make sure to restart all the pods which use kafka, to update the kafka-brokers value.

• The last step is to start the nginx-ingress service to again receive the requests.

• DB monitoring, alerting and debugging guidelines

• How to check if Infra is working as expected?

• How to monitor and setup alerts? Other debugging tools?

• Solutions to common problems and next steps

• Monitoring how-to

• Debugging

• Fixing/escalating

Overview

Unlike rolling upgrades, direct upgrades involve migrating from an older version to a newer one in a single coordinated operation.

This comprehensive guide outlines the step-by-step process for deploying an Elasticsearch 8.11.3 cluster with enhanced security features. The document not only covers the initial deployment of the cluster but also includes instructions for seamlessly migrating data from an existing Elasticsearch cluster to the new one, allowing for a direct upgrade.

Steps

1. Clone the DIGIT-DevOps repo and checkout to the branch <branch_name>.

git clone https://github.com/egovernments/DIGIT-DevOps.git
git checkout digit-lts-go
code .

1. If you want to make any changes to the elasticsearch cluster like namespaces etc. You'll find the helm chart for elastic search in the path provided below. In the below chart, security is enabled for elasticsearch. If you want to disable the security, please set the environment variable xpack.security.enabled as false in the helmchart statefulset template.

cd deploy-as-code/helm/backbone-services/elasticsearch-master
cd deploy-as-code/helm/backbone-services/elasticsearch-data

1. Deploy the Elastic Search Cluster using the below commands.

cd deploy-as-code/deployer
export KUBECONFIG=<path_to_kubeconfig>
kubectl config current-context
go run main.go deploy -e <env_file_name> elasticsearch-master
go run main.go deploy -e <env_file_name> elasticsearch-master

1. Check the pods status using the below command.

kubectl get pods -n <elasticsearch_namespace>

1. Once all pods are running, execute the below commands inside the playground pod to dump data from the old elasticsearch cluster and restore it to the new elasticsearch cluster.

# Copy the script and replace elasticsearch url's and authentication credentials and save it in your local
#!/bin/bash

# Elasticsearch cluster information
ELASTICSEARCH_OLD_URL="<old_elasticsearch_url>"
ELASTICSEARCH_NEW_URL="<new_elasticsearch_url>"

# Authentication credentials
USERNAME="elastic"
PASSWORD="<es_pwd>

#  Replace Elasticsearch cluster URL in elasticsearch_url 
ELASTICSEARCH_URL="http://elasticsearch-data-v1.es-cluster:9200"
# Provide the indices to take dump
EXCLUDE_INDEX_PATTERN="jaeger|monitor|kibana|fluentbit"
# Provide backup directory
BACKUP_DIR="backup"
# Provide indices output file
IDICES_OUTPUT="elasticsearch-indexes.txt"

mapfile -t INDICES < <(curl -s ${ELASTICSEARCH_OLD_URL}/_cat/indices | grep -v -E "(${EXCLUDE_INDEX_PATTERN})" | awk '{print $3}')

printf "%s\n" "${INDICES[@]}" > $IDICES_OUTPUT

# Create backup directory if it doesn't exist
mkdir -p "$BACKUP_DIR"

# Loop through each index and perform export
for INDEX in "${INDICES[@]}"; do
    OUTPUT_FILE="${BACKUP_DIR}/${INDEX}_mapping_backup.json"

    # Build the elasticdump command
    ELASTICDUMP_CMD="elasticdump \
        --input=http://${ELASTICSEARCH_OLD_URL}/${INDEX} \
        --output=${OUTPUT_FILE} \
        --type=mapping"

    # Execute the elasticdump command
    $ELASTICDUMP_CMD

    # Check if the elasticdump command was successful
    if [ $? -eq 0 ]; then
        echo "Backup of index ${INDEX} mapping completed successfully."
    else
        echo "Error backing up index ${INDEX}."
    fi
done

for INDEX in "${INDICES[@]}"; do
    OUTPUT_FILE="${BACKUP_DIR}/${INDEX}_data_backup.json"

    # Build the elasticdump command
    ELASTICDUMP_CMD="elasticdump \
        --input=http://${ELASTICSEARCH_OLD_URL}/${INDEX} \
        --output=${OUTPUT_FILE} \
        --type=data"

    # Execute the elasticdump command
    $ELASTICDUMP_CMD

    # Check if the elasticdump command was successful
    if [ $? -eq 0 ]; then
        echo "Backup of index ${INDEX} completed successfully."
    else
        echo "Error backing up index ${INDEX}."
    fi
done

for INDEX in "${INDICES[@]}"; do
    OUTPUT_FILE="${BACKUP_DIR}/${INDEX}_mapping_backup.json"
    
    # Process the mapping file to remove unsupported parameters

    PROCESSED_FILE="${BACKUP_DIR}/${INDEX}_mapping_processed.json"

    jq 'del(.mappings._default_, .mappings._meta, .mappings.dynamic_templates, .mappings.dynamic, .mappings.general) | .mappings = .mappings["_doc"]' "${INPUT_FILE}" > "${PROCESSED_FILE}"

    # Print the contents of the processed file for debugging

    echo "Contents of ${PROCESSED_FILE}:"

    cat "${PROCESSED_FILE}"

    # Build the elasticdump command
    ELASTICDUMP_CMD="elasticdump \
        --input=${PROCESSED_FILE} \
        --output=https://${USERNAME}:${PASSWORD}@${ELASTICSEARCH_NEW_URL}/${INDEX} \
        --type=mapping"

    # Execute the elasticdump command
    $ELASTICDUMP_CMD

    # Check if the elasticdump command was successful
    if [ $? -eq 0 ]; then
        echo "Restoring of index ${INDEX} mapping completed successfully."
    else
        echo "Error Restoring index ${INDEX}."
    fi
done

for INDEX in "${INDICES[@]}"; do
    OUTPUT_FILE="${BACKUP_DIR}/${INDEX}_data_backup.json"

    # Build the elasticdump command
    ELASTICDUMP_CMD="elasticdump \
        --input=${OUTPUT_FILE} \
        --output=https://${USERNAME}:${PASSWORD}@${ELASTICSEARCH_NEW_URL}/${INDEX} \
        --type=data"

    # Execute the elasticdump command
    $ELASTICDUMP_CMD

    # Check if the elasticdump command was successful
    if [ $? -eq 0 ]; then
        echo "Restoring of index ${INDEX} data completed successfully."
    else
        echo "Error Restoring index ${INDEX}."
    fi
done

kubectl get pods -n playground
kubectl cp <path_to_script_in_your_machine>/es-dump.sh playground/<playground_name>:<path_in_playground_pod>/es-dump.sh

# Execute into the playground pod shell and run the below command
kubectl exec -it <playground_pod_name> -n playground  bash

# Run the script which takes dump of your elasticsearch data using below command
cd <path_to_script_inside_playground_pod>
./es-dump.sh

1. Using the above script, you can take the data dump from the old cluster and restore it in the new elasticsearch in a single command.

2. After restoring the data successfully in the new elasticsearch cluster, check the cluster health and document count using the below command.

# Enter into elasticsearch pod
kubectl exec -it <elasticsearch_data_pod_name> -n <elasticsearch_namespace>  bash

# To check cluster health 
curl -X GET "<elasticsearch_url>:9200/_cat/health?v=true&pretty"

# To check documents count and indices status
curl <new_elasticsearch_url>:9200/_cat/indices?v

1. Now the deployment and restoring the data are completed successfully. It's time to change the es_url and indexer_url in egov-config configmap using the below command.

kubectl edit configmap egov-config --namespace egov

1. Restart all the pods which have a dependency on elasticsearch to pick a new elasticsearch_url.

Monitor, debug, fix

What are the metrics to track for Kafka, Postgres and ES?

How/what to track?

Use tracing to track core service APIs. Add info on Jaeger.

• How to monitor each and every service

• How to debug

• Potential fixes

Pre-reads

https://kafka.apache.org/intro https://zookeeper.apache.org/

Pre-requisites

• kubectl is a CLI to connect to the kubernetes cluster from your machine

• Install Visualstudio IDE Code for better code/configuration editing capabilities

• Git

Status check of Kafka Broker's

Using the below command you can able list down the Kafka brokers and their status

kubectl get pods -n kafka-cluster

• If Kafka brokers are in crashloopbackoff or Error status

  • Describe the brokers and look for error

    kubectl describe kafka-v2-0 -n kafka-cluster

  kubectl describe kafka-v2-1 -n kafka-cluster

  kubectl describe kafka-v2-2 -n kafka-cluster

• Check Kafka broker's logs for error

  kubectl logs -f kafka-v2-0 -n kafka-cluster

  kubectl logs -f kafka-v2-1 -n kafka-cluster

  kubectl logs -f kafka-v2-2 -n kafka-cluster

• If brokers are in crashloopbackoff due to disk space issues, follow the below document for the cleanup of the logs

  • Cleanup of Kafka logs

Status check of Zookeeper

Ensure Zookeeper pods are running without any errors in order to run Kafka brokers without a hitch

• If Zookeeper pods are in crashloopbackoff or Error status, Use the below commands to check the error

  • Describe the Zookeeper and look for error

    kubectl describe zookeeper-v2-0 -n zookeeper-cluster

    kubectl describe zookeeper-v2-1 -n zookeeper-cluster

    kubectl describe zookeeper-v2-2 -n zookeeper-cluster

  • Check Kafka broker's logs for error

    kubectl logs -f zookeeper-v2-0 -n zookeeper-cluster

    kubectl logs -f zookeeper-v2-1 -n zookeeper-cluster

    kubectl logs -f zookeeper-v2-2 -n zookeeper-cluster

Consumer offset is used to track the messages that are consumed by consumers in a consumer group. A topic can be consumed by many consumer groups and each consumer group will have many consumers. A topic is divided into multiple partitions.

A consumer in a consumer group is assigned to a partition. Only one consumer is assigned to a partition. A consumer can be assigned to consume multiple partitions.

Consumer offset is managed at the partition level per consumer group.

Why reset the consumer offset?

In some scenarios, consumers which consumed the messages from a Kafka partition could have resulted in errors and the consumption would have been incomplete. In such cases of consumption failures you may have a need to re-consume the messages which were previously consumed. In such instances you would have to reset the consumer offset to an earlier offset.

Follow the steps below if consumers stop consuming data from consumer group topics for any reason.

1. Get a Shell to a Kafka broker

   Copy

   kubectl get pods -n kafka-cluster

   Copy

   kubectl exec -it kafka-v2-0 -n kafka-cluster

2. Find the current consumer offset

   Use the kafka-consumer-groups along with the consumer group id followed by a describe.

   Copy

   kafka-consumer-groups --bootstrap-server kafka-v2.kafka-cluster:9092 --group <group_id> --describe

   You will see 2 entries related to offsets – CURRENT-OFFSET and LOG-END-OFFSET for the partitions in the topic for that consumer group. CURRENT-OFFSET is the current offset for the partition in the consumer group.

3. If you find out any topic lags that are not getting cleared then use the following steps to reset the consumer offset

   1. Scale down the respective consumer group service (eg. for egov-infra-persist you have to scale down the egov-persister service )

      Copy

      kubectl scale --replicas=0 deployment <deployment_name> -n egov

   2. Reset the consumer offset

      Use the kafka-consumer-groups to change or reset the offset. You would have to specify the topic, consumer group and use the –reset-offsets flag to change the offset.

      Copy

      kafka-consumer-groups --bootstrap-server kafka-v2.kafka-cluster:9092  --group <group_id>   --topic <topic_name> --reset-offsets --to-datetime 2021-06-25T21:22:39.306 --execute

      Reset offsets to offset from datetime. Format: ‘YYYY-MM-DDTHH:mm:SS.sss’

   3. Scale up the respective consumer group service

      Copy

      kubectl scale --replicas=2 deployment <deployment_name> -n egov

Backbone services - Kafka, DB

Infra

Core services

Applications

SRE Team's Monthly Tasks:

1. Cloud Cost - Monitoring/Optimization/Publishing

2. Infra Utilization Summary

3. History of deployments

4. History of Config changes

5. History of release (if any) post-release findings if any.

6. Monthly summary update

SRE Team's Weekly Tasks:

1. Cleanup logs

2. Backup logs

3. Weekly DB dump in case of SDC

4. ES Data backup

5. Publish Weekly Summary report/Come up with the format

6. Publish JIRA status

SRE Team's Daily Tasks:

1. Monitor the status of the environment and ensure every single service is running

2. Keep track of all tasks by creating tickets

3. Attend Daily scrums

4. Monitor the Prometheus Alters

• How to identify security issues - where to look

• Troubleshooting

• Solutions

Before installing Helm you need to know about Yaml files. Yaml file: Yaml files (Yet another Markup Language)are used to transmitting the data in web applications. Json file: Json files (JavaScript Object Notation) are a standard text based format to view the structured data of javascript object syntax.

Both Json and yaml files are used for transmitting data between web applications but the only difference is, In Yaml file there is indentation like python that shows the level of your code unlike Json

Prerequisites:

1. Git(please visit GitOps page if you didn't installed Git)

2. Install visual studio https://code.visualstudio.com/download IDE Code for better code visualization/editing capabilities

3. Install Golang https://go.dev/doc/install#download(required version:V1.13.3)

4. Kubectl(see working with kubernetes page to install kubectl)

• What is helm? helm can be defined as package manager for kubernetes It is used to deploy(to extend) the applications and services easily into kubernetes cluster in the form of Helm charts.

• what are helm charts? It consists set of templates and a file containing variables used to fill these templates based on the custom values and configurations.

• Why we are using helm in DIGIT?

1. Greatly improved productivity

2. Reduced complexity of deployments

3. More streamlined CI/CD pipeline

Helm charts are written in YAML and contain everything your developers need to deploy a container to a Kubernetes cluster.You may be used to creating Pods, Deployments, Services etc. in Kubernetes via the kubectl create command. This way of creating objects is indeed valid and great for learning purposes. However, when running Kubernetes in production you often want to have all your objects defined as .yaml files. This makes it easier for others to know what’s running in the cluster, and allows for your deployments to be version controlled.

To install Helm, follow the link provided below https://helm.sh/docs/intro/install/

• We have to make sure t

For any logs that appear to be overflowing and consuming disk space, you can use the following steps to clean up the logs from Kafka brokers

Note: Make sure the team is informed before doing this activity. This activity will delete the Kafka topic data

Steps

1. Backup list of log file names and their disk consumption data (optional)

   kubectl exec -it kafka-v2-0 -- du -h /opt/kafka-data/logs |tee backup_0.logs

   kubectl exec -it kafka-v2-1 -- du -h /opt/kafka-data/logs |tee backup_1.logs

   kubectl exec -it kafka-v2-2 -- du -h /opt/kafka-data/logs |tee backup_2.logs

2. Cleanup the logs

   kubectl exec -it kafka-v2-0 -- rm -rf /opt/kafka-data/logs/* -n kafka-cluster

   kubectl exec -it kafka-v2-1 -- rm -rf /opt/kafka-data/logs/* -n kafka-cluster

   kubectl exec -it kafka-v2-2 -- rm -rf /opt/kafka-data/logs/* -n kafka-cluster

3. If the pod is in crashlookbackoff state, and the storage is full, use the following workaround:

• Make a copy of the pod manifest

  kubectl get statefulsets kafka-v2 -n kafka-cluster -oyaml > manifest.yaml

• Scale down the Kafka statefulset replica count to zero

  kubectl scale statefulsets kafka-v2 -n kafka-cluster --replicas=0

• Make the following changes to the copy of the statefulsets manifest file

  • Modify the command line from:

containers:
- command:
  - sh
  - -exc
  - |
    export KAFKA_BROKER_ID=${HOSTNAME##*-} && \
    export KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://${POD_NAME}.kafka-v2-headless.${POD_NAMESPACE}:9092,EXTERNAL://${HOST_IP}:$((31090 + ${KAFKA_BROKER_ID})) && \
    exec /etc/confluent/docker/run

To

containers:
- command:
  - sh
  - -exc
  - |
    tail -f /dev/null

• Apply this statefulsets manifest and scale up statefulsets replica count to 3, the pod should be in a running state now and follow [step 2].

• Again scale down the Kafka statefulset replica count to zero

  kubectl scale statefulsets kafka-v2 --replicas=0 -n kafka-cluster

• Make the following changes to the copy of the statefulsets manifest file

  • Modify the command line from:

containers:
- command:
  - sh
  - -exc
  - |
    tail -f /dev/null

To

containers:
- command:
  - sh
  - -exc
  - |
    export KAFKA_BROKER_ID=${HOSTNAME##*-} && \
    export KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://${POD_NAME}.kafka-v2-headless.${POD_NAMESPACE}:9092,EXTERNAL://${HOST_IP}:$((31090 + ${KAFKA_BROKER_ID})) && \
    exec /etc/confluent/docker/run