Overview

DIGIT is the largest urban governance platform built for billions and billions of transactions between citizens and the state govt through various municipal services/integration. The platform is built with key capabilities like scale, speed, integration, configurable, customizable, extendable, multi-tenanted, security, etc. Here, we shall discuss the key requirements and capabilities.

Pre-requisites

Before proceeding to set up DIGIT, it is essential to know some of the key technical details about DIGIT, like architecture, tech stack and how it is packaged and deployed on various infrastructures. Some of these details are explained in the previous sections. Below are some of the key capabilities to know about DIGIT as a platform.

1. DIGIT is a collection of various services built as RESTFul APIs with OpenAPI standard

2. DIGIT is built as MSA (Microservices Architecture)

3. DIGIT services are packaged as containers and deployed as docker Images.

4. DIGIT is deployed on Kubernetes which abstracts any Cloud/Infra suitable and standardised for DIGIT deployment.

5. DIGIT deployment, configuration and customization are done through Helm Charts.

6. Kubernetes cluster setup is done through code like terraform/ansible suitably.

Why OpenAPI

The OpenAPI Specification (OAS) defines a standard, programming language-agnostic interface description for REST APIs, which allows both humans and computers to discover and understand the capabilities of a service without requiring access to source code, additional documentation, or inspection of network traffic. When properly defined via OpenAPI, a consumer can understand and interact with the remote service with a minimal amount of implementation logic. Similar to what interface descriptions have done for lower-level programming, the OpenAPI Specification removes the guesswork in calling a service.

Why Microservice Architecture

Microservices are nothing but breaking big beasts into smaller units that can independently be developed, enhanced and scaled as a categorized and layered stack that gives better control over each component of an application that exists in its own container, independently managed and updated. This means that developers can build applications from multiple components and program each component in the language best suited to its function, rather than having to choose a single less-than-ideal language to use for everything. Optimizing software all the way down to the components of the application helps you increase the quality of your products. No time and resources are wasted managing the effects of updating one application on another.

Why Containerized/Dockerized

Comparatively the best infra choice for running a microservices application architecture is application containers. Containers encapsulate a lightweight runtime environment for the application, presenting a consistent environment that can follow the application from the developer's desktop to testing to final production deployment, and you can run containers on cloud infra with physical or virtual machines.

Why Kubernetes

As most modern software developers can attest, containers have provided us with dramatically more flexibility for running cloud-native applications on physical and virtual infrastructure. Kubernetes allows you to deploy cloud-native applications anywhere and manage them exactly as you like everywhere. For more details refer to the above link that explains various advantages of Kubernetes.

Why Helm Charts

Kubernetes, the popular container orchestration system, is used extensively. However, it can become complex: you have to handle all of the objects (ConfigMaps, pods, etc.), and would also have to manage the releases. Both can be accomplished with Kubernetes Helm. It is a Kubernetes package manager designed to easily package, configure, and deploy applications and services onto Kubernetes clusters in a standard way, this helps the ecosystem to adopt the standard way of deployment and customization.

For being successful in the DIGIT Setup, below are certain requirements that need to be ascertained:

Skills Needed

• On-premise/private cloud accounts

  • Interface to access and provision required infra

  • In the case of SDC, NIC or private DC, it'll be VPN to an allocated VLAN

  • SSH access to the VMs/machines

• Infra Skills

  • Public cloud

    • Managed Kubernetes services like AKS or EKS or GKE on Azure, AWS and GCP respectively

  • Private Clouds (SDC, NIC)

    • Clouds like VMware, OpenStack, Nutanix and more, may or may not have Kubernetes as a managed service. If yes we may have to estimate only the worker nodes depending on the number of ULBs and DIGIT's municipal services that you opt.

    • In the absence of the above, you have to provision the Kubernetes cluster from the plain VMs as per the general Kubernetes setup instruction and add worker nodes.

• Operations Skills

  • Understanding of Linux, containers, VM Instances, Load Balancers, Security Groups/Firewalls, Nginx, DB Instance, Data Volumes

  • Experience with Kubernetes, Docker, Jenkins, Helm, Infra-as-code, Terraform

  • Experience in DevOps/SRE practice on microservices and modern infrastructure

High-level Action To Deploy DIGIT

1. Provisioning the Kubernetes Cluster in any of the

   • Commercial cloud or

   • Private State datacenter (SDC) or

   • National Cloud (NIC)

2. Setting up the persistent disk volumes to attach to DIGIT backbone stateful containers like

   • ZooKeeper

   • Kafka

   • Elastic Search

3. Setting up the Postgres DB

   • On a public cloud, provision a Postgres RDS instance.

   • Private cloud, provision a Postgres DB on a VM with the backup, HA/DRS

4. Preparing deployment configuration for required DIGIT services using Helm templates from the InfraOps like the following

   • Preparing DIGIT service helm templates to deploy on the Kubernetes cluster

   • K8s Secrets

   • K8s ConfigMaps

   • Environment variables of each microservices

5. Deploy the stable released version of DIGIT and the required services

6. Setting up Jenkins job to build, bake images and deploy the components for the rolling updates

7. Setup Application monitoring, Distributed Tracing, Alert management

Introduction

This document mainly aims to help the implementation team in configuring and customizing the eDCR engine as per the state/city rules and regulations. Samples are given for each of the activities that are to be done. This can be considered as a handbook on how to approach the DCR automated scrutiny for a newcomer.

Activities Involved

Below mentioned are the various activities to be done as part of eDCR customization and configurations for the state/city. Also, it mentioned the training needs for the end-users.

Layer Matrix/Template

Layer Matrix contains the colour and layer names to be applied while drawing a plan that needs to be scrutinized in the eDCR system. A sample is provided here. If there are new layers to be added or a new occupancy type to be considered, additions will have to be made to layers and colours. This needs to be done by a person having architecture and Auto Cad knowledge. The document needs to be updated with the new layers and colours and the same need to be shared with the architects.

Drawing Manual

Drawing manual maps list of layers used in marking for a feature. Here is a sample drawing manual for a client. Similarly, this customer-specific manual needs to be created.

Drawing Details

Drawing details provide additional details required for marking or drawing a part or full feature.

Drawing details can be understood and prepared using the column “Parameters /Additional details to be addressed in drawings “ column from this sheet.

Data Collection Template

Rules need to be read and understood from the state rule book and this information has to be translated into a data template. This will enable the customization team to validate and codify the rules efficiently. Here is a reference to a sample data collection template. This may need to be enhanced for any state-specific rules if it cannot fit into the sample one.

Configuration & Customization

Based on the rules documented in the data template, the implementation team needs to make the required configuration changes in the existing features. In cases where the default implementation is different from the state rules, customization needs to be done by the implementation team. If there are any rules in a state which are not covered in any of the features in eDCR, this needs to be coded afresh.

Example 1: where there is a change in the existing rule and the rule number-

Change in the FAR (Floor Area Ratio) in a particular state, where the expected value is 1.4 instead of 1.2 which is the default. Changes will need to be done in the FAR.java file for this.

Example 2: Where there is a new rule, say -distance from the monument -

If this rule is not currently available in eDCR, you will need to add a new file with the rule name, where the implementation of the rule will be written.

Help Videos

Videos are helping tools for architects to make a plan drawing as per the standards defined by the eDCR system. Implementation partners will be training the end-user which is an architect using these help videos. These are the Sample videos that are done for Kozhikode, Kerala state.

Overview

This page discusses the infrastructure requirements for DIGIT services. It also explains why DIGIT services are containerised and deployed on Kubernetes.

Requirements

DIGIT Infra is abstracted to Kubernetes which is an open-source containers orchestration platform that helps in abstracting a variety of infra types that are being available across each state, like Physical, VMs, on-premises clouds(VMware, OpenStack, Nutanix, etc.), commercial clouds (Google, AWS, Azure, etc.), SDC and NIC into a standard infra type. Essentially it unifies various infra types into a standard and single type of infrastructure and thus DIGIT becomes multi-cloud supported, portable, extensible, high-performant and scalable containerized workloads and services. This facilitates both declarative configuration and automation. Kubernetes services, eco-system, support and tools are widely available.

The basic need to provision Kubernetes Cluster

Kubernetes as such is a set of components that designated jobs of scheduling, controlling, monitoring

Master Cluster

• 3 or more machines running one of:

  • Ubuntu 16.04+

  • Debian 9

  • CentOS 7

  • RHEL 7

  • Container Linux (tested with 1576.4.0)

• 4 GB or more of RAM per machine (any less will leave little room for your apps)

• 2 CPUs or more

User Cluster

• 3 or more machines running one of:

  • Ubuntu 16.04+

  • Debian 9

  • CentOS 7

  • RHEL 7

  • Container Linux (tested with 1576.4.0)

• 2 GB or more of RAM per machine (any less will leave little room for your apps)

• 2 CPUs or more

• Full network connectivity between all machines in the cluster (public or private network is fine)

• Certain ports are open on your machines. See below for more details

• Swap disabled. You MUST disable swap in order for the Kubelet to work properly

Verify the MAC Address and product_uuid Are Unique for Every Node

• You can get the MAC address of the network interfaces using the command ip link or ifconfig -a

• The product_uuid can be checked by using the command sudo cat /sys/class/dmi/id/product_uuid

Check Network Adapters

If you have more than one network adapter, and your Kubernetes components are not reachable on the default route, we recommend you add IP route(s) so Kubernetes cluster addresses go via the appropriate adapter.

Check Required Ports

Master Cluster Master Node(s)

Worker Node(s)& User Cluster Worker Nodes

Any port numbers marked with * are overridable, so you will need to ensure any custom ports you provide are also open.

Complete Infra Specifications

Download the file below to view the DIGIT Rollout Program Governance structure and process details.

Introduction

Currently, ULBs follow a complete manual mass mode of providing various building-related approvals and related NOCs. This delays the approval process and creates a lot of inconvenience for applicants and architects. There is also a lack of transparency and accountability in this manual approach. Due to the complexity of scrutiny and approval processes and the involvement of various stakeholders, there are several challenges in the system. Some of the issues are-Complex rules and bye-laws and their interpretation, the time-consuming process of issuance of Planning Permission, increased chances of human errors due to manual submission and scrutiny, inadequate citizen interface and no single point of contact, etc.

The manual scrutiny of the building bye-laws is replaced with “Digit-DCR”.Digit-DCR is capable of reading the drawing, extracting parameter values, applying them to bylaws and generating the acceptance or rejection information in seconds. This process provides a leap of 15 days of cumbersome manual work to a few seconds. The Digit-DCR is even capable of marking mistakes and failing details. It converts details of drawing like Site plans, Floor plans, Elevation, etc to pdf and saves them in the system.

Prerequisites

Knowledge on following

1. Java

2. Basics of 2D Drawing like ( Top view, Front view, Side view, etc)

3. Basic Math

Functionality

High-level design

Sequence Diagram

Configuration and Set up

References and Notes

Layer Name Standards

The general Practice of the Architects is to draw diagrams in the default layer with the name “0” with portions stating Section Plan, Floor Plan, etc. With this approach, it is not possible to extract parts of the building for Bye-Laws validation. Hence Parts of the buildings should be drawn in specific layers. These layers also should have a standard format for machine reading.

For example, Plan will have a “Bathroom” and "Washbasin” which should be validated for dimensions and counts for Sanitation. Hence these sanitation-related parts should be drawn in a layer called “BLK_1_WATER_CLOSET” layer, where “BLK_1” states it is block 1 and “WATER_CLOSET” states it is a toilet. Another example is Toilets for the disabled should be validated floor wise Hence it should be defined as Floorwise layer names like “BLK_1_FLR_0_SP_WC” layer.

We cannot define a complete set of layer names with these combinations because it is based on the following parameters

1. Number of Blocks

2. Number of Floors in each Block

3. Number of Polygons on each floor

4. Number of levels

So when data is required at Plan Level, Block Level, Floor Level, or Height it is named and prefixed accordingly

Plan Level

At the Plan level, the layer names are used without any prefixes. The list of such layer names are

Block Level

Floor Level

Color Code Standards

Some parts of the building and its validation is based on Usage(Occupancy) types. For example, every floor in which all types of usage present have to be defined. Based on usage type parking has to be calculated. So only layer name is not sufficient to differentiate but colour also to include. Hence Floorwise built-up area will be defined in “BLK_1_FLR_0_BLT_UP_AREA” with index colour 2 for Residential type of usage. All colours are defined based on Occupancy type. The below list provides the colour codes for each occupancy-

Drawing Standards

The following components are used in the drawing. EDCR deals with only a defined set of drawing components.

Introduction

This article provides a DIGIT Infra overview, Guidelines for Operational Excellence while DIGIT is deployed on SDC, NIC or any commercial clouds along with the recommendations and . It helps to plan the procurement and build the necessary capabilities to deploy and implement DIGIT.

In shared control, the state program team/partners can consider these guidelines and must provide their own control implementation to the state’s cloud infrastructure and partners for standard and smooth operational excellence.

Operational Recommendations

DIGIT strongly recommends (SRE) principles as a key means to bridge development and operations by applying a software engineering mindset to the system and IT administration topics. In general, an SRE team is responsible for availability, latency, performance, efficiency, change management, monitoring, emergency response, and capacity planning.

Monitoring Tools Recommendations: Commercial clouds like , and GCP offer sophisticated monitoring solutions across various infra levels like CloudWatch and StackDriver, in the absence of such managed services to monitor we can look at various best practices and tools listed below which helps efficiently.

• , , , ****

• ,

Key Standard Operating Procedures (SOPs)

• Segregation of duties and responsibilities.

• SME and SPOCs for support along with the SLAs defined.

• to manage incidents, converge and collaborate on various operational issues.

• Monitoring dashboards at various levels like Infrastructure, Networks and applications.

• Transparency of monitoring data and collaboration between teams.

• Periodic remote sync-up meetings, acceptance and attendance to the meeting.

• Ability to see stakeholders' availability of calendar time to schedule meetings.

• Periodic (weekly, monthly) summary reports of the various infra, operations incident categories.

• Communication channels and synchronization on a regular basis and also upon critical issues, changes, upgrades, releases etc.

Segregation of Duties

While DIGIT is deployed at state cloud infrastructure, it is essential to identify and distinguish the responsibilities between Infrastructure, Operations and Implementation partners. Identify these teams and assign SPOC, define responsibilities and Incident management followed to visualize, track issues and manage dependencies between teams. Essentially these are monitored through dashboards and alerts are sent to the stakeholders proactively. eGov team can provide consultation and training on a need basis depending on any of the below categories.

• and State program team - Refers to the owner for the whole DIGIT implementation, application rollouts, and capacity building. Responsible for identifying and synchronizing the operating mechanism between the below teams.

• Implementation partner - Refers to the DIGIT Implementation, application performance monitoring for errors, logs scrutiny, TPS on peak load, distributed tracing, DB queries analysis, etc.

• Operations team - this team could be an extension of the implementation team and is responsible for DIGIT deployments, configurations, CI/CD, change management, traffic monitoring and alerting, log monitoring and dashboard, application security, DB Backups, application uptime, etc.

• **State IT/Cloud team -**Refers to state infra team for the Infra, network architecture, LAN network speed, internet speed, OS Licensing and upgrade, patch, compute, memory, disk, firewall, IOPS, security, access, SSL, DNS, data backups/recovery, snapshots, capacity monitoring dashboard.

Skills Required to Set up, Operate and Maintain DIGIT on SDC

Best practices for securing your Kubernetes cluster

Kubernetes has changed the way organizations deploy and run their applications, and it has created a significant shift in mindsets. While it has already gained a lot of popularity and more and more organizations are embracing the change, running Kubernetes in production requires care.

Although Kubernetes is open source and does have its share of vulnerabilities, making the right architectural decision can prevent a disaster from happening.

You need to have a deep level of understanding of how Kubernetes works and how to enforce the best practices so that you can run a secure, highly available, production-ready Kubernetes cluster.

Although Kubernetes is a robust container orchestration platform, the sheer level of complexity with multiple moving parts overwhelms all administrators.

That is the reason why Kubernetes has a large attack surface, and, therefore, hardening of the cluster is an absolute must if you are to run Kubernetes in production.

There are a massive number of configurations in K8s, and while you can configure a few things correctly, the chances are that you might misconfigure a few things.

I will describe a few best practices that you can adopt if you are running Kubernetes in production. Let’s find out.

Use a Managed Kubernetes Service if Possible

If you are running your Kubernetes cluster in the cloud, consider using a managed Kubernetes cluster such as Google Kubernetes Engine or Azure Kubernetes Service.

A managed cluster comes with some level of hardening already in place, and, therefore, there are fewer chances to misconfigure things. A managed cluster also makes upgrades easy, and sometimes automatic. It helps you manage your cluster with ease and provides monitoring and alerting out of the box.

Upgrade Kubernetes Frequently

Since Kubernetes is open source, vulnerabilities appear quickly and security patches are released regularly. You need to ensure that your cluster is up to date with the latest security patches and for that, add an upgrade schedule to your standard operating procedure.

Having a CI/CD pipeline that runs periodically for executing rolling updates for your cluster is a plus. You would not need to check for upgrades manually, and rolling updates would cause minimal disruption and downtime; also, there would be fewer chances to make mistakes.

That would make upgrades less of a pain. If you are using a managed Kubernetes cluster, your cloud provider can cover this aspect for you.

Patch and Harden Your OS

It goes without saying that you should patch and harden the operating system of your Kubernetes nodes. This would ensure that an attacker would have the least attack surface possible.

You should upgrade your OS regularly and ensure that it is up to date.

Enforce RBAC

Kubernetes post version 1.6 has role-based access control (RBAC) enabled by default. Ensure that your cluster has this enabled.

You also need to ensure that legacy attribute-based access control (ABAC) is disabled. Enforcing RBAC gives you several advantages as you can now control who can access your cluster and ensure that the right people have the right set of permissions.

RBAC does not end with securing access to the cluster by Kubectl clients but also by pods running within the cluster, nodes, proxies, scheduler, and volume plugins.

Only provide the required access to service accounts and ensure that the API server authenticates and authorizes them every time they make a request.

Use TLS

Running your API server on plain HTTP in production is a terrible idea. It opens your cluster to a man-in-the-middle attack and would open up multiple security holes.

Always use transport layer security (TLS) to ensure that communication between Kubectl clients and the API server is secure and encrypted.

Be aware of any non-TLS ports you expose for managing your cluster. Also ensure that internal clients such as pods running within the cluster, nodes, proxies, scheduler, and volume plugins use TLS to interact with the API server.

Segregate Resources in Namespaces

While it might be tempting to create all resources within your default namespace, it would give you tons of advantages if you use namespaces. Not only will it be able to segregate your resources in logical groups but it will also enable you to define security boundaries to resources in namespaces.

Namespaces logically behave as separate clusters within Kubernetes. You might want to create namespaces based on teams, or based on the type of resources, projects, or customers depending on your use case.

After that, you can do clever stuff like defining resource quotas, limit ranges, user permissions, and RBAC on the namespace layer.

Avoid binding ClusterRoles to users and service accounts, instead provide them namespace roles so that users have access only to their namespace and do not unintentionally misconfigure someone else’s resources.

Cluster Role and Namespace Role Bindings

Control Pod to Pod Traffic

You can use Kubernetes network policies that work as firewalls within your cluster. That would ensure that an attacker who gained access to a pod (especially the ones exposed externally) would not be able to access other pods from it.

You can create Ingress and Egress rules to allow traffic from the desired source to the desired target and deny everything else.

Kubernetes Network Policy

Create Separate User Accounts

By default, when you boot your cluster through kubeadm, you get access to the kubernetes-admin config file which is the superuser for performing all activities within your cluster.

Do not share this file with your team, instead, create a separate user account for every user and only provide the right access to them. Bear in mind that Kubernetes does not maintain an internal user directory, and therefore, you need to ensure that you have the right solution in place to create and manage your users.

Once you create the user, you can generate a private key and a certificate signing request for the user, and Kubernetes would sign and generate a CA cert for the user.

You can then securely share the CA certificate with the user. The user can then use the certificate within kubectl to authenticate with the API server securely.

Configuring User Accounts

Follow the Principle of Least Privilege

You can provide granular access to user and service accounts with RBAC. Let us consider a typical organization where you can have multiple roles, such as:

1. Application developers — These need access only to a namespace and not the entire cluster. Ensure that you provide them with access only to deploy their applications and troubleshoot them only within their namespace. You might want application developers with access to spin-only ClusterIP services and might wish to grant permissions only to network administrators to define ingresses for them.

2. Network administrators — You can provide network admins access to networking features such as ingresses, and privileges to spin up external services.

3. Cluster administrators — These are sysadmins whose main job is to administer the entire cluster. These are the only people that should have cluster-admin access and only the amount that is necessary for them to do their roles.

The above is not etched in stone, and you can have a different organisational policy and different roles, but the only thing to keep in mind here is that you need to enforce the principle of least privilege.

That means that individuals and teams should have only the right amount of access they need to perform their job, nothing less and nothing more.

Frequently Rotate Infrastructure Credentials

It does not stop with just issuing separate user accounts and using TLS to authenticate with the API server. It is an absolute must that you frequently rotate and issue credentials to your users.

Set up an automated system that periodically revokes the old TLS certificates and issues new ones to your user. That helps as you don’t want attackers to get hold of a TLS cert or a token and then make use of it indefinitely.

A bootstrap token, for example, needs to be revoked as soon as you finish with your activity. You can also make use of a credential management system such as HashiCorp Vault which can issue you with credentials when you need them and revoke them when you finish your work.

Use a Partitioned Approach to Secure Secrets

Imagine a scenario where an externally exposed web application is compromised, and someone has gained access to the pod. In that scenario, they would be able to access the secrets (such as private keys) and target the entire system.

The way to protect from this kind of attack is to have a sidecar container that stores the private key and responds to signing requests from the main container.

In case someone gets access to your login microservice, they would not be able to gain access to your private key, and therefore, it would not be a straightforward attack, giving you valuable time to protect yourself.

Partitioned Approach

Limit Resource Usage

The last thing you would want as a cluster admin is a situation where a poorly written microservice code that has a memory leak can take over a cluster node causing the Kubernetes cluster to crash. That is an extremely important and generally ignored area.

You can add a resource limit and requests on the pod level as a developer or the namespace as an administrator. You can use resource quotas to limit the amount of CPU, memory, or persistent disk a namespace can allocate.

It can also allow you to limit the number of pods, volumes, or services you can spin within a namespace. You can also make use of limit ranges that provide you with a minimum and maximum size of resources every unit of the cluster within the namespace can request.

That will limit users from seeking an unusually large amount of resources such as memory and CPU.

Specifying a default resource limit and request on a namespace level is generally a good idea as developers aren’t perfect. If they forget to specify a limit, then the default limit and requests would protect you from resource overrun.

Protect Your ETCD Cluster Like a Treasure Vault

The ETCD datastore is the primary source of data for your Kubernetes cluster. That is where all cluster information and the expected configuration are stored.

If someone gains access to your ETCD database, all security measures will go down the drain. They will have full control of your cluster, and they can do what they want by modifying the state in your ETCD datastore.

You should always ensure that only the API server can communicate with the ETCD datastore and only through TLS using a secure mutual auth. You can put your ETCD nodes behind a firewall and block all traffic except the ones originating from the API server.

Do not use the master ETCD for any other purpose but for managing your Kubernetes cluster and do not provide any other component access to the ETCD cluster.

Enable encryption of your secret data at rest. That is extremely important so that if someone gets access to your ETCD cluster, they should not be able to view your secrets by just doing a hex dump of your secrets.

Control Container Privileges

Containers run on nodes and therefore have some level of access to the host file system, however, the best way to reduce the attack surface is to architect your application in such a way that containers do not need to run as root.

Use pod security policies to restrict the pod to access HostPath volumes as that might result in getting access to the host filesystem. Administrators can use a restrictive pod policy so that anyone who gained access to one pod should not be able to access another pod from there.

Enable Auditing

Audit loggers are now a beta feature in Kubernetes, and I recommend you make use of it. That would help you troubleshoot and investigate what happened in case of an attack.

As a cluster admin dealing with a security incident, the last thing you would want is that you are unaware of what exactly happened with your cluster and who has done what.

Conclusion

Remember that the above are just some general best practices and they are not exhaustive. You are free to adjust and make changes based on your use case and ways of working for your team.

This section discusses the supported cloud environment for DIGIT services. It provides information on where and how DIGIT is deployed. Further, it offers guidelines on estimating the infrastructural requirements for cloud support.

Supported Cloud List:

• Google Cloud

• Azure

• AWS

• VSphere

• SDC

This section contains docs and information resources that guide you through the key DevOps concepts and their role in managing the DIGIT platform.

• Skills needed

• Resource requests & limits

• Readiness & liveness

• Troubleshooting

• CI/CD

• Security Practices

Compute Engine API

For access to the Compute Engine API, it has to be enabled at the Google APIs console.

User Roles

The user for the Google Service Account that has to be created has to have three roles:

• Compute Admin: roles/compute.admin

• Service Account User: roles/iam.serviceAccountUser

• Viewer: roles/viewer

Once the gcloud CLI is installed create a service account following the steps below:

# create new service account
gcloud iam service-accounts create k8c-cluster-provisioner

# get your service account id
gcloud iam service-accounts list
# get your project id
gcloud projects list

# create policy binding
gcloud projects add-iam-policy-binding YOUR_PROJECT_ID --member 'serviceAccount:YOUR_SERVICE_ACCOUNT_ID' --role='roles/compute.admin'
gcloud projects add-iam-policy-binding YOUR_PROJECT_ID --member 'serviceAccount:YOUR_SERVICE_ACCOUNT_ID' --role='roles/iam.serviceAccountUser' 
gcloud projects add-iam-policy-binding YOUR_PROJECT_ID --member 'serviceAccount:YOUR_SERVICE_ACCOUNT_ID' --role='roles/viewer'

Google Service Account

A Google Service Account for the platform should be created. Refer to Creating and managing service accounts. The result is a JSON file containing the fields

• type

• project_id

• private_key_id

• private_key

• client_email

• client_id

• auth_uri

• token_uri

• auth_provider_x509_cert_url

• client_x509_cert_url

The private key is BASE64 containing the newlines as non-escaped strings "\n”. So to avoid the resulting troubles the machine controller expects the full service account encoded in BASE64.

# create a new json key for your service account
gcloud iam service-accounts keys create --iam-account YOUR_SERVICE_ACCOUNT k8c-cluster-provisioner-sa-key.json
# create base64 encoded secret
base64 -w 0 ./k8c-cluster-provisioner-sa-key.json

Passing the Google Service Account

The base64 encoded secret of the service account will be passed in the field serviceAccount of the cloudProviderSpec of the machine deployment. The encoded secret can be entered in the UI field Service Account.

full-service

Overview

This page discusses the provisioning of the Kubernetes cluster which is an abstracted infrastructure requirement for DIGIT to be deployed. Learn how to provision infra-as-code on AWS using terraform.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "iam:GetInstanceProfile",
                "iam:ListInstanceProfiles"
            ],
            "Resource": "arn:aws:iam::YOUR_ACCOUNT_ID:instance-profile/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "iam:CreateRole",
                "iam:DeleteRole",
                "iam:DeleteRolePolicy",
                "iam:GetRole",
                "iam:ListAttachedRolePolicies",
                "iam:ListRolePolicies",
                "iam:PassRole",
                "iam:PutRolePolicy"
            ],
            "Resource": "arn:aws:iam::YOUR_ACCOUNT_ID:role/kubernetes-*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "iam:AddRoleToInstanceProfile",
                "iam:CreateInstanceProfile",
                "iam:DeleteInstanceProfile",
                "iam:GetInstanceProfile",
                "iam:RemoveRoleFromInstanceProfile"
            ],
            "Resource": "arn:aws:iam::YOUR_ACCOUNT_ID:instance-profile/kubernetes-*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:*",
                "elasticloadbalancing:CreateListener",
                "elasticloadbalancing:CreateRule",
                "elasticloadbalancing:CreateTargetGroup",
                "elasticloadbalancing:CreateLoadBalancer",
                "elasticloadbalancing:ConfigureHealthCheck",
                "elasticloadbalancing:DeleteListener",
                "elasticloadbalancing:DeleteRule",
                "elasticloadbalancing:DeleteTargetGroup",
                "elasticloadbalancing:DeleteLoadBalancer",
                "elasticloadbalancing:DeregisterTargets",
                "elasticloadbalancing:DescribeListeners",
                "elasticloadbalancing:DescribeRules",
                "elasticloadbalancing:DescribeTargetGroupAttributes",
                "elasticloadbalancing:DescribeTargetGroups",
                "elasticloadbalancing:DescribeTargetHealth",
                "elasticloadbalancing:DescribeLoadBalancers",
                "elasticloadbalancing:DescribeLoadBalancerAttributes",
                "elasticloadbalancing:ModifyListener",
                "elasticloadbalancing:ModifyRule",
                "elasticloadbalancing:ModifyTargetGroup",
                "elasticloadbalancing:ModifyTargetGroupAttributes",
                "elasticloadbalancing:ModifyLoadBalancerAttributes",
                "elasticloadbalancing:RegisterTargets",
                "elasticloadbalancing:RegisterInstancesWithLoadBalancer",
                "elasticloadbalancing:RemoveListenerCertificates",
                "elasticloadbalancing:SetIpAddressType",
                "elasticloadbalancing:SetRulePriorities",
                "elasticloadbalancing:SetSecurityGroups",
                "elasticloadbalancing:SetSubnets",
                "elasticloadbalancing:SetWebAcl",
                "sts:GetFederationToken"
            ],
            "Resource": "*"
        }
    ]
}

Overview

The Kubernetes vSphere driver contains bugs related to detaching volumes from offline nodes. See the Volume detach bug section for more details.

VM Images

When creating worker nodes for a user cluster, the user can specify an existing image. Defaults may be set in the datacenters.yaml.

Supported operating systems

• Ubuntu 18.04 ova

• CoreOS ova

• CentOS 7 qcow2

Importing the OVA

1. Go into the VSphere WebUI, select your data centre, right-click onto it and choose “Deploy OVF Template”

2. Fill in the “URL” field with the appropriate URL

3. Click through the dialogue until “Select storage”

4. Select the same storage you want to use for your machines

5. Select the same network you want to use for your machines

6. Leave everything in the “Customize Template” and “Ready to complete” dialogue as it is

7. Wait until the VM got fully imported and the “Snapshots” => “Create Snapshot” button is not greyed out anymore.

8. The template VM must have the disk.enable UUID flag set to 1, this can be done using the govc tool with the following command:

govc vm.change -e="disk.enableUUID=1" -vm='/PATH/TO/VM'

Importing the QCOW2

• Convert it to vmdk: qemu-img convert -f qcow2 -O vmdk CentOS-7-x86_64-GenericCloud.qcow2 CentOS-7-x86_64-GenericCloud.vmdk

• Upload it to a Datastore of your vSphere installation

• Create a new virtual machine that uses the uploaded vmdk as rootdisk.

Modifications

Modifications like Network, disk size, etc. must be done in the ova template before creating a worker node from it. If user clusters have dedicated networks, all user clusters, therefore, need a custom template.

VM Folder

During the creation of a user cluster, Kubermatic creates a dedicated VM folder in the root path on the Datastore (Defined in the datacenters.yaml). That folder will contain all worker nodes of a user cluster.

Credentials / Cloud-Config

Kubernetes needs to talk to the vSphere to enable Storage inside the cluster. For this, Kubernetes needs a config called cloud-config. This config contains all details to connect to a vCenter installation, including credentials.

As this Config must also be deployed onto each worker node of a user cluster, its recommended to have individual credentials for each user cluster.

Permissions

The VSphere user must have the following permissions on the correct resources

Seed Cluster

• Role k8c-storage-vmfolder-propagate

  • Granted at VM Folder and Template Folder, propagated

  • Permissions

    • Virtual machine

      • Change Configuration

        • Add existing disk

        • Add new disk

        • Add or remove the device

        • Remove disk

    • Folder

      • Create folder

      • Delete folder

• Role k8c-storage-datastore-propagate

  • Granted at Datastore, propagated

  • Permissions

    • Datastore

      • Allocate space

      • Low-level file operations

• Role Read-only (predefined)

  • Granted at …, not propagated

    • Datacenter

User Cluster

• Role k8c-user-vcenter

  • Granted at vcentre level, not propagated

  • Needed to customize VM during provisioning

  • Permissions

    • VirtualMachine

      • Provisioning

        • Modify customization specification

        • Read customization specifications

• Role k8c-user-datacenter

  • Granted at datacentre level, not propagated

  • Needed for cloning the template VM (obviously this is not done in a folder at this time)

  • Permissions

    • Datastore

      • Allocate space

      • Browse datastore

      • Low-level file operations

      • Remove file

    • vApp

      • vApp application configuration

      • vApp instance configuration

    • Virtual Machine

      • Change CPU count

      • Memory

      • Settings

    • Inventory

      • Create from existing

• Role k8c-user-cluster-propagate

  • Granted at the cluster level, propagated

  • Needed for upload of cloud-init.iso (Ubuntu and CentOS) or defining the Ignition config into Guestinfo (CoreOS)

  • Permissions

    • Host

      • Configuration

        • System Management

      • Local operations

        • Reconfigure virtual machine

    • Resource

      • Assign virtual machine to the resource pool

      • Migrate powered off the virtual machine

      • Migrate powered-on virtual machine

    • vApp

      • vApp application configuration

      • vApp instance configuration

• Role k8s-network-attach

  • Granted for each network that should be used

  • Permissions

    • Network

      • Assign network

• Role k8c-user-datastore-propagate

  • Granted at datastore/datastore cluster level, propagated

  • Permissions

    • Datastore

      • Allocate space

      • Browse datastore

      • Low-level file operations

• Role k8c-user-folder-propagate

  • Granted at VM Folder and Template Folder level, propagated

  • Needed for managing the node VMs

  • Permissions

    • Folder

      • Create folder

      • Delete folder

    • Global

      • Set custom attribute

    • Virtual machine

      • Change Configuration

      • Edit Inventory

      • Guest operations

      • Interaction

      • Provisioning

      • Snapshot management

The described permissions have been tested with vSphere 6.7 and might be different for other vSphere versions.

Volume Detach Bug

After a node is powered-off, the Kubernetes vSphere driver doesn’t detach disks associated with PVCs mounted on that node. This makes it impossible to reschedule pods using these PVCs until the disks are manually detached in vCenter.

Upstream Kubernetes has been working on the issue for a long time now and tracking it under the following tickets:

• https://github.com/kubernetes/kubernetes/issues/63577

• https://github.com/kubernetes/kubernetes/issues/61707

• https://github.com/kubernetes/kubernetes/issues/67900

• https://github.com/kubernetes/kubernetes/issues/71829

• https://github.com/kubernetes/kubernetes/issues/75342

Hiring DevOps Resources

Anyone involved in hiring DevOps engineers will realize that it is hard to find prospective candidates who have all the skills listed in this section.

Ultimately, the skill set needed for an incoming DevOps engineer would depend on the current and short-term focus of the operations team. A brand new team that is rolling out a new software service would require someone with good experience in infrastructure provisioning, deployment automation, and monitoring. A team that supports a stable product might require the service of an expert who could migrate home-grown automation projects to tools and processes around standard configuration management and continuous integration tools.

DevOps practice is a glue between engineering disciplines. An experienced DevOps engineer would end up working in a very broad swath of technology landscapes that overlaps with software development, system integration, and operations engineering.

An experienced DevOps engineer would be able to describe most of the technologies that are described in the following sections. This is a comprehensive list of DevOps skills for comparing one’s expertise and a reference template for acquiring new skills.

In theory, a template like this should be used only for assessing the current experience of a prospective hire. The needed skills can be picked up on jobs that demand deep knowledge in certain areas. Therefore, the focus should be to hire smart engineers who have a track record of picking up new skills, rolling out innovative projects at work, and contributing to reputed open-source projects.

1. Knowledge of Infrastructure

A DevOps engineer should have a good understanding of both classic (data centre-based) and cloud infrastructure components, even if the team has a dedicated infrastructure team.

A. Classic Infrastructure

This involves how real hardware (servers and storage devices) are racked, networked, and accessed from both the corporate network and the internet. It also involves the provisioning of shared storage to be used across multiple servers and the methods available for that, as well as infrastructure and methods for load balancing.

Virtualization Basics

• Hypervisors.

• Virtual machines.

• Object storage.

• Running virtual machines on PC and Mac (Vagrant, VMWare, etc.).

B. Cloud Infrastructure

Cloud infrastructure has to do with core cloud computing and storage components as they are implemented in one of the popular virtualization technologies (VMWare or OpenStack). It also involves the idea of an elastic infrastructure and the options available to implement it.

Networking Basics

• Network layers

• Routers, domain controllers, etc.

• Networks and subnets

• IP address

• VPN

• DNS

• Firewall

• IP tables

• Network access between applications (ACL)

• Networking in the cloud (i.e., Amazon AWS)

Load Balancing

• Load balancing infrastructure and methods

• Geographical load balancing

• Understanding of CDN

• Load balancing in the cloud

2. DevOps Toolchain

A DevOps engineer should have experience using specialized tools for implementing various DevOps processes. While Jenkins, Dockers, Kubernetes, Terraform, Ansible, and the like are known to most DevOps guys, other tools might be obscure or not very obvious (such as the importance of knowing one major monitoring tool in and out). Some tools like source code control systems, are shared with development teams.

The list here has only examples of basic tools. An experienced DevOps engineer would have used some application or tool from all or most of these categories.

Source Code Management (SCM) System

• Expert-level knowledge of an SCM system such as Git or Subversion.

• Knowledge of code branching best practices, such as Git-Flow.

• Knowledge of the importance of checking in Ops code to the SCM system.

• Experience using GitHub.

Bug Management System

• Experience using a major bug management system such as Bugzilla or Jira.

• Ability to have a workflow related to the bug filing and resolution process.

• Experience integrating SCM systems with the bug resolution process and using triggers or REST APIs.

Collaborative Documentation System

• Knowledge of Wiki basics.

• Experience using MediaWiki, Confluence, etc.

• Knowledge of why DevOps projects have to be documented.

• Knowledge of how documents were organized on a Wiki-based system.

Build and CI

• Experience building on Jenkins standalone, or dockerized.

• Experience using Jenkins as a Continuous Integration (CI) platform.

• CI/CD pipeline scripting using groovy

• Experience with CI platform features such as:

  • Integration with SCM systems.

  • Secret management and SSH-based access management.

  • Scheduling and chaining of build jobs.

  • Source-code change-based triggers.

  • Worker and slave nodes.

  • REST API support and Notification management.

Artefacts Management

• Should know what artefacts are and why they have to be managed.

• Experience using a standard artefact management system such as Artifactory.

• Experience caching third-party tools and dependencies in-house.

Configuration Management

• Should be able to explain configuration management.

• Experience using any Configuration Management Database (CMDB) system.

• Experience using open-source tools such as Cobbler for inventory management.

• Ability to do both agentlesschange-based and agent-driven enforcement of configuration.

• Experience using Ansible, Puppet, Chef, Cobbler, etc.

Orchestration and Deployment

• Knowledge of the workflow of released code getting into production.

• Ability to push code to production with the use of SSH-based tools such as Ansible.

• Ability to perform on-demand or Continuous Delivery (CD) of code from Jenkins.

• Ability to perform agent-driven code pull to update the production environment.

• Knowledge of deployment strategies, with or without an impact on the software service.

• Knowledge of code deployment in the cloud (using auto-scaling groups, machine images, etc.).

Monitoring

• Knowledge of all monitoring categories: system, platform, application, business, last-mile, log management, and meta-monitoring.

• Status-based monitoring with Nagios.

• Data-driven monitoring with Zabbix.

• Experience with last-mile monitoring, as done by Pingdom or Catchpoint.

• Experience doing log management with ELK.

• Experience monitoring SaaS solutions (i.e., Datadog and Loggly).

3. System Tools and Methods

To get an automation project up and running, a DevOps engineer builds new things such as configuration objects in an application and code snippets of full-blown programs. However, a major part of the work is glueing many things together at the system level on the given infrastructure. Such efforts are not different from traditional system integration work and, in my opinion, the ingenuity of an engineer at this level determines his or her real value on the team. It is easy to find cookbooks, recipes, and best practices for vendor-supported tools, but it would take experience working on diverse projects to gain the necessary skill set to implement robust integrations that have to work reliably in production.

Important system-level tools and techniques are listed here. The engineer should have knowledge about the following.

Access Management

• Users and groups on Linux.

• Use of service accounts for automation.

• Sudo commands, /etc/sudoers files, and passwordless access.

• Using LDAP and AD for access management.

• Remote access using SSH.

  • SSH keys and related topics.

  • SCP, SFTP, and related tools.

  • SSH key formats.

• Managing access using configuration management tools.

Password Management

• Use of GPG for password encryption.

• Tools for password management such as KeePass.

• MD5, KMS for encryption/decryption.

• Remote access with authentication from automation scripts.

• Managing API keys.

• Jenkins plugins for password management.

Build

• Basics of compilers such as node.js and Javac.

• Make and Makefile, npm, Maven, Gradle, etc.

• Code libraries in Node, Java, Python, React etc.

• Build artefacts such as JAR, WAR and node modules.

• Running builds from Jenkins.

Packaging

• Packaging files: ZIP, TAR, GZIP, etc.

• Packaging for deployment: RPM, Debian, DNF, Zypper, etc.

• Packaging for the cloud: AWS AMI, VMWare template, etc.

• Use of Packer.

• Docker and containers for microservices.

4. Artefacts Management

• Use of artefacts repository: Distribution and release of builds; meeting build and deployment dependencies

• Serving artefacts from a shared storage volume

• Mounting locations from cloud storage services such as AWS S3

• Artifactory as artefacts server

File Transfer

• SCP, Rsync, FTP, and SSL counterparts

• Via shared storage

• File transfer with cloud storage services such as AWS S3

Deployment

• Code pushing using system-level file transfer tools.

• Scripting using SSH libraries such as Paramiko.

• Orchestrating code pushes using configuration management tools.

Job Management

• Use of crontab.

• Running jobs in the background; use of Nohup.

• Use of screen to launch long-running jobs.

• Jenkins as a process manager.

Files and Storage

• Typical uses of the find, DF, DU, etc.

Linux Distributions

• A comparison of popular distributions.

• Checking OS release and system info.

• Package management differences.

• OS Internals and Commands

Text Processing

• Typical uses of SED, AWK, GREP, TR, etc.

• Scripting using Perl, Python.

• Regular expressions.

• Support for regular expressions in Perl and Python.

Troubleshooting Toolkit

Sample usages and steps to install these tools:

• NC

• Netstat

• Traceroute

• VMStat

• LSOF

• Top

• NSLookup

• Ping

• TCPDump

• Dig

• Sar

• Uptime

• IFConfig

• Route

Programming Primer for DevOps

One of the attributes that helps differentiate a DevOps engineer from other members in the operations team, like sysadmins, DBAs, and operations support staff, is his or her ability to write code. The coding and scripting skill is just one of the tools in the DevOps toolbox, but it's a powerful one that a DevOps engineer would maintain as part of practising his or her trade.

Coding is the last resort when things cannot be integrated by configuring and tweaking the applications and tools that are used in an automation project.

Scripting

Bash Scripting Essentials

Many times, a few lines of bash script could be the best glue code integrating two components in the whole software system. DevOps engineers should have basic shell scripting skills and Bash is the most popular right now.

Python

If a script has to deal with external systems and components or it's more than just a few lines of command lines and dealing with fairly complex logic, it might be better to write that script in an advanced scripting language like Python, Perl, or Ruby.

Knowledge of Python would make your life easier when dealing with DevOps applications such as Ansible, which uses Python syntax to define data structures and implement conditionals for defining configurations.

Web Programming

One of the categories of projects a DevOps engineer would end up doing is building dashboards. Though dashboarding features are found with most of the DevOps tools, those are specific to the application, and there will be a time when you may require to have a general-purpose dashboard with more dynamic content than just static links and text.

Another requirement is to build web UI for provisioning tools to present those as self-service tools to user groups.

In both these cases, deep web programming skills are not required. Knowledge of a web programming-friendly language such as PHP and a JavaScript/CSS/HTML library like Composer would be enough to get things started. It is also important for the DevOps engineer to know the full stack, in this case, LAMP, for building and running web apps.

Configuration Languages

Almost every application and tool that is used for building, deploying, and maintaining software systems use configuration files. While manual reading of these files might not require any expertise, a DevOps engineer should know how config files in such formats are created and parsed programmatically.

A DevOps engineer should have a good understanding of these formats:

• INI.

• XML.

• JSON.

• YAML.

The engineer should also know how these formats are parsed in his/her favourite scripting language.

REST API

The wide acceptance of REST API as a standard to expose features that other applications can use for system integration made it a feature requirement for any application that wants to be taken seriously. The knowledge of using REST API has become an important skill for DevOps engineers.

• HTTP/HTTPS: REST APIs are based on HTTP/HTTPS protocol and a solid understanding of its working is required. Knowledge of HTTP headers, status codes, and main verbs GET, POST, and PUT.

• REST API basics: Normal layout of APIs defined for an application.

• Curl and Wget: Command-line tools to access REST API and HTTP URLs. Some knowledge of the support available for HTTP protocol in scripting languages will be useful and that would be an indication of working with REST APIs.

• Authentication methods: Cookie-based and OAuth authentication; API keys; use of If-Match and If-None-Match set of HTTP headers for updates.

• API management tools: If the application you support provides an API for the users, most probably, its usage will be managed by some API Gateway tool. Though not an essential skill, experience in this area would be good if one works on the API provider side.

Programming With Data Repositories

There was a time when mere knowledge of programming with RDBMS was enough for an application developer and system integrator to manage application data. Now with the wide adoption of Big Data platforms like Hadoop and NoSQL systems to process and store data, a DevOps engineer needs varied requirements, from one project to another. Core skills are the following:

• RDBMS: MySQL, Postgres, etc. knowledge of one or more is important.

• Setting up and configuring PostGres: As an open-source database used with many other tools in the DevOps toolchain, consider this as a basic requirement for a DevOps engineer. If one hasn’t done this, he or she might not have done enough yet.

• Running queries from a Bash script: How to run a database query via a database client from a Bash script and use the output. MySQL is a good example.

• Database access from Perl/PHP/Python: All the major scripting languages provide modules to access databases and that can be used to write robust automation scripts. Examples are Perl DBI and Python’s MySQLdb module.

• DB Backups: Migration, Logging, monitoring and cleanup.

Programming for Cloud

Those who have built cloud infrastructure with a focus on automation and versioning should know some of these (or similar) tools:

• cloud-init: Cloud-init can be used to configure a virtual machine when it is spun up. This is very useful when a node is spun up from a machine image with baseline or even application software already baked in.

• AWS/Azure/GCloud CLI: If the application runs on a commercial cloud, knowledge of CLI is needed, which would be handy to put together simple automation scripts.

• Terraform: HashiCorp’s Terraform is an important tool if the focus would be to provision infrastructure as code (IaaS). Using this, infrastructure can be configured independently of the target cloud or virtualization platform.

• Ansible: It can be used to build machine images for a variety of virtualization technologies and cloud platforms, it is useful if the infrastructure is provisioned in a mixed or hybrid cloud environment.

Error Handling

In a rush to get things rolled out, one of the things left half-done is adding enough error handling in scripts. Automation scripts that are not robust can cause major production issues, which could impact the credibility of DevOps efforts itself. A DevOps engineer should be aware of the following best practices in error handling and logging:

• The importance of error handling in automated scripts.

• Error handling in Bash.

• Error handling in Python.

• Logging errors in application and system logs.

This section contains a list of documents elaborating on the key concepts aiding the deployment of the DIGIT platform.

Introduction

This document aims to put together all the items which will enable us to come up with a proper training plan for a partner team that will be working on the DIGIT platform.

Technical Prerequisites

Below listed are the technical skill sets that are required to work on the DIGIT stack. It is expected the team planning on attending training is well versed with the mentioned technologies before they attend eGov training sessions.

Development Team Skillset

• Open API Contract - Swagger2.0

• YAML/JSON

• Postman

• Postgres

• Java and REST APIS

• Basics of Elasticsearch

• Maven

• Springboot

• Kafka

• Zuul

• NodeJS, ReactJS

• WordPress

• PHP

DevOps Team Skillset

• Understanding of the microservice architecture.

• Experience with AWS, Azure, GCP, and NIC Cloud.

• Strong working knowledge of Linux, command, VM Instances, networking and storage.

• To create Kubernetes cluster on AWS, Azure, and GCP on NIC Cloud.

• Kubectl installation & commands (apply, get, edit, describe k8s objects)

• Terraform for infra-as-code for cluster or VM provisioning.

• Understanding of VM types, Linux OS types, LoadBalancer, VPC, Subnets, Security Groups, Firewall, Routing, DNS)

• Experience setting up CI like Jenkins and creating pipelines.

• Deployment strategies - Rolling updates, Canary, Blue/Green.

• Scripting - Shell, Groovy, Python and GoLang.

• Experience in Baking Containers and Dockers.

• Artifactory - Nexus, Verdaccio, DockerHub, etc.

• Experience with Kubernetes ingress, setting up SSL certificates and renewal

• Understanding of Zuul gateway

• Gitops, Git branching, PR review process. Rules, Hooks, etc.

• Experience in Helm, packaging and deploying.

• JBoss Wildfly, Apache, Nginx, Redis and Postgres.

Hardware Pre-requisites

Trainees are expected to have laptops/ desktops configured as mentioned below with all the software required to run the DIGIT application

• Laptop for hands-on training with 16GB RAM and OS preferably Ubuntu

• All developers need to have Git ids

• Install VSCode/IntelliJ/Eclipse

• Install Git

• Install JDK 8 update 112 or higher

• Install maven v3.2.x

• Install PostgreSQL v9.6

• Install Elastic Search v2.4.x

• Postman

Software Assets

There are knowledge assets available on the Net for general items and eGov assets for DIGIT services. Here you can find references to each of the topics of importance. It is mandated the trainees do a self-study of all the software mentioned in the prerequisites using the reference materials shared.

Introduction

This document aims to put together all the items which will enable us to come up with a proper training plan for a partner team that will be working on the eDCR service used for the plan scrutiny.

Technical Pre-requisites

Below listed are the technical skillsets that are required to work on eDCR service. It is expected that the team planning on attending training is well versed with the mentioned technologies before they attend eGov training sessions.

Development Team Skillset

• Java and REST APIS

• Postgres

• Maven

• Spring framework

• Basics of 2D CAD Drawings

• Git

• Postman

• YAML/JSON

DevOps Team Skillset

• Strong working knowledge of Linux, command, VM Instances, networking, storage

• The session, cache, and tokens handling (Redis-server)

• Understanding of VM types, Linux OS types, LoadBalancer, VPC, Subnets, Security Groups, Firewall, Routing, DNS

• Experience setting up CI like Jenkins and creating pipelines

• Artifactory - Nexus, verdaccio, DockerHub, etc

• Experience in setting up SSL certificates and renewal

• Gitops, Git branching, PR review process. Rules, Hooks, etc.

• JBoss Wildfly, Apache, Nginx, Redis and Postgres

Hardware prerequisites

Trainees are expected to have laptops/ desktops configured as mentioned below with all the software required to run the eDCR service application

• Laptop for hands-on training with 16GB RAM and OS preferably Ubuntu

• All developers need to have Git ids

• Install VSCode/IntelliJ/Eclipse

• Install Git

• Install JDK 8 update 112 or higher

• Install maven v3.2.x

• Install PostgreSQL v9.6

• Postman

• Install LibreCAD

• Application Server JBoss Wildfly v11.x

Software Assets

There are knowledge assets available on the Net for general items and eGov assets for DIGIT services. Here you can find references to each of the topics of importance. It is mandated the trainees do a self-study of all the software mentioned in the prerequisites using the reference materials shared.

​

Containerising applications and running them on Kubernetes doesn’t mean we can forget all about resource utilization. Our thought process may have changed because we can much more easily scale-out our application as demand increases, but many times we need to consider how our containers might fight with each other for resources. Resource Requests and Limits can be used to help stop the “noisy neighbour” problem in a Kubernetes Cluster.

Resource Requests

To put things simply, a resource request specifies the minimum amount of resources a container needs to successfully run. Thought of in another way, this is a guarantee from Kubernetes that you’ll always have this amount of either CPU or Memory allocated to the container.

Why would you worry about the minimum amount of resources guaranteed to a pod? Well, it's to help prevent one container from using up all the node’s resources and starving the other containers from CPU or memory. For instance, if I had two containers on a node, one container could request 100% of that node's processor. Meanwhile, the other container would likely not be working very well because the processor is being monopolized by its “noisy neighbour”.

What a resource request can do, is ensure that at least a small part of that processor’s time is reserved for both containers. This way if there is resource contention, each pod will have a guaranteed, minimum amount of resources in which to still function.

Resource Limits

As you might guess, a resource limit is the maximum amount of CPU or memory that can be used by a container. The limit represents the upper bounds of how much CPU or memory that a container within a pod can consume in a Kubernetes cluster, regardless of whether or not the cluster is under resource contention.

Limits prevent containers from taking up more resources on the cluster than you’re willing to let them.

Common Practices

As a general rule, all containers should have a request for memory and CPU before deploying to a cluster. This will ensure that if resources are running low, your container can still do the minimum amount of work to stay in a healthy state until those resources free up again (hopefully).

Limits are often used in conjunction with requests to create a “guaranteed pod”. This is where the request and limit are set to the same value. In that situation, the container will always have the same amount of CPU available to it, no more or less.

At this point, you may be thinking about adding a high “request” value to make sure you have plenty of resources available for your container. This might sound like a good idea, but have dramatic consequences for scheduling on the Kubernetes cluster. If you set a high CPU request, for example, 2 CPUs, then your pod will ONLY be able to be scheduled on Kubernetes nodes that have 2 full CPUs available that aren’t reserved by other pods’ requests. In the example below, the 2 vCPU pods couldn’t be scheduled on the cluster. However, if you were to lower the “request” amount to say 1 vCPU, it could.

Resource Requests and Limits – In Action

CPU Limit Example

Let us try out using a CPU limit on a pod and see what happens when we try to request more CPU than we’re allowed to have. Before we set the limit though, let us look at a pod with a single container under normal conditions. I’ve deployed a resource consumer container in my cluster and by default, you can see that I am using 1m CPU(cores) and 6 Mi(bytes) of memory.

NOTE: CPU is measured in millicores so 1000m = 1 CPU core. Memory is measured in Megabytes.

Ok, now that we have seen the “no-load” state, let us add some CPU load by making a request to the pod. Here, I’ve increased the CPU usage on the container to 400 millicores.

After the metrics start coming in, you can see that I’ve got roughly 400m used on the container as you’d expect to see.

Now I’ve deleted the container and we’ll edit the deployment manifest so that it has a limit on CPU.

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    run: resource-consumer
  name: resource-consumer
  namespace: default
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      run: resource-consumer
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        run: resource-consumer
    spec:
      containers:
      - image: theithollow/resource-consumer:v1
        imagePullPolicy: IfNotPresent
        name: resource-consumer
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        resources:
          requests:
            memory: "100Mi"
            cpu: "100m"
          limits:
            memory: "300Mi"
            cpu: "300m"
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30

After redeploying the container and again increasing my CPU load to 400m, we can see that the container is throttled to 300m instead. I’ve effectively “limited” the resources the container could consume from the cluster.

CPU Requests Example

OK, next, I’ve deployed two pods into my Kubernetes cluster and those pods are on the same worker node for a simple example of contention. I’ve got a guaranteed pod that has 1000m CPU set as a limit but also as a request. The other pod is unbounded, meaning there is no limit on how much CPU it can utilize.

After the deployment, each pod is really not using any resources as you can see here.

We make a request to increase the load on my non-guaranteed pod.

And if we look at the container's resources you can see that even though my container wants to use a 2000m CPU, it’s only actually using a 1000m CPU. The reason for this is that the guaranteed pod is guaranteed a 1000m CPU, whether it is actively using that CPU or not.

Summary

Kubernetes uses Resource Requests to set a minimum amount of resources for a given container so that it can be used if it needs it. You can also set a Resource Limit to set the maximum amount of resources a pod can utilize.

Taking these two concepts and using them together can ensure that your critical pods always have the resources that they need to stay healthy. They can also be configured to take advantage of shared resources within the cluster.

Be careful setting resource requests too high so your Kubernetes scheduler can still scheduler these pods. Good luck!

Things To Know When Deploying Kubernetes On SDC

Running Kubernetes on-premise gives a cloud-native experience or SDC becomes cloud-agnostic when it comes to the experience of Deploying DIGIT.

Whether States have their own on-premise data centre or have decided to forego the various managed cloud solutions, there are a few things one should know when getting started with on-premise K8s.

One should be familiar with Kubernetes and one should know that the consists of the Kube-apiserver, Kube-scheduler, Kube-controller-manager and an ETCD datastore. For managed cloud solutions like or , it also includes the cloud-controller-manager. This is the component that connects the cluster to external cloud services to provide networking, storage, authentication, and other feature support.

To successfully deploy a bespoke Kubernetes cluster and achieve a cloud-like experience on SDC, one needs to replicate all the same features you get with a managed solution. At a high level this means that we probably want to:

• Automate the deployment process

• Choose a networking solution

• Choose a storage solution

• Handle security and authentication

Let us look at each of these challenges individually, and we’ll try to provide enough of an overview to aid you in getting started.

Automating The Deployment Process

Using a tool like Ansible can make deploying Kubernetes clusters on-premise trivial.

When deciding to manage your own Kubernetes clusters, we need to set up a few proofs-of-concept (PoC) clusters to learn how everything works, perform performance and conformance tests, and try out different configuration options.

After this phase, automating the deployment process is an important if not necessary step to ensure consistency across any clusters you build. For this, you have a few options, but the most popular are:

• : a low-level tool that helps you bootstrap a minimum viable Kubernetes cluster that conforms to best practices

• : an Ansible playbook that helps deploy production-ready clusters

If you already using Ansible, Kubespray is a great option otherwise we recommend writing automation around Kubeadm using your preferred playbook tool after using it a few times. This will also increase your confidence and knowledge of the tooling surrounding Kubernetes.

Choosing A Network Solution

When designing clusters, choosing the right container networking interface (CNI) plugin can be the hardest part. This is because choosing a CNI that will work well with an existing network topology can be tough. Do you need BGP peering capabilities? Do you want an overlay network using vxlan? How close to bare-metal performance are you trying to get?

Choosing A Storage Solution

For a cloud-like experience, you’ll need to add a plugin to dynamically create persistent volume objects that match the user’s persistent volume claims. You can use dynamic provisioning to reclaim these volume objects after a resource has been deleted.

Handle Security And Authentication

As anyone familiar with security knows, this is a rabbit hole. You can always make your infrastructure more secure and should be investing in continual improvements.

Including different Kubernetes plugins can help build a secure, cloud-like experience for your users

When designing on-premise clusters you’ll have to decide where to draw the line. To really harden your cluster’s security you can add plugins like:

Other Considerations

Hope this has given you a good idea of deploying, networking, storage, and security for you to take the leap into deploying your own on-premise Kubernetes clusters. As we mentioned above, the team will want to build proof-of-concept clusters, run conformance and performance tests, and really become experts on Kubernetes if you’re going to be using it to run DIGIT on production.

We’ll leave you with a few other things the team should be thinking of:

• Externally backing up Kubernetes YAML, namespaces, and configuration files

• Running applications across clusters in an active-active configuration to allow for zero-downtime updates

• Running game days like deleting the CNI to measure and improve time-to-recovery

Prepare Azure Environment

For provisioning Kubernetes clusters with the Kubermatic needs a service account with (at least) the Azure role Contributor. Follow the steps below to create a matching service account.

Login to Azure and Get Basic Information

Login to Azure with az.

This command opens in your default browser window where you can authenticate. Get your subscription ID once you are logged in successfully.

Get your Tenant ID -

Create a new app using -

• Client ID: Take the value of appId

• Client Secret: Take the value of password

• Tenant ID: your tenant ID

• Subscription ID: your subscription ID

Overview

Since there are many DIGIT services and the development code is part of various git repos, you need to understand the concept of cicd-as-service which is open-sourced. This page also guides you through the process of creating a CI/CD pipeline.

As a developer - To integrate any new service/app to the CI/CD below is the starting point:

Once the desired service is ready for the integration: decide the service name, type of service, whether DB migration is required or not. While you commit the source code of the service to the git repository, the following file should be added with the relevant details which are mentioned as below:

Build-config.yml –It is present under build directory in each repository

This file contains the below details which are used for creating the automated Jenkins pipeline job for your newly created service.

While integrating a new service/app, the above content needs to be added in the build-config.yml file of that app repository. For example: If we are on-boarding a new service called egov-test, then the build-config.yml should be added as mentioned below.

If a job requires multiple images to be created (DB Migration) then it should be added as below,

Note - If a new repository is created then the build-config.yml should be created under the build folder and then the config values are added to it.

The git repository URL is then added to the Job Builder parameters

When the Jenkins Job => job builder is executed the CI Pipeline gets created automatically based on the above details in build-config.yml. Eg: egov-test job will be created under core-services folder in Jenkins because the “build-config was edited under core-services” And it should be the “master” branch only. Once the pipeline job is created, it can be executed for any feature branch with build parameters (Specifying which branch to be built – master or any feature branch).

As a result of the pipeline execution, the respective app/service docker image will be built and pushed to the Docker repository.

Continuous Integration (CI)

The Jenkins CI pipeline is configured and managed 'as code'.

If git repository URL is available build the Job-Builder Job

If the git repository URL is not available ask the DevOps team to add it.

Continuous Deployment (CD)

The services are deployed and managed on a Kubernetes cluster in cloud platforms like AWS, Azure, GCP, OpenStack, etc. Here, we use helm charts to manage and generate the Kubernetes manifest files and use them for further deployment to the respective Kubernetes cluster. Each service is created as charts which will have the below-mentioned files in it.

To deploy a new service, we need to create the helm chart for it. The chart should be created under the charts/helm directory in DIGIT-DevOps repository.

We have an automatic helm chart generator utility which needs to be installed on the local machine, the utility will prompt for user inputs about the newly developed service( app specifications) for creating the helm chart. The requested chart with the configuration values (created based on the inputs provided) will be created for the user.

‌ _Name of the service? test-service Application Type? NA Kubernetes health checks to be enabled? Yes Flyway DB migration container necessary? No Expose service to the internet? Yes Route through API gateway [zuul] No Context path? hello_‌

The generated chart will have the following files.

This chart can also be modified further based on user requirements.

The Deployment of manifests to the Kubernetes cluster is made very simple and easy. We have Jenkins Jobs for each state and environment-specific. We need to provide the image name or the service name in the respective Jenkins deployment job.

Enter a caption for this image (optional)

Enter a caption for this image (optional)

‌The deployment Jenkins job internally performs the following operations,‌

• Reads the image name or the service name given and finds the chart that is specific to it.

• Generates the Kubernetes manifests files from the chart using helm template engine.

• Execute the deployment manifest with the specified docker image(s) to the Kubernetes cluster.

Introduction

Security is always a difficult subject to approach either by the lack of experience; either by the fact you should know when the level of security is right for what you have to secure.

Security is a major concern when it comes to government systems and infra. As an architect, we can consider that working with technically educated people (engineers, experts) and tools (systems, frameworks, IDE) should prevent key VAPT issues.

However, it’s quite difficult to avoid, a certain infatuation from different categories of people to try to hack the systems.

Infra Security

1. Update to the latest version

There aren’t only bug fixes in each release but also new security measures to require advantage of them, we recommend working with the newest stable version.

Updates and support could also be harder than the new features offered in releases, so plan your updates a minimum of once a quarter. Significantly simplify updates can utilize the providers of managed Kubernetes solutions.

2. Enable role-based access control (RBAC)

Use RBAC (Role-Based Access Control) to regulate who can access and what rights they need. Usually, RBAC is enabled by default in version 1.6 and later (or later for a few providers), but if you’ve got been updated since then and didn’t change the configuration, you ought to double-check your settings.

However, enabling RBAC isn’t enough — it still must be used effectively. within the general case, the rights to the whole cluster (cluster-wide) should be avoided, giving preference to rights in certain namespaces. Avoid giving someone cluster administrator privileges even for debugging — it’s much safer to grant rights only necessary and from time to time.

If the appliance requires access to the Kubernetes API, create separate service accounts. and provides them with the minimum set of rights required for every use case. This approach is far better than giving an excessive amount of privilege to the default account within the namespace.

3. Use namespaces to set security boundaries

Creating separate namespaces is vital because of the first level of component isolation. it’s much easier to regulate security settings — for instance, network policies — when different types of workloads are deployed in separate namespaces.

4. Separate sensitive workloads

A good practice to limit the potential consequences of compromise is to run workloads with sensitive data on a fanatical set of machines. This approach reduces the risk of a less secure application accessing the application with sensitive data running in the same container executable environment or on the same host.

For example, a kubelet of a compromised node usually has access to the contents of secrets only if they are mounted on pods that are scheduled to be executed on the same node. If important secrets are often found on multiple cluster nodes, the attacker will have more opportunities to urge them.

Separation can be done using node pools (in the cloud or on-premises), as well as Kubernetes controlling mechanisms, such as namespaces, taints, tolerations, and others.

5. Protect access to cloud service metadata

Sensitive metadata — for instance, kubelet administrative credentials, are often stolen or used with malicious intent to escalate privileges during a cluster. For example, a recent find within Shopify’s bug bounty showed in detail how a user could exceed authority by receiving metadata from a cloud provider using specially generated data for one of the microservices.

The GKE metadata concealment function changes the mechanism for deploying the cluster in such how that avoids such a drag. And we recommend using it until a permanent solution is implemented.

6. Create and define cluster network policies

Network Policies — allow you to control access to the network in and out of containerized applications. To use them, you must have a network provider with support for such a resource. For managed Kubernetes solution providers such as Google Kubernetes Engine (GKE), support will need to be enabled.

Once everything is ready, start with simple default network policies — for example, blocking (by default) traffic from other namespaces.

7. Set the Pod Security Policy for the cluster

Pod Security Policy sets the default values ​​used to start workloads in the cluster. Consider defining a policy and enabling the Pod Security Policy admission controller: the instructions for these steps vary depending on the cloud provider or deployment model used.

In the beginning, you might want to disable the NET_RAW capability in containers to protect yourself from certain types of spoofing attacks.

8. Work on node security

To improve host security, you can follow these steps:

• Ensure that the host is securely and correctly configured. One way is CIS Benchmarks; Many products have an auto checker that automatically checks the system for compliance with these standards.

• Monitor the network availability of important ports. Ensure that the network is blocking access to the ports used by kubelet, including 10250 and 10255. Consider restricting access to the Kubernetes API server — with the exception of trusted networks. In clusters that did not require authentication and authorization in the kubelet API, attackers used to access such ports to launch cryptocurrency miners.

• Minimize administrative access to Kubernetes hosts Access to cluster nodes should in principle be limited: for debugging and solving other problems, as a rule, you can do without direct access to the node.

9. Enable Audit Logging

Make sure that audit logs are enabled and that you are monitoring for the occurrence of unusual or unwanted API calls in them, especially in the context of any authorization failures — such entries will have a message with the “Forbidden” status. Authorization failures can mean that an attacker is trying to take advantage of the credentials obtained.

Managed solution providers (including GKE) provide access to this data in their interfaces and can help you set up notifications in case of authorization failures.

Conclusion

Overview

This page explains why Kubernetes is required. It deep dives into the key benefits of using Kubernetes to run a large containerized platform like DIGIT in production environments.

The Big Why

Kubernetes project started in the year 2014 with . Kubernetes has now become the de facto standard for deploying containerized applications at scale in private, public and hybrid cloud environments. The largest public cloud platforms , , , and now provide managed services for Kubernetes. A few years back RedHat, Mesosphere, Pivotal, VMware, and Nutanix completely redesigned their implementation with Kubernetes and collaborated with the Kubernetes community to implement the next-generation container platform with incorporated key features of Kubernetes such as container grouping, overlay networking, layer 4 routing, secrets, etc. Today many organizations & technology providers adopting Kubernetes at a rapid phase.

Kubernetes Architecture

One of the fundamental design decisions which have been taken by this impeccable cluster manager is its ability to deploy existing applications that run on VMs without any changes to the application code. On a high level, any application that runs on VMs can be deployed on Kubernetes by simply containerizing its components. This is achieved by its core features; container grouping, container orchestration, overlay networking, container-to-container routing with layer 4 virtual IP-based routing system, service discovery, support for running daemons, deploying stateful application components, and most importantly the ability to extend the container orchestrator for supporting complex orchestration requirements.

On a very high-level Kubernetes provides a set of dynamically scalable hosts for running workloads using containers and uses a set of management hosts called masters for providing an API for managing the entire container infrastructure.

That's just a glimpse of what Kubernetes provides out of the box. The following few sections will go through its core features and explain how it can help applications be deployed on it in no time.

Application Deployment Model

A containerized application can be deployed on Kubernetes using a deployment definition by executing a simple CLI command as follows:

Service Discovery & Load Balancing

One of the key features of Kubernetes is its service discovery and internal routing model provided using SkyDNS and layer 4 virtual IP-based routing system. These features provide internal routing for application requests using services. A set of pods created via a replica set can be load balanced using a service within the cluster network. The services get connected to pods using selector labels. Each service will get assigned a unique IP address, a hostname derived from its name and route requests among the pods in a round-robin manner. The services will even provide an IP-hash-based routing mechanism for applications which may require session affinity. A service can define a collection of ports and the properties defined for the given service will apply to all the ports in the same way. Therefore, in a scenario where session affinity is only needed for a given port and where all the other ports are required to use round-robin-based routing, multiple services may need to be used.

How Services Internally Work

Kubernetes services have been implemented using a component called kube-proxy. A kube-proxy instance runs in each node and provides three proxy modes: Userspace, iptables and IPVS. The current default is iptables.

In the first proxy mode: userspace, kube-proxy itself will act as a proxy server and delegate requests accepted by an iptable rule to the backend pods. In this mode, kube-proxy will operate in the userspace and will add an additional hop to the message flow.

In the second proxy mode: iptables, the kube-proxy will create a collection of iptable rules for forwarding incoming requests from the clients directly to the ports of backend pods on the network layer without adding an additional hop in the middle. This proxy mode is much faster than the first mode because of operating in the kernel space and not adding an additional proxy server in the middle.

Internal/External Routing Separation

Kubernetes services can be exposed to external networks in two main ways. The first is using node ports by exposing dynamic ports on the nodes that forward traffic to the service ports. The second is using a load balancer configured via an ingress controller which can delegate requests to the services by connecting to the same overlay network. An ingress controller is a background process which may run in a container which listens to the Kubernetes API, and dynamically configure and reloads a given load balancer according to a given set of ingresses. An ingress defines the routing rules based on hostnames and context paths using services.

Once an application is deployed on Kubernetes using kubectl run command, it can be exposed to the external network via a load balancer as follows:

The above command will create a service of load balancer type and map it to the pods using the same selector label created when the pods were created. As a result, depending on how the Kubernetes cluster has been configured a load balancer service on the underlying infrastructure will get created for routing requests for the given pods either via the service or directly.

Usage Of Persistent Volumes

Applications that require persisting data on the filesystem may use volumes for mounting storage devices to ephemeral containers similar to how volumes are used with VMs. Kubernetes has properly designed this concept by loosely coupling physical storage devices with containers by introducing an intermediate resource called persistent volume claims (PVCs). A PVC defines the disk size, and disk type (ReadWriteOnce, ReadOnlyMany, ReadWriteMany) and dynamically links a storage device to a volume defined against a pod. The binding process can either be done in a static way using PVs or dynamically by using a persistent storage provider. In both approaches, a volume will get linked to a PV one-to-one and depending on the configuration given data will be preserved even if the pods get terminated. According to the disk type used multiple pods will be able to connect to the same disk and read/write.

Deploying Daemons On Nodes

Kubernetes provides a resource called DaemonSets for running a copy of a pod in each Kubernetes node as a daemon. Some of the use cases of DaemonSets are as follows:

• Deploy the cluster storage daemon such as glusterd , ceph on each node to provide persistence storage.

• Run the log collection daemon such as fluentd or logstash on every node for collecting container and Kubernetes component logs.

• Run the ingress controller pod on a collection of nodes for providing external routing.

Deploying Stateful Distributed Systems

One of the most difficult tasks of containerizing applications is the process of designing the deployment architecture of stateful distributed components. Stateless components can be easily containerized as they may not have a predefined startup sequence, clustering requirements, point-to-point TCP connections, unique network identifiers, graceful startup and termination requirements, etc. Systems such as databases, big data analysis systems, distributed key/value stores, and message brokers, may have complex distributed architectures that may require the above features. Kubernetes introduced StatefulSets resource for supporting such complex requirements.

On high-level StatefulSets are similar to ReplicaSets except that it provides the ability to handle the startup sequence of pods, and uniquely identify each pod for preserving its state while providing the following characteristics:

• Stable, unique network identifiers.

• Stable, persistent storage.

• Ordered, graceful deployment and scaling.

• Ordered, graceful deletion and termination.

• Ordered, automated rolling updates

Running Background Jobs

Deploying Databases

Configurations Management

Containers generally use environment variables for parameterizing their runtime configurations. However, typical enterprise applications use a considerable amount of configuration files for providing static configurations required for a given deployment. Kubernetes provides a fabulous way of managing such configuration files using a simple resource called ConfigMaps without bundling them into container images. ConfigMaps can be created using directories, files or literal values using the following CLI command:

Once a ConfigMap is created, it can be mounted to a pod using a volume mount. With this loosely coupled architecture, configurations of an already running system can be updated seamlessly just by updating the relevant ConfigMap and executing a rolling update process which I will explain in one of the next sections. It might be important to note that currently, ConfigMaps does not support nested folders, therefore if there are configuration files available in a nested directory structure of the application, a ConfigMap would need to be created for each directory level.

Credentials Management

Similar to ConfigMaps Kubernetes provides another valuable resource called Secrets for managing sensitive information such as passwords, OAuth tokens, and ssh keys. Otherwise updating that information on an already running system might require rebuilding the container images.

A secret can be created for managing basic auth credentials using the following way:

Once a secret is created, it can be read by a pod either using environment variables or volume mounts. Similarly, any other type of sensitive information can be injected into pods using the same approach.

Rolling Out Updates

The animated image above illustrates how application updates can be rolled out for an already running application using the blue/green deployment method without having to take a system downtime. This is another invaluable feature of Kubernetes which allows applications to seamlessly roll out security updates and backwards compatible changes without much effort. If the changes are not backwards compatible, a manual blue/green deployment might need to be executed using a separate deployment definition.

This approach allows a rollout to be executed for updating a container image using a simple CLI command:

Once a rollout is executed, the status of the rollout process can be checked as follows:

Using the same CLI command kubectl set image deployment an update can be rolled back to a previous state.

Autoscaling

Figure 10: Kubernetes Pod Autoscaling Model

Kubernetes allows pods to be manually scaled either using ReplicaSets or Deployments. The following CLI command can be used for this purpose:

Package Management

Figure 11: Helm and Kubeapps Hub

The Kubernetes community initiated a separate project for implementing a package manager for Kubernetes called Helm. This allows Kubernetes resources such as deployments, services, config maps, ingresses, etc to be templated and packaged using a resource called chart and allows them to be configured at the installation time using input parameters. More importantly, it allows existing charts to be reused when implementing installation packages using dependencies. Helm repositories can be hosted in public and private cloud environments for managing application charts. Helm provides a CLI for installing applications from a given Helm repository into a selected Kubernetes environment.

Conclusion

Kubernetes has been designed with over a decade of experience in running containerized applications at scale at Google. It has been already adopted by the largest public cloud vendors, and technology providers and is currently being embraced by most of the software vendors and enterprises as this article is written. It has even led to the inception of the Cloud Native Computing Foundation (CNCF) in the year 2015, was the first project to graduate under CNCF, and started streamlining the container ecosystem together with other container-related projects such as CNI, Containers, Envoy, Fluentd, gRPC, Jagger, Linkerd, Prometheus, RKT and Vitess. The key reasons for its popularity and to be endorsed at such a level might be its flawless design, collaborations with industry leaders, making it open-source, and always being open to ideas and contributions.

References

Overview

This page contains steps that are involved in building and deploying the application. FAQ related to various deployment and development issues are discussed here

Setup Using Auto-installer

• Clone the eGov repository (development is done on the develop branch).

1$ mkdir -p ${HOME}/egovgithub && cd egovgithub 2$ git clone -b develop --single-branch <https://github.com/egovernments/egov-smartcity-suite.git>

• First time setup which will install the stacks, build the source code, and deploys the artefact to Wildfly

1$ cd ${HOME}/egovgithub/egov-smartcity-suite && make all

• To install the prerequisites Phoenix stacks

1$ cd ${HOME}/egovgithub/egov-smartcity-suite && make install

• To build the source code base

1$ cd ${HOME}/egovgithub/egov-smartcity-suite && make build

• To deploy the artefact to WILDFLY

1$ cd ${HOME}/egovgithub/egov-smartcity-suite && make deploy

Manual Setup

Pre-requisites

• Install maven v3.2.x

• Install PostgreSQL v9.4

• Install Elastic Search v2.4.x

• Install Jboss Wildfly v10.x

• Install Git 2.8.3

• Install JDK 8 update 112 or higher

Database Setup

1. Create a database and user in postgres

2. Create a schema called generic

3. Execute ALTER ROLE <your_login_role> SET search_path TO generic,public;

Elastic Search Setup

Elastic search server properties need to be configured in elasticsearch.yml under <ELASTICSEARCH_INSTALL_DIR>/config1## Your local elasticsearch clustername, DO NOT use default clustername 2cluster.name: elasticsearch-<username> 3## This is the default port 4transport.tcp.port: 9300 5

NB: <username> user name of the logged-in system, enter the below command in terminal to find the username.1$ id -un

Building Source

1. Clone the eGov repository (development is done on the develop branch.

1$ mkdir egovgithub 2$ cd egovgithub 3$ git clone <https://github.com/egovernments/egov-smartcity-suite.git> 4$ git checkout develop

1. Change directory to <CLONED_REPO_DIR>/egov/egov-config/src/main/resources/config/ and create a file called egov-erp-<username>.properties and enter the following values based on your environment config.

1##comma separated list of host names 2elasticsearch.hosts=localhost 3elasticsearch.port=9300 4elasticsearch.cluster.name=elasticsearch-<username> 5

If required, you can override any default settings available in /egov/egov-egi/src/main/resources/config/application-config.properties by overriding the value in egov-erp-<username>.properties.

1. Change directory back to <CLONED_REPO_DIR>/egov

2. Run the following commands. This will clean, compile, test, migrate the database and generate ear artefact along with jars and wars appropriately.

1mvn clean package -s settings.xml -Ddb.user=<db_username> -Ddb.password=<db_password> -Ddb.driver=org.postgresql.Driver -Ddb.url=<jdbc_url>

Redis Server Setup