This guide provides a step-by-step guide to monitoring and operating the DIGIT Platform and services in production.

Overview

Role-based access control (RBAC) regulates access to a computer or network resources based on the roles of individual users within your organization.

RBAC authorization uses the rbac.authorization.k8s.io API group to drive authorization decisions, allowing you to configure policies through the Kubernetes API dynamically.

API objects

The RBAC API declares four Kubernetes objects: Role, ClusterRole, RoleBinding and ClusterRoleBinding. You can describe objects, or amend them, using tools such as kubectl, just like any other Kubernetes object.

Role & ClusterRole

An RBAC Role or ClusterRole contains rules that represent a set of permissions. Permissions are purely additive (there are no "deny" rules).

A Role always sets permissions within a particular namespace; when you create a Role, you have to specify the namespace it belongs in.

ClusterRole, by contrast, is a non-namespaced resource. The resources have different names (Role and ClusterRole) because a Kubernetes object always has to be either namespaced or not namespaced; it can't be both.

ClusterRoles have several uses. You can use a ClusterRole to:

1. define permissions on namespaced resources and be granted access within individual namespace(s)

2. define permissions on namespaced resources and be granted access across all namespaces

3. define permissions on cluster-scoped resources

If you want to define a role within a namespace, use a Role; if you want to define a role cluster-wide, use a ClusterRole.

ClusterRole In DIGIT

A ClusterRole can be used to grant the same permissions as a Role. Because ClusterRoles are cluster-scoped, you can also use them to grant access to:

• cluster-scoped resources (like nodes)

• non-resource endpoints (like /healthz)

• namespaced resources (like Pods), across all namespaces

  For example: you can use a ClusterRole to allow a particular user to run kubectl get pods --all-namespaces

Here is the Digit ClusterRole that can be used to grant read access and restricted admin access

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: digit-user
rules:
- apiGroups:
  - "extensions"
  resources:
  - deployments
  verbs:
  - patch
- apiGroups:
  - ""
  resources:
  - pods/portforward
  - pods/proxy 
  verbs:
  - create  
  - delete
---  
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: digit-admin
rules:
- apiGroups:
  - ""
  resources:
  - pods/portforward
  - pods/proxy 
  - pods/exec 
  verbs:
  - create  
  - delete
- apiGroups:
  - "batch"
  resources:
  - jobs 
  verbs:
  - create  
  - update
  - patch
  - list
- apiGroups: 
  - "apps"
  - "extensions"  
  resources: 
  - deployments
  verbs: 
  - patch 
  - get 
  - list
  - update    

RoleBinding & ClusterRoleBinding

A role binding grants the permissions defined in a role to a user or set of users. It holds a list of subjects (users, groups, or service accounts), and a reference to the role being granted. A RoleBinding grants permissions within a specific namespace whereas a ClusterRoleBinding grants that access cluster-wide.

A RoleBinding may reference any Role in the same namespace. Alternatively, a RoleBinding can reference a ClusterRole and bind that ClusterRole to the namespace of the RoleBinding. If you want to bind a ClusterRole to all the namespaces in your cluster, you use a ClusterRoleBinding.

RoleBinding In DIGIT

Here is the Digit rolebinading that we are using to grant access to group

{{- with index .Values "cluster-configs" "rbac" }}
{{- range $idx, $v := . }}                 // These iteration values are defined in the environment file
{{- range $nsidx, $nsval := .namespaces }}  // These iteration values are defined in the environment file 
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: digit-{{ $v.role }}
  namespace: {{ $nsval }}
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: digit-{{ $v.role }}
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: digit-user        
---
{{- end }}
{{- end }}
{{- end }}

A RoleBinding can also reference a ClusterRole to grant the permissions defined in that ClusterRole to resources inside the RoleBinding's namespace. This kind of reference lets you define a set of common roles across your cluster, and then reuse them within multiple namespaces.

For instance, even though the following RoleBinding refers to a ClusterRole, "dave" (the subject, case sensitive) will only be able to read Secrets in the "development" namespace, because the RoleBinding's namespace (in its metadata) is "development".

Define the RBAC config in the Environment file

You must add a namespace to a role section to grant access to a group of a namespace.

cluster-configs:
  rbac:
    - role: user
      namespaces: [egov] // digit-user ClusterRole would be granted to the playground and egov namespace. 
    - role: admin
      namespaces: [playground,egov]  // digit-admin ClusterRole would be granted to the playground and egov namespace. 

Overview

On this page, you will find the steps on how to create a database dump.

Steps

1. To create a database dump, execute the dump command (given below) in the playground pod.

   kubectl get pods -n playground

   kubectl exec <playground-pod-name> -it -n playground bash

2. Use the below command to take a backup.

   pg_dump -Fp --no-acl --no-owner --no-privileges -h <db-host> egov_db -U dbusername > backup.sql

   gzip backup.sql.gz backup.sql

3. Copy the zip file to your local machine using the below command.

   kubectl cp <playground-pod-name>:/backup.sql.gz backup.sql.gz -n playground

What is organization and why we are creating in GitHub:

An organization are shared accounts where businesses and open source projects can collaborate across many projects at once.There are three types accounts in GitHub

1. Personal accounts

2. Organization accounts

3. Enterprise accounts

• Here the main reason for creating organization account is, accounts can be shared among unlimited number of people and they can collaborate across many projects at once.

• Our organization name is eGovernments Foundation.

GitHub account creation:

• Go to

• After completing the process Your GitHub account will be created.

• Click on Sign Up. Create your account by using email and password. Then add Username.

• After completing the process Your GitHub account will be created.

Creating Organization in GitHub:

• After setting up the GitHub account, we have to create an organization. Here we can add the data or code in the form of repository. Creating a repository, we will see this topic next.

• Open Github and click on the "+" icon add top tight corner. You will see the option"new organization". click it.

• click on "create a free organization"and enter your organization name you want to create with email and then '"next"

• After Organization got created, you can see your organizations by clicking on "Accounts

• To create a new repository, click on + icon and New repository

• Create your with repo with any name based on your code. Make it as public. Then anyone can able to see your code.

• If you want to add a README file, click on add a README file. It is helpful to understand how does the code present in repo will be helpful.

• Next click on create repository.

Adding new SSH key to your GitHub account:

• Open Your "Command prompt" or "Terminal".

• Type below commands to generate SSH key

• Now a .ssh folder is created in your home directory. Go to that directory.

• copy the SSH key which we get after running the above commands.

• open GitHub and add this SSH key as shown below:

• open Settings and go to SSH and GPG keys

• Click on New SSH key and paste it. Click on Add SSH key.

• If you want check the private key, use

• GitHub is an open source tool which helps the developers to manage, store, track and control changes in their code. If we want to clone(copy) the data from GItHub we need to install Git.

• There are some alternatives for GitHub like GitLab, Bitbucket. But many developers prefer GitHub because it's more popular and they are used to the navigation. So we are using Git in DIGIT

• GitHub is used to create Individual projects.

Checking for Git:

To check whether Git is already installed in your systems, open in terminal.

• If you are in Mac, look for the command prompt application called "Terminal".

• If you are in Windows, open the windows command prompt or "Git Bash".

• Type the below command:

git version

Installing Git on Linux:

Ubuntu:

In Ubuntu using terminal we can directly install Git using terminal.

• Go to command prompt shell and run the following command to make sure everything is up-to-date.

sudo apt-get update

• After that run the following command to install Git.

sudo apt-get install git-all

• Once the command output has completed, verify the installation using

git version

Installing Git on Windows and Mac:

• Go to the following page to download the Git latest version: For Windows: https://gitforwindows.org/ For Mac: https://sourceforge.net/projects/git-osx-installer/files/git-2.23.0-intel-universal-mavericks.dmg/download?use_mirror=autoselect

• Once the installation is done, open the windows command prompt or Git Bash and type

git version

• You can fetch updates from or submit changes to the original repository with pull requests

• A fork often occurs when a developer becomes dissatisfied or disillusioned with the direction of a project and wants to detach their work from that of the original project.

Objective

The objective is to provide a clear guide for efficiently using DIGIT infrastructure on various platforms like SDC, NIC, or commercial clouds. This document outlines the infrastructure overview, operational guidelines, and recommendations, along with the segregation of duties (SoD). It helps to plan the procurement and build the necessary capabilities to deploy and implement DIGIT.

In a shared control scenario, the state program team must adhere to these guidelines and develop their own control implementation for the state's cloud infrastructure and collaborations with partners. This ensures standardized and smooth operational excellence in the overall system.

DIGIT Infrastructure Overview

DIGIT Platform is designed as a microservices architecture, using open-source technologies and containerized apps and services. DIGIT components/services are deployed as docker containers on a platform called Kubernetes, which provides flexibility for running cloud-native applications anywhere like physical or virtual infrastructure or hypervisor or HCI and so on. Kubernetes handles the work of scheduling containerized services onto a compute cluster and manages the workloads to ensure they run as intended. And it substantially simplifies the deployment and management of microservices.

Provisioning the Kubernetes cluster will vary across from commercial clouds to state data centres, especially in the absence of managed Kubernetes services like AWS, Azure, GCP and NIC. Kubernetes clusters can also be provisioned on state data centres with bare-metal, virtual machines, hypervisors, HCI, etc. However providing integrated networking, monitoring, logging, and alerting is critical for operating Kubernetes Clusters when it comes to State data centers. DIGIT Platform also offers add-ons to monitor Kubernetes cluster performance, logging, tracing, service monitoring and alerting, which the implementation team can take advantage.

Below are the useful links to understand Kubernetes:

• Understand containers, dockers

• What is Kubernetes and Architecture

• Requirements to setup Kubernetes cluster

• Provider Managed Cluster vs Self-provisioning Cluster

• DIGIT Deployment on Kubernetes

DIGIT On Kubernetes - High-Level Deployment Diagram

DIGIT Infra Specification On SDC or NIC Or Any Commercial Cloud

Operational Recommendations

DIGIT strongly recommends Site reliability engineering (SRE) principles as a key means to bridge development and operations gaps by applying a software engineering mindset to system and IT administration topics. In general, an SRE team is responsible for the availability, latency, performance, efficiency, change management, monitoring, emergency response, and capacity planning.

Monitoring Tools Recommendations

Commercial clouds like AWS, Azure and GCP offer sophisticated monitoring solutions across various infra levels like CloudWatch and StackDriver. In the absence of such managed services to monitor, we can look at various best practices and tools listed below which help in debugging and troubleshooting efficiently.

• Network monitoring, Diagnosis, Assessment, Identify the latency

• Systems

• Application monitoring

• Web application monitoring

• Alert management, Alert Types

• Distributed Tracing

• Telemetry

Key Standard Operating Procedures (SOPs)

• Segregation of duties and responsibilities.

• SME and SPOCs for L1.5 support along with the SLAs defined.

• Ticketing system to manage incidents, converge and collaborate on various operational issues.

• Monitoring dashboards at various levels like Infrastructure, networks and applications.

• Transparency of monitoring data and collaboration between teams.

• Periodic remote sync-up meetings, acceptance and attendance to the meeting.

• Ability to see stakeholders' availability of calendar time to schedule meetings.

• Periodic (weekly, monthly) summary reports of the various infra, operations incident categories.

• Communication channels and synchronization regularly and also upon critical issues, changes, upgrades, releases etc.

Segregation Of Duties

While DIGIT is deployed at state cloud infrastructure, it is essential to identify and distinguish the responsibilities between Infrastructure, Operations and Implementation partners. Identify these teams and assign SPOC, define responsibilities and ensure the Incident Management process is followed to visualize, track issues and manage dependencies between teams. Essentially these are monitored through dashboards and alerts are sent to the stakeholders proactively. eGov team can provide consultation and training on a need basis depending on any of the below categories.

State IT/Cloud team -Refers to state infra team for the Infra, network architecture, LAN network speed, internet speed, OS Licensing and upgrade, patch, compute, memory, disk, firewall, IOPS, security, access, SSL, DNS, data backups/recovery, snapshots, capacity monitoring dashboard.

• State program team - Refers to the owner for the whole DIGIT implementation, application rollouts, and capacity building. Responsible for identifying and synchronizing the operating mechanism between the below teams.

• Implementation partner - Refers to the DIGIT Implementation, application performance monitoring for errors, logs scrutiny, TPS on peak load, distributed tracing, DB queries analysis, etc.

• Operations team - this team could be an extension of the implementation team that is responsible for DIGIT deployments, configurations, CI/CD, change management, traffic monitoring and alerting, log monitoring and dashboard, application security, DB Backups, application uptime, etc.

Skills To Setup, Operate & Maintain DIGIT On SDC

Resource Requirement

DIGIT - Security Standards & Operational Recommendations

This section provides insights on security principles, security layers and the line of control that we focus on to prevent DIGIT security from the code, application, access, infra and operations. The target audience of this section are internal teams, partners, ecosystems and states to understand what security measures to be considered to secure DIGIT from an infrastructure and operations perspective .

DIGIT - Key Security Principles

Subscribe to the DIGIT applicable OWASP top 10 standard across various security layers.

1. Minimize attack surface area

2. Implement a strong identity foundation - Who accesses what and who does what.

3. Apply security at all possible layers

4. Automate security best practices

5. Separation of duties (SoD).

6. The principle of Least privilege (PoLP)

7. Templatized design - (Code, Images, Infra-as-code, Deploy-as-code, Conf-as-code, etc)

8. Align with MeiTY Standards to meet SDC Infra policies.

Security Layers & Line of Control

Application Layer

The presentation layer is likely to be the #1 attack vector for malicious individuals seeking to breach security defences like DDoS attacks, Malicious bots, Cross-Site Scripting (XSS) and SQL injection. Need to invest in web security testing with the powerful combination of tools, automation, process and speed that seamlessly integrates testing into software development, helping to eliminate vulnerabilities more effectively, deploy a web application firewall (WAF) that monitors and filters traffic to and from the application, blocking bad actors while safe traffic proceeds normally.

Key Security Measures

1. TLS-protocols/Encryption: Access control to secure authentication and authorization. All APIs that are exposed must have HTTPS certificates and encrypt all the communication between client and server with transport layer security (TLS).

2. Auth Tokens: An authorization framework that allows users to obtain admittance to a resource from the server. This is done using tokens in microservices security patterns: resource server, resource owner, authorization server, and client. These tokens are responsible for access to the resource before its expiry time. Also, Refresh Tokens that are responsible for requesting new access after the original token has expired.

3. Multi-factor Authentication: authorize users on the front end, which requires a username and password as well as another form of identity verification to offer users better protection by default as some aspects are harder to steal than others. For instance, using OTP for authentication takes microservice security to a whole new level.

4. Rate Limit/DDoS: denial-of-service attacks are the attempts to send an overwhelming number of service messages to cause application failure by concentrating on volumetric flooding of the network pipe. Such attacks can target the entire platform and network stack.

To prevent this:

• We should set a limit on how many requests in a given period can be sent to each API.

• If the number exceeds the limit, block access from a particular API, at least for some reasonable interval.

• Also, make sure to analyze the payload for threats.

• The incoming calls from a gateway API would also have to be rate-limited.

• Add filters to the router to drop packets from suspicious sources.

5. Cross-site scripting (XSS): scripts that are embedded in a webpage and executed on the client side, in a user’s browser, instead of on the server side. When applications take data from users and dynamically include it in webpages without validating the data properly, attackers can execute arbitrary commands and display arbitrary content in the user’s browser to gain access to account credentials.

How to prevent:

• Applications must validate data input to the web application from user browsers.

• All output from the web application to user browsers must be encoded.

• Users must have the option to disable client-side scripts.

6. Cross-Site Request Forgery (CSRF): is an attack whereby a malicious website will send a request to a web application that a user is already authenticated against from a different website. This way an attacker can access functionality in a target web application via the victim's already authenticated browser. Targets include web applications like social media, in-browser email clients, online banking and web interfaces for network devices. To prevent this CSRF tokens are appended to each request and associated to the user’s session. Such tokens should at a minimum be unique per user session, but can also be unique per request.

How to prevent:

• By including a challenge token with each request, the developer can ensure that the request is valid and not coming from a source other than the user.

8. SQL Injection (SQLi): allows attackers to control an application’s database – letting them access or delete data, change an application’s data-driven behaviour, and do other undesirable things – by tricking the application into sending unexpected SQL commands. SQL injections are among the most frequent threats to data security.

How to prevent:

• Using parameterized queries which specify placeholders for parameters so that the database will always treat them as data rather than part of an SQL command. Prepared statements and object-relational mappers (ORMs) make this easy for developers.

• Remediate SQLi vulnerabilities in legacy systems by escaping inputs before adding them to the query. Use this technique only where prepared statements or similar facilities are unavailable.

• Mitigate the impact of SQLi vulnerabilities by enforcing the least privilege on the database. Ensure that each application has its database credentials and that these credentials have the minimum rights the application needs.

Security In The Code

The primary causes of commonly exploited software vulnerabilities are consistent defects, bugs, and logic flaws in the code. Poor coding practices can create vulnerabilities in the system that can be exploited by cybercriminals.

What defines a security in the code:

1. White-box code analysis: As developers write code, the IDE needs to provide focused, real-time security feedback with white-box code analysis. It also helps developers remediate faster and learn on the job through positive reinforcement, remediation guidance, code examples, etc.

2. Static Code Analysis (SAST): A static analysis tool reviews program code, searching for application coding flaws, back doors or other malicious code that could give hackers access to critical data or customer information. However, most static analysis tools can only scan source code.

3: Vulnerability assessment: Vulnerability assessment for the third-party libraries/artefacts as part of CI and GitHub PR process. Test results are returned quickly and prioritized in a Fix-First Analysis that identifies both the most urgent flaws and the ones that can be fixed most quickly, allowing developers to optimize efforts and save additional resources.

4. Secure PII/Encrypt: Personally identifying information – to make sure that it is not being displayed as plain text. All the passwords and usernames must be masked during the storing in logs or records. However, adding extra encryption above TLS/HTTP won’t add protection for traffic travelling through the wire. It can only help a little bit at the point where TLS terminates, so it can protect sensitive data (such as passwords or credit card numbers) from accidental dumping into a request log. Extra encryption (RSA 2048+ or Blowfish) might help protect data against those attacks that aim at accessing the log data. But it will not help with those who try accessing the memory of the application servers or the main data storage.

5. Manual Penetration Testing: Some categories of vulnerabilities, such as authorization issues and business logic flaws, cannot be found with automated assessments and will always require a skilled penetration tester to identify them. Need to employ Manual Penetration Testing that uses proven practices to provide extensive and comprehensive security testing results for web, mobile, desktop, and back-end with detailed results, including attack simulations.

Libraries/Containers

Components, such as libraries, frameworks, container images, and other software modules, almost always run with full privileges. If a vulnerable component is exploited, such an attack can facilitate serious data loss or server takeover. Applications using components with known vulnerabilities may undermine application defences and enable a range of possible attacks and impacts.

Automating dependency checks for the libraries and container auditing, as well as using other container security processes as part of the CI periodically or as part of PRs can largely prevent these vulnerabilities. Subscribing to tools that comply with vulnerable library databases such as OSVDB, Node Security Project, CIS, National Vulnerability Database, and Docker Bench for Security can help identify and fix the vulnerabilities periodically. A private docker registry can help.

Data Security

Data Security involves putting in place specific controls, standard policies, and procedures to protect data from a range of issues, including:

• Enforced encryption: Encrypt, manage and secure data by safeguarding it in transit. Password-based, easy to use and very efficient.

• Unauthorized access: Blocking unauthorized access plays a central role in preventing data breaches. Implementing Strong Password Policy and MFA.

• Accidental loss: All data should be backed up. In the event of hardware or software failure, breach, or any other error to data; a backup allows it to continue with minimal interruption. Storing the files elsewhere can also quickly determine how much data was lost and/or corrupted.

• Destruction: Endpoint Detection and Response (EDR) – provides visibility and defensive measures on the endpoint itself, when attacks occur on endpoint devices this can eliminate gaining access systems and avoid destruction of the data.

Infra/Cloud

In microservices and the Cloud Native architectural approach, the explosion of ephemeral, containerized services that arise from scaling applications developed increases the complexity of delivery. Fortunately, Kubernetes was developed just for this purpose. It provides DevOps teams with an orchestration capability for managing the multitude of deployed services, with in-built automation, resilience, load balancing, and much more. It's perfect for the reliable delivery of Cloud Native applications. Below are some of the key areas to get more control to establish policies, procedures and safeguards through the implementation of a set of rules for compliance. These rules cover infra privacy, security, breach notification, enforcement, and an omnibus rule that deals with security compliance.

• Strong stance on authentication and authorization

• Role-Based Access Control (RBAC)

• Kubernetes infrastructure vulnerability scanning

• Hunting misplaced secrets

• Workload hardening from Pod Security to network policies

• Ingress Controllers for security best practices

• Constantly watch your Kubernetes deployments

• Find deviations from desired baselines

• Should alert or deny on policy violation

• Block/Whitelist (IP or DNS) connections before entering the workloads.

• Templatize the deployment/secrets configs and serve as config-as-code.

Network Security

Kubernetes brings new requirements for network security, because applications, that are designed to run on Kubernetes, are usually architected as microservices that rely on the network. They make API calls to each other. Steps must be taken to ensure proper security protocols are in place. The following are the key areas for implementing network security for a Kubernetes platform:

• Container Groups: Coupled communication between grouped containers, is achieved inside the Pod that contains one or more containers.

• Communication between Pods: Pods are the smallest unit of deployment in Kubernetes. A Pod can be scheduled on one of the many nodes in a cluster and has a unique IP address. Kubernetes places certain requirements on communication between Pods when the network has not been intentionally segmented. These requirements include:

• Containers should be able to communicate with other Pods without using network address translation (NAT).

• All the nodes in the cluster should be able to communicate with all the containers in the cluster.

• The IP address assigned to a container should be the same that is visible to other entities communicating with the container.

• Pods and Services: Since Pods are ephemeral in nature, an abstraction called a Service provides a long-lived virtual IP address that is tied to the service locator (e.g., a DNS name). Traffic destined for that service VIP is then redirected to one of the Pods and offers the service using that specific Pod’s IP address as the destination.

• Traffic Direction: Traffic is directed to Pods and services in the cluster via multiple mechanisms. The most common is via an ingress controller, which exposes one or more service VIPs to the external network. Other mechanisms include node ports and even publicly-addressed Pods.

Operational Security

It is a procedural security that manages risk and encourages to view of operations from the perspective of an adversary to protect sensitive information from falling into the wrong hands. Following are a few best practices to implement a robust, comprehensive operational security program:

• Implement precise change management processes: All changes should be logged and controlled so they can be monitored and audited.

• Restrict access to network devices using AAA authentication: a “need-to-know” is a rule of thumb regarding access and sharing of information.

• Least Privilege (PoLP): Give the minimum access necessary to perform their jobs.

• Implement dual control: Those who work on the tasks are not the same people in charge of security.

• Automate tasks: reduce the need for human intervention. Humans are the weakest link in any organization’s operational security initiatives because they make mistakes, overlook details, forget things, and bypass processes.

Incident response and disaster recovery planning: are always crucial components of a sound security posture, we must have a plan to identify risks, respond to them, and mitigate potential damages.

• In eGovernments Foundations we are having multiple number of Teams. we can create independent teams to manage repository permissions and mentions for groups of people.

• Only organization owners and maintainers can create team. Owners can also restrict creation permissions for all teams in an organization.

• First sign in to your organization github account.

• Once you sign in to your account and if you open view organization you can able to see the above page.

• Click on Teams. You will see the below image.

• Now, click on the New team

• Fill the details as shown in the below image:

• After creating team, you will able to see the below image.

• If you click on members.you can add members to your team by providing their github username or mail.

• Now, you have successfully created GitHub team.

Creating a branch inside repository:

• Go to the repository and click on new branch.

• Here I have created a branch named DIGIT

• After, go to that branch in the same repository.

Creating branch protection rule:

• Branch protection rule states that, how to manage the branch restrictions/permissions in GitHub.

NOTE : You must have admin access orelse you have to be a codeowner to make these changes for branch restrictions/permissions.

• Open https://github.com and choose any repository.Go to the main page. Click on settings.

• Click on branches

• If you click on the Edit rules you can able to see the rules which are applied for that branch.you should follow the rules when ever you are going to made any changes to that branch and pushing it.

• If you want to create new branch protection rule click on Add Rule.

The common restrictions we are following to merge branches are :

1.Requires pull request

2.Requires approvals from CODE OWNERS

• Only the CODE OWNERS can have access to merge and makes changes to these rules.

• People with admin or owner permissions can set up a CODEOWNERS file in a repository.

• The people you choose as code owners must have write permissions for the repository.

• When the code owner is a team, that team must be visible and it must have write permissions, even if all the individual members of the team already have write permissions directly, through organization membership, or through another team membership.

• For every branch there will be a CODEOWNER file. Only they can able to write the code and able to merge the pull requests.

Create CODEOWNER file:

• Go to any of your branch(DIGIT branch created previously) in a repository and click on new file and name it as CODEOWNERS

• Click on "Create a new branch for this commit and start a pull request" and click on propose new file

• Next click on Create pull request and then Merge pull request and confirm merge.

• Add the GitHub Id's of all the team or people whom you want to add.

• In the top right corner of GitHub.com, click your profile photo, then click Your organizations.

• Click the name of your organization

• After that click on People

• Next, Click on Invite member

• Type the username, full name, or email address of the person you want to invite and click Invite.

To provide access to a repository:

• Go to the repository and click on settings

• Next click on Collaborators and teams.

• Provide access to edit the code based on the user request.

Docker Hub: It is a service provided by Docker for finding and sharing container images with our team. Key features include: Private Repositories: Push and pull container images. Automated Builds: Automatically build container images from GitHub and Bitbucket and push them to Docker Hub.

Users get access to free public repositories for storing and sharing images or can choose a subscription plan for private repositories.

What is the use of Docker Hub repository?

Docker Hub repositories allow you share container images with your team, customers, or the Docker community at large. Docker images are pushed to Docker Hub through the docker push command. A single Docker Hub repository can hold many Docker images.

Docker Hub provides the following major features:

• Repositories: Push and Pull container images.

• Teams and Organization: Manage access to private repositories of contanier images.

• Docker Offical Images: Pull and use high-quality container images provided by Docker.

• Docker Verified Publisher Images: Pull and use high-quality container images provided by extrernal vendors.

• Builds: Automatically build container images from GitHub and push them to Docker Hub.

• Webhooks: Trigger actions after a successful to repository to integrate Docker Hub with other services.

The following steps containes instructions on how to easily get Login to Docker Hub.

Step 1: Sign up for Docker account.

• Follow the link below to create a Docker ID.

Step 2: Create your first repository.

• Click and create a Repository on the Docker Hub welcome page.

• Name it in <Your-username>.

• Set the visibility to private.

• Click create.

You have created your first repository.

Step 3: Download and Install Docker Desktop:

You will need to download Docker desktop to build, push and pull container images.

• Download and install Docker desktop by following link given below

• Sign in to the Docker desktop application using the Docker ID you have just created.

Step 4: Pull and run a container image from Docker Hub:

• Run the following command to pull the image from Docker Hub.

• Run the image locally.

Then the output will be similar to;

Step 5: Build and push a container image to Docker Hub from your computer:

• Start by creating a Dockerfile to specify your application.

• Run the command to build your Docker image.

• Run your Docker image locally.

• Login in to a Docker registry.

Options:

• Push your Docker image Docker Hub.

Your repository in Docker Hub should now display new Latest tags under Tags.

You can create and register an OAuth App under your personal account or under any organization you have administrative access to. While creating your OAuth app, remember to protect your privacy by only using information you consider public.

Note: A user or organization can own up to 100 OAuth apps.

• In the upper-right corner of any page, click your profile photo, then click Settings.

• In the left sidebar, click Developer settings.

• In the left sidebar, click OAuth Apps.

• Click New OAuth App.

Note: If you haven't created an app before, this button will say, Register a new application.

• In "Application name", type the name of your app.

Warning: Only use information in your OAuth app that you consider public. Avoid using sensitive data, such as internal URLs, when creating an OAuth App.

• In "Homepage URL", type the full URL to your app's website.

• Optionally, in "Application description", type a description of your app that users will see.

• In "Authorization callback URL", type the callback URL of your app.

Note: OAuth Apps cannot have multiple callback URLs, unlike GitHub Apps.

• If your OAuth App will use the device flow to identify and authorize users, click Enable Device Flow. For more information about the device flow, see "Authorizing OAuth Apps."

• Click Register application.

Terraform: Terraform is an open-source infrastructure as code software tool that enables you to safely and predictably create, change, and improve infrastructure.

what is Terraform is used for: Terraform is an IAC tool, used primarily by DevOps teams to automate various infrastructure tasks. The provisioning of cloud resources, for instance, is one of the main use cases of Terraform. It is a open-source provisioning tool written in the Go language and created by HashiCorp.

• To install Terraform, use the following link to download the zip file.

• As per our requirment we have to install a specific version which is 0.14.10.

• Install the unzip.

• Extract the downloaded file archive.

• Move the executable into a directory searched for executables.

• Run the below command to check whether the terraform is working.

Pre-requisites:

• Install Docker in your local machine.

• Docker hub account.

Procedure:

• To move the existing docker images from one account to another account by changing tags.

• First, we have to login to the docker account in which the images are present.

sudo docker login -u <user_name> -p <password>

• We need to pull the image from the docker container to local machine.

sudo docker pull <image_name>:<image_tag>

• Next, we have to change the tag name to our required docker container tag

sudo docker tag <image_id> <new_tag>/<image_name>:<image_tag>

• Now, we have our required images with tags in our local machine. We need to push these images from local machine to destination container. First, login to the destination account using the above docker login command and then push the image using below command.

docker push <image_name>:<image_tag>

• Once successfully pushed, if you check in your docker hub account the images will be present.

Once DIGIT is installed, check the health of the system to ensure it is ready for usage:

• Check to make sure all services are running. This can be done by fetching all k8s pods:

kubectl get pods

All pods should be in "running" state.

• Step 2

• Step 3

• DB monitoring, alerting and debugging guidelines

• How to check if Infra is working as expected?

• How to monitor and setup alerts? Other debugging tools?

• Solutions to common problems and next steps

• Monitoring how-to

• Debugging

• Fixing/escalating

Pre-requisites:

• Install Visualstudio IDE Code for better code/configuration editing capabilities

• Install Terraform v0.14.10.

• Install AWS CLI.

Customization

• Clone the DIGIT-DevOps repo

$ git clone https://github.com/egovernments/DIGIT-DevOps.git

• Here we are using AWS cloud service provider to create terraform infra. So, we are choosing sample-aws module (Terraform module is a collection of standard configuration files in a dedicated directory).

• Open sample-aws in visual studio using the below command.

$ code DIGIT-DevOps/tree/release/infra-as-code/terraform/sample-aws

• In that sample-aws module we can find the below terraform templates

main.tf
providers.tf
outputs.tf
variables.tf

• main.tf will contain the main set of configuration for your module.

• outputs.tf will contain the output definitions for your module. Module outputs are made available to the configuration using the module, so they are often used to pass information about the parts of your infrastructure defined by the module to other parts of your configuration.

• providers.tf allow terraform to interact with cloud providers,SAAS providers. In this sample-aws our provider is aws.

• variables.tf will contain the variable definitions for your module. When your module is used by others, the variables will be configured as arguments in the module block. Since all Terraform values must be defined, any variables that are not given a default value will become required arguments. Variables with default values can also be provided as module arguments, overriding the default value.

• To setup the DIGIT infra we made changes in variables.tf. Open variables.tf in visual studio using the below code.

$ code DIGIT-DevOps/tree/release/infra-as-code/terraform/sample-aws/variables.tf

• Change the values in variables.tf which are specified to replace based on our requirements.For example: cluster_name, network_availability_zones, availability_zones, ssh_key_name, db_name, db_username.

• After customizing the values in variables.tf configure the aws credentials using the below commands.

$ aws configure --profile <profile_name>

• Provide AWS access key id,AWS secret access key,Default region and Default output format.

• Set aws_session _token using the below command.

$ aws configure --profile <profile_name> set aws_session_token <session_token>

• To make sure that aws credentials are configured use the below command.

$ aws s3 ls

• The output should be similar to the below image.

• After that run the below commands in the terminal one after another.

$ terraform init
$ terraform apply
$ terraform plan

• terraform init is used to initialize your code to download the requirements mentioned in your code.

• terraform plan is used to review changes and choose whether to simply accept them or not.

• terraform apply is used to accept changes and apply them against real infrastructure.

• After successfully running these commands we are able to set up the infra in aws. We are able to see the config file which is used to deploy the environment.

• Want to destroy the terraform use the below command.

terraform destroy

• Kubernetes also known as K8s, is an open-source system for automating deployment, scaling, and management of containerized applications.kubectl, allows you to run commands against Kubernetes clusters.

• If you want to study about kubernetes in detail, open Kubernetes

Why kubectl is using in DIGIT?

There are some other tools like kubelet along with kubectl. kubectl is the command-line interface (CLI) tool for working with a Kubernetes cluster. Kubelet is the technology that applies, creates, updates, and destroys containers on a Kubernetes node.But the only difference is, using kubectl the developer can interacts with kubernetes cluster. So we are using kubectl in DIGIT.

To know kubernetes is installed:

kubectl version

To install or update kubectl:

In Windows:

• Download the kubectl latest release v1.25.0. or if you have curl installed use this command:

curl.exe -LO "https://dl.k8s.io/release/v1.25.0/bin/windows/amd64/kubectl.exe"

• To download curl follow the page and proceed the download with curl https://www.wikihow.com/Install-Curl-on-Windows

• Append or prepend the kubectl binary folder to your PATH environment variable. To perform this, complete the following steps:

• 1.Open the kubectl.exe folder in files and copy that folder.

• 2.Create a new folder in Local Disk(C:) with name Kube.

• 3. Paste the kubectl.exe folder there.

• 4. Open windows option Search for Advanced system settings

• 5. Click on Environmental variables and then System variables>Path>add and add your path name i.e c:\kube

• 6. Save it.

• Once you install kubectl, you can verify its version with the following command:

kubectl version --short --client

To install kubectl in linux:

• Open the below link to install kubectl in linux: https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/

To install kubectl in MacOs:

• Open the below link to install kubectl in macos: https://kubernetes.io/docs/tasks/tools/install-kubectl-macos/

Pre-Reads

Pre-requisites

• DIGIT uses golang (required v1.13.3) automated scripts to deploy the builds onto Kubernetes - Linux or Windows or Mac.

• kubectl is a CLI to connect to the kubernetes cluster from your machine

• Install Visualstudio IDE Code for better code/configuration editing capabilities

• Git

What is Cert-manager

Cert-manager adds certificates and certificate issuers as a resource types in kubernetes cluster,and simplifies the process of obtaining, renewing and using those certificates. It will ensure certificates are valid and up-to-date, and attempt to renew certificates at a configured time before expiring.

What is SSL Certificate

SSL Certificate is a digital certificate that authenticates a website's identity and enables encrypted connection. SSL stands for Secure Sockets Layer, a security protocol that creates an encrypted link between a web server and a web browser. SSL cetificates keeps internet connections secure and prevents criminals from reading or modifying information transferred between two systems.

• Cert-Manager can issue certificates from a variety of supported sources, including Let's Encrypt, HashiCorp Vault, and Venafi as well as private PKI.

• In eGov Organization we are using letsencrypt-prod,letsencrypt-staging as a certificate-issuer.

• First, we have to clone DIGIT-DevOps repo.

$ git clone https://github.com/egovernments/DIGIT-DevOps.git

• Check the cert-manager chart templates which contains yaml files of clusterissuer and clusterrole in the below link.

https://github.com/egovernments/DIGIT-DevOps/tree/release/config-as- code/helm/charts/backbone-services/cert-manager/templates

• If we want to override any values in the chart. Open values.yaml and customize the chart.

https://github.com/egovernments/DIGIT-DevOps/blob/release/config-as-code/helm/charts/backbone-services/cert-manager/values.yaml

• Open egov-demo template in the Visual Studio code.

$ code DIGIT-DevOps/config-as-code/environments/egov-demo.yaml

• Check whether the below configurations is present in your environment file. If not add these configurations in your environment file.

Deploying cert-manager

Run the following command to deploy only the cert-manager.

$ cd DIGIT-DevOps/deploy-as-code/deployer
$ go run main.go -c -e egov-demo 'cert-manager'

• After deploying check the certificate is issued or not using the below command.

$ kubectl get certificates -n <namespace_name>

• The following output will be displayed.

• Once the certificate is issued we can see it in secrets.

$ kubectl get secrets

• The following output will be displayed

• To know about the cluster-issuers used in our deployement we can use the following command.

$ kubectl get clusterissuers

• The following output will be displayed

• Options

• Infra level separation vs logical separation

• Recommendations

Multi-tenancy Benefits

Many tenants, one cluster (Multi-tenancy), Why use multi-tenancy and benefits of multi-tenancy?

Multi-tenancy is the more common option for several reasons, but affordability tops the list:

Cost efficiency: Sharing of resources, databases, and the application itself means lower costs per customer. There is no need to buy or manage additional infrastructure or software. All the tenants share the server and storage space, which proves to be cheaper as it promotes economies of scale

Fast, easy deployment: With no new infrastructure to worry about, set-up and onboarding are simple. For instance carving out resources for a new team/project

Built-in security: Isolation between the tenants

Optimum performance: Multi-tenancies allow improve operational efficiency such as speed, utilisation, etc.

High scalability: Service small customers (whose size may not warrant dedicated infrastructure) and large organization's (that need access to unlimited computing resources).

How to implement multi-tenancy?

1. Infra Isolation

(a) Namespaces

Namespaces are the primary unit of tenancy in Kubernetes. By themselves, they don’t do much except organize other objects — but almost all policies support namespaces by default

• Require cluster-level permissions to create

• Included in Kubernetes natively

• Official Kubernetes documentation on namespaces: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/

(b) RBAC: Role based access control for Kubernetes

Kubernetes includes a built-in role-based access control mechanism that enables you to configure fine-grained and specific sets of permissions that define how a given Google Cloud user, or group of users, can interact with any Kubernetes object in your cluster, or in a specific Namespace of your cluster.

• Kubernetes RBAC is enabled by default

• Official Kubernetes documentation on RBAC: https://kubernetes.io/docs/reference/access-authn-authz/rbac/

(C) Network Policies: To isolate traffic between the namespaces

Network policies allow you to limit connections between Pods. Therefore, using network policies provide better security by reducing the compromise radius.

• Network Policies are an application-centric construct which allow you to specify how a pod is allowed to communicate with various network “entities”

• Note that the network policies determine whether a connection is allowed, and they do not offer higher level features like authorization or secure transport (like SSL/TLS).

• Control traffic flow at the IP address or port level (OSI layer 3 or 4)

• Official Kubernetes documentation on Network Policies: https://kubernetes.io/docs/concepts/services-networking/network-policies/

2. Data Isolation:

There are multiple well-known strategies to implement this architecture, ranging from highly isolated (like single-tenant) to everything shared. We can implement multi-tenancy using any of the following approaches:

1. Database per Tenant: Each Tenant has its own database and is isolated from other tenants.

2. Shared Database, Shared Schema: All Tenants share a database and tables. Every table has a Column with the Tenant Identifier, that shows the owner of the row.

3. Shared Database, Separate Schema: All Tenants share a database, but have their own database schemas and tables.

Multi-tenancy Models

Overview

This page provides the steps to follow for upgrading Kafka Connect.

Steps

• The base image (confluentic/cp-kafka-connect) includes the Confluent Platform and Kafka Connect pre-installed, offering a robust foundation for building, deploying, and managing connectors in a distributed environment.

• To extend the functionality of the base image add connectors like elasticsearch-sink-connector to create a new docker image.

• Download the elasticsearch-sink-connector jar files on your local machine using the link .

• Create a Dockerfile based on the below sample code.

• Run the below command to build the docker image.

• Run the below command to rename the docker image.

• Push the image to the dockerhub using the below command.

• Replace the image tag in kafka-connect helm chart values.yaml and redploy the kafka-connect.

Overview

This page provides comprehensive documentation and instructions for implementing a rolling upgrade strategy for your Elasticsearch cluster.

Steps

Note: During the rolling upgrade, it is anticipated that there will be some downtime. Additionally, ensure to take an elasticdump of the Elasticsearch data using the script provided below in the playground pod.

• Copy the below script and save it as es-dump.sh. Replace the elasticsearch URL and the indices names in the script.

• Run the below commands in the terminal.

• Now, run the below command inside the playground pod.

Rolling upgrade from v6.6.2 to v7.17.15

Steps

1. List the elasticsearch pods and enter into any of the elasticsearch pod shells.

1. Scale down the replica count of elasticsearch master and data from 3 to 0.

1. Edit the Statefulset of elasticsearch master by replacing the docker image removing deprecated environment variables and adding compatible environment variables. Replace the elasticsearch image tag from 6.6.2 to 7.17.15. The below code provides the depraced environment variables and compatible environment variables.

1. Edit elasticsearch-master values.yaml file

1. Edit the Statefulset of elasticsearch data by replacing the docker image removing deprecated environment variables and adding compatible environment variables. Replace the elasticsearch image tag from 6.6.2 to 7.17.15.

1. Edit elasticsearch-data values.yaml file.

1. After making the changes, scale up the statefulsets of elasticsearch data and master.

1. After all pods are in running state, re-enable shard allocation and check cluster health.

You have successfully upgraded the elasticsearch cluster from v6.6.2 to v7.17.15 :)

ReIndexing the Indices:

After successfully upgrading the elasticsearch, reindex the indices present in elasticsearch using below script which are created in v6.6.2 or earlier.

Copy the below script and save it as es-reindex.sh. Replace the elasticsearch URL in the script.

Run the below commands in the terminal.

Now, run the below command inside the playground pod.

NOTE: Make Sure to delete jaeger indices as mapping is not supported in v8.11.3 and the indices which are created before v7.17.15 by reindexing. If the indices which are created in v6.6.2 or earlier are present then the upgradation from v7.17.15 to v8.11.3 may fail.

Rolling upgrade from v7.17.15 to v8.11.3 & security is disabled

Steps

1. Scale down the replica count of elasticsearch master and data from 3 to 0.

1. Edit the Statefulset of elasticsearch master by replacing the docker image removing deprecated environment variables and adding compatible environment variables. Replace the elasticsearch image tag from 7.17.15 to 8.11.3. The below code provides the compatible environment variables and if you are following a rolling upgrade then there are no deprecated environment variables from v7.17.15 to v8.11.3.

1. Edit the Statefulset of elasticsearch data by replacing the docker image removing deprecated environment variables and adding compatible environment variables. Replace the elasticsearch image tag from 7.17.15 to 8.11.3.

1. After making the changes, scale up the statefulsets of elasticsearch data and master.

1. After all pods are in running state, re-enable shard allocation and check cluster health.

Overview

Unlike rolling upgrades, direct upgrades involve migrating from an older version to a newer one in a single coordinated operation.

This comprehensive guide outlines the step-by-step process for deploying an Elasticsearch 8.11.3 cluster with enhanced security features. The document not only covers the initial deployment of the cluster but also includes instructions for seamlessly migrating data from an existing Elasticsearch cluster to the new one, allowing for a direct upgrade.

Steps

1. Clone the DIGIT-DevOps repo and checkout to the branch digit-lts-go.

1. If you want to make any changes to the elasticsearch cluster like namespaces etc. You'll find the helm chart for elastic search in the path provided below. In the below chart, security is enabled for elasticsearch. If you want to disable the security, please set the environment variable xpack.security.enabled as false in the helm chart statefulset template.

1. Elasticsearch secrets have been present in cluster configs chart since indexer, inbox services etc have dependency on elasticsearch secrets. Below is the template.

1. In cluster-configs values.yaml, add the namespaces in which you want to deploy the elasticsearch secrets.

1. Add the elasticsearch password in the env-secrets.yaml file, if not it will automatically creates a random password which will be updated everytime you deploy the elasticsearch.

1. Deploy the Elastic Search Cluster using the below commands.

1. Check the pods status using the below command.

1. Once all pods are running, execute the below commands inside the playground pod to dump data from the old elasticsearch cluster and restore it to the new elasticsearch cluster.

1. Using the above script, you can take the data dump from the old cluster and restore it in the new elasticsearch in a single command.

2. After restoring the data successfully in the new elasticsearch cluster, check the cluster health and document count using the below command.

1. Now the deployment and restoring the data are completed successfully. It's time to change the es_url and indexer_url in egov-config present under cluster-configs of the environment file. The same can be updated directly using the below command.

1. Restart all the pods which have a dependency on elasticsearchwith cluster-configs to pick a new elasticsearch_url.

Monitor, debug, fix

How/what to track?

What are the metrics to track for Kafka, Postgres and ES?

• How to monitor each and every service

• How to debug

• Potential fixes

• How to identify security issues - where to look

• Troubleshooting

• Solutions

Use tracing to track core service APIs. Add info on Jaeger.

Backbone services - Kafka, DB

Infra

Core services

Applications

Overview

Curator is a tool from Elastic (the company behind Elasticsearch) to help manage your Elasticsearch cluster. You can create, backup, and delete some indices, Curator helps make this process automated and repeatable. Curator is written in Python, so almost all operating systems support it. It can easily manage the huge number of logs written to the Elasticsearch cluster periodically by deleting them and thus helps you save disk space.

Steps

es-curator helm chart for SSL-enabled elastic search: https://github.com/egovernments/DIGIT-DevOps/tree/digit-lts-go/deploy-as-code/helm/charts/backbone-services/es-curator

es-curator helm chart for SSL disabled elastic search: https://github.com/egovernments/DIGIT-DevOps/tree/unified-env/deploy-as-code/helm/charts/backbone-services/es-curator

A very elegant way to configure and automate Elasticsearch Curator execution is using a YAML configuration. The ‘es-curator-values.yaml’ file

# Common Labels
labels:
  group: "es-curator"

cron:
  schedule: "45 18 * * *"  
# Container Configs
namespace: es-cluster
image:
  repository: "untergeek/curator"
  tag: 8.0.15
logs-cleanup-enabled: true
jaeger-cleanup-enabled: true
logs-to-retain-in-days: 7
memory_limits: 256Mi
args: [ "--config", "/etc/es-curator/config.yml", "/etc/es-curator/action_file.yml" ]

# Additional Container Envs
env: |
  - name: SERVER_PORT
    value: "8080"
  - name: JAVA_OPTS
    value: {{ index .Values "heap" | quote }}
  - name: ES_CLIENT_HOST
    valueFrom:
      configMapKeyRef:
        name: egov-config
        key: es-indexer-host 
  - name: ES_USERNAME
    valueFrom:
      secretKeyRef:
        name: elasticsearch-master-credentials
        key: username 
  - name: ES_PASSWORD
    valueFrom:
      secretKeyRef:
        name: elasticsearch-master-credentials
        key: password
  - name: ES_CLIENT_PORT
    value: "9200"  
  - name: LOG_LEVEL
    value: "DEBUG" 
  {{- if index .Values "logs-cleanup-enabled" }}                  
  - name: LOGS_CLEANUP_DISABLED
    value: "False"
  - name: RETAIN_LOGS_IN_DAYS
    value: {{ index .Values "logs-to-retain-in-days" | quote }}
  {{- end }}               
  {{- if index .Values "jaeger-cleanup-enabled" }}     
  - name: JAEGER_CLEANUP_DISABLED
    value: "False"
  - name: RETAIN_JAEGER_DATA_IN_DAYS
    value: "14"
  {{- end }}       
extraVolumes: |
  - name: config-volume
    configMap:
      name: {{ template "name" . }}-config
extraVolumeMounts: |
  - mountPath: /etc/es-curator
    name: config-volume     
resources: |
  requests:
    memory: {{ .Values.memory_limits | quote }}
  limits:
    memory: {{ .Values.memory_limits | quote }}

You can modify the above es-curator-infra-values.yaml according to the requirements, some modifications are suggested below:

  schedule: "45 18 * * *"

The above represents all the possible numbers for that position.

• Schedule Cron Job: In the above code, at line number 6, the Cron Job is Scheduled to run at 6:45 PM every day. You can schedule your Cron Job accordingly.

• RETAIN_LOGS_IN_DAYS: Specify the age of the logs to be deleted. In line 14 of the code, logs-to-retain-in-days indicate that logs older than 7 days will be deleted.

Pre-reads

This doc covers the steps on how to deploy an OpenTelemetry collector on Kubernetes. We will then use an OTEL instrumented (Go) application provided by OpenTelemetry to send traces to the Collector. From there, we will bring the trace data to a Jaeger collector. Finally, the traces will be visualised using the Jaeger UI.

This image shows the flow between the application, OpenTelemetry collector and Jaeger.

This OpenTelemetry repository provides a complete demo on how you can deploy OpenTelemetry on Kubernetes, we can use this as a starting point.

Pre-requisites

To start off, we need a Kubernetes cluster you can use any of your existing Kubernetes clusters that has got the apx 2vCPUs, 4GB RAM, and 100GB Storage.

Local Kubernetes Cluster Setup

Skip this in case you have the existing cluster.

In case, you don't have the ready Kubernetes but you have a good local machine with at least 4GB RAM left, you can use a local instance of Kind. The application will access this Kubernetes cluster through a NodePort (on port 30080). So make sure this port is free.

conn, err := grpc.DialContext(ctx, "localhost:30080", grpc.WithTransportCredentials(insecure.NewCredentials()), grpc.WithBlock())

To use NodePort with Kind, we need to first enable it.

Extra port mappings can be used to port forward to the kind nodes. This is a cross-platform option to get traffic into your kind cluster.

vim kind-config.yaml

kind: ClusterapiVersion: kind.x-k8s.io/v1alpha4nodes:- role: control-plane  # port forward 30080 on the host to 30080 on this node  extraPortMappings:  - containerPort: 30080    hostPort: 30080- role: worker

Create the cluster with: kind create cluster --config kind-config.yaml

Creating cluster "kind" ... ✓ Ensuring node image (kindest/node:v1.24.0) 🖼 ✓ Preparing nodes 📦 📦 ✓ Writing configuration 📜 ✓ Starting control-plane 🕹️ ✓ Installing CNI 🔌 ✓ Installing StorageClass 💾 ✓ Joining worker nodes 🚜Set kubectl context to "kind-kind"You can now use your cluster with:kubectl cluster-info --context kind-kindThanks for using kind! 😊

Once our Kubernetes cluster is up, we can start deploying Jaeger.

What is Jaeger?

Jaeger is an open-source distributed tracing system for tracing transactions between distributed services. It’s used for monitoring and troubleshooting complex microservices environments. By doing this, we can view traces and analyse the application’s behaviour.

Why do we need it?

Using a tracing system (like Jaeger) is especially important in microservices environments since they are considered a lot more difficult to debug than a single monolithic application.

Problems that Jaeger addresses?

• Distributed tracing monitoring

• Performance and latency optimisation

• Root cause analysis

• Service dependency analysis

Deploy Jaeger

To deploy Jaeger on the Kubernetes cluster, we can make use of the Jaeger operator.

Operators are pieces of software that ease the operational complexity of running another piece of software.

Deploy Jaeger Operator

You first install the Jaeger Operator on Kubernetes. This operator will then watch for new Jaeger custom resources (CR).

There are different ways of installing the Jaeger Operator on Kubernetes:

• using Helm

• using Deployment files

Before you start, pay attention to the Prerequisite section.

Since version 1.31 the Jaeger Operator uses webhooks to validate Jaeger custom resources (CRs). This requires an installed version of the cert-manager.

Installing Cert-Manager

cert-manager is a powerful and extensible X.509 certificate controller for Kubernetes and OpenShift workloads. It will obtain certificates from a variety of Issuers, both popular public Issuers as well as private Issuers, and ensure the certificates are valid and up-to-date, and will attempt to renew certificates at a configured time before expiry.

Installation of cert-manager of is very simple, just run:

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.9.1/cert-manager.yaml

By default, cert-manager will be installed into the cert-manager namespace.

You can verify the installation by following the instructions here

With cert-manager installed, let’s continue with the deployment of Jaeger

Installing Jaeger Operator Using Helm

Jump over to Artifact Hub and search for jaeger-operator

Add the Jaeger Tracing Helm repository:

helm repo add jaegertracing https://jaegertracing.github.io/helm-charts

To install the chart with the release name my-release (in the default namespace)

helm install my-release jaegertracing/jaeger-operator

You can also install a specific version of the helm chart:

helm install my-release jaegertracing/jaeger-operator --version 2.25.0

Verify that it’s installed on Kubernetes:

helm list -A

You can also deploy the Jaeger operator using deployment files.

kubectl create -f https://github.com/jaegertracing/jaeger-operator/releases/download/v1.36.0/jaeger-operator.yaml

At this point, there should be a jaeger-operator deployment available.

kubectl get deployment my-jaeger-operator

NAME                 READY   UP-TO-DATE   AVAILABLE   AGEmy-jaeger-operator   1/1     1            1           2m58s

The operator is now ready to create Jaeger instances.

Deploy Jaeger All-in-One

The operator that we just installed doesn’t do anything itself, it just means that we can create jaeger resources/instances that we want the jaeger operator to manage.

The simplest possible way to create a Jaeger instance is by deploying the All-in-one strategy, which installs the all-in-one image, and includes the agents, collector, query and the Jaeger UI in a single pod using in-memory storage.

Create a yaml file like the following. The name of the Jaeger instance will be simplest

vim simplest.yaml

apiVersion: jaegertracing.io/v1kind: Jaegermetadata:  name: simplest

kubectl apply -f simplest.yaml

After a little while, a new in-memory all-in-one instance of Jaeger will be available, suitable for quick demos and development purposes.

When the Jaeger instance is up and running, we can check the pods and services.

kubectl get pods

NAME                     READY STATUS    RESTARTS   AGEsimplest-656d7cf5c8-lff7b 1/1  Running   0          3m55s

kubectl get services

To get the pod name, query for the pods belonging to the simplest Jaeger instance:

Query the logs from the pod:

kubectl logs -l app.kubernetes.io/instance=simplest

{"level":"info","ts":1660155049.86027,"caller":"channelz/logging.go:50","msg":"[core]Channel Connectivity change to READY","system":"grpc","grpc_log":true}{"level":"info","ts":1660155049.8612773,"caller":"grpc/builder.go:120","msg":"Agent collector connection state change","dialTarget":":14250","status":"READY"}{"level":"info","ts":1660155049.8617437,"caller":"app/server.go:241","msg":"Starting HTTP server","port":16686,"addr":":16686"}{"level":"info","ts":1660155049.8621716,"caller":"app/server.go:260","msg":"Starting GRPC server","port":16685,"addr":":16685"}

Let’s open the Jaeger UI

Use port-forwarding to access the Jaeger UI

kubectl port-forward svc/simplest-query 16686:16686

Forwarding from 127.0.0.1:16686 -> 16686Forwarding from [::1]:16686 -> 16686

Jaeger UI

Deploy Open Telemetry Collector

To deploy the OpenTelemetry collector, we will use this otel-collector.yaml file as a starting point. The yaml file consists of a ConfigMap, Service and a Deployment.

vim otel-collector.yaml

Make sure to change the name of the jaeger collector (exporter) to match the one we deployed above. In our case, that would be:

exporters:      jaeger:        endpoint: "simplest-collector.default.svc.cluster.local:14250"

Also, pay attention to receivers. This part creates the receiver on the Collector side and opens up the port 4317 for receiving traces, which enables the application to send data to the OpenTelemetry Collector.

...  otel-collector-config: |    receivers:      otlp:        protocols:          grpc:            endpoint: "0.0.0.0:4317"...

Apply the file with: kubectl apply -f otel-collector.yaml

configmap/otel-collector-conf createdservice/otel-collector createddeployment.apps/otel-collector created

Verify that the OpenTelemetry Collector is up and running.

kubectl get deployment

kubectl logs deployment/otel-collector

"Everything is ready. Begin running and processing data."

Run Application

Time to send some trace data to our OpenTelemetry collector.

Remember, that the application access the Kubernetes cluster through a NodePort on port 30080. The Kubernetes service will bind the 4317 port used to access the OTLP receiver to port 30080 on the Kubernetes node.

By doing so, it makes it possible for us to access the Collector by using the static address <node-ip>:30080. In case you are running a local cluster, this will be localhost:30080. Source

This repository contains an (SDK) instrumented application written in Go, that simulates an application.

go run main.go

2022/08/10 20:31:37 Waiting for connection...2022/08/10 20:31:37 Doing really hard work (1 / 10)2022/08/10 20:31:38 Doing really hard work (2 / 10)2022/08/10 20:31:39 Doing really hard work (3 / 10)2022/08/10 20:31:40 Doing really hard work (4 / 10)2022/08/10 20:31:41 Doing really hard work (5 / 10)2022/08/10 20:31:42 Doing really hard work (6 / 10)2022/08/10 20:31:43 Doing really hard work (7 / 10)2022/08/10 20:31:44 Doing really hard work (8 / 10)2022/08/10 20:31:45 Doing really hard work (9 / 10)2022/08/10 20:31:46 Doing really hard work (10 / 10)2022/08/10 20:31:47 Done!

Viewing the data

Let’s check out the telemetry data generated by our sample application

Again, we can use port-forwarding to access Jaeger UI.

kubectl port-forward svc/simplest-query 16686:16686

Open the web-browser and go to http://127.0.0.1:16686/

Under Service select test-service to view the generated traces.

The service name is specified in the main.go file.

res, err := resource.New(ctx,  resource.WithAttributes(   // the service name used to display traces in backends   semconv.ServiceNameKey.String("test-service"),  ),

The application will access this Kubernetes cluster through a NodePort (on port 30080). The URL is specified here:

conn, err := grpc.DialContext(ctx, "localhost:30080", grpc.WithTransportCredentials(insecure.NewCredentials()), grpc.WithBlock()) if err != nil {  return nil, fmt.Errorf("failed to create gRPC connection to collector: %w", err) }

Done

This document has covered how we deploy an OpenTelemetry collector on Kubernetes. Then we sent trace data to this collector using an Otel SDK instrumented application written in Go. From there, the traces were sent to a Jaeger collector and visualised in Jaeger UI.

In this article, we’ll deploy ECK Operator using helm to the Kubernetes cluster and build a quick-ready solution for logging using Elasticsearch, Kibana, and Filebeat.

What is ECK?

Built on the Kubernetes Operator pattern, Elastic Cloud on Kubernetes (ECK) extends the basic Kubernetes orchestration capabilities to support the setup and management of Elasticsearch, Kibana, APM Server, Enterprise Search, Beats, Elastic Agent, and Elastic Maps Server on Kubernetes.

With Elastic Cloud on Kubernetes, we can streamline critical operations, such as:

1. Managing and monitoring multiple clusters

2. Scaling cluster capacity and storage

3. Performing safe configuration changes through rolling upgrades

4. Securing clusters with TLS certificates

5. Setting up hot-warm-cold architectures with availability zone awareness

Install ECK

1. In this case we use helmfile to manage the helm deployments: helmfile.yaml

repositories:
- name: elastic
  url: https://helm.elastic.co

releases:
  - name: eck-operator
    version: 2.2.0
    chart: elastic/eck-operator
    namespace: monitoring
    values:
    - ./eck-operator/values.yaml

2. But we can do that just with helm: Installation using helm

helm repo add elastic https://helm.elastic.co
helm install elastic-operator elastic/eck-operator -n monitoring --create-namespace

After that we can see that the ECK pod is running:

kubectl get po elastic-operator-0 -n monitoring

NAME                 READY   STATUS    RESTARTS   AGE
elastic-operator-0   1/1     Running   0          43s

The pod is up and running

Creating Elasticsearch, Kibana, and Filebeat resources

There are a lot of different applications in Elastic Stack, such as:

• Elasticsearch

• Kibana

• Beats (Filebeat/Metricbeat)

• APM Server

• Elastic Maps

• etc

In our case, we’ll use only the first three of them, because we just want to deploy a classical EFK stack.

Let’s deploy the following in the order:

1. Elasticsearch cluster: This cluster has 3 nodes, each node with 100Gi of persistent storage, and intercommunication with a self-signed TLS-certificate.

# This sample sets up an Elasticsearch cluster with 3 nodes.
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch-logging
  namespace: monitoring
spec:
  version: 8.2.0
  nodeSets:
  - name: default
    config:
      # most Elasticsearch configuration parameters are possible to set, e.g: node.attr.attr_name: attr_value
      node.roles: ["master", "data", "ingest", "ml"]
      # this allows ES to run on nodes even if their vm.max_map_count has not been increased, at a performance cost
      node.store.allow_mmap: true
    podTemplate:
      metadata:
        labels:
          # additional labels for pods
          purpose: logging
      spec:
        containers:
        - name: elasticsearch
          # specify resource limits and requests
          resources:
            limits:
              memory: 4Gi
              cpu: 2
          env:
          - name: ES_JAVA_OPTS
            value: "-Xms2g -Xmx2g"
    count: 3
    # request 100Gi of persistent data storage for pods in this topology element
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data # Do not change this name unless you set up a volume mount for the data path.
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 100Gi
        storageClassName: gp2
  http:
    tls:
      selfSignedCertificate:
        # add a list of SANs into the self-signed HTTP certificate
        subjectAltNames:
        - dns: elasticsearch-logging-es-http.monitoring.svc.cluster.local
        - dns: elasticsearch-logging-es-http.monitoring.svc
        - dns: "*.monitoring.svc"
        - dns: "*.monitoring.svc.cluster.local"