eGov Monitoring & Alerting Setup
This doc will cover how you can set up the monitoring and alerting on existing k8s cluster either with help of go lang script or Jenkins deployment Jobs.
Last updated
Was this helpful?
This doc will cover how you can set up the monitoring and alerting on existing k8s cluster either with help of go lang script or Jenkins deployment Jobs.
Last updated
Was this helpful?
is an open-source system monitoring and alerting toolkit originally built at
DIGIT uses (required v1.13.3) automated scripts to deploy the builds onto Kubernetes - or or
All DIGIT services are packaged using helm charts
is a CLI to connect to the kubernetes cluster from your machine
IDE Code for better code/configuration editing capabilities
Git
The default installation is intended to suit monitoring a kubernetes cluster the chart is deployed onto. It closely matches the kube-prometheus project.
service monitors to scrape internal kubernetes components
kube-apiserver
kube-scheduler
kube-controller-manager
etcd
kube-dns/coredns
kube-proxy
With the installation, the chart also includes dashboards and alerts.
Deployment steps:
Chose your env config file, if you are deploying monitoring and alerting into the qa environment chose qa.yaml similarly for uat, dev, and other environments.
Depending upon your environment config file update the configs repo branch (like for qa.yaml add qa branch and uat.yaml it would be UAT the branch)
3. Enable the serviceMonitor in the nginx-ingress configs which are available in the same <env>.yaml and redeploy the nginx-ingress.
go run main.go deploy -e <environment_name> -c 'nginx-ingress'
4. To enable alerting, Add alertmanager secret in <env>-secrets.yaml
If you want you can change the slack channel and other details like group_wait, group_interval, and repeat_interval according to your values.
5. You can deploy the prometheus-operator using one of the below methods.
1. Deploy using go lang deployer
go run main.go deploy -e <environment_name> -c 'prometheus-operator,grafana,prometheus-kafka-exporter'
2. Deploy using Jenkin’s deployment job. (here we are using deploy-to-dev, you can choose your environment specific deployment job )
Login to the dashboard and click on add panel
Set all required queries and apply the changes. Export the JSON file by clicking on the save dashboard
3. Go to the configs repo and select your branch. In the branch look for the monitoring-dashboards folder and update the existing *-dashboard.json with a newly exported JSON file.
chart includes multiple components and is suitable for a variety of use-cases.
Add the below grafana init container parameters to your
2. Add folder to the configs repo's branch which you selected in 1st step.
You can connect to the monitoring console at