Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Performance testing and benchmark
Since we have around 600 wards in Punjab, we decided to make the national dashboard service be robust enough to handle 600 requests simultaneously in a very short time (preferably midnight when the employees ingest that day’s data).
The national dashboard service Ingest API underwent extensive stress testing where the service was tested for 25 concurrent threads (users) targeting 100 requests per minute and each request payload containing 30 records (as the ingest API is a bulk API). The service passed all the tests with 0 per cent error with a throughput of 21 requests per second, 12671 total number of samples in 10 minutes which roughly equals 21 requests per second.
The stress test ran with the following configuration and reliably persisted each and every record that was ingested -
National-dashboard-ingest service -
Number of pods- 5
Number of threads for each pod- 25
Heap memory- 750 MB
Producer linger time- 1 ms
National dashboard Kafka pipeline service -
Number of pods- 3
Number of threads for each pod- 10
Heap memory- 512 MB
Producer linger time- 1 ms
This page provides steps to deploy Airflow DAG.
The Kubernetes environment is required for the deployment of Airflow.
Step 1: Clone the git repo for airflow, and update the values.yaml as per the requirement.
Step 2: Update the git repository URL and subpath for the directory in values.yaml
Example: the following params are updated as given below:
repo: "GitHub - pmidc-digit/utilities ”
repoSubPath: "egov-national-dashboard-accelerator/dag
branch: "develop
Step 3: Change the directory to airflow and update the helm. Update the helm repo locally and add the airflow repo to Helm using the command below:
helm repo add apache-airflow https://airflow.apache.org
The above command pulls the airflow repo and it is added to the local helm repository.
Step 4: Installing Apache airflow after updating the helm repositories
helm install airflow apache-airflow/airflow --namespace egov
The above command will take the updated repo details.
Step 5: Upgrade the changes made to values.yaml using the command below.
helm upgrade --install airflow apache-airflow/airflow -n airflow -f values.yaml
The above command updates the git repo, subpath and branch while deployment.
Step 6: Deployment is done pods service will start running with updated values.yaml
Latest files for the deployment: Attached below is the final "values.yaml" file. It syncs both the plugins and dags from the repo. Airflow Deployment
This document lists the steps required to follow to create national dashboard indexes.
Prior Knowledge of ElasticSearch
For understanding purposes, we are listing out the steps needed to be followed to create a property tax index for the national dashboard.
Open Kibana
Create an index using the following query -
PUT pt-national-dashboard
{}3. Add corresponding field mappings for the created index -
PUT pt-national-dashboard/_mapping/nss
{
"properties" : {
"assessedPropertiesForUsageCategory" : {
"type" : "long"
},
"assessments" : {
"type" : "long"
},
"cessForUsageCategory" : {
"type" : "long"
},
"createdBy" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"createdTime" : {
"type" : "long"
},
"date" : {
"type" : "date",
"format" : "dd-MM-yyyy HH:mm:ss||dd-MM-yyyy||epoch_millis||dd-MM-yyyy'T'HH:mm:ss.SSSZ"
},
"financialYear" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"interestForUsageCategory" : {
"type" : "long"
},
"lastModifiedBy" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"lastModifiedTime" : {
"type" : "long"
},
"module" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"penaltyForUsageCategory" : {
"type" : "long"
},
"propertiesRegisteredForFinancialYear" : {
"type" : "long"
},
"propertyTaxForUsageCategory" : {
"type" : "long"
},
"rebateForUsageCategory" : {
"type" : "long"
},
"region" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"state" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"todaysClosedApplications" : {
"type" : "long"
},
"todaysCollectionForUsageCategory" : {
"type" : "long"
},
"todaysTotalApplications" : {
"type" : "long"
},
"transactionsForUsageCategory" : {
"type" : "long"
},
"ulb" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"usageCategory" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"ward" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
}1. Deploy the Airflow on the state side to extract data from the Elastic Search server to push to the National Dashboard adaptor.
2. Configure the airflow with the required connection ids and the variables.
es_conn: To connect to the required ES server
digit-auth: To connect to the required API to insert data.
Variables: The credentials to connect to the API.
3. The index must contain data till the ward level and the structure must be similar to ensure the queries fetch the desired results.
4. Pull data from the DIGIT source only.
Builds deployed
national-dashboard-ingest-db: nda-patch-db6cb27b02-18
national-dashboard-kafka-pipeline:v0.0.1-762c61e743-3
Upyog devOps and MDMS changes
Add localisation for newly loaded Punjab tenants
Added NDA_SYSTEM role action mapping in MDMS and created a user with access rights of the same
Credentials: SYSTEMSU1 / eGov@123 - PG, user type: SYSTEM
Steps to trigger airflow DAG
In Airflow, a DAG –Directed Acyclic Graph – is a collection of the tasks you want to run, organized in a way that reflects their relationships and dependencies.
A DAG is defined in a Python script, which represents the DAGs structure (tasks and their dependencies) as code.
Manual Trigger
1.Log onto the Punjab Prod server using the credentials:
URL: Sign In - Airflow
username: admin
password: admin
2. Trigger the DAG by clicking on the “Trigger DAG with Config” option.
3. Enter a date and click on the Trigger button
Format {“date”: “dd-MM-yyyy”}
4. Click on the Log option and expand the DAG to view the logs. Choose a stage for any module.
Logs can also be viewed in the Elastic search index adaptor_logs
GET adaptor_logs/_search - the timestamp is provided based on the day for which the logs are being searched.
This DAG triggers every day at midnight for the previous day.
Bulk Insert For A Date Range
Execute the script to run the DAG for a date range for the staging NDB
sh iterate_over_date.sh <start-date> <end-date> ex: sh iterate_over_date.sh 01-03-2022 05-03-2022
date needs to be in the format of dd-mm-YYYY
range exclusive of the last date, [start-date, end-date). For instance: in the above example, the script will trigger DAG on 1st, 2nd, 3rd and 4th March. It will not be triggered on 5th March.
The overview of the adaptor service
The code is organised in the following code repo:
utilities/egov-national-dashboard-accelerator at develop · pmidc-digit/utilities
Refer to the folder structure below:
dags
plugins
The dags folder is the folder which contains the DAG code.
national_dashboard_template_latest.py(manual trigger)
national_dashboard_template_scheduled.py(scheduled)
The queries folder contains the ES queries for the individual modules
For adding a new module we need to add a module-specific file in the queries folder with the transform logic and then refer to the DAG code in both manual and trigger versions.
Druid DIGIT
Metabase DIGIT
https://metabase-qa.ifix.org.in/
eGov@4321
Code base: utilities/egov-revenue-maximization at develop · pmidc-digit/utilities
The command for executing the DAG:
{"start":"01-04-2022","end":"04-04-2022"}
Select the rev_max DAG right now deployed in the Punjab production
https://airflow.mseva.lgpunjab.gov.in/home
DAG has 3 steps
collect-data step:
The data is collected from ES and is converted to a JSON file.
join_data step:
The json file is converted to CSV.
The tables are joined like water and trade, water and property and trade and property.
The outcome is CSV files.
upload_data step:
The CSV files which are created are then uploaded to the druid by using the API.
The CSV files are converted to the format expected by the API and then uploaded to druid tables.
Postman collection for posting the data: https://www.getpostman.com/collections/74fcabc976b2b9baac93
| URL | UserID | Password | City | |
|---|---|---|---|---|
QA Test Cases - Test Cases Link
Elastic Search Queries - Elastic Search Queries
List of Index in Punjab - List Of Index in Punjab
Code Checkin path - Code Checkin Repo
JIRA link for the issue tracking - Dashboard for Issue Tracking
Postman Collection for the INGEST API - Postman Collection for Ingest API
National Dashboard KPI + Definitions- KPI+Definitions
Design and develop an adaptor service which extracts data from the state DIGIT installations and pushes it onto the National Dashboard instance periodically.
A list of tasks for this is tracked for the adaptor -
Adaptor to be deployed on state DIGIT installations
Periodically, the adaptor extracts data and aggregates it from the different DIGIT modules
Posts the data to the National Dashboard for the state
Bookkeeping is done for every adaptor data extract and pushes for audit and debugging
Out of scope: extraction from non-DIGIT sources
A national dashboard adaptor extracts data from the state DSS at a scheduled time which can be configured and then would ingest in the National Dashboard. The adaptor ingests data at the state/ULB/Ward level for each module on a daily basis. The adapter sends the data in a batch size of 50 to the national dashboard.
| ConnectionId | Connection Type | Host | |||
|---|---|---|---|---|---|
Refer to the sample below to delete the data for the month of July module-wise in the Postgres database:
delete from nss_ingest_data where dataKey like '%-07-2022:FIRE%';
delete from nss_ingest_data where dataKey like '%-07-2022:TL%';
delete from nss_ingest_data where dataKey like '%-07-2022:WS%';
delete from nss_ingest_data where dataKey like '%-07-2022:PGR%';
delete from nss_ingest_data where dataKey like '%-07-2022:PT%';
delete from nss_ingest_data where dataKey like '%-07-2022:MCOLLECT%';
Adjust the module and the date range accordingly.
Check the records before deleting.
Note: Deleting data from both ES and Postgres is mandated to avoid duplication of data.
GET firenoc-national-dashboard/_search
{
"query": {
"bool": {
"must": [
{
"range": {
"date": {
"gte": 1656613800000,
"lte": 1658773799000,
"format": "epoch_millis"
}
}
}
]
}
}
}
POST firenoc-national-dashboard/_delete_by_query
{
"query": {
"bool": {
"must": [
{
"range": {
"date": {
"gte": 1656613800000,
"lte": 1658773799000,
"format": "epoch_millis"
}
}
}
]
}
}
}
es_conn
ElasticSearch
elasticsearch-data-v1.es-cluster
9200
For the ES server
digit -auth-state
HTTP
https
For the auth api conenction - Staging
digit-auth
For the auth api conenction - UPYOG
Punjab Kibana QA
admin
24!jcZ]z"[$qZ%Fa
Punjab Kibana UAT
OAuth authentication
Punjab Dashboard - QA
STADSS
STADSS
Punjab
Punjab Dashboard - UAT
EMP9
eGov@123
testing
National DSS - QA
QADSS
eGov@123
amritsar
National DSS - DEV
amr001
eGov@123
amritsar
National DSS - UAT
NDSS1/NDSS2
eGov@123
PG
National Dashbaord - UPYOG
NDSS1
eGov@123
PG
AUTH API :Prod
SYSTEMSU3
eGov@123
ZWdvdi11c2VyLWNsaWVudDo=
pb
AUTH API credentials:DEV
amr001/EMPLOYEE
eGov@123
ZWdvdi11c2VyLWNsaWVudDo=
pb.amritsar
AUTH API:UPYOG
SYSTEMSU1/SYSTEM
eGov@123
pg
AUTH API:Staging
SYSTEMSU3/
SYSTEM
eGov@123
pg
The objectives of the national dashboard-ingest service are listed below.
To provide a one-stop framework for ingesting data regardless of a data source based on configuration.
To create provision for ingest based on module-wise requirements which directly or indirectly require aggregated data ingestion functionality.
Prior knowledge of Java/J2EE.
Prior knowledge of SpringBoot.
Prior knowledge of PostgreSQL.
Prior knowledge of REST APIs and related concepts like path parameters, headers, JSON etc.
Prior knowledge of ElasticSearch.
Step 1: Define the index name for the module as per your requirement in module.index.mapping key present in the configuration here - DIGIT-DevOps/qa.yaml at master · egovernments/DIGIT-DevOps .
Step 2: Define the allowed metrics for the module as per your requirement in module.fields.mapping key present in the configuration here - DIGIT-DevOps/qa.yaml at master · egovernments/DIGIT-DevOps
Step 3: Define the allowed group-by fields for the module as per your requirement in module.allowed.groupby.fields.mapping key present in the configuration here - DIGIT-DevOps/qa.yaml at master · egovernments/DIGIT-DevOps
Step 4: Define the master data index name as per your requirement in master.data.index key present in the configuration here - DIGIT-DevOps/qa.yaml at master · egovernments/DIGIT-DevOps
Step 5: Define the allowed metrics for the master data index as per your requirement in master.module.fields.mapping key present in the configuration here - https://github.com/egovernments/DIGIT-DevOps/blob/master/deploy-as-code/helm/environments/qa.yaml#L367
Step 6: Create Kafka connectors for all the modules that have been configured. A sample request for creating a trade license national dashboard Kafka connector is as follows -
curl --location --request POST 'http://kafka-connect.kafka-cluster:8083/connectors/' \
--header 'Content-Type: application/json' \
--data-raw '{
"name": "cms-case-es-sink9128",
"config": {
"connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
"connection.url": "http://elasticsearch-data-v1.es-cluster:9200/",
"type.name": "nss",
"topics": "tl-national-dashboard",
"key.ignore": true,
"schemas.enable": false,
"schema.ignore": true,
"value.converter.schemas.enable": false,
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
"value.converter": "org.apache.kafka.connect.json.JsonConverter",
"batch.size": 10,
"max.buffered.records": 500,
"flush.timeout.ms": 600000,
"retry.backoff.ms": 5000,
"read.timout.ms": 10000,
"linger.ms": 100,
"max.in.flight.requests": 2,
"errors.log.enable": true,
"errors.deadletterqueue.topic.name": "nss-es-failed",
"tasks.max": 1
}
}'7. Step 7: Run the national-dashboard-ingest application along with the national-dashboard-ingest-kafka-pipeline.
Config file - A YAML (xyz.yml) file which contains configuration for running national dashboard ingest.
API - A REST endpoint to post data based on the configuration.
When the national dashboard ingests metrics API is hit, all the data payload lookup keys are first checked against the db to determine whether they already exist or not. The db table currently being used for storing lookup keys is nss-ingest-data.
If the record for a given date and area details is not present, the payload is then flattened and pushed to nss-ingest-keydata topic.
National dashboard ingest Kafka pipeline consumer listens on nss-ingest-keydata topic and according to the module to which the data belongs, pushes it to the respective topic defined in the module.index.mapping key.
Once the national dashboard ingests the Kafka pipeline and pushes data to the respective topic, a Kafka connector then takes the flattened records from that topic and ingests into ElasticSearch.
Add configs for different modules required for National Dashboard Ingest Service and National Dashboard Kafka Pipeline service.
Deploy the latest version of National Dashboard Ingest and National dashboard Kafka pipeline service.
Add role-action mapping for APIs.
The national dashboard service is used to push aggregated data present in systems and persisting them onto elasticsearch on top of which dashboards can be built for visualizing and analyzing data.
Can perform service-specific business logic without impacting the other module.
In the future, if we want to expose the application to citizens then it can be done easily.
To integrate, the host of the national-dashboard-ingest-service module should be overwritten in the helm chart.
national-dashboard/metric/_ingest should be added as the search endpoint for the config added.
national-dashboard/masterdata/_ingest should be added as the search endpoint for the config added.
URI: The format of the ingest API to be used to ingest data using national-dashboard-ingest is as follows: national-dashboard/metric/_ingest
Example ingest request body -
{
"RequestInfo": {
"apiId": "asset-services",
"ver": null,
"ts": null,
"action": null,
"did": null,
"key": null,
"msgId": "search with from and to values",
"authToken": "82c7da0d-da73-4c35-8ea7-5b231369b4cd",
"userInfo": {
"id": 41737,
"uuid": "9a81233f-e212-4035-a831-320b70e93b82",
"userName": "NDSS1",
"name": "National Dashboard Viewer",
"mobileNumber": "7777888813",
"emailId": null,
"locale": null,
"type": "EMPLOYEE",
"roles": [
{
"name": "Employee",
"code": "SUPERUSER",
"tenantId": "pb.amritsar"
},
{
"name": "National Dashboard Admin",
"code": "NATADMIN",
"tenantId": "pb.amritsar"
},
{
"name": "FSM Employee Dashboard Viewer",
"code": "FSM_DASHBOARD_VIEWER",
"tenantId": "pb.amritsar"
},
{
"name": "National Dashboard Admin",
"code": "NATADMIN",
"tenantId": "pb"
}
],
"active": true,
"tenantId": "pb.amritsar",
"permanentCity": null
}
},
"Data": [
{
"date": "14-01-2017",
"module": "MCOLLECT",
"ward": "GODOWN AREA (BHABAT) - B14-SECTOR-13 - A1",
"ulb": "pb.amritsar",
"region": "Amritsar-MC",
"state": "Punjab",
"metrics": {
"numberOfCategories": 240,
"todaysCollection": [
{
"groupBy": "paymentMode",
"buckets": [
{
"name": "UPI",
"value": 70
},
{
"name": "CASH",
"value": 45
},
{
"name": "DEBIT_CARD",
"value": 20
}
]
},
{
"groupBy": "status",
"buckets": [
{
"name": "NEW",
"value": 70
},
{
"name": "DEPOSITED",
"value": 20
},
{
"name": "DISHONOURED",
"value": 45
}
]
},
{
"groupBy": "category",
"buckets": [
{
"name": "COMMON_MASTERS_HOARDING",
"value": 50
},
{
"name": "COMMON_MASTERS_ROAD_SHOW",
"value": 20
},
{
"name": "COMMON_MASTERS_UNIPOLLS",
"value": 5
},
{
"name": "COMMON_MASTERS_AUCTION_FEE",
"value": 25
},
{
"name": "COMMON_MASTERS_USER_FEES",
"value": 35
}
]
}
],
"numberOfReceipts": [
{
"groupBy": "status",
"buckets": [
{
"name": "NEW",
"value": 70
},
{
"name": "DEPOSITED",
"value": 105
},
{
"name": "DISHONOURED",
"value": 50
}
]
},
{
"groupBy": "paymentMode",
"buckets": [
{
"name": "CASH",
"value": 70
},
{
"name": "CHEQUE",
"value": 105
},
{
"name": "DEBIT_CARD",
"value": 35
},
{
"name": "ONLINE",
"value": 15
}
]
},
{
"groupBy": "category",
"buckets": [
{
"name": "COMMON_MASTERS_HOARDING",
"value": 50
},
{
"name": "COMMON_MASTERS_ROAD_SHOW",
"value": 40
},
{
"name": "COMMON_MASTERS_UNIPOLLS",
"value": 45
},
{
"name": "COMMON_MASTERS_AUCTION_FEE",
"value": 55
},
{
"name": "COMMON_MASTERS_USER_FEES",
"value": 35
}
]
}
],
"numberOfChallans": [
{
"groupBy": "challanStatus",
"buckets": [
{
"name": "PAID",
"value": 50
},
{
"name": "CANCELLED",
"value": 105
},
{
"name": "ACTIVE",
"value": 35
}
]
},
{
"groupBy": "category",
"buckets": [
{
"name": "COMMON_MASTERS_HOARDING",
"value": 50
},
{
"name": "COMMON_MASTERS_ROAD_SHOW",
"value": 40
},
{
"name": "COMMON_MASTERS_UNIPOLLS",
"value": 45
},
{
"name": "COMMON_MASTERS_AUCTION_FEE",
"value": 20
},
{
"name": "COMMON_MASTERS_USER_FEES",
"value": 35
}
]
}
]
}
}]
}The steps required for the creation of the index and adding the index mapping for the same can be found here - National Dashboard: Steps for Index Creation.
The following section contains module-wise index names, index mapping and ingest curls for ingesting data to national dashboard indexes.
a. Index - pt-national-dashboard
b. Index mapping -
"properties" : {
"assessedPropertiesForUsageCategory" : {
"type" : "long"
},
"assessments" : {
"type" : "long"
},
"cessForUsageCategory" : {
"type" : "long"
},
"createdBy" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"createdTime" : {
"type" : "long"
},
"date" : {
"type" : "date",
"format" : "dd-MM-yyyy HH:mm:ss||dd-MM-yyyy||epoch_millis||dd-MM-yyyy'T'HH:mm:ss.SSSZ"
},
"financialYear" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"interestForUsageCategory" : {
"type" : "long"
},
"lastModifiedBy" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"lastModifiedTime" : {
"type" : "long"
},
"module" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"penaltyForUsageCategory" : {
"type" : "long"
},
"propertiesRegisteredForFinancialYear" : {
"type" : "long"
},
"propertyTaxForUsageCategory" : {
"type" : "long"
},
"rebateForUsageCategory" : {
"type" : "long"
},
"region" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"state" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"todaysClosedApplications" : {
"type" : "long"
},
"noOfPropertiesPaidToday" : {
"type" : "long"
},
"todaysCollectionForUsageCategory" : {
"type" : "long"
},
"todaysTotalApplications" : {
"type" : "long"
},
"transactionsForUsageCategory" : {
"type" : "long"
},
"ulb" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"usageCategory" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"ward" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}c. Ingest curl -
curl --location --request POST 'https://qa.digit.org/national-dashboard/metric/_ingest' \
--header 'Content-Type: application/json' \
--data-raw '{
"RequestInfo": {
"apiId": "asset-services",
"ver": null,
"ts": null,
"action": null,
"did": null,
"key": null,
"msgId": "search with from and to values",
"authToken": "null",
"userInfo": {
"id": 11131,
"uuid": "0a8ef1d4-ef5c-4061-aea5-0ac4a410b87f",
"userName": "NDSS",
"name": "Lata",
"mobileNumber": "7897807878",
"emailId": null,
"locale": null,
"type": "EMPLOYEE",
"roles": [
{
"name": "National Dashboard Admin",
"code": "NATADMIN",
"tenantId": "pg"
}
],
"active": true,
"tenantId": "uk.rishikesh",
"permanentCity": null
}
},
"Data": [
{
"date": "23-03-2022",
"module": "PT",
"ward": "Block 1",
"ulb": "uk.rishikesh",
"region": "Rishikesh-MC",
"state": "Uttarakhand",
"metrics": {
"assessments": 29,
"todaysTotalApplications": 62,
"todaysClosedApplications": 21,
"propertiesRegistered": [
{
"groupBy": "financialYear",
"buckets": [
{
"name": "2018-19",
"value": 12
},
{
"name": "2019-20",
"value": 18
},
{
"name": "2020-21",
"value": 21
}
]
}
],
"assessedProperties": [
{
"groupBy": "usageCategory",
"buckets": [
{
"name": "RESIDENTIAL",
"value": 21
},
{
"name": "COMMERCIAL",
"value": 11
},
{
"name": "INDUSTRIAL",
"value": 13
}
]
}
],
"transactions": [
{
"groupBy": "usageCategory",
"buckets": [
{
"name": "RESIDENTIAL",
"value": 19
},
{
"name": "COMMERCIAL",
"value": 13
},
{
"name": "INDUSTRIAL",
"value": 13
}
]
}
],
"todaysCollection": [
{
"groupBy": "usageCategory",
"buckets": [
{
"name": "RESIDENTIAL",
"value": 16000
},
{
"name": "COMMERCIAL",
"value": 22500
},
{
"name": "INDUSTRIAL",
"value": 26000
}
]
}
],
"propertyTax": [
{
"groupBy": "usageCategory",
"buckets": [
{
"name": "RESIDENTIAL",
"value": 1200
},
{
"name": "COMMERCIAL",
"value": 2100
},
{
"name": "INDUSTRIAL",
"value": 100
}
]
}
],
"cess": [
{
"groupBy": "usageCategory",
"buckets": [
{
"name": "RESIDENTIAL",
"value": 1300
},
{
"name": "COMMERCIAL",
"value": 1900
},
{
"name": "INDUSTRIAL",
"value": 1000
}
]
}
],
"rebate": [
{
"groupBy": "usageCategory",
"buckets": [
{
"name": "RESIDENTIAL",
"value": -500
},
{
"name": "COMMERCIAL",
"value": -1200
},
{
"name": "INDUSTRIAL",
"value": -900
}
]
}
],
"penalty": [
{
"groupBy": "usageCategory",
"buckets": [
{
"name": "RESIDENTIAL",
"value": 1300
},
{
"name": "COMMERCIAL",
"value": 1500
},
{
"name": "INDUSTRIAL",
"value": 1500
}
]
}
],
"interest": [
{
"groupBy": "usageCategory",
"buckets": [
{
"name": "RESIDENTIAL",
"value": 1900
},
{
"name": "COMMERCIAL",
"value": 1800
},
{
"name": "INDUSTRIAL",
"value": 600
}
]
}
]
}
}
]
}'a. Index name - ws-national-dashboard
b. Index mapping -
"properties" : {
"channelType" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"connectionType" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"connectionsCreatedForChannelType" : {
"type" : "long"
},
"connectionsCreatedForConnectionType" : {
"type" : "long"
},
"createdBy" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"createdTime" : {
"type" : "long"
},
"date" : {
"type" : "date",
"format" : "dd-MM-yyyy HH:mm:ss||dd-MM-yyyy||epoch_millis||dd-MM-yyyy'T'HH:mm:ss.SSSZ"
},
"duration" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"lastModifiedBy" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"lastModifiedTime" : {
"type" : "long"
},
"meterType" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"module" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"paymentChannelType" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"pendingConnectionsForDuration" : {
"type" : "long"
},
"region" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"sewerageConnectionsForChannelType" : {
"type" : "long"
},
"sewerageConnectionsForUsageType" : {
"type" : "long"
},
"slaCompliance" : {
"type" : "long"
},
"state" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"taxHeads" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"tenantId" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"todaysClosedApplications" : {
"type" : "long"
},
"todaysCollectionForChannelType" : {
"type" : "long"
},
"todaysCollectionForConnectionType" : {
"type" : "long"
},
"todaysCollectionForPaymentChannelType" : {
"type" : "long"
},
"todaysCollectionForTaxHeads" : {
"type" : "long"
},
"todaysCollectionForUsageType" : {
"type" : "long"
},
"todaysCompletedApplicationsWithinSLA" : {
"type" : "long"
},
"todaysTotalApplications" : {
"type" : "long"
},
"transactions" : {
"type" : "long"
},
"ulb" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"usageType" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"ward" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"waterConnectionsForChannelType" : {
"type" : "long"
},
"waterConnectionsForMeterType" : {
"type" : "long"
},
"waterConnectionsForUsageType" : {
"type" : "long"
}
}c. Ingest curl -
curl --location --request POST 'https://qa.digit.org/national-dashboard/metric/_ingest' \
--header 'Content-Type: application/json' \
--data-raw '{
"RequestInfo": {
"apiId": "asset-services",
"ver": null,
"ts": null,
"action": null,
"did": null,
"key": null,
"msgId": "search with from and to values",
"authToken": "{{access_token}}",
"userInfo": {
"id": 11131,
"uuid": "0a8ef1d4-ef5c-4061-aea5-0ac4a410b87f",
"userName": "NDSS",
"name": "Lata",
"mobileNumber": "7897807878",
"emailId": null,
"locale": null,
"type": "EMPLOYEE",
"roles": [
{
"name": "National Dashboard Admin",
"code": "NATADMIN",
"tenantId": "pg"
}
],
"active": true,
"tenantId": "pb.amritsar",
"permanentCity": null
}
},
"Data": [
{
"date": "20-02-2022",
"module": "WS",
"ward": "Ward 1",
"ulb": "pb.amritsar",
"region": "Amritsar-MC",
"state": "Punjab",
"metrics": {
"transactions": 2345,
"connectionsCreated": [
{
"groupBy": "connectionType",
"buckets": [
{
"name": "WATER.METERED",
"value": 110
},
{
"name": "WATER.NONMETERED",
"value": 50
},
{
"name": "SEWERAGE",
"value": 50
}
]
},
{
"groupBy": "channelType",
"buckets": [
{
"name": "Counter",
"value": 60
},
{
"name": "Online",
"value": 70
},
{
"name": "System",
"value": 50
},
{
"name": "CSC",
"value": 30
}
]
}
],
"todaysCollection": [
{
"groupBy": "usageType",
"buckets": [
{
"name": "Domestic",
"value": 11000
},
{
"name": "Commercial",
"value": 5000
},
{
"name": "Institutional",
"value": 5600
},
{
"name": "Domestic SLC",
"value": 7200
},
{
"name": "Domestic Exempted",
"value": 3800
},
{
"name": "Commercial Motor",
"value": 4800
}
]
},
{
"groupBy": "paymentChannelType",
"buckets": [
{
"name": "System",
"value": 8000
},
{
"name": "Paytm",
"value": 7000
},
{
"name": "Field",
"value": 3700
},
{
"name": "Razorpay",
"value": 2200
},
{
"name": "PayU",
"value": 1000
},
{
"name": "BBPS",
"value": 1400
},
{
"name": "POS",
"value": 5100
},
{
"name": "Sewakendra",
"value": 5600
},
{
"name": "Freecharge",
"value": 3400
}
]
},
{
"groupBy": "taxHeads",
"buckets": [
{
"name": "INTEREST",
"value": 7000
},
{
"name": "LATE.CHARGES",
"value": 8500
},
{
"name": "ADVANCE",
"value": 3700
},
{
"name": "CURRENT.CHARGES",
"value": 12422
},
{
"name": "ARREAR.CHARGES",
"value": 5778
}
]
},
{
"groupBy": "connectionType",
"buckets": [
{
"name": "WATER.METERED",
"value": 18700
},
{
"name": "WATER.NONMETERED",
"value": 8500
},
{
"name": "SEWERAGE",
"value": 10200
}
]
}
],
"sewerageConnections": [
{
"groupBy": "channelType",
"buckets": [
{
"name": "ONLINE",
"value": 30
},
{
"name": "CSC",
"value": 11
},
{
"name": "SYSTEM",
"value": 9
}
]
},
{
"groupBy": "usageType",
"buckets": [
{
"name": "Domestic",
"value": 11
},
{
"name": "Commercial",
"value": 10
},
{
"name": "Residential",
"value": 14
},
{
"name": "Institutional",
"value": 6
},
{
"name": "Domestic Exempted",
"value": 9
}
]
}
],
"waterConnections": [
{
"groupBy": "channelType",
"buckets": [
{
"name": "Counter",
"value": 47
},
{
"name": "ONLINE",
"value": 53
},
{
"name": "CSC",
"value": 15
},
{
"name": "SYSTEM",
"value": 45
}
]
},
{
"groupBy": "usageType",
"buckets": [
{
"name": "Domestic",
"value": 123
},
{
"name": "Commercial",
"value": 22
},
{
"name": "Domestic Exempted",
"value": 15
}
]
},
{
"groupBy": "meterType",
"buckets": [
{
"name": "METERED",
"value": 110
},
{
"name": "NON.METERED",
"value": 50
}
]
}
],
"pendingConnections": [
{
"groupBy": "duration",
"buckets": [
{
"name": "0to3Days",
"value": 11
},
{
"name": "3to7Days",
"value": 50
},
{
"name": "7to15Days",
"value": 5
},
{
"name": "MoreThan15Days",
"value": 2
}
]
}
],
"slaCompliance": 24,
"todaysTotalApplications": 35,
"todaysClosedApplications": 33,
"todaysCompletedApplicationsWithinSLA": 46
}
}
]
}'a. Index name - firenoc-national-dashboard
b. Index mapping -
"properties" : {
"actualNOCIssuedByDeptForDepartment" : {
"type" : "long"
},
"actualNOCIssuedForDepartment" : {
"type" : "long"
},
"actualNOCIssuedForUsageType" : {
"type" : "long"
},
"applicationType" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"avgDaysToIssueActualNOCForDepartment" : {
"type" : "long"
},
"avgDaysToIssueProvisionalNOCForDepartment" : {
"type" : "long"
},
"createdBy" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"createdTime" : {
"type" : "long"
},
"date" : {
"type" : "date",
"format" : "dd-MM-yyyy HH:mm:ss||dd-MM-yyyy||epoch_millis"
},
"department" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"lastModifiedBy" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"lastModifiedTime" : {
"type" : "long"
},
"module" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"nocIssuedTodayForType" : {
"type" : "long"
},
"paymentMode" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"provisionalNOCIssuedForDepartment" : {
"type" : "long"
},
"region" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"slaComplianceActualForDepartment" : {
"type" : "long"
},
"slaComplianceProvisionalForDepartment" : {
"type" : "long"
},
"state" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"tenantId" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"todaysApplicationsForApplicationType" : {
"type" : "long"
},
"todaysApplicationsForDepartment" : {
"type" : "long"
},
"todaysClosedApplications" : {
"type" : "long"
},
"todaysCollectionForDepartment" : {
"type" : "long"
},
"todaysCollectionForPaymentMode" : {
"type" : "long"
},
"todaysCompletedApplicationsWithinSLA" : {
"type" : "long"
},
"type" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"ulb" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"usageType" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"ward" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}c. Ingest curl -
curl --location --request POST 'https://qa.digit.org/national-dashboard/metric/_ingest' \
--header 'Content-Type: application/json' \
--data-raw '{
"RequestInfo": {
"apiId": "asset-services",
"ver": null,
"ts": null,
"action": null,
"did": null,
"key": null,
"msgId": "search with from and to values",
"authToken": "81ee8cb1-713c-40b3-a935-d9ae66d547af",
"userInfo": {
"id": 11131,
"uuid": "0a8ef1d4-ef5c-4061-aea5-0ac4a410b87f",
"userName": "NDSS",
"name": "Lata",
"mobileNumber": "7897807878",
"emailId": null,
"locale": null,
"type": "EMPLOYEE",
"roles": [
{
"name": "National Dashboard Admin",
"code": "NATADMIN",
"tenantId": "pg"
}
],
"active": true,
"tenantId": "pb.amritsar",
"permanentCity": null
}
},
"Data": [
{
"date": "25-02-2022",
"module": "FIRENOC",
"ward": "Ward 1",
"ulb": "pb.amritsar",
"region": "Amritsar-MC",
"state": "Punjab",
"metrics": {
"todaysClosedApplications": 18,
"todaysCompletedApplicationsWithinSLA": 32,
"todaysApplications": [
{
"groupBy": "applicationType",
"buckets": [
{
"name": "ACTUAL",
"value": 30
},
{
"name": "PROVISIONAL",
"value": 54
}
]
},
{
"groupBy": "department",
"buckets": [
{
"name": "DEPARTMENT1",
"value": 35
},
{
"name": "DEPARTMENT2",
"value": 49
}
]
}
],
"todaysCollection": [
{
"groupBy": "paymentMode",
"buckets": [
{
"name": "CASH",
"value": 5465
},
{
"name": "CHEQUE",
"value": 1300
},
{
"name": "DEBIT_CARD",
"value": 1400
},
{
"name": "ONLINE",
"value": 7570
}
]
},
{
"groupBy": "department",
"buckets": [
{
"name": "DEPARTMENT1",
"value": 11767
},
{
"name": "DEPARTMENT2",
"value": 3968
}
]
}
],
"nocIssuedToday": [
{
"groupBy": "type",
"buckets": [
{
"name": "ACTUAL",
"value": 13
},
{
"name": "PROVISIONAL",
"value": 65
}
]
}
],
"provisionalNOCIssued": [
{
"groupBy": "department",
"buckets": [
{
"name": "DEPARTMENT1",
"value": 30
},
{
"name": "DEPARTMENT2",
"value": 23
}
]
}
],
"actualNOCIssued": [
{
"groupBy": "department",
"buckets": [
{
"name": "DEPARTMENT1",
"value": 56
},
{
"name": "DEPARTMENT2",
"value": 43
}
]
},
{
"groupBy": "usageType",
"buckets": [
{
"name": "GROUP_A_RESIDENTIAL.SUBDIVISIONA-1",
"value": 20
},
{
"name": "GROUP_B_EDUCATIONAL.SUBDIVISIONB-1",
"value": 30
},
{
"name": "GROUP_C_INSTITUTIONAL.SUBDIVISIONC-1",
"value": 50
},
{
"name": "GROUP_A_RESIDENTIAL.SUBDIVISIONA-2",
"value": 60
},
{
"name": "GROUP_B_EDUCATIONAL.SUBDIVISIONB-2",
"value": 70
},
{
"name": "GROUP_C_INSTITUTIONAL.SUBDIVISIONC-2",
"value": 80
}
]
}
],
"avgDaysToIssueProvisionalNOC": [
{
"groupBy": "department",
"buckets": [
{
"name": "DEPARTMENT1",
"value": 54
},
{
"name": "DEPARTMENT2",
"value": 43
}
]
}
],
"slaComplianceActual": [
{
"groupBy": "department",
"buckets": [
{
"name": "DEPARTMENT1",
"value": 65
},
{
"name": "DEPARTMENT2",
"value": 33
}
]
}
],
"slaComplianceProvisional": [
{
"groupBy": "department",
"buckets": [
{
"name": "DEPARTMENT1",
"value": 23
},
{
"name": "DEPARTMENT2",
"value": 34
}
]
}
],
"avgDaysToIssueActualNOC": [
{
"groupBy": "department",
"buckets": [
{
"name": "DEPARTMENT1",
"value": 43
},
{
"name": "DEPARTMENT2",
"value": 12
}
]
}
]
}
}
]
}'a. Index name - pgr-national-dashboard
b. Index mapping -
"properties" : {
"averageSolutionTimeForDepartment" : {
"type" : "long"
},
"category" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"channel" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"closedComplaints" : {
"type" : "long"
},
"completionRate" : {
"type" : "long"
},
"completionRateForDepartment" : {
"type" : "long"
},
"createdBy" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"createdTime" : {
"type" : "long"
},
"date" : {
"type" : "date",
"format" : "dd-MM-yyyy HH:mm:ss||dd-MM-yyyy||epoch_millis||dd-MM-yyyy'T'HH:mm:ss.SSSZ"
},
"department" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"lastModifiedBy" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"lastModifiedTime" : {
"type" : "long"
},
"module" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"region" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"resolvedComplaints" : {
"type" : "long"
},
"slaAchievement" : {
"type" : "long"
},
"slaAchievementForDepartment" : {
"type" : "long"
},
"state" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"status" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"todaysAssignedComplaintsForDepartment" : {
"type" : "long"
},
"todaysClosedComplaintsForDepartment" : {
"type" : "long"
},
"todaysComplaintsForCategory" : {
"type" : "long"
},
"todaysComplaintsForChannel" : {
"type" : "long"
},
"todaysComplaintsForDepartment" : {
"type" : "long"
},
"todaysComplaintsForStatus" : {
"type" : "long"
},
"todaysOpenComplaintsForDepartment" : {
"type" : "long"
},
"todaysReassignRequestedComplaintsForDepartment" : {
"type" : "long"
},
"todaysReassignedComplaintsForDepartment" : {
"type" : "long"
},
"todaysRejectedComplaintsForDepartment" : {
"type" : "long"
},
"todaysReopenedComplaintsForDepartment" : {
"type" : "long"
},
"todaysResolvedComplaintsForDepartment" : {
"type" : "long"
},
"ulb" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"uniqueCitizens" : {
"type" : "long"
},
"ward" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}c. Ingest curl -
curl --location --request POST 'https://qa.digit.org/national-dashboard/metric/_ingest' \
--header 'Content-Type: application/json' \
--data-raw '{
"RequestInfo": {
"apiId": "asset-services",
"ver": null,
"ts": null,
"action": null,
"did": null,
"key": null,
"msgId": "search with from and to values",
"authToken": "{{access_token}}",
"userInfo": {
"id": 11131,
"uuid": "0a8ef1d4-ef5c-4061-aea5-0ac4a410b87f",
"userName": "NDSS",
"name": "Lata",
"mobileNumber": "7897807878",
"emailId": null,
"locale": null,
"type": "EMPLOYEE",
"roles": [
{
"name": "National Dashboard Admin",
"code": "NATADMIN",
"tenantId": "pg"
}
],
"active": true,
"tenantId": "pb.amritsar",
"permanentCity": null
}
},
"Data": [
{
"date": "20-02-2022",
"module": "WS",
"ward": "Ward 1",
"ulb": "pb.amritsar",
"region": "Amritsar-MC",
"state": "Punjab",
"metrics": {
"transactions": 2345,
"connectionsCreated": [
{
"groupBy": "connectionType",
"buckets": [
{
"name": "WATER.METERED",
"value": 110
},
{
"name": "WATER.NONMETERED",
"value": 50
},
{
"name": "SEWERAGE",
"value": 50
}
]
},
{
"groupBy": "channelType",
"buckets": [
{
"name": "Counter",
"value": 60
},
{
"name": "Online",
"value": 70
},
{
"name": "System",
"value": 50
},
{
"name": "CSC",
"value": 30
}
]
}
],
"todaysCollection": [
{
"groupBy": "usageType",
"buckets": [
{
"name": "Domestic",
"value": 11000
},
{
"name": "Commercial",
"value": 5000
},
{
"name": "Institutional",
"value": 5600
},
{
"name": "Domestic SLC",
"value": 7200
},
{
"name": "Domestic Exempted",
"value": 3800
},
{
"name": "Commercial Motor",
"value": 4800
}
]
},
{
"groupBy": "paymentChannelType",
"buckets": [
{
"name": "System",
"value": 8000
},
{
"name": "Paytm",
"value": 7000
},
{
"name": "Field",
"value": 3700
},
{
"name": "Razorpay",
"value": 2200
},
{
"name": "PayU",
"value": 1000
},
{
"name": "BBPS",
"value": 1400
},
{
"name": "POS",
"value": 5100
},
{
"name": "Sewakendra",
"value": 5600
},
{
"name": "Freecharge",
"value": 3400
}
]
},
{
"groupBy": "taxHeads",
"buckets": [
{
"name": "INTEREST",
"value": 7000
},
{
"name": "LATE.CHARGES",
"value": 8500
},
{
"name": "ADVANCE",
"value": 3700
},
{
"name": "CURRENT.CHARGES",
"value": 12422
},
{
"name": "ARREAR.CHARGES",
"value": 5778
}
]
},
{
"groupBy": "connectionType",
"buckets": [
{
"name": "WATER.METERED",
"value": 18700
},
{
"name": "WATER.NONMETERED",
"value": 8500
},
{
"name": "SEWERAGE",
"value": 10200
}
]
}
],
"sewerageConnections": [
{
"groupBy": "channelType",
"buckets": [
{
"name": "ONLINE",
"value": 30
},
{
"name": "CSC",
"value": 11
},
{
"name": "SYSTEM",
"value": 9
}
]
},
{
"groupBy": "usageType",
"buckets": [
{
"name": "Domestic",
"value": 11
},
{
"name": "Commercial",
"value": 10
},
{
"name": "Residential",
"value": 14
},
{
"name": "Institutional",
"value": 6
},
{
"name": "Domestic Exempted",
"value": 9
}
]
}
],
"waterConnections": [
{
"groupBy": "channelType",
"buckets": [
{
"name": "Counter",
"value": 47
},
{
"name": "ONLINE",
"value": 53
},
{
"name": "CSC",
"value": 15
},
{
"name": "SYSTEM",
"value": 45
}
]
},
{
"groupBy": "usageType",
"buckets": [
{
"name": "Domestic",
"value": 123
},
{
"name": "Commercial",
"value": 22
},
{
"name": "Domestic Exempted",
"value": 15
}
]
},
{
"groupBy": "meterType",
"buckets": [
{
"name": "METERED",
"value": 110
},
{
"name": "NON.METERED",
"value": 50
}
]
}
],
"pendingConnections": [
{
"groupBy": "duration",
"buckets": [
{
"name": "0to3Days",
"value": 11
},
{
"name": "3to7Days",
"value": 50
},
{
"name": "7to15Days",
"value": 5
},
{
"name": "MoreThan15Days",
"value": 2
}
]
}
],
"slaCompliance": 24,
"todaysTotalApplications": 35,
"todaysClosedApplications": 33,
"todaysCompletedApplicationsWithinSLA": 46
}
}
]
}'a. Index name - tl-national-dashboard
b. Index mapping -
"properties" : {
"adhocPenalty" : {
"type" : "long"
},
"adhocRebate" : {
"type" : "long"
},
"applicationsMovedTodayForStatus" : {
"type" : "long"
},
"createdBy" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"createdTime" : {
"type" : "long"
},
"date" : {
"type" : "date",
"format" : "dd-MM-yyyy HH:mm:ss||dd-MM-yyyy||epoch_millis||dd-MM-yyyy'T'HH:mm:ss.SSSZ"
},
"lastModifiedBy" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"lastModifiedTime" : {
"type" : "long"
},
"module" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"region" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"state" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"status" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"tlTax" : {
"type" : "long"
},
"todaysApplications" : {
"type" : "long"
},
"todaysCollectionForTradeType" : {
"type" : "long"
},
"todaysLicenseIssuedWithinSLA" : {
"type" : "long"
},
"todaysTradeLicensesForStatus" : {
"type" : "long"
},
"tradeType" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"transactions" : {
"type" : "long"
},
"ulb" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"ward" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}c. Ingest curl -
curl --location --request POST 'https://qa.digit.org/national-dashboard/metric/_ingest' \
--header 'Content-Type: application/json' \
--data-raw '{
"RequestInfo": {
"apiId": "asset-services",
"ver": null,
"ts": null,
"action": null,
"did": null,
"key": null,
"msgId": "search with from and to values",
"authToken": "{{access_token_loc}}",
"userInfo": {
"id": 11131,
"uuid": "0a8ef1d4-ef5c-4061-aea5-0ac4a410b87f",
"userName": "NDSS",
"name": "Lata",
"mobileNumber": "7897807878",
"emailId": null,
"locale": null,
"type": "EMPLOYEE",
"roles": [
{
"name": "National Dashboard Admin",
"code": "NATADMIN",
"tenantId": "pg"
}
],
"active": true,
"tenantId": "uk.kedarnath",
"permanentCity": null
}
},
"Data": [
{
"date": "29-12-2021",
"module": "TL",
"ward": "Ward 2",
"ulb": "uk.kedarnath",
"region": "Kedarnath-MC",
"state": "Uttarakhand",
"metrics": {
"transactions": 29,
"todaysApplications": 179,
"tlTax": 50000,
"adhocPenalty": 6000,
"adhocRebate": 2000,
"todaysLicenseIssuedWithinSLA": 41,
"todaysCollection": [
{
"groupBy": "tradeType",
"buckets": [
{
"name": "BRICKFIELD",
"value": 21000
},
{
"name": "GROCERYSTORES",
"value": 20000
},
{
"name": "CHARCOAL_KLIN",
"value": 13000
}
]
}
],
"todaysTradeLicenses": [
{
"groupBy": "status",
"buckets": [
{
"name": "INITIATED",
"value": 31
},
{
"name": "APPLIED",
"value": 56
},
{
"name": "FIELDINSPECTION",
"value": 32
},
{
"name": "PENDINGAPPROVAL",
"value": 44
},
{
"name": "PENDINGPAYMENT",
"value": 29
},
{
"name": "APPROVED",
"value": 23
},
{
"name": "REJECTED",
"value": 1
},
{
"name": "EXPIRED",
"value": 1
}
]
}
],
"applicationsMovedToday": [
{
"groupBy": "status",
"buckets": [
{
"name": "INITIATED",
"value": 31
},
{
"name": "APPLIED",
"value": 56
},
{
"name": "FIELDINSPECTION",
"value": 32
},
{
"name": "PENDINGAPPROVAL",
"value": 44
},
{
"name": "PENDINGPAYMENT",
"value": 29
},
{
"name": "APPROVED",
"value": 23
},
{
"name": "REJECTED",
"value": 1
},
{
"name": "EXPIRED",
"value": 1
}
]
}
]
}
}
]
}'a. Index name - mcollect-national-dashboard
b. Index mapping -
"properties" : {
"category" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"challanStatus" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"createdBy" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"createdTime" : {
"type" : "long"
},
"date" : {
"type" : "date",
"format" : "dd-MM-yyyy HH:mm:ss||dd-MM-yyyy||epoch_millis||dd-MM-yyyy'T'HH:mm:ss.SSSZ"
},
"lastModifiedBy" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"lastModifiedTime" : {
"type" : "long"
},
"module" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"numberOfCategories" : {
"type" : "long"
},
"numberOfChallansForCategory" : {
"type" : "long"
},
"numberOfChallansForChallanStatus" : {
"type" : "long"
},
"numberOfReceiptsForCategory" : {
"type" : "long"
},
"numberOfReceiptsForPaymentMode" : {
"type" : "long"
},
"numberOfReceiptsForStatus" : {
"type" : "long"
},
"paymentMode" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"region" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"state" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"status" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"todaysCollectionForCategory" : {
"type" : "long"
},
"todaysCollectionForPaymentMode" : {
"type" : "long"
},
"todaysCollectionForStatus" : {
"type" : "long"
},
"ulb" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"ward" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}c. Ingest curl -
curl --location --request POST 'https://qa.digit.org/national-dashboard/metric/_ingest' \
--header 'Content-Type: application/json' \
--data-raw '{
"RequestInfo": {
"apiId": "asset-services",
"ver": null,
"ts": null,
"action": null,
"did": null,
"key": null,
"msgId": "search with from and to values",
"authToken": "{{access_token_loc}}",
"userInfo": {
"id": 11131,
"uuid": "0a8ef1d4-ef5c-4061-aea5-0ac4a410b87f",
"userName": "NDSS",
"name": "Lata",
"mobileNumber": "7897807878",
"emailId": null,
"locale": null,
"type": "EMPLOYEE",
"roles": [
{
"name": "National Dashboard Admin",
"code": "NATADMIN",
"tenantId": "pg"
}
],
"active": true,
"tenantId": "pb.amritsar",
"permanentCity": null
}
},
"Data": [
{
"date": "29-10-2021",
"module": "MCOLLECT",
"ward": "Ward 1",
"ulb": "pb.amritsar",
"region": "Amritsar-MC",
"state": "Punjab",
"metrics": {
"numberOfCategories": 9,
"todaysCollection": [
{
"groupBy": "paymentMode",
"buckets": [
{"name": "UPI","value":1765},{"name": "CASH","value":2415},{"name": "DEBIT_CARD","value":165}
]
},
{
"groupBy": "status",
"buckets": [
{"name": "NEW","value":1835},{"name": "DEPOSITED","value":1425},{"name": "DISHONOURED","value":1085}
]
},
{
"groupBy": "category",
"buckets": [
{"name": "COMMON_MASTERS_HOARDING","value":135},{"name": "COMMON_MASTERS_ROAD_SHOW","value":805},{"name": "COMMON_MASTERS_UNIPOLLS","value":795},{"name": "COMMON_MASTERS_AUCTION_FEE","value":655},{"name": "COMMON_MASTERS_USER_FEES","value":495},{"name": "COMMON_MASTERS_MUNICIPAL_SHOPS_RENT","value":305},{"name": "COMMON_MASTERS_PARKING_FEE","value":555},{"name": "COMMON_MASTERS_TOWER_ANNUAL_RENT","value":505},{"name": "COMMON_MASTERS_TOWER_INSTALLATION","value":95}
]
}
],
"numberOfReceipts": [
{
"groupBy": "status",
"buckets": [
{"name": "NEW","value":174},{"name": "DEPOSITED","value":284},{"name": "DISHONOURED","value":187}
]
},
{
"groupBy": "paymentMode",
"buckets": [
{"name": "CASH","value":163},{"name": "CHEQUE","value":140},{"name": "DEBIT_CARD","value":156},{"name": "ONLINE","value":186}
]
},
{
"groupBy": "category",
"buckets": [
{"name": "COMMON_MASTERS_HOARDING","value":56},{"name": "COMMON_MASTERS_ROAD_SHOW","value":51},{"name": "COMMON_MASTERS_UNIPOLLS","value":82},{"name": "COMMON_MASTERS_AUCTION_FEE","value":77},{"name": "COMMON_MASTERS_USER_FEES","value":46},{"name": "COMMON_MASTERS_MUNICIPAL_SHOPS_RENT","value":66},{"name": "COMMON_MASTERS_PARKING_FEE","value":63},{"name": "COMMON_MASTERS_TOWER_ANNUAL_RENT","value":103},{"name": "COMMON_MASTERS_TOWER_INSTALLATION","value":101}
]
}
],
"numberOfChallans": [
{
"groupBy": "challanStatus",
"buckets": [
{"name": "PAID","value":151},{"name": "CANCELLED","value":41},{"name": "ACTIVE","value":341}
]
},
{
"groupBy": "category",
"buckets": [
{"name": "COMMON_MASTERS_HOARDING","value":45},{"name": "COMMON_MASTERS_ROAD_SHOW","value":70},{"name": "COMMON_MASTERS_UNIPOLLS","value":71},{"name": "COMMON_MASTERS_AUCTION_FEE","value":65},{"name": "COMMON_MASTERS_USER_FEES","value":75},{"name": "COMMON_MASTERS_MUNICIPAL_SHOPS_RENT","value":30},{"name": "COMMON_MASTERS_PARKING_FEE","value":61},{"name": "COMMON_MASTERS_TOWER_ANNUAL_RENT","value":56},{"name": "COMMON_MASTERS_TOWER_INSTALLATION","value":60}
]
}
]
}
}
]
}'a. Index name - obps-national-dashboard
b. Index mapping -
"mappings" : {
"nss" : {
"properties" : {
"applicationsSubmitted" : {
"type" : "long"
},
"applicationsWithDeviation" : {
"type" : "long"
},
"averageDaysToIssueOC" : {
"type" : "long"
},
"averageDaysToIssuePermit" : {
"type" : "long"
},
"averageDeviation" : {
"type" : "long"
},
"createdBy" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"createdTime" : {
"type" : "long"
},
"date" : {
"type" : "date",
"format" : "dd-MM-yyyy HH:mm:ss||dd-MM-yyyy||epoch_millis||dd-MM-yyyy'T'HH:mm:ss.SSSZ"
},
"landAreaAppliedInSystemForBPA" : {
"type" : "long"
},
"lastModifiedBy" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"lastModifiedTime" : {
"type" : "long"
},
"module" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"ocIssued" : {
"type" : "long"
},
"ocPlansScrutinized" : {
"type" : "long"
},
"ocSubmitted" : {
"type" : "long"
},
"ocWithDeviation" : {
"type" : "long"
},
"occupancyType" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"paymentMode" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"permitsIssuedForOccupancyType" : {
"type" : "long"
},
"permitsIssuedForRiskType" : {
"type" : "long"
},
"permitsIssuedForSubOccupancyType" : {
"type" : "long"
},
"plansScrutinized" : {
"type" : "long"
},
"region" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"riskType" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"slaComplianceOC" : {
"type" : "long"
},
"slaCompliancePermit" : {
"type" : "long"
},
"state" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"subOccupancyType" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"tenantId" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"todaysClosedApplicationsOC" : {
"type" : "long"
},
"todaysClosedApplicationsPermit" : {
"type" : "long"
},
"todaysCollectionForPaymentMode" : {
"type" : "long"
},
"todaysCompletedApplicationsWithinSLAOC" : {
"type" : "long"
},
"todaysCompletedApplicationsWithinSLAPermit" : {
"type" : "long"
},
"ulb" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"ward" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
}
}c. Ingest curl -
curl --location --request POST 'https://qa.digit.org/national-dashboard/metric/_ingest' \
--header 'Content-Type: application/json' \
--data-raw '{
"RequestInfo": {
"apiId": "asset-services",
"ver": null,
"ts": null,
"action": null,
"did": null,
"key": null,
"msgId": "search with from and to values",
"authToken": "7112c6a3-41d4-42d0-9ae9-7d2ec08d1a4f"
},
"Data": [ {
"date": "05-01-2022",
"module": "OBPS",
"ward": "Ajit Nagar",
"ulb": "pb.amritsar",
"region": "amritsar-MC",
"state": "Punjab",
"metrics": {
"ocPlansScrutinized": 120,
"plansScrutinized": 540,
"ocSubmitted": 50,
"applicationsSubmitted": 50,
"ocIssued": 19,
"landAreaAppliedInSystemForBPA": 23442,
"averageDaysToIssuePermit": 10,
"averageDaysToIssueOC": 8,
"todaysClosedApplicationsOC":10 ,
"todaysCompletedApplicationsWithinSLAOC":5 ,
"todaysClosedApplicationsPermit": 20,
"todaysCompletedApplicationsWithinSLAPermit":10,
"slaComplianceOC": 20,
"slaCompliancePermit": 40,
"applicationsWithDeviation": 20,
"averageDeviation": 10,
"ocWithDeviation": 30,
"todaysCollection": [
{
"groupBy": "paymentMode",
"buckets": [
{
"name": "UPI",
"value": 10000
},
{
"name": "DEBIT.CARD",
"value": 15000
},
{
"name": "CREDIT.CARD",
"value": 8500
}
]
}
],
"permitsIssued": [
{
"groupBy": "riskType",
"buckets": [
{
"name": "LOW",
"value": 150
},
{
"name": "MEDIUM",
"value": 300
},
{
"name": "HIGH",
"value": 600
}
]
} ,
{
"groupBy": "occupancyType",
"buckets": [
{
"name": "RESIDENTIAL",
"value": 150
},
{
"name": "INSTITUTIONAL",
"value": 180
}
]
},
{
"groupBy": "subOccupancyType",
"buckets": [
{
"name": "RESIDENTIAL.INDIVIDUAL",
"value": 50
},
{
"name": "RESIDENTIAL.SHARED",
"value": 20
},
{
"name": "INSITUTIONAL.SHARED",
"value": 120
}
]
}
]
}
}
]
}'a. Index name - common-national-dashboard
b. Index mapping -
"properties" : {
"createdBy" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"createdTime" : {
"type" : "long"
},
"date" : {
"type" : "date",
"format" : "dd-MM-yyyy HH:mm:ss||dd-MM-yyyy||epoch_millis||dd-MM-yyyy'T'HH:mm:ss.SSSZ"
},
"lastModifiedBy" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"lastModifiedTime" : {
"type" : "long"
},
"liveUlbsCountForServiceModuleCode" : {
"type" : "long"
},
"module" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"onboardedUlbsCount" : {
"type" : "long"
},
"region" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"serviceModuleCode" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"slaAchievement" : {
"type" : "long"
},
"state" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"status" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"totalCitizensCount" : {
"type" : "long"
},
"totalLiveUlbsCount" : {
"type" : "long"
},
"totalUlbCount" : {
"type" : "long"
},
"ulb" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"ward" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}c. Ingest curl -
curl --location --request POST 'https://qa.digit.org/national-dashboard/metric/_ingest' \
--header 'Content-Type: application/json' \
--data-raw '{
"RequestInfo": {
"apiId": "asset-services",
"ver": null,
"ts": null,
"action": null,
"did": null,
"key": null,
"msgId": "search with from and to values",
"authToken": "ac024e05-2b0e-4341-b294-681c5d72fcaf",
"userInfo": {
"id": 11131,
"uuid": "0a8ef1d4-ef5c-4061-aea5-0ac4a410b87f",
"userName": "NDSS",
"name": "Lata",
"mobileNumber": "7897807878",
"emailId": null,
"locale": null,
"type": "EMPLOYEE",
"roles": [
{
"name": "National Dashboard Admin",
"code": "NATADMIN",
"tenantId": "pg"
}
],
"active": true,
"tenantId": "pb.amritsar",
"permanentCity": null
}
},
"Data": [
{
"date": "06-02-2022",
"module": "COMMON",
"ward": "Ward 1",
"ulb": "pb.amritsar",
"region": "Amritsar-MC",
"state": "Punjab",
"metrics": {
"status": "Live",
"onboardedUlbsCount": 12,
"totalCitizensCount": 1250,
"totalLiveUlbsCount": 15,
"totalUlbCount": 22,
"slaAchievement": 75,
"liveUlbsCount": [
{
"groupBy": "serviceModuleCode",
"buckets": [
{
"name": "PT",
"value": 12
},
{
"name": "TL",
"value": 10
},
{
"name": "FIRENOC",
"value": 8
},
{
"name": "PGR",
"value": 22
},
{
"name": "WS",
"value": 12
}
]
}
]
}
}
]
}'password
eGov@123
username
SYSTEMSU1
for Upyog
username_state
SYSTEMSU3
for staging
token
ZWdvdi11c2VyLWNsaWVudDo=
tenantid
pg
usertype
SYSYTEM
totalulb_url
for reading the ulb’s of punjab
Body: The body consists of 2 parts: RequestInfo and Data. Data is where the aggregated data to be ingested resides. The keys given under the metrics object here are metrics provided in module.fields.mapping present in the configuration here - DIGIT-DevOps/qa.yaml at master · egovernments/DIGIT-DevOps
