Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Browse through DIGIT core service configuration details
Create and modify workflow configuration
Each service integrated with egov-workflow-v2 service needs to first define the workflow configuration which describes the workflow states, the action that can be taken on these states, the user roles that can perform those actions, SLAs etc. This configuration is created using APIs and is stored in the DB. The configuration can be created at either the state level or the tenant level based on the requirements.
Before you proceed with the configuration, make sure the following pre-requisites are met -
egov-workflow-v2 service is up and running
Role action mapping is added for the BusinessService APIs
Create and modify workflow configuration
Configure state level as well BusinessService level SLA
Control access to workflow actions from the configuration
Validates if the flow defined in the configuration is complete during the creation
Deploy the latest version of egov-workflow-v2 service.
Add role action mapping for BusinessService APIs (preferably add _create and update only for SUPERUSER. Search can be added for CITIZEN and required employee roles like TL__CEMP etc.
Overwrite the egov.wf.statelevel flag (true for state level and false for tenant level).
Add businessService persister yaml path in persister configuration.
Create the businessService JSON based on product requirements. Following is a sample json of a simple 2-step workflow where an application can be applied by a citizen or counter employee and then can be either rejected or approved by the approver.
Once the businessService json is created add it in the request body of _create API of workflow and call the API to create the workflow.
To update the workflow first search the workflow object using _search API and then make changes in the businessService object and then call _update using the modified search result.
States cannot be removed using _update API as it will leave applications in that state in an invalid state. In such cases first, all the applications in that state should be moved forward or backward state and then the state should be disabled through DB directly.
The workflow configuration can be used by any module which performs a sequence of operations on an application/entity. It can be used to simulate and track processes in organisations to make them more efficient and increase accountability.
Integrating with workflow service provides a way to have a dynamic workflow configuration which can be easily modified according to the changing requirements. The modules don’t have to deal with any validations regarding workflow such as authorisation of the user to take an action if documents are required to be uploaded at a certain stage etc. since it will be automatically handled by egov-workflow-v2 service based on the defined configuration. It also automatically keeps updating SLAs for all applications which provide a way to track the time taken by an application to get processed.
To integrate, the host of egov-workflow-v2
should be overwritten in the helm chart
/egov-workflow-v2/egov-wf/businessservice/_search
should be added as the endpoint for searching workflow configuration. (Other endpoints are not required once workflow configuration is created)
The configuration can be fetched by calling
_search
API
Note: All the APIs are in the same Postman collection therefore the same link is added in each row.
Configure workflows as per requirements
Workflows are a series of steps that moves a process from one state to another state by actions performed by different kind of Actors - Humans, Machines, Time based events etc. to achieve a goal like onboarding an employee, or approve an application or granting a resource etc. The egov-workflow-v2 is a workflow engine which helps in performing these operations seamlessly using a predefined configuration.
Before you proceed with the documentation, make sure the following pre-requisites are met -
Java 8
Kafka server is up and running
egov-persister service is running and has a workflow persister config path added to it
PSQL server is running and a database is created to store workflow configuration and data
Always allow anyone with a role in the workflow state machine to view the workflow instances and comment on it
On the creation of workflow, it will appear in the inbox of all employees that have roles that can perform any state transitioning actions in this state.
Once an instance is marked to an individual employee it will appear only in that employee's inbox although point 1 will still hold true and all others participating in the workflow can still search it and act if they have the necessary action available to them
If the instance is marked to a person who cannot perform any state transitioning action, they can still comment/upload and mark to anyone else.
Overall SLA: SLA for the complete processing of the application/Entity
State-level SLA: SLA for a particular state in the workflow
Deploy the latest version of egov-workflow-v2 service
Add businessService persister yaml path in persister configuration
Add Role-Action mapping for BusinessService APIs
Overwrite the egov.wf.statelevel flag ( true for state level and false for tenant level)
Create businessService (workflow configuration) according to product requirements
Add Role-Action mapping for /processInstance/_search API
Add workflow persister yaml path in persister configuration
The workflow configuration can be used by any module which performs a sequence of operations on an application/Entity. It can be used to simulate and track processes in organisations to make it more efficient too and increase accountability.
Role-based workflow
An easy way of writing rule
File movement within workflow roles
To integrate, the host of egov-workflow-v2 should be overwritten in the helm chart.
/process/_search should be added as the search endpoint for searching workflow process Instance objects.
/process/_transition should be added to perform an action on an application. (It’s for internal use in modules and should not be added in Role-Action mapping).
The workflow configuration can be fetched by calling _search API to check if data can be updated or not in the current state.
Note: All the APIs are in the same Postman collection therefore the same link is added in each row.
Configure user data management services
User service is responsible for user data management and providing functionality to login and logout into the DIGIT system
Before you proceed with the configuration, make sure the following pre-requisites are met -
Java 8
The Kafka server is up and running
Encryption and MDMS services are running
PSQL server is running
The Redis database is running
Store, update and search user data
Provide authentication
Provide login and logout functionality into the DIGIT platform
Store user data PIIs in encrypted form
Setup the latest version of egov-enc-service and egov-mdms- service
Deploy the latest version of egov-user service
Add role-action mapping for APIs
The following application properties file in user service are configurable.
User data management and functionality to log in and log out into the DIGIT system using OTP and password.
Providing the following functionalities to citizen and employee-type users
Employee:
User registration
Search user
Update user details
Forgot password
Change password
User role mapping(Single ULB to multiple roles)
Enable employees to login into the DIGIT system based on a password.
Citizen:
Create user
Update user
Search user
User registration using OTP
OTP based login
To integrate, the host of egov-user should be overwritten in the helm chart.
Use /citizen/_create
endpoint for creating users into the system. This endpoint requires the user to validate his mobile number using OTP. First, the OTP is sent to the user's mobile number and then the OTP is sent as otpReference
in the request body.
Use /v1/_search
and /_search
endpoints to search users in the system depending on various search parameters.
Use /profile/_update
for updating the user profile. The user is validated (either through OTP-based validation or password validation) when this API is called.
/users/_createnovalidate
and /users/_updatenovalidate
are endpoints to create user data into the system without any validations (no OTP or password required). They should be strictly used only for creating/updating users internally and should not be exposed outside.
Forgot password: In case the user forgets the password it can be reset by first calling /user-otp/v1/_send
which generates and sends OTP to the employee’s mobile number. The password is then updated using this OTP by calling the API /password/nologin/_update
in which a new password along with the OTP is sent.
Use /password/_update
to update the existing password by logging in. Both old and new passwords are sent to the request body. Details of the API can be found in the attached swagger documentation.
Use /user/oauth/token
for generating tokens, /_logout
for logout and /_details
for getting user information from the token.
Multi-Tenant User: The multi-tenant user functionality allows users to perform actions across multiple ULBs. For example, employees belonging to Amritsar can perform the role of say Trade License Approver for Jalandhar by assigning them the tenant-level role of tenantId pb.jalandhar.
Following is an example of the user:
If an employee has a role with statelevel tenantId
they can perform actions corresponding to that role across all tenants.
Refresh Token: Whenever the /user/oauth/token
is called to generate the access_token
along with access_token,
one more token is generated called refresh_token
. The refresh token is used to generate a new access_token
whenever the existing one expires. Till the time the refresh token is valid, users will not have to log in even if their access_token
expires since this is generated using refresh_token
. The validity time of the refresh token is configurable and can be configured using the property: refresh.token.validity.in.minutes
There are two security policy models for user data - User and UserSelf.
In model User, the field attributes
contains a list of fields from the user object that needs to be secured and the field roleBasedDecryptionPolicy
is an attribute-level role-based policy. It defines visibility for each attribute.
UserSelf model contains the same structure of security policy but the User security model is used for Search API response and UserSelf is used for Create/Update API response.
The visibility of the PII data is based on the above MDMS configuration. There are three types of visibility mentioned in the config.
PLAIN - Show text in plain form.
MASKED - The returned text contains masked data. The masking pattern is applied as defined in the Masking Patterns master data.
NONE - The returned text does not contain any data. It contains strings like “Confidential Information”.
Any user can get plain access to the secured data (citizen’s PII) by requesting through the plainAccessRequest
parameter. It takes the following parameters:
recordId
- It is the unique identifier of the record that is requested for plain access.
fields
- It defines a list of attributes that are requested for plain access.
Configure role based user access and map actions to roles
DIGIT is an API-based platform where each API denotes a DIGIT resource. The primary job of Access Control Service (ACS) is to authorise end-user based on their roles and provide access to the DIGIT platform resources. Access control functionality is essentially based on below points:
Actions: Actions are events which are performed by a user. This can be an API end-point or front-end event. This is the MDMS master.
Roles: Roles are assigned to users. A single user can hold multiple roles. Roles are defined in MDMS masters.
Role-Action: Role actions are mapped between Actions and Roles. Based on roles, the action mapping access control service identifies applicable actions for the role.
Before you proceed with the configuration, make sure the following pre-requisites are met -
Java 8
MDMS service is up and running
Serve the applicable actions for a user based on user roles (To print menu three).
On each action performed by a user, access control looks at the roles for the user and validates actions mapping with the role.
Support tenant-level role action - For instance, an employee from Amritsar can have the role of APPROVER for other ULBs like Jalandhar and hence will be authorised to act as APPROVER in Jalandhar.
Deploy the latest version of the Access Control Service
Deploy MDMS service to fetch the Role Action Mappings
Define the roles:
Add the actions (URL)
Add the role action mapping
Any microservice which requires authorisation can leverage the functionalities provided by the access control service.
Any new microservice that is to be added to the platform will not have to worry about authorisation. It can just add its role action mapping in the master data and the Access Control Service will perform authorisation whenever the API for the microservice is called.
To integrate with Access Control Service the role action mapping has to be configured (added) in the MDMS service.
The service needs to call /actions/_authorize API of Access Control Service to check for authorisation of any request.
Configure bulk generation of PDF documents
The objective of the PDF generation service is to bulk generate pdf as per requirement.
Before proceeding with the documentation, ensure the following prerequisites are met:
Install npm
Ensure the Kafka server is operational
Confirm that the egov-persister service is running and configured with a pdf generation persister path
Verify that the PostgreSQL (PSQL) server is running, and a database is created to store filestore IDs and job IDs of generated PDFs
Provide a common framework to generate PDFs
Provide flexibility to customise the PDF as per the requirement
Provide functionality to add an image or QR code in a PDF
Provide functionality to generate PDFs in bulk
Provide functionality to specify the maximum number of records to be written in one PDF
Create data config and format config for a PDF according to product requirements.
Add data config and format config files in PDF configuration.
Add the file path of data and format config in the environment yaml file.
Deploy the latest version of pdf-service in a particular environment.
The PDF configuration can be used by any module which needs to show particular information in PDF format that can be printed/downloaded by the user.
Functionality to generate PDFs in bulk.
Avoid regeneration.
Support QR codes and Images.
Functionality to specify the maximum number of records to be written in one PDF.
Uploading generated PDF to filestore and return filestore id for easy access.
To download and print the required PDF _create API has to be called with the required key (For Integration with UI, please refer to the links in Reference Docs)
Note: All the APIs are in the same Postman collection, therefore, the same link is added in each row.
Configure workflows for a new product
Workflow is defined as a sequence of tasks that has to be performed on an application/Entity to process it. The egov-workflow-v2
is a workflow engine which helps in performing these operations seamlessly using a predefined configuration. We will discuss how to create this configuration for a new product in this document.
Before you proceed with the configuration, make sure the following pre-requisites are met -
egov-workflow-v2
service is up and running
Role action mapping is added for business service APIs
Create and modify workflow configuration according to the product requirements
Configure State level as well BusinessService level SLA to efficiently track the progress of the application
Control access to perform actions through configuration
Attributes | Description |
---|
Deploy the latest version of the egov-workflow-v2 service.
Add businessService persister yaml path in persister configuration.
Add role action mapping for BusinessService APIs.
Overwrite the egov.wf.statelevel flag (true for state level and false for tenant level).
The Workflow configuration has 3 levels of hierarchy:
BusinessService
State
Action
The top-level object is BusinessService which contains fields describing the workflow and a list of States that are part of the workflow. The businessService can be defined at the tenant level like pb.amritsar or at the state level like pb. All objects maintain an audit sub-object which keeps track of who is creating and updating and the time of it.
Each state object is a valid status for the application. The State object contains information about the state and what actions can be performed on it.
The action object is the last object in the hierarchy, it defines the name of the action and the roles that can perform the action.
The workflow should always start from the null state as the service treats new applications as having null as the initial state. eg:
In the action object whatever nextState is defined, the application will be sent to that state. It can be to another forward state or even some backward state from where the application has already passed (generally, such actions are named SENDBACK)
SENDBACKTOCITIZEN is a special keyword for an action name. This action sends back the application to the citizen’s inbox for him to take action. A new State should be created on which Citizen can take action and should be the nextState of this action. While calling this action from the module assignees should be enriched by the module with the UUIDs of the owners of the application
Note: All the APIs are in the same Postman collection therefore the same link is added in each row.
Configure escalation flows based on predefined criteria
The objective of this functionality is to provide a mechanism to trigger action on applications which satisfy certain predefined criteria.
Looking at sample use cases provided by the product team, the majority of use cases can be summarised as performing action ‘X’ on applications in state ‘Y’ and have exceeded the state SLA by ‘Z’ days. We can write one query builder which takes this state ‘Y’ and SLA exceeded by ‘Z’ as search params and then we can perform action X on the search response. This has been achieved by defining an MDMS config like below:
In the above configuration, we define the condition for triggering the escalation of applications. The above configuration triggers escalation for applications in RESOLVED
state which have exceeded stateSLA by more than 1.0
day and this triggers the escalation by performing CLOSERESOLVEDCOMPLAIN
on the applications. Once the applications are escalated the processInstances are pushed on the pgr-auto-escalation
topic. We have done a sample implementation for pgr-services, where we have updated the persister configuration to listen on this topic and update the complaint status accordingly.
The auto escalation for businessService PGR
will be triggered when the following API is called:
Note: The businessService is a path param. (For example, if the escalation has to be done for tl-services NewTL workflow the URL will be 'http://egov-workflow-v2.egov:8080/egov-workflow-v2/egov-wf/auto/NewTL/_escalate
').
These APIs have to be configured in the cron job config so that they can be triggered periodically as per requirements. Only the user with the role permission AUTO_ESCALATE
can trigger auto escalations. Hence, create the user with statelevel AUTO_ESCALATE
role permission and then add that user in the userInfo of the requestInfo. This step has to be done because the cron job does internal API calls and ZUUL will not enrich the userInfo.
For setting up an auto-escalation trigger, the workflow must be updated. For example, to add an auto escalate trigger on RESOLVED
state with action CLOSERESOLVEDCOMPLAIN
in PGR
businessService, we will have to search the businessService and add the following action in the actions array of RESOLVED
state and call update API.
Suppose an application gets auto-escalated from state ‘X' to state 'Y’, employees can look at these escalated applications through the escalate search API. The following sample cURL can be used to search auto-escalated applications of the PGR module belonging to Amritsar tenant -
Configure service to enable fetch and share of location details
A core application that provides location details of the tenant for which the services are being provided.
Before you proceed with the documentation, make sure the following pre-requisites are met -
Java 8
PSQL server is running and the database is created
Knowledge of egov-mdms service
egov-mdms service is running and all the required MDMS masters are loaded in it
The location information is also known as the boundary data of ULB
Boundary data can be of different hierarchies - ADMIN or ELECTION hierarchy defined by the Administrators and Revenue hierarchy defined by the Revenue department.
The election hierarchy has the locations divided into several types like zone, election wards, blocks, streets and localities. The Revenue hierarchy has the locations divided into a zone, ward, block and locality.
The model which defines the localities like zone, ward and etc is a boundary object which contains information like name, lat, long, parent or children boundary if any. The boundaries come under each other in a hierarchy. For instance, a zone contains wards, a ward contains blocks, and a block contains locality. The order in which the boundaries are contained in each other differs based on the tenants.
Environment Variables | Description |
---|
Add/Update the MDMS master file which contains boundary data of ULBs.
Add Role-Action mapping for the egov-location APIs.
Deploy/Redeploy the latest version of the egov-mdms service.
Fill the above environment variables in the egov-location with proper values.
Deploy the latest version of the egov-location service.
The boundary data has been moved to MDMS from the master tables in DB. The location service fetches the JSON from MDMS and parses it to the structure of the boundary object as mentioned above. A sample master would look like below.
The egov-location APIs can be used by any module which needs to store the location details of the tenant.
Get the boundary details based on boundary type and hierarchy type within the tenant boundary structure.
Get the geographical boundaries by providing appropriate GeoJson.
Get the tenant list in the given latitude and longitude.
To integrate, the host of egov-location should be overwritten in the helm chart.
/boundarys/_search should be added as the search endpoint for searching boundary details based on tenant Id, Boundary Type, Hierarchy Type etc.
/geography/_search should be added as the search endpoint. This method handles all requests related to geographical boundaries by providing appropriate GeoJson and other associated data based on tenantId or lat/long etc.
/tenant/_search should be added as the search endpoint. This method tries to resolve a given lat, long to a corresponding tenant, provided there exists a mapping between the reverse geocoded city to the tenant.
The MDMS tenant boundary master file should be loaded in the MDMS service.
Deploy workflow 2.0 in an environment where workflow is already running
In workflow 2.0 assignee is changed from an object to a list of objects.
To accommodate this change a new table named 'eg_wf_assignee_v2' is added that maps the processInstaceIds to assignee UUIDs. To deploy workflow 2.0 in an environment where workflow is already running assignee column needs to be migrated to the eg_wf_assignee_v2 table.
The following query does this migration:
Persister config for egov-workflow-v2 is updated. Insert query for the table eg_wf_assignee_v2 is added in egov-workflow-v2-persister.yml.
The latest updated config can be referred to from the below link:
The employee inbox has an added column to display the locality of the applications. This mapping of the application number to locality is fetched by calling the searcher API for the respective module. If a new module is integrated with workflow its searcher config should be added in the locality searcher yaml with module code as a name in the definition. All the search URLs and role action mapping details must be added to the MDMS.
The format of the URL is given below:
Sample request for TL:
The searcher yaml can be referred from the below link:
For sending back the application to citizens, the action with the key 'SENDBACKTOCITIZEN' must be added. The exact key should be used. The resultant state of the action should be a new state. If pointing to an existing state the action in that state will be visible to the CITIZEN even when the application reaches the state without Send Back as the workflow is role-based.
To update the businessService for Send Back feature, add the following state and action in the search response at required places and add call businessService update API. This assigns the UUID to the new state and action and creates the required references. The Resubmit action is added as optional action for counter employees to take action on behalf of the citizen.
State json:
Action json:
Each item in the above dropdown is displayed by adding an object in the link below -
For example:
{
"id": 1928,
"name": "rainmaker-common-tradelicence",
"url": "quickAction",
"displayName": "Search TL",
"orderNumber": 2,
"parentModule": "",
"enabled": true,
"serviceCode": "",
"code": "",
"path": "",
"navigationURL": "tradelicence/search",
"leftIcon": "places:business-center",
"rightIcon": "",
"queryParams": ""
}
id , url, displayName, navigationURL
Mandatory properties.
The value of the URL property should be “quickAction” as shown above.
Accordingly, add the role-actions:
{
"rolecode": "TL_CEMP",
"actionid": 1928,
"actioncode": "",
"tenantId": "pb"
}
SLA slots and the background colour of the SLA days remaining on the Inbox page are defined in the MDMS configuration as shown above.
For example: If the maximum SLA is 30 then, it has 3 slots
30 - 30*(1/3) => 20 - 30: will have green colour defined
0 - 20: will have yellow colour defined
<0: will have red colour defined
The colours are also configured in the MDMS.
For API /egov-workflow-v2/egov-wf/process/_transition: The field assignee of type User in ProcessInstance object is changed to a list of 'User' called assignees. User assignee --> List<User> assignees
For Citizen Sendback: When the action SENDBACKTOCITIZEN is called on the entity the module has to enrich the assignees with the UUIDs of the owners and creator of the entity.
Title |
---|
Title |
---|
Environment Variables | Description |
---|
For configuration details, refer to the links in .
Title |
---|
Title |
---|
Property | Value | Remarks |
---|
To know more about the encryption policy, refer to the document docs.
APIs |
---|
The details about the fields in the configuration can be found in the
Title |
---|
Title | Link |
---|
Environment Variables | Description |
---|
PDFMake: ( ):- for generating PDFs.
Mustache.js: ( ):- as templating engine to populate format as defined in format config, from request json based on mappings defined in data config.
For configuration details, refer to
Title |
---|
Title |
---|
For integration-related steps, refer to the document .
Title |
---|
Title |
---|
Attribute Name | Description |
---|
Title |
---|
APIs |
---|
Please refer to the for the location service to understand the structure of APIs and to have a visualisation of all internal APIs.
| The tenantId (ULB code) for which the workflow configuration is defined |
| The name of the workflow |
| The name of the module which uses this workflow configuration |
| The overall SLA to process the application (in milliseconds) |
| Name of the state |
| Status of the application when in the given state |
| Boolean flag representing if document are required to enter the state |
| Boolean flag representing if the state can be used as starting state in workflow |
| Boolean flag representing if the state is the leaf node or end state in the workflow configuration. (No Actions can be taken on states with this flag as true) |
| Boolean flag representing whether data can be updated in the application when taking action on the state |
| The current state on which action can be performed |
| The resultant state after action is performed |
| A list containing the roles which can perform the actions |
| Contains fields to audit edits on the data. (createdTime, createdBy,lastModifiedTIme,lastModifiedby) |
egov.wf.default.offset | The default value of offset in search |
egov.wf.default.limit | The default value of limit in search |
egov.wf.max.limit | The maximum number of records that are returned in search response |
egov.wf.inbox.assignedonly | Boolean flag if set to true default search will return records assigned to the user only, if false it will return all the records based on the user’s role. (default search is the search call when no query params are sent and based on the RequestInfo of the call, records are returned, it’s used to show applications in employee inbox) |
egov.wf.statelevel | Boolean flag set to true if a state-level workflow is required |
| Host name for MDMS service. |
| MDMS Search URL. |
| MDMS module which contain boundary master. |
| MDMS master file which contain boundary detail. |
tenantId | The tenantId (ULB code) for which the boundary data configuration is defined. |
moduleName | The name of the module where TenantBoundary master is present. |
TenantBoundary.hierarchyType.name | Unique name of the hierarchy type. |
TenantBoundary.hierarchyType.code | Unique code of the hierarchy type. |
TenantBoundary.boundary.id | Id of boundary defined for particular hierarchy. |
boundaryNum | Sequence number of boundary attribute defined for the particular hierarchy. |
name | Name of the boundary like Block 1 or Zone 1 or City name. |
localname | Local name of the boundary. |
longitude | Longitude of the boundary. |
latitude | Latitude of the boundary. |
label | Label of the boundary. |
code | Code of the boundary. |
children | Details of its sub-boundaries. |
| 10 | default search record number limit |
| true | whether citizen login otp based |
| false | whether employee login otp based |
| 123456 | fixed otp for citizen |
| false | allow fixed otp for citizen |
| true | whether otp compulsory for registration |
| 10080 | validity time of access token |
| 20160 | validity time of refresh token |
| 90 | expiry date of a password |
| 60 | unlock time |
| 30 | window size for counting attempts for lock |
| 5 | max failed login attempts before account is locked |
| pb |
| [LINK TO BE UPDATED] |
| Maximum number of records to be written in one PDF |
| Date timezone which will be used to convert epoch timestamp into date ( |
| Default value of localisation locale |
| Default value of localisation tenant |
| File path/URL'S of data config |
| File path/URL'S of format config |
Tenant represents a body in a system. In the municipal system, a state and its ULBs (Urban local bodies) are tenants. ULB represents a city or a town in a state. Tenant configuration is done in MDMS.
Before proceeding with the configuration, the following pre-requisites are met -
Knowledge of json and how to write a json is required.
Knowledge of MDMS is required.
User with permission to edit the git repository where MDMS data is configured.
On the login page, city name selection is required. Tenant added in MDMS shows in the city drop-down of the login page.
In reports or in the employee inbox page the details related to ULB are displayed from the fetched ULB data added in MDMS.
Modules i.e., TL, PT, MCS can be enabled based on the requirement of the tenant.
After adding the new tenant, the MDMS service needs to be restarted to read the newly added data.
Tenant is added in tenant.json. In MDMS, file tenant.json, under the tenant folder holds the details of the state and ULBs to be added in that state.
To enable tenants the above data should be pushed in tenant.json file. Here "ULB Grade" and City "Code" are important fields. ULB Grade can have a set of allowed values that determines the ULB type, (Municipal corporation (Nagar Nigam), Municipality (municipal council, municipal board, municipal committee) (Nagar Parishad), etc). City "Code" has to be unique to each tenant. This city-specific code is used in all transactions. Not permissible to change the code. If changed we will lose the data of the previous transactions done.
Naming Convention for Tenants Code
“Code”:“uk.citya” is StateTenantId.ULBTenantName"
"logoId": "https://s3.ap-south-1.amazonaws.com/uk-egov-assets/uk.citya/logo.png", Here the last section of the path should be "/<tenantId>/logo.png". If we use anything else, the logo will not be displayed on the UI. <tenantId> is the tenant code ie “uk.citya”.
Localization should be pushed for ULB grade and ULB name. The format is given below.
Localization for ULB Grade
Localization for ULB Name
Format of localization code for tenant name <MDMS_State_Tenant_Folder_Name>_<Tenants_Fille_Name>_<Tenant_Code> (replace dot with underscore)
Boundary data should be added for the new tenant.
MDMS stands for Master Data Management Service. MDMS is one of the applications in the eGov DIGIT core group of services. This service aims to reduce the time spent by developers on writing codes to store and fetch master data (primary data needed for module functionality ) which doesn’t have any business logic associated with them.
Before you proceed with the configuration, make sure the following pre-requisites are met -
Prior knowledge of Java/J2EE.
Prior knowledge of Spring Boot.
Prior knowledge of REST APIs and related concepts like path parameters, headers, JSON, etc.
Prior knowledge of Git.
Advanced knowledge of how to operate JSON data would be an added advantage to understanding the service.
The MDMS service reads the data from a set of JSON files from a pre-specified location.
It can either be an online location (readable JSON files from online) or offline (JSON files stored in local memory).
The JSON files are in a prescribed format and store the data on a map. The tenantID of the file serves as a key and a map of master data details as values.
Once the data is stored in the map the same can be retrieved by making an API request to the MDMS service. Filters can be applied in the request to retrieve data based on the existing fields of JSON.
For deploying the changes in MDMS data, the service needs to be restarted.
The changes in MDMS data could be adding new data, updating existing data, or deleting it.
The config JSON files to be written should follow the listed rules
The config files should have JSON extension
The file should mention the tenantId, module name, and master name first before defining the data
Example config JSON for “Billing Service”
Master Data Management Service is a core service made available on the DIGIT platform. It encapsulates the functionality surrounding Master Data Management. The service creates, updates and fetches Master Data pertaining to different modules. This eliminates the overhead of embedding the storage and retrieval logic of Master Data into other modules. The functionality is exposed via REST API.
Prior Knowledge of Java/J2EE, Spring Boot, and advanced knowledge of operating JSON data would be an added advantage to understanding the service.
The MDM service reads the data from a set of JSON files from a pre-specified location. It can either be an online location (readable JSON files from online) or offline (JSON files stored in local memory). The JSON files should conform to a prescribed format. The data is stored in a map and tenantID of the file serves as the key.
Once the data is stored in the map the same can be retrieved by making an API request to the MDM service. Filters can be applied in the request to retrieve data based on the existing fields of JSON.
The spring boot application needs lombok extension added to your IDE to load it. Once the application is up and running API requests can be posted to the URL.
The config JSON files to be written should follow the listed rules
The config files should have JSON extension.
The file should mention the tenantId, module name, and master name first before defining the data.
The Master Name in the above sample will be substituted by the actual name of the master data. The array succeeding it will contain the actual data.
Example config JSON for “Billing Service”
APIs are available to create, update and fetch master data pertaining to different modules. Refer to the segment below for quick details.
BasePath:/mdms/v1/[API endpoint] Method
POST /_create
Creates or Updates Master Data on GitHub as JSON files
MDMSCreateRequest
Request Info + MasterDetail — Details of the master data to be created or updated on GitHub.
MasterDetail
MdmsCreateResponse
Response Info
Method
POST /_search
This method fetches a list of masters for a specified module and tenantId.
MDMSCriteriaReq (mdms request) -
Request Info + MdmsCriteria — Details of module and master which need to be searched using MDMS.
MdmsCriteria
MdmsResponse
Response Info + Mdms
MDMS
Common Request/Response/Error Structures:
RequestInfo should be used to carry meta information about the requests to the server as described in the fields below. All DIGIT APIs will use requestinfo as a part of the request body to carry this meta information. Some of this information will be returned back from the server as part of the ResponseInfo in the response body to ensure correlation.
ResponseInfo should be used to carry metadata information about the response from the server. apiId, ver, and msgId in ResponseInfo should always correspond to the same values in the respective request's RequestInfo.
ErrorRes
All DIGIT APIs will return ErrorRes in case of failure which will carry ResponseInfo as metadata and Error object as an actual representation of the error. When the request processing status in the ResponseInfo is ‘FAILED’ the HTTP status code 400 is returned.
Configuring master data for a new module requires creating a new module in the master config file and adding master data. For better organizing, create all the master data files belonging to the module in the same folder. Organizing in the same folder is not mandatory it is based on the moduleName in the master data file.
Before you proceed with the configuration, make sure the following pre-requisites are met -
User with permission to edit the git repository where MDMS data is configured.
These data can be used to validate the incoming data.
After adding the new module data, the MDMS service needs to be restarted to read the newly added data.
The master config file is structured as below. Each key in the master config is a module and each key in the module is a master.
The new module can be added below the existing modules in the master config file.
Please check the link to create a new master Adding New Master
MDMS supports the configuration of data at different levels. While we enable a state there can be data that is common to all the ULBs of the state and data specific to each ULB. The data further can be configured at each module level as state-specific or ULB’s specific.
Before you proceed with the configuration, make sure the following pre-requisites are met -
Prior Knowledge of Java/J2EE.
Prior Knowledge of Spring Boot.
Prior Knowledge of REST APIs and related concepts like path parameters, headers, JSON, etc.
Prior knowledge of Git.
Advanced knowledge of operating JSON data would be an added advantage to understanding the service.
State Level Masters are maintained in a common folder.
ULB Level Masters are maintained in separate folders named after the ULB.
Module Specific State Level Masters are maintained by a folder named after the specific module that is placed outside the common folder.
For deploying the changes(adding new data, updating existing data or deletion) in MDMS, the MDMS service needs to be restarted.
The common master data across all ULBs and modules like department, designation, etc are placed under the common-masters folder which is under the tenant folder of the MDMS repository.
The common master data across all ULBs and are module-specific are placed in a folder named after each module. These folders are placed directly under the tenant folder.
Module data that are specific to each ULB like boundary data, interest, penalty, etc are configured at the ULB level. There will be a folder per ULB under the tenant folder and all the ULB’s module-specific data are placed under this folder.
The Indexer Service runs as a separate service. This service is designed to perform all the indexing tasks of the digit platform. The service reads records posted on specific Kafka topics and picks the corresponding index configuration from the yaml file provided by the respective module. The objective of the Indexer Service is listed below.
To provide a one-stop framework for indexing the data to elasticsearch.
To create provisions for indexing live data, reindexing from one index to the other and indexing legacy data from the data store.
Before you proceed with the configuration, make sure the following pre-requisites are met -
Prior knowledge of Java/J2EE
Prior knowledge of SpringBoot
Prior knowledge of Elasticsearch
Prior knowledge of REST APIs and related concepts like path parameters, headers, JSON etc.
Prior knowledge of Kafka and related concepts like Producer, Consumer, Topic etc.
Performs three major tasks namely: LiveIndex, Reindex and LegacyIndex.
LiveIndex: Task of indexing the live transaction data on the platform. This keeps the es data in sync with the DB.
Reindex: Task of indexing data from one index to the other. ES already provides this feature, the indexer does the same but with data transformation.
LegacyIndex: Task of indexing legacy data from the tables to ES.
Provides flexibility to index the entire object, a part of the object or an entirely different custom object all using one input JSON from modules.
Provides features for customizing index JSON by field mapping, field masking, data enrichment through external APIs and data denormalization using MDMS.
One-stop shop for all the es index requirements with easy-to-write and easy-to-maintain configuration files.
Designed as a consumer to save API overhead. The consumer configs are written from scratch to have complete control over consumer behaviour.
Step 1: Write the configuration as per your requirement. The structure of the config file is explained later in the same doc.
Step 3: Provide the absolute path of the checked-in file to DevOps, to add it to the file-read path of egov-indexer. The files are added to egov-indexer's environment manifest file for it to be read at the start-up of the application.
Step 4: Run the egov-indexer app. Since it is a consumer, it starts listening to the configured topics and indexes the data.
a) POST /{key}/_index
Receive data and index. There should be a mapping with the topic as {key} in index config files.
b) POST /_reindex
This is used to migrate data from one index to another index
c) POST /_legacyindex
This is to run the LegacyIndex job to index data from DB. In the request body, the URL of the service which would be called by the indexer service to pick data must be mentioned.
In legacy indexing and for collection-service records LiveIndex kafka-connect is used to do part of pushing records to elastic search. For more details please refer to the document mentioned in the document list.
eGov Payment Gateway acts as a liaison between eGov apps and external payment gateways facilitating payments, reconciliation of payments and lookup of transactions' status'.
Before you proceed with the documentation, make sure the following pre-requisites are met -
Java 8
Kafka server is up and running
egov-persister service is running and has pg service persister config path added in it
PSQL server is running and the database is created to store transaction data.
Create or initiate a transaction, to make a payment against a bill.
Make payment for multiple bill details [multi module] for a single consumer code at once.
Transaction to initiate a call to the transaction/_create API, various validations are carried out to ensure the sanctity of the request.
The response includes a generated transaction id and a redirect URL to the payment gateway itself.
Various validations are carried out to verify the authenticity of the request and the status is updated accordingly. If the transaction is successful, a receipt is generated for the same.
Reconciliation is carried out by two jobs scheduled via a Quartz clustered scheduler.
The early Reconciliation job is set to run every 15 minutes [configurable via app properties] and is aimed at reconciling transactions which were created 15 - 30 minutes ago and are in a PENDING state.
The daily Reconciliation job is set to run once per day and is aimed at reconciling all transactions that are in a PENDING state, except for ones which were created 30 minutes ago.
Axis, Phonepe and Paytm payment gateways are implemented.
The following properties in the application.properties file in egov-pg-service have to be added and set to default value after integrating with the new payment gateway. In the below table properties for AXIS bank, payment gateway is shown the same relevant property needs to be added for other payment gateways.
Deploy the latest version of egov-pg-service.
Add pg service persister yaml path in persister configuration.
The egov-pg-service acts as communication/contact between eGov apps and external payment gateways.
Record every transaction against a bill.
Record of payment for multiple bill details for a single consumer code at once.
To integrate, the host of egov-pg-service should be overwritten in helm chart
/pg-service/transaction/v1/_create should be added in the module to initiate a new payment transaction, on successful validation
/pg-service/transaction/v1/_update should be added as the update endpoint to updates an existing payment transaction. This endpoint is issued only by payment gateways to update the status of payments. It verifies the authenticity of the request with the payment gateway and forwards all query params received from a payment gateway
/pg-service/transaction/v1/_search should be added as the search endpoint for retrieving the current status of a payment in our system.
(Note: All the APIs are in the same postman collection therefore the same link is added in each row)
Documentation on NLP Chatbot
NLP [Natural Language Processing] is a branch of artificial intelligence that deals with the interactions between humans and computers. The primary task is to interpret the intent of the user (speech or text) and provide the user with the appropriate output. Challenges in Natural language frequently involve speech recognition, audio transcription, natural language understanding and natural language generation.
In this project, we are trying to empower regular chatbots with NLP. This will significantly increase user convenience and in turn, help us to enhance our customer reach. On the other hand, this is a cost-efficient approach as the number of dialogues between the user and the chatbot would also reduce and there would be a huge saving on messaging charges.
This is one of the most insightful use cases of this project. Instead of bombarding the user with subsequent messages consisting of a menu to select from, we can just ask the user to enter his query. Using intent classification and entity recognition, the chatbot can return the appropriate output.
Another area where this concept can be used is complaint classification. Instead of visiting a link and selecting a complaint category from a huge list of complaint categories, the user would at any time prefer to just type in the complaint and leave it to the chatbot to identify the category of the complaint using NLP techniques.
The third use case is regarding the city recognition algorithm. In the existing version of the Punjab UAT chatbot, the process of user locality recognition is a bit inconvenient. The user needs to visit a link, select his/her city from a drop-down menu consisting of around 170 cities, and then return back to WhatsApp to continue the chat further. Using NLP, we can just ask the user to enter his city name and we can detect the user's location using NLP techniques.
Preparing a virtual dataset
One of the major challenges in this project was the absence of a real-time dataset, which is of utmost importance in any NLP-based project. The first phase of the project consisted of recognizing the user intent and classifying whether the user wants to pay bills or he simply wants to retrieve receipts. The idea was to exploit the fact that the inputs corresponding to the intent of paying bills, on average have a certain number of past tense verbs and present tense verbs in the sentence. The same goes for the intent of retrieving the paid bills. The tense of the words was figured out using POS tagging and tokenization in-built functions in nltk.
The sentences present in the brown corpus of the nltk library were segregated into two files namely ‘paid.txt’ and ‘unpaid.txt’ The criteria used for segregation were as follows:
A. If a certain sentence consists of one or more words synonymous with ‘paid’ in its past tense, the sentence is added to ‘paid.txt’.
B. If the sentence consists of one or more words synonymous with ‘paid’ but in the present tense, the sentence is added to ‘unpaid.txt’.
C. If a certain sentence consists of one or more words synonymous with ‘unpaid’ in its past tense, the sentence is added to ‘unpaid.txt’.
D. If a certain sentence consists of one or more words synonymous with ‘unpaid’ in its present tense, the sentence is added to ‘paid.txt’.
Thus, we had our virtual dataset ready. Sentences consisting of negative words such as ‘not’, ‘non’ etc were taken care of in the n-gram analysis, which is explained in the subsequent parts of the documentation.
Classification:
The classifier used here is the Decision Tree Classifier available in the nltk library. ‘paid.txt’ and ‘unpaid.txt’ files were used as training data sets for the model. The following have been used as training features:
1. The number of simple past tense verbs in the sentence.
2. The number of past perfect tense verbs in the sentence.
3. The number of simple present tense verbs in the sentence.
4. The number of past perfect continuous tense verbs in the sentence.
5. The number of present perfect tense verbs in the sentence.
6. The number of words synonymous with ‘paid’.
7. The number of words synonymous with ‘unpaid’.
The classifier was then stored in a pickle file so that the classifier is not trained from scratch every time the program is executed, and hence a lot of compilation time is saved. The run time however remains the same.
Text pre-processing
The following refinements are carried out on the text before sending it through the classifier.
Remove all the punctuations in the sentence, or convert them into a string if possible. For example, convert n’t to not. Other punctuations such as , . / ? ; etc can be removed safely from the string.
Remove stop-words from the sentence. Stop-words in nltk are a list of trivial words like ‘I’, ‘me’, ‘myself’, ‘and’, ‘or’, ‘not’ etc. However, the words ‘are’, ‘to’, ‘be’ and ‘not’ are not removed from the sentence as they are useful in n-gram analysis.
If any character in the sentence appears consecutively more than two times, then remove the extra occurrences and limit the consecutive characters to two. For example, if there is ‘iiii’ present in the string, then remove the extra i’s and convert it to ‘ii’. This is done based on the fact that there are hardly any words in the English vocabulary where there are more than two consecutive repeating characters. Moreover, this also helps to remove the extra spaces in the input.
Convert the entire sentence to lowercase. As we know, Python is case-sensitive and so are the functions in many of the libraries.
The latter half of the text pre-processing comprises Fuzzy String Matching.
This is accomplished using the ratio function in the fuzzy-wuzzy library of Python. Each word in the sentence is checked against each word in the master list. (the master list is a list containing all the keywords used for training the classifier). If the match ratio is above a certain threshold, the word in the sentence is replaced by the word from the master list. Else, the word is replaced by its closest matching word from the English vocabulary.
Intent Recognition
After text pre-processing, the input sentence is checked for words matching with synonyms of ‘quit’. If such a word is found, then the program assumes the user to have entered the bill payment section mistakenly, and the program exits. Else, the program continues to be executed.
The input sentence is then checked for n-grams. N-grams are nothing but a set of n consecutive words in a sentence. For example, the list of 2-grams in the sentence “I want to pay property taxes” will look like [(‘I’, ‘want’), (‘want’, ‘to’), (‘to’, ‘pay’), (‘pay’, ‘property’), (‘property’, ‘taxes’)]. Some of the 2-grams considered for analysis are [ 'not paid', 'not complete', 'not done', 'not given']. Some of the 3-grams considered are ['not to be paid', 'not to be given']. It has been observed that the n-grams dominate the user intent more than the tenses of the verbs. Hence, the n-gram analysis is done prior to classification. If the text does not contain any relevant n-grams, the text is then classified using the Decision Tree Classifier.
Entity Recognition
Entity recognition is all about finding the entity mentioned in the user’s input statement. For example, in the statement “I want to pay property taxes”, the entity mentioned is ‘property’. At this point, we are dealing with three entities, namely ‘water and sewerage’, ‘property’, ‘and trade license’. Every word in the user's input is checked with every word in the list of entities. If the fuzzy match ratio between these words is above a certain threshold, the entity is marked. Then, out of all the marked entities, the entity with the highest fuzz ratio is chosen. For example, if the threshold is 60 and the fuzzy match ratios for the entities are as follows:
Water and sewerage = 65 Property= 75
Trade license= 70
Then ‘Property’ will be identified as the entity which the user is talking about. If none of the entities has a match ratio above the threshold, then the program prompts the user to re-enter the entity. For example, if the threshold in the above example is set to 80, then we see that none of the entities actually has a match ratio above 80 and hence the user is prompted to re-enter his input. The concept of fuzzy match ratio is used for dealing with spelling mistakes in the user’s input.
Audio transcription
Audio transcription is the process of converting an audio file to written text. Several pre-trained models are available in various languages in Python for audio transcription. In this project, I have used pydub and speech recognition libraries. The audio message sent on WhatsApp is transferred in the form of .ogg files. However, the Python libraries in use accept only .wav files. So, initially, we convert the received .ogg files to .wav files. The second important step is Audio Segmentation. Audio Segmentation is the process of breaking the audio files into chunks based on the pause between two words in the sentence. This helps to analyze the meaning of the sentence precisely and increases the accuracy of the system. In this way, the audio message is converted to plain text and further analysis is carried out in the same way as mentioned above.
Bill payment chatbot working:
Complaint classification chatbot working:
In the existing version of the chatbot, the user has to select his/her city from a drop-down menu by visiting the mSeva website. This significantly reduces user convenience as the user is required to constantly switch pages. Thus, we have implemented an algorithm which uses fuzzy matching and pattern recognition to recognize the city input by the user. A list comprising all the city names in English, Punjabi and Hindi was used as a reference tool for this project. Based on the user input, the cities having the highest match ratio with the input are being returned as the output list to select from. The user should then enter the option number of his /her desired city. If the user does not find their city in the list, he/she can go back to the main menu and start all over again. This API works in Hindi and Punjabi as well. A one-time integration was done with WhatsApp using the GupShup platform. Here are some snapshots of the type and search API.
As evident from the above discussions, we are now able to:
Classify user intent and recognize entities for facilitating bill payments/ receipt retrievals in multiple languages.
Categorize user complaints into appropriate categories based on the keywords present in the user input.
Locate the user city by using fuzzy string matching and pattern recognition requiring minimum efforts from the user’s side and the bot being implemented in English, Hindi and Punjabi as well.
Respond appropriately to audio messages from the user regarding bill payment queries.
However, the project has some limitations too. They are as follows:
Due to the scarcity of data, rigorous testing could not be performed and tangible accuracy metrics could not be produced.
The city recognition algorithm works on the basis of the information provided by the MDMS database. However, there are some anomalies in the data and some missing cities as well. Due to these reasons, the algorithm is not yet foolproof.
The audio transcription feature could not be implemented in regional languages due to some limitations in Python modules such as scarce data corpus in regional languages and a lack of pre-trained models in Indian languages.
Configure master data management service
The MDMS service aims to reduce the time spent by developers on writing codes to store and fetch master data (primary data needed for module functionality) which doesn’t have any business logic associated with them. Instead of writing APIs and creating tables in different services to store and retrieve data that is seldom changed, the MDMS service keeps them in a single location for all modules and provides data on demand with the help of no more than three lines of configuration.
Prior knowledge of Java/J2EE
Prior knowledge of Spring Boot
Prior knowledge of REST APIs and related concepts like path parameters, headers, JSON etc.
Prior knowledge of Git
Advanced knowledge of how to operate JSON data would be an added advantage to understanding the service
Adds master data for usage without the need to create master data APIs in every module.
Reads data from GIT directly with no dependency on any database services.
Environment Variables | Description |
---|---|
Deploy the latest version of the Mdms-service
Add conf path for the file location
Add master config JSON path
The MDMS service provides ease of access to master data for any service.
No time spent writing repetitive codes with no business logic.
To integrate, the host of egov-mdms-service should be overwritten in the helm chart
egov-mdms-service/v1/_search
should be added as the search endpoint for searching master data.
MDMS client from eGov snapshots should be added as mvn entity in pom.xml for ease of access since it provides MDMS request pojos.
Learn how to setup DIGIT master data.
Whenever any user logs an authorization token, a refresh token is generated for the user. Using the auth token the client can make rest API calls to the server to fetch data. The auth token has an expiry period. Once the authorization token expires, it cannot be used to make API calls. The client has to generate a new authorization token. This is done by authenticating the refresh token with the server which then generates and sends a new authorization token to the client. The refresh token avoids the need for the client to log in again whenever the authorization token expires.
Refresh token also has an expiry period and once it gets expired it cannot be used to generate new authorization tokens. The user has to log in again to get a new pair of authorization tokens and refresh tokens. Generally, the duration before the expiry of the refresh token is more as compared to that of authorization tokens. If the user logs out of the account both authorization tokens and refresh tokens become invalid.
Variables to configure expiry time:
Indexer uses a config file per module to store all the configurations pertaining to that module. The Indexer reads multiple such files at start-up to support indexing for all the configured modules. In config, we define source and, destination elastic search index names, custom mappings for data transformation and mappings for data enrichment.
Below is the sample configuration for indexing TL application creation data into elastic search.
The table below lists the key configuration variables.
For creating a new master in MDMS, create the JSON file with the master data and configure the newly created master in the master config file.
Before proceeding with the configuration, make sure the following pre-requisites are met -
User with permission to edit the git repository where MDMS data is configured.
After adding the new master, the MDMS service needs to be restarted to read the newly added data.
The new JSON file needs to contain 3 keys as shown in the below code snippet. The new master can be created either State-wise or ULB-wise. Tenant ID and config in the master config file determine this.
The master config file is structured as below. Each key in the master config is a module and each key in the module is a master.
Each master contains the following data and the keys are self-explanatory
In the existing version of the chatbot, for PGR complaint creation feature, the user has to select his/her city from a drop-down menu by visiting the mseva website. This significantly reduces user convenience as the user is required to constantly switch pages. To overcome the above inconvenience, nlp-engine service is used. The service has an algorithm that uses fuzzy matching and pattern recognition to recognise the city provided by the user as input. Based on the user input, the cities having the highest match ratio with the input are being returned as the output list. A list comprising all the city names in English, Punjabi and Hindi was used as a reference tool for this service.
Before you proceed with the documentation, make sure the following pre-requisites are met -
Python.
egov-mdms service is running and all the data related to the service are added in the mdms repository.
egov-running service is running.
Provides city fuzzy search feature which returns the list of cities having the highest match ratio with the input.
City fuzzy search can support input data in English, Hindi and Punjabi language.
Provides locality fuzzy search feature which returns the list of localities having the highest match ratio with the input.
Environment Variables | Description |
---|
Deploy the latest version of nlp-engine service.
Whitelist the city and locality fuzzy search API’s.
The nlp-engine service is used to locate the user city and locality by using fuzzy string matching and pattern recognition.
Currently integrated into the chatbots for locating user city and locality for complaint creation use case.
This feature functionality can be extended for the other entities and can be used for a fuzzy search of those different entities.
To integrate, the host of nlp-engine service module should be overwritten in the helm chart.
/nlp-engine/fuzzy/city
should be added as the fuzzy search endpoint for a city search.
/nlp-engine/fuzzy/locality
should be added as the fuzzy search endpoint for locality search.
(Note: All the APIs are in the same postman collection therefore same link is added in each row)
Goal: To onboard developers onto the XState-Chatbot code base so that they can modify existing flows or create new ones.
This document sticks to explaining the chatbot's core features and does not dive into the use cases implemented by the chatbot.
This chatbot handles basic form-filling within a chat flow by collecting user information and making API calls to the rainmaker backend services to meet user needs. It utilizes StateCharts, similar to State Machines, to manage the user's state in the chat flow and store provided information. XState, a JavaScript implementation of StateCharts, is employed to code all chat flows.
This chatbot does not have any Natural Language Processing component. In future, we have plans to extend the chatbot to add such features.
XState is a JavaScript implementation of StateCharts. There is detailed documentation available to study XState. Some of the XState concepts used in the Chatbot are listed below. Basic knowledge of these concepts is necessary. It can also be learned while going through the chat flow implementation of pilot use cases in PGR and Bills.
Actions
onEntry
Below are some tips for using XState, which have been followed throughout the pilot chat flows:
When transitioning to a state that is not at the same hierarchical level, assign a unique ID value. If the state has an ID, address it using the # qualifier in the target attribute.
Ensure that each state has a unique ID value to avoid duplicates. Having duplicate IDs can cause unexpected behavior in the application.
Surround any actions, such as onEntry, with 'assign' for effective handling of state changes and updates.
This includes almost all functions except the guard condition code snippets.
NodeJS
PostgreSQL
Kafka(optional)
Build a chat flow to facilitate a user to interact with rainmaker modules
Link a chat flow with back-end services
Deploy the latest version of xstate-chatbot
Configure /xstate-chatbot to be a whitelisted open endpoint in zuul
Below are some guidelines for coding chat interactions using XState:
Interaction with the user, such as sending a message or processing incoming messages, should be coded as states in the State Machine.
It's recommended to test any chat flow using the provided react-app. Follow the instructions in the README file of the react-app for testing.
Follow standard patterns when coding chat interactions. These patterns are explained below and can also be studied by browsing through the code of pilot use cases like PGR and Bills.
Chat states should only contain dialogue-specific code. Backend service-related code should be written in separate files, such as ...-service.js.
Code that doesn't involve asynchronous API calls can be written within the onEntry function or action.
If an API call is needed, follow the invoke-onDone pattern. Write asynchronous functions in separate service files, and process the consolidated data returned by these functions in the dialogue file's state.
Helper functions should be written in the dialog.js file. It's advisable to use these functions instead of writing custom logic in dialogue files whenever possible.
Apart from the chat flow and its backend service API calls, a few other components are present in the project. These components do NOT need to be modified to code any new chat flow or changed for an existing chat flow. These components with a short description for each are listed below:
Session Manager: It manages the sessions of all the users on a server. It stores, updates and reads the user’s state in a datastore. Based on the state of the user, it creates a state machine and sends the incoming message event to the state machine. It sanctifies the state data (any sensitive data like the name and mobile number of a user are removed) before storing it in the datastore.
Repository: It is the datastore that stores the user states. To reduce dependency, an in-memory repository is also provided configured using environment variables. So to run the chatbot service, Although PostgreSQL is not a strict requirement, using it as the repository provider is recommended.
Channel Provider: It supports multiple WhatsApp Providers. Any one of the providers is configured for use. A separate console
WhatsApp Provider is present for the developer to test the chatbot server locally. Additionally, a Postman collection is provided to simulate receiving messages from users to the server..
Localisation: Messages to be sent to the user are stored within the chatbot. Here, the localisation service is not used. The messages are typically present near the bottom of the dialogue files. A separate localisation-service.js is provided to retrieve the messages for the localisation codes not owned by the chatbot. For example, the PGR complaint types data is under the ownership of the PGR module, and the messages for such can be fetched from the egov-localization-service using the functions provided in the localization-service.js.
Service Provider: Simplifies initial dialogue development by allowing configuration of a dummy service instead of coding API calls to backend services. This can be set up via environment variables and modifying the service-loader.js file.
Telemetry: Chatbot logs telemetry events to a Kafka topic with the provision to mask any sensitive data before indexing the events onto ElasticSearch by egov-indexer. Below are the list of events that get logged:
Incoming message
Outgoing message
State transitions
This document aims to facilitate communication between the software developers and whoever is localising the chatbot messages. The goal is to make it clear and unambiguous.
to access the sheet containing all messages with the localisation codes.
The project is organised such that all the messages are contained within the files present inside the /machine directory. /service directory, which is present inside it, also includes files that could contain localization messages.
Guidelines to be followed by developers:
(According to the standard pattern followed in the project, all the localization messages will be present near the end of the file in a JavaScript object named “messages”.)
Developers will be the ones first filling up the sheet with codes (and the English version of the messages). Below are the guidelines to be followed when writing the codes in the sheet:
The standard separator to be used is . (dot)
The first part is the filename—Eg: “pgr.” when the filename is pgr.js.
Use “service.” as a prefix when the file is present inside the /service directory.
In the /service directory, filenames are like egov-pgr.js
For localization messages contained in those files, instead of writing “egov-pgr” just write “pgr”
So the prefix for such files would be “service.pgr.”
All the message bundles would be present in the “messages” object near the end of the file. They have been organized in a pattern in the JS object like fileComplaint.complaintType2Step.category.question
The corresponding localization code for such a message bundle in the sheet would be “pgr.fileComplaint.complaintType2Step.category.question”, where the first “pgr.” is added as the prefix for the file name.
Once the localisation codes have been written correctly (and the English version of the messages) in the sheet, it is easy to add the new message in the corresponding new column.
Some guidelines to follow when adding new messages:
The parameter names are written within {{}} (double curly brackets).
The content inside these curly brackets should be written in English even when writing messages for any new language.
The URL shortening service is used to shorten long URLs. There may be a requirement when we want to avoid sending very long URLs to the user via SMS, WhatsApp etc. This service compresses the URL.
Before you proceed with the configuration, make sure the following pre-requisites are met -
Prior knowledge of Java/J2EE
Prior knowledge of SpringBoot
Prior knowledge of REST APIs and related concepts like path parameters, headers, JSON etc.
Compress long URLs
Converted short URLs contain id, which is used by this service to identify and get longer URLs.
Environment Variable | Description |
---|
Deploy the latest version of the URL Shortening service
POST /egov-url-shortening/shortener
Receive long URLs and converts them to shorter URLs. Shortened URLs contain URLs to the endpoint mentioned next. When a user clicks on shortened URL, the user is redirected to a long URL.
GET /{id}
This shortened URL contains the path to this endpoint. The service uses the id used in the last endpoint to get the long URL. In response, the user is redirected to the long URL.
The is a revamped version of the , which provides functionality to the user to access PGR module services like filing complaints, tracking complaints and notifications from WhatsApp. It allows the user to view receipts and pay bills for Property, Trade Licence, Fire NOC, Water and Sewerage and BPA service module.
File PGR complaint
Track PGR complaint
Support images when filing complaints
Notifications to citizens when an employee performs any action on the complaint
Allow users to search and pay bills of different modules.
Allow users to search and view receipts of different modules.
Allow users to change the language of their choice to have a better experience.
Put user interactions on an elastic search for telemetry.
The XState chatbot can be integrated with any other module to improve the ease of searching and viewing bills/past payment receipts and to improve speed and convenience for bill payments. It can be integrated with the PGR module for easiness of creation and tracking of the complaint.
Increase in convenience and ease of making the bill payment.
Increase in no. of users opting for online payment.
Improvement in demand collection efficiency
Creating an additional channel for payment.
Remove dependency on mobile/web apps or counter.
Integration of New Whatsapp Provider
Whatsapp provider is a third-party service that acts as an intermediary between the user WhatsApp client and the XState-Chatbot server. All messages coming/going to/from the user pass through the WhatsApp provider. The chatbot calls the WhatsApp provider to send messages to the user. When a user responds to any WhatsApp message the WhatsApp provider calls the Chatbot service configured endpoint with details like messages sent by the user, the sender number etc.
If any new WhatsApp provider is to be used with a chatbot, code must be written to convert the provider’s incoming messages to the format that the chatbot understands. The final output from the chatbot is also converted to the WhatsApp provider’s API request format.
Currently, the XState-Chatbot service is using ValueFirst as the WhatsApp Provider. This will require provider-specific environment variables to be configured. If the provider changes then, all these environment variables will also change. A few of those environment variables are stored as secrets, so these values need to be configured in env-secrets.yaml.
As this is a revamped version of the chatbot service, all of the secrets should already be present. There is no need to create new secrets.
Configuration of PGR version in chatbot
pgrVersion
pgrUpdateTopic
To configure PGR v2 in XState chatbot then pgrVersion should be ‘v2' and pgrUpdateTopic should be 'update-pgr-request’.
Adding Information Images in PGR Complaint Creation and Open Search Information Image
To configure the filestoreid for an informational image follow the steps mentioned below
Download the images from the section Information Images for PGR and Open Search
For example:
a) if supportedLocales: process.env.SUPPORTED_LOCALES || 'en_IN,hi_IN' then valuefirst-notification-resolved-templateid: "12345,6789"
b) if supportedLocales: process.env.SUPPORTED_LOCALES || 'hi_IN,en_IN' then valuefirst-notification-resolved-templateid: "6789,12345"
(Note: Both lists should not be empty - they must contain at least one element)
Template messages with buttons are maintained in the same way as described in the previous section (Configuration of push notification template messages)
There are two types of button message
Quick Reply
Call To Action
The Value First document below provides more details.
Configuration of module for Bill payment and Receipt search
For example:
If the application modules are defined in the variable - bill-supported-modules: "WS, PT, TL" -
the defined modules appear in the bill payment and receipt search. In the given example, the modules Water and Sewerage, Property Tax, and Trade License appear for bill payment and receipt search.
Configuration of Telemetry File
Cron job mdms entry:
Information images for PGR and Open Search
Description |
---|
Title | Description |
---|---|
Description |
---|
Description |
---|
Input Field | Description | Mandatory | Data Type |
---|---|---|---|
Input Field | Description | Mandatory | Data Type |
---|---|---|---|
Input Field | Description | Mandatory | Data Type |
---|---|---|---|
Input Field | Description | Mandatory | Data Type |
---|---|---|---|
Output Field | Description | Mandatory | Data Type |
---|---|---|---|
Output Field | Description | Mandatory | Data Type |
---|---|---|---|
Description |
---|
ex: ///common-masters/ Here “pb” is the tenant folder name.
ex: ///TradeLicense/ Here “pb” is the tenant folder name and “TradeLicense“ is the module name.
ex: ////TradeLicense/ Here “amritsar“ is the ULB name and “TradeLicense“ is the module name. All the data specific to this module for the ULB are configured inside this folder.
Description |
---|
Description |
---|
Step 2: Check in the config file to a remote location preferably Github. Currently, we check the files into this folder -for dev
Click here to access the details.
Additional gateways can be added by implementing the interface. No changes are required to the core packages.
Property | Remarks |
---|
Title |
---|
Title |
---|
Title |
---|
Title |
---|
Param | Description |
---|
API | Description |
---|
Variable Name | Descriptions |
---|
Description |
---|
Add mdms configs required for nlp-engine service () and restart mdms service.
Title |
---|
APIs |
---|
Add to the egov-indexer to index all the telemetry messages
Environment Variable | Description |
---|
Other configuration details are mentioned as part of the .
Title |
---|
The integration of PGR with a chatbot can be enabled and disabled by making changes in this . By exporting the respective PGR service file, the PGR service feature can be sable and vice versa.
To configure the PGR module to use in Xstate-chatbot - the below variable values need to change in the as per the requirement.
Upload the image into the filestore server. Use the upload file API from the Postman collection .
For the PGR information image mention the filestore id in the environment file.
For open search, the information image mentions the filestore id in the environment file.
The integration of the Bill payment and receipt search feature with the chatbot is enabled and disabled by making changes in this . The payment and receipt search feature can be enabled and vice versa by exporting the respective bill service and receipt service file.
To configure the list of modules to appear as an option for payment and receipt, Add the module business service code to the list present in the file.
Add the message bundle, validation and service code for the locality searcher in and files.
Environmental Variables | Description |
---|
Add this in the and mention the filename in the respective .
Title |
---|
Title |
---|
tenantId
Serves as a Key
moduleName
Name of the module to which the master data belongs
MasterName
The Master Name will be substituted by the actual name of the master data. The array succeeding it will contain the actual data.
egov.mdms.conf.path
The default value of folder where master data files are stored
masters.config.url
The default value of the file URL which contains master-config values
egov-mdms sample data - https://github.com/egovernments/egov-mdms-data/tree/DEV/data - Connect to preview
tenantId
Unique id for a tenant.
Yes
String
filePath
file-path on git where master data is to be created or updated
Yes
String
masterName
Master Data name to be created or updated
Yes
String
masterData
content to be written on to the Config file
Yes
Object
tenantId
Unique id for a tenant
Yes
String
moduleDetails
module for which master data is required
Yes
Array
mdms
Array of modules
Yes
Array
apiId
unique API ID
Yes
String
ver
API version - for HTTP based request this will be same as used in path
Yes
String
ts
time in epoch format: int64
Yes
Long
action
API action to be performed like _create, _update, _search (denoting POST, PUT, GET) or _oauth etc
Yes
String
DID
Device ID from which the API is called
No
String
Key
API key (API key provided to the caller in case of server to server communication)
No
String
msgId
Unique request message id from the caller
Yes
String
requestorId
UserId of the user calling
No
String
authToken
//session/jwt/saml token/oauth token - the usual value that would go into HTTP bearer token
Yes
String
apiId
unique API ID
Yes
String
ver
API version
Yes
String
ts
response time in epoch format: int64
Yes
Long
resMsgId
unique response message-id (UUID) - will usually be the correlation id from the server
No
String
msgId
message-id of the request
No
String
status
status of request processing -
Enum: SUCCESSFUL (HTTP 201) or FAILED (HTTP 400)
Yes
String
code
Error Code will be a module-specific error label/code to identify the error. All modules should also publish the Error codes with their specific localized values in localization service to ensure clients can print locale-specific error messages. An example of an error code would be UserNotFound to indicate User Not Found by User/Authentication service. All services must declare their possible Error Codes with a brief description in the error response section of their API path.
Yes
String
message
English locale message of the error code. Clients should make a separate call to get the other locale description if configured with the service. Clients may choose to cache these locale-specific messages to enhance performance with a reasonable TTL (May be defined by the localization service based on tenant + module combination).
Yes
String
description
Optional long description of the error to help clients take remedial action. This will not be available as part of the localization service.
No
String
params
Some error messages may carry replaceable fields (say $1, $2) to provide more context to the message. E.g. Format related errors may want to indicate the actual field for which the format is invalid. Client's should use the values in the param array to replace those fields.
No
Array
| Duration in minutes for which the authorization token is valid |
| Duration in minutes for which the refresh token is valid |
| Used to start the session by generating Auth token and refresh token from username and password using grant_type as password. The same API can be used to generate new auth token from refresh token by using grant_type as refresh_token and sending the refresh token with key refresh_token |
| This API is used to end the session. The access token and refresh token will become invalid once this API is called. Auth token is sent as param in the API call |
serviceName | Name of the module to which this configuration belongs. |
summary | Summary of the module. |
version | Version of the configuration. |
mappings | List of definitions within the module. Every definition corresponds to one index requirement. Which means, every object received onto the kafka queue can be used to create multiple indexes, each of these indexes will need configuration, all such configurations belonging to one topic forms one entry in the mappings list. The keys listed henceforth together form one definition and multiple such definitions are part of this mappings key. |
topic | The topic on which the data is to be received to activate this particular configuration. |
configKey | Key to identify to what type of job is this config for. values: INDEX, REINDEX, LEGACYINDEX. INDEX: LiveIndex, REINDEX: Reindex, LEGACYINDEX: LegacyIndex. |
indexes | Key to configure multiple index configurations for the data received on a particular topic. Multiple indexes based on a different requirement can be created using the same object. |
name | Index name on the elastic search. (Index will be created if it doesn't exist with this name.) |
type | Document type within that index to which the index json has to go. (Elasticsearch uses the structure of index/type/docId to locate any file within index/type with id = docId) |
id | Takes comma-separated JsonPaths. The JSONPath is applied on the record received on the queue, the values hence obtained are appended and used as ID for the record. |
isBulk | Boolean key to identify whether the JSON received on the Queue is from a Bulk API. In simple words, whether the JSON contains a list at the top level. |
jsonPath | Key to be used in case of indexing a part of the input JSON and in case of indexing a custom json where the values for custom json are to be fetched from this part of the input. |
timeStampField | JSONPath of the field in the input which can be used to obtain the timestamp of the input. |
fieldsToBeMasked | A list of JSONPaths of the fields of the input to be masked in the index. |
customJsonMapping | Key to be used while building an entirely different object using the input JSON on the queue |
indexMapping | A skeleton/mapping of the JSON that is to be indexed. Note that, this JSON must always contain a key called "Data" at the top-level and the custom mapping begins within this key. This is only a convention to smoothen dashboarding on Kibana when data from multiple indexes have to be fetched for a single dashboard. |
fieldMapping | Contains a list of configurations. Each configuration contains keys to identify the field of the input JSON that has to be mapped to the fields of the index json which is mentioned in the key 'indexMapping' in the config. |
inJsonPath | JSONPath of the field from the input. |
outJsonPath | JSONPath of the field of the index json. |
externalUriMapping | Contains a list of configurations. Each configuration contains keys to identify the field of the input JSON that is to be enriched using APIs from the external services. The configuration for those APIs also is a part of this. |
path | URI of the API to be used. (it should be POST/_search API.) |
queryParam | Configuration of the query params to be used for the API call. It is a comma-separated key-value pair, where the key is the parameter name as per the API contract and value is the JSONPath of the field to be equated against this parameter. |
apiRequest | Request Body of the API. (Since we only use _search APIs, it should be only RequestInfo.) |
uriResponseMapping | Contains a list of configuration. Each configuration contains two keys: One is a JSONPath to identify the field from response, Second is also a JSONPath to map the response field to a field of the index json mentioned in the key 'indexMapping'. |
mdmsMapping | Contains a list of configurations. Each configuration contains keys to identify the field of the input JSON that is to be denormalized using APIs from the MDMS service. The configuration for those MDMS APIs also is a part of this. |
path | URI of the API to be used. (it should be POST/_search API.) |
moduleName | Module Name from MDMS. |
masterName | Master Name from MDMS. |
tenantId | Tenant id to be used. |
filter | Filter to be applied to the data to be fetched. |
filterMapping | Maps the field of input json to variables in the filter |
variable | Variable in the filter |
valueJsonpath | JSONPath of the input to be mapped to the variable. |
| Bollean lag to set the payment gateway active/inactive |
| Currency representation for merchant, default(INR) |
| Payment merchant Id |
| Secret key for payment merchant |
| User name to access the payment merchant for transaction |
| Password of the user tp access payment merchant |
| Access code |
| Pay command |
| commans status |
| Url for making payment |
| URL to get the status of the transaction |
| Contains the module name of mdms required for nlp-engine. |
| Contains the file name of mdms master file which contains the city names in various locale. |
| Contains the file name of mdms master file which contains the tenantid of the cities present in |
| Contains the state level tenantid |
| The provider through which WhatsApp messages are sent & received. An adapter for ValueFirst is written. If there is any new provider a separate adapter will have to be implemented. A default |
| The database used to store the chat state. Currently, an adapter for PostgreSQL is provided. An |
| If it’s value is configured to be eGov, it will call the backend rainmaker services. If the value is configured as Dummy, dummy data would be used rather than fetching data from APIs. Dummy option is provided for initial dialog development, and is only to be used locally. |
| A list of comma-separated locales supported by the chatbot. |
host.name | Host name to append in short URL |
db.persistance.enabled | The boolean flag stores the short URL in the database when the flag is set as TRUE. |
| The mobile number to be used on server |
| Username for configured number for sending messages to user through whatsapp provider API calls |
| Password for configured number for sending messages to user through whatsapp provider API calls |
| Maps API key to access geocoding feature |
| Contains state level tenantid value |
| This variable contains the list supported language in chatbot. If there is a need to add new language in chatbot, then its respective locale need to add in this list. |
| Contains PGR version value to use (i.e v1 or v2) |
| Depends on PGR version respective PGR update kafka topic name should mention here. Example: If |
| Limit for showing maximum number of bills on search. |
| Limit for showing maximum number of receipts on search. |
| Limit for showing maximum number of complaints on search. |
| Contains the list of modules to be use for bill payment and receipts search. |
| Contains the filestoreid of informational image, which shows how to share the user current location. |
| Contains the filestoreid of open search informational image, which shows how to use open search pay feature for bill payment |
| This variable contain fixed value of login password and otp. This value has to configured in env-secrets.yaml. |
| Boolean flag to enable and disable city / locality nlp search |
Configuration of the notification messages for a business service based on the channel for the same event.
For a specific action of the user, he/she will get an SMS and email as an acknowledgement.
Users can get SMS, Event, and email based on different channels.
The application allows one to either send different messages across all channels based on their actions.
To have this functionality for different business services, a channel names file was created and added to the MDMS data.
It contains information about the combination of different actions and channels for a particular business service. Example below:
The different channels are
SMS: ID (Mobile Number)
Event
Email: ID (Email ID)
This feature enables the functionality that checks for the channels present in the file and sends the notifications accordingly.
For the SMS event, it would send the SMS notification and log “Sending SMS Notification” for the Event it would log, “Event Notification Sent”, and for Email, it would log, “Email Notification Sent”.
To add/delete any particular channel for a business service -
Update channelNames array in the Channel List file and add/delete the channel you want a business service’s action to have.
Restart egov-mdms-service to apply the changes.
Configure your business service - follow the steps mentioned below.
Adding details about the particular action and the channels you want that action to trigger in the Channel List file in egov-mdms-data repository.
For any record that comes into the topic first, the service should fetch all the required data like user name, property id, mobile number, tenant id, etc from the record that we fetched from the topic.
Respected localization templates should be upserted before in the required environment using the localization service. Restart the localization service to have the newly updated templates available.
Then we should fetch the message content from localization and the service replaces the placeholders with the actual data.
Then put the record in whichever channel’s Kafka topic that the sms, event or email service is listening to.
Then the respective channel services (sms, event, email) will send out the notifications.
Document uploader will be used by ULB employees to upload the document which will then be visible to the citizens. In an effort to increase the engagement of citizens with mSeva platform, mSeva is providing this service to enable the citizens to view important documents related to their ULB such as acts, circulars, citizen charters etc.
Prior Knowledge of Java/J2EE.
Prior Knowledge of SpringBoot.
Prior Knowledge of PostgresSQL.
Prior Knowledge of REST APIs and related concepts like path parameters, headers, JSON etc.
Prior Knowledge of JSONQuery in Postgres. (Similar to PostgresSQL with a few aggregate functions)
Employees can perform all four operations i.e. creating, searching, updating and deleting the documents whereas the citizens can only search for the created documents.
For creating documents in a particular ULB, the document category that needs to be provided in the create API cURL has to be present in the document category MDMS file for the tenantId for which the document is getting uploaded.
A sample MDMS document category configuration file is available here.
In this MDMS configuration file, the key can be added and the allowed category types can be added in categoryList key.
Once a document is created in any ULB, the following attributes can be updated for that document:
ULB
Document name
Document category
Links
Attachments
Upon deleting any document, that document is soft deleted from the records i.e. that document’s active field is set to false.
/egov-document-uploader/egov-du/document/_create - Takes RequestInfo and DocumentEntity in request body. The document entity has all the parameters related to the document being inserted.
/egov-document-uploader/egov-du/document/_update - Allows editing of attributes related to an already existing document. Searches document based on its uuid and updates attributes.
/egov-document-uploader/egov-du/document/_search - Allows searching existing documents in the database. Takes search parameters in the URL and RequestInfo in the request body. For looking at documents that have been created/updated in a given time range, fromDate and toDate parameters can be used which accept epoch values.
For looking at documents that have been deleted, viewDeletedDocuments parameter should be used with “true” as the parameter value.
/egov-document-uploader/egov-du/document/_delete - Soft deletes an existing document from the database i.e. it makes the document inactive. It takes the DocumentEntity that needs to be deleted in the request body along with RequestInfo object.
Detailed API payloads for interacting with document service for all four endpoints is available in the postman collection here.
Click here to find the link to the swagger documentation.
Rainmaker has a reporting framework to configure new reports. As part of the report configuration, we have to write a native SQL query to get the required data for the report. So if the query takes huge time to execute or the query result has huge data, then it will impact the whole application's performance.
The following cases are where we can see the application performance issue because of heavy reports.
Filtering with long date range data or applying fewer filters which in turn returns huge data.
Join the multiple tables for getting required data and missing creating an index on join columns.
Implementing conditional logic inside the queries itself.
Writing multiple sub-queries inside a single query for getting the required data.
Because of heavy reports, the following are the impacts on the platform -
When we execute a complex query on the database, a thread from the connection pool will block to execute the query.
When threads from the connection pool are blocked completely, the application will become very slow for incoming requests.
When max request timeout is crossed, the API gateway will return a timeout error, But still, the connection thread on the database is active, Then all these types of idle threads will occupy database resources like memory, and CPU which in turns increase the load on the database.
Sometimes when running huge queries, the time taken by the query will lead to a broken pipe issue which causes more memory leaks and out-of-heap memory type issues. Because of this, the service will frequently restart automatically.
If a query returns huge data, the browser will become unresponsive and the application will become unresponsive.
Surveys are an important way to close the feedback loop on the services that are delivered by ULBs. Without proper surveys, ULB employees will never know how happy the citizens are, about the services they consume and what the areas of improvement are.
The surveys section will be used by ULB employees to create citizen surveys to make better-informed decisions while issuing new policies/ feedback on existing services.
Prior knowledge of Java/J2EE.
Prior knowledge of SpringBoot.
Prior knowledge of PostgresSQL.
Prior knowledge of REST APIs and related concepts like path parameters, headers, JSON etc.
Prior knowledge of JSONQuery in Postgres. (Similar to PostgresSQL with a few aggregate functions.)
This service allows -
Employees - to create, edit, and delete surveys
Employees - to share survey results with other employees/citizens
Citizens - to fill out surveys
Each survey entity will have questions associated with it. Questions can be of -
short answer type
long answer type
multiple answers type
checkbox answer type
date answer type
time answer type
Survey entities will have a collectCitizenInfo flag associated with them, if that flag is set to true, the mobile number and email address of the citizens responding to the survey will be captured. If it is set to false, responding to the survey will be completely anonymous.
Surveys will have a startDate and endDate associated with them. Within that period, those surveys will be shown in the active section. However, in case the survey has been manually marked inactive by the employee during its active period, that survey will be shown under the inactive surveys section. Citizens can look at active survey entities and submit their responses for that survey.
Once a survey is created, it can be searched with the following parameters -
tenantIds - To search surveys based on multiple ULBs
title - To search surveys based on survey names
postedBy - To search surveys based on the employee who created the survey
status - To search surveys based on status
Also, if any created survey needs to be updated -
Before the survey becomes active, users can edit all the fields.
Once the survey becomes Live user can only change
Survey Description
End date and time of the survey
In case of deletion of a survey, the survey will be soft deleted i.e. the ‘active’ boolean field will be set to false and it will not appear in search results.
APIs for interacting with survey service are available in the collection here.
To use the generic GET/POST SMS gateway, first, configure the service application properties sms.provider.class=Generic
This will set the generic interface to be used. This is the default implementation, which can work with most SMS providers. The generic implementation supports below -
GET or POST-based API
Supports query params, form data, JSON body
To configure the URL of the SMS provider use sms.provider.url property.
To configure the HTTP method use configure the sms.provider.requestType property to either GET or POST.
To configure form data or json api set sms.provider.contentType=application/x-www-form-urlencoded or sms.provider.contentType=application/json respectively.
To configure which data needs to be sent to the API set the below property:
sms.config.map={'uname':'$username', 'pwd': '$password', 'sid':'$senderid', 'mobileno':'$mobileno', 'content':'$message', 'smsservicetype':'unicodemsg', 'myParam': '$extraParam' , 'messageType': '$mtype'}
sms.category.map={'mtype': {'*': 'abc', 'OTP': 'def'}}
sms.extra.config.map={'extraParam': 'abc'}
sms.extra.config.map is not used currently and is only kept for custom implementation which requires data that does not need to be directly passed to the REST API call. sms.config.map is a map of parameters and their values.
Special variables that are mapped -
$username maps to sms.provider.username
$password maps to sms.provider.password
$senderid maps to sms.senderid
$mobileno maps to mobileNumber from kafka fetched message
$message maps to the message from the kafka fetched message
$<name> any variable that is not from above list, is first checked in sms.category.map and then in application.properties and then in the environment variable with full upper case and _ replacing -, space or .
So if you use sms.config.map={'u':'$username', 'p':'password'}. Then the API call will be passed <url>?u=<$username>&p=password
Message success delivery can be controlled using the below properties
sms.verify.response (default: false)
sms.print.response (default: false)
sms.verify.responseContains
sms.success.codes (default: 200,201,202)
sms.error.codes
If you want to verify some text in the API call response set sms.verify.response=true and sms.verify.responseContains to the text that should be contained in the response.
It is possible to whitelist or blacklist phone numbers to which the messages should be sent. This can be controlled using the below properties:
sms.blacklist.numbers
sms.whitelist.numbers
Both of them can be given a separate list of numbers or number patterns. To use patterns use X for any digit match and * for any number of digits match.
sms.blacklist.numbers=5*,9999999999,88888888XX will blacklist any phone number starting with 5, or the exact number 9999999999 and all numbers starting from 8888888800 to 8888888899
Few 3rd parties require a prefix of 0 or 91 or +91 with the mobile number. In such a case you can use the sms.mobile.prefix to automatically add the prefix to the mobile number coming in the message queue.
Through report service, useful data get shown for a specific module based on some given criteria like date, locality, financial year, etc.
For example, PT dump report of property tax service you have to select from date to date, financial year etc and based on the criteria we can see all the data fulfilling the criteria. In the response, we see all the details of a property which is paid between the given from date and to date. In case we select the financial year, the property which is paid for that specific financial year is visible.
Before you proceed with the configuration, make sure the following pre-requisites are met -
User with permission to edit the git repository where Reports are configured and knowledge on YAML.
Prior Knowledge of YAML.
Prior Knowledge of SQL queries.
Prior Knowledge of the relation between the tables for which module you are going to write a report.
Users can write queries (like SQL queries) for fetching real-time data to display in a UI application.
Users can apply filters like from date, to date, financial year, etc based on the report configuration.
Users can download the result in PDF and XLS format.
User can select or deselect the columns user wants to see.
User can choose the number of records he/she wants to see on a page.
Once the changes have been done in the report configuration file we have to restart the report service so the report service will read the new configuration.
To add a new report first add the file path in the reportFileLocationsv1. (In this file, the path of the report configuration files gets stored).
<Module Name>=file:///work-dir/configs/reports/config/<report file name>.yml
ex: pgr=file:///work-dir/configs/reports/config/pgr-reports.yml
Once the file path is added in the file reportFileLocationsv1, go to the folder /configs/reports/config.
Create a new file and name the file that you have given in the file reportFileLocationsv1.
Write the report configuration. Once it is done commit those changes.
Add the role and actions for the new report.
Restart the MDMS and report service.
Configuring a report for a module requires adding the required report configuration as per the standard format and with the minimum development time.
UI can have different types of filters such as date, dropdown etc.. and even the sum of a column can also be easily displayed in UI. Pagination and downloading the report in pdf format, xls format is already present in the report UI.
Type of reports which can be configured :
Count of applications
Statewide collections
Application status
Cancelled receipts
Migrated records / Data entry records
The limitation of this framework is for reports having requirements with complex queries with multiple joins as the report uses the query to fetch the data from the database, It is resource-intensive and response might be slow in those scenarios.
Before you proceed with the configuration, make sure the following pre-requisites are met -
User with permission to edit the git repository to add the report configuration.
User with permission to add action and role action in the MDMS.
Showcase the data in the required and cleaner format.
The UI is rendered with the help of configuration in the report and there is no extra effort in building UI for different reports.
For Implementation specific report requirements, customization is easy and turnaround time is less.
After adding the new report/ editing the existing report configuration in the respective module, the report service needs to be restarted.
Create a reports.yml file and add report configuration as per standard format.
Add the action and role action in the MDMS.
Add the GitHub raw path of the report.yml file in the report.config file.
Reporting Service is a service running independently on a separate server. The main objective of this service is to provide a common framework for generating reports. This service loads the report configuration from a yaml file at the run time and provides the report details by using a couple of APIs.
Before you proceed with the documentation, make sure the following pre-requisites are met -
Prior knowledge of Java/J2EE.
Prior knowledge of SpringBoot.
Advanced knowledge of PostgreSQL.
Encryption and MDMS services must be running.
Prior knowledge of REST APIs and related concepts like path parameters, headers, JSON etc.
JSON path for filtering required data from json objects.
Provides an easy way to add reports on the fly just by adding configurations without any coding effort.
Provides flexibility to customise result column names in the config.
Provides flexibility to fetch data from DB and also from some other services returning required json objects when its not possible to get all required data from DB.
Provides functionality to add filters as per requirements before actually fetching data for reports.
Provides user data in masked form. People with authorization will get plain data one by one.
Config file
A YAML (xyz.yml) file which contains configuration for report requirements.
API
A REST endpoint to fetch data based on the configuration.
Inline-table
If we also want to show data from some external service with data coming from DB in reports we use inline tables. The data from an external service is stored in an inline table and then used as any normal DB table to get data. This table is short-lived and stays only for the time when the query is being executed. It is never stored in DB. We provide JSON paths in an ordered manner corresponding to each column in the table. These JSON paths will be used to extract the required data from the external service’s response. For configs please see the ‘How to Use’ section.
Configuration: As mentioned above, the report service uses a config file per module to store all the configurations of reports pertaining to that module. Report service reads multiple such files at start-up to support reports of all the configured modules. The file contains the following keys:
reportName: name of the report, to be used with module name to identify any report config
summary: summary of the report
version: version of the report
moduleName: name of the module to which the report belongs to
externalService: To be used when some of the report data needs to be fetched from external service through inline tables. It contains the following fields
entity: JSON Path to filter json arrays(result to be turned into tables) from returned json
apiURL: API URL of the external service
keyOrder: order of JSON object keys to form table columns from JSON object arrays
tableName: name to be given to represent this transformed data which will be used as a table in the SQL query
sourceColumns: These represent the final data sent by the service on GET_DATA API call. The order of sourceColumns in the Config is the same as that of columns in the result. Each sourceColumns represents one column in the result. For each column, data is picked after executing the final SQL query formed after appending groupby, orderby, and search params into the base query.
name: name of the column to fetch data from query results, must be there in query results
label: custom column label
type: data type of column
source: module name
total: whether column total is required on the front end
searchParams:
name: name of search param. Must match variable used in search clause
label: the custom label for viewing on the front end
type: type of search params. If the type is ‘singlevaluelist’ then use the pattern to populate searchparams possible values to select from by the user Ex:-number, string, singlevaluelist etc.
source: module name
isMandatory: If the user must fill this searchparam before requesting report data
searchClause: SQL search clause for corresponding search params to filter results, to be appended in base query Ex:- AND fnoc.tenantId IN ($ulb). Here $ulb will be replaced by user inputs
Pattern: This field will be used only when ‘type’ is set to ‘singlevaluelist’. It is an external service URL combined with JSON Paths separated by ‘|’. The first JSON path is for codes and the second is for values. Values will be shown to the user in the dropdown. And codes corresponding to user selected value will be sent to the report service and will be used in searchClauses.
Query: Main/base query clause for fetching report data from DB and custom tables formed after fetching data from external service
Orderby: order by clause to be appended into base query
Groupby: group by clause to be appended into base query
additionalConfig: to provide additional custom configs which are not present above
Call the MDMS or any other API with the post method
Configuring the post object in the yaml itself like below.
externalService:
entity: $.MdmsRes.egf-master.FinancialYear
keyOrder: finYearRange,startingDate,endingDate,tenantId
tableName: tbl_financialyear
stateData: true
postObject:
tenantId: $tenantid
moduleDetails:
moduleName: egf-master
masterDetails:
name: FinancialYear filter: "[?(@.id IN [2,3] && @.active == true)]"
Keep the post object in a separate JSON file externally and call at runtime.
There are two API calls to report service ‘GET METADATA’ and ‘GET DATA’.
GET METADATA
This request to report service is made to get metadata for any report. The metadata contains information about search filters to be used in the report before actually sending a request to get actual data. The user-selected values are then used in the GET_DATA request to filter data.
endpoint: /report/{moduleName}/{report name}/metadata/_get
moduleName:- It is used to define the names of the module which contain current report
Body: The body consists of the following:
RequestInfo: Header details as used on the egov platform
tenantId: tenantId of ULB
reportName: name of the report to be used
Instance:
URL: https://{domain name}/report/rainmaker-tl/metadata/_get
Body:
GET DATA
This request-to-report service is used to get data for the report. Inputs given by the user for filters are sent in the request body. These filter values are used while querying data from DB.
endpoint: report/{moduleName}/{report name}/_get
moduleName: It is used to define the names of the module which contains the current repo
Body: The body consists of the following:
RequestInfo: Header details as used on the egov platform
tenantId: tenantId of ULB
reportName: name of the report to be used
The array of searchparams corresponds to each of the filled filters by the user. Each searchparam contains:-
Name: name of the filter
Input: user selected value
Instance:
URL: https://{domain name}/report/rainmaker-tl/_get
Body:
Write configuration as per your requirement. The structure of the config file is explained in the configuration details section.
Check-in the config file to a remote location preferably GitHub, currently we check the files into the folder here. Can't find links for the dev and QA environment.
Add the module name and corresponding report path in the same format as used here.
Provide the absolute path of the file mentioned in Point 3 to DevOps, to add it to the file-read path of the report service. The file will be added to the environment manifest file for it to be read at the start-up of the application.
Deploy the latest version of the report service app.
Add role-action mapping for APIs.
Use the module name as a path parameter in the URL of the requests for report service with the required request body.
The report service provides a common framework to generate reports and show the report details based on the search criteria.
Provide a common framework for generating reports
Provide functionality to create new ad-hoc reports with minimal efforts
Avoid writing code again in case of new report requirements
Makes possible to create reports with only knowledge of SQL and JSONPath
Provides metadata about the report.
Provides the data for the report.
Reload the configuration at runtime.
To integrate, the host of echallan-services module should be overwritten in the helm chart.
The API should be mentioned in ACCESSCONTROL-ACTIONS-TEST. Refer below example
Add role-action mapping for APIs.
Note: Refer to the guidelines for supporting user privacy in a module, for the changes required in the report config to support the encryption/decryption process.
Persister Service persists data in the database in a sync manner providing very low latency. The queries which have to be used to insert/update data in the database are written in yaml file. The values which have to be inserted are extracted from the json using jsonPaths defined in the same yaml configuration.
Below is a sample configuration which inserts data in a couple of tables.
The above configuration is used to insert data published on the kafka topic save-pgr-request
in the tables eg_pgr_service_v2
and eg_pgr_address_v2
. Similarly, the configuration can be written to update data. Following is a sample configuration:
The above configuration is used to update the data in tables. Similarly, the upsert operation can be done using ON CONFLICT() function in psql.
The table below describes each field variable in the configuration.
Persister service provides a framework to persist data in transactional fashion with low latency based on a config file. Removes repetitive and time consuming persistence code from other services.
Before you proceed with the documentation, make sure the following pre-requisites are met -
Prior Knowledge of Java/J2EE.
Prior Knowledge of SpringBoot.
Prior Knowledge of PostgresSQL.
Prior Knowledge of JSONQuery in Postgres. (Similar to PostgresSQL with a few aggregate functions.).
Kafka server is up and running.
Persist data asynchronously using kafka providing very low latency
Data is persisted in batch
All operations are transactional
Values in prepared statement placeholder are fetched using JsonPath
Easy reference to parent object using ‘{x}’ in jsonPath which substitutes the value of the variable x in the JsonPath with value of x for the child object.(explained in detail below in doc)
Supported data types ARRAY("ARRAY"), STRING("STRING"), INT("INT"),DOUBLE("DOUBLE"), FLOAT("FLOAT"), DATE("DATE"), LONG("LONG"),BOOLEAN("BOOLEAN"),JSONB("JSONB")
Persister uses configuration file to persist data. The key variables are described below:
serviceName: Name of the service to which this configuration belongs.
description: Description of the service.
version: the version of the configuration.
fromTopic: The kafka topic from which data is fetched
queryMaps: Contains the list of queries to be executed for the given data.
query: The query to be executed in form of prepared statement:
basePath: base of json object from which data is extrated
jsonMaps: Contains the list of jsonPaths for the values in placeholders.
jsonPath: The jsonPath to fetch the variable value.
To persist large quantity of data bulk setting in persister can be used. It is mainly used when we migrate data from one system to another. The bulk persister have the following two settings:
Any kafka topic containing data which has to be bulk persisted should have '-batch' appended at the end of topic name example: save-pt-assessment-batch.
Each persister config has a version attribute which signifies the service version, this version can contain custom DSL; defined here, GitHub - zafarkhaja/jsemver: Java implementation of the SemVer Specification
Every incoming request [via kafka] is expected to have a version attribute set, [jsonpath, $.RequestInfo.ver] if versioning is to be applied.
If the request version is absent or invalid [not semver] in the incoming request, then a default version defined by the following property in application.propertiesdefault.version=1.0.0
is used.
The request version is then matched against the loaded persister configs and applied appropriately.
Write configuration as per the requirement. Refer the example given earlier.
In the environment file, mention the file path of configuration under the variable egov.persist.yml.repo.path
while mentioning the file path we have to add file:///work-dir/
as prefix. for example: egov.persist.yml.repo.path = file:///work-dir/configs/egov-persister/abc-persister.yml
. If there are multiple file separate it with comma (,
) .
Deploy latest version of egov-persister service and push data on kafka topic specified in config to persist it in DB.
The persister configuration can be used by any module to store records in particular table of database.
Insert/Update Incoming Kafka messages to Database.
Add Modify kafka message before putting it into database.
Persist data asynchronously.
Data is persisted in batch.
Write configuration as per your requirement. Structure of the config file is explained above in the same document.
Check-in the config file to a remote location preferably github.
Provide the absolute path of the checked-in file to DevOps, to add it to the file-read path of egov-persister. The file will be added to egov-persister's environment manifest file for it to be read at start-up of the application.
Run the egov-persister app and push data on kafka topic specified in config to persist it in DB
The objective of this service is to create a common point to manage all the SMS notifications being sent out of the platform. Notification SMS service consumes SMS from the Kafka notification topic and processes them to send it to a third-party service. Modules like PT, TL, PGR etc make use of this service to send messages through the Kafka Queue.
Before you proceed with the documentation, make sure the following pre-requisites are met -
Prior knowledge of Java/J2EE
Prior knowledge of SpringBoot
Prior knowledge of Third party API integration
Prior knowledge of REST APIs and related concepts like path parameters, headers, JSON etc
Prior knowledge of Kafka and related concepts like Producer, Consumer, Topic etc.
Provide a common platform to send an SMS notification to the user.
Support localised SMS.
Easily configurable with a different SMS service provider.
This service is a consumer, which means it reads from the Kafka queue and does not provide a facility to be accessed through API calls, there’s no REST layer here. The producers willing to integrate with this consumer will be posting a JSON onto the topic configured at ‘kafka.topics.notification.sms.name’. The notification-sms service reads from the queue and sends the sms to the mentioned phone number using one of the SMS providers configured.
The implementation of the consumer is present in the directory src/main/java/org/egov/web/notification/sms/service/impl
.
These are the current providers available
Generic
Console
MSDG
The implementation to be used can be configured by setting sms.provider.class
.
The Console
implementation just prints the message mobile number and message to the console
This is the default implementation, which can work with most SMS providers. The generic implementation supports below
GET or POST-based API
Supports query params, form data, JSON Body
To configure the URL of the SMS provider use sms.provider.url
property To configure the HTTP method use this to configure the sms.provider.requestType
property to either GET
or POST
.
To configure form data or json API set sms.provider.contentType=application/x-www-form-urlencoded
or sms.provider.contentType=application/json
respectively
To configure which data needs to be sent to the API below property can be configured:
sms.config.map
={'uname':'$username', 'pwd': '$password', 'sid':'$senderid', 'mobileno':'$mobileno', 'content':'$message', 'smsservicetype':'unicodemsg', 'myParam': '$extraParam' , 'messageType': '$mtype'}
sms.category.map
={'mtype': {'*': 'abc', 'OTP': 'def'}}
sms.extra.config.map
={'extraParam': 'abc'}
sms.extra.config.map
is not used currently and is only kept for custom implementation which requires data that doesn't need to be directly passed to the REST API call
sms.config.map
is a map of parameters and their values
Special variables that are mapped
$username
maps to sms.provider.username
$password
maps to sms.provider.password
$senderid
maps to sms.senderid
$mobileno
maps to mobileNumber
from kafka fetched message
$message
maps to the message
from the kafka fetched message
$<name>
any variable that is not from the above list, is first checked in sms.category.map
and then in application.properties
and then in the environment variable with full upper case and _
replacing -
, space or .
So if you use sms.config.map={'u':'$username', 'p':'password'}
. Then the API call will be passed <url>?u=<$username>&p=password
Message Success or Failure
Message success delivery can be controlled using the below properties
sms.verify.response
(default: false)
sms.print.response
(default: false)
sms.verify.responseContains
sms.success.codes
(default: 200,201,202)
sms.error.codes
If you want to verify some text in the API call response set sms.verify.response=true
and sms.verify.responseContains
to the text that should be contained in the response.
Blacklisting or Whitelisting numbers
It is possible to whitelist or blacklist phone numbers to which the messages should be sent. This can be controlled using below properties:
sms.blacklist.numbers
sms.whitelist.numbers
Both of them can be given a ,
separated list of numbers or number patterns. To use patterns use X
for any digit match and *
for any number of digits match.
sms.blacklist.numbers=5*,9999999999,88888888XX
will blacklist any phone number starting with 5
, or the exact number 9999999999
and all numbers starting from 8888888800
to 8888888899
Prefixing
Few 3rd parties require a prefix of 0
or 91
or +91
with the mobile number. In such a case you can use sms.mobile.prefix
to automatically add the prefix to the mobile number coming into the message queue.
Error Handling
There are different topics to which the service will send messages. Below is a list of the same:
1kafka.topics.backup.sms 2kafka.topics.expiry.sms=egov.core.sms.expiry 3kafka.topics.error.sms=egov.core.sms.error
In an event of a failure to send an SMS, if kafka.topics.backup.sms
is specified, then the message will be pushed onto that topic.
Any SMS which expires due to Kafka lags, or some other internal issues, they will be passed to the topic configured in kafka.topics.expiry.sms
If a backup
the topic has not been configured, then in an event of an error the same will be delivered to kafka.topics.error.sms
Following are the properties in the application.properties file in the notification sms service which are configurable.
Add the variables present in the above table in a particular environment file
Deploy the latest version of egov-notification-sms service.
Notification SMS service consumes SMS from the Kafka notification topic and processes them to send it to a third-party service. Modules like PT, TL, PGR etc make use of this service to send messages through the Kafka Queue.
Provide an interface to send notification SMS on user mobile number.
Support SMS in various languages.
To integrate, create the SMS request body given in the example below. Provide the correct mobile number and message in the request body and send it to the Kafka topic:- egov.core.notification.sms
The notification-sms service reads from the queue and sends the sms to the mentioned phone number using one of the SMS providers configured.
The enc-client library is a supplementary java library provided to support encryption-related functionalities so that every service does not need to pre-process the request before calling the encryption service.
MDMS Service
Encryption Service
Kafka
The MDMS configurations explained below are fetched by this library at boot time. So after you make changes in the MDMS repo and restart the MDMS service, you would also need to RESTART THE SERVICE which has imported the enc-client library. For example, the report service is using the enc-client library so after making configuration changes to Security Policy pertaining to any report, you will have to restart the report service.
Encrypt a JSON Object - The encryptJson
function of the library takes any Java Object as an input and returns an object which has encrypted values of the selected fields. The fields to be encrypted are selected based on an MDMS Configuration.
This function requires the following parameters:
Java/JSON object - The oject whose fields will get encrypted.
Model - It is used to identify the MDMS configuration to be used to select fields of the provided object.
Tenant Id - The encryption key will be selected based on the passed tenantId.
Encrypt a Value - The encryptValue
function of the library can be used to encrypt single values. This method also required a tenantId parameter.
Decrypt a JSON Object - The decryptJson
function of the library takes any Java Object as an input and returns an object that has plain/masked or no values of the encrypted fields. The fields are identified based on the MDMS configuration. The returned value(plain/masked/null) of each of the attribute depends on the user’s role and if it is a PlainAccess
request or a normal request. These configurations are part of the MDMS.
This function required following parameters:
Java/JSON object - The object containing the encrypted values that are to be decrypted.
Model - It is used to select a configuration from the list of all available MDMS configurations.
Purpose - It is a string parameter that passes the reason of the decrypt request. It is used for Audit purposes.
RequestInfo - The requestInfo parameter serves multiple purposes:
User Role - A list of user roles are extracted from the requestInfo parameter.
PlainAccess Request - If the request is an explicit plain access request, it is to be passed as a part of the requestInfo. It will contain the fields that user is requesting for decryption and the id of record.
While decrypting Java object, this method also audits the request.
All the configurations related to enc-client library are stored in the MDMS. These master data are stored in DataSecurity
module. It has two types of configurations:
Masking Patterns
Security Policy
The masking patterns for different types of attributes(mobile number, name, etc.) are configurable in MDMS. It contains the following attributes:
patternId
- It is the unique pattern identifier. This id is referred to in the SecurityPolicy MDMS.
pattern
- This defines the actual pattern according to which the value will be masked.
The Security Policy master data contains the policy used to encrypt and decrypt JSON objects. Each of the Security Policy contains the following details:
model
- This is the unique identifier of the policy.
uniqueIdentifier
- The field defined here should uniquely identify records passed to the decryptJson
function.
attributes
- This defines a list of fields from the JSON object that needs to be secured.
roleBasedDecryptionPolicy
- This defines attribute-level role-based policy. It will define visibility for each attribute.
The visibility is an enum with the following options:
PLAIN - Show text in plain form.
MASKED - The returned text will contain masked data. The masking pattern will be applied as defined in the Masking Patterns master data.
NONE - The returned text will not contain any data. It would contain string like “Confidential Information”.
It defines what level of visibility should the decryptJson
function return for each attribute.
The Attribute defines a list of attributes of the model that are to be secured. The attribute is defined by the following parameters:
name
- This uniquely identifies the attribute out of the list of attributes for a given model.
jsonPath
- It is the json path of the attribute from the root of the model. This jsonPath is NOT the same as Jayway JsonPath library. This uses /
and *
to define the json paths.
patternId
- It refers to the pattern to be used for masking which is defined in the Masking Patterns master.
defaultVisibility
- It is an enum configuring the default level of visibility of that attribute. If the visibility is not defined for a given role, then this defaultVisibility will apply.
This parameter is used to define the unique identifier of that model. It is used for the purpose of auditing the access logs. (This attribute’s jsonPath should be at the root level of the model.)
It defines attribute-level access policies for a list of roles. It consists of the following parameters:
roles
- It defines a list of role codes for which the policy will get applied. Please make sure not to duplicate role codes anywhere in the other policy. Otherwise, any one of the policies will get chosen for that role code.
attributeAccessList
- It defines a list of attributes for which the visibility differs from the default for those roles.
There are two levels of visibility:
First level Visibility - It applies to normal search requests. The search response could have multiple records.
Second level Visibility - It is applied only when a user explicitly requests for plain access of a single record with a list of fields required in plain.
Second level visibility can be requested by passing plainAccessRequest
in the RequestInfo
.
Any user will be able to get plain access to the secured data(citizen’s PII) by requesting through the plainAccessRequest
parameter. It takes the following parameters:
recordId
- It is the unique identifier of the record that is requested for plain access.
fields
- It defines a list of attributes that are requested for plain access.
Every decrypt request is audited. Based on the uniqueIdentifier
defined as part of the Security Policy, it lists out the identifiers of the records that were decrypted as part of the request.
Each Audit Object contains the following attributes:
An eGov core application which provides locale-specific components and translating text for the eGov group of applications.
Prior knowledge of Java/J2EE.
Prior knowledge of Spring Boot.
Prior knowledge of REST APIs and related concepts like path parameters, headers, JSON etc.
Prior knowledge of Redis and Postgres.
The localization application stores the locale data in the format of key and value along with the module, tenantid and locale. The module defines which application of eGov owns the locale data and tenantId does the same for the tenant, locale is the specific locale for which data is being added.
The request can be posted through the post API with the above-mentioned variables in the request body.
Once posted the same data can be searched based on the module, locale and tenantId as keys.
The data posted to the localization service is permanently stored in the database and loaded into the Redis cache for easy access and every time new data is added to the application the Redis cache will be refreshed.
Deploy the latest version of the Localization Service.
Add role-action mapping for APIs.
The localization service is used to store key-value pairs of metadata in different languages for all miscellaneous/ad-hoc services which citizens avail from ULBs.
Can perform service-specific business logic without impacting the other module.
Provides the capability of having multiple languages in the module.
To integrate, the host of the localization-services module should be overwritten in the helm chart.
/localization/messages/v1/_upsert
should be added as the create endpoint for creating localization key-value pairs in the system
/localization/messages/v1/_search
should be added as the search endpoint. This method handles all requests to search existing records depending on different search criteria
Encryption Service is used to secure sensitive data that is being stored in the database.
Before you proceed with the documentation, make sure the following pre-requisites are met -
Java 8.
Kafka server is up and running.
Encryption Service offers following features :
Encrypt - The service will encrypt the data based on given input parameters and data to be encrypted. The encrypted data will be mandatorily of type string.
Decrypt - The decryption will happen solely based on the input data (any extra parameters are not required). The encrypted data will have identity of the key used at the time of encryption, the same key will be used for decryption.
Sign - Encryption Service can hash and sign the data which can be used as unique identifier of the data. This can also be used for searching gicen value from a datastore.
Verify - Based on the input sign and the claim, it can verify if the the given sign is correct for the provided claim.
Rotate Key - Encryption Service supports changing the key used for encryption. The old key will still remain with the service which will be used to decrypt old data. All the new data will be encrypted by the new key.
Following are the properties in application.properties file in egov-enc-service which are configurable.
Deploy the latest version of Encryption Service.
Add Role-Action mapping for API’s.
The Encryption service is used to encrypt sensitive data that needs to be stored in the database.
Can perform encryption without having to re-write encryption logic everytime in every service.
To integrate, host of encryption-services module should be overwritten in helm chart.
/crypto/v1/_encrypt
should be added as endpoint for encrypting input data in the system
/crypto/v1/_decrypt
should be added as the decryption endpoint.
/crypto/v1/_sign
should be added as the endpoint for providing signature for a given value.
/crypto/v1/_verify
should be added as the endpoint for verifying whether the signature for the provided value is correct.
/crypto/v1/_rotatekey
should be added as endpoint to deactivate the keys and generate new keys for a given tenant.
a) POST /crypto/v1/_encrypt
Encrypts the given input value/s OR values of the object.
b) POST /crypto/v1/_decrypt
Decrypts the given input value/s OR values of the object.
c) /crypto/v1/_sign
Provide signature for a given value.
d) POST /crypto/v1/_verify
Check if the signature is correct for the provided value.
e) POST /crypto/v1/_rotatekey
Deactivate the keys for the given tenant and generate new keys. It will deactivate both symmetric and asymmetric keys for the provided tenant.
An eGov core application which handles uploading different kinds of files to server including images and different document types.
Prior Knowledge of Java/J2EE.
Prior Knowledge of Spring Boot.
Prior Knowledge of REST APIs and related concepts like path parameters, headers, JSON etc.
Prior knowledge of aws and azure
The filestore application takes in a request object which contains an image/document or any kind of file and stores them in a disk/AWS/azure depending upon the configurations, additional implementations can be written for the app interface to interact with any other remote storages.
The request file to be uploaded is taken in form of multipart file part then saved to the storage and a uuid will returned as a unique identifier for that resource, which can be used to fetch the documents later.
Incase of images, the application will create three more additional copies of the file in the likes of large, medium and small for the usage of thumbnails or low quality images incase of mobile applications.
The search api takes the uuid, tenantid as mandatory url params and a few optional parameters and returns the presigned url of the files from the server, In case of images a single string containing multiple urls separated by commas will be returned representing different sizes of images stored.
The Application is present among the core group of applications available in the eGov-services git repository. The spring boot application but needs lombok extension added in your ide to load it. Once the application is up and running API requests can be posted to the url and ids can be generated.
**in case of intellij the plugin can be installed directly, for eclipse the lombok jar location has to be added in eclipse.ini file in this format -javaagent:lombok.jar.
For the API information please refer the swagger yaml
Then add the raw url of the API doc in the pop up.
The application needs at least one type of storage available for it to store the files either file-storage, AWS S3 or azure. More storage types can be added by extending the application interface also.
To work work any of the file storages there are some application properties which needs to be configured.
DiskStorage:
The mount path of the disk should be provided in the following variable to save files in the disc. file.storage.mount.path=path.
Following are the variables that needs to be populated based on the aws/azure account you are integrating with.
How to enable Minio SDC:
isS3Enabled=true(Should be true)
aws.secretkey={minio_secretkey}
aws.key={minio_accesskey}
fixed.bucketname=egov-rainmaker(Minio bucket name)
minio.source=minio
How to enable AWS S3:
isS3Enabled=true(Should be true)
aws.secretkey={s3_secretkey}
aws.key={s3_accesskey}
fixed.bucketname=egov-rainmaker(S3 bucket name)
minio.source=minio
AZURE:
isAzureStorageEnabled - informing the application whether Azure is available or not
azure.defaultEndpointsProtocol - type of protocol https
azure.accountName - name of the user account
azure.accountKey - secret key of the user account
NFS :
isnfsstorageenabled-informing the application whether NFS is available or not <True/False>
file.storage.mount.path - <NFS location, example /filestore>
source.disk - diskStorage - name of storage
disk.storage.host.url=<Main Domain URL>
Allowed formats to be uploaded The default format of the files is the use of set brackets with strings inside it - {"jpg", "png"}. Make sure to follow the same.
allowed.formats.map: {jpg:{'image/jpg','image/jpeg'},jpeg:{'image/jpeg','image/jpg'},png:{'image/png'},pdf:{'application/pdf'},odt:{'application/vnd.oasis.opendocument.text'},ods:{'application/vnd.oasis.opendocument.spreadsheet'},docx:{'application/x-tika-msoffice','application/x-tika-ooxml','application/vnd.oasis.opendocument.text'},doc:{'application/x-tika-msoffice','application/x-tika-ooxml','application/vnd.oasis.opendocument.text'},dxf:{'text/plain'},csv:{'text/plain'},txt:{'text/plain'},xlsx:{'application/x-tika-ooxml','application/x-tika-msoffice'},xls:{'application/x-tika-ooxml','application/x-tika-msoffice'}}
The key in the map is the visible extension of the file types, the values on the right in curly braces are the respective tika types of the file. these values can be found in tika website or by passing the file through tika functions.
Upload POST API to save the files in the server
Search Files GET API to retrieve files based only on id and tenantid
Search URLs GET API to retrieve pre-signed URLs for a given array of ids
Deploy the latest version of Filestore Service.
Add role-action mapping for APIs.
The filestore service is used to upload and store documents which citizens add while availing of services from ULBs.
Can perform file upload independently without having to add fileupload specific logic in each module.
To integrate, the host of the filestore module should be overwritten in the helm chart.
/filestore/v1/files
should be added as the endpoint for uploading files in the system
/filestore/v1/files/url
should be added as the search endpoint. This method handles all requests to search existing files depending on different search criteria
Our telemetry service is built upon Sunbird telemetry sdk. mSeva’s frontend React app pushes the telemetry JS file along with every response that it sends. Now, whenever a user interacts with any of the mSeva pages in any way, for e.g. entering values in a text box or when the window is loaded etc. The event will get triggered and will be recorded.
The telemetry API collection is available here.
The telemetry payload consists of an events array. In DIGIT, only 3 event types are used namely -
START - START signifies that events have now started getting collected for a particular page.
SUMMARY - SUMMARY signifies a collection of all the data that is required to get collected for that particular page. For example, time spent by the user on that page, times at which the user came into the page, left the page for another tab etc. are all recorded as part of the SUMMARY event
END - END event signifies the end of collecting events.
All these events keep getting collected and are bundled and sent when either the URL changes or the END event occurs.
The event data is captured and pushed onto a Kafka topic.
It goes into the processing pipeline where we make a topic to topic transfer of the data. So, the format for the events payload is checked, and then it is pushed to another topic for de-duplicating.
Similarly, the messages are unbundled and enriched via topic to topic transfer i.e. pick from one topic and push to another. In this case, there are two sinks, namely the Amazon S3 bucket and the ES bucket. To perform this topic to the topic transfer of data across the various components of the processing pipeline, Kafka streams (KStreams) are used which are nothing but a consumer and producer coupled together.
To push data to the S3 bucket, a Secor service is used developed by Pinterest to pick up JSON data and push it onto configured S3 buckets. Secor does not always create a new JSON file for any new data that it gets. There are two triggers for it, namely, reaching a particular threshold size or reaching a particular time threshold. To push data to the ES sink, Kafka Connect is used. Now, instead of making single API calls every time a message is received, the messages are again combined and persisted onto the ES index via bulk insert.
According to TRAI’s regulation on unsolicited commercial communication, all teleco's must verify every sms content before delivering it (Template scrubbing). For this all the businesses using SMS need to register Entities, SenderIDs, SMS templates in a centralised DLT portal.
Below are the steps to register the SMS template in a centralised DLT portal and to add the template in the SMS country portal (Service provider).
Step 1: Visit Airtel DLT portal(AIRTEL DLT ) and select your area of operation as Enterprise then click on next.
Step 2: Login into the portal by entering the proper credentials and OTP.
Please contact the HR manager for the credentials and the OTP.
Step 3: Now select the Template from option and then click on content templates.
Now click on Add button to go to next section
Step 4: Select the option mentioned in the image below.
Note: a) For placeholder text (dynamic text in message) mention {#var#} in the message. Each {#var#} can contains 0-30 character. If dynamic text is supposed to go more than 30 characters in length, then two {#var#} have to mention side by side. Now the dynamic text can be up to 60 characters. Example:- Hi Citizen, Click on this link to pay the bill {#var#}{#var#} EGOVS b) It is mandatory to mention EGOVS at the end of every message. c) Select Template message Type as Regional if the message is in other language rather than english.
After clicking on Save button, template is added in the portal, Now wait for the approval of the template. Once the template get approved save the template id and the message.
Step 5: Repeat process from Step 3 to Step 4 to register template in DLT portal.
Note: The below steps are to add approved templates in the SMS Country web portal. These steps might be different for other service providers but the data required for any service provider would be the same.
Step 6: Now, login into SMS Country portal (SMS Marketing Solution Provider | SMS & Voice Marketing APIs - SMSCountry ) by entering proper credentials.
Please contact the HR manager for the credentials.
Step 7: Select option Features, then click on Manage button under Template section.
Then click on Add DLT Template button.
Step 8: Mention the template id and message of approved template which we have saved earlier in step 4. And select sender id as EGOVFS
After adding all the above details click on Add Template button. Now the DLT approved template get added into SMS Country portal and it is ready to use.
Select ISLanguage check box if the message is in other language rather than english.
Step 9: Repeat process from Step 7 to Step 8 to add approved template in SMS Country portal.
Here are the articles in this section:
In a bid to avoid unnecessary repetition of codes which generate ids and to have central control over the logic so that the burden of maintenance is reduced from the developers. To create a config-based application which can be used without writing even a single line of coding.
Prior knowledge of Java/J2EE.
Prior knowledge of Spring Boot and Flyway.
Prior knowledge of REST APIs and related concepts like path parameters, headers, JSON etc.
The application exposes a Rest API to take in requests and provide the ids in response in the requested format. The request format varies from current date information, , random number, and sequence generated number. Id can be generated by providing a request with any of the above-mentioned information.
For example: The Id Amritsar-PT-2019/09/12-000454-99 contains
Amritsar - which is the name of the city
PT - a fixed string representing the module code(PROPERTY TAX)
2019/09/12 - date
000454 - sequence generated number
99 - random number
The id generated in the above-mentioned eg needs the following format
[city]-PT-[cy:yyyy/mm/dd]-[SEQ_SEQUENCE_NAME]-[d{4}]
Everything in the square brackets will be replaced with the appropriate values by the app.
ID-FORMAT Configuration
By default, the IDGen service will now read its configuration from MDMS. DB Configuration requires access to the DB, so the new preferred method for the configuration is MDMS. The configuration needs to be stored in common-masters/IdFormat.json in MDMS
It is recommended to have the IdFormat as a state-level master. To disable the configuration to be read from DB instead, the environment variable IDFORMAT_FROM_MDMS should be set to false.
ID-FORMAT-REPLACEABLES:
[FY:] - represents the financial year, the string will be replaced by the value of starting year and the last two numbers of the ending year separated by a hyphen. For instance: 2018-19 in case of the financial year 2018 to 2019.
[cy:] - any string that starts with cy will be considered as the date format. The values after the cy: is the format using which output will be generated.
[d{5}] - d represents the random number generator, the length of the random number can be specified in flower brackets next to d. If the value is not provided then the random length of 2 will be assigned.
[city] - The string city will be replaced by the city code provided by the respective ULB in location services.
[SEQ_*] - String starting with seq will be considered as sequence names, which will be queried to get the next seq number. If your sequence does not start with the namespace containing “SEQ” then the application will not consider it as a sequence. In the absence of the sequence from DB error will be thrown.
[tenantid] - replaces the placeholder with the tenantid passed in the request object.
[tenant_id] - replaces the placeholder with the tenantid passed in the request object. Replaces all `.` with `_`
[TENANT_ID] - replaces the placeholder with the tenantid passed in the request object. Replaces all `.` with `_`, and changes the case to upper case.
idName v/s format:
When you use both idName and format in a request. IDGEN will first try to see if the format for the given idName exists, if not then the format will be used.
STATE v/s ULB Level Sequences:
If you want a state-level sequence then you need to use a fixed sequence name
But if you want a ULB level sequence, the sequence name should be dynamic based on the tenantid as given in the below example.
Sequences & Its Creation
The SEQ_* replaceable used in id generation are by default expected for the sequence to already exist in the DB. But this behaviour can be changed and controlled using two environmental variables while deploying the service
AUTOCREATE_NEW_SEQ: Default to false. When set to true, this will auto-create sequences when the format has been derived using provided idName. Since the idName format comes from DB or MDMS, it is a trusted value and this value should be set to true. This will make sure that no DB configuration needs to be done as long as MDMS has been configured. It is recommended that each service using idgen should generate an ID using idName instead of just using passing the format directly. This will make sure that no DB configuration needs to be done for creating sequences.
AUTOCREATE_REQUEST_SEQ: Default to false. When set to true, this will auto-create sequences when the format has been derived using the format parameter from the request. This is recommended to keep as false, as anyone with access to idgen can create any number of sequences in DB and overload the DB. Though during the initial setup of an environment, this variable can be kept as true to create all the sequences when the initial flows are run from the UI to generate the sequences. And afterwards, the flags should be disabled.
Add MDMS configs required for ID Gen service and restart the MDMS service.
Deploy the latest version of the ID generation service.
Add role-action mapping for APIs.
The ID Gen service is used to generate unique ID numbers for all miscellaneous/ad-hoc services which citizens avail from ULBs.
Can perform service-specific business logic without impacting the other module.
Provides the capability of generating the unique identifier of the entity calling ID Gen service.
To integrate, the host of idgen-services module should be overwritten in the helm chart.
/egov-idgen/id/_generate
should be added as the endpoint for generating ID numbers in the system
The main reason to Setup Base Product Localization is because Digit system supports multiple languages. By setting-up Localization, we can have multiple language support to the UI. So, that user can easily understand the Digit Operations
Before you proceed with the configuration, make sure the following pre-requisites are met -
Before Starting the Localization setup one should have knowledge on React and eGov FrameWork.
Before setting-up Localization, make sure that all the Keys are pushed to the Create API and also get prepared with the Values that need to be added to the Localization key specific to particular languages that are being added in the product.
Make sure where to add the Localization in the Code.
Once the Localization is done, the user can view the Digit Screens in their own language to complete the whole application process easier as digit gives the user to select the language of their choice.
Once The key is added to the code as per requirement, Deployment can be done in the same way, how the code is being deployed.
Select a label which is needed to be localized from the Product code. Here is the example code for a header before setting-up Localization.
As we see the above which supports only the English language, To setup Localization to that header we need to the code in the following manner.
we can see below code is added when we compare with code before Localization setup.
{
labelName: "Trade Unit ",
labelKey: "TL_NEW_TRADE_DETAILS_TRADE_UNIT_HEADER"
},
Here the Values to the Key can be added by two methods either by using the localization Screen which is Developed Recently or by updating the values to the keys to create API using the postman application.
User Service stores the PII data in the database in encrypted form. So whichever service is reading that data directly from the database will have to decrypt the data before responding to the user. As of now, to the best of our knowledge following services are reading the user data from the database:
User Service
Report Service
Searcher Service
Enc-Client Library is a supplementary library with features like masking, auditing, etc. The above services call the functions from the library to decrypt the data read from the database.
Learn how to configure Localization service.
This documents highlights the changes need to be done in a module to support the user privacy feature.
Prior Knowledge of Java/J2EE.
Prior Knowledge of Spring Boot.
MDMS Service
Encryption Service
Prior Knowledge of React Js / Javascript
Upgrade services-common library version.
Upgrade tracer library version.
After upgrading the above library version, there might be chance that an issue occur for RequestInfo creation because of change in variable type, please go through all java files including JUnit test cases before pushing the changes.
The above library version need to be upgraded in the dependant module also. For Example: If water service call the property service and property service is calling user service for owner details then the version of above library need to be upgraded in property service pom file also.
At service level or calculator service level where demand generation process take place, there payer details value must be pass in plain form. And these user details must get through SYSTEM user.
Create a system user in the particular environment and make sure that system user has role INTERNAL_MICROSERVICE_ROLE
. Use the curl mentioned below:
Do not create the system user with INTERNAL_MICROSERVICE_ROLE
if it is already existed. Use the UUID of that user, to get plain result data.
Mention the uuid of the system user in the environment file and in the application.properties file of service example:
Environment file:
egov-internal-microservice-user-uuid: b5b2ac70-d347-4339-98f0-5349ce25f99f
Application properties file:
egov.internal.microservice.user.uuid=b5b2ac70-d347-4339-98f0-5349ce25f99f
Create a method/ function which call the user search API to get the user details in plain form, in that call pass the userInfo of the user with INTERNAL_MICROSERVICE_ROLE
For reference follow the below code snippet
For the report where PII data is present and decryption is required, such report definition must have UUID values from user table in the query result.
And the Source column section of the report definition must have mention of the UUID column and for that entry, showColumn
flag should be set as false.
And the base sql query in the report config should be written in such a way that in the query it must have the user uuid.
In the config, for name: uuid Here replace uuid with the name of the column containing the user’s uuid in the query’s output.
Example:
In the report config where PII data is present and decryption is required, add this new filed in the report config decryptionPathId
. And in the Security Policy mdms file, must have the model configuration from the particular report. And the value of key model
in Security Policy mdms file and decryptionPathId
in a report must be same.
Example:-
For the searcher result where PII data is present and decryption is required, such searcher definition must have UUID values from user table in the query result. The base sql query in the searcher config should be written in such a way that in the query it must have the user uuid.
I searcher result, where data are coming from user table and to get those value in decrypted form, then in searcher config these following filed decryptionPathId
must be present. And the Security Policy mdms file, must have the model configuration from the particular searcher config. And the value of key model
in Security Policy mdms file and decryptionPathId
in searcher config must be same.
To Do
Title |
---|
Title |
---|
Title |
---|
Variable Name | Description |
---|---|
Variable Name | Default Value | Description |
---|---|---|
Property | Value | Remarks |
---|---|---|
Title |
---|
Here is a link to a sample master data.
To know more about regular expression refer the below articles To test regular expression refer the below link.
Property | Default Value | Remarks |
---|
Title | Link |
---|
GOTO : and click on file -> import url
Incase the url is unavailable, please go to the of egov-services git repo and find the yaml for egov-filestroe.
minio.url=.backbone:9000(Minio server end point)
minio.url=
Title | Link |
---|
All content on this page by is licensed under a .
When any of these services reads the data from the database, it will be in encrypted form. Before responding to a request, they call the enc-client library to convert the data to plain/masked form. The data returned as part of the response should only be in plain/masked form. It should not contain any encrypted values. Detailed guidelines on how to write the necessary configurations are provided in document.
For details information on Security Policy MDMS file refer this .
Local Setup
serviceName
The module name to which the configuration belongs
version
Version of the config
description
Detailed description of the operations performed by the config
fromTopic
Kafka topic from which data has to be persisted in DB
isTransaction
Flag to enable/disable perform operations in Transaction fashion
query
Prepared Statements to insert/update data in DB
basePath
JsonPath of the object that has to be inserted/updated.
jsonPath
JsonPath of the fields that has to be inserted in table columns
type
Type of field
dbType
DB Type of the column in which field is to be inserted
persister.bulk.enabled
false
Switch to turn on or off the bulk kafka consumer
persister.batch.size
100
The batch size for bulk update
egov.core.notification.sms
It is the topic name to which the notification sms consumer would subscribe. Any module wanting to integrate with this consumer should post data on this topic only.
sms.provider.class
Generic
This property decides which SMS provider is to be used by the service to send messages. Currently, Console, MSDG and Generic have been implemented.
sms.provider.contentType
application/x-www-form-urlencoded
To configure form data or json api set sms.provider.contentType=application/x-www-form-urlencoded or sms.provider.contentType=application/json respectively
sms.provider.requestType
POST
Property to configure the http method used to call provider
sms.provider.url
URL of the provider. This will be given by the SMS provider only.
sms.provider.username
egovsms
Username as provided by the provider which is passed during the API call to the provider.
sms.provider.password
abc123
Password as provided by the provider which is passed during the API call to the provider. This has to be encrypted and stored
sms.senderid
EGOV
SMS sender id provided by the provider, this will show up as the sender on receiver’s phone.
sms.config.map
{'uname':'$username', 'pwd': '$password', 'sid':'$senderid', 'mobileno':'$mobileno', 'content':'$message', 'smsservicetype':'unicodemsg', 'myParam': '$extraParam' , 'messageType': '$mtype'}
Map of parameters to be passed to the API provider. This is provider-specific. $username maps to sms.provider.username
$password maps to sms.provider.password
$senderid maps to sms.senderid
$mobileno maps to mobileNumber from kafka fetched message
$message maps to the message from the kafka fetched message
$<name> any variable that is not from above list, is first checked in sms.category.map and then in application.properties and then in environment variable with full upper case and _ replacing -, space or
sms.category.map
{'mtype': {'*': 'abc', 'OTP': 'def'}}
replace any value in sms.config.map
sms.blacklist.numbers
5*,9999999999,88888888XX
For blacklisting, a “,” separated list of numbers or number patterns. To use patterns use X for any digit match and * for any number of digits match
sms.whitelist.numbers
5*,9999999999,88888888XX
For whitelisting, a “,” separated list of numbers or number patterns. To use patterns use X for any digit match and * for any number of digits match
sms.mobile.prefix
91
add the prefix to the mobile number coming in the message queue
| asd@#$@$!132123 | Master password for encryption/ decryption. It can be any string. |
| qweasdzx | A salt is random data that is used as an additional input to a one-way function that hashes data, a password or passphrase. It needs to be an alphanumeric string of length 8. |
| qweasdzxqwea | An initialization vector is a fixed-size input to a cryptographic primitive. It needs to be an alphanumeric string of length 12. |
| 256 | Default size of Symmetric key. |
| 1024 | Default size of Asymmetric key. |
| 12 | Default size of Initial vector. |
The objective of this service is to create a common point to manage all the email notifications being sent out of the platform. The notification email service consumes email requests from the Kafka notification topic and processes them to send them to a third-party service. Modules like PT, TL, PGR etc make use of this service to send messages through the Kafka Queue.
Before you proceed with the documentation, make sure the following pre-requisites are met -
Prior knowledge of Java/J2EE
Prior knowledge of SpringBoot
Prior knowledge of Third party API integration
Prior knowledge of REST APIs and related concepts like path parameters, headers, JSON etc
Prior knowledge of Kafka and related concepts like Producer, Consumer, Topic etc.
Provide a common platform to send email notifications to the user
Support localised email.
The egov-notification-mail is a consumer which listens to the egov.core.notification.email topic reads the message and generates email using SMTP Protocol. The services need the senders' email configured. On the other hand, if the senders' email is not configured, the services get the email id by internally calling egov-user service to fetch the email id. Once the email is generated, the content is localized by egov-localization service after which it is notified to the email id.
Deploy the latest version of the notification email service.
Make sure the consumer topic name for email service is added in deployment configs
The email notification service is used to send out email notifications for all miscellaneous/ad-hoc services which citizens avail from ULBs.
Can perform service-specific business logic without impacting the other module.
In the future, if we want to expose the application to citizens then it can be done easily.
To integrate, the client service should send email requests to email notification consumer topics.
An API Gateway provides a unified interface for a set of microservices so that clients do not need to know about all the details of microservice internals.
DIGIT uses Zuul as an edge service that proxies request to multiple back-end services. It provides a unified “front door” to our ecosystem. This allows any browser, mobile app or other user interface to consume underlying services.
Before you proceed with the documentation, make sure the following pre-requisites are met -
Java 8
egov-user service is running
egov-accesscontrol service is running
Provides easier API interface to clients
Can be used to prevent exposing the internal micro-services structure to the outside world.
Allows to refactor microservices without forcing the clients to refactor consuming logic
Can centralize cross-cutting concerns like security, monitoring, rate limiting etc
Zuul has mainly four types of filters that enable us to intercept the traffic in the different timelines of the request processing for any particular transaction. We can add any number of filters for a particular URL pattern.
pre-filters – are invoked before the request is routed.
post filters – are invoked after the request has been routed.
route filters – are used to route the request.
error filters – are invoked when an error occurs while handling the request.
Microservice authentication and security
Authorization
API Routing
Open APIs using Whitelisting
RBAC filter
Logout filter for the finance module
Property module tax calculation filter for firecess
Request enrichment filter:
Addition of co-relation id
Addition of authenticated user (userinfo) to requestInfo.
Error filter:
Error response formatting
Validation Filter to check if a tenant of a particular module is enabled or not.
Multitenancy Validation Filter. Take the tenant id from the Req body or Query Param and validate against the additional tenant role or primary tenant role.
DevOps efficiency: API response time for logging and sending notifications if it is taking more time.
Rate Throttling
Routing Property
For each service, below mentioned property has to be add-in routes.properties
Rate Limiting Property
For endpoints which require rate throttling, below mentioned property has to be added in limiter.properties
Deploy the latest version of the Zuul service.
Add Zuul routing context paths and service hostname in the configuration.
The Zuul service is used to act as an API gateway for services which citizens avail from ULBs.
Can perform service-specific business logic without impacting the other module.
Provides the capability of routing and authorizing users for accessing resources.
To integrate, the host of the Zuul module should be overwritten in the helm chart.
Performance testing and benchmark
Since we have around 600 wards in Punjab, we decided to make the national dashboard service be robust enough to handle 600 requests simultaneously in a very short time (preferably midnight when the employees ingest that day’s data).
The national dashboard service Ingest API underwent extensive stress testing where the service was tested for 25 concurrent threads (users) targeting 100 requests per minute and each request payload containing 30 records (as the ingest API is a bulk API). The service passed all the tests with 0 per cent error with a throughput of 21 requests per second, 12671 total number of samples in 10 minutes which roughly equals 21 requests per second.
The stress test ran with the following configuration and reliably persisted each and every record that was ingested -
National-dashboard-ingest service -
Number of pods- 5
Number of threads for each pod- 25
Heap memory- 750 MB
Producer linger time- 1 ms
National dashboard Kafka pipeline service -
Number of pods- 3
Number of threads for each pod- 10
Heap memory- 512 MB
Producer linger time- 1 ms
This document lists the steps required to follow to create national dashboard indexes.
Prior Knowledge of ElasticSearch
For understanding purposes, we are listing out the steps needed to be followed to create a property tax index for the national dashboard.
Open Kibana
Create an index using the following query -
3. Add corresponding field mappings for the created index -
This page provides steps to deploy Airflow DAG.
The Kubernetes environment is required for the deployment of Airflow.
Step 1: Clone the git repo for airflow, and update the values.yaml as per the requirement.
Step 2: Update the git repository URL and subpath for the directory in values.yaml
Example: the following params are updated as given below:
repo: "GitHub - pmidc-digit/utilities ”
repoSubPath: "egov-national-dashboard-accelerator/dag
branch: "develop
Step 3: Change the directory to airflow and update the helm. Update the helm repo locally and add the airflow repo to Helm using the command below:
helm repo add apache-airflow https://airflow.apache.org
The above command pulls the airflow repo and it is added to the local helm repository.
Step 4: Installing Apache airflow after updating the helm repositories
helm install airflow apache-airflow/airflow --namespace egov
The above command will take the updated repo details.
Step 5: Upgrade the changes made to values.yaml using the command below.
helm upgrade --install airflow apache-airflow/airflow -n airflow -f values.yaml
The above command updates the git repo, subpath and branch while deployment.
Step 6: Deployment is done pods service will start running with updated values.yaml
Latest files for the deployment: Attached below is the final "values.yaml" file. It syncs both the plugins and dags from the repo. Airflow Deployment
Refer to the sample below to delete the data for the month of July module-wise in the Postgres database:
delete from nss_ingest_data where dataKey like '%-07-2022:FIRE%';
delete from nss_ingest_data where dataKey like '%-07-2022:TL%';
delete from nss_ingest_data where dataKey like '%-07-2022:WS%';
delete from nss_ingest_data where dataKey like '%-07-2022:PGR%';
delete from nss_ingest_data where dataKey like '%-07-2022:PT%';
delete from nss_ingest_data where dataKey like '%-07-2022:MCOLLECT%';
Adjust the module and the date range accordingly.
Check the records before deleting.
Note: Deleting data from both ES and Postgres is mandated to avoid duplication of data.
Adding New Language to Digit System. You can refer the link provided for how languages are added in DIGIT |
API Swagger Documentation |
Notification service can notify the user through SMS and email for there action on DIGIT as an acknowledgement that their action has been successfully completed.
ex: actions like property create, TL create, etc.
To send SMS we need the help of 2 services, one on which the user is taking action and the second SMS service.
To send an email we need the help of 2 services, one on which the user is taking action and second email service.
Before you proceed with the configuration, make sure the following pre-requisites are met -
Prior Knowledge of Spring boot.
Prior Knowledge of Kafka.
Prior Knowledge of localization service.
For a specific action of the user, he/she will get a SMS and email as an acknowledgment.
Users can get SMS and email based on the localization language.
If you want to take action for a specific action on that action the service has to listen to a particular topic so that each time any record comes to the topic consumer will know that action has been taken and can trigger a notification for it.
ex: if you want to trigger a notification for Property create then the property service’s NotificationConsumer class should listen to topic egov.pt.assessment.create.topic so that each time any record comes to the topic egov.pt.assessment.create.topic NotificationConsumer will know that Property creates action that has been taken and can trigger a notification for it.
when any record comes into the topic first the service will fetch all the required data like user name, property id, mobile number, tenant id, etc from the record which we fetched from the topic.
Then we will fetch the message contain from localization and the service replaces the placeholders with the actual data.
Then put the record in SMS topic in which SMS service is listening.
email service is also listening to the same topic which SMS service is listening.
User-OTP service handles the OTP for user registration, user log in and password reset for a particular user.
Before you proceed with the documentation, make sure the following pre-requisites are met -
Java 8
egov-user service is running
egov-localization service is running
egov-otp service is running
The user-otp service sends the OTP to the user on login request, on password change request and during new user registration.
Deploy the latest version of user-otp.
Make sure egov-otp is running.
Add Role-Action mapping for APIs.
User-OTP service handles the OTP for user registration, user login and password reset for a particular user.
Can perform user registration, login, and password reset.
In the future, if we want to expose the application to citizens then it can be done easily.
To integrate, the host of user-otp module should be overwritten in the helm chart.
/user-otp/v1/_send
should be added as the endpoint for sending OTP to the user via sms or email
BasePath
/user-otp/v1/[API endpoint]
Method
a) POST /_send
This method sends the OTP to a user via sms or email based on the below parameters -
Following are the Producer topic.
egov.core.notification.sms.otp
:- This topic is used to send OTP to user mobile number.
org.egov.core.notification.email
:- This topic is used to send OTP to user email id.
The objective of egov-searcher service is listed below.
To provide a one-stop framework for searching data from multiple data-source based on configuration (Postgres, Elasticsearch etc).
To create provisions for implementing features based on ad-hoc requirements which directly or indirectly require a search functionality.
Prior knowledge of Java/J2EE
Prior knowledge of SpringBoot
Prior knowledge of PostgresSQL
Prior knowledge of REST APIs and related concepts like path parameters, headers, JSON etc.
Prior knowledge of JSONQuery in Postgres. (Similar to PostgresSQL with a few aggregate functions.)
Step 1: Write the configuration as per your requirement. The structure of the config file is explained later in the same doc.
Step 2: Check-in the config file to a remote location preferably GitHub, currently we check the files into this folder - https://github.com/egovernments/configs/tree/DEV/egov-searcher for DEV and QA and this folder - https://github.com/egovernments/configs/tree/UAT/egov-searcher for UAT.
Step 3: Provide the absolute path of the checked-in file to DevOps, to add it to the file-read path of egov-searcher. The file will be added to egov-searcher's environment manifest file for it to be read at the start-up of the application.
Step 4: Run the egov-searcher app, and use the modulename and definitionname parameters from the configuration as path parameters in the URL of the search API to fetch the required data.
Config file - A YAML (xyz.yml) file which contains configuration for search requirements.
API - A REST endpoint to fetch data based on the configuration.
Uses Postgres JSONQuery instead of SQL queries to fetch data from the Postgres DB.
JSONQuery: JSONQuery is one of the exclusive features of Postgres, It provides a way of fetching data from the DB as JSON instead of ResultSet format. This saves the time spent mapping ResultSet into required JSON formats on the functionality side. JSONQueries are similar to SQL queries with certain functions to internally map the ResultSet to JSON. SQL queries (SELECT queries to be precise) are passed as parameters to these functions, the SQL Query returns the ResultSet which is transformed into the JSON by these functions.
Some of the functions extensively used are: 1) row_to_json: This function takes a query as a parameter and converts the result into JSON. However, the query must return only one row in the response. Note that, JSONQuery functions operate on aliases, So, the query must be mapped to an alias and the alias is passed to the function as a parameter. For example: {"name": "egov", "age": "20"} 2) array_agg: This functions takes the output of row_to_json and aggregates it into an array of JSON. This is required when the query is returning multiple rows in the response. The query will be passed to row_to_json through an alias, this is further wrapped within array_agg to ensure all the rows returned by the query as converted to a JSONArray. For example: [{"name": "egov", "age": "20"},{"name": "egov", "age": "20"},{"name": "egov", "age": "20"}] 3) array_to_json: This transforms the result of array_agg into a single JSON and returns it. This way, the response of a JSONQuery will always be a single JSON with the JSONArray of results attached to a key. This function is more for the final transformation of the result. The result so obtained can be easily cast to any other format or operated on using the PGObject instance exposed by Postgres. For example: {"alias": [{"name": "egov", "age": "20"},{"name": "egov", "age": "20"},{"name": "egov", "age": "20"}]}
For more details about JSONQuery, please check: JSON Functions and Operators
Provides an easy way to set up search APIs on the fly by adding configurations without any coding effort.
Provides flexibility to build where clause as per requirement, with config keys for operators, conditional blocks and other query clauses.
Designed to use a specific URI for every search request, making role-based access control easy.
Fetches data in the form of JSON the format of which can be configured. This saves considerable effort in writing row mappers for every search result.
Add configs for different modules required for Searcher Service.
Deploy the latest version of Searcher Service.
Add role-action mapping for APIs.
The searcher service is used to search for data present in databases by running PSQL queries in the background.
Can perform service-specific business logic without impacting the other module.
In the future, if we want to expose the application to citizens then it can be done easily.
To integrate, the host of the searcher-service module should be overwritten in the helm chart.
searcher/{moduleName}/{searchName}/_get
should be added as the search endpoint for the config added.
URI: The format of the search API to be used to fetch data using egov-searcher is as follows: /egov-searcher/{moduleName}/{searchName}/_get
Every search call is identified by a combination of moduleName and searchName. Here, 'moduleName' is the name of the module as mentioned in the configuration file and 'searchName' is the name of the definition within the same module that needs to be used for our search requirement.
For instance, If I want to search all complaints on PGR I will use the URI - /egov-searcher/rainmaker-pgr-V2/serviceSearchWithDetails/_get
Body: The body consists of 2 parts: RequestInfo and searchCriteria. searchCriteria is where the search params are provided as key-value pairs. The keys given here are the ones to be mentioned in the 'jsonPath' configuration within the 'searchParams' key of the config file.
For instance, If I want to search complaints on PGR where serviceRequestId is 'ABC1234' and tenantId is 'pb.amritsar' the API body will be:
Chatbot service is a chatbot which provides functionality to the user to access PGR module services like filing complaints, tracking complaints, and notifications from WhatsApp. Currently, the citizen has three options - to start a conversation scan the QR code, give a missed call or directly send a message to configured WhatsApp number.
Prior knowledge of Java/J2EE.
Prior knowledge of SpringBoot.
Prior knowledge of PostgreSQL.
Prior knowledge of REST APIs and related concepts like path parameters, headers, JSON etc.
Prior knowledge of JSONQuery in Postgres. (Similar to PostgreSQL with a few aggregate functions.)
egov-user-chatbot
: For creating a user without name validation and logging in the user
egov-user
: For searching users
egov-localization
: The chatbot is made such that it will store localization codes and the actual text value will be fetched only at the end. This way we can provide multi-lingual support. Localization service is also used to construct messages from templates. This dependency can be eliminated if you want to pass values instead of localization codes.
egov-filestore
: It is a dependency if you want to send/receive any file. This includes sending PDF/Image files.
egov-url-shortening
: For shortening links sent to the user
egov-mdms-service
: For loading MDMS data
egov-location
: For loading locality data
rainmaker-pgr
: For creating/searching PGR complaints
Chatbot service allows citizens to access PGR service through WhatsApp. Citizens can provide all details required to create a PGR complaint through the question-and-answer method. The service continuously listens to the PGR update Kafka topic and sends notifications to users associated with the PGR record. On any message from a citizen which is forwarded by a WhatsApp provider, the chatbot processes the messages by passing messages through various stages. For instance - validations, enrichment, transformations etc and at the end sends the final response to the user by calling the endpoint of the WhatsApp provider.
There are two types of configurations for chatbot states:-
Configuration for each state in chatbot, ex:-
Graph adjacency list configuration:- to define flow between chatbot states,ex:-
High-Level diagram of chatbot interactions
Flow Diagram of Chatbot-User conversation
Flow Diagram of Chatbot notifications
Add configs required for chatbot service.
Deploy the latest version of the chatbot service.
Add role-action mapping for APIs.
The chatbot service is used to communicate with the users, lodge their complaints or provide miscellaneous/ad-hoc services which citizens avail from ULBs in an interactive way over WhatsApp.
Can perform service-specific business logic without impacting the other module.
In the future, if we want to expose the application to citizens then it can be done easily.
The workflow or Service-specific workflow can be enabled at the chatbot level at any time without changing the design.
To integrate, the host of the chatbot module should be overwritten in the helm chart.
POST /messages
should be added as the endpoint for receiving user-sent messages and forwarding them to the chatbot core logic for further processing and sending back the response.
GET /messages
should be added as the endpoint for receiving user-sent messages and forwarding them to chatbot core logic for further processing and sending back responses.
a) POST /messages
Receive the user-sent messages and forward them to chatbot core logic for further processing and sending back responses.
If the media_type
parameter value is text
then user input would be sent in the parameter text
, in other cases where media_type
have some other value ex:- image, location etc, the user input would be sent in a parameter media_data
b) GET /messages
Receive the user-sent message and forward it to the chatbot core logic for further processing and sending back responses.
If the media_type
parameter value is text
then user input would be sent in the parameter text
, in other cases where media_type
have some other value ex:- image, location etc, the user input would be sent in a parameter media_data
update-pgr-service
: used in update.pgr.service.topic
application property, the chatbot listens on this topic to listen for updates on PGR records and then to send notifications to users.
The service uses consumers for internal processing also between different stages.
send-message-localized
: chatbot sends data to this topic for telemetry indexing and for internal processing.
The service uses producers for internal processing also between different stages.
OTP Service is a core service that is available on the DIGIT platform. The service is used to authenticate the user in the platform. The functionality is exposed via REST API.
Before you proceed with the documentation, make sure the following pre-requisites are met -
Java 8
egov-otp is called internally by the user-otp service that fetches the mobile number and feeds to egov-otp to generate the 'n' DIGIT OTP.
The below properties define the OTP configurations -
a) egov.otp.length
: Number of digits in the OTP
b) egov.otp.ttl
: Controls the validity time frame of the OTP. The default value is 900 seconds. Another OTP generated within this time frame is also allowed.
c) egov.otp.encrypt
: Controls if the OTP is encrypted and stored in the table.
Deploy the latest version of egov-otp service.
Add role-action mapping for APIs.
The egov-otp service is used to authenticate the user in the platform.
Can perform user authentication without impacting the other module.
In the future, this application can be used in a standalone manner in any other platforms that require a user authentication system.
To integrate, the host of egov-otp module should be overwritten in the helm chart.
/otp/v1/_create
should be added as the create endpoint. Create OTP configuration API is an internal call from v1/_send endpoint. This endpoint is present in the user-otp service and removes the need for explicit calls.
/otp/v1/_validate
should be added as the validate endpoint. The OTP configuration end point validates the OTP with respect to the mobile number.
/otp/v1/_search
should be added as the search endpoint. This API searches the mobile number and OTP using uuid - mapping uuid to OTP reference number.
BasePath
/egov-otp/v1
Egov-otp service APIs - contains create, validate and search endpoints
a) POST /otp/v1/_create
- create OTP configuration this API is an internal call from v1/_send endpoint. This endpoint present in the user-otp service removes the need for explicit calls.
b) POST /otp/v1/_validate
- validate OTP configuration this endpoint is to validate the OTP with respect to the mobile number
c) POST /otp/v1/_search
- Search the mobile number and OTP using uuid, uuid using the OTP reference number
The objectives of the national dashboard-ingest service are listed below.
To provide a one-stop framework for ingesting data regardless of a data source based on configuration.
To create provision for ingest based on module-wise requirements which directly or indirectly require aggregated data ingestion functionality.
Prior knowledge of Java/J2EE.
Prior knowledge of SpringBoot.
Prior knowledge of PostgreSQL.
Prior knowledge of REST APIs and related concepts like path parameters, headers, JSON etc.
Prior knowledge of ElasticSearch.
Step 1: Define the index name for the module as per your requirement in module.index.mapping
key present in the configuration here - DIGIT-DevOps/qa.yaml at master · egovernments/DIGIT-DevOps .
Step 2: Define the allowed metrics for the module as per your requirement in module.fields.mapping
key present in the configuration here - DIGIT-DevOps/qa.yaml at master · egovernments/DIGIT-DevOps
Step 3: Define the allowed group-by fields for the module as per your requirement in module.allowed.groupby.fields.mapping
key present in the configuration here - DIGIT-DevOps/qa.yaml at master · egovernments/DIGIT-DevOps
Step 4: Define the master data index name as per your requirement in master.data.index
key present in the configuration here - DIGIT-DevOps/qa.yaml at master · egovernments/DIGIT-DevOps
Step 5: Define the allowed metrics for the master data index as per your requirement in master.module.fields.mapping
key present in the configuration here - https://github.com/egovernments/DIGIT-DevOps/blob/master/deploy-as-code/helm/environments/qa.yaml#L367
Step 6: Create Kafka connectors for all the modules that have been configured. A sample request for creating a trade license national dashboard Kafka connector is as follows -
7. Step 7: Run the national-dashboard-ingest application along with the national-dashboard-ingest-kafka-pipeline.
Config file - A YAML (xyz.yml) file which contains configuration for running national dashboard ingest.
API - A REST endpoint to post data based on the configuration.
When the national dashboard ingests metrics API is hit, all the data payload lookup keys are first checked against the db to determine whether they already exist or not. The db table currently being used for storing lookup keys is nss-ingest-data
.
If the record for a given date and area details is not present, the payload is then flattened and pushed to nss-ingest-keydata
topic.
National dashboard ingest Kafka pipeline consumer listens on nss-ingest-keydata
topic and according to the module to which the data belongs, pushes it to the respective topic defined in the module.index.mapping
key.
Once the national dashboard ingests the Kafka pipeline and pushes data to the respective topic, a Kafka connector then takes the flattened records from that topic and ingests into ElasticSearch.
Add configs for different modules required for National Dashboard Ingest Service and National Dashboard Kafka Pipeline service.
Deploy the latest version of National Dashboard Ingest and National dashboard Kafka pipeline service.
Add role-action mapping for APIs.
The national dashboard service is used to push aggregated data present in systems and persisting them onto elasticsearch on top of which dashboards can be built for visualizing and analyzing data.
Can perform service-specific business logic without impacting the other module.
In the future, if we want to expose the application to citizens then it can be done easily.
To integrate, the host of the national-dashboard-ingest-service module should be overwritten in the helm chart.
national-dashboard/metric/_ingest
should be added as the search endpoint for the config added.
national-dashboard/masterdata/_ingest
should be added as the search endpoint for the config added.
URI: The format of the ingest API to be used to ingest data using national-dashboard-ingest is as follows: national-dashboard/metric/_ingest
Example ingest request body -
The steps required for the creation of the index and adding the index mapping for the same can be found here - National Dashboard: Steps for Index Creation.
The following section contains module-wise index names, index mapping and ingest curls for ingesting data to national dashboard indexes.
a. Index - pt-national-dashboard
b. Index mapping -
c. Ingest curl -
a. Index name - ws-national-dashboard
b. Index mapping -
c. Ingest curl -
a. Index name - firenoc-national-dashboard
b. Index mapping -
c. Ingest curl -
a. Index name - pgr-national-dashboard
b. Index mapping -
c. Ingest curl -
a. Index name - tl-national-dashboard
b. Index mapping -
c. Ingest curl -
a. Index name - mcollect-national-dashboard
b. Index mapping -
c. Ingest curl -
a. Index name - obps-national-dashboard
b. Index mapping -
c. Ingest curl -
a. Index name - common-national-dashboard
b. Index mapping -
c. Ingest curl -
Service request allows users to define a service and then create a service against service definitions. A service definition can be a survey or a checklist which the users might want to define and a service against the service definition can be a response against the survey or a filled-out checklist.
Prior knowledge of Java/J2EE
Prior knowledge of SpringBoot
Prior knowledge of PostgreSQL
Prior knowledge of REST APIs and related concepts like path parameters, headers, JSON etc.
Users can -
Create and search service definitions.
Create and search services.
/service-request/service/definition/v1/_create - Takes RequestInfo and ServiceDefinition in request body. ServiceDefinition has all the parameters related to the service definition being created.
/service-request/service/definition/v1/_search - Allows searching of existing service definitions. Takes RequestInfo, ServiceDefinitionCriteria and Pagination objects in the request body.
/service-request/service/v1/_create - Takes RequestInfo and Service in the request body. Service has all the parameters related to the service being created against a particular ServiceDefinition.
/service-request/service/v1/_search - Allows searching of existing services created against service definitions. Takes RequestInfo, ServiceCriteria and Pagination objects in the request body.
The code is organised in the following code repo:
utilities/egov-national-dashboard-accelerator at develop · pmidc-digit/utilities
Refer to the folder structure below:
dags
plugins
The dags folder is the folder which contains the DAG code.
national_dashboard_template_latest.py(manual trigger)
national_dashboard_template_scheduled.py(scheduled)
The queries folder contains the ES queries for the individual modules
For adding a new module we need to add a module-specific file in the queries folder with the transform logic and then refer to the DAG code in both manual and trigger versions.
The overview of the adaptor service
Steps to trigger airflow DAG
In Airflow, a DAG
–Directed Acyclic Graph – is a collection of the tasks you want to run, organized in a way that reflects their relationships and dependencies.
A DAG is defined in a Python script, which represents the DAGs structure (tasks and their dependencies) as code.
Manual Trigger
1.Log onto the Punjab Prod server using the credentials:
URL: Sign In - Airflow
username: admin
password: admin
2. Trigger the DAG by clicking on the “Trigger DAG with Config” option.
3. Enter a date and click on the Trigger button
Format {“date”: “dd-MM-yyyy”}
4. Click on the Log option and expand the DAG to view the logs. Choose a stage for any module.
Logs can also be viewed in the Elastic search index adaptor_logs
GET adaptor_logs/_search - the timestamp is provided based on the day for which the logs are being searched.
This DAG triggers every day at midnight for the previous day.
Bulk Insert For A Date Range
Execute the script to run the DAG for a date range for the staging NDB
sh iterate_over_date.sh <start-date> <end-date> ex: sh iterate_over_date.sh 01-03-2022 05-03-2022
date needs to be in the format of dd-mm-YYYY
range exclusive of the last date, [start-date, end-date). For instance: in the above example, the script will trigger DAG on 1st, 2nd, 3rd and 4th March. It will not be triggered on 5th March.
URL | UserID | Password | City | |
---|---|---|---|---|
QA Test Cases - Test Cases Link
Elastic Search Queries - Elastic Search Queries
List of Index in Punjab - List Of Index in Punjab
Code Checkin path - Code Checkin Repo
JIRA link for the issue tracking - Dashboard for Issue Tracking
Postman Collection for the INGEST API - Postman Collection for Ingest API
National Dashboard KPI + Definitions- KPI+Definitions
Design and develop an adaptor service which extracts data from the state DIGIT installations and pushes it onto the National Dashboard instance periodically.
A list of tasks for this is tracked for the adaptor - Jira - Dashboard
Adaptor to be deployed on state DIGIT installations
Periodically, the adaptor extracts data and aggregates it from the different DIGIT modules
Posts the data to the National Dashboard for the state
Bookkeeping is done for every adaptor data extract and pushes for audit and debugging
Out of scope: extraction from non-DIGIT sources
A national dashboard adaptor extracts data from the state DSS at a scheduled time which can be configured and then would ingest in the National Dashboard. The adaptor ingests data at the state/ULB/Ward level for each module on a daily basis. The adapter sends the data in a batch size of 50 to the national dashboard.
1. Deploy the Airflow on the state side to extract data from the Elastic Search server to push to the National Dashboard adaptor.
2. Configure the airflow with the required connection ids and the variables.
es_conn: To connect to the required ES server
digit-auth: To connect to the required API to insert data.
Variables: The credentials to connect to the API.
3. The index must contain data till the ward level and the structure must be similar to ensure the queries fetch the desired results.
4. Pull data from the DIGIT source only.
Builds deployed
national-dashboard-ingest-db: nda-patch-db6cb27b02-18
national-dashboard-kafka-pipeline:v0.0.1-762c61e743-3
Upyog devOps and MDMS changes
- upyog devOps PR
- upyog mdms PR
Add localisation for newly loaded Punjab tenants
Added NDA_SYSTEM role action mapping in MDMS and created a user with access rights of the same
Credentials: SYSTEMSU1 / eGov@123 - PG, user type: SYSTEM
Druid DIGIT
Metabase DIGIT
eGov@4321
Code base:
The command for executing the DAG:
{"start":"01-04-2022","end":"04-04-2022"}
Select the rev_max DAG right now deployed in the Punjab production
DAG has 3 steps
collect-data step:
The data is collected from ES and is converted to a JSON file.
join_data step:
The json file is converted to CSV.
The tables are joined like water and trade, water and property and trade and property.
The outcome is CSV files.
upload_data step:
The CSV files which are created are then uploaded to the druid by using the API.
The CSV files are converted to the format expected by the API and then uploaded to druid tables.
The objective of the audit service is listed below -
To provide a one-stop framework for signing data i.e. creating an immutable data entry to track activities of an entity. Whenever an entity is created/updated/deleted the operation is captured in the data logs and is digitally signed to protect it from tampering.
Prior knowledge of Java/J2EE
Prior knowledge of SpringBoot
Prior knowledge of PostgreSQL
Prior knowledge of REST APIs and related concepts like path parameters, headers, JSON etc.
The audit service will be parsing all the persister configs so that it can process data received by the persister and create audit logs out of it.
Step 1: Add the following metrics to the existing persister configs -
Step 2: If a custom implementation of ConfigurableSignAndVerify
interface is present, provide the signing algorithm implementation name as a part of audit.log.signing.algorithm
property. For example, if the signing algorithm is HMAC, the property will be set as follows -
Step 3: Set egov.persist.yml.repo.path
this property to the location of persister configs.
Step 4: Run the audit-service application along with the persister service.
Definitions
Config file - A YAML (xyz.yml) file which contains persister configuration for running audit service.
API - A REST endpoint to post audit logs data.
When audit-service create API is hit, it will validate request size, keyValueMap and operationType.
Upon successful validation, it will choose the configured signer and sign entity data.
Once audit logs are signed and ready, it will send it to audit-create
topic.
Persister will listen on this topic and persist the audit logs.
Add the required keys for enabling audit service in persister configs.
Deploy the latest version of the Audit service and Persister service.
Add Role-Action mapping for APIs.
The audit service is used to push signed data for tracking each and every create/modify/delete operation done on database entities.
Can be used to have tamper-proof audit logs for all database transactions.
Replaying events in chronological order will lead to the current state of the entity in the database.
To integrate, the host of the audit-service module should be overwritten in the helm chart.
audit-service/log/v1/_create
should be added as the create endpoint for the config added.
audit-service/log/v1/_search
should be added as the search endpoint for the config added.
1. URI: The format of the API to be used to create audit logs using the audit service is as follows: audit-service/log/v1/_create
Body: The body consists of 2 parts: RequestInfo and AuditLogs.
Sample Request Body -
2. URI: The format of the API to be used to search audit logs using audit-service is as follows: audit-service/log/v1/_search
Body: The body consists RequestInfo and search criteria is passed as query params.
Sample curl for search -
The signed audit service provides a one-stop framework for signing data i.e. creating an immutable data entry to track activities of an entity. Whenever an entity is created/updated/deleted the operation is captured in the data logs and is digitally signed to protect it from tampering.
Infra configuration -
Number of concurrent users - 15000
Duration of bombarding service with requests ~ 17 minutes
Number of signed audit pod(s) - 1
The inbox service is an event-based service that fetches pre-aggregated data of municipal services and workflow performs complex search operations and returns applications and workflow data in a paginated manner. The service also returns the total count matching the search criteria.
The first step is to capture pre-aggregated events for the module which needs to be enabled in event-based inbox. This index needs to hold all the fields against which a search needs to be performed and any other fields which need to be shown in the UI when applications are listed.
Now, this service allows to search both the module objects as well as processInstance
(Workflow record) based on the provided criteria for any of the municipal services. For this, it uses a module-specific configuration. A sample configuration is given below -
Inside each query configuration object, we have the following keys -
module
- Module code for which inbox has been configured.
index
- Index where pre-aggregated events are stored.
allowedSearchCriteria
- This lists out various parameters on which searching is allowed for the given module
sortBy
- Specifies the field path inside the pre-aggregated record present in the index against which the result has to be sorted. Default order can be specified to ASC
or DESC
.
sourceFilterPathList
- This is a list which specifies the fields which should appear and which should not appear as part of the search result in order to avoid clutter and to improve query performance.
allowedSearchCriteria
- Within each object is going to have the following keys -
name
- Name of the incoming search parameter in the inbox request body.
path
- Path inside the pre-aggregated record present in the index against which the incoming search parameter needs to be matched.
isMandatory
- This specifies whether a particular parameter is mandatory to be sent in inbox search request or not.
operator
- This specifies which operator clause needs to be applied while forming ES queries. Currently, we support equality and range comparison operators only.
Once the configuration is added to this MDMS file, the MDMS service for that particular environment has to be restarted.
For modules where search has to be given on PII data like mobile number, a migration API needs to be written which will fetch data from the database, decrypt it, hash it using the encryption client and store it in the index to enable search.
For modules where a search has to be given on non-PII data, the indexer service’s _legacyIndex API can be invoked to move data to the new index.
Handler to manage and simplify service configurations
The objective of this module is to handle configurations of services in a centralised manner. The overall objective of the architecture is to simplify configuration handling and remove the need for manual pod restarts to refresh the configurations. The configurations that can be added by this handler are MDMS, pdf, persister, indexer, DSS, reports, workflow and localisation.
Currently, any developer who is working on a new service has to check in configuration at 6 different places (including reports, pdf templates etc.). After checking in the configuration the developer has to restart all of these 6 pods. Once this is done, the developer needs cURL for workflow and localisation service to insert their configuration/data. The whole process is long, tedious and error-prone (many times people forget to restart pods after adding config). This approach requires access to GitHub and Jenkins which is not possible in the case of a sandbox environment.
The handler will provide API endpoints to manage configurations. The structure of the API request is as follows:
The request contains an array of objects. Each object contains the module name and the configuration that has to be added.
The handler contains two APIs: _create and _update.
The _create API is used to add configuration for the first time. It works in an idempotent manner i.e if any configuration is already present it ignores it (will not add that particular config). This API is used to integrate modules. During application initialisation, the service calls the API to add the default configurations (if required).
The _update API is called for any further changes to the configuration.
Currently, services like Persister, Indexer, etc. use a git-sync container to load configuration files. Usually, the developer runs the services as sidecar containers. Each service requires one. The containers can be removed from all these services by enhancing the MDMS service. Using this approach only the MDMS service is required to run the git-sync pod.
Steps:
The service(s) loads the configuration files by calling the MDMS service.
MDMS service returns the requested configuration files.
Note:
We can prevent the existing resource loader logic to load from the resource folder. In addition to that, we can provide another type of resource loader that fetches from MDMS. Based on the environment variable it selects the loader.
Currently, MDMS data is in JSON format. We need to add support (if not present) for YAML files to serve the configuration from MDMS. (Another way is to change the configuration format from YAML to JSON).
The configuration handler integrates with Git and the required DIGIT services (workflow, localisation etc.) to add/update configuration. Once the configuration is added in Git, the pods using this configuration pull the latest changes. There are multiple approaches to achieving this. Below is the list of approaches with their pros and cons.
The git-sync runs as a sidecar container instead of an init container and periodically pulls the Git repo based on configured frequency. We can configure the frequency to around 1-2 mins.
Note: Git-sync checks the hash first and pulls the data only if the hash has changed.
Pros:
The approach is easy to implement without any need to write new code.
Cons:
Data is not refreshed instantly. There will be a lag of 1-2 mins. (Currently, it takes 3-4 min to refresh data as the pod needs to be redeployed).
Understand and modify the git-sync code to expose an endpoint for manual pull. Whenever the config handler uploads configs on Git it calls the API of the modified git-sync container to initiate the pull.
Pros:
Updated configurations will reflect within a few seconds.
Cons:
Will increase the development time and efforts.
Once the configuration is pulled by git-sync the services are notified to reload the configuration.
Git-sync provides webhooks which are called once the new hash is synced.
Add one /config/_refresh API in each of these services and configure them in git-sync. Whenever git-sync pulls data successfully it triggers _refresh API which in turn refreshes the configuration.
Since the _refresh API call goes to only one pod, only that pod will refresh the data. Because of this, we cannot use the local cache.
This requires a cluster-aware caching implementation and this can be done using the Redis instance that is already deployed. Move the configuration to Redis - even if one pod refreshes the cache it reflects in all the pods for that service.
For services like indexer and persister which load consumers from configuration, cache burst does not work. For this we need to use configuration versioning. In the cache, we store the configuration along with the hash associated with that version. Services like indexer and persister maintain a variable called localOperatingConfigHash in its heap memory. Whenever it fetches data from the Redis cache it compares the localOperatingConfigHash with the current hash(version) of the configuration. If it matches it goes ahead and performs the process. In case it is not the same, it reloads the consumers from the new configuration and updates the localOperatingConfigHash to the value it got from the Redis cache.
Another optimised approach is that the persister service fetches the configs when _refresh API is called. After fetching the config the service gets all the topics present in the configs and get a hash of that list of topics. This hash service is set in its local variable and adds it in the Redis along with the configs. When another pod fetches the config and hash from Redis it compares the hash with its local hash and based on that it restarts consumers. In this way, consumers are restarted only when topics are added or removed.
The tracer library can be updated to automatically connect with the config handler module during initialisation. The tracer module works by using a variable called tracer.config.path. This variable contains the path where the configuration json is present. By default, it points to the resource folder inside the code base, but for production use cases it can be overwritten with Git URL or any other valid file path as per requirement.
Helm Charts:
Should we think about adding the service helm chart as part of the module code base? We can think of generating it dynamically based on a moustache template.
2. Approaches to Deploy Config Handler (if we don’t want to provide _update API):
There are 3 ways we can deploy the config handler if we are trying to use it only by adding config during deployment:
Config handler code will be part of the tracer
We create a separate client for the config handler
We create a separate service and run it as an init container
If we want it to provide _update API as well then we need to deploy it as a separate microservice.
Title | Link |
---|---|
All content on this page by eGov Foundation is licensed under a Creative Commons Attribution 4.0 International License.
Title | Link |
---|---|
Input Field | Description | Mandatory | Data Type |
---|---|---|---|
Title | Link |
---|---|
Title | Link |
---|---|
Body: The body consists of 2 parts: RequestInfo and Data. Data is where the aggregated data to be ingested resides. The keys given under the metrics object here are metrics provided in module.fields.mapping
present in the configuration here - DIGIT-DevOps/qa.yaml at master · egovernments/DIGIT-DevOps
Detailed API payloads for interacting with Service Request for all four endpoints can be found in the following collection -
The link to the swagger documentation can be found below -
ConnectionId | Connection Type | Host | |||
---|---|---|---|---|---|
Postman collection for posting the data:
A new inbox service needs to be enabled via the configuration present in MDMS. The path to the MDMS configuration is - .
If any existing module needs to be migrated onto a new inbox, data needs to be , and configuration like the above needs to be written to enable these modules on the new inbox.
NotificationConsumer
API Postman Collection
tenantId
Unique id for a tenant.
Yes
String
mobileNumber
Mobile number of the user
Yes
String
type
OTP type ex: login/register/password reset
Yes
String
userType
Type of user ex: Citizen/Employee
No
String
API Swagger Documentation
API Swagger Documentation
es_conn
ElasticSearch
elasticsearch-data-v1.es-cluster
9200
For the ES server
digit -auth-state
HTTP
https
For the auth api conenction - Staging
digit-auth
For the auth api conenction - UPYOG
Punjab Kibana QA
admin
24!jcZ]z"[$qZ%Fa
Punjab Kibana UAT
OAuth authentication
Punjab Dashboard - QA
STADSS
STADSS
Punjab
Punjab Dashboard - UAT
EMP9
eGov@123
testing
National DSS - QA
QADSS
eGov@123
amritsar
National DSS - DEV
amr001
eGov@123
amritsar
National DSS - UAT
NDSS1/NDSS2
eGov@123
PG
National Dashbaord - UPYOG
NDSS1
eGov@123
PG
AUTH API :Prod
SYSTEMSU3
eGov@123
ZWdvdi11c2VyLWNsaWVudDo=
pb
AUTH API credentials:DEV
amr001/EMPLOYEE
eGov@123
ZWdvdi11c2VyLWNsaWVudDo=
pb.amritsar
AUTH API:UPYOG
SYSTEMSU1/SYSTEM
eGov@123
pg
AUTH API:Staging
SYSTEMSU3/
SYSTEM
eGov@123
pg
password
eGov@123
username
SYSTEMSU1
for Upyog
username_state
SYSTEMSU3
for staging
token
ZWdvdi11c2VyLWNsaWVudDo=
tenantid
pg
usertype
SYSYTEM
totalulb_url
for reading the ulb’s of punjab