Configuring a report for a module requires adding the required report configuration as per the standard format and with the minimum development time.
UI can have different types of filters such as date, dropdown etc.. and even the sum of a column can also be easily displayed in UI. Pagination and downloading the report in pdf format, xls format is already present in the report UI.
Type of reports which can be configured :
Count of applications
Statewide collections
Application status
Cancelled receipts
Migrated records / Data entry records
The limitation of this framework is for reports having requirements with complex queries with multiple joins as the report uses the query to fetch the data from the database, It is resource-intensive and response might be slow in those scenarios.
Before you proceed with the configuration, make sure the following pre-requisites are met -
User with permission to edit the git repository to add the report configuration.
User with permission to add action and role action in the MDMS.
Showcase the data in the required and cleaner format.
The UI is rendered with the help of configuration in the report and there is no extra effort in building UI for different reports.
For Implementation specific report requirements, customization is easy and turnaround time is less.
After adding the new report/ editing the existing report configuration in the respective module, the report service needs to be restarted.
Create a reports.yml file and add report configuration as per standard format.
Add the action and role action in the MDMS.
Add the GitHub raw path of the report.yml file in the report.config file.
| Title |
|---|
Through report service, useful data get shown for a specific module based on some given criteria like date, locality, financial year, etc.
For example, PT dump report of property tax service you have to select from date to date, financial year etc and based on the criteria we can see all the data fulfilling the criteria. In the response, we see all the details of a property which is paid between the given from date and to date. In case we select the financial year, the property which is paid for that specific financial year is visible.
Before you proceed with the configuration, make sure the following pre-requisites are met -
User with permission to edit the git repository where Reports are configured and knowledge on YAML.
Prior Knowledge of YAML.
Prior Knowledge of SQL queries.
Prior Knowledge of the relation between the tables for which module you are going to write a report.
Users can write queries (like SQL queries) for fetching real-time data to display in a UI application.
Users can apply filters like from date, to date, financial year, etc based on the report configuration.
Users can download the result in PDF and XLS format.
User can select or deselect the columns user wants to see.
User can choose the number of records he/she wants to see on a page.
Once the changes have been done in the report configuration file we have to restart the report service so the report service will read the new configuration.
<Module Name>=file:///work-dir/configs/reports/config/<report file name>.yml
ex: pgr=file:///work-dir/configs/reports/config/pgr-reports.yml
Create a new file and name the file that you have given in the file reportFileLocationsv1.
Write the report configuration. Once it is done commit those changes.
Add the role and actions for the new report.
Restart the MDMS and report service.
To add a new report first add the file path in the . (In this file, the path of the report configuration files gets stored).
Once the file path is added in the file reportFileLocationsv1, go to the folder .
| Title |
|---|
Rainmaker has a reporting framework to configure new reports. As part of the report configuration, we have to write a native SQL query to get the required data for the report. So if the query takes huge time to execute or the query result has huge data, then it will impact the whole application's performance.
The following cases are where we can see the application performance issue because of heavy reports.
Filtering with long date range data or applying fewer filters which in turn returns huge data.
Join the multiple tables for getting required data and missing creating an index on join columns.
Implementing conditional logic inside the queries itself.
Writing multiple sub-queries inside a single query for getting the required data.
Because of heavy reports, the following are the impacts on the platform -
When we execute a complex query on the database, a thread from the connection pool will block to execute the query.
When threads from the connection pool are blocked completely, the application will become very slow for incoming requests.
When max request timeout is crossed, the API gateway will return a timeout error, But still, the connection thread on the database is active, Then all these types of idle threads will occupy database resources like memory, and CPU which in turns increase the load on the database.
Sometimes when running huge queries, the time taken by the query will lead to a broken pipe issue which causes more memory leaks and out-of-heap memory type issues. Because of this, the service will frequently restart automatically.
If a query returns huge data, the browser will become unresponsive and the application will become unresponsive.
Reporting Service is a service running independently on a separate server. The main objective of this service is to provide a common framework for generating reports. This service loads the report configuration from a yaml file at the run time and provides the report details by using a couple of APIs.
Before you proceed with the documentation, make sure the following pre-requisites are met -
Prior knowledge of Java/J2EE.
Prior knowledge of SpringBoot.
Advanced knowledge of PostgreSQL.
Encryption and MDMS services must be running.
Prior knowledge of REST APIs and related concepts like path parameters, headers, JSON etc.
JSON path for filtering required data from json objects.
Provides an easy way to add reports on the fly just by adding configurations without any coding effort.
Provides flexibility to customise result column names in the config.
Provides flexibility to fetch data from DB and also from some other services returning required json objects when its not possible to get all required data from DB.
Provides functionality to add filters as per requirements before actually fetching data for reports.
Provides user data in masked form. People with authorization will get plain data one by one.
Config file
A YAML (xyz.yml) file which contains configuration for report requirements.
API
A REST endpoint to fetch data based on the configuration.
Inline-table
If we also want to show data from some external service with data coming from DB in reports we use inline tables. The data from an external service is stored in an inline table and then used as any normal DB table to get data. This table is short-lived and stays only for the time when the query is being executed. It is never stored in DB. We provide JSON paths in an ordered manner corresponding to each column in the table. These JSON paths will be used to extract the required data from the external service’s response. For configs please see the ‘How to Use’ section.
Configuration: As mentioned above, the report service uses a config file per module to store all the configurations of reports pertaining to that module. Report service reads multiple such files at start-up to support reports of all the configured modules. The file contains the following keys:
reportName: name of the report, to be used with module name to identify any report config
summary: summary of the report
version: version of the report
moduleName: name of the module to which the report belongs to
externalService: To be used when some of the report data needs to be fetched from external service through inline tables. It contains the following fields
entity: JSON Path to filter json arrays(result to be turned into tables) from returned json
apiURL: API URL of the external service
keyOrder: order of JSON object keys to form table columns from JSON object arrays
tableName: name to be given to represent this transformed data which will be used as a table in the SQL query
sourceColumns: These represent the final data sent by the service on GET_DATA API call. The order of sourceColumns in the Config is the same as that of columns in the result. Each sourceColumns represents one column in the result. For each column, data is picked after executing the final SQL query formed after appending groupby, orderby, and search params into the base query.
name: name of the column to fetch data from query results, must be there in query results
label: custom column label
type: data type of column
source: module name
total: whether column total is required on the front end
searchParams:
name: name of search param. Must match variable used in search clause
label: the custom label for viewing on the front end
type: type of search params. If the type is ‘singlevaluelist’ then use the pattern to populate searchparams possible values to select from by the user Ex:-number, string, singlevaluelist etc.
source: module name
isMandatory: If the user must fill this searchparam before requesting report data
searchClause: SQL search clause for corresponding search params to filter results, to be appended in base query Ex:- AND fnoc.tenantId IN ($ulb). Here $ulb will be replaced by user inputs
Pattern: This field will be used only when ‘type’ is set to ‘singlevaluelist’. It is an external service URL combined with JSON Paths separated by ‘|’. The first JSON path is for codes and the second is for values. Values will be shown to the user in the dropdown. And codes corresponding to user selected value will be sent to the report service and will be used in searchClauses.
Query: Main/base query clause for fetching report data from DB and custom tables formed after fetching data from external service
Orderby: order by clause to be appended into base query
Groupby: group by clause to be appended into base query
additionalConfig: to provide additional custom configs which are not present above
Call the MDMS or any other API with the post method
Configuring the post object in the yaml itself like below.
externalService:
entity: $.MdmsRes.egf-master.FinancialYear
keyOrder: finYearRange,startingDate,endingDate,tenantId
tableName: tbl_financialyear
stateData: true
postObject:
tenantId: $tenantid
moduleDetails:
moduleName: egf-master
masterDetails:
name: FinancialYear filter: "[?(@.id IN [2,3] && @.active == true)]"
Keep the post object in a separate JSON file externally and call at runtime.
There are two API calls to report service ‘GET METADATA’ and ‘GET DATA’.
GET METADATA
This request to report service is made to get metadata for any report. The metadata contains information about search filters to be used in the report before actually sending a request to get actual data. The user-selected values are then used in the GET_DATA request to filter data.
endpoint: /report/{moduleName}/{report name}/metadata/_get
moduleName:- It is used to define the names of the module which contain current report
Body: The body consists of the following:
RequestInfo: Header details as used on the egov platform
tenantId: tenantId of ULB
reportName: name of the report to be used
Instance:
URL: https://{domain name}/report/rainmaker-tl/metadata/_get
Body:
GET DATA
This request-to-report service is used to get data for the report. Inputs given by the user for filters are sent in the request body. These filter values are used while querying data from DB.
endpoint: report/{moduleName}/{report name}/_get
moduleName: It is used to define the names of the module which contains the current repo
Body: The body consists of the following:
RequestInfo: Header details as used on the egov platform
tenantId: tenantId of ULB
reportName: name of the report to be used
The array of searchparams corresponds to each of the filled filters by the user. Each searchparam contains:-
Name: name of the filter
Input: user selected value
Instance:
URL: https://{domain name}/report/rainmaker-tl/_get
Body:
Write configuration as per your requirement. The structure of the config file is explained in the configuration details section.
Check-in the config file to a remote location preferably GitHub, currently we check the files into the folder here. Can't find links for the dev and QA environment.
Add the module name and corresponding report path in the same format as used here.
Provide the absolute path of the file mentioned in Point 3 to DevOps, to add it to the file-read path of the report service. The file will be added to the environment manifest file for it to be read at the start-up of the application.
Deploy the latest version of the report service app.
Add role-action mapping for APIs.
Use the module name as a path parameter in the URL of the requests for report service with the required request body.
The report service provides a common framework to generate reports and show the report details based on the search criteria.
Provide a common framework for generating reports
Provide functionality to create new ad-hoc reports with minimal efforts
Avoid writing code again in case of new report requirements
Makes possible to create reports with only knowledge of SQL and JSONPath
Provides metadata about the report.
Provides the data for the report.
Reload the configuration at runtime.
To integrate, the host of echallan-services module should be overwritten in the helm chart.
The API should be mentioned in ACCESSCONTROL-ACTIONS-TEST. Refer below example
Add role-action mapping for APIs.
Note: Refer to the guidelines for supporting user privacy in a module, for the changes required in the report config to support the encryption/decryption process.
| Title |
|---|
Local Setup