To make sure that after enabling privacy, the system works as expected, we will require some configurations to be made in the environment. This document contains all the steps to ensure successful implementation and working of the Water & Sewerage module.
Steps
The following are the changes required to move the water and sewerage application to other environments:
Add the following json mappings in the existing mappings (parallel to water-services and sewerage-services key) for water-services and sewerage-services in kibana so that the PII data is not visible during search(The data do remain in the index and also search with respect to this happens as is).
In the params list in both the above curls, “tenantIds” param can either be provided with a single tenantId or a list of tenantIds for encrypting the data with respect to the provided tenantIds. However, to encrypt the data for all tenantIds in the system, tenantIds param itself should be removed.
To validate if the encryption is completed, you can check with the following dB queries:
select * from eg_ws_enc_audit order by createdtime desc;
select count(*) from eg_ws_id_enc_audit;
This query can validate whether all records are there or not. The count should match the total count of records in the eg_ws_connection table.
select * from eg_ws_id_enc_audit;
This can help you check what all properties have been updated so far. This table contains the id, applicationnumber, connectionnumber and tenantid.
Sample Index for point 5:
PUT water-services
{
}
Water & Sewerage Privacy Changes
Backend technical documentation
Overview
This document highlights the changes that need to be done in a Water and Sewerage module to support the privacy feature.
Pre-requisites
Prior knowledge of Java/J2EE
Prior knowledge of Spring Boot
MDMS service
Encryption service
MDMS Changes
Update the SecurityPolicy.json file
The Security Policy MDMS file must have the model configuration for fields to be encrypted/decrypted.
The following models have been used for W&S in the SecurityPolicy.json file:
For modules, there is no such hard rule or pattern for naming the model, but it should be related to that service.
ex: 1.) For user service we have two security policy models User which is used when a user tries to search other user data. The PII data will be available in masked, plain or encrypted form depending on the visibility set for an attribute in the MDMS.
The other model is UserSelf which is used when a user tries to search their own data. The data is available as per the configuration set there.
For report and searcher config the model name should be similar to the value that we are setting in the field decryptionPathId. ex:- Employee Report. Employee Report Security Model.
2.) For W&S we have 2 models -WnSConnection and WnSConnectionOwner which are meant for ConnectionHolderDetails.
2 different models for connectionHolder data are required since some data for connectionHolder comes from the W&S tables and the remaining is fetched from the User table. So to maintain consistency we had to have 2 different models.
For an attribute where its firstLevelVisibility is set as "Masked" whenever the respective search API is called without the plain Access Request, the masked value is returned as the API response for that attribute.
for ex:- if for mobile number attribute’s firstLevelVisibility is masked and its plain value is 9089243280 then in response, the value is displayed as ******3280. The masking pattern is defined in the MaskingPatternMDMS file and the pattern is picked up based on the patternId. Similarly, if the firstLevelVisibility is set as "ENCRYPTED" we will get the encrypted value of that plain data (which is present in DB) in response.
NOTE:Foradding of new attribute for encryption, the following things need to be kept in mind:
We do not have a direct approach to it, but a workaround is as follows:
We need to make sure which property has to be encrypted and what is the path of the property in the Request/Response object of W&S. If PII data is for connectionholder and is coming from WnS tables directly, then with the Proper name and path of the object we can add a new Property in existing model WnSConnection.
The inclusion of any new attribute here would need encryption of the old data for this new property.
For that, in MDMS, we will have to replace the existing model attributes with only new attributes and hit the _encryptOldData API. Once old data encryption is done, we can put back all the required attributes (old and +new) in the model.
Also, before starting the encryption of old data, we will have to check the latest record of the table eg_ws_enc_audit / eg_sw_enc_audit / eg_pt_enc_audit(for PT) . If the latest record has offset and record count value other than 0 then insert a random record with offset and record count as 0 and createdtime and encryptiontime as a current timestamp in millis in utc.
Any data other than that of connectionHolder would need a new model to be created and changes at the code level as well.
For old data encryption of new property other than that of connectionHolder, we will have to have code changes as well.
In ws-services encryption and decryption endpoints should be declared as follows:
#--------enable/disable ABAC in encryption----------#
water.decryption.abac.enabled=true
encryption.batch.value=500
encryption.offset.value=0
#-------Persister topics for oldDataEncryption-------#
egov.waterservice.oldDataEncryptionStatus.topic=ws-enc-audit
egov.waterservice.update.oldData.topic=update-ws-encryption
In sw-services encryption and decryption endpoints should be declared as follows:
#--------enable/disable ABAC in encryption----------#
sewerage.decryption.abac.enabled=true
encryption.batch.value=500
encryption.offset.value=0
#-------Persister topics for oldDataEncryption-------#
egov.sewerageservice.oldDataEncryptionStatus.topic=sw-enc-audit
egov.sewerageservice.update.oldData.topic=update-sw-encryption
EncryptionDecryptionUtil.java
We need an interfacing file for handling the encryption and decryption of attributes and for interacting with the enc-client library directly.
For reference follow the below code snippet:
package org.egov.waterconnection.util;
import com.fasterxml.jackson.databind.ObjectMapper;
import lombok.extern.slf4j.Slf4j;
import org.egov.common.contract.request.RequestInfo;
import org.egov.common.contract.request.User;
import org.egov.encryption.EncryptionService;
import org.egov.encryption.audit.AuditService;
import org.egov.tracer.model.CustomException;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Component;
import org.springframework.web.client.HttpClientErrorException;
import org.springframework.web.client.HttpServerErrorException;
import org.springframework.web.client.ResourceAccessException;
import java.io.IOException;
import java.util.*;
import static org.egov.waterconnection.constants.WCConstants.*;
@Slf4j
@Component
public class EncryptionDecryptionUtil {
private EncryptionService encryptionService;
@Autowired
private AuditService auditService;
@Autowired
private ObjectMapper objectMapper;
@Value(("${state.level.tenant.id}"))
private String stateLevelTenantId;
@Value(("${water.decryption.abac.enabled}"))
private boolean abacEnabled;
public EncryptionDecryptionUtil(EncryptionService encryptionService) {
this.encryptionService = encryptionService;
}
public <T> T encryptObject(Object objectToEncrypt, String key, Class<T> classType) {
try {
if (objectToEncrypt == null) {
return null;
}
T encryptedObject = encryptionService.encryptJson(objectToEncrypt, key, stateLevelTenantId, classType);
if (encryptedObject == null) {
throw new CustomException("ENCRYPTION_NULL_ERROR", "Null object found on performing encryption");
}
return encryptedObject;
} catch (Exception e) {
log.error("Unknown Error occurred while encrypting", e);
throw new CustomException("UNKNOWN_ERROR", "Unknown error occurred in encryption process");
}
}
public <E, P> P decryptObject(Object objectToDecrypt, String key, Class<E> classType, RequestInfo requestInfo) {
try {
boolean objectToDecryptNotList = false;
if (objectToDecrypt == null) {
return null;
} else if (requestInfo == null || requestInfo.getUserInfo() == null) {
User userInfo = User.builder().uuid("no uuid").type("EMPLOYEE").build();
requestInfo = RequestInfo.builder().userInfo(userInfo).build();
}
if (!(objectToDecrypt instanceof List)) {
objectToDecryptNotList = true;
objectToDecrypt = Collections.singletonList(objectToDecrypt);
}
Map<String, String> keyPurposeMap = getKeyToDecrypt(objectToDecrypt, key);
String purpose = keyPurposeMap.get("purpose");
if (key.equalsIgnoreCase(WNS_ENCRYPTION_MODEL) || key.equalsIgnoreCase(WNS_OWNER_ENCRYPTION_MODEL) || key.equalsIgnoreCase(WNS_PLUMBER_ENCRYPTION_MODEL))
key = keyPurposeMap.get("key");
P decryptedObject = (P) encryptionService.decryptJson(requestInfo, objectToDecrypt, key, purpose, classType);
if (decryptedObject == null) {
throw new CustomException("DECRYPTION_NULL_ERROR", "Null object found on performing decryption");
}
if (objectToDecryptNotList) {
decryptedObject = (P) ((List<E>) decryptedObject).get(0);
}
return decryptedObject;
} catch (IOException | HttpClientErrorException | HttpServerErrorException | ResourceAccessException e) {
log.error("Error occurred while decrypting", e);
throw new CustomException("DECRYPTION_SERVICE_ERROR", "Error occurred in decryption process");
} catch (Exception e) {
log.error("Unknown Error occurred while decrypting", e);
throw new CustomException("UNKNOWN_ERROR", "Unknown error occurred in decryption process");
}
}
public Map<String, String> getKeyToDecrypt(Object objectToDecrypt, String key) {
Map<String, String> keyPurposeMap = new HashMap<>();
if (!abacEnabled) {
if (key.equals(WNS_ENCRYPTION_MODEL) || key == null) {
keyPurposeMap.put("key", "WnSConnectionDecrypDisabled");
keyPurposeMap.put("purpose", "WnSConnectionDecryptionDisabled");
} else if (key.equals(WNS_OWNER_ENCRYPTION_MODEL)) {
keyPurposeMap.put("key", "WnSConnectionOwnerDecrypDisabled");
keyPurposeMap.put("purpose", "WnSConnectionDecryptionDisabled");
} else if (key.equals(WNS_PLUMBER_ENCRYPTION_MODEL)) {
keyPurposeMap.put("key", "WnSConnectionPlumberDecrypDisabled");
keyPurposeMap.put("purpose", "WnSConnectionPlumberDecrypDisabled");
}
} else {
if (key.equals(WNS_ENCRYPTION_MODEL) || key == null) {
keyPurposeMap.put("key", WNS_ENCRYPTION_MODEL);
keyPurposeMap.put("purpose", "WnSConnectionSearch");
} else if (key.equals(WNS_OWNER_ENCRYPTION_MODEL)) {
keyPurposeMap.put("key", WNS_OWNER_ENCRYPTION_MODEL);
keyPurposeMap.put("purpose", "WnSConnectionSearch");
} else if (key.equals(WNS_PLUMBER_ENCRYPTION_MODEL)) {
keyPurposeMap.put("key", WNS_PLUMBER_ENCRYPTION_MODEL);
keyPurposeMap.put("purpose", "WnSConnectionPlumberSearch");
}
}
return keyPurposeMap;
}
}
UnmaskingUtil.java file helps in extracting the plain data of the Owner from the User service.
This file also contains a method that swaps the PlumberInfo in a similar manner.
For reference follow the below code snippet:
package org.egov.waterconnection.util;
import java.util.*;
import java.util.function.Function;
import java.util.stream.Collectors;
import java.util.stream.StreamSupport;
import org.egov.common.contract.request.PlainAccessRequest;
import org.egov.common.contract.request.RequestInfo;
import org.egov.waterconnection.web.models.OwnerInfo;
import org.egov.waterconnection.service.UserService;
import org.egov.waterconnection.web.models.PlumberInfo;
import org.egov.waterconnection.web.models.WaterConnection;
import org.egov.waterconnection.web.models.users.UserDetailResponse;
import org.egov.waterconnection.web.models.users.UserSearchRequest;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;
import org.springframework.util.ObjectUtils;
import org.springframework.util.StringUtils;
@Component
public class UnmaskingUtil {
private static List<String> plainRequestFieldsList;
@Autowired
private UserService userService;
public void getOwnerDetailsUnmasked(WaterConnection waterConnection, RequestInfo requestInfo) {
PlainAccessRequest apiPlainAccessRequest = requestInfo.getPlainAccessRequest();
List<String> plainRequestFieldsList = getAllFieldsPlainAccessList();
PlainAccessRequest plainAccessRequest = PlainAccessRequest.builder()
.plainRequestFields(plainRequestFieldsList)
.build();
requestInfo.setPlainAccessRequest(plainAccessRequest);
UserSearchRequest userSearchRequest = userService.getBaseUserSearchRequest(waterConnection.getTenantId(), requestInfo);
Set<String> ownerIds = new HashSet<>();
for (OwnerInfo ownerInfo : waterConnection.getConnectionHolders()) {
String currentOwnerId = ownerInfo.getUuid();
/*
* once user module is updated to handle masked update
* users will be unmasked on need, currently all.
*/
ownerIds.clear();
ownerIds.add(currentOwnerId);
userSearchRequest.setUuid(ownerIds);
plainAccessRequest.setRecordId(currentOwnerId);
UserDetailResponse userDetailResponse = userService.getUser(userSearchRequest);
if(!ObjectUtils.isEmpty(userDetailResponse.getUser())) {
OwnerInfo unmaskedUser = userDetailResponse.getUser().get(0);
updateMaskedOwnerInfoWithUnmaskedFields(ownerInfo, unmaskedUser);
}
requestInfo.setPlainAccessRequest(apiPlainAccessRequest);
}
}
private void updateMaskedOwnerInfoWithUnmaskedFields(OwnerInfo ownerInfo, OwnerInfo unmaskedUser) {
if (ownerInfo.getFatherOrHusbandName().contains("*")) {
ownerInfo.setFatherOrHusbandName(unmaskedUser.getFatherOrHusbandName());
}
if (ownerInfo.getMobileNumber().contains("*")) {
ownerInfo.setMobileNumber(unmaskedUser.getMobileNumber());
}
if (ownerInfo.getCorrespondenceAddress().contains("*")) {
ownerInfo.setCorrespondenceAddress(unmaskedUser.getCorrespondenceAddress());
}
if (ownerInfo.getUserName().contains("*")) {
ownerInfo.setUserName(unmaskedUser.getUserName());
}
if (ownerInfo.getName().contains("*")) {
ownerInfo.setName(unmaskedUser.getName());
}
if (ownerInfo.getGender().contains("*")) {
ownerInfo.setGender(unmaskedUser.getGender());
}
}
public static List<String> getAllFieldsPlainAccessList() {
if (plainRequestFieldsList == null) {
plainRequestFieldsList = new ArrayList<>();
plainRequestFieldsList.add("mobileNumber");
plainRequestFieldsList.add("guardian");
plainRequestFieldsList.add("fatherOrHusbandName");
plainRequestFieldsList.add("correspondenceAddress");
plainRequestFieldsList.add("userName");
plainRequestFieldsList.add("name");
plainRequestFieldsList.add("gender");
}
return plainRequestFieldsList;
}
/**
* Method returns true then owner info is modified for update,
* it should be unmasked for unchanged field updates
*
* @param ownerInfo
* @return Boolean
*/
private Boolean shouldOwnerBeUnmaksed(OwnerInfo ownerInfo) {
return !(ownerInfo.getFatherOrHusbandName().contains("*") &&
ownerInfo.getMobileNumber().contains("*") &&
ownerInfo.getCorrespondenceAddress().contains("*") &&
ownerInfo.getUserName().contains("*") &&
ownerInfo.getName().contains("*") &&
ownerInfo.getGender().contains("*"));
}
/**
* Method returns unmasked PlumberInfo,
* it should be unmasked for unchanged field updates
*
* @param plumberInfos
* @param unmaskedPlumberInfos
*/
public boolean getUnmaskedPlumberInfo(List<PlumberInfo> plumberInfos, List<PlumberInfo> unmaskedPlumberInfos) {
Map<String, PlumberInfo> unmaskedPlumberInfoMap;
boolean isPlumberSwapped = false;
if (!ObjectUtils.isEmpty(unmaskedPlumberInfos)) {
unmaskedPlumberInfoMap = unmaskedPlumberInfos.stream().collect(Collectors.toMap(PlumberInfo::getId, Function.identity()));
if (!ObjectUtils.isEmpty(plumberInfos)) {
for (PlumberInfo plumberInfo : plumberInfos) {
if (!StringUtils.isEmpty(plumberInfo.getMobileNumber()) && plumberInfo.getMobileNumber().contains("*")) {
if (!StringUtils.isEmpty(unmaskedPlumberInfoMap.get(plumberInfo.getId()))) {
plumberInfo.setMobileNumber(unmaskedPlumberInfoMap.get(plumberInfo.getId()).getMobileNumber());
isPlumberSwapped = true;
}
}
}
}
}
return isPlumberSwapped;
}
}
During the search call the PII data needs to be masked in the response received.
For this, the searchCriteria needs to be encrypted before fetching details and eventually the data retrieved needs to be decrypted and then returned to the user.
Following changes need to be made in WaterServiceImpl.java
public List<WaterConnection> search(SearchCriteria criteria, RequestInfo requestInfo) {
List<WaterConnection> waterConnectionList;
//Creating copies of apiPlainAcessRequests for decryption process
//Any decryption process returns the requestInfo with only the already used plain Access Request fields
PlainAccessRequest apiPlainAccessRequest = null, apiPlainAccessRequestCopy = null;
if (!ObjectUtils.isEmpty(requestInfo.getPlainAccessRequest())) {
PlainAccessRequest plainAccessRequest = requestInfo.getPlainAccessRequest();
if (!StringUtils.isEmpty(plainAccessRequest.getRecordId()) && !ObjectUtils.isEmpty(plainAccessRequest.getPlainRequestFields())) {
apiPlainAccessRequest = new PlainAccessRequest(plainAccessRequest.getRecordId(), plainAccessRequest.getPlainRequestFields());
apiPlainAccessRequestCopy = new PlainAccessRequest(plainAccessRequest.getRecordId(), plainAccessRequest.getPlainRequestFields());
}
}
/* encrypt here */
if (criteria.getIsInternalCall()) {
criteria = encryptionDecryptionUtil.encryptObject(criteria, "WnSConnectionDecrypDisabled", SearchCriteria.class);
criteria = encryptionDecryptionUtil.encryptObject(criteria, "WnSConnectionPlumberDecrypDisabled", SearchCriteria.class);
} else {
criteria = encryptionDecryptionUtil.encryptObject(criteria, WNS_ENCRYPTION_MODEL, SearchCriteria.class);
criteria = encryptionDecryptionUtil.encryptObject(criteria, WNS_PLUMBER_ENCRYPTION_MODEL, SearchCriteria.class);
}
waterConnectionList = getWaterConnectionsList(criteria, requestInfo);
if (!StringUtils.isEmpty(criteria.getSearchType()) &&
criteria.getSearchType().equals(WCConstants.SEARCH_TYPE_CONNECTION)) {
waterConnectionList = enrichmentService.filterConnections(waterConnectionList);
/*if(criteria.getIsPropertyDetailsRequired()){
waterConnectionList = enrichmentService.enrichPropertyDetails(waterConnectionList, criteria, requestInfo);
}*/
}
if ((criteria.getIsPropertyDetailsRequired() != null) && criteria.getIsPropertyDetailsRequired()) {
waterConnectionList = enrichmentService.enrichPropertyDetails(waterConnectionList, criteria, requestInfo);
}
waterConnectionValidator.validatePropertyForConnection(waterConnectionList);
enrichmentService.enrichConnectionHolderDeatils(waterConnectionList, criteria, requestInfo);
enrichmentService.enrichProcessInstance(waterConnectionList, criteria, requestInfo);
// enrichmentService.enrichDocumentDetails(waterConnectionList, criteria, requestInfo);
requestInfo.setPlainAccessRequest(apiPlainAccessRequest);
/* decrypt here */
if (criteria.getIsInternalCall()) {
waterConnectionList = encryptionDecryptionUtil.decryptObject(waterConnectionList, "WnSConnectionPlumberDecrypDisabled", WaterConnection.class, requestInfo);
requestInfo.setPlainAccessRequest(apiPlainAccessRequestCopy);
return encryptionDecryptionUtil.decryptObject(waterConnectionList, "WnSConnectionDecrypDisabled", WaterConnection.class, requestInfo);
}
waterConnectionList = encryptionDecryptionUtil.decryptObject(waterConnectionList, WNS_PLUMBER_ENCRYPTION_MODEL, WaterConnection.class, requestInfo);
requestInfo.setPlainAccessRequest(apiPlainAccessRequestCopy);
return encryptionDecryptionUtil.decryptObject(waterConnectionList, WNS_ENCRYPTION_MODEL, WaterConnection.class, requestInfo);
}
plainAccessRequest contains :
1. recordId which will take the uuid of the user as an input for which PII data has to be unmasked, and
2. plainRequestFields will take an array of attributes that need to be unmasked. These attributes should comply with the attributes used in the Mdms SecurityPolicy.json’s models created.
For unmasking PlumberInfoMobileNumber, recordId will take the applicationno as value.
_create and _update API Changes
In _create and _update API calls the data needs to be encrypted before the data for application/connection is pushed to respective topics.
A sample code line for encrypting data in these calls is as follows:
Before using the privacy feature in any environment, the encryption of existing data in the dB should be done.
An API for the same is written in the service and needs to be triggered by port-forwarding the respective service pod.
The data encryption API uses the existing plainSearch method and the encryption shall take place tenantId-wise. If tenantId is not specified then all the tenants are picked from the MDMS repository and the encryption happens for all the tenants. However, if the tenantId is specified, then the encryption happens only for that tenantId.
The following method is added to execute the oldDataEncryption
In WaterController.java
/**
* Encrypts existing Water records
*
* @param requestInfoWrapper RequestInfoWrapper
* @param criteria SearchCriteria
* @return list of updated encrypted data
*/
/* To be executed only once */
@RequestMapping(value = "/_encryptOldData", method = RequestMethod.POST)
public ResponseEntity<WaterConnectionResponse> encryptOldData(@Valid @RequestBody RequestInfoWrapper requestInfoWrapper,
@Valid @ModelAttribute SearchCriteria criteria){
WaterConnectionResponse waterConnectionResponse = waterEncryptionService.updateOldData(criteria, requestInfoWrapper.getRequestInfo());
waterConnectionResponse.setResponseInfo(
responseInfoFactory.createResponseInfoFromRequestInfo(requestInfoWrapper.getRequestInfo(), true));
return new ResponseEntity<>(waterConnectionResponse, HttpStatus.OK);
}
There is no need to re-index the base indexes(viz. water-services/ sewerage-services) once the API for old data encryption is executed as this will happen during the API execution itself.
For any new data getting created, the new data will get saved in the encrypted format only in the indexes.
There are some changes in the Indexer file that need to be made for W&S to get the encrypted data from the external service like Property-service.
Note: Privacy decryption means: For any new application creation/ old application updation Internal data will be stored in the encrypted form itself, just that during decryption disabled it would give PLAIN text rather than masked one.
All the above changes need to be done for sw-services as well.
