Overview

DIGIT is designed to ensure it addresses all the issues linked to traditional application-based approach and enable sustainable digital transformation at speed and scale. DIGIT design principles also keep in mind the low state capacity in low and middle-income countries (LMICs) hence there is special emphasis on Minimal and Ease of Use.

Key Tenets

The foundational capabilities of the platform adhere to the following principles illustrated in the diagram below:

A brief explanation of each tenet is given below:

• Interoperable

  • Syntactic Interoperability

    1. Define common data formats and communication protocol

  • Semantic Interoperability

    1. Define common meanings of terms

• Unbundle

  • Loose Coupling

    1. Single responsibility

    2. Open closed principle

• Minimal

• Scalable

• Secure and Reliable

• Open and Standards Driven

• Ease of Use

• Inclusive

DIGIT core services are developed using the following technologies -

1. Java Spring Boot

2. Apache Kafka

3. Netflix Zuul

4. Postgres

5. Elastic Search

6. Docker

7. Kubernetes

8. Terraform

Click the links below to learn more about the open-source tools and API gateways used in DIGIT.

DIGIT consists of several reusable building blocks on top of which one can develop a citizen service delivery solution. Read on to learn more about the microservice building blocks architecture.

Microservice - Building Blocks

To develop a new service, one needs to create a microservice and make it available through the API gateway. The API gateway calls the User Service for authentication and the Access Service for authorisation. The service developer can configure the roles and map the roles and actions using the Master Data Management Service.

Citizen Dashboard - The service user interface can be developed as part of the Citizen Dashboard or can be an independent solution. The citizen can log in using a mobile number and OTP. They can apply for a new service using the service UI. The File Store Service allows users to upload relevant documentation.

The Persister Service stores the submitted application asynchronously into the registry. The PII data is encrypted using the Encryption Service before storing. All changes are digitally signed and logged using the Signed Audit Service (ongoing). The Indexer Service transforms and enriches the application data. The Indexer Service also strips the PII data and sends it to the Analytics Data Store(ElasticSearch).

The Dashboard Backend Service executes configured queries on the stripped data and makes the aggregated data available to the Dashboard Frontend for the administrator or employee view. The views are in accordance with the user role access which is also configurable.

The Billing Service generates a demand based on the calculation logic for the given service delivery. Based on this demand, a bill is generated against which the payment has to be made. The citizen can either make an online payment or can pay at the counter (offline payment). The Payment Gateway Service is called for online payments and this is integrated with third-party service providers. The service routes the citizen to the service provider's website and then back to the citizen's UI once the payment is successful.

The Collection service is the payment registry and records all the successful payments. For offline payments, a record is made in the collection service after the collection of the Cash/Cheque/DD/RTGS. The allowed payment modes are configurable. The PDF service is used to generate receipts based on a configurable template.

The service then triggers the Workflow Service to assign tasks for verification and approval. Workflows can be configured. The Employee Service allows employee registrations and enables them to log in using the Employee Dashboard. The dashboard displays the list of pending applications as per the employee's role. The employee can perform actions on these applications using the employee UI for the service.

As the status changes or actions are taken on the applications the relevant SMS and Email Notification Service sends the updates to the applicant. Once the application is approved, the applicant can download the final certificate which is generated using PDF Service.

DIGIT has multi-tenant configuration capabilities. The Location Service allows the configuration of the tenant hierarchy and maps multiple tenants like Country, State, District or State, Department, and Sub Department. Each tenant can have their own configurations for service, roles, workflows etc. This allows for variations across different agencies in tune with the local context.

The Language Service facilitates the support of multiple languages in DIGIT. This service stores the translations in multiple languages.

Various infrastructure resources required for DIGIT deployment can be provisioned from the AWS cloud manually. However, the best practice is to maintain consistency, automation, audit, reusability, and cost forecast. The use of infra-as-code using tools like terraform templates works well to this end.

Infrastructure Resources

1. Ingress Rules, WAF, SSL Termination

2. VPC

3. Subnets (Private/Public, CIDR, Security Groups)

4. EKS (Managed Kubernetes Service)

5. Worker Node Group (Autoscaling)

6. RDS (Managed Database Service)

7. NAT Gateway

8. Internet Gateway

9. Block Storage (EBS Volumes)

10. Object Storage (S3)

Continuous Integration/Continuous Deployment (CI/CD) software is the backbone of the modern DevOps environment. CI/CD bridges the gap between development and operations teams by automating the build, test, and deployment of applications.

Deployment Diagram

APIs - The Do's & Don'ts

The APIs developed on DIGIT adhere to specific conventions and principles. Below are the dos and don'ts to be observed when following these API principles.

1. Always define the Yaml for your APIs as the first thing using Open API 3 Standard ().

2. APIs path should be standardised as follows:

   • /{service}/{entity}/{version}/_create: This endpoint should be used to create the entity.

   • /{service}/{entity}/{version}/_update: This endpoint should be used to edit an entity which is already existing.

   • /{service}/{entity}/{version}/_search: This endpoint should be used to provide a search on the entity based on certain criteria.

   • /{service}/{entity}/{version}/_count: This endpoint should be provided to give a count of entities that match the given search criteria.

3. Always use POST for each of the endpoints.

4. Take most search parameters in the POST body only.

5. If query params for search need to be supported then make sure to have the same parameters in the POST body also and the POST body should take priority over query params.

6. Provide additional detail for objects in _create and _update APIs so that custom requirements can use these fields.

7. Each API should have a object in the request body at the top level.

8. Each API should have a object in the response body at the top level.

9. Mandatory fields should be the minimum for the APIs.

10. minLength and maxLength should be defined for each attribute.

11. Read-only fields should be called out.

12. Use common models already available in the platform in your APIs. Example -

    • (Citizen/ Employee/ Owner)

    • (Response sent in case of errors)

    • TODO: Add all the models here

13. For receiving files in an API, do not use binary file data. Instead, accept the file store ids.

14. If there is only one file to be uploaded and no persistence is needed, and no additional JSON data is to be posted, you can consider using direct file upload instead of using filestore id.

Overview

API Gateway provides a unified interface for a set of microservices so that clients do not need to know about all the details of microservices internals.

DIGIT uses as API Gateway. It serves as an edge service that proxies requests to multiple back-end services. It provides a unified “front door” to our ecosystem. This allows any browser, mobile app or any other user interface to consume underlying services.

Zuul Components

Zuul has mainly four types of filters that enable us to intercept the traffic in different timelines of the request processing for any particular transaction. We can add any number of filters for a particular URL pattern.

Zuul Features

1. Microservice authentication and security

2. Authorization

3. API Routing

4. Open APIs using Whitelisting

5. RBAC filter

6. Logout filter for the finance module

7. Property module tax calculation filter for firecess

8. Request enrichment filter:

   • Addition of co-relation id

   • Addition of authenticated user’s userinfo to requestInfo.

9. Error filter:

   • Error response formatting

Feature enhancements in the latest version -

1. Validation filter: to check if a tenant of a particular module is enabled or not.

2. Multi-tenancy validation filter: Take the tenant id from the Req body or Query Param and validate against the additional tenant role or primary tenant role.

3. DevOps efficiency: API response time logging and sending notifications if it is taking more time.

PHASE 1: REQUIREMENTS

• Call out the security requirements in PRD like a Sample functional requirement: user needs the ability to verify their contact information before they are able to renew their membership.

• Sample security consideration: users should be able to see only their own contact information and no one else’s.

PHASE 2: DESIGN

This phase translates in-scope requirements into a plan of what this should look like in the actual application. Here, functional requirements typically describe what should happen, while security requirements usually focus on what shouldn’t.

• Sample functional design: page should retrieve the user’s name, email, phone, and address from CUSTOMER_INFO table in the database and display it on the screen.

• Sample security concern: we must verify that the user has a valid session token before retrieving information from the database. If absent, the user should be redirected to the login page.

PHASE 3: DEVELOPMENT

When it’s time to actually implement the design and make it a reality, concerns usually shift to making sure the code is well-written from the security perspective. There are usually established secure coding guidelines as well as code reviews that double-check that these guidelines have been followed correctly. These code reviews can be either manual or automated using technologies such as .

That said, modern application developers can’t be concerned only with the code they write, because the vast majority of modern applications aren’t written from scratch. Instead, developers rely on existing functionality, usually provided by free open-source components to deliver new features and therefore value to the organization as quickly as possible. In fact, 90%+ of modern deployed applications are made of these open-source components. These open-source components are usually checked using tools.

Secure coding guidelines, in this case, may include:

• Using parameterized, read-only SQL queries to read data from the database and minimize chances that anyone can ever commandeer these queries for nefarious purposes

• Validating user inputs before processing data contained in them

• Sanitizing any data that’s being sent back out to the user from the database

• Checking open source libraries for vulnerabilities before using them

PHASE 4: VERIFICATION

The Verification phase is where applications go through a thorough testing cycle to ensure they meet the original design & requirements. This is also a great place to introduce automated security testing using a variety of technologies. The application is not deployed unless these tests pass. This phase often includes automated tools like CI/CD pipelines to control verification and release.

Verification at this phase may include:

• Automated tests that express the critical paths of your application

• Automated execution of application unit tests that verify the correctness of the underlying application

• Automated deployment tools that dynamically swap in application secrets to be used in a production environment

PHASE 5: MAINTENANCE & EVOLUTION

The story doesn’t end once the application is released. In fact, vulnerabilities that slipped through the cracks may be found in the application long after it’s been released. These vulnerabilities may be in the code developers wrote but are increasingly found in the underlying open-source components that comprise an application. This leads to an increase in the number of “zero-days”—previously unknown vulnerabilities that are discovered in production by the application’s maintainers.

These vulnerabilities then need to be patched by the development team, a process that may in some cases require significant rewrites of application functionality. Vulnerabilities at this stage may also come from other sources, such as external penetration tests conducted by ethical hackers or submissions from the public through what’s known as “bug bounty” programs. Addressing these types of production issues must be planned for and accommodated in future releases.

DIGIT also comes with three UI Frameworks

1. DIGIT UI Framework (React)

2. DIGIT UI Framework (Flutter)

3. DIGIT Dashboard UI Framework

Performance is crucial in today's web applications. A slow app feels buggy to the users and makes them flee it. Although performance is such an important topic, there are so many optimization vectors that it's easy to forget some of them. The intent of this checklist is to gather performance practices to apply when developing a web application.

While we apply these practices, we should always keep the following rules in mind:

• Don't optimize too early

• Don't let performance ruin productivity

• Always check an optimization is efficient by measuring performance before and after it

Objectives

• Define the performance metrics and objectives

  • The RAIL model is generally a good model to start with.

  • If speed is an advantage we want to have against competitors, know that users usually will feel we are faster if we are at least 20% faster than them.

• Plan out a loading sequence; this way we can define early what is important in the content, what to load first and what to load later

• Make a performance budget

  • The performance budget calculator is useful to estimate the budget depending on the performance we want to obtain. This one is nice too.

  • Remember that this budget takes compression into account.

  • Currently, the recommended budget is max. 170kb gzipped, 130kb if JS heavy, but it depends mostly on the user-centric objectives

Frontend

• If choosing between SPA frameworks, take in account features like server side rendering; these features will be hard to add later

• Take in account how much every library / framework will take on the performance budget; don't use too much of them

  • Bundlephobia can help we estimate the size of a new dependency.

• Make sure we need custom fonts before using them

  • This article can help

• Consider technologies like AMP and Instant Articles, but be aware of their pros and cons

  • Keep also in mind these solutions are not mandatory to obtain correct performance

Optimize images

Images represent in average ~60% of a page's weight, thus it's an important part to optimize.

• Use WebP compression format for browsers that accept it

• Use responsive images with img's' srcset and size attributes

• Optimize manually important images or script their optimization

• Lazy load images

• Replacing animated gifs by videos can reduce their size dramatically (details here)

Reduce code size

• Minimize the source code

• Use tree shaking (e.g. with webpack) to remove unused code

• If the bundled code file is too big, use code splitting to load only what's needed first and lazy load the rest

• Serving ES2015 code to browsers supporting it and ES5 code to browsers that don't, using type="module" and nomodule, can improve the bundle size and parsing time (details here)

Reduce number of requests

• Replace third parties components (like sharing buttons, maps...) by static components

  • Tools like the Simple sharing buttons generator can help.

• Cache requests client side using service workers

• Bundle common images using CSS sprites

Prepare next requests

We can use prefetching to prepare the browser to next requests and make them faster or even instant. This article is a few old but explains well the following techniques.

• Use dns-prefetch to resolve the domain of services we may need

• Use preconnect to do DNS lookup, TCP handshake and TLS negotiation with services we know we will need soon

• Use prefetch to request specific resources that are likely to be needed soon, like images and scripts

  • This technique makes an especially good combination with lazy loading.

• Use preload to request specific resources that will be needed in the current page, e.g. <script> tags at the end of the body

  • The difference between prefetch and preload is explained here.

• Use prerender to request and prerender pages that are very likely to be visited soon, like homepage or user dashboard

  • Be aware that this technique is quite heavy, make sure we know what we are doing before applying it.

Optimize time to rendering

Ideally the critical code should fit in 14KB in order to be server in the first TCP roundtrip (why 14KB?). These techniques help to achieve this goal.

• Inline critical CSS in the <head> of the pages

  • Tools like CriticalCSS and critical can help defining the site's critical CSS

• If critical CSS is inlined, we can then defer the CSS files loading

  • loadCSS can help achieving that

• When the CSS files are loaded, set a cookie to avoid using inlined critical CSS anymore; this way it will benefit from browser cache

  • More explanations on this technique here

• Defer scripts execution, especially social media buttons and ads

• Defer fonts loading; the technique is explained here

• Use server side rendering if making an SPA

Optimize fonts

• Subset the fonts using Font Squirrel's font generator

• Use WOFF2 with fallback to WOFF and OTF

Make animations smooth

• Prefer animating using CSS' transform and opacity; more explanations here

• If animating with JS, use requestAnimationFrame instead of setInterval

• Avoid animating during high network activity; for example, wait till the page is fully loaded.

User perceived performance

User-perceived performance is often disregarded but can be more important than actual performance. These techniques allow us to improve it.

• We should use a loader only on "long" / "heavy" tasks, i.e. tasks the user can imagine they are heavy (e.g. account creation)

• Instead, we can use animations to illustrate the transitions following the user's action, for example, the transition from one page to another

• Show app shell before content if needed; more explanations here

• If using JPG images, we can use progressive JPGs to improve their loading perception

• If not using especially JPG, we can replace the images by cheaper components until they are loaded

  • We can replace an image with a canvas filled with its main colour.

  • We can replace an image with a very lightweight, blurred version of it. This efficient technique is explained in this article from Facebook.

  • We can also simply use a low-quality version of it.

• Make an optimistic UI to make some interactions feel instant; a quick explanation can be found here

Backend

• When choosing a web framework/library/language, take into account the following points:

  • How fast is the library / underlying language (but be aware that benchmarks are usually biased)?

  • How easy will it be to handle concurrency?

  • Does it allow efficient resource management, e.g. using a connections pool or an event loop?

General

• Provide batch queries/transactions instead of making the client send multiple requests

• Identify & optimize slow resources

• Parallelize slow tasks

• Use relevant data structures

• Don't overuse serialization

• Generate static content when deploying so that it will be computed only once

• If possible, use jemalloc to improve memory allocation

Cache

• Use HTTP cache; the different caching techniques are explained in this guide

• Consider using ESI if the app is not a SPA

• Cache calls to other services using Redis, a reverse proxy...

• Cache data slow to compute and memoize slow functions

Don’t loose time with non-urgent tasks

• Defer tasks to workers using a queue or use event-based patterns like Event Sourcing

• Use UDP for immediate but not vital tasks like logging

Don’t lose time with errors

• Fail fast by validating request inputs as soon as possible

• Use the circuit breaker pattern to avoid waiting for timeout when needing another service

• On sensible resources, detect suspicious requests as attacks before actually handling them; attacks can cause heavy resources consumption

  • For example, detect a login request as part of a brute force attack before fetching user data from the database

Database

• Make sure we use the right DBMS for the needs

  • Usually, a relational database will cover most needs, but in some cases, a NoSQL database may be a better fit.

  • If hesitating between NoSQL solutions, this comparison may help us.

• First, use indexes smartly

  • Use the Index, Luke provides a complete guide on this subject for common RDBMS. here

• Tune the DB; tools like MySQLTuner and the experimental OtterTune can help

• Identify & optimize critical and slow queries (e.g. code that produces n+1 queries)

  • In most SQL databases, EXPLAIN can help by showing the execution plan for a query

  • In PostgreSQL, EXPLAIN ANALYZE can help further by executing the explained query

• If using pagination, use the last row instead of offset as a starting point; more explanations here

• Once we are sure the used DBMS is the good one for our needs, take advantage of its advanced features (e.g. materialized views in Oracle, hyperloglogs in Redis...)

• Don’t use ORM for complex queries, unless you know what you’re doing

• If possible, defer heavy tasks to moments of the day when there is less load on the database (at night for example) to save resources when needed

• If possible, enable jemalloc to improve memory allocation

• If using UUIDs, reorder them before storing them; more explanations here

• Try different storage engines

  • On many DBMSs RocksDB often gives interesting results.

Network & Infrastructure

• Serve static content using a CDN to shorten the distance between the client and the server

  • When using the CDN take into consideration features like HTTP/2 support, compression...

• Deploy the app on several datacenters, also to shorten the distance between the client and the server

• Serve resources compressed using Brotli if it's supported, Gzip otherwise

• Compress resources that are rarely changed using Zopfli

• Use HTTP/2 and its features like server push and enable HPACK to compress HTTP headers

• Use OCSP stapling to fasten TLS shaking

• Use 0-RTT resumption to avoid round trips during TLS negotiations

• Avoid redirects as they increase the number of needed requests

• If using a microservices architecture, bring services needing each other often closer, ideally in the same machine

  • Kubernetes' pods can help achieve this goal

Measuring

• Measure server-side performance; this is usually already done by the web framework

• Measure client-side performance; tools like Web Page Test can help

• Measure client-side performance by country; results may hugely differ from one to another

• Use tools like Lighthouse to audit the site

• Load test the servers as they probably won't have the same perfs under 10rps and 1000rps

  • Gatling and Locust are good tools to perform these tests.

• Keep track of queries to the databases to ease slow queries discovery

Miscellaneous

• Keep the dependencies up-to-date as their performance is often improved by their maintainers

• Take into account the Save-Data request header to serve lighter assets to clients with limited resources

To ensure smooth deployments and minimize potential issues, it's crucial to follow a thorough checklist. Here's a high-level overview:

1. Know what is going to get deployed

2. Excessive testing; preferably using a pipeline with automated tests

3. Know when is going to get deployed

4. Create backups

5. Deploy to production

6. Test on the live server

By adhering to this checklist, you can enhance control over deployments and reduce the likelihood of complications.

Know what is put in version control

• It is extremely important to know exactly what you put in version control. Make sure you mask passwords, or better put them in your file with environment variables, like a .env file that contains all environments variables.

• What’s less impactful, but definitely not what we want in .env, is to put in config files and version control.

Testing

• Before any piece of code gets deployed it should be tested thoroughly. Testing the code comes in different degrees.

• Code reviews should be part of the routine of every team just to have an extra pair of eyes look at what is being deployed.

• In the best-case scenario writing code should go hand-in-hand with writing automated tests for that same piece of code.

Pipelines

• Pipelines are flexible in a way that you could configure them however the most common components of a pipeline are build automation, test automation, and deployment automation.

• Ensure that every change to the code is releasable and as easy pushing a button.

1. Application Development

Health checks

• Containers have Readiness probes

• Containers crash when there's a fatal error

• Configure a passive Liveness probe

• Liveness probes values aren't the same as the Readiness

Apps are independent

• The Readiness probes are independent

• The app retries connecting to dependent services

Graceful shutdown

• The app doesn't shut down on SIGTERM, but it gracefully terminates connections

• The app still processes incoming requests in the grace period

• The CMD in the Dockerfile forwards the SIGTERM to the process

• Close all idle keep-alive sockets

Fault tolerance

• Run more than one replica for your Deployment

• Avoid Pods being placed into a single node

• Set Pod disruption budgets

Resources utilisation

• Set memory limits and requests for all containers

• Set CPU request to 1 CPU or below

• Disable CPU limits — unless you have a good use case

• The namespace has a LimitRange

• Set an appropriate Quality of Service (QoS) for Pods

Tagging resources

• Resources have technical labels defined

• Resources have business labels defined

• Resources have security labels defined

Logging

• The application logs to stdout and stderr

• Avoid sidecars for logging (if you can)

Scaling

• Containers do not store any state in their local filesystem

• Use the Horizontal Pod Autoscaler for apps with variable usage patterns

• Don't use the Vertical Pod Autoscaler while it's still in beta

• Use the Cluster Autoscaler if you have highly varying workloads

Configuration and secrets

• Externalise all configuration

• Mount Secrets as volumes, not environment variables

2. Governance

Namespace limits

• Namespaces have LimitRange

• Namespaces have ResourceQuotas

Pod security policies

• Enable Pod Security Policies

• Disable privileged containers

• Use a read-only filesystem in containers

• Prevent containers from running as root

• Limit capabilities

• Prevent privilege escalation

Network policies

• Enable network policies

• There's a conservative NetworkPolicy in every namespace

Role-Based Access Control (RBAC) policies

• Disable auto-mounting of the default ServiceAccount

• RBAC policies are set to the least amount of privileges necessary

• RBAC policies are granular and not shared

Custom policies

• Allow deploying containers only from known registries

• Enforce uniqueness in Ingress hostnames

• Only use approved domain names in the Ingress hostnames

3. Cluster Configuration

Approved Kubernetes configuration

• The cluster passes the CIS benchmark

• Disable metadata cloud providers metada API

Application security is a critical topic. Being a good engineer requires being aware of Application security best practices. We should practice defensive programming and ensure run security checks much earlier and more frequently during every phase of the software development. The earlier we catch vulnerabilities, the less dramatic and expensive those violations are to resolve. Waiting until release will just leave us nervous and unprepared. Delivering security alongside the continuous delivery of software, we'll identify security problems before they become hopelessly entangled in the application and therefore more difficult, and costly, to resolve.

Teams should ensure the following checks to deliver security.

1. Design considerations

2. Development considerations

3. CI/CD considerations

4. Testing considerations

5. Infra-security considerations

6. Deployment considerations

1. Design Considerations

Injecting Security in the design phase means addressing design decisions that take into account the perspective of an attacker that is trying to breach weaknesses and compromise the confidentiality, integrity and other important security aspects of your software.

• Protect Personally Identifiable Information (PII Data)

  • Personally identifiable information (PII) is any data that can be used to identify a specific individual

  • Protect Personally Identifiable Information in the applications by encrypting them

  • Follow the data privacy laws of the land

• Secure File uploads

  • Whitelist-allowed extensions. Only allow safe and critical extensions for business functionality

  • Ensure that input validation is applied before validating the extensions.

  • Set a filename length limit. Restrict the allowed characters if possible

  • Only allow authorised users to upload files

  • In the case of public access to the files, use a handler that gets mapped to filenames inside the application (someid -> file.ext)

  • Run the file through antivirus or a sandbox, if available, to validate that it does not contain malicious data

  • Ensure that any libraries used are securely configured and kept up to date

  • Protect the file upload from CSRF attacks

• Where possible, always log in and handle the exception

  • Input validation failures e.g. protocol violations, unacceptable encodings, invalid parameter names and values

  • Output validation failures e.g. database recordset mismatch, invalid data encoding

  • Authentication successes and failures, authorization (access control) failures

  • Session management failures e.g. cookie session identification value modification

  • Application errors and system events e.g. syntax and runtime errors, connectivity problems, performance issues, third-party service error messages, file system errors, file upload virus detection, configuration changes

  • Application and related systems startups and shutdowns, and logging initialization (starting, stopping or pausing)

  • Use of higher-risk functionality e.g. network connections, addition or deletion of users, changes to privileges, assigning users to tokens, adding or deleting tokens, use of systems administrative privileges, access by application administrators, all actions by users with administrative privileges, access to payment cardholder data, use of data encrypting keys, key changes, creation and deletion of system-level objects, data import and export including screen-based reports, submission of user-generated content especially file uploads

  • Legal and other opt-ins e.g. permissions for mobile phone capabilities, terms of use, terms & conditions, personal data usage consent, permission to receive marketing communications

• In the event of an authentication exception, using any of the authentication mechanisms (login, password reset or password recovery), an application must respond with a generic error message regardless of whether:

  • The user ID or password was incorrect.

  • The account does not exist.

  • The account is locked or disabled.

• In order to secure authorization-based transactions in such a system:

  • Authorization should be performed and enforced server-side

  • Transaction verification data should be generated server-side

  • Applications should prevent authorization credentials from brute-forcing

  • Transaction data should be protected against modification

  • Confidentiality of the transaction data should be protected during any client/server communications

  • When a transaction is executed, the system should check whether it was authorized

  • Authorization credentials should be valid only for a limited period of time

  • Authorization credentials should be unique for every operation

• Enable logging and monitoring of authentication functions to detect attacks/failures on a real-time basis

  • Ensure that all failures are logged and reviewed

  • Ensure that all password failures are logged and reviewed

  • Ensure that all account lockouts are logged and reviewed

• Cross-Site-Scripting (XSS)

  • Use templating engines or frameworks that automatically escape XSS by design, such as EJS, Pug, React, or Angular. - - Learn the limitations of each mechanism XSS protection and appropriately handle the use cases which are not covered

  • Escaping untrusted HTTP request data based on the context in the HTML output (body, attribute, JavaScript, CSS, or URL) will resolve Reflected and Stored XSS vulnerabilities

  • Applying context-sensitive encoding when modifying the browser document on the client-side acts against DOM XSS

  • Enabling a Content-Security Policy (CSP) as a defence-in-depth mitigating control against XSS

• DB Security

  1. Data Encryption - Encrypt database storage files and on-site/offsite backups.

     • Identify the specific data sets on which encryption should be enabled. Which databases? Which Rows? Which Columns?

     • Use different encryption keys to encrypt different datasets, Perhaps data that belongs to different clients sharing a single database?

  2. AUDITING - Monitor internal and external access to data that is considered sensitive.

     • Identify the specific data assets that require auditing: databases, tables, columns.

     • Track the username, originating from which server, accessing which specific dataset and when.

     • Special auditing is needed if applications users are separate from database users.

  3. DATA GOVERNANCE - Track and monitor data including as it moves across different data silos.

     • Make sure to tag, track and catalog datasets as they travel from one database to another. Establish policies and workflows with checks along the way.

     • Implement complete data lifecycle management policies across all databases including data provisioning, data cleansing and periodic dataset compliance checks.

  4. CLIENT SEGREGATION - In Multi-Tenant environments, multiple “clients” can share the same database server. These types of databases require special treatment across all security domains: encryption, auditing, authorization, etc…

     • Encrypt individual client data using different keys. implement strong authorization and authentication, using different users for each client and restricting access to subsets of the entire database.

     • Identify specific customers that cannot co-exist on the same database server.

  5. AUTHORIZATION - Enable polices on who can access which datasets, enable strict permissions.

     • Different users should only be allowed access to specific datasets within a database: down to the specific row and column levels. Never grant excessive permissions.

     • If application users and database users do not correlate, more sophisticated authorization needs to be configured to create a strong chain of identity.

     • Set requirements for sophisticated authorization that goes beyond simple users: filter origin of access to data – server IP, time/date, application, etc…

     • Do you have “super” / system” users that can access all data? Are they required? These types of users require special attention.

  6. AUTHENTICATION - Setup secure means by which users authenticate with the database.

     • Ensure strong authentication mechanisms such as single-sign on and restricting databases to only authenticate users from Directory Services and not using username/password combinations.

     • If “web users” authenticate with your application and the application authenticates with the database using different users or using username/password combinations, special attention must be placed on creating strong authentication chains

2. Development Considerations

• Lightweight, Static analysis (SAST) checking in the engineer’s IDE like VSCode, Eclipse, Intelli..

• Peer code reviews (for defensive coding and security vulnerabilities), Code reviewers should always consider security guidelines during code reviews.

• Audit your application's design, implementation and security with unit/integration tests coverage frequently

Authentication

• Don't use Basic Auth. Use standard authentication instead (e.g. JWT, OAuth).

• Don't reinvent the wheel in Authentication, token generation, password storage. Use the standards.

• Use Max Retry and jail features in Login.

• Use encryption on all sensitive data.

JWT (JSON Web Token)

• Use a random complicated key (JWT Secret) to make brute-forcing the token very hard.

• Don't extract the algorithm from the header. Force the algorithm in the backend (HS256 or RS256).

• Make token expiration (TTL, RTTL) as short as possible.

• Don't store sensitive data in the JWT payload, it can be decoded easily.

OAuth

• Always validate redirect_uri server-side to allow only whitelisted URLs.

• Always try to exchange for code and not tokens (don't allow response_type=token).

• Use state parameter with a random hash to prevent CSRF on the OAuth authentication process.

• Define the default scope, and validate scope parameters for each application.

Access

• Limit requests (Throttling) to avoid DDoS / brute-force attacks.

• Use HTTPS on the server side to avoid MITM (Man in the Middle Attack).

• Use HSTS header with SSL to avoid SSL Strip attacks.

• For private APIs, only allow access from whitelisted IPs/hosts.

• Use an API Gateway service to enable caching, Rate Limit policies (e.g. Quota, Spike Arrest, or Concurrent Rate Limit) and deploy APIs resources dynamically

Input

• Text areas and input fields for PII (name, email, address, phone number) and login credentials (username, password) should be prevented from being stored in the browser.

• Use the proper HTTP method according to the operation: GET (read), POST (create), PUT/PATCH (replace/update), and DELETE (to delete a record), and respond with 405 Method Not Allowed if the requested method isn't appropriate for the requested resource.

• Validate content-type on request Accept header (Content Negotiation) to allow only your supported format (e.g. application/xml, application/json, etc.) and respond with 406 Not Acceptable response if not matched.

• Validate user input to avoid common vulnerabilities (e.g. XSS, SQL-Injection, Remote Code Execution, etc.).

• Don't use any sensitive data (credentials, Passwords, security tokens, or API keys) in the URL, but use the standard Authorization header.

• Use an API Gateway service to enable caching, Rate Limit policies (e.g. Quota, Spike Arrest, or Concurrent Rate Limit) and deploy APIs resources dynamically.

• All the data exchanged with the REST API must be validated by the API.

Processing

• Check if all the endpoints are protected behind authentication to avoid a broken authentication process.

• Serialize your JSON (to avoid arbitrary JavaScript remote code execution)

• User own resource ID should be avoided. Use /me/orders instead of /user/654321/orders.

• Don't auto-increment IDs. Use UUID instead.

• If you are parsing XML files, make sure entity parsing is not enabled to avoid XXE (XML external entity attack).

• If you are parsing XML files, make sure entity expansion is not enabled to avoid Billion Laughs/XML bomb via exponential entity expansion attack.

• Use a CDN for file uploads.

• If you are dealing with a huge amount of data, use Workers and Queues to process as much as possible in the background and return a response fast to avoid HTTP Blocking.

• Do not forget to turn the DEBUG mode OFF.

Output

• Send X-Content-Type-Options: nosniff header.

• Send X-Frame-Options: deny header.

• Send Content-Security-Policy: default-src 'none' header.

• Remove fingerprinting headers - X-Powered-By, Server, X-AspNet-Version, etc.

• Force content-type for your response. If you return application/json, then your content-type response is application/json.

• Don't return sensitive data like credentials, Passwords, or security tokens.

• Return the proper status code according to the operation completed. (e.g. 200 OK, 400 Bad Request, 401 Unauthorized, 405 Method Not Allowed, etc.).

Logging

• Do not forget to turn the DEBUG mode OFF in production builds.

3. CI/CD Considerations

• Compile and build checks, ensuring the vulnerabilities are identified in third-party components, Libraries.

• Dashboard the findings and Generating Alerts on high-risk code.

• Automate the fixes by integrating the efficient tools that gates the security thresholds and raises possible fixes/suggestions.

• Automation of unit testing of security functions, with full coverage of code analysis.

• Audit your design and implementation with unit/integration tests coverage.

• Use a code review process and disregard self-approval.

• Ensure that all components of your services are statically scanned by AV software before pushing to production, including vendor libraries and other dependencies.

• Design a rollback solution for deployments.

4. Testing Considerations

Information Gathering

A successful web application security strategy fundamentally begins with an understanding of the interactions between the web server, users, and applications. While application deployment platforms vary, key vulnerabilities in infrastructure configuration act as a common weak link for threat actors to initiate an attack.

Some key application security information-gathering activities include:

• Identify technologies used

• Identify user roles

• Identify application entry points

• Identify client-side code

• Identify multiple versions/channels (e.g. web, mobile web, mobile app, web services)

• Identify co-hosted and related applications

• Identify all hostnames and ports

• Identify third-party hosted content

Configuration Management

A web server ecosystem is intrinsically complex with highly connected, heterogeneous services and components working together. Reviewing and managing the configuration of the server is, as a result, a very crucial aspect for maintaining robust security across multiple layers of an application.

Securing various configuration items of an application involves:

• Check for commonly used application and administrative URLs

• Check for old, backup and unreferenced files

• Check HTTP methods supported and Cross Site Tracing (XST)

• Test file extensions handling

• Test for security HTTP headers (e.g. CSP, X-Frame-Options, HSTS)

• Test for policies (e.g. Flash, Silverlight, robots)

• Test for non-production data in live environment, and vice-versa

• Check for sensitive data in client-side code (e.g. API keys, credentials)

Secure Transmission

• Check SSL Version, Algorithms, Key length

• Check for Digital Certificate Validity (Duration, Signature and CN)

• Check credentials only delivered over HTTPS

• Check that the login form is delivered over HTTPS

• Check session tokens only delivered over HTTPS

• Check if HTTP Strict Transport Security (HSTS) in use

Authentication

• Test for user enumeration

• Test for authentication bypass

• Test for bruteforce protection

• Test password quality rules

• Test remember me functionality

• Test for autocomplete on password forms/input

• Test password reset and/or recovery

• Test password change process

• Test CAPTCHA

• Test multi-factor authentication

• Test for logout functionality presence

• Test for cache management on HTTP (eg Pragma, Expires, Max-age)

• Test for default logins

• Test for user-accessible authentication history

• Test for out-of-channel notification of account lockouts and successful password changes

• Test for consistent authentication across applications with shared authentication schema / SSO

Session Management

Once a user is authenticated, their interaction with the server is managed within a session. Improperly managed sessions open doors for attackers to compromise access mechanisms by assuming those to be identities of legitimate users. More so, such compromised accesses are often taken advantage of by attack vectors that escalate privileges and penetrate deeper into the system. To avoid vulnerabilities within a session, the following processes are recommended to be tested as a best practice:

• Establish how session management is handled in the application (eg, tokens in cookies, token in URL)

• Check session tokens for cookie flags (httpOnly and secure)

• Check session cookie scope (path and domain)

• Check session cookie duration (expires and max-age)

• Check session termination after a maximum lifetime

• Check session termination after a relative timeout

• Check session termination after logout

• Test to see if users can have multiple simultaneous sessions

• Test session cookies for randomness

• Confirm that new session tokens are issued on login, role change and logout

• Test for consistent session management across applications with shared session management

• Test for session puzzling

• Test for CSRF and clickjacking

Authorization

• Test for path traversal

• Test for bypassing authorization schema

• Test for vertical Access control problems (a.k.a. Privilege Escalation)

• Test for horizontal Access control problems (between two users at the same privilege level)

• Test for missing authorization

Data Validation

• Test for Reflected Cross Site Scripting

• Test for Stored Cross Site Scripting

• Test for DOM based Cross Site Scripting

• Test for Cross Site Flashing

• Test for HTML Injection

• Test for SQL Injection

• Test for LDAP Injection

• Test for ORM Injection

• Test for XML Injection

• Test for XXE Injection

• Test for SSI Injection

• Test for XPath Injection

• Test for XQuery Injection

• Test for IMAP/SMTP Injection

• Test for Code Injection

• Test for Expression Language Injection

• Test for Command Injection

• Test for Overflow (Stack, Heap and Integer)

• Test for Format String

• Test for incubated vulnerabilities

• Test for HTTP Splitting/Smuggling

• Test for HTTP Verb Tampering

• Test for Open Redirection

• Test for Local File Inclusion

• Test for Remote File Inclusion

• Compare client-side and server-side validation rules

• Test for NoSQL injection

• Test for HTTP parameter pollution

• Test for auto-binding

• Test for Mass Assignment

• Test for NULL/Invalid Session Cookie

Denial of Service

• Test for anti-automation

• Test for account lockout

• Test for HTTP protocol DoS

• Test for SQL wildcard DoS

Business Logic

Hackers mostly leverage an application’s original programmed flow to orchestrate breaches and penetration attacks. As a result, it is recommended to assess the business and application’s configuration to identify vulnerabilities in code or business logic that could be used for potential exploits.

• Test for feature misuse

• Test for lack of non-repudiation

• Test for trust relationships

• Test for integrity of data

• Test segregation of duties

Cryptography

Cryptography ensures the secure exchange of information by using algorithms that transform human-readable data into a ciphertext-encrypted output. While doing so, the process establishes trust between the web server and network entities using security keys, making it an important mechanism for maintaining application security. Testing cryptography for maintaining application security involves:

• Check if the data which should be encrypted is not

• Check for wrong algorithms usage depending on the context

• Check for weak algorithms usage

• Check for proper use of salting

• Check for randomness functions

Risky Functionality - File Uploads

• Test that acceptable file types are whitelisted

• Test that file size limits, upload frequency and total file counts are defined and are enforced

• Test that file contents match the defined file type

• Test that all file uploads have Anti-Virus scanning in-place.

• Test that unsafe filenames are sanitised

• Test that uploaded files are not directly accessible within the web root

• Test that uploaded files are not served on the same hostname/port

• Test that files and other media are integrated with the authentication and authorisation schemas

Risky Functionality - Card Payment

• Test for known vulnerabilities and configuration issues on the Web Server and Web Application

• Test for default or guessable password

• Test for non-production data in a live environment, and vice-versa

• Test for Injection vulnerabilities

• Test for Buffer Overflows

• Test for Insecure Cryptographic Storage

• Test for Insufficient Transport Layer Protection

• Test for Improper Error Handling

• Test for all vulnerabilities with a CVSS v2 score > 4.0

• Test for Authentication and Authorization issues

• Test for CSRF

HTML 5

• Test Web Messaging

• Test for Web Storage SQL injection

• Check CORS implementation

• Check Offline Web Application

5. Infra-Security Considerations

Kubernetes Cluster/Infra Security

• Practice the principle of least privilege. (PoLP)

  • SSO and Identity Access Management to the Cloud Infrastructure

  • Role-Based Access Control (RBAC) through ‘Role Binding’

  • Grant only the necessary privileges.

  • Revoke unnecessary privileges from the PUBLIC user group.

  • Restrict permissions on run-time facilities.

• Pod Security – an admission control plugin to ensure that pods are admitted only when certain security guidelines are met.

• Network Security

  • Use a firewall.

  • Never poke a hole through a firewall.

  • Monitor who accesses your systems.

  • Check network IP addresses.

• Multi-tenancy

• Data Encryption and Secrets Management

• Regulatory Compliance

• Incident Response and Forensics

• Image Security and Vulnerability Scanning Enabled

• Security Groups - allow inbound traffic only on TCP port 443 (HTTPS).

• Logging Enabled

• Running Recent Version - Ensure that the clusters are using the latest stable version with the latest features, design updates and bug fixes, and benefit from better security and performance.

• Non-public Endpoints - ensure the server endpoints are accessible within VPCs

Application Access

• TLS for Kubernetes Ingress – external access to the ingress controller should be over TLS, and interaction between the ingress controller and application containers must utilize TLS too, despite the fact that there are situations where that isn’t required.

• Encrypt everything in Transit – the default behaviour ought to encrypt everything in transit. It is prudent to encrypt network traffic between containers.

6. Deployment Considerations

• OWASP A2: Broken Authentication

  • Require MFA/2FA for important services and accounts

  • Rotate passwords and access keys frequently, including SSH keys

  • Apply strong password policies, both for ops and in-application user management.

  • Do not ship or deploy the application with any default credentials, particularly for admin users or external services you depend on

  • Use only standard authentication methods like OAuth, OpenID, etc.  avoid basic authentication

  • Auth rate-limiting: Disallow more than X login attempts (including password recovery, etc.) in a period of Y

  • On login failure, don't let the user know whether the username or password verification failed, just return a common-auth error

  • Consider using a centralized user management system to avoid managing multiple accounts per employee (e.g. GitHub, AWS, Jenkins, etc) and to benefit from a battle-tested user management system

• OWASP A3: Sensitive Data Exposure

  • Only accept SSL/TLS connections, enforce Strict-Transport-Security using headers

  • Separate the network into segments (i.e. subnets) and ensure each node has the least necessary networking access permissions

  • Group all services/instances that need no internet access and explicitly disallow any outgoing connection (a.k.a private subnet)

  • Store all secrets in vault products like AWS KMS, HashiCorp Vault or Google Cloud KMS

  • Lockdown sensitive instance metadata using metadata

  • Encrypt data in transit when it leaves a physical boundary

  • Don't include secrets in log statements

  • Avoid showing plain passwords in the frontend, take necessary measures in the backend and never store sensitive information in plaintext

• OWASP A5:  Broken access control

  • Respect the principle of least privilege  -  every component and DevOps person should only have access to the necessary information and resources

  • Never work with the console/root (full-privilege) account except for account management

  • Run all instances/containers on behalf of a role/service account

  • Assign permissions to groups and not to users. This should make permission management easier and more transparent for most cases

• OWASP A6: Security Misconfiguration

  • Access to production environment internals is done through the internal network only, use SSH or other ways, but never expose internal services

  • Restrict internal network access  - explicitly set which resource can access other resources (e.g. network policy or subnets)

  • If using cookies, configure it to "secured" mode where it's being sent over SSL only

  • If using cookies, configure it for "same-site" only so only requests from same domain will get back the designated cookies

  • If using cookies, prefer "HttpOnly" configuration that prevents client-side JavaScript code from accessing the cookies

  • Protect each VPC with strict and restrictive access rules

  • Prioritize threats using any standard security threat modelling like STRIDE or DREAD

  • Protect against DDoS attacks using HTTP(S) and TCP load balancers

  • Perform periodic penetration tests by specialized agencies

• OWASP A7: Cross-Site-Scripting (XSS)

  • Use templating engines or frameworks that automatically escape XSS by design, such as EJS, Pug, React, or Angular. - - Learn the limitations of each mechanism XSS protection and appropriately handle the use cases which are not covered

  • Escaping untrusted HTTP request data based on the context in the HTML output (body, attribute, JavaScript, CSS, or URL) will resolve Reflected and Stored XSS vulnerabilities

  • Applying context-sensitive encoding when modifying the browser document on the client-side acts against DOM XSS

  • Enabling a Content-Security Policy (CSP) as a defence-in-depth mitigating control against XSS

• OWASP A9: Using Components With Known Security Vulnerabilities

  • Scan docker images for known vulnerabilities (using Docker's and other vendors' scanning services)

  • Enable automatic instance (machine) patching and upgrades to avoid running old OS versions that lack security patches

  • Provide the user with both 'id', 'access' and 'refresh' token so the access token is short-lived and renewed with the refresh token

  • Log and audit each API call to cloud and management services (e.g who deleted the S3 bucket?) using services like AWS CloudTrail

  • Run the security checker of your cloud provider (e.g. AWS security trust advisor)

• OWASP A10: Insufficient Logging & Monitoring

  • Alert on remarkable or suspicious auditing events like user login, new user creation, permission change, etc

  • Alert on irregular amount of login failures (or equivalent actions like forgot password)

  • Include the time and username that initiated the update in each DB record

DIGIT being an open source platform, all the tools and tech stack used to build, deploy and operate DIGIT - are also open source and community edition. The various tools used are listed below with the specific versions used and their short description.

Platform Tools

Dev Stack Tools

DevOps Stack Tools

Explore the different aspects of the DIGIT platform details:

Best Practices

Below are some simplified best practices to ensure security:

Detailed Guidelines

Privilege Escalation

Role Action mapping should be in accordance with the business requirements. No role should have access to resources that are not required by that role on UI. In the case of a CITIZEN role, extra precaution should be taken. If there is a business requirement for user-based access to entities instead of role-based access, validations should be added at the module level to verify the user id.

There is a possibility of horizontal/vertical escalation in the case of workflows. We have fixed this issue by validating the logged in user and the workflow item's owner. Allowing access if the logged-in user and the workflow owner are the same.

Failure To Restrict URL Access

We use pre-signed URLs, so we cannot restrict access to the file if someone gets the URL. As a security measure, we should restrict the value of the URL validity to as minimum as possible. The value is configurable and can be overwritten in helm charts using the following property:

filestore-url-validity

Insecure Direct Object References (IDOR)

API’s like /user/_search which exposes the personal data of users shouldn’t be directly exposed. We have removed access to the API from UI, the user information of logged in user can be instead fetched from /user/oauth/token which is now enhanced to return the required info.

Malicious File Upload Leads To Cross-Site Scripting

Unrestricted file upload is a serious security risk. To tackle this problem we have a bunch of security validations. The file extension, content and content type in the header are all validated. We define the allowed extensions and their corresponding content type as a map and is configurable using the following property:

  allowed-file-formats-map: "{jpg:{'image/jpg','image/jpeg'},jpeg:{'image/jpeg','image/jpg'},png:{'image/png'},pdf:{'application/pdf'},odt:{'application/vnd.oasis.opendocument.text'},ods:{'application/vnd.oasis.opendocument.spreadsheet'},docx:{'application/x-tika-msoffice','application/x-tika-ooxml','application/vnd.oasis.opendocument.text'},doc:{'application/x-tika-msoffice','application/x-tika-ooxml','application/vnd.oasis.opendocument.text'},dxf:{'text/plain','application/dxf','application/octet-stream','image/vnd.dxf','image/vnd.dxf; format=ascii','image/vnd.dxf; format=binary','image/vnd.dxb'},csv:{'text/plain'},txt:{'text/plain'},xlsx:{'application/x-tika-ooxml','application/x-tika-msoffice','application/vnd.ms-excel'},xls:{'application/x-tika-ooxml','application/x-tika-msoffice','application/vnd.ms-excel'}}"

Improper Authentication

Before adding endpoints in the whitelist or mixed endpoints list all security implications should be thought through, as there will be no authentication or authorisation of the request. It’s a good practice to add origin-based rate limiting to avoid DDoS attack.

Missing Account Lockout

The account locking mechanism is provided in the DIGIT platform and should never be disabled. It helps in mitigating brute force attacks.

Request Throttling Attack

All _create API’s should have some sort of rate throttling to avoid DDoS attack that can overwhelm the system. The rate-limiting can be achieved by configuring the endpoint in limiter.properties of zuul. Following is a sample configuration:

zuul.ratelimit.policy-list.tl-services[0].limit=5
zuul.ratelimit.policy-list.tl-services[0].quota=10000
zuul.ratelimit.policy-list.tl-services[0].refresh-interval=60
zuul.ratelimit.policy-list.tl-services[0].type[0]=url=/tl-services/v1/_create
zuul.ratelimit.policy-list.tl-services[0].type[1]=user

Weak Encoding Mechanism

Client secrets shouldn’t be sent in Base64 encoded format from UI, only for the server to server call the secret can be sent in Authorization header in Base64 format. We have removed client secret from the Authorization header in Digit Platform

Sensitive Information In URL

Avoid sending sensitive information in URL as a query param. Instead, it can be sent either in the request body or in headers. For example in the previous Digit version in _logout API, authToken was sent in query param as below:

https://uat.digit.org/user/logout?access_token=04f03b3b-3db1-4127-ab3a-dedce685a428&tenantId=pg.citya

The API is now enhanced to accept the authToken in Request Body as below :

{
    "RequestInfo": {
        "apiId": "Rainmaker",
        "ver": ".01",
        "ts": "",
        "action": "_logout",
        "did": "1",
        "key": "",
        "msgId": "20170310130900|en_IN",
        "authToken": "04f03b3b-3db1-4127-ab3a-dedce685a4281"
    },
    "access_token": "04f03b3b-3db1-4127-ab3a-dedce685a428"
}

Lack Of Automatic Session Expiration

Proper expiry time should be set for authToken validity. In case the authToken get’s stolen, it can be exploited only till the time the token is valid. The validity can be configured from the following two property in helm charts:

  access-token-validity: 15
  refresh-token-validity: 15

Concurrent Session

Multiple logins using the same user name and password should be avoided. Due to business requirement, we currently have not implemented this feature in DIGIT.

Improper Error Handling

Errors should not expose any sensitive data or detailed error message to the end-user. Hackers can use this information to attack the system. Use of e.printStackTrace() should be avoided instead the error should be logged using logger. Always return a custom error message to the end-user.

// INCORRECT
try{
  // Some code
}
catch(Exception e){
  e.printStackTrace()
}

// OK
try{
  // Some code
}
catch(Exception e){
  log.error("ERROR_CODE", e)
}

Improper Input Validation

All user inputs should be validated before storing in DB. If not done attacker could use malicious input to modify data or possibly alter control flow in unexpected ways, including arbitrary command execution. In Digit we use annotations to validate user data Following is a sample validation using annotations:

@NotNull
@SafeHtml
@Size(max=64)
@JsonProperty("tenantId")
private String tenantId

Mail Command Injection

Validate any user input that is used to create an email subject line from user input. And encode special characters after proper canonicalization, and in particular, any line break (CR / LF) characters in the input, that attackers may use to inject unexpected headers in the mail message sent to the server.

Use Of Hardcoded Credentials

Always overwrite sensitive values in application.properties during deployment. Never use hardcoded credentials. Generally, the sensitive values in application.properties will be like the DB user name and password, secrets etc. In Digit all the sensitive values are overwritten using kubernetes ConfigMaps. Following is examples of properties that we overwrite:

spring.datasource.url=jdbc:postgresql://localhost:5432/rainmaker_tl
spring.datasource.username=postgres
spring.datasource.password=postgres

Use of sensitive information into the configuration file

Make sure any sensitive information like authToken or secrets are not exposed in any configuration file.

Exclude unsanitized user input from format strings

If the format string is constructed with untrusted input, an attacker may produce unexpected application behaviour. It may cause an exception such as java.util.MissingFormatArgumentException to be thrown (which, if not cached, may lead to a denial-of-service condition), or information leak. Do not compose format strings from untrusted input. The arguments, except format string, to the formatting methods, are data and could contain format specifiers without any unexpected behaviour.

HTTP Parameter Pollution

Sanitize user input before using it in HTTP parameters. Concatenating unvalidated user input into a URL can allow an attacker to override the value of a request parameter

Standard pseudo-random number generators cannot withstand cryptographic attacks

There are two types of PRNGs: statistical and cryptographic. Statistical PRNGs provide useful statistical properties, but their output is highly predictable and form an easy to reproduce numeric stream that is unsuitable for use in cases where security depends on generated values being unpredictable. Cryptographic PRNGs address this problem by generating output that is more difficult to predict. In the latest release of Digit we have taken care of it by the following replacement:

Random random = new Random();

replaced with

SecureRandom random = new SecureRandom();

Weak cryptographic hash

The use of a weak hashing algorithm could compromise the original value. For example, if an authentication system takes an incoming password and generates a hash, then compares the hash to another hash stored in the authentication database, then the ability to create a collision could allow an attacker to provide an alternate password that produces the same target hash, bypassing authentication. It is recommended to use SHA-2 instead of SHA-1.

MessageDigest messageDigest1 = MessageDigest.getInstance(SHA_1); // INCORRECT

MessageDigest messageDigest2 = MessageDigest.getInstance(SHA_256); // OK

MessageDigest messageDigest3 = MessageDigest.getInstance("SHA-384"); // OK

Insecure SSL configuration

The problem with a flawed SSL configuration is that it may allow man-in-the-middle attacks and other issues. Following is a reference implementation:

SSLContext ctx = SSLContext.getInstance("SSL");
// FIXED, null here means to use the default (with proper validations) implementation
ctx.init(null, null, SecureRandom.getInstance("SHA1PRNG"));
HttpsURLConnection.setDefaultSSLSocketFactory(ctx.getSocketFactory());
URL url = new URL("https://spoofable.mybank.com/online-banking");
return url.openConnection().getInputStream();

Improper Neutralization of CRLF Sequences in HTTP Header

An attacker could inject line separators (CR/LF sequences) that could split the response message generated by the software into two messages. The second response is completely under the control of the attacker (intermediate web proxies may cache it), with could produce multiple conditions (web defacement, cache poisoning, cross-site scripting, or page hijacking). It is recommended to strip out any input which contains the %0d%0a URL encoded characters in HTTP request headers.

Avoid Capturing Java.Lang Security Exception

Security exceptions are related to security breaches such as denied permissions and improper use of the API. The code should not catch security exceptions except in specific cases where it may be necessary to mask some of these exceptions, for example, in the case of a log-on failure.

  // INCORRECT 
  catch (IllegalArgumentException e) {
                e.printStackTrace();
            } catch (SecurityException e) {
                e.printStackTrace();
            }
            
  //OK   
  catch (IllegalArgumentException e) {
                log.error("ERROR_CODE",e);
      }       

Always normalize system inputs

Not validating and normalizing the user input can lead to the execution of arbitrary code. An application’s strategy for avoiding cross-site scripting (XSS) vulnerabilities may include forbidding <script> tags in inputs. Such blacklisting mechanisms are a useful part of a security strategy, even though they are insufficient for complete input validation and sanitization. When implemented, this form of validation must be performed only after normalizing the input. We handle this by annotating all string user input field with @SafetHtml annotation

	@SafeHtml
	@JsonProperty("ownershipCategory")
	private String ownershipCategory;

Avoid the Command Throws within Finally

Throwing an exception from within a finally block will mask any exception which was previously thrown in the try or catch block, and the masked's exception message and stack trace will be lost.

Close Input and Output resources in finally block

Failure to properly close resources will result in a resource leak which could bring first the application on to their knees. Always closes any input stream in finally block.

// INCORRECT
public void readMdmsConfigFiles(String masterConfigUrl) {
        log.info("Loading master configs from: " + masterConfigUrl);
        Resource resource = resourceLoader.getResource(masterConfigUrl);
        try {
            masterConfigMap = objectMapper.readValue(resource.getInputStream(), new TypeReference<Map<String, Map<String, Object>>>() {
            });
        } catch (IOException e) {
            log.error("Exception while fetching service map for: ", e);
        }
    }

// OK
public void readMdmsConfigFiles(String masterConfigUrl) {
        log.info("Loading master configs from: " + masterConfigUrl);
        Resource resource = resourceLoader.getResource(masterConfigUrl);
        InputStream inputStream = null;
        try {
            inputStream = resource.getInputStream();
            masterConfigMap = objectMapper.readValue(inputStream, new TypeReference<Map<String, Map<String, Object>>>() {
            });
        } catch (IOException e) {
            log.error("Exception while fetching service map for: ", e);
        } finally {
            IOUtils.closeQuietly(inputStream);
        }
    }

Cross Site Request Forgery

CSRF attack forces a logged-on victim’s browser to send a request to a vulnerable web application, which then performs the chosen action on behalf of the victim. A successful CSRF exploit can compromise end user data and operation in the case of a normal user. If the targeted end user is the administrator account, this can compromise the entire web application. For this attack to be successful, the user must be logged into the application.

We have fixed these CSRF issues by adding a CSRF Filter in the security filter chain. All requests mapped pass through the csrf filter and the csrf token validation happens in the filter. All the requests except GET required to pass a csrf token.

applicationContext-security.xml

<bean id="csrfFilter" class="org.egov.infra.web.filter.CsrfFilter">
       <constructor-arg name="csrfTokenRepository">
            <bean class="org.springframework.security.web.csrf.HttpSessionCsrfTokenRepository"/>
        </constructor-arg>
        <property name="accessDeniedHandler" ref="accessDeniedHandler"/>
</bean>

applicationSecurityContext-<modulename>.xml

ex: applicationSecurityContext-egi.xml

<security:filter-chain pattern="/**"  filters="concurrentSessionFilter,securityContextPersistenceFilter,logoutFilter,authenticationProcessingFilter,headerWriterFilter,csrfFilter,securityContextHolderAwareRequestFilter,rememberMeAuthenticationFilter,anonymousAuthenticationFilter,exceptionTranslationFilter,filterSecurityInterceptor" />

Cross Site Scripting - Stored

Cross-site scripting is a web security vulnerability that allows an attacker to compromise the interactions that users have with a vulnerable application. It allows an attacker to circumvent the same origin policy, which is designed to segregate different websites from each other.

1. We have added a Validation Interceptor to validate the user input for all struts requests. Used OWASP AntiSamy policy to validate the user input.

2. Added entity level validation annotations to validate the entities used in spring requests.

struts.xml

<interceptor name="egov-validator" class="org.egov.infra.web.struts.interceptors.ValidationInterceptor" />

XSSValidator.java

public static String validate(String field, String input) {
        try {
            if (isBlank(input)) {
                return input;
            }
            CleanResults cr = getAntiSamy().scan(input, policy);
            if (!cr.getErrorMessages().isEmpty()) {
                if (LOG.isWarnEnabled())
                    LOG.warn(cr.getErrorMessages().toString());
                throw new ValidationException(field, "Invalid, contains unsafe value");
            }
            return input;
        } catch (PolicyException | ScanException exp) {
            throw new ApplicationRuntimeException("Error occurred while validating inputs", exp);
        }

    }

Insufficient Cookie Attributes

The application was missing the “Secure” cookie attribute which was carrying the session information (SessionID). The Secure attribute makes sure that the cookie will only be sent with requests made over an encrypted connection and an attacker will not be able to steal cookies by sniffing.

Added the missing secure cookie attribute

@Value("${secure.cookie}")
private boolean secureCookie;
	
@Bean
public CookieSerializer cookieSerializer() {
    DefaultCookieSerializer serializer = new DefaultCookieSerializer();
    serializer.setCookieName(SESSION_COOKIE_NAME);
    serializer.setCookiePath(SESSION_COOKIE_PATH);
    serializer.setUseSecureCookie(secureCookie);
    return serializer;
}

Code Injection

Application accepts the unvalidated user input. Interpreting user-controlled instructions at run-time can allow attackers to execute malicious code.

Fixed this issue by sanitizing the user input and removed eval().

function preventBack(){
	var sanitizedUrl = sanitizeHTML(document.URL);
	history.pushState(null, null, sanitizedUrl);
    window.addEventListener('popstate', function () {
        history.pushState(null, null, sanitizedUrl);
    });
}
function sanitizeHTML(text) {
	return jQuery('<div>').text(text).html();
}

Exclude unsanitized user input from format strings

Used format specifiers (%s, %d, %c etc) while constructing format strings instead of adding using the additional operator.

String validationMessage = String.format("Glcode: %1$s which  is mapped with Istrument type cheque in mdms is not exist in database",  accountCode.getGlcode());

cgvnNumber = String.format("%s/%s/%s%010d", vh.getFundId().getIdentifier(), getCgnType(vh.getType()), "CGVN",
				nextSequence);

Avoid data submissions to non-editable fields

Fixed this issue by using WebDataBinder.setDisallowedFields() to disallow id in request binding.

@InitBinder
public void initBinder(WebDataBinder binder) {
		binder.setDisallowedFields("id");
}

Potential Infinite Loops

Most of the loops present in the third party (YUI) libraries. So we have not changed this.

Avoid dangerous J2EE API, use replacements from security-focused libraries (like OWASP ESAPI)

Replaced the dangerous J2EE API with OWASP Enterprise Security API (ESAPI) equivalent APIs

pom.xml

<dependency>
            <groupId>org.owasp.esapi</groupId>
            <artifactId>esapi</artifactId>
            <version>2.2.3.1</version>
</dependency>

HTTPUtilities httpUtilities = ESAPI.httpUtilities();
httpUtilities.setCurrentHTTP(request, response);
httpUtilities.setHeader("Content-Type", mimeType);

referenceDocument = ESAPI.encoder().encodeForURL(referenceDocument);

ESAPI.httpUtilities().sendRedirect(request.getContextPath() + errorPage);

HTTPUtilities httpUtilities = ESAPI.httpUtilities();
httpUtilities.addHeader(response, "Cache-control", "no-cache, no-store, must-revalidate");
httpUtilities.addHeader(response, "Pragma", "no-cache");
httpUtilities.addHeader(response, "Expires", "-1");
httpUtilities.addHeader(response, "Vary", "*");

Do not allow external input to control resource identifiers

Sanitized the user input to fix this issue.

function sanitizeHTML(text) {
	return jQuery('<div>').text(text).html();
}

url: "/services/EGF/voucher/common-ajaxYearCode.action?departmentId="+sanitizeHTML(departmentid.value)
			+"&bankaccount="+sanitizeHTML(document.getElementById('bankaccount').value)

The setter method for an identifier property (id or composite-id) should be private

Changed the identifiers setter to private/protected

@Override
protected void setId(final Long id) {
  this.id = id;
}

private void setId(final Integer id) {
  this.id = id;
}

Frontend - Android APK

URL Redirection to Untrusted Site ('Open Redirect') It was observed that the application redirects the user’s browser to a URL provided in a user request, without proper validation of the user input. So it is recommended not to allow external input to modify the target URL. Perform a strict validation on the external input to ensure that the final URL is valid, appropriate for the application, and authorized

window.open(validateAndReturn(strDest), "myresults");
validateAndReturn(url){
Validation Logic to prevent only Allowed URL to Redirect to External Website
}

Exported activity must require permissions

Activity can be exported by setting its attribute exported=true, or adding an intent-filter not setting attribute exported=false. To receive or bind them permission must be required, otherwise, any apps can access the activity

It is recommended to set attribute exported=false or remove the intent-filter as default value without the intent is considered as false.

	<activity
			android:name="org.egovernment.mseva.MainActivity"
			android:launchMode="singleInstance"
			android:permission="dangerous"  // Set Permissions
			android:screenOrientation="portrait"> <!-- remove or alter as your apps requirement -->
			<intent-filter
				android:label="@string/app_name"
				android:exported="false">
				<action android:name="android.intent.action.VIEW" />
				<category android:name="android.intent.category.DEFAULT" />
				<category android:name="android.intent.category.BROWSABLE" />
				<data android:scheme="https"
					android:host="staging.digit.org"
					android:pathPattern=".*"
					android:pathPrefix="/" /> <!-- if you want only a specific directory from your website to be opened in the app through external links -->
			</intent-filter>
		</activity>

Potential code injection via WebView.addJavaScriptInterface()

Exposing Java objects to JavaScript could have negative security implications, such as code injection (allowing access to native phone functionality like sending SMS to premium numbers, accessing account information and sensitive data, etc.) Such code injection may, for example, do something like this (to launch a system command): exposedObj.getClass().forName('java.lang.Runtime').getMethod('getRuntime', null).invoke(null, null).exec(cmd) To avoid this security issue, you may remove addJavaScriptInterface() when possible

Either

(1) remove the JavaScript/Java bridge (addJavascriptInterface) or

(2) make sure that the content loaded by WebView is really trusted (and place a suppression for violations when needed), or

(3) upgrade to API level 17 or higher and place a @JavascriptInterface annotation on allowed public methods, as in the following code

	@JavascriptInterface    // Added Anotation to overcome this issue.
	public boolean isMsewaApp() {
		return true;
	}

Intent Manipulation

The application allows the user input to control Intent parameters that could enable an attacker to control the behaviour of the subsequent activity If untrusted input is inserted into certain parts of an Android Intent, without proper sanitization, a malicious user or app could force, via the tainted Intent, the execution of unintended code or inject malicious data in the vulnerable app. Certain Intent properties could change the expected semantics of the Intent, like setAction(), setClass(), setClassName(), or setComponent(). If the Intent is used to start an Activity or Service, for example, the attacker may change the expected element launched, with potential nefarious consequences It is recommended to validate untrusted input that does not proceed from app-controlled resources, and in particular user interface fields and other Intents coming from other apps.

	void loadView(String url, Boolean tab) {
		Intent request = getIntent(); // possibly coming from a malicious app
		ComponentName component = request.getComponent();
		if(  isValidComponent(component) ) {  // Validating Intent
		if (tab) {
			Intent intent = new Intent(Intent.ACTION_VIEW);
			intent.setData(Uri.parse(url));
			startActivity(intent);
		} else {
			webView.loadUrl(url);
		}}
	}

	private boolean isValidComponent(ComponentName component) {  // Validation Logic
		String activityName = component.getClassName();
		if(activityName.contains("MainActivity")){ 
			return true;
		}
		return false;
	}

Do not release debuggable apps

It was observed that android:debuggable is set to true. Android allows the attribute android:debuggable to be set to true in the manifest so that the app can be debugged. By default this attribute is disabled, i.e., it is set to false, but it may be set to true to help with debugging during the development of the app. However, an app should never be released with this attribute set to true as it enables users to gain access to details of the app that should be kept secure. The best practice is to set android:debuggable="false">

Enabling JavaScript is not recommended WebView consumes web content that can include HTML and JavaScript from an external URL, improper use can introduce common web security issues such as cross-site-scripting (XSS, or JavaScript injection).

Android includes several mechanisms to reduce the scope of these potential issues by limiting the capability of WebView to the minimum functionality required by your application.

WebView webView = (WebView)findViewById(R.id.webView);
 webView.getSettings().setJavaScriptEnabled(false); // FIXED, is the default value

Frontend- Web Application

Do not use eval() function, for security and performance reasons It was observed that the application uses eval(code)

If the end-user has control over evaluated code (because the code is concatenated with user data), this leads to 'script injection' vulnerabilities, which could end in well-known security attacks like cross-site scripting.

It is recommended to avoid the use of eval(code).

 <--  Wrong way using Eval -->

 if (wfSlaConfig) {
      if ((MAX_SLA - (MAX_SLA * eval(wfSlaConfig[0].slotPercentage)) <= sla) && sla <= MAX_SLA) {
        return wfSlaConfig[0].positiveSlabColor;
      } else if (0 < sla && sla < MAX_SLA - (MAX_SLA * eval(wfSlaConfig[0].slotPercentage))) {
        return wfSlaConfig[0].middleSlabColor;
      } else {
        return wfSlaConfig[0].negativeSlabColor;
      }
    }


<--  Correct way without Eval -->
if (wfSlaConfig) {
      if ((MAX_SLA - (MAX_SLA * wfSlaConfig[0].slotPercentage / 100) <= sla) && sla <= MAX_SLA) {
        return wfSlaConfig[0].positiveSlabColor;
      } else if (0 < sla && sla < MAX_SLA - (MAX_SLA * wfSlaConfig[0].slotPercentage / 100)) {
        return wfSlaConfig[0].middleSlabColor;
      } else {
        return wfSlaConfig[0].negativeSlabColor;
      }
    }

Do not use dangerouslySetInnerHTML property in React components.

dangerouslySetInnerHTML is React’s replacement for using innerHTML in the browser DOM. In general, setting HTML from code is risky because it’s easy to inadvertently expose your users to a cross-site scripting (XSS) attack.

Avoid dangerouslySetInnerHTML, equivalent to the risky innerHTML in DOM, when possible. Try using HTML directly from React, but use dangerouslySetInnerHTML and pass an object with a __html key.

Avoid post cross-document messages with an overly permissive target origin It was observed that the cross-document messaging feature has been used. The feature allows scripts to post messages to other windows. The corresponding API allows the user to specify the origin of the target window. However, caution should be taken when specifying the target origin because an overly permissive target origin will allow malicious script to communicate with the victim window in an inappropriate way.

Always specify an exact target origin, not *, when you use postMessage to send data to other windows. A malicious site can change the location of the window without your knowledge, and therefore it can intercept the data sent using postMessage.

We have Removed the instance as it was not required.

Potential infinite loops The application executing infinite loops using user inputs. would lead to denial of service.

It Is recommended to a proper termination condition for every lo0ps

All the user inputs included in the loop should have proper validation along with length check. 
while(true) {if(cond()) break;}
do { if(cond) break; } while(true);
function f() {
 while(true) {if(cond()) return;}
}
for(var i=0;i<P; i++) { ejecuta(); }
for(var i=Q;i<P; i+=3) { ejecuta(); }
for(var i=0,j=0;i<=N && j<=M; i++, ++j) { ejecuta(); }
for(var i=Q,j=E;i>=P && j>P2; i--, --j) { ejecuta(); }
for(var i=0;i<P; i--) { if(comprueba(i)) break; }
for(i=0;!(i<P); i--) { ejecuta(); }

Never use JavaScript 'history' object or navigation-based positioning functions Using 'history' (window.history, self.history) or navigation-based positioning functions (window.back(), window.forward()) is a bad practice for different reasons: * POST: If a visited page was generated with POST, the browser will emit a warning if the submitted data is not encoded in the URL. * PRIVACY: No application should know which pages a user visited (browsers will emit a security alert and block access). * POOR NAVIGATION LOGIC: The exact page sequence taken by users may not match programmer expectations

It is recommended not to use the navigation history.

 window.location.replace(targetUrl); // OK, not using navigation history