Kafka Consumer Producer Approach

Workflow Introduction

This document is to provide a high-level abstraction of Kafka workflow. Same documentation would be scaled to granular technical components as and when a grouping of services can be categorised from the ecosystem.

It’s important to note that we need to have closure engagements between development, solution engineering and QA automation teams to accommodate data streaming, staging and ecosystem workflows.

Kafka Abstraction Work Flow

Kafka SMS Consumers

egov.core.notification.sms : egov-notification-sms listens to this topic to get the data.

Kafka Email Consumers

egov.core.notification.email : egov-notification-mail listens to this topic to listen for the updates on emails and then to send notifications to user.

Kafka Producers

Following are the Producer topic:

egov.core.notification.sms.otp :- This topic is used to send OTP to user mobile number.

org.egov.core.notification.email :- This topic is used to send OTP to user email id.

Approach 1: User OTP

The user-otp service send the OTP to user on login request, on password change request and during new user registration.

1. /userOtp API - Invoke (PRODUCER) this API using Karate (user-otp).

2. Connect to the Kafka Server running on QA using karate with respective TOPIC.

3. Define a Consumer class for the OTP producer Topic for reading producer data. Consumers will listen to the topic and verify for a given mobile number for which OTP has been generated or not.

4. Persist OTP produced data across the test flow of User OTP API.

Actions to Explore and Data Collection

1. Full set of Producer and Consumer list required from End to End Eco-system

2. Filter grouping, Replication, High Availability, Broker Config Elements

Log Aggregation Solution

1. Log Aggregation Solution: Collect logs from a number of eGov services and then make them available to consumers in a standard format.