> For the complete documentation index, see [llms.txt](https://docs.digit.org/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.digit.org/health/design/architecture/low-level-design/services/health-services/console-services/project-factory-campaign-manager/project-factory.md).

# Project Factory

### Overview

When dealing with large-scale data creation on a server based on Excel input, the choice between Python and Node.js depends on factors like performance, ecosystem support, scalability, and ease of development.

Considering the various trade-offs between Java, Node.js, and Python (detailed in the sections below), **we chose a Node.js implementation for the Project Factory Service**. This service involves:

* Boundary creation based on the input Excel file
* Project creation for selected boundaries within a campaign
* Creation of entities such as facilities, users, and the necessary mappings between the created projects and these entities

We utilized the **exceljs** library to process the input Excel data for entity information. We have observed that the total data creation — such as project creation and project mapping — for **3000+ boundaries is completed within 15 minutes with high concurrency**.

***

### At a Glance

| Aspect                   | Detail                                                                                              |
| ------------------------ | --------------------------------------------------------------------------------------------------- |
| Purpose                  | Large-scale data creation on a server based on Excel input                                          |
| Chosen Technology        | Node.js                                                                                             |
| Excel Processing Library | exceljs                                                                                             |
| Key Operations           | Boundary creation, project creation, entity creation (facilities, users), project-entity mappings   |
| Performance Observed     | 3000+ boundaries processed within 15 minutes with high concurrency                                  |
| Why Node.js              | I/O-intensive task, high concurrency and scalability needs, streaming support for large Excel files |

***

### Dependencies

#### Language/Runtime Options Considered

**1. Java**

* Strengths: High performance for CPU-bound tasks (compiled, efficient JVM memory management); proven scalability for enterprise systems via multi-threading and frameworks like Spring Boot/WebFlux; strong stability and type safety for enterprise-grade needs.
* Weaknesses: Verbose development with more boilerplate/setup; non-blocking I/O requires additional frameworks (Netty, WebFlux), adding complexity; longer startup time and higher memory usage than Node.js; Excel processing (via Apache POI, JExcel) is more complex and resource-intensive for large datasets.

**2. Node.js**

* Advantages: Event-driven, non-blocking I/O — excellent for I/O-heavy operations like HTTP requests/API interactions; efficient concurrency via single-threaded event loop and async/await; Excel libraries (xlsx, exceljs) for reading/writing files, though less feature-rich than Python's pandas; native stream support for processing large files in chunks without loading them fully into memory; strong scalability under high loads due to lightweight architecture.
* Disadvantages: Lacks Python's mature data manipulation libraries (e.g., pandas), making complex data transformations less efficient; struggles with CPU-intensive tasks since single-threaded by default (mitigable with worker threads).

**3. Python**

* Advantages: Excellent Excel handling libraries (pandas, openpyxl, xlrd); strong data manipulation for large datasets (pandas, NumPy, Dask for parallel processing); built-in parallelism support (multiprocessing, concurrent.futures); ease of development due to simplicity and ecosystem; easy data export/integration via requests (HTTP) and sqlalchemy (databases).
* Disadvantages: GIL limits concurrency for I/O-heavy tasks (mitigable with asyncio/threading); higher memory usage for extremely large datasets; scaling to millions of concurrent requests requires more effort.

#### Decision Guidance

**Choose Node.js when:**

* The task is I/O-intensive (e.g., creating data on other servers via HTTP APIs)
* High concurrency and scalability are required
* Working with large Excel files where streaming avoids loading entire files into memory
* Already operating in a Node.js-based ecosystem

**Choose Python when:**

* Processing and transforming huge datasets in Excel efficiently
* The task involves heavy data manipulation or analytics
* Established libraries like pandas and openpyxl are preferred
* The use case is CPU-bound rather than I/O-bound

**Hybrid Approach (Optional):**

* Python for preprocessing and transforming large Excel files
* Node.js for efficient HTTP requests to create data on other servers

#### Final Recommendation

* Heavy Excel processing and transformations → Python
* Sending data concurrently to other servers → Node.js
* If both matter and comfort with Python exists → Python is often preferred for ecosystem and ease of data manipulation

#### Selected Dependency for Project Factory Service

* **Runtime:** Node.js
* **Library:** exceljs (for processing input Excel data for entity information)

***

### 4. Data Model

*Not specified in the source content.* The source describes the entities involved (boundaries, projects, facilities, users, and project-entity mappings) but does not provide schema-level or data-model details. This section can be filled in once the underlying data model/schema is available.

***

### 5. Business Flow

*Not explicitly detailed in the source content.* Based on the described operations, the business flow involves:

1. Input Excel file is provided containing boundary and entity information
2. Boundary creation is performed based on the input Excel file
3. Project creation for selected boundaries occurs within a campaign
4. Entities (facilities, users) are created
5. Mappings between the created projects and these entities are established

*Note: This is inferred solely from the operations listed in the source ("This service involves boundary creation based on the input Excel file, project creation for selected boundaries within a campaign, and the creation of entities such as facilities, users, and the necessary mappings between the created projects and these entities"). Further business flow detail was not present in the provided content.*

***

### 6. Process Flow

*Not explicitly detailed in the source content.* Based on the described technical approach:

1. Excel file input is processed using the **exceljs** library
2. Entity information is extracted from the Excel data
3. Data creation (boundaries, projects, facilities, users, mappings) is executed with **high concurrency**
4. Processing of 3000+ boundaries completes within **15 minutes**

*Note: This is inferred solely from the performance/technical details given in the source. Further process-level detail (e.g., specific steps, validations, error handling) was not present in the provided content.*


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.digit.org/health/design/architecture/low-level-design/services/health-services/console-services/project-factory-campaign-manager/project-factory.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.