High-Performance Data Processing Engine

A C++ data processing engine designed for large datasets and compute-intensive workflows, with Python utilities for generating benchmark inputs.

Highlights

• Streaming CSV-style input processing.
• Memory-conscious aggregation with standard library containers.
• Clear C++ module boundaries.
• Python benchmark data generator.

Build

mkdir -p build
c++ -std=c++20 -O2 src/main.cpp src/processor.cpp -o build/engine

Run

python3 python/generate_data.py > sample.csv
./build/engine sample.csv

Roadmap

• Add parallel chunk processing.
• Add binary columnar input support.
• Add benchmark suite with throughput and memory metrics.