The problem
Streaming systems have to handle unfamiliar datasets, queue pressure, ordering, verification, and live visibility — without being rewritten for each new input.
For: Engineers reviewing pipeline architecture and concurrency behavior.
What I built
A reusable CSV ingestion pipeline with bounded multiprocessing queues, signature verification, packet re-sequencing, running-average computation, observer-based telemetry, and a live dashboard. New datasets are described in config.json rather than coded against.
The hard part
Coordinating concurrency, backpressure visibility, sequence ordering, and generic schema mapping without collapsing the three layers into one script.
The tradeoff
Focused on CSV inputs to keep the generic pipeline clean, trading broader adapters for clearer architecture.
What this does not do
- CSV only. Any other source needs a new input plugin.
- No published throughput or latency numbers — the telemetry exists, the benchmark does not.