FMCG company · via Delman Data Lab · 2022–2024
Automated Daily Sales & Stock Reporting
Data Scientist
Client engagement. Names, data, and identifying details are withheld; the architecture is described at a generalized level.
Problem
Every day, modern-trade retail chains sent sales and stock files that admins unzipped and processed one by one, with three admins per chain. Building the daily report took hours, and human errors kept slipping in.
Approach
I built a pipeline that processes all incoming files at once: clean the raw files, combine them with existing records, validate file types and formats, replace values using user-defined patterns, transform the data into each report template, and generate the Excel reports. One admin now only uploads the data.
Architecture
Hover a component to trace its connections · drag to pan · use + / − to zoom
- Source
- Processing
- Data store
- Output
How It Works
- UploadOne admin uploads the daily sales and stock files from every retail chain.
- CleanFiles are unzipped and cleaned in parallel instead of one at a time.
- Combine & validateNew data is merged with existing records, and file types and formats are checked.
- TransformValues are replaced by user-defined patterns and mapped into each report template.
- GenerateThe daily reports are written to Excel, ready for the business team every morning.
What I Built
- The end-to-end pipeline: cleaning, combining with history, validation, pattern-based value replacement, template transformation, and Excel generation.
- Parallel processing of every retail chain’s files instead of one file at a time.
- Report templates that produce several daily reports from a single run.
Impact
Operational cost fell by 50% and daily processing time by 99%, from hours to minutes. No human errors were found after the switch, and the business could act on same-day data.