Retail
Overnight batch jobs that regularly ran into the working day replaced by incremental Spark pipelines, so store and head-office reports are ready within 30 minutes of the data arriving.
Point-of-sale, e-commerce, loyalty and supplier data came from more than 50 sources into an on-premise warehouse through hand-written SQL jobs. The nightly run took nine hours and failed often enough that regional managers stopped trusting morning reports.
Volumes were growing past two billion rows a day, and the finance team needed the same figures the stores saw, not a separate reconciliation each month.
How things ran when we started, and once the work shipped.
Approach
4 phases over 7 months, with a working demo at the end of every week.
Weeks 1–5
Catalogued every source, owner and schema, and agreed data contracts with the teams that produce them so upstream changes stop breaking the pipeline silently.
Weeks 4–12
Built an S3 and Delta Lake lakehouse on AWS with Terraform, and moved ingestion to incremental loads using change data capture from the store systems.
Weeks 10–24
Rewrote the transformations as tested PySpark jobs orchestrated by Airflow, running old and new side by side until outputs matched to the row.
Weeks 24–30
Pointed 500+ scheduled reports at the new warehouse layer, added data quality checks with alerts to owners, and switched off the legacy jobs.
Data lands incrementally, is validated before anyone sees it, and every report reads from the same curated layer.
“Regional managers now open the dashboard at 8am and trust what they see. Finance and operations finally argue about the business, not about whose numbers are right.”
Tell us where things stand today. On a 30-minute call an engineer will sketch how we would approach it, the timeline and a price range.