Skip to content

E-commerce

Real-time analytics and demand forecasting for an online marketplace

Weekly spreadsheet reports replaced by a live dashboard and a demand-forecasting model, with product recommendations served from the same event stream.

Client Project5 months
Client
Online marketplace, 2M events a day, UK
Industry
E-commerce
Region
United Kingdom
Duration
5 months
Team
12 engineers
  • Data engineering
  • AI features
  • Product engineering

Results

Faster analytics queries
10×
Revenue lift on recommended products
45%
Events processed a day
2M+
Dashboard query latency
<100 ms

The challenge

The commercial team ran the business on reports exported from the production database every Monday. Queries on the order tables took minutes and occasionally slowed checkout, so exports were limited to one run a week.

The client also wanted personalised recommendations on product pages, but had no event pipeline, no feature store and no one on staff who had put a model into production.

Before and after

How things ran when we started, and once the work shipped.

  • Before: Weekly reports built from spreadsheet exports
  • After: Live dashboard, refreshed every five minutes
  • Before: Reporting queries slowing down checkout
  • After: Reporting fully separated from production
  • Before: Bestseller list shown to every visitor
  • After: Personalised recommendations, A/B tested

Approach

How the work was done

4 phases over 5 months, with a working demo at the end of every week.

  1. 01

    Weeks 1–5

    Event tracking and pipeline

    Defined an event schema with the product team, instrumented web and app clients, and streamed events through Kinesis into S3 and Snowflake.

  2. 02

    Weeks 4–10

    Analytics model and dashboard

    Built dbt models for orders, inventory and customer cohorts, and a Next.js dashboard on top with role-based access for finance, buying and marketing.

  3. 03

    Weeks 9–18

    Forecasting and recommendations

    Trained a gradient-boosted demand forecast per SKU and a two-tower recommendation model, then A/B tested recommendations against the existing bestseller list.

  4. 04

    Weeks 18–22

    MLOps and handover

    Added weekly retraining, drift alerts and a model registry, and paired with the two data analysts the client hired so they could own it.

Architecture

Production databases are no longer queried for reporting. Everything flows from a single event stream into the warehouse, and models read features from the same source.

  • Client and server events sent to Amazon Kinesis
  • Raw events in S3, modelled in Snowflake with dbt
  • Feature store and training jobs on SageMaker
  • Recommendation API on ECS, cached in Redis, p95 under 60 ms
  • Next.js dashboard with row-level security per team

Stack

  • Kinesis
  • Snowflake
  • dbt
  • Python
  • TensorFlow
  • SageMaker
  • Next.js
  • Redis
“Our buyers used to wait until Monday to see what sold last week. Now they reorder from a forecast every morning, and the recommendations paid for the whole project within the first quarter.”
Head of DataOnline marketplace, UK

Have a project like this?

Tell us where things stand today. On a 30-minute call an engineer will sketch how we would approach it, the timeline and a price range.