Case Studies | SaaS B2B

Building the real-time data foundation for smarter auctions

Real-Time ETL Pipeline

About

A SaaS B2B business working with Cloud Combinator on AWS. The client is anonymised at their request.

Challenge

The challenge had four focus areas.

Ingesting live events at scale

Bid data needed to move continuously from the client's Pinot and Postgres sources into a streaming platform, capturing every auction event in real time rather than in slow, periodic batches.

Transforming data on the fly

Raw events are not directly useful. The pipeline had to perform real-time feature engineering and aggregation, deriving signals such as bid increments, bidder frequency, ad requests and winning bids as the data flowed through.

Storing for two very different jobs

The processed data had to serve both long-term, large-scale analytics and fast, low-latency lookups by auction or bidder, which are competing storage needs that a single store handles poorly.

A foundation the client could own

As the first phase of a larger plan, the pipeline had to be reliable, well documented, and fully understood by the client's engineers so they could maintain it and build the machine learning phase on top.

Solution

Cloud Combinator delivered the work as a sequence of clear workstreams, each with its own deliverable and sign-off, so the client could see progress and validate the pipeline as it took shape.

Real-time

Streaming bid-event processing

Exactly-

Processing guarantee via Flink checkpointing

Phase 1 of 2

Foundation for ML revenue optimisation

By the numbers:

  • Real-time - Streaming bid-event processing
  • Exactly- - Processing guarantee via Flink checkpointing
  • Phase 1 of 2 - Foundation for ML revenue optimisation
Changes

The project delivered its scope: a working, real-time ETL pipeline deployed into the client's own AWS environment, with the data foundation in place for the machine learning phase to follow.

  • Continuous ingestion — Bid events now stream from Pinot and Postgres into a managed Kafka backbone in real time, replacing slow, batch-bound movement of data.
  • Live feature engineering — Apache Flink derives signals such as bid increments, bidder frequency and winning bids as events flow, ready for analytics and modelling.
  • Fit-for-purpose storage — Amazon S3 handles large-scale analytics and archival while DynamoDB serves fast lookups by auction or bidder, each doing what it does best.
  • Reliable processing — Exactly-once guarantees mean every event is counted once, giving the client trustworthy data to build revenue models on.
  • Handover — A runbook, reference architecture and training session left the client's engineers able to operate, maintain and extend the pipeline independently.

With a dependable real-time data foundation in place, the client are set up for phase two, applying machine learning to predict optimal floor prices and uncover the bid patterns that drive more revenue from every auction.

AWS Stack

Amazon Managed Streaming

For Apache Kafka for real-time ingestion of live auction events.

Amazon Managed Service

For Apache Flink for real-time feature engineering and aggregation.

Amazon S3

For long-term, low-cost storage and batch analytics, with Amazon S3 Glacier for archival.

Amazon DynamoDB

For low-latency retrieval of processed metrics by auction or bidder.

Amazon Kinesis

Considered as an alternative streaming option during design.

YOU MIGHT LIKE

Related success stories

View all case studies

Case Studies | Insights

Utilising Language Recognition, Speed, and Enhanced Security to Make Social Media a Force for Good

  • Here, we take a detailed look at how the Cloud Combinator team collaborated with another cutting-edge AI service provider that provides intelligent systems to “make social media more social” for brands and users alike.
  • Arwen AI is a UK-based startup specialising in AI solutions to manage and enhance brands’ social media interactions. Founded in 2020 by Matt McGrory, Dr. David Cole, and Joel Bailey, Arwen. AI focuses on using AI to automatically detect and remove spam, toxic comments, and other unwanted content from social media platforms.
  • The team at Arwen have three core products. ‘Moderate’ is focused on identifying and removing toxic content from social media channels. ‘Engage’ helps brands identify and engage with meaningful conversations on social media, and ‘Customize’ allows brands to apply bespoke algorithms to their channels - creating an even more effective moderation and engagement.
Read more
CONTACT US

Ready to turn AI into impact?

We'll help you spot the highest-value opportunities, reduce risk around your first AI initiative, and define a clear path to results from day one.

Why talk to us:

Outcome-driven recommendations

AWS-recognised delivery expertise

Risk-aware AI adoption

Clear next step, not a sales pitch

Start with a focused 20-minute conversation about your goals — no pressure, no commitment.

We use cookies

This website uses cookies to enhance user experience and to analyze performance and traffic on our website.

See our Privacy Policy for details.