Case Studies | FinTech

From notebooks to an unified, automated data platform

Data Platform Migration

About

A FinTech business working with Cloud Combinator on AWS. The client is anonymised at their request.

Challenge

The migration concentrated on three focus areas needed to turn a split architecture into one dependable platform.

Move the AI pipeline into production

The four Bedrock agents, Classifier, Planner, Executor and Aggregator, had to leave the notebooks and run as production workloads. That meant refactoring notebook code into modular packages, containerising each agent, versioning the images in Amazon ECR, and integrating them into the existing Step Functions state machine so extraction runs automatically on new document arrival.

Unify the data platform

Registry scrapers, document extraction and satellite analytics ran on different tracks. The challenge was to bring them together under one S3 data lake with consistent prefix conventions and lifecycle policies, connect the isolated satellite pipeline to the main EventBridge and Step Functions orchestration, and track data lineage from raw source document through to the normalised Aurora tables, with the Glue Data Catalog and Athena validated across every dataset.

Automate delivery and operations

A production platform needs safe, repeatable delivery. Cloud Combinator had to build CI/CD with CodePipeline and CodeBuild for all workloads, define everything in AWS CDK with dev, staging and production parity, adopt blue/green deployment for zero-downtime agent updates, and instrument the platform with dashboards, alerting, cost tagging and audit logging.

Solution

Cloud Combinator sequenced the migration to protect the running platform, integrating the scrapers first, then the agents, then the satellite pipeline, before unifying the data lake and delivery pipelines.

4

AI agents moved from notebooks to production containers

3

Data pipelines unified under one orchestration

~10 weeks

Sequenced migration to acceptance

By the numbers:

  • 4 - AI agents moved from notebooks to production containers
  • 3 - Data pipelines unified under one orchestration
  • ~10 weeks - Sequenced migration to acceptance
Changes

The engagement delivered an unified, automated data platform against the success criteria in the Statement of Work, with acceptance measured by all four agents running in production containers, the satellite pipeline integrated, CI/CD operational, the unified S3 data lake in place and CloudWatch dashboards live. The figures below describe the delivered scope rather than an approved production benchmark.

  • AI extraction in production — The four agents run as versioned containers on ECS Fargate and Lambda, integrated into Step Functions and triggered automatically by EventBridge on new document arrival, ending manual notebook runs.
  • One unified data platform — Registry scrapers, document extraction and satellite analytics flow through a single S3 data lake into Aurora, with consistent prefixes, lifecycle policies and a Glue catalog validated through Athena.
  • Safe, automated delivery — CodePipeline and CodeBuild deliver every workload with dev, staging and production parity, and blue/green deployment enables zero-downtime updates to the agent tasks.
  • Observable and auditable — CloudWatch dashboards, SNS alerting, CloudTrail logging and per-pipeline cost tagging give the client full visibility, with data lineage tracked from raw source to Aurora.
  • Handover — The client received the platform as AWS CDK infrastructure as code with operational runbooks, documented CI/CD and rollback procedures, and recorded knowledge-transfer sessions for its team.

With every data source flowing through one automated, observable pipeline, the client can scale its underwriting throughput without manual notebook interventions, and has a Well-Architected foundation ready for future phases such as graph-based retrieval.

AWS Stack

Amazon ECS Fargate

And Amazon ECR for containerised, versioned execution of the agent and satellite workloads.

AWS Step Functions

And Amazon EventBridge for orchestration and event-driven triggering of the unified pipelines.

AWS Lambda

For serverless execution for agents suited to short-lived tasks.

Amazon S3

And AWS Glue for the unified data lake and catalog with Athena-validated schemas.

Amazon Aurora PostgreSQL

For the normalised data store for all pipeline outputs.

AWS CodePipeline

And CodeBuild for CI/CD with blue/green, zero-downtime deployments.

Amazon CloudWatch

And AWS CloudTrail for dashboards, alerting and audit logging across the platform.

YOU MIGHT LIKE

Related success stories

View all case studies

Case Studies | Insights

Utilising Language Recognition, Speed, and Enhanced Security to Make Social Media a Force for Good

  • Here, we take a detailed look at how the Cloud Combinator team collaborated with another cutting-edge AI service provider that provides intelligent systems to “make social media more social” for brands and users alike.
  • Arwen AI is a UK-based startup specialising in AI solutions to manage and enhance brands’ social media interactions. Founded in 2020 by Matt McGrory, Dr. David Cole, and Joel Bailey, Arwen. AI focuses on using AI to automatically detect and remove spam, toxic comments, and other unwanted content from social media platforms.
  • The team at Arwen have three core products. ‘Moderate’ is focused on identifying and removing toxic content from social media channels. ‘Engage’ helps brands identify and engage with meaningful conversations on social media, and ‘Customize’ allows brands to apply bespoke algorithms to their channels - creating an even more effective moderation and engagement.
Read more
CONTACT US

Ready to turn AI into impact?

We'll help you spot the highest-value opportunities, reduce risk around your first AI initiative, and define a clear path to results from day one.

Why talk to us:

Outcome-driven recommendations

AWS-recognised delivery expertise

Risk-aware AI adoption

Clear next step, not a sales pitch

Start with a focused 20-minute conversation about your goals — no pressure, no commitment.

We use cookies

This website uses cookies to enhance user experience and to analyze performance and traffic on our website.

See our Privacy Policy for details.