Proving trustworthy AI for due diligence answers

A FinTech business working with Cloud Combinator on AWS. The client is anonymised at their request.

The PoC concentrated on four areas: measuring answer accuracy, guaranteeing source traceability, eliminating hallucination, and comparing models fairly across varied datasets.

Cloud Combinator structured the PoC across weekly milestones, from infrastructure configuration through pipeline build, evaluation, multi-model testing and handover, with a final week of support for implementing in the client's own AWS environment. Delivery was collaborative throughout, with pair programming and code walkthroughs to upskill the client's team.

Acceptance was based on signing off delivery against the project success criteria, confirming the technical competence of the client's internal tech lead, and demonstrating that the AWS environment is suitable for continued development of the client's service. The PoC delivered a working, repeatable framework for measuring how trustworthy LLM-generated DDQ answers are, along with a comparative report across models.

• Trust made measurable — The framework scores model answers on correctness, traceability and hallucination rate, using RAG evaluation and Bedrock's LLM-as-a-judge with custom metrics.
• Hallucination guardrail proven — The PoC specifically validated the model's ability to suppress generation when no ground truth exists, the key safeguard for due diligence.
• Model choice de-risked — The same the client's dataset was run across multiple Bedrock models, including Claude 3.7 Sonnet and Claude Sonnet 3.0, producing a like-for-like comparison of accuracy, traceability, hallucination and latency.
• Team capability lifted — Collaborative pair programming, code walkthroughs and prompt documentation strengthened the client's internal LLM integration and AI workflow skills.
• Handover — Final CloudFormation, documented Lambda code, prompt templates and test harnesses were delivered in a shared repository, with a knowledge transfer session and a week of post-project support for implementation in the client's AWS environment.

With an evidence-based framework in hand, the client can decide with confidence which Bedrock model and evaluation mode best fit each client's data, and has a repeatable, Infrastructure-as-Code foundation to build on as it moves from proof of concept towards a production DDQ answering capability in its own AWS environment.