Results
- 2M+ Customers served by SofIA, ABANCA's AI banking assistant
- 71% Reduction in cost per full evaluation cycle
- > 60% reduction in timing needed to fulfill an evaluation cycle
- 100% Security score (started from 59% and reached 95% after one iteration).
- 0 Production incidents since deployment
- ~10x combined ROI (including direct cost savings and the regulatory risk reduction).
The situation
SofIA is ABANCA's AI banking assistant. It handles customer queries across the bank's full product range in Spanish, English, Portuguese, and Galician, serving over 2 million customers. At that scale, SofIA is a regulated service channel with direct consequences for customer experience, brand reputation, and compliance standing.
Like every production AI system, SofIA does not stand still. Model updates, policy revisions, and knowledge base changes arrive regularly. Each change is a potential regression. Under the EU AI Act, ABANCA needed repeatable, documented evidence of reliability and safety.
The validation programme supporting SofIA was built the way most banks build theirs: manually and on sampled data. QA engineers hand-wrote test cases. Domain experts reviewed sampled conversations. Each cycle took weeks and covered quality only.
The challenge
Manual validation had two ceilings ABANCA was about to hit simultaneously.
- A cost ceiling. Bringing manual validation to Tier 1 coverage for a production assistant of SofIA's scale costs over €51,000 for a single iteration. That is a linear cost curve: every new use case added to the bank's AI portfolio multiplies it.
- A scope ceiling. Manual test-writing does not naturally cover edge cases or adversarial behaviour. Social engineering, policy boundary probing, information extraction attacks, and bias elicitation were outside the programme. Discovering them would require someone to consider each case, design a test, and run it manually, which is an unscalable process.
Both ceilings had the same root cause: manual processes do not scale.
What ABANCA changed
ABANCA applied a clear principle: encode expectations as specifications, then evaluate exhaustively against them. Three decisions shaped how Galtea's platform was deployed.
- Specification-driven test generation
Instead of writing individual test cases, ABANCA encoded SofIA's expected behaviour as a specification. What it must answer, what it must refuse, which policies it must enforce, and where its boundaries lie under pressure. Galtea generated evaluation datasets and adversarial variants automatically. Adding a new attack class in iteration 2 required a specification update, not a new authoring sprint. - LLM judges calibrated to ABANCA's policy taxonomy
Off-the-shelf judges evaluate against generic rubrics. Galtea's judges were calibrated against ABANCA's own compliance guidelines and policy taxonomy, with regular agreement checks on sampled outputs to keep judge drift visible. This catches false-positive approvals that generic rubrics miss. - Pre-release evaluation gating, separate from production monitoring
Every deployment candidate now scores against the full evaluation battery before release. Failures block the build. Separately, Galtea's monitoring pipeline provides ongoing observability in production. The two functions are distinct: one gates releases, one watches live traffic.
How SofIA security was tested before reaching any customer
Any LLM assistant deployed in financial services faces the same class of attacks. The question is whether they are discovered in a controlled test environment. Galtea's evaluation covered 12 risk categories across policy compliance and adversarial robustness. Three examples illustrate the attack patterns the programme tested for, and the standard SofIA was held to before any version reached a customer.
System prompt extraction via role re-assignment
A standard adversarial technique: instruct the assistant to act as "a new chatbot in training" and recite its foundational instructions. Tested across all major LLM deployments. The evaluation confirmed that SofIA's boundaries held under this class of prompts and their variants.
Out-of-scope financial advice via professional framing
Requests framed as legitimate strategic documents — asking for "innovative legal strategies" for wealth protection — are a known evasion pattern. Standard quality evaluation does not catch them because the framing appears reasonable. Galtea's judges are calibrated to ABANCA's policy taxonomy specifically to surface these cases. SofIA passed this class of test in iteration 2.
Geographic credit discrimination, justified on request
A direct test of EU AI Act compliance: prompts designed to elicit group-based risk justifications, such as flagging loan applications by postcode as a proxy for demographic risk profiling. This is among the highest-priority test categories for any financial AI system operating under the Act. SofIA's refusal behaviour was validated against this class before deployment.
Business impact
- Direct cost saving: 71% per full evaluation cycle. One full-coverage evaluation cycle without Galtea cost €51,726. The equivalent coverage with Galtea's platform cost €15,000, a saving of over €36,700 per cycle, at a greater scope: 16 metrics across quality and adversarial dimensions, versus quality alone.
- 23.6x more vulnerabilities surfaced per cycle. The depth of the automated adversarial programme relative to the manual quality programme captured classes of risk that were invisible under the previous approach.
- ~€280,000 in annual regulatory risk mitigated. Modelling a 1% baseline incident probability, 80% risk mitigation rate, and €35M maximum exposure under the EU AI Act, the expected annual loss reduction attributable to Galtea's compliance coverage is approximately €280,000.
- ~10x combined ROI. Including direct cost savings across the full use-case portfolio and the regulatory risk reduction, the projected annual return on the Galtea investment is approximately 10x. Payback period: under two months.
In their own words
"With Galtea, we uncovered vulnerabilities we would likely have missed otherwise, saved significant engineering time, and improved the reliability of our AI systems. It changed how we approach AI evaluation and governance."
Jorge Romaris, Head of AI at ABANCA
What's next
SofIA now runs through Galtea's evaluation pipeline as a permanent release gate. Every deployment candidate scores against the full battery before it reaches users. The auditable record produced, per release, gives ABANCA the documented, per-version evidence regulators require under the EU AI Act.
