A researcher has tested a retrieval system designed to answer financial due-diligence questions using exact, checkable excerpts rather than free-form AI paraphrases. In a small study of Austrian banks’ annual reports, the full pipeline found a relevant passage in the top five results for all 12 answerable questions about one bank, but extracting the right answer remained the harder part.
Watch Desk analysis
What happened
Tuvshin Selenge describes the thesis project in a Hugging Face community article. It tackles a practical problem: a due-diligence answer may depend on a few words buried in hundreds of pages, where a fluent paraphrase can quietly lose a qualification or introduce a claim the report does not support.
The pipeline rewrites questions and generates alternative searches, combines semantic and keyword-based retrieval, reranks the results, then extracts text spans and checks that they appear in the source. Its evaluation uses 16 questions across four Austrian banks’ 2024 annual reports. Twelve questions were answerable and four were deliberately unanswerable.
Key findings
- Retrieval improved on the RBI test set
The full pipeline placed at least one relevant passage in its top five for 12 of 12 answerable questions, compared with 7 of 12 for the baseline. - Finding a passage is not the same as answering correctly
Recall@5 was 59.6%, and a high-overlap passage could still omit the exact rating or figure asked for. - Results varied across banks and models
On the three other banks, retrieval hit rates ranged from 90.9% to 100%; exact extraction and abstention were less consistent. - Abstention depends on the model and the test
GPT-5.1 abstained correctly on all four unanswerable RBI questions in the reported evaluation, while other models did so less reliably.
Why it matters
The distinction at the heart of this work is a useful one for anyone building AI tools around financial records: finding relevant text is only the first step. The system must also select a passage that actually answers the question, preserve its wording and know when the documents do not provide an answer.
The results are promising as a research prototype, not a green light for unattended financial due diligence. The evaluation is small, the system’s design choices were made using the RBI data, and the author notes that annotation differences affect comparisons across banks. Even a verified quotation can be the wrong quotation for the question. Paper trails, it turns out, still need readers.
Our read
This is a concrete example of AI being made more auditable by narrowing what it is allowed to say: retrieve evidence, extract it, check it, and leave the answer blank when appropriate. The strongest result is the improvement in retrieval; the caution is that evidence-shaped text is not automatically a correct answer. Treat the system as a way to help a person find and inspect supporting passages, not as a substitute for that person’s judgement.
What to watch
- Whether the approach is tested on larger datasets and more varied financial documents.
- Whether later evaluations measure answer correctness and abstention as carefully as retrieval.
- Whether the code and dataset allow others to reproduce the reported results.
Discussion spark: For financial due diligence, would you trust an AI assistant that quotes only text it can verify, or should a human still locate and select every supporting passage?
Sources and evidence
- From RAG to Verifiable Answers: Building a Multi-Stage Verbatim RAG Pipeline for Financial Due Diligence (6 October 2026, 19:24 UTC)
Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.