rag-normas: questions about the Pix rules

A generic chatbot answers about Pix from memory: limits that changed, the MED refund flow, what a bank may charge. This project answers only from the Banco Central's official text in force, says where each sentence comes from and measures how often it gets it right.

  • RAG
  • Evaluation
  • MCP

The path of a question

Question → BM25 + vector → RRF → Re-ranker → Free LLM → Answer [n]QuestionBM25 + vectorRRFRe-rankerFree LLMAnswer [n]
The index is one SQLite file built offline (FTS5 for BM25, sqlite-vec for vectors) and opened read-only. If the re-ranker says nothing retrieved is about the question, the answer is "NÃO ENCONTREI" and the LLM is never called.

The corpus

14 BCB acts in force about Pix: the consolidated Pix Regulation, fees, the penalties manual, value limits, Pix Automático and Agendado, cash withdrawal, onboarding, the DICT key directory. 649 chunks, one per article. Official acts are not copyrighted in Brazil (Lei 9.610/98, art. 8º, IV), and each chunk keeps the link to the official page.

The BCB publishes the consolidated text with every past wording struck through. The ingester drops the struck text, and the Pix Regulation goes from 56k to 29k words: without that, search would return revoked rules.

Evaluation

82 questions: 70 written by hand from randomly sampled articles (stratified by act, fixed seed), each with the article that answers it and a reference answer, plus 12 out of scope (Selic, TED, income tax, LGPD) to check that the system can say it does not know.

SearchRecall@1Recall@5MRR@10p50
BM250.570.900.700.5 ms
Vector (e5-small)0.540.760.664 ms
Hybrid (RRF)0.600.840.725 ms
Hybrid + re-ranker0.800.900.84315 ms

CI fails if Recall@5 or MRR@10 drop more than 0.03 below the baseline committed in the repo. BM25 does well here partly because the questions were written from the text: the numbers are there to catch regressions, not as a benchmark.

Answers

With a local model (qwen3.5:9b on Ollama) answering and another (gemma4:12b) judging: 66 of 70 questions answered, all with citations and 95% citing the expected article, no made-up citations, and all 12 out-of-scope questions answered "NÃO ENCONTREI". The 4 left unanswered are cases where search missed the right article, and the model said it did not find it instead of guessing.

What it costs to run

  • All on CPU, no torch: e5-small and an mMiniLM cross-encoder in int8 ONNX
  • A 5 MB SentencePiece tokenizer instead of tokenizer.json, which took 370 MB of RAM per instance (same ids, and a test checks that on the whole corpus)
  • A 512 MB, 0.5 CPU, read-only, non-root container: 404 MB peak and about 1 search per second with the re-ranker; without it, 261 MB and 74 searches per second
  • Free-tier LLMs only: Groq, then OpenRouter :free, then local Ollama

With the other projects

  • An MCP server with two tools: buscar_normas (search only, no LLM) and perguntar_normas_pix (answer with citations)
  • Each answer is a trace in genai-lens, and the caller's traceparent puts the spans inside the agent's trace

Run it locally

git clone https://github.com/vicgamelo/rag-normas
cd rag-normas && uv sync
uv run rag-normas models && uv run rag-normas build
uv run rag-normas search "devolução de Pix por fraude"

Limits

The questions were written by the author, and the re-ranker settings and abstention threshold were picked on this same set. The PDF manuals that complete the regulation (Manual de Tempos, DICT manual) are not in the corpus yet. This is not legal advice: the answer points to the article so you can check it.