rag-normas: questions about the Pix rules
A generic chatbot answers about Pix from memory: limits that changed, the MED refund flow, what a bank may charge. This project answers only from the Banco Central's official text in force, says where each sentence comes from and measures how often it gets it right.
The path of a question
The corpus
14 BCB acts in force about Pix: the consolidated Pix Regulation, fees, the penalties manual, value limits, Pix Automático and Agendado, cash withdrawal, onboarding, the DICT key directory. 649 chunks, one per article. Official acts are not copyrighted in Brazil (Lei 9.610/98, art. 8º, IV), and each chunk keeps the link to the official page.
The BCB publishes the consolidated text with every past wording struck through. The ingester drops the struck text, and the Pix Regulation goes from 56k to 29k words: without that, search would return revoked rules.
Evaluation
82 questions: 70 written by hand from randomly sampled articles (stratified by act, fixed seed), each with the article that answers it and a reference answer, plus 12 out of scope (Selic, TED, income tax, LGPD) to check that the system can say it does not know.
| Search | Recall@1 | Recall@5 | MRR@10 | p50 |
|---|---|---|---|---|
| BM25 | 0.57 | 0.90 | 0.70 | 0.5 ms |
| Vector (e5-small) | 0.54 | 0.76 | 0.66 | 4 ms |
| Hybrid (RRF) | 0.60 | 0.84 | 0.72 | 5 ms |
| Hybrid + re-ranker | 0.80 | 0.90 | 0.84 | 315 ms |
CI fails if Recall@5 or MRR@10 drop more than 0.03 below the baseline committed in the repo. BM25 does well here partly because the questions were written from the text: the numbers are there to catch regressions, not as a benchmark.
Answers
With a local model (qwen3.5:9b on Ollama) answering and another (gemma4:12b) judging: 66 of 70 questions answered, all with citations and 95% citing the expected article, no made-up citations, and all 12 out-of-scope questions answered "NÃO ENCONTREI". The 4 left unanswered are cases where search missed the right article, and the model said it did not find it instead of guessing.
What it costs to run
- All on CPU, no torch: e5-small and an mMiniLM cross-encoder in int8 ONNX
- A 5 MB SentencePiece tokenizer instead of tokenizer.json, which took 370 MB of RAM per instance (same ids, and a test checks that on the whole corpus)
- A 512 MB, 0.5 CPU, read-only, non-root container: 404 MB peak and about 1 search per second with the re-ranker; without it, 261 MB and 74 searches per second
- Free-tier LLMs only: Groq, then OpenRouter :free, then local Ollama
With the other projects
- An MCP server with two tools: buscar_normas (search only, no LLM) and perguntar_normas_pix (answer with citations)
- Each answer is a trace in genai-lens, and the caller's traceparent puts the spans inside the agent's trace
Run it locally
git clone https://github.com/vicgamelo/rag-normas cd rag-normas && uv sync uv run rag-normas models && uv run rag-normas build uv run rag-normas search "devolução de Pix por fraude"
Limits
The questions were written by the author, and the re-ranker settings and abstention threshold were picked on this same set. The PDF manuals that complete the regulation (Manual de Tempos, DICT manual) are not in the corpus yet. This is not legal advice: the answer points to the article so you can check it.