bear-app-rag Benchmark Results

Semantic vs keyword (SQLite LIKE) retrieval on a 25-note synthetic corpus with 23 eval queries across 7 query types.

Key finding: Semantic search improves recall by 0.40 on paraphrase queries and 0.13 on synonym queries. Exact-match recall ties at 1.00; the multi-concept group favors semantic recall and keyword first-hit rank.

Recall@5 by Query Type Recall@5 by Query Type Exact Match 1.00 1.00 Multi Concept 0.83 0.73 Paraphrase 1.00 0.60 Synonym 0.83 0.70 Tag Multi Or 1.00 0.67 Tag No Match 1.00 1.00 Tag Single 0.33 0.33 RAG Keyword (LIKE)
MRR by Query Type MRR by Query Type Exact Match 1.00 1.00 Multi Concept 0.84 0.90 Paraphrase 0.77 0.44 Synonym 1.00 0.70 Tag Multi Or 1.00 1.00 Tag No Match 1.00 1.00 Tag Single 1.00 1.00 RAG Keyword (LIKE)
Overall Metrics Overall Metrics Recall@5 0.90 0.75 MRR 0.91 0.79 Groundedness 0.87 0.80 RAG Keyword (LIKE)

Data source: tests/eval/results.json. Reproduce with uv run pytest -m eval -v.

Read the full story | Architecture tour | Source code