THE COMPARISON
What does another level add?
Inspect the implementation and the saved evidence together. A higher overall mean does not mean every question type improves.
Part of this page didn’t load.
An interactive part of the page is unavailable.
Reload to retry.
THE COMPARISON
Inspect the implementation and the saved evidence together. A higher overall mean does not mean every question type improves.
THE LADDER
77.7 percent Hit at 5
Hit@5
Page text, overlapping chunks and dense retrieval. A baseline you can follow.
Explore Naive79.3 percent Hit at 5
Hit@5
Markdown, OCR and hybrid retrieval. Adds conversation history in the hosted demo.
Explore Intermediate82.6 percent Hit at 5
Hit@5
Structured parsing, dense/sparse retrieval, reranking and parent context. Run this level locally.
Explore Advanced200 saved questions; 184 answerable. Processing, rewriting, retries and generation off. Overall Hit@5 differences are not statistically distinguishable in this dataset.
Page text, overlapping chunks and dense retrieval. A baseline you can follow.
Single-turn; previous questions are not passed to the model.
Explore NaiveTHE SAVED MEASUREMENTS
| Metric | Naive | Intermediate | Advanced |
|---|---|---|---|
| hit@1 | 46.2% | 45.1% | 59.8% |
| hit@3 | 68.5% | 71.2% | 77.7% |
| hit@5 | 77.7% | 79.3% | 82.6% |
| hit@10 | 88.0% | 86.4% | Not measured |
| mrr@5 | 0.7206 | 0.7538 | 0.8656 |
| recall@5 | 83.8% | 88.4% | 91.9% |
| location_hit@5 | 79.9% | 87.0% | 90.8% |
| Evidence coverage@5 | 83.1% | 85.4% | 88.8% |
Hit@5 requires the evaluator’s complete evidence rule. Recall@5 is keyword coverage; evidence coverage credits complete required entries. MRR@5 is the saved evaluator’s reciprocal-rank metric. Advanced Hit@10 is not measured because the returned output is capped at five parents.
TRADE-OFFS YOU CAN INSPECT
| Stage | Naive | Intermediate | Advanced |
|---|---|---|---|
| Input | Text PDFs | PDFs with text recognition | PDFs with structured parsing |
| Passages | Overlapping windows | Heading-aware passages | Smaller passages with surrounding context |
| Search | Meaning-based search | Meaning and keywords | Complementary search and reranking |
| Conversation | Independent questions | Recent conversation | No hosted conversation |
| Availability | Hosted session | Hosted session | Private local setup |