Evaluation
Retrieval ablation and answer-quality metrics over the golden set. The browser renders the committed artifact generated by eval/run_eval.py.
Real evaluation. Real evaluation over 13 indexed videos and 135 hand-labeled queries. Metrics are regression signals, not universal benchmark claims.
- Golden set
- eval/golden/expanded.jsonl
- Generated
- Jun 11, 2026, 9:01 PM ET
- Judge
- haiku
- Primary config
- production
Config:
Retrieval ablation · 135 golden queries · top-10
| Config | MRR | Timestamp@5s | No-answer F1 |
|---|---|---|---|
| Dense only | 83% | 77% | 71% |
| Dense + loose gate | 74% | 67% | 0% |
| Dense + strict gate | 67% | 60% | 34% |
| Hybrid BM25 | 71% | 60% | 0% |
| Hybrid + rerank | 74% | 64% | 0% |
| Hybrid + rewrite | 71% | 60% | 0% |
| Production | 79% | 73% | 71% |
Answer quality — Production
- Quality
- 84%
- Grounded
- 92%
- Correct
- 80%
- Useful
- 94%
117 answerable queries judged by haiku
No-answer gate
10
True refuse
1
Missed refuse
7
Over-refuse
117
True answer
Refusal precision 59% · recall 91%