How much does verified cricket evidence improve LLM answers?
We test each AI model twice on 250 sampled questions drawn from a 1,000-question cricket benchmark bank. Pass A: no data — the model answers from training knowledge alone. Pass B: CricketStudio's verified stats injected as context. The gap shows exactly how much structured cricket data improves AI accuracy.
Highest accuracy across all models when CricketStudio data is injected as context.
Started lowest, gained the most. CricketStudio data is the equaliser.
- ▸Question sent with no context — model answers from training data alone
- ▸250 sampled from a 1,000-question bank — cricket-qa-v2
- ▸Four types: single-entity facts, career arcs, compound conditions, causal debate
- ▸This score reflects what the model "knows" about cricket without any help
- ▸Same question, but CricketStudio's OKF knowledge bundle injected as context
- ▸Context source: okf.cricketstudio.ai/llms.txt
- ▸All OKF facts derived from ball-by-ball data with explicit provenance
- ▸The "Improvement" column shows how many accuracy points CricketStudio adds per model
The bank holds 1,000 questions across four difficulty levels; each weekly run samples 250 of them. Even the “simple” ones stump most LLMs without CricketStudio data — they require exact ball-by-ball aggregation that no model carries in training.
Stats about one player, team, or venue in a specific context. Sounds straightforward — but requires exact ball-by-ball aggregations that LLMs don't carry in training.
How has a player evolved across multiple IPL seasons? Peak vs decline questions need season-by-season breakdowns — impossible without structured time-series data.
Two or more conditions stacked together. Requires filtering the dataset to only the matches where both X and Y are true simultaneously.
Who is better? Why does X outperform in Y conditions? Good answers define the metric, time window, and era before concluding — not just picking a name.
- ▸Judge: Claude Haiku — scores each response binary 0 or 1
- ▸Partial answers counted correct when the core fact is accurate
- ▸Models run in parallel batches, rate-limited to avoid throttling
- ▸Run weekly · Mon 06:00 UTC · results committed to this repo
- ▸Methodology published — see benchmark design
Benchmark: cricket-qa-v2 · Judge: claude-haiku-4-5-20251001 · Benchmark methodology· Citation policy