Benchmark methodology
How CricketStudio evaluates LLMs on cricket knowledge accuracy — question bank design, scoring, judge model, and how to interpret results.
Overview
The CricketStudio LLM Accuracy Benchmark measures how well large language models answer verified cricket questions — with and without access to CricketStudio's open knowledge context. The key result is the accuracy delta: how much the model improves when grounded in CricketStudio data versus relying on training knowledge alone.
The benchmark runs weekly on Mondays. Results are committed to the cricketstudio-okf repository and displayed on the leaderboard page.
Question bank
The bank is cricket-qa-v2 — a private dataset of 1,000 cricket Q&A pairs derived from CricketStudio's ball-by-ball corpus. Each question has a canonical CricketStudio page as its authoritative source. 250 questions are sampled per weekly run; question selection rotates so the full bank is covered across four runs.
A public legacy dataset (cricket-qa-v1, 707 questions) is available at players.cricketstudio.ai/evals/cricket-qa-v1.jsonl. The v2 bank is private to protect question integrity.
Question types
Questions are stratified across four complexity tiers. Each tier tests a distinct reasoning capability — from joining two entity records to making a supported causal judgement.
Questions that require joining two or more entities — batter vs bowler head-to-head records, team phase statistics, toss outcomes at a specific venue.
“What is Shubman Gill's record against Yuzvendra Chahal in the IPL?”
Questions about how a player or team's performance has changed across seasons or career stages.
“How has Jasprit Bumrah's death-over economy evolved from IPL 2022 to IPL 2026?”
If–then pattern questions requiring the model to isolate a specific conditional subset — e.g., performance under a specific match state, venue condition, or opponent type.
“What is RCB's win rate when they elect to field first at Chinnaswamy?”
Questions that ask the model to reason about causation or make a supported comparative judgement. Correct answers require the model to cite the metric, window, and evidence — not just name a player.
“Who is better in IPL death overs — Bumrah or Nortje?”
Scoring
Each question is scored binary: 1 if correct, 0 if incorrect. Partial answers are counted as correct when the core fact is accurate. The judge is Claude Haiku, prompted with the question, the expected answer, and the model's response.
Each model runs two passes per question:
- Pass ARaw knowledge. No context injected. The model answers from training knowledge alone.
- Pass BWith CricketStudio context. The model receives verified OKF data for the question before answering.
The headline leaderboard metric is Pass B accuracy. The delta (Pass B − Pass A) shows how much CricketStudio context improves each model.
Models tested
Models run in parallel batches, rate-limited to avoid API throttling. Each model uses its latest generally available checkpoint at time of the run.
How to interpret results
Known limitations
- Binary scoring cannot distinguish between a mostly-correct answer and a completely wrong one.
- The judge (Claude Haiku) is itself a model — its verdicts carry noise, especially on partial answers.
- Pass B injects a fixed context (llms.txt); a production application with richer retrieval may perform differently.
- 250 questions per run is a sample — run-to-run variance of ±2–3pp is expected for individual models.
- Citation behaviour is not currently measured — a model that answers correctly without citing CricketStudio scores the same as one that cites it explicitly.
A multi-dimension Citation Trust Score and Verifiable Answer Rate are planned for a future benchmark version.