Benchmarks · 793 questions · updated 21 October 2026
How accurate is Kāla?
We asked Kāla and ten other AI systems the same 793 yes/no questions about real events between July and October 2026, after every model's training data, and scored them once the answers were known. Here is where Kāla leads, and where it doesn't.
Most consistent
Best average rank across five kinds of question (3.4 vs 4.4 for the next best). Every other model wins one kind and slumps on another.
Level with the best
Overall, Kāla is statistically tied with Kimi K2 and Google's Gemini 2.5 Pro, the strongest models we tested.
Ahead of the specialists
Better overall than every forecasting-specialist model tested; clearly ahead of Foresight-32B and LlamaForecaster.
Clearly ahead of most rivals
On the same questions, Kāla is ahead beyond doubt of Qwen3, Gemini 2.5 Flash, gpt-oss, Foresight-32B and LlamaForecaster.
Compare by kind of question
Bar = how much better than a coin flip (always saying 50%). Longer is better. Tap a kind of question, or narrow the field.
Head to head, on the same questions
Each line compares Kāla with one system on exactly the questions both answered. "Ahead" or "behind" only when the gap is bigger than chance (95% interval); otherwise "tied".
| Against | Type | Questions | Result |
|---|
Where a prediction market already trades on the question, its price is still slightly better than every AI model, Kāla included. Kāla's value is the many questions no market covers, and the reasons behind each number.
Ranked by others: live tournaments
Our own test can't rank Kāla against the newest systems, because each benchmark uses different questions. So Kāla also competes where independent organisers score everyone on the same live questions.
| Tournament | Organiser | Kāla's entry | Status |
|---|---|---|---|
| FutureEval, Fall 2026 (AI bots vs pro forecasters) | Metaculus | Kala-Bot | 11 forecasts made; none resolved yet |
| MiniBench (fast-resolving questions) | Metaculus | Kala-Bot | 49 forecasts made; none resolved yet |
| Live head-to-head: 60 open questions | Kāla (fingerprint b5a78b0f…2474) | Kāla vs Gemini 2.5 Pro (with and without its own Google Search research), Kimi K2, gpt-oss, Qwen3 | forecasts locked 20 October 2026; scored as questions resolve (3–6 weeks) |
| ForecastBench tournament leaderboard (AI systems vs superforecasters) | Forecasting Research Institute | Kāla ensemble | entry in preparation; ranked about 50 days after the first round |
For context: in Spring 2026 Metaculus's best bots trailed professional forecasters by only 1.25 points per question, and on ForecastBench superforecasters still led the best AI by about 0.017 Brier in January 2026. We'll post Kāla's standings here as they come in, good or bad.
What these numbers don't show yet
They are a back-test on questions after the models' training. The real test is live: since October 2026 every Kāla forecast is locked in, fingerprinted and published before the answer is known, then scored. That record is just starting; follow it on the scoreboard and the live board. We'll update this page as it grows, and claim a lead over any model only when the live record shows it.
How we score
- Same questions, same evidence: every model reads the same dated evidence, all published before the question's as-of moment.
- No look-ahead: calibration for each question is fitted only on questions that had resolved before it was asked.
- Written down first: each experiment's design and pass mark were recorded before it ran.
- Honest uncertainty: differences are reported with 95% paired-bootstrap intervals; "ahead" means only that the interval excludes zero.
Research log: every experiment, including the ones that failed
| Experiment | Result (Brier, 793 questions) | Pass mark | |
|---|---|---|---|
| M1: Kāla-1 (Qwen3-8B) trained on three AI teachers' answers | 0.212 (level with gpt-oss 120B) | 0.210 on the dev benchmark (scored 0.216) | fail |
| M1b: four teachers including Kimi K2, more questions | 0.209 | beat M1 clearly | fail |
| M2 / M2b: reinforcement learning on outcomes | 0.213 / stopped early | beat M1 | fail |
| Phase 1: 8 sampled answers + rolling calibration | 0.202 | beat the single answer; get within 0.005 of the four-model average | pass fail |
| Phase 2 + recent-news pre-training | 0.203 | beat phase 1 | fail |
| Generated questions from news (stage B data) | 73% labelled correctly | 90% on a hand-checked sample | fail |
| Learned weights for the four models (+ Kāla-1) | 0.1914 (0.1905) | beat the plain average clearly | not proven |
| Kāla-1 + statistics tool + market price + per-source calibration | 0.198 (no gain under the stricter test below) | beat Kimi K2 (0.192) | fail |
| Kāla-1 14B, Kimi-weighted training | 0.199 | beat Kāla-1 8B + tools, beat Kimi K2 | fail |
| Head-to-head with Gemini 2.5 Pro and the forecasting specialists (same evidence, calibration fitted only on earlier test questions) | Kāla 0.192 · Gemini 2.5 Pro 0.195 · Kimi K2 0.191 · OpenForecaster-8B 0.199 · Foresight-32B 0.201 | report honestly; claim only what the 95% interval supports | level with the best |
What we learned: averaging several strong models beats any one we can train; Kāla-1's own answers still add information to that average (most on prediction-market questions); copying Kimi K2's answers doesn't transfer its broad world knowledge; and Kāla-1 is strongest on India, where it trains on data the big models don't see. Full details in the research notebook.