Watch Kāla learn
A live view of a model while it trains on a cloud GPU. This run teaches a pocket-size, long-memory version of a 0.6B model by having it copy the original, layer by layer, then word by word. The page refreshes every 30 seconds.
The model, layer by layer
Words enter at the bottom and flow up through 28 layers. Each dot is a token moving through the network. Hover or tap a layer for details.
Learning curve
Lower is better: the gap between the new model and the original. Hover for exact values.
What the current gap looks like
An illustration, not model output: two next-word predictions that differ by exactly the current measured gap, for “The monsoon reached Kerala on …”.
Report card
Targets were written down before training started. Perplexity: lower is better (how surprised the model is by real text). The bars show how far each result is from the original; the black line is the allowed limit.
Why bother: memory
While it reads, a model stores a cache for every token in every layer. Full-attention layers keep all of it; converted layers keep only their window plus a fixed-size summary. Drag to change the length of the history.
32K tokens · – less memory
Each column is one layer: how many tokens it keeps. Approximate cache size in bf16 for Qwen3-0.6B (8 key-value heads × 128 dimensions); the summary state is counted in fp32.
The passkey test
A 5-digit code is hidden at 12 different depths in a long stretch of filler text, and the model is asked to repeat it. The question is whether the compact memory can still hold on to it.
Only the number found per length is logged, so the dots show the count, not which depths were missed.
What you're watching
Why: a normal model re-reads its whole history for every new word, so memory grows with every word. The converted layers instead look at the last 512 tokens and keep a small running summary of everything older, so their memory stays the same size however long the history gets.
1. Layer test. Each of the 28 layers is switched to the cheap version one at a time to see how much it hurts. The 6 that hurt most keep full attention (gold).
2. Imitate (stage 1). Each converted layer is trained to produce the same internal outputs the original layer did. The curve shows the average squared gap.
3. Copy the predictions (stage 2). The whole model learns to give the same next-word probabilities as the original, on texts of 2K, then 4K, then 8K tokens. The curve shows the KL divergence per token: how different the two predictions are; 0 means identical.
4. Final checks. Perplexity on held-out web text and books, and the passkey test at up to 48K tokens, past the 32K the original was trained on.
Results are logged in the research notebook, misses included.