The DWSLM 256M run, live, about a minute behind the GPU.

One pass over the corpus plus the rephrased data, on a single GPU. Everything that led up to it, from the learning-rate sweep to the anneal mix, is on the pre-run page.

Live data needs JavaScript; the plan below doesn't.

Waiting for the first update.

Fig. 1 Where the run is.

01The plan.

The 256M model trains for about 23.6B tokens: one pass over the corpus plus 289M rephrased ones. Context grows from 512 to 4,096 tokens, the learning rate holds and then decays over the last fifth, and from three quarters in, high-quality sources are sampled twice as often. Checkpoints are kept at both of those points.

model: 20 layers × 1,024, Kimi Delta local layers
batch: 512K tokens
precision: bf16
peak learning rate: 3.7e-3, MuonH
schedule: decay from 80%
anneal mix: high-quality ×2 from 75%

Waiting for the first update.

Fig. The plan against tokens so far. Context steps above the bar, anneal and decay below.

03What has happened.

Milestones, newest first. Steps skipped for a large gradient and routine checkpoints are counted, not listed.

04Scores.

Held-out bits per byte comes in every 500 steps, by source and language. Cloze accuracy is measured on a 500-example subset per task about every tenth of the run, and on the full suite at the end. Both fill in as they land.

Bits per byte by slice

Fig. Bits per byte by source and language, latest measurement.