Everything DWSLM tested before its main run.

Small models, each testing one choice, plus the benchmarks and the data work that came first. Every result is here with the decision it led to, and failed or lost runs stay up. The main run itself is on the dashboard.

Run data needs JavaScript; the results table doesn't.

Waiting for the first update.

Fig. 1 Runs before the main run, by outcome.

01Results and decisions.

What each test found and what it settled for the 256M main run. Scores are held-out bits per byte; lower is better. K numbers refer to the decision log.

test	result	decision	K
day-0 benchmarks	every kernel checked and timed; FlashAttention-2 beat cuDNN by 13 to 30% on the small GPU	compiled loss head; attention re-benched on the main GPU	G2, G4, K25
learning rate	1e-2 best at 8 layers (1.342); 1.300 at 20 layers against 1.326 at half and 1.328 at double	3.7e-3 for the main run's 23B tokens	K11, K15
depth transfer	the 20-layer optimum, 0.98e-2, matches the 8-layer proxy's	one LR at every depth; sublayer outputs scaled by 8 over the layer count	K8, K11
local layers	Kimi Delta 1.313, Gated DeltaNet 1.342, window 1.401	Kimi Delta	K12
optimizer	plain Muon 1.3225, MuonH 1.3421	MuonH kept	K13
StochasTok	costs 0.042; spelling up, counting and arithmetic down	out	K17, K20
init scale	half scale: Kimi Delta 1.306 vs 1.313, Gated DeltaNet 1.324 vs 1.342	half scale	K22
leave-one-out	dropping value embeddings costs 0.029; attention residuals 0.005 for 8% of the speed; the rest 0.005 or less	all kept	K22
32K probe	YaRN 1.038 vs 1.068 at 16K to 32K tokens, free at short range	YaRN for long context	K19
FP8 MLP	11.7% faster on the small GPU with no loss gap; no gain on the main GPU	bf16 for the main run	K21, K25, K30
Soft-MTP	not run; the paper loses to plain training below 689M	dropped for 256M, revisit at 1B	K9
COUNT	not run; needs a 30 to 50 GB table	dropped for 256M, revisit at 1B	K10
anneal mix	proxies lost to a rented-machine outage, then dropped	high-quality data at 2x from 75%, checkpoint kept to redo the last quarter	K23, K28
B5 rephrase	planned 1B tokens; stopped at 289M to start training sooner	289M rephrased tokens in the main run	K18, K27
speed work	kernels were 95% of the step; 134K to 182K tokens per second after fixes	bf16, one large micro-batch	K30

Fig. 2 Pre-run results and the decisions they led to. Full reasoning on the model page.

02Every run.

Pick a run to see it beside the rest of its test, and up close further down. A stale run hasn't reported in 10 minutes; a paused one is waiting on purpose.

run	state	tokens	loss	bpb	tok/s	finish
No runs yet.

Fig. 3 Every run, newest first. Loss and speed average the last 20 reports.

03One run up close.

Pick a run in the table above.

Latest events, newest first.

Bits per byte by slice

Fig. 4 Bits per byte by source and language.