DWSLM's first model has 256M parameters. It trains before the 1B, on the same 23.2B tokens, to find out which of the past year's ideas pay off on one GPU.

01Why 256M first.

One pass over the corpus at 256M parameters is about 90 tokens per parameter and roughly 3.6 × 10¹⁹ FLOPs. At the speeds we measured it takes about two days, a fraction of what the 1B needs, so a bug found late costs days instead of weeks.

The 256M isn't a scaled-down copy of the 1B. The 1B has to run in llama.cpp and MLX on ordinary GPUs and Macs, so it will reuse an architecture those runtimes already support, most likely Qwen3.8's. The 256M has no such constraint and can use layers that no runtime supports yet. Lessons about data, the optimizer and the schedule carry over to the 1B. Some of the layer choices won't.

Every choice is judged on loss per hour of training rather than loss per step. A change that lowers loss by 1% but makes each step 5% slower doesn't make the cut.

02Three local layers to every global one.

In each group of four, three local layers mix nearby tokens and one global layer attends over the whole context. The global layers do the long-range work, and the local ones are cheaper to run.

Three candidates for the local layers go head to head: sliding-window attention over 512 tokens, Gated DeltaNet (Yang et al.) and Kimi Delta Attention (Kimi Linear). The last two are linear attention, which keeps a fixed-size state instead of a cache that grows with the context. Qwen3.5 and Qwen3.8 use Gated DeltaNet in the same three-to-one pattern.

The evidence at our size is mixed. Olmo Hybrid replaced three of every four attention layers with Gated DeltaNet and reports about 2% lower bits per byte at 1B and 4% at 760M (Merrill et al.), but its pure Gated DeltaNet model also beat its transformer, which makes that baseline look weak. At 350M, Kimi Delta Attention trained with Muon had the lowest loss of the linear designs tested (Cerruti). MiniMax shipped a linear-attention hybrid and then went back to full attention for M2, citing weaknesses in multi-hop reasoning that small benchmarks missed (MiniMax).

Kernels for linear attention are also less mature than attention kernels, so a design that wins per step can still lose per hour. The contest is scored on the clock.

Kimi Delta won. At the tuned learning rate, it finished the 8-layer proxy at 1.313 bits per byte, Gated DeltaNet at 1.342 and the 512-token window at 1.401. It's also the slowest of the three, but by less than the kernels suggested: its kernels take about twice as long as Gated DeltaNet's (day 0), yet whole training steps were only about 6% slower, and about 27% slower than the window's. A 20-layer Kimi Delta run now checks that the result holds at full depth.

03The global layers.

Grouped-query attention with 8 query heads and 2 key-value heads, each 128 wide. Fewer, wider heads leave short-context loss about flat and improve long-context retrieval (Tokui et al.), and two key-value heads keep the cache small for local inference.

Each head's output passes through a learned sigmoid gate. In Qwen's tests the gate cut the share of attention dumped on the first token from 46.7% to 4.8%, and it kept a run stable at a learning rate where the ungated model diverged (Qiu et al.). Smaller activation outliers also make a model easier to quantize.

Queries and keys are normalized before attention. Positions use rotary embeddings with learned frequencies, which beat standard and partial rotary embeddings from 52M to 2.5B parameters (Karypis et al.).

There's one known risk. Ai2 found that normalizing queries and keys, combined with post-norm, cost about 6 points on a 32K-context benchmark at 7B (Bertsch et al.), and nobody has published YaRN context extension on learned rotary frequencies. Both get an early test on a proxy checkpoint stretched to 32K.

Both checks came back clean. An 8-layer proxy trained at 4,096 tokens was stretched to 32,768 on long documents. With YaRN, bits per byte on the last 16K tokens of each document fell to 1.038, against 1.068 without it, at no cost on short text. Dropping QK-norm made long-range loss slightly worse, 1.045 against 1.038, so it stays. One lesson from a false start: the extension has to begin at the base run's final learning rate. A first try at four times that rate raised the loss.

04Cheap additions.

Every other layer adds a learned per-token vector to its attention values. These value embeddings are table lookups, so they cost memory and almost no compute. Karpathy's nanochat log counts them among its biggest wins, and He et al. found lower loss at 135M and 780M (arXiv 2606.02780).

Normally each layer adds its output to one running sum. With attention residuals, each layer instead attends over the outputs of all earlier layers and takes what it needs. Luo et al. report validation loss 0.06 to 0.15 lower from 100M to 1B, with no added parameters (arXiv 2607.27230), and Kimi K3 uses the idea in production. At our size it's still one group's result.

The feed-forward layers use squared ReLU instead of SwiGLU, which needs two matrix multiplies instead of three, and it won on wall clock at both sizes in nanochat's tests. Output logits are soft-capped at about 20 inside the fused loss kernel to guard against late loss spikes. Input and output embeddings start tied and get untied two-thirds of the way through, a trick from the modded-nanogpt speedrun.

Leave-one-out runs on the final 8-layer recipe show which of these carry weight. Without value embeddings, bits per byte rose by 0.029. Without attention residuals, QK-norm, learned rotary frequencies or the residual scalars, each rose by 0.006 or less. All five stay in the main run. Attention residuals were the closest call: without them, training ran 7.9% faster for 0.005 more bits per byte, about even per hour of training at 8 layers, and their gain should grow at 20. A repeat of the full recipe with a different seed, which would have measured the noise, was lost when the rented machine running it went offline overnight.

05The data, as the model sees it.

One pass over the corpus. The 16 languages besides English keep their share. Cutting them would help English at 256M, but the 1B needs them and the tokenizer was built for them. Wikipedia, FineWeb-Edu, science and math are repeated 1.5 to 2 times. The noisiest OCR newspaper pages and filings are trimmed, and OCR documents full of rare garbage words are filtered at load time.

About 1B tokens, roughly 4% of the text, will be our best English sources rewritten by a small model as FAQs, tutorials, tables, math, Q&A and dialogue. A first pass got about halfway overnight before its machine went offline, and it will be redone before the main run. Roberts et al. find the best share of rewritten data grows with compute, from under 10% at 3 × 10¹⁸ FLOPs to 32% at 10²² (arXiv 2610.31867), and Niklaus et al. found rewriters above 1B parameters added nothing (arXiv 2604.13977).

Half of the code documents train as fill-in-the-middle, which costs nothing measurable in left-to-right ability (Bavarian et al.). Files from the same repository are stitched together for the long-context phase.

Documents never attend across each other; packing studies found that gives the lowest perplexity, most of all on code (Zhao et al.). Context grows from 512 to 4,096 tokens during training, which was both faster and better at 1B (Zhu et al.). StochasTok costs about 7% more compute, so it gets a proxy test before it goes in.

06Optimizer and schedule.

Weight matrices train with MuonH (Wen et al.); embeddings, norms and scalars use AdamW. Muon orthogonalizes each update. The Hyperball variant also holds weight and update norms fixed, which replaces weight decay, and its authors report reaching the same loss on 20 to 30% fewer tokens up to 1.2B.

That result is contested. Another group found Hyperball slower early, catching up only later (Xiao et al.), which is how a short proxy run would undersell it. Plain Muon runs alongside as a control. For scale, an independent benchmark found Muon about 1.3 times as token-efficient as well-tuned AdamW up to 520M, falling to 1.1 at 1.2B (Wen et al., 2025).

MuonH stays. On the 8-layer proxy, plain Muon finished about 0.02 bits per byte lower at its best learning rate (1.3225 at a peak of 2e-2, against MuonH's 1.3421 at 1e-2). A short proxy is where Xiao et al. also found Hyperball behind.

The learning rate warms up, holds, then decays over the last 20 to 25% of tokens to under 5% of peak (Fig. 7). Decaying close to zero beat stopping at 10% of peak in tests at 1B and below (Bergsma et al.). The flat stretch lets us fork the decay: two decays with different data mixes start from the same checkpoint, and the better one wins. During the decay, Wikipedia, science, math and clean code get 2 to 4 times their usual share, and an average of the weights over the decay is kept as a second candidate.

Batches are 512K tokens. Training runs in bf16; FP8 for the feed-forward layers gets one throughput test and stays only if it's more than 10% faster with no loss gap. It passed: 11.7% more tokens per second at the same loss (1.3116 bits per byte against 1.3130 in bf16), on the small GPU that ran the proxies. The GPU the main run uses showed no gain from FP8 (0.98 to 1.00 times the speed), so the main run trains in bf16.

07Proxy runs, in order.

Up to four days of small models before the main run. Small models can rank choices reliably enough: in AI2's DataDecide, rankings at 150M picked the 1B winner about 80% of the time (Magnusson et al.).

The learning rate comes first. Comparing designs at an untuned learning rate misleads, and after tuning learning rate and weight decay together, about 90% of one reported optimizer gain disappeared (Xin et al.). The rest follows expected payoff. Losing arms get dropped as results come in.

The last two steps check the stack as a whole. Leave-one-out runs remove each addition from the winning design to see what it's worth in combination, and a 32K probe tests the long-context risks from section 03.

So far, the learning-rate sweep put the best MuonH peak at 1e-2, half the planned default, and a 20-layer check landed in the same place: 1.300 bits per byte, against 1.326 at half that rate and 1.328 at double. The main run trains 23 times longer than the sweep, so it will peak lower, near 3.7e-3. Kimi Delta kept its lead at 20 layers (1.273 against Gated DeltaNet's 1.300). StochasTok cost about 0.04 bits per byte with either local layer. On a spelling probe it helped (0.43 against 0.35 accuracy), but it did worse on the first-letter, counting, addition and comparison probes, so it's out. Halving the initial weight scale helped Gated DeltaNet by 0.018 and Kimi Delta by 0.007, so the main run starts at half scale. Soft-MTP and COUNT are out for the 256M. On a full read of its paper, plain next-token training beat Soft-MTP at 167M and 332M, and it only pulled ahead from 689M. COUNT needs a large n-gram build in host memory, and most of its small-scale gain came from scaling the input. Both get another look at 1B. Kimi Delta won the local-layer contest and MuonH stays.

08How it gets scored.

The main number is bits per byte on held-out documents, per source and per language. It doesn't depend on the tokenizer, so it compares directly with other models, and AI2 found likelihood metrics like it rank small models more reliably than benchmark accuracy (Heineman et al.).

HellaSwag, ARC, PIQA, WinoGrande and MMLU come second, in cloze format and scored by likelihood, after a 13-gram overlap check against the corpus. The baselines are SmolLM2-360M, Gemma 3 270M, LFM2.5-350M and Qwen3.5-0.8B. All of them trained on far more data, some of it from the same public sources, which flatters their held-out scores.

Training metrics will stream to a live dashboard on this site, about a minute behind the run.

09After pretraining.

Supervised fine-tuning on outputs from much larger models, then on-policy distillation, with long and short reasoning traces mixed one to four. Then work in weight space: merging checkpoints, task vectors, and direct edits in the spirit of MIT CSAIL's weight-space programming experiments. None of that starts until the base model exists.