DWSLM-256M: model card and technical report

01Model overview.

02Architecture.

• Local layers use Kimi Delta Attention (KDA), a gated linear-attention layer with a fixed-size state per head. KDA won a bake-off on an 8-layer proxy at 1B tokens: 1.313 bits per byte against 1.342 for Gated DeltaNet and 1.401 for sliding-window attention, and it also led on loss per GPU-hour.
• Global layers (3, 7, 11, 15, 19) are grouped-query attention with QK-norm, a per-head sigmoid output gate and LeRoPE, rotary embeddings with one learned frequency scale per band.
• Residual stream: block attention residuals in the Kimi form. Each group of four layers reads a learned, per-token mix of the embedding and the finished groups, and learned scalars add the raw embedding back into each sublayer's input.
• Value embeddings: ten lookup tables, one per alternating layer, add a token-indexed vector into the attention values.
• Head: tied to the input embedding for the first two thirds of training, then untied. Logits are scaled by 1/d and soft-capped at 20.
• Init: CompleteP depth scaling, small nonzero output projections, and an init scale of 0.5.

Removing pieces one at a time from the 8-layer proxy cost: value embeddings +0.029 bits per byte, QK-norm +0.005, LeRoPE +0.003, the λ scalars +0.001. Attention residuals came out even per GPU-hour (+0.005 bits per byte, but 7.9% faster without them) and were kept because their reported gains grow with depth.

03Pretraining data.

The corpus is 15.7M documents from 18 openly licensed sources, 23.2B tokens:

By language, 25% is English, 26% is code and 49% is spread over 16 other languages. MinHash deduplication removed 181,089 near-duplicates and an OCR quality gate removed 2,129 pages of scanner noise.

• Rephrased data: 289M tokens of synthetic rewrites (FAQ, tutorial, table, math, Q&A and dialogue formats) of FineWeb-Edu, science and OpenWebMath documents, written by SmolLM2-1.7B-Instruct. They train beside the corpus at weight 1.
• Held out: about 0.1% of documents, stratified by source and language, are never trained on. All bits-per-byte numbers are measured on that split.
• Decontamination: every training document sharing a rare 13-token sequence with an evaluation item was dropped before training (2,802 documents).

04Pretraining.

The learning rate comes from a five-point sweep on the 8-layer proxy at 1B tokens (optimum 1 × 10⁻², confirmed at 20 layers), scaled to 23B tokens with HyperP's η ∝ T^-0.32 rule. 370 steps were skipped (0.8%), mostly on gradient spikes in the global layers' attention projections. Attention-logit checks at 14K and 29K steps stayed well inside our limits.

05Long-context extension (Base 32K).

Starting from the final 4K checkpoint, we trained 953 more steps (500M tokens) at 32K context on documents of at least 16K tokens, with YaRN ×8 over the learned rotary frequencies and a learning rate starting at the base's final floor (1.85 × 10⁻⁴).

Bits per byte by position in the document, early in the phase (step 95) and at its end:

Short-context quality held: overall held-out bits per byte went from 0.9963 to 0.9957 and the cloze average from 0.4063 to 0.4066. On the proxy, YaRN beat plain extension at 16K to 32K (1.038 against 1.068) and cost nothing below 8K.

06Post-training (Chat).

6.1 Data

Conversations are rendered with reserved role tokens (system, user, assistant, tool, call, end) into 4K windows. Loss falls only on assistant tokens, so the model learns to write its own turns and never anyone else's.

Agent traces dominate by volume. The three conversational sources are 7.2% of the loss tokens, upweighted on purpose to make the model chattier. Assistant turns in the Gemini, Fable and Open-SWE data were written by larger models; we use them under each dataset's terms.

Privacy processing. Every source was scanned for secrets (API keys, assigned credentials, email addresses) and those were redacted. In the Discord data, every participant's handle, every mention and channel tag, every URL and phone number, and a list of real names were replaced with "someone", and bot messages were dropped. Other people's messages appear only as the user side of a conversation, which carries no loss. Exact and prefix duplicates were removed across exports.

6.2 Fine-tuning

A full fine-tune of Base (4K): one pass over the mix, 706 steps of 524K tokens, the pretraining optimizers at a tenth of the pretraining learning rate (3.7 × 10⁻⁴), 2% warmup then linear decay to zero. It took 36 minutes. Loss on a held-out 1% of conversations per source, mostly agent traces, fell from 2.389 (the base model reading the same template) to 1.627.

6.3 Weight-space post-processing

After fine-tuning we tried three kinds of weight edits. The selection rules were written down before any result came in: candidates are scored on the even-numbered items of the cloze suite (selection half) and reported on the odd-numbered ones (report half). A candidate must stay within 3% of the best held-out loss (all sources, mostly agent traces); among those, the best selection-half score wins.

• Tail average: the mean of the last five kept checkpoints (steps 550 to 706). It matched the last step.
• Blends interpolate back toward the base, θ = base + α(tuned − base). Lower α buys back up to 0.4 points of cloze and gives up 14% of held-out loss. That trade isn't worth it here: the whole spread is inside the suite's noise (about ±0.4 points on the average), and the held-out loss is what the chat model is for.
• Loop edits. Under greedy decoding, 13.3% of replies to 128 held-out prompts fall into a loop (some 6-token sequence repeats three or more times). Following recent work on weight-space editing, we took the mean hidden state inside loops minus the mean over loop-free replies, and projected that direction out of either the output head or every write into the residual stream, at strengths 0.5 and 1. Every variant made looping worse: 17.2% and 36.7% at the head, 15.6% and 44.5% in the stream. All four were rejected. Our reading is that loop-free replies end and looping ones don't, so the contrast also captures the model's "time to stop" signal, and removing it removes the stop. A contrast between positions matched for distance from the end would avoid that.

Fine-tuning on chat data left the benchmark scores where pretraining put them. That is the expected result for a cloze suite: it measures what the model knows and how it reads the task's format, and chat data teaches neither. Section 6.4 moves them with data built for the task.

6.4 Benchmark train splits and task arithmetic (Chat v1.1)

The cloze suite never asks the model to write. It compares how likely the model finds each option, so a model that has never seen a task's format or subject matter can have nearly random preferences between options even when it knows the words. Version 1.1 teaches those formats with the benchmarks' own training data, then merges that change into the chat model.

Data. Every split the suite doesn't score on, in the eval's own prompt format, with loss only on the gold answer:

OpenBookQA's train split turned out to be inside MMLU's auxiliary set already. Two filters ran against every evaluation item: 940 items sharing a rare 13-token run were dropped, then 123 more whose question text (normalized) matches an eval question exactly, which catches short questions that 13-grams miss. A run trained before the second filter scored the same (0.4512 against 0.4513 on the full suite), so the filter changed nothing measurable.

Bench fine-tune. From Base, three passes over that data (248 steps of 524K tokens, 46 minutes), with the chat fine-tune's learning rate and schedule. Alone it scores 0.451 on the full suite, but its conversation is worse than the base model's.

Merge. Task arithmetic: θ = θ_base + (θ_chat − θ_base) + λ(θ_bench − θ_base), with θ_chat = Chat v1.0 and λ in {0.25, 0.5, 0.75, 1}. The rule, fixed before scoring: loss on held-out conversations (the Discord, reply-pair and Gemini sources) within 2% of Chat v1.0, overall held-out loss within 3%, then the best selection-half score; report the other half.

λ = 0.5 is the last weight inside the 2% limit (+1.8%). It adds 3.4 points on the report half, well outside the suite's noise. λ = 0.75 and 1 score higher still, but cost 4% and 8% of conversation loss. These numbers come from a second scoring pass with a different attention kernel; Base scores 0.4067 there against 0.4063 in section 7.1, so the two passes agree within 0.0004.

The gain is uneven in the way format training predicts: ARC-Easy +10 points and ARC-Challenge +3 (grade-school science, which MMLU's auxiliary set is full of), HellaSwag +1.6, PIQA +1.3, MMLU +0.8, WinoGrande flat. It reflects the tasks' format and subject matter, not broader ability, and it isn't comparable with models that trained on none of these splits.

Loop edits, again. On Chat v1.1, 12.5% of greedy test replies loop. We repeated the loop-direction edit with a second contrast: positions inside a loop against the same replies' positions before the loop, which removes the "reply is ending" signal. It did better than the original contrast (13.3% to 18.0% against 16.4% to 91.4%) but still never reduced looping, so no edit was kept again. Removing one direction doesn't stop this model from repeating itself.

07Evaluation.

7.1 Benchmarks

Length-normalized accuracy (log-likelihood per UTF-8 byte of each option), full evaluation sets, all models scored by the same harness in BF16:

Prompts follow lm-evaluation-harness and OLMES cloze formats; MMLU uses SmolLM's cloze prompt with no lettered choices. Splits: HellaSwag, PIQA and WinoGrande validation; ARC and MMLU test. The average is the unweighted mean of the six tasks, with a standard error of about ±0.4 points. Token counts for the baselines are from their own model cards where published. † Chat v1.1 trained on these benchmarks' train splits (section 6.4) and was scored in the second pass; the other columns saw none of that data.

MMLU at 27% is barely above chance (25%), and ARC-Challenge at 25% is at chance. The gap to the baselines is about what the token budgets predict.

7.2 Held-out text

Bits per byte on the held-out split of our corpus (lower is better). Each model tokenizes the text itself.

This comparison favors us in one way and the baselines in another. The split comes from our own training distribution, which no other model was tuned for. But the sources are public, and the baselines may have trained on these very documents (SmolLM2's corpus includes FineWeb-Edu and The Stack outright). Read it as a sanity check, not a ranking.

7.3 Chat

We ran no chat benchmark. At this size, judge-scored suites like MT-Bench sit at their floor. Two real replies from the demo, at the default settings:

User: What is the capital of France?
DWSLM-256M Chat: i mean

User: rust vs go
DWSLM-256M Chat: i'm going to say the same for this game, which is what i did today

The casual voice from the conversational data came through. The knowledge to back it up did not. Chat v1.1, greedy:

User: What is the capital of France?
DWSLM-256M Chat v1.1: Paris

User: rust vs go?
DWSLM-256M Chat v1.1: go is a better build for this

User: Which gas do plants take in from the air?
DWSLM-256M Chat v1.1: sodium nitrate

Shorter and more direct, as the benchmark data's one-line answers would suggest, and still wrong often.

08Safety and responsible use.

• No safety training. There was no refusal training, preference tuning or red-teaming, and no safety, bias or toxicity benchmark was run. The model refuses nothing, and it can produce offensive, false or nonsensical text, especially in the chat register it learned from Discord.
• Low capability. At this size it can't reliably follow multi-step instructions, write working code beyond fragments, or state technical facts accurately. That limits misuse, and it also means nothing it says should be trusted.
• Privacy. The chat data includes our own messages and agent sessions, processed as described in section 6.1. Small models memorize less than large ones, but the model could still reproduce fragments of its training conversations.
• The demo serves DWSLM-256M Chat v1.1 and keeps no conversations: the page holds the messages, and the server holds only a session slot. Three people can chat at once, the rest wait in a queue, and sessions last ten minutes. Inputs are capped in length and count, and the server logs request routes and status codes, never content or addresses.

09Limitations.

• Knowledge: well below models trained on a hundred times more data; MMLU and ARC-Challenge are at or near chance.
• Coherence: replies are short and often unrelated to the question. 12.5% of Chat v1.1's greedy replies loop; the demo samples at temperature 0.7 with a repetition penalty of 1.1, which hides most of it.
• Languages: half the pretraining corpus is not English, but nearly all the chat data is. Chat in other languages is untested.
• Context: the chat model has a 4K context. The 32K base is not chat-tuned yet, because our fine-tuning code doesn't load YaRN checkpoints.
• Benchmark scores: Chat v1.1's 3-point gain comes from training on the benchmarks' train splits. Compare it only with models that did the same; the base models' scores are the clean comparison.
• Evaluation coverage: cloze accuracy and bits per byte only. Generative, code and multilingual benchmarks are still to come.

10What's next.

DWSLM-256M is the first model in the DWSLM line. Its runs set the recipe for the 1B model, which will use a layout llama.cpp and MLX already support (most likely Qwen3.8's). Lessons carried forward: chunk long books and interleave them across the data order, since streaming them whole skewed some training steps; budget evaluation noise before reading small differences; fix selection rules before seeing results.

ARejected ideas and measured negatives.

BEvaluation details.

• Cloze: each option is scored as the log-likelihood of its text given the prompt; accuracy takes the highest raw score, and length-normalized accuracy divides by the option's UTF-8 bytes. The suite is decontaminated against the training corpus at 13 tokens.
• Bits per byte: total negative log-likelihood in bits over the total UTF-8 bytes of about 5M held-out tokens. Our model scores 4,096-token packed rows that restart at row edges; the baselines score each document from its start with up to a 32K window. Both choices slightly favor the baselines.
• Selection and report halves: even- and odd-indexed items of every task, fixed before any post-training candidate was scored.