DWS Research
DWSLM is a 1B-parameter language model we're building from scratch, to be trained on a single GPU.
The corpus, tokenizer, model code and training loop are all ours, and no pretrained weights are involved. The goal is decent scores on standard benchmarks, along with a written record of what it took to get there.
1B params · 23.2B tokens · 17 languages + code · 32,768-token vocab
- parameters, target
- 1B
- training tokens
- 23.2B
- documents
- 15.7M
- GPUs
- 1
00 data03 pretrain: now06 release
01Seven phases, three done.
Each phase produces a file that the next one consumes, so an error in one carries into everything after it. That's why the data and the tokenizer have exact counts and parity tests behind them (sections 02 and 03).
Evaluation won't wait for phase 06. The benchmarks start running with the first proxy model.
0215.7 million documents, all openly licensed. done
Eighteen sources: FineWeb-Edu and FineWeb-2 for the web, Wikipedia in 16 languages, The Stack v3 for permissively licensed code, public-domain books in ten languages, peS2o papers, SEC filings, old US newspapers and OpenWebMath.
Every document keeps its source, language and license. MinHash deduplication removed 181,089 near-duplicates, and an OCR quality gate removed 2,129 pages of scanner noise.
Half the tokens are in the 16 languages other than English, and a quarter are code (Fig. 3).
- documents
- 15,710,438
- size
- 38 GB, one parquet file
- tokens
- 23,207,662,201, counted exactly
- licenses
- ODC-By, CC BY-SA, CC0, public domain, permissive code
03A tokenizer built for this corpus. done
DWSLMTok32.3 is a byte-level BPE (byte-pair encoding) tokenizer with 32,768 tokens, built on GPT-4o's pretokenizer and trained on the corpus above. It changes three things from the standard recipe. Common Chinese characters get their own tokens, merges are spread across languages instead of going mostly to English, and digit runs split from the right so the groups line up with thousands.
Compared with plain BPE on the same corpus, each token carries 27% more Chinese text and 19% more Japanese. English and code each give up about 2.5%.
English · 41 bytes, 14 tokens
The quick brown fox counts 1234567 sheep.
Code · 62 bytes, 26 tokens
def fib(n):↵ return n if n < 2 else fib(n - 1) + fib(n - 2)
Hindi · 188 bytes, 27 tokens
अनुसंधान वह है जो मैं तब करता हूँ जब मुझे नहीं पता कि मैं क्या कर रहा हूँ।
04Now: training the 256M. now
Before the 1B run, we're training a 256M-parameter model on the same corpus, plus 289M rephrased tokens. That's one pass at about 90 tokens per parameter, a couple of days on one GPU, and its results decide what the 1B looks like.
The 256M is free to try layers from the past year's papers. The 1B has to run in llama.cpp and MLX, so it will reuse an architecture those runtimes already support, most likely Qwen3.8's. Small proxy runs came first. They settled the learning rate, the local layers, the init scale and the long-context recipe. The main run started on 11 November.
The 256M plan, and the papers behind it →
- params
- 256M first, 1B after
- tokens
- 23.2B, one epoch each
- ratio
- 90 / param at 256M, 23 at 1B
- compute
- ≈ 3.6 × 10¹⁹, then 1.4 × 10²⁰ FLOPs
- hardware
- 1 GPU
- wall clock
- ≈ 2 days, then to be set from the 256M
05What "decent" means.
Evaluation will use the standard suite (HellaSwag, ARC, PIQA, WinoGrande and MMLU) plus code and multilingual benchmarks, since a quarter of the corpus is code and half is in languages other than English.
The comparison is with open models around 1B parameters, and most of them trained on far more data. TinyLlama used 3 trillion tokens for 1.1B parameters, about 130 times our budget, and Llama 3.2 1B used up to 9 trillion (Fig. 6).
For DWSLM, decent means scoring close to models that saw a hundred times more data, not beating them.