DWSLMTok32.3 is DWSLM's tokenizer: 32,768 tokens of byte-level BPE (byte-pair encoding), trained on our own corpus.

It follows the GPT-4o design on purpose, because that's the boring option that works. Three changes from recent tokenizer research sit on top, all within the same 32k vocabulary.

byte fallback · o200k regex, right-aligned digits · Rust, with a byte-identical Python export

Fig. 1 All 32,768 ids, to scale, then the first 2,125 zoomed in.

01Try it.

This runs DWSLMTok32.3 in your browser from the same tokenizer.json the training shards were encoded with, next to GPT-2, Mistral and GPT-4o's o200k.

Text

Show tokens from

- tokens

- characters

- bytes/token

Fig. 2 One tint per token; hover over a token to see its id. A small number marks a character that took several tokens.

tokenizer	vocab	tokens	bytes/token

Fig. 3 The same text in four tokenizers; fewer tokens is better. o200k has six times our vocabulary, so treat it as a reference point.

0232,768 ids, three kinds.

ids	what
0-31	32 special tokens, 13 named, 19 reserved
32-287	one per byte, so nothing is unencodable
288-32,767	32,480 learned merges

Specials never come out of encode(); the training code injects them. There's no Unicode normalization anywhere, so decode(encode(text)) == text for any input, including text that spells out a special token's name.

Fig. 4 The 13 named specials, ids 0 to 12, including fill-in-the-middle tokens for code and a pair for reasoning traces. Reserving them now means post-training won't need to resize the embedding table.

03Chinese gets its own characters. 32.2

Plain BPE on this corpus (32.1) made only 377 Han characters single tokens, 57.6% coverage, so most Chinese characters cost two or three byte fragments.

32.2 forces the 1,556 most frequent ones in first. 90% of the Han characters in the corpus are now one token each. It costs 1,837 merges, under 6% of the vocabulary.

On its own, that packs 25.7% more Chinese text into each token and 17.4% more Japanese, and no other language loses more than 1%.

32.1 · 30 tokens

研究就是在不知道自己在做什么的时候所做的事。

32.3 · 22 tokens

研究就是在不知道自己在做什么的时候所做的事。

Fig. 5 "Research is what you do when you don't know what you're doing," in Chinese. 32.1 split eight of them into two byte fragments each.

04Merges, spread across languages. 32.2

BPE hands its merges to whatever is most frequent, which here means English and code: 60% of the training bytes. So before training, 32.2 scales each language's pair counts by p ∝ bytes^0.7.

English counts are scaled by 0.68, code by 0.72 and Chinese by 1.59. Only the tokenizer sees this reweighting; the training mix stays the same.

English and code lose about 2.5% and every other language gains, yet total training tokens still fall by 0.7% (Fig. 6). At α = 0.5 the gains grow but the total nets to zero, because English and code make up most of the bytes, so 32.2 uses 0.7.

Fig. 6 Bytes per token, 32.2 vs plain-BPE 32.1. Higher means fewer tokens for the same text. "all" weighs groups by corpus bytes, so it tracks total training tokens.

Fig. 7 Pair-count weight per language group at α = 0.7. The dashed line is unweighted.

05Numbers split from the right. 32.3

GPT-4o's pretokenizer chops digit runs into threes from the left, so 1234567 becomes 123|456|7 and the groups mean nothing. 32.3 splits from the right: 1|234|567. Every group is ones, thousands, millions.

TokEval reports that place-value alignment tracks math accuracy. The change costs 0.1% compression overall.

One side effect is that 2026 now splits as 2|026 instead of 202|6, which is two tokens either way.

text	o200k, 32.1	32.3
1234567	1234567	1234567
$1,000,000	$1,000,000	$1,000,000
2000年	2000年	2000年
2026	2026	2026

Fig. 8 Digit runs under both pretokenizers. Commas already produced the right-aligned split; bare runs did not.

06Sometimes a token gets split on purpose.

At train time the loader can replace a token, with probability 0.1, by a random pair of tokens whose bytes add up to it. " quick" becomes " qui" + "ck".

The StochasTok authors report that models trained this way learn what's inside their tokens (spelling, counting, digits) and depend less on a single canonical segmentation. The shards stay canonical, and decoding doesn't change.

The vocabulary has 61,525 valid splits. At p = 0.1, sequences run about 7% longer, so an epoch costs about 7% more compute. The method is from StochasTok (Sims et al., ICLR 2026).

canonical · 13 tokens

The quick brown fox jumps over the lazy dog.

one StochasTok draw · 17 tokens

The quick brown fox jumps over the lazy dog.

Fig. 9 Real splitter output at p = 0.25 so you can see it; training uses 0.1. Same bytes, different tokens.

07Two implementations, byte-identical.

The tokenizer is Rust: a hand-written pretokenizer scanner and heap-based BPE. Python gets it through a HuggingFace export, which runs a different regex engine and a different BPE. On 2,167 documents (37.6M ids), the two agree byte for byte.

The scanner matches the reference regex on a 300k-case fuzz.
The trainer matches brute-force BPE on 20 corpora built to break it.
Retraining with both 32.2 fixes off reproduces all 32,480 merges of 32.1, in order.
300 random documents decode back to the exact corpus text.

One trap: BPE ties must merge the leftmost pair. A heap that pops the rightmost one tokenizes runs like three spaces differently.

pretokenize: ~3 min, full corpus
train: 5 to 7 min
encode: 4 to 5 min, 23.2B tokens

Fig. 10 Measured wall-clock time per stage on a single machine. A full retrain and re-encode takes 12 to 15 minutes, so testing a tokenizer idea is cheap.

08Versions.

32.332.2 plus right-aligned digitscurrent

32.2Han coverage, merges balanced at α = 0.7superseded

32.1plain byte-level BPEretired

The artifact refuses to load with any other pretokenizer pattern, so a 32.2 file can't sneak into a 32.3 run.