Explorador de tokenizador BPE

Byte-pair encoding is how most language models turn text into a vocabulary of a few tens of thousands of tokens. Start from characters, repeatedly merge the most frequent adjacent pair, and stop when the vocabulary is large enough. Common words become single tokens; rare ones stay as short pieces.

How to use it. Pick a corpus preset (or paste your own), choose how many merges to learn, then press Train tokenizer. Nothing is trained until that click. The encode box then shows how the learned merges cut a test sentence into coloured chips; hover a chip for its vocabulary id. Copy token IDs as a JSON array when you want them elsewhere.

What the numbers mean. The vocabulary is base characters plus one entry per merge. Corpus tokens is the total length of the training set after each merge — it falls as the tokenizer compresses the corpus. Characters per token on the test string is the practical compression ratio for new text. Single-character tokens remaining after encoding are pieces the merges never absorbed.

The ▁ marker. Every word is prefixed with a leading-space character (▁), the same convention SentencePiece uses. That keeps “the” at the start of a sentence distinct from “ the” after a space, which is why modern tokenizers treat spaces as part of the token rather than as separators.

Try this. On the English preset, train with 0 merges and encode “learned learner”: every character is its own token. Raise merges to 80 and the shared stem “learn” collapses into one piece. Switch to the DNA corpus with lowercase off: long ACGT repeats become multi-base tokens quickly. On the code corpus, identifiers like tokenize and keywords like function absorb into single tokens while punctuation stays atomic.

Limitations. This is character BPE, not byte-level BPE with a 256-byte alphabet and fallback for unknown glyphs. There is no special-token vocabulary, no Unicode normalization beyond optional lowercasing, and training is capped at 40 000 characters / 400 merges so the UI stays responsive. Real GPT-style tokenizers use the same idea at larger scale with carefully chosen pre-tokenizers.