Character BPE tokenizer

This is a teaching model of character BPE. It learns frequent adjacent pairs of Unicode-code-point symbols. It is not a GPT tokenizer or a byte-level tokenizer, and its IDs have no meaning in an external language model.

Normalization. If selected during training, JavaScript's language-independent lowercase conversion runs first. Leading and trailing ECMAScript whitespace is removed and every remaining whitespace run becomes one ordinary space. No NFC or NFKC normalization is applied. The same trained lowercase setting is used when encoding. Every word, including the first, receives the word-start marker ▁; the marker does not distinguish the first word from later words. Literal ▁ in input and unpaired UTF-16 surrogates are rejected. NUL and other controls are allowed and displayed with escapes such as \u0000. Visible Unicode characters remain visible.

The raw and normalized corpus are each limited to 40,000 code points; text to encode is limited to 4,000. Lowercasing can increase code-point count, so both limits are checked. Nothing is silently truncated. Training requires nonempty normalized text. Empty encoding is valid and has zero tokens; its code-points-per-token ratio is undefined. Text code-point counts include normalized single spaces but exclude the added word-start markers. Markers do count as token symbols.

Training. The base alphabet is sorted by Unicode code-point order. In each round, every adjacent pair position contributes to frequency, including overlapping positions and repeated words. A pair needs frequency at least two. Ties compare the first token and then the second by code-point order, without locale-dependent or UTF-16 ordering. All occurrences of the chosen pair are replaced from left to right without overlap. Thus aaaa has three adjacent a+a positions but only two actual replacements. The merge table and selected point distinguish pair frequency from replacements and show the corpus token count after the merge.

IDs are zero-based and belong to unique token strings. A new merge appends a vocabulary entry only if its resulting string is new; otherwise it reuses the existing ID. The displayed full vocabulary includes every learned merge, even when only a prefix is active. The active prefix ranges from zero to all learned merges. The graph shows weighted corpus token counts at every prefix, not bytes or a compression ratio.

Encoding and IDs. Encode explicitly applies the selected trained ranks; the lowest available rank wins, with the leftmost pair breaking a rank tie. A prefix change updates an already encoded, unchanged text without retraining. Unknown code points remain visible as tokens with no ID. If any are unknown, copying IDs is disabled; there is no −1 ID or byte fallback. Known IDs decode to the normalized text, not the original whitespace or capitalization. Token pages show at most 100 items with IDs visible without hovering.

Corpus, merge-limit or lowercase edits discard the old model. Training runs in small portions and can be cancelled; edits, reset, hidden pages and navigation invalidate a pending job. Reset keeps the inputs. The current model, encoding and inputs remain in memory during page navigation in this session. Only valid inputs that fit the site's saved-state limit can survive a full reload. Oversized or invalid saved input is cleared rather than leaving an older saved corpus in place.

Conceptual sources: Sennrich, Haddow and Birch (2016) for BPE segmentation, and SentencePiece normalization documentation. This tool's normalization, explicit word prefixes, minimum frequency and tie order are its own stated conventions; it does not reproduce all behaviours of either implementation.