Let's build the GPT Tokenizer

Andrej Karpathy builds the GPT series tokenizer from scratch, exposing how byte-pair encoding creates persistent LLM limitations.

Core Concept

• Tokenization converts strings to integer tokens via a separate trained stage before the LLM (0:00-12:02).
• Byte Pair Encoding (BPE) starts with 256 UTF-8 bytes and iteratively merges the most frequent pairs (24:08-36:09).

Implementation Steps

• Count consecutive pairs, record merges, then apply them in order during encode (36:10-48:13).
• Regex patterns force splits between letters, numbers, and punctuation to prevent bad merges (1:00:17-1:12:20).

GPT-2 vs GPT-4

• GPT-4 adds case-insensitive apostrophe handling and caps number merges at three digits (1:12:22-1:24:24).
• Special tokens like EOT require embedding surgery and careful handling (1:24:27-1:36:28).

SentencePiece Alternative

• Operates on Unicode code points with byte fallback for rare characters (1:36:31-1:48:34).

Resulting LLM Quirks

• Arbitrary splits hurt spelling, arithmetic, Python indentation, and non-English text (1:48:34-2:00:35).
• Partial or untrained tokens produce unstable, out-of-distribution behavior (2:00:37-2:13:34).

00:00:00 Major Topic — Introduction to Tokenization in LLMs

Speaker: Karpathy

• Tokenization is the bridge between human text and model primitives: it translates strings into tokens and back again. It is a distinct stage in the LLM pipeline with its own training data, its own training algorithm (notably Byte Pair Encoding, BPE), and two core functions: encode() (strings -> tokens) and decode() (tokens -> strings).
• Tokenization is often overlooked, but many “quirks” and failures of LLMs trace back to tokenization. The video sets out to build the GPT-era tokenizer from scratch to illustrate these effects.
• The broader framing: in practice, modern LLMs do not operate on characters but on token chunks, and tokenization can dramatically affect context length, efficiency, and even the model’s apparent ability to spell, do math, or handle non-English text.

"The Tokenizer is a necessary and pervasive component of Large Language Models (LLMs), where it translates between strings and tokens." (Karpathy, 00:00:00)

• Visual cue from the lecture’s framing: tokenizers are a pre-processing stage with their own data and objective, separate from the Transformer itself. The speaker emphasizes that improvements or changes in tokenization can yield large downstream gains (or pains).

00:05:50 Major Topic — Tokenization by Example (Web UI) and Intuition

• Demonstration: live tokenization in a web UI (tiktokenizer) showing how a string is broken into tokens with GPT-2’s tokenizer.
• Key observation: a single word like "tokenization" may be represented by multiple tokens; spaces and even capitalization influence token boundaries. Tokens are not just metaphoric “words”; they are learned chunks that can combine letters, punctuation, spaces, and numbers in non-obvious ways.
• Practical insight: some tokens encode multiple characters; some tokens encode spaces; tokenization is highly language- and context-dependent, and different tokenizers (GPT-2 vs GPT-4) handle spaces and punctuation differently.

"this string that I pasted here is currently tokenizing into 300 tokens" (Karpathy, 00:05:50)

• The demo underscores a core theme: the representation is highly dependent on the tokenizer’s design. The same textual surface form can map to very different token ID sequences depending on the tokenizer.
• The example also introduces the notion that numbers, Python code, and other structured text can produce very long or very short token sequences depending on how spaces and digits are tokenized.

00:14:56 Major Topic — Strings in Python, Unicode Code Points

• Python strings are sequences of Unicode code points. A code point is a unique number assigned to a character in the Unicode standard (e.g., 'H' = 104). Unicode defines roughly 150,000 characters across many scripts.
• The ord() function in Python maps a single character to its code point integer. However, using code points directly as vocabulary would be enormous and unstable because Unicode is dynamic and large, so we don’t tokenize on raw code points.
• Rationale for encoding: to compress and stabilize the vocabulary, we use encodings. UTF-8 is a variable-length encoding (1–4 bytes per code point). UTF-16 and UTF-32 exist too, with their own trade-offs. UTF-8 is preferred on the Internet for backwards-compatibility and efficiency in most cases.
• The lecture emphasizes UTF-8 as the practical encoding for text, but raw UTF-8 bytes are still too granular (only 256 possible byte values) for efficient modeling with limited context length.

"Unicode code points are defined by the Unicode Consortium... roughly 150,000 characters..." (Karpathy, 00:14:56)

• The takeaway: you cannot effectively feed raw Unicode code points or raw UTF-8 bytes directly into a transformer with a reasonable context length; you need a learned tokenization approach (e.g., BPE) to compress the input into a manageable vocabulary.

00:18:15 Major Topic — Unicode Encodings and Trade-offs (UTF-8, UTF-16, UTF-32)

• UTF-8 is dominant because it is compact for common characters and backwards compatible with ASCII; it uses a variable number of bytes per code point.
• UTF-16, UTF-32 can be less space-efficient in many cases and have their own corner-case complexities.
• The video mentions broader references (e.g., the UTF-8 Everywhere Manifesto) to justify UTF-8 as the practical encoding in real-world systems.
• The “naive” approach of using raw bytes (UTF-8) as tokens would create a vocabulary of only 256 tokens, which is too small and would waste attention budget on long sequences; hence the need for a learned BPE-style compression.

“we would like to stick with utf8 encoding of these strings but not use raw bytes naively” (Karpathy, 00:18:15)

00:22:47 Major Topic — Daydreaming: Deleting Tokenization

• The speaker fantasizes about a world without tokenization, acknowledging it would simplify many issues—but then grounds the discussion in the reality that tokenization exists as a separate stage with architecture-specific trade-offs.
• This section primes the audience for why tokenization matters beyond cosmetic text processing: it influences context length, arithmetic ability, multilingual performance, and even how the model handles code.

00:23:50 Major Topic — Byte Pair Encoding (BPE) Algorithm Walkthrough

• Core idea: start with a very small vocabulary (e.g., 256 bytes for UTF-8), then iteratively find the most frequent adjacent pair of tokens in a corpus and merge them into a new token. Repeat to grow the vocabulary and compress the input sequence.
• Example walk-through (simplified): start with {A,B,C,D} (vocabulary size 4). If AA is most common, create a new token Z representing AA; replace occurrences of AA with Z; now the sequence length shrinks and the vocabulary grows (to 5). Reiterate merging the most frequent pair (e.g., AB -> Y), then (ZY -> X), etc.
• The result is a compressed sequence of tokens with a larger vocabulary than the initial 256, enabling shorter input sequences for the same text.
• The key takeaway: BPE yields a hierarchical, data-driven vocabulary that captures common subword units and words, balancing between byte-level granularity and whole-word tokens.

"Byte Pair Encoding (BPE) algorithm walkthrough" (Karpathy, 00:23:50)

00:27:02 Major Topic — Starting the Implementation

• The speaker moves from theory to code, using a text sample (a block post paragraph) to illustrate the encoding process: first encode to UTF-8 bytes, then convert to a list of integers representing each byte, and then start applying BPE merges.
• The goal is to implement the two core operations: encode (text -> token IDs) and decode (token IDs -> text).

00:28:35 Major Topic — Counting Consecutive Pairs and Finding the Most Common Pair

• The first concrete step: write a helper get_stats(tokens) that counts all consecutive pair occurrences in the token list. The dictionary maps a pair (e.g., (101, 32)) to its frequency.
• The pair with the highest frequency is a candidate for merging in the next step. The transcript shows how to print and inspect the most frequent pair, confirming e.g., (101, 32) corresponds to 'e' followed by a space in ASCII/decoded form.
• The code path demonstrates how to validate that the most common pair aligns with intuitive observations about common sequences in English text (like 'e ').

00:30:36 Major Topic — Merging the Most Common Pair

• The next step is to implement a function that takes a token list and merges all occurrences of the selected pair into a new token (e.g., 256). Then, you re-run the counting to find the next most common pair and merge it, continuing for a fixed number of steps to create the final vocabulary.
• The demonstration shows a successful first merge: (101, 32) -> 256; the list length decreases, indicating compression.
• A crucial note: the newly minted token (256) remains eligible for further merging, enabling hierarchical growth (a binary forest of merges) rather than a single tree.

00:34:58 Major Topic — Training the Tokenizer: Compression Ratio

• The video adds a loop to perform multiple merges (hyperparameter) to reach a target vocabulary size (e.g., 276 total tokens in the example, meaning 20 additional merges beyond the initial 256 bytes).
• The compression ratio is computed as original_length / merged_length (e.g., 616 bytes -> 596 tokens after 20 merges, ratio ~1.27). More merges yield higher compression but come with diminishing returns and larger embedding/softmax costs.
• The tokenizer is a separate construct from the LLM: the tokenizer has its own training corpus and training stage, and once trained, it provides encode/decode pathways to and from raw text.

⭐ HIGH PRIORITY — The tokenizer is a completely separate object from the LLM; training data, vocabulary, and merges are distinct from the language model itself.

00:39:20 Major Topic — Tokenizer/LLM Diagram and Practical Implications

• The tokenizer is a separate pre-processing stage, trained on its own corpus. The training data for the tokenizer can be language-specific and content-specific (e.g., code vs. natural language).
• The embedding table (for tokens) and the final LM head depend on the tokenizer’s vocabulary size. A larger vocabulary increases memory and computation in both embedding and the final projection layer, but reduces sequence length for the same text.
• There is a Goldilocks zone for vocabulary size: too small yields long sequences; too large yields sparse token usage and undertrained tokens.

00:42:47 Major Topic — Decoding Tokens to Strings (Decode)

• The process of decoding: build a vocab map from token IDs to the corresponding bytes (by concatenating the bytes of the leaves in the merge tree). Then, given a token sequence, map each token to its bytes and decode the concatenated bytes using UTF-8 to obtain the string.
• A subtle pitfall: not every byte sequence is valid UTF-8; invalid sequences cause decoding errors. In Python, you can use errors='replace' to gracefully handle invalid sequences, at the cost of possible garbled text. This mirrors real-world tokenizer outputs where some generated tokens may map to invalid encodings if the model produces out-of-distribution sequences.

> "try to implement this function yourself" (Karpathy, 00:47:xx area where decoding is implemented)

00:48:21 Major Topic — Encoding Strings to Tokens (Encode)

• Encoding is the reverse of decoding. Start with UTF-8 bytes, map them to the initial 256 byte tokens, and then repeatedly merge adjacent pairs according to the learned merges dictionary to produce a final token sequence.
• Important design detail: the merges are applied in a specific order, respecting the hierarchy of merges (top-down in the merge list) so that previously-created merges can participate in later merges (e.g., if 256 was used in an earlier merge, it can be involved in subsequent pairs).
• The encoding function must handle edge cases, including the possibility that a pair to be merged does not exist in the merges dictionary (signal with a special value to break early).
• A practical caution about corner cases: if the token sequence is too short (less than two), there may be nothing to merge; the encoder should handle these gracefully.

00:57:36 Major Topic — Regex Patterns to Force Splits Across Categories

• The speaker discusses how OpenAI’s GPT-2 tokenizer uses a set of regular expressions to split text into chunks by category (letters, numbers, punctuation, spaces, etc.). This chunking can force certain characters to stay in separate tokens and prevents cross-boundary merges (e.g., letters with following digits or punctuation).
• The regex pattern enforces language-typical boundaries (letters, numbers, punctuation) and introduces special handling for apostrophes and whitespace to avoid excessive cross-boundary merges. The approach reduces spurious merging across category boundaries, at the cost of more tokens in some cases.
• Caveats: tokenization is language- and character-set dependent; the exact pattern is not universal and can yield surprising results for non-English languages or unconventional punctuation (e.g., Unicode apostrophes).

01:11:38 Major Topic — tiktoken and GPT-2 vs GPT-4 Regex Variants

• The tiktoken library is OpenAI’s official tokenizer library for inference. It provides the GPT-2 and GPT-4 (and other) tokenizers with their respective tokenization rules, including regex patterns and special-case handling for whitespace and tokens.
• GPT-2 vs GPT-4 differences: GPT-4’s tokenizer uses different regex rules and tends to group whitespace more aggressively in Python and code, producing denser code tokens than GPT-2. The GPT-4 vocabulary is around 100K tokens, roughly double GPT-2’s vocabulary (~50K), achieving a denser encoding of the same text and allowing more context per token.
• The GPT-4 tokenizer changes in whitespace handling (e.g., grouping multiple spaces into a single token) reflect deliberate design choices to densify code and improve context efficiency.

⭐ HIGH PRIORITY — The tokenizer design (and its regex) is a major lever in how effectively code and natural language can be represented and attended to by the LM.

01:14:59 Major Topic — GPT-2 Encoder.py and OpenAI’s Released Material

• OpenAI released an encoder.py (GPT-2) and the associated vocab/bpe merges as a compact representation of their trained tokenizer. The provided encoder maps IDs to bytes and vice versa using the two saved artifacts: encoder.json (vocabulary mapping) and vocab.bpe (merges).
• The released code is inference-focused; training the tokenizer from scratch is not exposed in the release, so the video uses MinBPE-like steps to illustrate training and demonstrates how to reproduce similar behavior.
• The GPT-2 vs GPT-4 tokenizers differ in their regex rules and vocab size. OpenAI’s TikToken library exposes the difference in how whitespaces and numbers get tokenized across model generations.

01:18:26 Major Topic — Special Tokens, GPT-2 vs GPT-4 Differences

• Special tokens are extra IDs reserved for particular semantics (e.g., end-of-text, start-of-sequence, padding, or tokens used to mark prompts, messages, or system instructions).
• In GPT-2, there is a single special token (end-of-text) at the tail end of the vocabulary. In GPT-4, there are multiple special tokens (start/middle/end delimiters for various contexts, and additional markers for structured prompts).
• Special tokens are not just “extra vocabulary”; they often require model surgery (expanding the embedding matrix and the final projection layer) when added to a pretrained model.

01:25:28 Major Topic — MinBPE Exercise Time! Write Your Own GPT-4 Tokenizer

• The speaker introduces a hands-on exercise: implement a GPT-4-like tokenizer using the same core BPE machinery but with the GPT-4 regex and vocabulary rules. The goal is to reproduce GPT-4-style tokenization behavior and compare encoding/decoding results.
• The exercise is framed with a GitHub repo (minBPE) and a set of documents that guide the reader through step-by-step construction, including training, encoding, decoding, and testing against reference outputs.

01:28:42 Major Topic — SentencePiece Library and its Use in LLaMA-2

• SentencePiece is a different tokenizer toolkit that can train and encode using BPE (or other algorithms) with several design choices. It’s widely used in models like LLaMA-2 and various other families.
• SentencePiece operates at the code-point level and can optionally fallback to UTF-8 bytes for rare code points (byte fallback). It supports many hyperparameters and configurations, making it more flexible but more complex to tune than tiktoken.
• The speaker demonstrates training SentencePiece on a toy dataset and inspecting the resulting vocabulary and tokens, including how special tokens and byte fallback influence encoding of non-native characters.

01:43:27 Major Topic — How to Set Vocabulary Size and Revisit GPT Architecture

• Vocabulary size is a critical hyperparameter. The embedding matrix and the LM head scale with vocabulary size, which impacts memory, compute, and training stability.
• A larger vocabulary can shorten input sequences but increases parameter count and can reduce the per-token learning signal (rarer tokens). A smaller vocabulary reduces parameters but lengthens sequences, stressing the attention budget.
• The talk revisits a Transformer architecture built in a prior video, focusing on how BPE-vocab size interacts with embedding size, attention head count, and overall training stability.

01:48:11 Major Topic — Training New Tokens, Prompt Compression, and Multimodal Tokenization

• Training new tokens (e.g., for special tools or prompts) is possible via parameter-efficient techniques that add new embeddings and possibly small adapter-like layers, while freezing most of the base model.
• A concept called prompt compression (gist tokens) is discussed: introducing a small number of new tokens that condense a long prompt into a dense, learnable representation, enabling the model to reproduce the behavior of a long prompt with far fewer tokens.
• The video also touches multimodal tokenization: for images, videos, and audio, tokenization can be achieved through vector quantization or patch-based tokenization so that a Transformer can attend to cross-modal tokens alongside text tokens.

02:10:20 Major Topic — Final Recommendations and Reflections

• Tokenization is a separate stage with its own training data and objectives. It’s not just a preprocessing nicety; it can determine how much context you effectively have, how well you can represent non-English and code, and how efficiently you can train and run models.
• Reuse GPT-4’s tokenization vocabulary when possible for best compatibility and density; if training a tokenizer from scratch, the GPT-4-like approach (dense tokenization, proper whitespace handling) tends to be advantageous for Python and code.
• OpenAI’s tiktoken library is a robust inference-time tokenizer; for training from scratch, MinBPE (or a precise GPT-4-like BPE) plus SentencePiece as an alternative can be explored, keeping in mind the different design trade-offs.

02:12:50 Minor/Wrap-up Notes

• The video closes on a light note with a playful puzzle and an invitation to explore the video’s appendix resources and code.
• The overarching takeaway: tokenization is a deep and nuanced design space with concrete practical consequences for model performance, capabilities, and safety.

Notable Visuals and How to Recreate Them

• Web UI tokenization demo: a side-by-side view showing input text on the left and a color-coded token sequence on the right, with tokens labeled and spaces highlighted. Recreate by rendering each token as a colored block with a small label and color-coding per token ID.
• Diagram: Tokenizer vs. LLM pipeline as separate boxes with arrows: Text -> Tokenizer (encode) -> Tokens -> Transformer -> Tokens -> Decoder/LM Head -> Text (decode). Emphasize the data flow and the separation of concerns.
• BPE merging steps: a small schematic showing bytes or initial tokens as leaves, and merges producing a forest of binary merges culminating in a final vocabulary of merged tokens. Annotate with examples (e.g., AA -> Z, AB -> Y, etc.).
• Regex pattern anatomy: a schematic showing how GPT-2’s pattern splits input into categories (letters, numbers, punctuation, unknowns) and how this prevents cross-boundary merges.

Resources Mentioned

• OpenAI: GPT-2 paper (input representation and tokenization rationale)
• OpenAI: GPT-2 encoder.py and vocab.bpe (tokenizer artifacts)
• minBPE: Karpathy’s repository with a practical exercise for building a GPT-4-like tokenizer
• minBPE blog/exercise MD: exercise progression and guidance
• Google Colab: Colab notebook for the video
• tiktoken: OpenAI’s official tokenizer library (inference) and its GPT-2 vs GPT-4 regex patterns
• tiktokenizer (web app): live in-browser tokenization demo
• SentencePiece: Google’s tokenizer toolkit (training and inference)
• GPT-4 vs GPT-2: Regex differences and vocabulary sizes
• Regex tooling: regX patterns and extended patterns (re and re_ex) used to split text into units
• SORA: a paper referenced in the context of multimodal tokenization approaches
• UTF-8 Everywhere Manifesto: rationale for UTF-8 adoption across the web
• UTF-8, UTF-16, UTF-32: encoding trade-offs and their impact on tokenization
• End-of-text token: a special token often used to delimit documents during training
• Unk token and unknown code points: behavior when rare code points are encountered
• Gist tokens (prompt compression): a method to compress long prompts into a few trainable tokens
• Multimodal tokenization concepts: vector quantization for images/videos and soft tokens for more flexible representations
• Solid Gold Magikarp (tokenization quirk): a case study about tokenization-induced weird model behavior

Notable Lines (Verbatim, with Timestamps)

"The Tokenizer is a necessary and pervasive component of Large Language Models (LLMs), where it translates between strings and tokens." (Karpathy, 00:00:00)

"this string that I pasted here is currently tokenizing into 300 tokens" (Karpathy, 00:05:50)

"Byte Pair Encoding (BPE) algorithm walkthrough" (Karpathy, 00:23:50)

"Tokenizer/LLM diagram: it is a completely separate stage" (Karpathy, 00:39:20)

"minbpe exercise time! write your own GPT-4 tokenizer" (Karpathy, 01:25:28)

"special tokens, tiktoken handling of, GPT-2/GPT-4 differences" (Karpathy, 01:18:26)

"final recommendations" (Karpathy, 02:10:20)

"solid gold Magikarp" (Karpathy, 02:12:50)

Topics (Tags)

• Tokenization
• Byte-Pair Encoding
• GPT-2 and GPT-4 Tokenizers
• tiktoken / minBPE / SentencePiece
• Unicode and UTF encodings
• Special Tokens and End-of-Text
• Regex-based tokenization patterns
• Unknown tokens and unstable tokens
• Prompt compression and gist tokens
• Multimodal tokenization
• Model architecture implications of vocabulary size

Summary of the Deep-Read Analysis

• The video offers a hands-on tour from the textual surface to token sequences through BPE-driven vocabulary growth, illustrating how tokenization shapes what the Transformer sees, how long a sequence is, and how efficiently the model can learn.
• It emphasizes tokenizer as a standalone pre-processing stage with its own training corpus, the importance of vocabulary size, and the practical differences between GPT-2 and GPT-4 tokenizers (especially around whitespace handling and numeric tokenization).
• It also discusses practical tools (tiktoken, minBPE, SentencePiece) and the trade-offs among them, including special tokens and the need for model surgery when adding tokens to a pretrained model.
• Finally, it grounds the discussion in real-world quirks (e.g., language-specific tokenization, python code spacing, and the Mischief of “solid gold Magikarp”) to illustrate why tokenization remains a critical and nuanced challenge in scaling LLMs.