Building GPT from Scratch

Reference Lecture by Andrej Karpathy: YouTube Link

1. Introduction & Core Philosophy

The Core Premise: At its fundamental level, a Large Language Model (LLM) is an auto-complete system. Given a sequence of input text (context), it predicts the probability distribution of the next token.
The Transformer Backbone: Modern LLMs rely entirely on the Transformer architecture introduced in “Attention Is All You Need”, which uses self-attention mechanisms to dynamically weigh relationships between tokens across sequences.
Scope of Implementation: While massive models like ChatGPT incorporate petabytes of web data, multi-stage Supervised Fine-Tuning (SFT), and Reinforcement Learning from Human Feedback (RLHF), building a foundational GPT from scratch can be demonstrated locally on a tiny dataset (e.g., Shakespeare’s works, $\approx 1\text{MB}$) to understand every tensor operation under the hood.

2. Reading, Exploring, and Preparing Data

Before modeling, raw text must be ingested, inspected, and split into training and validation subsets to prevent memorization.

2.1. Ingesting Text Data

Using Python to read raw text:

with open("input.txt", "r", encoding="utf-8") as f:
    text = f.read()

# Extract all unique characters to form the vocabulary
chars = sorted(list(set(text)))
vocab_size = len(chars)
print(f"Vocab size: {vocab_size}")

2.2. Character-Level vs. Sub-Word Tokenization

Character-Level Tokenization: Used in educational builds (like nanoGPT on Shakespeare) where the vocabulary consists of individual characters (letters, punctuation, numbers).
Sub-Word Tokenization (BPE / SentencePiece): Modern production models (GPT-2/3/4, Llama) use Byte Pair Encoding (BPE) via libraries like tiktoken to group frequent character n-grams into single tokens, shrinking sequence lengths and improving efficiency.

# Character mapping dictionaries
stoi = {ch: i for i, ch in enumerate(chars)}
itos = {i: ch for i, ch in enumerate(chars)}

encode = lambda s: [stoi[c] for c in s]  # String to integer list
decode = lambda l: "".join([itos[i] for i in l])  # Integer list to string

2.3. Train / Validation Split

To verify that the model generalizes rather than merely memorizing the training corpus, the dataset is tokenized into a PyTorch tensor and split (typically 90% train, 10% validation):

import torch

data = torch.tensor(encode(text), dtype=torch.long)
n = int(0.9 * len(data))
train_data = data[:n]
val_data = data[n:]

2.4. Batching and Data Loaders

During training, the model doesn’t process the entire text at once. Instead, it samples random chunks of length block_size (context window) across batches (batch_size):

torch.manual_seed(1337)
batch_size = 4
block_size = 8  # maximum context length for predictions


def get_batch(split):
  data = train_data if split == "train" else val_data
  # Generate random starting indices for batch chunks
  ix = torch.randint(len(data) - block_size, (batch_size,))
  x = torch.stack([data[i : i + block_size] for i in ix])
  y = torch.stack([data[i + 1 : i + block_size + 1] for i in ix])
  return x, y

3. The Baseline: Bigram Language Model

As a starting point, Karpathy implements a simple Bigram Language Model where each token prediction depends only on the immediately preceding single token.

3.1. PyTorch Module Structure

import torch.nn as nn
from torch.nn import functional as F


class BigramLanguageModel(nn.Module):

  def __init__(self, vocab_size):
    super().__init__()
    # Each token reads off the logits for the next token from a lookup table
    self.token_embedding_table = nn.Embedding(vocab_size, vocab_size)

  def forward(self, idx, targets=None):
    # idx and targets are (B, T) tensors of integers
    logits = self.token_embedding_table(idx)  # (B, T, C) -> Batch, Time, Channels

    if targets is None:
      loss = None
    else:
      B, T, C = logits.shape
      logits_flat = logits.view(B * T, C)
      targets_flat = targets.view(B * T)
      loss = F.cross_entropy(logits_flat, targets_flat)

    return logits, loss

  def generate(self, idx, max_new_tokens):
    for _ in range(max_new_tokens):
      logits, loss = self(idx)
      # Focus only on the last time step
      logits = logits[:, -1, :]  # becomes (B, C)
      probs = F.softmax(logits, dim=-1)  # (B, C)
      idx_next = torch.multinomial(probs, num_samples=1)  # (B, 1)
      idx = torch.cat((idx, idx_next), dim=1)  # (B, T+1)
    return idx

4. The Crux: Building Self-Attention Step-by-Step

The central conceptual breakthrough of the lecture is demonstrating how context aggregation evolves from naive averaging to optimized matrix multiplication.

4.1. Version 1: Naive For-Loops (Averaging Past Context)

To compute the representation of a token as an average of all preceding tokens in the context window:

# Toy example: B=4 batches, T=8 time steps, C=2 channels
torch.manual_seed(1337)
B, T, C = 4, 8, 2
x = torch.randn(B, T, C)

xbow = torch.zeros((B, T, C))
for b in range(B):
  for t in range(T):
    xprev = x[b, : t + 1]  # (t, C)
    xbow[b, t] = torch.mean(xprev, 0)

Bottleneck: Explicit Python loops are extremely slow and do not scale.

4.2. Version 2: Matrix Multiplication Trick (Lower Triangular Weights)

Averaging can be expressed efficiently as matrix multiplication using a lower-triangular weight matrix:

wei = torch.tril(torch.ones(T, T))
wei = wei / wei.sum(1, keepdim=True)
xbow2 = wei @ x  # (T, T) @ (B, T, C) -> (B, T, C)

4.3. Version 3: Adding Softmax (Data-Dependent Affinity)

Instead of uniform averaging weights, tokens should dynamically calculate how much “attention” they want to pay to past tokens based on their content:

torch.manual_seed(42)
a = torch.tril(torch.ones(3, 3))
wei = torch.zeros(3, 3)
wei = wei.masked_fill(a == 0, float("-inf"))
wei = F.softmax(wei, dim=-1)

4.4. Version 4: Scaled Self-Attention (Queries, Keys, Values)

Self-attention introduces three core linear projections for every token:

Query ($Q$): What am I looking for?
Key ($K$): What do I contain?
Value ($V$): What information will I pass along if attended to?

class Head(nn.Module):
  """One head of self-attention"""

  def __init__(self, head_size, n_embd, block_size):
    super().__init__()
    self.key = nn.Linear(n_embd, head_size, bias=False)
    self.query = nn.Linear(n_embd, head_size, bias=False)
    self.value = nn.Linear(n_embd, head_size, bias=False)
    self.register_buffer("tril", torch.tril(torch.ones(block_size, block_size)))
    self.dropout = nn.Dropout(0.2)

  def forward(self, x):
    B, T, C = x.shape
    k = self.key(x)  # (B, T, hs)
    q = self.query(x)  # (B, T, hs)

    # Compute attention scores ("affinities")
    wei = (
        q @ k.transpose(-2, -1) * (k.shape[-1] ** -0.5)
    )  # (B, T, hs) @ (B, hs, T) -> (B, T, T)
    wei = wei.masked_fill(self.tril[:T, :T] == 0, float("-inf"))  # Causal mask
    wei = F.softmax(wei, dim=-1)
    wei = self.dropout(wei)

    # Perform the weighted aggregation of the values
    v = self.value(x)  # (B, T, hs)
    out = wei @ v  # (B, T, T) @ (B, T, hs) -> (B, T, hs)
    return out

Why Scale? ($\frac{1}{\sqrt{d_k}}$): Dividing by the square root of the head size prevents variance from growing too large before Softmax, which would otherwise push gradients close to zero.

5. Multi-Head Attention & The Complete Transformer Block

To let the model jointly attend to information from different representation subspaces simultaneously, multiple attention heads are concatenated:

class MultiHeadAttention(nn.Module):
  """Multiple heads of self-attention in parallel"""

  def __init__(self, num_heads, head_size, n_embd, block_size):
    super().__init__()
    self.heads = nn.ModuleList(
        [Head(head_size, n_embd, block_size) for _ in range(num_heads)]
    )
    self.proj = nn.Linear(n_embd, n_embd)
    self.dropout = nn.Dropout(0.2)

  def forward(self, x):
    out = torch.cat([h(x) for h in self.heads], dim=-1)
    out = self.dropout(self.proj(out))
    return out

5.1. Feed-Forward Networks & Residual Connections

Each transformer block pairs attention with a position-wise Feed-Forward Network, wrapped in Residual Connections (skip connections) and Layer Normalization:

class Block(nn.Module):
  """Transformer block: communication followed by computation"""

  def __init__(self, n_embd, n_head, block_size):
    super().__init__()
    head_size = n_embd // n_head
    self.sa = MultiHeadAttention(n_head, head_size, n_embd, block_size)
    self.ffwd = FeedForward(n_embd)
    self.ln1 = nn.LayerNorm(n_embd)
    self.ln2 = nn.LayerNorm(n_embd)

  def forward(self, x):
    x = x + self.sa(self.ln1(x))  # Residual connection around attention
    x = x + self.ffwd(self.ln2(x))  # Residual connection around feed-forward
    return x

6. Summary of Key Architectural Takeaways

Autoregressive Masking: Ensures future tokens cannot communicate with past tokens during training.
Positional Embeddings: Combined with token embeddings (nn.Embedding(block_size, n_embd)) to give the model awareness of token order.
Scaling Law Foundation: Transitioning from character-level Bigram baselines to multi-head stacked transformer blocks mirrors the exact scaling architecture utilized in state-of-the-art LLMs (GPT-2, GPT-3, and beyond).