Diving Deep into Large Language Models (LLMs)

Reference lecture by Andrej Karpathy: YouTube

Part 1: Pretraining & Neural Network Architecture

Introduction

Large Language Models (LLMs) such as ChatGPT present a deceptively simple interface: a single text box input yielding generated responses. Beneath this interface lies an extensive engineering and mathematical pipeline designed to build intuitive mental models of language processing.

Understanding LLMs requires dissecting how these models are built from scratch, examining every sequential pipeline stage, and analyzing the cognitive and psychological implications of their architectural constraints.

Pretraining Data: Scraping & Curation

The first phase of creating an LLM is Pretraining, which centers on downloading, filtering, and curating vast amounts of raw text from the internet.

+------------------+     +------------------+     +-------------------+     +------------------+
|   Common Crawl   | --> |  URL Filtering   | --> |  Text Extraction  | --> | Language Filter  |
|  (2.7B Webpages) |     | (Malware/Block)  |     |  (Strip HTML/CSS) |     |  (e.g., >65% EN) |
+------------------+     +------------------+     +-------------------+     +------------------+
                                                                                  |
                                                                                  v
+------------------+     +------------------+                             +------------------+
| FineWeb Dataset  | <-- |   PII Removal    | <-------------------------- |  Deduplication   |
|   (44 TB Text)   |     |  (SSN/Addresses) |                             +------------------+
+------------------+     +------------------+

Key Data Pipeline Stages

  1. Raw Web Scraping: The primary foundation for pretraining datasets is Common Crawl, an organization that has indexed over 2.7 billion webpages since 2007 by traversing hyperlinked internet networks starting from seed URLs.
  2. URL Filtering: Eliminates domains present on blocklists, including spam, malware, adult content, marketing networks, and hate speech platforms.
  3. Text Extraction: Converts raw HTML documents into plaintext by stripping markup tags, CSS scripts, and navigation UI metadata.
  4. Language Filtering: Applies language classifiers to keep high-quality content matching target distributions (e.g., HuggingFace’s FineWeb dataset filters out pages containing less than 65% English text).
  5. Deduplication & Quality Heuristics: Removes duplicated web content, boilerplate text, and low-quality machine-generated pages.
  6. Personally Identifiable Information (PII) Removal: Detects and redacts sensitive individual data, such as Social Security numbers, telephone numbers, and street addresses.

Despite the internet’s vast scope, aggressive curation reduces raw web data down to remarkably compact sizes. For example, the FineWeb dataset compresses down to approximately 44 terabytes of disk space—small enough to fit onto a single modern hard drive setup.

Tokenization

Neural networks cannot natively process raw string characters; they operate on one-dimensional sequences of discrete numeric tokens chosen from a finite vocabulary set.

Raw Text: "Hello world"
      │
      ▼ UTF-8 Encoding
Binary/Bytes: [0x48, 0x65, 0x6c, 0x6c, 0x6f, 0x20, 0x77, 0x6f, 0x72, 0x6c, 0x64] (256 Symbol Vocab)
      │
      ▼ Byte Pair Encoding (BPE)
Token IDs: [15339, 1917] (Vocabulary Size: 100,277)

The Trade-off: Vocabulary Size vs. Sequence Length

  • Raw Bit/Byte Representation: Representing text directly as UTF-8 bytes yields a tiny vocabulary size ($256$ possible symbols), but inflates sequence length, consuming context window capacity.
  • Byte Pair Encoding (BPE): BPE recursively merges common consecutive bytes or token pairs into new unique token IDs. This compresses sequence length while expanding vocabulary size.
  • Production Tokenizers: GPT-4 utilizes the cl100k_base tokenizer with a vocabulary size of 100,277 symbols.

Tokenizer Characteristics

Input String Token Count Token IDs Notes
"Hello world" 2 [15339, 1917] Includes space prefix in second token
"Helloworld" 2 [73, 20387] Distinct IDs due to missing space
" Hello world" 3 [220, 15339, 1917] Extra leading space creates explicit token 220

Neural Network Input/Output

During pretraining, neural networks model the statistical relationships of tokens within context sequences.

  1. Context Sampling: Random windows of token sequences are sampled from datasets (e.g., FineWeb’s 15-trillion token sequence). Context lengths range up to model-defined context limits ($N$).
  2. Probability Distribution Output: Given a context window $[t_1, t_2, \dots, t_k]$, the network outputs a probability vector over the entire vocabulary $V$ (e.g., 100,277 values) estimating $P(t_{k+1} \mid t_1, \dots, t_k)$.
  3. Loss Function & Gradient Updates: Predictions are compared against the ground-truth target token using Cross-Entropy Loss. Mathematical optimizations (backpropagation) adjust network weights to incrementally increase the target token’s probability.

Neural Network Internals

An LLM is a stateless mathematical function parameterized by billions or trillions of adjustable weights $W$.

\[\text{Output Probabilities} = f(X; W)\]
  • Parameters ($W$): Continuous numbers initialized randomly and tuned during training.
  • Statelessness: The network retains no biological memory or state updates across executions. Given identical inputs $X$ and fixed parameters $W$, internal mathematical operations execute identically.
  • Architecture: Modern models rely on the Transformer architecture, utilizing embedding tables, multi-head self-attention mechanisms, layer normalization, and multi-layer perceptrons (MLP).

Inference Dynamics

Inference is the autoregressive generation of output tokens from a fixed set of trained parameters:

  1. Provide an input token prefix sequence.
  2. Calculate the output probability distribution $P(t_{k+1})$.
  3. Sample a single token from the probability distribution using a stochastic sampling method (e.g., temperature sampling).
  4. Append the sampled token to the context sequence and repeat the loop.

Because inference relies on probabilistic coin-flipping over probability distributions, generations remain inherently stochastic and generative rather than deterministic text retrieval.

Part 2: Base Model Training & Inference (GPT-2 & Llama 3)

GPT-2: Historical Blueprint

OpenAI’s GPT-2 (2019) represents the foundational blueprint of modern LLM architectures. Although small by modern standards, its pipeline mirrors current state-of-the-art systems.

Parameter GPT-2 (2019) Modern Base Models (e.g., Llama 3.1)
Parameters 1.6 Billion 8B to 405 Billion+
Context Length 1,024 Tokens 128,000+ Tokens
Training Tokens ~100 Billion 15 Trillion+
Training Cost ~$40,000 (2019) ~$100–$600 (Reproduced via llm.c) 
      Training Step Update (1 Million Tokens Batch)
┌────────────────────────────────────────────────────────┐
│ Loss Value: 10.8 (Random) ---> 3.2 ---> 2.1 (Trained) │
└────────────────────────────────────────────────────────┘
                           │
                           ▼
  [Step 20 Inference]  --> "Since she is mine it's a part of..." (Gibberish)
  [Step 32,000 Final]  --> Coherent English structure & syntax

Hardware & GPU Compute Scaling

Training requires distributed GPU clusters (e.g., 8x NVIDIA H100 GPU nodes renting at ~$3/GPU/hour). Enterprise training deployments aggregate tens of thousands of GPUs into dedicated data centers to maximize parallel matrix operations.

Llama 3.1 Base Model Inference

Base models (such as Meta’s Llama 3.1 405B) are raw token simulators trained to complete internet text rather than act as helpful conversational assistants.

Characteristics of Base Models

  1. Gradients Held Fixed: During inference, parameter weights are frozen.
  2. Autocomplete Behavior: Prompting a base model with "What is 2+2?" will not yield an answer like "2+2 is 4". Instead, it completes the text based on web page patterns: ```text Input: What is 2+2? Output: What is 3+3? What is 4+4? (Simulating a math worksheet)


3. **Lossy Compression Metaphor**: A 405B parameter model acts like a lossy compression zip file of the 15-trillion token pretraining corpus. It retains a generalized representation of web knowledge rather than verbatim entries.
4. **Verbatim Regurgitation**: High-quality documents sampled multiple times during pretraining (such as Wikipedia entries) can be recited almost word-for-word from memory.
5. **Knowledge Cutoffs**: Queries regarding events occurring after dataset collection cutoffs result in plausible hallucinations grounded in pre-cutoff knowledge distributions.

---

## Base Model Prompting Techniques

Despite lacking conversational tuning, base models can perform complex tasks when prompted using structured context priming:

### Few-Shot Prompting (In-Context Learning)

```text
English: Teacher  | Korean: Sim
English: Apple    | Korean: Saegwa
English: Dog      | Korean: Gae
English: House    | Korean: Jib
English: Water    | Korean:

Result: The model detects the pattern and outputs Mul.

Assistant Persona Simulation via Structured Dialogue 

A conversational transcript between a helpful AI assistant and a human.
Human: Recommend top 5 landmarks in Paris.
Assistant: 1. Eiffel Tower 2. Louvre Museum 3. Arc de Triomphe...
Human: Why is the sky blue?
Assistant:

Result: Priming the prompt as a synthetic transcript forces the base model to simulate the assistant persona cleanly.

Part 3: Post-Training & Supervised Fine-Tuning (SFT)

Pretraining to Post-Training

Post-training transforms an unaligned Base Model (an internet text completer) into an Assistant Model capable of answering questions, following instructions, and refusing unsafe requests.

+-----------------------------------------------------------------------+
|                         PRETRAINING STAGE                             |
|  Data: 15 Trillion Web Tokens | Duration: ~3 Months | GPUs: Thousands |
+-----------------------------------------------------------------------+
                                   │
                                   ▼
                        [ Base Model Weights ]
                                   │
                                   ▼
+-----------------------------------------------------------------------+
|                         POST-TRAINING STAGE                           |
|  Data: Conversation Dialogue  | Duration: ~3 Hours  | Compute: Small  |
+-----------------------------------------------------------------------+
                                   │
                                   ▼
                      [ Assistant Model Weights ]

Post-Training Data & Formatting

Post-training uses Supervised Fine-Tuning (SFT) over curated dataset dialogues containing thousands of human-annotated multi-turn conversations.

Tokenizing Multi-Turn Dialogue

Conversations are converted into linear token sequences using structural formatting tokens (such as ChatML standards):

<|im_start|>user
What is 2+2?<|im_end|>
<|im_start|>assistant
2 + 2 equals 4.<|im_end|>
<|im_start|>user
What if it was multiplication?<|im_end|>
<|im_start|>assistant
  • Control Tokens: <|im_start|> and <|im_end|> are unique tokens added to the tokenizer vocabulary during post-training to demarcate structural turn boundaries.

Data Annotation Protocols & InstructGPT

First popularized by OpenAI’s InstructGPT (2022), initial SFT datasets relied on human annotators hired to write explicit prompt-response pairs following strict quality guidelines:

                  +-----------------------------------+
                  |  Human Labeling Guidelines        |
                  |  - Be Helpful, Honest, Harmless   |
                  |  - Refuse Malicious Requests      |
                  +-----------------------------------+
                                    │
                                    ▼
+-----------------------+                         +-----------------------+
|   User Prompt Ideas   |  ---> Annotator --->    | Ideal Assistant Responses
| (e.g., Code/Writing)  |                         |  (SFT Target Output)  |
+-----------------------+                         +-----------------------+

Modern Synthetic Data Evolution

In modern pipelines, human annotators no longer write all dialog responses from scratch. Frameworks like UltraChat utilize strong LLMs to generate vast synthetic conversational datasets, which are then filtered and edited by human experts.

Mental Model of an Assistant Model

When querying an assistant model, you are not communicating with a conscious entity. You are sampling from a statistical simulation trained to mimic an expert human annotator operating under corporate labeling guidelines.

Part 4: LLM Psychology, Capabilities & Constraints

Hallucinations

Hallucinations occur when an LLM outputs incorrect or fabricated facts with high confidence.

       [ User Query: "Who is Orson Kovats?" ]
                         │
                         ▼
   +-------------------------------------------+
   | Statistical Pattern Matching Triggered    |
   | Training data format: "Who is X?" -> Bio |
   +-------------------------------------------+
                         │
        ┌────────────────┴────────────────┐
        ▼                                 ▼
[ Naive Model (Falcon 7B) ]    [ Calibrated Model (Llama 3) ]
Outputs plausible fake bio     Internal uncertainty triggers
("1950s TV actor")             refusal: "I don't have records..."

Root Cause

SFT datasets historically contained mostly confident answers to questions starting with “Who is…” or “What is…“. Consequently, models learned to imitate the confident style of an answer even when internal activations indicated uncertainty.

Mitigation via Interrogation

  1. Interrogate base models empirically on factual text passages to locate boundary limits of their memorized parametric knowledge.
  2. Construct fine-tuning datasets that pair out-of-boundary queries with explicit refusals: “I don’t have access to information about X.”
  3. Connect internal uncertainty activations to explicit refusal outputs.

Tool Use & Memory Boundaries

To overcome parametric memory limits, models are trained to use external execution tools via specific token protocols.

User Context: "How many Stanley Cups did Dominic Hasek win?"
      │
      ▼
LLM Token Output:  Dominic Hasek Stanley Cups 
      │
      ▼ (Inference engine pauses, calls search API, injects text)
Injected Text: "Dominic Hasek won 2 Stanley Cups (1998, 2002)..."
      │
      ▼
LLM Continues: "Dominic Hasek won the Stanley Cup twice." [Citation 1]

Memory Hierarchy

Memory Type Location Properties Metaphor
Parametric Memory Fixed Parameter Weights ($W$) Vague, probabilistic long-term storage Reading a book 1 month ago
Working Memory Context Window Tokens Crisp, directly accessible active text Having the open book right in front of you

Knowledge of Self

LLMs have no intrinsic awareness of their identity or architecture. Answers to queries like “What model are you?” are entirely dictated by:

  1. Hardcoded identity strings written into system prompts.
  2. Fine-tuning dialogue examples provided during post-training.

Token-Level Computation Limits

LLMs execute a fixed amount of mathematical computation per output token.

\[\text{FLOPs per Token} \approx 2 \times N_{\text{params}}\]

Consequences

  • Complex Tasks Need Tokens to Think: For complex logic, math, or code tasks, an LLM cannot compute the answer in a single token. Prompting techniques (such as “Think step-by-step”) force the model to generate intermediate reasoning tokens, allocating more total compute to the problem.

Tokenization Traps: Spelling & Character Manipulation

Because transformers process subword tokens rather than raw character arrays, they struggle with character-level operations.

Word: "strawberry"
Tokenizer Mapping: ["straw", "berry"] -> Token IDs: [8238, 16801]
  • Spelling Failure: The model never sees the individual letters 's', 't', 'r', 'a', 'w', 'b', 'e', 'r', 'r', 'y'. Asking how many 'r's are in “strawberry” requires predicting character compositions hidden inside subword token embeddings.

Jagged Intelligence

LLM capability profiles differ significantly from human cognition:

  • Superhuman Capabilities: Immediate memorization of vast documentation, multi-language translation across dozens of tongues, and instantaneous code syntax generation.
  • Subhuman Blind Spots: Vulnerability to simple character-counting errors, logic traps without reasoning tokens, and subtle hallucination failures.

Part 5: Reinforcement Learning & DeepSeek-R1

Supervised Fine-Tuning vs. Reinforcement Learning

While SFT forces models to imitate human responses, Reinforcement Learning (RL) allows models to exceed human annotation ceilings by discovering optimal problem-solving pathways through self-play and automated verification.

SFT (Imitation Learning)          Reinforcement Learning (RL)
┌───────────────────────┐          ┌───────────────────────┐
│ Target: Copy Human    │          │ Target: Maximize      │
│ Labeler Examples      │          │ Verifiable Reward     │
└───────────────────────┘          └───────────────────────┘
  Cap: Human Labeler Level           Cap: Superhuman Discovery

Reinforcement Learning in Verifiable Domains

RL thrives in domains with unambiguous, programmatically verifiable feedback metrics (such as mathematics, code execution, and formal games).

[ Problem Prompt ] ---> [ Model Generates 10 Solutions ] 
                              │
                              ▼ Programmatic Verifier / Test Suite
                      [ Check Results ]
                        - Solution 3: PASS (Reward = +1.0)
                        - Solution 7: FAIL (Reward = -1.0)
                              │
                              ▼ Policy Gradient Update (PPO / GRPO)
                      [ Reinforce Paths Like Solution 3 ]

DeepSeek-R1 & Emergent Reasoning

DeepSeek-R1 demonstrates the power of applying RL directly to base/SFT models in reasoning domains without relying on human CoT annotations.

Emergent Properties from Pure RL

  • Spontaneous Thought Generation: Models naturally learn to allocate variable “thinking tokens” before committing to a final answer.
  • Self-Correction Patterns: The model learns internal verification behaviors (e.g., generating phrases like “Wait, let me re-check this calculation…” mid-generation).
  • Length Expansion: Average token usage per response increases autonomously to solve harder benchmark problems.

The AlphaGo Parallel

The trajectory of RL in language models mirrors DeepMind’s AlphaGo:

  1. Phase 1 (SFT Equivalent): AlphaGo initially trained on thousands of human expert Go games, capping performance near human levels.
  2. Phase 2 (RL Equivalent): AlphaGo played millions of self-play games against cloned versions of itself.
  3. Move 37 Moment: Free from human imitation constraints, AlphaGo discovered novel moves beyond traditional human Go strategy. Reasoning models trained with RL undergo an identical transformation.

RLHF: Non-Verifiable Domains

For subjective tasks lacking automated programmatic verifiers (such as writing essays or conversational responses), models rely on Reinforcement Learning from Human Feedback (RLHF).

Step 1: Train Reward Model           Step 2: PPO / DPO Optimization
┌─────────────────────────────┐      ┌─────────────────────────────┐
│ Prompt: "Write an essay..." │      │  Policy Model Generates     │
│  - Generation A             │      │  Response Candidates        │
│  - Generation B             │      └──────────────┬──────────────┘
│                             │                     │
│ Human Prefers: A > B        │                     ▼
│                             │      [ Evaluated by Reward Model ]
▼                             │                     │
[ Reward Model (RM) Trained ] │                     ▼
└─────────────────────────────┘      [ Update Policy Model Weights ]
  • Reward Model (RM): A neural network trained on human pairwise preference evaluations (Option A vs Option B) to output a scalar quality score.
  • Optimization Algorithms: Proximal Policy Optimization (PPO) or Direct Preference Optimization (DPO) adjust LLM parameters to maximize scores from the Reward Model while penalizing drift via KL-Divergence constraints.

Part 6: Ecosystem, Future Trends & Summary

Preview of Things to Come 

                       MODERN LLM EVOLUTION
                       
   Multimodal Tokens          Autonomous Agents          Test-Time Compute
┌─────────────────────┐    ┌─────────────────────┐    ┌─────────────────────┐
│ Native Processing:  │    │ Tool Execution Loops│    │ Allocate Inference  │
│ Audio, Images,      │ -> │ over Multi-Hour     │ -> │ Time dynamically    │
│ Video Streams       │    │ Workflows           │    │ for hard problems   │
└─────────────────────┘    └─────────────────────┘    └─────────────────────┘
  1. Native Multimodality: Moving away from separate modality wrappers to process text, image, audio, and video tokens unified within a single Transformer architecture.
  2. Agentic Workflows: Shifting from single-turn response generation to autonomous loops where models issue execution commands, inspect output, fix errors, and iterate toward long-term goals.
  3. Test-Time Compute Scaling: Shifting compute investment from pretraining to runtime inference. Harder problems receive significantly larger compute budgets at runtime, allowing models to reflect and verify longer before responding.

Track state-of-the-art developments across model performance and research publications using these core platforms:

  • LMSYS Chatbot Arena (lmarena.ai): Crowdsourced, blind A/B testing platform providing Elo rankings for open and proprietary language models.
  • Hugging Face Open LLM Leaderboard: Standardized benchmark evaluation tracking open-weights models.
  • LM Studio & Ollama: Local desktop frameworks for executing quantized open-weights models locally on user hardware.
  • ArXiv & AI News Newsletters: Repositories for tracking research papers on architectural optimizations, training methods, and dataset curation.

Grand Summary: The 3-Stage LLM Lifecycle 

Stage 1: Pretraining
┌────────────────────────────────────────────────────────────────────────┐
│ Raw Internet Data (15T+ Tokens) ---> Transformer Training (Thousands GPU│
│ Output: Base Model (Internet Token Simulator)                          │
└────────────────────────────────────────────────────────────────────────┘
                                   │
                                   ▼
Stage 2: Supervised Fine-Tuning (SFT)
┌────────────────────────────────────────────────────────────────────────┐
│ Conversation Dialogues ---> Supervised Alignment Training              │
│ Output: Assistant Model (Imitates Human Annotator Persona)             │
└────────────────────────────────────────────────────────────────────────┘
                                   │
                                   ▼
Stage 3: Reinforcement Learning (RL / RLHF)
┌────────────────────────────────────────────────────────────────────────┐
│ Reward Models / Verifier Systems ---> Policy Optimization (GRPO / DPO) │
│ Output: Reasoning Agent (Self-Correcting, Superhuman Problem Solver)   │
└────────────────────────────────────────────────────────────────────────┘
This site uses Just the Docs, a documentation theme for Jekyll.