AI Architecture · Grade 11 · Chapter 2

LLMs Under the Hood

From tokens and vector space embeddings to multi-head self-attention mechanisms and logits sampling.

Historical Context

The Paper That Changed Everything

In June 2017, eight researchers at Google published "Attention Is All You Need". They replaced sequential recurrent neural networks (RNNs) with a purely attention-driven architecture: the Transformer.

Discussion: Before 2017, AI translated sentences word-by-word sequentially. Why did parallel attention enable training models on the entire internet?

Learning Objectives

What We Will Master Today

Trace the complete Transformer execution pipeline from raw text to output token
Deconstruct subword tokenization (BPE) and vocabulary mapping
Perform high-dimensional embedding vector arithmetic & cosine similarity calculations
Understand the math of Query, Key, Value (Q, K, V) self-attention
Master sampling hyperparameters: Temperature, Top-K, and Top-P (Nucleus)

Prior Knowledge Connection

Grade 10 vs Grade 11 NLP Depth

Grade 10 Mindset

AI reads text, understands prompts, generates responses, and sometimes hallucinates facts.

Grade 11 CS Mindset

AI processes numerical tensor matrices, computes dot-product attention scores, projects token vectors, and samples probability distributions over vocabulary lists.

Pipeline Overview

The 6-Step LLM Processing Pipeline

1. Tokenization

Text -> Token IDs

2. Embedding

IDs -> Dense Vectors

3. Positional

Add order info

4. Self-Attention

Calculate context scores

5. Feed-Forward

FFN projection

6. Logits & Softmax

Sample next token

Step 1: Tokenization

Text to Numbers: Byte-Pair Encoding (BPE)

Computers cannot read text. BPE breaks words into frequent subword chunks:

Raw Text:  "Unbelievable cybersecurity breakthrough!"
Tokens:    ["Un", "believ", "able", " cyber", "security", " break", "through", "!"]
Token IDs: [3482, 18920, 521, 1492, 4921, 318, 920, 0]

Vocabulary Size (V): Typically 32,000 to 100,000 unique subword tokens.
Rule of Thumb: 1 token ≈ 0.75 English words (or ~4 characters).
    

Tokenization Pitfalls

Tokenizer Edge Cases & Hacks

High Cost / Inefficiency

Non-English scripts, indentation in Python, or complex math/emojis consume 3-5x more tokens per character!

Special Tokens

Control tokens like <|endoftext|>, <|im_start|> define conversation roles and context boundaries.

Python Code

Inspecting Tokens with Tiktoken

import tiktoken

# Load OpenAI's cl100k_base tokenizer
enc = tiktoken.get_encoding("cl100k_base")

text = "Tokenization is fascinating!"
tokens = enc.encode(text)

print("Token IDs:", tokens)
# [3069, 1634, 374, 18765, 0]

for token_id in tokens:
    print(f"{token_id} -> '{enc.decode([token_id])}'")
    

Step 2: Vector Space

High-Dimensional Embeddings

Each token ID is mapped to a high-dimensional vector of floating-point numbers (e.g. $D = 1536$ or $D = 4096$).

What is Semantic Geometry?

Words with similar meanings are located close together in geometric vector space. Words with similar relationships form parallel directional vectors!

Vector Arithmetic

Vector Math: Semantic Relationships

Mathematical operations on embedding vectors reveal conceptual relationships:

Vector Formula:
  V("King") - V("Man") + V("Woman") ≈ V("Queen")

Vector Similarity Math (Cosine Distance):
  cos(θ) = (A · B) / (||A|| * ||B||)

  If cos(θ) = 1.0 -> Identical direction / semantic meaning
  If cos(θ) = 0.0 -> Orthogonal / Unrelated concepts
  If cos(θ) = -1.0 -> Opposite meaning
    

Step 3: Positional Encoding

Injecting Sequence Order

Unlike RNNs, Transformers process all input tokens simultaneously in parallel. Without positional encoding, "dog bites man" and "man bites dog" look identical to the model!

Sinusoidal Encodings

Adds sine/cosine waves of varying frequencies to embedding vectors to encode absolute position $i$.

Rotary Position Embeddings (RoPE)

Rotates embedding vectors in 2D pairs based on relative position. Used in Llama and modern models.

Step 4: The Core Breakthrough

Self-Attention: Query, Key, Value

Self-Attention allows every token in a prompt to look at every other token and decide which ones are relevant.

Query Vector (Q)

"What am I looking for?" (Current token searching for context).

Key Vector (K)

"What content do I contain?" (Other tokens offering information).

Value Vector (V)

"If relevant, what information do I pass along?"

Attention Mathematics

Scaled Dot-Product Attention Equation

The foundational equation powering all modern Generative AI:

Attention(Q, K, V) = Softmax( (Q * K^T) / sqrt(d_k) ) * V

Step 1: Q * K^T  -> Compute raw similarity matrix between all pairs of tokens.
Step 2: / sqrt(d_k) -> Scale values to prevent vanishing gradients during training.
Step 3: Softmax() -> Convert raw scores into normalized probabilities (sum = 1.0).
Step 4: * V -> Compute weighted linear combination of Value vectors!
    

Attention Example

Resolving Ambiguity via Attention

Sentence: "The animal didn't cross the street because it was too tired."

What does "it" refer to? The Self-Attention layer calculates high dot-product attention scores between "it" and "animal" (and low scores for "street"), transferring contextual vector features!

Multi-Head Attention

Multiple Parallel Perspectives

A single attention head can only focus on one relationship at a time. Multi-Head Attention splits $Q, K, V$ into multiple independent heads (e.g. 32 heads):

Head 1 Focus

Tracks grammatical relationships (Subject -> Verb agreement).

Head 2 Focus

Tracks semantic references (Pronoun -> Noun resolution).

Step 5: Dense Processing

Feed-Forward Networks & Residuals

After attention contextualizes vectors, each token vector passes independently through a Feed-Forward Neural Network (FFN):

Residual Connections (Skip Connections): Prevents vanishing gradients across 80+ transformer layers by adding input tensor directly to output $x + f(x)$.
Layer Normalization: Stabilizes neural network activation values.
Non-Linear Activation (GELU / SwiGLU): Enables learning complex non-linear logic patterns.

Step 6: Token Generation

Logits to Probability Distribution

The final layer outputs raw unnormalized numbers called Logits for every token in the vocabulary ($V = 100,000$).

Raw Logits Output:
  " Paris":  14.2
  " France": 11.5
  " banana": -3.1

Softmax Formula: P(i) = exp(Logit_i) / sum(exp(Logit_j))

Resulting Probabilities:
  " Paris":  85.2%
  " France": 12.1%
  " banana":  0.0001%
    

Decoding & Sampling

Hyperparameter Control: Temperature

Temperature ($T$) scales raw logits before applying Softmax ($Logit / T$):

Low Temperature (T = 0.0 - 0.2)

Sharpen distribution! Model greedy-picks top probability token. Ideal for coding, math, and factual queries.

High Temperature (T = 0.8 - 1.2)

Flattens distribution! Increases probability of lower-ranked tokens. Creates creative writing, but risks hallucination.

Truncation Strategies

Top-K and Top-P (Nucleus) Sampling

Top-K Sampling

Restricts next-token choice strictly to the top $K$ most likely candidates (e.g. $K=50$). Ignores the long tail of low-probability tokens.

Top-P (Nucleus) Sampling

Accumulates tokens until their cumulative probability exceeds threshold $P$ (e.g. $P=0.90$). Dynamically adjusts candidate pool size!

Memory & Hardware

Context Window & KV-Cache Footprint

Why are long prompts expensive in hardware?

Memory Formula for KV-Cache:
  KV_Bytes = 2 * 2 * Layers * Heads * Dim * Sequence_Length * Batch_Size

For a 70B parameter model with 128k context:
  KV-Cache alone requires over 24 GB of High-Bandwidth VRAM (HBM) per session!
    

Smartboard Challenge

Predict the Next Token & Softmax

Prompt: "The capital of Japan is ___"

Task: Suppose raw logits are: Tokyo=10.0, Kyoto=8.0, Sushi=2.0.
Calculate which candidate gets chosen at $T=0.1$ vs $T=2.0$. Discuss how temperature alters output randomness.

Smartboard Math

2D Vector Dot-Product Calculation

Let Vector A (Apple) = [0.8, 0.6] and Vector B (Banana) = [0.9, 0.4]. Vector C (Car) = [-0.5, 0.8].

Dot Product (A · B) = (0.8 * 0.9) + (0.6 * 0.4) = 0.72 + 0.24 = 0.96 (High similarity!)
Dot Product (A · C) = (0.8 * -0.5) + (0.6 * 0.8) = -0.40 + 0.48 = 0.08 (Orthogonal/Unrelated)
    

Guided Python Coding

Simulating Softmax in Python

import math

def softmax(logits: list, temp: float = 1.0) -> list:
    # Scale by temperature
    scaled = [x / temp for x in logits]
    exp_vals = [math.exp(x) for x in scaled]
    total = sum(exp_vals)
    return [e / total for e in exp_vals]

logits = [4.0, 2.0, 1.0]
print("T=1.0:", [round(p, 3) for p in softmax(logits, temp=1.0)])
print("T=0.2:", [round(p, 3) for p in softmax(logits, temp=0.2)])
    

Question 1

What is the primary function of Byte-Pair Encoding (BPE)?

ATo encrypt prompts using AES-256

BTo convert raw text into frequent subword token IDs

CTo compute vector similarity scores between words

DTo eliminate low probability words from output

Click to reveal answer

Question 2

In Self-Attention, which matrix represents "what information a token contains"?

AQuery (Q)

BKey (K)

CValue (V)

DSoftmax (S)

Click to reveal answer

Question 3

What happens when Temperature is set close to 0.0?

AOutputs become highly random and creative

BOutput becomes deterministic, selecting highest logit

CThe context window expands to double size

DTokenization reverts to word-level splitting

Click to reveal answer

Question 4

Why do Transformers require Positional Encodings?

ABecause subword tokens do not have unique IDs

BBecause attention processes all tokens in parallel without sequence order

CTo compress vector embeddings into lower dimensions

DTo restrict vocabulary size to 32,000

Click to reveal answer

Question 5

What does Top-P (Nucleus) sampling do?

ASelects fixed top K number of tokens

BCuts off token pool once cumulative probability exceeds P

CDivides logits by temperature value

DHashes tokens into 1536-dimensional space

Click to reveal answer

Question 6

True or False: LLMs think and reason like human brains.

ATrue: Transformers form logical conceptual thought paths

BFalse: They are statistical next-token probability predictors trained on data

Click to reveal answer

Common Misconceptions

Busting LLM Myths

Myth: "An LLM retrieves information like a Google Search database."

Reality: LLMs do NOT store documents or web pages! Knowledge is stored lossily across billions of floating-point weight parameters. Generation is re-synthesizing probabilities!

Chapter Summary

Key Takeaways

Tokenization: Text -> Subword BPE integer tokens.
Embeddings: Tokens -> Vectors in high-dimensional semantic geometric space.
Self-Attention: $Q \cdot K^T$ matrix dot products calculate contextual relationships between tokens.
Sampling: Temperature & Top-P control output distribution determinism vs creativity.

Exit Ticket

Before You Leave

Written Prompt: Explain in 2 sentences why setting $T=0.0$ is ideal for generating Python code, but bad for writing a creative fantasy poem.