AI Engineering

Autograd computes every gradient; optimizers take the step.

Gradients of plain numerical functions, compiled for GPUs and TPUs.

Train a small neural network in the browser and watch the loss.

Linear models trained with stochastic gradient descent.

Plot error against model complexity.

Track train vs eval loss per run.

Early stopping on eval loss.

Train tiny networks in the browser.

The most used deep learning framework.

The visual video series.

OpenAI's fast BPE tokenizer for GPT models.

Train and run BPE, WordPiece and Unigram tokenizers.

Language-independent subword tokenizer used by Llama, Gemma and T5.

Paste a prompt and see its tokens for many models.

Count tokens for a Claude request before you send it.

The paper that introduced sinusoidal position encodings.

Rotary position embeddings, used by Llama, Mistral and Qwen.

Inspect any open model's embedding table and position settings in its config.

Explore word embeddings in 3D and find nearest neighbours.

The 2017 paper that introduced the transformer and scaled dot-product attention.

Interactive GPT-2 in the browser, from Georgia Tech's Polo Club.

Visualize attention heads of real models.

A small, readable GPT you can train yourself.

The fast, memory-efficient attention kernel most inference stacks use.

A GPT whose whole block fits on one screen of code.

A 3D walk through every matrix in a small GPT.

GPT-2 running in the browser, block by block.

Meta's reference code and configs: width, layers, heads and feed-forward size.

Dong et al., 2021: why stacks without residual connections collapse.

Hosted models with temperature, top-p, and logit bias.

Claude with temperature and sampling controls.

Run open models locally with the same sampling knobs.

Load models and call generate() with temperature and top-k.

High-throughput GPU inference server.

Local CPU/GPU inference for open weights.

PagedAttention stores the KV cache in pages to fit more users per GPU.

Cache classes for generation, including quantized and offloaded caches.

Local inference with a configurable context size and cache type.

Providers reuse the cached prefix of a repeated prompt and charge less for it.

The open web crawl behind most pretraining datasets.

15T tokens of filtered, deduplicated web text, with the recipe published.

Filtering and deduplication pipelines for large text datasets.

NVIDIA's framework for training large transformers across many GPUs.

PyTorch's reference for large-scale LLM pretraining.

SFT, reward modelling, PPO and DPO trainers for open models.

RLHF at scale with PPO, DPO and reward models.

Ouyang et al., 2022: SFT, a reward model and PPO.

Rafailov et al., 2023: preference tuning without a separate reward model.

Bai et al., 2022: AI feedback guided by written principles.

Gao et al., 2022: reward keeps rising while real quality falls.

Training compute-optimal LLMs (Hoffmann et al.).

The first LLM scaling laws.

Data on training compute over time.

Kalai et al., 2025: rare facts and guess-rewarding grading.

Answers that point to the exact passages in your documents.

Faithfulness and answer-relevance scores for RAG outputs.

How often popular models invent facts when summarizing.

Short factual questions graded right, wrong or not attempted.

Cache a long, unchanged prefix and pay a fraction to reuse it.

Automatic discount on repeated prompt prefixes.

Store a large context once and reference it across calls.

Self-hosted inference with automatic prefix caching.

Cheaper repeated long prompts.

Why mid-context facts get missed.

RAG pipelines and indexing.

Hosted embedding API for text.

Multilingual embedding models as an API.

Open embedding models you can run yourself.

Embeddings + vector store in one open-source package.

Vector similarity inside PostgreSQL.

Managed vector database for similarity search.

Open-source vector store with hybrid search.

Character, recursive, Markdown-header and token splitters.

Sentence, semantic and hierarchical chunking.

Parse PDFs, slides and HTML into titled elements before chunking.

A small chunking library with token, sentence and semantic chunkers.

BM25 plus dense vectors, with reciprocal rank fusion built in.

Open-source search with hybrid queries and normalization.

Vector database with a hybrid (BM25 + vector) query.

Vector database with sparse + dense vectors and fusion.

Search and ranking engine with multi-phase ranking.

Hosted cross-encoder reranker for search results.

Retrieval chains, loaders, and prompt templates.

Connectors, chunking, and query engines over your data.

Open-source RAG and search pipelines.

Parse PDFs and HTML into clean chunks for indexing.

Persist retrieved chunks with embeddings.

Hybrid keyword + vector retrieval in one index.

HyDE and multi-step query rewriting.

Generate several rephrasings and merge what they retrieve.

The pre-LLM fix: map user words to document words.

Anthropic's write-up of these six patterns (Dec 2024).

Graphs of model calls with loops, branches and state.

Agents, handoffs and guardrails.

Multi-agent conversations and orchestration.

Durable execution for long-running agent workflows.

Reasoning and acting, Yao et al.

Build agent loops as graphs.

Anthropic's agent loop with tools.

Test prompts against cases with assertions and model graders, from the command line.

Framework and registry of evals for language models.

The UK AI Security Institute's open evaluation framework.

Metrics for RAG: faithfulness, answer relevance, context recall.

Hosted evals, scoring and experiment comparison.

Tracing, datasets and evaluators for LLM apps.

precision_recall_fscore_support.

Precision and recall explained.

Score prompts against labelled tests.

LLM-rubric and pairwise grading in test configs.

Model-graded eval templates you can adapt.

Pairwise human votes, the ground truth many judges are checked against.

LoRA, QLoRA and other adapters for Transformers models.

Faster, lower-memory LoRA fine-tuning on one GPU.

Serve one base model with many adapters loaded per request.

Hu et al., 2021: Low-Rank Adaptation of Large Language Models.