Build a Large Language Model (From Scratch)

Book Sebastian Raschka LLMs

Build a Large Language Model (From Scratch) is a practical guide written by Sebastian Raschka, a leading AI researcher and educator. The book breaks down the complexities of generative AI and shows you how to design, train, and fine-tune your own GPT-like models step-by-step using PyTorch.

About this Book

This book teaches you how to construct a complete LLM from scratch. You will write code for every single component, from the initial text tokenization and embedding generation, through the multi-head attention mechanism and transformer architecture block, to pretraining, fine-tuning, and optimizing your models.

Key Topics Covered

  • Understanding Tokenizers, Word Embeddings, and Context Vectors
  • Implementing Causal and Multi-Head Attention Mechanisms
  • Coding a full GPT-style LLM Architecture in PyTorch
  • Pretraining on Unlabeled Corpora (Optimization & Training Loops)
  • Loading Pretrained weights (e.g. from OpenAI's GPT-2)
  • Instruction Fine-Tuning for Classification & Dialogue agents

Why this book is useful

It demystifies LLMs. Instead of treating models as API endpoints or pre-packaged wrappers, you learn the exact mechanics by typing code line-by-line. Crucially, the architectures are designed to run, train, and test on standard consumer laptops, making advanced AI development accessible to everyone.

Who should read it

Developers, machine learning engineers, and software architects who want a deep, practical understanding of transformer-based language models. Basic knowledge of Python and familiarity with deep learning concepts or PyTorch are helpful pre-requisites.

Get Started with Sebastian Raschka's Book

Read the book on Manning Publications and download the official repository containing all source code and Jupyter notebooks for free on GitHub.

Visit Manning Catalog Back to Resources