Amazon US-EAST-1 Outage in Details

My new book - Root cause, Stories from two decades of backend bugs

I wrote a new book that has been in the works for years. It is called Root Cause, and it is for those who enjoy the art of backend engineering. Early in my career, 20 years ago, I built backend and database applications without fully grasping their inner mechanics. Performance issues, race conditions, bugs, and even data corruption often left me lost. Since that day, I resolved to truly understand how systems work. From networking protocols and intermediary proxies to backend services and various database engines. I made it a habit to follow every request on its journey through the dark alleys of the network, down to the bowels of the database engine, meanwhile interacting with various kernel data structures in the process at every hop, and back. I became obsessed with understanding what happens behind the scenes in software. Not just what breaks, and how but also why and what was the source of the bleed.  Root Cause is a collection of the most interesting bugs I encountered, ranging from performance bottlenecks and non-deterministic crashes to subtle data inconsistencies and incorrect results. This book is for anyone curious about how production backend systems really behave under pressure, and how to debug them when they don’t. Even when you don’t have access to the source code. Root cause consists of 15 chapters, each is a story about a backend bug, with investigation, diagrams, a section of a fundamental concept until the root cause is revealed. Grab your copy here paperback or kindle ebook paperback https://amzn.to/4cKfZhe [https://amzn.to/4cKfZhe] ebook https://amzn.to/4cfQjJj [https://amzn.to/4cfQjJj]

5 Backend Design Patterns for Managing Threads and Sockets

In this video I introduce 5 different design patterns for building backend applications. Each mode explains how a socket listener is established, a connections are established and how threads and connections are managed to read, write and process requests.

Page Tables

Page tables provide the mapping between virtual memory and physical memory for each process. This means it needs to be as efficient and as fast as possible. I explore the inner workings of page tables in this episode. * 0:00 Intro * 2:00 Virtual Memory ⁃ ⁃ 8:00 MMU * 10:00 Page Tables ⁃ ⁃ ⁃ ⁃ ⁃ ⁃ ⁃ 11:30 Single Table Byte Addressability   ⁃ ⁃ ⁃ ⁃ ⁃ ⁃ ⁃ ⁃ 16:00 Single Table Page addressability  ⁃ ⁃ ⁃ ⁃ ⁃ 19:00 Multi-level Paging (Radix tree) ⁃ ⁃ 31:00 Huge Tables ⁃ ⁃ 33:00 TLB ⁃ ⁃ Summary

CPU and Kernel Page Faults

Page faults occurs when the process tries to access a memory that isn’t backed by a physical page kernel raises a fault which loads a page. It happens on first access, stack expansion, COW, swap and much more. However it comes with a cost.  In this episode of the backend engineering show I dissect the need and the cost page faults in the kernel.  * 0:00 Intro  * 4:00 Virtual memory * Abstraction of physical memory * Memory sharing * Allow more processes to run , unused go to disk * Numa, kernel can place memory near the cpu * 12:00 VMA areas * Text/code  * Data  * BSS * Heap * Stack * 19:50 Kernel mode * 25:30 What is a Page fault? * 30:30 First access page fault * 33:00 Stack Expansion page fault * 34:30 CoW page fault * 38:00 Swap page fault * 39:39 File backed page fault * 40:29 Permission page fault  * 45:30 Summary

Amazon US-EAST-1 Outage in Details

On October 19 2025 AWS experienced an outage that lasted over a day, 10 days later we finally got the root cause analysis and we know exactly what caused the DNS to fail0:00 Summary 5:30 How did Dynamo lost its DNS?13:41 EC2 Errors 16:16 Network Load Balancer ErrorsRCA here https://aws.amazon.com/message/101925/