Signals and Threads

Why Testing is Hard and How to Fix it with Will Wilson

Will Wilson is the founder and CEO of Antithesis, which is trying to change how people test software. The idea is that you run your application inside a special hypervisor environment that intelligently (and deterministically) explores the program’s state space, allowing you to pinpoint and replay the events leading to crashes, bugs, and violations of invariants. In this episode, he and Ron take a broad view of testing, considering not just “the unreasonable effectiveness of example-based tests” but also property-based testing, fuzzing, chaos testing, type systems, and formal methods. How do you blend these techniques to find the subtle, show-stopper bugs that will otherwise wake you up at 3am? As Will has discovered, making testing less painful is actually a tour of some of computer science’s most vexing and interesting problems. You can find the transcript for this episode on our website [https://signalsandthreads.com/why-testing-is-hard-and-how-to-fix-it]. Some links to topics that came up in the discussion: * Antithesis [https://antithesis.com/], Will’s company * FoundationDB’s deterministic simulation framework [https://apple.github.io/foundationdb/testing.html] * QuickCheck [https://hackage.haskell.org/package/QuickCheck] — the original Haskell property-based testing library, by Koen Claessen and John Hughes * Hypothesis [https://hypothesis.readthedocs.io/] — property-based testing for Python, created by David MacIver * QuviQ [https://www.quviq.com/] — John Hughes’ company commercializing QuickCheck, including automotive testing work * Netflix Chaos Monkey [https://netflix.github.io/chaosmonkey/] * Goodhart’s law [https://en.wikipedia.org/wiki/Goodhart%27s_law] — “When a measure becomes a target, it ceases to be a good measure” * CAP theorem [https://en.wikipedia.org/wiki/CAP_theorem] — the impossibility result for distributed systems that FoundationDB claims to have in some sense violated. * Paxos [https://en.wikipedia.org/wiki/Paxos_(computer_science)] — the consensus algorithm FoundationDB reimplemented from scratch * Large cardinals [https://en.wikipedia.org/wiki/Large_cardinal], an area Will studied before abandoning mathematics * Lyapunov exponent [https://en.wikipedia.org/wiki/Lyapunov_exponent] — measure of chaotic divergence * Chesterton’s fence [https://en.wikipedia.org/wiki/G._K._Chesterton#Chesterton's_fence] * The Story of the Flash Fill Feature in Excel [https://blog.sigplan.org/2021/09/14/the-story-of-the-flash-fill-feature-in-excel/] * Building a C compiler with a team of parallel Claudes [https://www.anthropic.com/engineering/building-c-compiler] * Barak Richman, “How Community Institutions Create Economic Advantage: Jewish Diamond Merchants in New York” [https://scholarship.law.duke.edu/faculty_scholarship/1639/]

Why ML Needs a New Programming Language with Chris Lattner

Chris Lattner is the creator of LLVM and led the development of the Swift language at Apple. With Mojo, he’s taking another big swing: How do you make the process of getting the full power out of modern GPUs productive and fun? In this episode, Ron and Chris discuss how to design a language that’s easy to use while still providing the level of control required to write state of the art kernels. A key idea is to ask programmers to fully reckon with the details of the hardware, but making that work manageable and shareable via a form of type-safe metaprogramming. The aim is to support both specialization to the computation in question as well as to the hardware platform. “Somebody has to do this work,” Chris says, “if we ever want to get to an ecosystem where one vendor doesn’t control everything.” You can find the transcript for this episode on our website [https://signalsandthreads.com/why-ml-needs-a-new-programming-language]. Some links to topics that came up in the discussion: * Democratizing AI compute [https://www.modular.com/blog/democratizing-compute-part-1-deepseeks-impact-on-ai] (an 11-part series) * Modular AI [https://www.modular.com/] * Mojo [https://www.modular.com/mojo] * MLIR [https://mlir.llvm.org/] * Swift [https://www.swift.org/]

The Thermodynamics of Trading with Daniel Pontecorvo

Daniel Pontecorvo runs the “physical engineering” team at Jane Street. This group blends architecture, mechanical engineering, electrical engineering, and construction management to build functional physical spaces. In this episode, Ron and Dan go deep on the challenge of heat exchange in a datacenter, especially in the face of increasingly dense power demands—and the analogous problem of keeping traders cool at their desks. Along the way they discuss the way ML is changing the physical constraints of computing; the benefits of having physical engineering expertise in-house; the importance of monitoring; and whether you really need Apollo-style CO2 scrubbers to ensure your office gets fresh air. You can find the transcript for this episode on our website [https://signalsandthreads.com/the-thermodynamics-of-trading]. Some links to topics that came up in the discussion: * ASHRAE [https://www.ashrae.org/] (American Society of Heating, Refrigerating and Air-Conditioning Engineers) * Some research [https://ehp.niehs.nih.gov/doi/10.1289/ehp.1510037] on [https://www.researchgate.net/publication/283944378_Effects_of_Exposure_to_Carbon_Dioxide_and_Human_Bioeffluents_on_Cognitive_Performance] CO2’s effects [https://pmc.ncbi.nlm.nih.gov/articles/PMC3548274/] on human performance, which motivated us to look into CO2 Scrubbers [https://en.wikipedia.org/wiki/Carbon_dioxide_scrubber] * The Open Compute Project [https://www.opencompute.org/] * Rail-Optimized [https://developer.nvidia.com/blog/doubling-all2all-performance-with-nvidia-collective-communication-library-2-12/] and Rail-only [https://people.csail.mit.edu/ghobadi/papers/rail_llm_hotnets_2023.pdf] network topologies. * Immersion cooling [https://en.wikipedia.org/wiki/Immersion_cooling], where you submerge a machine in a dielectric fluid!

Building Tools for Traders with Ian Henry

Ian Henry started his career at Warby Parker and Trello, building consumer apps for millions of users. Now he writes high-performance tools for a small set of experts on Jane Street’s options desk. In this episode, Ron and Ian explore what it’s like writing code at a company that has been “on its own parallel universe software adventure for the last twenty years.” Along the way, they go on a tour of Ian’s whimsical and sophisticated side projects—like Bauble, a playground for rendering trippy 3D shapes using signed distance functions—that have gone on to inform his work: writing typesafe frontend code for users who measure time in microseconds and prefer their UIs to be “six pixels high.” You can find the transcript for this episode on our website [https://signalsandthreads.com/building-tools-for-traders]. Some links to topics that came up in the discussion: * Bauble studio [https://bauble.studio/] * Janet for Mortals, by Ian Henry [https://janet.guide/] * What if writing tests was a joyful experience? [https://blog.janestreet.com/the-joy-of-expect-tests/]

Finding Signal in the Noise with In Young Cho

In Young Cho thought she was going to be a doctor but fell into a trading internship at Jane Street. Now she helps lead the research group’s efforts in machine learning. In this episode, In Young and Ron touch on the porous boundaries between trading, research, and software engineering, which require different sensibilities but are often blended in a single person. They discuss the tension between flexible research tools and robust production systems; the challenges of ML in a low-data, high-noise environment subject to frequent regime changes; and the shift from simple linear models to deep neural networks. You can find the transcript for this episode on our website [https://signalsandthreads.com//finding-signal-in-the-noise/].