Building LLM Agents: Evaluation, Safety, and Tool Use with Georgios Chouliaras

Building LLM Agents: Evaluation, Safety, and Tool Use with Georgios Chouliaras

From BERT to Agents: Building Production AI at Booking.com After seven years building ML systems that serve millions of travelers, Georgios Chouliaras has watched the field transform from hand-coded chatbot rules to autonomous agents—and he's learned which shiny new approaches actually work in production. Georgios Chouliaras, Senior Machine Learning Scientist at Booking.com, joins me to share hard-won insights from deploying AI at scale. His journey spans customer service chatbots that broke during COVID (because the training data didn't include "global pandemic"), company-wide ML best practices, and now the cutting edge of agent development. In this episode, we explore: * Why LLMs represent the biggest abstraction leap since high-level programming languages, and what control you sacrifice for that flexibility * The practical framework for deciding when LLMs beat classical ML (hint: it's not always about having text data) * How to build LLM judges that actually work: starting with binary labels, achieving annotator agreement before anything else, and why boundary cases matter most for few-shot examples * What's genuinely unsolved in agents right now, memory as lifelong learning and planning approaches that don't collapse under complexity Georgios challenges some popular assumptions: the REACT pattern everyone implements? He hasn't seen it consistently outperform simpler approaches. Massive parameter counts? Architecture and training data now matter more. His underhyped pick: straightforward function calling often beats elaborate agent architectures. The core takeaway: Use the simplest tool that solves your problem. Production users don't care if you're running a sophisticated multi-agent system, they care if it works. Connect with Georgios: * LinkedIn: https://www.linkedin.com/in/chouligi/ [https://www.linkedin.com/in/chouligi/] Connect with me: * LinkedIn: https://www.linkedin.com/in/christianbarra/ [https://www.linkedin.com/in/christianbarra/] Check out our awesome sponsor, dearmachines.com [https://dearmachines.com], QA AI Agents for Continuous Testing.

Building at Scale: From Flexible Team Models to AI-Powered Engineers with Jan Hegewald

What happens when you throw out traditional team structures and rebuild them around missions instead of functions? Jan Hegewald, VP of Engineering at Taxdoo and former engineering leader at Zalando and SumUp, shares how he dissolved static component teams in favor of flexible, outcome-focused "mission teams" that reformed every quarter around specific goals. The experiment worked brilliantly until it didn't. Jan brings nearly two decades of experience building high-volume systems across e-commerce, FinTech, and point-of-sale infrastructure. At Zalando, his teams processed hundreds of thousands of product offers during Black Friday. At SumUp, he built the point-of-sale system now visible in stores across Europe, Australia, and Brazil. His career offers a rare perspective on what actually transfers between startup culture and enterprise scale. In this episode, we explore: How mission-based teams increase ownership but create tricky problems around code quality, knowledge retention, and maintenance work, plus the practical solutions that worked (and the seniority level required to pull it off) Why treating QA as an enabling function rather than a gatekeeper changes the game for release velocity, even in heavily regulated FinTech environments The hidden operational costs of AI that teams overlook: model drift, retraining cycles, and why "garbage in, garbage out" still determines success more than any algorithm What engineering leaders should actually hire for now, why the "I only do backend" engineer is becoming obsolete and what holistic skill set replaces narrow specialization Jan also pushes back on AI hype with examples from failed experiments (merchant price recommendations that solved nothing) versus measurable wins (size recommendations that cut returns). His advice for teams pressured to "do AI": treat it like normal product development, iterate fast using modern tools, and stop experiments that don't deliver customer value. Plus: why the old career advice to "stay at one company forever" is now the riskiest move you can make. Connect with Jan:LinkedIn: https://www.linkedin.com/in/jan-hegewald/ [https://www.linkedin.com/in/jan-hegewald/]Link to the tshirt: https://www.amazon.de/dp/B0FBKJJX7S [https://www.amazon.de/dp/B0FBKJJX7S] Connect with me:LinkedIn: https://www.linkedin.com/in/christianbarra/ [https://www.linkedin.com/in/christianbarra/] Check out our awesome sponsor, dearmachines.com, QA AI Agents for Continuous Testing: https://www.dearmachines.com [https://www.dearmachines.com]

From IC to CTO: The Journey Through PagerDuty, Grammarly, and Building Breakfast with Roman Skvirskyi

⁠In this episode, I sit down with Roman, CTO of Breakfast, a social tech startup fostering offline human connections in Lisbon.⁠ Roman brings over a decade of tech experience, having navigated the trenches from Rails engineer to engineering manager at companies like Loca (later acquired by Capgemini) and Grammarly, before taking the helm as CTO at an early-stage startup. What We Cover The Consulting to Product Journey: We explore Roman's seven-year tenure at Loca, where he grew a PagerDuty account from 3 to 25 engineers, and his transition to Grammarly's 700-person organization. Roman candidly shares the mindset shift required when moving from keeping external customers happy to truly understanding internal users and being proactive about product decisions. Platform Team Transformation: Roman details how he transformed a critical feature team at Grammarly, one handling APIs for all applications, into a proper platform team. We discuss tackling dependencies, managing a Slack channel with 400 people asking questions, and the importance of clarifying team charter and identity. Startup CTO Challenges: We dive into the realities of leading a 4-person engineering team at Breakfast, juggling everything from hands-on backend development to architecture decisions and team mentoring. Roman shares insights on hiring for "the flame," prioritizing curiosity and drive over pure technical skills, and managing the lack of defensive layers that protect against burnout in larger companies. Leadership and Authenticity: Throughout our conversation, Roman emphasizes emotional intelligence, candor, and self-awareness as foundations of good leadership. He shares his daily meditation practice using the Waking Up app and how it helps him stay present and connected to objective reality. Connect with Roman: LinkedIn: https://www.linkedin.com/in/roman-skvirskyi/ Connect with me: LinkedIn: https://www.linkedin.com/in/christianbarra/ Check out our awesome sponsor, dearmachines.com, QA AI Agents for Continuous Testing: https://www.dearmachines.com

Is the Tech Talent War Really Over? A chat with Iwan Gulenko

Iwan Gulenk, founder of a Swiss tech recruitment agency, maps the new reality of tech hiring What’s inside: 1) Market reset and hiring rigor: After years of growth peaking in 2021, 2024 hammered agencies and slowed decisions. Inbound has exploded (think LinkedIn Easy Apply), time-to-fill is unpredictable, and exact-match hiring dominates. Iwan’s “life partner” analogy underscores the variability and risk focus. 2) Remote/hybrid onboarding realities: Juniors and mid-levels suffer without close guidance. Practical fixes: three short daily check-ins (morning/afternoon/evening) and paying for tooling that prevents no-shows and miscommunication—small frictions compound into failed ramp-ups. 3) Nearshoring’s new normal in Switzerland: SMEs now lose business without a nearshore bench—one 20-person firm did. Still, German/Swiss German needs and client culture shape who actually works. Meanwhile, cloud transformations continue, but cloud bills are biting; one client saw costs triple, creating demand for cost optimization. 4) Senior candidate playbook: Treat your resume as a sales document. The top third should signal location, phone, language, role seniority, tech stack, and outcomes. Don’t omit job titles. Read the ad—his team hid “Swiss cheese” to test attention, and many missed it. Freelancers moving to FTE must credibly explain long-term intent. Why it matters: Whether you’re hiring or job-hunting, the easy-money era is over. With innovation labs being cut and 2025 budgets still tight, advantage goes to teams that design thoughtful setups (including nearshore) and candidates who present crisply and execute. AI helps as tooling, but the human nuance still closes the deal. Connect with Iwan: - LinkedIn: https://www.linkedin.com/in/iwan-gulenko/ Connect with me: - LinkedIn: https://www.linkedin.com/in/christianbarra/ Check out our awesome sponsor, dearmachines.com, QA AI Agents for Continuous Testing: ⁠https://www.dearmachines.com⁠ [https://www.dearmachines.com]

Why Your Prompts Matter More Than You Think with Giuseppe Birardi

Giuseppe Birardi is CTO of Orma Lab, an Italian consultancy focused on R&D and industrial AI. A former researcher turned PM and developer, he also co-organizes the Python Bar community. What we cover: - The geometry of meaning: Embeddings turn words into vectors with direction and magnitude, allowing context to “pull” meanings (money-bank vs. river-bank). Dimensionality reduction compresses co-occurrence statistics into 500–700D spaces where vectors become transformative, not static points. - Inside transformers: Multi-head attention re-weights tokens to resolve ambiguity across a sequence, then MLP layers with ReLU activations “fold” the space—think approximating a circle with linear cuts after repeated folds—so nonlinear problems become linearly separable. - Prompt engineering as activation: How phrasing can turn on skills learned during training (e.g., TL;DR for summarization, “step by step” for task decomposition). Why chain-of-thought often simulates reasoning and can still hallucinate. - Probing the latent space: From mechanistic interpretability to feature-level observability in open models (e.g., GemmaScope) and why steering features is promising but not yet turnkey for production. Concrete example: apparent “decryption” often reflects seen patterns (like Caesar shifts) rather than true cryptanalysis. Why it matters: If you’re building RAG or agentic applications, these mental models help you design better prompts, set up experiments, choose SOTA models first, and then optimize cost/latency. Giuseppe also shares how Italian firms—via soft-finance-backed R&D—are moving real AI products into production across domains. If you want to learn more about the geometry of the latent space: https://www.lesswrong.com/posts/nfGZtKzz8WzxF3MAs/on-the-geometrical-nature-of-insight [https://www.lesswrong.com/posts/nfGZtKzz8WzxF3MAs/on-the-geometrical-nature-of-insight] Connect with Giuseppe: - LinkedIn: https://www.linkedin.com/in/giuseppe-birardi-18a7b011/ Connect with me: - LinkedIn: https://www.linkedin.com/in/christianbarra/ Check out our awesome sponsor, dearmachines.com, QA AI Agents for Continuous Testing: ⁠https://www.dearmachines.com⁠ [https://www.dearmachines.com]