What We're Getting Wrong About Running Experiments

How Top Companies Are Evolving Experimentation in 2025

What’s next for experimentation in 2025? We’re breaking down insights from the biggest names in tech—Netflix, Amazon, Meta, and more. Fresh from top experimentation conferences, we’ve identified three major trends set to shape the future of experimentation.  In this Episode: * Shifting toward self-serve capabilities for less-technical end users * The rise of automated "set it and forget it" experiments * Rethinking statistical guidelines to accelerate innovation Connect with us:  Ryan on LinkedIn [https://www.linkedin.com/in/ryanlucht/] Chetan on LinkedIn [https://www.linkedin.com/in/chetanvsharma/]

What We're Getting Wrong About Running Experiments

Eppo Head of Product Eric Metelka took to the stage in San Francisco at the 2024 Lenny & Friends Summit to share some big insights on where product teams are making mistakes in their experimentation programs. Drawing on case studies from industry leaders like Netflix, MSN and Booking.com, Eric explores how to rebuild trust in your data, accelerate experiment velocity, and foster a culture that thrives on bold, high-impact innovation.  In this Episode: * Discover how rigorous science ensures reliable results and builds trust with stakeholders. * Learn the secret to running faster, more impactful experiments using self-service tools. * Explore why bold changes drive growth—and how to manage risks with guardrails.

4 Stages of Experiment Maturity

Data-driven is the buzzword of the day—but what are the strategies that separate top-tier, data-driven companies from the rest? From chaotic, one-off tests to fully automated systems, we share the four stages that drive scalable growth, speed up learning, and make decision-making a breeze. In this episode, Lukas Goetz-Weiss, joins Ryan Lucht to explore each maturity stage, the common roadblocks, and the practical steps any business can take to level up. In this Episode: 1. Why most companies start with isolated, intuition-driven tests and the pitfalls of this chaotic approach. 2. Why companies gain structure but struggle with resource bottlenecks and decision-making delays. 3. How to build trust in experiment results, empower teams, and avoid pitfalls like p-hacking and underpowered tests. 4. Examples of companies that turn experimentation into a seamless, strategic advantage.

5 places to generate GREAT A/B test ideas

"When in doubt, test it out!" In today's episode of Outperform, we uncover why the best teams thrive on a culture of testing everything— powerful metrics like segment-level differences and the dynamic balance between best practices and bold experiments, showcasing why both small tweaks and big swings matter. Plus, how a well-balanced experiment portfolio and savvy management can supercharge your path to strategic innovation! In this Episode:  * Discover five unconventional methods for generating powerful A/B test hypotheses. * Learn how to harness historical company insights, customer feedback, and UX best practices to drive impactful business outcomes. * Explore the significance of a balanced experimentation portfolio, blending big and small bets to effectively transform your business and maintain a competitive edge.

3 Challenges in Creating a Culture of Experimentation

The power of experimentation is never clearer than when small changes spark massive breakthroughs. But how do you get your teams convinced to start testing many small changes in the first place?  We explore some common roadblocks that stand in the way of world-class cultures of experimentation. Ryan, Che, and Sven share insider insights from companies like Airbnb and Stitch Fix to give you actionable ideas and strategies in building your own experimentation program - and career. In this Episode:  * Taking calculated risks and consistently experimenting are key to achieving breakthroughs and driving success. * Standardizing processes and metrics minimizes friction, making it easier for teams to run experiments independently. * Leadership is essential in building a culture of experimentation, where successes are celebrated and failures are seen as valuable learning opportunities.