From Weeks to Minutes: How Prompt-Based Experimentation Changes Everything

Best Buy Experimentation: Ads vs. Conversion, Guardrail Metrics, and the “Job of the Page” (Kevin Washington)

What does experimentation look like when you’re optimising one of the biggest retail sites in the world? In this episode, we talk with Kevin Washington (former Director of Product Management over Experimentation at Best Buy) about how large e-commerce teams think about testing when revenue comes from multiple directions: purchase conversion, retail media, and marketplace growth. We cover: * The “job of the page” mindset (homepage vs. PLP vs. PDP) * Choosing primary metrics vs. guardrails (and why “purchase conversion” can mislead) * How ad placements can cannibalise core retail performance * Why short-term revenue wins can quietly damage long-term trust and LTV * Non-endemic ads, emergency testing, and what “passes the smell test” Note: This episode includes a screen-share walkthrough — watch the video version on YouTube for the full experience. Bonus: Kevin will be speaking at UNITE Summit.

From Weeks to Minutes: How Prompt-Based Experimentation Changes Everything

In this episode of the Conversion Podcast, Matt is joined by Colin, Vice President of North America at Kameleoon, to explore prompt-based experimentation (PBX) and how AI is transforming the way experiments are built and launched. Colin breaks down what PBX is, how it connects AI-driven “vibe coding” with experimentation, and why the build phase has long been the biggest bottleneck for experimentation teams. They discuss how PBX allows teams to create variants, tests, and prototypes directly on live websites using natural-language prompts, cutting build time from weeks or days down to minutes. The episode also covers where PBX fits into the experimentation lifecycle, the importance of keeping humans in the loop, current limitations of the approach, and what adoption looks like for teams just getting started. They close with a discussion of the Prompt-a-thon, an open challenge designed to explore how far prompt-based experimentation can be pushed. * What is prompt-based experimentation (PBX) is * How PBX differs from AI coding and vibe-coding tools * Why building experiments is the biggest friction point * Human-in-the-loop experimentation with AI * Adoption tips and limitations * The Prompt-a-thon

Inside Minecraft's Experimentation Strategy Part 2: Experimentation Adoption, and AI in the Product Lifecycle

Setting the scene for this one and diving right in. Tim talks about successful experimentation being cultivated, not designed or manufactured and why building adoption for experimentation (or any organisational change) comes down to a simple question: are you on the soapbox or are you driving the bus? Tim breaks down why correcting people (“you’re doing it wrong”) creates defensiveness, why the “soapbox” approach is the shortcut, and how the “bus ride” approach builds sustainable adoption. He shares an example from Minecraft, walking through a product lifecycle approach, observation, theory, test, bringing teams along for research, insights, solutioning tied to OKRs, prioritisation, experimentation, analysis, and implementation. They then shift into AI and experimentation: where Tim sees real value today (speed from large datasets to insights), where he’s sceptical (AI-generated high-fidelity designs and experiment variations), and where opportunities may be strongest (analytics, projections, predictive modelling, multi-armed bandits, variance reduction, and qualifying/disqualifying users). They close with Tim sharing why he opened time slots for people to chat about experimentation, why he was encouraged by his manager, and why LinkedIn is the best way to connect. What’s covered (from the conversation): * “Soapbox vs bus ride” and sustainable adoption * Defensiveness, reactance, and avoiding “shaming” in change efforts * Minecraft example: bringing teams through the product lifecycle with experimentation * Separating observation → theory → test * Tying solutioning to OKRs and prioritising experiments * AI in discovery: summarising large qualitative (and quantitative) datasets quickly * Spot-checking + context pitfalls (e.g., “creeper,” profanity interpreted as negative) * Prompt templates vs AI agents with guardrails * Scepticism on AI for full-fidelity designs and experiment variations * AI/ML in analytics: sample size, test duration, forecasting, bandits, variance reduction * Community learning + LinkedIn as the main contact channel

Inside Minecraft's Experimentation Strategy: Driving Measurable Growth with Tim Mehta

In this episode, Matt sits down with Tim Mehta, Experimentation PM at Minecraft, to discuss how one of the world's biggest games approaches growth and testing. Tim shares his journey from agency life to big tech (Microsoft, Lucid) and reveals the inner workings of Minecraft's experimentation program. We dive deep into:• The "Spillover Effect": Unique challenges of A/B testing in a multiplayer game environment and how to solve them.• Delivery vs. Growth Mindset: Why simply shipping features isn't enough and how to shift focus to measurable outcomes.• The Ladder of Evidence: Why tests backed by user research have a 50% success rate compared to the industry average.• 3 Pillars of Experimentation: Trustworthiness, Velocity, and Democratization—and why trust must always come first. Whether you're a Product Manager or Growth Marketer, Tim's insights on building a culture of evidence-based decision-making are invaluable.

Inside Adobe’s AI Experimentation Accelerator

Join host Matt Wright with Brent Kostak and David Arbour from Adobe as they reveal how the new Experimentation Accelerator is transforming digital experimentation through AI automation, causal insights, and agentic workflows. Learn how Adobe is helping brands scale testing velocity, uncover deeper experiment insights, and unify teams across marketing, product, and analytics. Featuring early beta results, real-world use cases, and practical advice for adopting AI-driven experimentation successfully.