In this episode of the Conversion Podcast, Matt is joined by Colin, Vice President of North America at Kameleoon, to explore prompt-based experimentation (PBX) and how AI is transforming the way experiments are built and launched.

Colin breaks down what PBX is, how it connects AI-driven “vibe coding” with experimentation, and why the build phase has long been the biggest bottleneck for experimentation teams. They discuss how PBX allows teams to create variants, tests, and prototypes directly on live websites using natural-language prompts, cutting build time from weeks or days down to minutes.

The episode also covers where PBX fits into the experimentation lifecycle, the importance of keeping humans in the loop, current limitations of the approach, and what adoption looks like for teams just getting started. They close with a discussion of the Prompt-a-thon, an open challenge designed to explore how far prompt-based experimentation can be pushed.

• What is prompt-based experimentation (PBX) is

• How PBX differs from AI coding and vibe-coding tools

• Why building experiments is the biggest friction point

• Human-in-the-loop experimentation with AI

• Adoption tips and limitations

• The Prompt-a-thon

Setting the scene for this one and diving right in. Tim talks about successful experimentation being cultivated, not designed or manufactured and why building adoption for experimentation (or any organisational change) comes down to a simple question: are you on the soapbox or are you driving the bus?

Tim breaks down why correcting people (“you’re doing it wrong”) creates defensiveness, why the “soapbox” approach is the shortcut, and how the “bus ride” approach builds sustainable adoption. He shares an example from Minecraft, walking through a product lifecycle approach, observation, theory, test, bringing teams along for research, insights, solutioning tied to OKRs, prioritisation, experimentation, analysis, and implementation.

They then shift into AI and experimentation: where Tim sees real value today (speed from large datasets to insights), where he’s sceptical (AI-generated high-fidelity designs and experiment variations), and where opportunities may be strongest (analytics, projections, predictive modelling, multi-armed bandits, variance reduction, and qualifying/disqualifying users).

They close with Tim sharing why he opened time slots for people to chat about experimentation, why he was encouraged by his manager, and why LinkedIn is the best way to connect.

What’s covered (from the conversation):

• “Soapbox vs bus ride” and sustainable adoption

• Defensiveness, reactance, and avoiding “shaming” in change efforts

• Minecraft example: bringing teams through the product lifecycle with experimentation

• Separating observation → theory → test

• Tying solutioning to OKRs and prioritising experiments

• AI in discovery: summarising large qualitative (and quantitative) datasets quickly

• Spot-checking + context pitfalls (e.g., “creeper,” profanity interpreted as negative)

• Prompt templates vs AI agents with guardrails

• Scepticism on AI for full-fidelity designs and experiment variations

• AI/ML in analytics: sample size, test duration, forecasting, bandits, variance reduction

• Community learning + LinkedIn as the main contact channel

In this episode, Matt sits down with Tim Mehta, Experimentation PM at Minecraft, to discuss how one of the world's biggest games approaches growth and testing. Tim shares his journey from agency life to big tech (Microsoft, Lucid) and reveals the inner workings of Minecraft's experimentation program.

We dive deep into:• The "Spillover Effect": Unique challenges of A/B testing in a multiplayer game environment and how to solve them.• Delivery vs. Growth Mindset: Why simply shipping features isn't enough and how to shift focus to measurable outcomes.• The Ladder of Evidence: Why tests backed by user research have a 50% success rate compared to the industry average.• 3 Pillars of Experimentation: Trustworthiness, Velocity, and Democratization—and why trust must always come first.

Whether you're a Product Manager or Growth Marketer, Tim's insights on building a culture of evidence-based decision-making are invaluable.