Experiment Nation: The Podcast – Details, episodes & analysis

In this episode of Experiment Nation, host Jason sits down with Namata Agarwal, a seasoned product designer with over 17 years of experience building digital products, websites, and apps.Together, they explore:- Ethical testing: how to balance user needs with business goals.- Leaks vs. puddles framework: a practical way to identify urgent vs. non-urgent product problems.- The role of design in CRO and experimentation.- Frameworks, guardrails, and collaboration strategies for building an experimentation culture.Whether you’re a CRO specialist, product manager, or experimentation nerd, this episode is packed with actionable insights you can bring to your own team.👉 Don’t forget to like, comment, and subscribe to stay updated on the latest in experimentation and growth!📖 Check out Prove It or Lose It: The (Mostly) No-Nonsense Guide to Surviving Experimentation Program Drama .⏱️ Chapters with Timestamps0:00 – Why design must be involved early0:22 – Welcome & intro to Jason0:43 – Guest intro: Namata Agarwal0:55 – Topics: Ethical testing & Leaks vs. Puddles1:18 – What is ethical testing?2:22 – Balancing business needs vs. user needs2:54 – Handling stakeholder pushback4:01 – Misaligned metrics and real solutions5:17 – The Leaks vs. Puddles analogy explained6:32 – Why better hypothesis building matters7:48 – Using Airtable & frameworks for test prioritization8:40 – Why experimentation must be collaborative9:24 – Reframing experiments around the user journey10:00 – Unpredictable user journeys & design challenges11:12 – Involving guest collaborators in experimentation11:50 – Mapping user journeys with Miro & qualitative research12:50 – Jobs To Be Done & other qualitative frameworks14:44 – Turning qualitative insights into hypotheses15:32 – How product design elevates CRO strategies16:11 – The importance of forms in experimentation17:36 – Why design should be included from day one18:20 – Design is not just visuals—it’s about impact19:11 – Governance, responsibility, and inclusivity in testing19:46 – Leak vs. Puddle framework deep dive22:58 – Running collaborative workshops25:45 – The “Stinky Fish” workshop format27:21 – Micro vs. macro frameworks for problem-solving28:05 – Balancing bold ideas vs. safe bets29:19 – Planning tests like an investment portfolio30:13 – Testing across 13 sites at scale30:26 – What Namata would A/B test in daily life31:57 – Art, painting, and creative balance32:42 – Martial arts & keeping calm32:56 – Wrap up & final thoughts33:14 – How to connect with Namata

In this episode of the Experiment Nation Podcast, host Ward Vanestra sits down with Ryan Lux, one of the first team members at Eppo and now Experimentation Evangelist at Datadog.They dive into:What Eppo is and how it helps companies experiment at scaleWhy data warehouses are critical for trustworthy resultsThe recent Datadog acquisition of Eppo and what it means for the futureAdvanced methods like CUPED (noise-cancelling for experiments) and contextual banditsHow to scale experimentation culture and get promoted by driving adoption, efficiency, and trustworthinessWhether you’re a CRO, growth leader, or experimentation enthusiast, this episode is packed with insights from someone who’s been shaping experimentation for over a decade.📘 Check out Experiment Nation’s new book: Prove It or Lose It: The Mostly No-Nonsense Guide to Surviving Experimentation Program Drama → experimentnation.com⏱️ Chapters0:00 – Why win rates don’t really increase0:20 – Introduction to Ryan Lucht & Eppo1:20 – What Eppo does and who it’s for2:18 – Experimentation at smaller companies3:18 – Datadog’s acquisition of Eppo3:59 – The data warehouse advantage6:03 – CUPED explained: noise-cancelling for experiments9:13 – Segments, personalization & pitfalls11:09 – Heterogeneous treatment effects (HTEs)12:06 – Personalization with contextual bandits14:00 – Why Datadog acquired Eppo16:22 – Data observability & product analytics17:52 – Experimentation and site performance19:46 – Content, culture, and experimentation career paths22:10 – Every experiment is valuable (win, lose, or flat)24:17 – OKRs for experimentation leaders25:47 – Three key OKRs: adoption, efficiency, trustworthiness28:14 – Knowledge building and learning plans29:34 – The vision: experiments as the ultimate insight system30:24 – Ryan’s Substack “Everything is an Experiment”31:02 – Closing thoughts & where to connect

Experimentation is a journey, not a destination. Don’t expect to build a culture of experimentation overnight.
Cross-functional teams are critical. Breaking down silos and bringing together engineers, product managers, and designers will create a unified vision and drive faster iteration.
Leadership buy-in is essential. A top-down approach to embedding experimentation ensures everyone is on board and resources are allocated strategically.
Don’t underestimate the importance of organizational rhythms. Establish clear processes and structures to ensure smooth collaboration and avoid friction.
Embrace the growing pains. Change is rarely easy. Be prepared for challenges and celebrate milestones along the way.

Florent Buisson is a decade-long behavioral economist who published a practical guide to behavioral data analysis in business. It's the book he wished he had when he started in analytics.

In this episode, he shares:

Plus other statistical concepts explained in a way even a five-year-old could understand.Why intuition can actually be a good thing in data analytics

If you need to know R or Python to understand your users

Why and when causal analytics are important

--

Florent Buisson is a behavioral economist with 10 years of experience in business, analytics, and behavioral science. He most recently started and led for four years the behavioral science team of one of the largest insurance companies in the US; he then published Behavioral Data Analysis with R and Python (O'Reilly), the book he wished he had had when he started. Florent holds a PhD in behavioral economics from the Sorbonne university in Paris.

Find Florent on LinkedIn: linkedin.com/in/florentbuisson

Read Florent's content on Medium: medium.com/@florent.buisson

About this series: Rommil Santiago, the founder of Experiment Nation shares his most recent thoughts about the industry and the craft of Experimentation in less than 5 minutes.

A/B Testing can be intimidating to the uninitiated, but it doesn’t have to be! I want to introduce testing concepts via a real-world scenario that we can all relate to: Dessert. I’ll cover what an MVT is, hypotheses, independent variables, dependent variables, results (of course), and design flaws. And if you need any more convincing, I have four words for you: Monte Cristo Bread Pudding.

Matt Green and Moshe Mikanovsky of Product for Product podcast help you navigate the plethora of A/B testing tools out there.

Whether you're in product or marketing, listen for a beginner-friendly comparison of tools like Google Optimize, Optimizely, and more.

**This episode, nor those of Product for Product, are endorsements of any of the discussed tools.**

Listen to Product for Product: spreaker.com/show/product-for-product

Find Matt on LinkedIn: /in/mattgreenanalytics

Find Moshe on LinkedIn: /in/mikanovsky

You can do better than testing button colors, but how do you decide what to test when, and what type of test to run? Roadmapping. This session teaches you to be the GPS navigator, helping your team optimize experimentation routes.  On this journey, you will identify how each path can drive desired impacts by turning a formidable trek into a series of attainable milestones.   

Presented by 

• Kenya Davis, Microsoft Digital Analytics


• Planning Program Manager, and Evolytics Decision Science Director Krissy Tripp.

Experiment Nation: The Podcast is a show on which we share Interviews with Experimenters from around the world, opinion pieces, and recent presentations with you - the experimentation community. We hope you enjoy the show, let’s get to it.

If you enjoy our content, consider signing up for Experiment Nation's bi-weekly newsletter here: https://buff.ly/31qO0AQ

Tracy is a Toronto-based growth marketer with a knack for conversion optimization. In a past life she’s QA tested “adult” eBooks and helped make it easier for Canadians to write their wills online. Tracy is also one of the co-hosts of the Experiment Nation podcast where she chats with optimizers at the top of their game.

Beyond the 9-5, you can catch her knitting, crushing empires in Civ6, or napping with her cats.