Explore every episode of the podcast Experiment Nation: The Podcast

What do you do when an experiment tanks conversion rate by 8% in less than a week?In this episode, Gerda and Rommil break down real CRO and experimentation scenarios submitted by the community, including failed tests, executive pressure, guardrail metrics, knowledge management, developer friction, and why experimentation teams always seem to end up back in spreadsheets.Check out: http://katsed.com/We also discuss:

* Root cause analysis for failed experiments

* How to calm leadership during testing disasters

* Guardrail metrics and control limits

* Why experimentation knowledge banks become messy

* Airtable vs spreadsheets vs enterprise tools

* Why developers and designers struggle with experimentation culture

* How to onboard technical teams into testing workflows

A practical conversation for experimentation leads, growth teams, product managers, CRO specialists, and anyone building an experimentation culture.Chapters00:00 Burger Menu Debate: Left vs Right01:28 Mystery Meat Navigation & UX History03:03 Why Modern Phones Break UX05:08 Community CRO Scenarios Begin06:18 Experiment Tanks Conversion by 8%07:33 Root Cause Analysis & Leadership Pressure09:17 QA, Tracking Errors & Failed Tests10:03 Building a CRO Analysis Tool12:18 Probabilistic Root Cause Detection14:26 Managing Panic During Bad Experiments15:21 Guardrail Metrics Explained16:40 Using Control Limits in Experimentation18:18 Building an Experimentation Knowledge Bank20:05 Why You Don’t Need AI for Everything21:08 Why Process Changes Create Admin Pain23:08 Migrating Between Experimentation Tools24:31 Why Teams Always Go Back to Spreadsheets26:50 CRO Teams vs Developers & Designers29:09 Why Early Experiments Should Be Simple30:41 Why Developers Hate “Throwaway Code”32:16 Designers vs Experiment Constraints33:02 Outro & Next Community Questions

Most teams say they “do research”… but very few actually do it well.In this episode, we break down what real customer research looks like in CRO and growth. From analyzing 200+ survey responses manually, to why AI still struggles with qualitative data, to the politics that slow everything down.We also get into:Why AI fails at nuanced research (and what it misses)How to actually analyze open-ended survey responsesThe biggest mistakes teams make with surveysWhy companies avoid talking to customers (and why that’s a red flag)How research becomes a political nightmare in larger orgsWhen you should sample vs analyze everythingHow research fuels better marketing and personalizationAnd yes… we somehow end up talking about billion dollars and laundry.If you’re in CRO, growth, product, or marketing, this is the stuff people don’t usually say out loud.👇 Timestamps00:00 Why content actually drives opportunities03:15 What “research” really means04:00 How many responses you actually need06:00 Why AI fails at qualitative analysis07:30 When to sample vs analyze everything09:00 Using research for marketing copy10:00 Why not talking to customers is a red flag11:00 Survey politics and internal resistance13:30 Bad survey questions and bias16:00 Personalization vs CRO (what’s the difference?)22:30 Personalizing outreach properly26:30 Why research projects spiral out of control29:00 How to actually make research a habit31:00 How to ship research faster (timeboxing)34:00 “Business jiu-jitsu” and fake urgency35:30 Final thoughts (and laundry)

In this episode, we dive into the realities of CRO (Conversion Rate Optimization) today — from AI anxiety to experimentation culture and how teams should actually be structured.

We start with a surprisingly philosophical take on photography and overthinking, then quickly shift into what’s really happening in the CRO and AI space right now.

We discuss:

* Why AI might be making everything look the same

* The growing fear (and noise) around AI in CRO

* How to structure CRO teams: centralized vs hybrid vs embedded

* Why experimentation still struggles with a “nice-to-have” perception

* What actually builds a strong experimentation culture

* How individual contributors can influence culture without authority

* And how to position CRO as a true growth engine, not just testing

If you work in growth, product, or experimentation, this is a candid look at what’s changing and what actually matters.

Chapters

0:00 – Photography, overthinking, and creative process1:26 – Back to CRO: what’s happening in the industry2:07 – AI concerns and job market anxiety2:45 – The “everything looks the same” AI problem4:00 – Too many AI tools, no real differentiation5:38 – What people are really asking about AI6:23 – AI fear vs reality (and layoffs)7:08 – Fear-mongering vs signal8:00 – Fear and decision-making9:00 – CRO team structures: centralized, hybrid, embedded9:31 – Why centralized works early10:08 – When to move to hybrid10:29 – Why embedded is hard11:04 – The role of leadership buy-in12:13 – Why experimentation gets cut first13:00 – CRO’s “nice-to-have” branding problem14:00 – Why CRO is easy to underestimate15:45 – Future-proofing: combine CRO with other skills16:54 – Building experimentation culture (the “laundry” analogy)18:19 – How to destroy experimentation culture19:23 – Why authority and buy-in matter20:25 – How ICs can influence culture21:34 – Reframing experimentation as empowerment23:09 – Becoming a catalyst across teams24:46 – Confidence and presence in CRO roles25:59 – CRO vs Growth: what’s the difference?27:24 – Where CRO fits inside growth teams28:02 – Moving CRO upstream into systems thinking29:00 – Closing thoughts

In this episode, we answer real, unfiltered questions from the CRO and experimentation community. We dig into how far AI can realistically help with experimentation, why revenue metrics like ARPU are a nightmare for MDE calculations, how to handle sketchy interview “audits,” and why ROI conversations never seem to end.No hype. No magic bullets. Just honest perspectives from people actually doing the work.Topics include:

* Where AI helps (and quietly breaks) CRO workflows

* How to think about MDEs for continuous metrics like ARPU

* Making time for research when you’re already drowning

* When “auditing a site” crosses into spec work

* Volume hiring for CRO roles

* Escaping (or not) the endless ROI loop

* Why some CRO agencies don’t have designers

Drop your own questions in the comments or check the links below to submit anonymously.- Ask us a question here: https://docs.google.com/forms/d/e/1FAIpQLSd8P5yCsO5NB6WPe204k1WExhH1Bs8w6nZxPRRn-BiAGozpaQ/viewform- Visit Koalatative here: https://www.youtube.com/@koalatative⏱️ Chapters00:00 – Intro & warm-up04:34 – How far can AI actually help in delivering CRO?11:19 – How do experimentation experts estimate MDE for continuous metrics like ARPU?15:28 – How do you make the most of research time when there’s no time to do anything?19:29 – How do you handle being asked to audit a site as part of a CRO interview?24:20 – How do you approach CRO in volume recruitment?28:21 – How do you get out of the endless ROI loop with stakeholders (and stop saying “CRO”)?31:46 – Why do some CRO agencies not have designers on the team?33:47 – Outro & announcements#cro #experimentation #abtesting #aiincro #growth #productmanagement #conversionrateoptimization #experimentationculture #careerintech #experimentationnation

A real conversation about why A/B testing doesn’t magically fix broken businesses.We break down the biggest CRO myths, why copying case studies fails, how experimentation tools really get chosen, and what actually causes friction between CRO, UX, and research teams.For CROs, experimenters, product, growth, UX, and analytics leaders.1:38 The biggest myth in A/B testing3:00 The one-person CRO team fallacy4:34 Why copying tests never works10:44 Choosing an experimentation platform14:39 Working with “difficult” clients

In this episode, we go off-script and cover everything from CRO strategy to existential questions.We talk through:How to recover after being shaken at workWhy long-form content still matters in experimentationCRO differences between B2B, SaaS, and ecommerceWhy website redesigns without A/B testing backfireHow beginners can gain real CRO experienceWhat it actually takes to get people to listen to dataAnd yes… the meaning of life (besides 42)This episode is less about “perfect answers” and more about how CRO, experimentation, and work actually play out in the real world.If you work in experimentation, product, growth, or analytics, this one will feel uncomfortably familiar.Chapters00:00 – How to recover after being shaken at workReflection, time, and physical reset after tough moments02:10 – End-of-year reflection and career resetsBurnout, misfit roles, and rediscovering intrinsic motivation04:25 – Why short-form content fails complex industriesAttention, doomscrolling, and why long-form still matters07:18 – Listener Q&A begins: CRO across business modelsB2B vs SaaS vs ecommerce testing realities12:02 – The hidden buyer problem in B2B CROResearchers vs decision-makers and misaligned incentives13:44 – Website redesigns without A/B testingPolitics, risk, and how to introduce testing anyway18:16 – How beginners can get real CRO experienceSide projects, free tools, and hands-on learning23:46 – The meaning of life (other than 42)Creating your own meaning rather than finding one26:58 – Getting people to listen to data at workInfluence, likability, and organizational dynamics31:24 – Culture fit, authority, and when to move onWhy sometimes it’s not you, it’s the environment32:23 – Wrap-up and next episode teaser

Koalatative's Gerda Thomas and Experiment Nation's Rommil Santiago answer questions from the CRO community.Chapters3:35 – Is there room in CRO for AI skeptics anymore?4:58 – How can I feel like I'm using my CRO skills for good?11:29 – AI is boring and speeding up the decline of civilization. Agree or disagree?15:22 – How do I tell my micromanaging manager to shut up?18:43 – Why are you the way that you are?23:48 – How do you share a complete story with clients so they actually hear both sides of the research?26:39 – How do you calculate ROI for my CRO program for a client?29:17 – When you have no ideas on what test to run where do you get ideas?

Koalatative's Gerda Thomas and Experiment Nation's Rommil Santiago answer CRO questions from the community.Chapters:1:51 – What's the one thing people on internet still refuse to automate or use AI for that you think they should?”5:07 – How do you measure the impact of your CRO program over time?10:37 – What to do when leadership has a win at all cost mentality, choosing short-term wins over long-term ones. How can a proper experimentation program thrive?15:27 – How should checks and balances happen in an organization on data integrity? Who should own those roles?20:10 – Should a testing tool have a warning that says, 'Your win rate is too high. Check your data type warning.'?22:49 – It seems like every day it gets harder to land a quality job in CRO with some of my colleagues taking up to a year to bounce back from a layoff. Is it worth joining the field anymore?

Koalatative's Gerda Thomas and Experiment Nation's Rommil Santiago answer your CRO questionsPart 1 of 3Chapters:8:05 – What is the right balance between big tests versus small and medium tests?12:47 – How many iterations per test will you run knowing the first test was negative but still see some relevant data toward the hypothesis?14:32 – In small traffic sites, is it best to run one A/B test or launch a test under pre-post test assumption?17:40 – What’s the one thing most people in experimentation still refuse to automate or use AI for that you think they should do it?-----Catch Gerda on Koalatative's channel here:

In this episode of Experiment Nation, host Jason Gossit sits down with Ameliano Blanca, a senior growth and product leader with 10+ years of experience driving revenue through data-driven experimentation.They dive deep into:How to build experimentation programs from the ground upThe balance between qualitative and quantitative insightsExperimentation in startups vs. mature organizationsCRO frameworks and the power of iterationHow to avoid "random testing mode" and adopt structured frameworksIf you’re a growth, product, or CRO professional, this episode is packed with actionable insights to help you scale experimentation programs, improve conversion rates, and create sustainable growth strategies.👉 Get our new book: Prove It or Lose It – The (Mostly) No-Nonsense Guide to Surviving Experimentation Program Drama at experimentnation.com and all major bookstores.📌 Chapters:0:00 – Why tests should be rerun after 12 months0:19 – Welcome & introduction with Jason Gossit0:35 – Meet Ameliano Blanca: growth & product leader1:32 – First steps in building experimentation programs2:42 – Measuring conversion funnels & revenue impact3:46 – Tools for gathering quantitative & qualitative insights5:05 – Heatmaps, surveys, and finding hidden gems6:00 – Drafting test ideas & prioritization7:19 – Using and adjusting the ICE framework9:50 – The “three strikes” rule for failed experiments11:16 – Experimentation in startups vs. mature companies13:00 – Why rerunning startup tests is critical14:57 – When to pause or rerun losing experiments15:29 – Complementing quantitative data with customer interviews17:16 – Running experiments at scale in larger organizations18:18 – Balancing speed vs. rigor in CRO testing20:04 – Defaulting experiments & tracking long-term results20:11 – CRO frameworks and the power of iteration22:26 – Why iteration is underrated in experimentation23:44 – Adapting frameworks to your company & industry25:18 – Advice for teams stuck in random testing mode27:39 – How CRO perspectives have evolved over 10 years28:39 – Growth, insights, and the rise of experimentation in strategy30:38 – Real-world example: financing simulator boosts conversions

In this episode of Experiment Nation, host Jason sits down with Namata Agarwal, a seasoned product designer with over 17 years of experience building digital products, websites, and apps.Together, they explore:- Ethical testing: how to balance user needs with business goals.- Leaks vs. puddles framework: a practical way to identify urgent vs. non-urgent product problems.- The role of design in CRO and experimentation.- Frameworks, guardrails, and collaboration strategies for building an experimentation culture.Whether you’re a CRO specialist, product manager, or experimentation nerd, this episode is packed with actionable insights you can bring to your own team.👉 Don’t forget to like, comment, and subscribe to stay updated on the latest in experimentation and growth!📖 Check out Prove It or Lose It: The (Mostly) No-Nonsense Guide to Surviving Experimentation Program Drama .⏱️ Chapters with Timestamps0:00 – Why design must be involved early0:22 – Welcome & intro to Jason0:43 – Guest intro: Namata Agarwal0:55 – Topics: Ethical testing & Leaks vs. Puddles1:18 – What is ethical testing?2:22 – Balancing business needs vs. user needs2:54 – Handling stakeholder pushback4:01 – Misaligned metrics and real solutions5:17 – The Leaks vs. Puddles analogy explained6:32 – Why better hypothesis building matters7:48 – Using Airtable & frameworks for test prioritization8:40 – Why experimentation must be collaborative9:24 – Reframing experiments around the user journey10:00 – Unpredictable user journeys & design challenges11:12 – Involving guest collaborators in experimentation11:50 – Mapping user journeys with Miro & qualitative research12:50 – Jobs To Be Done & other qualitative frameworks14:44 – Turning qualitative insights into hypotheses15:32 – How product design elevates CRO strategies16:11 – The importance of forms in experimentation17:36 – Why design should be included from day one18:20 – Design is not just visuals—it’s about impact19:11 – Governance, responsibility, and inclusivity in testing19:46 – Leak vs. Puddle framework deep dive22:58 – Running collaborative workshops25:45 – The “Stinky Fish” workshop format27:21 – Micro vs. macro frameworks for problem-solving28:05 – Balancing bold ideas vs. safe bets29:19 – Planning tests like an investment portfolio30:13 – Testing across 13 sites at scale30:26 – What Namata would A/B test in daily life31:57 – Art, painting, and creative balance32:42 – Martial arts & keeping calm32:56 – Wrap up & final thoughts33:14 – How to connect with Namata

In this episode of the Experiment Nation Podcast, host Ward Vanestra sits down with Ryan Lux, one of the first team members at Eppo and now Experimentation Evangelist at Datadog.They dive into:What Eppo is and how it helps companies experiment at scaleWhy data warehouses are critical for trustworthy resultsThe recent Datadog acquisition of Eppo and what it means for the futureAdvanced methods like CUPED (noise-cancelling for experiments) and contextual banditsHow to scale experimentation culture and get promoted by driving adoption, efficiency, and trustworthinessWhether you’re a CRO, growth leader, or experimentation enthusiast, this episode is packed with insights from someone who’s been shaping experimentation for over a decade.📘 Check out Experiment Nation’s new book: Prove It or Lose It: The Mostly No-Nonsense Guide to Surviving Experimentation Program Drama → experimentnation.com⏱️ Chapters0:00 – Why win rates don’t really increase0:20 – Introduction to Ryan Lucht & Eppo1:20 – What Eppo does and who it’s for2:18 – Experimentation at smaller companies3:18 – Datadog’s acquisition of Eppo3:59 – The data warehouse advantage6:03 – CUPED explained: noise-cancelling for experiments9:13 – Segments, personalization & pitfalls11:09 – Heterogeneous treatment effects (HTEs)12:06 – Personalization with contextual bandits14:00 – Why Datadog acquired Eppo16:22 – Data observability & product analytics17:52 – Experimentation and site performance19:46 – Content, culture, and experimentation career paths22:10 – Every experiment is valuable (win, lose, or flat)24:17 – OKRs for experimentation leaders25:47 – Three key OKRs: adoption, efficiency, trustworthiness28:14 – Knowledge building and learning plans29:34 – The vision: experiments as the ultimate insight system30:24 – Ryan’s Substack “Everything is an Experiment”31:02 – Closing thoughts & where to connect

In this episode of the Experiment Nation Podcast, host Richard Joe welcomes Manuel Da Costa, founder of Effective Experiments and now Efestra, to unpack a critical issue in the experimentation industry: the trust gap between practitioners and business decision-makers.🎯 Manuel shares how running more experiments isn’t the goal—making better decisions is. He introduces the Learning Loop, explains why most experimentation programs stall, and gives practical strategies for connecting experimentation insights to executive decisions.🔍 Whether you’re in CRO, product experimentation, or experimentation governance, this is a must-watch for moving beyond vanity metrics and toward real impact.📌 Connect with Manuel:📍 LinkedIn: https://www.linkedin.com/in/manueldacosta/#CRO #Experimentation #ABTesting #EffectiveExperiments #ProductManagement #DigitalMarketingChapter00:00 – Experiments don't matter unless they drive decisions 00:30 – Meet Manuel Da Costa, founder of Effective Experiments 01:45 – Manuel’s journey: from web design to CRO and Lean Startup 03:00 – How experimentation became his product 05:20 – The limitations of spreadsheets in experimentation 06:45 – Building Effective Experiments to streamline the workflow 09:10 – What is the "Trust Gap" in experimentation? 11:30 – Why 88% of experimentation programs aren’t transformative 13:00 – The survival mindset vs. strategic experimentation 16:00 – The echo chamber of experimentation advice 17:40 – Why velocity and win rate are not the real success metrics 19:00 – The problem with how learnings are shared 21:00 – How to communicate insights in one powerful slide 23:00 – Building organizational learning with the “Learning Loop” 25:00 – Step-by-step breakdown of the Learning Loop framework 27:15 – Why experimentation needs executive alignment 29:00 – How to reset a tactical experimentation program 30:40 – Introducing Efestra: a governance platform for experimentation 32:00 – Final thoughts and how to get in touch with Manuel 33:14 – About the book: *Prove It or Lose It*

In this episode, we dive into Speero’s shift from CRO to Experiment-Led Growth, how they structure their testing systems, where to start with a new client, and what makes a good growth bet. Plus, we discuss the power of insights, the rise of EXL meetups and the future of Speero.Chapters----00:00 - Introduction 00:18 - Meet the Hosts & Guest 01:11 - From CRO to Growth Experimentation 02:55 - Building a System for Experimentation 05:00 - Organizational Buy-in and Culture Shift 06:02 - Client Reactions and Collaboration 08:00 - Applying Experimentation Across the Business 10:00 - Strategy vs. Tactics 11:30 - Why Create a Strategy? 13:02 - Making Better Bets with Experimentation 15:05 - Breaking Down Big Initiatives 16:41 - Landing Pages, Messaging & Paid Spend 18:20 - Irreversible Risks and Thought Experiments 20:15 - Research Frameworks: Qual & Quant 21:47 - Copy Testing and Messaging Insights 23:48 - Combining Qualitative and Quantitative Testing 25:49 - Using Ads to Accelerate Learning 26:33 - EXL Meetups: Community and Format 28:00 - How to Host an EXL Meetup 30:04 - Upcoming Global Meetups 31:35 - How to Connect with Paul Randall 32:40 - Outro#cro #experimentnation #abtesting

This episode features Achille Vaulot, a CRO expert, discussing the evolution of conversion rate optimization (CRO) and the steps to building a mature program. He highlights the shift from basic A/B testing to a data-driven, strategic approach encompassing research, thorough analysis (including post-test analysis of both primary and secondary KPIs), and effective client communication. Achille emphasizes the importance of planning for both successful and unsuccessful tests, managing client expectations, and fostering a culture of experimentation within the client’s organization. He stresses the value of qualitative data alongside quantitative results and the importance of aligning CRO strategies with broader business goals. The conversation underscores the crucial role of education and collaboration in scaling a successful CRO program.

Ekaterina (Shpadareva) Gamsriegler, a seasoned growth and marketing expert, shared her insights on mobile app experimentation. Here are 5 key takeaways:

Mobile experimentation is more complex: Release cycles are longer, and dependencies on app versions can slow down learning.
Think beyond the app: Experimentation should encompass ads, store listing pages, and the app itself.
Borrow insights from other areas: Use data from paid advertising to inform store listing optimizations.
Interconnectivity is key: Changes in one area can impact others. Strong stakeholder management is crucial.
Document everything: Detailed documentation helps track results and avoid surprises.
Check out the full conversation on Experiment Nation’s YouTube channel!LinkedIn: https://www.linkedin.com/in/ekaterina-shpadareva-gamsriegler/

#mobilemarketing #appmarketing #experimentation #growthmarketing #experimentnation

Tracy Laranjo shares learnings from years of growing her career with Suchita Srinivasan, here are some key takeaways:

1. CRO vs Experimentation: CRO focuses on driving performance and results, while experimentation centers around decision-making and validating ideas.
2. Agency vs In-House: Tracy’s journey highlights the benefits of agency experience for rapid skill development and exposure to diverse challenges.
3. Building Your Experimental Brand: Don’t be afraid to be yourself online, even if it’s not the typical thought leader persona.
4. Adapting to Constant Change: Focus on fundamentals and building your skills; stay curious and open to new developments, but don’t fear the future.
5. CRO is for Everyone: While AB testing requires scale, CRO encompasses qualitative research and user interviews, making it accessible for businesses of all sizes.

This episode of Experiment Nation features an interview with Dr. Else van der Berg, a product management consultant for B2B SaaS startups and scale-ups. Dr. Else van der Berg discusses her process for helping companies become more outcome-driven, focusing on understanding the ideal customer profile (ICP) and its associated value. She also shares insights on common challenges faced during the testing phase, including participant recruitment, data analysis, and differentiating signal from noise.

Highlights
🎯 Dr. Else van der Berg emphasizes the importance of defining a narrow and well-defined ICP, emphasizing the need to focus on a specific customer segment.
🔎 She believes in a layered approach to testing, using a variety of methods, including interviews, landing page tests, and moderated user tests.
🗣️ Empathy plays a crucial role in interpreting user feedback, recognizing that data interpretation can be subjective and require a “gut feeling” alongside evidence.
📈 Dr. Else van der Berg stresses the importance of finding a mentor within the field and learning on the job to master product management techniques.
👩‍💻 She encourages aspiring product managers to reach out to her via LinkedIn to learn more and potentially connect for career advice.

We recently had the pleasure of chatting with Rishi, a Shopify product expert. Here are 5 key takeaways for optimizing your product pages:

1) Product pages are the Grand Central Station of conversions. They’re where buyers make the decision to buy or leave.
2) Pop-ups aren’t as effective as you think. Focus on valuable content, not annoying interruptions.
3) Content is king (but don’t be afraid to go long!). Detailed product descriptions are more effective than short ones.
4) Build trust above the fold. Establish your company’s credibility before highlighting product features.
5) Understand the filters buyers use. Address common concerns like “too good to be true” to build confidence.
Want to learn more? Check out Rishi’s website, frictionlesscommerce.com!

#ecommerce #productpages #conversionrateoptimization #marketing #experimentation

Black Friday is coming, and the pressure is on. But don’t worry, Ilan Hurwitz has some expert advice to help you navigate the chaos.

Here are five key takeaways from his chat with Tracy Laranjo:

1) Start now! Don’t wait until the last minute to get buy-in and start testing.
2) Speak the language of profit. Showcase the ROI of CRO to your stakeholders, focusing on increasing conversions and profits.
3) Don’t be afraid to experiment. Test offers, product placements, and even email campaigns outside of standard website A/B testing.
4) Focus on the customer journey. Don’t just look at your website. Analyze customer service tickets, reviews, and LTV to identify areas for improvement.
5) Think beyond discounting. Explore creative offers like gift with purchase, cross-sells, and subscription incentives to boost sales.

Get ready to conquer Black Friday!

#BlackFriday #CRO #ecommerce #marketing #experimentation

Experimentation is a journey, not a destination. Don’t expect to build a culture of experimentation overnight.
Cross-functional teams are critical. Breaking down silos and bringing together engineers, product managers, and designers will create a unified vision and drive faster iteration.
Leadership buy-in is essential. A top-down approach to embedding experimentation ensures everyone is on board and resources are allocated strategically.
Don’t underestimate the importance of organizational rhythms. Establish clear processes and structures to ensure smooth collaboration and avoid friction.
Embrace the growing pains. Change is rarely easy. Be prepared for challenges and celebrate milestones along the way.

In this podcast episode of Experiment Nation, hosted by Charlotte Bomford, guest Iqbal Ali, an experienced web designer, researcher, and comic book writer, discusses the concept of solution bias and the importance of identifying underlying problems before diving into solutions. Ali emphasizes using AI to help understand problem landscapes, avoiding hastily jumping to solutions, and improving experimentation processes. The conversation covers techniques like premortems and frameworks to minimize potential issues and ensure experiments address real user problems. Ali highlights the collaborative potential of AI tools in maintaining the connection between problem statements and solutions, advocating for a more thoughtful approach to A/B testing and problem-solving in product development.

Ruben de Boer, a lead consultant at Online Dialogue and the founder of Conversion Ideas, shares his approach to conversion rate optimization (CRO) and experimentation. With 15 years of experience, Ruben emphasizes the importance of data-driven decision-making and understanding human behavior. He highlights the shift from focusing solely on digital products to also optimizing organizational processes. Ruben provides insights into the challenges of implementing experimentation in organizations, the need for higher management support, and the significance of aligning with colleagues' goals and motivations. He advocates for a research-driven approach, continuous feedback, and fostering an experimentation culture within teams to drive better results.

Tymur Donets, a seasoned Growth PM and experimentation consultant, shared some powerful insights during his Experiment Nation talk. Here are five key takeaways:

Growth PMs are holistic: Don’t be siloed! Growth PMs work across the entire customer journey, ensuring marketing and product teams are aligned.
Commercial awareness is key: Both Growth PMs and CROs must understand the business impact of their work and how it contributes to the bottom line.
Empathy is vital: CROs need to understand the pressures their PM colleagues face and proactively offer support.
Leverage your expertise: If you’re a CRO, consider specializing in Growth PM to expand your impact and skillset.
Transparency is key: Clear communication between teams helps everyone understand their roles and collaborate effectively.

Find Tymur on LinkedIn: https://www.linkedin.com/in/tim-donets-2358364a

Chapters:
0:00 - Introduction and The Role of a Growth PM
0:58 - Growth PM vs. Product Manager
2:40 - Tymur’s Journey and How He Became a Growth PM
5:23 - Importance of Business Savvy and Data Literacy
8:30 - Bridging the Gap Between Marketing and Product Teams
12:42 - Experimenters as Facilitators
16:38 - The Complexity of a Product Manager’s Role
19:50 - CRO Responsibilities and Building Relationships with PMs
24:48 - Transitioning from CRO to PM
29:00 - Finding Growth PM Opportunities and Wrap-Up

#growthmarketing #experimentation #cro #productmanagement #collaboration

Kelly Wortham dropped some serious knowledge bombs in our recent Experiment Nation chat. Here’s what stood out:

- CRO isn’t just for marketers: Lawyers, project managers, even developers can thrive in this field. It’s all about a love of learning and adaptability.
- Don’t underestimate the power of mentorship: Kelly has THREE mentors, each focusing on different aspects of her career. Find your own advisors to help you grow!
- Certifications are nice, but experience trumps all: While CXL is great, hands-on testing platform experience is the real deal.
- UX psychology is CRUCIAL: Understand your users’ motivations and behaviors to create truly effective tests.

Simar, CEO and Founder of OptiPhoenix, shared invaluable insights on A/B testing and CRO at Experiment Nation. Here are 5 key takeaways:

Don’t rely solely on macro goals: Micro goals reveal user behavior, driving better testing strategies and informing future hypotheses.
Iterative testing trumps scrapping: Learn from failed tests by adjusting hypotheses and running new experiments, boosting your win rate.
QA is critical, but often neglected: Invest time in thorough QA, ensuring your test is accurate and integrated correctly.
Outsourcing AB test development can be beneficial: Consider outsourcing for faster scale-up, access to expert resources, and reduced development burden.
Prioritize experimentation within your workflow: Make A/B testing an integral part of your product development process, leading to better decisions and faster results.

Avishek Basu Mallick, a Senior PM at CheapOair, talks about:
- Experimentation at scale: Even with high traffic, rigorous experimentation is crucial.
- Idea generation: User research, NPS surveys, and internal stakeholders are valuable sources of experiment ideas.
- Communicating progress: Regular updates with leadership build trust and ensure alignment.
- Personalization is nuanced: Distinguish between customization, segmentation, and true personalization.
- Generative AI’s potential: Explore its use for simplifying the travel planning experience.

Michael St Laurent, a seasoned CRO expert, shared his insights on large-scale testing Here are 5 key takeaways for marketers:

Prioritization is key: Large-scale testing requires prioritizing tests based on potential impact and aligning with overall business goals.
AI is the future, but humans guide it: Mike emphasizes the power of AI in automating processes and analyzing data, but highlights the importance of human expertise and guidance.
Data quality is crucial: AI models rely heavily on the quality of data. Invest in building a robust data infrastructure to ensure accurate results.
People are the biggest challenge: Navigating internal politics and managing stakeholder expectations is often the biggest hurdle for large-scale testing.
Building a testing culture is a win: The ultimate goal of large-scale testing is to create a data-driven culture that fosters continuous improvement and better user experiences.

Ellie Hughes, a top CRO expert and Experimentor of the Year, shared some incredible insights in her recent Experiment Nation interview. Here are 5 takeaways to boost your experimentation game:

- Start Small and Stay Curious: Ellie emphasizes that experimentation is a journey, not a destination. Start small, stay curious, and never stop learning!
- Prioritize Value, Not Tasks: Focus your time on activities that truly drive client value, like uncovering insights and ideation, rather than spending hours on tedious tasks.
- Set Clear Expectations: Clearly define reporting timelines and communicate about peeking, managing client expectations, and setting the stage for success.
- Embrace Iteration: Experimentation is an iterative process. Be open to changing your process, adapting to new learnings, and continuously improving.
- Maximize Impact Through Big Ideas: Larger changes can be easier to detect, resulting in faster results and more impactful growth. Encourage clients to think big!

#experimentation #CRO #conversionrateoptimization #marketing #digitalmarketing #experimentnation

Join us for a deep dive into conversion rate optimization on low-traffic sites with these actionable insights Chris Mercer

Measure before optimizing: Understand the current user journey and behaviors to identify areas for improvement.
Test sequentially: Iterate faster on low-traffic websites by testing changes sequentially instead of using AB tests.
Focus on the right metrics: Prioritize actions that drive meaningful outcomes, such as lead generation or revenue.
Listen to the market: Gather user feedback to inform your optimization strategy and meet their needs.
Automate measurement: Leverage Google Analytics 4 to collect and analyze data effectively to guide decision-making.
#CRO #Experimentation #ConversionOptimization

Discover the power of experimenting to learn and win! 💪

Nick Murphy shares his expertise on building learning libraries. Here are 5 key takeaways:

- Learning libraries capture the “why” behind successful and unsuccessful tests.
- Organize your learning library with consistent and digestible fields.
- Use “broad” and “specific” learning columns for a layered understanding.
- Encourage cross-team collaboration by sharing insights company-wide.
- Learning libraries foster a culture of experimentation and growth.

In this episode, senior customer experience leader, Courtney LeBlanc, reveals the ideal partnership between CX and experimenters and shares invaluable insights on:

Forging powerful partnerships between CX and experimenters
Navigating the daily demands of a CX leader
Simplifying CX workflows with customer insights
... and so much more!

Don't miss out on this enlightening conversation from an expert in the CX game!

Find Courtney on LinkedIn ⤵

LinkedIn: https://www.linkedin.com/in/courtney-leblanc-426920b3/

00:00 Introduction and Mutual Appreciation
00:27 Meet Courtney: Customer Support and CX Expert
03:38 Building an Alliance with Customer Support
06:11 Challenges and Strategies in CX Collaboration
14:25 Data Analysis and Insights Sharing
23:01 The Ideal Partnership Between CX and Experimentation
24:47 Balancing Customer Insights and Business Priorities
26:46 The Emotional Impact of Customer Support
28:29 Communicating Customer Stories Effectively
30:04 Leveraging NPS Data for Product Improvement
32:08 Empowering Teams with Data Access
33:06 The Importance of Experiment Communication
37:53 Fostering Collaboration Between CX and Product Teams
48:58 Reflecting on Growth and Collaboration
50:22 Conclusion and Final Thoughts

How do you get Product Managers to adopt Experimentation? Shagun Aulakh, the Director of Product Management - Experimentation at American Express shares with Rommil here tips. Here are just a few of the takeaways:

1- Leadership advocacy is critical for embedding experimentation practices in a highly matrixed organization.
2 - Prioritize education and internal PR to influence and persuade teams to embrace experimentation.
3- Define success metrics for your experimentation program and map out a long-term plan for evolving maturity.
5 - Identify champions within the organization to inspire and motivate other teams to adopt experimentation practices.
5 - Democratize insights and share success stories to showcase the value and impact of experimentation.

Watch Shagun here:
Experimentation Elite in Birmingham UK: https://experimentationelite.com/#spe...
Growth Marketing Summit in Frankfurt, Germany: https://www.growthmarketingsummit.com/en

In this episode, we have Joey Herzberg who shares his thoughts on how AI fits in the experimentation life cycle, the utility of keeping easily referenced records of test results, and the importance of experimentation strategy with regard to sample ratios.

Learn about human-first engineering in this in-depth interview where Shawn David discusses strategies to organize work and to break down tasks into smaller digestible sessions. He explained the most effective ways for teams to discover the work that should be automated the most by focusing on the desires and strengths of individuals, to remove work that is menial specially to them. We explored the impacts on company culture that menial tasks can cause and discussed some of the pitfalls with new technologies as well as lessons to keep in mind while reviewing new tools and technologies.

Chapters

00:00 Introduction to Human First Engineering

00:55 Meet Sean David: The Pioneer of Human First Engineering

01:34 Breaking Down Tasks for Efficiency

09:56 The Importance of Time Tracking in Various Professions

16:51 Discovering Automation Opportunities

25:34 Maximizing Productivity: The Power of Uninterrupted Focus Time

26:59 Introducing Roman Santiago & Experiment Nation

27:13 The Impact of Menial Tasks on Team Mental Load

27:31 Exploring Company Culture and Employee Well-being

00:00 The Role of Menial Tasks in Shaping Company Culture

32:24 The Evolution of Work: From Henry Ford to Jeff Bezos

36:32 Navigating the Hype Around AI and Tools in the Workplace

47:45 Human-First Approach: Automating to Enhance Productivity

50:44 Closing Thoughts and Where to Find More

Jyoti Malik, the founder of Blue Bagels, built a successful CRO agency by only working with ethical brands. In this chat with Tracy Laranjo, she shares:
How to run CRO programs with purpose instead of random guesses
- Her favourite underrated user research methods
- Tips for creating a CRO strategy that succeeds in the long-term
... And much more.

Contact Jyoti:
Website: https://www.bluebagels.com/
LinkedIn: https://www.linkedin.com/in/jyotiisingh
Email: jyoti@bluebagels.com

Chapters
00:00 Introduction to Experiment Nation and Guest Introduction
00:48 Jyoti Malan's Journey and the Essence of Blue Bagels
02:26 The Philosophy Behind Purpose-Driven Brands
05:43 Crafting Purposeful CRO Programs
08:32 Identifying and Correcting Off-Track CRO Programs
17:36 Unconventional Research Methods in CRO
21:06 Adapting and Evolving CRO Strategies
24:41 Upcoming Projects and Closing Thoughts

Join Nima Yassini as he dives into experimentation culture, the journey of building a brand UX and digital marketing agency focused on conversion rate optimization, the integration of digital marketing and experimentation cultur, the struggles and benefits of adopting a culture of experimentation, and the need for leadership and reinvestment in the experimentation industry.

Connect with Nima here: https://www.linkedin.com/in/nimayassini/

Chapters
0:00 Opening Remarks and Guest Introduction
00:48 The Evolution of a Digital Marketing Agency
01:27 Merging UX Design with Experimentation
03:56 The Power of Data in Design Decisions
09:13 The Culture of Experimentation in Agencies vs. In-House
11:49 The Challenge of Embedding a Culture of Experimentation
13:36 The Future of the Experimentation Industry
25:12 Advice for Experimentation Practitioners
28:29 Closing Thoughts and Offers to Help the Industry

CEO and founder of Decision Layer, Solomon Kahn chats with Rommil Santiago about why he thinks that most companies should not be A/B testing and shares what we should instead. He also discusses his recent debate with Ron Kohavi on LinkedIn on this same topic.

Solomon's LinkedIn: https://www.linkedin.com/in/solomonkahn
Solomon's email: solomon.kahn@gmail.com

Original YouTube Video: https://www.youtube.com/watch?v=vqPylhO3ePU
Original LinkedIn Post: https://www.linkedin.com/posts/solomonkahn_the-case-against-ab-testing-yes-i-know-activity-7184577272685719552-hFAB
Ron's reply: https://www.linkedin.com/posts/ronnyk_abtesting-experimentguide-abtestingissues-activity-7185060367054565376-cAU9

It's that time of year again! It's time for Experiment Nation's 2024 CRO Salary Report. In this episode, we explore some of the report's highlights with Tracy Laranjo as well as ask her for her thoughts on how CROs and Experimenters can get paid fairly.

Find the CRO Salary report here: https://experimentnation.com/experiment-nations-2024-cro-salary-report/
Tracy's LinkedIn: https://www.linkedin.com/in/tracylaranjo/
Tracys email: hello@tracylaranjo.com

Zach Lebovics shares insights on experimentation and product management. He discusses the importance of documenting assumptions, testing the best-case scenario first, and understanding that experiment failure isn't solely tied to the outcome. Zach also shares personal experiences and learnings from working in companies like Zynga, Ritual, and Tonal, providing valuable tips for product managers.

Episode links
Zach's Etsy shop link: www.mindfulmix.co
Zach's Team Hub on Coda link: https://coda.io/@zach-lebovics/tonals-product-team-hub

Chapters
00:00 Unlocking the Power of Experimentation: Insights and Strategies
00:23 Meet Zach: A Journey from Product Manager to Experimentation
01:56 The Art of Optimization: From Personal Life to Professional Projects
05:33 Diving Deep into Experimentation: Lessons from Zynga to Tonal
12:21 The Three Pillars of Successful Experimentation
30:26 Wrapping Up: Key Takeaways and Future Directions

Shiva Manjanath, discusses with Ana Kerkovic the significance of collaboration with stakeholders and stakeholder management, optimizing CRO processes, transitioning from testing to experimentation, the impact of copy testing, and the need for research-backed strategies in CRO.

Chapters
-----
00:00 Unlocking Business Growth with Data00:24 Introducing Experiment Nation and Guest Shiva Manjunath00:45 Diving Deep into the World of CRO and Experimentation02:52 The Optimizer's Breakfast: A Routine for Success04:12 Anna Kerkovic Shares Her Journey and Passion for Psychology in CRO05:34 The Art of Persuasion: Manipulation or Motivation?06:29 Strategizing Beyond Spaghetti Testing: A Structured Approach to Experimentation14:15 Building Trust and Navigating Frustration in the CRO Process23:41 From Testing Ideas to Strategic Experimentation: The Journey of Learning31:06 The Importance of Copy Testing and the Future of CRO35:44 Wrapping Up: An Optimistic Outlook on CRO

Media.Monk's Juliana Jackson shares the truth about app development, optimization, and how web and app should work together.

Fractional Head of CRO, Tracy Laranjo talks about the power of Mentorship for Conversion Rate Optimizers (CROs). We cover:

Why would CROs need mentorship?
How do you find one?
When are you ready to be a mentor and how do you get that word out?
Would Tracy be a mentor for our listeners?

GrowthMentor: https://app.growthmentor.com/mentees/sign_up?ref=970f4f6f79

Contact Tracy:
LinkedIn: https://www.linkedin.com/in/tracylaranjo/
Email: hello@tracylaranjo.com

Chapters:
00:00 Introduction
02:34 Tracy's Journey to Self-Employment
06:01 The Value of CRO Mentorship
09:45 Finding the Right Mentor
13:09 The Mentor-Mentee Relationship
18:14 Transitioning from Mentee to Mentor
23:06 Tracy's Offer to Mentor

🔥 Check out this insightful interview with Bryce York, Director of Product Management at Tatari, on Experiment Nation’s Conference! Here are 5 key takeaways from the interview:

The similarities and differences between CRO and product management.
The importance of overlapping skills and work in both areas, especially in smaller companies.
Bryce’s background and journey from entrepreneurship to product management.
The significance of qualitative and quantitative data in experiment design and analysis.
The value of documentation and knowledge sharing in building an experiment-driven culture.
Watch the video to gain valuable insights into CRO, product management, and experimentation!

Chapters
00:00 - Introduction and Background
12:30 - Product Management vs. CRO
18:45 - Quantitative and Qualitative Data
25:20 - Minimum Viable Experiment vs. Minimum Viable Product
31:34 - Prioritization and Risk Assessment
34:38 - Getting Smart vs. Getting Lucky
38:54 - Experiment Log and Knowledge Sharing


#CRO #ProductManagement #Experimentation #MarketingSkills #DataDrivenMindset

Documentation. The unsung, yet incredibly important part of any experimentation program. Great documentation allows you to look back at past experiments and find patterns of what were the most effective levers to inform your testing roadmap and levers you probably can deprioritize. In today's episode of Experiment Nation, our own Charlotte April Bomford recently spoke to Edmund Beggs about the "Levers Framework" which allows Convert.com to deliver results for their clients in shorter amounts of time. If you'd like to learn more about this interesting framework, watch the full interview.

🚀 Exciting Insights from Simon Girardin about learning from failure!🚀

📊 Takeaways from the conversation with Simon Girardin, CRO Program Manager at Conversion Advocates:

1️⃣ Speak Stakeholder Language: Report results in terms of uplifts, decreases, and revenue. Ensure clarity for stakeholders who may not be as versed in the technical aspects.

2️⃣ Learning from Tests: Emphasize the importance of learning from every test. Capture value by reviewing data, segmenting results, and analyzing user behavior through session recordings and heatmaps.

3️⃣ Strategic Iteration: When faced with a winning result in a specific segment, be cautious about immediate implementation. Consider the long-term impact on the user experience and potential tech debt. Iteration should be intentional and aligned with overall business goals.

🔗 Follow Simon Gerarden on LinkedIn for more insightful content on strategic testing and experimentation.

🌐 Connect with Tracy Lano and Experiment Nation for more interviews and discussions with top conversion rate optimizers.

Chapters
00:00 Introduction and Welcome
00:23 Understanding the Importance of Learning from Past Experiments
00:56 The Role of Stakeholders in Experimentation
01:29 The Importance of Testing Strategically
02:34 The Process of Iterative Testing
04:34 The Importance of Segmentation in Testing
05:39 Interpreting Test Data
06:58 The Value of Inconclusive or Losing Tests
07:49 The Challenge of Iterating on a Test
09:32 The Importance of Prioritizing Hypotheses
11:40 The Process of Reviewing Test Results
15:00 The Role of Metrics in Testing
18:02 The Challenge of Implementing Test Wins
25:51 Conclusion and Final Thoughts

#ConversionOptimization #Experimentation #CRO #MarketingInsights

Koalatative's Gerda Vogt-Thomas asks Rommil Santiago questions about how he started Experiment Nation and how he manages it while working a full-time job at Optimizely.

Data-backed web design refers to the approach of utilizing empirical evidence and user insights to inform the design and optimization of websites. By analyzing data such as user behavior, click-through rates, conversion rates, and heatmaps, designers can make informed decisions about layout, navigation, content placement, and visual elements. A data-backed approach helps identify user preferences, pain points, and areas for improvement, resulting in enhanced user experiences. A/B testing, user surveys, and analytics tools are commonly employed to gather data and validate design choices. By leveraging data, web designers can create user-centric interfaces that drive engagement, conversions, and overall business success.

In this insightful video, Ishan Goel, transitioning from a software engineer to a researcher, delves into crucial CRO statistical concepts. He sheds light on statistical significance in experimentation, explores A/B testing intricacies, emphasizes the efficiency of sequential testing, and discusses challenges in stopping tests prematurely. The talk concludes by highlighting the intersection of experimentation and statistics in the realm of Conversion Rate Optimization.

In this episode, Mike Green discusses the use of diary studies in UX research, particularly in the context of public sector user research. Mike Green, a freelance user research lead based in the UK, shares his experiences using Google Forms, structuring incentives, and addressing challenges in recruitment for diary studies. The discussion emphasizes the longitudinal nature of diary studies, contrasting them with traditional usability tests, and highlights their effectiveness in capturing user feedback over a period of time. Mike also touches upon the importance of understanding user behavior, attitudinal responses, and the ethical considerations in dealing with vulnerable groups. The planning and design stage for diary studies is emphasized, stressing the need to align the research goals with the chosen methodology. Overall, the video aims to provide insights and tips for conducting impactful UX research using diary studies in the public sector.

Chapters
00:00 Introduction to Diary Studies with Mike Green
03:21 Understanding the Purpose of a Diary Study
06:45 Recruiting Participants for Your Diary Study
10:12 Analyzing Data from Diary Studies
14:30 Key Considerations for Diary Studies Success

#UXResearch #DiaryStudies #UserExperience #ResearchablePodcast