In this episode of Experiment Nation, host Jason sits down with Namata Agarwal, a seasoned product designer with over 17 years of experience building digital products, websites, and apps.Together, they explore:- Ethical testing: how to balance user needs with business goals.- Leaks vs. puddles framework: a practical way to identify urgent vs. non-urgent product problems.- The role of design in CRO and experimentation.- Frameworks, guardrails, and collaboration strategies for building an experimentation culture.Whether you’re a CRO specialist, product manager, or experimentation nerd, this episode is packed with actionable insights you can bring to your own team.👉 Don’t forget to like, comment, and subscribe to stay updated on the latest in experimentation and growth!📖 Check out Prove It or Lose It: The (Mostly) No-Nonsense Guide to Surviving Experimentation Program Drama .⏱️ Chapters with Timestamps0:00 – Why design must be involved early0:22 – Welcome & intro to Jason0:43 – Guest intro: Namata Agarwal0:55 – Topics: Ethical testing & Leaks vs. Puddles1:18 – What is ethical testing?2:22 – Balancing business needs vs. user needs2:54 – Handling stakeholder pushback4:01 – Misaligned metrics and real solutions5:17 – The Leaks vs. Puddles analogy explained6:32 – Why better hypothesis building matters7:48 – Using Airtable & frameworks for test prioritization8:40 – Why experimentation must be collaborative9:24 – Reframing experiments around the user journey10:00 – Unpredictable user journeys & design challenges11:12 – Involving guest collaborators in experimentation11:50 – Mapping user journeys with Miro & qualitative research12:50 – Jobs To Be Done & other qualitative frameworks14:44 – Turning qualitative insights into hypotheses15:32 – How product design elevates CRO strategies16:11 – The importance of forms in experimentation17:36 – Why design should be included from day one18:20 – Design is not just visuals—it’s about impact19:11 – Governance, responsibility, and inclusivity in testing19:46 – Leak vs. Puddle framework deep dive22:58 – Running collaborative workshops25:45 – The “Stinky Fish” workshop format27:21 – Micro vs. macro frameworks for problem-solving28:05 – Balancing bold ideas vs. safe bets29:19 – Planning tests like an investment portfolio30:13 – Testing across 13 sites at scale30:26 – What Namata would A/B test in daily life31:57 – Art, painting, and creative balance32:42 – Martial arts & keeping calm32:56 – Wrap up & final thoughts33:14 – How to connect with Namata

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

In this episode of the Experiment Nation Podcast, host Ward Vanestra sits down with Ryan Lux, one of the first team members at Eppo and now Experimentation Evangelist at Datadog.They dive into:What Eppo is and how it helps companies experiment at scaleWhy data warehouses are critical for trustworthy resultsThe recent Datadog acquisition of Eppo and what it means for the futureAdvanced methods like CUPED (noise-cancelling for experiments) and contextual banditsHow to scale experimentation culture and get promoted by driving adoption, efficiency, and trustworthinessWhether you’re a CRO, growth leader, or experimentation enthusiast, this episode is packed with insights from someone who’s been shaping experimentation for over a decade.📘 Check out Experiment Nation’s new book: Prove It or Lose It: The Mostly No-Nonsense Guide to Surviving Experimentation Program Drama → experimentnation.com⏱️ Chapters0:00 – Why win rates don’t really increase0:20 – Introduction to Ryan Lucht & Eppo1:20 – What Eppo does and who it’s for2:18 – Experimentation at smaller companies3:18 – Datadog’s acquisition of Eppo3:59 – The data warehouse advantage6:03 – CUPED explained: noise-cancelling for experiments9:13 – Segments, personalization & pitfalls11:09 – Heterogeneous treatment effects (HTEs)12:06 – Personalization with contextual bandits14:00 – Why Datadog acquired Eppo16:22 – Data observability & product analytics17:52 – Experimentation and site performance19:46 – Content, culture, and experimentation career paths22:10 – Every experiment is valuable (win, lose, or flat)24:17 – OKRs for experimentation leaders25:47 – Three key OKRs: adoption, efficiency, trustworthiness28:14 – Knowledge building and learning plans29:34 – The vision: experiments as the ultimate insight system30:24 – Ryan’s Substack “Everything is an Experiment”31:02 – Closing thoughts & where to connect

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Experimentation is a journey, not a destination. Don’t expect to build a culture of experimentation overnight.
Cross-functional teams are critical. Breaking down silos and bringing together engineers, product managers, and designers will create a unified vision and drive faster iteration.
Leadership buy-in is essential. A top-down approach to embedding experimentation ensures everyone is on board and resources are allocated strategically.
Don’t underestimate the importance of organizational rhythms. Establish clear processes and structures to ensure smooth collaboration and avoid friction.
Embrace the growing pains. Change is rarely easy. Be prepared for challenges and celebrate milestones along the way.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Florent Buisson is a decade-long behavioral economist who published a practical guide to behavioral data analysis in business. It's the book he wished he had when he started in analytics.

In this episode, he shares:

Plus other statistical concepts explained in a way even a five-year-old could understand.Why intuition can actually be a good thing in data analytics

If you need to know R or Python to understand your users

Why and when causal analytics are important

--

Florent Buisson is a behavioral economist with 10 years of experience in business, analytics, and behavioral science. He most recently started and led for four years the behavioral science team of one of the largest insurance companies in the US; he then published Behavioral Data Analysis with R and Python (O'Reilly), the book he wished he had had when he started. Florent holds a PhD in behavioral economics from the Sorbonne university in Paris.

Find Florent on LinkedIn: linkedin.com/in/florentbuisson

Read Florent's content on Medium: medium.com/@florent.buisson

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

About this series: Rommil Santiago, the founder of Experiment Nation shares his most recent thoughts about the industry and the craft of Experimentation in less than 5 minutes.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

A/B Testing can be intimidating to the uninitiated, but it doesn’t have to be! I want to introduce testing concepts via a real-world scenario that we can all relate to: Dessert. I’ll cover what an MVT is, hypotheses, independent variables, dependent variables, results (of course), and design flaws. And if you need any more convincing, I have four words for you: Monte Cristo Bread Pudding.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Matt Green and Moshe Mikanovsky of Product for Product podcast help you navigate the plethora of A/B testing tools out there.

Whether you're in product or marketing, listen for a beginner-friendly comparison of tools like Google Optimize, Optimizely, and more.

**This episode, nor those of Product for Product, are endorsements of any of the discussed tools.**

Listen to Product for Product: spreaker.com/show/product-for-product

Find Matt on LinkedIn: /in/mattgreenanalytics

Find Moshe on LinkedIn: /in/mikanovsky

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

You can do better than testing button colors, but how do you decide what to test when, and what type of test to run? Roadmapping. This session teaches you to be the GPS navigator, helping your team optimize experimentation routes.  On this journey, you will identify how each path can drive desired impacts by turning a formidable trek into a series of attainable milestones.   

Presented by 

• Kenya Davis, Microsoft Digital Analytics


• Planning Program Manager, and Evolytics Decision Science Director Krissy Tripp.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Experiment Nation: The Podcast is a show on which we share Interviews with Experimenters from around the world, opinion pieces, and recent presentations with you - the experimentation community. We hope you enjoy the show, let’s get to it.

If you enjoy our content, consider signing up for Experiment Nation's bi-weekly newsletter here: https://buff.ly/31qO0AQ

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Tracy is a Toronto-based growth marketer with a knack for conversion optimization. In a past life she’s QA tested “adult” eBooks and helped make it easier for Canadians to write their wills online. Tracy is also one of the co-hosts of the Experiment Nation podcast where she chats with optimizers at the top of their game.

Beyond the 9-5, you can catch her knitting, crushing empires in Civ6, or napping with her cats.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

About this series: Rommil Santiago, the founder of Experiment Nation shares his most recent thoughts about the industry and the craft of Experimentation in less than 5 minutes.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

eCommerce CRO veteran Johann Van Tonder shares what usually goes wrong when coming up with test ideas and how to fix it. We also discuss the merits of quantity vs. quality of ideas, the problem with brainstorming, and what to do instead for more effective ideation sessions.

Find Johann on Twitter: @amaclickclick

Learn more about AWA Digital: awa-digital.com

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Anil is a Digital Marketing and Analytics professional with 18+ years of experience. He has worked with companies ranging from startups to Fortune 500 and has helped them improve marketing results as well as develop data driven culture. He has vast experience in Web, Email, Social, Display, Search, Paid Search, and CRM. He holds a B. Tech in Electronics and Communication Engineering from India and an MBA from the University of Washington, Seattle. He is the founder of https://academy.optizent.com – A Digital Analytics training and education company.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

In this podcast episode of Experiment Nation, hosted by Charlotte Bomford, guest Iqbal Ali, an experienced web designer, researcher, and comic book writer, discusses the concept of solution bias and the importance of identifying underlying problems before diving into solutions. Ali emphasizes using AI to help understand problem landscapes, avoiding hastily jumping to solutions, and improving experimentation processes. The conversation covers techniques like premortems and frameworks to minimize potential issues and ensure experiments address real user problems. Ali highlights the collaborative potential of AI tools in maintaining the connection between problem statements and solutions, advocating for a more thoughtful approach to A/B testing and problem-solving in product development.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

About this series
Rommil Santiago, the founder of Experiment Nation shares his most recent thoughts about the industry and the craft of Experimentation in less than 5 minutes.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

When experimenting with our products, we need to find our users but it’s not always that simple! Where do we find the users? In this session Moshe will go over different types of products and different ways to find users for each of them. Coming out of this session, attendees will have some ideas on where to start looking for their users.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

About this series

Rommil Santiago, the founder of Experiment Nation shares his most recent thoughts about the industry and the craft of Experimentation in less than 5 minutes.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Although the two sample proportion test is well known within the marketing experimentation community, there are a host of categorical data analysis techniques that are amenable to the analysis of A\B test data. This presentation provides a quick overview of Chi-square analysis and a discussion of its use cases in marketing experimentation. Specifically, we discuss how to 1) test a\b\n experiments without inflating Type I error after multiple comparisons, 2) analyze tests with more than dichotomous outcome variables and 3) the process for calculating goodness-of-fit and tests-of-independence. Simple guidelines will be provided regarding test assumptions and use case scenarios.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Sina Fak is the Head of Optimization & Business Intelligence at ConversionAdvocates, the #3 top CRO agency in the world in 2020. Sina shares what businesses get wrong when they hire a shiny new in-house CRO strategist, what to look out for before taking an in-house CRO role, and when a business should consider going agency vs. In-house for CRO.

Learn more about ConversionAdvocates: https://conversionadvocates.com/

Find Sina Fak on LinkedIn: https://www.linkedin.com/in/sinafak/

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

If you liked this episode, don't forget to signup for Experiment Nation's bi-weekly newsletter here: https://buff.ly/31qO0AQ

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

We know that the key skillsets you need for any successful CRO / Experimentation program are: Research, Design, Development and QA. Whilst larger companies who can afford to hire all of those skilled people, for smaller companies, it becomes a big challenge. So much so, many companies exhaust their options of running smaller tests and then stop doing all kind of experimentation work.  In this session, I will be covering the areas where the companies can reduce their cost of experimentation, things that they will need to consider and increase their profitability of running a successful experimentation program. 

--- 

If you liked this episode, don't forget to signup for Experiment Nation's bi-weekly newsletter here: https://buff.ly/31qO0AQ

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Rishi Rawat is the product page guy. With his 13 years of running A/B tests, Rishi shares what makes an effective product description, his go-to method for segmenting product pages, and how to stand out to your visitors during their most “promiscuous state”.



Learn more about Rishi's work at frictionless-commerce.com.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

A lot of research and data analysis has gone in to answering this question. But yet, there’s no clear, definitive answer.  In attempt to resolve this question once-and-for-all, this session will present and interesting and surprising answer.  Applying the results from a real-life A/B test case study, digital marketers will be prompted to think not only about email send time, but also their audience and how both aspects impact conversions.

--- 

If you liked this episode, don't forget to signup for Experiment Nation's bi-weekly newsletter here: https://buff.ly/31qO0AQ

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Finally, remember to signup for Experiment Nation's bi-weekly newsletter here: https://buff.ly/31qO0AQ

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Ruben de Boer, a lead consultant at Online Dialogue and the founder of Conversion Ideas, shares his approach to conversion rate optimization (CRO) and experimentation. With 15 years of experience, Ruben emphasizes the importance of data-driven decision-making and understanding human behavior. He highlights the shift from focusing solely on digital products to also optimizing organizational processes. Ruben provides insights into the challenges of implementing experimentation in organizations, the need for higher management support, and the significance of aligning with colleagues' goals and motivations. He advocates for a research-driven approach, continuous feedback, and fostering an experimentation culture within teams to drive better results.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Welcome to "Where I Started", the interview mini-series in which we get behind the scenes and find out how some of today's experimenters got into CRO. This should be a fun little project that Ontrack Digital started with Experiment Nation to give you a bit of a getaway from all those test results. But also it can be a source of inspiration for those of you that want to get started in CRO and don't really know which path to take.

---

If you liked this episode, don't forget to signup for Experiment Nation's bi-weekly newsletter here: https://buff.ly/31qO0AQ

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

This session is about the conundrum that “data is everything” – a mentality circulating at large the past couple years. There’s so much data, and we often find ourselves simply staring at metrics looking for “+” or “-” to make decisions. The objective of this session is to draw on thinking strategies from physics, of how to approach CRO differently, without any real “data”, to then gleam positive impacts to CRO.

Finally, remember to signup for Experiment Nation's bi-weekly newsletter here: https://buff.ly/31qO0AQ

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

The success of your testing program depends on the quality of your hypotheses. But too many experimentation programs lack the processes and frameworks to move from spaghetti testing to strategic experimentation.  Haley shares two practical frameworks to help you form better hypotheses and objectively rank test ideas.

---

If you liked this episode, don't forget to signup for Experiment Nation's bi-weekly newsletter here: https://buff.ly/31qO0AQ

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Aditi Gupta covers the creation of a growth roadmap/ how to build alignment with stakeholders/ creating an org-wide impact.  

--- 

If you liked this episode, don't forget to signup for Experiment Nation's bi-weekly newsletter here: https://buff.ly/31qO0AQ

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Welcome to "Where I Started", the interview mini-series in which we get behind the scenes and find out how some of today's experimenters got into CRO. This should be a fun little project that Ontrack Digital started with Experiment Nation to give you a bit of a getaway from all those test results. But also it can be a source of inspiration for those of you that want to get started in CRO and don't really know which path to take.

---

If you liked this episode, don't forget to signup for Experiment Nation's bi-weekly newsletter here: https://buff.ly/31qO0AQ

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Speero's Director of CX Optimization, Edgar Spongolts, dives deep into the influential agency's Experimentation Maturity Audit and why maturity matters. Learn about the four pillars of a successful experimentation program and what it takes to reach the highest levels of maturity. Take the audit yourself here: https://speero.link/ExperimentNation
---
If you looked this episode, don't forget to signup for Experiment Nation's bi-weekly newsletter here: https://buff.ly/31qO0AQ

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Too often experiments are squandered. But if you put customer problems first and design the right type of test you can ensure every experiment is valuable to your business. The following talk will give your experimentation programs guardrails to make sure you’re not testing for testing sake. In this session Annika and Paul will share;  How to map out business goals and customer problems. How to pick the right type of experiment. e.g when to use tree testing, copy testing, 5 second tests, A/B tests etc. How to gain insights faster and deliver more effective experiments.

---

Don't forget to signup for Experiment Nation's bi-weekly newsletter here: https://buff.ly/31qO0AQ

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Experiment Nation: The Conference Reloaded

Everybody is doing it, so you have to do it too, right? As my mom used to say: you’re not everyone! Best practices are not experimentation and you should know that. I want to prove to you this is the easy way, but it’s not the right way for a healthy, profitable and experiment-oriented business. These tactics don’t last long.  In this session, we will discuss why, statically, best practices don’t work. It’s fun to know about the competitors, their results and ideas. It may generate new ideas and grab your attention when you’re starting to experiment, but that’s the only good thing. To have a great experiment culture, you need to learn from your tests. Let’s talk more about this?

-----

If you liked this interview, signup for Experiment Nation's bi-weekly newsletter to receive more great interviews like this, memes, editorials, and conference sessions in your inbox: https://lnkd.in/g6rM8w25

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Experiment Nation: The Conference Reloaded

CRO should be the foundation that all of your other digital marketing efforts rests upon. Through what we call our ‘Ripple Benefits’, I’d love to showcase just how CRO not only improves your conversion rates, but your SEO, Content Marketing, Paid Ads, and more. Testing is key and should be the foundation that drives where your efforts are focused.  -

-----

If you liked this interview, signup for Experiment Nation's bi-weekly newsletter to receive more great interviews like this, memes, editorials, and conference sessions in your inbox: https://lnkd.in/g6rM8w25

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Petra Lylander is a growth marketer who developed the experimentation process at Helsinki-based SaaS company, Aiven. In this episode she talks about process documentation for experimentation and how to get other teammates to care about it.

------

If you liked this interview, signup for Experiment Nation's bi-weekly newsletter to receive more great interviews like this, memes, editorials, and conference sessions in your inbox: https://lnkd.in/g6rM8w25

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Tymur Donets, a seasoned Growth PM and experimentation consultant, shared some powerful insights during his Experiment Nation talk. Here are five key takeaways:

Growth PMs are holistic: Don’t be siloed! Growth PMs work across the entire customer journey, ensuring marketing and product teams are aligned.
Commercial awareness is key: Both Growth PMs and CROs must understand the business impact of their work and how it contributes to the bottom line.
Empathy is vital: CROs need to understand the pressures their PM colleagues face and proactively offer support.
Leverage your expertise: If you’re a CRO, consider specializing in Growth PM to expand your impact and skillset.
Transparency is key: Clear communication between teams helps everyone understand their roles and collaborate effectively.

Find Tymur on LinkedIn: https://www.linkedin.com/in/tim-donets-2358364a

Chapters:
0:00 - Introduction and The Role of a Growth PM
0:58 - Growth PM vs. Product Manager
2:40 - Tymur’s Journey and How He Became a Growth PM
5:23 - Importance of Business Savvy and Data Literacy
8:30 - Bridging the Gap Between Marketing and Product Teams
12:42 - Experimenters as Facilitators
16:38 - The Complexity of a Product Manager’s Role
19:50 - CRO Responsibilities and Building Relationships with PMs
24:48 - Transitioning from CRO to PM
29:00 - Finding Growth PM Opportunities and Wrap-Up

#growthmarketing #experimentation #cro #productmanagement #collaboration

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Experiment Nation: The Conference Reloaded

AKA Why you should approach research like an Anthropologist  How ethnographic research helps you understand *why* people think the way they think (not just *what* they think or how they respond to triggers) to create more meaningful hypotheses and experiments.

------



If you liked this interview, signup for Experiment Nation's bi-weekly newsletter to receive more great interviews like this, memes, editorials, and conference sessions in your inbox: https://lnkd.in/g6rM8w25

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Experiment Nation: The Conference Reloaded

Experimentation post-conversion or how to switch from buying journey to customer journey.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Experiment Nation: The Conference Reloaded

A quick and dirty guide to pointing out bad practices that create friction to the customer when brands practice the company journey instead of the customer journey.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Nesse episódio nosso convidado André Gillich, Head of CRO, compartilha sua experiência na Jussi crescendo a cultura de experimentação, começando pela nova abordagem em que “CRO não é um profissional e sim um processo.”Exploramos o impacto da cultura “Agil” e o desafio do foco em outputs não outcomes.Fechamos o podcast de forma prática compartilhando iniciativas tomadas em nossas empresas para acelerar a flywheel de experimentação.

• Edição: João Pedro AlencarReferências citadas: 


• Mestre GP https://www.mestregp.com.br/


• Product arena https://productarena.io/


• It takes a flywheel to fly https://www.microsoft.com/en-us/research/group/experimentation-platform-exp/articles/it-takes-a-flywheel-to-fly-kickstarting-and-keeping-the-a-b-testing-momentum/


• Cursos PM3 cursospm3.com.br


• Naval at Tim Ferris https://open.spotify.com/episode/1yTaZvBKLTwVQHcNTvZ19i?si=wF_acs3ZT7qEGTCa4JUDlw


• Naval monologue https://open.spotify.com/episode/1TCwItjO5uj294IZ7MoZ10?si=3uWYsGhQStumCcRYBFJm5w


• How to get rich (without getting lucky) https://twitter.com/naval/status/1002103360646823936?lang=en


• Guess the test inspiration https://guessthetest.com/

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Steph walks us through the challenges of growing Specsavers’ CRO team and how she overcame them, her experimentation red flags, and her take on if CRO certifications are really worth it. Plus: A comparison of the CRO job market between Canada vs. the UK, including how well you can expect to be paid as an optimizer.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Experiment Nation: The Conference Reloaded

CRO, experimentation, A/B testing, research, analytics: these things are all parts of a whole which never really gets discussed. What we’re doing is replicating the scientific method in a business context, and yet we can’t see the wood for the trees. Our industry is focused on the individual elements of what we do and how we do it, but not why we do it and the overall philosophy for it.  This is part of the reason that businesses don’t give it the traction it deserves; senior business leaders are not interested in the execution, they’re interested in the overall benefit. Let’s take a step back and think about how to go beyond the detail and drive a more mature outlook on what we do!

We'd love your feedback. Tell us what you thought of the session and of our conference for a chance to win a free Customer Value Optimization course from Omniconvert.com and a chance to be featured on ExperimentNation.com  Send us your feedback here: https://forms.gle/Mzo9h33RZ94d471Q7

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Experiment Nation: The Conference Reloaded

We'd love your feedback. Tell us what you thought of the session and of our conference for a chance to win a free Customer Value Optimization course from Omniconvert.com and a chance to be featured on ExperimentNation.com  Send us your feedback here: https://forms.gle/Mzo9h33RZ94d471Q7

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Experiment Nation: The Conference Reloaded

We'd love your feedback. Tell us what you thought of the session and of our conference for a chance to win a free Customer Value Optimization course from Omniconvert.com and a chance to be featured on ExperimentNation.com  Send us your feedback here: https://forms.gle/Mzo9h33RZ94d471Q7

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Experiment Nation: The Conference Reloaded

As conversion specialists and data analysts, we strive for experimentation- and validation-driven culture.  To be successful, it is essential to understand that your job is not just to run A/B tests. A large proportion of our work should be dedicated to change management. However, change is hard.  In his presentation, Ruben will discuss the importance of change management guided by an understandable model that you can start using today. He will explain the model step-by-step, using the knowledge you already have. You will find many examples of things you should and should not do to cultivate a culture of experimentation in your organization.

We'd love your feedback. Tell us what you thought of the session and of our conference for a chance to win a free Customer Value Optimization course from Omniconvert.com and a chance to be featured on ExperimentNation.com  Send us your feedback here: https://forms.gle/Mzo9h33RZ94d471Q7

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Experiment Nation: The Conference Reloaded

We'd love your feedback. Tell us what you thought of the session and of our conference for a chance to win a free Customer Value Optimization course from Omniconvert.com and a chance to be featured on ExperimentNation.com  Send us your feedback here: https://forms.gle/Mzo9h33RZ94d471Q7

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Kelly Wortham dropped some serious knowledge bombs in our recent Experiment Nation chat. Here’s what stood out:

- CRO isn’t just for marketers: Lawyers, project managers, even developers can thrive in this field. It’s all about a love of learning and adaptability.
- Don’t underestimate the power of mentorship: Kelly has THREE mentors, each focusing on different aspects of her career. Find your own advisors to help you grow!
- Certifications are nice, but experience trumps all: While CXL is great, hands-on testing platform experience is the real deal.
- UX psychology is CRUCIAL: Understand your users’ motivations and behaviors to create truly effective tests.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Tim shares his tips for running low and high maturity experimentation programs, what it’s like to optimize in-house vs. agency, when it is (and is not) ok to make compromises, and to how make your program’s results actually drive action. Plus: Tim’s take on those controversial UI polls on LinkedIn.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Experiment Nation: The Conference Reloaded

Shiva will run through WHY you want to be running an experimentation program based on insight generation and not simply ‘winning’. He’ll also show you how to structure tests to support the growth of an insight-driven experimentation program.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Experiment Nation: The Conference Reloaded

The average success rate of most AB testing programs hovers around 12%. Yep, for every 100 experiments you run, only 12 of them will generate an increase in conversions. That is worse than a coin toss.  Too many companies start AB testing hoping that it will deliver a great increase in conversions. A few months later, they are bitterly disappointed.  In this session, I will share the lessons we learned by analyzing over 22,000 AB tests on 700 CRO projects. These lessons paint the common patterns that helped us deliver upwards of 50% success rates on AB testing programs.  The lessons provide a strategic roadmap for a CRO project as well as tactical details needed while running the day-to-day operations.  Key takeaways  Attendees will learn what it really takes to run a successful CRO program (Time, resource, management, etc). Attendees will learn the right balance between strategy and tactics when running an experimentation program. Attendees will see an actual project management board running a CRO program

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com