Explorez tous les épisodes du podcast Klaviyo Data Science Podcast

Games typically involve some combination of math, chance, strategy, and skill. In this two part mini-series, the Klaviyo Data Science Podcast investigates the role of math in games and how our relationship to them is changing given the increasing number of sophisticated AI Agents that can often beat us at our favorite games.

This podcast additionally covers concepts ranging from what makes a game ‘Solved’, to how the early development of Game Theory was influenced by the Cold War.The team also touches on a topic that will be expanded on in the next episode: why AI Agents are so sophisticated at certain games (e.g. Go), while struggling with other games (e.g. no stakes poker or basketball).

When they invent the robot LeBron James, it will be the end of human society.
- Michael Lawson

For more details, check out the ⁠⁠⁠full writeup on Medium⁠⁠⁠!

All companies start small. The lucky ones grow, and growth necessarily comes with change. This month on the Klaviyo Data Science Podcast, we look at Klaviyo’s growth through the past 8 years and the profound effects that growth has had on R&D, ranging from the tooling we use to the processes we employ to the very culture we strive to cultivate. Listen in to learn:

• How the day-to-day of some of our most senior engineers has changed
• Why common headaches like planning, deployment, and cross-team coordination can get harder as your org grows — and why they can get easier 
• What a single lunch can teach you about a company’s culture 

For more details, check out the ⁠⁠full writeup on Medium⁠⁠!

This month, we return to a classic Klaviyo Data Science Podcast series: books every data scientist (and software engineer) should read. This episode focuses on the Clean * duology by Robert C. Martin, which teaches the principles of both clean code and clean architecture. We’ve brought on two senior engineers at Klaviyo who’ve learned, practiced, and developed their own opinions on the lessons in these books. Listen in to learn:

• How to use these books to level up your own skills and the skills of your team
• Why the book’s spiciest opinions make sense, and where you might disagree with them in practice 
• What our panel’s deepest, most intimate thoughts on docstrings are

For more details, including links to these books, check out the ⁠full writeup on Medium⁠!

What should I do next? A common question, one that seems simple on the surface, but the answer, especially a more optimal answer, can be very difficult to uncover. It may involve information that the asker is not aware of, be unintuitive, or even be counter to our instincts.

This month, we discuss a new Klaviyo feature: Next Best Action. Utilizing Klaviyo’s knowledge of marketing best practices, we can recommend specific actions that are likely to be highly advantageous

All successful teams have at least one leader, and most have at least one manager. This episode, we dive into how leadership works on highly technical teams, how managing a highly technical team works, and why the two aren’t exactly the same thing. Listen along for more discussion about:

• The traits of highly effective leaders — and how that might look different on an engineering or data science team 
• How to know that a move into management fits you 
• Our guests’ best recommendations for management books, resources, and experiences

For more details, including links to the many resources our panel suggest to learn more about leadership and management, check out the full writeup on Medium!

What is agile methodology — and, just as importantly, what is it not? Whether you’re new to agile entirely or you stay up late pondering its most philosophical inner workings, if you want to know more about agile and how organizations can reap its benefits while avoiding its pitfalls, this is the episode for you. You’ll learn about a variety of topics, including:

• How to effectively compromise between the tenets of agile and the realities of building software — and how not to 
• When agile helps you pivot, and what that means for your customers 
• Our guests’ hottest takes about agile

For the full show notes, including who's who, see the ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Medium writeup⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠.

This month, the Klaviyo Data Science Podcast welcomes Evan Miller to deliver a seminar on his recently published paper, Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations! This episode is a mix of a live seminar Evan gave to the team at Klaviyo and an interview we conducted with him afterward.

Suppose you’re trying to understand the performance of an AI model — maybe one you built or fine-tuned and are comparing to state-of-the-art models, maybe one you’re considering loading up and using for a project you’re about to start. If you look at the literature today, you can get a sense of what the average performance for the model is on an evaluation or set of tasks. But often, that’s unfortunately the extent of what it’s possible to learn —there is much less emphasis placed on the variability or uncertainty inherent to those estimates. And as anyone who’s worked with a statistical model in the past can affirm, variability is a huge part of why you might choose to use or discard a model. 

This seminar explores how to best compute, summarize, and display estimates of variability for AI models. Listen along to hear about topics like:

• Why the Central Limit Theorem you learned about in Stats 101 is still relevant with the most advanced AI models developed today
• How to think about complications of classic assumptions, such as measurement error or clustering, in the AI landscape 
• When to do a sample size calculation for your AI model, and how to do it

About Evan Miller

You may already know our guest Evan Miller from his fantastic blog, which includes his celebrated A/B testing posts, such as “How not to run an A/B test.” You may also have used his A/B testing tools, such as the sample size calculator. Evan currently works as a research scientist at Anthropic. 

About Anthropic

Per Anthropic’s website:

You can find more information about Anthropic, including links to their social media accounts, on the company website.

Anthropic is an AI safety and research company based in San Francisco. Our interdisciplinary team has experience across ML, physics, policy, and product. Together, we generate research and create reliable, beneficial AI systems.

Special thanks to Chris Murphy at Klaviyo for organizing this seminar and making this episode possible! 

For the full show notes, including who's who, see the ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Medium writeup⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠.

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

2024 Year in Review

As the new year starts, we take a look back at 2024. We spoke to data scientists and people who work closely with data scientists, and we asked them all the question we ask every year: what is the coolest data science thing you learned about in 2024? You’ll hear a wide range of answers, including:

• How a rhyme can topple LLM security
• Why the Sequential Probably Ratio Test is better at measuring basketball ability than the NBA playoffs 
• How badly a non-specialized LLM could beat you at chess

For the full show notes, including who's who, see the ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Medium writeup⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠.

How we protect invention and ingenuity: Patents

Writing software often involves taking known patterns, combining and shaping them, and adding needed context or specialization related to the problem we’re trying to solve. Sometimes, that means writing something that’s effectively been written by someone else before. But sometimes, that means creating something new.

What should you do in a case where you’ve genuinely created something new? Perhaps more importantly, how do you know when you’re in that situation? 

This month, we explore one of the best tools to help answer both questions: the patent process. Listen along with your fearless co-hosts and a member of Klaviyo’s legal team to learn about what a patent is, why getting them matters, and how to get your own novel work patented, along with:

• Tips to keep in mind when applying for a patent 
• Knowing which software you should try to patent
• How to get your organization to buy into patents 

For the full show notes, including who's who, see the ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Medium writeup⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠.

Welcome to the November episode of the Klaviyo Data Science Podcast for 2024! In years past, November episodes reflected the chaotic Black Friday/Cyber Monday season by examining unique challenges of readiness, scale, and fundamental changes happening with little to no warning, as well as how those challenges were handled; this November is no different.

What happens when two of the largest email platforms make sweeping changes to their spam filters, providing a few short months of notice? Stress, uncertainty, and an opportunity for individuals and organization to rise to the challenge. In this month’s episode, we talk with analysts, engineers, and product managers to discuss Klaviyo’s journey to meet Yahoo and Google’s new Email Delivery Requirements — aka Yahoogle, the colloqial name for a new set of rules that must be followed by email senders to have their emails make it to inboxes and not go straight to the junk bin.

Listen in to hear more about:

• Yahoogle. It’s more than just a funny word!
• The challenges in operating a large-scale email sending system
• Advice and retrospectives about large, cross-functional projects and tight deadlines

For the full show notes, including who's who, see the ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Medium writeup⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠.

As most data scientists will tell you, there is no such thing as the single best model or the perfect model. Some work well in some circumstances but poorly in others, some present a specific tradeoff between factors like flexibility and explainability that is only useful in certain settings. Some are best set up to handle specific types of data that don’t arise in every single project. 

But at the same time, most data scientists would acknowledge that some models manage to stand out. Maybe it’s nostalgia, maybe it’s how powerful they are in some settings, maybe it’s another factor entirely — but for one reason or another, most data scientists will admit they have a soft spot for some models. That’s what we’re here to discuss this month: what is your favorite model? Listen in to hear more about:

• The benefits of simple models
• How learning about some models teaches you about entire fields of mathematics 
• What fitting a classification model has to teach us about epistemology 

For the full show notes, including who's who, see the ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Medium writeup⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠.

If you’re making software, especially data science-powered software, there’s a good chance one of your biggest goals is to empower stronger and deeper personalization for your users. Our topic for this month: how can you do even more than that? How can we make personalization not just robust, but both more effective and easier than the alternative? 

It’s not a simple task, but it is one that the team we interviewed this month has tackled. Listen in to hear more about:

• Why personalization is a matter of finishing your user’s… sandwiches 
• How to approach complex personalization features as a data scientist, a designer, or an engineer 
• What the game of Battleship can teach you about personalization

For the full show notes, including who's who, see the ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Medium writeup⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠.

It may come as a suprise to those of you reading this, but this milestone snuck up on me. I was surprised to realize we’d reached a full 50 episodes. What better time to take a moment to reflect and look back?

This episode is all about the Klaviyo Data Science Podcast. We talk through the history of the podcast, how we approach making episodes that matter to our listeners, our highlight episodes, and what we’ve learned through the years. You’ll hear about:

• Why you don’t need a fancy mic to get started
• How to approach talking about deep, technical work on the air
• What we have planned for the next 50 episodes

For the full show notes, including who's who, see the ⁠⁠⁠⁠⁠⁠⁠⁠⁠Medium writeup⁠⁠⁠⁠⁠⁠⁠⁠⁠.

A big part of growing and developing as a data scientist, or any other member of a data science team, is taking time to reflect, learn, and distill experiences into advice. This month, we’ve asked four senior members of the data science team to do exactly that: look back over their careers, reflect on what they know and what they wished they’d known earlier, and tell everyone what those lessons are. Listen to this advice-filled episode to hear:

• How taking a more scenic or indirect career route can impart valuable experiences 
• The growth and development opportunities that truly unlocked new phases of their careers, and how you can make the most of similar opportunities
• Why mistakes often make the best learning opportunities 

For the full show notes, including who's who, see the ⁠⁠⁠⁠⁠⁠⁠⁠Medium writeup⁠⁠⁠⁠⁠⁠⁠⁠.

Internationalizing your product

There are many aspects of product growth — reaching new heights for peak volume, reaching new levels of sustained daily volume, growing your feature set and the complexity of your code based, and many others. Dealing with growth in an intelligent and forward-looking way is never easy, but this month we deal with a type of growth that presents its own unique set of challenges: international growth, i.e. expanding the range of countries and languages your products are natively available in. 

This month, we talked with multiple members of the internationalization effort here at Klaviyo, from teams across our organization. You’ll hear about:

• How strings can be far more complex than they seem
• Why changing languages doesn’t just mean changing words 
• Where in the world your assumptions about language may break

For the full show notes, including who's who, see the ⁠⁠⁠⁠⁠⁠⁠Medium writeup⁠⁠⁠⁠⁠⁠⁠.

How real marketers use data science

We spend a lot of time on this podcast talking about how to build data science solutions. Implicit in many of those conversations is perhaps the most fundamental truth of product design and development: we build data science solutions because people use them. We aren’t doing this just for fun — the reason we spend so much time, effort, and energy to refine our solutions is that it actually matters to real people.

This month, we talk to some of those people. In particular, we sat down with two members of the team at Made In Cookware (http://madeincookware.com/) to discuss what makes their business unique, how they approach understanding and marketing to their customers, and how data science and AI help them do all of that. You’ll hear about:

• What kitchen knives can teach you about product design and development
• Which type of pan you should use to cook a steak, and how that can help you understand customer segmentation
• How AI saves real marketers real time while also giving them better results

About Made In

Made In Cookware (Made In) is a premium cookware brand based in Austin, TX. Founded in 2017 but born of a 4th-generation, family-owned kitchen supply business, Made In creates best-in-class cookware developed in partnership with the world’s finest chefs and foremost craftsmen. Today, you’ll find Made In products in more than 2,000 restaurants, in the hands of James Beard Award-winning chefs at Michelin-starred restaurants across the country, and in the kitchens of home cooks everywhere. Made In products have garnered over 100,000 5-star reviews, and the company was named one of Inc. Magazine’s best workplaces and Newsweek’s best online shops of 2024.

For the full show notes, including who's who, see the ⁠⁠⁠⁠⁠⁠⁠Medium writeup⁠⁠⁠⁠⁠⁠⁠.

An Introduction to ML Ops 

Building data science products requires many things we’ve discussed on this podcast before: insight, customer empathy, strategic thinking, flexibility, and a whole lot of determination. But it requires one more thing we haven’t talked about nearly as much: a stable, performant, and easy-to-use foundation. Setting up that foundation is the chief goal of the field of machine learning operations, aka ML Ops.

This month on the Klaviyo Data Science Podcast, we give a brief but thorough introduction to the field of ML Ops. You’ll hear about:

• How ML Ops is different from the similar fields of data science and DevOps
• What skills a successful ML Ops developer should have, and what an ML Ops developer’s day-to-day looks like
• Why concepts like “velocity” and “stability” have their own special nuances in the world of ML Ops

For the full show notes, including who's who, see the ⁠⁠⁠⁠⁠⁠Medium writeup⁠⁠⁠⁠⁠⁠.

In many ways, 2023 was the year of AI in tech, which is a double-edged sword. On the one hand, the basic technology is straightforwardly exciting — but on the other hand, with seemingly every technology solution scrambling to integrate a thin wrapper around ChatGPT, it’s hard to stand out in a saturated environment. This month on the Klaviyo Data Science Podcast, we dive into a case study of how to build AI products, SegmentsAI, and discuss the principles that go into making sure your AI-powered product shines — and, more importantly, actually helps your customers. You’ll hear about:

• How to know when AI is the right solution for the problem
• The unique technical challenges that come with building an AI product, from user testing to validation 
• The answer to the AI chicken-and-egg problem

“Why do this, why build another LLM feature? It seems like every website is rushing to get their name next to AI... How you break through the noise is to actually provide value to people, not novelty. Being able to help customers speed up or generate new, interesting segments that they otherwise wouldn’t? I think that’s valuable.”— Rob Huselid, Senior Data Scientist

For the full show notes, including who's who, see the ⁠⁠⁠⁠⁠Medium writeup⁠⁠⁠⁠⁠.

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

Equity, Diversity, and Inclusion

Equity, diversity, and inclusion (EDI) are more than just central principles of successful teams in data science and beyond — they’re also a rich field that presents interesting and challenging data science problems. This episode, we chat with two EDI specialists at Klaviyo about EDI, the data that powers it, and the challenges that come with using that data. You’ll hear about:

• Why EDI is a core part of both processes and products 
• How to work with self-reported data — and, sometimes, work around the fact that you don’t actually have the data you want
• Examples of EDI work in action

For the full show notes, including who's who, see the ⁠⁠⁠⁠Medium writeup⁠⁠⁠⁠.

2023 Year in Review

As the new year starts, we take a look back at 2023. We spoke to 11 data scientist and people who work closely with data scientists, and we asked them all the question we ask every year: what is the coolest data science thing you learned about in 2023? You’ll hear a wide range of answers, including:

• How data science moving to peripheral devices and becoming more accessible has huge implications for the future of the field
• Peculiarities of working with large language models, both in terms of the tasks they can carry out and how the process of working with them is more complicated than it seems at first
• How powerful simple techniques can be at even highly complex tasks

“You don’t have to have a PhD any longer to do data science. And I think that’s amazing and powerful, and it’s going to mean that the future is… where everybody is allowed to do data science stuff without having lots and lots of education.” 
 
 — Wayne Coburn, Director, Product Management

For the full show notes, including stories mentioned in the episode and who's who, see the ⁠⁠⁠Medium writeup⁠⁠⁠.

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

Knowing your customers

Customers are all unique, whether you’re building a data science product or selling an ecommerce product. In an ideal world, we’d be able to think about all of them on a truly one-on-one basis. Most of us can’t keep track of that many people in our brains, though, which is where the topic of today’s episode comes in: what is the best way to summarize an entire population of customers into a number of groups that is small enough to intuit but fine-grained enough to actually be useful in practice?

Listen along to learn more about:

• Why understanding your customers is the superpower that drives any successful product
• How simple-sounding concepts like recency can be trickier than expected
• How to build a technical solution that draws on a vast number of data stores

For the full show notes, including resources mentioned in the episode and who's who, see the ⁠⁠Medium writeup⁠⁠.

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

When Things Break

Welcome to the November episode of the Klaviyo Data Science Podcast for this year! November is a unique month for ecommerce, which makes it a unique month for any software solution built for ecommerce; it’s a tradition on this podcast to take the opportunity to celebrate some of those unique challenges.

In an ideal world, software and data science products would never break. We do not live in an ideal world, though, so an important question to answer is: what should you do when things do break? This month, we discuss incidents, incident response, and getting things back on track as quickly and effectively as possible to continue delivering value to your customers.

Listen along to learn more about:

• Why not all ways of recognizing something has gone wrong are created equal
• How to cut through disagreements when the stakes are at their highest
• What sorts of unique challenges data science breakages and incidents pose

For the full show notes, including resources mentioned in the episode and who's who, see the ⁠Medium writeup⁠.

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

Off the Happy Path

In most discussions about data science and data science features on this podcast, we make a basic, foundational assumption: the users whose data we are thinking about and customer experience we are trying to improve are, generally speaking, trying to use the platform in a way we recognize and approve of. Not all users of an application have this intention, and the data science behind detecting users who misuse a platform— and even abuse it — constitutes a complex and vast field of study.

Listen along to learn more about:

• Different types of human behaviors motivating platform misuse, and how that translates to different types of data
• What makes many-to-one problems so challenging
• Why keywords alone are not enough

For the full show notes, including who's who, see the Medium writeup.

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

Presenting your work for fun and profit

Presenting technical work is not something you automatically learn how to do — just like the technical skills themselves, it has to be learned and practiced, and opportunities to practice it can be hard to find. This episode, we discuss one opportunity that Klaviyo put together for its R&D teams this summer: the Klaviyo R&D Science Fair. Listen along to hear about:

• How, much like software development, explaining technical work is an iterative process
• The best ways to engage a crowd and get them interested in what you have to say
• The unique and powerful allure of scissors and glue guns

“We put together a little game: try to find all of the accessibility problems in this form, without using the tool that we built…. And then when they react, ‘oh my God, like that one was impossible, I don’t know how you expected me to find that,’ that’s when we can say: exactly! That’s why we needed this feature!”— Maya Nigrin, Senior Software Engineer

For the full show notes, including photos of the event, see the Medium writeup.

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

An introduction to production

What comes after you finish building a data science model? If you’re working on a software project, the answer likely involves that model serving customers in production. Understanding production is crucial for any data scientist or software engineer, so we spend this episode learning about best practices from three experienced Klaviyo engineers.

Listen along to learn more about:

• How to make sure your code is “battle-ready,” whether you’re working on a data science project or not
• Why error messages you think are safe to ignore may not actually be safe to ignore
• One key lesson for safely deploying your code, no matter what environment you work in

“That’s stuck with me through the years: there are these knock-on effects between things. Even if it’s not your code, you should still try to understand how it’s working and whether it can have a ripple effect that comes back and affects your code.”— Chris Conlon, Lead Software Engineer

Check out the full show notes on Medium!

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

Research is a core part of data science. But data science is far from alone in that respect — other fields rely on research just as heavily, and they have their own set of hypotheses, methods, complications, and concerns. This month, we talk to three Klaviyos about research they did before joining the team — both data science research and other kinds — to see what we can learn about conducting effective data science research.

Listen along to learn more about:

• What tiny iron meteorites teach us about the importance of using your results to tell a compelling story
• What data science research into commerce and policy teaches us about iterating on your research questions
• What rubber beams teach us about the importance of getting feedback early

“Everybody has a unique perspective could be the one that opens up a brand new door. You’re looking at doing specific algorithms, you’re looking at doing the research a specific way, but there could be an alternative path.” 

- Mike Galli, Data Scientist

See the full writeup on Medium!

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

Few parts of your product, application, or webpage are more crucial than the very initial experience. In a web application like Klaviyo, that means the home page. Everyone sees it every time they log on to do anything, and interactions with that page set the tone for everything that follows. Meaning: if you’re going to change the home page, you need to really know what you’re doing.

This month, we talk with the Klaviyo engineering team that did just that. We discuss many aspects of that redesign, including:

• How to get buy-in from teams you depend on without taking away your own independence
• The unique difficulties that come with large front-end engineering projects and smart data visualization
• How to filter through the noise when evaluating the success of a feature

“There are very few features ever been released in Klaviyo that have seen that sort of change… At the end of the day, if we can help our users complete tasks faster and more effectively, that’s our highest priority.”
- Griffin Drigotas, Senior Product Designer

See the full writeup on Medium!

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

The question is slightly tongue-in-cheek, but only slightly. Data science is a new field — while many people today are graduating with degrees in data science, the same was not true a decade ago. Many of the people who work (and will work) as data scientists were not classically trained as a data scientist, but as something else. This month, we examine that process: the process of working in a field that’s distinct from data science and becoming a data scientist.

We discuss several parts of that journey, including:

• What attracts someone to data science in the first place
• How to approach gaining the technical skills you need to get a data science job
• How similar some parts of the data scientist job are to washing dishes

Where do data scientists come from?“You really need to practice using these tools. I did my best to come up with excuses to use data science techniques in all my projects… maybe instead of trying to automate a workflow in Excel VBA, I’d try to automate it in python instead.”
- Steven Her, Data Scientist

Read the full writeup on Medium!

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

Back by popular demand: data science is a broad, deep field with an extraordinary amount to learn, and we’re here to help you learn it. We asked four members of the Data Science team at Klaviyo what one of their favorite data science books was, and we got four different answers. Listen on if you’ve wanted to know more ways to learn about:

• How to think about and employ the Bayesian framework (and corgis)
• Learning intro-to-intermediate coding skills necessary for data science work
• The theory that drives natural language processing
• The mindset of a data scientist in general

“it gives you a different lens to apply to different problems. And sometimes taking that different lens, suddenly a problem that was really hard to formulate using traditional frequentist statistics or machine learning techniques, suddenly it can be really easy to frame in this other way” - Tommy Blanchard, Senior Data Science Manager

Read the full writeup on Medium!

Listen to the full episode on Anchor, or in your favorite podcast distribution platform!

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

Starting from scratch

We’ve talked about a lot of aspects of data science on this podcast — building software features, conducting research, learning new methods and skills, recruiting new members — but there’s one we’ve always avoided: building a new team from the ground up. A large reason for that is personnel — while your cohosts may be intrepid, they are not experts in this area.

This month, we bring on two people who are: Eric Silberstein and Ezra Freedman, who founded the Data Science team at Klaviyo. We draw on their wealth of experience, knowledge, and lessons learned the hard way while founding a young team.

As you might expect, these lessons extend beyond data science teams in particular — whether you’re founding another team or starting a new business, or looking to join a team in its early stages, you might be able to learn from our discussions, such as:

• How setting concrete goals is key for a new team
• How to think about your first hire, and your next five
• How to steer a team through large organizational changes while maintaining its culture and essence

“When you view the world, do you think of it as ‘if-then’ statements, or do you tend to think of it as some sort of function to optimize? Our team needed both.”
- Eric Silberstein, VP of Data Science

Read the full writeup on Medium!

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

When the data science world changes

When you work in data science, it’s inevitable that the world will change for you. Sometimes it’s due to global events, macroeconomic trends, or sudden shifts in consumer behavior. Other times it’s due to new features added by a commonly-used piece of software. When your lifeblood is data, all of these can be equally shocking and disruptive.

This month, we discuss one of the latter cases: the changes to the world of email marketing data brought about by the iOS 15 privacy updates. We bring on a panel of product managers, data scientists, and software engineers to discuss:

• How one software update can drastically alter your data landscape
• How to do research while the world is changing, and how to test your conclusions while the ground truth is still in flux
• How using different sources of data can help you adapt your product to a new reality

“That was the biggest thing I came out of this with. Being first doesn’t really matter if what you’re delivering isn’t the right solution for your customers”— Nader Fotouhi, Lead Product Manager

Read the full writeup on Medium!

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

2022 Year in Review

As the new year starts, we take a look back at 2022. We spoke to 8 data scientist and people who work closely with data scientists, and we asked them all the same question: what is the coolest data science thing you learned about in 2022? You’ll hear about fascinating data science topics, including:

• Advances in AI, NLP, and data science in general in 2022
• How understanding data science and ML operations makes a recruiter’s job easier
• New ways to classify models, visualize data, and play video games

“I think these models have been really good for a long while. It’s a snowball effect of people realizing they’ve been good and seeing how much cool stuff can be built with them.”
- Robert Huselid, Data Scientist

Read the full writeup on Medium!

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

Tools of the Trade

We talk a lot on this podcast about the results of data science and software engineering work. We even talk about the process of doing data science and software engineering work. But one thing we haven’t shed much light on, until this month, is: what specific tools help a Data Science team — or any developer or data scientist similarly engaged in building a scalable and intelligent system — actually do their work? We asked several data scientists, machine learning engineers, software engineers, designers, and product managers the same question: what is your favorite tool that helps you do your job? You’ll hear all their answers in this episode, including:

• Why some well-known tools fully deserve the hype
• Specialized packages for specialized purposes
• How to slow down and really force yourself to think about the problem
• How to avoid analysis paralysis

“I like banging out unstyled web forms as much as the next back-end developer, but when you have the experience of spending all day in a tool, those ‘tiny’ things like icon consistency really pay off.”
— Zac Bentley, Lead Site Reliability Engineer II

Read the full show notes, meet this month's guests, and learn more about Klaviyo in our Medium writeup!

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

Anomaly Detection

It’s our third November on the Klaviyo Data Science Podcast, and if you work in ecommerce, you know that November means one thing: Black Friday and (usually) Cyber Monday, i.e. the month of the year where everything changes. Traditionally, we’ve talked about things that help prepare builders of software for when the world is about to change, such as infrastructure, readiness, scale-out testing, and other things along those lines. This year, we’re approaching it from another angle: ecommerce stores go through the exact same struggle every year. How can a platform like Klaviyo help prepare them for the unexpected? One answer: by automatically figuring out when unexpected things are happening, i.e., by detecting anomalous behavior. You’ll hear all about anomaly detection on this episode, including:

• How to pivot your research when your idea is valuable but your results aren’t providing that value
• How to label large swaths of data efficiently
• How to design algorithms for an extraordinarily diverse base of end users

“Imagine a sneaker company who does product drops compared to a regular ecommerce brand. Then imagine customers who are just starting up, with very low traffic…. It was definitely a challenge to generalize to the entire Klaviyo customer base.” — Harsh Mehta, Senior Machine Learning Engineer

Read the full show notes on Medium!

I’ll let you in on a secret: this podcast does not cover everything. We cover a wide array of projects, go into detail on a variety of aspects of them, and speak to a diverse panel of data scientists and people related to the data science world, but we still can’t cover everything. This month, to give you a taste of what we haven’t been able to showcase on this podcast, we’re asking six Klaviyos who work on or with the Data Science team one simple question: what is your favorite data science project you’ve worked on? You’ll hear about all of the following and more:

• How data science and product management can work together to maximize their strengths
• How two different viewpoints on the same project can illuminate different, equally fascinating parts of it
• An unexpectedly powerful way to use data about first names

“As a data scientist, you have to be curious and you have to be really agile, you have to pivot and ask a new question…. I learned a lot about how to leverage that to maximize outcomes when we work together.” 
— Alexandra Edelstein, Director of Product Management See the full show notes on Medium!

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

Using NLP to communicate at scale

Last episode, we discussed the history and practice of natural language processing, or NLP. This month, we’re here to discuss an exciting and cutting-edge application: using NLP to help businesses converse with their customers at scale. See the power of NLP in action as we talk with NLP experts on the Conversation AI team at Klaviyo about:

• How NLP enables a qualitative shift in how businesses communicate
• What intent classification is and why it matters
• Tips on tailoring NLP to a highly specific use case

“There’s a lot of ways to think about the term ‘intent’. One way is what is the customer saying, and you can assign some sort of value to that. But the real intent that we’re interested in is what response are they hoping to get.”
- David Lustig, Data Scientist

See the full show notes on Medium!

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

What’s the deal with natural language?

Natural language processing, or NLP, is one of the dominant forces in modern data science, and it’s produced a host of data science-powered products many people take for granted as a basic fact of life. It hasn’t always been so powerful or pervasive, though — NLP has a long and interesting history, and some of the advances powering today’s technology would have seemed like science fiction only decades ago. This month, we dive into the history and foundations of NLP, examining:

• Why natural languages are so difficult to work with in the first place
• Early attempts by mathematicians and data scientists to use natural languages, and why they failed
• What distinguishes today’s cutting edge models and allows them to succeed

“Language is the natural medium for humans to communicate in. So if you want to build a really immersive, interesting product, for especially non-data scientists to interact with, it almost has to involve NLP.” - Robert Huselid, Data Scientist

See the full show notes, including resources to learn more, on Medium.

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

Thinking big-picture with A/B testing

We’ve discussed A/B testing multiple times on this podcast, for good reason. But there’s an important angle we have yet to cover: in the life of a researcher or marketer, there’s no such thing as an A/B test. There’s an entire system of A/B tests run for specific purposes over time. What is the best way to construct a system of A/B tests to help you learn, improve, and grow over time? How does that translate into tenets to hold while building software to help people run A/B tests? We’ve brought on three members of the data science team at Klaviyo, and you’ll hear about A/B tests in a variety of ways, including:

• Real data-driven trends observed by successful A/B testers on Klaviyo
• Why up-front thinking and vision translate into long-term success
• Why dad jokes might be far more powerful than you think

“The more experimental you can be, the more creative you can be, the more you can learn about your customers to really deliver authentic experiences and see return on your investment.” - Woody Austin, Senior Machine Learning Engineer

Check out the full show notes on Medium for more information!

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

Using data science to help people write

Using machine learning models to generate text, images, and other creative objects is, as they say, a bit of a hot topic right now. There are examples of models like this in action all across the internet and across different fields and disciplines. Today, we discuss one of those fields in more depth: marketing. In particular, the Klaviyo data science team recently released the Subject Line Assistant tool, which helps marketers craft better subject lines. We take a close look at that tool, how it works, and the thinking behind it to examine what it looks like to use AI to help a human write. We’ve brought on four experts from Klaviyo, and you’ll hear about subject lines from a variety of angles, including:

• What a subject line is, and why it’s arguably the most important part of an email
• What holds people back from writing great subject lines and how the team went about solving those problems
• How a specialized human-in-the-loop model for a highly specific context can look

“Subject lines are a very unique type of text generation problem. We’re not asking for a short story where there’s a lot of leeway to really hit a home run — you have a limited amount of space to communicate a brand message, communicate what the email is communicating, make a connection with your audience, and encourage them to interact.” - Josh Villarreal, Data Scientist

Head over to the full show notes to see all the information about this episode!

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

Writing code for computers and people

No matter what sort of data science work you do, it’s fairly inevitable that you’ll have to write code to accomplish your goals. For substantial projects, it’s also fairly inevitable that you’ll have to work with other people to see them to completion. As anyone who’s dived into a legacy code base can tell you, writing code that other people (and yourself in the future) can understand is both an essential skill to have and a difficult practice to master. This episode, we talk specifics about improving your coding skills. We’ve brought on four software engineering experts from Klaviyo, and you’ll hear about writing good code from a variety of angles, including:

• What exactly is good code?
• The biggest misconceptions that come with writing code
• How to prepare for your first code review
• Our panel’s top tips for improving your coding skills, tailored to your level of experience

“You don’t have to make a perfect work of art. It doesn’t have to be bug-free. But it should absolutely be an act of polite and intelligible communication for the next person who will interpret what you create.” - Zac Bentley, Lead Site Reliability Engineer

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

What are data privacy and security?

Data privacy and security are huge and hugely important topics — in all likelihood, you already know a little about them if you’re reading this intro. But they are both crucial to any good data science work, and this month we explore the fundamentals of both topics: why data privacy and security are necessary to deliver the value you promise your customers, who they matter the most to, and how to build privacy and security into your own data science work. The panel includes some of the foremost experts on the topics at Klaviyo from data science, engineering, and security and risk governance, so you’ll get to hear about these topics from a variety of angles, including:

• How approaches to data privacy that seem intuitive can fail, and fail spectacularly
• The consequences of not taking privacy and security carefully enough
• How to make people actively want to work within the security environment you set up

“The worst case is that you violate your customers’ trust. And if you think about personal relationships you have where someone has violated your trust, it’s really hard to build that back.” - Dom Lombardi, Security Risk and Compliance Manager Learn More

• Privacy and security failures mentioned in the episode
   — The SWIFT hack of the Bank of Bangladesh
   — The CafePress data breach
• Differential privacy
  —Overview:  A non-technical primer from Nissim et al. 
   — Example: Apple’s DP Sketch algorithm
   — Example: Google’s RAPPOR
• Data Privacy
   — The Harvard Business Review’s New Rules of Data Privacy

For the full show notes, see the writeup on Medium.

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

Customer-focused research

This month, we focus on research — but specifically research that’s aimed at your customers, delivering the sort of insight they would try to glean by running experiments and analysis using their own data. In particular, we dive into two different case studies drawn from the recent topics explored by the Klaviyo data science team. You’ll hear about:

• Why customer-focused research can be some of your highest-impact work
• Whether or not to use emojis when you’re sending out an email
• How to react when you encounter surprising results in your research

“It was startling. It was the type of number that when you see it, you think: oh, what did I do wrong?”
- Mike Galli

See the full writeup, including links to the blog posts we mention, in the show notes on Medium.

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

2021 Year in Review

Once again, as the new year starts, we begin by recapping the old. Instead of diving deep into a specific topic, I asked 7 members of the Klaviyo data science team to give their personal highlight for 2021 as a year in data science. You’ll hear about fascinating data science topics, including:

• How companies used domain knowledge to hyper-charge their ranking algorithms
• Powerful estimating methods that account for covariance
• How 2021 provided new opportunities — and pitfalls— for state-of-the-art experimental analysis techniques

Be sure to check out the show notes in Medium to learn more about the topics we discuss in this episode!

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

Customer research: your secret weapon

You can study as much mathematical theory, invent as sophisticated a machine learning model, or write as clean production-ready code as you want — if you don’t make sure you’re solving the right problems to begin with, all that effort could be for nothing. It’s not a topic you learn about in most data science coursework, but understanding your end customer is a crucial part of being an effective data scientist. We spend this whole episode describing why and how to do great customer research. Topics include:

• Why customer research is such a big deal in the first place
• How talking with customers can drastically change your thinking
• How to run the perfect customer call

Be sure to check out the show notes in Medium to learn more about the topics we discuss in this episode!

If you have any questions, comments, or concerns, please contact me on Twitter.

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

Fuel for the Creative Fire

It’s no secret: being creative is hard. Creativity requires time and energy, at the bare minimum, and lacking creativity can spiral into writer’s block and other such conditions. That may be okay if you’re just sending out a tweet here or there — but what if your core user base consists of people who need to be creative, day in and day out? The Creative team at Klaviyo recently tackled the problem of helping users get inspired to create content, and I sat down to discuss the thinking that went into the resulting feature, Showcase. You’ll hear about the development process for Showcase, but also about the underlying problems that Showcase is trying to solve and the process of coming up with a solution like Showcase. Specific topics include:

• Using data science to answer questions that seem simple… even when they aren’t
• Ensuring data privacy in solutions that have to scale
• Controversial sandwiches, and why they make great marketing tools

“There are actually a lot of sites where you can subscribe to literally every single email that a company sends out… but you have no sense of: did these emails do well? What about them was good? Is this something I should copy? It’s just throwing out a bunch of data with no context or insight whatsoever.” 
— Charlie Natoli, Senior Data Scientist See the full episode writeup, including links and who's who, on Medium.

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

Slow Problems, Quick Solutions

We’ve devoted quite a bit of time on this podcast to robust, carefully tuned, and vetted-in-a-thousand-ways solutions. This episode, we venture beyond the land of neatly trimmed hedges and into the unknown, where scrappy solutions may be the only ones that are feasible — or even possible. And we’ll hear about settings where a quick calculation on a napkin can be the difference between success and failure — including the biggest weekend of the ecommerce year. You’ll hear about all that and more, including:

• How to solve Fermi problems (and possibly get put on a watch list)
• When quick calculations can save hours of painstaking work
• How even the simplest math can help you prepare for the most complex engineering challenges of the year

“We really are talking huge surges here… The systems you really want to watch out for, between the hours of 9 to 11 a.m. on Black Friday, move as much data as they had to move in the month of June.”
 — Zac Bentley, Lead Site Reliability Engineer

See the full show notes, including the statistical explanations of the paradoxes we discuss, on Medium.

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

Solving difficult problems with data science

This month, we talk with Shane Suazo, the founder of Plytrix Analytics, about using data science to drive efficient business growth. Shane and Plytrix work with Vital Proteins, and we dive deep into their story and highlight the places where using specific — and powerful — data science techniques helped accelerate a growth opportunity into a growth story. You’ll hear about all that and more, including:

• Establishing a single source of truth as a foundation for advanced analyses
• Preventing churn with minimal cost
• The most important advice for translating general data science techniques to the reality of a specific business

“It’s enabled Vital Proteins to send more timely messages with more relevant offers — offers that are better-tailored to our high-value customers specifically.”
 — Shane Suazo, Plytrix Links

• Learn more about Plytrix Analytics (Medium, LinkedIn, Twitter, Facebook)
• Full show notes on Medium

About Klaviyo

Klaviyo empowers creators to own their own destiny and helps growth-focused ecommerce brands drive more sales with super-targeted, highly relevant email, SMS, Facebook, and Instagram marketing. Interested? We’re always looking for great people to join our team.

Who’s who

• Michael Lawson, Senior Data Scientist
• Shane Suazo, Founder, Plytrix

Edited by: Michael Lawson

Logo by: Griffin Drigotas, Ally Hangartner from Klaviyo Design

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

(More) required reading for data science

A question we frequently get asked is: what books should I read to be a better data scientist/machine learning engineer? This may not surprise you, but there isn’t just one answer — in fact, we spent an entire episode talking about three ways to level up your data science knowledge and skills. This month, we’re back with three more:

• One of the foremost foundational texts for understanding machine learning models in a statistical way
• A survey course for a broad variety of machine learning models, with the opportunity to go in depth on topics like deep learning
• A foundational text in designing and analyzing experiments — both in ideal scenarios and in cases where the standard assumptions aren’t met

Mentioned this episode

We discuss the following books and courses in this episode:

• The Elements of Statistical Learning by Trevor Hastie, Robert Tibshirani, and Jerome Friedman: https://web.stanford.edu/~hastie/ElemStatLearn/
• Kirill Eremenko’s A-Z courses on data science, machine learning, artificial intelligence, and deep learning
• Field Experiments: Design, Analysis, and Interpretation by Alan Gerber and Donald Green: https://wwnorton.com/books/9780393979954

About Klaviyo

Klaviyo helps growth-focused ecommerce brands drive more sales with super-targeted, highly relevant email, Facebook, and Instagram marketing. Interested? We’re always looking for great people to join our team.

Who’s who

• Michael Lawson, Senior Data Scientist
• Nuvan Rathnayaka, Statistician at NoviSci
• Chad Furman, Senior Software Engineer
• David Lustig, Data Scientist

Edited by: Michael Lawson

Logo by: Griffin Drigotas, Ally Hangartner from Klaviyo Design

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

Getting real value from data science

This week, we talk with Ben Knox from Super Coffee and Gina Perrelli from Lunar Solar Group about using data science to motivate the growth of a business. No hypothetical business cases this week — Super Coffee is a real business with a real growth story, and we’re here to showcase the ways that they have partnered with Lunar Solar Group and used inquisitive problem-solving methods to answer questions core to Super Coffee’s business needs. You’ll hear about all that and more, including:

• How expert insight translates into valuable questions
• Dealing with findings that stumped even the experts
• The data science feature that has helped Super Coffee the most

Links

• Learn more about Super Coffee
• Learn more about Lunar Solar Group

Who’s who

• Michael Lawson, Senior Data Scientist
• Ben Knox, SVP Digital, Super Coffee
• Gina Perrelli (LinkedIn, Website), Co-Founder, Lunar Solar Group

Edited by: Michael Lawson

Logo by: Griffin Drigotas, Ally Hangartner from Klaviyo Design

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

Making your product experiments count

We’ve talked about quite a few aspects of data science on this podcast, but one that’s perhaps conspicuously absent so far is running experiments on your product. It’s no secret that experiments provide extraordinarily high-quality data to help you make decisions, but it’s also no secret that you only get good experimental results if you run good experiments. You’ll hear about running a good experiment and more, including:

• How experimentation fits into the design cycle
• What sorts of changes can drive unexpectedly large growth
• How to understand and adapt to counterintuitive results

Resources

• Evan Miller’s A/B testing guide: https://www.evanmiller.org/ab-testing/

Who’s who

• Michael Lawson, Senior Data Scientist
• Eric Gravlin, Lead Product Designer
• Hannah McGrath, Product Analyst II

Edited by: Michael Lawson, Aaron Goeglein

Logo by: Griffin Drigotas, Ally Hangartner from Klaviyo Design