Engineering Enablement by Abi Noda – Details, episodes & analysis

In this episode, Abi and Laura dive into the 2024 DX Core 4 benchmarks, sharing insights across data from 500+ companies. They discuss what these benchmarks mean for engineering leaders, how to interpret key metrics like the Developer Experience Index, and offer advice on how to best use benchmarking data in your organization.

Mentions and Links:

• DX core 4 benchmarks
• Measuring developer productivity with the DX Core 4
• Developer experience index (DXI)
• Will Larson’s article on the Core 4 and power of benchmarking data

Discussion points:

• (0:42) What benchmarks are for
• (3:44) Overview of the DX Core 4 benchmarks
• (6:07) PR throughput data 
• (11:05) Key insights related to startups and mobile teams 
• (14:54) Change fail rate data 
• (19:42) How to best use benchmarking data

In this episode, Abi and Laura introduce the DX Core 4, a new framework designed to simplify how organizations measure developer productivity. They discuss the evolution of productivity metrics, comparing Core 4 with frameworks like DORA, SPACE, and DevEx, and emphasize its focus on speed, effectiveness, quality, and impact. They explore why each metric was chosen, the importance of balancing productivity measures with developer experience, and how Core 4 can help engineering leaders align productivity goals with broader business objectives.   

Mentions and Links:

• Measuring developer productivity with the DX Core 4
• Laura’s developer productivity metrics course

Discussion Points:

• (2:42) Introduction to the DX Core 4
• (3:42) Identifying the Core 4's target audience and key stakeholders
• (4:38) Origins and purpose
• (9:20) Building executive alignment
• (14:15) Tying metrics to business value through output-oriented measures
• (24:45) Defining impact
• (32:42) Choosing between DORA, SPACE, and Core 4 frameworks

Click here to view the episode transcript.

This week we’re joined by Sean Mcllroy from Slack’s Release Engineering team to learn about how they’ve fully automated their deployment process. This conversation covers Slack’s original release process, key changes Sean’s team has made, and the latest challenges they’re working on today.

Mentions and links:

• Read Sean’s blog post, The Scary Thing About Deploys
• Follow Sean on LinkedIn

Time Stamps:

• (1:34): The Release Engineering team
• (2:13): How the monolith has served Slack 
• (3:24): How the deployment process used to work 
• (6:23): The complexity of the deploy itself
• (7:39): Early ideas for improving the deployment process
• (9:07): Why anomaly detection is challenging
• (10:32): What a Z-score is
• (13:23): Managing noise with Z-scores
• (16:49): Presenting this information to people that need it
• (19:54): Taking humans out of the process
• (23:13): Handling rollbacks
• (25:27): Not overloading developers with information
• (28:26): Handling large deployments

Click here to view the episode transcript.

This week’s episode is the recording of a live conversation between Abi and Chris Westerhold (Thoughtworks Head of Developer Experience). This conversation is useful for anyone early in their journey with developer portals or platforms: Abi and Chris discuss common approaches to solving these problems, pitfalls to avoid, building vs. buying, and more.

Mentions and Links

• Follow Chris on LinkedIn
• Watch the recording of this conversation 
• Watch part 2 of this conversation on the market landscape
• Learn about PlatformX, DX’s product mentioned in the conversation

Time Stamps:

• (3:09) Why there’s an increased interest in developer portals
• (5:33) Chris’ background with dev portals
• (6:37) Homegrown solutions for developer portals
• (9:22) How developer portal initiatives begin
• (11:24) Internal developer portal vs service catalogs and IDPs
• (16:18) Mistakes companies make with developer portals
• (21:05) Approaches to solving this problem
• (24:28) How can developer portals drive value
• (32:07) Common traps to avoid

Click here to listen to the episode transcript.

On this week's episode, Abi interviews Kent Wills, Director of Engineering Effectiveness at Yelp.  He shares insights into the evolution of their developer productivity efforts over the past decade. From tackling challenges with their monolithic architecture to scaling productivity initiatives for over 1,300 developers. Kent also touches on his experience in building a business case for developer productivity.

Discussion points:

• (1:42) Forming the developer productivity team
• (3:25) Naming the team engineering effectiveness
• (4:30) Getting leadership buy-in for focusing on this work
• (7:54) Managing code ownership in Yelp’s monolith
• (12:23) Supporting the design system
• (16:00) The business case for forming a dedicated team 
• (19:45) How to standardize 
• (23:50) How their approach to standardization might be different in another company
• (27:08) Demonstrating the value of their work 
• (32:21) Building an insights platform
• (38:47) How Yelp is using LLM’s

Mentions and Links

• Connect with Kent Wills on LinkedIn
• Watch Kent’s 2023 talk at Elevate
• Listen to the interview with Peter Seibel (“Let 1,000 flowers bloom”)
• Download the recently published benchmarks on developer productivity team headcount

This week we’re joined by Gail Carmichael, Principal Instructional Engineer at Splunk. At Splunk, Gail’s team is responsible for improving developer onboarding, which they do through a multi-day learning program. Here, Gail shares how this program works and how they measure developer onboarding. The conversation also covers what instructional engineers are generally, and how Gail demonstrates the impact of her team’s work. 

Discussion points:

• (1:16) The Engineering Enablement & Engagement Team at Splunk
• (8:01) What an Instructional Engineer is
• (14:36) The developer onboarding program at Splunk
• (16:05) Components of a good onboarding program
• (21:11) Why having an onboarding program matters
• (28:17) Measuring onboarding at Shopify (Gail’s previous company)
• (31:39) Measuring developer onboarding at Splunk

Mentions and Links

• Connect with Gail on LinkedIn
• Download the report on Developer productivity metrics at top tech companies

In this episode we’re joined by Adam Rogal, who leads Developer Productivity and Platform at DoorDash. Adam describes DoorDash’s journey with their internal developer portal, and gives advice for  other teams looking to follow a similar path. Adam also describes how his team delivered value quickly and drove adoption for their developer platform.

Discussion points:

• (1:47) Why DoorDash explored implementing a developer portal
• (6:59) The initial vision for the developer portal 
• 12:19 Funding ongoing development 
• 16:01 Deciding what to include in the portal 
• 19:15 Coming up with a name for the portal 
• 20:01 Advice for interested beginners
• 23:55 Putting together a business case
• 32:32 Getting adoption for the portal 
• 37:27 Driving initial awareness 
• 41:29 Getting feedback from developers
• 48:33 What Adam would have done differently

Mentions and links:

• Adam Rogal on LinkedIn
• Get started (API)
• New testing and monitoring tools

In this episode, Abi has a fascinating conversation with Rebecca Parsons, ThoughtWorks's CTO, Camilla Crispim, and Erik Dörnenburg on the ThoughtWorks Tech Radar. The trio begins with an overview of Tech Radar and its history before delving into the intricate process of creating each report involving multiple teams and stakeholders. The conversation concludes with a focus on the evolution of Tech Radar's design and process and potential future changes. This episode offers Tech Radar fans an exclusive behind-the-scenes look at its history and production.

Discussion points:

• 1:20-An introduction to the Tech Radar
• 6:06-Common terms used in this episode
• 6:27-The origin of the Tech Radar
• 8:50-Problems that the Tech Radar was aiming to solve
• 12:23-The impact on internal decision making-a tool for driving change
• 14:30-The teams philosophy behind Tech Radar
• 18:33-What sets the Tech Radar apart
• 21:11-Why maintaining independence is crucial for their audience
• 25:08-How Tech Radar publishes their reports
• 29:36-A look into Thoughtworks live meeting sessions
• 34:51-Tech Radars Git repository
• 42:20-Recent changes and upcoming shifts

Mentions and links:

• ThoughtWorks TechRadar
• Rebecca Parsons on LinkedIn
• Camilla Crispim on LinkedIn
• Erik Dörnenburg on LinkedIn
• Thoughtworks Git repository

This week's guest is Eirini Kalliamvakou, a staff researcher at GitHub focused on AI and developer experience. Eirini sits at the forefront of research into GitHub Copilot. Abi and Eirini discuss recent research on how AI coding assistance impacts developer productivity. They talk about how leaders should build business cases for AI tools. They also preview what's to come with AI tools and implications for how developer productivity is measured.

Discussion points:

• (1:49) Overview of GitHub’s research on AI
• (2:59) The research study on Copilot
• (4:48) Defining and measuring productivity for this study
• (7:44) Exact measures and factors studied
• (8:16) Key findings from the study
• (9:45) How the study was conducted 
• (11:17) Most surprising findings for the researchers
• (14:01) The motivation for conducting a follow-up study
• (15:34) How the follow-up study was conducted
• (18:42) Findings from the follow-up study
• (21:13) Is AI just hype? 
• (26:34) How to begin advocating for AI tools
• (34:44) How to translate data into dollars
• (37:06) How to roll out AI tools to an organization
• (38:47) The impact of AI on developer experience
• (43:24) Implications of AI on how we measure productivity

Mentions and links:

• Eirini Kalliamvakou on LinkedIn
• Research on the impact of Copilot 
• Crossing the Chasm by Geoffrey Moore

Christopher Sanson is a product manager at Airbnb who is dedicated to enhancing developer productivity and tooling. Today, we learn more about Airbnb's developer productivity team and how various teams use metrics, both within and outside the organization. From there, we dive even deeper into their measurement journey, highlighting their implementation of DORA metrics and the challenges they overcame throughout the process.

Discussion points:

• (2:43) Who is the developer productivity customer
• (4:49) The evolution of developer productivity at Airbnb
• (9:26) Approach before DORA metrics
• (14:29) Getting buy-in for DORA metrics
• (17:49) Planning how to deliver new metrics to the organization
• (21:12) How Airbnb calculates deployment frequency
• (23:29) Implementing a proof of concept
• (27:20) Statistical measurement strategies and tactics
• (31:11) Operationalizing developer productivity metrics
• (34:26) How Airbnb reviews data
• (35:41) How Airbnb uses DORA metrics

Mentions and links:

• Christopher Sanson on LinkedIn
• Christopher’s talk at DPE Summit
• How Top Companies Measure Developer Productivity