Retour
Explorez tous les épisodes du podcast Data Engineering Podcast
Plongez dans la liste complète des épisodes de Data Engineering Podcast. Chaque épisode est catalogué accompagné de descriptions détaillées, ce qui facilite la recherche et l'exploration de sujets spécifiques. Suivez tous les épisodes de votre podcast préféré et ne manquez aucun contenu pertinent.
| Titre | Date | Durée | |
|---|---|---|---|
| The Evolution of DataOps: Insights from DataKitchen's CEO | 04 Aug 2024 | 00:53:30 | |
Summary In this episode of the Data Engineering Podcast, host Tobias Macey welcomes back Chris Berg, CEO of DataKitchen, to discuss his ongoing mission to simplify the lives of data engineers. Chris explains the challenges faced by data engineers, such as constant system failures, the need for rapid changes, and high customer demands. Chris delves into the concept of DataOps, its evolution, and the misappropriation of related terms like data mesh and data observability. He emphasizes the importance of focusing on processes and systems rather than just tools to improve data engineering workflows. Chris also introduces DataKitchen's open-source tools, DataOps TestGen and DataOps Observability, designed to automate data quality validation and monitor data journeys in production. Announcements
Parting Question
| |||
| Achieving Data Reliability: The Role of Data Contracts in Modern Data Management | 28 Jul 2024 | 00:49:26 | |
Summary Data contracts are both an enforcement mechanism for data quality, and a promise to downstream consumers. In this episode Tom Baeyens returns to discuss the purpose and scope of data contracts, emphasizing their importance in achieving reliable analytical data and preventing issues before they arise. He explains how data contracts can be used to enforce guarantees and requirements, and how they fit into the broader context of data observability and quality monitoring. The discussion also covers the challenges and benefits of implementing data contracts, the organizational impact, and the potential for standardization in the field. Announcements
Parting Question
| |||
| Data Migration Strategies For Large Scale Systems | 27 May 2024 | 01:00:00 | |
Summary
Any software system that survives long enough will require some form of migration or evolution. When that system is responsible for the data layer the process becomes more challenging. Sriram Panyam has been involved in several projects that required migration of large volumes of data in high traffic environments. In this episode he shares some of the valuable lessons that he learned about how to make those projects successful. Announcements
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA Sponsored By:
| |||
| Analytics Engineering Without The Friction Of Complex Pipeline Development With Optimus and dbt | 30 Oct 2022 | 00:40:10 | |
Summary
One of the most impactful technologies for data analytics in recent years has been dbt. It’s hard to have a conversation about data engineering or analysis without mentioning it. Despite its widespread adoption there are still rough edges in its workflow that cause friction for data analysts. To help simplify the adoption and management of dbt projects Nandam Karthik helped create Optimus. In this episode he shares his experiences working with organizations to adopt analytics engineering patterns and the ways that Optimus and dbt were combined to let data analysts deliver insights without the roadblocks of complex pipeline management. Announcements
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA Sponsored By:
| |||
| How To Bring Agile Practices To Your Data Projects | 23 Oct 2022 | 01:12:18 | |
Summary
Agile methodologies have been adopted by a majority of teams for building software applications. Applying those same practices to data can prove challenging due to the number of systems that need to be included to implement a complete feature. In this episode Shane Gibson shares practical advice and insights from his years of experience as a consultant and engineer working in data about how to adopt agile principles in your data work so that you can move faster and provide more value to the business, while building systems that are maintainable and adaptable. Announcements
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA Sponsored By:
| |||
| Going From Transactional To Analytical And Self-managed To Cloud On One Database With MariaDB | 23 Oct 2022 | 00:52:04 | |
Summary
The database market has seen unprecedented activity in recent years, with new options addressing a variety of needs being introduced on a nearly constant basis. Despite that, there are a handful of databases that continue to be adopted due to their proven reliability and robust features. MariaDB is one of those default options that has continued to grow and innovate while offering a familiar and stable experience. In this episode field CTO Manjot Singh shares his experiences as an early user of MySQL and MariaDB and explains how the suite of products being built on top of the open source foundation address the growing needs for advanced storage and analytical capabilities. Announcements
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA | |||
| An Exploration Of The Open Data Lakehouse And Dremio's Contribution To The Ecosystem | 16 Oct 2022 | 00:50:44 | |
Summary
The "data lakehouse" architecture balances the scalability and flexibility of data lakes with the ease of use and transaction support of data warehouses. Dremio is one of the companies leading the development of products and services that support the open lakehouse. In this episode Jason Hughes explains what it means for a lakehouse to be "open" and describes the different components that the Dremio team build and contribute to. Announcements
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA | |||
| Speeding Up The Time To Insight For Supply Chains And Logistics With The Pathway Database That Thinks | 16 Oct 2022 | 01:02:36 | |
Summary
Logistics and supply chains are under increased stress and scrutiny in recent years. In order to stay ahead of customer demands, businesses need to be able to react quickly and intelligently to changes, which requires fast and accurate insights into their operations. Pathway is a streaming database engine that embeds artificial intelligence into the storage, with functionality designed to support the spatiotemporal data that is crucial for shipping and logistics. In this episode Adrian Kosowski explains how the Pathway product got started, how its design simplifies the creation of data products that support supply chain operations, and how developers can help to build an ecosystem of applications that allow businesses to accelerate their time to insight. Announcements
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA | |||
| Making The Open Data Lakehouse Affordable Without The Overhead At Iomete | 10 Oct 2022 | 00:55:24 | |
Summary
The core of any data platform is the centralized storage and processing layer. For many that is a data warehouse, but in order to support a diverse and constantly changing set of uses and technologies the data lakehouse is a paradigm that offers a useful balance of scale and cost, with performance and ease of use. In order to make the data lakehouse available to a wider audience the team at Iomete built an all-in-one service that handles management and integration of the various technologies so that you can worry about answering important business questions. In this episode Vusal Dadalov explains how the platform is implemented, the motivation for a truly open architecture, and how they have invested in integrating with the broader ecosystem to make it easy for you to get started. Announcements
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA | |||
| Investing In Understanding The Customer Journey At American Express | 10 Oct 2022 | 00:40:43 | |
Summary
For any business that wants to stay in operation, the most important thing they can do is understand their customers. American Express has invested substantial time and effort in their Customer 360 product to achieve that understanding. In this episode Purvi Shah, the VP of Enterprise Big Data Platforms at American Express, explains how they have invested in the cloud to power this visibility and the complex suite of integrations they have built and maintained across legacy and modern systems to make it possible. Announcements
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA | |||
| Gain Visibility And Insight Into Your Supply Chains Through Operational Analytics Powered By Roambee | 03 Oct 2022 | 01:00:04 | |
Summary
The global economy is dependent on complex and dynamic networks of supply chains powered by sophisticated logistics. This requires a significant amount of data to track shipments and operational characteristics of materials and goods. Roambee is a platform that collects, integrates, and analyzes all of that information to provide companies with the critical insights that businesses need to stay running, especially in a time of such constant change. In this episode Roambee CEO, Sanjay Sharma, shares the types of questions that companies are asking about their logistics, the technical work that they do to provide ways to answer those questions, and how they approach the challenge of data quality in its many forms. Announcements
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA | |||
| Make Data Lineage A Ubiquitous Part Of Your Work By Simplifying Its Implementation With Alvin | 03 Oct 2022 | 00:56:16 | |
Summary
Data lineage is something that has grown from a convenient feature to a critical need as data systems have grown in scale, complexity, and centrality to business. Alvin is a platform that aims to provide a low effort solution for data lineage capabilities focused on simplifying the work of data engineers. In this episode co-founder Martin Sahlen explains the impact that easy access to lineage information can have on the work of data engineers and analysts, and how he and his team have designed their platform to offer that information to engineers and stakeholders in the places that they interact with data. Announcements
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA | |||
| Power Your Real-Time Analytics Without The Headache Using Fivetran's Change Data Capture Integrations | 26 Sep 2022 | 00:49:37 | |
Summary
Data integration from source systems to their downstream destinations is the foundational step for any data product. With the increasing expecation for information to be instantly accessible, it drives the need for reliable change data capture. The team at Fivetran have recently introduced that functionality to power real-time data products. In this episode Mark Van de Wiel explains how they integrated CDC functionality into their existing product, discusses the nuances of different approaches to change data capture from various sources. Announcements
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA | |||
| Zenlytic Is Building You A Better Coworker With AI Agents | 19 May 2024 | 00:54:19 | |
Summary
The purpose of business intelligence systems is to allow anyone in the business to access and decode data to help them make informed decisions. Unfortunately this often turns into an exercise in frustration for everyone involved due to complex workflows and hard-to-understand dashboards. The team at Zenlytic have leaned on the promise of large language models to build an AI agent that lets you converse with your data. In this episode they share their journey through the fast-moving landscape of generative AI and unpack the difference between an AI chatbot and an AI agent. Announcements
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA Sponsored By:
| |||
| Build A Common Understanding Of Your Data Reliability Rules With Soda Core and Soda Checks Language | 26 Sep 2022 | 00:41:02 | |
Summary
Regardless of how data is being used, it is critical that the information is trusted. The practice of data reliability engineering has gained momentum recently to address that question. To help support the efforts of data teams the folks at Soda Data created the Soda Checks Language and the corresponding Soda Core utility that acts on this new DSL. In this episode Tom Baeyens explains their reasons for creating a new syntax for expressing and validating checks for data assets and processes, as well as how to incorporate it into your own projects. Announcements
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA | |||
| Operational Analytics To Increase Efficiency For Multi-Location Businesses With OpsAnalitica | 19 Sep 2022 | 01:32:03 | |
Summary
In order to improve efficiency in any business you must first know what is contributing to wasted effort or missed opportunities. When your business operates across multiple locations it becomes even more challenging and important to gain insights into how work is being done. In this episode Tommy Yionoulis shares his experiences working in the service and hospitality industries and how that led him to found OpsAnalitica, a platform for collecting and analyzing metrics on multi location businesses and their operational practices. He discusses the challenges of making data collection purposeful and efficient without distracting employees from their primary duties and how business owners can use the provided analytics to support their staff in their duties. Announcements
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA | |||
| Building A Shared Understanding Of Data Assets In A Business Through A Single Pane Of Glass With Workstream | 19 Sep 2022 | 00:54:52 | |
Summary
There is a constant tension in business data between growing siloes, and breaking them down. Even when a tool is designed to integrate information as a guard against data isolation, it can easily become a silo of its own, where you have to make a point of using it to seek out information. In order to help distribute critical context about data assets and their status into the locations where work is being done Nicholas Freund co-founded Workstream. In this episode he discusses the challenge of maintaining shared visibility and understanding of data work across the various stakeholders and his efforts to make it a seamless experience. Announcements
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA | |||
| Build Confidence In Your Data Platform With Schema Compatibility Reports That Span Systems And Domains Using Schemata | 12 Sep 2022 | 00:59:40 | |
Summary
Data engineering systems are complex and interconnected with myriad and often opaque chains of dependencies. As they scale, the problems of visibility and dependency management can increase at an exponential rate. In order to turn this into a tractable problem one approach is to define and enforce contracts between producers and consumers of data. Ananth Packildurai created Schemata as a way to make the creation of schema contracts a lightweight process, allowing the dependency chains to be constructed and evolved iteratively and integrating validation of changes into standard delivery systems. In this episode he shares the design of the project and how it fits into your development practices. Announcements
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA | |||
| Building Data Pipelines That Run From Source To Analysis And Activation With Hevo Data | 12 Sep 2022 | 00:57:16 | |
Summary
Any business that wants to understand their operations and customers through data requires some form of pipeline. Building reliable data pipelines is a complex and costly undertaking with many layered requirements. In order to reduce the amount of time and effort required to build pipelines that power critical insights Manish Jethani co-founded Hevo Data. In this episode he shares his journey from building a consumer product to launching a data pipeline service and how his frustrations as a product owner have informed his work at Hevo Data. Announcements
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA Sponsored By:
| |||
| A Reflection On Data Observability As It Reaches Broader Adoption | 05 Sep 2022 | 00:58:39 | |
Summary
Data observability is a product category that has seen massive growth and adoption in recent years. Monte Carlo is in the vanguard of companies who have been enabling data teams to observe and understand their complex data systems. In this episode founders Barr Moses and Lior Gavish rejoin the show to reflect on the evolution and adoption of data observability technologies and the capabilities that are being introduced as the broader ecosystem adopts the practices. Announcements
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA | |||
| Introduce Climate Analytics Into Your Data Platform Without The Heavy Lifting Using Sust Global | 05 Sep 2022 | 00:54:19 | |
Summary
The global climate impacts everyone, and the rate of change introduces many questions that businesses need to consider. Getting answers to those questions is challenging, because the climate is a multidimensional and constantly evolving system. Sust Global was created to provide curated data sets for organizations to be able to analyze climate information in the context of their business needs. In this episode Gopal Erinjippurath discusses the data engineering challenges of building and serving those data sets, and how they are distilling complex climate information into consumable facts so you don’t have to be an expert to understand it. Announcements
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA | |||
| An Exploration Of What Data Automation Can Provide To Data Engineers And Ascend's Journey To Make It A Reality | 29 Aug 2022 | 01:03:33 | |
Summary
The dream of every engineer is to automate all of their tasks. For data engineers, this is a monumental undertaking. Orchestration engines are one step in that direction, but they are not a complete solution. In this episode Sean Knapp shares his views on what constitutes proper automation and the work that he and his team at Ascend are doing to help make it a reality. Announcements
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA Sponsored By:
| |||
| Alumni Of AirBnB's Early Years Reflect On What They Learned About Building Data Driven Organizations | 28 Aug 2022 | 01:10:15 | |
Summary
AirBnB pioneered a number of the organizational practices that have become the goal of modern data teams. Out of that culture a number of successful businesses were created to provide the tools and methods to a broader audience. In this episode several almuni of AirBnB’s formative years who have gone on to found their own companies join the show to reflect on their shared successes, missed opportunities, and lessons learned. Announcements
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA | |||
| An Exploration Of The Expectations, Ecosystem, and Realities Of Real-Time Data Applications | 22 Aug 2022 | 01:06:20 | |
Summary
Data has permeated every aspect of our lives and the products that we interact with. As a result, end users and customers have come to expect interactions and updates with services and analytics to be fast and up to date. In this episode Shruti Bhat gives her view on the state of the ecosystem for real-time data and the work that she and her team at Rockset is doing to make it easier for engineers to build those experiences. Announcements
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA | |||
| Release Management For Data Platform Services And Logic | 12 May 2024 | 00:20:09 | |
Summary
Building a data platform is a substrantial engineering endeavor. Once it is running, the next challenge is figuring out how to address release management for all of the different component parts. The services and systems need to be kept up to date, but so does the code that controls their behavior. In this episode your host Tobias Macey reflects on his current challenges in this area and some of the factors that contribute to the complexity of the problem. Announcements
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA Sponsored By:
| |||
| Understanding The Role Of The Chief Data Officer | 22 Aug 2022 | 00:47:11 | |
Summary
The position of Chief Data Officer (CDO) is relatively new in the business world and has not been universally adopted. As a result, not everyone understands what the responsibilities of the role are, when you need one, and how to hire for it. In this episode Tracy Daniels, CDO of Truist, shares her journey into the position, her responsibilities, and her relationship to the data professionals in her organization. Announcements
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA | |||
| Bringing Automation To Data Labeling For Machine Learning With Watchful | 14 Aug 2022 | 01:20:29 | |
Summary
Data engineers have typically left the process of data labeling to data scientists or other roles because of its nature as a manual and process heavy undertaking, focusing instead on building automation and repeatable systems. Watchful is a platform to make labeling a repeatable and scalable process that relies on codifying domain expertise. In this episode founder Shayan Mohanty explains how he and his team are bringing software best practices and automation to the world of machine learning data preparation and how it allows data engineers to be involved in the process. Announcements
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA | |||
| Collecting And Retaining Contextual Metadata For Powerful And Effective Data Discovery | 14 Aug 2022 | 00:53:24 | |
Summary
Data is useless if it isn’t being used, and you can’t use it if you don’t know where it is. Data catalogs were the first solution to this problem, but they are only helpful if you know what you are looking for. In this episode Shinji Kim discusses the challenges of data discovery and how to collect and preserve additional context about each piece of information so that you can find what you need when you don’t even know what you’re looking for yet. Announcements
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA | |||
| Useful Lessons And Repeatable Patterns Learned From Data Mesh Implementations At AgileLab | 06 Aug 2022 | 00:48:31 | |
Summary
Data mesh is a frequent topic of conversation in the data community, with many debates about how and when to employ this architectural pattern. The team at AgileLab have first-hand experience helping large enterprise organizations evaluate and implement their own data mesh strategies. In this episode Paolo Platter shares the lessons they have learned in that process, the Data Mesh Boost platform that they have built to reduce some of the boilerplate required to make it successful, and some of the considerations to make when deciding if a data mesh is the right choice for you. Announcements
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA | |||
| Optimize Your Machine Learning Development And Serving With The Open Source Vector Database Milvus | 06 Aug 2022 | 00:58:52 | |
Summary
The optimal format for storage and retrieval of data is dependent on how it is going to be used. For analytical systems there are decades of investment in data warehouses and various modeling techniques. For machine learning applications relational models require additional processing to be directly useful, which is why there has been a growth in the use of vector databases. These platforms store direct representations of the vector embeddings that machine learning models rely on for computing relevant predictions so that there is no additional processing required to go from input data to inference output. In this episode Frank Liu explains how the open source Milvus vector database is implemented to speed up machine learning development cycles, how to think about proper storage and scaling of these vectors, and how data engineering and machine learning teams can collaborate on the creation and maintenance of these data sets. Announcements
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA | |||
| Interactive Exploratory Data Analysis On Petabyte Scale Data Sets With Arkouda | 31 Jul 2022 | 00:40:37 | |
Summary
Exploratory data analysis works best when the feedback loop is fast and iterative. This is easy to achieve when you are working on small datasets, but as they scale up beyond what can fit on a single machine those short iterations quickly become long and tedious. The Arkouda project is a Python interface built on top of the Chapel compiler to bring back those interactive speeds for exploratory analysis on horizontally scalable compute that parallelizes operations on large volumes of data. In this episode David Bader explains how the framework operates, the algorithms that are built into it to support complex analyses, and how you can start using it today. Announcements
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA | |||
| What "Data Lineage Done Right" Looks Like And How They're Doing It At Manta | 31 Jul 2022 | 01:05:18 | |
Summary
Data lineage is the roadmap for your data platform, providing visibility into all of the dependencies for any report, machine learning model, or data warehouse table that you are working with. Because of its centrality to your data systems it is valuable for debugging, governance, understanding context, and myriad other purposes. This means that it is important to have an accurate and complete lineage graph so that you don’t have to perform your own detective work when time is in short supply. In this episode Ernie Ostic shares the approach that he and his team at Manta are taking to build a complete view of data lineage across the various data systems in your organization and the useful applications of that information in the work of every data stakeholder. Announcements
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA | |||
| Re-Bundling The Data Stack With Data Orchestration And Software Defined Assets Using Dagster | 24 Jul 2022 | 00:58:14 | |
Summary
The current stage of evolution in the data management ecosystem has resulted in domain and use case specific orchestration capabilities being incorporated into various tools. This complicates the work involved in making end-to-end workflows visible and integrated. Dagster has invested in bringing insights about external tools’ dependency graphs into one place through its "software defined assets" functionality. In this episode Nick Schrock discusses the importance of orchestration and a central location for managing data systems, the road to Dagster’s 1.0 release, and the new features coming with Dagster Cloud’s general availability. Announcements
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA | |||
| Writing The Book That Offers A Single Reference For The Fundamentals Of Data Engineering | 24 Jul 2022 | 01:01:02 | |
Summary
Data engineering is a difficult job, requiring a large number of skills that often don’t overlap. Any effort to understand how to start a career in the role has required stitching together information from a multitude of resources that might not all agree with each other. In order to provide a single reference for anyone tasked with data engineering responsibilities Joe Reis and Matt Housley took it upon themselves to write the book "Fundamentals of Data Engineering". In this episode they share their experiences researching and distilling the lessons that will be useful to data engineers now and into the future, without being tied to any specific technologies that may fade from fashion. Announcements
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA | |||
| Making The Total Cost Of Ownership For External Data Manageable With Crux | 17 Jul 2022 | 01:07:12 | |
Summary
There are extensive and valuable data sets that are available outside the bounds of your organization. Whether that data is public, paid, or scraped it requires investment and upkeep to acquire and integrate it with your systems. Crux was built to reduce the total cost of acquisition and ownership for integrating external data, offering a fully managed service for delivering those data assets in the manner that best suits your infrastructure. In this episode Crux CTO Mark Etherington discusses the different costs involved in managing external data, how to think about the total return on investment for your data, and how the Crux platform is architected to reduce the toil involved in managing third party data. Announcements
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA Sponsored By:
| |||
| Barking Up The Wrong GPTree: Building Better AI With A Cognitive Approach | 05 May 2024 | 00:54:17 | |
Summary Artificial intelligence has dominated the headlines for several months due to the successes of large language models. This has prompted numerous debates about the possibility of, and timeline for, artificial general intelligence (AGI). Peter Voss has dedicated decades of his life to the pursuit of truly intelligent software through the approach of cognitive AI. In this episode he explains his approach to building AI in a more human-like fashion and the emphasis on learning rather than statistical prediction. Announcements
Parting Question
| |||
| Joe Reis Flips The Script And Interviews Tobias Macey About The Data Engineering Podcast | 17 Jul 2022 | 00:56:39 | |
Summary
Data engineering is a large and growing subject, with new technologies, specializations, and "best practices" emerging at an accelerating pace. This podcast does its best to explore this fractal ecosystem, and has been at it for the past 5+ years. In this episode Joe Reis, founder of Ternary Data and co-author of "Fundamentals of Data Engineering", turns the tables and interviews the host, Tobias Macey, about his journey into podcasting, how he runs the show behind the scenes, and the other things that occupy his time. Announcements
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA | |||
| Charting the Path of Riskified's Data Platform Journey | 10 Jul 2022 | 00:39:57 | |
Summary
Building a data platform is a journey, not a destination. Beyond the work of assembling a set of technologies and building integrations across them, there is also the work of growing and organizing a team that can support and benefit from that platform. In this episode Inbar Yogev and Lior Winner share the journey that they and their teams at Riskified have been on for their data platform. They also discuss how they have established a guild system for training and supporting data professionals in the organization. Announcements
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA | |||
| Maintain Your Data Engineers' Sanity By Embracing Automation | 10 Jul 2022 | 01:05:08 | |
Summary
Building and maintaining reliable data assets is the prime directive for data engineers. While it is easy to say, it is endlessly complex to implement, requiring data professionals to be experts in a wide range of disparate topics while designing and implementing complex topologies of information workflows. In order to make this a tractable problem it is essential that engineers embrace automation at every opportunity. In this episode Chris Riccomini shares his experiences building and scaling data operations at WePay and LinkedIn, as well as the lessons he has learned working with other teams as they automated their own systems. Announcements
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA | |||
| Be Confident In Your Data Integration By Quickly Validating Matching Records With data-diff | 03 Jul 2022 | 01:10:57 | |
Summary
The perennial challenge of data engineers is ensuring that information is integrated reliably. While it is straightforward to know whether a synchronization process succeeded, it is not always clear whether every record was copied correctly. In order to quickly identify if and how two data systems are out of sync Gleb Mezhanskiy and Simon Eskildsen partnered to create the open source data-diff utility. In this episode they explain how the utility is implemented to run quickly and how you can start using it in your own data workflows to ensure that your data warehouse isn’t missing any records from your source systems. Announcements
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA Special Guest: Gleb Mezhanskiy. | |||
| The View From The Lakehouse Of Architectural Patterns For Your Data Platform | 03 Jul 2022 | 00:58:44 | |
Summary
The ecosystem for data tools has been going through rapid and constant evolution over the past several years. These technological shifts have brought about corresponding changes in data and platform architectures for managing data and analytical workflows. In this episode Colleen Tartow shares her insights into the motivating factors and benefits of the most prominent patterns that are in the popular narrative; data mesh and the modern data stack. She also discusses her views on the role of the data lakehouse as a building block for these architectures and the ongoing influence that it will have as the technology matures. Announcements
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA | |||
| Bring Geospatial Analytics Across Disparate Datasets Into Your Toolkit With The Unfolded Platform | 27 Jun 2022 | 01:07:01 | |
Summary
The proliferation of sensors and GPS devices has dramatically increased the number of applications for spatial data, and the need for scalable geospatial analytics. In order to reduce the friction involved in aggregating disparate data sets that share geographic similarities the Unfolded team built a platform that supports working across raster, vector, and tabular data in a single system. In this episode Isaac Brodsky explains how the Unfolded platform is architected, their experience joining the team at Foursquare, and how you can start using it for analyzing your spatial data today. Announcements
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA Sponsored By:
| |||
| Strategies And Tactics For A Successful Master Data Management Implementation | 27 Jun 2022 | 01:09:08 | |
Summary
The most complicated part of data engineering is the effort involved in making the raw data fit into the narrative of the business. Master Data Management (MDM) is the process of building consensus around what the information actually means in the context of the business and then shaping the data to match those semantics. In this episode Malcolm Hawker shares his years of experience working in this domain to explore the combination of technical and social skills that are necessary to make an MDM project successful both at the outset and over the long term. Announcements
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA | |||
| Combining The Simplicity Of Spreadsheets With The Power Of Modern Data Infrastructure At Canvas | 19 Jun 2022 | 00:42:58 | |
Summary
Data analysis is a valuable exercise that is often out of reach of non-technical users as a result of the complexity of data systems. In order to lower the barrier to entry Ryan Buick created the Canvas application with a spreadsheet oriented workflow that is understandable to a wide audience. In this episode Ryan explains how he and his team have designed their platform to bring everyone onto a level playing field and the benefits that it provides to the organization. Announcements
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA | |||
| Level Up Your Data Platform With Active Metadata | 19 Jun 2022 | 00:52:36 | |
Summary
Metadata is the lifeblood of your data platform, providing information about what is happening in your systems. A variety of platforms have been developed to capture and analyze that information to great effect, but they are inherently limited in their utility due to their nature as storage systems. In order to level up their value a new trend of active metadata is being implemented, allowing use cases like keeping BI reports up to date, auto-scaling your warehouses, and automated data governance. In this episode Prukalpa Sankar joins the show to talk about the work she and her team at Atlan are doing to push this capability into the mainstream. Announcements
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA | |||
| Discover And De-Clutter Your Unstructured Data With Aparavi | 13 Jun 2022 | 00:49:12 | |
Summary
Unstructured data takes many forms in an organization. From a data engineering perspective that often means things like JSON files, audio or video recordings, images, etc. Another category of unstructured data that every business deals with is PDFs, Word documents, workstation backups, and countless other types of information. Aparavi was created to tame the sprawl of information across machines, datacenters, and clouds so that you can reduce the amount of duplicate data and save time and money on managing your data assets. In this episode Rod Christensen shares the story behind Aparavi and how you can use it to cut costs and gain value for the long tail of your unstructured data. Announcements
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA | |||
| Build Your Second Brain One Piece At A Time | 28 Apr 2024 | 00:50:10 | |
Summary Generative AI promises to accelerate the productivity of human collaborators. Currently the primary way of working with these tools is through a conversational prompt, which is often cumbersome and unwieldy. In order to simplify the integration of AI capabilities into developer workflows Tsavo Knott helped create Pieces, a powerful collection of tools that complements the tools that developers already use. In this episode he explains the data collection and preparation process, the collection of model types and sizes that work together to power the experience, and how to incorporate it into your workflow to act as a second brain. Announcements
Parting Question
| |||
| Hire And Scale Your Data Team With Intention | 13 Jun 2022 | 01:00:54 | |
Summary
Building a well rounded and effective data team is an iterative process, and the first hire can set the stage for future success or failure. Trupti Natu has been the first data hire multiple times and gone through the process of building teams across the different stages of growth. In this episode she shares her thoughts and insights on how to be intentional about establishing your own data team. Announcements
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA | |||
| Simplify Data Security For Sensitive Information With The Skyflow Data Privacy Vault | 06 Jun 2022 | 00:54:05 | |
Summary
The best way to make sure that you don’t leak sensitive data is to never have it in the first place. The team at Skyflow decided that the second best way is to build a storage system dedicated to securely managing your sensitive information and making it easy to integrate with your applications and data systems. In this episode Sean Falconer explains the idea of a data privacy vault and how this new architectural element can drastically reduce the potential for making a mistake with how you manage regulated or personally identifiable information. Announcements
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA | |||
| Bringing The Modern Data Stack To Everyone With Y42 | 06 Jun 2022 | 00:59:02 | |
Summary
Cloud services have made highly scalable and performant data platforms economical and manageable for data teams. However, they are still challenging to work with and manage for anyone who isn’t in a technical role. Hung Dang understood the need to make data more accessible to the entire organization and created Y42 as a better user experience on top of the "modern data stack". In this episode he shares how he designed the platform to support the full spectrum of technical expertise in an organization and the interesting engineering challenges involved. Announcements
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA Sponsored By:
| |||
© My Podcast Data