Data engineering has historically involved extracting data from disperate sources, transforming it to a standard layout, and then loading it into a new database for analytics. Usually these data engineering pipeline jobs would run on a schedule such as nightly or weekly. In today's fastpaced high-tech world however the need for data closer to real-time, meaning when it was first generated, is higher than ever. In today's episode we hear from Dustin Vannoy who is a consultant and blogger in the streaming data space about how to use Apache Spark, the most popular streaming analytics platform.
How to connect with Dustin:
- WEBSITE: https://dustinvannoy.com/
- TWITTER: / dustinvannoy
- LINKEDIN: / dustinvannoy
- YOUTUBE: / @dustinvannoy
Learn data skills at our academy and elevate your career. Start for free at https://ftdacademy.com/YT
Chapters:
0:00:00 Intro
Dustin's Background
Transitioning from legacy databases to Big Data and Streaming
Microbatching vs Streaming
What is Spark and why use it?
Apache Spark vs Data Bricks
Pay for a hosted Spark version or roll your own?
Databricks setup
How Databricks executes queries
Scaling approaches to Spark
Connecting to external databases in databricks
Visualizing data in Databricks
Using Spark for ETL work
What is real-time processing?
How to build a streaming job in Spark using Kafka
Streaming architecture overview
Pulling data from Kafka into Spark streaming
Why apps use Kafka
Why use Spark versus alternatives
What is Confluent?
Ways to learn Spark
How hard is Spark to learn?
Troubleshooting errors in Spark
How hard is it to transition to Spark from traditional databases?
Interviewing for a Spark job
Outro