Vanishing Gradients – Details, episodes & analysis

In this brief introduction, Hugo introduces the rationale behind launching a new data science podcast and gets excited about his upcoming guests: Jeremy Howard, Rachael Tatman, and Heather Nolis! Original music, bleeps, and blops by local Sydney legend PlaneFace (https://planeface.bandcamp.com/album/fishing-from-an-asteroid)!

Hugo speaks with Ravin Kumar,*Senior Research Data Scientist at Google Labs. Ravin’s career has taken him from building rockets at SpaceX to driving data science and technology at Sweetgreen, and now to advancing generative AI research and applications at Google Labs and DeepMind. His multidisciplinary experience gives him a rare perspective on building AI systems that combine technical rigor with practical utility. In this episode, we dive into: • Ravin’s fascinating career path, including the skills and mindsets needed to work effectively with AI and machine learning models at different stages of the pipeline. • How to build generative AI systems that are scalable, reliable, and aligned with user needs. • Real-world applications of generative AI, such as using open weight models such as Gemma to help a bakery streamline operations—an example of delivering tangible business value through AI. • The critical role of UX in AI adoption, and how Ravin approaches designing tools like Notebook LM with the user journey in mind. We also include a live demo where Ravin uses Notebook LM to analyze my website, extract insights, and even generate a podcast-style conversation about me. While some of the demo is visual, much can be appreciated through audio, and we’ve added a link to the video in the show notes for those who want to see it in action. We’ve also included the generated segment at the end of the episode for you to enjoy. LINKS The livestream on YouTube (https://www.youtube.com/live/ffS6NWqoo_k) Google Labs (https://labs.google/) Ravin's GenAI Handbook (https://ravinkumar.com/GenAiGuidebook/book_intro.html) Breadboard: A library for prototyping generative AI applications (https://breadboard-ai.github.io/breadboard/) As mentioned in the episode, Hugo is teaching a four-week course, Building LLM Applications for Data Scientists and SWEs, co-led with Stefan Krawczyk (Dagworks, ex-StitchFix). The course focuses on building scalable, production-grade generative AI systems, with hands-on sessions, $1,000+ in cloud credits, live Q&As, and guest lectures from industry experts. Listeners of Vanishing Gradients can get 25% off the course using this special link (https://maven.com/hugo-stefan/building-llm-apps-ds-and-swe-from-first-principles?promoCode=VG25) or by applying the code VG25 at checkout.

Hugo speaks about Lessons Learned from a Year of Building with LLMs with Eugene Yan from Amazon, Bryan Bischof from Hex, Charles Frye from Modal, Hamel Husain from Parlance Labs, and Shreya Shankar from UC Berkeley. These five guests, along with Jason Liu who couldn't join us, have spent the past year building real-world applications with Large Language Models (LLMs). They've distilled their experiences into a report of 42 lessons across operational, strategic, and tactical dimensions (https://applied-llms.org/), and they're here to share their insights. We’ve split this roundtable into 2 episodes and, in this second episode, we'll explore: An inside look at building end-to-end systems with LLMs; The experimentation mindset: Why it's the key to successful AI products; Building trust in AI: Strategies for getting stakeholders on board; The art of data examination: Why looking at your data is more crucial than ever; Evaluation strategies that separate the pros from the amateurs. Although we're focusing on LLMs, many of these insights apply broadly to data science, machine learning, and product development, more generally. LINKS The livestream on YouTube (https://www.youtube.com/live/c0gcsprsFig) The Report: What We’ve Learned From A Year of Building with LLMs (https://applied-llms.org/) About the Guests/Authors (https://applied-llms.org/about.html) <-- connect with them all on LinkedIn, follow them on Twitter, subscribe to their newsletters! (Seriously, though, the amount of collective wisdom here is 🤑 Your AI product needs evals by Hamel Husain (https://hamel.dev/blog/posts/evals/) Prompting Fundamentals and How to Apply them Effectively by Eugene Yan (https://eugeneyan.com/writing/prompting/) Fuck You, Show Me The Prompt by Hamel Husain (https://hamel.dev/blog/posts/prompt/) Vanishing Gradients on YouTube (https://www.youtube.com/channel/UC_NafIo-Ku2loOLrzm45ABA) Vanishing Gradients on Twitter (https://x.com/vanishingdata) Vanishing Gradients on Lu.ma (https://lu.ma/calendar/cal-8ImWFDQ3IEIxNWk)

Hugo speaks about Lessons Learned from a Year of Building with LLMs with Eugene Yan from Amazon, Bryan Bischof from Hex, Charles Frye from Modal, Hamel Husain from Parlance Labs, and Shreya Shankar from UC Berkeley. These five guests, along with Jason Liu who couldn't join us, have spent the past year building real-world applications with Large Language Models (LLMs). They've distilled their experiences into a report of 42 lessons across operational, strategic, and tactical dimensions (https://applied-llms.org/), and they're here to share their insights. We’ve split this roundtable into 2 episodes and, in this first episode, we'll explore: The critical role of evaluation and monitoring in LLM applications and why they're non-negotiable, including "evals" - short for evaluations, which are automated tests for assessing LLM performance and output quality; Why data literacy is your secret weapon in the AI landscape; The fine-tuning dilemma: when to do it and when to skip it; Real-world lessons from building LLM applications that textbooks won't teach you; The evolving role of data scientists and AI engineers in the age of AI. Although we're focusing on LLMs, many of these insights apply broadly to data science, machine learning, and product development, more generally. LINKS The livestream on YouTube (https://www.youtube.com/live/c0gcsprsFig) The Report: What We’ve Learned From A Year of Building with LLMs (https://applied-llms.org/) About the Guests/Authors (https://applied-llms.org/about.html) <-- connect with them all on LinkedIn, follow them on Twitter, subscribe to their newsletters! (Seriously, though, the amount of collective wisdom here is 🤑 Your AI product needs evals by Hamel Husain (https://hamel.dev/blog/posts/evals/) Prompting Fundamentals and How to Apply them Effectively by Eugene Yan (https://eugeneyan.com/writing/prompting/) Fuck You, Show Me The Prompt by Hamel Husain (https://hamel.dev/blog/posts/prompt/) Vanishing Gradients on YouTube (https://www.youtube.com/channel/UC_NafIo-Ku2loOLrzm45ABA) Vanishing Gradients on Twitter (https://x.com/vanishingdata) Vanishing Gradients on Lu.ma (https://lu.ma/calendar/cal-8ImWFDQ3IEIxNWk)

Hugo speaks with Alan Nichol, co-founder and CTO of Rasa, where they build software to enable developers to create enterprise-grade conversational AI and chatbot systems across industries like telcos, healthcare, fintech, and government. What's super cool is that Alan and the Rasa team have been doing this type of thing for over a decade, giving them a wealth of wisdom on how to effectively incorporate LLMs into chatbots - and how not to. For example, if you want a chatbot that takes specific and important actions like transferring money, do you want to fully entrust the conversation to one big LLM like ChatGPT, or secure what the LLMs can do inside key business logic? In this episode, they also dive into the history of conversational AI and explore how the advent of LLMs is reshaping the field. Alan shares his perspective on how supervised learning has failed us in some ways and discusses what he sees as the most overrated and underrated aspects of LLMs. Alan offers advice for those looking to work with LLMs and conversational AI, emphasizing the importance of not sleeping on proven techniques and looking beyond the latest hype. In a live demo, he showcases Rasa's Calm (Conversational AI with Language Models), which allows developers to define business logic declaratively and separate it from the LLM, enabling reliable execution of conversational flows. LINKS The livestream on YouTube (https://www.youtube.com/live/kMFBYC2pB30?si=yV5sGq1iuC47LBSi) Alan's Rasa CALM Demo: Building Conversational AI with LLMs (https://youtu.be/4UnxaJ-GcT0?si=6uLY3GD5DkOmWiBW) Alan on twitter.com (https://x.com/alanmnichol) Rasa (https://rasa.com/) CALM, an LLM-native approach to building reliable conversational AI (https://rasa.com/docs/rasa-pro/calm/) Task-Oriented Dialogue with In-Context Learning (https://arxiv.org/abs/2402.12234) 'We don’t know how to build conversational software yet' by Alan Nicol (https://medium.com/rasa-blog/we-don-t-know-how-to-build-conversational-software-yet-a18301db0e4b) Vanishing Gradients on Twitter (https://twitter.com/vanishingdata) Hugo on Twitter (https://twitter.com/hugobowne) Upcoming Livestreams Lessons from a Year of Building with LLMs (https://lu.ma/e8huz3s6?utm_source=vgan) VALIDATING THE VALIDATORS with Shreya Shanker (https://lu.ma/zz3qic45?utm_source=vgan)

Hugo speaks with Jason Liu, an independent consultant who uses his expertise in recommendation systems to help fast-growing startups build out their RAG applications. He was previously at Meta and Stitch Fix is also the creator of Instructor, Flight, and an ML and data science educator. They talk about how Jason approaches consulting companies across many industries, including construction and sales, in building production LLM apps, his playbook for getting ML and AI up and running to build and maintain such apps, and the future of tooling to do so. They take an inverted thinking approach, envisaging all the failure modes that would result in building terrible AI systems, and then figure out how to avoid such pitfalls. LINKS The livestream on YouTube (https://youtube.com/live/USTG6sQlB6s?feature=share) Jason's website (https://jxnl.co/) PyDdantic is all you need, Jason's Keynote at AI Engineer Summit, 2023 (https://youtu.be/yj-wSRJwrrc?si=JIGhN0mx0i50dUR9) How to build a terrible RAG system by Jason (https://jxnl.co/writing/2024/01/07/inverted-thinking-rag/) To express interest in Jason's Systematically improving RAG Applications course (https://q7gjsgfstrp.typeform.com/ragcourse?typeform-source=vg) Vanishing Gradients on Twitter (https://twitter.com/vanishingdata) Hugo on Twitter (https://twitter.com/hugobowne) Upcoming Livestreams Good Riddance to Supervised Learning with Alan Nichol (CTO and co-founder, Rasa) (https://lu.ma/gphzzyyn?utm_source=vgj) Lessons from a Year of Building with LLMs (https://lu.ma/e8huz3s6?utm_source=vgj)

Hugo speaks with Sebastian Raschka, a machine learning & AI researcher, programmer, and author. As Staff Research Engineer at Lightning AI, he focuses on the intersection of AI research, software development, and large language models (LLMs). How do you build LLMs? How can you use them, both in prototype and production settings? What are the building blocks you need to know about? ​In this episode, we’ll tell you everything you need to know about LLMs, but were too afraid to ask: from covering the entire LLM lifecycle, what type of skills you need to work with them, what type of resources and hardware, prompt engineering vs fine-tuning vs RAG, how to build an LLM from scratch, and much more. The idea here is not that you’ll need to use an LLM you’ve built from scratch, but that we’ll learn a lot about LLMs and how to use them in the process. Near the end we also did some live coding to fine-tune GPT-2 in order to create a spam classifier! LINKS The livestream on YouTube (https://youtube.com/live/qL4JY6Y5pmA) Sebastian's website (https://sebastianraschka.com/) Machine Learning Q and AI: 30 Essential Questions and Answers on Machine Learning and AI by Sebastian (https://nostarch.com/machine-learning-q-and-ai) Build a Large Language Model (From Scratch) by Sebastian (https://www.manning.com/books/build-a-large-language-model-from-scratch) PyTorch Lightning (https://lightning.ai/docs/pytorch/stable/) Lightning Fabric (https://lightning.ai/docs/fabric/stable/) LitGPT (https://github.com/Lightning-AI/litgpt) Sebastian's notebook for finetuning GPT-2 for spam classification! (https://github.com/rasbt/LLMs-from-scratch/blob/main/ch06/01_main-chapter-code/ch06.ipynb) The end of fine-tuning: Jeremy Howard on the Latent Space Podcast (https://www.latent.space/p/fastai) Our next livestream: How to Build Terrible AI Systems with Jason Liu (https://lu.ma/terrible-ai-systems?utm_source=vg) Vanishing Gradients on Twitter (https://twitter.com/vanishingdata) Hugo on Twitter (https://twitter.com/hugobowne)

Hugo speaks with Omoju Miller, a machine learning guru and founder and CEO of Fimio, where she is building 21st century dev tooling. In the past, she was Technical Advisor to the CEO at GitHub, spent time co-leading non-profit investment in Computer Science Education for Google, and served as a volunteer advisor to the Obama administration’s White House Presidential Innovation Fellows. We need open tools, open data, provenance, and the ability to build fully reproducible, transparent machine learning workflows. With the advent of closed-source, vendor-based APIs and compute becoming a form of gate-keeping, developer tools are at the risk of becoming commoditized and developers becoming consumers. We’ll talk about how ideas for escaping these burgeoning walled gardens. We’ll dive into What fully reproducible ML workflows would look like, including git for the workflow build process, The need for loosely coupled and composable tools that embrace a UNIX-like philosophy, What a much more scientific toolchain would look like, What a future open sources commons for Generative AI could look like, What an open compute ecosystem could look like, How to create LLMs and tooling so everyone can use them to build production-ready apps, And much more! LINKS The livestream on YouTube (https://www.youtube.com/live/n81PWNsHSMk?si=pgX2hH5xADATdJMu) Omoju on Twitter (https://twitter.com/omojumiller) Hugo on Twitter (https://twitter.com/hugobowne) Vanishing Gradients on Twitter (https://twitter.com/vanishingdata) Lu.ma Calendar that includes details of Hugo's European Tour for Outerbounds (https://lu.ma/Outerbounds) Blog post that includes details of Hugo's European Tour for Outerbounds (https://outerbounds.com/blog/ob-on-the-road-2024-h1/)

Hugo speaks with Johno Whitaker, a Data Scientist/AI Researcher doing R&D with answer.ai. His current focus is on generative AI, flitting between different modalities. He also likes teaching and making courses, having worked with both Hugging Face and fast.ai in these capacities. Johno recently reminded Hugo how hard everything was 10 years ago: “Want to install TensorFlow? Good luck. Need data? Perhaps try ImageNet. But now you can use big models from Hugging Face with hi-res satellite data and do all of this in a Colab notebook. Or think ecology and vision models… or medicine and multimodal models!” We talk about where we’ve come from regarding tooling and accessibility for foundation models, ML, and AI, where we are, and where we’re going. We’ll delve into What the Generative AI mindset is, in terms of using atomic building blocks, and how it evolved from both the data science and ML mindsets; How fast.ai democratized access to deep learning, what successes they had, and what was learned; The moving parts now required to make GenAI and ML as accessible as possible; The importance of focusing on UX and the application in the world of generative AI and foundation models; The skillset and toolkit needed to be an LLM and AI guru; What they’re up to at answer.ai to democratize LLMs and foundation models. LINKS The livestream on YouTube (https://youtube.com/live/hxZX6fBi-W8?feature=share) Zindi, the largest professional network for data scientists in Africa (https://zindi.africa/) A new old kind of R&D lab: Announcing Answer.AI (http://www.answer.ai/posts/2023-12-12-launch.html) Why and how I’m shifting focus to LLMs by Johno Whitaker (https://johnowhitaker.dev/dsc/2023-07-01-why-and-how-im-shifting-focus-to-llms.html) Applying AI to Immune Cell Networks by Rachel Thomas (https://www.fast.ai/posts/2024-01-23-cytokines/) Replicate -- a cool place to explore GenAI models, among other things (https://replicate.com/explore) Hands-On Generative AI with Transformers and Diffusion Models (https://www.oreilly.com/library/view/hands-on-generative-ai/9781098149239/) Johno on Twitter (https://twitter.com/johnowhitaker) Hugo on Twitter (https://twitter.com/hugobowne) Vanishing Gradients on Twitter (https://twitter.com/vanishingdata) SciPy 2024 CFP (https://www.scipy2024.scipy.org/#CFP) Escaping Generative AI Walled Gardens with Omoju Miller, a Vanishing Gradients Livestream (https://lu.ma/xonnjqe4)

Hugo speaks with Allen Downey, a curriculum designer at Brilliant, Professor Emeritus at Olin College, and the author of Think Python, Think Bayes, Think Stats, and other computer science and data science books. In 2019-20 he was a Visiting Professor at Harvard University. He previously taught at Wellesley College and Colby College and was a Visiting Scientist at Google. He is also the author of the upcoming book Probably Overthinking It! They discuss Allen's new book and the key statistical and data skills we all need to navigate an increasingly data-driven and algorithmic world. The goal was to dive deep into the statistical paradoxes and fallacies that get in the way of using data to make informed decisions. For example, when it was reported in 2021 that “in the United Kingdom, 70-plus percent of the people who die now from COVID are fully vaccinated,” this was correct but the implication was entirely wrong. Their conversation jumps into many such concrete examples to get to the bottom of using data for more than “lies, damned lies, and statistics.” They cover Information and misinformation around pandemics and the base rate fallacy; The tools we need to comprehend the small probabilities of high-risk events such as stock market crashes, earthquakes, and more; The many definitions of algorithmic fairness, why they can't all be met at once, and what we can do about it; Public health, the need for robust causal inference, and variations on Berkson’s paradox, such as the low-birthweight paradox: an influential paper found that that the mortality rate for children of smokers is lower for low-birthweight babies; Why none of us are normal in any sense of the word, both in physical and psychological measurements; The Inspection paradox, which shows up in the criminal justice system and distorts our perception of prison sentences and the risk of repeat offenders. LINKS The livestream on YouTube (https://youtube.com/live/G8LulD72kzs?feature=share) Allen Downey on Github (https://github.com/AllenDowney) Allen's new book Probably Overthinking It! (https://greenteapress.com/wp/probably-overthinking-it/) Allen on Twitter (https://twitter.com/AllenDowney) Prediction-Based Decisions and Fairness: A Catalogue of Choices, Assumptions, and Definitions by Mitchell et al. (https://arxiv.org/abs/1811.07867)