The Backend Engineering Show with Hussein Nasser – Details, episodes & analysis

Cloudflare built a global cache purge system that runs under 150 ms.

This is how they did it.

Using RockDB to maintain local CDN cache, and a peer-to-peer data center distributed system and clever engineering, they went from 1.5 second purge, down to 150 ms.

However, this isn’t full picture, because that 150 ms is just actually the P50. In this video I explore Clouldflare CDN work, how the old core-based centralized quicksilver, lazy purge work compared to the new coreless, decentralized active purge. In it I explore the pros and cons of both systems and give you my thoughts of this system.

0:00 Intro

4:25　From Core Base Lazy Purge to Coreless Active

12:50 CDN Basics

16:00 TTL Freshness

17:50 Purge

20:00 Core-Based Purge

24:00 Flexible Purges

26:36 Lazy Purge

30:00 Old Purge System Limitations

36:00 Coreless / Active Purge

39:00 LSM vs BTree

45:30 LSM Performance issues

48:00 How Active Purge Works

50:30 My thoughts about the new system

58:30 Summary

Cloudflare blog

https://blog.cloudflare.com/instant-purge/

Mentioned Videos

Cloudflare blog

https://blog.cloudflare.com/instant-purge/

Percentile Tail Latency Explained (95%, 99%) Monitor Backend performance with this metric

https://www.youtube.com/watch?v=3JdQOExKtUY

How Discord Stores Trillions of Messages | Deep Dive

https://www.youtube.com/watch?v=xynXjChKkJc

Fundamentals of Operating Systems Course

https://os.husseinnasser.com

Backend Troubleshooting Course

https://performance.husseinnasser.com

Fundamentals of Database Engineering udemy course https://databases.win MySQL has been having bumpy journey since 2018 with the release of the version 8.0. Critical crashes that made to the final product, significant performance regressions, and tons of stability and bugs issues. In this video I explore what happened to MySql, are these issues getting fixed? And what is the current state of MySQL at the end of 2024. 0:00 Intro 2:00 MySQL 8.0 vs 5.7 Performance 11:00 Critical Crash in 8.0.38, 8.4.1 and 9.0.0 15:40 Is 8.4 better than 8.0.36? 16:30 More Features = More Bugs 22:30 Summary and my thoughts resources https://x.com/MarkCallaghanDB/status/1786428909376164263 https://www.percona.com/blog/do-not-upgrade-to-any-version-of-mysql-after-8-0-37/ http://smalldatum.blogspot.com/2024/09/mysql-innodb-vs-sysbench-on-large-server.html https://www.percona.com/blog/mysql-8-0-vs-5-7-are-the-newer-versions-more-problematic/

https://oscourse.win

Allegro improved their Kafka produce tail latency by over 80% when they switched from ext4 to xfs. What I enjoyed most about this article is the detailed analysis and tweaking the team made to ext4 before considering switching to xfs. This is a classic case of how a good tech blog looks like in my opinion. 

0:00 Intro

0:30 Summary

2:35 How Kafka Works?

5:00 Producers Writes are Slow

7:10 Tracing Kafka Protocol 

12:00 Tracing Kernel System Calls

16:00 Journaled File Systems

21:00 Improving ext4

26:00 Switching to XFS

Blog

https://blog.allegro.tech/2024/03/kafka-performance-analysis.html

I asked a question on my community post   What port does your computer connect to when you visit google.com on your browser for the very first time?  The answer really depends on the network configuration and that's what I discuss in this video. The answer could be HTTP port 80, HTTPS port 443, DNS port 53 or other answers that aren't even listed here including DNS over HTTPS and DNS over TLS. Lets discuss

In this episode of the backend engineering show I discuss the ramification of index page splits which results in fragmented index yielding slow query performance when using indexes. I go through what a page is, how a page is read and written in the database and finally discuss the current solutions to address this problem such as reindexing, fill factor and choosing a good index key type. Get my Fundamentals of Database Engineering Udemy course https://database.husseinnasser.com 0:00 Intro 1:50 What is a Page? 5:00 How to Read and Write a page 8:45 Batching Writes and WAL 11:45 The Order of Content in the Page 16:00 Page Splits 21:30 Solving Index Fragmentation 28:30 Summary

Multitenancy refers to placing two or more clients, customers or tenants so they share a single resource. The idea behind Multitenancy is to save on resources instead of giving each client its own resource (storage, app, memory, network etc..). You slice each part and give it to dedicated client. The number one rule is those tenants shouldn’t be aware of the existence of other tenants, they need to be completed isolated, no one tenant can read or influence the other’s tenant share. If this is confusing don’t worry I’ll go through examples in this episode of the backend engineering show.

0:00 Intro

2:50 Ssd Multitenancy

6:45 Database Multitenancy

10:14 Networking Multitenancy (SDN)

16:55 VMs and Containers Multitenancy

19:00 Application Multitenancy

Scientists at the Physics and Engineering Department of the UK’s Lancaster University published a new paper describing ULTRARAM. A Low-Energy, High-Endurance, Compound-Semiconductor Memory on Silicon. I think this tech is a game changer but I still have my doubts. Let us discuss 0:00 Intro 0:30 RAM 4:00 SSD 11:00 UltraRAM 17:30 Doubts * RAM (Dynamic RAM) * Fast access with capacitors * random access * requires power to be refreshed * Flash NAND (SSD) * Pros fast random access (FTL) * Block storage * Requires high voltage (20V to erase/ 7.5 to program) * Low durability endurance which then (write amplification + garbage collection ) * UltraRAM uses a new novel approach to the memory cell design that only need 2.5 V to program/erase, low power which saves the memory cell lifetime https://www.sciencedirect.com/topics/engineering/fowler-nordheim-tunnelling * High endurance program/erase last longer * High retention * GC/WA not required * Limitations: still we know very little https://onlinelibrary.wiley.com/doi/10.1002/aelm.202101103  

enom my DNS registrar had an outage which took down my site husseinnasser.com lets discuss   0:00 Intro  5:00 How DNS Works 12:00 Outage report 16:00 Will I migrate my DNS? 21:40 Does decentralize mean anything anymore?

In this episode of the backend engineering show, I’ll discuss the low code movements and their pros and cons.

0:00 Intro

2:00 Evolution of Languages

8:40 Low Code Movement

10:00 Side Effect of Low Code

13:00 Leaky Abstractions

Fundamentals of Database Engineering udemy course (link redirects to udemy with coupon)

https://database.husseinnasser.com

Introduction to NGINX (link redirects to udemy with coupon)

https://nginx.husseinnasser.com

Python on the Backend (link redirects to udemy with coupon)

https://nginx.husseinnasser.com

These are some fundamental technologies that I want to explore in 2022.   0:00 The Way I learn 4:20 Database Engineering 7:00 Network Engineering  9:30 Hardware and Operating Systems 15:00 The problem with software engineering   Collateral Knowledge Video https://www.youtube.com/watch?v=6YKbVpWmeLM&t=3s  Become a Member on YouTube https://www.youtube.com/channel/UC_ML5xP23TOWKUcc-oAE_Eg/join 🔥 Members Only Content https://www.youtube.com/playlist?list=UUMO_ML5xP23TOWKUcc-oAE_Eg  Fundamentals of Database Engineering udemy course (link redirects to udemy with coupon) https://database.husseinnasser.com Introduction to NGINX  (link redirects to udemy with coupon) https://nginx.husseinnasser.com Python on the Backend (link redirects to udemy with coupon) https://nginx.husseinnasser.com