PyTorch Developer Podcast

Podcast details

Technical and general information from the podcast's RSS feed.

Site

RSS

Apple

Recent rankings

Latest chart positions across Apple Podcasts and Spotify rankings.

Shared links between episodes and podcasts

Links found in episode descriptions and other podcasts that share them.

See all

https://youtu.be/_qB2Ho1O3u4
2 shares
https://www.youtube.com/watch?v=-Z-17h3jG0A
2 shares
https://youtu.be/vbtGZL7IrAw?t=1037
1 share

RSS feed quality and score

Technical evaluation of the podcast's RSS feed quality and structure.

See all

Publication history

Monthly episode publishing history over the past years.

Latest published episodes

Recent episodes with titles, durations, and descriptions.

See all

Compiler collectives

Episode 83

Sunday, August 4, 2024 • Duration 16:33

Compiler collectives are a PT2 feature where by compiler instances across multiple ranks use NCCL collectives to communicate information to other instances. This is used to ensure we consistently decide if inputs or static or dynamic across all ranks. See also PR at https://github.com/pytorch/pytorch/pull/130935

Read moreCollapse

TORCH_TRACE and tlparse

Episode 82

Monday, April 29, 2024 • Duration 15:28

TORCH_TRACE and tlparse are a structured log and log parser for PyTorch 2. It gives useful information about what code was compiled and what the intermediate build products look like.

Inductor - Define-by-run IR

Episode 73

Wednesday, January 24, 2024 • Duration 12:06

Define-by-run IR is how Inductor defines the internal compute of a pointwise/reduction operation. It is characterized by a function that calls a number of functions in the 'ops' namespace, where these ops can be overridden by different handlers depending on what kind of semantic analysis you need to do. The ops Inductor supports include regular arithmetic operators, but also memory load/store, indirect indexing, masking and collective operations like reductions.

Read moreCollapse

Unsigned integers

Episode 72

Wednesday, January 17, 2024 • Duration 13:07

Traditionally, unsigned integer support in PyTorch was not great; we only support uint8. Recently, we added support for uint16, uint32 and uint64. Bare bones functionality works, but I'm entreating the community to help us build out the rest. In particular, for most operations, we plan to use PT2 to build anything else. But if you have an eager kernel you really need, send us a PR and we'll put it in. While most of the implementation was straightforward, there are some weirdnesses related to type promotion inconsistencies with numpy and dealing with the upper range of uint64. There is also upcoming support for sub-byte dtypes uint1-7, and these will exclusively be implemented via PT2.

Read moreCollapse

Inductor - IR

Episode 71

Tuesday, January 16, 2024 • Duration 18:00

Inductor IR is an intermediate representation that lives between ATen FX graphs and the final Triton code generated by Inductor. It was designed to faithfully represent PyTorch semantics and accordingly models views, mutation and striding. When you write a lowering from ATen operators to Inductor IR, you get a TensorBox for each Tensor argument which contains a reference to the underlying IR (via StorageBox, and then a Buffer/ComputedBuffer) that says how the Tensor was computed. The inner computation is represented via define-by-run, which allows for compact definition of IR representation, while still allowing you to extract an FX graph out if you desire. Scheduling then takes buffers of inductor IR and decides what can be fused. Inductor IR may have too many nodes, this would be a good thing to refactor in the future.

Read moreCollapse

Dynamo - VariableTracker

Episode 70

Friday, January 12, 2024 • Duration 15:55

I talk about VariableTracker in Dynamo. VariableTracker is Dynamo's representation of the Python. I talk about some recent changes, namely eager guards and mutable VT. I also tell you how to find the functionality you care about in VariableTracker (https://docs.google.com/document/d/1XDPNK3iNNShg07jRXDOrMk2V_i66u1hEbPltcsxE-3E/edit#heading=h.i6v7gqw5byv6).

Read moreCollapse

Unbacked SymInts

Episode 69

Tuesday, February 21, 2023 • Duration 21:31

This podcast goes over the basics of unbacked SymInts. You might want to listen to this one before listening to https://pytorch-dev-podcast.simplecast.com/episodes/zero-one-specialization Some questions we answer (h/t from Gregory Chanan):

- Are unbacked symints only for export? Because otherwise I could just break / wait for the actual size. But maybe I can save some retracing / graph breaks perf if I have them too? So the correct statement is "primarily" for export?

- Why am I looking into the broadcasting code at all? Naively, I would expect the export graph to be just a list of ATen ops strung together. Why do I recurse that far down? Why can't I annotate DONT_TRACE_ME_BRO?

- How does 0/1 specialization fit into this? I understand we may want to 0/1 specialize in a dynamic shape regime in "eager" mode (is there a better term?), but that doesn't seem to matter for export?

- So far we've mainly been talking about how to handle our own library code. There is a worry about pushing complicated constraints downstream, similar to torchscript. What constraints does this actually push?

Read moreCollapse

Zero-one specialization

Episode 68

Monday, February 20, 2023 • Duration 21:07

Mikey Dagistes joins me to ask some questions about the recent recent composability sync https://www.youtube.com/watch?v=NJV7YFbtoR4 where we discussed 0/1 specialization and its implications on export in PT2. What's the fuss all about? What do I need to understand about PT2 to understand why 0/1 specialization is a thing?

Read moreCollapse

torchdynamo

Episode 67

Tuesday, December 6, 2022 • Duration 25:35

What is torchdynamo? From a bird's eye view, what exactly does it do? What are some important things to know about it? How does it differ from other graph capture mechanisms?

For more reading, check out https://docs.google.com/document/d/13K03JN4gkbr40UMiW4nbZYtsw8NngQwrTRnL3knetGM/edit#

Read moreCollapse

PyTorch 2.0

Episode 66

Sunday, December 4, 2022 • Duration 17:51

Soumith's keynote on PT2.0: https://youtu.be/vbtGZL7IrAw?t=1037
PT2 Manifesto: https://docs.google.com/document/d/1tlgPcR2YmC3PcQuYDPUORFmEaBPQEmo8dsh4eUjnlyI/edit#
PT2 Architecture: https://docs.google.com/document/d/1wpv8D2iwGkKjWyKof9gFdTf8ISszKbq1tsMVm-3hSuU/edit#

Read moreCollapse

PyTorch Developer Podcast – Details, episodes & analysis

Podcast details

Recent rankings

Apple Podcasts

Spotify

Shared links between episodes and podcasts

RSS feed quality and score

Publication history

Latest published episodes

Related Shows Based on Content Similarities

Apple Podcasts

Spotify

Related Shows Based on Content Similarities

PyTorch Developer Podcast – Details, episodes & analysis

Podcast details

Recent rankings

Apple Podcasts

Spotify

Shared links between episodes and podcasts

Github99+

Other53

Youtube4

RSS feed quality and score

Publication history

Latest published episodes

Similar podcasts and content

Related Shows Based on Content Similarities

Apple Podcasts

Spotify

Related Shows Based on Content Similarities

Github99+

Other53

Youtube4