Keynote: Automatic Differentiation for Dummies

Automatic Differentiation for Dummies by Simon Peyton Jones

Automatic differentiation (AD) is clearly cool. And it has become useful (as well as cool) recently, because it essentially implements the back-propagation step that is key to learning in neural networks; and it is used extensively in many other machine learning contexts. But as well as being cool and useful, AD is also mysterious. There are forward and backward variants (the latter being mind-bending); there are tapes and mutation, delimited continuations, and more. It all makes me feel stupid.
In this talk I’ll explain a simple and elegant approach to AD from first principles, based on work by Conal Elliott. We’ll end up with an AD algorithm that takes 10 lines of code; that works for forward and backward AD; that works for matrix computations; and that is a source-to-source transformation whose output can be efficiently compiled without needing a runtime trace.

Part of ECOOP 2019 2019.ecoop.org/
Video by PL Presents, follow us at twitter.com/plpresents