Alexandre Day / Projects / Publications
Physics Reports 810, 1–124 (2019)

A high-bias, low-variance introduction to Machine Learning for physicists

Pankaj Mehta, Marin Bukov, Ching-Hao Wang, Alexandre G.R. Day, Clint Richardson, Charles K. Fisher, David J. Schwab

122 pages · 78 figures · 20 Python notebooks

TL;DR

An introduction to the core ideas of machine learning written in the language of physicists, from the bias–variance tradeoff to deep learning and energy-based models, with a strong emphasis on the connections to statistical physics. Every chapter comes with a Jupyter notebook on a physics dataset, so the ideas can be run, not just read.

What it covers

The review starts with the fundamentals that every other topic rests on: the bias–variance tradeoff, overfitting, regularization, generalization and gradient descent. It then works through supervised methods (linear and logistic regression, ensemble models, deep neural networks and convolutional networks) and unsupervised ones (clustering, data visualization, energy-based models such as MaxEnt and Restricted Boltzmann Machines, and variational methods including variational autoencoders).

Throughout, the emphasis is on the connections to statistical physics: free energies, phase transitions, mean-field theory and Monte Carlo sampling appear as natural tools for understanding why learning algorithms behave as they do. The review closes with an outlook on applying machine learning to physics and on open problems where physicists are well placed to contribute.

A short-tailed distribution p and a long-tailed distribution q, with points of equal probability density matched between them.
Figure 1: One of the review's illustrations: a short-tailed distribution p and a long-tailed distribution q. Matching points of equal density between the two shows how a heavy tail moves probability mass far from the centre, an effect that shows up repeatedly when reasoning about overfitting and generalization.

The notebooks

Twenty Python notebooks accompany the text and use physics-inspired datasets, including the Ising model and simulated proton–proton collision data, to implement the methods from scratch or with standard libraries. They are the fastest way to get a feel for the material and are maintained on GitHub, with an index on Pankaj Mehta's page.

Why it matters

Physicists arrive at machine learning with a lot of relevant intuition and an unfamiliar vocabulary. The review is meant as a self-contained on-ramp that uses the first to build the second, and it remains the way I would recommend a physicist start.