Heavy tails in machine learning
TML
2 April 2024 to 26 April 2024
Programme theme
HEAVY TAILS IN MACHINE LEARNING
The Stochastic Gradient Descent (SGD) algorithm is both fundamental and ubiquitous in Machine Learning applications. In recent years, heavy-tailed distributions have been observed in practice implementations of the SGD algorithm. Importantly, the heavy-tailed behaviour is generally not a consequence of the presence of a heavy-tailed distribution in the description of the model. It is not understood under what circumstances heavy tails arise in SGD and, when they do, what their effects on the performance of the SGD algorithm are.
The INI satellite programme at the Alan Turing Institute will initiate a research programme centered around the following questions:
-
When and how do heavy-tailed phenomena arise in general SGD algorithms?
-
How should the SGD algorithm be modified to make it efficient in the presence of heavy tails?
-
Can convergence properties of (possibly modified) SGD algorithms be analysed?
In the presence of heavy tails, can convergence guarantees for SGD methodologies be established rigorously and what are the implications for the algorithm’s design?
The programme will gather the Machine Learning and Applied Probability communities with the aim of tackling these questions using probabilistic methods. Three research-level lecture courses on SGD, stochastic stability and simulation of heavy tailed dynamics will run in the first week of the programme, followed by two weeks of deep-dive research into the questions above. The programme will conclude with a week-long workshop on heavy tails in Machine Learning.
This satellite programme will take place at The Alan Turing Institute in London.
Organisers
- Deniz Akyildiz Imperial College London
- Anita Behme Technische Universität Dresden
- Emilie Chouzenoux Université Paris Saclay
- Jorge Gonzalez-Cazares University of Warwick; The Alan Turing Institute
- Aleksandar Mijatovic University of Warwick; The Alan Turing Institute
Participants
- Deniz Akyildiz Imperial College London
- Jorge Gonzalez-Cazares University of Warwick; The Alan Turing Institute
- Aleksandar Mijatovic University of Warwick; The Alan Turing Institute
Supported By
