João Sacramento

Title: Gradient-based optimization emerges in trained neural networks

Abstract: The algorithms implemented by task-optimized neural networks are usually unknown to their designers. In my talk I will attempt to reverse engineer transformers trained to solve small-scale few-shot learning and sequential prediction tasks. It turns out that these trained neural networks often approach their tasks by constructing appropriate objective functions and then optimizing them using gradient-based methods within their forward dynamics, not unlike what a human machine learning practitioner would do. I will then extend these analyses to recurrent neural networks by establishing an equivalence result between the two model classes. I will discuss how our findings might help understand in-context learning in language models and finish with some speculations about learning in the cortex.

National TV: I spoke about links between neuroscience and artificial intelligence research at the 72nd episode of Sociedade Civil, a portuguese television show.

BrainGain 2023: I gave a general-audience talk explaining our research to Portuguese students interested in neuroscience and AI.

COSYNE 2023: We presented two posters at the main meeting and gave two talks at the Top-down interactions in the neocortex: Structure, function, plasticity and models workshop.

Swiss Computational Neuroscience Retreat 2023: Nicolas presented our least-control principle for learning at the 2nd edition of the Swiss Computational Neuroscience Retreat in Crans Montana.

Nicolas's talk for the Brain & AI group at Meta AI: Nicolas gave a talk presenting our least-control principle for learning to Jean-Rémi King's Brain & AI group at Meta AI on November 16.

Mathematics, Physics & Machine Learning seminar talk: I gave an IST Mathematics, Physics & Machine Learning seminar talk on November 10.

Panel discussion on lifelong learning: I participated on a panel discussion on Lifelong Learning Machines at NeurIPS 2022.

Visit to Mila: Simon, Alexander, Nicolas and I visited Blake Richards's lab at Mila.

Doctoral symposium at EPIA: Together with Fernando P. Santos and Henrique Lopes Cardoso I organized a doctoral symposium at EPIA, the Portuguese conference on artificial intelligence which will be held in Lisbon.

DeepMind talk by Johannes: Johannes gave a talk at DeepMind, London presenting our models and algorithms for continual learning and meta-learning.

MLSS^N 2022 lecture (video on YouTube): I gave a lecture with Alexander, Simon and Nicolas for the MLSS^N 2022 summer school in Krakow, Poland where we discussed bilevel optimization problems involving neural networks. We covered how to solve them with recurrent backpropagation, equilibrium propagation, and some of our own work on learning and meta-learning without error backpropagation. The lecture is now on YouTube.

Oxford seminar talk: I gave an Oxford NeuroTheory Forum seminar presenting our work on biologically-plausible meta-learning.

NAISys 2022 poster: Alexander presented ongoing work on our new principle for learning at the NAISys 2022 conference in Cold Spring Harbor, NY.

Swiss Computational Neuroscience Retreat 2022: Nicolas presented our work on biologically-plausible meta-learning at the Swiss Computational Neuroscience Retreat in Crans Montana.