Category: AI – This Quote
Catherine Olsson and colleagues, 2022
We present six complementary lines of evidence, arguing that induction heads may be the mechanistic source of general in-context learning.
Nelson Elhage and colleagues, 2022
This paper provides a toy model where polysemanticity can be fully understood, arising as a result of models storing additional sparse features in superposition.
Neel Nanda and colleagues, 2023
We argue that progress measures can be found via mechanistic interpretability: reverse-engineering learned behaviors into their individual components.
Collin Burns and colleagues, 2022
We propose directly finding latent knowledge inside the internal activations of a language model in a purely unsupervised way.
Chuan Guo and colleagues, 2017
We discover that modern neural networks, unlike those from a decade ago, are poorly calibrated.
Ian Osband and colleagues, 2016
We propose bootstrapped DQN, a simple algorithm that explores in a computationally and statistically efficient manner through use of randomized value functions.
Ilya Loshchilov and Frank Hutter, 2017
We propose a simple modification to recover the original formulation of weight decay regularization.
Balaji Lakshminarayanan and colleagues, 2016
We propose an alternative to Bayesian neural networks that is simple to implement and readily parallelizable.
Ari Holtzman and colleagues, 2019
Our findings motivate Nucleus Sampling, a simple but effective method to draw the best out of neural generation.
Hongyu Wang and colleagues, 2023
We introduce BitNet, a scalable and stable 1-bit Transformer architecture designed for large language models.
Yutao Sun and colleagues, 2023
We propose Retentive Network as a foundation architecture for large language models.
