Günter Klambauer and colleagues, 2017
We introduce self-normalizing neural networks to enable high-level abstract representations.
Don't Just Read it – Explore It
We introduce self-normalizing neural networks to enable high-level abstract representations.
We demonstrate the power of a simple combination of two common SSL methods: consistency regularization and pseudo-labeling.
We propose a novel simplified gating mechanism and investigate the impact of key architectural decisions.
Using a combination of exhaustive and reinforcement learning-based search, we discover multiple novel activation functions.
RMSNorm is computationally simpler and thus more efficient than LayerNorm.
We introduce grouped-query attention, a generalization of multi-query attention.
We therefore introduce a simpler and more efficient position method, Attention with Linear Biases.
We propose a variant called multi-query attention, where the keys and values are shared across all of the different attention heads.
We develop a novel solution, Zero Redundancy Optimizer, to optimize memory, vastly improving training speed while increasing the model size.
We implement a simple, efficient intra-layer model parallel approach that enables training transformer models with billions of parameters.
We propose GPTQ, a new one-shot weight quantization method based on approximate second-order information.
We propose SmoothQuant, a training-free, accuracy-preserving, and general-purpose post-training quantization solution.