The Multipolarity Report

The Transformer architecture has changed surprisingly little from the original paper in 2017 (over 7 years ago!). The diff: - The nonlinearity in the MLP has undergone some refinement. Almost every model uses some form of gated nonlinearity. A silu or gelu nonlinearity is common. - The placement of normalization layers. This tends to vary a little from architecture to architecture. Sometimes more normalization layers per Transformer block (e.g.Gemma 2). Sometimes keys and queries are normalized (e.g. Command+R). - The type of normalization layer. RMS norm is commonly used instead of Layer Norm. All of Llama 3, Phi 3 and Gemma 2 use RMS norm now. Seems like vanilla Layer Norm is becoming a little less common. - Group-query attention is now a staple as it really speeds up inference for larger KV cache's (e.g. longer prompts / generations). - And of course the positional encodings have changed from sinusoidal to rotary (aka RoPE). Not too much variation otherwise.