← DemosIn your browser

Inside attention

Type a sentence and see the causal attention weights, computed in your browser.

ml-from-scratch

Each row shows where a word "looks" when building its own representation: softmax(QKᵀ/√d + M). With the causal mask, each word only sees the ones before it, like a model generating text. The weights here are random (not a trained model); the point is the mechanism.