Attention demo

Attention forms scores S = QKᵀ/√dₖ, applies softmax across each query row, then computes O = AV. The selected query table shows the score, weight, value and weighted contribution. Values change the output, not the weights. Each allowed row sums to 1. A causal mask allows only keys up to and including the query; crossed cells have exactly zero weight. Their displayed scores are the original unmasked dot products.

This is an untrained toy. Five neutral token labels receive reproducible random vectors from the seed; Q = K = V = E. D features are split equally between H heads, so D must be divisible by H. There are no learned projections or position embeddings. Select any head and query without recalculating. Outputs from all heads are concatenated with Wᴼ = I; the details show the actual N × D matrix. The labels do not provide language understanding.

Patch mode uses one head and no causal mask. An image is stretched to 64 × 64 pixels over white, then split into sixteen 16 × 16 patches. Each patch averages all RGB channel values to v in [0,1], giving E = [v, 1−v, v/2, 1/4]. This channel mean is not photometric luminance. The grid uses row-major indices 4y + x. This illustrates patch attention, not a trained ViT or image recognition. Upload or Demo prepares the image; Calculate computes attention. Files are limited to 10 MiB, decoded images to 16 million pixels.

Model edits clear their results; head and query choices only change the view. Heatmap opacity uses the same square-root scale from 0 to 1 throughout (weight 0.25 is the midpoint); the table provides numeric values when colors are faint.

Attention Is All You Need, §3.2 · Vision Transformer, §3.1

Only input values are remembered in this browser. Calculated results and experiment histories are not saved.