{
  "paper": {
    "title": "Attention Is All You Need",
    "authors": [
      "Ashish Vaswani",
      "Noam Shazeer",
      "Niki Parmar",
      "Jakob Uszkoreit",
      "Llion Jones",
      "Aidan N. Gomez",
      "Łukasz Kaiser",
      "Illia Polosukhin"
    ],
    "equal_contribution": true,
    "year": 2017,
    "venue": "31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA",
    "arxiv": "1706.03762",
    "arxiv_version_in_source": "v7 [cs.CL] 2 Aug 2023",
    "pdf": "https://arxiv.org/pdf/1706.03762",
    "html": "https://ar5iv.labs.arxiv.org/html/1706.03762"
  },
  "claim": {
    "one_sentence": "We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.",
    "source": "Abstract"
  },
  "problem_and_hypothesis": {
    "problem": "Recurrent encoder–decoder models factor computation along symbol positions, so hidden state h_t depends on h_{t-1}. This sequential dependence precludes parallelization within training examples and becomes critical at longer sequence lengths (Section 1).",
    "prior_attention": "Attention already lets models capture dependencies regardless of distance, but in all but a few cases is used together with a recurrent network (Section 1, citing [2, 19, 27]).",
    "hypothesis": "A transduction model that relies entirely on attention (no sequence-aligned RNNs or convolution) can draw global dependencies, allow more parallelization, and still reach state-of-the-art translation quality (Sections 1–2).",
    "complexity_motivation": "In ConvS2S the operations relating two positions grow linearly with distance; in ByteNet they grow logarithmically. In the Transformer this is reduced to a constant number of operations, at the cost of averaging, which Multi-Head Attention is meant to counteract (Section 2, Section 3.2)."
  },
  "method_steps": [
    {
      "id": "encoder_decoder",
      "title": "Encoder–decoder stacks",
      "text": "Encoder maps (x_1..x_n) to z=(z_1..z_n); decoder auto-regressively emits (y_1..y_m). Both stacks use N=6 identical layers (Section 3, 3.1).",
      "refs": ["Section 3", "Section 3.1"]
    },
    {
      "id": "residual_norm",
      "title": "Residual + LayerNorm around each sub-layer",
      "text": "Output of each sub-layer is LayerNorm(x + Sublayer(x)). All sub-layers and embeddings produce d_model=512 (base) to allow residuals (Section 3.1).",
      "refs": ["Section 3.1"]
    },
    {
      "id": "scaled_dot_product",
      "title": "Scaled Dot-Product Attention",
      "text": "Attention(Q,K,V)=softmax(QK^T / sqrt(d_k)) V. Scaling by 1/sqrt(d_k) is to keep dot products from growing with d_k and saturating softmax (Section 3.2.1, eq. 1, footnote 4).",
      "refs": ["Section 3.2.1"]
    },
    {
      "id": "multi_head",
      "title": "Multi-Head Attention",
      "text": "h=8 parallel heads; per head d_k=d_v=d_model/h=64. Concatenate heads and project with W^O. Single-head averaging is said to inhibit jointly attending different subspaces (Section 3.2.2).",
      "refs": ["Section 3.2.2"]
    },
    {
      "id": "three_attentions",
      "title": "Three uses of attention",
      "text": "(1) Encoder self-attention: Q,K,V from previous encoder layer. (2) Masked decoder self-attention: cannot attend to subsequent positions (mask set to -∞ before softmax). (3) Encoder–decoder attention: queries from decoder, K,V from encoder output (Section 3.2.3).",
      "refs": ["Section 3.2.3"]
    },
    {
      "id": "ffn",
      "title": "Position-wise FFN",
      "text": "FFN(x)=max(0, xW_1+b_1)W_2+b_2; d_model=512, d_ff=2048 (base). Same transform at every position, different parameters per layer (Section 3.3, eq. 2).",
      "refs": ["Section 3.3"]
    },
    {
      "id": "embeddings",
      "title": "Shared embeddings and softmax",
      "text": "Learned embeddings of size d_model; embedding matrix shared with the pre-softmax linear map, similar to Press & Wolf [30]. Embedding weights multiplied by sqrt(d_model) (Section 3.4).",
      "refs": ["Section 3.4"]
    },
    {
      "id": "positional_encoding",
      "title": "Sinusoidal positional encoding",
      "text": "PE(pos,2i)=sin(pos/10000^{2i/d_model}), PE(pos,2i+1)=cos(pos/10000^{2i/d_model}). Wavelengths from 2π to 10000·2π. Chosen so PE_{pos+k} can be a linear function of PE_pos. Learned PE gave nearly identical Table 3 row (E) results; sinusoids kept for possible length extrapolation (Section 3.5).",
      "refs": ["Section 3.5", "Table 3 row (E)"]
    },
    {
      "id": "optimizer",
      "title": "Adam + warmup schedule",
      "text": "Adam β1=0.9, β2=0.98, ε=10^{-9}. lrate = d_model^{-0.5} · min(step^{-0.5}, step · warmup^{-1.5}), warmup_steps=4000 (Section 5.3, eq. 3).",
      "refs": ["Section 5.3"]
    },
    {
      "id": "regularization",
      "title": "Residual dropout and label smoothing",
      "text": "Dropout on each sub-layer output before residual add, and on embedding+PE sums. Base P_drop=0.1. Label smoothing ε_ls=0.1 (hurts perplexity, improves BLEU) (Section 5.4).",
      "refs": ["Section 5.4"]
    }
  ],
  "figures": [
    {
      "id": "fig1",
      "label": "Figure 1",
      "location": "page 3 (extract page marker after Section 3 intro)",
      "proves": "The whole model is two residual stacks (encoder left, decoder right) of self-attention + FFN; the only cross-stack link is encoder–decoder multi-head attention. Recurrence and convolution do not appear in the diagram.",
      "not_a_caption": true
    },
    {
      "id": "fig2",
      "label": "Figure 2",
      "location": "Section 3.2 / page 4",
      "proves": "Left: scaled dot-product is MatMul → scale → (optional mask) → softmax → MatMul with V. Right: multi-head is h independent projections, not one fat attention — concatenation + W^O is the only mixing across heads.",
      "not_a_caption": true
    },
    {
      "id": "table1",
      "label": "Table 1",
      "location": "Section 3.5 / page 6",
      "proves": "Self-attention has O(1) sequential operations and O(1) maximum path length, versus O(n) for recurrence and O(log_k n) path for dilated convolution. Complexity is O(n^2 · d) vs recurrent O(n · d^2).",
      "not_a_caption": true
    },
    {
      "id": "fig3",
      "label": "Figure 3 (appendix)",
      "location": "page 13",
      "proves": "Encoder self-attention in layer 5 of 6 can follow a long-distance dependency: heads on 'making' complete 'making … more difficult'.",
      "not_a_caption": true
    },
    {
      "id": "fig4",
      "label": "Figure 4 (appendix)",
      "location": "page 14",
      "proves": "Two heads in layer 5 appear to do anaphora: attention from 'its' is sharp (heads 5 and 6).",
      "not_a_caption": true
    },
    {
      "id": "fig5",
      "label": "Figure 5 (appendix)",
      "location": "page 15",
      "proves": "Different heads at layer 5 learn different structural behaviors on the same sentence, not a single averaged alignment.",
      "not_a_caption": true
    }
  ],
  "numbers": [
    {"value": "28.4 BLEU", "what": "Transformer (big) WMT 2014 EN–DE newstest2014", "where": "Abstract; Table 2; Section 6.1"},
    {"value": "41.8 BLEU", "what": "Transformer (big) WMT 2014 EN–FR, stated as new single-model SOTA", "where": "Abstract; Table 2"},
    {"value": "41.0 BLEU", "what": "Same EN–FR big model, different number in running text", "where": "Section 6.1 — UNRESOLVED INTERNAL INCONSISTENCY with Abstract/Table 2 41.8"},
    {"value": "27.3 BLEU / 38.1 BLEU", "what": "Transformer (base) EN–DE / EN–FR", "where": "Table 2"},
    {"value": ">2 BLEU over previous best (including ensembles) on EN–DE", "what": "Abstract; Section 6.1 says more than 2.0 BLEU", "where": "Abstract; Section 6.1"},
    {"value": "3.5 days on eight GPUs / eight P100 GPUs", "what": "Big-model training wall time / hardware", "where": "Abstract; Section 5.2; Section 6.1"},
    {"value": "twelve hours on eight P100 GPUs", "what": "Introduction claim for reaching a new SOTA after training", "where": "Section 1 (matches base 100,000 steps / 12 hours in Section 5.2)"},
    {"value": "N=6, d_model=512, d_ff=2048, h=8, d_k=d_v=64", "what": "Base architecture", "where": "Sections 3.1–3.3; Table 3 base row"},
    {"value": "65 × 10^6 params", "what": "Base parameter count", "where": "Table 3"},
    {"value": "big: N=6, d_model=1024, d_ff=4096, h=16, P_drop=0.3, 300K steps, 213 × 10^6 params", "what": "Big configuration (unlisted values = base)", "where": "Table 3 bottom row"},
    {"value": "dev PPL 4.92 / BLEU 25.8 (base); PPL 4.33 / BLEU 26.4 (big)", "what": "EN–DE newstest2013 development, per-wordpiece, no checkpoint averaging for variations", "where": "Table 3"},
    {"value": "4.5 million sentence pairs, ~37k shared BPE vocab", "what": "WMT 2014 EN–DE data", "where": "Section 5.1"},
    {"value": "36M sentences, 32k word-piece vocab", "what": "WMT 2014 EN–FR data", "where": "Section 5.1"},
    {"value": "~25,000 source tokens and 25,000 target tokens per batch", "what": "Batching", "where": "Section 5.1"},
    {"value": "0.4 s/step base; 1.0 s/step big; 100,000 / 300,000 steps", "what": "Schedule on 8× P100", "where": "Section 5.2"},
    {"value": "3.3e18 FLOPs (base), 2.3e19 FLOPs (big)", "what": "Estimated training cost, both language pairs share one number in Table 2", "where": "Table 2"},
    {"value": "ByteNet 23.75; GNMT+RL 24.6; ConvS2S 25.16; MoE 26.03; GNMT ens. 26.30; ConvS2S ens. 26.36 EN–DE BLEU", "what": "Table 2 baselines EN–DE", "where": "Table 2"},
    {"value": "Deep-Att+PosUnk 39.2; GNMT+RL 39.92; ConvS2S 40.46; MoE 40.56; Deep-Att ens. 40.4; GNMT ens. 41.16; ConvS2S ens. 41.29 EN–FR BLEU", "what": "Table 2 baselines EN–FR", "where": "Table 2"},
    {"value": "beam size 4, length penalty α=0.6; last 5 (base) / 20 (big) checkpoints averaged", "what": "MT decoding", "where": "Section 6.1"},
    {"value": "EN–FR big used P_drop=0.1 instead of 0.3", "what": "Exception to Table 3 big dropout", "where": "Section 6.1"},
    {"value": "single-head 0.9 BLEU worse than best head setting (dev)", "what": "Table 3 row (A) commentary", "where": "Section 6.2"},
    {"value": "4-layer Transformer, d_model=1024; WSJ 23 F1 91.3 (WSJ-only) and 92.7 (semi-supervised)", "what": "English constituency parsing", "where": "Section 6.3; Table 4"},
    {"value": "WSJ ~40K sentences; semi-supervised ~17M sentences; vocab 16K / 32K; beam 21, α=0.3; max output = input+300", "what": "Parsing setup", "where": "Section 6.3"},
    {"value": "P100 sustained estimate 9.5 TFLOPS (also K80 2.8, K40 3.7, M40 6.0)", "what": "FLOP accounting footnote", "where": "Section 6.1 footnote 5"}
  ],
  "limitations": [
    "Self-attention pays O(n^2 · d) and averages over positions, which the authors call reduced effective resolution; Multi-Head Attention is the proposed countermeasure, not a proof that resolution is fully restored (Sections 2, 3.2, 4).",
    "For very long sequences they explicitly defer restricted / local attention (neighborhood r, path O(n/r)) to future work (Section 4).",
    "Decoder generation remains auto-regressive; 'making generation less sequential' is listed as a research goal (Section 3, Conclusion).",
    "EN–FR big model uses a different dropout (0.1 vs 0.3) than the Table 3 big row; base EN–FR BLEU 38.1 is below several prior single models in Table 2 — the SOTA claim for FR is for the big model (Table 2, Section 6.1).",
    "Parsing experiments used only a small search over dropout, LR and beam; the model does not beat RNNG (Dyer et al., 2016) 93.3 generative F1 or Luong et al. multi-task 93.0 (Section 6.3, Table 4).",
    "Label smoothing ε_ls=0.1 is stated to hurt perplexity while helping BLEU — PPL and BLEU are not interchangeable quality metrics here (Section 5.4).",
    "Table 3 perplexities are per-wordpiece under their BPE and 'should not be compared to per-word perplexities' (Table 3 caption).",
    "Internal inconsistency: Section 6.1 says EN–FR big BLEU 41.0; Abstract and Table 2 say 41.8."
  ],
  "neighbor_papers": [
    {"cite": 13, "name": "LSTM", "authors": "Hochreiter & Schmidhuber, 1997", "role": "SOTA sequence modeling baseline the introduction is leaving"},
    {"cite": 7, "name": "GRU / gated recurrent", "authors": "Chung et al., 2014", "role": "Named with LSTM as established recurrent SOTA"},
    {"cite": 35, "name": "Seq2Seq", "authors": "Sutskever, Vinyals & Le, 2014", "role": "Encoder–decoder transduction template"},
    {"cite": 5, "name": "RNN Encoder–Decoder", "authors": "Cho et al., 2014", "role": "Encoder–decoder structure citation"},
    {"cite": 2, "name": "Additive attention NMT", "authors": "Bahdanau, Cho & Bengio, 2014", "role": "Attention + comparison point for additive vs dot-product"},
    {"cite": 38, "name": "GNMT + RL", "authors": "Wu et al., 2016", "role": "MT baseline and beam/length-penalty source; word-piece vocab"},
    {"cite": 24, "name": "Attention-based NMT", "authors": "Luong, Pham & Manning, 2015", "role": "Prior encoder–decoder attention work"},
    {"cite": 18, "name": "ByteNet", "authors": "Kalchbrenner et al., 2017", "role": "Conv baseline; logarithmic path length; Table 2 23.75 EN–DE"},
    {"cite": 9, "name": "ConvS2S", "authors": "Gehring et al., 2017", "role": "Conv seq2seq; linear path; Table 2; also learned PE comparison [9]"},
    {"cite": 16, "name": "Extended Neural GPU", "authors": "Kaiser & Bengio, 2016", "role": "Parallel conv-style reduction of sequential compute"},
    {"cite": 17, "name": "Neural GPUs", "authors": "Kaiser & Sutskever, 2016", "role": "Compared in Section 2 as non-Transformer parallel model"},
    {"cite": 32, "name": "MoE", "authors": "Shazeer et al., 2017", "role": "Table 2 baseline; also cited for conditional computation in intro"},
    {"cite": 39, "name": "Deep-Att + PosUnk", "authors": "Zhou et al., 2016", "role": "Table 2 EN–FR baseline / ensemble"},
    {"cite": 34, "name": "End-to-end Memory Networks", "authors": "Sukhbaatar et al., 2015", "role": "Recurrent attention without sequence-aligned recurrence"},
    {"cite": 27, "name": "Decomposable Attention", "authors": "Parikh et al., 2016", "role": "Rare case of attention not tied to an RNN (Section 1)"},
    {"cite": 4, "name": "LSTMN", "authors": "Cheng, Dong & Lapata, 2016", "role": "Prior self-attention / intra-attention use"},
    {"cite": 11, "name": "ResNet residuals", "authors": "He et al., 2016", "role": "Residual connection around sub-layers"},
    {"cite": 1, "name": "Layer Normalization", "authors": "Ba, Kiros & Hinton, 2016", "role": "Post-residual norm"},
    {"cite": 20, "name": "Adam", "authors": "Kingma & Ba, 2015", "role": "Optimizer"},
    {"cite": 31, "name": "BPE subwords", "authors": "Sennrich, Haddow & Birch, 2015", "role": "EN–DE tokenization (text also says byte-pair [3] in 5.1 — [3] is Britz et al.; BPE paper is [31])"},
    {"cite": 3, "name": "Britz et al. architecture exploration", "authors": "Britz et al., 2017", "role": "Cited for additive vs scaled dot-product at large d_k; Section 5.1 attributes BPE to [3] — likely a citation slip"},
    {"cite": 30, "name": "Output embedding tying", "authors": "Press & Wolf, 2016", "role": "Shared embedding / pre-softmax weights"},
    {"cite": 36, "name": "Label smoothing", "authors": "Szegedy et al., 2015", "role": "ε_ls=0.1"},
    {"cite": 33, "name": "Dropout", "authors": "Srivastava et al., 2014", "role": "Residual dropout"},
    {"cite": 12, "name": "Long-term dependency path length", "authors": "Hochreiter et al., 2001", "role": "Why shorter max path helps (Table 1 motivation)"},
    {"cite": 8, "name": "RNNG", "authors": "Dyer et al., 2016", "role": "Parsing SOTA the Transformer does not surpass (93.3 F1 generative)"},
    {"cite": 37, "name": "Grammar as a Foreign Language", "authors": "Vinyals & Kaiser et al., 2015", "role": "RNN seq2seq parsing baseline; small-data regime citation"},
    {"cite": 29, "name": "BerkeleyParser", "authors": "Petrov et al., 2006", "role": "WSJ-only parser the 4-layer Transformer beats (90.4 vs 91.3)"}
  ],
  "unverified_or_inconsistent": [
    "Section 6.1 EN–FR big BLEU 41.0 vs Abstract/Table 2 41.8 — both appear in the same v7 PDF; this page reports both and does not pick a 'true' score.",
    "Section 5.1 says sentences 'encoded using byte-pair encoding [3]'; [3] in the reference list is Britz et al. 2017, while BPE is [31] Sennrich et al. 2015. Reported as a citation inconsistency, not resolved.",
    "Table 2 FLOP cells for some ensemble rows wrap across lines in pdftotext (1.8·10^20, 7.7·10^19 reconstructed from layout + nearby exponents). Treat ensemble FLOPs as slightly OCR-fragile; BLEU columns are clear.",
    "Appendix Figures 3–5 are qualitative attention plots; this page does not reproduce paper attention weights (the heatmap is a labeled didactic toy)."
  ]
}