假設你想把客服文字分成「退款」和「物流」：一種做法是準備標記資料，訓練一個分類器；另一種做法是把分類規則和幾個例子放進文字，讓語言模型接續寫出類別。
兩者看起來都能輸出標籤，改變模型的方法卻不同。前者用 loss 和梯度更新參數；後者可以保持參數不變，只改變本次計算所依據的 context。從 2018 年的 GPT，到 2019 年 GPT-2、2020 年 GPT-3，一條值得理解的主線，就是研究者如何把「預測下一個 token」拿來處理愈來愈多任務。
本篇先把歷史與資訊流接起來，再用三個手設 Q／K／V 向量和一組固定 logits，實際核對 causal mask 與標籤位移。這不是 GPT 訓練、checkpoint 推論或分類品質評測。前一篇 BERT 的預訓練與微調 談雙向表徵；這裡轉向自回歸模型如何把 prefix 變成下一個 token 的分布。
三代研究，改變了什麼使用方式？
先抓住每份論文主要回答的問題，不把三年排成「模型變大，所以自然變聰明」的單一故事。
| 研究 | 主要使用方式 | 任務適應的關鍵 |
|---|---|---|
| GPT，2018 | 語言模型預訓練，再監督微調 | 下游標籤透過梯度改變參數 |
| GPT-2，2019 | 探索語言模型的 zero-shot 任務遷移 | 不為測試任務另改參數或架構，以文字提供任務脈絡 |
| GPT-3，2020 | 系統比較 zero／one／few-shot | 推論時給指令與不同數量示例，保持權重不變 |
第一份論文標題是 Improving Language Understanding by Generative Pre-Training；「GPT-1」是方便回顧系列時使用的稱呼。它的核心是兩階段遷移，不是宣稱預訓練從這一年才出現。GPT-2 進一步問：大量文字中的任務模式，能否透過語言建模轉成不另做任務微調的能力？GPT-3 則把 context 裡放多少示例，變成清楚區分的評估設定。原 GPT §3、GPT-2 §2、GPT-3 §2 是這段歷史比較的依據。
這不表示前一代完全沒有 zero-shot 探索，也不表示後來就不能微調。原 GPT 已分析部分 zero-shot 行為；GPT-3 論文也明確說模型原則上可以微調，只是該研究聚焦於不做任務權重更新的評估。這裡比較的是研究重心與實驗設定，並非替模型能力畫絕對邊界。
下一個 token：把整段文字拆成條件機率
令一段文字經 tokenizer 後得到 x₁, x₂, …, xT。自回歸語言模型把聯合機率分解成：
pθ(x₁, …, xT) = ∏ₜ pθ(xₜ | x₁, …, xₜ₋₁)
常見訓練目標：
L = −(1/T) ∑ₜ log pθ(xₜ | x₁, …, xₜ₋₁)
這裡用平均負對數似然說明一段有效 token 的目標；實際 batch、padding、文件邊界與 loss reduction 還要由實作明定。θ 是模型參數，條件部分是已知 prefix。token 是 tokenizer 的單位，可能是一段字、一部分單字或 bytes 的合併；不必等於一個中文字或英文單字。
訓練文字本身就能提供下一個 token 作為標籤，所以不必為每一個位置請人填答案。這也是早期論文稱 unsupervised pre-training、今天常以 self-supervised 任務理解它的原因。不過，能從原文產生標籤，只解決監督訊號的來源；它沒有消除語料品質、重複內容、評估污染與運算成本問題。
假設教學序列是 A B，並約定起始與結束符號，輸入和目標可以對齊成：
<BOS>AABB<EOS><BOS>／<EOS> 是本篇教學約定，不代表每個歷史 GPT checkpoint 都有相同特殊 token 名稱或 ID。關鍵是位置對齊：消耗 A 的位置要預測後面的 B，而不是把自己當作下一個答案。
訓練時，真實 prefix 已經在資料裡，可以把整段 shifted inputs 一起送入模型；各位置靠 causal mask 限制可見範圍。這是 teacher forcing 的要點：位置上的歷史 token 來自訓練資料，不必等模型逐字猜出它，才計算後面位置的 loss。
「可以平行計算多個訓練位置」與「預測時不能看未來答案」並不衝突。前者是計算安排，後者是資訊流約束；如果只把 labels 移位，卻讓 attention 讀到未來輸入，仍可能把答案洩漏進表徵。
Decoder-only 裡，資料如何走到詞彙分布？
用長度 T、hidden width d 和詞彙大小 Vocab 來看，一條簡化資料流是：token IDs 查 embedding，加入位置資訊，經過多層 Transformer blocks，得到每個位置的 hidden state，再映射成詞彙 logits。
token IDs [T]
embedding + position [T, d]
causal Transformer blocks [T, d]
vocabulary logits [T, Vocab]
softmax over vocabulary [T, Vocab]
表中省略 batch 軸。softmax 是沿詞彙軸，把每個位置的 logits 轉成下一 token 分布；attention 內部的 softmax 則沿可見 key 位置正規化，兩者不是同一個分布。前者回答「接哪個 token」，後者決定「這個 query 位置如何加權其他位置的 value」。
右圖可以用來讀「先做什麼、加回哪條 residual」，但仍有簡化。各頭算出的結果需要合併再通過輸出投影，圖上沒有另列 concat 標籤；Q、K、V 與縮放點積的細節要接回前面的 Self-Attention 教學。原圖寫 Positional Encoding，也不能據此推定所有 GPT 都用正弦位置編碼：2018 原 GPT 使用的是 learned position embeddings。
最外層的 head 同樣要辨認。把 hidden states 送進類別分類 head，可以得到客服類別；接詞彙輸出層，才是本節討論的 next-token logits。body 的表徵與 head 的任務不可只靠「Transformer」這個名稱混在一起。
Causal mask 限制讀取範圍，生成則逐步延長 prefix
以 query 位置 i、key 位置 j 表示縮放點積分數：
score[i,j] = dot(Q[i], K[j]) / sqrt(dk)
causal scores[i,j] = score[i,j] if j <= i
−∞ if j > i
weights[i,:] = softmax(causal scores[i,:])
output[i] = ∑ⱼ weights[i,j] V[j]
未來位置在 softmax 前被排除，所以其權重為零。這裡允許對角線：消耗第 i 個輸入 token 的位置，可以使用自己來預測下一個 token。它不等於使用下一個答案。標籤是否 shifted、mask 是否 causal，必須一起檢查。
生成時則不同。給定 prompt，模型先算出目前最後位置的下一 token 分布；用指定解碼規則選出一個 token，把它接到 prefix，再預測下一個。可以選最大機率，也可以依採樣規則抽取；「做一次 softmax」本身不決定採用哪種規則，更不保證選到事實正確的答案。
這也解釋 teacher forcing 與生成之間的差別。訓練時後續位置通常看到真實歷史 token；生成時，新的 prefix 包含剛才自己選出的 token，前一步的偏差可能影響後面的條件分布。停止可以依模型與呼叫端約定的結束 token、長度或其他條件決定；圖中的句點只是此例下一 token，並非所有生成一遇到句點就停止。
本文的程式只核對單層 attention 資訊流和一個位置的 CE，不含上述生成迴圈。原始 GPT 論文的任務評估也不一定都用自由生成；例如可以比較多個候選答案的條件機率，再依指定規則選擇。把所有評估都想成聊天續寫，會漏掉這些任務介面。
2018：先學語言模型，再讓任務標籤改寫參數
2018 原 GPT 先在 BooksCorpus 做語言模型預訓練，再把預訓練網路接到監督任務。模型已從連續文字得到初始化參數，下游標籤再改變這些參數，使表徵朝分類、蘊涵判斷或選擇題等目標調整。
例如要做客服分類，可以把一段文字送進 decoder，以指定讀取位置的表徵接分類線性層，讓類別 loss 反向傳播。這是說明遷移流程的自訂情境，並非原論文的客服實驗。原論文 §3.2 還描述在微調目標中加入加權語言模型輔助目標；不能把它概括成永遠只訓練新 head。
原 GPT 的另一個做法是把結構化任務轉成 token 序列。句對可以用 delimiter 串接；問答與常識選擇題則把文件、問題與各候選答案分別組成序列，計算候選的輸出。這讓不同任務能大量共用預訓練架構，但不代表所有任務從此有完全相同的輸入編排、讀取位置或 loss。
此論文的模型是 12 層 decoder、hidden width 768、12 heads、feed-forward 內層 3072，使用 learned positions、40,000 次 BPE merges，訓練片段長度 512。這些數字是歷史設定，讀者不必為理解流程逐一背誦；真正要分清的是：預訓練提供起點，監督微調用下游資料更新參數。 詳見原 GPT §3.1–3.3、§4.1。
2019：讓語言模型用文字脈絡處理不同任務
GPT-2 把研究問題往前推一步：如果預訓練文字裡本來就有問題與答案、雙語片段、摘要等模式，語言模型能否藉由文字脈絡，在沒有為測試任務另外訓練或微調的條件下執行任務？這是該論文探索 zero-shot task transfer 的方向，不是說任何任務都已可靠解決。
「zero-shot」也不等於沒有輸入規則、提示或任務格式。可以把問題、文件、任務描述放進 context，或設計候選答案評分方式；沒有的是針對該評估任務另做參數更新。是否用 prompt 與是否用梯度微調，是兩個不同問題。
資料也改變了。論文的 WebText 從 Reddit 上至少 3 karma 的外連出發，蒐集約 4,500 萬個連結，經去重與清理後留下超過 800 萬份文件、約 40GB 文字；該版本不含 2017 年 12 月後建立的連結，並移除 Wikipedia。這是其特定資料建置方式，不能寫成直接把整個網路或 Common Crawl 原封不動拿來訓練。
Tokenization 採 byte-level BPE，提供從 UTF-8 bytes 組合較常見字串的路徑；它仍有合併規則與預切分限制，不是拿掉 tokenizer，也不是每個 token 恆等於一個 byte。架構則改動 LayerNorm 位置、增加最後 LayerNorm、調整初始化，context 從 512 增為 1024。論文列出的最大模型為 1,542M 參數、48 層、width 1600，常以約 1.5B 稱呼。資料、切分、架構與規模一起變動，因此不能把論文間的能力差異全部歸因於單一參數量。GPT-2 §2.1–2.3 提供這些設定。
2020：示例放進 context，與微調是不同的適應方式
GPT-3 的研究把 zero-shot、one-shot、few-shot 區分得更清楚。依該論文的設定，三者推論時都不更新權重；差別主要在 context 是否放入任務 demonstrations，以及放幾個。
下面是本站自訂的 few-shot prompt，沒有送進模型執行，也沒有實測回覆。示例的目的，是把規則與格式具體化：
把客服訊息分為「退款」或「物流」，只輸出一個類別。
訊息：商品已退回，請問何時退費？
類別：退款
訊息：包裹已出貨，還查不到配送進度。
類別：物流
訊息：款項已扣，但取消訂單後還沒退回。
類別：
最後一個空白位置是待續寫答案，不是另一個已給定的正確 demonstration。若只保留第一個示例，就是 one-shot；若拿掉兩個示例、保留指令與最後問題，就是 zero-shot。這些改變都可以只動輸入文字，不觸發 optimizer。
「權重不變」不代表模型內部完全不變。新的 prompt 會產生新的 hidden states、attention 分布與下一 token 分布；那些是這次前向計算的中間結果，不能與持久參數更新混為一談。換 prompt 能改變行為，但不等於同時把新知永久寫進模型權重。
原 GPT-3 論文將 in-context learning 理解為推論時的示例條件化，並未裁定它究竟是從零學會新任務，還是辨識預訓練時已見過的模式。示例順序、格式、內容與任務都可能影響結果；多放示例不保證單調變好，也不保證答案正確。要比較 prompt，就需要固定評估資料、分開選擇與最終測試，而不是從幾次好看的回覆直接宣稱「學會了」。這與先前的資料切分與洩漏原則相通。
原論文 §2.4 通常從 task training set 抽 demonstrations；有些任務因資料切分不同，改用 dev 等來源。作者會在可用的 dev 上選部分設定，再做 test 評估。這是明定的評估協定，不是把待評分項目的正確答案塞進 prompt。本文自訂 prompt 沒有任何 benchmark 分數，亦不主張能複製原論文的效果。GPT-3 §2、§2.4 有完整定義與協定。
175B、300B、2048，三個數字在量不同東西
看 GPT-3 時，容易把「很大」的幾個數字混在一起。它們分別描述模型、訓練與一次可見上下文：
訓練 tokens 是累計取樣量，資料來源還有不同混合權重，部分語料會重複取樣。它與「獨特語料有多少 token」不是同一個數字。context 又是另一個限制：指令、示例與待處理文字需要佔用輸入空間，生成亦受模型與呼叫端的總長度安排限制；它不是模型只學過 2048 個 tokens。
原論文訓練多種大小，而「GPT-3」常專指最大的 175B 模型。它沿用 GPT-2 路線，卻不是單純把同一張 dense attention 圖放大：§2.1 說明層間交替使用 dense 與 locally banded sparse attention。圖 2 只是一般結構，圖 1 與下面三點例子則採完整 causal 可見位置，不是 GPT-3 每層 sparse pattern 的精確重現。
這篇的歷史範圍止於三份原始研究所討論的語言模型與使用方式。不能從這些設定直接推導現代聊天產品的指令遵循、工具使用或可靠性；那些還牽涉其他訓練方法、資料、系統介面與評估。此處有用的基礎是：模型如何得到下一 token 的機率，以及任務是在更新權重，還是在改變 context。GPT-3 表 2.1–2.2、§2.1–2.3 是數字與架構的依據。
動手核對：未來 K／V 改變，前面位置會跟著變嗎？
這個小程式只用 Python 標準函式庫，固定三個 2D queries、keys、values，沒有語料、tokenizer、神經網路參數或亂數。矩陣列是 query 位置，欄是 key 位置。先計算 causal 與不遮罩的 attention，再只改第三列 K／V，觀察前兩個位置。
第一個位置遮罩後只能看到第一個 key，權重就是 [1, 0, 0]。第二個 query 是 [0, 1]，可見 keys 的分數為 [0, 1/√2]；softmax 約為 [0.330238450673, 0.669761549327]，所以其 output 是對應 values 的加權和。
causal weights =
[[1, 0, 0],
[0.330238450673, 0.669761549327, 0],
[0.333333333333, 0.333333333333, 0.333333333333]]
causal output =
[[1, 0],
[0.330238450673, 0.669761549327],
[2, 1.666666666667]]
第三列剛好等權重，是因為本例第三個 query 與三個 keys 的點積都相同；這不是 causal attention 一般都均勻加權的性質。把最後一列 key 改成 [-1.5, 2.5]、value 改成 [-3, 6]，固定 Q 不動，前兩個 causal outputs 的最大差異均為 0；未遮罩時，兩個位置的最大差異分別約為 2.870649503931、3.047476721048。
這是一個受控的 K／V 擾動：它刻意固定 Q，不是把真實 token 換掉後重新由 embeddings 產生整組 Q／K／V，更不是完整多層 GPT 的行為測試。它能驗的是此算式與遮罩下的資訊依賴。
同一支程式另用固定詞彙順序 [BOS, A, B, EOS]、位置 0 的 logits [-0.2, 1.4, 0.3, -1.0]，核對 CE：
logsumexp(logits) ≈ 1.885806574215
正確 next-token label A： CE ≈ 1.885806574215 − 1.4 = 0.485806574215
錯用未移位 label BOS： CE ≈ 1.885806574215 − (−0.2) = 2.085806574215
此例正確 label 的 CE 較低，是因為刻意給了 A 較高 logit；換一組 logits，數值大小可以反過來。正確位移的理由是任務定義為預測下一 token，不是「位移後 loss 必然較低」。這段也只計算一個位置的 CE，沒有示範整個 batch 的 masked mean、反向傳播或學習。
以下完整程式可存為 causal_shift_and_attention.py，在已有 Python 3 的環境執行：
python3 causal_shift_and_attention.py > stdout.json
保存的實際執行環境為 Python 3.10.12，沒有第三方依賴、套件安裝、網路、模型下載或亂數。輸出 JSON 的重點值如上；程式保留到小數 12 位，其他環境最後幾位的浮點差異須分開判讀，未宣稱所有平台逐 byte 相同。程式裡直接重寫一次 logsumexp 的檢查是同公式的冗餘核對；另有獨立查核使用 PyTorch CPU float64 的 attention／CE backend 交叉驗算，但執行下方程式本身不需要安裝 PyTorch。
"""Tiny deterministic example of causal attention and shifted next-token targets.
Standard library only. This is not a GPT checkpoint, training run, ICL test,
quality evaluation, or benchmark.
"""
from __future__ import annotations
import json
import math
import platform
from typing import Sequence
Vector = list[float]
Matrix = list[Vector]
def dot(left: Sequence[float], right: Sequence[float]) -> float:
return sum(a * b for a, b in zip(left, right))
def softmax(logits: Sequence[float]) -> Vector:
"""Numerically stable softmax across the provided keys."""
top = max(logits)
exps = [math.exp(value - top) for value in logits]
total = sum(exps)
return [value / total for value in exps]
def attention(
queries: Matrix, keys: Matrix, values: Matrix, causal: bool
) -> tuple[Matrix, Matrix]:
"""Return [query,key] weights and one weighted-value vector per query."""
key_depth = len(queries[0])
if any(len(row) != key_depth for row in queries + keys):
raise ValueError("query/key vector depths must match")
if len(keys) != len(values):
raise ValueError("key and value sequence lengths must match")
all_weights: Matrix = []
outputs: Matrix = []
for query_index, query in enumerate(queries):
scores = [dot(query, key) / math.sqrt(key_depth) for key in keys]
if causal:
scores = [
score if key_index <= query_index else float("-inf")
for key_index, score in enumerate(scores)
]
weights = softmax(scores)
output = [
sum(weights[j] * values[j][dim] for j in range(len(values)))
for dim in range(len(values[0]))
]
all_weights.append(weights)
outputs.append(output)
return all_weights, outputs
def logsumexp(logits: Sequence[float]) -> float:
top = max(logits)
return top + math.log(sum(math.exp(value - top) for value in logits))
def cross_entropy(logits: Sequence[float], label_index: int) -> float:
return logsumexp(logits) - logits[label_index]
def rounded(values, digits: int = 12):
if isinstance(values, (int, float)):
return round(float(values), digits)
return [rounded(value, digits) for value in values]
def main() -> None:
# Hand-set three-token Q/K/V arrays. Row t is the vector for token t.
# There are no learned parameters, corpus, or random draws in this example.
queries: Matrix = [
[1.0, 0.0],
[0.0, 1.0],
[1.0, 1.0],
]
keys: Matrix = [
[1.0, 0.0],
[0.0, 1.0],
[2.0, -1.0],
]
values: Matrix = [
[1.0, 0.0],
[0.0, 1.0],
[5.0, 4.0],
]
causal_weights, causal_output = attention(queries, keys, values, causal=True)
full_weights, full_output = attention(queries, keys, values, causal=False)
# Change only the future (last-position) K and V vectors.
changed_keys = [row[:] for row in keys]
changed_values = [row[:] for row in values]
changed_keys[-1] = [-1.5, 2.5]
changed_values[-1] = [-3.0, 6.0]
changed_causal_weights, changed_causal_output = attention(
queries, changed_keys, changed_values, causal=True
)
changed_full_weights, changed_full_output = attention(
queries, changed_keys, changed_values, causal=False
)
prefix_causal_diffs = [
max(abs(a - b) for a, b in zip(causal_output[i], changed_causal_output[i]))
for i in range(2)
]
prefix_unmasked_diffs = [
max(abs(a - b) for a, b in zip(full_output[i], changed_full_output[i]))
for i in range(2)
]
causal_future_weights = [
causal_weights[q][k] for q in range(3) for k in range(q + 1, 3)
]
row_sums = [sum(row) for row in causal_weights]
assert all(abs(total - 1.0) < 1e-14 for total in row_sums)
assert all(weight == 0.0 for weight in causal_future_weights)
assert max(prefix_causal_diffs) < 1e-14
assert min(prefix_unmasked_diffs) > 1e-6
# Decoder inputs are shifted right by one relative to their next-token labels.
teacher_forced_inputs = ["<BOS>", "A", "B"]
targets = ["A", "B", "<EOS>"]
assert len(teacher_forced_inputs) == len(targets)
# One fixed position's vocabulary logits. This is a numeric CE illustration,
# not a model prediction. Vocab order: BOS, A, B, EOS.
vocabulary = ["<BOS>", "A", "B", "<EOS>"]
position_zero_logits = [-0.2, 1.4, 0.3, -1.0]
correct_target_index = vocabulary.index(targets[0])
incorrect_unshifted_target_index = vocabulary.index(teacher_forced_inputs[0])
log_z = logsumexp(position_zero_logits)
correct_ce = cross_entropy(position_zero_logits, correct_target_index)
unshifted_ce = cross_entropy(
position_zero_logits, incorrect_unshifted_target_index
)
# Independent scalar check written directly from the log-sum-exp definition.
max_logit = max(position_zero_logits)
direct_log_z = max_logit + math.log(
sum(math.exp(logit - max_logit) for logit in position_zero_logits)
)
direct_correct_ce = direct_log_z - position_zero_logits[1]
direct_unshifted_ce = direct_log_z - position_zero_logits[0]
assert abs(log_z - direct_log_z) < 1e-15
assert abs(correct_ce - direct_correct_ce) < 1e-15
assert abs(unshifted_ce - direct_unshifted_ce) < 1e-15
assert correct_ce < unshifted_ce
result = {
"scope": (
"Three fixed 2D Q/K/V vectors; scaled dot-product attention; causal "
"mask and unmasked comparison; one teacher-forced target-shift CE "
"example. No model, training, corpus, checkpoint, network, ICL "
"evaluation, task-quality claim, or benchmark."
),
"environment": {
"python": platform.python_version(),
"dependencies": "Python standard library only",
"randomness": "none",
"network_or_install": "none",
},
"attention": {
"sequence_length": 3,
"key_depth": 2,
"scaling": "dot(Q_i,K_j) / sqrt(2)",
"matrix_axes": "rows are query positions; columns are key positions",
"Q": queries,
"K": keys,
"V": values,
"causal_weights": rounded(causal_weights),
"causal_weight_row_sums": rounded(row_sums),
"causal_output": rounded(causal_output),
"future_key_weights": rounded(causal_future_weights),
"unmasked_weights": rounded(full_weights),
"unmasked_output": rounded(full_output),
"future_perturbation": {
"changed_last_key": changed_keys[-1],
"changed_last_value": changed_values[-1],
"causal_output_first_two_before": rounded(causal_output[:2]),
"causal_output_first_two_after": rounded(changed_causal_output[:2]),
"max_abs_differences_first_two_causal": rounded(prefix_causal_diffs),
"causal_weights_after": rounded(changed_causal_weights),
"max_abs_differences_first_two_unmasked": rounded(prefix_unmasked_diffs),
"unmasked_output_after": rounded(changed_full_output),
"unmasked_weights_after": rounded(changed_full_weights),
},
"checks": {
"every_causal_weight_row_sums_to_one": True,
"future_key_weights_are_exactly_zero": True,
"changing_future_KV_leaves_earlier_causal_outputs_unchanged": True,
"same_perturbation_changes_earlier_unmasked_outputs": True,
},
},
"teacher_forcing_shift": {
"input_tokens": teacher_forced_inputs,
"next_token_targets": targets,
"alignment": [
{"input": token, "target": target}
for token, target in zip(teacher_forced_inputs, targets)
],
"vocabulary_order": vocabulary,
"position_zero_fixed_logits": position_zero_logits,
"logsumexp": rounded(log_z),
"correct_shifted_label": {
"target": targets[0],
"formula": "logsumexp(logits) - logit[target_index]",
"cross_entropy": rounded(correct_ce),
},
"incorrect_unshifted_negative_control": {
"wrong_target": teacher_forced_inputs[0],
"why_wrong": "It asks the position consuming BOS to predict BOS, rather than the next token A.",
"cross_entropy": rounded(unshifted_ce),
},
"independent_scalar_check": {
"logsumexp": rounded(direct_log_z),
"correct_shifted_ce": rounded(direct_correct_ce),
"unshifted_negative_control_ce": rounded(direct_unshifted_ce),
"matches_implementation_within_1e-15": True,
},
},
"result_note": (
"All values arise from explicitly fixed toy vectors/logits. The "
"example isolates a causal information-flow rule and next-token "
"label alignment; it does not model GPT training or behavior."
),
}
print(json.dumps(result, ensure_ascii=False, indent=2, allow_nan=False))
if __name__ == "__main__":
main()
跑完後，先看 future_key_weights 是否全為 0、causal_weight_row_sums 是否全為 1，再看 future_perturbation 的前兩個 causal 差異，最後核對 CE label 是 A。這比只看到程式沒報錯更能檢查你是否理解 mask、軸與位移。
如果回到開頭的客服任務，可以用三個問題辨認自己正在做什麼：分類標籤有沒有進入 loss 並更新參數？答案示例是否只存在於這次 prompt？目前報告的是實際評估結果，還是手設示意？能把這三件事分清，才不會把一次看似合理的續寫，誤讀成已完成微調或已證明模型學會新任務。
來源與延伸閱讀
- 2018 原 GPT：Radford 等，Improving Language Understanding by Generative Pre-Training，官方 PDF，本篇歷史方法與設定依 §3.1–3.3、§4.1；不把 GPT 視為預訓練的最早起點。
- 2019 GPT-2：Radford 等，Language Models are Unsupervised Multitask Learners，官方 PDF，依 §2.1–2.3 與 zero-shot 設定。WebText 連結 cutoff 為 December 2017。
- 2020 GPT-3：Brown 等，Language Models are Few-Shot Learners，arXiv 與版本紀錄，依 Introduction 的定義、§2–2.4、表 2.1–2.2。原論文結果需回到其任務與評估協定，不能當本站小程式成效。
- 圖 1：D2L 原章〈Large-Scale Pretraining with Transformers〉，四作者 Aston Zhang、Zachary C. Lipton、Mu Li、Alexander J. Smola；固定來源 SVG；原圖與本次改作適用 CC BY-SA 4.0。
- 圖 2、手機版與封面：Marxav 原 PNG 作品頁，Marxav，own work，CC0 1.0。這是獨立 Commons 作品的授權，不是拿教材授權替其他架構圖清權。
- 圖 3：Hugging Face〈Behind the pipeline〉；transformer_and_head 原 SVG 固定版本。圖 4：Hugging Face〈How do Transformers work?〉；causal_modeling 原 SVG 固定版本。作者為 Hugging Face course authors；該固定圖片 dataset 的 README 自行聲明 Apache-2.0，完整 Apache 2.0 副本與本次歸屬／變更紀錄如下。
- 本篇的原創教學貢獻是按參數更新與 context 條件化重組歷史、對齊圖中資料流、自訂客服 prompt，以及解讀固定 Q／K／V 與 CE 的可重跑算術範圍。未下載或執行 GPT 權重，未訓練語料、測試 ICL 品質、重跑原論文 benchmark，亦沒有以換句話說取代來源署名。
圖像授權副本
以下完整 Apache License 2.0 副本適用於所列 HF 圖片及本次改作，不把 D2L 的 CC BY-SA 或 Marxav 的 CC0 改成 Apache。其後完整保留本次歸屬與變更紀錄：它是本站整理的 publication notice，不冒稱上游另有 NOTICE，也不代表 Hugging Face 為本站背書。
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
Publisher attribution/change notice for DS49
This notice is supplied by Sunny Data Science; it is not an upstream NOTICE file.
Original figures: causal_modeling.svg and transformer_and_head.svg.
Authors: Hugging Face course authors.
Image repository: https://huggingface.co/datasets/huggingface-course/documentation-images
Fixed image revision: 96e71ad956904da670e6827274fc2c9b735b09d5
Source context: https://huggingface.co/learn/llm-course/chapter1/4 and https://huggingface.co/learn/llm-course/chapter2/2
License: Apache License, Version 2.0. Full copy accompanies these figures.
Changes: original SVG rendered on white and proportionally resized; selected whole-first mobile details rearrange original pixels with white spacing. No new diagram labels, arrows, model outputs, or generated artwork.
The fixed image-dataset README declares Apache-2.0; its complete fixed tree contains no upstream NOTICE file. No endorsement is implied.
留言0
登入後就能留言。
登入