Encoder、Decoder 與 Tokenizer：同樣的架構為何有不同任務？

「BERT 用來理解，GPT 用來生成，T5 則兩者都能做。」這種記法很快，卻容易在實作時卡住：為什麼 GPT 也能接分類層？為什麼同樣是一串 token，有的模型預測被遮住的位置，有的預測下一個 token？而 tokenizer 裡寫的 Model，又是不是我們剛學的 Transformer？

要回答這些問題，可以沿著資料走一次：文字先被轉為 token IDs，IDs 查出向量，再依注意力遮罩交換資訊；最後，輸出層與訓練標籤決定模型正在學什麼。Encoder 或 Decoder 的名字只描述其中一部分，不能單獨決定模型能完成哪些任務。

本文先把這幾層拆開，再看 BERT、GPT、T5 的三張原圖。後半段用固定小詞表實際執行 BPE、WordPiece 與 Unicode 正規化；這些程式驗證的是切分機制，沒有下載或訓練語言模型。位置、因果遮罩與平行計算的基礎，可先回看上一篇 Transformer 教學。

先分清 tokenizer、骨幹與輸出層

假設要處理「這本書很好看」。從輸入到答案，至少有四個不同問題：

怎麼表示文字？ Tokenizer 決定切成哪些片段，再把片段對應到整數 ID；一個 token 不必是一個中文字或一個完整英文單字。
每個位置能讀到什麼？ Embedding 把 ID 轉成向量後，注意力的可見範圍決定各位置能讀哪些向量；位置資訊與網路層也參與表徵計算。
要求模型預測什麼？ 是某個被選中的原字詞、下一個 token，還是一段缺失文字？這由資料製作、標籤與 loss 決定。
怎麼把表徵變成答案？ 分類 head 可以輸出幾個類別的 logits；語言模型 head 則輸出詞表中各 token 的 logits。生成時還要有選 token、接回輸入與停止的程序。

因此，同一個 Transformer 骨幹可以接不同的 head；同一種 tokenizer 也可能供不同架構使用。只有檔案或類別名稱相似，並不表示預訓練權重、遮罩、詞表與任務可以任意互換。

Tokenizer 的 normalization、pre-tokenization、model 與 postprocessor 四階段 — 圖 1｜Hugging Face 原圖先把字母轉小寫，再做預切分、子詞切分與後處理；此處 Model 指 tokenizer 的切分算法，不是 Transformer。原圖的 U 只變成 u，並沒有展開成 you；tday→td／##ay、CLS／SEP 是圖中示意，不保證任一 BERT checkpoint 都切出相同結果。來源：Hugging Face 課程〈Normalization and pre-tokenization〉，Hugging Face course authors；圖片 dataset 固定版本依 Apache 2.0 提供。本文將完整 SVG 轉為白底 PNG；手機先列全圖，再重用各原 stage 標籤與其原 before→arrow→after 區域直排，邊界狀態會重複，只加白色間隔，沒有新增字詞或箭頭。封面把完整原圖等比置於白底 1600×900，沒有裁切或新增圖元。完整授權與本次歸屬／變更紀錄附於文末。

這張圖也提醒我們：切分算法只是 tokenizer 流程的一環。 同一個子詞詞表，搭配不同的正規化或預切分，就可能處理出不同輸入。後處理加入的特殊符號，也必須和模型預期一致。ID 是詞表索引，數字大小本身沒有「比較重要」或「比較接近某個意思」的含義。

BERT：讓同位置的表徵預測被選中的原 token

在常見的 BERT encoder-only 配置中，有效輸入位置可以互相注意。這讓某個位置的表徵同時受到左邊與右邊內容影響，適合拿來做分類、逐位置標註或抽取式問答等任務。

BERT 預訓練的 masked language modeling（MLM）會先選一些位置作為預測目標，再改動輸入。圖中的原句是 I love this red car；把 love 換成 mask 後，模型要在那個位置預測原來的 love。它不是把所有標籤整體往左或往右移一格。

BERT 遮詞預測與雙向注意力可見範圍 — 圖 2｜左圖輸入的 mask 對應上方原 token love；右圖表示各位置可以讀取所有有效輸入位置，藍格是可見性，不是實測注意力權重。這是預訓練示意，不代表所有下游推論都要放 mask，也不表示每個輸出位置都要算 MLM loss。來源：D2L〈Large-Scale Pretraining with Transformers〉，Aston Zhang、Zachary C. Lipton、Mu Li、Alexander J. Smola，CC BY-SA 4.0。本文保留完整原 SVG 轉為白底 PNG；手機先列全圖，再依序放大原左架構與右矩陣，保留原標籤與軸，沒有補畫連線，跨區對照以全圖為準；改作圖同採 CC BY-SA 4.0。

用教材符號列出位置對齊，可以寫成：

輸入：  <cls>   I   <mask>   this   red   car
目標：    —    —    love      —     —     —

破折號表示這個小示意沒有選那些位置作 MLM 計分目標，不是要把字面上的破折號餵給模型。教材的小寫角括號是示意寫法，也不是要指定真實 BERT 的 token 字串或 ID。

原 BERT 的做法更細：隨機選取約 15% 的 token 位置；在這些被選中的位置中，80% 換成 [MASK]、10% 換成隨機 token、10% 保持原樣。即使某個被選中的 token 沒改字面內容，它仍可能是需要預測的標籤。因此，「有哪些位置算 loss」與「輸入裡哪些字是 mask」不能當成同一張表。原版另外有 next sentence prediction 目標，這也不等於 segment ID 或注意力方向。原 BERT 方法

做評論分類時，常見路徑是把整段文字送入 BERT，取 [CLS] 的最終表徵，再由分類 head 輸出類別 logits。此時不用為了套用 BERT 而任意遮掉評論內容。若分類共有三類，分類 head 可以輸出三個 logits；MLM head 則是在選定位置對整個詞表做預測，兩者的輸出意義不同。

GPT：用已給的前綴預測下一個 token

常見的 GPT decoder-only 配置採因果注意力：一個輸入位置能讀到自己和之前的輸入，不能讀後方尚未給定的位置。訓練語言模型時，這個位置產生的 logits 用來預測下一個 token。

GPT 的 next-token 輸入標籤位移與因果注意力 — 圖 3｜左圖輸入 bos、I、like、this、book，對應上方目標 I、like、this、book、eos，已畫出一格位移。右矩陣的縱軸是要預測的輸出詞，橫軸是已右移的輸入；最下面預測 I 的那列只讀 bos。因果設定包含當下輸入位置，不能把「不看未來標籤」誤寫成「輸入對角線也不能看」。來源：D2L〈Large-Scale Pretraining with Transformers〉，Aston Zhang、Zachary C. Lipton、Mu Li、Alexander J. Smola，CC BY-SA 4.0。本文將完整原 SVG 轉為白底 PNG；手機先列全圖，再放大原左架構與右矩陣，保留 bos／eos、原軸與位移，沒有新增圖元；改作圖同採 CC BY-SA 4.0。

把圖中的一筆訓練資料平放，關係會更清楚：

模型輸入： <bos>    I     like    this    book
預測目標：    I    like   this    book   <eos>

在訓練時，後面的位置可以使用真實的已給前綴，並在因果遮罩下同時計算各位置的預測；生成時則還沒有未來答案，必須把選出的 token 接回去，再繼續產生下一個。一次前向能平行計算多個訓練位置，不表示自回歸生成能同時知道所有未來 token。

這種預訓練方式也沒有禁止分類。原始 GPT 論文就描述了在預訓練模型上加入輸出層，以序列最終位置的表徵預測分類標籤。只是要讓那個 head 和骨幹依對應資料與目標適配，不能把「可以接分類層」當成任意現成權重已經完成你的分類任務。原 GPT 的 supervised fine-tuning

T5：讀取完整來源，再生成另一段目標

Encoder–decoder 配置把兩條序列分開：Encoder 表示來源；Decoder 在已給的目標前綴上做因果計算，同時經 cross-attention 讀取來源表徵。來源與目標長度可以不同。

T5 把不同任務整理成 text-to-text：輸入文字，輸出目標文字。分類也能以生成標籤字串的方式處理，例如生成某個允許的類別名稱；這和直接輸出固定幾類的分類 logits 是不同的任務表達方式。原 T5 方法也要求判別生成文字是否落在有效標籤集合，而不是把任何字串都當作合法分類結果。T5 的輸入與輸出格式

在本篇採用的 span-corruption 預訓練示意中，來源不是一般翻譯句子。原句有兩段被拿走：love，以及 red car；每段用一個 sentinel 標記替代，目標則串接被拿走的片段及其標記。

T5 缺失片段目標與 encoder、cross、decoder 三種可見範圍 — 圖 4｜左圖來源為 I、X、this、Y，目標為 X、love、Y、red、car、Z；原圖用角括號標示 sentinel。右下方表示 encoder 輸入互看，右上左側表示 decoder 可讀完整來源，右上三角區表示目標位置的因果可見範圍。右圖 decoder 橫縱軸都用目標文字標位置，沒有畫出實際右移後的 decoder_input_ids 或 start ID，不能照軸文字把同位置的目標答案當成 decoder 輸入。末 sentinel Z 也不能直接等同實際 checkpoint 的 EOS token。來源：D2L〈Large-Scale Pretraining with Transformers〉，Aston Zhang、Zachary C. Lipton、Mu Li、Alexander J. Smola，CC BY-SA 4.0。本文保留完整原 SVG 轉為白底 PNG；手機先列全圖，再放大原左架構與右矩陣，完整保留軸與 sentinel，跨區關係依全圖解讀，沒有補畫位移或連線；改作圖同採 CC BY-SA 4.0。

用完整文字列出來源與目標，可以避免把 span corruption 誤當成「輸出原句每一個字」：

原句：         I love this red car
Encoder輸入：  I <X> this <Y>
目標：         <X> love <Y> red car <Z>

若用 teacher forcing，Decoder 的概念輸入是目標向右移一格：

Decoder輸入： START  <X>  love  <Y>  red  car
預測目標：     <X>  love  <Y>   red  car  <Z>

START 只是這裡的說明符號，不是承諾實際 ID。這張表也只對齊圖中的六個目標位置，不另外假造 checkpoint 的 EOS 或 padding 配置。前面可見的 span sentinel 用來界定缺失片段；真正的停止符號、起始符號和 IDs 要依所用 tokenizer 與模型設定核對。

三種 mask 與標籤位移要分開查

容易混在一起的其實是三件事：改動輸入內容、限制注意力連接，以及選擇哪些位置計分。 把某個字換成 [MASK]，不等於把那個位置從 attention 中移除；PAD 是否能被其他位置讀取，也不會自動決定 loss 是否計算 PAD 的標籤。

同樣地，資料表中的位移與 API 在內部做的位移，不能重複。以下是對固定 Hugging Face Transformers v4.45.2 原始碼的行為查核，不是本站執行過 BERT／GPT2／T5 模型：

BertForMaskedLM 直接將各位置 logits 對上同位置 labels；呼叫者仍須指定哪些位置不計分。這不是 next-token shift。
GPT2LMHeadModel 的 loss 內部用前面的 logits 對後一格 labels。因此，使用這個接口時，不能外面先錯移一次、裡面再照原約定移第二次；手寫已位移輸入／目標的交叉熵，則是另一套需要清楚定義的接口約定。
T5ForConditionalGeneration 在提供 labels、但沒有提供 decoder inputs 時，才呼叫 _shift_right 建立 decoder 輸入；loss 仍對原 labels。該版本 helper 讀設定中的 decoder start ID，並把 labels 中的 -100 在輸入那一側換成合法的 PAD ID。起始位置可能使用 PAD 的 ID，卻仍是有效起始訊號，不能只靠「ID 等於 PAD」便把它遮掉。

這些約定可直接查固定版 BERT loss、GPT2 loss 與 T5 shift／loss。其中 -100 是這些 loss 接口的忽略標籤約定，不是可拿去查 embedding 的 token ID。

也不要只憑 Encoder／Decoder 的類別名稱猜 mask。可見性由實際傳入的遮罩、模型設定與運算路徑共同決定；改一個旗標不會自動帶來沒有訓練過的任務能力。選模型時，應一起核對預訓練任務、輸出 head、資料格式與現有權重，再判斷如何適配你的任務。

BPE 和 WordPiece：同樣的詞表也可能切得不同

回到 tokenizer，先把「學出詞表」和「用固定詞表切新文字」分開。下面的小例完全不訓練詞表，而是手動指定內容，直接核對推論規則。

BPE 使用有順序的 merge rules。令詞表包含 a、b、c、ab、bc，新字串為 abc：

初始片段：a | b | c

規則順序一：先合 b+c，再合 a+b
結果：      a | bc

規則順序二：先合 a+b，再合 b+c
結果：      ab | c

第一種情況合併 b 和 c 之後，已經沒有相鄰的 a、b 可供第二條規則合併。兩種規則順序使用的是同一個詞表，結果卻不同；所以，從左邊只挑詞表中最長的前綴，並不能取代這裡的 deterministic BPE merge rank。詞表告訴你「有哪些 token」，merge order 還告訴你「先做哪次合併」。

WordPiece 的固定詞表推論則可以用 greedy longest-match 理解：在當前位置找最長的合法 token；非詞首片段使用 ## 這類 continuation prefix。本例詞表為 [UNK]、ab、a、##bc、##c、b，實際執行結果如下：

輸入	輸出 token	原因
`abc`	`ab, ##c`	詞首先取最長 ab，再接合法 continuation
`bc`	`b, ##c`	詞首沒有 bc，先取 b
`c`	`[UNK]`	詞表只有 continuation 的 ##c，沒有詞首 c
`abx`	`[UNK]`	雖先配到 ab，剩下 x 無法匹配，整個預切分詞變成 UNK

abx 這列尤其值得看：結果不是 ab, [UNK]。在此 WordPiece 配置中，一個 pretokenized word 的任何剩餘部分失敗，就會放棄先前的局部成功片段，整詞輸出一個 UNK。程式另外測試 abc bc：先按空白分成兩詞，各自切分，不跨詞合併。這只是本例明選的 WhitespaceSplit，不是所有 tokenizer 或中文文字都必然遵守的切分方式。

BPE 本例的 a|bc 是 IDs [1,5]；換順序的 ab|c 是 [4,3]。WordPiece 的 ab|##c 則是 [1,4]。這些 ID 都由本例手設，兩個詞表即使出現相同整數，也不代表對應相同文字；更不能拿來代替真實 BERT 或 GPT 的詞表 ID。

這裡沒有比較哪種算法比較懂語意。尤其 WordPiece 的推論最長匹配，不等於詞表訓練算法；HF 課程也明示 Google 沒有公開原始 training implementation，課程的訓練評分示範是依文獻重建，不能把它或某個同名 trainer 當成已完整重現原 BERT 訓練。BPE 官方課程、WordPiece 官方課程

字元、byte 與 token 是三種不同計數

字串顯示起來很短，編碼後未必只有同樣多的 bytes；token 數又取決於 tokenizer。實際執行 Python 的 Unicode 字串與 strict UTF-8 編碼，得到：

字串	Codepoints	UTF-8 hex	Bytes
`A`	1	`41`	1
`中`	1	`e4 b8 ad`	3
預組合的 `é`	1	`c3 a9`	2
`e` 加 U+0301	2	`65 cc 81`	3

最後兩列可以呈現很相近的字形，但 codepoint 序列不同。Python 的 len 在這些字串上計算 codepoints，不是 bytes，也不是一般使用者眼中所有可見字形的數量。上表的每個原字串都能透過 strict UTF-8 encode／decode 精確還原；這驗證的是編碼層，不是某個 tokenizer decoder 已做 roundtrip。UTF-8 定義

接著，程式另外使用明確的 NFD → Lowercase → StripAccents 正規化，三種原文 É、E 加 U+0301、e 都變成同一個 e。既然不同原文碰撞到相同結果，就無法只憑結果恢復原大小寫、重音與組合形式。另一個獨立例子是 NFKC 把全形 Ａ 與圈號 ① 組成的字串，轉成 A1。

這兩組正規化是獨立示範，沒有偷偷套用在前面的 BPE 或 WordPiece 配置。真實 tokenizer 是否轉小寫、移除重音、保留空白，要查自己的 checkpoint 設定；不能把 uncased 的行為推到所有 BERT，也不能把「可還原正規化後文字」當成「可還原原始輸入」。

在 CPU 重跑這些切分機制

下面是完整程式。實際環境是 Python 3.10.12、tokenizers 0.22.2；Python 標準庫 unicodedata 的版本為 13.0.0。這不表示套件內部全部 Unicode 資料表也使用同一版本。兩個新程序執行皆 exit 0、沒有 stderr，輸出的 JSON 逐 byte 相同。

若已有相依套件，把下方程式存成 tokenizer_mechanics.py，執行 python3 tokenizer_mechanics.py。依賴只有 tokenizers==0.22.2；若自己環境尚未安裝，可參考 python3 -m pip install tokenizers==0.22.2。安裝指令是設定示例，本次沒有另外測試新虛擬環境或安裝流程，也沒有安裝 Transformers、SentencePiece 或下載模型。

成功時，JSON 中 bpe.same_vocabulary_two_merge_orders 會依序出現 a,bc 與 ab,c；wordpiece.cases 可核對上表與雙詞例；normalization_separate_from_tokenizers 下的 distinct_originals_collide 為 true。完整 tokens、IDs、merge trace、codepoints 與 UTF-8 bytes 都會輸出，方便逐項追查。

可讀參照函式只處理非空的 ASCII 字母，刻意沒有包裝成通用 tokenizer；第三方套件的能力並不受這個小函式限制。本文沒有訓練詞表、測語言品質或執行架構模型，也沒有測 token offsets、特殊符號 postprocessor 或 tokenizer decoder；不要從幾個成功案例延伸成全 Unicode、全輸入或所有模型都能無損還原的保證。

"""Fixed-vocabulary tokenizer mechanics, not pretrained language-model inference.

Requires tokenizers==0.22.2. No training, models, network, offsets, special-token
postprocessor, architecture run, or tokenizer quality benchmark is performed.
"""
from __future__ import annotations

import json
import platform
import unicodedata
import tokenizers
from tokenizers import Tokenizer, models, normalizers, pre_tokenizers


def ascii_word(word: str) -> None:
    if not word or not word.isascii() or not word.isalpha():
        raise ValueError("The readable reference algorithms accept nonempty ASCII letters only")


def ranked_bpe_reference(word: str, ordered_merges: list[tuple[str, str]]):
    """Small deterministic reference, restricted to ASCII and no dropout."""
    ascii_word(word)
    ranks = {pair: rank for rank, pair in enumerate(ordered_merges)}
    pieces = list(word)
    trace = []
    while True:
        candidates = [
            (ranks[(left, right)], pos)
            for pos, (left, right) in enumerate(zip(pieces, pieces[1:]))
            if (left, right) in ranks
        ]
        if not candidates:
            return pieces, trace
        rank, pos = min(candidates)  # Lowest rank first; leftmost on a tie.
        trace.append({"before": pieces[:], "pair": pieces[pos:pos + 2],
                      "rank": rank, "position": pos})
        pieces[pos:pos + 2] = [pieces[pos] + pieces[pos + 1]]
        trace[-1]["after"] = pieces[:]


def longest_prefix_reference(word: str, vocabulary: dict[str, int]):
    """A vocabulary-only contrast; this is deliberately NOT the BPE algorithm."""
    ascii_word(word)
    pieces, start = [], 0
    while start < len(word):
        matches = [token for token in vocabulary if word.startswith(token, start)]
        if not matches:
            raise ValueError("No vocabulary prefix for this contrast case")
        piece = max(matches, key=len)
        pieces.append(piece)
        start += len(piece)
    return pieces


def wordpiece_reference(word: str, vocabulary: dict[str, int]):
    """Longest match with ## on noninitial pieces; whole word UNK on failure."""
    ascii_word(word)
    pieces, trace, start = [], [], 0
    while start < len(word):
        prefix = "" if start == 0 else "##"
        match = next((prefix + word[start:end]
                      for end in range(len(word), start, -1)
                      if prefix + word[start:end] in vocabulary), None)
        if match is None:
            trace.append({"unmatched_suffix": word[start:],
                          "discarded_successful_prefix_pieces": pieces[:]})
            return ["[UNK]"], trace
        trace.append({"position": start, "matched": match})
        pieces.append(match)
        start += len(match) - len(prefix)
    return pieces, trace


def describe_unicode(text: str):
    encoded = text.encode("utf-8", errors="strict")
    return {
        "text": text,
        "unicode_codepoints": [f"U+{ord(char):04X}" for char in text],
        "python_codepoint_count": len(text),
        "utf8_bytes_hex": encoded.hex(" "),
        "utf8_byte_count": len(encoded),
        "utf8_decode_roundtrip": encoded.decode("utf-8") == text,
    }


def main() -> None:
    bpe_vocab = {"[UNK]": 0, "a": 1, "b": 2, "c": 3, "ab": 4, "bc": 5}
    orders = [[("b", "c"), ("a", "b")], [("a", "b"), ("b", "c")]]
    bpe_cases = []
    for merges in orders:
        tokenizer = Tokenizer(models.BPE(
            vocab=bpe_vocab, merges=merges, unk_token="[UNK]",
            dropout=None, ignore_merges=False, byte_fallback=False,
        ))
        # No normalizer or pre-tokenizer: this single ASCII word is unchanged.
        observed = tokenizer.encode("abc", add_special_tokens=False)
        expected, trace = ranked_bpe_reference("abc", merges)
        assert observed.tokens == expected
        assert observed.ids == [bpe_vocab[token] for token in expected]
        bpe_cases.append({"ordered_merges_rank_zero_first": merges,
                          "input": "abc", "tokens": observed.tokens,
                          "ids": observed.ids, "reference_trace": trace})
    contrast = longest_prefix_reference("abc", bpe_vocab)
    assert bpe_cases[0]["tokens"] == ["a", "bc"]
    assert bpe_cases[1]["tokens"] == contrast == ["ab", "c"]

    wp_vocab = {"[UNK]": 0, "ab": 1, "a": 2, "##bc": 3, "##c": 4, "b": 5}
    wp = Tokenizer(models.WordPiece(
        vocab=wp_vocab, unk_token="[UNK]", continuing_subword_prefix="##",
        max_input_chars_per_word=100,
    ))
    wp.pre_tokenizer = pre_tokenizers.WhitespaceSplit()
    wp_cases = []
    for text in ["abc", "bc", "c", "abx", "abc bc"]:
        words = text.split()  # These fixed samples contain only ASCII spaces.
        references = [wordpiece_reference(word, wp_vocab) for word in words]
        expected = [piece for pieces, _ in references for piece in pieces]
        observed = wp.encode(text, add_special_tokens=False)
        assert observed.tokens == expected
        assert observed.ids == [wp_vocab[token] for token in expected]
        wp_cases.append({"input": text, "whitespace_pretokens": words,
                         "tokens": observed.tokens, "ids": observed.ids,
                         "per_word_reference_traces": [trace for _, trace in references]})
    assert wp_cases[0]["tokens"] == ["ab", "##c"]
    assert wp_cases[2]["tokens"] == wp_cases[3]["tokens"] == ["[UNK]"]

    unicode_cases = [describe_unicode(text) for text in ["A", "中", "é", "e\u0301"]]
    # These are separate demonstrations, not a hidden part of either tokenizer.
    normalizer = normalizers.Sequence([
        normalizers.NFD(), normalizers.Lowercase(), normalizers.StripAccents(),
    ])
    normalization = []
    for text in ["É", "E\u0301", "e"]:
        normalized = normalizer.normalize_str(text)
        # Independent standard-library check for these Latin samples only.
        reference = "".join(c for c in unicodedata.normalize("NFD", text).lower()
                            if unicodedata.category(c) != "Mn")
        assert normalized == reference == "e"
        normalization.append({"input": describe_unicode(text),
                              "normalized": describe_unicode(normalized),
                              "input_equals_normalized": text == normalized})
    compatible_input = "Ａ①"
    compatible_output = normalizers.NFKC().normalize_str(compatible_input)
    assert compatible_output == unicodedata.normalize("NFKC", compatible_input) == "A1"

    result = {
        "scope": "Tiny fixed vocabularies/merges; BPE and WordPiece inference only, plus independent Unicode normalization examples. No tokenizer training, downloaded vocabulary/model, Transformers/SentencePiece run, architecture inference, quality benchmark or offsets.",
        "environment": {"python": platform.python_version(),
                        "tokenizers": tokenizers.__version__,
                        "python_unicodedata_version": unicodedata.unidata_version,
                        "execution": "CPU host; no GPU or neural model"},
        "bpe": {"vocabulary_ids_hand_assigned": bpe_vocab,
                "options": {"dropout": None, "ignore_merges": False,
                            "byte_fallback": False, "normalizer": None,
                            "pre_tokenizer": None, "special_token_postprocessor": None},
                "same_vocabulary_two_merge_orders": bpe_cases,
                "vocabulary_only_longest_prefix_contrast": contrast,
                "meaning": "The same vocabulary permits a|bc or ab|c. Ranked merges, not vocabulary longest-prefix alone, determine these deterministic BPE results. This is not byte-level BPE and has no all-Unicode coverage claim."},
        "wordpiece": {"vocabulary_ids_hand_assigned": wp_vocab,
                      "options": {"continuation_prefix": "##", "normalizer": None,
                                  "pre_tokenizer": "WhitespaceSplit",
                                  "max_input_chars_per_word": 100,
                                  "special_token_postprocessor": None},
                      "cases": wp_cases,
                      "meaning": "##c is legal as a continuation but no plain c token exists at word start. For abx the matched ab prefix is discarded: one entire pretokenized word becomes [UNK]. No claim about original Google WordPiece training or trainer scoring."},
        "unicode_codepoints_and_utf8": unicode_cases,
        "normalization_separate_from_tokenizers": {
            "pipeline": ["NFD", "Lowercase", "StripAccents"],
            "cases": normalization,
            "distinct_originals_collide": len({row["input"]["text"] for row in normalization}) == 3 and len({row["normalized"]["text"] for row in normalization}) == 1,
            "separate_NFKC_compatibility_case": {"input": describe_unicode(compatible_input),
                                                 "normalized": describe_unicode(compatible_output)},
            "meaning": "UTF-8 encode/decode is exact for these valid strings. Normalization maps different originals to the same output, so the normalized string alone cannot recover original case, accent or composition. NFKC additionally folds the shown compatibility forms. Codepoint count is not a general visible-grapheme count. No token offsets or decoder roundtrip is tested."},
    }
    print(json.dumps(result, ensure_ascii=False, indent=2, allow_nan=False))


if __name__ == "__main__":
    main()

把這些差別帶回任務

假設你手上是客服訊息，先問預期答案的形式，再檢查模型與資料：若要固定幾個類別，可以用合適的序列表徵接分類 head；若要產生摘要或另一種長度的文字，便要定義生成目標、已給條件與停止方式。BERT、GPT、T5 都提供了常見路徑，但架構名字不會替你完成資料標籤、微調或品質評估。

Tokenizer 則是這條路徑的輸入契約。正規化、預切分、詞表、merge order、特殊符號和 ID，應和所用權重配套。本文的手設 ID 只為看清機制；換成真實任務時，不能把它們直接餵進另一個 checkpoint 的 embedding。新增 token 也不會自動替模型建立正確的新表徵。

讀一段模型範例時，可以依序追查：輸入文字變成了哪些 IDs？每個位置能讀哪些輸入？logits 要預測的是同位置、下一位置，還是另一條目標序列？忽略標籤與位移由誰處理？這些問題通常比先背「理解模型／生成模型」更能找到資料流的錯誤。

原始方法與圖像來源

本文新增的是小型可重跑切分案例、逐步對齊與圖意解釋；三種架構的原圖與方法分別來自以下來源。沒有拿圖中的藍格當本站實跑權重，也沒有重現原論文模型的品質結果。

圖 1 與封面：HF〈Normalization and pre-tokenization〉；圖片原件由 documentation-images 固定版本提供，其 dataset README 自行聲明 Apache 2.0。作者為 Hugging Face course authors；這不是只憑程式碼倉庫推定圖片授權。完整 Apache 2.0 副本與本次歸屬／變更紀錄在下一節。
圖 2、3、4：D2L〈Large-Scale Pretraining with Transformers〉，使用 D2L 英文教材固定版本的三張原作者向量示意。四作者為 Aston Zhang、Zachary C. Lipton、Mu Li、Alexander J. Smola，依 CC BY-SA 4.0 使用；轉檔與手機裁切改作圖片維持相同授權。這份 grant 不擴張到章中另有 paper credit 的照片或論文圖。
原始方法：BERT、GPT、T5。本文連結原文，未採其論文原圖。BERT 的位置選取、GPT 的監督式分類適配、T5 的 text-to-text 與 span 目標，各應依原方法的範圍理解。
Tokenizer 推論：HF BPE 課程、WordPiece 課程；實際套件行為對照固定 tokenizers v0.22.2 BPE、merge rank 實作與 WordPiece。本文使用手設詞表推論，沒有重現課程或 Google 的詞表訓練。
本文 HF 模型 API 說明固定於 Transformers v4.45.2 的前述原始碼；本機實跑只有 tokenizer 小程式，不包含該套件模型或權重。圖像的 Apache 2.0 與 D2L 的 CC BY-SA 4.0 分開適用，不混成一種授權。

圖像授權副本

以下為圖 1、手機改作及完整圖封面所需的 Apache License 2.0 完整副本。這份授權適用於所列 HF 圖片，並不把 D2L 三張圖改標為 Apache。其後保留本次圖像歸屬與變更紀錄；該紀錄由本次製作整理，不冒稱上游另外提供了 NOTICE 文件，也不表示 HF 對本站背書。

                                 Apache License
                           Version 2.0, January 2004
                        http://www.apache.org/licenses/

   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION

   1. Definitions.

      "License" shall mean the terms and conditions for use, reproduction,
      and distribution as defined by Sections 1 through 9 of this document.

      "Licensor" shall mean the copyright owner or entity authorized by
      the copyright owner that is granting the License.

      "Legal Entity" shall mean the union of the acting entity and all
      other entities that control, are controlled by, or are under common
      control with that entity. For the purposes of this definition,
      "control" means (i) the power, direct or indirect, to cause the
      direction or management of such entity, whether by contract or
      otherwise, or (ii) ownership of fifty percent (50%) or more of the
      outstanding shares, or (iii) beneficial ownership of such entity.

      "You" (or "Your") shall mean an individual or Legal Entity
      exercising permissions granted by this License.

      "Source" form shall mean the preferred form for making modifications,
      including but not limited to software source code, documentation
      source, and configuration files.

      "Object" form shall mean any form resulting from mechanical
      transformation or translation of a Source form, including but
      not limited to compiled object code, generated documentation,
      and conversions to other media types.

      "Work" shall mean the work of authorship, whether in Source or
      Object form, made available under the License, as indicated by a
      copyright notice that is included in or attached to the work
      (an example is provided in the Appendix below).

      "Derivative Works" shall mean any work, whether in Source or Object
      form, that is based on (or derived from) the Work and for which the
      editorial revisions, annotations, elaborations, or other modifications
      represent, as a whole, an original work of authorship. For the purposes
      of this License, Derivative Works shall not include works that remain
      separable from, or merely link (or bind by name) to the interfaces of,
      the Work and Derivative Works thereof.

      "Contribution" shall mean any work of authorship, including
      the original version of the Work and any modifications or additions
      to that Work or Derivative Works thereof, that is intentionally
      submitted to Licensor for inclusion in the Work by the copyright owner
      or by an individual or Legal Entity authorized to submit on behalf of
      the copyright owner. For the purposes of this definition, "submitted"
      means any form of electronic, verbal, or written communication sent
      to the Licensor or its representatives, including but not limited to
      communication on electronic mailing lists, source code control systems,
      and issue tracking systems that are managed by, or on behalf of, the
      Licensor for the purpose of discussing and improving the Work, but
      excluding communication that is conspicuously marked or otherwise
      designated in writing by the copyright owner as "Not a Contribution."

      "Contributor" shall mean Licensor and any individual or Legal Entity
      on behalf of whom a Contribution has been received by Licensor and
      subsequently incorporated within the Work.

   2. Grant of Copyright License. Subject to the terms and conditions of
      this License, each Contributor hereby grants to You a perpetual,
      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
      copyright license to reproduce, prepare Derivative Works of,
      publicly display, publicly perform, sublicense, and distribute the
      Work and such Derivative Works in Source or Object form.

   3. Grant of Patent License. Subject to the terms and conditions of
      this License, each Contributor hereby grants to You a perpetual,
      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
      (except as stated in this section) patent license to make, have made,
      use, offer to sell, sell, import, and otherwise transfer the Work,
      where such license applies only to those patent claims licensable
      by such Contributor that are necessarily infringed by their
      Contribution(s) alone or by combination of their Contribution(s)
      with the Work to which such Contribution(s) was submitted. If You
      institute patent litigation against any entity (including a
      cross-claim or counterclaim in a lawsuit) alleging that the Work
      or a Contribution incorporated within the Work constitutes direct
      or contributory patent infringement, then any patent licenses
      granted to You under this License for that Work shall terminate
      as of the date such litigation is filed.

   4. Redistribution. You may reproduce and distribute copies of the
      Work or Derivative Works thereof in any medium, with or without
      modifications, and in Source or Object form, provided that You
      meet the following conditions:

      (a) You must give any other recipients of the Work or
          Derivative Works a copy of this License; and

      (b) You must cause any modified files to carry prominent notices
          stating that You changed the files; and

      (c) You must retain, in the Source form of any Derivative Works
          that You distribute, all copyright, patent, trademark, and
          attribution notices from the Source form of the Work,
          excluding those notices that do not pertain to any part of
          the Derivative Works; and

      (d) If the Work includes a "NOTICE" text file as part of its
          distribution, then any Derivative Works that You distribute must
          include a readable copy of the attribution notices contained
          within such NOTICE file, excluding those notices that do not
          pertain to any part of the Derivative Works, in at least one
          of the following places: within a NOTICE text file distributed
          as part of the Derivative Works; within the Source form or
          documentation, if provided along with the Derivative Works; or,
          within a display generated by the Derivative Works, if and
          wherever such third-party notices normally appear. The contents
          of the NOTICE file are for informational purposes only and
          do not modify the License. You may add Your own attribution
          notices within Derivative Works that You distribute, alongside
          or as an addendum to the NOTICE text from the Work, provided
          that such additional attribution notices cannot be construed
          as modifying the License.

      You may add Your own copyright statement to Your modifications and
      may provide additional or different license terms and conditions
      for use, reproduction, or distribution of Your modifications, or
      for any such Derivative Works as a whole, provided Your use,
      reproduction, and distribution of the Work otherwise complies with
      the conditions stated in this License.

   5. Submission of Contributions. Unless You explicitly state otherwise,
      any Contribution intentionally submitted for inclusion in the Work
      by You to the Licensor shall be under the terms and conditions of
      this License, without any additional terms or conditions.
      Notwithstanding the above, nothing herein shall supersede or modify
      the terms of any separate license agreement you may have executed
      with Licensor regarding such Contributions.

   6. Trademarks. This License does not grant permission to use the trade
      names, trademarks, service marks, or product names of the Licensor,
      except as required for reasonable and customary use in describing the
      origin of the Work and reproducing the content of the NOTICE file.

   7. Disclaimer of Warranty. Unless required by applicable law or
      agreed to in writing, Licensor provides the Work (and each
      Contributor provides its Contributions) on an "AS IS" BASIS,
      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
      implied, including, without limitation, any warranties or conditions
      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
      PARTICULAR PURPOSE. You are solely responsible for determining the
      appropriateness of using or redistributing the Work and assume any
      risks associated with Your exercise of permissions under this License.

   8. Limitation of Liability. In no event and under no legal theory,
      whether in tort (including negligence), contract, or otherwise,
      unless required by applicable law (such as deliberate and grossly
      negligent acts) or agreed to in writing, shall any Contributor be
      liable to You for damages, including any direct, indirect, special,
      incidental, or consequential damages of any character arising as a
      result of this License or out of the use or inability to use the
      Work (including but not limited to damages for loss of goodwill,
      work stoppage, computer failure or malfunction, or any and all
      other commercial damages or losses), even if such Contributor
      has been advised of the possibility of such damages.

   9. Accepting Warranty or Additional Liability. While redistributing
      the Work or Derivative Works thereof, You may choose to offer,
      and charge a fee for, acceptance of support, warranty, indemnity,
      or other liability obligations and/or rights consistent with this
      License. However, in accepting such obligations, You may act only
      on Your own behalf and on Your sole responsibility, not on behalf
      of any other Contributor, and only if You agree to indemnify,
      defend, and hold each Contributor harmless for any liability
      incurred by, or claims asserted against, such Contributor by reason
      of your accepting any such warranty or additional liability.

   END OF TERMS AND CONDITIONS

   APPENDIX: How to apply the Apache License to your work.

      To apply the Apache License to your work, attach the following
      boilerplate notice, with the fields enclosed by brackets "[]"
      replaced with your own identifying information. (Don't include
      the brackets!)  The text should be enclosed in the appropriate
      comment syntax for the file format. We also recommend that a
      file or class name and description of purpose be included on the
      same "printed page" as the copyright notice for easier
      identification within third-party archives.

   Copyright [yyyy] [name of copyright owner]

   Licensed under the Apache License, Version 2.0 (the "License");
   you may not use this file except in compliance with the License.
   You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

   Unless required by applicable law or agreed to in writing, software
   distributed under the License is distributed on an "AS IS" BASIS,
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   See the License for the specific language governing permissions and
   limitations under the License.

本次 HF 圖像歸屬與變更紀錄完整副本：

HF TOKENIZATION PIPELINE — ATTRIBUTION AND CHANGES

Hugging Face course authors; documentation-images dataset.
Source dataset revision: 96e71ad956904da670e6827274fc2c9b735b09d5
Source diagram: https://huggingface.co/datasets/huggingface-course/documentation-images/resolve/96e71ad956904da670e6827274fc2c9b735b09d5/en/chapter6/tokenization_pipeline.svg
Source chapter: https://huggingface.co/learn/llm-course/chapter6/4
License: Apache License 2.0. The full license is in Apache-2.0.txt.
Changes: rendered original SVG as a white-background PNG; mobile version retains the complete view then reuses original stage labels and before/arrow/after strips; repeated shared states and white spacing only. Cover fits the complete original on white1600x900; no crop or new artwork.
The fixed dataset tree contains no upstream NOTICE file; this is this publication's attribution/change record, not an invented upstream NOTICE or a claim of HF endorsement.