课程介绍
CS336:从零开始构建属于自己的大模型。
本课程学习过程参考以下资料:
- B站课程链接:https://www.bilibili.com/video/BV1msTD6CE6j/?vd_source=66414b75f6db8197e95def95e244d230&p=1
- Github学习框架:https://github.com/bcefghj/learn-cs336/blob/main/docs/02-BPE%E5%88%86%E8%AF%8D%E5%99%A8%E5%8E%9F%E7%90%86%E4%B8%8E%E5%AE%9E%E7%8E%B0.md
- Stanford原始网站: https://cs336.stanford.edu/
Lesson 01
Overview & Tokenization
The Bitter Lesson
Wrong interpretation: scale is all that matters, algorithms don’t matter.
Right interpretation: algorithms that scale is what matters.
accracy=efficiency * resources
Tokenization
Tokenizer
把string转化为tokens序列
下面就是一个string2tokens的例子,其中tokens包含的是每个token的indices。
string = "Hello, 🌍! 你好!"
tokens = [15496, 11, 995, 0]
与tokens对应的是Vocabulary(词表),代表了所有有可能出现的tokens,indices就是token在Vocabulary里面的序号。
string->tokens称为encode,反之tokens->string则称为decode
从string到tokens,整个序列的元素数量减少了,我们把string的字节数与tokens数的比例称为Compression_ratio,用来衡量Tokenizer的性能
Three types of tokenizer
Character-based
Each Unicode Character can be converted into a code point (integer) via ord, and it can be converted back via chr
assert ord("a") == 97
assert ord("🌍") == 127757
assert chr(97) == "a"
assert chr(127757) == "🌍"
But there are approximately 150k unicode characters,
Vocabulary_size = max(indices) + 1
- So the vocabulary_size is too large
- And many chrs are too rare, which is inefficient use of the vocabulary.
Bytes-based
Unicode strings can be represented as a sequence of bytes, which can be represented by integers between 0 and 255.
assert bytes("a", encoding="utf-8") == b"a"
assert bytes("🌍", encoding="utf-8") == b"\xf0\x9f\x8c\x8d"
Vocabulary_size = 256, but Compression_ratio = 1, which means the sequences will be too long
The context of transformer is limited, so it is not good.
Words-based
First, split strings to words. Regular Expression keeps all alphanumeric characters together.
string = "I'll say supercalifragilisticexpialidocious!"
segments = regex.findall(r"\w+|.", string) # @inspect segments
Then turn it into tokenizer. We can build a mapping from each segment into an integer.
Problems:
- number of words is huge (like for Unicode characters).
- Many words are rare and the model won’t learn much about them.
- This doesn’t obviously provide a fixed vocabulary size.
New words we haven’t seen during training get a special UNK token, which is ugly and can mess up perplexity calculations.
BPE
Basic idea: train the tokenizer on raw text to automatically determine the vocabulary.
让Tokenizer自动决定如何构建vocabulary
Intuition: common sequences of characters are represented by a single token, rare sequences are represented by many tokens.
首先,我们计算每对相邻的token对(初始每个token都是一个字节)的出现次数,然后把出现次数最多的token对合并,作为vocab[++max_indices]的token。再重新统计,合并下一个token对,知道合并次数满足要求或者到达Vocabulary容量上限为止。