跳转到正文

技术笔记

We will know. We must konw.

列表阅读
技术 / 全部笔记

从零开始,搭建一个属于自己的博客

这是一篇建站示例文章,用于展示中文排版与代码样式,可直接修改或删除。

为什么是一个静态博客

个人博客的核心是内容。把文章保存为 Markdown,再生成静态页面,写作过程简单,迁移和备份也方便。

这里选择 AstroPaper 作为基础,保留阅读、搜索、标签归档和 RSS,把视觉调整为暖灰纸张底色、衬线字体和居中的文章标题。

在本地运行

安装依赖后,启动开发服务器:

npm run dev

构建可以部署的静态文件:

npm run build
npm run preview

构建结果在 dist/ 目录。预览构建结果时,静态搜索也能正常工作。

做减法

中文正文使用本地宋体或衬线字体,代码使用等宽字体。页面不需要大幅动画,也不需要持续闪烁的光标。

让内容容易被读到,让写作容易坚持下去。

Stanford CS336

课程介绍

CS336:从零开始构建属于自己的大模型。

本课程学习过程参考以下资料:

  1. B站课程链接:https://www.bilibili.com/video/BV1msTD6CE6j/?vd_source=66414b75f6db8197e95def95e244d230&p=1
  2. Github学习框架:https://github.com/bcefghj/learn-cs336/blob/main/docs/02-BPE%E5%88%86%E8%AF%8D%E5%99%A8%E5%8E%9F%E7%90%86%E4%B8%8E%E5%AE%9E%E7%8E%B0.md
  3. Stanford原始网站: https://cs336.stanford.edu/

Lesson 01

Overview & Tokenization

The Bitter Lesson

Wrong interpretation: scale is all that matters, algorithms don’t matter.

Right interpretation: algorithms that scale is what matters.

accracy=efficiency * resources

Tokenization

Tokenizer

把string转化为tokens序列 下面就是一个string2tokens的例子,其中tokens包含的是每个token的indices。

string = "Hello, 🌍! 你好!"
tokens = [15496, 11, 995, 0]

与tokens对应的是Vocabulary(词表),代表了所有有可能出现的tokens,indices就是token在Vocabulary里面的序号。

string->tokens称为encode,反之tokens->string则称为decode

从string到tokens,整个序列的元素数量减少了,我们把string的字节数与tokens数的比例称为Compression_ratio,用来衡量Tokenizer的性能

Three types of tokenizer

Character-based

Each Unicode Character can be converted into a code point (integer) via ord, and it can be converted back via chr

assert ord("a") == 97
assert ord("🌍") == 127757
assert chr(97) == "a"
assert chr(127757) == "🌍"

But there are approximately 150k unicode characters,

Vocabulary_size = max(indices) + 1
  • So the vocabulary_size is too large
  • And many chrs are too rare, which is inefficient use of the vocabulary.

Bytes-based

Unicode strings can be represented as a sequence of bytes, which can be represented by integers between 0 and 255.

assert bytes("a", encoding="utf-8") == b"a"
assert bytes("🌍", encoding="utf-8") == b"\xf0\x9f\x8c\x8d"

Vocabulary_size = 256, but Compression_ratio = 1, which means the sequences will be too long The context of transformer is limited, so it is not good.

Words-based

First, split strings to words. Regular Expression keeps all alphanumeric characters together.

string = "I'll say supercalifragilisticexpialidocious!"
segments = regex.findall(r"\w+|.", string)  # @inspect segments

Then turn it into tokenizer. We can build a mapping from each segment into an integer.

Problems:

  • number of words is huge (like for Unicode characters).
  • Many words are rare and the model won’t learn much about them.
  • This doesn’t obviously provide a fixed vocabulary size.

New words we haven’t seen during training get a special UNK token, which is ugly and can mess up perplexity calculations.

BPE

Basic idea: train the tokenizer on raw text to automatically determine the vocabulary.

让Tokenizer自动决定如何构建vocabulary

Intuition: common sequences of characters are represented by a single token, rare sequences are represented by many tokens.

首先,我们计算每对相邻的token对(初始每个token都是一个字节)的出现次数,然后把出现次数最多的token对合并,作为vocab[++max_indices]的token。再重新统计,合并下一个token对,知道合并次数满足要求或者到达Vocabulary容量上限为止。