In [1]:

from transformers import AutoTokenizer

from transformers import AutoTokenizer

Encode dari text kedalam token menggunakan tokenisasi LLama3.2 v Tiktoken¶

More resource :

• https://en.wikipedia.org/wiki/Byte-pair_encoding
• https://github.com/openai/tiktoken
• https://github.com/google/sentencepiece/blob/master/python/README.md
• https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb
• https://github.com/huggingface/transformers/issues/31187
• https://huggingface.co/docs/transformers/en/model_doc/llama3

Hasil tokenisasi model Llama-3.2 sama persis dengan hasil tokenisasi GPT-3.5 dan GPT-4 (playground)

Kita akan menyandingkan antara hasil tokenizer transfomer dan library tiktoken dari OpenAI library, berikut adalah daftar Encoding Name

Tokenizer menggunakan transformer AutoTokenizerm¶

In [2]:

tonkenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-3B")

tonkenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-3B")

In [3]:

text = """The new policy will require all employees to attend a mandatory meeting on Friday at 2 PM. The team is expected to arrive early
and be prepared to discuss the upcoming project deadline. Meanwhile, the marketing department has been instructed to create a new
social media campaign by the end of the week. The sales team will also receive additional training on product features and
benefits. The CEO has announced plans to visit the office next month to meet with employees and discuss company growth.

In related news, the IT department is upgrading the server system to improve network security. This upgrade is expected to
enhance data storage capacity and reduce downtime for critical applications. Additionally, the HR department will be hosting a
team-building event next Saturday at 10 AM, where employees can participate in fun activities and bond with colleagues outside of
work.

The company's commitment to sustainability has also been highlighted, with plans to implement energy-efficient lighting and
reduce waste in all facilities. The new policy is designed to promote employee well-being, productivity, and company success."""
result_encode_llama = tonkenizer.encode(text)
print(result_encode_llama)

text = """The new policy will require all employees to attend a mandatory meeting on Friday at 2 PM. The team is expected to arrive early and be prepared to discuss the upcoming project deadline. Meanwhile, the marketing department has been instructed to create a new social media campaign by the end of the week. The sales team will also receive additional training on product features and benefits. The CEO has announced plans to visit the office next month to meet with employees and discuss company growth. In related news, the IT department is upgrading the server system to improve network security. This upgrade is expected to enhance data storage capacity and reduce downtime for critical applications. Additionally, the HR department will be hosting a team-building event next Saturday at 10 AM, where employees can participate in fun activities and bond with colleagues outside of work. The company's commitment to sustainability has also been highlighted, with plans to implement energy-efficient lighting and reduce waste in all facilities. The new policy is designed to promote employee well-being, productivity, and company success.""" result_encode_llama = tonkenizer.encode(text) print(result_encode_llama)

[128000, 791, 502, 4947, 690, 1397, 682, 8420, 311, 9604, 264, 23911, 6574, 389, 6740, 520, 220, 17, 5975, 13, 578, 2128, 374, 3685, 311, 17782, 4216, 198, 438, 387, 10235, 311, 4358, 279, 14827, 2447, 22143, 13, 26982, 11, 279, 8661, 9476, 706, 1027, 42075, 311, 1893, 264, 502, 198, 23191, 3772, 4901, 555, 279, 842, 315, 279, 2046, 13, 578, 6763, 2128, 690, 1101, 5371, 5217, 4967, 389, 2027, 4519, 323, 198, 68244, 1220, 13, 578, 12432, 706, 7376, 6787, 311, 4034, 279, 5274, 1828, 2305, 311, 3449, 449, 8420, 323, 4358, 2883, 6650, 382, 644, 5552, 3754, 11, 279, 8871, 9476, 374, 47035, 279, 3622, 1887, 311, 7417, 4009, 4868, 13, 1115, 14234, 374, 3685, 311, 198, 40967, 685, 828, 5942, 8824, 323, 8108, 75954, 369, 9200, 8522, 13, 23212, 11, 279, 23096, 9476, 690, 387, 20256, 264, 198, 9376, 52499, 1567, 1828, 7884, 520, 220, 605, 6912, 11, 1405, 8420, 649, 16136, 304, 2523, 7640, 323, 11049, 449, 18105, 4994, 315, 198, 1816, 382, 791, 2883, 596, 15507, 311, 41329, 706, 1101, 1027, 27463, 11, 449, 6787, 311, 4305, 4907, 73916, 18186, 323, 198, 27369, 12571, 304, 682, 13077, 13, 578, 502, 4947, 374, 6319, 311, 12192, 9548, 1664, 33851, 11, 26206, 11, 323, 2883, 2450, 13]

Tokenizer menggunakan Tiktoken¶

In [4]:

import tiktoken

import tiktoken

In [5]:

encoding = tiktoken.get_encoding('cl100k_base')
result_encode_tiktoken = encoding.encode(text)
print(result_encode_tiktoken)

encoding = tiktoken.get_encoding('cl100k_base') result_encode_tiktoken = encoding.encode(text) print(result_encode_tiktoken)

[791, 502, 4947, 690, 1397, 682, 8420, 311, 9604, 264, 23911, 6574, 389, 6740, 520, 220, 17, 5975, 13, 578, 2128, 374, 3685, 311, 17782, 4216, 198, 438, 387, 10235, 311, 4358, 279, 14827, 2447, 22143, 13, 26982, 11, 279, 8661, 9476, 706, 1027, 42075, 311, 1893, 264, 502, 198, 23191, 3772, 4901, 555, 279, 842, 315, 279, 2046, 13, 578, 6763, 2128, 690, 1101, 5371, 5217, 4967, 389, 2027, 4519, 323, 198, 68244, 1220, 13, 578, 12432, 706, 7376, 6787, 311, 4034, 279, 5274, 1828, 2305, 311, 3449, 449, 8420, 323, 4358, 2883, 6650, 382, 644, 5552, 3754, 11, 279, 8871, 9476, 374, 47035, 279, 3622, 1887, 311, 7417, 4009, 4868, 13, 1115, 14234, 374, 3685, 311, 198, 40967, 685, 828, 5942, 8824, 323, 8108, 75954, 369, 9200, 8522, 13, 23212, 11, 279, 23096, 9476, 690, 387, 20256, 264, 198, 9376, 52499, 1567, 1828, 7884, 520, 220, 605, 6912, 11, 1405, 8420, 649, 16136, 304, 2523, 7640, 323, 11049, 449, 18105, 4994, 315, 198, 1816, 382, 791, 2883, 596, 15507, 311, 41329, 706, 1101, 1027, 27463, 11, 449, 6787, 311, 4305, 4907, 73916, 18186, 323, 198, 27369, 12571, 304, 682, 13077, 13, 578, 502, 4947, 374, 6319, 311, 12192, 9548, 1664, 33851, 11, 26206, 11, 323, 2883, 2450, 13]

Apakah llama 3.2 menggunakan BPE tokeniser tiktoken ?¶

Tidak dijelaskan secara terang (berdasarkan yg saya cari dari internet) yang mengatakan Llama3.2 menggunakan tiktoken dengan encoding name cl100k_base yang sama juga digunakan oleh model GPT-3.5 dan GPT-4 ^1

Informasi yang saya baca dari mengatakan "LLaMA and similar Meta models use SentencePiece with Byte-Pair Encoding (BPE)." ^2

---

Footer

1. https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb
2. https://www.linkedin.com/pulse/transforming-ai-workflows-llama-32-simplified-text-chunking-karan-hwbqc/

In [6]:

result_encode_tiktoken == result_encode_llama[1:]

result_encode_tiktoken == result_encode_llama[1:]

Out[6]:

True

Saat saya mencari informasi terkait tokenisasi, saya menemukan Augustas Macijauskas, some random people out there yang memperhatikan hal yang sama, tokenisasi yang digunakan Llam3 dan GPT-4. Percobaan yang beliau lakukan membandingkan tokenisasi langsung dari modelnya, dan memeriksa kesamaan vocabulary dari setiap mode kesimpulannya yang dia dapat adalah ^3;

1. Tokenisasi HuggingFace GPT-4 mirip dengan tiktoken (Benar, OpenAI secara terbuka menyampaikan tokenizer yang digunakan)
2. Llam3 dan GPT-4 memiliki vocabulary yang mirip, dikatakan 100256 token pertama sejenis.

Kesimpulan yang bersangkutan tepat adanya, karena ketika disandingkan hasil tokenisasi tiktoken menggunakan encode cl100k_base dan tokenisasi dari pretrained model hasilnya sama persis (setidaknya menggunakan sample teks yang saya gunakan).

---

3. https://augustasmacijauskas.github.io/personal-website/notes/llama-3-vs-gpt-4-tokenizer/llama-3-vs-gpt-4-tokenizer.html

Trying another language¶

In [7]:

text = "请你基于以下「评估标准」"
result_tiktoken_encoding_two = encoding.encode(text)
result_llama_encoding_tokenizer_two = tonkenizer.encode(text)
print(result_tiktoken_encoding_two)
print(result_llama_encoding_tokenizer_two[1:])

text = "请你基于以下「评估标准」" result_tiktoken_encoding_two = encoding.encode(text) result_llama_encoding_tokenizer_two = tonkenizer.encode(text) print(result_tiktoken_encoding_two) print(result_llama_encoding_tokenizer_two[1:])

[15225, 57668, 75146, 35304, 88852, 13177, 64479, 17885, 108, 31944, 12870, 228, 10646]
[15225, 57668, 125510, 88852, 13177, 64479, 112494, 110778, 10646]

Saya juga menemukan diskusi yang serupa. Pada diskusi tersebut sang author menanyakan, mengapa hasil tokenisasi antara tiktoken dengan seri Llma3 berbeda. Sang author menyandingkan hasil encoding pada kalimat berikut (请你基于以下「评估标准」). Pertanyaan tersebut terjawab, dan jawab tersebut menjadi penerangan bagi saya yang saat ini sedang belajar mengenai tokenisasi. Dibelakang layar, tokenisasi pada model Llama3 menggunakan tokenizer yang berbeda. Walaupun, ketika saya mencoba menyandingkan hasil tokenisasi dari text inggris sepanjan 1138 karakter didapatkan hasil tokenisasi yang sama.^4

Sekarang yang menjadi pertanyaan adalah, tokenizer apa yang digunakan oleh Meta dalam mengembangkan Llama 3 series khususnya ? beberapa sumber mengatakan menggunakan SentencePiece, namun sejauh ini saya belum berhasil menggunakannya.

Wait, tepat sebelum saya ingin menutup laptop, saya kembali membaca dokumen model resmi dari Llama3, entah mengapa saya terlewat pada saat pencarian awal. Untuk tokensiasi, Llama3 menggunakan model Byte Per-Encoding pada tiktoken dengan dengan sentuhan yang berbeda. Dimana Llama3 alih-alih memecah satu kata menjadi beberapa token, tokenisasi tersebut akan menghitungnya menjadi satu token jika kata tersebut ada didalam vocabulary dari Llama. Mari kita coba

---

4. https://github.com/huggingface/transformers/issues/34403

In [8]:

text_final = "Istiklal"
tiktoken_result_final = encoding.encode(text_final)
llama_tokenizer_result_final = tonkenizer.encode(text_final)
print(tiktoken_result_final)
print(llama_tokenizer_result_final[1:])

print()
print("Tiktoken")
for a in tiktoken_result_final:
    print(f"token {a} : Decode {encoding.decode([a])}")
print()
for a in llama_tokenizer_result_final[1:]:
    print(f"token {a} : Decode {tonkenizer.decode([a])}")
print()

text_final = "Istiklal" tiktoken_result_final = encoding.encode(text_final) llama_tokenizer_result_final = tonkenizer.encode(text_final) print(tiktoken_result_final) print(llama_tokenizer_result_final[1:]) print() print("Tiktoken") for a in tiktoken_result_final: print(f"token {a} : Decode {encoding.decode([a])}") print() for a in llama_tokenizer_result_final[1:]: print(f"token {a} : Decode {tonkenizer.decode([a])}") print()

[40, 267, 1609, 75, 278]
[40, 267, 1609, 122591]

Tiktoken
token 40 : Decode I
token 267 : Decode st
token 1609 : Decode ik
token 75 : Decode l
token 278 : Decode al

token 40 : Decode I
token 267 : Decode st
token 1609 : Decode ik
token 122591 : Decode lal