FazBrowse GitHub Viewer | Trending | | Home

Tools: [Download Repo ZIP] [Original HTTPS Page]

Commits · wangzhaode/tokenizer.cpp · GitHub

Commits

Commits on Jan 4, 2026

feat(test): add hy-mt.
wangzhaode
committed
8638367

Commits on Dec 27, 2025

perf(core): optimize BPE lookup/byte_map and ensure thread safety
wangzhaode
committed
a9e88dd
chore(ci): remove arm64 runner from main branch
wangzhaode
committed
37ecc22
feat(test): add test_simple
wangzhaode
committed
2cf05f2
chore(ci): trigger workflow on all pull requests
wangzhaode
committed
fed88d7
chore(tests): parallelize test asset generation for faster CI
wangzhaode
committed
edf97d6
Merge pull request #1 from mpj1234/main
wangzhaode
authored
3b07a26

Commits on Dec 26, 2025

Commits on Dec 23, 2025

feat(json): switch JSON backend to ujson with RapidJSON support
wangzhaode
committed
4fcd3d8
feat(test): add execution time statistics for test cases
wangzhaode
committed
1619cc2
update(jinja): update jinja to new.
wangzhaode
committed
192b86c
feat(tokenizer): fix decoding and normalization for BERT, BGE, GTE, and Albert
wangzhaode
committed
8dd9c6d
update(docs): using C++11.
wangzhaode
committed
76136b8
feat(docs): add README, design document and CI workflow
wangzhaode
committed
2947801
feat(core): initialize tokenizer.cpp with BPE support
wangzhaode
committed
bbfdf5d

Back | FazBrowse Home | New Git URL