Search results

Precompiled normalizer diverges from SentencePiece: shortest-prefix match + discarded remainder silently drops codepoints

Status: Open.

#2334 In huggingface/tokenizers;

· imhmhm opened on Aug 14, 2026

BpeTrainer emits a merge for a pair that occurs zero times (negative pair_counts sign-extended by as u64)

Status: Open.

#2320 In huggingface/tokenizers;

· Abraham-y opened on Aug 7, 2026

[BUG] | The docs listing page for tokenizers is throwing 404

Status: Open.

#2236 In huggingface/tokenizers;

· JayaKrishnaNamburu opened on Jul 23, 2026

[Feature/Security] Support trusted and untrusted special-token spans

Status: Open.

#2225 In huggingface/tokenizers;

· kexinoh opened on Jul 21, 2026

EncodingVisualizer: data-* attributes built from token text are not HTML-escaped (XSS gap left by #1937)

Status: Open.

#2210 In huggingface/tokenizers;

· ErenAta16 opened on Jul 17, 2026

BPE build() panics on load for a crafted tokenizer.json (merge concatenation exceeds the scratch buffer)

Status: Open.

#2198 In huggingface/tokenizers;

· professor-moody opened on Jul 14, 2026

Precompiled normalizer uses grapheme-level shortest-prefix matching and diverges from SentencePiece on NFD input

Status: Open.

#2116 In huggingface/tokenizers;

· Incheonkirin opened on Jun 17, 2026

Proposal: emit UserWarning from Split.pre_tokenize_str() when it produces zero matches (catches silent str vs Regex confusion)

Status: Open.

#2109 In huggingface/tokenizers;

· vamsin07 opened on Jun 16, 2026

[Security] Load-time process abort when loading a malicious tokenizer.json (BPE merge buffer overrun)

Status: Open.

#2094 In huggingface/tokenizers;

· geo-chen opened on Jun 10, 2026

Release 0.23.1 introduces breaking change in Encoding.overflowing

Status: Open.

#2091 In huggingface/tokenizers;

· alistairewj opened on Jun 8, 2026

Two incorrect type annotations affect input typing to tokenizers.Tokenizer.encode()

Status: Open.

#2088 In huggingface/tokenizers;

· JEHoctor opened on Jun 7, 2026

Reproducible WordPiece encode bench shows ~5x speedup on 123B fixture — methodology question for maintainers

Status: Open.

#2075 In huggingface/tokenizers;

· SMC17 opened on May 28, 2026