[gorzell]

Summary

• Migrate the benchmark-only Hugging Face tokenizer dependency to tokenizers 1.0.0-rc.4, removing paste and number_prefix from the lockfile.
• Adapt the benchmark loader and encoding helpers to the read-only pipeline API.
• Add an offline unit test for removing the pre-tokenizer while preserving encoding.

Validation

• cargo check -p bpe-benchmarks
• cargo test -p bpe-benchmarks --lib
• cargo test -p bpe-benchmarks --test equivalence --no-run
• The Hugging Face equivalence tests and benchmarks need model downloads; this environment could not resolve huggingface.co, so please run the benchmarks locally.

[Copilot]

🟡 Changes recommended

The performance benchmarks still call the removed encode_fast API, preventing the benchmark target and CI’s all-target build from compiling.

2 open issues

• Benchmarks still call removed encode_fast API · New
• Offline regression test is skipped by CI · New

What changed in this PR

Updates the benchmark-only Hugging Face tokenizer dependency to remove unmaintained transitive dependencies, without changing the core BPE library.

Changes:

• Pins tokenizers to 1.0.0-rc.4, removing paste and number_prefix from the lockfile.
• Adapts tokenizer loading and shared encoding helpers to the read-only pipeline API.
• Adds an offline regression test for pre-tokenizer removal.

File	Description
crates/​bpe/​benchmarks/​performance.rs	Migrates an encoding benchmark to the shared helper.
crates/​bpe/​benchmarks/​lib.rs	Adds pipeline loading, encoding helpers, and an offline test.
crates/​bpe/​benchmarks/​equivalence.rs	Uses the shared helper for equivalence checks.
crates/​bpe/​benchmarks/​Cargo.toml	Updates tokenizer dependencies and feature selection.
Cargo.lock	Records the updated dependency graph.

_{Review effort: Balanced (auto)}