Installation
About
Interfaces with the 'Hugging Face' tokenizers library to provide implementations of today's most used tokenizers such as the 'Byte-Pair Encoding' algorithm
| github.com/mlverse/tok | |
| System requirements | Rust tool chain w/ cargo, libclang/llvm-config |
| Bug report | File report |
Key Metrics
Downloads
| Yesterday | 14 0% |
| Last 7 days | 60 -17% |
| Last 30 days | 197 +40% |
| Last 90 days | 426 -21% |
| Last 365 days | 1.646 |