tokenizers.bpe

Byte Pair Encoding Text Tokenization

CRAN Package

Unsupervised text tokenizer focused on computational efficiency. Wraps the 'YouTokenToMe' library https://github.com/VKCOM/YouTokenToMe which is an implementation of fast Byte Pair Encoding https://aclanthology.org/P16-1162/.

  • Version0.1.3
  • R versionunknown
  • LicenseMPL-2.0
  • Needs compilation?Yes
  • Last release09/15/2023

Documentation

Team

Insights

Last 30 days

The following line graph shows the downloads per day. You can hover over the graph to see the exact number of downloads per day.

Last 365 days

The following line graph shows the downloads per day. You can hover over the graph to see the exact number of downloads per day.

Data provided by CRAN

Binaries

Dependencies

  • Imports1 package
  • Linking To1 package
  • Reverse Suggests3 packages