Factual package intelligence from PyPI

bpetokenizer 1.2.1 package intelligence

A Byte Pair Encoding (BPE) tokenizer, which algorithmically follows along the GPT tokenizer(tiktoken), allows you to train your own tokenizer. The tokenizer is capable of handling special tokens and uses a customizable regex pattern for tokenization(includes the gpt4 regex patter

pip install bpetokenizer

Current version
1.2.1
Python requirement
<3.13,>=3.9
License
MIT
Distribution type
Pure Python wheel
Release files
2 (1 wheels, 1 source)
Download size
486.4 KiB
Release history
9 releases with files
Median release cadence
0 days

bpetokenizer dependencies

PyPI declares 3 unique dependency rules for this release. Environment markers are shown when supplied by the project.

  1. regex

Compatibility and release files

bpetokenizer publishes 1 wheel and 1 source archive for version 1.2.1. Wheel platform tags: any.

No version-specific Python classifiers are declared.

Release activity and sources

PyPI lists 9 releases with files. The first dated release is ; 9 releases fall within the 365 days preceding the latest dated release. The current release files were uploaded on . The preceding dated release was .