Factual package intelligence from PyPI

tokeniser-py-lite 0.1.1 package intelligence

A custom tokeniser with a 131,072-token vocabulary derived from 0.5B (val) and 1B (val+test) tokens in SlimPajama. Lite version of the tokeniser-py library. Uses a novel token generation algorithm and a dynamic programming-based segmentation method for fast, interpretable tokenis

pip install tokeniser-py-lite

Current version
0.1.1
Python requirement
Not declared
License
MIT
Distribution type
Pure Python wheel
Release files
2 (1 wheels, 1 source)
Download size
9.4 MiB
Release history
2 releases with files
Median release cadence
0 days

tokeniser-py-lite dependencies

PyPI declares 3 unique dependency rules for this release. Environment markers are shown when supplied by the project.

  1. numpy
  2. regex
  3. tqdm

Compatibility and release files

tokeniser-py-lite publishes 1 wheel and 1 source archive for version 0.1.1. Wheel platform tags: any.

No version-specific Python classifiers are declared.

Metadata completeness

PyPI does not currently declare: Python requirement. PyDeps marks these fields as unknown instead of guessing values.

Release activity and sources

PyPI lists 2 releases with files. The first dated release is ; 2 releases fall within the 365 days preceding the latest dated release. The current release files were uploaded on . The preceding dated release was .