Factual package intelligence from PyPI

tokscope 0.1.0 package intelligence

Measure how efficiently LLM tokenizers handle your language: fertility, UNK rate, and token premium across HuggingFace, tiktoken, and SentencePiece tokenizers

pip install tokscope

Current version
0.1.0
Python requirement
>=3.9
License
MIT License
Distribution type
Pure Python wheel
Release files
2 (1 wheels, 1 source)
Download size
23.1 KiB
Release history
1 releases with files
Median release cadence
Not enough dated releases

tokscope dependencies

PyPI declares 12 unique dependency rules for this release. Environment markers are shown when supplied by the project.

This release declares no required runtime dependencies in its PyPI metadata.

Compatibility and release files

tokscope publishes 1 wheel and 1 source archive for version 0.1.0. Wheel platform tags: any.

Declared Python classifiers: 3.10, 3.11, 3.12, 3.13, 3.9.

Security snapshot for tokscope 0.1.0

An OSV query completed on and found 0 known vulnerabilities affecting this version. 0 advisories are classified as critical. 0 advisories appear in the CISA Known Exploited Vulnerabilities catalog.

Release activity and sources

PyPI lists 1 release with files. The first dated release is ; 1 release falls within the 365 days preceding the latest dated release. The current release files were uploaded on .