Tokenizer

Public

Fast and customizable text tokenization library with BPE and SentencePiece support

MIT License

Updated Feb 2, 2026

Created Feb 14, 2017

329 stars

79 forks

18 watchers

8 open issues

Languages

Codebase composition by bytes

Top Contributors

550 commits

25 commits

9 commits

8 commits

5 commits

5 commits

2 commits

2 commits

1 commits

1 commits

Recent Stargazers

almost 9 years ago

almost 9 years ago

over 8 years ago

over 8 years ago

over 8 years ago

over 8 years ago

over 8 years ago

DA davidalbertonogueira

over 8 years ago

TO tokestermw

almost 8 years ago

SP Spacebody

over 7 years ago

Repository Complexity

Approximate file count and repository size.

Files-

Size (KB)1,813

Issues & Activity

Open vs closed issues and recent activity peak.

Open Issues 7Closed Issues 93

Open Issues100

Activity Peak

Most recent day with highest activityFeb 2, 2026 • 1 events

Stars & Forks Over Time

Cumulative stars and forks based on recent history.

GitHub Analytics

Tokenizer