Claude Tokenizer
🔤
2
How Claude tokenizes text, next to GPT and open models
Peter Szemraj PRO
pszemraj
AI & ML interests
metallic intuition
Recent Activity
liked a Space 2 days ago
sanderland/claude-tokenizer liked a model 2 days ago
microsoft/Mage-ViT commentedon an article 4 days ago
Transformers Model Integration ContractsOrganizations
liked a
Space 2 days ago
liked a
model 2 days ago
commented on Transformers Model Integration Contracts 4 days ago
t5/umt5 in a blog post in 2026, never thought I’d see the day 😍
upvoted an article 4 days ago
Article
Transformers Model Integration Contracts
tarekziade
• • 16 feat: publish explicit CPU weight variants
1
#1 opened 11 days ago
by 
pszemraj
liked a
dataset 10 days ago
upvoted a paper 10 days ago
liked a
model 11 days ago
commented on FineBooks: are open OCR models good enough to unlock historical knowledge? 11 days ago
Good stuff. I'm working on a CPU-friendly implementation in Rust and C with a friend, btw https://github.com/amazingvince/falcon-ocr-cpu
There's currently an open PR that we're going to merge for multi-page stuff
commented on FineBooks: are open OCR models good enough to unlock historical knowledge? 12 days ago
This is the 1.0 I believe. Almost no one knows about 1.5 because they use the same repo 🤷
published a
bucket 14 days ago
updated a
bucket 15 days ago
upvoted a paper 17 days ago
updated a
bucket 20 days ago
published a
bucket 23 days ago
upvoted an article 24 days ago
Article
UltraBERT: One Encoder for Classification and Retrieval
dmis-lab
• • 67 upvoted a paper 24 days ago
liked 2
models 24 days ago
liked a
dataset 25 days ago