Post
351
tokenizers 0.23.3 is out!https://github.com/huggingface/tokenizers/releases/tag/v0.23.3
The release is python-only and fixes a few very annoying issues
🏗️ Building on HF
Simon Brandeis
sbrandeis
AI & ML interests
None yet
Recent Activity
liked a model about 9 hours ago
moonshotai/Kimi-K3.1-Pro liked a model about 9 hours ago
ccore/Qwen3.5-4B-3D-Coder liked a model about 9 hours ago
Pika-Labs/Pika-H3-FlexOrganizations
Posts 2
https://github.com/huggingface/tokenizers/releases/tag/v0.23.3
The release is python-only and fixes a few very annoying issues
Papers I want to read, at some point.
-
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
Paper • 2108.12409 • Published • 5 -
YaRN: Efficient Context Window Extension of Large Language Models
Paper • 2309.00071 • Published • 84 -
MIMIC-IT: Multi-Modal In-Context Instruction Tuning
Paper • 2306.05425 • Published • 11 -
Music ControlNet: Multiple Time-varying Controls for Music Generation
Paper • 2311.07069 • Published • 44
Papers I want to read, at some point. Focused on Reinforcement Learning papers.
-
Deep reinforcement learning from human preferences
Paper • 1706.03741 • Published • 4 -
Training language models to follow instructions with human feedback
Paper • 2203.02155 • Published • 26 -
Direct Preference-based Policy Optimization without Reward Modeling
Paper • 2301.12842 • Published -
Woodpecker: Hallucination Correction for Multimodal Large Language Models
Paper • 2310.16045 • Published • 18
Papers I want to read, at some point.
-
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
Paper • 2108.12409 • Published • 5 -
YaRN: Efficient Context Window Extension of Large Language Models
Paper • 2309.00071 • Published • 84 -
MIMIC-IT: Multi-Modal In-Context Instruction Tuning
Paper • 2306.05425 • Published • 11 -
Music ControlNet: Multiple Time-varying Controls for Music Generation
Paper • 2311.07069 • Published • 44
Papers I want to read, at some point. Focused on Reinforcement Learning papers.
-
Deep reinforcement learning from human preferences
Paper • 1706.03741 • Published • 4 -
Training language models to follow instructions with human feedback
Paper • 2203.02155 • Published • 26 -
Direct Preference-based Policy Optimization without Reward Modeling
Paper • 2301.12842 • Published -
Woodpecker: Hallucination Correction for Multimodal Large Language Models
Paper • 2310.16045 • Published • 18
spaces 4
Running
tokenizers v1 compatibility
🧩
Compare tokenizers v1 compatibility with previous releases
Running
Tokenmaxxing BERT
🏎
Step through huggingface tokenizers normalizers
Running
8
Tokenizers in your browser
🚀
tokenizers 1.0.0-rc.2 performance overview
Running
Quickdraw
🐢
Play a drawing game where AI guesses your sketches