Papers
arxiv:2606.10375

SIDInspector: A Mapping-First Diagnostic Resource for Semantic-ID Tokenizers

Published on Aug 11
Authors:
,
,

Abstract

SIDInspector provides a diagnostic framework to evaluate semantic identifier tokenizer mappings for generative recommendation by measuring aliasing, prefix alignment, and structural properties independently of downstream model performance.

Semantic-ID (SID) tokenizers are increasingly reused as standalone artifacts in generative recommendation: an exported item-to-code mapping becomes the address space that a later sequence generator must use. These mappings rarely come with a common inspection interface, so coverage gaps, full-code aliasing, behaviorally weak prefixes, tail compression, and prefix fan-out are often found only after downstream training. We present SIDInspector, a mapping-first diagnostic resource for SID tokenizer artifacts. SIDInspector defines a small adapter contract over item mappings, metadata, interactions, and optional generator traces; validates the contract; and reports mapping-level probes for utilization, aliasing, neighborhood alignment, popularity allocation, and structural cost, with hooks for temporal churn and generator traces. SIDInspector characterizes exported mappings independently of downstream leaderboard scores. The release covers four tokenizer lines: GRID-style RQ-KMeans under two capacity settings and a bounded ReSID/GAOQ export in a same-item Musical protocol, plus released LETTER and LC-Rec item-index artifacts. In that protocol, D2 aliasing falls from 0.977 to 0.779 under added GRID-style capacity and to 0.000 for ReSID/GAOQ, while D3 remains below a deterministic category-prefix control (0.055-0.154 versus 0.447). SIDInspector exposes capacity, addressability, and behavioral prefix alignment as distinct artifact properties before method-level evaluation. Additional domain checks show the same diagnostic distinction. Bounded fixed-reranker probes connect prefix alignment to candidate exposure, while final ranking quality remains a downstream model question.

Community

Sign up or log in to comment

Get this paper in your agent:

hf papers read 2606.10375
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2606.10375 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2606.10375 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2606.10375 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.