LowRankArena: A Standardized Evaluation Platform for SVD-Based LLM Compression

LowRankArena is a standardized evaluation platform and checkpoint zoo for low-rank, singular value decomposition (SVD)-based compression of large language models. It is designed to make compression methods comparable under shared model, task, compression-budget, and serving configurations.

This is a multi-model checkpoint repository, not a single directly loadable model. Select a model, method, and keep ratio from the directory tree or the machine-readable checkpoint catalog.

Paper · Benchmark code · Checkpoint catalog · Evaluation results · Audits & reproducibility index

LowRankArena standardizes artifacts, evaluation, and serving measurements across representative reproducible SVD-based compression methods. The current checkpoint zoo contains more than 3 TiB of artifacts.

Why LowRankArena?

Comparing low-rank LLM compression methods is difficult when published results use different model families, task versions, compression-ratio definitions, numerical precisions, retraining recipes, and serving stacks. LowRankArena controls these variables so that a reported difference is attributable as closely as possible to the compression method being evaluated.

The platform currently provides:

• A checkpoint zoo: 313 derived checkpoints and 11 reconstruction artifacts across eight model families in the current catalog.
• Representative SVD-based methods: ASVD, SVD-LLM, DoBi-SVD, Basis Sharing, and MoDeGPT, including method variants where applicable.
• Shared compression budgets: uniform-precision parameter keep ratios, with the principal paper comparisons at 0.8, 0.6, and 0.4.
• Matched quality evaluation: language-modeling perplexity, seven zero-shot multiple-choice tasks, and capability-specific mathematical and instruction-following tasks.
• Matched serving evaluation: time to first token (TTFT), inter-token latency (ITL), end-to-end latency, and prompt/output throughput under identical serving configurations.
• Structured-pruning references: standardized pruning artifacts and results for matched-budget comparison.

Standardized compression budget

For a dense matrix (W_l \in \mathbb{R}^{m_l \times n_l}) replaced by low-rank factors of rank (k_l), LowRankArena defines the global parameter keep ratio as

$$r=\frac{\sum _l{k}_l(m_l+n_l)}{\sum _l{m}_l{n}_l},\mathrm{prec}({\widehat{W}}_l)=\mathrm{prec}(W_l)\mathrm{.}r\; =\; \backslash frac\{\backslash sum\_l\; k\_l(m\_l+n\_l)\}\{\backslash sum\_l\; m\_l\; n\_l\},\; \backslash qquad\; \backslash operatorname\{prec\}(\backslash widehat\{W\}\_l)=\backslash operatorname\{prec\}(W\_l).$$

The equal-precision constraint separates low-rank subspace selection from bit-width reduction, quantization, and hidden dense fallbacks. Comparisons are additionally separated by recovery regime; results without retraining are not mixed with LoRA-recovered or otherwise healed checkpoints.

Main findings

1. Method rankings are architecture- and budget-dependent

The relative ordering of methods changes across backbones, and it can change again as the compression budget tightens. There is no stable architecture-invariant winner in the evaluated public, reproducible subset.

The compact leaderboard below reports Q Ret., the arithmetic mean of dense-normalized retention on WikiText-2 perplexity, C4 perplexity, multiple-choice average, MathQA, and MMLU-Math. Higher is better; values should be interpreted only within the shared LowRankArena protocol.

Method	LLaMA-1-7B 0.8	Llama-3.1-8B 0.8	Qwen3-8B-Base 0.8	LLaMA-1-7B 0.6	Llama-3.1-8B 0.6	Qwen3-8B-Base 0.6
ASVD	0.787	0.304	0.696	0.458	0.326	0.252
SVD-LLM v1	0.767	0.520	0.584	0.615	0.329	0.380
DoBi-SVD	0.732	0.341	0.299	0.560	0.325	0.284
Basis Sharing	0.747	0.531	0.585	0.622	0.330	0.409
MoDeGPT	0.880	0.766	0.594	0.690	0.446	0.400

2. Multiple-choice accuracy can hide a capability cliff

In the matched no-retraining comparison below, multiple-choice accuracy can remain near its floor while C4 perplexity has already degraded sharply. Perplexity is therefore an important sensitivity check rather than an interchangeable substitute for downstream accuracy.

This result should not be read as a universal statement about post-training recovery: LoRA healing or method-specific retraining can materially alter the comparison.

3. Nominal low-rank savings do not guarantee end-to-end speedup

The serving results show strong workload dependence. Low-rank execution can improve compute-bound prefill and TTFT—up to 4.20× in the evaluated configuration—while decode-heavy end-to-end latency and throughput often improve only slightly or regress. Hybrid designs that retain dense operators can behave differently from fully factorized implementations.

Checkpoint zoo

All released checkpoint payloads live under one namespace:

Collection	Repository path
Low-rank checkpoints	checkpoints/low_rank/
Structured-pruning checkpoints	checkpoints/pruning/

The shared layout is checkpoints/collection/model/method/variant_ratio. Checkpoint-level artifact_metadata.json and compatibility.json record provenance, structural loading information, and backend validation scope.

Load any self-contained checkpoint (recommended)

Snapshot one checkpoint directory, then load that local directory. This is the universal path for custom-code checkpoints in this multi-model repository and does not clone unrelated model weights.

from pathlib import Path

from huggingface_hub import snapshot_download
from transformers import AutoModelForCausalLM

repo_id = "Duke-CEI-SVD/LowRankArena"
checkpoint = "checkpoints/low_rank/llama31_8b/basis_sharing/default_0.6"
snapshot = Path(
    snapshot_download(
        repo_id=repo_id,
        revision="main",  # Pin a release tag or commit for exact reproduction.
        allow_patterns=[f"{checkpoint}/**"],
    )
)
model = AutoModelForCausalLM.from_pretrained(
    snapshot / checkpoint,
    trust_remote_code=True,
    local_files_only=True,
    device_map="auto",
)

Direct Basis Sharing loading

The self-contained Llama-3.1 and Qwen3 Basis Sharing exports also support the shorter native repo_id plus subfolder form:

from transformers import AutoModelForCausalLM

checkpoint = "checkpoints/low_rank/llama31_8b/basis_sharing/default_0.6"
model = AutoModelForCausalLM.from_pretrained(
    "Duke-CEI-SVD/LowRankArena",
    subfolder=checkpoint,
    revision="main",  # Pin a release tag or commit for exact reproduction.
    trust_remote_code=True,
    device_map="auto",
)

This direct form applies to the AutoModel-compatible Basis Sharing exports under llama31_70b, llama31_8b, llama31_8b_instruct, qwen3_8b, and qwen3_8b_base. Older factorized or reconstruction-only artifacts retain their artifact-specific workflows.

Some upstream base models are gated. Accept their terms and authenticate before downloading derived checkpoints. Inspect included code and pin revision before enabling trust_remote_code=True.

Reproducing the benchmark

Evaluation and serving code is maintained in the LowRankArena GitHub repository. Use the code revision, environment, task version, prompt format, recovery regime, and serving workload recorded with each result when reproducing a number. The results/ directory contains repository-side evaluation artifacts.

Intended use

LowRankArena is intended for research on low-rank LLM compression, controlled comparison of compression methods, reproduction of the accompanying paper, and evaluation of quality/serving trade-offs. It is not an endorsement of a single method or a claim to cover every low-rank compression technique.

Limitations

• The benchmark covers a representative publicly reproducible subset of SVD-based methods, not the entire literature.
• Conclusions are conditional on the evaluated models, keep ratios, tasks, recovery regimes, precision, hardware, kernels, and serving workloads.
• Multiple-choice aggregate scores can saturate near chance and conceal severe language-modeling degradation.
• Nominal parameter or FLOP reductions do not directly predict memory traffic, kernel efficiency, or user-visible serving latency.
• Derived checkpoints retain the limitations and potential risks of their upstream base models and training data.

Licenses and use terms

This Hugging Face repository is labeled other because it is a mixed-license artifact collection rather than a single MIT-licensed model:

• LowRankArena-authored benchmark code is released under the MIT License.
• The Croissant catalog metadata is licensed under CC BY 4.0; that catalog license does not apply to referenced model weights.
• Each derived checkpoint retains the applicable upstream base-model license, acceptable-use policy, access conditions, and any method-specific terms.

The repository label does not override those terms. Consult the license matrix, checkpoint metadata, and the relevant upstream model card before use or redistribution.

Citation

If you use LowRankArena, please cite the arXiv paper:

@misc{shao2026lowrankarenastandardizedevaluationplatform,
  title         = {LowRankArena: A Standardized Evaluation Platform for SVD-Based LLM Compression},
  author        = {Zishan Shao and Lixun Zhang and Kangning Cui and Wenhao Wu and Jinhee Kim and Yixiao Wang and Ting Jiang and Hancheng Ye and Qinsi Wang and Fan Yang and Danyang Zhuo and Yiran Chen and Hai Li},
  year          = {2026},
  eprint        = {2608.26389},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2608.26389}
}

Questions and corrections may be sent to zishan.shao@duke.edu.