AI & ML interests
None yet
Recent Activity
repliedto savoji's post about 22 hours ago SO-Arena scored 21 System One models, and the most accurate one ranks only seventh on reliability.
System One models pick from fixed options in one forward pass, so teams use them for routing, tool choice and verification. A leaderboard score does not say whether the pick survives shuffled options or a missing right answer. SO-Arena (2,919 tasks, 11 domains) measures that separately.
š What's new
ā¢ Reliability R = sqrt(S*N). S is stability over 12 option reorderings. N = 2AUC-1 on spotting that the correct option was removed
ā¢ JevAny-27B is the most reliable System One model yet ranks eleventh in capability
ā¢ 14 of 18 models reach S above 0.8, but N runs from near chance to about 0.8
š The numbers
ā¢ GPT-6 Sol (82.3) and Luna (79.7) lead. The best System One models, GLiDE and KnowLine-31B, score 70.2
ā¢ Open-weight System One models answer within 0.3 s per task on one H100
ā¢ Knowledge, sentiment, retrieval: comparable to Luna, with little gain from a bigger backbone or thinking
ā¢ Math: thinking lifts StartLux Auto from 50.8 to 92.5. In intent, safety, tools and web, the larger backbone helps far more
ā ļø Caveats
ā¢ The backbone gain mixes size with Qwen3.5 vs Qwen3.8. Thinking is tested on one model
ā¢ Two closed System Two references only, and no independent replication yet
Before adopting a model, check per-domain scores, then reorder options and drop the correct one on your own task. The MIT repo ships the scoring and reliability tooling.
https://huggingface.co/papers/2611.15777
https://github.com/holdrain/SO-Arena
Do you test decision models for order sensitivity before putting them in an agent?
š Read it elsewhere
ā¢ The deep dive on Medium: https://medium.com/p/e923d2f58e85
ā¢ The long read on Substack: https://sajjadpakdamansavoji.substack.com/p/so-arena-finds-the-most-accurate
ā¢ The thread on X: https://x.com/saavoji/status/2121665100981444943
ā¢ Every version on sajjadpsavoji.github.io: https://sajjadpsavoji.github.io/blog#so-arena-finds repliedto mlboydaisuke's post 5 days ago Turn one photo into a 3D object you can spin around, sides and back included, entirely on your phone.
On a Galaxy S26, Idol (speridlabs) on LiteRT now makes a splat of 65,536 Gaussians in 158 s with its new default form, or of 262,144 Gaussians in 267 s with the first release's form, which stays in the repo (the first photo after start, on a cool phone). Checked against the author's PyTorch pipeline, renders from four sides agree at a PSNR of 32.3ā34.4 dB for the default and 41.4ā46.6 dB for the larger form.
Graphs, a Python runner and an Android sample: https://huggingface.co/litert-community/Idol Organizations
None yet
