AX-Ray Leaderboard
π©»
40
AI/AX Safety Diagnostics and Assessment
π In a Training Loop
VIDRAFT_LAB
SeaWolf-AI
AI & ML interests
Contact: arxivgpt@gmail.com
Recent Activity
updated a Space about 4 hours ago
FINAL-Bench/AX-RAY updated a Space about 8 hours ago
FINAL-Bench/gate-tetris repliedto their post about 14 hours ago
The cost of a judging gate is usually quoted as a number. This puts it on a Tetris board.
Three boards get the same piece order, and on every move the same proposal and the same noise β a paired comparison. The gate decides one thing: keep this move, or draw again. Each board gets the same 60 seconds of gate time.
The text-writing gates get through 15β22 moves. The generation-free gate gets through 40β50. The boards that stop simply run out of clock.
It does not win on accuracy: on the same 2,018-question LODO set, JEV scores AUC 0.7350 against ZTC-Judge-27B's 0.7289. The separation is elsewhere. Clock β 2.1 s vs 0.0615 s per call, and on a 200-candidate agent screen one judging call measured 3.206 s generative vs 0.033 s readout, same server. Calibration β a gate is a threshold, and at ECE 0.4985 (vs ZTC 0.0245) a threshold stops carrying information. Mechanism β a text judge can name option 42 when there is no option 42; a scoring readout cannot. Not a lower error rate. No path.
The curve in the ZTC panel is real online fitting, scored prequentially β predict first, learn after β with base weights untouched. Not recursive self-improvement.
Limits, also stated on the page: Laya's AUC and latency are not our measurements and are set equal to JEV's, so calibration is the only measured axis it differs on. The page is a simulation driven by measured constants.
KO / EN / ZH.
https://huggingface.co/spaces/FINAL-Bench/Tetris-JEV-LAYA-ZTC
https://huggingface.co/FINAL-Bench/ZTC-Judge-27BOrganizations
updated a
Space about 4 hours ago
updated a
Space about 8 hours ago
replied to their post about 14 hours ago
Sharp read, thanks for going through the source.
Both points are in. All three gates now set their threshold by the same rule, so the board isolates judging quality and the clock. Online learning is separated from the judging. You're right that the deployable label is a delayed outcome (a line cleared a few moves later); we'll treat that on its own.
Laya's measured latency (0.015 s) is wired in too. On the updated board, over 1,000 paired matches in the same 60 s, ZTC clears 3.6 more lines than JEV and 0.9 more than Laya.
The page is live with the changes. Give it a run.
updated a
Space about 14 hours ago
reacted to their post with πππ₯ 1 day ago
Post
2951
The cost of a judging gate is usually quoted as a number. This puts it on a Tetris board.
Three boards get the same piece order, and on every move the same proposal and the same noise β a paired comparison. The gate decides one thing: keep this move, or draw again. Each board gets the same 60 seconds of gate time.
The text-writing gates get through 15β22 moves. The generation-free gate gets through 40β50. The boards that stop simply run out of clock.
It does not win on accuracy: on the same 2,018-question LODO set, JEV scores AUC 0.7350 against ZTC-Judge-27B's 0.7289. The separation is elsewhere. Clock β 2.1 s vs 0.0615 s per call, and on a 200-candidate agent screen one judging call measured 3.206 s generative vs 0.033 s readout, same server. Calibration β a gate is a threshold, and at ECE 0.4985 (vs ZTC 0.0245) a threshold stops carrying information. Mechanism β a text judge can name option 42 when there is no option 42; a scoring readout cannot. Not a lower error rate. No path.
The curve in the ZTC panel is real online fitting, scored prequentially β predict first, learn after β with base weights untouched. Not recursive self-improvement.
Limits, also stated on the page: Laya's AUC and latency are not our measurements and are set equal to JEV's, so calibration is the only measured axis it differs on. The page is a simulation driven by measured constants.
KO / EN / ZH.
FINAL-Bench/Tetris-JEV-LAYA-ZTC
FINAL-Bench/ZTC-Judge-27B
Three boards get the same piece order, and on every move the same proposal and the same noise β a paired comparison. The gate decides one thing: keep this move, or draw again. Each board gets the same 60 seconds of gate time.
The text-writing gates get through 15β22 moves. The generation-free gate gets through 40β50. The boards that stop simply run out of clock.
It does not win on accuracy: on the same 2,018-question LODO set, JEV scores AUC 0.7350 against ZTC-Judge-27B's 0.7289. The separation is elsewhere. Clock β 2.1 s vs 0.0615 s per call, and on a 200-candidate agent screen one judging call measured 3.206 s generative vs 0.033 s readout, same server. Calibration β a gate is a threshold, and at ECE 0.4985 (vs ZTC 0.0245) a threshold stops carrying information. Mechanism β a text judge can name option 42 when there is no option 42; a scoring readout cannot. Not a lower error rate. No path.
The curve in the ZTC panel is real online fitting, scored prequentially β predict first, learn after β with base weights untouched. Not recursive self-improvement.
Limits, also stated on the page: Laya's AUC and latency are not our measurements and are set equal to JEV's, so calibration is the only measured axis it differs on. The page is a simulation driven by measured constants.
KO / EN / ZH.
FINAL-Bench/Tetris-JEV-LAYA-ZTC
FINAL-Bench/ZTC-Judge-27B
- 5 replies
posted an update 1 day ago
Post
2951
The cost of a judging gate is usually quoted as a number. This puts it on a Tetris board.
Three boards get the same piece order, and on every move the same proposal and the same noise β a paired comparison. The gate decides one thing: keep this move, or draw again. Each board gets the same 60 seconds of gate time.
The text-writing gates get through 15β22 moves. The generation-free gate gets through 40β50. The boards that stop simply run out of clock.
It does not win on accuracy: on the same 2,018-question LODO set, JEV scores AUC 0.7350 against ZTC-Judge-27B's 0.7289. The separation is elsewhere. Clock β 2.1 s vs 0.0615 s per call, and on a 200-candidate agent screen one judging call measured 3.206 s generative vs 0.033 s readout, same server. Calibration β a gate is a threshold, and at ECE 0.4985 (vs ZTC 0.0245) a threshold stops carrying information. Mechanism β a text judge can name option 42 when there is no option 42; a scoring readout cannot. Not a lower error rate. No path.
The curve in the ZTC panel is real online fitting, scored prequentially β predict first, learn after β with base weights untouched. Not recursive self-improvement.
Limits, also stated on the page: Laya's AUC and latency are not our measurements and are set equal to JEV's, so calibration is the only measured axis it differs on. The page is a simulation driven by measured constants.
KO / EN / ZH.
FINAL-Bench/Tetris-JEV-LAYA-ZTC
FINAL-Bench/ZTC-Judge-27B
Three boards get the same piece order, and on every move the same proposal and the same noise β a paired comparison. The gate decides one thing: keep this move, or draw again. Each board gets the same 60 seconds of gate time.
The text-writing gates get through 15β22 moves. The generation-free gate gets through 40β50. The boards that stop simply run out of clock.
It does not win on accuracy: on the same 2,018-question LODO set, JEV scores AUC 0.7350 against ZTC-Judge-27B's 0.7289. The separation is elsewhere. Clock β 2.1 s vs 0.0615 s per call, and on a 200-candidate agent screen one judging call measured 3.206 s generative vs 0.033 s readout, same server. Calibration β a gate is a threshold, and at ECE 0.4985 (vs ZTC 0.0245) a threshold stops carrying information. Mechanism β a text judge can name option 42 when there is no option 42; a scoring readout cannot. Not a lower error rate. No path.
The curve in the ZTC panel is real online fitting, scored prequentially β predict first, learn after β with base weights untouched. Not recursive self-improvement.
Limits, also stated on the page: Laya's AUC and latency are not our measurements and are set equal to JEV's, so calibration is the only measured axis it differs on. The page is a simulation driven by measured constants.
KO / EN / ZH.
FINAL-Bench/Tetris-JEV-LAYA-ZTC
FINAL-Bench/ZTC-Judge-27B
- 5 replies
liked a
Space 1 day ago
published a
Space 1 day ago
updated 2
models 3 days ago
updated a
Space 3 days ago
updated 2
models 3 days ago
liked 2
models 4 days ago
reacted to their post with ππ€β€οΈ 4 days ago
Post
3137
Ask a language model how confident it is and you get an AUC of 0.5000. Exactly a coin flip. We measured it across 2,018 items.
FINAL-Bench/gate-tetris
https://huggingface.co/blog/FINAL-Bench/ztc
Collection: https://huggingface.co/collections/FINAL-Bench/ztc-models-jev-ecosystems
Zero-Token Confidence (ZTC) reads it. One forward pass over the model's hidden state returns a calibrated probability that the answer is correct. Zero generated tokens.
It sits at the top of the shared board. Same 2,018 items, same harness for every entry: ZTC on Darwin-397B 0.7394, JEV 0.7335, ZTC-Judge-27B 0.7255, a surface baseline that reads only answer length and formatting 0.7036, Lynx 8B 0.5157, the model's own self-reported confidence 0.5000, HHEM 0.4852. First and third place both emit nothing at all.
The number worth staring at is 0.7036. That is a baseline reading no content whatsoever, just how long the answer is and how it is formatted. Any verifier scoring below it is not reading content either.
On speed, one gate call costs 0.0615 seconds, measured on four B200s across 2,000 items. Generating a single candidate answer takes 1.631 seconds, so the gate is 26 times cheaper than the work it guards. A verifier that generates competes with your agent for the same budget. A verifier that only reads can be attached to every action instead of a sampled few.
We built it so you can watch it decide. Three lanes receive the same stream of proposed actions and the same time budget. One has no gate and must execute everything. One uses a text-reading verifier. One uses ZTC. Right action plus one, wrong action minus one, hold zero. Over 400 matches: no gate minus 3.9, text verifier plus 13.0, ZTC plus 29.1, with ZTC taking 98 percent of matches. Gating lifts executed accuracy from 49 percent to 65 percent.
FINAL-Bench/gate-tetris
https://huggingface.co/blog/FINAL-Bench/ztc
Collection: https://huggingface.co/collections/FINAL-Bench/ztc-models-jev-ecosystems
Zero-Token Confidence (ZTC) reads it. One forward pass over the model's hidden state returns a calibrated probability that the answer is correct. Zero generated tokens.
It sits at the top of the shared board. Same 2,018 items, same harness for every entry: ZTC on Darwin-397B 0.7394, JEV 0.7335, ZTC-Judge-27B 0.7255, a surface baseline that reads only answer length and formatting 0.7036, Lynx 8B 0.5157, the model's own self-reported confidence 0.5000, HHEM 0.4852. First and third place both emit nothing at all.
The number worth staring at is 0.7036. That is a baseline reading no content whatsoever, just how long the answer is and how it is formatted. Any verifier scoring below it is not reading content either.
On speed, one gate call costs 0.0615 seconds, measured on four B200s across 2,000 items. Generating a single candidate answer takes 1.631 seconds, so the gate is 26 times cheaper than the work it guards. A verifier that generates competes with your agent for the same budget. A verifier that only reads can be attached to every action instead of a sampled few.
We built it so you can watch it decide. Three lanes receive the same stream of proposed actions and the same time budget. One has no gate and must execute everything. One uses a text-reading verifier. One uses ZTC. Right action plus one, wrong action minus one, hold zero. Over 400 matches: no gate minus 3.9, text verifier plus 13.0, ZTC plus 29.1, with ZTC taking 98 percent of matches. Gating lifts executed accuracy from 49 percent to 65 percent.