Absolute Beast
After the initial review and testing I realized the existing benchmark tools are somewhat lacking, many are saturated, some of the graders have issues and some of the tests themselves are traps in the OSS tool call testing.
I've built my own full testing suite now, including multiturn highly complex code generation, edit, bug fixing etc, and I will never share the internals to prevent any contamination from ever occurring, but its comprehensive on REAL work in a REAL codebase, spanning rust, python, typescript, vllm, a full inference client stack (think LM Studio + Claude Code + Open Router + Parts of Cursor merged into a single application).
Initial tests are vastly better than existing benchmarks, it clearly stratifies the model tiers with failure rates comensurate to my personal experience with how "smart" a model is. I will publish numbers and test concepts/desriptions for understanding but currently here is the order of the models that have undergone evaluation:
Fable
Opus
Sonnet
711
Haiku
.
.
[large gap]
.
.
Qwen35B
When given semi-technical/layman prompts, the gap from 711 to sonnet is tighter than haiku to 711.
THE REAL NEWS: When given a detailed spec, it matches opus at implementations. If you don't let it do the architecture and give it just implementation, we have a small monster. In the hands of a SWE it is highly capable, highly.
I am using 711 to back my molested version of OpenClaude, to further trim cruft in OpenClaude out and make it a less cancerous application. 711 is doing great, the codebase is >1m LoC, and even with 512k context, it's holding together deep in context window. Invokes agents without issue, handles file search navigation, updates, linting, git, merge conflicts etc with grace. I put it solidly in the Sonnet 3.5 (when sonnet was actually a coding beast vs opus 4) era.
Your recipe has created essentially the Qwen3.7 we were never given IMHO.
Thank you very much for this, really appreciate the thorough testing!
As I mentioned in previous threads, this was put together with what was there at the time, Heretic or not.
I will ask Armando if he could build the same Fable traces on the common ablit base(why mess with a good training regimen), and of course David is working on tightening up the components so we can rebuild this as fully Heretic.
Meanwhile, 711 is not the ceiling.
I noticed you compared with Tess--a great model by itself by the way--that helped the 711 become 712
https://huggingface.co/nightmedia/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-Tess-mxfp8-mlx
Gul Dukat comes out to play :)
The model is a bit more RP-friendly on lower quants, even on mxfp4
arc arc/e boolq hswag obkqa piqa wino
mxfp8 0.712,0.879,0.911,0.792,0.508,0.823,0.764
qx86-hi 0.701,0.877,0.911,0.794,0.518,0.823,0.758
qx64-hi 0.706,0.873,0.909,0.795,0.512,0.823,0.752
mxfp4 0.706,0.873,0.910,0.790,0.496,0.817,0.761
1M
qx64-hi 0.706,0.873,0.909,0.795,0.512,0.823,0.752
Quant Perplexity Peak Memory Tokens/sec
mxfp8 3.797 ± 0.024 34.74 GB 183
qx86-hi 3.756 ± 0.023 33.25 GB 170
qx64-hi 3.765 ± 0.023 27.03 GB 181
mxfp4 3.876 ± 0.024 21.26 GB 176
1M
mxfp8 3.803 ± 0.024 34.70 GB 169
qx64-hi 3.769 ± 0.023 26.99 GB 172
mxfp4 3.876 ± 0.024 21.26 GB 176
To put metrics in context, the 711:
arc arc/e boolq hswag obkqa piqa wino
mxfp8 0.711,0.879,0.910,0.790,0.514,0.823,0.763
qx86-hi 0.696,0.876,0.912,0.791,0.518,0.824,0.760
qx64-hi 0.702,0.873,0.909,0.794,0.514,0.822,0.750
mxfp4 0.701,0.873,0.909,0.786,0.488,0.813,0.759
Quant Perplexity Peak Memory Tokens/sec
mxfp8 3.783 ± 0.023 34.74 GB 203
qx86-hi 3.735 ± 0.023 33.25 GB 183
qx64-hi 3.747 ± 0.023 27.03 GB 194
mxfp4 3.854 ± 0.024 21.30 GB 197
Source is available
https://huggingface.co/nightmedia/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-Tess
I've been doing some of my experimental training, based on AEON using unsloth, mostly as a pipeline and correctness test for the new AMD unsloth software. Working on developing a specialist version and seeing what is possible from the 27B base is a great help.
Very much looking forward to David's larger 40B with the magic applied to it.
PS: I am willing to help train on my rig, doesnt buy you more video memory, (128 VRAM, 256 System), but does buy a ton more compute power 😔 If interested in just DM me and we can work something out.
https://huggingface.co/nightmedia/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-Tess-mxfp8-mlx
Gul Dukat comes out to play :)
The model is a bit more RP-friendly on lower quants, even on mxfp4
arc arc/e boolq hswag obkqa piqa wino mxfp8 0.712,0.879,0.911,0.792,0.508,0.823,0.764 qx86-hi 0.701,0.877,0.911,0.794,0.518,0.823,0.758 qx64-hi 0.706,0.873,0.909,0.795,0.512,0.823,0.752 mxfp4 0.706,0.873,0.910,0.790,0.496,0.817,0.761 1M qx64-hi 0.706,0.873,0.909,0.795,0.512,0.823,0.752 Quant Perplexity Peak Memory Tokens/sec mxfp8 3.797 ± 0.024 34.74 GB 183 qx86-hi 3.756 ± 0.023 33.25 GB 170 qx64-hi 3.765 ± 0.023 27.03 GB 181 mxfp4 3.876 ± 0.024 21.26 GB 176 1M mxfp8 3.803 ± 0.024 34.70 GB 169 qx64-hi 3.769 ± 0.023 26.99 GB 172 mxfp4 3.876 ± 0.024 21.26 GB 176
@nightmedia can you share what you ran this against to get the peak memory and TG on that mxfp8 ? i'm sorry, i dont really understand the test types there, but i'm wondering what machine you use, what mlx runtime loader, and what that TG is under what context and if you had PP numbers as well.
All the tests are done in Instruct mode--in think mode the test numbers will be naturally lower.
This is probably an issue that everyone has in reproducing metrics: set the model in Instruct mode in the chat_template.jinja
{%- set enable_thinking = false %}
The perplexity numbers are not affected by the instruct flag, but they are visibly affected by the RoPE, with one notable exception for this model, mxfp4. The mxfp4 peforms at the exactly same numbers in 256k context and 1M context. In the other model, this effect appears in qx64-hi, showing the sweet spot of the model