DoktorMincs/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-MTP-W4A16-AutoRound

Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-MTP — W4A16 AutoRound

4-bit weight-only quantization (W4A16, int4 symmetric, group_size 128) of DavidAU/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-MTP using AutoRound via llm-compressor 0.12 + auto-round 0.13.

• Quantized: all language-model Linear layers (400 modules: attention, MLP and GatedDeltaNet projections) → compressed-tensors pack-quantized (served by vLLM's Marlin kernels).
• Preserved in BF16 (untouched, bit-identical to the base model):
  • Vision tower (model.visual.*, 333 tensors)
  • MTP / Multi-Token Prediction head (mtp.*, 15 tensors) — enables speculative decoding
  • lm_head, embeddings, norms, conv1d, and the GatedDeltaNet in_proj_a/in_proj_b projections (output dim 48 < group_size 128)
• Calibration: 128 samples × 2048 tokens from neuralmagic/LLM_compression_calibration with the model's own chat template, 200 AutoRound iterations per layer.
• Size: ~19.5 GB (base BF16: ~52 GB).

Usage (vLLM ≥ 0.26)

Production example (4× RTX 3090, tool calling + reasoning parser + MTP speculative decoding). Replace the local path with the repo id DoktorMincs/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-MTP-W4A16-AutoRound to load directly from the Hub:

export CUDA_DEVICE_ORDER=PCI_BUS_ID
export OMP_NUM_THREADS=4
export CUDA_VISIBLE_DEVICES=0,1,2,3
export PYTORCH_ALLOC_CONF=expandable_segments:True
export NCCL_P2P_DISABLE=1
export FLASHINFER_DISABLE_VERSION_CHECK=1

exec vllm serve \
    /root/models/DoktorMincs/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-MTP-W4A16-AutoRound \
    --served-model-name "qwen3.6-27b" \
    --tensor-parallel-size 4 \
    --max-model-len 163840 \
    --override-generation-config '{"temperature": 0.6, "top_p": 0.95, "top_k": 20, "min_p": 0.0, "presence_penalty": 0.0, "repetition_penalty": 1.0}' \
    --tool-call-parser qwen3_coder \
    --reasoning-parser qwen3 \
    --enable-auto-tool-choice \
    --trust-remote-code \
    --enable-prefix-caching \
    --max-num-batched-tokens 8192 \
    --gpu-memory-utilization 0.88 \
    --disable-custom-all-reduce \
    --max-num-seqs 16 \
    --host 0.0.0.0 \
    --port 3434 \
    --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}'

• Vision inputs work normally (image/video); the vision tower runs in BF16.
• The MTP draft head loads from the same checkpoint for speculative decoding.
• Validated on vLLM 0.26 (text, MTP speculative decoding, vision).

Notes

• Architecture: Qwen3_5ForConditionalGeneration (hybrid: 64 layers, 3:1 GatedDeltaNet linear-attention : full-attention, 27-block ViT, 1 MTP layer).
• chat_template-instruct.jinja is the author's alternate template shipped with the base model.
• Calibration data is general (math/code/logic/science QA); for heavy RP/creative use, a domain-matched calibration set may further improve fidelity.