vLLM W4A16?

#4
by hiutaky - opened

Is planned a W4A16 ?

would really appreciate Intel Autoround quant for this.

Or a BF16 version that the usual suspects could built upon them. cyankiwi/Intel/RedHat... for the Blackwell-less

Update: I'm working on a PR for vLLM to run the NVFP4 on the Ampere ( 30XX ). It's actually working but the KV context is pretty small ( < 20 000 tokens ). Once I can get a proper KV size I'll publish the PR or a fork.

Edit 1: reached 64k context by removing the vision tower and optimizing the batch size. Also enabled Dspark: the drafter increased the decode speed by +23 / +61%, also the occupied weight size 0,6GB.

I hope they do end up releasing an unquantized checkpoint or something, was hyped to try out the model till I saw nvfp4 😒

I hope they do end up releasing an unquantized checkpoint or something, was hyped to try out the model till I saw nvfp4 😒

You can now run NVFP4 with DSpark on sm86 Ampere RTX 30XX with vLLM

https://huggingface.co/thinkingmachines/Inkling-2-Mini/discussions/20

Sign up or log in to comment