Add runntime int4 and int8 weights
#2
by kamkacz - opened
model/runntime/{int4,int8}: the int4 and int8 files from model/safetensors, repacked to runntime's quant format (int4 nibbles as two's complement, runntime.quant metadata added). No weight value changed. runntime loads these, plus model/safetensors/fp32 and model/tokenizer.json as they are.
kamkacz changed pull request status to open
benito47 changed pull request status to merged