Was this really trained during QAT using a symmetric 4bit quant with only 15/16 values used?
Can you confirm the original QAT model was trained using symmetric 4bit values with two zeros, eg:
-7, -6, ..., -1, 0, 0, +1, ..., +6, +7
See my discussion starting here about converting this model to use Q4_0 in llama.cpp:
https://github.com/ggml-org/llama.cpp/pull/17064#issuecomment-3520544778
I worry that the QAT might have used asymmetric 4bit values like:
-7, -6, ..., -1, 0, +1, ..., +7, +8
and then have been converted to vllm's compressed-tensors afterwards...
It also seems very suspicious to me that out of all the 32-element blocks I haven't found a single one where the range of 4bit values doesn't take up the full {-7, ..., +7} range. I would have thought that at least some blocks during QAT training would have moved the range to {-6, ..., +6} or less.
After some more experiments this afternoon, it maybe looks like it's not two zeros specifically, but just one end of the range never gets used, eg:
-7, -6, ..., -1, 0, +1, ..., +7
So one of the 16 int4 values isn't used?
I
After some more experiments this afternoon, it maybe looks like it's not two zeros specifically, but just one end of the range never gets used, eg:
-7, -6, ..., -1, 0, +1, ..., +7So one of the 16
int4values isn't used?
If this is true, does it mean that one-sixteenth of the space remains unused?
We used symmetric quantization and didn't use -8 to avoid extra quant bias.