k-quants - #1684

Merged

ggerganov merged 32 commits into

Jun 5, 2023

Commits on Jun 3, 2023

Starting to add k-quantization to ggml
Iwan Kawrakow
committed
Adding Q3_K and Q8_K (de)-quantization
Iwan Kawrakow
committed
Q3_K now working on CUDA and AVX2/scalar
Iwan Kawrakow
committed
Some improvement for Q3_K on CUDA
Iwan Kawrakow
committed
Some more CUDA optimizations for Q3_K
Iwan Kawrakow
committed
Adding Q4_K - scalar, AVX2, CUDA
Iwan Kawrakow
committed
Adding Q6_K - scalar, AVX2, CUDA
Iwan Kawrakow
committed
Adding Q5_K - scalar, AVX2, CUDA
Iwan Kawrakow
committed
Per convention, all QX_K quantizations use Q5_K for output.weight
Iwan Kawrakow
committed
Adding quantization mixes
Iwan Kawrakow
committed
Quantization mixes: didn't quite get what I wanted in the last commit
Iwan Kawrakow
committed
Q4_K dot product for ARM_NEON
Iwan Kawrakow
committed
Q6_K dot product for ARM_NEON
Iwan Kawrakow
committed
Q5_K dot product for ARM_NEON
Iwan Kawrakow
committed
Adding Q3_K dot for ARM_NEON
Iwan Kawrakow
committed
A very slightly faster ARM_NEON Q3_K dot
Iwan Kawrakow
committed
Adding Q2_K - just CUDA for now
Iwan Kawrakow
committed
Adding scalar and AVX2 Q2_K dot
Iwan Kawrakow
committed
Adding ARM_NEON Q2_K dot
Iwan Kawrakow
committed
A slightly faster ARM_NEON Q2_K dot
Iwan Kawrakow
committed
Fixed bug in Q2_K CUDA dot product kernel
Iwan Kawrakow
committed
Don't print zeros/NaNs when no count histogram has been collected
Iwan Kawrakow
committed
A 10% faster CUDA vector dot kernel for Q3_K
Iwan Kawrakow
committed
A slightly daster Q4_K AVX2 dot product
Iwan Kawrakow
committed
A slightly faster ARM_NEON A4_K dot product
Iwan Kawrakow
committed
Minor
Iwan Kawrakow
committed
Fix quantization error test
Iwan Kawrakow
committed
Fix docker build
Iwan Kawrakow
committed

Commits on Jun 4, 2023

Added forgotten ggml.o dependence on k_quants.h to the Makefile
Iwan Kawrakow
committed
Had unintentionally committed the Makefile with -Ofast enabled
Iwan Kawrakow
committed

Commits on Jun 5, 2023