Google

Google

Google/Gemma 4 31B IT

Quantizations

Quant	Quantized by	Size	Decode	Prefill	Score	Actions
q3_k_m	Unsloth Unsloth	13.7 GB	N/A	N/A	N/A	Run
q4_k_m	Unsloth Unsloth	17.1 GB	11.7 tok/s	100.3 tok/s	Runs poorly	Run
q4_k_m	Unsloth Unsloth	17.1 GB	N/A	N/A	N/A	Run
q4_k_m	ggml ggml	17.4 GB	N/A	N/A	N/A	Run
ud-q4_k_xl	Unsloth Unsloth	17.5 GB	11.1 tok/s	97.7 tok/s	Runs poorly	Run
q5_k_m	Unsloth Unsloth	20.2 GB	N/A	N/A	N/A	Run
ud-4bit	Unsloth Unsloth	20.6 GB	9.4 tok/s	92.3 tok/s	Runs poorly	Run
q8_0	Unknown	30.4 GB	7.3 tok/s	102.8 tok/s	Barely runs	N/A
8bit	MLX Community MLX Community	31.4 GB	6.9 tok/s	91.1 tok/s	Barely runs	Run

Device Comparison

Results include trials with 4,096 input tokens and 1,024 output tokens only.