Quant | Actions | ||||||
|---|---|---|---|---|---|---|---|
 mlx_lm0.31.3 | 66.7 tok/s | 608.3 tok/s | 21.00 GB 33% | ||||
| mlx_lm0.31.3 | 59.9 tok/s | 872.6 tok/s | 22.00 GB 46% | ||||
 llama.cppb8950 | 22.7 tok/s | 252.9 tok/s | 2.93 GB 18% | ||||
| mlx_lm0.31.2 | 113.0 tok/s | 1,616.7 tok/s | 21.00 GB 16% | ||||
| mlx_lm0.31.2 | 111.9 tok/s | 1,549.2 tok/s | 21.00 GB 16% | ||||
| llama.cppb8890 | 74.1 tok/s | 1,106.9 tok/s | 16.94 GB 13% | ||||
| mlx_lm0.31.3 | 95.8 tok/s | 2,360.1 tok/s | 21.00 GB 33% | ||||
| mlx_lm0.31.3 | 90.2 tok/s | 2,150.3 tok/s | 21.00 GB 33% | ||||
| mlx_lm0.31.2 | 40.2 tok/s | 305.2 tok/s | 17.00 GB 53% | ||||
| mlx_lm0.31.2 | 35.3 tok/s | 641.6 tok/s | 21.00 GB 66% |
Quant | Actions | ||
|---|---|---|---|
66.7 tok/s | |||
59.9 tok/s | |||
22.7 tok/s | |||
113.0 tok/s | |||
111.9 tok/s | |||
74.1 tok/s | |||
95.8 tok/s | |||
90.2 tok/s | |||
40.2 tok/s | |||
35.3 tok/s |