Quant | Actions | ||||||
|---|---|---|---|---|---|---|---|
 mlx_lm0.31.3 | 13.1 tok/s | 180.3 tok/s | 7.50 GB 47% | ||||
| mlx_lm0.31.3 | 106.4 tok/s | 4,184.5 tok/s | 5.92 GB 16% | ||||
 llama.cppb9110 | 93.3 tok/s | 3,378.0 tok/s | 5.08 GB 16% | ||||
| mlx_lm0.31.2 | 91.8 tok/s | 7,841.7 tok/s | 7.29 GB 11% | ||||
| mlx_lm0.31.2 | 23.6 tok/s | 349.6 tok/s | 5.52 GB 17% | ||||
| mlx_lm0.31.2 | 40.3 tok/s | 760.3 tok/s | 6.08 GB 34% | ||||
| llama.cppb8796 | 50.2 tok/s | 2,608.7 tok/s | 7.95 GB 50% | ||||
| llama.cppb8680 | 7.8 tok/s | 38.4 tok/s | 7.23 GB 45% | ||||
| mlx_lm0.31.2 | 15.6 tok/s | 455.5 tok/s | 9.05 GB 57% | ||||
| mlx_lm0.31.2 | 77.3 tok/s | 3,650.1 tok/s | 5.83 GB 12% |
Quant | Actions | ||
|---|---|---|---|
13.1 tok/s | |||
106.4 tok/s | |||
93.3 tok/s | |||
91.8 tok/s | |||
23.6 tok/s | |||
40.3 tok/s | |||
50.2 tok/s | |||
7.8 tok/s | |||
15.6 tok/s | |||
77.3 tok/s |