Quant | Actions | ||||||
|---|---|---|---|---|---|---|---|
 llama.cppb9110 | 93.3 tok/s | 3,378.0 tok/s | 5.08 GB 16% | ||||
 mlx_lm0.31.2 | 91.8 tok/s | 7,841.7 tok/s | 7.29 GB 11% | ||||
| mlx_lm0.31.2 | 23.6 tok/s | 349.6 tok/s | 5.52 GB 17% | ||||
| mlx_lm0.31.2 | 40.3 tok/s | 760.3 tok/s | 6.08 GB 34% | ||||
| llama.cppb8796 | 50.2 tok/s | 2,608.7 tok/s | 7.95 GB 50% | ||||
| llama.cppb8680 | 7.8 tok/s | 38.4 tok/s | 7.23 GB 45% | ||||
| mlx_lm0.31.2 | 15.6 tok/s | 455.5 tok/s | 9.05 GB 57% | ||||
| mlx_lm0.31.2 | 77.3 tok/s | 3,650.1 tok/s | 5.83 GB 12% | ||||
| mlx_lm0.31.2 | 47.7 tok/s | 3,422.8 tok/s | 11.00 GB 23% | ||||
| llama.cppb8680 | 65.7 tok/s | 1,429.8 tok/s | 6.75 GB 14% |
Quant | Actions | ||
|---|---|---|---|
93.3 tok/s | |||
91.8 tok/s | |||
23.6 tok/s | |||
40.3 tok/s | |||
50.2 tok/s | |||
7.8 tok/s | |||
15.6 tok/s | |||
77.3 tok/s | |||
47.7 tok/s | |||
65.7 tok/s |