Quant | ||||||
|---|---|---|---|---|---|---|
q8_0  | llama.cppb8793 | 40.2 tok/s | 789.9 tok/s | 4.86 GB 8% | ||
 mlx_lm0.31.1 | 106.6 tok/s | 1,228.9 tok/s | 5.00 GB 4% | |||
| mlx_lm0.31.1 | 30.6 tok/s | 357.4 tok/s | 4.60 GB 29% | |||
| llama.cppb8610 | 83.9 tok/s | 1,251.3 tok/s | 4.78 GB 10% | |||
| mlx_lm0.31.1 | 138.2 tok/s | 1,476.4 tok/s | 4.83 GB 10% | |||
| llama.cppb8610 | 85.6 tok/s | 1,329.2 tok/s | 3.61 GB 8% | |||
| mlx_lm0.31.1 | 35.9 tok/s | 392.0 tok/s | 4.52 GB 28% | |||
| mlx_lm0.31.1 | 52.6 tok/s | 293.5 tok/s | 4.66 GB 29% | |||
| mlx_lm0.31.1 | 88.6 tok/s | 940.6 tok/s | 4.87 GB 14% | |||
| mlx_lm0.31.1 | 80.4 tok/s | 768.6 tok/s | 4.76 GB 20% |
Quant | Actions | ||
|---|---|---|---|
q8_0 | 40.2 tok/s | ||
106.6 tok/s | |||
30.6 tok/s | |||
83.9 tok/s | |||
138.2 tok/s | |||
85.6 tok/s | |||
35.9 tok/s | |||
52.6 tok/s | |||
88.6 tok/s | |||
80.4 tok/s |