Quant | Actions | ||||||
|---|---|---|---|---|---|---|---|
 mlx_lm0.31.3 | 54.9 tok/s | 488.1 tok/s | 16.00 GB 50% | ||||
| mlx_lm0.31.3 | 54.6 tok/s | 489.2 tok/s | 16.00 GB 50% | ||||
 llama.cppb8950 | 54.9 tok/s | 649.8 tok/s | 10.14 GB 21% | ||||
| mlx_lm0.31.3 | 51.1 tok/s | 2,139.2 tok/s | 28.00 GB 44% | ||||
| mlx_lm0.31.3 | 72.4 tok/s | 2,255.2 tok/s | 16.00 GB 25% | ||||
| mlx_lm0.31.2 | 71.5 tok/s | 1,154.4 tok/s | 29.00 GB 23% | ||||
| mlx_lm0.31.2 | 97.3 tok/s | 1,163.5 tok/s | 17.00 GB 13% | ||||
| mlx_lm0.31.2 | 32.3 tok/s | 1,042.9 tok/s | 17.00 GB 53% | ||||
| llama.cppb8796 | 7.3 tok/s | 125.3 tok/s | 14.18 GB 89% | ||||
| mlx_lm0.31.2 | 25.5 tok/s | 231.7 tok/s | 15.00 GB 47% |
Quant | Actions | ||
|---|---|---|---|
54.9 tok/s | |||
54.6 tok/s | |||
54.9 tok/s | |||
51.1 tok/s | |||
72.4 tok/s | |||
71.5 tok/s | |||
97.3 tok/s | |||
32.3 tok/s | |||
7.3 tok/s | |||
25.5 tok/s |