Quant | Actions | ||||||
|---|---|---|---|---|---|---|---|
 mlx_lm0.31.3 | 100.8 tok/s | 1,230.7 tok/s | 17.00 GB 27% | ||||
 llama.cppb8920 | 60.6 tok/s | 1,542.2 tok/s | 9.15 GB 14% | ||||
| llama.cppb9430 | 12.2 tok/s | 220.0 tok/s | 0.98 GB 4% | ||||
| mlx_lm0.31.3 | 54.9 tok/s | 488.1 tok/s | 16.00 GB 50% | ||||
| mlx_lm0.31.3 | 54.6 tok/s | 489.2 tok/s | 16.00 GB 50% | ||||
| llama.cppb8950 | 54.9 tok/s | 649.8 tok/s | 10.14 GB 21% | ||||
| mlx_lm0.31.3 | 51.1 tok/s | 2,139.2 tok/s | 28.00 GB 44% | ||||
| mlx_lm0.31.3 | 72.4 tok/s | 2,255.2 tok/s | 16.00 GB 25% | ||||
| mlx_lm0.31.2 | 71.5 tok/s | 1,154.4 tok/s | 29.00 GB 23% | ||||
| mlx_lm0.31.2 | 97.3 tok/s | 1,163.5 tok/s | 17.00 GB 13% |
Quant | Actions | ||
|---|---|---|---|
100.8 tok/s | |||
60.6 tok/s | |||
12.2 tok/s | |||
54.9 tok/s | |||
54.6 tok/s | |||
54.9 tok/s | |||
51.1 tok/s | |||
72.4 tok/s | |||
71.5 tok/s | |||
97.3 tok/s |