Quant | ||||||
|---|---|---|---|---|---|---|
 llama.cppb8680 | 13.1 tok/s | 194.9 tok/s | 0.26 GB 3% | |||
| llama.cppb8680 | 40.2 tok/s | 698.6 tok/s | 0.52 GB 2% | |||
| llama.cppb8680 | 55.4 tok/s | 844.4 tok/s | 1.34 GB 8% | |||
| llama.cppb8680 | 29.6 tok/s | 354.2 tok/s | 1.87 GB 8% | |||
| llama.cppb8680 | 79.6 tok/s | 2,470.5 tok/s | 4.24 GB 7% | |||
208.3 tok/s | 12,775.9 tok/s | 2.69 GB 12% | ||||
| llama.cppb8680 | 36.2 tok/s | 613.8 tok/s | 3.72 GB 15% | |||
 mlx_lm0.31.2 | 81.9 tok/s | 2,135.3 tok/s | 4.23 GB 13% | |||
| llama.cppb8672 | 24.6 tok/s | 173.5 tok/s | 4.23 GB 28% | |||
| llama.cppb8679 | 12.0 tok/s | 50.6 tok/s | 4.23 GB 14% |
Quant | Actions | ||
|---|---|---|---|
13.1 tok/s | |||
40.2 tok/s | |||
55.4 tok/s | |||
29.6 tok/s | |||
79.6 tok/s | |||
208.3 tok/s | |||
36.2 tok/s | |||
81.9 tok/s | |||
24.6 tok/s | |||
12.0 tok/s |