whatcani.run on an M4 Pro with 24 GB RAM?
Find the best models and how to run them locally, based on real data from tokens across trials from people.Models
Results include trials with 4,096 input tokens and 1,024 output tokens only.
Actions | ||||||||
|---|---|---|---|---|---|---|---|---|
 llama.cpp | 293.1 tok/s | 7,251.8 tok/s | 565 ms | 0.43 GB 2% | 101 run | Runs great | ||
 mlx_lm | 241.1 tok/s | 3,686.7 tok/s | 1,102 ms | 2.46 GB 10% | 505 runs | Runs great | ||
| llama.cpp | 44.4 tok/s | 595.1 tok/s | 6.87 sec | 1.99 GB 8% | 101 run | Runs well | ||
| llama.cpp | 41.9 tok/s | 590.7 tok/s | 6.82 sec | 1.86 GB 8% | 303 runs | Runs well | ||
| llama.cpp | 35.3 tok/s | 469.5 tok/s | 8.25 sec | 2.82 GB 12% | 101 run | Runs ok | ||
| llama.cpp | 24.3 tok/s | 298.0 tok/s | 13.72 sec | 1.74 GB 7% | 101 run | Runs ok | ||
| mlx_lm | 46.8 tok/s | 355.4 tok/s | 11.44 sec | 7.17 GB 30% | 101 run | Runs ok | ||
| mlx_lm | 81.7 tok/s | 671.3 tok/s | 6.07 sec | 13.00 GB 54% | 101 run | Runs ok | ||
| mlx_lm | 15.1 tok/s | 100.4 tok/s | 40.69 sec | 18.00 GB 75% | 101 run | Barely runs | ||
| mlx_lm | 14.7 tok/s | 96.4 tok/s | 41.47 sec | 18.00 GB 75% | 101 run | Barely runs |
| Actions | |||
|---|---|---|---|
293.1 tok/s | Runs great | ||
241.1 tok/s | Runs great | ||
44.4 tok/s | Runs well | ||
41.9 tok/s | Runs well | ||
35.3 tok/s | Runs ok | ||
24.3 tok/s | Runs ok | ||
46.8 tok/s | Runs ok | ||
81.7 tok/s | Runs ok | ||
15.1 tok/s | Barely runs | ||
14.7 tok/s | Barely runs |