https://github.com/jundot/omlx/releases/tag/v0.5.0.dev2
Among the main features, Lightning MTP stands out.
I tried running a benchmark with the new quantized oQe model.
The output token speed is in the 30s.
Since it's 4-bit quantization, I'll have to test it through actual coding to see how well it performs.
oMLX - LLM inference, optimized for your Mac
https://github.com/jundot/omlx
Benchmark Model: Qwen3.6-27B-oQ4e-mtp
Engine: Auto
================================================================================
Single Request Results
--------------------------------------------------------------------------------
Test TTFT(ms) TPOT(ms) pp TPS tg TPS E2E(s) Throughput Peak Mem
pp1024/tg128 2286.0 31.90 447.9 tok/s 31.6 tok/s 6.356 181.3 tok/s 16.74 GB
pp4096/tg128 8983.6 31.02 455.9 tok/s 32.5 tok/s 12.948 326.2 tok/s 18.16 GB
pp8192/tg128 18706.4 34.90 437.9 tok/s 28.9 tok/s 23.166 359.1 tok/s 19.20 GB
pp16384/tg128 48031.7 34.05 341.1 tok/s 29.6 tok/s 52.402 315.1 tok/s 19.51 GB
pp32768/tg128 123238.3 39.08 265.9 tok/s 25.8 tok/s 128.237 256.5 tok/s 21.28 GB
