https://github.com/jundot/omlx/pull/3161
If you look at this PR, the 2bit peak memory usage is 37gb.
The method is to put nmap on the ssd and keep only the model in ram.
Since I have a 64GB Mac, it seems possible to shoot with 2bit haha.
But looking at the speed, it's 30t/s on M5Max, so I expect it to be halved on M5Pro, around 15t/s.
If A6B 2bit is not smarter than 27B, there doesn't seem to be much point in using it.
That's a shame.
