After trying various configurations
llamacpp is twice as good as ollama. Token processing speed is much faster with MoE. Most of the crashing issues that existed before have also been resolved.
Qwen3.6 35B A3B can run on IQ2M, Qwen 3.6 27B can run on IQ3_S or XXS, and Gemma4 26B A4B can run on IQ3_S or XXS (Gemma4 26B A4B with IQ3_XXS allows for smooth multimodal operation)
The rest, such as 8B and 14B, etc., naturally work well. Once you try MoE, you'll want to keep using it...
MTP is not possible with MoE for the above 35B or 26B models. I tried Qwen3.6 IQ2M because it had some memory left, but llamacpp crashes when the token count exceeds about 2K. According to Reddit, at least 2GB of additional memory is required. 8B models and similar ones should be sufficient for MTP.
I tried heretic, uncensored, and other tuning models, but ultimately the unsloth model worked best.
Qwen3.6 27B only achieves about 13-15 tps, making it difficult to use. Realistically, 35B A3B with IQ2M (TG 50-60tps) and Gemma4 26B A4B with IQ3_XSS (TG 50-55tps) are optimal.
Both can handle CTX up to 128K, but performance drops to around 20-30 tps when it exceeds 80-90K.
Of course, the probability of nonsense also increases. Especially with Qwen3.6 on IQ2M (sentence repetition occurs).
Ultimately, I'm using Gemma4 26B-A4B unsloth IQ3_XXS with multimodal enabled.
Even if you input a context of up to 128K and run cline, it won't crash or stop, although it may slow down when it exceeds 80-90K.
Using multimodal consumes all available memory, so SSD swapping occurs intermittently, making it slower. However, there is performance degradation even when running in memory, so I leave it as is. (Swapping about 1GB when reaching 128K)
If you don't like swapping, you can reduce the context to 64K or remove multimodal.
Still, Gemma4 doesn't have sentence repetition with IQ3.
I think it's usable as a hobby coding agent. With multimodal enabled, it analyzes graphs and tables when you provide them.
Of course, to do this on BC250, you need to run BC250 headless and only serve llamacpp, with everything else handled by other servers like openwebui, opencode, or rag.
Personally, I think BC250 is the best choice for those who are interested in hobby agent AI.
Both M1 ultra and BC250 have the same bandwidth, and both achieve TG 50-60tps when running Qwen 3.6 35B A3B with 100% GPU offload.
Isn't this great?
