I wanted to try running a local LLM, so I installed Ollama, but SAC (Smart App Control) kept blocking it, so I gave up and started using LM Studio to experience local LLMs. At first I tried it with a single 3070ti, but since I wanted to run larger models, I added a 1660ti that was sitting idle to my computer. LM Studio doesn't handle different multi-GPU setups very well. So I started using llama.cpp directly. When using smaller models, there are almost no issues, but multimodal (image and speech recognition) throws errors. I tested several options, but with a 3070+1660 setup it was impossible to use stably. When running just the 3070ti alone, it operates stably.
Another issue is instability when using larger models like Gemma4-26B-A4B. When I had logs analyzed by Gemini or Codex, the 3070 runs fast with CUDA+tensor cores, but the 1660 has CUDA but no tensor cores, so llama.cpp has some trouble handling both simultaneously. After testing this back and forth, I boldly abandoned CUDA/tensor core operations and switched to running with Vulkan, and finally succeeded in operating it stably. In exchange, processing speed became a bit slower. (As an aside, before switching to Vulkan, ChatGPT said Vulkan wouldn't solve it, it's a hardware problem so there's nothing to be done, and strongly recommended using only the 3070. You know nothing.)
And even with Vulkan, multimodal support is not possible. Image and audio recognition can only be done with a single GPU. I still don't know if this is because llama.cpp doesn't support it or if I selected the options incorrectly.
These days I'm thinking about it every day—whether to upgrade my graphics card or not. Since I'm just running it for fun, I'm putting up with it, but I have no idea how it will turn out.
