I have a main PC with a 5080 installed and an Allyx with a 9070xt as an eGPU, both being used separately for local LLMs. I suddenly thought, "If I combine these, wouldn't the memory double?!"
I immediately turned on Claude to see if it was possible.
Without even searching the web, it gave me an immediate answer: "No".
Recently, I've been finding it hard to trust Claude's answers, so when I said, "I checked," I ended up searching anyway.
It seems there have been some successful cases, but combining CUDA and ROCm is difficult.
Since I don't believe anything unless I try it myself, I checked various things. When building llama.cpp from the source, it seems that if you include the ROCm graphics card during the build process, it will be built together.
AMD doesn't properly support the WSL environment, so I had to install and build it on Windows, which is an unfamiliar environment and made me very frustrated.
When error messages appeared in the middle, I asked Claude and ChatGPT. Although Claude gave me answers, it kept trying to make me give up.
Somehow, I succeeded in getting both graphics cards running simultaneously, but the speed was not as fast as the 9070xt alone (80 toks)!
I was shocked and wondered what the problem was, so I turned on lmstudio.
When I changed the ROCm runtime to the Vulkan runtime, it recognized both the 5080 and 9070xt and worked well. I thought, "Do I have to use LMStudio?"
Still, with the mindset of trying everything I could, I rebuilt llama using Vulkan again.
After setting the llama-server configuration to CUDA0,Vulkan0 and testing it again, the speed was not as fast as the 5080 alone, but it was about halfway between the 5080 and 9070xt (100~110 toks, Qwen3.6-35B-Q5, ctx 130k).
As I started adjusting the settings and testing with curl using the turboquant option, garbage values appeared...
I checked what was wrong, and it seems that turboquant has some problems with Vulkan. After changing it to q4_0, the issue was resolved.
The biggest goal of using a higher quantization version of the model I used to use with IQ3_S has been achieved.
Although I initially started building Local LLM on a Mac mini, it was too limited in terms of what I could do with 24GB. The toks didn't come out either, and since I was using it as my main machine, I couldn't allocate memory solely to the local LLM...
The biggest problem is electricity costs, but... with a 9950x3d + 9070xt + 5080, wouldn't that be about the same cost as a Claude Max 100 subscription? Haha, but what can I do? It was a fun experience.
One thing I realized through this build is that AI tends to discourage users by saying they can't do something or that it's difficult.
ChatGPT is especially bad at this. It explains things in a long-winded way, draws its own conclusions, and often ends with "I can tell you ~~ if you want". It makes me feel uncomfortable.Claude has a tendency to do the same, but not as severely.
Still, I'm proud of myself for overcoming the AI's limitations.
