I finally switched from the troublesome hybrid 3070ti (8gb) + 1660ti (6gb) combination
to a pure 5060ti 16gb card.
When running Gemma4-26B-A4B (Q4_K_M GGUF) at a context of 32k, the old hybrid combination gave
about 12t/s, but with the standalone 5060ti it's showing around 30t/s. Impressive.
With about 1/3 of the total layers loaded into main memory, I'd say that's a usable speed worth sticking with.
(To fit all the layers into VRAM, you'd probably need something like a 3090 or another 90-series card with 24GB.)
What's a bit funny is that for a multimodal test, I cropped part of the Cyberpunk 2077 cover and asked it to identify it —
Gemma4-26B-A4B got the game name right on the first try, but Gemma4-E4B couldn't get it right and answered that it was likely Apex Legends. They were shown the same image but gave different answers. I guess there's no getting around having fewer parameters. Gemma4-E4B is said to have about 7.5B parameters.
By the way, when using a local LLM, having multimodal support is really quite convenient. If you get curious about something while doing other things, you can just capture that part and ask about it directly without having to type out an explanation, and when you want output in a certain format, you don't need to describe it or make a sample — you can just roughly sketch it in Paint and say "output it like this."
