I manage an in-house local LLM serving work.
RTX PRO 6000 * 2...
Qwen3.6-35b-A3b (MTP applied) - 200~300TPS
Gemma4-26b-A4b - 40TPS
This is about the performance.
I'm setting up vLLM and SGlang, but setting up Pro 6000 was quite tricky.
But somehow, after applying MTP, the speed is over 300TPS haha
Before, I used to tinker with A6000 and A6000ada with TP=2 or 4 to put up large language models.
Since a single 96Gb has some room, I'm looking elsewhere.
I'm setting up various MCPs on OpenWebUI and using them,
and 4 are running with Hermes Agent + Slack bot..
I'm also crawling news and updating Slack channels...
What else should I try? haha
Any recommendations?
