생각보다 벤치마크 상으로는 잘 뽑힌 것 처럼 보이네요.
특히 Muse-Glimmer를 끼워넣어서 확인 사살을... ㅠㅠ
다들 내려받는 중인지 속도가 엄청 느리네요. unsloth 버전으로 받는 중인데 걸어놓고 자고 일어나면 받아져 있을지 모르겠습니다...
https://huggingface.co/Qwen/Qwen3.8-27B
Benchmark Results
Text Performance
| Qwen3.8-27B | Qwen3.6-27B | Qwen3.7-Plus | Muse Glimmer-30B | Opus4.6 Max |
|---|
Coding |
Agentic terminal coding Terminal Bench 2.1 (Terminus) | 73.0 | 63.4 | 64.0 | 51.7 | 78.2 |
Agentic coding SWE-bench Pro | 61.7 | 53.5 | 57.6 | 51.2 | 53.4 |
Repo-level code generation NL2Repo-Bench | 42.3 | 36.2 | 41.1 | -- | 47.6 |
Agentic coding DeepSWE 1.1 | 42.2 | 13.3 | 14.2 | -- | -- |
Software engineering QwenSWEBench | 79.0 | 49.3 | 59.2 | -- | 63.8 |
Agent |
Long-horizon office work CoWorkBench | 70.7 | 61.0 | 65.1 | -- | 68.2 |
Professional job tasks JobBench | 33.4 | 21.8 | 27.6 | -- | -- |
Frontier agentic tasks Agents' Last Exam | Pass@1 20.4 Score 42.9 | Pass@1 10.6 Score 27.3 | Pass@1 13.2 Score 33.6 | -- | -- |
General |
Instruction following IFBench | 79.5 | 69.1 | 79.1 | 77.0 | 62.5 |
Scientific reasoning GPQA Diamond | 89.2 | 87.8 | 90.3 | 83.5 | 91.3 |
Multidisciplinary reasoning HLE | 30.8 | 24.0 | 34.7 | 22.0 | 40.0 |
Competitive coding LiveCodeBench v6 | 90.3 | 83.9 | 89.6 | -- | 88.8 |
SWE-bench Pro: Except for Opus4.6 Max, which uses the officially reported score, all models are evaluated with the Claude Code harness at temp=1.0, top_p=0.95, and a 256K context window. Problematic tasks were corrected, and all baseline models were re-evaluated on the refined benchmark.
NL2Repo-Bench: Evaluated with the Claude Code harness. To prevent reward hacking, we disable Bash commands that attempt to access the specific repository, such as pip download, pip install, and git clone.
DeepSWE 1.1: Evaluated with the Claude Code harness at temp=1.0, top_p=0.95, and a 256K context window.
QwenSWEBench: In-house coding benchmark for evaluating models' software engineering capabilities. Evaluated with the Claude Code harness. Reporting avg@3 with an 8-hour timeout, max_tokens=32,768, temperature=1.0, and a 256K context window.
CoWorkBench: In-house cowork benchmark for evaluating long-horizon tasks across computer science, finance, law, medical, and other productivity domains.
HLE: Judged by GPT-4o.
The best result in each row is shown in bold.
Empty cells (--) indicate that results are not yet available or not applicable.
VL Performance
| Qwen3.8-27B | Qwen3.6-27B | Qwen3.7-Plus | Muse Glimmer-30B | Opus4.6 Max |
|---|
Agentic Multimodal Intelligence |
Computer use OSWorld-Verified | 84.3 | 63.9 | 73.3 | 65.9 | 72.7 |
Browser use WebArena-Verified | 64.8 | 48.8 | 55.3 | -- | -- |
Mobile use AndroidWorld | 81.9 | 70.3 | 81.0 | -- | 62.0 |
Application recreation RecreationBench | 47.1 | 29.8 | 30.2 | -- | -- |
Multimodal tool use ClawEval-MM | Pass@3 57.4 Average 56.9 | Pass@3 42.6 Average 50.4 | Pass@3 57.4 Average 60.1 | -- | Pass@3 52.5 Average 54.7 |
Multimodal software engineering SWE-MM | 38.6 | 25.7 | 30.0 | -- | 27.1 |
Visual web development Vision2Web | 62.9 | 45.0 | 42.1 | -- | -- |
General Multimodal Intelligence |
Visual math problem solving MathVision | Without CI 90.0 With CI 94.6 | Without CI 85.1 | Without CI 90.3 | -- | Without CI 65.5 |
General visual reasoning BabyVision | Without CI 65.7 With CI 85.6 | Without CI 28.9 | Without CI 64.7 With CI 70.4 | -- | Without CI 12.6 |
Scientific chart analysis CharXiv (RQ) | Without CI 83.7 With CI 90.2 | Without CI 78.4 | Without CI 85.8 With CI 85.9 | 78.8 | Without CI 66.0 |
Document intelligence OmniDocBench 1.5 | 91.1 | 89.4 | 91.4 | 75.8 | 86.6 |
Real-world perception RealWorldQA | 85.9 | 84.1 | 86.9 | -- | 73.9 |
Embodied intelligence ERQA | 65.5 | 62.5 | 69.8 | -- | 40.8 |
MathVision, BabyVision, and CharXiv (RQ): Where both settings are available, cells report “Without CI” and “With CI” separately; otherwise, only the available setting is shown. A small number of incorrect ground-truth annotations in MathVision and CharXiv (RQ) were corrected following manual verification, and all reported scores on those benchmarks were computed using the corrected annotations.
MathVision: Qwen3.8-27B is evaluated using the fixed prompt: “Please reason step by step, and put your final answer within \boxed{}.” For the remaining models, we report the higher score from two prompt variants—one with and one without the \boxed{} formatting requirement.
WebArena-Verified: Scores are computed with the official WebArena-Verified grader under the OSWorld scaffold.
RecreationBench: An in-house, long-horizon application-recreation benchmark designed to evaluate hybrid-agent capabilities across five platforms: desktop (Ubuntu, macOS, and Windows), mobile (Android), and the web.
ClawEval-MM: Scores are reported as “Pass@3 / average score.” Pass@3 is the percentage of tasks passed in at least one of three trials; the average score is the mean benchmark score across the three trials.
Vision2Web: Scores are averaged across the frontend, webpage, and website categories. Evaluations use the Claude Code harness and are judged by gpt-5.4-2026-03-05.
SWE-MM: Scores are evaluated on the Claude Code harness using the public dev split of SWE-bench Multimodal, with the modifications described in Appendix 8.3 of the Claude Opus 4.7 system card.
Empty cells (--) indicate that results are not yet available or not applicable.
▶ 원문 출처: https://huggingface.co/Qwen/Qwen3.8-27B
