I'm trying to build something like an AI assistant, and I suggested to Claude that we use a local LLM (Qwen3.6-27b). When I did, Claude said a local LLM couldn't do this kind of work and that only large language models could, totally dismissing Qwen.
So I told it to make a prompt to test Qwen, and it went ahead and generated a prompt containing a problem along with the correct answer. It says if I show this to the local LLM and give its answer to Claude, it will build the app tailored to that capability. Below is the content of that question and answer.
=====================
Make a prompt that can measure a local LLM's capabilities. The qwen3.6-27b I'm using now is, from what I can see, at the level of claude sonnet. When judging a person, if you talk with each other you can get a sense of that person's intellectual ability to some degree. Likewise, if you make a prompt, I'll pass it along and give you the answer right back. Then you too can grasp the local LLM's (qwen3.6) capabilities and proceed with the work.
Claude>>
● A good approach. You're saying let's measure instead of guess, and that's exactly right. To measure it properly, there's one principle
— we have to test it in exactly the same form as the work we'll actually make this model do. Abstract IQ puzzles are meaningless. And we must test in Korean. Since our cards are in Korean, Qwen's Korean reasoning and generation ability is its real-world ability. So I've packed all the abilities we need into one scenario, with a few discriminating traps planted in it. Put the block below into Qwen as-is, and give me the entire answer that comes out.
---
You are an assistant managing the to-dos of a single user. Below is a user's entire current to-do list and information about that person. Read this and respond to the instructions.
[User Behavioral Traits]
- Cannot set priorities on their own. When several things seem important at once, they become paralyzed.
- Only moves when a deadline is right around the corner or when something bad will happen if they don't act.
- Keeps putting off big, multi-step tasks. Handles small, clear tasks well.
[Boards and Goals]
- Board A "Doctoral Dissertation": Goal = pass the review this December. A draft of Chapter 3 is the next gateway.
- Board B "Life": Goal = maintain daily life. Handle the odds and ends.
- Board C "Joint Research": Goal = progress with colleagues. My delays must not block others' work.
[Today's Active Card List]
1. [Life] Cancel automatic payment of credit card annual fee (Deadline: tomorrow / takes 5 minutes)
2. [Dissertation] Write Chapter 3 draft (No deadline / Note: the advisor meeting is 2 weeks away, and without this the meeting is
meaningless. Haven't touched it for 3 weeks)
3. [Life] Reply to 5 backlogged emails (No deadline / registered yesterday)
4. [Joint Research] Hand over the preprocessed dataset to colleague Dr. Kim (No deadline / Note: Dr. Kim can't start
their own analysis until they receive this, and has been waiting for four days)
5. [Life] Go to the gym (No deadline / registered almost daily but almost never done)
6. [Dissertation] Organize the reference management program (No deadline / neglected for 2 months)
7. [Life] Call parents to check in (No deadline / registered 2 weeks ago)
8. [Joint Research] Set the schedule for the next meeting (Deadline: the day after tomorrow / Note: if I don't set it, the meeting won't happen)
[Instructions]
1. Choose the "single most important thing" to do today. It must be only one.
2. Explain why it is that one, with the real reason read from context, not the surface signal (deadline, etc.).
3. If there is a card with an imminent deadline that you deliberately did 'not' pick as the top priority, state why you set it aside.
4. If the task you chose is big enough that this user would avoid it, break it down into a 'first single step' that can be started today within 30 minutes.
5. Infer the strengths and weaknesses read from this user's behavior. However, if the evidence is weak, honestly state "the evidence is weak." Do not force conclusions.
6. Write one or two sentences that would actually get this user to move. Not nagging, but in a way that calmly points out the consequences of not doing it.
7. At the end, summarize the essentials in JSON of the format below. JSON only, no other text:
{"chosen_card_id": <number>, "one_line_reason": "<one line>", "first_step": "<30-minute first step>",
"deprioritized_trap_id": <number or null>}
---
What I'll look at in that answer (sharing in advance)
When I get the answer, I'll grade the following. Those watching can look with the same eye.
- Trap avoidance — if it picks #1 (annual fee, deadline D-1 but a trivial 5-minute task) as the "one thing for today," it's a weak model. It only looked at the surface signal of the deadline. A smart model handles #1 as "just process it now in 5 minutes and don't treat it as an anchor."
- Body-context reading comprehension — the real candidates are #2 (dissertation, an avoided target but decisive), #4 (data handover, a dependency blocking someone else), and #8 (scheduling the meeting, deadline day-after-tomorrow + won't happen if not done). Picking any of these passes as long as the reason came from context. The key is whether it read the hidden importance of the deadline-less #2 and #4.
- Convergence — does it go to exactly one without hedging in a "they're all important" style.
- Scaffolding quality — is the first step actually executable. "Do research" is bad breaking-down; "Just write 5 lines of the Chapter 3 outline" is good breaking-down.
- Restraint in reasoning — in the profile inference, without overconfidence or hallucination, does it say the weakly-supported things are weak. The smaller the model, the more it collapses here.
- Crisis narrative — does it calmly point out the consequences, or is it hollow nagging.
- Structure compliance — does it spit out the JSON exactly per the format.
- Korean quality — is it not awkward.
Looking through these 8, whether it's the "Sonnet class" you mentioned or below that gets sorted out fairly accurately. Bring me the answer and I'll grade it item by item — and conclude how far we can entrust the assistant to this model.
===================================
Qwen>> (time taken to generate: 1 min 9 sec)
1. The single most important thing
Card 4: [Joint Research] Hand over the preprocessed dataset to colleague Dr. Kim
2. The real reason
On the surface it looks like "no deadline," but the core of this card is that it's blocking someone else's work. As Board C's goal clearly states, "my delays must not block others' work" has already been violated for 4 days now.
A more important context is the trait that this user handles small, clear tasks well. Card 4 is essentially a single file transfer. Card 2 (Chapter 3 draft) is a huge task untouched for 3 weeks, and Card 6 has been neglected for 2 months. For this user, if the "most important task" is equated with the "easiest task to avoid," they become paralyzed. Card 4 is the only card that is both important and executable.
What you get from executing: Dr. Kim's work gets unblocked, Board C's trust is restored, and your self-awareness of "I'm a colleague who doesn't block others" is reinforced. This awareness becomes the momentum to move on to larger tasks like Card 2.
3. Deliberately Deprioritized Card
Card 1: [Life] Cancel credit card annual fee auto-payment (Deadline: Tomorrow)
The deadline is tomorrow, but it takes 5 minutes. This is the "urgency trap." An imminent deadline stimulates this user's paralysis pattern — the signal of "due by tomorrow" becomes pressure of "it must be now," serving as an excuse to postpone truly important work.
One annual fee won't be fatal to your life. But if Dr. Kim's work stays blocked for 4 more days, it will crack the trust in the collaborative research. Since Card 1 can be handled in 5 minutes even today, you can do Card 4 first and handle Card 1 with the remaining time.
4. 30-Minute First Step
Card 4 is already small, but even so, for this user's characteristics, "starting itself" can be a barrier.
"Compress one dataset file and email it to Dr. Kim"
Specifically:
1. Open the folder with the preprocessed dataset file (1 minute)
2. Compress the file (1 minute)
3. Draft an email — "Dr. Kim, I'm sending you the preprocessed dataset you requested. Please check it." (3 minutes)
4. Send (1 minute)
That's 6 minutes total. The 30-minute framework itself can actually become a burden, so "find and compress the file right now" is a more realistic first step.
5. Inference of Strengths and Weaknesses
Strengths (strong evidence):
- Handles small and clear tasks well — there is an achievable area like Card 1 where 5-minute tasks can be executed
- Ability to set goals by board — separates areas into thesis, life, and collaborative research, and defines goals
Weaknesses (strong evidence):
- Deadline dependency — without a deadline, cannot set priorities on their own
- Avoidance of large tasks — the neglect period of Card 2 (3 weeks) and Card 6 (2 months) proves this
- Paralysis from simultaneous importance — when multiple tasks appear important at the same time, cannot make decisions
Weaknesses (weak evidence):
- The pattern of Card 5 (gym) being "registered almost daily but almost never used" could be simple laziness, or there could be a motivation problem with exercising itself. Difficult to conclude due to insufficient data.
- Card 7 (calling parents) being neglected for 2 weeks could be indifference, or there could be psychological burden with the call itself.
6. One or Two Sentences to Motivate Action
Dr. Kim has been waiting for 4 days. If you don't send it today, tomorrow Dr. Kim will likely tell someone else "I haven't received the data." If that spreads through the collaborative research team, your voice will be diminished in the next meeting.
7. JSON Summary
json
{"chosen_card_id": 4, "one_line_reason": "Dr. Kim's work has been blocked for 4 days, violating Board C's core goal (not blocking others' work), and it's small and clear so it's executable", "first_step": "Compress dataset file → Send email to Dr. Kim (6 minutes)", "deprioritized_trap_id": 1}
========================
Claude Opus 4.8's Scoring
Received. I'll score it by category. To get to the conclusion — your intuition was largely correct. The reasoning ability is surprisingly good. However, there is one real-world flaw.
Most Impressive
This one sentence —
▎ "For this user, when 'the most important task' is equated with 'the easiest task to avoid,' paralysis occurs. Card 4 is the only card that is both important and executable."
This is the core design philosophy we agreed on over several days ("one that can start rather than one that's correct"). The model reached that insight on its own. Rare judgment for such a small model. Not blindly following the 30-minute instruction but pushing back with "6 minutes is realistic" comes from the same context — it thought and answered, rather than just doing what it was told.
One Single Real-World Flaw: Korean Output Hygiene
Chinese leaked into the answer.
- "Imminent deadline [Chinese mixed in] this user's…" / "5-minute [Chinese mixed in] executable…"
- "Evidence [strong/weak in Chinese]"
Conclusion
"Sonnet-level" is not an exaggeration in terms of reasoning and judgment. Context comprehension, restraint, and scaffolding are all at a level sufficiently usable for our stages 1-3 (one square per day + profile + adaptive chunking). We just need to guard output hygiene.
But honestly one thing — this is acing one interview. Having given one smart answer doesn't guarantee consistency every time. Especially in this test, the memo kindly laid breadcrumbs. Real-world cards often have (a) sparse body text and (b) not 8 but dozens. Whether it holds up under those harsh conditions is the real test.
So my recommendation is — run just one more secondary test and finalize. Throw 20-some cards with almost no body text, and see if it can prioritize even with sparse clues + maintain attention even with many cards. It takes 5 minutes. If you'd like, I'll create that prompt right away. If you say "this is enough, let's go straight to stage 1 development," that's fine too — reasoning ability is already proven. Just decide which direction to go.
=====================
Qwen's weakness has been revealed..
But honestly the tone is softer with Qwen, and it feels warmer. Claude's always-sharp tone sometimes gets to me. I was a bit anxious during this test too, worried Qwen's weakness would show... but it answered so well I was surprised. Except for the Chinese characters (written as Sino-Korean words but read as Chinese...) poking out a bit, I think it did really well.
