Why one-off vibe checks stop being useful
The moment Sonnet 5 became available, it was worth poking at. That’s the problem with model releases now. They show up fast, the demos look polished, and the first few prompts can make almost anything seem smarter than it really is. A model can sound sharp in a chat window and still stumble as soon as the work turns repetitive, visual, or annoyingly specific. That gap between “seems good” and “actually earns its place” is where most casual reactions fall apart.
In a Sonnet 5 review, the temptation is to react to the first burst of performance and call it a day. I get why people do it. The early outputs are often clean, quick, and persuasive in the way a good first date is persuasive. But shipping decisions are rarely based on charm. They’re based on whether the tool keeps doing the job when the novelty wears off and the task gets less polite.
A model that feels impressive for ten minutes can still be the wrong one to pay for every day.
Pricing makes that even trickier. Sonnet 5 doesn’t sit at the luxury end of the bill. It lands in the middle tier, which sounds reasonable until you remember that “reasonable” only matters if the model’s output holds up under real use. If you’re paying less than the flagship models, you still want a clear return: better code, better writing, better agent behavior, or at least fewer stupid mistakes. If the gains are fuzzy, the invoice starts looking less friendly.
That’s why quick taste tests have limited value. They tell you how a model behaves on a handful of prompts you happened to try at 11 p.m. They do not tell you how it performs across the same tasks tomorrow, or next week, or after the vendor quietly changes the weights and everyone pretends nothing happened. Repeatable evaluation is the real standard. If you can’t rerun the same tasks with the same inputs and compare the outputs, you’re mostly collecting impressions, not evidence.
A proper AI model benchmark gives those impressions some structure. It turns “this feels better” into something closer to a decision. That doesn’t mean spreadsheets solve everything. They don’t. Human judgment still matters, especially when a model is asked to write product specs, spin up prototypes, or manage work that drifts across tools and time zones. But a benchmark does stop you from trusting your memory, and memory is a slippery little liar when the latest release is shiny.
This article takes that problem in two practical directions. First, it shows how to build a personal benchmark that can be rerun every time a new model arrives. Then it moves into a different but related question: what happens when the model isn’t just answering prompts, but doing real work you have to manage from a phone?
How to build a benchmark you can run again
The trick is to stop treating evals like a one-night stand with a shiny model release. A good benchmark should feel a little boring in the best way possible: same inputs, same rubric, same tasks, every time. That way, when a new model shows up with a higher price tag and a nicer launch post, you’re not stuck relying on memory, mood, or the last demo that impressed you in the first five minutes.
A practical place to start is your own history. Open old session logs, past drafts, half-finished prototypes, support threads, the stuff you actually touched in real work. Those are much better raw materials than toy prompts about bananas or pirate captains or whatever the internet has decided is “an LLM challenge” this week. If you build software, pull examples from product specs, bug fixes, small refactors, browser tasks, and the kind of odd little follow-up work that autonomous coding agents tend to wander into when nobody is looking. If you write a lot, include editing and tone-shift tasks. If you manage teams, include message drafting and decision summaries. The point is to test the work you really do.
If you can’t run the same test again next month, you probably built a demo, not a benchmark.
Once you’ve got a pool of real tasks, freeze them. Keep the prompt text fixed. Keep the scoring rubric fixed. Keep the task list fixed for long enough to make comparisons meaningful. That sounds obvious until you’ve watched a benchmark drift into mush because someone “just tweaked” one task, then another, then a third, until the results mean almost nothing. You want a stable reference point, not a moving target with good intentions.
A balanced set of tasks usually works better than a single monster prompt. In practice, that might mean one PRD-writing task, one prototype-generation task, one agent-style task that checks whether the model can carry out a sequence without losing the plot, and one or two personality or communication checks. A model that writes clean product language might still break the moment it has to wire up a form. Another might be fine at code but sound oddly brittle or evasive in a customer-facing reply. Different tasks catch different failures, which is the whole point.
Building the scoring harness itself doesn’t need to become a weekend project. With Claude Code or a similar coding assistant, a simple HTML scoring page and a JSON export script can be knocked out in under an hour. Nothing fancy. Just a clean interface where you can open a generation, read the prompt, see the output, and record a score without hunting through files like an exhausted archaeologist. The JSON export matters because it keeps the results portable. You can compare runs later, sort by model, and stop relying on screenshots that disappear into the void of your desktop.
The manual scoring pass should stay quick. Run dozens of generations, then give each one a gut rating from one to five. Add a short note when something looks weird, promising, broken, or unexpectedly good. Those notes matter more than they first appear to. A score of 2 tells you very little on its own, but “missed the form constraint,” “prototype loaded but layout collapsed,” or “tone was sharp but too verbose” gives you something you can actually compare across releases. That is the difference between a vague feeling and a usable record.
For the benchmark described here, the test set covered 64 generations across five models. That’s enough to surface patterns without turning the whole exercise into a full-time job. You do not need a giant research lab to get signal. You need consistent inputs, a simple scoring habit, and enough volume to see whether one model is genuinely better or just got lucky on a single prompt.
A reusable benchmark also changes how you think about new releases. Instead of asking, “Did this model impress me today?” you can ask, “Did it beat the last one on the tasks I actually care about?” That question is a lot less glamorous. It’s also a lot harder to fool.
What the Sonnet 5 blind test revealed
Once the benchmark was fixed and the outputs were hidden behind anonymous labels, the picture got less tidy than a launch post would suggest. Sonnet 5 did well enough to stay in the conversation, but it didn’t stroll to the top just because it was new and priced like a sensible upgrade. That’s the awkward bit with model releases: a fresh name and a friendlier cost can make a system look like an obvious buy, until it has to sit beside older models on the same tasks and earn its keep line by line.
The automated judges made that process look cleaner than it was. In practice, the LLM-based scoring kept compressing results into a narrow band. Good answers and merely acceptable ones ended up too close together, which made the ranking feel polite in a way that was slightly suspicious. A model could be sloppy in one place and still get waved through because the judge was generous, or because it missed the part a human would notice in ten seconds. That’s the trouble with machine judges when the tasks involve layout, usability, or whether a prototype actually works. They can read the text, but they don’t always feel the breakage.
Human review caught the obvious problems faster. Broken prototypes showed up right away. So did layout constraints that had been ignored, even when the underlying text sounded fine. A response might have looked coherent on paper, then turned into a misaligned mess the moment it met a browser. The human pass didn’t need much persuasion there. It saw the issue, marked it down, moved on.
When the task has a visual or workflow edge, a careful human usually spots the failure before the judge does.
That gap mattered because the automated ranking and the human ranking were not just slightly different. They were far enough apart to change the story entirely. Sonnet 5 might have looked better under the blended or machine-weighted view, but once the human scores carried more weight, some of the older models climbed back into place. The final ordering changed in a way that felt less glamorous and more believable. That’s probably the right outcome, even if it’s less fun for anyone hoping the newest model will flatten every chart on contact.
Sonnet 4.6 also refused to become irrelevant. Even when it didn’t land at the very top overall, it remained the model that felt best for day-to-day agent work. The tone was better. The responses felt more usable. It was quicker to work with in the annoying, real-world sense that matters when you’re iterating all day and don’t want to wrestle the model into sounding like a person who has seen a spreadsheet before. For an agent that needs to keep moving, that kind of responsiveness can matter more than a small edge in an abstract score.
The task-by-task recommendations ended up being more useful than a single winner anyway. For PRD writing, the stronger models still had the edge. They handled structure, tradeoffs, and product language with more confidence, and that showed up in the drafts. For prototyping and chattier work, a different model fit better, especially when the job asked for quick back-and-forth rather than polished prose. Dense codebase navigation pushed in yet another direction. Some models handled broader search and file context more cleanly, which made them better at finding the right seams in a large project.
A few tasks, though, were too easy to say much about. If every model can sail through a prompt, the benchmark stops teaching you anything. Those are the ones that should get replaced. Harder tasks, messier constraints, and more realistic failure modes would tell you more than a pile of comfortable wins. Otherwise you end up measuring politeness instead of competence, and that’s a bad trade.
By the end of the blind test, the message was fairly plain. Sonnet 5 was viable, sometimes strong, and worth a look, but it wasn’t a universal upgrade. The better question wasn’t “is it newer?” It was “which jobs does it actually do better, and where do the older models still fit?” That turns out to be the useful part.
Managing coding agents from a phone
Once the comparison work is done, the job changes shape. You stop acting like an agent prompter, firing off one-off instructions and watching the next response. Instead, you start behaving more like an agent manager, which is a less glamorous title but a lot less annoying in practice. The difference is simple: prompting works when you’re sitting there babysitting every turn. Management works when the task can run for hours, get interrupted, and pick up again after lunch, a commute, or a missed notification.
That shift matters because coding agents are often asynchronous by nature. A model can draft a branch, open a file, hit a wall, ask for a decision, and then sit there waiting. If your setup only lives in a local terminal, the whole thing tends to stall the moment you step away. Put the same workflow on a cloud VPS, give it shell access, wire in Linear for task tracking, and let mobile control handle the rest, and suddenly the thing keeps moving while you’re nowhere near a desk. It’s less magical than people make it sound. It’s mostly just plumbing that doesn’t fall over the first time you leave the house.
The real trick isn’t making agents smarter every minute. It’s making them usable when you’re not watching them.
That is where mobile AI management becomes practical instead of theoretical. A phone isn’t the place where the work happens. It’s the place where you approve the next step, answer a question, or kill a task that’s gone weird. That sounds modest, but it changes the whole rhythm. A bug fix can start on a VPS, get assigned in Linear, continue through shell commands, then get checked from a phone while you’re in a cab or standing in line for coffee. No ceremony. No dramatic handoff.
Symphony fits into this setup as a behavior spec, not some mysterious orchestration layer with a shiny name and fog machine. It’s basically a detailed Markdown file that tells the agent how to behave, what to prioritize, what not to forget, and when to stop improvising. That may sound underwhelming, and honestly, that’s part of the appeal. Fancy orchestration can become a pile of hidden logic. A plain spec is easier to inspect, edit, and argue with when the agent does something stupid, which it still will from time to time.
Cost tracking needs the same plain treatment. Once tasks start burning through millions of tokens, the bill stops being a background annoyance and starts acting like a management signal. Some jobs might stay relatively small. Others can balloon past a hundred million tokens if the agent keeps exploring dead ends, re-reading context, or looping on a stubborn UI. Watching token cost per task gives you a way to compare work, spot waste, and decide when a job deserves a tighter scope. Without that, it’s easy to confuse activity with progress. Agents are very good at looking busy.
Skills files should stay short for the same reason. Long, sprawling instructions tend to accrete contradictions. One file says to be terse, another says to be explanatory, a third tells the agent to preserve legacy patterns even when those patterns are the bug. The result is predictable: confusion wrapped in confidence. Shorter skills files are easier to refresh, and refreshing them matters because models pick up stale habits quickly. If a rule hasn’t been revisited in a while, it might be guiding nothing except yesterday’s workflow.
When the UI gets fuzzy, better senses help. Screenshots can show layout problems that text alone misses. Visual diffs make it easier to see what changed after a patch. Video can reveal timing issues, loading states, or a button that appears only after a slow hover. Those extra inputs often keep an agent moving when the interface stops being neatly readable. Without them, it may just stare at a blank prompt and pretend the problem is philosophical.
Browser-enabled automation gets especially interesting when the task is narrow and slightly nerdy. One example is scanning collectible card listings for mispriced inventory, then comparing the listing against a reference market before acting. That’s a dull sentence, which is exactly why it works. A human can do it, sure, but a well-wired agent can do the repetitive passes faster, flag odd cases, and keep a watchlist open while you do something better with your afternoon. Not every use case needs to be grand. Sometimes the win is finding a card priced like a common bulk pull when it clearly isn’t.
That’s the operational side of agent work in a nutshell: fewer dramatic prompts, more routine management. The model may do the typing, but the system only starts to feel usable when you can steer it from a phone, keep an eye on cost, and tighten the instructions before they get mushy.
The practical playbook for teams shipping with AI
Once you’ve moved past one-off experiments and started running agents for real work, the rules get a lot less glamorous and a lot more useful. Hype fades fast. A model that looks brilliant in a demo can still fumble a layout, miss a constraint, or wander off in a browser tab like it’s late for lunch. That’s why repeatable benchmarks beat gut feel every time. They give you something stable to compare against when a new release drops, pricing changes, or a vendor suddenly claims the model is “best for coding” before the week is even over.
A good eval setup doesn’t need to be fancy. It needs to be stubborn. Keep the tasks frozen. Keep the rubric fixed. Run the same jobs again when the next model arrives. If the result improves on paper but falls apart on the things your team actually does, that’s not progress. That’s just a nicer screenshot.
The boring parts of AI operations are usually the parts that save you from expensive mistakes.
Human review still catches the stuff automated judging misses, especially when the work touches interfaces, screenshots, or multi-step flows. A model can write a clean explanation and still break the prototype. It can produce code that looks tidy in a diff and miss the one interaction that matters. A person scanning the output will notice those failures faster than a judge model that’s grading generously and calling it a day. That doesn’t mean you need to inspect everything forever. It means you should know where the machine is shaky and send a person there first.
For small businesses and solo operators, this setup can do a surprising amount of work. A single owner can use AI to draft product descriptions, prep support replies, sort leads, track content ideas, or keep tabs on operational noise without hiring a whole back office. The trick is not magic. It’s developer tools, a remote management habit, and a willingness to treat the system like software instead of a personality cult. If the workflow is set up cleanly, one person can supervise a lot more output than feels reasonable at first.
That gets even more useful in information-heavy jobs. Inventory management is a good example. So is personal monitoring, where the task is less about creativity and more about watching for changes, exceptions, and patterns over time. Agents can handle the repetitive checking, then hand back the cases that need a human decision. They’re also decent at the dull middle ground: scanning records, flagging mismatches, checking whether a listing went stale, or comparing yesterday’s numbers to today’s. Not glamorous work. Very useful work.
The lasting pattern is simple. Keep a stable eval process so you know what changed. Keep a lightweight management stack so you can steer the agent when it goes sideways. Then prune weak tasks, retire the easy ones that stopped telling you anything, and update the rubric before your benchmark turns into a museum exhibit. Models will keep changing. Your job is to make sure your process changes on purpose, not by accident.
