I already paid for Codex, Gemini, and Claude. I wanted one place to give them work, let them challenge each other, and bring back something I could actually trust, without spending my day as the human API between three chat windows.
I built TeamLoop with Codex to do that from one conversation. The tests on this page helped decide which model should build, review, investigate, or stay out of a particular job.
So what actually runs?October 12 update: 15 dated changes
Quick reality check: this is one practitioner’s structured experiment using synthetic tasks, isolated runs, browser checks, blind scoring, and every failure I could preserve. It is not a scientific benchmark, an employer evaluation, or a universal model ranking. This was my personal experiment, not an official Glean benchmark.
October 26: Argon current-work trials and the watch-log repair.
October results and nine new runnable examples: original replay, Claude recovery, and Haiku versus 6.1.
My subscriptions gave me access to a lot of capability, spread across separate apps and usage limits. I wanted Codex to coordinate that capability while I kept one conversation. Spending my weekends testing models was how I worked out the assignments. It was not supposed to become the entire personality of this project.
Where the idea came from: Claudex Loop got me thinking about having Claude and Codex challenge each other's work. I wanted to bring Gemini into that loop, then add smaller and specialized models when they earned a place. So I built a separate system with Codex: TeamLoop. Claudex Loop supplied the starting workflow idea. TeamLoop has its own runners, context handling, routing rules, and verification.
Independent reviews can run alongside each other. A fast model can handle a bounded draft while the lead works elsewhere. The handoff has to earn its time back; adding another model can also make a small job slower. If all I added was coordination, I invented a meeting.
I can bring different strengths to one project: implementation, design direction, deep architecture, or a fresh attempt to break the idea. A demanding job can get a stronger specialist and more time.
The builder gets another set of eyes. Then Codex checks the criticism against the brief, the source, and actual tests. Browser checks, calculations, and reproducible failures carry more weight than confident agreement. Two models confidently agreeing is still not a test.
Tokens are the chunks of material a model reads and writes. Local tools can extract relevant content and summarize passing checks before a model reads it. Specialists get the selected evidence and assignment, so expensive context goes toward the difficult part.
Suitable jobs can use the allowances attached to my different subscriptions. That gives me more options when one is tight. Each provider still has its own limits, and optional API models have their own access and billing rules.
I explain the job in Codex. The lead chooses the useful workers, passes context, and brings the findings back. I keep the desktop interface and the decision, without becoming the person copying the same brief into three chat windows.
And yes, I get to red-team everything. A plan, a design, an explanation, an architecture, a finished build. Red-teaming means deliberately asking where it breaks, what it assumes, and what the builder missed. TeamLoop makes that a repeatable part of the work. The depth matches the stakes: routine checks can stay with Codex, while consequential work gets an independent critic or a wider review.
The tests gave these assignments an evidence base. They also showed me where delegation was fragile, where a specialist helped, and where the handoff was more trouble than it was worth.
These are the operating benefits the system is designed to pursue. The field study informs task fit and verification choices; it does not establish a universal speedup or a fixed percentage of tokens saved.
A friend asked me this several times before I realized I had explained nearly everything except what the software actually was. Fair criticism. Here is the actual answer.
TeamLoop is a custom orchestration harness running on my Windows computer. It consists of a Codex skill, JavaScript programs run by Node.js, JSON routing rules, typed execution plans, and local files that preserve context and results. I operate it through the Codex desktop app.
A harness is the code and rules around the models that organize their work. Node.js is the program that executes the JavaScript outside a web browser. My computer runs the coordination software; the selected AI models run in their providers' clouds.
The lead works out the requirements, chooses the useful specialists, and tells me who is doing what.
A typed plan records dependencies, failure rules, and the concurrency cap. Local tools prepare the selected evidence.
The runner starts only typed review stages, reaches the selected providers, and records each stage lifecycle. The lead only starts the routes the job needs.
Results and logs come back to local files. Codex checks findings, applies accepted changes, and returns the answer and artifacts in our conversation.
The runners do not synchronize three shared brains. Each worker gets an explicit assignment and selected evidence. It does not automatically see the other models' chats or the whole Codex conversation. Keeping the context consistent is part of the lead's job.
The current TeamLoop core has no database server, MCP server, or separate hosted dashboard. It is skill instructions plus executable local code and files. It lives outside the repository being worked on, so the same system can help across projects. This website explains the system; it does not run the orchestration.
SKILL.md, route-team.mjs, and JSON policies describing model roles, tools, and execution targets.team-loop.mjs starts subprocesses such as claude -p, the Antigravity CLI, or codex exec. Provider adapters translate the same assignment into each program's expected format. Optional NVIDIA reviewers use an explicitly selected API route.config.json selects installed executables and the state folder. state/packets holds task snapshots; state/runs holds individual review results; state/plan-runs holds plan snapshots, status, and lifecycle events. CONTEXT.md and DECISIONS.md preserve accepted task knowledge.Implementation described from the installed skill, routing script, provider adapters, and local record layout inspected on September 21, 2026. Provider access and available models can change.
Run logs are great after the fact. I wanted one glance to tell me who is working, who is stuck, and who is waiting on me. So every agent session on my computer now shows up as a little 8-bit character, working away in a pixel-art arcade.
The Dream Machine is TeamLoop Live: a local, read-only display of my own agent sessions. Codex tasks, Claude Code sessions and TeamLoop reviewers each get a procedurally drawn 8-bit character, and each crew gets its own arcade room.
There are no sprite files. Every character compiles from a DNA recipe, a short list of traits that code turns into pixels. There are 129 recipes and 125 public characters, and any name you type compiles into a brand new one. Same name, same character, every time.
Second edition / 68 pagesThe ManualAn NES-style instruction booklet: the case study, the process, every character, the rebuilt rooms, the code, and Character Select, where you type any name and the machine builds a new challenger.Read the Manual
The Manual lives here at /dream-machine/ and at its original home, starforge-dream-machine.netlify.app. Its code is on GitHub.
The benchmark chapters below are frozen records of what happened. This is the living part: the current TeamLoop assignment map after the latest evidence.
The operator-selected Codex lead: scopes, owns context, implements, integrates, verifies and adjudicates. Handles evidence preparation and the predeclared subscription availability fallback. GPT-6 Sol is removed from active routing by operator decision; historical results remain unchanged. Other specialists stay task-selected.
Tool-enabled multimodal production, ambitious web builds, asset research, and difficult end-to-end work. Used after cheaper tools prepare the evidence because this thing eats well.
The active bounded builder uses the exact gpt-6.1-terra pin at Medium for operational tools, customer-facing interfaces and scoped implementation. 6.1 Sol remains lead and verifier. Two matching text smokes do not replace historical GPT-5.6 Terra grades; included subscription savings remain unverified.
Exploratory drafts and bounded alternatives under explicit acceptance criteria. Useful breadth, followed by lead-owned QA.
Ambitious interface/design work in an isolated worktree, with selected-file Read/Write/Edit, incremental saves, a small final receipt and the full 45-minute allowance. GPT-6.1 Sol owns scope, browser QA, integration and promotion. No automatic fallback.
Pinned independent critique for language, requirements, accessibility, design coherence and selected complex synthesis. GPT-6.1 Sol verifies findings and retains final authority.
Pinned rare architecture, product vision and editorial synthesis. Selected design ownership uses the separate durable-file design lane, not a whole-HTML reviewer response.
Completeness, edge states and evidence synthesis. Native subscription video is preferred only when its transport is separately verified; the text-only headless runner does not watch footage.
Disposable breadth, alternative directions, and fast first passes. Cheap enough to explore, never exempt from verification.
Public or synthetic source-heavy engineering review, lifecycle and concurrency analysis, and architecture-versus-implementation checks. High is the default. Codex verifies every claim.
One bounded patch with a fixed test and cheap restart. It is not an automatic route and should not inherit a whole build halfway through.
Interesting direction reconstruction, but not reliable enough for production visual judgment. Available for a genuinely different tie-breaker.
A different model family for rare architecture disputes. The tested delivery did not earn a normal lane.
The tested GLM endpoint was retired. Scripted Kimi subscription use remains blocked pending provider permission; this is an automation boundary, not a model-quality judgment.
Experimental creative comparison only when explicitly selected. Its pinned identity must match and subscription access must work with usage credits off. No retry or substitution. Local guard tests passed; creative quality and provider availability remain unproven.
Tools are part of the team without pretending to be models. Local scripts extract, convert, diff, lint, test, and package evidence first. The Forge Worker handles private local preparation when appropriate. A model only gets work that still requires judgment.
Availability is a route condition, not a philosophical crisis. If a specialist is unavailable, the job returns to Sol or a named fallback. TeamLoop never silently swaps in paid API usage or sends private material to an external specialist.
Opus 5.5 High scored 91/100; Argon Medium scored 69/100 on the original model-comparison console brief. Both completed through the same constrained host file-action contract. Opus was better at evidence-based decision support; Argon invented audit details and left score-bound and verdict-label defects. This new workflow does not replace the historical one-shot grades or promote Argon to default design work.
| Configuration | Accepted primary | Frozen HTML | Masked quality /100 |
|---|---|---|---|
| Opus 5.5 High / revision 3 | Completed; 3 native calls, 3 host turns, 4 actions | 45,668 bytes | 91 |
| Argon Medium / revision 5 | Completed; 4 unique native calls, 4 host turns, 8 actions | 76,060 bytes | 69 |
The original brief, seed data and six-category rubric stayed fixed. Each primary had 45 minutes total, up to 32 intentional host turns and four selected Read/Write/Edit actions per turn. Native calls used the remaining total deadline; the host replayed the complete action/result transcript. There was no native operational tool, Compute, OS, network, browser, MCP, delegation or adaptive lead QA feedback. This is explicit host orchestration, not CLI session continuity or a shipped automatic production route. Native effort labels are not equal compute.
The quality difference: Opus says missing evidence details were not supplied. Argon invents validated gate logs, a specific accessibility finding and a category breakdown absent from the brief. Its verdict labels still name the original pair after selection changes. After the score lock, root confirmed that five inputs of 100 display 500 / 100 without invalid flags; the locked 69 was unchanged. Opus's invalid numeric draft remained a smaller clarity issue. Scores used the original 30/20/15/15/10/10 rubric, random artifact IDs and a lock before identity join, with source-backed lead adjudication. Download both category-score rows.
What was actually checked: startup, fictional failed-candidate exclusion, live score/evidence separation, filtering, an Enter-key swap with notes/assessment association, and comparison changes worked in the observed flows. Viewport checks reported no horizontal page overflow at 1440x900 and 390x844. Both apps reported a download, but actual downloaded JSON contents remain unverified after browser observation limits; no blocked URL was bypassed. Full keyboard, screen-reader and cross-browser coverage remain unverified. A working core is not a blanket browser pass.
Zero-call setup refusals, an Argon diagnostic-preview comparison bug and an Opus passive-metadata observer abort remain separate infrastructure records. Argon later encountered a real transient audit-file double-link and a concurrent host-check race. Narrow study-local physical-audit, FIFO and drain corrections passed 37 offline tests plus 100 synthetic writes with concurrent checks. Its original complete actions were mechanically replayed without quality regeneration or QA feedback. The original 45-minute clock included host-repair downtime, and Argon completed before that original deadline. Guards were not weakened. One Opus observer-abort call has unknown final usage and is outside the accepted-primary count; replayed Argon source calls are counted once. Wall-clock speed ranking across these interruptions would mislead.
Claude's rounded session meter moved 9% to 12%, with weekly usage unchanged at 38%; those shared meters and provider token counts do not establish normalized quota cost, dollars billed or subscription savings. Policy 1.14.0, active Terra 6.1, 6.1 Sol as lead and the pinned Opus/Sonnet roles remain unchanged. Existing Argon review consideration does not become default design ownership. Full method and evidence record.
Use synthetic inputs only. These original artifacts preserve the observed defects. Neither is a production tool, and downloaded-file contents remain unverified for both.
Opus 5.5 High
Core observed controls worked and missing audit details stay explicitly unknown. An invalid numeric draft remains a clarity issue. No exhaustive accessibility or actual export-file verification is claimed.
Inspect unchanged buildArgon Medium
Invented audit details, unsupported category decomposition, stale verdict labels and a 500/100 score-bound defect remain visible. These were not repaired or hidden before publication.
Inspect unchanged buildOne practical configuration per contender, one task and AI-assisted scores are field evidence, not a universal model ranking. This was my personal experiment, not an official Glean benchmark.
Active bounded Terra work now selects GPT-6.1 Terra Medium under policy 1.14.0. GPT-6.1 Sol Medium remains lead. The operator accepted this version swap with included subscription savings explicitly unverified. Pinned Opus 5.5 design/architecture and Sonnet 5.5 critique roles stay unchanged.
| Requested configuration | Matched text smoke | Elapsed |
|---|---|---|
| GPT-5.6 Terra Medium | Same two seeded defects; correct minimal recommendations | 9.134s |
| GPT-6.1 Terra Medium | Same two seeded defects; correct minimal recommendations | 6.656s |
Both October 27 calls used the same small tool-free source packet and 90-second deadline. Exact requested pins were recorded, but the backend did not separately return an actual model identity. Two observations establish a small compatible path, not broad quality improvement, a latency distribution, or tested image/computer-use capability. Historical GPT-5.6 Terra scores, dated comparisons and example apps remain labeled as originally tested.
Credit prices are not a subscription-saving measurement. The official Standard credit rates, checked October 28, are 50 / 5 / 300 credits per million input / cached-input / output tokens for 5.6 Terra, versus 25 / 1.25 / 125 for 6.1 Terra. Lower equal-token credit rates do not prove the same change in included subscription usage. The rounded weekly meter stayed at 50% before and after each tiny call and cannot resolve either call or establish a saving ratio. No purchase, overflow or paid API fallback was used.
What changes operationally: active local model selections and bounded-builder routing use gpt-6.1-terra at Medium; 6.1 Sol still scopes, integrates and verifies. The portable toolkit resolves the adopter's explicit provider/model configuration and does not install personal defaults. Dated change record and operating snapshot. Unfinished Argon/Opus design work is outside this update.
Once TeamLoop had a purpose, I needed evidence for its assignments. Raw capability mattered, obviously, but task fit, clean delivery, verification cost, and whether I could recover from a failure mattered just as much. Here is what the testing changed.
The strongest default builder was not always the strongest architect, critic, visual thinker, or fast source of alternatives. Once I looked at the work instead of the logos, routing by task made way more sense than crowning one brand.
A model can make exceptional work when it finishes and still be a rough operational default. The picture changed as soon as failed attempts and broken browser behavior counted too.
Timeouts, wrapper failures, transport limits, and context exhaustion can all look like model weakness. I kept those failures and separated the model, provider, harness, and verification problems when the evidence allowed it.
Several interfaces looked polished and still had dead controls, runtime exceptions, or resets that did not actually reset. Browser checks caught things the screenshots and very confident prose did not.
I did not pour every result into one cursed leaderboard. Each study answered a narrower routing question and kept its own date, conditions, failures, and limits.
Eight configurations, three working-interface briefs, blind quality scoring, and browser gates.
September 4, 2026Five configurations, three intentionally dense builds, and separate quality versus strict delivery scores.
September 5, 2026Fairer ceilings, frozen primaries, neutral repair packets, and deterministic browser checks.
September 5, 2026Visual criticism, an engineering artifact, and a systems audit against incumbents.
September 7, 2026A surgical repair, a bounded UI job, and the broad-repository task that found its ceiling.
September 11-12, 2026Engineering audits, real repair, giant context, browser delivery, caching, and orchestration economics.
October 24, 202629 short attempts, corrected oracles and candidate findings; separate from the original roster tests.
October 24-25, 2026All 65 scheduled primary outcomes, separate repairs and later CLI failure qualification.
October 25, 2026Eight captured failure streams, native continuation limits, delivered Sonnet and recovered Opus; no matched quality ranking.
October 25, 2026Three deliveries, zero complete functional passes; scores 70, 49 and 64 under a separate tool contract.
October 25, 2026Four primaries with matched useful tools: estimator 26 versus 94; comparison 87 versus 95. No Haiku promotion.
October 26, 202615 saved artifacts rescored with workflow and delivery limits preserved; 6.1 stays lead, Opus and Sonnet retain deliberate specialist roles.
October 26, 2026Nine frozen text reviews, six current-work attempts and a verified final-watch repair. Argon remains a candidate; 6.1 leads. NVIDIA failure is separate transport evidence.
October 28, 2026Two completed original-console artifacts through a separate matched host workflow: Opus 91 and Argon 69; evidence defects, infrastructure recoveries and incomplete export verification stay explicit.
Every comparison started with the same bounded brief. I isolated the runs, scored them against written criteria, and checked the finished work in a real browser. If something needed repair or salvage, I recorded that instead of quietly swapping in the fixed version and pretending we nailed it the first time.
Short functional builds established a baseline. Deep-end builds tested ceiling. Corrected assisted runs tested repair. Specialist, Spark, and DeepSeek placements tested narrowly claimed jobs.
Completed-work quality asks, “How good was it when it came back?” Reliability-adjusted value asks, “What did I actually get from every run I scheduled?” I need both answers. They are not interchangeable.
For the original September cohorts, models made artifacts and contributed critique, but they did not vote on the winner. I made the final calls from the requirements, evidence, and browser results. The October 24-25 replay below uses separately labeled AI-assisted masked scoring; I have not personally scored it.
I shuffled completed artifacts behind random IDs. The scoring view showed the task, screenshots, browser evidence, and blind ID, but not the provider or model. I did not open the identity map until the score file was written.
The scoring interface hid model names, provider labels, aggregate results, and every other model’s score. Every artifact inside a task set used the same frozen rubric.
Incomplete attempts never entered the blind gallery. Their runner status already existed before scoring, and I classified the failure from logs afterward. A timeout means no artifact was scored. It does not mean the model’s ideas were bad.
Every configuration inside a task set got the same self-contained brief, seed data, observable hooks, and delivery constraints. These summaries include the central question and what success had to look like without publishing unrelated implementation artifacts.
Question: Where should an AI adoption lead intervene this week?
Question: What is the customer paying for, and what changed?
Question: What did each model add, and is it worth routing similar work there again?
Question: What is the fictional fleet doing, what may break, and which intervention improves the outcome?
Question: How does a plausible AI transformation fail through interacting human and technical dynamics?
Question: Which strange-but-plausible future is worth building toward, and what would people do inside it?
Bounded trial, 100 points: functional correctness 30; visual hierarchy and readability 20; interaction clarity 15; responsive accessibility 15; originality and domain fit 10; implementation quality 10.
Long-running trial, 100 points: functional system integrity 25; conceptual depth 20; ideation and originality 15; information architecture 15; interaction design 10; visual ambition and execution 10; earned complexity 5.
Automated checks were gates and evidence, not the whole score. Quality scoring excluded speed. A failed or invalid attempt received zero only in reliability-adjusted or strict operational calculations.
Eight configurations each got the same three functional interface tasks. I put every raw outcome beside the rounded summary because three observations are three observations. Giving the mean a decimal does not magically turn it into a law of nature.
On smaller screens, swipe the tables sideways. With a keyboard, Tab to a table and use the left and right arrow keys.
| Configuration | Three task outcomes | Completed | Completed-work mean | Reliability-adjusted | Mean completed time |
|---|---|---|---|---|---|
| Codex Sol | 97 · 93 · 93 | 3 / 3 | 94 | 94 | 1.7 min |
| Codex Terra | 92 · 96 · 83 | 3 / 3 | 90 | 90 | 1.4 min |
| Gemini Flash High | 92 · 89 · 88 | 3 / 3 | 90 | 90 | 2.3 min |
| Gemini Flash Medium | 83 · 91 · 88 | 3 / 3 | 87 | 87 | 0.9 min |
| Codex Luna | 93 · 87 · 79 | 3 / 3 | 86 | 86 | 1.5 min |
| Claude Sonnet | 85 · 84 · 86 | 3 / 3 | 85 | 85 | 1.9 min |
| Claude Opus | 83 · timeout · 78 | 2 / 3 | 81 | 54 | 1.7 min |
| Gemini Pro High | timeout · 79 · timeout | 1 / 3 | 79 | 26 | 1.7 min |
Outcome order is adoption operations, transactional estimator, then model comparison. Scores are task-specific judgments out of 100. Summary values are rounded to whole points. I recorded speed, but it was not part of the quality score.
sum(scored completed attempts) ÷ count(scored completed attempts)This answers, “When this configuration returned something I could score, how good was it on these tasks?”
sum(scores, with incomplete attempts = 0) ÷ 3 scheduled attemptsThis answers, “How much scored work came back from everything I scheduled?” Non-delivery counts because operationally, of course it does.
n = 3 tasks per configurationThese numbers describe this trial. They do not estimate general model performance, and a one-point difference does not prove one model is better.
Five configurations got the same three intentionally dense builds with twelve-minute ceilings. The strict operational score counted only clean deliveries that also passed the browser startup check. Salvages and browser failures got zero in that metric, even when the underlying artifact had real value.
| Configuration | Three task outcomes | Completed | Completed-work mean | Strict operational | Practical read |
|---|---|---|---|---|---|
| Gemini Flash Medium | 90 · 53* · 88 | 3 / 3 | 77 | 59 (178÷3) | Best strict return here; one delivered artifact had a broken operating core. |
| Codex Sol | 56* · 80 · 90 | 3 / 3 | 75 | 57 (170÷3) | Strong lead behavior and one category win; one serious browser failure prevented a clean sweep. |
| Gemini Flash High | 89 · 87† · failed | 2 / 3 | 88 | 30 (89÷3) | Highest completed-work ceiling; one salvage and one failed delivery reduced strict value. |
| Claude Sonnet | failed · 84† · 89 | 2 / 3 | 87 | 30 (89÷3) | Excellent surviving design and framing; one salvage and one timeout made one-shot delivery fragile. |
| Claude Opus | 81† · 80 · 52†* | 3 / 3 | 71 | 27 (80÷3) | All three artifacts were scoreable, but only one arrived clean and passed the browser gate. |
Outcome order is agent fleet, failure strategy, then futures exploration. * Browser issue: zero in strict operational score. † Salvaged from a wrapper or transport failure: scored for completed-work quality, zero in strict operational score. Rounded summaries describe this trial. They are not statistically stable.
sum(score only when delivery = clean AND browser = pass) ÷ 3Failed, salvaged, or browser-failing attempts contribute zero. This measures the delivery contract I tested, not pure model intelligence.
81 salvaged → 0 · 80 clean/pass → 80 · 52 salvaged/browser issue → 0; 80 ÷ 3 = 26.7 → displayed 27This is why three scoreable Opus artifacts can coexist with a strict operational score of 27.
delivery status + wrapper status + browser evidenceI did not treat transport and harness failures as judgments about the content. The attempt appendix shows the attribution used for every row.
The first deep test exposed failures I had caused with brittle wrappers and unfair ceilings. I fixed the preventable parts, kept every primary attempt frozen, and gave failed browser builds a separately labeled repair pass. That measured whether the models could finish under fairer operating conditions without quietly rewriting history.
All twelve completed. Four passed browser integrity on the first delivery.
Seven completed and three passed. Every successful repair was in the design-direction task.
That total includes separately labeled repairs. It does not replace the four primary passes.
| Configuration | Primary completed | Primary browser pass | Mean primary time | Mean primary artifact | Mean reported tokens | Repair result |
|---|---|---|---|---|---|---|
| Gemini Flash High | 4 / 4 | 2 / 4 | 4.21 min | 109.0 KiB | 133,686 | 2 attempted, 2 completed, 1 passed |
| Gemini Flash Medium | 4 / 4 | 0 / 4 | 2.56 min | 108.6 KiB | 65,779 | 4 attempted, 3 completed, 1 passed |
| Codex Sol | 4 / 4 | 2 / 4 | 7.13 min | 68.8 KiB | 35,602 | 2 attempted, 2 completed, 1 passed |
Provider token reports are included as operational observations, not directly comparable cost units. Different providers count and cache context differently.
| Configuration | Completed primaries | Primary browser pass | Complete repairs | Passed repairs | Practical read |
|---|---|---|---|---|---|
| Claude Sonnet | 1 / 4 | 0 / 4 | 0 / 1 | 0 / 1 | Strong ideas did not become dependable delivery in this lane. |
| Claude Opus | 3 / 4 | 0 / 4 | 3 / 3 | 2 / 3 | Fragile first delivery, but the strongest repair recovery in the matched round. |
Muse, GLM-5.2, and Nemotron 3 Ultra did not need the whole general test suite. They got tasks built around their claimed specialties, then faced the TeamLoop models already doing those jobs.
A controlled campaign A/B set tested whether a reviewer could reconstruct the intended direction, notice concrete defects, and make the same exact selections as the reference judgment.
| Configuration | Input | Time | Reported tokens | Exact selection result | Score |
|---|---|---|---|---|---|
| Muse | Contact sheet, run 1 | 18.6 sec | 10,405 | Four ties | Not scored |
| Muse | Contact sheet, run 2 | 14.8 sec | 9,807 | Four ties | Not scored |
| Claude Sonnet | Contact sheet salvage | 78.5 sec | 15,564 | Tie leaning B, B, B, tie | 55 |
| Muse | Eight full-resolution WebP files | 23.8 sec | 21,153 | B, tie, tie, tie | 43 |
| Claude Sonnet | Eight full-resolution WebP files | 79.0 sec | 25,888 | Four ties | 34 |
Expected selections were B, B, B, B. Muse reconstructed the intended direction well, scoring 23 of 25 there, but a false “cream paper” observation drove three wrong or tied choices. An eight-PNG request failed with HTTP 500 after 5.3 seconds at about 12.4 MiB. Gemini Flash High was unavailable through the tested text-only transport, which was a transport limitation, not a visual-quality result. No production visual winner was declared.
| Configuration | Time | Delivery | Verification | Attribution |
|---|---|---|---|---|
| Codex Sol | 5:03.5 | Complete envelope, broken JavaScript | Three-literal mechanical salvage passed 15 / 15 DOM simulation checks | Model-delivered artifact failed; salvage recorded separately |
| GLM-5.2 | 0.5 sec | No artifact | Not testable | Endpoint retired with HTTP 410 |
| Claude Opus | 15:03.4 | No artifact | Not testable | Timeout at the configured ceiling |
No production engineering winner. Sol produced recoverable work, but the primary did not pass. GLM was operationally unavailable. Opus did not deliver inside the test window.
| Configuration | Time | Score | Roster consequence |
|---|---|---|---|
| Claude Opus | 6:06.7 | 89 | Earned the scarce deep-architecture escalation. |
| Gemini Flash High | 2:51 | 76 | Stayed the faster everyday completeness and breadth auditor. |
| Nemotron 3 Ultra | 1:42.8 | 8 | Different-family wildcard only. A second NVIDIA attempt returned HTTP 503 after 45.1 seconds. |
A score makes more sense when you can click the thing, try the controls, and see what the rubric was reacting to. These are original test artifacts, not cleaned-up recreations. Five show what worked. One shows exactly why a polished screenshot was never enough.
Codex Sol · bounded trial
The strongest decision-first experience in the opening set. Try the department and risk filters, open the details, and follow the recommendation into an actionable brief.
Run the build
Codex Terra · bounded trial
The clearest customer-facing pricing story. Change quantities, add or remove work, and watch the selective tax, overhead, profit, and total recalculate.
Run the build
Codex Sol · bounded trial
The best evidence-versus-judgment framing, with one seeded-content assertion still failing. Select two models, edit the scoring, swap sides, and inspect the export.
Run the build
Gemini Flash Medium · long-running trial
The strongest functioning mission simulator. Play or step the clock, change routing policy, apply an intervention, and inspect how state and evidence move together.
Run the build
Gemini Flash High · long-running trial
The wrapper failed, the complete HTML was recoverable, and the browser checks passed. Compose a scenario, alter stressors, run the simulation, and inspect mitigations and adversary lenses.
Run the build
Codex Sol · long-running trial
This is the reason browser testing became part of the method. The control room looks credible. The JavaScript had other plans. Runtime exceptions leave the central operating surface empty. Open it beside the working fleet simulator.
Inspect the failureWhy there is no Futures Lab here: those artifacts used project-specific branding from the original test. I kept them out of this neutral public edition instead of rewriting historical output and pretending it was what the model originally delivered.
I tested GPT-5.3-Codex-Spark later as a possible specialist. It passed a real surgical repair and finished most of a bounded UI correction job. Then I gave it a broad repository scout, watched it run for nearly nine minutes, exhaust context during compaction, and return no report. Point taken.
“Separate limits” is not the same thing as “free throughput.” The extra capacity only matters if the job can finish before the context and short-window ceilings become the real problem.
The useful move was not replacing the sustained builder. It was defining a narrow reserve lane where partial work is cheap to throw away or mechanically easy to resume.
The new model got the job it claimed to want: source-heavy engineering review, a real historical repair, giant-context retrieval, browser artifact delivery, caching, and agent orchestration. It earned a roster spot, but only after the test drew a very sharp fence around that spot.
One point behind Opus at 89 and twelve ahead of Flash High at 76 on the same audit rubric.
One file changed. An independent lead rerun passed both focused tests.
A lower-bound total across direct and harness-derived usage. Some timed-out routes exposed no final usage.
| Effort | Time | Estimated off-peak cost | Findings | Practical read |
|---|---|---|---|---|
| None | 0:26 | $0.0200 | 9 | Fast, noisy, and speculative. More findings did not mean more truth. |
| Low | 1:19 | $0.0273 | 5 | Useful compression, but not enough depth for the consequential lane. |
| High | 3:17 | $0.0426 | 8 | Best default balance of coverage, cost, and evidence. |
| Max | 6:41 | $0.0518 | 5 | Best evidence discipline. Reserve for consequential architecture or a targeted recheck. |
The original Max attempt remains a failed attempt because its 32K response budget was consumed by reasoning before a usable answer arrived. The corrected Max run used a fairer ceiling and scored 88. The correction is labeled; it does not overwrite the failure.
| Lane | Result | Time and usage | What it proved | What it did not prove |
|---|---|---|---|---|
| Historical code repair | Pass | 5:58 / about $0.0386 | It could make a narrow evidence-backed repair in a real codebase. | Four earlier setup attempts failed before provider input because isolated Windows worktrees lacked dependencies. |
| Large-repository scout | Qualified pass | 2:45 / about $0.0349 / 1,024 words | Fast source reconnaissance inside a 1,400-word cap. | It made one material async-waterfall classification error that later review caught. |
| Large HTML build 1 | Browser fail | Primary delivered | It could produce substantial code. | The production contract still failed in the browser. |
| Large HTML build 2 | Browser fail | Primary delivered | The first miss was not a one-off transport accident. | It did not earn final UI production. |
| Neutral repair | Browser fail | 1 repair attempted | A repair opportunity was available. | The repair still did not pass. Other giant responses also broke the required JSON envelope. |
| Visual criticism | 69 / 100 | Completed | Strong direction reconstruction. | Only one of four exact choices; a false cream-paper claim distorted the rest. |
| Prompt-cache replay | Cache worked | 1,280 of 1,510 input tokens cached | Repeated context could be reused. | Cost rose slightly from $0.0259 to $0.0262 because the second answer was longer. |
| Nominal packet | Actual input | Time | Estimated cost | Result |
|---|---|---|---|---|
| 128K | 132,369 tokens | 6:12 | $0.0677 | Completed |
| 512K | 537,618 tokens | 4:21 | $0.1134 | Completed |
| 800K | 818,477 tokens | 5:28 | $0.1630 | Completed |
Finishing an 818K-token packet is impressive. The giant packets still did not beat the curated 113K packet. Context capacity is a seat count, not a requirement to invite the whole internet.
| Mode | Time | Estimated cost | Result | Operational read |
|---|---|---|---|---|
| Single agent | 5:57 | $0.0766 | Strong 4,500-word report after 110 tool calls | Best default for one coherent source audit. |
| Autonomous four-worker swarm | 2:04 | $0.0995 | Failed; coordinator exited while workers were still running | Autonomous orchestration did not earn a route. |
| Explicit assisted four-worker team | 5:42 | $0.1503 | Best report | Nearly twice the single-agent spend and only fifteen seconds faster. |
TeamLoop can now be led from Claude Code as well as Codex. 5 dated changes since the October 12 update: one public software change, one evidence record and three local operating records. No routing role, provider rule or model assignment changed; the operating policy stays 1.11.0.
| Effective date | Change and status | Observed effect and boundary |
|---|---|---|
| Claude Code joins as a second lead Implemented software Public toolkit and private installation | TeamLoop can now be led from a Claude Code session as well as from Codex. Claude Opus 5.5 leads in Claude Code; GPT-6 Sol Medium keeps the lead in Codex. Both use the same runner, packets, receipts, guardrails and run ledger. The portable installer gained a Claude Code host option, and one installation test brings the portable suite from 108 to 109. The Claude Code lead prefers a reviewer from another model family and does not use the Claude Bridge, which stays a Codex-led builder path. | |
| First Claude-led review Evidence Evidence only; private run records | One frozen private packet went from the Claude Code lead to Codex GPT-6 Sol at medium effort through the standard runner: 35 seconds, verdict revise, three high and three medium findings about a local log reader’s redaction and labeling. The lead accepted five, accepted one in part with a stated reason, fixed them and added four regression tests. Reported usage: 40,908 input and 1,943 output tokens. One review shows the lead change works; it is not a quality comparison between leads or models. | |
| Provider executable moved after an update Operating observation and repair Private configuration; documentation note | A Codex desktop update moved its CLI into a new versioned folder. One reviewer run failed at process start after 17 ms, before any model call. The lead checked the new executable’s publisher signature, changed only that path, kept a backup and started a new, disclosed run on the same packet; both records stay in the ledger. The portable runner still refuses to search for executables or fall back to a shell. | |
| Who sent a reviewer, in the local display Implemented locally Local read-only display; not in portable preview | The local display places each reviewer run in the room of the lead session, Codex or Claude Code, whose own run or dispatch command named that reviewer’s packet. Packet paths already carry the task id, so no field was added to run records. Command evidence replaces the configured default room only when it exists; titles and recency are never used. | |
| Plain activity labels in the local display Implemented locally Local read-only display; not in portable preview | Each agent in the display gets a job label (research, build, test, design pass, red team and others), one current line and per-turn lines such as “Edited 3 files” or “Tested 2 times: 1 passed, 1 failed”, worked out from records the tools already keep. Claude Code sessions and their subagents appear beside Codex tasks and TeamLoop reviewers. No model call summarizes anything; prompts, messages, reasoning, tool output and file contents are never shown. |
Measurement boundary: one review and one installation test show that the lead change works. They are not a comparison between leads, a token, quota or billing saving, or a quality result. The six historical benchmark families, original model labels and scores remain frozen. This is personal-project field evidence, not an official Glean benchmark or a universal model ranking.
Public toolkit change record / The Dream Machine, a case study of the local display
15 material changes since the sites' September 21 content cutoff: operating changes, engineering work and evidence updates are labeled separately below. GitHub had already shipped compact digests on September 24. Its portable preview and the installed private runtime have different capability boundaries.
Current operating record: policy 1.11.0, effective October 5. GPT-6 Sol Medium owns the lead. Fable is an explicitly selected candidate. The registry holds 48 ready variants, 2 manual local pilots and 5 deferred or per-project capabilities; the existing 7 automatic lanes still select at most 3 per task.
| Effective date | Change and status | Observed effect and boundary |
|---|---|---|
| Availability fallback Accepted operating change Private installed runtime only Operating decision or implementation recorded | One predeclared subscription switch after a classified capacity or temporary-provider failure before meaningful output. Same packet, both attempts retained. Timeouts and quality failures are excluded. | |
| GPT-6 Sol Medium default Operator decision Private model policy only Operating decision or implementation recorded | The local lead, evidence pass and permitted fallback changed from GPT-5.6 Sol High. This was an operating choice, not a new controlled quality comparison. | |
| Local observation and power controls Implemented locally; recorded September 26 Local read-only display; not in portable preview Evidence recorded; fixes dated September 26 | Multi-root discovery and a reversible desktop power switch were retained. Nine files were verified for three audit repairs: absolute-path redaction, unreadable-rollout phase reset, and cancelled-worker departure. Twenty-three bridge tests pass. Power and scope changes were completed before this record date. The nine-file check compared hashes before local installation. The 23 bridge checks were rerun October 12. | |
| Selective-control audit Evidence and decision Evidence and decision only Evidence or decision recorded | Rejected an extra lead checkpoint and whole-system Jev adoption. Deterministic substitution must remove existing work; no new classifier or standing routing layer was installed. | |
| Reviewed architecture explainer Documentation and evidence Local documentation only Evidence or decision recorded | A self-contained technical explanation received one independent Gemini review; two mobile findings were repaired. Final QA covered three widths and 13 links. It distinguishes public typed plans from the private bounded availability wrapper; no shared-memory or pooled-quota claim. | |
| Compact private receipts Implemented software Private measurement; portable receipts improved separately Operating decision or implementation recorded | The same 43 frozen runs went from 44,552 to 26,340 serialized bytes: 18,212 fewer, or 40.9%. Scoped failures and unfinished records remain visible. These are serialized bytes, not measured tokens, quota, billing or quality. | |
| Mimic metadata audit Evidence only Evidence only Evidence or decision recorded | 242 delivery-status records were inspected. The 43 recent records (September 19 to October 3 UTC) contained 34 completed and 9 failed deliveries; this same frozen set supports the receipt comparison. An earlier experiment changed 44,552 to 25,189 serialized bytes (43.5% fewer), distinct from the implemented 40.9% result. Bytes only. | |
| Maintained-engine audit Evidence and staged plan Evidence and local staged program only Evidence or decision recorded | 16 catalog groups and 61 source mechanisms inspected, with 3 adjacent checks excluded. Seven implementation stages exist: stages 0 and 1 are engineering-verified; stages 2 through 6 remain queued. Verified here means selected engineering gates passed, not proven ordinary-use benefit. Upstream test suites were not executed. | |
| Declared evidence coverage Implemented software Private runtime and portable preview Operating decision or implementation recorded | Required files or original physical line ranges must be covered before preparation. A synthetic packet shrank from 8,142 to 1,773 UTF-8 bytes (78.2%), preserving 3 of 3 declared requirements in the coverage checker. The lead still judges semantic sufficiency; source drift blocks dispatch. Bytes are not token or billing savings. | |
| Pinned Fable candidate Experimental gate Private experimental route only Operating decision or implementation recorded | Explicit selection only, exact returned-model identity, fresh subscription guards, no retry or substitution. There is no successful provider call or creative-quality result supporting promotion. | |
| Creative delivery trial Evidence only Uncontrolled delivery evidence only Evidence or decision recorded | Astra produced 31 SVG studies and 10 dashboard concepts; the lead added 2 boards. Fable and Opus each timed out after about 12 minutes with no output artifact or returned identity. Their creative quality remains unscored. Astra was tool-enabled production; Claude received bounded tool-free reviews. Conditions were uncontrolled. Output counts and empty timeouts are not a quality, speed or reliability comparison; provider cause remains unknown. | |
| Program coordinator guard Scoped implementation Local implementation program only Operating decision or implementation recorded | Exclusive single-host cooperative ownership and state comparison protect the implementation program. Sixteen focused tests and two continuation checkpoints support this narrow scope, not a universal runtime lock. | |
| Bounded readiness command Manual local capability Manual local utility; not in portable preview Operating decision or implementation recorded | One authorized DNS/HTTPS target, declared deadline and pending states, cancellation and a final receipt. Sixteen tests and a four-transition controlled replay passed; natural wait samples remain 0 of 3 planned. | |
| HTML preparation receipt Manual local pilot Manual local pilot; not in portable preview Operating decision or implementation recorded | Twenty-four tests cover hashed evidence and declared desktop checks. Unrun mobile, functional, visual and other checks remain partial. Natural-use samples are zero; receipts are not reusable cache entries. | |
| Procedural room and character expansion Implemented locally; September 26 to October 11 Local display only; no activity or assets exported Work span September 26 to October 11; final record October 11 | Authored recipes grew 26 to 42 (+61.5%); rooms grew 4 to 5 (+25%); gallery specimens grew 31 to 36. Fifteen legacy auras gained fine animated rendering. No runtime model calls or per-avatar sprite library were added. Private activity and homage assets remain local. These counts describe rendering, not productivity. Saved browser QA supports local implementation; no new visual-quality benchmark or physical-device verification is claimed. Ten planned ordinary-use observations remain pending; mobile/PWA remains conditional. |
Measurement boundary: receipt comparison: 44,552 to 26,340 bytes, 18,212 fewer. Synthetic packet: 8,142 to 1,773 UTF-8 bytes, 6,369 fewer. These are payload measurements, not measured token, quota, billing or quality savings. Readiness and HTML receipt work are manual local pilots. Later program stages remain queued; no reusable evidence cache is shipped.
The six historical benchmark families, original model labels and scores remain frozen. The September 21 boundary is a verified content cutoff, not an inferred deployment timestamp. This is personal-project field evidence, not an official Glean benchmark or a universal model ranking.
I do not rewrite an old table every time a model gets promoted, retired, or bonks its head on a context window. Each study stays frozen. This ledger records what changed in the operating system afterward.
Sol became the lead and strongest default in this task set. Terra earned the fast sustained-builder lane. Luna stayed a bounded variant builder. Sonnet earned human-facing critique. Flash High and Medium split completeness from rapid breadth.
The team stopped treating delivery and quality as one number. Strong salvages remained valuable, but a clean browser pass became its own operating requirement.
Repairs became an explicit second lane instead of an invisible rewrite. Preventable infrastructure aborts were separated from model failures. Opus showed unusually strong repair recovery, while no model earned immunity from browser QA.
Opus earned the rare consequential architecture escalation. Muse and Nemotron remained different-family wildcards. GLM-5.2 left the available roster after its endpoint returned retired.
Spark earned a reserve lane for atomic, one-turn patches with fixed acceptance criteria and a deterministic test. It did not join automatic routing.
DeepSeek earned the public and synthetic engineering-audit lane at High effort. Its visual, browser-delivery, and autonomous-swarm results kept it out of final UI work and orchestration authority.
A temporary, expiring Claude implementation overlay entered a six-day field trial after exact-file synthetic and real-repository gates passed. Standard routing stayed canonical. Claude drafts inside detached worktrees; Codex still scopes, verifies, promotes, commits, and pushes. Failed builds return the system to standard with no silent retry or paid overflow.
The temporary implementation overlay expired and was explicitly retired. Standard role-based TeamLoop routing is active again. Claude remains available for its usual review and specialist work, with usage credits off and a fresh 24-hour account check required before a Claude worker run.
TeamLoop gained a native typed plan for multi-review runs. Plans validate dependencies, cap concurrency, distinguish required from advisory failures, and retain schema-versioned lifecycle events. The lead still adjudicates. Plans cannot run shell commands, import remote tasks, retry workers, or substitute providers.
A Gemini capacity failure exposed a dead lane, not a judgment failure. Normal Claude and Gemini reviews may now switch once to Codex Sol only when capacity or a temporary provider outage occurs before meaningful output. The exact packet is reused, both attempts stay visible, and timeouts, policy failures, bad output, quality chasing, paid APIs, DeepSeek, and wildcard lanes still fail closed.
Mark selected GPT-6 Sol at Medium effort for the local lead, evidence preparation and availability fallback on September 25, after checking the then-current official guidance. This is an operating choice, not a new TeamLoop head-to-head benchmark or a measured savings result. Astra stays task-selected and frozen GPT-5.6 results retain their original labels.
Implemented locally; recorded September 26. Multi-root discovery and a reversible desktop power switch were retained. Nine files were verified for three audit repairs: absolute-path redaction, unreadable-rollout phase reset, and cancelled-worker departure. Twenty-three bridge tests pass.
Evidence and decision. Rejected an extra lead checkpoint and whole-system Jev adoption. Deterministic substitution must remove existing work; no new classifier or standing routing layer was installed.
Documentation and evidence. A self-contained technical explanation received one independent Gemini review; two mobile findings were repaired. Final QA covered three widths and 13 links. It distinguishes public typed plans from the private bounded availability wrapper; no shared-memory or pooled-quota claim.
Implemented software. The same 43 frozen runs went from 44,552 to 26,340 serialized bytes: 18,212 fewer, or 40.9%. Scoped failures and unfinished records remain visible.
Evidence only. 242 records were inspected; the recent 43 contained 34 completed and 9 failed deliveries. An initial digest experiment cut bytes by 43.5%; that is separate from the implemented 40.9% result.
Evidence and staged plan. 16 catalog groups and 61 source mechanisms inspected, with 3 adjacent checks excluded. Seven implementation stages exist: stages 0 and 1 are engineering-verified; stages 2 through 6 remain queued.
The operator requested a bounded creative comparison with Fable. TeamLoop now permits explicit selection of its pinned model behind an experimental gate, exact returned-model checks, and existing subscription and tools-off guards. Default routing stays unchanged. Local guard tests passed, but there is no successful provider call or creative-quality result yet. A paid-only availability conflict stops the lane without retry or substitution.
Astra produced 31 SVG studies and 10 dashboard concepts; the lead added two boards. The Fable and Opus attempts timed out after roughly 12 minutes with no stdout, stderr, artifact, returned model identity, or usage. The Fable receipt reports a timeout stop reason while retaining runner-or-unknown as its failure-class field. These are delivery and observability results, not quality scores or proof of a provider cause. Default routing and Fable's candidate status stay unchanged. No quality score or promotion follows from this trial.
Implemented software. Required files or original physical line ranges must be covered before preparation. A synthetic packet shrank from 8,142 to 1,773 UTF-8 bytes, preserving 3 of 3 requirements. Source drift blocks dispatch.
Scoped implementation. Exclusive single-host cooperative ownership and state comparison protect the implementation program. Sixteen focused tests and two continuation checkpoints support this narrow scope, not a universal runtime lock.
Manual local capability. One authorized DNS/HTTPS target, declared deadline and pending states, cancellation and a final receipt. Sixteen tests and a four-transition controlled replay passed; natural wait samples remain 0 of 3 planned.
Manual local pilot. Twenty-four tests cover hashed evidence and declared desktop checks. Unrun mobile, functional, visual and other checks remain partial. Natural-use samples are zero; receipts are not reusable cache entries.
Implemented locally; September 26 to October 11. Authored recipes grew 26 to 42 (+61.5%); rooms grew 4 to 5 (+25%); gallery specimens grew 31 to 36. Fifteen legacy auras gained fine animated rendering. No runtime model calls or per-avatar sprite library were added. Private activity and homage assets remain local.
Implemented locally. Read-only readiness checks now precede Claude reviews; expired billing verification, deliberate cancellation and timeout are distinct from model failure. One small synthetic Sonnet smoke completed with a validated response. This verifies current transport readiness only; model assignments and the 24-hour billing boundary are unchanged. Historical evidence is preserved.
Implemented software. TeamLoop can now be led from a Claude Code session as well as from Codex. Claude Opus 5.5 leads in Claude Code; GPT-6 Sol Medium keeps the lead in Codex. Both use the same runner, packets, receipts, guardrails and run ledger. The portable installer gained a Claude Code host option, and one installation test brings the portable suite from 108 to 109. The Claude Code lead prefers a reviewer from another model family and does not use the Claude Bridge, which stays a Codex-led builder path.
Evidence. One frozen private packet went from the Claude Code lead to Codex GPT-6 Sol at medium effort through the standard runner: 35 seconds, verdict revise, three high and three medium findings about a local log reader’s redaction and labeling. The lead accepted five, accepted one in part with a stated reason, fixed them and added four regression tests. Reported usage: 40,908 input and 1,943 output tokens. One review shows the lead change works; it is not a quality comparison between leads or models.
Operating observation and repair. A Codex desktop update moved its CLI into a new versioned folder. One reviewer run failed at process start after 17 ms, before any model call. The lead checked the new executable’s publisher signature, changed only that path, kept a backup and started a new, disclosed run on the same packet; both records stay in the ledger. The portable runner still refuses to search for executables or fall back to a shell.
Implemented locally. The local display places each reviewer run in the room of the lead session, Codex or Claude Code, whose own run or dispatch command named that reviewer’s packet. Packet paths already carry the task id, so no field was added to run records. Command evidence replaces the configured default room only when it exists; titles and recency are never used.
Implemented locally. Each agent in the display gets a job label (research, build, test, design pass, red team and others), one current line and per-turn lines such as “Edited 3 files” or “Tested 2 times: 1 passed, 1 failed”, worked out from records the tools already keep. Claude Code sessions and their subagents appear beside Codex tasks and TeamLoop reviewers. No model call summarizes anything; prompts, messages, reasoning, tool output and file contents are never shown.
Operational repair, not a new model ranking. Two reviewer runs ended at their deadlines without an accepted final response. A deeper local investigation found generation activity and conflicting instructions: the schema exposed a finish completion protocol while the agent and runner prohibited all tools. On CLI 1.4.5, a synthetic arithmetic check returned an empty partial SUCCESS after 183.736 seconds; after explicitly permitting schema-only finish, a new check on the same assertion, model, effort and deadline completed in 17.428 seconds. The CLI deadline warning is now classified as a timeout even with exit code zero. Operational tools remain denied. This small before/after check supports the contract conflict as a contributing cause; service variability is not ruled out, and historical scores and routing stay unchanged.
The original frozen tasks were replayed with matched current controls and the same 45-minute ceiling for every design and review cell. This is a dated replay of existing benchmark families, not a rewrite of historical scores. Sixty-five primary configurations covered six newer OpenAI/Anthropic versions and two matched controls. Opus 5.5 led the three bounded design briefs at 94.33/100; GPT-6.1 Sol scored 94 on the source-grounded engineering lab. Larger builds exposed substantial non-delivery in the Claude CLI route. Accepted quality, failed delivery and single neutral repairs remain separate. Internal Codex reviewers supplied masked rubric judgments; the lead froze scores before joining identities. Mark did not personally score this replay. Candidate evidence only. Current roster, effort, fallback, privacy and paid-overflow rules stay unchanged; this local update has not been published.
The operator chooses the lead they trust to follow instructions and work well with them. A benchmark does not override that operating preference. New routes use GPT-6.1 Sol, and the standard runner refuses GPT-6 Sol. No provider, billing, privacy or automatic specialist-promotion rules change. Historical benchmark results remain unchanged. The Haiku trial is still a plan.
The Claude continuation audit qualified large-build ability claims. Durable files recovered useful work but did not prove a native fix. Separate OpenAI file-plus-JavaScript trials all failed the full gate. In a matched four-primary feature trial, Haiku scored 26 and 87 versus GPT-6.1 Sol at 94 and 95. No automatic Haiku promotion; the operator-selected 6.1 lead stays. Nine unchanged synthetic examples make the strengths and defects inspectable.
Explicit operator activation: pinned Sonnet 5.5 High for critique and Opus 5.5 High for architecture and selected durable-file design ownership, with the full 45-minute window. GPT-6.1 Sol remains lead; GPT-6 Sol excluded; Haiku not promoted. Small live compatibility checks returned both exact model identities. An initial Opus postflight rejected cloud-sync hard links; unsynced worktree placement was corrected without weakening the guard. These are operating checks, not new quality scores.
A long file-design attempt hit the harness event-stream capture limit before its native final receipt. Repeated file snapshots in tool-result metadata filled the 8 MiB budget. The failed attempt stays recorded as an infrastructure abort. File-design capture is now bounded at 32 MiB; artifact, receipt, tool and provider guards remain unchanged. Synthetic streams verify acceptance above the old limit and rejection above the new one. Separate completed deliveries stayed below 8 MiB, so they do not prove a causal recovery. No score, winner or routing change follows.
Fresh calibrated quality scores put the completed Opus artifacts at 85.5-86.5 and Sonnet at 81-85.5. Working 6.1 examples remain strong; its failed fleet primary and separate repair stay visible. Unequal generation tools, partial mobile verification and small-sample AI judgment prevent a universal ranking. Five unchanged outputs join the gallery; 6.1 stays lead.
Builder 1.0.6 screens original receipt bytes before redaction, including EOF without a newline. Limits are explicit: 16 KiB overall, summary 4096 characters, up to 16 uncertainties of 512 characters. Exact single-fence formatting tolerance does not repair malformed JSON or weaken tool, identity and billing guards.
Argon delivered3/3current reviews; Flash2/3 with one HTTP500. Argon won the two delivered paired utility judgments and found a final-watch gap, independently reproduced and repaired. Candidate status and6.1lead remain unchanged.
The operator accepted GPT-6.1 Terra Medium for active bounded work under policy 1.14.0, keeping GPT-6.1 Sol Medium as lead. Both Terra versions found the same two defects in a small matched text smoke. Published credit rates are lower, but the rounded unchanged usage meter cannot establish included subscription savings. Historical 5.6 scores stay frozen.
A new matched host file-action cohort kept the original console task and scored Opus 5.5 High 91 against Argon Medium 69. Argon invented audit details and retained score-bound and verdict-label defects. Infrastructure recoveries preserve original evidence without quality regeneration. Actual exported-file contents remain unverified for both. No automatic design promotion or historical score rewrite follows.
The practical result was a small routing discipline. One lead owns the context and judgment. Deterministic tools do deterministic work. Another model only enters when what it might add is worth the cost of explaining the job and integrating the answer.
Keep implementation, testing, and final synthesis together. Add a critic only when the stakes or ambiguity are worth another handoff.
Specialists get bounded evidence packets. One lead brings the answers together and stays accountable for the result.
When models disagree, I check the evidence. I do not count hands. The second model is there to notice something different, not form a tiny robot committee.
Conversion, extraction, linting, browser checks, and structured diffs happen deterministically whenever they can.
These rows are the public reconstruction layer behind the summaries. “Clean” only describes delivery. It does not mean the work was good. Later studies use deterministic pass conditions where a visual score would make no sense. Private prompts, credentials, run identifiers, and proprietary artifacts stay out.
| Date | Task | Configuration | Score | Delivery | Browser result | Failure attribution |
|---|---|---|---|---|---|---|
| 2026-09-04 | Adoption operations | Claude Opus | 83 | Clean | Pass (6/6 checks) | None |
| 2026-09-04 | Transactional estimator | Claude Opus | None | Failed | No artifact | Timeout at configured ceiling; content unscored |
| 2026-09-04 | Model comparison | Claude Opus | 78 | Clean | Issues (6/7 checks) | Delivered artifact missed a required browser check |
| 2026-09-04 | Adoption operations | Claude Sonnet | 85 | Clean | Pass (6/6 checks) | None |
| 2026-09-04 | Transactional estimator | Claude Sonnet | 84 | Clean | Pass (7/7 checks) | None |
| 2026-09-04 | Model comparison | Claude Sonnet | 86 | Clean | Pass (7/7 checks) | None |
| 2026-09-04 | Adoption operations | Codex Luna | 93 | Clean | Pass (6/6 checks) | None |
| 2026-09-04 | Transactional estimator | Codex Luna | 87 | Clean | Pass (7/7 checks) | None |
| 2026-09-04 | Model comparison | Codex Luna | 79 | Clean | Issues (4/7 checks) | Delivered artifact missed required browser checks |
| 2026-09-04 | Adoption operations | Codex Sol | 97 | Clean | Pass (6/6 checks) | None |
| 2026-09-04 | Transactional estimator | Codex Sol | 93 | Clean | Pass (7/7 checks) | None |
| 2026-09-04 | Model comparison | Codex Sol | 93 | Clean | Issues (6/7 checks) | Delivered artifact missed a required browser check |
| 2026-09-04 | Adoption operations | Codex Terra | 92 | Clean | Pass (6/6 checks) | None |
| 2026-09-04 | Transactional estimator | Codex Terra | 96 | Clean | Pass (7/7 checks) | None |
| 2026-09-04 | Model comparison | Codex Terra | 83 | Clean | Issues (6/7 checks) | Delivered artifact missed a required browser check |
| 2026-09-04 | Adoption operations | Gemini Pro High | None | Failed | No artifact | Timeout waiting for a complete response; content unscored |
| 2026-09-04 | Transactional estimator | Gemini Pro High | 79 | Clean | Pass (7/7 checks) | None |
| 2026-09-04 | Model comparison | Gemini Pro High | None | Failed | No artifact | Timeout waiting for a complete response; content unscored |
| 2026-09-04 | Adoption operations | Gemini Flash High | 92 | Clean | Pass (6/6 checks) | None |
| 2026-09-04 | Transactional estimator | Gemini Flash High | 89 | Clean | Pass (7/7 checks) | None |
| 2026-09-04 | Model comparison | Gemini Flash High | 88 | Clean | Pass (7/7 checks) | None |
| 2026-09-04 | Adoption operations | Gemini Flash Medium | 83 | Clean | Issues (6/6 checks, 1 exception) | Delivered artifact raised a browser runtime exception |
| 2026-09-04 | Transactional estimator | Gemini Flash Medium | 91 | Clean | Pass (7/7 checks) | None |
| 2026-09-04 | Model comparison | Gemini Flash Medium | 88 | Clean | Pass (7/7 checks) | None |
| Date | Task | Configuration | Score | Delivery | Browser result | Failure attribution |
|---|---|---|---|---|---|---|
| 2026-09-04 | Agent fleet simulator | Claude Opus | 81 | Salvaged | Pass | Wrapper parse failure; complete HTML salvaged and separately scored |
| 2026-09-04 | Failure strategy simulator | Claude Opus | 80 | Clean | Pass | None |
| 2026-09-04 | Futures exploration lab | Claude Opus | 52 | Salvaged | Issues | Wrapper parse failure; salvaged artifact also failed the browser gate |
| 2026-09-04 | Agent fleet simulator | Claude Sonnet | None | Failed | No artifact | Timeout at configured ceiling; no artifact scored |
| 2026-09-04 | Failure strategy simulator | Claude Sonnet | 84 | Salvaged | Pass | Wrapper parse failure; complete HTML salvaged and separately scored |
| 2026-09-04 | Futures exploration lab | Claude Sonnet | 89 | Clean | Pass | None |
| 2026-09-04 | Agent fleet simulator | Codex Sol | 56 | Clean | Issues | Delivered artifact had runtime exceptions and a broken operating core |
| 2026-09-04 | Failure strategy simulator | Codex Sol | 80 | Clean | Pass | None |
| 2026-09-04 | Futures exploration lab | Codex Sol | 90 | Clean | Pass | None |
| 2026-09-04 | Agent fleet simulator | Gemini Flash High | 89 | Clean | Pass | None |
| 2026-09-04 | Failure strategy simulator | Gemini Flash High | 87 | Salvaged | Pass | Wrapper escaping failure; complete HTML salvaged and separately scored |
| 2026-09-04 | Futures exploration lab | Gemini Flash High | None | Failed | No artifact | Provider or harness rejected unexpected activity; no artifact scored |
| 2026-09-04 | Agent fleet simulator | Gemini Flash Medium | 90 | Clean | Pass | None |
| 2026-09-04 | Failure strategy simulator | Gemini Flash Medium | 53 | Clean | Issues | Delivered artifact had a broken operating core |
| 2026-09-04 | Futures exploration lab | Gemini Flash Medium | 88 | Clean | Pass | None |
| Date | Task | Configuration | Score or oracle | Delivery | Verification result | Failure attribution |
|---|---|---|---|---|---|---|
| 2026-09-07 | Surgical historical repair | GPT-5.3-Codex-Spark · medium | 2/2 focused tests | Clean | Pass | None |
| 2026-09-07 | Three-turn UI implementation and repair | GPT-5.3-Codex-Spark · medium | 37/38 final checks | Qualified | One reset-state defect remained | Delivered work; one acceptance criterion remained unresolved |
| 2026-09-07 | Broad repository scout · initial dispatch | GPT-5.3-Codex-Spark · medium | None | Aborted | No report | Provider quota rejection before content work; not a model-quality result |
| 2026-09-07 | Broad repository scout · authorized post-reset attempt | GPT-5.3-Codex-Spark · medium | None | Failed | No report | Context exhausted during remote compaction after about 8m50s |
| Date | Phase | Task | Configuration | Delivery | Browser result | Failure attribution |
|---|---|---|---|---|---|---|
| 2026-09-05 | Primary | Agent Fleet | Codex Sol high | Complete | Pass | None |
| 2026-09-05 | Primary | Futures Lab | Codex Sol high | Complete | Pass | None |
| 2026-09-05 | Primary | Failure Simulator | Codex Sol high | Complete | Fail | Inline JavaScript syntax error; operating core did not load |
| 2026-09-05 | Primary | Design Direction | Codex Sol high | Complete | Fail | Horizontal overflow at 390 px |
| 2026-09-05 | Primary | Agent Fleet | Gemini Flash High | Complete | Fail | Horizontal overflow at 390 px |
| 2026-09-05 | Primary | Futures Lab | Gemini Flash High | Complete | Pass | None |
| 2026-09-05 | Primary | Failure Simulator | Gemini Flash High | Complete | Pass | None |
| 2026-09-05 | Primary | Design Direction | Gemini Flash High | Complete | Fail | Horizontal overflow at 390 px |
| 2026-09-05 | Primary | Agent Fleet | Gemini Flash Medium | Complete | Fail | Missing DOM element at startup and mobile overflow |
| 2026-09-05 | Primary | Futures Lab | Gemini Flash Medium | Complete | Fail | Runtime data-collision exception |
| 2026-09-05 | Primary | Failure Simulator | Gemini Flash Medium | Complete | Fail | Multiple null-element exceptions; results and export failed |
| 2026-09-05 | Primary | Design Direction | Gemini Flash Medium | Complete | Fail | Mobile overflow and one inert navigation control |
| 2026-09-05 | Repair | Failure Simulator | Codex Sol high | Complete | Fail | Syntax fixed; missing result elements still raised exceptions |
| 2026-09-05 | Repair | Design Direction | Codex Sol high | Complete | Pass | None |
| 2026-09-05 | Repair | Agent Fleet | Gemini Flash High | Complete | Fail | Repair introduced literal escaped newlines and runtime failures |
| 2026-09-05 | Repair | Design Direction | Gemini Flash High | Complete | Pass | None |
| 2026-09-05 | Repair | Agent Fleet | Gemini Flash Medium | Complete | Fail | Required intervention behavior remained absent |
| 2026-09-05 | Repair | Futures Lab | Gemini Flash Medium | Complete | Fail | Required nondefault lens interaction remained inert |
| 2026-09-05 | Repair | Failure Simulator | Gemini Flash Medium | Incomplete | No artifact | Provider error after partial replacement; not rerun |
| 2026-09-05 | Repair | Design Direction | Gemini Flash Medium | Complete | Pass | None |
| 2026-09-05 | Primary | Agent Fleet | Claude Sonnet high | Timeout | No artifact | Full twelve-minute ceiling reached |
| 2026-09-05 | Primary | Futures Lab | Claude Sonnet high | Timeout | No artifact | Full twelve-minute ceiling reached |
| 2026-09-05 | Primary | Failure Simulator | Claude Sonnet high | Timeout | No artifact | Full twelve-minute ceiling reached |
| 2026-09-05 | Primary | Design Direction | Claude Sonnet high | Complete | Fail | Mobile overflow and unobserved export |
| 2026-09-05 | Repair | Design Direction | Claude Sonnet high | Timeout | No artifact | Full twelve-minute ceiling reached |
| 2026-09-05 | Primary | Agent Fleet | Claude Opus high | Complete | Fail | Mobile overflow |
| 2026-09-05 | Repair | Agent Fleet | Claude Opus high | Complete | Pass | None |
| 2026-09-05 | Primary | Futures Lab | Claude Opus high | Timeout | No artifact | Full twelve-minute ceiling reached |
| 2026-09-05 | Primary | Failure Simulator | Claude Opus high | Complete | Fail | Adversary-lens control produced no state change |
| 2026-09-05 | Repair | Failure Simulator | Claude Opus high | Complete | Fail | Same adversary control remained inert |
| 2026-09-05 | Primary | Design Direction | Claude Opus high | Complete | Fail | Mobile overflow |
| 2026-09-05 | Repair | Design Direction | Claude Opus high | Complete | Pass | None |
Fourteen setup records failed before usable provider work and were excluded from model comparison. Their failure classes and the harness corrections are summarized in the study chapter above.
| Date | Arena | Configuration | Score or oracle | Delivery | Failure attribution or finding |
|---|---|---|---|---|---|
| 2026-09-05 | Visual contact sheet 1 | Muse | Four ties | Complete | No decisive ranking |
| 2026-09-05 | Visual contact sheet 2 | Muse | Four ties | Complete | No decisive ranking |
| 2026-09-05 | Visual contact sheet | Claude Sonnet | 55 | Deterministic salvage | One exact B lean and two exact B choices |
| 2026-09-05 | Eight PNG visual set | Muse | None | Failed | HTTP 500 at about 12.4 MiB |
| 2026-09-05 | Eight WebP visual set | Muse | 43 | Complete | One of four exact selections |
| 2026-09-05 | Eight WebP visual set | Claude Sonnet | 34 | Complete | Four ties |
| 2026-09-05 | Visual transport | Gemini Flash High | Not scored | Unavailable | Tested headless adapter was text-only; not a model failure |
| 2026-09-05 | Engineering artifact | Codex Sol | Primary fail | Complete | Broken JavaScript |
| 2026-09-05 | Engineering salvage | Mechanical edit | 15 / 15 checks | Complete | Three literal corrections; not credited to primary delivery |
| 2026-09-05 | Engineering artifact | GLM-5.2 | None | Failed | Endpoint retired with HTTP 410 |
| 2026-09-05 | Engineering artifact | Claude Opus | None | Timeout | No artifact at 15:03.4 |
| 2026-09-05 | Systems audit | Claude Opus | 89 | Complete | Deep architecture winner |
| 2026-09-05 | Systems audit | Gemini Flash High | 76 | Complete | Faster everyday completeness reviewer |
| 2026-09-05 | Systems audit | Nemotron 3 Ultra | 8 | Complete | Did not earn a normal route; another NVIDIA request returned HTTP 503 |
| Date | Lane | Effort or mode | Score or oracle | Delivery | Failure attribution or finding |
|---|---|---|---|---|---|
| 2026-09-11 | Systems audit | None | 9 findings | Complete | Noisy and speculative |
| 2026-09-11 | Systems audit | Low | 5 findings | Complete | Useful but shallow for consequential review |
| 2026-09-11 | Systems audit | High | 8 findings | Complete | Best default balance |
| 2026-09-11 | Systems audit | Max, original | None | Failed | 32K response ceiling consumed by reasoning |
| 2026-09-11 | Systems audit | Max, corrected | 88 / 100 | Complete | Best evidence discipline; correction kept separate |
| 2026-09-11 | Historical repair | High | 2 / 2 tests | Complete | Pass after one-file change |
| 2026-09-11 | Historical repair setup | Four pre-provider attempts | Excluded | Infrastructure-aborted | Isolated Windows worktrees lacked dependencies |
| 2026-09-11 | Large-repo scout | High | Qualified pass | Complete | One material async-waterfall classification error |
| 2026-09-12 | Visual criticism | High | 69 / 100 | Complete | Strong direction reconstruction; one of four exact choices |
| 2026-09-12 | Large HTML build 1 | High | 0 browser passes | Complete | Primary failed browser QA |
| 2026-09-12 | Large HTML build 2 | High | 0 browser passes | Complete | Primary failed browser QA |
| 2026-09-12 | HTML repair | High | 0 browser passes | Complete | Neutral repair still failed |
| 2026-09-12 | Long context | 132,369 tokens | Completed | Complete | No transport failure |
| 2026-09-12 | Long context | 537,618 tokens | Completed | Complete | No transport failure |
| 2026-09-12 | Long context | 818,477 tokens | Completed | Complete | Did not beat the curated 113K packet |
| 2026-09-12 | Cache replay | High | 1,280 / 1,510 cached | Complete | Cost rose slightly because output grew |
| 2026-09-12 | Agent audit | Single agent | Strong report | Complete | Best default operating mode |
| 2026-09-12 | Agent audit | Autonomous four-worker swarm | None | Failed | Coordinator exited while workers remained active |
| 2026-09-12 | Agent audit | Explicit assisted four-worker team | Best report | Complete | Nearly twice the spend and only fifteen seconds faster than single |
Bounded and long-running trial dates are UTC calendar dates from recorded run starts. Later dates follow their preserved study logs. Scores only appear when a scored rubric applied. I did not count a hidden retry, repair, or automatic provider substitution as an original success.
| Configuration | Migration targets | Synthesis targets | Systems /100 | Systems time |
|---|---|---|---|---|
| 6.1 High | 5/5 | 8/8 | 87 | 124.97s |
| Argon Medium | 5/5 | 8/8 | 61 | 66.65s |
| Flash High | 5/5; overstatement | 7 full, 1 partial | 39 | 350.95s |
All nine text primaries completed. Same original systems evidence and 45-minute ceiling; migration and synthesis retained their separate six-minute screen limits. Two masked OpenAI panels locked judgments before identities were joined. Systems scores and short-screen coverage are different metrics, with no pooled mean. Both Gemini reviews wrongly said supplied Host files were absent. Saved prompts contain all 12 complete sources, with no native truncation warning; outgoing Google bodies and backend-attended context are unverified. Token ratios cannot settle that question. All three reviewers missed the authoritative queue-update handler.
One review of current receipt/capture code, one of published claims and one of Gemini completion integration. Public source snapshot ec9ae37, identical paired packets and operational-tools-off contract, 15 minutes per attempt. These are current-work utility judgments, not the old build benchmark or a universal model leaderboard.
| Task | Configuration | Delivery | Review utility /100 | Time |
|---|---|---|---|---|
| Gemini integration audit | Flash High | Completed | 52 | 176.64s |
| Gemini integration audit | Argon Medium | Completed | 81 | 46.00s |
| Public claim audit | Flash High | Completed | 59 | 224.89s |
| Public claim audit | Argon Medium | Completed | 97 | 43.05s |
| Receipt/capture audit | Flash High | HTTP 500; failed | Unscored | 198.11s |
| Receipt/capture audit | Argon Medium | Completed | 100 | 54.16s |
Argon's 100-point receipt review was a precise, useful no-defect conclusion, not a production certification. Its claim review avoided invented contradictions. Flash's claim review treated dated history and limited-scope documents as contradictions. Argon found the useful final-watch seam in the Gemini audit, but mandatory deep equality and array-shape remedies were not adopted. Flash supplied conditional identity/test observations while overstating image and possible future-timeout issues. Five completed reviews were graded behind random IDs using utility, evidence precision, false-positive control, remedies and coverage; the failed control received no quality score. Lead source checks and arithmetic followed the locked AI-assisted panel.
A synthetic child could write a disallowed marker to its watch log and exit between polling ticks. Both process helpers returned success without checking that final log. Actual-child reproductions confirmed the gap; this does not prove a live Gemini bypass, paid spend or operational action. The corrected public and installed helpers settle a pending poll and inspect the final configured watch log before accepting normal completion. Filesystem probes have a 1,000 ms deadline, fail closed and isolate late results. Earlier stops remain authoritative and no termination is attempted against an exited child. Four regression fixtures bring the public suite to 149 passing tests; independent review and installed-path checks support the repair. Original trial outputs remain frozen.
The Flash receipt control ended with a native INTERNAL HTTP 500 after generation. Startup OAuth warnings contaminated its generic authentication label; they did not establish terminal authentication failure. The runner stopped before the advertised native retry. After a fresh read-only preflight, only the four unattempted original cells resumed; all six planned attempts ran once, without substitution. The first packet preparation was blocked before inference by synthetic secret-screening fixtures. The relevant EOF tests were supplied verbatim as declared original lines 1-57; unrelated credential-marker tests were excluded, with the full original hash preserved.
Antigravity CLI 1.4.7 reports one configured unsandboxed rule spelling invalid/ignored and says it grants nothing. Configuration checks and mocks do not prove seven effective native denials or a universal OS sandbox. Strict mode, identity/settings checks and observed event rejection remain in place. Gemini headless is text-only; native effort labels are not equal compute. Completed CLI model labels are not independent backend provenance.
The earlier NVIDIA branch remains separate: one Super VL canary failed its returned-model guard, but the native identity value was not retained. Missing field, alias and wrong model remain possible. Four scored NVIDIA calls stayed held; no quality result or new Muse checkpoint is claimed. No new NVIDIA call was made for these Argon trials.
Operating consequence: 6.1 leads; Opus and Sonnet retain their pinned roles. Argon is available only through the qualified task-local evaluation path, outside the canonical installed allowlist and automatic routing. Candidate evidence does not change policy v1.13.0. Single attempts, AI-assisted judgments, unmeasured lead labor and shared subscription meters cannot establish general superiority or quota savings.
Download all nine screen rows · Download all six current-work attempts · Runtime and evidence change record.
This was my personal experiment, not an official Glean benchmark. Historical cohorts and their scores remain unchanged.
GPT-6.1 Sol Medium stays lead. Opus 5.5 earns deliberate design and deep-modeling work; Sonnet 5.5 remains an independent critic and selected synthesis worker. The new completed Claude builds show real capability, without establishing a model-wide sweep or changing the historical scores.
| Exact model tested / High | Fleet | Failure Garden | Futures |
|---|---|---|---|
| Opus 5.5 | 85.5; receipt recovery | 86.5; corrected fresh attempt | 86.5 |
| Sonnet 5.5 | 81; artifact quality only, receipt failed | 85.5 | 83.5; earlier one-shot artifact |
| GPT-6.1 Sol | 39.5 primary / 89 repaired | 91 | 91 |
| GPT-5.6 Terra | Not tested | 62.5; file plus JavaScript | Not tested |
| GPT-6 Luna | 49 | 39.5; original one-shot | 67 |
Fresh artifact judgments out of 100, not a matched model-only ranking. Luna's separate file-plus-JavaScript Failure Garden scored 47 and still failed startup/Run. 6.1's repaired score does not erase its failed primary. No untested Terra score is invented. Download all 15 artifact rows and category scores.
Three masked task reviewers read 15 complete sources, browser observations and desktop evidence under the original seven-category rubric. A separate masked visual reviewer calibrated all 15 opening views at the same requested 1536 by 960 desktop size. Final scores retain six frozen nonvisual components and replace only the 10-point visual component. Raw panel scores and calibrated values are both in the CSV. Scores and input hashes were locked before identity joining. This is AI-assisted judgment, not human ratification or statistical evidence.
The original brief and rubric bytes, High effort, 45-minute outer allowance and 512 KiB HTML limit were retained. New Claude artifacts used selected-file Read/Write/Edit without a computation tool; original 6.1 and Luna controls and Sonnet's earlier Futures artifact used whole-HTML JSON completion; Terra and the supplemental Luna artifact had selected files plus isolated JavaScript. Re-scoring reduces judge drift but cannot remove these workflow differences or normal generation variation. Speed and length earn no quality bonus. No grand average across unequal task coverage is presented.
Actual build outcomes: Sonnet Failure Garden completed in 40.59 minutes, Opus Futures in 29.05, and the separately approved corrected Opus Failure Garden in 40.45. All three completed natively and through their original wrapper, without receipt salvage. An earlier Opus call hit our 8 MiB stream-capture guard at 42.69 minutes; that infrastructure abort and its saved artifact remain separate. The successful fresh generation does not prove a deterministic recovery caused solely by a cap change.
Where the points went: Opus had excellent verification/rework, delayed detection and original hybrid institutions. It also had stale-state/export issues and an observed Futures defect where reordering the same cards changed detected tensions from two to zero. Sonnet supplied sampled failure replay and useful tradeoffs, with chronology/probability and state-freshness limits. Working 6.1 artifacts were better integrated, while its fleet primary could not initialize and its Futures atlas still displayed NaN numbering. Terra's basic loop worked but deeper mechanics mostly followed scalar pressure; Luna's attractive shells repeatedly outran their implementation.
Verification boundaries: native identity, allowed-file changes, immutable inputs, structural checks and core browser transitions were inspected. Opening desktop captures were standardized. Mobile visual usability remains unverified because viewport geometry and screenshot evidence conflicted. Actual downloaded export contents, full keyboard/OS reduced-motion behavior and exhaustive numerical validation remain outside verified coverage. A score or startup badge is not a production-readiness certificate.
File-design stream capture is bounded at 32 MiB, with the legacy builder still at 8 MiB. Builder 1.0.6 validates the original final receipt before redaction, including a terminal line without a final newline. The receipt is limited to 16,384 raw UTF-8 bytes, a 4,096-character summary and at most 16 uncertainty strings of 512 characters each. The worker is told to return raw JSON. The receiver tolerates exactly one complete lowercase json fence, while refusing surrounding prose, malformed JSON, extra fields and oversized content. It does not extract, repair or retry a failed response. Artifact, identity, billing, tools, path, deadline and deliberate-promotion guards stay intact. Software fixtures and an independent release review support the implementation; they do not rewrite old receipts.
Portable runtime and evidence change record. Argon and Nemotron 3.5 were identified as separate candidates, outside this OpenAI/Anthropic-only execution cohort. Their absence here is not a measured quality result.
These are byte-for-byte synthetic artifacts, including documented defects and the malformed-receipt fleet candidate. No cleanup repairs or hidden best-of selection. Use synthetic inputs only.
Opus 5.5 High
Verification queues and rework have depth; export freshness and soft budget limits remain.
Inspect unchanged build
Sonnet 5.5 High
Useful simulation with a malformed final receipt. Saved app is not a clean route success.
Inspect unchanged build
Opus 5.5 High
Rich detection and containment; pending export and comparison-cache weaknesses remain.
Inspect unchanged build
Sonnet 5.5 High
Sampled failure replay and constrained portfolios; chronology and probability language need care.
Inspect unchanged build
Opus 5.5 High
Distinctive hybrid institutions; reordering can hide deck tensions. Bug preserved.
Inspect unchanged buildEarlier galleries and their historical scores remain unchanged. This was my personal experiment, not an official Glean benchmark.
Policy v1.13.0 is active locally. GPT-6.1 Sol Medium stays lead. Sonnet is pinned to claude-sonnet-5-5 at High for independent critique. Opus is pinned to claude-opus-5-5 at High for rare architecture and selected ambitious interface/design ownership. GPT-6 Sol remains excluded. Haiku has no automatic feature-owner promotion.
The selected design command gives Opus the full 45-minute window, selected-file Read/Write/Edit and an isolated worktree outside the cloud-sync folder. It saves progress incrementally and returns a small final receipt rather than a whole HTML document in structured completion. Deterministic checks run outside the worker. The 6.1 lead owns browser QA, integration and deliberate promotion. Worker OS commands, browser/network, MCP and delegation stay disabled. There is no automatic retry, provider switch or patch promotion.
Live compatibility checks returned both exact 5.5 identities. Sonnet identified the seeded null-input defect. Opus saved the synthetic interface and progress files; declared deterministic checks and the lead's browser interaction check passed. The first Opus attempt was retained as an environment/postflight failure: cloud-sync software created extra hard links. Worktree placement was corrected, keeping the hard-link rejection intact, and a separately recorded smoke then passed. A small smoke proves this path can run; it is not a new model-quality benchmark, a guarantee on large designs or a native continuation fix.
The same design runtime and a disabled-by-default example profile are included in the public toolkit. Personal accounts, billing evidence and machine paths stay private. The legacy Claude Bridge overlay remains disabled locally. Subscription checks and spending safeguards remain in force.
GPT-6.1 Sol Medium leads TeamLoop. GPT-6 Sol is excluded from active workers and fallback. This is the operator's trust and instruction-following decision, not a claim that 6.1 won every benchmark. Historical GPT-6 Sol records remain unchanged. The later explicit specialist activation above records the pinned Sonnet/Opus roles in roster v1.13.0; the benchmark scores remain frozen.
Opus 5.5 led the original three bounded designs at 94.33/100 and reached 95 after the separate permitted design-direction repair. Sonnet 5.5 improved substantially over Sonnet 5 on the full-resolution critique, 73 versus 29, while both newer Claude models still missed source-audit coverage. Haiku's early promise justified a fair feature-ownership test; that test exposed one startup failure and one useful but imperfect app. There is no single honest all-purpose winner.
Short synthetic screen · All 65 original replay outcomes · Claude failure audit and recovery · OpenAI file-backed cohort · Haiku versus 6.1 · Nine new runnable examples
The October work includes a 29-attempt screen, 65 scheduled original-replay primaries, recovery diagnostics and pilots, three accepted OpenAI file-plus-JavaScript primaries, and four matched feature primaries. These counts are different units, not one pooled benchmark. Masked scores are AI-assisted judgments, not Mark's personal ratings. This was my personal experiment, not an official Glean benchmark.
Twenty-nine attempts across eleven small task packets completed on October 24: twelve OpenAI and seventeen Anthropic. This was a screen for bounded repairs, task consoles, evidence discipline and planted review defects. It was not the original roster benchmark, so the original tasks were subsequently replayed separately. No hidden retries or model-output repairs were used.
| Matched comparison | Observed outcome | Interpretation |
|---|---|---|
| Sonnet 5 vs 5.5, High | Same six accessibility, five lifecycle and six invitation targets; three-task time 156.5s vs 48.9s | 5.5 stronger critic candidate here; both need naming-precision qualification. |
| Sonnet 5.5 Medium vs High | Same corrected issue coverage on two tasks; 26.3s vs 34.2s combined | Cached Medium repeats confound effort-versus-speed inference. |
| Opus 5.5 vs Sonnet 5.5 High | Both found five lifecycle and five migration targets; 61.7s vs 49.1s | No unique Opus target on these explicit short cases. |
| GPT-6 Sol vs GPT-6.1 Sol Medium | Both passed exercised repair, console and evidence criteria; 87.2s vs 162.1s | No pass/fail advantage or 6.1 speed advantage in this screen. |
| GPT-6 Luna vs GPT-5.6 Terra Medium | Both passed pagination and console checks; strict money 26/26 vs 25/26 | The Terra boundary used exact BigInt arithmetic; ambiguous oracle case excluded from capability ranking. |
| Haiku 5.5 vs Sonnet 5.5 Medium | Both exact extraction and three requirements defects; Haiku added two unsupported compaction contradictions | Candidate constrained checking, not autonomous contradiction judgment. Combined time 32.2s vs 21.5s. |
| Requested model / effort | Attempts completed |
|---|---|
| claude-sonnet-5 high | 3 |
| claude-opus-5-5 high | 2 |
| gpt-6-sol medium | 3 |
| gpt-6.1-sol medium | 3 |
| claude-sonnet-5-5 high | 4 |
| claude-sonnet-5-5 medium | 5 |
| claude-haiku-5-5 medium | 3 |
| gpt-6-luna medium | 3 |
| gpt-5.6-terra medium | 3 |
All 29 outputs were runner-valid. The functional suite recorded 155 mechanical passes and one disputed money interpretation across 156 cases; four console apps passed the exercised browser checks. The accessible-name oracle was corrected symmetrically: a placeholder supplied a browser fallback name, so an absent name could not be asserted. Pure-function checks used compilation and a DOM stub, not full browser validation of every repair demo. The lead knew assignments; assertions were not fully preregistered. Exact requested Claude identities were returned; Codex backend identity was not independently attested. Candidate recommendations did not automatically change routing.
The original large-build delivery results describe the tested CLI route. They do not establish Claude's quality ceiling. The installed Claude Code CLI 2.1.310 could exhaust a response on thinking, discard a standalone incomplete thinking block during request normalization, and then ask the model to resume that missing thought. Source-and-trace inspection found this pattern in eight captured non-delivery streams, covering twenty transitions. Two older buffered timeouts lacked enough evidence to name an internal cause.
This is not a blanket StructuredOutput block: successful controls used the same completion contract. Thinking sharing a message with a tool call can survive normalization. The traces do not explain why the first response spent its budget thinking, and they are not direct captures of outgoing HTTP bodies. A wrapper also flattened native failures into a generic worker-exit message, making diagnosis harder.
A small baseline diagnostic reproduced four thinking-only responses, three continuation nudges and no deliverable. Explicit recovery guidance returned valid JSON but an unknown answer, so it did not prove task recovery. No verified native fix or remove-the-output-cap switch was established. The response ceiling was already 128,000 tokens including thinking. Durable files preserve work across boundaries; they do not prove this CLI behavior fixed.
| File-only Failure Garden pilot | Outcome / time | What survived validation | Remaining defect |
|---|---|---|---|
| Opus 5.5 High | 45-minute deadline; recovered HTML, not completed; unscored | 185,049-byte app; core 600-trial monthly model works | Missing final completion; phone overflow; mitigation comparison uses different trial counts; independence explanation overclaims. |
| Sonnet 5.5 High | Completed in 34m09s; unscored pilot | 210,689-byte app; seeded monthly model and core flows work; fits tested widths | Export can combine pending seed/trial settings with old results. Uncertainty and representative-path qualifications remain. |
| GPT-6 Sol control | Not qualified; no quality result | Infrastructure evidence only | Could not establish the agreed file-only tool contract. |
Same original Failure Garden brief, High effort, 45 minutes and 512 KiB. Selected file Read/Write/Edit only, without a computation tool. These are separate pilot artifacts, not replacements for frozen primary scores. Sonnet's stale export is a confirmed provenance defect, not a production-ready pass. The Opus snapshot is recoverable work, not a completed success. An earlier conversational claim that Desktop Code delivered an estimator was corrected: no supported Desktop model call ran. There is no Desktop benchmark result.
The next Failure Garden cohort added isolated pure JavaScript computation to selected file Read/Write/Edit. GPT-6 Sol, GPT-6 Luna and GPT-5.6 Terra each received the same original brief and rubric, High effort, a 45-minute ceiling and a 512 KiB HTML cap. No OS commands, browser, network, imports or delegation were authorized. No lead source repairs or best-of selection followed.
| Requested configuration | Time | Blind quality /100 | Full functional gate | Observed behavior |
|---|---|---|---|---|
| GPT-6 Sol High | 7m23s | 70 | Fail | Deepest model of the trio, but default and two other KPI modes produce NaN; phone overflows. |
| GPT-6 Luna High | 5m53s | 49 | Fail | Undefined names break startup and result rendering. Export salvages finite ranges and dissent, but no representative path. |
| GPT-5.6 Terra High | 6m10s | 64 | Fail | Finite repeatable results, replay and export; resource limits are soft penalties, numerical mechanics shallow, tablet overflow. |
Sol omitted a zero default for a governance coefficient. Luna had multiple missing functions/state references, so repairing one typo would not establish correctness. Terra allowed $12.6M and 24 capacity units against $12M and 18; its replay was an authored template rather than a selected simulation trial.
One approval-blocked Sol setup attempt is preserved as infrastructure, excluded from model quality; the declared replacement used the same corrected contract as Luna and Terra. Requested model pins were recorded, but terminal backend identity was unavailable. Masked AI-assisted scoring used full sources, nine screenshots and observed checks. All three scores are quality judgments, not pass certificates. This file-plus-JavaScript cohort is not matched to the file-only Claude pilot. It cannot establish a cross-provider winner or isolate workflow effects from generation variation. Historical GPT-6 Sol remains visible here but is excluded from active TeamLoop routing.
GPT-6.1 Sol stays the default builder. Haiku did not earn an automatic feature-owner route. The first four primaries compared complete features, not tiny microtasks: one estimator and one evidence-aware comparison/import app per configuration. All four delivered HTML and valid manifests. A delivered file can still be a broken product.
| Feature | Practical configuration | Blind quality /100 | Approx. time | Observed result |
|---|---|---|---|---|
| Estimator | Haiku 5.5 Default | 26 | 4m | Startup exception: totals have data-testid attributes but code looks for missing IDs. No rows; displayed amounts stay at zero. |
| Estimator | GPT-6.1 Sol Medium | 94 | 3m09s | Passed five money-oracle cases, eight invalid-input cases, undo, committed-state export and unchanged iframe integration. |
| Comparison / import | GPT-6.1 Sol Medium | 95 | 4m29s | Passed observed state, import and provenance checks, including strict precision rejection. |
| Comparison / import | Haiku 5.5 Default | 87 | 2m44s | Useful core flows, larger readable controls and clear explanations. Accepts 1.000000001 then rounds to 1.00, violating the two-decimal contract. |
Each pair received identical frozen brief, schema and fixture files, the same shared Read/Write/Edit/Compute service, isolated writable output files, 128 KiB and 45 minutes. Computation used fresh isolated JavaScript sessions with empty tool lists. Twelve Python and eight Node qualification checks covered boundaries and regression behavior. No model-operated OS, browser, network, imports or delegation were permitted.
Provider presentation was not byte-identical: Codex still required its native routing host, whose hidden computation cannot be fully audited in JSONL. Exact Claude identities were returned; 6.1 was requested and pinned but not terminal-verified. Default Haiku versus Medium 6.1 tests practical configurations, not equal latent compute or pure model-only superiority.
Both comparison apps handled zero versus null, selected means, six invalid imports atomically, imported claims versus verified evidence, safe text labels, undo, reset and round-trip export. The near-hundredth precision case was triggered by source review and then applied identically to both; it was supplemental, not an originally enumerated fixture. All four apps fit wide, tablet and phone views. Ten-step history and unchanged iframe checks passed on the three functioning apps. Natural Tab-only navigation across the entire comparison apps and screen-reader behavior remain unverified.
The rubric reviewer received masked sources, observations and twelve screenshots; scores were locked before joining identities. No speed or token bonus applied. One nine-second 6.1 invocation was interrupted by a watchdog misreading permission prose in a successful Read as an error. That infrastructure receipt stays excluded; a declared replacement supplies the primary. The diagnostic-only fix did not change inputs, permissions, tools or model settings. Haiku's broken estimator is preserved without repair.
One attempt per configuration on two features is a small practical trial. Haiku's comparison score is eight points behind 6.1, outside the predeclared five-point tolerance, with a known data-fidelity defect. Active lead minutes were not directly measured, so no 25% labor-saving claim is justified. Earlier Haiku bounded-design strengths still count; they do not establish a dependable calculator specialty. A narrow draft role remains a candidate. No second stage or automatic promotion followed.
These synthetic HTML artifacts are byte-for-byte copies of the retained outputs. No cleanup repairs. Open the paired feature apps side by side, or inspect the recovered simulators. Scores belong to their stated cohort; unscored pilot snapshots do not get an invented grade. Known broken apps deliberately remain broken.
GPT-6.1 Sol Medium
Money, invalid drafts, undo and committed-state exports worked in observed checks.
Inspect original build
Haiku 5.5 Default
A serious arithmetic design is unreachable because startup looks for IDs absent from its own HTML.
Inspect original build
GPT-6.1 Sol Medium
Try selection, assessment, imports, evidence provenance and undo. Strict precision rejection passed.
Inspect original build
Haiku 5.5 Default
Readable controls and detailed hints. Strict import precision fails for a near-hundredth value.
Inspect original build
Sonnet 5.5 High
A rich monthly simulator. Changing pending trial settings before export can mislabel old results.
Inspect original build
Opus 5.5 High
Useful 600-trial model survived. No final completion; phone overflow and counterfactual qualifications remain.
Inspect original build
GPT-6 Sol High (historical)
Rich feedback model, broken default NaN results and phone overflow. Removed from active routing.
Inspect original build
GPT-6 Luna High
Undefined names prevent normal operation. Finite export fragments are salvage, not end-to-end success.
Inspect original build
GPT-5.6 Terra High
Finite repeated results and replay, but budget/capacity are soft penalties and the tablet layout overflows.
Inspect original buildThese are experimental local-style demos, not production tools. Use synthetic inputs. A successful screenshot is not a functional pass. The earlier six examples remain in the original gallery above.
Keep one accountable 6.1 lead for scoping, integration and verification. Use complete, testable feature contracts when exploring specialist delegation. Preserve intermediate files for recovery, but verify the actual product before accepting delivery. Select Opus design and Sonnet critique candidates by the task; neither a timeout nor a high screenshot score settles the question alone. Haiku stays a narrow draft candidate rather than an unattended calculator owner.
Historical cohorts retain their exact contracts. The file-only Claude pilot and file-plus-JavaScript OpenAI pilot cannot support a common leaderboard. The newer four-primary feature trial matched authorized useful tools, with explicit transport limitations. None of these single-attempt studies establishes a latency distribution, broad model superiority or a causal version improvement.
| Worker reporting only | Input | Output including reasoning | Scope limit |
|---|---|---|---|
| OpenAI file + JS cohort | 2,119,129 total; 1,940,352 cached subset | 89,404 total; 15,440 reasoning subset | Includes blocked setup; excludes lead/review overhead. |
| Sonnet file-only delivery | 54 fresh; 395,714 cache creation; 6,458,864 cache read | 283,804 total; 167,998 thinking subset | Opus deadline terminal usage unavailable, not zero. |
| Haiku feature pair | 16 fresh; 108,995 cache creation; 69,898 cache read | 100,113 total; 68,080 thinking subset | Two accepted primaries only. |
| 6.1 feature pair | 407,728 total; 355,968 cached subset | 24,017 total; 3,237 reasoning subset | Interrupted invocation terminal usage unknown; supervision excluded. |
Cached input and reasoning are subsets where labeled; do not add them twice. Provider reporting categories differ. Subscription allowance was checked around calls, with Claude credits and auto-reload off; no paid API fallback or overflow was used. Account-wide rounded meters include unrelated activity and do not establish per-model quota cost. No defensible conversion from these tokens to subscription dollars, percentage savings or lead labor savings is claimed. Raw prompts, native traces, account screenshots and private run identifiers are not published.
The original frozen tasks were replayed with matched current controls and the same 45-minute ceiling for every design and review cell. This is a dated replay of existing benchmark families, not a rewrite of historical scores.
Sixty-five primary configurations covered six newer OpenAI/Anthropic versions and two matched controls. Opus 5.5 led the three bounded design briefs at 94.33/100; GPT-6.1 Sol scored 94 on the source-grounded engineering lab. Larger builds exposed substantial non-delivery in the Claude CLI route. The subsequent technical audit qualifies those failures as comparisons of underlying model ability; see the separate recovery chapter. Accepted quality, failed delivery and single neutral repairs remain separate. Internal Codex reviewers supplied masked rubric judgments; the lead froze scores before joining identities. Mark did not personally score this replay.
Bounded designs retain CLI default effort; larger builds and specialists use explicit High. An existing Opus default preference may resolve differently and is not returned by the CLI. The initial Opus High bounded supplement stays outside primary means. No speed or length bonus was awarded. Eight original WebP references and all original specialist source snapshots were preserved.
Quality below is the original rubric score for an accepted output. Missing quality is unscored; delivery-adjusted contribution is zero only for the scheduled failed or invalid delivery. Failed gates can coexist with partial quality credit. Repairs are shown separately, including unsuccessful repairs. The gate column refers to the corrected Deep browser protocol; specialists retain their separate checks.
Download the 65-row public replay CSV. Run identifiers are excluded. Read the later CLI qualification before interpreting large-build non-delivery.
| Task | Model / effort | Primary delivery | Quality /100 | Delivery-adjusted /100 | Primary gate | Single neutral repair |
|---|---|---|---|---|---|---|
| design-ops-command-center | gpt-6.1-sol / default | completed | 92 | 92 | N/A | Not applied |
| design-ops-command-center | gpt-6-sol / default | completed | 87 | 87 | N/A | Not applied |
| design-ops-command-center | gpt-6-luna / default | completed | 77 | 77 | N/A | Not applied |
| design-ops-command-center | claude-opus-5-5 / default | completed | 94 | 94 | N/A | Not applied |
| design-ops-command-center | claude-sonnet-5-5 / default | completed | 87 | 87 | N/A | Not applied |
| design-ops-command-center | claude-haiku-5-5 / default | completed | 85 | 85 | N/A | Not applied |
| design-renovation-estimator | gpt-6.1-sol / default | completed | 44 | 44 | N/A | Not applied |
| design-renovation-estimator | gpt-6-sol / default | completed | 88 | 88 | N/A | Not applied |
| design-renovation-estimator | gpt-6-luna / default | completed | 74 | 74 | N/A | Not applied |
| design-renovation-estimator | claude-opus-5-5 / default | completed | 96 | 96 | N/A | Not applied |
| design-renovation-estimator | claude-sonnet-5-5 / default | completed | 86 | 86 | N/A | Not applied |
| design-renovation-estimator | claude-haiku-5-5 / default | completed | 90 | 90 | N/A | Not applied |
| design-team-loop-console | gpt-6.1-sol / default | completed | 96 | 96 | N/A | Not applied |
| design-team-loop-console | gpt-6-sol / default | completed | 90 | 90 | N/A | Not applied |
| design-team-loop-console | gpt-6-luna / default | completed | 86 | 86 | N/A | Not applied |
| design-team-loop-console | claude-opus-5-5 / default | completed | 93 | 93 | N/A | Not applied |
| design-team-loop-console | claude-sonnet-5-5 / default | completed | 81 | 81 | N/A | Not applied |
| design-team-loop-console | claude-haiku-5-5 / default | completed | 87 | 87 | N/A | Not applied |
| assisted-agent-fleet | gpt-6.1-sol / high | completed | 46 | 46 | Fail | completed; quality 93; gate Pass |
| assisted-agent-fleet | gpt-6-sol / high | completed | 84 | 84 | Pass | Not applied |
| assisted-agent-fleet | gpt-6-luna / high | completed | 59 | 59 | Fail | completed; quality 62; gate Fail |
| assisted-agent-fleet | claude-opus-5-5 / high | timed_out | Unscored | 0 | Not delivered | Not applied |
| assisted-agent-fleet | claude-sonnet-5-5 / high | timed_out | Unscored | 0 | Not delivered | Not applied |
| assisted-agent-fleet | claude-haiku-5-5 / high | failed | Unscored | 0 | Not delivered | Not applied |
| assisted-failure-simulator | gpt-6.1-sol / high | completed | 92 | 92 | Pass | Not applied |
| assisted-failure-simulator | gpt-6-sol / high | completed | 84 | 84 | Pass | Not applied |
| assisted-failure-simulator | gpt-6-luna / high | completed | 37 | 37 | Fail | completed; quality 31; gate Fail |
| assisted-failure-simulator | claude-opus-5-5 / high | timed_out | Unscored | 0 | Not delivered | Not applied |
| assisted-failure-simulator | claude-sonnet-5-5 / high | timed_out | Unscored | 0 | Not delivered | Not applied |
| assisted-failure-simulator | claude-haiku-5-5 / high | failed | Unscored | 0 | Not delivered | Not applied |
| assisted-futures-lab | gpt-6.1-sol / high | completed | 92 | 92 | Pass | Not applied |
| assisted-futures-lab | gpt-6-sol / high | completed | 84 | 84 | Fail | completed; quality 84; gate Fail |
| assisted-futures-lab | gpt-6-luna / high | completed | 74 | 74 | Pass | Not applied |
| assisted-futures-lab | claude-opus-5-5 / high | timed_out | Unscored | 0 | Not delivered | Not applied |
| assisted-futures-lab | claude-sonnet-5-5 / high | completed | 88 | 88 | Pass (adjudicated lens probe) | Not applied |
| assisted-futures-lab | claude-haiku-5-5 / high | failed | Unscored | 0 | Not delivered | Not applied |
| assisted-design-direction | gpt-6.1-sol / high | completed | 96.5 | 96.5 | Pass | Not applied |
| assisted-design-direction | gpt-6-sol / high | completed | 90 | 90 | Fail | completed; quality 86; gate Pass |
| assisted-design-direction | gpt-6-luna / high | completed | 83 | 83 | Fail | completed; quality 85.5; gate Pass |
| assisted-design-direction | claude-opus-5-5 / high | completed | 89.5 | 89.5 | Fail | completed; quality 95; gate Pass |
| assisted-design-direction | claude-sonnet-5-5 / high | completed | 91 | 91 | Fail | completed; quality 94; gate Pass |
| assisted-design-direction | claude-haiku-5-5 / high | completed | 89 | 89 | Fail | completed; quality 92.5; gate Pass |
| design-ops-command-center | gpt-5.6-sol / default | completed | 83 | 83 | N/A | Not applied |
| design-ops-command-center | claude-sonnet-5 / default | completed | 81 | 81 | N/A | Not applied |
| design-renovation-estimator | gpt-5.6-sol / default | completed | 81 | 81 | N/A | Not applied |
| design-renovation-estimator | claude-sonnet-5 / default | completed | 85 | 85 | N/A | Not applied |
| design-team-loop-console | gpt-5.6-sol / default | completed | 84 | 84 | N/A | Not applied |
| design-team-loop-console | claude-sonnet-5 / default | completed | 88 | 88 | N/A | Not applied |
| assisted-agent-fleet | gpt-5.6-sol / high | completed | 72 | 72 | Pass | Not applied |
| assisted-agent-fleet | claude-sonnet-5 / high | completed | 29 | 29 | Fail | completed; quality 63; gate Fail |
| assisted-failure-simulator | gpt-5.6-sol / high | completed | 69 | 69 | Fail | completed; quality 74; gate Fail |
| assisted-failure-simulator | claude-sonnet-5 / high | timed_out | Unscored | 0 | Not delivered | Not applied |
| assisted-futures-lab | gpt-5.6-sol / high | completed | 89 | 89 | Pass (adjudicated lens probe) | Not applied |
| assisted-futures-lab | claude-sonnet-5 / high | timed_out | Unscored | 0 | Not delivered | Not applied |
| assisted-design-direction | gpt-5.6-sol / high | completed | 89 | 89 | Fail | completed; quality 91.5; gate Pass |
| assisted-design-direction | claude-sonnet-5 / high | completed | 78 | 78 | Fail | completed; quality 78; gate Fail |
| specialist-glm-task-surface-lab | gpt-6.1-sol / high | completed | 94 | 94 | N/A | Not applied |
| specialist-glm-task-surface-lab | gpt-6-sol / high | completed | 78.5 | 78.5 | N/A | Not applied |
| specialist-glm-task-surface-lab | gpt-5.6-sol / high | completed | 80.5 | 80.5 | N/A | Not applied |
| specialist-glm-task-surface-lab | claude-opus-5-5 / high | completed | 79 | 79 | N/A | Not applied |
| specialist-nemotron-vs-opus-task-surface-audit | claude-opus-5-5 / high | completed | 66 | 66 | N/A | Not applied |
| specialist-nemotron-vs-opus-task-surface-audit | claude-sonnet-5-5 / high | completed | 70 | 70 | N/A | Not applied |
| specialist-muse-vs-sonnet-starforge-assisted-webp | claude-opus-5-5 / high | completed | 84 | 84 | N/A | Not applied |
| specialist-muse-vs-sonnet-starforge-assisted-webp | claude-sonnet-5-5 / high | completed | 73 | 73 | N/A | Not applied |
| specialist-muse-vs-sonnet-starforge-assisted-webp | claude-sonnet-5 / high | completed | 29 | 29 | N/A | Not applied |
80 benchmark executions include planned primaries, one separately labeled bounded effort supplement, single neutral repairs and one additional execution after external app/server interruption. That interruption and its declared replacement count as one planned configuration. Two tiny maintenance diagnostics are outside this count. No incomplete fragment was promoted into an accepted artifact. A final eligibility audit found genuine mobile overflow previously overlooked while resolving a harmless control-discovery issue; the affected model then received its same single neutral repair. Earlier locked judgments were preserved, and the added output was scored by a fresh reviewer without creator or process-role context.
Completed Claude identities were checked. OpenAI receipts retain exact requested IDs but do not independently attest the backend identity. Some Claude CLI attempts hit a 45-minute deadline or repeated internal output-limit continuation; these are operating-route outcomes, not proof of intrinsic inability. Earlier opaque deadline causes remain unknown. Passive logging changed during the replay without changing model-visible prompts, acceptance caps or deadlines. A clarified completion canary did not reproduce the failures and was not adopted.
Masking was limited: the lead knew dispatch identities. Two recorded identity/control hints caused entire direction and engineering cohorts to be reassigned to a fresh reviewer. A human-author footer in two generated outputs was treated as unsupported attribution, not verified authorship or a creator-model identity; no creator inference was made. Historical human-scored cohorts remain unchanged. This replay used AI-assisted rubric judgments, and different historical deadlines prevent causal version-improvement claims. Token metadata is not a subscription bill; no paid API, overflow or silent provider substitution was used.
Fable 5.1 already has both an October 3 creative success and an October 5 timeout; it is not newly untested. Astra 6 already has compound real-work evidence. Opus 4.8 remains historical evidence with current access unverified. Argon is general purpose, but outside this two-provider execution.
This is useful as a transparent account of one practitioner learning how to assign work. It is not evidence that the same ordering will repeat across other organizations, prompts, dates, or interfaces.
Freeze the tasks. Preserve every attempt. Separate quality from reliability. Test what actually works. Classify infrastructure failures. Then turn the findings into routing rules you are willing to change.
The exact ordering. Models, subscription limits, wrappers, and interfaces change constantly. Samples were small and tasks synthetic. Earlier human-scored cohorts and the later masked AI-assisted judgments retain their separate methods.
Bounded-trial scoring used written rubrics covering functional integrity, hierarchy, interaction, accessibility, originality, and implementation. Long-running scoring separated completed-artifact quality, portfolio value, and strict operational delivery. Repairs and salvages kept their labels. I recorded timing and provider-reported usage when the interface exposed them, but I did not pretend token counts from different providers were directly comparable cost units.
This public edition contains no employer or client data, credentials, account identifiers, personal filesystem paths, private prompts, or proprietary work products. This was my personal experiment, not an official Glean benchmark.