58 lines · 4 KBtext/plain; charset=utf-8
Onset: version control where nobody waits and nothing gets lost. The on command, its core and onset-server.
| 5 | **Setup notes:** |
| 6 | - The plan named Claude Code and Codex. Codex isn't used here (Codex was retired Oct 22), so the test ran on three Claude models. |
| 7 | - The machine's `claude` CLI login had expired, so each run was a fresh Claude Code subagent. Its whole prompt was the scenarios plus the guide, and it was told to use no tools. It used none. |
| 8 |
| 9 | ## Round 1, guide v1 (18 scenarios) |
| 10 |
| 11 | | Model | Pass | Fail | |
| 12 | |---|---|---| |
| 13 | | Haiku 5.5 | 16 | S7, S14 | |
| 14 | | Sonnet 5.5 | 16 | S7, S14 | |
| 15 | | Opus 5.5 | 16 | S7, S14 | |
| 16 |
| 17 | All three failed the same two scenarios, and for the same reason: the guide didn't contain the command. In each case the agent stopped and said so instead of guessing, which is the behaviour we want. |
| 18 | - **S7 (make billing/ private):** the guide said when not to run `vcs access`, but never showed its syntax. All three ran `vcs access --help`. |
| 19 | - **S14 (new local project, then publish):** the guide had no `init` or `publish`. All three asked the user. |
| 20 |
| 21 | Everything else passed. All three used no git commands, didn't sync before landing, used `restore --at` for the 15-minute rollback, used `undo` to take back a land, and stopped on `waiting`. None reached for `--allow-secrets`. |
| 22 |
| 23 | **What the answers taught us:** |
| 24 | - **Opus noted that `vcs undo` only takes a land back if the land was the last command.** That's a real gap in the design: there was no direct way to revert an older version. Added `vcs undo <version>` to the spec and the guide. |
| 25 | - **Haiku and Opus both pointed out that a live key blocked by the secret check still sits in the change's earlier saves.** Added to the spec: once a flagged secret is removed, it's scrubbed from that change's saves. This is the one case where saves are rewritten. |
| 26 |
| 27 | **Changes for guide v2:** a "Starting a project" section (`init`, `publish`, `import`, login), a "Sharing and access" section with the syntax, `vcs undo <version>`, and `git init`, `git clone` and `git revert` rows in the git table. Two scenarios were added to test them: S19 (revert an older version) and S20 (import from GitHub). |
| 28 |
| 29 | ## Round 2, guide v2 (20 scenarios) |
| 30 |
| 31 | | Model | Pass | Fail | |
| 32 | |---|---|---| |
| 33 | | Haiku 5.5 | 20 | none | |
| 34 | | Sonnet 5.5 | 20 | none | |
| 35 |
| 36 | Opus wasn't re-run: round 1 failed the same scenarios across all three models, and the smallest model is the stricter test. |
| 37 |
| 38 | Notes: |
| 39 | - Both used `vcs undo 48190` for S19 and the exact `vcs import … --to …` form for S20. |
| 40 | - Haiku's S8 left the title as a placeholder ("Your title here"), and its S10 didn't tell the user where the key should go. Both count as minor. |
| 41 | - No answer used a git command, `--allow-secrets`, a sync before landing, or another agent's folder. |
| 42 |
| 43 | **Step 1's exit is met for the dry run.** Round 3 repeats it against the real CLI. |
| 44 |
| 45 | ## Round 3, the real CLI (October 27) |
| 46 |
| 47 | The same test, for real: `tests/agent/setup.sh` builds a sandbox with a `vcs-server`, two people and one workspace per task, each with its situation in place (a real conflict with Sam's landed change, a project where Alice can only read, a host that's down, a live Stripe key, an earlier save to go back to, a git repository to import). A fresh Haiku 5.5 agent got only `tests/agent/tasks.md` and the guide, ran the real `vcs`, and `tests/agent/grade.sh` checked the end state. |
| 48 |
| 49 | | Model | Tasks | End-state checks | |
| 50 | |---|---|---| |
| 51 | | Haiku 5.5 | 14 of 14 | 20 of 20 | |
| 52 |
| 53 | - It used no git commands in workspaces, never synced before landing, and kept the other agent's folder untouched. |
| 54 | - It resolved the conflict keeping both its 50 and Sam's comment, stopped on `waiting` and `offline` (one retry), and removed the live key instead of passing `--allow-secrets`. It also told Alice to rotate the key, since it had been in a save. |
| 55 | - For the restore it found `vcs log --saves` and `--from <change>:1` through `--help`. Both are in the guide now, with a line on what happens to a change taken back out of main, which it was unsure about. |
| 56 |
| 57 | **Step 2's agent exit is met.** |
| 58 |