# Dry-run results

Step 1's exit test, from `reports/Upstream version control plan Oct 2026.md`. A fresh agent gets only `../GUIDE.md` and `dry-run-scenarios.md`, with no tools and no repository, and writes the commands it would run. Answers are graded against `dry-run-key.md`.

**Setup notes:**
- The plan named Claude Code and Codex. Codex isn't used here (Codex was retired Oct 22), so the test ran on three Claude models.
- The machine's `claude` CLI login had expired, so each run was a fresh Claude Code subagent. Its whole prompt was the scenarios plus the guide, and it was told to use no tools. It used none.

## Round 1, guide v1 (18 scenarios)

| Model | Pass | Fail |
|---|---|---|
| Haiku 5.5 | 16 | S7, S14 |
| Sonnet 5.5 | 16 | S7, S14 |
| Opus 5.5 | 16 | S7, S14 |

All three failed the same two scenarios, and for the same reason: the guide didn't contain the command. In each case the agent stopped and said so instead of guessing, which is the behaviour we want.
- **S7 (make billing/ private):** the guide said when not to run `vcs access`, but never showed its syntax. All three ran `vcs access --help`.
- **S14 (new local project, then publish):** the guide had no `init` or `publish`. All three asked the user.

Everything else passed. All three used no git commands, didn't sync before landing, used `restore --at` for the 15-minute rollback, used `undo` to take back a land, and stopped on `waiting`. None reached for `--allow-secrets`.

**What the answers taught us:**
- **Opus noted that `vcs undo` only takes a land back if the land was the last command.** That's a real gap in the design: there was no direct way to revert an older version. Added `vcs undo <version>` to the spec and the guide.
- **Haiku and Opus both pointed out that a live key blocked by the secret check still sits in the change's earlier saves.** Added to the spec: once a flagged secret is removed, it's scrubbed from that change's saves. This is the one case where saves are rewritten.

**Changes for guide v2:** a "Starting a project" section (`init`, `publish`, `import`, login), a "Sharing and access" section with the syntax, `vcs undo <version>`, and `git init`, `git clone` and `git revert` rows in the git table. Two scenarios were added to test them: S19 (revert an older version) and S20 (import from GitHub).

## Round 2, guide v2 (20 scenarios)

| Model | Pass | Fail |
|---|---|---|
| Haiku 5.5 | 20 | none |
| Sonnet 5.5 | 20 | none |

Opus wasn't re-run: round 1 failed the same scenarios across all three models, and the smallest model is the stricter test.

Notes:
- Both used `vcs undo 48190` for S19 and the exact `vcs import … --to …` form for S20.
- Haiku's S8 left the title as a placeholder ("Your title here"), and its S10 didn't tell the user where the key should go. Both count as minor.
- No answer used a git command, `--allow-secrets`, a sync before landing, or another agent's folder.

**Step 1's exit is met for the dry run.** Step 2 repeats these scenarios against the real CLI.