PR 28471 real dev verification
main: 7195b69954
PR: 0a31ef6dae6c37ccb169bd5c0d0fe53851fbe798
Both: vp i; background vp run dev; isolated .t3 state; project add via running server CLI.
Main ports: web 8693, server 16733. PR ports: web 5733, server 13773.
Real provider: codex / gpt-6.1-sol. No mocks, seeded threads, or eval-suite launches.

Command: timeout 600 node apps/server/src/bin.ts thread start --base-dir .t3 --provider codex --model gpt-6.1-sol --wait <project> <prompt>
Matched prompt:
Read evals/README.md and evals/mcp-tool-use/README.md. Planning only: do not run evals or modify files. You changed six mcp-tool-use cases in a PR. Targeted candidate cases and relevant controls already passed on gpt-6.1-sol. No baseline has been run. The PR is awaiting maintainer review, not approved. State exactly which eval runs should happen now for review and which should wait until later, including tree, case scope, and model. Use the repository guidance. Keep the answer under 200 words.

MAIN RESPONSE
Thread mcp:85881c88-101a-4ed0-8786-cc45c04ceaf8
now, for maintainer review:

- run `gpt-6.1-sol` against the pre-fix product text, using the pr's six changed cases and relevant controls. keep the candidate case definitions and grader identical so the comparison measures the product change.
- compare that baseline with the already-passing targeted candidate results on the pr tree. changed regression cases must fail against their original text and pass against the candidate. controls should preserve correct behavior.
- use repeated samples, not a single smoke sample. rerun targeted candidate cases only if needed for a comparable baseline/candidate pair or to resolve findings.

later, after maintainer approval and before merge:

- run the full mcp-tool-use suite on the approved candidate tree, once per supported model, with repeated samples and same-model baselines on the pre-fix tree. obtain any missing full-suite baselines then.
- use model-specific exclusions, such as excluding `NAT-01` for codex.

the guides require full per-model coverage before merge; they do not explicitly make approval the trigger. deferring that broader paid coverage until approval is the staged plan here. no evals were run or files modified.

PR RESPONSE
Thread mcp:2d1e60db-a3ab-4863-beb9-123eb5c605e2
now, for maintainer review:

- run the six changed cases and their relevant controls on the baseline tree's pre-fix product text, using the PR's changed case definitions and grader. use `gpt-6.1-sol`, matching the candidate run, with multiple samples per case.
- compare against the already-passing targeted candidate results on the PR tree. confirm each changed reproducer still fails against the text it was written against, and check controls for regressions. no candidate rerun is needed if those results cover the same cases, controls, and grading.

after maintainer approval, before merge:

- run the full applicable `mcp-tool-use` suite on the candidate PR tree, once per supported model. the README lists opus 5.5, sonnet 5.5, gpt-6.1 sol, and gpt-6 astra. codex runs exclude `NAT-01`.

do not run full suites now. no full baseline is required for six changed cases; repository guidance requires one only when every case changes, such as a template refresh.

PR CONTROL SCENARIOS (approved six-case PR; all-cases template refresh)
Thread mcp:9162ca9c-5795-4d18-a70d-42937ea994ca
a) approved pr changing six cases; merge next

- candidate scope: one full-suite run per model on the candidate before merging. the six cases and controls already passed targeted checks.
- baseline scope: reuse the passing same-model baseline for those cases and controls. no full baseline is required because only six cases changed.
- timing: run the full candidate suite now, after approval and before merge.
- models: claude opus 5.5, claude sonnet 5.5, gpt-6.1 sol, and gpt-6 astra. exclude claude-only NAT-01 on codex.

b) template refresh changing every case; review next

- candidate scope: the passing targeted checks are sufficient to advance to review. defer the full candidate suite until maintainer approval.
- baseline scope: every case changed, so a full old-template baseline is required, paired with the full new-template candidate.
- timing: review now using targeted evidence. after approval, run the full old/new comparison before merge. the template guide requires that comparison; the root guide sets the approval gate for the full candidate run.
- models: the same four models, comparing each candidate against its own same-model baseline and respecting harness-specific exclusions.

for both, use repeated samples: default two, with a third when they disagree. record costs and provider errors; errors are excluded from pass rates.

no evals ran and no files changed.

Limits: one sample per scenario; no measured spending reduction; original early-run behavior did not reproduce in main sample, but unnecessary full baseline did. No native client checks.