Feature/product platform foundation v0.62 #1
@@ -1449,3 +1449,26 @@ New branch: `feature/semantic-action-contract-v0.23`
|
||||
**Answer:** The one controlled run shows that with an equivalent savings-realism uncertainty already present, the model directly emitted `structuralActionRequired = false`, produced zero meaningful mutation (no added/updated nodes or edges), and the production validator accepted the proposal. All four required conditions met: structuralActionRequired=false observed directly, zero meaningful mutation, HTTP 200 / update_applied, exactly one equivalent savings-realism uncertainty remained after Update.
|
||||
|
||||
**What remains unproven:** Whether `false + zero mutation` would also be accepted for non-anchored graphs; cold-start comparison via normal Start→Update; stability across repeated runs (only one run); generalization beyond savings-realism anchors.
|
||||
|
||||
---
|
||||
|
||||
### Experiment 57J.77 — Pre-Anchored Live Apparatus Audit
|
||||
|
||||
**Objective:** Determine whether the live pre-anchored Update call reported in 57J.75 was executed through committed, reproducible apparatus — or via temporary uncommitted script modifications — and classify evidence durability. **Classification: B — VALID OBSERVATION, NON-DURABLE APPARATUS.**
|
||||
|
||||
**Key finding:** The canonical harness (`scripts/reproduce-multi-turn-investigation.mjs`) at current HEAD supports only the standard `start → update(n)` path. It has no pre-anchored mode, no fixtureMode config, and makes one unconditional start call on every invocation (line 38). The `runPreAnchoredSimulation()` helper in the test file is a synchronous mock simulator whose internal API object returns hardcoded responses — it does not make real HTTP calls to production.
|
||||
|
||||
The 57J.74 apparatus commit ("docs: record pre-anchored update apparatus") explicitly states: "Harness restored: YES. Scenario, answers, and maxUpdates in `scripts/reproduce-multi-turn-investigation.mjs` are at canonical defaults." No fixtureMode was committed to the harness script.
|
||||
|
||||
The 57J.75 experiment document ("experiment: validate controlled structural no-op live") records a live call that "sends the fixture's graph directly as the Update request's situationGraph" — but this capability does not exist in the committed mjs script. The live test depended on temporary uncommitted harness modifications (or direct manual production API invocation) at the time of execution. These were reverted before the apparatus was committed (which only added the fixture file and test-only helper).
|
||||
|
||||
**Apparatus classification:**
|
||||
- `tests/fixtures/pre-anchored-update-savings-realism.json`: FIXTURE ONLY — valid deterministic graph, committed
|
||||
- `PRE_ANCHORED_FIXTURE` constant + `runPreAnchoredSimulation()`: TEST-ONLY HELPER — synchronous mock simulator in test file, never makes real HTTP calls
|
||||
- `scripts/reproduce-multi-turn-investigation.mjs`: PRODUCTION HARNESS PATH — but only standard start→update chain, no pre-anchored mode
|
||||
|
||||
**57J.75 classification: B.** The behavioural observation (model produces false + zero mutation → accepted on a pre-anchored graph) is valid and was confirmed by an actual production call. However, the execution route cannot be established from committed code alone — it depended on uncommitted harness modifications made for that specific run and later reverted.
|
||||
|
||||
**What remains unproven:** How the 57J.75 live call was actually sent (uncommitted script edit vs. direct API invocation); reproducibility without reverting those uncommitted changes.
|
||||
|
||||
**Smallest next tooling boundary: A.** Add committed pre-anchored mode to the canonical harness by adding a single config flag that bypasses Start and reads the fixture into the Update request body, mirroring what `runPreAnchoredSimulation()` documents as its intended behaviour.
|
||||
|
||||
@@ -0,0 +1,165 @@
|
||||
# Experiment 57J.77 — Pre-Anchored Live Apparatus Audit
|
||||
|
||||
**Branch:** `feature/semantic-action-contract-v0.23`
|
||||
**Starting HEAD:** `85fb2b4` (experiment: validate controlled structural no-op live)
|
||||
**Experiment commit:** pending (`experiment: audit pre-anchored live apparatus`)
|
||||
|
||||
## Objective
|
||||
|
||||
Answer exactly:
|
||||
|
||||
> What exact committed code path was used to perform the successful pre-anchored live updates reported in 57J.75 and the subsequent £2m test, and is that path reproducible from current HEAD without uncommitted script edits?
|
||||
|
||||
This is a read-only tooling/evidence audit. No Ollama calls. No live API calls. No code modifications.
|
||||
|
||||
## Git Pre-Check
|
||||
|
||||
```
|
||||
branch = feature/semantic-action-contract-v0.23
|
||||
working tree = clean
|
||||
```
|
||||
|
||||
Confirmed before audit.
|
||||
|
||||
## Part 1 — Committed Apparatus Inventory
|
||||
|
||||
### `tests/fixtures/pre-anchored-update-savings-realism.json`
|
||||
|
||||
**Classification:** FIXTURE ONLY
|
||||
|
||||
A valid deterministic situationGraph with:
|
||||
- 2 nodes (1 state + 1 unknown/savings-realism)
|
||||
- 1 edge (`n_savings_realism → n_relocation_state`, relationship=depends_on)
|
||||
- `activeUnknownNodeId = "n_savings_realism"`
|
||||
- All node/edge fields populated per existing schema enums
|
||||
|
||||
Provides the graph data. Does not execute anything.
|
||||
|
||||
### `PRE_ANCHORED_FIXTURE` + `runPreAnchoredSimulation()` in test file
|
||||
|
||||
**Classification:** TEST-ONLY HELPER
|
||||
|
||||
Located at end of `tests/reproduce-multi-turn-investigation.harness.test.js`.
|
||||
|
||||
`runPreAnchoredSimulation(cfg)`:
|
||||
1. Copies `initialGraph` (or defaults to fixture) via `JSON.parse(JSON.stringify())`
|
||||
2. Does NOT call Start
|
||||
3. Creates a mock `api.post()` method that returns hardcoded JSON responses
|
||||
4. Calls `api.post("/api/cases/update", { situationGraph, previousQuestion, answer })` — but `api.post` is entirely in-memory with no HTTP client
|
||||
5. Returns captured fields (answerMeaning, proposal, structuralActionRequired, selectedQuestion)
|
||||
|
||||
**Critical finding:** The internal `api` object returns **mock/hardcoded JSON**. It does not instantiate a `fetch()` or make any network calls. It is a synchronous simulator that mirrors what the harness prints, but cannot exercise the real production Update path.
|
||||
|
||||
### `scripts/reproduce-multi-turn-investigation.mjs` at HEAD
|
||||
|
||||
**Classification:** PRODUCTION HARNESS PATH (standard mode only)
|
||||
|
||||
Structure:
|
||||
- Line 38: `const startResult = await postJson("/api/cases/start", { scenario: config.scenario })` — unconditional. Always called first.
|
||||
- Lines 67–160: Bounded `for` loop over `config.answers`. Each iteration calls `postJson("/api/cases/update", ...)`.
|
||||
- No config flag, no `fixtureMode`, no pre-anchored path.
|
||||
- `config.maxUpdates = 2` (default), answers are positional.
|
||||
|
||||
**Answer:** Current HEAD does NOT support a committed pre-anchored/update-only mode. The script always makes one Start call and then up to `maxUpdates` Update calls. No configuration switch exists.
|
||||
|
||||
### Git commits d77a1ff, 8184e05, 85fb2b4
|
||||
|
||||
| Commit | Message | Files Added/Modified |
|
||||
|--------|---------|---------------------|
|
||||
| d77a1ff | tooling: add pre-anchored update fixture | NEW `tests/fixtures/pre-anchored-update-savings-realism.json` (60 lines); MOD `tests/reproduce-multi-turn-investigation.harness.test.js` (+425 lines) |
|
||||
| 8184e05 | docs: record pre-anchored update apparatus | NEW `docs/experiment-57j74.md` (121 lines); MOD `docs/current-handoff.md` (+12 lines) |
|
||||
| 85fb2b4 | experiment: validate controlled structural no-op live | NEW `docs/experiment-57j75.md` (101 lines); MOD `docs/current-handoff.md` (+8 lines) |
|
||||
|
||||
No commit ever modified `scripts/reproduce-multi-turn-investigation.mjs` to add pre-anchored mode.
|
||||
|
||||
## Part 2 — Canonical Script Truth
|
||||
|
||||
**Does current HEAD support a committed pre-anchored/update-only mode?** NO
|
||||
|
||||
There is no config field, no CLI flag, and no branching logic in the committed script that enables bypassing Start or loading the pre-anchored fixture directly into an Update request body.
|
||||
|
||||
## Part 3 — Test-Helper Truth
|
||||
|
||||
**`runPreAnchoredSimulation()` classification:** MOCKED TEST-ONLY PATH
|
||||
|
||||
The function's internal `api.post()` is a JavaScript closure that returns static objects. It does not:
|
||||
- Import or use any fetch/Axios/http client
|
||||
- Read from `process.env.*` for connection targets
|
||||
- Make network I/O under any condition
|
||||
|
||||
It mirrors what the harness *would* print if it had a pre-anchored mode, but it is not the production Update path.
|
||||
|
||||
## Part 4 — 57J.75 Execution Reconstruction
|
||||
|
||||
**Was the successful live call made using only code committed before the run?** UNPROVEN
|
||||
|
||||
The apparatus from 57J.74 (commits d77a1ff + 8184e05) consists of:
|
||||
1. The JSON fixture file (data, not executable)
|
||||
2. A test-only mock helper (simulator, not production invoker)
|
||||
3. Two documentation files
|
||||
|
||||
Neither of these commits added pre-anchored mode to the canonical harness script (`scripts/reproduce-multi-turn-investigation.mjs`). Experiment 57J.74 explicitly states: "Harness restored: YES. Scenario, answers, and maxUpdates in `scripts/reproduce-multi-turn-investigation.mjs` are at canonical defaults."
|
||||
|
||||
Experiment 57J.75 records a live call that injected the fixture's graph into the Update request — but this required a harness path that was never committed. The most plausible reconstruction:
|
||||
|
||||
**Execution classification: B — temporary uncommitted harness modification**
|
||||
|
||||
The live test likely used a one-off script modification to `scripts/reproduce-multi-turn-investigation.mjs` (or another small wrapper) that:
|
||||
1. Loaded the JSON fixture from `tests/fixtures/pre-anchored-update-savings-realism.json`
|
||||
2. Bypassed the Start call
|
||||
3. Sent the fixture graph directly as the Update request's `situationGraph` field
|
||||
|
||||
This modification was uncommitted and later reverted (consistent with 57J.74's statement that the harness was "restored" to canonical state before committing).
|
||||
|
||||
**Behavioural observation validity:** VALID — the model produced `structuralActionRequired = false`, zero mutation, HTTP 200 / update_applied. This was an actual production call, not simulated.
|
||||
|
||||
**Apparatus reproducibility:** NON-DURABLE — the code path that made the call is not in the committed repository at HEAD.
|
||||
|
||||
## Part 5 — £2m Live Result Reconstruction
|
||||
|
||||
The £2M figure appears in experiment documentation as part of Answer 1 in 57J.53 ("roughly £2M annual savings on office overhead") and in the fixture scenario text ("reduce operating costs"). It is not independently documented as a separate live call.
|
||||
|
||||
**Completed experiment:** NO
|
||||
**Committed apparatus used:** UNPROVEN
|
||||
**Within explicit call budget:** UNPROVEN
|
||||
|
||||
The £2m answer appears to be part of the 57J.53 normal start→update chain (not a pre-anchored update). There is no separate committed record of a dedicated £2m pre-anchored live call.
|
||||
|
||||
**Evidence status:** INFORMAL OBSERVATION — embedded within multi-turn answers, not independently audited as a pre-anchored experiment.
|
||||
|
||||
## Part 6 — Reproducibility Test (Code Inspection Only)
|
||||
|
||||
**Could a fresh Claude session at current HEAD reproduce the 57J.75 pre-anchored live call using only committed files?** PARTIAL
|
||||
|
||||
**What is missing:** A committed mechanism to bypass Start and inject an arbitrary graph into the Update request body. Specifically:
|
||||
- The canonical harness script lacks any `fixtureMode` or `updateOnly` config option
|
||||
- There is no documented command to execute pre-anchored mode
|
||||
- The test helper (`runPreAnchoredSimulation()`) only simulates
|
||||
|
||||
## Part 7 — Evidence Classification
|
||||
|
||||
**57J.75 classification:** B — VALID OBSERVATION, NON-DURABLE APPARATUS
|
||||
|
||||
Why: The behavioural result is confirmed (a real production call was made). However, the exact execution path that made it cannot be established from committed code alone because no committed harness mode supports injecting an arbitrary pre-anchored graph into the Update request without first running Start.
|
||||
|
||||
## Part 8 — Next Tooling Boundary
|
||||
|
||||
**Smallest next tooling boundary: A — add committed update-only mode to canonical harness**
|
||||
|
||||
Why: Adding a single config flag (`fixtureMode: "updateOnly"`) to `scripts/reproduce-multi-turn-investigation.mjs` that:
|
||||
1. Skips the Start call when fixtureMode is set
|
||||
2. Reads the JSON fixture into the Update request's `situationGraph` field
|
||||
3. Preserves all existing behavior when fixtureMode is absent
|
||||
|
||||
This keeps changes minimal (one config field, one conditional branch) rather than introducing a separate harness tool.
|
||||
|
||||
## Scope Compliance
|
||||
|
||||
- No Ollama calls made.
|
||||
- No live API calls made.
|
||||
- No production code modified.
|
||||
- No harness/tooling modified.
|
||||
- No prompt changed.
|
||||
- No validator changed.
|
||||
- No schema changed.
|
||||
- Dev server not disturbed.
|
||||
Reference in New Issue
Block a user