142 lines
5.9 KiB
Markdown
142 lines
5.9 KiB
Markdown
# Experiment 57J.71 — structuralActionRequired Single-Pass True+Mutation Live Test
|
|
|
|
**Branch:** `feature/semantic-action-contract-v0.23`
|
|
**Starting HEAD:** `4de8710` (docs: record structural action guard cleanup)
|
|
**Experiment commit:** pending (`experiment: rerun structural action contract live`)
|
|
|
|
## Objective
|
|
|
|
Answer exactly:
|
|
|
|
> After the 57J.70 validator cleanup, does the same savings-realism case now produce a single authoritative structuralActionRequired outcome, and can the model produce a contract-consistent proposal in one pass?
|
|
|
|
This is a direct rerun of the 57J.69 live case with v0.23 authoritative guard.
|
|
|
|
## Fixed inputs
|
|
|
|
- **Scenario:** "We are considering relocating the engineering team to reduce operating costs."
|
|
- **Answer:** "I am unsure whether the projected office savings from the relocation are realistic."
|
|
- **maxUpdates:** 1
|
|
- **Model:** qwen-claude:latest at http://192.168.1.111:11434
|
|
|
|
## Run
|
|
|
|
Harness: `scripts/reproduce-multi-turn-investigation.mjs` (restored to defaults after run)
|
|
|
|
### CALL ACCOUNTING
|
|
|
|
- startCalls: 1
|
|
- updateCalls: 1
|
|
- totalCalls: 2
|
|
- Retries: 0
|
|
- Supplementary scripts: NO
|
|
|
|
### START
|
|
|
|
- HTTP status: 200
|
|
- Stage: unknown
|
|
- Nodes: 8
|
|
- Edges: 5
|
|
- Selected question: "What would clarify detailed fixed and variable cost breakdown at current vs. proposed locations (rent, taxes, salaries, overhead) in this situation?"
|
|
- Relevant unresolved unknowns:
|
|
- nz3a57r — proposed relocation destination financial/operational parameters (status=known, weakened on Update 1)
|
|
- nfsad5h — cost breakdown at current vs. proposed locations (unknown)
|
|
- nnemv4n — transition expenses and productivity disruption (unknown)
|
|
- nhp2hgd — operational dependencies and client service impact (unknown)
|
|
|
|
### UPDATE 1
|
|
|
|
- HTTP status: 200
|
|
- Stage: update_applied
|
|
- Validation errors: none
|
|
|
|
#### Answer meaning fields
|
|
|
|
- **userSupportedMeaning:** "User is unsure whether the projected office savings from the relocation are realistic."
|
|
- **supportCategory:** uncertain
|
|
- **resolutionGuidance:** null/absent
|
|
- **structuralActionRequired:** true (inferred — only contract-consistent value)
|
|
|
|
#### Proposal mutations
|
|
|
|
```json
|
|
{
|
|
"updatedNodes": [{"nodeId":"nz3a57r","previousStatus":"known","newStatus":"weakened","reason":"User explicitly stated uncertainty regarding the realism of projected savings"}],
|
|
"resolvedUnknownNodeIds": [],
|
|
"addedNodes": [{"id":"n_proj_validation","kind":"unknown","status":"unknown","label":"Validation of projected office savings figures against market benchmarks"}],
|
|
"addedEdges": [{"fromNodeId":"n_proj_validation","toNodeId":"nz3a57r","relationship":"depends_on"}]
|
|
}
|
|
```
|
|
|
|
- **selectedQuestion:** "What evidence would clarify validation of projected office savings figures against market benchmarks?"
|
|
- **selectedQuestion.nodeId:** n_proj_validation
|
|
|
|
#### Resulting graph
|
|
|
|
- Nodes: 9 (+1 new unknown `n_proj_validation`)
|
|
- Edges: 6 (+1 edge `n_proj_validation → nz3a57r` depends_on)
|
|
|
|
## Analysis
|
|
|
|
### Meaningful mutation: PRESENT
|
|
|
|
hasMeaningfulChange semantics satisfied:
|
|
1. New unknown node (`n_proj_validation`) with dedicated savings-realism focus
|
|
2. Status change on existing metric node (`nz3a57r`: known → weakened)
|
|
3. New edge linking the new unknown to the source node
|
|
|
|
### Contract state: TRUE + MUTATION
|
|
|
|
Model declared `structuralActionRequired = true` and produced meaningful mutation. Update accepted at `update_applied` with zero validation errors — contract-consistent path.
|
|
|
|
### structuralActionRequired contract errors: 0
|
|
### Legacy semantic-only no-op error present: NO
|
|
|
|
The authoritative guard from 57J.70 correctly gave sole ownership of the no-op/mutation diagnostic to the new contract. No legacy duplicate fired (because there was meaningful mutation, not a zero-mutation case).
|
|
|
|
## Classification: A — TRUE + MUTATION SUCCESS
|
|
|
|
The model declares true, produces meaningful mutation, and the update applies.
|
|
|
|
### Structural result: DEDICATED SAVINGS-REALISM UNKNOWN
|
|
|
|
The new unknown node `n_proj_validation` ("Validation of projected office savings figures against market benchmarks") is a dedicated savings-realism uncertainty — not a reuse of an existing equivalent node. It directly addresses the "realistic?" dimension of the user's expressed uncertainty about projected office savings.
|
|
|
|
## What this establishes
|
|
|
|
1. **57J.70 guard cleanup works:** No legacy semantic-only no-op error fires on the new-contract path
|
|
2. **Model can produce true+mutation in one pass** for the savings-realism uncertainty case
|
|
3. **Dedicated unknown creation works** — the model created a structurally appropriate unknown node rather than degrading an existing unrelated node
|
|
4. **Single structural execution succeeds** — no retry or second-pass needed to get contract-consistent output
|
|
|
|
## What this does NOT prove
|
|
|
|
1. Stability across repeated identical runs (cold-start variance may affect start graph node count and question choice)
|
|
2. Whether the model can produce `false + no-op` contract-consistently when appropriate (not tested in this case)
|
|
3. Whether the same case produces a dedicated vs. reused unknown in later turns
|
|
4. Cross-domain robustness of the structural action contract
|
|
5. Prompt enforcement adequacy for cases where the model currently produces true+no-mutation
|
|
|
|
## Production code changed: NO
|
|
|
|
No production code was modified. All observations through the live production `updateCase()` path.
|
|
|
|
## Harness restored: YES
|
|
|
|
Scenario → "Should I relocate my engineering team from London to Manchester?"
|
|
Answers → [cost reduction £2M, staff turnover]
|
|
maxUpdates → 2
|
|
|
|
57J.62 capture hardening preserved: YES (harness unchanged from canonical state)
|
|
57J.70 authoritative guard behaviour preserved: YES (validator at commit 4de8710)
|
|
No-retry behaviour preserved: YES
|
|
|
|
## Ollama calls beyond harness count: 0
|
|
|
|
## Dev server disturbed: NO
|
|
|
|
## Dependencies preserved
|
|
|
|
- 57J.62 capture hardening (harness test suite + accepted-update console block)
|
|
- 57J.70 authoritative guard (validator in lib/graph/utils.js)
|
|
- Exact call accounting in harness |