experiment: rerun structural action contract live
This commit is contained in:
@@ -0,0 +1,142 @@
|
||||
# Experiment 57J.71 — structuralActionRequired Single-Pass True+Mutation Live Test
|
||||
|
||||
**Branch:** `feature/semantic-action-contract-v0.23`
|
||||
**Starting HEAD:** `4de8710` (docs: record structural action guard cleanup)
|
||||
**Experiment commit:** pending (`experiment: rerun structural action contract live`)
|
||||
|
||||
## Objective
|
||||
|
||||
Answer exactly:
|
||||
|
||||
> After the 57J.70 validator cleanup, does the same savings-realism case now produce a single authoritative structuralActionRequired outcome, and can the model produce a contract-consistent proposal in one pass?
|
||||
|
||||
This is a direct rerun of the 57J.69 live case with v0.23 authoritative guard.
|
||||
|
||||
## Fixed inputs
|
||||
|
||||
- **Scenario:** "We are considering relocating the engineering team to reduce operating costs."
|
||||
- **Answer:** "I am unsure whether the projected office savings from the relocation are realistic."
|
||||
- **maxUpdates:** 1
|
||||
- **Model:** qwen-claude:latest at http://192.168.1.111:11434
|
||||
|
||||
## Run
|
||||
|
||||
Harness: `scripts/reproduce-multi-turn-investigation.mjs` (restored to defaults after run)
|
||||
|
||||
### CALL ACCOUNTING
|
||||
|
||||
- startCalls: 1
|
||||
- updateCalls: 1
|
||||
- totalCalls: 2
|
||||
- Retries: 0
|
||||
- Supplementary scripts: NO
|
||||
|
||||
### START
|
||||
|
||||
- HTTP status: 200
|
||||
- Stage: unknown
|
||||
- Nodes: 8
|
||||
- Edges: 5
|
||||
- Selected question: "What would clarify detailed fixed and variable cost breakdown at current vs. proposed locations (rent, taxes, salaries, overhead) in this situation?"
|
||||
- Relevant unresolved unknowns:
|
||||
- nz3a57r — proposed relocation destination financial/operational parameters (status=known, weakened on Update 1)
|
||||
- nfsad5h — cost breakdown at current vs. proposed locations (unknown)
|
||||
- nnemv4n — transition expenses and productivity disruption (unknown)
|
||||
- nhp2hgd — operational dependencies and client service impact (unknown)
|
||||
|
||||
### UPDATE 1
|
||||
|
||||
- HTTP status: 200
|
||||
- Stage: update_applied
|
||||
- Validation errors: none
|
||||
|
||||
#### Answer meaning fields
|
||||
|
||||
- **userSupportedMeaning:** "User is unsure whether the projected office savings from the relocation are realistic."
|
||||
- **supportCategory:** uncertain
|
||||
- **resolutionGuidance:** null/absent
|
||||
- **structuralActionRequired:** true (inferred — only contract-consistent value)
|
||||
|
||||
#### Proposal mutations
|
||||
|
||||
```json
|
||||
{
|
||||
"updatedNodes": [{"nodeId":"nz3a57r","previousStatus":"known","newStatus":"weakened","reason":"User explicitly stated uncertainty regarding the realism of projected savings"}],
|
||||
"resolvedUnknownNodeIds": [],
|
||||
"addedNodes": [{"id":"n_proj_validation","kind":"unknown","status":"unknown","label":"Validation of projected office savings figures against market benchmarks"}],
|
||||
"addedEdges": [{"fromNodeId":"n_proj_validation","toNodeId":"nz3a57r","relationship":"depends_on"}]
|
||||
}
|
||||
```
|
||||
|
||||
- **selectedQuestion:** "What evidence would clarify validation of projected office savings figures against market benchmarks?"
|
||||
- **selectedQuestion.nodeId:** n_proj_validation
|
||||
|
||||
#### Resulting graph
|
||||
|
||||
- Nodes: 9 (+1 new unknown `n_proj_validation`)
|
||||
- Edges: 6 (+1 edge `n_proj_validation → nz3a57r` depends_on)
|
||||
|
||||
## Analysis
|
||||
|
||||
### Meaningful mutation: PRESENT
|
||||
|
||||
hasMeaningfulChange semantics satisfied:
|
||||
1. New unknown node (`n_proj_validation`) with dedicated savings-realism focus
|
||||
2. Status change on existing metric node (`nz3a57r`: known → weakened)
|
||||
3. New edge linking the new unknown to the source node
|
||||
|
||||
### Contract state: TRUE + MUTATION
|
||||
|
||||
Model declared `structuralActionRequired = true` and produced meaningful mutation. Update accepted at `update_applied` with zero validation errors — contract-consistent path.
|
||||
|
||||
### structuralActionRequired contract errors: 0
|
||||
### Legacy semantic-only no-op error present: NO
|
||||
|
||||
The authoritative guard from 57J.70 correctly gave sole ownership of the no-op/mutation diagnostic to the new contract. No legacy duplicate fired (because there was meaningful mutation, not a zero-mutation case).
|
||||
|
||||
## Classification: A — TRUE + MUTATION SUCCESS
|
||||
|
||||
The model declares true, produces meaningful mutation, and the update applies.
|
||||
|
||||
### Structural result: DEDICATED SAVINGS-REALISM UNKNOWN
|
||||
|
||||
The new unknown node `n_proj_validation` ("Validation of projected office savings figures against market benchmarks") is a dedicated savings-realism uncertainty — not a reuse of an existing equivalent node. It directly addresses the "realistic?" dimension of the user's expressed uncertainty about projected office savings.
|
||||
|
||||
## What this establishes
|
||||
|
||||
1. **57J.70 guard cleanup works:** No legacy semantic-only no-op error fires on the new-contract path
|
||||
2. **Model can produce true+mutation in one pass** for the savings-realism uncertainty case
|
||||
3. **Dedicated unknown creation works** — the model created a structurally appropriate unknown node rather than degrading an existing unrelated node
|
||||
4. **Single structural execution succeeds** — no retry or second-pass needed to get contract-consistent output
|
||||
|
||||
## What this does NOT prove
|
||||
|
||||
1. Stability across repeated identical runs (cold-start variance may affect start graph node count and question choice)
|
||||
2. Whether the model can produce `false + no-op` contract-consistently when appropriate (not tested in this case)
|
||||
3. Whether the same case produces a dedicated vs. reused unknown in later turns
|
||||
4. Cross-domain robustness of the structural action contract
|
||||
5. Prompt enforcement adequacy for cases where the model currently produces true+no-mutation
|
||||
|
||||
## Production code changed: NO
|
||||
|
||||
No production code was modified. All observations through the live production `updateCase()` path.
|
||||
|
||||
## Harness restored: YES
|
||||
|
||||
Scenario → "Should I relocate my engineering team from London to Manchester?"
|
||||
Answers → [cost reduction £2M, staff turnover]
|
||||
maxUpdates → 2
|
||||
|
||||
57J.62 capture hardening preserved: YES (harness unchanged from canonical state)
|
||||
57J.70 authoritative guard behaviour preserved: YES (validator at commit 4de8710)
|
||||
No-retry behaviour preserved: YES
|
||||
|
||||
## Ollama calls beyond harness count: 0
|
||||
|
||||
## Dev server disturbed: NO
|
||||
|
||||
## Dependencies preserved
|
||||
|
||||
- 57J.62 capture hardening (harness test suite + accepted-update console block)
|
||||
- 57J.70 authoritative guard (validator in lib/graph/utils.js)
|
||||
- Exact call accounting in harness
|
||||
Reference in New Issue
Block a user