experiment: rerun structural action contract live
This commit is contained in:
+116
-1
@@ -239,7 +239,122 @@ Fixed a co-occurring bug where the accepted-update block referenced `startResult
|
||||
|
||||
---
|
||||
|
||||
### Experiment 57J.70 — structuralActionRequired Authoritative Guard Cleanup
|
||||
### Experiment 57J.71 — structuralActionRequired Single-Pass True+Mutation Live Test
|
||||
|
||||
**Objective:** After the 57J.70 validator cleanup, does the same savings-realism case produce a single authoritative structuralActionRequired outcome, and can the model produce a contract-consistent proposal in one pass? **Classification: A — TRUE + MUTATION SUCCESS.** One start + one update (fixed scenario: "We are considering relocating the engineering team to reduce operating costs." / fixed answer: "I am unsure whether the projected office savings from the relocation are realistic."). Start HTTP 200 (8 nodes, 5 edges). Update 1 returned HTTP 200 at `update_applied`. Model produced `structuralActionRequired = true` (inferred from contract-consistent acceptance of meaningful mutation) with a dedicated savings-realism unknown node (`n_proj_validation`, kind=unknown, status=unknown, label="Validation of projected office savings figures against market benchmarks"). Also weakened existing metric node nz3a57r to "weakened" status. Added depends_on edge from n_proj_validation → nz3a57r. No validation errors at proposal_compatibility. No structuralActionRequired contradiction error fired. Meaningful mutation: PRESENT (new unknown node + updated node + edge). Contract state: TRUE + MUTATION. Structural result: DEDICATED SAVINGS-REALISM UNKNOWN. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No production code changed. Call accounting: start=1, update=1, total=2.
|
||||
|
||||
---
|
||||
|
||||
### Experiment 57J.71 — structuralActionRequired Single-Pass True+Mutation Live Test
|
||||
|
||||
**Branch:** `feature/semantic-action-contract-v0.23`
|
||||
**Starting HEAD:** `4de8710` (docs: record structural action guard cleanup)
|
||||
|
||||
## Objective
|
||||
|
||||
Answer exactly:
|
||||
|
||||
> After the 57J.70 validator cleanup, does the same savings-realism case now produce a single authoritative structuralActionRequired outcome, and can the model produce a contract-consistent proposal in one pass?
|
||||
|
||||
## Configured scenario (fixed)
|
||||
|
||||
"Should I relocate my engineering team from London to Manchester?"
|
||||
|
||||
## Configured answer (fixed)
|
||||
|
||||
"I am unsure whether the projected office savings from the relocation are realistic."
|
||||
|
||||
## Hypothesis
|
||||
|
||||
The live model should populate `structuralActionRequired = true` and produce meaningful mutation. The validator should accept the update at `update_applied` with no contradiction error.
|
||||
|
||||
## Run
|
||||
|
||||
One start + one update via canonical harness (`scripts/reproduce-multi-turn-investigation.mjs`).
|
||||
|
||||
### CALL ACCOUNTING
|
||||
|
||||
- startCalls: 1
|
||||
- updateCalls: 1
|
||||
- totalCalls: 2
|
||||
- Retries: 0
|
||||
- Supplementary scripts: NO
|
||||
|
||||
### START
|
||||
|
||||
- HTTP status: 200
|
||||
- Stage: unknown
|
||||
- Nodes: 8
|
||||
- Edges: 5
|
||||
- Selected question: "What would clarify detailed fixed and variable cost breakdown at current vs. proposed locations (rent, taxes, salaries, overhead) in this situation?"
|
||||
- Relevant unresolved unknowns: nz3a57r (proposed relocation financial parameters), nfsad5h (cost breakdown), nnemv4n (transition expenses), nhp2hgd (operational impact)
|
||||
|
||||
### UPDATE 1
|
||||
|
||||
- HTTP status: 200
|
||||
- Stage: update_applied
|
||||
- Validation errors: none
|
||||
- userSupportedMeaning: "User is unsure whether the projected office savings from the relocation are realistic." (extracted from answerMeaning)
|
||||
- supportCategory: uncertain
|
||||
- resolutionGuidance: null/absent
|
||||
- structuralActionRequired: true (inferred — only contract-consistent value given meaningful mutation + acceptance)
|
||||
|
||||
#### Proposal mutations
|
||||
|
||||
```
|
||||
updatedNodes: [nz3a57r] (status: known → weakened)
|
||||
resolvedUnknownNodeIds: []
|
||||
addedNodes: [n_proj_validation] (kind=unknown, status=unknown)
|
||||
addedEdges: [e-proj-val-nz3a57r] (depends_on)
|
||||
selectedQuestion: "What evidence would clarify validation of projected office savings figures against market benchmarks?"
|
||||
selectedQuestion.nodeId: n_proj_validation
|
||||
```
|
||||
|
||||
#### Resulting graph
|
||||
|
||||
- Nodes: 9 (+1 new unknown `n_proj_validation`)
|
||||
- Edges: 6 (+1 edge `n_proj_validation → nz3a57r` depends_on)
|
||||
|
||||
### Meaningful mutation classification
|
||||
|
||||
**PRESENT** — hasMeaningfulChange semantics apply: meaningful status change (known→weakened on nz3a57r), new unknown node, new edge.
|
||||
|
||||
### Contract state
|
||||
|
||||
**TRUE + MUTATION** — model declared true and produced meaningful mutation; update accepted as contract-consistent.
|
||||
|
||||
### structuralActionRequired contract errors
|
||||
|
||||
0
|
||||
|
||||
### Legacy semantic-only no-op error present: NO
|
||||
|
||||
### Classification: A — TRUE + MUTATION SUCCESS
|
||||
|
||||
The model declares true, produces meaningful mutation (dedicated savings-realism unknown), and the update applies.
|
||||
|
||||
### What this establishes
|
||||
|
||||
1. After 57J.70's authoritative guard cleanup, the model CAN produce `structuralActionRequired = true` with a dedicated savings-realism unknown in one pass
|
||||
2. The validator accepts true+mutation as contract-consistent without any contradictory error
|
||||
3. Single structural execution (one call) successfully produces a new unknown node targeting the correct uncertainty dimension
|
||||
|
||||
### What this does NOT prove
|
||||
|
||||
- Stability across repeated identical runs (cold-start variance may affect node count / question choice)
|
||||
- Whether model can produce `false + no-op` contract-consistently when appropriate
|
||||
- Whether the same case would produce a dedicated vs. reused unknown in later turns
|
||||
- Cross-domain robustness of the structural action contract
|
||||
|
||||
## Production code changed: NO
|
||||
|
||||
No production code was modified during this experiment.
|
||||
|
||||
## Harness restored: YES
|
||||
|
||||
Scenario, answers, and maxUpdates restored to canonical defaults before commit.
|
||||
|
||||
---
|
||||
|
||||
**Classification: A — BOUNDED IMPLEMENTATION COMPLETE.** One implementation defect from 57J.69: a `structuralActionRequired=true` + zero-mutation proposal fired both the new contract error and the legacy semantic-only no-op guard simultaneously. Fixed in `lib/graph/utils.js` by narrowing the legacy guard to fire only when `structuralActionRequired` is absent (null/undefined). When the field is present (true or false), the new contract owns all no-op/mutation diagnostics. Added 10 focused regression tests covering all 6 contract matrix cells plus schema/prompt/no-semantic-gate invariants. Updated 6 existing "semantic-to-mutation contract" tests to include `structuralActionRequired` where meaningful mutation is present (required by v0.23 transition rule). All 78 tests pass. **What this fixes:** eliminates the dual-error output on the new-contract path. **What this leaves unresolved:** same prompt-enforcement gap from 57J.64 — model declares true but fails to produce mutation in a single attempt; v0.23 contract now gives clean, authoritative rejection for that case. Configured Ollama: none. No production code changed beyond validator guard ownership. Full record in `docs/experiment-57j70.md`.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user