177 lines
8.2 KiB
Markdown
177 lines
8.2 KiB
Markdown
# Experiment 57J.84 — Direct Meaning/Action Field Observation on Anchored £2m Update
|
|
|
|
**Branch:** `feature/semantic-action-contract-v0.23`
|
|
**Starting HEAD:** `6a04d62` (docs: record accepted answer-meaning capture)
|
|
|
|
## Objective
|
|
|
|
For the anchored savings-realism case with new £2m/year information, what does the model directly populate for `userSupportedMeaning` and `structuralActionRequired`, and is the accepted proposal contract-consistent?
|
|
|
|
Reruns the 57J.82 case only to replace inference with direct observation.
|
|
|
|
## Fixed Starting Graph
|
|
|
|
Pre-anchored fixture (`tests/fixtures/pre-anchored-update-savings-realism.json`):
|
|
|
|
```
|
|
id: n_savings_realism
|
|
label: Are the projected office savings from relocation realistic?
|
|
status: unknown
|
|
kind: unknown
|
|
value: null
|
|
```
|
|
|
|
Exactly one equivalent unresolved savings-realism uncertainty before Update: **YES**
|
|
|
|
## Fixed Answer
|
|
|
|
"The projected office saving is about £2 million per year based on eliminating the current lease cost, but I am still unsure whether that estimate is realistic."
|
|
|
|
## Configured Model
|
|
|
|
- Ollama base URL: http://192.168.1.111:11434
|
|
- Model: qwen-claude:latest
|
|
|
|
## Run
|
|
|
|
```bash
|
|
FIXTURE_MODE=updateOnly \
|
|
ANSWER_2="The projected office saving is about £2 million per year based on eliminating the current lease cost, but I am still unsure whether that estimate is realistic." \
|
|
CONFIDENCE_ENGINE_BASE_URL=http://127.0.0.1:3000 \
|
|
node scripts/reproduce-multi-turn-investigation.mjs
|
|
```
|
|
|
|
## CALL ACCOUNTING
|
|
|
|
- startCalls: 0
|
|
- updateCalls: 1
|
|
- totalCalls: 1
|
|
- Retries: 0
|
|
- Supplementary scripts: NO
|
|
|
|
## PRE-ANCHORED FIXTURE
|
|
|
|
- savings-realism node id: `n_savings_realism`
|
|
- label: "Are the projected office savings from relocation realistic?"
|
|
- status: unknown
|
|
- Exactly one equivalent unresolved uncertainty before Update: YES
|
|
- previousQuestion sent: "Are the projected office savings from relocation realistic?" (derived from fixture.unresolved_question)
|
|
|
|
## UPDATE
|
|
|
|
**HTTP status:** 200 (update succeeded — no rejection; mutation applied)
|
|
**Stage:** UNAVAILABLE (harness updateOnly path does not explicitly print stage for accepted updates, but successful response with mutations confirms `update_applied`)
|
|
**Validation errors:** none
|
|
|
|
### Direct field capture
|
|
|
|
- **userSupportedMeaning:** null (answerMeaning present at top level of response but all fields — userSupportedMeaning, possibleInference, supportCategory, resolutionGuidance — resolved to null)
|
|
- **possibleInference:** null
|
|
- **supportCategory:** null
|
|
- **resolutionGuidance:** null
|
|
- **structuralActionRequired:** null
|
|
|
|
### Proposal mutations
|
|
|
|
```
|
|
updatedNodes: []
|
|
resolvedUnknownNodeIds: []
|
|
addedNodes: [{"id":"n_lease_savings_claim","label":"Projected savings from lease elimination claim","description":"Claim that eliminating the current London lease yields approximately £2 million in annual office savings.","kind":"reported_claim","status":"provisional","confidence":"medium","value":2000000,"unit":"GBP/year","dependsOn":[],"affects":["n_savings_realism"],"parentId":null,"childIds":["n_savings_realism"]}]
|
|
addedEdges: [{"id":"e-claim-to-realism","fromNodeId":"n_lease_savings_claim","toNodeId":"n_savings_realism","relationship":"supports","confidence":"medium","description":"The specific savings claim directly feeds into the uncertainty regarding its realism."}]
|
|
selectedQuestion: "What evidence would clarify are the projected office savings from relocation realistic?"
|
|
selectedQuestion.nodeId: "n_savings_realism"
|
|
```
|
|
|
|
### Resulting persistent graph (3 nodes, 2 edges)
|
|
|
|
```
|
|
node: id=n_relocation_state, kind=state, label=Engineering team relocation consideration, status=provisional
|
|
node: id=n_savings_realism, kind=unknown, label=Are the projected office savings from relocation realistic?, status=unknown
|
|
node: id=n_lease_savings_claim, kind=reported_claim, label=Projected savings from lease elimination claim, status=provisional
|
|
edge: from=n_savings_realism, to=n_relocation_state, relationship=depends_on
|
|
edge: from=n_lease_savings_claim, to=n_savings_realism, relationship=supports
|
|
```
|
|
|
|
## Meaning fidelity
|
|
|
|
**UNAVAILABLE** — `userSupportedMeaning` was not directly populated. The model implicitly captured answer meaning through structural graph changes (new reported_claim node) rather than explicit semantic field population.
|
|
|
|
## Meaningful mutation: PRESENT
|
|
|
|
The model produced a non-trivial structural change: new reported_claim node with structured value (£2,000,000 GBP/year) and supporting edge to the existing savings-realism unknown node. This satisfies hasMeaningfulChange semantics.
|
|
|
|
## Direct contract state: D — MUTATION WITHOUT USER-SUPPORTED MEANING
|
|
|
|
```
|
|
userSupportedMeaning = null/UNAVAILABLE
|
|
structuralActionRequired = null
|
|
meaningful mutation = PRESENT
|
|
```
|
|
|
|
## £2m/year information: REPRESENTED
|
|
|
|
Captured as value=2,000,000 with unit=GBP/year on the new reported_claim node. Full precision preserved ("£2 million" → 2,000,000).
|
|
|
|
## Lease-cost basis: REPRESENTED
|
|
|
|
Description explicitly states "Claim that eliminating the current London lease yields approximately £2 million in annual office savings." — both £2m value and lease-elimination basis are structured on the node.
|
|
|
|
## Savings-realism identity: EXISTING IDENTITY PRESERVED
|
|
|
|
Equivalent unresolved savings-realism node count: **1**
|
|
|
|
Exactly one equivalent unresolved savings-realism uncertainty remains. The original `n_savings_realism` persisted unchanged (status=unknown, value=null). No duplicate created.
|
|
|
|
## Classification: D — MUTATION WITHOUT USER-SUPPORTED MEANING
|
|
|
|
The model produced a meaningful structural mutation (new evidence node with £2m/year + lease-basis data) but did not populate any explicit semantic extraction fields (`userSupportedMeaning`, `possibleInference`, `supportCategory` all null). The structural action boolean (`structuralActionRequired`) was also not populated.
|
|
|
|
## Why:
|
|
|
|
The model implicitly represented the user's answer through graph structure rather than explicit meaning fields. It created a new reported_claim node capturing the £2m/year savings figure and lease-elimination basis, then linked it as supporting evidence to the existing savings-realism uncertainty. This is a structurally faithful representation of the answer — but without `userSupportedMeaning` or other semantic field population, there is no direct observable meaning extraction to evaluate for fidelity.
|
|
|
|
## Did model directly emit userSupportedMeaning: NO
|
|
|
|
## Did model directly emit structuralActionRequired: NO (null)
|
|
|
|
## Was meaningful new information structurally preserved: YES
|
|
|
|
£2m/year and lease-cost basis both fully represented on the new reported_claim node.
|
|
|
|
## Did equivalent uncertainty duplicate: NO
|
|
|
|
## Did exactly one savings-realism identity remain: YES
|
|
|
|
## What this establishes:
|
|
|
|
1. The model can implicitly capture answer meaning through structural graph mutation even when explicit semantic fields are not populated
|
|
2. A new evidence node with structured numeric value (£2M GBP/year) and descriptive lease-basis was created as supporting evidence for the existing savings-realism uncertainty
|
|
3. Identity preservation holds — the original `n_savings_realism` node remains untouched
|
|
4. The model can produce contract-consistent true+mutation (through implicit representation) even without explicit userSupportedMeaning field population
|
|
5. The structuralActionRequired null gap persists: the model creates meaningful mutation but does not populate the boolean declaration
|
|
|
|
## What this does NOT prove:
|
|
|
|
- Whether the model can explicitly populate `userSupportedMeaning` alongside structural mutation in a single response
|
|
- Whether the implicit-meaning-through-mutation pattern is stable across repeated runs
|
|
- Whether this behavior generalizes to other domains or answer types
|
|
- Whether the £2M/year structured value survives full end-to-end graph queries (no downstream query tested)
|
|
|
|
## Production code changed: NO
|
|
|
|
## Prompt changed during experiment: NO
|
|
|
|
## Harness/tooling changed: NO
|
|
|
|
## Canonical committed update-only mode used: YES
|
|
|
|
## 57J.83 direct accepted meaning capture exercised: YES
|
|
|
|
The harness correctly captured `answerMeaning` from the production response path — fields were present at top level but all null, confirming the model did not populate them.
|
|
|
|
## Ollama calls beyond harness count: 0
|
|
|
|
## Dev server disturbed: NO
|
|
|
|
## Documentation updated: YES (`docs/experiment-57j84.md` + `docs/current-handoff.md`)
|