experiment: validate structured semantic fidelity live

This commit is contained in:
2026-08-11 17:18:59 +01:00
parent f156bf5ef8
commit 5947ccb642
2 changed files with 122 additions and 0 deletions
+17
View File
@@ -990,3 +990,20 @@ Implemented the bounded Option C from 57J.50 on branch `feature/structured-seman
**Deferred intentionally:** `must_resolve` target-specific enforcement remains deferred because the current proposal structure does not safely identify the answered/targeted unknown in every case without inventing new linkage.
**Tests:** Required deterministic suites pass, plus one directly related proposal-parse suite updated for the new schema boundary. No live model calls. No Ollama. Full record in `docs/experiment-57j51.md`.
### Experiment 57J.52 — Structured Semantic Fidelity Live Verification
**Classification: A — V0.20 STRUCTURED PATH WORKS**
One fresh live run through the production `startCase()``updateCase()` path. Fixed scenario: "We are considering relocating the engineering team to reduce operating costs." Fixed answer: "I am unsure whether the projected office savings from the relocation are realistic."
**Key evidence:**
- `supportCategory = "uncertain"` (populated, valid enum) — STRUCTURED path active, not LEGACY FALLBACK.
- `resolutionGuidance = "may_resolve"` (populated).
- Meaning: FAITHFUL — model preserved uncertainty without strengthening or degrading.
- New unknown node `nf3g7m2` ("Realism of projected office savings from relocation") added with `depends_on` edge to summary — meaningful graph structure produced.
- The old `unsure``uncertain` lexical mismatch does NOT determine the outcome: structured `supportCategory` is authoritative, bypassing lexical derivation entirely.
**What this establishes:** v0.20's structured semantic fidelity path executes live on the real production path. The model populates structured fields from a "I am unsure..." answer, the validator trusts those populated fields over lexical comparison, and meaningful graph structure is produced.
**What this does NOT prove:** Stability across repeated runs; behavior for answers outside existing categories; `must_remain_unresolved` enforcement in practice; end-to-end investigation viability past Update 2+. Full record in `docs/experiment-57j52.md`.
+105
View File
@@ -0,0 +1,105 @@
# Experiment 57J.52 — Structured Semantic Fidelity Live Verification
**Branch:** `feature/structured-semantic-fidelity-v0.20`
**Starting HEAD:** `f156bf5e9a3f53f7d0b438e96b9c75f9d4f1ab29` (closest to feature/structured-semantic-fidelity-v0.20)
**Experiment commit:** pending
## Objective
Answer exactly: does v0.20 populate and use structured semantic fidelity live? Does it avoid the old `unsure``uncertain` lexical false-positive while still producing meaningful graph structure?
## Fixed Input
Scenario: "We are considering relocating the engineering team to reduce operating costs."
Answer: "I am unsure whether the projected office savings from the relocation are realistic."
## Configuration
Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434
Dev server: REUSED EXISTING
## Call Accounting
startCalls: 1
updateCalls: 1
totalCalls: 2
Supplementary scripts used: NO
Retries: 0
## START
HTTP: 200 | stage: unknown
Nodes: 8
Edges: 5
Selected question: "What was the comparable state before detailed breakdown of current operating costs versus projected costs in the new location(s)?"
## UPDATE 1
HTTP: 200
Stage: update_applied
First error: none
Nodes: 9 (+1)
Edges: 6 (+1)
Selected question: "What would clarify realism of projected office savings from relocation in this situation?"
## ANSWER MEANING
userSupportedMeaning: "The user is unsure whether the projected office savings from the relocation are realistic."
possibleInference: "Overestimating these savings would undermine the primary goal of lowering operating costs."
supportCategory: "uncertain"
resolutionGuidance: "may_resolve"
Meaning classification: FAITHFUL
Structured path: STRUCTURED
resolutionGuidance populated: YES
## STRUCTURAL PROPOSAL
updatedNodes: []
resolvedUnknownNodeIds: []
addedNodes: [{ id: "nf3g7m2", label: "Realism of projected office savings from relocation", kind: "unknown", status: "unknown", dependsOn: ["n11dav1"] }]
addedEdges: [{ id: "e-unk-nf3g7m2", fromNodeId: "nf3g7m2", toNodeId: "n11dav1", relationship: "depends_on" }]
Structural action: ADD NEW UNKNOWN
## RESULT
Classification: A — V0.20 STRUCTURED PATH WORKS
Why:
- `supportCategory = "uncertain"` is populated and valid (STRUCTURED).
- `resolutionGuidance = "may_resolve"` is populated.
- Meaning is FAITHFUL: the model captured the user's uncertainty without strengthening or degrading.
- The old `unsure``uncertain` lexical mismatch does NOT occur because structured fields are authoritative — v0.20 bypasses lexical derivation entirely when structured fields are populated.
- A new unknown node "Realism of projected office savings from relocation" was added to the graph with a `depends_on` edge to the summary state node — meaningful structural representation.
## Critical Evidence
Did outcome depend on "unsure" vs "uncertain": NO
The structured `supportCategory = "uncertain"` is authoritative; lexical comparison of "unsure" vs "uncertain" never occurs in this path.
## What this establishes
1. v0.20's structured semantic fidelity path executes live and correctly populates `supportCategory` from the user answer expressing uncertainty ("I am unsure...").
2. The model returns `supportCategory = "uncertain"` (not null), triggering the structured path over legacy lexical fallback.
3. `resolutionGuidance = "may_resolve"` is also populated.
4. A new unknown node is added to the graph with meaningful structural content derived from the answer's uncertainty dimension.
5. The old `unsure`/`uncertain` lexical false-positive is eliminated on the structured path.
## What this does NOT prove
1. Whether `supportCategory = "uncertain"` also works when the model instead returns a different category for this or other answers.
2. Stability of structured population across repeated identical runs.
3. Behavior with answers that don't naturally map to existing categories (e.g., pure preference, conditional trade-off).
4. Whether `must_remain_unresolved` is enforced correctly in practice (not tested by this answer — the model returned "may_resolve" not "must_remain_unresolved").
5. End-to-end investigation viability past Update 2+.
## Production code changed: NO
## Harness restored: YES