From 5947ccb642b1cc55f17855d84abb8aa64ec366e7 Mon Sep 17 00:00:00 2001 From: robbond Date: Tue, 11 Aug 2026 17:18:59 +0100 Subject: [PATCH] experiment: validate structured semantic fidelity live --- docs/current-handoff.md | 17 +++++++ docs/experiment-57j52.md | 105 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 122 insertions(+) create mode 100644 docs/experiment-57j52.md diff --git a/docs/current-handoff.md b/docs/current-handoff.md index d1ff51d..fabe55f 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -990,3 +990,20 @@ Implemented the bounded Option C from 57J.50 on branch `feature/structured-seman **Deferred intentionally:** `must_resolve` target-specific enforcement remains deferred because the current proposal structure does not safely identify the answered/targeted unknown in every case without inventing new linkage. **Tests:** Required deterministic suites pass, plus one directly related proposal-parse suite updated for the new schema boundary. No live model calls. No Ollama. Full record in `docs/experiment-57j51.md`. + +### Experiment 57J.52 — Structured Semantic Fidelity Live Verification + +**Classification: A — V0.20 STRUCTURED PATH WORKS** + +One fresh live run through the production `startCase()` → `updateCase()` path. Fixed scenario: "We are considering relocating the engineering team to reduce operating costs." Fixed answer: "I am unsure whether the projected office savings from the relocation are realistic." + +**Key evidence:** +- `supportCategory = "uncertain"` (populated, valid enum) — STRUCTURED path active, not LEGACY FALLBACK. +- `resolutionGuidance = "may_resolve"` (populated). +- Meaning: FAITHFUL — model preserved uncertainty without strengthening or degrading. +- New unknown node `nf3g7m2` ("Realism of projected office savings from relocation") added with `depends_on` edge to summary — meaningful graph structure produced. +- The old `unsure` → `uncertain` lexical mismatch does NOT determine the outcome: structured `supportCategory` is authoritative, bypassing lexical derivation entirely. + +**What this establishes:** v0.20's structured semantic fidelity path executes live on the real production path. The model populates structured fields from a "I am unsure..." answer, the validator trusts those populated fields over lexical comparison, and meaningful graph structure is produced. + +**What this does NOT prove:** Stability across repeated runs; behavior for answers outside existing categories; `must_remain_unresolved` enforcement in practice; end-to-end investigation viability past Update 2+. Full record in `docs/experiment-57j52.md`. diff --git a/docs/experiment-57j52.md b/docs/experiment-57j52.md new file mode 100644 index 0000000..6604429 --- /dev/null +++ b/docs/experiment-57j52.md @@ -0,0 +1,105 @@ +# Experiment 57J.52 — Structured Semantic Fidelity Live Verification + +**Branch:** `feature/structured-semantic-fidelity-v0.20` +**Starting HEAD:** `f156bf5e9a3f53f7d0b438e96b9c75f9d4f1ab29` (closest to feature/structured-semantic-fidelity-v0.20) +**Experiment commit:** pending + +## Objective + +Answer exactly: does v0.20 populate and use structured semantic fidelity live? Does it avoid the old `unsure` → `uncertain` lexical false-positive while still producing meaningful graph structure? + +## Fixed Input + +Scenario: "We are considering relocating the engineering team to reduce operating costs." +Answer: "I am unsure whether the projected office savings from the relocation are realistic." + +## Configuration + +Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434 +Dev server: REUSED EXISTING + +## Call Accounting + +startCalls: 1 +updateCalls: 1 +totalCalls: 2 + +Supplementary scripts used: NO +Retries: 0 + +## START + +HTTP: 200 | stage: unknown +Nodes: 8 +Edges: 5 +Selected question: "What was the comparable state before detailed breakdown of current operating costs versus projected costs in the new location(s)?" + +## UPDATE 1 + +HTTP: 200 +Stage: update_applied +First error: none + +Nodes: 9 (+1) +Edges: 6 (+1) +Selected question: "What would clarify realism of projected office savings from relocation in this situation?" + +## ANSWER MEANING + +userSupportedMeaning: "The user is unsure whether the projected office savings from the relocation are realistic." +possibleInference: "Overestimating these savings would undermine the primary goal of lowering operating costs." +supportCategory: "uncertain" +resolutionGuidance: "may_resolve" + +Meaning classification: FAITHFUL + +Structured path: STRUCTURED + +resolutionGuidance populated: YES + +## STRUCTURAL PROPOSAL + +updatedNodes: [] +resolvedUnknownNodeIds: [] +addedNodes: [{ id: "nf3g7m2", label: "Realism of projected office savings from relocation", kind: "unknown", status: "unknown", dependsOn: ["n11dav1"] }] +addedEdges: [{ id: "e-unk-nf3g7m2", fromNodeId: "nf3g7m2", toNodeId: "n11dav1", relationship: "depends_on" }] + +Structural action: ADD NEW UNKNOWN + +## RESULT + +Classification: A — V0.20 STRUCTURED PATH WORKS + +Why: +- `supportCategory = "uncertain"` is populated and valid (STRUCTURED). +- `resolutionGuidance = "may_resolve"` is populated. +- Meaning is FAITHFUL: the model captured the user's uncertainty without strengthening or degrading. +- The old `unsure` → `uncertain` lexical mismatch does NOT occur because structured fields are authoritative — v0.20 bypasses lexical derivation entirely when structured fields are populated. +- A new unknown node "Realism of projected office savings from relocation" was added to the graph with a `depends_on` edge to the summary state node — meaningful structural representation. + +## Critical Evidence + +Did outcome depend on "unsure" vs "uncertain": NO + +The structured `supportCategory = "uncertain"` is authoritative; lexical comparison of "unsure" vs "uncertain" never occurs in this path. + +## What this establishes + +1. v0.20's structured semantic fidelity path executes live and correctly populates `supportCategory` from the user answer expressing uncertainty ("I am unsure..."). +2. The model returns `supportCategory = "uncertain"` (not null), triggering the structured path over legacy lexical fallback. +3. `resolutionGuidance = "may_resolve"` is also populated. +4. A new unknown node is added to the graph with meaningful structural content derived from the answer's uncertainty dimension. +5. The old `unsure`/`uncertain` lexical false-positive is eliminated on the structured path. + +## What this does NOT prove + +1. Whether `supportCategory = "uncertain"` also works when the model instead returns a different category for this or other answers. +2. Stability of structured population across repeated identical runs. +3. Behavior with answers that don't naturally map to existing categories (e.g., pure preference, conditional trade-off). +4. Whether `must_remain_unresolved` is enforced correctly in practice (not tested by this answer — the model returned "may_resolve" not "must_remain_unresolved"). +5. End-to-end investigation viability past Update 2+. + +## Production code changed: NO + +## Harness restored: YES +