From 174e581c23646bd3f9dc607407b12dac373526c4 Mon Sep 17 00:00:00 2001 From: robbond Date: Tue, 11 Aug 2026 14:35:50 +0100 Subject: [PATCH] experiment: convergence-test uncertainty action selection --- docs/current-handoff.md | 30 +++++++ docs/experiment-57j47.md | 187 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 217 insertions(+) create mode 100644 docs/experiment-57j47.md diff --git a/docs/current-handoff.md b/docs/current-handoff.md index 37c2999..1f6bb40 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -831,3 +831,33 @@ One explicit instruction-order rule in Additional Guidance: check existing unres **What is intentionally left unresolved:** Semantic threshold for "represents the same uncertainty" (relies on model capability + validator safety net). Live production validation (requires next experiment pass). Multi-turn tuning. Edge-connection strategy for Case B new unknowns. Configured Ollama: none used. Production code changed: prompt + tests only. Dev server disturbed: NO. + +### Experiment 57J.47 — Convergence Test: Existing-First Uncertainty Fallback Live + +**Objective:** Does v0.19 produce a faithful structural graph action for one explicit unresolved uncertainty instead of another no-op? + +**Fixed scenario:** "We are considering relocating the engineering team to reduce operating costs." +**Fixed answer:** "I am unsure whether the projected office savings from the relocation are realistic." + +**Live-call count:** start: 1, update: 1, total: 2 + +**START:** HTTP 200 | stage: unknown | nodes: 7 | edges: 4 | question: "What would clarify current detailed breakdown of engineering operating costs and geographic distribution in this situation?" + +**UPDATE 1:** HTTP 422 | stage: proposal_compatibility | error: "answerMeaning.userSupportedMeaning overstates a raw answer that remains uncertain." + +rejectedProposalSnapshot: +- userSupportedMeaning: "The user is currently uncertain whether the projected office savings from the relocation are realistic." (STRENGTHENED — third-person assertion of mental state + temporal specificity "currently" beyond first-person "I am unsure") +- possibleInference: null +- addedNodes: [{id: "ns82kx9", kind: unknown, label: "Projected office savings from the relocation lack validation"}] (genuinely new — ADD NEW UNKNOWN structural action) +- addedEdges: [{fromNodeId: "ns82kx9", toNodeId: "nv759rs", relationship: depends_on}] +- updatedNodes: [] | resolvedUnknownNodeIds: [] + +**Meaning classification:** STRENGTHENED (third-person assertion of user mental state + temporal specificity overstates first-person uncertainty) + +**Structural action:** ADD NEW UNKNOWN — a genuinely new unknown directly representing savings realism was created. This is the FIRST time in experiments 57J.36-47 that the model produced a meaningful structural mutation for this type of uncertainty answer. + +**Classification: D — STRENGTHENING BLOCKS TEST.** Meaning is strengthened; the test cannot establish v0.19's effectiveness for the semantic-to-mutation boundary. However, structural progress was achieved (one new unknown + one edge) — the faithful no-op pattern has been broken. The remaining blocker is meaning extraction (first-person uncertainty → third-person assertion), not the existing-first rule. + +**Key finding:** v0.19's existing-first fallback now produces meaningful graph mutation for explicit unresolved uncertainty. New blocking boundary: first-person uncertainty statements are being converted to third-person assertions, caught by proposal_compatibility as semantic strengthening. Convergence test result is non-A; prompt tuning sequence terminates here per convergence rule. + +Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. 2 live calls total. No production code changed. No prompt change during experiment. Harness restored to original scenario/answers. diff --git a/docs/experiment-57j47.md b/docs/experiment-57j47.md new file mode 100644 index 0000000..b561218 --- /dev/null +++ b/docs/experiment-57j47.md @@ -0,0 +1,187 @@ +# Experiment 57J.47 — Convergence Test: Existing-First Uncertainty Fallback Live + +**Branch:** `feature/semantic-to-mutation-contract-v0.19` +**Starting HEAD:** `94ca1b9` docs: experiment 57J.46 record and handoff update + +## Objective + +Answer exactly: + +> For one explicit unresolved uncertainty, does v0.19 now produce a faithful structural graph action instead of another no-op? + +This is the convergence test for the current prompt-tuning sequence. + +If the same faithful no-op still occurs, do not diagnose or propose v0.20. Report it and stop. + +## Hypothesis + +v0.19 gives the model a two-step structural action rule: + +``` +if equivalent unresolved unknown exists: + reuse/refine it +otherwise: + add a new unknown representing the uncertainty +``` + +Therefore faithful meaning should no longer end with zero meaningful graph mutation. + +## Pre-written expectation + +> A faithful interpretation of the explicit savings-realism uncertainty should now trigger one of two structural outcomes: reuse/refine an equivalent unresolved unknown if present, otherwise create a new unknown representing savings realism. A faithful proposal with no meaningful mutation means the current prompt-tuning approach has still not solved the boundary. + +## Configuration + +**Configured model:** qwen-claude:latest at http://192.168.1.111:11434 +**Dev server:** REUSED EXISTING (HTTP 200, Next.js running) +**Fixed scenario:** "We are considering relocating the engineering team to reduce operating costs." +**Fixed answer:** "I am unsure whether the projected office savings from the relocation are realistic." +**Harness:** `scripts/reproduce-multi-turn-investigation.mjs` (maxUpdates=1, answers=[fixed answer]) + +## CALL ACCOUNTING + +startCalls: 1 +updateCalls: 1 +totalCalls: 2 + +Supplementary scripts used: NO +Retries: 0 + +## START + +HTTP status: 200 | stage: unknown + +Nodes: 7 +Edges: 4 +Selected question: "What would clarify current detailed breakdown of engineering operating costs and geographic distribution in this situation?" + +Start graph (cold-start from scenario): +- n-baseline-costs (unknown/observation) — baseline cost state +- nv759rs (state/knowledge) — relocation consideration state anchor +- n-engineering-operating-costs (unknown/observation) — engineering cost detail gap +- 4 additional nodes (scenario reconstruction scaffolding) + +No analysis of cold-start variance. + +## UPDATE 1 + +HTTP status: 422 +Stage: proposal_compatibility +First error: "answerMeaning.userSupportedMeaning overstates a raw answer that remains uncertain." + +Nodes: 7 (unchanged — mutation not applied) +Edges: 4 (unchanged — mutation not applied) +Selected question: null + +### rejectedProposalSnapshot + +```json +{ + "answerMeaning": { + "userSupportedMeaning": "The user is currently uncertain whether the projected office savings from the relocation are realistic.", + "possibleInference": null + }, + "updatedNodes": [], + "resolvedUnknownNodeIds": [], + "addedNodes": [ + { + "id": "ns82kx9", + "kind": "unknown", + "label": "Projected office savings from the relocation lack validation", + "description": "The realism of projected office savings is unverified, so that the potential cost reduction cannot be validated for the relocation strategy.", + "parentId": null, + "dependsOn": [], + "affects": [], + "childIds": [] + } + ], + "addedEdges": [ + { + "fromNodeId": "ns82kx9", + "toNodeId": "nv759rs", + "relationship": "depends_on" + } + ] +} +``` + +## ANSWER MEANING + +userSupportedMeaning: "The user is currently uncertain whether the projected office savings from the relocation are realistic." +possibleInference: null + +### Meaning classification: STRENGTHENED + +Raw answer: "I am unsure whether the projected office savings from the relocation are realistic." (first-person uncertainty statement) +Extracted meaning: "The user is currently uncertain whether..." (third-person assertion about user's mental state + temporal specificity "currently") + +The model converted first-person uncertainty ("I am unsure") into third-person assertion ("The user is currently uncertain"). This adds two elements beyond the raw answer: +1. **Perspective shift:** Asserting what "the user" feels/thinks, rather than preserving the first-person expression. +2. **Temporal specificity:** Adding "currently" — an explicit temporal qualifier not present in the raw answer. + +This goes slightly beyond preserving "only uncertainty over whether projected office savings are realistic." The extracted meaning introduces a stronger claim (assertion about user mental state) that was not established by the raw answer. + +## STRUCTURAL PROPOSAL + +updatedNodes: [] (none — empty array) +resolvedUnknownNodeIds: [] (none — empty array) +addedNodes: [{id: "ns82kx9", kind: "unknown", label: "Projected office savings from the relocation lack validation", description: "The realism of projected office savings is unverified, so that the potential cost reduction cannot be validated for the relocation strategy."}] +addedEdges: [{fromNodeId: "ns82kx9", toNodeId: "nv759rs", relationship: "depends_on"}] + +### Meaningful updated/refined existing uncertainty: NO + +updatedNodes is empty. No existing unknown was meaningfully modified. + +### Meaningful new uncertainty added: YES + +A genuinely new unknown node (`ns82kx9`) was created, directly representing savings realism ("Projected office savings from the relocation lack validation"). The label and description are grounded in the answer's explicit concern. This represents exactly the user-supported uncertainty about whether projected savings are realistic. + +### Structural action: ADD NEW UNKNOWN + +The proposal added a new unknown node (with one depends_on edge to the state anchor) representing savings realism. The existing-first rule found no equivalent existing unresolved unknown for savings realism, so the fallback-to-add path was correctly exercised. + +## Classification: D — STRENGTHENING BLOCKS TEST + +**Meaning is STRENGTHENED.** The structural action (ADD NEW UNKNOWN) represents exactly the savings-realism uncertainty and is meaningful. However, the meaning extraction is not purely faithful (third-person assertion + temporal specificity overstates the raw first-person uncertainty). Therefore this test does not establish v0.19's effectiveness for the semantic-to-mutation boundary. + +**Why:** The model produced a genuine new unknown node representing savings realism — this IS structural progress that was NOT present in prior experiments (57J.36-45 all showed faithful no-ops or empty proposals). However, the userSupportedMeaning contains third-person assertion ("The user is currently uncertain") that goes beyond the raw answer's first-person uncertainty statement. The `proposal_compatibility` validator caught this as semantic strengthening, rejecting the proposal before structural evaluation. + +**Did v0.19 solve the faithful semantic-to-mutation failure in this run:** NO + +The test cannot determine whether v0.19 solves the boundary because meaning was strengthened, which blocks the test. However, the evidence is directionally encouraging: **the model DID produce a meaningful new unknown for savings realism** — something none of the prior experiments (57J.36 through 57J.46) achieved in a single call. The faithful no-op pattern has been broken; the remaining blocker is the meaning extraction boundary, not the structural action selection. + +## What this establishes: + +1. **The existing-first uncertainty fallback rule works at the structural level.** When no equivalent exists, the model adds a genuinely new unknown directly representing savings realism — exactly what the v0.19 prompt was designed to produce. +2. **The faithful no-op is no longer the default output.** This run produced one added node and one added edge. Prior experiments (57J.36-45) consistently returned zero structural mutations for the same type of uncertainty answer. +3. **A new blocking issue emerges at the meaning extraction boundary:** first-person uncertainty statements ("I am unsure") are being converted to third-person assertions ("The user is currently uncertain"), which the semantic fidelity guard correctly flags as strengthening. + +## What it does NOT prove: + +- That v0.19 reliably produces faithful meaning from first-person uncertainty across repeated runs. +- That the new unknown node's label/description would survive if meaning were faithful. +- That later turns in the investigation remain productive after this type of rejection. +- That the "currently" temporal specificity issue generalizes to other answer types. + +## CONVERGENCE DECISION + +Further prompt tuning justified by this run: NO + +If result is non-A: +Return to architecture discussion before any v0.20 change. + +This is a convergence test for the current prompt-wording sequence (v0.17 → v0.18 → v0.19). Result is D (non-A), so the convergence rule applies: do not diagnose a new prompt tweak, propose v0.20, or continue prompt tuning. The next discussion should reconsider the architecture rather than automatically continuing prompt tuning. + +Production code changed during experiment: NO +Prompt changed during experiment: NO +Canonical harness restored: YES +Hardened no-retry behaviour preserved: YES +Dev server disturbed: NO +Ollama calls beyond harness count: 0 + +## Documentation + +- Created: `docs/experiment-57j47.md` +- Handoff updated: appended to `docs/current-handoff.md` + +Git status after documentation: (dirty — doc file uncommitted)