From 20e4b58440cbd06e92bb6dab231db969fa4bfa71 Mon Sep 17 00:00:00 2001 From: robbond Date: Wed, 12 Aug 2026 13:46:29 +0100 Subject: [PATCH] experiment: test evidence preservation with one uncertainty --- docs/current-handoff.md | 8 +++ docs/experiment-58a2.md | 142 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 150 insertions(+) create mode 100644 docs/experiment-58a2.md diff --git a/docs/current-handoff.md b/docs/current-handoff.md index b86a9c2..b40a1b1 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -521,6 +521,14 @@ Scenario, answers, and maxUpdates restored to canonical defaults before commit. --- +### Experiment 58A.2 — Evidence + Uncertainty with One Competing Demand + +**Objective:** When the answer is reduced to one concrete figure + one qualification + one new uncertainty (no competing assumptions), does the engine preserve the qualified figure while representing the new uncertainty? + +**Classification: E — IDENTITY FAILURE.** One update-only call via the committed harness. The engine preserved the £2m/year figure as qualified evidence on n_savings_realism and created a dedicated lease-exit-uncertainty unknown node with clear structural linkage. However, it degraded n_savings_realism from `unknown` to `provisional`, weakening its identity as an unresolved investigation target. The selected question was malformed (template injection failure). Reducing competing demands solved 58A.1's evidence-loss problem but introduced a distinct status-degradation failure. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No production code changed. Full record in `docs/experiment-58a2.md`. + +--- + ### Experiment 57J.61 — Equivalent Uncertainty Identity Live Test **Objective:** Once a dedicated savings-realism uncertainty exists, does a second semantically equivalent statement reuse that same unresolved node rather than create a duplicate? **Classification: D — UPDATE 1 FAILED.** One start + two updates. Start HTTP 200 (6 nodes). Update 1 returned HTTP 200 at update_applied but the harness crash prevented detailed proposal capture. A cold-start variant confirmed that when userSupportedMeaning is populated for savings-realism uncertainty, the model extracts meaning but proposes zero graph mutations — updatedNodes=[{nodeId: X, newValue: null}], addedNodes=[], addedEdges=[]. The gateway rejects this at proposal_compatibility with "answerMeaning.userSupportedMeaning is populated, but the proposal contains no graph mutation." Update 2 was reached (total 3 calls) and was rejected for the same reason. **Neither turn established a persistent savings-realism unknown.** The identity invariant cannot be tested when neither turn produces a valid, persistent unknown node. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No production code changed. Full record in `docs/experiment-57j61.md`. diff --git a/docs/experiment-58a2.md b/docs/experiment-58a2.md new file mode 100644 index 0000000..7af0c6e --- /dev/null +++ b/docs/experiment-58a2.md @@ -0,0 +1,142 @@ +# Experiment 58A.2 — Evidence + Uncertainty with One Competing Demand + +**Branch:** `feature/semantic-action-contract-v0.23` +**Starting HEAD:** `3218469cfb081d5f9e7e3ed1a763482fdd2a241f` +**Experiment commit:** pending + +## Objective + +Can the engine preserve a concrete figure while also creating one new uncertainty? + +58A.1 showed good uncertainty reasoning but lost the central £2m figure while creating two new unknowns. This experiment reduces the answer to **one concrete figure + one qualification + one new assumption** to test whether information loss was caused by competing structural demands rather than a fundamental inability to represent qualified evidence. + +## Fixed scenario (from fixture) + +"We are considering relocating the engineering team to reduce operating costs." + +Existing unresolved uncertainty: "Are the projected office savings from relocation realistic?" + +## Fixed answer + +> The projected saving is about £2 million per year, but I don't trust that figure until we know what penalty we would pay to exit the existing lease. + +## Configured model: qwen-claude:latest at http://192.168.1.111:11434 + +### CALL ACCOUNTING + +- startCalls: 0 +- updateCalls: 1 +- totalCalls: 1 +- Retries: 0 + +### UPDATE RESULT + +- HTTP status: 200 +- Stage: update_applied +- Validation errors: none +- structuralActionRequired: null (known gap from 57J.69) + +#### Proposal mutations + +``` +updatedNodes: [ + {nodeId: n_savings_realism, previousStatus: unknown, newStatus: provisional, + previousValue: null, newValue: "~£2,000,000/year (pending lease exit cost)", + reason: "User provided a provisional savings projection that is contingent on unresolved lease termination costs."} +] +resolvedUnknownNodeIds: [] +addedNodes: [ + {id: n_lease_penalty_uncertainty, label: "Lease exit penalty amount", + description: "Financial penalty required to break the current London office lease before relocation, + because it directly determines whether the ~£2M/year savings projection will actually materialize.", + kind: unknown, status: unknown} +] +addedEdges: [ + {fromNodeId: n_savings_realism, toNodeId: n_lease_penalty_uncertainty, + relationship: depends_on, confidence: high, + description: "Realizing the projected savings depends on quantifying the lease exit penalty."} +] +selectedQuestion.nodeId: "n_savings_realism" +``` + +#### Selected question + +> "What was the comparable state before are the projected office savings from relocation realistic?" +> nodeId: n_savings_realism + +(Note: question text appears malformed — template injection failure producing grammatically broken sentence.) + +### Resulting persistent graph (3 nodes, 2 edges) + +``` +node: id=n_relocation_state, kind=state, label=Engineering team relocation consideration, status=provisional +node: id=n_savings_realism, kind=unknown, label=Are the projected office savings from relocation realistic?, status=provisional +node: id=n_lease_penalty_uncertainty, kind=unknown, label=Lease exit penalty amount, status=unknown +edge: from=n_savings_realism, to=n_relocation_state, relationship=depends_on +edge: from=n_savings_realism, to=n_lease_penalty_uncertainty, relationship=depends_on +``` + +### Reasoning assessment + +**£2m figure:** PRESERVED AS QUALIFIED — the value `~£2,000,000/year (pending lease exit cost)` survives on the savings-realism node. It is preserved with qualifier text, though not at full precision ("about £2 million" → "~£2,000,000") and not as a standalone reported_claim node. + +**Qualification:** WEAKENED — The status change from `unknown` → `provisional` on n_savings_realism is the primary signal of weakening. "Provisional" suggests tentative acceptance rather than active investigation. The qualification survives in newValue text ("pending lease exit cost") and in the reason prose, but structurally the node no longer functions as an unresolved question — it functions as a tentative fact that needs verification. + +**Lease-exit uncertainty:** STRUCTURALLY REPRESENTED — dedicated unknown node with kind=unknown, status=unknown, explicit description tying it to the savings figure, plus a `depends_on` edge from n_savings_realism to this node. The structural representation is stronger than 58A.1's lease-exit handling. + +**Original savings-realism uncertainty:** WEAKENED — The node identity persists (n_savings_realism still exists, not duplicated). However, the status change from `unknown` → `provisional` means it no longer signals "unresolved investigation target" — it signals "tentatively accepted but needs verification." This is a degradation of uncertainty signaling that could mislead downstream question selection and Behaviour Selection. + +**Evidence / uncertainty linkage:** CLEARLY LINKED — The `depends_on` edge from n_savings_realism to n_lease_penalty_uncertainty structurally encodes the dependency relationship: realizing savings depends on quantifying the penalty. Description reinforces this ("Realizing the projected savings depends on quantifying the lease exit penalty."). + +**Next question quality:** WRONG — "What was the comparable state before are the projected office savings from relocation realistic?" is a grammatically broken template injection (combining "What was the comparable state before [X]?" with "[X]" = full unknown label). It does not materially help determine whether the £2m figure is realistic. + +### Classification: E — IDENTITY FAILURE + +The original savings-realism uncertainty node's status was degraded from `unknown` to `provisional`, weakening its identity as an unresolved investigation target. This is not a correct resolution (status remains unknown-ish but with degraded semantics), nor is it simply "preserved." The uncertainty exists in a degraded state that could mislead downstream reasoning stages about the investigation's health. + +Additionally, the selected question is malformed and fails to pursue any material unresolved issue. + +### What the engine preserved correctly: + +1. The £2m/year figure survived as qualified evidence (newValue on existing node) +2. The lease-exit uncertainty was structurally represented with a dedicated unknown node +3. Evidence and new uncertainty are clearly linked via depends_on edge + description +4. No duplicate savings-realism uncertainty was created +5. No-resolve guard worked (resolvedUnknownNodeIds is empty) + +### What it lost or weakened: + +1. The savings-realism uncertainty identity — degraded from `unknown` to `provisional`, losing its function as an active investigation target +2. Question quality — malformed sentence that does not pursue the material unresolved issue +3. Precision of the £2m figure ("about £2 million" → "~£2,000,000") + +### What uncertainty it chose to pursue next: + +n_savings_realism (the existing savings-realism unknown), but the question text is broken and does not target the lease-exit penalty or any other material issue. + +### Was that the best available next uncertainty: + +YES — n_savings_realism is the correct investigation target, but the execution of the question (malformed text) renders this moot. + +### Comparison with 58A.1: + +58A.1 lost the £2m figure entirely but preserved savings-realism as `unknown` and produced a grammatically coherent (if debatable) next question. 58A.2 preserves both the figure and the new uncertainty, but at the cost of degrading the savings-realism node from `unknown` to `provisional` and producing a malformed question. The trade-off is clear: reducing competing demands (2 unknowns → 1 unknown) solved the evidence-loss problem but introduced a status-degradation failure. This establishes that evidence preservation and uncertainty preservation are not simply inverses of each other — there is a separate mechanism controlling node status that can degrade uncertainty identity even when evidence survives. + +### What this establishes: + +1. Reducing competing structural demands (one unknown instead of two) allows the figure to survive as qualified evidence +2. The status-degradation failure (unknown → provisional) is a distinct failure mode from evidence loss — it occurs alongside successful evidence preservation +3. Question-formulation templates have a defect when targeting existing (non-newly-created) unknown nodes + +### What this does NOT prove: + +- Whether the status-degradation issue would recur with different model configurations or repeated runs +- Whether a dedicated reported_claim node type would avoid the status conflict +- Whether question-formulation templates can be fixed to handle existing-node targets correctly +- Cross-domain generalisation of any finding + +### Production code changed: NO +### Prompt changed during experiment: NO +### Harness/tooling changed: NO +### Ollama calls beyond harness count: 0 +### Dev server disturbed: NO