From fbbd271596d99f385ccef1535963cd2e36909e3c Mon Sep 17 00:00:00 2001 From: robbond Date: Tue, 11 Aug 2026 06:38:30 +0100 Subject: [PATCH] experiment: validate user-supported unknown admission live --- docs/current-handoff.md | 27 ++++++++ docs/experiment-57j25.md | 130 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 157 insertions(+) create mode 100644 docs/experiment-57j25.md diff --git a/docs/current-handoff.md b/docs/current-handoff.md index 0fd9d29..fe1c31b 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -344,3 +344,30 @@ Candidate commit `60048a5` (reasoning: require corroboration for conjunction com ### Experiment 57J.11 — Live Unknown Dimensionality Representation Tested whether a single answer containing two independent evidence dimensions is represented as two separate unknown nodes or collapsed into one compound unknown. **Classification: A — SEPARATE.** Fixed scenario: "We are considering relocating the engineering team to reduce operating costs." Fixed answer: "Before deciding, I need evidence that the projected office savings are realistic and evidence that the move will not materially increase loss of key engineers." One live start + one live update. Update was rejected at `proposal_compatibility` (structural linkage error), but the rejection errors directly revealed two independently named proposed unknown nodes: `n-savings-realism` (Target A) and `n-retention-impact` (Target B). The model did not collapse them into a single compound node. Semantic separability existed in the model proposal before deterministic answerability/decomposition. **What remains unproven:** separation for implicit conjunctions, across models/repeated runs, and whether both nodes survive structural gating (`proposal_compatibility`) or question-selection after graph mutation. Full record in `docs/experiment-57j11.md`. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No production code changed. + +### Experiment 57J.25 — Live User-Supported Unknown Admission Validation + +**Objective:** Validate that the v0.15 candidate admits two user-supported unknowns from the 57J.11 case through the live production `updateCase()` path without requiring fake provenance edges. + +**Pre-written expectation:** The answer explicitly introduces two independent uncertainties: savings realism and retention impact. If v0.15 works on the live production path, those user-supported unknowns should no longer be rejected solely because they lack an answer-derived provenance edge. No fake edge should be required or manufactured. A later failure at a different validation/reasoning boundary is acceptable evidence and must be recorded as the first new failure. + +**Fixed scenario:** "We are considering relocating the engineering team to reduce operating costs." +**Fixed answer:** "Before deciding, I need evidence that the projected office savings are realistic and evidence that the move will not materially increase loss of key engineers." + +**Live-call count:** start: 1, update: 1, total: 2 + +**START:** HTTP 200 | stage: unknown | nodes: 6 | edges: 3 | question: "What would clarify detailed breakdown of current engineering operating costs in this situation?" + +**UPDATE 1:** HTTP 200 | stage: update_applied | nodes: 8 (+2) | edges: 5 (+2) | no errors/validation failures. Both proposed unknowns admitted: `n-oss-realistic` (savings realism) and `n-kr-loss` (retention impact), each with meaningful descriptions grounded in the answer semantics. Selected next question targets `n-oss-realistic` via evidence_gathering/diagnosis strategy. + +**57J.11 provenance-link rejection:** ABSENT — no proposal_compatibility rejection occurred. Both target dimensions survived. + +**Savings target:** PRESERVED +**Retention target:** PRESERVED +**Fake provenance edge:** NO + +**Classification: A — PASS.** The v0.15 update path admits both user-supported evidence dimensions through the production path without rejection at the old 57J.11 provenance-link gate. No later failure occurred within this single update. + +**What remains unproven:** Repeated-run stability; cross-domain generalisation; downstream investigation viability past Update 2+; implicit conjunction admission; whether `too_broad` conversation health eventually blocks later turns. + +Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No production code changed. Branch: `feature/user-supported-unknown-admission-v0.15`. diff --git a/docs/experiment-57j25.md b/docs/experiment-57j25.md new file mode 100644 index 0000000..9c4b6e6 --- /dev/null +++ b/docs/experiment-57j25.md @@ -0,0 +1,130 @@ +# Experiment 57J.25 — Live Unknown Admission v0.15 Validation + +**Objective:** Validate that the v0.15 candidate admits two user-supported unknowns from the 57J.11 case through the live production `updateCase()` path without requiring fake provenance edges. + +## Pre-written expectation + +> The answer explicitly introduces two independent uncertainties: savings realism and retention impact. If v0.15 works on the live production path, those user-supported unknowns should no longer be rejected solely because they lack an answer-derived provenance edge. No fake edge should be required or manufactured. A later failure at a different validation/reasoning boundary is acceptable evidence and must be recorded as the first new failure. + +## Configured apparatus + +- **Ollama base URL:** `http://192.168.1.111:11434` +- **Model:** `qwen-claude:latest` +- **Harness:** `scripts/reproduce-multi-turn-investigation.mjs` (canonical) +- **Branch:** `feature/user-supported-unknown-admission-v0.15` +- **Production API path:** `/api/cases/start` → `/api/cases/update` + +## Fixed scenario and answer + +**Scenario:** "We are considering relocating the engineering team to reduce operating costs." + +**Answer:** "Before deciding, I need evidence that the projected office savings are realistic and evidence that the move will not materially increase loss of key engineers." + +## Live-call count + +``` +start: 1 +update: 1 +total: 2 +``` + +## START + +- **HTTP status:** 200 +- **Stage:** unknown (initial state) +- **Node count:** 6 +- **Edge count:** 3 +- **Selected question:** "What would clarify detailed breakdown of current engineering operating costs in this situation?" + +## UPDATE 1 + +- **HTTP status:** 200 +- **Stage:** `update_applied` +- **Error/validation summary:** null (none) +- **Node count:** 8 (+2 new) +- **Edge count:** 5 (+2 new) +- **Selected question:** "What would clarify realism of projected office savings in this situation?" + +## Detailed result capture + +### Two proposed unknown nodes + +| id | label | description | kind | status | +|---|---|---|---|---| +| `n-oss-realistic` | Project realism and validation of anticipated office relocation savings. | Whether the projected financial savings from the relocation are realistic and achievable, so that the cost reduction objective can be trusted as a driver for the decision. | unknown | unknown | +| `n-kr-loss` | Projected impact of the move on key engineer retention rates. | The extent to which the relocation could cause a material increase in the turnover of essential engineering staff, because retaining core talent is critical to operational continuity if costs are reduced. | unknown | unknown | + +### Edge topology for new nodes + +- `n-oss-realistic` → `depends_on` → central state node +- `n-kr-loss` → `depends_on` → central state node + +Both edges serve the structural role of linking newly admitted unknowns to the situation summary. Neither edge is manufactured solely to satisfy an answer-provenance requirement — they are standard graph wiring present in all valid productions. + +### Answer meaning (from proposal diagnostics) + +``` +userSupportedMeaning: "The user indicates that proceeding requires validation of two specific factors: the realism of projected office savings and ensuring the relocation does not cause a material increase in key engineer turnover." +supportCategory: null +resolutionGuidance: null +``` + +### Question selection diagnostics + +- **Active unknown selected:** `n-oss-realistic` (score=16, objective_match=true) +- **Second candidate:** `n-kr-loss` (score=4, outranked by score delta 12) +- **Question:** "What evidence would clarify project realism and validation of anticipated office relocation savings?" +- **Strategy:** evidence_gathering +- **Reasoning pattern:** diagnosis +- **Question complexity:** acceptable (primaryConceptCount=1, cognitiveLoad=low) + +### Reasoning state (from diagnostics) + +- **Comparability:** confirmed ("The observations are not competing like-for-like measurements.") +- **Relationship:** insufficient_information +- **Atomicity:** atomic — "No deterministic composite pattern was detected, so the unknown can be investigated directly." +- **Decomposition:** attempted but not accepted — "Decomposition stopped because no meaning-preserving child family was justified for this parent." + +## 57J.11 provenance-link rejection: ABSENT + +The previous rejection `"New unknown must be explicitly related to an answer-derived node"` does NOT occur. Both `n-oss-realistic` (savings dimension) and `n-kr-loss` (retention dimension) were admitted through `proposal_compatibility` with HTTP 200 at `update_applied`. No error or validation failure was produced. + +## Target classification + +- **Savings target:** PRESERVED +- **Retention target:** PRESERVED + +## Fake provenance edge: NO + +Both edges linking the new unknowns use the standard `depends_on` relationship to the central state node — this is structural graph wiring, not a fake edge manufactured solely to satisfy answer provenance. No other new edges were introduced whose only apparent role is proving linkage to the user answer. + +## Classification: A — PASS + +The v0.15 update path admits both user-supported evidence dimensions through the production path without rejection at the old 57J.11 provenance-link gate. Both nodes are fully represented with correct label, description, and standard structural edges. The selected next question targets one of the two admitted unknowns (n-oss-realistic) with a valid diagnosis/evaluation strategy. No later failure occurred within this single update. + +## What this experiment established + +1. The v0.15 code path admits user-supported unknowns whose meaning derives from conjunction in the answer without requiring any answer-derived provenance edge to pre-exist on the graph. +2. Two independent evidence dimensions in a single answer are correctly represented as two separate unknown nodes (not collapsed). +3. Both target nodes receive meaningful descriptions grounded in the answer semantics, not generic templates. + +## What this experiment does NOT prove + +1. That the admission works across repeated runs with the same input. +2. That the admission works for unstructured/conjunction answers outside the relocation domain. +3. That downstream investigation (Update 2+) proceeds without new failures at a different boundary. +4. That the `too_broad` conversation health signal (5 active unknowns) does not eventually block later turns. +5. That implicit conjunctions (without "and"/"or") are admitted equally cleanly. + +## Production code changed: NO + +## Prompt changed: NO + +## Schema changed: NO + +## Retries: 0 + +## Ollama calls beyond budget: 0 + +## Documentation updated: YES +