experiment: validate user-supported unknown admission live

This commit is contained in:
2026-08-11 06:38:30 +01:00
parent 2e20d30890
commit fbbd271596
2 changed files with 157 additions and 0 deletions
+27
View File
@@ -344,3 +344,30 @@ Candidate commit `60048a5` (reasoning: require corroboration for conjunction com
### Experiment 57J.11 — Live Unknown Dimensionality Representation
Tested whether a single answer containing two independent evidence dimensions is represented as two separate unknown nodes or collapsed into one compound unknown. **Classification: A — SEPARATE.** Fixed scenario: "We are considering relocating the engineering team to reduce operating costs." Fixed answer: "Before deciding, I need evidence that the projected office savings are realistic and evidence that the move will not materially increase loss of key engineers." One live start + one live update. Update was rejected at `proposal_compatibility` (structural linkage error), but the rejection errors directly revealed two independently named proposed unknown nodes: `n-savings-realism` (Target A) and `n-retention-impact` (Target B). The model did not collapse them into a single compound node. Semantic separability existed in the model proposal before deterministic answerability/decomposition. **What remains unproven:** separation for implicit conjunctions, across models/repeated runs, and whether both nodes survive structural gating (`proposal_compatibility`) or question-selection after graph mutation. Full record in `docs/experiment-57j11.md`. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No production code changed.
### Experiment 57J.25 — Live User-Supported Unknown Admission Validation
**Objective:** Validate that the v0.15 candidate admits two user-supported unknowns from the 57J.11 case through the live production `updateCase()` path without requiring fake provenance edges.
**Pre-written expectation:** The answer explicitly introduces two independent uncertainties: savings realism and retention impact. If v0.15 works on the live production path, those user-supported unknowns should no longer be rejected solely because they lack an answer-derived provenance edge. No fake edge should be required or manufactured. A later failure at a different validation/reasoning boundary is acceptable evidence and must be recorded as the first new failure.
**Fixed scenario:** "We are considering relocating the engineering team to reduce operating costs."
**Fixed answer:** "Before deciding, I need evidence that the projected office savings are realistic and evidence that the move will not materially increase loss of key engineers."
**Live-call count:** start: 1, update: 1, total: 2
**START:** HTTP 200 | stage: unknown | nodes: 6 | edges: 3 | question: "What would clarify detailed breakdown of current engineering operating costs in this situation?"
**UPDATE 1:** HTTP 200 | stage: update_applied | nodes: 8 (+2) | edges: 5 (+2) | no errors/validation failures. Both proposed unknowns admitted: `n-oss-realistic` (savings realism) and `n-kr-loss` (retention impact), each with meaningful descriptions grounded in the answer semantics. Selected next question targets `n-oss-realistic` via evidence_gathering/diagnosis strategy.
**57J.11 provenance-link rejection:** ABSENT — no proposal_compatibility rejection occurred. Both target dimensions survived.
**Savings target:** PRESERVED
**Retention target:** PRESERVED
**Fake provenance edge:** NO
**Classification: A — PASS.** The v0.15 update path admits both user-supported evidence dimensions through the production path without rejection at the old 57J.11 provenance-link gate. No later failure occurred within this single update.
**What remains unproven:** Repeated-run stability; cross-domain generalisation; downstream investigation viability past Update 2+; implicit conjunction admission; whether `too_broad` conversation health eventually blocks later turns.
Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No production code changed. Branch: `feature/user-supported-unknown-admission-v0.15`.
+130
View File
@@ -0,0 +1,130 @@
# Experiment 57J.25 — Live Unknown Admission v0.15 Validation
**Objective:** Validate that the v0.15 candidate admits two user-supported unknowns from the 57J.11 case through the live production `updateCase()` path without requiring fake provenance edges.
## Pre-written expectation
> The answer explicitly introduces two independent uncertainties: savings realism and retention impact. If v0.15 works on the live production path, those user-supported unknowns should no longer be rejected solely because they lack an answer-derived provenance edge. No fake edge should be required or manufactured. A later failure at a different validation/reasoning boundary is acceptable evidence and must be recorded as the first new failure.
## Configured apparatus
- **Ollama base URL:** `http://192.168.1.111:11434`
- **Model:** `qwen-claude:latest`
- **Harness:** `scripts/reproduce-multi-turn-investigation.mjs` (canonical)
- **Branch:** `feature/user-supported-unknown-admission-v0.15`
- **Production API path:** `/api/cases/start``/api/cases/update`
## Fixed scenario and answer
**Scenario:** "We are considering relocating the engineering team to reduce operating costs."
**Answer:** "Before deciding, I need evidence that the projected office savings are realistic and evidence that the move will not materially increase loss of key engineers."
## Live-call count
```
start: 1
update: 1
total: 2
```
## START
- **HTTP status:** 200
- **Stage:** unknown (initial state)
- **Node count:** 6
- **Edge count:** 3
- **Selected question:** "What would clarify detailed breakdown of current engineering operating costs in this situation?"
## UPDATE 1
- **HTTP status:** 200
- **Stage:** `update_applied`
- **Error/validation summary:** null (none)
- **Node count:** 8 (+2 new)
- **Edge count:** 5 (+2 new)
- **Selected question:** "What would clarify realism of projected office savings in this situation?"
## Detailed result capture
### Two proposed unknown nodes
| id | label | description | kind | status |
|---|---|---|---|---|
| `n-oss-realistic` | Project realism and validation of anticipated office relocation savings. | Whether the projected financial savings from the relocation are realistic and achievable, so that the cost reduction objective can be trusted as a driver for the decision. | unknown | unknown |
| `n-kr-loss` | Projected impact of the move on key engineer retention rates. | The extent to which the relocation could cause a material increase in the turnover of essential engineering staff, because retaining core talent is critical to operational continuity if costs are reduced. | unknown | unknown |
### Edge topology for new nodes
- `n-oss-realistic``depends_on` → central state node
- `n-kr-loss``depends_on` → central state node
Both edges serve the structural role of linking newly admitted unknowns to the situation summary. Neither edge is manufactured solely to satisfy an answer-provenance requirement — they are standard graph wiring present in all valid productions.
### Answer meaning (from proposal diagnostics)
```
userSupportedMeaning: "The user indicates that proceeding requires validation of two specific factors: the realism of projected office savings and ensuring the relocation does not cause a material increase in key engineer turnover."
supportCategory: null
resolutionGuidance: null
```
### Question selection diagnostics
- **Active unknown selected:** `n-oss-realistic` (score=16, objective_match=true)
- **Second candidate:** `n-kr-loss` (score=4, outranked by score delta 12)
- **Question:** "What evidence would clarify project realism and validation of anticipated office relocation savings?"
- **Strategy:** evidence_gathering
- **Reasoning pattern:** diagnosis
- **Question complexity:** acceptable (primaryConceptCount=1, cognitiveLoad=low)
### Reasoning state (from diagnostics)
- **Comparability:** confirmed ("The observations are not competing like-for-like measurements.")
- **Relationship:** insufficient_information
- **Atomicity:** atomic — "No deterministic composite pattern was detected, so the unknown can be investigated directly."
- **Decomposition:** attempted but not accepted — "Decomposition stopped because no meaning-preserving child family was justified for this parent."
## 57J.11 provenance-link rejection: ABSENT
The previous rejection `"New unknown must be explicitly related to an answer-derived node"` does NOT occur. Both `n-oss-realistic` (savings dimension) and `n-kr-loss` (retention dimension) were admitted through `proposal_compatibility` with HTTP 200 at `update_applied`. No error or validation failure was produced.
## Target classification
- **Savings target:** PRESERVED
- **Retention target:** PRESERVED
## Fake provenance edge: NO
Both edges linking the new unknowns use the standard `depends_on` relationship to the central state node — this is structural graph wiring, not a fake edge manufactured solely to satisfy answer provenance. No other new edges were introduced whose only apparent role is proving linkage to the user answer.
## Classification: A — PASS
The v0.15 update path admits both user-supported evidence dimensions through the production path without rejection at the old 57J.11 provenance-link gate. Both nodes are fully represented with correct label, description, and standard structural edges. The selected next question targets one of the two admitted unknowns (n-oss-realistic) with a valid diagnosis/evaluation strategy. No later failure occurred within this single update.
## What this experiment established
1. The v0.15 code path admits user-supported unknowns whose meaning derives from conjunction in the answer without requiring any answer-derived provenance edge to pre-exist on the graph.
2. Two independent evidence dimensions in a single answer are correctly represented as two separate unknown nodes (not collapsed).
3. Both target nodes receive meaningful descriptions grounded in the answer semantics, not generic templates.
## What this experiment does NOT prove
1. That the admission works across repeated runs with the same input.
2. That the admission works for unstructured/conjunction answers outside the relocation domain.
3. That downstream investigation (Update 2+) proceeds without new failures at a different boundary.
4. That the `too_broad` conversation health signal (5 active unknowns) does not eventually block later turns.
5. That implicit conjunctions (without "and"/"or") are admitted equally cleanly.
## Production code changed: NO
## Prompt changed: NO
## Schema changed: NO
## Retries: 0
## Ollama calls beyond budget: 0
## Documentation updated: YES