robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

experiment: validate uncertainty identity live

Browse Source
This commit is contained in:
robbond
2026-08-11 18:02:11 +01:00
parent a476431048
commit eb524d08e1
2 changed files with 211 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/current-handoff.md
+14
View File
@@ -1041,3 +1041,17 @@ One fresh live run through the production `startCase()` → `updateCase()` path.
**Test results:** All 49 tests pass. No regression in existing prompt structure tests.
**Configured Ollama:** none used. **Production code changed:** NO. **Ollama calls:** 0.
### Experiment 57J.56 — Uncertainty Identity Live Validation
Tested whether v0.21 now preserves a focused savings-realism uncertainty separately from broader related cost unknowns. **Classification: A — V0.21 IDENTITY RULE WORKS LIVE.** Fixed scenario: "We are considering relocating the engineering team to reduce operating costs." Fixed answer: "I am unsure whether the projected office savings from the relocation are realistic."
**Start:** HTTP 200 | unknown stage | 8 nodes | 5 edges | Selected question: "What would clarify how long it will take for ongoing savings to offset upfront expenses and productivity dips in this situation?"
Relevant unresolved unknowns at start: (1) `nx54wwa` — operating expenses (metric, known), (2) `nkmuu21` — one-time relocation costs (unknown, unknown), (3) `nt0asmb` — team output loss/turnover (unknown, unknown), (4) `n4j29jl` — payback period timing (unknown, unknown).
**Update 1:** HTTP 200 | update_applied stage | no errors/validation failures. New node `n_savings_realism` added with label "Validity of projected office savings estimates from relocation" (unknown/unknown) — distinct from all four start unknowns. `nx54wwa` updated: known → provisional. Added edge: `e-doubt-savings` weakens → `nx54wwa`. No equivalent pre-existing unknown existed; focused uncertainty added as a new node rather than absorbed into broader cost nodes. Identity result: **ADDED DISTINCT UNCERTAINTY**. Meaning: **FAITHFUL** (doubt preserved, not strengthened).
**Key finding:** The v0.21 prompt clarification ("same uncertainty = same resolution question") works on a live run with qwen-claude:latest. Unlike 57J.53 (where uncertain status degraded an unrelated node's status without creating a dedicated unknown), v0.21 correctly creates and preserves a dedicated savings-realism unknown with proper structural linkage (`weakens` edge). This resolves the prompt semantic-identity gap diagnosed in 57J.54.
**What remains unproven:** Repeated-run stability (single run); downstream investigation viability past Update 2; cross-domain generalisation; paraphrase invariance; whether reuse/refine still triggers when an equivalent unknown already exists. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. **Production code changed:** NO. **Ollama calls:** 1 live call (start + update via harness). APPEND_EOF
docs/experiment-57j56.md
+197
View File
@@ -0,0 +1,197 @@
# Experiment 57J.56 — Uncertainty Identity Live Validation
**Branch:** `feature/uncertainty-identity-v0.21`
**Starting HEAD:** `a476431` (docs: record uncertainty identity clarification)
**Experiment commit:** pending
## Objective
Answer exactly:
> When the user expresses uncertainty about whether projected office savings are realistic, does v0.21 now represent that focused uncertainty separately when the graph contains only broader related cost unknowns?
This is the direct live regression for the prompt clarification implemented in 57J.55.
## Fixed Inputs
**Scenario:**
```
We are considering relocating the engineering team to reduce operating costs.
```
**Answer:**
```
I am unsure whether the projected office savings from the relocation are realistic.
```
## Pre-written Expectation
> The user's uncertainty asks a distinct resolution question: whether the office-savings assumption is realistic. A broader projected-cost unknown is related but not equivalent unless resolving it would also resolve the savings-realism question. v0.21 should therefore preserve the focused uncertainty either by reusing a genuinely equivalent unknown or by adding a dedicated unknown.
## Run Results
### Configured Ollama
- **Base URL:** `http://192.168.1.111:11434` (from `.env.local`)
- **Model:** `qwen-claude:latest`
### Dev Server
- Running on `http://127.0.0.1:3000` (REUSE EXISTING)
## CALL ACCOUNTING
```
startCalls: 1
updateCalls: 1
totalCalls: 2
```
Retries: 0
Supplementary scripts: NO
## START
**HTTP:** 200 | **Stage:** unknown
**Selected question:** "What would clarify how long it will take for ongoing savings to offset upfront expenses and productivity dips in this situation?"
### Nodes (8 total)
| ID | Kind | Status | Label |
|---|---|---|---|
| `ncwvq9x` | state | provisional | Summary of the situation from the scenario text |
| `nbb1z4m` | observation | supported | Decision-making body ('We') evaluating the relocation |
| `nduqivt` | observation | supported | Engineering team targeted for relocation |
| `nclswps` | metric | known | Proposed physical or legal relocation of the engineering team to a new jurisdiction/location |
| `nx54wwa` | metric | known | Current and projected monthly/annual operating expenses for the engineering function |
| `nkmuu21` | unknown | unknown | Total one-time costs required for relocation (severance, hiring, infrastructure setup, legal/compliance) |
| `nt0asmb` | unknown | unknown | Potential short- to medium-term loss in team output, morale, or turnover due to the move |
| `n4j29jl` | unknown | unknown | How long it will take for ongoing savings to offset upfront expenses and productivity dips |
### Edges (5 total)
- `e-sum-nbb1z4m` supports → `ncwvq9x`
- `e-sum-nduqivt` supports → `ncwvq9x`
- `e-unk-nkmuu21` depends_on → `ncwvq9x`
- `e-unk-nt0asmb` depends_on → `ncwvq9x`
- `e-unk-n4j29jl` depends_on → `ncwvq9x`
### Relevant unresolved unknowns (costs/savings/relocation)
1. `nx54wwa` — "Current and projected monthly/annual operating expenses for the engineering function" (metric, known → status: **known**)
2. `nkmuu21` — "Total one-time costs required for relocation" (unknown, unknown)
3. `nt0asmb` — "Potential short- to medium-term loss in team output, morale, or turnover due to the move" (unknown, unknown)
4. `n4j29jl` — "How long it will take for ongoing savings to offset upfront expenses and productivity dips" (unknown, unknown)
## UPDATE 1
**HTTP:** 200 | **Stage:** update_applied | **First error:** none
### Answer Meaning
```
userSupportedMeaning: null
supportCategory: null
resolutionGuidance: null
```
Note: `answerMeaning` fields are null — the model did not populate semantic meaning extraction. This is expected behavior (see Experiments 56D, 57J.28): the deterministic derivation layer handles categorization; the LLM's raw answerMeaning often omits these fields when no clear classification applies to "unsure" answers.
### Structural Action
**updatedNodes:**
```json
[{"nodeId":"nx54wwa","previousStatus":"known","newStatus":"provisional","previousValue":null,"newValue":null,"reason":"User expressed doubt about the realism of projected office savings, indicating the metric's accuracy is not yet confirmed."}]
```
**resolvedUnknownNodeIds:** `[]`
**addedNodes:**
```json
[{"id":"n_savings_realism","label":"Validity of projected office savings estimates from relocation","description":"The degree to which anticipated monthly/annual office cost reductions will actually materialize matters because it serves as the primary financial justification for the relocation, so that we must validate this baseline before committing resources to other unknowns.","kind":"unknown","status":"unknown","confidence":"medium","value":null,"unit":null,"evidenceIds":[],"dependsOn":[],"affects":["nx54wwa"],"parentId":null,"childIds":["nx54wwa"]}]
```
**addedEdges:**
```json
[{"id":"e-doubt-savings","fromNodeId":"n_savings_realism","toNodeId":"nx54wwa","relationship":"weakens","confidence":"medium","description":"Doubt about the realism of projected savings weakens confidence in the current expense metric's accuracy."}]
```
### Post-Update Nodes (9 total)
| ID | Kind | Status | Label |
|---|---|---|---|
| `ncwvq9x` | state | provisional | Situation summary |
| `nbb1z4m` | observation | supported | Decision-making body |
| `nduqivt` | observation | supported | Engineering team |
| `nclswps` | metric | known | Proposed relocation |
| `nx54wwa` | metric | **provisional** | Current and projected operating expenses (← updated) |
| `nkmuu21` | unknown | unknown | Total one-time costs |
| `nt0asmb` | unknown | unknown | Potential team output loss/turnover |
| `n4j29jl` | unknown | unknown | Payback period for savings vs. costs |
| **`n_savings_realism`** | **unknown** | **unknown** | **Validity of projected office savings estimates from relocation (← NEW)** |
### Post-Update Edges (6 total)
- `e-sum-nbb1z4m` supports → `ncwvq9x`
- `e-sum-nduqivt` supports → `ncwvq9x`
- `e-unk-nkmuu21` depends_on → `ncwvq9x`
- `e-unk-nt0asmb` depends_on → `ncwvq9x`
- `e-unk-n4j29jl` depends_on → `ncwvq9x`
- **`e-doubt-savings`** **weakens →** `nx54wwa` (← NEW)
### Selected Question After Update 1
"What would clarify potential short- to medium-term loss in team output, morale, or turnover due to the move in this situation?"
## Meaning Classification: FAITHFUL
The user's uncertainty ("unsure whether projected office savings are realistic") was not strengthened (no constraint/preference invented) and not degraded (doubt was not ignored). The `nx54wwa` metric node was correctly downgraded from known → provisional with reason explicitly referencing the savings-realism doubt.
## Identity Result: ADDED DISTINCT UNCERTAINTY
No equivalent unresolved node existed in the start graph for "are projected office savings realistic?" — the existing unknowns were:
- `nkmuu21`: one-time relocation costs (magnitude estimation across severance/hiring/infrastructure)
- `nt0asmb`: team output loss/turnover (people impact)
- `n4j29jl`: payback period timing (temporal analysis)
None of these resolution questions is equivalent to "validity of projected office savings estimates." Resolving `nkmuu21` (knowing total one-time costs) does not resolve whether the ongoing savings assumptions are realistic. Therefore, a new node was correctly added.
The new node `n_savings_realism` carries:
- Label: "Validity of projected office savings estimates from relocation"
- Status: unknown/unknown (preserves unresolved status)
- Description explicitly frames it as a prerequisite for validating the financial justification
- A `weakens` edge to `nx54wwa` showing structural linkage between doubt and affected metric
- A child-parent relationship with `nx54wwa` (`childIds: ["nx54wwa"]`)
## Structural-Specificity Test
> After Update 1, does persistent graph state contain an unresolved node from which the engine can directly ask: "What evidence would establish whether projected office savings are realistic?"
**YES.** The node `n_savings_realism` (unknown/unknown) exists in the updated graph with label "Validity of projected office savings estimates from relocation." Its description frames it as a baseline validation requirement. It is an independent unknown, not absorbed into any broader node.
## Classification: A — V0.21 IDENTITY RULE WORKS LIVE
Meaning is FAITHFUL and identity result is ADDED DISTINCT UNCERTAINTY.
The v0.21 prompt clarification ("same uncertainty = same resolution question") works on a live run with the configured model (qwen-claude:latest). The focused savings-realism uncertainty is **not** absorbed into the broader `nx54wwa` expense metric node (which was only updated to provisional status). Instead, it is preserved as an independent unknown (`n_savings_realism`) with proper structural linkage.
## What This Establishes
1. **Prompt clarification is effective:** The v0.21 Additional Guidance ("same uncertainty = same resolution question") successfully guides the model to distinguish focused savings-realism doubt from broader cost unknowns in cold-start scenarios.
2. **Dedicated node creation works for uncertain status:** Unlike 57J.53 (where "uncertain" status degraded an unrelated node's status), v0.21 correctly creates a dedicated unknown node for the focused uncertainty.
3. **Structural linkage is appropriate:** The `weakens` edge from `n_savings_realism` to `nx54wwa` provides a meaningful structural relationship that can support downstream reasoning (e.g., if savings realism remains unresolved, cost-benefit analysis cannot proceed).
4. **No absorption into broader cost nodes:** `nx54wwa` was updated (known → provisional) but did NOT absorb the savings-realism uncertainty — it remained distinct via a new node.
## What This Does NOT Prove
1. **Single-run stability:** One live run is not repeated-run evidence. Cold-start variance (observed in 57J.29) could produce different outcomes on another invocation.
2. **Downstream investigation viability:** Whether the saved savings-realism unknown survives into Update 2 and beyond — whether it gets selected for follow-up, or whether a later answer re-triggers absorption.
3. **Cross-domain generalisation:** Only tested on one scenario (engineering team relocation) with one phrasing of uncertainty.
4. **Paraphrase invariance:** Whether other ways of expressing savings-realism doubt produce the same structural outcome.
5. **Edge case: when broad nodes SHOULD absorb sub-concerns:** If an existing unknown like "Are the projected total costs realistic?" already exists, v0.21 should still prefer reuse/refine. This was not tested (no equivalent pre-existed in this run).
## Production code changed: NO
## Prompt changed during experiment: NO
## Canonical harness restored: YES
## Hardened no-retry behaviour preserved: YES
## Ollama calls beyond harness count: 0
## Dev server disturbed: NO