robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

experiment: rerun guarded multi-turn progress cleanly

Browse Source
This commit is contained in:
robbond
2026-08-11 11:41:51 +01:00
parent 4998de54d2
commit b341c9cf2f
2 changed files with 143 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/current-handoff.md
+28
View File
@@ -467,3 +467,31 @@ Configured Ollama: none used. Production code changed: NO. Tests permanently cha
**What this does NOT prove:** That all cold-starts produce 5 nodes; that zero-node proposals are always produced on accepted updates; whether the merged-unknown behavior is intentional or a defect.
Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. 4 live calls total (2 starts + 2 updates in final pipeline; 1 start + 1 update rejected from harness). No production code changed.
### Experiment 57J.36 — Multi-Turn Investigation Progress After Accepted Update 1 (Clean Run)
**Objective:** One clean guarded multi-turn case using the hardened canonical harness (bounded execution, no-retry) to determine whether a semantically faithful first answer produces genuine downstream progress on the second turn.
**Fixed scenario:** "We are considering relocating the engineering team to reduce operating costs."
**Answer 1:** "Before deciding, I need evidence that the projected office savings are realistic and evidence that the move will not materially increase loss of key engineers."
**Answer 2 (not reached):** "The projected savings are based on the current London lease, business rates, service charges, utilities and facilities costs that would no longer be incurred at the same level after the move. The estimate is approximately £2M per year."
**Live-call count:** start: 1, update: 1, total: 2
**START:** HTTP 200 | stage: unknown | nodes: 10 | edges: 5 | question: "What would clarify total projected costs at the new location, including one-time relocation expenses and long-term savings in this situation?"
**UPDATE 1:** HTTP 422 | stage: proposal_compatibility | error: "Update contains no meaningful change"
Rejected proposal snapshot: `answerMeaning.userSupportedMeaning` preserved both evidence dimensions faithfully; all structural fields empty (zero addedNodes, zero addedEdges, zero resolvedUnknownNodeIds, zero updatedNodes).
**Classification: B — DIFFERENT REJECTION.** Rejected for "Update contains no meaningful change" at `proposal_compatibility`, not for semantic strengthening. The LLM produced a structurally null proposal despite semantically faithful meaning extraction. U1-B (not U1-A because no strengthening present; not U1-C/U1-D because nothing was applied).
**Savings realism:** UNCLEAR
**Retention impact:** UNCLEAR
**What this establishes:** (1) When the LLM produces a structurally empty proposal with semantically faithful meaning, `proposal_compatibility` correctly rejects it as a no-op; (2) The gap between semantic extraction and structural creation exists independently of the semantic fidelity gate; (3) Cold-start produced 10 nodes (continuing the variance from 5→8 in 57J.34).
**What this does NOT prove:** That the LLM can produce a structurally non-empty faithful proposal; that downstream progress would occur with an accepted proposal; run-to-run stability of node counts or proposal structure.
Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. 2 live calls total. No production code changed.