From 580b2a122e356d2d01bd7e75f01ff1fddb013427 Mon Sep 17 00:00:00 2001 From: robbond Date: Fri, 4 Sep 2026 15:56:00 +0100 Subject: [PATCH] docs(confidence-engine): record v0.61 decomposition experiment 2 --- docs/current-handoff.md | 100 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 100 insertions(+) diff --git a/docs/current-handoff.md b/docs/current-handoff.md index 59b5460..93f20b2 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -1069,6 +1069,106 @@ The `@/` alias is a legitimate architectural choice that should not be changed. --- +## v0.61 Experiment 2 — Initial Decomposition Coverage (Scenario Reuse) + +**Status: B — USEFUL BUT MATERIAL UNCERTAINTY LOST** + +### Execution + +| Item | Value | +|---|---| +| Branch | `feature/initial-decomposition-v0.61` | +| Starting HEAD | `6425540 test(confidence-engine): prove direct helper production seam` | +| Route | POST /api/cases/start | +| Requests made | 1 | +| Successful results | 1 | +| Retries | 0 | +| Execution errors | NONE | +| Model | `qwen-claude:latest` | +| Response duration | 77,984 ms | +| Validation status | valid | +| Prompt version | v0.2 | + +### Observed Decomposition + +**Summary:** "A manufacturing business owner is evaluating whether to invest £120k in automated quality inspection amid rising complaint counts and recent operational changes, but lacks rate-based and categorical data to determine if a genuine quality or delivery problem exists." + +**Graph topology:** 14 nodes (1 state, 3 observation, 3 metric, 2 relationship, 3 unknown, 2 assumption), 6 edges. + +**Unknown investigative paths:** +1. `nvb2z51` — Current and historical complaint rate per unit produced is unknown +2. `nn7tkbd` — Whether the new CRM system changed the threshold or definition of a valid complaint compared to the old system +3. `n9fhzkz` — Defect and delivery failure rates specifically attributed to the new supplier or weekend shift are unknown + +**Other materially relevant graph content:** +- `nqpcnaa` (relationship): Complaint volume rose less than production volume (35% vs 40%), suggesting the complaint rate per unit may have decreased or remained stable +- `netiuwi` (relationship): Complaint categories changed three months ago due to CRM updates, decoupling historical complaint data from current classification methods +- `npf0jdh` (assumption): No genuine quality degradation exists +- `n1kszew` (assumption): A true quality or delivery issue exists, likely triggered by new supplier or weekend shift + +**Note on selected question:** The automatic unknown selection process produced a decomposition-rejection in this run. The `selectedQuestion` is null; the system generated a reconstruction question but rejected it as not authoritative, deferring to graph-backed pipeline. Three unresolved unknowns remain at initial decomposition — no auto-selection/prioritisation occurred beyond tie-breaking via scoring heuristics. + +### Semantic Reference Evaluation (A–F) + +**A — facts and explicit unknowns: CLEARLY PRESERVED** +Evidence: All six material facts present as distinct nodes (complaints +35%, production +40%, late delivery/minor defects, CRM tagging change, supplier change, weekend shift). £120k decision node present. Three explicit unknowns preserved. No invented facts. + +**B — normalization uncertainty: CLEARLY PRESERVED** +Evidence: Node `nqpcnaa` explicitly captures the 35% vs 40% divergence as a relationship node with status "supported" and confidence "medium". Notes that complaint rate per unit may have decreased or remained stable despite higher absolute numbers. This is distinct from arithmetic resolution — it frames the question without resolving it. + +**C — measurement/comparability: PARTIALLY PRESERVED** +Evidence: Node `netiuwi` captures CRM categories changed decoupling historical data. However, this is merged with a threshold/definition unknown (`nn7tkbd`) rather than remaining as its own independent material concern. The distinctness of "measurement instrument changed" from "rate normalization needed" is partially compressed. + +**D — nature of the problem: PARTIALLY PRESERVED** +Evidence: "Late delivery" and "minor product defects" are bundled into a single observation node (`nqjt07e`). The distinction between a delivery/logistics problem and a quality/defect problem is not preserved at sufficient granularity to guide targeted investigation. + +**E — causal/source ambiguity: CLEARLY PRESERVED** +Evidence: Supplier change, weekend shift, CRM/tagging, and production volume changes all present in the graph without any edge asserting causality between them and the complaint increase. Node `nwg8ma4` bundles these as known metric changes; assumptions (`npf0jdh`, `n1kszew`) frame competing hypotheses without resolution. + +**F — intervention fit: PARTIALLY PRESERVED** +Evidence: £120k automation cost node exists (`nubgwjr`). However, no graph content explicitly connects the proposed intervention to what it would or would not address (e.g., automated inspection does not resolve late delivery). The gap between "there might be a quality problem" and "£120k automated inspection is the right response" is not surfaced. + +### Failure Patterns + +| Pattern | Result | Evidence | +|---|---|---| +| F1 — premature conclusion | NO | Both competing assumptions presented; no resolution asserted | +| F2 — invented evidence | NO | All facts trace to scenario text | +| F3 — material qualification lost | YES — late delivery/defect distinction (D) compressed; intervention-fit gap (F) not surfaced as explicit unknown | +| F4 — steering/prioritisation | NO | No "first/most important/primary/priority" language in graph or summary | +| F5 — action recommendation | NO | Summary uses neutral framing ("evaluating whether to invest"); no buy/wait directive | +| F6 — redundant Open Questions | NO | Three unknowns are materially distinct | +| F7 — premature arithmetic resolution | NO | 35% vs 40% divergence framed as open question, not resolved | +| F8 — speculative proliferation | NO | Two assumptions are direct negations of each other (minimal viable pair) | +| F9 — unsupported action implication | NO | No explicit recommendation to buy or wait; "waiting" only implied by null selectedQuestion | + +### Classification: B — USEFUL BUT MATERIAL UNCERTAINTY LOST + +**Reason:** The decomposition is useful and restrained — all facts are preserved, no steering, no premature conclusions. Three materially distinct unknowns at initial decomposition. However, dimensions D (nature of the problem) and F (intervention fit) are partially lost through compression: the late-delivery/defect distinction collapses into generic "complaints", and the £120k intervention's potential mismatch with delivery problems is not surfaced as an explicit uncertainty channel. This is a material compression, though less severe than Experiment 1 where CRM comparability and intervention fit were fully compressed. + +### Comparison with Clean Experiment 1 + +**Shared dimensions:** Both experiments lose CRM measurement/comparability distinctness (present but merged into threshold question in Exp 2, fully compressed in Exp 1). Both lose intervention-fit specificity — £120k automation is listed as a fact but not connected to what it would/wouldn't address. Neither experiment steers toward any investigation path. + +**Different dimensions:** Experiment 2 **surfaces normalization uncertainty** via node `nqpcnaa` (35% vs 40% rate divergence) — this was absent in Experiment 1. This is a material difference: the explicit framing that "complaints rose less than production" preserves a channel of inquiry that was missing before. + +**What two observations support:** The engine's initial decomposition captures normalization where it previously did not, suggesting partial semantic coverage improvement or scenario-sensitive behaviour. Both experiments consistently compress CRM measurement distinctness and intervention-fit gaps. + +**What two observations do NOT establish:** Systematic instability, model inconsistency, reliable coverage, or reproducible improvement patterns. Two observations are insufficient to establish either direction. + +### Git + +**Documentation commit:** `docs(confidence-engine): record v0.61 decomposition experiment 2` +**Working tree:** clean after commit + +### Next Evidence Question + +One neutral evidence question only: + +> Does the current initial decomposition preserve the late-delivery vs quality-defect distinction as separate investigative paths when the scenario explicitly frames them as distinct complaint categories, and does the decomposition surface what a proposed intervention would NOT address? + +--- + ## Next restart point > v0.60 is complete. Report is established as the culmination of an Investigation. No next product boundary is currently selected. Begin the next session by choosing the next unresolved user/product reasoning boundary from current product behaviour and founding principles, rather than continuing storage migration or assuming an old backlog item is next.