From a0f90e8885369525166d000157fcf65af8b59511 Mon Sep 17 00:00:00 2001 From: robbond Date: Sun, 6 Sep 2026 13:28:55 +0100 Subject: [PATCH] docs(confidence-engine): record matched provider reconstruction evidence --- docs/current-handoff.md | 104 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 104 insertions(+) diff --git a/docs/current-handoff.md b/docs/current-handoff.md index c2baa2d..fc13d24 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -1755,6 +1755,110 @@ The initial decomposition represents core factual uncertainty channels (normalis --- +## v0.61.3 — Matched Provider Reconstruction Evidence Checkpoint + +**Status: DOCUMENTATION-ONLY CHECKPOINT** + +**Purpose:** Record six matched reconstruction-comparison observations (Qwen + Terra) against the canonical manufacturing fixture before moving downstream. No live model calls. No production changes. + +### Fixture under observation + +``` +tests/fixtures/manufacturing-complaints-scenario.json +``` + +Identical across all runs. + +### Production prompt / schema frozen + +- **Prompt:** `reconstruct-v0.5` (production default) +- **Rule:** Rule 5a +- **Salience check:** semantic-preservation +- **Schema:** canonical reconstruction schema +- **Path:** reconstruction-only experiment path + +### Qwen / Ollama evidence (3 matched observations) + +| Channel | Finding | +|---|---| +| A — complaints +35%, production +40%, complaint-rate-per-unit uncertainty | semantically stable | +| B — CRM tagging / complaint-count comparability uncertainty | semantically stable | +| C — late-delivery versus minor-product-defect distinction | meaning preserved, representation varied | +| D1 — supplier-related defect uncertainty | meaning preserved, but supplier and shift frequently recombined | +| D2 — weekend-shift-related defect uncertainty | meaning preserved, but supplier and shift frequently recombined | +| E — intervention-fit dependency (whether £120k automated inspection is appropriate depends on understanding the actual quality problem) | core dependency preserved, graph topology varied | + +**Important observed pattern:** compound supplier/shift unknown appeared in all three Qwen runs. + +**Representation variance (Qwen):** late-delivery / defect graph granularity varied; supplier / weekend-shift node granularity varied; investment dependency topology varied; overall node count varied. + +**Wording of concern:** Qwen shows clear representation variance and a recurring semantic-compression tendency around independently investigable supplier/shift distinctions. This is documented as an observation, not yet classified as harmful or proven defective. + +### Terra / OpenAI evidence (3 matched observations) + +| Channel | Finding | +|---|---| +| A — complaints +35%, production +40% | stable | +| B — CRM comparability | stable | +| C — late-delivery vs defects | semantically stable and separately represented in all three | +| D1 — supplier-related defect uncertainty | stable | +| D2 — weekend-shift-related defect uncertainty | stable | +| E — intervention-fit dependency | core dependency stable | + +**All three Terra runs:** + +| Check | Result | +|---|---| +| Supplier transition distinct | YES | +| Weekend-shift transition distinct | YES | +| Supplier unknown distinct | YES | +| Weekend-shift unknown distinct | YES | +| Compound supplier/shift unknown | NO | +| Unsupported expansion | NO | +| Causal strengthening | NO | + +**Structural/topological variance in Terra runs:** node counts varied; encoding between observedStates / differences / transitions varied; description granularity varied; E topology varied while preserving the dependency. Do not interpret this as deterministic graph output. + +### Cross-provider conclusion + +> Exact graph topology is not stable for either provider and should not be treated as a reconstruction invariant. The more important invariant is preservation and independent recoverability of supplied distinctions and uncertainties. + +> Terra showed stronger stability in preserving independently investigable C and D1/D2 distinctions as separate graph structures. Qwen preserved their meaning but more frequently compressed them into combined representations. + +**Not claimed:** provider superiority (sample too small, one fixture), prompt defect or fix, production provider migration decision. + +### Cost / development interpretation + +> Terra is currently useful as a paid reference/comparator. Qwen remains valuable as the low-cost/local development and potential runtime provider. The practical question is whether Qwen's semantic compression causes a downstream product consequence. + +No pricing assumptions or commercial projections added. + +### Next product boundary + +> **Does Qwen's recurring compression of independently investigable distinctions materially degrade the downstream Confidence Engine investigation flow?** + +Examples of downstream consequences to investigate later: an uncertainty becomes inaccessible as an Open Question; supplier and shift cannot be investigated independently; answer integration loses attribution; Current Understanding becomes less faithful; later graph updates cannot recover the original distinction. + +This experiment is **NOT** designed or run in this checkpoint task. + +### Live calls during this checkpoint + +- Qwen/Ollama: 0 +- Terra/OpenAI: 0 +- Application API: 0 + +### Production / prompt / schema changed + +- Production code changed: NO +- Prompt changed: NO +- Schema changed: NO + +### Git + +**Commit message:** `docs(confidence-engine): record matched provider reconstruction evidence` + +--- + ## v0.61 active restart point The current handoff direction is the repeated-same-input decomposition stability experiment described in the **active design** section at the top of §v0.61.