docs(confidence-engine): record matched provider reconstruction evidence

2026-09-06 13:28:55 +01:00
parent d24ad48f62
commit a0f90e8885
1 changed files with 104 additions and 0 deletions
@@ -1755,6 +1755,110 @@ The initial decomposition represents core factual uncertainty channels (normalis

 ---

+## v0.61.3 — Matched Provider Reconstruction Evidence Checkpoint
+
+**Status: DOCUMENTATION-ONLY CHECKPOINT**
+
+**Purpose:** Record six matched reconstruction-comparison observations (Qwen + Terra) against the canonical manufacturing fixture before moving downstream. No live model calls. No production changes.
+
+### Fixture under observation
+
+```
+tests/fixtures/manufacturing-complaints-scenario.json
+```
+
+Identical across all runs.
+
+### Production prompt / schema frozen
+
+- **Prompt:** `reconstruct-v0.5` (production default)
+- **Rule:** Rule 5a
+- **Salience check:** semantic-preservation
+- **Schema:** canonical reconstruction schema
+- **Path:** reconstruction-only experiment path
+
+### Qwen / Ollama evidence (3 matched observations)
+
+| Channel | Finding |
+|---|---|
+| A — complaints +35%, production +40%, complaint-rate-per-unit uncertainty | semantically stable |
+| B — CRM tagging / complaint-count comparability uncertainty | semantically stable |
+| C — late-delivery versus minor-product-defect distinction | meaning preserved, representation varied |
+| D1 — supplier-related defect uncertainty | meaning preserved, but supplier and shift frequently recombined |
+| D2 — weekend-shift-related defect uncertainty | meaning preserved, but supplier and shift frequently recombined |
+| E — intervention-fit dependency (whether £120k automated inspection is appropriate depends on understanding the actual quality problem) | core dependency preserved, graph topology varied |
+
+**Important observed pattern:** compound supplier/shift unknown appeared in all three Qwen runs.
+
+**Representation variance (Qwen):** late-delivery / defect graph granularity varied; supplier / weekend-shift node granularity varied; investment dependency topology varied; overall node count varied.
+
+**Wording of concern:** Qwen shows clear representation variance and a recurring semantic-compression tendency around independently investigable supplier/shift distinctions. This is documented as an observation, not yet classified as harmful or proven defective.
+
+### Terra / OpenAI evidence (3 matched observations)
+
+| Channel | Finding |
+|---|---|
+| A — complaints +35%, production +40% | stable |
+| B — CRM comparability | stable |
+| C — late-delivery vs defects | semantically stable and separately represented in all three |
+| D1 — supplier-related defect uncertainty | stable |
+| D2 — weekend-shift-related defect uncertainty | stable |
+| E — intervention-fit dependency | core dependency stable |
+
+**All three Terra runs:**
+
+| Check | Result |
+|---|---|
+| Supplier transition distinct | YES |
+| Weekend-shift transition distinct | YES |
+| Supplier unknown distinct | YES |
+| Weekend-shift unknown distinct | YES |
+| Compound supplier/shift unknown | NO |
+| Unsupported expansion | NO |
+| Causal strengthening | NO |
+
+**Structural/topological variance in Terra runs:** node counts varied; encoding between observedStates / differences / transitions varied; description granularity varied; E topology varied while preserving the dependency. Do not interpret this as deterministic graph output.
+
+### Cross-provider conclusion
+
+> Exact graph topology is not stable for either provider and should not be treated as a reconstruction invariant. The more important invariant is preservation and independent recoverability of supplied distinctions and uncertainties.
+
+> Terra showed stronger stability in preserving independently investigable C and D1/D2 distinctions as separate graph structures. Qwen preserved their meaning but more frequently compressed them into combined representations.
+
+**Not claimed:** provider superiority (sample too small, one fixture), prompt defect or fix, production provider migration decision.
+
+### Cost / development interpretation
+
+> Terra is currently useful as a paid reference/comparator. Qwen remains valuable as the low-cost/local development and potential runtime provider. The practical question is whether Qwen's semantic compression causes a downstream product consequence.
+
+No pricing assumptions or commercial projections added.
+
+### Next product boundary
+
+> **Does Qwen's recurring compression of independently investigable distinctions materially degrade the downstream Confidence Engine investigation flow?**
+
+Examples of downstream consequences to investigate later: an uncertainty becomes inaccessible as an Open Question; supplier and shift cannot be investigated independently; answer integration loses attribution; Current Understanding becomes less faithful; later graph updates cannot recover the original distinction.
+
+This experiment is **NOT** designed or run in this checkpoint task.
+
+### Live calls during this checkpoint
+
+- Qwen/Ollama: 0
+- Terra/OpenAI: 0
+- Application API: 0
+
+### Production / prompt / schema changed
+
+- Production code changed: NO
+- Prompt changed: NO
+- Schema changed: NO
+
+### Git
+
+**Commit message:** `docs(confidence-engine): record matched provider reconstruction evidence`
+
+---
+
 ## v0.61 active restart point

 The current handoff direction is the repeated-same-input decomposition stability experiment described in the **active design** section at the top of §v0.61.