refactor(confidence-engine): compact current-handoff to operational snapshot; archive v0.61 experiment history to ch19
- Reduce docs/current-handoff.md from 2003 → 185 lines (90% reduction) - Move all initial-decomposition v0.61 experiment narrative to docs/archive/experiments/vol-1-chapters/ch19/initial-decomposition-v0.61.md - Update design-evolution/README.md with ch19 Era 8 entry - Add CURRENT MVP DIRECTION section (frozen; OpenAI investigation is next question) - Fix stale branch reference in current-project-state.md - Update Return-to-Work Summary to reflect v0.61 completion - Update Verification Marker for v0.60/v0.61 status
This commit is contained in:
@@ -0,0 +1,159 @@
|
||||
# Ch19 — Initial Decomposition Hardening (v0.61)
|
||||
|
||||
**Status: COMPLETE (frozen)**
|
||||
**Branch:** `feature/initial-decomposition-v0.61`
|
||||
**Final HEAD:** `5878ce4` experiment(confidence-engine): add reconstruction-only helper flag
|
||||
|
||||
## v0.61 Objective
|
||||
|
||||
How stable is the initial semantic decomposition of the same fixed scenario across repeated runs using the same model, prompt version and production API path?
|
||||
|
||||
## Fixed manufacturing scenario (used throughout)
|
||||
|
||||
```
|
||||
I run a small manufacturing business. Customer complaints have risen by 35% over the last six months, while production volume increased by 40%.
|
||||
|
||||
Most complaints mention late delivery or minor product defects, but our complaint categories changed when we introduced a new CRM tagging system three months ago. During the same period we also changed one supplier and introduced a weekend production shift.
|
||||
|
||||
I am deciding whether to spend about £120,000 on automated quality inspection now or wait until we understand whether there is actually a quality problem.
|
||||
|
||||
I do not yet know the complaint rate per unit, whether defect rates differ by shift or supplier, or whether the new tagging system changed what gets counted as a complaint.
|
||||
```
|
||||
|
||||
## v0.61.3 — Matched Provider Reconstruction Evidence Checkpoint
|
||||
|
||||
**Status:** DOCUMENTATION-ONLY CHECKPOINT
|
||||
**Purpose:** Record six matched reconstruction-comparison observations (Qwen + Terra) against the canonical manufacturing fixture before closing out. No live model calls. No production changes.
|
||||
|
||||
### Production prompt / schema frozen
|
||||
- **Prompt:** `reconstruct-v0.5` (production default)
|
||||
- **Rule:** Rule 5a
|
||||
- **Salience check:** semantic-preservation
|
||||
- **Schema:** canonical reconstruction schema
|
||||
- **Path:** reconstruction-only experiment path
|
||||
|
||||
### Qwen / Ollama evidence (3 matched observations)
|
||||
|
||||
| Channel | Finding |
|
||||
|---|---|
|
||||
| A — complaints +35%, production +40%, complaint-rate-per-unit uncertainty | semantically stable |
|
||||
| B — CRM tagging / complaint-count comparability uncertainty | semantically stable |
|
||||
| C — late-delivery versus minor-product-defect distinction | meaning preserved, representation varied |
|
||||
| D1 — supplier-related defect uncertainty | meaning preserved, but supplier and shift frequently recombined |
|
||||
| D2 — weekend-shift-related defect uncertainty | meaning preserved, but supplier and shift frequently recombined |
|
||||
| E — intervention-fit dependency | core dependency preserved, graph topology varied |
|
||||
|
||||
**Important observed pattern:** compound supplier/shift unknown appeared in all three Qwen runs.
|
||||
|
||||
### Terra / OpenAI evidence (3 matched observations)
|
||||
|
||||
| Channel | Finding |
|
||||
|---|---|
|
||||
| A — complaints +35%, production +40% | stable |
|
||||
| B — CRM comparability | stable |
|
||||
| C — late-delivery vs defects | semantically stable and separately represented in all three |
|
||||
| D1 — supplier-related defect uncertainty | stable |
|
||||
| D2 — weekend-shift-related defect uncertainty | stable |
|
||||
| E — intervention-fit dependency | core dependency stable |
|
||||
|
||||
All three Terra runs: supplier transition distinct=YES, weekend-shift distinct=YES, compound supplier/shift=NO, unsupported expansion=NO, causal strengthening=NO.
|
||||
|
||||
### Cross-provider conclusion
|
||||
|
||||
> Exact graph topology is not stable for either provider and should not be treated as a reconstruction invariant. The more important invariant is preservation and independent recoverability of supplied distinctions and uncertainties.
|
||||
|
||||
> Terra showed stronger stability in preserving independently investigable C and D1/D2 distinctions as separate graph structures. Qwen preserved their meaning but more frequently compressed them into combined representations.
|
||||
|
||||
## v0.61 Experiment 5 — Repeated Same-Input Initial Decomposition Stability
|
||||
|
||||
**Status:** INCOMPLETE — five runs required; obtained four successful plus one validation failure.
|
||||
|
||||
### Execution
|
||||
- **Route:** POST /api/cases/start
|
||||
- **Requests made:** 5 (4 successful, 1 validation failure)
|
||||
- **Model:** `qwen-claude:latest`
|
||||
- **Prompt version:** v0.2
|
||||
|
||||
### Structure Range
|
||||
|
||||
| | Nodes | Unknowns | Assumptions |
|
||||
|---|---|---|---|
|
||||
| Run 1 | 16 | 3 | 3 |
|
||||
| Run 2 | 20 | 7 | 2 |
|
||||
| Run 3 | 21 | 3 | 3 |
|
||||
| Run 4 | 14 | 3 | 2 |
|
||||
|
||||
Unknown count: 3–7. Assumption count: 2–3. Total node count: 14–21.
|
||||
|
||||
### Semantic Channel Frequency (Runs 1–4)
|
||||
|
||||
```
|
||||
A Normalisation: 4/4 present, 0/4 partial, 0/4 absent
|
||||
B CRM comparability: 4/4 present, 0/4 partial, 0/4 absent
|
||||
C Delivery vs defects: 1/4 present, 3/4 partial, 0/4 absent
|
||||
D Supplier/shift/source: 4/4 present, 0/4 partial, 0/4 absent
|
||||
E Intervention fit: 1/4 present, 0/4 partial, 3/4 absent
|
||||
```
|
||||
|
||||
### Key findings
|
||||
|
||||
- Run 2 introduced speculative subdivisions not grounded in source text (customer segment variation, QC team capability shifts, responsibility distribution impact, batch-size/equipment utilization correlation).
|
||||
- No run contained steering language, unsupported causal claims, premature arithmetic conclusions, or action recommendations.
|
||||
- The £120k intervention was represented neutrally in all runs.
|
||||
|
||||
### Conclusion
|
||||
|
||||
Initial decomposition represents core factual uncertainty channels (normalisation, CRM comparability, supplier/shift ambiguity) to varying degrees. Primary instability: intervention-fit mapping and speculative subdivision risk. Graph topology varies materially (14–21 nodes).
|
||||
|
||||
## Focused-deconstruction plumbing fix (historical)
|
||||
|
||||
**Root cause:** focused route passed full provider envelope `{ response, providerApiPath, providerExecution }` into `validateFocusedDeconstructSchema()`. Validator expected semantic fields at top level but they lived on `wrapper.response`, not the wrapper. All six fields appeared absent → structured-output 502.
|
||||
|
||||
**Fix:** Route extracts `const deconstruction = wrapper.response` and passes inner object to validation and serialization. Provider diagnostics preserved but do not interfere with semantic fields.
|
||||
|
||||
**Tests:** `tests/focused-deconstruct-boundary.test.js` — 48 focused-investigation-boundary tests pass on first run. Previous two pre-fix semantic runs remain invalid as semantic evidence.
|
||||
|
||||
### Post-fix repeatability (3 controlled Qwen/Ollama runs)
|
||||
|
||||
| Metric | Result |
|
||||
|---|---|
|
||||
| Qwen/Ollama calls | 3 |
|
||||
| HTTP 200 | 3/3 |
|
||||
| schema valid | 3/3 |
|
||||
| supplier evidence preservation | 3/3 PASS |
|
||||
| weekend-shift uncertainty preservation | 3/3 PASS |
|
||||
| epistemic separation | 3/3 PASS |
|
||||
| unsupported inference | 0/3 |
|
||||
| steering | 0/3 |
|
||||
|
||||
**Product interpretation:** For the fixed compound supplier/weekend-shift case, Qwen's initial compression did not prevent focused-investigation stage from repeatedly recovering the epistemic distinction once substantive user evidence was supplied. This supports tolerating some initial representation compression when meaning survives downstream.
|
||||
|
||||
This is **not yet generalised to a production invariant.**
|
||||
|
||||
## v0.61 — Closed experiment boundaries
|
||||
|
||||
| Boundary | Status |
|
||||
|---|---|
|
||||
| Repeated-same-input decomposition experiments | FROZEN |
|
||||
| Qwen/Terra reconstruction comparison | FROZEN |
|
||||
| Initial prompt refinement | FROZEN |
|
||||
| Supplier/weekend-shift decomposition experiments | FROZEN |
|
||||
| Relationship-preservation experiments | FROZEN |
|
||||
| Causal-fidelity experiments | FROZEN |
|
||||
|
||||
## v0.4 initial reconstruction — implementation status
|
||||
|
||||
**Status:** COMPLETE — deterministic tests pass, one live smoke accepted
|
||||
**Prompt:** `prompts/reconstruct-v0.4.md` (replaced by v0.5 in production default)
|
||||
**Tests:** 89/89 passed on first run, zero reruns
|
||||
|
||||
Key principles proven: provenance-preserving decomposition, relationship preservation, explicit stop boundary, interpretation separation. All documented and verified.
|
||||
|
||||
## Git
|
||||
|
||||
- **Documentation commit:** `docs(confidence-engine): record focused deconstruction repeatability`
|
||||
- **Working tree:** clean after this session's commit
|
||||
|
||||
---
|
||||
|
||||
*This chapter records the initial-decomposition v0.61 experiment line. The line is frozen for the current MVP stage. See `docs/current-handoff.md` §CURRENT MVP DIRECTION.*
|
||||
+112
-1930
File diff suppressed because it is too large
Load Diff
@@ -47,7 +47,7 @@ These are what currently affect the working engine:
|
||||
- **Question formulation** — remains an available capability (graph-backed question generation for selected nodes);
|
||||
- Scenario API (analyseScenario / updateCase);
|
||||
- Investigation turn cycle orchestration;
|
||||
- **Reasoning-fidelity v0.8 (completed):** user-supported meaning cannot silently outrun the raw answer at the mutation boundary; evidence-resolvable uncertainty and user-owned ambiguity are routed differently at question formulation. A–F regression boundaries closed for this pass. See `docs/current-handoff.md` for closeout details.
|
||||
- **Reasoning-fidelity v0.8 (completed, frozen for current MVP):** user-supported meaning cannot silently outrun the raw answer at the mutation boundary; evidence-resolvable uncertainty and user-owned ambiguity are routed differently at question formulation. A–F regression boundaries closed for this pass. See `docs/current-handoff.md` for closeout details.
|
||||
|
||||
> **NOTE on investigation ownership:** The user currently owns which unresolved
|
||||
> investigation/question to pursue. Selector-led compulsory next-question
|
||||
@@ -55,7 +55,7 @@ These are what currently affect the working engine:
|
||||
> is also paused. Question formulation remains available as a capability but its
|
||||
> output is not automatically enforced as the user's required next step.
|
||||
|
||||
### Route architecture (v0.55)
|
||||
### Route architecture (UX/product lineage v0.51–v0.60)
|
||||
|
||||
Three distinct routes, each with clear ownership:
|
||||
|
||||
@@ -125,7 +125,7 @@ Multi-investigation identity is durable (`uuidv4`). Legacy singleton compatibili
|
||||
|
||||
## 6. Work Currently Paused
|
||||
|
||||
- Engine experiments advanced through Experiment 43 (Clarify readiness diagnostic confirming zero Clarify eligibility across all real fixtures; orienting-based rule identified as dead code; too_broad trigger validly narrow but untested in fixtures).
|
||||
- Engine experiments advanced through Experiment 43. Initial decomposition hardening (v0.61) completed and frozen for current MVP stage — see `docs/current-handoff.md` §CURRENT MVP DIRECTION.
|
||||
- UI experiments are paused;
|
||||
- Knowledge-management experiments are complete (confirmed by Experiment 38 cold-start validation);
|
||||
- Nothing historical has been deleted or archived yet.
|
||||
@@ -150,16 +150,19 @@ Historical documents are retained under `docs/archive/` and should be opened onl
|
||||
|
||||
## 8. Return-to-Work Summary
|
||||
|
||||
Engine experiments advanced through Experiment 43, which diagnosed Clarify's absence across all real fixtures (zero eligibility in 10 turns). The orienting-based Clarify rule is dead code — the assessor never produces phase=orienting. The too_broad trigger is validly narrow but untested by any fixture. Summarise and Pause remain operational from Exp 42. Behaviour Selection remains passive and isolated.
|
||||
Engine experiments advanced through Experiment 43 (Clarify readiness). UX/product development reached v0.60 (multi-investigation structurally complete) + v0.61 (initial decomposition hardening, frozen for MVP). Reasoning-fidelity v0.8 closed.
|
||||
|
||||
UX/product development reached v0.55: Portfolio / Investigation / Report route separation, persisted report lifecycle (first visit = one synthesis; subsequent visits = zero), confirmation-gated restart, focused-presentation ownership per question, empty Done + Re-open semantics, zero-Open-Questions milestone. All verified via targeted Vitest and live Playwright.
|
||||
Key current state:
|
||||
- Multi-investigation architecture structurally complete (v0.60): durable IDs, listInvestigations, restart container-preserve, report freshness
|
||||
- Initial decomposition frozen (v0.61): semantically stable enough for MVP; exact topology not invariant; see CURRENT MVP DIRECTION in handoff
|
||||
- Focused deconstruction plumbing fixed and verified (48/48 tests)
|
||||
- Passive classifiers operational but isolated
|
||||
|
||||
First document to read: **`docs/current-handoff.md`** (methodology continuity + current state), then `docs/current-project-state.md`, then `docs/project-knowledge-inventory.md`. Consult `.claude/architecture-guardrails.md` before any code changes. The full experiment history remains available in `docs/design-evolution/README.md` but is no longer default reading — load only when a specific question requires it.
|
||||
|
||||
## Verification Marker
|
||||
|
||||
Implementation status last checked against source: Experiment 43.
|
||||
The current-state document was verified as accurate by focused code inspection of API routes, orchestrator imports/calls, and cross-module traces for all passive classifiers. No corrections were required.
|
||||
Implementation status last checked against source: Experiment 43 + v0.61 apparatus (tsx helper, reconstruction-only seam, focused-deconstruction schema fix). Multi-investigation architecture verified at v0.60g2+ and structurally complete. The current-state document was verified as accurate by focused code inspection of API routes, orchestrator imports/calls, and cross-module traces for all passive classifiers. No corrections were required.
|
||||
|
||||
## 10. Post-v0.8 Methodology Learning
|
||||
|
||||
|
||||
@@ -73,6 +73,12 @@ All chapters are exact contiguous extracts from the former monolith at `docs/des
|
||||
|----|------|-------------|---------|
|
||||
| 18 | [ch18/55D-separate-stated-vs-inferred-meaning-through-v058-provenance.md](../archive/experiments/vol-1-chapters/ch18/55D-separate-stated-vs-inferred-meaning-through-v058-provenance.md) | 9983–10322 | Experiments 55D–55F and v0.51–v0.58 product provenance. |
|
||||
|
||||
### Era 8 — Initial Decomposition Hardening (frozen)
|
||||
|
||||
| Ch | Path | Source Lines | Summary |
|
||||
|----|------|-------------|---------|
|
||||
| 19 | [ch19/initial-decomposition-v0.61.md](../archive/experiments/vol-1-chapters/ch19/initial-decomposition-v0.61.md) | — | v0.61 initial decomposition: repeated-same-input stability, Qwen/Terra comparison, focused-deconstruction plumbing fix. Status: frozen for current MVP stage. |
|
||||
|
||||
---
|
||||
|
||||
## Structure Notes
|
||||
|
||||
Reference in New Issue
Block a user