robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: map reasoning requirements to production path

Browse Source
This commit is contained in:
robbond
2026-08-08 11:08:37 +01:00
parent 162ead2d69
commit 8c1036ecd0
4 changed files with 264 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/design-evolution-log.md
+40
View File
@@ -10212,3 +10212,43 @@ Consolidate findings from Experiments 53–55D into a compact, implementation-re
### Conclusion
This round of semantic experimentation is closed. The requirements synthesis in `docs/reasoning-refinement-requirements.md` provides a bounded starting point for tomorrow's implementation pass. No live inference was performed. No production code, prompts, schemas, graph files, Behaviour Selection rules, or UI code were modified. The mechanism from Experiment 55D avoided the specific weak-priority strengthening defect in this tested run; broader stability remains unproven.
## Experiment 55F — Reasoning Requirements Production Path Map (2026-08-08)
### Objective
Map how reasoning requirements R1–R8 are actually supported (or unsupported) by the existing production code path, using source-inspection only. Trace the answer-to-reasoning flow through prompt building, LLM response parsing and normalization, and graph mutation. Identify which gaps have structural carriers in current schemas and which require new schema fields or logic at specific line locations. This exercise is explicitly NOT architecture design or implementation — it documents what exists today so tomorrow's Codex pass starts from accurate information.
### Context Documents Reviewed
- `docs/reasoning-refinement-requirements.md` (R1–R8 requirements, regression pack A–F);
- `docs/current-handoff.md` (handoff state after 55E);
- `lib/graph/orchestrator.js` — updateCase code path and LLM/provider integration;
- `lib/graph/schema.js` — situationNodeSchema, graphUpdateSchema, updateCaseRequestSchema;
- `lib/graph/update-proposal.js` — parseGraphUpdateProposal with normalization;
- `lib/graph/prompt-builder.js` — buildGraphUpdatePrompt with answer embedding;
- `lib/graph/apply-proposal.js` — applyValidatedProposal and deriveReasoningStateOverride;
- `lib/graph/builder.js` — initial graph construction (not used in update cycles).
### Findings
**Production update path:** user answer → buildGraphUpdatePrompt → LLM provider → parseGraphUpdateProposal → applyValidatedProposal. The full chain was traced with line-number precision for each transition.
**Confirmed gap on provenance:** `situationNodeSchema` has no provenance fields (no source/inference annotation). `graphUpdateSchema` also lacks provenance fields. `updateCaseRequestSchema` carries the raw answer but provides no semantic-meaning fields. Evidence records built during startCase are not returned alongside graph state during update cycles.
**Confirmed gap on meaning preservation:** The answer string in `applyValidatedProposal` reaches only `deriveReasoningStateOverride` at line 2875 and is used solely for a narrow comparability confirmation check. After that point, only the structural graph state (already containing the LLM's interpretation) flows forward — not the original answer meaning.
**Confirmed support:** Existing relationship types distinguish evidence vs clarification needs. Structural validation gates maintain integrity. Decomposition quality gates exist on child unknowns. Null selectedQuestion is structurally valid.
**All eight requirements assessed individually** in a cross-reference matrix showing which have any support (prompt, parse/normalize, application, schema) and where gaps are located.
### Key Unresolved Items
- Whether provenance fields should be added to `situationNodeSchema`, `graphUpdateSchema`, or both;
- How meaning preservation verification compares original answer text against proposed graph changes;
- Where in the four-step pipeline (schema → prompt → parse → mutation) semantic-meaning carriers must enter;
- Whether the current approach (two-field interpretation contract from 55D) is viable given the lack of schema carrier, or if a different mechanism is required.
### Conclusion
Source-inspection-only exercise completed. The production path does not carry semantic meaning — it carries structural graph changes that represent the LLM's interpretation of the answer. Every R1–R8 requirement depends on mechanisms absent from the current code path. A complete cross-reference with specific line-location gap targets is in `docs/reasoning-production-path-map.md`. No live inference was performed. No production code, prompts, schemas, graph files, Behaviour Selection rules, or UI code were modified. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: docs/reasoning-production-path-map.md for the full gap analysis and specific line-location targets. Status pending Rob's review.