docs: consolidate reasoning refinement requirements
This commit is contained in:
@@ -10147,7 +10147,7 @@ All 4 live inference calls completed successfully. Four answers tested through o
|
||||
|
||||
### Historical Comparison Result
|
||||
|
||||
Compared to Experiment 55C's finding that Stage 1 strengthened "Risk matters more to me." into language about "preference/trade-off rather than absolute hard constraint," Experiment 55D shows the two-field separation can prevent that strengthening when it matters most: weak-priority answers stayed as relative importance in statedMeaning while stronger interpretations were placed separately in possibleInference. The mechanism works for the specific failure mode identified in 55C, but broader stability and downstream consumption remain untested.
|
||||
Compared to Experiment 55C's finding that Stage 1 strengthened "Risk matters more to me." into language about "preference/trade-off rather than absolute hard constraint," Experiment 55D shows the two-field separation avoided the specific weak-priority strengthening defect in this tested run: weak-priority answers stayed as relative importance in statedMeaning while stronger interpretations were placed separately in possibleInference. The mechanism handled the specific failure mode successfully in this probe, but broader stability and downstream consumption remain untested.
|
||||
|
||||
### Documentation Updated
|
||||
|
||||
@@ -10179,3 +10179,36 @@ This experiment created one new test file only. No stated-vs-inferred clarificat
|
||||
### Return-to-Work Note (Experiment 55D)
|
||||
|
||||
55C showed preserved meaning can protect later resolution, but weak-priority meaning was already strengthened in Stage 1. 55D isolated that first interpretation step using a single-call stated-vs-inferred separation with four fixed answers across risk preference cases. Weak priority stayed as relative importance only (direct improvement over 55C's constraint-language strengthening). Conditionality survived through the conditional trade-off case. Explicit and uncertain controls stayed clean — no unsupported meaning leaked into statedMeaning. Stronger implications were kept separate in possibleInference for Cases 1 and 2, though Cases 3 and 4 showed unnecessary inference generation (hygiene issue, not leakage). This does not yet prescribe production architecture. Graph, Behaviour Selection, UI and production remain untouched. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7. First test/file to inspect when resuming: tests/reconstruction/semantic-clarification-stated-vs-inferred.test.js for the full experiment and results. Status pending Rob's review.
|
||||
|
||||
## Experiment 55E — Reasoning Refinement Requirements Synthesis (2026-08-08)
|
||||
|
||||
### Objective
|
||||
|
||||
Consolidate findings from Experiments 53–55D into a compact, implementation-ready reasoning contract for the next production pass. Stop opening new semantic sub-problems for this round.
|
||||
|
||||
### Context Documents Reviewed
|
||||
|
||||
- `docs/current-handoff.md` (current state and handoff);
|
||||
- Experiments 53, 54K–54Z, 55A–55D in `docs/design-evolution-log.md`;
|
||||
- Created: `docs/reasoning-refinement-requirements.md`.
|
||||
|
||||
### Synthesis Output
|
||||
|
||||
- **8 reasoning requirements** retained (R1–R8), all supported by at least one recorded experiment observation;
|
||||
- **7 known failure patterns** recorded from the experiment history;
|
||||
- **8 known good behaviours** recorded as regression expectations;
|
||||
- **6 regression cases** selected (A–F) covering: weak priority, conditional trade-off, non-answer/uncertainty, explicit hard constraint, evidence-resolvable disagreement, and user-owned ambiguity;
|
||||
- **10 open questions** explicitly retained to prevent premature architecture.
|
||||
|
||||
### Key Unresolved Items
|
||||
|
||||
- Stability across larger case sets and other models;
|
||||
- Exact production representation (graph integration pending);
|
||||
- Downstream consumption of inference fields;
|
||||
- Behaviour Selection and UI integration timing;
|
||||
- Performance/latency implications;
|
||||
- One call versus multiple calls for semantic separation.
|
||||
|
||||
### Conclusion
|
||||
|
||||
This round of semantic experimentation is closed. The requirements synthesis in `docs/reasoning-refinement-requirements.md` provides a bounded starting point for tomorrow's implementation pass. No live inference was performed. No production code, prompts, schemas, graph files, Behaviour Selection rules, or UI code were modified. The mechanism from Experiment 55D avoided the specific weak-priority strengthening defect in this tested run; broader stability remains unproven.
|
||||
|
||||
Reference in New Issue
Block a user