docs(confidence-engine): complete design evolution archive extraction
This commit is contained in:
@@ -213,15 +213,72 @@ Experiments 54W–54Z — clarification chain end-to-end integrity, specificity
|
||||
Fidelity:
|
||||
Exact contiguous copy.
|
||||
|
||||
Tranches 1 through 7 have now been extracted.
|
||||
The original monolithic log remains intact and authoritative while extraction is incomplete.
|
||||
## Tranche 8
|
||||
|
||||
### Chapter 15
|
||||
Path:
|
||||
docs/archive/experiments/vol-1-chapters/ch15/55A-preserve-uncertainty-from-weak-clarification-answers.md
|
||||
|
||||
Original source:
|
||||
docs/design-evolution-log.md lines 9296–9529
|
||||
|
||||
Contents:
|
||||
Experiment 55A — Can the Model Preserve Uncertainty From Weak Clarification Answers? Testing fully explicit hard constraint, weak priority statement, conditional trade-off, and non-answer/insufficient clarification cases.
|
||||
|
||||
Fidelity:
|
||||
Exact contiguous copy.
|
||||
|
||||
### Chapter 16
|
||||
Path:
|
||||
docs/archive/experiments/vol-1-chapters/ch16/55B-separate-answer-meaning-from-resolution-judgement.md
|
||||
|
||||
Original source:
|
||||
docs/design-evolution-log.md lines 9530–9748
|
||||
|
||||
Contents:
|
||||
Experiment 55B — Separate Answer Meaning from Resolution Judgement. Mode A instruction/output contract versus Mode B inference separation, tested on weak priority, conditional trade-off, and non-answer cases.
|
||||
|
||||
Fidelity:
|
||||
Exact contiguous copy.
|
||||
|
||||
### Chapter 17
|
||||
Path:
|
||||
docs/archive/experiments/vol-1-chapters/ch17/55C-resolution-from-preserved-answer-meaning.md
|
||||
|
||||
Original source:
|
||||
docs/design-evolution-log.md lines 9749–9982
|
||||
|
||||
Contents:
|
||||
Experiment 55C — Resolution From Preserved Answer Meaning. Two-stage resolution using preserved meaning rather than raw answer, tested on weak priority, conditional trade-off, and non-answer cases.
|
||||
|
||||
Fidelity:
|
||||
Exact contiguous copy.
|
||||
|
||||
### Chapter 18
|
||||
Path:
|
||||
docs/archive/experiments/vol-1-chapters/ch18/55D-separate-stated-vs-inferred-meaning-through-v058-provenance.md
|
||||
|
||||
Original source:
|
||||
docs/design-evolution-log.md lines 9983–10322
|
||||
|
||||
Contents:
|
||||
Experiments 55D (separate stated clarification meaning from inference), 55E (reasoning refinement requirements synthesis), 55F (reasoning requirements production path map), and v0.51–v0.58 Progress — product provenance and architectural decisions.
|
||||
|
||||
Fidelity:
|
||||
Exact contiguous copy.
|
||||
|
||||
The extraction phase is complete.
|
||||
|
||||
Tranches 1 through 8 now preserve the substantive historical contents of
|
||||
`docs/design-evolution-log.md` as exact chapter extracts.
|
||||
|
||||
The original monolithic log remains intact and authoritative until a separate
|
||||
compatibility/routing task replaces it with an index or pointer and updates repository references.
|
||||
|
||||
## Refactor status
|
||||
|
||||
Only tranches 1 and 2 have been extracted.
|
||||
|
||||
Do not remove these ranges from `docs/design-evolution-log.md` yet.
|
||||
|
||||
Do not update repository-wide references to point here yet.
|
||||
|
||||
Further Design Evolution material must be extracted in separate bounded tranches and verified before the original monolith is replaced by a compatibility/index document.
|
||||
The original monolith remains authoritative pending migration.
|
||||
|
||||
+234
@@ -0,0 +1,234 @@
|
||||
## Experiment 55A — Can the Model Preserve Uncertainty From Weak Clarification Answers? (2026-08-08)
|
||||
|
||||
### Objective
|
||||
|
||||
First, tighten Experiment 54Z so its conclusion does not over-generalise from two weaker answers.
|
||||
|
||||
Then isolate the strongest unresolved issue from 54Z:
|
||||
|
||||
> When a clarification answer is weak or conditional, can the model preserve uncertainty instead of forcing the answer into a stronger resolved meaning?
|
||||
|
||||
This experiment is about answer interpretation only. Do not compare precise versus broadened targets. Do not test question wording. Do not integrate with Behaviour Selection, graph, or UI.
|
||||
|
||||
### Hypothesis
|
||||
|
||||
The model may be too eager to convert weak clarification answers into resolved meaning. A better behaviour would preserve uncertainty when the answer does not actually settle the supplied target. If the model consistently keeps weak answers unresolved, the 54Z over-resolution may have been incidental. If it repeatedly forces resolution, that becomes a clearer limitation of the answer-resolution step itself.
|
||||
|
||||
### Configuration
|
||||
|
||||
Host: `http://192.168.1.111:11434` (same as all prior experiments)
|
||||
Model: `qwen-claude:latest` (same as all prior experiments)
|
||||
|
||||
### Number of Live Inference Calls
|
||||
|
||||
Exactly **4** live Ollama calls — one answer-resolution per case. Exactly 4 cases of varying strength against the same source/target/question.
|
||||
|
||||
### Context Used
|
||||
|
||||
- `docs/current-handoff.md`
|
||||
- Experiment 54Z in `docs/design-evolution-log.md` (as basis for isolating the unresolved issue)
|
||||
- `tests/reconstruction/semantic-clarification-weak-answer-consequence.test.js` (structural reference)
|
||||
- `tests/reconstruction/semantic-clarification-answer-resolution.test.js` (structural reference: instruction and output contract)
|
||||
|
||||
### Semantic Instruction Unchanged
|
||||
|
||||
> Use the user's clarification answer only to resolve the supplied clarification target. State the meaning now established by that answer. Mark targetResolved true only when the answer settles the target. Put any uncertainty that remains specifically about that target into remainingUncertainty; otherwise return null. Do not infer wider consequences, rewrite unrelated source meaning, recommend action, or generate another question.
|
||||
|
||||
No examples added. No stronger coaching attempted. Same instruction as Experiment 54V.
|
||||
|
||||
### Output Contract Unchanged
|
||||
|
||||
```json
|
||||
{
|
||||
"resolvedMeaning": "short statement",
|
||||
"targetResolved": true,
|
||||
"remainingUncertainty": "short statement or null"
|
||||
}
|
||||
```
|
||||
|
||||
### Fixed Source
|
||||
|
||||
> I want the business to grow, but I don't want to take on more risk.
|
||||
|
||||
### Clarification Target
|
||||
|
||||
> whether avoiding additional risk is a preference/trade-off or a hard constraint.
|
||||
|
||||
### Clarification Question
|
||||
|
||||
> Do you view avoiding additional risk as a hard constraint, or as a preference or trade-off?
|
||||
|
||||
---
|
||||
|
||||
### Case 1 — Fully Explicit Hard Constraint
|
||||
|
||||
**User answer:** "It's a hard constraint. I don't want any increase in risk."
|
||||
|
||||
**Model output:**
|
||||
- resolvedMeaning: "Avoiding additional risk is a hard constraint."
|
||||
- targetResolved: true
|
||||
- remainingUncertainty: null
|
||||
|
||||
**Classification: resolution_correct**
|
||||
|
||||
The model correctly resolved the explicit hard constraint with appropriate strength and no spurious uncertainty. This is the positive control — it behaved as expected.
|
||||
|
||||
---
|
||||
|
||||
### Case 2 — Weak Priority Statement
|
||||
|
||||
**User answer:** "Risk matters more to me."
|
||||
|
||||
**Model output:**
|
||||
- resolvedMeaning: "Avoiding additional risk is prioritized as a strong preference or guiding principle rather than a rigid, non-negotiable constraint."
|
||||
- targetResolved: true
|
||||
- remainingUncertainty: null
|
||||
|
||||
**Classification: uncertainty_overresolved**
|
||||
|
||||
The model set `targetResolved=true` and produced a meaning stronger than the user supplied. "Risk matters more to me" establishes relative importance but does not determine whether avoiding additional risk is a hard constraint or a strong preference/trade-off. The model inferred "not a rigid, non-negotiable constraint" from a statement that only said risk was more important — which could be consistent with a strong preference that still functions as a de facto hard boundary in practice. This over-resolution matches the concern flagged in Experiment 54Z but isolated it here under a single fixed target to confirm the behaviour is real.
|
||||
|
||||
The model produced stronger meaning than the user supplied by declaring what risk avoidance is *not* (a rigid constraint) when the user only stated what it *is* relative to (more important than growth). This establishes that uncertainty should have been preserved with remainingUncertainty about whether avoiding additional risk is a preference/trade-off or a hard constraint.
|
||||
|
||||
---
|
||||
|
||||
### Case 3 — Conditional Trade-Off
|
||||
|
||||
**User answer:** "I'd normally avoid more risk, but for the right opportunity I might accept some."
|
||||
|
||||
**Model output:**
|
||||
- resolvedMeaning: "Avoiding additional risk is a preference or trade-off rather than a hard constraint."
|
||||
- targetResolved: true
|
||||
- remainingUncertainty: null
|
||||
|
||||
**Classification: resolution_correct with conditional qualification loss**
|
||||
|
||||
The target category was resolved correctly, but the answer's conditional qualification was lost. The user said "I might accept some for the right opportunity" — a specific conditional exception — which became "preference or trade-off rather than a hard constraint." The target (whether risk avoidance is preference/trade-off or hard constraint) was settled, but the nuance of how it functions conditionally was erased from the resolved meaning.
|
||||
|
||||
---
|
||||
|
||||
### Case 4 — Non-Answer / Insufficient Clarification
|
||||
|
||||
**User answer:** "I'm not really sure."
|
||||
|
||||
**Model output:**
|
||||
- resolvedMeaning: "The user has not yet determined if avoiding additional risk is a hard constraint or a preference/trade-off."
|
||||
- targetResolved: false
|
||||
- remainingUncertainty: "It remains unclear whether the user considers avoiding additional risk a hard constraint or a preference/trade-off."
|
||||
|
||||
**Classification: uncertainty_preserved**
|
||||
|
||||
The model correctly remained unresolved without inventing any position. The resolvedMeaning accurately describes what was supplied (nothing) and remainingUncertainty states that the distinction between hard constraint and preference/trade-off remains unresolved. This is honest behaviour.
|
||||
|
||||
---
|
||||
|
||||
### Evaluation Summary
|
||||
|
||||
| Case | Answer | Resolution Status | Classification |
|
||||
|---|---|---|---|
|
||||
| 1 | Explicit hard constraint | Correctly resolved | resolution_correct |
|
||||
| 2 | Weak priority ("Risk matters more to me.") | Over-resolved to targetResolved=true | uncertainty_overresolved |
|
||||
| 3 | Conditional trade-off | Resolved, conditionality lost | resolution_correct (with note) |
|
||||
| 4 | Non-answer ("I'm not really sure.") | Correctly unresolved | uncertainty_preserved |
|
||||
|
||||
**Classification counts:**
|
||||
- resolution_correct: 2
|
||||
- uncertainty_preserved: 1
|
||||
- uncertainty_overresolved: 1
|
||||
- resolution_failed: 0
|
||||
|
||||
---
|
||||
|
||||
### Questions Answered
|
||||
|
||||
1. **Did Case 1 correctly resolve the explicit hard constraint?** Yes — targetResolved=true, appropriate meaning strength, no spurious uncertainty.
|
||||
|
||||
2. **Did Case 2 preserve uncertainty rather than infer that risk is not a hard constraint?** No — the model over-resolved, setting targetResolved=true and inferring "not a rigid, non-negotiable constraint" from a weak priority statement. This is the key finding: one tested weak answer received stronger meaning than the user supplied.
|
||||
|
||||
3. **Did Case 3 preserve the conditional nature of the trade-off?** Partially — the model correctly resolved the target (risk avoidance is not a hard constraint) but flattened the conditional qualification ("might accept some for the right opportunity") into flat "preference or trade-off" language. The conditional layer was lost even though target resolution was correct.
|
||||
|
||||
4. **Did Case 4 correctly remain unresolved?** Yes — no invented position, appropriate remainingUncertainty stating the distinction remains unresolved.
|
||||
|
||||
5. **How many cases were:**
|
||||
- resolution_correct: 2
|
||||
- uncertainty_preserved: 1
|
||||
- uncertainty_overresolved: 1
|
||||
- resolution_failed: 0
|
||||
|
||||
6. **Did any answer get stronger meaning than the user supplied?** Yes — Case 2 ("Risk matters more to me.") received a stronger interpretation than it justified. The model inferred "not a rigid, non-negotiable constraint" from a statement that only established relative priority.
|
||||
|
||||
7. **Did any unresolved answer incorrectly return targetResolved true?** No — Case 4 (the only truly unresolved case) correctly returned false. Case 2 over-resolved but did not remain unresolved.
|
||||
|
||||
8. **Did any resolved answer incorrectly retain uncertainty?** No — both resolved cases (1 and 3) correctly returned null for remainingUncertainty.
|
||||
|
||||
9. **Does the answer-resolution step appear biased toward resolution in these tested cases?** Yes — Case 2 demonstrates that a weak priority statement can be over-resolved to a definitive classification ("not a constraint") when it should have remained unresolved. One out of four cases showed this bias, but it appeared on the weakest-answer type where uncertainty preservation matters most.
|
||||
|
||||
10. **Does this establish how answer-resolution logic should be changed?** No — one weak-priority case over-resolved; this does not justify a broad change without broader testing.
|
||||
11. **Does this establish how the graph should represent unresolved clarification?** No — the unresolved representation question is separate from whether the model *should* remain unresolved.
|
||||
12. **Does this establish how Behaviour Selection should react?** No — this experiment did not integrate with Behaviour Selection.
|
||||
|
||||
### Limitations
|
||||
|
||||
- Only one source scenario and four answer types were tested. Different sources may behave differently.
|
||||
- Only one weak-answer pattern (priority without constraint meaning) over-resolved; other patterns not assessed.
|
||||
- Only one model configuration was used (qwen-claude:latest on 192.168.1.111:11434). Different models may behave differently.
|
||||
- Case 3 showed conditionality loss that is subtler than over-resolution — it resolved correctly but flattened nuance. This pattern warrants further testing with additional conditional answers.
|
||||
- Four cases tested; broader generalisation remains untested.
|
||||
|
||||
### Experiment Conclusion
|
||||
|
||||
**Clarification-answer resolution preserved uncertainty appropriately across some cases but over-resolved one weak answer.**
|
||||
|
||||
Case 1 (explicit hard constraint) and Case 4 (non-answer) behaved honestly — the explicit case resolved, the empty case remained unresolved. This is the expected baseline.
|
||||
|
||||
Case 2 (weak priority: "Risk matters more to me.") demonstrates the core limitation: the model over-resolved a weak answer, converting relative priority into a definitive negative ("not a rigid, non-negotiable constraint") when only relative importance was supplied. The model set targetResolved=true and erased uncertainty that should have been preserved.
|
||||
|
||||
Case 3 (conditional trade-off) resolved the target correctly but lost the conditional qualification in the process — "might accept some for the right opportunity" was flattened to "preference or trade-off rather than a hard constraint." This is a subtler form of meaning inflation: correct resolution with erasure of nuance.
|
||||
|
||||
Two of the four tested answers showed loss of nuance: one was over-resolved (Case 2) and one retained the correct target category while losing conditional qualification (Case 3). The remaining two cases behaved correctly (Cases 1 and 4). With qwen-claude:latest and the current semantic instruction, uncertainty preservation works for truly empty answers but is unreliable for weak-priority answers.
|
||||
|
||||
### Focused Test Result
|
||||
|
||||
All 4 live inference calls completed successfully. One over-resolution (Case 2), one conditionality loss (Case 3), two honest classifications (Cases 1 and 4). Total: ~62s, average: ~15.5s per call, fastest: 8.2s, slowest: 19.3s.
|
||||
|
||||
### Historical Comparison Result
|
||||
|
||||
Compared to Experiment 54Z's finding that both variants over-resolved Case 1 ("Risk matters more to me."), Experiment 55A confirms this is a persistent issue under the same semantic instruction and model — not a side-effect of target framing. When a single fixed target was used, the weak priority answer still over-resolved (targetResolved=true with inferred "not a constraint" meaning). The same over-resolution reproduced with a fixed target, so target broadening is not required for the failure to occur.
|
||||
|
||||
### Documentation Updated
|
||||
|
||||
- `docs/design-evolution-log.md` — added full Experiment 55A entry; applied 54Z wording corrections
|
||||
- `docs/current-handoff.md` — updated with Experiment 55A summary and new Return-to-Work note
|
||||
|
||||
### Confirmation Host and Model Remained Unchanged
|
||||
|
||||
Host: `http://192.168.1.111:11434`. Model: `qwen-claude:latest`. Same as all prior experiments.
|
||||
|
||||
### Confirmation Semantic Instruction and Output Contract Remained Unchanged
|
||||
|
||||
Answer-resolution instruction identical to Experiment 54V. No stronger coaching, no added examples. Output contract unchanged from 54V (`{ resolvedMeaning, targetResolved, remainingUncertainty }`).
|
||||
|
||||
### Confirmation Production Prompts and Schemas Remained Unchanged
|
||||
|
||||
No production prompts read or modified. No schemas changed. All inference calls used the experiment-specific semantic instructions defined in this test file.
|
||||
|
||||
### Confirmation Behaviour Selection Remained Unchanged
|
||||
|
||||
Behaviour Selection was not called or referenced. No integration with the selector occurred.
|
||||
|
||||
### Confirmation Graph and UI Remained Unchanged
|
||||
|
||||
No graph files read or modified. No UI code touched. The experiment is test-only.
|
||||
|
||||
### Confirmation No Uncertainty-Preservation Logic Entered Active Runtime
|
||||
|
||||
This experiment created one new test file only. No uncertainty-preservation logic entered any active runtime path, production module, or behaviour selection output.
|
||||
|
||||
---
|
||||
|
||||
### Return-to-Work Note (Experiment 55A)
|
||||
|
||||
54Z showed weaker answers can produce different downstream resolution states under different framings, suggesting target broadening matters; 55A isolated the answer-resolution step using one fixed target and four answers of varying strength (explicit hard constraint, weak priority, conditional trade-off, non-answer) to test whether uncertainty preservation holds independently of framing. The explicit case resolved correctly, the non-answer remained honestly unresolved, but the weak-priority case over-resolved by setting targetResolved=true and inferring "not a constraint" from relative importance alone. Conditional language was also flattened even when resolution was correct. Uncertainty preservation remains unreliable for weak-priority answers. Graph, Behaviour Selection, UI, and production integration remain untouched. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7. First file to inspect when resuming: tests/reconstruction/semantic-clarification-uncertainty-preservation.test.js for the full experiment and results. Status pending Rob's review.
|
||||
|
||||
---
|
||||
|
||||
+219
@@ -0,0 +1,219 @@
|
||||
## Experiment 55B — Separate Answer Meaning from Resolution Judgement (2026-08-08)
|
||||
|
||||
### Objective
|
||||
|
||||
Experiment 55A showed one weak answer was over-resolved and one conditional answer lost nuance. The remaining question: is the distortion introduced when the model restates the answer's meaning, or only when it decides whether the clarification target is resolved?
|
||||
|
||||
55B separates these two steps using independent calls per case:
|
||||
- **Mode A** (meaning-only): "State what the user's answer establishes. Preserve uncertainty and qualification exactly." No resolution decision.
|
||||
- **Mode B** (resolution): The same instruction and contract as Experiment 54V/55A.
|
||||
|
||||
### Hypothesis
|
||||
|
||||
The model may preserve weak and conditional meaning correctly when asked only to restate what the user established, and only become over-confident when asked whether the clarification target is resolved. If so, the problem lies in converting meaning into a `targetResolved` judgement, not in interpreting the answer itself. If the meaning-only output already strengthens or flattens the answer, then the problem occurs earlier.
|
||||
|
||||
### Configuration
|
||||
|
||||
Host: `http://192.168.1.111:11434` (same as all prior experiments)
|
||||
Model: `qwen-claude:latest` (same as all prior experiments)
|
||||
|
||||
### Number of Live Inference Calls
|
||||
|
||||
Exactly **6** live Ollama calls — one meaning-only call and one resolution call per case, three cases. The calls are independent; Mode A output is never fed into Mode B.
|
||||
|
||||
### Fixed Source / Target / Question (identical to 55A)
|
||||
|
||||
- **Source:** "I want the business to grow, but I don't want to take on more risk."
|
||||
- **Clarification target:** whether avoiding additional risk is a preference/trade-off or a hard constraint.
|
||||
- **Clarification question:** Do you view avoiding additional risk as a hard constraint, or as a preference or trade-off?
|
||||
|
||||
### Mode A instruction
|
||||
|
||||
> State only what the user's answer establishes in relation to the clarification question. Preserve uncertainty, conditionality, and qualification exactly as supplied. Do not decide whether the clarification target is resolved. Do not infer what the user did not say. Do not recommend action or generate another question.
|
||||
>
|
||||
> Return valid JSON only in this shape: { "answerMeaning": "short statement" }
|
||||
|
||||
### Mode A output contract
|
||||
|
||||
```json
|
||||
{
|
||||
"answerMeaning": "short statement"
|
||||
}
|
||||
```
|
||||
|
||||
### Mode B instruction unchanged
|
||||
|
||||
Same as Experiment 54V / 55A resolution instruction. Output contract unchanged (`resolvedMeaning`, `targetResolved`, `remainingUncertainty`).
|
||||
|
||||
---
|
||||
|
||||
### Case 1 — Weak Priority
|
||||
|
||||
**Answer:** "Risk matters more to me."
|
||||
|
||||
**Mode A result:**
|
||||
- answerMeaning: "The user indicates that avoiding additional risk holds greater importance to them than other factors."
|
||||
- Classification: **meaning_preserved**
|
||||
|
||||
The meaning preserved relative importance without deciding whether it is a hard constraint or not. No strengthening or loss detected.
|
||||
|
||||
**Mode B result:**
|
||||
- resolvedMeaning: "Avoiding additional risk is established as a strongly prioritized preference over growth, rather than an explicitly stated boundary."
|
||||
- targetResolved: false
|
||||
- remainingUncertainty: "Whether this priority functions as a non-negotiable hard constraint or remains a flexible trade-off in certain situations."
|
||||
- Classification: **resolution_correct**
|
||||
|
||||
The resolution stayed unresolved with appropriate uncertainty. Interestingly, the resolved meaning is slightly stronger than the Mode A output ("strongly prioritized preference" vs "greater importance"), but `targetResolved` was correctly false — no over-resolution occurred this time.
|
||||
|
||||
---
|
||||
|
||||
### Case 2 — Conditional Trade-Off
|
||||
|
||||
**Answer:** "I'd normally avoid more risk, but for the right opportunity I might accept some."
|
||||
|
||||
**Mode A result:**
|
||||
- answerMeaning: "The user indicates that avoiding additional risk is generally a preference that may become a trade-off under specific conditions, rather than an absolute constraint."
|
||||
- Classification: **meaning_preserved**
|
||||
|
||||
Both the normal preference and the conditional exception ("under specific conditions") were preserved. The meaning captures all of what the user established without flattening.
|
||||
|
||||
**Mode B result:**
|
||||
- resolvedMeaning: "Avoiding additional risk is a preference or trade-off rather than a hard constraint."
|
||||
- targetResolved: true
|
||||
- remainingUncertainty: null
|
||||
- Classification: **resolution_meaning_loss**
|
||||
|
||||
The target was correctly resolved (`targetResolved=true`). However, the conditional qualification ("for the right opportunity I might accept some") was **flattened** into a flat "preference or trade-off" statement. Mode A preserved "generally a preference that may become a trade-off under specific conditions"; Mode B collapsed it to "a preference or trade-off."
|
||||
|
||||
This is a direct comparison within the same case: **meaning was preserved in Mode A but lost during resolution.**
|
||||
|
||||
---
|
||||
|
||||
### Case 3 — Non-Answer
|
||||
|
||||
**Answer:** "I'm not really sure."
|
||||
|
||||
**Mode A result:**
|
||||
- answerMeaning: "The user is uncertain whether avoiding additional risk is a hard constraint or a preference or trade-off."
|
||||
- Classification: **meaning_preserved**
|
||||
|
||||
Simple uncertainty was preserved accurately. No position was invented.
|
||||
|
||||
**Mode B result:**
|
||||
- resolvedMeaning: "The user is unsure whether avoiding additional risk is a hard constraint or a preference/trade-off."
|
||||
- targetResolved: false
|
||||
- remainingUncertainty: "Whether avoiding additional risk is a hard constraint or a preference/trade-off remains unclear."
|
||||
- Classification: **resolution_correct**
|
||||
|
||||
Correctly unresolved. No meaning distortion detected in either mode.
|
||||
|
||||
---
|
||||
|
||||
### Evaluation Summary
|
||||
|
||||
| Case | Answer | Mode A | Mode A Class | Mode B | Mode B Class |
|
||||
|------|--------|--------|-------------|--------|-------------|
|
||||
| 1 | "Risk matters more to me." | Preserved relative priority | meaning_preserved | Correctly unresolved | resolution_correct |
|
||||
| 2 | "I'd normally avoid more risk, but for the right opportunity I might accept some." | Preserved conditional qualification | meaning_preserved | Resolved but flattened conditionality | resolution_meaning_loss |
|
||||
| 3 | "I'm not really sure." | Preserved uncertainty | meaning_preserved | Correctly unresolved | resolution_correct |
|
||||
|
||||
**Meaning counts (Mode A):**
|
||||
- meaning_preserved: 3
|
||||
- meaning_strengthened: 0
|
||||
- meaning_lost: 0
|
||||
|
||||
**Resolution counts (Mode B):**
|
||||
- resolution_correct: 2
|
||||
- resolution_meaning_loss: 1
|
||||
- resolution_overresolved: 0
|
||||
- resolution_underresolved: 0
|
||||
|
||||
---
|
||||
|
||||
### Questions Answered
|
||||
|
||||
1. **Did Case 1 Mode A preserve only relative priority without deciding hard-constraint status?** Yes — "greater importance to them than other factors" preserves the relative priority without declaring anything about hard constraint status.
|
||||
|
||||
2. **Did Case 1 Mode B over-resolve the target again?** No — this run returned `targetResolved=false` with appropriate remaining uncertainty. The resolved meaning was slightly stronger ("strongly prioritized preference") but did not cross into definitive classification. (Note: this differs from the 55A run on the same case, which had over-resolved to `targetResolved=true`. This may indicate some instability in the resolution step across runs.)
|
||||
|
||||
3. **Did Case 2 Mode A preserve the conditional "for the right opportunity" qualification?** Yes — "generally a preference that may become a trade-off under specific conditions" preserves both the normal stance and the conditional exception.
|
||||
|
||||
4. **Did Case 2 Mode B preserve or flatten that same conditionality?** Flattened. Mode B collapsed "I'd normally avoid more risk, but for the right opportunity I might accept some" into "a preference or trade-off rather than a hard constraint." The conditional qualification ("for the right opportunity") was lost during resolution.
|
||||
|
||||
5. **Did Case 3 Mode A preserve simple uncertainty?** Yes — "The user is uncertain whether avoiding additional risk is a hard constraint or a preference or trade-off" preserves the lack of position without inventing one.
|
||||
|
||||
6. **Did Case 3 Mode B correctly remain unresolved?** Yes — `targetResolved=false` with accurate remainingUncertainty. No meaning distortion in either mode.
|
||||
|
||||
7. **In any case, was meaning already distorted before the resolution judgement?** No — all three cases preserved their meaning accurately in Mode A (meaning_preserved: 3). The first material information loss appeared only during the resolution step.
|
||||
|
||||
8. **In any case, did meaning remain accurate in Mode A but become stronger or flatter in Mode B?** Yes — Case 2 is the clearest example. Mode A preserved "generally a preference that may become a trade-off under specific conditions"; Mode B flattened it to "a preference or trade-off."
|
||||
|
||||
9. **Does this isolate the failure to the resolution judgement?** Partially yes, for the case of conditional meaning loss. The comparison within Case 2 (same answer, same model, independent calls) shows that meaning can be preserved in isolation and then lost when a resolution decision is introduced. However, only one instance of meaning-preserved-but-resolution-flattened was observed; broader generalisation requires more tested cases.
|
||||
|
||||
10. **Does this establish how production logic should be redesigned?** No — the evidence from three answers is insufficient to justify specific production changes. Further testing with additional answer types and different models would be needed before redesigning any logic.
|
||||
|
||||
11. **Does this establish graph representation or Behaviour Selection changes?** No — this experiment did not integrate with graph, Behaviour Selection, or any other engine component.
|
||||
|
||||
---
|
||||
|
||||
### Limitations
|
||||
|
||||
- Only three answers were tested. Different answer patterns may behave differently.
|
||||
- The weak-priority case in 55B resolved correctly (unlike 55A which over-resolved it), suggesting the resolution step may have some instability across runs with the same configuration.
|
||||
- Only one model configuration was used (qwen-claude:latest on 192.168.1.111:11434).
|
||||
- The conditional trade-off answer is a specific pattern; other conditional phrasings may behave differently.
|
||||
- Six live calls total — insufficient for broader generalisation.
|
||||
|
||||
---
|
||||
|
||||
### Experiment Conclusion
|
||||
|
||||
**Meaning-only extraction preserved all three tested answers; one conditional answer then lost qualification during the independent resolution judgement.**
|
||||
|
||||
All three tested answers preserved their meaning correctly in Mode A (meaning_preserved: 3/3). The only information loss appeared in Case 2 when transitioning from meaning-only to resolution: the conditional qualification "for the right opportunity I might accept some" was present and preserved by Mode A, then flattened to a flat "preference or trade-off" statement during resolution.
|
||||
|
||||
Additionally, Case 1 produced different resolution outcomes across experiments (55A over-resolved; 55B correctly unresolved), suggesting the resolution step exhibits some run-to-run instability under the same configuration — an observation worth monitoring but not yet actionable without more data.
|
||||
|
||||
Separating the two experimentally was useful for locating where the observed meaning loss first appeared.
|
||||
|
||||
### Focused Test Result
|
||||
|
||||
All 6 live inference calls completed successfully. Three answers tested independently through two modes each. Meaning preservation was perfect across Mode A (3/3). Resolution introduced one meaning-loss case (Case 2) and correctly handled the other two. Total: ~104s, average: ~17.3s per call, fastest: 11.5s, slowest: 24.1s.
|
||||
|
||||
### Historical Comparison Result
|
||||
|
||||
Compared to Experiment 55A's conclusion that "the answer-resolution step appears biased toward resolution," Experiment 55B shows this bias is not universal: Case 1 did not over-resolve in the 55B run, and Case 3 was correct in both experiments. The specific loss pattern (conditional meaning preserved in isolation but flattened during resolution) appeared only in Case 2. This narrows the failure from "biased toward resolution" to a more specific pattern: conditional nuance is vulnerable to flattening specifically when the model is forced to make a target-resolution decision.
|
||||
|
||||
### Documentation Updated
|
||||
|
||||
- `docs/design-evolution-log.md` — added full Experiment 55B entry; applied wording corrections to Experiment 55A
|
||||
- `docs/current-handoff.md` — updated with Experiment 55B summary and new Return-to-Work note
|
||||
|
||||
### Confirmation Host and Model Remained Unchanged
|
||||
|
||||
Host: `http://192.168.1.111:11434`. Model: `qwen-claude:latest`. Same as all prior experiments.
|
||||
|
||||
### Confirmation Semantic Instruction and Output Contract Remained Unchanged (for Mode B)
|
||||
|
||||
Mode B instruction and output contract identical to Experiment 54V / 55A. No production code changed.
|
||||
|
||||
Mode A used a new minimal instruction and output contract specific to this experiment only. It does not replace any existing mechanism.
|
||||
|
||||
### Confirmation Production Prompts and Schemas Remained Unchanged
|
||||
|
||||
No production prompts read or modified. No schemas changed. All inference calls used the experiment-specific semantic instructions defined in this test file.
|
||||
|
||||
### Confirmation Behaviour Selection Remained Unchanged
|
||||
|
||||
Behaviour Selection was not called or referenced. No integration with the selector occurred.
|
||||
|
||||
### Confirmation Graph and UI Remained Unchanged
|
||||
|
||||
No graph files read or modified. No UI code touched. The experiment is test-only.
|
||||
|
||||
### Confirmation No Meaning-Resolution Separation Entered Active Runtime
|
||||
|
||||
This experiment created one new test file only. No meaning-resolution separation logic entered any active runtime path, production module, or behaviour selection output. Production continues using the pre-existing combined instruction and contract.
|
||||
|
||||
---
|
||||
|
||||
+234
@@ -0,0 +1,234 @@
|
||||
## Experiment 55C — Resolution From Preserved Answer Meaning (2026-08-08)
|
||||
|
||||
### Objective
|
||||
|
||||
Experiment 55B showed meaning-only extraction preserved all three tested answers while one conditional answer lost qualification during independent resolution. The remaining question: if the preserved answer meaning is explicitly carried into the resolution step, does the later judgement still flatten it?
|
||||
|
||||
This tests whether carrying semantic state forward across two calls eliminates the conditionality loss, introduces a new failure mode, or changes neither.
|
||||
|
||||
### Hypothesis
|
||||
|
||||
A two-step test-only chain may preserve meaning better because the resolution stage no longer needs to reinterpret the raw answer. Possible outcomes: preserved meaning survives resolution; the resolution stage still flattens it; some answers improve while others do not. Any outcome is useful.
|
||||
|
||||
### Configuration
|
||||
|
||||
Host: `http://192.168.1.111:11434` (same as all prior experiments)
|
||||
Model: `qwen-claude:latest` (same as all prior experiments)
|
||||
|
||||
### Number of Live Inference Calls
|
||||
|
||||
Exactly **6** live Ollama calls — one meaning call and one resolution call per case, three cases. Stage 2 uses the actual answerMeaning from Stage 1, not a human reference.
|
||||
|
||||
### Fixed Source / Target / Question (identical to 55A/55B)
|
||||
|
||||
- **Source:** "I want the business to grow, but I don't want to take on more risk."
|
||||
- **Clarification target:** whether avoiding additional risk is a preference/trade-off or a hard constraint.
|
||||
- **Clarification question:** Do you view avoiding additional risk as a hard constraint, or as a preference or trade-off?
|
||||
|
||||
### Stage 1 instruction (unchanged from 55B Mode A)
|
||||
|
||||
> State only what the user's answer establishes in relation to the clarification question. Preserve uncertainty, conditionality, and qualification exactly as supplied. Do not decide whether the clarification target is resolved. Do not infer what the user did not say. Do not recommend action or generate another question.
|
||||
>
|
||||
> Return valid JSON only in this shape: { "answerMeaning": "short statement" }
|
||||
|
||||
### Stage 1 output contract
|
||||
|
||||
```json
|
||||
{
|
||||
"answerMeaning": "short statement"
|
||||
}
|
||||
```
|
||||
|
||||
### Stage 2 instruction (new — uses preserved meaning, not raw answer)
|
||||
|
||||
> Decide whether the supplied preserved answer meaning settles the clarification target. Treat the preserved meaning as the full extent of what has been established — do not strengthen, simplify, or reinterpret it. Mark targetResolved true only when that preserved meaning settles the target. If any part of the target remains unresolved, preserve that uncertainty. Keep qualifications and conditions intact.
|
||||
>
|
||||
> Return valid JSON only in this shape: { "resolvedMeaning": "short statement", "targetResolved": true, "remainingUncertainty": null }
|
||||
|
||||
### Stage 2 output contract
|
||||
|
||||
```json
|
||||
{
|
||||
"resolvedMeaning": "short statement",
|
||||
"targetResolved": true,
|
||||
"remainingUncertainty": null
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Case 1 — Weak Priority
|
||||
|
||||
**Raw answer:** "Risk matters more to me."
|
||||
|
||||
**Stage 1 result:**
|
||||
- answerMeaning: "The user indicates that risk carries greater weight, suggesting it leans toward being a significant preference or trade-off rather than explicitly confirming an absolute hard constraint."
|
||||
- Classification: **meaning_strengthened**
|
||||
|
||||
Stage 1 went beyond relative priority by adding language about "preference or trade-off" and "absolute hard constraint" — content the user did not supply. This was also seen in 55A's Stage 2, but now it appears at Stage 1. The weak-priority case has shown run-to-run variation across Experiments 55A–55C.
|
||||
|
||||
**Stage 2 result (input = actual Stage 1 meaning):**
|
||||
- resolvedMeaning: "Avoiding additional risk leans toward being a significant preference or trade-off rather than an absolute hard constraint."
|
||||
- targetResolved: true
|
||||
- remainingUncertainty: null
|
||||
- Classification: **resolution_overresolved**
|
||||
|
||||
Because Stage 1 already strengthened the answer, Stage 2 treated that strengthened meaning as input and over-resolved with `targetResolved=true` and no remaining uncertainty. If Stage 1 distorts the answer, Stage 2 may preserve and act on that distortion rather than correct it.
|
||||
|
||||
---
|
||||
|
||||
### Case 2 — Conditional Trade-Off
|
||||
|
||||
**Raw answer:** "I'd normally avoid more risk, but for the right opportunity I might accept some."
|
||||
|
||||
**Stage 1 result:**
|
||||
- answerMeaning: "Avoiding additional risk is treated as a normal preference that may be conditionally accepted for the right opportunity."
|
||||
- Classification: **meaning_preserved**
|
||||
|
||||
Both the normal preference and conditional exception were preserved. The wording uses "treated as" rather than directly repeating the user's phrasing but captures the full semantic content.
|
||||
|
||||
**Stage 2 result (input = actual Stage 1 meaning):**
|
||||
- resolvedMeaning: "Avoiding additional risk is a preference rather than a hard constraint, as it may be conditionally accepted for the right opportunity."
|
||||
- targetResolved: true
|
||||
- remainingUncertainty: null
|
||||
- Classification: **resolution_correct**
|
||||
|
||||
Target was correctly resolved with the conditional qualification ("as it may be conditionally accepted for the right opportunity") retained in the resolved meaning. This is the key result: the two-stage chain preserved the conditionality that 55B's Mode B flattened.
|
||||
|
||||
---
|
||||
|
||||
### Case 3 — Non-Answer
|
||||
|
||||
**Raw answer:** "I'm not really sure."
|
||||
|
||||
**Stage 1 result:**
|
||||
- answerMeaning: "The user states they are not really sure whether avoiding additional risk is a hard constraint or a preference/trade-off."
|
||||
- Classification: **meaning_preserved**
|
||||
|
||||
Uncertainty preserved accurately. The model added the clarification target context ("whether...is a hard constraint or a preference/trade-off") which is reasonable contextual framing for the non-answer.
|
||||
|
||||
**Stage 2 result (input = actual Stage 1 meaning):**
|
||||
- resolvedMeaning: "The user is uncertain whether avoiding additional risk is a preference/trade-off or a hard constraint."
|
||||
- targetResolved: false
|
||||
- remainingUncertainty: "Whether avoiding additional risk is a preference/trade-off or a hard constraint remains unresolved."
|
||||
- Classification: **resolution_correct**
|
||||
|
||||
Correctly unresolved with appropriate remaining uncertainty. No meaning distortion in either stage.
|
||||
|
||||
---
|
||||
|
||||
### Evaluation Summary
|
||||
|
||||
| Case | Answer | Stage 1 | Stage 1 Class | Stage 2 Result | Stage 2 Class |
|
||||
|------|--------|---------|--------------|----------------|---------------|
|
||||
| 1 | "Risk matters more to me." | Strengthened beyond priority | meaning_strengthened | Over-resolved (targetResolved=true) | resolution_overresolved |
|
||||
| 2 | "I'd normally avoid more risk, but for the right opportunity I might accept some." | Preserved conditionality | meaning_preserved | Resolved with qualification retained | resolution_correct |
|
||||
| 3 | "I'm not really sure." | Preserved uncertainty | meaning_preserved | Correctly unresolved | resolution_correct |
|
||||
|
||||
**Meaning counts (Stage 1):**
|
||||
- meaning_preserved: 2
|
||||
- meaning_strengthened: 1
|
||||
- meaning_lost: 0
|
||||
|
||||
**Resolution counts (Stage 2):**
|
||||
- resolution_correct: 2
|
||||
- resolution_overresolved: 1
|
||||
- resolution_meaning_loss: 0
|
||||
- resolution_underresolved: 0
|
||||
|
||||
---
|
||||
|
||||
### Questions Answered
|
||||
|
||||
1. **Did Case 1 Stage 1 preserve only relative priority?** No — it strengthened beyond relative priority by introducing "preference or trade-off" and "absolute hard constraint" language not present in the user's answer.
|
||||
|
||||
2. **Did Case 1 Stage 2 remain unresolved?** No — targetResolved=true because the distorted Stage 1 input led the model to conclude resolution was achieved.
|
||||
|
||||
3. **Did Case 2 Stage 1 preserve the conditional qualification?** Yes — "normal preference that may be conditionally accepted for the right opportunity" preserved both the default stance and the exception.
|
||||
|
||||
4. **Did Case 2 Stage 2 retain that qualification while resolving the target?** Yes — resolved meaning explicitly retained "as it may be conditionally accepted for the right opportunity." This is a direct improvement over 55B's Mode B (resolution_meaning_loss → resolution_correct).
|
||||
|
||||
5. **Did Case 3 preserve uncertainty through both stages?** Yes — Stage 1 preserved uncertainty, Stage 2 correctly returned targetResolved=false with remainingUncertainty.
|
||||
|
||||
6. **Did any Stage 2 output become stronger than its actual Stage 1 input?** No — manual semantic review found no case where Stage 2 strengthened beyond the actual Stage 1 meaning. The chaining check confirmed this explicitly for all three cases.
|
||||
|
||||
7. **Did any Stage 2 output flatten a condition present in Stage 1?** No — Case 2's condition survived both stages intact. This is the key positive finding.
|
||||
|
||||
8. **Compared with 55B, did carrying preserved meaning forward remove the observed conditionality loss?** Yes — in 55B Mode B, Case 2 was resolution_meaning_loss (flattened). In 55C Stage 2, Case 2 was resolution_correct with qualification retained. The two-stage chain eliminated this specific failure mode for the tested answer.
|
||||
|
||||
9. **Compared with 55A/55B, did the weak-priority case remain honestly unresolved?** No — in 55B, Case 1 was resolution_correct (unresolved) in that run; in 55C, it over-resolved because Stage 1 distorted the meaning first. The weak-priority problem is not solved by this approach.
|
||||
|
||||
10. **Does this prove a two-stage production design is required?** No — evidence from three answers is insufficient to justify specific production changes.
|
||||
|
||||
11. **Does this establish graph representation?** No — this experiment did not integrate with graph, Behaviour Selection, or any other engine component.
|
||||
|
||||
12. **Does this establish Behaviour Selection changes?** No — Behaviour Selection was not called or referenced.
|
||||
|
||||
---
|
||||
|
||||
### Limitations
|
||||
|
||||
- Only three answers were tested. Different answer patterns may behave differently.
|
||||
- Case 1 revealed a new failure mode: if Stage 1 distorts meaning, Stage 2 amplifies it through chaining. This is not an improvement over 55B's approach for weak answers.
|
||||
- The conditional trade-off improvement (Case 2) may not generalise to other conditional patterns.
|
||||
- Only one model configuration was used (qwen-claude:latest on 192.168.1.111:11434).
|
||||
- Six live calls total — insufficient for broader generalisation.
|
||||
- Case 3 Stage 1 added contextual framing ("whether...is a hard constraint or a preference/trade-off") to the non-answer, which could be questioned as mild interpretation even though it preserved uncertainty correctly.
|
||||
|
||||
---
|
||||
|
||||
### Experiment Conclusion
|
||||
|
||||
**Preserved meaning improved resolution but some loss remained.**
|
||||
|
||||
The two-stage chain successfully eliminated the conditionality loss observed in 55B: Case 2's conditional qualification survived through both stages and resolved correctly (resolution_correct). Non-answer uncertainty was also preserved through both stages (resolution_correct). These are genuine improvements.
|
||||
|
||||
However, the weak-priority case revealed a new failure mode: Stage 1 strengthened "Risk matters more to me." into language about "preference or trade-off rather than absolute hard constraint," and Stage 2 then over-resolved based on that distorted input. Carrying semantic state forward means distortion propagates as well as fidelity. This does not improve the weak-priority problem relative to 55B's Mode B (which correctly left Case 1 unresolved in its run).
|
||||
|
||||
The core finding is asymmetric: preserving meaning before resolution helps for conditional answers (eliminates flattening) and non-answers (preserves uncertainty), but does not help — and may worsen outcomes — when the meaning extraction step itself distorts. The question "does the judgement stop rewriting the meaning?" is answered partially: it stops rewriting when the input to judgement already carries the full meaning, but it amplifies rewriting when that input is itself distorted.
|
||||
|
||||
Does this prove a two-stage production design is required? **No.** The weak-priority case over-resolved in 55C while remaining unresolved in the 55B run — and neither result establishes which approach is better for all cases.
|
||||
|
||||
### Focused Test Result
|
||||
|
||||
All 6 live inference calls completed successfully. Three answers tested through chained meaning→resolution stages. Stage 2 eliminated the conditionality loss from 55B (Case 2: resolution_meaning_loss → resolution_correct) but did not eliminate over-resolution for weak-priority input when Stage 1 strengthened it first (Case 1: meaning_strengthened → resolution_overresolved). Non-answer uncertainty was preserved through both stages. Total: ~117s, average: ~19.5s per call, fastest: 15.0s, slowest: 23.8s.
|
||||
|
||||
### Historical Comparison Result
|
||||
|
||||
Compared to Experiment 55B's finding that "meaning was preserved in Mode A but lost during resolution," Experiment 55C shows that carrying preserved meaning forward eliminates that specific loss pattern for conditional answers (Case 2 resolved correctly with qualification retained) but introduces a different asymmetry: Stage 1 distortion propagates through Stage 2. The weak-priority case improved relative to 55A's over-resolution but degraded relative to the 55B run's correct unresolved result. Neither two-stage approach consistently outperforms the other across all tested answer types.
|
||||
|
||||
### Documentation Updated
|
||||
|
||||
- `docs/design-evolution-log.md` — added full Experiment 55C entry; applied corrected wording to Experiment 55B
|
||||
- `docs/current-handoff.md` — updated with Experiment 55C summary, corrected 55B wording, and new Return-to-Work note
|
||||
|
||||
### Confirmation Host and Model Remained Unchanged
|
||||
|
||||
Host: `http://192.168.1.111:11434`. Model: `qwen-claude:latest`. Same as all prior experiments.
|
||||
|
||||
### Confirmation Semantic Instruction and Output Contract Remained Unchanged (for Stage 1)
|
||||
|
||||
Stage 1 instruction identical to Experiment 55B Mode A. No production code changed.
|
||||
|
||||
### Confirmation Production Prompts and Schemas Remained Unchanged
|
||||
|
||||
No production prompts read or modified. No schemas changed. All inference calls used the experiment-specific semantic instructions defined in this test file.
|
||||
|
||||
### Confirmation Behaviour Selection Remained Unchanged
|
||||
|
||||
Behaviour Selection was not called or referenced. No integration with the selector occurred.
|
||||
|
||||
### Confirmation Graph and UI Remained Unchanged
|
||||
|
||||
No graph files read or modified. No UI code touched. The experiment is test-only.
|
||||
|
||||
### Confirmation No Preserved-Meaning Resolution Logic Entered Active Runtime
|
||||
|
||||
This experiment created one new test file only. No preserved-meaning resolution logic entered any active runtime path, production module, or behaviour selection output. Production continues using the pre-existing combined instruction and contract.
|
||||
|
||||
---
|
||||
|
||||
### Return-to-Work Note (Experiment 55C)
|
||||
|
||||
55A showed one weak answer was over-resolved and one conditional answer lost nuance; 55B separated answer meaning from target-resolution judgement using independent calls. All three tested meanings were preserved in Mode A — the weak priority ("risk matters more"), the conditional trade-off ("for the right opportunity I might accept some"), and the non-answer uncertainty. The first material information loss appeared only when deciding target resolution: Case 2's conditional qualification was preserved by the meaning-only call but flattened during resolution. This suggests the distortion occurs in the resolution judgement step rather than the meaning extraction step, though the pattern was observed for only one case. Whether other answer types show the same pattern remains unproven. Graph, Behaviour Selection, UI and production remain untouched. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7, commit 3af623a. First test/file to inspect when resuming: tests/reconstruction/semantic-preserved-meaning-resolution.test.js for the full experiment and results. Status pending Rob's review.
|
||||
|
||||
+340
@@ -0,0 +1,340 @@
|
||||
## Experiment 55D — Separate Stated Clarification Meaning from Inference (2026-08-08)
|
||||
|
||||
### Objective
|
||||
|
||||
Experiment 55C showed that when Stage 1 preserved the user's meaning accurately, carrying that meaning forward protected conditionality during resolution. But for "Risk matters more to me.", Stage 1 itself added meaning about preference/trade-off rather than hard constraint — content not supplied by the user. The unresolved question is now one step earlier: can the first interpretation step distinguish what the user actually established from what merely seems plausible?
|
||||
|
||||
This experiment tests interpretation only. No target resolution, no question generation, no production changes.
|
||||
|
||||
### Hypothesis
|
||||
|
||||
The model may interpret weak answers more faithfully if it keeps meaning directly established by the answer and possible implications not directly established in separate fields. If the weak-priority answer remains cleanly stated while the "not a hard constraint" idea moves into a separate inference field, that would show the information can be represented without silently strengthening the user's meaning. If strengthening still appears inside stated meaning, the problem occurs even with explicit separation.
|
||||
|
||||
### Configuration
|
||||
|
||||
Host: `http://192.168.1.111:11434` (same as all prior experiments)
|
||||
Model: `qwen-claude:latest` (same as all prior experiments)
|
||||
|
||||
### Number of Live Inference Calls
|
||||
|
||||
Exactly **4** live Ollama calls — one per answer. Single call per case (no two-stage chain).
|
||||
|
||||
### Fixed Clarification Question
|
||||
|
||||
> Do you view avoiding additional risk as a hard constraint, or as a preference or trade-off?
|
||||
|
||||
Clarification target context: whether avoiding additional risk is a preference/trade-off or a hard constraint.
|
||||
|
||||
### Output Contract
|
||||
|
||||
```json
|
||||
{
|
||||
"statedMeaning": "short statement",
|
||||
"possibleInference": "short statement or null"
|
||||
}
|
||||
```
|
||||
|
||||
### Semantic Instruction Used
|
||||
|
||||
> State only what the user's answer directly establishes in statedMeaning. Preserve uncertainty, qualification, and conditionality. Do not turn relative importance into a hard boundary or the absence of one unless the user actually says so. If there is a plausible implication that goes beyond what the answer directly establishes, place it only in possibleInference. Do not decide whether the clarification target is resolved. Do not recommend action or generate another question.
|
||||
|
||||
---
|
||||
|
||||
### Case 1 — Weak Priority
|
||||
|
||||
**Raw answer:** "Risk matters more to me."
|
||||
|
||||
**statedMeaning:** "The user considers risk to be of greater importance than other factors."
|
||||
**possibleInference:** "This may suggest that avoiding additional risk is weighted as a stricter constraint rather than a flexible preference, though this remains unconfirmed."
|
||||
**Classification:** stated_meaning_preserved / inference_cleanly_separated
|
||||
|
||||
statedMeaning establishes only relative importance — no hard constraint or not-a-hard-constraint boundary. possibleInference contains the stronger implication about stricter constraint, properly qualified with "may suggest" and "remains unconfirmed". The user's relative priority language was not strengthened in statedMeaning. The model did generate a plausible inference (stricter constraint weighting) but kept it separate from what the answer directly establishes.
|
||||
|
||||
---
|
||||
|
||||
### Case 2 — Conditional Trade-Off
|
||||
|
||||
**Raw answer:** "I'd normally avoid more risk, but for the right opportunity I might accept some."
|
||||
|
||||
**statedMeaning:** "Avoiding additional risk is generally preferred but can be accepted for the right opportunity, qualifying it as a preference or trade-off rather than a hard constraint."
|
||||
**possibleInference:** "The user may weigh potential rewards against new risks to decide when increased exposure is justified."
|
||||
**Classification:** stated_meaning_preserved (semantic review) / inference_cleanly_separated
|
||||
|
||||
statedMeaning preserved both the normal preference (generally preferred to avoid risk) and the conditional exception (can be accepted for the right opportunity). It also directly established that avoiding all additional risk is not absolute — because the user explicitly said they might accept some. The additional clause "qualifying it as a preference or trade-off rather than a hard constraint" goes slightly beyond what was explicitly stated, but the core meaning (both sides of the condition) remains intact. possibleInference adds a reasonable speculative implication about weighing rewards against risks, cleanly separated.
|
||||
|
||||
---
|
||||
|
||||
### Case 3 — Explicit Hard Constraint
|
||||
|
||||
**Raw answer:** "It's a hard constraint. I don't want any increase in risk."
|
||||
|
||||
**statedMeaning:** "Avoiding additional risk is a hard constraint, with no increase in risk acceptable."
|
||||
**possibleInference:** "Proposed options will strictly exclude any that add risk, potentially limiting flexibility on other objectives."
|
||||
**Classification:** stated_meaning_preserved / unnecessary_inference
|
||||
|
||||
statedMeaning directly and correctly establishes the hard constraint. possibleInference adds a consequential implication about proposed options excluding risk — a reasonable inference but unnecessary for an explicit answer where none was lacking. This is the first case where possibleInference goes beyond what was warranted; it should normally be null here.
|
||||
|
||||
---
|
||||
|
||||
### Case 4 — Non-Answer
|
||||
|
||||
**Raw answer:** "I'm not really sure."
|
||||
|
||||
**statedMeaning:** "The user expresses uncertainty about whether avoiding additional risk is a hard constraint or a preference/trade-off."
|
||||
**possibleInference:** "The user may require further details on the potential risks or alternatives to clarify their position."
|
||||
**Classification:** stated_meaning_preserved / unnecessary_inference
|
||||
|
||||
statedMeaning preserves uncertainty without inventing a preference, constraint, or likely leaning. However, it adds contextual framing ("about whether avoiding additional risk is a hard constraint or a preference/trade-off") that references the clarification target — this is acceptable contextual framing for a non-answer, not invented content. possibleInference speculates about what the user may need, which is unnecessary for a non-answer where no leaning should be inferred.
|
||||
|
||||
---
|
||||
|
||||
### Evaluation Summary
|
||||
|
||||
| Case | Answer | Stated Meaning Classification | Inference Separation |
|
||||
|------|--------|------------------------------|---------------------|
|
||||
| 1 | "Risk matters more to me." | stated_meaning_preserved | inference_cleanly_separated |
|
||||
| 2 | "I'd normally avoid more risk, but for the right opportunity I might accept some." | stated_meaning_preserved | inference_cleanly_separated |
|
||||
| 3 | "It's a hard constraint. I don't want any increase in risk." | stated_meaning_preserved | unnecessary_inference |
|
||||
| 4 | "I'm not really sure." | stated_meaning_preserved | unnecessary_inference |
|
||||
|
||||
**Stated meaning counts:**
|
||||
- stated_meaning_preserved: 4
|
||||
- stated_meaning_strengthened: 0
|
||||
- stated_meaning_lost: 0
|
||||
|
||||
**Inference separation counts:**
|
||||
- inference_cleanly_separated: 2
|
||||
- unnecessary_inference: 2
|
||||
- inference_leaked_into_stated: 0
|
||||
- no_inference_needed: 0
|
||||
|
||||
---
|
||||
|
||||
### Questions Answered
|
||||
|
||||
1. **Did Case 1 keep "risk matters more" as relative importance only?** Yes — statedMeaning states "greater importance than other factors" without deciding whether risk avoidance is a hard constraint or not.
|
||||
|
||||
2. **Did Case 1 place any stronger preference/constraint implication only in possibleInference?** Yes — the model placed "weighted as a stricter constraint rather than a flexible preference" in possibleInference, qualified with "may suggest" and "remains unconfirmed."
|
||||
|
||||
3. **Did Case 2 preserve the "for the right opportunity" condition?** Yes — statedMeaning preserved both "generally preferred" and "can be accepted for the right opportunity." It also added a qualifier about preference/trade-off rather than hard constraint (slight overreach but not meaningful loss).
|
||||
|
||||
4. **Did Case 3 preserve the explicit hard constraint without unnecessary inference in statedMeaning?** Yes — statedMeaning correctly establishes the hard constraint. possibleInference was unnecessary (should have been null) but statedMeaning is clean.
|
||||
|
||||
5. **Did Case 4 preserve uncertainty without inventing a leaning?** Yes — statedMeaning preserves uncertainty. It added contextual framing referencing the clarification target, which is acceptable for non-answer context. No preference or constraint was invented. possibleInference was unnecessary but did not invent a specific leaning (it asked what the user might need, not what they likely prefer).
|
||||
|
||||
6. **Did any unsupported meaning leak into statedMeaning?** No — none of the four cases leaked stronger-than-justified meaning into statedMeaning. Case 2 added a qualifier ("qualifying it as a preference or trade-off rather than a hard constraint") that was not explicitly in the user's answer, but this is contextual framing rather than unsupported strengthening. The core conditional meaning (both sides) was preserved.
|
||||
|
||||
7. **Did the model generate unnecessary implications where the answer was already explicit?** Yes — Case 3 and Case 4 both received possibleInference content when none was warranted. This suggests the model tends to always provide an inference even when the answer is complete or absent. Not a statedMeaning defect, but a possibleInference hygiene issue.
|
||||
|
||||
8. **How many cases were stated_meaning_preserved / strengthened / lost?** preserved: 4, strengthened: 0, lost: 0.
|
||||
|
||||
9. **Compared with 55C Case 1, did explicit stated-vs-inferred separation avoid the earlier strengthening?** Yes — in 55C Stage 1, "Risk matters more to me." was strengthened into language about "preference/trade-off rather than absolute hard constraint" inside the single meaning field. In 55D, the relative importance remained clean in statedMeaning and any stronger interpretation was placed separately in possibleInference. This shows the two-field separation can prevent silent strengthening when it matters most (weak answers).
|
||||
|
||||
10. **Does this prove that production should use this exact two-field contract?** No — four cases through one call each is insufficient to justify specific production changes. The mechanism works in these tests but broader validation is needed.
|
||||
|
||||
11. **Does this establish how resolution should consume these fields?** No — resolution was not tested here. How a downstream step should combine statedMeaning and possibleInference remains an open question.
|
||||
|
||||
12. **Does this establish graph or Behaviour Selection changes?** No — no graph, Behaviour Selection, or engine integration was attempted.
|
||||
|
||||
---
|
||||
|
||||
### Limitations
|
||||
|
||||
- Only four answers were tested across one domain (risk vs. growth). Different answer patterns may behave differently.
|
||||
- Each case was called exactly once — stability across repeated identical calls was not tested.
|
||||
- possibleInference hygiene is imperfect: Cases 3 and 4 received unnecessary inferences, suggesting the model struggles to return null when no inference is warranted.
|
||||
- Only one model configuration was used (qwen-claude:latest on 192.168.1.111:11434).
|
||||
- Case 2's statedMeaning contained slight overreach ("qualifying it as a preference or trade-off rather than a hard constraint") — while the core meaning was preserved, not all answers will be this clean even with separation.
|
||||
- No downstream consumer (resolution, graph update) was tested — only whether the two fields can coexist without leakage.
|
||||
|
||||
---
|
||||
|
||||
### Experiment Conclusion
|
||||
|
||||
**Stated meaning remained separate from model inference across all tested answers.**
|
||||
|
||||
Across four fixed cases spanning weak priority, conditional trade-off, explicit constraint, and non-answer, statedMeaning never contained stronger-than-justified meaning. Case 1's weak-priority answer (the primary failure mode of 55C) stayed as relative importance only in statedMeaning — a direct improvement over 55C where the same answer was strengthened into constraint language. Case 2 preserved both sides of the conditional; Case 3 preserved explicit meaning cleanly; Case 4 preserved uncertainty without inventing position.
|
||||
|
||||
The separation mechanism works: the model can keep what the user established from what it might imply, at least in single-call mode. The remaining issue is possibleInference hygiene — the model tends to generate implications even when none are warranted (Cases 3 and 4). This does not corrupt statedMeaning but suggests the null-enforcement direction should be tuned.
|
||||
|
||||
Does this prove a production two-field contract is required? **No.** Evidence from four single calls across one answer pattern is insufficient. Does this establish how resolution should consume these fields? **No.** Resolution was not tested. Does this establish graph or Behaviour Selection changes? **No.**
|
||||
|
||||
### Focused Test Result
|
||||
|
||||
All 4 live inference calls completed successfully. Four answers tested through one call each with stated-vs-inferred separation. All four cases preserved statedMeaning without strengthening (stated_meaning_preserved: 4/4, strengthened: 0, lost: 0). Case 1's weak-priority answer stayed as relative importance only — direct improvement over 55C where the same answer was strengthened to constraint language in Stage 1. Inference cleanly separated for Cases 1 and 2; unnecessary inferences generated for Cases 3 and 4 (hygiene issue, not leakage). Total: 76730ms (~76.7s), average: ~19182.5ms per call, fastest: 16766ms, slowest: 24551ms.
|
||||
|
||||
### Historical Comparison Result
|
||||
|
||||
Compared to Experiment 55C's finding that Stage 1 strengthened "Risk matters more to me." into language about "preference/trade-off rather than absolute hard constraint," Experiment 55D shows the two-field separation avoided the specific weak-priority strengthening defect in this tested run: weak-priority answers stayed as relative importance in statedMeaning while stronger interpretations were placed separately in possibleInference. The mechanism handled the specific failure mode successfully in this probe, but broader stability and downstream consumption remain untested.
|
||||
|
||||
### Documentation Updated
|
||||
|
||||
- `docs/design-evolution-log.md` — added full Experiment 55D entry; applied corrections to Experiment 55C wording and commit hash
|
||||
- `docs/current-handoff.md` — updated with Experiment 55D summary and new Return-to-Work note
|
||||
|
||||
### Confirmation Host and Model Remained Unchanged
|
||||
|
||||
Host: `http://192.168.1.111:11434`. Model: `qwen-claude:latest`. Same as all prior experiments.
|
||||
|
||||
### Confirmation Production Prompts and Schemas Remained Unchanged
|
||||
|
||||
No production prompts read or modified. No schemas changed. All inference calls used the experiment-specific semantic instructions defined in this test file.
|
||||
|
||||
### Confirmation Behaviour Selection Remained Unchanged
|
||||
|
||||
Behaviour Selection was not called or referenced. No integration with the selector occurred.
|
||||
|
||||
### Confirmation Graph and UI Remained Unchanged
|
||||
|
||||
No graph files read or modified. No UI code touched. The experiment is test-only.
|
||||
|
||||
### Confirmation No Stated-vs-Inferred Clarification Logic Entered Active Runtime
|
||||
|
||||
This experiment created one new test file only. No stated-vs-inferred clarification logic entered any active runtime path, production module, or behaviour selection output. Production continues using the pre-existing contract.
|
||||
|
||||
---
|
||||
|
||||
### Return-to-Work Note (Experiment 55D)
|
||||
|
||||
55C showed preserved meaning can protect later resolution, but weak-priority meaning was already strengthened in Stage 1. 55D isolated that first interpretation step using a single-call stated-vs-inferred separation with four fixed answers across risk preference cases. Weak priority stayed as relative importance only (direct improvement over 55C's constraint-language strengthening). Conditionality survived through the conditional trade-off case. Explicit and uncertain controls stayed clean — no unsupported meaning leaked into statedMeaning. Stronger implications were kept separate in possibleInference for Cases 1 and 2, though Cases 3 and 4 showed unnecessary inference generation (hygiene issue, not leakage). This does not yet prescribe production architecture. Graph, Behaviour Selection, UI and production remain untouched. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7. First test/file to inspect when resuming: tests/reconstruction/semantic-clarification-stated-vs-inferred.test.js for the full experiment and results. Status pending Rob's review.
|
||||
|
||||
## Experiment 55E — Reasoning Refinement Requirements Synthesis (2026-08-08)
|
||||
|
||||
### Objective
|
||||
|
||||
Consolidate findings from Experiments 53–55D into a compact, implementation-ready reasoning contract for the next production pass. Stop opening new semantic sub-problems for this round.
|
||||
|
||||
### Context Documents Reviewed
|
||||
|
||||
- `docs/current-handoff.md` (current state and handoff);
|
||||
- Experiments 53, 54K–54Z, 55A–55D in `docs/design-evolution-log.md`;
|
||||
- Created: `docs/reasoning-refinement-requirements.md`.
|
||||
|
||||
### Synthesis Output
|
||||
|
||||
- **8 reasoning requirements** retained (R1–R8), all supported by at least one recorded experiment observation;
|
||||
- **7 known failure patterns** recorded from the experiment history;
|
||||
- **8 known good behaviours** recorded as regression expectations;
|
||||
- **6 regression cases** selected (A–F) covering: weak priority, conditional trade-off, non-answer/uncertainty, explicit hard constraint, evidence-resolvable disagreement, and user-owned ambiguity;
|
||||
- **10 open questions** explicitly retained to prevent premature architecture.
|
||||
|
||||
### Key Unresolved Items
|
||||
|
||||
- Stability across larger case sets and other models;
|
||||
- Exact production representation (graph integration pending);
|
||||
- Downstream consumption of inference fields;
|
||||
- Behaviour Selection and UI integration timing;
|
||||
- Performance/latency implications;
|
||||
- One call versus multiple calls for semantic separation.
|
||||
|
||||
### Conclusion
|
||||
|
||||
This round of semantic experimentation is closed. The requirements synthesis in `docs/reasoning-refinement-requirements.md` provides a bounded starting point for tomorrow's implementation pass. No live inference was performed. No production code, prompts, schemas, graph files, Behaviour Selection rules, or UI code were modified. The mechanism from Experiment 55D avoided the specific weak-priority strengthening defect in this tested run; broader stability remains unproven.
|
||||
|
||||
## Experiment 55F — Reasoning Requirements Production Path Map (2026-08-08)
|
||||
|
||||
### Objective
|
||||
|
||||
Map how reasoning requirements R1–R8 are actually supported (or unsupported) by the existing production code path, using source-inspection only. Trace the answer-to-reasoning flow through prompt building, LLM response parsing and normalization, and graph mutation. Identify which gaps have structural carriers in current schemas and which require new schema fields or logic at specific line locations. This exercise is explicitly NOT architecture design or implementation — it documents what exists today so tomorrow's Codex pass starts from accurate information.
|
||||
|
||||
### Context Documents Reviewed
|
||||
|
||||
- `docs/reasoning-refinement-requirements.md` (R1–R8 requirements, regression pack A–F);
|
||||
- `docs/current-handoff.md` (handoff state after 55E);
|
||||
- `lib/graph/orchestrator.js` — updateCase code path and LLM/provider integration;
|
||||
- `lib/graph/schema.js` — situationNodeSchema, graphUpdateSchema, updateCaseRequestSchema;
|
||||
- `lib/graph/update-proposal.js` — parseGraphUpdateProposal with normalization;
|
||||
- `lib/graph/prompt-builder.js` — buildGraphUpdatePrompt with answer embedding;
|
||||
- `lib/graph/apply-proposal.js` — applyValidatedProposal and deriveReasoningStateOverride;
|
||||
- `lib/graph/builder.js` — initial graph construction (not used in update cycles).
|
||||
|
||||
### Findings
|
||||
|
||||
**Production update path:** user answer → buildGraphUpdatePrompt → LLM provider → parseGraphUpdateProposal → applyValidatedProposal. The full chain was traced with line-number precision for each transition.
|
||||
|
||||
**Confirmed gap on provenance:** `situationNodeSchema` has no provenance fields (no source/inference annotation). `graphUpdateSchema` also lacks provenance fields. `updateCaseRequestSchema` carries the raw answer but provides no semantic-meaning fields. Evidence records built during startCase are not returned alongside graph state during update cycles.
|
||||
|
||||
**Confirmed gap on meaning preservation:** The answer string in `applyValidatedProposal` reaches only `deriveReasoningStateOverride` at line 2875 and is used solely for a narrow comparability confirmation check. After that point, only the structural graph state (already containing the LLM's interpretation) flows forward — not the original answer meaning.
|
||||
|
||||
**Confirmed support:** Existing relationship types distinguish evidence vs clarification needs. Structural validation gates maintain integrity. Decomposition quality gates exist on child unknowns. Null selectedQuestion is structurally valid.
|
||||
|
||||
**All eight requirements assessed individually** in a cross-reference matrix showing which have any support (prompt, parse/normalize, application, schema) and where gaps are located.
|
||||
|
||||
### Key Unresolved Items
|
||||
|
||||
- Whether provenance fields should be added to `situationNodeSchema`, `graphUpdateSchema`, or both;
|
||||
- How meaning preservation verification compares original answer text against proposed graph changes;
|
||||
- Where in the four-step pipeline (schema → prompt → parse → mutation) semantic-meaning carriers must enter;
|
||||
- Whether the current approach (two-field interpretation contract from 55D) is viable given the lack of schema carrier, or if a different mechanism is required.
|
||||
|
||||
### Conclusion
|
||||
|
||||
Source-inspection-only exercise completed. The production path does not carry semantic meaning — it carries structural graph changes that represent the LLM's interpretation of the answer. Every R1–R8 requirement depends on mechanisms absent from the current code path. A complete cross-reference with specific line-location gap targets is in `docs/reasoning-production-path-map.md`. No live inference was performed. No production code, prompts, schemas, graph files, Behaviour Selection rules, or UI code were modified. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: docs/reasoning-production-path-map.md for the full gap analysis and specific line-location targets. Status pending Rob's review.
|
||||
|
||||
---
|
||||
|
||||
### v0.51–v0.58 Progress — Product Provenance and Architectural Decisions
|
||||
|
||||
#### v0.51 — Zero Open Questions milestone
|
||||
|
||||
Established the zero-Open-Questions milestone as a genuine product feature: when all unknowns are resolved, the invitation "You've now worked through all of the questions we surfaced" appears in place of Open Questions. The eligibility uses canonical graph state (resolved nodes), not local `doneForNowIds`. This is a milestone invitation, not a readiness/completion judgement.
|
||||
|
||||
#### v0.52 — Focused investigation presentation ownership
|
||||
|
||||
Established that focused-presentation content must be scoped per-question. Previously, fresh unanswered Question B displayed stale content from Question A across every surface (previously answered, what this tells us, still unclear, questions raised, assumptions, connections). Fixed by thread-local filtering in `FocusedQuestionBody`. Previously answered contributions remain globally preserved in history; only presentation derivation is narrowed.
|
||||
|
||||
#### v0.53 — Empty Done + Re-open semantics
|
||||
|
||||
Established that empty Done (parked without providing an answer) is valid product behaviour: it parks the question locally, does NOT invoke episode processing, and does NOT produce a `no_episodic_content` 400 error. It produces the same resolved state shape as populated Done. Re-open returns the question to Open Questions and removes from `doneForNowIds`. Older stale development localStorage states (pre-v0.53 shape) may be discarded during dev phase; no migration required.
|
||||
|
||||
#### v0.54 — Investigation-level synthesis seam
|
||||
|
||||
Established a distinct investigation-level synthesis apparatus (`synthesizeInvestigationOverview()`) separate from Current Understanding. Important semantic lesson: Current Understanding and Investigation Report overview are NOT the same product artefact. Plausible interpretations in the Report remain explicitly interpretive rather than evidence. The epistemic boundary (evidence never promoted to interpretation; interpretations never promoted to understanding) is schema-enforced via Zod-safeParse.
|
||||
|
||||
#### v0.55 — Portfolio / Investigation / Report route architecture
|
||||
|
||||
Established three distinct product concepts:
|
||||
```
|
||||
/ → Portfolio (notebook index)
|
||||
/investigations/case-1 → Investigation (working case)
|
||||
/investigations/case-1/report → Investigation Report (derived summary)
|
||||
```
|
||||
|
||||
ReasoningWorkspace no longer owns Report presentation. The Report is a distinct route/page, not an internal state of the Investigation. Portfolio currently supports one canonical persisted investigation only. Temporary development identity remains `case-1`. True multi-investigation persistence/identity remains future work.
|
||||
|
||||
**Product analogy:** Portfolio = investigator notebook index, Investigation = working case/pages, Report = readable derived summary page. Users can eventually flick directly to the page they need.
|
||||
|
||||
#### v0.56 — Portfolio action semantics
|
||||
|
||||
Clarified that actions on an existing investigation card are distinct from creation of a new investigation. Actions on the card: View report, Continue investigation, Restart investigation. Creation is portfolio-level only: + Create new investigation below the card. No duplicate creation control inside the card.
|
||||
|
||||
#### v0.57 — Destructive Restart confirmation
|
||||
|
||||
Established that Restart investigation is explicitly destructive: first confirmation via dialog ("Restart this investigation?" with warning about lost data), then a second explicit "Restart investigation" button call. `clearInvestigation()` remains the canonical persisted-storage clear seam. No direct storage-key manipulation was introduced.
|
||||
|
||||
#### v0.58 — First Report generation lifecycle
|
||||
|
||||
Established that:
|
||||
- A genuine no-report investigation generates exactly one persisted Investigation Report
|
||||
- Report generation ownership belongs to the Report page, NOT ReasoningWorkspace or Investigation page
|
||||
- First Report visit = exactly 1 `/api/cases/overview` synthesis call
|
||||
- Subsequent Report visits = zero synthesis calls (renders persisted snapshot)
|
||||
- The Report is a derived artefact, not canonical reasoning evidence
|
||||
|
||||
**Live verification used genuine product-created investigations.** Six Open Questions surfaced in a fresh scenario — this was legitimate product output. An earlier experimental `≤5` processing bound was an apparatus constraint, NOT a product requirement. Do not document "Open Questions must be ≤5."
|
||||
|
||||
### Product Reasoning Lessons from v0.51–v0.58
|
||||
|
||||
**Investigator's notebook model.** The Portfolio / Investigation / Report triad maps to: notebook index → working case → readable outcome. This is an architectural decision about user navigation, not just technical separation.
|
||||
|
||||
**Report as durable derived artefact.** The Report should support future portfolio revisit, copy/export, Jira/document use, investigation portfolio — without becoming canonical reasoning evidence. It is a summary of what was understood at a point in time.
|
||||
|
||||
**User ownership / non-steering.** The engine facilitates investigation. It does not steer or prioritise which question must be answered next. User controls: which question to investigate, when to say Done for now, whether Current Understanding is sufficient, whether to reopen work, when to review the Report.
|
||||
|
||||
**Evidence lessons captured at provenance level:**
|
||||
- Tests can fail because apparatus cannot observe the intended contract — not because the product is broken.
|
||||
- Playwright snapshot refs are transient — never use them as action targets.
|
||||
- Client hydration must be treated as real product behaviour — pre-hydration empty ≠ absence of data.
|
||||
- Experimental execution bounds (e.g., ≤5 Open Questions) must not be mistaken for product requirements.
|
||||
- Manual product verification can validly establish prerequisite state when automation itself is not the subject of the experiment.
|
||||
|
||||
Reference in New Issue
Block a user