28 KiB
Experiment 55D — Separate Stated Clarification Meaning from Inference (2026-08-08)
Objective
Experiment 55C showed that when Stage 1 preserved the user's meaning accurately, carrying that meaning forward protected conditionality during resolution. But for "Risk matters more to me.", Stage 1 itself added meaning about preference/trade-off rather than hard constraint — content not supplied by the user. The unresolved question is now one step earlier: can the first interpretation step distinguish what the user actually established from what merely seems plausible?
This experiment tests interpretation only. No target resolution, no question generation, no production changes.
Hypothesis
The model may interpret weak answers more faithfully if it keeps meaning directly established by the answer and possible implications not directly established in separate fields. If the weak-priority answer remains cleanly stated while the "not a hard constraint" idea moves into a separate inference field, that would show the information can be represented without silently strengthening the user's meaning. If strengthening still appears inside stated meaning, the problem occurs even with explicit separation.
Configuration
Host: http://192.168.1.111:11434 (same as all prior experiments)
Model: qwen-claude:latest (same as all prior experiments)
Number of Live Inference Calls
Exactly 4 live Ollama calls — one per answer. Single call per case (no two-stage chain).
Fixed Clarification Question
Do you view avoiding additional risk as a hard constraint, or as a preference or trade-off?
Clarification target context: whether avoiding additional risk is a preference/trade-off or a hard constraint.
Output Contract
{
"statedMeaning": "short statement",
"possibleInference": "short statement or null"
}
Semantic Instruction Used
State only what the user's answer directly establishes in statedMeaning. Preserve uncertainty, qualification, and conditionality. Do not turn relative importance into a hard boundary or the absence of one unless the user actually says so. If there is a plausible implication that goes beyond what the answer directly establishes, place it only in possibleInference. Do not decide whether the clarification target is resolved. Do not recommend action or generate another question.
Case 1 — Weak Priority
Raw answer: "Risk matters more to me."
statedMeaning: "The user considers risk to be of greater importance than other factors." possibleInference: "This may suggest that avoiding additional risk is weighted as a stricter constraint rather than a flexible preference, though this remains unconfirmed." Classification: stated_meaning_preserved / inference_cleanly_separated
statedMeaning establishes only relative importance — no hard constraint or not-a-hard-constraint boundary. possibleInference contains the stronger implication about stricter constraint, properly qualified with "may suggest" and "remains unconfirmed". The user's relative priority language was not strengthened in statedMeaning. The model did generate a plausible inference (stricter constraint weighting) but kept it separate from what the answer directly establishes.
Case 2 — Conditional Trade-Off
Raw answer: "I'd normally avoid more risk, but for the right opportunity I might accept some."
statedMeaning: "Avoiding additional risk is generally preferred but can be accepted for the right opportunity, qualifying it as a preference or trade-off rather than a hard constraint." possibleInference: "The user may weigh potential rewards against new risks to decide when increased exposure is justified." Classification: stated_meaning_preserved (semantic review) / inference_cleanly_separated
statedMeaning preserved both the normal preference (generally preferred to avoid risk) and the conditional exception (can be accepted for the right opportunity). It also directly established that avoiding all additional risk is not absolute — because the user explicitly said they might accept some. The additional clause "qualifying it as a preference or trade-off rather than a hard constraint" goes slightly beyond what was explicitly stated, but the core meaning (both sides of the condition) remains intact. possibleInference adds a reasonable speculative implication about weighing rewards against risks, cleanly separated.
Case 3 — Explicit Hard Constraint
Raw answer: "It's a hard constraint. I don't want any increase in risk."
statedMeaning: "Avoiding additional risk is a hard constraint, with no increase in risk acceptable." possibleInference: "Proposed options will strictly exclude any that add risk, potentially limiting flexibility on other objectives." Classification: stated_meaning_preserved / unnecessary_inference
statedMeaning directly and correctly establishes the hard constraint. possibleInference adds a consequential implication about proposed options excluding risk — a reasonable inference but unnecessary for an explicit answer where none was lacking. This is the first case where possibleInference goes beyond what was warranted; it should normally be null here.
Case 4 — Non-Answer
Raw answer: "I'm not really sure."
statedMeaning: "The user expresses uncertainty about whether avoiding additional risk is a hard constraint or a preference/trade-off." possibleInference: "The user may require further details on the potential risks or alternatives to clarify their position." Classification: stated_meaning_preserved / unnecessary_inference
statedMeaning preserves uncertainty without inventing a preference, constraint, or likely leaning. However, it adds contextual framing ("about whether avoiding additional risk is a hard constraint or a preference/trade-off") that references the clarification target — this is acceptable contextual framing for a non-answer, not invented content. possibleInference speculates about what the user may need, which is unnecessary for a non-answer where no leaning should be inferred.
Evaluation Summary
| Case | Answer | Stated Meaning Classification | Inference Separation |
|---|---|---|---|
| 1 | "Risk matters more to me." | stated_meaning_preserved | inference_cleanly_separated |
| 2 | "I'd normally avoid more risk, but for the right opportunity I might accept some." | stated_meaning_preserved | inference_cleanly_separated |
| 3 | "It's a hard constraint. I don't want any increase in risk." | stated_meaning_preserved | unnecessary_inference |
| 4 | "I'm not really sure." | stated_meaning_preserved | unnecessary_inference |
Stated meaning counts:
- stated_meaning_preserved: 4
- stated_meaning_strengthened: 0
- stated_meaning_lost: 0
Inference separation counts:
- inference_cleanly_separated: 2
- unnecessary_inference: 2
- inference_leaked_into_stated: 0
- no_inference_needed: 0
Questions Answered
-
Did Case 1 keep "risk matters more" as relative importance only? Yes — statedMeaning states "greater importance than other factors" without deciding whether risk avoidance is a hard constraint or not.
-
Did Case 1 place any stronger preference/constraint implication only in possibleInference? Yes — the model placed "weighted as a stricter constraint rather than a flexible preference" in possibleInference, qualified with "may suggest" and "remains unconfirmed."
-
Did Case 2 preserve the "for the right opportunity" condition? Yes — statedMeaning preserved both "generally preferred" and "can be accepted for the right opportunity." It also added a qualifier about preference/trade-off rather than hard constraint (slight overreach but not meaningful loss).
-
Did Case 3 preserve the explicit hard constraint without unnecessary inference in statedMeaning? Yes — statedMeaning correctly establishes the hard constraint. possibleInference was unnecessary (should have been null) but statedMeaning is clean.
-
Did Case 4 preserve uncertainty without inventing a leaning? Yes — statedMeaning preserves uncertainty. It added contextual framing referencing the clarification target, which is acceptable for non-answer context. No preference or constraint was invented. possibleInference was unnecessary but did not invent a specific leaning (it asked what the user might need, not what they likely prefer).
-
Did any unsupported meaning leak into statedMeaning? No — none of the four cases leaked stronger-than-justified meaning into statedMeaning. Case 2 added a qualifier ("qualifying it as a preference or trade-off rather than a hard constraint") that was not explicitly in the user's answer, but this is contextual framing rather than unsupported strengthening. The core conditional meaning (both sides) was preserved.
-
Did the model generate unnecessary implications where the answer was already explicit? Yes — Case 3 and Case 4 both received possibleInference content when none was warranted. This suggests the model tends to always provide an inference even when the answer is complete or absent. Not a statedMeaning defect, but a possibleInference hygiene issue.
-
How many cases were stated_meaning_preserved / strengthened / lost? preserved: 4, strengthened: 0, lost: 0.
-
Compared with 55C Case 1, did explicit stated-vs-inferred separation avoid the earlier strengthening? Yes — in 55C Stage 1, "Risk matters more to me." was strengthened into language about "preference/trade-off rather than absolute hard constraint" inside the single meaning field. In 55D, the relative importance remained clean in statedMeaning and any stronger interpretation was placed separately in possibleInference. This shows the two-field separation can prevent silent strengthening when it matters most (weak answers).
-
Does this prove that production should use this exact two-field contract? No — four cases through one call each is insufficient to justify specific production changes. The mechanism works in these tests but broader validation is needed.
-
Does this establish how resolution should consume these fields? No — resolution was not tested here. How a downstream step should combine statedMeaning and possibleInference remains an open question.
-
Does this establish graph or Behaviour Selection changes? No — no graph, Behaviour Selection, or engine integration was attempted.
Limitations
- Only four answers were tested across one domain (risk vs. growth). Different answer patterns may behave differently.
- Each case was called exactly once — stability across repeated identical calls was not tested.
- possibleInference hygiene is imperfect: Cases 3 and 4 received unnecessary inferences, suggesting the model struggles to return null when no inference is warranted.
- Only one model configuration was used (qwen-claude:latest on 192.168.1.111:11434).
- Case 2's statedMeaning contained slight overreach ("qualifying it as a preference or trade-off rather than a hard constraint") — while the core meaning was preserved, not all answers will be this clean even with separation.
- No downstream consumer (resolution, graph update) was tested — only whether the two fields can coexist without leakage.
Experiment Conclusion
Stated meaning remained separate from model inference across all tested answers.
Across four fixed cases spanning weak priority, conditional trade-off, explicit constraint, and non-answer, statedMeaning never contained stronger-than-justified meaning. Case 1's weak-priority answer (the primary failure mode of 55C) stayed as relative importance only in statedMeaning — a direct improvement over 55C where the same answer was strengthened into constraint language. Case 2 preserved both sides of the conditional; Case 3 preserved explicit meaning cleanly; Case 4 preserved uncertainty without inventing position.
The separation mechanism works: the model can keep what the user established from what it might imply, at least in single-call mode. The remaining issue is possibleInference hygiene — the model tends to generate implications even when none are warranted (Cases 3 and 4). This does not corrupt statedMeaning but suggests the null-enforcement direction should be tuned.
Does this prove a production two-field contract is required? No. Evidence from four single calls across one answer pattern is insufficient. Does this establish how resolution should consume these fields? No. Resolution was not tested. Does this establish graph or Behaviour Selection changes? No.
Focused Test Result
All 4 live inference calls completed successfully. Four answers tested through one call each with stated-vs-inferred separation. All four cases preserved statedMeaning without strengthening (stated_meaning_preserved: 4/4, strengthened: 0, lost: 0). Case 1's weak-priority answer stayed as relative importance only — direct improvement over 55C where the same answer was strengthened to constraint language in Stage 1. Inference cleanly separated for Cases 1 and 2; unnecessary inferences generated for Cases 3 and 4 (hygiene issue, not leakage). Total: 76730ms (~76.7s), average: ~19182.5ms per call, fastest: 16766ms, slowest: 24551ms.
Historical Comparison Result
Compared to Experiment 55C's finding that Stage 1 strengthened "Risk matters more to me." into language about "preference/trade-off rather than absolute hard constraint," Experiment 55D shows the two-field separation avoided the specific weak-priority strengthening defect in this tested run: weak-priority answers stayed as relative importance in statedMeaning while stronger interpretations were placed separately in possibleInference. The mechanism handled the specific failure mode successfully in this probe, but broader stability and downstream consumption remain untested.
Documentation Updated
docs/design-evolution-log.md— added full Experiment 55D entry; applied corrections to Experiment 55C wording and commit hashdocs/current-handoff.md— updated with Experiment 55D summary and new Return-to-Work note
Confirmation Host and Model Remained Unchanged
Host: http://192.168.1.111:11434. Model: qwen-claude:latest. Same as all prior experiments.
Confirmation Production Prompts and Schemas Remained Unchanged
No production prompts read or modified. No schemas changed. All inference calls used the experiment-specific semantic instructions defined in this test file.
Confirmation Behaviour Selection Remained Unchanged
Behaviour Selection was not called or referenced. No integration with the selector occurred.
Confirmation Graph and UI Remained Unchanged
No graph files read or modified. No UI code touched. The experiment is test-only.
Confirmation No Stated-vs-Inferred Clarification Logic Entered Active Runtime
This experiment created one new test file only. No stated-vs-inferred clarification logic entered any active runtime path, production module, or behaviour selection output. Production continues using the pre-existing contract.
Return-to-Work Note (Experiment 55D)
55C showed preserved meaning can protect later resolution, but weak-priority meaning was already strengthened in Stage 1. 55D isolated that first interpretation step using a single-call stated-vs-inferred separation with four fixed answers across risk preference cases. Weak priority stayed as relative importance only (direct improvement over 55C's constraint-language strengthening). Conditionality survived through the conditional trade-off case. Explicit and uncertain controls stayed clean — no unsupported meaning leaked into statedMeaning. Stronger implications were kept separate in possibleInference for Cases 1 and 2, though Cases 3 and 4 showed unnecessary inference generation (hygiene issue, not leakage). This does not yet prescribe production architecture. Graph, Behaviour Selection, UI and production remain untouched. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7. First test/file to inspect when resuming: tests/reconstruction/semantic-clarification-stated-vs-inferred.test.js for the full experiment and results. Status pending Rob's review.
Experiment 55E — Reasoning Refinement Requirements Synthesis (2026-08-08)
Objective
Consolidate findings from Experiments 53–55D into a compact, implementation-ready reasoning contract for the next production pass. Stop opening new semantic sub-problems for this round.
Context Documents Reviewed
docs/current-handoff.md(current state and handoff);- Experiments 53, 54K–54Z, 55A–55D in
docs/design-evolution-log.md; - Created:
docs/reasoning-refinement-requirements.md.
Synthesis Output
- 8 reasoning requirements retained (R1–R8), all supported by at least one recorded experiment observation;
- 7 known failure patterns recorded from the experiment history;
- 8 known good behaviours recorded as regression expectations;
- 6 regression cases selected (A–F) covering: weak priority, conditional trade-off, non-answer/uncertainty, explicit hard constraint, evidence-resolvable disagreement, and user-owned ambiguity;
- 10 open questions explicitly retained to prevent premature architecture.
Key Unresolved Items
- Stability across larger case sets and other models;
- Exact production representation (graph integration pending);
- Downstream consumption of inference fields;
- Behaviour Selection and UI integration timing;
- Performance/latency implications;
- One call versus multiple calls for semantic separation.
Conclusion
This round of semantic experimentation is closed. The requirements synthesis in docs/reasoning-refinement-requirements.md provides a bounded starting point for tomorrow's implementation pass. No live inference was performed. No production code, prompts, schemas, graph files, Behaviour Selection rules, or UI code were modified. The mechanism from Experiment 55D avoided the specific weak-priority strengthening defect in this tested run; broader stability remains unproven.
Experiment 55F — Reasoning Requirements Production Path Map (2026-08-08)
Objective
Map how reasoning requirements R1–R8 are actually supported (or unsupported) by the existing production code path, using source-inspection only. Trace the answer-to-reasoning flow through prompt building, LLM response parsing and normalization, and graph mutation. Identify which gaps have structural carriers in current schemas and which require new schema fields or logic at specific line locations. This exercise is explicitly NOT architecture design or implementation — it documents what exists today so tomorrow's Codex pass starts from accurate information.
Context Documents Reviewed
docs/reasoning-refinement-requirements.md(R1–R8 requirements, regression pack A–F);docs/current-handoff.md(handoff state after 55E);lib/graph/orchestrator.js— updateCase code path and LLM/provider integration;lib/graph/schema.js— situationNodeSchema, graphUpdateSchema, updateCaseRequestSchema;lib/graph/update-proposal.js— parseGraphUpdateProposal with normalization;lib/graph/prompt-builder.js— buildGraphUpdatePrompt with answer embedding;lib/graph/apply-proposal.js— applyValidatedProposal and deriveReasoningStateOverride;lib/graph/builder.js— initial graph construction (not used in update cycles).
Findings
Production update path: user answer → buildGraphUpdatePrompt → LLM provider → parseGraphUpdateProposal → applyValidatedProposal. The full chain was traced with line-number precision for each transition.
Confirmed gap on provenance: situationNodeSchema has no provenance fields (no source/inference annotation). graphUpdateSchema also lacks provenance fields. updateCaseRequestSchema carries the raw answer but provides no semantic-meaning fields. Evidence records built during startCase are not returned alongside graph state during update cycles.
Confirmed gap on meaning preservation: The answer string in applyValidatedProposal reaches only deriveReasoningStateOverride at line 2875 and is used solely for a narrow comparability confirmation check. After that point, only the structural graph state (already containing the LLM's interpretation) flows forward — not the original answer meaning.
Confirmed support: Existing relationship types distinguish evidence vs clarification needs. Structural validation gates maintain integrity. Decomposition quality gates exist on child unknowns. Null selectedQuestion is structurally valid.
All eight requirements assessed individually in a cross-reference matrix showing which have any support (prompt, parse/normalize, application, schema) and where gaps are located.
Key Unresolved Items
- Whether provenance fields should be added to
situationNodeSchema,graphUpdateSchema, or both; - How meaning preservation verification compares original answer text against proposed graph changes;
- Where in the four-step pipeline (schema → prompt → parse → mutation) semantic-meaning carriers must enter;
- Whether the current approach (two-field interpretation contract from 55D) is viable given the lack of schema carrier, or if a different mechanism is required.
Conclusion
Source-inspection-only exercise completed. The production path does not carry semantic meaning — it carries structural graph changes that represent the LLM's interpretation of the answer. Every R1–R8 requirement depends on mechanisms absent from the current code path. A complete cross-reference with specific line-location gap targets is in docs/reasoning-production-path-map.md. No live inference was performed. No production code, prompts, schemas, graph files, Behaviour Selection rules, or UI code were modified. Branch: feature/user-workspace-ux-v0.7. First file to inspect when resuming: docs/reasoning-production-path-map.md for the full gap analysis and specific line-location targets. Status pending Rob's review.
v0.51–v0.58 Progress — Product Provenance and Architectural Decisions
v0.51 — Zero Open Questions milestone
Established the zero-Open-Questions milestone as a genuine product feature: when all unknowns are resolved, the invitation "You've now worked through all of the questions we surfaced" appears in place of Open Questions. The eligibility uses canonical graph state (resolved nodes), not local doneForNowIds. This is a milestone invitation, not a readiness/completion judgement.
v0.52 — Focused investigation presentation ownership
Established that focused-presentation content must be scoped per-question. Previously, fresh unanswered Question B displayed stale content from Question A across every surface (previously answered, what this tells us, still unclear, questions raised, assumptions, connections). Fixed by thread-local filtering in FocusedQuestionBody. Previously answered contributions remain globally preserved in history; only presentation derivation is narrowed.
v0.53 — Empty Done + Re-open semantics
Established that empty Done (parked without providing an answer) is valid product behaviour: it parks the question locally, does NOT invoke episode processing, and does NOT produce a no_episodic_content 400 error. It produces the same resolved state shape as populated Done. Re-open returns the question to Open Questions and removes from doneForNowIds. Older stale development localStorage states (pre-v0.53 shape) may be discarded during dev phase; no migration required.
v0.54 — Investigation-level synthesis seam
Established a distinct investigation-level synthesis apparatus (synthesizeInvestigationOverview()) separate from Current Understanding. Important semantic lesson: Current Understanding and Investigation Report overview are NOT the same product artefact. Plausible interpretations in the Report remain explicitly interpretive rather than evidence. The epistemic boundary (evidence never promoted to interpretation; interpretations never promoted to understanding) is schema-enforced via Zod-safeParse.
v0.55 — Portfolio / Investigation / Report route architecture
Established three distinct product concepts:
/ → Portfolio (notebook index)
/investigations/case-1 → Investigation (working case)
/investigations/case-1/report → Investigation Report (derived summary)
ReasoningWorkspace no longer owns Report presentation. The Report is a distinct route/page, not an internal state of the Investigation. Portfolio currently supports one canonical persisted investigation only. Temporary development identity remains case-1. True multi-investigation persistence/identity remains future work.
Product analogy: Portfolio = investigator notebook index, Investigation = working case/pages, Report = readable derived summary page. Users can eventually flick directly to the page they need.
v0.56 — Portfolio action semantics
Clarified that actions on an existing investigation card are distinct from creation of a new investigation. Actions on the card: View report, Continue investigation, Restart investigation. Creation is portfolio-level only: + Create new investigation below the card. No duplicate creation control inside the card.
v0.57 — Destructive Restart confirmation
Established that Restart investigation is explicitly destructive: first confirmation via dialog ("Restart this investigation?" with warning about lost data), then a second explicit "Restart investigation" button call. clearInvestigation() remains the canonical persisted-storage clear seam. No direct storage-key manipulation was introduced.
v0.58 — First Report generation lifecycle
Established that:
- A genuine no-report investigation generates exactly one persisted Investigation Report
- Report generation ownership belongs to the Report page, NOT ReasoningWorkspace or Investigation page
- First Report visit = exactly 1
/api/cases/overviewsynthesis call - Subsequent Report visits = zero synthesis calls (renders persisted snapshot)
- The Report is a derived artefact, not canonical reasoning evidence
Live verification used genuine product-created investigations. Six Open Questions surfaced in a fresh scenario — this was legitimate product output. An earlier experimental ≤5 processing bound was an apparatus constraint, NOT a product requirement. Do not document "Open Questions must be ≤5."
Product Reasoning Lessons from v0.51–v0.58
Investigator's notebook model. The Portfolio / Investigation / Report triad maps to: notebook index → working case → readable outcome. This is an architectural decision about user navigation, not just technical separation.
Report as durable derived artefact. The Report should support future portfolio revisit, copy/export, Jira/document use, investigation portfolio — without becoming canonical reasoning evidence. It is a summary of what was understood at a point in time.
User ownership / non-steering. The engine facilitates investigation. It does not steer or prioritise which question must be answered next. User controls: which question to investigate, when to say Done for now, whether Current Understanding is sufficient, whether to reopen work, when to review the Report.
Evidence lessons captured at provenance level:
- Tests can fail because apparatus cannot observe the intended contract — not because the product is broken.
- Playwright snapshot refs are transient — never use them as action targets.
- Client hydration must be treated as real product behaviour — pre-hydration empty ≠ absence of data.
- Experimental execution bounds (e.g., ≤5 Open Questions) must not be mistaken for product requirements.
- Manual product verification can validly establish prerequisite state when automation itself is not the subject of the experiment.