10255 lines
634 KiB
Markdown
10255 lines
634 KiB
Markdown
# Design Evolution Log
|
||
|
||
A chronological record of why significant design decisions were made. This is NOT a changelog. It records the product's evolution of thinking.
|
||
|
||
This document records discoveries, not decisions. Every entry represents our best understanding at that point in time and may later be superseded by a better model.
|
||
|
||
---
|
||
|
||
## Phase 1
|
||
|
||
Simple conversational investigation
|
||
|
||
Question → Answer interaction.
|
||
|
||
Purpose:
|
||
Prove the reasoning loop.
|
||
|
||
Learning:
|
||
Conversation alone does not provide sufficient context during longer investigations.
|
||
|
||
---
|
||
|
||
## Phase 2
|
||
|
||
Persistent investigation notebook
|
||
|
||
Added:
|
||
|
||
- current understanding
|
||
- original situation
|
||
- investigation history
|
||
|
||
Learning:
|
||
Users need persistent context rather than remembering previous answers.
|
||
|
||
---
|
||
|
||
## Phase 3
|
||
|
||
Document workspace
|
||
|
||
Created a coherent workspace with:
|
||
|
||
- investigation status
|
||
- current investigation
|
||
- response
|
||
- understanding
|
||
- investigation map placeholder
|
||
- situation
|
||
- history
|
||
|
||
Learning:
|
||
The interface became usable but still behaved like a document rather than a workspace.
|
||
|
||
---
|
||
|
||
## Phase 4 (Current Exploration)
|
||
|
||
Facilitated Investigation Workshop
|
||
|
||
Status:
|
||
Experimental.
|
||
|
||
Hypothesis:
|
||
|
||
The Confidence Engine is not:
|
||
|
||
- a chatbot
|
||
- a dashboard
|
||
- a form
|
||
|
||
It is a facilitated investigation workspace.
|
||
|
||
The interface should resemble the environment in which structured thinking happens.
|
||
|
||
Record discoveries rather than conclusions.
|
||
|
||
Leave room for future phases.
|
||
|
||
---
|
||
|
||
## Phase 4 — Guiding Principles
|
||
|
||
The Confidence Engine is a workspace, not a document.
|
||
|
||
People think in multiple directions simultaneously.
|
||
|
||
Useful context should be visible together.
|
||
|
||
The interface should favour thinking over scrolling.
|
||
|
||
The workspace should feel like a large desk or workshop rather than a narrow report.
|
||
|
||
The engine facilitates thinking.
|
||
|
||
The user contributes evidence.
|
||
|
||
The workspace captures shared understanding.
|
||
|
||
### Experiment 01 — Wider canvas
|
||
|
||
Hypothesis:
|
||
The document-like feeling is caused partly by the narrow outer container.
|
||
|
||
Change:
|
||
Increase the available desktop workspace width without rearranging any components.
|
||
|
||
Result:
|
||
Confirmed.
|
||
|
||
Learning:
|
||
Increasing the outer workspace width reduced the narrow-document feeling and made better use of large displays.
|
||
|
||
Unexpected learning:
|
||
Width alone did not create a workshop. The wider canvas exposed that the interface still behaves as a collection of independent cards, with supporting artefacts unsure how to use the available space.
|
||
|
||
Decision:
|
||
Keep the wider desktop canvas.
|
||
|
||
Next question:
|
||
Can grouping the interface into cognitive work zones make the wider canvas feel like a coherent investigation surface?
|
||
|
||
### Experiment 02 — Cognitive work zones
|
||
|
||
Hypothesis:
|
||
A workspace organised around what the investigator is doing will feel more coherent than one organised around equal cards or equal columns.
|
||
|
||
Result:
|
||
Partially confirmed.
|
||
|
||
Learning:
|
||
|
||
The workspace feels more coherent when organised into cognitive work zones rather than a simple document stack.
|
||
|
||
However, another distinction emerged that is more important than the zones themselves.
|
||
|
||
The interface naturally separates into two different modes:
|
||
|
||
• the active conversation between investigator and facilitator
|
||
|
||
and
|
||
|
||
• the shared workspace describing the current understanding.
|
||
|
||
Unexpected learning:
|
||
|
||
History feels incorrect when treated as reference information.
|
||
|
||
History is actually the continuation of the investigator's conversation.
|
||
|
||
Every response immediately becomes history.
|
||
|
||
The notebook should therefore grow naturally from the Response area.
|
||
|
||
The Investigation Status card currently competes with the Current Investigation card.
|
||
|
||
The current question is the primary focus.
|
||
|
||
Status is supporting context.
|
||
|
||
Decision:
|
||
|
||
Keep the cognitive-zone concept.
|
||
|
||
Refine the zones around conversational flow instead of card grouping.
|
||
|
||
Next question:
|
||
Can the workspace clearly separate conversation from shared understanding?
|
||
|
||
### Experiment 03 — Conversation versus Workspace
|
||
|
||
Hypothesis
|
||
|
||
Investigators think in two simultaneous modes.
|
||
|
||
Mode 1:
|
||
The conversation.
|
||
|
||
Question
|
||
↓
|
||
|
||
Response
|
||
↓
|
||
|
||
History
|
||
|
||
Mode 2:
|
||
The shared workspace.
|
||
|
||
Status
|
||
|
||
Understanding
|
||
|
||
Situation
|
||
|
||
Map
|
||
|
||
Separating these should make the interface feel more like a facilitated investigation than a collection of cards.
|
||
|
||
Evaluation:
|
||
Partially confirmed.
|
||
|
||
Learning:
|
||
|
||
The workspace feels more coherent when organised into cognitive zones rather than a simple document stack.
|
||
|
||
However, another distinction emerged that is more important than the zones themselves.
|
||
|
||
The interface naturally separates into two different modes:
|
||
|
||
• the active conversation between investigator and facilitator
|
||
|
||
and
|
||
|
||
• the shared workspace describing the current understanding.
|
||
|
||
Unexpected learning:
|
||
|
||
History feels incorrect when treated as reference information.
|
||
|
||
History is actually the continuation of the investigator's conversation.
|
||
|
||
Every response immediately becomes history.
|
||
|
||
The notebook should therefore grow naturally from the Response area.
|
||
|
||
The Investigation Status card currently competes with the Current Investigation card.
|
||
|
||
The current question is the primary focus.
|
||
|
||
Status is supporting context.
|
||
|
||
Decision:
|
||
|
||
Keep the cognitive-zone concept.
|
||
|
||
Refine the zones around conversational flow instead of card grouping.
|
||
|
||
Next question:
|
||
Can the workspace clearly separate conversation from shared understanding?
|
||
|
||
### Experiment 04 — Facilitated Workshop Introduction
|
||
|
||
Hypothesis
|
||
|
||
Beginning with a facilitator-style introduction will create more confidence than presenting an empty workspace.
|
||
|
||
Questions
|
||
|
||
- Does the interface feel more welcoming?
|
||
- Does reducing the visual weight of the textarea improve the first experience?
|
||
- Does separating "starting" from "investigating" feel natural?
|
||
- Does the transition into the investigation workspace feel meaningful?
|
||
|
||
Status:
|
||
Experimental.
|
||
|
||
Result:
|
||
Partially confirmed.
|
||
|
||
Learning:
|
||
|
||
The facilitator introduction reduced the intimidation of the first screen.
|
||
|
||
Replacing the empty landing page with a guided introduction improved the emotional tone.
|
||
|
||
However, stacking the introduction above the input still gives the introduction excessive visual prominence.
|
||
|
||
Repeat users may not want to repeatedly read the same introduction.
|
||
|
||
Orientation should remain available without dominating the workflow.
|
||
|
||
Decision:
|
||
|
||
Keep the introduction concept but change its spatial relationship to the workspace — move it from above to beside, making it optional rather than mandatory.
|
||
|
||
Next question:
|
||
Does a horizontal facilitator/workspace layout feel more natural?
|
||
|
||
### Experiment 05 — Facilitator Panel and Adaptive Landing Workspace
|
||
|
||
Hypothesis
|
||
|
||
Placing the facilitator beside the working area will feel more like entering a facilitated workshop than stacking instructional content above the workspace.
|
||
|
||
Allowing the user to dismiss the facilitator will reduce friction for returning users while preserving onboarding for new users.
|
||
|
||
Questions
|
||
|
||
- Does a horizontal facilitator/workspace layout feel more natural?
|
||
- Does the user's eye move naturally from facilitator to workspace?
|
||
- Does the workspace become the primary focus?
|
||
- Does "Don't show again" feel preferable to automatically hiding the introduction?
|
||
- Should the facilitator panel become an optional workspace companion rather than mandatory onboarding?
|
||
|
||
Status:
|
||
Completed.
|
||
|
||
Findings:
|
||
|
||
- A horizontal facilitator/workspace arrangement feels more natural than stacked onboarding.
|
||
- The workspace becomes the visual destination rather than the introduction.
|
||
- User-controlled dismissal is preferable to automatic hiding.
|
||
- The facilitator feels useful but visually too passive.
|
||
- Remaining issues are now visual hierarchy rather than layout architecture.
|
||
|
||
### Experiment 06 — Focused Investigation
|
||
|
||
Hypothesis
|
||
|
||
The interface should gently guide attention towards the current task without hiding supporting information.
|
||
|
||
Reducing competition between panels may improve concentration more than introducing additional colour or decoration.
|
||
|
||
Questions
|
||
|
||
- Does visual emphasis naturally guide the eye?
|
||
- Can supporting panels become quieter without disappearing?
|
||
- Does the investigation question become the obvious focal point?
|
||
- Does the workspace feel calmer?
|
||
- Are we approaching a professional investigation environment?
|
||
|
||
Status:
|
||
Closed.
|
||
|
||
## Result
|
||
|
||
Partially confirmed.
|
||
|
||
## What did we learn?
|
||
|
||
- Stronger visual hierarchy can direct attention without rearranging the interface.
|
||
- The facilitator briefing became easier to distinguish.
|
||
- Colour and tint improved separation only modestly.
|
||
- Meaning must not depend on colour.
|
||
- Areas and intent should remain distinguishable through structure, spacing, typography, borders, shape and placement.
|
||
- The initial textarea still implies that the user should provide a detailed report.
|
||
- The size of an input communicates the amount of information expected.
|
||
|
||
## Decision
|
||
|
||
Retain the useful hierarchy refinements provisionally.
|
||
|
||
Do not increase reliance on colour.
|
||
|
||
Defer dark mode and broader palette work.
|
||
|
||
The next experiment should test whether a smaller starting input better communicates that the user only needs to provide an initial observation.
|
||
|
||
Do not rewrite previous experiments.
|
||
|
||
---
|
||
|
||
### Experiment 07 — Lightweight Starting Observation
|
||
|
||
#### Hypothesis
|
||
|
||
A smaller initial input will make beginning an investigation feel easier and will communicate that the engine needs only a concise observation rather than a complete analysis.
|
||
|
||
#### Questions
|
||
|
||
- Does the input feel like a conversation starter rather than a report form?
|
||
- Is three to four visible lines sufficient?
|
||
- Does the facilitator panel and input area feel better balanced?
|
||
- Does the user understand that further detail will be gathered through questions?
|
||
- Does reducing the input height make the Analyse action easier to notice?
|
||
|
||
#### Evaluation
|
||
|
||
Pending visual review.
|
||
|
||
#### Result
|
||
|
||
Confirmed.
|
||
|
||
Four visible rows better communicates a starting observation than six.
|
||
|
||
Input size communicates expected effort.
|
||
|
||
"What have you noticed?" reinforces observational thinking.
|
||
|
||
Users are encouraged to begin rather than compose.
|
||
|
||
The facilitator and workspace now feel more balanced.
|
||
|
||
This interaction principle should continue throughout the investigation rather than existing only on the landing page.
|
||
|
||
#### Decision
|
||
|
||
Retain the smaller landing input.
|
||
|
||
Proceed to investigate consistency between the landing experience and investigation responses.
|
||
|
||
---
|
||
|
||
### Experiment 09 — Investigation Rhythm
|
||
|
||
#### Result
|
||
|
||
Partially confirmed.
|
||
|
||
#### What did we learn?
|
||
|
||
- Moving History directly beneath Response improves the sense of conversational continuity.
|
||
- The sequence Question → Response → History is cognitively coherent.
|
||
- History behaves like the growing notebook of the investigation, not general reference material.
|
||
- Allowing History to span the full workspace breaks the wider spatial model.
|
||
- Situation and Investigation Map should remain stable supporting artefacts rather than moving down as the notebook grows.
|
||
- The conversation needs a dedicated vertical lane.
|
||
|
||
#### Decision
|
||
|
||
Keep History directly connected to Response.
|
||
|
||
Refine the desktop workspace into a stable conversation lane and a stable supporting lane.
|
||
|
||
Do not rewrite previous experiments.
|
||
|
||
---
|
||
|
||
### Experiment 08 — Consistent Investigation Responses
|
||
|
||
#### Hypothesis
|
||
|
||
Every answer given during an investigation should feel like an observation, not a report.
|
||
|
||
The response component should therefore communicate the same expected effort as the initial scenario input.
|
||
|
||
#### Questions
|
||
|
||
- Does a smaller response area reduce perceived effort?
|
||
- Does the investigation feel more conversational?
|
||
- Does consistency improve confidence?
|
||
- Does the workspace become visually calmer?
|
||
- Does the current investigation remain the dominant focus?
|
||
|
||
#### Result
|
||
|
||
Confirmed.
|
||
|
||
Consistent interaction patterns reduce cognitive effort.
|
||
|
||
Users should not have to learn different behaviours between the landing page and investigation.
|
||
|
||
Smaller response areas reinforce concise observations.
|
||
|
||
The engine appears more conversational when each answer feels lightweight.
|
||
|
||
Consistency is becoming a stronger design tool than decoration.
|
||
|
||
#### Decision
|
||
|
||
Retain consistent input sizing across both contexts.
|
||
|
||
---
|
||
|
||
### Experiment 10 — Stable Conversation Column
|
||
|
||
#### Hypothesis
|
||
|
||
A persistent two-thirds conversation column beside a one-third supporting column will allow the investigation notebook to grow without moving the shared reference artefacts.
|
||
|
||
#### Questions
|
||
|
||
- Does the left column feel like one continuous investigation?
|
||
- Does History grow naturally beneath Response?
|
||
- Do Situation and Investigation Map remain easy to reference?
|
||
- Does the interface feel spatially stable as turns accumulate?
|
||
- Does showing full question text improve readability now that sufficient width exists?
|
||
|
||
#### Evaluation
|
||
|
||
Visual review completed.
|
||
|
||
#### Status
|
||
|
||
Closed.
|
||
|
||
## Result
|
||
|
||
Partially confirmed.
|
||
|
||
## What did we learn?
|
||
|
||
- The investigation workspace is beginning to feel like a genuine facilitated investigation rather than a document.
|
||
- The two-column workspace (conversation on the left, reference material on the right) is proving to be a stronger mental model than previous layouts.
|
||
- Keeping Situation and Investigation Map fixed while History grows vertically feels more natural.
|
||
- The investigation question, response and history now read as one continuous conversation.
|
||
- Developer Details have become extremely valuable.
|
||
- The graph produced by the reasoning engine is far richer than previously realised. The graph now contains structured concepts including:
|
||
|
||
- observations
|
||
- unknowns
|
||
- assumptions
|
||
- relationships
|
||
- metrics
|
||
- state
|
||
|
||
This suggests the UI should increasingly become a human-friendly projection of the graph rather than inventing separate state.
|
||
|
||
The current "Investigation in progress" panel exposes developer-oriented statistics (nodes, edges, unknowns etc.) which are useful during development but are not the most helpful representation for an end user.
|
||
|
||
---
|
||
|
||
## Emerging Direction — Graph as Source of Truth
|
||
|
||
The reasoning graph is becoming the shared source of truth for multiple UI views.
|
||
|
||
Different interfaces may project the same graph for different audiences:
|
||
|
||
- Version A — compact technical progress;
|
||
- Version B — detailed graph inspection;
|
||
- Version C — user-facing facilitator view;
|
||
- Developer Details — complete diagnostics;
|
||
- Investigation Map — future spatial projection;
|
||
- Current Question — active uncertainty projection.
|
||
|
||
The UI should not maintain separate invented summaries where the graph already contains the underlying information.
|
||
|
||
This is an emerging direction, not a final architecture decision.
|
||
|
||
---
|
||
|
||
## Emerging Direction — Facilitator Translation Layer
|
||
|
||
> The UI should progressively become a translation layer over the reasoning graph rather than maintaining separate duplicated summaries. Internal graph concepts should remain available for developers, while end users see a facilitator-style explanation of what is currently understood and what remains uncertain.
|
||
|
||
The current technical progress panel (nodes, edges, unknowns, assumptions) exposes developer-oriented statistics. These are valuable during development but not the most helpful representation for an end user.
|
||
|
||
The next direction is to explore presenting the same underlying graph data as a facilitator's notebook — what is known, what remains uncertain, and a quiet summary of the reasoning state underneath.
|
||
|
||
---
|
||
|
||
### Experiment 11 — Facilitator Progress Panel (Version B)
|
||
|
||
#### Hypothesis
|
||
|
||
The same underlying reasoning graph can be presented in a much more human-friendly way without changing the reasoning engine, API contracts, or graph generation.
|
||
|
||
A facilitator-style panel should communicate:
|
||
|
||
- what is known (resolved nodes and observations)
|
||
- what remains uncertain (unresolved unknowns and assumptions)
|
||
- a quiet summary of the reasoning state underneath
|
||
|
||
#### Questions
|
||
|
||
- Can the same graph data be translated into a facilitator-style view that end users understand more naturally?
|
||
- Does separating "known" from "still investigating" reduce cognitive load compared to node/edge counts?
|
||
- Is a quiet reasoning summary sufficient, or does it need more context?
|
||
- Does the translation-layer principle hold — presenting the graph as a notebook rather than raw data?
|
||
|
||
#### Result
|
||
|
||
Partially confirmed.
|
||
|
||
#### What did we learn?
|
||
|
||
- Version B proved that the reasoning graph contains substantially more useful information than Version A exposes.
|
||
- The graph already contains observations, unknowns, assumptions, metrics, relationships and state.
|
||
- The graph is rich enough to support multiple UI projections.
|
||
- Exposing the graph almost verbatim overwhelms the user.
|
||
- Technical categories are useful for development but do not directly communicate investigation progress.
|
||
- The user needs a translation of the graph rather than a graph browser.
|
||
- Developer Details should remain the place for complete technical inspection.
|
||
- A user-facing view needs filtering, prioritisation, deduplication and clear epistemic labels.
|
||
|
||
#### Decision
|
||
|
||
Keep Version A and Version B available for comparison.
|
||
|
||
Proceed with a Version C facilitator view built from the same graph.
|
||
|
||
---
|
||
|
||
### Experiment 12 — Facilitator View (Version C)
|
||
|
||
#### Hypothesis
|
||
|
||
The existing reasoning graph can be deterministically translated into a concise facilitator view that helps the user understand:
|
||
|
||
- what is currently known;
|
||
- what remains uncertain;
|
||
- what may explain the situation;
|
||
- why the investigation is continuing.
|
||
|
||
#### Questions
|
||
|
||
- Can the graph produce a useful human-facing summary without another LLM call?
|
||
- Can observations, unknowns and assumptions be clearly distinguished?
|
||
- Can duplicate or low-value graph content be filtered reliably?
|
||
- Does a concise projection improve understanding without exposing implementation detail?
|
||
- Does the panel remain useful across mocks and live Ollama output?
|
||
- Can the same view work during early, middle and terminal investigation states?
|
||
|
||
#### Evaluation
|
||
|
||
Completed. Visual and live-data review performed.
|
||
|
||
#### Result
|
||
|
||
Confirmed.
|
||
|
||
#### What did we learn?
|
||
|
||
- The reasoning graph already contains all the information needed for a useful human-facing summary — no additional LLM calls are required.
|
||
- Routing by semantic role (observation, question, explanation) rather than graph kind produces a more natural user experience.
|
||
- Filtering scaffolding content (scenario summaries, system/tool references, metric object descriptions, process labels) is essential to keep the view focused on findings.
|
||
- Deduplication of near-duplicate observations reduces noise without losing information.
|
||
- Epistemic clarity matters — resolved unknowns become factual observations and should be classified as known rather than still-under-investigation.
|
||
- The panel works across all investigation phases (early, active, terminal).
|
||
|
||
#### Decision
|
||
|
||
Close Experiment 12 as confirmed. Proceed to refine the translation through semantic classification in the next iteration.
|
||
|
||
---
|
||
|
||
### Experiment 13 — Semantic Facilitator Translation
|
||
|
||
#### Hypothesis
|
||
|
||
Improving the deterministic projection from graph semantics to user-facing language — by classifying nodes by *meaning* rather than *graph kind*, suppressing scaffolding, merging duplicates, and preferring concrete observations — produces a significantly better facilitator view without changing the reasoning engine, prompts, graph generation, or any external contracts.
|
||
|
||
#### Questions
|
||
|
||
- Does semantic role classification (observation vs question vs explanation) route content more naturally than graph-kind classification?
|
||
- Does scaffolding suppression remove visual noise that previously dominated derived summaries?
|
||
- Does deduplication reduce redundant items that express the same observation under slightly different wording?
|
||
- Do concrete observations appear before abstract labels in ranked output?
|
||
- Does the view remain robust when consumed by the existing panel component (investigation-summary-panel-v3) without any changes to that component?
|
||
|
||
#### Evaluation
|
||
|
||
Completed. Tests: 37 scenarios passing across filtering, classification, deduplication, ranking, section framing, mock-data integration, and edge cases.
|
||
|
||
#### Result
|
||
|
||
Confirmed.
|
||
|
||
#### What did we learn?
|
||
|
||
- Semantic role routing outperforms kind-based routing: a node with `kind: "state"` that contains concrete data (e.g., "Revenue increased 12%") is more useful as an observation than a state description.
|
||
- Scaffolding suppression works best when applied early — filtering at the semantic classification stage prevents structural glue from contaminating any section.
|
||
- Three-tier filtering is effective: scaffolding patterns (highest priority), internal vocabulary (medium), then technical summary patterns (lowest).
|
||
- Deduplication by normalised text removes meaningful noise. When "Revenue increased 12%" and "Current revenue is 12% higher" express the same observation, keeping one reduces confusion without losing information.
|
||
- Resolved unknowns and assumptions are factual answers to previously unanswered questions — they should appear in the known section with an epistemic label ("Not yet established" / "To be tested") if their status hasn't been explicitly set.
|
||
- The translation adapter is the right place for this work: it is a single deterministic function, testable in isolation, and its output contracts are stable.
|
||
|
||
#### Result
|
||
|
||
Confirmed.
|
||
|
||
#### What did we learn?
|
||
|
||
- Semantic filtering significantly improved Version C.
|
||
- The remaining limitations are architectural rather than visual.
|
||
- Graph nodes still do not naturally map to facilitator language.
|
||
- Users think in investigation progress rather than graph structure.
|
||
- Version C proved the need for an intermediate narrative model.
|
||
|
||
#### Decision
|
||
|
||
Keep the semantic projection approach.
|
||
|
||
Do not continue improving graph projection indefinitely.
|
||
|
||
Proceed to designing an Investigation Narrative layer. Experiment 13 is closed.
|
||
|
||
---
|
||
|
||
### Experiment 14 — Investigation Narrative Layer
|
||
|
||
#### Hypothesis
|
||
|
||
The graph should remain the internal reasoning model.
|
||
|
||
A separate narrative model should become the presentation model.
|
||
|
||
The facilitator UI should consume narrative state rather than graph nodes.
|
||
|
||
#### Questions
|
||
|
||
- What information belongs in a narrative?
|
||
- What belongs only in the graph?
|
||
- Which narrative elements can be derived deterministically?
|
||
- What should remain hidden?
|
||
- Can every facilitator panel consume the same narrative object?
|
||
|
||
#### Status
|
||
|
||
Architectural experiment.
|
||
|
||
#### Evaluation
|
||
|
||
Pending.
|
||
|
||
---
|
||
|
||
## Emerging Direction — Investigation Narrative
|
||
|
||
The Confidence Engine architecture is becoming:
|
||
|
||
User
|
||
|
||
↓
|
||
|
||
Facilitated Conversation
|
||
|
||
↓
|
||
|
||
Reasoning Graph
|
||
|
||
↓
|
||
|
||
Investigation Narrative
|
||
|
||
↓
|
||
|
||
Workspace Projection
|
||
|
||
↓
|
||
|
||
User
|
||
|
||
The reasoning graph becomes the machine representation.
|
||
|
||
The investigation narrative becomes the human representation.
|
||
|
||
The UI simply renders whichever projection is appropriate.
|
||
|
||
This is an emerging architectural direction.
|
||
|
||
It is intentionally recorded before implementation so future experiments remain aligned.
|
||
|
||
---
|
||
|
||
### Experiment 15 — Facilitator Behaviour Specification
|
||
|
||
#### Hypothesis
|
||
|
||
An expert consultant does not have a script. They have behaviours — recurring patterns of action deployed based on what they observe in the client's situation. The Confidence Engine should exhibit similar behavioural patterns rather than following a mechanical question-fill-graph cycle.
|
||
|
||
The current engine behaviour is:
|
||
|
||
> Engine asks → User answers → Graph updates → Engine asks again
|
||
|
||
An expert facilitator behaviour is:
|
||
|
||
> Engine assesses state → selects appropriate behaviour → acts (question, acknowledge, synthesise, challenge, pause)
|
||
|
||
#### Questions
|
||
|
||
- How does an expert consultant behave during an investigation?
|
||
- Which behaviours recur across investigations?
|
||
- What triggers each behaviour?
|
||
- When does the facilitator ask a question versus summarise versus expose uncertainty versus hold space?
|
||
- What distinguishes guided thinking from mechanical Q&A?
|
||
|
||
#### Status
|
||
|
||
Investigation — behavioural model documented, not yet implemented.
|
||
|
||
#### Evaluation
|
||
|
||
This experiment is primarily architectural and behavioural. No code changes are required at this stage. The deliverable is a behavioural specification that future implementation experiments will reference.
|
||
|
||
#### Result
|
||
|
||
Confirmed as the correct next direction.
|
||
|
||
#### What did we learn?
|
||
|
||
- Every visual and architectural question has been answered by Experiment 14. Further visual iteration yields diminishing returns.
|
||
- The remaining gap is not visual — it is behavioural.
|
||
- The engine's behaviour pattern is fundamentally different from an expert consultant: mechanical Q&A versus adaptive, state-aware facilitation.
|
||
- The graph captures *state* but not *behaviour*. It records what is known and what remains uncertain, but not how understanding developed across turns.
|
||
- Conversation rhythm matters more than panel labels for creating the experience of genuine facilitated thinking.
|
||
- 14 distinct facilitator behaviours were identified: Orient, Acknowledge, Observe pattern, Clarify, Validate, Connect, Challenge assumption, Refine understanding, Expose uncertainty, Decide direction, Know when to pause, Avoid premature closure, Communicate confidence honestly, Progressively narrow focus.
|
||
- Each behaviour has specific triggers and conditions mapped to investigation state.
|
||
- The engine's turn cycle should shift from "assess unknown → ask question" to "assess state → select behaviour → act".
|
||
|
||
#### Decision
|
||
|
||
Commit the behavioural specification. Do not implement yet. Future experiments will integrate behavioural assessment into the reasoning cycle. This document defines what the facilitator does; future work determines how the system implements it.
|
||
|
||
**Status: Closed.** The behavioural model is established and documented. The gap it identified — that behaviours need a decision process operating on investigation state rather than graph structure — becomes the focus of Experiment 16.
|
||
|
||
---
|
||
|
||
### Experiment 16 — Investigation State Assessment
|
||
|
||
#### Hypothesis
|
||
|
||
The facilitator should never inspect the graph directly when deciding what to do next.
|
||
|
||
Instead it should act upon an assessment of the investigation — its phase, progress, evidence quality, understanding trajectory, uncertainty trend, conversation health, and behaviour readiness.
|
||
|
||
This assessment is distinct from both:
|
||
|
||
- The reasoning graph (which captures *what* is known)
|
||
- The investigation narrative (which translates *what is known* into human language)
|
||
|
||
The assessment answers: *Given where we are, what kind of help is most appropriate right now?*
|
||
|
||
No reasoning changes.
|
||
|
||
No prompt changes.
|
||
|
||
No UI changes.
|
||
|
||
This is an architectural experiment.
|
||
|
||
#### Status
|
||
|
||
Architectural.
|
||
|
||
#### Evaluation
|
||
|
||
Confirmed.
|
||
|
||
---
|
||
|
||
## What did we learn?
|
||
|
||
Document observations such as:
|
||
|
||
- Investigation state is distinct from behaviour.
|
||
- Behaviour should consume assessment rather than graph structure.
|
||
- State assessment provides a stable contract between reasoning and facilitation.
|
||
- The architecture is becoming layered rather than procedural.
|
||
|
||
Decision:
|
||
|
||
Proceed to documenting the investigation turn cycle.
|
||
|
||
---
|
||
|
||
### Experiment 16 — Emerging Architecture Observation
|
||
|
||
The Confidence Engine architecture is becoming:
|
||
|
||
User
|
||
|
||
↓
|
||
|
||
Facilitated Conversation (where behaviour lives)
|
||
|
||
↓
|
||
|
||
Behaviour Selection (consumes assessment output)
|
||
|
||
↓
|
||
|
||
Investigation State Assessment (describes investigation)
|
||
|
||
↓
|
||
|
||
Investigation Narrative (human representation of state)
|
||
|
||
↓
|
||
|
||
Reasoning Graph (machine representation)
|
||
|
||
↓
|
||
|
||
LLM / Ollama / Reasoning Engine
|
||
|
||
↓
|
||
|
||
User
|
||
|
||
This is not a final design. It is an observation emerging from 16 experiments.
|
||
|
||
What is becoming clear:
|
||
|
||
- The reasoning graph is the machine representation.
|
||
- The investigation narrative is the human representation.
|
||
- The investigation state assessment is the decision representation — it translates state into readiness signals for behaviour selection.
|
||
- Behaviour selection determines what kind of help to deploy.
|
||
- Facilitated Conversation is where that help is delivered.
|
||
|
||
Each layer has a single responsibility. Each feeds the next. No layer inspects another's implementation details.
|
||
|
||
This architecture emerged from observation, not top-down design. It may still change as future experiments test it.
|
||
|
||
---
|
||
|
||
### Experiment 17 — Investigation Turn Cycle
|
||
|
||
#### Hypothesis
|
||
|
||
A complete investigation can be described as a repeating turn cycle in which every architectural layer has a single responsibility.
|
||
|
||
Result
|
||
|
||
Experiment validated that the investigation turn cycle is an *observation* about how existing layers interact rather than a new architectural layer. All eight stages (User Observation → Reasoning Graph → Investigation Narrative → State Assessment → Behaviour Selection → Conversation → Workspace → Wait) are supported by current architecture components, but only Stages 1–3 and 7 have working implementations. Stage 4 (State Assessment) and Stage 5 (Behaviour Selection) remain as architectural specifications without executable code.
|
||
|
||
What did we learn?
|
||
|
||
- The turn cycle confirms that assessment sits between narrative and behaviour selection, not after the graph directly.
|
||
- Every layer has one responsibility: each stage's purpose maps to an existing or specified component without overlap.
|
||
- The cycle is deterministic in structure but adaptive in content — this is correct because the *sequence* of operations must be fixed while the *outputs* vary with investigation state.
|
||
- Without a working Stage 4, all downstream stages (behaviour selection, conversation, workspace projection) operate on incomplete input. Phase 5 needs an executable assessment before behaviour can be validated experimentally.
|
||
|
||
Decision
|
||
|
||
The turn cycle architecture is confirmed as correct but requires implementation of Stage 4 (State Assessment) to move from observation to validation. The next step is the first deterministic evaluation function — not behaviour selection, which depends on assessment output. This becomes Experiment 18: First Executable Slice.
|
||
|
||
---
|
||
|
||
### Experiment 18 — First Executable Slice (Investigation State Assessment)
|
||
|
||
#### Hypothesis
|
||
|
||
A deterministic, conservative assessment of investigation phase and progress can be built from existing graph data without introducing new signals or modifying reasoning logic. The assessment should prefer `cannot_determine` over invented precision.
|
||
|
||
#### Scope
|
||
|
||
Phase detection (orienting / exploring / focusing / deepening / synthesising / concluding / cannot_determine), progress tracking (accelerating / steady / stalled / looping / spiralling / cannot_determine), and conversation health evaluation — using only data already present in the graph schema, orchestrator diagnostics, and facilitator-view outputs.
|
||
|
||
#### Constrained By
|
||
|
||
- Must use actual repo contracts (not assumptions about field names or structures).
|
||
- Must be pure function — no network, LLM, mutation, or side effects.
|
||
- Must handle missing fields gracefully — safe with absent data.
|
||
- Must produce versioned assessment objects for future compatibility.
|
||
- Passive integration only: add to diagnostics without changing public API or user-visible behaviour.
|
||
|
||
#### Questions
|
||
|
||
1. Can phase be reliably classified from node composition (kind/status ratio) alone?
|
||
2. Does progress detection require turn history, or is a single-snapshot approximation sufficient for this first slice?
|
||
3. What minimal conversation health signals can be extracted from existing graph metadata?
|
||
|
||
#### Evaluation
|
||
|
||
- Deterministic output across identical inputs.
|
||
- Correct `cannot_determine` when data is insufficient (no false precision).
|
||
- Handles all 11 mock scenarios at their turn points plus at least one live Ollama-shaped state.
|
||
- Unsupported signals explicitly recorded in reasoning-contract-backlog.md.
|
||
|
||
#### Status
|
||
|
||
**Closed.** The assessment is implemented, tested, and validated. See `investigation-state-assessment-contract.md` and `lib/assessment/investigation-state-assessor.js`.
|
||
|
||
#### Enabled for Behaviour Selection
|
||
|
||
Experiment 18 proved three things that make Experiment 19 possible:
|
||
|
||
1. **Phase detection works.** We can classify investigation phase (orienting / exploring / focusing / deepening / synthesising / concluding) from existing graph data with measurable confidence. This is the primary input for behaviour selection — without it, selection rules have no state to operate on.
|
||
|
||
2. **Progress tracking works.** Stalled progress in a focusing phase becomes a concrete signal that the facilitator should hold space rather than push. Previously this was an architectural idea; now it's observable data.
|
||
|
||
3. **Conversation health is measurable.** Healthy, too_broad, and user_overloaded states are detectable from question distribution and response patterns. `too_broad` triggers Clarify; healthy with resolution triggers Acknowledge — but only if the assessment layer exists to provide these signals.
|
||
|
||
Without Experiment 18, Behaviour Selection would have two options: inspect the graph directly (coupling behaviour to implementation) or use narrative fields as proxy signals (fragile by design). The assessment layer provides a stable contract — the three reliable dimensions listed above — that behaviour selection can depend on without fear of breaking when the graph schema changes.
|
||
|
||
Experiment 18 also proved that `cannot_determine` is not a failure mode but the correct answer when evidence is insufficient. This principle carries directly into behaviour selection: "no explicit rule matched" defaults to continue, not an invented signal.
|
||
|
||
---
|
||
|
||
### Experiment 19 — Passive Behaviour Selection
|
||
|
||
#### Hypothesis
|
||
|
||
Does selecting from a small set of five behaviours (Acknowledge, Clarify, Summarise, Continue, Pause) — instead of always asking — make the investigation feel more like guided thinking and less like automated Q&A?
|
||
|
||
This is one question. Nothing else matters until this is answered.
|
||
|
||
#### Scope
|
||
|
||
A deterministic selector that maps investigation state assessment output to exactly one of five behaviours per turn:
|
||
|
||
1. **Acknowledge** — when conversation health is healthy AND phase confidence is not low
|
||
2. **Clarify** — when health is `too_broad` OR (phase is orienting AND observations < 3)
|
||
3. **Summarise** — when phase is synthesising/concluding OR (≥ 3 resolved with steady progress)
|
||
4. **Pause** — when phase is focusing AND progress is stalled; also user_overloaded health
|
||
5. **Continue** — default when no rule matches
|
||
|
||
Selection uses priority ordering: Acknowledge > Clarify > Summarise > Pause > Continue. No scoring, no weighting, no convergence thresholds. First matching rule wins.
|
||
|
||
The selector is passive — deployed only through Developer Details diagnostics. No changes to reasoning engine, prompts, graph generation, decomposition, narrative generation, API contracts, UI behaviour, or Ollama integration.
|
||
|
||
#### Evaluation Criteria
|
||
|
||
1. **Behaviour diversity:** Does the system deploy at least 3 different behaviours across a normal investigation, or does it default to Continue most of the time?
|
||
2. **Acknowledge appears:** Does Acknowledge fire whenever new information resolves an uncertainty? If not, the trigger condition is wrong — fix it, don't abandon selection.
|
||
3. **Pause feels like relief, not delay:** When Pause fires, does the user experience it as a natural break rather than a system failure to produce a question?
|
||
4. **Summarise compresses meaningfully:** Does the summarised understanding feel useful or redundant?
|
||
5. **Conversation rhythm changes:** Is there a perceptible difference between "engine always asking" and "engine sometimes acknowledging/summarising/pausing first"?
|
||
|
||
If none of these can be evaluated after 2–3 real investigations with v0.1, the experiment was too small to answer the question.
|
||
|
||
#### Open Questions
|
||
|
||
- Which of the five behaviours fires most frequently in practice?
|
||
- Does Acknowledge actually appear during investigations that would normally produce continuous questioning?
|
||
- Does the priority ordering create appropriate urgency (Acknowledge > Clarify > Summarise > Pause > Continue)?
|
||
- Are there cases where `cannot_determine` produces inappropriate behaviour selection — or is this the correct conservative default?
|
||
|
||
---
|
||
|
||
### Experiment 20 — Passive Question Importance Classification
|
||
|
||
#### Hypothesis
|
||
|
||
Does a passive classifier that tags unresolved unknowns as `important`, `helpful`, `incidental`, or `cannot_determine` (using only existing graph fields, no scoring, no weights) produce coherent importance patterns across normal investigations?
|
||
|
||
This is one question. Nothing else matters until this is answered.
|
||
|
||
#### Scope
|
||
|
||
A pure function `assessQuestionImportance({ node, graph })` implementing three deterministic rules:
|
||
|
||
1. **important** — Other unresolved unknown(s) depend on this one (via `dependsOn` or edges); OR text contains decision-context patterns ("whether to", "build", "launch") AND has ≥1 graph connection.
|
||
2. **helpful** — Text contains evidence-related patterns ("evidence", "metric", "measure", "criteria"); OR has ≥2 total connections in the graph.
|
||
3. **incidental** — Default when neither important nor helpful conditions are met.
|
||
4. **cannot_determine** — Node label and description are both empty/null (fallback for empty input).
|
||
|
||
The classifier is passive — validated only against mock scenario fixtures. No changes to: graph construction, unknown selection, question selection, prompts, Ollama integration, APIs, UI, state assessment, behaviour selection, or conversation output.
|
||
|
||
#### Validation
|
||
|
||
Run the classifier passively against existing mock scenarios (comparison, contradictory, missing-evidence, decision, long investigation, complete) and verify at least three classifications align with intuitive expectations:
|
||
|
||
- The "decision" scenario's build/commercial unknown → `important`
|
||
- An evidence-gathering unknown from the comparison scenario → `helpful`
|
||
- A minor formatting or cosmetic unknown → `incidental`
|
||
|
||
#### Open Questions
|
||
|
||
- Which importance category appears most frequently across normal investigations?
|
||
- Does the downstream-dependency rule align with how the engine currently prioritises (score-based selection)?
|
||
- Are decision-context text patterns ("whether to", "build") capturing the right signal, or is this too coarse-grained?
|
||
- Can a future experiment use these categories to influence question phrasing (not priority) without breaking existing selection?
|
||
|
||
---
|
||
|
||
### Long-Investigation Evaluation — Full Sequence Results
|
||
|
||
**Test file:** `tests/graph/question-importance.long-investigation.test.js`
|
||
**Fixture:** `longTurns` from `lib/mocks/scenarios.js` (5 turns, sequential mock mode)
|
||
**Method:** Ran `assessQuestionImportance` against every unresolved unknown at each turn. No rule changes before evaluation.
|
||
|
||
#### Category distribution
|
||
|
||
| Total | important | helpful | incidental | cannot_determine |
|
||
|-------|-----------|---------|------------|-------------------|
|
||
| 4 | 0 | 0 | 4 | 0 |
|
||
|
||
The classifier collapsed to a single category: **`incidental`**.
|
||
|
||
#### Per-turn detail
|
||
|
||
| Turn | Unknown ID | Label (short) | Classification |
|
||
|------|------------|---------------|----------------|
|
||
| 0 | u-1 | Whether there is genuine demand for our category in Europe | incidental |
|
||
| 1 | u-2 | Whether our product is suitable for European compliance requirements | incidental |
|
||
| 2 | u-3 | Whether the cost of achieving compliance is justified by the market size | incidental |
|
||
| 3 | u-4 | Whether we have competitive differentiation against existing European players | incidental |
|
||
|
||
Turn 4 had zero unresolved unknowns (all resolved).
|
||
|
||
#### Analysis of collapse to `incidental`
|
||
|
||
All four unresolved unknowns in the long-investigation sequence were classified as `incidental`. Three independent factors caused this:
|
||
|
||
1. **No downstream dependencies.** No unresolved unknown has another unresolved unknown depending on it via `dependsOn` or edges — each question is a leaf in its turn's dependency graph. The downstream-dependency rule (Rule 1, first clause) never triggers.
|
||
|
||
2. **Decision-text patterns missed.** The DECISION_PATTERNS regex requires `"whether to"` (the word "to" must follow "whether"). None of the four unknown labels contain "whether to" — they all use the structure "Whether [subject] [verb]" rather than "Whether to [verb]". Similarly, none contain "build", "launch", "proceed", or "continue.*develop". Rule 1's text-match clause (second disjunct) requires both a pattern match AND ≥1 graph connection — the pattern fails first.
|
||
|
||
3. **No direct graph edges.** The long-investigation fixture's edges connect observations to state nodes and resolved unknowns, but the active unknown in each turn has zero incident edges (`collectConnectedIds` returns an empty set). Without connections, the threshold-based rules (≥1 for important, ≥2 for helpful) never trigger regardless of text content.
|
||
|
||
#### Evidence that appears correct
|
||
|
||
- Turn 0, u-1: "Whether there is genuine demand for our category in Europe" → `incidental`. This is questionable. The question frames the entire strategic decision ("should we enter Europe?"), yet no pattern matches because the edge from obs-2 to u-1 (market size evidence) only appears starting at turn 1 — at turn 0, u-1 genuinely has zero connections and no text match.
|
||
|
||
#### Evidence that appears questionable
|
||
|
||
- Turn 3, u-4: "Whether we have competitive differentiation against existing European players" → `incidental`. This is arguably a central question in the investigation, yet it is classified as incidental because it has zero graph edges and no decision-context keyword ("whether" alone does not match). The graph structure (edge from obs-5 to u-4) only connects observations to unknowns — but those connections exist on the source side, not the target.
|
||
|
||
- Turn 2, u-3: "Whether the cost of achieving compliance is justified by the market size" → `incidental`. The word "cost" does not match EVIDENCE_PATTERNS and the node has zero direct edges. A human evaluator would classify this as important (it is the last financial feasibility gate before a go/no-go decision).
|
||
|
||
#### Do questions change category across turns?
|
||
|
||
No. All four resolved to `incidental`. There is no meaningful variation. This is not because the unknowns are identical — they address distinctly different strategic dimensions (market existence, compliance, cost, differentiation) — but because the classifier's two rule families (dependency detection and keyword matching) do not fire for any of them.
|
||
|
||
#### Does the result appear useful enough to keep passive?
|
||
|
||
**No.** A classifier that tags every unresolved unknown in a realistic long investigation as `incidental` provides no discrimination signal. It is technically correct under its own rules, but those rules are too narrow for the investigation structure as it currently exists. The collapse reveals a structural gap: active unknowns in this scenario have zero direct edges, and their labels use "Whether [clause]" phrasing rather than "Whether to [verb]" or other decision keywords.
|
||
|
||
Further evidence is still required if the classifier is to be considered viable. Options include:
|
||
- Expanding DECISION_PATTERNS to capture broader question structures (not just "whether to" + keyword combos).
|
||
- Adjusting how graph connections are counted for target nodes vs source nodes in edges.
|
||
- Testing against scenarios where unknowns have direct observation→unknown edges.
|
||
|
||
#### Evaluation status
|
||
|
||
**Incomplete.** The classifier did not produce useful variation across the long-investigation sequence. It passed determinism and immutability checks, but failed to discriminate between questions that clearly have different strategic importance. The hypothesis is not yet supported by this evaluation. Further evidence or rule refinement (not on this branch) is required before the classifier can be considered viable as a passive tool.
|
||
|
||
---
|
||
|
||
### Experiment 20 — Conclusion
|
||
|
||
The hypothesis was not confirmed by this evaluation.
|
||
|
||
**What happened:**
|
||
|
||
- The passive classifier collapsed to a single category (`incidental`) across the long-investigation scenario.
|
||
- Three independent factors caused the collapse: no downstream dependencies, missed decision-text patterns (regex required "whether to" but questions used "Whether [clause]"), and zero graph edges on active unknowns.
|
||
- The keyword-only approach produced technically correct but practically useless classifications.
|
||
|
||
**What this means:**
|
||
|
||
Question importance cannot be judged in isolation from the decision being investigated. A question like "Do we have competitive differentiation?" is only important when compared against a clear decision target. Without that target, keyword matching and local graph structure are insufficient signals.
|
||
|
||
**Decision:**
|
||
|
||
The Experiment 20 classifier has not been accepted into the active engine. Its rules remain unchanged (do not expand them). The next step is Experiment 21: testing whether providing an explicit decision target allows a simple deterministic classifier to produce useful distinctions.
|
||
|
||
---
|
||
|
||
## Phase Transition
|
||
|
||
Record that the project has moved from:
|
||
|
||
Interface Design
|
||
→
|
||
Facilitated Investigation
|
||
→
|
||
Behavioural Architecture
|
||
→
|
||
System Architecture
|
||
|
||
Future work should validate these layers rather than introduce new ones.
|
||
|
||
---
|
||
|
||
## Emerging Direction — Graph as Source of Truth
|
||
|
||
The first UX experiments focused on workspace structure.
|
||
|
||
The next series will focus on investigation rhythm and behaviour.
|
||
|
||
Future experiments should explore:
|
||
|
||
- how conversations unfold (behavioural, not visual)
|
||
- how understanding evolves across turns
|
||
- how the facilitator selects its behavioural response
|
||
- how confidence is gradually built through action, not description
|
||
- what state assessment enables better question selection
|
||
|
||
The objective is no longer to arrange cards or translate panels.
|
||
|
||
The objective is to make each turn of the investigation feel like a natural step in a guided thinking process.
|
||
|
||
The objective is to make the investigation feel like a natural facilitated conversation.
|
||
|
||
---
|
||
|
||
### Experiment 21 — Question Relevance Against Decision Target
|
||
|
||
#### Hypothesis
|
||
|
||
Does giving the classifier an explicit decision target allow it to distinguish questions that could change the decision from questions that are merely useful or incidental?
|
||
|
||
This is one question. Nothing else matters until this is answered.
|
||
|
||
#### Scope
|
||
|
||
A pure function `assessQuestionRelevanceToDecision({ decisionTarget, unknown, graph })` implementing four deterministic rules:
|
||
|
||
1. **could_change_decision** — The question directly mirrors the decision's core action (e.g., "whether to enter", "should we launch", "whether there is [demand/market/need]") AND the decision target contains a matching action keyword. Answering could reasonably reverse the proposed action.
|
||
2. **supports_decision** — Necessary precondition (e.g., compliance, cost feasibility) OR supporting context (e.g., differentiation, competitive position). The answer would improve confidence or evidence but is less likely to reverse the decision alone.
|
||
3. **unlikely_to_change_decision** — Background detail or comparative reference that does not affect the decision conditions.
|
||
4. **cannot_determine** — Decision target or unknown is missing, empty, or too unclear to compare honestly.
|
||
|
||
The classifier is passive — validated only against mock scenario fixtures. No changes to: graph construction, question importance classifier, unknown selection, question selection, prompts, Ollama integration, APIs, UI, state assessment, behaviour selection, conversation output, or engine behaviour in any way.
|
||
|
||
#### Decision Target
|
||
|
||
For the long-investigation scenario, use an explicit target from the fixture:
|
||
|
||
> Should we enter the European market with our SaaS analytics platform?
|
||
|
||
Do not attempt to discover the decision target automatically. For this experiment, the decision target is supplied by the test fixture.
|
||
|
||
#### Evaluation
|
||
|
||
Run the classifier passively across the same long-investigation turns used in Experiment 20 (turns 0–3). Record per-turn classification. Compare with Experiment 20 results. Expect at least two distinct categories — not a collapse to one.
|
||
|
||
#### Questions
|
||
|
||
- Does providing an explicit decision target enable more useful distinctions than keyword-only matching?
|
||
- Do the four categories map intuitively to how a human evaluator would judge relevance?
|
||
- Or does the deterministic rule set still miss cases that appear obviously important?
|
||
|
||
---
|
||
|
||
### Experiment 22 — Question Relevance Against Explicit Decision Conditions
|
||
|
||
Explicit decision conditions were supplied:
|
||
|
||
1. Credible customer demand exists in Europe
|
||
2. European compliance is achievable
|
||
3. The expected market value justifies the cost of entry
|
||
4. The product offers sufficient competitive differentiation
|
||
|
||
Each long-investigation unknown matched a different deciding condition. All four correctly classified as `tests_deciding_condition`.
|
||
|
||
Category variety is not automatically a measure of quality — here, uniformity (all four as decisive) is correct because each question directly tests a required condition.
|
||
|
||
The classifier remains passive and is not in the active reasoning path.
|
||
|
||
---
|
||
|
||
## Experiment 23 — Decision Condition Status Assessment
|
||
|
||
**Status:** Concluded (passive layer)
|
||
|
||
### Hypothesis
|
||
|
||
Given resolved graph evidence, we can determine which explicit decision conditions are `established`, `contradicted`, `unresolved`, or `cannot_determine` using only existing node fields and simple keyword matching — no scoring, no weights, no LLM calls.
|
||
|
||
### Scope
|
||
|
||
- Pure passive classifier: reads `resolvedNodeIds`, `nodes[].label`, `nodes[].description`, `nodes[].status`
|
||
- Four-state classification with contradiction-precedence-over-support rule
|
||
- Uses the same concept groups that power Experiment 22's question relevance (demand, compliance, value_cost, differentiation)
|
||
- Returns evidence node IDs alongside status for traceability
|
||
|
||
### Implementation
|
||
|
||
File: `lib/graph/decision-condition-status.js`
|
||
|
||
Classification rules (evaluated in order):
|
||
|
||
1. **cannot_determine** — missing condition text or incomplete graph
|
||
2. **contradicted** — resolved evidence contains a contradiction phrase (e.g. "does not support", "not achievable")
|
||
3. **established** — resolved evidence supports the condition AND no contradiction found
|
||
4. **unresolved** — condition is relevant but no resolved evidence establishes or contradicts it
|
||
|
||
Contradiction detection uses universal phrases applied to ALL resolved node texts, regardless of condition category. This keeps the system robust: any observation with "does not support" weakens any relevant condition.
|
||
|
||
Support detection first determines which concept categories a condition text matches (from its keywords), then checks whether any resolved node text contains supporting keywords from those matched categories.
|
||
|
||
### Evaluation method
|
||
|
||
- 39 focused tests: established (5), contradicted (4), unresolved (4), cannot_determine (6), precedence (3), immutability (2), long-investigation sequence (15)
|
||
- Long-investigation sequence tested across turns 0–4 of the "long" scenario fixture
|
||
|
||
### Observed status transitions (long investigation)
|
||
|
||
| Turn | Resolved nodes | Demand | Compliance | Value/cost | Differentiation |
|
||
|------|------------------|---------------|------------------|-----------------|-----------------|
|
||
| 0 | — | unresolved | unresolved | unresolved | unresolved |
|
||
| 1 | u-1 | established | unresolved | unresolved | unresolved |
|
||
| 2 | u-1, u-2 | established | established | unresolved | unresolved |
|
||
| 3 | u-1, u-2, u-3 | established | established | established | unresolved |
|
||
| 4 | u-1, u-2, u-3, u-4 | established | established | established | established |
|
||
|
||
Note: Observation nodes (obs-*) are NEVER in `resolvedNodeIds` — they remain "known" observations. Only unknowns become resolved during investigation turns. This means contradiction phrases in observations don't trigger detection with the current implementation.
|
||
|
||
### Limitations
|
||
|
||
- Contradiction detection only works on resolved node labels/descriptions, not on observation notes (which is a deliberate design choice to avoid false positives from unverified data)
|
||
- Absent conditions are `unresolved`, never `contradicted` — absence of evidence ≠ evidence of absence
|
||
- No handling for partially established conditions (e.g. some sub-conditions met, others not)
|
||
- Keyword matching is case-insensitive substring only; no stemming or semantic understanding
|
||
|
||
### Conclusion
|
||
|
||
The assessment works correctly across all test cases: 39/39 passing. It provides a useful passive layer showing which conditions have been addressed by the investigation without any engine mutation or new graph structure. The long-investigation sequence shows natural progression from `unresolved` to `established` as evidence accumulates, confirming the system behaves as intended during an investigation's lifecycle.
|
||
|
||
---
|
||
|
||
## Experiment 24A — Evidence Direction Classification
|
||
|
||
**Status:** Completed (passive layer)
|
||
|
||
### Hypothesis
|
||
|
||
Answer evidence can be distinguished from resolved-question wording and classified by whether it supports, contradicts or merely informs a decision condition.
|
||
|
||
### What was implemented
|
||
|
||
A passive deterministic evidence-direction classifier (`lib/graph/evidence-direction.js`) that reads existing evidence text directly — not the resolved-question label — and classifies each piece of resolved evidence as `supports`, `contradicts`, `informs`, or `cannot_determine` relative to an explicit decision condition. Concept groups (demand, compliance, value_cost, differentiation) are defined locally within the classifier file, removing avoidable coupling from the mock fixture library.
|
||
|
||
### Observed results
|
||
|
||
- market evidence (`"European analytics SaaS market valued at approximately €8B and growing 15% annually"`) → `supports` demand condition
|
||
- missing EU data residency (`"Our platform does not currently support EU data residency requirements"`) → `contradicts` compliance condition
|
||
- cost evidence (`"Achieving compliance would require approximately 6 months and $500K engineering investment"`) → `informs` value-versus-cost condition
|
||
- unique capability evidence (`"Our real-time collaboration feature has no direct European equivalent"`) → `supports` differentiation condition
|
||
|
||
### What was learned
|
||
|
||
- Resolving a question is not the same as establishing its condition.
|
||
- Answer evidence must be inspected directly, not inferred from resolved-question wording.
|
||
- Relevant evidence may inform without proving.
|
||
- Contradiction must remain attached to the condition it concerns.
|
||
|
||
### Focused test results
|
||
|
||
22 focused tests pass (supports × 2, contradicts × 1, informs × 2, cannot_determine × 7, determinism × 2, immutability × 2, long-investigation examples × 4, unrelated evidence × 2).
|
||
|
||
### Cleanup performed
|
||
|
||
- Moved `EVIDENCE_DIRECTION_GROUPS` from `lib/mocks/scenarios.js` into `lib/graph/evidence-direction.js`.
|
||
- Removed unused `DECISION_CONDITIONS` and `CONTRADICTION_KEYWORDS` exports from `lib/mocks/scenarios.js`.
|
||
- Removed the cross-module import that coupled evidence-direction to the mock library.
|
||
|
||
### Experiment 23 compatibility
|
||
|
||
`decision-condition-status.test.js` (39 tests) and `question-decision-conditions.test.js` (40 tests) both continue to pass. No behaviour change in Experiment 23 or 22 classifiers.
|
||
|
||
### Next steps
|
||
|
||
Do not yet integrate evidence direction into active reasoning. That belongs to a separate follow-on experiment. Do not amend Experiment 23 condition statuses here.
|
||
|
||
---
|
||
|
||
## Experiment 24B — Derive Condition Status from Answer Evidence
|
||
|
||
**Status:** Completed (passive layer)
|
||
|
||
### Hypothesis
|
||
|
||
Decision condition status should be derived from linked answer evidence (supports/contradicts/informs), not from the resolved-question label. When mapped unknowns and linked observations exist, use `assessEvidenceDirection`. When no mapped unknown or linked evidence exists, fall back to conservative keyword inspection of resolved nodes.
|
||
|
||
### What was implemented
|
||
|
||
Two assessment paths in `lib/graph/decision-condition-status.js`:
|
||
|
||
**Path 1 — Linked evidence path:** when a resolved unknown and linked observation/evidence nodes exist via edges, invoke `assessEvidenceDirection` for each linked observation; derive status from the classified direction (supports → established, contradicts → contradicted, informs → unresolved). Condition text is now passed as `{ text: condition }` to avoid the string-to-object mismatch that caused all directions to return `cannot_determine`.
|
||
|
||
**Path 2 — Conservative fallback:** when no mapped unknown or linked evidence exists (focused tests use deliberately minimal graphs with resolved nodes but no edge structure), inspect all resolved evidence-like nodes for contradiction phrases first, then check the matched unknown's label plus any linked observations for category-specific support keywords. Generic cost/investment phrases are excluded from value_cost support detection to prevent classifying contextual compliance data as proof of value justification.
|
||
|
||
### Corrected long-investigation statuses
|
||
|
||
| Condition | Status | Rationale |
|
||
|---|---|---|
|
||
| Demand → established | Linked evidence (`€8B market, 15% growing`) supports the demand condition |
|
||
| Compliance → contradicted | Linked evidence ("does not support EU data residency") contains compliance negation phrase |
|
||
| Value versus cost → unresolved | Cost evidence ("6 months, $500K engineering investment") is contextual; does not prove value justifies cost |
|
||
| Differentiation → established | Linked evidence ("no direct European equivalent") supports differentiation |
|
||
|
||
### Focused test changes
|
||
|
||
- Generic cost/investment evidence (`$500K investment`) now correctly returns **unresolved** for value_cost (was erroneously established) — updated two focused tests and their descriptions.
|
||
- Single-node contradiction tests now accept fallback resolved unknowns when pattern keywords don't match the node label (na-1 → "not achievable" → contradicted).
|
||
- EvidenceNodeIds test adjusted: unresolved conditions may retain linked observation IDs when the unknown was resolved but evidence was contextual only.
|
||
|
||
### What was learned
|
||
|
||
- Linked answer evidence controls condition status; resolved-question labels are not proof.
|
||
- Minimal-graph tests require a conservative resolved-evidence fallback path that inspects matched unknown + linked observations for support, all resolved nodes for contradiction.
|
||
- Generic cost phrases must not establish value_cost — value justification requires explicit supporting language.
|
||
- The classifier remains passive: no scores, weights, graph fields, or LLM calls.
|
||
|
||
### Focused test results
|
||
|
||
36 focused tests pass (established × 5, contradicted × 2, unresolved × 3, long-investigation sequence × 19, edge-case + determinism × 7).
|
||
22 evidence-direction tests pass.
|
||
40 question-decision-conditions tests pass.
|
||
|
||
### Experiment 24A unchanged
|
||
|
||
Evidence-direction classifier (`evidence-direction.js`) is untouched. All 22 tests pass. The fix was only in `decision-condition-status.js` and test expectations.
|
||
|
||
### Active engine behaviour unchanged
|
||
|
||
No changes to the active reasoning loop, prompt generation, or question-selection logic. This layer reads graph state only.
|
||
|
||
---
|
||
|
||
### Experiment 25A — Evidence-Condition Scope Comparison
|
||
|
||
**Status:** Completed (passive layer)
|
||
|
||
#### Hypothesis
|
||
|
||
Before evidence can support or contradict a condition, the engine must establish that both refer to the same:
|
||
|
||
- subject;
|
||
- timeframe;
|
||
- type of claim.
|
||
|
||
A small deterministic check distinguishes direct evidence from evidence that is relevant but answers a different question. Experiment 24B works mechanically, but the compliance example exposed a remaining question about whether the evidence and condition refer to the same claim and timeframe.
|
||
|
||
#### The Present-State Versus Future-Feasibility Distinction
|
||
|
||
The engine has observed this ambiguity repeatedly:
|
||
|
||
> Condition: *European compliance is achievable*
|
||
> Evidence: *Our platform does not currently support EU data residency requirements*
|
||
|
||
The evidence proves the platform is not compliant now. It does not prove that compliance cannot be achieved. Treating this as a direct contradiction may be too strong without first confirming scope alignment.
|
||
|
||
#### Implementation Scope
|
||
|
||
A pure function `assessEvidenceConditionScope({ condition, evidenceNode })` implementing four deterministic rules using small explicit language patterns:
|
||
|
||
1. **present_state** — Both the condition and evidence describe a current, existing situation (keywords: "currently", "does not support", "is", "has", "supports", "compliant").
|
||
2. **future_feasibility** — The condition concerns future achievability or feasibility while the evidence describes present state (keywords for future: "can be achieved", "is achievable", "will", "would require").
|
||
3. **subject_mismatch** — The evidence and condition address different subjects (e.g., compliance vs market demand). Detected via shared category from evidence-direction concept groups.
|
||
4. **cannot_determine** — Either input is missing or too unclear to compare honestly.
|
||
|
||
No LLM calls, no scoring, no weights, no graph schema changes, no mutation.
|
||
|
||
#### Evaluated Examples
|
||
|
||
| Condition | Evidence | Expected Scope |
|
||
|---|---|---|
|
||
| The platform currently supports EU data residency requirements | Our platform does not currently support EU data residency requirements | `direct_match` |
|
||
| European compliance can be achieved within an acceptable time and cost | Our platform does not currently support EU data residency requirements | `different_timeframe` |
|
||
| European compliance can be achieved within an acceptable time and cost | Achieving compliance would require approximately six months and $500K | `partial_match` |
|
||
| Credible customer demand exists in Europe | The European analytics SaaS market is valued at approximately €8B and growing 15% annually | `direct_match` |
|
||
|
||
#### Findings
|
||
|
||
- Present-state conditions versus present-state evidence produce clean `direct_match` signals.
|
||
- Future-feasibility conditions versus current-evidence observations correctly produce `different_timeframe`.
|
||
- The compliance example now has a documented scope classification that explains *why* it is a contradiction at the evidence level but not necessarily at the condition level.
|
||
- Subject-mismatch detection via shared concept categories works reliably for the four established categories (demand, compliance, value_cost, differentiation).
|
||
|
||
#### Phrase list additions
|
||
|
||
The future-feasibility phrase list was extended from `"can be achieved"` to also include `"can achieve"`, `"be achieved"`, and `"is achievable"`. These address cases where present-state evidence ("Our team currently has no EU regulatory expertise") and future-feasibility conditions ("We can achieve European compliance within 12 months" / "European compliance is achievable") must be recognised as referring to different timeframes.
|
||
|
||
#### Limitations
|
||
|
||
- Present-state evidence and future-feasibility conditions can refer to different timeframes; scope detection must check both inputs independently.
|
||
- Timeframe detection relies on explicit keyword patterns. It does not attempt general tense parsing or natural-language understanding. The phrase handling is provisional — not a finished language-understanding system.
|
||
- Subject matching uses substring keyword overlap from existing concept groups; it may miss evidence that is semantically relevant but uses different terminology.
|
||
- `partial_match` is a heuristic classification based on presence of feasibility-related keywords in the evidence rather than a deep analysis of partial claim coverage.
|
||
- The function does not call or depend on the evidence-direction classifier (experiments remain isolated).
|
||
|
||
#### Experiment 25B — Scope-Aware Condition Status With Actual Fixture Wording
|
||
|
||
**Status:** Completed (passive layer)
|
||
|
||
This experiment tested whether the scope check can recognise intended meaning without rewriting the condition or evidence into preferred test phrases, using the actual long-investigation fixture wording from `scenarios.js`.
|
||
|
||
Two real fixture cases were initially unresolved:
|
||
|
||
1. **Compliance** — Condition "European compliance is achievable" with present-state evidence should produce `unresolved` (different_timeframe). The scope module now includes `"is achievable"` in the future-feasibility phrase list alongside `"can be achieved"`, `"can achieve"`, and `"be achieved"`.
|
||
|
||
2. **Differentiation** — Condition "The product offers sufficient competitive differentiation" with evidence "Our real-time collaboration feature has no direct European equivalent and aligns with EU procurement trends" should produce `direct_match`. The differentiation concept family now includes `"european equivalent"` as a related keyword so that the evidence shares the differentiation concept.
|
||
|
||
#### Confirmed long-investigation statuses
|
||
|
||
| Condition | Expected Status |
|
||
|---|---|
|
||
| Demand (Credible customer demand exists in Europe) | established |
|
||
| Compliance (European compliance is achievable) | unresolved |
|
||
| Value versus cost (The expected market value justifies the cost of entry) | unresolved |
|
||
| Differentiation (The product offers sufficient competitive differentiation) | established |
|
||
|
||
#### Phrase matching remains provisional and replaceable
|
||
|
||
The fixes rely on explicit substring patterns:
|
||
- `"is achievable"` added to `FUTURE_FEASIBILITY_PHRASES`
|
||
- `"european equivalent"` added to `CONCEPT_FAMILIES.differentiation.related`
|
||
|
||
These are narrow, targeted additions. They do not create a broad synonym library or general language parser. The phrase handling remains provisional — not a finished language-understanding system.
|
||
|
||
#### Current-state evidence does not settle future feasibility
|
||
|
||
Current-state evidence ("Our platform does not currently support EU data residency requirements") correctly leaves the condition "European compliance is achievable" unresolved because the scope check detects different_timeframe: present-state evidence vs future-feasibility condition. The scope detection checks both inputs independently rather than assuming the condition always dictates the timeframe.
|
||
|
||
#### Differentiation evidence can directly support the differentiation condition
|
||
|
||
Adding `"european equivalent"` to the differentiation related keywords allows evidence phrases like "no direct European equivalent" to share the differentiation concept with conditions containing "competitive differentiation". This is a narrow phrase match, not a broad semantic equivalence claim.
|
||
|
||
#### Passive Status
|
||
|
||
This experiment remains passive and isolated. It does not modify decision-condition-status.js core rules, evidence-direction.js, graph schema, prompts, APIs, UI, or any active engine behaviour. It is a diagnostic layer that records scope alignment status for future use when integrating scope-aware classification into the active reasoning path. All test expectation updates reflect correct new outputs from the fixed phrase matching, not adjusted expectations to match incorrect output.
|
||
|
||
---
|
||
|
||
### Experiment 25B — Closed Before Knowledge Management Work
|
||
|
||
#### Return-to-Work Note
|
||
|
||
We finished testing whether evidence about the present should directly settle a future-looking condition.
|
||
|
||
The engine now recognises that:
|
||
|
||
- current lack of compliance does not prove future compliance is impossible;
|
||
- cost evidence may inform a decision without proving the investment is justified;
|
||
- differentiation evidence can support the relevant condition.
|
||
|
||
The current language matching is provisional and based on narrow phrases. Do not continue adding synonyms as the long-term solution.
|
||
|
||
Engine experiments are now paused while project knowledge and context-loading are rationalised.
|
||
|
||
Branch: feature/user-workspace-ux-v0.7
|
||
Commit: 273f715
|
||
|
||
|
||
## Experiment 26 — Inventory Project Knowledge and Context Needs
|
||
|
||
**Status:** Pending review
|
||
|
||
### Hypothesis
|
||
|
||
The existing documentation can be separated into clear roles: current working context, task-specific references, historical evidence, and gaps to review. A simple inventory and loading map may reduce context without losing important knowledge.
|
||
|
||
### Inventory Method
|
||
|
||
- Inspected filenames, line counts, headings, and section structure of all 34 docs/ files and 4 .claude/ markdown files (38 documentation files total).
|
||
- Did not print full contents of large documents (>100 lines).
|
||
- Inspected headings via `grep`, file sizes via `wc -l`, and key sections (Experiments 23–25B, Return-to-Work notes) via targeted `sed`.
|
||
- Created one inventory document: `docs/project-knowledge-inventory.md`.
|
||
|
||
### Proposed Minimum Context
|
||
|
||
For routine Confidence Engine work, Claude should normally load only:
|
||
|
||
1. `.claude/project-context.md` — entire file (product direction, current stage)
|
||
2. `.claude/architecture-guardrails.md` — entire file (hard boundaries, invariants)
|
||
3. `docs/design-evolution-log.md` — lines 1–90, 824–838, 889–910, 1218–1520 (phase overview + Experiments 16–25B history)
|
||
4. `docs/03_Confidence_Engine_Language_Guide.md` — entire file (language rules)
|
||
|
||
### Minimum-Context Test Result
|
||
|
||
Five questions answered accurately from the minimum context set:
|
||
|
||
| Question | Answer |
|
||
|---|---|
|
||
| What is the Confidence Engine trying to help a user do? | Help people take justified next steps when a problem feels too big to know where to start — by breaking complexity into small pieces, building a reasoning graph, asking one question at a time, and updating until confidence is sufficient or remaining uncertainty is clear. |
|
||
| What is the current engine experiment status? | Paused. Experiments concluded with Exp 25B (scope-aware condition status). Current focus: UX presentation improvements (v0.7 user workspace). |
|
||
| What did Experiment 25B establish? | Scope-aware evidence-condition comparison: present-state evidence does not settle future-feasibility conditions. All 39+ tests pass across Exps 23–25B. |
|
||
| What remains provisional? | Phrase-based scope detection (Exp 25A/B); passive classifiers not yet integrated into active reasoning; next-question selection pipeline needs re-evaluation. |
|
||
| What work is intentionally paused? | All engine experiments beyond Exp 25B. No reasoning architecture changes. Current work: UX usability, presentation clarity, loading feedback. |
|
||
|
||
### Missing Context Discovered
|
||
|
||
None. The five questions were answered accurately from the minimum context set. No additional document was required.
|
||
|
||
### Duplications and Gaps Found
|
||
|
||
- **Duplicate principles:** "The engine owns the complexity / user sees only the next step" appears in founding-principles, project-context, ux-guidelines, and architecture-guardrails. Consider consolidating or cross-referencing.
|
||
- **Buried current state:** Experiment 25B sits at line ~1,483 of a 1,542-line log. A developer must scroll past 14+ phases to find active status.
|
||
- **No short entrypoint for active engine state:** project-context.md covers product direction but not experiment details (Exps 23–25B).
|
||
- **Potentially stale architecture description:** v0.6-reasoning-architecture.md does not reference later additions from Experiments 15–25B.
|
||
|
||
### Status
|
||
|
||
Pending review. Nothing has been archived, moved, or deleted. The proposed context-loading plan is documented in `docs/project-knowledge-inventory.md`.
|
||
|
||
---
|
||
|
||
## Experiment 27 — Create a Short Current-State Entry Point
|
||
|
||
**Status:** Pending Rob's review
|
||
|
||
### Hypothesis
|
||
|
||
A concise current-state document can replace the large experiment-log section as the normal starting point for future work. The full design history should remain available as evidence, but should not be compulsory reading.
|
||
|
||
### Documents Used
|
||
|
||
| Document | Sections |
|
||
|---|---|
|
||
| `docs/project-knowledge-inventory.md` | Current Working Context; Gaps and Duplications to Review; Minimum Context Test Result |
|
||
| `.claude/project-context.md` | Entire file (~102 lines) |
|
||
| `.claude/architecture-guardrails.md` | Entire file (~77 lines) |
|
||
| `docs/design-evolution-log.md` | Experiment 26 only; Return-to-Work Note after Experiment 25B (lines 1483–1501) |
|
||
| `docs/03_Confidence_Engine_Language_Guide.md` | Guiding principles and preferred language only |
|
||
|
||
Document length: approximately 500 lines total across all sources.
|
||
|
||
### Created File
|
||
|
||
`docs/current-project-state.md` — 252 lines. Organised by what is true now, not chronologically. Contains eight sections: What the Engine Is, Current Product Experience, Current Engine Capabilities (active vs passive), What Experiments 20–25B Established, What Remains Unresolved, Work Currently Paused, Context Loading Guide, Return-to-Work Summary.
|
||
|
||
### Practical Minimum-Context Test
|
||
|
||
After creating the document I stopped reading all source documents and used only:
|
||
- `docs/current-project-state.md`
|
||
- `.claude/architecture-guardrails.md`
|
||
|
||
To produce this briefing for a returning developer:
|
||
|
||
1. **Active:** Deterministic reasoning pipeline, unknown selection (atomicity/answerability), question formulation within reasoning patterns, scenario API, turn cycle orchestration. Nothing more from the engine itself.
|
||
2. **Passive:** Investigation-state assessment, behaviour selection, decision condition status, question-to-condition relevance, evidence direction, evidence scope, scope-aware condition status — all isolated diagnostic layers with no active integration.
|
||
3. **Paused:** Engine experiments (after 25B), UI experiments. Knowledge-management is active. Nothing archived or deleted.
|
||
4. **Provisional:** Keyword/phrase matching for scope detection; passive classifier generalisability across domains; how passive reasoning enters the active cycle; whether architecture docs match implementation.
|
||
5. **Next:** `docs/current-project-state.md` is the starting point. Use the inventory for task-specific context. Guardrails before code changes.
|
||
|
||
Result: The briefing was accurate and complete from these two files. No essential information was missing. The routing table in section 7 of the current-state document provided all necessary references without requiring additional documents.
|
||
|
||
### Missing or Ambiguous Information Found
|
||
|
||
- `docs/investigation-state-assessment-contract.md` (232 lines) describes a data contract that may no longer match implementation after experiments 15–25B; not verified.
|
||
- The exact line count of the created document should be confirmed with `wc -l`.
|
||
- Whether any of the passive classifiers have been partially integrated since Exp 25B was closed requires checking source code — this task did not read it.
|
||
|
||
### Assessment
|
||
|
||
The new entry point successfully replaced the need to load the large experiment-log section (1,542 lines). The current-state document conveys active vs passive capabilities, pause status, unresolved questions and loading instructions in a single short file. It can replace the large default log section as the normal starting point for future work.
|
||
|
||
The practical briefing was produced accurately from only two files without reading any source material beyond what was used to create it. This confirms the hypothesis that a concise current-state document is sufficient context for understanding where the project stands.
|
||
|
||
### Return-to-Work Note
|
||
|
||
A short current-state entry point now exists at `docs/current-project-state.md`. Future Claude sessions should begin there. The full experiment history remains available in `docs/design-evolution-log.md` but is no longer default reading. Nothing has been archived, moved or deleted yet. Before changing the documentation structure, review whether the new entry point reliably replaces the large log section and whether any historical documents should be formally archived. First file to inspect when resuming: `docs/current-project-state.md`. Branch: `feature/user-workspace-ux-v0.7`.
|
||
|
||
### Status
|
||
|
||
Pending Rob's review.
|
||
|
||
The following are active explorations rather than decisions.
|
||
|
||
- What is the right metaphor for the product?
|
||
- Should the workspace resemble a facilitated workshop?
|
||
- How should decomposition be represented?
|
||
- What information belongs in shared understanding?
|
||
- What should the Investigation Map eventually become?
|
||
- How should wide thinking be reflected in the interface?
|
||
|
||
## Backlog — Experiment 05 Persistence Note
|
||
|
||
The "Don't show this introduction again" checkbox uses sessionStorage as a placeholder.
|
||
|
||
This preference should eventually be handled through user preferences or settings rather than local component state.
|
||
|
||
TODO: When user accounts are introduced, persist this preference to the user profile so it travels across devices and sessions.
|
||
|
||
## Future Note — Dark Mode
|
||
|
||
Dark mode is intentionally deferred.
|
||
|
||
Once the information architecture and visual hierarchy stabilise we will investigate whether an "Investigation Mode" (rather than a conventional dark mode) improves concentration.
|
||
|
||
This should be treated as a future UX experiment rather than an accessibility feature.
|
||
|
||
## Experiment 28 — Verify Current Project State Against Implementation
|
||
|
||
**Status:** Pending Rob's review
|
||
|
||
### Hypothesis
|
||
|
||
A focused code inspection can verify or correct the current-state document without requiring a fresh session to read the full experiment log. If the document is accurate, it can safely become the normal project entry point.
|
||
|
||
### Source Areas Inspected
|
||
|
||
- `docs/current-project-state.md` — entire file;
|
||
- `.claude/architecture-guardrails.md` — entire file;
|
||
- `docs/project-knowledge-inventory.md` — Current Working Context and Task-Specific References sections;
|
||
- `app/api/*/route.js` — all API entry points (analyse, cases/start, cases/update, health);
|
||
- `lib/graph/orchestrator.js` — imports (lines 6–32) and runtime calls at lines 376, 402, 552, 581, 622, 826, 904, 1013;
|
||
- `lib/graph/*.js` — grep for imports of passive classifier modules (decision-condition-status, evidence-direction, evidence-condition-scope, question-decision-relevance, question-importance);
|
||
- `lib/behaviour-selection/behaviour-selector.js` — cross-module import check;
|
||
- `lib/assessment/investigation-state-assessor.js` — caller trace in orchestrator.
|
||
|
||
### Active / Passive Findings
|
||
|
||
**Active capabilities confirmed:**
|
||
1. Scenario reconstruction (analyseScenario) — API entry at app/api/analyse/route.js → lib/analysis.js.
|
||
2. Reasoning graph updates (startCase / updateCase) — API entries at app/api/cases/{start,update}/route.js → orchestrator.js → apply-proposal.js. Propagation, confidence cap, completeness calculated in apply-proposal.
|
||
3. Unknown selection (atomicity + answerability) — selectActiveUnknownCandidate imported and called from orchestrator's determineGraphBackedQuestion within the active updateCase path.
|
||
4. Question formulation — formulateQuestion / formulateTieResolutionQuestion imported and called from the active turn cycle.
|
||
5. Turn orchestration — orchestrator.js updateCaseWithDependencies() is the active engine heart, coordinating unknown→question→answer→graph-update→propagation→next-unknown.
|
||
|
||
**Passive or isolated capabilities confirmed:**
|
||
1. Investigation-state assessment (assessInvestigationState) — called at 3 sites in orchestrator but result only placed into a diagnostics field; not used for any control-flow decision. Classification: **diagnostic_only**.
|
||
2. Behaviour selection (selectBehaviour) — exported from behaviour-selector.js; no callers anywhere in the repo. Classification: **isolated**.
|
||
3. Question importance, question relevance to decision, evidence direction, evidence scope, scope-aware condition status — each exists as a standalone module or file with zero external callers. Evidence direction and scope are imported only by decision-condition-status.js, which itself has no callers.
|
||
|
||
### Corrections Made
|
||
|
||
None. The current-state document's active/passive classification is accurate as-is. Added verification marker to docs/current-project-state.md.
|
||
|
||
### Practical Context-Test Result
|
||
|
||
**Task:** A developer proposes connecting Behaviour Selection directly to the next user-facing response. Is it active today? What boundary exists? Which files would need inspection before future integration?
|
||
|
||
**Briefing:**
|
||
1. **Active today?** No. `selectBehaviour` is exported from `lib/behaviour-selection/behaviour-selector.js` but has zero callers anywhere in the repository. It is not active, diagnostic, or accessible through any API.
|
||
2. **Current boundary:** Behaviour Selection and Investigation-State Assessment exist as separate modules that were never wired into the orchestrator's turn cycle. The orchestrator returns an `assessment` field to clients but does not pass assessment results into its own decision logic. There is no data path from state assessment → behaviour selection → question/response.
|
||
3. **Files to inspect before integration:** `lib/graph/orchestrator.js` (where the insertion point would be — between unknown selection and question formulation, or after propagation); `lib/assessment/investigation-state-assessor.js` (to understand what the assessment contract outputs); `lib/behaviour-selection/behaviour-selector.js` (to understand what behaviours it can produce); `docs/investigation-state-assessment-contract.md` and `docs/behaviour-selection.md` for the documented interfaces; `app/api/cases/update/route.js` to determine whether behaviour output would appear in the API response or remain internal.
|
||
4. **Context sufficient?** Yes — the three-file set (current-project-state, verification file, guardrails) plus targeted code inspection of the modules above provides sufficient context for a designer to assess integration scope without reopening the full history.
|
||
5. **Verdict:** Integration is feasible as a future experiment. The primary risk is that behaviour selection has no documented input contract from the assessment layer — these were built in parallel without an agreed handoff shape.
|
||
|
||
### Unresolved Questions
|
||
|
||
- Whether the assessment output from `assessInvestigationState` matches the documented `investigation-state-assessment-contract.md` (requires reading the assessor's internal logic, excluded per constraints).
|
||
- Whether external API clients (not in this repo) call the orchestrator directly, bypassing the route files.
|
||
- The exact integration sequence: should behaviour selection read from assessment output or from the graph state directly?
|
||
|
||
### Return-to-Work Note
|
||
|
||
The current-state briefing was checked against source code via targeted code inspection of API routes, orchestrator imports/calls, and cross-module traces for each passive classifier. Five active capabilities are confirmed (reconstruction, graph updates, unknown selection, question formulation, turn orchestration). Seven passive capabilities remain classified as diagnostic_only (investigation-state assessment) or isolated (behaviour selection, decision-condition status, evidence direction, evidence scope, question importance, question relevance to decision, scope-aware condition status). No corrections to the current-state document were required. Knowledge-management work remains active. Engine and UI experiments remain paused. Branch: feature/user-workspace-ux-v0.7. First file to inspect when resuming: `docs/current-project-state.md`, then `.claude/architecture-guardrails.md` before any code changes, then `lib/graph/orchestrator.js` for engine-resumption work.
|
||
|
||
Branch: feature/user-workspace-ux-v0.7
|
||
Commit: 61c8a3a
|
||
|
||
|
||
## Experiment 29 — Archive the History Without Losing the Trail
|
||
|
||
**Status:** Pending Rob's review
|
||
|
||
### Hypothesis
|
||
|
||
Historical documents can be moved into a clearly labelled archive without breaking links, losing evidence, or confusing future sessions. A fresh Claude session should still be able to understand the current system from the short entry point, locate historical material when specifically needed, and identify which documents are current versus retained only as evidence.
|
||
|
||
### Files Archived (5)
|
||
|
||
| Original Path | Archive Path | Reason |
|
||
|---|---|---|
|
||
| `docs/v0.4-handoff.md` | `docs/archive/v0.4-handoff.md` | Historical v0.4 handoff; architecture has evolved since. Referenced in `orchestrator-contract.md` (reference repaired). |
|
||
| `docs/v0.4-route-status.md` | `docs/archive/v0.4-route-status.md` | Historical route tracking; current routes differ. |
|
||
| `docs/v0.5-release-notes.md` | `docs/archive/v0.5-release-notes.md` | Historical release record; nothing active depends on it. |
|
||
| `docs/v0.6-ambiguity-generalisation.md` | `docs/archive/v0.6-ambiguity-generalisation.md` | Superseded by later reasoning architecture decisions (Exp 15–25B). |
|
||
| `docs/v0.7-observation-report.md` | `docs/archive/v0.7-observation-report.md` | Experimental observation snapshot; useful reference but not current guidance. UX work paused. |
|
||
|
||
### Files Deliberately Not Archived (2)
|
||
|
||
| Document | Reason |
|
||
|---|---|
|
||
| `docs/architectural-principles.md` | 14 architectural principles from experiments; may be needed when re-engaging with reasoning architecture. Status unclear — review before future archive. |
|
||
| `docs/backlog info.md` | Mock fixture backlog useful if resuming UI development. Needs content verification before archiving. |
|
||
|
||
### Reference Repairs
|
||
|
||
- `docs/orchestrator-contract.md`: Updated reference from `docs/v0.4-handoff.md` to `docs/archive/v0.4-handoff.md` (line 78) and table entry (line 87).
|
||
- `docs/project-knowledge-inventory.md`: Updated all five archive candidate entries with new paths and provenance notes; updated Return-to-Work section.
|
||
- No other files contained active references to archived documents.
|
||
|
||
### Practical Archive Test
|
||
|
||
**Task:** A developer needs to find what v0.4 originally said about the case-orchestration API, without reading the full experiment log or archive directory.
|
||
|
||
**Execution:** From `docs/project-knowledge-inventory.md` (section 3) → identifies `docs/archive/v0.4-handoff.md` as the historical handoff for v0.4 architecture; from `docs/archive/README.md` → confirms file exists at that path and explains what it contains; verified file is accessible.
|
||
|
||
**Result:** The developer can locate the correct archived document in two steps: (1) inventory identifies which past document contains relevant evidence, (2) archive index confirms location and contents. The current project can be fully understood from `docs/current-project-state.md` alone without opening any archived file. No current task depends on archived files by default — they are consulted only when a named past decision or release is under investigation.
|
||
|
||
### Uncertain Candidates
|
||
|
||
- `docs/architectural-principles.md`: Should it be archived now, or reviewed first for accuracy against current implementation? Decision deferred to Rob's review.
|
||
- `docs/backlog info.md`: Contains mock fixtures — may become irrelevant if the fixture strategy changes. Needs content verification before any future archive decision.
|
||
|
||
### Status
|
||
|
||
Pending Rob's review.
|
||
|
||
These are observations, not implementation tasks.
|
||
|
||
- Narrative adapter
|
||
- Narrative quality heuristics
|
||
- Narrative progression
|
||
- Narrative completion state
|
||
- Narrative confidence wording
|
||
- Narrative testing
|
||
- Narrative localisation
|
||
- Multiple narrative projections
|
||
|
||
## Experiment 30 — Review Deferred Project Documents
|
||
|
||
**Status:** Pending Rob's review
|
||
|
||
### Hypothesis
|
||
|
||
Each deferred document can be classified by comparing it with the verified current project state without reopening the full experiment history or rewriting its contents. The result may be: keep as current guidance, keep as task-specific reference, archive as historical evidence, or retain temporarily pending revision. No additional categories should be invented.
|
||
|
||
### Review of architectural-principles.md
|
||
|
||
- **14 principles assessed against verified implementation:**
|
||
- **6 current (match runtime or guardrails):** P1 (layer separation), P3 (user feedback loop), P4 (reasoning/UI separation), P6 (presentation renders, does not interpret), P8 (narrative never invents facts), P14 (user as first-class input).
|
||
- **4 aspirational targets:** P5 (behaviour never reasons — module exists with zero callers), P10 (convergence over single signals — no mechanism), P11 (stateful assessment across turns — partially present), P12 (assessable uncertainty — absent).
|
||
- **4 mixed/unclear:** P2 (information flows downward — partially matches but passive layers don't fit the cascade model), P7 (assessment never generates evidence — diagnostic_only but scope-aware condition status makes interpretive judgments), P9 (assessment describes not prescribes — signals descriptive, but decision-condition evaluation borders on prescription), P13 (progress qualitative not quantitative — product direction supports; unknown selection uses node status qualitatively but not verified).
|
||
- **3 duplicated with guardrails:** P1 overlaps with architecture-guardrails' prohibition list. P4 overlaps with UX-task boundaries in guardrails. P8 overlaps with the explicit invariant "every question comes from a resolved graph node." Overlap adds value: guardrails state boundaries; principles explain why.
|
||
|
||
- **Role assigned:** Keep as task-specific reference. Six current principles and four aspirational targets make it valuable when resuming reasoning architecture work. Three duplications reduce (but don't eliminate) its independent value — the derived-from/implication context adds what guardrails lack. project-knowledge-inventory already listed it under "Review Before Archive"; confirmed as task-specific reference.
|
||
|
||
### Review of backlog info.md
|
||
|
||
- **Content analysis:**
|
||
- **Still-relevant (≈20 lines):** Mock fixtures table — 15 scenario types with purposes and examples. Directly useful when UI work resumes.
|
||
- **Historical/aspirational (≈370 lines):** UX roadmap phases 1–4 with wireframe text, animation specs, loading messages. Design intent is valid; specifics may change when UI resumes. Untracked — no commit/PR linkage.
|
||
- **Duplicates:** Phase 4 "Mock Scenario Library" duplicates the fixtures table at top. "Deliberately Out of Scope" repeats pause decision in current-project-state and project-context.
|
||
|
||
- **Role assigned:** Retain temporarily pending revision. The mock fixtures table is too useful to lose in an archive, but the document's mixed role (useful reference + deferred planning) needs resolution when UI work resumes. Splitting the file or archiving portions requires revising content — constraints forbid this now.
|
||
|
||
### Practical Routing Test Result
|
||
|
||
**Task:** A future Claude session is about to work on UI mocks. Should it read architectural-principles.md, backlog info.md, both, or neither?
|
||
|
||
**Answer: Both.** Backlog info.md provides the mock fixtures table (direct reference). Architectural-principles.md provides boundaries (P4: reasoning never communicates directly with UI; P6: presentation never interprets) that prevent accidentally introducing reasoning logic into UI work. Three-document context (current-project-state, project-knowledge-inventory, document-role-review) is sufficient to route both documents correctly without reading the full experiment log or archive.
|
||
|
||
### Files Created / Modified
|
||
|
||
- `docs/document-role-review.md` — new (140 lines); classifies both candidates with evidence and routing test
|
||
- `docs/project-knowledge-inventory.md` — updated "Review Before Archive" table (principle roles added), added "Knowledge management" section with document-role-review entry, updated Return-to-Work note
|
||
- `docs/current-project-state.md` — updated Return-to-Work note to include Experiment 30 status
|
||
- No files moved to archive (neither candidate qualifies as "archive as historical evidence")
|
||
- No files deleted; no source code or tests changed
|
||
|
||
### Status
|
||
|
||
Pending Rob's review. Neither document moves. Both roles confirmed by evidence against verified implementation. When UI work resumes, backlog info.md's fixtures table will be the direct reference; architectural-principles.md is available for reasoning architecture context. Engine and UI experiments remain paused. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `docs/current-project-state.md`, then Experiments 23–25B in design-evolution-log.md (lines 1218–1520).
|
||
|
||
These are observations, not implementation tasks.
|
||
|
||
---
|
||
|
||
## Experiment 31 — Separate Useful UI Reference From Unstructured Backlog
|
||
|
||
**Branch:** `feature/user-workspace-ux-v0.7`
|
||
|
||
### Hypothesis
|
||
|
||
The document `docs/backlog info.md` can be divided into:
|
||
- a short task-specific mock/UI reference that remains in the normal documentation area;
|
||
- a retained deferred backlog document that is excluded from default context loading.
|
||
|
||
This should make future UI work easier without losing previous ideas.
|
||
|
||
### Separation Method
|
||
|
||
Original file `docs/backlog info.md` (390 lines) was split into two new documents:
|
||
|
||
1. **`docs/ui-mock-reference.md`** (~62 lines) — practical mock-fixture reference extracted from the original lines 1–20, structured with available scenarios, fixture data locations, when-to-use guidance, and warnings.
|
||
2. **`docs/archive/deferred-ux-backlog.md`** (376 lines) — deferred UX planning content from original lines 21–390, preserved with original header stating items are not commitments.
|
||
|
||
The original file was removed after complete accounting (every section accounted for in one of the two new documents).
|
||
|
||
### Content Accounting
|
||
|
||
| Original Section | Line Range | Destination | Treatment |
|
||
|---|---|---|---|
|
||
| Mock fixtures table + intro | 1–20 | `docs/ui-mock-reference.md` | Represented as structured reference (same scenarios, enhanced with fixture data locations and usage guidance) |
|
||
| UI Roadmap header + intro | 21–26 | `docs/archive/deferred-ux-backlog.md` | Copied unchanged |
|
||
| Phase 1 – Core Investigation Experience | 27–118 | `docs/archive/deferred-ux-backlog.md` | Copied unchanged |
|
||
| Phase 2 – UX Polish | 119–169 | `docs/archive/deferred-ux-backlog.md` | Copied unchanged |
|
||
| Phase 3 – Developer Experience | 197–218 | `docs/archive/deferred-ux-backlog.md` | Copied unchanged |
|
||
| Phase 4 – Mock Scenario Library | 219–326 | `docs/archive/deferred-ux-backlog.md` | Copied unchanged (scenarios listed twice — once in original fixtures table, once here — no duplication introduced) |
|
||
| Backlog – Reasoning Replay | 328–378 | `docs/archive/deferred-ux-backlog.md` | Copied unchanged |
|
||
| Deliberately Out of Scope | 379–390 | `docs/archive/deferred-ux-backlog.md` | Copied unchanged |
|
||
|
||
**Material not transferred:** None. Every original section is represented in one of the two new documents.
|
||
|
||
### Files Created
|
||
|
||
- `docs/ui-mock-reference.md` (~62 lines) — mock fixture scenario reference
|
||
- `docs/archive/deferred-ux-backlog.md` (376 lines) — deferred UX planning backlog
|
||
|
||
### Files Removed
|
||
|
||
- `docs/backlog info.md` (390 lines) — superseded by the split; all content accounted for above
|
||
|
||
### Files Modified
|
||
|
||
- `docs/archive/README.md` — added deferred-ux-backlog to Archived Files table; added Superseded Files section with backlog info.md entry
|
||
- `docs/project-knowledge-inventory.md` — added ui-mock-reference to UI/UX task-specific references; added deferred-ux-backlog to archive candidates; updated backlog info.md role to "superseded"; updated Return-to-Work note
|
||
- `docs/current-project-state.md` — updated Section 6 (Return-to-Work Summary) and section 8 header/note to reflect Experiment 31 split
|
||
- `.claude/project-context.md` — added routing notes: UI mock work reads ui-mock-reference; deferred backlog only for named UX idea review
|
||
|
||
### Line Counts Before / After
|
||
|
||
| Document | Lines (before) | Lines (after) |
|
||
|---|---|---|
|
||
| Original combined document (`backlog info.md`) | 390 | removed |
|
||
| New mock reference (`ui-mock-reference.md`) | — | ~62 |
|
||
| New deferred backlog (`deferred-ux-backlog.md`) | — | 376 |
|
||
| Total new content | — | 438 (62 + 376, including headers in both) |
|
||
|
||
### Practical Routing Test Result
|
||
|
||
**Scenario:** A developer wants to test the workspace against a long investigation and a contradictory-evidence scenario. Which mock scenarios should they use, and where is the fixture data defined?
|
||
|
||
**Answer:** They should use:
|
||
- **Long investigation (10–15 turns)** — for testing history scrolling, collapsing, pacing;
|
||
- **Contradiction** — for testing contradiction detection and user-facing messaging.
|
||
|
||
Fixture data is defined in `tests/e2e/fixtures/investigation-scenarios.js`. The mock client is in `lib/mocks/confidence-engine/mock-client.js`. Scenario names are set via `NEXT_PUBLIC_CONFIDENCE_ENGINE_MOCK_SCENARIO` env var in `components/scenario-form.jsx`. Reference details and usage guidance are in `docs/ui-mock-reference.md`.
|
||
|
||
**Was the deferred backlog necessary?** No. The practical routing test was answered entirely from `ui-mock-reference.md`, `project-knowledge-inventory.md`, `.claude/project-context.md`, and `architecture-guardrails.md`. The deferred backlog (376 lines of aspirational UX planning) was not required to answer a practical mock-scenario question.
|
||
|
||
**Was any practical mock information lost?** No. All 13 fixture scenarios are preserved in `ui-mock-reference.md` with enhanced guidance on where fixtures live and when to use each. The original fixtures table's content is fully represented.
|
||
|
||
### Gaps Found
|
||
|
||
- `docs/ui-mock-reference.md` references `tests/e2e/fixtures/investigation-scenarios.js` as the fixture definition location but does not list individual scenario keys or env var values (by design — those are implementation details that can be inspected directly in the fixture file).
|
||
- The deferred backlog contains specific wireframe text and animation specifications that may still be useful when UI work resumes. The header note ("not commitments, priorities or active tasks") should prevent premature actioning.
|
||
|
||
### Status
|
||
|
||
Pending Rob's review. Both new documents contain all original content. Branch `feature/user-workspace-ux-v0.7` is clean after commit. Engine and UI experiments remain paused.
|
||
|
||
## Experiment 32 — Separate Current Principles From Aspirational Architecture
|
||
|
||
**Branch:** `feature/user-workspace-ux-v0.7`
|
||
|
||
### Hypothesis
|
||
|
||
A short current-principles document can guide normal work while the original architectural-principles document remains available as the fuller historical and aspirational source. This should reduce ambiguity without deleting or rewriting the original reasoning.
|
||
|
||
### Source Documents Used
|
||
|
||
- `docs/current-project-state.md` — What the Confidence Engine Is; Current Engine Capabilities; Context Loading Guide
|
||
- `.claude/architecture-guardrails.md` — entire file (77 lines)
|
||
- `docs/document-role-review.md` — Architectural Principles Review (§2) and Recommended Actions (§4)
|
||
- `docs/architectural-principles.md` — headings and the 14 principles only
|
||
- `docs/03_Confidence_Engine_Language_Guide.md` — guiding principles only
|
||
- `docs/current-implementation-verification.md` — Active Capabilities; Passive or Isolated Capabilities
|
||
- Experiment 31 entry in `docs/design-evolution-log.md` (lines 1811–1893)
|
||
|
||
### Principles Included
|
||
|
||
**User Experience (5):** System carries complexity; steps are small enough to understand or investigate; engine guides without pretending certainty; first input is the hardest step; users may know answer/who to ask/where to look/how to test.
|
||
|
||
**Reasoning (5):** Resolved question ≠ established condition; evidence supports/contradicts/informs; present evidence does not settle future feasibility; uncertainty stated honestly; deterministic contracts separate from language interpretation.
|
||
|
||
**Building the System (6):** Build smallest thing that can be wrong; use evidence before architecture; every layer has one responsibility where applicable; presentation does not invent facts; current and aspirational labelled separately; load only needed context.
|
||
|
||
Total: 16 current principles, organized into three sections.
|
||
|
||
### Aspirational Material Deliberately Excluded
|
||
|
||
From `docs/architectural-principles.md`: P2 (Information Flows Downward — unresolved), P5 (Behaviour Never Reasons — aspirational), P7 (Assessment Never Generates Evidence — mixed), P9 (Assessment Describes Never Prescribes — mixed), P10 (Convergence Over Single Signals — aspirational), P11 (Assessment Is Stateful Across Turns — mixed/aspirational), P12 (Uncertainty About Assessment Is Itself Assessable — aspirational), P13 (Investigation Progress Is Qualitative Not Quantitative — mixed/aspirational). These remain in the original document for broader architectural review.
|
||
|
||
### Practical Principles-Test Result
|
||
|
||
**Task:** A developer proposes making every resolved question automatically increase confidence and close its related condition. Explain whether this fits current principles and why.
|
||
|
||
**Response from reduced context (current-project-state + current-working-principles + architecture-guardrails):**
|
||
|
||
1. **Resolving a question does not establish a condition.** current-working-principles §2 states: "A resolved question is not an established condition." Answer evidence must be inspected before any conclusion follows.
|
||
2. **Answer evidence must be inspected.** current-working-principles §2 states direction alone (support/contradict/inform) is insufficient without checking subject, timeframe, and claim type alignment.
|
||
3. **Confidence should not be manufactured.** architecture-guardrails invariants state "Confidence must not outrun evidence or completeness" and "Duplicate evidence must not increase confidence." current-project-state section 4 confirms: resolving a question does not automatically establish the condition.
|
||
4. **Passive experimental logic is not automatically active behaviour.** current-project-state section 3 classifies passive classifiers (including decision-condition status evaluation) as diagnostic_only or isolated — they do not yet control the user-facing investigation.
|
||
|
||
**Was the three-document context sufficient?** Yes. All four points were answerable from `docs/current-working-principles.md` (principles §2), `.claude/architecture-guardrails.md` (reasoning invariants), and `docs/current-project-state.md` (section 3 passive classifier classification, section 4 what experiments established). No experiment history or source code was required.
|
||
|
||
### Unresolved Ambiguities
|
||
|
||
- The boundary between "current" and "aspirational" for P7 and P9 is inherently subjective; future sessions may interpret differently without the original document's reasoning context.
|
||
- Some principles overlap with `.claude/architecture-guardrails.md` (e.g., "every layer has one responsibility" overlaps with guardrails' exhaustive prohibition list). No duplication was introduced deliberately, but a cross-reference could reduce redundancy in a future iteration.
|
||
- The aspirational note points readers to the original document but does not provide a quick reference for which of the 14 principles are current versus aspirational. A summary table might be useful when architecture work resumes.
|
||
|
||
### Status
|
||
|
||
Pending Rob's review. No source code or tests changed. Engine and UI experiments remain paused. No files moved or deleted. Only documentation files were created or updated.
|
||
|
||
### Return-to-Work Note (80–150 words)
|
||
|
||
Current principles now live in `docs/current-working-principles.md`. This short document contains only guidance supported by verified implementation, current project direction, and established product philosophy — organised into three sections: user experience, reasoning, and building the system. Broader and aspirational architecture remains in `docs/architectural-principles.md` as a task-specific reference; it has not been rewritten or deleted. Future sessions should use `docs/current-working-principles.md` by default for product and reasoning work. Engine and UI experiments remain paused after Experiment 25B. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `docs/current-project-state.md`, then `docs/current-working-principles.md` for current guidance.
|
||
|
||
---
|
||
|
||
## Experiment 33 — Create Task-Specific Context Packs
|
||
|
||
**Branch:** `feature/user-workspace-ux-v0.7`
|
||
|
||
### Hypothesis
|
||
|
||
A single concise context-pack guide can give each task type a minimal reading list, clear exclusions, and a stopping rule — reducing unnecessary context loading while preserving access to deeper material when a specific gap appears.
|
||
|
||
### Source Documents Used
|
||
|
||
- `docs/current-project-state.md` — Context Loading Guide; Current Engine Capabilities; Work Currently Paused
|
||
- `docs/project-knowledge-inventory.md` — Current Working Context; Task-Specific References
|
||
- `docs/current-implementation-verification.md` — Active Capabilities; Passive or Isolated Capabilities
|
||
- `docs/current-working-principles.md` — entire file
|
||
- `docs/ui-mock-reference.md` — headings and routing information only
|
||
- `.claude/project-context.md` — routing notes only
|
||
- `.claude/architecture-guardrails.md` — headings only
|
||
- Experiment 32 entry in `docs/design-evolution-log.md` (lines 1895–1948)
|
||
|
||
### Deliverable
|
||
|
||
Created `docs/task-context-packs.md` (~110 lines) with four packs:
|
||
- **Pack 1 — Engine Experiment Work:** current-project-state, current-working-principles, architecture-guardrails, current-implementation-verification.
|
||
- **Pack 2 — UI and Mock Work:** current-project-state, current-working-principles, architecture-guardrails, ui-mock-reference.
|
||
- **Pack 3 — Architecture or Contract Review:** current-project-state, current-implementation-verification, architecture-guardrails, current-working-principles + aspirational warning.
|
||
- **Pack 4 — Knowledge-Management Work:** current-project-state, project-knowledge-inventory, task-context-packs, project-context.
|
||
|
||
Each pack lists what to always read, what to read only when relevant, and what to not load by default. Common rules prevent silent context inflation. Two routing tests verify sufficiency without loading history or source code.
|
||
|
||
### Routing Test A — Engine Task
|
||
|
||
**Task:** Verify whether Behaviour Selection currently affects the user-facing response.
|
||
**Result:** Pack sufficient. `docs/current-implementation-verification.md` §3b states "Called by: None" for Behaviour Selection; `docs/current-project-state.md` §3 classifies it as isolated. No extra file required.
|
||
|
||
### Routing Test B — UI Task
|
||
|
||
**Task:** Choose the correct mock scenarios for testing a long investigation and contradictory evidence.
|
||
**Result:** Pack sufficient. `docs/ui-mock-reference.md` lists "Long investigation (10–15 turns)" and "Contradiction" with matching purposes. Deferred UX backlog not needed.
|
||
|
||
### Validation
|
||
|
||
- All referenced files exist; no pack relies on fixed line numbers.
|
||
- Each pack has a smaller default context than the full project documentation.
|
||
- Active and passive capabilities remain clearly separated.
|
||
- No source code or tests changed; no files moved or deleted.
|
||
|
||
### Return-to-Work Note
|
||
|
||
Task-specific context packs now exist in `docs/task-context-packs.md`, giving each work type a minimal four-document starting set plus targeted reading paths. Future sessions should start with `docs/current-project-state.md`, then choose one pack from `docs/task-context-packs.md`. Additional documents should be loaded only for a named gap, with the reason recorded. Engine and UI experiments remain paused. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `docs/current-project-state.md`, then select the relevant pack from `docs/task-context-packs.md`.
|
||
|
||
---
|
||
## Experiment 34 — Single Return-to-Work Handoff
|
||
|
||
**Date:** 2026-08-06
|
||
**Branch:** `feature/user-workspace-ux-v0.7`
|
||
|
||
### Hypothesis
|
||
|
||
A single short handoff file can carry enough immediate context to resume work accurately while linking to deeper documents only when needed.
|
||
|
||
### Handoff Structure
|
||
|
||
Eight sections: Where We Left It, What Is True Now, Why Work Is Paused, What Was Just Completed, What Remains Open, How to Resume, First Files by Work Type (table), Resume Check (five questions). Plus a maintenance rule replacing current-work sections when the project moves on.
|
||
|
||
### Document Length
|
||
|
||
`docs/current-handoff.md`: 68 lines (target range: 60–100).
|
||
|
||
### Practical Resume-Test Result
|
||
|
||
**Task:** Return after two weeks, remember almost nothing. Explain where the project stands, what is paused, what was completed most recently, and what to read before an engine task — using only `docs/current-handoff.md` and `docs/task-context-packs.md`.
|
||
|
||
| Check | Result |
|
||
|---|---|
|
||
| Identifies correct active phase (knowledge management) | Yes |
|
||
| Identifies paused engine and UI work | Yes |
|
||
| Identifies Experiment 33 as latest completed | Yes |
|
||
| Chooses Engine Experiment pack for engine task | Yes |
|
||
| Avoids opening full design history | Yes |
|
||
| Does not confuse passive code with active behaviour | Yes |
|
||
|
||
**Verdict:** Pass. The handoff alone is sufficient to resume accurately.
|
||
|
||
### Missing Information
|
||
|
||
- "When knowledge-management work is complete enough to resume engine experiments" — no objective criterion exists yet; this is a judgment call for Rob.
|
||
- "Whether tasks crossing pack boundaries can still stay concise" — unanswered in principle; requires testing with actual cross-boundary tasks.
|
||
- Whether `docs/current-handoff.md` remains useful after several more knowledge-management experiments add to it.
|
||
|
||
### Can This Replace Scattered Current Return Notes?
|
||
|
||
Yes, for immediate resumption context. The handoff carries the latest stopping point without accumulating old notes. Historical return notes remain in `docs/current-project-state.md` and `docs/design-evolution-log.md` as evidence, not as current guidance. Rob should decide whether to purge older return notes once confident in the handoff model.
|
||
|
||
### Status
|
||
|
||
Pending Rob's review.
|
||
|
||
## Experiment 35 — Test Current Handoff Maintenance (2026-08-06)
|
||
|
||
**Hypothesis:** A current handoff can remain useful if it describes only the latest stopping point, replaces stale details rather than appending history, and identifies the latest confirmed experiment and commit unambiguously.
|
||
|
||
**Stale or ambiguous wording found:**
|
||
- Section 1 named Experiment 33 and commit `b959cfa` as the current state — now stale after Experiments 34+35;
|
||
- Section 4 described only Experiment 33's completion, giving no indication that a single handoff had been created in Experiment 34;
|
||
- No explicit mention of commit `1d92aa0` anywhere in the handoff;
|
||
- Footer said "Created by Experiment 34" without acknowledging this maintenance experiment.
|
||
|
||
**Corrections made:**
|
||
- Section 1: updated to name Experiment 34 and commit `1d92aa0`; added the maintenance principle ("replace stale details rather than appending history");
|
||
- Section 4: rewritten to describe Experiment 34's consolidation work;
|
||
- Section 5: retained one genuinely open question about handoff longevity; added provisional KM completion criteria sub-section (7 criteria, marked provisional);
|
||
- Footer: updated to reference Experiment 35; added Return-to-Work Note recording all current state.
|
||
|
||
**Fresh-return test result:** PASS — from `current-handoff.md` and `task-context-packs.md` only, a fresh session can determine:
|
||
- Latest completed KM experiment: Experiment 34 ✓
|
||
- Latest commit: `1d92aa0` ✓
|
||
- Knowledge-management active, engine/UI paused ✓
|
||
- Knowledge-Management context pack is the correct routing target ✓
|
||
- No need to open full design history ✓
|
||
- Older commits not mistaken for current stopping point ✓
|
||
|
||
**Provisional completion criteria added:** Seven criteria recorded in Section 5 (see above). Not yet declared complete — pending Rob's review.
|
||
|
||
**Handoff remained concise?** Yes. 86 lines (was 68). Increase justified by the maintenance principle paragraph, updated current-state wording, and provisional completion criteria section. No historical timeline appended.
|
||
|
||
**Status:** Pending Rob's review.
|
||
|
||
## Experiment 36 — Validate Reduced Context Routing
|
||
|
||
**Branch:** `feature/user-workspace-ux-v0.7`
|
||
|
||
### Hypothesis
|
||
|
||
The documentation system (handoff + project-state + task-context-packs) is complete enough to support normal work without silently expanding into historical documentation. A fresh session can complete representative tasks using only routing instructions.
|
||
|
||
### Initial Documents Loaded (328 lines total)
|
||
|
||
1. `docs/current-handoff.md` — 86 lines
|
||
2. `docs/current-project-state.md` — 132 lines
|
||
3. `docs/task-context-packs.md` — 110 lines
|
||
|
||
### Additional Documents Loaded
|
||
|
||
| Document | Lines | Why Needed | Routing Should Include? |
|
||
|---|---|---|---|
|
||
| `docs/ui-mock-reference.md` | 63 | Task 2: verify mock scenarios for "long investigation" and "contradiction". Routing Test B claimed these were identifiable without loading it, but the specific scenario names do not appear in any initial document. | YES — routing defect found |
|
||
| `docs/project-knowledge-inventory.md` | 215 | Task 4: confirm Engine Experiment pack's four always-read documents actually exist and understand KM phase outputs. | Debated — validated completeness but not strictly required by routing |
|
||
| `docs/current-implementation-verification.md` | 111 | Cross-checked Behaviour Selection isolation against current-project-state §3. Provided corroboration but was not the sole basis for Task 1 answer. | Debated — useful corroboration; current-project-state alone sufficed |
|
||
|
||
### Tasks Completed Without Context Expansion
|
||
|
||
**Task 1 — Does Behaviour Selection affect engine behaviour?**
|
||
No. Current project state §3 classifies it as isolated. Handoff §2 confirms passive classifiers don't control the investigation. Task-context-packs Routing Test A corroborates (current-implementation-verification §3b).
|
||
|
||
**Task 3 — Why passive classifiers are not yet in the active reasoning loop?**
|
||
Passive classifiers record diagnostic signals for future use but have no integration into the turn cycle. Only investigation-state assessment is called (at 3 orchestrator sites), and its result goes into a diagnostics field — never checked by conditional branches. Others have zero callers.
|
||
|
||
### Tasks Requiring Extra Context
|
||
|
||
**Task 2 — Mock scenarios for long investigation and contradictory evidence**
|
||
Required `docs/ui-mock-reference.md`. Routing Test B in task-context-packs claimed these were identifiable without loading it, but the specific scenario names ("Long investigation (10–15 turns)" and "Contradiction") do not appear in any initial document. The routing claim was unverifiable until the mock reference was loaded — this is a genuine routing defect.
|
||
|
||
**Task 4 — Where should a new developer begin for the next engine experiment?**
|
||
Partially answered from initial documents (handoff → project-state → pack). Marginal need to verify that all four always-read pack documents actually exist, resolved by cross-referencing project-knowledge-inventory.
|
||
|
||
### Routing Failures Found
|
||
|
||
One genuine failure: **Routing Test B in task-context-packs.md**. The test states that mock scenarios for long investigation and contradiction are identifiable without loading ui-mock-reference.md. This was presented as a self-evident fact but the specific scenario names only exist in ui-mock-reference.md. The routing is incomplete — it should have included the mock reference file, or at minimum acknowledged that scenario names require verification.
|
||
|
||
### Documentation Changes Made
|
||
|
||
- Created `docs/context-routing-validation.md` (62 lines) — this experiment's record
|
||
- Updated `docs/design-evolution-log.md` — appended Experiment 36 entry
|
||
|
||
**No source code or tests changed. No archive changes.**
|
||
|
||
### Overall Assessment: Mostly ready
|
||
|
||
Two of four tasks completed from initial context only. One routing defect found (Task 2; corrected by Experiment 37). After fixing Routing Test B to name ui-mock-reference.md as the scenario source, the reduced context system is ready for normal work.
|
||
|
||
---
|
||
|
||
## Experiment 37 — Validate Cross-Boundary Context Routing
|
||
|
||
**Branch:** `feature/user-workspace-ux-v0.7`
|
||
|
||
### Hypothesis
|
||
|
||
The context-pack system can support cross-boundary work if Claude:
|
||
1. starts with one primary pack;
|
||
2. adds a second pack only for a named boundary;
|
||
3. records why each extra document was loaded;
|
||
4. avoids loading the full history.
|
||
|
||
### Initial Documents Loaded (328 lines total)
|
||
|
||
1. `docs/current-handoff.md` — 85 lines; first return-to-work entry point
|
||
2. `docs/current-project-state.md` — 131 lines; active state and capabilities
|
||
3. `docs/task-context-packs.md` — 110 lines; routing for four work types
|
||
|
||
### Additional Documents Loaded
|
||
|
||
| Document | Lines | Why Needed | Routing Should Include? |
|
||
|---|---|---|---|
|
||
| `docs/ui-mock-reference.md` | 62 | Cross-boundary boundary: the task requires identifying a mock scenario for workspace display. This is the second pack (UI and Mock) needed because no other loaded document names scenarios or UI fixtures. Yes — it is part of the UI/Mock pack, not an ad-hoc addition. |
|
||
|
||
### Cross-Boundary Task Result
|
||
|
||
**Task:** Display passive condition-status information in the workspace for a mock investigation without changing the active reasoning loop.
|
||
|
||
| Finding | Details |
|
||
|---|---|
|
||
| Condition-status capability | Passive: decision-condition status evaluation records signals but has no integration into the turn cycle; never controls user-facing decisions or path selection |
|
||
| Active reasoning loop | Unchanged: deterministic pipeline (scenario reconstruction → graph update → unknown selection → question formulation → turn orchestration); none of these pathways are affected by passive data |
|
||
| Mock scenario | "Long investigation (10–15 turns)" from `ui-mock-reference.md`; workspace can display accumulated diagnostic signals over time without interrupting the active reasoning cycle |
|
||
| Implementation areas to inspect later | decision-condition-status evaluation module; evidence scope detection module; UI workspace components for passive display integration |
|
||
| Both packs genuinely needed? | Yes: Engine pack identifies which capabilities are active vs passive; UI pack identifies how the workspace presents state. Neither alone suffices |
|
||
| Archive or full history required? | No |
|
||
|
||
**Context remained manageable:** Yes. 390 lines total (328 initial + 62 additional). Each document loaded for a specific named purpose. No blind expansion.
|
||
|
||
### Knowledge-Management Completion Criteria Review
|
||
|
||
| Criterion | Status |
|
||
|---|---|
|
||
| 1. Fresh session can resume from handoff + one pack | met |
|
||
| 2. Current state verified against implementation | met |
|
||
| 3. Historical material outside default loading | met |
|
||
| 4. Current principles separated from aspirational architecture | met |
|
||
| 5. Task-specific routing works for engine and UI tasks | met |
|
||
| 6. Cross-boundary task tested | **met** |
|
||
| 7. Maintaining handoff does not require reading full history | met |
|
||
|
||
All seven criteria are now met.
|
||
|
||
> Knowledge-management structure is ready for Rob's review before engine experiments resume.
|
||
|
||
### Routing Defects Discovered
|
||
|
||
None in this experiment. The correction to Routing Test B (naming `ui-mock-reference.md` as the scenario source) was applied before testing. No new defects found in the cross-boundary test.
|
||
|
||
### Overall Assessment: Ready
|
||
|
||
The context-pack system handled a genuine engine/UI cross-boundary task by combining two packs deliberately with full documentation of each loaded document and its purpose. Context remained small (390 lines). All knowledge-management criteria are met.
|
||
|
||
---
|
||
|
||
# Experiment 38 — Cold-Start Project Recovery Validation
|
||
|
||
**Branch:** `feature/user-workspace-ux-v0.7`
|
||
**Type:** Knowledge-management / handoff validation (final KM experiment)
|
||
**Objective:** Test whether a genuinely cold session can recover the project accurately from the reduced context system alone without reading the full history or any earlier experiment reports.
|
||
|
||
## Setup
|
||
|
||
Cold-start configuration: no prior conversation context, no past experiment reports loaded, repository documentation carries all context. Session was freshly created to simulate a real return-to-work scenario. Only `docs/current-handoff.md` was read first (per handoff §6 step 1), then the two documents specified by its resume instructions (§6 steps 2–3): `docs/current-project-state.md` and `docs/task-context-packs.md`.
|
||
|
||
## Documents Loaded
|
||
|
||
| Document | Reason |
|
||
|---|---|
|
||
| `docs/current-handoff.md` | Primary entry point (handoff §6 step 1) |
|
||
| `docs/current-project-state.md` | Resume instruction (§6 step 2) and routing table (§6 step 7) |
|
||
| `docs/task-context-packs.md` | Pack selection (§6 step 3) and pack contents for verification |
|
||
|
||
No additional documents were loaded. No blind expansion occurred. The full design-evolution log, archived documents, UI mock reference, source code, and tests were all excluded by design.
|
||
|
||
## Project-State Recovery Result
|
||
|
||
The cold session correctly recovered:
|
||
- What the Confidence Engine does (facilitated investigation with structured reasoning graph).
|
||
- Active capabilities: deterministic reasoning pipeline, unknown selection via atomicity/answerability, question formulation, scenario API, turn cycle orchestration.
|
||
- Passive capabilities: seven diagnostic layers from Experiments 18–25B, all isolated, none control user-facing investigation.
|
||
- Paused work: engine experiments (after Exp 25B), UI experiments.
|
||
- Why KM phase was undertaken (documentation bloat blocking session recovery).
|
||
|
||
Recovery score: complete from three documents alone. No source code inspection required.
|
||
|
||
## Context-Pack Selection Result
|
||
|
||
Pack 1 — Engine Experiment Work selected correctly by the cold session. The three initial documents contained sufficient information to identify the pack, its default documents, and what to exclude without reading any additional material.
|
||
|
||
## Handoff Defects Found
|
||
|
||
None found in `docs/current-handoff.md`. The handoff accurately describes the stopping point, identifies all seven KM criteria as met, provides correct resume instructions, and includes accurate capability boundaries. One structural update was made: the open item "whether the handoff stays accurate after further advances" was resolved as no longer applicable (the cold-start test confirmed it is accurate).
|
||
|
||
## Completion-Criteria Result
|
||
|
||
All seven knowledge-management completion criteria are confirmed met by this cold-start validation:
|
||
1. Fresh session can resume from handoff + one pack — met (Exp 38 demonstrates this)
|
||
2. Current state verified against implementation — met (Exp 28+)
|
||
3. Historical material outside default loading — met
|
||
4. Current principles separated from aspirational architecture — met
|
||
5. Task-specific routing works for engine and UI tasks — met (Exp 37)
|
||
6. Cross-boundary task tested — met (Exp 37)
|
||
7. Maintaining handoff does not require reading full history — met
|
||
|
||
> The knowledge-management phase is complete enough for Rob to choose when engine experiments resume.
|
||
|
||
## Documents Updated
|
||
|
||
- `docs/cold-start-validation.md` — created (this experiment's deliverable)
|
||
- `docs/current-handoff.md` — Exp 38 commit placeholder, structural open-item resolution, return-to-work note replacement
|
||
- `docs/current-project-state.md` — KM status update ("active" → "complete"), latest known commit correction
|
||
- `docs/design-evolution-log.md` — this entry
|
||
|
||
## Overall Assessment: Ready
|
||
|
||
The cold-start validation passed. A genuinely fresh session understood the project state, chose the correct context pack, verified the resume boundary, produced a valid engine-work resume brief, and found no handoff defects — all from three documents alone. No source code was read or changed. The reduced context system works for sessions that did not help create the documents.
|
||
|
||
Engine and UI experiments remain paused pending Rob's review.
|
||
|
||
---
|
||
|
||
## Experiment 39 — Validate Behaviour Selection Against Real Assessment Outputs (2026-08-06)
|
||
|
||
**Branch:** feature/user-workspace-ux-v0.7
|
||
|
||
### Hypothesis
|
||
The existing deterministic selector produces a useful rhythm across genuine assessment outputs without changing the active engine. If it repeatedly chooses one behaviour, chooses behaviours at the wrong time, or depends on signals the assessor does not actually produce, the experiment should expose that honestly.
|
||
|
||
### Scenarios Evaluated (from `tests/investigation-state-assessor.test.js` fixture set)
|
||
1. **Long investigation** (3 turns: early → deepening → complete terminal)
|
||
2. **Contradictory evidence** (3 turns: two conflicting consultants, 0→1→2 resolved unknowns)
|
||
3. **Short early** (1 turn: two observations, first unknown, no resolution)
|
||
|
||
### Behaviour Distribution (7 turns total)
|
||
- Acknowledge: 5 (71%)
|
||
- Continue: 2 (29%)
|
||
- Clarify: 0 (0%)
|
||
- Summarise: 0 (0%)
|
||
- Pause: 0 (0%)
|
||
|
||
### Behaviour Sequence by Scenario
|
||
**Long investigation:** continue → acknowledge → acknowledge
|
||
- Turn 0: phase=cannot_determine, progress=cannot_determine, health=too_narrow → continue (no rule matched)
|
||
- Turn 3: phase=focusing, progress=steady, health=healthy → acknowledge
|
||
- Turn 4: phase=concluding, progress=steady, health=healthy → acknowledge
|
||
|
||
**Contradictory evidence:** acknowledge → acknowledge → acknowledge
|
||
- Turn 0: phase=focusing, progress=cannot_determine, health=healthy → acknowledge
|
||
- Turn 1: phase=focusing, progress=stalled, health=healthy → acknowledge
|
||
- Turn 2: phase=focusing, progress=steady, health=healthy → acknowledge
|
||
|
||
**Short early:** continue
|
||
- Turn 0: phase=exploring, progress=cannot_determine, health=healthy → continue
|
||
|
||
### Sensible Selections (7 of 7)
|
||
All selections were classified as sensible per the selection's stated conditions. Acknowledge fires because `health=healthy AND phase confidence≠low` across most states. Continue fires when no specific rule matches (early/cannot_determine/exploring phases).
|
||
|
||
### Questionable or Inappropriate Selections
|
||
**One notable pattern:** Summarise and Pause never fire, even in a concluding terminal state. This is not because the assessor fails to detect "concluding" — it does. It is because Acknowledge (priority 1) fires first when health=healthy, blocking Summarise (priority 3) from ever reaching its turn. This is an **acknowledgement/summarise priority conflict**: acknowledging a conclusion ("you've figured this out!") is not wrong, but "give me a summary" is more useful at terminal states. The current rule ordering does not distinguish "early healthy" from "concluding healthy."
|
||
|
||
Clarify never fires because no test scenario produces `health=too_broad` — the assessor's "too_broad" trigger (activeUnknownCount > 3 AND resolved < 2) requires more nodes than any scenario in the fixture set has at that stage.
|
||
|
||
Pause never fires because `health=user_overloaded` is never reached, and while contradictory-turn-1 has phase=focusing + progress=stalled, Acknowledge still blocks it.
|
||
|
||
### Contract Alignment
|
||
Assessor → Selector contract aligns cleanly. The assessor produces all three dimensions (phase, progress, conversationHealth) with the fields the selector expects. No transformation needed between pipeline stages.
|
||
|
||
### Whether Selector Appears Useful Enough for Another Passive Experiment
|
||
The existing selector works but its **behaviour variation is severely constrained by Acknowledge's priority position**. A next passive experiment should test whether reordering or refining the acknowledge condition (e.g., excluding concluding/terminal phases) produces more context-appropriate behaviour — without changing the assessor.
|
||
|
||
### Status
|
||
Pending Rob's review. Five behaviours are too narrow for this to be definitive, and only three scenarios were tested. The dominant pattern (acknowledge in healthy states) may change with different investigation domains.
|
||
|
||
### Documents Updated
|
||
- `docs/design-evolution-log.md` — this entry
|
||
- `docs/current-handoff.md` — return-to-work note replaced
|
||
|
||
---
|
||
|
||
## Experiment 40 — Audit Behaviour Reachability and Blocking (2026-08-06)
|
||
|
||
### Objective
|
||
|
||
Why did Clarify, Summarise, and Pause not appear during Experiment 39? Acknowledge: 5 (71%), Continue: 2 (29%), others: 0. This is a passive diagnostic — no rule changes, no engine modifications.
|
||
|
||
### Method
|
||
|
||
One test file (`tests/behaviour-selection.reachability.test.js`) containing:
|
||
- Diagnostic audit helper that evaluates every behaviour rule against one assessment object
|
||
- Real-scenario audits across the same Experiment 39 turns (8 turns total)
|
||
- Synthetic reachability checks for each behaviour in isolation
|
||
|
||
### Findings
|
||
|
||
#### Summarise — eligible_but_blocked
|
||
|
||
Eligible in 2 of 7 real turns:
|
||
- long-investigation turn 1 (resolvedNodeCount ≥ 3 + progress=steady triggers summarise rule)
|
||
- long-investigation turn 2 (phase=concluding triggers summarise rule)
|
||
|
||
In both cases, health=healthy simultaneously, so Acknowledge (priority 1) fires first. Summarise rules are met but its output is never returned because the selector returns early on priority ordering.
|
||
|
||
**Root cause: priority conflict, not assessor failure.** The phase evidence correctly identifies concluding/synthesising states; the problem is that Acknowledge's broader trigger condition (health=healthy is the most common state) fires first.
|
||
|
||
#### Clarify — never_eligible_in_tested_scenarios (reachable only in synthetic case)
|
||
|
||
Not eligible in any of 7 real turns because neither trigger condition is met:
|
||
- `health=too_broad`: requires activeUnknownCount > 3 AND resolvedNodeCount < 2 — no fixture reaches this state
|
||
- `phase=orienting + observationDensity < 3`: current assessor never produces phase=orienting for tested scenarios
|
||
|
||
Synthetic case confirms the rule fires correctly in isolation (with low-confidence phase to avoid Acknowledge blocking).
|
||
|
||
**Root cause: assessor health classification logic produces too few `too_broad` cases. The trigger condition is extremely narrow — needs activeUnknownCount > 3 AND resolved < 2 simultaneously.**
|
||
|
||
#### Pause — eligible_but_blocked
|
||
|
||
Eligible in 1 of 7 real turns:
|
||
- contradictory-evidence turn 1 (phase=focusing + progress=stalled triggers pause rule)
|
||
|
||
In this case, health=healthy simultaneously, so Acknowledge blocks it. The second pause trigger (`health=user_overloaded`) is never met because the assessor never produces that state.
|
||
|
||
**Root cause: same priority conflict as Summarise. One of two rules fires in real data but gets blocked by Acknowledge's earlier position.**
|
||
|
||
### Synthetic Reachability Confirmation
|
||
|
||
All five behaviours are independently reachable when isolated from Acknowledge:
|
||
- ✅ acknowledge — healthy + confident phase
|
||
- ✅ clarify — too_broad health (with low-confidence phase to avoid Acknowledge)
|
||
- ✅ summarise — synthesising/concluding phase (without healthy health)
|
||
- ✅ pause — focusing+stalled or user_overloaded (without healthy health)
|
||
- ✅ continue — no rules match
|
||
|
||
### Classifications
|
||
|
||
| Behaviour | Classification | Primary Cause |
|
||
|---|---|---|
|
||
| Summarise | eligible_but_blocked | Acknowledge priority 1 fires first when health=healthy |
|
||
| Clarify | never_eligible_in_tested_scenarios (reachable only in synthetic) | `too_broad` trigger too narrow for test scenarios; `orienting+low obs` not produced by assessor |
|
||
| Pause | eligible_but_blocked | Acknowledge priority 1 fires first when health=healthy; `user_overloaded` never produced |
|
||
|
||
### Impact on Prior Finding (Exp 39)
|
||
|
||
Experiment 39 concluded "the Acknowledge→Summarise priority conflict prevents Summarise from firing." Experiment 40 confirms this and adds that **Pause faces the same blocking** (1 eligible turn, blocked). Clarify's absence is fundamentally different: its rules are not triggered at all in tested scenarios.
|
||
|
||
This means any fix must address two distinct problems:
|
||
1. Priority conflict affecting Summarise AND Pause (same cause)
|
||
2. Narrow trigger conditions for Clarify and the `user_overloaded` health state
|
||
|
||
### Test Results
|
||
|
||
- `tests/behaviour-selection.reachability.test.js`: 33 passed (new diagnostic file)
|
||
- `tests/behaviour-selection.test.js`: 51 passed (no regressions)
|
||
- `tests/behaviour-selection.real-assessment.test.js`: 16 passed (shared fixtures intact)
|
||
- `tests/investigation-state-assessor.test.js`: 51 passed (assessor unchanged)
|
||
|
||
### Documents Updated
|
||
- `docs/design-evolution-log.md` — this entry
|
||
- `docs/current-handoff.md` — return-to-work note replaced
|
||
|
||
## Experiment 41 — Compare Acknowledge Priority Alternatives (2026-08-06)
|
||
|
||
### Purpose
|
||
|
||
Experiment 40 confirmed Summarise and Pause are eligible_but_blocked by Acknowledge's priority-1 position. Two passive alternatives were compared without modifying production code:
|
||
|
||
**Variant A** — Reorder rules so specific behaviours (Summarise, Pause) evaluate before Acknowledge. The idea is that if a more specific behaviour fires first, it captures the terminal/stalled states where Acknowledge should not fire.
|
||
|
||
**Variant B** — Keep existing priority order but exclude Acknowledge from firing when phase=concluding/synthesising, progress=stalled, or health=user_overloaded. The idea is to gate Acknowledge rather than reorder everything.
|
||
|
||
### Method
|
||
|
||
Both variants were implemented as test-only functions in `tests/behaviour-selection.counterfactual.test.js`. Each variant was evaluated against the same 7 real assessment turns from Experiments 39/40 across 3 scenarios. All five behaviours confirmed independently reachable synthetically. No production rules changed.
|
||
|
||
### Assessor Outputs (7 real turns)
|
||
|
||
| # | Scenario | Turn | Phase (conf) | Progress | Health | Existing |
|
||
|---|---|---|---|---|---|---|
|
||
| 1 | long-investigation | 0 | cannot_determine(low) | cannot_determine | too_narrow | continue |
|
||
| 2 | long-investigation | 3 | focusing(high) | steady | healthy | acknowledge |
|
||
| 3 | long-investigation | 4 | concluding(high) | steady | healthy | acknowledge |
|
||
| 4 | contradictory-evidence | 0 | focusing(high) | cannot_determine | healthy | acknowledge |
|
||
| 5 | contradictory-evidence | 1 | focusing(high) | stalled | healthy | acknowledge |
|
||
| 6 | contradictory-evidence | 2 | focusing(high) | steady | healthy | acknowledge |
|
||
| 7 | short-early | 0 | exploring(low) | cannot_determine | healthy | continue |
|
||
|
||
### Results on Real Scenarios
|
||
|
||
| Turn | Existing | Variant A | Variant B | Change? |
|
||
|---|---|---|---|---|
|
||
| long-investigation t3 | acknowledge | summarise | acknowledge | V-A: side-effect |
|
||
| long-investigation t4 | acknowledge | **summarise** | **summarise** | **convergent ✓** |
|
||
| contradictory-evidence t1 | acknowledge | **pause** | **pause** | **convergent ✓** |
|
||
| All others | unchanged | unchanged | unchanged | — |
|
||
|
||
### Divergence Analysis
|
||
|
||
**Variant A diverges from Variant B at long-investigation turn 3.** Variant A produces `summarise` because its `resolvedNodeCount >= 3 && steady` rule fires at priority 1 without phase context. The assessor confirms this is a focusing-phase state (not synthesising/concluding) where the user needs acknowledgment, not compression. This is a false-positive for summarisation — a side-effect of Variant A's priority reordering.
|
||
|
||
**Variant B correctly preserves Acknowledge** at long-investigation t3 because:
|
||
1. The exclusion list only includes `synthesising`, `concluding`, `stalled`, and `user_overloaded` — not focusing
|
||
2. SummariseV2 itself has a phase gate (`phase.value === "synthesising"`) that prevents false-fire in focusing states
|
||
3. Acknowledge at priority 1 wins because no exclusion applies
|
||
|
||
### Key Findings
|
||
|
||
1. **Both variants converge on the same two genuine changes:** `concluding → summarise` and `stalled → pause`. This was the experiment's primary question, and both approaches answer it correctly.
|
||
|
||
2. **Variant A introduces a false-positive:** The `resolvedNodeCount >= 3 && steady` rule fires in focusing-phase states without phase context, causing premature summarisation when Acknowledge would be more useful.
|
||
|
||
3. **Variant B has cleaner boundaries:** Explicit exclusion conditions prevent unwanted side-effects while preserving Acknowledge's role as the default healthy-state behaviour.
|
||
|
||
4. **Distribution shift (both variants):**
|
||
- Existing: acknowledge 71%, continue 29%
|
||
- Variant A: acknowledge 29%, summarise 29%, pause 14%, continue 29%
|
||
- Variant B: acknowledge 43%, summarise 14%, pause 14%, continue 29%
|
||
- Variant B preserves more Acknowledge because it doesn't remove the default healthy-state behaviour entirely
|
||
|
||
5. **Variant B is architecturally cleaner** for this problem space because it adds a targeted gate to one rule rather than reordering five priority levels — each of which would need individual review for side-effects.
|
||
|
||
### Test Results
|
||
|
||
- `tests/behaviour-selection.counterfactual.test.js`: 44 passed (new diagnostic file)
|
||
- `tests/behaviour-selection.reachability.test.js`: 33 passed (no regressions)
|
||
- `tests/behaviour-selection.real-assessment.test.js`: 16 passed (shared fixtures intact)
|
||
- `tests/behaviour-selection.test.js`: 51 passed (no regressions)
|
||
|
||
### Decision Criteria
|
||
|
||
| Criterion | Variant A | Variant B |
|
||
|---|---|---|
|
||
| Fixes concluding state | ✓ summarise | ✓ summarise |
|
||
| Fixes stalled state | ✓ pause | ✓ pause |
|
||
| No false-positive changes | ✗ long-t3 → summarise | ✓ preserved acknowledge |
|
||
| Implementation complexity | Simple reordering | Small gate function |
|
||
| Maintains Acknowledge for healthy focus states | ? (depends on future review) | ✓ explicit preservation |
|
||
|
||
### Recommendation
|
||
|
||
**Variant B is preferred.** Both variants correctly identify the two genuine changes needed. Variant B has no false-positives, cleaner architectural boundaries (targeted exclusion vs priority reordering), and better preserves the existing Acknowledge default for healthy focusing states where it is appropriate. A recommended implementation would:
|
||
|
||
1. Keep existing priority order
|
||
2. Add `isAcknowledgeExcluded()` function with conditions: phase∈{synthesising, concluding}, progress=stalled, health=user_overloaded
|
||
3. Gate Acknowledge through this exclusion before selecting it at priority 1
|
||
|
||
### Documents Updated
|
||
- `docs/design-evolution-log.md` — this entry
|
||
- `docs/current-handoff.md` — return-to-work note replaced
|
||
|
||
|
||
## Experiment 41 — Conclusion
|
||
|
||
**Variant B was preferred because it changed only the two intended turns without introducing a false-positive in a focusing state. Variant A produced an early summarise in a focusing phase and was discarded. No production rule changed during Experiment 41.** The implementation of Variant B's exclusion gate is the subject of Experiment 42.
|
||
|
||
---
|
||
|
||
## Experiment 42 — Implement Narrow Acknowledge Exclusion (Variant B) (2026-08-06)
|
||
|
||
### Hypothesis
|
||
|
||
Applying a narrow exclusion gate to Acknowledge — excluding it when phase is synthesising or concluding, progress is stalled, or conversation health is user_overloaded — will reduce the two identified false-Acknowledge selections (concluding → summarise, stalled → pause) without introducing any unintended behaviour changes in other tested turns.
|
||
|
||
### Exact Exclusion Rule
|
||
|
||
`isAcknowledgeExcluded(assessment)` returns `true` when:
|
||
- `phase.value` is `synthesising` or `concluding`; OR
|
||
- `progress.value` is `stalled`; OR
|
||
- `conversationHealth.value` is `user_overloaded`.
|
||
|
||
When excluded, Acknowledge does not fire and the selector proceeds to the next priority rule. The gate qualifies the trigger; it does not replace it.
|
||
|
||
### Two Changed Turns
|
||
|
||
| Turn | Scenario | Phase | Progress | Health | Before | After |
|
||
|------|----------|-------|----------|--------|--------|-------|
|
||
| long-investigation t4 | concluding long-investigation | concluding(high) | steady | healthy | acknowledge | **summarise** |
|
||
| contradictory-evidence t1 | stalled contradictory-evidence | focusing(high) | stalled | healthy | acknowledge | **pause** |
|
||
|
||
### Five Preserved Turns
|
||
|
||
| Turn | Scenario | Phase | Progress | Health | Behaviour (unchanged) |
|
||
|------|----------|-------|----------|--------|----------------------|
|
||
| long-investigation t0 | cannot_determine(low) | cannot_determine | too_narrow | continue |
|
||
| long-investigation t3 | focusing(high) | steady | healthy | acknowledge |
|
||
| contradictory-evidence t0 | focusing(high) | cannot_determine | healthy | acknowledge |
|
||
| contradictory-evidence t2 | focusing(high) | steady | healthy | acknowledge |
|
||
| short-early t0 | exploring(low) | cannot_determine | healthy | continue |
|
||
|
||
### Final Behaviour Distribution (7 real assessment turns)
|
||
|
||
- Acknowledge: 3
|
||
- Summarise: 1
|
||
- Pause: 1
|
||
- Continue: 2
|
||
- Clarify: 0
|
||
|
||
### Integration Status
|
||
|
||
The production Behaviour Selection module (`lib/behaviour-selection/behaviour-selector.js`) was changed to include the `isAcknowledgeExcluded()` gate. However, **active user-facing engine behaviour did not change** because Behaviour Selection remains isolated with no runtime caller — it is exported but never imported by any code in the repository.
|
||
|
||
### Clarify Status
|
||
|
||
`Clarify` remains unresolved and was not modified in this experiment. Its trigger conditions (`health=too_broad` or `phase=orienting + obs<3`) require states that no tested scenario produces. This remains an open question for future work.
|
||
|
||
### Selector Output Shape
|
||
|
||
The selector output shape did not change. The exclusion gate returns `null` from `selectAcknowledge`, which is the existing early-return mechanism used when a rule does not match. No new fields, no restructuring of the return object.
|
||
|
||
### Assessor and Fixtures
|
||
|
||
Assessor logic did not change. Fixtures did not change. Priority order did not change.
|
||
|
||
### Test Results
|
||
|
||
All 151 relevant tests passed across:
|
||
- `tests/behaviour-selection.test.js`: 51 (no regressions)
|
||
- `tests/behaviour-selection.reachability.test.js`: 33 (updated for new exclusion gate)
|
||
- `tests/behaviour-selection.counterfactual.test.js`: 44 (from Exp 41, no changes)
|
||
- `tests/behaviour-selection.real-assessment.test.js`: 16 (shared fixtures intact)
|
||
|
||
Tests were not rerun as part of this documentation-only closure. The recorded result comes from the implementation commit (05d3d96).
|
||
|
||
### Limitations
|
||
|
||
- Only seven real assessment turns across three scenarios were evaluated; other investigation domains may exhibit different patterns.
|
||
- `health=user_overloaded` is excluded by rule but never produced by any current assessor fixture — it is untested in practice.
|
||
- `Clarify` remains deferred because no scenario produces the narrow trigger conditions it requires.
|
||
- The selector remains isolated with no runtime caller; there is no live user-facing validation.
|
||
|
||
### Result
|
||
|
||
**Confirmed within the tested scenarios.** Variant B correctly changes only the two intended turns and preserves all five others. No unintended side-effects were observed.
|
||
|
||
### Documents Updated
|
||
|
||
- `docs/design-evolution-log.md` — this entry
|
||
- `docs/current-handoff.md` — return-to-work note replaced
|
||
|
||
## Experiment 43 — Audit Clarify Readiness Signals (2026-08-06)
|
||
|
||
### Hypothesis
|
||
|
||
The existing investigation-state-assessor never produces states that trigger the production Clarify rule in any tested scenario. Clarify is absent from Behaviour Selection not because of a selector defect but because no current fixture represents the genuinely unclear-scoped investigations that its triggers are designed for.
|
||
|
||
### Diagnostic Test File
|
||
|
||
A focused diagnostic test was created at `tests/behaviour-selection.clarify-readiness.test.js` with 31 assertions auditing every turn across all existing assessor and reachability fixtures. It inspects:
|
||
- Phase value distribution (focusing, exploring, concluding, synthesising, deepening, cannot_determine)
|
||
- Conversation health values (healthy, too_narrow, too_broad, user_overloaded)
|
||
- Observation density per turn
|
||
- Clarify eligibility via the exact production rule in `selectClarify`
|
||
|
||
### Audit Scope
|
||
|
||
| Source | Scenarios | Turns Inspected |
|
||
|--------|-----------|-----------------|
|
||
| `investigation-state-assessor.test.js` | 7 | 7 (one per scenario) |
|
||
| `behaviour-selection.reachability.test.js` | 3 | 3 (contradictory-evidence t0, t1, t2) |
|
||
| **Total** | **10** | **10 real-turn assessments** |
|
||
|
||
### Q1 — Does the assessor ever produce `too_broad`?
|
||
|
||
**No.** Zero scenarios across all test fixtures produce `conversationHealth.value === "too_broad"`.
|
||
|
||
The `too_broad` trigger requires `activeUnknownCount > 3 AND resolvedNodeIds.length < 2`. Every existing scenario starts with exactly one active unknown (the single unresolved question the investigation is about), and the assessor never produces a state where more than three unrelated unknowns coexist without resolution.
|
||
|
||
### Q2 — Does the assessor ever produce `phase.value === "orienting"`?
|
||
|
||
**No.** Zero scenarios produce orienting. The five phase values produced by the assessor are: concluding, synthesising, focusing, exploring, deepening, and cannot_determine. **`orienting` is not a possible output of any assessor code path.** It does not appear in `assessPhase()`.
|
||
|
||
### Q3 — Does orienting ever coincide with observation density < 3?
|
||
|
||
**Never applicable.** Since the assessor never produces orienting, this condition cannot arise in real data. The orienting-based Clarify trigger is dead code within the tested scenarios (and likely in production until a scenario change introduces orienting).
|
||
|
||
### Q4 — How many turns are Clarify-eligible?
|
||
|
||
**Zero of 10 turns.** Both Clarify rules evaluate to false for every assessed turn:
|
||
- Rule 1 (`too_broad` health): false in all 10 turns
|
||
- Rule 2 (`orienting + obs<3`): false in all 10 turns (orienting never appears)
|
||
|
||
### Q5 — What are the closest existing signals to a genuine Clarify need?
|
||
|
||
Two signals approach clarification but do not match its intent:
|
||
|
||
| Signal | Turns | Meaning | Maps to Clarify? |
|
||
|--------|-------|---------|-----------------|
|
||
| `too_narrow` health | 1 (long-turn-0) | Insufficient contextual evidence for a narrow investigation | No — too_narrow means "needs more data," not "scope is unclear" |
|
||
| `exploring` phase with low obs density | 1 (complete-turn-0) | Early-stage investigation with sparse observations | No — this signals the start of an investigation, not scope confusion |
|
||
|
||
### Q6 — Signal reliability assessment for future Clarify rule design
|
||
|
||
| Signal | Reliability for Clarify intent |
|
||
|--------|-------------------------------|
|
||
| `too_narrow` health | Low reliability. It reliably indicates insufficient context for question formulation but conflates "too little information" with "unclear scope." The assessor's own description: "The investigation needs more contextual evidence before the current question can be answered effectively." This is about quantity, not clarity. |
|
||
| `exploring` + low obs density | Low reliability. It reliably indicates an early-stage investigation but does not distinguish between "well-scoped investigation in early phase" and "unclear investigation needing anchoring." Both map to exploring. |
|
||
|
||
### Q7 — Is Clarify's absence appropriate for current fixtures?
|
||
|
||
**Yes.** Every existing fixture represents a well-defined, focused investigation with a clear central statement:
|
||
- "Comparing two products before purchase decision" (single question, single dimension)
|
||
- "Evaluating European market entry" (single strategic question)
|
||
- "Evaluating $2M procurement against conflicting expert advice" (single decision context)
|
||
|
||
A genuinely unclear-scoped investigation would need one of:
|
||
- A central statement so vague the system cannot classify it into any phase
|
||
- Multiple unrelated threads at startup with no clear priority anchor
|
||
- Contradictory framing where the situation itself is ambiguous
|
||
|
||
No current fixture represents these states. **Clarify's absence is appropriate because the existing scenarios are genuinely well-scoped, not because the selector is broken.**
|
||
|
||
### Phase Distribution Across All 10 Turns
|
||
|
||
| Phase | Count | Scenarios |
|
||
|-------|-------|-----------|
|
||
| focusing | 7 | comparison t0,t1,t2; long t3; contradictory t0,t1,t2 |
|
||
| cannot_determine | 1 | long t0 |
|
||
| concluding | 1 | long t4 |
|
||
| exploring | 1 | complete t0 |
|
||
|
||
No synthesising, deepening, or orienting phases observed.
|
||
|
||
### Production Clarify Trigger — Exact Rule Match
|
||
|
||
```js
|
||
// selectClarify (behaviour-selector.js lines 65-83)
|
||
function selectClarify(assessment) {
|
||
// Rule A: broad scope detected
|
||
if (assessment.conversationHealth.value === "too_broad") return clarify;
|
||
// Rule B: early orientation with sparse data
|
||
if (assessment.phase.value === "orienting" && assessment.phase.evidence?.observationDensity < 3) return clarify;
|
||
return null;
|
||
}
|
||
```
|
||
|
||
**Rule A trigger:** `conversationHealth.value === "too_broad"` — zero occurrences in tested scenarios.
|
||
**Rule B trigger:** `phase.value === "orienting"` — never produced by assessor; **dead code path.**
|
||
|
||
### Focused Test Results (Experiment 43)
|
||
|
||
- Total tests: **31**
|
||
- Passed: **31**
|
||
- Failed: **0**
|
||
|
||
All diagnostics confirm zero Clarify eligibility across the complete set of real-world fixtures.
|
||
|
||
### Regression / Validation Results
|
||
|
||
| Test File | Tests | Result | Notes |
|
||
|-----------|-------|--------|-------|
|
||
| `tests/behaviour-selection.clarify-readiness.test.js` | 31 | ✓ Pass | New diagnostic file — no regression possible |
|
||
| `tests/behaviour-selection.test.js` | 51 | ✓ Pass | Zero regressions from any prior experiments |
|
||
| `tests/behaviour-selection.reachability.test.js` | 33 | ✓ Pass | Clarify still eligible in 0 real turns; synthetically reachable |
|
||
| `tests/investigation-state-assessor.test.js` | 51 | ✓ Pass | Assessor behavior unchanged |
|
||
|
||
### Limitations
|
||
|
||
- The audit covers all existing test fixtures but not every possible investigation domain. Different problem domains (legal disputes, medical triage, multi-party procurement) may produce different assessor states.
|
||
- `too_broad` requires very specific conditions (>3 active unknowns with <2 resolved) that no current fixture exercises. A fixture designed specifically to trigger it would validate the health classifier path.
|
||
- The orienting phase was never produced by any assessor code path in the entire test suite, suggesting a design gap: either orienting was removed from the assessor without updating the selector, or it was never implemented as an active phase value.
|
||
|
||
### Conclusion
|
||
|
||
**Clarify is absent from Behaviour Selection because no current scenario genuinely needs clarification — not because of a selector defect.** The two production rules are well-formed but their trigger conditions (too_broad health and orienting phase) represent states that the assessor either cannot produce (orienting) or does not produce in any tested fixture (too_broad).
|
||
|
||
**Two distinct issues identified:**
|
||
1. **Dead code path**: The orienting-based Clarify rule never activates because the assessor produces six phase values but none is `orienting`. This is a design inconsistency worth correcting — either add orienting as a real phase or remove that rule from the selector.
|
||
2. **Narrow trigger threshold**: The too_broad condition (`activeUnknownCount > 3 AND resolvedNodeIds < 2`) is validly narrow but never exercised by any fixture. If Clarify should fire earlier in investigations, the threshold should be relaxed; if it should only fire for genuinely lost investigations, it should stay as-is and a dedicated fixture should validate it.
|
||
|
||
### Documents Updated
|
||
|
||
- `docs/design-evolution-log.md` — this entry
|
||
- `docs/current-handoff.md` — return-to-work note replaced
|
||
|
||
## Experiment 44 — Assessor Against Unclear Starting Point (2026-08-06)
|
||
|
||
### Objective
|
||
|
||
Create one deliberately unclear investigation fixture and test whether the existing Investigation State Assessor produces any signal that justifies Clarify.
|
||
|
||
### Hypothesis
|
||
|
||
A deliberately unclear starting scenario may expose one of three outcomes:
|
||
1. The assessor already produces `too_broad`.
|
||
2. The assessor produces another existing signal that reasonably represents the need to clarify.
|
||
3. The assessor has no suitable signal for unclear framing.
|
||
|
||
### Fixture Description
|
||
|
||
**File:** `tests/investigation-state-assessor.unclear-start.test.js` (test-only, not imported anywhere else)
|
||
|
||
The fixture represents:
|
||
- A vague central statement that admits uncertainty: *"The business feels stuck. Sales are uneven, staff are frustrated, customers ask for different things, and I'm not sure what the real problem is."*
|
||
- Five competing unknown threads (customer demand, staff capacity, product direction, pricing, operations) with no priority anchor
|
||
- Only one observation (the only concrete data point)
|
||
- Zero resolved evidence nodes
|
||
- No selected question (no established direction)
|
||
- All existing graph fields only (id, label, description, kind, status, confidence, evidenceIds, dependsOn, affects, childIds)
|
||
- Five `kind: "unknown"` nodes and one `kind: "observation"` node
|
||
|
||
### Returned Assessment Signals
|
||
|
||
| Signal | Value | Confidence |
|
||
|--------|-------|------------|
|
||
| Phase | `cannot_determine` | low |
|
||
| Phase signals | "Insufficient data for phase classification" | — |
|
||
| Progress | `cannot_determine` | low |
|
||
| Progress signals | "Insufficient data for progress assessment" | — |
|
||
| Conversation health | **`too_broad`** | medium |
|
||
| Health signals | "5 active unknowns with fewer than 2 resolved items"; "Investigation may be spreading too thin" | — |
|
||
|
||
Detailed evidence:
|
||
- Phase evidence: resolvedNodeCount=0, activeUnknownCount=5, observationDensity=1, evidenceDepth="shallow"
|
||
- Progress evidence: turnCount=0, recentResolutionsLastTurn=0
|
||
- Health evidence: activeUnknownCount=5, resolvedNodeRatio=null, hasActiveQuestion=false
|
||
|
||
### Clarify Eligibility
|
||
|
||
**Clarify became eligible via Rule A.** The production `selectClarify` rule fires because `conversationHealth.value === "too_broad"`.
|
||
|
||
The production selector (`selectBehaviour`) returned:
|
||
- behaviour: `"clarify"`
|
||
- confidence: `"high"`
|
||
- reason: "Conversation health is too broad — investigation may be spreading too thin. Narrow focus through a specific clarification question."
|
||
|
||
### Interpretation
|
||
|
||
**Classification: `assessor_recognises_unclear_start`**
|
||
|
||
The assessor produced `too_broad` from the unclear-start fixture, which directly maps to Clarify's intent (genuinely unclear scope requiring anchoring). The signal honestly reflects the starting situation: five competing unknowns with no resolved evidence and no established direction.
|
||
|
||
### What the Assessor Recognised
|
||
|
||
1. Multiple active unknowns without sufficient resolution triggered `too_broad` health classification.
|
||
2. The assessor correctly recorded 5 active unknowns in both phase and health evidence sections.
|
||
3. Observation density (1) was correctly reported as shallow.
|
||
4. Phase confidence remained low due to insufficient data for any meaningful classification.
|
||
|
||
### What the Assessor Failed to Recognise
|
||
|
||
1. **`orienting` phase**: Still not produced by the assessor. The orienting-based Clarify rule remains dead code, unchanged from Experiment 43's finding.
|
||
2. **Early-stage clarification need**: The `too_broad` trigger only fires after >3 unknowns accumulate — it does not catch a situation with fewer competing threads that is still genuinely unclear in framing.
|
||
|
||
### Limitations
|
||
|
||
- Only one fixture was tested. Different vague-scenario configurations may produce different results.
|
||
- The `too_broad` trigger depends on having more than 3 active unknowns with fewer than 2 resolved — this specific threshold was exercised, but other boundary conditions (e.g., exactly 4 unknowns, or 5 unknowns with 1 resolved) were not tested.
|
||
- The fixture uses the assessor's existing `too_broad` definition which conflates "many unknowns" with "unclear scope." A genuinely unclear scenario with only 2–3 competing threads may not trigger this signal.
|
||
|
||
### Status
|
||
|
||
Pending Rob's review. Experiment 43 remains closed — its conclusion that a deliberately unclear fixture was required is confirmed by this experiment, which successfully exercises the previously untested `too_broad` health path.
|
||
|
||
### Focused Test Results
|
||
|
||
| Test File | Tests | Result |
|
||
|-----------|-------|--------|
|
||
| `tests/investigation-state-assessor.unclear-start.test.js` | 23 | ✓ Pass |
|
||
|
||
### Regression / Validation Results
|
||
|
||
| Test File | Tests | Result | Notes |
|
||
|-----------|-------|--------|-------|
|
||
| `tests/behaviour-selection.clarify-readiness.test.js` | 31 | ✓ Pass | Zero regressions |
|
||
| `tests/investigation-state-assessor.test.js` | 51 | ✓ Pass | Zero regressions |
|
||
| `tests/behaviour-selection.test.js` | 51 | ✓ Pass | Zero regressions |
|
||
|
||
### Production Assessor Status
|
||
|
||
**Unchanged.** The assessor produced the expected `too_broad` signal from the unclear fixture, confirming the health classifier path works correctly. No code was modified.
|
||
|
||
### Closure
|
||
|
||
Experiment 44 is **closed**. Conclusion: the assessor recognises an extreme unclear start; too_broad and Clarify are reachable; the useful boundary remained unknown.
|
||
|
||
---
|
||
|
||
## Experiment 45 — Where Does "Too Broad" Begin? (2026-08-06)
|
||
|
||
### Objective
|
||
|
||
Test how the existing assessor's `too_broad` threshold behaves as an unclear starting scenario grows from two competing unknowns to five, all with identical base inputs. Passive boundary experiment only — no production code changes.
|
||
|
||
### Hypothesis
|
||
|
||
| Active unknowns | Expected health |
|
||
|---:|---|
|
||
| 2 | not `too_broad` |
|
||
| 3 | not `too_broad` |
|
||
| 4 | `too_broad` |
|
||
| 5 | `too_broad` |
|
||
|
||
### Fixture-Control Method
|
||
|
||
One test-only fixture builder creates the same vague starting situation varying only the number of competing unknowns:
|
||
- Same central statement; same single observation; zero resolved items (base); no selected question; no active direction; same node shapes and confidence values.
|
||
- Only the count of `kind: "unknown"` nodes differs.
|
||
|
||
### Results: Two Through Five Active Unknowns
|
||
|
||
| Active unknowns | Health | Confidence | Phase | Progress | Clarify eligible | Selector |
|
||
|---:|---|---|---|---|---:|---|
|
||
| 2 | `cannot_determine` | low | `cannot_determine` (low) | `cannot_determine` (low) | No | continue (low) |
|
||
| 3 | `cannot_determine` | low | `cannot_determine` (low) | `cannot_determine` (low) | No | continue (low) |
|
||
| 4 | `too_broad` | medium | `cannot_determine` (low) | `cannot_determine` (low) | Yes | clarify (high) |
|
||
| 5 | `too_broad` | medium | `cannot_determine` (low) | `cannot_determined` (low) | Yes | clarify (high) |
|
||
|
||
### Results: Four Unknowns + Resolved Items
|
||
|
||
| Active unknowns | Resolved | Health | Confidence | Clarify eligible |
|
||
|---:|---:|---|---|---:|
|
||
| 4 | 0 | `too_broad` | medium | Yes |
|
||
| 4 | 1 | `too_broad` | medium | Yes |
|
||
| 4 | 2 | `cannot_determine` | low | No |
|
||
|
||
### Human-Sense Review
|
||
|
||
- **Two competing threads:** Still appears ambiguous rather than clearly manageable. The assessor returns `cannot_determine`, not `healthy`. This is honest — two unknowns with one observation and no question genuinely leave the state unclear.
|
||
- **Three competing threads:** Appears ambiguous or already confused. The assessor still returns `cannot_determine`. This feels correct — three competing threads with minimal context is genuinely uncertain, not healthy.
|
||
- **Four competing threads:** Appears genuinely too broad. The transition from three (uncertain) to four (too_broad) feels believable — a real investigator would start losing focus at this point.
|
||
- **Five competing threads:** Clearly justifies clarification. Matches Experiment 44's result; no surprise.
|
||
- **Transition between three and four:** Understandable. Three threads with one observation is "not enough to decide"; four adds the tipping point where the spread becomes problematic.
|
||
- **Confidence language:** `too_broad` confidence is `medium` for both four and five unknowns. The signals are specific ("4 active unknowns with fewer than 2 resolved items"), so medium confidence is honest — it does not overstate certainty.
|
||
|
||
### Boundary Classification
|
||
|
||
| Transition | Classification | Rationale |
|
||
|---|---|---|
|
||
| 2→3 | `believable` | Both remain `cannot_determine`; the gap between "manageable" and "confused" genuinely sits around here |
|
||
| 3→4 | `believable` | Four competing threads with no resolution is a believable tipping point for losing focus |
|
||
| Resolution threshold (<2 resolved) | `believable` | The binary boundary (1 stays too_broad, 2 clears it) aligns with the design intent of "sufficient context to narrow" |
|
||
|
||
### Usefulness of Active-Unknown Count as a Proxy
|
||
|
||
Active-unknown count acts as a **useful but coarse** proxy for scope confusion. It works because:
|
||
1. In the tested scenarios, more unknowns directly correlates with genuine ambiguity.
|
||
2. The resolved-item gate prevents premature too_broad flags on investigations making progress.
|
||
3. It avoids subjective measurement of "how confused is the user."
|
||
|
||
However, it cannot distinguish between:
|
||
- Four unknowns about one decision (genuinely broad) versus four unknowns across a multi-decision comparison (expected).
|
||
- A well-formed investigation with natural branching versus an unfocused investigation losing its way.
|
||
|
||
### Questionable or Unsupported Findings
|
||
|
||
1. **Health defaults to `cannot_determine` rather than `healthy` for 2–3 unknowns.** This is mechanically correct (no active question means the "healthy" rule doesn't fire) but arguably should produce `healthy` when the state is simply an early-stage investigation with a few threads, not just insufficient data.
|
||
2. **The experiment uses synthetic boundary fixtures.** These cannot validate whether a real user would feel the same confusion at exactly these thresholds. The boundary may be mechanically correct but conceptually misaligned in some domains.
|
||
3. **All unknowns share identical labels and confidence values.** A more differentiated scenario (some high-confidence, some low) might behave differently.
|
||
|
||
### Experiment Conclusion
|
||
|
||
**Current boundary is mechanically clear but conceptually uncertain.**
|
||
|
||
The threshold sits exactly between three and four active unknowns. This mechanical boundary behaves predictably: no too_broad below it, too_broad above it, resolved items gate correctly. However, whether this aligns with genuine user confusion (not just code behaviour) cannot be determined from synthetic fixtures alone. The experiment confirms that Clarify switches on at the same boundary as too_broad, and that resolving two items does switch too_broad off.
|
||
|
||
### Limitations
|
||
|
||
- Synthetic fixture only; no real-user validation possible from this experiment.
|
||
- All unknowns have identical shapes and confidence — real scenarios mix high/low confidence differently.
|
||
- Only one central statement used; different domains may require different thresholds.
|
||
- Does not test whether the `cannot_determine` health for 2–3 unknowns is a bug or a feature.
|
||
|
||
### Status
|
||
|
||
Pending Rob's review. No production behaviour changed. The next logical step would be: (a) validate whether `cannot_determine` health for 2–3 unknowns should instead be `healthy`, or (b) test real-user scenarios to confirm the three→four boundary feels right in practice.
|
||
|
||
### Focused Test Results
|
||
|
||
| Test File | Tests | Result |
|
||
|-----------|-------|--------|
|
||
| `tests/investigation-state-assessor.too-broad-boundary.test.js` | 32 | ✓ Pass |
|
||
|
||
### Regression / Validation Results
|
||
|
||
| Test File | Tests | Result | Notes |
|
||
|-----------|-------|--------|-------|
|
||
| `tests/investigation-state-assessor.unclear-start.test.js` | 23 | ✓ Pass | Zero regressions |
|
||
| `tests/behaviour-selection.clarify-readiness.test.js` | 31 | ✓ Pass | Zero regressions |
|
||
| `tests/investigation-state-assessor.test.js` | 51 | ✓ Pass | Zero regressions |
|
||
| `tests/behaviour-selection.test.js` | 51 | ✓ Pass | Zero regressions |
|
||
|
||
### Production Assessor Status
|
||
|
||
**Unchanged.** No code was modified. The assessor produced the expected results from synthetic boundary fixtures only.
|
||
|
||
---
|
||
|
||
## Experiment 45 — Closure
|
||
|
||
The threshold is mechanically clear; active-unknown count is a coarse proxy; semantic coherence remained untested.
|
||
|
||
---
|
||
|
||
## Experiment 46 — Does "Too Broad" Mean Too Many Questions, or Too Many Unrelated Questions? (2026-08-06)
|
||
|
||
### Objective
|
||
|
||
Test whether the current `too_broad` assessment can distinguish between:
|
||
- several questions that all support one clear investigation; and
|
||
- several questions that belong to competing, unrelated lines of enquiry.
|
||
|
||
This is a passive diagnostic experiment. No production code changes.
|
||
|
||
### Hypothesis
|
||
|
||
Two fixtures with the same number of active unknowns may receive the same `too_broad` result even when one is coherent and the other is genuinely scattered. If so, active-unknown count is a useful warning signal but not enough on its own to describe scope confusion.
|
||
|
||
### Context Pack Used
|
||
|
||
Engine Experiment Work pack (Pack 1). Documents loaded:
|
||
- `docs/current-project-state.md`, `docs/current-working-principles.md`, `.claude/architecture-guardrails.md`, `docs/current-implementation-verification.md`
|
||
- `lib/assessment/investigation-state-assessor.js` (conversation-health logic only)
|
||
- `lib/behaviour-selection/behaviour-selector.js` (Clarify rule only)
|
||
- `tests/investigation-state-assessor.too-broad-boundary.test.js`
|
||
- `tests/investigation-state-assessor.unclear-start.test.js`
|
||
- Experiment 45 section in `docs/design-evolution-log.md`
|
||
|
||
No additional documents loaded.
|
||
|
||
### Controlled Structural Variables
|
||
|
||
Both fixtures share identical structural properties:
|
||
- 4 active unknown nodes
|
||
- 0 resolved nodes
|
||
- 1 observation node (status=known, confidence=medium)
|
||
- No selected question
|
||
- No active direction / central decision node
|
||
- Zero edges (no dependency or relationship data)
|
||
- Total node count: 5
|
||
- Identical node shapes and confidence values
|
||
|
||
### Coherent Fixture Summary
|
||
|
||
Central topic: "Should we launch the new service in the North West?"
|
||
|
||
Four unknowns all contributing to one decision:
|
||
1. Whether customer demand exists in the North West region
|
||
2. What price point the North West market would accept
|
||
3. Whether delivery infrastructure can support the North West region
|
||
4. Whether regulatory requirements allow operation in the North West
|
||
|
||
All four are legitimate, related questions about a single investigation. A human reviewer would classify this as a well-structured early investigation, not a confused one.
|
||
|
||
### Scattered Fixture Summary
|
||
|
||
Central topic: "The business feels stuck and I do not know where to begin."
|
||
|
||
Four unknowns from competing, unrelated threads:
|
||
1. Whether customer demand has shifted toward cheaper alternatives (customer strategy)
|
||
2. Whether staff conflict is the primary cause of reduced productivity (HR/operations)
|
||
3. Whether relocating the office would attract a different talent pool (real estate/recruiting)
|
||
4. Whether product pricing is aligned with competitor offerings (product/marketing)
|
||
|
||
Each unknown belongs to a separate domain of enquiry. A human reviewer would classify this as genuinely scattered — no clear shared decision target.
|
||
|
||
### Assessor and Selector Results
|
||
|
||
| Dimension | Coherent Fixture | Scattered Fixture |
|
||
|---|---|---|
|
||
| Phase | `cannot_determine` (low) | `cannot_determine` (low) |
|
||
| Progress | `cannot_determine` (low) | `cannot_determine` (low) |
|
||
| Health | `too_broad` (medium) | `too_broad` (medium) |
|
||
| Active unknown count | 4 | 4 |
|
||
| Resolved count | 0 | 0 |
|
||
| Clarify eligible | Yes | Yes |
|
||
| Selector behaviour | clarify (high) | clarify (high) |
|
||
|
||
### Key Findings
|
||
|
||
1. **Both fixtures return `too_broad`** — identical health result despite one being coherent and one scattered.
|
||
2. **Clarify becomes eligible in both** via Rule A (health === too_broad). Identical eligibility.
|
||
3. **The assessor does not distinguish coherent breadth from scattered breadth anywhere** — all assessed fields are identical between fixtures (JSON comparison confirmed).
|
||
4. **Existing dependency or relationship fields do not influence the health result** — the `too_broad` rule at line 450 references only `activeUnknownCount` and resolved count, never edges, dependsOn, affects, or childIds.
|
||
5. **Active-unknown count alone determines too_broad in both cases** — 4 > 3 and resolved < 2 triggers the same result regardless of semantic coherence.
|
||
|
||
### Human-Sense Review
|
||
|
||
- **Coherent fixture:** `too_broad` is **questionable**. Four unknowns contributing to one decision is breadth, not confusion. The label conflates "many questions" with "scattered focus."
|
||
- **Scattered fixture:** `too_broad` is **believable**. Four unrelated threads genuinely represent scope confusion. The label matches plain-English intuition.
|
||
|
||
### Was Coherence Detected?
|
||
|
||
**No.** The assessor produces identical results for both fixtures. It has no mechanism to detect whether active unknowns share a common decision target or belong to competing threads. Only the count (4) and resolution status (0) matter.
|
||
|
||
### Limitations
|
||
|
||
- Two synthetic fixtures; cannot validate against real-user scenarios or real-domain nuance.
|
||
- Zero edges means we did not test whether adding graph relationships would change results (that is outside scope).
|
||
- The 3→4 boundary was not re-tested here; it was established in Experiment 45.
|
||
- Synthetic labels may not capture how humans distinguish coherent from scattered breadth in practice.
|
||
|
||
### Conclusion
|
||
|
||
**Count is useful but cannot distinguish coherence.** Active-unknown count produces the correct signal for both coherent and scattered investigations, but for the wrong reason in the coherent case. The `too_broad` label is mechanically predictable but semantically imprecise — it flags breadth regardless of whether that breadth has structure.
|
||
|
||
### Questionable or Unsupported Findings
|
||
|
||
1. Both fixtures have 0 resolved items, which also forces phase and progress to `cannot_determine`. This makes the fixtures structurally very early-stage; a real investigation would likely have some resolved context by the time it accumulates four unknowns.
|
||
2. The "questionable" classification for the coherent fixture is a human judgment — one person might judge four related questions as genuinely manageable, not too broad.
|
||
|
||
### Status
|
||
|
||
**Closed.** Rob reviewed and confirmed the hypothesis: graph relationship structure provides a testable coherence signal that the existing assessor ignores.
|
||
|
||
### Focused Test Results
|
||
|
||
| Test File | Tests | Result |
|
||
|-----------|-------|--------|
|
||
| `tests/investigation-state-assessor.scope-coherence.test.js` | 47 | ✓ Pass |
|
||
|
||
### Regression / Validation Results
|
||
|
||
| Test File | Tests | Result | Notes |
|
||
|-----------|-------|--------|-------|
|
||
| `tests/investigation-state-assessor.too-broad-boundary.test.js` | 32 | ✓ Pass | Zero regressions |
|
||
| `tests/investigation-state-assessor.unclear-start.test.js` | 23 | ✓ Pass | Zero regressions |
|
||
| `tests/investigation-state-assessor.test.js` | 51 | ✓ Pass | Zero regressions |
|
||
| `tests/behaviour-selection.test.js` | 51 | ✓ Pass | Zero regressions |
|
||
|
||
### Production Assessor Status
|
||
|
||
**Unchanged.** The assessor produced identical results for both fixtures, confirming it uses only structural counts. No code was modified.
|
||
|
||
## Experiment 47 — Shared-Anchor Coherence Diagnostic (2026-08-06)
|
||
|
||
### Objective
|
||
|
||
Test whether existing graph relationships (`dependsOn`, `affects`, `parentId`, `childIds` on nodes; `fromNodeId`/`toNodeId` + `relationship` on edges) can distinguish coherent investigations (multiple unknowns sharing one anchor) from scattered investigations (multiple unknowns with separate anchors). This builds on Exp 46's finding that count alone cannot make this distinction.
|
||
|
||
This is a passive diagnostic experiment. No production code changes.
|
||
|
||
### Hypothesis
|
||
|
||
An existing SituationGraph for a coherent investigation will show a structural pattern — multiple unknown nodes referencing the same anchor node — that does not appear in scattered investigations where each unknown references a different anchor or no anchor at all. A diagnostic inspection of relationship fields can detect this pattern without modifying the assessor or introducing new scoring logic.
|
||
|
||
### Context Pack Used
|
||
|
||
Engine Experiment Work pack (Pack 1). Documents loaded:
|
||
- `docs/current-project-state.md`, `docs/current-working-principles.md`, `.claude/architecture-guardrails.md`, `docs/current-implementation-verification.md`
|
||
- `lib/assessment/investigation-state-assessor.js` (to verify assessor output)
|
||
- `tests/investigation-state-assessor.scope-coherence.test.js` (Exp 46, for context)
|
||
- Experiment 45 and 46 sections in `docs/design-evolution-log.md`
|
||
|
||
No additional documents loaded.
|
||
|
||
### Three Controlled Fixtures
|
||
|
||
| Property | Fixture A (shared) | Fixture B (separate) | Fixture C (none) |
|
||
|---|---|---|---|
|
||
| Nodes | 6 (1 obs + 1 ctx + 4 unk) | 6 (1 obs + 1 ctx + 4 unk) | 6 (1 obs + 1 ctx + 4 unk) |
|
||
| Edges | 5 | 1 | 0 |
|
||
| Active unknowns | 4 | 4 | 4 |
|
||
| Resolved | 0 | 0 | 0 |
|
||
| Observations | 1 | 1 | 1 |
|
||
| Relationship pattern | All unknowns reference ctx-1 | Each unknown references ctx-1 differently (or not at all) | No relationship fields populated |
|
||
| Diagnostic result | `shared_anchor` → [ctx-1] | `separate_anchors` → [ctx-1] | `insufficient_data` → [] |
|
||
|
||
### Relationship Fields Inspected by the Diagnostic Helper
|
||
|
||
The test-only helper `inspectSharedUnknownAnchor` inspects:
|
||
1. **`dependsOn`** on unknown nodes — direct dependency to an anchor
|
||
2. **`affects`** on unknown nodes — inverse relationship (unknown targets the decision/anchor)
|
||
3. **`parentId`** on unknown nodes — hierarchical parent reference
|
||
4. **`childIds`** on existing nodes — inverse child reference from anchor side
|
||
5. **Edge `fromNodeId`/`toNodeId` + `relationship`** — directional support edges between unknowns and anchors
|
||
|
||
The helper collects all referenced node IDs from these fields across all active unknowns, checks for a common intersection (shared_anchor), separate union (separate_anchors), or no data (insufficient_data).
|
||
|
||
### Existing-Scenario Results
|
||
|
||
Inspected three real scenarios from Experiments 39-46:
|
||
- **comparison-turn-2** (Exp 39/41/45 path): `insufficient_data` — fewer than two active unknowns
|
||
- **long-turn-3** (Exp 45 path): `insufficient_data` — fewer than two active unknowns
|
||
- **live-ollama-state** (Exp 46 test shape): `insufficient_data` — fewer than two active unknowns
|
||
|
||
All three return `insufficient_data`, confirming that real investigation data so far lacks the relationship structure needed for coherence detection. The diagnostic helper requires at least two active unknowns to run, and even then the existing data has no populated relationship fields on unknown nodes.
|
||
|
||
### Assessor Output Identity Verification
|
||
|
||
All three fixtures produce identical assessor output because:
|
||
1. Identical total node count (6) → same `scoreToConfidence(totalNodes)`
|
||
2. Identical active unknown count (4) and resolved count (0) → same health, phase, progress
|
||
3. The assessor does not inspect any relationship fields in its `too_broad` rule
|
||
|
||
### Key Findings
|
||
|
||
1. **The diagnostic helper successfully distinguishes all three fixtures** — shared_anchor vs separate_anchors vs insufficient_data works correctly against controlled data.
|
||
2. **All three fixtures return `too_broad` from the assessor** — identical health, phase, progress, Clarify eligibility, and selector behaviour (clarify) across all fixtures.
|
||
3. **Existing real-scenario graphs lack relationship structure on unknowns** — all three tested scenarios from Experiments 39-46 return `insufficient_data`. Unknown nodes have empty/missing `dependsOn`, `affects`, `parentId`, and `childIds` fields in current production data.
|
||
4. **The assessor's `too_broad` rule at line 450 does not use any relationship fields** — only `activeUnknownCount > 3 && resolved < 2`. The diagnostic result does not affect the output (confirmed by JSON comparison).
|
||
|
||
### Limitations
|
||
|
||
- One test-only helper; no production integration attempted or required.
|
||
- Existing-scenario results reflect a sample of three scenarios from Experiments 39-46 — larger datasets may contain relationship data not present in these fixtures.
|
||
- The diagnostic uses graph topology (shared vs separate anchors) but does not attempt semantic analysis of unknown labels/descriptions. Coherence may have additional signals beyond structural sharing.
|
||
- No new graph mutation or schema changes were made; the experiment relies entirely on existing fields.
|
||
|
||
### Conclusion
|
||
|
||
**A coherence signal exists in the data model.** A diagnostic helper inspecting relationship topology can distinguish shared-anchor from scattered investigations with controlled fixtures. However, real-scenario graphs lack populated relationship fields on unknown nodes, so the signal is currently undetectable in production data. This means the gap is not purely in assessment logic — it also requires upstream data quality: when an investigation adds new unknowns, their `dependsOn`/`affects` relationships must be populated to make the coherence signal visible.
|
||
|
||
### Status
|
||
|
||
Pending Rob's review. No production code or graph schema modified.
|
||
|
||
### Focused Test Results
|
||
|
||
| Test File | Tests | Result |
|
||
|-----------|-------|--------|
|
||
| `tests/investigation-state-assessor.shared-anchor.test.js` | 26 | ✓ Pass |
|
||
|
||
### Regression / Validation Results
|
||
|
||
| Test File | Tests | Result | Notes |
|
||
|-----------|-------|--------|-------|
|
||
| `tests/investigation-state-assessor.scope-coherence.test.js` | 47 | ✓ Pass | Zero regressions |
|
||
| `tests/investigation-state-assessor.test.js` | 51 | ✓ Pass | Zero regressions |
|
||
|
||
### Production Assessor Status
|
||
|
||
**Unchanged.** The assessor produced identical results across all three fixtures (verified by JSON comparison), confirming it does not use relationship fields in its assessment.
|
||
|
||
## Experiment 48 — Audit Unknown Relationship Population (2026-08-06)
|
||
|
||
Experiment 48 was a passive implementation audit asking whether the active graph-construction path actually populates relationship information on unknown nodes that could later support a shared-anchor coherence check (the signal discovered in Experiment 47).
|
||
|
||
**Constraints:** No production code changes. No schema changes. No assessor or test modifications. Only one new test file created. Three cases audited: (A) multiple unknowns from one investigation, (B) unknowns across separate updates, (C) child/decomposed unknowns if supported.
|
||
|
||
### Audit Findings
|
||
|
||
| Production Path | Populates `dependsOn`? | Populates `affects`? | Populates `parentId`? | Edges Created? |
|
||
|---|---|---|---|---|
|
||
| **Path 1: `buildInitialGraph`** | ✗ — always empty `[]` | ✗ — always empty `[]` | ✗ — always `null` | ✓ (to summary node, relationship=`depends_on`) |
|
||
| **Path 2: Emergent unknowns via `buildEmergentReasoningUnknown`** | ✓ — populated with `relatedNodeIds` | ✓ — set to `reasoningState` label | ✓ — set to `relationshipNode?.id ?? null` | ✓ (with `fromNodeId`, `toNodeId`, `relationship`) |
|
||
| **Path 3: Decomposition children via `buildCompositeUnknownChildren`** | ✓ — from template's `dependsOnLabels` | N/A (not set here) | ✓ — set to `parentNode.id` | ✓ (with relationship) |
|
||
|
||
Additionally, `applyGraphUpdate()` auto-creates/updates `dependsOn` and `childIds` arrays when edges are added (schema enforcement), but does NOT populate `affects` or `parentId`.
|
||
|
||
### Focused Test Results
|
||
|
||
| Test File | Tests | Result |
|
||
|-----------|-------|--------|
|
||
| `tests/graph/unknown-relationship-population.test.js` | 16 | ✓ Pass |
|
||
|
||
**Case A (multiple unknowns from one investigation):** 3 unknown nodes created. All have empty relationship fields (`dependsOn: []`, `affects: []`, `parentId: null`, `childIds: []`). Edges exist to summary node. **Diagnosis: insufficient_data for shared-anchor detection.**
|
||
|
||
**Case B (unknowns across separate updates):** After applying one resolved update via `applyValidatedProposal`, fewer than two active unknowns remain in the fixture. The path IS exercised (production code runs correctly) but only creates emergent unknowns when there are comparable observations to compare — a single-resolution scenario does not trigger this.
|
||
|
||
**Case C (child/decomposed unknowns):** Not supported without additional setup. Decomposition (`runDeterministicDecomposition`) requires an active unknown with a compound question selected. Neither Case A nor the tested Case B update path triggers decomposition. The production code exists and IS correct, but is only reachable through a multi-turn flow not exercised by this audit's fixture construction.
|
||
|
||
### Answering the Seven Questions
|
||
|
||
1. **Does buildInitialGraph populate dependsOn/affects/parentId on unknown nodes?** No — all three are empty/null. Only edges exist linking unknowns to summary node.
|
||
|
||
2. **Does applyValidatedProposal populate relationship fields when it creates new unknowns?** Yes — `buildEmergentReasoningUnknown` populates both `dependsOn` and `parentId`, and edges with proper `fromNodeId`/`toNodeId`/`relationship`. `buildCompositeUnknownChildren` (decomposition) also populates `parentId`.
|
||
|
||
3. **Does the existing-production path support creating graphs with multiple unknowns having a shared-anchor topology?** Partially — only when emergent reasoning is triggered by comparable observations within a single update. Initial graph build does not produce shared anchors. Decomposition children share parent as anchor but require multi-turn flow to reach.
|
||
|
||
4. **Can the diagnostic helper correctly classify graphs produced by real production paths?** Only for Case B-style outputs where at least two active unknowns have populated `dependsOn` or `affects` arrays pointing to the same node. For Case A (initial build), it returns `separate_anchors` if nodes have edge-derivable references, or `insufficient_data` if no cross-references exist at all.
|
||
|
||
5. **Which production path creates usable shared-anchor data?** Only emergent unknown creation via `buildEmergentReasoningUnknown` in `applyValidatedProposal`. This occurs when the system detects comparable observations and classifies their relationship as a reasoning state (confirmed, likely_inference, or uncertain).
|
||
|
||
6. **Is there any gap between what synthetic fixtures can represent and what production code actually produces?** Yes — synthetic fixtures manually set relationship fields to match intent. Production code only populates them through emergent reasoning when specific comparison conditions are met. The gap is not in the schema (fields exist) but in the triggering logic for their population.
|
||
|
||
7. **What data quality improvement enables shared-anchor detection?** Ensuring that whenever `buildInitialGraph` creates multiple unknowns, they inherit a common reference from the reconstruction input — either by having a shared contradiction node or a central summary node whose ID is stored in each unknown's `dependsOn`. Currently only edges point to the summary; the edge-to-field conversion would need to happen in Path 1.
|
||
|
||
### Evaluation Conclusion
|
||
|
||
**Insufficient Data** — The production path *does* populate relationship fields correctly when it creates emergent unknowns (Path 2), but shared-anchor detection requires at least two active unknowns with shared references, and the initial build path (Path 1) produces empty relationship fields exclusively. Shared-anchor coherence is structurally supportable in existing data only through the emergent-unknown path, which requires a multi-turn scenario to reach within this audit's constraints.
|
||
|
||
### Pending Rob's review. No production code or graph schema modified.
|
||
|
||
**Commit:** pending (experiment: audit unknown relationship population)
|
||
|
||
## Experiment 49 — Test Production Shared-Anchor Pattern (2026-08-07)
|
||
|
||
Experiment 49 asked whether any sequence of real production updates creates two or more active unknowns that reference the same populated relationship anchor. No production code changed. Only a new test file and diagnostic.
|
||
|
||
### Approach
|
||
|
||
Three production-path scenarios tested via `applyValidatedProposal`:
|
||
- **Case A**: Start with comparable observations + existing unknown → resolve it (triggers emergent reasoning) → then resolve the next active unknown → inspect for shared anchor between remaining unknowns.
|
||
- **Case B**: Identical approach from a separate fixture baseline.
|
||
- **Cases C–F**: Diagnostic controls — verified shared-anchor detection works on controlled fixtures, schema compliance holds, decomposition children share parent anchor correctly, and resolving one node doesn't mutate another's fields (immunity).
|
||
|
||
### Results
|
||
|
||
**All 36 tests pass.** The production-path cases (A & B) consistently returned `separate_anchors` or `insufficient_data`, not `shared_anchor`. Key observations:
|
||
|
||
- After first update in both Cases A and B: only one active unknown typically remains — the diagnostic correctly returns `insufficient_data` (< 2 active).
|
||
- When two active unknowns do exist after emergent reasoning, they reference *different* anchor nodes (separate anchors), not the same one.
|
||
- The diagnostic correctly identifies shared anchors on controlled fixtures (Cases C & D pass as expected).
|
||
- Schema compliance: all production-created nodes and edges pass `situationNodeSchema`/`situationEdgeSchema` validation.
|
||
|
||
### Why No Shared Anchor Emerges
|
||
|
||
The production flow creates at most one emergent reasoning unknown per update, via `buildEmergentReasoningUnknown`. For two unknowns to share an anchor, they would need to independently reference the same relationship node — but each call generates a unique ID and references different source nodes. The path exists (via parentId/populated dependsOn) but the *triggering logic* in `applyValidatedProposal` never produces coexisting active unknowns that point to the same anchor in any tested scenario.
|
||
|
||
### Answering the Seven Questions
|
||
|
||
1. **Can two active unknowns share an anchor via production updates?** No — not in any tested sequence. Each emergent reasoning creates a new unique node with distinct references.
|
||
|
||
2. **Does the diagnostic distinguish shared vs scattered patterns when both exist?** Yes (Cases C, D confirm). It returns `shared_anchor` for identical parentId/dependsOn intersections and `separate_anchors` otherwise.
|
||
|
||
3. **Is shared-anchor detection structurally possible in existing data?** Yes — fields populate correctly via Path 2 (emergent reasoning) and Path 3 (decomposition). The gap is not capability but triggering conditions.
|
||
|
||
4. **What production sequence would be needed to test this further?** A multi-turn flow where two independent investigations on the same relationship node trigger concurrent emergent reasoning before either unknown is resolved.
|
||
|
||
5. **Which production path creates usable shared-anchor data?** Path 2 (emergent reasoning) and Path 3 (decomposition children) both populate fields correctly, but neither produces coexisting anchors in tested scenarios.
|
||
|
||
6. **Is there a gap between what synthetic fixtures can represent and what production actually produces?** Yes — synthetic fixtures set relationship fields directly; production requires specific comparative observation triggers to populate them.
|
||
|
||
7. **What data quality improvement enables shared-anchor detection?** The existing emergent-reasoning path already works. A multi-turn scenario with coexisting unresolved unknowns referencing the same relationship node would be needed to verify shared-anchor coherence end-to-end.
|
||
|
||
### Evaluation Conclusion
|
||
|
||
**No shared anchor found in production update sequences tested.** Both Cases A and B returned `separate_anchors` or `insufficient_data`. The structural capability exists (fields populate correctly via emergent reasoning), but the triggering logic never produces coexisting active unknowns referencing the same anchor within a single testable flow. Shared-anchor coherence is theoretically supportable but empirically unobserved in tested production sequences.
|
||
|
||
### Test Results Summary
|
||
|
||
| Test File | Tests | Passed |
|
||
|---|---|---|
|
||
| `shared-anchor-production-path.test.js` (Exp 49) | 36 | 36 |
|
||
| `unknown-relationship-population.test.js` (Exp 48) | 16 | 16 |
|
||
| `investigation-state-assessor.shared-anchor.test.js` (Exp 47) | 26 | 26 |
|
||
|
||
### Pending Rob's review. No production code or graph schema modified.
|
||
|
||
**Commit:** pending (experiment: test production shared-anchor pattern)
|
||
|
||
## Experiment 50 — Are Shared Graph Edges Meaningful Coherence, or Just Generic Wiring? (2026-08-07)
|
||
|
||
Experiment 50 tested whether the shared edge structure created by `buildInitialGraph` tells us that unknowns belong to one coherent investigation, or merely reflects standard graph construction plumbing. This was a passive diagnostic — no production code changed.
|
||
|
||
### Approach
|
||
|
||
Two test-only reconstruction inputs passed through the identical real `buildInitialGraph` path:
|
||
- **Case A (Coherent)**: One clear decision ("expand into North West") with four domain-aligned unknowns (demand, pricing, delivery capacity, regulatory requirements).
|
||
- **Case B (Scattered)**: One vague statement ("business feels stuck") with four unrelated unknowns (customer demand shift, staff conflict, office relocation, product pricing).
|
||
|
||
A test-only helper `inspectUnknownEdgeAnchors` inspected for each graph: directly connected node IDs, edge relationship/type, whether all unknowns connect to one common node, the anchor's node kind, and whether the anchor is specific or generic. Three existing production-backed fixtures (from Exp 48/Exp 39) were also audited.
|
||
|
||
### Coherent Input Edge Result
|
||
|
||
- Unknown count: 4
|
||
- Edge count: 4 (one `depends_on` per unknown)
|
||
- Common edge anchor: one node, kind=`state`, label = reconstruction.summary
|
||
- Diagnostic result: `shared_generic_anchor`
|
||
- Node-level relationship fields: all empty (dependsOn=[], affects=[], parentId=null)
|
||
|
||
### Scattered Input Edge Result
|
||
|
||
- Unknown count: 4
|
||
- Edge count: 4 (one `depends_on` per unknown)
|
||
- Common edge anchor: one node, kind=`state`, label = reconstruction.summary
|
||
- Diagnostic result: `shared_generic_anchor`
|
||
- Node-level relationship fields: all empty (dependsOn=[], affects=[], parentId=null)
|
||
|
||
### Cross-Case Comparison
|
||
|
||
Both coherent and scattered inputs produced **identical edge topology**: every unknown connects via a `depends_on` edge to the same summary node. The anchor is always kind=`state`. No structural difference exists between them in production-created graphs.
|
||
|
||
### Common Anchors Found
|
||
|
||
In all cases tested (both Exp 50 cases plus three existing production-backed fixtures), shared anchors are summary/situation nodes created from `reconstruction.summary`. Kind is always `state`. They serve as the generic structural container for every initial unknown, regardless of whether the unknowns are semantically coherent.
|
||
|
||
### Common Anchor Node Types
|
||
|
||
`state` — this is the reconstruction summary node. It functions as a structural container/wiring target in the production graph, not as a subject-matter-specific anchor.
|
||
|
||
### Edge Relationship Labels Observed
|
||
|
||
`depends_on` (from unknown → summary) and `supports` (from observation/state → summary). Neither label carries semantic coherence information.
|
||
|
||
### Node-Level Relationship Fields Observed
|
||
|
||
Empty from `buildInitialGraph`: all active unknowns have `dependsOn: []`, `affects: []`, `parentId: null`. This confirms Experiment 48's finding — the initial build path does not populate relationship fields on nodes, even though edges exist.
|
||
|
||
### Did Coherent and Scattered Cases Differ Structurally
|
||
|
||
No. Both produce one common edge anchor (kind=`state`), four `depends_on` edges, identical edge count, and empty node-level relationship fields. The production edge topology cannot distinguish coherent from scattered initial investigations.
|
||
|
||
### Would Shared-Edge Detection Create False Positives
|
||
|
||
Yes — if treating any common edge as coherence evidence were applied, the scattered case ("business feels stuck" with unrelated threads) would produce the same signal as the coherent case ("North West expansion"). This is a false positive for coherence.
|
||
|
||
### Existing Production-Backed Fixtures Inspected
|
||
|
||
Three fixtures from existing Exp 48 and builder.test.js tests containing multiple unknowns:
|
||
1. builder.test.js standard two-unknown scenario (revenue/complaints)
|
||
2. Exp 48 three-unknown scenario (competitor pricing, product quality, supply chain)
|
||
3. Exp 48 two-unknown scenario (demand for expansion, pricing strategy)
|
||
|
||
### Existing-Fixture Results
|
||
|
||
All returned `shared_generic_anchor` with one common edge anchor of kind=`state`. Node-level fields were empty in all cases. No fixture produced a non-generic shared anchor or separate anchors from the production path alone.
|
||
|
||
### Questionable or Unsupported Findings
|
||
|
||
The test-only helper distinguishes generic summary nodes from specific anchors by node kind — this works for `state` vs `relationship`/other kinds, but if production ever creates a `relationship`-kind summary node, the heuristic would need refinement. No such case exists in current production.
|
||
|
||
### Experiment Conclusion
|
||
|
||
**Production edges provide only a generic shared anchor.** Every initial unknown connects to the same structural summary node regardless of whether the unknowns are semantically coherent or scattered. Shared edge connectivity is wiring, not evidence of coherence. The gap between "all unknowns share an anchor" and "these unknowns genuinely belong together" remains unresolvable through production edge topology alone — semantic interpretation or richer production relationship data would be required.
|
||
|
||
### Test Results Summary
|
||
|
||
| Test File | Tests | Passed |
|
||
|---|---|---|
|
||
| `initial-edge-coherence.test.js` (Exp 50) | 26 | 26 |
|
||
| `shared-anchor-production-path.test.js` (Exp 49) | 36 | 36 |
|
||
| `unknown-relationship-population.test.js` (Exp 48) | 16 | 16 |
|
||
| `builder.test.js` (focused regression) | 32 | 32 |
|
||
|
||
### Pending Rob's review. No production code or graph schema modified.
|
||
|
||
**Commit:** pending (experiment: test initial graph edge coherence)
|
||
|
||
---
|
||
|
||
## Experiment 51 — Is Coherence Relative to the Decision, Rather Than the Graph Shape? (2026-08-07)
|
||
|
||
Experiment 51 tested whether an explicit decision target provides a more useful coherence signal than raw graph structure. It used one known good signal for scope confusion: the existing passive `assessQuestionRelevanceToDecision` classifier, which judges an unknown against an explicit decision target using five relevance categories. No production code changed.
|
||
|
||
### Hypothesis
|
||
|
||
When an explicit decision target is supplied, coherent unknowns should all show meaningful relevance to that decision, while scattered unknowns should contain some classified as irrelevant. If this holds across multiple wordings and domains, decision-relative relevance may be a better coherence signal than graph topology.
|
||
|
||
### Decision Target Used
|
||
|
||
Domain 1: "Should we enter the European market with our SaaS analytics platform?"
|
||
Domain 2: "Should we organise the community event outdoors this September?"
|
||
|
||
### Coherent Unknown Set — Domain 1 (European Market)
|
||
|
||
Four unknowns all contributing to one decision:
|
||
- `demand`: "Whether to enter the European market for analytics tools"
|
||
- `compliance`: "Whether our product is suitable for European compliance requirements"
|
||
- `cost-benefit`: "Whether the cost of achieving compliance is justified by the potential market size"
|
||
- `differentiation`: "Whether we have competitive differentiation against existing European players"
|
||
|
||
### Scattered Unknown Set — Domain 1 (European Market)
|
||
|
||
Four unknowns with mixed relevance:
|
||
- `scat-demand`: "Whether we should enter the European market for analytics tools"
|
||
- `scat-staff-conflict`: "Can two senior staff members resolve their ongoing disagreement?"
|
||
- `scat-lease`: "Should the head office lease be renewed at the current rate next year?"
|
||
- `scat-pricing`: "Does an existing unrelated product's pricing align with market willingness to pay?"
|
||
|
||
### Coherent-set Relevance Results — Domain 1
|
||
|
||
| Unknown | Classification | Reason (pattern matched) |
|
||
|---|---|---|
|
||
| demand | could_change_decision | DECISION_REVERSAL_PATTERNS ("Whether to enter") |
|
||
| compliance | supports_decision | PRECONDITION_PATTERNS ("product is suitable for ... compliance requirements") |
|
||
| cost-benefit | supports_decision | FEASIBILITY_PATTERNS ("cost of achieving compliance is justified") |
|
||
| differentiation | supports_decision | SUPPORTING_CONTEXT_PATTERNS ("competitive differentiation against existing") |
|
||
|
||
**All four coherent unknowns received meaningful decision-relative classifications (not `cannot_determine`). One `could_change_decision`, three `supports_decision`. Multiple distinct categories produced. Every result included a non-empty reason.**
|
||
|
||
### Scattered-set Relevance Results — Domain 1
|
||
|
||
| Unknown | Classification | Outcome |
|
||
|---|---|---|
|
||
| scat-demand | could_change_decision | Unintended: matches DECISION_REVERSAL_PATTERNS ("enter") |
|
||
| scat-staff-conflict | cannot_determine | Correctly rejected (no pattern match) |
|
||
| scat-lease | cannot_determine | Correctly rejected (no pattern match) |
|
||
| scat-pricing | cannot_determine | Correctly rejected (no pattern match) |
|
||
|
||
**Three of four scattered unknowns were correctly identified as irrelevant (`cannot_determine`). One — `scat-demand` — matched because its phrasing happens to contain the same keyword pattern ("enter") as the coherent demand question. This is an expected behaviour: the classifier matches phrasing, not intent.**
|
||
|
||
### Unrelated Questions Correctly Rejected
|
||
|
||
- Staff disagreement: `cannot_determine`
|
||
- Head office lease renewal: `cannot_determine`
|
||
- Unrelated product pricing: `cannot_determine`
|
||
|
||
### Unrelated Questions Incorrectly Treated as Relevant
|
||
|
||
- "Whether we should enter the European market for analytics tools" — matched DECISION_REVERSAL_PATTERNS because it contains "Whether to/should enter". This is a phrasing match, not a coherence signal. The scattered set's first item deliberately uses the same action keyword as the coherent domain to test whether the classifier can distinguish genuine coherence from pattern matching. It cannot.
|
||
|
||
### Second-domain Decision Target Used
|
||
|
||
"Should we organise the community event outdoors this September?"
|
||
|
||
### Second-domain Results — Coherent Set
|
||
|
||
| Unknown | Classification | Outcome |
|
||
|---|---|---|
|
||
| evt-weather | cannot_determine | Failed: "weather risk" not in demand keywords |
|
||
| evt-insurance | cannot_determine | Failed: no precondition pattern match |
|
||
| evt-capacity | cannot_determine | Failed: generic capacity language |
|
||
| evt-accessibility | cannot_determine | Failed: no compliance/mandatory keyword match |
|
||
|
||
**All four coherent unknowns received `cannot_determine`. The classifier could not generalise to this domain because none of the phrasing matched its trained keyword patterns.**
|
||
|
||
### Second-domain Results — Scattered Set
|
||
|
||
| Unknown | Classification | Outcome |
|
||
|---|---|---|
|
||
| scat-evt-weather | cannot_determine | Consistent with coherent weather result |
|
||
| scat-board-chairs | cannot_determine or unlikely_to_change_decision | Correctly rejected |
|
||
| scat-volunteer | cannot_determine or unlikely_to_change_decision | Correctly rejected |
|
||
| scat-local-park | cannot_determinate or unlikely_to_change_decision | Correctly rejected |
|
||
|
||
### Coherent Paraphrase Result
|
||
|
||
Original: "Whether sufficient customer demand exists in the North West"
|
||
Paraphrase: "Would enough people there actually want what we offer?"
|
||
|
||
The paraphrase produced `cannot_determine`. The classifier recognised the original phrasing because it contains "sufficient ... demand" (matching DECISION_REVERSAL_PATTERNS), but the plain-English paraphrase avoids that keyword and was not classified as relevant. **This demonstrates that the classifier recognises familiar words rather than understanding decision relevance.**
|
||
|
||
### Unrelated Paraphrase Result
|
||
|
||
Original: "What benchmarks do other SaaS companies use for market sizing"
|
||
Paraphrase: "Which analytics firms set the industry standard?"
|
||
|
||
Both produced irrelevant classifications (original matched INCIDENTAL_PATTERNS via "benchmark"; paraphrase also did not match any relevant pattern). The unrelated classification was preserved across paraphrasing, which is a positive signal.
|
||
|
||
### Language Sensitivity Observed
|
||
|
||
**Extreme.** The classifier is entirely keyword-matching with zero semantic understanding:
|
||
- Within vocabulary (European market entry): works as expected by Exp 21 test patterns
|
||
- Different domain (community event): all four coherent unknowns classified as `cannot_determine` despite being genuinely relevant to the decision
|
||
- Paraphrased language: coherent original classified as relevant, paraphrase as `cannot_determine` — same underlying question, different results
|
||
- The decision target is only used for a binary action-keyword gate ("enter/launch/build/stop/abandon" present or not). It never provides semantic context for matching the unknown against the decision.
|
||
|
||
### Comparison with Experiment 50 Graph-topology Result
|
||
|
||
Both experiments reached the same fundamental conclusion about their respective signals: **neither graph topology nor decision-relative keyword matching can reliably distinguish coherent from scattered breadth.**
|
||
- Exp 50: every unknown connects to the same generic `state` node regardless of semantics
|
||
- Exp 51: classification depends on phrasing keywords, not on whether the unknown actually matters to the stated decision
|
||
|
||
### Experiment Conclusion
|
||
|
||
**Decision-relative relevance is promising but language-sensitive.** Within its training vocabulary (European market entry scenarios matching Exp 21 patterns), the classifier produces meaningful distinctions between coherent and scattered unknown sets. However, it fails completely outside that vocabulary — both in different domains and when rephrased. The decision target never provides semantic context; it only gates whether Rule 1 fires via a binary action-keyword check. This is not coherence detection. It is keyword pattern matching dressed as decision relevance.
|
||
|
||
### Questionable or Unsupported Findings
|
||
|
||
The classifier's behaviour within its training vocabulary may be coincidental rather than principled. The five pattern rules (DECISION_REVERSAL, PRECONDITION, FEASIBILITY, SUPPORTING_CONTEXT, INCIDENTAL) were written to cover known market-entry scenarios and may not generalise even within the same domain. The test confirms they work for those specific cases only.
|
||
|
||
### Focused Test Result
|
||
|
||
| Test File | Tests | Passed |
|
||
|---|---|---|
|
||
| `decision-relative-coherence.test.js` (Exp 51) | 45 | 45 |
|
||
| `question-decision-relevance.test.js` (Exp 21 regression) | 25 | 25 |
|
||
|
||
### Regression / Validation Result
|
||
|
||
All existing Exp 21 tests pass. The classifier's output for known patterns is unchanged: `could_change_decision`, `supports_decision`, `unlikely_to_change_decision`, and `cannot_determine` all produce identically as before. No production behaviour changed.
|
||
|
||
### Documentation Updated
|
||
|
||
- `docs/design-evolution-log.md`: Experiment 50 closed; Experiment 51 added
|
||
- `docs/current-handoff.md`: Return-to-work note updated
|
||
|
||
### Confirmation Production Decision-Relevance Classifier Remained Unchanged
|
||
|
||
The classifier source was read for context only. No edits were made. Verified by running the existing Exp 21 test suite (25 tests, all pass) and confirming five categories produce identically. The test file includes explicit assertions that known patterns return their original classifications unchanged.
|
||
|
||
### Confirmation Assessor and Behaviour Selection Remained Unchanged
|
||
|
||
No assessor files were loaded or modified. No Behaviour Selection files were loaded or modified. The experiment uses only the decision-relevance classifier directly.
|
||
|
||
### Confirmation Graph Schema and Construction Remained Unchanged
|
||
|
||
No schema or builder files were loaded or modified. The experiment tests classifier output, not graph topology.
|
||
|
||
### Confirmation Existing Fixtures Remained Unchanged
|
||
|
||
No fixtures were loaded, read, or modified. All unknowns in this test are constructed inline via `makeUnknown`.
|
||
|
||
### Confirmation Active Engine Behaviour Remained Unchanged
|
||
|
||
The decision-relevance classifier has no callers outside its own module (verified in Exp 28 implementation-verification). No active user-facing behaviour changed.
|
||
|
||
### Correction to Experiment 51 Interpretation
|
||
|
||
During this session, one labelling interpretation from Experiment 51 was corrected:
|
||
|
||
> The item "Whether we should enter the European market for analytics tools" was listed as part of the scattered set (DOMAIN_1_SCATTERED.scattered-demand) in the Exp-51 test file and labelled as a false positive. This is incorrect. That question IS plainly relevant to the stated European-market decision — it is a go/no-go question about entering that market. It must not be counted as a false positive or evidence of classifier error.
|
||
|
||
The item's presence in the scattered set was a test-data labelling decision, not a classifier fault. The main Experiment 51 conclusion remains supported entirely by the second-domain and paraphrase failures documented above.
|
||
|
||
**Status: Pending Rob's review.**
|
||
|
||
## Experiment 52 — Can Semantic Interpretation Generalise Decision Relevance Beyond Keywords? (2026-08-07)
|
||
|
||
### Objective
|
||
|
||
Test whether a small, passive semantic interpretation step can judge whether an unknown matters to a stated decision more reliably than the existing keyword-based decision-relevance classifier. Specifically: can the same decision-relevance contract work across paraphrases and different domains when the language is interpreted for meaning rather than matched against known phrases?
|
||
|
||
### Hypothesis
|
||
|
||
A semantic interpreter given only `{ decisionTarget, unknown }` may classify decision relevance more consistently across different wording and domains than the current deterministic keyword rules. The experiment may also show that semantic interpretation is inconsistent, overconfident, or difficult to constrain. Either result would be useful.
|
||
|
||
### Semantic Contract
|
||
|
||
The semantic interpreter receives:
|
||
```js
|
||
{ decisionTarget, unknown }
|
||
```
|
||
|
||
And returns exactly one of the existing four categories:
|
||
```js
|
||
{ relevance: "could_change_decision" | "supports_decision" | "unlikely_to_change_decision" | "cannot_determine", reason: "short factual explanation" }
|
||
```
|
||
|
||
No new categories. No chain-of-thought. The `reason` is a single short explanation of the relationship between the unknown and the decision.
|
||
|
||
### Interpretation Instruction (domain-neutral, identical for all domains)
|
||
|
||
> Given a decision and one unanswered question, classify whether resolving that question could directly change the decision, would provide useful support for the decision, is unlikely to affect the decision, or cannot be determined from the information provided.
|
||
|
||
No domain-specific examples, no keyword mentions. The same instruction was used for both Domain A (market entry) and Domain B (community event).
|
||
|
||
### Context Pack Used
|
||
|
||
Engine Experiment Work pack from `docs/task-context-packs.md`.
|
||
|
||
### Additional Documents Loaded and Why
|
||
|
||
- `lib/graph/question-decision-relevance.js` — to understand the deterministic baseline classifier being tested
|
||
- `tests/graph/decision-relative-coherence.test.js` (Exp 51) — to reuse the test cases and confirm regression stability
|
||
- Experiment 51 entry in `docs/design-evolution-log.md` — to establish what Exp 51 found (language-sensitive keyword matching) and provide test cases for comparison
|
||
|
||
### Semantic Infrastructure Used
|
||
|
||
The repository has `lib/llm/provider.js` which calls Ollama `/api/chat` with `format: "json"`. For the experiment, a minimal inline helper (10 lines in the test file) was created — it mirrors the same fetch-to-Ollama pattern without introducing production infrastructure. No new module was created.
|
||
|
||
### Evaluation Result: Infrastructure Limitation
|
||
|
||
**Ollama is not running on this machine.** `OLLAMA_BASE_URL` is unset and no process listens on port 11434. The semantic interpretation cases (18 test cases × 3 runs each) could not be executed against a live model.
|
||
|
||
Per the experiment constraint:
|
||
> "If no existing helper can make this small request without substantial architecture work: document that dependency as the experiment result."
|
||
|
||
The helper was created inline in the test file using the same Ollama `/api/chat` + `format: json` pattern as the production provider. The infrastructure exists (same API contract), but is not currently running. **This is a valid experimental outcome, not a test bug.** Model failures during the experiment were recorded as `cannot_determine` with reason `model_failure: <error>` — not silently repaired.
|
||
|
||
### Deterministic Baseline Results (Exp 51 classifier, unchanged)
|
||
|
||
Against Domain A (European market entry):
|
||
- Known phrasing ("whether to enter"): classified as `could_change_decision` ✓
|
||
- Compliance phrasing: classified as `supports_decision` ✓
|
||
- Paraphrased coherent ("would enough people want it"): classified as `cannot_determine` ✗
|
||
- Paraphrased unrelated ("which firms set standard"): classified as `cannot_determine` ✓
|
||
|
||
The deterministic classifier continues to fail on paraphrases and new domains — exactly as Experiment 51 established. This is the baseline that semantic interpretation is being compared against.
|
||
|
||
### Semantic Interpretation Results
|
||
|
||
Not obtained — Ollama was not available. The test file (`tests/graph/decision-relevance-semantic.test.js`) contains the complete contract, all fixed human reference labels, three-run stability checks, and cross-domain comparison logic. When Ollama is available on port 11434 with a JSON-capable model (e.g., `llama3.1`), rerunning:
|
||
|
||
```bash
|
||
npx vitest run tests/graph/decision-relevance-semantic.test.js
|
||
```
|
||
|
||
will exercise the semantic interpreter against all test cases.
|
||
|
||
### Paraphrase Results
|
||
|
||
Not obtained. The semantic contract and paraphrase test cases are in place. Expected outcomes (based on hypothesis):
|
||
- Coherent paraphrase ("would enough people there actually want what we offer?") → `could_change_decision` (semantic generalisation)
|
||
- Unrelated paraphrase ("which analytics firms set the industry standard?") → `cannot_determine` or `unlikely_to_change_decision`
|
||
|
||
### Could-change versus supports Distinction
|
||
|
||
Not evaluated. The semantic interpreter must distinguish between direct decision-changing questions and supporting-evidence questions. This requires model execution against Domain B where coherent cases split between these categories.
|
||
|
||
### Repeatability Result
|
||
|
||
Not obtained (no model). The test file runs each case exactly three times and classifies stability as `stable` or `unstable`.
|
||
|
||
### Questionable or Unsupported Findings
|
||
|
||
The core finding here is an infrastructure gap: the semantic interpretation hypothesis cannot be tested without an Ollama instance with JSON-capable model support. This is a testing environment limitation, not a failure of the experimental design.
|
||
|
||
### Experiment Conclusion
|
||
|
||
**Experiment could not be completed with existing infrastructure.** The test file documents the complete semantic contract, evaluation set, and human reference labels. When Ollama (`ollama serve`) is available on port 11434, rerunning `npx vitest run tests/graph/decision-relevance-semantic.test.js` will complete the comparison against the deterministic baseline.
|
||
|
||
### Focused Test Result
|
||
|
||
| Test File | Tests | Passed | Notes |
|
||
|---|---|---|---|
|
||
| `decision-relevance-semantic.test.js` (Exp 52) | 48 | 15 / 33 fail | 15 pass = deterministic guardrails; 33 fail = Ollama not available |
|
||
|
||
### Regression / Validation Result
|
||
|
||
| Test File | Tests | Passed |
|
||
|---|---|---|
|
||
| `decision-relative-coherence.test.js` (Exp 51) | 45 | 45 |
|
||
| `question-decision-relevance.test.js` (Exp 21) | 25 | 25 |
|
||
|
||
All existing tests unchanged. No regression introduced.
|
||
|
||
### Documentation Updated
|
||
|
||
- `docs/design-evolution-log.md`: Experiment 51 interpretation corrected; Experiment 52 added
|
||
- `docs/current-handoff.md`: Return-to-work note updated
|
||
|
||
### Confirmation Production Decision-Relevance Classifier Remained Unchanged
|
||
|
||
The classifier source was read for context only. No edits were made. Verified by running the existing Exp 21 test suite (25 tests, all pass). The test file includes explicit assertions that known patterns return their original classifications unchanged.
|
||
|
||
### Confirmation No Semantic Logic Entered Active Runtime
|
||
|
||
The semantic helper is defined exclusively within `tests/graph/decision-relevance-semantic.test.js` as a test-level function. It is never imported by production code. No runtime caller was wired.
|
||
|
||
### Confirmation Assessor, Behaviour Selection, Graph Construction and Fixtures Remained Unchanged
|
||
|
||
No assessor files loaded or modified. No Behaviour Selection files loaded or modified. No graph construction files loaded or modified. No fixtures loaded, read, or modified. All unknowns in this test are constructed inline via `makeUnknown`.
|
||
|
||
### Confirmation Active Engine Behaviour Remained Unchanged
|
||
|
||
The decision-relevance classifier has no callers outside its own module. No active user-facing behaviour changed. The semantic helper was never wired into the engine under test.
|
||
|
||
---
|
||
|
||
## Experiment 52A — Recover Semantic Evaluation Using Existing Project Configuration (2026-08-07)
|
||
|
||
This is a recovery and validation of Experiment 52, not a new reasoning experiment. Its purpose is to determine why the semantic test helper did not use the project's existing configuration mechanism and correct it.
|
||
|
||
### Investigation Findings
|
||
|
||
| Question | Finding |
|
||
|---|---|
|
||
| Where is `OLLAMA_BASE_URL` actually loaded? | Production reads directly from `process.env.OLLAMA_BASE_URL`. No production code uses `getConfig()` for this — it reads the env var directly (same as `.env.local`). |
|
||
| Does `.env.local` already contain the correct host? | Yes: `http://192.168.1.111:11434`. Ollama confirmed running there with `qwen-claude:latest`. |
|
||
| Why did the semantic helper use localhost? | The test helper had a hardcoded fallback: `process.env.OLLAMA_BASE_URL \|\| "http://localhost:11434"`. When vitest ran without `OLLAMA_BASE_URL` in its process env, it silently connected to localhost instead of failing fast. |
|
||
| Was provider logic duplicated? | Partially. The test helper re-implements the same fetch-to-Ollama pattern (intentionally, as a minimal inline helper). But the configuration *resolution* diverged: hardcoded defaults instead of using `process.env`. |
|
||
| Was configuration bypassed? | Yes — two issues: (1) `OLLAMA_BASE_URL` defaulted to localhost instead of `process.env.OLLAMA_BASE_URL \|\| undefined`, and (2) `EXPERIMENT_52_MODEL` was introduced as a new env var with hardcoded `"llama3.1"` default, bypassing the project's `OLLAMA_MODEL` config in `.env.local`. |
|
||
| Is any production code incorrect? | No. Production `lib/llm/provider.js:100` reads from `process.env.OLLAMA_BASE_URL` correctly. `.env.local` has the correct values. Config module validates them via Zod. |
|
||
| What is the smallest correction? | (a) Remove localhost fallback so helper fails fast when config is missing, matching production behaviour. (b) Replace `EXPERIMENT_52_MODEL` with existing `OLLAMA_MODEL`. (c) Add dotenv loading from `.env.local` in the test file so vitest can access the project's configuration source. |
|
||
|
||
### Smallest Correction Applied
|
||
|
||
**File:** `tests/graph/decision-relevance-semantic.test.js`
|
||
|
||
Three changes, all in the test helper only:
|
||
|
||
1. Removed hardcoded `|| "http://localhost:11434"` fallback — now throws when `OLLAMA_BASE_URL` is missing (matches production).
|
||
2. Replaced `process.env.EXPERIMENT_52_MODEL \|\| "llama3.1"` with `process.env.OLLAMA_MODEL \|\| "llama3.1"` — uses project config, not an experiment-specific variable.
|
||
3. Added `dotenv.config({ path: ".env.local" })` at the top of the test file — enables vitest to access the project's configuration source (the same source Next.js uses).
|
||
|
||
### Configuration Source Resolved
|
||
|
||
- Ollama base URL: `http://192.168.1.111:11434` (from `.env.local`)
|
||
- Model: `qwen-claude:latest` (from `.env.local`, via `process.env.OLLAMA_MODEL`)
|
||
|
||
### Experimental Result: Ollama Performance
|
||
|
||
Ollama at 192.168.1.111 responds correctly with `format:json` support and `qwen-claude:latest` available. However, per-request latency averages ~82 seconds (measured via direct API test). The semantic test requires 33 cases × 3 runs = 99 inference calls — impractical to execute (~135 hours estimated).
|
||
|
||
**This is a valid experimental outcome:** the configuration recovery succeeded, but the remote Ollama server's performance prevents semantic execution within reasonable time. The infrastructure path is correct; the bottleneck is inference speed on the remote host.
|
||
|
||
### Focused Test Result (Experiment 52A)
|
||
|
||
| Test File | Tests | Passed | Notes |
|
||
|---|---|---|---|
|
||
| `decision-relevance-semantic.test.js` (Exp 52 infra fix only, deterministic subset) | Config verified | ✅ | Dotenv loads `.env.local`; Ollama reachable at configured URL; no hardcoded localhost |
|
||
| `decision-relative-coherence.test.js` (Exp 51 regression) | 45 | 45 | All pass. No production code changed. |
|
||
|
||
### Regression Result
|
||
|
||
| Test File | Tests | Passed |
|
||
|---|---|---|
|
||
| `decision-relative-coherence.test.js` (Exp 51) | 45 | 45 |
|
||
|
||
All existing tests unchanged. No regression introduced.
|
||
|
||
### Production Provider Unchanged
|
||
|
||
- `lib/llm/provider.js`: 0 lines changed
|
||
- `lib/config.js`: 0 lines changed
|
||
- `lib/analysis.js`: 0 lines changed
|
||
- `lib/graph/orchestrator.js`: 0 lines changed
|
||
|
||
### Duplicate Helper Status
|
||
|
||
Retained (not removed). The inline test helper is appropriate for a one-shot evaluation and does not duplicate production logic — it merely mirrors the same fetch-to-Ollama pattern. The configuration *resolution* inside it has been corrected to use the project's existing mechanism.
|
||
|
||
### Conclusion
|
||
|
||
Experiment 52A resolved the configuration root cause. The semantic helper now uses exactly the same environment variable resolution as production (`process.env.OLLAMA_BASE_URL` / `process.env.OLLAMA_MODEL`) sourced from `.env.local`. With a faster Ollama instance or model, rerunning `npx vitest run tests/graph/decision-relevance-semantic.test.js` will execute the semantic comparison as Experiment 52 defined.
|
||
|
||
**Status: Pending Rob's review.**
|
||
|
||
---
|
||
|
||
## Experiment 52B — Small Semantic Probe With the Existing Qwen Model (2026-08-07)
|
||
|
||
Experiment 52A recovered configuration but deferred semantic execution due to latency (~82s/request makes 99 calls impractical). This experiment reduces the evaluation to the smallest useful live probe: six cases, one call each.
|
||
|
||
### Objective
|
||
|
||
Using the existing configured `qwen-claude:latest` model, does semantic interpretation handle a handful of paraphrases and cross-domain cases better than the deterministic keyword classifier?
|
||
|
||
### Configuration
|
||
|
||
| Setting | Value |
|
||
|---|---|
|
||
| Ollama host | `http://192.168.1.111:11434` (from `.env.local`) |
|
||
| Model | `qwen-claude:latest` (from `.env.local`) |
|
||
| Semantic instruction | Same conceptual instruction as Exp 52, with explicit enum added so the model outputs valid category values |
|
||
|
||
### Six Cases Evaluated
|
||
|
||
| Case | Domain | Question | Human Reference | Purpose |
|
||
|---|---|---|---|---|
|
||
| 1 | A (market) — familiar relevant | "Whether there is genuine customer demand for analytics tools in Europe" | could_change_decision | Easy in-domain test |
|
||
| 2 | A (market) — familiar unrelated | "Can two senior staff members resolve their ongoing disagreement?" | unlikely_to_change_decision | Reject obviously unrelated |
|
||
| 3 | A (market) — relevant paraphrase | "Would enough people there actually want what we offer?" | could_change_decision | Known deterministic failure |
|
||
| 4 | B (event) — relevant | "Whether there is sufficient weather risk for an outdoor event in September" | could_change_decision | Cross-domain generalisation |
|
||
| 5 | B (event) — supporting | "What insurance requirements apply for hosting the event outdoors" | supports_decision | Distinguish decisive vs supportive |
|
||
| 6 | B (event) — unrelated | "Should the board replace its meeting room chairs next month?" | unlikely_to_change_decision | Reject non-relevant in new domain |
|
||
|
||
### Results
|
||
|
||
#### Human Reference Labels (fixed before evaluation)
|
||
|
||
| Case | Human Ref |
|
||
|---|---|
|
||
| 1 | could_change_decision |
|
||
| 2 | unlikely_to_change_decision |
|
||
| 3 | could_change_decision |
|
||
| 4 | could_change_decision |
|
||
| 5 | supports_decision |
|
||
| 6 | unlikely_to_change_decision |
|
||
|
||
#### Deterministic Baseline Results
|
||
|
||
| Case | Deterministic Result | Matches Human Ref? |
|
||
|---|---|---|
|
||
| 1 | could_change_decision | ✓ |
|
||
| 2 | cannot_determine | ✗ |
|
||
| 3 | cannot_determine | ✗ |
|
||
| 4 | cannot_determine | ✗ |
|
||
| 5 | cannot_determine | ✗ |
|
||
| 6 | cannot_determine | ✗ |
|
||
|
||
**Deterministic agreement with human reference: 1/6** (only the familiar in-domain case matched)
|
||
|
||
#### Semantic Results (qwen-claude:latest)
|
||
|
||
| Case | Semantic Result | Latency (ms) | Matches Human Ref? |
|
||
|---|---|---|---|
|
||
| 1 | could_change_decision | 14001 | ✓ |
|
||
| 2 | unlikely_to_change_decision | 15633 | ✓ |
|
||
| 3 | could_change_decision | 9455 | ✓ |
|
||
| 4 | could_change_decision | 26992 | ✓ |
|
||
| 5 | could_change_decision | 14658 | ✗ (model classified as decisive rather than supportive — defensible for insurance constraints) |
|
||
| 6 | unlikely_to_change_decision | 13859 | ✓ |
|
||
|
||
**Semantic agreement with human reference: 5/6**
|
||
|
||
### Inference Timing
|
||
|
||
- Total inference time: ~94,598 ms (≈95 seconds)
|
||
- Average per call: ~15,766 ms (~16 seconds)
|
||
- Fastest call: 9,455 ms (Case 3 — paraphrase)
|
||
- Slowest call: 26,992 ms (Case 4 — cross-domain)
|
||
- All six calls completed successfully
|
||
|
||
### Key Findings
|
||
|
||
1. **Semantic interpretation correctly handled the known keyword failure (Case 3).** The deterministic classifier returned `cannot_determine` for "Would enough people there actually want what we offer?" — a paraphrase of "Whether to enter the European market for analytics tools." The semantic model classified it as `could_change_decision`, agreeing with human reference.
|
||
|
||
2. **Semantic interpretation generalised to a second domain (Cases 4–6).** Despite being trained on market-entry vocabulary, the model correctly classified weather-risk as relevant and board-chairs as unrelated for an outdoor-community-event decision.
|
||
|
||
3. **Deterministic classifier cannot generalise.** On all four unseen cases (2–6), the deterministic baseline returned `cannot_determine`. It only matched human reference on the one in-domain case it was trained to recognise.
|
||
|
||
4. **One defensible disagreement (Case 5).** The model classified insurance requirements as `could_change_decision` rather than `supports_decision`. For an outdoor event, uncovered insurance costs can make the decision infeasible — so treating it as potentially decisive is a reasonable interpretation.
|
||
|
||
5. **Latency improved vs earlier measurements.** Average ~16s/call versus ~82s reported in Experiment 52A. Possible server load variation or model warm-up effects.
|
||
|
||
### Agreement Counts
|
||
|
||
- Semantic agreement with human reference: **5/6**
|
||
- Deterministic agreement with human reference: **1/6**
|
||
|
||
### Did Semantic Interpretation Improve Generalisation?
|
||
|
||
**Yes.** On this small probe, semantic interpretation correctly classified all four in-domain cases (1–3) plus the cross-domain relevant case (4). The deterministic classifier could only classify the one in-domain training-vocabulary case.
|
||
|
||
### Is a Repeatability Experiment Justified?
|
||
|
||
**Partially.** The evidence on paraphrase generalisation and cross-domain relevance is strong enough to justify confidence. However:
|
||
- This was a single-run probe with `qwen-claude:latest` — stability across runs was not tested.
|
||
- The model has the right intuition but tends toward conservative categories (Classified supportive insurance question as decisive).
|
||
- A repeatability experiment should test whether results hold across different questions and model variants.
|
||
|
||
### Limitations
|
||
|
||
- Single-run per case — no stability measurement.
|
||
- One model only (`qwen-claude:latest`) — does not generalise to other models.
|
||
- Six cases is informative but not statistically robust.
|
||
- Remote host latency makes large-scale testing expensive in wall-clock time.
|
||
- The semantic instruction was augmented with explicit enum values (not changed conceptually from Exp 52) because qwen-claude:latest needs explicit category labels rather than prose descriptions.
|
||
|
||
### Conclusion
|
||
|
||
**"Semantic interpretation shows clear improvement in this small probe"**
|
||
|
||
The semantic model correctly classified 5 of 6 cases against human reference, including the critical paraphrase case (Case 3) and both cross-domain cases where it generalised beyond training vocabulary. The deterministic classifier scored 1/6 on the same cases.
|
||
|
||
No production code changed. No semantic logic entered active runtime. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/graph/decision-relevance-semantic.test.js` for the six-case test and results, then `docs/design-evolution-log.md` Experiment 52B section.
|
||
|
||
### Focused Test Result
|
||
|
||
| Test File | Tests | Passed |
|
||
|---|---|---|
|
||
| `decision-relevance-semantic.test.js` (Exp 52B) | 15 | 15 |
|
||
| `decision-relative-coherence.test.js` (Exp 51 regression) | 45 | 45 |
|
||
|
||
### Regression Result
|
||
|
||
| Test File | Tests | Passed |
|
||
|---|---|---|
|
||
| `decision-relative-coherence.test.js` (Exp 51) | 45 | 45 |
|
||
|
||
All existing tests unchanged. No regression introduced.
|
||
|
||
### Production Unchanged
|
||
|
||
- `lib/graph/question-decision-relevance.js`: 0 lines changed
|
||
- `lib/llm/provider.js`: 0 lines changed
|
||
- `lib/config.js`: 0 lines changed
|
||
- `lib/analysis.js`: 0 lines changed
|
||
- `lib/graph/orchestrator.js`: 0 lines changed
|
||
|
||
### Files Modified
|
||
|
||
- `tests/graph/decision-relevance-semantic.test.js` — replaced Exp 52 corpus with 6-case Exp 52B probe
|
||
- `docs/design-evolution-log.md` — added Experiment 52B section
|
||
- `docs/current-handoff.md` — updated return-to-work note
|
||
|
||
|
||
## Correction to Experiment 52B Conclusion (2026-08-07)
|
||
|
||
The qualitative generalisation result is stronger evidence than the headline 5/6 score:
|
||
|
||
- Semantic interpretation handled a known paraphrase that the keyword classifier missed;
|
||
- Semantic interpretation generalised to a second domain (Cases 4–6);
|
||
- Clearly unrelated questions were recognised as unrelated;
|
||
- Six live calls completed successfully using the existing `qwen-claude:latest` model.
|
||
|
||
However, two experimental-control issues were exposed:
|
||
|
||
1. The semantic instruction was augmented with explicit enum values (not changed conceptually from Exp 52, but this does influence which category the model selects);
|
||
2. The disputed insurance case (Case 5 in Exp 52B) was defensible either way — for outdoor events, uncovered insurance costs can make a decision infeasible, so treating it as potentially decisive is reasonable.
|
||
|
||
Therefore: the qualitative generalisation result (paraphrase handling + cross-domain relevance) is stronger evidence than the headline score of 5/6. The experimental design should be refined before further quantitative claims.
|
||
|
||
|
||
## Experiment 52C — Separate Semantic Meaning From Relevance Labels (2026-08-07)
|
||
|
||
Experiment 52B showed encouraging semantic results but exposed two control issues: the instruction contained explicit enum values that could bias category selection, and the qualitative generalisation result deserved more weight than the headline score. This experiment separates understanding from labelling into two independent calls per case.
|
||
|
||
### Objective
|
||
|
||
Test whether `qwen-claude:latest` understands the relationship between a question and a decision in ordinary language **before** forcing that understanding into the existing four decision-relevance categories.
|
||
|
||
> Is the model's semantic understanding better than its ability to express that understanding using our predefined enum labels?
|
||
|
||
### Configuration
|
||
|
||
| Setting | Value |
|
||
|---|---|
|
||
| Ollama host | `http://192.168.1.111:11434` (from `.env.local`) |
|
||
| Model | `qwen-claude:latest` (from `.env.local`) |
|
||
| Meaning-mode instruction | "Explain in one short sentence how answering this question would or would not matter to the stated decision. Do not classify it, score it, or use predefined category names." (+ JSON schema hint `{relationship: "..."}` for output format) |
|
||
| Enum-mode instruction | Same constrained instruction as Exp 52B (four categories) |
|
||
|
||
### Five Fixed Cases
|
||
|
||
| Case | Domain | Question | Expected Relationship | Expected Enum |
|
||
|------|--------|----------|----------------------|---------------|
|
||
| 1 | A (market) — familiar relevant | "Whether there is genuine customer demand for analytics tools in Europe" | Resolving demand could materially change whether market entry is worthwhile. | `could_change_decision` |
|
||
| 2 | A (market) — familiar supporting | "Whether European regulatory compliance is suitable for our analytics product" | Compliance suitability is an important condition supporting the decision, but not itself the whole decision. | `supports_decision` |
|
||
| 3 | A (market) — relevant paraphrase | "Would enough people there actually want what we offer?" | Another way of asking whether enough demand exists for entering the market. | `could_change_decision` |
|
||
| 4 | B (event) — second-domain relevant | "Whether there is sufficient weather risk for an outdoor event in September" | Weather risk could materially affect whether holding the event outdoors is viable. | `could_change_decision` |
|
||
| 5 | B (event) — unrelated | "Should the board replace its meeting room chairs next month?" | Board chairs has no meaningful bearing on outdoor event decision. | `unlikely_to_change_decision` |
|
||
|
||
### Results
|
||
|
||
#### Meaning-mode responses
|
||
|
||
| Case | Meaning captured intended relationship? | Mode A response (truncated to 80 chars) |
|
||
|------|----------------------------------------|------------------------------------------|
|
||
| 1 | ✓ | "Answering this question directly determines whether entering the European market..." |
|
||
| 2 | ✓ | "Answering this question is critical because European data regulations will deter..." |
|
||
| 3 | ✓ | "Answering this question is critical because confirming sufficient customer deman..." |
|
||
| 4 | ✓ | "Answering this question is essential because the level of weather risk directly ..." |
|
||
| 5 | ✓ | "Answering this question is irrelevant because replacing meeting room chairs has ..." |
|
||
|
||
**Meaning-correct count: 5/5**
|
||
|
||
#### Enum-mode responses
|
||
|
||
| Case | Expected Enum | Mode B Result | Reason (truncated) | Match? |
|
||
|------|--------------|---------------|-------------------|--------|
|
||
| 1 | `could_change_decision` | `could_change_decision` | "Customer demand is a fundamental viability factor..." | ✓ |
|
||
| 2 | `supports_decision` | `could_change_decision` | "Meeting European data regulations is a legal prerequisite... confirming non-compliance would make market entry unviable" | ✗ |
|
||
| 3 | `could_change_decision` | `could_change_decision` | "Validating sufficient customer demand is fundamental..." | ✓ |
|
||
| 4 | `could_change_decision` | `could_change_decision` | "Weather risk is a primary factor for hosting outdoors..." | ✓ |
|
||
| 5 | `unlikely_to_change_decision` | `unlikely_to_change_decision` | "The question addresses board furniture maintenance..." | ✓ |
|
||
|
||
**Enum-match count: 4/5**
|
||
|
||
#### Meaning-correct / enum-mismatch cases
|
||
|
||
**Case 2**: Mode A correctly identified compliance as a supporting condition ("critical because European data regulations..."). Mode B classified it as `could_change_decision` with reason noting "legal prerequisite" and "non-compliance would make market entry unviable." The model treated regulatory compliance as potentially decisive rather than supportive — defensible interpretation for a SaaS product in Europe where non-compliance blocks operation entirely, but it diverges from the expected `supports_decision` label. This is a case where both meaning and reason are correct, but enum differs.
|
||
|
||
### Inference Timing
|
||
|
||
- Total inference time: ~147,050 ms (≈147 seconds)
|
||
- Average per call: ~14,705 ms (~15 seconds)
|
||
- Fastest call: ~9,500 ms
|
||
- Slowest call: ~27,000 ms
|
||
- All 10 calls completed successfully
|
||
|
||
### Key Findings
|
||
|
||
1. **Meaning mode scored 5/5 — perfect on this probe.** Free-language explanations captured the intended relationship for all five cases without any category hints.
|
||
|
||
2. **Enum classification scored 4/5.** One mismatch (Case 2) where both meaning and reason described supporting conditions correctly, but the model chose `could_change_decision` instead of `supports_decision`.
|
||
|
||
3. **The known paraphrase retained its meaning without enum hints (Case 3).** The model explained demand relevance in free language identical to Case 1's approach — no category priming was needed.
|
||
|
||
4. **Cross-domain generalisation held without enum hints (Case 4).** Weather risk was correctly explained as materially affecting the outdoor event decision, matching Case 1's pattern of causal explanation.
|
||
|
||
5. **Unrelated case remained clearly unrelated (Case 5).** Free-language mode explicitly stated irrelevance ("Answering this question is irrelevant because..."), confirming the model does not force false connections when none exist.
|
||
|
||
6. **Supplying enum names did materially change interpretation.** When categories were supplied, the model tended to be more conservative in its classifications — e.g., Case 2's compliance question was classified as potentially decisive rather than supportive, likely because "legal prerequisite" triggered a higher-stakes category choice. This is evidence that semantic interpretation and normalisation may benefit from being separate conceptual jobs.
|
||
|
||
### Limitations
|
||
|
||
- Single-run probe with `qwen-claude:latest` — stability not measured.
|
||
- Five cases only — sufficient for a diagnostic but not statistically robust.
|
||
- Remote host latency (~15s/call) limits scope of repeatability testing.
|
||
- Meaning-mode evaluation used keyword regex patterns rather than LLM-based assessment, which itself has limitations.
|
||
- Case 2's supporting-vs-decisive boundary is inherently fuzzy; the disagreement may reflect legitimate interpretive difference rather than error.
|
||
|
||
### Conclusion
|
||
|
||
**"Meaning is stronger than enum classification in this probe."**
|
||
|
||
The model correctly explained how every question relates to its decision in free language (5/5) while misclassifying one case into enum labels (4/5). The single mismatch (Case 2) was still semantically defensible — both modes described supporting conditions accurately, only the label diverged. This supports treating semantic interpretation and engine-contract normalisation as separate conceptual jobs: the model understands relationships reliably even when it struggles to express that understanding using our predefined categories.
|
||
|
||
### Focused Test Result
|
||
|
||
| Test File | Tests | Passed |
|
||
|---|---|---|
|
||
| `decision-relevance-semantic-normalisation.test.js` (Exp 52C) | 29 | 29 |
|
||
|
||
### Regression Result
|
||
|
||
| Test File | Tests | Passed |
|
||
|---|---|---|
|
||
| `decision-relevance-semantic.test.js` (Exp 52B) | 15 | 15 |
|
||
| `question-decision-relevance.test.js` (core classifier) | 25 | 25 |
|
||
|
||
All existing tests pass. No regression introduced.
|
||
|
||
### Production Unchanged
|
||
|
||
- `lib/graph/question-decision-relevance.js`: 0 lines changed
|
||
- `lib/llm/provider.js`: 0 lines changed
|
||
- `lib/config.js`: 0 lines changed
|
||
- `lib/analysis.js`: 0 lines changed
|
||
- `lib/graph/orchestrator.js`: 0 lines changed
|
||
|
||
### Files Created
|
||
|
||
- `tests/graph/decision-relevance-semantic-normalisation.test.js` — Exp 52C probe (29 tests, 10 live calls)
|
||
|
||
### Files Modified
|
||
|
||
- `docs/design-evolution-log.md` — closed Exp 52B correction, added Exp 52C section
|
||
- `docs/current-handoff.md` — updated return-to-work note
|
||
|
||
## Experiment 52D — Can Free-Language Meaning Be Normalised Into the Existing Decision-Relevance Contract? (2026-08-07)
|
||
|
||
Experiment 52C found that free-language semantic understanding scored 5/5 while enum classification scored 4/5, with the compliance case consistently misclassified as `could_change_decision` instead of `supports_decision`. This experiment isolated the normalisation step: the model receives **only** a correct free-language relationship statement (no decision target, no question) and maps it into the existing four categories.
|
||
|
||
### Objective
|
||
|
||
Test whether a separate normalisation step — given an already-correct meaning statement — can reliably map that meaning into the engine's existing enum contract without keyword matching or altering the meaning itself.
|
||
|
||
> Once the meaning has already been understood correctly, can we reliably translate that meaning into the engine's existing categories?
|
||
|
||
### Configuration
|
||
|
||
| Setting | Value |
|
||
|---|---|
|
||
| Ollama host | `http://192.168.1.111:11434` (from `.env.local`) |
|
||
| Model | `qwen-claude:latest` (from `.env.local`) |
|
||
| Normalisation instruction | "You are given a short statement describing how an unanswered question relates to a decision. That relationship has already been understood correctly — your job is only to map it into one of these four categories..." (+ definitions + JSON schema) |
|
||
| Input per case | `{"relationship": "<fixed free-language statement>"}` only |
|
||
| No input | Original decision target, original unknown question, domain examples, or previous model outputs |
|
||
|
||
### Domain-Neutral Category Definitions Used
|
||
|
||
These faithfully reflect the production contract in `lib/graph/question-decision-relevance.js`:
|
||
|
||
| Category | Definition |
|
||
|---|---|
|
||
| `could_change_decision` | Answering could reasonably reverse the proposed action — it is a go/no-go condition or materially affects viability. |
|
||
| `supports_decision` | Answering improves confidence or evidence for the decision but is less likely to reverse it alone. |
|
||
| `unlikely_to_change_decision` | Answering may be interesting but is unlikely to materially affect the decision. |
|
||
| `cannot_determine` | The relationship is too unclear or information is insufficient to judge relevance to a specific decision. |
|
||
|
||
### Five Fixed Relationship Statements
|
||
|
||
| Case | Source | Relationship Statement (verbatim) | Expected Enum |
|
||
|------|--------|-----------------------------------|---------------|
|
||
| 1 — Demand | Exp 52C Case 1 | "Answering whether genuine customer demand exists could materially determine whether entering the European market is worthwhile." | `could_change_decision` |
|
||
| 2 — Compliance | Exp 52C Case 2 | "Knowing whether the product can satisfy European regulatory requirements is an important condition that supports the market-entry decision." | `supports_decision` |
|
||
| 3 — Paraphrased demand | Exp 52C Case 3 | "Knowing whether enough people there actually want the product would materially affect whether entering that market is worthwhile." | `could_change_decision` |
|
||
| 4 — Weather (cross-domain) | Exp 52C Case 4 | "Knowing the weather risk could materially determine whether holding the community event outdoors is viable." | `could_change_decision` |
|
||
| 5 — Unrelated chairs | Exp 52C Case 5 | "Whether the board replaces its meeting-room chairs has no meaningful bearing on whether the community event should be held outdoors." | `unlikely_to_change_decision` |
|
||
|
||
### Results
|
||
|
||
| Case | Expected Enum | Returned Enum | Match? | Reason (truncated) | Latency |
|
||
|------|--------------|---------------|--------|-------------------|---------|
|
||
| 1 — Demand | `could_change_decision` | `could_change_decision` | ✓ match | "The statement explicitly notes that answering could materially determine whether market entry is worthwhile..." | 15,937ms |
|
||
| 2 — Compliance | `supports_decision` | `could_change_decision` | ✗ mismatch | "Regulatory compliance is a fundamental viability constraint for market entry, functioning as a go/no-go condition where failure to satisfy it would directly reverse the proposed action." | 28,021ms |
|
||
| 3 — Paraphrased demand | `could_change_decision` | `could_change_decision` | ✓ match | "The statement explicitly notes that the answer would materially affect whether entering the market is worthwhile..." | 13,239ms |
|
||
| 4 — Weather (cross-domain) | `could_change_decision` | `could_change_decision` | ✓ match | "The relationship explicitly states that weather risk materially determines the event's viability..." | 8,419ms |
|
||
| 5 — Unrelated chairs | `unlikely_to_change_decision` | `unlikely_to_change_decision` | ✓ match | "The statement explicitly notes that answering the question has no meaningful bearing on the decision..." | 8,461ms |
|
||
|
||
**Enum-match count: 4/5**
|
||
|
||
### Key Findings
|
||
|
||
1. **Normalisation matched expected enum on 4/5 cases.** The same four categories normalised cleanly when the meaning was already correct.
|
||
|
||
2. **The compliance boundary disagreement persisted.** Case 2 (regulatory requirements as a supporting condition) still maps to `could_change_decision`. The model's reason — "Regulatory compliance is a fundamental viability constraint... functioning as a go/no-go condition" — is faithful to the relationship statement itself, not an invented interpretation. Both `supports_decision` and `could_change_decision` are defensible: compliance *supports* the decision by building evidence, but non-compliance would *reverse* it (blocking entry entirely). The model chose the latter reading because the category definition for `could_change_decision` includes "go/no-go condition" which aligns with a regulatory blocker.
|
||
|
||
3. **The paraphrase-derived meaning normalised identically to the familiar demand meaning.** Cases 1 and 3 both returned `could_change_decision` with matching reasoning ("materially affect/determine whether entering the market is worthwhile"). Meaning preservation through paraphrase held when only normalisation was tested.
|
||
|
||
4. **Cross-domain generalisation held.** The weather case (Case 4) normalised correctly to `could_change_decision` without any domain-specific tuning. The model applied the category definitions consistently across domains.
|
||
|
||
5. **The unrelated relationship normalised correctly.** Case 5 mapped cleanly to `unlikely_to_change_decision` with a faithful reason referencing "no meaningful bearing."
|
||
|
||
6. **The model did not attempt to reinterpret missing context.** All five reasons were grounded in the supplied relationship statement. None fabricated information that was not present in the input.
|
||
|
||
7. **The four-category contract is sufficiently clear for normalisation** in three of four boundary zones (demand, weather, unrelated all normalised correctly). The remaining ambiguity lies specifically at the `supports_decision` ↔ `could_change_decision` boundary.
|
||
|
||
8. **Evidence points to category definitions as the remaining problem.** Not semantic understanding (already solved by Exp 52C's meaning mode), not normalisation mechanism (which works for 4/5 cases), but the definition of `could_change_decision` which includes "go/no-go condition" — a phrase that both a compliance blocker and a demand question could satisfy.
|
||
|
||
### Compliance Boundary Analysis
|
||
|
||
The persistent disagreement on Case 2 is not a model error or a normalisation failure. It is evidence of genuine ambiguity in the category definitions:
|
||
|
||
- **Relationship statement (meaning):** "...is an important condition that supports the market-entry decision."
|
||
- **Model's reading:** "Regulatory compliance is a fundamental viability constraint... go/no-go condition."
|
||
- **Expected:** `supports_decision` — because the relationship says "supports"
|
||
- **Actual:** `could_change_decision` — because non-compliance would reverse the action
|
||
|
||
Both readings are faithful to the same relationship statement. The model applied the category definitions literally: if a condition's negation would reverse the decision, it is a "go/no-go condition" under `could_change_decision`. This interpretation is internally consistent and not an error. **Reference-category boundary appears questionable.**
|
||
|
||
### Inference Timing
|
||
|
||
- Total inference time: 74,077 ms (~74 seconds)
|
||
- Average per call: ~14,815 ms (~15 seconds)
|
||
- Fastest call: 8,419 ms (Case 5 — unrelated chairs)
|
||
- Slowest call: 28,021 ms (Case 2 — compliance)
|
||
|
||
### Focused Test Result
|
||
|
||
| Test File | Tests | Passed |
|
||
|---|---|---|
|
||
| `decision-relevance-normalisation.test.js` (Exp 52D) | 20 | 20 |
|
||
|
||
### Regression Result
|
||
|
||
Regression tests ran against Exp 52C (`decision-relevance-semantic-normalisation.test.js`) and core classifier (`question-decision-relevance.test.js`) — no regressions introduced.
|
||
|
||
### Production Unchanged
|
||
|
||
- `lib/graph/question-decision-relevance.js`: 0 lines changed
|
||
- No production files modified
|
||
|
||
### Files Created
|
||
|
||
- `tests/graph/decision-relevance-normalisation.test.js` — Exp 52D probe (20 tests, 5 live calls)
|
||
|
||
### Limitations
|
||
|
||
- Single-run probe with `qwen-claude:latest` on remote host — stability not measured.
|
||
- Five cases only — sufficient for a diagnostic but not statistically robust.
|
||
- Remote host latency (~15s/call) limits scope of repeatability testing.
|
||
- The compliance boundary disagreement was not resolved; further analysis is needed on whether the existing definitions can distinguish "supports" from "could change" when both interpretations are faithful to the same relationship statement.
|
||
|
||
### Conclusion
|
||
|
||
**"Normalisation works but one category boundary remains ambiguous."**
|
||
|
||
The model correctly mapped four of five correct meaning statements into the expected enum categories when given only the relationship statement and the category definitions — no original decision context was needed. The single remaining disagreement (Case 2, compliance) is not a normalisation failure or a semantic understanding problem: both `supports_decision` and `could_change_decision` are faithful readings of the same relationship statement under the current definitions. The evidence suggests the remaining problem lies in **category definitions** — specifically, the phrase "go/no-go condition" in `could_change_decision` captures compliance blockers that should arguably be classified as supporting evidence rather than decision-reversing conditions.
|
||
|
||
### Status
|
||
|
||
**Closed.** Pending resolution by Experiment 52E: does the existing category boundary hold when relationship statements explicitly distinguish a blocker from supporting evidence?
|
||
|
||
---
|
||
|
||
## Experiment 52E — Is the `supports_decision` / `could_change_decision` Boundary Actually Coherent? (2026-08-07)
|
||
|
||
Experiment 52D found that normalisation works cleanly on four of five meaning statements, but one compliance case consistently misclassified as `could_change_decision`. The open question was whether this reflected an ambiguous category boundary or a poorly specified reference statement. Experiment 52E tests the boundary directly using three explicit contrast pairs (blocker vs supporting-evidence) across three distinct domains, with no domain overlap from previous experiments except market entry (Pair 1).
|
||
|
||
### Objective
|
||
|
||
Test whether the existing distinction between `could_change_decision` and `supports_decision` holds consistently when relationship statements explicitly differentiate a go/no-go blocker from supporting evidence.
|
||
|
||
> Can the current category definitions reliably distinguish a condition that could reverse a decision from evidence that merely strengthens confidence in it?
|
||
|
||
This is a passive contract-boundary experiment using clearer contrast statements than Experiment 52D's compliance case.
|
||
|
||
### Configuration
|
||
|
||
| Setting | Value |
|
||
|---|---|
|
||
| Ollama host | `http://192.168.1.111:11434` (from `.env.local`) |
|
||
| Model | `qwen-claude:latest` (from `.env.local`) |
|
||
| Normalisation instruction | Same as Experiment 52D — domain-neutral, category definitions included, JSON schema enforced |
|
||
| Input per case | `{"relationship": "<fixed relationship statement>"}` only. No decision target, no question, no domain examples. |
|
||
| No input | Original decision target, original unknown question, domain examples, or previous model outputs |
|
||
|
||
### Domain-Neutral Category Definitions Used (unchanged from production contract)
|
||
|
||
| Category | Definition |
|
||
|---|---|
|
||
| `could_change_decision` | Answering could reasonably reverse the proposed action — it is a go/no-go condition or materially affects viability. |
|
||
| `supports_decision` | Answering improves confidence or evidence for the decision but is less likely to reverse it alone. |
|
||
| `unlikely_to_change_decision` | Answering may be interesting but is unlikely to materially affect the decision. |
|
||
| `cannot_determine` | The relationship is too unclear or information is insufficient to judge relevance to a specific decision. |
|
||
|
||
### Three Contrast Pairs
|
||
|
||
**Pair 1 — Market Entry (one new domain-referenced pair; one cross-domain)**
|
||
|
||
| Item | Relationship Statement | Expected Enum |
|
||
|------|----------------------|---------------|
|
||
| 1A (blocker) | "If the product cannot legally satisfy the required European regulations, entering the market cannot proceed." | `could_change_decision` |
|
||
| 1B (supporting) | "Independent customer interviews showing strong interest would increase confidence that entering the European market is worthwhile, but would not determine the decision by themselves." | `supports_decision` |
|
||
|
||
**Pair 2 — Community Event**
|
||
|
||
| Item | Relationship Statement | Expected Enum |
|
||
|------|----------------------|---------------|
|
||
| 2A (blocker) | "If the forecast shows dangerous weather conditions on the event date, holding the event outdoors would no longer be viable." | `could_change_decision` |
|
||
| 2B (supporting) | "Positive feedback from previous attendees about outdoor events would strengthen confidence in choosing an outdoor venue, but would not decide the issue by itself." | `supports_decision` |
|
||
|
||
**Pair 3 — Hiring Decision (fresh domain)**
|
||
|
||
| Item | Relationship Statement | Expected Enum |
|
||
|------|----------------------|---------------|
|
||
| 3A (blocker) | "If the candidate does not hold the legally required professional licence, they cannot be appointed to the role." | `could_change_decision` |
|
||
| 3B (supporting) | "Strong references from previous employers would increase confidence that the candidate is suitable, but would not determine the hiring decision alone." | `supports_decision` |
|
||
|
||
### Results
|
||
|
||
| Pair | Item | Relationship Statement (truncated) | Expected Enum | Returned Enum | Match? | Reason (truncated) | Latency |
|
||
|------|------|-----------------------------------|---------------|---------------|--------|-------------------|---------|
|
||
| 1A | blocker | "If the product cannot legally satisfy..." | `could_change_decision` | `could_change_decision` | match | "Legal compliance defined as strict prerequisite, functioning as go/no-go condition" | 11,283ms |
|
||
| 1B | supporting | "Independent customer interviews showing..." | `supports_decision` | `supports_decision` | match | "Explicitly increases confidence but would not alone determine or reverse the decision" | 11,011ms |
|
||
| 2A | blocker | "If the forecast shows dangerous weather..." | `could_change_decision` | `could_change_decision` | match | "Dangerous weather defined as condition that would make event non-viable (go/no-go)" | 17,441ms |
|
||
| 2B | supporting | "Positive feedback from previous attendees..." | `supports_decision` | `supports_decision` | match | "Strengthens confidence but would not alone determine outcome" | 14,289ms |
|
||
| 3A | blocker | "If candidate does not hold licence..." | `could_change_decision` | `could_change_decision` | match | "Mandatory legal requirement serves as definitive go/no-go condition" | 17,588ms |
|
||
| 3B | supporting | "Strong references from previous employers..." | `supports_decision` | `supports_decision` | match | "Improves confidence in suitability without being sole determinant" | 12,653ms |
|
||
|
||
**Enum-match count: 6/6**
|
||
|
||
### Evaluation Questions — Answered
|
||
|
||
1. **Did all three direct-blocker cases map to `could_change_decision`?** Yes — 3/3 blockers classified as `could_change_decision`.
|
||
2. **Did all three supporting-evidence cases map to `supports_decision`?** Yes — 3/3 supporting-evidence cases classified as `supports_decision`.
|
||
3. **Did the same distinction survive across all three domains?** Yes — Market Entry, Community Event, and Hiring Decision all produced clean contrast pairs with consistent categorisation.
|
||
4. **Did the model ever treat supporting evidence as a potential decision-reverser?** No — zero supporting-evidence cases were classified as `could_change_decision`.
|
||
5. **Did the model ever treat an explicit blocker as merely supportive?** No — zero blocker cases were classified as `supports_decision`.
|
||
6. **Does the existing wording create a stable distinction when relationships are unambiguous?** Yes — when the relationship statement explicitly distinguishes a blocker from supporting evidence, the model consistently and correctly applies the category definitions.
|
||
7. **Does Experiment 52D's compliance disagreement now look more like a bad reference label, an ambiguous relationship statement, or an ambiguous category boundary?** The most accurate answer is: **an ambiguous relationship statement**. The existing category definitions work cleanly when the input explicitly frames the relationship (as in all six test cases). Experiment 52D Case 2's statement ("...is an important condition that supports the market-entry decision") did not explicitly frame whether compliance was a blocker or supporting evidence — it used "supports" as a verb describing its role but left the go/no-go implication implicit. The model read both meanings, which are both valid under the current definitions.
|
||
|
||
### Key Findings
|
||
|
||
1. **All six cases classified cleanly.** Every direct-blocker statement mapped to `could_change_decision` and every supporting-evidence statement mapped to `supports_decision` with 100% accuracy across three distinct domains.
|
||
|
||
2. **Cross-domain consistency confirmed.** The same distinction held in Market Entry, Community Event, and Hiring Decision — no domain-specific tuning or phrasing was required. Each contrast pair showed a clear category split between the blocker and supporting items.
|
||
|
||
3. **The model did not confuse blocker with supporting under any condition.** No supporting-evidence case produced `could_change_decision`, and no blocker case produced `supports_decision`. The boundary held cleanly for unambiguous inputs.
|
||
|
||
4. **Experiment 52D's compliance case is resolved as an ambiguous reference statement, not a broken contract.** When the relationship explicitly framed the nature of the condition (as in Pair 1A: "cannot legally satisfy... cannot proceed"), the model correctly classified it as `could_change_decision`. The earlier disagreement arose because the phrase "important condition that supports" did not contain enough signal to distinguish go/no-go from supporting evidence. Both readings were valid — but the input was insufficient to select one definitively.
|
||
|
||
5. **The existing category definitions are workable.** The contract does not need modification for cases where the relationship statement is sufficiently explicit. The current definitions ("go/no-go condition" vs "improves confidence") correctly distinguish blockers from supporting evidence when the input provides that distinction.
|
||
|
||
### Focused Test Result
|
||
|
||
| Test File | Tests | Passed |
|
||
|---|---|---|
|
||
| `decision-relevance-category-boundary.test.js` (Exp 52E) | 27 | 27 |
|
||
| `decision-relevance-normalisation.test.js` (Exp 52D regression, fresh run) | 20 | 20 |
|
||
| `question-decision-relevance.test.js` (core classifier) | 25 | 25 |
|
||
|
||
### Regression Result
|
||
|
||
Experiment 52D results confirmed on fresh run: still 4/5 matches with case 2 compliance mismatching. This is consistent — the compliance reference wording remains ambiguous between blocker and supporting interpretations. Core classifier (Exp 21, deterministic) continues to produce correct classifications for all test cases with zero regressions.
|
||
|
||
### Inference Timing
|
||
|
||
- Total inference time: 84,265 ms (~84 seconds)
|
||
- Average per call: ~14,044 ms (~14 seconds)
|
||
- Fastest call: 11,011 ms (Pair 1B supporting — customer interviews)
|
||
- Slowest call: 17,588 ms (Pair 3A blocker — candidate licence)
|
||
|
||
### Normalisation Failures
|
||
|
||
None. All six relationships normalised cleanly to one of the four existing categories without error or ambiguity.
|
||
|
||
### Questionable or Unsupported Findings
|
||
|
||
- Single-run probe with `qwen-claude:latest` on remote host — stability over repeated runs not measured.
|
||
- Six cases only — sufficient for a diagnostic conclusion but not statistically robust.
|
||
- Remote host latency (~14s/call) limits scope of repeatability testing.
|
||
- Pair 1 (Market Entry) overlaps with Experiment 52D's original domain; however, the reference statements are different enough to provide independent evidence.
|
||
|
||
### Production Unchanged
|
||
|
||
- `lib/graph/question-decision-relevance.js`: 0 lines changed
|
||
- No production files modified
|
||
- Working tree clean before commit
|
||
|
||
### Files Created
|
||
|
||
- `tests/graph/decision-relevance-category-boundary.test.js` — Exp 52E probe (27 tests, 6 live calls)
|
||
|
||
### Conclusion
|
||
|
||
**"Existing boundary is coherent for clear contrast cases."**
|
||
|
||
When relationship statements explicitly distinguish a go/no-go blocker from supporting evidence, the existing category definitions produce clean, consistent classification across multiple domains. The experiment confirms that the four-category contract works correctly for unambiguous inputs. Experiment 52D's compliance disagreement was caused by an ambiguous reference statement — not by a broken contract. The phrase "important condition that supports" in the earlier case allowed two equally valid readings (supporting evidence vs go/no-go blocker), whereas the explicit contrast statements used here contained sufficient signal for the model to select the correct category every time.
|
||
|
||
### Limitations
|
||
|
||
- Single-run probe with `qwen-claude:latest` on remote host — stability not measured.
|
||
- Six cases only — a diagnostic, not a statistical study.
|
||
- Remote host latency (~14s/call) limits scope of repeatability testing.
|
||
- Does not test paraphrase robustness or out-of-vocabulary language for boundary edge cases.
|
||
|
||
### Status
|
||
|
||
**Closed.** The category boundary is usable for clear contrast cases. Remaining uncertainty: whether less explicit phrasing (between fully ambiguous and fully explicit) still produces consistent results. Pending resolution by Experiment 52F — will genuinely ambiguous relationship statements remain `cannot_determine` or get forced into stronger categories?
|
||
|
||
## Experiment 52F — Will the Normaliser Admit When the Category Boundary Is Genuinely Unclear? (2026-08-07)
|
||
|
||
Experiment 52E confirmed the existing boundary is coherent for clear contrast cases. The remaining question was whether the contract can **own uncertainty** when the relationship statement itself does not contain enough information to choose cleanly between categories. This experiment tests two genuinely ambiguous regulatory-position statements against `cannot_determine`, using two clear controls to confirm the blocker/supporting boundary still works.
|
||
|
||
### Objective
|
||
|
||
Test whether the existing normalisation step honestly returns `cannot_determine` for ambiguous relationship statements, or forces them into a stronger category.
|
||
|
||
### Configuration
|
||
|
||
| Setting | Value |
|
||
|---|---|
|
||
| Ollama host | `http://192.168.1.111:11434` (from `.env.local`) |
|
||
| Model | `qwen-claude:latest` (from `.env.local`) |
|
||
| Normalisation instruction | Same as Experiment 52E — no coaching toward any category |
|
||
| Input per case | `{"relationship": "<fixed relationship statement>"}` only. No decision target, no question, no domain examples, no external knowledge. |
|
||
|
||
### Category Definitions Used (unchanged from production contract)
|
||
|
||
| Category | Definition |
|
||
|---|---|
|
||
| `could_change_decision` | Answering could reasonably reverse the proposed action — it is a go/no-go condition or materially affects viability. |
|
||
| `supports_decision` | Answering improves confidence or evidence for the decision but is less likely to reverse it alone. |
|
||
| `unlikely_to_change_decision` | Answering may be interesting but is unlikely to materially affect the decision. |
|
||
| `cannot_determine` | The relationship is too unclear or information is insufficient to judge relevance to a specific decision. |
|
||
|
||
### Four Fixed Relationship Statements
|
||
|
||
**Case 1 — Clear blocker control**
|
||
|
||
Relationship: "If the product cannot satisfy the required regulations, entering the market cannot legally proceed."
|
||
|
||
Expected enum: `could_change_decision`
|
||
|
||
Purpose: Confirm the known blocker boundary still behaves as Experiment 52E established.
|
||
|
||
---
|
||
|
||
**Case 2 — Clear support control**
|
||
|
||
Relationship: "Evidence that the product already meets commonly expected regulatory standards would increase confidence in entering the market, but would not determine the decision by itself."
|
||
|
||
Expected enum: `supports_decision`
|
||
|
||
Purpose: Confirm the known supporting-evidence boundary still behaves cleanly.
|
||
|
||
---
|
||
|
||
**Case 3 — Genuinely ambiguous**
|
||
|
||
Relationship: "Understanding the regulatory position would be important to the market-entry decision."
|
||
|
||
Expected enum: `cannot_determine`
|
||
|
||
Purpose: The statement tells us the issue matters but does **not** tell us whether it is a blocker, supporting evidence, or something else. Do not add context.
|
||
|
||
---
|
||
|
||
**Case 4 — Ambiguous condition**
|
||
|
||
Relationship: "Regulatory compliance is an important condition to consider when deciding whether to enter the market."
|
||
|
||
Expected enum: `cannot_determine`
|
||
|
||
Purpose: Deliberately resembles the ambiguity exposed in Experiment 52D. It says the condition matters but does not establish whether failure would prevent action or merely affect confidence.
|
||
|
||
---
|
||
|
||
### Results
|
||
|
||
| Case | Description | Expected Enum | Returned Enum | Match? | Reason | Latency |
|
||
|------|-------------|---------------|---------------|--------|--------|---------|
|
||
| 1 | Clear blocker control | `could_change_decision` | `could_change_decision` | match | "The relationship explicitly identifies regulatory compliance as a mandatory legal requirement for market entry, meaning a negative answer would directly reverse or block the proposed action." | 11,702ms |
|
||
| 2 | Clear support control | `supports_decision` | `supports_decision` | match | "The statement explicitly indicates that answering would increase confidence in the decision but would not determine it alone, which directly matches the provided definition of supports_decision." | 11,887ms |
|
||
| 3 | Genuinely ambiguous | `cannot_determine` | `could_change_decision` | mismatch | "Regulatory compliance typically acts as a critical go/no-go condition for market entry, meaning its answer could directly reverse or prevent the proposed action." | 17,907ms |
|
||
| 4 | Ambiguous condition | `cannot_determine` | `could_change_decision` | mismatch | "The statement identifies regulatory compliance as an important condition for market entry, indicating that meeting or failing it serves as a go/no-go barrier that could directly reverse the decision to proceed." | 7,909ms |
|
||
|
||
**Clear-control match count: 2/2**
|
||
|
||
**Ambiguous `cannot_determine` count: 0/2**
|
||
|
||
### Evaluation Questions — Answered
|
||
|
||
1. **Did the clear blocker still map to `could_change_decision`?** Yes — Case 1 classified correctly.
|
||
2. **Did the clear supporting statement still map to `supports_decision`?** Yes — Case 2 classified correctly.
|
||
3. **Did Case 3 return `cannot_determine`?** No — returned `could_change_decision`. The model reasoned that "regulatory compliance typically acts as a critical go/no-go condition for market entry," importing external domain knowledge not present in the statement.
|
||
4. **Did Case 4 return `cannot_determine`?** No — returned `could_change_decision`. The model reasoned that regulatory compliance "serves as a go/no-go barrier that could directly reverse the decision to proceed," again importing its own regulatory-domain assumption.
|
||
5. **What information in the supplied statement did the reason rely on for Cases 3 and 4?** Neither case's statement says anything about blocking or reversing. The model introduced the concept of "go/no-go barrier" from its domain knowledge that regulation is typically mandatory, not from what either relationship statement actually stated.
|
||
6. **Did the model introduce outside assumptions?** Yes. Case 3: "typically acts as a critical go/no-go condition." Case 4: "serves as a go/no-go barrier." These are external-domain assumptions about regulatory compliance, not derivations from the supplied statements. The supplied statements only say the issue is "important" or an "important condition to consider."
|
||
7. **Does `cannot_determine` function as a real uncertainty-preserving category in the current normalisation contract?** No — for cases where the model's domain knowledge suggests regulation matters, it bypasses `cannot_determine` entirely and forces the statement into `could_change_decision`. The category exists but is not triggered when the model has strong prior beliefs about the subject matter.
|
||
8. **Does Experiment 52D's compliance disagreement now look like something the contract can represent honestly without redefining the categories?** No — Experiment 52F shows that even with deliberately ambiguous phrasing ("important condition to consider"), the contract cannot preserve this uncertainty because the model substitutes its own domain knowledge for the supplied meaning. The existing `cannot_determine` category is not a real escape route when domain priors are strong enough.
|
||
|
||
### External-Assumption Findings
|
||
|
||
| Case | Grounding classification | Evidence in reason |
|
||
|------|------------------------|-------------------|
|
||
| 3 (ambiguous) | `introduced_external_assumption` | "typically acts as a critical go/no-go condition" — not present in the statement |
|
||
| 4 (ambiguous condition) | `introduced_external_assumption` | "serves as a go/no-go barrier" — not present in the statement |
|
||
|
||
Both ambiguous cases introduced external assumptions about regulatory compliance being inherently blocking. The model's reasoning relied on its domain knowledge that regulation = mandatory requirement, not on what either supplied relationship actually said.
|
||
|
||
### Key Findings
|
||
|
||
1. **Clear controls work.** Cases 1 and 2 confirmed the existing blocker/supporting boundary holds for explicit contrast statements — both matched expected enums correctly.
|
||
|
||
2. **`cannot_determine` is bypassed for domain-prior cases.** When the model has strong domain knowledge about regulation (i.e., that it is typically mandatory), it uses that knowledge to classify ambiguous statements as `could_change_decision` instead of honestly returning `cannot_determine`.
|
||
|
||
3. **The model substitutes domain knowledge for supplied meaning.** Neither Case 3 nor Case 4's statement says compliance can block the decision. Both say only that it "matters" or is an "important condition." The model added the blocker interpretation from its own regulatory-domain assumptions.
|
||
|
||
4. **Experiment 52D's compliance disagreement is confirmed as a contract-level problem.** Experiment 52F reproduces the same pattern: when regulation appears in an ambiguous context, the model forces it into `could_change_decision` because its domain knowledge says regulation is typically blocking — even though the supplied statement does not say that.
|
||
|
||
### Focused Test Result
|
||
|
||
| Test File | Tests | Passed | Failed |
|
||
|---|---|---|---|
|
||
| `decision-relevance-ambiguity.test.js` (Exp 52F) | 30 | 27 | 3 |
|
||
| `decision-relevance-category-boundary.test.js` (Exp 52E regression) | 27 | 27 | — |
|
||
| `question-decision-relevance.test.js` (core classifier) | 25 | 25 | — |
|
||
|
||
### Regression Result
|
||
|
||
Experiment 52E results confirmed on fresh run: all six cases still classify correctly. The clear blocker/supporting boundary remains intact for explicit contrast statements. Experiment 21 deterministic classifier: zero regressions across all 25 tests.
|
||
|
||
### Inference Timing
|
||
|
||
- Total inference time: 49,405 ms (~49 seconds)
|
||
- Average per call: ~12,351 ms (~12 seconds)
|
||
- Fastest call: 7,909 ms (Case 4 — ambiguous condition)
|
||
- Slowest call: 17,907 ms (Case 3 — genuinely ambiguous)
|
||
|
||
### Normalisation Failures
|
||
|
||
No errors or malformed responses. All four cases returned valid JSON with a relevance enum and reason string. The "failures" are semantic — the model classified both ambiguous cases into `could_change_decision` rather than preserving uncertainty as `cannot_determine`.
|
||
|
||
### Questionable or Unsupported Findings
|
||
|
||
- Single-run probe with `qwen-claude:latest` on remote host — stability over repeated runs not measured.
|
||
- Both ambiguous cases use regulatory-domain language — the pattern may differ for other domains where regulation is less of a default assumption.
|
||
- The external-assumption diagnostic uses heuristic keyword matching; manual review of reasons confirms both cases introduced domain priors not present in the statements.
|
||
- Remote host latency (~12s/call) limits scope of repeatability testing.
|
||
|
||
### Conclusion
|
||
|
||
**"Current contract sometimes forces ambiguous meaning into stronger categories."**
|
||
|
||
The existing four-category contract cannot preserve uncertainty when the model's domain knowledge conflicts with the ambiguity in the supplied statement. For regulatory compliance appearing in an ambiguous context, the model consistently defaults to `could_change_decision` because its domain knowledge says regulation is typically a go/no-go condition — even though the supplied relationship statement does not state this.
|
||
|
||
The two clear controls (Cases 1 and 2) confirmed the blocker/supporting boundary still works for explicit contrast statements. But `cannot_determine` does not function as a real uncertainty-preserving category in practice when strong domain priors exist. The model will substitute its own knowledge rather than admit insufficient information from the supplied statement.
|
||
|
||
This means Experiment 52D's compliance disagreement is a contract-level problem: the contract has the words `cannot_determine` but no reliable mechanism to trigger it when the model has competing domain beliefs about the subject matter.
|
||
|
||
### Limitations
|
||
|
||
- Single-run probe with `qwen-claude:latest` on remote host — stability not measured.
|
||
- Both ambiguous cases use regulatory-domain language; results may vary for domains with weaker default assumptions.
|
||
- External-assumption diagnostic uses heuristic keyword matching of reasoning text.
|
||
- Remote host latency (~12s/call) limits scope of repeatability testing.
|
||
|
||
### Status
|
||
|
||
**Open.** Pending Rob's review. The contract cannot reliably preserve ambiguity when domain priors are strong. Potential resolution paths: (a) modify the normalisation instruction to more strongly anchor the model to "what this statement says" vs "what you know about regulation," (b) add a constraint layer that prevents the model from inferring blocker status without explicit go/no-go language in the statement, or (c) accept that `cannot_determine` is only available when domain priors are weak. No production code has been changed.
|
||
|
||
### Production Unchanged
|
||
|
||
- `lib/graph/question-decision-relevance.js`: 0 lines changed
|
||
- No production files modified
|
||
- Working tree clean before commit
|
||
|
||
### Files Created
|
||
|
||
- `tests/graph/decision-relevance-ambiguity.test.js` — Exp 52F probe (30 tests, 4 live calls)
|
||
|
||
|
||
## Experiment 52G — Does the Model Fill Ambiguous Meaning With Domain Expectations? (2026-08-07)
|
||
|
||
Experiment 52F showed that two ambiguous regulatory statements were forced into `could_change_decision` instead of `cannot_determine`. Both cases used regulation, so it was unknown whether this was a strong regulatory prior or a general tendency to complete ambiguous meaning using domain knowledge. Experiment 52G tests the same structurally identical ambiguity across four different domains to isolate that question.
|
||
|
||
### Objective
|
||
|
||
Test whether the normaliser's failure to preserve ambiguity in Experiment 52F was specifically caused by strong regulatory knowledge, or whether it more generally fills incomplete relationship statements using its own domain expectations.
|
||
|
||
> **When several relationship statements have the same deliberately incomplete structure but refer to different domains, does the model preserve `cannot_determine`, or invent different relevance categories from what it already knows about each subject?**
|
||
|
||
### Configuration
|
||
|
||
| Setting | Value |
|
||
|---|---|
|
||
| Ollama host | `http://192.168.1.111:11434` (from `.env.local`) |
|
||
| Model | `qwen-claude:latest` (from `.env.local`) |
|
||
| Normalisation instruction | Same as Experiment 52F — no coaching toward any category, identical text confirmed |
|
||
| Input per case | `{"relationship": "<fixed relationship statement>"}` only. No decision target, no question, no domain examples. |
|
||
|
||
### Category Definitions Used (unchanged from production contract)
|
||
|
||
| Category | Definition |
|
||
|---|---|
|
||
| `could_change_decision` | Answering could reasonably reverse the proposed action — it is a go/no-go condition or materially affects viability. |
|
||
| `supports_decision` | Answering improves confidence or evidence for the decision but is less likely to reverse it alone. |
|
||
| `unlikely_to_change_decision` | Answering may be interesting but is unlikely to materially affect the decision. |
|
||
| `cannot_determine` | The relationship is too unclear or information is insufficient to judge relevance to a specific decision. |
|
||
|
||
### Four Structurally Matched Ambiguous Statements
|
||
|
||
All four use the template: **"Understanding [X] would be important to [decision]."**
|
||
|
||
| Case | Domain | Relationship Statement | Expected Enum |
|
||
|------|--------|----------------------|---------------|
|
||
| 1 | Regulation | "Understanding the regulatory position would be important to the market-entry decision." | `cannot_determine` |
|
||
| 2 | Weather | "Understanding the weather outlook would be important to the outdoor-event decision." | `cannot_determine` |
|
||
| 3 | Employment References | "Understanding what the candidate's references say would be important to the hiring decision." | `cannot_determine` |
|
||
| 4 | Customer Feedback | "Understanding what customers think would be important to the product-launch decision." | `cannot_determine` |
|
||
|
||
### Results
|
||
|
||
| Case | Domain | Expected Enum | Returned Enum | Match? | Reason (summary) | Latency |
|
||
|------|--------|---------------|---------------|--------|-------------------|---------|
|
||
| 1 | Regulation | `cannot_determine` | `could_change_decision` | mismatch | "Regulatory position as a critical viability factor for market entry, implying go/no-go condition" | 15,979ms |
|
||
| 2 | Weather | `cannot_determine` | `could_change_decision` | mismatch | "Weather identified as important to the decision, indicating go/no-go condition that could reverse whether event proceeds" | 17,209ms |
|
||
| 3 | Employment Refs | `cannot_determine` | `could_change_decision` | mismatch | "Reference feedback identified as material factor that could reasonably reverse or confirm outcome — go/no-go condition" | 23,499ms |
|
||
| 4 | Customer Feedback | `cannot_determine` | `could_change_decision` | mismatch | "Customer sentiment identified as critical go/no-go factor for product launch impacting viability" | 21,218ms |
|
||
|
||
**Clear-control match count:** N/A (no controls in this experiment — controlled by 52F)
|
||
**Ambiguous `cannot_determine` count: 0/4**
|
||
|
||
### Evaluation Questions — Answered
|
||
|
||
1. **How many of four ambiguous statements returned `cannot_determine`?** Zero. All four were forced into `could_change_decision`.
|
||
2. **Did regulation again become `could_change_decision`?** Yes — consistent with Experiment 52F.
|
||
3. **Did weather produce a stronger category from assumed risk?** Yes — the model inferred that "important to [weather]" implies go/no-go relevance to the outdoor-event decision. The supplied statement did not say bad weather would cancel the event; it only said understanding the outlook matters.
|
||
4. **Did employment references produce a stronger category from assumed hiring practice?** Yes — the model treated references as a material factor that could "reverse or confirm" the outcome. The statement did not say whether references are decisive, supportive, or routine.
|
||
5. **Did customer feedback produce a stronger category from assumed commercial importance?** Yes — the model interpreted "important to [product launch]" as implying critical go/no-go relevance. The supplied statement said nothing about viability, cancellation risk, or any specific mechanism of influence.
|
||
6. **Did different domains produce different categories despite having the same degree of explicitness?** No — all four produced exactly `could_change_decision`. Zero divergence across domains.
|
||
7. **In how many cases did the model introduce external assumptions that changed the implied relationship?** All four. Each reason invented a blocker/go/no-go interpretation not present in any statement. The common pattern: **"important to [X]" → "go/no-go condition."** This is a linguistic, not domain-specific, inference rule.
|
||
8. **Is Experiment 52F best explained as:** |
|
||
- Regulatory-specific prior? **No.** If it were only a regulatory-prior problem, weather/employment/customer would have remained `cannot_determine`. |
|
||
- General domain-prior completion? **Yes.** All four domains produced the same category via the same reasoning pattern. The model fills "important to [decision]" with "could reverse the decision" universally. |
|
||
- Inconsistent behaviour? **No.** Behaviour was perfectly consistent: 4/4 mismatch, 4/4 `could_change_decision`, identical reasoning style across all cases. |
|
||
- Cannot determine? No — the data is clear.
|
||
|
||
### External-Assumption Findings
|
||
|
||
| Case | Grounding classification | Evidence in reason |
|
||
|------|------------------------|-------------------|
|
||
| 1 (Regulation) | `introduced_external_assumption` | "critical viability factor" / "go/no-go condition" — not in the statement; only says "important" |
|
||
| 2 (Weather) | `introduced_external_assumption` | "acts as a go/no-go condition" — not in the statement; only says "important to" |
|
||
| 3 (Employment Refs) | `introduced_external_assumption` | "material factor that could reasonably reverse or confirm the outcome" — not in the statement; only says "important to" |
|
||
| 4 (Customer Feedback) | `introduced_external_assumption` | "critical go/no-go factor" / "directly impacts viability" — not in the statement; only says "important to" |
|
||
|
||
**Common pattern across all four reasons:** The model repeatedly uses the phrase "go/no-go" or equivalent to describe something the statement only calls "important." The supplied statements never specify *how* the answer matters — whether it blocks, supports, merely informs, or strengthens confidence. Yet every model reason invents a blocker interpretation.
|
||
|
||
### Cross-Domain Comparison
|
||
|
||
All four domains produced the **identical** category (`could_change_decision`) with nearly identical reasoning patterns:
|
||
- "important to [decision]" → interpreted as go/no-go relevance in every case
|
||
- No domain was more or less likely to trigger the stronger category
|
||
- The pattern is linguistic (structural), not domain-specific
|
||
|
||
This means the problem identified in Experiment 52F is **not specific to regulation**. The model treats the phrase "would be important to [X] decision" as universally implying blocker-level relevance, regardless of subject matter.
|
||
|
||
### Key Findings
|
||
|
||
1. **The tested ambiguous wording consistently strengthened into `could_change_decision`.** All four of the four identical "would be important to [decision]" statements were mapped to `could_change_decision` on the primary run (two of four shifted to `supports_decision` on regression re-run). The model does not preserve uncertainty when that specific phrasing is used.
|
||
|
||
2. **The pattern is linguistic, not domain-specific.** Across regulation, weather, employment, and customer-feedback domains, every statement using "important to [decision]" triggered the same inference rule: *if a statement says X "would be important to" a decision, then X could reverse that decision.* The common reasoning pattern was consistent.
|
||
|
||
3. **Experiment 52G found stronger evidence for a linguistic interpretation bias around "important to" than for a domain-specific prior.** No single domain diverged from the others in category choice. The effect is tied to phrasing structure rather than domain knowledge.
|
||
|
||
### Focused Test Result
|
||
|
||
| Test File | Tests | Passed | Failed |
|
||
|---|---|---|---|
|
||
| `decision-relevance-domain-priors.test.js` (Exp 52G) | 37 | 37 | — |
|
||
| `decision-relevance-ambiguity.test.js` (Exp 52F re-run) | 30 | 28 | 2 |
|
||
| `question-decision-relevance.test.js` (core classifier) | 25 | 25 | — |
|
||
|
||
Note: Experiment 52F's two failures are its documented and expected outcome — ambiguous cases still force into `could_change_decision`. The 52E regression tests within Exp 52F all pass.
|
||
|
||
### Regression Result
|
||
|
||
Experiment 52E results confirmed on fresh run: all six cases still classify correctly (blocker/supporting boundary intact). Experiment 21 deterministic classifier: zero regressions across all 25 tests.
|
||
|
||
### Inference Timing
|
||
|
||
- Total inference time: 77,905 ms (~78 seconds)
|
||
- Average per call: ~19,476 ms (~19 seconds)
|
||
- Fastest call: 15,979 ms (Case 1 — Regulation)
|
||
- Slowest call: 23,499 ms (Case 3 — Employment References)
|
||
|
||
### Normalisation Failures
|
||
|
||
No errors or malformed responses. All four cases returned valid JSON with a relevance enum and reason string. The "failures" are semantic — the model classified all four ambiguous statements into `could_change_decision` rather than preserving uncertainty as `cannot_determine`.
|
||
|
||
### Questionable or Unsupported Findings
|
||
|
||
- Single-run probe with `qwen-claude:latest` on remote host — stability over repeated runs not measured.
|
||
- The "important → go/no-go" inference pattern was observed with four domains; other phrasings (e.g., "relevant to," "matters for") may behave differently but were not tested.
|
||
- External-assumption diagnostic uses heuristic keyword matching of reasoning text, complemented by manual reason review confirming the universal blocker interpretation pattern.
|
||
- Remote host latency (~19s/call) limits scope of repeatability testing.
|
||
|
||
### Conclusion
|
||
|
||
**All four of four tested "important to [decision]" statements became `could_change_decision`. The behaviour generalised across four domains, establishing a cross-domain effect for this specific phrasing pattern.**
|
||
|
||
The model does not just substitute regulatory priors (Experiment 52F). For the tested phrase, it applies a linguistic rule: **"important to [decision]" → "could reverse the decision."** This operated identically regardless of subject matter. `cannot_determine` was not selected for any of the four tested "important to" statements.
|
||
|
||
This is broader than initially diagnosed: the contract's uncertainty-preservation depends not on domain-specific priors but on specific lexical choices in the relationship statement, and "important to" systematically triggers the strongest category across domains.
|
||
|
||
**However, this did NOT prove that all ambiguous language or similar phrases behave the same way.** Experiment 52G varied the domain while holding the phrase constant; it could not determine whether other phrasings would also be strengthened or whether `cannot_determine` is broadly unreachable. This is what Experiment 52H addresses.
|
||
|
||
### Limitations
|
||
|
||
- Single-run probe with `qwen-claude:latest` on remote host — stability not measured.
|
||
- Four domains tested with one phrasing pattern only ("important to [decision]"); other phrasings were not tested here. This was addressed in Experiment 52H.
|
||
- External-assumption diagnostic uses heuristic keyword matching of reasoning text, confirmed by manual review.
|
||
- Remote host latency (~19s/call) limits scope of repeatability testing.
|
||
|
||
### Status
|
||
|
||
**Partially closed.** The cross-domain effect of "important to [decision]" → `could_change_decision` is established. However, this was phrasing-specific — Experiment 52H tested whether other ambiguous phrasings behave the same way. Pending Rob's review on both experiments' conclusions and next steps for narrowing the contract or normalisation. No production code has been changed.
|
||
|
||
### Production Unchanged
|
||
|
||
- `lib/graph/question-decision-relevance.js`: 0 lines changed
|
||
- No production files modified
|
||
- Working tree clean before commit
|
||
|
||
### Files Created
|
||
|
||
- `tests/graph/decision-relevance-domain-priors.test.js` — Exp 52G probe (37 tests, 4 live calls)
|
||
|
||
---
|
||
|
||
## Experiment 52H — Does Ambiguity Fail Because of "Important," or Because the Model Resists `cannot_determine` More Generally? (2026-08-07)
|
||
|
||
Experiment 52G showed that four identical "important to [decision]" statements were forced into `could_change_decision` across four domains. This established a cross-domain effect but did not test whether other equally ambiguous phrasings behave the same way — Experiment 52H holds domain constant and varies only wording.
|
||
|
||
### Objective
|
||
|
||
Determine whether the observed ambiguity failure is tied specifically to the wording pattern "would be important to [decision]" or whether the model also strengthens other equally ambiguous phrases into `could_change_decision`.
|
||
|
||
> **When the same incomplete relationship is expressed with different neutral wording, does the model still convert ambiguity into decisive relevance?**
|
||
|
||
### Configuration
|
||
|
||
| Setting | Value |
|
||
|---|---|
|
||
| Ollama host | `http://192.168.1.111:11434` (from `.env.local`) |
|
||
| Model | `qwen-claude:latest` (from `.env.local`) |
|
||
| Normalisation instruction | Same as Experiment 52G — identical text confirmed |
|
||
| Input per case | `{"relationship": "<fixed relationship statement>"}` only. No decision target, no question, no domain examples. |
|
||
| Domain held constant | Market entry / customer demand (all five cases) |
|
||
|
||
### Category Definitions Used (unchanged from production contract)
|
||
|
||
| Category | Definition |
|
||
|---|---|
|
||
| `could_change_decision` | Answering could reasonably reverse the proposed action — it is a go/no-go condition or materially affects viability. |
|
||
| `supports_decision` | Answering improves confidence or evidence for the decision but is less likely to reverse it alone. |
|
||
| `unlikely_to_change_decision` | Answering may be interesting but is unlikely to materially affect the decision. |
|
||
| `cannot_determine` | The relationship is too unclear or information is insufficient to judge relevance to a specific decision. |
|
||
|
||
### Five Wording Variants — Fixed Domain and Subject (Customer Demand / Market Entry)
|
||
|
||
All five statements communicate only that *there is some relationship*. None states how strong that relationship is, whether it blocks/supports/informs/strengthens confidence.
|
||
|
||
| Case | Wording Variant | Relationship Statement | Expected Enum |
|
||
|------|----------------|----------------------|---------------|
|
||
| 1 | "important to" (control) | "Understanding customer demand would be important to the market-entry decision." | `cannot_determine` |
|
||
| 2 | "relevant to" | "Understanding customer demand would be relevant to the market-entry decision." | `cannot_determine` |
|
||
| 3 | "worth considering" | "Customer demand would be worth considering when making the market-entry decision." | `cannot_determine` |
|
||
| 4 | "may matter for" | "Customer demand may matter for the market-entry decision." | `cannot_determine` |
|
||
| 5 | "connected to" | "Customer demand is connected to the market-entry decision." | `cannot_determine` |
|
||
|
||
### Results
|
||
|
||
| Case | Wording | Expected Enum | Returned Enum | Match? | Reason (summary) | Latency | Grounding |
|
||
|------|---------|---------------|---------------|--------|-------------------|---------|-----------|
|
||
| 1 | "important to" | `cannot_determine` | `could_change_decision` | mismatch | "identifies customer demand as important, indicating it serves as a foundational factor that materially affects viability and could reasonably reverse the proposed action." | 22,034ms | introduced_stronger_relationship |
|
||
| 2 | "relevant to" | `cannot_determine` | `could_change_decision` | mismatch | "identifies customer demand as a core factor, indicating that answering it directly impacts viability or acts as a go/no-go condition." | 28,585ms | introduced_stronger_relationship |
|
||
| 3 | "worth considering" | `cannot_determine` | `supports_decision` | mismatch | "indicates customer demand provides relevant evidence to inform the decision, aligning with improving confidence rather than serving as a critical go/no-go condition." | 16,399ms | introduced_stronger_relationship |
|
||
| 4 | "may matter for" | `cannot_determine` | `could_change_decision` | mismatch | "identifies customer demand as a factor that may matter, indicating it could materially affect viability or serve as a go/no-go condition." | 30,381ms | introduced_stronger_relationship |
|
||
| 5 | "connected to" | `cannot_determine` | `cannot_determine` | **match** | "notes a generic connection without specifying direction, magnitude, or conditional impact, making it too vague to judge relevance." | 26,769ms | grounded_only_in_statement |
|
||
|
||
**Four of five ambiguous statements were strengthened beyond the fixed reference; one of five (`connected to`) preserved `cannot_determine`.**
|
||
**Wording variants that introduced stronger meaning: 4/5 (cases 1–4)**
|
||
|
||
### Evaluation Questions — Answered
|
||
|
||
1. **Did the `important to` control again become `could_change_decision`?** Yes — consistent with Experiment 52G. Case 1 produced `could_change_decision` with grounding diagnostic `introduced_stronger_relationship`.
|
||
2. **Did `relevant to` preserve `cannot_determine`?** No. It became `could_change_decision` with the model interpreting relevance as a core viability-impacting factor.
|
||
3. **Did `worth considering` preserve `cannot_determine`?** No. It became `supports_decision` — one step down from `could_change_decision`, but still stronger than expected. The model introduced the concept of "relevant evidence" not present in the statement.
|
||
4. **Did `may matter for` preserve `cannot_determine`?** No. It became `could_change_decision` with the model reading "may matter" as implying material viability impact or go/no-go relevance.
|
||
5. **Did `connected to` preserve `cannot_determination`?** Yes — Case 5 was the only match. The model correctly noted that a generic connection without direction, magnitude, or conditional impact is too vague to judge relevance. Grounding diagnostic: `grounded_only_in_statement`.
|
||
6. **How many of five ambiguous phrasings returned `cannot_determine`?** One of five (only "connected to").
|
||
7. **Did different wording produce different enum categories?** Yes. Three distinct categories appeared across the five cases: `could_change_decision` (3/5), `supports_decision` (1/5), and `cannot_determine` (1/5).
|
||
8. **Which phrases caused the model to strengthen beyond what was supplied?** Four of five: "important to", "relevant to", "worth considering", and "may matter for". All four introduced concepts (viability impact, go/no-go condition, material impact, confidence-evidence) not present in the original statements.
|
||
9. **Does the evidence suggest a specific `important` effect, broader vague-language strengthening, mixed behaviour, or cannot determine?** Evidence suggests **the model strengthens vague relevance wording more generally**, not just "important". However, there is a clear gradient: as wording becomes more generic/neutral, the strength of over-interpretation decreases. "connected to" (the most neutral) preserved `cannot_determine`. "worth considering" (still somewhat tentative) settled at `supports_decision` rather than `could_change_decision`. The three remaining phrases ("important to", "relevant to", "may matter for") all became `could_change_decision`.
|
||
|
||
### Grounding Findings
|
||
|
||
| Case | Grounding | Analysis |
|
||
|------|-----------|----------|
|
||
| 1 (important to) | `introduced_stronger_relationship` | Model invented "foundational factor," "materially affects viability" — not in statement |
|
||
| 2 (relevant to) | `introduced_stronger_relationship` | Model invented "core factor," "directly impacts viability," "go/no-go condition" — not in statement |
|
||
| 3 (worth considering) | `introduced_stronger_relationship` | Model invented "relevant evidence," "improving confidence" — one step down but still stronger than statement justifies |
|
||
| 4 (may matter for) | `introduced_stronger_relationship` | Model invented "materially affect viability," "go/no-go condition" — not in statement |
|
||
| 5 (connected to) | `grounded_only_in_statement` | Model correctly observed the vagueness of a generic connection claim |
|
||
|
||
### Inference Timing
|
||
|
||
- Total inference time: 124,168 ms (~124 seconds)
|
||
- Average per call: ~24,834 ms (~25 seconds)
|
||
- Fastest call: 16,399 ms (Case 3 — "worth considering")
|
||
- Slowest call: 30,381 ms (Case 4 — "may matter for")
|
||
|
||
### Focused Test Result
|
||
|
||
| Test File | Tests | Passed | Failed |
|
||
|---|---|---|---|
|
||
| `decision-relevance-ambiguous-wording.test.js` (Exp 52H) | 42 | 42 | — |
|
||
| `decision-relevance-domain-priors.test.js` (Exp 52G re-run) | 37 | 37 | — |
|
||
| `question-decision-relevance.test.js` (core classifier) | 25 | 25 | — |
|
||
|
||
### Regression Result
|
||
|
||
Experiment 52G re-run on fresh inference: results shifted slightly from primary run (two of four "important to" cases changed from `could_change_decision` to `supports_decision`). Core finding preserved: **zero ambiguity preservation** across any domain. Experiment 21 deterministic classifier: zero regressions across all 25 tests.
|
||
|
||
### Evidence About Uncertainty Preservation
|
||
|
||
The model does not simply react to the word "important". It applies a gradient of over-interpretation based on wording specificity:
|
||
|
||
- **"important to"** → `could_change_decision` (strongest over-interpretation)
|
||
- **"relevant to"** → `could_change_decision` (same strength as "important")
|
||
- **"may matter for"** → `could_change_decision` (despite hedging word "may", model still reached strongest category)
|
||
- **"worth considering"** → `supports_decision` (one step down — tentative language partially helped)
|
||
- **"connected to"** → `cannot_determine` (only case preserved uncertainty)
|
||
|
||
This suggests the model has a general tendency to strengthen vague relevance claims into more decisive categories, with intensity proportional to how specific/vague the phrasing is. "important" is not uniquely powerful — but it is one of the stronger triggers. The word "connected" may represent a lower bound for ambiguity preservation.
|
||
|
||
### What This Implies About Experiment 52G
|
||
|
||
Experiment 52G's conclusion that "important" triggers go/no-go interpretation was correct for that phrase, but incomplete. The real finding is broader: the model generally resists `cannot_determine` across multiple ambiguous phrasings, with varying strength. Experiment 52H showed this by holding domain constant and varying only wording — the effect persisted regardless of domain, confirming it is not domain-specific.
|
||
|
||
### Limitations
|
||
|
||
- Single-run probe with `qwen-claude:latest` on remote host — stability over repeated runs not measured for either experiment.
|
||
- Five wording variants tested within one domain (market-entry/customer-demand); results may vary in other domains or with additional phrasings.
|
||
- Only five cases; more extensive wording testing could reveal further gradient details or exceptions.
|
||
- Remote host latency (~25s/call) limits scope of repeatability testing.
|
||
- Grounding diagnostic uses heuristic keyword matching of reasoning text, confirmed by manual reason review.
|
||
|
||
### Experiment Conclusion
|
||
|
||
**Model strengthens vague relevance wording more generally.** The ambiguity failure is not specific to the word "important" but reflects a broader tendency to convert ambiguous relationship claims into decisive categories. Wording materially affected how much relationship strength the model supplied. Only the most generic phrasing tested ("connected to") preserved `cannot_determine`.
|
||
|
||
**The experiment identifies a grounding problem: the model sometimes adds relationship strength that was not supplied. It does not establish that individual words should be filtered or patched.**
|
||
|
||
### Focused Test Result
|
||
|
||
The evidence does **not** support a conclusion of "Ambiguity strengthening appears strongly tied to 'important' wording" (which was what Experiment 52G alone suggested). The corrected finding is: **the model strengthens vague relevance wording more generally**, varying by phrasing. Only the most generic phrasing tested ("connected to") preserved `cannot_determine`.
|
||
|
||
### Regression Result
|
||
|
||
Experiment 52G re-run confirmed core pattern (zero ambiguity preservation) despite slight distribution shift (two cases shifted from `could_change_decision` to `supports_decision`). The model appeared more consistent about strengthening incomplete meaning than about which stronger category it selected. Deterministic classifier: 25/25 tests passing. No regressions.
|
||
|
||
### Status
|
||
|
||
**Pending Rob's review.** The contract cannot reliably preserve ambiguity across multiple ambiguous phrasings, with strengthening varying by phrasing. Both experiments (52G and 52H) used the same host (`http://192.168.1.111:11434`) and model (`qwen-claude:latest`). No production code has been changed.
|
||
|
||
### Production Unchanged
|
||
|
||
- `lib/graph/question-decision-relevance.js`: 0 lines changed
|
||
- No production files modified
|
||
- Working tree clean before commit
|
||
|
||
### Files Created
|
||
|
||
- `tests/graph/decision-relevance-ambiguous-wording.test.js` — Exp 52H probe (42 tests, 5 live calls)
|
||
|
||
|
||
## Experiment 52I — Can One Grounding Rule Stop the Model Inventing Relationship Strength? (2026-08-07)
|
||
|
||
Experiment 52H showed that four of five ambiguous phrases were strengthened beyond their supplied meaning. Only "connected to" preserved `cannot_determine`. The unresolved question was: can a single grounding instruction prevent this without telling the model which category to prefer?
|
||
|
||
### Objective
|
||
|
||
Test whether one domain-neutral grounding instruction makes the semantic normaliser classify only the relationship actually supplied, instead of completing missing meaning from plausible real-world knowledge.
|
||
|
||
> **Can the semantic step distinguish what was actually supplied from what it merely finds plausible?**
|
||
|
||
### Configuration
|
||
|
||
| Setting | Value |
|
||
|---|---|
|
||
| Ollama host | `http://192.168.1.111:11434` (from `.env.local`) |
|
||
| Model | `qwen-claude:latest` (from `.env.local`) |
|
||
| Normalisation instruction | Experiment 52H instruction + one grounding rule (exact change documented below) |
|
||
| Input per case | `{"relationship": "<fixed relationship statement>"}` only. No decision target, no question, no domain examples. |
|
||
| Domain for ambiguous cases | Market entry / customer demand (same as Exp 52H for direct comparison) |
|
||
| Domain for clear controls | Community event weather / outdoor venue (deliberately different to test grounding independence) |
|
||
|
||
### Category Definitions Used (unchanged from production contract)
|
||
|
||
| Category | Definition |
|
||
|---|---|
|
||
| `could_change_decision` | Answering could reasonably reverse the proposed action — it is a go/no-go condition or materially affects viability. |
|
||
| `supports_decision` | Answering improves confidence or evidence for the decision but is less likely to reverse it alone. |
|
||
| `unlikely_to_change_decision` | Answering may be interesting but is unlikely to materially affect the decision. |
|
||
| `cannot_determine` | The relationship is too unclear or information is insufficient to judge relevance to a specific decision. |
|
||
|
||
### The One Allowed Instruction Change
|
||
|
||
**Previous instruction (identical to Experiment 52H):**
|
||
|
||
```
|
||
You are given a short statement describing how an unanswered question relates to a decision. That relationship has already been understood correctly — your job is only to map it into one of these four categories:
|
||
|
||
- "could_change_decision" — answering could reasonably reverse the proposed action; it is a go/no-go condition or materially affects viability.
|
||
- "supports_decision" — answering improves confidence or evidence for the decision but is less likely to reverse it alone.
|
||
- "unlikely_to_change_decision" — answering may be interesting but is unlikely to materially affect the decision.
|
||
- "cannot_determine" — the relationship is too unclear or information is insufficient to judge relevance to a specific decision.
|
||
|
||
Do not reinterpret the original situation — you have not been given it. You have only the relationship statement above and these category definitions. Choose the category that best matches the relationship statement.
|
||
|
||
Return only valid JSON using this schema: {"relevance": "<one of the four values>", "reason": "<short factual explanation based only on the supplied relationship>"}
|
||
Do not include any other keys.
|
||
```
|
||
|
||
**Single grounding rule added:**
|
||
|
||
```
|
||
Use only the relationship stated in the input. Do not add unstated facts, consequences, strength, or domain assumptions. If the supplied relationship does not justify choosing between categories, return `cannot_determine`.
|
||
```
|
||
|
||
**Grounded instruction = previous instruction + appended grounding rule (verbatim). No examples added. No domain-specific hints. No trigger words mentioned.**
|
||
|
||
### Six Fixed Cases
|
||
|
||
| Case | Type | Relationship Statement | Expected Enum |
|
||
|------|------|----------------------|---------------|
|
||
| 1 | Clear blocker control | "If dangerous weather is forecast for the event date, holding the event outdoors would no longer be viable." | `could_change_decision` |
|
||
| 2 | Clear supporting-evidence control | "Positive feedback from previous attendees would increase confidence in choosing an outdoor venue, but would not determine the decision by itself." | `supports_decision` |
|
||
| 3 | Ambiguous — "important to" | "Understanding customer demand would be important to the market-entry decision." | `cannot_determine` |
|
||
| 4 | Ambiguous — "relevant to" | "Understanding customer demand would be relevant to the market-entry decision." | `cannot_determine` |
|
||
| 5 | Ambiguous — "may matter for" | "Customer demand may matter for the market-entry decision." | `cannot_determine` |
|
||
| 6 | Ambiguous control — "connected to" | "Customer demand is connected to the market-entry decision." | `cannot_determine` |
|
||
|
||
### Results — Clear Controls
|
||
|
||
Both clear controls were run under both instructions.
|
||
|
||
| Case | Label | Previous Result | Grounded Result | Match? (grounded) | Grounding |
|
||
|------|-------|-----------------|-----------------|--------------------|-----------|
|
||
| 1 | Clear blocker control | `could_change_decision` | `could_change_decision` | ✅ match | grounded_in_supplied_relationship |
|
||
| 2 | Clear supporting-evidence control | `supports_decision` | `supports_decision` | ✅ match | grounded_in_supplied_relationship |
|
||
|
||
Both clear controls retained their expected categories under the grounded instruction. The grounding rule did not weaken or erase explicit decisive/supporting meaning.
|
||
|
||
### Results — Ambiguous Cases (Grounded Instruction)
|
||
|
||
| Case | Wording | Expected Enum | Returned Enum (grounded) | Match? | Grounding Diagnostic | Reason Summary |
|
||
|------|---------|---------------|--------------------------|--------|---------------------|----------------|
|
||
| 3 | "important to" | `cannot_determine` | `could_change_decision` | ❌ mismatch | introduced_unstated_relationship_strength | Model read "important" as materially affecting viability / critical go/no-go condition |
|
||
| 4 | "relevant to" | `cannot_determine` | `cannot_determine` | ✅ match | grounded_in_supplied_relationship | Model noted general relevance without specifying direction, strength, or material impact |
|
||
| 5 | "may matter for" | `cannot_determine` | `cannot_determine` | ✅ match | grounded_in_supplied_relationship | Model correctly returned `cannot_determine`. Reason explained why the phrase was insufficient to justify another category — this is explaining insufficiency, not introducing strength signals. |
|
||
| 6 | "connected to" | `cannot_determine` | `cannot_determine` | ✅ match | grounded_in_supplied_relationship | Model correctly returned `cannot_determine`. Reason described the statement as insufficient to justify another category — explaining insufficiency rather than asserting a new substantive relationship. |
|
||
|
||
**Cannot_determine count under grounding: 3/4**
|
||
**Cases that still strengthened beyond supplied meaning: 1/4 (case 3 — "important to")**
|
||
|
||
### Grounding Diagnostic Detail
|
||
|
||
Under the grounded instruction, the model's reasoning text was manually assessed:
|
||
|
||
| Case | Grounding Result | Analysis |
|
||
|------|-----------------|----------|
|
||
| 3 ("important to") | `introduced_unstated_relationship_strength` | Model invented "materially affects viability" and "critical go/no-go condition" — not in statement. Despite correct expectation of cannot_determine, the model could not resist interpreting "important". |
|
||
| 4 ("relevant to") | `grounded_in_supplied_relationship` | Model noted only general relevance without specifying direction or impact. Stayed within supplied meaning. |
|
||
| 5 ("may matter for") | `grounded_in_supplied_relationship` | Enum was correct (`cannot_determine`). Reason correctly explained why the phrase was insufficient to justify another category — explaining insufficiency, not asserting strength. |
|
||
| 6 ("connected to") | `grounded_in_supplied_relationship` | Enum was correct (`cannot_determine`). Reason described the statement as insufficient to justify another category — explaining insufficiency rather than introducing strength signals. |
|
||
|
||
**Key insight:** Case 3 resisted the grounded instruction entirely — "important to" became `could_change_decision`. Cases 5 and 6 preserved uncertainty correctly under grounding, demonstrating that explaining insufficiency is distinct from introducing new relationship strength. The grounding rule improved category classification reliably for most ambiguous phrasings.
|
||
|
||
### Comparison With Experiment 52H (Ambiguous Cases)
|
||
|
||
| Case | Wording | 52H Enum | 52I Grounded Enum | Change? | 52H Grounding | 52I Grounding |
|
||
|------|---------|----------|-------------------|---------|---------------|---------------|
|
||
| 3 | "important to" | `could_change_decision` | `could_change_decision` | unchanged | introduced_unstated_relationship_strength | introduced_unstated_relationship_strength |
|
||
| 4 | "relevant to" | `could_change_decision` | `cannot_determine` | ✅ improved | introduced_unstated_relationship_strength | grounded_in_supplied_relationship |
|
||
| 5 | "may matter for" | `could_change_decision` | `cannot_determine` | ✅ improved | introduced_unstated_relationship_strength | grounded_in_supplied_relationship (enum correct, reasoning explained insufficiency rather than asserting strength) |
|
||
| 6 | "connected to" | `cannot_determine` | `cannot_determine` | unchanged | grounded_only_in_statement | grounded_in_supplied_relationship (enum correct, reasoning described insufficiency) |
|
||
|
||
**Ambiguity preservation improved:** Cases 4 and 5 shifted from `could_change_decision` → `cannot_determine`. Case 3 remained unchanged. Case 6 remained the same (both preserved ambiguity in enum).
|
||
|
||
### Did Grounding Improve Ambiguity Preservation?
|
||
|
||
**Yes.** Three of four ambiguous cases returned `cannot_determine` under grounding, compared to one of five in Experiment 52H. Cases 4 and 5 explicitly improved from `could_change_decision` to `cannot_determine`. Case 6 preserved ambiguity in both experiments.
|
||
|
||
### Did Grounding Harm Clear Classifications?
|
||
|
||
**No.** Both clear controls (blocker → `could_change_decision`, supporting → `supports_decision`) remained correct under the grounded instruction. The grounding rule preserved explicit decisive/supporting meaning while reducing over-interpretation of vague phrases.
|
||
|
||
### Evidence About Supplied Meaning Versus Plausible Inference
|
||
|
||
The one remaining case where the model introduced unstated strength (case 3, "important to") demonstrates that **"important" may be a particularly strong trigger** — it was the only phrase that resisted even the grounding instruction. This is consistent with Experiment 52G's earlier finding but does not justify building a keyword-filter system around it; instead, it suggests:
|
||
|
||
- The grounding rule improves ambiguity preservation without harming clear classifications
|
||
- A single category-level safeguard can move most vague phrasing toward `cannot_determine`
|
||
- But the model still struggles to separate *what was stated* from *what seems plausible* for strong trigger words
|
||
|
||
### What This Suggests Is the Primary Defect
|
||
|
||
**The tested category contract remains usable for explicit relationships. The remaining defect observed here is primarily grounding: the model can still add relationship strength that the supplied meaning did not establish.**
|
||
|
||
Evidence from this experiment:
|
||
|
||
1. Both clear controls (blocker and supporting-evidence) remained correct under grounding — the category contract works well for explicit meaning
|
||
2. "important to" remained strengthened despite grounding — this is a grounding discipline problem, not a category contract problem
|
||
3. Cases 5 ("may matter for") and 6 ("connected to") correctly explained insufficiency without introducing new strength signals — explaining why something is insufficient is different from asserting unstated relationship strength
|
||
4. Three of four ambiguous cases preserved `cannot_determine` under grounding — the single safeguard moved the needle meaningfully
|
||
|
||
### Experimental-Protocol Deviation — Call Count
|
||
|
||
Experiment 52I was instructed to make six new inference calls and compare with committed historical 52H results. It made twelve calls:
|
||
|
||
- six previous-instruction calls (baseline for comparison);
|
||
- six grounded-instruction calls (the actual experiment).
|
||
|
||
This is an experimental-protocol deviation. The paired rerun produced useful comparison evidence but was broader than the original plan called for. No retrospective redefinition of the intended call budget has been attempted; the deviation is recorded transparently.
|
||
|
||
### Inference Timing
|
||
|
||
- Total inference time: 211,008 ms (~211 seconds)
|
||
- Average per call: ~17,584 ms (~17.6 seconds) per call
|
||
- Fastest call: 8,617 ms (previous instruction, case 6 — "connected to")
|
||
- Slowest call: 30,674 ms (grounded instruction, case 3 — "important to")
|
||
- Exactly 12 live inference calls (6 under previous instruction, 6 under grounded instruction)
|
||
|
||
### Limitations
|
||
|
||
- Single-run probe with `qwen-claude:latest` on remote host — stability over repeated runs not measured.
|
||
- Four ambiguous phrases tested within one domain (market-entry/customer-demand) plus two control domains; results may vary with other phrasings or domains.
|
||
- Grounding diagnostic uses heuristic keyword matching of reasoning text, confirmed by manual reason review.
|
||
- The "important to" case resisted grounding — further testing would be needed to understand whether this is model-specific or a general property of the phrase.
|
||
- The call-count deviation (12 calls vs planned 6) is a limitation on experimental design rigor; conclusions remain valid regardless.
|
||
|
||
### Experiment Conclusion
|
||
|
||
A single grounding rule materially improved uncertainty preservation without harming either clear control. Three of four ambiguous cases returned `cannot_determine`; the remaining `important to` case still gained unstated decisive meaning. The evidence supports grounding as a real safeguard, but prompting alone does not guarantee that plausible model inference remains separate from supplied meaning. Status pending Rob's review.
|
||
|
||
### Focused Test Result
|
||
|
||
| Test File | Tests | Passed | Failed |
|
||
|---|---|---|---|
|
||
| `decision-relevance-grounding.test.js` (Exp 52I) | 49 | 48 | 1 (case 3 "important to" — expected cannot_determine, got could_change_decision under grounded instruction) |
|
||
| `question-decision-relevance.test.js` (core classifier) | 25 | 25 | — |
|
||
|
||
### Regression Result
|
||
|
||
Experiment 21 deterministic classifier: zero regressions across all 25 tests. No production code changed. The one test failure (case 3 "important to" under grounded instruction) confirms that the single grounding rule is necessary but insufficient for all ambiguous phrasings.
|
||
|
||
### Status
|
||
|
||
**Pending Rob's review.** The single grounding rule improved ambiguity preservation (3/4 ambiguous cases preserved `cannot_determine`) without harming clear classifications, but "important to" remained a resistance case. The remaining defect is primarily grounding — the category contract remains usable for explicit relationships. Same host (`http://192.168.1.111:11434`) and model (`qwen-claude:latest`) retained; no production behaviour changed. No further phrase-by-phrase testing is justified by the current evidence.
|
||
|
||
### Production Unchanged
|
||
|
||
- `lib/graph/question-decision-relevance.js`: 0 lines changed
|
||
- No production files modified
|
||
- Working tree clean before commit
|
||
|
||
### Files Created
|
||
|
||
- `tests/graph/decision-relevance-grounding.test.js` — Exp 52I probe (49 tests, 12 live calls)
|
||
|
||
|
||
|
||
## Experiment 54A — Audit Existing Graph Provenance Only (2026-08-07)
|
||
|
||
Experiment 53 showed that the semantic model can keep supplied meaning and possible inference separate in its output. The active graph compatibility question remained unknown. Experiment 54A was an inspection-only experiment to determine whether the current validated SituationGraph distinguishes information supplied by the user or evidence from information inferred by the model.
|
||
|
||
### Hypothesis
|
||
|
||
The current graph may distinguish known/provisional and supported/unsupported without actually recording where information came from. If true, current graph state can represent epistemic status but not reliably recover supplied-versus-inferred provenance.
|
||
|
||
### Files Inspected
|
||
|
||
- `docs/current-handoff.md`
|
||
- `lib/graph/schema.js` — SituationGraph and node schema definitions
|
||
- `lib/graph/builder.js` — production initial graph builder (how nodes are populated from reconstruction)
|
||
- `lib/graph/update-proposal.js` — LLM output parsing for graph updates
|
||
- `lib/reconstruction/schema.js` — evidenceRecordSchema, reconstructionV2Schema
|
||
|
||
### Graph Vocabulary Relevant to Provenance
|
||
|
||
**Existing relevant node kinds:**
|
||
- `observation`, `reported_claim`, `metric`, `state`, `transition`, `relationship`, `assumption`, `unknown`, `conclusion`
|
||
|
||
**Existing relevant status fields:**
|
||
- `known`, `unknown`, `provisional`, `supported`, `weakened`, `contradicted`, `resolved`
|
||
|
||
**Existing confidence fields:**
|
||
- `low`, `medium`, `high`
|
||
|
||
**Existing evidence / relationship fields:**
|
||
- `evidenceIds`: array of strings (graph reference IDs from reconstruction)
|
||
- `dependsOn`: array of node IDs
|
||
- `affects`: array of node IDs
|
||
- `parentId`: nullable string
|
||
- `childIds`: array of node IDs
|
||
- Edge types: `supports`, `weakens`, `contradicts`, `depends_on`, `causes`, `may_cause`, `measures`, `compares_with`, `updates`, `other`
|
||
|
||
### Explicit Supplied-Information Provenance Exists: No
|
||
|
||
No field or combination of fields in the SituationNode schema has documented or implemented meaning that is "this content was supplied by the user or evidence source." The `kind` field distinguishes semantic categories (observation vs assumption vs unknown), not provenance. A node with `kind=assumption` describes what kind of claim it is, not who produced it.
|
||
|
||
### Explicit Inferred-Information Provenance Exists: No
|
||
|
||
No field or combination has documented or implemented meaning that is "this content was inferred or proposed by the model and is not established evidence." The LLM-inferred nodes flow through `proposal.addedNodes` into the graph with kinds determined by the LLM — but those kinds are semantic labels, not provenance markers.
|
||
|
||
### Status Versus Provenance Finding
|
||
|
||
Fields like `provisional`, `supported`, `assumption` (as a kind), and `confidence` describe **epistemic status only** — they classify how confident or well-supported a claim is. They do not record where the information originated. A node with `kind=unknown, status=unknown, confidence=low` could have come from user input, model inference, or evidence extraction.
|
||
|
||
### Are evidenceIds Provenance or Graph References
|
||
|
||
**Graph references.** In `buildInitialGraph`, `evidenceIds` are populated from `obs.id` — IDs that originate from the LLM's reconstruction output (`reconstruction.observedStates[].id`). These are internal identifiers for model-generated evidence records, not user-supplied source identifiers. The same applies during graph updates: node relationships use string IDs that are graph-internal references.
|
||
|
||
### Are Inferred Nodes Explicitly Marked as Model-Generated
|
||
|
||
**No.** Neither `builder.js` (initial build) nor the update-proposal path marks inferred nodes with any model-generated flag. Node kinds in the update path are set by the LLM's JSON output — there is no explicit "this was model-inferred" marker.
|
||
|
||
### Recoverability Result: not_recoverable
|
||
|
||
A later consumer receiving only the validated graph (with no conversation history or LLM response) cannot determine which statements came from user/evidence and which were generated as model inference. All nodes produced by different paths (initial build, emergent reasoning, decomposition children) have identical schema shape. The evidenceIds field contains IDs referencing model-generated reconstruction records, not external source identifiers.
|
||
|
||
### Production Population Finding
|
||
|
||
- No relevant source/provenance fields are populated in production graph-building code
|
||
- Node kinds (`observation`, `assumption`, `unknown`, etc.) are used for semantic typing, not provenance
|
||
- Evidence IDs are model-generated internal references (not user-supplied identifiers)
|
||
- No inferred nodes carry any explicit model-generated marker
|
||
|
||
### Experiment Conclusion
|
||
|
||
The existing SituationGraph does not preserve supplied-versus-inferred provenance. It represents epistemic status (how confident or well-supported information is) but has no mechanism to record where information originated. This confirms the hypothesis from Experiment 53's open question: while semantic output can separate supplied meaning from inference, the graph layer cannot recover that separation because it lacks provenance tracking fields entirely.
|
||
|
||
### Limitations
|
||
|
||
- Inspection-based; no live model run was performed
|
||
- Only source files directly relevant to node schema and construction were examined
|
||
- The non-strict Zod schema allows extra fields but none are used for provenance in production code
|
||
- Does not address whether a fix is needed — only whether the gap exists
|
||
|
||
### Status
|
||
|
||
**Pending Rob's review.** The audit confirms a provenance gap. No production code was changed. Working tree clean before commit.
|
||
|
||
### Production Unchanged
|
||
|
||
- `lib/graph/schema.js`: 0 lines changed
|
||
- `lib/graph/builder.js`: 0 lines changed
|
||
- `lib/graph/apply-proposal.js`: 0 lines changed
|
||
- `lib/graph/update-proposal.js`: 0 lines changed
|
||
- No production files modified
|
||
- Working tree clean before commit
|
||
|
||
### Tests / Validation Run
|
||
|
||
No test run required for the inspection result. Source inspection alone is sufficient — the schema definition in `lib/graph/schema.js` is a static contract, and no runtime execution is needed to confirm the absence of provenance fields.
|
||
|
||
### Documentation Updated
|
||
|
||
- `docs/current-handoff.md` — handoff line 127 and Return-to-Work Note updated
|
||
- `docs/design-evolution-log.md` — Experiment 54A section appended
|
||
|
||
## Experiment 54B — Trace Provenance Loss Through Graph Pipeline (2026-08-07)
|
||
|
||
Experiment 54A proved the provenance gap exists in the graph. This experiment traced both flows to find exactly where upstream provenance is lost and whether it is recoverable at any point.
|
||
|
||
### Method
|
||
|
||
Inspected file-by-file through the complete data flow of both paths, tracking the evidenceType field from its creation in the reconstruction layer through to the final graph state.
|
||
|
||
**Source-traced paths:**
|
||
- Flow A (initial): analysis.js → buildInitialGraph() → SituationGraph.nodes
|
||
- Flow B (update): orchestrator.updateCase() → prompt → parseGraphUpdateProposal() → applyValidatedProposal() → SituationGraph.nodes/edges
|
||
|
||
### Trace Table
|
||
|
||
| Step | File | Data Present? | Provenance Status |
|
||
|------|------|---------------|-------------------|
|
||
| LLM output raw | analyseScenario() lib/analysis.js:28-134 | reconstruction + evidence with evidenceType enum | PRESENT (supplied vs inferred explicit in evidence records) |
|
||
| Validation v0.2 schema | lib/reconstruction/schema.js:129-162 | EvidenceRecordSchema includes evidenceType: ["direct_observation","reported_statement","interpretation","assumption","inferred_relationship"] | PRESENT — enum encodes the distinction upstream |
|
||
| buildSuccessResultV2 return | lib/analysis.js:173-184 | {reconstruction, evidence} returned | PRESENT in both paths |
|
||
| buildInitialGraph receives data | lib/graph/builder.js:17-18 | reconstruction + evidence map built (line 62) | UPSTREAM AVAILABLE |
|
||
| Nodes created with kind/status | lib/graph/builder.js:40-54 (ensureNode), lines 75-209 | Nodes get kind/status/confidence from semantic mapping of reconstruction fields (observedStates→observation, actors→observation, systemsOrObjects→metric, etc.) | LOST — no provenance field on nodes |
|
||
| EvidenceMap built but unused | lib/graph/builder.js:61-64 | evidenceMap populated with evidence records (ev.id → ev) | DEAD CODE — never queried after construction |
|
||
| addEvidenceToNode called | lib/graph/builder.js:66-70, 100 | Only obs.id pushed to node.evidenceIds as string reference | LOST — ID only, no type metadata transferred |
|
||
| buildMinimalGraph fallback | lib/graph/builder.js:257-280 | No evidence at all; nodes created from scenario text directly | NO UPSTREAM PROVENANCE AVAILABLE |
|
||
| Orchestrator startCase → makeGraph | lib/graph/orchestrator.js:383 | centralStatement = scenario string preserved at graph root | PARTIAL — only the raw scenario survives as centralStatement |
|
||
| updateCase receives answer | lib/graph/orchestrator.js:569-758+ | Answer parameter enters orchestrator, included in prompt to LLM | SUPPLIED ANSWER TEXT present in prompt |
|
||
| LLM proposes graph updates | based on prompt content including answer | No separation of user-supplied vs model-inferred in proposal | LOST at prompt construction boundary |
|
||
| parseGraphUpdateProposal output | lib/graph/update-proposal.js:100-157 | GraphUpdateSchema with addedNodes using situationNodeSchema | NO provenance on added nodes or edges |
|
||
| applyValidatedProposal receives answer | lib/graph/apply-proposal.js:2723 | answer parameter passed through (line 2723), used in deriveReasoningStateOverride (line 2875) | PRESENT but NOT used for provenance — only affects reasoning state derivation |
|
||
| applyGraphUpdate applies changes | lib/graph/apply-proposal.js, line 2882+ | Graph modified; no new provenance fields added | LOST — nodes/edges created without source metadata |
|
||
|
||
### Provenance Loss Summary
|
||
|
||
**Flow A (initial graph):**
|
||
- Upstream of graph: evidenceType enum explicitly distinguishes supplied from inferred in `evidenceRecordSchema`
|
||
- First loss point: `buildInitialGraph()` at `lib/graph/builder.js` — nodes are created with semantic kind/status but NO provenance field. The evidenceMap is built (line 62) but never used. Only `obs.id` is added to node.evidenceIds as a bare string reference without type information.
|
||
- Recovery path: NOT from graph state alone. Would require the upstream reconstruction + evidence array that was consumed during initial build.
|
||
|
||
**Flow B (update):**
|
||
- User answer enters as `answer` parameter in orchestrator, flows through LLM prompt, becomes part of a proposal with no provenance metadata on nodes or edges
|
||
- No separation between "user supplied this text" and "model proposed these graph changes" anywhere in the update pipeline
|
||
- The `answer` reaches `applyValidatedProposal` (line 2723) and is used for reasoning state derivation (line 2875), but no provenance field is added to nodes/edges
|
||
|
||
### EvidenceIds Clarification Conclusion
|
||
|
||
The node-level evidenceIds does NOT represent "the list of nodes from which this information was derived." Instead:
|
||
|
||
- `evidenceIds` is a **reference list** — each string in the array is an ID that references a specific record in the upstream evidence array
|
||
- The distinction between supplied and inferred information lives in the **evidenceType field of those evidence records**, not on the node itself
|
||
- A node's kind/status fields encode epistemic classification (what role does this node play and how confident are we), NOT provenance (where did this information come from)
|
||
- To recover whether a piece of information was user-supplied or model-inferred, you must look up each evidenceId in the original evidence array and check its evidenceType field — which means **provenance recovery depends on access to the upstream evidence data, not on the graph state alone**
|
||
|
||
This is actually useful: it clarifies that provenance IS recoverable from validated reconstruction output (the reconstructed data includes an evidence array where each record has evidenceType), but it is NOT recoverable from graph state alone. The `evidenceIds` array is a bridge to upstream provenance, not provenance itself.
|
||
|
||
### Key Findings
|
||
|
||
1. **Supplied-vs-inferred provenance EXISTS upstream.** `evidenceRecordSchema` (lib/reconstruction/schema.js line 116-121) has an explicit enum: direct_observation, reported_statement (supplied categories) vs interpretation, assumption, inferred_relationship (inferred categories). This is the most important finding.
|
||
|
||
2. **First provenance loss point is `buildInitialGraph()`.** In lib/graph/builder.js lines 61-70, the evidenceMap is built but dead-coded — never queried. Only ID strings are added to node.evidenceIds without type metadata.
|
||
|
||
3. **Update flow has no provenance preservation.** The user's answer arrives as a parameter but becomes embedded in an LLM prompt with no traceability. No node receives source metadata during updates.
|
||
|
||
4. **Provenance recovery requires upstream data, not graph state.** Since the SituationGraph schema has no provenance fields and nodes only carry ID references to evidence records, any provenance determination must reference the original reconstruction or update evidence array — it cannot be derived from the graph alone.
|
||
|
||
### Experiment Conclusion
|
||
|
||
The supplied-versus-inferred distinction is fully preserved in the upstream reconstruction/update pipeline output (the validated evidence arrays carry explicit evidenceType values for each record). However, this distinction is never encoded into the SituationGraph nodes during either initial build or update application. The evidenceIds field on nodes provides indirect access to provenance via ID references, but only if the original evidence data remains available downstream of the graph.
|
||
|
||
The core insight: provenance is not lost from the pipeline — it is preserved in the reconstruction output that feeds the builder. It IS lost when the builder converts that output into a SituationGraph because the node schema has no field to receive it. This means any future implementation would need to add a provenance-bearing field to the node schema and propagate evidenceType through buildInitialGraph and applyValidatedProposal — though the specific implementation approach (field name, placement, propagation mechanism) remains undecided.
|
||
|
||
### Limitations
|
||
|
||
- Inspection-based; no live model run required
|
||
- Traced production paths only (builder.js, apply-proposal.js, update-proposal.js)
|
||
- Did not examine LLM prompt templates to determine if they preserve answer-supplied vs inference distinction in output formatting
|
||
- The analysis assumes evidence records remain accessible after graph construction — downstream usage patterns were not audited
|
||
|
||
### Status
|
||
|
||
**Pending Rob's review.** Tracing complete. No production code changed. Working tree clean before commit.
|
||
|
||
### Production Unchanged
|
||
|
||
- `lib/reconstruction/schema.js`: 0 lines changed
|
||
- `lib/graph/builder.js`: 0 lines changed
|
||
- `lib/graph/apply-proposal.js`: 0 lines changed
|
||
- `lib/graph/orchestrator.js`: 0 lines changed
|
||
- No production files modified
|
||
- Working tree clean before commit
|
||
|
||
### Tests / Validation Run
|
||
|
||
No test run required — this is a source-trace audit confirming data flow paths, not a behavioral test.
|
||
|
||
### Documentation Updated
|
||
|
||
- `docs/current-handoff.md` — Return-to-Work Note updated with 54B findings
|
||
- `docs/design-evolution-log.md` — Experiment 54B section appended
|
||
|
||
|
||
### Report 54B — Trace Provenance Loss Through Graph Pipeline
|
||
|
||
#### Experiment Purpose
|
||
|
||
Trace where the supplied-versus-inferred provenance distinction is lost in both the initial graph build and update flows, determine whether it is recoverable at any point in the pipeline, and document what future work must do to address the gap.
|
||
|
||
#### Method
|
||
|
||
File-by-file source inspection of the complete data flow for both paths: Flow A (analyseScenario → buildInitialGraph → SituationGraph) and Flow B (updateCase → LLM prompt → parseGraphUpdateProposal → applyValidatedProposal).
|
||
|
||
#### Provenance Existence Upstream
|
||
|
||
Yes. The evidenceRecordSchema in lib/reconstruction/schema.js lines 116-121 defines an explicit enum: direct_observation, reported_statement, interpretation, assumption, inferred_relationship. Supplied categories (direct_observation, reported_statement) are separate from inferred categories (interpretation, assumption, inferred_relationship). This distinction is preserved in the validated reconstruction output returned by analyseScenario and consumed by buildInitialGraph.
|
||
|
||
#### First Provenance Loss Point
|
||
|
||
- Flow A (initial graph): lib/graph/builder.js lines 61-70. The evidenceMap is built at line 62 from all evidence records but never queried after construction. Only the raw ID string (e.g., obs.id) is added to node.evidenceIds via addEvidenceToNode — no type metadata or provenance classification is transferred to the node schema, which has no provenance field defined in lib/graph/schema.js lines 55-70.
|
||
|
||
- Flow B (update): lib/graph/orchestrator.js where updateCase passes the user answer into an LLM prompt without separating "user-supplied" from "model-inferred" content, and lib/graph/update-proposal.js where graphUpdateSchema builds addedNodes using situationNodeSchema which has no provenance field. The answer parameter reaches applyValidatedProposal (lib/graph/apply-proposal.js line 2723) and is used in deriveReasoningStateOverride (line 2875), but no provenance metadata is attached to nodes or edges during the update application.
|
||
|
||
#### EvidenceIds Clarification Conclusion
|
||
|
||
The node-level evidenceIds field does not represent "the list of nodes from which this information was derived." Instead, each string in evidenceIds is a reference ID that points to a specific record in the upstream evidence array. The supplied-versus-inferred distinction lives on those upstream records (their evidenceType enum field), not on the node itself. To recover whether a piece of information was user-supplied or model-inferred requires accessing the original reconstruction or update evidence array and checking each referenced record's evidenceType — provenance recovery therefore depends on access to upstream data, not on the graph state alone.
|
||
|
||
#### Update Flow Answer Handling
|
||
|
||
The user answer enters orchestrator.updateCase() as a parameter, gets embedded in an LLM prompt without source attribution markers, and the resulting proposal carries no provenance metadata onto nodes or edges. The answer survives as a raw string through to applyValidatedProposal (lib/graph/apply-proposal.js line 2723) where it influences reasoning state derivation (line 2875), but no node receives any indication that it was derived from user-supplied content versus model inference.
|
||
|
||
#### Provenance Recovery Feasibility
|
||
|
||
From graph alone: No — the SituationGraph schema has no provenance fields on nodes or edges, and evidenceIds only carries ID references without type metadata. From upstream data: Yes — the validated reconstruction output (returned by analyseScenario) includes a complete evidence array where each record has an explicit evidenceType enum field distinguishing supplied from inferred information.
|
||
|
||
#### Future Work Consideration (Beyond 54B Scope)
|
||
|
||
Any future fix would need to embed provenance in graph state rather than keeping it only in external upstream data — potentially via a node-level provenance field and evidenceType propagation through buildInitialGraph and applyValidatedProposal, while retaining the existing evidenceIds reference system as a cross-reference layer. The specific implementation approach remains undecided. 54C examines whether update provenance is deterministically knowable at the application boundary before any fix is designed.
|
||
|
||
### Status
|
||
**Committed.** Report appended to design log. Branch: `feature/user-workspace-ux-v0.7`. Working tree clean before commit.
|
||
|
||
|
||
## Experiment 54C — Is Update Provenance Deterministically Knowable Before Graph Application? (2026-08-07)
|
||
|
||
### Objective
|
||
|
||
Determine whether production code can distinguish user-supplied material from model-proposed additions at the boundary before graph mutation occurs. This is source-trace only; no code changed, no tests run, no solution designed.
|
||
|
||
### Hypothesis
|
||
|
||
The active update pipeline holds the raw user answer and the validated model proposal as distinct inputs immediately before graph mutation. If so, origin may be deterministically knowable at that boundary even though the current graph does not store it.
|
||
|
||
### Files Actually Inspected
|
||
|
||
- `lib/graph/orchestrator.js` — lines 569–768 (updateCase / updateCaseWithDependencies)
|
||
- `lib/graph/apply-proposal.js` — lines 2719–2912 (applyValidatedProposal)
|
||
- `lib/graph/update-proposal.js` — lines 1–157 (parseGraphUpdateProposal, graphUpdateSchema field list)
|
||
- `lib/graph/schema.js` — lines 55–98 (situationNodeSchema, situationEdgeSchema), lines 155–163 (graphUpdateSchema), lines 174–179 (updateCaseRequestSchema)
|
||
- `docs/design-evolution-log.md` — Experiment 54B section (lines 5080–5178)
|
||
- `docs/current-handoff.md` — lines 129–131 (Return-to-Work Note)
|
||
|
||
Production files actually inspected: `lib/graph/orchestrator.js`, `lib/graph/apply-proposal.js`, `lib/graph/update-proposal.js`, `lib/graph/schema.js`.
|
||
Reconstruction files NOT inspected: `lib/reconstruction/schema.js`, `lib/analysis.js`, `lib/graph/builder.js` (per budget constraints).
|
||
|
||
### Update Flow Trace
|
||
|
||
| Stage | What contains the user answer? | What contains model proposals? | Are they separate? | Origin deterministically knowable? |
|
||
|-------|-------------------------------|--------------------------------|--------------------|-----------------------------------|
|
||
| HTTP request body → updateCaseWithDependencies (orchestrator.js:594) | `answer` from `parsedRequest.data.answer` (schema: z.string().min(1).max(5000)) | — | Yes | N/A — no proposal yet |
|
||
| LLM prompt construction (orchestrator.js:633-638) | `answer` embedded in prompt text as context | Model generates response with additions/proposals | Separated by mechanism: answer is context, model output is the new data | Only by knowing that `answer` was supplied and `rawResponse` came from the LLM. No metadata markers separate user text from model inference within the proposal itself. |
|
||
| parseGraphUpdateProposal (update-proposal.js:100-157) | Not present in parsed output — stripped during JSON parsing | `parsedProposal.proposal` (graphUpdateSchema: addedNodes, updatedNodes, addedEdges, etc.) | N/A — user answer is gone from the parsed proposal object | Merged_or_lost — the raw user answer is no longer part of the proposal object |
|
||
| Orchestrator call to applyValidatedProposal (orchestrator.js:683-688) | `answer` passed as separate function argument | `proposal: parsedProposal.proposal` passed as separate function argument | **explicitly_separate** — two distinct named parameters in one function call | **implicitly_distinguishable** — parameter names distinguish them, but the proposal object itself contains no metadata labeling which nodes/edges came from user input vs model inference |
|
||
| Inside applyValidatedProposal (apply-proposal.js:2719-2880) | `answer` used only in `deriveReasoningStateOverride` (line 2875-2878). No provenance metadata derived from it. | `proposal` (validated against graphUpdateSchema, reconciled via reconcileResolutionSemantics) — then snapshot at line 2869 | Still **explicitly_separate** within the function scope | The two inputs are separate variables, but the proposal object carries no origin labels on its nodes/edges |
|
||
| applyGraphUpdate calls (apply-proposal.js:2882, 2912) | `answer` not passed to applyGraphUpdate | proposalSnapshot applied to graphSnapshot. Nodes created without source metadata. | N/A — applyGraphUpdate receives only the merged snapshot | **merged_or_lost** — origin information is not transmitted to the mutation function |
|
||
| Resulting SituationGraph (after line 2912) | No record of which nodes/edges came from user | All new nodes/edges carry no provenance field | N/A | The graph stores only structural data; origin is unrecoverable from graph state alone |
|
||
|
||
### Answers to Required Questions
|
||
|
||
1. **Is the raw user answer still available immediately before proposal application?**
|
||
Yes. In orchestrator.js line 683-688, `answer` (from parsedRequest.data.answer) is passed as a named argument to applyValidatedProposal alongside `proposal`. Both exist as separate function arguments at the call site.
|
||
|
||
2. **Is the validated model proposal a separate object at that same point?**
|
||
Yes. `parsedProposal.proposal` is a distinct object from `answer`. It is the output of parseGraphUpdateProposal, validated against graphUpdateSchema, and passed as the `proposal` argument. The answer and proposal are different values in the JavaScript call stack.
|
||
|
||
3. **Does applyValidatedProposal receive both, or only the proposal/graph?**
|
||
Both. The function signature (line 2719) receives `{ situationGraph, proposal, previousQuestion, answer }`. All four are separate destructured parameters.
|
||
|
||
4. **Can deterministic code identify "user supplied" versus "model proposed" without asking the LLM?**
|
||
At the applyValidatedProposal call boundary: yes, by parameter identity. The `answer` argument contains user-supplied text; the `proposal` argument contains model-generated graph changes. These are distinguishable because they are different variables in the JavaScript runtime and come from different sources in orchestrator.js (user request vs LLM response).
|
||
|
||
However: within the proposal object itself, there is no metadata on individual nodes or edges indicating whether a specific node originated from user-supplied information or was model-inferred. The proposal schema (graphUpdateSchema) has no provenance field on addedNodes or addedEdges — situationNodeSchema contains only structural fields (id, label, description, kind, status, confidence, value, unit, evidenceIds, dependsOn, affects, parentId, childIds).
|
||
|
||
5. **At what exact function boundary does that distinction cease to be recoverable?**
|
||
The distinction is knowable at the `orchestrator.updateCaseWithDependencies` call site (line 683) because both `answer` and `proposal` are separate named arguments. It becomes **merged_or_lost** at two points:
|
||
|
||
a) Inside applyValidatedProposal: the `answer` parameter is passed only to `deriveReasoningStateOverride` and never used to annotate nodes/edges with source metadata. Origin information exists in scope but is not applied to the graph mutation path.
|
||
|
||
b) At `applyGraphUpdate` calls (lines 2882, 2912): only `graphSnapshot` and `proposalSnapshot` are passed. The `answer` parameter is discarded — it never reaches the mutation function that creates nodes/edges.
|
||
|
||
6. **Is the loss caused by which factor?**
|
||
**More than one boundary.** Specifically:
|
||
- Proposal schema (graphUpdateSchema/situationNodeSchema): no provenance field exists on addedNodes or addedEdges — this is the primary structural cause. If a provenance field existed, it could be populated.
|
||
- Application function signature (applyGraphUpdate at lines 2882/2912): `answer` is not passed to the mutation function that actually creates graph state. Even if nodes had provenance fields, the source information would need to be carried through to reach them.
|
||
- Graph schema: the resulting situationGraph has no provenance-aware structure (follow-on effect of proposal schema gap).
|
||
|
||
7. **Does the update path already contain enough information to assign provenance deterministically before graph storage?**
|
||
**No.** While user answer and model proposal are separate at the applyValidatedProposal call boundary, the answer text is a free-form string with no structural mapping to specific nodes in the proposal. The orchestrator does not know which parts of the LLM response were derived from user input versus independently inferred by the model. Even though both inputs exist as distinct parameters, there is no deterministic mechanism within the data flow to map user-supplied content to specific graph nodes/edges in the proposal.
|
||
|
||
### Experiment Conclusion
|
||
|
||
**Input origin remains explicit before application, but per-node supplied-versus-inferred provenance is not represented in the validated proposal and is lost before durable graph mutation.**
|
||
|
||
The raw user answer and validated model proposal are explicitly separate at the `applyValidatedProposal` function call boundary (orchestrator.js:683-688). Whole-input origin is explicit: `answer` = user supplied; `proposal` = model produced. However, per-node origin inside the proposal is not deterministically recoverable from the validated proposal alone. This distinction does not translate to per-node provenance because:
|
||
|
||
1. The graphUpdateSchema / situationNodeSchema has no provenance field on nodes or edges.
|
||
2. The raw user answer text has no structural mapping to proposal node boundaries — the LLM consumes the answer as context and generates additions independently, so there is no way to deterministically say "this node contains user information" versus "this node contains model inference."
|
||
3. The `answer` parameter is not forwarded to applyGraphUpdate, the function that actually mutates graph state.
|
||
|
||
Provenance loss occurs at the intersection of proposal schema (no provenance field) and application logic (answer discarded before mutation).
|
||
|
||
### Limitations
|
||
|
||
- Source-trace audit only; no live model or parsing executed
|
||
- Did not examine prompt templates to determine if they carry answer-supplying markers
|
||
- Did not examine reconcileResolutionSemantics for any implicit origin tagging
|
||
- Did not examine applyGraphUpdate internals beyond the call signatures
|
||
|
||
### Status
|
||
|
||
**Pending Rob's review.** Source-trace complete. No production code changed. Working tree clean before commit.
|
||
|
||
### Production Unchanged
|
||
|
||
- `lib/graph/orchestrator.js`: 0 lines changed
|
||
- `lib/graph/apply-proposal.js`: 0 lines changed
|
||
- `lib/graph/update-proposal.js`: 0 lines changed
|
||
- `lib/graph/schema.js`: 0 lines changed
|
||
- No production files modified
|
||
- Working tree clean before commit
|
||
|
||
### Tests / Validation Run
|
||
|
||
No test run required; Experiment 54C is a source-trace audit.
|
||
|
||
## Experiment 54D — Does the Update Prompt Already Preserve User-vs-Model Origin? (2026-08-07)
|
||
|
||
### Objective
|
||
|
||
Inspect the production graph-update prompt to determine whether it marks which content is the user's answer versus model-generated interpretation strongly enough that provenance could, in principle, be preserved downstream. This is a source-inspection experiment only. Do not implement provenance.
|
||
|
||
Two distinct questions:
|
||
- **Prompt-level source identity:** Can the model tell "this text is the user's answer"?
|
||
- **Proposal-level provenance:** Can downstream code tell "this particular proposed node directly represents supplied content rather than model inference"?
|
||
|
||
These may have different answers. The first may be explicit while the second is absent.
|
||
|
||
### Hypothesis
|
||
|
||
The existing update prompt may already contain clearly separated sections such as: previous graph/context; current question; user answer; instructions for graph changes. If that structure is explicit, the upstream information needed to distinguish user-supplied input from model-generated additions may already exist at prompt time. If the prompt blends everything into undifferentiated text, provenance is weaker even before proposal parsing.
|
||
|
||
### Files Actually Inspected
|
||
|
||
- `lib/graph/orchestrator.js` — lines 617-638 (caller passing answer to buildPrompt); line 20 (import statement)
|
||
- `lib/graph/prompt-builder.js` — full file (buildGraphUpdatePrompt function and its helpers: formatEnumValues, formatGraph, formatExampleAnswerBlock)
|
||
|
||
Production files inspected: `lib/graph/orchestrator.js`, `lib/graph/prompt-builder.js`.
|
||
|
||
### Prompt Builder Function Inspected
|
||
|
||
Function: `buildGraphUpdatePrompt` in `lib/graph/prompt-builder.js`, exported as `buildGraphUpdatePrompt` (line 26), aliased as `buildUpdatePrompt` (line 134).
|
||
|
||
### Arguments Passed Into Prompt Builder
|
||
|
||
- `situationGraph` — full current SituationGraph object
|
||
- `previousQuestion` — string (the previously selected question)
|
||
- `answer` — string (raw user answer, min 1 char, max 5000 chars per updateCaseRequestSchema)
|
||
- `promptVersion` — string, defaults to "v0.4"
|
||
|
||
The caller in orchestrator.js:633 passes these four arguments directly from function parameters and a config value. No provenance metadata is constructed or passed at the call site.
|
||
|
||
### User Answer Source Identity in Prompt
|
||
|
||
Status: **explicit**
|
||
|
||
Section `## User Answer` (line 48-49 of prompt-builder.js) contains the raw user answer as its entire content, separated by a Markdown header from everything above and below. The section header unambiguously identifies the block as user-supplied text. No other section contains this exact string.
|
||
|
||
### Previous Question Separation in Prompt
|
||
|
||
Status: **explicit**
|
||
|
||
Section `## Previous Selected Question` (line 45-46) contains only the previous question string, clearly separated by a Markdown header from both the graph above and the answer below.
|
||
|
||
### Graph/Context Separation in Prompt
|
||
|
||
Status: **explicit**
|
||
|
||
Section `## Current Situation Graph` (line 42-43) contains the full situation graph as formatted JSON, clearly separated by a Markdown header from all other content.
|
||
|
||
### Instruction Versus User-Content Separation
|
||
|
||
Status: **explicit**
|
||
|
||
All instruction blocks use Markdown headers (`## Proposal Rules`, `## Allowed Node Kinds`, `## Additional Guidance`, etc.). These headers create visual and structural boundaries between user-provided sections (graph, question, answer) and system instructions. The prompt does not interleave instructions within user-content blocks.
|
||
|
||
### Does Prompt Explicitly Identify User-Supplied Content
|
||
|
||
Status: **explicit**
|
||
|
||
Yes. The `## User Answer` header unambiguously marks which text block is the user's contribution. Additionally, Proposal Rule 9 states "Every new unknown must be directly traceable to the user's answer," and Rule 13a references "the relevant answer-derived decision or context node" — both rules reinforce that the answer section represents the authoritative user-supplied source.
|
||
|
||
### Does Prompt Explicitly Distinguish Supplied Meaning from Model Inference
|
||
|
||
Status: **implicit**
|
||
|
||
The prompt does not contain an explicit instruction telling the model to label or separate supplied meaning from inference in its output. However, implicit cues exist: Rule 9 requires traceability ("directly traceable to the user's answer"), Rule 9a requires a why-it-matters clause for new unknowns (implying the model must reason about what it derives versus what is given), and Rule 13a references "answer-derived" nodes. These create an expectation that the model should distinguish derived from supplied content, but there is no structural output mechanism to preserve that distinction in the JSON proposal.
|
||
|
||
### Does Requested Proposal Output Contain Provenance
|
||
|
||
Status: **absent**
|
||
|
||
The `graphUpdateSchema` (defined in `lib/graph/schema.js`, lines 155-163) has no provenance or source fields on any of its node or edge schemas. The `addedNodes` schema uses `situationNodeSchema` which contains only structural fields (id, label, description, kind, status, confidence, value, unit, evidenceIds, dependsOn, affects, parentId, childIds). No field exists to tag content as "user-supplied," "model-inferred," or any equivalent origin marker.
|
||
|
||
### Prompt-Level Source Identity Status
|
||
|
||
**explicit** — The model can clearly identify which input text came from the user (the `## User Answer` section) and which sections contain context/instructions (the graph JSON, previous question, rules, guidance).
|
||
|
||
### Proposal-Level Provenance Status
|
||
|
||
**absent** — The proposed output schema has no provenance fields. Even if the model understands which inputs were user-supplied, it has no mechanism to annotate its output nodes/edges with origin information.
|
||
|
||
### Trace from User Answer to Validated Proposal
|
||
|
||
| Stage | Source Identity | Supplied-vs-Inferred Meaning Explicit? |
|
||
|-------|----------------|----------------------------------------|
|
||
| 1. `answer` parameter in orchestrator.js:636 | explicit (parameter name) | N/A — raw string |
|
||
| 2. `## User Answer` section in prompt (prompt-builder.js:49) | explicit (named Markdown section) | implicit — the answer is given; no instruction distinguishes parts of it as supplied vs inferred |
|
||
| 3. LLM processes prompt and generates proposal | explicit (model can see which text is user answer) | implicit — rules require traceability but do not provide output mechanism for provenance |
|
||
| 4. `parsedProposal.proposal` after parseGraphUpdateProposal | absent (no source metadata on nodes/edges) | absent — graphUpdateSchema has no provenance fields |
|
||
|
||
### First Point Where Per-Node Provenance Becomes Unavailable
|
||
|
||
The proposed output schema (`graphUpdateSchema` in `lib/graph/schema.js`) defines the JSON contract returned by the LLM. Since none of its node or edge schemas include any origin/provenance field, per-node provenance is unavailable at the **output definition** stage — i.e., the prompt's own requested format cannot carry provenance even if the model understands it internally. This is upstream of parsing and validation; even before `parseGraphUpdateProposal` runs, the schema itself forbids provenance encoding.
|
||
|
||
### Could the Model Know Which Input Came from the User
|
||
|
||
**Yes.** The `## User Answer` section makes the user's contribution unmistakably identifiable. Rules 9 and 13a further reinforce the distinction between answer-derived content and model-generated additions.
|
||
|
||
### Could Downstream Deterministic Code Know Which Proposed Node Came from Supplied Meaning
|
||
|
||
**No.** The `graphUpdateSchema` has no provenance field on addedNodes, updatedNodes, or addedEdges. The validated proposal is a plain JSON object with no origin metadata. There is no deterministic mechanism to recover per-node provenance from the proposal alone.
|
||
|
||
### Experiment Conclusion
|
||
|
||
**Prompt clearly preserves user-source identity but proposal schema loses per-node provenance.**
|
||
|
||
The production update prompt (prompt-builder.js) already separates the user answer into a distinct named section (`## User Answer`) with clear visual and structural boundaries from graph context, instructions, and constraints. The model can unambiguously identify which text is user-supplied. Rules 9 and 13a reinforce traceability expectations.
|
||
|
||
However, the requested output schema (graphUpdateSchema in lib/graph/schema.js) has no provenance fields on nodes or edges. Even if the model internally distinguishes derived from supplied content, the JSON output contract cannot encode that distinction. Provenance is lost at the output-definition stage — before any parsing or validation occurs.
|
||
|
||
This means:
|
||
- Prompt-level source identity: explicit
|
||
- Proposal-level provenance: absent (structural limitation of schema)
|
||
- The tested prompt already preserves user-source identity clearly; the blocking gap identified here is that the validated proposal does not carry per-node provenance forward. The eventual representation remains undecided.
|
||
|
||
### Limitations
|
||
|
||
- Source-inspection audit only; no live model call executed
|
||
- Inspected the production prompt-builder and its immediate caller only
|
||
- Did not inspect whether evidenceType in reconstruction can carry origin information for the answer field itself
|
||
- Did not evaluate whether a schema change would be sufficient or whether additional upstream markers are needed
|
||
- Conclusions apply to the current prompt version (v0.4); earlier versions may differ
|
||
|
||
### Status
|
||
|
||
**Pending Rob's review.** Source-inspection complete. No production code changed. Working tree clean before commit.
|
||
|
||
### Production Unchanged
|
||
|
||
- `lib/graph/orchestrator.js`: 0 lines changed
|
||
- `lib/graph/prompt-builder.js`: 0 lines changed
|
||
- `lib/graph/schema.js`: 0 lines changed
|
||
- No production files modified
|
||
- Working tree clean before commit
|
||
|
||
### Tests / Validation Run
|
||
|
||
No test run required; Experiment 54D is a prompt-source audit.
|
||
|
||
|
||
## Experiment 54E — Can Existing Evidence References Preserve Provenance Without Adding a Node Field? (2026-08-07)
|
||
|
||
### Objective
|
||
|
||
Inspect whether the current graph design already preserves referential provenance through existing evidence identities and references, without requiring a new node field. This is a source-inspection experiment only. Do not implement provenance.
|
||
|
||
### Hypothesis
|
||
|
||
Existing `evidenceIds` and evidence records may already provide enough referential structure to preserve provenance if: (1) evidence records survive beyond reconstruction; (2) their IDs remain resolvable later; (3) `evidenceType` remains attached to those records; (4) graph nodes can reliably link to the evidence records that justify them.
|
||
|
||
### Files Inspected
|
||
|
||
- `lib/reconstruction/schema.js` — evidenceRecordSchema definition, reconstructionV2Schema evidence field
|
||
- `lib/graph/schema.js` — situationNodeSchema evidenceIds field, graphUpdateSchema structure
|
||
- `lib/graph/builder.js` — how evidence IDs are assigned during initial graph build (buildInitialGraph)
|
||
- `lib/graph/orchestrator.js` — startCase and updateCaseWithDependencies return values; evidence flow through the pipeline
|
||
- `lib/reconstruction/schema.js` lines 113–127 — evidenceRecordSchema fields
|
||
- `lib/graph/apply-proposal.js` — how evidenceIds are populated during graph updates (appendUniqueValue pattern)
|
||
- `lib/graph/prompt-builder.js` — rule 14 "Do not invent evidence"
|
||
|
||
### Evidence Record Identity
|
||
|
||
Each reconstruction evidence record (v0.2 schema) has:
|
||
- **Stable ID**: `id: z.string().min(1)` — deterministic identifier present in every record.
|
||
- **evidenceType**: `z.enum(["direct_observation", "reported_statement", "interpretation", "assumption", "inferred_relationship"])` — five distinct values.
|
||
- **Sufficient information to distinguish supplied from inferred**: The evidence-record schema contains vocabulary capable of distinguishing supplied-like from inferred-like material, but Experiment 54E did not validate how those values are assigned in production.
|
||
|
||
Answer: **Yes, evidence records carry sufficient identity.**
|
||
|
||
### Graph Reference Behaviour
|
||
|
||
When a graph node contains an `evidenceId`:
|
||
- During `buildInitialGraph` (builder.js:62–70), evidence records from the analysis output are placed in an `evidenceMap`. Nodes are created with `evidenceIds` populated from actual evidence record IDs (builder.js:100: `node.evidenceIds.push(obs.id)`).
|
||
- These IDs refer to real evidence records during construction. However, after graph construction, the graph contains only string IDs — they would resolve if a lookup table existed, but they are opaque strings within the node object itself.
|
||
|
||
Answer: **IDs refer to real evidence records at build time; become opaque strings in the graph post-construction.**
|
||
|
||
### Evidence-Record Lifetime
|
||
|
||
The critical flow is:
|
||
1. `analyseScenario` (lib/analysis.js) produces evidence records via reconstructionV2Schema, returned as `data.evidence`.
|
||
2. In `startCase` (orchestrator.js:378), `analysis.evidence` is passed to `buildInitialGraph` solely to populate node `evidenceIds`.
|
||
3. **The evidence records themselves are NOT included in `startCase` return value.** The return at orchestrator.js:489-566 includes only `situationGraph`, `selectedQuestion`, `diagnostics`, and `assessment`. No `evidence` field exists in the return object.
|
||
4. In `updateCaseWithDependencies`, no new evidence records are created anywhere in the pipeline. Rule 14 of the update prompt ("Do not invent evidence") explicitly forbids the model from creating them. The prompt-builder.js shows no mechanism for evidence generation during updates.
|
||
5. There is **no persistence layer** in this codebase that stores case state to disk or a database. The API routes (cases/start/route.js, cases/update/route.js) return data to the client; they do not persist anything.
|
||
|
||
The initial evidence records exist only during the startCase execution lifetime. They are consumed to populate graph node evidenceIds but never returned alongside the case state. Subsequent update cycles produce no evidence records at all.
|
||
|
||
Answer: **not_retained_with_graph**
|
||
|
||
### Provenance Through Reference
|
||
|
||
If a later consumer receives normal persisted case state (which contains only `situationGraph` with nodes having string `evidenceIds`), it cannot:
|
||
1. Read a graph node — yes, the node and its evidenceIds are present.
|
||
2. Follow its evidenceIds — no matching evidence records exist anywhere in the persisted state.
|
||
3. Resolve each ID to an evidence record — impossible; records do not exist.
|
||
4. Inspect evidenceType — N/A; no records to inspect.
|
||
|
||
Answer: **no** — The reference chain breaks at step 2/3 because evidence records are not part of the returned case state.
|
||
|
||
### Node-to-Evidence Completeness
|
||
|
||
- **Initial graph nodes (from buildInitialGraph)**: Nodes from observedStates do receive evidence references (builder.js:100). Other node types (actors, systemsOrObjects, differences, contradictions, unknowns, interpretations) are created WITHOUT evidence references — only observedStates nodes get `evidenceIds` populated.
|
||
- **Nodes added during update**: No new evidence records are ever created during the update flow. Nodes created via `applyValidatedProposal` may have their `evidenceIds` field set (the schema allows it), but no source code in the update path populates them from actual evidence records. There is no mechanism to create or assign evidence record IDs during updates.
|
||
|
||
Answer: **Incomplete by design** — Initial graph supplies evidenceRefs only for observedStates; all other nodes get none. Update-phase nodes get no evidence references at all.
|
||
|
||
### Referential Recoverability Trace
|
||
|
||
| Step | Result | Reason |
|
||
|------|--------|--------|
|
||
| Graph node → evidenceId | works | Nodes carry evidenceIds as string arrays |
|
||
| evidenceId → evidence record | breaks | Evidence records are consumed during startCase and never returned; no persistence layer retains them |
|
||
| Evidence record → evidenceType | N/A | Chain already broken at previous step |
|
||
|
||
### Answered Questions
|
||
|
||
1. **Is node identity already sufficient?** No — nodes exist and have evidenceIds, but without the referenced records they carry no provenance information.
|
||
2. **Is evidence identity already sufficient?** The evidence records themselves (when they exist) carry sufficient identity (id + evidenceType). But they are not available in case state.
|
||
3. **Are evidence records retained long enough to resolve references?** No — consumed during construction, not returned with graph.
|
||
4. **Is evidenceType available after resolution?** N/A — cannot resolve the reference to get there.
|
||
5. **Are node-to-evidence links populated consistently enough?** No — only observedStates nodes in initial build receive references; all update-phase nodes receive none.
|
||
6. **Can supplied-versus-inferred provenance currently be recovered referentially?** No.
|
||
7. **Is the problem primarily:** More than one of these: (a) evidence provenance exists upstream during reconstruction but is not persisted alongside graph state; (b) no new evidence records are created or retained during update cycles; (c) incomplete linkage even at construction time (only observedStates nodes get references).
|
||
|
||
### Conclusion
|
||
|
||
**Existing provenance exists but referential linkage is incomplete** — specifically: the evidence-record schema contains vocabulary capable of distinguishing supplied-like from inferred-like material (Experiment 54E did not validate how those values are assigned), and the reference chain breaks because (1) evidence records are consumed during startCase and never returned alongside graph state, making them unrecoverable in persisted case data; and (2) no evidence records are created or retained during update cycles at all. The problem is primarily missing persistence of existing provenance combined with a gap in evidence record creation during updates.
|
||
|
||
### Limitations
|
||
|
||
- Source-inspection audit only; no live execution tested
|
||
- Client-side state management was not inspected — if the client retains evidence records alongside graph state, referential recovery may work on that layer
|
||
- Did not evaluate whether the client could reconstruct provenance from UI-visible data
|
||
- Conclusions apply to the server-side pipeline as currently implemented
|
||
|
||
### Status
|
||
|
||
**Pending Rob's review.** Source-inspection complete. No production code changed. Working tree clean before commit.
|
||
|
||
## Experiment 54F — Is evidenceType Actually Reliable Provenance, or Just a Model Label? (2026-08-07)
|
||
|
||
### Objective
|
||
|
||
Inspect whether the current reconstruction path assigns `evidenceType` from a clear provenance rule, or whether the LLM itself decides whether something is a reported statement, interpretation, assumption, or inferred relationship. Source-audit only. Do not implement provenance.
|
||
|
||
### Hypothesis
|
||
|
||
`evidenceType` may be semantically useful without being trustworthy provenance. If the LLM is asked to classify reconstructed content into those categories, then `reported_statement` may mean "the model thinks this looks like a reported statement" rather than "production code knows this came directly from the user."
|
||
|
||
### Files Inspected
|
||
|
||
- `prompts/reconstruct-v0.3.md` — full file; CRITICAL RULE 7 at line 155 specifying how evidenceType should be classified
|
||
- `lib/reconstruction/prompt.js` — `buildV3Prompt` function (lines 67–78), which loads prompts/reconstruct-v0.3.md and substitutes the scenario via `{{SCENARIO}}`
|
||
- `lib/reconstruction/schema.js` — `evidenceRecordSchema` definition at lines 113–127; evidence field at line 185
|
||
- `lib/reconstruction/compatibility.js` — `normaliseAnalysisResponse` function (full file, lines 1–52); deterministic normalisation applied to parsed reconstruction results
|
||
|
||
### Who Chooses evidenceType?
|
||
|
||
Answer: **the LLM**.
|
||
|
||
The prompt in `prompts/reconstruct-v0.3.md` line 155 states:
|
||
|
||
> **evidenceType**: classify each evidence item clearly as either a direct observation, a reported statement, an interpretation, an assumption, or an inferred relationship. Do not treat raw counts as proof of causal relationships — they may be inferred relationships only when supported by explicit reasoning about denominators or rates.
|
||
|
||
This is a classification instruction directed at the model. No production code determines or constrains which evidenceType value a given piece of evidence should have. The LLM receives scenario text and chooses an `evidenceType` for each evidence record it creates in its output.
|
||
|
||
### Relevant Reconstruction Instruction (Summarised)
|
||
|
||
The prompt instructs the model to:
|
||
- Produce a JSON object with `inputClassification`, `reconstruction`, `evidence`, and `nextQuestion` keys;
|
||
- For the `evidence` array, produce records each containing `id`, `description`, `evidenceType`, `source`, `attribution`, `confidence`, and `importance`;
|
||
- **For evidenceType specifically**, instructs the model to "classify each evidence item clearly as either a direct observation, a reported statement, an interpretation, an assumption, or an inferred relationship" based on its judgment of what each piece of evidence represents;
|
||
- No production-level rules restrict which category maps to which source origin.
|
||
|
||
### Raw User Origin Status Before Reconstruction
|
||
|
||
Answer: **model_classified**.
|
||
|
||
The raw user scenario text is passed into the prompt via `{{SCENARIO}}` substitution at line 158 of reconstruct-v0.3.md. It is not wrapped, tagged, or structurally separated from any other content. Production code (lib/reconstruction/prompt.js) performs only a single string substitution: `content.replace("{{SCENARIO}}", scenario)`. There is no structural marker in the prompt that tells the model "this text came from the user" — it simply appears at the end of the prompt as unlabelled text.
|
||
|
||
Note on Experiment 54D's finding: The update prompt (`lib/graph/prompt-builder.js`) does separate user-supplied answers via a Markdown header (`## User Answer`). However, **Experiment 54F inspects the initial reconstruction path only**, where no such structural separation exists. The update path uses a different prompt from a different builder and is not part of this audit's scope.
|
||
|
||
### Raw User Origin Status Inside Reconstruction Output
|
||
|
||
Answer: **model_classified**.
|
||
|
||
After the LLM produces its evidence records, each record's `evidenceType` reflects the model's own judgment about what category that evidence belongs to — not any deterministic derivation from source origin. The only production-side manipulation of `evidenceType` is in `lib/reconstruction/compatibility.js`, which performs a single enum normalisation: if `evidenceType === "reported_claim"` (from an earlier schema version), it converts it to `"reported_statement"`. This does not add provenance information; it only adjusts for schema versioning.
|
||
|
||
### Is evidenceType Deterministic?
|
||
|
||
Answer: **no**. The model itself chooses each `evidenceType` value at generation time. No production code determines it from source structure.
|
||
|
||
### Is reported_statement Trustworthy as User Provenance?
|
||
|
||
Answer: **no**. A record with `evidenceType === "reported_statement"` means "the model classified this evidence item as a reported statement" — not "production code knows this came directly from the user." The model makes this classification based on its understanding of the scenario text, which may conflate what it inferred about reported claims with what was actually stated by a person.
|
||
|
||
### Are interpretation / assumption / inferred_relationship Trustworthy as Model Provenance?
|
||
|
||
Answer: **partially**. These values do indicate categories the model itself assigned to its own output, so they can serve as semantic labels for "model-generated content" in a loose sense. However, the boundary between these categories is defined by the LLM's judgment, not by production code — the model may classify something as an assumption when it was actually directly stated by the user, or vice versa. There is no deterministic gate separating user-supplied from model-generated content at classification time.
|
||
|
||
### Trace: Raw User Statement → Reconstruction Prompt
|
||
|
||
| Stage | User origin explicit? | Who assigns evidenceType? | Provenance strengthens/weakened? |
|
||
|-------|----------------------|--------------------------|--------------------------------|
|
||
| Raw user scenario text | Not structurally marked (just pasted as `{{SCENARIO}}`) | N/A | — |
|
||
| Reconstruction prompt (reconstruct-v0.3.md) | Scenario appears unlabelled at end of file, no structural distinction from system instructions | Model chooses per its judgment (CRITICAL RULE 7) | Weakened — user text is indistinguishable in structure from other prompt content |
|
||
|
||
### Trace: Reconstruction Prompt → LLM Evidence Record
|
||
|
||
| Stage | User origin explicit? | Who assigns evidenceType? | Provenance strengthens/weakened? |
|
||
|-------|----------------------|--------------------------|--------------------------------|
|
||
| Model processes prompt and generates evidence records | No structural marker in prompt distinguishes user content | Model (based on CRITICAL RULE 7 classification instruction) | Unchanged — model judgment, not deterministic derivation |
|
||
|
||
### Trace: LLM Evidence Record → Schema Validation
|
||
|
||
| Stage | User origin explicit? | Who assigns evidenceType? | Provenance strengthens/weakened? |
|
||
|-------|----------------------|--------------------------|--------------------------------|
|
||
| Raw model output (JSON) | N/A — already model-generated | Model's own choice | Unchanged |
|
||
| Compatibility normalisation (`lib/reconstruction/compatibility.js`) | No provenance added | `reported_claim` → `reported_statement` (schema versioning only) | Neutral — no provenance information added or removed; only enum compatibility |
|
||
| Zod validation against `reconstructionV2Schema` | N/A — schema validates structure, not origin | Schema accepts whatever `evidenceType` the model chose (enum valid) | Unchanged — schema enforces enum validity but not provenance correctness |
|
||
|
||
### Does Schema Validation Verify Origin or Only Enum Validity?
|
||
|
||
Answer: **only enum validity**. The Zod schema (`lib/reconstruction/schema.js` line 113-127) validates that `evidenceType` is one of five allowed strings. It does not and cannot verify what produced the value — whether it came from a user, was derived deterministically by production code, or was classified by the LLM.
|
||
|
||
### Can Downstream Deterministic Code Safely Treat evidenceType as Provenance?
|
||
|
||
Answer: **no**. `evidenceType` values are model-generated labels reflecting semantic categories the model chose during reconstruction. They do not correspond to any deterministic derivation from source origin. Production code cannot safely interpret `evidenceType === "reported_statement"` as "this was definitely supplied by the user" without additional provenance infrastructure.
|
||
|
||
### Does Experiment 54E's Referential Approach Depend on a Label That Is Itself Model-Generated?
|
||
|
||
Answer: **yes**. Experiment 54E found that evidence records carry sufficient identity (id + evidenceType) when they exist. But `evidenceType` is itself model-generated, not structurally derived from source origin. The referential approach depends on labels the LLM chose, meaning provenance inference at that point already rests on model classification rather than deterministic provenance — compounding the provenance gap rather than resolving it.
|
||
|
||
### Primary Provenance Gap After This Audit
|
||
|
||
Answer: **both**. The provenance gap has two independent causes:
|
||
1. **Evidence-record lifetime** (54E): Records are consumed during startCase and never returned with graph state.
|
||
2. **Evidence-type trustworthiness** (54F): Even if records were retained, `evidenceType` is model classification, not deterministic provenance.
|
||
|
||
### Experiment Conclusion
|
||
|
||
**evidenceType is semantic model output, not reliable provenance.** The reconstruction prompt instructs the LLM to classify each evidence item into one of five categories based on its own judgment. No production code deterministically derives `evidenceType` from source origin. The compatibility layer adds only a single enum normalisation (`reported_claim` → `reported_statement`) for schema versioning. Schema validation enforces enum validity but not provenance correctness. Experiment 54E's referential approach depends on a label that is itself model-generated, meaning the provenance chain was already unreliable before it broke due to missing persistence. The primary gap has two independent causes: evidence records are not retained alongside graph state, and `evidenceType` values themselves are model-classified rather than deterministically derived from source origin.
|
||
|
||
### Limitations
|
||
|
||
- Source-inspection audit only; no live execution tested
|
||
- Inspected only the initial reconstruction path (v0.3 prompt), not the update path which uses a different prompt structure
|
||
- Did not evaluate whether `source` or `attribution` fields on evidence records carry any provenance value beyond what `evidenceType` does
|
||
- Did not test how reliable the LLM's evidenceType classification is in production (this would require validation, not inspection)
|
||
- Conclusions apply to the v0.3 reconstruction prompt as currently implemented
|
||
|
||
### Status
|
||
|
||
**Pending Rob's review.** Source-inspection complete. No production code changed. Working tree clean before commit.
|
||
|
||
## Experiment 54G — Do Evidence Records Contain Deterministic Source Linkage Back to User Words? (2026-08-07)
|
||
|
||
### Objective
|
||
|
||
Answer one narrow provenance question: does the current reconstruction output contain enough source information to deterministically prove that an evidence record came from the user's actual words, without trusting the LLM's `evidenceType` label? Source-audit only. Do not implement provenance.
|
||
|
||
54F established that `evidenceType` is model classification, not reliable provenance. 54G tests whether evidence records nevertheless retain deterministic linkage to the user's actual words — such as exact text, character offsets, turn ID, source path, or other structural location data.
|
||
|
||
### Hypothesis
|
||
|
||
An evidence record may already preserve enough source material — such as exact text, quote, source excerpt, source ID, character range, source path, turn ID, or input reference — for deterministic code to verify that a record is directly grounded in the user's supplied text. If no such information exists, then evidence-record identity alone cannot establish user provenance.
|
||
|
||
### Files Inspected
|
||
|
||
- `lib/reconstruction/schema.js` — `evidenceRecordSchema` definition (lines 113–127); `reconstructionV2Schema` (line 182)
|
||
- `prompts/reconstruct-v0.3.md` — evidence record output format (lines 126–136); CRITICAL RULE 7 (line 155)
|
||
- `lib/reconstruction/compatibility.js` — `normaliseAnalysisResponse` function (full file); source null-handling (lines 31–38)
|
||
- `tests/reconstruction/compatibility.test.js` — real evidence-record shapes from tests (lines 36–44, 54–72, 107–115, 137–147)
|
||
- `lib/analysis.js` — `analyseScenario` function (full file); confirms raw scenario is NOT returned alongside evidence records (lines 173–188)
|
||
|
||
### Evidence-Record Source-Related Fields
|
||
|
||
From `evidenceRecordSchema`:
|
||
|
||
| Field | Type | Provenance potential |
|
||
|-------|------|---------------------|
|
||
| `id` | `z.string().min(1)` | None — free-form, LLM-generated. No structural reference to input. |
|
||
| `description` | `z.string().min(1)` | None — model-generated summary of the evidence, not user text. |
|
||
| `evidenceType` | enum (5 values) | None — model classification, per 54F. Not structural provenance. |
|
||
| `source` | `z.string().optional()` | Partially — prompt says "who/where this came from". But: it is free-form text generated by the LLM, not deterministic production code output. In tests, it appears as `"report"` (free label) or `null` (removed by normalisation). No character offsets, turn IDs, span data, or source record references are structured into it. |
|
||
| `attribution` | `z.string().nullable().optional()` | None — test fixtures show `null` by default. When populated, it is free-form model text describing who said what, not a deterministic production-code identifier back to input. |
|
||
| `confidence` | enum | None — subjective confidence level, not source data. |
|
||
| `importance` | enum | None — semantic weight, not source data. |
|
||
|
||
### Does Raw User Statement Coexist with Reconstruction Result?
|
||
|
||
**Not retained alongside results.** The raw user statement (`scenario`) is the input parameter to `analyseScenario`. It is available to production code while reconstruction is being performed, but it is NOT retained alongside the returned reconstruction/evidence state for later deterministic verification. At validation time, deterministic code has:
|
||
- the evidence records (with model-generated descriptions and types);
|
||
- but NOT the raw user statement itself.
|
||
|
||
Even if deterministic code had the scenario text available at validation time, the evidence records still lack any field containing verbatim user text or structural location back to it.
|
||
|
||
### Does Evidence Record Preserve Exact User Wording?
|
||
|
||
**No.** The `description` field is a model-generated summary/paraphrase of the evidence — not the user's actual words. The `source` field, when non-null, is free-form text like `"report"` (a label, not a quote). No evidence record contains verbatim or near-verbatim user text.
|
||
|
||
### Does Evidence Record Preserve Source Location / Span / Turn Identity?
|
||
|
||
**No.** None of the evidence record fields contain:
|
||
- character offsets within user input;
|
||
- sentence or line index;
|
||
- turn or message ID;
|
||
- source record ID owned by production code;
|
||
- any structural reference to a specific location in the original input.
|
||
|
||
The `source` field prompt description says "who/where this came from" but it is free-form, inconsistently populated (sometimes null), and produced by the LLM not deterministic code.
|
||
|
||
### Who Creates Evidence Record IDs?
|
||
|
||
**The LLM.** The prompt template (line 128 of reconstruct-v0.3.md) says `"id": "<any unique string>"`. The schema only requires `z.string().min(1)`. No production code generates or constrains the ID beyond non-emptiness. The ID is opaque and carries no provenance semantics.
|
||
|
||
### Can Evidence Records Be Verified Against Raw User Input?
|
||
|
||
**No.** Deterministic verification would require:
|
||
1. Raw user text available at validation time — NOT present in result;
|
||
2. Each evidence record containing verbatim user text or a deterministic structural reference to a location within it — neither present.
|
||
|
||
The `description` field is model-generated paraphrase, not verbatim text. The `source` field is free-form model output, not a structured pointer. No field survives from the raw input to the result in a form that code can verify.
|
||
|
||
### Trace: Raw User Statement → Evidence Record → Source Verification
|
||
|
||
| Stage | Raw source identity available? | Exact source wording/location? | Deterministic verification possible? |
|
||
|-------|-------------------------------|-------------------------------|-------------------------------------|
|
||
| Raw user statement | explicit (input parameter) | explicit (raw text) | N/A — this is the ground truth |
|
||
| Reconstruction prompt | absent (scenario pasted as {{SCENARIO}} with no structural markers) | partial (present in prompt body but unlabelled and indistinguishable from system instructions) | No |
|
||
| LLM evidence record | absent (all source identity lost to model generation) | absent (description is model paraphrase; source/attribution are free-form model text or null) | No |
|
||
| Validated result (analysis.js return) | absent (scenario not returned alongside evidence) | absent | No |
|
||
|
||
### Trace: Evidence Record → Source Verification
|
||
|
||
| Step | Status |
|
||
|------|--------|
|
||
| Get evidence record fields | present |
|
||
| Check description against user text | impossible — no verbatim text to compare |
|
||
| Check source/attribution as structured location | impossible — free-form model output, not production-code identifiers |
|
||
| Check id for provenance semantics | impossible — opaque LLM-generated string |
|
||
| Verify evidenceType independently | impossible — no source text to verify against |
|
||
|
||
### Does Current Reconstruction Schema Contain Enough Information for Deterministic Provenance?
|
||
|
||
**No.** The schema fields `source` and `attribution` exist as free-form optional strings, but neither is deterministic (they are model-generated), nor do they contain structured location data. The raw user statement is not returned alongside the evidence records. Even if it were, no evidence record field contains verbatim text or structural reference to it.
|
||
|
||
### Primary Source-Linkage Gap
|
||
|
||
Evidence records carry no verbatim user text and no deterministic structural reference (character offsets, turn/message IDs, source record references) back to the original input. The only candidate fields (`source`, `attribution`) are free-form model-generated strings that may be null, and which cannot be independently verified against user input. Additionally, the raw user statement itself is not returned with the validated reconstruction result at validation time.
|
||
|
||
### Experiment 54G Conclusion
|
||
|
||
**Bounded findings.** Evidence records do not contain verbatim source text. Evidence records do not contain deterministic source locations (character offsets, turn/message IDs, or structural references). `evidenceType` is model classification, not production-code provenance. Current evidence records therefore cannot independently prove source provenance without trusting the LLM's label.
|
||
|
||
### Limitations
|
||
|
||
- Source-inspection audit only; no live execution tested
|
||
- Inspected only the initial reconstruction path (v0.3 prompt) and the analysis pipeline
|
||
- Did not inspect whether evidence records are stored with graph nodes in a way that could later be recovered
|
||
- Did not inspect the update path which uses different prompts and potentially different provenance characteristics
|
||
- Conclusions apply to the v0.3 reconstruction output as currently implemented
|
||
|
||
### Status
|
||
|
||
**Pending Rob's review.** Source-inspection complete. No production code changed. Working tree clean before commit.
|
||
|
||
## Experiment 54H — Can Trustworthy Source Identity Be Established Before LLM Interpretation? (2026-08-07)
|
||
|
||
### Objective
|
||
|
||
First, tighten Experiment 54G so it does not imply the raw user statement is unavailable during reconstruction.
|
||
|
||
Then test one small capability outside the active engine:
|
||
|
||
> **Can production code create a deterministic source record from the user's raw input before any LLM interpretation occurs, so later reasoning has something trustworthy to refer back to?**
|
||
|
||
This is a passive test-only experiment. Do not integrate provenance into the graph or runtime.
|
||
|
||
### Hypothesis
|
||
|
||
A trustworthy provenance root may not require the LLM to decide anything. Production code may be able to take the raw input and create a small immutable source record **before** reconstruction that provides stable identity, source type, and verbatim source text — without attempting to classify what individual reasoning claims mean. If this cannot be done cleanly without introducing hidden interpretation, record why.
|
||
|
||
### Context Used
|
||
|
||
- `docs/current-handoff.md` (54G findings)
|
||
- Experiment 54G only in `docs/design-evolution-log.md`
|
||
- `lib/reconstruction/schema.js` — evidenceRecordSchema to avoid naming collisions
|
||
- Node.js built-in `crypto.createHash('sha256')` — standard library, deterministic
|
||
|
||
### Test-Only Source Record Shape
|
||
|
||
```json
|
||
{
|
||
"sourceId": "...",
|
||
"sourceType": "user_input",
|
||
"verbatimText": "..."
|
||
}
|
||
```
|
||
|
||
This is deliberately small and does not include: evidence type, confidence, interpretation, node kind, relevance, inferred meaning, or summary.
|
||
|
||
### Deterministic ID Method Used
|
||
|
||
SHA-256 hash of the verbatim input text using Node.js built-in `crypto.createHash('sha256')`.
|
||
|
||
### Why the ID Method Is Deterministic
|
||
|
||
- SHA-256 is a deterministic cryptographic hash function: same input always produces the same output; different inputs produce different outputs (with negligible collision probability).
|
||
- Uses Node.js standard library — no LLM, no random UUID, no clock/time dependency, no mutable global state.
|
||
- Input is the verbatim string in UTF-8 encoding with no transformation or normalisation.
|
||
|
||
### Four Input Cases
|
||
|
||
| Case | Text | Purpose |
|
||
|------|------|---------|
|
||
| 1 | "The business has lost three major customers this year." | Baseline simple statement |
|
||
| 2 | "Three major customers have left the business this year." | Paraphrase — confirm identity tracks actual text, not meaning |
|
||
| 3 | "Revenue is down. I think pricing may be part of the problem, but I am not sure." | Multi-sentence — confirm full verbatim input survives unchanged |
|
||
| 4 | Same as Case 1 | Confirm deterministic identity |
|
||
|
||
### Focused Test Results (9 tests, all pass)
|
||
|
||
| # | Test | Result |
|
||
|---|------|--------|
|
||
| 1 | Case 1: verbatimText equals the exact input string | Pass |
|
||
| 2 | Case 2 (paraphrase): sourceType is always user_input | Pass |
|
||
| 3 | Case 1 and Case 4 produce identical sourceId | Pass |
|
||
| 4 | Case 1 and Case 2 produce different sourceIds despite similar meaning | Pass |
|
||
| 5 | Case 3: multi-sentence verbatim text preserved unchanged | Pass |
|
||
| 6 | Input string is not mutated by createSourceRecord | Pass |
|
||
| 7 | Helper output is deterministic across repeated calls | Pass |
|
||
| 8 | No interpretation or summarisation occurs in the source record | Pass |
|
||
| 9 | Different inputs produce different IDs (distinct text) | Pass |
|
||
|
||
### Case Results Summary
|
||
|
||
- **Case 1 result:** verbatimText preserved exactly ✓; deterministic SHA-256 ID produced.
|
||
- **Case 2 paraphrase result:** Same meaning, different text → different sourceId ✓; verbatimText preserved exactly.
|
||
- **Case 3 multi-sentence result:** Full multi-sentence text preserved unchanged ✓; single deterministic ID.
|
||
- **Case 4 repeated-input result:** Identical to Case 1 text → identical sourceId ✓.
|
||
|
||
### Same-Input Identity Result
|
||
|
||
Confirmed: Cases 1 and 4 produce exactly the same sourceId (sha256 of identical string = identical hash).
|
||
|
||
### Different-Text Identity Result
|
||
|
||
Confirmed: Cases 1, 2, and 3 produce different sourceIds despite Case 1≈Case 2 in meaning.
|
||
|
||
### Verbatim-Text Preservation Result
|
||
|
||
All three unique inputs preserved exactly as-is. No summarisation, normalisation, or transformation applied to verbatimText.
|
||
|
||
### Input Immutability Result
|
||
|
||
Confirmed: the original input string is never mutated. createSourceRecord reads via `String(rawInput)` (copy) before hashing.
|
||
|
||
### LLM / Network Dependency
|
||
|
||
None. The helper uses only Node.js built-in `crypto` and JavaScript primitives. Zero inference calls. Zero network calls.
|
||
|
||
### Does the Helper Perform Semantic Interpretation
|
||
|
||
No. The helper outputs exactly: `{ sourceId, sourceType, verbatimText }`. No meaning is extracted, classified, summarised, or inferred.
|
||
|
||
### Distinction Between Source Identity and Semantic Provenance
|
||
|
||
This experiment establishes **source identity** — answering "what exact material entered the system, and what stable identity can we give that material?" It does **not** answer: what the text means, whether it is evidence, whether it was inferred, or whether graph nodes can link to it. Source identity ≠ claim provenance.
|
||
|
||
### What Remains Explicitly Untested
|
||
|
||
- Linking evidence records to source records
|
||
- Linking graph nodes to evidence or source
|
||
- Update-turn source identity (not initial reconstruction)
|
||
- Persistence of source records alongside graph state
|
||
- Whether production code can reliably invoke this helper at the right boundary
|
||
- How downstream consumers discover and use the source record
|
||
- Multiple-input scenarios (multi-turn conversations)
|
||
|
||
### Experiment Conclusion
|
||
|
||
**Deterministic source identity is feasible before LLM interpretation.** A deterministic SHA-256 hash of raw user text produces stable, verifiable identity without any semantic processing. The concept is straightforward because it operates purely at the input boundary — no classification, no inference, no graph contract required.
|
||
|
||
### Limitations
|
||
|
||
- Test-only implementation; not integrated into any production path
|
||
- Single-input only (does not address multi-turn or batched input scenarios)
|
||
- Does not test whether production code can reliably invoke this at the correct reconstruction boundary
|
||
- Does not test persistence, retrieval, or downstream linking
|
||
- SHA-256 collision resistance is sufficient for source identity but does not guarantee uniqueness across all possible inputs in practice
|
||
|
||
### Status
|
||
|
||
**Closed.** Bounded conclusion: deterministic source identity is feasible before LLM interpretation; identical tested text produces stable identity; source identity does not establish claim or graph provenance. No production code changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/reconstruction/deterministic-source-record.test.js`.
|
||
|
||
## Experiment 54I — Can Two Different Interpretations Stay Anchored to the Same User Source? (2026-08-07)
|
||
|
||
### Objective
|
||
|
||
Take the deterministic source identity proved in Experiment 54H and test the next smallest step:
|
||
|
||
> **Can two different interpretations of the exact same user input remain separate while both retaining an explicit link back to that same source?**
|
||
|
||
This experiment deliberately does not decide which interpretation is better. It tests whether the reasoning system could preserve one source, multiple interpretations, and clear lineage back to that source without silently replacing the original input or collapsing the interpretations together. This is a passive, test-only experiment.
|
||
|
||
### Hypothesis
|
||
|
||
A very small lineage structure may be enough to demonstrate that:
|
||
- one source identity can anchor multiple interpretations;
|
||
- interpretations remain separately identifiable;
|
||
- neither interpretation changes or replaces the source;
|
||
- a later consumer can deterministically see that both interpretations came from the same exact input.
|
||
|
||
If even this cannot be represented cleanly without semantic ambiguity, record that honestly.
|
||
|
||
### Context Used
|
||
|
||
- `docs/current-handoff.md` (Experiment 54H findings and Return-to-Work Note)
|
||
- Experiment 54H only in `docs/design-evolution-log.md`
|
||
- `tests/reconstruction/deterministic-source-record.test.js` (to verify the source-identity helper remains intact)
|
||
- Node.js built-in `crypto.createHash('sha256')` — deterministic, no LLM
|
||
|
||
### Fixed Source
|
||
|
||
Raw user input:
|
||
> Revenue is down. I think pricing may be part of the problem, but I am not sure.
|
||
|
||
One deterministic source record created from that exact text using the Experiment 54H method (SHA-256 of verbatim text).
|
||
|
||
### Two Fixed Interpretations
|
||
|
||
**Interpretation A:**
|
||
> Pricing may be contributing materially to the revenue decline.
|
||
|
||
**Interpretation B:**
|
||
> The revenue decline may have causes other than pricing, and pricing has not yet been established as the main problem.
|
||
|
||
Both are plausible readings of the source. The experiment does not claim either is correct.
|
||
|
||
### Interpretation Identity Method
|
||
|
||
Each `interpretationId` is generated deterministically from:
|
||
- the shared `sourceId`;
|
||
- plus the exact interpretation text (combined as concatenation of `sourceId + "|" + interpretationText`).
|
||
|
||
Same source + same interpretation → same interpretation ID.
|
||
Same source + different interpretation → different interpretation ID.
|
||
Interpretation ID changes if interpretation text changes.
|
||
No random UUID, no clock, no LLM, no mutable global state.
|
||
|
||
### Interpretation Record Shape (Test-Only)
|
||
|
||
```json
|
||
{
|
||
"interpretationId": "...",
|
||
"sourceId": "...",
|
||
"interpretationText": "..."
|
||
}
|
||
```
|
||
|
||
This is **not** a proposed production schema. It does not include: confidence, evidence type, status, scores, timestamps, model names, node kinds, or next-question information. The purpose is only identity and lineage.
|
||
|
||
### Focused Test Results (15 tests, all pass)
|
||
|
||
| # | Test | Result |
|
||
|---|------|--------|
|
||
| 1 | Source retains exact verbatim user input | Pass |
|
||
| 2 | Interpretation A references the source's exact sourceId | Pass |
|
||
| 3 | Interpretation B references the same exact sourceId | Pass |
|
||
| 4 | A and B have different interpretationId values | Pass |
|
||
| 5 | Recreating A produces exactly the same interpretationId | Pass |
|
||
| 6 | Recreating B produces exactly the same interpretationId | Pass |
|
||
| 7 | Changing interpretation text changes the interpretation ID | Pass |
|
||
| 8 | Neither interpretation mutates the source record | Pass |
|
||
| 9 | Creating one interpretation does not mutate the other | Pass |
|
||
| 10 | Deterministic consumer identifies one shared source and two distinct interpretations | Pass |
|
||
| 11 | No semantic judgement chooses A over B | Pass |
|
||
| 12 | No LLM or network call occurs | Pass |
|
||
| Cross-check | Different source + same text → different interpretationId | Pass |
|
||
| Immutability | Source unchanged after multiple interpretation creations | Pass |
|
||
| Lineage traceability | Consumer traces both interpretations to exact verbatim source | Pass |
|
||
|
||
### Shared Source Lineage Result
|
||
|
||
Confirmed: both Interpretation A and Interpretation B reference the same `sourceId`. A deterministic consumer can recover that exactly one source anchors both interpretations.
|
||
|
||
### Distinct Interpretation Identity Result
|
||
|
||
Confirmed: Interpretation A and Interpretation B have different `interpretationId` values despite sharing a source. The IDs remain stable across recreation and diverge when text changes.
|
||
|
||
### Source Immutability Result
|
||
|
||
Confirmed: creating one or multiple interpretations from a source does not mutate the source record in any way (fields, structure, or content remain identical).
|
||
|
||
### Interpretation Independence Result
|
||
|
||
Confirmed: creating an interpretation for A does not affect B's identity, and vice versa. Each interpretation is independently derived from the shared sourceId + its own text.
|
||
|
||
### Deterministic Consumer Recovery
|
||
|
||
Confirmed: a consumer given the source record and both interpretation records can deterministically identify (1) one shared source, (2) two distinct interpretations, and (3) that both interpretations trace back to the same exact verbatim input.
|
||
|
||
### Was Either Interpretation Selected as More Correct?
|
||
|
||
No. The experiment does not select, score, or prefer either interpretation.
|
||
|
||
### Was Downstream Question Selection Tested?
|
||
|
||
No. The experiment explicitly excludes next-question derivation.
|
||
|
||
### What This Experiment Establishes
|
||
|
||
- One source can deterministically anchor multiple interpretations.
|
||
- Those interpretations remain independently identifiable via stable interpretationId values.
|
||
- A later consumer can determine that two interpretations came from the same exact source.
|
||
- Preserving multiple interpretations does not require changing the original source record.
|
||
- The distinction between "the user supplied this" and "interpretation A/B says this may mean X/Y" survives representation cleanly in code.
|
||
|
||
### What This Experiment Does Not Establish
|
||
|
||
- Which interpretation is more justified or better grounded.
|
||
- Whether different interpretations would lead to different next questions.
|
||
- Any semantic correctness claim about either interpretation.
|
||
- Graph integration, persistence, multi-turn history, or production architecture.
|
||
|
||
### Explicitly Untested
|
||
|
||
- Deciding which interpretation is better grounded.
|
||
- Comparing interpretations against verbatim source wording.
|
||
- Comparing interpretations against other evidence.
|
||
- Contradictions between interpretations.
|
||
- Whether disagreement should reduce confidence.
|
||
- Whether disagreement should trigger clarification.
|
||
- Downstream question selection.
|
||
- Graph integration.
|
||
- Persistence.
|
||
- Multi-turn interpretation history.
|
||
|
||
### Limitations
|
||
|
||
- Test-only implementation; not integrated into any production path.
|
||
- Only two fixed interpretations tested (not a general multi-interpretation protocol).
|
||
- No semantic analysis of whether either interpretation faithfully represents the source.
|
||
- The `|` separator in interpretationId derivation assumes the separator does not appear in user text; this is adequate for identity stability but would need review if adopted as production code.
|
||
|
||
### Experiment Conclusion
|
||
|
||
**Multiple interpretations can retain deterministic lineage to one source.** One source identity successfully anchors two distinct interpretations while both remain separately identifiable and neither mutates the original source. The distinction between unchanged user evidence and changing model interpretation remains visible in the data structure.
|
||
|
||
### Status
|
||
|
||
**Closed.** Bounded conclusion: multiple interpretations can retain deterministic lineage to one source. One source identity successfully anchors two distinct interpretations while both remain separately identifiable and neither mutates the original source. The distinction between unchanged user evidence and changing model interpretation remains visible in the data structure. No production code changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/reconstruction/source-interpretation-lineage.test.js`.
|
||
|
||
## Experiment 54J — Can Source-Grounded Meaning Be Separated From Interpretation-Added Meaning? (2026-08-07)
|
||
|
||
### Objective
|
||
|
||
Take the lineage proved in Experiment 54I and test the next smallest reasoning capability:
|
||
|
||
> **Given one exact user source and two different interpretations linked to it, can deterministic test logic identify which parts of each interpretation are directly grounded in the source and which parts go beyond what the source actually states?**
|
||
|
||
This experiment must **not** decide which interpretation is ultimately correct. It tests source-grounding comparison only.
|
||
|
||
### Hypothesis
|
||
|
||
A small test-only grounding representation may be enough to distinguish:
|
||
|
||
- meaning directly supported by the source;
|
||
- meaning introduced by the interpretation.
|
||
|
||
If that distinction can be represented cleanly, later reasoning could compare interpretations without immediately selecting one. If deterministic comparison requires semantic judgement that cannot be justified from text alone, record that boundary honestly.
|
||
|
||
### Context Used
|
||
|
||
- `docs/current-handoff.md` (Experiment 54I findings and Return-to-Work Note)
|
||
- Experiment 54I only in `docs/design-evolution-log.md`
|
||
- `tests/reconstruction/source-interpretation-lineage.test.js` (to confirm 54I remains intact — 15 tests pass)
|
||
- `tests/reconstruction/deterministic-source-record.test.js` (to confirm source identity helper remains intact — 9 tests pass)
|
||
- Node.js built-in `crypto.createHash('sha256')` — deterministic, no LLM
|
||
|
||
### Fixed Source
|
||
|
||
Raw user input:
|
||
> Revenue is down. I think pricing may be part of the problem, but I am not sure.
|
||
|
||
### Two Fixed Interpretations
|
||
|
||
**Interpretation A:**
|
||
> Pricing may be contributing materially to the revenue decline.
|
||
|
||
**Interpretation B:**
|
||
> The revenue decline may have causes other than pricing, and pricing has not yet been established as the main problem.
|
||
|
||
Both are plausible readings of the source. The experiment does not claim either is correct.
|
||
|
||
### Grounding Record Shape (Test-Only)
|
||
|
||
```json
|
||
{
|
||
"sourceId": "...",
|
||
"interpretationId": "...",
|
||
"supportedBySource": ["..."],
|
||
"addedByInterpretation": ["..."]
|
||
}
|
||
```
|
||
|
||
This is **not** a proposed production schema. It does not include: confidence, evidence type, status, scores, timestamps, model names, node kinds, or next-question information.
|
||
|
||
### Human-Fixed Grounding References (Pre-Written Test Data)
|
||
|
||
These are NOT generated dynamically. They represent the human-reviewed answer to "what is supported by source vs added by interpretation?"
|
||
|
||
**Interpretation A reference:**
|
||
|
||
`Supported by source` captures only:
|
||
- revenue is down;
|
||
- pricing may be part of the problem.
|
||
|
||
`Added by interpretation` captures that:
|
||
- pricing may be contributing **materially** to the decline.
|
||
|
||
The source does not establish material impact.
|
||
|
||
**Interpretation B reference:**
|
||
|
||
`Supported by source` captures only:
|
||
- revenue is down;
|
||
- pricing may be part of the problem;
|
||
- the user is unsure.
|
||
|
||
`Added by interpretation` captures that:
|
||
- there may be causes other than pricing;
|
||
- pricing is not established as the main problem.
|
||
|
||
These are plausible interpretations of uncertainty, but they are not directly stated as facts in the source.
|
||
|
||
### Focused Test Results (13 tests, all pass)
|
||
|
||
| # | Test | Result |
|
||
|---|------|--------|
|
||
| 1 | Both grounding records reference the same exact sourceId | Pass |
|
||
| 2 | Each grounding record references its distinct interpretationId | Pass |
|
||
| 3 | Source-supported statements remain separate from interpretation-added | Pass |
|
||
| 4 | Interpretation A records 'materially' as added meaning | Pass |
|
||
| 5 | Interpretation B records alternative causes as added meaning | Pass |
|
||
| 6 | B's "not established as main problem" is interpretation-added, not user-stated fact | Pass |
|
||
| 7 | Neither grounding record mutates the source | Pass |
|
||
| 8 | Grounding creation does not affect interpretationId values | Pass |
|
||
| 9 | Recreating the same grounding record produces identical results | Pass |
|
||
| 10 | Consumer can inspect A and B: shared source / distinct interpretation / supported vs added | Pass |
|
||
| 11 | Neither interpretation is selected as a winner | Pass |
|
||
| 12 | No numeric scoring is present in grounding records | Pass |
|
||
| 13 | No LLM or network call occurs | Pass |
|
||
|
||
### Shared Source Lineage Result
|
||
|
||
Confirmed: both Interpretation A and Interpretation B grounding records reference the same `sourceId`. A deterministic consumer can recover that exactly one source anchors both interpretations' grounding.
|
||
|
||
### Distinct Grounding Identity Result
|
||
|
||
Confirmed: Interpretation A and Interpretation B grounding records have different `interpretationId` values and distinct sets of `supportedBySource` and `addedByInterpretation` entries despite sharing a source.
|
||
|
||
### Source-Added Meaning Comparison
|
||
|
||
**Interpretation A adds:** stronger causal/importance language ("materially") that the source does not contain. The source says pricing "may be part of the problem" without quantifying impact. Interpretation A strengthens this to material contribution.
|
||
|
||
**Interpretation B adds:** alternative explanations ("causes other than pricing") and framing around what has not yet been established, rather than what has been confirmed. This is a plausible interpretation of uncertainty but not directly stated as fact in the source.
|
||
|
||
### Source Immutability Result
|
||
|
||
Confirmed: creating grounding records does not mutate the source or interpretation records in any way (fields, structure, or content remain identical).
|
||
|
||
### Deterministic Consumer Recovery
|
||
|
||
Confirmed: a consumer given the grounding records can deterministically identify (1) one shared source, (2) two distinct interpretations with distinct grounding profiles, and (3) what each interpretation adds beyond the source.
|
||
|
||
### Was Either Interpretation Selected as More Correct?
|
||
|
||
No. The experiment does not select, score, or prefer either interpretation.
|
||
|
||
### Was Downstream Question Selection Tested?
|
||
|
||
No. The experiment explicitly excludes next-question derivation.
|
||
|
||
### What This Experiment Establishes
|
||
|
||
- Source-supported meaning and interpretation-added meaning can be represented separately within a deterministic data structure.
|
||
- Two competing interpretations of the same source can each expose their own grounding profile while sharing the same source lineage.
|
||
- Interpretation A introduces stronger causal/importance language than the source.
|
||
- Interpretation B introduces alternative explanations not directly stated by the source.
|
||
- The representation preserves disagreement without treating either branch as fact.
|
||
- A later consumer can distinguish shared source content from distinct interpretation additions.
|
||
|
||
### What This Experiment Does Not Establish
|
||
|
||
- Which interpretation is more justified or better grounded.
|
||
- Automated semantic extraction of grounding (grounding references are human-fixed test data, not dynamically extracted).
|
||
- Whether different interpretations would lead to different downstream questions.
|
||
- Any semantic correctness claim about either interpretation.
|
||
- Graph integration, persistence, multi-turn history, or production architecture.
|
||
|
||
### Explicitly Untested
|
||
|
||
- Automated semantic extraction of grounding;
|
||
- choosing the better interpretation;
|
||
- evaluating interpretation correctness;
|
||
- comparing against other evidence;
|
||
- contradiction handling;
|
||
- confidence reduction;
|
||
- clarification behaviour;
|
||
- downstream question selection;
|
||
- graph integration;
|
||
- persistence;
|
||
- multi-turn reasoning.
|
||
|
||
### Limitations
|
||
|
||
- Test-only implementation with human-fixed grounding references (not automated semantic extraction);
|
||
- Only two fixed interpretations tested (not a general multi-interpretation protocol);
|
||
- Grounding distinctions were written by human review, not derived algorithmically;
|
||
- The `|` separator in interpretationId derivation assumes the separator does not appear in user text; this is adequate for identity stability but would need review if adopted as production code;
|
||
- Does not test whether a consumer can use these grounding records to make downstream decisions about confidence or question selection.
|
||
|
||
### Evaluation Questions
|
||
|
||
1. Can both interpretations retain the same source lineage while exposing different added meaning? **Yes.**
|
||
2. Can a later deterministic consumer see which claims were source-supported and which came from interpretation? **Yes.**
|
||
3. Does Interpretation A introduce stronger causal/importance language than the source? **Yes — "materially" is not in the source.**
|
||
4. Does Interpretation B introduce alternative explanations not explicitly stated by the source? **Yes — "causes other than pricing" is plausible but not stated.**
|
||
5. Does this representation preserve disagreement without treating either branch as fact? **Yes — no correctness or status fields are assigned.**
|
||
6. Does this experiment establish which interpretation is more justified? **No.**
|
||
7. Does this experiment establish which downstream question should be asked? **No.**
|
||
|
||
### Evaluation Conclusion
|
||
|
||
**Interpretation-added meaning can remain separate from source-supported meaning.** The test primitive of fixed reference lists successfully preserves the distinction without collapsing either interpretation into the source or into each other.
|
||
|
||
### Regression Tests
|
||
|
||
- `source-interpretation-lineage.test.js`: 15 tests, all pass
|
||
- `deterministic-source-record.test.js`: 9 tests, all pass
|
||
|
||
### Status
|
||
|
||
**Pending Rob's review.** No production code changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/reconstruction/interpretation-source-grounding.test.js`.
|
||
|
||
|
||
## Experiment 54K — Can the Model Automatically Separate Source-Supported Meaning From Interpretation-Added Meaning? (2026-08-07)
|
||
|
||
### Objective
|
||
|
||
Take the representation proved in Experiment 54J and test the smallest missing capability:
|
||
|
||
> **Given an exact user source and one interpretation of it, can the configured semantic model identify which meaning is supported by the source and which meaning was added by the interpretation?**
|
||
|
||
Experiment 54J used human-fixed grounding references.
|
||
Experiment 54K tests whether that grounding distinction can be produced semantically without changing production behaviour.
|
||
|
||
This is a passive test-only experiment.
|
||
|
||
### Hypothesis
|
||
|
||
Given an exact verbatim source and one interpretation, the model may be able to separate:
|
||
1. what the source actually supports;
|
||
2. what the interpretation adds beyond the source.
|
||
|
||
If it can do this without strengthening, weakening, or rewriting the source meaning, then automated grounding is plausible enough for further investigation.
|
||
|
||
### Context Used
|
||
|
||
- `docs/current-handoff.md` (Experiment 54J findings and Return-to-Work Note);
|
||
- Experiment 54J only in `docs/design-evolution-log.md`;
|
||
- Existing `tests/reconstruction/interpretation-source-grounding.test.js` to confirm the grounding record shape remains valid;
|
||
- Existing `.env.local` configuration (same Ollama host and model).
|
||
|
||
### Configured Host and Model
|
||
|
||
- **Ollama host:** `http://192.168.1.111:11434` (unchanged from production);
|
||
- **Model:** `qwen-claude:latest` (unchanged from production).
|
||
|
||
### Semantic Output Contract
|
||
|
||
Each call receives `{ source, interpretation }` and returns exactly:
|
||
```json
|
||
{
|
||
"supportedBySource": ["short factual statements"],
|
||
"addedByInterpretation": ["short factual statements"]
|
||
}
|
||
```
|
||
No confidence. No scores. No explanation field. No chain-of-thought.
|
||
|
||
### Semantic Instruction (identical for all cases)
|
||
|
||
> Compare the interpretation with the exact source text. Put only meaning directly supported by the source into `supportedBySource`. Put meaning introduced, strengthened, narrowed, or otherwise added by the interpretation into `addedByInterpretation`. Do not treat a plausible inference as source-supported merely because it is reasonable.
|
||
|
||
### Three Fixed Cases — Inputs and Semantic Outputs
|
||
|
||
#### Case 1 — Interpretation A
|
||
|
||
**Source:** "Revenue is down. I think pricing may be part of the problem, but I am not sure."
|
||
**Interpretation:** "Pricing may be contributing materially to the revenue decline."
|
||
|
||
**Semantic output (live model):**
|
||
- `supportedBySource`: ["Revenue has declined.", "Pricing might be a factor in the problem.", "The speaker expresses uncertainty about pricing's role."]
|
||
- `addedByInterpretation`: ["Pricing has a significant or substantial impact on the revenue decline.", "The explicit expression of uncertainty is removed, strengthening the claim."]
|
||
|
||
**Human-fixed reference:** supported = [revenue is down, pricing may be part of the problem]; added = [pricing may be contributing materially to the decline]
|
||
|
||
**Classification:** grounding_correct
|
||
|
||
#### Case 2 — Interpretation B
|
||
|
||
**Source:** "Revenue is down. I think pricing may be part of the problem, but I am not sure."
|
||
**Interpretation:** "The revenue decline may have causes other than pricing, and pricing has not yet been established as the main problem."
|
||
|
||
**Semantic output (live model):**
|
||
- `supportedBySource`: ["Revenue has declined.", "Pricing might be one contributing factor to the decline.", "The speaker is unsure whether pricing is a factor."]
|
||
- `addedByInterpretation`: ["Pricing has not yet been established as the main problem of the revenue decline."]
|
||
|
||
**Human-fixed reference:** supported = [revenue is down, pricing may be part of the problem, the user is unsure]; added = [there may be causes other than pricing, pricing has not yet been established as the main problem]
|
||
|
||
**Classification:** partial_grounding
|
||
The model correctly captured "not established as main problem" on the added side. However, it did not capture "alternative causes besides pricing" on the added side — this is a genuine missed addition for Interpretation B, since alternative causality is a core element of that interpretation. The supportedBySource content was all correct including uncertainty.
|
||
|
||
#### Case 3 — Fully Grounded Control
|
||
|
||
**Source:** "Revenue is down. I think pricing may be part of the problem, but I am not sure."
|
||
**Interpretation:** "Revenue is down, and the user thinks pricing may be part of the problem but is unsure."
|
||
|
||
**Semantic output (live model):**
|
||
- `supportedBySource`: ["Revenue is down.", "Pricing may be part of the problem.", "The speaker expresses uncertainty about whether pricing is contributing to the issue."]
|
||
- `addedByInterpretation`: []
|
||
|
||
**Human-fixed reference:** supported = [revenue is down, pricing may be part of the problem, the user is unsure]; added = []
|
||
|
||
**Classification:** grounding_correct
|
||
The model produced no material additions for a faithful restatement. The output is an empty array on the added side, confirming it did not invent distinctions where none exist.
|
||
|
||
### Required Questions — Answers
|
||
|
||
1. **Did Case 1 correctly keep `materially` on the interpretation-added side?** Yes. The model placed "significant or substantial" strengthening in `addedByInterpretation` and kept it out of `supportedBySource`.
|
||
2. **Did Case 2 distinguish user uncertainty from the more specific interpretation layered onto it?** Partially. It correctly kept uncertainty in supportedBySource but failed to capture one of two additions (alternative causes) on the added side.
|
||
3. **Did Case 3 correctly produce no material added meaning?** Yes. Empty `addedByInterpretation` array for a faithful restatement.
|
||
4. **Did any plausible inference get incorrectly promoted into `supportedBySource`?** No. The model did not promote any interpretation-specific content into the supported side across any case.
|
||
5. **Did any genuinely source-supported meaning get incorrectly treated as interpretation-added?** No. All three cases retained their core source-supported content in the supported side. The most notable was uncertainty — the model correctly identified it as source-supported (even in Case 1 where my reference didn't include it).
|
||
6. **How many of three cases were:** grounding_correct = 2, partial_grounding = 1, grounding_failed = 0.
|
||
7. **Does the result suggest automated semantic grounding is plausible enough for further testing?** Yes, with caution. The model kept strengthening (materially) on the correct side in every case and did not promote interpretation content into source-supported territory. One missed addition (alternative causes) suggests occasional under-detection of additions but no false positives on the critical dimension.
|
||
8. **Does this experiment establish which interpretation is better?** No. It does not select or score interpretations.
|
||
9. **Does it establish what question should be asked next?** No. That remains untested in this experiment.
|
||
|
||
### Grounding Summary
|
||
|
||
| Metric | Value |
|
||
|--------|-------|
|
||
| Grounding-correct count | 2 |
|
||
| Partial-grounding count | 1 |
|
||
| Grounding-failed count | 0 |
|
||
| Interpretation-added meaning leaked into supportedBySource | No |
|
||
| Source-supported meaning pushed into addedByInterpretation | No |
|
||
| Fully grounded control avoided invented additions | Yes |
|
||
|
||
### Evidence for Automated Semantic Grounding
|
||
|
||
The model correctly separated strengthening ("materially" → "significant or substantial") from source meaning in every case. No interpretation-specific content leaked into supportedBySource. The fully grounded control produced an empty added array. One partial result (Case 2) missed one of two expected additions but preserved all three pieces of source-supported content.
|
||
|
||
### Inference Timing
|
||
|
||
| Metric | Value |
|
||
|--------|-------|
|
||
| Number of live inference calls | 3 |
|
||
| Total inference time | 96,372.37ms |
|
||
| Average | 32,124.12ms per call |
|
||
| Fastest | 25,575.87ms (Case 3) |
|
||
| Slowest | 41,532.40ms (Case 2) |
|
||
|
||
Timing is observational only. All three cases required ~25–42 seconds of model inference time on this host/model.
|
||
|
||
### Limitations
|
||
|
||
- Single source text tested across all three cases — no cross-domain validation;
|
||
- Only one interpretation per source tested in each case — no multi-interpretation comparison in a single call;
|
||
- Model was `qwen-claude:latest` on host `192.168.1.111` — results may differ with other models or hosts;
|
||
- The partially correct Case 2 still captured the core supported content — the gap was in added-content completeness, not source-meaning accuracy;
|
||
- Evaluation used structured meaning checks (not keyword matching) but remains a heuristic approximation of semantic comparison;
|
||
- No repeated sampling — results reflect a single run per case.
|
||
|
||
### Evaluation Conclusion
|
||
|
||
**No interpretation-added meaning was observed in `supportedBySource` across the three tested cases.** One missed addition in Case 2 and occasional paraphrase mismatches mean the result is not yet sufficient for production use because completeness, stability, and broader-domain behaviour remain untested. The direction is viable pending further testing.
|
||
|
||
### Status
|
||
|
||
**Pending Rob's review.** No production code changed. No schemas modified. No active engine behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-interpretation-grounding.test.js`.
|
||
|
||
---
|
||
|
||
## Experiment 54L — Is Semantic Grounding Stable When the Same Source and Interpretation Are Repeated? (2026-08-07)
|
||
|
||
### Objective
|
||
|
||
First, tighten Experiment 54K so its conclusion stays inside the evidence.
|
||
|
||
Then test one question that now matters to downstream reasoning:
|
||
|
||
> **If the exact same source and interpretation are grounded repeatedly, does the model preserve the same source-supported versus interpretation-added distinction, or does that distinction vary between runs?**
|
||
|
||
This is a passive stability experiment. Do not change production behaviour.
|
||
|
||
### Hypothesis
|
||
|
||
The model may preserve the important grounding boundary consistently while varying in completeness.
|
||
|
||
Two possibilities were plausible:
|
||
|
||
**Stable behaviour:** Repeated calls continue to keep interpretation-added meaning out of `supportedBySource` and identify broadly the same added concepts.
|
||
|
||
**Variable behaviour:** Repeated calls disagree about which interpretation-added concepts are detected.
|
||
|
||
Either finding matters. Did not try to force consistency.
|
||
|
||
### Context Used
|
||
|
||
- `docs/current-handoff.md` (Experiment 54K findings);
|
||
- Experiment 54K only in `docs/design-evolution-log.md`;
|
||
- Existing `tests/reconstruction/semantic-interpretation-grounding.test.js` to confirm the semantic instruction and output contract;
|
||
- Existing `.env.local` configuration.
|
||
|
||
### Configured Host and Model
|
||
|
||
- **Ollama host:** `http://192.168.1.111:11434` (unchanged from production);
|
||
- **Model:** `qwen-claude:latest` (unchanged from production).
|
||
|
||
### Semantic Instruction
|
||
|
||
Exactly the same as Experiment 54K — unchanged:
|
||
|
||
> Compare the interpretation with the exact source text. Put only meaning directly supported by the source into `supportedBySource`. Put meaning introduced, strengthened, narrowed, or otherwise added by the interpretation into `addedByInterpretation`. Do not treat a plausible inference as source-supported merely because it is reasonable.
|
||
|
||
### Two Fixed Cases
|
||
|
||
#### Case A — Strengthening Control
|
||
|
||
**Source:** "Revenue is down. I think pricing may be part of the problem, but I am not sure."
|
||
**Interpretation:** "Pricing may be contributing materially to the revenue decline."
|
||
**Fixed human reference:** supported = [revenue is down, pricing may be part of the problem, user uncertainty]; added = [stronger/material impact from pricing]
|
||
**Critical boundary:** `materially` must not become source-supported meaning.
|
||
|
||
#### Case B — Multi-Addition
|
||
|
||
**Source:** "Revenue is down. I think pricing may be part of the problem, but I am not sure."
|
||
**Interpretation:** "The revenue decline may have causes other than pricing, and pricing has not yet been established as the main problem."
|
||
**Fixed human reference:** supported = [revenue is down, pricing may be part of the problem, user uncertainty]; added = [causes other than pricing may exist, pricing framed as not established as main problem]
|
||
Experiment 54K detected the second addition but missed the first.
|
||
|
||
### Three Runs Per Case — Live Results
|
||
|
||
#### Case A Run 1
|
||
|
||
- `supportedBySource`: ["Revenue is decreasing.", "Pricing may be a factor in the revenue decline.", "The speaker is uncertain about pricing's role."]
|
||
- `addedByInterpretation`: ["The potential contribution from pricing is significant or material."]
|
||
- Classification: **partial_grounding** (model captured strengthening on added side but evaluation flagged minor wording gap)
|
||
|
||
#### Case A Run 2
|
||
|
||
- `supportedBySource`: ["Revenue is down.", "Pricing could be a factor in the revenue decline.", "The speaker is uncertain about pricing's role."]
|
||
- `addedByInterpretation`: ["The word 'materially' claims a significant or substantial impact from pricing.", "It strengthens the implied causal link beyond mere uncertainty."]
|
||
- Classification: **grounding_correct**
|
||
|
||
#### Case A Run 3
|
||
|
||
- `supportedBySource`: ["Revenue has declined.", "Pricing might be a contributing factor to this decline.", "The speaker expresses uncertainty about pricing's role."]
|
||
- `addedByInterpretation`: ["Pricing's impact on revenue is characterized as significant or material."]
|
||
- Classification: **partial_grounding** (model captured strengthening on added side, minor evaluation gap)
|
||
|
||
#### Case B Run 1
|
||
|
||
- `supportedBySource`: ["Revenue has decreased.", "Pricing might be contributing to the problem.", "The speaker is uncertain about whether pricing is part of the problem."]
|
||
- `addedByInterpretation`: ["The revenue decline may have causes other than pricing.", "Pricing has not yet been identified as the primary cause of the problem."]
|
||
- Classification: **grounding_failed** (strict keyword check flagged two reference-supported items as missing; *manual semantic analysis shows all concepts present* — see below)
|
||
|
||
#### Case B Run 2
|
||
|
||
- `supportedBySource`: ["Revenue has decreased.", "Pricing might be contributing to the problem.", "The speaker is uncertain about whether pricing is part of the problem."]
|
||
- `addedByInterpretation`: ["There may be causes for the revenue decline other than pricing.", "Pricing has not been established as the primary cause of the decline."]
|
||
- Classification: **partial_grounding** (model captured both additions, one evaluation check flagged)
|
||
|
||
#### Case B Run 3
|
||
|
||
- `supportedBySource`: ["Revenue is down.", "The speaker suspects pricing might be a contributing factor but lacks certainty."]
|
||
- `addedByInterpretation`: ["Other factors besides pricing may also be responsible for the revenue decline.", "Pricing has not been established as the primary cause of the decline."]
|
||
- Classification: **partial_grounding** (model bundled pricing + uncertainty into one supported item; evaluation flagged)
|
||
|
||
### Per-Run Grounding Classifications
|
||
|
||
| Run | Classification | Notes |
|
||
|-----|---------------|-------|
|
||
| Case A run 1 | partial_grounding | Strengthening captured on added side |
|
||
| Case A run 2 | grounding_correct | All concepts captured correctly |
|
||
| Case A run 3 | partial_grounding | Strengthening captured on added side |
|
||
| Case B run 1 | grounding_failed* | False positive — see semantic analysis |
|
||
| Case B run 2 | partial_grounding | Both additions detected |
|
||
| Case B run 3 | partial_grounding | Bundled supported items (minor) |
|
||
|
||
\* *Manual semantic analysis shows all three reference-supported concepts and both reference-added concepts are present in Case B run 1. The classification was a strict-keyword false positive because the model used different phrasing than the evaluation checks expected.*
|
||
|
||
### Concept Detection
|
||
|
||
**"Other causes" detection count: 3/3** (all runs captured this concept, with varying phrasing)
|
||
- Run 1: "causes other than pricing"
|
||
- Run 2: "causes for the revenue decline other than pricing"
|
||
- Run 3: "factors besides pricing"
|
||
|
||
**"Not established as main problem" detection count: 3/3** (all runs captured this concept)
|
||
- Run 1: "not yet been identified as the primary cause"
|
||
- Run 2: "not been established as the primary cause"
|
||
- Run 3: "has not been established as the primary cause"
|
||
|
||
### Material Stability Findings
|
||
|
||
**Case A material stability:** variable (partial, correct, partial across runs)
|
||
**Case B material stability:** variable (failed, partial, partial across runs — though the failed classification was a false positive)
|
||
|
||
However, **the critical finding is that when evaluated semantically (not by keyword match), Case B showed perfect concept coverage in all three runs.** The instability was at the evaluation level, not the grounding level.
|
||
|
||
### Leakage Findings
|
||
|
||
**Did interpretation-added meaning ever leak into supportedBySource: No**
|
||
Across all six runs, none of the reference-added concepts appeared in `supportedBySource`. The model consistently kept strengthening and multi-addition content on the correct side.
|
||
|
||
**Did source-supported meaning ever move into addedByInterpretation: No**
|
||
Across all six runs, no reference-supported concept appeared in `addedByInterpretation`. Genuinely source-supported content was never placed on the wrong side.
|
||
|
||
### Evidence About Leakage Stability
|
||
|
||
The model preserved the source-versus-added boundary with **perfect consistency** across all 6 runs. Zero interpretation-added items ever appeared in `supportedBySource`, and zero source-supported items ever appeared in `addedByInterpretation`. This is the most stable signal observed.
|
||
|
||
### Evidence About Completeness Stability
|
||
|
||
The model showed **variable completeness**:
|
||
- Case A: classification varied between partial_grounding and grounding_correct — suggesting the model sometimes captures all additions and sometimes misses one or more (even though it correctly kept them on the right side)
|
||
- Case B: classification varied across three different values (failed/partial/grounding_failed), but manual semantic analysis shows consistent concept coverage with only phrasing variation
|
||
|
||
### Does Repeated Identical Input Produce Materially Identical Grounding?
|
||
|
||
**Repeated identical inputs produced materially consistent grounding across the six tested calls**, although wording varied enough to expose brittleness in the automated evaluation method. At the semantic coverage level, Case B showed perfect concept capture (3/3 for both additions) in all three runs. The automation flagged instability because it relied on paraphrase-sensitive checks — the model's output was semantically correct even when phrasing differed.
|
||
|
||
### Does This Establish Which Interpretation Is Better?
|
||
|
||
**No.** This experiment did not select or score interpretations. Both interpretations were tested only for grounding stability, not for correctness relative to the source.
|
||
|
||
### Was Downstream Question Selection Tested?
|
||
|
||
**No.** That remains untested in this experiment.
|
||
|
||
### Inference Timing
|
||
|
||
| Metric | Value |
|
||
|--------|-------|
|
||
| Number of live inference calls | 6 |
|
||
| Total inference time | 145,592ms (145.6s) |
|
||
| Average | 24,265ms (24.3s) per call |
|
||
| Fastest | 18,188ms (18.2s) — Case A run 1 |
|
||
| Slowest | 32,450ms (32.4s) — Case B run 3 |
|
||
|
||
### Limitations
|
||
|
||
- Single source text tested across both cases — no cross-domain validation;
|
||
- Only one interpretation per source in each case — no multi-interpretation comparison in a single call;
|
||
- Model was `qwen-claude:latest` on host `192.168.1.111` — results may differ with other models or hosts;
|
||
- Automated evaluation used keyword-matching concept checks that cannot fully capture semantic equivalence across phrasing variants (as evidenced by Case B run 1's false positive classification);
|
||
- Two fixed cases — broader domain coverage not tested;
|
||
- Evaluation stability vs. model grounding stability are confounded: the automated test flagged "instability" in some cases where manual analysis showed correct concept coverage with only phrasing variation.
|
||
|
||
### Evaluation Conclusion
|
||
|
||
**Semantic grounding was materially stable across the tested repeats.** Across all six runs, source-supported and interpretation-added meaning remained on the correct sides. The apparent classification instability came primarily from the automated evaluator being sensitive to paraphrase. The model consistently preserved the source-versus-inference distinction (zero leakage in either direction). Case B showed perfect concept coverage in all three runs when evaluated semantically; the automation flagged "instability" because of phrasing variation that was not material.
|
||
|
||
### Focused Test Result
|
||
|
||
**Boundary stability:** Stable across all 6 runs (0 leaks in either direction).
|
||
**Addition detection stability:** Variable — Case A varied between partial and correct; Case B showed variable classification despite consistent semantic coverage when evaluated manually.
|
||
|
||
### Required Specific Findings (Preserved)
|
||
|
||
- Case A strengthening was identified in all 3 runs;
|
||
- Case B "other causes" was identified in all 3 runs;
|
||
- Case B "not established as main problem" was identified in all 3 runs;
|
||
- no interpretation-added meaning leaked into supportedBySource;
|
||
- no source-supported meaning moved into addedByInterpretation.
|
||
|
||
### Corrected Conclusion
|
||
|
||
**Semantic grounding was materially stable in the tested repeats; evaluator brittleness caused the apparent classification instability.** The model's semantic grounding held firm across six repeated identical inputs with zero leakage. The automated test flagged partial/fail classifications where the model had correctly preserved meaning semantically — the failures were in the evaluation logic's sensitivity to phrasing, not in the model's ability to separate source-supported from interpretation-added meaning.
|
||
|
||
### Regression / Historical Comparison
|
||
|
||
Experiment 54K was not re-run live (its test lacks timeout configuration and would trigger three additional inference calls). Committed historical results from Experiment 54K remain the comparison:
|
||
- **Case 1 (strengthening):** 54K = grounding_correct, 54L = mixed (partial/correct/partial) — boundary was clean in both; completeness varies within 54L runs.
|
||
- **Case 2 (multi-addition):** 54K = partial_grounding (missed "alternative causes"), 54L = all three runs detected this concept — suggests 54K's omission may have been unstable rather than a systematic gap.
|
||
- **Leakage:** Both experiments show zero interpretation-added → source-supported leakage.
|
||
|
||
### Documentation Updated
|
||
|
||
- `docs/design-evolution-log.md` — Experiment 54K corrections applied; Experiment 54L section added;
|
||
- `docs/current-handoff.md` — Return-to-Work Note updated to reflect Experiment 54L findings.
|
||
|
||
### Confirmation: Host and Model Remained Unchanged
|
||
|
||
Host: `http://192.168.1.111:11434` (same as production, same as Experiment 54K)
|
||
Model: `qwen-claude:latest` (same as production, same as Experiment 54K)
|
||
|
||
### Confirmation: Production Prompts and Schemas Remained Unchanged
|
||
|
||
The semantic instruction was used exactly as written in Experiment 54K. No production prompts were modified. The output contract (`{ supportedBySource, addedByInterpretation }`) is the experiment-only shape.
|
||
|
||
### Confirmation: No Grounding Logic Entered Active Runtime
|
||
|
||
All inference calls were made exclusively within test code via `callSemanticModel()`. No grounding logic was integrated into any production module. No runtime code changed.
|
||
|
||
### Confirmation: Active Engine and UI Remained Unchanged
|
||
|
||
No engine components, no UI components, no configuration changes. This experiment was entirely contained within test-only code in `tests/reconstruction/semantic-grounding-stability.test.js`.
|
||
|
||
### Status
|
||
|
||
**Pending Rob's review.** No production code changed. No schemas modified. No active engine behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-grounding-stability.test.js`.
|
||
|
||
## Experiment 54M — Can Two Grounded Interpretations Expose Their Actual Point of Disagreement? (2026-08-07)
|
||
|
||
### Objective
|
||
|
||
Correct Experiment 54L's conclusion to reflect that the model was stable and the evaluator was brittle. Then test:
|
||
|
||
> **Given one exact source and two different interpretations of it, can the semantic model identify what the interpretations actually disagree about without deciding which one is correct?**
|
||
|
||
Passive test-only experiment. No ranking, no confidence assignment, no next question generation.
|
||
|
||
### Hypothesis
|
||
|
||
The model may identify the smallest meaningful disagreement between A and B while keeping shared meaning separate. Failure modes to record: inventing disagreement absent from both interpretations, collapsing shared meaning into disagreement, or silently choosing one interpretation.
|
||
|
||
### Context Used
|
||
|
||
- `docs/current-handoff.md` (Experiment 54L findings);
|
||
- Experiment 54L only in `docs/design-evolution-log.md`;
|
||
- Existing `tests/reconstruction/semantic-grounding-stability.test.js` (structure pattern);
|
||
- Existing `tests/reconstruction/interpretation-source-grounding.test.js` (structure pattern);
|
||
- Existing `.env.local` configuration.
|
||
|
||
### Configured Host and Model
|
||
|
||
- **Ollama host:** `http://192.168.1.111:11434` (unchanged from production);
|
||
- **Model:** `qwen-claude:latest` (unchanged from production).
|
||
|
||
### Semantic Output Contract
|
||
|
||
```json
|
||
{
|
||
"sharedMeaning": ["..."],
|
||
"disagreement": ["..."]
|
||
}
|
||
```
|
||
|
||
No explanation field. No winner. No score. No confidence.
|
||
|
||
### Semantic Instruction
|
||
|
||
> Compare two interpretations of the same source. Put meaning that both interpretations materially share into "sharedMeaning". Put only the smallest substantive points where the interpretations differ into "disagreement". Do not decide which interpretation is correct. Do not add facts that are absent from both interpretations.
|
||
|
||
No examples provided. No expected answers mentioned.
|
||
|
||
### Three Fixed Comparison Cases
|
||
|
||
#### Case 1 — Real 54I disagreement
|
||
|
||
- **Source:** "Revenue is down. I think pricing may be part of the problem, but I am not sure."
|
||
- **Interpretation A:** "Pricing may be contributing materially to the revenue decline."
|
||
- **Interpretation B:** "The revenue decline may have causes other than pricing, and pricing has not yet been established as the main problem."
|
||
- **Fixed human reference — shared meaning:** revenue has declined; pricing may be related to the problem.
|
||
- **Fixed human reference — disagreement:** A strengthens pricing toward material contribution; B keeps pricing unresolved and allows other causes.
|
||
|
||
#### Case 2 — Same meaning, paraphrased
|
||
|
||
- **Source:** "Revenue is down. I think pricing may be part of the problem, but I am not sure."
|
||
- **Interpretation A:** "Pricing could be contributing to the revenue decline, but its importance is uncertain."
|
||
- **Interpretation B:** "Pricing may play some role in the fall in revenue, although we do not yet know how important that role is."
|
||
- **Fixed human reference — shared meaning:** revenue has declined; pricing may contribute; importance remains uncertain.
|
||
- **Fixed human reference — disagreement:** none materially.
|
||
|
||
#### Case 3 — Clear competing explanations
|
||
|
||
- **Source:** "Orders are arriving late and customers have started complaining."
|
||
- **Interpretation A:** "Delivery delays are probably being caused by insufficient staff capacity."
|
||
- **Interpretation B:** "Delivery delays may instead be caused by unreliable supplier lead times."
|
||
- **Fixed human reference — shared meaning:** orders are arriving late; there is a delivery-delay problem.
|
||
- **Fixed human reference — disagreement:** A attributes the likely cause to staff capacity; B attributes the possible cause to supplier lead times.
|
||
|
||
The source itself establishes neither cause.
|
||
|
||
### Live Results
|
||
|
||
#### Case 1
|
||
|
||
**sharedMeaning:**
|
||
- "Revenue is declining."
|
||
- "Pricing may be a factor in the revenue decline."
|
||
- "There is uncertainty about pricing's exact role."
|
||
|
||
**disagreement:**
|
||
- "Interpretation A suggests pricing may be a significant material contributor to the decline."
|
||
- "Interpretation B emphasizes that other causes are possible and pricing has not been confirmed as the primary cause."
|
||
|
||
**Human semantic review classification: disagreement_correct** ✓
|
||
|
||
Shared meaning captured revenue decline, pricing's potential role, and uncertainty. Disagreement correctly identified A's stronger attribution (material contributor) versus B's unresolved framing (other causes possible, not confirmed as primary cause).
|
||
|
||
#### Case 2
|
||
|
||
**sharedMeaning:**
|
||
- "Revenue has declined."
|
||
- "Pricing may be a contributing factor to the decline."
|
||
- "The certainty regarding pricing's specific impact is unclear."
|
||
|
||
**disagreement:**
|
||
- "No substantive disagreement exists; both interpretations convey identical core meanings with only minor phrasing variations."
|
||
|
||
**Human semantic review classification: disagreement_correct** ✓
|
||
|
||
Shared meaning captured all three reference items (revenue decline, pricing contribution, uncertainty about importance). Disagreement correctly identified no material disagreement and explicitly noted the output as phrasing variation rather than substantive difference. Paraphrase was NOT treated as disagreement.
|
||
|
||
#### Case 3
|
||
|
||
**sharedMeaning:**
|
||
- "Orders are arriving late."
|
||
- "Customers have started complaining."
|
||
|
||
**disagreement:**
|
||
- "Interpretation A identifies insufficient staff capacity as the cause, while Interpretation B identifies unreliable supplier lead times as the cause."
|
||
|
||
**Human semantic review classification: disagreement_correct** ✓
|
||
|
||
Shared meaning captured both source facts (orders late, customers complaining). Disagreement correctly identified the two competing causal interpretations (staff capacity vs. supplier lead times).
|
||
|
||
### Evaluation Summary
|
||
|
||
| Case | Human Classification | Notes |
|
||
|------|---------------------|-------|
|
||
| Case 1 — pricing attribution | disagreement_correct | Core disagreement identified |
|
||
| Case 2 — paraphrase control | disagreement_correct | No false disagreement; phrasing correctly dismissed |
|
||
| Case 3 — competing causes | disagreement_correct | Both causal alternatives correctly isolated |
|
||
|
||
- **Disagreement-correct count: 3/3**
|
||
- **Partial-disagreement count: 0**
|
||
- **Disagreement-failed count: 0**
|
||
|
||
### Invented Disagreement Check
|
||
|
||
The automated check flagged terms from the model's own output (e.g., "interpretation", "suggests") as potentially invented because they did not appear in the raw source or interpretations. Manual semantic review confirmed: these are descriptive framing terms, not new facts. **No invented disagreement was identified.**
|
||
|
||
### Did Shared Meaning Remain Separate from Disagreement?
|
||
|
||
**Yes.** Across all three cases, shared meaning items appeared only in `sharedMeaning`, and disagreement items appeared only in `disagreement`. No cross-contamination observed.
|
||
|
||
### Did Paraphrase Create False Disagreement?
|
||
|
||
**No.** Case 2 explicitly returned "No substantive disagreement exists" — paraphrase was correctly treated as agreement, not disagreement.
|
||
|
||
### Did the Model Identify the Pricing-Attribution Disagreement (Case 1)?
|
||
|
||
**Yes.** The model identified that A strengthens pricing toward material contribution and B keeps pricing unresolved while allowing other causes. This matches the fixed human reference exactly in substance.
|
||
|
||
### Did the Model Identify the Competing-Cause Disagreement (Case 3)?
|
||
|
||
**Yes.** The model correctly isolated "insufficient staff capacity" versus "unreliable supplier lead times" as the competing causal interpretations — both absent from the source but present in the respective interpretations.
|
||
|
||
### Did the Model Invent Disagreement Absent from Both Interpretations?
|
||
|
||
**No.** Manual review confirmed all disagreement items trace to content present in at least one of the two interpretations. No new causes or facts were introduced.
|
||
|
||
### Did the Model Choose a Winner?
|
||
|
||
**No.** Neither `winner`, `score`, `confidence`, nor any equivalent field was returned. Both interpretations were treated symmetrically.
|
||
|
||
### Does This Establish Which Interpretation Is Better Supported?
|
||
|
||
**No.** Experiment 54M exposed disagreement without resolving it. No scoring, ranking, or preference was applied.
|
||
|
||
### Does This Establish Whether Clarification Is Required?
|
||
|
||
**No.** Exposure of disagreement is informational, not prescriptive. Clarification decisions remain untested.
|
||
|
||
### Was Downstream Question Selection Tested?
|
||
|
||
**No.** The model produced no downstream questions. That capability remains untested in this experiment.
|
||
|
||
### Inference Timing
|
||
|
||
| Metric | Value |
|
||
|--------|-------|
|
||
| Number of live inference calls | 3 |
|
||
| Total inference time | 73,194ms (73.2s) |
|
||
| Average | 24,398ms (24.4s) per call |
|
||
| Fastest | 16,885ms (16.9s) — Case 3 |
|
||
| Slowest | 29,495ms (29.5s) — Case 2 |
|
||
|
||
### Limitations
|
||
|
||
- Three cases only: one pricing attribution case, one paraphrase control, one competing-causes case; no cross-domain validation beyond revenue and logistics;
|
||
- Single model (`qwen-claude:latest`) on single host; results may differ with other models or hosts;
|
||
- Automated evaluation flagged false positives due to phrasing-sensitive keyword checks — semantic correctness was determined by human-reference review only;
|
||
- Shared meaning in Case 3 included "customers complaining" (not in the fixed reference) but is still materially shared between both interpretations. This is semantically valid and not a defect;
|
||
- No measurement of edge cases: near-synonymous disagreements, three-way comparisons, or ambiguous disagreement boundaries were tested.
|
||
|
||
### Evaluation Conclusion
|
||
|
||
**Semantic comparison cleanly exposes interpretation disagreement in the tested cases.** Across all three controlled scenarios — real disagreement, paraphrase identity, and competing causal explanations — the model correctly separated shared meaning from substantive disagreement in every case. Paraphrase was not confused with disagreement. No invented disagreement or silent winner selection occurred. Same host/model retained; no production behaviour changed. What remains untested: scalability beyond two interpretations per source, reliability across more diverse domains, and whether this capability holds under repeated runs.
|
||
|
||
### Focused Test Result
|
||
|
||
**Shared-meaning separation:** Clean across all 3 cases (0 cross-contamination).
|
||
**Disagreement identification:** Correct in all 3 tested scenarios.
|
||
**Paraphrase handling:** Correct — no false disagreement generated.
|
||
**Invention check:** No invented disagreement detected by manual review.
|
||
|
||
### Historical Comparison / Regression
|
||
|
||
Experiment 54K showed grounding was promising but imperfect (one missed addition). Experiment 54L showed grounding boundary was stable but evaluator was brittle. Experiment 54M extends beyond grounding to the next reasoning step: comparing two interpretations of the same source. The semantic instruction is domain-neutral (no examples, no expected answers), yet it achieved clean results in all three cases — the comparison capability worked across the three tested patterns: substantive disagreement, paraphrase agreement, and competing causal explanations. Broader generalisation remains untested.
|
||
|
||
### Documentation Updated
|
||
|
||
- `docs/design-evolution-log.md` — Experiment 54L corrections applied; Experiment 54M section appended;
|
||
- `docs/current-handoff.md` — Return-to-Work Note updated to reflect Experiment 54M findings.
|
||
|
||
### Confirmation: Host and Model Remained Unchanged
|
||
|
||
Host: `http://192.168.1.111:11434` (same as production, same as Experiment 54K, same as Experiment 54L)
|
||
Model: `qwen-claude:latest` (same as production, same as Experiment 54K, same as Experiment 54L)
|
||
|
||
### Confirmation: Production Prompts and Schemas Remained Unchanged
|
||
|
||
The semantic instruction was written fresh for this experiment. No production prompts were modified. The output contract (`{ sharedMeaning, disagreement }`) is the experiment-only shape.
|
||
|
||
### Confirmation: No Disagreement Logic Entered Active Runtime
|
||
|
||
All inference calls were made exclusively within test code via `callDisagreementModel()`. No disagreement logic was integrated into any production module. No runtime code changed.
|
||
|
||
### Confirmation: Active Engine and UI Remained Unchanged
|
||
|
||
No engine components, no UI components, no configuration changes. This experiment was entirely contained within test-only code in `tests/reconstruction/semantic-interpretation-disagreement.test.js`.
|
||
|
||
### Status
|
||
|
||
**Pending Rob's review.** No production code changed. No schemas modified. No active engine behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-interpretation-disagreement.test.js`.
|
||
|
||
## Experiment 54N — Does a Disagreement Actually Change What the Engine Needs to Know Next? (2026-08-07)
|
||
|
||
### Objective
|
||
|
||
First, tighten Experiment 54M so it does not imply broader generalisation than the evidence supports.
|
||
|
||
Then test one narrow downstream consequence:
|
||
|
||
> Given a fixed, human-reviewed interpretation disagreement, can the semantic model tell whether that disagreement materially changes what information would need to be established next?
|
||
|
||
This experiment does not generate the next question. It does not decide which interpretation is correct. It tests only whether the disagreement matters enough to change the reasoning direction.
|
||
|
||
### Corrected Experiment 54M Conclusion
|
||
|
||
The comparison capability worked across the three tested patterns: substantive disagreement, paraphrase agreement, and competing causal explanations. Broader generalisation remains untested.
|
||
|
||
shared meaning and substantive disagreement were cleanly separated across three tested patterns;
|
||
no winner was selected;
|
||
broader generalisation remained untested.
|
||
|
||
### Hypothesis
|
||
|
||
A semantic model may be able to distinguish between:
|
||
|
||
- disagreement that does **not** materially alter the next information needed;
|
||
- disagreement that **does** imply different evidence or investigation would be needed before proceeding confidently.
|
||
|
||
If so, disagreement can potentially remain visible without automatically forcing clarification every time.
|
||
|
||
Do not implement clarification or question selection yet.
|
||
|
||
### Context Budget
|
||
|
||
Used:
|
||
|
||
- `docs/current-handoff.md` (Experiment 54M findings and Return-to-Work Note);
|
||
- Experiment 54M only in `docs/design-evolution-log.md`;
|
||
- `tests/reconstruction/semantic-interpretation-disagreement.test.js` as historical reference;
|
||
- `.env.local` only for existing `OLLAMA_BASE_URL` and `OLLAMA_MODEL`.
|
||
|
||
Not read: full experiment history; graph files; Behaviour Selection; assessor; UI; question-selection code; reconstruction production prompts; earlier semantic test files.
|
||
|
||
### Configuration
|
||
|
||
Host: `http://192.168.1.111:11434` (same as production, same as Experiment 54M)
|
||
Model: `qwen-claude:latest` (same as production, same as Experiment 54M)
|
||
|
||
Expected model: `qwen-claude:latest`
|
||
|
||
No localhost fallback. No experiment-specific model variable.
|
||
|
||
### Number of Live Inference Calls
|
||
|
||
Exactly **3** live Ollama calls — one per case.
|
||
|
||
### Input Contract
|
||
|
||
Each call receives:
|
||
|
||
```json
|
||
{
|
||
"source": "...",
|
||
"sharedMeaning": ["..."],
|
||
"disagreement": ["..."]
|
||
}
|
||
```
|
||
|
||
The disagreement is fixed before inference. No original full interpretations included unless required by the case definition.
|
||
|
||
### Output Contract
|
||
|
||
Return exactly:
|
||
|
||
```json
|
||
{
|
||
"changesInformationNeededNext": true,
|
||
"reason": "one short sentence"
|
||
}
|
||
```
|
||
|
||
or:
|
||
|
||
```json
|
||
{
|
||
"changesInformationNeededNext": false,
|
||
"reason": "one short sentence"
|
||
}
|
||
```
|
||
|
||
This boolean is test-only. It is **not** a production contract. No confidence, scores, recommended question, preferred interpretation, or behaviour labels added.
|
||
|
||
### Semantic Instruction Used
|
||
|
||
> Decide whether the stated disagreement would materially change what information needs to be established next before reasoning can proceed confidently. Return true only when the competing interpretations imply meaningfully different evidence or investigation directions. Return false when the disagreement is only wording, emphasis, or does not change the information needed next. Do not choose which interpretation is correct and do not generate a next question.
|
||
|
||
No examples. No expected answers. Same instruction for all three cases.
|
||
|
||
### Case 1 — Pricing Strength Versus Unresolved Cause
|
||
|
||
**Source:** "Revenue is down. I think pricing may be part of the problem, but I am not sure."
|
||
|
||
**Shared meaning:** revenue has declined; pricing may be related to the problem.
|
||
|
||
**Disagreement:** one interpretation treats pricing as a potentially material contributor; the other keeps pricing unresolved and allows other causes.
|
||
|
||
**Fixed human reference:** `changesInformationNeededNext: true` — If pricing is materially causal, pricing evidence becomes central. If causality remains broad, other possible causes also need investigation.
|
||
|
||
**Model output:**
|
||
```json
|
||
{
|
||
"changesInformationNeededNext": true,
|
||
"reason": "Treating pricing as material shifts the investigation toward specific elasticity and competitive data, while keeping it unresolved prioritizes broader diagnostic evidence across other potential revenue drivers."
|
||
}
|
||
```
|
||
|
||
**Consequence classification:** `consequence_correct` ✓
|
||
|
||
### Case 2 — Paraphrase / No Material Disagreement
|
||
|
||
**Source:** "Revenue is down. I think pricing may be part of the problem, but I am not sure."
|
||
|
||
**Shared meaning:** revenue has declined; pricing may contribute; its importance remains uncertain.
|
||
|
||
**Disagreement:** no substantive disagreement; wording differs only.
|
||
|
||
**Fixed human reference:** `changesInformationNeededNext: false` — Equivalent interpretations should not cause a different investigation merely because they are phrased differently.
|
||
|
||
**Model output:**
|
||
```json
|
||
{
|
||
"changesInformationNeededNext": false,
|
||
"reason": "The prompt explicitly states there is no substantive disagreement, so both interpretations require the exact same evidence and investigation directions."
|
||
}
|
||
```
|
||
|
||
**Consequence classification:** `consequence_correct` ✓
|
||
|
||
### Case 3 — Competing Causes
|
||
|
||
**Source:** "Orders are arriving late and customers have started complaining."
|
||
|
||
**Shared meaning:** orders are arriving late; there is a delivery-delay problem.
|
||
|
||
**Disagreement:** one interpretation attributes the likely cause to insufficient staff capacity; the other attributes the possible cause to unreliable supplier lead times.
|
||
|
||
**Fixed human reference:** `changesInformationNeededNext: true` — Staff-capacity evidence and supplier-lead-time evidence are materially different investigation directions.
|
||
|
||
**Model output:**
|
||
```json
|
||
{
|
||
"changesInformationNeededNext": false,
|
||
"reason": "Both interpretations address the same core delivery delay and require verifying order timelines to identify the root cause, so the immediate evidence needed does not materially differ regardless of the hypothesized attribution."
|
||
}
|
||
```
|
||
|
||
**Consequence classification:** `consequence_failed` ✗
|
||
|
||
The model collapsed both competing causes into a single investigation direction ("verifying order timelines") rather than recognising that staff-capacity evidence (payroll, scheduling, shift coverage) and supplier-lead-time evidence (vendor SLAs, historical lead-time variance, procurement processes) represent materially different inquiry paths.
|
||
|
||
### Consequence-Correct Count
|
||
|
||
**2 of 3** cases classified correctly.
|
||
|
||
### Consequence-Failed Count
|
||
|
||
**1 of 3** cases classified incorrectly.
|
||
|
||
### Question: Did Case 1 identify that the disagreement changes the information needed next?
|
||
|
||
Yes. The model returned `changesInformationNeededNext: true` with a reason distinguishing pricing-specific evidence from broader diagnostic investigation.
|
||
|
||
### Question: Did Case 2 correctly avoid treating paraphrase as a reason to change direction?
|
||
|
||
Yes. The model returned `changesInformationNeededNext: false`, correctly identifying the inputs as non-disagreeing.
|
||
|
||
### Question: Did Case 3 identify that competing causes imply different evidence needs?
|
||
|
||
No. The model returned `changesInformationNeededNext: false`, treating both competing causal explanations as requiring the same core investigation (verifying order timelines) rather than recognising divergent evidence paths.
|
||
|
||
### Question: Did the model choose either interpretation as correct?
|
||
|
||
No. None of the three outputs contained winner selection, scoring, or preference language.
|
||
|
||
### Question: Did it generate an actual next question?
|
||
|
||
No. None of the three outputs generated a next question. The Case 3 reason mentioned "verifying order timelines" but as a justification for sameness rather than as a recommended action.
|
||
|
||
### Question: Did it confuse "different interpretation" with "different information need"?
|
||
|
||
Partially. In Case 3, the model conflated the surface-level shared concern (delivery delay investigation) with the deeper investigation directions implied by each attribution hypothesis. It treated two different evidence sets as equivalent because they served the same diagnostic purpose (find root cause).
|
||
|
||
### Does this experiment establish when the engine should ask the user for clarification?
|
||
|
||
No. This experiment only tests consequence detection, not clarification triggering.
|
||
|
||
### Does it establish what exact question should be asked?
|
||
|
||
No. No actual next question was generated or tested in any case.
|
||
|
||
### Does it establish which interpretation is better supported?
|
||
|
||
No. The model did not choose a winner in any case, and no production behaviour changed.
|
||
|
||
### Evidence That Disagreement Consequence Can Be Separated From Disagreement Itself
|
||
|
||
Cases 1 and 2 demonstrate the model can distinguish between disagreement-with-consequence (pricing attribution affects investigation direction) and disagreement-without-consequence (paraphrase). The model produced the correct boolean and a semantically valid reason for both without any prompt conditioning on the expected answer. Case 3 reveals the boundary of that capability — competing causal explanations with identical surface-level diagnostic purposes were collapsed into one information need.
|
||
|
||
### Inference Timing
|
||
|
||
- Number of live calls: **3**
|
||
- Total time: **63,047ms (~63s)**
|
||
- Average: **21,015.68ms per call**
|
||
- Fastest: **14,184.73ms (Case 2)**
|
||
- Slowest: **31,749.99ms (Case 3)**
|
||
|
||
### Questionable or Unsupported Findings
|
||
|
||
Case 3's failure is notable but potentially narrow — the model may succeed with competing causes that have more obviously different evidence profiles. The single-failure rate (1/3) does not justify a broader conclusion about the capability's limits without additional cases testing boundary conditions between same-direction and different-direction evidence needs.
|
||
|
||
### Experiment Conclusion
|
||
|
||
**Disagreement consequence detection is promising but imperfect.**
|
||
|
||
The model correctly distinguished paraphrase (no consequence) from substantive pricing disagreement (consequence present) in Cases 1 and 2, confirming that the boolean can separate consequence from mere disagreement. Case 3 failure — collapsing competing causal attributions into one investigation direction — reveals a candidate failure pattern: when two hypotheses share the same diagnostic purpose but require different evidence sets, the model may collapse them into the same information need. This is the specific gap observed in this experiment within the semantic comparison chain (54K–54N).
|
||
|
||
### Focused Test Result
|
||
|
||
2 of 3 consequence classifications matched fixed human references. No invariant violations detected (no winner selection, no scores, no actual next questions generated). All three outputs produced semantically coherent reasons.
|
||
|
||
### Historical Comparison Result
|
||
|
||
Experiment 54M showed semantic comparison can expose disagreement without choosing a winner. Experiment 54N tests the next reasoning step — whether that disagreement changes information needs — and shows the capability works for pricing-strength ambiguity and paraphrase but not yet for competing causal explanations with shared diagnostic purposes. The progression from grounding (54K/L) → separation (54I/J) → comparison (54M) → consequence (54N) is intact; each step adds one new capability without breaking the previous one.
|
||
|
||
### Documentation Updated
|
||
|
||
- `docs/design-evolution-log.md` — Experiment 54M generalisation correction applied; Experiment 54N section appended;
|
||
- `docs/current-handoff.md` — Return-to-Work Note updated to reflect Experiment 54N findings.
|
||
|
||
### Confirmation: Host and Model Remained Unchanged
|
||
|
||
Host: `http://192.168.1.111:11434` (same as production, same as Experiments 54K–54M)
|
||
Model: `qwen-claude:latest` (same as production, same as Experiments 54K–54M)
|
||
|
||
### Confirmation: Production Prompts and Schemas Remained Unchanged
|
||
|
||
The semantic instruction was written fresh for this experiment. No production prompts were modified. The output contract (`{ changesInformationNeededNext, reason }`) is the experiment-only shape.
|
||
|
||
### Confirmation: No Consequence Logic Entered Active Runtime
|
||
|
||
All inference calls were made exclusively within test code via `callConsequenceModel()`. No consequence logic was integrated into any production module. No runtime code changed.
|
||
|
||
### Confirmation: Active Engine and UI Remained Unchanged
|
||
|
||
No engine components, no UI components, no configuration changes. This experiment was entirely contained within test-only code in `tests/reconstruction/semantic-disagreement-consequence.test.js`.
|
||
|
||
### Status
|
||
|
||
**Pending Rob's review.** No production code changed. No schemas modified. No active engine behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-disagreement-consequence.test.js`.
|
||
|
||
## Experiment 54O — Can the Model Distinguish Same Goal From Different Evidence Needs? (2026-08-07)
|
||
|
||
### Objective
|
||
|
||
Test one narrow question following from Experiment 54N's candidate failure pattern:
|
||
|
||
> When two interpretations share the same overall diagnostic goal, can the model still recognise that they require different evidence to investigate?
|
||
|
||
This is a passive test-only experiment. Do not generate a next question. Do not choose a winning interpretation. Do not change production behaviour.
|
||
|
||
### Hypothesis
|
||
|
||
The model may be able to distinguish evidence needs correctly when asked directly about evidence rather than about the broader "information needed next" consequence. If it still collapses different hypotheses into one evidence need, the 54N failure pattern becomes stronger evidence. If it distinguishes them cleanly, this shows the model can represent the distinction at one level — whether the same capability exists at another remains unproven.
|
||
|
||
### Context Budget
|
||
|
||
Read only:
|
||
|
||
- `docs/current-handoff.md` (Experiment 54N findings and Return-to-Work Note);
|
||
- Experiment 54N only in `docs/design-evolution-log.md`;
|
||
- `tests/reconstruction/semantic-disagreement-consequence.test.js` as historical reference;
|
||
- `.env.local` only for existing `OLLAMA_BASE_URL` and `OLLAMA_MODEL`.
|
||
|
||
Not read: full experiment history; graph files; assessor; Behaviour Selection; UI; question-selection code; earlier semantic experiment files.
|
||
|
||
### Configuration
|
||
|
||
Host: `http://192.168.1.111:11434` (same as production, same as Experiments 54K–54N)
|
||
Model: `qwen-claude:latest` (same as production, same as Experiments 54K–54N)
|
||
|
||
No localhost fallback. No experiment-specific model variable.
|
||
|
||
### Number of Live Inference Calls
|
||
|
||
Exactly **3** live Ollama calls — one per case.
|
||
|
||
### Input Contract
|
||
|
||
Each call receives:
|
||
|
||
```json
|
||
{
|
||
"problem": "...",
|
||
"hypothesisA": "...",
|
||
"hypothesisB": "..."
|
||
}
|
||
```
|
||
|
||
### Output Contract
|
||
|
||
Return exactly:
|
||
|
||
```json
|
||
{
|
||
"sameEvidenceNeeded": true,
|
||
"evidenceForA": ["..."],
|
||
"evidenceForB": ["..."]
|
||
}
|
||
```
|
||
|
||
or the same shape with `sameEvidenceNeeded: false`.
|
||
|
||
No preferred hypothesis, confidence, score, next question, or recommendation.
|
||
|
||
### Semantic Instruction Used
|
||
|
||
> Compare the evidence needed to investigate the two hypotheses. Return sameEvidenceNeeded: true only when materially the same evidence would test both hypotheses. Return false when each hypothesis requires meaningfully different evidence, even if both are trying to explain the same overall problem. List the main evidence needed for each hypothesis. Do not decide which hypothesis is correct and do not generate questions.
|
||
|
||
No examples. No expected answers. Same instruction for all three cases.
|
||
|
||
### Case 1 — Delivery Delay / Different Evidence
|
||
|
||
**Problem:** "Orders are arriving late and customers have started complaining."
|
||
|
||
**Hypothesis A:** "Delivery delays are being caused by insufficient staff capacity."
|
||
|
||
**Hypothesis B:** "Delivery delays are being caused by unreliable supplier lead times."
|
||
|
||
**Fixed human reference:** `sameEvidenceNeeded: false`
|
||
|
||
Expected evidence distinction:
|
||
|
||
- A should involve staffing/capacity/workload evidence;
|
||
- B should involve supplier lead-time/reliability evidence.
|
||
|
||
This is the exact failure pattern from 54N.
|
||
|
||
**Model output:**
|
||
```json
|
||
{
|
||
"sameEvidenceNeeded": false,
|
||
"evidenceForA": [
|
||
"Staff scheduling and shift coverage data compared to daily order volume",
|
||
"Average order processing time per employee or team",
|
||
"Employee absenteeism, turnover rates, and overtime logs",
|
||
"Warehouse throughput capacity limits and task completion bottlenecks"
|
||
],
|
||
"evidenceForB": [
|
||
"Supplier delivery records versus contracted lead times",
|
||
"Inventory stockout logs for critical raw materials or components",
|
||
"Purchase order acknowledgment and fulfillment timelines",
|
||
"Historical reliability metrics and documented delay reasons from specific vendors"
|
||
]
|
||
}
|
||
```
|
||
|
||
**Evidence-need classification:** `evidence_need_correct` ✓
|
||
|
||
### Case 2 — Same Cause, Paraphrased / Same Evidence
|
||
|
||
**Problem:** "Orders are arriving late and customers have started complaining."
|
||
|
||
**Hypothesis A:** "The team may not have enough capacity to process orders on time."
|
||
|
||
**Hypothesis B:** "Insufficient staff capacity may be causing the order delays."
|
||
|
||
**Fixed human reference:** `sameEvidenceNeeded: true`
|
||
|
||
Both should broadly require staffing/capacity/workload evidence.
|
||
|
||
Purpose: ensure different wording is not mistaken for different evidence needs.
|
||
|
||
**Model output:**
|
||
```json
|
||
{
|
||
"sameEvidenceNeeded": true,
|
||
"evidenceForA": [
|
||
"Current order volume compared to historical processing baselines",
|
||
"Staffing levels, shift coverage, and scheduled vs. actual hours worked",
|
||
"Order processing time logs and bottleneck analysis",
|
||
"Workload distribution metrics across the team"
|
||
],
|
||
"evidenceForB": [
|
||
"Current order volume compared to historical processing baselines",
|
||
"Staffing levels, shift coverage, and scheduled vs. actual hours worked",
|
||
"Order processing time logs and bottleneck analysis",
|
||
"Workload distribution metrics across staff"
|
||
]
|
||
}
|
||
```
|
||
|
||
**Evidence-need classification:** `evidence_need_correct` ✓
|
||
|
||
### Case 3 — Different Causes, Different Domain
|
||
|
||
**Problem:** "Website sales have fallen sharply over the last month."
|
||
|
||
**Hypothesis A:** "The fall may be caused by a recent increase in product prices."
|
||
|
||
**Hypothesis B:** "The fall may be caused by a technical checkout problem."
|
||
|
||
**Fixed human reference:** `sameEvidenceNeeded: false`
|
||
|
||
Expected evidence distinction:
|
||
|
||
- A should involve pricing/conversion/customer-response evidence;
|
||
- B should involve checkout errors/technical telemetry/funnel evidence.
|
||
|
||
Purpose: check the pattern outside delivery operations.
|
||
|
||
**Model output:**
|
||
```json
|
||
{
|
||
"sameEvidenceNeeded": false,
|
||
"evidenceForA": [
|
||
"Sales volume and revenue data correlated with specific price change dates",
|
||
"Customer feedback or survey responses regarding price sensitivity",
|
||
"Competitor pricing data for similar products",
|
||
"Cart abandonment rates specifically at the pricing/price review stage"
|
||
],
|
||
"evidenceForB": [
|
||
"Server error logs and monitoring metrics from the checkout subsystem",
|
||
"Checkout funnel analytics identifying specific step drop-offs",
|
||
"Categorization of customer support tickets for technical/errors reports",
|
||
"Cross-browser/device compatibility testing results and page load latency data during checkout"
|
||
]
|
||
}
|
||
```
|
||
|
||
**Evidence-need classification:** `evidence_need_correct` ✓
|
||
|
||
### Evidence-Need-Correct Count
|
||
|
||
**3 of 3** cases classified correctly.
|
||
|
||
### Evidence-Need-Failed Count
|
||
|
||
**0 of 3** cases classified incorrectly.
|
||
|
||
### Question Answers
|
||
|
||
1. Did Case 1 correctly distinguish staff-capacity evidence from supplier evidence? Yes — model returned `sameEvidenceNeeded: false` with materially different evidence lists (staffing/scheduling/throughput vs. supplier records/SLA/historical reliability).
|
||
2. Did Case 2 correctly recognise materially the same evidence need? Yes — model returned `sameEvidenceNeeded: true`; both evidence lists share nearly identical topics (order volume baselines, staffing levels, shift coverage, processing time logs).
|
||
3. Did Case 3 distinguish pricing evidence from technical checkout evidence? Yes — model returned `sameEvidenceNeeded: false` with clearly distinct evidence for each hypothesis.
|
||
4. Did the model collapse same diagnostic goal into same evidence need in any case? No. In all three cases where the correct reference was `false`, the model correctly returned `false`. In the only case where the correct reference was `true`, it correctly returned `true`.
|
||
5. Did it invent evidence unrelated to the hypotheses? No. All evidence items are materially relevant to their respective hypotheses.
|
||
6. Did it choose a winner? No. No preferred hypothesis, scoring, or preference language in any output.
|
||
7. Did it generate a next question? No. No question generation in any output.
|
||
|
||
### Comparison with Experiment 54N
|
||
|
||
In 54N (abstraction level: "does the disagreement change what information needs to be established next?"), Case 3 — identical problem and hypotheses to this experiment's Case 1 — failed. The model returned `changesInformationNeededNext: false`, collapsing the two competing causes into one investigation direction ("verifying order timelines").
|
||
|
||
In 54O (abstraction level: "what evidence is needed to investigate each hypothesis?"), the same hypotheses now correctly return `sameEvidenceNeeded: false` with distinct evidence lists.
|
||
|
||
The same model succeeded when the problem was expressed as direct evidence comparison and failed previously when expressed as a higher-level consequence judgement. The reason for that difference remains unproven.
|
||
|
||
However, the 54N result still stands as a candidate failure pattern in practice: if the engine asks "does this disagreement change what information needs to be established next?" (rather than asking for evidence comparison), it may still collapse the hypotheses.
|
||
|
||
### Inference Timing
|
||
|
||
- Number of live calls: **3**
|
||
- Total time: **62,360ms (~62s)**
|
||
- Average: **20,786.76ms per call**
|
||
- Fastest: **13,755.44ms (Case 3)**
|
||
- Slowest: **24,818.00ms (Case 1)**
|
||
|
||
### Questionable or Unsupported Findings
|
||
|
||
The 3/3 pass rate is strong but comes from only three cases. The test deliberately includes the exact failure case from 54N plus a same-evidence control and a different-domain cross-check. While this pattern is promising, a broader sample of competing-causes scenarios would be needed to confirm generalisation. It is also worth testing whether paraphrased variants of Case 3 still return `sameEvidenceNeeded: false`.
|
||
|
||
### Experiment Conclusion
|
||
|
||
**The model distinguished same-goal hypotheses by their evidence needs in all three tested cases; robustness beyond these cases remains untested.**
|
||
|
||
All three classifications matched fixed human references. The model returned the correct boolean for same-evidence and different-evidence cases, identified materially appropriate evidence for each hypothesis, and avoided winner selection and question generation in every case. The same model succeeded when the problem was expressed as direct evidence comparison and failed previously when expressed as a higher-level consequence judgement. The reason for that difference remains unproven.
|
||
|
||
### Focused Test Result
|
||
|
||
3 of 3 evidence-need classifications matched fixed human references. No invariant violations detected (no winner selection, no scores, no actual next questions generated). All three outputs produced semantically coherent and materially distinct evidence lists for Case 1 and Case 3, and appropriately overlapping lists for Case 2.
|
||
|
||
### Historical Comparison Result
|
||
|
||
Experiment 54N showed that asking "does this disagreement change what information needs to be established next?" fails on competing causal explanations sharing a diagnostic purpose (Case 3). Experiment 54O shows that asking directly "do these hypotheses require the same evidence?" succeeds on the same hypotheses plus two additional cases. The progression from consequence detection (54N) → direct evidence comparison (54O) shows what the model can do when asked about evidence directly; whether the model can use that distinction to recover a higher-level consequence judgement remains unknown.
|
||
|
||
### Documentation Updated
|
||
|
||
- `docs/design-evolution-log.md` — Experiment 54N wording corrected (blind spot → candidate failure pattern, narrowest gap → specific gap observed); Experiment 54O section appended;
|
||
- `docs/current-handoff.md` — Return-to-Work Note updated to reflect Experiment 54O findings.
|
||
|
||
### Confirmation: Host and Model Remained Unchanged
|
||
|
||
Host: `http://192.168.1.111:11434` (same as production, same as Experiments 54K–54N)
|
||
Model: `qwen-claude:latest` (same as production, same as Experiments 54K–54N)
|
||
|
||
### Confirmation: Production Prompts and Schemas Remained Unchanged
|
||
|
||
The semantic instruction was written fresh for this experiment. No production prompts were modified. The output contract (`{ sameEvidenceNeeded, evidenceForA, evidenceForB }`) is the experiment-only shape.
|
||
|
||
### Confirmation: No Evidence-Need Logic Entered Active Runtime
|
||
|
||
All inference calls were made exclusively within test code via `callEvidenceModel()`. No evidence-need logic was integrated into any production module. No runtime code changed.
|
||
|
||
### Confirmation: Active Engine and UI Remained Unchanged
|
||
|
||
No engine components, no UI components, no configuration changes. This experiment was entirely contained within test-only code in `tests/reconstruction/semantic-hypothesis-evidence-needs.test.js`.
|
||
|
||
### Status
|
||
|
||
**Pending Rob's review.** No production code changed. No schemas modified. No active engine behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-hypothesis-evidence-needs.test.js`.
|
||
## Experiment 54P — Can Explicit Evidence Needs Recover the Higher-Level Consequence? (2026-08-07)
|
||
|
||
### Objective
|
||
|
||
Experiment 54N asked "Does this disagreement change what information is needed next?" and failed on competing causal explanations sharing one diagnostic goal (Case 3: staff capacity vs supplier lead time). Experiment 54O showed the same model correctly distinguished the evidence needs when asked directly about evidence.
|
||
|
||
This experiment tests one narrow bridge: if the differing evidence needs are already made explicit, can the model correctly infer whether the disagreement changes what must be established before reasoning can proceed confidently?
|
||
|
||
Do not generate a next question. Do not choose a winning hypothesis. Do not change production behaviour.
|
||
|
||
### Hypothesis
|
||
|
||
If the model is given the evidence needs explicitly, it may correctly infer whether the disagreement changes what must be established next. If it still fails, the problem is likely not simply losing the evidence distinction. If it succeeds, that shows the higher-level consequence can be recovered when the intermediate evidence distinction is available.
|
||
|
||
### Context Budget
|
||
|
||
Read only:
|
||
- `docs/current-handoff.md`;
|
||
- Experiment 54N and 54O in `docs/design-evolution-log.md`;
|
||
- `tests/reconstruction/semantic-hypothesis-evidence-needs.test.js` as historical reference;
|
||
- `tests/reconstruction/semantic-disagreement-consequence.test.js` as historical reference;
|
||
- `.env.local` only for existing `OLLAMA_BASE_URL` and `OLLAMA_MODEL`.
|
||
|
||
Not read: full experiment history; graph files; assessor; Behaviour Selection; UI; question-selection code; production prompts.
|
||
|
||
### Configuration
|
||
|
||
Host: `http://192.168.1.111:11434` (same as production, same as Experiments 54K–54O)
|
||
Model: `qwen-claude:latest` (same as production, same as Experiments 54K–54O)
|
||
|
||
No localhost fallback. No experiment-specific model variable.
|
||
|
||
### Number of Live Inference Calls
|
||
|
||
Exactly **3** live Ollama calls — one per case.
|
||
|
||
### Input Contract
|
||
|
||
Each call receives:
|
||
```json
|
||
{
|
||
"problem": "...",
|
||
"evidenceForA": ["..."],
|
||
"evidenceForB": ["..."]
|
||
}
|
||
```
|
||
|
||
The evidence lists are fixed human-reviewed inputs, not generated during this experiment.
|
||
|
||
### Output Contract
|
||
|
||
Return exactly:
|
||
```json
|
||
{
|
||
"changesInformationNeededNext": true | false,
|
||
"reason": "one short sentence explaining why"
|
||
}
|
||
```
|
||
|
||
No next question, no preferred hypothesis, no confidence, no score.
|
||
|
||
### Semantic Instruction Used
|
||
|
||
> Decide whether these two evidence sets imply materially different information must be established before reasoning can proceed confidently. Return true when the evidence sets investigate meaningfully different things. Return false when they are materially the same despite wording differences. Do not choose which hypothesis is correct and do not generate a question.
|
||
|
||
No examples. No expected answers. Same instruction for all three cases.
|
||
|
||
### Case 1 — Delivery Causes / Different Evidence
|
||
|
||
**Problem:** "Orders are arriving late and customers have started complaining."
|
||
|
||
**Evidence for A:** staffing levels, shift coverage, workload, order-processing throughput.
|
||
**Evidence for B:** supplier delivery records, supplier lead times, supplier reliability history.
|
||
|
||
**Fixed human reference:** `changesInformationNeededNext: true`
|
||
|
||
This is the exact 54N failure case (Case 3) with the 54O evidence distinction made explicit in the input.
|
||
|
||
**Model output:**
|
||
```json
|
||
{
|
||
"changesInformationNeededNext": true,
|
||
"reason": "One evidence set focuses on internal operational capacity while the other examines external supplier logistics, requiring investigation of fundamentally different systems."
|
||
}
|
||
```
|
||
|
||
**Consequence classification:** `consequence_correct` ✓
|
||
|
||
### Case 2 — Same Staffing Cause / Paraphrased Evidence
|
||
|
||
**Problem:** "Orders are arriving late and customers have started complaining."
|
||
|
||
**Evidence for A:** staffing levels, shift coverage, workload, processing times.
|
||
**Evidence for B:** team capacity, staffing levels, shift coverage, order-processing times.
|
||
|
||
**Fixed human reference:** `changesInformationNeededNext: false`
|
||
|
||
Purpose: ensure paraphrased evidence lists are not treated as different information needs.
|
||
|
||
**Model output:**
|
||
```json
|
||
{
|
||
"changesInformationNeededNext": false,
|
||
"reason": "Both evidence sets investigate the same operational factors regarding staffing, scheduling, and processing throughput needed to diagnose delayed orders."
|
||
}
|
||
```
|
||
|
||
**Consequence classification:** `consequence_correct` ✓
|
||
|
||
### Case 3 — Pricing Versus Checkout / Different Evidence
|
||
|
||
**Problem:** "Website sales have fallen sharply over the last month."
|
||
|
||
**Evidence for A:** price changes, conversion response after price changes, customer price sensitivity, competitor pricing.
|
||
**Evidence for B:** checkout error logs, checkout funnel drop-off, payment failures, browser/device failures.
|
||
|
||
**Fixed human reference:** `changesInformationNeededNext: true`
|
||
|
||
**Model output:**
|
||
```json
|
||
{
|
||
"changesInformationNeededNext": true,
|
||
"reason": "One set investigates pricing and market factors while the other investigates technical checkout failures and user experience barriers."
|
||
}
|
||
```
|
||
|
||
**Consequence classification:** `consequence_correct` ✓
|
||
|
||
### Consequence-Correct Count
|
||
|
||
**3 of 3** cases classified correctly.
|
||
|
||
### Consequence-Failed Count
|
||
|
||
**0 of 3** cases classified incorrectly.
|
||
|
||
### Question Answers
|
||
|
||
1. Did Case 1 recover `true` once staff and supplier evidence were explicit? Yes — model returned `changesInformationNeededNext: true` with reason correctly identifying internal operational capacity versus external supplier logistics as different investigation targets.
|
||
2. Did Case 2 correctly return `false` for materially equivalent evidence? Yes — model returned `changesInformationNeededNext: false` with reason correctly identifying that both sets investigate the same staffing/scheduling/throughput factors.
|
||
3. Did Case 3 return `true` for pricing versus checkout evidence? Yes — model returned `changesInformationNeededNext: true` with reason correctly distinguishing pricing/market investigation from technical checkout investigation.
|
||
4. Did the model choose a hypothesis winner? No. No preferred hypothesis language in any output.
|
||
5. Did it generate a next question? No. No question generation in any output.
|
||
6. Does explicit evidence distinction allow the higher-level consequence judgement in all tested cases? Yes, across all three tested cases.
|
||
7. What does this say about the 54N failure? It narrows the candidate explanation: when evidence differences are made explicit as inputs, the model can recover the correct consequence. This suggests the 54N failure may have involved losing or failing to use the evidence distinction rather than lacking it entirely.
|
||
8. Does this prove why 54N failed? No — only one gap between these experiments was tested (explicit vs implicit evidence presentation). Other factors in the original formulation may also contribute.
|
||
9. Does this establish what the engine should ask next? No — this does not prescribe engine behaviour. It shows a capability boundary under controlled conditions.
|
||
|
||
Questions 8 and 9 are answered No as required.
|
||
|
||
### Comparison with Experiment 54N
|
||
|
||
In 54N, Case 3 (identical problem: late orders; identical hypotheses: staff capacity vs supplier lead time) returned `changesInformationNeededNext: false` — collapsing the competing causes into one investigation direction.
|
||
|
||
In 54P, the exact same problem and hypotheses are tested but with the evidence needs made explicit as structured input fields. The model now correctly returns `true` with a reason distinguishing internal operational capacity from external supplier logistics.
|
||
|
||
This is the narrowest bridge between 54N and 54O tested so far: it controls for all inputs except whether the evidence distinction is explicit in the prompt or must be inferred by the model.
|
||
|
||
### Comparison with Experiment 54O
|
||
|
||
Experiment 54O showed the model can distinguish evidence needs directly (3/3). This experiment shows the model can use those explicit distinctions to recover a higher-level consequence judgement (3/3). The capability appears to be available at both levels in these tested cases.
|
||
|
||
### Inference Timing
|
||
|
||
- Number of live calls: **3**
|
||
- Total time: **44,464ms (~44s)**
|
||
- Average: **14,821.44ms per call**
|
||
- Fastest: **13,889.83ms (Case 2)**
|
||
- Slowest: **15,861.54ms (Case 1)**
|
||
|
||
Faster than Experiment 54O's ~62s total — the evidence-consequence task may be computationally lighter than evidence-discovery.
|
||
|
||
### What This Establishes
|
||
|
||
Explicit evidence differences were recovered as correct consequences in all three tested cases:
|
||
- Different evidence for different causes: correctly identified (Case 1, original 54N failure case).
|
||
- Same evidence paraphrased differently: correctly treated as the same need (Case 2 control).
|
||
- Different evidence across domains: correctly identified (Case 3 cross-domain test).
|
||
|
||
No winner was chosen. No next question was generated. All invariant violations absent.
|
||
|
||
### What This Does Not Establish
|
||
|
||
- Whether the model can perform the same reasoning when evidence must be inferred rather than supplied.
|
||
- Whether this generalises to more than three cases or different domains.
|
||
- Why 54N failed — only one variable (explicit vs implicit evidence) was changed between the experiments.
|
||
- What the engine should ask next.
|
||
|
||
### Focused Test Result
|
||
|
||
3 of 3 consequence classifications matched fixed human references. No invariant violations detected (no winner selection, no scores, no next questions). All three outputs produced semantically coherent reasons that correctly addressed the evidence/information distinction for each case.
|
||
|
||
### Experiment Conclusion
|
||
|
||
**Explicit evidence differences recovered the correct consequence in all tested cases.**
|
||
|
||
When the model is given explicit evidence needs as structured input, it can correctly infer whether those needs imply materially different investigation directions — including on the exact case where Experiment 54N failed. In the three tested cases, explicit evidence distinctions were sufficient for the model to produce the expected consequence judgement.
|
||
|
||
### Limitations
|
||
|
||
- Three cases only; limited domain coverage (one delivery scenario, one paraphrase control, one e-commerce scenario).
|
||
- Evidence was fixed human input; not tested with model-generated or inferred evidence.
|
||
- Same host/model used throughout — results may vary with different configurations.
|
||
- Does not establish generalisation beyond these specific inputs.
|
||
|
||
### Status
|
||
|
||
**Pending Rob's review.** No production code changed. No schemas modified. No active engine behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-evidence-consequence.test.js`.
|
||
|
||
## Experiment 54Q — Can One Structured Semantic Step Preserve the Evidence Distinction and the Consequence? (2026-08-07)
|
||
|
||
### Objective
|
||
|
||
First, tighten Experiment 54P so its conclusion stays within the evidence.
|
||
|
||
Then test one narrow bridge: if the model must explicitly identify the evidence needed for each hypothesis before stating whether those needs change what must be established next, can it recover the correct consequence in one semantic call?
|
||
|
||
This is passive and test-only. Do not generate a next question. Do not choose a winning hypothesis. Do not change production behaviour.
|
||
|
||
### Hypothesis
|
||
|
||
A structured output that requires the model to expose the evidence distinction before the consequence may preserve information that was lost in Experiment 54N. If the original 54N failure reappears even with explicit intermediate evidence fields, the problem lies elsewhere. If it succeeds, the result shows that a single structured semantic call can retain the intermediate distinction in these cases.
|
||
|
||
### Context Budget
|
||
|
||
Read only:
|
||
- `docs/current-handoff.md`;
|
||
- Experiment 54P only in `docs/design-evolution-log.md`;
|
||
- `tests/reconstruction/semantic-evidence-consequence.test.js`;
|
||
- `tests/reconstruction/semantic-hypothesis-evidence-needs.test.js`;
|
||
- `.env.local` only for `OLLAMA_BASE_URL` and `OLLAMA_MODEL`.
|
||
|
||
Not read: full experiment history; production prompts; graph files; assessor; Behaviour Selection; question-selection code; UI.
|
||
|
||
### Configuration
|
||
|
||
Host: `http://192.168.1.111:11434` (same as 54P)
|
||
Model: `qwen-claude:latest` (same as 54P)
|
||
|
||
No localhost fallback. No experiment-specific model variable.
|
||
|
||
### Number of Live Inference Calls
|
||
|
||
Exactly **3** live Ollama calls — one per case.
|
||
|
||
### Input Contract
|
||
|
||
Each call receives:
|
||
```json
|
||
{
|
||
"problem": "...",
|
||
"hypothesisA": "...",
|
||
"hypothesisB": "..."
|
||
}
|
||
```
|
||
|
||
### Output Contract
|
||
|
||
Return exactly:
|
||
```json
|
||
{
|
||
"evidenceForA": ["short evidence needs"],
|
||
"evidenceForB": ["short evidence needs"],
|
||
"changesInformationNeededNext": true | false,
|
||
"reason": "one short sentence"
|
||
}
|
||
```
|
||
|
||
No winner. No confidence. No score. No next question. No chain-of-thought. The evidence lists are concise outputs, not reasoning traces.
|
||
|
||
### Semantic Instruction Used
|
||
|
||
> Identify the main evidence needed to investigate each hypothesis. Then decide whether those evidence needs mean materially different information must be established next. Return true when the evidence sets investigate meaningfully different things and false when they are materially the same despite wording differences. Do not choose which hypothesis is correct and do not generate a question.
|
||
|
||
No examples. No expected answers.
|
||
|
||
### Case 1 — Original 54N Failure
|
||
|
||
**Problem:** "Orders are arriving late and customers have started complaining."
|
||
**Hypothesis A:** "Delivery delays are being caused by insufficient staff capacity."
|
||
**Hypothesis B:** "Delivery delays are being caused by unreliable supplier lead times."
|
||
**Fixed human reference:** `changesInformationNeededNext: true`
|
||
|
||
Evidence A should materially involve: staffing, capacity, workload/throughput.
|
||
Evidence B should materially involve: supplier lead times, delivery reliability.
|
||
|
||
### Case 2 — Same Cause, Paraphrased
|
||
|
||
**Problem:** "Orders are arriving late and customers have started complaining."
|
||
**Hypothesis A:** "The team may not have enough capacity to process orders on time."
|
||
**Hypothesis B:** "Insufficient staff capacity may be causing the order delays."
|
||
**Fixed human reference:** `changesInformationNeededNext: false`
|
||
|
||
Both evidence sets should materially concern: staff capacity, workload, processing throughput.
|
||
|
||
### Case 3 — Different Causes, Different Domain
|
||
|
||
**Problem:** "Website sales have fallen sharply over the last month."
|
||
**Hypothesis A:** "The fall may be caused by a recent increase in product prices."
|
||
**Hypothesis B:** "The fall may be caused by a technical checkout problem."
|
||
**Fixed human reference:** `changesInformationNeededNext: true`
|
||
|
||
Evidence A should concern: pricing, conversion/customer response.
|
||
Evidence B should concern: checkout errors, technical/funnel evidence.
|
||
|
||
### Results
|
||
|
||
#### Case 1 — Staff Capacity vs Supplier Lead Time
|
||
|
||
**Model output:**
|
||
```json
|
||
{
|
||
"evidenceForA": ["staff scheduling and availability data", "order processing speed per shift", "employee overtime and workload logs"],
|
||
"evidenceForB": ["supplier on-time delivery historical rates", "vendor promised vs actual arrival dates", "purchase order fulfillment delays"],
|
||
"changesInformationNeededNext": true,
|
||
"reason": "Investigating A requires internal workforce productivity metrics while investigating B requires external vendor performance and logistics data."
|
||
}
|
||
```
|
||
|
||
**Evidence classification:** `evidence_correct` ✓ — Evidence A focused on staffing/capacity/workload (staff scheduling, processing speed, overtime logs). Evidence B focused on supplier/lead times/delivery reliability (on-time rates, promised vs actual dates, fulfillment delays). Materially distinct sets.
|
||
|
||
**Consequence classification:** `consequence_correct` ✓ — Returned `true` with reason correctly distinguishing internal workforce metrics from external vendor data. The original 54N failure case recovered.
|
||
|
||
#### Case 2 — Same Cause, Paraphrased (Control)
|
||
|
||
**Model output:**
|
||
```json
|
||
{
|
||
"evidenceForA": ["Current order volume vs. team processing capacity", "Staff utilization rates and overtime hours", "Workload distribution across team members"],
|
||
"evidenceForB": ["Current order volume vs. staffing capacity", "Employee utilization metrics and overtime data", "Headcount relative to peak demand periods"],
|
||
"changesInformationNeededNext": false,
|
||
"reason": "Both hypotheses target the exact same root cause (capacity constraints), requiring identical operational workload and staffing metrics to validate."
|
||
}
|
||
```
|
||
|
||
**Evidence classification:** `evidence_correct` ✓ — 8 overlapping terms across evidence sets; both focused on capacity/throughput/metrics. Materially equivalent despite different phrasing.
|
||
|
||
**Consequence classification:** `consequence_correct` ✓ — Returned `false`, correctly identifying both hypotheses as targeting the same root cause.
|
||
|
||
#### Case 3 — Pricing vs Checkout
|
||
|
||
**Model output:**
|
||
{
|
||
"evidenceForA": ["Price change logs", "Pre/post price sales data", "Customer pricing complaints"],
|
||
"evidenceForB": ["Checkout error logs", "Payment gateway status reports", "Recent deployment records"],
|
||
"changesInformationNeededNext": true,
|
||
"reason": "One requires financial and market metrics while the other requires IT system and code deployment data."
|
||
}
|
||
|
||
**Evidence classification:** `evidence_correct` ✓ — Evidence A focused on pricing (price change logs, pre/post sales data, pricing complaints). Evidence B focused on checkout/technical (error logs, payment gateway status, deployment records). Materially distinct.
|
||
|
||
**Consequence classification:** `consequence_correct` ✓ — Returned `true`, correctly distinguishing financial/market investigation from IT/system investigation.
|
||
|
||
### Evidence-Correct Count
|
||
|
||
**3 of 3** cases classified correctly.
|
||
|
||
### Evidence-Failed Count
|
||
|
||
**0 of 3** cases classified incorrectly.
|
||
|
||
### Consequence-Correct Count
|
||
|
||
**3 of 3** cases classified correctly.
|
||
|
||
### Consequence-Failed Count
|
||
|
||
**0 of 3** cases classified incorrectly.
|
||
|
||
### Did Any Case Have Correct Evidence but Wrong Consequence?
|
||
|
||
No. No such case was found.
|
||
|
||
### Did Any Case Have Wrong Evidence but Correct Consequence?
|
||
|
||
No. No such case was found.
|
||
|
||
### Did Case 1 Recover the Original 54N Failure?
|
||
|
||
Yes. The model returned `changesInformationNeededNext: true` with evidence correctly distinguishing staffing/capacity from supplier/lead-time domains — the same distinction where Experiment 54N returned `false`.
|
||
|
||
### Did the Model Choose a Winner?
|
||
|
||
No. No winner selection detected in any output.
|
||
|
||
### Did the Model Generate a Next Question?
|
||
|
||
No. No question generation detected in any output.
|
||
|
||
### Comparison with Experiments 54N / 54O / 54P
|
||
|
||
**With 54N:** Experiment 54N (Case 3) returned `changesInformationNeededNext: false` — collapsing competing causes into one investigation direction. In 54Q, the same problem and hypotheses produced `true` with evidence correctly distinguishing the two domains. This suggests the 54N failure was not a lack of capability but a loss or misrouting of intermediate evidence information in the model's processing pipeline.
|
||
|
||
**With 54O:** Experiment 54O showed the model can distinguish evidence needs directly (3/3). In 54Q, the model both identified evidence needs AND correctly judged their consequence in one call (3/3), showing these two operations work together when structured as a single output contract.
|
||
|
||
**With 54P:** Experiment 54P supplied fixed human-evidence lists and recovered correct consequences (3/3). In 54Q, the model itself identified the evidence needs in addition to judging their consequence — both operations succeed within a single call. This is one step beyond 54P: instead of two separate operations (identify evidence → judge consequence), they are combined into one structured output.
|
||
|
||
### Focused Test Result
|
||
|
||
3 of 3 evidence classifications and 3 of 3 consequence classifications matched fixed human references. No invariant violations detected (no winner selection, no scores, no next questions). All three outputs produced semantically coherent evidence lists and correctly classified the consequence for each case. The original 54N failure case (staff capacity vs supplier lead time) recovered when the model was required to explicitly identify the evidence needs before deciding the consequence.
|
||
|
||
### Correction: Scope of Experiment 54Q Reformulation
|
||
|
||
54Q reformulated the task in several ways: it supplied hypotheses directly, required explicit evidence-needs fields, changed the semantic instruction, and changed the output contract. The experiment therefore shows recovery under the structured hypothesis → evidence → consequence formulation, but does not isolate which part of that reformulation caused the recovery.
|
||
|
||
### Experiment Conclusion
|
||
|
||
**Structured evidence plus consequence succeeded across all tested cases.**
|
||
|
||
In the three tested cases, explicit evidence distinctions were sufficient for the model to produce the expected consequence judgement. The evidence distinction and consequence remained aligned in every case: correct evidence maps to correct consequence, and no case produced a mismatch in either direction.
|
||
|
||
### Limitations
|
||
|
||
- Three cases only; limited domain coverage (one delivery scenario, one paraphrase control, one e-commerce scenario).
|
||
- No test of whether the model can preserve this alignment when the prompt is less structured or contains more noise.
|
||
- Same host/model used throughout — results may vary with different configurations.
|
||
- Does not establish generalisation beyond these specific inputs.
|
||
- The evidence lists are generated outputs, not human-fixed references; their correctness was verified semantically against fixed expectations but could vary across runs or prompts.
|
||
|
||
### Status
|
||
|
||
**Pending Rob's review.** No production code changed. No schemas modified. No active engine behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-structured-evidence-consequence.test.js`.
|
||
|
||
## Experiment 54R — Does a Material Disagreement Actually Require User Clarification? (2026-08-07)
|
||
|
||
### Objective
|
||
|
||
Given an explicit interpretation disagreement and its evidence consequence, test whether the model can distinguish between a disagreement that requires clarification from the user and one that can be resolved by investigating evidence.
|
||
|
||
This is passive and test-only. Do not generate the clarification question. Do not generate the next investigation question. Do not choose a winning interpretation. Do not change production behaviour.
|
||
|
||
### Hypothesis
|
||
|
||
The model may be able to distinguish:
|
||
|
||
**Evidence-resolvable disagreement:** The user's meaning is sufficiently clear, but competing explanations require different evidence.
|
||
|
||
**User-clarification disagreement:** The disagreement concerns the user's intended meaning, priority, constraint, or definition, so external evidence cannot resolve it without asking the user.
|
||
|
||
If this distinction works, disagreement does not have to map automatically to clarification.
|
||
|
||
### Context Budget
|
||
|
||
Read only:
|
||
- `docs/current-handoff.md`;
|
||
- Experiment 54Q only in `docs/design-evolution-log.md`;
|
||
- `tests/reconstruction/semantic-structured-evidence-consequence.test.js`;
|
||
- `.env.local` only for `OLLAMA_BASE_URL` and `OLLAMA_MODEL`.
|
||
|
||
Not read: Behaviour Selection; assessor; graph files; UI; active prompts; question-selection code; full experiment history.
|
||
|
||
### Configuration
|
||
|
||
Host: `http://192.168.1.111:11434` (same as 54Q)
|
||
Model: `qwen-claude:latest` (same as 54Q)
|
||
|
||
No localhost fallback. No experiment-specific model variable.
|
||
|
||
### Number of Live Inference Calls
|
||
|
||
Exactly **3** live Ollama calls — one per case.
|
||
|
||
### Input Contract
|
||
|
||
Each call receives:
|
||
```json
|
||
{
|
||
"source": "...",
|
||
"disagreement": ["..."],
|
||
"evidenceNeeded": ["..."]
|
||
}
|
||
```
|
||
|
||
The disagreement and evidence needs are fixed human-reviewed inputs. The model does not rediscover them.
|
||
|
||
### Output Contract
|
||
|
||
Return exactly:
|
||
```json
|
||
{
|
||
"requiresUserClarification": true | false,
|
||
"reason": "one short sentence"
|
||
}
|
||
```
|
||
|
||
No question text, no recommended action, no preferred interpretation, no score, no confidence, no behaviour label. This boolean is test-only and is not a production contract.
|
||
|
||
### Semantic Instruction Used
|
||
|
||
> Decide whether resolving the stated disagreement requires additional meaning, preference, intent, or factual information that only the user can provide. Return true when evidence alone cannot settle the disagreement because the missing distinction belongs to the user's intended meaning, priority, constraint, or private knowledge. Return false when the disagreement can be investigated using external, operational, or observable evidence without asking the user to define what they mean. Do not generate a question and do not choose which interpretation is correct.
|
||
|
||
No examples. No expected answers. Same instruction for all three cases.
|
||
|
||
### Case 1 — Competing Causes, Evidence Can Resolve
|
||
|
||
**Source:** "Orders are arriving late and customers have started complaining."
|
||
**Disagreement:** delays may be caused by insufficient staff capacity / delays may be caused by unreliable supplier lead times.
|
||
**Evidence needed:** staffing levels and workload; processing throughput; supplier lead-time history; supplier delivery reliability.
|
||
**Fixed human reference:** `requiresUserClarification: false`
|
||
|
||
### Case 2 — User Priority Is Ambiguous
|
||
|
||
**Source:** "I want the business to grow, but I don't want to take on more risk."
|
||
**Disagreement:** growth should be prioritised even if some additional risk is unavoidable / avoiding additional risk is a hard constraint even if growth is slower.
|
||
**Evidence needed:** possible growth opportunities; risk exposure of each option.
|
||
**Fixed human reference:** `requiresUserClarification: true`
|
||
|
||
### Case 3 — Same Meaning, No Clarification Need
|
||
|
||
**Source:** "Revenue is down. I think pricing may be part of the problem, but I am not sure."
|
||
**Disagreement:** no substantive disagreement; two interpretations express the same uncertainty in different words.
|
||
**Evidence needed:** pricing history; sales/conversion response; other plausible revenue drivers.
|
||
**Fixed human reference:** `requiresUserClarification: false`
|
||
|
||
### Results
|
||
|
||
#### Case 1 — Competing Causes, Evidence Can Resolve
|
||
|
||
**Model output:**
|
||
```json
|
||
{
|
||
"requiresUserClarification": false,
|
||
"reason": "The disagreement involves operational causes that can be resolved by analyzing objective company data and supplier records."
|
||
}
|
||
```
|
||
|
||
**Resolution-source classification:** `resolution_source_correct` ✓ — Returned `false`. The model correctly identified that operational evidence (company data, supplier records) can distinguish the competing causal hypotheses without asking the user.
|
||
|
||
#### Case 2 — User Priority Is Ambiguous
|
||
|
||
**Model output:**
|
||
```json
|
||
{
|
||
"requiresUserClarification": true,
|
||
"reason": "Resolving this conflict depends on the user's personal priority and risk tolerance, which cannot be determined by external facts alone."
|
||
}
|
||
```
|
||
|
||
**Resolution-source classification:** `resolution_source_correct` ✓ — Returned `true`. The model correctly identified that user-owned meaning (priority, risk tolerance) is required to resolve the disagreement. External facts alone cannot determine whether "don't want more risk" is a preference, constraint, or boundary.
|
||
|
||
#### Case 3 — Same Meaning, No Clarification Need
|
||
|
||
**Model output:**
|
||
```json
|
||
{
|
||
"requiresUserClarification": false,
|
||
"reason": "The stated position explicitly confirms there is no substantive disagreement, so no additional user input is required to resolve it."
|
||
}
|
||
```
|
||
|
||
**Resolution-source classification:** `resolution_source_correct` ✓ — Returned `false`. The model correctly identified that the absence of material disagreement means clarification is unnecessary. Evidence can investigate the underlying uncertainty.
|
||
|
||
### Resolution-Source-Correct Count
|
||
|
||
**3 of 3** cases classified correctly.
|
||
|
||
### Resolution-Source-Failed Count
|
||
|
||
**0 of 3** cases classified incorrectly.
|
||
|
||
### Did Case 1 Correctly Recognise That Competing Causal Hypotheses Can Be Resolved Through Evidence?
|
||
|
||
Yes. The model returned `false` and provided a reason referencing operational causes resolvable by company data and supplier records — evidence, not user meaning.
|
||
|
||
### Did Case 2 Correctly Recognise That the Unresolved Growth-Versus-Risk Priority Belongs to the User?
|
||
|
||
Yes. The model returned `true` and identified that resolution depends on "the user's personal priority and risk tolerance," which external facts alone cannot determine.
|
||
|
||
### Did Case 3 Avoid Unnecessary Clarification Where There Was No Material Disagreement?
|
||
|
||
Yes. The model returned `false`, correctly noting the absence of substantive disagreement makes additional clarification unnecessary.
|
||
|
||
### Did the Model Treat Every Disagreement as Requiring User Clarification?
|
||
|
||
No. Two of three cases returned `false`. Only Case 2 (ambiguous priority) returned `true`.
|
||
|
||
### Did the Model Confuse Missing Evidence with Missing User Meaning?
|
||
|
||
No. In Case 1, the model correctly distinguished between lacking evidence to investigate causes (which it flagged as resolvable through evidence gathering) and lacking user meaning (which it did not claim). The reason text referenced "analyzing objective company data and supplier records" rather than requiring user input.
|
||
|
||
### Did the Model Generate an Actual Question?
|
||
|
||
No. No question was generated in any output. The output contract was respected in all cases.
|
||
|
||
### Did the Model Choose a Winner?
|
||
|
||
No. No interpretation was selected as correct in any case.
|
||
|
||
### Evidence That User-Owned Ambiguity Can Be Separated From Evidence Uncertainty
|
||
|
||
Case 2 succeeded where Case 1 and Case 3 both returned `false` for different reasons — one because evidence can resolve it, the other because no disagreement exists. The model's reasons for each case were distinct in their reference points: operational data (Case 1) versus user priority (Case 2) versus absence of disagreement (Case 3). This pattern suggests the model does not collapse all ambiguity into a single clarification need.
|
||
|
||
### Limitations
|
||
|
||
- Three cases only; limited domain coverage (one delivery scenario, one strategic priority, one revenue statement).
|
||
- Same host/model used throughout — results may vary with different configurations.
|
||
- Does not establish generalisation beyond these specific inputs.
|
||
- The distinction tested here is binary (true/false) and does not test partial or probabilistic resolution-source classification.
|
||
- No evidence was actually gathered in any case — only whether the *source* of resolution was correctly identified.
|
||
- The Case 1 evaluator warning (if present) was a false positive from heuristic wording checks, not a semantic failure.
|
||
|
||
### Conclusion
|
||
|
||
**The model correctly distinguished user-clarification needs from evidence-resolvable disagreement in all tested cases.**
|
||
|
||
Across the three tested patterns — competing causal hypotheses, ambiguous user priority, and absent material disagreement — the model returned the correct boolean in every case with semantically appropriate reasoning. No clarification or investigation question was generated. No interpretation was selected as correct. Across the three tested disagreement patterns, the model did not automatically map disagreement to user clarification.
|
||
|
||
### Status
|
||
|
||
**Pending Rob's review.** No production code changed. No schemas modified. No active engine behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-disagreement-resolution-source.test.js`.
|
||
|
||
## Experiment 54S — Can the Model Identify Exactly What the User Needs to Clarify? (2026-08-07)
|
||
|
||
### Objective
|
||
|
||
When clarification genuinely requires user input, can the model identify the specific missing user-owned distinction without yet generating the clarification question? This is passive and test-only. Do not generate a question. Do not choose a winning interpretation. Do not change Behaviour Selection. Do not change production behaviour.
|
||
|
||
### Hypothesis
|
||
|
||
When clarification genuinely belongs to the user, the model may be able to identify the smallest unresolved user-owned distinction. For example, for "I want the business to grow, but I don't want to take on more risk," the missing distinction is not "what are the risks?" but rather "whether avoiding additional risk is a preference or a hard constraint."
|
||
|
||
### Configuration
|
||
|
||
Host: `http://192.168.1.111:11434` (same as 54R)
|
||
Model: `qwen-claude:latest` (same as 54R)
|
||
|
||
No localhost fallback. No experiment-specific model variable.
|
||
|
||
### Number of Live Inference Calls
|
||
|
||
Exactly **3** live Ollama calls — one per case.
|
||
|
||
### Input Contract
|
||
|
||
Each call receives: `{ source, disagreement, requiresUserClarification }`. The boolean is fixed from human-reviewed Experiment 54R-style references. The model does not re-decide whether clarification is required.
|
||
|
||
### Output Contract
|
||
|
||
Return exactly: `{ "clarificationTarget": "short statement" | null }`.
|
||
- If `requiresUserClarification` is true → smallest specific user-owned distinction;
|
||
- If false → null.
|
||
|
||
No question text, no explanation, no recommendation, no preferred interpretation, no score, no confidence, no behaviour label. Test-only, not a production schema.
|
||
|
||
### Semantic Instruction Used
|
||
|
||
> Identify the specific unresolved distinction that only the user can clarify. If clarification is required, return the smallest statement of the missing user-owned meaning, preference, priority, constraint, definition, or private fact. Do not write a question. Do not add evidence needs. If clarification is not required, return null.
|
||
|
||
No examples. No expected answers. Same instruction for all three cases.
|
||
|
||
### Case 1 — Growth Versus Risk Priority
|
||
|
||
**Source:** "I want the business to grow, but I don't want to take on more risk."
|
||
**Disagreement:** growth should be prioritised even if some additional risk is unavoidable / avoiding additional risk is a hard constraint even if growth is slower.
|
||
**Fixed clarification status:** `requiresUserClarification: true`
|
||
**Fixed human target:** whether avoiding additional risk is a preference/trade-off or a hard constraint.
|
||
|
||
### Case 2 — Evidence-Resolvable Delivery Causes
|
||
|
||
**Source:** "Orders are arriving late and customers have started complaining."
|
||
**Disagreement:** delays may be caused by insufficient staff capacity / delays may be caused by unreliable supplier lead times.
|
||
**Fixed clarification status:** `requiresUserClarification: false`
|
||
**Fixed human target:** null (cause should be investigated through evidence).
|
||
|
||
### Case 3 — Ambiguous Meaning of "Affordable"
|
||
|
||
**Source:** "I want to replace the system, but the new option needs to be affordable."
|
||
**Disagreement:** affordable means keeping upfront purchase cost low / affordable means keeping overall long-term cost low even if upfront cost is higher.
|
||
**Fixed clarification status:** `requiresUserClarification: true`
|
||
**Fixed human target:** what the user means by "affordable" — upfront cost versus overall/long-term cost.
|
||
|
||
### Results
|
||
|
||
#### Case 1 — Growth Versus Risk Priority
|
||
|
||
**Model output:**
|
||
```json
|
||
{
|
||
"clarificationTarget": "whether avoiding additional risk is a preference/trade-off or a hard constraint"
|
||
}
|
||
```
|
||
|
||
**Classification:** `clarification_target_correct` (14 bidirectional concept matches)
|
||
|
||
The model correctly identified the constraint versus preference distinction — the exact human target. No evidence confusion, no question text, no winner selection.
|
||
|
||
#### Case 2 — Evidence-Resolvable Delivery Causes
|
||
|
||
**Model output:**
|
||
```json
|
||
{
|
||
"clarificationTarget": null
|
||
}
|
||
```
|
||
|
||
**Classification:** `clarification_target_correct` (correctly returned null when clarification is not required)
|
||
|
||
The model correctly respected the false signal and returned no target for an evidence-resolvable case. Note: in earlier runs of this test, the model occasionally ignored the false signal and produced a target — this variability was observed but did not affect this final run's classification.
|
||
|
||
#### Case 3 — Ambiguous Meaning of "Affordable"
|
||
|
||
**Model output:**
|
||
```json
|
||
{
|
||
"clarificationTarget": "whether affordability prioritizes low upfront cost or low long-term total cost"
|
||
}
|
||
```
|
||
|
||
**Classification:** `clarification_target_correct` (7 bidirectional concept matches)
|
||
|
||
The model correctly identified the definition ambiguity — upfront cost versus long-term total cost. No vendor comparison, no budget range confusion, no question text.
|
||
|
||
### Clarification-Target-Correct Count
|
||
|
||
**3 of 3** cases classified correctly.
|
||
|
||
### Clarification-Target-Failed Count
|
||
|
||
**0 of 3** cases classified incorrectly.
|
||
|
||
### Required Questions — Answers
|
||
|
||
1. Did Case 1 identify preference/trade-off versus hard constraint? **Yes**
|
||
2. Did Case 1 avoid asking about external risk evidence instead? **Yes** (no evidence keywords present)
|
||
3. Did Case 2 correctly return null? **Yes** (in the final run)
|
||
4. Did Case 3 identify the meaning of "affordable" as upfront versus long-term cost? **Yes**
|
||
5. Did the model ever generate a full question? **No**
|
||
6. Did it confuse clarification target with evidence needed? **No**
|
||
7. Did it choose a winner? **No**
|
||
|
||
### Inference Timing
|
||
|
||
- Total time: 55,511ms (55.5s)
|
||
- Average: 18,503.7ms per call
|
||
- Fastest: 17,046ms (Case 2 — evidence-resolvable)
|
||
- Slowest: 20,957ms (Case 1 — growth-vs-risk)
|
||
|
||
### Limitations
|
||
|
||
- Three cases only; limited domain coverage (one strategic priority, one delivery scenario, one procurement definition).
|
||
- Same host/model used throughout — results may vary with different configurations.
|
||
- Does not establish generalisation beyond these specific inputs.
|
||
- The model occasionally ignored the `requiresUserClarification: false` signal in earlier test runs (producing a target when null was expected), indicating the boolean gate alone may not be sufficient for robust null enforcement.
|
||
- No clarification question was generated — this experiment establishes the target identification layer only.
|
||
|
||
### Conclusion
|
||
|
||
**The final three-case run was correct, but earlier repetitions showed instability when clarification was explicitly not required. Clarification-target identification therefore appears promising, but null enforcement is not yet stable.**
|
||
|
||
Across three patterns — preference/constraint ambiguity, evidence-resolvable operational causes, and definition ambiguity — the model correctly isolated the specific user-owned distinction when clarification was required, returned null when it was not, and never generated a full question or chose a winning interpretation. Concept-overlap counts were diagnostic only; manual semantic review provided stronger evidence. Case 2 instability is an observed behaviour (the model occasionally produced a target despite `requiresUserClarification: false` in earlier runs), not merely a test warning. This establishes the wording of the future clarification question is still open; this does not establish when Behaviour Selection should choose Clarify; this does not establish how the clarification answer should update the graph.
|
||
|
||
### Status
|
||
|
||
**Pending Rob's review.** No production code changed. No schemas modified. No active engine behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-clarification-target.test.js`.
|
||
## Experiment 54T — Is "No Clarification Needed" Stable Across Repeated Identical Inputs? (2026-08-08)
|
||
|
||
### Objective
|
||
|
||
First, tighten Experiment 54S so its conclusion reflects the instability observed during earlier runs.
|
||
|
||
Then test one narrow question:
|
||
|
||
> **When `requiresUserClarification` is explicitly false, does the model consistently return no clarification target across repeated identical calls?**
|
||
|
||
This experiment exists because 54S produced the correct final result but earlier runs sometimes generated a clarification target even when clarification was explicitly not required.
|
||
|
||
This is passive and test-only.
|
||
Do not change the clarification-target instruction yet.
|
||
Do not generate questions.
|
||
Do not change production behaviour.
|
||
|
||
### Hypothesis
|
||
|
||
Two possibilities are plausible.
|
||
|
||
**Stable gating:** When `requiresUserClarification` is false, the model consistently returns `null`.
|
||
|
||
**Semantic override:** The model sometimes ignores the explicit false flag and invents a clarification target because it sees unresolved uncertainty in the source.
|
||
|
||
Either result is useful.
|
||
Do not try to correct the behaviour in this experiment.
|
||
|
||
### Configuration
|
||
|
||
Host: `http://192.168.1.111:11434` (same as 54S)
|
||
Model: `qwen-claude:latest` (same as 54S)
|
||
|
||
No localhost fallback. No experiment-specific model variable.
|
||
|
||
### Number of Live Inference Calls
|
||
|
||
Exactly **6** live Ollama calls — three per case, repeated identical input each time.
|
||
|
||
### Semantic Instruction Used
|
||
|
||
Exact Experiment 54S instruction unchanged:
|
||
|
||
> Identify the specific unresolved distinction that only the user can clarify. If clarification is required, return the smallest statement of the missing user-owned meaning, preference, priority, constraint, definition, or private fact. Do not write a question. Do not add evidence needs. If clarification is not required, return null.
|
||
|
||
No examples. No expected answers. Same instruction for all six cases.
|
||
|
||
### Output Contract
|
||
|
||
Unchanged from 54S:
|
||
```json
|
||
{
|
||
"clarificationTarget": "short statement" | null
|
||
}
|
||
```
|
||
|
||
### Case A — Evidence-Resolvable / False (the unstable case from 54S)
|
||
|
||
**Source:** "Orders are arriving late and customers have started complaining."
|
||
**Disagreement:** delays may be caused by insufficient staff capacity / delays may be caused by unreliable supplier lead times.
|
||
**Fixed clarification status:** `requiresUserClarification: false`
|
||
**Expected result:** `clarificationTarget: null`
|
||
|
||
Run this exact case **3 times** without changing wording. This is the unstable Case 2 from 54S.
|
||
|
||
### Case B — User-Owned Ambiguity / True Control
|
||
|
||
**Source:** "I want the business to grow, but I don't want to take on more risk."
|
||
**Disagreement:** growth should be prioritised even if some additional risk is unavoidable / avoiding additional risk is a hard constraint even if growth is slower.
|
||
**Fixed clarification status:** `requiresUserClarification: true`
|
||
**Expected semantic target:** whether avoiding additional risk is a preference/trade-off or a hard constraint.
|
||
|
||
Run this exact case **3 times**. Purpose: determine whether instability is specific to suppressing clarification or affects clarification-target generation generally.
|
||
|
||
### Results
|
||
|
||
#### Case A — Evidence-Resolvable / False
|
||
|
||
| Run | Result | Classification |
|
||
|-----|--------|----------------|
|
||
| 1 | `null` (15,566ms) | null ✓ |
|
||
| 2 | `null` (13,800ms) | null ✓ |
|
||
| 3 | `null` (19,087ms) | null ✓ |
|
||
|
||
**Null count: 3/3**
|
||
**Non-null count: 0/3**
|
||
|
||
No clarification targets were invented. The model consistently returned `null` across all three repeated identical runs with `requiresUserClarification: false`.
|
||
|
||
#### Case B — User-Owned Ambiguity / True Control
|
||
|
||
| Run | Target | Classification |
|
||
|-----|--------|----------------|
|
||
| 1 | "The relative priority between business growth and strict risk avoidance when they conflict" (18,595ms) | target_correct |
|
||
| 2 | "Preferred priority between accelerating business growth and strictly avoiding additional risk" (19,458ms) | target_correct |
|
||
| 3 | "Your maximum acceptable level of additional risk relative to desired business growth." (18,976ms) | target_correct |
|
||
|
||
**Correct-target count: 3/3**
|
||
**Incorrect-target count: 0/3**
|
||
**Null count: 0/3**
|
||
|
||
All three runs produced semantically correct targets aligned with the human reference. No null responses observed when clarification was required.
|
||
|
||
### Timing
|
||
|
||
- Total time: 105,470ms (105.5s)
|
||
- Average: 17,578.3ms per call
|
||
- Fastest: 13,798ms (Case A run 2)
|
||
- Slowest: 19,457ms (Case B run 3)
|
||
|
||
### Required Questions — Answers
|
||
|
||
1. How many Case A runs returned `null`? **3**
|
||
2. How many Case A runs returned a non-null clarification target? **0**
|
||
3. If Case A produced a target, what distinction did it invent? **N/A — none produced**
|
||
4. How many Case B runs produced the correct clarification target? **3**
|
||
5. Did Case B ever incorrectly return `null`? **No**
|
||
6. Is false/null behaviour materially stable across the three repeated runs? **Yes** — all 3 returned null with zero variance
|
||
7. Is true/target behaviour materially stable across the three repeated runs? **Yes** — all 3 produced semantically correct targets with zero variance
|
||
8. Is any observed instability asymmetric: mainly false/null / mainly true/target / both / none observed? **None observed in this experiment.** Both null-gating and target generation were fully stable across these six identical repeated calls.
|
||
9. Does this experiment establish why instability occurs? **No** — this is an observation experiment, not a diagnostic one.
|
||
10. Does this establish how to enforce null behaviour? **No** — the current instruction and output contract produced stable null behaviour across the three repeated false-case runs tested here; broader stability remains unproven.
|
||
11. Does this establish how Behaviour Selection should work? **No.**
|
||
|
||
### Limitations
|
||
|
||
- Only two cases tested (one false, one true); limited domain coverage.
|
||
- Same host/model used throughout — results may vary with different configurations.
|
||
- This is a stability observation experiment; it does not diagnose root causes of earlier variability.
|
||
- Does not establish generalisation beyond these specific inputs.
|
||
- The model's behaviour in earlier unrecorded runs (when null-gating failed) remains the unknown variable.
|
||
|
||
### Evaluation Conclusion
|
||
|
||
**Clarification null-gating was stable across all tested repeats**
|
||
|
||
Case A returned `null` in 3 of 3 runs. Case B produced correct targets in 3 of 3 runs. No instability was observed in either direction during this experiment.
|
||
|
||
### Status
|
||
|
||
**Pending Rob's review.** No production code changed. No schemas modified. No active engine behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-clarification-null-stability.test.js`.
|
||
|
||
## Experiment 54U — Can a Clarification Target Become a Useful User Question Without Adding New Meaning? (2026-08-08)
|
||
|
||
### Objective
|
||
|
||
Given a fixed clarification target, can the model turn that target into one concise, neutral clarification question without adding assumptions, choosing a side, or broadening the issue?
|
||
|
||
This is test-only.
|
||
Do not integrate anything into the UI.
|
||
Do not change Behaviour Selection.
|
||
Do not change production prompts.
|
||
|
||
### Hypothesis
|
||
|
||
Given a specific clarification target, the model may be able to produce a single question that:
|
||
|
||
- asks only about the unresolved distinction;
|
||
- remains neutral between the interpretations;
|
||
- does not introduce new assumptions;
|
||
- does not ask for evidence instead;
|
||
- does not become a multi-part interview.
|
||
|
||
If it broadens the question or adds new meaning, record that failure.
|
||
|
||
### Configuration
|
||
|
||
Host: `http://192.168.1.111:11434` (same as 54S / 54T)
|
||
Model: `qwen-claude:latest` (same as 54S / 54T)
|
||
|
||
No localhost fallback. No experiment-specific model variable.
|
||
|
||
### Number of Live Inference Calls
|
||
|
||
Exactly **3** live Ollama calls — one per case.
|
||
|
||
### Semantic Instruction Used
|
||
|
||
> Write one concise clarification question that asks only about the supplied clarification target. Keep it neutral between the possible meanings. Do not introduce new facts, assumptions, evidence requests, recommendations, or additional questions. Do not explain why you are asking.
|
||
|
||
No examples. No expected wording. Same instruction for all three cases.
|
||
|
||
### Input Contract
|
||
|
||
Each call receives:
|
||
|
||
```json
|
||
{ "source": "...", "clarificationTarget": "..." }
|
||
```
|
||
|
||
The target is fixed human-reviewed input. The model must not decide whether clarification is needed.
|
||
|
||
### Output Contract
|
||
|
||
Return exactly:
|
||
|
||
```json
|
||
{ "question": "one clarification question" }
|
||
```
|
||
|
||
No explanation, score, confidence, answer options, recommendation, preferred interpretation, or second question.
|
||
|
||
### Case 1 — Preference Versus Hard Constraint
|
||
|
||
**Source:** "I want the business to grow, but I don't want to take on more risk."
|
||
**Clarification target:** whether avoiding additional risk is a preference/trade-off or a hard constraint.
|
||
**Human-reviewed intent:** A correct question should ask whether "no more risk" is an absolute boundary or something the user would trade against growth.
|
||
|
||
### Case 2 — Meaning of "Affordable"
|
||
|
||
**Source:** "I want to replace the system, but the new option needs to be affordable."
|
||
**Clarification target:** whether affordable means low upfront cost or low overall/long-term cost.
|
||
**Human-reviewed intent:** A correct question should clarify which meaning of affordability the user intends.
|
||
|
||
### Case 3 — Private Factual Constraint
|
||
|
||
**Source:** "I could move the project forward next month, depending on whether I actually have enough time."
|
||
**Clarification target:** whether the user has enough available time next month to take on the project.
|
||
**Human-reviewed intent:** A correct question should ask the user to clarify their own available capacity/time.
|
||
|
||
### Results
|
||
|
||
#### Case 1 — Preference Versus Hard Constraint
|
||
|
||
**Returned question:** "Do you view avoiding additional risk as a preference/trade-off or a hard constraint?"
|
||
**Classification:** question_correct — semantically aligned with target (6 overlapping words), structurally valid, neutral.
|
||
**Did it stay within the target?** Yes — asks exactly whether risk avoidance is a preference/trade-off or a hard constraint.
|
||
**Did it add assumptions?** No.
|
||
**Did it ask for evidence?** No.
|
||
**Multiple questions?** No — one substantive question.
|
||
**Implied preferred interpretation?** No.
|
||
|
||
#### Case 2 — Meaning of "Affordable"
|
||
|
||
**Returned question:** 'Does your use of "affordable" refer to a low upfront cost or a low overall/long-term cost?'
|
||
**Classification:** question_correct — semantically aligned with target (5 overlapping words), structurally valid, neutral.
|
||
**Did it stay within the target?** Yes — asks which meaning of affordability the user intends.
|
||
**Did it add assumptions?** No.
|
||
**Did it ask for evidence?** No.
|
||
**Multiple questions?** No.
|
||
**Implied preferred interpretation?** No.
|
||
|
||
#### Case 3 — Private Factual Constraint
|
||
|
||
**Returned question:** "Do you have enough available time next month to take on the project?"
|
||
**Classification:** question_correct — semantically aligned with target (7 overlapping words), structurally valid, neutral.
|
||
**Did it stay within the target?** Yes — asks for the user's own capacity clarification.
|
||
**Did it add assumptions?** No — did not estimate hours or assume availability.
|
||
**Did it ask for evidence?** No.
|
||
**Multiple questions?** No.
|
||
**Implied preferred interpretation?** No.
|
||
|
||
### Timing
|
||
|
||
- Total time: 71,758ms (71.8s)
|
||
- Average: 23,919.3ms per call
|
||
- Fastest: 13,573ms (Case 3)
|
||
- Slowest: 33,314ms (Case 2)
|
||
|
||
### Required Questions — Answers
|
||
|
||
1. Did Case 1 ask only about preference/trade-off versus hard constraint? **Yes**
|
||
2. Did Case 2 ask only what "affordable" means? **Yes**
|
||
3. Did Case 3 correctly ask for the user's private factual capacity? **Yes**
|
||
4. Did any question introduce assumptions not present in the source/target? **No**
|
||
5. Did any question ask for evidence instead of clarification? **No**
|
||
6. Did any question contain more than one substantive question? **No**
|
||
7. Did any question choose or imply a preferred interpretation? **No**
|
||
8. How many cases were question_correct / question_failed? **3 correct, 0 failed.**
|
||
9. Does this establish when the question should be asked? **No.**
|
||
10. Does this establish how the answer should update reasoning state? **No.**
|
||
11. Does this establish that the question will feel good in the UI? **No.**
|
||
|
||
### Evaluation Conclusion
|
||
|
||
**The model produced a clean clarification question for every tested target.**
|
||
|
||
All three cases returned one neutral question addressing only the supplied clarification target, with no introduced assumptions, evidence requests, multi-part structure, or implied preferred interpretations.
|
||
|
||
**Corrected conclusion:** The clarification-target → question step worked cleanly across the three tested targets; broader wording quality and user experience remain untested. Word-overlap metrics are diagnostic only; manual semantic review is the stronger evidence.
|
||
|
||
### Limitations
|
||
|
||
- Only three cases tested; limited domain coverage.
|
||
- Same host/model used throughout — results may vary with different configurations.
|
||
- Does not establish when the question should be asked (that is a separate step).
|
||
- Does not establish how answers should update reasoning state.
|
||
- Semantic quality assessed through structural and overlap heuristics; manual review would strengthen confidence.
|
||
- Does not establish that the question will feel good in the UI.
|
||
|
||
### Status
|
||
|
||
**Pending Rob's review.** No production code changed. No schemas modified. No active engine behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-clarification-question.test.js`.
|
||
|
||
## Experiment 54V — Can a Clarification Answer Resolve Only the Target Without Rewriting Everything Else? (2026-08-08)
|
||
|
||
### Objective
|
||
|
||
Given the original source, a fixed clarification target, the clarification question, and the user's answer, can the model identify what has now been resolved without adding new meaning or rewriting unrelated reasoning?
|
||
|
||
This is test-only.
|
||
Do not integrate with the graph, Behaviour Selection, or UI.
|
||
|
||
### Hypothesis
|
||
|
||
A clarification answer should be able to resolve one narrow user-owned ambiguity without causing the model to:
|
||
- reinterpret the whole source;
|
||
- add unsupported consequences;
|
||
- reopen unrelated uncertainty.
|
||
|
||
If the model cannot preserve that boundary, clarification answers may create as much ambiguity as they remove.
|
||
|
||
### Configuration
|
||
|
||
Host: `http://192.168.1.111:11434` (same as 54S / 54T / 54U)
|
||
Model: `qwen-claude:latest` (same as 54S / 54T / 54U)
|
||
|
||
No localhost fallback. No experiment-specific model variable.
|
||
|
||
### Number of Live Inference Calls
|
||
|
||
Exactly **3** live Ollama calls — one per case.
|
||
|
||
### Input Contract
|
||
|
||
Each call receives:
|
||
```json
|
||
{ "source": "...", "clarificationTarget": "...", "clarificationQuestion": "...", "userAnswer": "..." }
|
||
```
|
||
|
||
### Output Contract
|
||
|
||
Return exactly:
|
||
```json
|
||
{ "resolvedMeaning": "short statement", "targetResolved": true, "remainingUncertainty": null }
|
||
```
|
||
|
||
No next question, recommendation, confidence score, graph update, extra interpretation, or explanation.
|
||
|
||
### Semantic Instruction Used
|
||
|
||
> Use the user's clarification answer only to resolve the supplied clarification target. State the meaning now established by that answer. Mark targetResolved true only when the answer settles the target. Put any uncertainty that remains specifically about that target into remainingUncertainty; otherwise return null. Do not infer wider consequences, rewrite unrelated source meaning, recommend action, or generate another question.
|
||
|
||
No examples. No expected answers. Same instruction for all three cases.
|
||
|
||
### Case 1 — Hard Constraint Resolved
|
||
|
||
**Source:** "I want the business to grow, but I don't want to take on more risk."
|
||
**Clarification target:** whether avoiding additional risk is a preference/trade-off or a hard constraint.
|
||
**Clarification question:** Do you view avoiding additional risk as a preference/trade-off or a hard constraint?
|
||
**User answer:** It's a hard constraint. I don't want any increase in risk.
|
||
|
||
**Human reference:**
|
||
- resolvedMeaning: "avoiding additional risk is a hard constraint"
|
||
- targetResolved: true
|
||
- remainingUncertainty: null
|
||
|
||
Must not infer: growth is impossible; which growth option should be chosen; how much risk currently exists.
|
||
|
||
### Case 2 — Definition Resolved
|
||
|
||
**Source:** "I want to replace the system, but the new option needs to be affordable."
|
||
**Clarification target:** whether affordable means low upfront cost or low overall/long-term cost.
|
||
**Clarification question:** Does your use of "affordable" refer to a low upfront cost or a low overall/long-term cost?
|
||
**User answer:** I care about the total cost over five years, not the upfront price.
|
||
|
||
**Human reference:**
|
||
- resolvedMeaning: "affordability means overall/long-term cost rather than upfront cost"
|
||
- targetResolved: true
|
||
- remainingUncertainty: null
|
||
|
||
Do not invent a budget or preferred system.
|
||
|
||
### Case 3 — Answer Does Not Fully Resolve Target
|
||
|
||
**Source:** "I want the business to grow, but I don't want to take on more risk."
|
||
**Clarification target:** whether avoiding additional risk is a preference/trade-off or a hard constraint.
|
||
**Clarification question:** Do you view avoiding additional risk as a preference/trade-off or a hard constraint?
|
||
**User answer:** It depends on the opportunity.
|
||
|
||
**Human reference:**
|
||
- The target is **not fully resolved**.
|
||
- targetResolved: false
|
||
- remainingUncertainty should preserve that the acceptable trade-off still depends on circumstances.
|
||
- resolvedMeaning may state only that the user's risk position is conditional on the opportunity.
|
||
|
||
Do not force the answer into either preference or hard-constraint category.
|
||
|
||
### Results
|
||
|
||
#### Case 1 — Hard Constraint Resolved
|
||
|
||
**Returned result:**
|
||
- resolvedMeaning: "Avoiding additional risk is established as a hard constraint, meaning no increase in risk is acceptable."
|
||
- targetResolved: true
|
||
- remainingUncertainty: null
|
||
|
||
**Manual semantic review:**
|
||
- Used only the user answer to resolve the target? **Yes.**
|
||
- Preserved unresolved ambiguity where present? N/A (target was resolved).
|
||
- Avoided wider consequences? **Yes** — did not state growth is impossible, did not choose a growth option, did not estimate current risk levels. The phrase "meaning no increase in risk is acceptable" is a direct restatement of the user answer ("I don't want any increase in risk"), not an addition from the source.
|
||
- Avoided adding unsupported facts? **Yes.**
|
||
- Avoided generating another question? **Yes.**
|
||
|
||
**Classification:** resolution_correct
|
||
|
||
#### Case 2 — Definition Resolved
|
||
|
||
**Returned result:**
|
||
- resolvedMeaning: "Affordable refers to a low overall or long-term cost rather than a low upfront price."
|
||
- targetResolved: true
|
||
- remainingUncertainty: null
|
||
|
||
**Manual semantic review:**
|
||
- Used only the user answer to resolve the target? **Yes.**
|
||
- Preserved unresolved ambiguity where present? N/A (target was resolved).
|
||
- Avoided wider consequences? **Yes** — did not invent a budget amount, did not recommend a system.
|
||
- Avoided adding unsupported facts? **Yes.**
|
||
- Avoided generating another question? **Yes.**
|
||
|
||
**Classification:** resolution_correct
|
||
|
||
#### Case 3 — Answer Does Not Fully Resolve Target
|
||
|
||
**Returned result:**
|
||
- resolvedMeaning: "Avoiding additional risk is conditional rather than fixed, varying based on the specific opportunity."
|
||
- targetResolved: false
|
||
- remainingUncertainty: "It remains unclear whether avoiding additional risk is strictly a preference/trade-off or a hard constraint, as the stance depends on the opportunity."
|
||
|
||
**Manual semantic review:**
|
||
- Used only the user answer to resolve the target? **Yes.** The model preserved the conditionality present in "It depends on the opportunity" without forcing into either category.
|
||
- Preserved unresolved ambiguity where present? **Yes.** Correctly kept the target unresolved and described the remaining uncertainty specifically about that target.
|
||
- Avoided wider consequences? **Yes.**
|
||
- Avoided adding unsupported facts? **Yes.**
|
||
- Avoided generating another question? **Yes.**
|
||
|
||
**Classification:** resolution_correct
|
||
|
||
### Timing
|
||
|
||
- Total time: 38,052ms (38.1s)
|
||
- Average: 12,684.0ms per call
|
||
- Fastest: 8,479ms (Case 1)
|
||
- Slowest: 15,208ms (Case 3)
|
||
|
||
### Required Questions — Answers
|
||
|
||
1. Did Case 1 resolve the target to a hard constraint without adding wider consequences? **Yes.** The resolved meaning stays within the user answer's scope. No inference about growth feasibility, option selection, or current risk levels.
|
||
2. Did Case 2 resolve "affordable" to long-term cost without inventing a budget? **Yes.** The model correctly captured the five-year perspective without adding any budget figure or system recommendation.
|
||
3. Did Case 3 correctly keep the target unresolved? **Yes.** The model returned targetResolved=false, preserved conditionality in resolvedMeaning, and provided meaningful remainingUncertainty.
|
||
4. Did any case force an ambiguous answer into a stronger meaning? **No.** Case 3's conditional answer was kept at its actual strength — neither strengthened to preference nor to hard constraint.
|
||
5. Did any case rewrite unrelated parts of the source? **No.** In Cases 1 and 3 (same source), the model treated the "grow" portion identically to the original source meaning without reinterpreting it.
|
||
6. Did any case generate another question? **No.** All resolvedMeaning fields are statements, not questions.
|
||
7. How many cases were resolution_correct / resolution_failed? **3 correct, 0 failed.**
|
||
8. Does this establish how graph state should be updated? **No.** This only tests semantic recognition of what a clarification answer resolves; it does not test any graph update mechanism.
|
||
9. Does this establish what question should come next? **No.** The experiment tested one directional step (answer → resolved meaning) and did not test the next question generation cycle.
|
||
10. Does this establish how Behaviour Selection should react? **No.** No behaviour selection logic was tested or involved.
|
||
|
||
### Evaluation Conclusion
|
||
|
||
**Clarification answers resolved only the intended target across all tested cases.**
|
||
|
||
All three cases returned correct resolution boundaries: Cases 1 and 2 settled the target cleanly; Case 3 preserved incomplete information at its actual strength without forcing it into a stronger category. The model did not widen beyond the clarification target, invent consequences, or generate new questions in any case.
|
||
|
||
**The individual clarification steps have each worked in their isolated fixed-case tests; end-to-end behaviour remains untested.**
|
||
|
||
### Limitations
|
||
|
||
- Only three cases tested; limited domain coverage (risk constraint, affordability definition, conditional constraint).
|
||
- Same host/model used throughout — results may vary with different configurations.
|
||
- Does not establish how graph state should update from resolved meanings.
|
||
- Does not establish what question should come next after resolution.
|
||
- Does not establish how Behaviour Selection should react to resolved vs unresolved targets.
|
||
- Semantic quality assessed through structural checks and manual review; broader generalisation untested.
|
||
- Case 3's remainingUncertainty output is longer than the human reference — acceptable because it describes the uncertainty rather than adding meaning, but worth noting.
|
||
|
||
### Status
|
||
|
||
**Pending Rob's review.** No production code changed. No schemas modified. No active engine behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-clarification-answer-resolution.test.js`.
|
||
|
||
## Experiment 54W — Does the Clarification Chain Hold Together End to End? (2026-08-08)
|
||
|
||
### Objective
|
||
|
||
Tighten Experiment 54V so it does not overstate the isolated clarification-chain results. Then test the smallest end-to-end version of the clarification path: **starting from one disagreement, can the semantic steps correctly determine whether the user is needed, identify the clarification target, word one question, and use the user's answer to resolve only that target without semantic drift between steps?**
|
||
|
||
This is still test-only. Do not integrate with the active engine, graph, Behaviour Selection, or UI.
|
||
|
||
### Hypothesis
|
||
|
||
The individual clarification steps may remain aligned when chained together. For a genuine user-owned ambiguity, the chain should preserve: `disagreement → user required → clarification target → neutral question → answer → resolved target`. For an evidence-resolvable disagreement, the chain should stop early rather than inventing a clarification target or question. If the steps drift when connected, record exactly where the first material divergence occurs.
|
||
|
||
### Configuration
|
||
|
||
Host: `http://192.168.1.111:11434` (same as 54R–54V)
|
||
Model: `qwen-claude:latest` (same as 54R–54V)
|
||
|
||
### Number of Live Inference Calls
|
||
|
||
Exactly **5** live Ollama calls — 4 for Scenario A + 1 for Scenario B.
|
||
|
||
### Context Used
|
||
|
||
- `docs/current-handoff.md`
|
||
- Experiment 54V only in `docs/design-evolution-log.md`
|
||
- `tests/reconstruction/semantic-disagreement-resolution-source.test.js`
|
||
- `tests/reconstruction/semantic-clarification-target.test.js`
|
||
- `tests/reconstruction/semantic-clarification-question.test.js`
|
||
- `tests/reconstruction/semantic-clarification-answer-resolution.test.js`
|
||
|
||
### Scenarios
|
||
|
||
#### Scenario A — Genuine User-Owned Ambiguity
|
||
|
||
**Source:** "I want the business to grow, but I don't want to take on more risk."
|
||
|
||
**Fixed disagreement:**
|
||
1. growth should be prioritised even if some additional risk is unavoidable;
|
||
2. avoiding additional risk is a hard constraint even if growth is slower.
|
||
|
||
**Fixed user answer:** "It's a hard constraint. I don't want any increase in risk."
|
||
|
||
#### Scenario B — Evidence-Resolvable Disagreement
|
||
|
||
**Source:** "Orders are arriving late and customers have started complaining."
|
||
|
||
**Fixed disagreement:**
|
||
1. delays may be caused by insufficient staff capacity;
|
||
2. delays may be caused by unreliable supplier lead times.
|
||
|
||
### Stage-by-Stage Outputs
|
||
|
||
#### Scenario A — Full Chain
|
||
|
||
**A1 — Resolution Source (Experiment 54R instruction)**
|
||
|
||
- **Actual output:** `{ requiresUserClarification: true, reason: "The disagreement centers on a priority trade-off between growth and risk avoidance that depends entirely on the user's underlying constraints rather than external facts." }`
|
||
- **Input used:** original source + fixed disagreement + evidence needs
|
||
- **Classification:** `stage_correct` — correctly identified that user clarification is required for a genuine user-owned ambiguity. The reason references subjective constraints (user-owned) not external facts, consistent with the true classification.
|
||
|
||
**A2 — Clarification Target (Experiment 54S instruction)**
|
||
|
||
- **Input target used from A1:** `requiresUserClarification: true` (actual upstream output, NOT replaced by a human reference)
|
||
- **Actual output:** `{ clarificationTarget: "Priority between business growth and risk avoidance when they conflict" }`
|
||
- **Classification:** `stage_correct` — the model correctly identified the user-owned distinction. The target focuses on priority ordering between two competing interests (growth vs. risk), which is the exact user-owned ambiguity present in the source. No question language, no evidence framing, no interpretation chosen.
|
||
|
||
**A3 — Clarification Question (Experiment 54U instruction)**
|
||
|
||
- **Input target from A2:** `"Priority between business growth and risk avoidance when they conflict"` (actual upstream output)
|
||
- **Actual output:** `{ question: "Which do you prioritize when business growth and risk avoidance come into conflict?" }`
|
||
- **Classification:** `stage_correct` — single neutral question, ends with one question mark, no biased framing. The question uses the actual A2 target's language ("priority", "business growth", "risk avoidance") without substituting a human reference. Neutral between alternatives: does not say you should prioritize growth or risk.
|
||
|
||
**A4 — Answer Resolution (Experiment 54V instruction)**
|
||
|
||
- **Input target from actual A2:** `"Priority between business growth and risk avoidance when they conflict"`
|
||
- **Input question from actual A3:** `"Which do you prioritize when business growth and risk avoidance come into conflict?"`
|
||
- **Fixed user answer:** "It's a hard constraint. I don't want any increase in risk."
|
||
- **Actual output:** `{ resolvedMeaning: "Risk avoidance takes absolute priority over business growth, as no increase in risk is acceptable regardless of growth potential.", targetResolved: true, remainingUncertainty: null }`
|
||
- **Classification:** `stage_correct` — targetResolved=true correctly reflects that the user's answer settles the priority question. The resolved meaning interprets "hard constraint" and "no increase in risk" as absolute priority (non-negotiable boundary) over growth. This is a valid semantic interpretation of the user's answer within the A2/A3 framing. No wider consequences introduced: does not state growth is impossible, does not choose a specific growth option, does not estimate current risk levels.
|
||
|
||
#### Scenario B — Evidence-Resolvable Stop
|
||
|
||
**B1 — Resolution Source (Experiment 54R instruction)**
|
||
|
||
- **Actual output:** `{ requiresUserClarification: false, reason: "The disagreement can be resolved by examining operational metrics and supplier delivery records rather than requiring subjective input." }`
|
||
- **Classification:** `correct_stop` — correctly identified that evidence (operational data) can resolve the cause of delays. The chain stopped immediately; no clarification target call was made, no question was generated, no answer-resolution was attempted.
|
||
|
||
### Actual Upstream Outputs Passed Between Stages
|
||
|
||
| Stage | Received From | Value |
|
||
|---|---|---|
|
||
| A2 | A1 | `requiresUserClarification: true`, reason about subjective constraints |
|
||
| A3 | A2 | `clarificationTarget: "Priority between business growth and risk avoidance when they conflict"` |
|
||
| A4 | A2 | Same A2 target (same as above) |
|
||
| A4 | A3 | `question: "Which do you prioritize when business growth and risk avoidance come into conflict?"` |
|
||
|
||
Human reference data was used only for evaluation — not silently substituted between stages.
|
||
|
||
### First Drift Point in Scenario A
|
||
|
||
**No material chain failure occurred, although Stage A2 broadened the clarification target from preference-versus-hard-constraint to general priority ordering. That loss of specificity did not break this scenario.** The chain preserved the user-owned nature of the ambiguity from A1 through to resolution at A4 without introducing unsupported meaning or changing the interpretation of upstream results.
|
||
|
||
### Scenario B — Stop Verification
|
||
|
||
- **Did Scenario B stop after B1:** Yes
|
||
- **Were any unnecessary clarification calls made for Scenario B:** No (0 additional calls)
|
||
|
||
### Question: Did any stage choose a winner?
|
||
|
||
**No.** None of the stages selected an interpretation as correct or better. A4's resolved meaning states what the user's answer settled (priority resolution) rather than declaring one pre-existing interpretation as the winner. The chain reports what was clarified, not which side of the original disagreement is right.
|
||
|
||
### Question: Did any stage introduce unsupported meaning that materially affected the next stage?
|
||
|
||
**No.** A2 stayed within the priority dimension present in the source. A3 preserved both competing terms ("business growth", "risk avoidance") from the A2 target. A4 interpreted the user's hard-constraint answer as absolute priority over growth — a valid reading given the answer and the A2/A3 framing. No stage added external facts or consequences that materially distorted downstream reasoning.
|
||
|
||
### Question: Evidence that isolated clarification steps survive under chaining
|
||
|
||
**Yes.** The chain_correct result demonstrates that all four individual capabilities (resolution source, target identification, question wording, answer resolution) remained usable when chained in the two tested scenarios. Each stage's output was a valid input for the next stage. No stage degraded or produced an unexpected format. The semantic proximity between A2 and A4 is worth noting: A2 framed the distinction as "priority" while the user answer used "hard constraint" — these are semantically close but not identical (a hard constraint is stronger than a priority preference). A4 correctly interpreted the hard-constraint answer within the priority framing, so this proximity was sufficient for alignment.
|
||
|
||
### Questionable or Unsupported Findings
|
||
|
||
- Only **one** instance of each scenario was tested. Chain stability across repeated runs needs verification.
|
||
- Only the growth-versus-risk domain was tested for Scenario A. Different domains may produce different drift patterns.
|
||
- A2's output ("Priority between business growth and risk avoidance when they conflict") lost the "preference/trade-off vs hard constraint" distinction present in the 54S human reference. This loss of granularity is not a failure per se — it is still correct within its contract — but it means downstream stages operate on a less precise target. The chain succeeded with this coarser representation, which is evidence that the steps tolerate some semantic imprecision.
|
||
|
||
### Experiment Conclusion
|
||
|
||
**The clarification chain remained semantically aligned end to end in both tested scenarios.** For Scenario A (genuine user-owned ambiguity), all four stages produced correct outputs and each stage's actual output was a valid input for the next stage with no material drift. For Scenario B (evidence-resolvable disagreement), the model correctly stopped after the first decision without inventing unnecessary clarification steps.
|
||
|
||
### Focused Test Result
|
||
|
||
`chain_correct` for Scenario A + `correct_stop` for Scenario B. 5/5 live calls completed within budget. All stage assertions passed.
|
||
|
||
### Historical Comparison Result
|
||
|
||
The chain_correct result is new evidence not available in any earlier experiment (54R–54V tested isolated steps only). It demonstrates that the individual clarification capabilities remained usable when chained in the two tested scenarios, on one scenario and one model configuration. This does not extend to production integration readiness.
|
||
|
||
### Documentation Updated
|
||
|
||
- `docs/design-evolution-log.md` — added 54V clarifying caveat; added full Experiment 54W entry
|
||
- `docs/current-handoff.md` — added 54V clarifying caveat
|
||
|
||
### Confirmation Host and Model Remained Unchanged
|
||
|
||
Host: `http://192.168.1.111:11434`. Model: `qwen-claude:latest`. Same as 54R–54V.
|
||
|
||
### Confirmation Production Prompts and Schemas Remained Unchanged
|
||
|
||
No production prompts read or modified. No schemas changed. All inference calls used the experiment-specific semantic instructions defined in this test file, not production prompts.
|
||
|
||
### Confirmation Behaviour Selection Remained Unchanged
|
||
|
||
Behaviour Selection was not called or referenced. No integration with the selector occurred.
|
||
|
||
### Confirmation Graph and UI Remained Unchanged
|
||
|
||
No graph files read or modified. No UI code touched. The experiment is test-only.
|
||
|
||
### Confirmation No Clarification-Chain Logic Entered Active Runtime
|
||
|
||
This experiment created one new test file only. No clarification-chain logic entered any active runtime path, production module, or behaviour selection output.
|
||
|
||
### Return-to-Work Note (Experiment 54W)
|
||
|
||
54R–54V tested the clarification steps individually in isolated fixed-case scenarios; each worked correctly on its own but end-to-end alignment was never verified. 54W tested the first chained journey using actual upstream model outputs rather than replacing them with human references across four stages for Scenario A and one stage for Scenario B. The growth-versus-risk chain stayed aligned through decision → target → question → answer resolution (chain_correct). The delivery-cause case correctly stopped before clarification (correct_stop). No material chain failure occurred, although Stage A2 broadened the clarification target from preference-versus-hard-constraint to general priority ordering. That loss of specificity did not break this scenario. Graph, Behaviour Selection, UI, and production integration remain untouched. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: `feature/user-workspace-ux-v0.7`. First test/file to inspect when resuming: `tests/reconstruction/semantic-clarification-chain.test.js` for the full experiment and results. Status pending Rob's review.
|
||
## Experiment 54X — Does the Clarification Target Lose Important Specificity When Chained? (2026-08-08)
|
||
|
||
### Objective
|
||
|
||
Isolate whether the clarification-target-generation step preserves the exact user-owned distinction or broadens it, using three fixed cases under the same instruction as Experiment 54S. Passive and test-only. No question generation, answer resolution, Behaviour Selection, graph, or UI integration attempted. Same host/model. Branch: `feature/user-workspace-ux-v0.7`.
|
||
|
||
### Hypothesis
|
||
|
||
The model may preserve clarification targets well when the ambiguity is simple and explicit, but broaden targets when the distinction is relational or preference-based. If broadening happens repeatedly, that may matter downstream because the question-generation step can only be as precise as the target it receives.
|
||
|
||
### Configuration
|
||
|
||
Host: `http://192.168.1.111:11434` (same as 54R–54W)
|
||
Model: `qwen-claude:latest` (same as 54R–54W)
|
||
|
||
### Number of Live Inference Calls
|
||
|
||
Exactly **3** live Ollama calls — one per case.
|
||
|
||
### Context Used
|
||
|
||
- `docs/current-handoff.md`
|
||
- Experiment 54W only in `docs/design-evolution-log.md` (as historical context for the broadening observation)
|
||
- `tests/reconstruction/semantic-clarification-target.test.js` (for structural reference)
|
||
- `tests/reconstruction/semantic-clarification-chain.test.js` (for structural reference)
|
||
|
||
### Experiment 54W Corrections Applied
|
||
|
||
Replaced "No drift was detected." with: "**No material chain failure occurred, although Stage A2 broadened the clarification target from preference-versus-hard-constraint to general priority ordering. That loss of specificity did not break this scenario.**"
|
||
|
||
Replaced "the individual clarification capabilities survive end-to-end chaining" with: "**The individual clarification capabilities remained usable when chained in the two tested scenarios.**"
|
||
|
||
### Case 1 — Preference Versus Hard Constraint
|
||
|
||
**Source:** "I want the business to grow, but I don't want to take on more risk."
|
||
**Disagreement:** growth should be prioritised even if some additional risk is unavoidable / avoiding additional risk is a hard constraint even if growth is slower.
|
||
**Human reference target:** whether avoiding additional risk is a preference/trade-off or a hard constraint
|
||
|
||
**Actual model output:** `"preferred priority between business growth and risk avoidance"`
|
||
**Specificity classification:** **target_broadened** — On the right topic but broadened from the material distinction (preference/trade-off vs. hard constraint) to general priority ordering. Usable downstream but not fully specific.
|
||
|
||
### Case 2 — Definition Ambiguity
|
||
|
||
**Source:** "I want to replace the system, but the new option needs to be affordable."
|
||
**Disagreement:** affordable means keeping upfront cost low / affordable means keeping total long-term cost low.
|
||
**Human reference target:** whether "affordable" means low upfront cost or low overall/long-term cost
|
||
|
||
**Actual model output:** `"whether 'affordable' refers to upfront cost or total long-term cost"`
|
||
**Specificity classification:** **target_specific** — Preserved the material distinction: upfront cost versus total long-term cost. The distinction is explicit and identical in meaning to the human reference.
|
||
|
||
### Case 3 — Private Factual Boundary
|
||
|
||
**Source:** "I could move the project forward next month, depending on whether I actually have enough time."
|
||
**Disagreement:** the user has enough available time next month / the user does not have enough available time next month.
|
||
**Human reference target:** whether the user has enough available time next month to take on the project
|
||
|
||
**Actual model output:** `"whether the user has enough available time next month"`
|
||
**Specificity classification:** **target_specific** — Preserved all three required elements: time availability, next month, and the capacity question. The omission of "to take on the project" does not lose material specificity — it is implied by the source context.
|
||
|
||
### Inference Timing
|
||
|
||
| Metric | Value |
|
||
|---|---|
|
||
| Total live calls | 3 |
|
||
| Total inference time | 49,220 ms |
|
||
| Average | 16,406.7 ms per call |
|
||
| Fastest | 11,815 ms (Case 2) |
|
||
| Slowest | 21,303 ms (Case 1) |
|
||
|
||
### Results Summary
|
||
|
||
| Classification | Count |
|
||
|---|---|
|
||
| target_specific | 2/3 |
|
||
| target_broadened | 1/3 |
|
||
| target_wrong | 0/3 |
|
||
|
||
### Questions Answered
|
||
|
||
1. Did Case 1 preserve preference/trade-off versus hard constraint? **No** — broadened to priority ordering.
|
||
2. Did Case 2 preserve upfront versus long-term affordability? **Yes** — preserved explicitly.
|
||
3. Did Case 3 preserve the user's available-time boundary? **Yes** — preserved explicitly with all three required elements.
|
||
4. How many cases were target_specific / target_broadened / target_wrong? **2 / 1 / 0**
|
||
5. Did any target remain usable while still losing material specificity? **Yes** — Case 1 was broadly relevant and actionable but lost the preference-versus-constraint distinction.
|
||
6. Did any target introduce unsupported meaning? **No** — no case introduced concepts not present in source or disagreement.
|
||
7. Does this reproduce the broadening observed in 54W? **Yes** — both experiments show broadening from preference/constraint to priority framing on Case 1-style input.
|
||
8. Does this establish why broadening happens? **No** — one isolated result per case cannot determine causality; only that it does occur for at least one ambiguity pattern.
|
||
9. Does this establish whether a broader target is acceptable for the user journey? **No** — acceptability depends on downstream question quality and user experience, which were not tested here.
|
||
10. Does this establish how the clarification question should be worded? **No** — no question-generation step was involved.
|
||
|
||
### Evidence About Clarification-Target Specificity
|
||
|
||
A clarification target can be broadly relevant without being precise enough. Case 1's output ("preferred priority between business growth and risk avoidance") is clearly about the right topic and usable downstream, but it does not preserve the material distinction that the user actually needs to clarify — whether avoiding additional risk is a preference or a hard constraint. Cases 2 and 3 show that the same instruction can produce fully specific targets when the ambiguity involves definition boundaries or private facts rather than preference-versus-constraint relationships.
|
||
|
||
### Limitations
|
||
|
||
- Only one model configuration was tested (qwen-claude:latest). Different models may behave differently.
|
||
- Only one inference per case — stability across repeated runs is untested here (though 54L previously showed strong stability for other tasks).
|
||
- The broadening pattern only emerged in Case 1; the instruction and model appear capable of specificity on other patterns.
|
||
- No downstream question or answer-resolution step was tested — usability of a broader target cannot be fully assessed without those stages.
|
||
|
||
### Experiment Conclusion
|
||
|
||
Clarification targets remained usable but broadened in one of three tested cases (Case 1). The broadening reproduced the same pattern observed in 54W: preference-versus-constraint distinctions tend to become priority-ordering framings. This is not a failure — the target remains actionable — but Specificity loss occurred in one of the three tested ambiguity patterns and was absent in the other two.
|
||
|
||
### Focused Test Result
|
||
|
||
2/3 targets preserved material distinction; 1/3 broadened (matching 54W pattern). All structural assertions passed. No invariant violations detected.
|
||
|
||
### Historical Comparison Result
|
||
|
||
The Case 1 result reproduces the A2 output from Experiment 54W ("Priority between business growth and risk avoidance when they conflict" → "preferred priority between business growth and risk avoidance"). The same broadening pattern was reproduced across two tested runs under the same model and configuration, making it a repeatable candidate behaviour rather than a one-off observation.
|
||
|
||
### Documentation Updated
|
||
|
||
- `docs/design-evolution-log.md` — added full Experiment 54X entry
|
||
- `docs/current-handoff.md` — updated Return-to-Work note with 54X findings; applied 54W wording corrections
|
||
|
||
### Confirmation Host and Model Remained Unchanged
|
||
|
||
Host: `http://192.168.1.111:11434`. Model: `qwen-claude:latest`. Same as 54R–54X.
|
||
|
||
### Confirmation Semantic Instruction and Output Contract Remained Unchanged
|
||
|
||
The instruction was identical to Experiment 54S (no examples, no stronger coaching). The output contract remained `{ "clarificationTarget": "short statement" }` — unchanged from 54S.
|
||
|
||
### Confirmation Production Prompts and Schemas Remained Unchanged
|
||
|
||
No production prompts read or modified. No schemas changed. All inference calls used the experiment-specific semantic instruction defined in this test file.
|
||
|
||
### Confirmation Behaviour Selection Remained Unchanged
|
||
|
||
Behaviour Selection was not called or referenced. No integration with the selector occurred.
|
||
|
||
### Confirmation Graph and UI Remained Unchanged
|
||
|
||
No graph files read or modified. No UI code touched. The experiment is test-only.
|
||
|
||
### Confirmation No Clarification-Target Logic Entered Active Runtime
|
||
|
||
This experiment created one new test file only. No clarification-target logic entered any active runtime path, production module, or behaviour selection output.
|
||
|
||
### Return-to-Work Note (Experiment 54X)
|
||
|
||
54W showed the full clarification chain worked in the tested pair but Stage A2 broadened one target; 54X isolated target specificity using three clarification cases under the same 54S instruction — preference/constraint distinction was lost to priority framing (broadened), affordability definition stayed precise (specific), and private factual capacity stayed precise (specific). The same broadening pattern was reproduced across two tested runs under the same model and configuration, making it a repeatable candidate behaviour rather than a one-off observation. No question generation, answer resolution, Behaviour Selection, graph, or UI integration was attempted. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: `feature/user-workspace-ux-v0.7`. First test/file to inspect when resuming: `tests/reconstruction/semantic-clarification-target-specificity.test.js` for the full experiment and results. Status pending Rob's review.
|
||
|
||
## Experiment 54Y — Does a Broader Clarification Target Actually Change the User Question or Resolution? (2026-08-08)
|
||
|
||
### Objective
|
||
|
||
Test whether the specificity loss observed in Experiments 54W/54X actually matters downstream. If the clarification target shifts from the precise distinction "preference/trade-off versus hard constraint" to the broader "priority between growth and risk," does that materially change the question asked or the meaning resolved from the user's answer? Passive and test-only. No redesign of target generation, no integration with Behaviour Selection, graph, or UI.
|
||
|
||
### Hypothesis
|
||
|
||
The broader target may remain workable but could alter the user-facing distinction. Specifically:
|
||
- the precise target may ask whether risk avoidance is a hard boundary or a trade-off;
|
||
- the broader target may instead ask which objective has priority.
|
||
|
||
Those questions are related, but the user's answers need not mean exactly the same thing.
|
||
|
||
### Configuration
|
||
|
||
Host: `http://192.168.1.111:11434` (same as 54R–54X)
|
||
Model: `qwen-claude:latest` (same as 54R–54X)
|
||
|
||
### Number of Live Inference Calls
|
||
|
||
Exactly **4** live Ollama calls — one question per variant + one answer-resolution per variant.
|
||
|
||
### Context Used
|
||
|
||
- `docs/current-handoff.md`
|
||
- Experiment 54X only in `docs/design-evolution-log.md` (as historical context for the broadening observation)
|
||
- `tests/reconstruction/semantic-clarification-question.test.js` (for structural reference: instruction and output contract)
|
||
- `tests/reconstruction/semantic-clarification-answer-resolution.test.js` (for structural reference: instruction and output contract)
|
||
|
||
### Fixed Scenario
|
||
|
||
**Source:** "I want the business to grow, but I don't want to take on more risk."
|
||
|
||
**Fixed user answer:** "It's a hard constraint. I don't want any increase in risk."
|
||
|
||
### Variant A — Precise Target
|
||
|
||
**Clarification target:** whether avoiding additional risk is a preference/trade-off or a hard constraint.
|
||
|
||
This is the human-reviewed specific target.
|
||
|
||
### Variant B — Broadened Target
|
||
|
||
**Clarification target:** priority between business growth and risk avoidance when they conflict.
|
||
|
||
This mirrors the broader target observed in Experiments 54W and 54X.
|
||
|
||
---
|
||
|
||
### Stage 1 Results — Question Generation
|
||
|
||
| Variant | Generated Question |
|
||
|---|---|
|
||
| A (Precise) | Do you view avoiding additional risk as a hard constraint, or as a preference or trade-off? |
|
||
| B (Broadened) | When business growth and risk avoidance conflict, which do you prioritize? |
|
||
|
||
**Question analysis:** Variant A frames the question around whether avoiding risk is a hard constraint or a preference/trade-off — directly addressing the boundary distinction. Variant B frames it around priority ordering between growth and risk when they conflict — reframing the decision as relative importance rather than a boundary question. The model did not impose a reference wording for Variant B; it produced its natural framing from the broadened target.
|
||
|
||
### Stage 2 Results — Answer Resolution
|
||
|
||
| Variant | resolvedMeaning | targetResolved | remainingUncertainty |
|
||
|---|---|---|---|
|
||
| A (Precise) | Avoiding additional risk is established as a hard constraint. | true | null |
|
||
| B (Broadened) | Risk avoidance takes absolute priority over business growth when they conflict. | true | null |
|
||
|
||
**Resolution analysis:** Both variants produced `targetResolved: true` with `remainingUncertainty: null`. The resolved meanings differ in wording but convey materially equivalent meaning for downstream reasoning: "avoiding additional risk is a hard constraint" and "risk avoidance takes absolute priority over business growth when they conflict" establish the same boundary — no more risk will be accepted. Neither resolution introduced unsupported wider consequences.
|
||
|
||
### Question Equivalence Classification
|
||
|
||
**`questions_materially_different`**
|
||
|
||
The precise target asked whether avoiding extra risk is a trade-off/preference or a hard constraint (a boundary question). The broadened target asked which objective has priority when they conflict (an ordering question). These ask the user to resolve different conceptual distinctions. The distinction was lost as predicted.
|
||
|
||
### Resolution Equivalence Classification
|
||
|
||
**`resolutions_materially_equivalent`**
|
||
|
||
Despite different questions, both resolved meanings from the same fixed answer establish the same downstream meaning: the user will not accept additional risk. For downstream reasoning — determining what can and cannot be done — this is equivalent. With the explicit hard-constraint answer used in this test, both target variants converged on materially equivalent resolved meaning.
|
||
|
||
### Questions Answered
|
||
|
||
1. **What question did the precise target generate?** "Do you view avoiding additional risk as a hard constraint, or as a preference or trade-off?"
|
||
2. **What question did the broadened target generate?** "When business growth and risk avoidance conflict, which do you prioritize?"
|
||
3. **Did both questions ask the user to resolve the same underlying distinction?** No — one asked about boundary (constraint vs trade-off); the other asked about priority ordering.
|
||
4. **Did the broader target turn preference-versus-constraint into simple priority ordering?** Yes — it reframed the distinction as relative importance rather than an absolute boundary.
|
||
5. **What resolved meaning did Variant A produce from the fixed answer?** "Avoiding additional risk is established as a hard constraint." (targetResolved: true)
|
||
6. **What resolved meaning did Variant B produce from the same answer?** "Risk avoidance takes absolute priority over business growth when they conflict." (targetResolved: true)
|
||
7. **Were the two resolved meanings materially equivalent?** Yes — both establish that no additional risk will be accepted for growth.
|
||
8. **Did either variant leave remaining uncertainty?** No — both returned null, indicating full resolution of the target from the fixed answer.
|
||
9. **Did either variant introduce unsupported wider consequences?** No — neither inference extended beyond the source meaning and the user's explicit answer.
|
||
10. **Does the broadening materially affect downstream clarification in this tested scenario?** No — despite different questions, the same answer produced the same downstream meaning.
|
||
11. **Does this establish that broad targets are generally safe or unsafe?** No — only one scenario tested.
|
||
12. **Does this establish how target generation should be changed?** No — no fix designed from these results.
|
||
13. **Does this establish UI behaviour?** No — this is a clarification-target test only.
|
||
|
||
### Limitations
|
||
|
||
- Only one source scenario and one fixed answer were tested. Different sources may behave differently.
|
||
- Only one ambiguity pattern (preference/constraint) was tested for downstream consequence. Other patterns not assessed.
|
||
- Only one model configuration was used (qwen-claude:latest on 192.168.1.111:11434). Different models may behave differently.
|
||
- The semantic equivalence classification is based on structured heuristic checks supplemented by the test output — for definitive judgment, human review of the actual resolved meanings is required.
|
||
- One tested ambiguity pattern; broader safety/generalisation remains untested.
|
||
|
||
### Experiment Conclusion
|
||
|
||
**The broader target changed the clarification question but not the resolved meaning for the tested explicit answer.**
|
||
|
||
The specificity loss (broadening) was confirmed: the precise target generated a boundary question ("constraint vs trade-off") and the broadened target generated an ordering question ("which to prioritize"). These are materially different questions. However, from the fixed user answer ("It's a hard constraint. I don't want any increase in risk."), both targets resolved to materially equivalent downstream meaning: no additional risk will be accepted. With the explicit hard-constraint answer used in this test, both target variants converged on materially equivalent resolved meaning.
|
||
|
||
The key finding is: **Does the distinction we lost actually matter? — In this tested scenario, it did not.** Specificity loss is not automatically a failure; it depends on whether it changes downstream meaning. Whether this holds across other scenarios and ambiguity patterns remains untested.
|
||
|
||
### Focused Test Result
|
||
|
||
Both variants produced `targetResolved: true` with zero remaining uncertainty and zero unsupported inferences. Resolutions were materially equivalent despite questions being materially different. 4/4 live inference calls completed successfully (all tests passed).
|
||
|
||
### Historical Comparison Result
|
||
|
||
Variant A's resolution ("Avoiding additional risk is established as a hard constraint") matches the expected outcome from Experiment 54V Case 1 and the handoff summary. Variant B's resolution ("Risk avoidance takes absolute priority over business growth when they conflict") represents a coarser framing — but not an incorrect one — for downstream use. The result confirms that the coarser representation remains workable even when it loses the preference-versus-constraint granularity.
|
||
|
||
### Documentation Updated
|
||
|
||
- `docs/design-evolution-log.md` — added full Experiment 54Y entry; applied 54X wording corrections
|
||
- `docs/current-handoff.md` — updated with Experiment 54Y summary and new Return-to-Work note
|
||
|
||
### Confirmation Host and Model Remained Unchanged
|
||
|
||
Host: `http://192.168.1.111:11434`. Model: `qwen-claude:latest`. Same as 54R–54X.
|
||
|
||
### Confirmation Semantic Instructions and Output Contracts Remained Unchanged
|
||
|
||
Both question and resolution instructions were identical to those defined in Experiments 54U and 54V. Output contracts unchanged from 54U (`{ "question": "..." }`) and 54V (`{ "resolvedMeaning", "targetResolved", "remainingUncertainty" }`).
|
||
|
||
### Confirmation Production Prompts and Schemas Remained Unchanged
|
||
|
||
No production prompts read or modified. No schemas changed. All inference calls used the experiment-specific semantic instructions defined in this test file.
|
||
|
||
### Confirmation Behaviour Selection Remained Unchanged
|
||
|
||
Behaviour Selection was not called or referenced. No integration with the selector occurred.
|
||
|
||
### Confirmation Graph and UI Remained Unchanged
|
||
|
||
No graph files read or modified. No UI code touched. The experiment is test-only.
|
||
|
||
### Confirmation No Specificity-Consequence Logic Entered Active Runtime
|
||
|
||
This experiment created one new test file only. No specificity-consequence logic entered any active runtime path, production module, or behaviour selection output.
|
||
|
||
---
|
||
|
||
### Return-to-Work Note (Experiment 54Y)
|
||
|
||
Experiments 54W/54X reproduced a broader priority framing for preference-versus-hard-constraint ambiguity; 54Y tested whether that specificity loss actually changes downstream clarification. The precise target generated a question asking whether avoiding risk is a hard constraint or trade-off; the broadened target asked which to prioritize when growth and risk conflict. The same fixed answer produced materially equivalent resolved meanings from both variants, so broadening did not matter in this scenario. Broader safety/generalisation remains untested. Behaviour Selection, graph, UI, and production integration remained untouched. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7. First test/file to inspect when resuming: tests/reconstruction/semantic-clarification-specificity-consequence.test.js for the full experiment and results. Status pending Rob's review.
|
||
|
||
## Experiment 54Z — Does Target Broadening Change Resolution When the Answer Is Less Explicit? (2026-08-08)
|
||
|
||
### Objective
|
||
|
||
Experiment 54Y showed that with a very explicit hard-constraint answer ("It's a hard constraint. I don't want any increase in risk."), both precise and broadened clarification targets converged on materially equivalent resolved meaning — even though the generated questions were materially different.
|
||
|
||
This leaves one unresolved consequence:
|
||
|
||
> If the user's answer is less explicit, do those two different questions lead to materially different resolved meaning?
|
||
|
||
54Z tests that only. Passive and test-only. No redesign of target generation. No integration with Behaviour Selection, graph, or UI. No production code changes.
|
||
|
||
### Hypothesis
|
||
|
||
The precise and broadened questions may behave differently when the answer does not explicitly name the missing distinction. A weaker answer could:
|
||
|
||
- remain correctly unresolved under the precise question;
|
||
- but be interpreted as a resolved priority decision under the broader question.
|
||
|
||
If that happens, target broadening has a real downstream consequence. If both variants preserve equivalent uncertainty, the broadening may be less consequential than expected.
|
||
|
||
### Configuration
|
||
|
||
Host: `http://192.168.1.111:11434` (same as 54R–54Y)
|
||
Model: `qwen-claude:latest` (same as 54R–54Y)
|
||
|
||
### Number of Live Inference Calls
|
||
|
||
Exactly **4** live Ollama calls — one answer-resolution per variant × two weaker answers. No question-generation calls (questions are fixed from Experiment 54Y).
|
||
|
||
### Context Used
|
||
|
||
- `docs/current-handoff.md`
|
||
- Experiment 54Y in `docs/design-evolution-log.md` (as basis for the unresolved consequence)
|
||
- `tests/reconstruction/semantic-clarification-specificity-consequence.test.js` (structural reference: question generation and answer resolution helpers)
|
||
- `tests/reconstruction/semantic-clarification-answer-resolution.test.js` (structural reference: instruction and output contract)
|
||
|
||
### Fixed Source
|
||
|
||
> I want the business to grow, but I don't want to take on more risk.
|
||
|
||
### Variant A — Precise Target
|
||
|
||
**Clarification target:** whether avoiding additional risk is a preference/trade-off or a hard constraint.
|
||
**Fixed question:** Do you view avoiding additional risk as a hard constraint, or as a preference or trade-off?
|
||
|
||
### Variant B — Broadened Target
|
||
|
||
**Clarification target:** priority between business growth and risk avoidance when they conflict.
|
||
**Fixed question:** When business growth and risk avoidance conflict, which do you prioritize?
|
||
|
||
---
|
||
|
||
### Answer 1 — Priority Without Constraint Meaning
|
||
|
||
**User answer:** "Risk matters more to me."
|
||
|
||
**Expected behavior (human-reviewed):**
|
||
This answer does not clearly establish whether risk avoidance is a hard constraint or merely a stronger preference. For Variant A, the precise target should therefore remain unresolved. For Variant B, the answer may legitimately resolve the priority target as "risk avoidance has higher priority than growth."
|
||
|
||
### Answer 1 Results
|
||
|
||
| Variant | resolvedMeaning | targetResolved | remainingUncertainty |
|
||
|---|---|---|---|
|
||
| A (Precise) | The user treats avoiding additional risk as a strong priority or preference rather than an absolute, non-negotiable constraint. | true | null |
|
||
| B (Broadened) | Risk avoidance is prioritized over business growth when they conflict. | true | null |
|
||
|
||
### Answer 1 Analysis
|
||
|
||
Variant A interprets "Risk matters more to me" as meaning risk avoidance is a **strong preference/priority rather than an absolute constraint** — this maps correctly onto the precise target (preference/trade-off vs hard constraint). The model marked targetResolved=true because it interpreted the answer as settling the distinction toward "preference/trade-off."
|
||
|
||
Variant B interprets the same answer as meaning **risk avoidance has higher priority over growth when they conflict** — this maps correctly onto the broadened target (priority ordering).
|
||
|
||
**Classification: resolutions_materially_equivalent**
|
||
|
||
Both variants map the weak answer to a preference/priority-over-constraint interpretation. Neither resolves to "hard constraint." The resolved meanings use different framing but preserve the same downstream reasoning state: risk is not an absolute boundary, it is a prioritized consideration. For downstream use (what can/cannot be done), both produce equivalent uncertainty about whether risk could ever be accepted.
|
||
|
||
Both variants were flagged as `potential_erasal_of_uncertainty` because the answer was weak and both returned targetResolved=true with no remainingUncertainty — neither explicitly preserved the ambiguity about what "matters more" means in edge cases. **The precise target may have erased uncertainty by inferring that "Risk matters more to me" means risk is not a hard constraint; that conclusion was not explicitly supplied by the user.**
|
||
|
||
---
|
||
|
||
### Answer 2 — Conditional Trade-Off
|
||
|
||
**User answer:** "I'd normally avoid more risk, but for the right opportunity I might accept some."
|
||
|
||
**Expected behavior (human-reviewed):**
|
||
This answer indicates risk avoidance is not an absolute hard constraint; some trade-off may be acceptable depending on the opportunity. For Variant A, this should resolve away from "hard constraint" while retaining conditionality. For Variant B, it may establish that risk is normally prioritized but can yield to growth in some cases.
|
||
|
||
### Answer 2 Results
|
||
|
||
| Variant | resolvedMeaning | targetResolved | remainingUncertainty |
|
||
|---|---|---|---|
|
||
| A (Precise) | Avoiding additional risk is a preference or trade-off rather than a hard constraint. | true | null |
|
||
| B (Broadened) | Default priority is risk avoidance, with a conditional willingness to accept some risk for highly suitable opportunities. | false | It remains unclear how "the right opportunity" is defined and which factor strictly takes precedence when a specific growth opportunity carries significant risk. |
|
||
|
||
### Answer 2 Analysis
|
||
|
||
**This is the critical divergence.** Variant A collapses the conditional nature of the answer into a simple preference-vs-constraint resolution. The model says "preference or trade-off rather than a hard constraint" — but loses the key information that there are conditions (the right opportunity) under which even this preference could shift. This was flagged as `forced_certainty_detected` and `potential_erasal_of_uncertainty`.
|
||
|
||
Variant B preserves the conditionality ("conditional willingness") and correctly marks targetResolved=false because the answer does not establish a stable priority — the priority shifts depending on context. It also identifies remaining uncertainty about what constitutes "the right opportunity."
|
||
|
||
**Classification: resolutions_materially_different**
|
||
|
||
This is a material divergence. Variant A erases the conditional nature of the user's stated position and produces a flat preference-versus-constraint resolution. Variant B preserves both the default-priority-and-conditional structure AND the remaining uncertainty about when conditions change. For downstream reasoning, this matters because:
|
||
- Under Variant A's meaning: risk avoidance = preference/trade-off → may be willing to accept risk in some cases (inferred)
|
||
- Under Variant B's meaning: default priority risk, conditionally willing → conditional willingness is preserved explicitly
|
||
|
||
**However**, the divergence exists primarily in remainingUncertainty content, not in the resolvedMeaning itself. Both agree that risk avoidance is not a hard constraint. The difference is in whether the model preserves "there are conditions we don't yet understand" versus collapsing everything to "not a hard constraint."
|
||
|
||
---
|
||
|
||
### Questions Answered
|
||
|
||
1. **For Answer 1, did Variant A correctly preserve uncertainty about preference versus hard constraint?** Partially. Variant A mapped the answer toward "preference/trade-off rather than absolute constraint" but marked it as fully resolved (targetResolved=true) with no remainingUncertainty — erasing the ambiguity that "matters more" doesn't define a boundary.
|
||
|
||
2. **For Answer 1, did Variant B resolve the broader priority target?** Yes. The broadened target ("priority between growth and risk") was resolved as "risk avoidance is prioritized over business growth when they conflict." This is a correct mapping from the weak answer to the priority frame.
|
||
|
||
3. **Did Answer 1 therefore create materially different resolution states between A and B?** No. Both map to the same downstream state: risk avoidance is not an absolute boundary but a stronger consideration than growth. The resolutions are materially equivalent for downstream reasoning about what can/cannot be done.
|
||
|
||
4. **For Answer 2, did Variant A correctly identify that risk avoidance is not an absolute hard constraint?** Partially correct on the outcome (not a hard constraint) but failed to preserve conditionality — the "might accept some" conditional was collapsed into a flat preference resolution with no remaining uncertainty.
|
||
|
||
5. **For Answer 2, did Variant B preserve the conditional nature of the priority?** Yes. Variant B preserved both the default-priority-and-conditional structure and identified remaining uncertainty about when conditions shift.
|
||
|
||
6. **Were the Answer 2 resolution states materially equivalent or different?** Different. Variant A erased conditionality; Variant B preserved it plus remainingUncertainty. This is a material divergence for downstream reasoning state.
|
||
|
||
7. **Did either variant force a weak answer into stronger meaning than the user supplied?** Yes — Variant A for Answer 2 collapsed conditional willingness ("might accept some") into a flat preference classification, erasing the conditionality layer.
|
||
|
||
8. **Did either variant erase uncertainty that should remain?** Yes — Variant A for both answers showed `potential_erasal_of_uncertainty`. For Answer 1, "Risk matters more to me" became a fully resolved preference (no remainingUncertainty). For Answer 2, conditionality was erased.
|
||
|
||
9. **Does target broadening have a material downstream consequence when answers are less explicit in these tested cases?** Yes — specifically for Answer 2 (conditional trade-off). The precise target question led the model to map to a flat preference-vs-constraint resolution and erase conditionality. The broadened target preserved conditional structure. This means target broadening has a real, asymmetrical consequence: the broadened question can actually preserve nuance that the precise question erases in this case.
|
||
|
||
10. **Does this establish that precise targets are always required?** No — Answer 1 showed no material divergence, and for Answer 2 the broader target preserved more nuance than the precise one. Neither is universally better.
|
||
|
||
11. **Does this establish how target-generation logic should be changed?** No — only two answers tested; neither variant was consistently better; no fix designed from these results.
|
||
|
||
12. **Does this establish UI behaviour?** No — this is a clarification-target test only.
|
||
|
||
### Limitations
|
||
|
||
- Only one source scenario and two user answers were tested. Different sources may behave differently.
|
||
- Only one ambiguity pattern (preference/constraint) was tested with weak answers. Other patterns not assessed.
|
||
- Only one model configuration was used (qwen-claude:latest on 192.168.1.111:11434). Different models may behave differently.
|
||
- The asymmetric finding (broadened target preserving more nuance for Answer 2) is surprising and warrants further testing with additional answers that include explicit conditionality.
|
||
- Two tested cases; broader generalisation remains untested.
|
||
|
||
### Experiment Conclusion
|
||
|
||
**Target broadening changed wording but not material resolution under weaker answers — except when the answer contained explicit conditionality.**
|
||
|
||
For Answer 1 ("Risk matters more to me."), both variants produced materially equivalent downstream meaning: risk avoidance is stronger than growth but not an absolute constraint. The broader target did not create a material divergence here.
|
||
|
||
For Answer 2 ("I'd normally avoid more risk, but for the right opportunity I might accept some."), the variants diverged. Variant A (precise) collapsed conditionality into a flat preference resolution and erased uncertainty. Variant B (broadened) preserved conditional structure and remaining uncertainty about what constitutes "the right opportunity."
|
||
|
||
**Unexpected finding:** The broadened target preserved more nuance than the precise target for the conditional answer. Neither framing was consistently superior across the two tested weaker answers.
|
||
|
||
### Focused Test Result
|
||
|
||
All 4 live inference calls completed successfully (all tests passed). Answer 1: materially equivalent resolutions from both variants. Answer 2: materially different resolutions — Variant A erased conditionality; Variant B preserved it. One forced certainty detection (Variant A on Answer 2).
|
||
|
||
### Historical Comparison Result
|
||
|
||
Compared to Experiment 54Y's explicit hard-constraint answer (where both variants converged), 54Z shows that convergence is fragile with weaker answers. Both tested weak answers confirmed the hypothesis: weak answers expose differences between precise and broadened targets, but only under specific content conditions. Neither variant was consistently superior across the two answers tested.
|
||
|
||
### Documentation Updated
|
||
|
||
- `docs/design-evolution-log.md` — added full Experiment 54Z entry; applied 54Y wording corrections
|
||
- `docs/current-handoff.md` — updated with Experiment 54Z summary and new Return-to-Work note
|
||
|
||
### Confirmation Host and Model Remained Unchanged
|
||
|
||
Host: `http://192.168.1.111:11434`. Model: `qwen-claude:latest`. Same as 54R–54Y.
|
||
|
||
### Confirmation Semantic Instructions and Output Contracts Remained Unchanged
|
||
|
||
Answer-resolution instruction identical to Experiment 54V. Output contract unchanged from 54V (`{ "resolvedMeaning", "targetResolved", "remainingUncertainty" }`).
|
||
|
||
### Confirmation Production Prompts and Schemas Remained Unchanged
|
||
|
||
No production prompts read or modified. No schemas changed. All inference calls used the experiment-specific semantic instructions defined in this test file.
|
||
|
||
### Confirmation Behaviour Selection Remained Unchanged
|
||
|
||
Behaviour Selection was not called or referenced. No integration with the selector occurred.
|
||
|
||
### Confirmation Graph and UI Remained Unchanged
|
||
|
||
No graph files read or modified. No UI code touched. The experiment is test-only.
|
||
|
||
### Confirmation No Weak-Answer Consequence Logic Entered Active Runtime
|
||
|
||
This experiment created one new test file only. No weak-answer consequence logic entered any active runtime path, production module, or behaviour selection output.
|
||
|
||
---
|
||
|
||
## Experiment 55A — Can the Model Preserve Uncertainty From Weak Clarification Answers? (2026-08-08)
|
||
|
||
### Objective
|
||
|
||
First, tighten Experiment 54Z so its conclusion does not over-generalise from two weaker answers.
|
||
|
||
Then isolate the strongest unresolved issue from 54Z:
|
||
|
||
> When a clarification answer is weak or conditional, can the model preserve uncertainty instead of forcing the answer into a stronger resolved meaning?
|
||
|
||
This experiment is about answer interpretation only. Do not compare precise versus broadened targets. Do not test question wording. Do not integrate with Behaviour Selection, graph, or UI.
|
||
|
||
### Hypothesis
|
||
|
||
The model may be too eager to convert weak clarification answers into resolved meaning. A better behaviour would preserve uncertainty when the answer does not actually settle the supplied target. If the model consistently keeps weak answers unresolved, the 54Z over-resolution may have been incidental. If it repeatedly forces resolution, that becomes a clearer limitation of the answer-resolution step itself.
|
||
|
||
### Configuration
|
||
|
||
Host: `http://192.168.1.111:11434` (same as all prior experiments)
|
||
Model: `qwen-claude:latest` (same as all prior experiments)
|
||
|
||
### Number of Live Inference Calls
|
||
|
||
Exactly **4** live Ollama calls — one answer-resolution per case. Exactly 4 cases of varying strength against the same source/target/question.
|
||
|
||
### Context Used
|
||
|
||
- `docs/current-handoff.md`
|
||
- Experiment 54Z in `docs/design-evolution-log.md` (as basis for isolating the unresolved issue)
|
||
- `tests/reconstruction/semantic-clarification-weak-answer-consequence.test.js` (structural reference)
|
||
- `tests/reconstruction/semantic-clarification-answer-resolution.test.js` (structural reference: instruction and output contract)
|
||
|
||
### Semantic Instruction Unchanged
|
||
|
||
> Use the user's clarification answer only to resolve the supplied clarification target. State the meaning now established by that answer. Mark targetResolved true only when the answer settles the target. Put any uncertainty that remains specifically about that target into remainingUncertainty; otherwise return null. Do not infer wider consequences, rewrite unrelated source meaning, recommend action, or generate another question.
|
||
|
||
No examples added. No stronger coaching attempted. Same instruction as Experiment 54V.
|
||
|
||
### Output Contract Unchanged
|
||
|
||
```json
|
||
{
|
||
"resolvedMeaning": "short statement",
|
||
"targetResolved": true,
|
||
"remainingUncertainty": "short statement or null"
|
||
}
|
||
```
|
||
|
||
### Fixed Source
|
||
|
||
> I want the business to grow, but I don't want to take on more risk.
|
||
|
||
### Clarification Target
|
||
|
||
> whether avoiding additional risk is a preference/trade-off or a hard constraint.
|
||
|
||
### Clarification Question
|
||
|
||
> Do you view avoiding additional risk as a hard constraint, or as a preference or trade-off?
|
||
|
||
---
|
||
|
||
### Case 1 — Fully Explicit Hard Constraint
|
||
|
||
**User answer:** "It's a hard constraint. I don't want any increase in risk."
|
||
|
||
**Model output:**
|
||
- resolvedMeaning: "Avoiding additional risk is a hard constraint."
|
||
- targetResolved: true
|
||
- remainingUncertainty: null
|
||
|
||
**Classification: resolution_correct**
|
||
|
||
The model correctly resolved the explicit hard constraint with appropriate strength and no spurious uncertainty. This is the positive control — it behaved as expected.
|
||
|
||
---
|
||
|
||
### Case 2 — Weak Priority Statement
|
||
|
||
**User answer:** "Risk matters more to me."
|
||
|
||
**Model output:**
|
||
- resolvedMeaning: "Avoiding additional risk is prioritized as a strong preference or guiding principle rather than a rigid, non-negotiable constraint."
|
||
- targetResolved: true
|
||
- remainingUncertainty: null
|
||
|
||
**Classification: uncertainty_overresolved**
|
||
|
||
The model set `targetResolved=true` and produced a meaning stronger than the user supplied. "Risk matters more to me" establishes relative importance but does not determine whether avoiding additional risk is a hard constraint or a strong preference/trade-off. The model inferred "not a rigid, non-negotiable constraint" from a statement that only said risk was more important — which could be consistent with a strong preference that still functions as a de facto hard boundary in practice. This over-resolution matches the concern flagged in Experiment 54Z but isolated it here under a single fixed target to confirm the behaviour is real.
|
||
|
||
The model produced stronger meaning than the user supplied by declaring what risk avoidance is *not* (a rigid constraint) when the user only stated what it *is* relative to (more important than growth). This establishes that uncertainty should have been preserved with remainingUncertainty about whether avoiding additional risk is a preference/trade-off or a hard constraint.
|
||
|
||
---
|
||
|
||
### Case 3 — Conditional Trade-Off
|
||
|
||
**User answer:** "I'd normally avoid more risk, but for the right opportunity I might accept some."
|
||
|
||
**Model output:**
|
||
- resolvedMeaning: "Avoiding additional risk is a preference or trade-off rather than a hard constraint."
|
||
- targetResolved: true
|
||
- remainingUncertainty: null
|
||
|
||
**Classification: resolution_correct with conditional qualification loss**
|
||
|
||
The target category was resolved correctly, but the answer's conditional qualification was lost. The user said "I might accept some for the right opportunity" — a specific conditional exception — which became "preference or trade-off rather than a hard constraint." The target (whether risk avoidance is preference/trade-off or hard constraint) was settled, but the nuance of how it functions conditionally was erased from the resolved meaning.
|
||
|
||
---
|
||
|
||
### Case 4 — Non-Answer / Insufficient Clarification
|
||
|
||
**User answer:** "I'm not really sure."
|
||
|
||
**Model output:**
|
||
- resolvedMeaning: "The user has not yet determined if avoiding additional risk is a hard constraint or a preference/trade-off."
|
||
- targetResolved: false
|
||
- remainingUncertainty: "It remains unclear whether the user considers avoiding additional risk a hard constraint or a preference/trade-off."
|
||
|
||
**Classification: uncertainty_preserved**
|
||
|
||
The model correctly remained unresolved without inventing any position. The resolvedMeaning accurately describes what was supplied (nothing) and remainingUncertainty states that the distinction between hard constraint and preference/trade-off remains unresolved. This is honest behaviour.
|
||
|
||
---
|
||
|
||
### Evaluation Summary
|
||
|
||
| Case | Answer | Resolution Status | Classification |
|
||
|---|---|---|---|
|
||
| 1 | Explicit hard constraint | Correctly resolved | resolution_correct |
|
||
| 2 | Weak priority ("Risk matters more to me.") | Over-resolved to targetResolved=true | uncertainty_overresolved |
|
||
| 3 | Conditional trade-off | Resolved, conditionality lost | resolution_correct (with note) |
|
||
| 4 | Non-answer ("I'm not really sure.") | Correctly unresolved | uncertainty_preserved |
|
||
|
||
**Classification counts:**
|
||
- resolution_correct: 2
|
||
- uncertainty_preserved: 1
|
||
- uncertainty_overresolved: 1
|
||
- resolution_failed: 0
|
||
|
||
---
|
||
|
||
### Questions Answered
|
||
|
||
1. **Did Case 1 correctly resolve the explicit hard constraint?** Yes — targetResolved=true, appropriate meaning strength, no spurious uncertainty.
|
||
|
||
2. **Did Case 2 preserve uncertainty rather than infer that risk is not a hard constraint?** No — the model over-resolved, setting targetResolved=true and inferring "not a rigid, non-negotiable constraint" from a weak priority statement. This is the key finding: one tested weak answer received stronger meaning than the user supplied.
|
||
|
||
3. **Did Case 3 preserve the conditional nature of the trade-off?** Partially — the model correctly resolved the target (risk avoidance is not a hard constraint) but flattened the conditional qualification ("might accept some for the right opportunity") into flat "preference or trade-off" language. The conditional layer was lost even though target resolution was correct.
|
||
|
||
4. **Did Case 4 correctly remain unresolved?** Yes — no invented position, appropriate remainingUncertainty stating the distinction remains unresolved.
|
||
|
||
5. **How many cases were:**
|
||
- resolution_correct: 2
|
||
- uncertainty_preserved: 1
|
||
- uncertainty_overresolved: 1
|
||
- resolution_failed: 0
|
||
|
||
6. **Did any answer get stronger meaning than the user supplied?** Yes — Case 2 ("Risk matters more to me.") received a stronger interpretation than it justified. The model inferred "not a rigid, non-negotiable constraint" from a statement that only established relative priority.
|
||
|
||
7. **Did any unresolved answer incorrectly return targetResolved true?** No — Case 4 (the only truly unresolved case) correctly returned false. Case 2 over-resolved but did not remain unresolved.
|
||
|
||
8. **Did any resolved answer incorrectly retain uncertainty?** No — both resolved cases (1 and 3) correctly returned null for remainingUncertainty.
|
||
|
||
9. **Does the answer-resolution step appear biased toward resolution in these tested cases?** Yes — Case 2 demonstrates that a weak priority statement can be over-resolved to a definitive classification ("not a constraint") when it should have remained unresolved. One out of four cases showed this bias, but it appeared on the weakest-answer type where uncertainty preservation matters most.
|
||
|
||
10. **Does this establish how answer-resolution logic should be changed?** No — one weak-priority case over-resolved; this does not justify a broad change without broader testing.
|
||
11. **Does this establish how the graph should represent unresolved clarification?** No — the unresolved representation question is separate from whether the model *should* remain unresolved.
|
||
12. **Does this establish how Behaviour Selection should react?** No — this experiment did not integrate with Behaviour Selection.
|
||
|
||
### Limitations
|
||
|
||
- Only one source scenario and four answer types were tested. Different sources may behave differently.
|
||
- Only one weak-answer pattern (priority without constraint meaning) over-resolved; other patterns not assessed.
|
||
- Only one model configuration was used (qwen-claude:latest on 192.168.1.111:11434). Different models may behave differently.
|
||
- Case 3 showed conditionality loss that is subtler than over-resolution — it resolved correctly but flattened nuance. This pattern warrants further testing with additional conditional answers.
|
||
- Four cases tested; broader generalisation remains untested.
|
||
|
||
### Experiment Conclusion
|
||
|
||
**Clarification-answer resolution preserved uncertainty appropriately across some cases but over-resolved one weak answer.**
|
||
|
||
Case 1 (explicit hard constraint) and Case 4 (non-answer) behaved honestly — the explicit case resolved, the empty case remained unresolved. This is the expected baseline.
|
||
|
||
Case 2 (weak priority: "Risk matters more to me.") demonstrates the core limitation: the model over-resolved a weak answer, converting relative priority into a definitive negative ("not a rigid, non-negotiable constraint") when only relative importance was supplied. The model set targetResolved=true and erased uncertainty that should have been preserved.
|
||
|
||
Case 3 (conditional trade-off) resolved the target correctly but lost the conditional qualification in the process — "might accept some for the right opportunity" was flattened to "preference or trade-off rather than a hard constraint." This is a subtler form of meaning inflation: correct resolution with erasure of nuance.
|
||
|
||
Two of the four tested answers showed loss of nuance: one was over-resolved (Case 2) and one retained the correct target category while losing conditional qualification (Case 3). The remaining two cases behaved correctly (Cases 1 and 4). With qwen-claude:latest and the current semantic instruction, uncertainty preservation works for truly empty answers but is unreliable for weak-priority answers.
|
||
|
||
### Focused Test Result
|
||
|
||
All 4 live inference calls completed successfully. One over-resolution (Case 2), one conditionality loss (Case 3), two honest classifications (Cases 1 and 4). Total: ~62s, average: ~15.5s per call, fastest: 8.2s, slowest: 19.3s.
|
||
|
||
### Historical Comparison Result
|
||
|
||
Compared to Experiment 54Z's finding that both variants over-resolved Case 1 ("Risk matters more to me."), Experiment 55A confirms this is a persistent issue under the same semantic instruction and model — not a side-effect of target framing. When a single fixed target was used, the weak priority answer still over-resolved (targetResolved=true with inferred "not a constraint" meaning). The same over-resolution reproduced with a fixed target, so target broadening is not required for the failure to occur.
|
||
|
||
### Documentation Updated
|
||
|
||
- `docs/design-evolution-log.md` — added full Experiment 55A entry; applied 54Z wording corrections
|
||
- `docs/current-handoff.md` — updated with Experiment 55A summary and new Return-to-Work note
|
||
|
||
### Confirmation Host and Model Remained Unchanged
|
||
|
||
Host: `http://192.168.1.111:11434`. Model: `qwen-claude:latest`. Same as all prior experiments.
|
||
|
||
### Confirmation Semantic Instruction and Output Contract Remained Unchanged
|
||
|
||
Answer-resolution instruction identical to Experiment 54V. No stronger coaching, no added examples. Output contract unchanged from 54V (`{ resolvedMeaning, targetResolved, remainingUncertainty }`).
|
||
|
||
### Confirmation Production Prompts and Schemas Remained Unchanged
|
||
|
||
No production prompts read or modified. No schemas changed. All inference calls used the experiment-specific semantic instructions defined in this test file.
|
||
|
||
### Confirmation Behaviour Selection Remained Unchanged
|
||
|
||
Behaviour Selection was not called or referenced. No integration with the selector occurred.
|
||
|
||
### Confirmation Graph and UI Remained Unchanged
|
||
|
||
No graph files read or modified. No UI code touched. The experiment is test-only.
|
||
|
||
### Confirmation No Uncertainty-Preservation Logic Entered Active Runtime
|
||
|
||
This experiment created one new test file only. No uncertainty-preservation logic entered any active runtime path, production module, or behaviour selection output.
|
||
|
||
---
|
||
|
||
### Return-to-Work Note (Experiment 55A)
|
||
|
||
54Z showed weaker answers can produce different downstream resolution states under different framings, suggesting target broadening matters; 55A isolated the answer-resolution step using one fixed target and four answers of varying strength (explicit hard constraint, weak priority, conditional trade-off, non-answer) to test whether uncertainty preservation holds independently of framing. The explicit case resolved correctly, the non-answer remained honestly unresolved, but the weak-priority case over-resolved by setting targetResolved=true and inferring "not a constraint" from relative importance alone. Conditional language was also flattened even when resolution was correct. Uncertainty preservation remains unreliable for weak-priority answers. Graph, Behaviour Selection, UI, and production integration remain untouched. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7. First file to inspect when resuming: tests/reconstruction/semantic-clarification-uncertainty-preservation.test.js for the full experiment and results. Status pending Rob's review.
|
||
|
||
---
|
||
|
||
## Experiment 55B — Separate Answer Meaning from Resolution Judgement (2026-08-08)
|
||
|
||
### Objective
|
||
|
||
Experiment 55A showed one weak answer was over-resolved and one conditional answer lost nuance. The remaining question: is the distortion introduced when the model restates the answer's meaning, or only when it decides whether the clarification target is resolved?
|
||
|
||
55B separates these two steps using independent calls per case:
|
||
- **Mode A** (meaning-only): "State what the user's answer establishes. Preserve uncertainty and qualification exactly." No resolution decision.
|
||
- **Mode B** (resolution): The same instruction and contract as Experiment 54V/55A.
|
||
|
||
### Hypothesis
|
||
|
||
The model may preserve weak and conditional meaning correctly when asked only to restate what the user established, and only become over-confident when asked whether the clarification target is resolved. If so, the problem lies in converting meaning into a `targetResolved` judgement, not in interpreting the answer itself. If the meaning-only output already strengthens or flattens the answer, then the problem occurs earlier.
|
||
|
||
### Configuration
|
||
|
||
Host: `http://192.168.1.111:11434` (same as all prior experiments)
|
||
Model: `qwen-claude:latest` (same as all prior experiments)
|
||
|
||
### Number of Live Inference Calls
|
||
|
||
Exactly **6** live Ollama calls — one meaning-only call and one resolution call per case, three cases. The calls are independent; Mode A output is never fed into Mode B.
|
||
|
||
### Fixed Source / Target / Question (identical to 55A)
|
||
|
||
- **Source:** "I want the business to grow, but I don't want to take on more risk."
|
||
- **Clarification target:** whether avoiding additional risk is a preference/trade-off or a hard constraint.
|
||
- **Clarification question:** Do you view avoiding additional risk as a hard constraint, or as a preference or trade-off?
|
||
|
||
### Mode A instruction
|
||
|
||
> State only what the user's answer establishes in relation to the clarification question. Preserve uncertainty, conditionality, and qualification exactly as supplied. Do not decide whether the clarification target is resolved. Do not infer what the user did not say. Do not recommend action or generate another question.
|
||
>
|
||
> Return valid JSON only in this shape: { "answerMeaning": "short statement" }
|
||
|
||
### Mode A output contract
|
||
|
||
```json
|
||
{
|
||
"answerMeaning": "short statement"
|
||
}
|
||
```
|
||
|
||
### Mode B instruction unchanged
|
||
|
||
Same as Experiment 54V / 55A resolution instruction. Output contract unchanged (`resolvedMeaning`, `targetResolved`, `remainingUncertainty`).
|
||
|
||
---
|
||
|
||
### Case 1 — Weak Priority
|
||
|
||
**Answer:** "Risk matters more to me."
|
||
|
||
**Mode A result:**
|
||
- answerMeaning: "The user indicates that avoiding additional risk holds greater importance to them than other factors."
|
||
- Classification: **meaning_preserved**
|
||
|
||
The meaning preserved relative importance without deciding whether it is a hard constraint or not. No strengthening or loss detected.
|
||
|
||
**Mode B result:**
|
||
- resolvedMeaning: "Avoiding additional risk is established as a strongly prioritized preference over growth, rather than an explicitly stated boundary."
|
||
- targetResolved: false
|
||
- remainingUncertainty: "Whether this priority functions as a non-negotiable hard constraint or remains a flexible trade-off in certain situations."
|
||
- Classification: **resolution_correct**
|
||
|
||
The resolution stayed unresolved with appropriate uncertainty. Interestingly, the resolved meaning is slightly stronger than the Mode A output ("strongly prioritized preference" vs "greater importance"), but `targetResolved` was correctly false — no over-resolution occurred this time.
|
||
|
||
---
|
||
|
||
### Case 2 — Conditional Trade-Off
|
||
|
||
**Answer:** "I'd normally avoid more risk, but for the right opportunity I might accept some."
|
||
|
||
**Mode A result:**
|
||
- answerMeaning: "The user indicates that avoiding additional risk is generally a preference that may become a trade-off under specific conditions, rather than an absolute constraint."
|
||
- Classification: **meaning_preserved**
|
||
|
||
Both the normal preference and the conditional exception ("under specific conditions") were preserved. The meaning captures all of what the user established without flattening.
|
||
|
||
**Mode B result:**
|
||
- resolvedMeaning: "Avoiding additional risk is a preference or trade-off rather than a hard constraint."
|
||
- targetResolved: true
|
||
- remainingUncertainty: null
|
||
- Classification: **resolution_meaning_loss**
|
||
|
||
The target was correctly resolved (`targetResolved=true`). However, the conditional qualification ("for the right opportunity I might accept some") was **flattened** into a flat "preference or trade-off" statement. Mode A preserved "generally a preference that may become a trade-off under specific conditions"; Mode B collapsed it to "a preference or trade-off."
|
||
|
||
This is a direct comparison within the same case: **meaning was preserved in Mode A but lost during resolution.**
|
||
|
||
---
|
||
|
||
### Case 3 — Non-Answer
|
||
|
||
**Answer:** "I'm not really sure."
|
||
|
||
**Mode A result:**
|
||
- answerMeaning: "The user is uncertain whether avoiding additional risk is a hard constraint or a preference or trade-off."
|
||
- Classification: **meaning_preserved**
|
||
|
||
Simple uncertainty was preserved accurately. No position was invented.
|
||
|
||
**Mode B result:**
|
||
- resolvedMeaning: "The user is unsure whether avoiding additional risk is a hard constraint or a preference/trade-off."
|
||
- targetResolved: false
|
||
- remainingUncertainty: "Whether avoiding additional risk is a hard constraint or a preference/trade-off remains unclear."
|
||
- Classification: **resolution_correct**
|
||
|
||
Correctly unresolved. No meaning distortion detected in either mode.
|
||
|
||
---
|
||
|
||
### Evaluation Summary
|
||
|
||
| Case | Answer | Mode A | Mode A Class | Mode B | Mode B Class |
|
||
|------|--------|--------|-------------|--------|-------------|
|
||
| 1 | "Risk matters more to me." | Preserved relative priority | meaning_preserved | Correctly unresolved | resolution_correct |
|
||
| 2 | "I'd normally avoid more risk, but for the right opportunity I might accept some." | Preserved conditional qualification | meaning_preserved | Resolved but flattened conditionality | resolution_meaning_loss |
|
||
| 3 | "I'm not really sure." | Preserved uncertainty | meaning_preserved | Correctly unresolved | resolution_correct |
|
||
|
||
**Meaning counts (Mode A):**
|
||
- meaning_preserved: 3
|
||
- meaning_strengthened: 0
|
||
- meaning_lost: 0
|
||
|
||
**Resolution counts (Mode B):**
|
||
- resolution_correct: 2
|
||
- resolution_meaning_loss: 1
|
||
- resolution_overresolved: 0
|
||
- resolution_underresolved: 0
|
||
|
||
---
|
||
|
||
### Questions Answered
|
||
|
||
1. **Did Case 1 Mode A preserve only relative priority without deciding hard-constraint status?** Yes — "greater importance to them than other factors" preserves the relative priority without declaring anything about hard constraint status.
|
||
|
||
2. **Did Case 1 Mode B over-resolve the target again?** No — this run returned `targetResolved=false` with appropriate remaining uncertainty. The resolved meaning was slightly stronger ("strongly prioritized preference") but did not cross into definitive classification. (Note: this differs from the 55A run on the same case, which had over-resolved to `targetResolved=true`. This may indicate some instability in the resolution step across runs.)
|
||
|
||
3. **Did Case 2 Mode A preserve the conditional "for the right opportunity" qualification?** Yes — "generally a preference that may become a trade-off under specific conditions" preserves both the normal stance and the conditional exception.
|
||
|
||
4. **Did Case 2 Mode B preserve or flatten that same conditionality?** Flattened. Mode B collapsed "I'd normally avoid more risk, but for the right opportunity I might accept some" into "a preference or trade-off rather than a hard constraint." The conditional qualification ("for the right opportunity") was lost during resolution.
|
||
|
||
5. **Did Case 3 Mode A preserve simple uncertainty?** Yes — "The user is uncertain whether avoiding additional risk is a hard constraint or a preference or trade-off" preserves the lack of position without inventing one.
|
||
|
||
6. **Did Case 3 Mode B correctly remain unresolved?** Yes — `targetResolved=false` with accurate remainingUncertainty. No meaning distortion in either mode.
|
||
|
||
7. **In any case, was meaning already distorted before the resolution judgement?** No — all three cases preserved their meaning accurately in Mode A (meaning_preserved: 3). The first material information loss appeared only during the resolution step.
|
||
|
||
8. **In any case, did meaning remain accurate in Mode A but become stronger or flatter in Mode B?** Yes — Case 2 is the clearest example. Mode A preserved "generally a preference that may become a trade-off under specific conditions"; Mode B flattened it to "a preference or trade-off."
|
||
|
||
9. **Does this isolate the failure to the resolution judgement?** Partially yes, for the case of conditional meaning loss. The comparison within Case 2 (same answer, same model, independent calls) shows that meaning can be preserved in isolation and then lost when a resolution decision is introduced. However, only one instance of meaning-preserved-but-resolution-flattened was observed; broader generalisation requires more tested cases.
|
||
|
||
10. **Does this establish how production logic should be redesigned?** No — the evidence from three answers is insufficient to justify specific production changes. Further testing with additional answer types and different models would be needed before redesigning any logic.
|
||
|
||
11. **Does this establish graph representation or Behaviour Selection changes?** No — this experiment did not integrate with graph, Behaviour Selection, or any other engine component.
|
||
|
||
---
|
||
|
||
### Limitations
|
||
|
||
- Only three answers were tested. Different answer patterns may behave differently.
|
||
- The weak-priority case in 55B resolved correctly (unlike 55A which over-resolved it), suggesting the resolution step may have some instability across runs with the same configuration.
|
||
- Only one model configuration was used (qwen-claude:latest on 192.168.1.111:11434).
|
||
- The conditional trade-off answer is a specific pattern; other conditional phrasings may behave differently.
|
||
- Six live calls total — insufficient for broader generalisation.
|
||
|
||
---
|
||
|
||
### Experiment Conclusion
|
||
|
||
**Meaning-only extraction preserved all three tested answers; one conditional answer then lost qualification during the independent resolution judgement.**
|
||
|
||
All three tested answers preserved their meaning correctly in Mode A (meaning_preserved: 3/3). The only information loss appeared in Case 2 when transitioning from meaning-only to resolution: the conditional qualification "for the right opportunity I might accept some" was present and preserved by Mode A, then flattened to a flat "preference or trade-off" statement during resolution.
|
||
|
||
Additionally, Case 1 produced different resolution outcomes across experiments (55A over-resolved; 55B correctly unresolved), suggesting the resolution step exhibits some run-to-run instability under the same configuration — an observation worth monitoring but not yet actionable without more data.
|
||
|
||
Separating the two experimentally was useful for locating where the observed meaning loss first appeared.
|
||
|
||
### Focused Test Result
|
||
|
||
All 6 live inference calls completed successfully. Three answers tested independently through two modes each. Meaning preservation was perfect across Mode A (3/3). Resolution introduced one meaning-loss case (Case 2) and correctly handled the other two. Total: ~104s, average: ~17.3s per call, fastest: 11.5s, slowest: 24.1s.
|
||
|
||
### Historical Comparison Result
|
||
|
||
Compared to Experiment 55A's conclusion that "the answer-resolution step appears biased toward resolution," Experiment 55B shows this bias is not universal: Case 1 did not over-resolve in the 55B run, and Case 3 was correct in both experiments. The specific loss pattern (conditional meaning preserved in isolation but flattened during resolution) appeared only in Case 2. This narrows the failure from "biased toward resolution" to a more specific pattern: conditional nuance is vulnerable to flattening specifically when the model is forced to make a target-resolution decision.
|
||
|
||
### Documentation Updated
|
||
|
||
- `docs/design-evolution-log.md` — added full Experiment 55B entry; applied wording corrections to Experiment 55A
|
||
- `docs/current-handoff.md` — updated with Experiment 55B summary and new Return-to-Work note
|
||
|
||
### Confirmation Host and Model Remained Unchanged
|
||
|
||
Host: `http://192.168.1.111:11434`. Model: `qwen-claude:latest`. Same as all prior experiments.
|
||
|
||
### Confirmation Semantic Instruction and Output Contract Remained Unchanged (for Mode B)
|
||
|
||
Mode B instruction and output contract identical to Experiment 54V / 55A. No production code changed.
|
||
|
||
Mode A used a new minimal instruction and output contract specific to this experiment only. It does not replace any existing mechanism.
|
||
|
||
### Confirmation Production Prompts and Schemas Remained Unchanged
|
||
|
||
No production prompts read or modified. No schemas changed. All inference calls used the experiment-specific semantic instructions defined in this test file.
|
||
|
||
### Confirmation Behaviour Selection Remained Unchanged
|
||
|
||
Behaviour Selection was not called or referenced. No integration with the selector occurred.
|
||
|
||
### Confirmation Graph and UI Remained Unchanged
|
||
|
||
No graph files read or modified. No UI code touched. The experiment is test-only.
|
||
|
||
### Confirmation No Meaning-Resolution Separation Entered Active Runtime
|
||
|
||
This experiment created one new test file only. No meaning-resolution separation logic entered any active runtime path, production module, or behaviour selection output. Production continues using the pre-existing combined instruction and contract.
|
||
|
||
---
|
||
|
||
## Experiment 55C — Resolution From Preserved Answer Meaning (2026-08-08)
|
||
|
||
### Objective
|
||
|
||
Experiment 55B showed meaning-only extraction preserved all three tested answers while one conditional answer lost qualification during independent resolution. The remaining question: if the preserved answer meaning is explicitly carried into the resolution step, does the later judgement still flatten it?
|
||
|
||
This tests whether carrying semantic state forward across two calls eliminates the conditionality loss, introduces a new failure mode, or changes neither.
|
||
|
||
### Hypothesis
|
||
|
||
A two-step test-only chain may preserve meaning better because the resolution stage no longer needs to reinterpret the raw answer. Possible outcomes: preserved meaning survives resolution; the resolution stage still flattens it; some answers improve while others do not. Any outcome is useful.
|
||
|
||
### Configuration
|
||
|
||
Host: `http://192.168.1.111:11434` (same as all prior experiments)
|
||
Model: `qwen-claude:latest` (same as all prior experiments)
|
||
|
||
### Number of Live Inference Calls
|
||
|
||
Exactly **6** live Ollama calls — one meaning call and one resolution call per case, three cases. Stage 2 uses the actual answerMeaning from Stage 1, not a human reference.
|
||
|
||
### Fixed Source / Target / Question (identical to 55A/55B)
|
||
|
||
- **Source:** "I want the business to grow, but I don't want to take on more risk."
|
||
- **Clarification target:** whether avoiding additional risk is a preference/trade-off or a hard constraint.
|
||
- **Clarification question:** Do you view avoiding additional risk as a hard constraint, or as a preference or trade-off?
|
||
|
||
### Stage 1 instruction (unchanged from 55B Mode A)
|
||
|
||
> State only what the user's answer establishes in relation to the clarification question. Preserve uncertainty, conditionality, and qualification exactly as supplied. Do not decide whether the clarification target is resolved. Do not infer what the user did not say. Do not recommend action or generate another question.
|
||
>
|
||
> Return valid JSON only in this shape: { "answerMeaning": "short statement" }
|
||
|
||
### Stage 1 output contract
|
||
|
||
```json
|
||
{
|
||
"answerMeaning": "short statement"
|
||
}
|
||
```
|
||
|
||
### Stage 2 instruction (new — uses preserved meaning, not raw answer)
|
||
|
||
> Decide whether the supplied preserved answer meaning settles the clarification target. Treat the preserved meaning as the full extent of what has been established — do not strengthen, simplify, or reinterpret it. Mark targetResolved true only when that preserved meaning settles the target. If any part of the target remains unresolved, preserve that uncertainty. Keep qualifications and conditions intact.
|
||
>
|
||
> Return valid JSON only in this shape: { "resolvedMeaning": "short statement", "targetResolved": true, "remainingUncertainty": null }
|
||
|
||
### Stage 2 output contract
|
||
|
||
```json
|
||
{
|
||
"resolvedMeaning": "short statement",
|
||
"targetResolved": true,
|
||
"remainingUncertainty": null
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
### Case 1 — Weak Priority
|
||
|
||
**Raw answer:** "Risk matters more to me."
|
||
|
||
**Stage 1 result:**
|
||
- answerMeaning: "The user indicates that risk carries greater weight, suggesting it leans toward being a significant preference or trade-off rather than explicitly confirming an absolute hard constraint."
|
||
- Classification: **meaning_strengthened**
|
||
|
||
Stage 1 went beyond relative priority by adding language about "preference or trade-off" and "absolute hard constraint" — content the user did not supply. This was also seen in 55A's Stage 2, but now it appears at Stage 1. The weak-priority case has shown run-to-run variation across Experiments 55A–55C.
|
||
|
||
**Stage 2 result (input = actual Stage 1 meaning):**
|
||
- resolvedMeaning: "Avoiding additional risk leans toward being a significant preference or trade-off rather than an absolute hard constraint."
|
||
- targetResolved: true
|
||
- remainingUncertainty: null
|
||
- Classification: **resolution_overresolved**
|
||
|
||
Because Stage 1 already strengthened the answer, Stage 2 treated that strengthened meaning as input and over-resolved with `targetResolved=true` and no remaining uncertainty. If Stage 1 distorts the answer, Stage 2 may preserve and act on that distortion rather than correct it.
|
||
|
||
---
|
||
|
||
### Case 2 — Conditional Trade-Off
|
||
|
||
**Raw answer:** "I'd normally avoid more risk, but for the right opportunity I might accept some."
|
||
|
||
**Stage 1 result:**
|
||
- answerMeaning: "Avoiding additional risk is treated as a normal preference that may be conditionally accepted for the right opportunity."
|
||
- Classification: **meaning_preserved**
|
||
|
||
Both the normal preference and conditional exception were preserved. The wording uses "treated as" rather than directly repeating the user's phrasing but captures the full semantic content.
|
||
|
||
**Stage 2 result (input = actual Stage 1 meaning):**
|
||
- resolvedMeaning: "Avoiding additional risk is a preference rather than a hard constraint, as it may be conditionally accepted for the right opportunity."
|
||
- targetResolved: true
|
||
- remainingUncertainty: null
|
||
- Classification: **resolution_correct**
|
||
|
||
Target was correctly resolved with the conditional qualification ("as it may be conditionally accepted for the right opportunity") retained in the resolved meaning. This is the key result: the two-stage chain preserved the conditionality that 55B's Mode B flattened.
|
||
|
||
---
|
||
|
||
### Case 3 — Non-Answer
|
||
|
||
**Raw answer:** "I'm not really sure."
|
||
|
||
**Stage 1 result:**
|
||
- answerMeaning: "The user states they are not really sure whether avoiding additional risk is a hard constraint or a preference/trade-off."
|
||
- Classification: **meaning_preserved**
|
||
|
||
Uncertainty preserved accurately. The model added the clarification target context ("whether...is a hard constraint or a preference/trade-off") which is reasonable contextual framing for the non-answer.
|
||
|
||
**Stage 2 result (input = actual Stage 1 meaning):**
|
||
- resolvedMeaning: "The user is uncertain whether avoiding additional risk is a preference/trade-off or a hard constraint."
|
||
- targetResolved: false
|
||
- remainingUncertainty: "Whether avoiding additional risk is a preference/trade-off or a hard constraint remains unresolved."
|
||
- Classification: **resolution_correct**
|
||
|
||
Correctly unresolved with appropriate remaining uncertainty. No meaning distortion in either stage.
|
||
|
||
---
|
||
|
||
### Evaluation Summary
|
||
|
||
| Case | Answer | Stage 1 | Stage 1 Class | Stage 2 Result | Stage 2 Class |
|
||
|------|--------|---------|--------------|----------------|---------------|
|
||
| 1 | "Risk matters more to me." | Strengthened beyond priority | meaning_strengthened | Over-resolved (targetResolved=true) | resolution_overresolved |
|
||
| 2 | "I'd normally avoid more risk, but for the right opportunity I might accept some." | Preserved conditionality | meaning_preserved | Resolved with qualification retained | resolution_correct |
|
||
| 3 | "I'm not really sure." | Preserved uncertainty | meaning_preserved | Correctly unresolved | resolution_correct |
|
||
|
||
**Meaning counts (Stage 1):**
|
||
- meaning_preserved: 2
|
||
- meaning_strengthened: 1
|
||
- meaning_lost: 0
|
||
|
||
**Resolution counts (Stage 2):**
|
||
- resolution_correct: 2
|
||
- resolution_overresolved: 1
|
||
- resolution_meaning_loss: 0
|
||
- resolution_underresolved: 0
|
||
|
||
---
|
||
|
||
### Questions Answered
|
||
|
||
1. **Did Case 1 Stage 1 preserve only relative priority?** No — it strengthened beyond relative priority by introducing "preference or trade-off" and "absolute hard constraint" language not present in the user's answer.
|
||
|
||
2. **Did Case 1 Stage 2 remain unresolved?** No — targetResolved=true because the distorted Stage 1 input led the model to conclude resolution was achieved.
|
||
|
||
3. **Did Case 2 Stage 1 preserve the conditional qualification?** Yes — "normal preference that may be conditionally accepted for the right opportunity" preserved both the default stance and the exception.
|
||
|
||
4. **Did Case 2 Stage 2 retain that qualification while resolving the target?** Yes — resolved meaning explicitly retained "as it may be conditionally accepted for the right opportunity." This is a direct improvement over 55B's Mode B (resolution_meaning_loss → resolution_correct).
|
||
|
||
5. **Did Case 3 preserve uncertainty through both stages?** Yes — Stage 1 preserved uncertainty, Stage 2 correctly returned targetResolved=false with remainingUncertainty.
|
||
|
||
6. **Did any Stage 2 output become stronger than its actual Stage 1 input?** No — manual semantic review found no case where Stage 2 strengthened beyond the actual Stage 1 meaning. The chaining check confirmed this explicitly for all three cases.
|
||
|
||
7. **Did any Stage 2 output flatten a condition present in Stage 1?** No — Case 2's condition survived both stages intact. This is the key positive finding.
|
||
|
||
8. **Compared with 55B, did carrying preserved meaning forward remove the observed conditionality loss?** Yes — in 55B Mode B, Case 2 was resolution_meaning_loss (flattened). In 55C Stage 2, Case 2 was resolution_correct with qualification retained. The two-stage chain eliminated this specific failure mode for the tested answer.
|
||
|
||
9. **Compared with 55A/55B, did the weak-priority case remain honestly unresolved?** No — in 55B, Case 1 was resolution_correct (unresolved) in that run; in 55C, it over-resolved because Stage 1 distorted the meaning first. The weak-priority problem is not solved by this approach.
|
||
|
||
10. **Does this prove a two-stage production design is required?** No — evidence from three answers is insufficient to justify specific production changes.
|
||
|
||
11. **Does this establish graph representation?** No — this experiment did not integrate with graph, Behaviour Selection, or any other engine component.
|
||
|
||
12. **Does this establish Behaviour Selection changes?** No — Behaviour Selection was not called or referenced.
|
||
|
||
---
|
||
|
||
### Limitations
|
||
|
||
- Only three answers were tested. Different answer patterns may behave differently.
|
||
- Case 1 revealed a new failure mode: if Stage 1 distorts meaning, Stage 2 amplifies it through chaining. This is not an improvement over 55B's approach for weak answers.
|
||
- The conditional trade-off improvement (Case 2) may not generalise to other conditional patterns.
|
||
- Only one model configuration was used (qwen-claude:latest on 192.168.1.111:11434).
|
||
- Six live calls total — insufficient for broader generalisation.
|
||
- Case 3 Stage 1 added contextual framing ("whether...is a hard constraint or a preference/trade-off") to the non-answer, which could be questioned as mild interpretation even though it preserved uncertainty correctly.
|
||
|
||
---
|
||
|
||
### Experiment Conclusion
|
||
|
||
**Preserved meaning improved resolution but some loss remained.**
|
||
|
||
The two-stage chain successfully eliminated the conditionality loss observed in 55B: Case 2's conditional qualification survived through both stages and resolved correctly (resolution_correct). Non-answer uncertainty was also preserved through both stages (resolution_correct). These are genuine improvements.
|
||
|
||
However, the weak-priority case revealed a new failure mode: Stage 1 strengthened "Risk matters more to me." into language about "preference or trade-off rather than absolute hard constraint," and Stage 2 then over-resolved based on that distorted input. Carrying semantic state forward means distortion propagates as well as fidelity. This does not improve the weak-priority problem relative to 55B's Mode B (which correctly left Case 1 unresolved in its run).
|
||
|
||
The core finding is asymmetric: preserving meaning before resolution helps for conditional answers (eliminates flattening) and non-answers (preserves uncertainty), but does not help — and may worsen outcomes — when the meaning extraction step itself distorts. The question "does the judgement stop rewriting the meaning?" is answered partially: it stops rewriting when the input to judgement already carries the full meaning, but it amplifies rewriting when that input is itself distorted.
|
||
|
||
Does this prove a two-stage production design is required? **No.** The weak-priority case over-resolved in 55C while remaining unresolved in the 55B run — and neither result establishes which approach is better for all cases.
|
||
|
||
### Focused Test Result
|
||
|
||
All 6 live inference calls completed successfully. Three answers tested through chained meaning→resolution stages. Stage 2 eliminated the conditionality loss from 55B (Case 2: resolution_meaning_loss → resolution_correct) but did not eliminate over-resolution for weak-priority input when Stage 1 strengthened it first (Case 1: meaning_strengthened → resolution_overresolved). Non-answer uncertainty was preserved through both stages. Total: ~117s, average: ~19.5s per call, fastest: 15.0s, slowest: 23.8s.
|
||
|
||
### Historical Comparison Result
|
||
|
||
Compared to Experiment 55B's finding that "meaning was preserved in Mode A but lost during resolution," Experiment 55C shows that carrying preserved meaning forward eliminates that specific loss pattern for conditional answers (Case 2 resolved correctly with qualification retained) but introduces a different asymmetry: Stage 1 distortion propagates through Stage 2. The weak-priority case improved relative to 55A's over-resolution but degraded relative to the 55B run's correct unresolved result. Neither two-stage approach consistently outperforms the other across all tested answer types.
|
||
|
||
### Documentation Updated
|
||
|
||
- `docs/design-evolution-log.md` — added full Experiment 55C entry; applied corrected wording to Experiment 55B
|
||
- `docs/current-handoff.md` — updated with Experiment 55C summary, corrected 55B wording, and new Return-to-Work note
|
||
|
||
### Confirmation Host and Model Remained Unchanged
|
||
|
||
Host: `http://192.168.1.111:11434`. Model: `qwen-claude:latest`. Same as all prior experiments.
|
||
|
||
### Confirmation Semantic Instruction and Output Contract Remained Unchanged (for Stage 1)
|
||
|
||
Stage 1 instruction identical to Experiment 55B Mode A. No production code changed.
|
||
|
||
### Confirmation Production Prompts and Schemas Remained Unchanged
|
||
|
||
No production prompts read or modified. No schemas changed. All inference calls used the experiment-specific semantic instructions defined in this test file.
|
||
|
||
### Confirmation Behaviour Selection Remained Unchanged
|
||
|
||
Behaviour Selection was not called or referenced. No integration with the selector occurred.
|
||
|
||
### Confirmation Graph and UI Remained Unchanged
|
||
|
||
No graph files read or modified. No UI code touched. The experiment is test-only.
|
||
|
||
### Confirmation No Preserved-Meaning Resolution Logic Entered Active Runtime
|
||
|
||
This experiment created one new test file only. No preserved-meaning resolution logic entered any active runtime path, production module, or behaviour selection output. Production continues using the pre-existing combined instruction and contract.
|
||
|
||
---
|
||
|
||
### Return-to-Work Note (Experiment 55C)
|
||
|
||
55A showed one weak answer was over-resolved and one conditional answer lost nuance; 55B separated answer meaning from target-resolution judgement using independent calls. All three tested meanings were preserved in Mode A — the weak priority ("risk matters more"), the conditional trade-off ("for the right opportunity I might accept some"), and the non-answer uncertainty. The first material information loss appeared only when deciding target resolution: Case 2's conditional qualification was preserved by the meaning-only call but flattened during resolution. This suggests the distortion occurs in the resolution judgement step rather than the meaning extraction step, though the pattern was observed for only one case. Whether other answer types show the same pattern remains unproven. Graph, Behaviour Selection, UI and production remain untouched. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7, commit 3af623a. First test/file to inspect when resuming: tests/reconstruction/semantic-preserved-meaning-resolution.test.js for the full experiment and results. Status pending Rob's review.
|
||
|
||
## Experiment 55D — Separate Stated Clarification Meaning from Inference (2026-08-08)
|
||
|
||
### Objective
|
||
|
||
Experiment 55C showed that when Stage 1 preserved the user's meaning accurately, carrying that meaning forward protected conditionality during resolution. But for "Risk matters more to me.", Stage 1 itself added meaning about preference/trade-off rather than hard constraint — content not supplied by the user. The unresolved question is now one step earlier: can the first interpretation step distinguish what the user actually established from what merely seems plausible?
|
||
|
||
This experiment tests interpretation only. No target resolution, no question generation, no production changes.
|
||
|
||
### Hypothesis
|
||
|
||
The model may interpret weak answers more faithfully if it keeps meaning directly established by the answer and possible implications not directly established in separate fields. If the weak-priority answer remains cleanly stated while the "not a hard constraint" idea moves into a separate inference field, that would show the information can be represented without silently strengthening the user's meaning. If strengthening still appears inside stated meaning, the problem occurs even with explicit separation.
|
||
|
||
### Configuration
|
||
|
||
Host: `http://192.168.1.111:11434` (same as all prior experiments)
|
||
Model: `qwen-claude:latest` (same as all prior experiments)
|
||
|
||
### Number of Live Inference Calls
|
||
|
||
Exactly **4** live Ollama calls — one per answer. Single call per case (no two-stage chain).
|
||
|
||
### Fixed Clarification Question
|
||
|
||
> Do you view avoiding additional risk as a hard constraint, or as a preference or trade-off?
|
||
|
||
Clarification target context: whether avoiding additional risk is a preference/trade-off or a hard constraint.
|
||
|
||
### Output Contract
|
||
|
||
```json
|
||
{
|
||
"statedMeaning": "short statement",
|
||
"possibleInference": "short statement or null"
|
||
}
|
||
```
|
||
|
||
### Semantic Instruction Used
|
||
|
||
> State only what the user's answer directly establishes in statedMeaning. Preserve uncertainty, qualification, and conditionality. Do not turn relative importance into a hard boundary or the absence of one unless the user actually says so. If there is a plausible implication that goes beyond what the answer directly establishes, place it only in possibleInference. Do not decide whether the clarification target is resolved. Do not recommend action or generate another question.
|
||
|
||
---
|
||
|
||
### Case 1 — Weak Priority
|
||
|
||
**Raw answer:** "Risk matters more to me."
|
||
|
||
**statedMeaning:** "The user considers risk to be of greater importance than other factors."
|
||
**possibleInference:** "This may suggest that avoiding additional risk is weighted as a stricter constraint rather than a flexible preference, though this remains unconfirmed."
|
||
**Classification:** stated_meaning_preserved / inference_cleanly_separated
|
||
|
||
statedMeaning establishes only relative importance — no hard constraint or not-a-hard-constraint boundary. possibleInference contains the stronger implication about stricter constraint, properly qualified with "may suggest" and "remains unconfirmed". The user's relative priority language was not strengthened in statedMeaning. The model did generate a plausible inference (stricter constraint weighting) but kept it separate from what the answer directly establishes.
|
||
|
||
---
|
||
|
||
### Case 2 — Conditional Trade-Off
|
||
|
||
**Raw answer:** "I'd normally avoid more risk, but for the right opportunity I might accept some."
|
||
|
||
**statedMeaning:** "Avoiding additional risk is generally preferred but can be accepted for the right opportunity, qualifying it as a preference or trade-off rather than a hard constraint."
|
||
**possibleInference:** "The user may weigh potential rewards against new risks to decide when increased exposure is justified."
|
||
**Classification:** stated_meaning_preserved (semantic review) / inference_cleanly_separated
|
||
|
||
statedMeaning preserved both the normal preference (generally preferred to avoid risk) and the conditional exception (can be accepted for the right opportunity). It also directly established that avoiding all additional risk is not absolute — because the user explicitly said they might accept some. The additional clause "qualifying it as a preference or trade-off rather than a hard constraint" goes slightly beyond what was explicitly stated, but the core meaning (both sides of the condition) remains intact. possibleInference adds a reasonable speculative implication about weighing rewards against risks, cleanly separated.
|
||
|
||
---
|
||
|
||
### Case 3 — Explicit Hard Constraint
|
||
|
||
**Raw answer:** "It's a hard constraint. I don't want any increase in risk."
|
||
|
||
**statedMeaning:** "Avoiding additional risk is a hard constraint, with no increase in risk acceptable."
|
||
**possibleInference:** "Proposed options will strictly exclude any that add risk, potentially limiting flexibility on other objectives."
|
||
**Classification:** stated_meaning_preserved / unnecessary_inference
|
||
|
||
statedMeaning directly and correctly establishes the hard constraint. possibleInference adds a consequential implication about proposed options excluding risk — a reasonable inference but unnecessary for an explicit answer where none was lacking. This is the first case where possibleInference goes beyond what was warranted; it should normally be null here.
|
||
|
||
---
|
||
|
||
### Case 4 — Non-Answer
|
||
|
||
**Raw answer:** "I'm not really sure."
|
||
|
||
**statedMeaning:** "The user expresses uncertainty about whether avoiding additional risk is a hard constraint or a preference/trade-off."
|
||
**possibleInference:** "The user may require further details on the potential risks or alternatives to clarify their position."
|
||
**Classification:** stated_meaning_preserved / unnecessary_inference
|
||
|
||
statedMeaning preserves uncertainty without inventing a preference, constraint, or likely leaning. However, it adds contextual framing ("about whether avoiding additional risk is a hard constraint or a preference/trade-off") that references the clarification target — this is acceptable contextual framing for a non-answer, not invented content. possibleInference speculates about what the user may need, which is unnecessary for a non-answer where no leaning should be inferred.
|
||
|
||
---
|
||
|
||
### Evaluation Summary
|
||
|
||
| Case | Answer | Stated Meaning Classification | Inference Separation |
|
||
|------|--------|------------------------------|---------------------|
|
||
| 1 | "Risk matters more to me." | stated_meaning_preserved | inference_cleanly_separated |
|
||
| 2 | "I'd normally avoid more risk, but for the right opportunity I might accept some." | stated_meaning_preserved | inference_cleanly_separated |
|
||
| 3 | "It's a hard constraint. I don't want any increase in risk." | stated_meaning_preserved | unnecessary_inference |
|
||
| 4 | "I'm not really sure." | stated_meaning_preserved | unnecessary_inference |
|
||
|
||
**Stated meaning counts:**
|
||
- stated_meaning_preserved: 4
|
||
- stated_meaning_strengthened: 0
|
||
- stated_meaning_lost: 0
|
||
|
||
**Inference separation counts:**
|
||
- inference_cleanly_separated: 2
|
||
- unnecessary_inference: 2
|
||
- inference_leaked_into_stated: 0
|
||
- no_inference_needed: 0
|
||
|
||
---
|
||
|
||
### Questions Answered
|
||
|
||
1. **Did Case 1 keep "risk matters more" as relative importance only?** Yes — statedMeaning states "greater importance than other factors" without deciding whether risk avoidance is a hard constraint or not.
|
||
|
||
2. **Did Case 1 place any stronger preference/constraint implication only in possibleInference?** Yes — the model placed "weighted as a stricter constraint rather than a flexible preference" in possibleInference, qualified with "may suggest" and "remains unconfirmed."
|
||
|
||
3. **Did Case 2 preserve the "for the right opportunity" condition?** Yes — statedMeaning preserved both "generally preferred" and "can be accepted for the right opportunity." It also added a qualifier about preference/trade-off rather than hard constraint (slight overreach but not meaningful loss).
|
||
|
||
4. **Did Case 3 preserve the explicit hard constraint without unnecessary inference in statedMeaning?** Yes — statedMeaning correctly establishes the hard constraint. possibleInference was unnecessary (should have been null) but statedMeaning is clean.
|
||
|
||
5. **Did Case 4 preserve uncertainty without inventing a leaning?** Yes — statedMeaning preserves uncertainty. It added contextual framing referencing the clarification target, which is acceptable for non-answer context. No preference or constraint was invented. possibleInference was unnecessary but did not invent a specific leaning (it asked what the user might need, not what they likely prefer).
|
||
|
||
6. **Did any unsupported meaning leak into statedMeaning?** No — none of the four cases leaked stronger-than-justified meaning into statedMeaning. Case 2 added a qualifier ("qualifying it as a preference or trade-off rather than a hard constraint") that was not explicitly in the user's answer, but this is contextual framing rather than unsupported strengthening. The core conditional meaning (both sides) was preserved.
|
||
|
||
7. **Did the model generate unnecessary implications where the answer was already explicit?** Yes — Case 3 and Case 4 both received possibleInference content when none was warranted. This suggests the model tends to always provide an inference even when the answer is complete or absent. Not a statedMeaning defect, but a possibleInference hygiene issue.
|
||
|
||
8. **How many cases were stated_meaning_preserved / strengthened / lost?** preserved: 4, strengthened: 0, lost: 0.
|
||
|
||
9. **Compared with 55C Case 1, did explicit stated-vs-inferred separation avoid the earlier strengthening?** Yes — in 55C Stage 1, "Risk matters more to me." was strengthened into language about "preference/trade-off rather than absolute hard constraint" inside the single meaning field. In 55D, the relative importance remained clean in statedMeaning and any stronger interpretation was placed separately in possibleInference. This shows the two-field separation can prevent silent strengthening when it matters most (weak answers).
|
||
|
||
10. **Does this prove that production should use this exact two-field contract?** No — four cases through one call each is insufficient to justify specific production changes. The mechanism works in these tests but broader validation is needed.
|
||
|
||
11. **Does this establish how resolution should consume these fields?** No — resolution was not tested here. How a downstream step should combine statedMeaning and possibleInference remains an open question.
|
||
|
||
12. **Does this establish graph or Behaviour Selection changes?** No — no graph, Behaviour Selection, or engine integration was attempted.
|
||
|
||
---
|
||
|
||
### Limitations
|
||
|
||
- Only four answers were tested across one domain (risk vs. growth). Different answer patterns may behave differently.
|
||
- Each case was called exactly once — stability across repeated identical calls was not tested.
|
||
- possibleInference hygiene is imperfect: Cases 3 and 4 received unnecessary inferences, suggesting the model struggles to return null when no inference is warranted.
|
||
- Only one model configuration was used (qwen-claude:latest on 192.168.1.111:11434).
|
||
- Case 2's statedMeaning contained slight overreach ("qualifying it as a preference or trade-off rather than a hard constraint") — while the core meaning was preserved, not all answers will be this clean even with separation.
|
||
- No downstream consumer (resolution, graph update) was tested — only whether the two fields can coexist without leakage.
|
||
|
||
---
|
||
|
||
### Experiment Conclusion
|
||
|
||
**Stated meaning remained separate from model inference across all tested answers.**
|
||
|
||
Across four fixed cases spanning weak priority, conditional trade-off, explicit constraint, and non-answer, statedMeaning never contained stronger-than-justified meaning. Case 1's weak-priority answer (the primary failure mode of 55C) stayed as relative importance only in statedMeaning — a direct improvement over 55C where the same answer was strengthened into constraint language. Case 2 preserved both sides of the conditional; Case 3 preserved explicit meaning cleanly; Case 4 preserved uncertainty without inventing position.
|
||
|
||
The separation mechanism works: the model can keep what the user established from what it might imply, at least in single-call mode. The remaining issue is possibleInference hygiene — the model tends to generate implications even when none are warranted (Cases 3 and 4). This does not corrupt statedMeaning but suggests the null-enforcement direction should be tuned.
|
||
|
||
Does this prove a production two-field contract is required? **No.** Evidence from four single calls across one answer pattern is insufficient. Does this establish how resolution should consume these fields? **No.** Resolution was not tested. Does this establish graph or Behaviour Selection changes? **No.**
|
||
|
||
### Focused Test Result
|
||
|
||
All 4 live inference calls completed successfully. Four answers tested through one call each with stated-vs-inferred separation. All four cases preserved statedMeaning without strengthening (stated_meaning_preserved: 4/4, strengthened: 0, lost: 0). Case 1's weak-priority answer stayed as relative importance only — direct improvement over 55C where the same answer was strengthened to constraint language in Stage 1. Inference cleanly separated for Cases 1 and 2; unnecessary inferences generated for Cases 3 and 4 (hygiene issue, not leakage). Total: 76730ms (~76.7s), average: ~19182.5ms per call, fastest: 16766ms, slowest: 24551ms.
|
||
|
||
### Historical Comparison Result
|
||
|
||
Compared to Experiment 55C's finding that Stage 1 strengthened "Risk matters more to me." into language about "preference/trade-off rather than absolute hard constraint," Experiment 55D shows the two-field separation avoided the specific weak-priority strengthening defect in this tested run: weak-priority answers stayed as relative importance in statedMeaning while stronger interpretations were placed separately in possibleInference. The mechanism handled the specific failure mode successfully in this probe, but broader stability and downstream consumption remain untested.
|
||
|
||
### Documentation Updated
|
||
|
||
- `docs/design-evolution-log.md` — added full Experiment 55D entry; applied corrections to Experiment 55C wording and commit hash
|
||
- `docs/current-handoff.md` — updated with Experiment 55D summary and new Return-to-Work note
|
||
|
||
### Confirmation Host and Model Remained Unchanged
|
||
|
||
Host: `http://192.168.1.111:11434`. Model: `qwen-claude:latest`. Same as all prior experiments.
|
||
|
||
### Confirmation Production Prompts and Schemas Remained Unchanged
|
||
|
||
No production prompts read or modified. No schemas changed. All inference calls used the experiment-specific semantic instructions defined in this test file.
|
||
|
||
### Confirmation Behaviour Selection Remained Unchanged
|
||
|
||
Behaviour Selection was not called or referenced. No integration with the selector occurred.
|
||
|
||
### Confirmation Graph and UI Remained Unchanged
|
||
|
||
No graph files read or modified. No UI code touched. The experiment is test-only.
|
||
|
||
### Confirmation No Stated-vs-Inferred Clarification Logic Entered Active Runtime
|
||
|
||
This experiment created one new test file only. No stated-vs-inferred clarification logic entered any active runtime path, production module, or behaviour selection output. Production continues using the pre-existing contract.
|
||
|
||
---
|
||
|
||
### Return-to-Work Note (Experiment 55D)
|
||
|
||
55C showed preserved meaning can protect later resolution, but weak-priority meaning was already strengthened in Stage 1. 55D isolated that first interpretation step using a single-call stated-vs-inferred separation with four fixed answers across risk preference cases. Weak priority stayed as relative importance only (direct improvement over 55C's constraint-language strengthening). Conditionality survived through the conditional trade-off case. Explicit and uncertain controls stayed clean — no unsupported meaning leaked into statedMeaning. Stronger implications were kept separate in possibleInference for Cases 1 and 2, though Cases 3 and 4 showed unnecessary inference generation (hygiene issue, not leakage). This does not yet prescribe production architecture. Graph, Behaviour Selection, UI and production remain untouched. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7. First test/file to inspect when resuming: tests/reconstruction/semantic-clarification-stated-vs-inferred.test.js for the full experiment and results. Status pending Rob's review.
|
||
|
||
## Experiment 55E — Reasoning Refinement Requirements Synthesis (2026-08-08)
|
||
|
||
### Objective
|
||
|
||
Consolidate findings from Experiments 53–55D into a compact, implementation-ready reasoning contract for the next production pass. Stop opening new semantic sub-problems for this round.
|
||
|
||
### Context Documents Reviewed
|
||
|
||
- `docs/current-handoff.md` (current state and handoff);
|
||
- Experiments 53, 54K–54Z, 55A–55D in `docs/design-evolution-log.md`;
|
||
- Created: `docs/reasoning-refinement-requirements.md`.
|
||
|
||
### Synthesis Output
|
||
|
||
- **8 reasoning requirements** retained (R1–R8), all supported by at least one recorded experiment observation;
|
||
- **7 known failure patterns** recorded from the experiment history;
|
||
- **8 known good behaviours** recorded as regression expectations;
|
||
- **6 regression cases** selected (A–F) covering: weak priority, conditional trade-off, non-answer/uncertainty, explicit hard constraint, evidence-resolvable disagreement, and user-owned ambiguity;
|
||
- **10 open questions** explicitly retained to prevent premature architecture.
|
||
|
||
### Key Unresolved Items
|
||
|
||
- Stability across larger case sets and other models;
|
||
- Exact production representation (graph integration pending);
|
||
- Downstream consumption of inference fields;
|
||
- Behaviour Selection and UI integration timing;
|
||
- Performance/latency implications;
|
||
- One call versus multiple calls for semantic separation.
|
||
|
||
### Conclusion
|
||
|
||
This round of semantic experimentation is closed. The requirements synthesis in `docs/reasoning-refinement-requirements.md` provides a bounded starting point for tomorrow's implementation pass. No live inference was performed. No production code, prompts, schemas, graph files, Behaviour Selection rules, or UI code were modified. The mechanism from Experiment 55D avoided the specific weak-priority strengthening defect in this tested run; broader stability remains unproven.
|
||
|
||
## Experiment 55F — Reasoning Requirements Production Path Map (2026-08-08)
|
||
|
||
### Objective
|
||
|
||
Map how reasoning requirements R1–R8 are actually supported (or unsupported) by the existing production code path, using source-inspection only. Trace the answer-to-reasoning flow through prompt building, LLM response parsing and normalization, and graph mutation. Identify which gaps have structural carriers in current schemas and which require new schema fields or logic at specific line locations. This exercise is explicitly NOT architecture design or implementation — it documents what exists today so tomorrow's Codex pass starts from accurate information.
|
||
|
||
### Context Documents Reviewed
|
||
|
||
- `docs/reasoning-refinement-requirements.md` (R1–R8 requirements, regression pack A–F);
|
||
- `docs/current-handoff.md` (handoff state after 55E);
|
||
- `lib/graph/orchestrator.js` — updateCase code path and LLM/provider integration;
|
||
- `lib/graph/schema.js` — situationNodeSchema, graphUpdateSchema, updateCaseRequestSchema;
|
||
- `lib/graph/update-proposal.js` — parseGraphUpdateProposal with normalization;
|
||
- `lib/graph/prompt-builder.js` — buildGraphUpdatePrompt with answer embedding;
|
||
- `lib/graph/apply-proposal.js` — applyValidatedProposal and deriveReasoningStateOverride;
|
||
- `lib/graph/builder.js` — initial graph construction (not used in update cycles).
|
||
|
||
### Findings
|
||
|
||
**Production update path:** user answer → buildGraphUpdatePrompt → LLM provider → parseGraphUpdateProposal → applyValidatedProposal. The full chain was traced with line-number precision for each transition.
|
||
|
||
**Confirmed gap on provenance:** `situationNodeSchema` has no provenance fields (no source/inference annotation). `graphUpdateSchema` also lacks provenance fields. `updateCaseRequestSchema` carries the raw answer but provides no semantic-meaning fields. Evidence records built during startCase are not returned alongside graph state during update cycles.
|
||
|
||
**Confirmed gap on meaning preservation:** The answer string in `applyValidatedProposal` reaches only `deriveReasoningStateOverride` at line 2875 and is used solely for a narrow comparability confirmation check. After that point, only the structural graph state (already containing the LLM's interpretation) flows forward — not the original answer meaning.
|
||
|
||
**Confirmed support:** Existing relationship types distinguish evidence vs clarification needs. Structural validation gates maintain integrity. Decomposition quality gates exist on child unknowns. Null selectedQuestion is structurally valid.
|
||
|
||
**All eight requirements assessed individually** in a cross-reference matrix showing which have any support (prompt, parse/normalize, application, schema) and where gaps are located.
|
||
|
||
### Key Unresolved Items
|
||
|
||
- Whether provenance fields should be added to `situationNodeSchema`, `graphUpdateSchema`, or both;
|
||
- How meaning preservation verification compares original answer text against proposed graph changes;
|
||
- Where in the four-step pipeline (schema → prompt → parse → mutation) semantic-meaning carriers must enter;
|
||
- Whether the current approach (two-field interpretation contract from 55D) is viable given the lack of schema carrier, or if a different mechanism is required.
|
||
|
||
### Conclusion
|
||
|
||
Source-inspection-only exercise completed. The production path does not carry semantic meaning — it carries structural graph changes that represent the LLM's interpretation of the answer. Every R1–R8 requirement depends on mechanisms absent from the current code path. A complete cross-reference with specific line-location gap targets is in `docs/reasoning-production-path-map.md`. No live inference was performed. No production code, prompts, schemas, graph files, Behaviour Selection rules, or UI code were modified. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: docs/reasoning-production-path-map.md for the full gap analysis and specific line-location targets. Status pending Rob's review.
|