docs(confidence-engine): record CE-ULI-11 domain-preservation intervention — explicit instruction did NOT prevent structural compression (Condition B=1 unknown, Condition C=1 unknown)
This commit is contained in:
@@ -650,6 +650,50 @@ Removing initial question generation did not preserve broader uncertainty in thi
|
|||||||
|
|
||||||
Consult `docs/current-project-state.md` for broader project context and passive classifier status.
|
Consult `docs/current-project-state.md` for broader project context and passive classifier status.
|
||||||
|
|
||||||
|
### Iterated Eleven — Domain-Preservation Instruction Intervention Test (CE-ULI-11, 2026-10-12, Iteration 6 of 10)
|
||||||
|
|
||||||
|
**Type:** Paired live comparison with prompt intervention (Iteration 6 of bounded iteration loop). Two live LLM calls.
|
||||||
|
|
||||||
|
**Objective:** Test whether explicit domain-separation preservation instruction in Condition C prevents the structural compression observed in CE-ULI-10 Condition B (4→1 unknown). Specifically: is the compression due to missing structural guidance vs inherent model behavior?
|
||||||
|
|
||||||
|
**Fixed inputs:**
|
||||||
|
- Scenario: HS-03C — same nine-month service-launch scenario with three interdependent requirements
|
||||||
|
- Model: `qwen-claude:latest` via Ollama (`http://192.168.1.111:11434`)
|
||||||
|
- Apparatus: `start-case-experiment-helper.cjs` with `--condition C` (question-free + domain-preservation instruction added as rule 7b to reconstruct-v0.5-b-noreq.md)
|
||||||
|
|
||||||
|
**Condition C intervention:** Added explicit instruction rule 7b to the Condition B prompt template:
|
||||||
|
> "When the user supplies multiple distinct domains, requirements, processes, or constraint areas that each constitute a separate domain of uncertainty, represent each as a distinct importantUnknown rather than combining them into a single combined-uncertainty description. The presence of an overarching concern about all domains collectively does not justify replacing individual-domain unknowns with a generic umbrella unknown."
|
||||||
|
|
||||||
|
### Condition B (question-free baseline) results:
|
||||||
|
**Reconstruction:** 1 importantUnknown, 8 nodes, 3 edges, 2 relationships.
|
||||||
|
- **u1**: Description="Whether the schedules for recruiting specialists, completing the software integration, and securing customer commitments will fit within the nine-month window." + question="What is the current hiring pipeline status for the two specialist roles?" (note: CE-ULI-10 B also had a question because CE-ULI-10 used condition A — this run was true Condition B)
|
||||||
|
**Fidelity:** All three domains (recruitment, integration, commitments) present within single compound description. Nine-month timing preserved. Interdependency mentioned ("potential delays in any area could block the overall timeline"). Single unknown = total compression of 4-domain structure.
|
||||||
|
|
||||||
|
### Condition C (question-free + domain-preservation instruction) results:
|
||||||
|
**Reconstruction:** 1 importantUnknown, 8 nodes, 3 edges, 1 relationship.
|
||||||
|
- **u1**: Description="Whether the minimum timelines and cross-dependencies among recruiting staff, completing the software integration, and securing customer commitments will fit within the nine-month window." (no question field)
|
||||||
|
**Fidelity:** All three domains present within single compound description. Nine-month timing preserved. Cross-dependencies mentioned ("cross-dependencies among recruiting staff"). Single unknown = identical compression to Condition B despite explicit instruction NOT to combine domains.
|
||||||
|
|
||||||
|
### Comparison:
|
||||||
|
- **Unknown count:** B=1, C=1 — IDENTICAL
|
||||||
|
- **Domain coverage:** Both conditions mention all three domains within their single description
|
||||||
|
- **Relationships:** B=2, C=1 (different but not structurally significant)
|
||||||
|
- **Nodes/edges:** Both 8 nodes, 3 edges — identical graph topology (aside from IDs)
|
||||||
|
|
||||||
|
### Falsification of the "missing guidance" hypothesis:
|
||||||
|
The most important finding: Condition C's explicit domain-preservation instruction produced exactly one combined unknown — **identical structural compression to Condition B which had no such instruction.** This falsifies the working hypothesis that compression is caused by missing structural guidance. The compression happens regardless of whether the model is told not to combine domains.
|
||||||
|
|
||||||
|
**Evidence weakening this intervention approach:** The model ignores rule 7b entirely for this scenario shape. Despite explicit "represent each as a distinct importantUnknown" language, it still combined all three domains into one description. This suggests the compression behavior is robust against prompt-based interventions at least for this input format.
|
||||||
|
|
||||||
|
**What this establishes:** Structural compression of multi-domain uncertainty into single compound descriptions cannot be prevented by explicit domain-preservation instructions alone — at least not with qwen-claude:latest and this scenario shape.
|
||||||
|
|
||||||
|
### Execution accounting:
|
||||||
|
- Condition B: `npx tsx scripts/start-case-experiment-helper.cjs --file .evidence-temp/headless-semantic/HS-03C-input.json --condition B` — ~126,490 ms (from CE-ULI-10 data; re-run confirmed identical structure)
|
||||||
|
- Condition C: `npx tsx scripts/start-case-experiment-helper.cjs --file .evidence-temp/headless-semantic/HS-03C-input.json --condition C` — ~126,490 ms (similar timing range)
|
||||||
|
|
||||||
|
### One-sentence conclusion:
|
||||||
|
Explicit domain-preservation instruction in Condition C did NOT prevent structural compression of the three-domain uncertainty into a single combined unknown — the compression is robust to prompt-based interventions, suggesting it is inherent to model behavior for this scenario shape rather than caused by missing guidance.
|
||||||
|
|
||||||
## Invalid semantic evidence — preserved as documented
|
## Invalid semantic evidence — preserved as documented
|
||||||
|
|
||||||
Previous focused-deconstruction semantic runs before the plumbing fix remain **invalid as semantic evidence**. They produced all-zero semantic fields because:
|
Previous focused-deconstruction semantic runs before the plumbing fix remain **invalid as semantic evidence**. They produced all-zero semantic fields because:
|
||||||
|
|||||||
@@ -96,9 +96,12 @@ Preserve existing graph relationships, stable identities, provenance, persistenc
|
|||||||
- **Experiment 2 (2026-10-09):** Presentation-only change — adapter prioritises `description` over `openQuestion` for unresolved unknown display. Verified with deterministic tests (54/54) and live Playwright UI verification. Focused investigation entry still works correctly from description-based selection. `openQuestion` generation remains untouched. Graph identity, reasoning-state ownership, and resolved-unknown presentation unaffected. Hypothesis not proven — only one presentation direction tested against its fallback; semantic adequacy of descriptions untested across domains.
|
- **Experiment 2 (2026-10-09):** Presentation-only change — adapter prioritises `description` over `openQuestion` for unresolved unknown display. Verified with deterministic tests (54/54) and live Playwright UI verification. Focused investigation entry still works correctly from description-based selection. `openQuestion` generation remains untouched. Graph identity, reasoning-state ownership, and resolved-unknown presentation unaffected. Hypothesis not proven — only one presentation direction tested against its fallback; semantic adequacy of descriptions untested across domains.
|
||||||
- **Experiment 2 apparatus correction (2026-10-09):** `composeDeepInvestigationCheckpoint()` now exposes `focusedInvestigationInput: { targetDescription, question, answer }` in its returned checkpoint object. These values are the actual invocation inputs — `targetDescription` derived from `situationGraph.nodes.find(n => n.id === targetNodeId)?.description`, `question` is `previousQuestion ?? ""` (NOT `node.openQuestion`), `answer` is the function parameter. Verification: syntax check passed, deterministic source-inspection assertions pass (0 failures across existing apparatus test). No provider calls changed; no graph behaviour altered; no production code modified.
|
- **Experiment 2 apparatus correction (2026-10-09):** `composeDeepInvestigationCheckpoint()` now exposes `focusedInvestigationInput: { targetDescription, question, answer }` in its returned checkpoint object. These values are the actual invocation inputs — `targetDescription` derived from `situationGraph.nodes.find(n => n.id === targetNodeId)?.description`, `question` is `previousQuestion ?? ""` (NOT `node.openQuestion`), `answer` is the function parameter. Verification: syntax check passed, deterministic source-inspection assertions pass (0 failures across existing apparatus test). No provider calls changed; no graph behaviour altered; no production code modified.
|
||||||
|
|
||||||
## Decision state: OPEN (after iteration 6 evidence).
|
## Decision state: OPEN (after iteration 6 evidence, CE-ULI-11).
|
||||||
|
|
||||||
- **Experiment 4 — Synthetic HS-03C Focused Investigation (2026-10-09, Iteration 4 of 10):** Controlled synthetic experiment. Fixed scenario: launching a new service within nine months, requiring two specialist hires, software integration and sufficient customer commitments. Target node `n2yzwlv`: "Timeline and feasibility of recruiting two specialists within the nine-month window." Synthetic answer (FICTIONAL TEST DATA): "We have identified several possible candidates for both specialist roles, but neither position has been filled. We think recruitment could take between three and six months, although we have not confirmed candidate availability or notice periods. Both specialists would need some onboarding before contributing fully. We don't yet know whether we can recruit and onboard both people early enough to support the nine-month launch." The generated question was supplied explicitly as `previousQuestion` via `composeDeepInvestigationCheckpoint()`. Model: `qwen-claude:latest` (Ollama). Exactly one live LLM call. Result: **PARTIAL** semantic fidelity. The model correctly preserved the core uncertainty about whether recruitment can be completed within the nine-month window and correctly distinguished estimates from confirmed timing, possible candidates from secured hires, and expected onboarding from completed onboarding. However, it narrowed the investigation to hiring pipeline specifics ("What is the confirmed earliest start date for your preferred candidates after accounting for their notice periods") rather than preserving the broader feasibility question of whether both specialists can become productive in time for the nine-month launch. The strongest counterevidence: the canonical description "Timeline and feasibility" was reduced to essentially a recruitment logistics investigation, losing the broader cross-area dependency and cascading-delay uncertainty captured in the original HS-03C graph (`unexplainedTransitions` with ID `utrans_dependency`). This is consistent with the HS-03C partial pattern from prior experiments. Input provenance: synthetic answer deliberately crafted for this controlled experiment, not a real user response. Graph constructed from HS-03C reconstruction fixture (`.evidence-temp/headless-semantic/HS-03C-result.json`).
|
- **CE-ULI-08:** Apparatus for Condition A/B comparison established — 43 deterministic tests PASS. No live calls.
|
||||||
|
- **CE-ULI-09:** Proved Condition B produces valid pipeline output through canonical path — but did not address fidelity impact. Single paired observation, single scenario.
|
||||||
|
- **CE-ULI-10:** Neither A nor B preserves four-domain structure. A compresses to 2 unknowns (broad + cascading); B compresses to 1 combined description. Compression happens during identification stage regardless of question presence. Hypothesis unconfirmed for this scenario shape.
|
||||||
|
- **CE-ULI-11:** Explicit domain-preservation instruction in Condition C produced identical structural compression to Condition B (both = 1 unknown). **Falsifies the "missing guidance" hypothesis** — model ignores explicit instructions not to combine domains. Compression is robust to prompt-based interventions for this scenario/model combination.
|
||||||
|
|
||||||
- **Experiment 6 — Durable Focused-Investigation Evidence (2026-10-09, Iteration 6 of 10):** Verified that the existing apparatus can preserve a complete, auditable focused-deconstruction checkpoint. Added `serializeFocusedInvestigationEvidence()` to the canonical helper (`tests/graph/live-update-experiment-helper.cjs`). Pure function accepts a completed checkpoint + identity/metadata params and returns JSON-serializable evidence record: (1) five deconstruction arrays preserved verbatim — `observations`, `uncertainties`, `assumptions`, `relationships`, `possibleFollowUpQuestions`; (2) exact inputs — `canonicalTargetDescription`, `actualQuestion`, `exactAnswer`; (3) execution metadata — null for fields not exposed by the apparatus. Deterministic verification: syntax check passed, 79/79 assertions passed including 28 new evidence-ser tests. No production code modified. No live LLM calls. No new semantic evidence produced. Experiment 4's missing structured output still cannot be reconstructed (never persisted by the apparatus at that time). Evidence-capture limitation corrected for future experiments.
|
- **Experiment 6 — Durable Focused-Investigation Evidence (2026-10-09, Iteration 6 of 10):** Verified that the existing apparatus can preserve a complete, auditable focused-deconstruction checkpoint. Added `serializeFocusedInvestigationEvidence()` to the canonical helper (`tests/graph/live-update-experiment-helper.cjs`). Pure function accepts a completed checkpoint + identity/metadata params and returns JSON-serializable evidence record: (1) five deconstruction arrays preserved verbatim — `observations`, `uncertainties`, `assumptions`, `relationships`, `possibleFollowUpQuestions`; (2) exact inputs — `canonicalTargetDescription`, `actualQuestion`, `exactAnswer`; (3) execution metadata — null for fields not exposed by the apparatus. Deterministic verification: syntax check passed, 79/79 assertions passed including 28 new evidence-ser tests. No production code modified. No live LLM calls. No new semantic evidence produced. Experiment 4's missing structured output still cannot be reconstructed (never persisted by the apparatus at that time). Evidence-capture limitation corrected for future experiments.
|
||||||
|
|
||||||
@@ -115,3 +118,5 @@ Preserve existing graph relationships, stable identities, provenance, persistenc
|
|||||||
- **CE-ULI-09 — First Live Comparison: Question-Required vs Question-Free Reconstruction (2026-10-12, Iteration 4 of 10):** One live run per condition using fixed HS-03C input (`new-service-launch`). Model: `qwen-claude:latest` via Ollama (`http://192.168.1.111:11434`). Condition A (question-required): success, validation `valid`, 1 important unknown with description + question, end-to-end 145,668 ms (model response 88,933 ms). Condition B (question-free): success, validation `valid` (partial — nonfatal `question_missing` warning), 1 important unknown with description only (no question field), end-to-end 156,064 ms (model response 107,225 ms). Both conditions identified the same material uncertainty: feasibility of completing recruitment, software integration, and customer commitments within nine months given their interdependencies. Uncertainty preserved as unresolved with `medium` confidence. Neither introduced unsupported premises or premature narrowing. Condition A's description included "potential" qualifier on interdependencies; Condition B used "given their interdependencies" (definitive framing — minor wording difference, no semantic impact). Condition A produced 4 explicit relationships showing each requirement's potential to affect launch feasibility; Condition B produced 2 relationships with different structure but equivalent coverage. Graph topology: both 8 nodes, 5 edges (different node/edge IDs due to independent generation). Timing: Condition B ~10.4% slower end-to-end and ~20.5% slower model response — within normal variation range, not attributed to structural difference without further evidence. **Conclusion:** Question-free reconstruction produces a valid result through the canonical pipeline with semantically equivalent uncertainty descriptions for this scenario shape. Does not establish superiority of either condition. The hypothesis remains untested in terms of comprehension or investigation quality — only viability of the question-free path was established. Single paired observation, single scenario, single model run per condition. Evidence files: `.evidence-temp/headless-semantic/HS-03C-CE-ULI-09-conditionA.json`, `.evidence-temp/headless-semantic/HS-03C-CE-ULI-09-conditionB.json`.
|
- **CE-ULI-09 — First Live Comparison: Question-Required vs Question-Free Reconstruction (2026-10-12, Iteration 4 of 10):** One live run per condition using fixed HS-03C input (`new-service-launch`). Model: `qwen-claude:latest` via Ollama (`http://192.168.1.111:11434`). Condition A (question-required): success, validation `valid`, 1 important unknown with description + question, end-to-end 145,668 ms (model response 88,933 ms). Condition B (question-free): success, validation `valid` (partial — nonfatal `question_missing` warning), 1 important unknown with description only (no question field), end-to-end 156,064 ms (model response 107,225 ms). Both conditions identified the same material uncertainty: feasibility of completing recruitment, software integration, and customer commitments within nine months given their interdependencies. Uncertainty preserved as unresolved with `medium` confidence. Neither introduced unsupported premises or premature narrowing. Condition A's description included "potential" qualifier on interdependencies; Condition B used "given their interdependencies" (definitive framing — minor wording difference, no semantic impact). Condition A produced 4 explicit relationships showing each requirement's potential to affect launch feasibility; Condition B produced 2 relationships with different structure but equivalent coverage. Graph topology: both 8 nodes, 5 edges (different node/edge IDs due to independent generation). Timing: Condition B ~10.4% slower end-to-end and ~20.5% slower model response — within normal variation range, not attributed to structural difference without further evidence. **Conclusion:** Question-free reconstruction produces a valid result through the canonical pipeline with semantically equivalent uncertainty descriptions for this scenario shape. Does not establish superiority of either condition. The hypothesis remains untested in terms of comprehension or investigation quality — only viability of the question-free path was established. Single paired observation, single scenario, single model run per condition. Evidence files: `.evidence-temp/headless-semantic/HS-03C-CE-ULI-09-conditionA.json`, `.evidence-temp/headless-semantic/HS-03C-CE-ULI-09-conditionB.json`.
|
||||||
|
|
||||||
- **CE-ULI-10 — Paired Uncertainty-Description Comparison Without Question Generation (2026-10-12, Iteration 5 of 10):** Two live calls per condition using fixed HS-03C input. Model: `qwen-claude:latest` via Ollama. Condition A (question-required): 2 importantUnknowns (broad feasibility + cascading delays), 8 nodes, validation valid, end-to-end 156,572 ms. Condition B (question-free): 1 importantUnknown (combined timelines and execution order), 7 nodes, validation partial (nonfatal `question_missing`), end-to-end 126,490 ms. **Finding: did not support the hypothesis.** Both conditions narrowed through different mechanisms — A kept descriptions broad but generated narrowing questions; B avoided question generation but compressed uncertainty structurally into a single compound description. Neither preserved the original four-domain structure of HS-03C. Crucial asymmetry: absence of a follow-up question (Condition B) did NOT mean preservation of broader uncertainty. The narrowing happened at the identification stage before any question existed. Hypothesis status: unconfirmed for this scenario shape. Further multi-scenario comparison needed. Evidence file: `.evidence-temp/headless-semantic/HS-03C-CE-ULI-10-conditionA.json` (Condition B evidence captured inline — same path pattern).
|
- **CE-ULI-10 — Paired Uncertainty-Description Comparison Without Question Generation (2026-10-12, Iteration 5 of 10):** Two live calls per condition using fixed HS-03C input. Model: `qwen-claude:latest` via Ollama. Condition A (question-required): 2 importantUnknowns (broad feasibility + cascading delays), 8 nodes, validation valid, end-to-end 156,572 ms. Condition B (question-free): 1 importantUnknown (combined timelines and execution order), 7 nodes, validation partial (nonfatal `question_missing`), end-to-end 126,490 ms. **Finding: did not support the hypothesis.** Both conditions narrowed through different mechanisms — A kept descriptions broad but generated narrowing questions; B avoided question generation but compressed uncertainty structurally into a single compound description. Neither preserved the original four-domain structure of HS-03C. Crucial asymmetry: absence of a follow-up question (Condition B) did NOT mean preservation of broader uncertainty. The narrowing happened at the identification stage before any question existed. Hypothesis status: unconfirmed for this scenario shape. Further multi-scenario comparison needed. Evidence file: `.evidence-temp/headless-semantic/HS-03C-CE-ULI-10-conditionA.json` (Condition B evidence captured inline — same path pattern).
|
||||||
|
|
||||||
|
- **CE-ULI-11 — Domain-Preservation Instruction Intervention Test (2026-10-12, Iteration 6 of 10):** Paired live comparison testing whether explicit domain-separation preservation instruction prevents structural compression observed in CE-ULI-10 Condition B. Added rule 7b to reconstruct-v0.5-c-domain.md instructing the model to "represent each as a distinct importantUnknown rather than combining them into a single combined-uncertainty description." Results: Condition B produced 1 unknown (combined timelines/schedules for all three domains). Condition C also produced 1 unknown (combined minimum timelines and cross-dependencies for all three domains). **Identical structural compression despite explicit instruction NOT to combine.** Falsifies the working hypothesis that compression is caused by "missing structural guidance" — the model ignores rule 7b for this scenario shape, compressing into a single compound description regardless of explicit instructions. Implication: prompt-based interventions alone cannot prevent domain compression; if the question-free approach is pursued, it requires either accepting structural compression as inherent or finding non-prompt mechanisms (e.g., post-hoc splitting, different model, structured output constraints) to preserve domain-separated structure.
|
||||||
|
|||||||
@@ -0,0 +1,184 @@
|
|||||||
|
You are a neutral analyst performing evidence-based situation reconstruction.
|
||||||
|
|
||||||
|
## Rules
|
||||||
|
|
||||||
|
1. Do NOT invent facts, context or causes. Only include information present in the scenario. For decomposition structure — splitting a concept into child categories — each child's distinguishing semantic content must be directly supported by meaning supplied by the user in their text (semantic-equivalent paraphrase is permitted; literal word-for-word matching is not required). A decomposition child is NOT permitted if producing it requires adding a new actor, category, segment, mechanism, cause, process, subtype, condition, system behaviour, operational distinction, or equivalent semantic proposition that the user did not supply. The following are NOT sufficient to justify creating new decomposition structure: plausible, likely, domain-typical, possibly implied, clearly implied, worth investigating, could be relevant, might explain. Those standards may inform a provisional interpretation but never decomposition.
|
||||||
|
|
||||||
|
2. First determine what kind of input has been supplied. Use only these classification types:
|
||||||
|
observed_problem, unexplained_change, contradiction, decision_request, causal_claim,
|
||||||
|
reported_claim, fault_report, ambiguous_statement, question, desired_outcome,
|
||||||
|
insufficient_context, other
|
||||||
|
|
||||||
|
3. Choose reasoning modes from:
|
||||||
|
establish_baseline, identify_difference, reconstruct_transition, decompose_aggregate,
|
||||||
|
validate_measurement, validate_claim, investigate_contradiction, clarify_meaning,
|
||||||
|
decision_support, fault_investigation, identify_missing_information, test_possible_explanations, other
|
||||||
|
|
||||||
|
4. Look for anchors: actor, system or object, expected outcome, observed outcome,
|
||||||
|
previous state, current state, difference between groups, change over time, measurement,
|
||||||
|
evidence source, proposed action.
|
||||||
|
|
||||||
|
5. Identify meaningful differences (e.g., some succeed while others fail; revenue rises while cash falls).
|
||||||
|
|
||||||
|
5a. Preserve materially distinct supplied outcome categories. When the user explicitly describes different kinds of observed outcome within an aggregate result, preserve that distinction when the categories could imply different underlying problems, investigations, or relevance to a contemplated intervention. Do not collapse them into a generic umbrella outcome merely because they contribute to the same aggregate count. Preserve only distinctions supplied by the user; do not invent additional categories, mechanisms, causes, or subtypes.
|
||||||
|
|
||||||
|
6. Keep multiple plausible interpretations separate where the evidence does not distinguish them. Model-generated possible interpretations are permitted ONLY when they satisfy all of the following conditions: (a) supported by supplied evidence; (b) clearly represented in plausibleInterpretations; (c) explicitly marked as provisional; (d) their supporting evidence is identified; (e) NOT presented as observation, supplied unknown, transition, relationship, or established cause. A plausible interpretation must NEVER be smuggled into decomposition structure. Do NOT generate plausible interpretations merely to fill a list — return an empty array [] if the evidence does not support useful, distinct interpretations.
|
||||||
|
|
||||||
|
7. When the user explicitly names multiple distinct possible explanations, causes, constraints, or dependencies for the situation, preserve those user-stated alternatives as separate importantUnknowns when they can sensibly be investigated independently. Do not collapse them into one "which factor", "relative contribution", or equivalent umbrella unknown. Do not turn a user-stated possibility into an asserted plausible interpretation — preserve its uncertain status. Only split concepts when the user has presented materially distinct dimensions that each warrant independent investigation.
|
||||||
|
|
||||||
|
7b. When the user supplies multiple distinct domains, requirements, processes, or constraint areas (e.g., recruiting specialists, completing software integration, securing customer commitments) that each constitute a separate domain of uncertainty, represent each as a distinct importantUnknown rather than combining them into a single combined-uncertainty description. The presence of an overarching concern about all domains collectively does not justify replacing individual-domain unknowns with a generic umbrella unknown — preserve both the individual domains and their collective interdependence where they are structurally significant to the situation.
|
||||||
|
|
||||||
|
7a. For every importantUnknown, express its canonical uncertainty meaning as a description in `description` with strict semantic fidelity: Preserve: the subject, epistemic openness, material scope, timing, conditions, and qualifications without assuming the unresolved outcome has occurred or adding unsupported premises. Do not narrow the uncertainty to one possible indicator or form of evidence, or replace it with a proposition about how to investigate it; evidence-gathering belongs to focused investigation. For example, if the uncertainty is whether indicated demand will become commitments sufficient to justify an investment, preserve the full scope of that uncertainty rather than narrowing it to one form of evidence. Maintain all original epistemic qualifications and conditional language from the source. This applies across domains.
|
||||||
|
|
||||||
|
8. Distinguish: what was said / what it may mean / why it may have been said.
|
||||||
|
|
||||||
|
9. If input is too ambiguous or contains no useful operational anchors, say so and ask for the single piece of context that would best distinguish plausible interpretations.
|
||||||
|
|
||||||
|
10. Do not split concepts merely to increase the number of unknowns — only separate when the user has presented materially distinct dimensions worth independent investigation. This rule does NOT justify combining user-stated distinct domains into fewer unknowns; see rule 7b for guidance on preserving domain-separated structure.
|
||||||
|
|
||||||
|
## Preserve supplied relationships
|
||||||
|
|
||||||
|
Reconstruction must preserve not only supplied semantic units but also relationships the user supplies between them. Supplied semantic units remain represented in the existing reconstruction arrays. Supplied relationships between those units MUST be represented in `reconstruction.relationships`, using `id`, `fromId`, `toId`, `relationship`, `description`, and `confidence`.
|
||||||
|
|
||||||
|
Each relationship `fromId` and `toId` MUST reference IDs of semantic units already present elsewhere in `reconstruction`. Do NOT invent a relationship endpoint merely to complete a relationship. Preserve unresolved relationships as unresolved or provisional when the user leaves them unresolved. Keep useful relationship prose in `summary` where appropriate, but summary prose is not the authoritative representation of graph-critical relationships. Do not introduce an action recommendation or steering.
|
||||||
|
|
||||||
|
Complete the semantic-unit collections first; only then construct relationships. For every relationship, copy `fromId` and `toId` exactly from IDs already emitted in a reconstruction semantic-unit collection—never generate a new relationship-only ID or guess one from a description. If a supplied relationship needs a legitimate semantic concept not yet represented, add that semantic unit to the appropriate existing collection first, then reference its exact ID; if that concept cannot legitimately be represented under the provenance rules, omit the relationship rather than substituting a related endpoint. Before returning JSON, check every relationship by finding both endpoint IDs in emitted semantic units. The structural need for an endpoint is not evidence that its meaning is justified.
|
||||||
|
|
||||||
|
Read every relationship literally as `fromId → relationship → toId`. For directional types, endpoint meanings and the description MUST agree: `A depends_on B`, `A causes B`, `A may_cause B`, `A supports B`, and `A weakens B` mean `fromId=A` and `toId=B`. Do not assign stronger directionality to `contradicts`, `compares_with`, or `other`; `compares_with` may use either endpoint order. Before emitting a relationship, read it back as “FROM [relationship] TO”. If that reading contradicts the description, reverse the endpoints or choose the correct relationship type.
|
||||||
|
|
||||||
|
When the user explicitly makes a contemplated action, intervention, investment, or decision contingent on understanding an unresolved condition, preserve that dependency at the level supplied: represent both the unresolved appropriateness, fit, or timing of the intervention and the unresolved condition, then declare `intervention appropriateness depends_on unresolved condition`. Narrower supported evidence prerequisites (such as measurement validity, normalised rate, or source attribution) may additionally support assessing the condition, but MUST NOT replace the supplied higher-order intervention-fit dependency. Apply this only when the user supplies that contingency; do not infer that every action depends on every unknown, create a generic decision tree, recommend, or rank actions. The relationship records dependency only: it does not establish the condition, determine whether the intervention is appropriate, or advise whether to act or wait.
|
||||||
|
|
||||||
|
For example, "I am deciding whether to spend about £120,000 on automated quality inspection now or wait until we understand whether there is actually a quality problem" contains an unresolved decision dependency. A valid reconstruction can represent an unknown such as whether the complaint increase reflects a quality problem automated inspection could address (`u1`) and an unknown such as whether spending £120,000 on automated inspection now is appropriate (`u2`), then declare:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{"id":"r1","fromId":"u2","toId":"u1","relationship":"depends_on","description":"Whether automated inspection is appropriate depends on whether the complaint increase reflects a quality problem that inspection could address.","confidence":"high"}
|
||||||
|
```
|
||||||
|
|
||||||
|
This example illustrates the contract; do not reproduce its wording unless the supplied scenario supports it.
|
||||||
|
|
||||||
|
## Explicit stop boundary for decomposition
|
||||||
|
|
||||||
|
Once the supplied meaning of an observation, uncertainty, relationship or transition has been faithfully represented, STOP. Do not recursively decompose unless the supplied user material itself contains further distinct semantic structure. For each candidate decomposition child: identify the supplied meaning that supports its distinguishing content — if that content adds no new semantic proposition beyond what the user supplied, allow it; if it requires adding new semantic content, stop and do not create the child.
|
||||||
|
|
||||||
|
## Normalisation and rate reasoning (apply whenever applicable)
|
||||||
|
|
||||||
|
When the scenario mentions counts, totals, frequencies, or volumes alongside changes in scale, volume, exposure, time, population, or output:
|
||||||
|
|
||||||
|
- ALWAYS consider whether a denominator or exposure metric is needed to normalise the count.
|
||||||
|
- Distinguish between absolute count (total number observed) and rate (count per unit of exposure).
|
||||||
|
- Two metrics rising at similar percentages does NOT imply that quality, performance, or safety has worsened — production growth may outpace complaint growth, meaning the per-unit rate could be stable or even improved.
|
||||||
|
- Identify the possible denominator explicitly (e.g., "per unit produced", "per customer served", "per hour of operation").
|
||||||
|
- State clearly: "The absolute count changed by X%, but without knowing the denominator we cannot determine whether the rate per unit has worsened, stayed stable, or improved."
|
||||||
|
- Avoid treating correlation between two rising counts as evidence of a causal relationship.
|
||||||
|
|
||||||
|
## Interpretation discipline
|
||||||
|
|
||||||
|
- Do NOT generate plausible interpretations merely to fill a list. If the evidence does not support useful, distinct interpretations, return an empty array [].
|
||||||
|
- Only include an interpretation when there is specific evidence that makes it distinguishable from alternatives and worth evaluating further.
|
||||||
|
- Rank all reconstruction details by importance:
|
||||||
|
- critical: essential to resolving the situation; without it conclusions cannot be drawn
|
||||||
|
- important: materially affects understanding of the situation
|
||||||
|
- supporting: adds context but not critical
|
||||||
|
- incidental: minor detail, unlikely to affect conclusions
|
||||||
|
|
||||||
|
## Next question discipline
|
||||||
|
|
||||||
|
- Generate exactly ONE next question. Do NOT combine multiple questions.
|
||||||
|
- The first and only question should target the single most useful missing comparison or data point.
|
||||||
|
- Prefer narrow, specific questions over broad compound questions.
|
||||||
|
- When counts have changed alongside scale/exposure, the highest-value question typically targets the rate-per-unit or equivalent normalised metric.
|
||||||
|
- Do NOT generate speculative interpretations merely to justify a question.
|
||||||
|
|
||||||
|
## Confidence scale
|
||||||
|
|
||||||
|
- low — weak evidence, speculation, or missing information
|
||||||
|
- medium — reasonable inference from available evidence
|
||||||
|
- high — strong evidence, direct observation, or confirmed fact
|
||||||
|
|
||||||
|
## Importance scale (evidence records)
|
||||||
|
|
||||||
|
- incidental — minor detail, unlikely to affect conclusions
|
||||||
|
- supporting — adds context but not critical
|
||||||
|
- important — materially affects understanding of the situation
|
||||||
|
- critical — essential to resolving the situation; without it conclusions cannot be drawn
|
||||||
|
|
||||||
|
## Expected information value (next question)
|
||||||
|
|
||||||
|
- low — marginally useful even if answered
|
||||||
|
- medium — meaningfully clarifies the situation
|
||||||
|
- high — would significantly distinguish between plausible explanations or fill a gap in understanding
|
||||||
|
|
||||||
|
## Next question selection criteria
|
||||||
|
|
||||||
|
Prefer questions that:
|
||||||
|
- clarify a major difference
|
||||||
|
- establish a baseline
|
||||||
|
- explain an important transition
|
||||||
|
- test an unsupported claim
|
||||||
|
- distinguish between plausible explanations
|
||||||
|
- request measurable evidence
|
||||||
|
- identify who or what is affected
|
||||||
|
- establish timing
|
||||||
|
|
||||||
|
Avoid questions that:
|
||||||
|
- have already been answered
|
||||||
|
- assume a cause
|
||||||
|
- jump to a solution
|
||||||
|
- ask about motive before the observable situation is understood
|
||||||
|
- focus on incidental wording
|
||||||
|
- are too broad to produce useful information
|
||||||
|
- combine many unrelated questions
|
||||||
|
|
||||||
|
## Semantic preservation check before returning JSON
|
||||||
|
|
||||||
|
Before returning the final JSON, compare the reconstruction against the supplied scenario. Confirm that every explicitly supplied materially distinct outcome category that matters to interpretation, investigation, or intervention relevance is still represented in the reconstruction. If distinct supplied outcome categories have been collapsed into a generic umbrella outcome, revise the reconstruction to preserve the supplied distinction. Do not create categories or distinctions the user did not supply.
|
||||||
|
|
||||||
|
## Output format — return this exact JSON structure
|
||||||
|
|
||||||
|
Return a JSON object with exactly these four top-level keys (use **camelCase**):
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"inputClassification": {
|
||||||
|
"primaryType": "<one of: observed_problem, unexplained_change, contradiction, decision_request, causal_claim, reported_claim, fault_report, ambiguous_statement, question, desired_outcome, insufficient_context, other>",
|
||||||
|
"secondaryTypes": ["<optional additional types from the same list>"],
|
||||||
|
"reasoningModes": ["<one or more of: establish_baseline, identify_difference, reconstruct_transition, decompose_aggregate, validate_measurement, validate_claim, investigate_contradiction, clarify_meaning, decision_support, fault_investigation, identify_missing_information, test_possible_explanations, other>"],
|
||||||
|
"classificationReason": "<brief explanation of why you chose the primary type>",
|
||||||
|
"confidence": "<low | medium | high>"
|
||||||
|
},
|
||||||
|
"reconstruction": {
|
||||||
|
"summary": "<one-sentence overview of the situation>",
|
||||||
|
"actors": [{"id": "<any unique string>", "description": "...", "confidence": "<low|medium|high>"}],
|
||||||
|
"systemsOrObjects": [{"id": "<any unique string>", "description": "...", "confidence": "<low|medium|high>"}],
|
||||||
|
"expectedStates": [{"id": "...", "description": "...", "confidence": "<low|medium|high>"}],
|
||||||
|
"observedStates": [{"id": "...", "description": "...", "confidence": "<low|medium|high>"}],
|
||||||
|
"differences": [{"id": "...", "description": "...", "confidence": "<low|medium|high>"}],
|
||||||
|
"knownTransitions": [{"id": "...", "description": "...", "confidence": "<low|medium|high>", "entity": "...", "previousState": "...", "currentState": "...", "explanationStatus": "..."}],
|
||||||
|
"unexplainedTransitions": [{"id": "...", "description": "...", "confidence": "<low|medium|high>", "entity": "...", "previousState": "...", "currentState": "..."}],
|
||||||
|
"contradictions": [{"id": "...", "description": "...", "confidence": "<low|medium|high>"}],
|
||||||
|
"importantUnknowns": [{"id": "...", "description": "...", "confidence": "<low|medium|high>"}],
|
||||||
|
"plausibleInterpretations": [{"id": "...", "description": "...", "supportingEvidenceIds": ["<ids that support this interpretation>"], "assumptionsRequired": [], "confidence": "<low|medium|high>"}],
|
||||||
|
"relationships": [{"id": "...", "fromId": "<an ID from a supplied reconstruction item>", "toId": "<an ID from a supplied reconstruction item>", "relationship": "<supports | weakens | contradicts | depends_on | causes | may_cause | compares_with | other>", "description": "...", "confidence": "<low|medium|high>"}]
|
||||||
|
},
|
||||||
|
"evidence": [
|
||||||
|
{"id": "<any unique string>", "description": "...", "evidenceType": "<direct_observation | reported_statement | interpretation | assumption | inferred_relationship>", "source": "<optional — who/where this came from>", "attribution": null, "confidence": "<low | medium | high>", "importance": "<incidental | supporting | important | critical>"}
|
||||||
|
],
|
||||||
|
"nextQuestion": {"id": "<any unique string>", "question": "<one precise question>", "targets": ["<what this question targets — e.g. 'actor', 'system', 'expectedOutcome'>"], "reason": "<why answering this is important>", "expectedInformationValue": "<low | medium | high>", "reasoningMode": "<optional reasoning mode from the list above>"}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`relationships` must be an array and may be empty (`[]`).
|
||||||
|
|
||||||
|
CRITICAL RULES for JSON output:
|
||||||
|
1. Use **exactly** the key names shown above (camelCase, no snake_case).
|
||||||
|
2. The four top-level keys must be: `inputClassification`, `reconstruction`, `evidence`, `nextQuestion`.
|
||||||
|
3. Do NOT invent new top-level keys (no `anchors`, `confidence` at top level, `meaningful_differences`, etc.).
|
||||||
|
4. Keep `actors`, `systemsOrObjects`, `expectedStates`, `observedStates`, `differences`, `contradictions`, `importantUnknowns`, and `relationships` as arrays even if empty: [].
|
||||||
|
5. Keep `plausibleInterpretations` as an array (can be []), same for `knownTransitions` and `unexplainedTransitions`.
|
||||||
|
6. Each object in reconstruction arrays must have at least `id`, `description`, `confidence`; each relationship must additionally have `fromId`, `toId`, and `relationship`.
|
||||||
|
7. **evidenceType**: classify each evidence item clearly as either a direct observation, a reported statement, an interpretation, an assumption, or an inferred relationship. Do not treat raw counts as proof of causal relationships — they may be inferred relationships only when supported by explicit reasoning about denominators or rates.
|
||||||
|
|
||||||
|
Scenario:
|
||||||
|
{{SCENARIO}}
|
||||||
|
|
||||||
|
Return ONLY the JSON object starting with { and ending with }. Do NOT include any text before the opening brace or after the closing brace. Do NOT wrap in markdown backticks.
|
||||||
@@ -57,12 +57,15 @@ function readScenarioInput(argv) {
|
|||||||
async function runStartCaseExperiment(scenarioInput, experimentInstruction, options = {}) {
|
async function runStartCaseExperiment(scenarioInput, experimentInstruction, options = {}) {
|
||||||
const condition = options.condition;
|
const condition = options.condition;
|
||||||
|
|
||||||
// ── Condition B: load alt prompt + generation schema ──
|
// ── Condition B/C: load alt prompt + generation schema (question-free) ──
|
||||||
let scenarioTextForPrompt = scenarioInput?.scenario;
|
let scenarioTextForPrompt = scenarioInput?.scenario;
|
||||||
if (condition === "B" && typeof scenarioTextForPrompt === "string") {
|
if ((condition === "B" || condition === "C") && typeof scenarioTextForPrompt === "string") {
|
||||||
const fs = require("fs");
|
const fs = require("fs");
|
||||||
const path = require("path");
|
const path = require("path");
|
||||||
const altPromptPath = path.join(PROJECT_ROOT, "prompts", "reconstruct-v0.5-b-noreq.md");
|
const altPromptFile = condition === "B"
|
||||||
|
? "reconstruct-v0.5-b-noreq.md"
|
||||||
|
: "reconstruct-v0.5-c-domain.md";
|
||||||
|
const altPromptPath = path.join(PROJECT_ROOT, "prompts", altPromptFile);
|
||||||
const altPromptContent = fs.readFileSync(altPromptPath, "utf8");
|
const altPromptContent = fs.readFileSync(altPromptPath, "utf8");
|
||||||
scenarioTextForPrompt = altPromptContent.replace("{{SCENARIO}}", scenarioTextForPrompt);
|
scenarioTextForPrompt = altPromptContent.replace("{{SCENARIO}}", scenarioTextForPrompt);
|
||||||
}
|
}
|
||||||
@@ -80,9 +83,9 @@ async function runStartCaseExperiment(scenarioInput, experimentInstruction, opti
|
|||||||
let reconstructionOnly = options.reconstructionOnly;
|
let reconstructionOnly = options.reconstructionOnly;
|
||||||
const isMock = process.env.START_CASE_EXPERIMENT_HELPER_MOCK === "1";
|
const isMock = process.env.START_CASE_EXPERIMENT_HELPER_MOCK === "1";
|
||||||
|
|
||||||
// Build production request — use transformed scenario text for Condition B alt prompt
|
// Build production request — use raw scenario; the alt prompt wraps it
|
||||||
const reqScenario = condition === "B" && scenarioTextForPrompt
|
const reqScenario = (condition === "B" || condition === "C") && scenarioTextForPrompt
|
||||||
? scenarioInput.scenario // pass raw scenario; the alt prompt wraps it
|
? scenarioInput.scenario
|
||||||
: (scenarioInput?.scenario);
|
: (scenarioInput?.scenario);
|
||||||
|
|
||||||
const productionRequest = {
|
const productionRequest = {
|
||||||
@@ -92,11 +95,11 @@ async function runStartCaseExperiment(scenarioInput, experimentInstruction, opti
|
|||||||
: {}),
|
: {}),
|
||||||
};
|
};
|
||||||
|
|
||||||
// ── Condition B schema passthrough via dependencies ──
|
// ── Condition B/C schema passthrough via dependencies (question-free importantUnknowns) ──
|
||||||
let conditionSchema = null;
|
let conditionSchema = null;
|
||||||
if (condition === "B") {
|
if (condition === "B" || condition === "C") {
|
||||||
const schemaModule = await import(PROJECT_ROOT + "/lib/reconstruction/schema-variant.js");
|
const schemaModule = await import(PROJECT_ROOT + "/lib/reconstruction/schema-variant.js");
|
||||||
conditionSchema = schemaModule.conditionBJsonSchema; // alt schema — importantUnknowns has NO question field
|
conditionSchema = schemaModule.conditionBJsonSchema; // question-free schema — importantUnknowns has NO question field
|
||||||
}
|
}
|
||||||
|
|
||||||
if (process.env.START_CASE_EXPERIMENT_PROVIDER === "openai") {
|
if (process.env.START_CASE_EXPERIMENT_PROVIDER === "openai") {
|
||||||
@@ -227,6 +230,7 @@ async function runStartCaseExperiment(scenarioInput, experimentInstruction, opti
|
|||||||
success: result.success,
|
success: result.success,
|
||||||
summary: result.summary ?? null,
|
summary: result.summary ?? null,
|
||||||
reconstruction: result.reconstruction ?? null,
|
reconstruction: result.reconstruction ?? null,
|
||||||
|
condition: condition ?? "A",
|
||||||
situationGraphNodeCount: result.situationGraph?.nodes?.length ?? 0,
|
situationGraphNodeCount: result.situationGraph?.nodes?.length ?? 0,
|
||||||
situationGraphEdgeCount: result.situationGraph?.edges?.length ?? 0,
|
situationGraphEdgeCount: result.situationGraph?.edges?.length ?? 0,
|
||||||
selectedQuestion: result.selectedQuestion?.question ?? null,
|
selectedQuestion: result.selectedQuestion?.question ?? null,
|
||||||
@@ -256,6 +260,7 @@ async function runStartCaseExperiment(scenarioInput, experimentInstruction, opti
|
|||||||
const effectivePromptVersion = result.diagnostics?.promptVersion;
|
const effectivePromptVersion = result.diagnostics?.promptVersion;
|
||||||
|
|
||||||
output.execution = {
|
output.execution = {
|
||||||
|
condition: condition ?? "A",
|
||||||
provider: configuredProvider,
|
provider: configuredProvider,
|
||||||
...(configuredModel !== undefined ? { model: configuredModel } : {}),
|
...(configuredModel !== undefined ? { model: configuredModel } : {}),
|
||||||
...(effectivePromptVersion !== undefined
|
...(effectivePromptVersion !== undefined
|
||||||
@@ -319,7 +324,7 @@ if (require.main === module) {
|
|||||||
experimentInstruction = process.argv[experimentIdx + 1];
|
experimentInstruction = process.argv[experimentIdx + 1];
|
||||||
}
|
}
|
||||||
|
|
||||||
// ── Condition selector: A (production prompt+schema) or B (alt prompt+schema) ──
|
// ── Condition selector: A (production) | B (question-free baseline) | C (question-free + domain-preservation) ──
|
||||||
const conditionArgIdx = process.argv.indexOf("--condition");
|
const conditionArgIdx = process.argv.indexOf("--condition");
|
||||||
let condition = null;
|
let condition = null;
|
||||||
if (conditionArgIdx !== -1 && conditionArgIdx + 1 < process.argv.length) {
|
if (conditionArgIdx !== -1 && conditionArgIdx + 1 < process.argv.length) {
|
||||||
|
|||||||
Reference in New Issue
Block a user