Compare commits
3
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
b71e3a8267 | ||
|
|
d89e0816b6 | ||
|
|
fd5241c6af |
@@ -550,6 +550,7 @@ These results are documented as **historical experiment evidence**. The v0.61 li
|
||||
|---|---|
|
||||
| Product evolution v0.51–v0.60 | `docs/design-evolution/README.md` (progressive loading) |
|
||||
| Methodology / RTO axioms | `docs/current-working-principles.md` §0 (A1–A12) |
|
||||
| Uncertainty-led initial investigation hypothesis | `docs/uncertainty-led-investigation-hypothesis.md` — **PROPOSED / UNTESTED — no implementation decision made** |
|
||||
| Architecture guardrails | `.claude/architecture-guardrails.md` |
|
||||
| Task routing by work type | `docs/task-context-packs.md` |
|
||||
| September 8 product/economics checkpoint | `docs/confidence-engine-product-checkpoint-2026-09-08.md` |
|
||||
|
||||
@@ -0,0 +1,96 @@
|
||||
# Confidence Engine — Uncertainty-Led Investigation Hypothesis
|
||||
|
||||
**Status:** PROPOSED / UNTESTED — no implementation decision made
|
||||
|
||||
**Date:** 9 October 2026
|
||||
|
||||
**Purpose:** Durable project-context checkpoint / Build → Break → Learn → STOP
|
||||
|
||||
## Purpose and status
|
||||
|
||||
This is a durable methodology hypothesis, not an approved architecture or product requirement. It records a possible simplification of Confidence Engine before further optimisation of initial Open Question wording, preserving the reasoning, evidence, limitations, competing explanations, falsification conditions, and explicit decision boundary.
|
||||
|
||||
## Origin: what prompted the hypothesis
|
||||
|
||||
Confidence Engine currently identifies canonical uncertainties during reconstruction and also generates natural-language Open Questions intended to express those uncertainties. The SituationGraph carries these representations into the user experience.
|
||||
|
||||
HS-01 observed one partially faithful question out of three: a future customer-commitment sufficiency uncertainty was narrowed to whether written commitments already existed. After a v0.5 prompt-contract refinement, HS-02 observed three faithful questions in one repeat of the same manufacturing case. That single run does not establish causality or general reliability.
|
||||
|
||||
HS-03 tested four scenarios: six faithful and four partially faithful transformations across ten uncertainties, with all four partial cases in HS-03C (a nine-month service launch with interacting recruitment, integration, and customer-commitment conditions). The generated questions tended to move from underlying feasibility, timing, or dependency uncertainties to present evidence, schedules, or bottlenecks. HS-03C also exposed a separate possible deconstruction omission: an overarching launch-feasibility uncertainty was not surfaced as its own canonical Open Question.
|
||||
|
||||
These results support investigating the framing boundary. They do not prove that question wording is the only problem, that canonical descriptions are always adequate, or that the proposed interaction is superior.
|
||||
|
||||
## The hypothesis
|
||||
|
||||
Confidence Engine may not need to convert each initially identified canonical uncertainty into a question. It may be more faithful and useful to present uncertainties as uncertainties, allowing the user to choose which one to explore.
|
||||
|
||||
The initial stage answers: “What remains uncertain?” The focused-investigation stage asks: “What can we explore to understand it better?” Contextual, evidence-seeking questions belong naturally inside the latter.
|
||||
|
||||
An uncertainty can remain unresolved throughout investigation. The user’s Current Understanding can develop, and the nature or degree of uncertainty can decrease, increase, or change. The user’s confidence in an outcome is not the same as confidence in understanding the situation; confidence, sufficiency, readiness to act, and action remain the user’s judgement.
|
||||
|
||||
## Illustrative interaction, not a specification
|
||||
|
||||
Initial presentation: “We have identified three areas of uncertainty in your situation. You can explore whichever you find useful.” Each item displays its canonical uncertainty with enough context to recognise its meaning; no generated interrogative is required.
|
||||
|
||||
Example uncertainty: “Timeline and feasibility of recruiting two specialists within the nine-month window.”
|
||||
|
||||
After the user selects it, focused investigation may ask: “What recruitment activity has already taken place?” or “What is known about the availability of suitable specialists?” Such questions may help investigate feasibility, but they are not equivalent replacements for the original uncertainty.
|
||||
|
||||
The user can answer questions, pause with “Done for now”, review an updated Current Understanding, defer the uncertainty, and revisit it. There is no mandated priority, compulsory resolution, or confidence score.
|
||||
|
||||
## Why this might simplify the system
|
||||
|
||||
It could remove the canonical-description → initial-question semantic transformation where HS-01 and HS-03C exhibited drift. That may reduce the need for further prompt tightening, fixed dictionaries, dynamic semantic dictionaries, or a separate semantic-equivalence layer for that transformation.
|
||||
|
||||
It does not remove the need for LLM interpretation during deconstruction or contextual reasoning during focused investigation. It also does not guarantee that the original uncertainty was correctly identified or well expressed.
|
||||
|
||||
Cross-domain and eventual multilingual flexibility may benefit from not imposing a fixed catalogue of question forms, but that benefit remains a design expectation, not an observed result.
|
||||
|
||||
## Continuity with Return to Origin
|
||||
|
||||
The originating methodology is an investigation process rather than a question-answer service. The user is the investigator; the graph organises and remembers the work; multiple uncertainties may coexist; progress is not synonymous with answering every question.
|
||||
|
||||
This hypothesis does not replace or reverse Return to Origin. It proposes a clearer boundary between identifying uncertainties and investigating them, consistent with user-selected focus, persistent Current Understanding, user-owned confidence, and “Done for now”.
|
||||
|
||||
Preserve existing graph relationships, stable identities, provenance, persistence, focused-investigation flow, and user ownership unless evidence shows a change is necessary.
|
||||
|
||||
## Competing explanations, failure risks, and falsification conditions
|
||||
|
||||
- The current initial question may provide genuine clarity or accessibility that a terse canonical description lacks. If direct uncertainty presentation reduces comprehension or confidence in choosing a focus, the hypothesis is weakened or falsified.
|
||||
- Focused investigation may rely on the generated initial question as input, an identity cue, or a semantic constraint. If removing or bypassing it degrades contextual questions, graph continuity, or Current Understanding, the hypothesis is weakened or falsified.
|
||||
- Some canonical descriptions may drift, omit overarching uncertainties, or use jargon. If direct presentation merely exposes or amplifies those defects, the proposed simplification does not solve the underlying problem.
|
||||
- The difference between a canonical uncertainty and a user-friendly noninterrogative title or description may still require a small linguistic transformation. Any such transformation must preserve meaning and be tested rather than assumed safe.
|
||||
- Changes to uncertainty or confidence over time must not be automatically inferred as numeric metrics or claims of resolution.
|
||||
- If the existing question transformation consistently improves clarity without material semantic loss across representative domains, retaining it may be preferable.
|
||||
|
||||
## Build → Break → Learn → STOP: proposed experiment
|
||||
|
||||
**BUILD** — First perform a zero-live-call, read-only architecture trace: reconstruction fields → SituationGraph → initial uncertainty display → focused-investigation entry → contextual questions → Current Understanding → “Done for now”. Identify every consumer of the initial question field. Do not modify production code in this phase.
|
||||
|
||||
**BUILD** — If the trace supports it, create the smallest reversible prototype that presents the canonical uncertainty directly and starts the existing focused investigation from it. Retain the existing flow and data for comparison; avoid schema migrations or broad refactors.
|
||||
|
||||
**BREAK** — Try to falsify the hypothesis. Test HS-03C, an ambiguous or jargon-heavy uncertainty, a nonbusiness scenario, and a case where the generated initial question arguably improves clarity. Check whether users understand what they are choosing, whether deeper questions remain contextual, and whether graph identity and Current Understanding remain coherent.
|
||||
|
||||
**LEARN** — Compare semantic fidelity, comprehension, investigation continuity, and user control against current behaviour. Record failures and successes, distinguishing observation from interpretation. Decide to adopt, revise, or reject only after evidence review.
|
||||
|
||||
**STOP** — No prompt-contract change, dictionary, semantic agent, UI refactor, architecture refactor, or production implementation is authorised by this document. Stop at the first apparatus failure or material contradiction and record it separately.
|
||||
|
||||
## Decision boundary and open questions
|
||||
|
||||
- Can focused investigation operate directly from canonical uncertainty and graph context without depending on the generated initial question?
|
||||
- Are canonical uncertainty descriptions consistently comprehensible, or do they need a distinct noninterrogative presentation layer?
|
||||
- How should the UI indicate that an uncertainty is being explored, parked, clarified, or still unresolved without equating status with certainty?
|
||||
- What evidence would demonstrate that uncertainty-led presentation improves rather than merely changes the experience?
|
||||
|
||||
**Decision state:** OPEN.
|
||||
|
||||
**Next permitted action:** Read-only repository trace and evidence audit; no live LLM calls and no production edits.
|
||||
|
||||
**Implementation authority:** None. This document must not be treated as an accepted product requirement or architectural decision.
|
||||
|
||||
## Evidence anchors and provenance
|
||||
|
||||
- Return to Origin methodology context, 18 August 2026: investigation rather than Q&A; user chooses investigation focus; graph as organiser and memory; Build → Break → Learn → STOP.
|
||||
- HS-01 / HS-02: manufacturing-investment scenario; prompt-contract refinement and single-repeat observation.
|
||||
- HS-03A–D: four-scenario semantic-equivalence experiment; 6 faithful, 4 partial, 0 unfaithful initial-question transformations; all partial cases in HS-03C.
|
||||
- Current implementation and branch/HEAD details are repository facts to verify at execution time, not facts established by this hypothesis.
|
||||
@@ -26,7 +26,7 @@ You are a neutral analyst performing evidence-based situation reconstruction.
|
||||
|
||||
7. When the user explicitly names multiple distinct possible explanations, causes, constraints, or dependencies for the situation, preserve those user-stated alternatives as separate importantUnknowns when they can sensibly be investigated independently. Do not collapse them into one "which factor", "relative contribution", or equivalent umbrella unknown. Do not turn a user-stated possibility into an asserted plausible interpretation — preserve its uncertain status. Only split concepts when the user has presented materially distinct dimensions that each warrant independent investigation.
|
||||
|
||||
7a. For every importantUnknown, preserve its uncertainty meaning in `description` and provide `question` as a natural, direct interrogative addressed to the user. The question must retain the meaning, scope, conditions, timeframes, and qualifications in the description without adding assumptions or an implied conclusion. Do not use fragments or indirect proposition forms such as "Whether …". This applies across domains.
|
||||
7a. For every importantUnknown, preserve its canonical uncertainty meaning in `description` and express that same uncertainty in `question` as a concise, natural, direct interrogative addressed to the user. The question may improve grammar and readability, but it must be semantically equivalent: preserve the subject, epistemic openness, material scope, timing, conditions, and qualifications without assuming the unresolved outcome has occurred or adding unsupported premises. Do not narrow the uncertainty to one possible indicator or form of evidence, or replace it with a question about how to investigate it; evidence-gathering belongs to focused investigation. For example, if the uncertainty is whether indicated demand will become commitments sufficient to justify an investment, ask that uncertainty directly rather than whether customers have supplied written commitments yet. Do not use fragments or indirect proposition forms such as "Whether …". This applies across domains.
|
||||
|
||||
8. Distinguish: what was said / what it may mean / why it may have been said.
|
||||
|
||||
|
||||
@@ -67,6 +67,12 @@ async function runStartCaseExperiment(scenarioInput, experimentInstruction, opti
|
||||
let reconstructionModelName = options.reconstructionModelName;
|
||||
let reconstructionOnly = options.reconstructionOnly;
|
||||
const isMock = process.env.START_CASE_EXPERIMENT_HELPER_MOCK === "1";
|
||||
const productionRequest = {
|
||||
scenario: scenarioInput?.scenario,
|
||||
...(scenarioInput?.promptVersion !== undefined
|
||||
? { promptVersion: scenarioInput.promptVersion }
|
||||
: {}),
|
||||
};
|
||||
|
||||
if (process.env.START_CASE_EXPERIMENT_PROVIDER === "openai") {
|
||||
if (!process.env.OPENAI_API_KEY) {
|
||||
@@ -95,13 +101,51 @@ async function runStartCaseExperiment(scenarioInput, experimentInstruction, opti
|
||||
}
|
||||
return {
|
||||
success: true,
|
||||
situationGraph: { nodes: [], edges: [], activeUnknownNodeId: null, resolvedNodeIds: [], currentSummary: "test" },
|
||||
reconstruction: {
|
||||
importantUnknowns: [
|
||||
{
|
||||
id: "unknown-1",
|
||||
description: "Canonical uncertainty from the deterministic test double",
|
||||
confidence: "medium",
|
||||
question: "What would resolve the deterministic uncertainty?",
|
||||
},
|
||||
],
|
||||
},
|
||||
situationGraph: {
|
||||
nodes: [
|
||||
{
|
||||
id: "graph-unknown-1",
|
||||
label: "Canonical uncertainty",
|
||||
description: "Canonical uncertainty from the deterministic test double",
|
||||
openQuestion: "What would resolve the deterministic uncertainty?",
|
||||
status: "unknown",
|
||||
kind: "unknown",
|
||||
},
|
||||
{
|
||||
id: "graph-state-1",
|
||||
label: "Known state",
|
||||
description: "A non-unknown node",
|
||||
status: "supported",
|
||||
kind: "state",
|
||||
},
|
||||
],
|
||||
edges: [],
|
||||
activeUnknownNodeId: null,
|
||||
resolvedNodeIds: [],
|
||||
currentSummary: "test",
|
||||
},
|
||||
assessment: { phase: "initial", progress: 0 },
|
||||
selectedQuestion: null,
|
||||
summary: null,
|
||||
diagnostics: {
|
||||
validationStatus: "mock",
|
||||
modelName: "inline-test-double",
|
||||
promptVersion: productionRequest.promptVersion ?? "v0.5",
|
||||
presentationValidationStatus: "valid",
|
||||
presentationValidationIssues: [],
|
||||
compatibilityApplied: false,
|
||||
compatibilityChanges: [],
|
||||
compatibilityWarnings: [],
|
||||
reconstructionOnly: dependencies.reconstructionOnly === true,
|
||||
graphReferenceValidation: { valid: true, errors: [] },
|
||||
},
|
||||
@@ -125,7 +169,7 @@ async function runStartCaseExperiment(scenarioInput, experimentInstruction, opti
|
||||
let result;
|
||||
const endToEndStartedAt = Date.now();
|
||||
try {
|
||||
result = await startCase(scenarioInput, {
|
||||
result = await startCase(productionRequest, {
|
||||
reconstructionProvider,
|
||||
reconstructionModelName,
|
||||
reconstructionOnly,
|
||||
@@ -147,7 +191,7 @@ async function runStartCaseExperiment(scenarioInput, experimentInstruction, opti
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
const output = {
|
||||
success: result.success,
|
||||
summary: result.summary ?? null,
|
||||
reconstruction: result.reconstruction ?? null,
|
||||
@@ -162,6 +206,47 @@ async function runStartCaseExperiment(scenarioInput, experimentInstruction, opti
|
||||
error: result.success ? null : (result.error ?? "unknown"),
|
||||
statusCode: result.statusCode ?? (result.success ? 200 : 500),
|
||||
};
|
||||
|
||||
if (reconstructionOnly) {
|
||||
if (scenarioInput?.experimentId !== undefined) {
|
||||
output.experimentId = scenarioInput.experimentId;
|
||||
}
|
||||
if (scenarioInput?.scenarioId !== undefined) {
|
||||
output.scenarioId = scenarioInput.scenarioId;
|
||||
}
|
||||
|
||||
const configuredProvider =
|
||||
process.env.START_CASE_EXPERIMENT_PROVIDER === "openai" ? "openai" : "ollama";
|
||||
const configuredModel =
|
||||
result.diagnostics?.modelName ??
|
||||
reconstructionModelName ??
|
||||
(configuredProvider === "ollama" ? process.env.OLLAMA_MODEL : undefined);
|
||||
const effectivePromptVersion = result.diagnostics?.promptVersion;
|
||||
|
||||
output.execution = {
|
||||
provider: configuredProvider,
|
||||
...(configuredModel !== undefined ? { model: configuredModel } : {}),
|
||||
...(effectivePromptVersion !== undefined
|
||||
? { promptVersion: effectivePromptVersion }
|
||||
: {}),
|
||||
};
|
||||
|
||||
if (Array.isArray(result.situationGraph?.nodes)) {
|
||||
output.graphUnknownNodes = result.situationGraph.nodes
|
||||
.filter((node) => node.kind === "unknown")
|
||||
.map((node) => ({
|
||||
id: node.id,
|
||||
label: node.label,
|
||||
description: node.description,
|
||||
...(node.openQuestion !== undefined
|
||||
? { openQuestion: node.openQuestion }
|
||||
: {}),
|
||||
...(node.status !== undefined ? { status: node.status } : {}),
|
||||
}));
|
||||
}
|
||||
}
|
||||
|
||||
return output;
|
||||
}
|
||||
|
||||
if (require.main === module) {
|
||||
|
||||
@@ -11,7 +11,7 @@
|
||||
|
||||
import { describe, expect, it, beforeAll, afterAll, vi } from "vitest";
|
||||
import { execFile } from "child_process";
|
||||
import { writeFile, unlink } from "fs/promises";
|
||||
import { readFile, writeFile, unlink } from "fs/promises";
|
||||
import { join, dirname } from "path";
|
||||
import { fileURLToPath } from "url";
|
||||
import { createRequire } from "module";
|
||||
@@ -89,16 +89,29 @@ function parseOutput(result) {
|
||||
|
||||
describe("start-case-experiment-helper.cjs apparatus", () => {
|
||||
let tmpFile = "";
|
||||
let metadataFile = "";
|
||||
|
||||
beforeAll(async () => {
|
||||
// Create a temporary scenario fixture for file-input tests
|
||||
tmpFile = join(rootDir, "tests", "fixtures", "_helper_scenario_temp.json");
|
||||
await writeFile(tmpFile, JSON.stringify({ scenario: "Scenario from file" }), "utf-8");
|
||||
metadataFile = join(rootDir, "tests", "fixtures", "_helper_semantic_scenario_temp.json");
|
||||
await writeFile(
|
||||
metadataFile,
|
||||
JSON.stringify({
|
||||
experimentId: "semantic-open-question-001",
|
||||
scenarioId: "manufacturing-expansion-001",
|
||||
scenario: "Scenario with semantic inspection metadata",
|
||||
promptVersion: "v0.5",
|
||||
}),
|
||||
"utf-8",
|
||||
);
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
// Clean up temp fixture
|
||||
try { await unlink(tmpFile); } catch {}
|
||||
try { await unlink(metadataFile); } catch {}
|
||||
});
|
||||
|
||||
// ── A — Positional scenario input ───────────────────────────────
|
||||
@@ -157,6 +170,141 @@ describe("start-case-experiment-helper.cjs apparatus", () => {
|
||||
expect(result.exitCode).toBe(0);
|
||||
expect(parseOutput(result).diagnostics.reconstructionOnly).toBe(true);
|
||||
});
|
||||
|
||||
it("preserves experiment metadata and exposes semantic evidence", async () => {
|
||||
const result = await runHelper(["--reconstruction-only", "--file", metadataFile]);
|
||||
expect(result.exitCode).toBe(0);
|
||||
const parsed = parseOutput(result);
|
||||
|
||||
expect(parsed.experimentId).toBe("semantic-open-question-001");
|
||||
expect(parsed.scenarioId).toBe("manufacturing-expansion-001");
|
||||
expect(parsed.execution).toEqual({
|
||||
provider: "ollama",
|
||||
model: "inline-test-double",
|
||||
promptVersion: "v0.5",
|
||||
});
|
||||
expect(parsed.reconstruction.importantUnknowns).toEqual([
|
||||
{
|
||||
id: "unknown-1",
|
||||
description: "Canonical uncertainty from the deterministic test double",
|
||||
confidence: "medium",
|
||||
question: "What would resolve the deterministic uncertainty?",
|
||||
},
|
||||
]);
|
||||
expect(parsed.graphUnknownNodes).toEqual([
|
||||
{
|
||||
id: "graph-unknown-1",
|
||||
label: "Canonical uncertainty",
|
||||
description: "Canonical uncertainty from the deterministic test double",
|
||||
openQuestion: "What would resolve the deterministic uncertainty?",
|
||||
status: "unknown",
|
||||
},
|
||||
]);
|
||||
expect(parsed.diagnostics).toMatchObject({
|
||||
validationStatus: "mock",
|
||||
presentationValidationStatus: "valid",
|
||||
presentationValidationIssues: [],
|
||||
graphReferenceValidation: { valid: true, errors: [] },
|
||||
compatibilityApplied: false,
|
||||
compatibilityChanges: [],
|
||||
compatibilityWarnings: [],
|
||||
});
|
||||
});
|
||||
|
||||
it("passes only production-supported request fields and calls startCase once", async () => {
|
||||
const { runStartCaseExperiment } = require(helperPath);
|
||||
const importantUnknowns = [
|
||||
{
|
||||
id: "unknown-a",
|
||||
description: "Canonical uncertainty A",
|
||||
confidence: "high",
|
||||
question: "What resolves uncertainty A?",
|
||||
},
|
||||
];
|
||||
const startCase = vi.fn(async () => ({
|
||||
success: true,
|
||||
reconstruction: { importantUnknowns },
|
||||
situationGraph: {
|
||||
nodes: [
|
||||
{
|
||||
id: "unknown-node-a",
|
||||
label: "Canonical uncertainty A",
|
||||
description: "Canonical uncertainty A",
|
||||
openQuestion: "What resolves uncertainty A?",
|
||||
status: "unknown",
|
||||
kind: "unknown",
|
||||
},
|
||||
{
|
||||
id: "observation-node-a",
|
||||
label: "Known observation",
|
||||
description: "Known observation",
|
||||
status: "supported",
|
||||
kind: "observation",
|
||||
},
|
||||
],
|
||||
edges: [],
|
||||
},
|
||||
selectedQuestion: null,
|
||||
summary: "test",
|
||||
diagnostics: {
|
||||
modelName: "configured-model",
|
||||
promptVersion: "v0.5",
|
||||
validationStatus: "valid",
|
||||
presentationValidationStatus: "partial",
|
||||
presentationValidationIssues: [{ category: "test-diagnostic" }],
|
||||
graphReferenceValidation: { valid: true, errors: [] },
|
||||
compatibilityApplied: true,
|
||||
compatibilityChanges: ["normalised"],
|
||||
compatibilityWarnings: ["warning"],
|
||||
},
|
||||
}));
|
||||
|
||||
const result = await runStartCaseExperiment(
|
||||
{
|
||||
experimentId: "experiment-a",
|
||||
scenarioId: "scenario-a",
|
||||
scenario: "Scenario A",
|
||||
promptVersion: "v0.5",
|
||||
ignoredMetadata: "must not cross the production boundary",
|
||||
},
|
||||
null,
|
||||
{ startCase, reconstructionOnly: true },
|
||||
);
|
||||
|
||||
expect(startCase).toHaveBeenCalledTimes(1);
|
||||
expect(startCase).toHaveBeenCalledWith(
|
||||
{ scenario: "Scenario A", promptVersion: "v0.5" },
|
||||
expect.objectContaining({ reconstructionOnly: true }),
|
||||
);
|
||||
expect(result.experimentId).toBe("experiment-a");
|
||||
expect(result.scenarioId).toBe("scenario-a");
|
||||
expect(result.reconstruction.importantUnknowns).toBe(importantUnknowns);
|
||||
expect(result.graphUnknownNodes).toEqual([
|
||||
{
|
||||
id: "unknown-node-a",
|
||||
label: "Canonical uncertainty A",
|
||||
description: "Canonical uncertainty A",
|
||||
openQuestion: "What resolves uncertainty A?",
|
||||
status: "unknown",
|
||||
},
|
||||
]);
|
||||
expect(result.execution).toEqual({
|
||||
provider: "ollama",
|
||||
model: "configured-model",
|
||||
promptVersion: "v0.5",
|
||||
});
|
||||
expect(result.diagnostics).toEqual({
|
||||
modelName: "configured-model",
|
||||
promptVersion: "v0.5",
|
||||
validationStatus: "valid",
|
||||
presentationValidationStatus: "partial",
|
||||
presentationValidationIssues: [{ category: "test-diagnostic" }],
|
||||
graphReferenceValidation: { valid: true, errors: [] },
|
||||
compatibilityApplied: true,
|
||||
compatibilityChanges: ["normalised"],
|
||||
compatibilityWarnings: ["warning"],
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
// ── C — Environment loading ─────────────────────────────────────
|
||||
@@ -295,6 +443,19 @@ describe("start-case-experiment-helper.cjs apparatus", () => {
|
||||
const parsed = parseOutput(result);
|
||||
expect(parsed.error).toBeNull();
|
||||
});
|
||||
|
||||
it("preserves the existing non-reconstruction-only output contract", async () => {
|
||||
const result = await runHelper(["Compatibility scenario"]);
|
||||
const parsed = parseOutput(result);
|
||||
|
||||
expect(parsed).not.toHaveProperty("experimentId");
|
||||
expect(parsed).not.toHaveProperty("scenarioId");
|
||||
expect(parsed).not.toHaveProperty("execution");
|
||||
expect(parsed).not.toHaveProperty("graphUnknownNodes");
|
||||
expect(parsed).toHaveProperty("situationGraphNodeCount");
|
||||
expect(parsed).toHaveProperty("situationGraphEdgeCount");
|
||||
expect(parsed).toHaveProperty("endToEndElapsedMs");
|
||||
});
|
||||
});
|
||||
|
||||
// ── E — Failure output (startCase throws) ───────────────────────
|
||||
@@ -399,6 +560,11 @@ describe("start-case-experiment-helper.cjs apparatus", () => {
|
||||
expect(typeof parsed.error).toBe("string");
|
||||
}
|
||||
});
|
||||
|
||||
it("contains no investigation-persistence invocation", async () => {
|
||||
const source = await readFile(helperPath, "utf-8");
|
||||
expect(source).not.toMatch(/investigation-storage|server-investigation-persistence|saveInvestigation|\/api\/investigations/);
|
||||
});
|
||||
});
|
||||
|
||||
// ── H — Import-only mode (production seam verification) ────────
|
||||
|
||||
@@ -211,6 +211,23 @@ describe("v0.5 prompt", () => {
|
||||
expect(result.prompt).toContain("Default prompt scenario");
|
||||
});
|
||||
|
||||
it("requires Open Questions to express the canonical uncertainty without taking over focused investigation", async () => {
|
||||
const { prompt } = await buildPrompt("Semantic equivalence scenario", "v0.5");
|
||||
|
||||
expect(prompt).toContain("canonical uncertainty meaning in `description`");
|
||||
expect(prompt).toContain("express that same uncertainty in `question`");
|
||||
expect(prompt).toContain("concise, natural, direct interrogative");
|
||||
expect(prompt).toContain("semantically equivalent");
|
||||
expect(prompt).toContain("subject, epistemic openness, material scope, timing, conditions, and qualifications");
|
||||
expect(prompt).toContain("without assuming the unresolved outcome has occurred or adding unsupported premises");
|
||||
expect(prompt).toContain("Do not narrow the uncertainty to one possible indicator or form of evidence");
|
||||
expect(prompt).toContain("evidence-gathering belongs to focused investigation");
|
||||
expect(prompt).toContain('"importantUnknowns"');
|
||||
expect(prompt).toContain('"description": "<the canonical uncertainty meaning>"');
|
||||
expect(prompt).toContain('"question": "<a natural direct question for the user>"');
|
||||
expect(prompt).toContain("Return ONLY the JSON object");
|
||||
});
|
||||
|
||||
it("keeps explicit v0.4 unchanged", async () => {
|
||||
const result = await buildPrompt("Explicit v0.4 scenario", "v0.4");
|
||||
|
||||
@@ -218,6 +235,8 @@ describe("v0.5 prompt", () => {
|
||||
expect(result.prompt).toContain(
|
||||
"with no dedicated schema field MUST be preserved explicitly in summary",
|
||||
);
|
||||
expect(result.prompt).not.toContain("semantically equivalent");
|
||||
expect(result.prompt).not.toContain("evidence-gathering belongs to focused investigation");
|
||||
});
|
||||
|
||||
it("loads the v0.5 relationship contract", async () => {
|
||||
|
||||
Reference in New Issue
Block a user