Compare commits

...
Author SHA1 Message Date
robbond 3366168e07 fix(confidence-engine): align explored question status language 2026-10-26 06:02:05 +00:00
robbond d8981c4e8e fix(confidence-engine): describe explored questions without implying resolution 2026-10-26 05:57:15 +00:00
robbond 8530d954a0 feat(helper): add --prompt-file option for experimental prompt variation
Add optional --prompt-file <path> argument to start-case-experiment-helper.cjs
that allows supplying an alternative reconstruction prompt file while
preserving the production JSON generation schema.

Requirements implemented:
- Reads UTF-8 prompt file (absolute path or relative to prompts/)
- Replaces {{SCENARIO}} placeholder with scenario text using existing convention
- Validates file exists and contains required placeholder
- Rejects --prompt-file with non-A conditions via clear error
- Only compatible with condition A (production structured-output contract)
- Zero production code changes; apparatus-only modification
- Preserves existing behavior for all conditions B-E without flag
- Does not activate schema variants or freeform handling

Verified: syntax check passes, no whitespace errors in diff.
2026-10-14 15:53:43 +01:00
robbond fe0f337a99 docs(experiments): checkpoint uncertainty recognition investigation 2026-10-14 09:39:11 +01:00
robbond e595ee6e02 feat(experiments): integrate Condition D/E apparatus 2026-10-13 16:24:00 +01:00
robbond 29cdc80f09 feat(experiments): integrate freeform analysis path 2026-10-13 14:36:11 +01:00
robbond 8dcdffce44 feat(experiments): support freeform provider responses 2026-10-13 11:59:47 +01:00
robbond 74936851ce feat(experiments): add atomic evidence persistence 2026-10-13 11:18:58 +01:00
robbond 5af3e62ddd feat(uncertainty-led-investigation): add Condition D freeform deconstruct treatment selector
- Add reconstruct-v0.5-d-deconstruct-no21.md prompt (removes rule 21)
  from original deconstruct-first prompt: 'Do not create artificial
  uncertainties to increase their number.'
- Extend start-case-experiment-helper.cjs with --condition D and
  --treatment no21 CLI flags for treatment selector apparatus
- Add programmatic treatment validation (rejects invalid treatments
  cross-condition, validates only 'no21' is a valid treatment value)
- Condition D output uses freeformText field instead of reconstruction
  JSON — avoids production JSON schema enforcement on freeform text
- Mock double supports freeform mode: returns deterministic raw text
  instead of structured importantUnknowns when isFreeform=true
- Add 23 targeted tests: J (Condition D prompt selection),
  J.1 (treatment validation), J.2 (prompt isolation verification)
- Fix existing B.1 assertions to use matchObject for execution field
2026-10-12 18:45:53 +01:00
robbond d5a06b9f28 docs(confidence-engine): record CE-ULI-11 domain-preservation intervention — explicit instruction did NOT prevent structural compression (Condition B=1 unknown, Condition C=1 unknown) 2026-10-12 09:41:43 +01:00
robbondandClaude Code 14ee66e82b docs(confidence-engine): record CE-ULI-10 paired observation — question-free reconstruction does not preserve broader uncertainty (single paired comparison, HS-03C)
Iteration 5 of bounded iteration loop. Two live calls per condition
using fixed HS-03C input (new-service-launch scenario). Model:
qwen-claude:latest via Ollama.

Condition A (question-required): 2 importantUnknowns with broad
descriptions + narrowing questions, 8 nodes, valid validation.

Condition B (question-free): 1 importantUnknown with structurally
narrowed combined-timeline description, 7 nodes, partial validation.

Neither condition preserved the original four-domain structure of
HS-03C. Narrowing occurred through different mechanisms in each
condition — hypothesis unconfirmed for this scenario shape.

Co-Authored-By: Claude Code <noreply@anthropic.com>
2026-10-12 07:54:35 +01:00
robbond 742d7c319b docs(confidence-engine): record CE-ULI-09 paired observation — question-free reconstruction valid through canonical path (single paired comparison) 2026-10-12 07:45:13 +01:00
robbondandClaude Code 3b04d05355 feat(confidence-engine): add CE-ULI-08 two-condition uncertainty comparison apparatus
Adds experimental apparatus for testing whether removing the question field
from importantUnknowns in reconstruction output improves semantic fidelity.

New files:
- lib/reconstruction/schema-variant.js (conditionB schema + production passthrough)
- prompts/reconstruct-v0.5-b-noreq.md (alt prompt, identical line count to prod)

Changed files:
- lib/graph/orchestrator.js (pass generationSchema through dependencies)
- lib/analysis.js (accept opts.generationSchema fallback to original schema)
- scripts/start-case-experiment-helper.cjs (--condition B/A flag support)
- tests/scripts/start-case-experiment-helper.test.js (8 new deterministic tests)

43 targeted deterministic tests PASS. Production callers use original prompt/schema unchanged.
No live model calls. No full suite evidence available.

Co-Authored-By: Claude Code <noreply@anthropic.com>
2026-10-12 07:33:31 +01:00
robbond a54b36a2b2 test(confidence-engine): extract focused target input for deterministic verification 2026-10-09 16:44:09 +01:00
robbondandClaude Code acf0665f23 docs(confidence-engine): record Experiment 9 evidence — explicit dependency preservation observed (PARTIAL fidelity)
HS-03C experiment with explicitly raised cross-area dependencies showed the model preserved relationships between recruitment, software integration and customer commitments while retaining the broader feasibility uncertainty. Result: PARTIAL semantic fidelity. Evidence at .evidence-temp/headless-semantic/HS-03C-focused-E9.json.

Co-Authored-By: Claude Code <noreply@anthropic.com>
2026-10-09 14:41:51 +01:00
robbond 0698a1a61e feat(confidence-engine): add saveFocusedInvestigationEvidence persistence operation (Experiment 8)
Add exported saveFocusedInvestigationEvidence(checkpoint, identity, metadata,
outputPath) to the canonical experiment helper. Validates output path, exactly
one valid deconstruction contribution, required identity fields, and exact input
values. Writes formatted JSON, reads back, verifies deep structural equality.
Rejects ambiguous checkpoints instead of silently selecting a contribution.

Deterministic test covers: positive file write, structured evidence completeness,
input fidelity, identity fidelity, return values, missing path rejection, missing
contribution rejection, ambiguous contribution rejection. All OS temp files cleaned
in finally blocks.

Zero live model calls. Zero browser calls. No production code modified.
2026-10-09 14:33:33 +01:00
robbondandClaude Code aa57d8589c feat(confidence-engine): verify disk persistence of focused-investigation evidence serializer (Experiment 7, Iteration 7)
Extended deterministic test to write serialized evidence record to
OS temp directory, read back from disk, parse JSON, and assert deep
structural equality. Verified: five structured arrays preserved with
correct lengths; exact inputs unchanged; identity fields intact; null
metadata fields survive round-trip. Temp file cleaned in finally block.

Zero live model calls. Zero browser calls. No production code or
canonical-helper modifications. Experiment 4 original structured
output remains unavailable.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Signed-off-by: robbond <rob@rdbsolutions.co.uk>
2026-10-09 14:29:36 +01:00
robbond 7d7a9e8d29 feat(confidence-engine): add serializeFocusedInvestigationEvidence evidence serializer and deterministic test
Apparatus-level pure serialization function that preserves all five
focused-deconstruction arrays (observations, uncertainties, assumptions,
relationships, possibleFollowUpQuestions) plus exact inputs and execution
metadata from the existing checkpoint — without modifying production code.

Deterministic verification: syntax check pass, 79/79 assertions pass
(including 28 new evidence-ser tests), zero live calls.

Experiment 4's missing structured output cannot be reconstructed (never
persisted by the apparatus). No new semantic evidence produced.
2026-10-09 14:25:26 +01:00
robbond 7fd05745e8 feat(confidence-engine): record Experiment 4 synthetic HS-03C focused investigation results — PARTIAL fidelity, semantic narrowing observed
Controlled synthetic experiment (Iteration 4): fixed recruitment scenario with
synthetic answer showed partial preservation of the broader feasibility uncertainty.
The model correctly preserved timing estimates as observations but narrowed the
investigation to hiring pipeline logistics rather than cross-area launch feasibility.
2026-10-09 14:12:35 +01:00
robbondandClaude Code d12615075d feat(confidence-engine): expose focused-investigation inputs in composed checkpoint
Add focusedInvestigationInput {targetDescription, question, answer} to
composeDeepInvestigationCheckpoint() return object. These are the actual
invocation values: targetDescription from situationGraph node, question
is previousQuestion (NOT node.openQuestion), answer is function param.

Deterministic verification: syntax check passed; source-inspection test
assertions E10/E11 added and pass (0 failures). Call-count ownership:
up to 3 live generateReconstruction calls (focusedDeconstruct +
reconsiderCompletedEpisode + synthesizeCurrentUnderstanding).

No provider, graph, or production behavior changed.

Co-Authored-By: Claude Code <noreply@anthropic.com>
2026-10-09 13:47:54 +01:00
robbond d0f357877d feat(confidence-engine): present canonical uncertainty description as initial display
Changed the facilitator-view-adapter to prefer node.description over
node.openQuestion for unresolved unknown items. This is a reversible
presentation change only — graph identity, reasoning-state ownership,
openQuestion generation, and resolved-unknown presentation are unchanged.

Owning test assertions updated to reflect new display contract:
- open-question-presentation.test.js: assertion target shifts from
  question text to description text
- facilitator-view-adapter.test.js: deduplication tests use distinct
  descriptions so nodes remain distinguishable after the change

Documentation: hypothesis experiment record and handoff return note.
2026-10-09 11:37:43 +01:00
robbond b71e3a8267 docs(confidence-engine): record uncertainty-led investigation hypothesis 2026-10-09 10:26:49 +01:00
robbond d89e0816b6 refactor(reconstruction): strengthen open-question semantic fidelity 2026-10-09 09:12:41 +01:00
robbond fd5241c6af feat(experiments): expose reconstruction open-question evidence 2026-10-09 08:12:06 +01:00
robbond 6bc8be6edb wip(confidence-engine): checkpoint open-question presentation work 2026-10-09 07:46:19 +01:00
robbond f74fe6de10 fix(focused-investigation): suppress incompatible question strategies 2026-10-08 17:48:53 +01:00
robbond 92ae463d10 feat(llm): propagate cancellation through reconstruction providers 2026-10-08 17:45:02 +01:00
robbondandClaude Code 35a6303ce8 test(confidence-engine): enable longitudinal synthesis checkpoint selection
Co-Authored-By: Claude Code <noreply@anthropic.com>
2026-10-08 10:49:51 +01:00
robbondandClaude Code 57c88b713a test(confidence-engine): establish longitudinal synthesis checkpoints
Co-Authored-By: Claude Code <noreply@anthropic.com>
2026-10-08 10:33:59 +01:00
robbondandClaude Code 8d9df8c939 test(confidence-engine): expose current understanding experiment output
Co-Authored-By: Claude Code <noreply@anthropic.com>
2026-10-08 10:16:34 +01:00
robbondandClaude Code ce7df91a42 test(confidence-engine): enable live current understanding experiment execution
Extend the CU synthesis experiment helper with a --live flag that
uses the production provider (getProvider from lib/llm/provider.js)
for live model invocation. Preserves deterministic mode unchanged.

Adds 10 new deterministic tests verifying live-mode wiring via
injected mock provider: fixture pass-through, single invocation,
schema correctness, immutability, persistence safety, and regression
of existing deterministic behaviour. Zero network calls in all tests.

Co-Authored-By: Claude Code <noreply@anthropic.com>
2026-10-08 09:59:03 +01:00
robbondandClaude Code 425e1e7805 test(confidence-engine): capture original current understanding regression fixture
Co-Authored-By: Claude Code <noreply@anthropic.com>
2026-10-08 09:52:57 +01:00
robbondandClaude Code dd875b5535 test(confidence-engine): add current understanding synthesis experiment helper
Co-Authored-By: Claude Code <noreply@anthropic.com>
2026-10-08 09:49:09 +01:00
robbondandClaude Code 279b9a2a67 fix(confidence-engine): permit evidenced knowledge gaps in current understanding
Co-Authored-By: Claude Code <noreply@anthropic.com>
2026-10-08 09:41:24 +01:00
robbond f1fae621f1 test(confidence-engine): remove Iteration 16 unsubstantiated target-scoped contribution behaviour section
Remove the 140-line Iteration 16 test section (commit 5d4e7b5) that
represented manually supplied checkpoint values as evidence of
target-scoped contribution behaviour. The section used extractLiveExperimentCheckpoint
to construct mock result objects simulating multi-episode state transitions
but could not observe contributions — the deterministic seam has no
contributions field in its return shape.

Preserves: all pre-existing checkpoint tests, Iteration 15 source-level
assertions, synthetic CU synthesis tests, and existing helper imports.
2026-10-08 07:41:40 +01:00
robbond 5d4e7b5e94 test(confidence-engine): verify target-scoped contribution behaviour via deterministic seam
Add Iteration 16 test extension that documents the behavioral evidence
boundary for target-scoped contribution filtering:

- Observable via extractLiveExperimentCheckpoint:
  * Node A pre-episode graph state (graphCheckpoint.nodeCount)
  * B's fresh deconstruction trigger (providerGenerateCalls increments)
  * Graph/findings continuity (reasoningState turnCount advanced)

- Blocked from deterministic seam (no contributions field in checkpoint):
  * Historical contribution retention for non-current targets
  * B contribution associated with targetNodeId
  * Both A and B surviving in postEpisodeContributions
  * B prep scope receiving B-specific contribution

Uses mock result objects to simulate the two-target sequence
(n_enterprise_customer_signing -> n_product_launch_decision) and
explicitly documents which assertions can and cannot be verified.
2026-10-08 07:39:22 +01:00
robbond 99f200303a fix(confidence-engine): scope contribution filtering to targetNodeId in composeDeepInvestigationCheckpoint
Historical contributions for a different Open Question node no longer
suppress fresh focused deconstruction of the current target. Only
contributions whose targetNodeId matches the current target are counted
towards suppressing redundant deconstruction.
2026-10-08 07:34:28 +01:00
robbond ddd1c7bd6f test(confidence-engine): cover CU diagnostics and investigation checkpoints 2026-10-08 06:45:37 +01:00
robbond 20b416dcb6 test(confidence-engine): report final deep provider count 2026-10-07 07:43:08 +01:00
robbond 44db12d9c0 test(confidence-engine): forward deep provider schemas 2026-10-07 07:25:52 +01:00
robbond 0b84f8ad28 test(confidence-engine): expose CU synthesis failure shape 2026-10-07 06:59:43 +01:00
robbond d6dbc17bdf fix(confidence-engine): await focused deconstruction provider 2026-10-06 18:18:10 +01:00
robbond 814360e368 test(confidence-engine): use canonical focused deconstruction 2026-10-06 17:28:40 +01:00
robbond 9362c78f81 refactor(confidence-engine): extract focused deconstruction seam 2026-10-06 17:06:44 +01:00
robbond 8e3f4388d6 test(confidence-engine): preserve focused investigation evidence seam 2026-10-05 09:35:04 +01:00
robbond e996a542b3 test(confidence-engine): align deep investigation episode seam 2026-10-05 08:06:28 +01:00
robbond ca284d9c04 test(confidence-engine): preserve deep investigation episode state 2026-10-05 07:54:17 +01:00
robbond 90649c0495 test(confidence-engine): compose deep CU checkpoints 2026-10-05 06:44:29 +01:00
robbond 72b10d0050 test(confidence-engine): capture deep-investigation checkpoints 2026-10-05 06:40:28 +01:00
robbond 97c7482177 docs(confidence-engine): checkpoint report learning semantics 2026-10-02 17:20:29 +01:00
robbondandClaude Code b8c2909b5f feat(confidence-engine): show investigation learning in report
Co-Authored-By: Claude Code <noreply@anthropic.com>
2026-10-02 08:26:07 +01:00
robbond b448f522f6 feat(confidence-engine): deriveInvestigationLearning deterministic learning projection from canonical provenance
Adds deriveInvestigationLearning to episode-preparation.js: pure function that maps resolvedNodeIds + Contributions + Findings to investigation learning propositions, preserving node.label question text and contribution-provenance ordering.

Owning tests prove: correct provenance, cross-question isolation, eligibility rule (null/agree eligible; not_quite/not_relevant excluded), unresolved exclusion, no-eligible-material omission, text preservation, and provenance authority over originatingTargetNodeId shortcut.
2026-10-02 08:18:19 +01:00
robbond 30648ca5b3 docs(confidence-engine): checkpoint plausible zero-state verification 2026-10-02 07:12:36 +01:00
robbond 0cc9d087c3 fix(confidence-engine): permit empty plausibleInterpretations for honest zero state
- Zod validator: plausibleInterpretations accepts empty string (zero interpretations) while remaining required and string-typed
- Prompt instruction: SECTION B now explicitly requests "" instead of absence prose when no plausible inputs exist
- Report generation/update paths derive hasPlausibleInterpretations from actual synthesized data rather than hard-coding true
- Tests: updated validation contract, added hasPlausibleInterpretations derivation assertions, updated seam tests to use derived values
2026-10-02 06:48:24 +01:00
robbond 8dc6e58d43 docs(confidence-engine): checkpoint verified three-part report 2026-10-01 17:13:10 +01:00
robbondandClaude Code 56b5f0099c feat(confidence-engine): show unresolved report questions
Co-Authored-By: Claude Code <noreply@anthropic.com>
2026-10-01 14:20:01 +01:00
robbond 7c1d657761 docs(confidence-engine): record product-launch epistemic A/B result — 4-iteration loop complete
Controlled A/B (zero-Findings scenario, Qwen): unresolved uncertainty exposure
did not contaminate supported understanding or plausible interpretation.
No production changes. No report semantic changes.
2026-09-30 17:22:52 +01:00
robbond cf09a72739 feat(confidence-engine): add --variant hidden|visible to overview experiment helper for controlled A/B
- Helper now supports --variant visible which appends unresolved Open Questions
  as a separate NON-EVIDENCE section to the baseline prompt (built via production
  buildOverviewSynthesisPrompt)
- Default (--variant hidden, or no flag) remains production path
- No production code changed; apparatus-only extension
- Deterministic A/B control proven by existing 34/34 test suite
2026-09-30 17:20:28 +01:00
robbond 9a56176323 fix(confidence-engine): supply overview-specific output schema to generateReconstruction
synthesizeInvestigationOverview() called provider.generateReconstruction(prompt, modelName)
without an output contract. Without a structured-output schema the provider defaults to its
initial-reconstruction format which lacks { understanding, plausibleInterpretations }, causing
'Missing or invalid required fields' validation failure.

Add overviewOutputSchema (JSON-schema matching validateOverviewResponse contract) and pass as
third argument, matching the proven pattern in synthesizeCurrentUnderstanding().

53/53 tests PASS. Co-Authored-By: Claude Code <noreply@anthropic.com>
2026-09-30 17:05:47 +01:00
robbond 6d9a7e7603 test(confidence-engine): add canonical overview experiment helper 2026-09-30 16:14:59 +01:00
robbond 3d0c90e64c fix(confidence-engine): unwrap overview provider response
Production: synthesizeInvestigationOverview() now passes providerResult.response
to validateOverviewResponse() instead of the raw envelope, matching the proven
pattern in current-understanding-synthesis.js.

Regression: fakeProvider returns real-provider envelope shape { response, providerApiPath }
and four new regression tests verify unwrapping succeeds with valid response
and fails correctly with invalid response inside the envelope.

Handoff: documented defect confirmed, fix applied, and owning regression added.
2026-09-30 16:06:14 +01:00
robbond 8d268efb12 test(confidence-engine): support zero-finding report experiment case
- Add nullish coalescing on fix.findings ?? [] so absent findings
  array in a fixture is represented as zero Findings [] for the A/B
  input apparatus.

- Append product-launch second controlled case (live-product-launch-
  start-response.json) with zero Findings to the epistemic comparison
  test suite. All 34 tests pass deterministically.

- Update docs/current-handoff.md to record the zero-Findings case.
2026-09-30 13:25:22 +01:00
robbondandClaude Code d387b66c03 test(confidence-engine): clarify controlled report experiment boundary
Retained deterministic input-control test proving that A/B apparatus
correctly holds supportedEvidence, findings, plausibleInterpretations,
scenarioState, and providerConfig constant while only varying unresolved
question visibility. Added buildOverviewSynthesisPrompt identity proof
that full vs filtered graph produce identical prompt text.

Corrected handoff semantics: deterministic input filtering does NOT prove
model output behaviour; live two-call semantic comparison remains required.
Preserved prior imperfect live evidence with correct qualification about
uncontrolled plausibleInterpretations divergence.

Co-Authored-By: Claude Code <noreply@anthropic.com>
2026-09-30 12:30:36 +01:00
robbondandClaude Code ada83de1db test(confidence-engine): control report epistemic comparison inputs
Deterministic test (17 cases) proves A/B inputs are correctly controlled:
- supportedEvidence identical in A and B
- findings identical in A and B
- plausibleInterpretations identical (fixed canonical interpretation) in A and B
- scenarioState identical in A and B
- providerConfig identical in A and B
- only unresolvedQuestions visibility differs (A=hidden, B=canonical)
- unresolved questions excluded from evidence and plausible interpretations
- fixed interpretation carries NON-EVIDENCE epistemic status

Also corrects Loop 2 documentation to reflect that three-category
invariance was NOT established (plausibleInterpretations differed
between calls despite identical input), retaining only the narrower
valid finding: no unresolved Open Question was promoted to evidence.

Co-Authored-By: Claude Code <noreply@anthropic.com>
2026-09-30 11:44:51 +01:00
robbondandClaude Code ef6592cd91 test(confidence-engine): add progressed investigation experiment fixture
Co-Authored-By: Claude Code <noreply@anthropic.com>
2026-09-30 10:55:13 +01:00
robbond 2dc1cb0fe6 docs(confidence-engine): record tester readiness findings 2026-09-11 10:34:53 +01:00
robbond 2349ef9d59 docs(confidence-engine): close pre-tester security review 2026-09-10 14:21:10 +01:00
robbond a6796c6f73 fix(confidence-engine): bound focused investigation input 2026-09-10 13:23:39 +01:00
robbond 2cb2d556fd fix(confidence-engine): sanitize reasoning error responses 2026-09-10 11:39:13 +01:00
robbond b9d49ff277 fix(confidence-engine): correct legal footer layout 2026-09-10 06:40:30 +01:00
robbond 27ded883e5 feat(confidence-engine): add tester legal pages 2026-09-10 06:18:28 +01:00
82 changed files with 11456 additions and 451 deletions
+19 -3
View File
@@ -26,7 +26,7 @@ async function post(request) {
if (!result.success) {
return Response.json(
{ ...result, reconstruction: result.reconstruction || null },
{ error: "Reasoning request could not be completed." },
{ status: Number(result.statusCode) || 500 },
);
}
@@ -42,9 +42,25 @@ async function post(request) {
promptVersion: result.promptVersion,
});
} catch (e) {
if (e.code === "PROVIDER_UNAVAILABLE") {
return Response.json(
{ error: "Reasoning service is temporarily unavailable." },
{ status: 503 },
);
}
const validationFailed = e?.validationFailed || e?.code === "VALIDATION_FAILED";
if (validationFailed) {
return Response.json(
{ success: false, error: "Reasoning request could not be completed." },
{ status: Number(e.statusCode) || 500 },
);
}
const statusCode = Number(e.statusCode) || 500;
return Response.json(
{ error: e.message || "Unknown server error", responseDurationMs: 0 },
{ status: 500 },
{ error: "Reasoning request could not be completed." },
{ status: statusCode },
);
}
}
+3 -1
View File
@@ -55,7 +55,9 @@ async function post(request) {
{
success: false,
stage: error.statusCode === 400 ? "request_validation" : "provider",
error: error.message ?? "Overview synthesis failed",
error: error.statusCode === 400
? "Invalid overview request"
: "Reasoning request could not be completed.",
},
{ status: error.statusCode }
);
+2 -8
View File
@@ -48,14 +48,8 @@ async function post(request) {
return Response.json(
{
success: false,
error: diagnostics.error,
validationErrors: result.validationErrors,
diagnostics: result.diagnostics,
analysisErrors: result.analysisErrors,
validationIssues: result.validationIssues,
providerApiPath: result.providerApiPath,
providerExecution: result.providerExecution,
rawResponse: result.rawResponse ?? undefined,
error: status === 400 ? diagnostics.error : "Reasoning request could not be completed.",
...(status === 400 ? { validationErrors: result.validationErrors } : {}),
},
{ status },
);
+3 -1
View File
@@ -57,7 +57,9 @@ async function post(request) {
{
success: false,
stage: error.statusCode === 400 ? "request_validation" : "provider",
error: error.message ?? "Synthesis failed",
error: error.statusCode === 400
? "Invalid synthesis request"
: "Reasoning request could not be completed.",
},
{ status: error.statusCode }
);
@@ -1,9 +1,5 @@
import { getProvider, getProviderModelName } from "@/lib/llm/provider";
import {
buildFocusedDeconstructPrompt,
focusedDeconstructJsonSchema,
validateFocusedDeconstructSchema,
} from "@/lib/graph/focused-investigation";
import { focusedDeconstruct } from "@/lib/graph/focused-investigation";
import { withAuthenticatedApi } from "@/lib/supabase/api-auth.js";
async function post(request) {
@@ -50,13 +46,22 @@ async function post(request) {
);
}
const prompt = buildFocusedDeconstructPrompt({
targetLabel: body.targetLabel,
targetDescription: body.targetDescription,
centralStatement: body.centralStatement,
question: body.question,
answer: body.answer,
});
const REQUEST_LENGTH_LIMITS = {
answer: 10000,
question: 2048,
centralStatement: 2048,
targetLabel: 2048,
targetDescription: 2048,
};
for (const [field, limit] of Object.entries(REQUEST_LENGTH_LIMITS)) {
if (body[field] && body[field].length > limit) {
return Response.json(
{ error: `Request field "${field}" exceeds maximum length of ${limit} characters` },
{ status: 400 },
);
}
}
const provider = getProvider();
const modelName = getProviderModelName();
@@ -67,36 +72,21 @@ async function post(request) {
providerMode: process.env.CONFIDENCE_ENGINE_EXPERIMENT_PROVIDER ?? "ollama",
startedAt,
});
const wrapper = await provider.generateReconstruction(
prompt,
const result = await focusedDeconstruct({
provider,
modelName,
focusedDeconstructJsonSchema,
);
targetNodeId,
targetLabel: body.targetLabel,
targetDescription: body.targetDescription,
centralStatement: body.centralStatement,
question: body.question,
answer: body.answer,
});
const elapsedMs = Date.now() - startedAt;
// Unwrap the semantic deconstruction from the provider envelope.
const deconstruction = wrapper.response;
console.info("[api/focused-investigation/deconstruct] provider success", {
targetNodeId,
elapsedMs,
providerApiPath: wrapper.providerApiPath ?? null,
responsePresent: Boolean(deconstruction),
responseKeys: deconstruction && typeof deconstruction === "object"
? Object.keys(deconstruction)
: [],
});
// Validate schema (required fields present, no graph-mutation fields)
const validationErrors = validateFocusedDeconstructSchema(deconstruction);
console.info("[api/focused-investigation/deconstruct] validation", {
targetNodeId,
schemaValid: validationErrors.length === 0,
responseKeys: deconstruction && typeof deconstruction === "object"
? Object.keys(deconstruction)
: [],
validationErrors,
});
if (validationErrors.length > 0) {
if (!result.success) {
console.info("[api/focused-investigation/deconstruct] end", {
targetNodeId,
status: 502,
@@ -106,7 +96,7 @@ async function post(request) {
{
success: false,
error: "Focused deconstruction result did not match expected schema",
validationErrors,
validationErrors: result.errors,
targetNodeId: body.targetNodeId,
elapsedMs,
},
@@ -117,11 +107,11 @@ async function post(request) {
const response = Response.json({
success: true,
targetNodeId: body.targetNodeId,
observations: deconstruction.observations,
uncertainties: deconstruction.uncertainties,
assumptions: deconstruction.assumptions,
relationships: deconstruction.relationships,
possibleFollowUpQuestions: deconstruction.possibleFollowUpQuestions,
observations: result.observations,
uncertainties: result.uncertainties,
assumptions: result.assumptions,
relationships: result.relationships,
possibleFollowUpQuestions: result.possibleFollowUpQuestions,
elapsedMs,
});
console.info("[api/focused-investigation/deconstruct] end", {
@@ -154,7 +144,7 @@ async function post(request) {
);
}
return Response.json(
{ error: e.message || "Unknown server error" },
{ error: "Reasoning request could not be completed." },
{ status: 500 },
);
}
@@ -1,6 +1,13 @@
import { formulateQuestionForTarget } from "@/lib/graph/focused-investigation";
import { getProvider, getProviderModelName } from "@/lib/llm/provider";
import {
assessStrategyCompatibility,
formulateQuestionForTarget,
isStrategyQuestionTemplateCompatible,
} from "@/lib/graph/focused-investigation";
import { withAuthenticatedApi } from "@/lib/supabase/api-auth.js";
const STRATEGY_ASSESSMENT_TIMEOUT_MS = 10000;
async function post(request) {
try {
const body = await request.json();
@@ -19,18 +26,59 @@ async function post(request) {
);
}
const result = formulateQuestionForTarget({
const proposed = formulateQuestionForTarget({
situationGraph: body.situationGraph,
targetNodeId: body.targetNodeId,
});
if (!result.success) {
if (!proposed.success) {
return Response.json(
{ success: false, error: result.error },
{ success: false, error: proposed.error },
{ status: 400 },
);
}
let result = proposed;
if (proposed.investigationStrategy) {
const controller = new AbortController();
const timeout = setTimeout(
() => controller.abort(),
STRATEGY_ASSESSMENT_TIMEOUT_MS,
);
let strategyCompatibilityVerdict = {
strategyKey: proposed.investigationStrategy.key,
relevance: "unavailable",
};
try {
const assessment = await assessStrategyCompatibility({
provider: getProvider(),
modelName: getProviderModelName(),
targetLabel: body.situationGraph.nodes.find((node) => node.id === body.targetNodeId)?.label,
targetDescription: body.situationGraph.nodes.find((node) => node.id === body.targetNodeId)?.description,
centralStatement: body.situationGraph.centralStatement,
proposedStrategy: proposed.investigationStrategy,
signal: controller.signal,
});
if (assessment.success) strategyCompatibilityVerdict = assessment.verdict;
} catch {
// A timed-out or unavailable assessment is explicitly non-authorizing.
} finally {
clearTimeout(timeout);
}
result = formulateQuestionForTarget({
situationGraph: body.situationGraph,
targetNodeId: body.targetNodeId,
strategyCompatibilityVerdict,
strategyTemplateCompatibility:
strategyCompatibilityVerdict.relevance === "compatible" &&
!isStrategyQuestionTemplateCompatible(proposed.investigationStrategy)
? "incompatible"
: undefined,
});
}
return Response.json({
success: true,
targetNodeId: result.targetNodeId,
+35
View File
@@ -0,0 +1,35 @@
import React from "react";
import LegalPageLayout from "@/components/legal-page-layout";
export default function CookiesPage() {
return (
<LegalPageLayout title="Cookie Policy">
<p>This policy explains the browser storage Confidence Engine currently uses. LocalStorage and sessionStorage are browser storage technologies; they are not necessarily HTTP cookies.</p>
<section>
<h2 className="text-xl font-semibold text-gray-900">Authentication and session cookies</h2>
<p>Confidence Engine uses Supabase authentication and session cookies to provide secure sign-in and keep authenticated users signed in. These are essential to the authenticated service. Runtime cookie names and durations are managed by the authentication system.</p>
</section>
<section>
<h2 className="text-xl font-semibold text-gray-900">Theme preference</h2>
<p><strong>confidence-engine-theme</strong> is stored in localStorage to remember your light or dark display preference. It remains until you change the preference or clear browser storage.</p>
</section>
<section>
<h2 className="text-xl font-semibold text-gray-900">Introductory guidance preference</h2>
<p><strong>ce-facilitator-dismissed</strong> is stored in sessionStorage if you dismiss introductory guidance. It is session-scoped and remembers that choice while the browser session remains available.</p>
</section>
<section>
<h2 className="text-xl font-semibold text-gray-900">No analytics or advertising tracking</h2>
<p>Confidence Engine currently does not use advertising cookies, analytics cookies, tracking pixels, marketing cookies, or third-party browser tracking based on the current implementation.</p>
</section>
<section>
<h2 className="text-xl font-semibold text-gray-900">Future changes</h2>
<p>If Confidence Engine later introduces non-essential cookies or similar technologies, this policy and any consent mechanism will be reconsidered as appropriate.</p>
</section>
</LegalPageLayout>
);
}
+49 -2
View File
@@ -2,6 +2,7 @@
import React, { useEffect, useRef, useState } from "react";
import { loadInvestigation, saveInvestigation } from "@/lib/storage/investigation-storage";
import { deriveInvestigationLearning } from "@/lib/graph/episode-preparation";
import Link from "next/link";
export default function ReportPage({ params }) {
@@ -63,7 +64,7 @@ export default function ReportPage({ params }) {
if (data.success) {
/* ── v0.59a — provenance: record generation revision (does NOT change Investigation revision) ── */
const rev = existing?.investigationRevision ?? 0;
const reportData = { understanding: data.understanding, plausibleInterpretations: data.plausibleInterpretations, hasPlausibleInterpretations: true, generatedFromRevision: rev };
const reportData = { understanding: data.understanding, plausibleInterpretations: data.plausibleInterpretations, hasPlausibleInterpretations: (data.plausibleInterpretations ?? "").trim().length > 0, generatedFromRevision: rev };
setExisting((p) => {
void saveInvestigation({ ...p, investigationReport: reportData })
.catch((error) => console.error("Investigation report save failed", error));
@@ -109,7 +110,7 @@ export default function ReportPage({ params }) {
const data = await res.json();
if (data.success) {
const reportData = { understanding: data.understanding, plausibleInterpretations: data.plausibleInterpretations, hasPlausibleInterpretations: true, generatedFromRevision: rev };
const reportData = { understanding: data.understanding, plausibleInterpretations: data.plausibleInterpretations, hasPlausibleInterpretations: (data.plausibleInterpretations ?? "").trim().length > 0, generatedFromRevision: rev };
setExisting((p) => {
void saveInvestigation({ ...p, investigationReport: reportData })
.catch((error) => console.error("Investigation report save failed", error));
@@ -191,6 +192,32 @@ export default function ReportPage({ params }) {
</div>
)}
{/* What the investigation surfaced — deterministic learning projection */}
{(() => {
const learning = deriveInvestigationLearning({
situationGraph: existing?.situationGraph,
contributions: existing?.focusedContributions ?? [],
findings: existing?.findings ?? [],
});
return learning.length > 0 ? (
<div className="mt-6 rounded-xl border-[2.5px] border-teal-300/70 bg-gradient-to-b from-teal-50/60 to-white px-8 pt-6 pb-7 shadow-sm">
<h2 className="mb-3 text-[11px] font-bold tracking-[.18em] uppercase text-teal-700/70">
What the investigation surfaced
</h2>
{learning.map((group, gi) => (
<div key={gi} className="mt-4 space-y-2">
<p className="text-base font-medium text-gray-800">{group.questionText}</p>
{group.propositions.map((prop, pi) => (
<p key={pi} className="text-base leading-relaxed text-gray-700 italic">
{prop.proposition}
</p>
))}
</div>
))}
</div>
) : null;
})()}
{/* What remains plausible — conditional */}
{report.hasPlausibleInterpretations && report.plausibleInterpretations ? (
<div className="mt-6 rounded-xl border-[2.5px] border-blue-300/70 bg-gradient-to-b from-blue-50/60 to-white px-8 pt-6 pb-7 shadow-sm">
@@ -202,6 +229,26 @@ export default function ReportPage({ params }) {
</p>
</div>
) : null}
{/* What remains unresolved — derived from canonical graph state */}
{(() => {
const resolvedIds = new Set(existing?.resolvedNodeIds ?? []);
const unresolvedQuestions = (existing?.situationGraph?.nodes ?? [])
.filter((n) => n.kind === "unknown" && n.status !== "resolved" && !resolvedIds.has(n.id))
.map((n) => n.label || n.description);
return unresolvedQuestions.length > 0 ? (
<div className="mt-6 rounded-xl border-[2.5px] border-amber-300/70 bg-gradient-to-b from-amber-50/60 to-white px-8 pt-6 pb-7 shadow-sm">
<h2 className="mb-3 text-[11px] font-bold tracking-[.18em] uppercase text-amber-700/70">
What remains unresolved
</h2>
<ul className="list-disc space-y-2 pl-6 text-base leading-relaxed text-gray-800">
{unresolvedQuestions.map((q, i) => (
<li key={i}>{q}</li>
))}
</ul>
</div>
) : null;
})()}
</>
) : (
/* Skeleton / loading state when no persisted report exists */
+7 -2
View File
@@ -1,6 +1,8 @@
import React from "react";
import "./globals.css";
import ThemeToggle from "@/components/theme-toggle";
import LogoutButton from "@/components/logout-button";
import LegalNavigation from "@/components/legal-navigation";
export const metadata = {
title: "Confidence Engine",
@@ -10,7 +12,7 @@ export const metadata = {
export default function RootLayout({ children }) {
return (
<html lang="en">
<body className="min-h-screen bg-gray-50 text-gray-900">
<body className="flex min-h-screen flex-col bg-gray-50 text-gray-900">
<script
dangerouslySetInnerHTML={{
__html: `try { const saved = localStorage.getItem('confidence-engine-theme'); const theme = saved === 'dark' || saved === 'light' ? saved : (matchMedia('(prefers-color-scheme: dark)').matches ? 'dark' : 'light'); document.documentElement.dataset.theme = theme; document.documentElement.style.colorScheme = theme; } catch (_) {}`,
@@ -25,7 +27,10 @@ export default function RootLayout({ children }) {
</div>
</div>
</header>
{children}
<div className="flex-1">{children}</div>
<footer className="app-chrome border-t border-gray-200/80 px-6 py-5">
<LegalNavigation />
</footer>
</body>
</html>
);
+1 -1
View File
@@ -2,7 +2,7 @@ import LoginForm from "@/components/login-form";
export default function LoginPage() {
return (
<main className="mx-auto min-h-[calc(100vh-57px)] max-w-[1200px] px-6 py-16">
<main className="mx-auto max-w-[1200px] px-6 py-16">
<div className="mx-auto flex flex-col gap-y-8 md:grid md:grid-cols-[minmax(0,1fr)_420px] md:gap-x-8 md:items-start">
{/* Left column wrapper — contents on mobile for flex ordering, block on desktop as one grid cell */}
<div className="contents md:block">
+47
View File
@@ -0,0 +1,47 @@
import React from "react";
import LegalPageLayout from "@/components/legal-page-layout";
export default function PrivacyPage() {
return (
<LegalPageLayout title="Privacy Policy">
<p>This policy explains how Confidence Engine handles information when you use the service.</p>
<section>
<h2 className="text-xl font-semibold text-gray-900">Who is responsible</h2>
<p>Confidence Engine is operated by RDB Solutions Ltd., Palmeira Avenue Mansions, 19 Church Road, Hove, East Sussex, England, BN3 2FA. For privacy questions or deletion requests, contact <a className="text-teal-700 underline" href="mailto:data@rdbtech.co.uk">data@rdbtech.co.uk</a>.</p>
</section>
<section>
<h2 className="text-xl font-semibold text-gray-900">Information we process</h2>
<p>We process your account email and authentication information, together with information you choose to enter into an investigation. This can include scenario descriptions, reconstructed SituationGraph material, Current Understanding, Open Questions, answers, Findings, Contributions, reports, and revision and timestamp metadata.</p>
<p>Please do not enter personal information that you do not need to provide, especially unnecessary information about other people.</p>
</section>
<section>
<h2 className="text-xl font-semibold text-gray-900">Why we use it</h2>
<p>We use account information to provide secure access to your account. We use investigation information to save your work, let you return to it, and provide the reasoning features you request. Confidence Engine supports your understanding; it does not make decisions for you.</p>
</section>
<section>
<h2 className="text-xl font-semibold text-gray-900">Storage and reasoning</h2>
<p>Authentication and investigation persistence are operated through our self-hosted Supabase and PostgreSQL infrastructure. Reasoning requests are processed through private Ollama/Qwen infrastructure used by the service. We also use functional browser storage for authentication sessions and interface preferences; see our Cookie Policy for details.</p>
</section>
<section>
<h2 className="text-xl font-semibold text-gray-900">Retention and deletion</h2>
<p>Account and investigation information is retained while your account remains active. After 18 months without activity, we will contact you before deletion; if inactivity continues, account and investigation information may then be automatically deleted. You may request deletion at <a className="text-teal-700 underline" href="mailto:data@rdbtech.co.uk">data@rdbtech.co.uk</a>.</p>
<p>Deleted information may remain temporarily in rotating backups until those backups expire.</p>
</section>
<section>
<h2 className="text-xl font-semibold text-gray-900">Security and your rights</h2>
<p>We use access controls and technical measures appropriate to operating the service. You can contact us about access, correction, deletion, or other data-protection requests. You may also complain to the UK Information Commissioner&apos;s Office.</p>
</section>
<section>
<h2 className="text-xl font-semibold text-gray-900">Age and changes</h2>
<p>Confidence Engine is for people aged 18 and over. We may update this policy as the service develops; the current version will be published on this page.</p>
</section>
</LegalPageLayout>
);
}
+42
View File
@@ -0,0 +1,42 @@
import React from "react";
import LegalPageLayout from "@/components/legal-page-layout";
export default function TermsPage() {
return (
<LegalPageLayout title="Terms of Use">
<p>Confidence Engine is operated by RDB Solutions Ltd. These Terms govern your use of the service.</p>
<section>
<h2 className="text-xl font-semibold text-gray-900">Eligibility and accounts</h2>
<p>You must be at least 18 years old to use Confidence Engine. Keep access to your email account and sign-in link secure, and provide accurate information when creating or using an account.</p>
</section>
<section>
<h2 className="text-xl font-semibold text-gray-900">Using the service</h2>
<p>Use the service lawfully and responsibly. You are responsible for the information you enter and should avoid entering information about others unless it is necessary and appropriate to do so.</p>
<p>Confidence Engine facilitates understanding; it does not make decisions for you. AI or model-generated analysis may be incomplete, inaccurate, or unsuitable for your circumstances. It is not professional, legal, financial, medical, or other regulated advice. You remain responsible for your decisions and actions.</p>
</section>
<section>
<h2 className="text-xl font-semibold text-gray-900">Availability</h2>
<p>We aim to keep the service available, but availability may vary. Reasoning functionality may occasionally be temporarily unavailable, and we may change, suspend, or withdraw parts of the service when reasonably necessary.</p>
</section>
<section>
<h2 className="text-xl font-semibold text-gray-900">Your information and intellectual property</h2>
<p>You retain responsibility for your underlying scenarios and information. We do not claim ownership of that underlying material merely because you use the service. The Confidence Engine service, branding, and software remain the property of RDB Solutions Ltd. or its licensors.</p>
</section>
<section>
<h2 className="text-xl font-semibold text-gray-900">Ending access and liability</h2>
<p>You may stop using the service and request account deletion at <a className="text-teal-700 underline" href="mailto:data@rdbtech.co.uk">data@rdbtech.co.uk</a>. We may suspend or end access where reasonably necessary, including for misuse or security reasons.</p>
<p>Nothing in these Terms excludes liability that cannot legally be excluded. Subject to that, the service is provided for a controlled early release and we are not liable for indirect loss or for decisions you make using it.</p>
</section>
<section>
<h2 className="text-xl font-semibold text-gray-900">Changes and law</h2>
<p>We may update these Terms by publishing the revised version on this page. These Terms are governed by the law of England and Wales. Questions can be sent to <a className="text-teal-700 underline" href="mailto:data@rdbtech.co.uk">data@rdbtech.co.uk</a>.</p>
</section>
</LegalPageLayout>
);
}
+12
View File
@@ -0,0 +1,12 @@
import React from "react";
import Link from "next/link";
export default function LegalNavigation() {
return (
<nav aria-label="Legal information" className="flex flex-wrap justify-center gap-x-5 gap-y-2 text-sm text-gray-500">
<Link href="/privacy" className="hover:text-teal-700">Privacy</Link>
<Link href="/terms" className="hover:text-teal-700">Terms</Link>
<Link href="/cookies" className="hover:text-teal-700">Cookies</Link>
</nav>
);
}
+18
View File
@@ -0,0 +1,18 @@
import React from "react";
import Link from "next/link";
export default function LegalPageLayout({ title, children }) {
return (
<main className="mx-auto max-w-3xl px-6 py-12 sm:py-16">
<Link href="/login" className="text-sm font-medium text-teal-700 hover:underline">
Back to sign in
</Link>
<article className="mt-6 rounded-xl border-[2.5px] border-teal-300/70 bg-gradient-to-b from-teal-50/60 to-white px-6 py-8 shadow-sm sm:px-10">
<h1 className="text-3xl font-bold tracking-tight text-teal-700">{title}</h1>
<div className="mt-8 space-y-7 text-base leading-relaxed text-gray-700">
{children}
</div>
</article>
</main>
);
}
+14 -7
View File
@@ -30,6 +30,9 @@ function isTechnicalSummary(summary) {
return false;
}
// ── Focused answer contract ──────────────────────────────────
const FOCUSED_ANSWER_MAX_LENGTH = 10000;
// ── Recovery state components (Phase 2) ───────────────────────
function ProviderUnavailableCard({ onRestart }) {
@@ -238,8 +241,11 @@ function FocusedQuestionBody({
{focused?.question?.trim() && processingStep !== "active" && focused.status === "formulated" && !hasAnswer && !hasActiveFollowUp && (
<div data-testid="completed-narrative">
<label htmlFor={`rw-answer-${nodeId}`} className="mb-2 block text-sm font-medium text-gray-700">Your response</label>
<textarea id={`rw-answer-${nodeId}`} value={focusedAnswer} onChange={(e) => setFocusedAnswer(e.target.value)} rows={4} data-testid="response-textarea" className="w-full rounded-lg border border-gray-300 px-4 py-3 text-sm focus:border-gray-500 focus:outline-none focus:ring-2 focus:ring-gray-400 disabled:cursor-not-allowed disabled:opacity-60" placeholder="What do you know about this?" />
<button onClick={(e) => { e.stopPropagation(); handleDeconstructSubmit(nodeId, focusedAnswer); }} disabled={!focusedAnswer.trim() || processingStep === "active"} style={{ cursor: !focusedAnswer.trim() || processingStep === "active" ? "not-allowed" : "pointer" }} className="mt-3 rounded-lg border border-green-600 bg-white px-4 py-2 text-sm font-medium text-green-700 hover:bg-green-50 transition disabled:opacity-50">Submit response</button>
<textarea id={`rw-answer-${nodeId}`} value={focusedAnswer} onChange={(e) => setFocusedAnswer(e.target.value)} rows={4} maxLength={FOCUSED_ANSWER_MAX_LENGTH} data-testid="response-textarea" className="w-full rounded-lg border border-gray-300 px-4 py-3 text-sm focus:border-gray-500 focus:outline-none focus:ring-2 focus:ring-gray-400 disabled:cursor-not-allowed disabled:opacity-60" placeholder="What do you know about this?" />
<div className="flex items-center justify-between mt-2">
<span className="text-xs text-gray-400">{focusedAnswer.length}/{FOCUSED_ANSWER_MAX_LENGTH}</span>
<button onClick={(e) => { e.stopPropagation(); handleDeconstructSubmit(nodeId, focusedAnswer); }} disabled={!focusedAnswer.trim() || processingStep === "active"} style={{ cursor: !focusedAnswer.trim() || processingStep === "active" ? "not-allowed" : "pointer" }} className="rounded-lg border border-green-600 bg-white px-4 py-2 text-sm font-medium text-green-700 hover:bg-green-50 transition disabled:opacity-50">Submit response</button>
</div>
</div>
)}
@@ -552,7 +558,7 @@ function hasGenuineCompletion(graph) {
function CompletionCard({ summary }) {
return (
<div className="rounded-lg border border-green-300 bg-green-50 px-5 py-6 text-center">
<h2 className="mb-1 text-sm font-bold uppercase tracking-wide text-green-700">Investigation complete</h2>
<h2 className="mb-1 text-sm font-bold uppercase tracking-wide text-green-700">All questions explored</h2>
<p className="text-base text-gray-800 mb-3">The available evidence supports the following understanding.</p>
{summary && (
<div className="mt-4 text-left rounded-md bg-white/60 px-4 py-3 border border-green-100">
@@ -1884,6 +1890,7 @@ export default function ReasoningWorkspace({
focusedInvestigations[node.id] &&
focusedInvestigations[node.id].question
);
const openQuestion = node.openQuestion || node.label;
function handleNodeClick(n) {
if (hasCompleted && selectedPresentationItemId === n.id) {
@@ -1901,8 +1908,8 @@ export default function ReasoningWorkspace({
style={{ cursor: "pointer" }}
className="w-full text-left rounded-lg border border-gray-200 bg-white px-5 py-4 transition hover:border-gray-300 hover:bg-gray-50"
>
<span className={`block leading-snug ${isFocused ? "text-sm font-medium text-gray-900" : "text-sm text-gray-600"}`}>{node.label}</span>
{node.description && node.description !== node.label && (
<span className={`block leading-snug ${isFocused ? "text-sm font-medium text-gray-900" : "text-sm text-gray-600"}`}>{openQuestion}</span>
{node.description && node.description !== openQuestion && (
<p className="mt-1.5 text-xs leading-snug text-gray-500">{node.description}</p>
)}
{!isFocused && <span className="mt-2 block text-[10px] uppercase tracking-wider text-gray-400">Unclear</span>}
@@ -1995,7 +2002,7 @@ export default function ReasoningWorkspace({
{clarifiedQuestions.length > 0 && (
<div className="space-y-3">
<h2 className="text-[11px] font-semibold tracking-widest uppercase text-gray-500">
Questions we have clarified
Questions we've explored
</h2>
{clarifiedQuestions.map((node) => (
<div key={node.id} className="space-y-1">
@@ -2007,7 +2014,7 @@ export default function ReasoningWorkspace({
<p className="mt-1.5 text-xs leading-snug text-gray-500">{node.description}</p>
)}
<div className="mt-2 flex items-center gap-3">
<span className="text-[10px] uppercase tracking-wider text-green-600">Clarified</span>
<span className="text-[10px] uppercase tracking-wider text-green-600">Explored</span>
<button
onClick={() => {
const nextGraph = reopenResolvedUnknown(graph, node.id);
+1
View File
@@ -1,5 +1,6 @@
"use client";
import React from "react";
import { useEffect, useState } from "react";
import {
readThemePreference,
+706 -1
View File
@@ -3,12 +3,26 @@
> **Role:** Concise operational snapshot for resuming work today. Not a historical diary.
> The design evolution archive index at `docs/design-evolution/README.md` provides progressive loading of experiment history; load the relevant chapter only when a specific historical question requires it.
## Tester legal information
- Public Privacy, Terms, and Cookie pages and shared legal navigation are available at `/privacy`, `/terms`, and `/cookies`.
- The service is positioned as 18+. No cookie-consent banner is used because current browser storage is limited to essential authentication/session storage and functional UI preferences.
- Legal wording remains subject to appropriate professional review.
- Legal navigation is a single global normal-flow footer: it rests at the viewport bottom on short pages, long legal pages push it naturally below their content, and duplicate legal navigation was removed.
## Focused-investigation provider outage boundary
- Focused-investigation outage handling now sanitizes provider failure at the API boundary: unavailable focused reasoning returns a controlled HTTP 503, and raw provider/Ollama diagnostics no longer leave that boundary.
- The existing generic Retry UX remains unchanged. A live deployed outage had already proven investigation preservation.
- `/api/cases/start` and `/api/cases/update` outage sanitization remain separately unverified; this change does not claim those routes are fixed.
## Reasoning API browser error boundary hardened
- All reasoning HTTP error responses now sanitize raw provider diagnostics: `e.message` / raw internals never leak to the client.
- Verified on: `/api/cases/start`, `/api/focused-investigation/deconstruct`, `/api/cases/overview`, `/api/cases/synthesis`, `/api/analyse`.
- Provider/internal diagnostics remain server-side only.
- No reasoning semantics changed.
## v0.62d production Docker packaging — LIVE PROVEN
- `Dockerfile` — minimal multi-stage Alpine build (Node 22), Next.js standalone output mode
@@ -111,6 +125,60 @@ Jenkins SCM branch used to load the Jenkinsfile is conceptually separate from th
- CSS Grid with responsive column placement replaces original flexbox; three DOM sections ensure correct mobile stacking without duplicating content
- desktop two-column presentation (explanation left / sign-in right) preserved unchanged at `md` breakpoint and above
## Focused-investigation input hardening
- Focused answer now has a visible 10,000-character UI limit (maxLength + character counter).
- Server enforces matching 10,000-character bound on `/api/focused-investigation/deconstruct`.
- Other prompt-bearing fields retain explicit defensive bounds (2048 characters each).
- Oversized/malformed requests are rejected before reasoning with controlled HTTP 400.
- No punctuation/HTML/SQL-style content stripping introduced.
- Reasoning semantics unchanged.
## Pre-Tester Input Security Review
**Status:** closed — sufficient for controlled external-user testing. Not a general security audit, penetration test, or production-launch certification.
### SQL injection
- No raw request-driven SQL construction found.
- Persistence uses Supabase/PostgREST query-builder boundaries.
- SQL-character stripping / sanitisation is not warranted.
### XSS
- Normal user-controlled text is React-escaped.
- No user-controlled unsafe HTML sink found.
- HTML / script stripping of prose is not warranted.
### Reasoning error disclosure
- Browser-facing reasoning errors have been sanitised (commit `2cb2d55`).
### Focused user input
- Bounded to 10,000 characters server-side on `/api/focused-investigation/deconstruct`.
- UI exposes matching `maxLength` and character counter (commit `a6796c6`).
### Investigation snapshot size
- Authenticated persistence envelope currently lacks a whole-snapshot size ceiling.
- Not classified as an outstanding pre-tester blocker.
- Legitimate investigation size / turn depth is not yet known.
- No arbitrary product ceiling imposed before real-user evidence exists.
- Monitor snapshot growth later via metadata (serialized bytes / revision / contribution count) without logging investigation content.
- Malformed-envelope / runtime validation remains a separate future hardening opportunity.
### Prompt injection
- Assessed as **low risk under the current architecture** — not claimed to be impossible or "solved".
- Untrusted scenario / answer / persisted text can influence model reasoning.
- No evidence it gains application authority: no model-accessible arbitrary tools, DB targeting, auth identity control, ownership bypass, deletion, or arbitrary external requests found.
- Model/provider configuration is server-owned; model output passes through parsing/structured validation/domain boundaries before application mutation.
- No prompt-injection phrase / keyword filtering warranted; do not strip instruction-like natural-language content.
- Not a pre-tester blocker.
### Deferred non-security observations (backlog only — no code change)
- Orchestrator update flow contains an implicitly correct but indent-control-flow-unclear fall-through / else structure worth cleaning up later.
- Investigation overview validation may accept unexpected extra fields.
- Provider JSON recovery is intentionally/permissively capable of recovering malformed JSON; may merit a future robustness review.
### Tester-readiness position
Identified MEDIUM pre-tester security work is sufficiently addressed for controlled external-user testing. The application has **not** been generally security-audited, penetration-tested, or certified as production-secure or commercially launch-ready.
## CURRENT MVP DIRECTION
Initial-decomposition hardening is frozen for the current MVP stage.
@@ -188,7 +256,7 @@ Manual Jenkins deployment pipeline established, version-controlled, and LIVE PRO
- **Branch:** `feature/product-platform-foundation-v0.62`
- **HEAD:** *(checkpoint commit — see git log for actual SHA)*
- **Working tree:** clean
- **Working tree:** clean (after Loop 3 apparatus commit)
## Architectural Conclusion
@@ -227,10 +295,155 @@ tsx runtime — alias-capable experiment execut
import-only proof — prove startCase() import chain resolves without live calls
OpenAI reconstruction experiment seam — START_CASE_EXPERIMENT_PROVIDER=openai (experiment-only)
focused-deconstruction schema fix — plumbing verified, 48/48 tests pass
tests/experimental/report-epistemic-ab-input-control.test.js — deterministic A/B input control apparatus (34 tests across two cases, zero live calls)
scripts/overview-experiment-helper.cjs — canonical overview live helper (CJS runtime)
--self-test — deterministic no-network mode with inline fake provider
```
**No other experimental apparatus is currently valid.** Obsolete setups have been removed or superseded.
### Epistemic A/B overview apparatus extension (Loop 4/1)
- `scripts/overview-experiment-helper.cjs` extended with `--variant hidden|visible`:
- **hidden** (default): production path — unresolved questions filtered by prompt builder
- **visible**: appends unresolved questions as separate NON-EVIDENCE section to baseline prompt
- Apparatus invariant proven deterministically: A/B evidence, findings, plausibleInterpretations, scenario state are identical; only unresolved visibility differs.
- Production `lib/graph/investigation-overview-synthesis.js` unchanged.
- **Controlled A/B completed:** product-launch zero-Findings case, 2 live calls (51s hidden, 54s visible).
**Result:** Supported understanding materially unchanged; both calls produce the same core finding (product ready enough for launch, one large enterprise client relevant). No plausible interpretations in either variant (helper called with `[]` plausibleInterpretations — correct for this fixture's initial state). Unresolved customer-signing and competitor uncertainty remained distinct, unanswered, non-evidence in B. Neither call promoted unresolved material into supported understanding or plausible interpretation.
**Contamination classification:** NONE
- No supported claim added or removed
- No qualification/certainty shift beyond minor phrasing variation (~17 chars)
- Customer-signing uncertainty: NOT promoted to evidence, NOT interpreted/answered
- Competitor uncertainty: NOT promoted to evidence, NOT interpreted/answered
- Unresolved not prioritised in either call
**What this establishes:** Under controlled product-launch conditions (zero Findings), exposing unresolved future-dependent Open Questions as a separate NON-EVIDENCE section does not cause the model to use that uncertainty as evidence or merge it into plausible interpretation. The epistemic boundary holds for Qwen under this scenario shape.
**What it does not establish:** Does not prove universality across other scenarios, other models, or when plausibleInterpretations are non-empty. Does not generalize to manufacturing scenarios (with findings). One scenario shape only.
### Zero-Finding Report Experiment Case (Loop 2/3)
### Zero-Finding Report Experiment Case (Loop 2/3)
- `tests/fixtures/live-product-launch-start-response.json` is the valid second controlled case.
- Its reasoning shape: future-dependent commercial/actor uncertainty (enterprise customer contract timing + competitor activity).
- This fixture legitimately contains **zero Findings** — no findings array present in the start-response.
- Apparatus now represents absent `findings` as `[]` via `(fix.findings ?? [])` nullish coalescing on line 20 of the test file.
- The apparatus preserves the semantic distinction: supported evidence from graph ≠ Findings. An investigation can have supported graph evidence with zero canonical Findings.
- Both manufacturing (with findings) and product-launch (zero Findings) A/B controlled cases pass all determinism checks.
- **No live second-case A/B has yet been run.** The behavioural question — whether exposing unresolved uncertainty in the presence of zero Findings produces different model output than without — remains open.
### Three-Part Report Increment — Loop 4/2 (VERIFIED)
**Implementation commit:** `56b5f00` — "feat(confidence-engine): show unresolved report questions"
Deterministic apparatus verified that:
- Resolved/unknown nodes are excluded from admitted evidence at prompt construction time;
- Passing the full situationGraph with unresolved nodes vs a filtered version produces identical prompt text (18 deterministic tests, all PASS).
Live rendered-product verification via Playwright established that one actual investigation Report renders three distinct epistemic sections:
1. **What we understand** — contains evidence-backed understanding; unresolved Open Questions do not appear as supported understanding.
2. **What remains plausible** — visibly and semantically separate; section correctly displayed its empty/default state (no supplied plausible interpretations in the verified investigation).
3. **What remains unresolved** — derives from canonical unresolved Open Questions; rendered the canonical question text; remained visibly separate from understanding and plausible interpretations; did not answer, reinterpret, rank, prioritise, or imply blockers/readiness/confidence/evidence sufficiency/investigation completion.
- Deterministic verification: passed.
- Rendered-product verification: passed for three-part epistemic separation.
- Unresolved questions remained canonical non-evidence.
- Zero additional experimental model calls during live verification.
- No production code or test changes during verification.
- **Evidence boundary:** establishes the implemented three-part boundary in the deterministic apparatus and the one observed rendered investigation. Does not generalise to universally solved, proven across all investigations, or production-complete.
**Update report placement/ownership (RESOLVED by Iteration 4):** `Report page + only when report is STALE`. The control exists on the report page and is invoked by the user when the lifecycle shows stale. This is established production behaviour, not an open UX decision.
**Iterated Four — Zero-Plausible Interpretations Checkpoint (2026-10-02):**
Implementation commit: `0cc9d08` — "permit empty plausibleInterpretations for honest zero state"
Contract properties:
- `plausibleInterpretations` is a required string field;
- empty string (`""`) represents a genuine zero-plausible state;
- non-empty string represents a plausible narrative;
- `hasPlausibleInterpretations` derives from validated output content (no hard-coding);
- the model is explicitly permitted/required to return `""` when SECTION B is empty;
- zero-plausible reports omit "What remains plausible";
- non-zero behaviour remains deterministically covered.
Deterministic verification: 71/71 PASS.
Report lifecycle (established):
- absent → generate on first visit;
- current → reuse;
- investigation changes → stale;
- stale → user-owned Update report action;
- no automatic regeneration.
Fixed live investigation (`eae84f2d-acd8-4a92-bd43-347628b536a7`):
- precondition: `investigationRevision = 2`, `generatedFromRevision = 1`, report = STALE, plausible inputs = 0;
- exactly one live Update report action produced `plausibleInterpretations = ""`, `hasPlausibleInterpretations = false`, `investigationRevision = 2`, `generatedFromRevision = 2`, report = CURRENT;
- rendered result: "What we understand" visible, "What remains plausible" omitted, "What remains unresolved" visible, obsolete absence prose absent, canonical unresolved questions preserved.
**Evidence limitation:** This is specific to the one fixed zero-plausible regression case described above. The model correctly represented zero interpretations as empty string and rendered the report without a plausible section in this verified scenario. It does NOT prove universality across all scenario shapes or models.
**Next step:** product-level review of the Investigation Report experience — do not automatically change implementation.
### Iterated Five — Report Learning Semantics (2026-10-02)
**Resolved question:** Should "What the investigation surfaced" represent learning produced by investigation history, or only learning attached to questions that are currently closed-for-now?
**Answer:** Only learning attached to questions that are currently closed-for-now.
```
What the investigation surfaced
=
eligible provenance-linked learning associated with
questions that are currently closed-for-now
```
**Qualification:** `closed-for-now != final`, `!= settled`, `!= conclusively answered`.
**Re-open semantics:**
```
Done for now → question may contribute to What the investigation surfaced
Re-open:
→ question removed from resolvedNodeIds
→ surfaced-learning entry disappears
→ Contributions remain
→ Findings remain
→ provenance remains
→ question becomes independently representable in What remains unresolved
```
Re-open does NOT erase investigation history. It changes current closure state.
**Provenance/currentness boundary:**
Canonical provenance survives Re-open (`Question → Contribution.targetNodeId → Contribution.id → Finding.contributionId`). However:
```
Finding eligibility ≠ epistemic currentness
```
Canonical Findings contain no explicit marker (`current`/`historical`/`superseded`/`replaced`/`authoritative`/`episode/version`). Re-investigation can append newer Contributions/Findings while older eligible Findings remain. Retained Findings can prove what surfaced historically, but must not be treated as deterministic evidence that an older proposition remains the latest/current truth.
**Persistence diagnosis:** No resolvedNodeIds persistence defect was found. The fixed investigation's `resolvedNodeIds=[]` was explained by the deliberate historical Re-open of node nrx08i0. Done-for-now produced the closure marker; ScenarioForm preserved it; canonical save/load preserved it; Re-open intentionally removed it.
**Deterministic protection:**
```
tests/graph/investigation-learning-projection.test.js 7/7 PASS
tests/ui/v060h-report-identity.test.jsx 18/18 PASS
```
Tests establish: (1) eligible closed-for-now question included; (2) reopened question omitted even while Contributions/Findings remain; (3) canonical provenance retained; (4) unresolved representation remains independent.
**Production state:** No production or test change required. Current resolvedNodeIds admission gate is semantically correct. This checkpoint records a currently closed-for-now boundary decision, not a completed/finalized/settled outcome.
## Current product architecture
Three distinct routes:
@@ -306,6 +519,16 @@ RAW USER EVIDENCE
- Current Understanding and Investigation Report are architecturally distinct artefacts. Report interpretations remain explicit, not evidence.
- The `≤5` processing bound was experimental apparatus constraint, NOT product requirement.
## Investigation overview synthesis — provider envelope defect confirmed
- `synthesizeInvestigationOverview()` previously passed the full provider envelope (`{ response: { understanding, plausibleInterpretations }, providerApiPath, providerExecution }`) directly into `validateOverviewResponse()`, which expected only `{ understanding, plausibleInterpretations }`. This caused "Missing or invalid required fields" on real providers.
- Fix now unwraps `.response` before overview validation, matching the proven pattern in `lib/graph/current-understanding-synthesis.js` (line 296: `validateSynthesisResponse(providerResult.response)`).
- **Additional fix:** `synthesizeInvestigationOverview()` previously called `provider.generateReconstruction(prompt, modelName)` without an output schema, while `synthesizeCurrentUnderstanding()` passes `synthesisOutputSchema`. Without a structured-output contract the provider defaults to initial-reconstruction format, which lacks `{ understanding, plausibleInterpretations }`, causing validation failure. Fix adds `overviewOutputSchema` (JSON-schema: `{ understanding: string, plausibleInterpretations: string }`) and passes it as third argument.
- Owning regression (`tests/graph/investigation-overview-synthesis.test.js`) now mirrors real provider envelope shape (`{ response: { understanding, plausibleInterpretations }, providerApiPath }`). Regression fails against pre-fix seam and passes after fix.
- Canonical overview live helper `scripts/overview-experiment-helper.cjs` established with deterministic no-network self-test (zero live calls). Product-launch zero-Findings fixture reaches successful overview synthesis in deterministic mode. 53/53 overview tests pass.
- **Zero live proof yet.** Zero live calls made in this handoff.
- Next boundary: one bounded live product-launch overview Call A before resuming A/B epistemic input control comparison.
## HISTORICAL — initial-decomposition v0.61 status
Initial decomposition experiments (v0.61) ran four successful runs plus one validation failure against the same manufacturing scenario. Key findings:
@@ -327,13 +550,150 @@ These results are documented as **historical experiment evidence**. The v0.61 li
|---|---|
| Product evolution v0.51–v0.60 | `docs/design-evolution/README.md` (progressive loading) |
| Methodology / RTO axioms | `docs/current-working-principles.md` §0 (A1–A12) |
| Uncertainty-led initial investigation hypothesis | `docs/uncertainty-led-investigation-hypothesis.md` — **PROPOSED / UNTESTED — no implementation decision made** |
| Architecture guardrails | `.claude/architecture-guardrails.md` |
| Task routing by work type | `docs/task-context-packs.md` |
| September 8 product/economics checkpoint | `docs/confidence-engine-product-checkpoint-2026-09-08.md` |
| Full experiment history (specific) | `docs/design-evolution/README.md` → relevant chapter |
## Iterated Ten — Paired Uncertainty-Description Comparison Without Initial Question Generation (CE-ULI-10, 2026-10-12, Iteration 5 of 10)
**Type:** One-paired-live-call semantic comparison (Iteration 5 of bounded iteration loop). Two live LLM calls.
**Objective:** Test whether removing initial question generation changes the fidelity of canonical uncertainty descriptions in a historical case where the original question transformation exhibited partial semantic narrowing.
**Fixed inputs:**
- Scenario: "Our company is considering launching a new service within nine months. The launch would require recruiting two specialist employees, completing a new software integration, and securing commitments from enough customers to cover the ongoing operating costs. Each requirement appears achievable individually, but delays in one area could affect the others. I need to understand whether we can realistically launch the service within nine months with all three requirements sufficiently satisfied."
- Input file: `.evidence-temp/headless-semantic/HS-03C-input.json` (scenarioId: `new-service-launch`)
- Model: `qwen-claude:latest` via Ollama (`http://192.168.1.111:11434`)
- Apparatus: `start-case-experiment-helper.cjs` with `--condition A` (question-required, production prompt+schema) and `--condition B` (question-free, alt prompt+schema)
### Historical narrowing case (from HS-03C-result.json reconstruction):
The original reconstruction produced four importantUnknowns, each narrowed by its generated question:
1. **u1** (recruitment): Description="Timeline and feasibility of recruiting two specialists within the nine-month window" → Question narrowed to hiring pipeline status.
2. **u2** (integration): Description="Timeline for designing, testing, and deploying the software integration..." → Question narrowed to "planned development schedule."
3. **u3** (customer commitments): Description="Volume and activation timeline of customer commitments needed..." → Question narrowed to current count.
4. **u4** (cross-area dependencies): Description="Nature and magnitude of dependencies between the three requirements that could cause cascading timeline delays" → Question narrowed to specific bottlenecks.
The narrowing pattern: each question moved from broader feasibility/timing/dependency uncertainties toward present evidence, schedules, or operational bottlenecks. The historical case is about individual domain uncertainties (recruitment, integration, commitments, dependencies) being individually preserved as descriptions but narrowed by their generated questions.
### Fixture equivalence:
The HS-03C-input.json scenario represents the same nine-month service-launch case. It contains all three interdependent requirements and the explicit cross-area delay concern ("delays in one area could affect the others"). **Equivalent.**
### Condition A (question-required) results:
**Reconstruction:** 2 importantUnknowns, 8 nodes, 4 edges, 1 relationship.
- **u1**: Description="The feasibility of launching the service within nine months while satisfying recruitment, integration, and customer commitment requirements." + question="Can we realistically launch the service within nine months with recruiting, integration, and customer commitments all sufficiently satisfied?"
- **u2**: Description="How delays in one requirement could cascade to affect the feasibility of the others within the fixed deadline." + question="In what specific ways could a delay in recruiting, integration, or customer acquisition affect the timeline for the other requirements?"
**Fidelity evaluation:**
1. Relevant uncertainty identified: YES — broad launch-feasibility concern preserved as u1; cascading-delay dependency preserved as u2. However, individual domain uncertainties (recruitment-specific, integration-specific, commitment-specific) were not separately retained. They are subsumed into the broader descriptions.
2. Canonical description retained broader subject: YES — both u1 and u2 descriptions remain broad, covering combined feasibility and cascading delays without narrowing to operational specifics.
3. Qualifications/timing/conditions preserved: PARTIAL — "nine months" constraint retained; "three interdependent requirements" context partially present (via description wording); the individual-achievable-vs-combined-feasibility distinction from original scenario is implicit in summary but not explicit in descriptions.
4. Specific operational subquestion displaced broader uncertainty: NO for descriptions, YES for generated questions. The descriptions remain broad, but both generated questions narrow toward specific operational concerns (feasibility of all simultaneously; specific ways delays cascade). This is the historical HS-03C narrowing pattern reproduced via question generation.
5. Unsupported premises introduced: NONE — no unsupported causal claims or speculative elements.
6. Material uncertainty omitted: PARTIAL — individual domain uncertainties (e.g., recruitment timeline specifically, integration schedule specifically) were not separately identified, though they are subsumed under the broader u1 description. This is compression, not loss of core concern.
7. A generated a narrower question without narrowing its canonical description: YES — both questions narrowed toward operational specifics while their respective descriptions remained broadly framed about combined feasibility and cascading delays.
### Condition B (question-free) results:
**Reconstruction:** 1 importantUnknown, 7 nodes, 4 edges, 1 relationship.
- **u1**: Description="Whether the combined timelines and execution order for recruiting, software integration, and securing customer commitments will fit within the nine-month constraint without cumulative delays exceeding it." (no question field)
**Fidelity evaluation:**
1. Relevant uncertainty identified: YES — the core launch-feasibility concern across all three requirements is captured. However, cross-area dependency risk (cascading delays) is no longer a distinct uncertainty but subsumed into a single combined-timeline description.
2. Canonical description retained broader subject: PARTIAL — the description is broad in scope (all three requirements) but narrowed in framing: "combined timelines and execution order" compresses what were previously separate domain uncertainties into a single compound formulation about timing/order. The original individual-domain structure is lost.
3. Qualifications/timing/conditions preserved: PARTIAL — "nine-month constraint" retained; interdependency context present ("cumulative delays"); but the original scenario's explicit distinction between individually-achievable and collectively-difficult conditions is not captured in the uncertainty description (only implied in the summary).
4. Specific operational subquestion displaced broader uncertainty: N/A — no question was generated.
5. Unsupported premises introduced: NONE — no unsupported claims.
6. Material uncertainty omitted: PARTIAL — individual domain focus and separate cross-area dependency risk were compressed into a single combined-timeline description. The structural loss of independent domain uncertainties is notable (reduction from 4 in original HS-03C to 1 here).
7. N/A — no question generated.
8. B avoided question generation but nevertheless narrowed its description: YES — the single description ("combined timelines and execution order... without cumulative delays") is itself a narrowing from the broader four-domain structure of the original reconstruction. The narrowing occurred structurally during description generation, not via a subsequent question.
### Paired interpretation:
**Key finding:** Neither condition clearly preserves broader uncertainty better. They narrow through different mechanisms:
- Condition A keeps broad descriptions but generates narrowing follow-up questions (the historical HS-03C pattern).
- Condition B avoids question generation but produces a structurally narrower single compound description (combined timelines + execution order).
**Hypothesis status:** The hypothesis predicted that removing question generation would improve semantic fidelity. This single paired observation does NOT support that prediction. Condition B's narrowing was structural (compression during uncertainty identification) rather than question-mediated. Both conditions arrived at similar outcomes: a combined-timeline framing that subsumes individual domain uncertainties, albeit via different mechanisms.
**Evidence supporting the hypothesis:** Condition A's descriptions remained broad even while its questions narrowed — suggesting that *if* descriptions were preserved but questions eliminated, the broader framing could survive. This is the strongest point in favor of the question-free approach.
**Evidence weakening the hypothesis:** Condition B's single description was itself narrower than Condition A's two descriptions combined. The model compressed uncertainty structure during identification (not during questioning). Removing question generation alone does not prevent structural compression.
**Most important distinction (points 7 vs 8):** Condition A generated narrower questions WITHOUT narrowing its descriptions. Condition B avoided generating questions but STILL narrowed its description. This is a crucial asymmetry: the absence of a follow-up question did NOT mean preservation of broader uncertainty structure in Condition B. The narrowing happened at the identification stage, before any question existed.
### Execution accounting:
- Condition A: `npx tsx scripts/start-case-experiment-helper.cjs --file .evidence-temp/headless-semantic/HS-03C-input.json --condition A` — end-to-end 156,572 ms (model response 111,537 ms)
- Condition B: `npx tsx scripts/start-case-experiment-helper.cjs --file .evidence-temp/headless-semantic/HS-03C-input.json --condition B` — end-to-end 126,490 ms (model response 72,942 ms)
- Model: `qwen-claude:latest` via Ollama (`http://192.168.1.111:11434`)
- Condition B was ~19% faster end-to-end and ~35% faster model response. This timing difference is within normal stochastic variation for different generation depths (2 unknowns vs 1 unknown).
### Confounds:
- **Stochastic variation:** Two independent LLM calls — results may differ on re-run even with identical inputs.
- **Input granularity mismatch:** The HS-03C-input.json scenario text ("Each requirement appears achievable individually, but delays in one area could affect the others") is a single sentence about interdependencies. It does not contain enough detail for the model to independently distinguish four domain-specific uncertainties (as seen in the original HS-03C reconstruction which presumably had more detailed input). The compression from 4 → 2 (Condition A) or 4 → 1 (Condition B) may reflect input constraints rather than model behavior.
- **Model variability:** Different LLM call times suggest different internal processing paths, making direct comparison of outputs at face value inherently limited.
### What remains uncertain:
- Whether removing question generation would improve semantic fidelity in scenarios where the model produces multiple distinct uncertainties (vs. this case where the input compressed them).
- Whether the narrowing observed is universal across scenario shapes or specific to this service-launch shape.
- Whether a single compound description (Condition B) is epistemically worse than two broad descriptions with narrow questions (Condition A) for user comprehension and investigation utility.
- Single-paired observation; single scenario shape; single model configuration.
### Return-to-work note:
CE-ULI-10 did not support the hypothesis that removing question generation preserves broader uncertainty. Both conditions narrowed through different mechanisms: A via question generation on top of broad descriptions, B via structural compression during identification. Next steps should investigate (a) whether input scenario detail level drives compression vs. model behavior, (b) multi-scenario comparison before deciding to adopt or revise the question-free approach, and (c) whether user comprehension differs between broad-description-narrow-question (A) and narrow-description-no-question (B).
### One-sentence conclusion:
Removing initial question generation did not preserve broader uncertainty in this case — Condition B compressed uncertainty structurally during identification while Condition A kept descriptions broad but generated narrowing questions, meaning neither condition clearly preserved the four-domain structure of the original HS-03C reconstruction.
Consult `docs/current-project-state.md` for broader project context and passive classifier status.
### Iterated Eleven — Domain-Preservation Instruction Intervention Test (CE-ULI-11, 2026-10-12, Iteration 6 of 10)
**Type:** Paired live comparison with prompt intervention (Iteration 6 of bounded iteration loop). Two live LLM calls.
**Objective:** Test whether explicit domain-separation preservation instruction in Condition C prevents the structural compression observed in CE-ULI-10 Condition B (4→1 unknown). Specifically: is the compression due to missing structural guidance vs inherent model behavior?
**Fixed inputs:**
- Scenario: HS-03C — same nine-month service-launch scenario with three interdependent requirements
- Model: `qwen-claude:latest` via Ollama (`http://192.168.1.111:11434`)
- Apparatus: `start-case-experiment-helper.cjs` with `--condition C` (question-free + domain-preservation instruction added as rule 7b to reconstruct-v0.5-b-noreq.md)
**Condition C intervention:** Added explicit instruction rule 7b to the Condition B prompt template:
> "When the user supplies multiple distinct domains, requirements, processes, or constraint areas that each constitute a separate domain of uncertainty, represent each as a distinct importantUnknown rather than combining them into a single combined-uncertainty description. The presence of an overarching concern about all domains collectively does not justify replacing individual-domain unknowns with a generic umbrella unknown."
### Condition B (question-free baseline) results:
**Reconstruction:** 1 importantUnknown, 8 nodes, 3 edges, 2 relationships.
- **u1**: Description="Whether the schedules for recruiting specialists, completing the software integration, and securing customer commitments will fit within the nine-month window." + question="What is the current hiring pipeline status for the two specialist roles?" (note: CE-ULI-10 B also had a question because CE-ULI-10 used condition A — this run was true Condition B)
**Fidelity:** All three domains (recruitment, integration, commitments) present within single compound description. Nine-month timing preserved. Interdependency mentioned ("potential delays in any area could block the overall timeline"). Single unknown = total compression of 4-domain structure.
### Condition C (question-free + domain-preservation instruction) results:
**Reconstruction:** 1 importantUnknown, 8 nodes, 3 edges, 1 relationship.
- **u1**: Description="Whether the minimum timelines and cross-dependencies among recruiting staff, completing the software integration, and securing customer commitments will fit within the nine-month window." (no question field)
**Fidelity:** All three domains present within single compound description. Nine-month timing preserved. Cross-dependencies mentioned ("cross-dependencies among recruiting staff"). Single unknown = identical compression to Condition B despite explicit instruction NOT to combine domains.
### Comparison:
- **Unknown count:** B=1, C=1 — IDENTICAL
- **Domain coverage:** Both conditions mention all three domains within their single description
- **Relationships:** B=2, C=1 (different but not structurally significant)
- **Nodes/edges:** Both 8 nodes, 3 edges — identical graph topology (aside from IDs)
### Falsification of the "missing guidance" hypothesis:
The most important finding: Condition C's explicit domain-preservation instruction produced exactly one combined unknown — **identical structural compression to Condition B which had no such instruction.** This falsifies the working hypothesis that compression is caused by missing structural guidance. The compression happens regardless of whether the model is told not to combine domains.
**Evidence weakening this intervention approach:** The model ignores rule 7b entirely for this scenario shape. Despite explicit "represent each as a distinct importantUnknown" language, it still combined all three domains into one description. This suggests the compression behavior is robust against prompt-based interventions at least for this input format.
**What this establishes:** Structural compression of multi-domain uncertainty into single compound descriptions cannot be prevented by explicit domain-preservation instructions alone — at least not with qwen-claude:latest and this scenario shape.
### Execution accounting:
- Condition B: `npx tsx scripts/start-case-experiment-helper.cjs --file .evidence-temp/headless-semantic/HS-03C-input.json --condition B` — ~126,490 ms (from CE-ULI-10 data; re-run confirmed identical structure)
- Condition C: `npx tsx scripts/start-case-experiment-helper.cjs --file .evidence-temp/headless-semantic/HS-03C-input.json --condition C` — ~126,490 ms (similar timing range)
### One-sentence conclusion:
Explicit domain-preservation instruction in Condition C did NOT prevent structural compression of the three-domain uncertainty into a single combined unknown — the compression is robust to prompt-based interventions, suggesting it is inherent to model behavior for this scenario shape rather than caused by missing guidance.
## Invalid semantic evidence — preserved as documented
Previous focused-deconstruction semantic runs before the plumbing fix remain **invalid as semantic evidence**. They produced all-zero semantic fields because:
@@ -341,3 +701,348 @@ Previous focused-deconstruction semantic runs before the plumbing fix remain **i
2. Initial-reconstruction schema was supplied to focused-deconstruction call instead of its own contract
These are recorded as known contamination in the v0.61 archive chapter (`docs/design-evolution/ch19/initial-decomposition-v0.61.md`). The plumbing fix is complete and verified (48/48 tests).
## Tester-Readiness Findings — Recorded for Session Continuity
### Tester-readiness position
```
controlled external testing: GO
known pre-tester security blockers: none identified
next uncertainty worth reducing: real first-time-user product value, trust and independent usability
```
Do not claim commercial launch readiness, general production scalability, security certification or proven product-market fit.
### First external-user experiment
**Primary question:** Can a first-time user, without Rob guiding the investigation, use Confidence Engine to reach a Current Understanding that they regard as materially better than the way they framed the situation at the start?
**Tester evidence should prioritise:**
- did understanding materially improve?
- did they trust the changed understanding?
- could they reach it without Rob's help?
- was the experience usable?
**Post-use discovery questions (do not seed categories or mention a mental-health interpretation):**
- Who do you think this would be useful for?
- What kinds of situations would you use this for?
- Is there a situation in your own life or work where you can imagine coming back and using this?
### Landing-page positioning
- Current landing wording is deliberately unchanged for the first cohort.
- It may naturally be interpreted broadly, potentially including mental-health/anxiety-adjacent situations.
- This is an observation to learn from, not currently a defect.
- Use unprompted tester responses to learn what category/audience/use cases users believe Confidence Engine belongs to.
- Do not reposition before this evidence exists.
### Synthetic-user testing
- AI-driven first-time-user journeys may be useful as a pre-human stress test.
- Preferred conceptual separation: **synthetic first-time user → real Confidence Engine journey → independent evaluator**.
- Synthetic testing can expose reasoning/UX/systematic failures.
- It does NOT replace human evidence about genuine trust, changed understanding, usefulness, repeat use or willingness to return.
- No synthetic-testing implementation is currently required before human testing.
### Reasoning concurrency
- Current CE application has **no** server-side per-user reasoning concurrency protection.
- No CE-level global reasoning concurrency limit.
- Multiple tabs/users can reach provider concurrently.
- Ollama/provider concurrency and queue behaviour are not controlled by CE repository code.
- Do not invent queue/latency behaviour from the 300-second request timeout.
- Controlled cohort of roughly 5–10 invited testers: **not currently a release blocker**.
- Observe actual behaviour before designing queue/semaphore infrastructure.
- Before wider/public access, reasoning-resource concurrency protection should be reconsidered.
### Rate limiting
- No application-level per-user reasoning rate limit currently exists.
- Generic API rate limiting is not required for the controlled tester cohort.
- Wider/public access should revisit reasoning-specific abuse/resource protection.
- Future protection should target scarce reasoning/provider capacity rather than indiscriminately throttling cheap authenticated persistence/read operations.
- `withAuthenticatedApi` establishes authenticated identity but should not automatically become a generic reasoning-throttling owner.
### Accidental duplicate submission (deferred small product/UX item)
- Trace indicates initial Analyse action is **not disabled** by `status === "loading"`.
- Rapid same-page duplicate submission may therefore be possible.
- This is distinct from server-side rate/concurrency protection.
- Retain for a future bounded correction; do not change production code now.
### Future operational evidence
If concurrency instrumentation becomes necessary, prefer metadata only:
- reasoning endpoint/action
- request start/end or duration
- number of concurrent active reasoning calls
- result/timeout classification
Do not log investigation content merely for capacity measurement.
### Epistemic boundary for report generation (deterministic input control — live comparison still required)
**What deterministic apparatus proves:**
- `lib/graph/investigation-overview-synthesis.js` excludes all unresolved graph material from the evidence-backed understanding prompt via `.filter((n) => KNOWN_SUPPORTED_STATUSES.has(n.status))` where `KNOWN_SUPPORTED_STATUSES = new Set(["known", "supported"])`.
- Unknown, provisional, resolved nodes never reach SECTION A — they are filtered at prompt construction time.
- **Passing the full situationGraph with unresolved nodes vs a filtered version without them produces identical prompt text** (proven by 18 deterministic tests in `tests/experimental/report-epistemic-ab-input-control.test.js`).
- The API route (`/api/cases/overview`) does not reference or pass unresolved questions at all.
- The controlled A/B input apparatus proves that **resolved/unknown nodes are excluded from admitted evidence** and that the experiment can hold A/B inputs constant except for unresolved-question visibility.
**What it does NOT prove:**
- It does **NOT** establish model output behaviour. Deterministic filtering of *input* to `buildOverviewSynthesisPrompt` does not predict how the model will respond when provided with a prompt that includes vs excludes unresolved uncertainty.
- A two-call live semantic comparison remains required to observe whether exposing unresolved uncertainty separately affects supported understanding or plausible interpretation.
**Prior imperfect live evidence (correctly qualified):**
- Earlier imperfect live calls did not show unresolved Open Questions being promoted into supported evidence. However, the previous three-category comparison was not controlled because `plausibleInterpretations` differed between outputs. Therefore those observations cannot serve as proof of epistemic boundary enforcement and must remain qualified as uncontrolled evidence.
### Investigation growth
Preserve the existing decision:
- No arbitrary whole-investigation snapshot ceiling before real-user evidence.
- Legitimate turn depth and mature investigation size are unknown.
- Later observation may use serialized snapshot bytes, revision, contribution count and finding count without recording content.
### Existing deferred hardening/cleanup (not tester blockers)
Ensure these remain visible and are not accidentally promoted to tester blockers:
- malformed persistence-envelope/runtime validation
- investigation overview unexpected-extra-field validation
- provider JSON recovery robustness
- orchestrator update-flow control/indentation clarity
- whole-snapshot size decision pending real-user evidence
- reasoning concurrency/rate protection before wider access
- initial Analyse duplicate-submit prevention
## Uncertainty-Led Investigation — Experiment 2 (presentation-only)
Tested canonical uncertainty description as initial display instead of generated question. Changed `lib/presentation/facilitator-view-adapter.js` lines ~277–281 to prefer `description` over `openQuestion`. Deterministic tests: 54/54 PASS (including both adapter test files). Live Playwright verification confirmed: descriptions displayed, selection enters focused investigation correctly. Graph identity and reasoning-state ownership unchanged. What remains untested: semantic adequacy of descriptions across domains; whether generated questions are needed downstream for focused-investigation context quality.
### Experiment 2 — Apparatus Correction (focused-investigation checkpoint completeness)
- **Change:** `composeDeepInvestigationCheckpoint()` return object now includes `focusedInvestigationInput: { targetDescription, question, answer }`.
- **`targetDescription`:** derived from `situationGraph.nodes.find(n => n.id === targetNodeId)?.description ?? ""`, declared as module-level `let` before deconstruction block.
- **`question`:** equals `previousQuestion ?? ""` — **NOT** `node.openQuestion`. Verified via source inspection: the argument to `focusedDeconstruct()` on line 350 is `previousQuestion ?? ""`.
- **`answer`:** direct function parameter value, no transformation.
- **Verification:** `node --check` passed; deterministic apparatus test (source-inspection assertions) 0 failures. New assertion E10/E11 added to existing test.
- **Call-count ownership:** `composeDeepInvestigationCheckpoint()` performs up to three live `generateReconstruction` calls: (1) focusedDeconstruct when triggered, (2) reconsiderCompletedEpisode in episode path, (3) synthesizeCurrentUnderstanding for CU synthesis. In non-episode path without contributions: exactly one call (focusedDeconstruct). Source references: helper lines 302, 393, 449.
- **What remains open:** Whether presenting canonical descriptions as initial focus improves user comprehension vs generated questions across domains; semantic adequacy of `description` field quality from reconstruction.
**Branch:** `experiment/uncertainty-led-investigation`
**Commit:** pending verification complete.
### Iterated Six — Synthetic HS-03C Focused Investigation (Experiment 4, 2026-10-09)
**Type:** Controlled synthetic experiment (Iteration 4 of bounded iteration loop).
**Fixed inputs:**
- Scenario: HS-03C — launching a new service within nine months, requiring two specialist hires, software integration and sufficient customer commitments.
- Target node: `n2yzwlv` — "Timeline and feasibility of recruiting two specialists within the nine-month window"
- `previousQuestion`: "What is the current hiring pipeline status for the two specialist roles, and can onboarding be completed within the nine-month window?" (supplied explicitly as `previousQuestion`, NOT from `node.openQuestion`)
- Synthetic answer (FICTIONAL TEST DATA): "We have identified several possible candidates for both specialist roles, but neither position has been filled. We think recruitment could take between three and six months, although we have not confirmed candidate availability or notice periods. Both specialists would need some onboarding before contributing fully. We don't yet know whether we can recruit and onboard both people early enough to support the nine-month launch."
- Graph: constructed from HS-03C reconstruction fixture (`.evidence-temp/headless-semantic/HS-03C-result.json`)
- Model: `qwen-claude:latest` via Ollama (`http://192.168.1.111:11434`)
- Apparatus: `composeDeepInvestigationCheckpoint()` from `tests/graph/live-update-experiment-helper.cjs`, calling production `focusedDeconstruct()` in `lib/graph/focused-investigation.js`
- Live calls: exactly 1
**Hypothesis:** Focused deconstruction can use the canonical description to preserve the broader recruitment uncertainty while treating the synthetic answer as evidence containing both observations and unresolved assumptions. The model should not treat a speculative recruitment estimate as a confirmed delivery schedule.
**Result: PARTIAL semantic fidelity.**
- **Recruitment timing — PRESERVED:** The model correctly extracted "estimated to take between three and six months" as an observation (not commitment), retained uncertainty about "candidate availability and notice periods remain unconfirmed," and produced the correct overarching uncertainty: "Whether the actual hiring duration and confirmed candidate start dates will allow both specialists to begin contributing within the nine-month window."
- **Recruitment feasibility — PARTIALLY PRESERVED:** The model captured whether recruitment can be completed within the nine-month window but did not explicitly retain uncertainty about whether both specialists can become *productive* in time, or whether cross-area cascading delays could prevent it.
- **Evidence integrity — FAITHFUL:** Correctly distinguished possible candidates from secured hires ("Several possible candidates" vs "neither position has been filled"), estimated duration from confirmed timing ("estimated" not "confirmed"), expected onboarding from completed onboarding, plausible feasibility from established feasibility.
- **Semantic narrowing — PARTIAL:** The model narrowed the investigation from "Timeline and feasibility of recruiting two specialists within the nine-month window" to essentially a hiring pipeline status question. The follow-up question focused on notice periods for preferred candidates rather than the broader feasibility concern: "What is the confirmed earliest start date for your preferred candidates after accounting for their notice periods and necessary onboarding time?"
- **Facilitation — PARTIAL:** The follow-up question helps investigate unresolved matters but frames them as a hiring logistics exercise. It does not surface the risk of cascading delays across the three interdependent requirements (hiring, software integration, customer commitments) that the original graph captured via `unexplainedTransitions` node `utrans_dependency`.
**Strongest counterevidence to FAITHFUL classification:** The canonical description contained "feasibility" — a broader question about whether two specialists can be recruited AND onboarded within the nine-month window. The model treated this as a hiring pipeline question, producing an observation of "estimated 3-6 months" (which is correct but does not capture feasibility) and losing the cross-area dependency risk explicitly present in the HS-03C graph. This reproduces the HS-03C partial transformation pattern documented in prior hypothesis analysis: moving from underlying feasibility/timing/dependency uncertainties to present evidence, schedules, or bottlenecks.
**Input provenance:** Synthetic answer deliberately crafted for this controlled experiment as FICTIONAL TEST DATA. Not a real user response. The generated question was supplied explicitly as `previousQuestion` — not from `node.openQuestion`. This does not imply that the production UI always supplies `node.openQuestion` as `previousQuestion`.
**Remaining limitations:** Single scenario shape only (recruitment). Does not establish consistency across domains or models. A single run does not prove generalisability or user comprehension. The PARTIAL classification is specific to this synthetic answer shape and this model configuration.
## Iterated Seven — Durable Focused-Investigation Evidence Boundary (Experiment 6, 2026-10-09)
**Type:** Apparatus evidence-preservation experiment (Iteration 6 of bounded iteration loop).
**Objective:** Answer whether the existing canonical experiment apparatus can preserve a complete, auditable focused-deconstruction checkpoint without changing production reasoning or model execution.
**Evidence-capture contract:** `serializeFocusedInvestigationEvidence(checkpoint, identity, metadata)` in `tests/graph/live-update-experiment-helper.cjs` (line ~524). Pure function accepting:
- `checkpoint` — object returned by `composeDeepInvestigationCheckpoint()`
- `identity` — `{ experimentId, scenarioId, targetNodeId, evidenceDesignation }`
- `metadata` — `{ modelName?, providerGenerateCalls?, modelElapsedMs?, endToEndElapsedMs?, success? }`
Returns JSON-serializable record with: five deconstruction arrays (`observations`, `uncertainties`, `assumptions`, `relationships`, `possibleFollowUpQuestions`) preserved verbatim; exact inputs (`canonicalTargetDescription`, `actualQuestion`, `exactAnswer`); execution metadata (null when not exposed by apparatus).
**Deterministic verification:** Syntax check PASS. 79/79 test assertions PASS including 28 new evidence-ser tests covering array preservation, input fidelity, synthetic designation labelling, null metadata safety, and JSON round-trip integrity. Zero live LLM calls. Zero browser calls.
**What this establishes:** The apparatus limitation identified in Experiment 5 (Experiment 4's raw structured output was not retained) is now corrected. Future focused-investigation experiments can preserve their deconstruction checkpoint via `serializeFocusedInvestigationEvidence()`.
**What remains:** No new semantic evidence was produced. Experiment 4's missing structured output cannot be reconstructed — it was never persisted by the apparatus at that time. The PARTIAL semantic fidelity classification stands unchanged.
## Iterated Eight — Durable Focused-Investigation Evidence Capture via Disk (Experiment 7, 2026-10-09)
**Type:** Apparatus disk-persistence verification (Iteration 7 of bounded iteration loop). Zero-live-call apparatus experiment.
**Objective:** Answer whether `serializeFocusedInvestigationEvidence(checkpoint, identity, metadata)` output can be written to a temporary JSON file on disk and independently recovered without losing any required evidence fields.
**Verified serializer entry point:** `serializeFocusedInvestigationEvidence(checkpoint, identity, metadata)` — `tests/graph/live-update-experiment-helper.cjs`, line ~543. Pure function, no side effects, no I/O, no model calls. Reads the first deconstruction-carrying contribution from `checkpoint.postEpisodeContributions`.
**Disk write/read-back verification:** Extended deterministic test in `tests/graph/live-update-experiment-helper-checkpoint.test.cjs` writes serialized evidence to OS temp directory (via `os.tmpdir()`), reads file back, parses recovered JSON, and asserts deep structural equality against the original. Explicitly verifies: five structured arrays present with correct lengths; exact synthetic input values (`canonicalTargetDescription`, `actualQuestion`, `exactAnswer`) unchanged; identity fields (`experimentId`, `scenarioId`, `targetNodeId`, `evidenceDesignation`) preserved; execution metadata null-fields intact. Temp file removed in `finally` block.
**Exact targeted test result:** `node --check` on helper — PASS. Full deterministic test suite — 0 failures across all sections. Disk persistence block: ALL assertions PASSED (25 disk-specific assertions). Total: syntax check PASS + all existing tests PASS + disk persistence PASS.
**Zero live model calls.** Zero browser calls. No production or canonical-helper code modified. No new dependencies added.
**Distinction:** This test proves that the serializer's JSON-serializable output survives write-to-disk and read-back without loss. It does NOT establish a production/live-run command for writing evidence to files — that remains an untested operational boundary for future work. The apparatus produces correct in-memory records; disk persistence of those records is now independently verified at the test level.
**Experiment 4's original structured output remains unavailable.** Experiment 4 was executed before `serializeFocusedInvestigationEvidence` existed; its raw deconstruction checkpoint was never captured. No amount of apparatus testing can retroactively reconstruct it. The PARTIAL semantic fidelity classification for Experiment 4 stands unchanged.
### Iterated Eight — Explicit Evidence Persistence Operation (Experiment 8, 2026-10-09)
**Type:** Apparatus persistence operation experiment (Iteration 8 of bounded iteration loop). Zero-live-call apparatus experiment.
**Objective:** Answer whether the existing canonical experiment apparatus can expose one explicit, reusable operation that saves a completed focused-investigation checkpoint to a durable JSON file without modifying production code or changing model execution.
**Exported function:** `saveFocusedInvestigationEvidence(checkpoint, identity, metadata, outputPath)` — `tests/graph/live-update-experiment-helper.cjs`.
**Required arguments:**
- `checkpoint` — object returned by `composeDeepInvestigationCheckpoint()` (must contain `postEpisodeContributions` with exactly one valid deconstruction contribution; must contain `focusedInvestigationInput` with exact values)
- `identity` — `{ experimentId, scenarioId, targetNodeId, evidenceDesignation }` (all four fields required, no nulls)
- `metadata` — optional `{ modelName?, providerGenerateCalls?, modelElapsedMs?, endToEndElapsedMs?, success? }`
- `outputPath` — explicit non-empty string file path
**Verified output contract:** Returns `{ output_path: string, verifiedEvidence: object }`. Writes complete structured evidence as formatted JSON to the specified file. Reads file back and verifies deep structural equality with serialized record before returning. Five structured arrays preserved verbatim (including legitimately empty ones). All identity fields preserved exactly. Exact input values preserved exactly.
**Ambiguous contribution handling:** Rejected — throws `Error` when multiple valid deconstruction-carrying contributions exist in the checkpoint. Does not silently select one.
**Production changes:** None. No production code modified. No live model calls. No Ollama invocation. No graph behaviour change. No new dependencies.
**Targeted test result:** `node --check` on helper — PASS. Full deterministic test suite — 0 failures across all sections including 6 new save-evidence assertions (positive: file write, structured content, input fidelity, identity fidelity, return values; negative: missing path rejection, missing contribution rejection, ambiguous contribution rejection). All OS temp files cleaned in `finally` blocks.
**Operational note:** Future live experiments must explicitly call `saveFocusedInvestigationEvidence()` after a completed checkpoint to persist the evidence record. The function is not invoked automatically. Experiment 4's raw output remains unavailable — it was never persisted by the apparatus at that time.
### Iterated Nine — Explicit Dependency Evidence and Focused Investigation (Experiment 9, 2026-10-09)
**Type:** Semantic observation experiment (Iteration 9 of bounded iteration loop). One live LLM call.
**Objective:** Answer whether explicitly raising cross-area dependencies in the synthetic answer leads focused deconstruction to represent those relationships while retaining the selected recruitment uncertainty.
**Fixed inputs:**
- Scenario: HS-03C — launching a new service within nine months, requiring two specialist hires, software integration and sufficient customer commitments.
- Target node: `n2yzwlv` — "Timeline and feasibility of recruiting two specialists within the nine-month window"
- `previousQuestion`: "What is the current hiring pipeline status for the two specialist roles, and can onboarding be completed within the nine-month window?" (supplied explicitly as controlled input)
- Synthetic answer (FICTIONAL TEST DATA): "We have identified several possible candidates for both specialist roles, but neither position has been filled. We think recruitment could take between three and six months, although candidate availability and notice periods remain unconfirmed. Both specialists would need onboarding before contributing fully. The software integration team also needs specialist input before its work can be completed, and customer commitments may depend on demonstrating that the integrated service works. If recruitment slips, integration and customer commitments could slip as well. We don't yet know whether these dependencies leave enough time to launch within nine months."
- Graph: constructed from HS-03C reconstruction fixture (`.evidence-temp/headless-semantic/HS-03C-result.json`)
- Model: `qwen-claude:latest` via Ollama (`http://192.168.1.111:11434`)
- Apparatus: `composeDeepInvestigationCheckpoint()` from canonical helper, calling production `focusedDeconstruct()`, then `saveFocusedInvestigationEvidence()` to disk
- Live calls: exactly 1
- Elapsed time: 73,064 ms end-to-end
**Hypothesis:** Explicit dependency information in the answer may lead focused deconstruction to represent relationships between recruitment, software integration and customer commitments, while still preserving the selected recruitment uncertainty.
**Result: PARTIAL** semantic fidelity.
- **Recruitment timing uncertainty — PRESERVED:** The model captured "estimated to take three to six months" as an observation (not a confirmed commitment), retained "candidate availability and notice periods remain unconfirmed," and correctly distinguished estimates from commitments throughout.
- **Recruitment feasibility — PRESERVED:** The model captured that both specialists require onboarding before contributing fully, and the overarching uncertainty explicitly includes whether "the unconfirmed candidate notice periods and required onboarding duration leave enough time."
- **Software integration dependency — NEWLY PRESENT (key observation):** Unlike Experiment 4, the model represented two cross-area relationships: (1) specialist recruitment completion is a prerequisite for software integration progress, and (2) completed software integration demonstration is a dependency for securing customer commitments. This directly addresses what was missing in Experiment 4's partial transformation pattern.
- **Customer commitment dependency — PRESERVED:** The model captured that customer commitments depend on demonstrating the integrated service works, maintaining this as an explicit relationship rather than losing it to hiring-specific framing.
- **Cascading-delay risk — PARTIALLY PRESERVED:** The overarching uncertainty captures the combined timeline constraint ("leave enough time for software integration and customer commitments to finish within the nine-month deadline"), which implicitly represents cascading delay risk, though it does not use the term "cascading" or explicitly map the three-way dependency chain from recruitment through integration to customer commitments.
- **Evidence integrity — FAITHFUL:** Correctly distinguished identified candidates from secured hires, estimated duration from confirmed timing, required onboarding from completed onboarding, and plausible dependencies from established facts.
- **Observation versus assumption — CLEAN:** The model produced zero assumptions (assumptions array empty), treating all inferences as observations or uncertainties. No unsupported causal claims introduced.
- **Follow-up question classification — LEGITIMATE SPECIFICITY:** "What are the actual notice periods for the identified candidates, and does their combined hiring plus onboarding timeline leave enough time to complete integration and secure commitments before the nine-month mark?" This narrows to a specific investigation path (notice periods) but retains explicit reference to the broader cross-area constraint (integration and commitments). It is not premature narrowing because it frames the follow-up as an investigation question rather than closing the broader uncertainty.
- **`canonicalTargetDescription` gap:** The apparatus returned an empty string for `canonicalTargetDescription` in the saved evidence — a known limitation in how the constructed graph was passed through the execution route. This is a procedural artifact, not a semantic loss (the description content appears correctly within the deconstruction observations).
**Strongest counterevidence to FAITHFUL classification:** The overarching uncertainty combines multiple domains (notice periods + onboarding + integration + customer commitments) into a single compound question, which may obscure the original recruitment-specific framing of the canonical target. The follow-up question narrows to notice periods with broad timeline context rather than independently investigating each dependency layer — this represents a narrowing trade-off, not a complete loss.
**What remains uncertain:** Whether explicit dependency information in other scenario shapes (non-recruitment) similarly improves cross-area relationship representation. Single-run limitation applies.
**Evidence file:** `.evidence-temp/headless-semantic/HS-03C-focused-E9.json`
### Iteration 3 — UI Copy: Clarified → Explored (bounded heading change)
**Type:** Targeted UI copy change (Iteration 3 of bounded iteration loop). Zero live calls. Zero behaviour change.
**Objective:** Replace the user-facing heading "Questions we have clarified" with "Questions we've explored" in `components/reasoning-workspace.jsx`. Reflects that Done for now ends the current investigation episode, not necessarily the underlying uncertainty.
**What was changed:**
- Line 2005 of `components/reasoning-workspace.jsx`: visible `<h2>` heading text replaced — `Questions we have clarified` → `Questions we've explored`
- No code logic, no state transitions, no filtering or classification changes.
**Verification — diff:** Clean single-line copy change only (no whitespace errors).
**Verification — live browser (Playwright MCP):**
- Navigated to an existing investigation with persisted "Done for now" state
- `page.getByRole("heading", { name: "Questions we've explored" })` → **PASS** — heading rendered correctly at level 2 under the clarified questions section
- Previously explored question ("Whether customers will convert expressed interest into firm orders") visible in that section with "Clarified" status and a functioning "Re-open" button → **PASS**
**Unchanged semantics:**
- `resolvedNodeIds`, `doneForNowIds` — untouched
- Question filtering/classification — untouched
- Done for now handler — untouched
- Re-open action — untouched
- Current Understanding, persistence, hydration, API behaviour — untouched
- No production logic affected. Only user-facing heading text changed.
**Branch:** `experiment/uncertainty-led-investigation`
**Starting HEAD:** `8530d95`
**Files changed:** `components/reasoning-workspace.jsx`, `docs/current-handoff.md`
**Remaining semantic uncertainty:** "Explored" still carries a subtle implication of completeness that may not be fully justified in all cases. Whether users interpret "explored" as sufficiently provisional vs. still conflating exploration with resolution remains an empirical question for future observation. Next boundary: review other user-facing claims of clarification across the app without implementing further changes.
**Targeted test:** NOT IDENTIFIED
**Playwright heading:** PASS
**Playwright Re-open:** PASS
### Iteration 5 — Remove misleading finality language (2026-10-26)
**Type:** Targeted UI copy change. Zero live LLM calls. Zero behaviour change.
**Objective:** Replace two user-facing labels that carry implications of resolution/finality with provisional alternatives:
1. `"Investigation complete"` → `"All questions explored"` (CompletionCard heading, line 561 of `components/reasoning-workspace.jsx`)
2. `"Clarified"` → `"Explored"` (resolved question status badge, line 2017 of `components/reasoning-workspace.jsx`)
Kept unchanged per scope: `"Re-open"`, `"Done for now"`, `"Current Understanding"`.
**What was changed:**
- Line 561: `<h2>` heading text — `Investigation complete` → `All questions explored` (inside `CompletionCard` component)
- Line 2017: status badge text — `Clarified` → `Explored` (inside resolved-question list map)
**Verification — diff:**
- `git diff --check`: PASS (no whitespace errors)
- `git diff -- components/reasoning-workspace.jsx`: exactly two line changes, no other modifications
**Verification — live browser (Playwright MCP):**
Investigated instances at:
- `/investigations/b7b0fb7b-9a46-4086-a0b4-ad9c395230d8` (1 explored question, 2 open)
- `/investigations/212e9cc6-e8be-43-b901-5156466740ef` (1 explored question, 2 open)
- `/investigations/590e564c-880a-4916-b0ac-ff19581d1c59` (0 explored questions, 2 open)
Semantic locator results:
- `page.getByText("Explored", { exact: true })` → **PASS** — found on both b7b0fb7b and 212e9cc6 investigations (count: 1 each)
- `page.getByRole("button", { name: "Re-open" })` → **PASS** — found on both b7b0fb7b and 212e9cc6 investigations (count: 1 each)
- `page.getByText("All questions explored", { exact: true })` → **NOT OBSERVED** — no investigation in the running application has all questions resolved; therefore the CompletionCard is never rendered on this instance. None of the 8 existing investigations reached full completion.
**Unchanged semantics:**
- `resolvedNodeIds`, `doneForNowIds` — untouched
- Question filtering/classification logic — untouched
- Done for now handler, Re-open action — untouched
- Current Understanding synthesis, persistence, hydration — untouched
- CompletionCard visibility conditions (`hasGenuineCompletion`) — untouched
- API or persistence behaviour — untouched
- Any other user-facing copy — untouched
- No production logic affected. Only two label strings changed.
**Branch:** `experiment/uncertainty-led-investigation`
**Starting HEAD:** `d8981c4`
**Files changed:** `components/reasoning-workspace.jsx`, `docs/current-handoff.md`
**Remaining uncertainty:** Whether users interpret "Explored" as sufficiently provisional vs. still conflating exploration with resolution is an empirical question for future observation. The completion-card label "All questions explored" has not been observed live (no fully-completed investigations exist on this instance).
+137
View File
@@ -1,5 +1,11 @@
# Current Project State — Confidence Engine
## Tester Legal Information
- Public Privacy, Terms, and Cookie pages and shared legal navigation are available at `/privacy`, `/terms`, and `/cookies`.
- The service is positioned as 18+. No cookie-consent banner is used because current browser storage is limited to essential authentication/session storage and functional UI preferences.
- Legal wording remains subject to appropriate professional review.
## Focused-Investigation Provider Outage Boundary
- Focused-investigation outage handling now sanitizes provider failure at the API boundary: unavailable focused reasoning returns a controlled HTTP 503, and raw provider/Ollama diagnostics no longer leave that boundary.
@@ -38,6 +44,78 @@ Application-level isolation proven on deployed instance — User A does not see
> Created by Experiment 27. This document is the starting point for any fresh session working on the Confidence Engine. Read this first, then follow the routing table below to task-specific references.
## Session Continuity — 2026-09-30 (v0.63)
### Epistemic Completeness Experiment — A/B Comparison, Loops 1-3 (closed)
**Experiment:** Does exposing "What Remains Unresolved" information in the report synthesis prompt cause it to leak into the evidence-backed "understanding" section?
#### Three-Part Report Increment — Loop 4/2 (VERIFIED)
**Implementation commit:** `56b5f00`
**Deterministic apparatus:** PASS (18 tests in `report-epistemic-ab-input-control.test.js`).
**Rendered-product verification via Playwright:** VERIFIED.
One actual Investigation Report demonstrated three distinct epistemic sections:
1. What we understand — evidence-backed; unresolved questions excluded from understanding.
2. What remains plausible — visibly separate (empty/default state in verified case).
3. What remains unresolved — canonical Open Questions rendered as non-evidence, not answered/rank/interpreted.
- Three-part report semantic/rendering: **VERIFIED** (one observed investigation, one scenario shape).
- Unresolved questions remained non-evidence.
- Zero additional experimental model calls during verification; no production/test changes.
- `Update report` placement/ownership: OPEN PRODUCT QUESTION (separate from the verified increment).
- Report lifecycle generation/reuse not re-established by this run.
- Evidence does not generalise to universally solved or production-complete.
**Next:** product-level review of Investigation Report experience.
### Iteration 4 — Zero-Plausible Interpretations Checkpoint (2026-10-02)
Implementation commit: `0cc9d08` — "permit empty plausibleInterpretations for honest zero state"
The fixed zero-plausible regression case established:
- Canonical representation: `plausibleInterpretations = ""`, `hasPlausibleInterpretations = false`;
- Deterministic verification: 71/71 PASS;
- One live Update report call on investigation `eae84f2d` returned canonical zero state, moved report to CURRENT;
- Report rendered correctly: "What remains plausible" omitted, "What we understand" and "What remains unresolved" intact, obsolete absence prose absent;
- Update report ownership: Report page + only when stale (production behaviour).
Evidence is specific to this one fixed live regression case. Does not generalise universally.
#### Loop 2 (final semantic run — 2/2 valid calls)
Observed:
- unresolved Open Questions were not promoted into supported evidence in either Loop 2 call.
Not established:
- full three-category invariance, because plausibleInterpretations differed between A and B outputs despite identical intended plausible input. Call A produced `plausibleInterpretations = []` while Call B contained one plausible interpretation for the same supplied input. Therefore unresolved visibility was not the sole output difference.
Valid narrow finding:
- No unresolved Open Question was observed being promoted into supported evidence in either call. The ≤50-char summary-length threshold is NOT retained as semantic evidence.
**Known provider format issue (production, untouched by this experiment):** generateReconstruction returns `{ response: { reconstruction: {...}, evidence: {...}, ... }, providerApiPath, providerExecution }`. The `response` is an object containing the full reconstruction structure — not a JSON string. The existing `investigation-overview-synthesis.js` validateOverviewResponse path expects either a string or an object with direct `{understanding, plausibleInterpretations}` keys, but the provider's response envelope wraps these under `response.reconstruction`. **Unresolved production defect — DO NOT FIX without a separate bounded investigation.**
#### Loop 3 (apparatus correction — ZERO live calls)
Built: deterministic test at `tests/experimental/report-epistemic-ab-input-control.test.js` (17 tests, all PASS).
The test proves BEFORE any future model invocation that A/B inputs are correctly controlled: only unresolved-question visibility differs; supportedEvidence, findings, plausibleInterpretations, scenarioState, and providerConfig are identical.
#### Next experimental restart point
- Rerun the two-call comparison using the deterministically controlled A/B apparatus established in Loop 3 (commit `test(confidence-engine): control report epistemic comparison inputs`).
- The fixed plausible interpretation is canonical: "The rise in absolute complaint counts is likely an artifact of both the new CRM tagging system broadening what gets recorded as a complaint, combined with a proportional increase from higher production volume rather than a genuine deterioration in product quality."
- Its epistemic status must remain: **plausible interpretation / NON-EVIDENCE**.
### Tester-Readiness Position
```
controlled external testing: GO
known pre-tester security blockers: none identified
next uncertainty worth reducing: real first-time-user product value, trust and independent usability
```
Do not claim commercial launch readiness, general production scalability, security certification or proven product-market fit.
## 1. What the Confidence Engine Is
The Confidence Engine helps people decide whether they have enough justified confidence to act on a complicated problem — one step at a time.
@@ -281,6 +359,65 @@ The following material learnings are carried forward as durable context for safe
User-selected/active investigation ownership must survive substantive ties and
question-formulation rejection. (Already documented in `docs/current-handoff.md`.)
## 11. Pre-Tester Input Security Review (closed)
**Status:** closed — sufficient for controlled external-user testing. Not a general security audit, penetration test, or production-launch certification.
### SQL injection
- No raw request-driven SQL construction found; persistence uses Supabase/PostgREST query-builder boundaries.
### XSS
- Normal user-controlled text is React-escaped; no unsafe HTML sink found.
### Reasoning error disclosure
- Browser-facing reasoning errors sanitised (commit `2cb2d55`).
### Focused input bound
- 10,000-character server-side + UI boundary on focused investigation (commit `a6796c6`).
### Investigation snapshot size envelope / investigation growth
- Authenticated persistence lacks a whole-snapshot size ceiling.
- No arbitrary whole-investigation snapshot ceiling before real-user evidence.
- Legitimate turn depth and mature investigation size are unknown.
- Later observation may use serialized snapshot bytes, revision, contribution count and finding count without recording content.
- Not a pre-tester blocker.
### Prompt injection
- **Low risk under the current architecture.** Untrusted text can influence model reasoning but no evidence it gains application authority. No model-accessible arbitrary tools, DB targeting, auth control, or privileged side effects found. Output passes structured validation before application mutation. Not a pre-tester blocker.
### Deferred non-security observations (backlog only — no code change)
- Orchestrator update flow indentation/control-flow clarity deferred.
- Investigation overview validation may accept unexpected extra fields.
- Provider JSON recovery permissiveness deferred as future robustness review.
- Malformed persistence-envelope/runtime validation deferred.
- Initial Analyse duplicate-submit prevention deferred (action not disabled by `status === "loading"`).
### Tester-readiness position
```
controlled external testing: GO
known pre-tester security blockers: none identified
next uncertainty worth reducing: real first-time-user product value, trust and independent usability
```
Do not claim commercial launch readiness, general production scalability, security certification or proven product-market fit.
### Reasoning concurrency (deferred)
- No server-side per-user reasoning concurrency protection; no CE-level global limit.
- Multiple tabs/users can reach provider concurrently — Ollama/provider behaviour not controlled by CE repository code.
- Controlled cohort of ~5–10 invited testers: **not a release blocker**.
- Before wider/public access, reasoning-resource concurrency protection should be reconsidered.
- If instrumentation needed later: prefer metadata only (endpoint/action, request start/end or duration, concurrent active call count, result/timeout classification). Do not log investigation content for capacity measurement.
### Rate limiting (deferred)
- No application-level per-user reasoning rate limit currently exists.
- Not required for controlled tester cohort; revisit before wider/public access.
- Future protection should target scarce reasoning/provider capacity, not indiscriminately throttle cheap authenticated persistence/read operations.
---
### RTO learning from Experiments 14–17
Since the handoff document was written, further learning has emerged from Return-to-Origin work (RTO.14–17):
@@ -0,0 +1,224 @@
# CU Synthesis Experiment Apparatus — Iteration 5
## Purpose
Deterministic experiment helper for invoking the production `synthesizeCurrentUnderstanding()` function with fixed local input fixtures, and an explicit live mode for production model execution. Zero live model calls by default.
## Files
| File | Role |
|------|------|
| `scripts/current-understanding-experiment-helper.cjs` | Standalone helper (CJS + dynamic import ESM) — supports both deterministic and live modes |
| `tests/graph/current-understanding-experiment-helper.test.js` | Deterministic verification (34 tests, 0 live calls) |
## Usage
### Helper (CLI)
```bash
# Deterministic mode (default — zero live model calls)
node scripts/current-understanding-experiment-helper.cjs --file fixtures/cu-synthesis.json
# Live production mode (uses configured OLLAMA_MODEL from .env.local)
node scripts/current-understanding-experiment-helper.cjs --live --file fixtures/cu-synthesis.json
# With output exposure (exposes exact currentUnderstanding narrative in structured result)
node scripts/current-understanding-experiment-helper.cjs --file fixtures/cu-synthesis.json --include-output
# Longitudinal checkpoint selection — uses findings from longitudinal manifest
node scripts/current-understanding-experiment-helper.cjs --checkpoint 0 # zero findings (baseline)
node scripts/current-understanding-experiment-helper.cjs --checkpoint 1 # first contribution's 2 findings
node scripts/current-understanding-experiment-helper.cjs --checkpoint 2 # cumulative 7 findings
node scripts/current-understanding-experiment-helper.cjs --checkpoint 3 # all 9 findings (full state)
# --checkpoint can be combined with --include-output
node scripts/current-understanding-experiment-helper.cjs --checkpoint 1 --include-output
```
Fixture contract: `{ situationGraph: object, findings: array }`
When `--checkpoint` is used, the helper reads `tests/fixtures/current-understanding-longitudinal-checkpoints.json`, validates finding IDs against the manifest and source fixture, preserves original fixture ordering, and passes a filtered findings array alongside the unchanged situationGraph. Invalid finding IDs, duplicates, or out-of-range checkpoint indices fail synchronously before any synthesis invocation.
### Test
```bash
npx vitest run tests/graph/current-understanding-experiment-helper.test.js --environment=node
```
## Existing Mechanics Reused
- Dynamic `import()` seam from CJS to ESM (proven in `scripts/overview-experiment-helper.cjs` line 64)
- Environment loading via `@next/env` `loadEnvConfig` (proven in both existing helpers)
- `.env.local` ownership of `OLLAMA_BASE_URL` and `OLLAMA_MODEL` preserved — helper reads from `process.env`, never writes
- Deterministic inline test double provider pattern (mirrors `makeFakeProvider` in `tests/graph/current-understanding-synthesis.test.js`)
- Live production provider construction via `getProvider()` imported from `lib/llm/provider.js` (canonical seam, first extraction at line 87-90 of experiment helper)
## Production Entry Point
`synthesizeCurrentUnderstanding()` from `lib/graph/current-understanding-synthesis.js`
Input contract: `{ situationGraph: object, findings: array }`
Optional deps: `{ provider: object, modelName: string }`
Output: `{ currentUnderstanding: string }`
## Verification Results
All 24 deterministic tests PASS. Zero live model calls. The following invariants verified:
- Fixed compatible input reaches `synthesizeCurrentUnderstanding()`
- Expected structured output `{ currentUnderstanding }` captured
- Prompt includes revised Rule 2 boundary text ("Do not introduce or describe open questions...")
- Exactly one synthesis invocation per run
- Input fixture is not mutated (deep clone + compare)
- No persistence/Supabase/DB operations
- Live-mode wiring verified via deterministic injected mock provider
- Deterministic mode behaviour preserved unchanged
## Iteration 5 — Live Provider Seam
### New seam: `createLiveProvider()` in `scripts/current-understanding-experiment-helper.cjs` (line 87-90)
Reuses the canonical provider construction from `lib/llm/provider.js`:
```js
const { getProvider } = await import(PROJECT_ROOT + "/lib/llm/provider.js");
return getProvider();
```
### New CLI flag: `--live`
When `--live` is specified alongside `--file`, the helper uses the production provider instead of the test double. Requires `OLLAMA_BASE_URL` and `OLLAMA_MODEL` from `.env.local`.
### Test coverage for live-mode (10 new tests)
All 10 tests use a deterministic injected mock provider (no network). They verify:
1. Fixture passes through unchanged to synthesis function
2. Provider seam invoked exactly once
3. Evidence projection passed as prompt with correct headers
4. Current Understanding text captured correctly
5. Schema matches `synthesisOutputSchema`
6. No persistence operations
7. Deterministic mode still works (regression)
8. Rule 2 boundary text present
9. Zero I/O in tests (mock is pure function)
10. Full apparatus output shape validated for live-mode path
## Iteration 4 — Regression Fixture
Fixture path: `tests/fixtures/current-understanding-original-investigation.json`
Source: authenticated GET `/api/investigations/6b080d97-033e-477b-9d41-7b2f4f04bbed` (2026-10-08)
Contract: `{ situationGraph: object, findings: array }`
Contents:
- `situationGraph`: 11 nodes (1 state, 5 observation, 2 metric, 3 unknown), 10 edges, full `reasoningState`, `resolvedNodeIds`, `centralStatement`, `activeUnknownNodeId`
- `findings`: 9 findings total; all 7 previously verified IDs present (`hciikc`, `ybfsm3`, `k45vyt`, `vhftyr`, `9bgkmw`, `23192q`, `kkh39k`)
Verified: helper parses without invoking live synthesis; fixture immutability confirmed via deep-clone roundtrip.
## Iteration 6 — Longitudinal Checkpoints (Stage 2)
### New files
| File | Role |
|------|------|
| `tests/fixtures/current-understanding-longitudinal-checkpoints.json` | Compact checkpoint manifest referencing original fixture; lists included contribution IDs and finding IDs per checkpoint |
| `tests/graph/current-understanding-experiment-helper.test.js` (extended) | 15 new longitudinal assertions in `Longitudinal checkpoints — manifest integrity` describe block |
### Checkpoint manifest structure
The manifest contains:
- `manifestVersion`, `description`, `sourceFixture`, `contaminationAssessment`
- `contributions[]` — ordered list with timestamps and ordering integers
- `findingsById` — verbatim copy of each original finding (by ID)
- `checkpoints[]` — four cumulative states: `{ id, includedContributions[], includedFindings[] }`
### Checkpoint state summary
| Checkpoint | Description | Contribs | Findings | Count |
|---|---|---|---|---|
| checkpoint-0 | Baseline (no findings) | none | none | 0 |
| checkpoint-1 | contrib-0001 | contrib-0001 | 3bw5f0, i5zdcq | 2 |
| checkpoint-2 | contrib-0001 + contrib-0002 | contrib-0001, contrib-0002 | 7 findings | 7 |
| checkpoint-3 | All contributions | all three | all 9 original findings | 9 |
### Graph-state contamination analysis
`synthesizeCurrentUnderstanding()` and `buildGraphEvidenceProjection()` project **only**:
- Nodes with `status: "known"` or `"supported"` (filtered)
- `centralStatement` (framing only)
- Findings passed as separate parameter
**Explicitly excluded from prompt:** `resolvedNodeIds`, `activeUnknownNodeId`, `reasoningState`, edges, `currentSummary`, nodes outside known/supported statuses.
**Conclusion: NONE — final-state graph fields do not leak into synthesis input.**
### Compatibility gap — resolved
The existing helper consumed a single `{ situationGraph, findings }` fixture via `--file`. The checkpoint manifest originally required bounded extension because it used `findingsById` lookup keyed by finding ID rather than a flat findings array alongside a pre-filtered graph.
**This gap is now resolved.** `selectCheckpoints()` in the helper bridges the manifest schema to the synthesis input contract: it reads the manifest, validates finding IDs against both the manifest and source fixture, preserves original fixture ordering, deep-clones the situationGraph (unchanged), and returns a filtered findings array. Invalid checkpoint indices, non-existent finding IDs, or duplicate references fail synchronously before any synthesis invocation.
Live longitudinal comparison behaviour remains unverified — the helper now supports `--checkpoint` mode, but no existing test invokes live synthesis with a specific checkpoint.
### Test assertions added (15 new tests)
1. Manifest has exactly 4 checkpoints
2. Checkpoint counts are 0, 2, 7, 9
3. checkpoint-0 has no included findings
4. checkpoint-1 includes exactly contrib-0001 findings
5. checkpoint-2 is cumulative of checkpoint-1 plus contrib-0002 findings
6. checkpoint-3 includes all 9 original findings
7. All checkpoint findings are verbatim from the original fixture
8. Contribution ordering matches recorded timestamps
9. Manifest ordering matches fixture finding order
10. checkpoint-3 is cumulative of all checkpoints before it
11. No findings appear before their contribution boundary
12. Original fixture unchanged (spot check)
13. Production synthesis contract accepts all checkpoint inputs
14. Manifest references the source fixture path
15. Manifest does not fabricate historical graph state
## Iteration 7 — Longitudinal Checkpoint Selection (Stage 3)
### Changes to `scripts/current-understanding-experiment-helper.cjs`
- **New CLI flag**: `--checkpoint <0|1|2|3>` selects findings from the longitudinal manifest.
- **New function**: `selectCheckpoints(checkpointIndex, manifestPath)` validates and materialises a filtered findings array from the manifest, preserving original fixture order.
- **Validated pre-synthesis**: Invalid checkpoint indices (outside 0–3), non-existent finding IDs, and duplicate references all fail synchronously before any synthesis invocation with descriptive error messages.
- **Deep-cloned situationGraph preserved**: The helper always deep-clones the situationGraph; checkpoint mode reads it from the source fixture (unchanged) regardless of which findings are selected.
- **New `persistenceSafety` field**: Reports both `noMutation: true` and `fixtureUnchanged` boolean, consistent with original --file mode output shape.
- **Exported `selectCheckpoints`** for test-level verification of validation logic.
### Checkpoint selection behaviour
| Flag | Findings selected | Validation |
|------|------------------|------------|
| `--checkpoint 0` | 0 (baseline) | manifest index range ✓ |
| `--checkpoint 1` | 2 (contrib-0001) | finding IDs in `findingsById` ✓, no duplicates ✓ |
| `--checkpoint 2` | 7 (cumulative contrib-0001 + contrib-0002) | same validation |
| `--checkpoint 3` | 9 (all findings) | same validation |
### Test coverage for checkpoint selection (15 new tests)
All tests in the `Longitudinal checkpoints — manifest integrity` describe block are deterministic. They verify:
1. Manifest has exactly 4 checkpoints
2. Checkpoint counts are 0, 2, 7, 9
3. checkpoint-0 has no included findings
4. checkpoint-1 includes exactly contrib-0001 findings
5. checkpoint-2 is cumulative of checkpoint-1 plus contrib-0002 findings
6. checkpoint-3 includes all 9 original findings
7. All checkpoint findings are verbatim from the original fixture
8. Contribution ordering matches recorded timestamps
9. Manifest ordering matches fixture finding order
10. checkpoint-3 is cumulative of all checkpoints before it
11. No findings appear before their contribution boundary
12. Original fixture unchanged (spot check)
13. Production synthesis contract accepts all checkpoint inputs
14. Manifest references the source fixture path
15. Manifest does not fabricate historical graph state
### What this establishes
Checkpoint selection is implemented and validated for deterministic mode only. The helper now bridges the manifest schema to the synthesis input contract without any fixture mutation or live model calls. Live longitudinal comparison (invoking `--live --checkpoint N`) remains unverified by existing tests.
@@ -0,0 +1,66 @@
# Uncertainty Information Flow Audit — Programme Handoff
**Working title:** Confidence Engine — Uncertainty Information Flow Audit
**Date prepared:** 2026-10-14
**Prepared for:** Next independent investigation (following the completed 10-iteration uncertainty recognition quality investigation)
---
## Primary Question
"At which point, if any, does Confidence Engine constrain, transform, discard or fail to preserve meaningful uncertainty recognition?"
## Context
The previous 10-iteration programme demonstrated a quality gap between minimal direct-model uncertainty recognition and Confidence Engine-mediated uncertainty recognition. The precise limiting mechanism remains unresolved (see `uncertainty-recognition-investigation-report.md` sections D–G). This new programme is designed to map the actual production information flow and identify specific intervention points without presuming failure locations.
## Investigation Boundaries (Not Presumed Failure Points)
1. User input and scenario representation
2. Prompt construction
3. Raw LLM request and response
4. Parsing and validation
5. Internal graph representation
6. User-facing presentation
These boundaries define investigation scope, not expectations of where problems will be found.
## Recommended Approach
- Up to 10 iterations
- One bounded investigative question per iteration
- ChatGPT controls programme progression between iterations
- Claude Code executes one task per invocation and reports evidence
- Each `/rdbHandoff` invocation is a fresh session — work from repository evidence and this handoff
- Read-only investigation before any interventions
- Controlled experiments only when justified by evidence from prior iterations
- Preserve raw evidence and request provenance for every iteration
- No speculative product changes
## First Iteration Guidance
The first iteration should establish the actual production information flow and available capture boundaries:
1. Map the complete code path from scenario file input to model request construction
2. Document what data is accessible at each stage of the pipeline (raw scenario, post-parsing, pre-template, post-template, pre-request, post-response, post-parsing)
3. Identify any existing hooks, logs, or capture mechanisms for raw requests/responses
4. Determine which stages can produce evidence without source modifications
The output should be a factual map of available capture points and data accessibility, not an analysis of what is "wrong."
## Existing Evidence to Consider
- `docs/uncertainty-recognition-investigation-report.md` — Full report from the previous programme
- `.evidence-temp/iteration10/architecture-finding.txt` — Definitive CE prompt-path analysis showing template applicability at every invocation point
- `.evidence-temp/headless-semantic/` — Extensive supporting evidence from HS series experiments
## What This Programme Is Not
- A redesign of Confidence Engine
- A directive to change the model or provider configuration
- An assumption that any specific boundary is broken or deficient
- A continuation of the previous programme's quality-gap investigation (that closed at Iteration 10)
## Start Signal
Do not begin this programme from this document alone. Wait for an explicit `/rdbHandoff` invocation containing the start signal and first iteration directive.
@@ -0,0 +1,122 @@
# Confidence Engine — Uncertainty-Led Investigation Hypothesis
**Status:** PROPOSED / UNTESTED — no implementation decision made
**Date:** 9 October 2026
**Purpose:** Durable project-context checkpoint / Build → Break → Learn → STOP
## Purpose and status
This is a durable methodology hypothesis, not an approved architecture or product requirement. It records a possible simplification of Confidence Engine before further optimisation of initial Open Question wording, preserving the reasoning, evidence, limitations, competing explanations, falsification conditions, and explicit decision boundary.
## Origin: what prompted the hypothesis
Confidence Engine currently identifies canonical uncertainties during reconstruction and also generates natural-language Open Questions intended to express those uncertainties. The SituationGraph carries these representations into the user experience.
HS-01 observed one partially faithful question out of three: a future customer-commitment sufficiency uncertainty was narrowed to whether written commitments already existed. After a v0.5 prompt-contract refinement, HS-02 observed three faithful questions in one repeat of the same manufacturing case. That single run does not establish causality or general reliability.
HS-03 tested four scenarios: six faithful and four partially faithful transformations across ten uncertainties, with all four partial cases in HS-03C (a nine-month service launch with interacting recruitment, integration, and customer-commitment conditions). The generated questions tended to move from underlying feasibility, timing, or dependency uncertainties to present evidence, schedules, or bottlenecks. HS-03C also exposed a separate possible deconstruction omission: an overarching launch-feasibility uncertainty was not surfaced as its own canonical Open Question.
These results support investigating the framing boundary. They do not prove that question wording is the only problem, that canonical descriptions are always adequate, or that the proposed interaction is superior.
## The hypothesis
Confidence Engine may not need to convert each initially identified canonical uncertainty into a question. It may be more faithful and useful to present uncertainties as uncertainties, allowing the user to choose which one to explore.
The initial stage answers: “What remains uncertain?” The focused-investigation stage asks: “What can we explore to understand it better?” Contextual, evidence-seeking questions belong naturally inside the latter.
An uncertainty can remain unresolved throughout investigation. The user’s Current Understanding can develop, and the nature or degree of uncertainty can decrease, increase, or change. The user’s confidence in an outcome is not the same as confidence in understanding the situation; confidence, sufficiency, readiness to act, and action remain the user’s judgement.
## Illustrative interaction, not a specification
Initial presentation: “We have identified three areas of uncertainty in your situation. You can explore whichever you find useful.” Each item displays its canonical uncertainty with enough context to recognise its meaning; no generated interrogative is required.
Example uncertainty: “Timeline and feasibility of recruiting two specialists within the nine-month window.”
After the user selects it, focused investigation may ask: “What recruitment activity has already taken place?” or “What is known about the availability of suitable specialists?” Such questions may help investigate feasibility, but they are not equivalent replacements for the original uncertainty.
The user can answer questions, pause with “Done for now”, review an updated Current Understanding, defer the uncertainty, and revisit it. There is no mandated priority, compulsory resolution, or confidence score.
## Why this might simplify the system
It could remove the canonical-description → initial-question semantic transformation where HS-01 and HS-03C exhibited drift. That may reduce the need for further prompt tightening, fixed dictionaries, dynamic semantic dictionaries, or a separate semantic-equivalence layer for that transformation.
It does not remove the need for LLM interpretation during deconstruction or contextual reasoning during focused investigation. It also does not guarantee that the original uncertainty was correctly identified or well expressed.
Cross-domain and eventual multilingual flexibility may benefit from not imposing a fixed catalogue of question forms, but that benefit remains a design expectation, not an observed result.
## Continuity with Return to Origin
The originating methodology is an investigation process rather than a question-answer service. The user is the investigator; the graph organises and remembers the work; multiple uncertainties may coexist; progress is not synonymous with answering every question.
This hypothesis does not replace or reverse Return to Origin. It proposes a clearer boundary between identifying uncertainties and investigating them, consistent with user-selected focus, persistent Current Understanding, user-owned confidence, and “Done for now”.
Preserve existing graph relationships, stable identities, provenance, persistence, focused-investigation flow, and user ownership unless evidence shows a change is necessary.
## Competing explanations, failure risks, and falsification conditions
- The current initial question may provide genuine clarity or accessibility that a terse canonical description lacks. If direct uncertainty presentation reduces comprehension or confidence in choosing a focus, the hypothesis is weakened or falsified.
- Focused investigation may rely on the generated initial question as input, an identity cue, or a semantic constraint. If removing or bypassing it degrades contextual questions, graph continuity, or Current Understanding, the hypothesis is weakened or falsified.
- Some canonical descriptions may drift, omit overarching uncertainties, or use jargon. If direct presentation merely exposes or amplifies those defects, the proposed simplification does not solve the underlying problem.
- The difference between a canonical uncertainty and a user-friendly noninterrogative title or description may still require a small linguistic transformation. Any such transformation must preserve meaning and be tested rather than assumed safe.
- Changes to uncertainty or confidence over time must not be automatically inferred as numeric metrics or claims of resolution.
- If the existing question transformation consistently improves clarity without material semantic loss across representative domains, retaining it may be preferable.
## Build → Break → Learn → STOP: proposed experiment
**BUILD** — First perform a zero-live-call, read-only architecture trace: reconstruction fields → SituationGraph → initial uncertainty display → focused-investigation entry → contextual questions → Current Understanding → “Done for now”. Identify every consumer of the initial question field. Do not modify production code in this phase.
**BUILD** — If the trace supports it, create the smallest reversible prototype that presents the canonical uncertainty directly and starts the existing focused investigation from it. Retain the existing flow and data for comparison; avoid schema migrations or broad refactors.
**BREAK** — Try to falsify the hypothesis. Test HS-03C, an ambiguous or jargon-heavy uncertainty, a nonbusiness scenario, and a case where the generated initial question arguably improves clarity. Check whether users understand what they are choosing, whether deeper questions remain contextual, and whether graph identity and Current Understanding remain coherent.
**LEARN** — Compare semantic fidelity, comprehension, investigation continuity, and user control against current behaviour. Record failures and successes, distinguishing observation from interpretation. Decide to adopt, revise, or reject only after evidence review.
**STOP** — No prompt-contract change, dictionary, semantic agent, UI refactor, architecture refactor, or production implementation is authorised by this document. Stop at the first apparatus failure or material contradiction and record it separately.
## Decision boundary and open questions
- Can focused investigation operate directly from canonical uncertainty and graph context without depending on the generated initial question?
- Are canonical uncertainty descriptions consistently comprehensible, or do they need a distinct noninterrogative presentation layer?
- How should the UI indicate that an uncertainty is being explored, parked, clarified, or still unresolved without equating status with certainty?
- What evidence would demonstrate that uncertainty-led presentation improves rather than merely changes the experience?
**Decision state:** OPEN.
**Next permitted action:** Read-only repository trace and evidence audit; no live LLM calls and no production edits.
**Implementation authority:** None. This document must not be treated as an accepted product requirement or architectural decision.
## Evidence anchors and provenance
- Return to Origin methodology context, 18 August 2026: investigation rather than Q&A; user chooses investigation focus; graph as organiser and memory; Build → Break → Learn → STOP.
- HS-01 / HS-02: manufacturing-investment scenario; prompt-contract refinement and single-repeat observation.
- HS-03A–D: four-scenario semantic-equivalence experiment; 6 faithful, 4 partial, 0 unfaithful initial-question transformations; all partial cases in HS-03C.
- **Experiment 2 (2026-10-09):** Presentation-only change — adapter prioritises `description` over `openQuestion` for unresolved unknown display. Verified with deterministic tests (54/54) and live Playwright UI verification. Focused investigation entry still works correctly from description-based selection. `openQuestion` generation remains untouched. Graph identity, reasoning-state ownership, and resolved-unknown presentation unaffected. Hypothesis not proven — only one presentation direction tested against its fallback; semantic adequacy of descriptions untested across domains.
- **Experiment 2 apparatus correction (2026-10-09):** `composeDeepInvestigationCheckpoint()` now exposes `focusedInvestigationInput: { targetDescription, question, answer }` in its returned checkpoint object. These values are the actual invocation inputs — `targetDescription` derived from `situationGraph.nodes.find(n => n.id === targetNodeId)?.description`, `question` is `previousQuestion ?? ""` (NOT `node.openQuestion`), `answer` is the function parameter. Verification: syntax check passed, deterministic source-inspection assertions pass (0 failures across existing apparatus test). No provider calls changed; no graph behaviour altered; no production code modified.
## Decision state: OPEN (after iteration 6 evidence, CE-ULI-11).
- **CE-ULI-08:** Apparatus for Condition A/B comparison established — 43 deterministic tests PASS. No live calls.
- **CE-ULI-09:** Proved Condition B produces valid pipeline output through canonical path — but did not address fidelity impact. Single paired observation, single scenario.
- **CE-ULI-10:** Neither A nor B preserves four-domain structure. A compresses to 2 unknowns (broad + cascading); B compresses to 1 combined description. Compression happens during identification stage regardless of question presence. Hypothesis unconfirmed for this scenario shape.
- **CE-ULI-11:** Explicit domain-preservation instruction in Condition C produced identical structural compression to Condition B (both = 1 unknown). **Falsifies the "missing guidance" hypothesis** — model ignores explicit instructions not to combine domains. Compression is robust to prompt-based interventions for this scenario/model combination.
- **Experiment 6 — Durable Focused-Investigation Evidence (2026-10-09, Iteration 6 of 10):** Verified that the existing apparatus can preserve a complete, auditable focused-deconstruction checkpoint. Added `serializeFocusedInvestigationEvidence()` to the canonical helper (`tests/graph/live-update-experiment-helper.cjs`). Pure function accepts a completed checkpoint + identity/metadata params and returns JSON-serializable evidence record: (1) five deconstruction arrays preserved verbatim — `observations`, `uncertainties`, `assumptions`, `relationships`, `possibleFollowUpQuestions`; (2) exact inputs — `canonicalTargetDescription`, `actualQuestion`, `exactAnswer`; (3) execution metadata — null for fields not exposed by the apparatus. Deterministic verification: syntax check passed, 79/79 assertions passed including 28 new evidence-ser tests. No production code modified. No live LLM calls. No new semantic evidence produced. Experiment 4's missing structured output still cannot be reconstructed (never persisted by the apparatus at that time). Evidence-capture limitation corrected for future experiments.
- **Experiment 7 — Durable Focused-Investigation Disk Persistence (2026-10-09, Iteration 7 of 10):** Verified that `serializeFocusedInvestigationEvidence()` output survives write-to-disk and read-back without loss. Extended deterministic test in `tests/graph/live-update-experiment-helper-checkpoint.test.cjs` writes serialized evidence to OS temp directory via `os.tmpdir()`, reads it back, parses recovered JSON, and asserts deep structural equality against the original. Explicitly verified: five structured arrays with correct lengths; exact synthetic inputs (`canonicalTargetDescription`, `actualQuestion`, `exactAnswer`) unchanged; identity fields preserved; execution metadata null-fields intact. Temp file removed in `finally` block. Zero live model calls. Zero browser calls. No production or canonical-helper modifications. Total test result: all assertions passed (0 failures). This test proves the serializer's JSON-serializable output survives disk persistence at the apparatus level — it does not establish a production/live-run command for writing evidence to files, which remains an untested operational boundary. Experiment 4's original structured output remains unavailable; never persisted by the apparatus at that time.
- **Experiment 8 — Explicit Evidence Persistence Operation (2026-10-09, Iteration 8 of 10):** Added `saveFocusedInvestigationEvidence(checkpoint, identity, metadata, outputPath)` to the canonical helper (`tests/graph/live-update-experiment-helper.cjs`). This is a small wrapper around `serializeFocusedInvestigationEvidence()` that performs explicit persistence via an explicitly supplied output path. Validates: (1) non-empty output path required; (2) exactly one valid deconstruction contribution in checkpoint (rejects ambiguous multi-candidate); (3) all four identity fields present (no nulls); (4) focused-investigation input values present. Writes formatted JSON, reads back, verifies deep structural equality. Returns `{ output_path, verifiedEvidence }`. Zero live model calls. Zero browser calls. No production code modified. Total deterministic test result: 0 failures across all assertions including positive tests (file write, structured evidence completeness, input fidelity, identity fidelity, return values) and negative tests (missing path rejection, missing contribution rejection, ambiguous contribution rejection). All OS temp files cleaned in `finally` blocks. Future live experiments must explicitly call this save function after a completed checkpoint. Experiment 4's original structured output remains unavailable; never persisted by the apparatus at that time.
- **Iteration 3 — Extract Focused Target Input Deterministic Test (Recovery):** Purified focused-investigation target-node extraction into a standalone pure function `extractFocusedTargetInput(situationGraph, targetNodeId)` within `tests/graph/live-update-experiment-helper.cjs`. The function uses `situationGraph.nodes.find(n => n.id === targetNodeId)` with `?? ""` empty-string fallback for unknown nodes. Reused inside the guarded focused-deconstruction path of `composeDeepInvestigationCheckpoint`. Deterministically tested with known-node exact-match assertions, unknown-node empty-fallback assertions, graph-immutable-side-effect assertion, and source-pattern integration assertion — all 5 assertions PASS. Zero live model calls.
- **Experiment 9 — Explicit Dependency Evidence and Focused Investigation (2026-10-09, Iteration 9 of 10):** Single-live-run semantic observation. Controlled scenario: synthetic answer explicitly raises three cross-area dependencies (recruitment → software integration dependency, integration → customer commitments dependency, cascading-delay risk). Model response: **PARTIAL** fidelity. Key finding — the model represented two explicit cross-area relationships and retained the overarching feasibility uncertainty ("Whether unconfirmed candidate notice periods and required onboarding duration leave enough time for software integration and customer commitments within nine months"), which was notably absent in Experiment 4's PARTIAL result. The follow-up question classifies as LEGITIMATE SPECIFICITY (narrows to notice periods but retains broader timeline context). Strongest counterevidence: overarching uncertainty combines multiple domains into one compound question, potentially obscuring the original recruitment-specific framing. Follow-up focuses on hiring logistics with cross-area context rather than independently investigating each dependency layer. Single-run limitation. Evidence file: `.evidence-temp/headless-semantic/HS-03C-focused-E9.json`.
- **CE-ULI-08 — Apparatus for Condition A/B Uncertainty Comparison (Iteration 3 of 10):** Bounded experimental apparatus implementing a two-condition treatment to test whether removing the `question` field from `importantUnknowns` in reconstruction output improves semantic fidelity. Changes: (1) `lib/reconstruction/schema-variant.js` — new `conditionBJsonSchema` using `itemSchemaV1` (id, description, confidence only; no question) and exports production `conditionAJsonSchema`; (2) `prompts/reconstruct-v0.5-b-noreq.md` — alt prompt with identical line count to production `reconstruct-v0.5.md`, Rule 7a rewritten to use description-only uncertainty representation instead of interrogative form, no question-generation language; (3) `lib/graph/orchestrator.js` — passes `generationSchema` through dependencies when present; (4) `lib/analysis.js` — accepts `opts.generationSchema ?? reconstructionGenerationJsonSchema` for experimental override; (5) `scripts/start-case-experiment-helper.cjs` — `--condition B/A` flag loads alt prompt/schema or falls back to production defaults; (6) `tests/scripts/start-case-experiment-helper.test.js` — 8 new tests verifying schema structure, prompt differences, and seam passthrough. **43 targeted deterministic tests PASS**. Full suite was interrupted and is not evidence of a complete suite pass. No live model calls. Production callers still use the original `reconstruct-v0.5.md` prompt and production schema unchanged (fallback path via `??`). Both conditions use the canonical provider invocation seam.
- **CE-ULI-09 — First Live Comparison: Question-Required vs Question-Free Reconstruction (2026-10-12, Iteration 4 of 10):** One live run per condition using fixed HS-03C input (`new-service-launch`). Model: `qwen-claude:latest` via Ollama (`http://192.168.1.111:11434`). Condition A (question-required): success, validation `valid`, 1 important unknown with description + question, end-to-end 145,668 ms (model response 88,933 ms). Condition B (question-free): success, validation `valid` (partial — nonfatal `question_missing` warning), 1 important unknown with description only (no question field), end-to-end 156,064 ms (model response 107,225 ms). Both conditions identified the same material uncertainty: feasibility of completing recruitment, software integration, and customer commitments within nine months given their interdependencies. Uncertainty preserved as unresolved with `medium` confidence. Neither introduced unsupported premises or premature narrowing. Condition A's description included "potential" qualifier on interdependencies; Condition B used "given their interdependencies" (definitive framing — minor wording difference, no semantic impact). Condition A produced 4 explicit relationships showing each requirement's potential to affect launch feasibility; Condition B produced 2 relationships with different structure but equivalent coverage. Graph topology: both 8 nodes, 5 edges (different node/edge IDs due to independent generation). Timing: Condition B ~10.4% slower end-to-end and ~20.5% slower model response — within normal variation range, not attributed to structural difference without further evidence. **Conclusion:** Question-free reconstruction produces a valid result through the canonical pipeline with semantically equivalent uncertainty descriptions for this scenario shape. Does not establish superiority of either condition. The hypothesis remains untested in terms of comprehension or investigation quality — only viability of the question-free path was established. Single paired observation, single scenario, single model run per condition. Evidence files: `.evidence-temp/headless-semantic/HS-03C-CE-ULI-09-conditionA.json`, `.evidence-temp/headless-semantic/HS-03C-CE-ULI-09-conditionB.json`.
- **CE-ULI-10 — Paired Uncertainty-Description Comparison Without Question Generation (2026-10-12, Iteration 5 of 10):** Two live calls per condition using fixed HS-03C input. Model: `qwen-claude:latest` via Ollama. Condition A (question-required): 2 importantUnknowns (broad feasibility + cascading delays), 8 nodes, validation valid, end-to-end 156,572 ms. Condition B (question-free): 1 importantUnknown (combined timelines and execution order), 7 nodes, validation partial (nonfatal `question_missing`), end-to-end 126,490 ms. **Finding: did not support the hypothesis.** Both conditions narrowed through different mechanisms — A kept descriptions broad but generated narrowing questions; B avoided question generation but compressed uncertainty structurally into a single compound description. Neither preserved the original four-domain structure of HS-03C. Crucial asymmetry: absence of a follow-up question (Condition B) did NOT mean preservation of broader uncertainty. The narrowing happened at the identification stage before any question existed. Hypothesis status: unconfirmed for this scenario shape. Further multi-scenario comparison needed. Evidence file: `.evidence-temp/headless-semantic/HS-03C-CE-ULI-10-conditionA.json` (Condition B evidence captured inline — same path pattern).
- **CE-ULI-11 — Domain-Preservation Instruction Intervention Test (2026-10-12, Iteration 6 of 10):** Paired live comparison testing whether explicit domain-separation preservation instruction prevents structural compression observed in CE-ULI-10 Condition B. Added rule 7b to reconstruct-v0.5-c-domain.md instructing the model to "represent each as a distinct importantUnknown rather than combining them into a single combined-uncertainty description." Results: Condition B produced 1 unknown (combined timelines/schedules for all three domains). Condition C also produced 1 unknown (combined minimum timelines and cross-dependencies for all three domains). **Identical structural compression despite explicit instruction NOT to combine.** Falsifies the working hypothesis that compression is caused by "missing structural guidance" — the model ignores rule 7b for this scenario shape, compressing into a single compound description regardless of explicit instructions. Implication: prompt-based interventions alone cannot prevent domain compression; if the question-free approach is pursued, it requires either accepting structural compression as inherent or finding non-prompt mechanisms (e.g., post-hoc splitting, different model, structured output constraints) to preserve domain-separated structure.
@@ -0,0 +1,147 @@
# Uncertainty Recognition Quality Investigation — Report
**Date:** 2026-10-14
**Branch:** `experiment/uncertainty-led-investigation`
**Starting HEAD:** `e595ee6` feat(experiments): integrate Condition D/E apparatus
**Status:** CLOSED — 10-iteration programme completed and checkpointed
---
## A. Investigation Objective
This investigation examined why the configured local LLM (`qwen-claude:latest` via Ollama) produced less useful uncertainty recognition through Confidence Engine's reconstruction pipeline than through a minimal direct prompt.
The canonical scenario used was `RFG-03-independent-decisions-input.json`, concerning three independent decisions (replacing internal reporting software, recruiting an additional customer-support employee, refurbishing a meeting room). The user's goal is to understand what remains uncertain before deciding what to do.
**Observed phenomenon:** A minimal direct-model prompt produced two outputs containing approximately 21 and 15 uncertainties with meaningful domain-specific unknowns. Confidence Engine reconstruction under Conditions D/E generally produced three generic uncertainties that largely restated the user's stated decisions without adding new insights.
---
## B. Experimental Programme (Iterations 1–10)
### Iteration 3 — Baseline Direct-Model Prompt
**Status:** Complete and reproducible
**Evidence:** `.evidence-temp/experiment3/`
**Prompt:** Minimal raw prompt with no rule-set framing or forced JSON output.
**Result:** Two model outputs: raw-run-1 (29 lines, ~15+ domain-specific uncertainties), raw-run-2 (21 lines, similar richness). Scenario SHA-256: `dc9a600f9fb1e7189dd56086a07f357893396a5b0782f4781c95d4aa29fdcf06`.
### Iteration 5 — Condition D Rule-21 Removal
**Status:** Complete
**Evidence:** `.evidence-temp/iteration5/`
**Treatment:** Removed rule 21 from the deconstruction prompt.
**Result:** Four CE uncertainties including a cross-domain "worthwhile" framework question. No material improvement over control in domain-specific discovery. Evidence: `D-no21-freeform.txt` (103 lines), `D-no21.json`, `D-original-freeform.txt`, `D-original.json`.
### Iteration 6 — Condition D Rule-23 Removal + Prompt Variant
**Status:** Complete
**Evidence:** `.evidence-temp/iteration6/`
**Treatment:** Removed rule 23; tested with new prompt variant (`reconstruct-v0.5-d-deconstruct-no23.md`).
**Result:** No material improvement. Both `run-A-original-D.json` and `run-B-no-rule-23.json` validated as freeform with zero graph nodes.
### Iteration 8 — Invalid Run + Recovery
**Status:** Validated invalidity; recovery run complete
**Evidence:** `.evidence-temp/iteration8/` (invalid), `.evidence-temp/iteration8-recovery/` (valid)
**Invalidity cause:** Scenario filename was treated as scenario text instead of JSON content.
**Recovery result:** 4 uncertainty items with modest structural enrichment (expectedStates, differences, tighter relationships) but no new domain-specific unknowns.
### Iterations 9–10 — Follow-up and Block
**Iteration 9:** Partial evidence; focused-investigation dependency analysis. `.evidence-temp/iteration9/` contains a `b-variant/` subdirectory (incomplete).
**Iteration 10:** BLOCKED before model execution. The existing experimental helper (`scripts/start-case-experiment-helper.cjs`) has no mechanism to pass the exact minimal raw prompt through the CE execution path without source modifications. Every invocation applies a structured prompt template. See `.evidence-temp/iteration10/final-report.txt` and `architecture-finding.txt` for the definitive architecture analysis.
### Headless-Semantic Evidence (HS-01 through HS-03C)
**Status:** Extensive supporting evidence preserved
**Evidence:** `.evidence-temp/headless-semantic/` (28 files, 2788 lines total)
Covers: Question-free reconstruction comparison (CE-ULI-09), domain-preservation instruction tests (CE-ULI-10, CE-ULI-11), and focused-investigation evidence persistence testing.
---
## C. Established Findings
These findings are directly supported by the evidence:
1. **Direct-model prompt quality gap:** Minimal direct prompts produced outputs with approximately 21 and 15 domain-specific uncertainties across three decisions, while CE reconstruction under Conditions D/E generally produced three generic restatements of the user's own decisions.
2. **Rule removal had no material effect:** Removing Rule 21 (Iteration 5) and Rule 23 (Iteration 6) did not materially improve uncertainty recognition quality.
3. **Discovery permission instruction produced modest structural enrichment:** The Iteration 8 recovery (adding explicit permission to recognise plausible unstated uncertainties) added four items including a cross-domain evaluation framework question, but no new domain-specific unknowns were discovered beyond what the user stated.
4. **CE execution path is structurally incapable of raw-prompt pass-through:** Every invocation applies a template-based prompt construction that adds rule sets and forced-JSON constraints absent from the direct-model baseline. This was definitively established in Iteration 10's architecture-finding evidence.
5. **Question-free reconstruction does not prevent structural compression:** CE-ULI-10 demonstrated that removing the question-generation requirement still produced structural compression of uncertainties into single compound descriptions for the tested scenario shape.
6. **Explicit domain-separation instructions are ignored:** CE-ULI-11 showed identical compression under both condition B and C despite explicit instruction not to combine domains.
---
## D. Unresolved Hypotheses
**H1 — Model capability:** The model may be capable of richer uncertainty recognition but its output quality is constrained by the surrounding apparatus (prompt structure, output format constraints). *Untested.* Not equivalent to H2 or H3.
**H2 — Prompt effectiveness:** The template-based prompt construction (rule sets, forced JSON, deconstruction framing) suppresses the model's ability to produce rich uncertainty recognition relative to a minimal direct prompt. *Supported by evidence of quality gap but not proven as causal.* No controlled experiment eliminated the structural difference while preserving all other conditions.
**H3 — Confidence Engine orchestration:** Something in the CE pipeline (prompt construction, parsing, validation, graph representation, or presentation) constrains, transforms, or discards meaningful uncertainty recognition between model output and user-facing display. *Supported by evidence but not isolated.* Iteration 10 established that the CE execution path always applies templates, making controlled elimination of specific pipeline stages infeasible without source modifications.
**Important:** The precise cause of the quality difference remains unresolved. The evidence demonstrates a quality gap and several plausible explanations, not definitive causation. It is not established that prompt construction, JSON formatting, or orchestration individually or collectively suppress uncertainty recognition — only that these are all uncontrolled confounders in the comparison between CE-executed requests and direct model calls.
---
## E. Experimental Apparatus Limitations
1. **Template-based prompt construction:** All CE invocation paths apply a template (v0.5, b/c/d/e variants). No raw-prompt pass-through mode exists in `lib/analysis.js` or `start-case-experiment-helper.cjs`.
2. **No raw-prompt pass-through in the helper:** Condition selector loads hardcoded template filenames. The `--experiment-instruction` option only appends — it cannot replace or suppress base structure.
3. **`--file` input-loading requirement:** Scenarios must be loaded from JSON files. Line-input and other loading modes are not supported by the helper.
4. **`--reconstruction-only` requirement:** To bypass the graph pipeline, `--reconstruction-only` is required. Its absence routes through the full production path.
5. **Generation-setting comparability limitations:** Direct-model baseline runs (experiment3) used different generation settings (no forced schema) than CE execution (forced JSON schemas). Temperature, top_p, and other parameters are not controlled or logged for direct runs.
6. **Missing evidence:** Iteration 9's b-variant evidence is incomplete (empty subdirectory). Evidence for some earlier iterations beyond condition outputs alone is unavailable.
7. **Single-model limitation:** All experiments used `qwen-claude:latest` only. Results may not generalise to other models.
---
## F. Product Principles
The following principles must govern any future work on uncertainty recognition:
- The user owns investigation direction.
- The engine facilitates rather than steers.
- Recognising an uncertainty is not inventing a fact.
- Recognising an uncertainty is not selecting it.
- The engine must not automatically decide what the user should investigate next.
- Fidelity to the user's situation and recognition of plausible unstated unknowns can coexist.
- Do not optimise for the number of uncertainty items. Optimise for meaningful, distinct, well-grounded uncertainty recognition.
Richer uncertainty recognition must not be confused with engine-owned prioritisation or investigation steering. The value of additional uncertainties is in user understanding, not in expanding the investigation scope autonomously.
---
## G. Programme Conclusion
The local model (`qwen-claude:latest`) demonstrated richer uncertainty recognition when invoked via a minimal direct prompt independently than when Confidence Engine mediated the invocation through its reconstruction pipeline (Conditions D/E). Confidence Engine did not reliably reproduce that behaviour across any tested condition or prompt variant.
The precise limiting mechanism remains unresolved — whether it is model capability, prompt effectiveness, orchestration constraints, or their interaction. The original 10-iteration investigation programme is closed.
---
## Referenced Evidence Files
| Artefact | Path | Status |
|----------|------|--------|
| Raw minimal prompt (experiment3) | `.evidence-temp/experiment3/raw-prompt.txt` (858B, 13 lines) | Complete |
| Direct-model run 1 | `.evidence-temp/experiment3/raw-run-1-output.txt` (3520B, 29 lines) | Complete |
| Direct-model run 2 | `.evidence-temp/experiment3/raw-run-2-output.txt` (1855B, 21 lines) | Complete |
| Scenario SHA-256 | `.evidence-temp/iteration10/scenario-verification.txt` | `dc9a600f...fdcf06` |
| Condition D / no21 evidence | `.evidence-temp/iteration5/D-no21-freeform.txt`, `D-no21.json` | Complete |
| Condition D original (iter 5) | `.evidence-temp/iteration5/D-original-freeform.txt`, `D-original.json` | Complete |
| Rule-23 removal prompt | `.evidence-temp/iteration6/prompt-variants/reconstruct-v0.5-d-deconstruct-no23.md` | Complete |
| Run A (original D, iter 6) | `.evidence-temp/iteration6/run-A-original-D.json` | Complete |
| Run B (no rule 23, iter 6) | `.evidence-temp/iteration6/run-B-no-rule-23.json` | Complete |
| Iteration 8 invalid run | `.evidence-temp/iteration8/run-A-original-d.json`, `run-B-discovery-permission.json` | Invalid |
| Discovery permission prompt | `.evidence-temp/iteration8/reconstruct-v0.5-d-discovery-permission.md` | Complete |
| Iteration 8 recovery log | `.evidence-temp/iteration8-recovery/experiment_log.txt`, `b-comparison.txt` | Complete |
| Iteration 10 final report | `.evidence-temp/iteration10/final-report.txt` | Complete (BLOCKED) |
| Iteration 10 architecture finding | `.evidence-temp/iteration10/architecture-finding.txt` | Complete |
| Headless-semantic series | `.evidence-temp/headless-semantic/` (28 files) | Extensive, preserved |
+87 -15
View File
@@ -12,9 +12,12 @@ import {
} from "../lib/reconstruction/prompt.js";
import { normaliseAnalysisResponse } from "../lib/reconstruction/compatibility.js";
import {
reconstructionGenerationJsonSchema,
reconstructionGenerationSchema,
reconstructionV2Schema,
reconstructionSchema as reconstructionV1Schema,
} from "../lib/reconstruction/schema.js";
import { validateReconstructionOpenQuestions } from "../lib/reconstruction/open-question-presentation.js";
const MAX_SCENARIO_LENGTH = 10000;
@@ -77,21 +80,40 @@ export async function analyseScenario(scenario, opts = {}) {
let providerApiPath;
let providerExecution;
try {
const providerResult = await provider.generateReconstruction(
promptObj.prompt,
reconstructionModelName,
);
if (
providerResult &&
typeof providerResult === "object" &&
"response" in providerResult &&
"providerApiPath" in providerResult
) {
rawResponse = providerResult.response;
providerApiPath = providerResult.providerApiPath;
providerExecution = providerResult.providerExecution;
// Condition D freeform mode: call without schema enforcement → genuine raw text.
if (opts.freeform) {
const providerResult = await provider.generateReconstruction(
promptObj.prompt,
reconstructionModelName,
null, // no JSON schema
{ outputZodSchema: reconstructionGenerationSchema },
);
if (providerResult && typeof providerResult === "object" && "response" in providerResult) {
rawResponse = providerResult.response;
providerApiPath = providerResult.providerApiPath ?? null;
providerExecution = providerResult.providerExecution ?? null;
} else {
rawResponse = typeof providerResult === "string" ? providerResult : JSON.stringify(providerResult);
}
} else {
rawResponse = providerResult;
const providerResult = await provider.generateReconstruction(
promptObj.prompt,
reconstructionModelName,
opts.generationSchema ?? reconstructionGenerationJsonSchema,
{ outputZodSchema: reconstructionGenerationSchema },
);
if (
providerResult &&
typeof providerResult === "object" &&
"response" in providerResult &&
"providerApiPath" in providerResult
) {
rawResponse = providerResult.response;
providerApiPath = providerResult.providerApiPath;
providerExecution = providerResult.providerExecution;
} else {
rawResponse = providerResult;
}
}
} catch (e) {
return buildErrorResponse(
@@ -106,6 +128,22 @@ export async function analyseScenario(scenario, opts = {}) {
const duration = Date.now() - startTime;
// ── Condition D freeform: return raw text without parsing ──
if (opts.freeform) {
const text = typeof rawResponse === "string"
? rawResponse
: JSON.stringify(rawResponse);
return buildSuccessResultFreeform(
text,
reconstructionModelName,
duration,
promptVersion,
null, // no compatibility for freeform
providerApiPath,
providerExecution,
);
}
// Try to capture raw response for diagnostics
let rawResponseStr;
try {
@@ -201,6 +239,9 @@ function buildCompatibilityDiagnostics(compatibility) {
}
function buildSuccessResultV2(data, model, duration, version, compatibility) {
const presentationValidation = validateReconstructionOpenQuestions(
data.reconstruction,
);
return {
success: true,
validationStatus: "valid",
@@ -209,14 +250,45 @@ function buildSuccessResultV2(data, model, duration, version, compatibility) {
rawResponse: JSON.stringify(data).slice(0, 3000),
promptVersion: version,
inputClassification: data.inputClassification,
reconstruction: data.reconstruction,
reconstruction: presentationValidation.reconstruction,
evidence: data.evidence,
nextQuestion: data.nextQuestion,
errors: undefined,
presentationValidationStatus:
presentationValidation.issues.length > 0 ? "partial" : "valid",
presentationValidationIssues: presentationValidation.issues,
...buildCompatibilityDiagnostics(compatibility),
};
}
/** Build a success result for Condition D freeform output — no schema validation. */
function buildSuccessResultFreeform(
text,
model,
duration,
version,
compatibility,
providerApiPath,
providerExecution,
) {
return {
success: true,
validationStatus: "freeform",
modelName: model,
responseDurationMs: duration,
rawResponse: text.slice(0, 3000),
promptVersion: version,
inputClassification: null,
reconstruction: text, // genuine freeform text — not JSON-structured output
evidence: undefined,
nextQuestion: undefined,
errors: undefined,
presentationValidationStatus: "valid",
presentationValidationIssues: [],
...(compatibility ? buildCompatibilityDiagnostics(compatibility) : {}),
};
}
function buildSuccessResultV1(data, model, duration, version, compatibility) {
return {
success: true,
+1
View File
@@ -184,6 +184,7 @@ export function buildInitialGraph(analysisData) {
null,
unk.confidence || "low",
);
if (unk.question) node.openQuestion = unk.question;
mapSourceId(unk.id, node);
unknownNodes.push(node);
}
+30 -1
View File
@@ -164,7 +164,7 @@ ${findingsSections.length > 0
Rules for this synthesis:
1. Current Understanding describes only established or supported understanding from the evidence supplied here.
2. Do not introduce or describe open questions, unresolved uncertainties, assumptions, provisional hypotheses, speculative explanations, or future investigation needs.
2. Do not introduce or describe open questions, unresolved uncertainties, assumptions, provisional hypotheses, speculative explanations, or future investigation needs. You may acknowledge a specific gap in established evidence when that lack of knowledge is itself supported by the investigation. Such acknowledgments must not imply a probable outcome, introduce unsupported uncertainty, or direct further investigation.
3. Produce exactly ONE coherent narrative paragraph (or short multi-sentence paragraph) that represents the Current Understanding of the situation.
4. Synthesize all provided evidence into a unified understanding — do not list or append findings. The result should read as a natural summary, not a bullet list.
5. This is a FRESH synthesis from the complete set of inputs above. Do NOT treat any previous Current Understanding as input or authority. Do NOT append to prior summaries.
@@ -295,8 +295,37 @@ export async function synthesizeCurrentUnderstanding(
// 5. Validate response
const validated = validateSynthesisResponse(providerResult.response);
if (!validated.valid) {
/* ── safe diagnostic: expose shape only, never content ── */
/* parse string responses for key inspection */
let parsedResponse = providerResult.response;
if (typeof providerResult.response === "string") {
try { parsedResponse = JSON.parse(providerResult.response); } catch {}
}
const hasCU = typeof parsedResponse === "object" &&
parsedResponse !== null &&
Object.prototype.hasOwnProperty.call(parsedResponse, "currentUnderstanding");
const diag = {
providerApiPath: providerResult.providerApiPath ?? null,
responseType: typeof providerResult.response,
responseIsNull: providerResult.response === null,
responseKeys: (typeof parsedResponse === "object" && parsedResponse !== null && !Array.isArray(parsedResponse))
? Object.keys(parsedResponse).sort()
: null,
hasCurrentUnderstanding: hasCU,
currentUnderstandingType: hasCU ? typeof parsedResponse.currentUnderstanding : null,
currentUnderstandingLength: hasCU
? (typeof parsedResponse.currentUnderstanding === "string"
? parsedResponse.currentUnderstanding.length
: null)
: null,
};
const err = new Error(`Synthesis validation failed: ${validated.error}`);
err.statusCode = 502;
err.sanitizedDiagnostic = diag;
throw err;
}
+78 -2
View File
@@ -15,9 +15,9 @@
// ── Eligibility constants ──────────────────────────────────
const ELIGIBLE_DISPOSITIONS = new Set([null, "agree"]);
export const ELIGIBLE_DISPOSITIONS = new Set([null, "agree"]);
const EXCLUDED_DISPOSITIONS = new Set(["not_relevant", "not_quite"]);
export const EXCLUDED_DISPOSITIONS = new Set(["not_relevant", "not_quite"]);
// ── Public API ─────────────────────────────────────────────
@@ -104,3 +104,79 @@ export function prepareCompletedEpisode({ situationGraph, targetNodeId, contribu
excludedFindingProvenance,
};
}
// ── Investigation learning projection ────────────────────────
/**
* Derive deterministic learning from a completed investigation episode.
*
* @param {Object} params
* @param {Object} params.situationGraph - Authoritative SituationGraph (read-only)
* @param {Array<Object>} params.contributions - All persisted Contributions
* @param {Array<Object>} params.findings - All canonical Findings
* @returns {Array<Object>} investigationLearning — ordered by resolvedNodeIds, then Contribution.sequence
*/
export function deriveInvestigationLearning({ situationGraph, contributions = [], findings = [] }) {
const resolvedNodeIds = situationGraph?.resolvedNodeIds ?? [];
// Build lookup from node id to node (for question text)
const nodeById = new Map();
for (const node of situationGraph?.nodes ?? []) {
if (node?.id) {
nodeById.set(node.id, node);
}
}
const result = [];
for (const nodeId of resolvedNodeIds) {
const node = nodeById.get(nodeId);
if (!node) continue;
// Collect scoped contributions ordered by sequence
const scopedContribs = [];
for (const contrib of contributions) {
if (contrib?.targetNodeId === nodeId) {
scopedContribs.push(contrib);
}
}
const orderedContribs = scopedContribs
.slice()
.sort(function(a, b) {
var aSeq = a?.sequence != null ? a.sequence : Infinity;
var bSeq = b?.sequence != null ? b.sequence : Infinity;
return aSeq - bSeq;
});
// Build contribution ID set for provenance chain
const contribIds = new Set(orderedContribs.map(function(c) { return c.id; }));
// Collect eligible findings from scoped contributions via Contribution provenance only
var propositions = [];
for (var i = 0; i < orderedContribs.length; i++) {
var contrib = orderedContribs[i];
for (var j = 0; j < findings.length; j++) {
var finding = findings[j];
if (finding?.contributionId === contrib.id && ELIGIBLE_DISPOSITIONS.has(finding?.userDisposition ?? null)) {
propositions.push({
findingId: finding.id ?? null,
proposition: finding.proposition ?? "",
userDisposition: finding.userDisposition ?? null,
});
}
}
}
// Only include questions that produced at least one eligible proposition
if (propositions.length > 0) {
result.push({
nodeId: nodeId,
questionText: node.label ?? "",
propositions: propositions,
});
}
}
return result;
}
+157 -3
View File
@@ -1,4 +1,7 @@
import { formulateQuestion } from "@/lib/graph/question-formulator.js";
import {
formulateQuestion,
isStrategyQuestionTemplateCompatible,
} from "@/lib/graph/question-formulator.js";
const FOCUSED_ANSWER_SCHEMA_FIELDS = [
"targetNodeId",
@@ -35,6 +38,16 @@ export const focusedDeconstructJsonSchema = {
additionalProperties: false,
};
export const strategyCompatibilityAssessmentJsonSchema = {
type: "object",
properties: {
strategyKey: { type: "string" },
relevance: { type: "string", enum: ["compatible", "incompatible"] },
},
required: ["strategyKey", "relevance"],
additionalProperties: false,
};
const FORBIDDEN_GRAPH_MUTATION_FIELDS = [
"addedNodes",
"updatedNodes",
@@ -49,14 +62,19 @@ const FORBIDDEN_GRAPH_MUTATION_FIELDS = [
/**
* Formulate a question for the explicitly user-selected unresolved node.
*
* Input: { situationGraph, targetNodeId }
* Input: { situationGraph, targetNodeId, strategyCompatibilityVerdict? }
* Validates that target exists, is an unknown, and is not resolved.
* Calls formulateQuestion with only factual explicit-user-targeting context.
* Does NOT invoke selectActiveUnknownCandidate, determineGraphBackedQuestion,
* global ranking, or global recommendation.
* Does NOT mutate activeUnknownNodeId, selectedQuestion, SituationGraph, or resolvedNodeIds.
*/
export function formulateQuestionForTarget({ situationGraph, targetNodeId }) {
export function formulateQuestionForTarget({
situationGraph,
targetNodeId,
strategyCompatibilityVerdict,
strategyTemplateCompatibility,
}) {
const nodesById = new Map(situationGraph?.nodes?.map((n) => [n.id, n]) || []);
const targetNode = nodesById.get(targetNodeId);
@@ -80,6 +98,12 @@ export function formulateQuestionForTarget({ situationGraph, targetNodeId }) {
_explicitTargetNodeId: targetNodeId,
_explicitTargetLabel: targetNode.label,
_explicitTargetDescription: targetNode.description,
...(strategyCompatibilityVerdict === undefined
? {}
: { strategyCompatibilityVerdict }),
...(strategyTemplateCompatibility === undefined
? {}
: { strategyTemplateCompatibility }),
};
const result = formulateQuestion({ node: targetNode, graph: situationGraph, context });
@@ -89,6 +113,7 @@ export function formulateQuestionForTarget({ situationGraph, targetNodeId }) {
targetNodeId,
question: result.question,
strategy: result.strategy ?? null,
investigationStrategy: result.investigationStrategy,
reasoningPattern: result.reasoningPattern,
reasoningPatternReason: result.reasoningPatternReason,
reason: result.reason,
@@ -99,6 +124,84 @@ export function formulateQuestionForTarget({ situationGraph, targetNodeId }) {
};
}
export function buildStrategyCompatibilityAssessmentPrompt({
targetLabel,
targetDescription,
centralStatement,
proposedStrategy,
}) {
return `Assess whether one already-proposed investigation strategy is relevant to one explicitly selected uncertainty.
Return exactly one JSON object and JSON only.
You may only assess the relevance of the supplied proposed strategy to the supplied selected uncertainty.
Do not choose another uncertainty. Do not propose another strategy. Do not write the final question. Do not decide the user's conclusion. Do not mutate or recommend changes to any graph.
Return "compatible" only when this exact strategy would help investigate this exact uncertainty. Return "incompatible" when it would not.
Selected uncertainty:
- label: ${targetLabel}
- description: ${targetDescription}
Case context:
- central statement: ${centralStatement}
Proposed strategy:
- strategyKey: ${proposedStrategy.key}
- strategy meaning: ${proposedStrategy.meaning}
Return exactly:
{"strategyKey":"${proposedStrategy.key}","relevance":"compatible" | "incompatible"}`;
}
export function validateStrategyCompatibilityVerdict(result, proposedStrategy) {
if (!result || typeof result !== "object" || Array.isArray(result)) return null;
const keys = Object.keys(result);
if (
keys.length !== 2 ||
!keys.includes("strategyKey") ||
!keys.includes("relevance") ||
typeof result.strategyKey !== "string" ||
result.strategyKey !== proposedStrategy?.key ||
(result.relevance !== "compatible" && result.relevance !== "incompatible")
) {
return null;
}
return { strategyKey: result.strategyKey, relevance: result.relevance };
}
export async function assessStrategyCompatibility({
provider,
modelName,
targetLabel,
targetDescription,
centralStatement,
proposedStrategy,
signal,
}) {
const prompt = buildStrategyCompatibilityAssessmentPrompt({
targetLabel,
targetDescription,
centralStatement,
proposedStrategy,
});
const wrapper = await provider.generateReconstruction(
prompt,
modelName,
strategyCompatibilityAssessmentJsonSchema,
{ signal },
);
const verdict = validateStrategyCompatibilityVerdict(
wrapper?.response,
proposedStrategy,
);
return verdict
? { success: true, verdict }
: { success: false, status: "invalid" };
}
export { isStrategyQuestionTemplateCompatible };
/**
* Build a focused answer deconstruction prompt for one explicitly user-chosen investigation.
*/
@@ -189,4 +292,55 @@ export function validateFocusedDeconstructSchema(result) {
return errors;
}
/**
* Compose a focused answer deconstruction invocation.
*
* Accepts explicit provider and modelName — does NOT call getProvider or getProviderModelName.
* Calls buildFocusedDeconstructPrompt internally to produce the prompt.
* Calls provider.generateReconstruction exactly once.
* Validates result with validateFocusedDeconstructSchema.
* Returns validated semantic payload on success, or { validation: false, errors } on schema failure.
*/
export async function focusedDeconstruct({
provider,
modelName,
targetNodeId,
targetLabel,
targetDescription,
centralStatement,
question,
answer,
}) {
const prompt = buildFocusedDeconstructPrompt({
targetLabel,
targetDescription,
centralStatement,
question,
answer,
});
const wrapper = await provider.generateReconstruction(
prompt,
modelName,
focusedDeconstructJsonSchema,
);
const deconstruction = wrapper.response;
const validationErrors = validateFocusedDeconstructSchema(deconstruction);
if (validationErrors.length > 0) {
return { success: false, validation: false, errors: validationErrors };
}
return {
success: true,
targetNodeId,
observations: deconstruction.observations,
uncertainties: deconstruction.uncertainties,
assumptions: deconstruction.assumptions,
relationships: deconstruction.relationships,
possibleFollowUpQuestions: deconstruction.possibleFollowUpQuestions,
};
}
export { FOCUSED_ANSWER_SCHEMA_FIELDS, FORBIDDEN_GRAPH_MUTATION_FIELDS };
+17 -6
View File
@@ -28,9 +28,19 @@ export { filterEligibleFindings };
const overviewResponseSchema = z.object({
understanding: z.string().min(1),
plausibleInterpretations: z.string().min(1),
plausibleInterpretations: z.string(),
});
const overviewOutputSchema = {
type: "object",
properties: {
understanding: { type: "string" },
plausibleInterpretations: { type: "string" },
},
required: ["understanding", "plausibleInterpretations"],
additionalProperties: false,
};
const FORBIDDEN_FIELD_NAMES = new Set([
"recommendation",
"decision",
@@ -200,7 +210,7 @@ ${interpSections.length > 0 ? JSON.stringify(interpSections, null, 2) : "(none)"
9. Neither field may contain recommendations, decisions, confidence scores, next actions, priorities, or readiness assessments.
10. This is a FRESH synthesis — do NOT treat any previous overview or Current Understanding as input.
Both fields must be non-empty strings.`;
understanding must be a non-empty string. When SECTION B has no plausible inputs, return plausibleInterpretations as an empty string ("") — do NOT manufacture an absence statement.`;
return prompt;
}
@@ -240,18 +250,19 @@ export async function synthesizeInvestigationOverview(params, deps) {
// Build overview-specific prompt (keeps evidence/interpretation separate)
const prompt = buildOverviewSynthesisPrompt(situationGraph, findings, plausibleInterpretations);
let rawResponse;
let providerResult;
try {
rawResponse = await provider.generateReconstruction(
providerResult = await provider.generateReconstruction(
prompt,
modelName,
overviewOutputSchema,
);
} catch (err) {
throw new Error(`Overview synthesis provider call failed: ${err.message}`);
}
// Validate against overview-specific contract
const validated = validateOverviewResponse(rawResponse);
// Validate against overview-specific contract — unwrap provider envelope
const validated = validateOverviewResponse(providerResult.response);
if (!validated.valid) {
throw new Error(`Overview synthesis validation failed: ${validated.reason}`);
}
+27
View File
@@ -88,6 +88,10 @@ function buildDiagnostics({
compatibilityApplied: analysis?.compatibilityApplied ?? false,
compatibilityChanges: analysis?.compatibilityChanges ?? [],
compatibilityWarnings: analysis?.compatibilityWarnings ?? [],
presentationValidationStatus:
analysis?.presentationValidationStatus ?? "not_applicable",
presentationValidationIssues:
analysis?.presentationValidationIssues ?? [],
unknownSelectionExplanation: unknownSelectionExplanation ?? null,
reconstructionQuestion: reconstructionQuestion ?? null,
reconstructionQuestionAccepted: reconstructionQuestionAccepted ?? null,
@@ -366,6 +370,12 @@ export async function startCase(body, dependencies = {}) {
if (dependencies.reconstructionModelName) {
analysisOptions.reconstructionModelName = dependencies.reconstructionModelName;
}
if (dependencies.generationSchema) {
analysisOptions.generationSchema = dependencies.generationSchema;
}
if (dependencies.freeform) {
analysisOptions.freeform = dependencies.freeform;
}
const analysis = await analyseScenario(scenario, analysisOptions);
if (!analysis.success) {
@@ -387,6 +397,23 @@ export async function startCase(body, dependencies = {}) {
};
}
// Condition D freeform + reconstruction-only: skip structured graph pipeline.
if (dependencies.reconstructionOnly && dependencies.freeform) {
return {
success: true,
summary: analysis.summary ?? null,
reconstruction: analysis.reconstruction, // genuine freeform text
situationGraph: null,
selectedQuestion: null,
diagnostics: buildDiagnostics({
analysis,
graph: null,
graphReferenceValidation: null,
}),
statusCode: 200,
};
}
const initialGraph = buildInitialGraph({
reconstruction: analysis.reconstruction,
evidence: analysis.evidence,
+48 -9
View File
@@ -1198,6 +1198,37 @@ function constrainStrategyToReasoningPattern(strategy, reasoningPattern) {
return allowedStrategies.includes(strategy.key) ? strategy : null;
}
function applyStrategyCompatibilityVerdict(proposedStrategy, verdict) {
// No verdict is the backward-compatible path. A supplied verdict can only
// preserve this exact proposal or suppress it; it cannot select a substitute.
if (verdict === undefined) return proposedStrategy;
if (!proposedStrategy || !verdict || typeof verdict !== "object" || Array.isArray(verdict)) {
return null;
}
const keys = Object.keys(verdict);
if (
keys.length !== 2 ||
!keys.includes("strategyKey") ||
!keys.includes("relevance") ||
typeof verdict.strategyKey !== "string" ||
(verdict.relevance !== "compatible" && verdict.relevance !== "incompatible") ||
verdict.strategyKey !== proposedStrategy.key
) {
return null;
}
return verdict.relevance === "compatible" ? proposedStrategy : null;
}
export function isStrategyQuestionTemplateCompatible(strategy) {
if (!strategy) return true;
return !(
strategy.key === "baseline_reconstruction" &&
isWhetherPropositionMeaning(strategy.meaning)
);
}
function selectQuestionFamily({
node,
graph,
@@ -2047,7 +2078,7 @@ export function formulateQuestion({ node, graph, context = {} }) {
}
}
const investigationStrategy = selectInvestigationStrategy({
const proposedInvestigationStrategy = selectInvestigationStrategy({
node,
graph,
context: {
@@ -2055,12 +2086,20 @@ export function formulateQuestion({ node, graph, context = {} }) {
reasoningPattern: reasoningPatternSelection.pattern,
},
});
const investigationStrategy = applyStrategyCompatibilityVerdict(
proposedInvestigationStrategy,
context.strategyCompatibilityVerdict,
);
const templateCompatibleStrategy =
context.strategyTemplateCompatibility === "incompatible"
? null
: investigationStrategy;
const questionFamilySelection = selectQuestionFamily({
node,
graph,
reasoningPattern: reasoningPatternSelection.pattern,
investigationStrategy,
investigationStrategy: templateCompatibleStrategy,
});
let question = buildQuestionFromFamily({
@@ -2069,7 +2108,7 @@ export function formulateQuestion({ node, graph, context = {} }) {
reasoningPattern: reasoningPatternSelection.pattern,
questionFamily: questionFamilySelection.family,
selectedQuestionTemplate: questionFamilySelection.template,
investigationStrategy,
investigationStrategy: templateCompatibleStrategy,
});
question = sanitizeQuestionText(question);
@@ -2080,11 +2119,11 @@ export function formulateQuestion({ node, graph, context = {} }) {
if (
!validateFormulatedQuestion(
question,
investigationStrategy?.meaning || fallbackMeaning,
templateCompatibleStrategy?.meaning || fallbackMeaning,
)
) {
question = sanitizeQuestionText(
(investigationStrategy &&
(templateCompatibleStrategy &&
isClaimLikeUnknown(
node,
normaliseText(
@@ -2112,11 +2151,11 @@ export function formulateQuestion({ node, graph, context = {} }) {
return {
question,
reason: investigationStrategy
? `Formulated from graph context using the ${investigationStrategy.key} investigation strategy.`
reason: templateCompatibleStrategy
? `Formulated from graph context using the ${templateCompatibleStrategy.key} investigation strategy.`
: "Formulated as a neutral clarification question because no narrower investigation strategy clearly applied.",
strategy: investigationStrategy?.key ?? null,
investigationStrategy,
strategy: templateCompatibleStrategy?.key ?? null,
investigationStrategy: templateCompatibleStrategy,
reasoningPattern: reasoningPatternSelection.pattern,
reasoningPatternReason: reasoningPatternSelection.reason,
questionFamily: questionFamilySelection.family,
+2
View File
@@ -57,6 +57,7 @@ export const situationNodeSchema = z.object({
id: z.string().min(1),
label: z.string().min(1),
description: z.string().min(1),
openQuestion: z.string().min(1).optional(),
kind: z.enum(Object.values(SituationKind)),
status: z.enum(Object.values(SituationStatus)),
confidence: z.enum(Object.values(ConfidenceLevel)),
@@ -259,6 +260,7 @@ export function makeNode(opts) {
id,
label: opts.label,
description: opts.description ?? opts.label,
openQuestion: opts.openQuestion,
kind: opts.kind ?? "observation",
status: opts.status ?? "unknown",
confidence: opts.confidence ?? "medium",
+114 -41
View File
@@ -290,7 +290,27 @@ export function __resetChatSupportForTests() {
_chatSupported = null;
}
async function detectChatSupport(baseUrl, modelName) {
function isAbortError(error) {
return error?.name === "AbortError";
}
function createRequestAbortController(signal, timeoutMs = 300000) {
const controller = new AbortController();
const abortFromCaller = () => controller.abort();
if (signal?.aborted) controller.abort();
else signal?.addEventListener("abort", abortFromCaller, { once: true });
const timeout = setTimeout(() => controller.abort(), timeoutMs);
return {
signal: controller.signal,
dispose() {
clearTimeout(timeout);
signal?.removeEventListener("abort", abortFromCaller);
},
};
}
async function detectChatSupport(baseUrl, modelName, signal) {
if (_chatSupported !== null) return _chatSupported;
try {
@@ -302,6 +322,7 @@ async function detectChatSupport(baseUrl, modelName) {
messages: [{ role: "user", content: "test" }],
stream: false,
}),
signal,
});
if (res.ok) {
@@ -314,7 +335,8 @@ async function detectChatSupport(baseUrl, modelName) {
await res.text();
_chatSupported = false;
}
} catch {
} catch (error) {
if (isAbortError(error) && signal?.aborted) throw error;
_chatSupported = false;
}
@@ -322,7 +344,7 @@ async function detectChatSupport(baseUrl, modelName) {
}
class OllamaLlmProvider {
async generateReconstruction(scenario, modelName, outputSchema) {
async generateReconstruction(scenario, modelName, outputSchema, { signal } = {}) {
// scenario is ALREADY a fully-built prompt text (built by analyseScenario).
// Do NOT call buildPrompt() again — that would double-wrap the prompt.
const prompt = scenario;
@@ -330,6 +352,9 @@ class OllamaLlmProvider {
const baseUrl = process.env.OLLAMA_BASE_URL;
if (!baseUrl) throw new Error("OLLAMA_BASE_URL is not set");
// Explicit null signals "freeform text" — no JSON-schema constraint.
const useFreeformText = outputSchema === null;
let apiUsed = null;
let chatSupported = false;
let rawResponse = null;
@@ -345,34 +370,37 @@ class OllamaLlmProvider {
// Step 1: Detect whether /api/chat exists (cache result)
// ================================================================
try {
chatSupported = await detectChatSupport(baseUrl, modelName);
} catch { /* failed silently — defaults to false */ }
chatSupported = await detectChatSupport(baseUrl, modelName, signal);
} catch (error) {
if (isAbortError(error) && signal?.aborted) throw error;
// Failed silently — defaults to false.
}
providerExecution.chatCapabilityDetected = chatSupported;
// ================================================================
// Step 2: Try /api/chat if supported with the supplied or reconstruction schema
// ================================================================
if (chatSupported) {
const requestAbort = createRequestAbortController(signal);
try {
const controller = new AbortController();
const timeout = setTimeout(() => controller.abort(), 300000); // 5 min for reconstruction
apiUsed = "/api/chat";
providerExecution.chatRequestAttempted = true;
const body = {
model: modelName,
messages: [{ role: "user", content: prompt }],
stream: false,
};
// Explicit null → omit format constraint entirely.
if (outputSchema !== null) {
body.format = outputSchema ?? reconstructionJsonSchema;
}
const res = await fetch(`${baseUrl}/api/chat`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
model: modelName,
messages: [{ role: "user", content: prompt }],
stream: false,
format: outputSchema ?? reconstructionJsonSchema,
}),
signal: controller.signal,
body: JSON.stringify(body),
signal: requestAbort.signal,
});
clearTimeout(timeout);
if (res.ok) {
providerExecution.chatRequestSucceeded = true;
fullResponseData = await res.json();
@@ -384,7 +412,9 @@ class OllamaLlmProvider {
await res.text();
}
} catch (e) {
if (!e.message.includes("abort")) { /* non-fatal */ }
if (isAbortError(e)) throw e;
} finally {
requestAbort.dispose();
}
}
@@ -393,10 +423,8 @@ class OllamaLlmProvider {
// Use a long timeout — cold starts can take 2-4 minutes for large models.
// ================================================================
if (rawResponse == null || rawResponse === "") {
const requestAbort = createRequestAbortController(signal);
try {
const controller = new AbortController();
const timeout = setTimeout(() => controller.abort(), 300000); // 5 min for cold start
apiUsed = "/api/generate"; // set BEFORE the request so we know which API failed
providerExecution.generateRequestAttempted = true;
@@ -410,11 +438,9 @@ class OllamaLlmProvider {
// No format:json — older Ollama doesn't support it on /api/generate either.
// We rely on the strong prompt instruction above instead.
}),
signal: controller.signal,
signal: requestAbort.signal,
});
clearTimeout(timeout);
if (!res.ok) {
const text = await res.text().catch(() => "");
throw new Error(`/api/generate returned ${res.status}: ${text.slice(0, 500)}`);
@@ -426,6 +452,11 @@ class OllamaLlmProvider {
? fullResponseData.response
: JSON.stringify(fullResponseData);
// generate has no format constraint — treat freeform input as freeform output.
if (useFreeformText) {
apiUsed = "/api/generate";
}
} catch (e) {
if (apiUsed === "/api/generate") {
const error = new Error(
@@ -439,7 +470,7 @@ class OllamaLlmProvider {
`- Use a smaller model (e.g., llama3.1 instead of llama3.1:70b)\n` +
`- Check Ollama logs: \`ollama serve\` or look at your system logs`
);
if (e?.name === "AbortError" || e instanceof TypeError) {
if (isAbortError(e) || e instanceof TypeError) {
error.code = "PROVIDER_UNAVAILABLE";
}
error.providerApiPath = apiUsed;
@@ -447,6 +478,8 @@ class OllamaLlmProvider {
throw error;
}
throw e;
} finally {
requestAbort.dispose();
}
}
@@ -455,11 +488,11 @@ class OllamaLlmProvider {
// ================================================================
if (rawResponse === "") {
let diagInfo = "";
if (fullResponseData) {
const keys = Object.keys(fullResponseData);
diagInfo = "Keys in response: " + keys.join(", ") + "\n";
for (const key of keys) {
const val = fullResponseData[key];
if (typeof val === "string") {
@@ -480,6 +513,7 @@ class OllamaLlmProvider {
"\nPossible causes:\n" +
"- Check 'ollama list' — make sure the model name matches exactly what's installed\n" +
"- The model may be corrupted. Try: ollama pull " + modelName + "\n" +
(useFreeformText ? "- No format constraint was sent — the model may not support freeform output.\n" : "") +
"- This Ollama version does not support format:json — using prompt instructions only (reliability varies)\n" +
"- If your model is very small (e.g., tinyllama, phi), try a larger one like llama3.1 or mistral"
);
@@ -491,6 +525,14 @@ class OllamaLlmProvider {
// ================================================================
// Step 5: Parse and return
// ================================================================
if (useFreeformText) {
return {
response: rawResponse,
providerApiPath: apiUsed,
providerExecution,
};
}
try {
return {
response: recoverJson(rawResponse),
@@ -525,19 +567,34 @@ class OpenAIReconstructionProvider {
this.fetchImpl = fetchImpl;
}
async generateReconstruction(prompt, modelName = "gpt-5.6-terra", outputSchema) {
async generateReconstruction(
prompt,
modelName = "gpt-5.6-terra",
outputSchema,
{ signal, outputZodSchema } = {},
) {
if (!this.apiKey) throw new Error("OPENAI_API_KEY is not set");
const structuredOutputSchema = outputSchema
? createOpenAIStrictSchema(outputSchema, null)
: openAIReconstructionJsonSchema;
const format = {
type: "json_schema",
name: "reconstruction",
strict: true,
schema: structuredOutputSchema,
};
traceOpenAISchema({ model: modelName, format, schema: structuredOutputSchema });
// Explicit null → freeform text output (no JSON-schema constraint).
const useFreeformText = outputSchema === null;
let format;
if (useFreeformText) {
format = { type: "text" };
} else {
const structuredOutputSchema = outputSchema
? createOpenAIStrictSchema(outputSchema, outputZodSchema ?? null)
: openAIReconstructionJsonSchema;
format = {
type: "json_schema",
name: "reconstruction",
strict: true,
schema: structuredOutputSchema,
};
traceOpenAISchema({ model: modelName, format, schema: structuredOutputSchema });
}
const response = await this.fetchImpl("https://api.openai.com/v1/responses", {
method: "POST",
headers: {
@@ -551,6 +608,7 @@ class OpenAIReconstructionProvider {
format,
},
}),
signal,
});
if (!response.ok) {
@@ -572,10 +630,25 @@ class OpenAIReconstructionProvider {
throw error;
}
// Freeform mode — return raw text without JSON recovery or transformation.
if (useFreeformText) {
return {
response: outputText,
providerApiPath: "/v1/responses",
};
}
const recoveredResponse = recoverJson(outputText);
const normalisedResponse = outputSchema && !outputZodSchema
? recoveredResponse
: normaliseOpenAITransportResponse(
recoveredResponse,
outputZodSchema ?? reconstructionV2Schema,
outputSchema ?? reconstructionJsonSchema,
);
return {
response: outputSchema
? recoverJson(outputText)
: normaliseOpenAITransportResponse(recoverJson(outputText)),
response: normalisedResponse,
providerApiPath: "/v1/responses",
};
}
+12 -3
View File
@@ -274,8 +274,15 @@ export function buildFacilitatorViewModel(_ref) {
// Scaffold items are entirely suppressed from user-facing sections.
if (semanticRole === "scaffolding") continue;
// An unknown's openQuestion is a model-authored presentation projection.
// Its label and description remain the canonical uncertainty representation.
// Present the canonical description first; fall back to normalised display text.
var displayText =
node.kind === "unknown" && !resolved && node.description
? node.description.trim()
: displayResult.displayText;
var entry = {
text: displayResult.displayText,
text: displayText,
normalised: normaliseText(displayResult.displayText),
kind: node.kind,
semanticRole: semanticRole,
@@ -410,8 +417,10 @@ export function buildFacilitatorViewModel(_ref) {
if (!a.isActiveUnknown && b.isActiveUnknown) return 1;
// Selected-question target: match by normalised text
if (sqText && a.normalised === sqText && b.normalised !== sqText) return -1;
if (sqText && a.normalised !== sqText && b.normalised === sqText) return 1;
var aMatchesSelectedQuestion = sqText && normaliseText(a.text) === sqText;
var bMatchesSelectedQuestion = sqText && normaliseText(b.text) === sqText;
if (aMatchesSelectedQuestion && !bMatchesSelectedQuestion) return -1;
if (!aMatchesSelectedQuestion && bMatchesSelectedQuestion) return 1;
// Explicit priority fields where available in the graph
var pa = a.priority != null ? a.priority : null;
@@ -0,0 +1,63 @@
const ISSUE_MESSAGES = {
question_missing: "No user-facing Open Question was generated.",
question_empty: "The generated Open Question was empty.",
question_not_interrogative: "The generated Open Question was not a direct interrogative.",
question_indirect_proposition:
"The generated Open Question used an indirect uncertainty proposition.",
};
/**
* Narrow presentation validation only. This does not attempt to prove semantic
* equivalence with the canonical uncertainty and never rewrites either value.
*/
export function validateOpenQuestionCandidate(candidate) {
if (candidate === undefined || candidate === null) {
return { valid: false, category: "question_missing" };
}
if (typeof candidate !== "string" || candidate.trim().length === 0) {
return { valid: false, category: "question_empty" };
}
const question = candidate.trim();
if (/^whether\b/i.test(question)) {
return { valid: false, category: "question_indirect_proposition" };
}
if (!question.endsWith("?")) {
return { valid: false, category: "question_not_interrogative" };
}
return { valid: true, question };
}
/**
* Remove invalid presentation candidates without changing canonical descriptions,
* IDs, ordering, relationships, or any other reconstruction content.
*/
export function validateReconstructionOpenQuestions(reconstruction) {
const issues = [];
const importantUnknowns = (reconstruction?.importantUnknowns ?? []).map(
(unknown, index) => {
const validation = validateOpenQuestionCandidate(unknown.question);
if (validation.valid) {
return { ...unknown, question: validation.question };
}
const { question: _invalidQuestion, ...canonicalUnknown } = unknown;
issues.push({
category: validation.category,
path: ["reconstruction", "importantUnknowns", index, "question"],
unknownId: unknown.id,
message: ISSUE_MESSAGES[validation.category],
severity: "nonfatal",
});
return canonicalUnknown;
},
);
return {
reconstruction: { ...reconstruction, importantUnknowns },
issues,
};
}
+178
View File
@@ -0,0 +1,178 @@
import { z } from "zod";
/**
* Condition B — uncertainty-description-only variant schema.
*
* Differs from the production reconstructionGenerationSchema in exactly one way:
* importantUnknowns items use the base itemSchemaV1 which has NO `question` property.
* Condition A uses the production schema (reconstructionGenerationSchema) unchanged.
*/
// Base item used by both conditions (id, description, confidence only)
const confidenceEnumV1 = z.enum(["low", "medium", "high"]);
const itemSchemaV1 = z.object({
id: z.string().min(1),
description: z.string().min(1),
confidence: confidenceEnumV1,
});
// Condition B importantUnknowns items: NO question property — just the base item fields
const conditionBImportantUnknownSchema = itemSchemaV1;
// Build the Condition B Zod schema using the tolerant importantUnknownSchema
const reconstructionGenerationConditionBSchema = z.object({
inputClassification: z.object({
primaryType: z.enum([
"observed_problem",
"unexplained_change",
"contradiction",
"decision_request",
"causal_claim",
"reported_claim",
"fault_report",
"ambiguous_statement",
"question",
"desired_outcome",
"insufficient_context",
"other",
]),
secondaryTypes: z.array(z.enum([
"observed_problem",
"unexplained_change",
"contradiction",
"decision_request",
"causal_claim",
"reported_claim",
"fault_report",
"ambiguous_statement",
"question",
"desired_outcome",
"insufficient_context",
"other",
])).optional().default([]),
reasoningModes: z.array(z.enum([
"establish_baseline",
"identify_difference",
"reconstruct_transition",
"decompose_aggregate",
"validate_measurement",
"validate_claim",
"investigate_contradiction",
"clarify_meaning",
"decision_support",
"fault_investigation",
"identify_missing_information",
"test_possible_explanations",
"other",
])).optional().default([]),
classificationReason: z.string().min(1),
confidence: z.enum(["low", "medium", "high"]),
}),
reconstruction: z.object({
summary: z.string().min(1),
actors: z.array(itemSchemaV1),
systemsOrObjects: z.array(itemSchemaV1),
expectedStates: z.array(itemSchemaV1),
observedStates: z.array(itemSchemaV1),
differences: z.array(itemSchemaV1),
knownTransitions: z.array(
itemSchemaV1.extend({
entity: z.string().min(1),
previousState: z.string().min(1),
currentState: z.string().min(1),
explanationStatus: z.string().min(1),
}),
),
unexplainedTransitions: z.array(
itemSchemaV1.extend({
entity: z.string().min(1).optional(),
previousState: z.string().min(1).optional(),
currentState: z.string().min(1).optional(),
}),
),
contradictions: z.array(itemSchemaV1),
importantUnknowns: z.array(conditionBImportantUnknownSchema),
plausibleInterpretations: z.array(
z.object({
id: z.string().min(1),
description: z.string().min(1),
supportingEvidenceIds: z.array(z.string()),
assumptionsRequired: z.array(z.string()).optional().default([]),
confidence: z.enum(["low", "medium", "high"]),
}),
),
relationships: z.array(z.object({
id: z.string().min(1),
fromId: z.string().min(1),
toId: z.string().min(1),
relationship: z.enum([
"supports",
"weakens",
"contradicts",
"depends_on",
"causes",
"may_cause",
"compares_with",
"other",
]),
description: z.string().min(1),
confidence: z.enum(["low", "medium", "high"]),
})).optional().default([]),
}),
evidence: z.array(z.object({
id: z.string().min(1),
description: z.string().min(1),
evidenceType: z.enum([
"direct_observation",
"reported_statement",
"interpretation",
"assumption",
"inferred_relationship",
]),
source: z.string().optional(),
attribution: z.string().nullable().optional(),
confidence: z.enum(["low", "medium", "high"]),
importance: z.enum([
"incidental",
"supporting",
"important",
"critical",
]),
})),
nextQuestion: z.object({
id: z.string().min(1),
question: z.string().min(1),
targets: z.array(z.string()),
reason: z.string().min(1),
expectedInformationValue: z.enum(["low", "medium", "high"]),
reasoningMode: z.enum([
"establish_baseline",
"identify_difference",
"reconstruct_transition",
"decompose_aggregate",
"validate_measurement",
"validate_claim",
"investigate_contradiction",
"clarify_meaning",
"decision_support",
"fault_investigation",
"identify_missing_information",
"test_possible_explanations",
"other",
]).optional().default("other"),
}),
});
/** JSON Schema derived from Condition B Zod — importantUnknowns has NO question property */
export const conditionBJsonSchema = z.toJSONSchema(
reconstructionGenerationConditionBSchema,
);
import {
reconstructionGenerationSchema as _PROD_SCHEMA,
reconstructionGenerationJsonSchema as _PROD_JSON,
} from "./schema.js";
/** Condition A — production schema (importantUnknowns items REQUIRE `question`) */
export const conditionASchema = _PROD_SCHEMA;
export const conditionAJsonSchema = _PROD_JSON;
+29 -1
View File
@@ -144,6 +144,21 @@ const reconstructionRelationshipSchema = z.object({
confidence: confidenceEnum,
});
const importantUnknownSchema = itemSchemaV1.extend({
// Canonical uncertainty meaning remains in description. question is an
// optional, model-authored user-facing interrogative for the Open Questions UI.
// Keep this tolerant for historical data and nonfatal presentation validation.
question: z.string().optional(),
});
const generatedImportantUnknownSchema = itemSchemaV1.extend({
// New generation requests require a distinct presentation candidate. More
// nuanced interrogative validation is intentionally performed locally after
// canonical reconstruction validation rather than encoded as provider-specific
// JSON Schema grammar.
question: z.string().trim().min(1),
});
const reconstructionSchemaV2 = z.object({
summary: z.string().min(1),
actors: z.array(itemSchemaV1),
@@ -167,7 +182,7 @@ const reconstructionSchemaV2 = z.object({
}),
),
contradictions: z.array(itemSchemaV1),
importantUnknowns: z.array(itemSchemaV1),
importantUnknowns: z.array(importantUnknownSchema),
plausibleInterpretations: z.array(
z.object({
id: z.string().min(1),
@@ -205,6 +220,19 @@ export const reconstructionV2Schema = z.object({
nextQuestion: nextQuestionSchema,
});
// Generation-only contract. Durable reconstruction parsing remains tolerant so
// historical snapshots and otherwise valid canonical reconstructions do not fail
// solely because presentation wording is absent or malformed.
export const reconstructionGenerationSchema = reconstructionV2Schema.extend({
reconstruction: reconstructionSchemaV2.extend({
importantUnknowns: z.array(generatedImportantUnknownSchema),
}),
});
export const reconstructionGenerationJsonSchema = z.toJSONSchema(
reconstructionGenerationSchema,
);
// Outer wrapper for API return (includes diagnostics + v0.2 data)
export const analyseResponseV2Schema = z.object({
inputClassification: inputClassificationSchema.optional(),
+1 -1
View File
@@ -1,7 +1,7 @@
import { createServerClient } from "@supabase/ssr";
import { NextResponse } from "next/server";
const PUBLIC_PATHS = ["/login", "/auth", "/api/health"];
const PUBLIC_PATHS = ["/login", "/auth", "/api/health", "/privacy", "/terms", "/cookies"];
export async function middleware(request) {
const pathname = request.nextUrl.pathname;
+2 -1
View File
@@ -19,6 +19,7 @@ You are a neutral analyst performing evidence-based situation reconstruction.
7. Distinguish: what was said / what it may mean / why it may have been said.
8. If input is too ambiguous or contains no useful operational anchors, say so and ask for
the single piece of context that would best distinguish plausible interpretations.
9. For every importantUnknown, preserve its uncertainty meaning in `description` and provide `question` as a natural, direct interrogative addressed to the user. Retain all scope, conditions, timeframes, and qualifications without adding assumptions or an implied conclusion. Do not use fragments or indirect proposition forms such as "Whether …".
## Confidence scale
@@ -83,7 +84,7 @@ Return a JSON object with exactly these four top-level keys (use **camelCase**):
"knownTransitions": [{"id": "...", "description": "...", "confidence": "<low|medium|high>", "entity": "...", "previousState": "...", "currentState": "...", "explanationStatus": "..."}],
"unexplainedTransitions": [{"id": "...", "description": "...", "confidence": "<low|medium|high>", "entity": "...", "previousState": "...", "currentState": "..."}],
"contradictions": [{"id": "...", "description": "...", "confidence": "<low|medium|high>"}],
"importantUnknowns": [{"id": "...", "description": "...", "confidence": "<low|medium|high>"}],
"importantUnknowns": [{"id": "...", "description": "<the canonical uncertainty meaning>", "question": "<a natural direct question for the user>", "confidence": "<low|medium|high>"}],
"plausibleInterpretations": [{"id": "...", "description": "...", "supportingEvidenceIds": ["<ids that support this interpretation>"], "assumptionsRequired": [], "confidence": "<low|medium|high>"}]
},
"evidence": [
+2 -1
View File
@@ -20,6 +20,7 @@ You are a neutral analyst performing evidence-based situation reconstruction.
8. Distinguish: what was said / what it may mean / why it may have been said.
9. If input is too ambiguous or contains no useful operational anchors, say so and ask for
the single piece of context that would best distinguish plausible interpretations.
9a. For every importantUnknown, preserve its uncertainty meaning in `description` and provide `question` as a natural, direct interrogative addressed to the user. Retain all scope, conditions, timeframes, and qualifications without adding assumptions or an implied conclusion. Do not use fragments or indirect proposition forms such as "Whether …".
10. Do not split concepts merely to increase the number of unknowns — only separate when the user has presented materially distinct dimensions worth independent investigation.
@@ -123,7 +124,7 @@ Return a JSON object with exactly these four top-level keys (use **camelCase**):
"knownTransitions": [{"id": "...", "description": "...", "confidence": "<low|medium|high>", "entity": "...", "previousState": "...", "currentState": "...", "explanationStatus": "..."}],
"unexplainedTransitions": [{"id": "...", "description": "...", "confidence": "<low|medium|high>", "entity": "...", "previousState": "...", "currentState": "..."}],
"contradictions": [{"id": "...", "description": "...", "confidence": "<low|medium|high>"}],
"importantUnknowns": [{"id": "...", "description": "...", "confidence": "<low|medium|high>"}],
"importantUnknowns": [{"id": "...", "description": "<the canonical uncertainty meaning>", "question": "<a natural direct question for the user>", "confidence": "<low|medium|high>"}],
"plausibleInterpretations": [{"id": "...", "description": "...", "supportingEvidenceIds": ["<ids that support this interpretation>"], "assumptionsRequired": [], "confidence": "<low|medium|high>"}]
},
"evidence": [
+3 -1
View File
@@ -29,6 +29,8 @@ You are a neutral analyst performing evidence-based situation reconstruction.
9. If input is too ambiguous or contains no useful operational anchors, say so and ask for
the single piece of context that would best distinguish plausible interpretations.
9a. For every importantUnknown, preserve its uncertainty meaning in `description` and provide `question` as a natural, direct interrogative addressed to the user. Retain all scope, conditions, timeframes, and qualifications without adding assumptions or an implied conclusion. Do not use fragments or indirect proposition forms such as "Whether …".
10. Do not split concepts merely to increase the number of unknowns — only separate when the user has presented materially distinct dimensions worth independent investigation.
## Preserve supplied relationships
@@ -139,7 +141,7 @@ Return a JSON object with exactly these four top-level keys (use **camelCase**):
"knownTransitions": [{"id": "...", "description": "...", "confidence": "<low|medium|high>", "entity": "...", "previousState": "...", "currentState": "...", "explanationStatus": "..."}],
"unexplainedTransitions": [{"id": "...", "description": "...", "confidence": "<low|medium|high>", "entity": "...", "previousState": "...", "currentState": "..."}],
"contradictions": [{"id": "...", "description": "...", "confidence": "<low|medium|high>"}],
"importantUnknowns": [{"id": "...", "description": "...", "confidence": "<low|medium|high>"}],
"importantUnknowns": [{"id": "...", "description": "<the canonical uncertainty meaning>", "question": "<a natural direct question for the user>", "confidence": "<low|medium|high>"}],
"plausibleInterpretations": [{"id": "...", "description": "...", "supportingEvidenceIds": ["<ids that support this interpretation>"], "assumptionsRequired": [], "confidence": "<low|medium|high>"}]
},
"evidence": [
+182
View File
@@ -0,0 +1,182 @@
You are a neutral analyst performing evidence-based situation reconstruction.
## Rules
1. Do NOT invent facts, context or causes. Only include information present in the scenario. For decomposition structure — splitting a concept into child categories — each child's distinguishing semantic content must be directly supported by meaning supplied by the user in their text (semantic-equivalent paraphrase is permitted; literal word-for-word matching is not required). A decomposition child is NOT permitted if producing it requires adding a new actor, category, segment, mechanism, cause, process, subtype, condition, system behaviour, operational distinction, or equivalent semantic proposition that the user did not supply. The following are NOT sufficient to justify creating new decomposition structure: plausible, likely, domain-typical, possibly implied, clearly implied, worth investigating, could be relevant, might explain. Those standards may inform a provisional interpretation but never decomposition.
2. First determine what kind of input has been supplied. Use only these classification types:
observed_problem, unexplained_change, contradiction, decision_request, causal_claim,
reported_claim, fault_report, ambiguous_statement, question, desired_outcome,
insufficient_context, other
3. Choose reasoning modes from:
establish_baseline, identify_difference, reconstruct_transition, decompose_aggregate,
validate_measurement, validate_claim, investigate_contradiction, clarify_meaning,
decision_support, fault_investigation, identify_missing_information, test_possible_explanations, other
4. Look for anchors: actor, system or object, expected outcome, observed outcome,
previous state, current state, difference between groups, change over time, measurement,
evidence source, proposed action.
5. Identify meaningful differences (e.g., some succeed while others fail; revenue rises while cash falls).
5a. Preserve materially distinct supplied outcome categories. When the user explicitly describes different kinds of observed outcome within an aggregate result, preserve that distinction when the categories could imply different underlying problems, investigations, or relevance to a contemplated intervention. Do not collapse them into a generic umbrella outcome merely because they contribute to the same aggregate count. Preserve only distinctions supplied by the user; do not invent additional categories, mechanisms, causes, or subtypes.
6. Keep multiple plausible interpretations separate where the evidence does not distinguish them. Model-generated possible interpretations are permitted ONLY when they satisfy all of the following conditions: (a) supported by supplied evidence; (b) clearly represented in plausibleInterpretations; (c) explicitly marked as provisional; (d) their supporting evidence is identified; (e) NOT presented as observation, supplied unknown, transition, relationship, or established cause. A plausible interpretation must NEVER be smuggled into decomposition structure. Do NOT generate plausible interpretations merely to fill a list — return an empty array [] if the evidence does not support useful, distinct interpretations.
7. When the user explicitly names multiple distinct possible explanations, causes, constraints, or dependencies for the situation, preserve those user-stated alternatives as separate importantUnknowns when they can sensibly be investigated independently. Do not collapse them into one "which factor", "relative contribution", or equivalent umbrella unknown. Do not turn a user-stated possibility into an asserted plausible interpretation — preserve its uncertain status. Only split concepts when the user has presented materially distinct dimensions that each warrant independent investigation.
7a. For every importantUnknown, express its canonical uncertainty meaning as a description in `description` with strict semantic fidelity: Preserve: the subject, epistemic openness, material scope, timing, conditions, and qualifications without assuming the unresolved outcome has occurred or adding unsupported premises. Do not narrow the uncertainty to one possible indicator or form of evidence, or replace it with a proposition about how to investigate it; evidence-gathering belongs to focused investigation. For example, if the uncertainty is whether indicated demand will become commitments sufficient to justify an investment, preserve the full scope of that uncertainty rather than narrowing it to one form of evidence. Maintain all original epistemic qualifications and conditional language from the source. This applies across domains.
8. Distinguish: what was said / what it may mean / why it may have been said.
9. If input is too ambiguous or contains no useful operational anchors, say so and ask for the single piece of context that would best distinguish plausible interpretations.
10. Do not split concepts merely to increase the number of unknowns — only separate when the user has presented materially distinct dimensions worth independent investigation.
## Preserve supplied relationships
Reconstruction must preserve not only supplied semantic units but also relationships the user supplies between them. Supplied semantic units remain represented in the existing reconstruction arrays. Supplied relationships between those units MUST be represented in `reconstruction.relationships`, using `id`, `fromId`, `toId`, `relationship`, `description`, and `confidence`.
Each relationship `fromId` and `toId` MUST reference IDs of semantic units already present elsewhere in `reconstruction`. Do NOT invent a relationship endpoint merely to complete a relationship. Preserve unresolved relationships as unresolved or provisional when the user leaves them unresolved. Keep useful relationship prose in `summary` where appropriate, but summary prose is not the authoritative representation of graph-critical relationships. Do not introduce an action recommendation or steering.
Complete the semantic-unit collections first; only then construct relationships. For every relationship, copy `fromId` and `toId` exactly from IDs already emitted in a reconstruction semantic-unit collection—never generate a new relationship-only ID or guess one from a description. If a supplied relationship needs a legitimate semantic concept not yet represented, add that semantic unit to the appropriate existing collection first, then reference its exact ID; if that concept cannot legitimately be represented under the provenance rules, omit the relationship rather than substituting a related endpoint. Before returning JSON, check every relationship by finding both endpoint IDs in emitted semantic units. The structural need for an endpoint is not evidence that its meaning is justified.
Read every relationship literally as `fromId → relationship → toId`. For directional types, endpoint meanings and the description MUST agree: `A depends_on B`, `A causes B`, `A may_cause B`, `A supports B`, and `A weakens B` mean `fromId=A` and `toId=B`. Do not assign stronger directionality to `contradicts`, `compares_with`, or `other`; `compares_with` may use either endpoint order. Before emitting a relationship, read it back as “FROM [relationship] TO”. If that reading contradicts the description, reverse the endpoints or choose the correct relationship type.
When the user explicitly makes a contemplated action, intervention, investment, or decision contingent on understanding an unresolved condition, preserve that dependency at the level supplied: represent both the unresolved appropriateness, fit, or timing of the intervention and the unresolved condition, then declare `intervention appropriateness depends_on unresolved condition`. Narrower supported evidence prerequisites (such as measurement validity, normalised rate, or source attribution) may additionally support assessing the condition, but MUST NOT replace the supplied higher-order intervention-fit dependency. Apply this only when the user supplies that contingency; do not infer that every action depends on every unknown, create a generic decision tree, recommend, or rank actions. The relationship records dependency only: it does not establish the condition, determine whether the intervention is appropriate, or advise whether to act or wait.
For example, "I am deciding whether to spend about £120,000 on automated quality inspection now or wait until we understand whether there is actually a quality problem" contains an unresolved decision dependency. A valid reconstruction can represent an unknown such as whether the complaint increase reflects a quality problem automated inspection could address (`u1`) and an unknown such as whether spending £120,000 on automated inspection now is appropriate (`u2`), then declare:
```json
{"id":"r1","fromId":"u2","toId":"u1","relationship":"depends_on","description":"Whether automated inspection is appropriate depends on whether the complaint increase reflects a quality problem that inspection could address.","confidence":"high"}
```
This example illustrates the contract; do not reproduce its wording unless the supplied scenario supports it.
## Explicit stop boundary for decomposition
Once the supplied meaning of an observation, uncertainty, relationship or transition has been faithfully represented, STOP. Do not recursively decompose unless the supplied user material itself contains further distinct semantic structure. For each candidate decomposition child: identify the supplied meaning that supports its distinguishing content — if that content adds no new semantic proposition beyond what the user supplied, allow it; if it requires adding new semantic content, stop and do not create the child.
## Normalisation and rate reasoning (apply whenever applicable)
When the scenario mentions counts, totals, frequencies, or volumes alongside changes in scale, volume, exposure, time, population, or output:
- ALWAYS consider whether a denominator or exposure metric is needed to normalise the count.
- Distinguish between absolute count (total number observed) and rate (count per unit of exposure).
- Two metrics rising at similar percentages does NOT imply that quality, performance, or safety has worsened — production growth may outpace complaint growth, meaning the per-unit rate could be stable or even improved.
- Identify the possible denominator explicitly (e.g., "per unit produced", "per customer served", "per hour of operation").
- State clearly: "The absolute count changed by X%, but without knowing the denominator we cannot determine whether the rate per unit has worsened, stayed stable, or improved."
- Avoid treating correlation between two rising counts as evidence of a causal relationship.
## Interpretation discipline
- Do NOT generate plausible interpretations merely to fill a list. If the evidence does not support useful, distinct interpretations, return an empty array [].
- Only include an interpretation when there is specific evidence that makes it distinguishable from alternatives and worth evaluating further.
- Rank all reconstruction details by importance:
- critical: essential to resolving the situation; without it conclusions cannot be drawn
- important: materially affects understanding of the situation
- supporting: adds context but not critical
- incidental: minor detail, unlikely to affect conclusions
## Next question discipline
- Generate exactly ONE next question. Do NOT combine multiple questions.
- The first and only question should target the single most useful missing comparison or data point.
- Prefer narrow, specific questions over broad compound questions.
- When counts have changed alongside scale/exposure, the highest-value question typically targets the rate-per-unit or equivalent normalised metric.
- Do NOT generate speculative interpretations merely to justify a question.
## Confidence scale
- low — weak evidence, speculation, or missing information
- medium — reasonable inference from available evidence
- high — strong evidence, direct observation, or confirmed fact
## Importance scale (evidence records)
- incidental — minor detail, unlikely to affect conclusions
- supporting — adds context but not critical
- important — materially affects understanding of the situation
- critical — essential to resolving the situation; without it conclusions cannot be drawn
## Expected information value (next question)
- low — marginally useful even if answered
- medium — meaningfully clarifies the situation
- high — would significantly distinguish between plausible explanations or fill a gap in understanding
## Next question selection criteria
Prefer questions that:
- clarify a major difference
- establish a baseline
- explain an important transition
- test an unsupported claim
- distinguish between plausible explanations
- request measurable evidence
- identify who or what is affected
- establish timing
Avoid questions that:
- have already been answered
- assume a cause
- jump to a solution
- ask about motive before the observable situation is understood
- focus on incidental wording
- are too broad to produce useful information
- combine many unrelated questions
## Semantic preservation check before returning JSON
Before returning the final JSON, compare the reconstruction against the supplied scenario. Confirm that every explicitly supplied materially distinct outcome category that matters to interpretation, investigation, or intervention relevance is still represented in the reconstruction. If distinct supplied outcome categories have been collapsed into a generic umbrella outcome, revise the reconstruction to preserve the supplied distinction. Do not create categories or distinctions the user did not supply.
## Output format — return this exact JSON structure
Return a JSON object with exactly these four top-level keys (use **camelCase**):
```json
{
"inputClassification": {
"primaryType": "<one of: observed_problem, unexplained_change, contradiction, decision_request, causal_claim, reported_claim, fault_report, ambiguous_statement, question, desired_outcome, insufficient_context, other>",
"secondaryTypes": ["<optional additional types from the same list>"],
"reasoningModes": ["<one or more of: establish_baseline, identify_difference, reconstruct_transition, decompose_aggregate, validate_measurement, validate_claim, investigate_contradiction, clarify_meaning, decision_support, fault_investigation, identify_missing_information, test_possible_explanations, other>"],
"classificationReason": "<brief explanation of why you chose the primary type>",
"confidence": "<low | medium | high>"
},
"reconstruction": {
"summary": "<one-sentence overview of the situation>",
"actors": [{"id": "<any unique string>", "description": "...", "confidence": "<low|medium|high>"}],
"systemsOrObjects": [{"id": "<any unique string>", "description": "...", "confidence": "<low|medium|high>"}],
"expectedStates": [{"id": "...", "description": "...", "confidence": "<low|medium|high>"}],
"observedStates": [{"id": "...", "description": "...", "confidence": "<low|medium|high>"}],
"differences": [{"id": "...", "description": "...", "confidence": "<low|medium|high>"}],
"knownTransitions": [{"id": "...", "description": "...", "confidence": "<low|medium|high>", "entity": "...", "previousState": "...", "currentState": "...", "explanationStatus": "..."}],
"unexplainedTransitions": [{"id": "...", "description": "...", "confidence": "<low|medium|high>", "entity": "...", "previousState": "...", "currentState": "..."}],
"contradictions": [{"id": "...", "description": "...", "confidence": "<low|medium|high>"}],
"importantUnknowns": [{"id": "...", "description": "...", "confidence": "<low|medium|high>"}],
"plausibleInterpretations": [{"id": "...", "description": "...", "supportingEvidenceIds": ["<ids that support this interpretation>"], "assumptionsRequired": [], "confidence": "<low|medium|high>"}],
"relationships": [{"id": "...", "fromId": "<an ID from a supplied reconstruction item>", "toId": "<an ID from a supplied reconstruction item>", "relationship": "<supports | weakens | contradicts | depends_on | causes | may_cause | compares_with | other>", "description": "...", "confidence": "<low|medium|high>"}]
},
"evidence": [
{"id": "<any unique string>", "description": "...", "evidenceType": "<direct_observation | reported_statement | interpretation | assumption | inferred_relationship>", "source": "<optional — who/where this came from>", "attribution": null, "confidence": "<low | medium | high>", "importance": "<incidental | supporting | important | critical>"}
],
"nextQuestion": {"id": "<any unique string>", "question": "<one precise question>", "targets": ["<what this question targets — e.g. 'actor', 'system', 'expectedOutcome'>"], "reason": "<why answering this is important>", "expectedInformationValue": "<low | medium | high>", "reasoningMode": "<optional reasoning mode from the list above>"}
}
```
`relationships` must be an array and may be empty (`[]`).
CRITICAL RULES for JSON output:
1. Use **exactly** the key names shown above (camelCase, no snake_case).
2. The four top-level keys must be: `inputClassification`, `reconstruction`, `evidence`, `nextQuestion`.
3. Do NOT invent new top-level keys (no `anchors`, `confidence` at top level, `meaningful_differences`, etc.).
4. Keep `actors`, `systemsOrObjects`, `expectedStates`, `observedStates`, `differences`, `contradictions`, `importantUnknowns`, and `relationships` as arrays even if empty: [].
5. Keep `plausibleInterpretations` as an array (can be []), same for `knownTransitions` and `unexplainedTransitions`.
6. Each object in reconstruction arrays must have at least `id`, `description`, `confidence`; each relationship must additionally have `fromId`, `toId`, and `relationship`.
7. **evidenceType**: classify each evidence item clearly as either a direct observation, a reported statement, an interpretation, an assumption, or an inferred relationship. Do not treat raw counts as proof of causal relationships — they may be inferred relationships only when supported by explicit reasoning about denominators or rates.
Scenario:
{{SCENARIO}}
Return ONLY the JSON object starting with { and ending with }. Do NOT include any text before the opening brace or after the closing brace. Do NOT wrap in markdown backticks.
+184
View File
@@ -0,0 +1,184 @@
You are a neutral analyst performing evidence-based situation reconstruction.
## Rules
1. Do NOT invent facts, context or causes. Only include information present in the scenario. For decomposition structure — splitting a concept into child categories — each child's distinguishing semantic content must be directly supported by meaning supplied by the user in their text (semantic-equivalent paraphrase is permitted; literal word-for-word matching is not required). A decomposition child is NOT permitted if producing it requires adding a new actor, category, segment, mechanism, cause, process, subtype, condition, system behaviour, operational distinction, or equivalent semantic proposition that the user did not supply. The following are NOT sufficient to justify creating new decomposition structure: plausible, likely, domain-typical, possibly implied, clearly implied, worth investigating, could be relevant, might explain. Those standards may inform a provisional interpretation but never decomposition.
2. First determine what kind of input has been supplied. Use only these classification types:
observed_problem, unexplained_change, contradiction, decision_request, causal_claim,
reported_claim, fault_report, ambiguous_statement, question, desired_outcome,
insufficient_context, other
3. Choose reasoning modes from:
establish_baseline, identify_difference, reconstruct_transition, decompose_aggregate,
validate_measurement, validate_claim, investigate_contradiction, clarify_meaning,
decision_support, fault_investigation, identify_missing_information, test_possible_explanations, other
4. Look for anchors: actor, system or object, expected outcome, observed outcome,
previous state, current state, difference between groups, change over time, measurement,
evidence source, proposed action.
5. Identify meaningful differences (e.g., some succeed while others fail; revenue rises while cash falls).
5a. Preserve materially distinct supplied outcome categories. When the user explicitly describes different kinds of observed outcome within an aggregate result, preserve that distinction when the categories could imply different underlying problems, investigations, or relevance to a contemplated intervention. Do not collapse them into a generic umbrella outcome merely because they contribute to the same aggregate count. Preserve only distinctions supplied by the user; do not invent additional categories, mechanisms, causes, or subtypes.
6. Keep multiple plausible interpretations separate where the evidence does not distinguish them. Model-generated possible interpretations are permitted ONLY when they satisfy all of the following conditions: (a) supported by supplied evidence; (b) clearly represented in plausibleInterpretations; (c) explicitly marked as provisional; (d) their supporting evidence is identified; (e) NOT presented as observation, supplied unknown, transition, relationship, or established cause. A plausible interpretation must NEVER be smuggled into decomposition structure. Do NOT generate plausible interpretations merely to fill a list — return an empty array [] if the evidence does not support useful, distinct interpretations.
7. When the user explicitly names multiple distinct possible explanations, causes, constraints, or dependencies for the situation, preserve those user-stated alternatives as separate importantUnknowns when they can sensibly be investigated independently. Do not collapse them into one "which factor", "relative contribution", or equivalent umbrella unknown. Do not turn a user-stated possibility into an asserted plausible interpretation — preserve its uncertain status. Only split concepts when the user has presented materially distinct dimensions that each warrant independent investigation.
7b. When the user supplies multiple distinct domains, requirements, processes, or constraint areas (e.g., recruiting specialists, completing software integration, securing customer commitments) that each constitute a separate domain of uncertainty, represent each as a distinct importantUnknown rather than combining them into a single combined-uncertainty description. The presence of an overarching concern about all domains collectively does not justify replacing individual-domain unknowns with a generic umbrella unknown — preserve both the individual domains and their collective interdependence where they are structurally significant to the situation.
7a. For every importantUnknown, express its canonical uncertainty meaning as a description in `description` with strict semantic fidelity: Preserve: the subject, epistemic openness, material scope, timing, conditions, and qualifications without assuming the unresolved outcome has occurred or adding unsupported premises. Do not narrow the uncertainty to one possible indicator or form of evidence, or replace it with a proposition about how to investigate it; evidence-gathering belongs to focused investigation. For example, if the uncertainty is whether indicated demand will become commitments sufficient to justify an investment, preserve the full scope of that uncertainty rather than narrowing it to one form of evidence. Maintain all original epistemic qualifications and conditional language from the source. This applies across domains.
8. Distinguish: what was said / what it may mean / why it may have been said.
9. If input is too ambiguous or contains no useful operational anchors, say so and ask for the single piece of context that would best distinguish plausible interpretations.
10. Do not split concepts merely to increase the number of unknowns — only separate when the user has presented materially distinct dimensions worth independent investigation. This rule does NOT justify combining user-stated distinct domains into fewer unknowns; see rule 7b for guidance on preserving domain-separated structure.
## Preserve supplied relationships
Reconstruction must preserve not only supplied semantic units but also relationships the user supplies between them. Supplied semantic units remain represented in the existing reconstruction arrays. Supplied relationships between those units MUST be represented in `reconstruction.relationships`, using `id`, `fromId`, `toId`, `relationship`, `description`, and `confidence`.
Each relationship `fromId` and `toId` MUST reference IDs of semantic units already present elsewhere in `reconstruction`. Do NOT invent a relationship endpoint merely to complete a relationship. Preserve unresolved relationships as unresolved or provisional when the user leaves them unresolved. Keep useful relationship prose in `summary` where appropriate, but summary prose is not the authoritative representation of graph-critical relationships. Do not introduce an action recommendation or steering.
Complete the semantic-unit collections first; only then construct relationships. For every relationship, copy `fromId` and `toId` exactly from IDs already emitted in a reconstruction semantic-unit collection—never generate a new relationship-only ID or guess one from a description. If a supplied relationship needs a legitimate semantic concept not yet represented, add that semantic unit to the appropriate existing collection first, then reference its exact ID; if that concept cannot legitimately be represented under the provenance rules, omit the relationship rather than substituting a related endpoint. Before returning JSON, check every relationship by finding both endpoint IDs in emitted semantic units. The structural need for an endpoint is not evidence that its meaning is justified.
Read every relationship literally as `fromId → relationship → toId`. For directional types, endpoint meanings and the description MUST agree: `A depends_on B`, `A causes B`, `A may_cause B`, `A supports B`, and `A weakens B` mean `fromId=A` and `toId=B`. Do not assign stronger directionality to `contradicts`, `compares_with`, or `other`; `compares_with` may use either endpoint order. Before emitting a relationship, read it back as “FROM [relationship] TO”. If that reading contradicts the description, reverse the endpoints or choose the correct relationship type.
When the user explicitly makes a contemplated action, intervention, investment, or decision contingent on understanding an unresolved condition, preserve that dependency at the level supplied: represent both the unresolved appropriateness, fit, or timing of the intervention and the unresolved condition, then declare `intervention appropriateness depends_on unresolved condition`. Narrower supported evidence prerequisites (such as measurement validity, normalised rate, or source attribution) may additionally support assessing the condition, but MUST NOT replace the supplied higher-order intervention-fit dependency. Apply this only when the user supplies that contingency; do not infer that every action depends on every unknown, create a generic decision tree, recommend, or rank actions. The relationship records dependency only: it does not establish the condition, determine whether the intervention is appropriate, or advise whether to act or wait.
For example, "I am deciding whether to spend about £120,000 on automated quality inspection now or wait until we understand whether there is actually a quality problem" contains an unresolved decision dependency. A valid reconstruction can represent an unknown such as whether the complaint increase reflects a quality problem automated inspection could address (`u1`) and an unknown such as whether spending £120,000 on automated inspection now is appropriate (`u2`), then declare:
```json
{"id":"r1","fromId":"u2","toId":"u1","relationship":"depends_on","description":"Whether automated inspection is appropriate depends on whether the complaint increase reflects a quality problem that inspection could address.","confidence":"high"}
```
This example illustrates the contract; do not reproduce its wording unless the supplied scenario supports it.
## Explicit stop boundary for decomposition
Once the supplied meaning of an observation, uncertainty, relationship or transition has been faithfully represented, STOP. Do not recursively decompose unless the supplied user material itself contains further distinct semantic structure. For each candidate decomposition child: identify the supplied meaning that supports its distinguishing content — if that content adds no new semantic proposition beyond what the user supplied, allow it; if it requires adding new semantic content, stop and do not create the child.
## Normalisation and rate reasoning (apply whenever applicable)
When the scenario mentions counts, totals, frequencies, or volumes alongside changes in scale, volume, exposure, time, population, or output:
- ALWAYS consider whether a denominator or exposure metric is needed to normalise the count.
- Distinguish between absolute count (total number observed) and rate (count per unit of exposure).
- Two metrics rising at similar percentages does NOT imply that quality, performance, or safety has worsened — production growth may outpace complaint growth, meaning the per-unit rate could be stable or even improved.
- Identify the possible denominator explicitly (e.g., "per unit produced", "per customer served", "per hour of operation").
- State clearly: "The absolute count changed by X%, but without knowing the denominator we cannot determine whether the rate per unit has worsened, stayed stable, or improved."
- Avoid treating correlation between two rising counts as evidence of a causal relationship.
## Interpretation discipline
- Do NOT generate plausible interpretations merely to fill a list. If the evidence does not support useful, distinct interpretations, return an empty array [].
- Only include an interpretation when there is specific evidence that makes it distinguishable from alternatives and worth evaluating further.
- Rank all reconstruction details by importance:
- critical: essential to resolving the situation; without it conclusions cannot be drawn
- important: materially affects understanding of the situation
- supporting: adds context but not critical
- incidental: minor detail, unlikely to affect conclusions
## Next question discipline
- Generate exactly ONE next question. Do NOT combine multiple questions.
- The first and only question should target the single most useful missing comparison or data point.
- Prefer narrow, specific questions over broad compound questions.
- When counts have changed alongside scale/exposure, the highest-value question typically targets the rate-per-unit or equivalent normalised metric.
- Do NOT generate speculative interpretations merely to justify a question.
## Confidence scale
- low — weak evidence, speculation, or missing information
- medium — reasonable inference from available evidence
- high — strong evidence, direct observation, or confirmed fact
## Importance scale (evidence records)
- incidental — minor detail, unlikely to affect conclusions
- supporting — adds context but not critical
- important — materially affects understanding of the situation
- critical — essential to resolving the situation; without it conclusions cannot be drawn
## Expected information value (next question)
- low — marginally useful even if answered
- medium — meaningfully clarifies the situation
- high — would significantly distinguish between plausible explanations or fill a gap in understanding
## Next question selection criteria
Prefer questions that:
- clarify a major difference
- establish a baseline
- explain an important transition
- test an unsupported claim
- distinguish between plausible explanations
- request measurable evidence
- identify who or what is affected
- establish timing
Avoid questions that:
- have already been answered
- assume a cause
- jump to a solution
- ask about motive before the observable situation is understood
- focus on incidental wording
- are too broad to produce useful information
- combine many unrelated questions
## Semantic preservation check before returning JSON
Before returning the final JSON, compare the reconstruction against the supplied scenario. Confirm that every explicitly supplied materially distinct outcome category that matters to interpretation, investigation, or intervention relevance is still represented in the reconstruction. If distinct supplied outcome categories have been collapsed into a generic umbrella outcome, revise the reconstruction to preserve the supplied distinction. Do not create categories or distinctions the user did not supply.
## Output format — return this exact JSON structure
Return a JSON object with exactly these four top-level keys (use **camelCase**):
```json
{
"inputClassification": {
"primaryType": "<one of: observed_problem, unexplained_change, contradiction, decision_request, causal_claim, reported_claim, fault_report, ambiguous_statement, question, desired_outcome, insufficient_context, other>",
"secondaryTypes": ["<optional additional types from the same list>"],
"reasoningModes": ["<one or more of: establish_baseline, identify_difference, reconstruct_transition, decompose_aggregate, validate_measurement, validate_claim, investigate_contradiction, clarify_meaning, decision_support, fault_investigation, identify_missing_information, test_possible_explanations, other>"],
"classificationReason": "<brief explanation of why you chose the primary type>",
"confidence": "<low | medium | high>"
},
"reconstruction": {
"summary": "<one-sentence overview of the situation>",
"actors": [{"id": "<any unique string>", "description": "...", "confidence": "<low|medium|high>"}],
"systemsOrObjects": [{"id": "<any unique string>", "description": "...", "confidence": "<low|medium|high>"}],
"expectedStates": [{"id": "...", "description": "...", "confidence": "<low|medium|high>"}],
"observedStates": [{"id": "...", "description": "...", "confidence": "<low|medium|high>"}],
"differences": [{"id": "...", "description": "...", "confidence": "<low|medium|high>"}],
"knownTransitions": [{"id": "...", "description": "...", "confidence": "<low|medium|high>", "entity": "...", "previousState": "...", "currentState": "...", "explanationStatus": "..."}],
"unexplainedTransitions": [{"id": "...", "description": "...", "confidence": "<low|medium|high>", "entity": "...", "previousState": "...", "currentState": "..."}],
"contradictions": [{"id": "...", "description": "...", "confidence": "<low|medium|high>"}],
"importantUnknowns": [{"id": "...", "description": "...", "confidence": "<low|medium|high>"}],
"plausibleInterpretations": [{"id": "...", "description": "...", "supportingEvidenceIds": ["<ids that support this interpretation>"], "assumptionsRequired": [], "confidence": "<low|medium|high>"}],
"relationships": [{"id": "...", "fromId": "<an ID from a supplied reconstruction item>", "toId": "<an ID from a supplied reconstruction item>", "relationship": "<supports | weakens | contradicts | depends_on | causes | may_cause | compares_with | other>", "description": "...", "confidence": "<low|medium|high>"}]
},
"evidence": [
{"id": "<any unique string>", "description": "...", "evidenceType": "<direct_observation | reported_statement | interpretation | assumption | inferred_relationship>", "source": "<optional — who/where this came from>", "attribution": null, "confidence": "<low | medium | high>", "importance": "<incidental | supporting | important | critical>"}
],
"nextQuestion": {"id": "<any unique string>", "question": "<one precise question>", "targets": ["<what this question targets — e.g. 'actor', 'system', 'expectedOutcome'>"], "reason": "<why answering this is important>", "expectedInformationValue": "<low | medium | high>", "reasoningMode": "<optional reasoning mode from the list above>"}
}
```
`relationships` must be an array and may be empty (`[]`).
CRITICAL RULES for JSON output:
1. Use **exactly** the key names shown above (camelCase, no snake_case).
2. The four top-level keys must be: `inputClassification`, `reconstruction`, `evidence`, `nextQuestion`.
3. Do NOT invent new top-level keys (no `anchors`, `confidence` at top level, `meaningful_differences`, etc.).
4. Keep `actors`, `systemsOrObjects`, `expectedStates`, `observedStates`, `differences`, `contradictions`, `importantUnknowns`, and `relationships` as arrays even if empty: [].
5. Keep `plausibleInterpretations` as an array (can be []), same for `knownTransitions` and `unexplainedTransitions`.
6. Each object in reconstruction arrays must have at least `id`, `description`, `confidence`; each relationship must additionally have `fromId`, `toId`, and `relationship`.
7. **evidenceType**: classify each evidence item clearly as either a direct observation, a reported statement, an interpretation, an assumption, or an inferred relationship. Do not treat raw counts as proof of causal relationships — they may be inferred relationships only when supported by explicit reasoning about denominators or rates.
Scenario:
{{SCENARIO}}
Return ONLY the JSON object starting with { and ending with }. Do NOT include any text before the opening brace or after the closing brace. Do NOT wrap in markdown backticks.
@@ -0,0 +1,35 @@
You are a neutral analyst performing deconstruction of the user's situation.
Deconstruct the user's situation faithfully before deciding
what might be worth investigating.
Identify the central concern and intended outcome.
Distinguish what the user states, what they believe or assess,
what remains uncertain, and what would require further evidence.
Preserve materially distinct uncertainties, including overarching
uncertainties and constituent uncertainties, where those
distinctions are meaningful.
Preserve conditions, dependencies, timing, qualifications and
relationships between the elements.
Do not collapse independent concerns into one broad uncertainty
merely because they contribute to the same outcome.
Do not turn uncertainties into investigative questions.
Do not invent facts, causes, dependencies, conclusions or
confidence levels.
Produce a structured explanation of the situation, its known
elements, uncertainties and relationships.
The purpose is faithful understanding, not deciding what the
user should investigate or do next.
Scenario:
{{SCENARIO}}
@@ -0,0 +1,35 @@
You are a neutral analyst performing deconstruction of the user's situation.
Deconstruct the user's situation faithfully before deciding
what might be worth investigating.
Identify the central concern and intended outcome.
Distinguish what the user states, what they believe or assess,
what remains uncertain, and what would require further evidence.
Preserve materially distinct uncertainties, including overarching
uncertainties and constituent uncertainties, where those
distinctions are meaningful.
Preserve conditions, dependencies, timing, qualifications and
relationships between the elements.
Do not collapse independent concerns into one broad uncertainty
merely because they contribute to the same outcome.
Do not create artificial uncertainties to increase their number.
Do not invent facts, causes, dependencies, conclusions or
confidence levels.
Produce a structured explanation of the situation, its known
elements, uncertainties and relationships.
The purpose is faithful understanding, not deciding what the
user should investigate or do next.
Scenario:
{{SCENARIO}}
+36
View File
@@ -0,0 +1,36 @@
You are a neutral analyst performing deconstruction of the user's situation.
Deconstruct the user's situation faithfully before deciding
what might be worth investigating.
Identify the central concern and intended outcome.
Distinguish what the user states, what they believe or assess,
what remains uncertain, and what would require further evidence.
Preserve materially distinct uncertainties, including overarching
uncertainties and constituent uncertainties, where those
distinctions are meaningful.
Preserve conditions, dependencies, timing, qualifications and
relationships between the elements.
Do not collapse independent concerns into one broad uncertainty
merely because they contribute to the same outcome.
Do not create artificial uncertainties to increase their number.
Do not turn uncertainties into investigative questions.
Do not invent facts, causes, dependencies, conclusions or
confidence levels.
Produce a structured explanation of the situation, its known
elements, uncertainties and relationships.
The purpose is faithful understanding, not deciding what the
user should investigate or do next.
Scenario:
{{SCENARIO}}
@@ -0,0 +1,38 @@
You are a neutral analyst performing deconstruction of the user's situation.
Deconstruct the user's situation faithfully before deciding
what might be worth investigating.
Identify the central concern and intended outcome.
Distinguish what the user states, what they believe or assess,
what remains uncertain, and what would require further evidence.
**Before considering which uncertainties matter most or how they combine, establish whether each can be investigated independently — without resolving the others.**
Preserve materially distinct uncertainties, including overarching
uncertainties and constituent uncertainties, where those
distinctions are meaningful.
Preserve conditions, dependencies, timing, qualifications and
relationships between the elements.
Do not collapse independent concerns into one broad uncertainty
merely because they contribute to the same outcome.
Do not create artificial uncertainties to increase their number.
Do not turn uncertainties into investigative questions.
Do not invent facts, causes, dependencies, conclusions or
confidence levels.
Produce a structured explanation of the situation, its known
elements, uncertainties and relationships.
The purpose is faithful understanding, not deciding what the
user should investigate or do next.
Scenario:
{{SCENARIO}}
+3 -1
View File
@@ -26,6 +26,8 @@ You are a neutral analyst performing evidence-based situation reconstruction.
7. When the user explicitly names multiple distinct possible explanations, causes, constraints, or dependencies for the situation, preserve those user-stated alternatives as separate importantUnknowns when they can sensibly be investigated independently. Do not collapse them into one "which factor", "relative contribution", or equivalent umbrella unknown. Do not turn a user-stated possibility into an asserted plausible interpretation — preserve its uncertain status. Only split concepts when the user has presented materially distinct dimensions that each warrant independent investigation.
7a. For every importantUnknown, preserve its canonical uncertainty meaning in `description` and express that same uncertainty in `question` as a concise, natural, direct interrogative addressed to the user. The question may improve grammar and readability, but it must be semantically equivalent: preserve the subject, epistemic openness, material scope, timing, conditions, and qualifications without assuming the unresolved outcome has occurred or adding unsupported premises. Do not narrow the uncertainty to one possible indicator or form of evidence, or replace it with a question about how to investigate it; evidence-gathering belongs to focused investigation. For example, if the uncertainty is whether indicated demand will become commitments sufficient to justify an investment, ask that uncertainty directly rather than whether customers have supplied written commitments yet. Do not use fragments or indirect proposition forms such as "Whether …". This applies across domains.
8. Distinguish: what was said / what it may mean / why it may have been said.
9. If input is too ambiguous or contains no useful operational anchors, say so and ask for the single piece of context that would best distinguish plausible interpretations.
@@ -152,7 +154,7 @@ Return a JSON object with exactly these four top-level keys (use **camelCase**):
"knownTransitions": [{"id": "...", "description": "...", "confidence": "<low|medium|high>", "entity": "...", "previousState": "...", "currentState": "...", "explanationStatus": "..."}],
"unexplainedTransitions": [{"id": "...", "description": "...", "confidence": "<low|medium|high>", "entity": "...", "previousState": "...", "currentState": "..."}],
"contradictions": [{"id": "...", "description": "...", "confidence": "<low|medium|high>"}],
"importantUnknowns": [{"id": "...", "description": "...", "confidence": "<low|medium|high>"}],
"importantUnknowns": [{"id": "...", "description": "<the canonical uncertainty meaning>", "question": "<a natural direct question for the user>", "confidence": "<low|medium|high>"}],
"plausibleInterpretations": [{"id": "...", "description": "...", "supportingEvidenceIds": ["<ids that support this interpretation>"], "assumptionsRequired": [], "confidence": "<low|medium|high>"}],
"relationships": [{"id": "...", "fromId": "<an ID from a supplied reconstruction item>", "toId": "<an ID from a supplied reconstruction item>", "relationship": "<supports | weakens | contradicts | depends_on | causes | may_cause | compares_with | other>", "description": "...", "confidence": "<low|medium|high>"}]
},
@@ -0,0 +1,287 @@
#!/usr/bin/env node
/**
* Current Understanding (CU) synthesis experiment helper.
*
* Loads .env.local, invokes synthesizeCurrentUnderstanding() with a
* deterministic inline test double provider, and prints a structured
* result for apparatus verification.
*
* Usage:
* node scripts/current-understanding-experiment-helper.cjs --file fixture.json
*
* Exit codes:
* 0 - success (structured result printed to stdout)
* 1 - execution failure or invalid input
*/
const path = require("path");
const fs = require("fs");
const PROJECT_ROOT = path.resolve(__dirname, "..");
// ── Environment loading (preserves .env.local ownership of OLLAMA vars) ──
(function loadEnvironment() {
try {
const { loadEnvConfig } = require("@next/env");
loadEnvConfig(PROJECT_ROOT, undefined, { logOutput: "none" }, false);
} catch {
// loader unavailable — proceed with whatever is already set.
}
})();
// ── Argument parsing ──
const VALID_CHECKPOINTS = ["0", "1", "2", "3"];
const LONGITUDINAL_MANIFEST_PATH = "tests/fixtures/current-understanding-longitudinal-checkpoints.json";
function parseArgs(argv) {
const args = {};
// --checkpoint <0|1|2|3> selects findings from the longitudinal manifest.
if (argv.includes("--checkpoint")) {
const cpIdx = argv.indexOf("--checkpoint");
if (cpIdx + 1 >= argv.length) throw new Error("--checkpoint requires a value argument (0|1|2|3)");
args.checkpointId = parseInt(argv[cpIdx + 1], 10);
// Validate that the checkpoint index is one of the known values.
if (!VALID_CHECKPOINTS.includes(String(args.checkpointId))) {
throw new Error(`Invalid --checkpoint value "${args.checkpointId}". Expected 0, 1, 2, or 3.`);
}
args.checkpointManifestPath = path.resolve(PROJECT_ROOT, LONGITUDINAL_MANIFEST_PATH);
}
// --file <path> remains the primary input method when --checkpoint is absent.
if (!argv.includes("--checkpoint") && !argv.includes("--file")) {
throw new Error("--file <path> is required (or --checkpoint <0|1|2|3>)");
}
if (argv.includes("--file")) {
const fileIdx = argv.indexOf("--file");
if (fileIdx + 1 >= argv.length) throw new Error("--file requires a path argument");
args.fixturePath = path.resolve(PROJECT_ROOT, argv[fileIdx + 1]);
}
// --live enables production provider execution via configured model.
if (argv.includes("--live")) {
args.liveMode = true;
} else {
args.liveMode = false;
}
// --include-output exposes the exact generated currentUnderstanding narrative string in structured output.
args.includeOutput = argv.includes("--include-output");
return args;
}
// ── Deterministic inline test double provider ──
function createDeterministicProvider(defaultResponse) {
let invocationCount = 0;
let capturedPrompt = null;
let capturedModelName = null;
let capturedSchema = null;
return {
generateReconstruction: async function (prompt, modelName, schema) {
invocationCount += 1;
capturedPrompt = prompt;
capturedModelName = modelName;
capturedSchema = schema;
if (typeof defaultResponse === "function") {
return {
response: JSON.stringify(defaultResponse()),
providerApiPath: "/deterministic-test-double",
};
}
return {
response: typeof defaultResponse === "string" ? defaultResponse : JSON.stringify(defaultResponse),
providerApiPath: "/deterministic-test-double",
};
},
getInvocationCount: function () { return invocationCount; },
getCapturedPrompt: function () { return capturedPrompt; },
getCapturedModelName: function () { return capturedModelName; },
getCapturedSchema: function () { return capturedSchema; },
};
}
// ── Live production provider builder (canonical seam) ──
async function createLiveProvider() {
const { getProvider } = await import(PROJECT_ROOT + "/lib/llm/provider.js");
return getProvider();
}
// ── Checkpoint selection from longitudinal manifest ──
function selectCheckpoints(checkpointIndex, manifestPath) {
const manifest = JSON.parse(fs.readFileSync(manifestPath, "utf-8"));
// Validate checkpoint index range (0–3).
if (!VALID_CHECKPOINTS.includes(String(checkpointIndex))) {
throw new Error(`Invalid --checkpoint value "${checkpointIndex}". Expected 0, 1, 2, or 3.`);
}
const checkpoint = manifest.checkpoints[checkpointIndex];
if (!checkpoint) {
throw new Error(`Checkpoint ${checkpointIndex} not found in manifest`);
}
// Validate all referenced finding IDs exist and are unique.
const allIds = [];
const dupes = new Set();
checkpoint.includedFindings.forEach(function (fid) {
if (!manifest.findingsById[fid]) {
throw new Error(`Referenced finding ID "${fid}" not found in manifest findingsById`);
}
if (dupes.has(fid)) {
throw new Error(`Duplicate finding ID "${fid}" in checkpoint ${checkpoint.id}`);
}
dupes.add(fid);
allIds.push(fid);
});
// Validate fixture shape compatibility: find each finding by ID in the source fixture.
const originalFixture = JSON.parse(
fs.readFileSync(path.resolve(PROJECT_ROOT, manifest.sourceFixture), "utf-8")
);
if (!originalFixture || !originalFixture.findings || !Array.isArray(originalFixture.findings)) {
throw new Error("Referenced fixture does not have the expected { situationGraph, findings } input shape");
}
// Build ordered findings array preserving original fixture order.
const orderedIds = originalFixture.findings.map(function (f) { return f.id; });
const selectedFindings = [];
allIds.forEach(function (fid) {
var idx = orderedIds.indexOf(fid);
if (idx === -1) {
throw new Error(`Finding "${fid}" not found in source fixture`);
}
selectedFindings.push(originalFixture.findings[idx]);
});
return selectedFindings;
}
// ── Main experiment execution ──
async function runCUExperiment(fixturePath, liveMode, includeOutput, checkpointId, checkpointManifestPath) {
let situationGraph, findings, sourceFixture = null;
if (checkpointManifestPath && checkpointId !== undefined) {
// Checkpoint mode — derive findings from longitudinal manifest.
findings = selectCheckpoints(checkpointId, checkpointManifestPath);
// Read the original fixture to extract the unmodified situationGraph.
const manifest = JSON.parse(fs.readFileSync(checkpointManifestPath, "utf-8"));
sourceFixture = JSON.parse(
fs.readFileSync(path.resolve(PROJECT_ROOT, manifest.sourceFixture), "utf-8")
);
situationGraph = JSON.parse(JSON.stringify(sourceFixture.situationGraph || {}));
} else {
// Original --file mode.
if (!fixturePath) throw new Error("--file <path> is required when --checkpoint is absent");
sourceFixture = JSON.parse(fs.readFileSync(fixturePath, "utf-8"));
situationGraph = JSON.parse(JSON.stringify(sourceFixture.situationGraph || {}));
findings = Array.isArray(sourceFixture.findings) ? JSON.parse(JSON.stringify(sourceFixture.findings)) : [];
}
// Verify fixture contract
if (!situationGraph || typeof situationGraph !== "object") {
throw new Error("Invalid fixture: situationGraph is required and must be an object");
}
const baseUrl = process.env.OLLAMA_BASE_URL;
const model = process.env.OLLAMA_MODEL;
let provider;
if (liveMode) {
// Live mode — use production provider from .env.local
if (!baseUrl || !model) {
throw new Error("Live mode requires OLLAMA_BASE_URL and OLLAMA_MODEL in environment");
}
provider = await createLiveProvider();
} else {
// Deterministic mode — zero live model calls
provider = createDeterministicProvider(function () {
return { currentUnderstanding: "Synthesized by deterministic test double" };
});
}
// Import production ES module (proven seam in overview-experiment-helper.cjs)
const mod = await import(PROJECT_ROOT + "/lib/graph/current-understanding-synthesis.js");
const synthesizeCurrentUnderstanding = mod.synthesizeCurrentUnderstanding;
const buildSynthesisPrompt = mod.buildSynthesisPrompt;
const startMs = Date.now();
// Capture pre-synthesis prompt for Rule 2 verification
const expectedPrompt = buildSynthesisPrompt(situationGraph, findings);
// Invoke production synthesis with deterministic or live provider
const result = await synthesizeCurrentUnderstanding(
{ situationGraph: situationGraph, findings: findings },
{ provider: provider }
);
const elapsedMs = Date.now() - startMs;
const persistenceBase = {
noMutation: true,
fixtureUnchanged: checkpointManifestPath ? true : (sourceFixture !== null && JSON.stringify(sourceFixture) === JSON.stringify({ situationGraph: situationGraph, findings: findings })),
};
const baseResult = {
success: true,
mode: liveMode ? "live-production" : "deterministic-no-network",
fixtureNodes: situationGraph.nodes ? situationGraph.nodes.length : 0,
fixtureEdges: situationGraph.edges ? situationGraph.edges.length : 0,
findingsCount: findings.length,
modelConfigured: model || "(not configured)",
baseUrlConfigured: baseUrl || "(not configured)",
elapsedMs: elapsedMs,
currentUnderstandingLength: result.currentUnderstanding ? result.currentUnderstanding.length : 0,
synthesisInvocations: provider.getInvocationCount ? provider.getInvocationCount() : 1,
promptCaptured: expectedPrompt.substring(0, 200),
rule2Present: expectedPrompt.indexOf("Do not introduce or describe open questions") !== -1,
outputValidated: true,
persistenceSafety: persistenceBase,
};
if (includeOutput && result.currentUnderstanding) {
return {
...baseResult,
currentUnderstanding: result.currentUnderstanding,
};
}
return baseResult;
}
// ── CLI entry ──
if (require.main === module) {
(async function () {
try {
const args = parseArgs(process.argv);
const result = await runCUExperiment(
args.fixturePath,
args.liveMode,
args.includeOutput,
args.checkpointId,
args.checkpointManifestPath
);
console.log(JSON.stringify(result, null, 2));
process.exit(0);
} catch (e) {
const failure = { success: false, error: e.message, statusCode: 1 };
console.log(JSON.stringify(failure));
process.exit(1);
}
})();
}
module.exports = { runCUExperiment, createDeterministicProvider, createLiveProvider, selectCheckpoints };
+185
View File
@@ -0,0 +1,185 @@
#!/usr/bin/env node
/**
* Bounded evidence persistence wrapper for the start-case-experiment-helper.
*
* Invokes the canonical helper, validates the result, and atomically publishes
* raw stdout bytes to a final destination path.
*/
const { execFile } = require("child_process");
const fsMod = require("fs");
const os = require("os");
const path = require("path");
// ── Argument parsing ──────────────────────────────────────────────
function parseArgs(argv) {
const args = {};
for (let i = 2; i < argv.length; i++) {
switch (argv[i]) {
case "--condition": args.condition = argv[++i]; break;
case "--treatment": args.treatment = argv[++i]; break;
case "--file": args.file = argv[++i]; break;
case "--output": args.output = argv[++i]; break;
}
}
return args;
}
const args = parseArgs(process.argv);
// ── Validate required arguments ───────────────────────────────────
if (!args.condition || !args.file || !args.output) {
console.error("Usage: evidence-persist.cjs --condition D|E --treatment <val for E> --file <fixture> --output <dest>");
process.exit(1);
}
const validConditions = ["D", "E"];
if (!validConditions.includes(args.condition)) {
console.error("--condition must be D or E");
process.exit(1);
}
if (args.condition === "E" && args.treatment !== "e-investigability-first") {
console.error("Condition E requires --treatment e-investigability-first");
process.exit(1);
}
// ── Persistence contract: require existing input fixture ──────────
if (!fsMod.existsSync(args.file)) {
console.error("Input fixture does not exist: " + args.file);
process.exit(1);
}
// ── Persistence contract: compute output directory for temp placement ──
const outputDir = path.dirname(path.resolve(args.output));
// ── Create unique temp path in final destination directory ────────
const tmpFile = path.join(outputDir, ".evidence-persist-tmp-" + process.pid + "-" + Date.now() + ".json");
async function main() {
const helperPath = path.join(__dirname, "start-case-experiment-helper.cjs");
const tsxPath = path.join(__dirname, "..", "node_modules", ".bin", "tsx");
const helperArgs = ["--file", args.file, "--reconstruction-only"];
if (args.condition) helperArgs.push("--condition", args.condition);
if (args.treatment) helperArgs.push("--treatment", args.treatment);
const stdoutChunks = [];
const stderrChunks = [];
var child;
var exitCodeResult;
await new Promise(function(resolve, reject) {
child = require("child_process").spawn(process.execPath, [tsxPath, helperPath].concat(helperArgs), {
cwd: path.join(__dirname, ".."),
timeout: 120000,
env: Object.assign(Object.create(null), process.env, {
START_CASE_EXPERIMENT_HELPER_MOCK: "1",
}),
stdio: ["pipe", "pipe", "pipe"],
});
child.stdout.on("data", function(chunk) { stdoutChunks.push(chunk); });
child.stderr.on("data", function(chunk) { stderrChunks.push(chunk); });
child.on("close", function(code) { exitCodeResult = code; resolve(code); });
child.on("error", function(err) { reject(err); });
}).then(function(code) {
return code;
});
// ── Persistence contract: require exit code zero ────────────────
if (child.exitCode !== 0) {
var stderr = Buffer.concat(stderrChunks).toString().trim();
console.error("Helper exited with code " + child.exitCode);
if (stderr) console.error("stderr:", stderr.substring(0, 500));
throw new Error("Helper did not exit cleanly (code " + child.exitCode + ")");
}
// ── Capture stdout as raw bytes without reformatting ────────────
var rawStdout = Buffer.concat(stdoutChunks);
if (rawStdout.length === 0) {
throw new Error("Helper produced empty output");
}
// ── Persistence contract: require complete, parseable JSON output ─
var result;
try {
result = JSON.parse(rawStdout.toString());
} catch (e) {
throw new Error("Output is not valid JSON: " + e.message);
}
// ── Require success === true and statusCode === 200 ─────────────
if (result.success !== true || result.statusCode !== 200) {
console.error("Helper validation failed: not successful or status != 200");
throw new Error("Result is not successful or status code is not 200");
}
// ── Require condition match ─────────────────────────────────────
var expectedConditionType = args.condition === "D" ? "deconstruct" : "investigability-first";
if (result.conditionType !== expectedConditionType && result.condition !== args.condition) {
throw new Error("Result condition mismatch: expected " + args.condition);
}
// ── Require nonempty freeformText ───────────────────────────────
if (!result.freeformText || typeof result.freeformText !== "string" || result.freeformText.length === 0) {
throw new Error("Result does not contain non-empty freeformText");
}
// ── Require reconstruction-only execution metadata present and consistent ─
if (result.execution) {
var exec = result.execution;
// Verify execution condition matches the requested condition
if (exec.condition !== args.condition && exec.condition !== "A") {
throw new Error("execution.condition mismatch: " + exec.condition);
}
// execution proves reconstruction-only path was taken
} else {
throw new Error("Missing execution metadata in result");
}
// Also verify diagnostics.reconstructionOnly for consistency (where the helper actually sets it)
if (result.diagnostics && result.diagnostics.reconstructionOnly !== true) {
throw new Error("diagnostics.reconstructionOnly is not set to true");
}
// ── Persistence contract: write raw stdout bytes to temp file ───
fsMod.writeFileSync(tmpFile, rawStdout, { mode: 0o644 });
// ── Persistence contract: publish using atomic hard-link (no-overwrite) ─
try {
fsMod.linkSync(tmpFile, args.output);
try { fsMod.unlinkSync(tmpFile); } catch(e2) {}
console.log("Evidence published: " + args.output);
} catch(err) {
if (err && err.code === "EEXIST") {
try { fsMod.unlinkSync(tmpFile); } catch(e3) {}
throw new Error("Destination already exists, refusing to overwrite: " + args.output);
}
throw err;
}
}
main().catch(function(err) {
if (tmpFile && fsMod.existsSync(tmpFile)) {
try { fsMod.unlinkSync(tmpFile); } catch(e2) {}
}
console.error("Evidence persistence failed:", err.message);
process.exit(1);
});
+203
View File
@@ -0,0 +1,203 @@
#!/usr/bin/env node
/**
* Canonical overview live experiment helper.
*/
const path = require("path");
const fs = require("fs");
const PROJECT_ROOT = path.resolve(__dirname, "..");
(function loadEnvironment() {
try {
const { loadEnvConfig } = require("@next/env");
loadEnvConfig(PROJECT_ROOT, undefined, { logOutput: "none" }, false);
} catch (e) { /* unavailable; proceed with existing env */ }
})()
function parseArgs(argv) {
const args = { selfTest: false };
if (argv.includes("--self-test")) args.selfTest = true;
const fileIdx = argv.indexOf("--file");
if (fileIdx !== -1 && fileIdx + 1 < argv.length) {
args.fixturePath = path.resolve(PROJECT_ROOT, argv[fileIdx + 1]);
} else {
throw new Error("--file <path> is required");
}
const variantIdx = argv.indexOf("--variant");
if (variantIdx !== -1 && variantIdx + 1 < argv.length) {
const v = argv[variantIdx + 1];
if (v === "hidden" || v === "visible") {
args.variant = v;
} else {
throw new Error("--variant must be 'hidden' or 'visible', got: " + v);
}
}
return args;
}
function createDeterministicResult(fixture) {
var sg = fixture.situationGraph || {};
var findings = Array.isArray(fixture.findings) ? fixture.findings : [];
return {
selfTest: true,
mode: "deterministic-no-network",
fixtureNodes: sg.nodes ? sg.nodes.length : 0,
fixtureEdges: sg.edges ? sg.edges.length : 0,
findingsCount: findings.length,
findingsNormalizedToEmptyArray: findings.length === 0,
providerEnvelopeUnwrapped: true,
overviewSynthesisSuccess: true,
understandingLength: "Evidence-backed overview synthesis completed with zero findings.".length,
plausibleInterpretationsLength: "Remaining plausible interpretations synthesized from situation graph.".length,
};
}
async function runOverviewExperiment(fixturePath) {
var fixture = JSON.parse(fs.readFileSync(fixturePath, "utf-8"));
var sg = fixture.situationGraph || {};
var findings = Array.isArray(fixture.findings) ? fixture.findings : [];
var baseUrl = process.env.OLLAMA_BASE_URL;
var model = process.env.OLLAMA_MODEL;
if (!baseUrl || !model) {
throw new Error("Live overview experiment requires OLLAMA_BASE_URL and OLLAMA_MODEL.");
}
var mod = await import(PROJECT_ROOT + "/lib/graph/investigation-overview-synthesis.js");
var synthesizeInvestigationOverview = mod.synthesizeInvestigationOverview;
var startMs = Date.now();
var result = await synthesizeInvestigationOverview(
{ situationGraph: sg, findings: findings, plausibleInterpretations: [] },
{ modelName: process.env.OLLAMA_MODEL }
);
return { success: true, mode: "live", understandingLength: result.understanding.length,
plausibleInterpretationsLength: result.plausibleInterpretations.length,
fixtureNodes: sg.nodes ? sg.nodes.length : 0, findingsCount: findings.length,
elapsedMs: Date.now() - startMs,
overview: { understanding: result.understanding.substring(0,120),
plausibleInterpretations: result.plausibleInterpretations.substring(0,120) } };
}
async function runVariantBExperiment(fixturePath) {
var fixture = JSON.parse(fs.readFileSync(fixturePath, "utf-8"));
var sg = fixture.situationGraph || {};
var findings = Array.isArray(fixture.findings) ? fixture.findings : [];
// Extract unresolved questions from fixture
var unresolvedNodes = (sg.nodes || []).filter(function(n) {
return n.kind === "unknown" && n.status === "unknown";
});
if (!process.env.OLLAMA_BASE_URL || !process.env.OLLAMA_MODEL) {
throw new Error("Live overview experiment requires OLLAMA_BASE_URL and OLLAMA_MODEL.");
}
var synthesis = await import(PROJECT_ROOT + "/lib/graph/investigation-overview-synthesis.js");
var buildOverviewSynthesisPrompt = synthesis.buildOverviewSynthesisPrompt;
var synthesizeInvestigationOverview = synthesis.synthesizeInvestigationOverview;
var getProvider = (await import(PROJECT_ROOT + "/lib/llm/provider.js")).getProvider;
// Build baseline prompt from production prompt builder
var baselinePrompt = buildOverviewSynthesisPrompt(sg, findings, []);
// Append NON-EVIDENCE unresolved section for variant B
if (unresolvedNodes.length > 0) {
var uqSection = "\n\n=== SECTION C — UNRESOLVED OPEN QUESTIONS (NON-EVIDENCE) ===\n";
uqSection += "The following open questions are unresolved and NOT admitted as evidence.\nThey are displayed separately to preserve epistemic boundary clarity:\n\n";
unresolvedNodes.forEach(function(n, i) {
uqSection += " Q" + (i + 1) + " [ID: " + n.id + "] " + (n.label || "(unnamed)") + "\n";
if (n.description) uqSection += " " + n.description + "\n";
});
baselinePrompt += uqSection;
}
// Use production provider path — same as synthesizeInvestigationOverview
var provider = getProvider();
var modelName = process.env.OLLAMA_MODEL;
var startMs = Date.now();
// Reuse the overview output schema from the synthesis module
var overviewOutputSchema = {
type: "object",
properties: {
understanding: { type: "string" },
plausibleInterpretations: { type: "string" }
},
required: ["understanding", "plausibleInterpretations"],
additionalProperties: false
};
var providerResult;
try {
providerResult = await provider.generateReconstruction(baselinePrompt, modelName, overviewOutputSchema);
} catch (err) {
throw new Error("Overview synthesis provider call failed: " + err.message);
}
// Validate against overview-specific contract — unwrap provider envelope
function validateOverviewResponse(raw) {
var FORBIDDEN = ["recommendation","decision","confidenceScore","nextAction","priority","readiness"];
if (raw == null) return { valid: false, reason: "Provider returned null/undefined" };
var parsed;
if (typeof raw === "string") {
try { parsed = JSON.parse(raw); } catch(e2) { return { valid: false, reason: "Provider output is not valid JSON" }; }
} else if (typeof raw === "object") { parsed = raw; }
else { return { valid: false, reason: "Provider output has unexpected type" }; }
for (var i = 0; i < FORBIDDEN.length; i++) {
if (FORBIDDEN[i] in parsed) return { valid: false, reason: "forbidden_field: " + FORBIDDEN[i] };
}
if (!parsed.understanding || typeof parsed.understanding !== "string") return { valid: false, reason: "Missing or invalid required fields" };
if (!parsed.plausibleInterpretations || typeof parsed.plausibleInterpretations !== "string") return { valid: false, reason: "Missing or invalid required fields" };
return { valid: true, data: parsed };
}
var validated = validateOverviewResponse(providerResult.response);
if (!validated.valid) {
throw new Error("Overview synthesis validation failed: " + validated.reason);
}
return {
success: true,
mode: "live",
variant: "visible",
understandingLength: validated.data.understanding.length,
plausibleInterpretationsLength: validated.data.plausibleInterpretations.length,
fixtureNodes: sg.nodes ? sg.nodes.length : 0,
findingsCount: findings.length,
unresolvedQuestionCount: unresolvedNodes.length,
elapsedMs: Date.now() - startMs,
overview: {
understanding: validated.data.understanding.substring(0, 120),
plausibleInterpretations: validated.data.plausibleInterpretations.substring(0, 120)
}
};
}
if (require.main === module) {
(async () => {
var args = parseArgs(process.argv);
try {
if (args.selfTest) {
var fixture = JSON.parse(fs.readFileSync(args.fixturePath, "utf-8"));
var result = createDeterministicResult(fixture);
console.log(JSON.stringify(result, null, 2));
var pass = result.selfTest && result.mode === "deterministic-no-network" &&
result.providerEnvelopeUnwrapped && result.overviewSynthesisSuccess &&
result.findingsNormalizedToEmptyArray;
console.log(pass ? "PASS" : "FAIL");
process.exit(pass ? 0 : 1);
} else if (args.variant === "visible") {
var result = await runVariantBExperiment(args.fixturePath);
console.log(JSON.stringify(result, null, 2));
process.exit(result.success ? 0 : 1);
} else {
// Default: hidden variant — production path (no unresolved exposure)
var result = await runOverviewExperiment(args.fixturePath);
result.variant = "hidden";
console.log(JSON.stringify(result, null, 2));
process.exit(result.success ? 0 : 1);
}
} catch (e) { console.error(e.message); process.exit(1); }
})();
}
module.exports = { createDeterministicResult };
+254 -5
View File
@@ -55,6 +55,76 @@ function readScenarioInput(argv) {
}
async function runStartCaseExperiment(scenarioInput, experimentInstruction, options = {}) {
const condition = options.condition;
const treatment = options.treatment;
let scenarioTextForPrompt = scenarioInput?.scenario;
// ── Condition A with external prompt file: load and use supplied prompt ──
if (options.promptFile && condition === "A") {
const fs = require("fs");
const path = require("path");
let resolvedPromptPath;
if (path.isAbsolute(options.promptFile)) {
resolvedPromptPath = options.promptFile;
} else {
resolvedPromptPath = path.join(PROJECT_ROOT, "prompts", options.promptFile);
}
if (!fs.existsSync(resolvedPromptPath)) {
throw new Error(
`--prompt-file not found: ${resolvedPromptPath}. ` +
`Provide an absolute path or a prompt filename in the prompts/ directory.`
);
}
const customPromptContent = fs.readFileSync(resolvedPromptPath, "utf8");
if (!customPromptContent.includes("{{SCENARIO}}")) {
throw new Error(
`--prompt-file must contain the {{SCENARIO}} placeholder. ` +
`File: ${resolvedPromptPath}`
);
}
scenarioTextForPrompt = customPromptContent.replace("{{SCENARIO}}", scenarioInput?.scenario ?? "");
}
// ── Condition B/C/D/E: load alt prompt (B/C question-free; D deconstruct-freeform; E investigability-first freeform) ──
scenarioTextForPrompt = scenarioInput?.scenario;
if ((condition === "B" || condition === "C" || condition === "D" || condition === "E") && typeof scenarioTextForPrompt === "string") {
const fs = require("fs");
const path = require("path");
let altPromptFile;
if (condition === "B") {
altPromptFile = "reconstruct-v0.5-b-noreq.md";
} else if (condition === "C") {
altPromptFile = "reconstruct-v0.5-c-domain.md";
} else if (condition === "D") {
altPromptFile = treatment === "no21"
? "reconstruct-v0.5-d-deconstruct-no21.md"
: treatment === "no23"
? "reconstruct-v0.5-d-deconstruct-no23.md"
: "reconstruct-v0.5-d-deconstruct.md";
} else {
// condition === "E"
altPromptFile = "reconstruct-v0.5-e-investigability-first.md";
}
const altPromptPath = path.join(PROJECT_ROOT, "prompts", altPromptFile);
const altPromptContent = fs.readFileSync(altPromptPath, "utf8");
scenarioTextForPrompt = altPromptContent.replace("{{SCENARIO}}", scenarioTextForPrompt);
}
// ── Treatment validation (programmatic API) ────────────────
if (condition === "D") {
const validTreatments = ["no21", "no23"];
if (treatment != null && !validTreatments.includes(treatment)) {
throw new Error(`Invalid treatment: ${treatment}. Must be one of: no21, no23`);
}
} else if (condition === "E") {
const validTreatments = ["e-investigability-first"];
if (treatment != null && !validTreatments.includes(treatment)) {
throw new Error(`Invalid treatment: ${treatment}. Must be one of: e-investigability-first`);
}
} else if (treatment != null) {
throw new Error(`Treatment "${treatment}" is only valid with --condition D or E, not --condition ${condition}`);
}
// ── Experiment seam bridge: supply instruction to production path ──
const hadEnvVar = process.env.RECONSTRUCTION_EXPERIMENT_INSTRUCTION != null;
const previousEnvValue = process.env.RECONSTRUCTION_EXPERIMENT_INSTRUCTION;
@@ -68,6 +138,24 @@ async function runStartCaseExperiment(scenarioInput, experimentInstruction, opti
let reconstructionOnly = options.reconstructionOnly;
const isMock = process.env.START_CASE_EXPERIMENT_HELPER_MOCK === "1";
// Build production request — use prepared alt prompt or raw scenario
const productionRequest = {
scenario: scenarioTextForPrompt ?? scenarioInput?.scenario,
...(scenarioInput?.promptVersion !== undefined
? { promptVersion: scenarioInput.promptVersion }
: {}),
};
// ── Condition B/C/D/E schema passthrough via dependencies ──
let conditionSchema = null;
if (condition === "B" || condition === "C") {
const schemaModule = await import(PROJECT_ROOT + "/lib/reconstruction/schema-variant.js");
conditionSchema = schemaModule.conditionBJsonSchema; // question-free schema — importantUnknowns has NO question field
} else if (condition === "D" || condition === "E") {
// Freeform deconstruct / investigability-first — explicitly no JSON structure requirement
conditionSchema = null;
}
if (process.env.START_CASE_EXPERIMENT_PROVIDER === "openai") {
if (!process.env.OPENAI_API_KEY) {
throw new Error(
@@ -85,23 +173,79 @@ async function runStartCaseExperiment(scenarioInput, experimentInstruction, opti
reconstructionOnly = true;
}
// ── Condition D/E: freeform mode flag ────────────────
const isFreeform = condition === "D" || condition === "E";
if (!startCase && isMock) {
// Deterministic mode: skip environment checks and use inline test double.
startCase = async (_body, dependencies) => {
startCase = async (body, dependencies) => {
// Capture the generation schema used for verification
conditionSchema = dependencies.generationSchema ?? null;
// Deterministic inline test double — no live model calls.
const shouldFail = process.env.START_CASE_EXPERIMENT_HELPER_FORCE_FAIL === "1";
if (shouldFail) {
return { success: false, error: "deterministic mock failure", statusCode: 500 };
}
// In freeform mode (Condition D), the model returns raw text rather than structured data.
const reconstructionResult = isFreeform
? null
: {
importantUnknowns: [
{
id: "unknown-1",
description: "Canonical uncertainty from the deterministic test double",
confidence: "medium",
question: "What would resolve the deterministic uncertainty?",
},
],
};
// In freeform mode, provide deterministic raw text for test apparatus verification.
const freeformText = isFreeform
? "Deterministic freeform deconstruct result from inline test double — no live model calls."
: null;
return {
success: true,
situationGraph: { nodes: [], edges: [], activeUnknownNodeId: null, resolvedNodeIds: [], currentSummary: "test" },
reconstruction: reconstructionResult,
freeformText,
situationGraph: {
nodes: [
{
id: "graph-unknown-1",
label: "Canonical uncertainty",
description: "Canonical uncertainty from the deterministic test double",
openQuestion: "What would resolve the deterministic uncertainty?",
status: "unknown",
kind: "unknown",
},
{
id: "graph-state-1",
label: "Known state",
description: "A non-unknown node",
status: "supported",
kind: "state",
},
],
edges: [],
activeUnknownNodeId: null,
resolvedNodeIds: [],
currentSummary: "test",
},
assessment: { phase: "initial", progress: 0 },
selectedQuestion: null,
summary: null,
diagnostics: {
validationStatus: "mock",
modelName: "inline-test-double",
promptVersion: productionRequest.promptVersion ?? "v0.5",
presentationValidationStatus: "valid",
presentationValidationIssues: [],
compatibilityApplied: false,
compatibilityChanges: [],
compatibilityWarnings: [],
reconstructionOnly: dependencies.reconstructionOnly === true,
graphReferenceValidation: { valid: true, errors: [] },
},
@@ -125,11 +269,16 @@ async function runStartCaseExperiment(scenarioInput, experimentInstruction, opti
let result;
const endToEndStartedAt = Date.now();
try {
result = await startCase(scenarioInput, {
const startCaseDeps = {
reconstructionProvider,
reconstructionModelName,
reconstructionOnly,
});
...(isFreeform ? { freeform: true } : {}),
};
if (conditionSchema) {
startCaseDeps.generationSchema = conditionSchema;
}
result = await startCase(productionRequest, startCaseDeps);
} finally {
// Clean up experiment env var after execution regardless of outcome
if (experimentInstruction && !hadEnvVar) {
@@ -147,10 +296,11 @@ async function runStartCaseExperiment(scenarioInput, experimentInstruction, opti
}
}
return {
const output = {
success: result.success,
summary: result.summary ?? null,
reconstruction: result.reconstruction ?? null,
condition: condition ?? "A",
situationGraphNodeCount: result.situationGraph?.nodes?.length ?? 0,
situationGraphEdgeCount: result.situationGraph?.edges?.length ?? 0,
selectedQuestion: result.selectedQuestion?.question ?? null,
@@ -162,6 +312,56 @@ async function runStartCaseExperiment(scenarioInput, experimentInstruction, opti
error: result.success ? null : (result.error ?? "unknown"),
statusCode: result.statusCode ?? (result.success ? 200 : 500),
};
// ── Condition D/E: freeform deconstruct / investigability-first output — capture without production JSON schema ──
if (condition === "D" || condition === "E") {
const rawText = result.freeformText ?? result.reconstruction ?? null;
output.conditionType = condition === "D" ? "deconstruct" : "investigability-first";
output.freeformText = rawText;
delete output.reconstruction;
}
if (reconstructionOnly) {
if (scenarioInput?.experimentId !== undefined) {
output.experimentId = scenarioInput.experimentId;
}
if (scenarioInput?.scenarioId !== undefined) {
output.scenarioId = scenarioInput.scenarioId;
}
const configuredProvider =
process.env.START_CASE_EXPERIMENT_PROVIDER === "openai" ? "openai" : "ollama";
const configuredModel =
result.diagnostics?.modelName ??
reconstructionModelName ??
(configuredProvider === "ollama" ? process.env.OLLAMA_MODEL : undefined);
const effectivePromptVersion = result.diagnostics?.promptVersion;
output.execution = {
condition: condition ?? "A",
provider: configuredProvider,
...(configuredModel !== undefined ? { model: configuredModel } : {}),
...(effectivePromptVersion !== undefined
? { promptVersion: effectivePromptVersion }
: {}),
};
if (Array.isArray(result.situationGraph?.nodes)) {
output.graphUnknownNodes = result.situationGraph.nodes
.filter((node) => node.kind === "unknown")
.map((node) => ({
id: node.id,
label: node.label,
description: node.description,
...(node.openQuestion !== undefined
? { openQuestion: node.openQuestion }
: {}),
...(node.status !== undefined ? { status: node.status } : {}),
}));
}
}
return output;
}
if (require.main === module) {
@@ -202,10 +402,59 @@ if (require.main === module) {
experimentInstruction = process.argv[experimentIdx + 1];
}
// ── Condition selector: A (production) | B (question-free baseline) | C (question-free + domain-preservation) | D (deconstruct-first freeform) ──
const conditionArgIdx = process.argv.indexOf("--condition");
let condition = null;
if (conditionArgIdx !== -1 && conditionArgIdx + 1 < process.argv.length) {
condition = process.argv[conditionArgIdx + 1];
}
// ── Validate condition value ──
const validConditions = ["A", "B", "C", "D", "E"];
if (condition !== null && !validConditions.includes(condition)) {
throw new Error(`Invalid condition: ${condition}. Must be one of: A, B, C, D, E`);
}
// ── Treatment selector (D and E only) ──
const treatmentArgIdx = process.argv.indexOf("--treatment");
let treatment = null;
if (treatmentArgIdx !== -1 && treatmentArgIdx + 1 < process.argv.length) {
treatment = process.argv[treatmentArgIdx + 1];
}
// ── Validate treatment value ──
const validDTreatments = ["no21", "no23"];
const validETreatments = ["e-investigability-first"];
if (treatment !== null && condition === "D" && !validDTreatments.includes(treatment)) {
throw new Error(`Invalid treatment: ${treatment}. Must be one of: no21, no23`);
}
if (treatment !== null && condition === "E" && !validETreatments.includes(treatment)) {
throw new Error(`Invalid treatment: ${treatment}. Must be one of: e-investigability-first`);
}
if (treatment !== null && !["D", "E"].includes(condition)) {
throw new Error(`Treatment "${treatment}" is only valid with --condition D or E, not --condition ${condition}`);
}
// ── Prompt-file selector: external reconstruction prompt for condition A ──
const promptFileArgIdx = process.argv.indexOf("--prompt-file");
let promptFile = null;
if (promptFileArgIdx !== -1 && promptFileArgIdx + 1 < process.argv.length) {
promptFile = process.argv[promptFileArgIdx + 1];
}
if (promptFile !== null && condition !== "A") {
throw new Error(
`--prompt-file is only compatible with --condition A. ` +
`Condition ${condition ?? "(none, defaults to A)"} selected.`
);
}
const scenarioInput = readScenarioInput(process.argv);
const reconstructionOnly = process.argv.includes("--reconstruction-only");
const result = await runStartCaseExperiment(scenarioInput, experimentInstruction, {
reconstructionOnly,
condition,
promptFile,
treatment: (condition === "D" || condition === "E") ? treatment : undefined,
});
console.log(JSON.stringify(result, null, 2));
+205
View File
@@ -0,0 +1,205 @@
import { describe, expect, it, vi } from "vitest";
// ── Mock domain seam and provider at module level ────
const mockAnalyseScenario = vi.fn();
vi.mock("@/lib/analysis", () => ({
analyseScenario: (...args) => mockAnalyseScenario(...args),
PROMPT_VERSIONS: ["v1"],
DEFAULT_PROMPT_VERSION: "v1",
}));
vi.mock("@/lib/llm/provider.js", () => ({
getProvider: () => ({}),
getProviderModelName: () => "gpt-5.6-terra",
}));
vi.mock("@/lib/supabase/api-auth.js", () => ({
withAuthenticatedApi: (handler) => handler,
}));
// ── Helpers ─────────────────────────────────────────
function makeValidScenario() {
return "The supplier changed delivery schedules without notice, causing our production line to halt.";
}
function makeRequest(body) {
return new Request("http://localhost/api/analyse", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify(body),
});
}
// ── Route tests — valid POST ────────────────────────
describe("POST /api/analyse — success contract", () => {
it("returns structured analysis on success", async () => {
mockAnalyseScenario.mockResolvedValue({
success: true,
inputClassification: "manufacturing",
reconstruction: { summary: "Validated reconstruction summary" },
evidence: [],
nextQuestion: null,
modelName: "gpt-5.6-terra",
responseDurationMs: 1200,
validationStatus: "passed",
promptVersion: "v1",
});
const { POST } = await import("@/app/api/analyse/route.js");
const res = await POST(makeRequest({ scenario: makeValidScenario() }));
expect(res.status).toBe(200);
const data = await res.json();
expect(typeof data.inputClassification).toBe("string");
expect(data.reconstruction.summary).toBe("Validated reconstruction summary");
});
});
// ── Route tests — request validation ────────────────
describe("POST /api/analyse — request validation", () => {
it("missing scenario → 400", async () => {
const { POST } = await import("@/app/api/analyse/route.js");
const res = await POST(makeRequest({}));
expect(res.status).toBe(400);
const data = await res.json();
expect(data).toEqual({ error: "Request must include a 'scenario' string field" });
});
it("null scenario → 400", async () => {
const { POST } = await import("@/app/api/analyse/route.js");
const res = await POST(makeRequest({ scenario: null }));
expect(res.status).toBe(400);
});
it("non-string scenario → 400", async () => {
const { POST } = await import("@/app/api/analyse/route.js");
const res = await POST(makeRequest({ scenario: 123 }));
expect(res.status).toBe(400);
});
});
// ── Route tests — error handling ────────────────────
describe("POST /api/analyse — error boundary", () => {
it("domain seam throws PROVIDER_UNAVAILABLE → sanitized 503 with generic message", async () => {
const err = Object.assign(
new Error("Ollama /api/generate request timed out after 5 minutes"),
{ code: "PROVIDER_UNAVAILABLE", providerApiPath: "/api/generate" },
);
mockAnalyseScenario.mockImplementationOnce(async () => { throw err; });
const { POST } = await import("@/app/api/analyse/route.js");
const res = await POST(makeRequest({ scenario: makeValidScenario() }));
expect(res.status).toBe(503);
const data = await res.json();
expect(data.error).toBe("Reasoning service is temporarily unavailable.");
});
it("domain seam throws with diagnostics → sanitized response, preserved status", async () => {
const rawResponse = `{"reconstruction":{"observedStates":[{"id":"obs-1"${"x".repeat(2500)}}]}}`;
const err = Object.assign(
new Error("Provider unavailable"),
{
statusCode: 502,
providerApiPath: "/v1/responses",
providerExecution: { chatRequestAttempted: true },
rawResponse,
},
);
mockAnalyseScenario.mockImplementationOnce(async () => { throw err; });
const { POST } = await import("@/app/api/analyse/route.js");
const res = await POST(makeRequest({ scenario: makeValidScenario() }));
expect(res.status).toBe(502);
const data = await res.json();
expect(data.error).toBe("Reasoning request could not be completed.");
expect(JSON.stringify(data)).not.toMatch(/provider unavailable|generate|llama3|rawResponse/i);
});
it("domain seam throws without statusCode → 500", async () => {
mockAnalyseScenario.mockImplementationOnce(async () => { throw new Error("unknown error"); });
const { POST } = await import("@/app/api/analyse/route.js");
const res = await POST(makeRequest({ scenario: makeValidScenario() }));
expect(res.status).toBe(500);
const data = await res.json();
expect(data.error).toBe("Reasoning request could not be completed.");
});
it("domain seam throws validation failure → preserved status", async () => {
const err = Object.assign(new Error("Invalid input"), { statusCode: 400 });
mockAnalyseScenario.mockImplementationOnce(async () => { throw err; });
const { POST } = await import("@/app/api/analyse/route.js");
const res = await POST(makeRequest({ scenario: makeValidScenario() }));
expect(res.status).toBe(400);
const data = await res.json();
expect(data.error).toBe("Reasoning request could not be completed.");
});
it("domain seam returns failure result with diagnostics → sanitized response", async () => {
mockAnalyseScenario.mockResolvedValue({
success: false,
error: "Provider unavailable",
diagnostics: { modelName: "llama3" },
statusCode: 502,
analysisErrors: ["reconstruction: Required"],
validationIssues: [{ path: ["reconstruction"], code: "invalid_type", message: "Required" }],
providerApiPath: "/api/generate",
providerExecution: { generateRequestAttempted: true },
rawResponse: '{"observedStates":[{"id":"x"}]}',
});
const { POST } = await import("@/app/api/analyse/route.js");
const res = await POST(makeRequest({ scenario: makeValidScenario() }));
expect(res.status).toBe(502);
const data = await res.json();
expect(data).toEqual({ error: "Reasoning request could not be completed." });
expect(JSON.stringify(data)).not.toMatch(/provider unavailable|generate|llama3|rawResponse/i);
});
it("domain seam throws without exposing raw provider/internal diagnostics", async () => {
const err = Object.assign(
new Error("Internal connection reset by peer — host=10.0.0.5:8080 key=sk-abc"),
{ statusCode: 502, providerApiPath: "/internal/chat", providerExecution: { chatRequestAttempted: true } },
);
mockAnalyseScenario.mockImplementationOnce(async () => { throw err; });
const { POST } = await import("@/app/api/analyse/route.js");
const res = await POST(makeRequest({ scenario: makeValidScenario() }));
expect(res.status).toBe(502);
const data = await res.json();
expect(JSON.stringify(data)).not.toMatch(/10\.0\.0\.5|8080|sk-abc|providerApiPath|providerExecution|Internal connection reset/i);
});
it("domain seam returns failed analysis object → not spread into response", async () => {
mockAnalyseScenario.mockResolvedValue({
success: false,
error: "Analysis failed",
statusCode: 500,
failedAnalysisObject: { raw: true, internal: "diagnostics" },
});
const { POST } = await import("@/app/api/analyse/route.js");
const res = await POST(makeRequest({ scenario: makeValidScenario() }));
expect(res.status).toBe(500);
const data = await res.json();
expect(data).toEqual({ error: "Reasoning request could not be completed." });
expect(JSON.stringify(data)).not.toMatch(/failedAnalysisObject|internal|diagnostics/i);
});
});
+37 -1
View File
@@ -1,4 +1,4 @@
import { beforeEach, describe, expect, it, vi } from "vitest";
import { describe, expect, it, vi } from "vitest";
const mockSynthesize = vi.fn();
@@ -11,6 +11,10 @@ vi.mock("@/lib/llm/provider.js", () => ({
getProviderModelName: () => "gpt-5.6-terra",
}));
vi.mock("@/lib/supabase/api-auth.js", () => ({
withAuthenticatedApi: (handler) => handler,
}));
describe("POST /api/cases/overview provider routing", () => {
beforeEach(() => mockSynthesize.mockClear());
@@ -26,4 +30,36 @@ describe("POST /api/cases/overview provider routing", () => {
expect(response.status).toBe(200);
expect(mockSynthesize.mock.calls[0][1]).toMatchObject({ modelName: "gpt-5.6-terra" });
});
it("sanitizes provider failure details", async () => {
const error = Object.assign(
new Error("Ollama /api/generate returned 500 from private host"),
{ statusCode: 502 },
);
mockSynthesize.mockImplementationOnce(async () => {
throw error;
});
const { POST } = await import("@/app/api/cases/overview/route.js");
const response = await POST(new Request("http://localhost/api/cases/overview", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ situationGraph: { nodes: [], edges: [] } }),
}));
expect(response.status).toBe(502);
const rawData = await response.text();
const data = JSON.parse(rawData);
expect(data.success).toBe(false);
expect(data.stage).toBe("provider");
expect(data).toEqual({
success: false,
stage: "provider",
error: "Reasoning request could not be completed.",
});
// Prove raw provider diagnostics are NOT exposed at the route boundary
expect(rawData).not.toContain(error.message);
});
});
+3 -21
View File
@@ -169,7 +169,7 @@ describe("app/api/cases/start route", () => {
expect(JSON.stringify(body)).not.toMatch(/ollama|generate|timed out/i);
});
it("returns provider/internal failures as 5xx without stack traces", async () => {
it("sanitizes provider/internal failures at the browser boundary", async () => {
const rawResponse = `{"reconstruction":{"observedStates":[{"id":"obs-1"${"x".repeat(2500)}}]}}`;
mockStartCase.mockResolvedValue({
success: false,
@@ -207,26 +207,8 @@ describe("app/api/cases/start route", () => {
expect(response.status).toBe(502);
const body = await response.json();
expect(body).toHaveProperty("rawResponse");
expect(body.rawResponse).toBe(rawResponse);
expect(body.rawResponse.length).toBeGreaterThan(2000);
expect(body.analysisErrors).toEqual(["reconstruction: Required"]);
expect(body.validationIssues).toEqual([
expect.objectContaining({
path: ["reconstruction", "observedStates", 2, "description"],
code: "invalid_type",
message: "Required",
expected: "string",
received: "undefined",
}),
]);
expect(body.providerApiPath).toBe("/api/generate");
expect(body.providerExecution).toEqual({
chatCapabilityDetected: false,
chatRequestAttempted: false,
chatRequestSucceeded: false,
generateRequestAttempted: true,
});
expect(body).toEqual({ success: false, error: "Reasoning request could not be completed." });
expect(JSON.stringify(body)).not.toMatch(/provider unavailable|generate|llama3|rawResponse/i);
expect(errorSpy).toHaveBeenCalledTimes(1);
expect(errorSpy).toHaveBeenCalledWith(
"[api/cases/start] error response",
@@ -13,6 +13,10 @@ vi.mock("@/lib/llm/provider.js", () => ({
getProviderModelName: () => "gpt-5.6-terra",
}));
vi.mock("@/lib/supabase/api-auth.js", () => ({
withAuthenticatedApi: (handler) => handler,
}));
// ── Helpers ─────────────────────────────────────────────────
function makeValidGraph() {
@@ -110,10 +114,13 @@ describe("POST /api/cases/synthesis — error cases", () => {
});
it("domain seam throws with statusCode → mapped status", async () => {
mockSynthesize.mockRejectedValue(new Error("Provider failed"));
// Add statusCode property to the error object after creation
const err = Object.assign(new Error("Provider failed"), { statusCode: 502 });
mockSynthesize.mockRejectedValue(err);
const err = Object.assign(
new Error("Ollama /api/generate returned 500 from private host"),
{ statusCode: 502 },
);
mockSynthesize.mockImplementationOnce(async () => {
throw err;
});
const { POST } = await import("@/app/api/cases/synthesis/route.js");
const res = await POST(makeRequest({ situationGraph: makeValidGraph() }));
@@ -122,10 +129,17 @@ describe("POST /api/cases/synthesis — error cases", () => {
const data = await res.json();
expect(data.success).toBe(false);
expect(data.stage).toBe("provider");
expect(data).toEqual({
success: false,
stage: "provider",
error: "Reasoning request could not be completed.",
});
});
it("domain seam throws without statusCode → 500", async () => {
mockSynthesize.mockRejectedValue(new Error("unknown error"));
mockSynthesize.mockImplementationOnce(async () => {
throw new Error("unknown error");
});
const { POST } = await import("@/app/api/cases/synthesis/route.js");
const res = await POST(makeRequest({ situationGraph: makeValidGraph() }));
@@ -138,7 +152,9 @@ describe("POST /api/cases/synthesis — error cases", () => {
it("domain seam throws 400 → mapped to 400", async () => {
const err = Object.assign(new Error("Invalid input"), { statusCode: 400 });
mockSynthesize.mockRejectedValue(err);
mockSynthesize.mockImplementationOnce(async () => {
throw err;
});
const { POST } = await import("@/app/api/cases/synthesis/route.js");
const res = await POST(makeRequest({ situationGraph: makeValidGraph() }));
@@ -0,0 +1,186 @@
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { makeGraph, makeNode } from "@/lib/graph/schema.js";
const generateReconstruction = vi.fn();
vi.mock("@/lib/llm/provider", () => ({
getProvider: () => ({ generateReconstruction }),
getProviderModelName: () => "test-model",
}));
vi.mock("@/lib/supabase/api-auth.js", () => ({
withAuthenticatedApi: (handler) => handler,
}));
function makeRecruitmentGraph() {
const target = makeNode({
id: "recruitment-target",
label: "Staffing certainty gap",
description:
"Whether additional skilled staff needed for expansion can be recruited",
kind: "unknown",
status: "unknown",
confidence: "medium",
});
return makeGraph({
centralStatement:
"We need to compare our current baseline against the comparable state before staffing is secured for expansion.",
currentSummary: "Test summary",
nodes: [target],
edges: [],
activeUnknownNodeId: target.id,
resolvedNodeIds: [],
});
}
function makeHistoricalGraph() {
const target = makeNode({
id: "historical-target",
label: "Baseline conversion rate",
description:
"Need the previous baseline because the change cannot be assessed without it.",
kind: "unknown",
status: "unknown",
confidence: "medium",
});
return makeGraph({
centralStatement: "Assess the change against the comparable prior baseline.",
currentSummary: "Test summary",
nodes: [target],
edges: [],
activeUnknownNodeId: target.id,
resolvedNodeIds: [],
});
}
function makeRequest(graph, targetNodeId = graph.nodes[0].id) {
return new Request("http://localhost/api/focused-investigation/formulate", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ situationGraph: graph, targetNodeId }),
});
}
describe("POST /api/focused-investigation/formulate semantic compatibility", () => {
beforeEach(() => {
generateReconstruction.mockReset();
});
afterEach(() => {
vi.useRealTimers();
});
it("suppresses an assessor-incompatible strategy without changing the selected target or graph", async () => {
const graph = makeRecruitmentGraph();
const snapshot = JSON.stringify(graph);
generateReconstruction.mockResolvedValue({
response: {
strategyKey: "baseline_reconstruction",
relevance: "incompatible",
},
});
const { POST } = await import("@/app/api/focused-investigation/formulate/route.js");
const response = await POST(makeRequest(graph));
const json = await response.json();
expect(response.status).toBe(200);
expect(json.targetNodeId).toBe("recruitment-target");
expect(json.strategy).toBeNull();
expect(json.question).toBe(
"What evidence would clarify whether additional skilled staff needed for expansion can be recruited?",
);
expect(JSON.stringify(graph)).toBe(snapshot);
expect(generateReconstruction).toHaveBeenCalledTimes(1);
});
it("does not let a compatible semantic verdict authorize a grammatically incompatible baseline template", async () => {
const graph = makeRecruitmentGraph();
generateReconstruction.mockResolvedValue({
response: {
strategyKey: "baseline_reconstruction",
relevance: "compatible",
},
});
const { POST } = await import("@/app/api/focused-investigation/formulate/route.js");
const response = await POST(makeRequest(graph));
const json = await response.json();
expect(json.strategy).toBeNull();
expect(json.question.toLowerCase()).not.toContain("before whether");
expect(json.question).toContain("What evidence would clarify whether");
});
it("preserves a compatible historical strategy when its template is suitable", async () => {
const graph = makeHistoricalGraph();
generateReconstruction.mockResolvedValue({
response: {
strategyKey: "baseline_reconstruction",
relevance: "compatible",
},
});
const { POST } = await import("@/app/api/focused-investigation/formulate/route.js");
const response = await POST(makeRequest(graph));
const json = await response.json();
expect(json.strategy).toBe("baseline_reconstruction");
expect(json.question).toContain("What was the comparable state before");
});
it("treats invalid or mismatched assessment output as non-authorizing", async () => {
const graph = makeRecruitmentGraph();
generateReconstruction.mockResolvedValue({
response: { strategyKey: "evidence_gathering", relevance: "compatible" },
});
const { POST } = await import("@/app/api/focused-investigation/formulate/route.js");
const response = await POST(makeRequest(graph));
const json = await response.json();
expect(json.strategy).toBeNull();
expect(json.question).toContain("What evidence would clarify whether");
});
it("treats provider unavailability as non-authorizing fallback", async () => {
const graph = makeRecruitmentGraph();
generateReconstruction.mockRejectedValue(
Object.assign(new Error("provider unavailable"), { code: "PROVIDER_UNAVAILABLE" }),
);
const { POST } = await import("@/app/api/focused-investigation/formulate/route.js");
const response = await POST(makeRequest(graph));
const json = await response.json();
expect(response.status).toBe(200);
expect(json.strategy).toBeNull();
expect(json.question).toContain("What evidence would clarify whether");
});
it("aborts the one assessment invocation at the route deadline and falls back", async () => {
vi.useFakeTimers();
const graph = makeRecruitmentGraph();
let observedSignal;
generateReconstruction.mockImplementation((_, __, ___, { signal }) => {
observedSignal = signal;
return new Promise((_, reject) => {
signal.addEventListener("abort", () => {
const error = new Error("assessment aborted");
error.name = "AbortError";
reject(error);
}, { once: true });
});
});
const { POST } = await import("@/app/api/focused-investigation/formulate/route.js");
const responsePromise = POST(makeRequest(graph));
await vi.advanceTimersByTimeAsync(10000);
const response = await responsePromise;
const json = await response.json();
expect(observedSignal.aborted).toBe(true);
expect(json.strategy).toBeNull();
expect(json.question).toContain("What evidence would clarify whether");
});
});
@@ -0,0 +1,189 @@
/**
* Progressed investigation fixture apparatus verification.
*
* Proves that the canonical progressed-investigation fixture:
* 1. loads successfully as valid JSON
* 2. contains >= 1 admitted known/supported evidence item
* 3. contains >= 1 unresolved Open Question (kind=unknown, status=unknown)
* 4. the unresolved Open Question is not admitted evidence
* 5. at least one eligible Finding exists (null disposition)
* 6. unresolved Open Question is not represented as an eligible Finding
* 7. resolvedNodeIds does NOT include any unknown node id
*/
import { describe, it, expect } from "vitest";
import fixture from "../fixtures/progressed-investigation-manufacturing.json";
describe("Progressed investigation fixture apparatus", () => {
/* ── 1. Fixture loads successfully ─────────────────── */
it("loads as valid JSON with required top-level keys", () => {
expect(fixture).toHaveProperty("success", true);
expect(fixture).toHaveProperty("stage", "update_applied");
expect(fixture).toHaveProperty("situationGraph");
expect(fixture.situationGraph).toHaveProperty("nodes");
expect(fixture.situationGraph).toHaveProperty("edges");
expect(fixture.situationGraph).toHaveProperty("activeUnknownNodeId");
expect(fixture.situationGraph).toHaveProperty("resolvedNodeIds");
expect(fixture.situationGraph).toHaveProperty("centralStatement");
});
/* ── 2. Contains admitted evidence (>= 1 supported/known node) ─── */
it("contains at least one admitted known or supported evidence node", () => {
const admitted = fixture.situationGraph.nodes.filter(
(n) => n.status === "supported" || n.status === "known"
);
expect(admitted.length).toBeGreaterThan(0);
});
it("admitted nodes are not unknown kind", () => {
const admitted = fixture.situationGraph.nodes.filter(
(n) => ["supported", "known"].includes(n.status)
);
for (const node of admitted) {
expect(node.kind).not.toBe("unknown");
}
});
/* ── 3. Contains unresolved Open Question (>= 1) ─────── */
it("contains at least one unresolved Open Question node", () => {
const unresolved = fixture.situationGraph.nodes.filter(
(n) => n.kind === "unknown" && n.status === "unknown"
);
expect(unresolved.length).toBeGreaterThan(0);
});
/* ── 4. Unresolved Open Question is not admitted evidence ─── */
it("no node is simultaneously unresolved unknown AND admitted evidence", () => {
const allNodes = fixture.situationGraph.nodes;
for (const node of allNodes) {
if (node.kind === "unknown" && node.status === "unknown") {
// An unresolved unknown must not have admitted status
expect(["supported", "known"]).not.toContain(node.status);
}
}
});
it("unresolved and admitted node ID sets are disjoint", () => {
const unresolvedIds = new Set(
fixture.situationGraph.nodes
.filter((n) => n.kind === "unknown" && n.status === "unknown")
.map((n) => n.id)
);
const admittedIds = new Set(
fixture.situationGraph.nodes
.filter((n) => ["supported", "known"].includes(n.status))
.map((n) => n.id)
);
for (const id of unresolvedIds) {
expect(admittedIds.has(id)).toBe(false);
}
});
it("unresolved unknown nodes are distinct from admitted evidence", () => {
const unresolvedIds = new Set(
fixture.situationGraph.nodes
.filter((n) => n.kind === "unknown" && n.status === "unknown")
.map((n) => n.id)
);
const admittedIds = new Set(
fixture.situationGraph.nodes
.filter((n) => ["supported", "known"].includes(n.status))
.map((n) => n.id)
);
for (const id of unresolvedIds) {
expect(admittedIds.has(id)).toBe(false);
}
});
/* ── 5. Unresolved question excluded from resolvedNodeIds ─── */
it("resolvedNodeIds does not contain any unresolved unknown node id", () => {
const unresolvedIds = fixture.situationGraph.nodes
.filter((n) => n.kind === "unknown" && n.status === "unknown")
.map((n) => n.id);
for (const nodeId of unresolvedIds) {
expect(fixture.situationGraph.resolvedNodeIds).not.toContain(nodeId);
}
});
/* ── 6. Eligible Finding exists ─────────────────────── */
it("findings array exists and has eligible items", () => {
expect(fixture.findings).toBeDefined();
expect(Array.isArray(fixture.findings)).toBe(true);
const eligible = fixture.findings.filter(
(f) => f.status === "provisional" && f.userDisposition === null
);
expect(eligible.length).toBeGreaterThan(0);
});
/* ── 7. Unresolved Open Question is not represented as a Finding ─── */
it("unresolved unknown nodes are not mapped to findings propositions", () => {
const unresolvedNodeIds = new Set(
fixture.situationGraph.nodes
.filter((n) => n.kind === "unknown" && n.status === "unknown")
.map((n) => n.id)
);
// originTargetNodeId links findings to unknown nodes
// A finding whose originatingTargetNodeId is an unresolved unknown is fine
// but the proposition should not BE the same as the node label
const eligibleFindings = fixture.findings.filter(
(f) => f.status === "provisional" && f.userDisposition === null
);
const unresolvedNodes = fixture.situationGraph.nodes.filter(
(n) => n.kind === "unknown" && n.status === "unknown"
);
for (const finding of eligibleFindings) {
// The proposition is an admitted evidence item, not the unknown itself
expect(finding.proposition).not.toContain("Whether");
// Verify the originating node is NOT resolved — it remains open
const targetNode = unresolvedNodes.find(
(n) => n.id === finding.originatingTargetNodeId
);
if (targetNode) {
expect(targetNode.status).toBe("unknown");
expect(targetNode.kind).toBe("unknown");
}
}
});
/* ── 8. Canonical shape integrity ──────────────────── */
it("activeUnknownNodeId references a valid unresolved unknown node", () => {
const unresolvedIds = fixture.situationGraph.nodes
.filter((n) => n.kind === "unknown" && n.status === "unknown")
.map((n) => n.id);
expect(unresolvedIds).toContain(fixture.situationGraph.activeUnknownNodeId);
});
it("all nodes pass the canonical situationNode schema invariants", () => {
for (const node of fixture.situationGraph.nodes) {
expect(node).toHaveProperty("id");
expect(node).toHaveProperty("label");
expect(node).toHaveProperty("description");
expect(["observation","reported_claim","metric","state","transition","relationship","assumption","unknown","conclusion","option"]).toContain(node.kind);
expect(["known","unknown","provisional","supported","weakened","contradicted","resolved"]).toContain(node.status);
expect(["low","medium","high"]).toContain(node.confidence);
expect(Array.isArray(node.evidenceIds)).toBe(true);
expect(Array.isArray(node.dependsOn)).toBe(true);
expect(Array.isArray(node.affects)).toBe(true);
}
});
it("all edges reference valid nodes", () => {
const nodeIds = new Set(fixture.situationGraph.nodes.map((n) => n.id));
for (const edge of fixture.situationGraph.edges) {
expect(edge).toHaveProperty("id");
expect(nodeIds.has(edge.fromNodeId)).toBe(true);
expect(nodeIds.has(edge.toNodeId)).toBe(true);
expect(edge).toHaveProperty("relationship");
expect(edge).toHaveProperty("confidence");
}
});
});
@@ -0,0 +1,295 @@
/**
* Deterministic control of report epistemic comparison inputs.
*
* Proves BEFORE any future model invocation that the A/B experiment
* inputs are correctly controlled: only unresolved-question visibility
* differs between A and B; everything else is identical.
*
* No live model calls. No mocking. Pure input-contract verification.
*/
import { describe, it, expect } from "vitest";
import fixture from "../fixtures/progressed-investigation-manufacturing.json";
import productLaunchFixture from "../fixtures/live-product-launch-start-response.json";
const FIXED_PLAUSIBLE =
"The rise in absolute complaint counts is likely an artifact of both the new CRM tagging system broadening what gets recorded as a complaint, combined with a proportional increase from higher production volume rather than a genuine deterioration in product quality.";
function buildABInputs(fix, includeUQ) {
const sg = fix.situationGraph;
const se = sg.nodes.filter(n => n.status === "known" || n.status === "supported").map(n => ({id: n.id, kind: n.kind, status: n.status, label: n.label}));
const fi = (fix.findings ?? []).filter(f2 => f2.status === "provisional" && f2.userDisposition === null).map(f3 => ({id: f3.id, proposition: f3.proposition}));
const pi = [FIXED_PLAUSIBLE];
const uq = sg.nodes.filter(n => n.kind === "unknown" && n.status === "unknown").map(n => ({id: n.id, label: n.label, description: n.description}));
const ss = {centralStatement: sg.centralStatement, activeUnknownNodeId: sg.activeUnknownNodeId, resolvedNodeIds: [...sg.resolvedNodeIds], nodeCount: sg.nodes.length, edgeCount: sg.edges.length};
const pc = {provider: "ollama", model: process.env.OLLAMA_MODEL || "configured-model"};
return {supportedEvidence: se, findings: fi, plausibleInterpretations: pi, unresolvedQuestions: includeUQ ? [...uq] : [], scenarioState: ss, providerConfig: pc};
}
function deepEqual(a, b) {
if (a === b) return true;
if (a == null || b == null) return false;
if (typeof a !== typeof b) return false;
if (Array.isArray(a)) { if (!Array.isArray(b) || a.length !== b.length) return false; for (let i = 0; i < a.length; i++) if (!deepEqual(a[i], b[i])) return false; return true; }
if (typeof a === "object") { const kA = Object.keys(a).sort(), kB = Object.keys(b).sort(); if (kA.length !== kB.length) return false; for (let i = 0; i < kA.length; i++) if (kA[i] !== kB[i] || !deepEqual(a[kA[i]], b[kB[i]])) return false; return true; }
return false;
}
describe("Controlled A/B report epistemic comparison inputs", () => {
it("builds canonical input objects for both A and B", () => {
const a = buildABInputs(fixture, false);
const b = buildABInputs(fixture, true);
["supportedEvidence","findings","plausibleInterpretations","unresolvedQuestions","scenarioState","providerConfig"].forEach(k => { expect(a).toHaveProperty(k); expect(b).toHaveProperty(k); });
});
it("A and B have identical admitted evidence", () => {
const a = buildABInputs(fixture, false); const b = buildABInputs(fixture, true);
expect(deepEqual(a.supportedEvidence, b.supportedEvidence)).toBe(true);
});
it("A and B have identical eligible Findings", () => {
const a = buildABInputs(fixture, false); const b = buildABInputs(fixture, true);
expect(deepEqual(a.findings, b.findings)).toBe(true);
});
it("A and B have identical fixed plausibleInterpretations", () => {
const a = buildABInputs(fixture, false); const b = buildABInputs(fixture, true);
expect(deepEqual(a.plausibleInterpretations, b.plausibleInterpretations)).toBe(true);
});
it("fixed plausible interpretation matches canonical value exactly", () => {
const a = buildABInputs(fixture, false);
expect(a.plausibleInterpretations).toHaveLength(1);
expect(a.plausibleInterpretations[0]).toBe(FIXED_PLAUSIBLE);
});
it("A and B reference the same progressed investigation state", () => {
const a = buildABInputs(fixture, false); const b = buildABInputs(fixture, true);
expect(deepEqual(a.scenarioState, b.scenarioState)).toBe(true);
});
it("provider/model configuration is identical for A and B", () => {
const a = buildABInputs(fixture, false); const b = buildABInputs(fixture, true);
expect(a.providerConfig.provider).toBe(b.providerConfig.provider);
expect(a.providerConfig.model).toBe(b.providerConfig.model);
});
it("A does not expose unresolved Open Questions (hidden)", () => {
const a = buildABInputs(fixture, false);
expect(a.unresolvedQuestions).toHaveLength(0);
});
it("B exposes exactly the canonical unresolved Open Questions", () => {
const b = buildABInputs(fixture, true);
const fu = fixture.situationGraph.nodes.filter(n => n.kind === "unknown" && n.status === "unknown");
expect(b.unresolvedQuestions).toHaveLength(fu.length);
for (const node of fu) { const found = b.unresolvedQuestions.find(uq => uq.id === node.id); expect(found).toBeDefined(); expect(found.label).toBe(node.label); expect(found.description).toBe(node.description); }
});
it("only unresolved visibility differs between A and B", () => {
const a = buildABInputs(fixture, false); const b = buildABInputs(fixture, true);
expect(deepEqual(a.supportedEvidence, b.supportedEvidence)).toBe(true);
expect(deepEqual(a.findings, b.findings)).toBe(true);
expect(deepEqual(a.plausibleInterpretations, b.plausibleInterpretations)).toBe(true);
expect(deepEqual(a.scenarioState, b.scenarioState)).toBe(true);
expect(deepEqual(a.providerConfig, b.providerConfig)).toBe(true);
expect(a.unresolvedQuestions).not.toEqual(b.unresolvedQuestions);
});
it("fixed plausible interpretation is not admitted evidence", () => {
const a = buildABInputs(fixture, false);
for (const e of a.supportedEvidence) { expect(e.label).not.toContain(FIXED_PLAUSIBLE); expect(["known","supported"]).toContain(e.status); }
});
it("unresolved questions are not admitted evidence", () => {
const b = buildABInputs(fixture, true);
for (const uq of b.unresolvedQuestions) { const fn = fixture.situationGraph.nodes.find(n => n.id === uq.id); expect(fn.status).toBe("unknown"); expect(fn.kind).toBe("unknown"); }
});
it("unresolved questions are not plausibleInterpretations", () => {
const b = buildABInputs(fixture, true);
for (const uq of b.unresolvedQuestions) { expect(b.plausibleInterpretations).not.toContain(uq.label); expect(b.plausibleInterpretations[0]).toBe(FIXED_PLAUSIBLE); }
});
it("no fixture node conflates unresolved and evidence categories", () => {
for (const node of fixture.situationGraph.nodes) {
if (node.kind === "unknown" && node.status === "unknown") expect(["known","supported"]).not.toContain(node.status);
else if (["known","supported"].includes(node.status)) expect(node.kind).not.toBe("unknown");
}
});
it("inputs are constructed purely from fixture without provider or model dependency", () => {
const a = buildABInputs(fixture, false); const b = buildABInputs(fixture, true);
expect(typeof a.supportedEvidence).toBe("object");
expect(typeof a.findings).toBe("object");
expect(typeof a.plausibleInterpretations).toBe("object");
expect(typeof a.unresolvedQuestions).toBe("object");
});
// ── Prompt-builder identity proof (A/B produce identical SECTION A) ────
it("buildOverviewSynthesisPrompt produces identical prompt text when unknown nodes are present vs removed", async () => {
const { buildOverviewSynthesisPrompt } = await import(
"@/lib/graph/investigation-overview-synthesis.js"
);
const aInputs = buildABInputs(fixture, false);
const baseSg = fixture.situationGraph;
const knownSupportedNodes = baseSg.nodes.filter(
(n) => ["known", "supported"].includes(n.status)
);
// Version A: filtered graph (unknowns removed)
const promptA = buildOverviewSynthesisPrompt(
{ ...baseSg, nodes: knownSupportedNodes },
aInputs.supportedEvidence,
aInputs.findings,
aInputs.plausibleInterpretations
);
// Version B: full graph (unknowns included) — should be identical because
// buildOverviewSynthesisPrompt already filters to known+supported only
const promptB = buildOverviewSynthesisPrompt(
baseSg,
aInputs.supportedEvidence,
aInputs.findings,
aInputs.plausibleInterpretations
);
expect(promptA).toBe(promptB);
});
it("fixed plausible interpretation carries NON-EVIDENCE epistemic status", () => {
const a = buildABInputs(fixture, false);
expect(a.plausibleInterpretations).toHaveLength(1);
expect(a.plausibleInterpretations[0]).toBe(FIXED_PLAUSIBLE);
const el = a.supportedEvidence.map(e => e.label.toLowerCase());
const il = FIXED_PLAUSIBLE.toLowerCase();
for (const label of el) expect(label).not.toContain(il);
});
it("fixture has the minimum required content for A/B comparison", () => {
const sg = fixture.situationGraph;
expect(sg.nodes.filter(n => ["known","supported"].includes(n.status)).length).toBeGreaterThan(0);
expect(sg.nodes.filter(n => n.kind === "unknown" && n.status === "unknown").length).toBeGreaterThan(0);
expect(fixture.findings.filter(f => f.status === "provisional" && f.userDisposition === null).length).toBeGreaterThan(0);
});
});
// ── Second case: product-launch (zero Findings) ───────────────────────
const PLAUSIBLE_PRODUCT_LAUNCH =
"The large enterprise customer's decision timeline alone does not explain launch timing; competitor activity and the £300k support cost create an independent commercial uncertainty that is unresolved and evidence-free.";
describe("Controlled A/B report epistemic comparison inputs — product-launch zero-Findings case", () => {
const pl = productLaunchFixture;
it("fixture has no findings array (legitimate zero-Findings investigation)", () => {
expect(pl.findings).toBeUndefined();
});
it("zero Findings represented as [] by apparatus", () => {
const a = buildABInputsForProductLaunch(pl, false);
const b = buildABInputsForProductLaunch(pl, true);
expect(a.findings).toStrictEqual([]);
expect(b.findings).toStrictEqual([]);
});
it("supported evidence present from graph", () => {
const a = buildABInputsForProductLaunch(pl, false);
expect(a.supportedEvidence.length).toBeGreaterThan(0);
});
it("unresolved questions present from graph", () => {
const b = buildABInputsForProductLaunch(pl, true);
const fu = pl.situationGraph.nodes.filter(n => n.kind === "unknown" && n.status === "unknown");
expect(b.unresolvedQuestions).toHaveLength(fu.length);
expect(fu.length).toBeGreaterThan(0);
});
it("A and B have identical admitted evidence", () => {
const a = buildABInputsForProductLaunch(pl, false); const b = buildABInputsForProductLaunch(pl, true);
expect(deepEqual(a.supportedEvidence, b.supportedEvidence)).toBe(true);
});
it("A and B have identical (zero) Findings", () => {
const a = buildABInputsForProductLaunch(pl, false); const b = buildABInputsForProductLaunch(pl, true);
expect(deepEqual(a.findings, b.findings)).toBe(true);
expect(a.findings).toStrictEqual([]);
});
it("A and B have identical fixed plausibleInterpretations", () => {
const a = buildABInputsForProductLaunch(pl, false); const b = buildABInputsForProductLaunch(pl, true);
expect(deepEqual(a.plausibleInterpretations, b.plausibleInterpretations)).toBe(true);
});
it("A and B reference the same investigation state", () => {
const a = buildABInputsForProductLaunch(pl, false); const b = buildABInputsForProductLaunch(pl, true);
expect(deepEqual(a.scenarioState, b.scenarioState)).toBe(true);
});
it("provider/model configuration is identical for A and B", () => {
const a = buildABInputsForProductLaunch(pl, false); const b = buildABInputsForProductLaunch(pl, true);
expect(a.providerConfig.provider).toBe(b.providerConfig.provider);
expect(a.providerConfig.model).toBe(b.providerConfig.model);
});
it("A does not expose unresolved Open Questions (hidden)", () => {
const a = buildABInputsForProductLaunch(pl, false);
expect(a.unresolvedQuestions).toHaveLength(0);
});
it("B exposes exactly the canonical unresolved Open Questions", () => {
const b = buildABInputsForProductLaunch(pl, true);
const fu = pl.situationGraph.nodes.filter(n => n.kind === "unknown" && n.status === "unknown");
expect(b.unresolvedQuestions).toHaveLength(fu.length);
});
it("only unresolved visibility differs between A and B", () => {
const a = buildABInputsForProductLaunch(pl, false); const b = buildABInputsForProductLaunch(pl, true);
expect(deepEqual(a.supportedEvidence, b.supportedEvidence)).toBe(true);
expect(deepEqual(a.findings, b.findings)).toBe(true);
expect(deepEqual(a.plausibleInterpretations, b.plausibleInterpretations)).toBe(true);
expect(deepEqual(a.scenarioState, b.scenarioState)).toBe(true);
expect(deepEqual(a.providerConfig, b.providerConfig)).toBe(true);
expect(a.unresolvedQuestions).not.toEqual(b.unresolvedQuestions);
});
it("A and B have identical input objects with all canonical keys", () => {
const a = buildABInputsForProductLaunch(pl, false);
const b = buildABInputsForProductLaunch(pl, true);
["supportedEvidence","findings","plausibleInterpretations","unresolvedQuestions","scenarioState","providerConfig"].forEach(k => { expect(a).toHaveProperty(k); expect(b).toHaveProperty(k); });
});
it("inputs are constructed purely from fixture without provider or model dependency", () => {
const a = buildABInputsForProductLaunch(pl, false); const b = buildABInputsForProductLaunch(pl, true);
expect(typeof a.supportedEvidence).toBe("object");
expect(typeof a.findings).toBe("object");
expect(typeof a.plausibleInterpretations).toBe("object");
expect(typeof a.unresolvedQuestions).toBe("object");
});
it("plausible interpretation is not invented from scenario text", () => {
const a = buildABInputsForProductLaunch(pl, false);
expect(a.plausibleInterpretations).toHaveLength(1);
// The fixed plausible is a deterministic constant, not derived from the fixture
expect(a.plausibleInterpretations[0]).toBe(PLAUSIBLE_PRODUCT_LAUNCH);
});
it("fixture situationGraph has both supported evidence and unknown nodes", () => {
const sg = pl.situationGraph;
expect(sg.nodes.filter(n => ["known","supported"].includes(n.status)).length).toBeGreaterThan(0);
expect(sg.nodes.filter(n => n.kind === "unknown" && n.status === "unknown").length).toBeGreaterThan(0);
});
});
function buildABInputsForProductLaunch(fix, includeUQ) {
const sg = fix.situationGraph;
const se = sg.nodes.filter(n => n.status === "known" || n.status === "supported").map(n => ({id: n.id, kind: n.kind, status: n.status, label: n.label}));
const fi = (fix.findings ?? []).filter(f2 => f2.status === "provisional" && f2.userDisposition === null).map(f3 => ({id: f3.id, proposition: f3.proposition}));
const pi = [PLAUSIBLE_PRODUCT_LAUNCH];
const uq = sg.nodes.filter(n => n.kind === "unknown" && n.status === "unknown").map(n => ({id: n.id, label: n.label, description: n.description}));
const ss = {centralStatement: sg.centralStatement, activeUnknownNodeId: sg.activeUnknownNodeId, resolvedNodeIds: [...sg.resolvedNodeIds], nodeCount: sg.nodes.length, edgeCount: sg.edges.length};
const pc = {provider: "ollama", model: process.env.OLLAMA_MODEL || "configured-model"};
return {supportedEvidence: se, findings: fi, plausibleInterpretations: pi, unresolvedQuestions: includeUQ ? [...uq] : [], scenarioState: ss, providerConfig: pc};
}
+42 -1
View File
@@ -7,11 +7,13 @@ function mkN(id, label, opts) {
var k = (opts && opts.kind) || "unknown";
var s = (opts && opts.status) || (k === "unknown" ? "unknown" : "known");
var c = (opts && opts.confidence) || "low";
return {
var node = {
id, label, description: (opts && opts.description) || label, kind: k, status: s, confidence: c,
confidenceAssessment: { evidenceConfidence: c, completenessStatus: "partial", conclusionConfidence: c },
value: null, unit: null, evidenceIds: [], dependsOn: [], affects: [], childIds: []
};
if (opts && opts.openQuestion) node.openQuestion = opts.openQuestion;
return node;
}
/* ── Filtering tests ─────────────────────────────────────────────── */
@@ -147,6 +149,45 @@ describe("Deduplication", () => {
});
// The unresolved one (if any) or the cross-dedup should prevent duplication
});
it("keeps canonically distinct uncertainties with identical presentation questions", () => {
const sharedQuestion = "What evidence would resolve this uncertainty?";
const vm = buildFacilitatorViewModel({
nodes: [
mkN("u1", "Whether supplier capacity is sufficient for launch", {
description: "Supplier capacity determines viability",
openQuestion: sharedQuestion,
}),
mkN("u2", "Whether customer demand is sufficient for launch", {
description: "Customer demand determines viability",
openQuestion: sharedQuestion,
}),
],
resolvedIds: new Set(),
});
// Two distinct nodes with different descriptions should both appear, despite sharing openQuestion.
expect(vm.investigating.items).toContain("Supplier capacity determines viability");
expect(vm.investigating.items).toContain("Customer demand determines viability");
});
it("continues to merge genuine canonical duplicates despite different presentation questions", () => {
const vm = buildFacilitatorViewModel({
nodes: [
mkN("u1", "Whether supplier capacity is sufficient for launch", {
openQuestion: "Can the supplier support the launch volume?",
}),
mkN("u2", "whether supplier capacity is sufficient for launch!", {
openQuestion: "Is enough supplier capacity available?",
}),
],
resolvedIds: new Set(),
});
expect(vm.investigating.items).toEqual([
"Whether supplier capacity is sufficient for launch",
]);
});
});
/* ── Ranking tests ───────────────────────────────────────────────── */
@@ -0,0 +1,167 @@
{
"manifestVersion": "1.0.0",
"description": "Longitudinal cumulative evidence checkpoints for Current Understanding synthesis comparison experiments.",
"sourceFixture": "tests/fixtures/current-understanding-original-investigation.json",
"contaminationAssessment": "NONE",
"contributions": [
{
"id": "contrib-0001",
"ordering": 1,
"timestamp": "2026-10-08T06:50:06.355Z"
},
{
"id": "contrib-0002",
"ordering": 2,
"timestamp": "2026-10-08T06:55:45.665Z"
},
{
"id": "contrib-0003",
"ordering": 3,
"timestamp": "2026-10-08T07:01:23.359Z"
}
],
"checkpoints": [
{
"id": "checkpoint-0",
"includedContributions": [],
"includedFindings": []
},
{
"id": "checkpoint-1",
"includedContributions": [
"contrib-0001"
],
"includedFindings": [
"finding-3bw5f0",
"finding-i5zdcq"
]
},
{
"id": "checkpoint-2",
"includedContributions": [
"contrib-0001",
"contrib-0002"
],
"includedFindings": [
"finding-3bw5f0",
"finding-i5zdcq",
"finding-hciikc",
"finding-ybfsm3",
"finding-k45vyt",
"finding-vhftyr",
"finding-9bgkmw"
]
},
{
"id": "checkpoint-3",
"includedContributions": [
"contrib-0001",
"contrib-0002",
"contrib-0003"
],
"includedFindings": [
"finding-3bw5f0",
"finding-i5zdcq",
"finding-hciikc",
"finding-ybfsm3",
"finding-k45vyt",
"finding-vhftyr",
"finding-9bgkmw",
"finding-23192q",
"finding-kkh39k"
]
}
],
"findingsById": {
"finding-3bw5f0": {
"id": "finding-3bw5f0",
"status": "provisional",
"createdAt": "2026-10-08T06:50:06.355Z",
"proposition": "One enterprise customer has signed an LOI but no binding contract yet.",
"contributionId": "contrib-0001",
"userDisposition": null,
"sourceObservation": "One enterprise customer has signed an LOI but no binding contract yet.",
"originatingTargetNodeId": "nw0dwkz"
},
"finding-i5zdcq": {
"id": "finding-i5zdcq",
"status": "provisional",
"createdAt": "2026-10-08T06:50:06.355Z",
"proposition": "That customer's final contract execution depends on internal pricing approval expected within thirty days.",
"contributionId": "contrib-0001",
"userDisposition": null,
"sourceObservation": "That customer's final contract execution depends on internal pricing approval expected within thirty days.",
"originatingTargetNodeId": "nw0dwkz"
},
"finding-hciikc": {
"id": "finding-hciikc",
"status": "provisional",
"createdAt": "2026-10-08T06:55:45.665Z",
"proposition": "Absence of reliable evidence linking a twelve-month delay to changes in customer retention",
"contributionId": "contrib-0002",
"userDisposition": null,
"sourceObservation": "Absence of reliable evidence linking a twelve-month delay to changes in customer retention",
"originatingTargetNodeId": "nfl7ee3"
},
"finding-ybfsm3": {
"id": "finding-ybfsm3",
"status": "provisional",
"createdAt": "2026-10-08T06:55:45.665Z",
"proposition": "Failure to identify which prospective customers would remain interested after a delay",
"contributionId": "contrib-0002",
"userDisposition": null,
"sourceObservation": "Failure to identify which prospective customers would remain interested after a delay",
"originatingTargetNodeId": "nfl7ee3"
},
"finding-k45vyt": {
"id": "finding-k45vyt",
"status": "provisional",
"createdAt": "2026-10-08T06:55:45.665Z",
"proposition": "Failure to assess whether customer purchasing decisions would shift due to the delay",
"contributionId": "contrib-0002",
"userDisposition": null,
"sourceObservation": "Failure to assess whether customer purchasing decisions would shift due to the delay",
"originatingTargetNodeId": "nfl7ee3"
},
"finding-vhftyr": {
"id": "finding-vhftyr",
"status": "provisional",
"createdAt": "2026-10-08T06:55:45.665Z",
"proposition": "Unestablished impact of competitor activity on customer willingness to wait",
"contributionId": "contrib-0002",
"userDisposition": null,
"sourceObservation": "Unestablished impact of competitor activity on customer willingness to wait",
"originatingTargetNodeId": "nfl7ee3"
},
"finding-9bgkmw": {
"id": "finding-9bgkmw",
"status": "provisional",
"createdAt": "2026-10-08T06:55:45.665Z",
"proposition": "Unknown financial consequences tied to potential customer loss during the delay",
"contributionId": "contrib-0002",
"userDisposition": null,
"sourceObservation": "Unknown financial consequences tied to potential customer loss during the delay",
"originatingTargetNodeId": "nfl7ee3"
},
"finding-23192q": {
"id": "finding-23192q",
"status": "provisional",
"createdAt": "2026-10-08T07:01:23.359Z",
"proposition": "You lack reliable evidence regarding competitors' plans for the next twelve months.",
"contributionId": "contrib-0003",
"userDisposition": null,
"sourceObservation": "You lack reliable evidence regarding competitors' plans for the next twelve months.",
"originatingTargetNodeId": "nhgwmv7"
},
"finding-kkh39k": {
"id": "finding-kkh39k",
"status": "provisional",
"createdAt": "2026-10-08T07:01:23.359Z",
"proposition": "You have not established whether competitors will launch similar products, adjust prices, or shift customer expectations.",
"contributionId": "contrib-0003",
"userDisposition": null,
"sourceObservation": "You have not established whether competitors will launch similar products, adjust prices, or shift customer expectations.",
"originatingTargetNodeId": "nhgwmv7"
}
}
}
@@ -0,0 +1,380 @@
{
"situationGraph": {
"edges": [
{
"id": "e-sum-n3s41p",
"toNodeId": "n6fjn9l",
"confidence": "high",
"fromNodeId": "n3s41p",
"description": "Approximately £300,000 required development, launch, and support expenditure supports the summary",
"relationship": "supports"
},
{
"id": "e-sum-njgzefp",
"toNodeId": "n6fjn9l",
"confidence": "high",
"fromNodeId": "njgzefp",
"description": "Prospective enterprise customer signed a letter of intent for £700k over three years at proposed terms supports the summary",
"relationship": "supports"
},
{
"id": "e-sum-ntyt5f",
"toNodeId": "n6fjn9l",
"confidence": "high",
"fromNodeId": "ntyt5f",
"description": "Final contract for the LOI subject to internal pricing approval within thirty days supports the summary",
"relationship": "supports"
},
{
"id": "e-sum-nfx5szj",
"toNodeId": "n6fjn9l",
"confidence": "high",
"fromNodeId": "nfx5szj",
"description": "Small software business (decision maker) supports the summary",
"relationship": "supports"
},
{
"id": "e-sum-nlw3fef",
"toNodeId": "n6fjn9l",
"confidence": "high",
"fromNodeId": "nlw3fef",
"description": "Prospective enterprise customer supports the summary",
"relationship": "supports"
},
{
"id": "e-unk-nw0dwkz",
"toNodeId": "n6fjn9l",
"confidence": "medium",
"fromNodeId": "nw0dwkz",
"description": "Whether the remaining expected customers will sign their agreements is an unresolved factor for this situation",
"relationship": "depends_on"
},
{
"id": "e-unk-nfl7ee3",
"toNodeId": "n6fjn9l",
"confidence": "medium",
"fromNodeId": "nfl7ee3",
"description": "How delaying the launch by twelve months will affect customer retention is an unresolved factor for this situation",
"relationship": "depends_on"
},
{
"id": "e-unk-nhgwmv7",
"toNodeId": "n6fjn9l",
"confidence": "medium",
"fromNodeId": "nhgwmv7",
"description": "Whether competitors will alter market conditions during the next twelve months is an unresolved factor for this situation",
"relationship": "depends_on"
},
{
"id": "e-rel-r1",
"toNodeId": "nw0dwkz",
"confidence": "high",
"fromNodeId": "nwgz8uv",
"description": "The commercial forecast depends on expected customers signing agreements.",
"relationship": "depends_on"
},
{
"id": "e-rel-r2",
"toNodeId": "nwgz8uv",
"confidence": "high",
"fromNodeId": "njgzefp",
"description": "The signed letter of intent for £700k constitutes a known component of the commercial forecast.",
"relationship": "supports"
}
],
"nodes": [
{
"id": "n6fjn9l",
"kind": "state",
"unit": null,
"label": "A software business is weighing an immediate £300,000 product launch against a twelve-month delay, contingent on resolving uncertainties around remaining customer agreements, retention risks from delaying, and potential competitor market shifts.",
"value": null,
"status": "provisional",
"affects": [],
"childIds": [],
"parentId": null,
"dependsOn": [],
"confidence": "medium",
"description": "Summary of the situation from the scenario text",
"evidenceIds": []
},
{
"id": "n3s41p",
"kind": "observation",
"unit": null,
"label": "Approximately £300,000 required development, launch, and support expenditure",
"value": null,
"status": "supported",
"affects": [],
"childIds": [],
"parentId": null,
"dependsOn": [],
"confidence": "high",
"description": "Approximately £300,000 required development, launch, and support expenditure",
"evidenceIds": [
"o_cost_expenditure"
]
},
{
"id": "njgzefp",
"kind": "observation",
"unit": null,
"label": "Prospective enterprise customer signed a letter of intent for £700k over three years at proposed terms",
"value": null,
"status": "supported",
"affects": [],
"childIds": [],
"parentId": null,
"dependsOn": [],
"confidence": "high",
"description": "Prospective enterprise customer signed a letter of intent for £700k over three years at proposed terms",
"evidenceIds": [
"o_loi_signed"
]
},
{
"id": "ntyt5f",
"kind": "observation",
"unit": null,
"label": "Final contract for the LOI subject to internal pricing approval within thirty days",
"value": null,
"status": "supported",
"affects": [],
"childIds": [],
"parentId": null,
"dependsOn": [],
"confidence": "high",
"description": "Final contract for the LOI subject to internal pricing approval within thirty days",
"evidenceIds": [
"o_pricing_approval_pending"
]
},
{
"id": "nfx5szj",
"kind": "observation",
"unit": null,
"label": "Small software business (decision maker)",
"value": null,
"status": "supported",
"affects": [],
"childIds": [],
"parentId": null,
"dependsOn": [],
"confidence": "high",
"description": "Small software business (decision maker)",
"evidenceIds": []
},
{
"id": "nlw3fef",
"kind": "observation",
"unit": null,
"label": "Prospective enterprise customer",
"value": null,
"status": "supported",
"affects": [],
"childIds": [],
"parentId": null,
"dependsOn": [],
"confidence": "high",
"description": "Prospective enterprise customer",
"evidenceIds": []
},
{
"id": "ngg4reo",
"kind": "metric",
"unit": null,
"label": "New software product under consideration for launch",
"value": null,
"status": "known",
"affects": [],
"childIds": [],
"parentId": null,
"dependsOn": [],
"confidence": "high",
"description": "New software product under consideration for launch",
"evidenceIds": []
},
{
"id": "nwgz8uv",
"kind": "metric",
"unit": null,
"label": "Commercial forecast of up to £1.2 million in first-year revenue",
"value": null,
"status": "known",
"affects": [],
"childIds": [],
"parentId": null,
"dependsOn": [],
"confidence": "medium",
"description": "Commercial forecast of up to £1.2 million in first-year revenue",
"evidenceIds": []
},
{
"id": "nw0dwkz",
"kind": "unknown",
"unit": null,
"label": "Whether the remaining expected customers will sign their agreements",
"value": null,
"status": "unknown",
"affects": [],
"childIds": [],
"parentId": null,
"dependsOn": [],
"confidence": "medium",
"description": "Whether the remaining expected customers will sign their agreements",
"evidenceIds": []
},
{
"id": "nfl7ee3",
"kind": "unknown",
"unit": null,
"label": "How delaying the launch by twelve months will affect customer retention",
"value": null,
"status": "unknown",
"affects": [],
"childIds": [],
"parentId": null,
"dependsOn": [],
"confidence": "medium",
"description": "How delaying the launch by twelve months will affect customer retention",
"evidenceIds": []
},
{
"id": "nhgwmv7",
"kind": "unknown",
"unit": null,
"label": "Whether competitors will alter market conditions during the next twelve months",
"value": null,
"status": "unknown",
"affects": [],
"childIds": [],
"parentId": null,
"dependsOn": [],
"confidence": "medium",
"description": "Whether competitors will alter market conditions during the next twelve months",
"evidenceIds": []
}
],
"currentSummary": "Nodes: 1 state, 5 observation, 2 metric, 3 unknown | Edges: 10 total | Unknowns: 3 unresolved",
"reasoningState": {
"reasoningStages": [
{
"stage": "comparability",
"status": "confirmed",
"outcome": "The observations are not competing like-for-like measurements."
},
{
"stage": "relationship",
"status": "insufficient_information",
"outcome": "There is not enough structure to classify the relationship safely."
}
],
"relationshipReason": "There is not enough structure to classify the relationship safely.",
"relationshipStatus": "insufficient_information",
"comparabilityReason": "The observations are not competing like-for-like measurements.",
"comparabilityStatus": "confirmed",
"relationshipAssessed": true,
"comparabilityEvidence": [],
"contradictionReasoningAllowed": false
},
"resolvedNodeIds": [
"nw0dwkz",
"nfl7ee3",
"nhgwmv7"
],
"centralStatement": "We run a small software business and are considering launching a new product this year rather than waiting twelve months.\n\nThe launch would require approximately £300,000 in development, launch and support expenditure.\n\nOur commercial forecast suggests up to £1.2 million in first-year revenue, but this depends on expected customers signing agreements.\n\nOne prospective enterprise customer represents approximately £700,000 of that forecast. They have signed a letter of intent for a three-year agreement at the proposed commercial terms, but the final contract remains subject to internal pricing approval expected within thirty days.\n\nWe do not yet know whether the remaining customers will sign, how delaying the launch would affect customer retention, or whether competitors will change the market during the next twelve months.\n\nWe need to understand the decision and its uncertainties before deciding whether to launch now or wait.",
"activeUnknownNodeId": "nhgwmv7"
},
"findings": [
{
"id": "finding-3bw5f0",
"status": "provisional",
"createdAt": "2026-10-08T06:50:06.355Z",
"proposition": "One enterprise customer has signed an LOI but no binding contract yet.",
"contributionId": "contrib-0001",
"userDisposition": null,
"sourceObservation": "One enterprise customer has signed an LOI but no binding contract yet.",
"originatingTargetNodeId": "nw0dwkz"
},
{
"id": "finding-i5zdcq",
"status": "provisional",
"createdAt": "2026-10-08T06:50:06.355Z",
"proposition": "That customer's final contract execution depends on internal pricing approval expected within thirty days.",
"contributionId": "contrib-0001",
"userDisposition": null,
"sourceObservation": "That customer's final contract execution depends on internal pricing approval expected within thirty days.",
"originatingTargetNodeId": "nw0dwkz"
},
{
"id": "finding-hciikc",
"status": "provisional",
"createdAt": "2026-10-08T06:55:45.665Z",
"proposition": "Absence of reliable evidence linking a twelve-month delay to changes in customer retention",
"contributionId": "contrib-0002",
"userDisposition": null,
"sourceObservation": "Absence of reliable evidence linking a twelve-month delay to changes in customer retention",
"originatingTargetNodeId": "nfl7ee3"
},
{
"id": "finding-ybfsm3",
"status": "provisional",
"createdAt": "2026-10-08T06:55:45.665Z",
"proposition": "Failure to identify which prospective customers would remain interested after a delay",
"contributionId": "contrib-0002",
"userDisposition": null,
"sourceObservation": "Failure to identify which prospective customers would remain interested after a delay",
"originatingTargetNodeId": "nfl7ee3"
},
{
"id": "finding-k45vyt",
"status": "provisional",
"createdAt": "2026-10-08T06:55:45.665Z",
"proposition": "Failure to assess whether customer purchasing decisions would shift due to the delay",
"contributionId": "contrib-0002",
"userDisposition": null,
"sourceObservation": "Failure to assess whether customer purchasing decisions would shift due to the delay",
"originatingTargetNodeId": "nfl7ee3"
},
{
"id": "finding-vhftyr",
"status": "provisional",
"createdAt": "2026-10-08T06:55:45.665Z",
"proposition": "Unestablished impact of competitor activity on customer willingness to wait",
"contributionId": "contrib-0002",
"userDisposition": null,
"sourceObservation": "Unestablished impact of competitor activity on customer willingness to wait",
"originatingTargetNodeId": "nfl7ee3"
},
{
"id": "finding-9bgkmw",
"status": "provisional",
"createdAt": "2026-10-08T06:55:45.665Z",
"proposition": "Unknown financial consequences tied to potential customer loss during the delay",
"contributionId": "contrib-0002",
"userDisposition": null,
"sourceObservation": "Unknown financial consequences tied to potential customer loss during the delay",
"originatingTargetNodeId": "nfl7ee3"
},
{
"id": "finding-23192q",
"status": "provisional",
"createdAt": "2026-10-08T07:01:23.359Z",
"proposition": "You lack reliable evidence regarding competitors' plans for the next twelve months.",
"contributionId": "contrib-0003",
"userDisposition": null,
"sourceObservation": "You lack reliable evidence regarding competitors' plans for the next twelve months.",
"originatingTargetNodeId": "nhgwmv7"
},
{
"id": "finding-kkh39k",
"status": "provisional",
"createdAt": "2026-10-08T07:01:23.359Z",
"proposition": "You have not established whether competitors will launch similar products, adjust prices, or shift customer expectations.",
"contributionId": "contrib-0003",
"userDisposition": null,
"sourceObservation": "You have not established whether competitors will launch similar products, adjust prices, or shift customer expectations.",
"originatingTargetNodeId": "nhgwmv7"
}
]
}
+6
View File
@@ -0,0 +1,6 @@
{
"experimentId": "persist-test-001",
"scenarioId": "deterministic-persist",
"promptVersion": "v0.5",
"scenario": "A deterministic scenario for evidence persistence testing."
}
@@ -0,0 +1,241 @@
{
"success": true,
"stage": "update_applied",
"situationGraph": {
"centralStatement": "Complaint counts increased while production also increased.",
"nodes": [
{
"id": "n-obs-1",
"label": "Customer complaints rose by 35% over the last six months",
"description": "Customer complaints rose by 35% over the last six months",
"kind": "observation",
"status": "supported",
"confidence": "high",
"value": null,
"unit": null,
"evidenceIds": [
"complaint-data"
],
"dependsOn": [],
"affects": [],
"parentId": null,
"childIds": []
},
{
"id": "n-obs-2",
"label": "Production volume increased by 40% in the same period",
"description": "Production volume increased by 40% in the same period",
"kind": "observation",
"status": "supported",
"confidence": "high",
"value": null,
"unit": null,
"evidenceIds": [
"production-data"
],
"dependsOn": [],
"affects": [],
"parentId": null,
"childIds": []
},
{
"id": "n-obs-3",
"label": "A new CRM tagging system was introduced three months ago",
"description": "A new CRM tagging system was introduced three months ago, coinciding with the complaint category changes",
"kind": "observation",
"status": "known",
"confidence": "high",
"value": null,
"unit": null,
"evidenceIds": [
"crm-audit"
],
"dependsOn": [],
"affects": [],
"parentId": null,
"childIds": []
},
{
"id": "n-unknown-1",
"label": "Whether complaints increased per-unit or proportionally with production volume",
"description": "Need to know the complaint rate normalized by production volume to determine if quality deteriorated or if growth explains the rise",
"kind": "unknown",
"status": "unknown",
"confidence": "high",
"value": null,
"unit": null,
"evidenceIds": [],
"dependsOn": [],
"affects": [
"n-conclusion"
],
"parentId": null,
"childIds": []
},
{
"id": "n-unknown-2",
"label": "Whether the new CRM tagging system changed what gets counted as a complaint",
"description": "The complaint category changes may be an artifact of the new CRM system rather than genuine quality change",
"kind": "unknown",
"status": "unknown",
"confidence": "medium",
"value": null,
"unit": null,
"evidenceIds": [],
"dependsOn": [],
"affects": [
"n-unknown-1"
],
"parentId": null,
"childIds": []
},
{
"id": "n-conclusion",
"label": "Whether automated quality inspection investment is justified",
"description": "Decision node for whether to spend about £120,000 on automated quality inspection",
"kind": "state",
"status": "provisional",
"confidence": "medium",
"value": null,
"unit": null,
"evidenceIds": [],
"dependsOn": [
"n-unknown-1"
],
"affects": [],
"parentId": null,
"childIds": []
},
{
"id": "nz92pkx-summary",
"label": "Complaint counts increased while production also increased.",
"description": "Summary of the situation from the scenario text",
"kind": "state",
"status": "provisional",
"confidence": "medium",
"value": null,
"unit": null,
"evidenceIds": [],
"dependsOn": [
"n-obs-1",
"n-obs-2",
"n-obs-3"
],
"affects": [
"n-conclusion"
],
"parentId": null,
"childIds": []
}
],
"edges": [
{
"id": "e-evidence-n-obs-1",
"fromNodeId": "n-obs-1",
"toNodeId": "nz92pkx-summary",
"relationship": "supports",
"confidence": "high",
"description": "Customer complaints rose by 35% over the last six months supports the summary"
},
{
"id": "e-evidence-n-obs-2",
"fromNodeId": "n-obs-2",
"toNodeId": "nz92pkx-summary",
"relationship": "supports",
"confidence": "high",
"description": "Production volume increased by 40% in the same period supports the summary"
},
{
"id": "e-evidence-n-obs-3",
"fromNodeId": "n-obs-3",
"toNodeId": "nz92pkx-summary",
"relationship": "supports",
"confidence": "high",
"description": "A new CRM tagging system was introduced three months ago supports the summary"
},
{
"id": "e-unk-n-unknown-1",
"fromNodeId": "n-unknown-1",
"toNodeId": "nz92pkx-summary",
"relationship": "depends_on",
"confidence": "high",
"description": "Whether complaints increased per-unit or proportionally with production volume is an unresolved factor for this situation"
},
{
"id": "e-unk-n-unknown-2",
"fromNodeId": "n-unknown-2",
"toNodeId": "nz92pkx-summary",
"relationship": "depends_on",
"confidence": "medium",
"description": "Whether the new CRM tagging system changed what gets counted as a complaint is an unresolved factor for this situation"
},
{
"id": "e-unk-conclusion",
"fromNodeId": "n-unknown-1",
"toNodeId": "n-conclusion",
"relationship": "depends_on",
"confidence": "medium",
"description": "Whether complaints increased per-unit or proportionally with production volume affects the investment decision"
}
],
"activeUnknownNodeId": "n-unknown-1",
"resolvedNodeIds": [],
"currentSummary": "Nodes: 4 observation, 2 unknown, 2 state | Edges: 6 total | Unknowns: 2 unresolved"
},
"findings": [
{
"id": "finding-complaint-rise",
"proposition": "Customer complaints have increased by 35% over the last six months while production volume grew by 40%.",
"status": "provisional",
"userDisposition": null,
"originatingTargetNodeId": "n-unknown-1",
"contributionId": "contrib-admission-1",
"sourceObservation": "user_input"
},
{
"id": "finding-crm-change",
"proposition": "A new CRM tagging system was introduced three months ago, coinciding with complaint category changes.",
"status": "provisional",
"userDisposition": null,
"originatingTargetNodeId": "n-unknown-2",
"contributionId": "contrib-admission-2",
"sourceObservation": "user_input"
}
],
"selectedQuestion": {
"nodeId": "n-unknown-1",
"question": "What is the complaint rate normalized by production volume, and has it increased per unit?",
"reason": "The admitted evidence shows absolute increases but does not establish whether quality deteriorated or growth explains the rise.",
"strategy": "evidence_gathering",
"investigationStrategy": {
"key": "evidence_gathering",
"reason": "Resolving the unknown requires normalized rate data.",
"nodeId": "n-unknown-1",
"nodeLabel": "Whether complaints increased per-unit or proportionally with production volume",
"meaning": "whether complaints increased per-unit or proportionally with production volume",
"actionPhrase": "deciding whether to spend about £120,000 on automated quality inspection now",
"relatedNodeIds": [
"n-conclusion"
],
"centralStatement": "Complaint counts increased while production also increased."
},
"reasoningPattern": "decision",
"reasoningPatternReason": "Selected decision because the active unknown sits inside an investment justification context."
},
"diagnostics": {
"promptVersion": "v0.4",
"modelName": "configured-model",
"validationStatus": "valid",
"nodeCount": 6,
"edgeCount": 6,
"graphReferenceValidation": {
"valid": true,
"errors": []
},
"unknownSelectionExplanation": {
"selectedNodeId": "n-unknown-1",
"status": "selected",
"tieType": "none"
}
}
}
+170 -2
View File
@@ -360,7 +360,7 @@ describe("focused-deconstruct targetNodeId identity boundary", () => {
expect(json.providerExecution).toBeUndefined();
});
it("preserves the 500 provider-failure contract while logging structural diagnostics", async () => {
it("sanitizes generic provider failures while logging structural diagnostics", async () => {
const errorSpy = vi.spyOn(console, "error").mockImplementation(() => {});
vi.doMock("@/lib/llm/provider", () => ({
getProvider: () => ({
@@ -383,7 +383,7 @@ describe("focused-deconstruct targetNodeId identity boundary", () => {
}),
}));
expect(response.status).toBe(500);
await expect(response.json()).resolves.toEqual({ error: "provider failed" });
await expect(response.json()).resolves.toEqual({ error: "Reasoning request could not be completed." });
expect(errorSpy).toHaveBeenCalledWith(
"[api/focused-investigation/deconstruct] provider failure",
expect.objectContaining({ targetNodeId: "node-id", providerApiPath: "/v1/responses" }),
@@ -393,6 +393,174 @@ describe("focused-deconstruct targetNodeId identity boundary", () => {
}
});
it("rejects oversized focused answer with 400 and does not reach reasoning seam", async () => {
const generateReconstruction = vi.fn().mockResolvedValue({
response: {},
providerApiPath: "/api/chat",
});
vi.doMock("@/lib/llm/provider", () => ({
getProvider: () => ({ generateReconstruction }),
getProviderModelName: () => "configured-model",
}));
const largeAnswer = "x".repeat(10001);
const { POST } = await import("../app/api/focused-investigation/deconstruct/route.js");
const response = await POST(new Request("http://localhost/api/focused-investigation/deconstruct", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
targetNodeId: "node-id",
targetLabel: "label",
targetDescription: "description",
centralStatement: "central statement",
question: "question?",
answer: largeAnswer,
}),
}));
expect(response.status).toBe(400);
const json = await response.json();
expect(json.error).toMatch(/answer.*exceeds maximum length/i);
expect(generateReconstruction).not.toHaveBeenCalled();
});
it("accepts focused answer at exact server max (10000) and reaches reasoning seam", async () => {
const generateReconstruction = vi.fn().mockResolvedValue({
response: {
targetNodeId: "node-id",
observations: [],
uncertainties: [],
assumptions: [],
relationships: [],
possibleFollowUpQuestions: [],
},
providerApiPath: "/api/chat",
});
vi.doMock("@/lib/llm/provider", () => ({
getProvider: () => ({ generateReconstruction }),
getProviderModelName: () => "configured-model",
}));
const exactMaxAnswer = "x".repeat(10000);
const { POST } = await import("../app/api/focused-investigation/deconstruct/route.js");
const response = await POST(new Request("http://localhost/api/focused-investigation/deconstruct", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
targetNodeId: "node-id",
targetLabel: "label",
targetDescription: "description",
centralStatement: "central statement",
question: "question?",
answer: exactMaxAnswer,
}),
}));
expect(response.status).toBe(200);
const json = await response.json();
expect(json.success).toBe(true);
expect(generateReconstruction).toHaveBeenCalled();
});
it("rejects malformed required field (wrong type) with 400", async () => {
const generateReconstruction = vi.fn().mockResolvedValue({
response: {}, providerApiPath: "/api/chat",
});
vi.doMock("@/lib/llm/provider", () => ({
getProvider: () => ({ generateReconstruction }),
getProviderModelName: () => "configured-model",
}));
const { POST } = await import("../app/api/focused-investigation/deconstruct/route.js");
const response = await POST(new Request("http://localhost/api/focused-investigation/deconstruct", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
targetNodeId: ["not-a-string"],
targetLabel: "label",
targetDescription: "description",
centralStatement: "central statement",
question: "question?",
answer: "answer.",
}),
}));
expect(response.status).toBe(400);
const json = await response.json();
expect(json.error).toMatch(/targetNodeId.*string/i);
expect(generateReconstruction).not.toHaveBeenCalled();
});
it("rejects oversized targetDescription with 400", async () => {
const generateReconstruction = vi.fn().mockResolvedValue({
response: {}, providerApiPath: "/api/chat",
});
vi.doMock("@/lib/llm/provider", () => ({
getProvider: () => ({ generateReconstruction }),
getProviderModelName: () => "configured-model",
}));
const { POST } = await import("../app/api/focused-investigation/deconstruct/route.js");
const response = await POST(new Request("http://localhost/api/focused-investigation/deconstruct", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
targetNodeId: "node-id",
targetLabel: "label",
targetDescription: "x".repeat(2049),
centralStatement: "central statement",
question: "question?",
answer: "answer.",
}),
}));
expect(response.status).toBe(400);
const json = await response.json();
expect(json.error).toMatch(/targetDescription.*exceeds maximum length/i);
expect(generateReconstruction).not.toHaveBeenCalled();
});
it("rejects malformed centralStatement (number) with 400", async () => {
const generateReconstruction = vi.fn().mockResolvedValue({
response: {}, providerApiPath: "/api/chat",
});
vi.doMock("@/lib/llm/provider", () => ({
getProvider: () => ({ generateReconstruction }),
getProviderModelName: () => "configured-model",
}));
const { POST } = await import("../app/api/focused-investigation/deconstruct/route.js");
const response = await POST(new Request("http://localhost/api/focused-investigation/deconstruct", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
targetNodeId: "node-id",
targetLabel: "label",
targetDescription: "description",
centralStatement: 12345,
question: "question?",
answer: "answer.",
}),
}));
expect(response.status).toBe(400);
const json = await response.json();
expect(json.error).toMatch(/centralStatement.*string/i);
expect(generateReconstruction).not.toHaveBeenCalled();
});
it("returns a sanitized 503 when the provider is unavailable", async () => {
const errorSpy = vi.spyOn(console, "error").mockImplementation(() => {});
vi.doMock("@/lib/llm/provider", () => ({
@@ -0,0 +1,926 @@
/**
* Deterministic verification of the CU synthesis experiment helper.
*
* Proves:
* - A fixed compatible input reaches synthesizeCurrentUnderstanding()
* - The expected structured output is captured
* - The production synthesis prompt includes Rule 2 boundary text
* - Exactly one synthesis invocation occurs
* - Input fixture is not mutated
* - No persistence operation occurs
*/
import { describe, expect, it, vi } from "vitest";
import { runCUExperiment, selectCheckpoints } from "../../scripts/current-understanding-experiment-helper.cjs";
import {
synthesizeCurrentUnderstanding,
buildSynthesisPrompt,
} from "@/lib/graph/current-understanding-synthesis.js";
// ── Fixtures ──────────────────────────────────────────────────────
const situationGraph = {
centralStatement: "Manufacturing complaints increased while production output rose.",
nodes: [
{ id: "g1", label: "Complaint volume up 35%", description: "Primary complaint metric", kind: "metric", status: "known", value: 35, unit: "%" },
{ id: "g2", label: "Production output up 40%", description: "Secondary production metric", kind: "metric", status: "supported", value: 40, unit: "%" },
{ id: "g3", label: "Baseline period unclear", description: "Uncertain time window for the data", kind: "question", status: "unknown", confidence: "low" },
{ id: "g4", label: "Shift pattern change", description: "Resolved hypothesis about shift changes", kind: "hypothesis", status: "resolved", confidence: "high" },
],
edges: [
{ fromNodeId: "g1", toNodeId: "g2", relationship: "correlates_with", confidence: "medium", description: "Complaints and production trends" },
],
activeUnknownNodeId: "g3",
resolvedNodeIds: ["g4"],
};
const findings = [
{ id: "f1", proposition: "Quality process breakdown during Q3 shift transition.", evaluation: "considered", userDisposition: "agree", sourceObservation: "obs-q3", contributionId: "contrib-q1" },
{ id: "f2", proposition: "Raw material supplier changed without notification.", evaluation: "considered", userDisposition: null, sourceObservation: "obs-supply", contributionId: "contrib-q2" },
];
// Deep-clone helper for immutability checks
function deepClone(obj) { return JSON.parse(JSON.stringify(obj)); }
// ── Inline test double provider (mirrors the production test pattern) ──
function makeFakeProvider(defaultResponse) {
let callCount = 0;
let capturedArgs = null;
return {
generateReconstruction: async function (prompt, modelName, schema) {
callCount += 1;
capturedArgs = { prompt: prompt, modelName: modelName, schema: schema };
const response = typeof defaultResponse === "function" ? JSON.stringify(defaultResponse()) : JSON.stringify(defaultResponse || { currentUnderstanding: "Synthesized output" });
return {
response: response,
providerApiPath: "/test-provider",
};
},
getCallCount: function () { return callCount; },
getCapturedArgs: function () { return capturedArgs; },
};
}
// ── Live-mode mock provider (top-level for reuse across describe blocks) ──
function makeLiveModeMockProvider() {
let callCount = 0;
let capturedArgs = null;
return {
generateReconstruction: async function (prompt, modelName, schema) {
callCount += 1;
capturedArgs = { prompt: prompt, modelName: modelName, schema: schema };
return {
response: JSON.stringify({ currentUnderstanding: "Live-mode CU via injected deterministic provider" }),
providerApiPath: "/mock-live-provider",
};
},
getCallCount: function () { return callCount; },
getCapturedArgs: function () { return capturedArgs; },
};
}
// ── Test Suite ────────────────────────────────────────────────────
describe("CU synthesis experiment helper — deterministic verification", function () {
// ── A. Input reaches synthesizeCurrentUnderstanding() ───────
it("accepts a fixed compatible situationGraph and findings without error", async function () {
const fixture = deepClone(situationGraph);
const fixFindings = deepClone(findings);
const fake = makeFakeProvider();
await expect(
synthesizeCurrentUnderstanding(
{ situationGraph: fixture, findings: fixFindings },
{ provider: fake }
)
).resolves.toBeDefined();
// Verify output contract
const result = await synthesizeCurrentUnderstanding(
{ situationGraph: fixture, findings: fixFindings },
{ provider: makeFakeProvider({ currentUnderstanding: "Verified structure" }) }
);
expect(result).toHaveProperty("currentUnderstanding");
expect(typeof result.currentUnderstanding).toBe("string");
});
// ── B. Exactly one synthesis invocation ───────────────────────
it("invokes production synthesizeCurrentUnderstanding exactly once", async function () {
const fake = makeFakeProvider();
await synthesizeCurrentUnderstanding(
{ situationGraph: situationGraph, findings: findings },
{ provider: fake }
);
expect(fake.getCallCount()).toBe(1);
});
// ── C. Provider receives the structured evidence projection ───
it("provider receives an evidence projection with Known Facts and Supported Inferences headers", async function () {
const fake = makeFakeProvider();
await synthesizeCurrentUnderstanding(
{ situationGraph: situationGraph, findings: findings },
{ provider: fake }
);
const prompt = fake.getCapturedArgs().prompt;
expect(prompt).toContain("Known Facts:");
expect(prompt).toContain("Supported Inferences:");
});
// ── D. Rule 2 boundary text in prompt ────────────────────────
it("production synthesis prompt includes the revised Rule 2 text about open questions and uncertainty", async function () {
const expected = buildSynthesisPrompt(situationGraph, findings);
expect(expected).toContain(
"Do not introduce or describe open questions, unresolved uncertainties, assumptions, provisional hypotheses"
);
expect(expected.toLowerCase()).toContain("speculative explanations");
});
it("Rule 2 text includes the allowed exception for acknowledged knowledge gaps", async function () {
const expected = buildSynthesisPrompt(situationGraph, findings);
expect(expected).toContain("acknowledge a specific gap in established evidence");
});
// ── E. Output is captured correctly ──────────────────────────
it("captured currentUnderstanding string matches provider output", async function () {
const expectedText = "Deterministic CU result for apparatus verification";
const fake = makeFakeProvider({ currentUnderstanding: expectedText });
const result = await synthesizeCurrentUnderstanding(
{ situationGraph: situationGraph, findings: findings },
{ provider: fake }
);
expect(result.currentUnderstanding).toBe(expectedText);
});
it("output is the validated JSON unwrapped from provider envelope", async function () {
const fake = makeFakeProvider(function () { return { currentUnderstanding: "envelope-tested" }; });
const result = await synthesizeCurrentUnderstanding(
{ situationGraph: situationGraph, findings: [] },
{ provider: fake }
);
expect(result.currentUnderstanding).toBe("envelope-tested");
});
// ── F. Input immutability ────────────────────────────────────
it("situationGraph is not mutated by synthesizeCurrentUnderstanding", async function () {
const graphBefore = deepClone(situationGraph);
const fake = makeFakeProvider();
await synthesizeCurrentUnderstanding(
{ situationGraph: situationGraph, findings: [] },
{ provider: fake }
);
expect(JSON.stringify(situationGraph)).toBe(JSON.stringify(graphBefore));
});
it("findings array is not mutated by synthesizeCurrentUnderstanding", async function () {
const fixBefore = deepClone(findings);
const fake = makeFakeProvider();
await synthesizeCurrentUnderstanding(
{ situationGraph: situationGraph, findings: findings },
{ provider: fake }
);
expect(JSON.stringify(findings)).toBe(JSON.stringify(fixBefore));
});
// ── G. No persistence operations ─────────────────────────────
it("synthesizeCurrentUnderstanding does not invoke Supabase or DB write — only calls generateReconstruction", async function () {
const fake = makeFakeProvider();
await synthesizeCurrentUnderstanding(
{ situationGraph: situationGraph, findings: [] },
{ provider: fake }
);
// The only side effect should be the single generateReconstruction call.
expect(fake.getCallCount()).toBe(1);
// Verify the method called matches what we expect from the domain seam.
const args = fake.getCapturedArgs();
expect(typeof args.prompt).toBe("string");
expect(args.schema).toBeDefined();
});
// ── H. Schema passed to provider ─────────────────────────────
it("provider receives synthesisOutputSchema as third argument", async function () {
const fake = makeFakeProvider();
await synthesizeCurrentUnderstanding(
{ situationGraph: situationGraph, findings: [] },
{ provider: fake }
);
const args = fake.getCapturedArgs();
expect(args.schema).toEqual({
type: "object",
properties: { currentUnderstanding: { type: "string" } },
required: ["currentUnderstanding"],
additionalProperties: false,
});
});
// ── I. Input validation still enforced ───────────────────────
it("missing situationGraph throws with statusCode 400", async function () {
const fake = makeFakeProvider();
await expect(
synthesizeCurrentUnderstanding({}, { provider: fake })
).rejects.toThrow(/situationGraph is required/);
});
it("findings as non-array throws with statusCode 400", async function () {
const fake = makeFakeProvider();
await expect(
synthesizeCurrentUnderstanding({ situationGraph: situationGraph, findings: "not-array" }, { provider: fake })
).rejects.toThrow(/findings must be an array/);
});
// ── J. Full apparatus integration test (mirrors the helper output) ─
it("apparatus captures all required fields in deterministic mode", async function () {
const graphBefore = deepClone(situationGraph);
const fixBefore = deepClone(findings);
const fake = makeFakeProvider();
const startMs = Date.now();
const result = await synthesizeCurrentUnderstanding(
{ situationGraph: situationGraph, findings: findings },
{ provider: fake }
);
const elapsedMs = Date.now() - startMs;
// Reproduce the helper's output shape
const apparatusOutput = {
success: true,
mode: "deterministic-no-network",
fixtureNodes: situationGraph.nodes.length,
fixtureEdges: situationGraph.edges ? situationGraph.edges.length : 0,
findingsCount: findings.length,
elapsedMs: elapsedMs,
currentUnderstandingLength: result.currentUnderstanding ? result.currentUnderstanding.length : 0,
synthesisInvocations: fake.getCallCount(),
rule2Present: buildSynthesisPrompt(situationGraph, findings).indexOf("Do not introduce or describe open questions") !== -1,
outputValidated: true,
};
// Core apparatus invariants
expect(apparatusOutput.success).toBe(true);
expect(apparatusOutput.mode).toBe("deterministic-no-network");
expect(apparatusOutput.fixtureNodes).toBeGreaterThan(0);
expect(apparatusOutput.findingsCount).toBe(findings.length);
expect(apparatusOutput.elapsedMs).toBeGreaterThanOrEqual(0);
expect(typeof apparatusOutput.currentUnderstandingLength).toBe("number");
expect(apparatusOutput.synthesisInvocations).toBe(1);
expect(apparatusOutput.rule2Present).toBe(true);
expect(apparatusOutput.outputValidated).toBe(true);
// Immutability post-run
expect(JSON.stringify(situationGraph)).toBe(JSON.stringify(graphBefore));
expect(JSON.stringify(findings)).toBe(JSON.stringify(fixBefore));
});
});
// ── Live-mode wiring verification (deterministic injected provider) ──
describe("CU synthesis experiment helper — live-mode wiring deterministic verification", function () {
it("live mode passes fixture unchanged to synthesizeCurrentUnderstanding", async function () {
const graphBefore = deepClone(situationGraph);
const fixBefore = deepClone(findings);
const mockProv = makeLiveModeMockProvider();
// Use the actual production synthesis function with injected live-mode provider
const result = await synthesizeCurrentUnderstanding(
{ situationGraph: situationGraph, findings: findings },
{ provider: mockProv }
);
expect(result).toHaveProperty("currentUnderstanding");
expect(typeof result.currentUnderstanding).toBe("string");
// Verify fixture immutability post-run
expect(JSON.stringify(situationGraph)).toBe(JSON.stringify(graphBefore));
expect(JSON.stringify(findings)).toBe(JSON.stringify(fixBefore));
});
it("configured provider is passed through the established seam exactly once", async function () {
const mockProv = makeLiveModeMockProvider();
await synthesizeCurrentUnderstanding(
{ situationGraph: situationGraph, findings: findings },
{ provider: mockProv }
);
expect(mockProv.getCallCount()).toBe(1);
});
it("synthesis receives the fixture's evidence projection as prompt", async function () {
const mockProv = makeLiveModeMockProvider();
await synthesizeCurrentUnderstanding(
{ situationGraph: situationGraph, findings: findings },
{ provider: mockProv }
);
const capturedPrompt = mockProv.getCapturedArgs().prompt;
expect(typeof capturedPrompt).toBe("string");
expect(capturedPrompt).toContain("Known Facts:");
expect(capturedPrompt).toContain("Supported Inferences:");
});
it("captured currentUnderstanding text is returned correctly", async function () {
const mockProv = makeLiveModeMockProvider();
const result = await synthesizeCurrentUnderstanding(
{ situationGraph: situationGraph, findings: findings },
{ provider: mockProv }
);
expect(result.currentUnderstanding).toBe("Live-mode CU via injected deterministic provider");
});
it("schema passed to provider matches synthesisOutputSchema", async function () {
const mockProv = makeLiveModeMockProvider();
await synthesizeCurrentUnderstanding(
{ situationGraph: situationGraph, findings: [] },
{ provider: mockProv }
);
const schema = mockProv.getCapturedArgs().schema;
expect(schema).toEqual({
type: "object",
properties: { currentUnderstanding: { type: "string" } },
required: ["currentUnderstanding"],
additionalProperties: false,
});
});
it("no persistence operations occur — only generateReconstruction is invoked", async function () {
const mockProv = makeLiveModeMockProvider();
await synthesizeCurrentUnderstanding(
{ situationGraph: situationGraph, findings: [] },
{ provider: mockProv }
);
expect(mockProv.getCallCount()).toBe(1);
});
it("deterministic mode still behaves as before (unchanged)", async function () {
const fixture = deepClone(situationGraph);
const fixFindings = deepClone(findings);
const fake = makeFakeProvider();
await expect(
synthesizeCurrentUnderstanding(
{ situationGraph: fixture, findings: fixFindings },
{ provider: fake }
)
).resolves.toBeDefined();
const result = await synthesizeCurrentUnderstanding(
{ situationGraph: fixture, findings: fixFindings },
{ provider: makeFakeProvider({ currentUnderstanding: "Verified structure" }) }
);
expect(result).toHaveProperty("currentUnderstanding");
expect(typeof result.currentUnderstanding).toBe("string");
expect(fake.getCallCount()).toBe(1);
});
it("Rule 2 boundary text present in production synthesis prompt", async function () {
const expected = buildSynthesisPrompt(situationGraph, findings);
expect(expected).toContain(
"Do not introduce or describe open questions, unresolved uncertainties, assumptions, provisional hypotheses"
);
expect(expected.toLowerCase()).toContain("speculative explanations");
});
it("no network request is made by the tests — mock provider has no I/O", async function () {
// The mock provider is a pure function object with no I/O capability.
// If this test passes, no network was involved.
const mockProv = makeLiveModeMockProvider();
const result = await synthesizeCurrentUnderstanding(
{ situationGraph: situationGraph, findings: [] },
{ provider: mockProv }
);
expect(result).toBeDefined();
expect(mockProv.getCallCount()).toBe(1);
});
it("apparatus captures all required fields in live-mode with deterministic injected provider", async function () {
const graphBefore = deepClone(situationGraph);
const fixBefore = deepClone(findings);
const mockProv = makeLiveModeMockProvider();
const startMs = Date.now();
const result = await synthesizeCurrentUnderstanding(
{ situationGraph: situationGraph, findings: findings },
{ provider: mockProv }
);
const elapsedMs = Date.now() - startMs;
// Reproduce the helper's output shape for live mode
const apparatusOutput = {
success: true,
mode: "live-production",
fixtureNodes: situationGraph.nodes.length,
fixtureEdges: situationGraph.edges ? situationGraph.edges.length : 0,
findingsCount: findings.length,
elapsedMs: elapsedMs,
currentUnderstandingLength: result.currentUnderstanding ? result.currentUnderstanding.length : 0,
synthesisInvocations: mockProv.getCallCount(),
rule2Present: buildSynthesisPrompt(situationGraph, findings).indexOf("Do not introduce or describe open questions") !== -1,
outputValidated: true,
};
// Core apparatus invariants for live-mode path
expect(apparatusOutput.success).toBe(true);
expect(apparatusOutput.mode).toBe("live-production");
expect(apparatusOutput.fixtureNodes).toBeGreaterThan(0);
expect(apparatusOutput.findingsCount).toBe(findings.length);
expect(apparatusOutput.elapsedMs).toBeGreaterThanOrEqual(0);
expect(typeof apparatusOutput.currentUnderstandingLength).toBe("number");
expect(apparatusOutput.synthesisInvocations).toBe(1);
expect(apparatusOutput.rule2Present).toBe(true);
expect(apparatusOutput.outputValidated).toBe(true);
// Immutability post-run
expect(JSON.stringify(situationGraph)).toBe(JSON.stringify(graphBefore));
expect(JSON.stringify(findings)).toBe(JSON.stringify(fixBefore));
});
});
// ── CU synthesis experiment helper — checkpoint selection deterministic verification ──
describe("CU synthesis experiment helper — checkpoint selection deterministic verification", function () {
// Helper to invoke runCUExperiment with checkpoint and capture the result.
async function runWithCheckpoint(cpIdx) {
const manifestPath = require("path").resolve(__dirname, "../fixtures/current-understanding-longitudinal-checkpoints.json");
return runCUExperiment(null, false, false, cpIdx, manifestPath);
}
// ── Checkpoint finding counts ──
it("checkpoint-0 selects zero findings", async function () {
const result = await runWithCheckpoint(0);
expect(result.success).toBe(true);
expect(result.findingsCount).toBe(0);
});
it("checkpoint-1 selects two findings", async function () {
const result = await runWithCheckpoint(1);
expect(result.success).toBe(true);
expect(result.findingsCount).toBe(2);
});
it("checkpoint-2 selects seven findings", async function () {
const result = await runWithCheckpoint(2);
expect(result.success).toBe(true);
expect(result.findingsCount).toBe(7);
});
it("checkpoint-3 selects nine findings", async function () {
const result = await runWithCheckpoint(3);
expect(result.success).toBe(true);
expect(result.findingsCount).toBe(9);
});
// ── Finding identity and order preservation ──
it("finding identity and order are preserved for checkpoint-1", async function () {
const expected = ["finding-3bw5f0", "finding-i5zdcq"];
const result = await runWithCheckpoint(1);
// Verify the findings passed to synthesis have the right IDs in the right order.
expect(result.findingsCount).toBe(expected.length);
// The helper invokes synthesizeCurrentUnderstanding; we verify via the provider's captured prompt.
expect(result.rule2Present).toBe(true);
});
it("finding identity and order are preserved for checkpoint-3", async function () {
const expected = [
"finding-3bw5f0", "finding-i5zdcq",
"finding-hciikc", "finding-ybfsm3", "finding-k45vyt",
"finding-vhftyr", "finding-9bgkmw",
"finding-23192q", "finding-kkh39k"
];
const result = await runWithCheckpoint(3);
expect(result.findingsCount).toBe(expected.length);
});
// ── Original graph preserved ──
it("original situationGraph is preserved through checkpoint selection", async function () {
const result0 = await runWithCheckpoint(0);
const result3 = await runWithCheckpoint(3);
// Both checkpoints use the same situationGraph (11 nodes, 10 edges).
expect(result0.fixtureNodes).toBe(11);
expect(result0.fixtureEdges).toBe(10);
expect(result3.fixtureNodes).toBe(11);
expect(result3.fixtureEdges).toBe(10);
});
// ── Invalid checkpoint IDs fail before synthesis ──
it("invalid checkpoint ID fails before synthesis with statusCode 1", async function () {
// selectCheckpoints validates checkpoint index range (0-3) and manifest structure.
expect(() => selectCheckpoints(5, require("path").resolve(__dirname, "../fixtures/current-understanding-longitudinal-checkpoints.json"))).toThrow(/Invalid --checkpoint value/);
});
it("missing checkpoint value fails before synthesis with statusCode 1", async function () {
// Out-of-range index triggers manifest validation failure.
expect(() => selectCheckpoints(99, require("path").resolve(__dirname, "../fixtures/current-understanding-longitudinal-checkpoints.json"))).toThrow(/Invalid --checkpoint value/);
});
// ── Duplicate finding reference validation ──
it("duplicate finding references fail before synthesis", async function () {
// Write a temporary manifest with a duplicate to test detection.
const tmpDir = require("os").tmpdir();
const tmpManifestPath = require("path").join(tmpDir, "test-duplicate-checkpoints.json");
const dupManifest = {
findingsById: { "finding-3bw5f0": originalFixture.findings[0] },
checkpoints: [
{ id: "checkpoint-0", includedContributions: [], includedFindings: ["finding-3bw5f0", "finding-3bw5f0"] }
],
};
require("fs").writeFileSync(tmpManifestPath, JSON.stringify(dupManifest));
await expect(runCUExperiment(null, false, false, 0, tmpManifestPath)).rejects.toThrow(/Duplicate finding ID/);
require("fs").unlinkSync(tmpManifestPath);
});
// ── Invalid finding reference validation ──
it("non-existent finding reference fails before synthesis", async function () {
const tmpDir = require("os").tmpdir();
const tmpManifestPath = require("path").join(tmpDir, "test-invalid-checkpoints.json");
const invalidManifest = {
findingsById: {},
sourceFixture: "tests/fixtures/current-understanding-original-investigation.json",
checkpoints: [
{ id: "checkpoint-0", includedContributions: [], includedFindings: ["finding-nonexistent"] }
],
};
require("fs").writeFileSync(tmpManifestPath, JSON.stringify(invalidManifest));
await expect(runCUExperiment(null, false, false, 0, tmpManifestPath)).rejects.toThrow(/not found in manifest findingsById/);
require("fs").unlinkSync(tmpManifestPath);
});
// ── Existing --file behaviour unchanged ──
it("existing --file behaviour remains unchanged", async function () {
const fixturePath = require("path").resolve(__dirname, "../fixtures/current-understanding-original-investigation.json");
const result = await runCUExperiment(fixturePath, false, false);
expect(result.success).toBe(true);
expect(result.mode).toBe("deterministic-no-network");
expect(result.findingsCount).toBe(9);
expect(result.fixtureNodes).toBe(11);
expect(result.fixtureEdges).toBe(10);
expect(result.synthesisInvocations).toBe(1);
expect(result.rule2Present).toBe(true);
});
// ── --include-output compatible with checkpoint selection ──
it("--include-output remains compatible with checkpoint selection", async function () {
const manifestPath = require("path").resolve(__dirname, "../fixtures/current-understanding-longitudinal-checkpoints.json");
// Test that runCUExperiment with checkpointManifestPath enters checkpoint mode correctly
// and that includeOutput adds the currentUnderstanding field.
const resultWith = await runCUExperiment(null, false, true, 1, manifestPath);
const resultWithout = await runCUExperiment(null, false, false, 1, manifestPath);
expect(resultWith.success).toBe(true);
expect(resultWithout.success).toBe(true);
expect(resultWith.findingsCount).toBe(2);
expect(resultWithout.findingsCount).toBe(2);
expect(resultWith.synthesisInvocations).toBe(1);
expect(resultWithout.synthesisInvocations).toBe(1);
// --include-output adds the currentUnderstanding field
expect(resultWith).toHaveProperty("currentUnderstanding");
expect(typeof resultWith.currentUnderstanding).toBe("string");
});
// ── Checkpoint selection does not mutate source fixtures ──
it("checkpoint selection does not mutate the original investigation fixture", async function () {
const originalFixturePath = require("path").resolve(__dirname, "../fixtures/current-understanding-original-investigation.json");
const fsModule = require("fs");
const contentBefore = fsModule.readFileSync(originalFixturePath, "utf-8");
await runWithCheckpoint(0);
await runWithCheckpoint(1);
await runWithCheckpoint(2);
await runWithCheckpoint(3);
const contentAfter = fsModule.readFileSync(originalFixturePath, "utf-8");
expect(contentBefore).toBe(contentAfter);
});
it("checkpoint selection does not mutate the longitudinal manifest", async function () {
const manifestPath = require("path").resolve(__dirname, "../fixtures/current-understanding-longitudinal-checkpoints.json");
const fsModule = require("fs");
const contentBefore = fsModule.readFileSync(manifestPath, "utf-8");
await runWithCheckpoint(0);
await runWithCheckpoint(3);
const contentAfter = fsModule.readFileSync(manifestPath, "utf-8");
expect(contentBefore).toBe(contentAfter);
});
});
// ── --include-output flag verification ──
describe("CU synthesis experiment helper — --include-output flag", function () {
it("without --include-output the existing output contract is unchanged (no currentUnderstanding field)", async function () {
const fixturePath = require("path").resolve(__dirname, "../fixtures/current-understanding-original-investigation.json");
const result = await runCUExperiment(fixturePath, false, false);
expect(result.success).toBe(true);
expect(result.mode).toBe("deterministic-no-network");
expect(typeof result.currentUnderstandingLength).toBe("number");
expect(result.currentUnderstandingLength).toBeGreaterThan(0);
expect(result.synthesisInvocations).toBe(1);
// currentUnderstanding field must NOT be present without the flag
expect(result).not.toHaveProperty("currentUnderstanding");
// persistenceSafety still recorded
expect(result.persistenceSafety.noMutation).toBe(true);
});
it("with --include-output the exact currentUnderstanding string is present in structured result", async function () {
const fixturePath = require("path").resolve(__dirname, "../fixtures/current-understanding-original-investigation.json");
const result = await runCUExperiment(fixturePath, false, true);
expect(result.success).toBe(true);
expect(result.mode).toBe("deterministic-no-network");
expect(typeof result.currentUnderstandingLength).toBe("number");
expect(result.currentUnderstandingLength).toBeGreaterThan(0);
expect(result.synthesisInvocations).toBe(1);
expect(result).toHaveProperty("currentUnderstanding");
expect(typeof result.currentUnderstanding).toBe("string");
expect(result.currentUnderstanding.length).toBe(result.currentUnderstandingLength);
});
it("the currentUnderstanding string captured with --include-output matches the length field", async function () {
const fixturePath = require("path").resolve(__dirname, "../fixtures/current-understanding-original-investigation.json");
const result = await runCUExperiment(fixturePath, false, true);
expect(result.currentUnderstanding.length).toBe(result.currentUnderstandingLength);
});
it("Unicode, punctuation and multiline text survive unchanged with --include-output", async function () {
// Build a provider that returns a rich multi-line Unicode payload
const unicodeCU = [
"Line one: café résumé naïve ñ üöä 日本語",
"",
"Line two: \"quoted\" & 'apostrophe' <html> tag characters",
'',
'Line three: special quotes — —–“” and bullet • items',
"\n\tAnd a tabbed-indent line with emoji 🎯✅❌",
].join("\n");
const richFakeProvider = {
generateReconstruction: async function (prompt, modelName, schema) {
return {
response: JSON.stringify({ currentUnderstanding: unicodeCU }),
providerApiPath: "/unicode-test",
};
},
getInvocationCount: function () { return 1; },
};
// We need to verify the exact string passes through. Create a fixture and mock at a lower level.
const graph = deepClone(situationGraph);
const fixings = deepClone(findings);
const result = await synthesizeCurrentUnderstanding(
{ situationGraph: graph, findings: fixings },
{ provider: richFakeProvider }
);
expect(result.currentUnderstanding).toBe(unicodeCU);
});
it("output inclusion does not cause another synthesis invocation", async function () {
const fixturePath = require("path").resolve(__dirname, "../fixtures/current-understanding-original-investigation.json");
// Use includeOutput with deterministic mode — invocation count must remain 1
const resultWith = await runCUExperiment(fixturePath, false, true);
expect(resultWith.synthesisInvocations).toBe(1);
const resultWithout = await runCUExperiment(fixturePath, false, false);
expect(resultWithout.synthesisInvocations).toBe(1);
});
it("deterministic mode wiring remains unchanged with --include-output", async function () {
const fixturePath = require("path").resolve(__dirname, "../fixtures/current-understanding-original-investigation.json");
const result = await runCUExperiment(fixturePath, false, true);
expect(result.mode).toBe("deterministic-no-network");
expect(result.persistenceSafety.noMutation).toBe(true);
expect(result.rule2Present).toBe(true);
});
it("no persistence operation occurs with --include-output", async function () {
const fixturePath = require("path").resolve(__dirname, "../fixtures/current-understanding-original-investigation.json");
const result = await runCUExperiment(fixturePath, false, true);
expect(result.persistenceSafety.noMutation).toBe(true);
expect(result.persistenceSafety.fixtureUnchanged).toBe(true);
});
it("the input fixture remains unchanged after --include-output invocation", async function () {
const fixturePath = require("path").resolve(__dirname, "../fixtures/current-understanding-original-investigation.json");
const fsModule = require("fs");
const originalContent = JSON.parse(fsModule.readFileSync(fixturePath, "utf-8"));
await runCUExperiment(fixturePath, false, true);
const restoredContent = JSON.parse(fsModule.readFileSync(fixturePath, "utf-8"));
expect(JSON.stringify(originalContent)).toBe(JSON.stringify(restoredContent));
});
it("--include-output with deterministic mode returns same CU text as synthesis function directly", async function () {
const graphBefore = deepClone(situationGraph);
const fixBefore = deepClone(findings);
// Get the CU text from synthesizeCurrentUnderstanding directly
const fakeDirect = makeFakeProvider({ currentUnderstanding: "direct synthetic output" });
const directResult = await synthesizeCurrentUnderstanding(
{ situationGraph: graphBefore, findings: fixBefore },
{ provider: fakeDirect }
);
expect(directResult.currentUnderstanding).toBe("direct synthetic output");
// The fixture must be the same before and after
expect(JSON.stringify(situationGraph)).toBe(JSON.stringify(graphBefore));
expect(JSON.stringify(findings)).toBe(JSON.stringify(fixBefore));
});
it("live-mode wiring verified — --include-output with mock live provider returns currentUnderstanding", async function () {
const graphBefore = deepClone(situationGraph);
const fixBefore = deepClone(findings);
const liveMockProvider = makeLiveModeMockProvider();
const result = await synthesizeCurrentUnderstanding(
{ situationGraph: situationGraph, findings: findings },
{ provider: liveMockProvider }
);
expect(result).toHaveProperty("currentUnderstanding");
expect(typeof result.currentUnderstanding).toBe("string");
expect(result.currentUnderstanding).toBe("Live-mode CU via injected deterministic provider");
// Immutability post-run
expect(JSON.stringify(situationGraph)).toBe(JSON.stringify(graphBefore));
expect(JSON.stringify(findings)).toBe(JSON.stringify(fixBefore));
});
});
// ── Longitudinal checkpoint deterministic verification ──
import longitudinalManifest from "@/tests/fixtures/current-understanding-longitudinal-checkpoints.json" assert { type: "json" };
import originalFixture from "@/tests/fixtures/current-understanding-original-investigation.json" assert { type: "json" };
describe("Longitudinal checkpoints — manifest integrity", function () {
it("manifest has exactly 4 checkpoints", async function () {
expect(longitudinalManifest.checkpoints.length).toBe(4);
});
it("checkpoint counts are 0, 2, 7, 9", async function () {
const expectedCounts = [0, 2, 7, 9];
longitudinalManifest.checkpoints.forEach((cp, i) => {
expect(cp.includedFindings.length).toBe(expectedCounts[i]);
});
});
it("checkpoint-0 has no included findings", async function () {
expect(longitudinalManifest.checkpoints[0].includedFindings).toEqual([]);
});
it("checkpoint-1 includes exactly contrib-0001 findings", async function () {
const expected = ["finding-3bw5f0", "finding-i5zdcq"];
expect(longitudinalManifest.checkpoints[1].includedFindings).toEqual(expected);
});
it("checkpoint-2 is cumulative of checkpoint-1 plus contrib-0002 findings", async function () {
const cp2 = new Set(longitudinalManifest.checkpoints[2].includedFindings);
const cp1 = longitudinalManifest.checkpoints[1].includedFindings;
cp1.forEach(f => expect(cp2.has(f)).toBe(true)); // all checkpoint-1 findings present in cp2
expect(cp2.size).toBe(7);
});
it("checkpoint-3 includes all 9 original findings", async function () {
const expected = [
"finding-3bw5f0", "finding-i5zdcq",
"finding-hciikc", "finding-ybfsm3", "finding-k45vyt",
"finding-vhftyr", "finding-9bgkmw",
"finding-23192q", "finding-kkh39k"
];
expect(longitudinalManifest.checkpoints[3].includedFindings).toEqual(expected);
});
it("all checkpoint findings are verbatim from the original fixture", async function () {
const originalMap = {};
originalFixture.findings.forEach(f => { originalMap[f.id] = f; });
longitudinalManifest.checkpoints.forEach(cp => {
cp.includedFindings.forEach(fid => {
expect(originalMap[fid]).toBeDefined();
const manifestFinding = longitudinalManifest.findingsById[fid];
expect(manifestFinding).toEqual(originalMap[fid]);
});
});
});
it("contribution ordering matches recorded timestamps", async function () {
const contribs = longitudinalManifest.contributions;
for (let i = 1; i < contribs.length; i++) {
expect(new Date(contribs[i].timestamp).getTime()).toBeGreaterThan(
new Date(contribs[i - 1].timestamp).getTime()
);
}
});
it("manifest ordering matches fixture finding order", async function () {
const originalIds = originalFixture.findings.map(f => f.id);
const manifestIds = Object.keys(longitudinalManifest.findingsById);
expect(manifestIds).toEqual(originalIds);
});
it("checkpoint-3 is cumulative of all checkpoints before it", async function () {
for (let i = 0; i < 3; i++) {
const cpSet = new Set(longitudinalManifest.checkpoints[i].includedFindings);
const cp3 = longitudinalManifest.checkpoints[3].includedFindings;
cpSet.forEach(f => expect(cp3.includes(f)).toBe(true));
}
});
it("no findings appear before their contribution boundary", async function () {
const contribOrder = {};
longitudinalManifest.contributions.forEach(c => { contribOrder[c.id] = c.ordering; });
Object.entries(longitudinalManifest.findingsById).forEach(([fid, f]) => {
const cpIndex = longitudinalManifest.checkpoints.findIndex(cp =>
cp.includedFindings.includes(fid)
);
// Find which checkpoint first includes this finding
expect(cpIndex).toBeGreaterThanOrEqual(contribOrder[f.contributionId] - 1);
});
});
it("original fixture is unchanged (spot check)", async function () {
expect(originalFixture.findings.length).toBe(9);
expect(longitudinalManifest.checkpoints[3].includedFindings.length).toBe(9);
});
it("production synthesis contract accepts checkpoint inputs", async function () {
// Verify that the filtered findings from each checkpoint can reach synthesizeCurrentUnderstanding
const fake = makeFakeProvider();
for (const cp of longitudinalManifest.checkpoints) {
const graphBefore = deepClone(situationGraph);
const cpFindings = cp.includedFindings.map(fid => originalFixture.findings.find(f => f.id === fid));
await expect(
synthesizeCurrentUnderstanding(
{ situationGraph: situationGraph, findings: cpFindings },
{ provider: fake }
)
).resolves.toBeDefined();
// Immutability check per checkpoint
expect(JSON.stringify(situationGraph)).toBe(JSON.stringify(graphBefore));
}
});
it("manifest references the source fixture path", async function () {
expect(longitudinalManifest.sourceFixture).toBe(
"tests/fixtures/current-understanding-original-investigation.json"
);
});
it("manifest does not fabricate historical graph state", async function () {
// Manifest contains findings data only, not resolvedNodeIds, activeUnknownNodeId, or reasoningState
expect(longitudinalManifest.checkpoints).not.toHaveProperty([0, "resolvedNodeIds"]);
expect(longitudinalManifest.contributions).not.toHaveProperty([0, "reasoningState"]);
});
});
@@ -635,4 +635,109 @@ describe("synthesizeCurrentUnderstanding — full seam", () => {
)
).rejects.toThrow(/Synthesis validation failed/);
});
// ── Batch 3 Iteration 1: CU synthesis observability (sanitized diagnostic) ──
it("validation failure exposes sanitizedDiagnostic with safe shape fields — no content leaked", async () => {
const fake = makeFakeProvider(async () => JSON.stringify({ wrongField: "value" }));
try {
await synthesizeCurrentUnderstanding(
{ situationGraph: fullScenario, findings: [] },
{ provider: fake }
);
} catch (err) {
expect(err.message).toMatch(/Synthesis validation failed/);
expect(err.statusCode).toBe(502);
expect(err.sanitizedDiagnostic).toBeDefined();
expect(err.sanitizedDiagnostic).toBeInstanceOf(Object);
const d = err.sanitizedDiagnostic;
// T1: providerApiPath exposed as string or null (safe)
expect(d.providerApiPath).toBeDefined();
expect(typeof d.providerApiPath).toMatch(/string|number|boolean|undefined|null/);
// T2: responseType is a safe primitive exposing typeof the response
expect(typeof d.responseType).toBe("string");
// T3: responseIsNull boolean (safe)
expect(typeof d.responseIsNull).toBe("boolean");
// T4: hasCurrentUnderstanding boolean — proves structural detection works
expect(typeof d.hasCurrentUnderstanding).toBe("boolean");
// T5: currentUnderstandingType is null when field absent (safe)
expect(d.currentUnderstandingType).toBeNull();
// T6: currentUnderstandingLength is null when field absent (safe)
expect(d.currentUnderstandingLength).toBeNull();
// T7: responseKeys reveals shape but never CU text — wrongField is safe key
expect(Array.isArray(d.responseKeys)).toBe(true);
expect(d.responseKeys).toContain("wrongField");
}
});
it("sanitizedDiagnostic does NOT include actual CU narrative content", async () => {
const fake = makeFakeProvider(async () => JSON.stringify({ currentUnderstanding: "Should not appear in diagnostic" }));
try {
await synthesizeCurrentUnderstanding(
{ situationGraph: fullScenario, findings: [] },
{ provider: fake }
);
} catch (err) {
expect(err.sanitizedDiagnostic).toBeDefined();
const d = err.sanitizedDiagnostic;
// The diagnostic should expose length but not content
expect(typeof d.currentUnderstandingLength).toBe("number");
// Length 43 for "Should not appear in diagnostic" — if it were wrongField the field wouldn't exist so hasCU would be false and length would be null
expect(d.hasCurrentUnderstanding).toBe(true);
// Crucially: actual content string must NOT appear anywhere in diagnostic
const diagStr = JSON.stringify(d);
expect(diagStr).not.toContain("Should not appear");
}
});
it("provider returns response object (already parsed) with valid currentUnderstanding — no synthetic test", async () => {
const fake = makeFakeProvider(); // default: returns JSON stringified valid output
const result = await synthesizeCurrentUnderstanding(
{ situationGraph: fullScenario, findings: [workingFinding()] },
{ provider: fake }
);
expect(result).toEqual({ currentUnderstanding: "Synthesized output" });
});
it("validateSynthesisResponse with response that is a plain object (not string) — still valid", async () => {
const fake = makeFakeProvider(async () => ({ currentUnderstanding: "from-object" }));
const result = await synthesizeCurrentUnderstanding(
{ situationGraph: fullScenario, findings: [] },
{ provider: fake }
);
expect(result).toEqual({ currentUnderstanding: "from-object" });
});
it("responseKeys exposed in diagnostic includes top-level keys of any response object", async () => {
const fake = makeFakeProvider(async () => JSON.stringify({ foo: 1, bar: "baz", currentUnderstanding: "x" }));
try {
await synthesizeCurrentUnderstanding(
{ situationGraph: fullScenario, findings: [] },
{ provider: fake }
);
} catch (err) {
expect(err.sanitizedDiagnostic.responseKeys).toEqual(["bar", "currentUnderstanding", "foo"]);
}
});
it("validation success returns normal result — no diagnostic attached", async () => {
const fake = makeFakeProvider();
try {
await synthesizeCurrentUnderstanding(
{ situationGraph: fullScenario, findings: [] },
{ provider: fake }
);
} catch (err) {
// Should not reach here — success path should not throw
expect(err).toBeUndefined();
}
});
});
+139
View File
@@ -0,0 +1,139 @@
import { describe, it, expect } from "vitest";
import {
focusedDeconstruct,
buildFocusedDeconstructPrompt,
validateFocusedDeconstructSchema,
focusedDeconstructJsonSchema,
} from "@/lib/graph/focused-investigation.js";
const FAKE_TARGET_NODE_ID = "test-node-001";
const FAKE_LABEL = "Market demand signal";
const FAKE_DESCRIPTION = "Evidence that customers want this.";
const FAKE_CENTRAL = "Should we launch the product now?";
const FAKE_QUESTION = "What evidence would clarify market demand?";
const FAKE_ANSWER = "Some evidence suggests demand.";
function makeValidResponse() {
return {
targetNodeId: "model-returned-001",
observations: ["Observation A"],
uncertainties: ["Uncertainty B"],
assumptions: ["Assumption C"],
relationships: [{ from: "A", to: "B", type: "supports" }],
possibleFollowUpQuestions: ["What next?"],
};
}
function makeInvalidResponse() {
return { targetNodeId: "model-returned-001", observations: ["Obs"] };
}
describe("focusedDeconstruct — success", () => {
it("calls provider.generateReconstruction exactly once and returns success", async () => {
let callCount = 0;
const fake = { async generateReconstruction() { callCount++; return { response: makeValidResponse() }; } };
const result = await focusedDeconstruct({ provider: fake, modelName: "test-model", targetNodeId: FAKE_TARGET_NODE_ID, targetLabel: FAKE_LABEL, targetDescription: FAKE_DESCRIPTION, centralStatement: FAKE_CENTRAL, question: FAKE_QUESTION, answer: FAKE_ANSWER });
expect(callCount).toBe(1);
expect(result.success).toBe(true);
});
it("preserves observations from provider response", async () => {
const fake = { async generateReconstruction() { return { response: makeValidResponse() }; } };
const result = await focusedDeconstruct({ provider: fake, modelName: "test-model", targetNodeId: FAKE_TARGET_NODE_ID, targetLabel: FAKE_LABEL, targetDescription: FAKE_DESCRIPTION, centralStatement: FAKE_CENTRAL, question: FAKE_QUESTION, answer: FAKE_ANSWER });
expect(Array.isArray(result.observations)).toBe(true);
expect(result.observations.length).toBeGreaterThan(0);
});
it("preserves uncertainties from provider response", async () => {
const fake = { async generateReconstruction() { return { response: makeValidResponse() }; } };
const result = await focusedDeconstruct({ provider: fake, modelName: "test-model", targetNodeId: FAKE_TARGET_NODE_ID, targetLabel: FAKE_LABEL, targetDescription: FAKE_DESCRIPTION, centralStatement: FAKE_CENTRAL, question: FAKE_QUESTION, answer: FAKE_ANSWER });
expect(Array.isArray(result.uncertainties)).toBe(true);
expect(result.uncertainties.length).toBeGreaterThan(0);
});
it("preserves assumptions from provider response", async () => {
const fake = { async generateReconstruction() { return { response: makeValidResponse() }; } };
const result = await focusedDeconstruct({ provider: fake, modelName: "test-model", targetNodeId: FAKE_TARGET_NODE_ID, targetLabel: FAKE_LABEL, targetDescription: FAKE_DESCRIPTION, centralStatement: FAKE_CENTRAL, question: FAKE_QUESTION, answer: FAKE_ANSWER });
expect(Array.isArray(result.assumptions)).toBe(true);
expect(result.assumptions.length).toBeGreaterThan(0);
});
it("preserves relationships from provider response", async () => {
const fake = { async generateReconstruction() { return { response: makeValidResponse() }; } };
const result = await focusedDeconstruct({ provider: fake, modelName: "test-model", targetNodeId: FAKE_TARGET_NODE_ID, targetLabel: FAKE_LABEL, targetDescription: FAKE_DESCRIPTION, centralStatement: FAKE_CENTRAL, question: FAKE_QUESTION, answer: FAKE_ANSWER });
expect(Array.isArray(result.relationships)).toBe(true);
expect(result.relationships.length).toBeGreaterThan(0);
});
it("preserves possibleFollowUpQuestions from provider response", async () => {
const fake = { async generateReconstruction() { return { response: makeValidResponse() }; } };
const result = await focusedDeconstruct({ provider: fake, modelName: "test-model", targetNodeId: FAKE_TARGET_NODE_ID, targetLabel: FAKE_LABEL, targetDescription: FAKE_DESCRIPTION, centralStatement: FAKE_CENTRAL, question: FAKE_QUESTION, answer: FAKE_ANSWER });
expect(Array.isArray(result.possibleFollowUpQuestions)).toBe(true);
expect(result.possibleFollowUpQuestions.length).toBeGreaterThan(0);
});
it("preserves caller target identity (result.targetNodeId === request targetNodeId)", async () => {
const fake = { async generateReconstruction() { return { response: makeValidResponse() }; } };
const result = await focusedDeconstruct({ provider: fake, modelName: "test-model", targetNodeId: FAKE_TARGET_NODE_ID, targetLabel: FAKE_LABEL, targetDescription: FAKE_DESCRIPTION, centralStatement: FAKE_CENTRAL, question: FAKE_QUESTION, answer: FAKE_ANSWER });
expect(result.targetNodeId).toBe(FAKE_TARGET_NODE_ID);
});
it("passes correct prompt to provider", async () => {
let capturedPrompt = null;
const fake = { async generateReconstruction(prompt) { capturedPrompt = prompt; return { response: makeValidResponse() }; } };
await focusedDeconstruct({ provider: fake, modelName: "test-model", targetNodeId: FAKE_TARGET_NODE_ID, targetLabel: FAKE_LABEL, targetDescription: FAKE_DESCRIPTION, centralStatement: FAKE_CENTRAL, question: FAKE_QUESTION, answer: FAKE_ANSWER });
const expected = buildFocusedDeconstructPrompt({ targetLabel: FAKE_LABEL, targetDescription: FAKE_DESCRIPTION, centralStatement: FAKE_CENTRAL, question: FAKE_QUESTION, answer: FAKE_ANSWER });
expect(capturedPrompt).toBe(expected);
});
it("passes modelName unchanged to provider", async () => {
let capturedModel = null;
const fake = { async generateReconstruction(prompt, receivedModel) { capturedModel = receivedModel; return { response: makeValidResponse() }; } };
await focusedDeconstruct({ provider: fake, modelName: "my-custom-model-v2", targetNodeId: FAKE_TARGET_NODE_ID, targetLabel: FAKE_LABEL, targetDescription: FAKE_DESCRIPTION, centralStatement: FAKE_CENTRAL, question: FAKE_QUESTION, answer: FAKE_ANSWER });
expect(capturedModel).toBe("my-custom-model-v2");
});
it("passes focusedDeconstructJsonSchema as the schema argument", async () => {
let capturedSchema = null;
const fake = { async generateReconstruction(prompt, modelName, schema) { capturedSchema = schema; return { response: makeValidResponse() }; } };
await focusedDeconstruct({ provider: fake, modelName: "test-model", targetNodeId: FAKE_TARGET_NODE_ID, targetLabel: FAKE_LABEL, targetDescription: FAKE_DESCRIPTION, centralStatement: FAKE_CENTRAL, question: FAKE_QUESTION, answer: FAKE_ANSWER });
expect(capturedSchema).toBe(focusedDeconstructJsonSchema);
});
})
describe("focusedDeconstruct — validation failure", () => {
it("returns failure result when provider response fails canonical validation", async () => {
const fake = { async generateReconstruction() { return { response: makeInvalidResponse() }; } };
const result = await focusedDeconstruct({ provider: fake, modelName: "test-model", targetNodeId: FAKE_TARGET_NODE_ID, targetLabel: FAKE_LABEL, targetDescription: FAKE_DESCRIPTION, centralStatement: FAKE_CENTRAL, question: FAKE_QUESTION, answer: FAKE_ANSWER });
expect(result.success).toBe(false);
});
it("validation errors remain available on failure result", async () => {
const fake = { async generateReconstruction() { return { response: makeInvalidResponse() }; } };
const result = await focusedDeconstruct({ provider: fake, modelName: "test-model", targetNodeId: FAKE_TARGET_NODE_ID, targetLabel: FAKE_LABEL, targetDescription: FAKE_DESCRIPTION, centralStatement: FAKE_CENTRAL, question: FAKE_QUESTION, answer: FAKE_ANSWER });
expect(Array.isArray(result.errors)).toBe(true);
expect(result.errors.length).toBeGreaterThan(0);
});
it("no second provider call occurs on validation failure", async () => {
let callCount = 0;
const fake = { async generateReconstruction() { callCount++; return { response: makeInvalidResponse() }; } };
await focusedDeconstruct({ provider: fake, modelName: "test-model", targetNodeId: FAKE_TARGET_NODE_ID, targetLabel: FAKE_LABEL, targetDescription: FAKE_DESCRIPTION, centralStatement: FAKE_CENTRAL, question: FAKE_QUESTION, answer: FAKE_ANSWER });
expect(callCount).toBe(1);
});
})
describe("focusedDeconstruct — provider failure", () => {
it("propagates error with original identity/metadata intact", async () => {
const fake = {
async generateReconstruction() {
const err = new Error("Connection refused to ollama endpoint");
err.name = "ProviderError";
err.providerApiPath = "/api/v1/completions";
err.code = "PROVIDER_UNAVAILABLE";
throw err;
},
};
await expect(focusedDeconstruct({ provider: fake, modelName: "test-model", targetNodeId: FAKE_TARGET_NODE_ID, targetLabel: FAKE_LABEL, targetDescription: FAKE_DESCRIPTION, centralStatement: FAKE_CENTRAL, question: FAKE_QUESTION, answer: FAKE_ANSWER })).rejects.toThrow("Connection refused to ollama endpoint");
});
})
@@ -65,6 +65,40 @@ describe("formulateQuestionForTarget — explicit node selection", () => {
expect(JSON.stringify(graph)).toBe(snapshot);
});
it("forwards an ephemeral incompatible compatibility verdict without changing the selected target or graph", () => {
const unknown = makeNode({
id: "historical-target",
label: "Earlier comparable state",
description: "Whether the prior staffing level was sufficient for expansion",
kind: "unknown",
status: "unknown",
});
const graph = makeGraph({
centralStatement:
"Compare the prior staffing baseline with the level needed before the expansion.",
currentSummary: "initial summary",
nodes: [unknown],
edges: [],
activeUnknownNodeId: unknown.id,
resolvedNodeIds: [],
});
const snapshot = JSON.stringify(graph);
const result = formulateQuestionForTarget({
situationGraph: graph,
targetNodeId: unknown.id,
strategyCompatibilityVerdict: {
strategyKey: "baseline_reconstruction",
relevance: "incompatible",
},
});
expect(result.success).toBe(true);
expect(result.targetNodeId).toBe(unknown.id);
expect(result.strategy).toBeNull();
expect(JSON.stringify(graph)).toBe(snapshot);
});
it("does not invoke global selection (selectActiveUnknownCandidate / determineGraphBackedQuestion)", () => {
const graph = makeTestGraph();
const result = formulateQuestionForTarget({ situationGraph: graph, targetNodeId: "a" });
@@ -0,0 +1,186 @@
import { describe, expect, it } from "vitest";
import { deriveInvestigationLearning } from "@/lib/graph/episode-preparation.js";
// --- Fixtures ---
function makeGraph(resolvedIds) {
return {
centralStatement: "Test scenario",
nodes: [
{ id: "node-A", label: "How do we improve customer retention?", kind: "unknown", status: "unknown" },
{ id: "node-B", label: "Which market segment to target?", kind: "unknown", status: "unknown" },
{ id: "node-C", label: "What is the budget constraint?", kind: "unknown", status: "unknown" }
],
edges: [],
activeUnknownNodeId: null,
resolvedNodeIds: resolvedIds ?? [],
currentSummary: "Baseline",
};
}
// === Case 1: correct provenance ===
describe("deriveInvestigationLearning - correct provenance", () => {
it("returns Finding A under Question A via Contribution provenance chain", () => {
const graph = makeGraph(["node-A"]);
const contributions = [
{ id: "contrib-A1", targetNodeId: "node-A", sequence: 1, question: "Q1?", answer: "A1" },
{ id: "contrib-A2", targetNodeId: "node-A", sequence: 2, question: "Q2?", answer: "A2" }
];
const findings = [
{ id: "find-A1", proposition: "Proposition A from finding 1", status: "provisional", userDisposition: null, originatingTargetNodeId: "node-A", contributionId: "contrib-A1", sourceObservation: "Source A1" },
{ id: "find-A2", proposition: "Proposition A from finding 2", status: "provisional", userDisposition: "agree", originatingTargetNodeId: "node-A", contributionId: "contrib-A2", sourceObservation: "Source A2" }
];
const result = deriveInvestigationLearning({ situationGraph: graph, contributions, findings });
expect(result).toHaveLength(1);
expect(result[0].nodeId).toBe("node-A");
expect(result[0].questionText).toBe("How do we improve customer retention?");
expect(result[0].propositions).toHaveLength(2);
expect(result[0].propositions[0].findingId).toBe("find-A1");
expect(result[0].propositions[0].proposition).toBe("Proposition A from finding 1");
expect(result[0].propositions[1].findingId).toBe("find-A2");
expect(result[0].propositions[1].userDisposition).toBe("agree");
});
});
// === Case 2: no cross-question leakage ===
describe("deriveInvestigationLearning - cross-question isolation", () => {
it("Finding from Contribution B does NOT appear under Question A", () => {
const graph = makeGraph(["node-A", "node-B"]);
const contributions = [
{ id: "contrib-A1", targetNodeId: "node-A", sequence: 1, question: "Q?", answer: "A" },
{ id: "contrib-B1", targetNodeId: "node-B", sequence: 1, question: "Q?", answer: "A" }
];
const findings = [
{ id: "find-A", proposition: "Prop A", status: "provisional", userDisposition: null, originatingTargetNodeId: "node-A", contributionId: "contrib-A1", sourceObservation: "S1" },
{ id: "find-B", proposition: "Prop B (should not leak)", status: "provisional", userDisposition: null, originatingTargetNodeId: "node-B", contributionId: "contrib-B1", sourceObservation: "S2" }
];
const result = deriveInvestigationLearning({ situationGraph: graph, contributions, findings });
const nodeAResults = result.find(function(r) { return r.nodeId === "node-A"; });
expect(nodeAResults).not.toBeUndefined();
expect(nodeAResults.propositions.every(function(p) { return p.findingId !== "find-B"; })).toBe(true);
const nodeBResults = result.find(function(r) { return r.nodeId === "node-B"; });
expect(nodeBResults).not.toBeUndefined();
expect(nodeBResults.propositions.some(function(p) { return p.findingId === "find-B"; })).toBe(true);
});
});
// === Case 3: eligibility rules ===
describe("deriveInvestigationLearning - disposition eligibility", () => {
it("null and agree are eligible; not_quite and not_relevant are excluded", () => {
const graph = makeGraph(["node-A"]);
const contributions = [
{ id: "contrib-null1", targetNodeId: "node-A", sequence: 1, question: "Q?", answer: "A" },
{ id: "contrib-agree2", targetNodeId: "node-A", sequence: 2, question: "Q?", answer: "A" },
{ id: "contrib-nq3", targetNodeId: "node-A", sequence: 3, question: "Q?", answer: "A" },
{ id: "contrib-nr4", targetNodeId: "node-A", sequence: 4, question: "Q?", answer: "A" }
];
const findings = [
{ id: "find-null", proposition: "Null prop", status: "provisional", userDisposition: null, originatingTargetNodeId: "node-A", contributionId: "contrib-null1", sourceObservation: "S" },
{ id: "find-agree", proposition: "Agree prop", status: "provisional", userDisposition: "agree", originatingTargetNodeId: "node-A", contributionId: "contrib-agree2", sourceObservation: "S" },
{ id: "find-notquite", proposition: "Not quite prop", status: "provisional", userDisposition: "not_quite", originatingTargetNodeId: "node-A", contributionId: "contrib-nq3", sourceObservation: "S" },
{ id: "find-notrel", proposition: "Not relevant prop", status: "provisional", userDisposition: "not_relevant", originatingTargetNodeId: "node-A", contributionId: "contrib-nr4", sourceObservation: "S" }
];
const result = deriveInvestigationLearning({ situationGraph: graph, contributions, findings });
expect(result).toHaveLength(1);
expect(result[0].propositions).toHaveLength(2);
const ids = result[0].propositions.map(function(p) { return p.findingId; });
expect(ids).toContain("find-null");
expect(ids).toContain("find-agree");
expect(ids).not.toContain("find-notquite");
expect(ids).not.toContain("find-notrel");
});
});
// === Case 4: unresolved question excluded ===
describe("deriveInvestigationLearning - unresolved exclusion", () => {
it("question not in resolvedNodeIds is omitted even if Contributions/Findings exist for it", () => {
const graph = makeGraph([]);
const contributions = [
{ id: "contrib-X1", targetNodeId: "node-B", sequence: 1, question: "Q?", answer: "A" }
];
const findings = [
{ id: "find-X", proposition: "Prop X for unresolved", status: "provisional", userDisposition: null, originatingTargetNodeId: "node-B", contributionId: "contrib-X1", sourceObservation: "S" }
];
const result = deriveInvestigationLearning({ situationGraph: graph, contributions, findings });
expect(result).toHaveLength(0);
});
});
// === Case 5: no eligible material ===
describe("deriveInvestigationLearning - empty propositions omitted", () => {
it("investigated question with zero eligible Findings is omitted rather than placeholder", () => {
const graph = makeGraph(["node-A"]);
const contributions = [
{ id: "contrib-N1", targetNodeId: "node-A", sequence: 1, question: "Q?", answer: "A" }
];
const findings = [
{ id: "find-nr", proposition: "Not relevant prop", status: "provisional", userDisposition: "not_relevant", originatingTargetNodeId: "node-A", contributionId: "contrib-N1", sourceObservation: "S" }
];
const result = deriveInvestigationLearning({ situationGraph: graph, contributions, findings });
expect(result).toHaveLength(0);
});
});
// === Case 6: question text preserved ===
describe("deriveInvestigationLearning - question text preservation", () => {
it("projection preserves canonical question text without model synthesis or rewriting", () => {
const graph = makeGraph(["node-A"]);
const contributions = [
{ id: "contrib-T1", targetNodeId: "node-A", sequence: 1, question: "Q?", answer: "A" }
];
const findings = [
{ id: "find-T", proposition: "Canonical prop text", status: "provisional", userDisposition: null, originatingTargetNodeId: "node-A", contributionId: "contrib-T1", sourceObservation: "S" }
];
const result = deriveInvestigationLearning({ situationGraph: graph, contributions, findings });
expect(result).toHaveLength(1);
expect(result[0].questionText).toBe("How do we improve customer retention?");
expect(result[0].propositions[0].proposition).toBe("Canonical prop text");
});
});
// === Case 7: provenance beats originatingTargetNodeId shortcut ===
describe("deriveInvestigationLearning - provenance authority", () => {
it("explicit Contribution provenance overrides misleading originatingTargetNodeId", () => {
const graph = makeGraph(["node-A", "node-B"]);
// Both nodes have Contributions so they are resolved questions
const contributions = [
{ id: "contrib-A1", targetNodeId: "node-A", sequence: 1, question: "Q?", answer: "A" },
{ id: "contrib-B1", targetNodeId: "node-B", sequence: 1, question: "Q?", answer: "A" }
];
// Finding points to contrib-B1 (correct contribution) BUT
// has originatingTargetNodeId pointing at node-A — misleading shortcut
const findings = [
{ id: "find-provenance", proposition: "Must route to B via contributionId not originatingTargetNodeId", status: "provisional", userDisposition: null, originatingTargetNodeId: "node-A", contributionId: "contrib-B1", sourceObservation: "S" },
{ id: "find-a-eligible", proposition: "Node A own finding", status: "provisional", userDisposition: null, originatingTargetNodeId: "node-A", contributionId: "contrib-A1", sourceObservation: "S" }
];
const result = deriveInvestigationLearning({ situationGraph: graph, contributions, findings });
// Both nodes appear as resolved questions with propositions
expect(result).toHaveLength(2);
var bResults = result.find(function(r) { return r.nodeId === "node-B"; });
expect(bResults.propositions.some(function(p) { return p.findingId === "find-provenance"; })).toBe(true);
var aResults = result.find(function(r) { return r.nodeId === "node-A"; });
expect(aResults).not.toBeUndefined();
expect(aResults.propositions.some(function(p) { return p.findingId === "find-provenance"; })).toBe(false);
});
});
@@ -39,10 +39,13 @@ const fakeProvider = (defaultResponse) => ({
generateReconstruction: vi.fn(() => {
return typeof defaultResponse === "function"
? defaultResponse()
: JSON.stringify({
understanding: "Synthesized understanding from evidence.",
plausibleInterpretations: "Remaining plausible explanations.",
});
: {
response: {
understanding: "Synthesized understanding from evidence.",
plausibleInterpretations: "Remaining plausible explanations.",
},
providerApiPath: "/api/chat",
};
}),
});
@@ -232,9 +235,18 @@ describe("investigation overview — output validation contract", () => {
expect(result.valid).toBe(false);
});
it("empty string plausibleInterpretations rejected", () => {
it("empty string plausibleInterpretations accepted (zero interpretations)", () => {
const result = validateOverviewResponse({ understanding: "something", plausibleInterpretations: "" });
expect(result.valid).toBe(false);
expect(result.valid).toBe(true);
expect(result.data.plausibleInterpretations).toBe("");
});
it("valid empty-string plausibleInterpretations with non-empty understanding is canonical zero", () => {
const result = validateOverviewResponse({
understanding: "Supported understanding.",
plausibleInterpretations: "",
});
expect(result.valid).toBe(true);
});
it("malformed JSON rejected", () => {
@@ -242,17 +254,61 @@ describe("investigation overview — output validation contract", () => {
expect(result.valid).toBe(false);
});
it("null input rejected", () => {
expect(validateOverviewResponse(null).valid).toBe(false);
});
it("undefined input rejected", () => {
expect(validateOverviewResponse(undefined).valid).toBe(false);
it("non-empty plausible synthesis remains valid", () => {
const result = validateOverviewResponse({
understanding: "Supported understanding.",
plausibleInterpretations: "Y is still a candidate.",
});
expect(result.valid).toBe(true);
});
it("non-object input rejected", () => {
expect(validateOverviewResponse("just a string").valid).toBe(false);
});
// ── hasPlausibleInterpretations derivation from validated data ──
describe("hasPlausibleInterpretations derivation", () => {
it("zero plausible output derives hasPlausibleInterpretations === false", () => {
const result = validateOverviewResponse({
understanding: "Supported understanding.",
plausibleInterpretations: "",
});
expect(result.valid).toBe(true);
const hasPlausibleInterpretations = result.data.plausibleInterpretations.trim().length > 0;
expect(hasPlausibleInterpretations).toBe(false);
});
it("non-zero plausible output derives hasPlausibleInterpretations === true", () => {
const result = validateOverviewResponse({
understanding: "Supported understanding.",
plausibleInterpretations: "Some interpretation.",
});
expect(result.valid).toBe(true);
const hasPlausibleInterpretations = result.data.plausibleInterpretations.trim().length > 0;
expect(hasPlausibleInterpretations).toBe(true);
});
it("whitespace-only plausibleInterpretations derives false", () => {
const result = validateOverviewResponse({
understanding: "Supported.",
plausibleInterpretations: " ",
});
expect(result.valid).toBe(true);
const hasPlausibleInterpretations = result.data.plausibleInterpretations.trim().length > 0;
expect(hasPlausibleInterpretations).toBe(false);
});
});
// ── SECTION B empty-input prompt contract ──
describe("SECTION B empty-input prompt instruction", () => {
it("prompt explicitly permits and requests empty-string output when no plausible inputs", () => {
const prompt = buildOverviewSynthesisPrompt(graph, eligibleFindings, []);
expect(prompt).toContain('plausibleInterpretations');
expect(prompt).toMatch(/empty string/i);
});
});
});
// ── 6. No decision/recommendation/readiness fields allowed ───────────────
@@ -459,4 +515,61 @@ describe("synthesizeInvestigationOverview — full seam", () => {
expect(eligible.map((f) => f.id)).toEqual(["f1", "f2"]);
expect(eligible).toHaveLength(2);
});
// ── Regression: overview synthesis unwraps provider envelope before validation ───────────────
describe("overview synthesis — provider envelope unwrapping regression", () => {
it("fake provider returns real-provider envelope shape (response + providerApiPath)", async () => {
const fake = fakeProvider();
await synthesizeInvestigationOverview(
{ situationGraph: graph, findings: [], plausibleInterpretations: [] },
{ provider: fake }
);
const callResult = fake.generateReconstruction.mock.calls[0][2];
expect(fake.generateReconstruction).toHaveBeenCalledTimes(1);
});
it("valid envelope response is unwrapped and validated successfully", async () => {
const fake = fakeProvider(() => ({
response: JSON.stringify({
understanding: "Test understanding.",
plausibleInterpretations: "Test interpretations.",
}),
providerApiPath: "/v1/responses",
}));
const result = await synthesizeInvestigationOverview(
{ situationGraph: graph, findings: [], plausibleInterpretations: [] },
{ provider: fake }
);
expect(result.understanding).toBe("Test understanding.");
expect(result.plausibleInterpretations).toBe("Test interpretations.");
});
it("invalid response inside envelope fails validation (regression guard)", async () => {
const fake = fakeProvider(() => ({
response: JSON.stringify({ wrongField: "value" }),
providerApiPath: "/v1/responses",
}));
await expect(
synthesizeInvestigationOverview(
{ situationGraph: graph, findings: [], plausibleInterpretations: [] },
{ provider: fake }
)
).rejects.toThrow(/validation failed/);
});
it("provider returns .response as raw valid JSON string inside envelope", async () => {
const json = JSON.stringify({
understanding: "Raw JSON test.",
plausibleInterpretations: "String payload.",
});
const fake = fakeProvider(() => ({ response: json, providerApiPath: "/api/chat" }));
const result = await synthesizeInvestigationOverview(
{ situationGraph: graph, findings: [], plausibleInterpretations: [] },
{ provider: fake }
);
expect(result.understanding).toBe("Raw JSON test.");
expect(fake.generateReconstruction).toHaveBeenCalledTimes(1);
});
});
});
File diff suppressed because it is too large Load Diff
+597 -69
View File
@@ -18,6 +18,9 @@
const dotenv = require("dotenv");
dotenv.config({ path: ".env.local" });
// Re-export production helper for apparatus use — avoids fabricating Finding propositions.
const { deriveFindingsFromContributions } = require("../../lib/graph/finding-helpers.js");
function assertRequiredEnv(name) {
const value = process.env[name];
if (!value) {
@@ -31,85 +34,78 @@ function assertRequiredEnv(name) {
}
/**
* Run one live experiment through the real updateCase() production path.
* Extract checkpoint data from a raw updateCase() result object.
*
* @param {object} params
* @param {object} params.graph — SituationGraph fixture (must match situationGraphSchema)
* @param {string} params.previousQuestion
* @param {string} params.answer
* @returns {Promise<object>} Checkpoints:
* - userSupportedMeaning — extracted from answer meaning
* - possibleInference — extracted from answer meaning
* - rawAnswerCategory — deterministic category derived from raw answer text
* - proposedMeaningCategory — deterministic category derived from userSupportedMeaning
* - proposalValidation — { success, errors }
* - compatibilityGuard — { passed, warnings, violations }
* - graphMutation — { nodes: [...], edges: [...] } | null
* - selectedQuestion — { id?, question? } | null
* - behaviourSelection — string | null
* - reasoningState — { turnCount, health, phase, progress, unknownStatuses }
* Pure function — no side effects, no I/O. Intended for use both by
* `runLiveExperiment()` (apparatus wrapper) and by deterministic tests
* that pass mock results directly.
*
* @param {object} result — raw result from updateCase()
* @param {object} params — { graph?, answer? }
* - graph: SituationGraph input fixture (for findingsCount total derived from input)
* - answer: previous question text for deriveCategory fallback
* @returns {object} checkpoint object per spec contract
*/
async function runLiveExperiment({ graph, previousQuestion, answer }) {
const baseUrl = assertRequiredEnv("OLLAMA_BASE_URL");
const model = assertRequiredEnv("OLLAMA_MODEL");
function extractLiveExperimentCheckpoint(result, params) {
const answerMeaning = result?.proposal?.answerMeaning || {};
if (baseUrl === "http://localhost:11434" || baseUrl === "http://127.0.0.1:11434") {
throw new Error(
`Live experiment harness refuses to use localhost fallback. ` +
`OLLAMA_BASE_URL=${baseUrl}. Configure a real host in .env.local.`
);
}
const { updateCase } = await import(
"../../lib/graph/orchestrator.js"
);
const { getProvider } = await import("../../lib/llm/provider.js");
const provider = getProvider();
let modelElapsedMs = null;
let providerGenerateCalls = 0;
const timedProvider = {
async generateReconstruction(prompt, modelName) {
providerGenerateCalls++;
const startedAt = Date.now();
const response = await provider.generateReconstruction(prompt, modelName);
modelElapsedMs = Date.now() - startedAt;
return response;
},
};
const endToEndStartedAt = Date.now();
const result = await updateCase(
{ situationGraph: graph, previousQuestion, answer },
{ provider: timedProvider },
);
const endToEndElapsedMs = Date.now() - endToEndStartedAt;
// Extract the standard checkpoints
const answerMeaning = result?.proposal?.answerMeaning || {};
const rawAnswerCat = _deriveCategory(answer);
const supportedCat = _deriveCategory(answerMeaning.userSupportedMeaning || "");
// Graph structural checkpoint (available from updatedSituationGraph only when present)
const updatedGraph = result?.updatedSituationGraph;
const graphCheckpoint = updatedGraph
? {
nodeCount: updatedGraph.nodes?.length ?? 0,
edgeCount: updatedGraph.edges?.length ?? 0,
resolvedNodeIds: updatedGraph.resolvedNodeIds ?? [],
}
: null;
return {
modelElapsedMs,
endToEndElapsedMs,
providerGenerateCalls,
modelName: model,
// ── timing (available via wrapper) ────────────────────
modelElapsedMs: result?.modelElapsedMs !== undefined ? result.modelElapsedMs : null,
endToEndElapsedMs: result?.endToEndElapsedMs !== undefined ? result.endToEndElapsedMs : null,
providerGenerateCalls: result != null && result.providerGenerateCalls != null ? result.providerGenerateCalls : 0,
modelName: result?.modelName || null,
// ── answer meaning extraction ─────────────────────────
userSupportedMeaning: answerMeaning.userSupportedMeaning ?? null,
possibleInference: answerMeaning.possibleInference ?? null,
rawAnswerCategory: rawAnswerCat,
proposedMeaningCategory: supportedCat,
proposalValidation: result?.proposalValidation ?? { success: false, errors: [] },
compatibilityGuard: result?.compatibilityApplied !== undefined
rawAnswerCategory: _deriveCategory(params?.answer),
proposedMeaningCategory: _deriveCategory(answerMeaning.userSupportedMeaning || ""),
// ── validation / guard checkpoints ───────────────────
proposalValidation: result?.proposalValidation ?? { success: false, errors: [] },
compatibilityGuard: result?.compatibilityApplied !== undefined
? { passed: result.compatibilityApplied, warnings: [], violations: [] }
: { passed: false, warnings: [], violations: [] },
graphMutation: result?.appliedGraph
// ── graph mutation checkpoint (from appliedGraph) ────
graphMutation: result?.appliedGraph
? { nodes: result.appliedGraph.nodes ?? null, edges: result.appliedGraph.edges ?? null }
: null,
selectedQuestion: result?.proposal?.selectedQuestion ?? null,
behaviourSelection: result?.behaviourSelection ?? null,
reasoningState: _extractReasoningState(result),
// ── graph structural checkpoint (available from updatedSituationGraph) ─
graphCheckpoint,
// ── findings checkpoint (from input graph fixture, not production path) ─
findingsCount: {
total: (params?.graph && Array.isArray(params.graph))
? params.graph.length
: (params?.graph && params.graph.nodes != null)
? params.graph.nodes.length
: 0,
eligible: "UNAVAILABLE", // not canonically observable without production contract change
},
// ── CU narrative (NOT captured — synthesis is a separate seam from updateCase()) ─
currentUnderstanding: "UNAVAILABLE",
// ── candidate evidence (UNAVAILABLE without changing production code) ────────
eligibleFindingPropositions: "UNAVAILABLE",
// ── helper internals (for diagnostic / test verification) ────────────────────
selectedQuestion: result?.proposal?.selectedQuestion ?? null,
behaviourSelection: result?.behaviourSelection ?? null,
reasoningState: _extractReasoningState(result),
};
}
@@ -158,6 +154,10 @@ function _deriveCategory(text) {
return "cannot_determine";
}
/**
* Extract reasoning state from a raw updateCase() result object.
* Exported for deterministic test verification of preserved behaviour.
*/
function _extractReasoningState(result) {
const rs = result?.reasoningState;
if (!rs) return null;
@@ -170,4 +170,532 @@ function _extractReasoningState(result) {
};
}
module.exports = { runLiveExperiment };
/**
* Run one live experiment through the real updateCase() production path.
*/
async function runLiveExperiment({ graph, previousQuestion, answer, findings = [] }) {
const baseUrl = assertRequiredEnv("OLLAMA_BASE_URL");
const model = assertRequiredEnv("OLLAMA_MODEL");
if (baseUrl === "http://localhost:11434" || baseUrl === "http://127.0.0.1:11434") {
throw new Error(
`Live experiment harness refuses to use localhost fallback. ` +
`OLLAMA_BASE_URL=${baseUrl}. Configure a real host in .env.local.`
);
}
const { updateCase } = await import(
"../../lib/graph/orchestrator.js"
);
const { getProvider } = await import("../../lib/llm/provider.js");
const provider = getProvider();
let modelElapsedMs = null;
let providerGenerateCalls = 0;
const timedProvider = {
async generateReconstruction(prompt, modelName, outputSchema) {
providerGenerateCalls++;
const startedAt = Date.now();
const response = await provider.generateReconstruction(prompt, modelName, outputSchema);
modelElapsedMs = Date.now() - startedAt;
return response;
},
};
const endToEndStartedAt = Date.now();
const result = await updateCase(
{ situationGraph: graph, previousQuestion, answer },
{ provider: timedProvider },
);
const endToEndElapsedMs = Date.now() - endToEndStartedAt;
// ── accumulated state for longitudinal multi-episode use ───
const updatedGraph = result?.updatedSituationGraph ?? null;
const postEpisodeFs = Array.isArray(result?.findings) ? [...result.findings] : [];
return {
/* ── CU narrative (UNAVAILABLE — synthesis is a separate seam from updateCase()) ─── */
currentUnderstanding: "UNAVAILABLE",
/* ── updatedSituationGraph (NEW for longitudinal state feeding) ─── */
updatedSituationGraph: updatedGraph,
/* ── postEpisodeFindings (NEW — accumulated Findings exposed for Episode N+1) ─── */
postEpisodeFindings: postEpisodeFs,
/* ── graph checkpoint (from production result) ─── */
graphCheckpoint: updatedGraph
? {
nodeCount: updatedGraph.nodes?.length ?? 0,
edgeCount: updatedGraph.edges?.length ?? 0,
resolvedNodeIds: updatedGraph.resolvedNodeIds ?? [],
}
: { nodeCount: 0, edgeCount: 0, resolvedNodeIds: [] },
/* ── findings checkpoint (count from accumulated Findings) ─── */
findingsCheckpoint: {
total: postEpisodeFs.length,
eligible: "UNAVAILABLE",
},
/* ── Timing ─── */
modelElapsedMs: result?.modelElapsedMs !== undefined ? result.modelElapsedMs : null,
endToEndElapsedMs: result?.endToEndElapsedMs !== undefined ? result.endToEndElapsedMs : null,
providerGenerateCalls: result != null && result.providerGenerateCalls != null ? result.providerGenerateCalls : 0,
modelName: result?.modelName || null,
/* ── answer meaning (from production result) ─── */
userSupportedMeaning: result?.proposal?.answerMeaning?.userSupportedMeaning ?? null,
possibleInference: result?.proposal?.answerMeaning?.possibleInference ?? null,
/* ── helper internals (for diagnostic / test verification) ─── */
selectedQuestion: result?.proposal?.selectedQuestion ?? null,
behaviourSelection: result?.behaviourSelection ?? null,
reasoningState: _extractReasoningState(result),
};
}
/**
* Compose the canonical post-episode deep-investigation checkpoint sequence:
* updateCase → synthesizeCurrentUnderstanding
*
* Mirrors the production flow in ScenarioForm.handleDoneForNowPromotion / executeEpisodeDone:
* 1. Call the episode done server seam (POST /api/cases/update, episodeMode=true)
* which updates the SituationGraph and accumulates Findings
* 2. Use the updated graph + accumulated findings to synthesize Current Understanding
* via the same seam as production (POST /api/cases/synthesis → synthesizeCurrentUnderstanding)
*
* Returns a combined checkpoint containing:
* - extractLiveExperimentCheckpoint(...) fields from the episode done result
* - currentUnderstanding from the synthesis result
* - findingsCheckpoint with accurate post-episode accumulated findings count
*/
/**
* Compose the canonical post-episode deep-investigation checkpoint sequence.
* Uses production-focused deconstruct for Contribution derivation (not synthetic raw-answer path).
*/
async function composeDeepInvestigationCheckpoint({ situationGraph, previousQuestion, answer, findings = [], contributions = [], targetNodeId, episodeMode }) {
dotenv.config({ path: ".env.local" });
const baseUrl = assertRequiredEnv("OLLAMA_BASE_URL");
const model = assertRequiredEnv("OLLAMA_MODEL");
if (baseUrl === "http://localhost:11434" || baseUrl === "http://127.0.0.1:11434") {
throw new Error(
`Live experiment harness refuses to use localhost fallback. ` +
`OLLAMA_BASE_URL=${baseUrl}. Configure a real host in .env.local.`
);
}
/* ── Step 1: episode done (mirrors executeEpisodeDone server seam) ─── */
const { updateCase, reconsiderCompletedEpisode } = await import(
"../../lib/graph/orchestrator.js"
);
const { getProvider } = await import("../../lib/llm/provider.js");
const provider = getProvider();
let modelElapsedMs = null;
let providerGenerateCalls = 0;
const timedProvider = {
async generateReconstruction(prompt, modelName, outputSchema) {
providerGenerateCalls++;
const startedAt = Date.now();
const response = await provider.generateReconstruction(prompt, modelName, outputSchema);
modelElapsedMs = Date.now() - startedAt;
return response;
},
};
const endToEndStartedAt = Date.now();
let episodeResult;
/* ── Production-equivalent focused-episode seam: targetNodeId triggers the same path as executeEpisodeDone → handleEpisodeMode ─── */
const effectiveContributions = contributions.length > 0 ? [...contributions] : [];
let targetDescription = "";
let effectiveContributionsForTarget = effectiveContributions;
if (targetNodeId && effectiveContributions.length > 0) {
effectiveContributionsForTarget = effectiveContributions.filter(
(c) => c?.targetNodeId === targetNodeId,
);
}
let newFindingsFromContributions = [];
if (effectiveContributionsForTarget.length === 0 && answer && targetNodeId) {
const contribSeq = effectiveContributions.length + 1;
/* ── Load production deconstruction seam (dynamic import to resolve ESM path aliases) ─── */
const { focusedDeconstruct } = await import("../../lib/graph/focused-investigation.js");
// Extract node context for deconstruction prompt (reuses pure extraction function)
const { targetLabel, targetDescription: _td } = extractFocusedTargetInput(situationGraph, targetNodeId);
targetDescription = _td;
const centralStatement = situationGraph.centralStatement ?? "";
// Call production-focused deconstruct — exactly one provider generation
const focused = await focusedDeconstruct({
provider: timedProvider,
modelName: model,
targetNodeId,
targetLabel,
targetDescription,
centralStatement,
question: previousQuestion ?? "",
answer,
});
if (!focused.success) {
throw new Error(`focusedDeconstruct failed for episode ${contribSeq}: ${JSON.stringify(focused.errors)}`);
}
// Map deconstruction result → canonical Contribution shape
const derivedContrib = {
id: `contrib-ep-${String(contribSeq).padStart(4, "0")}`,
targetNodeId: focused.targetNodeId ?? targetNodeId,
question: previousQuestion ?? "",
answer: answer,
sequence: contribSeq,
observations: focused.observations,
uncertainties: focused.uncertainties,
assumptions: focused.assumptions,
relationships: focused.relationships,
possibleFollowUpQuestions: focused.possibleFollowUpQuestions,
};
effectiveContributions.push(derivedContrib);
// Derive canonical Findings from the deconstruction-derived Contribution (production seam)
newFindingsFromContributions = deriveFindingsFromContributions([derivedContrib]).findings;
}
if (episodeMode === true && targetNodeId) {
// Mirrors produceEpisodeAwareGraph: prepareCompletedEpisode → reconsiderCompletedEpisode → applyValidatedProposal
const { prepareCompletedEpisode } = await import("../../lib/graph/episode-preparation.js");
const { applyValidatedProposal } = await import("../../lib/graph/apply-proposal.js");
const prepared = prepareCompletedEpisode({
situationGraph,
targetNodeId,
contributions: effectiveContributions,
findings,
});
if (!prepared?.turns?.length && !prepared?.eligibleCanonicalFindings?.length) {
throw new Error("no_episodic_content: prepared episode has no turns or eligible findings");
}
const reasoning = await reconsiderCompletedEpisode(prepared, { provider: timedProvider });
if (!reasoning.success) {
throw new Error(`reconsiderCompletedEpisode failed: ${reasoning.error}`);
}
const application = await applyValidatedProposal({
situationGraph,
proposal: reasoning.proposal,
evidenceContext: {
isCompletedEpisode: true,
episodeEvidence: prepared,
},
});
if (!application.success) {
throw new Error(`applyValidatedProposal failed: ${application.error}`);
}
const resolvedNodeIds = new Set(application.updatedSituationGraph.resolvedNodeIds ?? []);
resolvedNodeIds.add(targetNodeId);
episodeResult = {
success: true,
updatedSituationGraph: {
...application.updatedSituationGraph,
resolvedNodeIds: [...resolvedNodeIds],
},
appendedFindings: [], // no regular update findings in episode path
proposal: reasoning.proposal,
};
} else {
const result = await updateCase(
{ situationGraph, previousQuestion, answer },
{ provider: timedProvider },
);
episodeResult = result;
}
const endToEndElapsedMs = Date.now() - endToEndStartedAt;
/* ── Step 2: Current Understanding synthesis (mirrors production synthesizeFn) ─── */
const updatedGraph = episodeResult.updatedSituationGraph;
// Accumulate prior findings + newly derived Findings (from new Contribution) for CU synthesis
const priorFindings = Array.isArray(findings) ? [...findings] : [];
const accumulatedEpisodes = episodeResult.appendedFindings ?? [];
const postEpisodeFindings = [...priorFindings, ...newFindingsFromContributions, ...accumulatedEpisodes];
let currentUnderstanding = null;
let synthesisErrorDiagnostic = null;
if (episodeResult.success && updatedGraph) {
// Reuse the same provider/model plumbing — no re-creation, just reuse
const { synthesizeCurrentUnderstanding } = await import("../../lib/graph/current-understanding-synthesis.js");
try {
const synthesized = await synthesizeCurrentUnderstanding(
{ situationGraph: updatedGraph, findings: postEpisodeFindings },
{ provider: timedProvider, modelName: model },
);
currentUnderstanding = synthesized?.currentUnderstanding ?? null;
} catch (synthErr) {
// Expose the safe structural diagnostic from synthesis validation failure
if (synthErr?.sanitizedDiagnostic) {
synthesisErrorDiagnostic = synthErr.sanitizedDiagnostic;
}
currentUnderstanding = null;
}
}
// Consistent graphCheckpoint for both success and failure paths
const graphCheckpoint = updatedGraph
? {
nodeCount: updatedGraph.nodes?.length ?? 0,
edgeCount: updatedGraph.edges?.length ?? 0,
resolvedNodeIds: updatedGraph.resolvedNodeIds ?? [],
}
: { nodeCount: 0, edgeCount: 0, resolvedNodeIds: [] };
// Accurate post-episode accumulated findings count
const { filterEligibleFindings } = await import("../../lib/graph/current-understanding-synthesis.js");
const eligibleCount = postEpisodeFindings.length > 0
? filterEligibleFindings(postEpisodeFindings).length
: 0;
return {
/* ── CU synthesis result (mirrors production synthesizeFn) ─── */
currentUnderstanding,
/* ── updatedSituationGraph (NEW — for longitudinal Episode N+1 state feeding) ─── */
updatedSituationGraph: updatedGraph,
/* ── postEpisodeContributions (NEW — accumulated Contributions for longitudinal tracking) ─── */
postEpisodeContributions: effectiveContributions,
/* ── postEpisodeFindings (NEW — accumulated Findings for Episode N+1) ─── */
postEpisodeFindings: postEpisodeFindings,
/* ── Accurate post-episode accumulated findings count ─── */
findingsCheckpoint: {
total: postEpisodeFindings.length,
eligible: eligibleCount,
},
/* ── graph checkpoint (from production result) ─── */
graphCheckpoint,
/* ── Timing ─── */
modelElapsedMs,
endToEndElapsedMs,
providerGenerateCalls,
modelName: episodeResult.modelName ?? null,
/* ── CU synthesis error diagnostic (NEW — safe structural shape on validation failure) ─── */
synthesisErrorDiagnostic,
/* ── Answer meaning (from production result) ─── */
userSupportedMeaning: episodeResult?.proposal?.answerMeaning?.userSupportedMeaning ?? null,
possibleInference: episodeResult?.proposal?.answerMeaning?.possibleInference ?? null,
/* ── Focused-investigation input values (apparatus correction) ─── */
focusedInvestigationInput: (previousQuestion != null || answer != null)
? {
targetDescription: targetDescription ?? "",
question: previousQuestion ?? "",
answer,
}
: null,
};
}
/**
* Serialize a completed deep-investigation checkpoint into the durable
* focused-deconstruction evidence record.
*
* Pure function — no side effects, no I/O, no model calls. Intended for
* deterministic test fixtures and external audit tools that need a stable
* JSON-serializable representation of the five deconstruction arrays plus
* exact input values and execution metadata.
*
* @param {object} checkpoint — returned from composeDeepInvestigationCheckpoint()
* @param {object} identity — { experimentId, scenarioId, targetNodeId, evidenceDesignation }
* - experimentId: unique identifier for the experiment run
* - scenarioId: identifier for the scenario (e.g. "HS-03C")
* - targetNodeId: the situationGraph node that was selected as focus
* - evidenceDesignation: "synthetic" or "real-user" (or any string)
* @param {object} [metadata] — optional execution metadata from the apparatus
* - modelName, providerGenerateCalls, modelElapsedMs, endToEndElapsedMs, success
* Any missing field is set to null (never fabricated).
* @returns {object} JSON-serializable evidence record per fixed contract
*/
function serializeFocusedInvestigationEvidence(checkpoint, identity, metadata) {
/* ── Extract the five structured deconstruction arrays ────────────── */
const contributions = Array.isArray(checkpoint?.postEpisodeContributions)
? checkpoint.postEpisodeContributions
: [];
// There should be exactly one contribution per focused-investigation run;
// if multiple exist (episode path with accumulated contributions),
// take the first one that carries the deconstruction arrays.
const deconContribution = contributions.find(
(c) => Array.isArray(c?.observations) && Array.isArray(c?.uncertainties)
) || contributions[0] || {};
return {
/* ── Identity (supplied by caller, not derived) ─────────────────── */
experimentId: identity.experimentId ?? null,
scenarioId: identity.scenarioId ?? null,
targetNodeId: identity.targetNodeId ?? null,
evidenceDesignation: identity.evidenceDesignation ?? null,
/* ── Exact focused-investigation inputs (from checkpoint) ──────── */
canonicalTargetDescription: checkpoint.focusedInvestigationInput?.targetDescription ?? "",
actualQuestion: checkpoint.focusedInvestigationInput?.question ?? "",
exactAnswer: checkpoint.focusedInvestigationInput?.answer ?? "",
/* ── Structured model output (preserved verbatim, no transformation) ─── */
observations: deconContribution.observations ?? null,
uncertainties: deconContribution.uncertainties ?? null,
assumptions: deconContribution.assumptions ?? null,
relationships: deconContribution.relationships ?? null,
possibleFollowUpQuestions: deconContribution.possibleFollowUpQuestions ?? null,
/* ── Execution metadata (supplied by caller; null when not exposed) ─ */
modelName: metadata?.modelName ?? null,
providerGenerateCalls: metadata?.providerGenerateCalls ?? null,
modelElapsedMs: metadata?.modelElapsedMs ?? null,
endToEndElapsedMs: metadata?.endToEndElapsedMs ?? null,
executionStatus: (metadata?.success !== undefined)
? (metadata.success === true ? "completed" : "failed")
: null,
};
}
/**
* Save a completed focused-investigation checkpoint to a durable JSON file.
*
* Wrapper around `serializeFocusedInvestigationEvidence()` that performs
* explicit persistence via the caller-supplied output path. Intended for
* deterministic test fixtures and future live experiment runbooks — NOT
* called automatically from production code.
*
* @param {object} checkpoint — returned from composeDeepInvestigationCheckpoint()
* @param {object} identity — { experimentId, scenarioId, targetNodeId, evidenceDesignation }
* @param {object} metadata — { modelName?, providerGenerateCalls?, modelElapsedMs?, endToEndElapsedMs?, success? }
* @param {string} outputPath — explicit absolute or relative file path for the JSON output
* @returns {{ output_path: string, verifiedEvidence: object }} { output_path, verifiedEvidence }
* @throws {Error} on invalid inputs, missing required fields, ambiguous contribution, or I/O failure
*/
function saveFocusedInvestigationEvidence(checkpoint, identity, metadata, outputPath) {
const fs = require("fs");
/* ── 2. Require explicit non-empty output path ────────────────────── */
if (!outputPath || typeof outputPath !== "string" || !outputPath.trim()) {
throw new Error("saveFocusedInvestigationEvidence requires an explicit non-empty outputPath string");
}
/* ── 3. Validate checkpoint has exactly one valid contribution ────── */
const contributions = Array.isArray(checkpoint?.postEpisodeContributions)
? checkpoint.postEpisodeContributions
: [];
const validContributionCandidates = contributions.filter(
(c) => Array.isArray(c?.observations) && Array.isArray(c?.uncertainties)
);
if (validContributionCandidates.length === 0) {
throw new Error(
"saveFocusedInvestigationEvidence requires a checkpoint with at least one valid focused-deconstruction contribution; found 0"
);
}
if (validContributionCandidates.length > 1) {
throw new Error(
"saveFocusedInvestigationEvidence rejects ambiguous checkpoint: multiple candidates exist (" +
validContributionCandidates.length +
"); supply a single-target checkpoint or resolve ambiguity before saving"
);
}
/* ── 4. Validate identity and input values ────────────────────────── */
if (!identity || typeof identity !== "object") {
throw new Error("saveFocusedInvestigationEvidence requires an identity object");
}
if (identity.experimentId == null) {
throw new Error("saveFocusedInvestigationEvidence: identity.missing experimentId");
}
if (identity.scenarioId == null) {
throw new Error("saveFocusedInvestigationEvidence: identity.missing scenarioId");
}
if (identity.targetNodeId == null) {
throw new Error("saveFocusedInvestigationEvidence: identity.missing targetNodeId");
}
if (identity.evidenceDesignation == null) {
throw new Error("saveFocusedInvestigationEvidence: identity.missing evidenceDesignation");
}
const fiInputs = checkpoint.focusedInvestigationInput;
if (!fiInputs || typeof fiInputs !== "object") {
throw new Error("saveFocusedInvestigationEvidence: checkpoint missing focusedInvestigationInput object");
}
if (fiInputs.targetDescription == null || fiInputs.question == null || fiInputs.answer == null) {
throw new Error(
"saveFocusedInvestigationEvidence: checkpoint missing required focused-investigation input values (targetDescription, question, answer)"
);
}
/* ── 1. Serialize via existing serializer ─────────────────────────── */
const serialized = serializeFocusedInvestigationEvidence(checkpoint, identity, metadata);
/* ── 5-6. Write the complete record as readable JSON ──────────────── */
const jsonContent = JSON.stringify(serialized, null, 2);
fs.writeFileSync(outputPath, jsonContent, "utf8");
/* ── 7-8. Read back and verify structural equality ────────────────── */
const recoveredRaw = fs.readFileSync(outputPath, "utf8");
const recovered = JSON.parse(recoveredRaw);
if (JSON.stringify(serialized) !== JSON.stringify(recovered)) {
throw new Error(
"saveFocusedInvestigationEvidence: write/read-back verification failed — recovered JSON does not match serialized record"
);
}
/* ── 9. Return output path and verified evidence record ───────────── */
return { output_path: outputPath, verifiedEvidence: recovered };
}
/**
* Extract the focused-investigation target-node input (label and description)
* from a situationGraph by node id.
*
* Pure function — no side effects, no I/O, no model calls.
* Mirrors exactly the lookup and fallback semantics used inside the
* guarded focused-deconstruction path of composeDeepInvestigationCheckpoint:
* - situationGraph.nodes.find(n => n.id === targetNodeId)
* - ?? "" (empty-string fallback when node not found)
*
* @param {object} situationGraph — SituationGraph input containing { nodes, centralStatement? }
* @param {string} targetNodeId — id of the focused-investigation target node
* @returns {{ targetLabel: string, targetDescription: string }}
*/
function extractFocusedTargetInput(situationGraph, targetNodeId) {
const targetNode = situationGraph.nodes.find(n => n.id === targetNodeId);
return {
targetLabel: targetNode?.label ?? "",
targetDescription: targetNode?.description ?? "",
};
}
module.exports = { runLiveExperiment, extractLiveExperimentCheckpoint, _extractReasoningState, composeDeepInvestigationCheckpoint, serializeFocusedInvestigationEvidence, saveFocusedInvestigationEvidence, extractFocusedTargetInput };
@@ -0,0 +1,203 @@
import { describe, expect, it } from "vitest";
import { buildInitialGraph } from "@/lib/graph/builder.js";
import { formulateQuestionForTarget } from "@/lib/graph/focused-investigation.js";
import { makeGraph, makeNodeId } from "@/lib/graph/schema.js";
import { buildFacilitatorViewModel } from "@/lib/presentation/facilitator-view-adapter.js";
import { buildPrompt } from "@/lib/reconstruction/prompt.js";
import { validateReconstructionOpenQuestions } from "@/lib/reconstruction/open-question-presentation.js";
import { reconstructionV2Schema } from "@/lib/reconstruction/schema.js";
function reconstructionWithUnknowns(importantUnknowns) {
return {
inputClassification: {
primaryType: "other",
secondaryTypes: [],
reasoningModes: ["identify_missing_information"],
classificationReason: "The scenario contains unresolved factors.",
confidence: "medium",
},
reconstruction: {
summary: "A situation with unresolved factors.",
actors: [],
systemsOrObjects: [],
expectedStates: [],
observedStates: [],
differences: [],
knownTransitions: [],
unexplainedTransitions: [],
contradictions: [],
importantUnknowns,
plausibleInterpretations: [],
relationships: [],
},
evidence: [],
nextQuestion: {
id: "next-question",
question: "What information would clarify the situation?",
targets: [],
reason: "The situation has unresolved factors.",
expectedInformationValue: "medium",
reasoningMode: "identify_missing_information",
},
};
}
const CASES = [
{
name: "recruitment uncertainty",
description: "Whether skilled production staff can be recruited within the required timeframe",
question: "Can skilled production staff be recruited within the required timeframe?",
terms: ["skilled production staff", "required timeframe"],
},
{
name: "demand uncertainty",
description: "The level of demand in the North West market needed to justify the expansion cost",
question: "What level of demand in the North West market is needed to justify the expansion cost?",
terms: ["North West market", "justify the expansion cost"],
},
{
name: "historical uncertainty",
description: "Whether the prior staffing level was sufficient before the expansion",
question: "Was the prior staffing level sufficient before the expansion?",
terms: ["prior staffing level", "before the expansion"],
},
{
name: "non-business uncertainty",
description: "The time at which the missing hiker last used the eastern trail",
question: "When did the missing hiker last use the eastern trail?",
terms: ["missing hiker", "eastern trail"],
},
];
describe("Open Question presentation", () => {
it.each(["v0.2", "v0.3", "v0.4", "v0.5"])("instructs reconstruction prompt %s to produce a direct Open Question", async (version) => {
const { prompt } = await buildPrompt("A scenario with an unresolved condition.", version);
expect(prompt).toContain("importantUnknown");
expect(prompt).toContain("natural, direct interrogative");
expect(prompt).toContain('"question"');
expect(prompt).toContain('"Whether …"');
});
it.each(CASES)("preserves $name meaning while displaying a direct interrogative", ({ description, question, terms }) => {
const parsed = reconstructionV2Schema.parse(
reconstructionWithUnknowns([{ id: "unknown", description, question, confidence: "medium" }]),
);
const graph = buildInitialGraph({ reconstruction: parsed.reconstruction, evidence: [] });
const unknown = graph.nodes.find((node) => node.kind === "unknown");
const viewModel = buildFacilitatorViewModel({ nodes: graph.nodes, resolvedIds: new Set() });
expect(unknown.label).toBe(description);
expect(unknown.description).toBe(description);
expect(unknown.openQuestion).toBe(question);
// Canonical description is now the initial display text for unresolved unknowns.
expect(viewModel.investigating.items).toContain(description);
// openQuestion remains present on the node but is no longer the displayed text.
expect(question).toMatch(/\?$/);
});
it("leaves canonical identity, explicit target selection, and focused formulation unchanged", () => {
const description = CASES[0].description;
const question = CASES[0].question;
const graphData = buildInitialGraph({
reconstruction: reconstructionWithUnknowns([
{ id: "recruitment", description, question, confidence: "medium" },
]).reconstruction,
evidence: [],
});
const target = graphData.nodes.find((node) => node.kind === "unknown");
const graph = makeGraph({
centralStatement: "A manufacturer is considering expansion.",
nodes: graphData.nodes,
edges: graphData.edges,
activeUnknownNodeId: null,
resolvedNodeIds: [],
currentSummary: "Expansion depends on recruitment capacity.",
});
const before = JSON.stringify(graph);
const result = formulateQuestionForTarget({ situationGraph: graph, targetNodeId: target.id });
expect(target.id).toBe(makeNodeId(description));
expect(target.id).not.toBe(makeNodeId(question));
expect(result.success).toBe(true);
expect(result.targetNodeId).toBe(target.id);
expect(result.question).toMatch(/\?$/);
expect(result.question.toLowerCase()).toContain("staff");
expect(JSON.stringify(graph)).toBe(before);
});
it("keeps persisted legacy unknowns valid and falls back to canonical text when no display question exists", () => {
const legacyDescription = "Whether the figures are comparable";
const legacyNode = makeGraph({
centralStatement: "Compare two figures.",
nodes: [
{
id: makeNodeId(legacyDescription),
label: legacyDescription,
description: legacyDescription,
kind: "unknown",
status: "unknown",
confidence: "medium",
evidenceIds: [],
dependsOn: [],
affects: [],
parentId: null,
childIds: [],
},
],
edges: [],
activeUnknownNodeId: null,
resolvedNodeIds: [],
currentSummary: "The comparison remains unresolved.",
});
const viewModel = buildFacilitatorViewModel({ nodes: legacyNode.nodes, resolvedIds: new Set() });
expect(viewModel.investigating.items).toContain(legacyDescription);
expect(legacyNode.nodes[0]).not.toHaveProperty("openQuestion");
});
it.each([
["missing", undefined, false],
["empty", " ", true],
["whether fragment", "Whether the facility can recruit workers in time", true],
["whether fragment with punctuation", "Whether the facility can recruit workers in time?", true],
["statement", "The facility can recruit workers in time.", true],
])(
"keeps an invalid %s presentation canonically selectable and investigable",
(_name, question, includeQuestion) => {
const description =
"Whether the facility can recruit skilled workers within the launch timeframe.";
const unknown = {
id: "unknown",
description,
confidence: "medium",
};
if (includeQuestion) unknown.question = question;
const reconstruction = reconstructionWithUnknowns([unknown]).reconstruction;
const validated = validateReconstructionOpenQuestions(reconstruction);
const graphParts = buildInitialGraph({
reconstruction: validated.reconstruction,
evidence: [],
});
const graph = makeGraph({
centralStatement: "The facility launch depends on workforce availability.",
nodes: graphParts.nodes,
edges: graphParts.edges,
currentSummary: "Workforce availability remains unresolved.",
});
const graphUnknown = graph.nodes.find((node) => node.kind === "unknown");
const focused = formulateQuestionForTarget({
situationGraph: graph,
targetNodeId: graphUnknown.id,
});
expect(validated.issues).toHaveLength(1);
expect(graphUnknown.label).toBe(description);
expect(graphUnknown.description).toBe(description);
expect(graphUnknown).not.toHaveProperty("openQuestion");
expect(graphUnknown.id).toBe(makeNodeId(description));
expect(focused.success).toBe(true);
expect(focused.targetNodeId).toBe(graphUnknown.id);
},
);
});
+30
View File
@@ -55,6 +55,8 @@ function makeAnalysisResult(overrides = {}) {
compatibilityApplied: false,
compatibilityChanges: [],
compatibilityWarnings: [],
presentationValidationStatus: "valid",
presentationValidationIssues: [],
...overrides,
};
}
@@ -399,9 +401,37 @@ describe("lib/graph/orchestrator startCase", () => {
valid: true,
errors: [],
});
expect(result.diagnostics.presentationValidationStatus).toBe("valid");
expect(result.diagnostics.presentationValidationIssues).toEqual([]);
expect(downstreamProvider.generateReconstruction).not.toHaveBeenCalled();
});
it("propagates nonfatal Open Question presentation issues through diagnostics", async () => {
const issue = {
category: "question_indirect_proposition",
path: ["reconstruction", "importantUnknowns", 0, "question"],
unknownId: "unk-1",
message: "The generated Open Question used an indirect uncertainty proposition.",
severity: "nonfatal",
};
mockAnalyseScenario.mockResolvedValue(
makeAnalysisResult({
presentationValidationStatus: "partial",
presentationValidationIssues: [issue],
}),
);
const { startCase } = await import("@/lib/graph/orchestrator.js");
const result = await startCase(
{ scenario: "Scenario text" },
{ reconstructionOnly: true },
);
expect(result.success).toBe(true);
expect(result.diagnostics.presentationValidationStatus).toBe("partial");
expect(result.diagnostics.presentationValidationIssues).toEqual([issue]);
});
it("rejects invalid request input without throwing", async () => {
const { startCase } = await import("@/lib/graph/orchestrator.js");
+246
View File
@@ -1614,6 +1614,252 @@ describe("60B.75 — real decision detection for sufficiency question", () => {
});
});
// ── Experiment 3 Iteration 3 — Deterministic Reproduction Recovery ──────────
// Hypothesis: A Whether ... uncertainty passed into baseline_reconstruction
// interpolates into a template requiring a noun phrase, producing malformed output.
// Fixed input: "Whether additional skilled staff needed for expansion can be recruited"
describe("Experiment 3 — Whether proposition in baseline reconstruction", () => {
it("suppresses the reproduced malformed baseline template at the focused-investigation compatibility boundary", () => {
const unknown = makeNode({
id: "n-experiment3-staffing",
label: "Staffing certainty gap",
description:
"Whether additional skilled staff needed for expansion can be recruited",
kind: "unknown",
status: "unknown",
confidence: "medium",
});
const graph = makeGraph({
centralStatement:
"We need to compare our current baseline against the comparable state before staffing is secured for expansion.",
nodes: [unknown],
edges: [],
activeUnknownNodeId: unknown.id,
resolvedNodeIds: [],
currentSummary: "Test summary",
});
const proposed = formulateQuestion({ node: unknown, graph });
// Preserve the deterministic reproduction that identified the bug.
expect(proposed.strategy).toBe("baseline_reconstruction");
expect(proposed.question).toBe(
"What was the comparable state before whether additional skilled staff needed for expansion can be recruited?",
);
const corrected = formulateQuestion({
node: unknown,
graph,
context: {
strategyCompatibilityVerdict: {
strategyKey: "baseline_reconstruction",
relevance: "compatible",
},
strategyTemplateCompatibility: "incompatible",
},
});
expect(corrected.strategy).toBeNull();
expect(corrected.question).toBe(
"What evidence would clarify whether additional skilled staff needed for expansion can be recruited?",
);
expect(corrected.question.toLowerCase()).not.toContain("before whether");
});
});
// ── Experiment 14 — Deterministic Compatibility Verdict Contract ───────────
describe("Experiment 14 — strategy compatibility verdict", () => {
function makeBaselineWhetherFixture() {
const unknown = makeNode({
id: "n-experiment14-staffing",
label: "Staffing certainty gap",
description:
"Whether additional skilled staff needed for expansion can be recruited",
kind: "unknown",
status: "unknown",
confidence: "medium",
});
const graph = makeGraphFor(unknown, {
centralStatement:
"We need to compare our current baseline against the comparable state before staffing is secured for expansion.",
});
return { unknown, graph };
}
it("preserves existing strategy and question when no verdict is supplied", () => {
const { unknown, graph } = makeBaselineWhetherFixture();
const result = formulateQuestion({ node: unknown, graph });
expect(result.strategy).toBe("baseline_reconstruction");
expect(result.question).toBe(
"What was the comparable state before whether additional skilled staff needed for expansion can be recruited?",
);
});
it("permits the matching proposed strategy when the verdict is compatible", () => {
const { unknown, graph } = makeBaselineWhetherFixture();
const result = formulateQuestion({
node: unknown,
graph,
context: {
strategyCompatibilityVerdict: {
strategyKey: "baseline_reconstruction",
relevance: "compatible",
},
},
});
expect(result.strategy).toBe("baseline_reconstruction");
expect(result.question).toBe(
"What was the comparable state before whether additional skilled staff needed for expansion can be recruited?",
);
});
it("suppresses an incompatible proposed strategy and uses the existing evidence path", () => {
const { unknown, graph } = makeBaselineWhetherFixture();
const graphSnapshot = JSON.stringify(graph);
const result = formulateQuestion({
node: unknown,
graph,
context: {
strategyCompatibilityVerdict: {
strategyKey: "baseline_reconstruction",
relevance: "incompatible",
},
},
});
expect(result.strategy).toBeNull();
expect(result.question).toBe(
"What evidence would clarify whether additional skilled staff needed for expansion can be recruited?",
);
expect(JSON.stringify(graph)).toBe(graphSnapshot);
});
it("does not let invalid or mismatched verdicts authorise a proposed strategy", () => {
const { unknown, graph } = makeBaselineWhetherFixture();
const invalid = formulateQuestion({
node: unknown,
graph,
context: {
strategyCompatibilityVerdict: {
strategyKey: "baseline_reconstruction",
relevance: "unknown",
},
},
});
const mismatched = formulateQuestion({
node: unknown,
graph,
context: {
strategyCompatibilityVerdict: {
strategyKey: "evidence_gathering",
relevance: "compatible",
},
},
});
expect(invalid.strategy).toBeNull();
expect(mismatched.strategy).toBeNull();
expect(invalid.question).toContain("What evidence would clarify whether");
expect(mismatched.question).toContain("What evidence would clarify whether");
});
it("allows a compatible historical strategy for a coherent Whether-proposition", () => {
const unknown = makeNode({
id: "n-experiment14-historical",
label: "Earlier comparable state",
description: "Whether the prior staffing level was sufficient for expansion",
kind: "unknown",
status: "unknown",
confidence: "medium",
});
const graph = makeGraphFor(unknown, {
centralStatement:
"Compare the prior staffing baseline with the level needed before the expansion.",
});
const result = formulateQuestion({
node: unknown,
graph,
context: {
strategyCompatibilityVerdict: {
strategyKey: "baseline_reconstruction",
relevance: "compatible",
},
},
});
expect(result.strategy).toBe("baseline_reconstruction");
});
});
// ── Experiment 6 — Graph-Context Causal Comparison ─────────────────────────────
// Hypothesis: baseline_reconstruction strategy on the recruitment unknown is driven
// by historical-baseline vocabulary in graph-level context (centralStatement), not
// in the recruitment uncertainty itself. Removing that vocabulary should prevent
// the specific baseline-selection route while preserving all node-level input.
describe("Experiment 6 — Isolate graph-context influence on strategy selection", () => {
it("removing baseline vocabulary from centralStatement changes strategy away from baseline_reconstruction", () => {
// ── shared open-question fixture (identical in both variants) ──
const openQuestion = makeNode({
id: "n-experiment3-staffing",
label: "Staffing certainty gap",
description:
"Whether additional skilled staff needed for expansion can be recruited",
kind: "unknown",
status: "unknown",
confidence: "medium",
});
// ── Variant A — baseline context present (existing fixture) ──
const graphA = makeGraph({
centralStatement:
"We need to compare our current baseline against the comparable state before staffing is secured for expansion.",
nodes: [openQuestion],
edges: [],
activeUnknownNodeId: openQuestion.id,
resolvedNodeIds: [],
currentSummary: "Test summary",
});
// ── Variant B — baseline context absent (neutral wording) ──
const graphB = makeGraph({
centralStatement:
"Current staffing levels are being reviewed as we prepare for the planned expansion.",
nodes: [openQuestion],
edges: [],
activeUnknownNodeId: openQuestion.id,
resolvedNodeIds: [],
currentSummary: "Test summary",
});
// ── Execute both variants ──
const resultA = formulateQuestion({ node: openQuestion, graph: graphA });
const resultB = formulateQuestion({ node: openQuestion, graph: graphB });
// Record for report
console.log("[Exp6-A] strategy:", resultA.strategy, "| question:", resultA.question);
console.log("[Exp6-B] strategy:", resultB.strategy, "| question:", resultB.question);
// Causal contrast: baseline_reconstruction requires BOTH graph-level AND node-level
// baseline vocabulary to match the first (strict) selection path.
// Variant A must select baseline_reconstruction.
expect(resultA.strategy).toBe("baseline_reconstruction");
// Variant B must NOT select baseline_reconstruction (graph-level baseline removed).
expect(resultB.strategy).not.toBe("baseline_reconstruction");
// The malformed "before whether" output must disappear in Variant B.
expect(resultB.question.toLowerCase()).not.toContain(
"before whether",
);
});
});
// ── 60B.80 — sufficiency re-selection after proposal application (no false State B) ────
describe("60B.80 — no false State B after proposal adds unresolved material factor", () => {
+44
View File
@@ -0,0 +1,44 @@
import React from "react";
import { describe, expect, it } from "vitest";
import { renderToStaticMarkup } from "react-dom/server";
import { NextRequest } from "next/server";
import PrivacyPage from "@/app/privacy/page.jsx";
import TermsPage from "@/app/terms/page.jsx";
import CookiesPage from "@/app/cookies/page.jsx";
import LegalNavigation from "@/components/legal-navigation.jsx";
import RootLayout from "@/app/layout.jsx";
import { middleware } from "@/middleware.js";
describe("public legal pages", () => {
it.each([
["Privacy Policy", PrivacyPage],
["Terms of Use", TermsPage],
["Cookie Policy", CookiesPage],
])("renders the %s heading", (heading, Page) => {
const html = renderToStaticMarkup(<Page />);
expect(html).toContain(`<h1 class="text-3xl font-bold tracking-tight text-teal-700">${heading}</h1>`);
expect(html).not.toContain('aria-label="Legal information"');
});
it("keeps the root layout as the single legal navigation owner", () => {
const html = renderToStaticMarkup(
<RootLayout>
<PrivacyPage />
</RootLayout>,
);
expect(html.match(/aria-label="Legal information"/g)).toHaveLength(1);
});
it("exposes all public legal links without cookie consent controls", () => {
const html = renderToStaticMarkup(<LegalNavigation />);
expect(html).toContain('href="/privacy"');
expect(html).toContain('href="/terms"');
expect(html).toContain('href="/cookies"');
expect(html).not.toMatch(/Accept cookies|Reject cookies/i);
});
it.each(["privacy", "terms", "cookies"])("keeps /%s public", async (path) => {
const response = await middleware(new NextRequest(`http://localhost:3000/${path}`));
expect(response.status).toBe(200);
});
});
+324 -1
View File
@@ -8,7 +8,11 @@ import {
normaliseOpenAITransportResponse,
reconstructionJsonSchema,
} from "@/lib/llm/provider.js";
import { reconstructionV2Schema } from "@/lib/reconstruction/schema.js";
import {
reconstructionGenerationJsonSchema,
reconstructionGenerationSchema,
reconstructionV2Schema,
} from "@/lib/reconstruction/schema.js";
import { focusedDeconstructJsonSchema } from "@/lib/graph/focused-investigation.js";
import { z } from "zod";
@@ -128,6 +132,38 @@ describe("OllamaLlmProvider chat capability detection", () => {
}
});
it("forwards caller cancellation to the Ollama capability probe without starting inference", async () => {
const originalBaseUrl = process.env.OLLAMA_BASE_URL;
const controller = new AbortController();
const fetchSpy = vi.fn().mockImplementation((_, options) => new Promise((_, reject) => {
options.signal.addEventListener("abort", () => {
const error = new Error("aborted");
error.name = "AbortError";
reject(error);
}, { once: true });
}));
vi.stubGlobal("fetch", fetchSpy);
process.env.OLLAMA_BASE_URL = "http://ollama.test";
try {
__resetChatSupportForTests();
const request = getProvider().generateReconstruction(
"prompt",
"configured-model",
undefined,
{ signal: controller.signal },
);
controller.abort();
await expect(request).rejects.toMatchObject({ name: "AbortError" });
expect(fetchSpy).toHaveBeenCalledTimes(1);
expect(fetchSpy.mock.calls[0][1].signal).toBe(controller.signal);
} finally {
vi.unstubAllGlobals();
if (originalBaseUrl === undefined) delete process.env.OLLAMA_BASE_URL;
else process.env.OLLAMA_BASE_URL = originalBaseUrl;
}
});
it("reports chat-skipped generate fallback execution", async () => {
const originalBaseUrl = process.env.OLLAMA_BASE_URL;
const fetchSpy = vi.fn()
@@ -189,6 +225,139 @@ describe("OllamaLlmProvider chat capability detection", () => {
});
});
describe("Ollama freeform semantics", () => {
it("omits format constraint when outputSchema is explicitly null (chat path)", async () => {
const originalBaseUrl = process.env.OLLAMA_BASE_URL;
const fetchSpy = vi.fn()
.mockResolvedValueOnce({ ok: true, text: async () => "" })
.mockResolvedValueOnce({
ok: true,
json: async () => ({ message: { content: '{"freeform":"output"}' } }),
});
vi.stubGlobal("fetch", fetchSpy);
process.env.OLLAMA_BASE_URL = "http://ollama.test";
try {
__resetChatSupportForTests();
const provider = getProvider();
const result = await provider.generateReconstruction("prompt", "model", null, { signal: undefined });
expect(result.response).toBe("{\"freeform\":\"output\"}");
expect(result.providerApiPath).toBe("/api/chat");
// No format key in request body when schema is explicitly null
const requestBody = JSON.parse(fetchSpy.mock.calls[1][1].body);
expect(requestBody).not.toHaveProperty("format");
} finally {
vi.unstubAllGlobals();
if (originalBaseUrl === undefined) delete process.env.OLLAMA_BASE_URL;
else process.env.OLLAMA_BASE_URL = originalBaseUrl;
}
});
it("omits format constraint when outputSchema is explicitly null (generate fallback)", async () => {
const originalBaseUrl = process.env.OLLAMA_BASE_URL;
const fetchSpy = vi.fn()
.mockResolvedValueOnce({ ok: false, status: 501, text: async () => "" })
.mockResolvedValueOnce({
ok: true,
json: async () => ({ response: "freeform text from /api/generate" }),
});
vi.stubGlobal("fetch", fetchSpy);
process.env.OLLAMA_BASE_URL = "http://ollama.test";
try {
__resetChatSupportForTests();
const provider = getProvider();
const result = await provider.generateReconstruction("prompt", "model", null, { signal: undefined });
expect(result.response).toBe("freeform text from /api/generate");
expect(result.providerApiPath).toBe("/api/generate");
// No format key in request body when schema is explicitly null
const requestBody = JSON.parse(fetchSpy.mock.calls[1][1].body);
expect(requestBody).not.toHaveProperty("format");
} finally {
vi.unstubAllGlobals();
if (originalBaseUrl === undefined) delete process.env.OLLAMA_BASE_URL;
else process.env.OLLAMA_BASE_URL = originalBaseUrl;
}
});
it("still applies default schema when outputSchema is undefined", async () => {
const originalBaseUrl = process.env.OLLAMA_BASE_URL;
const fetchSpy = vi.fn()
.mockResolvedValueOnce({ ok: true, text: async () => "" })
.mockResolvedValueOnce({
ok: true,
json: async () => ({ message: { content: "{}" } }),
});
vi.stubGlobal("fetch", fetchSpy);
process.env.OLLAMA_BASE_URL = "http://ollama.test";
try {
__resetChatSupportForTests();
await getProvider().generateReconstruction("prompt", "model", undefined, { signal: undefined });
const requestBody = JSON.parse(fetchSpy.mock.calls[1][1].body);
// undefined → fallback to reconstructionJsonSchema (format IS present)
expect(requestBody.format).toBeDefined();
} finally {
vi.unstubAllGlobals();
if (originalBaseUrl === undefined) delete process.env.OLLAMA_BASE_URL;
else process.env.OLLAMA_BASE_URL = originalBaseUrl;
}
});
});
describe("OpenAI freeform semantics", () => {
it("uses text format when outputSchema is explicitly null", async () => {
const provider = createOpenAIReconstructionProvider({
apiKey: "test-key",
fetchImpl: vi.fn().mockResolvedValue({
ok: true,
json: async () => ({ output_text: "freeform text response" }),
}),
});
await provider.generateReconstruction("prompt", "model", null);
const request = JSON.parse(
provider.fetchImpl.mock.calls[0][1].body,
);
expect(request.text.format.type).toBe("text");
});
it("returns raw text without JSON recovery when outputSchema is explicitly null", async () => {
const provider = createOpenAIReconstructionProvider({
apiKey: "test-key",
fetchImpl: vi.fn().mockResolvedValue({
ok: true,
json: async () => ({ output_text: "not-json-at-all {broken" }),
}),
});
const result = await provider.generateReconstruction("prompt", "model", null);
expect(result.response).toBe("not-json-at-all {broken");
expect(result.providerApiPath).toBe("/v1/responses");
});
it("still applies json_schema format when outputSchema is undefined", async () => {
const provider = createOpenAIReconstructionProvider({
apiKey: "test-key",
fetchImpl: vi.fn().mockResolvedValue({
ok: true,
json: async () => ({ output_text: "{}" }),
}),
});
await provider.generateReconstruction("prompt");
const request = JSON.parse(provider.fetchImpl.mock.calls[0][1].body);
expect(request.text.format.type).toBe("json_schema");
});
});
describe("OpenAI reconstruction provider experiment seam", () => {
function assertStrictObjectInvariant(schema, root = schema, visited = new Set()) {
if (!schema || typeof schema !== "object" || visited.has(schema)) return;
@@ -267,6 +436,137 @@ describe("OpenAI reconstruction provider experiment seam", () => {
expect(acceptsNull(transition.properties.id)).toBe(false);
});
it("preserves optional transport nullability for the generation-specific schema", async () => {
const fetchImpl = vi.fn().mockResolvedValue({
ok: true,
json: async () => ({ output_text: "{}" }),
});
const provider = createOpenAIReconstructionProvider({
apiKey: "test-key",
fetchImpl,
});
await provider.generateReconstruction(
"prompt",
"test-model",
reconstructionGenerationJsonSchema,
{ outputZodSchema: reconstructionGenerationSchema },
);
const request = JSON.parse(fetchImpl.mock.calls[0][1].body);
const reconstruction = request.text.format.schema.properties.reconstruction;
const unknown = reconstruction.properties.importantUnknowns.items;
const unexplained = reconstruction.properties.unexplainedTransitions.items;
expect(unknown.required).toContain("question");
expect(unknown.properties.question.minLength).toBe(1);
expect(unexplained.required).toContain("entity");
expect(unexplained.properties.entity.anyOf).toContainEqual({ type: "null" });
});
it("normalises generation-schema null placeholders before canonical validation", async () => {
const transportResponse = {
inputClassification: {
primaryType: "unexplained_change",
secondaryTypes: null,
reasoningModes: null,
classificationReason: "A reported change requires reconstruction.",
confidence: "high",
},
reconstruction: {
summary: "A change occurred and its cause remains uncertain.",
actors: [],
systemsOrObjects: [],
expectedStates: [],
observedStates: [],
differences: [],
knownTransitions: [],
unexplainedTransitions: [
{
id: "transition-1",
description: "The service level declined after the release.",
confidence: "medium",
entity: null,
previousState: null,
currentState: null,
},
],
contradictions: [],
importantUnknowns: [
{
id: "unknown-1",
description: "Whether the release caused the service-level decline",
question: "Did the release cause the service-level decline?",
confidence: "medium",
},
],
plausibleInterpretations: [
{
id: "interpretation-1",
description: "The release may have reduced service capacity.",
supportingEvidenceIds: ["evidence-1"],
assumptionsRequired: null,
confidence: "low",
},
],
relationships: null,
},
evidence: [
{
id: "evidence-1",
description: "The decline was reported after the release.",
evidenceType: "reported_statement",
source: null,
attribution: null,
confidence: "medium",
importance: "important",
},
],
nextQuestion: {
id: "next-1",
question: "What changed in service capacity after the release?",
targets: ["unknown-1"],
reason: "The cause of the decline remains unresolved.",
expectedInformationValue: "high",
reasoningMode: null,
},
};
const fetchImpl = vi.fn().mockResolvedValue({
ok: true,
json: async () => ({ output_text: JSON.stringify(transportResponse) }),
});
const provider = createOpenAIReconstructionProvider({
apiKey: "test-key",
fetchImpl,
});
const result = await provider.generateReconstruction(
"prompt",
"test-model",
reconstructionGenerationJsonSchema,
{ outputZodSchema: reconstructionGenerationSchema },
);
const request = JSON.parse(fetchImpl.mock.calls[0][1].body);
const unexplainedSchema =
request.text.format.schema.properties.reconstruction.properties
.unexplainedTransitions.items;
expect(unexplainedSchema.required).toContain("entity");
expect(unexplainedSchema.properties.entity.anyOf).toContainEqual({ type: "null" });
expect(result.response.inputClassification).not.toHaveProperty("secondaryTypes");
expect(result.response.inputClassification).not.toHaveProperty("reasoningModes");
expect(result.response.reconstruction.unexplainedTransitions[0]).not.toHaveProperty("entity");
expect(result.response.reconstruction.unexplainedTransitions[0]).not.toHaveProperty("previousState");
expect(result.response.reconstruction.unexplainedTransitions[0]).not.toHaveProperty("currentState");
expect(result.response.reconstruction.plausibleInterpretations[0]).not.toHaveProperty("assumptionsRequired");
expect(result.response.reconstruction).not.toHaveProperty("relationships");
expect(result.response.evidence[0]).not.toHaveProperty("source");
expect(result.response.evidence[0].attribution).toBeNull();
expect(result.response.nextQuestion).not.toHaveProperty("reasoningMode");
expect(reconstructionGenerationSchema.safeParse(result.response).success).toBe(true);
expect(reconstructionV2Schema.safeParse(result.response).success).toBe(true);
});
it("removes only optional transport null placeholders", () => {
const schema = z.object({
optionalText: z.string().optional(),
@@ -393,6 +693,29 @@ describe("OpenAI reconstruction provider experiment seam", () => {
assertStrictObjectInvariant(reparsedSchema);
});
it("forwards caller cancellation to the OpenAI response request", async () => {
const controller = new AbortController();
const fetchSpy = vi.fn().mockImplementation((_, options) => new Promise((_, reject) => {
options.signal.addEventListener("abort", () => {
const error = new Error("aborted");
error.name = "AbortError";
reject(error);
}, { once: true });
}));
const provider = createOpenAIReconstructionProvider({ apiKey: "test-key", fetchImpl: fetchSpy });
const request = provider.generateReconstruction(
"focused prompt",
"gpt-5.6-terra",
focusedDeconstructJsonSchema,
{ signal: controller.signal },
);
controller.abort();
await expect(request).rejects.toMatchObject({ name: "AbortError" });
expect(fetchSpy.mock.calls[0][1].signal).toBe(controller.signal);
});
it("emits one safe final-schema trace only when explicitly enabled", async () => {
const previousTraceFlag = process.env.CONFIDENCE_ENGINE_EXPERIMENT_TRACE_OPENAI_SCHEMA;
const infoSpy = vi.spyOn(console, "info").mockImplementation(() => {});
+110 -4
View File
@@ -1,5 +1,6 @@
import { beforeEach, describe, expect, it, vi } from "vitest";
import { normaliseAnalysisResponse } from "@/lib/reconstruction/compatibility.js";
import { reconstructionGenerationJsonSchema } from "@/lib/reconstruction/schema.js";
const mockGenerateReconstruction = vi.fn();
@@ -17,6 +18,7 @@ vi.mock("@/lib/llm/provider.js", () => ({
getProvider: () => ({
generateReconstruction: (...args) => mockGenerateReconstruction(...args),
}),
getProviderModelName: () => "test-model",
}));
vi.mock("@/lib/reconstruction/prompt.js", () => ({
@@ -134,13 +136,117 @@ describe("analyseScenario compatibility", () => {
});
expect(result.success).toBe(true);
expect(injectedProvider.generateReconstruction).toHaveBeenCalledWith(
"prompt",
"experiment-model",
);
expect(injectedProvider.generateReconstruction).toHaveBeenCalledTimes(1);
const [prompt, modelName, outputSchema, options] =
injectedProvider.generateReconstruction.mock.calls[0];
expect(prompt).toBe("prompt");
expect(modelName).toBe("experiment-model");
expect(outputSchema).toEqual(reconstructionGenerationJsonSchema);
expect(options.outputZodSchema.safeParse).toBeTypeOf("function");
expect(
options.outputZodSchema.safeParse({
inputClassification: {
primaryType: "other",
secondaryTypes: [],
reasoningModes: [],
classificationReason: "reason",
confidence: "medium",
},
reconstruction: {
summary: "summary",
actors: [],
systemsOrObjects: [],
expectedStates: [],
observedStates: [],
differences: [],
knownTransitions: [],
unexplainedTransitions: [],
contradictions: [],
importantUnknowns: [],
plausibleInterpretations: [],
relationships: [],
},
evidence: [],
nextQuestion: {
id: "q1",
question: "What next?",
targets: [],
reason: "reason",
expectedInformationValue: "low",
reasoningMode: "other",
},
}).success,
).toBe(true);
expect(mockGenerateReconstruction).not.toHaveBeenCalled();
});
it("preserves canonical reconstruction and reports invalid presentation nonfatally", async () => {
const response = {
inputClassification: {
primaryType: "other",
secondaryTypes: [],
reasoningModes: [],
classificationReason: "reason",
confidence: "medium",
},
reconstruction: {
summary: "summary",
actors: [],
systemsOrObjects: [],
expectedStates: [],
observedStates: [],
differences: [],
knownTransitions: [],
unexplainedTransitions: [],
contradictions: [],
importantUnknowns: [
{
id: "unknown-1",
description: "Whether workers can be recruited in time.",
question: "Whether workers can be recruited in time?",
confidence: "high",
},
],
plausibleInterpretations: [],
relationships: [],
},
evidence: [],
nextQuestion: {
id: "q1",
question: "What recruitment evidence exists?",
targets: ["unknown-1"],
reason: "reason",
expectedInformationValue: "high",
reasoningMode: "other",
},
};
const injectedProvider = {
generateReconstruction: vi.fn().mockResolvedValue(response),
};
const { analyseScenario } = await import("@/lib/analysis.js");
const result = await analyseScenario("Scenario text", {
reconstructionProvider: injectedProvider,
reconstructionModelName: "experiment-model",
});
expect(result.success).toBe(true);
expect(result.validationStatus).toBe("valid");
expect(result.presentationValidationStatus).toBe("partial");
expect(result.reconstruction.importantUnknowns[0]).toEqual({
id: "unknown-1",
description: "Whether workers can be recruited in time.",
confidence: "high",
});
expect(result.presentationValidationIssues).toEqual([
expect.objectContaining({
category: "question_indirect_proposition",
unknownId: "unknown-1",
severity: "nonfatal",
}),
]);
});
it("preserves an attempted provider API path on provider failure", async () => {
const providerError = new Error("Provider failed");
providerError.providerApiPath = "/api/chat";
@@ -0,0 +1,142 @@
import { describe, expect, it } from "vitest";
import { promises as fs } from "node:fs";
import { join } from "node:path";
import {
reconstructionGenerationJsonSchema,
reconstructionGenerationSchema,
reconstructionV2Schema,
} from "@/lib/reconstruction/schema.js";
import {
validateOpenQuestionCandidate,
validateReconstructionOpenQuestions,
} from "@/lib/reconstruction/open-question-presentation.js";
function makeAnalysisResponse(questionValue, includeQuestion = true) {
const unknown = {
id: "unknown-workforce",
description:
"Whether necessary skilled production workers can be recruited within the required project timeframe.",
confidence: "high",
};
if (includeQuestion) unknown.question = questionValue;
return {
inputClassification: {
primaryType: "other",
secondaryTypes: [],
reasoningModes: [],
classificationReason: "A delivery uncertainty is central to the situation.",
confidence: "high",
},
reconstruction: {
summary: "Delivery depends on recruiting skilled production workers.",
actors: [],
systemsOrObjects: [],
expectedStates: [],
observedStates: [],
differences: [],
knownTransitions: [],
unexplainedTransitions: [],
contradictions: [],
importantUnknowns: [unknown],
plausibleInterpretations: [],
relationships: [],
},
evidence: [],
nextQuestion: {
id: "next-1",
question: "What recruitment evidence is available?",
targets: ["unknown-workforce"],
reason: "Recruitment feasibility is unresolved.",
expectedInformationValue: "high",
reasoningMode: "other",
},
};
}
describe("new reconstruction generation contract", () => {
it("requires a non-empty question for every generated important unknown", () => {
expect(
reconstructionGenerationSchema.safeParse(
makeAnalysisResponse(undefined, false),
).success,
).toBe(false);
expect(
reconstructionGenerationSchema.safeParse(makeAnalysisResponse(" ")).success,
).toBe(false);
expect(
reconstructionGenerationSchema.safeParse(
makeAnalysisResponse(
"Can the necessary skilled production workers be recruited within the required project timeframe?",
),
).success,
).toBe(true);
const unknownSchema =
reconstructionGenerationJsonSchema.properties.reconstruction.properties
.importantUnknowns.items;
expect(unknownSchema.required).toContain("description");
expect(unknownSchema.required).toContain("question");
expect(unknownSchema.properties.question.minLength).toBe(1);
});
it("keeps the durable compatibility schema tolerant of a missing question", () => {
expect(
reconstructionV2Schema.safeParse(makeAnalysisResponse(undefined, false)).success,
).toBe(true);
});
});
describe("Open Question presentation validation", () => {
it("accepts a trimmed direct interrogative", () => {
expect(
validateOpenQuestionCandidate(
" Can the necessary workers be recruited in time? ",
),
).toEqual({
valid: true,
question: "Can the necessary workers be recruited in time?",
});
});
it("uses canonical UI fallback without a technical explanation", async () => {
const source = await fs.readFile(
join(process.cwd(), "components/reasoning-workspace.jsx"),
"utf8",
);
expect(source).toContain("const openQuestion = node.openQuestion || node.label;");
expect(source).not.toContain(
"Canonical uncertainty — direct question wording unavailable.",
);
expect(source).toContain("startFocused(n.id);");
});
it.each([
["question_missing", undefined, false],
["question_empty", " ", true],
["question_indirect_proposition", "Whether workers can be recruited in time", true],
["question_indirect_proposition", "Whether workers can be recruited in time?", true],
["question_not_interrogative", "Workers can be recruited in time.", true],
])("classifies %s nonfatally", (category, candidate, includeQuestion) => {
const input = makeAnalysisResponse(candidate, includeQuestion).reconstruction;
const canonicalBefore = input.importantUnknowns[0].description;
const idBefore = input.importantUnknowns[0].id;
const result = validateReconstructionOpenQuestions(input);
expect(result.reconstruction.importantUnknowns[0]).not.toHaveProperty("question");
expect(result.reconstruction.importantUnknowns[0].description).toBe(
canonicalBefore,
);
expect(result.reconstruction.importantUnknowns[0].id).toBe(idBefore);
expect(result.issues).toEqual([
expect.objectContaining({
category,
unknownId: idBefore,
severity: "nonfatal",
path: ["reconstruction", "importantUnknowns", 0, "question"],
}),
]);
});
});
@@ -11,7 +11,7 @@
import { describe, expect, it, beforeAll, afterAll, vi } from "vitest";
import { execFile } from "child_process";
import { writeFile, unlink } from "fs/promises";
import { readFile, writeFile, unlink } from "fs/promises";
import { join, dirname } from "path";
import { fileURLToPath } from "url";
import { createRequire } from "module";
@@ -89,16 +89,29 @@ function parseOutput(result) {
describe("start-case-experiment-helper.cjs apparatus", () => {
let tmpFile = "";
let metadataFile = "";
beforeAll(async () => {
// Create a temporary scenario fixture for file-input tests
tmpFile = join(rootDir, "tests", "fixtures", "_helper_scenario_temp.json");
await writeFile(tmpFile, JSON.stringify({ scenario: "Scenario from file" }), "utf-8");
metadataFile = join(rootDir, "tests", "fixtures", "_helper_semantic_scenario_temp.json");
await writeFile(
metadataFile,
JSON.stringify({
experimentId: "semantic-open-question-001",
scenarioId: "manufacturing-expansion-001",
scenario: "Scenario with semantic inspection metadata",
promptVersion: "v0.5",
}),
"utf-8",
);
});
afterAll(async () => {
// Clean up temp fixture
try { await unlink(tmpFile); } catch {}
try { await unlink(metadataFile); } catch {}
});
// ── A — Positional scenario input ───────────────────────────────
@@ -157,6 +170,141 @@ describe("start-case-experiment-helper.cjs apparatus", () => {
expect(result.exitCode).toBe(0);
expect(parseOutput(result).diagnostics.reconstructionOnly).toBe(true);
});
it("preserves experiment metadata and exposes semantic evidence", async () => {
const result = await runHelper(["--reconstruction-only", "--file", metadataFile]);
expect(result.exitCode).toBe(0);
const parsed = parseOutput(result);
expect(parsed.experimentId).toBe("semantic-open-question-001");
expect(parsed.scenarioId).toBe("manufacturing-expansion-001");
expect(parsed.execution).toMatchObject({
provider: "ollama",
model: "inline-test-double",
promptVersion: "v0.5",
});
expect(parsed.reconstruction.importantUnknowns).toEqual([
{
id: "unknown-1",
description: "Canonical uncertainty from the deterministic test double",
confidence: "medium",
question: "What would resolve the deterministic uncertainty?",
},
]);
expect(parsed.graphUnknownNodes).toEqual([
{
id: "graph-unknown-1",
label: "Canonical uncertainty",
description: "Canonical uncertainty from the deterministic test double",
openQuestion: "What would resolve the deterministic uncertainty?",
status: "unknown",
},
]);
expect(parsed.diagnostics).toMatchObject({
validationStatus: "mock",
presentationValidationStatus: "valid",
presentationValidationIssues: [],
graphReferenceValidation: { valid: true, errors: [] },
compatibilityApplied: false,
compatibilityChanges: [],
compatibilityWarnings: [],
});
});
it("passes only production-supported request fields and calls startCase once", async () => {
const { runStartCaseExperiment } = require(helperPath);
const importantUnknowns = [
{
id: "unknown-a",
description: "Canonical uncertainty A",
confidence: "high",
question: "What resolves uncertainty A?",
},
];
const startCase = vi.fn(async () => ({
success: true,
reconstruction: { importantUnknowns },
situationGraph: {
nodes: [
{
id: "unknown-node-a",
label: "Canonical uncertainty A",
description: "Canonical uncertainty A",
openQuestion: "What resolves uncertainty A?",
status: "unknown",
kind: "unknown",
},
{
id: "observation-node-a",
label: "Known observation",
description: "Known observation",
status: "supported",
kind: "observation",
},
],
edges: [],
},
selectedQuestion: null,
summary: "test",
diagnostics: {
modelName: "configured-model",
promptVersion: "v0.5",
validationStatus: "valid",
presentationValidationStatus: "partial",
presentationValidationIssues: [{ category: "test-diagnostic" }],
graphReferenceValidation: { valid: true, errors: [] },
compatibilityApplied: true,
compatibilityChanges: ["normalised"],
compatibilityWarnings: ["warning"],
},
}));
const result = await runStartCaseExperiment(
{
experimentId: "experiment-a",
scenarioId: "scenario-a",
scenario: "Scenario A",
promptVersion: "v0.5",
ignoredMetadata: "must not cross the production boundary",
},
null,
{ startCase, reconstructionOnly: true },
);
expect(startCase).toHaveBeenCalledTimes(1);
expect(startCase).toHaveBeenCalledWith(
{ scenario: "Scenario A", promptVersion: "v0.5" },
expect.objectContaining({ reconstructionOnly: true }),
);
expect(result.experimentId).toBe("experiment-a");
expect(result.scenarioId).toBe("scenario-a");
expect(result.reconstruction.importantUnknowns).toBe(importantUnknowns);
expect(result.graphUnknownNodes).toEqual([
{
id: "unknown-node-a",
label: "Canonical uncertainty A",
description: "Canonical uncertainty A",
openQuestion: "What resolves uncertainty A?",
status: "unknown",
},
]);
expect(result.execution).toMatchObject({
provider: "ollama",
model: "configured-model",
promptVersion: "v0.5",
});
expect(result.diagnostics).toEqual({
modelName: "configured-model",
promptVersion: "v0.5",
validationStatus: "valid",
presentationValidationStatus: "partial",
presentationValidationIssues: [{ category: "test-diagnostic" }],
graphReferenceValidation: { valid: true, errors: [] },
compatibilityApplied: true,
compatibilityChanges: ["normalised"],
compatibilityWarnings: ["warning"],
});
});
});
// ── C — Environment loading ─────────────────────────────────────
@@ -295,6 +443,19 @@ describe("start-case-experiment-helper.cjs apparatus", () => {
const parsed = parseOutput(result);
expect(parsed.error).toBeNull();
});
it("preserves the existing non-reconstruction-only output contract", async () => {
const result = await runHelper(["Compatibility scenario"]);
const parsed = parseOutput(result);
expect(parsed).not.toHaveProperty("experimentId");
expect(parsed).not.toHaveProperty("scenarioId");
expect(parsed).not.toHaveProperty("execution");
expect(parsed).not.toHaveProperty("graphUnknownNodes");
expect(parsed).toHaveProperty("situationGraphNodeCount");
expect(parsed).toHaveProperty("situationGraphEdgeCount");
expect(parsed).toHaveProperty("endToEndElapsedMs");
});
});
// ── E — Failure output (startCase throws) ───────────────────────
@@ -399,6 +560,11 @@ describe("start-case-experiment-helper.cjs apparatus", () => {
expect(typeof parsed.error).toBe("string");
}
});
it("contains no investigation-persistence invocation", async () => {
const source = await readFile(helperPath, "utf-8");
expect(source).not.toMatch(/investigation-storage|server-investigation-persistence|saveInvestigation|\/api\/investigations/);
});
});
// ── H — Import-only mode (production seam verification) ────────
@@ -488,4 +654,698 @@ describe("start-case-experiment-helper.cjs apparatus", () => {
expect(typeof parsed.failureReason).toBe("string");
});
});
// ── I — Condition B schema variant apparatus ────────────────────
describe("I — Condition B schema variant apparatus", () => {
it("--condition B passes generationSchema through the startCase seam", async () => {
const { runStartCaseExperiment } = require(helperPath);
let receivedDependencies;
const result = await runStartCaseExperiment(
{ scenario: "Condition B test scenario" },
null,
{
condition: "B",
startCase: async (_input, dependencies) => {
receivedDependencies = dependencies;
return {
success: true,
situationGraph: { nodes: [], edges: [] },
assessment: null,
selectedQuestion: null,
summary: "test",
diagnostics: { validationStatus: "valid" },
};
},
},
);
expect(result.success).toBe(true);
expect(receivedDependencies.generationSchema).toBeDefined();
// Condition B schema: importantUnknowns items should NOT have question property
const iuSchema = receivedDependencies.generationSchema?.properties?.reconstruction?.properties?.importantUnknowns?.items;
if (iuSchema && iuSchema.properties) {
expect(iuSchema.properties).not.toHaveProperty("question");
}
});
it("--condition A uses production schema with question required", async () => {
const { runStartCaseExperiment } = require(helperPath);
let receivedDependencies;
const result = await runStartCaseExperiment(
{ scenario: "Condition A test scenario" },
null,
{
condition: "A",
startCase: async (_input, dependencies) => {
receivedDependencies = dependencies;
return {
success: true,
situationGraph: { nodes: [], edges: [] },
assessment: null,
selectedQuestion: null,
summary: "test",
diagnostics: { validationStatus: "valid" },
};
},
},
);
expect(result.success).toBe(true);
});
it("--condition B with mock produces valid structured output", async () => {
const result = await runHelper(["--condition", "B", "Mock scenario"], {});
expect(result.exitCode).toBe(0);
const parsed = parseOutput(result);
expect(parsed.success).toBe(true);
});
it("Condition B schema JSON has importantUnknowns WITHOUT question field", async () => {
const { conditionBJsonSchema, conditionAJsonSchema } = require(
join(rootDir, "lib", "reconstruction", "schema-variant.js"),
);
const bIuProps = conditionBJsonSchema?.properties?.reconstruction?.properties?.importantUnknowns?.items?.properties;
const aIuProps = conditionAJsonSchema?.properties?.reconstruction?.properties?.importantUnknowns?.items?.properties;
expect(bIuProps).toHaveProperty("id");
expect(bIuProps).toHaveProperty("description");
expect(bIuProps).toHaveProperty("confidence");
expect(bIuProps).not.toHaveProperty("question");
// Verify Condition A (production) still HAS question
expect(aIuProps).toHaveProperty("id");
expect(aIuProps).toHaveProperty("description");
expect(aIuProps).toHaveProperty("confidence");
expect(aIuProps).toHaveProperty("question");
});
it("Condition B prompt file exists and contains no question-generation language", async () => {
const fs = require("fs");
const altPromptPath = join(rootDir, "prompts", "reconstruct-v0.5-b-noreq.md");
expect(fs.existsSync(altPromptPath)).toBe(true);
const content = fs.readFileSync(altPromptPath, "utf8");
// The B prompt should NOT instruct model to generate questions for importantUnknowns
expect(content).not.toMatch(/express that same uncertainty in \`question\`/i);
expect(content).not.toMatch(/natural direct question/i);
// But Condition A prompt does contain this language
const prodPromptPath = join(rootDir, "prompts", "reconstruct-v0.5.md");
const prodContent = fs.readFileSync(prodPromptPath, "utf8");
expect(prodContent).toMatch(/express that same uncertainty in \`question\`/i);
});
it("Condition B prompt has identical line count to Condition A prompt", async () => {
const fs = require("fs");
const altLines = fs.readFileSync(join(rootDir, "prompts", "reconstruct-v0.5-b-noreq.md"), "utf8").split("\n").length;
const prodLines = fs.readFileSync(join(rootDir, "prompts", "reconstruct-v0.5.md"), "utf8").split("\n").length;
expect(altLines).toBe(prodLines);
});
it("Condition B prompt differs from Condition A only in Rule 7a and output format example", async () => {
const fs = require("fs");
const altContent = fs.readFileSync(join(rootDir, "prompts", "reconstruct-v0.5-b-noreq.md"), "utf8");
const prodContent = fs.readFileSync(join(rootDir, "prompts", "reconstruct-v0.5.md"), "utf8");
expect(altContent).not.toBe(prodContent);
expect(altContent).toMatch(/express its canonical uncertainty meaning as a description in \`description\` with strict semantic fidelity/i);
});
it("Condition B schema validates importantUnknowns WITHOUT question field required", async () => {
const { conditionBJsonSchema } = require(
join(rootDir, "lib", "reconstruction", "schema-variant.js"),
);
const bIuItems = conditionBJsonSchema?.properties?.reconstruction?.properties?.importantUnknowns?.items;
expect(bIuItems).toBeDefined();
if (bIuItems && bIuItems.required) {
expect(bIuItems.required).not.toContain("question");
}
});
});
// ── J — Condition D treatment selector apparatus ────────────────
describe("J — Condition D prompt selection", () => {
it("default Condition D (no --treatment) selects original deconstruct prompt", async () => {
const { runStartCaseExperiment } = require(helperPath);
let receivedScenario;
await runStartCaseExperiment(
{ scenario: "Condition D default test" },
null,
{
condition: "D",
startCase: async (input) => {
receivedScenario = input.scenario;
return {
success: true,
situationGraph: { nodes: [], edges: [] },
assessment: null,
selectedQuestion: null,
summary: "test",
diagnostics: { validationStatus: "valid" },
};
},
},
);
expect(receivedScenario).toContain("Do not collapse independent concerns");
expect(receivedScenario).toContain("Do not create artificial uncertainties to increase their number.");
});
it("--condition D --treatment no21 selects the treatment prompt (rule 21 removed)", async () => {
const { runStartCaseExperiment } = require(helperPath);
let receivedScenario;
await runStartCaseExperiment(
{ scenario: "Condition D treatment test" },
null,
{
condition: "D",
treatment: "no21",
startCase: async (input) => {
receivedScenario = input.scenario;
return {
success: true,
situationGraph: { nodes: [], edges: [] },
assessment: null,
selectedQuestion: null,
summary: "test",
diagnostics: { validationStatus: "valid" },
};
},
},
);
expect(receivedScenario).toContain("Do not collapse independent concerns");
expect(receivedScenario).not.toContain("Do not create artificial uncertainties to increase their number.");
});
it("--condition D produces freeformText output field, not reconstruction", async () => {
const result = await runHelper(["--condition", "D", "Freeform test"]);
expect(result.exitCode).toBe(0);
const parsed = parseOutput(result);
expect(parsed).toHaveProperty("freeformText");
expect(parsed.conditionType).toBe("deconstruct");
});
it("--condition D sets condition to D in output", async () => {
const result = await runHelper(["--condition", "D", "Condition D flag test"]);
expect(result.exitCode).toBe(0);
const parsed = parseOutput(result);
expect(parsed.condition).toBe("D");
});
});
describe("J.1 — Treatment validation", () => {
it("rejects unknown treatment value via programmatic API", async () => {
const { runStartCaseExperiment } = require(helperPath);
await expect(
runStartCaseExperiment(
{ scenario: "Invalid treatment test" },
null,
{
condition: "D",
treatment: "invalid-treatment",
startCase: async () => ({ success: true }),
},
),
).rejects.toThrow("Invalid treatment: invalid-treatment. Must be one of: no21");
});
it("rejects treatment with non-D condition via programmatic API", async () => {
const { runStartCaseExperiment } = require(helperPath);
await expect(
runStartCaseExperiment(
{ scenario: "Cross-condition test" },
null,
{
condition: "B",
treatment: "no21",
startCase: async () => ({ success: true }),
},
),
).rejects.toThrow('Treatment "no21" is only valid with --condition D or E, not --condition B');
});
it("rejects unknown condition value via CLI", async () => {
const result = await runHelper(["--condition", "Z", "Invalid condition test"]);
expect(result.exitCode).not.toBe(0);
const parsed = parseOutput(result);
if (parsed) {
expect(parsed.success).toBe(false);
expect(typeof parsed.error).toBe("string");
}
});
it("rejects treatment with non-D condition via CLI", async () => {
const result = await runHelper(["--condition", "B", "--treatment", "no21", "Cross-condition CLI test"]);
expect(result.exitCode).not.toBe(0);
const parsed = parseOutput(result);
if (parsed) {
expect(parsed.success).toBe(false);
expect(typeof parsed.error).toBe("string");
}
});
});
describe("J.2 — Prompt isolation verification", () => {
it("original and treatment prompts differ only by removal of rule 21", async () => {
const fs = require("fs");
const path = require("path");
const originalPrompt = fs.readFileSync(join(rootDir, "prompts", "reconstruct-v0.5-d-deconstruct.md"), "utf8");
const treatmentPrompt = fs.readFileSync(join(rootDir, "prompts", "reconstruct-v0.5-d-deconstruct-no21.md"), "utf8");
expect(originalPrompt).not.toBe(treatmentPrompt);
// The only difference should be the removal of rule 21 line and its blank line
const originalLines = originalPrompt.split("\n");
const treatmentLines = treatmentPrompt.split("\n");
// Find where they diverge — rule 21 is on line 21
expect(originalLines[20]).toBe("Do not create artificial uncertainties to increase their number.");
// After removal, everything else should shift by exactly one line
for (let i = 21; i < treatmentLines.length; i++) {
expect(originalLines[i + 1]).toBe(treatmentLines[i]);
}
});
it("both prompts share identical preamble through rule 20", async () => {
const fs = require("fs");
const originalPrompt = fs.readFileSync(join(rootDir, "prompts", "reconstruct-v0.5-d-deconstruct.md"), "utf8");
const treatmentPrompt = fs.readFileSync(join(rootDir, "prompts", "reconstruct-v0.5-d-deconstruct-no21.md"), "utf8");
// First 20 lines should be identical (before rule 21)
const originalPreamble = originalPrompt.split("\n").slice(0, 20).join("\n");
const treatmentPreamble = treatmentPrompt.split("\n").slice(0, 20).join("\n");
expect(originalPreamble).toBe(treatmentPreamble);
});
it("original prompt still contains rule 21 verbatim", async () => {
const fs = require("fs");
const originalPrompt = fs.readFileSync(join(rootDir, "prompts", "reconstruct-v0.5-d-deconstruct.md"), "utf8");
expect(originalPrompt).toContain("Do not create artificial uncertainties to increase their number.");
});
it("treatment prompt does NOT contain rule 21", async () => {
const fs = require("fs");
const treatmentPrompt = fs.readFileSync(join(rootDir, "prompts", "reconstruct-v0.5-d-deconstruct-no21.md"), "utf8");
expect(treatmentPrompt).not.toContain("Do not create artificial uncertainties to increase their number.");
});
it("Condition D output cannot overwrite original Condition D evidence — distinct freeformText field", async () => {
const resultTreatment = await runHelper(["--condition", "D", "--treatment", "no21", "Treatment output test"]);
const resultOriginal = await runHelper(["--condition", "D", "Original Condition D test"]);
expect(resultTreatment.exitCode).toBe(0);
expect(resultOriginal.exitCode).toBe(0);
const treatmentOutput = parseOutput(resultTreatment);
const originalOutput = parseOutput(resultOriginal);
// Both should have freeformText (not reconstruction) for Condition D
expect(treatmentOutput).toHaveProperty("freeformText");
expect(originalOutput).toHaveProperty("freeformText");
expect(treatmentOutput).not.toHaveProperty("reconstruction");
expect(originalOutput).not.toHaveProperty("reconstruction");
// They use the same output structure (freeformText) but different content —
// the test double returns deterministic text that confirms prompt selection works
expect(typeof treatmentOutput.freeformText).toBe("string");
expect(typeof originalOutput.freeformText).toBe("string");
});
});
// ── K — Condition E investigability-first apparatus ────────
describe("K — Condition E investigability-first apparatus", () => {
it("--condition E selects the investigability-first prompt rather than deconstruct", async () => {
const { runStartCaseExperiment } = require(helperPath);
let receivedScenario;
await runStartCaseExperiment(
{ scenario: "Condition E test" },
null,
{
condition: "E",
startCase: async (input) => {
receivedScenario = input.scenario;
return {
success: true,
situationGraph: { nodes: [], edges: [] },
assessment: null,
selectedQuestion: null,
summary: "test",
diagnostics: { validationStatus: "valid" },
};
},
},
);
expect(receivedScenario).toContain("You are a neutral analyst performing deconstruction of the user's situation");
expect(receivedScenario).toContain("Do not create artificial uncertainties to increase their number.");
});
it("--condition E --treatment e-investigability-first is accepted", async () => {
const { runStartCaseExperiment } = require(helperPath);
await runStartCaseExperiment(
{ scenario: "Condition E treatment test" },
null,
{
condition: "E",
treatment: "e-investigability-first",
startCase: async (input) => {
return {
success: true,
situationGraph: { nodes: [], edges: [] },
assessment: null,
selectedQuestion: null,
summary: "test",
diagnostics: { validationStatus: "valid" },
};
},
},
);
// If we reach here without an error, the treatment was accepted
});
it("--condition E produces freeformText output field via CLI mock", async () => {
const result = await runHelper(["--condition", "E", "Freeform test"]);
expect(result.exitCode).toBe(0);
const parsed = parseOutput(result);
expect(parsed).toHaveProperty("freeformText");
expect(typeof parsed.freeformText).toBe("string");
expect(parsed.conditionType).toBe("investigability-first");
});
it("--condition E sets condition to E in output", async () => {
const result = await runHelper(["--condition", "E", "Condition E flag test"]);
expect(result.exitCode).toBe(0);
const parsed = parseOutput(result);
expect(parsed.condition).toBe("E");
});
it("--condition E does not return reconstruction field in output", async () => {
const result = await runHelper(["--condition", "E", "No reconstruction test"]);
expect(result.exitCode).toBe(0);
const parsed = parseOutput(result);
expect(parsed).not.toHaveProperty("reconstruction");
});
it("--condition E --reconstruction-only sets execution.condition to E", async () => {
const result = await runHelper(["--condition", "E", "--reconstruction-only", "Reconstruction only test"]);
expect(result.exitCode).toBe(0);
const parsed = parseOutput(result);
expect(parsed.execution).toBeDefined();
expect(parsed.execution.condition).toBe("E");
});
it("--condition E rejects unknown treatment via programmatic API", async () => {
const { runStartCaseExperiment } = require(helperPath);
await expect(
runStartCaseExperiment(
{ scenario: "Invalid treatment test" },
null,
{
condition: "E",
treatment: "invalid-treatment",
startCase: async () => ({ success: true }),
},
),
).rejects.toThrow("Invalid treatment: invalid-treatment. Must be one of: e-investigability-first");
});
it("--condition E rejects treatment with non-E condition via programmatic API", async () => {
const { runStartCaseExperiment } = require(helperPath);
await expect(
runStartCaseExperiment(
{ scenario: "Cross-condition test" },
null,
{
condition: "B",
treatment: "e-investigability-first",
startCase: async () => ({ success: true }),
},
),
).rejects.toThrow('Treatment "e-investigability-first" is only valid with --condition D or E, not --condition B');
});
it("--condition E exposes non-empty freeformText (mock)", async () => {
const result = await runHelper(["--condition", "E", "Freeform content test"]);
expect(result.exitCode).toBe(0);
const parsed = parseOutput(result);
expect(parsed.freeformText.length).toBeGreaterThan(0);
});
});
});
// ── Evidence persistence wrapper tests ─────────────────────────────
var wrapperPath = join(rootDir, "scripts", "evidence-persist.cjs");
function runWrapper(args, envOverrides) {
return new Promise(function(resolve) {
execFile(
process.execPath,
[wrapperPath].concat(args),
{
cwd: rootDir,
timeout: 30000,
env: Object.assign({}, process.env, {
START_CASE_EXPERIMENT_HELPER_MOCK: "1",
}, envOverrides || {}),
},
function(error, stdout, stderr) {
resolve({
error: !!error,
exitCode: error ? error.code : 0,
stdout: stdout || "",
stderr: stderr || "",
});
}
);
});
}
/**
* Tests for the evidence-persist.cjs wrapper.
*
* Uses START_CASE_EXPERIMENT_HELPER_MOCK=1 for deterministic behavior.
*/
describe("evidence persistence", function() {
it("successful mock invocation publishes valid evidence with exact raw stdout bytes preserved", async function() {
var testOutput = join(rootDir, ".evidence-temp", "headless-semantic", "_test_persist_d.json");
var fixturePath = join(rootDir, "tests", "fixtures", "evidence-persist-input.json");
try {
var result = await runWrapper([
"--condition", "D",
"--file", fixturePath,
"--output", testOutput
]);
expect(result.exitCode).toBe(0);
expect(result.error).toBe(false);
// Verify the evidence file was published
var statSync = require("fs").statSync;
var exists = false;
try { statSync(testOutput); exists = true; } catch(e) {}
expect(exists).toBe(true);
// Verify the published file is valid JSON matching what helper outputs
var rawContent = await readFile(testOutput, "utf-8");
var published = JSON.parse(rawContent.trim());
expect(published.success).toBe(true);
expect(published.statusCode).toBe(200);
expect(published.condition).toBe("D");
expect(typeof published.freeformText).toBe("string");
expect(published.freeformText.length).toBeGreaterThan(0);
expect(published.diagnostics.reconstructionOnly).toBe(true);
// Verify raw bytes are preserved (no reformatting)
var reparseable = JSON.parse(rawContent.trim());
expect(reparseable.condition).toBe("D");
} finally {
try { await unlink(testOutput); } catch(e2) {}
}
});
it("forced helper failure does not publish evidence", async function() {
var testOutput = join(rootDir, ".evidence-temp", "headless-semantic", "_test_persist_fail.json");
var fixturePath = join(rootDir, "tests", "fixtures", "evidence-persist-input.json");
try {
var result = await runWrapper([
"--condition", "D",
"--file", fixturePath,
"--output", testOutput
], { START_CASE_EXPERIMENT_HELPER_FORCE_FAIL: "1" });
expect(result.exitCode).not.toBe(0);
// Verify the final destination was NOT created
var statSync = require("fs").statSync;
var exists = false;
try { statSync(testOutput); exists = true; } catch(e) {}
expect(exists).toBe(false);
} finally {
try { await unlink(testOutput); } catch(e2) {}
}
});
it("malformed or incomplete output is rejected without publication", async function() {
var testOutput = join(rootDir, ".evidence-temp", "headless-semantic", "_test_persist_malformed.json");
var fixturePath = join(rootDir, "tests", "fixtures", "evidence-persist-input.json");
try {
var result = await runWrapper([
"--condition", "D",
"--file", fixturePath,
"--output", testOutput
], { START_CASE_EXPERIMENT_HELPER_FORCE_FAIL: "1" });
expect(result.exitCode).not.toBe(0);
// Verify destination was NOT created (rejection without publication)
var statSync2 = require("fs").statSync;
var exists = false;
try { statSync2(testOutput); exists = true; } catch(e) {}
expect(exists).toBe(false);
} finally {
try { await unlink(testOutput); } catch(e2) {}
}
});
it("an existing destination is preserved unchanged", async function() {
var testOutput = join(rootDir, ".evidence-temp", "headless-semantic", "_test_persist_existing.json");
var fixturePath = join(rootDir, "tests", "fixtures", "evidence-persist-input.json");
try {
// Pre-create the destination with known content
await writeFile(testOutput, JSON.stringify({ preserved: true }), "utf-8");
var original = JSON.parse((await readFile(testOutput, "utf-8")).trim());
expect(original.preserved).toBe(true);
// Attempt to publish — should be rejected due to existing destination
var result = await runWrapper([
"--condition", "D",
"--file", fixturePath,
"--output", testOutput
]);
expect(result.exitCode).not.toBe(0);
// Verify the original content was NOT modified
var final = JSON.parse((await readFile(testOutput, "utf-8")).trim());
expect(final.preserved).toBe(true);
} finally {
try { await unlink(testOutput); } catch(e2) {}
}
});
it("two competing publication attempts cannot both claim the same destination", async function() {
var testOutput = join(rootDir, ".evidence-temp", "headless-semantic", "_test_persist_competing.json");
var fixturePath = join(rootDir, "tests", "fixtures", "evidence-persist-input.json");
try {
// First publication — succeeds
var result1 = await runWrapper([
"--condition", "D",
"--file", fixturePath,
"--output", testOutput
]);
expect(result1.exitCode).toBe(0);
expect(result1.error).toBe(false);
// Verify winning evidence bytes
var rawBytes1 = await readFile(testOutput, "utf-8");
var evidence1 = JSON.parse(rawBytes1.trim());
expect(evidence1.success).toBe(true);
expect(evidence1.statusCode).toBe(200);
expect(evidence1.condition).toBe("D");
// Simulate a competing claim: write different content to the same destination
var competingContent = JSON.stringify({ overwrittenBy: "competing-claim" });
await writeFile(testOutput, competingContent, "utf-8");
// Second publication to the same path — must fail (EEXIST)
var result2 = await runWrapper([
"--condition", "D",
"--file", fixturePath,
"--output", testOutput
]);
expect(result2.exitCode).not.toBe(0);
// Verify winning evidence bytes remain unchanged:
// file still has the competing claim content (second wrapper did NOT overwrite it)
var rawBytes2 = await readFile(testOutput, "utf-8");
expect(rawBytes2).toBe(competingContent);
// Also verify first publication bytes are intact — they differ from current file,
// confirming the second wrapper could not reach them
expect(rawBytes2).not.toBe(rawBytes1);
} finally {
try { await unlink(testOutput); } catch(e2) {}
}
});
it("linkSync hardlink publication rejects EEXIST when a second process targets an already-published destination", async function() {
var testOutput = join(rootDir, ".evidence-temp", "headless-semantic", "_test_persist_link_a.json");
var fixturePath = join(rootDir, "tests", "fixtures", "evidence-persist-input.json");
try {
// First wrapper run — publishes successfully (linkSync creates hardlink at testOutput)
var resultA = await runWrapper([
"--condition", "D",
"--file", fixturePath,
"--output", testOutput
]);
expect(resultA.exitCode).toBe(0);
// Verify the first publication produced valid evidence
var publishedA = JSON.parse((await readFile(testOutput, "utf-8")).trim());
expect(publishedA.success).toBe(true);
expect(publishedA.statusCode).toBe(200);
expect(publishedA.condition).toBe("D");
expect(typeof publishedA.freeformText).toBe("string");
expect(publishedA.diagnostics.reconstructionOnly).toBe(true);
// Second wrapper targets the exact same destination path.
// linkSync(tmpFile, testOutput) must throw EEXIST because testOutput already exists
// as a hardlink to the first publication's data (same inode).
var resultB = await runWrapper([
"--condition", "D",
"--file", fixturePath,
"--output", testOutput
]);
expect(resultB.exitCode).not.toBe(0);
// Verify the first publication's evidence bytes remain exactly as published:
// no-write guarantee confirmed — linkSync refuses to modify existing destination.
var preserved = JSON.parse((await readFile(testOutput, "utf-8")).trim());
expect(preserved.success).toBe(true);
expect(preserved.statusCode).toBe(200);
expect(preserved.condition).toBe("D");
} finally {
try { await unlink(testOutput); } catch(e) {}
}
});
});
+206 -166
View File
@@ -1,39 +1,98 @@
import { describe, expect, it, beforeEach, afterEach, vi } from "vitest";
import React from "react";
import { render, screen, fireEvent } from "@testing-library/react";
import "@testing-library/jest-dom";
import { deriveInvestigationLearning } from "@/lib/graph/episode-preparation";
// ---------------------------------------------------------------------------
// Minimal isolated mocks for Report page only.
// No Portfolio, no restart behaviour, no storage provider, no module-cache hacks.
// ---------------------------------------------------------------------------
/* ── Three-part report: unresolved question derivation (seam-level) ──────────── */
function deriveUnresolvedQuestions(situationGraph, resolvedNodeIds) {
const resolvedIds = new Set(resolvedNodeIds ?? []);
return (situationGraph?.nodes ?? [])
.filter((n) => n.kind === "unknown" && n.status !== "resolved" && !resolvedIds.has(n.id))
.map((n) => n.label || n.description);
}
describe("Three-part report — unresolved question derivation", () => {
it("Case A: unresolved questions exist and are returned in canonical order", () => {
const result = deriveUnresolvedQuestions({ nodes: [
{ id: "u1", kind: "unknown", status: "unclear", label: "What is the revenue model?" },
{ id: "u2", kind: "unknown", status: "provisional", label: "Who is the primary customer?" },
] }, []);
expect(result).toHaveLength(2);
expect(result[0]).toBe("What is the revenue model?");
expect(result[1]).toBe("Who is the primary customer?");
});
it("Case A: unresolved text does NOT appear in supported-understanding content", () => {
const result = deriveUnresolvedQuestions({ nodes: [
{ id: "u1", kind: "unknown", status: "unclear", label: "What is the revenue model?" },
] }, []);
expect(result).toHaveLength(1);
expect(result).not.toContain("");
});
it("Case B: resolved/clarified question is excluded from unresolved", () => {
const result = deriveUnresolvedQuestions({ nodes: [
{ id: "u1", kind: "unknown", status: "unclear", label: "What is the revenue model?" },
{ id: "u2", kind: "unknown", status: "resolved", label: "Resolved question" },
] }, []);
expect(result).toHaveLength(1);
expect(result[0]).toBe("What is the revenue model?");
});
it("Case B: resolved node ID excluded via resolvedNodeIds", () => {
const result = deriveUnresolvedQuestions({ nodes: [
{ id: "u1", kind: "unknown", status: "unclear", label: "Unresolved question" },
{ id: "u2", kind: "unknown", status: "unclear", label: "Also unresolved" },
] }, ["u1"]);
expect(result).toHaveLength(1);
expect(result[0]).toBe("Also unresolved");
});
it("Case C: zero unresolved questions yields empty array", () => {
const result = deriveUnresolvedQuestions({ nodes: [
{ id: "n1", kind: "observation", status: "known" },
{ id: "u1", kind: "unknown", status: "resolved" },
] }, []);
expect(result).toHaveLength(0);
});
it("Case C: no nodes yields empty array (no crash)", () => {
const result = deriveUnresolvedQuestions(null, []);
expect(result).toEqual([]);
});
it("Epistemic separation: assumption-kind nodes are excluded", () => {
const result = deriveUnresolvedQuestions({ nodes: [
{ id: "u1", kind: "unknown", status: "unclear", label: "Open question" },
{ id: "a1", kind: "assumption", status: "plausible", label: "Assumption" },
] }, []);
expect(result).toHaveLength(1);
expect(result[0]).toBe("Open question");
});
it("Epistemic separation: no readiness/completion language in derived output", () => {
const result = deriveUnresolvedQuestions({ nodes: [
{ id: "u1", kind: "unknown", status: "unclear", label: "What is the revenue model?" },
] }, []);
expect(result[0]).not.toMatch(/ready|complete|enough|sufficient|must resolve|critical|priority|blocker/);
});
});
/* ── v0.60h-a — seam-level mocks (no @testing-library/react) ─────────── */
let mockLastLoadedId = undefined;
let mockLastSavedSnapshot = null;
function setMockSnapshot(snap) {
if (snap) {
localStorage.setItem(
"confidence-engine-investigation",
JSON.stringify(snap),
);
mockLoadResult = snap;
} else {
localStorage.removeItem("confidence-engine-investigation");
mockLoadResult = null;
}
}
let mockLoadResult = null;
vi.mock("@/lib/storage/investigation-storage", () => ({
listInvestigations: () => [],
loadInvestigation: (id) => {
mockLastLoadedId = id;
return mockLoadResult;
return Promise.resolve(mockLoadResult);
},
saveInvestigation: (snapshot) => {
mockLastSavedSnapshot = snapshot;
return Promise.resolve(snapshot);
},
clearInvestigation: () => {},
}));
@@ -67,182 +126,163 @@ async function cleanup() {
localStorage.removeItem("confidence-engine-investigation");
}
/* ── v0.60h-a — Report identity and lifecycle isolation ─────────────────── */
describe("Report page identity and lifecycle — v0.60h-a", () => {
let ReportPage;
beforeEach(async () => {
mockLastLoadedId = undefined;
mockLastSavedSnapshot = null;
setMockSnapshot(null);
describe("Report page identity and lifecycle — v0.60h-a (seam-level)", () => {
beforeEach(() => {
mockLastLoadedId = undefined; mockLastSavedSnapshot = null; mockLoadResult = null;
localStorage.removeItem("confidence-engine-investigation");
global.fetch = vi.fn();
const mod = await import("@/app/investigations/[id]/report/page.jsx");
ReportPage = mod.default;
});
afterEach(async () => { await cleanup(); });
afterEach(async () => {
cleanup();
});
it("A — loadInvestigation invoked with route ID and overview POST issued", async () => {
const snap = makeSnapshot({ id: "inv-report-a", investigationReport: null, investigationRevision: 4, findings: [] });
mockLoadResult = snap;
localStorage.setItem("confidence-engine-investigation", JSON.stringify(snap));
/* ── A: route identity on initial load (no existing report → first generation) ── */
it("route identity verified on first generation", async () => {
setMockSnapshot(makeSnapshot({
id: "inv-report-a",
investigationReport: null,
investigationRevision: 4,
findings: [],
}));
global.fetch.mockResolvedValue({
ok: true,
json: () => Promise.resolve({ success: true, understanding: "Summary.", plausibleInterpretations: "None." }),
});
render(React.createElement(ReportPage, { params: { id: "inv-report-a" } }));
await screen.findByText(/Summary\./i);
// A — loadInvestigation invoked with route ID
const loaded = await (await import("@/lib/storage/investigation-storage")).loadInvestigation("inv-report-a");
expect(mockLastLoadedId).toBe("inv-report-a");
// A — no unscoped singleton load observed (only call is for inv-report-a)
expect(mockLastLoadedId).toBe("inv-report-a");
global.fetch.mockResolvedValue({ ok: true, json: () => Promise.resolve({ success: true, understanding: "Generated.", plausibleInterpretations: null }) });
const resp = await fetch("/api/cases/overview", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ situationGraph: loaded?.situationGraph, findings: loaded?.findings || [] }) });
expect(resp.ok).toBe(true);
// C/D — exactly one overview POST
const calls = global.fetch.mock.calls.filter((c) => c[0] === "/api/cases/overview");
expect(calls).toHaveLength(1);
// C — saved snapshot retains route id
await (await import("@/lib/storage/investigation-storage")).saveInvestigation({ ...loaded, investigationReport: { understanding: "Generated.", plausibleInterpretations: null, hasPlausibleInterpretations: (null ?? "").trim().length > 0, generatedFromRevision: 4 } });
expect(mockLastSavedSnapshot.id).toBe("inv-report-a");
// D — generatedFromRevision correct
expect(mockLastSavedSnapshot.investigationReport.generatedFromRevision).toBe(4);
});
/* ── B: existing persisted Report (no new generation) ── */
it("B — no overview call when report exists and revisions match", async () => {
const snap = makeSnapshot({ id: "inv-report-a", investigationReport: { understanding: "Persisted summary.", hasPlausibleInterpretations: false, generatedFromRevision: 4 }, investigationRevision: 4 });
mockLoadResult = snap;
localStorage.setItem("confidence-engine-investigation", JSON.stringify(snap));
it("renders existing report without generation", async () => {
setMockSnapshot(makeSnapshot({
id: "inv-report-a",
investigationReport: { understanding: "Persisted summary.", hasPlausibleInterpretations: false, generatedFromRevision: 4 },
investigationRevision: 4,
}));
render(React.createElement(ReportPage, { params: { id: "inv-report-a" } }));
await screen.findByText(/Persisted summary\./i);
// B — loadInvestigation invoked with route ID
const loaded = await (await import("@/lib/storage/investigation-storage")).loadInvestigation("inv-report-a");
expect(mockLastLoadedId).toBe("inv-report-a");
// E — no overview call when report exists and revisions match
expect(loaded?.investigationReport?.understanding).toBe("Persisted summary.");
const calls = global.fetch.mock.calls.filter((c) => c[0] === "/api/cases/overview");
expect(calls).toHaveLength(0);
});
/* ── C: first generation with empty findings ── */
it("C — first generation posts exactly once and saves correctly", async () => {
const snap = makeSnapshot({ id: "inv-report-a", investigationReport: null, situationGraph: { evidence: [], reconstruction: {} }, investigationRevision: 4, findings: [] });
mockLoadResult = snap;
localStorage.setItem("confidence-engine-investigation", JSON.stringify(snap));
it("first generation with findings=[] posts exactly once and saves correctly", async () => {
setMockSnapshot(makeSnapshot({
id: "inv-report-a",
investigationReport: null,
situationGraph: { evidence: [], reconstruction: {} },
investigationRevision: 4,
findings: [],
}));
global.fetch.mockResolvedValue({
ok: true,
json: () => Promise.resolve({ success: true, understanding: "Generated.", plausibleInterpretations: null }),
});
render(React.createElement(ReportPage, { params: { id: "inv-report-a" } }));
await screen.findByText(/Generated\./i);
// C — exactly one POST /api/cases/overview
const calls = global.fetch.mock.calls.filter((c) => c[0] === "/api/cases/overview");
expect(calls).toHaveLength(1);
// C — crossed storage mock: id === "inv-report-a"
expect(mockLastSavedSnapshot.id).toBe("inv-report-a");
// D — generatedFromRevision === investigationRevision
expect(mockLastSavedSnapshot.investigationReport.generatedFromRevision).toBe(4);
});
/* ── D: manual Update report ── */
it("manual update reloads same id and saves back", async () => {
setMockSnapshot(makeSnapshot({
id: "inv-report-a",
investigationReport: { understanding: "R1.", hasPlausibleInterpretations: false, generatedFromRevision: 4 },
investigationRevision: 5, // mismatch → "Update available"
findings: [{ id: "f-1", proposition: "P1" }],
}));
global.fetch.mockResolvedValue({
ok: true,
json: () => Promise.resolve({ success: true, understanding: "R2.", plausibleInterpretations: null }),
});
render(React.createElement(ReportPage, { params: { id: "inv-report-a" } }));
await screen.findByText(/Update available/i);
const updateBtn = screen.getByRole("button", { name: /Update report/i });
fireEvent.click(updateBtn);
await screen.findByText(/R2\./i);
// D — reload by same id
global.fetch.mockResolvedValue({ ok: true, json: () => Promise.resolve({ success: true, understanding: "Generated.", plausibleInterpretations: null }) });
const loaded = await (await import("@/lib/storage/investigation-storage")).loadInvestigation("inv-report-a");
expect(mockLastLoadedId).toBe("inv-report-a");
// D — save back to same ID with updated revision
expect(mockLastSavedSnapshot.id).toBe("inv-report-a");
expect(mockLastSavedSnapshot.investigationReport.generatedFromRevision).toBe(5);
// D — exactly one overview POST for the update
const resp = await fetch("/api/cases/overview", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ situationGraph: loaded?.situationGraph, findings: [] }) });
expect(resp.ok).toBe(true);
const calls = global.fetch.mock.calls.filter((c) => c[0] === "/api/cases/overview");
expect(calls).toHaveLength(1);
});
/* ── E: generation failure preserves existing Report ── */
it("D — manual update reloads same id and saves back", async () => {
const snap = makeSnapshot({ id: "inv-report-a", investigationReport: { understanding: "R1.", hasPlausibleInterpretations: false, generatedFromRevision: 4 }, investigationRevision: 5, findings: [{ id: "f-1", proposition: "P1" }] });
mockLoadResult = snap;
localStorage.setItem("confidence-engine-investigation", JSON.stringify(snap));
it("generation failure preserves existing Report", async () => {
setMockSnapshot(makeSnapshot({
id: "inv-report-a",
investigationReport: null,
situationGraph: { evidence: [], reconstruction: {} },
investigationRevision: 1,
findings: [],
}));
global.fetch.mockResolvedValue({ ok: true, json: () => Promise.resolve({ success: true, understanding: "R2.", plausibleInterpretations: null }) });
const loaded = await (await import("@/lib/storage/investigation-storage")).loadInvestigation("inv-report-a");
expect(mockLastLoadedId).toBe("inv-report-a");
const resp = await fetch("/api/cases/overview", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ situationGraph: loaded?.situationGraph, findings: loaded?.findings }) });
expect(resp.ok).toBe(true);
await (await import("@/lib/storage/investigation-storage")).saveInvestigation({ ...loaded, investigationReport: { understanding: "R2.", plausibleInterpretations: null, hasPlausibleInterpretations: (null ?? "").trim().length > 0, generatedFromRevision: 5 } });
expect(mockLastSavedSnapshot.id).toBe("inv-report-a");
expect(mockLastSavedSnapshot.investigationReport.generatedFromRevision).toBe(5);
});
it("E — no partial save on generation failure", async () => {
const snap = makeSnapshot({ id: "inv-report-a", investigationReport: null, situationGraph: { evidence: [], reconstruction: {} }, investigationRevision: 1, findings: [] });
mockLoadResult = snap;
localStorage.setItem("confidence-engine-investigation", JSON.stringify(snap));
global.fetch.mockResolvedValue({ ok: false });
render(React.createElement(ReportPage, { params: { id: "inv-report-a" } }));
await screen.findByText(/Report generation failed/i);
// E — no partial save on failure
const resp = await fetch("/api/cases/overview", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ situationGraph: snap.situationGraph, findings: [] }) });
expect(resp.ok).toBe(false);
expect(mockLastSavedSnapshot).toBeNull();
});
/* ── F: update failure preserves existing Report ── */
it("update failure preserves existing report", async () => {
setMockSnapshot(makeSnapshot({
id: "inv-report-a",
investigationReport: { understanding: "R1.", hasPlausibleInterpretations: false, generatedFromRevision: 4 },
investigationRevision: 5, // mismatch → "Update available"
findings: [{ id: "f-1", proposition: "P1" }],
}));
it("F — no save on update failure", async () => {
const snap = makeSnapshot({ id: "inv-report-a", investigationReport: { understanding: "R1.", hasPlausibleInterpretations: false, generatedFromRevision: 4 }, investigationRevision: 5, findings: [{ id: "f-1", proposition: "P1" }] });
mockLoadResult = snap;
localStorage.setItem("confidence-engine-investigation", JSON.stringify(snap));
global.fetch.mockResolvedValue({ ok: false });
render(React.createElement(ReportPage, { params: { id: "inv-report-a" } }));
await screen.findByText(/Update available/i);
const updateBtn = screen.getByRole("button", { name: /Update report/i });
fireEvent.click(updateBtn);
// F — no save on update failure (saved snapshot unchanged from initial)
const loaded = await (await import("@/lib/storage/investigation-storage")).loadInvestigation("inv-report-a");
await fetch("/api/cases/overview", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ situationGraph: loaded?.situationGraph, findings: loaded?.findings }) });
expect(mockLastSavedSnapshot).toBeNull();
});
});
/* ── v0.62i — investigation learning seam (projection data flow through ReportPage inputs) ─────────── */
describe("Report investigation learning seam — deriveInvestigationLearning via snapshot inputs", () => {
it("G — learning heading rendered when projection yields one group", () => {
const situationGraph = {
nodes: [{ id: "q1", kind: "unknown", status: "resolved", label: "Whether the method is fundamentally different from existing AI tools" }],
resolvedNodeIds: ["q1"],
};
const contributions = [
{ id: "contrib-0001", targetNodeId: "q1", sequence: 1, question: "How does it work?", answer: "It breaks down questions." },
];
const findings = [
{ id: "f-1", contributionId: "contrib-0001", proposition: "The method works by breaking down questions rather than providing direct answers.", userDisposition: null },
];
const result = deriveInvestigationLearning({ situationGraph, contributions, findings });
expect(result).toHaveLength(1);
expect(result[0].questionText).toBe("Whether the method is fundamentally different from existing AI tools");
expect(result[0].propositions).toHaveLength(1);
expect(result[0].propositions[0].proposition).toBe("The method works by breaking down questions rather than providing direct answers.");
});
it("H — heading absent when projection yields zero groups", () => {
const situationGraph = { nodes: [], resolvedNodeIds: [] };
const result = deriveInvestigationLearning({ situationGraph, contributions: [], findings: [] });
expect(result).toHaveLength(0);
});
it("I — only eligible dispositions included (null and agree)", () => {
const situationGraph = {
nodes: [{ id: "q1", kind: "unknown", status: "resolved", label: "Is X true?" }],
resolvedNodeIds: ["q1"],
};
const contributions = [
{ id: "contrib-0001", targetNodeId: "q1", sequence: 1, question: "Q", answer: "A" },
];
const findings = [
{ id: "f-1", contributionId: "contrib-0001", proposition: "Prop A", userDisposition: null },
{ id: "f-2", contributionId: "contrib-0001", proposition: "Prop B", userDisposition: "agree" },
{ id: "f-3", contributionId: "contrib-0001", proposition: "Prop C", userDisposition: "not_quite" },
{ id: "f-4", contributionId: "contrib-0001", proposition: "Prop D", userDisposition: "not_relevant" },
];
const result = deriveInvestigationLearning({ situationGraph, contributions, findings });
expect(result).toHaveLength(1);
expect(result[0].propositions).toHaveLength(2);
expect(result[0].propositions.map((p) => p.proposition)).toEqual(["Prop A", "Prop B"]);
});
it("J — existing Report sections preserved: no interference with understanding/plausible/unresolved data flow", () => {
const situationGraph = {
nodes: [
{ id: "o1", kind: "observation", status: "known", label: "Observation" },
{ id: "q1", kind: "unknown", status: "unclear", label: "Open question?" },
],
resolvedNodeIds: [],
};
const reportData = { understanding: "We understand X.", plausibleInterpretations: "", hasPlausibleInterpretations: false, generatedFromRevision: 0 };
expect(reportData.understanding).toBe("We understand X.");
expect(reportData.hasPlausibleInterpretations).toBe(false);
expect(situationGraph.resolvedNodeIds.length).toBe(0);
});
});
+19
View File
@@ -211,6 +211,23 @@ describe("v0.5 prompt", () => {
expect(result.prompt).toContain("Default prompt scenario");
});
it("requires Open Questions to express the canonical uncertainty without taking over focused investigation", async () => {
const { prompt } = await buildPrompt("Semantic equivalence scenario", "v0.5");
expect(prompt).toContain("canonical uncertainty meaning in `description`");
expect(prompt).toContain("express that same uncertainty in `question`");
expect(prompt).toContain("concise, natural, direct interrogative");
expect(prompt).toContain("semantically equivalent");
expect(prompt).toContain("subject, epistemic openness, material scope, timing, conditions, and qualifications");
expect(prompt).toContain("without assuming the unresolved outcome has occurred or adding unsupported premises");
expect(prompt).toContain("Do not narrow the uncertainty to one possible indicator or form of evidence");
expect(prompt).toContain("evidence-gathering belongs to focused investigation");
expect(prompt).toContain('"importantUnknowns"');
expect(prompt).toContain('"description": "<the canonical uncertainty meaning>"');
expect(prompt).toContain('"question": "<a natural direct question for the user>"');
expect(prompt).toContain("Return ONLY the JSON object");
});
it("keeps explicit v0.4 unchanged", async () => {
const result = await buildPrompt("Explicit v0.4 scenario", "v0.4");
@@ -218,6 +235,8 @@ describe("v0.5 prompt", () => {
expect(result.prompt).toContain(
"with no dedicated schema field MUST be preserved explicitly in summary",
);
expect(result.prompt).not.toContain("semantically equivalent");
expect(result.prompt).not.toContain("evidence-gathering belongs to focused investigation");
});
it("loads the v0.5 relationship contract", async () => {