Compare commits

..
Author SHA1 Message Date
robbond 3366168e07 fix(confidence-engine): align explored question status language 2026-10-26 06:02:05 +00:00
robbond d8981c4e8e fix(confidence-engine): describe explored questions without implying resolution 2026-10-26 05:57:15 +00:00
robbond 8530d954a0 feat(helper): add --prompt-file option for experimental prompt variation
Add optional --prompt-file <path> argument to start-case-experiment-helper.cjs
that allows supplying an alternative reconstruction prompt file while
preserving the production JSON generation schema.

Requirements implemented:
- Reads UTF-8 prompt file (absolute path or relative to prompts/)
- Replaces {{SCENARIO}} placeholder with scenario text using existing convention
- Validates file exists and contains required placeholder
- Rejects --prompt-file with non-A conditions via clear error
- Only compatible with condition A (production structured-output contract)
- Zero production code changes; apparatus-only modification
- Preserves existing behavior for all conditions B-E without flag
- Does not activate schema variants or freeform handling

Verified: syntax check passes, no whitespace errors in diff.
2026-10-14 15:53:43 +01:00
robbond fe0f337a99 docs(experiments): checkpoint uncertainty recognition investigation 2026-10-14 09:39:11 +01:00
robbond e595ee6e02 feat(experiments): integrate Condition D/E apparatus 2026-10-13 16:24:00 +01:00
robbond 29cdc80f09 feat(experiments): integrate freeform analysis path 2026-10-13 14:36:11 +01:00
robbond 8dcdffce44 feat(experiments): support freeform provider responses 2026-10-13 11:59:47 +01:00
robbond 74936851ce feat(experiments): add atomic evidence persistence 2026-10-13 11:18:58 +01:00
robbond 5af3e62ddd feat(uncertainty-led-investigation): add Condition D freeform deconstruct treatment selector
- Add reconstruct-v0.5-d-deconstruct-no21.md prompt (removes rule 21)
  from original deconstruct-first prompt: 'Do not create artificial
  uncertainties to increase their number.'
- Extend start-case-experiment-helper.cjs with --condition D and
  --treatment no21 CLI flags for treatment selector apparatus
- Add programmatic treatment validation (rejects invalid treatments
  cross-condition, validates only 'no21' is a valid treatment value)
- Condition D output uses freeformText field instead of reconstruction
  JSON — avoids production JSON schema enforcement on freeform text
- Mock double supports freeform mode: returns deterministic raw text
  instead of structured importantUnknowns when isFreeform=true
- Add 23 targeted tests: J (Condition D prompt selection),
  J.1 (treatment validation), J.2 (prompt isolation verification)
- Fix existing B.1 assertions to use matchObject for execution field
2026-10-12 18:45:53 +01:00
16 changed files with 1623 additions and 65 deletions
+3 -3
View File
@@ -558,7 +558,7 @@ function hasGenuineCompletion(graph) {
function CompletionCard({ summary }) {
return (
<div className="rounded-lg border border-green-300 bg-green-50 px-5 py-6 text-center">
<h2 className="mb-1 text-sm font-bold uppercase tracking-wide text-green-700">Investigation complete</h2>
<h2 className="mb-1 text-sm font-bold uppercase tracking-wide text-green-700">All questions explored</h2>
<p className="text-base text-gray-800 mb-3">The available evidence supports the following understanding.</p>
{summary && (
<div className="mt-4 text-left rounded-md bg-white/60 px-4 py-3 border border-green-100">
@@ -2002,7 +2002,7 @@ export default function ReasoningWorkspace({
{clarifiedQuestions.length > 0 && (
<div className="space-y-3">
<h2 className="text-[11px] font-semibold tracking-widest uppercase text-gray-500">
Questions we have clarified
Questions we've explored
</h2>
{clarifiedQuestions.map((node) => (
<div key={node.id} className="space-y-1">
@@ -2014,7 +2014,7 @@ export default function ReasoningWorkspace({
<p className="mt-1.5 text-xs leading-snug text-gray-500">{node.description}</p>
)}
<div className="mt-2 flex items-center gap-3">
<span className="text-[10px] uppercase tracking-wider text-green-600">Clarified</span>
<span className="text-[10px] uppercase tracking-wider text-green-600">Explored</span>
<button
onClick={() => {
const nextGraph = reopenResolvedUnknown(graph, node.id);
+80 -2
View File
@@ -965,6 +965,84 @@ Returns JSON-serializable record with: five deconstruction arrays (`observations
**Evidence file:** `.evidence-temp/headless-semantic/HS-03C-focused-E9.json`
### Iteration 3 — Extract Focused Target Input (apparatus recovery)
### Iteration 3 — UI Copy: Clarified → Explored (bounded heading change)
The focused-investigation target-node extraction was purified into a standalone pure function `extractFocusedTargetInput(situationGraph, targetNodeId)` in the canonical helper. Deterministic test verified: known-node exact-match (1/1), unknown-node empty-string fallback (2/2), graph-immutable-side-effect (1/1), source-pattern integration verification (1/1) — total 5/5 PASS. Zero live model calls. Extraction was tested deterministically; the original Experiment 9 runtime graph and root cause remain unestablished.
**Type:** Targeted UI copy change (Iteration 3 of bounded iteration loop). Zero live calls. Zero behaviour change.
**Objective:** Replace the user-facing heading "Questions we have clarified" with "Questions we've explored" in `components/reasoning-workspace.jsx`. Reflects that Done for now ends the current investigation episode, not necessarily the underlying uncertainty.
**What was changed:**
- Line 2005 of `components/reasoning-workspace.jsx`: visible `<h2>` heading text replaced — `Questions we have clarified` → `Questions we've explored`
- No code logic, no state transitions, no filtering or classification changes.
**Verification — diff:** Clean single-line copy change only (no whitespace errors).
**Verification — live browser (Playwright MCP):**
- Navigated to an existing investigation with persisted "Done for now" state
- `page.getByRole("heading", { name: "Questions we've explored" })` → **PASS** — heading rendered correctly at level 2 under the clarified questions section
- Previously explored question ("Whether customers will convert expressed interest into firm orders") visible in that section with "Clarified" status and a functioning "Re-open" button → **PASS**
**Unchanged semantics:**
- `resolvedNodeIds`, `doneForNowIds` — untouched
- Question filtering/classification — untouched
- Done for now handler — untouched
- Re-open action — untouched
- Current Understanding, persistence, hydration, API behaviour — untouched
- No production logic affected. Only user-facing heading text changed.
**Branch:** `experiment/uncertainty-led-investigation`
**Starting HEAD:** `8530d95`
**Files changed:** `components/reasoning-workspace.jsx`, `docs/current-handoff.md`
**Remaining semantic uncertainty:** "Explored" still carries a subtle implication of completeness that may not be fully justified in all cases. Whether users interpret "explored" as sufficiently provisional vs. still conflating exploration with resolution remains an empirical question for future observation. Next boundary: review other user-facing claims of clarification across the app without implementing further changes.
**Targeted test:** NOT IDENTIFIED
**Playwright heading:** PASS
**Playwright Re-open:** PASS
### Iteration 5 — Remove misleading finality language (2026-10-26)
**Type:** Targeted UI copy change. Zero live LLM calls. Zero behaviour change.
**Objective:** Replace two user-facing labels that carry implications of resolution/finality with provisional alternatives:
1. `"Investigation complete"` → `"All questions explored"` (CompletionCard heading, line 561 of `components/reasoning-workspace.jsx`)
2. `"Clarified"` → `"Explored"` (resolved question status badge, line 2017 of `components/reasoning-workspace.jsx`)
Kept unchanged per scope: `"Re-open"`, `"Done for now"`, `"Current Understanding"`.
**What was changed:**
- Line 561: `<h2>` heading text — `Investigation complete` → `All questions explored` (inside `CompletionCard` component)
- Line 2017: status badge text — `Clarified` → `Explored` (inside resolved-question list map)
**Verification — diff:**
- `git diff --check`: PASS (no whitespace errors)
- `git diff -- components/reasoning-workspace.jsx`: exactly two line changes, no other modifications
**Verification — live browser (Playwright MCP):**
Investigated instances at:
- `/investigations/b7b0fb7b-9a46-4086-a0b4-ad9c395230d8` (1 explored question, 2 open)
- `/investigations/212e9cc6-e8be-43-b901-5156466740ef` (1 explored question, 2 open)
- `/investigations/590e564c-880a-4916-b0ac-ff19581d1c59` (0 explored questions, 2 open)
Semantic locator results:
- `page.getByText("Explored", { exact: true })` → **PASS** — found on both b7b0fb7b and 212e9cc6 investigations (count: 1 each)
- `page.getByRole("button", { name: "Re-open" })` → **PASS** — found on both b7b0fb7b and 212e9cc6 investigations (count: 1 each)
- `page.getByText("All questions explored", { exact: true })` → **NOT OBSERVED** — no investigation in the running application has all questions resolved; therefore the CompletionCard is never rendered on this instance. None of the 8 existing investigations reached full completion.
**Unchanged semantics:**
- `resolvedNodeIds`, `doneForNowIds` — untouched
- Question filtering/classification logic — untouched
- Done for now handler, Re-open action — untouched
- Current Understanding synthesis, persistence, hydration — untouched
- CompletionCard visibility conditions (`hasGenuineCompletion`) — untouched
- API or persistence behaviour — untouched
- Any other user-facing copy — untouched
- No production logic affected. Only two label strings changed.
**Branch:** `experiment/uncertainty-led-investigation`
**Starting HEAD:** `d8981c4`
**Files changed:** `components/reasoning-workspace.jsx`, `docs/current-handoff.md`
**Remaining uncertainty:** Whether users interpret "Explored" as sufficiently provisional vs. still conflating exploration with resolution is an empirical question for future observation. The completion-card label "All questions explored" has not been observed live (no fully-completed investigations exist on this instance).
@@ -0,0 +1,66 @@
# Uncertainty Information Flow Audit — Programme Handoff
**Working title:** Confidence Engine — Uncertainty Information Flow Audit
**Date prepared:** 2026-10-14
**Prepared for:** Next independent investigation (following the completed 10-iteration uncertainty recognition quality investigation)
---
## Primary Question
"At which point, if any, does Confidence Engine constrain, transform, discard or fail to preserve meaningful uncertainty recognition?"
## Context
The previous 10-iteration programme demonstrated a quality gap between minimal direct-model uncertainty recognition and Confidence Engine-mediated uncertainty recognition. The precise limiting mechanism remains unresolved (see `uncertainty-recognition-investigation-report.md` sections D–G). This new programme is designed to map the actual production information flow and identify specific intervention points without presuming failure locations.
## Investigation Boundaries (Not Presumed Failure Points)
1. User input and scenario representation
2. Prompt construction
3. Raw LLM request and response
4. Parsing and validation
5. Internal graph representation
6. User-facing presentation
These boundaries define investigation scope, not expectations of where problems will be found.
## Recommended Approach
- Up to 10 iterations
- One bounded investigative question per iteration
- ChatGPT controls programme progression between iterations
- Claude Code executes one task per invocation and reports evidence
- Each `/rdbHandoff` invocation is a fresh session — work from repository evidence and this handoff
- Read-only investigation before any interventions
- Controlled experiments only when justified by evidence from prior iterations
- Preserve raw evidence and request provenance for every iteration
- No speculative product changes
## First Iteration Guidance
The first iteration should establish the actual production information flow and available capture boundaries:
1. Map the complete code path from scenario file input to model request construction
2. Document what data is accessible at each stage of the pipeline (raw scenario, post-parsing, pre-template, post-template, pre-request, post-response, post-parsing)
3. Identify any existing hooks, logs, or capture mechanisms for raw requests/responses
4. Determine which stages can produce evidence without source modifications
The output should be a factual map of available capture points and data accessibility, not an analysis of what is "wrong."
## Existing Evidence to Consider
- `docs/uncertainty-recognition-investigation-report.md` — Full report from the previous programme
- `.evidence-temp/iteration10/architecture-finding.txt` — Definitive CE prompt-path analysis showing template applicability at every invocation point
- `.evidence-temp/headless-semantic/` — Extensive supporting evidence from HS series experiments
## What This Programme Is Not
- A redesign of Confidence Engine
- A directive to change the model or provider configuration
- An assumption that any specific boundary is broken or deficient
- A continuation of the previous programme's quality-gap investigation (that closed at Iteration 10)
## Start Signal
Do not begin this programme from this document alone. Wait for an explicit `/rdbHandoff` invocation containing the start signal and first iteration directive.
@@ -0,0 +1,147 @@
# Uncertainty Recognition Quality Investigation — Report
**Date:** 2026-10-14
**Branch:** `experiment/uncertainty-led-investigation`
**Starting HEAD:** `e595ee6` feat(experiments): integrate Condition D/E apparatus
**Status:** CLOSED — 10-iteration programme completed and checkpointed
---
## A. Investigation Objective
This investigation examined why the configured local LLM (`qwen-claude:latest` via Ollama) produced less useful uncertainty recognition through Confidence Engine's reconstruction pipeline than through a minimal direct prompt.
The canonical scenario used was `RFG-03-independent-decisions-input.json`, concerning three independent decisions (replacing internal reporting software, recruiting an additional customer-support employee, refurbishing a meeting room). The user's goal is to understand what remains uncertain before deciding what to do.
**Observed phenomenon:** A minimal direct-model prompt produced two outputs containing approximately 21 and 15 uncertainties with meaningful domain-specific unknowns. Confidence Engine reconstruction under Conditions D/E generally produced three generic uncertainties that largely restated the user's stated decisions without adding new insights.
---
## B. Experimental Programme (Iterations 1–10)
### Iteration 3 — Baseline Direct-Model Prompt
**Status:** Complete and reproducible
**Evidence:** `.evidence-temp/experiment3/`
**Prompt:** Minimal raw prompt with no rule-set framing or forced JSON output.
**Result:** Two model outputs: raw-run-1 (29 lines, ~15+ domain-specific uncertainties), raw-run-2 (21 lines, similar richness). Scenario SHA-256: `dc9a600f9fb1e7189dd56086a07f357893396a5b0782f4781c95d4aa29fdcf06`.
### Iteration 5 — Condition D Rule-21 Removal
**Status:** Complete
**Evidence:** `.evidence-temp/iteration5/`
**Treatment:** Removed rule 21 from the deconstruction prompt.
**Result:** Four CE uncertainties including a cross-domain "worthwhile" framework question. No material improvement over control in domain-specific discovery. Evidence: `D-no21-freeform.txt` (103 lines), `D-no21.json`, `D-original-freeform.txt`, `D-original.json`.
### Iteration 6 — Condition D Rule-23 Removal + Prompt Variant
**Status:** Complete
**Evidence:** `.evidence-temp/iteration6/`
**Treatment:** Removed rule 23; tested with new prompt variant (`reconstruct-v0.5-d-deconstruct-no23.md`).
**Result:** No material improvement. Both `run-A-original-D.json` and `run-B-no-rule-23.json` validated as freeform with zero graph nodes.
### Iteration 8 — Invalid Run + Recovery
**Status:** Validated invalidity; recovery run complete
**Evidence:** `.evidence-temp/iteration8/` (invalid), `.evidence-temp/iteration8-recovery/` (valid)
**Invalidity cause:** Scenario filename was treated as scenario text instead of JSON content.
**Recovery result:** 4 uncertainty items with modest structural enrichment (expectedStates, differences, tighter relationships) but no new domain-specific unknowns.
### Iterations 9–10 — Follow-up and Block
**Iteration 9:** Partial evidence; focused-investigation dependency analysis. `.evidence-temp/iteration9/` contains a `b-variant/` subdirectory (incomplete).
**Iteration 10:** BLOCKED before model execution. The existing experimental helper (`scripts/start-case-experiment-helper.cjs`) has no mechanism to pass the exact minimal raw prompt through the CE execution path without source modifications. Every invocation applies a structured prompt template. See `.evidence-temp/iteration10/final-report.txt` and `architecture-finding.txt` for the definitive architecture analysis.
### Headless-Semantic Evidence (HS-01 through HS-03C)
**Status:** Extensive supporting evidence preserved
**Evidence:** `.evidence-temp/headless-semantic/` (28 files, 2788 lines total)
Covers: Question-free reconstruction comparison (CE-ULI-09), domain-preservation instruction tests (CE-ULI-10, CE-ULI-11), and focused-investigation evidence persistence testing.
---
## C. Established Findings
These findings are directly supported by the evidence:
1. **Direct-model prompt quality gap:** Minimal direct prompts produced outputs with approximately 21 and 15 domain-specific uncertainties across three decisions, while CE reconstruction under Conditions D/E generally produced three generic restatements of the user's own decisions.
2. **Rule removal had no material effect:** Removing Rule 21 (Iteration 5) and Rule 23 (Iteration 6) did not materially improve uncertainty recognition quality.
3. **Discovery permission instruction produced modest structural enrichment:** The Iteration 8 recovery (adding explicit permission to recognise plausible unstated uncertainties) added four items including a cross-domain evaluation framework question, but no new domain-specific unknowns were discovered beyond what the user stated.
4. **CE execution path is structurally incapable of raw-prompt pass-through:** Every invocation applies a template-based prompt construction that adds rule sets and forced-JSON constraints absent from the direct-model baseline. This was definitively established in Iteration 10's architecture-finding evidence.
5. **Question-free reconstruction does not prevent structural compression:** CE-ULI-10 demonstrated that removing the question-generation requirement still produced structural compression of uncertainties into single compound descriptions for the tested scenario shape.
6. **Explicit domain-separation instructions are ignored:** CE-ULI-11 showed identical compression under both condition B and C despite explicit instruction not to combine domains.
---
## D. Unresolved Hypotheses
**H1 — Model capability:** The model may be capable of richer uncertainty recognition but its output quality is constrained by the surrounding apparatus (prompt structure, output format constraints). *Untested.* Not equivalent to H2 or H3.
**H2 — Prompt effectiveness:** The template-based prompt construction (rule sets, forced JSON, deconstruction framing) suppresses the model's ability to produce rich uncertainty recognition relative to a minimal direct prompt. *Supported by evidence of quality gap but not proven as causal.* No controlled experiment eliminated the structural difference while preserving all other conditions.
**H3 — Confidence Engine orchestration:** Something in the CE pipeline (prompt construction, parsing, validation, graph representation, or presentation) constrains, transforms, or discards meaningful uncertainty recognition between model output and user-facing display. *Supported by evidence but not isolated.* Iteration 10 established that the CE execution path always applies templates, making controlled elimination of specific pipeline stages infeasible without source modifications.
**Important:** The precise cause of the quality difference remains unresolved. The evidence demonstrates a quality gap and several plausible explanations, not definitive causation. It is not established that prompt construction, JSON formatting, or orchestration individually or collectively suppress uncertainty recognition — only that these are all uncontrolled confounders in the comparison between CE-executed requests and direct model calls.
---
## E. Experimental Apparatus Limitations
1. **Template-based prompt construction:** All CE invocation paths apply a template (v0.5, b/c/d/e variants). No raw-prompt pass-through mode exists in `lib/analysis.js` or `start-case-experiment-helper.cjs`.
2. **No raw-prompt pass-through in the helper:** Condition selector loads hardcoded template filenames. The `--experiment-instruction` option only appends — it cannot replace or suppress base structure.
3. **`--file` input-loading requirement:** Scenarios must be loaded from JSON files. Line-input and other loading modes are not supported by the helper.
4. **`--reconstruction-only` requirement:** To bypass the graph pipeline, `--reconstruction-only` is required. Its absence routes through the full production path.
5. **Generation-setting comparability limitations:** Direct-model baseline runs (experiment3) used different generation settings (no forced schema) than CE execution (forced JSON schemas). Temperature, top_p, and other parameters are not controlled or logged for direct runs.
6. **Missing evidence:** Iteration 9's b-variant evidence is incomplete (empty subdirectory). Evidence for some earlier iterations beyond condition outputs alone is unavailable.
7. **Single-model limitation:** All experiments used `qwen-claude:latest` only. Results may not generalise to other models.
---
## F. Product Principles
The following principles must govern any future work on uncertainty recognition:
- The user owns investigation direction.
- The engine facilitates rather than steers.
- Recognising an uncertainty is not inventing a fact.
- Recognising an uncertainty is not selecting it.
- The engine must not automatically decide what the user should investigate next.
- Fidelity to the user's situation and recognition of plausible unstated unknowns can coexist.
- Do not optimise for the number of uncertainty items. Optimise for meaningful, distinct, well-grounded uncertainty recognition.
Richer uncertainty recognition must not be confused with engine-owned prioritisation or investigation steering. The value of additional uncertainties is in user understanding, not in expanding the investigation scope autonomously.
---
## G. Programme Conclusion
The local model (`qwen-claude:latest`) demonstrated richer uncertainty recognition when invoked via a minimal direct prompt independently than when Confidence Engine mediated the invocation through its reconstruction pipeline (Conditions D/E). Confidence Engine did not reliably reproduce that behaviour across any tested condition or prompt variant.
The precise limiting mechanism remains unresolved — whether it is model capability, prompt effectiveness, orchestration constraints, or their interaction. The original 10-iteration investigation programme is closed.
---
## Referenced Evidence Files
| Artefact | Path | Status |
|----------|------|--------|
| Raw minimal prompt (experiment3) | `.evidence-temp/experiment3/raw-prompt.txt` (858B, 13 lines) | Complete |
| Direct-model run 1 | `.evidence-temp/experiment3/raw-run-1-output.txt` (3520B, 29 lines) | Complete |
| Direct-model run 2 | `.evidence-temp/experiment3/raw-run-2-output.txt` (1855B, 21 lines) | Complete |
| Scenario SHA-256 | `.evidence-temp/iteration10/scenario-verification.txt` | `dc9a600f...fdcf06` |
| Condition D / no21 evidence | `.evidence-temp/iteration5/D-no21-freeform.txt`, `D-no21.json` | Complete |
| Condition D original (iter 5) | `.evidence-temp/iteration5/D-original-freeform.txt`, `D-original.json` | Complete |
| Rule-23 removal prompt | `.evidence-temp/iteration6/prompt-variants/reconstruct-v0.5-d-deconstruct-no23.md` | Complete |
| Run A (original D, iter 6) | `.evidence-temp/iteration6/run-A-original-D.json` | Complete |
| Run B (no rule 23, iter 6) | `.evidence-temp/iteration6/run-B-no-rule-23.json` | Complete |
| Iteration 8 invalid run | `.evidence-temp/iteration8/run-A-original-d.json`, `run-B-discovery-permission.json` | Invalid |
| Discovery permission prompt | `.evidence-temp/iteration8/reconstruct-v0.5-d-discovery-permission.md` | Complete |
| Iteration 8 recovery log | `.evidence-temp/iteration8-recovery/experiment_log.txt`, `b-comparison.txt` | Complete |
| Iteration 10 final report | `.evidence-temp/iteration10/final-report.txt` | Complete (BLOCKED) |
| Iteration 10 architecture finding | `.evidence-temp/iteration10/architecture-finding.txt` | Complete |
| Headless-semantic series | `.evidence-temp/headless-semantic/` (28 files) | Extensive, preserved |
+77 -16
View File
@@ -80,23 +80,40 @@ export async function analyseScenario(scenario, opts = {}) {
let providerApiPath;
let providerExecution;
try {
const providerResult = await provider.generateReconstruction(
promptObj.prompt,
reconstructionModelName,
opts.generationSchema ?? reconstructionGenerationJsonSchema,
{ outputZodSchema: reconstructionGenerationSchema },
);
if (
providerResult &&
typeof providerResult === "object" &&
"response" in providerResult &&
"providerApiPath" in providerResult
) {
rawResponse = providerResult.response;
providerApiPath = providerResult.providerApiPath;
providerExecution = providerResult.providerExecution;
// Condition D freeform mode: call without schema enforcement → genuine raw text.
if (opts.freeform) {
const providerResult = await provider.generateReconstruction(
promptObj.prompt,
reconstructionModelName,
null, // no JSON schema
{ outputZodSchema: reconstructionGenerationSchema },
);
if (providerResult && typeof providerResult === "object" && "response" in providerResult) {
rawResponse = providerResult.response;
providerApiPath = providerResult.providerApiPath ?? null;
providerExecution = providerResult.providerExecution ?? null;
} else {
rawResponse = typeof providerResult === "string" ? providerResult : JSON.stringify(providerResult);
}
} else {
rawResponse = providerResult;
const providerResult = await provider.generateReconstruction(
promptObj.prompt,
reconstructionModelName,
opts.generationSchema ?? reconstructionGenerationJsonSchema,
{ outputZodSchema: reconstructionGenerationSchema },
);
if (
providerResult &&
typeof providerResult === "object" &&
"response" in providerResult &&
"providerApiPath" in providerResult
) {
rawResponse = providerResult.response;
providerApiPath = providerResult.providerApiPath;
providerExecution = providerResult.providerExecution;
} else {
rawResponse = providerResult;
}
}
} catch (e) {
return buildErrorResponse(
@@ -111,6 +128,22 @@ export async function analyseScenario(scenario, opts = {}) {
const duration = Date.now() - startTime;
// ── Condition D freeform: return raw text without parsing ──
if (opts.freeform) {
const text = typeof rawResponse === "string"
? rawResponse
: JSON.stringify(rawResponse);
return buildSuccessResultFreeform(
text,
reconstructionModelName,
duration,
promptVersion,
null, // no compatibility for freeform
providerApiPath,
providerExecution,
);
}
// Try to capture raw response for diagnostics
let rawResponseStr;
try {
@@ -228,6 +261,34 @@ function buildSuccessResultV2(data, model, duration, version, compatibility) {
};
}
/** Build a success result for Condition D freeform output — no schema validation. */
function buildSuccessResultFreeform(
text,
model,
duration,
version,
compatibility,
providerApiPath,
providerExecution,
) {
return {
success: true,
validationStatus: "freeform",
modelName: model,
responseDurationMs: duration,
rawResponse: text.slice(0, 3000),
promptVersion: version,
inputClassification: null,
reconstruction: text, // genuine freeform text — not JSON-structured output
evidence: undefined,
nextQuestion: undefined,
errors: undefined,
presentationValidationStatus: "valid",
presentationValidationIssues: [],
...(compatibility ? buildCompatibilityDiagnostics(compatibility) : {}),
};
}
function buildSuccessResultV1(data, model, duration, version, compatibility) {
return {
success: true,
+20
View File
@@ -373,6 +373,9 @@ export async function startCase(body, dependencies = {}) {
if (dependencies.generationSchema) {
analysisOptions.generationSchema = dependencies.generationSchema;
}
if (dependencies.freeform) {
analysisOptions.freeform = dependencies.freeform;
}
const analysis = await analyseScenario(scenario, analysisOptions);
if (!analysis.success) {
@@ -394,6 +397,23 @@ export async function startCase(body, dependencies = {}) {
};
}
// Condition D freeform + reconstruction-only: skip structured graph pipeline.
if (dependencies.reconstructionOnly && dependencies.freeform) {
return {
success: true,
summary: analysis.summary ?? null,
reconstruction: analysis.reconstruction, // genuine freeform text
situationGraph: null,
selectedQuestion: null,
diagnostics: buildDiagnostics({
analysis,
graph: null,
graphReferenceValidation: null,
}),
statusCode: 200,
};
}
const initialGraph = buildInitialGraph({
reconstruction: analysis.reconstruction,
evidence: analysis.evidence,
+57 -18
View File
@@ -352,6 +352,9 @@ class OllamaLlmProvider {
const baseUrl = process.env.OLLAMA_BASE_URL;
if (!baseUrl) throw new Error("OLLAMA_BASE_URL is not set");
// Explicit null signals "freeform text" — no JSON-schema constraint.
const useFreeformText = outputSchema === null;
let apiUsed = null;
let chatSupported = false;
let rawResponse = null;
@@ -382,15 +385,19 @@ class OllamaLlmProvider {
try {
apiUsed = "/api/chat";
providerExecution.chatRequestAttempted = true;
const body = {
model: modelName,
messages: [{ role: "user", content: prompt }],
stream: false,
};
// Explicit null → omit format constraint entirely.
if (outputSchema !== null) {
body.format = outputSchema ?? reconstructionJsonSchema;
}
const res = await fetch(`${baseUrl}/api/chat`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
model: modelName,
messages: [{ role: "user", content: prompt }],
stream: false,
format: outputSchema ?? reconstructionJsonSchema,
}),
body: JSON.stringify(body),
signal: requestAbort.signal,
});
@@ -445,6 +452,11 @@ class OllamaLlmProvider {
? fullResponseData.response
: JSON.stringify(fullResponseData);
// generate has no format constraint — treat freeform input as freeform output.
if (useFreeformText) {
apiUsed = "/api/generate";
}
} catch (e) {
if (apiUsed === "/api/generate") {
const error = new Error(
@@ -476,11 +488,11 @@ class OllamaLlmProvider {
// ================================================================
if (rawResponse === "") {
let diagInfo = "";
if (fullResponseData) {
const keys = Object.keys(fullResponseData);
diagInfo = "Keys in response: " + keys.join(", ") + "\n";
for (const key of keys) {
const val = fullResponseData[key];
if (typeof val === "string") {
@@ -501,6 +513,7 @@ class OllamaLlmProvider {
"\nPossible causes:\n" +
"- Check 'ollama list' — make sure the model name matches exactly what's installed\n" +
"- The model may be corrupted. Try: ollama pull " + modelName + "\n" +
(useFreeformText ? "- No format constraint was sent — the model may not support freeform output.\n" : "") +
"- This Ollama version does not support format:json — using prompt instructions only (reliability varies)\n" +
"- If your model is very small (e.g., tinyllama, phi), try a larger one like llama3.1 or mistral"
);
@@ -512,6 +525,14 @@ class OllamaLlmProvider {
// ================================================================
// Step 5: Parse and return
// ================================================================
if (useFreeformText) {
return {
response: rawResponse,
providerApiPath: apiUsed,
providerExecution,
};
}
try {
return {
response: recoverJson(rawResponse),
@@ -553,17 +574,27 @@ class OpenAIReconstructionProvider {
{ signal, outputZodSchema } = {},
) {
if (!this.apiKey) throw new Error("OPENAI_API_KEY is not set");
const structuredOutputSchema = outputSchema
? createOpenAIStrictSchema(outputSchema, outputZodSchema ?? null)
: openAIReconstructionJsonSchema;
const format = {
type: "json_schema",
name: "reconstruction",
strict: true,
schema: structuredOutputSchema,
};
traceOpenAISchema({ model: modelName, format, schema: structuredOutputSchema });
// Explicit null → freeform text output (no JSON-schema constraint).
const useFreeformText = outputSchema === null;
let format;
if (useFreeformText) {
format = { type: "text" };
} else {
const structuredOutputSchema = outputSchema
? createOpenAIStrictSchema(outputSchema, outputZodSchema ?? null)
: openAIReconstructionJsonSchema;
format = {
type: "json_schema",
name: "reconstruction",
strict: true,
schema: structuredOutputSchema,
};
traceOpenAISchema({ model: modelName, format, schema: structuredOutputSchema });
}
const response = await this.fetchImpl("https://api.openai.com/v1/responses", {
method: "POST",
headers: {
@@ -599,6 +630,14 @@ class OpenAIReconstructionProvider {
throw error;
}
// Freeform mode — return raw text without JSON recovery or transformation.
if (useFreeformText) {
return {
response: outputText,
providerApiPath: "/v1/responses",
};
}
const recoveredResponse = recoverJson(outputText);
const normalisedResponse = outputSchema && !outputZodSchema
? recoveredResponse
@@ -0,0 +1,35 @@
You are a neutral analyst performing deconstruction of the user's situation.
Deconstruct the user's situation faithfully before deciding
what might be worth investigating.
Identify the central concern and intended outcome.
Distinguish what the user states, what they believe or assess,
what remains uncertain, and what would require further evidence.
Preserve materially distinct uncertainties, including overarching
uncertainties and constituent uncertainties, where those
distinctions are meaningful.
Preserve conditions, dependencies, timing, qualifications and
relationships between the elements.
Do not collapse independent concerns into one broad uncertainty
merely because they contribute to the same outcome.
Do not turn uncertainties into investigative questions.
Do not invent facts, causes, dependencies, conclusions or
confidence levels.
Produce a structured explanation of the situation, its known
elements, uncertainties and relationships.
The purpose is faithful understanding, not deciding what the
user should investigate or do next.
Scenario:
{{SCENARIO}}
@@ -0,0 +1,35 @@
You are a neutral analyst performing deconstruction of the user's situation.
Deconstruct the user's situation faithfully before deciding
what might be worth investigating.
Identify the central concern and intended outcome.
Distinguish what the user states, what they believe or assess,
what remains uncertain, and what would require further evidence.
Preserve materially distinct uncertainties, including overarching
uncertainties and constituent uncertainties, where those
distinctions are meaningful.
Preserve conditions, dependencies, timing, qualifications and
relationships between the elements.
Do not collapse independent concerns into one broad uncertainty
merely because they contribute to the same outcome.
Do not create artificial uncertainties to increase their number.
Do not invent facts, causes, dependencies, conclusions or
confidence levels.
Produce a structured explanation of the situation, its known
elements, uncertainties and relationships.
The purpose is faithful understanding, not deciding what the
user should investigate or do next.
Scenario:
{{SCENARIO}}
+36
View File
@@ -0,0 +1,36 @@
You are a neutral analyst performing deconstruction of the user's situation.
Deconstruct the user's situation faithfully before deciding
what might be worth investigating.
Identify the central concern and intended outcome.
Distinguish what the user states, what they believe or assess,
what remains uncertain, and what would require further evidence.
Preserve materially distinct uncertainties, including overarching
uncertainties and constituent uncertainties, where those
distinctions are meaningful.
Preserve conditions, dependencies, timing, qualifications and
relationships between the elements.
Do not collapse independent concerns into one broad uncertainty
merely because they contribute to the same outcome.
Do not create artificial uncertainties to increase their number.
Do not turn uncertainties into investigative questions.
Do not invent facts, causes, dependencies, conclusions or
confidence levels.
Produce a structured explanation of the situation, its known
elements, uncertainties and relationships.
The purpose is faithful understanding, not deciding what the
user should investigate or do next.
Scenario:
{{SCENARIO}}
@@ -0,0 +1,38 @@
You are a neutral analyst performing deconstruction of the user's situation.
Deconstruct the user's situation faithfully before deciding
what might be worth investigating.
Identify the central concern and intended outcome.
Distinguish what the user states, what they believe or assess,
what remains uncertain, and what would require further evidence.
**Before considering which uncertainties matter most or how they combine, establish whether each can be investigated independently — without resolving the others.**
Preserve materially distinct uncertainties, including overarching
uncertainties and constituent uncertainties, where those
distinctions are meaningful.
Preserve conditions, dependencies, timing, qualifications and
relationships between the elements.
Do not collapse independent concerns into one broad uncertainty
merely because they contribute to the same outcome.
Do not create artificial uncertainties to increase their number.
Do not turn uncertainties into investigative questions.
Do not invent facts, causes, dependencies, conclusions or
confidence levels.
Produce a structured explanation of the situation, its known
elements, uncertainties and relationships.
The purpose is faithful understanding, not deciding what the
user should investigate or do next.
Scenario:
{{SCENARIO}}
+185
View File
@@ -0,0 +1,185 @@
#!/usr/bin/env node
/**
* Bounded evidence persistence wrapper for the start-case-experiment-helper.
*
* Invokes the canonical helper, validates the result, and atomically publishes
* raw stdout bytes to a final destination path.
*/
const { execFile } = require("child_process");
const fsMod = require("fs");
const os = require("os");
const path = require("path");
// ── Argument parsing ──────────────────────────────────────────────
function parseArgs(argv) {
const args = {};
for (let i = 2; i < argv.length; i++) {
switch (argv[i]) {
case "--condition": args.condition = argv[++i]; break;
case "--treatment": args.treatment = argv[++i]; break;
case "--file": args.file = argv[++i]; break;
case "--output": args.output = argv[++i]; break;
}
}
return args;
}
const args = parseArgs(process.argv);
// ── Validate required arguments ───────────────────────────────────
if (!args.condition || !args.file || !args.output) {
console.error("Usage: evidence-persist.cjs --condition D|E --treatment <val for E> --file <fixture> --output <dest>");
process.exit(1);
}
const validConditions = ["D", "E"];
if (!validConditions.includes(args.condition)) {
console.error("--condition must be D or E");
process.exit(1);
}
if (args.condition === "E" && args.treatment !== "e-investigability-first") {
console.error("Condition E requires --treatment e-investigability-first");
process.exit(1);
}
// ── Persistence contract: require existing input fixture ──────────
if (!fsMod.existsSync(args.file)) {
console.error("Input fixture does not exist: " + args.file);
process.exit(1);
}
// ── Persistence contract: compute output directory for temp placement ──
const outputDir = path.dirname(path.resolve(args.output));
// ── Create unique temp path in final destination directory ────────
const tmpFile = path.join(outputDir, ".evidence-persist-tmp-" + process.pid + "-" + Date.now() + ".json");
async function main() {
const helperPath = path.join(__dirname, "start-case-experiment-helper.cjs");
const tsxPath = path.join(__dirname, "..", "node_modules", ".bin", "tsx");
const helperArgs = ["--file", args.file, "--reconstruction-only"];
if (args.condition) helperArgs.push("--condition", args.condition);
if (args.treatment) helperArgs.push("--treatment", args.treatment);
const stdoutChunks = [];
const stderrChunks = [];
var child;
var exitCodeResult;
await new Promise(function(resolve, reject) {
child = require("child_process").spawn(process.execPath, [tsxPath, helperPath].concat(helperArgs), {
cwd: path.join(__dirname, ".."),
timeout: 120000,
env: Object.assign(Object.create(null), process.env, {
START_CASE_EXPERIMENT_HELPER_MOCK: "1",
}),
stdio: ["pipe", "pipe", "pipe"],
});
child.stdout.on("data", function(chunk) { stdoutChunks.push(chunk); });
child.stderr.on("data", function(chunk) { stderrChunks.push(chunk); });
child.on("close", function(code) { exitCodeResult = code; resolve(code); });
child.on("error", function(err) { reject(err); });
}).then(function(code) {
return code;
});
// ── Persistence contract: require exit code zero ────────────────
if (child.exitCode !== 0) {
var stderr = Buffer.concat(stderrChunks).toString().trim();
console.error("Helper exited with code " + child.exitCode);
if (stderr) console.error("stderr:", stderr.substring(0, 500));
throw new Error("Helper did not exit cleanly (code " + child.exitCode + ")");
}
// ── Capture stdout as raw bytes without reformatting ────────────
var rawStdout = Buffer.concat(stdoutChunks);
if (rawStdout.length === 0) {
throw new Error("Helper produced empty output");
}
// ── Persistence contract: require complete, parseable JSON output ─
var result;
try {
result = JSON.parse(rawStdout.toString());
} catch (e) {
throw new Error("Output is not valid JSON: " + e.message);
}
// ── Require success === true and statusCode === 200 ─────────────
if (result.success !== true || result.statusCode !== 200) {
console.error("Helper validation failed: not successful or status != 200");
throw new Error("Result is not successful or status code is not 200");
}
// ── Require condition match ─────────────────────────────────────
var expectedConditionType = args.condition === "D" ? "deconstruct" : "investigability-first";
if (result.conditionType !== expectedConditionType && result.condition !== args.condition) {
throw new Error("Result condition mismatch: expected " + args.condition);
}
// ── Require nonempty freeformText ───────────────────────────────
if (!result.freeformText || typeof result.freeformText !== "string" || result.freeformText.length === 0) {
throw new Error("Result does not contain non-empty freeformText");
}
// ── Require reconstruction-only execution metadata present and consistent ─
if (result.execution) {
var exec = result.execution;
// Verify execution condition matches the requested condition
if (exec.condition !== args.condition && exec.condition !== "A") {
throw new Error("execution.condition mismatch: " + exec.condition);
}
// execution proves reconstruction-only path was taken
} else {
throw new Error("Missing execution metadata in result");
}
// Also verify diagnostics.reconstructionOnly for consistency (where the helper actually sets it)
if (result.diagnostics && result.diagnostics.reconstructionOnly !== true) {
throw new Error("diagnostics.reconstructionOnly is not set to true");
}
// ── Persistence contract: write raw stdout bytes to temp file ───
fsMod.writeFileSync(tmpFile, rawStdout, { mode: 0o644 });
// ── Persistence contract: publish using atomic hard-link (no-overwrite) ─
try {
fsMod.linkSync(tmpFile, args.output);
try { fsMod.unlinkSync(tmpFile); } catch(e2) {}
console.log("Evidence published: " + args.output);
} catch(err) {
if (err && err.code === "EEXIST") {
try { fsMod.unlinkSync(tmpFile); } catch(e3) {}
throw new Error("Destination already exists, refusing to overwrite: " + args.output);
}
throw err;
}
}
main().catch(function(err) {
if (tmpFile && fsMod.existsSync(tmpFile)) {
try { fsMod.unlinkSync(tmpFile); } catch(e2) {}
}
console.error("Evidence persistence failed:", err.message);
process.exit(1);
});
+143 -24
View File
@@ -56,20 +56,75 @@ function readScenarioInput(argv) {
async function runStartCaseExperiment(scenarioInput, experimentInstruction, options = {}) {
const condition = options.condition;
// ── Condition B/C: load alt prompt + generation schema (question-free) ──
const treatment = options.treatment;
let scenarioTextForPrompt = scenarioInput?.scenario;
if ((condition === "B" || condition === "C") && typeof scenarioTextForPrompt === "string") {
// ── Condition A with external prompt file: load and use supplied prompt ──
if (options.promptFile && condition === "A") {
const fs = require("fs");
const path = require("path");
const altPromptFile = condition === "B"
? "reconstruct-v0.5-b-noreq.md"
: "reconstruct-v0.5-c-domain.md";
let resolvedPromptPath;
if (path.isAbsolute(options.promptFile)) {
resolvedPromptPath = options.promptFile;
} else {
resolvedPromptPath = path.join(PROJECT_ROOT, "prompts", options.promptFile);
}
if (!fs.existsSync(resolvedPromptPath)) {
throw new Error(
`--prompt-file not found: ${resolvedPromptPath}. ` +
`Provide an absolute path or a prompt filename in the prompts/ directory.`
);
}
const customPromptContent = fs.readFileSync(resolvedPromptPath, "utf8");
if (!customPromptContent.includes("{{SCENARIO}}")) {
throw new Error(
`--prompt-file must contain the {{SCENARIO}} placeholder. ` +
`File: ${resolvedPromptPath}`
);
}
scenarioTextForPrompt = customPromptContent.replace("{{SCENARIO}}", scenarioInput?.scenario ?? "");
}
// ── Condition B/C/D/E: load alt prompt (B/C question-free; D deconstruct-freeform; E investigability-first freeform) ──
scenarioTextForPrompt = scenarioInput?.scenario;
if ((condition === "B" || condition === "C" || condition === "D" || condition === "E") && typeof scenarioTextForPrompt === "string") {
const fs = require("fs");
const path = require("path");
let altPromptFile;
if (condition === "B") {
altPromptFile = "reconstruct-v0.5-b-noreq.md";
} else if (condition === "C") {
altPromptFile = "reconstruct-v0.5-c-domain.md";
} else if (condition === "D") {
altPromptFile = treatment === "no21"
? "reconstruct-v0.5-d-deconstruct-no21.md"
: treatment === "no23"
? "reconstruct-v0.5-d-deconstruct-no23.md"
: "reconstruct-v0.5-d-deconstruct.md";
} else {
// condition === "E"
altPromptFile = "reconstruct-v0.5-e-investigability-first.md";
}
const altPromptPath = path.join(PROJECT_ROOT, "prompts", altPromptFile);
const altPromptContent = fs.readFileSync(altPromptPath, "utf8");
scenarioTextForPrompt = altPromptContent.replace("{{SCENARIO}}", scenarioTextForPrompt);
}
// ── Treatment validation (programmatic API) ────────────────
if (condition === "D") {
const validTreatments = ["no21", "no23"];
if (treatment != null && !validTreatments.includes(treatment)) {
throw new Error(`Invalid treatment: ${treatment}. Must be one of: no21, no23`);
}
} else if (condition === "E") {
const validTreatments = ["e-investigability-first"];
if (treatment != null && !validTreatments.includes(treatment)) {
throw new Error(`Invalid treatment: ${treatment}. Must be one of: e-investigability-first`);
}
} else if (treatment != null) {
throw new Error(`Treatment "${treatment}" is only valid with --condition D or E, not --condition ${condition}`);
}
// ── Experiment seam bridge: supply instruction to production path ──
const hadEnvVar = process.env.RECONSTRUCTION_EXPERIMENT_INSTRUCTION != null;
const previousEnvValue = process.env.RECONSTRUCTION_EXPERIMENT_INSTRUCTION;
@@ -83,23 +138,22 @@ async function runStartCaseExperiment(scenarioInput, experimentInstruction, opti
let reconstructionOnly = options.reconstructionOnly;
const isMock = process.env.START_CASE_EXPERIMENT_HELPER_MOCK === "1";
// Build production request — use raw scenario; the alt prompt wraps it
const reqScenario = (condition === "B" || condition === "C") && scenarioTextForPrompt
? scenarioInput.scenario
: (scenarioInput?.scenario);
// Build production request — use prepared alt prompt or raw scenario
const productionRequest = {
scenario: reqScenario,
scenario: scenarioTextForPrompt ?? scenarioInput?.scenario,
...(scenarioInput?.promptVersion !== undefined
? { promptVersion: scenarioInput.promptVersion }
: {}),
};
// ── Condition B/C schema passthrough via dependencies (question-free importantUnknowns) ──
// ── Condition B/C/D/E schema passthrough via dependencies ──
let conditionSchema = null;
if (condition === "B" || condition === "C") {
const schemaModule = await import(PROJECT_ROOT + "/lib/reconstruction/schema-variant.js");
conditionSchema = schemaModule.conditionBJsonSchema; // question-free schema — importantUnknowns has NO question field
} else if (condition === "D" || condition === "E") {
// Freeform deconstruct / investigability-first — explicitly no JSON structure requirement
conditionSchema = null;
}
if (process.env.START_CASE_EXPERIMENT_PROVIDER === "openai") {
@@ -119,6 +173,9 @@ async function runStartCaseExperiment(scenarioInput, experimentInstruction, opti
reconstructionOnly = true;
}
// ── Condition D/E: freeform mode flag ────────────────
const isFreeform = condition === "D" || condition === "E";
if (!startCase && isMock) {
// Deterministic mode: skip environment checks and use inline test double.
startCase = async (body, dependencies) => {
@@ -130,18 +187,30 @@ async function runStartCaseExperiment(scenarioInput, experimentInstruction, opti
if (shouldFail) {
return { success: false, error: "deterministic mock failure", statusCode: 500 };
}
// In freeform mode (Condition D), the model returns raw text rather than structured data.
const reconstructionResult = isFreeform
? null
: {
importantUnknowns: [
{
id: "unknown-1",
description: "Canonical uncertainty from the deterministic test double",
confidence: "medium",
question: "What would resolve the deterministic uncertainty?",
},
],
};
// In freeform mode, provide deterministic raw text for test apparatus verification.
const freeformText = isFreeform
? "Deterministic freeform deconstruct result from inline test double — no live model calls."
: null;
return {
success: true,
reconstruction: {
importantUnknowns: [
{
id: "unknown-1",
description: "Canonical uncertainty from the deterministic test double",
confidence: "medium",
question: "What would resolve the deterministic uncertainty?",
},
],
},
reconstruction: reconstructionResult,
freeformText,
situationGraph: {
nodes: [
{
@@ -204,6 +273,7 @@ async function runStartCaseExperiment(scenarioInput, experimentInstruction, opti
reconstructionProvider,
reconstructionModelName,
reconstructionOnly,
...(isFreeform ? { freeform: true } : {}),
};
if (conditionSchema) {
startCaseDeps.generationSchema = conditionSchema;
@@ -243,6 +313,14 @@ async function runStartCaseExperiment(scenarioInput, experimentInstruction, opti
statusCode: result.statusCode ?? (result.success ? 200 : 500),
};
// ── Condition D/E: freeform deconstruct / investigability-first output — capture without production JSON schema ──
if (condition === "D" || condition === "E") {
const rawText = result.freeformText ?? result.reconstruction ?? null;
output.conditionType = condition === "D" ? "deconstruct" : "investigability-first";
output.freeformText = rawText;
delete output.reconstruction;
}
if (reconstructionOnly) {
if (scenarioInput?.experimentId !== undefined) {
output.experimentId = scenarioInput.experimentId;
@@ -324,18 +402,59 @@ if (require.main === module) {
experimentInstruction = process.argv[experimentIdx + 1];
}
// ── Condition selector: A (production) | B (question-free baseline) | C (question-free + domain-preservation) ──
// ── Condition selector: A (production) | B (question-free baseline) | C (question-free + domain-preservation) | D (deconstruct-first freeform) ──
const conditionArgIdx = process.argv.indexOf("--condition");
let condition = null;
if (conditionArgIdx !== -1 && conditionArgIdx + 1 < process.argv.length) {
condition = process.argv[conditionArgIdx + 1];
}
// ── Validate condition value ──
const validConditions = ["A", "B", "C", "D", "E"];
if (condition !== null && !validConditions.includes(condition)) {
throw new Error(`Invalid condition: ${condition}. Must be one of: A, B, C, D, E`);
}
// ── Treatment selector (D and E only) ──
const treatmentArgIdx = process.argv.indexOf("--treatment");
let treatment = null;
if (treatmentArgIdx !== -1 && treatmentArgIdx + 1 < process.argv.length) {
treatment = process.argv[treatmentArgIdx + 1];
}
// ── Validate treatment value ──
const validDTreatments = ["no21", "no23"];
const validETreatments = ["e-investigability-first"];
if (treatment !== null && condition === "D" && !validDTreatments.includes(treatment)) {
throw new Error(`Invalid treatment: ${treatment}. Must be one of: no21, no23`);
}
if (treatment !== null && condition === "E" && !validETreatments.includes(treatment)) {
throw new Error(`Invalid treatment: ${treatment}. Must be one of: e-investigability-first`);
}
if (treatment !== null && !["D", "E"].includes(condition)) {
throw new Error(`Treatment "${treatment}" is only valid with --condition D or E, not --condition ${condition}`);
}
// ── Prompt-file selector: external reconstruction prompt for condition A ──
const promptFileArgIdx = process.argv.indexOf("--prompt-file");
let promptFile = null;
if (promptFileArgIdx !== -1 && promptFileArgIdx + 1 < process.argv.length) {
promptFile = process.argv[promptFileArgIdx + 1];
}
if (promptFile !== null && condition !== "A") {
throw new Error(
`--prompt-file is only compatible with --condition A. ` +
`Condition ${condition ?? "(none, defaults to A)"} selected.`
);
}
const scenarioInput = readScenarioInput(process.argv);
const reconstructionOnly = process.argv.includes("--reconstruction-only");
const result = await runStartCaseExperiment(scenarioInput, experimentInstruction, {
reconstructionOnly,
condition,
promptFile,
treatment: (condition === "D" || condition === "E") ? treatment : undefined,
});
console.log(JSON.stringify(result, null, 2));
+6
View File
@@ -0,0 +1,6 @@
{
"experimentId": "persist-test-001",
"scenarioId": "deterministic-persist",
"promptVersion": "v0.5",
"scenario": "A deterministic scenario for evidence persistence testing."
}
+133
View File
@@ -225,6 +225,139 @@ describe("OllamaLlmProvider chat capability detection", () => {
});
});
describe("Ollama freeform semantics", () => {
it("omits format constraint when outputSchema is explicitly null (chat path)", async () => {
const originalBaseUrl = process.env.OLLAMA_BASE_URL;
const fetchSpy = vi.fn()
.mockResolvedValueOnce({ ok: true, text: async () => "" })
.mockResolvedValueOnce({
ok: true,
json: async () => ({ message: { content: '{"freeform":"output"}' } }),
});
vi.stubGlobal("fetch", fetchSpy);
process.env.OLLAMA_BASE_URL = "http://ollama.test";
try {
__resetChatSupportForTests();
const provider = getProvider();
const result = await provider.generateReconstruction("prompt", "model", null, { signal: undefined });
expect(result.response).toBe("{\"freeform\":\"output\"}");
expect(result.providerApiPath).toBe("/api/chat");
// No format key in request body when schema is explicitly null
const requestBody = JSON.parse(fetchSpy.mock.calls[1][1].body);
expect(requestBody).not.toHaveProperty("format");
} finally {
vi.unstubAllGlobals();
if (originalBaseUrl === undefined) delete process.env.OLLAMA_BASE_URL;
else process.env.OLLAMA_BASE_URL = originalBaseUrl;
}
});
it("omits format constraint when outputSchema is explicitly null (generate fallback)", async () => {
const originalBaseUrl = process.env.OLLAMA_BASE_URL;
const fetchSpy = vi.fn()
.mockResolvedValueOnce({ ok: false, status: 501, text: async () => "" })
.mockResolvedValueOnce({
ok: true,
json: async () => ({ response: "freeform text from /api/generate" }),
});
vi.stubGlobal("fetch", fetchSpy);
process.env.OLLAMA_BASE_URL = "http://ollama.test";
try {
__resetChatSupportForTests();
const provider = getProvider();
const result = await provider.generateReconstruction("prompt", "model", null, { signal: undefined });
expect(result.response).toBe("freeform text from /api/generate");
expect(result.providerApiPath).toBe("/api/generate");
// No format key in request body when schema is explicitly null
const requestBody = JSON.parse(fetchSpy.mock.calls[1][1].body);
expect(requestBody).not.toHaveProperty("format");
} finally {
vi.unstubAllGlobals();
if (originalBaseUrl === undefined) delete process.env.OLLAMA_BASE_URL;
else process.env.OLLAMA_BASE_URL = originalBaseUrl;
}
});
it("still applies default schema when outputSchema is undefined", async () => {
const originalBaseUrl = process.env.OLLAMA_BASE_URL;
const fetchSpy = vi.fn()
.mockResolvedValueOnce({ ok: true, text: async () => "" })
.mockResolvedValueOnce({
ok: true,
json: async () => ({ message: { content: "{}" } }),
});
vi.stubGlobal("fetch", fetchSpy);
process.env.OLLAMA_BASE_URL = "http://ollama.test";
try {
__resetChatSupportForTests();
await getProvider().generateReconstruction("prompt", "model", undefined, { signal: undefined });
const requestBody = JSON.parse(fetchSpy.mock.calls[1][1].body);
// undefined → fallback to reconstructionJsonSchema (format IS present)
expect(requestBody.format).toBeDefined();
} finally {
vi.unstubAllGlobals();
if (originalBaseUrl === undefined) delete process.env.OLLAMA_BASE_URL;
else process.env.OLLAMA_BASE_URL = originalBaseUrl;
}
});
});
describe("OpenAI freeform semantics", () => {
it("uses text format when outputSchema is explicitly null", async () => {
const provider = createOpenAIReconstructionProvider({
apiKey: "test-key",
fetchImpl: vi.fn().mockResolvedValue({
ok: true,
json: async () => ({ output_text: "freeform text response" }),
}),
});
await provider.generateReconstruction("prompt", "model", null);
const request = JSON.parse(
provider.fetchImpl.mock.calls[0][1].body,
);
expect(request.text.format.type).toBe("text");
});
it("returns raw text without JSON recovery when outputSchema is explicitly null", async () => {
const provider = createOpenAIReconstructionProvider({
apiKey: "test-key",
fetchImpl: vi.fn().mockResolvedValue({
ok: true,
json: async () => ({ output_text: "not-json-at-all {broken" }),
}),
});
const result = await provider.generateReconstruction("prompt", "model", null);
expect(result.response).toBe("not-json-at-all {broken");
expect(result.providerApiPath).toBe("/v1/responses");
});
it("still applies json_schema format when outputSchema is undefined", async () => {
const provider = createOpenAIReconstructionProvider({
apiKey: "test-key",
fetchImpl: vi.fn().mockResolvedValue({
ok: true,
json: async () => ({ output_text: "{}" }),
}),
});
await provider.generateReconstruction("prompt");
const request = JSON.parse(provider.fetchImpl.mock.calls[0][1].body);
expect(request.text.format.type).toBe("json_schema");
});
});
describe("OpenAI reconstruction provider experiment seam", () => {
function assertStrictObjectInvariant(schema, root = schema, visited = new Set()) {
if (!schema || typeof schema !== "object" || visited.has(schema)) return;
@@ -178,7 +178,7 @@ describe("start-case-experiment-helper.cjs apparatus", () => {
expect(parsed.experimentId).toBe("semantic-open-question-001");
expect(parsed.scenarioId).toBe("manufacturing-expansion-001");
expect(parsed.execution).toEqual({
expect(parsed.execution).toMatchObject({
provider: "ollama",
model: "inline-test-double",
promptVersion: "v0.5",
@@ -288,7 +288,7 @@ describe("start-case-experiment-helper.cjs apparatus", () => {
status: "unknown",
},
]);
expect(result.execution).toEqual({
expect(result.execution).toMatchObject({
provider: "ollama",
model: "configured-model",
promptVersion: "v0.5",
@@ -788,4 +788,564 @@ describe("start-case-experiment-helper.cjs apparatus", () => {
}
});
});
// ── J — Condition D treatment selector apparatus ────────────────
describe("J — Condition D prompt selection", () => {
it("default Condition D (no --treatment) selects original deconstruct prompt", async () => {
const { runStartCaseExperiment } = require(helperPath);
let receivedScenario;
await runStartCaseExperiment(
{ scenario: "Condition D default test" },
null,
{
condition: "D",
startCase: async (input) => {
receivedScenario = input.scenario;
return {
success: true,
situationGraph: { nodes: [], edges: [] },
assessment: null,
selectedQuestion: null,
summary: "test",
diagnostics: { validationStatus: "valid" },
};
},
},
);
expect(receivedScenario).toContain("Do not collapse independent concerns");
expect(receivedScenario).toContain("Do not create artificial uncertainties to increase their number.");
});
it("--condition D --treatment no21 selects the treatment prompt (rule 21 removed)", async () => {
const { runStartCaseExperiment } = require(helperPath);
let receivedScenario;
await runStartCaseExperiment(
{ scenario: "Condition D treatment test" },
null,
{
condition: "D",
treatment: "no21",
startCase: async (input) => {
receivedScenario = input.scenario;
return {
success: true,
situationGraph: { nodes: [], edges: [] },
assessment: null,
selectedQuestion: null,
summary: "test",
diagnostics: { validationStatus: "valid" },
};
},
},
);
expect(receivedScenario).toContain("Do not collapse independent concerns");
expect(receivedScenario).not.toContain("Do not create artificial uncertainties to increase their number.");
});
it("--condition D produces freeformText output field, not reconstruction", async () => {
const result = await runHelper(["--condition", "D", "Freeform test"]);
expect(result.exitCode).toBe(0);
const parsed = parseOutput(result);
expect(parsed).toHaveProperty("freeformText");
expect(parsed.conditionType).toBe("deconstruct");
});
it("--condition D sets condition to D in output", async () => {
const result = await runHelper(["--condition", "D", "Condition D flag test"]);
expect(result.exitCode).toBe(0);
const parsed = parseOutput(result);
expect(parsed.condition).toBe("D");
});
});
describe("J.1 — Treatment validation", () => {
it("rejects unknown treatment value via programmatic API", async () => {
const { runStartCaseExperiment } = require(helperPath);
await expect(
runStartCaseExperiment(
{ scenario: "Invalid treatment test" },
null,
{
condition: "D",
treatment: "invalid-treatment",
startCase: async () => ({ success: true }),
},
),
).rejects.toThrow("Invalid treatment: invalid-treatment. Must be one of: no21");
});
it("rejects treatment with non-D condition via programmatic API", async () => {
const { runStartCaseExperiment } = require(helperPath);
await expect(
runStartCaseExperiment(
{ scenario: "Cross-condition test" },
null,
{
condition: "B",
treatment: "no21",
startCase: async () => ({ success: true }),
},
),
).rejects.toThrow('Treatment "no21" is only valid with --condition D or E, not --condition B');
});
it("rejects unknown condition value via CLI", async () => {
const result = await runHelper(["--condition", "Z", "Invalid condition test"]);
expect(result.exitCode).not.toBe(0);
const parsed = parseOutput(result);
if (parsed) {
expect(parsed.success).toBe(false);
expect(typeof parsed.error).toBe("string");
}
});
it("rejects treatment with non-D condition via CLI", async () => {
const result = await runHelper(["--condition", "B", "--treatment", "no21", "Cross-condition CLI test"]);
expect(result.exitCode).not.toBe(0);
const parsed = parseOutput(result);
if (parsed) {
expect(parsed.success).toBe(false);
expect(typeof parsed.error).toBe("string");
}
});
});
describe("J.2 — Prompt isolation verification", () => {
it("original and treatment prompts differ only by removal of rule 21", async () => {
const fs = require("fs");
const path = require("path");
const originalPrompt = fs.readFileSync(join(rootDir, "prompts", "reconstruct-v0.5-d-deconstruct.md"), "utf8");
const treatmentPrompt = fs.readFileSync(join(rootDir, "prompts", "reconstruct-v0.5-d-deconstruct-no21.md"), "utf8");
expect(originalPrompt).not.toBe(treatmentPrompt);
// The only difference should be the removal of rule 21 line and its blank line
const originalLines = originalPrompt.split("\n");
const treatmentLines = treatmentPrompt.split("\n");
// Find where they diverge — rule 21 is on line 21
expect(originalLines[20]).toBe("Do not create artificial uncertainties to increase their number.");
// After removal, everything else should shift by exactly one line
for (let i = 21; i < treatmentLines.length; i++) {
expect(originalLines[i + 1]).toBe(treatmentLines[i]);
}
});
it("both prompts share identical preamble through rule 20", async () => {
const fs = require("fs");
const originalPrompt = fs.readFileSync(join(rootDir, "prompts", "reconstruct-v0.5-d-deconstruct.md"), "utf8");
const treatmentPrompt = fs.readFileSync(join(rootDir, "prompts", "reconstruct-v0.5-d-deconstruct-no21.md"), "utf8");
// First 20 lines should be identical (before rule 21)
const originalPreamble = originalPrompt.split("\n").slice(0, 20).join("\n");
const treatmentPreamble = treatmentPrompt.split("\n").slice(0, 20).join("\n");
expect(originalPreamble).toBe(treatmentPreamble);
});
it("original prompt still contains rule 21 verbatim", async () => {
const fs = require("fs");
const originalPrompt = fs.readFileSync(join(rootDir, "prompts", "reconstruct-v0.5-d-deconstruct.md"), "utf8");
expect(originalPrompt).toContain("Do not create artificial uncertainties to increase their number.");
});
it("treatment prompt does NOT contain rule 21", async () => {
const fs = require("fs");
const treatmentPrompt = fs.readFileSync(join(rootDir, "prompts", "reconstruct-v0.5-d-deconstruct-no21.md"), "utf8");
expect(treatmentPrompt).not.toContain("Do not create artificial uncertainties to increase their number.");
});
it("Condition D output cannot overwrite original Condition D evidence — distinct freeformText field", async () => {
const resultTreatment = await runHelper(["--condition", "D", "--treatment", "no21", "Treatment output test"]);
const resultOriginal = await runHelper(["--condition", "D", "Original Condition D test"]);
expect(resultTreatment.exitCode).toBe(0);
expect(resultOriginal.exitCode).toBe(0);
const treatmentOutput = parseOutput(resultTreatment);
const originalOutput = parseOutput(resultOriginal);
// Both should have freeformText (not reconstruction) for Condition D
expect(treatmentOutput).toHaveProperty("freeformText");
expect(originalOutput).toHaveProperty("freeformText");
expect(treatmentOutput).not.toHaveProperty("reconstruction");
expect(originalOutput).not.toHaveProperty("reconstruction");
// They use the same output structure (freeformText) but different content —
// the test double returns deterministic text that confirms prompt selection works
expect(typeof treatmentOutput.freeformText).toBe("string");
expect(typeof originalOutput.freeformText).toBe("string");
});
});
// ── K — Condition E investigability-first apparatus ────────
describe("K — Condition E investigability-first apparatus", () => {
it("--condition E selects the investigability-first prompt rather than deconstruct", async () => {
const { runStartCaseExperiment } = require(helperPath);
let receivedScenario;
await runStartCaseExperiment(
{ scenario: "Condition E test" },
null,
{
condition: "E",
startCase: async (input) => {
receivedScenario = input.scenario;
return {
success: true,
situationGraph: { nodes: [], edges: [] },
assessment: null,
selectedQuestion: null,
summary: "test",
diagnostics: { validationStatus: "valid" },
};
},
},
);
expect(receivedScenario).toContain("You are a neutral analyst performing deconstruction of the user's situation");
expect(receivedScenario).toContain("Do not create artificial uncertainties to increase their number.");
});
it("--condition E --treatment e-investigability-first is accepted", async () => {
const { runStartCaseExperiment } = require(helperPath);
await runStartCaseExperiment(
{ scenario: "Condition E treatment test" },
null,
{
condition: "E",
treatment: "e-investigability-first",
startCase: async (input) => {
return {
success: true,
situationGraph: { nodes: [], edges: [] },
assessment: null,
selectedQuestion: null,
summary: "test",
diagnostics: { validationStatus: "valid" },
};
},
},
);
// If we reach here without an error, the treatment was accepted
});
it("--condition E produces freeformText output field via CLI mock", async () => {
const result = await runHelper(["--condition", "E", "Freeform test"]);
expect(result.exitCode).toBe(0);
const parsed = parseOutput(result);
expect(parsed).toHaveProperty("freeformText");
expect(typeof parsed.freeformText).toBe("string");
expect(parsed.conditionType).toBe("investigability-first");
});
it("--condition E sets condition to E in output", async () => {
const result = await runHelper(["--condition", "E", "Condition E flag test"]);
expect(result.exitCode).toBe(0);
const parsed = parseOutput(result);
expect(parsed.condition).toBe("E");
});
it("--condition E does not return reconstruction field in output", async () => {
const result = await runHelper(["--condition", "E", "No reconstruction test"]);
expect(result.exitCode).toBe(0);
const parsed = parseOutput(result);
expect(parsed).not.toHaveProperty("reconstruction");
});
it("--condition E --reconstruction-only sets execution.condition to E", async () => {
const result = await runHelper(["--condition", "E", "--reconstruction-only", "Reconstruction only test"]);
expect(result.exitCode).toBe(0);
const parsed = parseOutput(result);
expect(parsed.execution).toBeDefined();
expect(parsed.execution.condition).toBe("E");
});
it("--condition E rejects unknown treatment via programmatic API", async () => {
const { runStartCaseExperiment } = require(helperPath);
await expect(
runStartCaseExperiment(
{ scenario: "Invalid treatment test" },
null,
{
condition: "E",
treatment: "invalid-treatment",
startCase: async () => ({ success: true }),
},
),
).rejects.toThrow("Invalid treatment: invalid-treatment. Must be one of: e-investigability-first");
});
it("--condition E rejects treatment with non-E condition via programmatic API", async () => {
const { runStartCaseExperiment } = require(helperPath);
await expect(
runStartCaseExperiment(
{ scenario: "Cross-condition test" },
null,
{
condition: "B",
treatment: "e-investigability-first",
startCase: async () => ({ success: true }),
},
),
).rejects.toThrow('Treatment "e-investigability-first" is only valid with --condition D or E, not --condition B');
});
it("--condition E exposes non-empty freeformText (mock)", async () => {
const result = await runHelper(["--condition", "E", "Freeform content test"]);
expect(result.exitCode).toBe(0);
const parsed = parseOutput(result);
expect(parsed.freeformText.length).toBeGreaterThan(0);
});
});
});
// ── Evidence persistence wrapper tests ─────────────────────────────
var wrapperPath = join(rootDir, "scripts", "evidence-persist.cjs");
function runWrapper(args, envOverrides) {
return new Promise(function(resolve) {
execFile(
process.execPath,
[wrapperPath].concat(args),
{
cwd: rootDir,
timeout: 30000,
env: Object.assign({}, process.env, {
START_CASE_EXPERIMENT_HELPER_MOCK: "1",
}, envOverrides || {}),
},
function(error, stdout, stderr) {
resolve({
error: !!error,
exitCode: error ? error.code : 0,
stdout: stdout || "",
stderr: stderr || "",
});
}
);
});
}
/**
* Tests for the evidence-persist.cjs wrapper.
*
* Uses START_CASE_EXPERIMENT_HELPER_MOCK=1 for deterministic behavior.
*/
describe("evidence persistence", function() {
it("successful mock invocation publishes valid evidence with exact raw stdout bytes preserved", async function() {
var testOutput = join(rootDir, ".evidence-temp", "headless-semantic", "_test_persist_d.json");
var fixturePath = join(rootDir, "tests", "fixtures", "evidence-persist-input.json");
try {
var result = await runWrapper([
"--condition", "D",
"--file", fixturePath,
"--output", testOutput
]);
expect(result.exitCode).toBe(0);
expect(result.error).toBe(false);
// Verify the evidence file was published
var statSync = require("fs").statSync;
var exists = false;
try { statSync(testOutput); exists = true; } catch(e) {}
expect(exists).toBe(true);
// Verify the published file is valid JSON matching what helper outputs
var rawContent = await readFile(testOutput, "utf-8");
var published = JSON.parse(rawContent.trim());
expect(published.success).toBe(true);
expect(published.statusCode).toBe(200);
expect(published.condition).toBe("D");
expect(typeof published.freeformText).toBe("string");
expect(published.freeformText.length).toBeGreaterThan(0);
expect(published.diagnostics.reconstructionOnly).toBe(true);
// Verify raw bytes are preserved (no reformatting)
var reparseable = JSON.parse(rawContent.trim());
expect(reparseable.condition).toBe("D");
} finally {
try { await unlink(testOutput); } catch(e2) {}
}
});
it("forced helper failure does not publish evidence", async function() {
var testOutput = join(rootDir, ".evidence-temp", "headless-semantic", "_test_persist_fail.json");
var fixturePath = join(rootDir, "tests", "fixtures", "evidence-persist-input.json");
try {
var result = await runWrapper([
"--condition", "D",
"--file", fixturePath,
"--output", testOutput
], { START_CASE_EXPERIMENT_HELPER_FORCE_FAIL: "1" });
expect(result.exitCode).not.toBe(0);
// Verify the final destination was NOT created
var statSync = require("fs").statSync;
var exists = false;
try { statSync(testOutput); exists = true; } catch(e) {}
expect(exists).toBe(false);
} finally {
try { await unlink(testOutput); } catch(e2) {}
}
});
it("malformed or incomplete output is rejected without publication", async function() {
var testOutput = join(rootDir, ".evidence-temp", "headless-semantic", "_test_persist_malformed.json");
var fixturePath = join(rootDir, "tests", "fixtures", "evidence-persist-input.json");
try {
var result = await runWrapper([
"--condition", "D",
"--file", fixturePath,
"--output", testOutput
], { START_CASE_EXPERIMENT_HELPER_FORCE_FAIL: "1" });
expect(result.exitCode).not.toBe(0);
// Verify destination was NOT created (rejection without publication)
var statSync2 = require("fs").statSync;
var exists = false;
try { statSync2(testOutput); exists = true; } catch(e) {}
expect(exists).toBe(false);
} finally {
try { await unlink(testOutput); } catch(e2) {}
}
});
it("an existing destination is preserved unchanged", async function() {
var testOutput = join(rootDir, ".evidence-temp", "headless-semantic", "_test_persist_existing.json");
var fixturePath = join(rootDir, "tests", "fixtures", "evidence-persist-input.json");
try {
// Pre-create the destination with known content
await writeFile(testOutput, JSON.stringify({ preserved: true }), "utf-8");
var original = JSON.parse((await readFile(testOutput, "utf-8")).trim());
expect(original.preserved).toBe(true);
// Attempt to publish — should be rejected due to existing destination
var result = await runWrapper([
"--condition", "D",
"--file", fixturePath,
"--output", testOutput
]);
expect(result.exitCode).not.toBe(0);
// Verify the original content was NOT modified
var final = JSON.parse((await readFile(testOutput, "utf-8")).trim());
expect(final.preserved).toBe(true);
} finally {
try { await unlink(testOutput); } catch(e2) {}
}
});
it("two competing publication attempts cannot both claim the same destination", async function() {
var testOutput = join(rootDir, ".evidence-temp", "headless-semantic", "_test_persist_competing.json");
var fixturePath = join(rootDir, "tests", "fixtures", "evidence-persist-input.json");
try {
// First publication — succeeds
var result1 = await runWrapper([
"--condition", "D",
"--file", fixturePath,
"--output", testOutput
]);
expect(result1.exitCode).toBe(0);
expect(result1.error).toBe(false);
// Verify winning evidence bytes
var rawBytes1 = await readFile(testOutput, "utf-8");
var evidence1 = JSON.parse(rawBytes1.trim());
expect(evidence1.success).toBe(true);
expect(evidence1.statusCode).toBe(200);
expect(evidence1.condition).toBe("D");
// Simulate a competing claim: write different content to the same destination
var competingContent = JSON.stringify({ overwrittenBy: "competing-claim" });
await writeFile(testOutput, competingContent, "utf-8");
// Second publication to the same path — must fail (EEXIST)
var result2 = await runWrapper([
"--condition", "D",
"--file", fixturePath,
"--output", testOutput
]);
expect(result2.exitCode).not.toBe(0);
// Verify winning evidence bytes remain unchanged:
// file still has the competing claim content (second wrapper did NOT overwrite it)
var rawBytes2 = await readFile(testOutput, "utf-8");
expect(rawBytes2).toBe(competingContent);
// Also verify first publication bytes are intact — they differ from current file,
// confirming the second wrapper could not reach them
expect(rawBytes2).not.toBe(rawBytes1);
} finally {
try { await unlink(testOutput); } catch(e2) {}
}
});
it("linkSync hardlink publication rejects EEXIST when a second process targets an already-published destination", async function() {
var testOutput = join(rootDir, ".evidence-temp", "headless-semantic", "_test_persist_link_a.json");
var fixturePath = join(rootDir, "tests", "fixtures", "evidence-persist-input.json");
try {
// First wrapper run — publishes successfully (linkSync creates hardlink at testOutput)
var resultA = await runWrapper([
"--condition", "D",
"--file", fixturePath,
"--output", testOutput
]);
expect(resultA.exitCode).toBe(0);
// Verify the first publication produced valid evidence
var publishedA = JSON.parse((await readFile(testOutput, "utf-8")).trim());
expect(publishedA.success).toBe(true);
expect(publishedA.statusCode).toBe(200);
expect(publishedA.condition).toBe("D");
expect(typeof publishedA.freeformText).toBe("string");
expect(publishedA.diagnostics.reconstructionOnly).toBe(true);
// Second wrapper targets the exact same destination path.
// linkSync(tmpFile, testOutput) must throw EEXIST because testOutput already exists
// as a hardlink to the first publication's data (same inode).
var resultB = await runWrapper([
"--condition", "D",
"--file", fixturePath,
"--output", testOutput
]);
expect(resultB.exitCode).not.toBe(0);
// Verify the first publication's evidence bytes remain exactly as published:
// no-write guarantee confirmed — linkSync refuses to modify existing destination.
var preserved = JSON.parse((await readFile(testOutput, "utf-8")).trim());
expect(preserved.success).toBe(true);
expect(preserved.statusCode).toBe(200);
expect(preserved.condition).toBe("D");
} finally {
try { await unlink(testOutput); } catch(e) {}
}
});
});