Compare commits
7
Commits
d5a06b9f28
...
8530d954a0
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
8530d954a0 | ||
|
|
fe0f337a99 | ||
|
|
e595ee6e02 | ||
|
|
29cdc80f09 | ||
|
|
8dcdffce44 | ||
|
|
74936851ce | ||
|
|
5af3e62ddd |
@@ -0,0 +1,66 @@
|
||||
# Uncertainty Information Flow Audit — Programme Handoff
|
||||
|
||||
**Working title:** Confidence Engine — Uncertainty Information Flow Audit
|
||||
**Date prepared:** 2026-10-14
|
||||
**Prepared for:** Next independent investigation (following the completed 10-iteration uncertainty recognition quality investigation)
|
||||
|
||||
---
|
||||
|
||||
## Primary Question
|
||||
|
||||
"At which point, if any, does Confidence Engine constrain, transform, discard or fail to preserve meaningful uncertainty recognition?"
|
||||
|
||||
## Context
|
||||
|
||||
The previous 10-iteration programme demonstrated a quality gap between minimal direct-model uncertainty recognition and Confidence Engine-mediated uncertainty recognition. The precise limiting mechanism remains unresolved (see `uncertainty-recognition-investigation-report.md` sections D–G). This new programme is designed to map the actual production information flow and identify specific intervention points without presuming failure locations.
|
||||
|
||||
## Investigation Boundaries (Not Presumed Failure Points)
|
||||
|
||||
1. User input and scenario representation
|
||||
2. Prompt construction
|
||||
3. Raw LLM request and response
|
||||
4. Parsing and validation
|
||||
5. Internal graph representation
|
||||
6. User-facing presentation
|
||||
|
||||
These boundaries define investigation scope, not expectations of where problems will be found.
|
||||
|
||||
## Recommended Approach
|
||||
|
||||
- Up to 10 iterations
|
||||
- One bounded investigative question per iteration
|
||||
- ChatGPT controls programme progression between iterations
|
||||
- Claude Code executes one task per invocation and reports evidence
|
||||
- Each `/rdbHandoff` invocation is a fresh session — work from repository evidence and this handoff
|
||||
- Read-only investigation before any interventions
|
||||
- Controlled experiments only when justified by evidence from prior iterations
|
||||
- Preserve raw evidence and request provenance for every iteration
|
||||
- No speculative product changes
|
||||
|
||||
## First Iteration Guidance
|
||||
|
||||
The first iteration should establish the actual production information flow and available capture boundaries:
|
||||
|
||||
1. Map the complete code path from scenario file input to model request construction
|
||||
2. Document what data is accessible at each stage of the pipeline (raw scenario, post-parsing, pre-template, post-template, pre-request, post-response, post-parsing)
|
||||
3. Identify any existing hooks, logs, or capture mechanisms for raw requests/responses
|
||||
4. Determine which stages can produce evidence without source modifications
|
||||
|
||||
The output should be a factual map of available capture points and data accessibility, not an analysis of what is "wrong."
|
||||
|
||||
## Existing Evidence to Consider
|
||||
|
||||
- `docs/uncertainty-recognition-investigation-report.md` — Full report from the previous programme
|
||||
- `.evidence-temp/iteration10/architecture-finding.txt` — Definitive CE prompt-path analysis showing template applicability at every invocation point
|
||||
- `.evidence-temp/headless-semantic/` — Extensive supporting evidence from HS series experiments
|
||||
|
||||
## What This Programme Is Not
|
||||
|
||||
- A redesign of Confidence Engine
|
||||
- A directive to change the model or provider configuration
|
||||
- An assumption that any specific boundary is broken or deficient
|
||||
- A continuation of the previous programme's quality-gap investigation (that closed at Iteration 10)
|
||||
|
||||
## Start Signal
|
||||
|
||||
Do not begin this programme from this document alone. Wait for an explicit `/rdbHandoff` invocation containing the start signal and first iteration directive.
|
||||
@@ -0,0 +1,147 @@
|
||||
# Uncertainty Recognition Quality Investigation — Report
|
||||
|
||||
**Date:** 2026-10-14
|
||||
**Branch:** `experiment/uncertainty-led-investigation`
|
||||
**Starting HEAD:** `e595ee6` feat(experiments): integrate Condition D/E apparatus
|
||||
**Status:** CLOSED — 10-iteration programme completed and checkpointed
|
||||
|
||||
---
|
||||
|
||||
## A. Investigation Objective
|
||||
|
||||
This investigation examined why the configured local LLM (`qwen-claude:latest` via Ollama) produced less useful uncertainty recognition through Confidence Engine's reconstruction pipeline than through a minimal direct prompt.
|
||||
|
||||
The canonical scenario used was `RFG-03-independent-decisions-input.json`, concerning three independent decisions (replacing internal reporting software, recruiting an additional customer-support employee, refurbishing a meeting room). The user's goal is to understand what remains uncertain before deciding what to do.
|
||||
|
||||
**Observed phenomenon:** A minimal direct-model prompt produced two outputs containing approximately 21 and 15 uncertainties with meaningful domain-specific unknowns. Confidence Engine reconstruction under Conditions D/E generally produced three generic uncertainties that largely restated the user's stated decisions without adding new insights.
|
||||
|
||||
---
|
||||
|
||||
## B. Experimental Programme (Iterations 1–10)
|
||||
|
||||
### Iteration 3 — Baseline Direct-Model Prompt
|
||||
**Status:** Complete and reproducible
|
||||
**Evidence:** `.evidence-temp/experiment3/`
|
||||
**Prompt:** Minimal raw prompt with no rule-set framing or forced JSON output.
|
||||
**Result:** Two model outputs: raw-run-1 (29 lines, ~15+ domain-specific uncertainties), raw-run-2 (21 lines, similar richness). Scenario SHA-256: `dc9a600f9fb1e7189dd56086a07f357893396a5b0782f4781c95d4aa29fdcf06`.
|
||||
|
||||
### Iteration 5 — Condition D Rule-21 Removal
|
||||
**Status:** Complete
|
||||
**Evidence:** `.evidence-temp/iteration5/`
|
||||
**Treatment:** Removed rule 21 from the deconstruction prompt.
|
||||
**Result:** Four CE uncertainties including a cross-domain "worthwhile" framework question. No material improvement over control in domain-specific discovery. Evidence: `D-no21-freeform.txt` (103 lines), `D-no21.json`, `D-original-freeform.txt`, `D-original.json`.
|
||||
|
||||
### Iteration 6 — Condition D Rule-23 Removal + Prompt Variant
|
||||
**Status:** Complete
|
||||
**Evidence:** `.evidence-temp/iteration6/`
|
||||
**Treatment:** Removed rule 23; tested with new prompt variant (`reconstruct-v0.5-d-deconstruct-no23.md`).
|
||||
**Result:** No material improvement. Both `run-A-original-D.json` and `run-B-no-rule-23.json` validated as freeform with zero graph nodes.
|
||||
|
||||
### Iteration 8 — Invalid Run + Recovery
|
||||
**Status:** Validated invalidity; recovery run complete
|
||||
**Evidence:** `.evidence-temp/iteration8/` (invalid), `.evidence-temp/iteration8-recovery/` (valid)
|
||||
**Invalidity cause:** Scenario filename was treated as scenario text instead of JSON content.
|
||||
**Recovery result:** 4 uncertainty items with modest structural enrichment (expectedStates, differences, tighter relationships) but no new domain-specific unknowns.
|
||||
|
||||
### Iterations 9–10 — Follow-up and Block
|
||||
**Iteration 9:** Partial evidence; focused-investigation dependency analysis. `.evidence-temp/iteration9/` contains a `b-variant/` subdirectory (incomplete).
|
||||
**Iteration 10:** BLOCKED before model execution. The existing experimental helper (`scripts/start-case-experiment-helper.cjs`) has no mechanism to pass the exact minimal raw prompt through the CE execution path without source modifications. Every invocation applies a structured prompt template. See `.evidence-temp/iteration10/final-report.txt` and `architecture-finding.txt` for the definitive architecture analysis.
|
||||
|
||||
### Headless-Semantic Evidence (HS-01 through HS-03C)
|
||||
**Status:** Extensive supporting evidence preserved
|
||||
**Evidence:** `.evidence-temp/headless-semantic/` (28 files, 2788 lines total)
|
||||
Covers: Question-free reconstruction comparison (CE-ULI-09), domain-preservation instruction tests (CE-ULI-10, CE-ULI-11), and focused-investigation evidence persistence testing.
|
||||
|
||||
---
|
||||
|
||||
## C. Established Findings
|
||||
|
||||
These findings are directly supported by the evidence:
|
||||
|
||||
1. **Direct-model prompt quality gap:** Minimal direct prompts produced outputs with approximately 21 and 15 domain-specific uncertainties across three decisions, while CE reconstruction under Conditions D/E generally produced three generic restatements of the user's own decisions.
|
||||
|
||||
2. **Rule removal had no material effect:** Removing Rule 21 (Iteration 5) and Rule 23 (Iteration 6) did not materially improve uncertainty recognition quality.
|
||||
|
||||
3. **Discovery permission instruction produced modest structural enrichment:** The Iteration 8 recovery (adding explicit permission to recognise plausible unstated uncertainties) added four items including a cross-domain evaluation framework question, but no new domain-specific unknowns were discovered beyond what the user stated.
|
||||
|
||||
4. **CE execution path is structurally incapable of raw-prompt pass-through:** Every invocation applies a template-based prompt construction that adds rule sets and forced-JSON constraints absent from the direct-model baseline. This was definitively established in Iteration 10's architecture-finding evidence.
|
||||
|
||||
5. **Question-free reconstruction does not prevent structural compression:** CE-ULI-10 demonstrated that removing the question-generation requirement still produced structural compression of uncertainties into single compound descriptions for the tested scenario shape.
|
||||
|
||||
6. **Explicit domain-separation instructions are ignored:** CE-ULI-11 showed identical compression under both condition B and C despite explicit instruction not to combine domains.
|
||||
|
||||
---
|
||||
|
||||
## D. Unresolved Hypotheses
|
||||
|
||||
**H1 — Model capability:** The model may be capable of richer uncertainty recognition but its output quality is constrained by the surrounding apparatus (prompt structure, output format constraints). *Untested.* Not equivalent to H2 or H3.
|
||||
|
||||
**H2 — Prompt effectiveness:** The template-based prompt construction (rule sets, forced JSON, deconstruction framing) suppresses the model's ability to produce rich uncertainty recognition relative to a minimal direct prompt. *Supported by evidence of quality gap but not proven as causal.* No controlled experiment eliminated the structural difference while preserving all other conditions.
|
||||
|
||||
**H3 — Confidence Engine orchestration:** Something in the CE pipeline (prompt construction, parsing, validation, graph representation, or presentation) constrains, transforms, or discards meaningful uncertainty recognition between model output and user-facing display. *Supported by evidence but not isolated.* Iteration 10 established that the CE execution path always applies templates, making controlled elimination of specific pipeline stages infeasible without source modifications.
|
||||
|
||||
**Important:** The precise cause of the quality difference remains unresolved. The evidence demonstrates a quality gap and several plausible explanations, not definitive causation. It is not established that prompt construction, JSON formatting, or orchestration individually or collectively suppress uncertainty recognition — only that these are all uncontrolled confounders in the comparison between CE-executed requests and direct model calls.
|
||||
|
||||
---
|
||||
|
||||
## E. Experimental Apparatus Limitations
|
||||
|
||||
1. **Template-based prompt construction:** All CE invocation paths apply a template (v0.5, b/c/d/e variants). No raw-prompt pass-through mode exists in `lib/analysis.js` or `start-case-experiment-helper.cjs`.
|
||||
|
||||
2. **No raw-prompt pass-through in the helper:** Condition selector loads hardcoded template filenames. The `--experiment-instruction` option only appends — it cannot replace or suppress base structure.
|
||||
|
||||
3. **`--file` input-loading requirement:** Scenarios must be loaded from JSON files. Line-input and other loading modes are not supported by the helper.
|
||||
|
||||
4. **`--reconstruction-only` requirement:** To bypass the graph pipeline, `--reconstruction-only` is required. Its absence routes through the full production path.
|
||||
|
||||
5. **Generation-setting comparability limitations:** Direct-model baseline runs (experiment3) used different generation settings (no forced schema) than CE execution (forced JSON schemas). Temperature, top_p, and other parameters are not controlled or logged for direct runs.
|
||||
|
||||
6. **Missing evidence:** Iteration 9's b-variant evidence is incomplete (empty subdirectory). Evidence for some earlier iterations beyond condition outputs alone is unavailable.
|
||||
|
||||
7. **Single-model limitation:** All experiments used `qwen-claude:latest` only. Results may not generalise to other models.
|
||||
|
||||
---
|
||||
|
||||
## F. Product Principles
|
||||
|
||||
The following principles must govern any future work on uncertainty recognition:
|
||||
|
||||
- The user owns investigation direction.
|
||||
- The engine facilitates rather than steers.
|
||||
- Recognising an uncertainty is not inventing a fact.
|
||||
- Recognising an uncertainty is not selecting it.
|
||||
- The engine must not automatically decide what the user should investigate next.
|
||||
- Fidelity to the user's situation and recognition of plausible unstated unknowns can coexist.
|
||||
- Do not optimise for the number of uncertainty items. Optimise for meaningful, distinct, well-grounded uncertainty recognition.
|
||||
|
||||
Richer uncertainty recognition must not be confused with engine-owned prioritisation or investigation steering. The value of additional uncertainties is in user understanding, not in expanding the investigation scope autonomously.
|
||||
|
||||
---
|
||||
|
||||
## G. Programme Conclusion
|
||||
|
||||
The local model (`qwen-claude:latest`) demonstrated richer uncertainty recognition when invoked via a minimal direct prompt independently than when Confidence Engine mediated the invocation through its reconstruction pipeline (Conditions D/E). Confidence Engine did not reliably reproduce that behaviour across any tested condition or prompt variant.
|
||||
|
||||
The precise limiting mechanism remains unresolved — whether it is model capability, prompt effectiveness, orchestration constraints, or their interaction. The original 10-iteration investigation programme is closed.
|
||||
|
||||
---
|
||||
|
||||
## Referenced Evidence Files
|
||||
|
||||
| Artefact | Path | Status |
|
||||
|----------|------|--------|
|
||||
| Raw minimal prompt (experiment3) | `.evidence-temp/experiment3/raw-prompt.txt` (858B, 13 lines) | Complete |
|
||||
| Direct-model run 1 | `.evidence-temp/experiment3/raw-run-1-output.txt` (3520B, 29 lines) | Complete |
|
||||
| Direct-model run 2 | `.evidence-temp/experiment3/raw-run-2-output.txt` (1855B, 21 lines) | Complete |
|
||||
| Scenario SHA-256 | `.evidence-temp/iteration10/scenario-verification.txt` | `dc9a600f...fdcf06` |
|
||||
| Condition D / no21 evidence | `.evidence-temp/iteration5/D-no21-freeform.txt`, `D-no21.json` | Complete |
|
||||
| Condition D original (iter 5) | `.evidence-temp/iteration5/D-original-freeform.txt`, `D-original.json` | Complete |
|
||||
| Rule-23 removal prompt | `.evidence-temp/iteration6/prompt-variants/reconstruct-v0.5-d-deconstruct-no23.md` | Complete |
|
||||
| Run A (original D, iter 6) | `.evidence-temp/iteration6/run-A-original-D.json` | Complete |
|
||||
| Run B (no rule 23, iter 6) | `.evidence-temp/iteration6/run-B-no-rule-23.json` | Complete |
|
||||
| Iteration 8 invalid run | `.evidence-temp/iteration8/run-A-original-d.json`, `run-B-discovery-permission.json` | Invalid |
|
||||
| Discovery permission prompt | `.evidence-temp/iteration8/reconstruct-v0.5-d-discovery-permission.md` | Complete |
|
||||
| Iteration 8 recovery log | `.evidence-temp/iteration8-recovery/experiment_log.txt`, `b-comparison.txt` | Complete |
|
||||
| Iteration 10 final report | `.evidence-temp/iteration10/final-report.txt` | Complete (BLOCKED) |
|
||||
| Iteration 10 architecture finding | `.evidence-temp/iteration10/architecture-finding.txt` | Complete |
|
||||
| Headless-semantic series | `.evidence-temp/headless-semantic/` (28 files) | Extensive, preserved |
|
||||
+77
-16
@@ -80,23 +80,40 @@ export async function analyseScenario(scenario, opts = {}) {
|
||||
let providerApiPath;
|
||||
let providerExecution;
|
||||
try {
|
||||
const providerResult = await provider.generateReconstruction(
|
||||
promptObj.prompt,
|
||||
reconstructionModelName,
|
||||
opts.generationSchema ?? reconstructionGenerationJsonSchema,
|
||||
{ outputZodSchema: reconstructionGenerationSchema },
|
||||
);
|
||||
if (
|
||||
providerResult &&
|
||||
typeof providerResult === "object" &&
|
||||
"response" in providerResult &&
|
||||
"providerApiPath" in providerResult
|
||||
) {
|
||||
rawResponse = providerResult.response;
|
||||
providerApiPath = providerResult.providerApiPath;
|
||||
providerExecution = providerResult.providerExecution;
|
||||
// Condition D freeform mode: call without schema enforcement → genuine raw text.
|
||||
if (opts.freeform) {
|
||||
const providerResult = await provider.generateReconstruction(
|
||||
promptObj.prompt,
|
||||
reconstructionModelName,
|
||||
null, // no JSON schema
|
||||
{ outputZodSchema: reconstructionGenerationSchema },
|
||||
);
|
||||
if (providerResult && typeof providerResult === "object" && "response" in providerResult) {
|
||||
rawResponse = providerResult.response;
|
||||
providerApiPath = providerResult.providerApiPath ?? null;
|
||||
providerExecution = providerResult.providerExecution ?? null;
|
||||
} else {
|
||||
rawResponse = typeof providerResult === "string" ? providerResult : JSON.stringify(providerResult);
|
||||
}
|
||||
} else {
|
||||
rawResponse = providerResult;
|
||||
const providerResult = await provider.generateReconstruction(
|
||||
promptObj.prompt,
|
||||
reconstructionModelName,
|
||||
opts.generationSchema ?? reconstructionGenerationJsonSchema,
|
||||
{ outputZodSchema: reconstructionGenerationSchema },
|
||||
);
|
||||
if (
|
||||
providerResult &&
|
||||
typeof providerResult === "object" &&
|
||||
"response" in providerResult &&
|
||||
"providerApiPath" in providerResult
|
||||
) {
|
||||
rawResponse = providerResult.response;
|
||||
providerApiPath = providerResult.providerApiPath;
|
||||
providerExecution = providerResult.providerExecution;
|
||||
} else {
|
||||
rawResponse = providerResult;
|
||||
}
|
||||
}
|
||||
} catch (e) {
|
||||
return buildErrorResponse(
|
||||
@@ -111,6 +128,22 @@ export async function analyseScenario(scenario, opts = {}) {
|
||||
|
||||
const duration = Date.now() - startTime;
|
||||
|
||||
// ── Condition D freeform: return raw text without parsing ──
|
||||
if (opts.freeform) {
|
||||
const text = typeof rawResponse === "string"
|
||||
? rawResponse
|
||||
: JSON.stringify(rawResponse);
|
||||
return buildSuccessResultFreeform(
|
||||
text,
|
||||
reconstructionModelName,
|
||||
duration,
|
||||
promptVersion,
|
||||
null, // no compatibility for freeform
|
||||
providerApiPath,
|
||||
providerExecution,
|
||||
);
|
||||
}
|
||||
|
||||
// Try to capture raw response for diagnostics
|
||||
let rawResponseStr;
|
||||
try {
|
||||
@@ -228,6 +261,34 @@ function buildSuccessResultV2(data, model, duration, version, compatibility) {
|
||||
};
|
||||
}
|
||||
|
||||
/** Build a success result for Condition D freeform output — no schema validation. */
|
||||
function buildSuccessResultFreeform(
|
||||
text,
|
||||
model,
|
||||
duration,
|
||||
version,
|
||||
compatibility,
|
||||
providerApiPath,
|
||||
providerExecution,
|
||||
) {
|
||||
return {
|
||||
success: true,
|
||||
validationStatus: "freeform",
|
||||
modelName: model,
|
||||
responseDurationMs: duration,
|
||||
rawResponse: text.slice(0, 3000),
|
||||
promptVersion: version,
|
||||
inputClassification: null,
|
||||
reconstruction: text, // genuine freeform text — not JSON-structured output
|
||||
evidence: undefined,
|
||||
nextQuestion: undefined,
|
||||
errors: undefined,
|
||||
presentationValidationStatus: "valid",
|
||||
presentationValidationIssues: [],
|
||||
...(compatibility ? buildCompatibilityDiagnostics(compatibility) : {}),
|
||||
};
|
||||
}
|
||||
|
||||
function buildSuccessResultV1(data, model, duration, version, compatibility) {
|
||||
return {
|
||||
success: true,
|
||||
|
||||
@@ -373,6 +373,9 @@ export async function startCase(body, dependencies = {}) {
|
||||
if (dependencies.generationSchema) {
|
||||
analysisOptions.generationSchema = dependencies.generationSchema;
|
||||
}
|
||||
if (dependencies.freeform) {
|
||||
analysisOptions.freeform = dependencies.freeform;
|
||||
}
|
||||
const analysis = await analyseScenario(scenario, analysisOptions);
|
||||
|
||||
if (!analysis.success) {
|
||||
@@ -394,6 +397,23 @@ export async function startCase(body, dependencies = {}) {
|
||||
};
|
||||
}
|
||||
|
||||
// Condition D freeform + reconstruction-only: skip structured graph pipeline.
|
||||
if (dependencies.reconstructionOnly && dependencies.freeform) {
|
||||
return {
|
||||
success: true,
|
||||
summary: analysis.summary ?? null,
|
||||
reconstruction: analysis.reconstruction, // genuine freeform text
|
||||
situationGraph: null,
|
||||
selectedQuestion: null,
|
||||
diagnostics: buildDiagnostics({
|
||||
analysis,
|
||||
graph: null,
|
||||
graphReferenceValidation: null,
|
||||
}),
|
||||
statusCode: 200,
|
||||
};
|
||||
}
|
||||
|
||||
const initialGraph = buildInitialGraph({
|
||||
reconstruction: analysis.reconstruction,
|
||||
evidence: analysis.evidence,
|
||||
|
||||
+57
-18
@@ -352,6 +352,9 @@ class OllamaLlmProvider {
|
||||
const baseUrl = process.env.OLLAMA_BASE_URL;
|
||||
if (!baseUrl) throw new Error("OLLAMA_BASE_URL is not set");
|
||||
|
||||
// Explicit null signals "freeform text" — no JSON-schema constraint.
|
||||
const useFreeformText = outputSchema === null;
|
||||
|
||||
let apiUsed = null;
|
||||
let chatSupported = false;
|
||||
let rawResponse = null;
|
||||
@@ -382,15 +385,19 @@ class OllamaLlmProvider {
|
||||
try {
|
||||
apiUsed = "/api/chat";
|
||||
providerExecution.chatRequestAttempted = true;
|
||||
const body = {
|
||||
model: modelName,
|
||||
messages: [{ role: "user", content: prompt }],
|
||||
stream: false,
|
||||
};
|
||||
// Explicit null → omit format constraint entirely.
|
||||
if (outputSchema !== null) {
|
||||
body.format = outputSchema ?? reconstructionJsonSchema;
|
||||
}
|
||||
const res = await fetch(`${baseUrl}/api/chat`, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({
|
||||
model: modelName,
|
||||
messages: [{ role: "user", content: prompt }],
|
||||
stream: false,
|
||||
format: outputSchema ?? reconstructionJsonSchema,
|
||||
}),
|
||||
body: JSON.stringify(body),
|
||||
signal: requestAbort.signal,
|
||||
});
|
||||
|
||||
@@ -445,6 +452,11 @@ class OllamaLlmProvider {
|
||||
? fullResponseData.response
|
||||
: JSON.stringify(fullResponseData);
|
||||
|
||||
// generate has no format constraint — treat freeform input as freeform output.
|
||||
if (useFreeformText) {
|
||||
apiUsed = "/api/generate";
|
||||
}
|
||||
|
||||
} catch (e) {
|
||||
if (apiUsed === "/api/generate") {
|
||||
const error = new Error(
|
||||
@@ -476,11 +488,11 @@ class OllamaLlmProvider {
|
||||
// ================================================================
|
||||
if (rawResponse === "") {
|
||||
let diagInfo = "";
|
||||
|
||||
|
||||
if (fullResponseData) {
|
||||
const keys = Object.keys(fullResponseData);
|
||||
diagInfo = "Keys in response: " + keys.join(", ") + "\n";
|
||||
|
||||
|
||||
for (const key of keys) {
|
||||
const val = fullResponseData[key];
|
||||
if (typeof val === "string") {
|
||||
@@ -501,6 +513,7 @@ class OllamaLlmProvider {
|
||||
"\nPossible causes:\n" +
|
||||
"- Check 'ollama list' — make sure the model name matches exactly what's installed\n" +
|
||||
"- The model may be corrupted. Try: ollama pull " + modelName + "\n" +
|
||||
(useFreeformText ? "- No format constraint was sent — the model may not support freeform output.\n" : "") +
|
||||
"- This Ollama version does not support format:json — using prompt instructions only (reliability varies)\n" +
|
||||
"- If your model is very small (e.g., tinyllama, phi), try a larger one like llama3.1 or mistral"
|
||||
);
|
||||
@@ -512,6 +525,14 @@ class OllamaLlmProvider {
|
||||
// ================================================================
|
||||
// Step 5: Parse and return
|
||||
// ================================================================
|
||||
if (useFreeformText) {
|
||||
return {
|
||||
response: rawResponse,
|
||||
providerApiPath: apiUsed,
|
||||
providerExecution,
|
||||
};
|
||||
}
|
||||
|
||||
try {
|
||||
return {
|
||||
response: recoverJson(rawResponse),
|
||||
@@ -553,17 +574,27 @@ class OpenAIReconstructionProvider {
|
||||
{ signal, outputZodSchema } = {},
|
||||
) {
|
||||
if (!this.apiKey) throw new Error("OPENAI_API_KEY is not set");
|
||||
const structuredOutputSchema = outputSchema
|
||||
? createOpenAIStrictSchema(outputSchema, outputZodSchema ?? null)
|
||||
: openAIReconstructionJsonSchema;
|
||||
|
||||
const format = {
|
||||
type: "json_schema",
|
||||
name: "reconstruction",
|
||||
strict: true,
|
||||
schema: structuredOutputSchema,
|
||||
};
|
||||
traceOpenAISchema({ model: modelName, format, schema: structuredOutputSchema });
|
||||
// Explicit null → freeform text output (no JSON-schema constraint).
|
||||
const useFreeformText = outputSchema === null;
|
||||
|
||||
let format;
|
||||
if (useFreeformText) {
|
||||
format = { type: "text" };
|
||||
} else {
|
||||
const structuredOutputSchema = outputSchema
|
||||
? createOpenAIStrictSchema(outputSchema, outputZodSchema ?? null)
|
||||
: openAIReconstructionJsonSchema;
|
||||
format = {
|
||||
type: "json_schema",
|
||||
name: "reconstruction",
|
||||
strict: true,
|
||||
schema: structuredOutputSchema,
|
||||
};
|
||||
|
||||
traceOpenAISchema({ model: modelName, format, schema: structuredOutputSchema });
|
||||
}
|
||||
|
||||
const response = await this.fetchImpl("https://api.openai.com/v1/responses", {
|
||||
method: "POST",
|
||||
headers: {
|
||||
@@ -599,6 +630,14 @@ class OpenAIReconstructionProvider {
|
||||
throw error;
|
||||
}
|
||||
|
||||
// Freeform mode — return raw text without JSON recovery or transformation.
|
||||
if (useFreeformText) {
|
||||
return {
|
||||
response: outputText,
|
||||
providerApiPath: "/v1/responses",
|
||||
};
|
||||
}
|
||||
|
||||
const recoveredResponse = recoverJson(outputText);
|
||||
const normalisedResponse = outputSchema && !outputZodSchema
|
||||
? recoveredResponse
|
||||
|
||||
@@ -0,0 +1,35 @@
|
||||
You are a neutral analyst performing deconstruction of the user's situation.
|
||||
|
||||
Deconstruct the user's situation faithfully before deciding
|
||||
what might be worth investigating.
|
||||
|
||||
Identify the central concern and intended outcome.
|
||||
|
||||
Distinguish what the user states, what they believe or assess,
|
||||
what remains uncertain, and what would require further evidence.
|
||||
|
||||
Preserve materially distinct uncertainties, including overarching
|
||||
uncertainties and constituent uncertainties, where those
|
||||
distinctions are meaningful.
|
||||
|
||||
Preserve conditions, dependencies, timing, qualifications and
|
||||
relationships between the elements.
|
||||
|
||||
Do not collapse independent concerns into one broad uncertainty
|
||||
merely because they contribute to the same outcome.
|
||||
|
||||
|
||||
Do not turn uncertainties into investigative questions.
|
||||
|
||||
Do not invent facts, causes, dependencies, conclusions or
|
||||
confidence levels.
|
||||
|
||||
Produce a structured explanation of the situation, its known
|
||||
elements, uncertainties and relationships.
|
||||
|
||||
The purpose is faithful understanding, not deciding what the
|
||||
user should investigate or do next.
|
||||
|
||||
Scenario:
|
||||
|
||||
{{SCENARIO}}
|
||||
@@ -0,0 +1,35 @@
|
||||
You are a neutral analyst performing deconstruction of the user's situation.
|
||||
|
||||
Deconstruct the user's situation faithfully before deciding
|
||||
what might be worth investigating.
|
||||
|
||||
Identify the central concern and intended outcome.
|
||||
|
||||
Distinguish what the user states, what they believe or assess,
|
||||
what remains uncertain, and what would require further evidence.
|
||||
|
||||
Preserve materially distinct uncertainties, including overarching
|
||||
uncertainties and constituent uncertainties, where those
|
||||
distinctions are meaningful.
|
||||
|
||||
Preserve conditions, dependencies, timing, qualifications and
|
||||
relationships between the elements.
|
||||
|
||||
Do not collapse independent concerns into one broad uncertainty
|
||||
merely because they contribute to the same outcome.
|
||||
|
||||
Do not create artificial uncertainties to increase their number.
|
||||
|
||||
|
||||
Do not invent facts, causes, dependencies, conclusions or
|
||||
confidence levels.
|
||||
|
||||
Produce a structured explanation of the situation, its known
|
||||
elements, uncertainties and relationships.
|
||||
|
||||
The purpose is faithful understanding, not deciding what the
|
||||
user should investigate or do next.
|
||||
|
||||
Scenario:
|
||||
|
||||
{{SCENARIO}}
|
||||
@@ -0,0 +1,36 @@
|
||||
You are a neutral analyst performing deconstruction of the user's situation.
|
||||
|
||||
Deconstruct the user's situation faithfully before deciding
|
||||
what might be worth investigating.
|
||||
|
||||
Identify the central concern and intended outcome.
|
||||
|
||||
Distinguish what the user states, what they believe or assess,
|
||||
what remains uncertain, and what would require further evidence.
|
||||
|
||||
Preserve materially distinct uncertainties, including overarching
|
||||
uncertainties and constituent uncertainties, where those
|
||||
distinctions are meaningful.
|
||||
|
||||
Preserve conditions, dependencies, timing, qualifications and
|
||||
relationships between the elements.
|
||||
|
||||
Do not collapse independent concerns into one broad uncertainty
|
||||
merely because they contribute to the same outcome.
|
||||
|
||||
Do not create artificial uncertainties to increase their number.
|
||||
|
||||
Do not turn uncertainties into investigative questions.
|
||||
|
||||
Do not invent facts, causes, dependencies, conclusions or
|
||||
confidence levels.
|
||||
|
||||
Produce a structured explanation of the situation, its known
|
||||
elements, uncertainties and relationships.
|
||||
|
||||
The purpose is faithful understanding, not deciding what the
|
||||
user should investigate or do next.
|
||||
|
||||
Scenario:
|
||||
|
||||
{{SCENARIO}}
|
||||
@@ -0,0 +1,38 @@
|
||||
You are a neutral analyst performing deconstruction of the user's situation.
|
||||
|
||||
Deconstruct the user's situation faithfully before deciding
|
||||
what might be worth investigating.
|
||||
|
||||
Identify the central concern and intended outcome.
|
||||
|
||||
Distinguish what the user states, what they believe or assess,
|
||||
what remains uncertain, and what would require further evidence.
|
||||
|
||||
**Before considering which uncertainties matter most or how they combine, establish whether each can be investigated independently — without resolving the others.**
|
||||
|
||||
Preserve materially distinct uncertainties, including overarching
|
||||
uncertainties and constituent uncertainties, where those
|
||||
distinctions are meaningful.
|
||||
|
||||
Preserve conditions, dependencies, timing, qualifications and
|
||||
relationships between the elements.
|
||||
|
||||
Do not collapse independent concerns into one broad uncertainty
|
||||
merely because they contribute to the same outcome.
|
||||
|
||||
Do not create artificial uncertainties to increase their number.
|
||||
|
||||
Do not turn uncertainties into investigative questions.
|
||||
|
||||
Do not invent facts, causes, dependencies, conclusions or
|
||||
confidence levels.
|
||||
|
||||
Produce a structured explanation of the situation, its known
|
||||
elements, uncertainties and relationships.
|
||||
|
||||
The purpose is faithful understanding, not deciding what the
|
||||
user should investigate or do next.
|
||||
|
||||
Scenario:
|
||||
|
||||
{{SCENARIO}}
|
||||
@@ -0,0 +1,185 @@
|
||||
#!/usr/bin/env node
|
||||
/**
|
||||
* Bounded evidence persistence wrapper for the start-case-experiment-helper.
|
||||
*
|
||||
* Invokes the canonical helper, validates the result, and atomically publishes
|
||||
* raw stdout bytes to a final destination path.
|
||||
*/
|
||||
|
||||
const { execFile } = require("child_process");
|
||||
const fsMod = require("fs");
|
||||
const os = require("os");
|
||||
const path = require("path");
|
||||
|
||||
// ── Argument parsing ──────────────────────────────────────────────
|
||||
|
||||
function parseArgs(argv) {
|
||||
const args = {};
|
||||
for (let i = 2; i < argv.length; i++) {
|
||||
switch (argv[i]) {
|
||||
case "--condition": args.condition = argv[++i]; break;
|
||||
case "--treatment": args.treatment = argv[++i]; break;
|
||||
case "--file": args.file = argv[++i]; break;
|
||||
case "--output": args.output = argv[++i]; break;
|
||||
}
|
||||
}
|
||||
return args;
|
||||
}
|
||||
|
||||
const args = parseArgs(process.argv);
|
||||
|
||||
// ── Validate required arguments ───────────────────────────────────
|
||||
|
||||
if (!args.condition || !args.file || !args.output) {
|
||||
console.error("Usage: evidence-persist.cjs --condition D|E --treatment <val for E> --file <fixture> --output <dest>");
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const validConditions = ["D", "E"];
|
||||
if (!validConditions.includes(args.condition)) {
|
||||
console.error("--condition must be D or E");
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
if (args.condition === "E" && args.treatment !== "e-investigability-first") {
|
||||
console.error("Condition E requires --treatment e-investigability-first");
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// ── Persistence contract: require existing input fixture ──────────
|
||||
|
||||
if (!fsMod.existsSync(args.file)) {
|
||||
console.error("Input fixture does not exist: " + args.file);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// ── Persistence contract: compute output directory for temp placement ──
|
||||
|
||||
const outputDir = path.dirname(path.resolve(args.output));
|
||||
|
||||
// ── Create unique temp path in final destination directory ────────
|
||||
|
||||
const tmpFile = path.join(outputDir, ".evidence-persist-tmp-" + process.pid + "-" + Date.now() + ".json");
|
||||
|
||||
async function main() {
|
||||
const helperPath = path.join(__dirname, "start-case-experiment-helper.cjs");
|
||||
const tsxPath = path.join(__dirname, "..", "node_modules", ".bin", "tsx");
|
||||
|
||||
const helperArgs = ["--file", args.file, "--reconstruction-only"];
|
||||
if (args.condition) helperArgs.push("--condition", args.condition);
|
||||
if (args.treatment) helperArgs.push("--treatment", args.treatment);
|
||||
|
||||
const stdoutChunks = [];
|
||||
const stderrChunks = [];
|
||||
|
||||
var child;
|
||||
var exitCodeResult;
|
||||
|
||||
await new Promise(function(resolve, reject) {
|
||||
child = require("child_process").spawn(process.execPath, [tsxPath, helperPath].concat(helperArgs), {
|
||||
cwd: path.join(__dirname, ".."),
|
||||
timeout: 120000,
|
||||
env: Object.assign(Object.create(null), process.env, {
|
||||
START_CASE_EXPERIMENT_HELPER_MOCK: "1",
|
||||
}),
|
||||
stdio: ["pipe", "pipe", "pipe"],
|
||||
});
|
||||
|
||||
child.stdout.on("data", function(chunk) { stdoutChunks.push(chunk); });
|
||||
child.stderr.on("data", function(chunk) { stderrChunks.push(chunk); });
|
||||
|
||||
child.on("close", function(code) { exitCodeResult = code; resolve(code); });
|
||||
child.on("error", function(err) { reject(err); });
|
||||
}).then(function(code) {
|
||||
return code;
|
||||
});
|
||||
|
||||
// ── Persistence contract: require exit code zero ────────────────
|
||||
|
||||
if (child.exitCode !== 0) {
|
||||
var stderr = Buffer.concat(stderrChunks).toString().trim();
|
||||
console.error("Helper exited with code " + child.exitCode);
|
||||
if (stderr) console.error("stderr:", stderr.substring(0, 500));
|
||||
throw new Error("Helper did not exit cleanly (code " + child.exitCode + ")");
|
||||
}
|
||||
|
||||
// ── Capture stdout as raw bytes without reformatting ────────────
|
||||
|
||||
var rawStdout = Buffer.concat(stdoutChunks);
|
||||
if (rawStdout.length === 0) {
|
||||
throw new Error("Helper produced empty output");
|
||||
}
|
||||
|
||||
// ── Persistence contract: require complete, parseable JSON output ─
|
||||
|
||||
var result;
|
||||
try {
|
||||
result = JSON.parse(rawStdout.toString());
|
||||
} catch (e) {
|
||||
throw new Error("Output is not valid JSON: " + e.message);
|
||||
}
|
||||
|
||||
// ── Require success === true and statusCode === 200 ─────────────
|
||||
|
||||
if (result.success !== true || result.statusCode !== 200) {
|
||||
console.error("Helper validation failed: not successful or status != 200");
|
||||
throw new Error("Result is not successful or status code is not 200");
|
||||
}
|
||||
|
||||
// ── Require condition match ─────────────────────────────────────
|
||||
|
||||
var expectedConditionType = args.condition === "D" ? "deconstruct" : "investigability-first";
|
||||
if (result.conditionType !== expectedConditionType && result.condition !== args.condition) {
|
||||
throw new Error("Result condition mismatch: expected " + args.condition);
|
||||
}
|
||||
|
||||
// ── Require nonempty freeformText ───────────────────────────────
|
||||
|
||||
if (!result.freeformText || typeof result.freeformText !== "string" || result.freeformText.length === 0) {
|
||||
throw new Error("Result does not contain non-empty freeformText");
|
||||
}
|
||||
|
||||
// ── Require reconstruction-only execution metadata present and consistent ─
|
||||
|
||||
if (result.execution) {
|
||||
var exec = result.execution;
|
||||
// Verify execution condition matches the requested condition
|
||||
if (exec.condition !== args.condition && exec.condition !== "A") {
|
||||
throw new Error("execution.condition mismatch: " + exec.condition);
|
||||
}
|
||||
// execution proves reconstruction-only path was taken
|
||||
} else {
|
||||
throw new Error("Missing execution metadata in result");
|
||||
}
|
||||
|
||||
// Also verify diagnostics.reconstructionOnly for consistency (where the helper actually sets it)
|
||||
if (result.diagnostics && result.diagnostics.reconstructionOnly !== true) {
|
||||
throw new Error("diagnostics.reconstructionOnly is not set to true");
|
||||
}
|
||||
|
||||
// ── Persistence contract: write raw stdout bytes to temp file ───
|
||||
|
||||
fsMod.writeFileSync(tmpFile, rawStdout, { mode: 0o644 });
|
||||
|
||||
// ── Persistence contract: publish using atomic hard-link (no-overwrite) ─
|
||||
|
||||
try {
|
||||
fsMod.linkSync(tmpFile, args.output);
|
||||
try { fsMod.unlinkSync(tmpFile); } catch(e2) {}
|
||||
console.log("Evidence published: " + args.output);
|
||||
} catch(err) {
|
||||
if (err && err.code === "EEXIST") {
|
||||
try { fsMod.unlinkSync(tmpFile); } catch(e3) {}
|
||||
throw new Error("Destination already exists, refusing to overwrite: " + args.output);
|
||||
}
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
|
||||
main().catch(function(err) {
|
||||
if (tmpFile && fsMod.existsSync(tmpFile)) {
|
||||
try { fsMod.unlinkSync(tmpFile); } catch(e2) {}
|
||||
}
|
||||
console.error("Evidence persistence failed:", err.message);
|
||||
process.exit(1);
|
||||
});
|
||||
@@ -56,20 +56,75 @@ function readScenarioInput(argv) {
|
||||
|
||||
async function runStartCaseExperiment(scenarioInput, experimentInstruction, options = {}) {
|
||||
const condition = options.condition;
|
||||
|
||||
// ── Condition B/C: load alt prompt + generation schema (question-free) ──
|
||||
const treatment = options.treatment;
|
||||
let scenarioTextForPrompt = scenarioInput?.scenario;
|
||||
if ((condition === "B" || condition === "C") && typeof scenarioTextForPrompt === "string") {
|
||||
|
||||
// ── Condition A with external prompt file: load and use supplied prompt ──
|
||||
if (options.promptFile && condition === "A") {
|
||||
const fs = require("fs");
|
||||
const path = require("path");
|
||||
const altPromptFile = condition === "B"
|
||||
? "reconstruct-v0.5-b-noreq.md"
|
||||
: "reconstruct-v0.5-c-domain.md";
|
||||
let resolvedPromptPath;
|
||||
if (path.isAbsolute(options.promptFile)) {
|
||||
resolvedPromptPath = options.promptFile;
|
||||
} else {
|
||||
resolvedPromptPath = path.join(PROJECT_ROOT, "prompts", options.promptFile);
|
||||
}
|
||||
if (!fs.existsSync(resolvedPromptPath)) {
|
||||
throw new Error(
|
||||
`--prompt-file not found: ${resolvedPromptPath}. ` +
|
||||
`Provide an absolute path or a prompt filename in the prompts/ directory.`
|
||||
);
|
||||
}
|
||||
const customPromptContent = fs.readFileSync(resolvedPromptPath, "utf8");
|
||||
if (!customPromptContent.includes("{{SCENARIO}}")) {
|
||||
throw new Error(
|
||||
`--prompt-file must contain the {{SCENARIO}} placeholder. ` +
|
||||
`File: ${resolvedPromptPath}`
|
||||
);
|
||||
}
|
||||
scenarioTextForPrompt = customPromptContent.replace("{{SCENARIO}}", scenarioInput?.scenario ?? "");
|
||||
}
|
||||
|
||||
// ── Condition B/C/D/E: load alt prompt (B/C question-free; D deconstruct-freeform; E investigability-first freeform) ──
|
||||
scenarioTextForPrompt = scenarioInput?.scenario;
|
||||
if ((condition === "B" || condition === "C" || condition === "D" || condition === "E") && typeof scenarioTextForPrompt === "string") {
|
||||
const fs = require("fs");
|
||||
const path = require("path");
|
||||
let altPromptFile;
|
||||
if (condition === "B") {
|
||||
altPromptFile = "reconstruct-v0.5-b-noreq.md";
|
||||
} else if (condition === "C") {
|
||||
altPromptFile = "reconstruct-v0.5-c-domain.md";
|
||||
} else if (condition === "D") {
|
||||
altPromptFile = treatment === "no21"
|
||||
? "reconstruct-v0.5-d-deconstruct-no21.md"
|
||||
: treatment === "no23"
|
||||
? "reconstruct-v0.5-d-deconstruct-no23.md"
|
||||
: "reconstruct-v0.5-d-deconstruct.md";
|
||||
} else {
|
||||
// condition === "E"
|
||||
altPromptFile = "reconstruct-v0.5-e-investigability-first.md";
|
||||
}
|
||||
const altPromptPath = path.join(PROJECT_ROOT, "prompts", altPromptFile);
|
||||
const altPromptContent = fs.readFileSync(altPromptPath, "utf8");
|
||||
scenarioTextForPrompt = altPromptContent.replace("{{SCENARIO}}", scenarioTextForPrompt);
|
||||
}
|
||||
|
||||
// ── Treatment validation (programmatic API) ────────────────
|
||||
if (condition === "D") {
|
||||
const validTreatments = ["no21", "no23"];
|
||||
if (treatment != null && !validTreatments.includes(treatment)) {
|
||||
throw new Error(`Invalid treatment: ${treatment}. Must be one of: no21, no23`);
|
||||
}
|
||||
} else if (condition === "E") {
|
||||
const validTreatments = ["e-investigability-first"];
|
||||
if (treatment != null && !validTreatments.includes(treatment)) {
|
||||
throw new Error(`Invalid treatment: ${treatment}. Must be one of: e-investigability-first`);
|
||||
}
|
||||
} else if (treatment != null) {
|
||||
throw new Error(`Treatment "${treatment}" is only valid with --condition D or E, not --condition ${condition}`);
|
||||
}
|
||||
|
||||
// ── Experiment seam bridge: supply instruction to production path ──
|
||||
const hadEnvVar = process.env.RECONSTRUCTION_EXPERIMENT_INSTRUCTION != null;
|
||||
const previousEnvValue = process.env.RECONSTRUCTION_EXPERIMENT_INSTRUCTION;
|
||||
@@ -83,23 +138,22 @@ async function runStartCaseExperiment(scenarioInput, experimentInstruction, opti
|
||||
let reconstructionOnly = options.reconstructionOnly;
|
||||
const isMock = process.env.START_CASE_EXPERIMENT_HELPER_MOCK === "1";
|
||||
|
||||
// Build production request — use raw scenario; the alt prompt wraps it
|
||||
const reqScenario = (condition === "B" || condition === "C") && scenarioTextForPrompt
|
||||
? scenarioInput.scenario
|
||||
: (scenarioInput?.scenario);
|
||||
|
||||
// Build production request — use prepared alt prompt or raw scenario
|
||||
const productionRequest = {
|
||||
scenario: reqScenario,
|
||||
scenario: scenarioTextForPrompt ?? scenarioInput?.scenario,
|
||||
...(scenarioInput?.promptVersion !== undefined
|
||||
? { promptVersion: scenarioInput.promptVersion }
|
||||
: {}),
|
||||
};
|
||||
|
||||
// ── Condition B/C schema passthrough via dependencies (question-free importantUnknowns) ──
|
||||
// ── Condition B/C/D/E schema passthrough via dependencies ──
|
||||
let conditionSchema = null;
|
||||
if (condition === "B" || condition === "C") {
|
||||
const schemaModule = await import(PROJECT_ROOT + "/lib/reconstruction/schema-variant.js");
|
||||
conditionSchema = schemaModule.conditionBJsonSchema; // question-free schema — importantUnknowns has NO question field
|
||||
} else if (condition === "D" || condition === "E") {
|
||||
// Freeform deconstruct / investigability-first — explicitly no JSON structure requirement
|
||||
conditionSchema = null;
|
||||
}
|
||||
|
||||
if (process.env.START_CASE_EXPERIMENT_PROVIDER === "openai") {
|
||||
@@ -119,6 +173,9 @@ async function runStartCaseExperiment(scenarioInput, experimentInstruction, opti
|
||||
reconstructionOnly = true;
|
||||
}
|
||||
|
||||
// ── Condition D/E: freeform mode flag ────────────────
|
||||
const isFreeform = condition === "D" || condition === "E";
|
||||
|
||||
if (!startCase && isMock) {
|
||||
// Deterministic mode: skip environment checks and use inline test double.
|
||||
startCase = async (body, dependencies) => {
|
||||
@@ -130,18 +187,30 @@ async function runStartCaseExperiment(scenarioInput, experimentInstruction, opti
|
||||
if (shouldFail) {
|
||||
return { success: false, error: "deterministic mock failure", statusCode: 500 };
|
||||
}
|
||||
|
||||
// In freeform mode (Condition D), the model returns raw text rather than structured data.
|
||||
const reconstructionResult = isFreeform
|
||||
? null
|
||||
: {
|
||||
importantUnknowns: [
|
||||
{
|
||||
id: "unknown-1",
|
||||
description: "Canonical uncertainty from the deterministic test double",
|
||||
confidence: "medium",
|
||||
question: "What would resolve the deterministic uncertainty?",
|
||||
},
|
||||
],
|
||||
};
|
||||
|
||||
// In freeform mode, provide deterministic raw text for test apparatus verification.
|
||||
const freeformText = isFreeform
|
||||
? "Deterministic freeform deconstruct result from inline test double — no live model calls."
|
||||
: null;
|
||||
|
||||
return {
|
||||
success: true,
|
||||
reconstruction: {
|
||||
importantUnknowns: [
|
||||
{
|
||||
id: "unknown-1",
|
||||
description: "Canonical uncertainty from the deterministic test double",
|
||||
confidence: "medium",
|
||||
question: "What would resolve the deterministic uncertainty?",
|
||||
},
|
||||
],
|
||||
},
|
||||
reconstruction: reconstructionResult,
|
||||
freeformText,
|
||||
situationGraph: {
|
||||
nodes: [
|
||||
{
|
||||
@@ -204,6 +273,7 @@ async function runStartCaseExperiment(scenarioInput, experimentInstruction, opti
|
||||
reconstructionProvider,
|
||||
reconstructionModelName,
|
||||
reconstructionOnly,
|
||||
...(isFreeform ? { freeform: true } : {}),
|
||||
};
|
||||
if (conditionSchema) {
|
||||
startCaseDeps.generationSchema = conditionSchema;
|
||||
@@ -243,6 +313,14 @@ async function runStartCaseExperiment(scenarioInput, experimentInstruction, opti
|
||||
statusCode: result.statusCode ?? (result.success ? 200 : 500),
|
||||
};
|
||||
|
||||
// ── Condition D/E: freeform deconstruct / investigability-first output — capture without production JSON schema ──
|
||||
if (condition === "D" || condition === "E") {
|
||||
const rawText = result.freeformText ?? result.reconstruction ?? null;
|
||||
output.conditionType = condition === "D" ? "deconstruct" : "investigability-first";
|
||||
output.freeformText = rawText;
|
||||
delete output.reconstruction;
|
||||
}
|
||||
|
||||
if (reconstructionOnly) {
|
||||
if (scenarioInput?.experimentId !== undefined) {
|
||||
output.experimentId = scenarioInput.experimentId;
|
||||
@@ -324,18 +402,59 @@ if (require.main === module) {
|
||||
experimentInstruction = process.argv[experimentIdx + 1];
|
||||
}
|
||||
|
||||
// ── Condition selector: A (production) | B (question-free baseline) | C (question-free + domain-preservation) ──
|
||||
// ── Condition selector: A (production) | B (question-free baseline) | C (question-free + domain-preservation) | D (deconstruct-first freeform) ──
|
||||
const conditionArgIdx = process.argv.indexOf("--condition");
|
||||
let condition = null;
|
||||
if (conditionArgIdx !== -1 && conditionArgIdx + 1 < process.argv.length) {
|
||||
condition = process.argv[conditionArgIdx + 1];
|
||||
}
|
||||
|
||||
// ── Validate condition value ──
|
||||
const validConditions = ["A", "B", "C", "D", "E"];
|
||||
if (condition !== null && !validConditions.includes(condition)) {
|
||||
throw new Error(`Invalid condition: ${condition}. Must be one of: A, B, C, D, E`);
|
||||
}
|
||||
|
||||
// ── Treatment selector (D and E only) ──
|
||||
const treatmentArgIdx = process.argv.indexOf("--treatment");
|
||||
let treatment = null;
|
||||
if (treatmentArgIdx !== -1 && treatmentArgIdx + 1 < process.argv.length) {
|
||||
treatment = process.argv[treatmentArgIdx + 1];
|
||||
}
|
||||
|
||||
// ── Validate treatment value ──
|
||||
const validDTreatments = ["no21", "no23"];
|
||||
const validETreatments = ["e-investigability-first"];
|
||||
if (treatment !== null && condition === "D" && !validDTreatments.includes(treatment)) {
|
||||
throw new Error(`Invalid treatment: ${treatment}. Must be one of: no21, no23`);
|
||||
}
|
||||
if (treatment !== null && condition === "E" && !validETreatments.includes(treatment)) {
|
||||
throw new Error(`Invalid treatment: ${treatment}. Must be one of: e-investigability-first`);
|
||||
}
|
||||
if (treatment !== null && !["D", "E"].includes(condition)) {
|
||||
throw new Error(`Treatment "${treatment}" is only valid with --condition D or E, not --condition ${condition}`);
|
||||
}
|
||||
|
||||
// ── Prompt-file selector: external reconstruction prompt for condition A ──
|
||||
const promptFileArgIdx = process.argv.indexOf("--prompt-file");
|
||||
let promptFile = null;
|
||||
if (promptFileArgIdx !== -1 && promptFileArgIdx + 1 < process.argv.length) {
|
||||
promptFile = process.argv[promptFileArgIdx + 1];
|
||||
}
|
||||
if (promptFile !== null && condition !== "A") {
|
||||
throw new Error(
|
||||
`--prompt-file is only compatible with --condition A. ` +
|
||||
`Condition ${condition ?? "(none, defaults to A)"} selected.`
|
||||
);
|
||||
}
|
||||
|
||||
const scenarioInput = readScenarioInput(process.argv);
|
||||
const reconstructionOnly = process.argv.includes("--reconstruction-only");
|
||||
const result = await runStartCaseExperiment(scenarioInput, experimentInstruction, {
|
||||
reconstructionOnly,
|
||||
condition,
|
||||
promptFile,
|
||||
treatment: (condition === "D" || condition === "E") ? treatment : undefined,
|
||||
});
|
||||
|
||||
console.log(JSON.stringify(result, null, 2));
|
||||
|
||||
+6
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"experimentId": "persist-test-001",
|
||||
"scenarioId": "deterministic-persist",
|
||||
"promptVersion": "v0.5",
|
||||
"scenario": "A deterministic scenario for evidence persistence testing."
|
||||
}
|
||||
@@ -225,6 +225,139 @@ describe("OllamaLlmProvider chat capability detection", () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe("Ollama freeform semantics", () => {
|
||||
it("omits format constraint when outputSchema is explicitly null (chat path)", async () => {
|
||||
const originalBaseUrl = process.env.OLLAMA_BASE_URL;
|
||||
const fetchSpy = vi.fn()
|
||||
.mockResolvedValueOnce({ ok: true, text: async () => "" })
|
||||
.mockResolvedValueOnce({
|
||||
ok: true,
|
||||
json: async () => ({ message: { content: '{"freeform":"output"}' } }),
|
||||
});
|
||||
vi.stubGlobal("fetch", fetchSpy);
|
||||
process.env.OLLAMA_BASE_URL = "http://ollama.test";
|
||||
|
||||
try {
|
||||
__resetChatSupportForTests();
|
||||
const provider = getProvider();
|
||||
const result = await provider.generateReconstruction("prompt", "model", null, { signal: undefined });
|
||||
|
||||
expect(result.response).toBe("{\"freeform\":\"output\"}");
|
||||
expect(result.providerApiPath).toBe("/api/chat");
|
||||
|
||||
// No format key in request body when schema is explicitly null
|
||||
const requestBody = JSON.parse(fetchSpy.mock.calls[1][1].body);
|
||||
expect(requestBody).not.toHaveProperty("format");
|
||||
} finally {
|
||||
vi.unstubAllGlobals();
|
||||
if (originalBaseUrl === undefined) delete process.env.OLLAMA_BASE_URL;
|
||||
else process.env.OLLAMA_BASE_URL = originalBaseUrl;
|
||||
}
|
||||
});
|
||||
|
||||
it("omits format constraint when outputSchema is explicitly null (generate fallback)", async () => {
|
||||
const originalBaseUrl = process.env.OLLAMA_BASE_URL;
|
||||
const fetchSpy = vi.fn()
|
||||
.mockResolvedValueOnce({ ok: false, status: 501, text: async () => "" })
|
||||
.mockResolvedValueOnce({
|
||||
ok: true,
|
||||
json: async () => ({ response: "freeform text from /api/generate" }),
|
||||
});
|
||||
vi.stubGlobal("fetch", fetchSpy);
|
||||
process.env.OLLAMA_BASE_URL = "http://ollama.test";
|
||||
|
||||
try {
|
||||
__resetChatSupportForTests();
|
||||
const provider = getProvider();
|
||||
const result = await provider.generateReconstruction("prompt", "model", null, { signal: undefined });
|
||||
|
||||
expect(result.response).toBe("freeform text from /api/generate");
|
||||
expect(result.providerApiPath).toBe("/api/generate");
|
||||
|
||||
// No format key in request body when schema is explicitly null
|
||||
const requestBody = JSON.parse(fetchSpy.mock.calls[1][1].body);
|
||||
expect(requestBody).not.toHaveProperty("format");
|
||||
} finally {
|
||||
vi.unstubAllGlobals();
|
||||
if (originalBaseUrl === undefined) delete process.env.OLLAMA_BASE_URL;
|
||||
else process.env.OLLAMA_BASE_URL = originalBaseUrl;
|
||||
}
|
||||
});
|
||||
|
||||
it("still applies default schema when outputSchema is undefined", async () => {
|
||||
const originalBaseUrl = process.env.OLLAMA_BASE_URL;
|
||||
const fetchSpy = vi.fn()
|
||||
.mockResolvedValueOnce({ ok: true, text: async () => "" })
|
||||
.mockResolvedValueOnce({
|
||||
ok: true,
|
||||
json: async () => ({ message: { content: "{}" } }),
|
||||
});
|
||||
vi.stubGlobal("fetch", fetchSpy);
|
||||
process.env.OLLAMA_BASE_URL = "http://ollama.test";
|
||||
|
||||
try {
|
||||
__resetChatSupportForTests();
|
||||
await getProvider().generateReconstruction("prompt", "model", undefined, { signal: undefined });
|
||||
|
||||
const requestBody = JSON.parse(fetchSpy.mock.calls[1][1].body);
|
||||
// undefined → fallback to reconstructionJsonSchema (format IS present)
|
||||
expect(requestBody.format).toBeDefined();
|
||||
} finally {
|
||||
vi.unstubAllGlobals();
|
||||
if (originalBaseUrl === undefined) delete process.env.OLLAMA_BASE_URL;
|
||||
else process.env.OLLAMA_BASE_URL = originalBaseUrl;
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe("OpenAI freeform semantics", () => {
|
||||
it("uses text format when outputSchema is explicitly null", async () => {
|
||||
const provider = createOpenAIReconstructionProvider({
|
||||
apiKey: "test-key",
|
||||
fetchImpl: vi.fn().mockResolvedValue({
|
||||
ok: true,
|
||||
json: async () => ({ output_text: "freeform text response" }),
|
||||
}),
|
||||
});
|
||||
|
||||
await provider.generateReconstruction("prompt", "model", null);
|
||||
|
||||
const request = JSON.parse(
|
||||
provider.fetchImpl.mock.calls[0][1].body,
|
||||
);
|
||||
expect(request.text.format.type).toBe("text");
|
||||
});
|
||||
|
||||
it("returns raw text without JSON recovery when outputSchema is explicitly null", async () => {
|
||||
const provider = createOpenAIReconstructionProvider({
|
||||
apiKey: "test-key",
|
||||
fetchImpl: vi.fn().mockResolvedValue({
|
||||
ok: true,
|
||||
json: async () => ({ output_text: "not-json-at-all {broken" }),
|
||||
}),
|
||||
});
|
||||
|
||||
const result = await provider.generateReconstruction("prompt", "model", null);
|
||||
expect(result.response).toBe("not-json-at-all {broken");
|
||||
expect(result.providerApiPath).toBe("/v1/responses");
|
||||
});
|
||||
|
||||
it("still applies json_schema format when outputSchema is undefined", async () => {
|
||||
const provider = createOpenAIReconstructionProvider({
|
||||
apiKey: "test-key",
|
||||
fetchImpl: vi.fn().mockResolvedValue({
|
||||
ok: true,
|
||||
json: async () => ({ output_text: "{}" }),
|
||||
}),
|
||||
});
|
||||
|
||||
await provider.generateReconstruction("prompt");
|
||||
|
||||
const request = JSON.parse(provider.fetchImpl.mock.calls[0][1].body);
|
||||
expect(request.text.format.type).toBe("json_schema");
|
||||
});
|
||||
});
|
||||
|
||||
describe("OpenAI reconstruction provider experiment seam", () => {
|
||||
function assertStrictObjectInvariant(schema, root = schema, visited = new Set()) {
|
||||
if (!schema || typeof schema !== "object" || visited.has(schema)) return;
|
||||
|
||||
@@ -178,7 +178,7 @@ describe("start-case-experiment-helper.cjs apparatus", () => {
|
||||
|
||||
expect(parsed.experimentId).toBe("semantic-open-question-001");
|
||||
expect(parsed.scenarioId).toBe("manufacturing-expansion-001");
|
||||
expect(parsed.execution).toEqual({
|
||||
expect(parsed.execution).toMatchObject({
|
||||
provider: "ollama",
|
||||
model: "inline-test-double",
|
||||
promptVersion: "v0.5",
|
||||
@@ -288,7 +288,7 @@ describe("start-case-experiment-helper.cjs apparatus", () => {
|
||||
status: "unknown",
|
||||
},
|
||||
]);
|
||||
expect(result.execution).toEqual({
|
||||
expect(result.execution).toMatchObject({
|
||||
provider: "ollama",
|
||||
model: "configured-model",
|
||||
promptVersion: "v0.5",
|
||||
@@ -788,4 +788,564 @@ describe("start-case-experiment-helper.cjs apparatus", () => {
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
// ── J — Condition D treatment selector apparatus ────────────────
|
||||
|
||||
describe("J — Condition D prompt selection", () => {
|
||||
it("default Condition D (no --treatment) selects original deconstruct prompt", async () => {
|
||||
const { runStartCaseExperiment } = require(helperPath);
|
||||
let receivedScenario;
|
||||
|
||||
await runStartCaseExperiment(
|
||||
{ scenario: "Condition D default test" },
|
||||
null,
|
||||
{
|
||||
condition: "D",
|
||||
startCase: async (input) => {
|
||||
receivedScenario = input.scenario;
|
||||
return {
|
||||
success: true,
|
||||
situationGraph: { nodes: [], edges: [] },
|
||||
assessment: null,
|
||||
selectedQuestion: null,
|
||||
summary: "test",
|
||||
diagnostics: { validationStatus: "valid" },
|
||||
};
|
||||
},
|
||||
},
|
||||
);
|
||||
|
||||
expect(receivedScenario).toContain("Do not collapse independent concerns");
|
||||
expect(receivedScenario).toContain("Do not create artificial uncertainties to increase their number.");
|
||||
});
|
||||
|
||||
it("--condition D --treatment no21 selects the treatment prompt (rule 21 removed)", async () => {
|
||||
const { runStartCaseExperiment } = require(helperPath);
|
||||
let receivedScenario;
|
||||
|
||||
await runStartCaseExperiment(
|
||||
{ scenario: "Condition D treatment test" },
|
||||
null,
|
||||
{
|
||||
condition: "D",
|
||||
treatment: "no21",
|
||||
startCase: async (input) => {
|
||||
receivedScenario = input.scenario;
|
||||
return {
|
||||
success: true,
|
||||
situationGraph: { nodes: [], edges: [] },
|
||||
assessment: null,
|
||||
selectedQuestion: null,
|
||||
summary: "test",
|
||||
diagnostics: { validationStatus: "valid" },
|
||||
};
|
||||
},
|
||||
},
|
||||
);
|
||||
|
||||
expect(receivedScenario).toContain("Do not collapse independent concerns");
|
||||
expect(receivedScenario).not.toContain("Do not create artificial uncertainties to increase their number.");
|
||||
});
|
||||
|
||||
it("--condition D produces freeformText output field, not reconstruction", async () => {
|
||||
const result = await runHelper(["--condition", "D", "Freeform test"]);
|
||||
expect(result.exitCode).toBe(0);
|
||||
const parsed = parseOutput(result);
|
||||
expect(parsed).toHaveProperty("freeformText");
|
||||
expect(parsed.conditionType).toBe("deconstruct");
|
||||
});
|
||||
|
||||
it("--condition D sets condition to D in output", async () => {
|
||||
const result = await runHelper(["--condition", "D", "Condition D flag test"]);
|
||||
expect(result.exitCode).toBe(0);
|
||||
const parsed = parseOutput(result);
|
||||
expect(parsed.condition).toBe("D");
|
||||
});
|
||||
});
|
||||
|
||||
describe("J.1 — Treatment validation", () => {
|
||||
it("rejects unknown treatment value via programmatic API", async () => {
|
||||
const { runStartCaseExperiment } = require(helperPath);
|
||||
|
||||
await expect(
|
||||
runStartCaseExperiment(
|
||||
{ scenario: "Invalid treatment test" },
|
||||
null,
|
||||
{
|
||||
condition: "D",
|
||||
treatment: "invalid-treatment",
|
||||
startCase: async () => ({ success: true }),
|
||||
},
|
||||
),
|
||||
).rejects.toThrow("Invalid treatment: invalid-treatment. Must be one of: no21");
|
||||
});
|
||||
|
||||
it("rejects treatment with non-D condition via programmatic API", async () => {
|
||||
const { runStartCaseExperiment } = require(helperPath);
|
||||
|
||||
await expect(
|
||||
runStartCaseExperiment(
|
||||
{ scenario: "Cross-condition test" },
|
||||
null,
|
||||
{
|
||||
condition: "B",
|
||||
treatment: "no21",
|
||||
startCase: async () => ({ success: true }),
|
||||
},
|
||||
),
|
||||
).rejects.toThrow('Treatment "no21" is only valid with --condition D or E, not --condition B');
|
||||
});
|
||||
|
||||
it("rejects unknown condition value via CLI", async () => {
|
||||
const result = await runHelper(["--condition", "Z", "Invalid condition test"]);
|
||||
expect(result.exitCode).not.toBe(0);
|
||||
const parsed = parseOutput(result);
|
||||
if (parsed) {
|
||||
expect(parsed.success).toBe(false);
|
||||
expect(typeof parsed.error).toBe("string");
|
||||
}
|
||||
});
|
||||
|
||||
it("rejects treatment with non-D condition via CLI", async () => {
|
||||
const result = await runHelper(["--condition", "B", "--treatment", "no21", "Cross-condition CLI test"]);
|
||||
expect(result.exitCode).not.toBe(0);
|
||||
const parsed = parseOutput(result);
|
||||
if (parsed) {
|
||||
expect(parsed.success).toBe(false);
|
||||
expect(typeof parsed.error).toBe("string");
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe("J.2 — Prompt isolation verification", () => {
|
||||
it("original and treatment prompts differ only by removal of rule 21", async () => {
|
||||
const fs = require("fs");
|
||||
const path = require("path");
|
||||
const originalPrompt = fs.readFileSync(join(rootDir, "prompts", "reconstruct-v0.5-d-deconstruct.md"), "utf8");
|
||||
const treatmentPrompt = fs.readFileSync(join(rootDir, "prompts", "reconstruct-v0.5-d-deconstruct-no21.md"), "utf8");
|
||||
|
||||
expect(originalPrompt).not.toBe(treatmentPrompt);
|
||||
|
||||
// The only difference should be the removal of rule 21 line and its blank line
|
||||
const originalLines = originalPrompt.split("\n");
|
||||
const treatmentLines = treatmentPrompt.split("\n");
|
||||
|
||||
// Find where they diverge — rule 21 is on line 21
|
||||
expect(originalLines[20]).toBe("Do not create artificial uncertainties to increase their number.");
|
||||
|
||||
// After removal, everything else should shift by exactly one line
|
||||
for (let i = 21; i < treatmentLines.length; i++) {
|
||||
expect(originalLines[i + 1]).toBe(treatmentLines[i]);
|
||||
}
|
||||
});
|
||||
|
||||
it("both prompts share identical preamble through rule 20", async () => {
|
||||
const fs = require("fs");
|
||||
const originalPrompt = fs.readFileSync(join(rootDir, "prompts", "reconstruct-v0.5-d-deconstruct.md"), "utf8");
|
||||
const treatmentPrompt = fs.readFileSync(join(rootDir, "prompts", "reconstruct-v0.5-d-deconstruct-no21.md"), "utf8");
|
||||
|
||||
// First 20 lines should be identical (before rule 21)
|
||||
const originalPreamble = originalPrompt.split("\n").slice(0, 20).join("\n");
|
||||
const treatmentPreamble = treatmentPrompt.split("\n").slice(0, 20).join("\n");
|
||||
expect(originalPreamble).toBe(treatmentPreamble);
|
||||
});
|
||||
|
||||
it("original prompt still contains rule 21 verbatim", async () => {
|
||||
const fs = require("fs");
|
||||
const originalPrompt = fs.readFileSync(join(rootDir, "prompts", "reconstruct-v0.5-d-deconstruct.md"), "utf8");
|
||||
expect(originalPrompt).toContain("Do not create artificial uncertainties to increase their number.");
|
||||
});
|
||||
|
||||
it("treatment prompt does NOT contain rule 21", async () => {
|
||||
const fs = require("fs");
|
||||
const treatmentPrompt = fs.readFileSync(join(rootDir, "prompts", "reconstruct-v0.5-d-deconstruct-no21.md"), "utf8");
|
||||
expect(treatmentPrompt).not.toContain("Do not create artificial uncertainties to increase their number.");
|
||||
});
|
||||
|
||||
it("Condition D output cannot overwrite original Condition D evidence — distinct freeformText field", async () => {
|
||||
const resultTreatment = await runHelper(["--condition", "D", "--treatment", "no21", "Treatment output test"]);
|
||||
const resultOriginal = await runHelper(["--condition", "D", "Original Condition D test"]);
|
||||
|
||||
expect(resultTreatment.exitCode).toBe(0);
|
||||
expect(resultOriginal.exitCode).toBe(0);
|
||||
|
||||
const treatmentOutput = parseOutput(resultTreatment);
|
||||
const originalOutput = parseOutput(resultOriginal);
|
||||
|
||||
// Both should have freeformText (not reconstruction) for Condition D
|
||||
expect(treatmentOutput).toHaveProperty("freeformText");
|
||||
expect(originalOutput).toHaveProperty("freeformText");
|
||||
expect(treatmentOutput).not.toHaveProperty("reconstruction");
|
||||
expect(originalOutput).not.toHaveProperty("reconstruction");
|
||||
|
||||
// They use the same output structure (freeformText) but different content —
|
||||
// the test double returns deterministic text that confirms prompt selection works
|
||||
expect(typeof treatmentOutput.freeformText).toBe("string");
|
||||
expect(typeof originalOutput.freeformText).toBe("string");
|
||||
});
|
||||
});
|
||||
|
||||
// ── K — Condition E investigability-first apparatus ────────
|
||||
|
||||
describe("K — Condition E investigability-first apparatus", () => {
|
||||
it("--condition E selects the investigability-first prompt rather than deconstruct", async () => {
|
||||
const { runStartCaseExperiment } = require(helperPath);
|
||||
let receivedScenario;
|
||||
|
||||
await runStartCaseExperiment(
|
||||
{ scenario: "Condition E test" },
|
||||
null,
|
||||
{
|
||||
condition: "E",
|
||||
startCase: async (input) => {
|
||||
receivedScenario = input.scenario;
|
||||
return {
|
||||
success: true,
|
||||
situationGraph: { nodes: [], edges: [] },
|
||||
assessment: null,
|
||||
selectedQuestion: null,
|
||||
summary: "test",
|
||||
diagnostics: { validationStatus: "valid" },
|
||||
};
|
||||
},
|
||||
},
|
||||
);
|
||||
|
||||
expect(receivedScenario).toContain("You are a neutral analyst performing deconstruction of the user's situation");
|
||||
expect(receivedScenario).toContain("Do not create artificial uncertainties to increase their number.");
|
||||
});
|
||||
|
||||
it("--condition E --treatment e-investigability-first is accepted", async () => {
|
||||
const { runStartCaseExperiment } = require(helperPath);
|
||||
|
||||
await runStartCaseExperiment(
|
||||
{ scenario: "Condition E treatment test" },
|
||||
null,
|
||||
{
|
||||
condition: "E",
|
||||
treatment: "e-investigability-first",
|
||||
startCase: async (input) => {
|
||||
return {
|
||||
success: true,
|
||||
situationGraph: { nodes: [], edges: [] },
|
||||
assessment: null,
|
||||
selectedQuestion: null,
|
||||
summary: "test",
|
||||
diagnostics: { validationStatus: "valid" },
|
||||
};
|
||||
},
|
||||
},
|
||||
);
|
||||
|
||||
// If we reach here without an error, the treatment was accepted
|
||||
});
|
||||
|
||||
it("--condition E produces freeformText output field via CLI mock", async () => {
|
||||
const result = await runHelper(["--condition", "E", "Freeform test"]);
|
||||
expect(result.exitCode).toBe(0);
|
||||
const parsed = parseOutput(result);
|
||||
expect(parsed).toHaveProperty("freeformText");
|
||||
expect(typeof parsed.freeformText).toBe("string");
|
||||
expect(parsed.conditionType).toBe("investigability-first");
|
||||
});
|
||||
|
||||
it("--condition E sets condition to E in output", async () => {
|
||||
const result = await runHelper(["--condition", "E", "Condition E flag test"]);
|
||||
expect(result.exitCode).toBe(0);
|
||||
const parsed = parseOutput(result);
|
||||
expect(parsed.condition).toBe("E");
|
||||
});
|
||||
|
||||
it("--condition E does not return reconstruction field in output", async () => {
|
||||
const result = await runHelper(["--condition", "E", "No reconstruction test"]);
|
||||
expect(result.exitCode).toBe(0);
|
||||
const parsed = parseOutput(result);
|
||||
expect(parsed).not.toHaveProperty("reconstruction");
|
||||
});
|
||||
|
||||
it("--condition E --reconstruction-only sets execution.condition to E", async () => {
|
||||
const result = await runHelper(["--condition", "E", "--reconstruction-only", "Reconstruction only test"]);
|
||||
expect(result.exitCode).toBe(0);
|
||||
const parsed = parseOutput(result);
|
||||
expect(parsed.execution).toBeDefined();
|
||||
expect(parsed.execution.condition).toBe("E");
|
||||
});
|
||||
|
||||
it("--condition E rejects unknown treatment via programmatic API", async () => {
|
||||
const { runStartCaseExperiment } = require(helperPath);
|
||||
|
||||
await expect(
|
||||
runStartCaseExperiment(
|
||||
{ scenario: "Invalid treatment test" },
|
||||
null,
|
||||
{
|
||||
condition: "E",
|
||||
treatment: "invalid-treatment",
|
||||
startCase: async () => ({ success: true }),
|
||||
},
|
||||
),
|
||||
).rejects.toThrow("Invalid treatment: invalid-treatment. Must be one of: e-investigability-first");
|
||||
});
|
||||
|
||||
it("--condition E rejects treatment with non-E condition via programmatic API", async () => {
|
||||
const { runStartCaseExperiment } = require(helperPath);
|
||||
|
||||
await expect(
|
||||
runStartCaseExperiment(
|
||||
{ scenario: "Cross-condition test" },
|
||||
null,
|
||||
{
|
||||
condition: "B",
|
||||
treatment: "e-investigability-first",
|
||||
startCase: async () => ({ success: true }),
|
||||
},
|
||||
),
|
||||
).rejects.toThrow('Treatment "e-investigability-first" is only valid with --condition D or E, not --condition B');
|
||||
});
|
||||
|
||||
it("--condition E exposes non-empty freeformText (mock)", async () => {
|
||||
const result = await runHelper(["--condition", "E", "Freeform content test"]);
|
||||
expect(result.exitCode).toBe(0);
|
||||
const parsed = parseOutput(result);
|
||||
expect(parsed.freeformText.length).toBeGreaterThan(0);
|
||||
});
|
||||
});
|
||||
|
||||
});
|
||||
|
||||
// ── Evidence persistence wrapper tests ─────────────────────────────
|
||||
|
||||
var wrapperPath = join(rootDir, "scripts", "evidence-persist.cjs");
|
||||
|
||||
function runWrapper(args, envOverrides) {
|
||||
return new Promise(function(resolve) {
|
||||
execFile(
|
||||
process.execPath,
|
||||
[wrapperPath].concat(args),
|
||||
{
|
||||
cwd: rootDir,
|
||||
timeout: 30000,
|
||||
env: Object.assign({}, process.env, {
|
||||
START_CASE_EXPERIMENT_HELPER_MOCK: "1",
|
||||
}, envOverrides || {}),
|
||||
},
|
||||
function(error, stdout, stderr) {
|
||||
resolve({
|
||||
error: !!error,
|
||||
exitCode: error ? error.code : 0,
|
||||
stdout: stdout || "",
|
||||
stderr: stderr || "",
|
||||
});
|
||||
}
|
||||
);
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Tests for the evidence-persist.cjs wrapper.
|
||||
*
|
||||
* Uses START_CASE_EXPERIMENT_HELPER_MOCK=1 for deterministic behavior.
|
||||
*/
|
||||
|
||||
describe("evidence persistence", function() {
|
||||
|
||||
it("successful mock invocation publishes valid evidence with exact raw stdout bytes preserved", async function() {
|
||||
var testOutput = join(rootDir, ".evidence-temp", "headless-semantic", "_test_persist_d.json");
|
||||
var fixturePath = join(rootDir, "tests", "fixtures", "evidence-persist-input.json");
|
||||
|
||||
try {
|
||||
var result = await runWrapper([
|
||||
"--condition", "D",
|
||||
"--file", fixturePath,
|
||||
"--output", testOutput
|
||||
]);
|
||||
|
||||
expect(result.exitCode).toBe(0);
|
||||
expect(result.error).toBe(false);
|
||||
|
||||
// Verify the evidence file was published
|
||||
var statSync = require("fs").statSync;
|
||||
var exists = false;
|
||||
try { statSync(testOutput); exists = true; } catch(e) {}
|
||||
expect(exists).toBe(true);
|
||||
|
||||
// Verify the published file is valid JSON matching what helper outputs
|
||||
var rawContent = await readFile(testOutput, "utf-8");
|
||||
var published = JSON.parse(rawContent.trim());
|
||||
expect(published.success).toBe(true);
|
||||
expect(published.statusCode).toBe(200);
|
||||
expect(published.condition).toBe("D");
|
||||
expect(typeof published.freeformText).toBe("string");
|
||||
expect(published.freeformText.length).toBeGreaterThan(0);
|
||||
expect(published.diagnostics.reconstructionOnly).toBe(true);
|
||||
|
||||
// Verify raw bytes are preserved (no reformatting)
|
||||
var reparseable = JSON.parse(rawContent.trim());
|
||||
expect(reparseable.condition).toBe("D");
|
||||
} finally {
|
||||
try { await unlink(testOutput); } catch(e2) {}
|
||||
}
|
||||
});
|
||||
|
||||
it("forced helper failure does not publish evidence", async function() {
|
||||
var testOutput = join(rootDir, ".evidence-temp", "headless-semantic", "_test_persist_fail.json");
|
||||
var fixturePath = join(rootDir, "tests", "fixtures", "evidence-persist-input.json");
|
||||
|
||||
try {
|
||||
var result = await runWrapper([
|
||||
"--condition", "D",
|
||||
"--file", fixturePath,
|
||||
"--output", testOutput
|
||||
], { START_CASE_EXPERIMENT_HELPER_FORCE_FAIL: "1" });
|
||||
|
||||
expect(result.exitCode).not.toBe(0);
|
||||
|
||||
// Verify the final destination was NOT created
|
||||
var statSync = require("fs").statSync;
|
||||
var exists = false;
|
||||
try { statSync(testOutput); exists = true; } catch(e) {}
|
||||
expect(exists).toBe(false);
|
||||
} finally {
|
||||
try { await unlink(testOutput); } catch(e2) {}
|
||||
}
|
||||
});
|
||||
|
||||
it("malformed or incomplete output is rejected without publication", async function() {
|
||||
var testOutput = join(rootDir, ".evidence-temp", "headless-semantic", "_test_persist_malformed.json");
|
||||
var fixturePath = join(rootDir, "tests", "fixtures", "evidence-persist-input.json");
|
||||
|
||||
try {
|
||||
var result = await runWrapper([
|
||||
"--condition", "D",
|
||||
"--file", fixturePath,
|
||||
"--output", testOutput
|
||||
], { START_CASE_EXPERIMENT_HELPER_FORCE_FAIL: "1" });
|
||||
|
||||
expect(result.exitCode).not.toBe(0);
|
||||
|
||||
// Verify destination was NOT created (rejection without publication)
|
||||
var statSync2 = require("fs").statSync;
|
||||
var exists = false;
|
||||
try { statSync2(testOutput); exists = true; } catch(e) {}
|
||||
expect(exists).toBe(false);
|
||||
} finally {
|
||||
try { await unlink(testOutput); } catch(e2) {}
|
||||
}
|
||||
});
|
||||
|
||||
it("an existing destination is preserved unchanged", async function() {
|
||||
var testOutput = join(rootDir, ".evidence-temp", "headless-semantic", "_test_persist_existing.json");
|
||||
var fixturePath = join(rootDir, "tests", "fixtures", "evidence-persist-input.json");
|
||||
|
||||
try {
|
||||
// Pre-create the destination with known content
|
||||
await writeFile(testOutput, JSON.stringify({ preserved: true }), "utf-8");
|
||||
var original = JSON.parse((await readFile(testOutput, "utf-8")).trim());
|
||||
expect(original.preserved).toBe(true);
|
||||
|
||||
// Attempt to publish — should be rejected due to existing destination
|
||||
var result = await runWrapper([
|
||||
"--condition", "D",
|
||||
"--file", fixturePath,
|
||||
"--output", testOutput
|
||||
]);
|
||||
|
||||
expect(result.exitCode).not.toBe(0);
|
||||
|
||||
// Verify the original content was NOT modified
|
||||
var final = JSON.parse((await readFile(testOutput, "utf-8")).trim());
|
||||
expect(final.preserved).toBe(true);
|
||||
} finally {
|
||||
try { await unlink(testOutput); } catch(e2) {}
|
||||
}
|
||||
});
|
||||
|
||||
it("two competing publication attempts cannot both claim the same destination", async function() {
|
||||
var testOutput = join(rootDir, ".evidence-temp", "headless-semantic", "_test_persist_competing.json");
|
||||
var fixturePath = join(rootDir, "tests", "fixtures", "evidence-persist-input.json");
|
||||
|
||||
try {
|
||||
// First publication — succeeds
|
||||
var result1 = await runWrapper([
|
||||
"--condition", "D",
|
||||
"--file", fixturePath,
|
||||
"--output", testOutput
|
||||
]);
|
||||
expect(result1.exitCode).toBe(0);
|
||||
expect(result1.error).toBe(false);
|
||||
|
||||
// Verify winning evidence bytes
|
||||
var rawBytes1 = await readFile(testOutput, "utf-8");
|
||||
var evidence1 = JSON.parse(rawBytes1.trim());
|
||||
expect(evidence1.success).toBe(true);
|
||||
expect(evidence1.statusCode).toBe(200);
|
||||
expect(evidence1.condition).toBe("D");
|
||||
|
||||
// Simulate a competing claim: write different content to the same destination
|
||||
var competingContent = JSON.stringify({ overwrittenBy: "competing-claim" });
|
||||
await writeFile(testOutput, competingContent, "utf-8");
|
||||
|
||||
// Second publication to the same path — must fail (EEXIST)
|
||||
var result2 = await runWrapper([
|
||||
"--condition", "D",
|
||||
"--file", fixturePath,
|
||||
"--output", testOutput
|
||||
]);
|
||||
expect(result2.exitCode).not.toBe(0);
|
||||
|
||||
// Verify winning evidence bytes remain unchanged:
|
||||
// file still has the competing claim content (second wrapper did NOT overwrite it)
|
||||
var rawBytes2 = await readFile(testOutput, "utf-8");
|
||||
expect(rawBytes2).toBe(competingContent);
|
||||
|
||||
// Also verify first publication bytes are intact — they differ from current file,
|
||||
// confirming the second wrapper could not reach them
|
||||
expect(rawBytes2).not.toBe(rawBytes1);
|
||||
} finally {
|
||||
try { await unlink(testOutput); } catch(e2) {}
|
||||
}
|
||||
});
|
||||
|
||||
it("linkSync hardlink publication rejects EEXIST when a second process targets an already-published destination", async function() {
|
||||
var testOutput = join(rootDir, ".evidence-temp", "headless-semantic", "_test_persist_link_a.json");
|
||||
var fixturePath = join(rootDir, "tests", "fixtures", "evidence-persist-input.json");
|
||||
|
||||
try {
|
||||
// First wrapper run — publishes successfully (linkSync creates hardlink at testOutput)
|
||||
var resultA = await runWrapper([
|
||||
"--condition", "D",
|
||||
"--file", fixturePath,
|
||||
"--output", testOutput
|
||||
]);
|
||||
expect(resultA.exitCode).toBe(0);
|
||||
|
||||
// Verify the first publication produced valid evidence
|
||||
var publishedA = JSON.parse((await readFile(testOutput, "utf-8")).trim());
|
||||
expect(publishedA.success).toBe(true);
|
||||
expect(publishedA.statusCode).toBe(200);
|
||||
expect(publishedA.condition).toBe("D");
|
||||
expect(typeof publishedA.freeformText).toBe("string");
|
||||
expect(publishedA.diagnostics.reconstructionOnly).toBe(true);
|
||||
|
||||
// Second wrapper targets the exact same destination path.
|
||||
// linkSync(tmpFile, testOutput) must throw EEXIST because testOutput already exists
|
||||
// as a hardlink to the first publication's data (same inode).
|
||||
var resultB = await runWrapper([
|
||||
"--condition", "D",
|
||||
"--file", fixturePath,
|
||||
"--output", testOutput
|
||||
]);
|
||||
expect(resultB.exitCode).not.toBe(0);
|
||||
|
||||
// Verify the first publication's evidence bytes remain exactly as published:
|
||||
// no-write guarantee confirmed — linkSync refuses to modify existing destination.
|
||||
var preserved = JSON.parse((await readFile(testOutput, "utf-8")).trim());
|
||||
expect(preserved.success).toBe(true);
|
||||
expect(preserved.statusCode).toBe(200);
|
||||
expect(preserved.condition).toBe("D");
|
||||
} finally {
|
||||
try { await unlink(testOutput); } catch(e) {}
|
||||
}
|
||||
});
|
||||
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user