confidence-engine/docs/experiment-57j64.md

Experiment 57J.64 — Semantic-to-Mutation Action Ownership Diagnosis

Branch: feature/selected-question-contract-v0.22 Starting HEAD: f022d6f (experiment: rerun equivalent uncertainty identity with hardened capture)

Objective

Answer exactly:

Given that prior live runs have sometimes created a dedicated savings-realism unknown and other runs have produced only userSupportedMeaning with zero meaningful mutation, what production contract allows both outcomes for the same semantic class?

Do not diagnose this as generic "model variance" unless the production contract truly leaves both outcomes valid.

---

Part 1 — Prompt Contract Trace

Rules governing the structural-mutation obligation

Rule	Text (summary)
#6	MUST express consequential meaning through structural mutation. answerMeaning alone is not sufficient.
#7	Add new unknown only for genuinely new decision/claim/object/measurement/dependency/unresolved term.
#9–9a	Every new unknown traceable to answer with why-it-matters clause.
#11	No duplicate unknowns.
#13–13a	Connect new unknowns via edge to existing nodes.
#16	When adding new unresolved unknowns, MUST include selectedQuestion.
Additional Guidance (line 2)	If no equivalent node exists → add a new unknown; do not use edge alone.
Additional Guidance (line 8)	answerMeaning.preserves semantic fidelity while structural mutation handles graph progress

Is there a legitimate path to zero structural mutation?

Answer: PARTIAL

The prompt gives two overlapping obligations:

1. Rule #6: When userSupportedMeaning contains consequential information → MUST mutate structurally.
2. Additional Guidance (line 8): If rule #6 does not apply → return empty arrays.

These overlap because the model must decide whether rule #6 applies. The prompt provides no deterministic test for "consequential" or "unresolved uncertainty that is not already represented." The model can legitimately reason: "rule #6 does not apply — the answer doesn't introduce a genuinely new unknown" → empty arrays.

This creates a legitimate escape hatch even though it contradicts the outcome of 57J.60 (where the same semantic class produced structural mutation). No prompt ambiguity exists per se — rule #6 is unambiguous in its "MUST" language. But the model must make the threshold decision ("is this consequential?") without a deterministic reference, and that decision point is where variance enters.

---

Part 2 — Action-Selection Completeness

Does the prompt provide a complete decision sequence?

Answer: PARTIAL

The Effective sequence implied by rules #6 + Additional Guidance line 2 is:

1. Does userSupportedMeaning contain unresolved uncertainty?
2. Is an equivalent unresolved node already present?
3. If yes → reuse/refine existing.
4. If no → add a new unknown.
5. Provide selectedQuestion if new unresolved unknown added.

This is complete as a model instruction. But it's not enforced by code. The prompt does not say: "If step 2 returns true, you must produce addedNodes containing the new unknown and an addedEdge connecting it." The prompt tells the model what to do — but if the model skips to step "empty arrays" at any point, the validator only rejects, it doesn't correct.

Remaining escape hatch

answerMeaning.userSupportedMeaning populated
supportCategory = null (model did not populate structured field)
resolutionGuidance = null (model did not populate structured field)
addedNodes = []
addedEdges = []

This proposal is valid JSON, semantically consistent with the raw answer, structurally minimal — but violates rule #6's MUST obligation. The validator rejects it post-hoc. There is no pre-validation path that catches this before the model sends it.

---

Part 3 — Structured-Field Influence

Does the structural-mutation obligation depend on supportCategory/resolutionGuidance being populated?

Answer: NO (contractually) / PARTIAL (operationally)

Contractually: The mutation obligation derives entirely from rule #6 and userSupportedMeaning text content. The prompt instructs supportCategory/population in rules 28–32 as a model output requirement, not as an input that gates downstream behavior.

Operationally: In practice, when the model produces supportCategory=uncertain (as in 57J.60), it also tends to produce structural mutation. When supportCategory=null (as in 57J.63), zero mutation occurs. The question is whether this correlation is causal.

Tracing the actual pipeline:

1. Model produces response with answerMeaning + proposal.
2. Orchestrator passes both to validator.
3. Validator checks answerMeaning.userSupportedMeaning for structural progress (line 886 of utils.js). If populated with zero mutation → reject.
4. getAnswerMeaningProfile() derives fallback category/resolutionGuidance from userSupportedMeaning text if model did not populate them.
5. Derived values (category: "uncertain", resolutionGuidance: "must_remain_unresolved") are used only for downstream validator cross-checks (e.g., must_not_resolve when must_remain_unresolved).

No code path uses derived supportCategory/resolutionGuidance to mandate structural mutation. The structured fields only feed into the deterministic profile, which is then used for compatibility checking — not for generating mutations.

The apparent correlation between populated structured fields and successful structural mutation is a model-behavior pattern, not an architectural dependency. When the model commits to a category label, it has already decided what semantic action it's taking. The correlation reflects downstream consistency of the model's own output rather than any enforcement mechanism in the engine.

---

Part 4 — Validator Role

What can the validator do?

Capability	Answer
Transform semantic meaning into structure	NO
Choose update-vs-add	NO
Repair a missing unknown	NO
Trigger regeneration	NO

Validator classification: ENFORCEMENT ONLY

The validator's entire role is rejection: it rejects proposals that violate constraints (empty arrays when meaning populated, stronger-than-raw meaning, resolution against must_remain_unresolved, etc.). It has zero recovery/repair capability. After rejection, the experiment apparatus stops — no regeneration, no second attempt, no automated repair.

---

Part 5 — Model Responsibility Boundary

Who owns the actual choice: reuse existing / add new / emit nothing?

Answer: MODEL

The deterministic engine provides:

1. The graph state (input).
2. Prompt instructions (output contract).
3. Post-hoc validation (rejection of invalid proposals).

But it does NOT contain:

• Deterministic decision logic for action selection.
• Any function that translates userSupportedMeaning + derived category into a concrete proposal (addedNodes/updatedNodes/resolvedUnknownNodeIds).
• A bounded repair mechanism when the model's proposal fails validation.

The model produces both answerMeaning AND structural mutation independently. The validator checks consistency but does not bridge gaps.

Can the model violate invariant and simply receive rejection?

Answer: YES

The proposal format contract allows valid JSON with populated answerMeaning and empty structural arrays. This is syntactically valid, semantically consistent with the raw answer, and structurally minimal. The validator's rejection happens after the model's response is parsed — there is no pre-validation check that requires structural action before the prompt is sent.

---

Part 6 — Architecture Classification

Classification: E — MIXED

Three independently verifiable factors contribute:

B — MODEL NONCOMPLIANCE WITH COMPLETE CONTRACT

The prompt does unambiguously require structural action (rule #6 "MUST") and structured fields (rules 28–32). The model occasionally returns supportCategory=null + zero mutation, which violates both sets of instructions. This is genuine noncompliance, not a contract gap.

C — STRUCTURED-FIELD DEPENDENCY GAP

Reliability correlates with structured field population because:

• When the model populates supportCategory=uncertain, it has semantically committed to an action class that implies structural work → downstream consistency follows.
• When supportCategory=null, the deterministic fallback derives the same category (category: "uncertain") but this derivation only feeds validator cross-checks — not action selection. The gap between derivation and action is the reliability problem.

D — VALIDATOR/RECOVERY ARCHITECTURE GAP

The model owns action selection entirely. Deterministic code rejects invalid output but has no bounded recovery path (no regeneration, no repair, no deterministic fallback mutation). This means every noncompliant proposal is a hard stop, not a transient failure state.

---

Part 7 — Anti-Keyword Architecture Decision

Option 1 — More prompt wording

Would another prompt clarification add a genuinely missing rule?

NO. Rule #6 already uses "MUST" for the structural-mutation obligation. Rules 28–32 already mandate structured field population. Additional wording would be incremental, not boundary-crossing.

Option 2 — Deterministic raw-text semantics

Would detecting words like "unsure", "uncertain", "whether", "need evidence" create keyword-dictionary reasoning?

YES. Any system that maps lexical signals directly to structural actions bypasses semantic understanding and reverts to pattern-matching, which is precisely what the current architecture was designed to avoid.

Option 3 — Structured semantic action contract

Could the model be required to explicitly state an action classification such as "existing unknown already supports" or "need new unknown", with deterministic code enforcing the corresponding mutation?

VIABLE WITH EXISTING STRUCTURE. The derived profile (category + resolutionGuidance) already exists and captures the necessary classification. Adding a small mutationIntent field to the model output contract — one of: reuse_existing, add_new_unknown, resolve_existing, no_change_needed — with deterministic enforcement (when userSupportedMeaning is populated, mutationIntent must be non-null; when it's add_new_unknown, addedNodes must be non-empty) would close the gap without inventing new taxonomies.

Option 4 — Bounded proposal repair

Could rejection of "faithful userSupportedMeaning + zero mutation" trigger one bounded repair attempt?

ARCHITECTURALLY VIABLE. This would require: (1) detecting the semantic-only-no-op error specifically, (2) re-sending the prompt with an explicit note that structural action is required (not just rejection), and (3) a strict call budget limit. The existing harness architecture supports bounded retry patterns — it's just been explicitly forbidden by protocol for experiments. For production, this is architecturally viable.

---

Part 8 — Smallest Next Production Boundary

Recommended next boundary: B — structured action-contract implementation

Why smaller and safer than alternatives:

• Smaller than A (prompt clarification): Prompt wording changes are the most fragile form of fix — they depend on model compliance every turn. The contract gap is architectural, not linguistic. Adding a mutationIntent field to the output schema (one enum value per actionable case) closes the gap at the data-contract layer, where it can be validated deterministically before acceptance.

• Smaller than C (bounded proposal repair): Repair adds a second API call, which increases latency and introduces new failure modes (the model may still refuse to mutate on retry). A contract-level fix prevents the noncompliant output from being accepted in the first place.

• Safer than D (deterministic orchestration change): Deterministically generating mutations based on derived semantics risks reverting to keyword-dictionary reasoning. The structured action-contract keeps semantic understanding in the model while adding a deterministic enforcement layer on the output, not the input.

This is the smallest boundary because it changes only the output contract shape (one new optional field) and the validator (reject null mutationIntent when userSupportedMeaning is populated). It does not modify the reasoning pipeline, the prompt, or the scoring system.

---

Convergence

PROMPT-ONLY PATH EXHAUSTED: YES

The next production boundary is a structured action-contract extension: adding a deterministic mutationIntent field to the model output contract that explicitly states which structural action the answerMeaning implies (e.g., "add_new_unknown", "reuse_existing", "resolve_existing", "no_change"), validated by code before proposal acceptance. This moves the semantics-to-mutation bridge from prompt-instruction-reliance to contract-enforcement, without resorting to deterministic keyword detection or model regeneration loops.

---

Production code changed

NO

Prompt changed: NO

Validator changed: NO

Schema changed: NO

Tests changed: NO

Ollama calls: 0

Dev server disturbed: NO