242 lines
19 KiB
Markdown
242 lines
19 KiB
Markdown
# Experiment 57J.42 — Structural-Mutation MUST Rule: Prompt Conflict Diagnosis
|
||
|
||
**Branch:** `feature/semantic-to-mutation-contract-v0.17`
|
||
**Starting HEAD:** 6aea0bd experiment: isolate semantic-to-mutation contract live
|
||
**Experiment commit:** pending
|
||
|
||
## Objective
|
||
|
||
Answer exactly:
|
||
|
||
> Why can the model still produce faithful `userSupportedMeaning` with zero structural mutation despite the new v0.17 MUST rule? Is another prompt instruction conflicting with, weakening, or making that obligation operationally ambiguous?
|
||
|
||
57J.41 already proved the live failure — a faithful proposal with zero structural fields across all four categories. This is a read-only prompt-contract diagnosis. No Ollama calls. No API calls. No code/prompt/test changes.
|
||
|
||
## Context Route
|
||
|
||
Read files:
|
||
1. `docs/current-handoff.md` (57J.41 entry)
|
||
2. `lib/graph/prompt-builder.js` — the complete assembled graph-update prompt
|
||
3. `tests/graph/prompt-builder.test.js` — focused tests on the MUST rule
|
||
|
||
**Source read budget:** ~300 lines of prompt-builder.js + ~180 lines of test file.
|
||
|
||
## Controlled Case Walkthrough
|
||
|
||
**Meaning:** "The user is unsure whether the projected office savings from the relocation are realistic."
|
||
**possibleInference:** null
|
||
**Graph assumption:** current graph does NOT obviously contain a node named "realism of projected office savings" or semantic equivalent. The start produced 6 nodes — these are broad (operating costs, cost structure, etc.) but not an exact match for "savings realism".
|
||
|
||
### What the prompt clearly requires
|
||
|
||
Walking through rule-by-rule as the model would:
|
||
|
||
**Step 1: Extract meaning.** Rule #26–30 apply. The answer says the user is unsure about savings realism. This goes into `userSupportedMeaning` per rules #26 and #30 (direct uncertainty). ✓ Clear obligation.
|
||
|
||
**Step 2: Assess consequentiality.** Rule #6 triggers — the userSupportedMeaning contains unresolved uncertainty ("unsure whether projected office savings are realistic") which is consequential to the case (relocation decision). The prompt says MUST express its effect through structural mutation. ✓ Obligation exists.
|
||
|
||
**Step 3: Choose action path.** Four options listed by rule #6:
|
||
a) update/refine existing structure
|
||
b) resolve an existing unknown
|
||
c) a genuinely new unknown
|
||
d) a justified relationship
|
||
|
||
The model must decide which of these four paths to take. This is where ambiguity arises (see below).
|
||
|
||
## RELEVANT PROMPT RULES
|
||
|
||
### 1. Rule #6 — The v0.17 MUST Rule
|
||
**Location:** prompt-builder.js line ~95, "Proposal Rules" section
|
||
**Strength:** **MUST** ("you MUST express its effect through structural mutation")
|
||
**Effect on structural mutation:** ENCOURAGES + OBLIGATES
|
||
**Meaning:** If userSupportedMeaning contains consequential unresolved uncertainty not already represented, MUST express it structurally. AnswerMeaning alone is insufficient. Four acceptable forms: update/refine existing, resolve existing unknown, genuinely new unknown, or justified relationship.
|
||
|
||
### 2. Rule #7 — New Unknown Restriction
|
||
**Location:** prompt-builder.js line ~96, "Proposal Rules" section
|
||
**Strength:** **MUST NOT** (restrictive boundary on *adding* nodes)
|
||
**Effect on structural mutation:** RESTRICTS (specifically the "add new unknown" path)
|
||
**Meaning:** Add new unknown nodes ONLY when the answer introduces a "new decision, claim, object, measure, dependency, or unresolved term directly relevant to the case." The word "adds" — does an uncertainty about realism qualify as an "unresolved term"? Unclear. This is ambiguous for our controlled case because the user didn't introduce a new *concept* — they expressed doubt about an already-mentioned one (projected office savings, which was implicit in the relocation question).
|
||
|
||
### 3. Rule #5 — Resolve Existing Unknown First
|
||
**Location:** prompt-builder.js line ~94
|
||
**Strength:** **SHOULD** ("Resolve the answered unknown first when the answer supports it")
|
||
**Effect on structural mutation:** NEUTRAL → ENCOURAGES (for resolve path)
|
||
**Meaning:** If the answer supports resolving an existing unknown, do so first. Our controlled case does NOT answer any question — it expresses uncertainty about a concept. Rule #5 is inapplicable here.
|
||
|
||
### 4. Additional Guidance Bullet A — Clarification Preference
|
||
**Location:** prompt-builder.js line ~125
|
||
**Strength:** **SHOULD** ("prefer updatedNodes and resolvedUnknownNodeIds over creating duplicate nodes")
|
||
**Effect on structural mutation:** ENCOURAGES (update/resolve path)
|
||
**Meaning:** If the answer only clarifies an existing unknown, prefer updating/resolving. Our controlled case is NOT clarification of an existing unknown — it's introducing a new dimension of uncertainty. This bullet is inapplicable.
|
||
|
||
### 5. Additional Guidance Bullet B — Empty Arrays Permission
|
||
**Location:** prompt-builder.js line ~132
|
||
**Strength:** **PERMITS** ("return empty arrays for every category")
|
||
**Effect on structural mutation:** PERMITS NO-OP (direct conflict with rule #6)
|
||
**Meaning:** "If the answer does not justify a change, return empty arrays for every category." This is the critical conflicting instruction. It provides an escape hatch: if the model decides nothing justifies a change, it may return all-empty arrays including semantic-only content via answerMeaning.
|
||
|
||
### 6. Additional Guidance Bullet C — Semantic Preservation
|
||
**Location:** prompt-builder.js line ~132 (final bullet)
|
||
**Strength:** **PERMITS/ENCOURAGES** ("Use answerMeaning to preserve the answer's direct meaning even when the graph change remains unresolved")
|
||
**Effect on structural mutation:** PERMITS semantic-only output
|
||
**Meaning:** Explicitly encourages using answerMeaning for semantic preservation "even when the graph change remains unresolved." This is permissive of the exact pattern that v0.17 sought to eliminate — populated `answerMeaning` with zero structure.
|
||
|
||
### 7. Rule #4 — AddedNodes Scope
|
||
**Location:** prompt-builder.js line ~93
|
||
**Strength:** **MUST NOT** ("Use addedNodes only for genuinely new concepts")
|
||
**Effect on structural mutation:** RESTRICTS (the "add new unknown" path)
|
||
**Meaning:** New nodes require "genuinely new concepts." The user's uncertainty about savings realism might not qualify as a "new concept" — it's an epistemic state about something already discussed.
|
||
|
||
### 8. Rule #9 — Traceability Requirement
|
||
**Location:** prompt-builder.js line ~98
|
||
**Strength:** **MUST** ("directly traceable to the user's answer")
|
||
**Effect on structural mutation:** ENCOURGES (requires grounded structure)
|
||
**Meaning:** New unknowns must be traceable and explain why they matter. This is clear and achievable but adds complexity cost to the "add" path.
|
||
|
||
### 9. Rule #20 — Null Question Condition
|
||
**Location:** prompt-builder.js line ~111
|
||
**Strength:** **MUST** ("Return selectedQuestion as null only when no consequential unresolved unknown remains")
|
||
**Effect on structural mutation:** NEUTRAL → INDIRECTLY ENCOURAGES mutation
|
||
**Meaning:** Since consequential unresolved uncertainty exists (per rule #6 assessment), the model should NOT return `selectedQuestion: null`. But this doesn't tell it HOW to structure — it only constrains question output.
|
||
|
||
### 10. Rule #26 — User-Supported Meaning Fidelity
|
||
**Location:** prompt-builder.js line ~117
|
||
**Strength:** **MUST** ("state only what the user's answer directly supports")
|
||
**Effect on structural mutation:** NEUTRAL (semantic field constraint)
|
||
**Meaning:** Keep `userSupportedMeaning` faithful. This is what the model did correctly.
|
||
|
||
## CONFLICT CHECKS
|
||
|
||
### Pattern A — MUST vs restrictive "only when"
|
||
**YES** — Partial conflict. Rule #6 says MUST structurally represent consequential meaning. Rule #7 restricts new unknown nodes to cases where the answer introduces "a new decision, claim, object, measure, dependency, or unresolved term." The controlled case (unsure about savings realism) falls in a grey zone: it's not clearly any of those enumerated items. It's an epistemic state (doubt) about something already mentioned. Rule #6 creates the obligation; rule #7 restricts the most natural action (adding a new node). The model cannot satisfy both without knowing which existing node to update.
|
||
|
||
### Pattern B — semantic preservation without structural mapping
|
||
**YES** — The prompt tells the model what the answer means (rules #26-30) but does not provide a decision procedure for choosing among: update existing / resolve existing / add new unknown / add edge. Rule #6 lists the four options but provides no selection criteria or fallback ordering. This is operationally ambiguous when no single path is obviously correct.
|
||
|
||
### Pattern C — duplicate avoidance causing paralysis
|
||
**YES** — Partial. Additional Guidance Bullet A encourages preferring updates over new nodes. Rule #4 says "genuinely new concepts" for addedNodes. Rule #11 prohibits duplicates. Combined, these make the model risk-averse about adding any structure. If it can't find a clearly matching existing node to update AND doesn't feel confident the concept is "genuinely new" (vs. overlapping with existing cost-related nodes), the safest path is no mutation at all.
|
||
|
||
### Pattern D — fidelity/inference paralysis
|
||
**YES** — Partial. Rules #26, #27, and #9 create a high bar: every structural element must be directly traceable to the answer, any stronger interpretation goes in possibleInference, new unknowns must state "why it matters." For a simple uncertainty ("unsure whether realistic"), producing a grounded node with justification is non-trivial when no existing anchor exists. The model may prefer faithfulness without mutation over risking an inferred structural relationship.
|
||
|
||
### Pattern E — surviving semantic-only permission
|
||
**YES** — Clear conflict. Additional Guidance Bullet B states: "If the answer does not justify a change, return empty arrays for every category." Additionally, the final bullet says: "Use answerMeaning to preserve the answer's direct meaning even when the graph change remains unresolved." These two instructions together explicitly permit (and in the case of the last one, encourage) the exact pattern that v0.17's MUST rule was designed to eliminate: populated `answerMeaning` with zero structural mutation. The "does not justify a change" condition can be satisfied if the model interprets rule #7 narrowly — since no enumerated category (decision/claim/object/measure/dependency/unresolved term) is clearly present, nothing justifies a change.
|
||
|
||
### Pattern F — selected-question requirements compete with mutation
|
||
**YES** — Partial. Rules #16 and #20 together tell the model: if unresolved unknowns exist, you may select a question about one; return null only when no consequential unresolved unknown remains. The model can satisfy this by producing a `selectedQuestion` (identifying the uncertainty as a question) WITHOUT any structural mutation — the rule constrains question output but doesn't mandate the structure underlying the question's target node. A model can reason: "I've identified the question (satisfying rule #16/#20). The graph already contains 'operating costs' which I'll use as the nodeId reference. No new structure needed." This satisfies rules #16-20 without touching structural mutation at all.
|
||
|
||
## CONTROLLED CASE
|
||
|
||
**Meaning:** "The user is unsure whether the projected office savings from the relocation are realistic."
|
||
**possibleInference:** null
|
||
**Graph assumption:** no obvious exact node named "realism of projected office savings"
|
||
|
||
### What does the prompt clearly require?
|
||
|
||
**STRUCTURAL OBLIGATION EXISTS BUT ACTION CHOICE IS AMBIGUOUS**
|
||
|
||
Why: Rule #6 creates a MUST obligation for structural mutation. But rules #4, #7, and Additional Guidance provide three separate restrictions that each independently justify choosing no mutation:
|
||
- Rule #7: The answer doesn't clearly introduce a "new decision/claim/object/measure/dependency/unresolved term" — it's doubt about an existing concept.
|
||
- Rule #4: "genuinely new concepts" standard is unclear for epistemic state about known topic.
|
||
- Additional Guidance: "If the answer does not justify a change, return empty arrays" provides explicit escape hatch.
|
||
|
||
The four paths under rule #6 (update/refine/resolve/add) are listed without decision criteria. Without an obviously matching existing node to update, and with no clear permission to add a new unknown, the model faces genuine action-selection ambiguity despite knowing mutation is required.
|
||
|
||
Additionally, Additional Guidance lines 132 explicitly permit semantic-only output, creating a direct MUST vs PERMIT conflict.
|
||
|
||
## EVALUATED DIAGNOSIS OPTIONS
|
||
|
||
### A — RULE IS CLEAR, MODEL SIMPLY FAILED
|
||
**Rejected.** The prompt contains multiple restrictions (rules #4, #7) and permissive escape hatches (Additional Guidance bullets B and C) that provide independent justification for choosing no mutation. This is not a case of ignoring clear instructions.
|
||
|
||
### B — OBLIGATION CLEAR, ACTION CHOICE AMBIGUOUS
|
||
**Partial fit.** The action-selection ambiguity is real and present — rules list four paths without decision criteria. However, this diagnosis is incomplete because it doesn't account for the direct permissive conflicts in Additional Guidance (Pattern E).
|
||
|
||
### C — PROMPT CONFLICT
|
||
**Selected.** Multiple instructions materially conflict with the v0.17 MUST rule:
|
||
- Pattern A (rule #6 vs rule #7): obligation exists but new-node path is restricted by enumeration
|
||
- Pattern E (Additional Guidance bullet B/C): explicitly permits the exact semantic-only no-op pattern that MUST rules out
|
||
- Pattern C (rules #4, #11 + Additional Guidance): duplicate avoidance creates paralysis
|
||
- Pattern D (rules #9, #26, #27): fidelity requirements make structural creation complex
|
||
|
||
These are not edge cases — they are the primary conditions the controlled case exercises. The v0.17 MUST rule is contradicted by surviving permissive instructions at equal prompt hierarchy level (both in "Proposal Rules" and "Additional Guidance" sections).
|
||
|
||
### D — NO-OP STILL PERMITTED
|
||
**Subsumed by C.** Pattern E shows that a no-op is indeed still permitted via Additional Guidance bullets B and C. However, this is itself a manifestation of the broader Prompt Conflict diagnosis.
|
||
|
||
## Provider-Agnostic Check
|
||
|
||
**YES — CONTRACT LEVEL**
|
||
|
||
The same ambiguity/conflict would plausibly affect OpenAI, Anthropic, Gemini, or any other model. The conflict exists at the instruction-contract level: multiple instructions with different obligation strengths (MUST vs PERMIT) operate in tension, and the prompt provides no priority ordering between them. All major models trained to follow instruction hierarchies would face the same ambiguity when MUST creates an obligation and PERMIT/SHOULD provides an escape route for a plausible reading of a restrictive condition.
|
||
|
||
## TEST ADEQUACY
|
||
|
||
### Current prompt tests classification: TEXT PRESENCE ONLY
|
||
|
||
### What they prove:
|
||
- The exact text "MUST express its effect through structural mutation" exists in the assembled prompt
|
||
- The four permitted action forms (update/refine, resolve existing unknown, genuinely new unknown) are present as text
|
||
- "answerMeaning alone is not sufficient" exists as text
|
||
- Rules 4, 7, 8, 18 are present via text matching
|
||
- User-supported meaning vs possibleInference separation instructions exist
|
||
|
||
### What they do not prove:
|
||
- The complete prompt has no conflicting permissive guidance (no test checks for Additional Guidance bullets B/C)
|
||
- The update-vs-add fallback is operationally clear (no test exercises action-selection ambiguity)
|
||
- Rule #7's restrictive boundary doesn't undermine rule #6's obligation
|
||
- The model actually follows the MUST rule when it conflicts with other instructions
|
||
- Any end-to-end prompt coherence
|
||
|
||
The 57J.39 tests only verify that the new MUST sentence was inserted into the prompt text. They do not test whether that sentence survives the full instruction context uncontradicted.
|
||
|
||
## Classification: C — PROMPT CONFLICT
|
||
|
||
### Why:
|
||
|
||
Multiple independent prompt instructions create conditions where zero structural mutation is a defensible, even encouraged, interpretation of the full prompt — despite rule #6's MUST obligation. The conflict patterns A through F are all materially present, not hypothetical. Additional Guidance bullets B and C provide the most direct contradiction by explicitly permitting semantic-only output with empty structural arrays, using the exact same escape condition ("if the answer does not justify a change") that rules #4 and #7 help establish.
|
||
|
||
## Primary owner of 57J.41 failure: PROMPT CONFLICT
|
||
|
||
The model faithfully extracted meaning (correct under rules #26-30). The v0.17 MUST rule exists in the prompt (rule #6). But surviving permissive instructions (Additional Guidance) and restrictive gates (rules #4, #7) provide independent justification for choosing no mutation. This is not model failure — it is a contract-level instruction conflict.
|
||
|
||
## Smallest prompt boundary requiring correction:
|
||
|
||
**One line:** Additional Guidance bullet at line ~132 of prompt-builder.js:
|
||
> "If the answer does not justify a change, return empty arrays for every category."
|
||
|
||
This bullet must either be removed or modified to explicitly condition on rule #6 — i.e., only permit empty arrays when userSupportedMeaning does NOT contain consequential unresolved uncertainty (i.e., when rule #6 does not trigger). Without this fix, the MUST vs PERMIT conflict remains live.
|
||
|
||
**Second line:** Additional Guidance bullet:
|
||
> "Use answerMeaning to preserve the answer's direct meaning even when the graph change remains unresolved."
|
||
|
||
This must be modified or removed because it explicitly encourages semantic-only output in the exact scenario rule #6 mandates structural mutation.
|
||
|
||
These two bullets are ~10 words total. Removing or conditioning them is the minimal correction that resolves Pattern E (and cascades to weaken Patterns C and D).
|
||
|
||
## What this establishes:
|
||
|
||
1. The v0.17 contract fix (Option B from 57J.38) successfully converts the original silent accepted no-op into an explicitly rejected proposal with specific diagnostic. This confirms rule #6 exists in the prompt text and the validator fires on the structural fact.
|
||
2. Rule #6 alone is insufficient to produce compliant proposals because it conflicts with permissive instructions at equal hierarchy level. The model has multiple defensible paths to zero mutation.
|
||
3. The conflict is provider-agnostic — it exists at the instruction-contract level, not in any specific model's interpretation.
|
||
4. Test coverage for the v0.17 contract is limited to text presence, not semantic coherence of the full prompt.
|
||
|
||
## What this does NOT establish:
|
||
|
||
1. That fixing the identified bullet will restore faithful meaning → structural mutation. The remaining ambiguity (action-selection under rule #6's four paths) might still block some cases.
|
||
2. Whether adding decision criteria for action selection (update vs resolve vs add vs edge) would fully resolve the issue.
|
||
3. Whether the restrictive conditions in rules #4 and #7 should be relaxed rather than Additional Guidance being tightened.
|
||
4. How this interacts with other experiments (decomposition, answerability, provenance).
|
||
|
||
## Production code changed: NO
|
||
|
||
## Prompt changed: NO
|
||
|
||
## Tests changed: NO
|
||
|
||
## Ollama calls made: 0
|
||
|
||
## Dev server disturbed: NO
|
||
|
||
## Documentation updated: YES
|