robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

experiment: choose structural action-selection rule

Browse Source
This commit is contained in:
robbond
2026-08-11 14:01:42 +01:00
parent acd1928ebb
commit 96b855b8e7
2 changed files with 243 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/current-handoff.md
+38
View File
@@ -773,3 +773,41 @@ Configured Ollama: none used. Production code changed: prompt + tests only. Dev
**Classification: B — SAME FAITHFUL NO-OP.** The prompt contradiction was removed but faithful uncertainty alone still produced zero structural mutation for this model on this scenario. The action-selection gap under rule #6 (identified by 57J.42 as intentionally left unresolved) remains unaddressed.
Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No production code changed.
### Experiment 57J.45 — Choose Structural Action-Selection Rule
**Objective:** When rule #6 requires structural progress for a faithful unresolved uncertainty, choose the smallest provider-agnostic instruction that distinguishes update-of-existing from add-new-unknown.
**Pre-written expectation:** The current prompt (v0.18) has the ingredients to distinguish Case A (equivalent unknown exists → update/refine) from Case B (no equivalent → add new unknown), but lacks an explicit fallback sequence. Rule #6's four structural options are listed without decision criteria; rules #4, #7, and #11 collectively push toward "nothing justifies a change." Result: NO — the prompt does not distinguish Case A from Case B.
**Option A — EXISTING-FIRST FALLBACK:**
- Case A correct: YES (explicit first step checks existing unresolved nodes)
- Case B correct: YES ("otherwise" clause creates new unknown)
- Duplicate risk: LOW (validator safety net + deterministic order)
- Related-node overwrite risk: MEDIUM (semantic judgment required but bounded by existing contracts #4/#11)
- No-action risk: LOW (eliminates ambiguity that caused 57J.41/57J.44 failures)
- New semantic classifier: NO | New schema/state: NO | Validator change: NO | Provider-specific: NO
**Option B — NEW-UNKNOWN DEFAULT:**
- Case A correct: NO ("exact duplicate" requires undefined mechanism)
- Case B correct: YES (default-to-add works)
- Duplicate risk: HIGH (no "exact vs merely related" mechanism exists)
- Related-node overwrite risk: LOW (only creates, does not overwrite)
- No-action risk: MEDIUM (less than current but "exact duplicate" is underspecified)
- New semantic classifier: YES (required for "exact duplicate" definition) | New schema/state: NO | Validator change: YES | Provider-specific: NO
**Option C — GENERAL STRUCTURAL CHOICE:**
- Case A correct: PARTIAL (depends on model judgment across 4 unweighted options)
- Case B correct: PARTIAL (model may choose any option; evidence shows no-action preference)
- Duplicate risk: MEDIUM | Overwrite risk: MEDIUM | No-action risk: HIGH (essentially current state — confirmed failure in 57J.41/57J.44)
**Action-space narrowing for controlled case:**
- Edge-only sufficient: NO (edges connect nodes but cannot represent new uncertainty without a node)
- Resolution applicable: NO (user expresses uncertainty, not resolution of existing unknown)
- Relevant actions: update/refine (Case A) or add unknown (Case B)
**Recommended option: A — EXISTING-FIRST FALLBACK**
One explicit instruction-order rule in Additional Guidance: check existing unresolved nodes first; if none represents the same uncertainty, create a new unknown. Deterministic order replaces ambiguous choice-list. No new classifiers, schema state, or validator changes required. Existing rule #11 and deterministic validator remain as safety net.
**Status:** Design complete. READY FOR BOUNDED IMPLEMENTATION: YES. Implementation requires one Additional Guidance bullet in prompt-builder.js plus 8 focused deterministic regressions in tests. Full record in `docs/experiment-57j45.md`.
docs/experiment-57j45.md
+205
View File
@@ -0,0 +1,205 @@
# Experiment 57J.45 — Choose Structural Action-Selection Rule
**Branch:** `feature/semantic-to-mutation-contract-v0.18`
**Starting HEAD:** `acd1928` (experiment: validate conflict-free mutation prompt live)
**Experiment commit:** pending
## Objective
Answer exactly:
> When rule #6 requires structural progress for a faithful unresolved uncertainty, what is the smallest provider-agnostic instruction that tells the model when to update existing structure versus add a new unknown?
57J.44 established that the direct prompt contradiction is gone, but the model can still preserve meaning faithfully and produce no meaningful graph action. This task chooses the next bounded implementation without reopening the no-op validator.
## Context route (read-only)
- `docs/current-handoff.md` — current-project state
- `docs/experiment-57j44.md` — most recent live test result
- `lib/graph/prompt-builder.js` — complete graph-update rules
- `tests/graph/prompt-builder.test.js` — focused prompt tests
- Duplicate/semantic-match helper: existing rule #11 ("Do not add duplicate unknowns") and Additional Guidance line 125 ("prefer updatedNodes and resolvedUnknownNodeIds over creating duplicate nodes")
## Controlled case
```
The user is unsure whether the projected office savings from the relocation are realistic.
possibleInference = null
```
### Case A — equivalent uncertainty already exists
Graph contains an unresolved unknown materially representing whether projected relocation savings are realistic.
Desired: DO NOT ADD DUPLICATE; use/refine/update existing structure.
### Case B — no equivalent uncertainty exists
Graph contains general relocation/cost nodes but no unresolved node materially representing savings realism.
Desired: CREATE STRUCTURAL REPRESENTATION OF THE NEW UNCERTAINTY.
No edge required unless a genuine relationship is established by the answer.
## Existing contract check
### Current prompt content:
- **Genuinely new concepts:** Rule #4 — "Use addedNodes only for genuinely new concepts."
- **Duplicate unknowns:** Rule #11 — "Do not add duplicate unknowns."
- **Update/refine existing nodes:** Additional Guidance line 125 — "prefer updatedNodes and resolvedUnknownNodeIds over creating duplicate nodes."
- **New unresolved terms:** Rule #7 — "Add new unknown nodes only when the answer introduces a new decision, claim, object, measure, dependency, or unresolved term directly relevant to the case."
- **Rule #6 structural-progress rule (current):** "If answerMeaning.userSupportedMeaning contains consequential information or unresolved uncertainty that is not already represented in the graph, you MUST express its effect through structural mutation. This may be an update/refinement of existing structure, resolution of an existing unknown, a genuinely new unknown, or a justified relationship."
- **Rule #6 trigger condition:** "not already represented in the graph" — this is the ambiguous term that must be interpreted by the model to distinguish Case A from Case B.
### Does the prompt already contain enough information to distinguish Case A from Case B?
**NO** — necessary distinction is absent. The prompt requires the model to decide what "not already represented" means, but provides no instruction-order rule: should it check for an existing equivalent first (Case A path) or attempt a new node creation and catch duplicates at validation time (Case B path)? Rule #7's restrictive enumeration combined with rules #4 and #11 actually pushes the model toward "nothing justifies a change" when facing a simple uncertainty. The four structural options in rule #6 are listed without decision criteria or fallback ordering, confirming the ambiguity diagnosed in 57J.42 and reproduced in 57J.41/57J.44 live runs.
## Evaluate Option A — EXISTING-FIRST FALLBACK
### One explicit action-order rule:
```
When rule #6 applies:
1. If an existing unresolved node already represents the same uncertainty, update/refine that existing structure rather than adding a duplicate.
2. Otherwise add a new unknown that directly represents the unresolved uncertainty.
```
Do not require an edge solely for provenance.
#### Evaluation:
- **Case A correct:** YES — explicit first step is to check existing unresolved nodes
- **Case B correct:** YES — "otherwise" clause creates new unknown explicitly
- **Duplicate risk:** LOW — deterministic validator catches any miss; rule order prevents unnecessary duplication attempts
- **Risk of overwriting a merely-related existing node:** MEDIUM — the model must judge whether an existing node "already represents the same uncertainty." This is a semantic judgment, not a lexical match. However, this is exactly what rules #4 and #11 already require the model to do, so it's within the existing contract. The risk is bounded because (a) update/refine can add detail without overwriting, (b) new unknown with clear why-it-matters clause makes it traceable even if a partial overlap exists, (c) deterministic validator prevents true duplicates.
- **Risk of another no-action proposal:** LOW — eliminates the primary ambiguity that caused 57J.41/57J.44 failures. The instruction order is deterministic: check existing first, create second. No room for "nothing applies" escape because rule #6 still fires (unresolved uncertainty not yet in graph = case B).
- **Requires new semantic classifier:** NO — uses the model's existing ability to read the graph and compare semantics; deterministic validator remains safety net
- **Requires new graph/schema state:** NO
- **Requires validator change:** NO
## Evaluate Option B — NEW-UNKNOWN DEFAULT
```
When rule #6 applies to explicit unresolved uncertainty:
add a new unknown unless an exact duplicate already exists
```
Existing non-exact related nodes do not block new unknown creation.
#### Evaluation:
- **Case A correct:** NO — "exact duplicate" is stricter than what the current prompt allows. Rule #11 already says "Do not add duplicate unknowns" without defining "duplicate." Option B adds no mechanism to determine whether something is an "exact duplicate" versus "merely related." If the graph contains a partially-related uncertainty about savings (not exact), option B would create a second node — the same duplication problem this exercise seeks to prevent.
- **Case B correct:** YES — default-to-add works for genuinely new uncertainties
- **Duplicate risk:** HIGH — no mechanism distinguishes "exact duplicate" from "merely related"; current prompt has no deterministic duplicate definition beyond validator post-hoc detection
- **Risk of overwriting a merely-related existing node:** LOW — does not create nodes, so no overwrite occurs; only creates new nodes that may overlap
- **Risk of another no-action proposal:** MEDIUM — but less than current because it defaults to creation. However, the "exact duplicate" term is undefined and would need semantic matching logic
- **Requires new semantic classifier:** YES — "exact duplicate" requires a mechanism the current prompt does not provide
- **Requires new graph/schema state:** NO (but arguably needs one for the classification)
- **Requires validator change:** YES — must enforce the exact-duplicate vs merely-related distinction deterministically
## Evaluate Option C — GENERAL STRUCTURAL CHOICE
Keep all four existing structural options but add explanatory examples and leave the model to choose.
#### Evaluation:
- **Case A correct:** PARTIAL — depends on the model interpreting "update/refine" correctly for equivalent uncertainties. No instruction order given, so model must independently weigh four options
- **Case B correct:** PARTIAL — model may choose any of four options; evidence from 57J.41/57J.44 shows it chooses "no action" when the structural decision is ambiguous
- **Duplicate risk:** MEDIUM — without an explicit check-first step, duplication depends on model judgment across four unweighted options
- **Risk of overwriting a merely-related existing node:** MEDIUM — same as current prompt; no change
- **Risk of another no-action proposal:** HIGH — this is essentially the current state. 57J.41 and 57J.44 both failed under the four-option approach where no action was chosen
- **Requires new semantic classifier:** NO
- **Requires new graph/schema state:** NO
- **Requires validator change:** NO
## ACTION-SPACE CHECK
### Is `add relationship` a sensible standalone response to the controlled case?
**EDGE-ONLY SUFFICIENT: NO**
If no existing unknown node represents the savings-realism uncertainty, an edge alone cannot represent it. Edges connect nodes; they do not create representational capacity. A relationship from a state node to nothing new is empty — it has no target for the uncertainty. If there IS an equivalent unknown (Case A), then `add relationship` could be part of updating that structure, but by itself it does not represent the uncertainty.
### Is `resolve existing` applicable to the controlled case?
**RESOLUTION APPLICABLE: NO**
Resolution applies when the user's answer resolves a distinction previously encoded as an unresolved unknown. In the controlled case, the user expresses uncertainty ("I am unsure whether..."), not a resolution. There is nothing to resolve in Case B (no existing equivalent). In Case A, the user's uncertainty might inform refinement of an existing node but does not constitute resolution unless the answer explicitly states "X is definitely true/false" about that node's content.
### Effect on action space:
Two relevant actions remain for the controlled case:
1. **update/refine** (Case A path)
2. **add unknown** (Case B path)
Four nominal options narrowed to two by the controlled-case semantics.
## Recommendation
### CHOSEN: A — EXISTING-FIRST FALLBACK
#### Why:
Option A provides a deterministic instruction order that directly addresses the failure mode confirmed in 57J.41 and 57J.44. The problem was not missing semantic information but missing priority: when rule #6 fires, the model must first check whether an equivalent unresolved node exists before considering new structure creation. This is the smallest possible rule change — one explicit two-step sequence — that resolves the ambiguity without adding classifiers, schema state, or validator changes.
Option B fails because "exact duplicate" cannot be determined without a new semantic-matching mechanism (which contradicts the critical semantic boundary). Option C preserves the exact ambiguity that caused the failure.
#### Convergence:
The instruction order must be deterministic: check → act. Not options → choose. Not semantics → match. This rule preserves all existing contracts: duplicate detection still uses the deterministic validator as safety net; provider-agnostic design is maintained because the model's existing semantic access to the graph handles the "represents the same uncertainty" judgment that rules #4 and #11 already require.
### Does recommendation add deterministic semantic matching?
**NO** — the model's prompt-level semantic comparison of graph node content to answer semantics is within its existing capability (rules #4 and #11 already require this). Deterministic validator remains the post-hoc safety net for true duplicates.
### Does recommendation preserve provider-agnostic design?
**YES** — no provider-specific language, routing, or classification added.
### Does recommendation preserve duplicate protection?
**YES** — existing rule #11 and deterministic validator unchanged. The instruction order reduces (not eliminates) duplication attempts but does not weaken detection.
### Does recommendation require validator change?
**NO** — prompt-only change in Additional Guidance.
## Ready for bounded implementation: YES
### Exact prompt boundary:
One bullet added to Additional Guidance in `lib/graph/prompt-builder.js`, replacing or supplementing the existing guidance about preferring updates (line 125 area):
```text
When rule #6 applies: first check whether an existing unresolved node already represents the same uncertainty. If so, update/refine that existing structure rather than creating a duplicate. If no such node exists, add a new unknown that directly represents the unresolved uncertainty; do not create an edge alone to represent it.
```
### Required deterministic regressions:
1. equivalent existing unresolved unknown → prefer existing structure, no duplicate;
2. no equivalent unknown → explicit unresolved uncertainty must be represented as a new unknown;
3. merely related state/cost node does not count as representing the uncertainty itself;
4. answerMeaning alone remains insufficient;
5. possibleInference alone does not create unknowns;
6. resolution path remains available when the user actually resolves an existing unknown;
7. edge-only mutation is not presented as sufficient to represent a previously unrepresented uncertainty;
8. existing duplicate validator remains unchanged.
### Stop condition for implementation:
Prompt text and focused tests pass. No Ollama calls required for this step. Next bounded experiment (live) would follow in a separate task if the model produces Case A or Case B correctly on one fresh run each.
### What this intentionally leaves unresolved:
- The semantic threshold for "represents the same uncertainty" (relying on model capability + deterministic validator safety net);
- Live production validation of Option A (requires next experiment pass);
- Whether the instruction order needs tuning across multiple turns;
- Edge-connection strategy for new unknowns created under Case B.
## Documentation
This file: `docs/experiment-57j45.md`
Handoff update: appended to `docs/current-handoff.md`
Commit: `experiment: choose structural action-selection rule`
No merge.
---
*Read-only implementation-design task. No production code changed. No prompt changed. No tests changed. Zero Ollama calls. Dev server undisturbed.*