Feature/product platform foundation v0.62 #1
+59
-11
@@ -69,10 +69,11 @@ Experiment 39 resumed reasoning experiments with a passive validation of Behavio
|
||||
Experiment 40 diagnosed the root causes: Summarise and Pause fire their rules in real data but are always blocked by Acknowledge's priority-1 position (priority conflict, not assessor failure). Clarify's triggers never activate in tested scenarios due to the `too_broad` health condition being extremely narrow. All five behaviours confirmed independently reachable in synthetic isolation. No rules changed.
|
||||
|
||||
Experiment 41 compared two passive alternatives for reducing Acknowledge dominance:
|
||||
|
||||
- Variant A (priority reordering): evaluate Summarise/Pause before Acknowledge — introduces false-positive summarise in focusing phase
|
||||
- Variant B (Acknowledge exclusions): keep priority, gate Acknowledge when phase=concluding/synthesising or progress=stalled or health=user_overloaded — recommended
|
||||
- Both variants converge on the same two genuine changes: concluding→summarise and stalled→pause
|
||||
Experiment 42 implemented Variant B's narrow Acknowledge exclusion gate in the production selector (commit `05d3d96`). Summarise now appears at conclusion; Pause now appears when stalled. All other tested turns remain unchanged. Behaviour Selection remains passive and isolated with no runtime caller — active user-facing engine behaviour did not change.
|
||||
Experiment 42 implemented Variant B's narrow Acknowledge exclusion gate in the production selector (commit `05d3d96`). Summarise now appears at conclusion; Pause now appears when stalled. All other tested turns remain unchanged. Behaviour Selection remains passive and isolated with no runtime caller — active user-facing engine behaviour did not change.
|
||||
|
||||
Experiment 43 audited Clarify readiness across all 10 real assessment turns in existing fixtures. Zero turns produced Clarify-eligible states. Two findings: (1) the orienting-based Clarify rule is dead code because the assessor never produces phase=orienting, and (2) the too_broad trigger requires conditions no fixture exercises. Branch: `feature/user-workspace-ux-v0.7`.
|
||||
|
||||
@@ -151,12 +152,12 @@ Provisional criteria for review (all confirmed met by Experiment 38 cold-start t
|
||||
|
||||
## 7. First Files by Work Type
|
||||
|
||||
| Work type | Start with |
|
||||
|---|---|
|
||||
| Engine experiment | Engine Experiment pack |
|
||||
| UI or mock work | UI and Mock pack |
|
||||
| Work type | Start with |
|
||||
| ------------------------------- | ----------------------------- |
|
||||
| Engine experiment | Engine Experiment pack |
|
||||
| UI or mock work | UI and Mock pack |
|
||||
| Architecture or contract review | Architecture or Contract pack |
|
||||
| Knowledge management | Knowledge-Management pack |
|
||||
| Knowledge management | Knowledge-Management pack |
|
||||
|
||||
## 8. Resume Check
|
||||
|
||||
@@ -170,7 +171,7 @@ Answer before continuing:
|
||||
|
||||
---
|
||||
|
||||
*Created by Experiment 34. Updated by Experiments 38–53, 54A–54Z, 55A–55F, 56D–56H, 56L–56M, v0.8 closeout. Branch: `feature/reasoning-fidelity-v0.8`. First-pass reasoning-fidelity v0.8 complete to A–F scope.*
|
||||
_Created by Experiment 34. Updated by Experiments 38–53, 54A–54Z, 55A–55F, 56D–56H, 56L–56M, v0.8 closeout. Branch: `feature/reasoning-fidelity-v0.8`. First-pass reasoning-fidelity v0.8 complete to A–F scope._
|
||||
|
||||
---
|
||||
|
||||
@@ -226,7 +227,7 @@ The first implementation pass added proposal-level `answerMeaning` with a pre-mu
|
||||
|
||||
### Experiment 56B Summary — Regression B Live Run After Normalisation
|
||||
|
||||
Commit `36faf70` added normalization for `conditional_qualification → conditional_tradeoff`, but a live Regression B run returned a *different* variant: `supportCategory: "conditional_preference"`. The existing normalisation map does not cover this value. Two independent Zod errors occurred: (1) `conditional_preference` not in the supportCategory enum, and (2) `resolutionGuidance` was free-text instead of an enum value. **Run-to-run model variation confirmed** — the same fixed input produced `conditional_qualification` in Ex 56A and `conditional_preference` in Ex 56B. The pre-mutation guard remains unreachable because proposal_validation rejects first. Failure classification: `FAIL — normalization / proposal contract`. Branch: `feature/reasoning-fidelity-v0.8`. File to inspect when resuming: `docs/experiment-56b.md`. Status pending Rob's review.
|
||||
Commit `36faf70` added normalization for `conditional_qualification → conditional_tradeoff`, but a live Regression B run returned a _different_ variant: `supportCategory: "conditional_preference"`. The existing normalisation map does not cover this value. Two independent Zod errors occurred: (1) `conditional_preference` not in the supportCategory enum, and (2) `resolutionGuidance` was free-text instead of an enum value. **Run-to-run model variation confirmed** — the same fixed input produced `conditional_qualification` in Ex 56A and `conditional_preference` in Ex 56B. The pre-mutation guard remains unreachable because proposal_validation rejects first. Failure classification: `FAIL — normalization / proposal contract`. Branch: `feature/reasoning-fidelity-v0.8`. File to inspect when resuming: `docs/experiment-56b.md`. Status pending Rob's review.
|
||||
|
||||
### Experiment 56D Summary — Regression B via Real Production Path
|
||||
|
||||
@@ -392,7 +393,7 @@ Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No producti
|
||||
|
||||
### Experiment 57J.32 — Inspect Rejected Proposal Live Variance (EXPERIMENT INTERRUPTED)
|
||||
|
||||
**Protocol breach: YES.** The execution harness drifted beyond the agreed budget. Only valid bounded evidence from the first 3 trials is retained in `docs/experiment-57j32.md`. All post-trial activity is excluded from conclusions. **Classification: F — MULTIPLE DIFFERENCES.** Using `rejectedProposalSnapshot` directly, accepted vs rejected proposals differ in both answerMeaning framing (prescriptive "decision is conditional on" vs neutral "requires evidence") AND structural content (accepted = 2 addedNodes + 2 addedEdges; rejected = 0 addedNodes + 0 addedEdges). Neither single cause can be independently isolated. The rejected trial's userSupportedMeaning did not faithfully preserve the raw answer: "The decision is conditional on" prescribes what the *decision* requires rather than reporting the user's information need ("Before deciding, I need..."). **rejectedProposalSnapshot worked as designed** (57J.31 verified). Status: interrupted mid-execution; evidence retained in `docs/experiment-57j32.md`. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. Production code unchanged.
|
||||
**Protocol breach: YES.** The execution harness drifted beyond the agreed budget. Only valid bounded evidence from the first 3 trials is retained in `docs/experiment-57j32.md`. All post-trial activity is excluded from conclusions. **Classification: F — MULTIPLE DIFFERENCES.** Using `rejectedProposalSnapshot` directly, accepted vs rejected proposals differ in both answerMeaning framing (prescriptive "decision is conditional on" vs neutral "requires evidence") AND structural content (accepted = 2 addedNodes + 2 addedEdges; rejected = 0 addedNodes + 0 addedEdges). Neither single cause can be independently isolated. The rejected trial's userSupportedMeaning did not faithfully preserve the raw answer: "The decision is conditional on" prescribes what the _decision_ requires rather than reporting the user's information need ("Before deciding, I need..."). **rejectedProposalSnapshot worked as designed** (57J.31 verified). Status: interrupted mid-execution; evidence retained in `docs/experiment-57j32.md`. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. Production code unchanged.
|
||||
|
||||
### Experiment 57J.30 — Proposal-Boundary Live Variance
|
||||
|
||||
@@ -427,6 +428,7 @@ Configured Ollama: none used. Production code changed: NO. Tests permanently cha
|
||||
**Approach:** Bounded execution configuration (`maxUpdates`) + explicit call accounting (startCalls/updateCalls/totalCalls counters reflecting actual API invocations) + rejection-immediate-stop semantics + rejectedProposalSnapshot preservation for v0.16 diagnostic visibility.
|
||||
|
||||
**Changes to canonical harness (`scripts/reproduce-multi-turn-investigation.mjs`):**
|
||||
|
||||
- Hardcoded `Start → Update 1 → Update 2` sequence replaced with configurable bounded loop (`config.maxUpdates`)
|
||||
- Call accounting added: `calls.startCalls`, `calls.updateCalls`, reported as `totalCalls`
|
||||
- Rejection returns immediately; no retry path exists for any semantic outcome (proposal_compatibility, validation failure, etc.)
|
||||
@@ -434,6 +436,7 @@ Configured Ollama: none used. Production code changed: NO. Tests permanently cha
|
||||
- Every update call is explicit in the loop; `config.answers[i]` maps positionally to `Update i+1`
|
||||
|
||||
**Tests added (`tests/reproduce-multi-turn-investigation.harness.test.js`):** 8 deterministic cases via synchronous simulation mirror of harness logic — all pass (0 Ollama calls, no dev-server needed). Test cases:
|
||||
|
||||
1. Start success → exactly 1 Start call.
|
||||
2. Start failure → exactly 1 Start call, no retry.
|
||||
3. Update success → exactly 1 Update call.
|
||||
@@ -502,6 +505,7 @@ Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. 2 live call
|
||||
**Method:** Analyzed prompt instructions (`lib/graph/prompt-builder.js`), schema defaults (`lib/graph/schema.js`), validator logic (`lib/graph/utils.js` line 868–885), application pipeline (`lib/graph/apply-proposal.js` line 3174, 3252–3270), and existing test coverage. No Ollama calls. No live API.
|
||||
|
||||
**Findings:**
|
||||
|
||||
- **Prompt contract is AMBIGUOUS:** Rule #6 requires inspecting for new uncertainty but rule #7 ("Add new unknown nodes only when...") is a restriction, not a requirement. Additional Guidance explicitly permits semantic-only proposals via `answerMeaning`.
|
||||
- **Schema contract PERMITS the combination:** `graphUpdateSchema` allows populated `answerMeaning` + zero structural mutation (all array fields default to `[]`). No cross-field constraint exists.
|
||||
- **Validator contract REJECTS it:** `hasMeaningfulChange` checks only structural fields (addedNodes, updatedNodes status/value changes, addedEdges, removedEdgeIds). `answerMeaning` is not considered meaningful change.
|
||||
@@ -513,12 +517,12 @@ Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. 2 live call
|
||||
|
||||
Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. Production code changed: NO. Prompt changed: NO. Tests changed: NO. Dev server disturbed: NO. Ollama calls: 0.
|
||||
|
||||
|
||||
### Experiment 57J.38 — Semantic-to-Mutation Contract Fix Selection
|
||||
|
||||
**Objective:** Choose the smallest safe contract change preventing faithful semantic-only no-op proposals (populated `answerMeaning` with consequential uncertainty + zero structural mutation).
|
||||
|
||||
**Three options evaluated:**
|
||||
|
||||
- **A (prompt-only):** Single MUST rule in Additional Guidance clarifying structural mutation requirement. Fixes ambiguity but relies entirely on model compliance.
|
||||
- **B (prompt + validator):** Same prompt rule PLUS a deterministic validator check targeting `userSupportedMeaning` text + empty structural fields. Produces specific actionable error.
|
||||
- **C (schema cross-field constraint):** Zod `.refine()` rejecting any populated `answerMeaning` object with zero structural mutation.
|
||||
@@ -526,6 +530,7 @@ Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. Production
|
||||
**Key finding on Option C:** Breaks Case 5 (possibleInference only). Schema cannot distinguish "new consequential meaning" from "inference-only" without semantic analysis. Any populated answerMeaning object triggers rejection regardless of content type.
|
||||
|
||||
**Controlled case results summary:**
|
||||
|
||||
- Case 1 (genuinely new uncertainty): All options reject as expected. B has best diagnostic visibility.
|
||||
- Case 2 (already represented): All options correctly reject.
|
||||
- Case 3 (update/resolve existing): All options correctly allow structural update to existing node.
|
||||
@@ -540,6 +545,7 @@ Configured Ollama: none used. Production code changed: NO. Prompt changed: NO. T
|
||||
**READY FOR BOUNDED IMPLEMENTATION: YES**
|
||||
|
||||
Exact implementation boundary:
|
||||
|
||||
1. One MUST rule in prompt Additional Guidance (replaces line 132 of prompt-builder.js)
|
||||
2. One deterministic check in `validateGraphUpdate()` after `hasMeaningfulChange` (utils.js)
|
||||
3. Six regression tests: populated-meaning-zero-mutation rejection, prompt text verification, null-answerMeaning preserved, possibleInference-only not forced, update-existing-node valid, resolve-path valid
|
||||
@@ -553,6 +559,7 @@ Exact implementation boundary:
|
||||
Implemented the bounded Option B from 57J.38 with ownership correction: prompt owns structural materialization obligation; validator owns only the structural fact that `answerMeaning` alone is not graph progress.
|
||||
|
||||
**Production changes:**
|
||||
|
||||
- **prompt-builder.js** rule #6 replaced: added explicit MUST structural-materialization rule. userSupportedMeaning must be expressed through graph structure — update/refine existing, resolve unknown, new unknown, or justified relationship. answerMeaning alone is insufficient.
|
||||
- **utils.js** validateGraphUpdate(): added specific diagnostic "answerMeaning.userSupportedMeaning is populated, but the proposal contains no graph mutation" inside the `!hasMeaningfulChange` path, ordered before generic no-op error for exact same proposal.
|
||||
|
||||
@@ -577,7 +584,7 @@ Ollama calls: 0. Dev server disturbed: NO. Branch: feature/semantic-to-mutation-
|
||||
|
||||
**UPDATE 1:** HTTP 422 | stage: proposal_compatibility | error: "answerMeaning.userSupportedMeaning introduces a stronger reasoning category than the raw answer establishes."
|
||||
|
||||
Rejected proposal snapshot: `userSupportedMeaning`: "Proceeding with the relocation decision is explicitly conditional on obtaining verified evidence that projected office savings are realistic and that key engineer retention is preserved." — this converts the user's information-need statement ("I need evidence that...") into prescriptive conditional language about what the *decision* requires, introducing a `conditional_qualification` meaning stronger than the raw answer. `possibleInference`: null. Structural proposal: 1 addedNode (`n_prereq_constraint`, kind=assumption) + 1 addedEdge (depends_on to state node).
|
||||
Rejected proposal snapshot: `userSupportedMeaning`: "Proceeding with the relocation decision is explicitly conditional on obtaining verified evidence that projected office savings are realistic and that key engineer retention is preserved." — this converts the user's information-need statement ("I need evidence that...") into prescriptive conditional language about what the _decision_ requires, introducing a `conditional_qualification` meaning stronger than the raw answer. `possibleInference`: null. Structural proposal: 1 addedNode (`n_prereq_constraint`, kind=assumption) + 1 addedEdge (depends_on to state node).
|
||||
|
||||
**Meaning fidelity classification: STRENGTHENED.** The model transformed a neutral information-need ("Before deciding, I need evidence...") into prescriptive constraint language ("Proceeding...is explicitly conditional on obtaining verified evidence"). This introduces a stronger reasoning category not supported by the raw answer.
|
||||
|
||||
@@ -599,6 +606,7 @@ Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. 2 live call
|
||||
**Starting HEAD:** 712c0c4 docs: experiment 57J.39 record and handoff update
|
||||
|
||||
**Pre-written expectation:**
|
||||
|
||||
> The raw answer contains two explicit unresolved evidence needs: projected savings realism and key-engineer retention impact. If v0.17 closes the semantic-to-mutation contract gap, a faithful `userSupportedMeaning` should no longer be accompanied by a completely empty structural proposal. The model should either update/refine existing relevant graph structure, resolve relevant structure, or add justified new structure.
|
||||
> A semantic-strengthening rejection remains a valid protected outcome and does not count as failure of v0.17.
|
||||
|
||||
@@ -658,11 +666,13 @@ Mutation type: ADD NODE
|
||||
This run did not test the v0.17 contract's core question because the model never produced a faithful `userSupportedMeaning` to begin with. The strengthening occurred before reaching the mutation boundary, so v0.17's MUST rule was never exercised in this run. A faithful semantic-only no-op (if it exists under v0.17) is neither reproduced nor disproved here.
|
||||
|
||||
**What this clean run establishes:**
|
||||
|
||||
1. The configured model maps "Before deciding, I need evidence..." to prescriptive conditional framing on this scenario — a repeatable strengthening pattern observed in earlier experiments (57J.32, 57J.33).
|
||||
2. The existing semantic-fidelity guard catches this class of strengthening at proposal_compatibility.
|
||||
3. When meaning is strengthened and rejected, the model's structural proposal is built on the strengthened meaning (not a faithful translation), making mutation analysis irrelevant to v0.17.
|
||||
|
||||
**What it does NOT prove:**
|
||||
|
||||
1. That the configured model produces faithful `userSupportedMeaning` on this scenario under repeated runs.
|
||||
2. That v0.17's MUST rule prevents the faithful semantic-only no-op when meaning is genuinely preserved.
|
||||
3. That strengthening avoidance would occur with different phrasing, domain, or model.
|
||||
@@ -692,6 +702,7 @@ This run did not test the v0.17 contract's core question because the model never
|
||||
**UPDATE 1:** HTTP 422 | stage: proposal_compatibility | First error: "answerMeaning.userSupportedMeaning is populated, but the proposal contains no graph mutation. answerMeaning alone does not constitute graph progress."
|
||||
|
||||
**Rejected Proposal Snapshot:**
|
||||
|
||||
- `userSupportedMeaning`: "The user is unsure whether the projected office savings from the relocation are realistic." — **FAITHFUL** (preserves uncertainty, no conditional/constraint language)
|
||||
- `possibleInference`: null
|
||||
- All structural fields: empty (zero updatedNodes, zero resolvedUnknownNodeIds, zero addedNodes, zero addedEdges)
|
||||
@@ -701,6 +712,7 @@ This run did not test the v0.17 contract's core question because the model never
|
||||
**Key finding:** v0.17 successfully blocks the original failure (accepted semantic-only no-op) but does not establish a positive path for faithful meaning to produce graph progress. The open question remains: is there any valid pathway where faithful meaning translates to structural mutation, or does the MUST rule universally block it?
|
||||
|
||||
Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No production code changed. Harness restored. No-retry preserved. Dev server disturbed: NO.
|
||||
|
||||
### Experiment 57J.42 — Structural-Mutation MUST Rule Prompt Conflict Diagnosis
|
||||
|
||||
**Objective:** Diagnose why the model produces faithful `userSupportedMeaning` with zero structural mutation despite the v0.17 MUST rule. Read-only prompt-contract analysis of assembled prompt + tests. No Ollama calls.
|
||||
@@ -708,6 +720,7 @@ Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No producti
|
||||
**Method:** Inspected all prompt rules from `lib/graph/prompt-builder.js`, analyzed action-selection ambiguity, checked six conflict patterns (A–F), reviewed test adequacy in `tests/graph/prompt-builder.test.js`.
|
||||
|
||||
**Relevant rule conflicts found:**
|
||||
|
||||
- Rule #6 (MUST) vs Additional Guidance bullet B (PERMIT empty arrays if "answer does not justify a change"): direct MUST vs PERMIT conflict. Rule #7's restrictive enumeration + rules #4 ("genuinely new concepts") + #11 (no duplicates) help the model decide nothing justifies a change, triggering the escape hatch in Additional Guidance.
|
||||
- Additional Guidance bullet C: "Use answerMeaning to preserve the answer's direct meaning even when the graph change remains unresolved" explicitly encourages semantic-only output — the exact pattern v0.17 sought to eliminate.
|
||||
- Action-selection ambiguity under rule #6: four acceptable forms listed without decision criteria or fallback ordering.
|
||||
@@ -727,6 +740,7 @@ Configured Ollama: none used. Production code changed: NO. Prompt changed: NO. T
|
||||
**Objective:** Apply the smallest proven correction from 57J.42: replace two Additional Guidance bullets that conflicted with rule #6's MUST rule, plus add one clarifying bullet and seven focused tests. Bounded prompt-contract task only. Not solving update-vs-add action selection.
|
||||
|
||||
**Production changes:**
|
||||
|
||||
- `lib/graph/prompt-builder.js` — Replaced two conflicting Additional Guidance bullets (line ~131-132):
|
||||
- Old: "If the answer does not justify a change, return empty arrays for every category." → Now conditioned on rule #6 not applying.
|
||||
- Old: "Use answerMeaning to preserve the answer's direct meaning even when the graph change remains unresolved." → Replaced with explicit semantic-fidelity vs structural-mutation separation.
|
||||
@@ -734,15 +748,18 @@ Configured Ollama: none used. Production code changed: NO. Prompt changed: NO. T
|
||||
- `tests/graph/prompt-builder.test.js` — Added 7 focused tests covering all seven required coverage areas.
|
||||
|
||||
**What this guarantees:**
|
||||
|
||||
1. Empty-array permission is explicitly subordinate to rule #6 — MUST vs PERMIT contradiction eliminated.
|
||||
2. `answerMeaning` can no longer substitute for graph mutation per the corrected bullet text.
|
||||
3. All existing contracts preserved: duplicate avoidance, genuinely-new-concepts protection, fidelity/inference separation, traceability, update/refine preference.
|
||||
|
||||
**What is intentionally left unresolved:**
|
||||
|
||||
1. Action selection under rule #6 (update vs add) — confirmed ambiguous by 57J.42.
|
||||
2. Live production validation — requires next experiment pass.
|
||||
|
||||
**Test results:**
|
||||
|
||||
- prompt-builder.test.js: 22/22 pass (7 new + 15 pre-existing)
|
||||
- utils.test.js: 68/68 pass
|
||||
- apply-proposal.test.js: 64/64 pass
|
||||
@@ -764,6 +781,7 @@ Configured Ollama: none used. Production code changed: prompt + tests only. Dev
|
||||
**START:** HTTP 200 | stage: unknown | nodes: 6 | edges: 3 | question: "What would clarify current operating costs for the present location versus projected post-relocation costs and one-time relocation expenses in this situation?"
|
||||
|
||||
**UPDATE 1:** HTTP 422 | stage: proposal_compatibility | error: "answerMeaning.userSupportedMeaning is populated, but the proposal contains no graph mutation. answerMeaning alone does not constitute graph progress."
|
||||
|
||||
- Nodes: 6 | Edges: 3 (unchanged)
|
||||
|
||||
**Rejected Proposal Snapshot:** `userSupportedMeaning`: "The user is unsure whether the projected office savings from the relocation are realistic." — FAITHFUL (no strengthening). `possibleInference`: about inflated projections negating benefits. All structural mutation fields empty (zero addedNodes, zero addedEdges, zero resolvedUnknownNodeIds, zero updatedNodes with meaningful change).
|
||||
@@ -781,6 +799,7 @@ Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No producti
|
||||
**Pre-written expectation:** The current prompt (v0.18) has the ingredients to distinguish Case A (equivalent unknown exists → update/refine) from Case B (no equivalent → add new unknown), but lacks an explicit fallback sequence. Rule #6's four structural options are listed without decision criteria; rules #4, #7, and #11 collectively push toward "nothing justifies a change." Result: NO — the prompt does not distinguish Case A from Case B.
|
||||
|
||||
**Option A — EXISTING-FIRST FALLBACK:**
|
||||
|
||||
- Case A correct: YES (explicit first step checks existing unresolved nodes)
|
||||
- Case B correct: YES ("otherwise" clause creates new unknown)
|
||||
- Duplicate risk: LOW (validator safety net + deterministic order)
|
||||
@@ -789,6 +808,7 @@ Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No producti
|
||||
- New semantic classifier: NO | New schema/state: NO | Validator change: NO | Provider-specific: NO
|
||||
|
||||
**Option B — NEW-UNKNOWN DEFAULT:**
|
||||
|
||||
- Case A correct: NO ("exact duplicate" requires undefined mechanism)
|
||||
- Case B correct: YES (default-to-add works)
|
||||
- Duplicate risk: HIGH (no "exact vs merely related" mechanism exists)
|
||||
@@ -797,11 +817,13 @@ Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No producti
|
||||
- New semantic classifier: YES (required for "exact duplicate" definition) | New schema/state: NO | Validator change: YES | Provider-specific: NO
|
||||
|
||||
**Option C — GENERAL STRUCTURAL CHOICE:**
|
||||
|
||||
- Case A correct: PARTIAL (depends on model judgment across 4 unweighted options)
|
||||
- Case B correct: PARTIAL (model may choose any option; evidence shows no-action preference)
|
||||
- Duplicate risk: MEDIUM | Overwrite risk: MEDIUM | No-action risk: HIGH (essentially current state — confirmed failure in 57J.41/57J.44)
|
||||
|
||||
**Action-space narrowing for controlled case:**
|
||||
|
||||
- Edge-only sufficient: NO (edges connect nodes but cannot represent new uncertainty without a node)
|
||||
- Resolution applicable: NO (user expresses uncertainty, not resolution of existing unknown)
|
||||
- Relevant actions: update/refine (Case A) or add unknown (Case B)
|
||||
@@ -818,10 +840,12 @@ One explicit instruction-order rule in Additional Guidance: check existing unres
|
||||
**Starting HEAD:** `acd1928` (HEAD of v0.18 at branch point)
|
||||
|
||||
**What changed:**
|
||||
|
||||
- `lib/graph/prompt-builder.js` — Added one Additional Guidance bullet: "When rule #6 applies to explicitly unresolved uncertainty: first check whether an existing unresolved node already represents the same uncertainty; if so, update/refine that existing structure rather than adding a duplicate; if no such node exists, add a new unknown that directly represents the unresolved uncertainty; do not use an edge alone to represent a previously unrepresented uncertainty."
|
||||
- `tests/graph/prompt-builder.test.js` — Added 14 focused tests verifying: existing-first ordering, reuse path, fallback-to-add, related-node-insufficient, edge-only-prohibited, possibleInference separation, resolution path preserved, duplicate contract preserved, scope uncertainty-only, fidelity/traceability preserved, noop validator untouched, no semantic classifier added.
|
||||
|
||||
**Test results:**
|
||||
|
||||
- prompt-builder.test.js: 37 tests pass (23 existing + 14 new)
|
||||
- utils.test.js: 68 tests pass (regression confirmation)
|
||||
- Total: 105 tests pass, 0 failures
|
||||
@@ -846,6 +870,7 @@ Configured Ollama: none used. Production code changed: prompt + tests only. Dev
|
||||
**UPDATE 1:** HTTP 422 | stage: proposal_compatibility | error: "answerMeaning.userSupportedMeaning overstates a raw answer that remains uncertain."
|
||||
|
||||
rejectedProposalSnapshot:
|
||||
|
||||
- userSupportedMeaning: "The user is currently uncertain whether the projected office savings from the relocation are realistic." (HUMAN ASSESSMENT: minor non-material paraphrase — see 57J.48; VALIDATOR: lexically rejected because "uncertain" absent from deriveAnswerMeaningProfile detection patterns)
|
||||
- possibleInference: null
|
||||
- addedNodes: [{id: "ns82kx9", kind: unknown, label: "Projected office savings from the relocation lack validation"}] (genuinely new — ADD NEW UNKNOWN structural action)
|
||||
@@ -867,10 +892,12 @@ Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. 2 live call
|
||||
**Objective:** Determine whether the captured rejection of "The user is currently uncertain..." → rejected as overstatement when raw answer was "I am unsure..." is semantically justified or a lexical false positive. Fully deterministic — zero Ollama calls.
|
||||
|
||||
**Captured pair:**
|
||||
|
||||
- Raw answer: `"I am unsure whether the projected office savings from the relocation are realistic."`
|
||||
- userSupportedMeaning: `"The user is currently uncertain whether the projected office savings from the relocation are realistic."`
|
||||
|
||||
**Deterministic trace:**
|
||||
|
||||
1. `deriveAnswerMeaningProfile(raw)` → category=`uncertain` (matches pattern `"unsure"` at line 2880)
|
||||
2. `deriveAnswerMeaningProfile(userSupportedMeaning)` → category=`other` (pattern gap: `"uncertain"` not detected)
|
||||
3. `validateAnswerMeaningCompatibilityWithRawAnswer`: line 2966-2970 fires because `raw=uncertain` && `supported≠uncertain` → error: "overstates a raw answer that remains uncertain."
|
||||
@@ -878,6 +905,7 @@ Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. 2 live call
|
||||
**Inverse test (same semantics, different word):** Using `"unsure"` instead of `"uncertain"` in userSupportedMeaning → category=`uncertain`, errors=none. Both words express identical uncertainty.
|
||||
|
||||
**Semantic equivalence check:**
|
||||
|
||||
- Uncertainty preserved: YES
|
||||
- Decision condition added: NO
|
||||
- Hard constraint added: NO
|
||||
@@ -934,6 +962,7 @@ Read-only design decision selecting the smallest safe migration path from keywor
|
||||
**Decision rationale:** Option C eliminates lexical dictionary dependence for all protected categories where structured fields are populated, while preserving backwards compatibility through null-to-lexical fallback for legacy proposals. It retains model-trust safeguards via cross-field deterministic consistency checks (resolutionGuidance vs proposal state, enum validity) without requiring the validator to re-infer English semantics from raw text or invent new classifiers.
|
||||
|
||||
**Exact next implementation boundary:** One branch — `feature/structured-semantic-fidelity-v0.20`. Four file modifications:
|
||||
|
||||
1. `lib/graph/prompt-builder.js` — rule 28 replacement (enforce + instruct) + output contract addition (enum value listing for the two fields);
|
||||
2. `lib/graph/schema.js` — `z.enum(Object.values(answerSupportCategory))` and `z.enum(Object.values(answerResolutionGuidance))` constraints on the two zod fields;
|
||||
3. `lib/graph/apply-proposal.js` — migrate `validateAnswerMeaningAlignment()` to read structured values first, lexical fallback for null legacy proposals only; add schema-invariant consistency checks;
|
||||
@@ -942,3 +971,22 @@ Read-only design decision selecting the smallest safe migration path from keywor
|
||||
No new taxonomy. No provider-specific code. No embeddings. No synonym lists. No new LLM call pattern.
|
||||
|
||||
Full results in `docs/experiment-57j50.md`.
|
||||
|
||||
### Experiment 57J.51 — Structured Semantic Fidelity Implementation
|
||||
|
||||
**Classification: E — IMPLEMENTATION COMPLETE**
|
||||
|
||||
Implemented the bounded Option C from 57J.50 on branch `feature/structured-semantic-fidelity-v0.20`.
|
||||
|
||||
**Production changes:**
|
||||
|
||||
- `lib/graph/schema.js` now enum-constrains `answerMeaning.supportCategory` and `answerMeaning.resolutionGuidance` while preserving nullability.
|
||||
- `lib/graph/prompt-builder.js` now instructs the model to populate the structured semantic fields using the existing allowed values and exposes those exact values in the prompt contract.
|
||||
- `lib/graph/apply-proposal.js` now treats populated structured semantic fields as the primary fidelity authority, skips lexical category verification on that populated path, and retains lexical derivation only as legacy fallback when structured fields are null.
|
||||
- Structured consistency now deterministically rejects `resolutionGuidance = must_remain_unresolved` when the proposal resolves an unknown.
|
||||
|
||||
**Captured false positive resolved:** The exact `unsure` raw answer + structured `supportCategory=uncertain` case no longer fails on the populated structured path. No synonym rule was added.
|
||||
|
||||
**Deferred intentionally:** `must_resolve` target-specific enforcement remains deferred because the current proposal structure does not safely identify the answered/targeted unknown in every case without inventing new linkage.
|
||||
|
||||
**Tests:** Required deterministic suites pass, plus one directly related proposal-parse suite updated for the new schema boundary. No live model calls. No Ollama. Full record in `docs/experiment-57j51.md`.
|
||||
|
||||
@@ -0,0 +1,145 @@
|
||||
# Experiment 57J.51 — Structured Semantic Fidelity Implementation
|
||||
|
||||
**Branch:** `feature/structured-semantic-fidelity-v0.20`
|
||||
**Starting HEAD:** `b6a232ff6f56b5f1af49d94bb2881190b5bf8345`
|
||||
**Production commit:** `7d06cd3c473cee64c2c371c1e1af1c466cdc32dd`
|
||||
|
||||
## Objective
|
||||
|
||||
Implement Option C from Experiment 57J.50:
|
||||
|
||||
> Use existing structured semantic fields (`supportCategory`, `resolutionGuidance`) as the primary fidelity contract when populated, enforce their allowed enum values, validate only structured cross-field consistency, and retain current lexical derivation only as a temporary fallback when those fields are null.
|
||||
|
||||
## Scope Implemented
|
||||
|
||||
### 1. Schema
|
||||
|
||||
`lib/graph/schema.js`
|
||||
|
||||
- Constrained `answerMeaning.supportCategory` to `z.enum(Object.values(answerSupportCategory)).nullable().optional()`;
|
||||
- Constrained `answerMeaning.resolutionGuidance` to `z.enum(Object.values(answerResolutionGuidance)).nullable().optional()`;
|
||||
- Preserved transitional nullability on both fields;
|
||||
- Reused existing enum constants — no new taxonomy added.
|
||||
|
||||
### 2. Prompt
|
||||
|
||||
`lib/graph/prompt-builder.js`
|
||||
|
||||
- Exposed allowed values for both structured semantic fields in the output contract;
|
||||
- Replaced the old “optional descriptive hints only” instruction with structured population guidance;
|
||||
- Instructed the model to:
|
||||
- populate `supportCategory` whenever the answer fits an existing category,
|
||||
- use `other` when none of the protected categories applies,
|
||||
- avoid leaving `supportCategory` null merely because wording is uncertain,
|
||||
- populate `resolutionGuidance` when one of the existing resolution states genuinely applies,
|
||||
- keep `resolutionGuidance` null only when no existing state actually applies;
|
||||
- Used the existing `formatEnumValues()` helper;
|
||||
- Added no provider-specific wording.
|
||||
|
||||
### 3. Validator — structured first
|
||||
|
||||
`lib/graph/apply-proposal.js`
|
||||
|
||||
- Added `getAnswerMeaningProfile(answerMeaning)` to unify:
|
||||
- structured `supportCategory` / `resolutionGuidance` when populated,
|
||||
- lexical derivation only when those structured fields are null;
|
||||
- Updated `validateAnswerMeaningCompatibilityWithRawAnswer()` so populated structured semantic fields bypass raw-text lexical category verification entirely;
|
||||
- Updated `validateAnswerMeaningAlignment()` so:
|
||||
- structured fields are authoritative when populated,
|
||||
- lexical fallback remains active only for legacy null cases.
|
||||
|
||||
### 4. Non-lexical consistency
|
||||
|
||||
Implemented one deterministic structured consistency check now:
|
||||
|
||||
- `resolutionGuidance = must_remain_unresolved` + proposal resolves an unknown → reject with:
|
||||
- `Proposal resolves an unknown even though answerMeaning.resolutionGuidance is must_remain_unresolved.`
|
||||
|
||||
Deferred one check intentionally:
|
||||
|
||||
- `must_resolve` target-specific enforcement was **deferred** because the current proposal structure does not safely identify the answered/targeted unknown in every valid case without inventing new linkage.
|
||||
|
||||
### 5. possibleInference
|
||||
|
||||
- Preserved current behaviour: `possibleInference` remains non-authoritative;
|
||||
- It does not independently justify mutation;
|
||||
- No validator path was added that treats it as authoritative structure.
|
||||
|
||||
## Captured False Positive
|
||||
|
||||
The exact `unsure` → `uncertain` populated structured-path false positive is now removed.
|
||||
|
||||
### Captured case
|
||||
|
||||
```text
|
||||
raw answer:
|
||||
I am unsure whether the projected office savings from the relocation are realistic.
|
||||
|
||||
userSupportedMeaning:
|
||||
The user is currently uncertain whether the projected office savings from the relocation are realistic.
|
||||
|
||||
supportCategory:
|
||||
uncertain
|
||||
|
||||
resolutionGuidance:
|
||||
must_remain_unresolved
|
||||
```
|
||||
|
||||
### Outcome
|
||||
|
||||
- **Passes** on the populated structured path;
|
||||
- Does **not** depend on synonym logic;
|
||||
- `unsure` vs `uncertain` wording is irrelevant when structured category is present.
|
||||
|
||||
## Tests Added / Updated
|
||||
|
||||
Focused deterministic coverage added or updated in:
|
||||
|
||||
- `tests/graph/schema.test.js`
|
||||
- `tests/graph/prompt-builder.test.js`
|
||||
- `tests/graph/apply-proposal.test.js`
|
||||
- `tests/graph/update-proposal.test.js` (directly related parse-boundary suite due to new enum enforcement)
|
||||
|
||||
### Required outcomes
|
||||
|
||||
1. raw `unsure` + structured `supportCategory=uncertain` does not produce old lexical mismatch rejection — **PASS**
|
||||
2. equivalent paraphrase wording does not change category acceptance when structured category is populated — **PASS**
|
||||
3. invalid `supportCategory` rejected by schema — **PASS**
|
||||
4. invalid `resolutionGuidance` rejected by schema — **PASS**
|
||||
5. `must_remain_unresolved` + relevant resolution mutation rejected — **PASS**
|
||||
6. `must_resolve` + unresolved target rejected if safely implementable — **DEFERRED**
|
||||
7. null structured fields still use existing lexical fallback — **PASS**
|
||||
8. populated `conditional_tradeoff` and `explicit_hard_constraint` use structured path without lexical verification — **PASS**
|
||||
9. `possibleInference` remains non-authoritative — **PASS**
|
||||
10. no new synonym/regex/keyword logic was added — **PASS**
|
||||
|
||||
## Commands Run
|
||||
|
||||
```bash
|
||||
npx vitest run tests/graph/schema.test.js tests/graph/apply-proposal.test.js tests/graph/prompt-builder.test.js
|
||||
npx vitest run tests/graph/update-proposal.test.js
|
||||
```
|
||||
|
||||
## What this now guarantees
|
||||
|
||||
1. Populated structured semantic fields are now the primary fidelity contract.
|
||||
2. The engine no longer re-derives protected semantic categories lexically when those structured fields are populated.
|
||||
3. Invalid structured category/resolution values fail at schema parse time.
|
||||
4. `must_remain_unresolved` is enforced through deterministic structured consistency rather than English keyword matching.
|
||||
5. Legacy null structured proposals still follow the old lexical fallback path during transition.
|
||||
|
||||
## What remains intentionally unresolved
|
||||
|
||||
1. Safe deterministic enforcement of `must_resolve` against a specific target unknown without inventing new linkage.
|
||||
2. Population reliability of structured fields in live model runs.
|
||||
3. Full retirement of the lexical fallback path once structured population is proven reliable.
|
||||
|
||||
## Constraints respected
|
||||
|
||||
- No new semantic taxonomy;
|
||||
- No synonym or regex expansion;
|
||||
- No new semantic classifier;
|
||||
- No new LLM call;
|
||||
- No provider integration changes;
|
||||
- No Ollama calls;
|
||||
- No graph redesign.
|
||||
Reference in New Issue
Block a user