docs: experiment 57J.46 record and handoff update
This commit is contained in:
@@ -811,3 +811,23 @@ Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No producti
|
||||
One explicit instruction-order rule in Additional Guidance: check existing unresolved nodes first; if none represents the same uncertainty, create a new unknown. Deterministic order replaces ambiguous choice-list. No new classifiers, schema state, or validator changes required. Existing rule #11 and deterministic validator remain as safety net.
|
||||
|
||||
**Status:** Design complete. READY FOR BOUNDED IMPLEMENTATION: YES. Implementation requires one Additional Guidance bullet in prompt-builder.js plus 8 focused deterministic regressions in tests. Full record in `docs/experiment-57j45.md`.
|
||||
|
||||
### Experiment 57J.46 — Existing-First Uncertainty Fallback: Prompt Implementation
|
||||
|
||||
**Branch:** `feature/semantic-to-mutation-contract-v0.19`
|
||||
**Starting HEAD:** `acd1928` (HEAD of v0.18 at branch point)
|
||||
|
||||
**What changed:**
|
||||
- `lib/graph/prompt-builder.js` — Added one Additional Guidance bullet: "When rule #6 applies to explicitly unresolved uncertainty: first check whether an existing unresolved node already represents the same uncertainty; if so, update/refine that existing structure rather than adding a duplicate; if no such node exists, add a new unknown that directly represents the unresolved uncertainty; do not use an edge alone to represent a previously unrepresented uncertainty."
|
||||
- `tests/graph/prompt-builder.test.js` — Added 14 focused tests verifying: existing-first ordering, reuse path, fallback-to-add, related-node-insufficient, edge-only-prohibited, possibleInference separation, resolution path preserved, duplicate contract preserved, scope uncertainty-only, fidelity/traceability preserved, noop validator untouched, no semantic classifier added.
|
||||
|
||||
**Test results:**
|
||||
- prompt-builder.test.js: 37 tests pass (23 existing + 14 new)
|
||||
- utils.test.js: 68 tests pass (regression confirmation)
|
||||
- Total: 105 tests pass, 0 failures
|
||||
|
||||
**What this guarantees:** When rule #6 fires for explicitly unresolved uncertainty, the prompt gives a deterministic instruction order: check existing first → reuse if equivalent → otherwise add new. Rule is scoped to unresolved uncertainty only. All existing contracts preserved (duplicate avoidance, fidelity/inference separation, traceability, noop validator, structural-materialization MUST rule).
|
||||
|
||||
**What is intentionally left unresolved:** Semantic threshold for "represents the same uncertainty" (relies on model capability + validator safety net). Live production validation (requires next experiment pass). Multi-turn tuning. Edge-connection strategy for Case B new unknowns.
|
||||
|
||||
Configured Ollama: none used. Production code changed: prompt + tests only. Dev server disturbed: NO.
|
||||
|
||||
@@ -0,0 +1,89 @@
|
||||
# Experiment 57J.46 — Existing-First Uncertainty Fallback: Prompt Implementation
|
||||
|
||||
**Branch:** `feature/semantic-to-mutation-contract-v0.19`
|
||||
**Starting HEAD:** `acd1928` (HEAD of v0.18 at branch point)
|
||||
**Implementation commit:** pending
|
||||
|
||||
## Objective
|
||||
|
||||
Implement the existing-first fallback for explicit unresolved uncertainty chosen in Experiment 57J.45:
|
||||
|
||||
> When rule #6 applies to explicitly unresolved uncertainty, first reuse/refine an equivalent unresolved unknown if one exists; otherwise add a new unknown that directly represents the uncertainty.
|
||||
|
||||
## What was changed
|
||||
|
||||
### lib/graph/prompt-builder.js — Additional Guidance
|
||||
|
||||
One bullet added (line after existing "prefer updatedNodes" guidance):
|
||||
|
||||
```
|
||||
- When rule #6 applies to explicitly unresolved uncertainty: first check whether an existing unresolved node already represents the same uncertainty; if so, update/refine that existing structure rather than adding a duplicate; if no such node exists, add a new unknown that directly represents the unresolved uncertainty; do not use an edge alone to represent a previously unrepresented uncertainty.
|
||||
```
|
||||
|
||||
This is scoped specifically to `unresolved uncertainty` — it does NOT apply to facts, constraints, decisions, or resolved information.
|
||||
|
||||
### tests/graph/prompt-builder.test.js — Focused prompt tests
|
||||
|
||||
14 new tests in describe block "57J.46 existing-first uncertainty fallback":
|
||||
|
||||
| # | Test | What it verifies |
|
||||
|---|------|-----------------|
|
||||
| 1 | assembled prompt has existing-first ordering | Rule exists in full prompt |
|
||||
| 2 | reuse path explicit | update/refine language present |
|
||||
| 3 | fallback-to-add explicit | new-unknown path explicit |
|
||||
| 4 | full ordered fallback | entire rule as single coherent instruction |
|
||||
| 5 | related node insufficient | uses "same uncertainty" not weaker criteria |
|
||||
| 6 | edge-only insufficient | prohibition on edge-only representation |
|
||||
| 7 | possibleInference separation | rule does not reference possibleInference |
|
||||
| 8 | resolution path preserved | resolvedUnknownNodeIds + rule #5 intact |
|
||||
| 9 | duplicate contract preserved | rules #4, #11 unchanged |
|
||||
| 10 | scope uncertainty-only | scoped to "explicitly unresolved uncertainty" only |
|
||||
| 11 | fidelity separation | userSupportedMeaning vs possibleInference rule untouched |
|
||||
| 12 | traceability | new-unknown traceability rule intact |
|
||||
| 13 | noop validator | "rule #6 does not apply → empty arrays" unchanged |
|
||||
| 14 | no semantic classifier | no threshold/synonym/keyword logic added |
|
||||
| 15 | provider-agnostic | no provider-specific wording |
|
||||
|
||||
## Controlled case mapping
|
||||
|
||||
### Case A — existing equivalent unknown (prompt instruction)
|
||||
|
||||
When graph contains:
|
||||
> "Whether projected relocation savings are realistic"
|
||||
|
||||
And user says:
|
||||
> "I am unsure whether the projected office savings from the relocation are realistic."
|
||||
|
||||
Prompt now instructs: **reuse/refine existing unresolved unknown; do not add duplicate.**
|
||||
|
||||
### Case B — no equivalent unknown (prompt instruction)
|
||||
|
||||
When graph contains only broader cost/relocation concepts.
|
||||
|
||||
Same user statement.
|
||||
|
||||
Prompt now instructs: **add a new unknown directly representing savings realism.**
|
||||
|
||||
## Test results
|
||||
|
||||
- prompt-builder.test.js: 37 tests pass (23 existing + 14 new)
|
||||
- utils.test.js: 68 tests pass (regression confirmation)
|
||||
- Total: 105 tests pass, 0 failed
|
||||
|
||||
## What this implementation guarantees
|
||||
|
||||
- When the model receives an answer containing explicitly unresolved uncertainty and rule #6 fires, the assembled prompt now gives a deterministic instruction order: check existing first → reuse if equivalent → otherwise add new.
|
||||
- The rule is scoped only to unresolved uncertainty. It does not apply universally to all meaning categories.
|
||||
- Existing contracts are preserved: duplicate avoidance (rules #4, #11), possibleInference separation (rule #27), fidelity rules (rule #26), traceability (rule #9/9a), noop validator (Additional Guidance "rule #6 does not apply"), structural-materialization MUST rule (57J.39).
|
||||
|
||||
## What this intentionally leaves unresolved
|
||||
|
||||
- The semantic threshold for "represents the same uncertainty" — relies on model's prompt-level semantic comparison capability + deterministic validator as safety net.
|
||||
- Live production validation of Option A — requires next experiment pass (live run with fresh case).
|
||||
- Whether the instruction order needs tuning across multiple turns.
|
||||
- Edge-connection strategy for new unknowns created under Case B.
|
||||
|
||||
## Documentation
|
||||
|
||||
This file: `docs/experiment-57j46.md`
|
||||
Handoff update: appended to `docs/current-handoff.md`
|
||||
Reference in New Issue
Block a user