robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

experiment: audit provenance in update prompt

Browse Source
This commit is contained in:
robbond
2026-08-07 14:07:28 +01:00
parent ec167f2689
commit 90472de766
2 changed files with 152 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/current-handoff.md
+6 -3
View File
@@ -18,6 +18,7 @@
- Passive experimental classifiers from Experiments 18–25B remain isolated diagnostic layers; none control the user-facing investigation. Behaviour Selection was passively evaluated against real assessment outputs in Experiment 39 — it produced all valid behaviours but with skewed distribution (Acknowledge 71%).
- Keyword and phrase-based scope detection remains provisional scaffolding.
- `docs/current-project-state.md` is the main entry point for active project state.
- Experiment 54D confirmed the production update prompt explicitly separates the user answer (## User Answer section) but the proposal schema has no provenance field — source identity at prompt level is explicit, per-node provenance at output level is absent.
- `docs/task-context-packs.md` chooses the minimum context documents for each work type.
## 3. Why Work Is Paused
@@ -72,6 +73,8 @@ Experiment 52C separated free-language semantic understanding from enum normalis
Experiment 52D isolated enum normalisation from semantic understanding: five fixed meaning statements (no decision target or question in the input) were mapped to the existing four-category contract via one live model call each. Four of five normalised to the expected enum. The compliance boundary case persisted — the model classified a "supports" relationship as `could_change_decision`, exposing genuine ambiguity between these two categories under the current definitions. The existing contract appears clear enough for a separate normalisation step; the remaining problem lies in category definitions, not semantic understanding or normalisation mechanism. Same Qwen model (`qwen-claude:latest`) and host (`http://192.168.1.111:11434`) were retained throughout. No production behaviour changed. What remains uncertain: whether the `supports_decision` ↔ `could_change_decision` boundary can be clarified without restructuring the contract, and whether the discrepancy holds under repeated runs. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/graph/decision-relevance-normalisation.test.js` for results.
Experiment 53 proved semantic separation of supplied meaning from possible inference is achievable. Experiment 54A confirmed the SituationGraph cannot recover provenance from graph state alone. Experiment 54B traced supplied-versus-inferred distinction upstream to evidenceRecordSchema but found it lost at buildInitialGraph because the node schema has no provenance field. Experiment 54C inspected the normal answer-update boundary: whole-input origin is explicit (answer = user supplied; proposal = model produced) but per-node provenance inside the proposal is not deterministically recoverable from the validated proposal alone. Experiment 54D audited the production update prompt: it clearly separates the user answer (## User Answer section) and instructions, so prompt-level source identity is explicit; however the proposed output schema has no provenance fields on nodes or edges, so per-node provenance at output level is absent — the gap is a schema deficiency, not a prompt-design problem.
## 5. What Remains Open
- The `too_broad` boundary sits exactly between three and four active unknowns; it is mechanically clear but conceptually uncertain — whether it aligns with genuine user confusion requires real-scenario validation;
@@ -124,8 +127,8 @@ Answer before continuing:
---
*Created by Experiment 34. Updated by Experiments 38–53, 54A. Branch: `feature/user-workspace-ux-v0.7`.*
*Created by Experiment 34. Updated by Experiments 38–53, 54A–54D. Branch: `feature/user-workspace-ux-v0.7`.*
### Return-to-Work Note (Experiment 53 → 54A → 54B → 54C)
### Return-to-Work Note (Experiment 53 → 54A → 54B → 54C → 54D)
Experiment 53 proved semantic separation was possible; 54A confirmed the SituationGraph cannot recover provenance from graph state alone; 54B showed supplied-versus-inferred distinction exists upstream in evidenceRecordSchema but is lost at buildInitialGraph because the node schema has no provenance field and evidenceIds carries only bare ID references. 54C inspected the normal answer-update boundary and found that while the raw user answer and validated model proposal are explicitly separate parameters at applyValidatedProposal's call site (orchestrator.js:683), they cannot be deterministically mapped to individual graph nodes within the proposal — the LLM independently generates additions from answer context with no provenance mapping. First function/file to inspect when resuming: `lib/graph/orchestrator.js` line 633 (buildGraphUpdatePrompt) to assess whether prompt structure carries any origin markers.
Experiment 54C clarified that whole-input origin remains deterministically knowable before application (answer = user supplied, proposal = model produced) but per-node provenance is lost because the validated proposal schema carries no per-node origin markers. Experiment 54D audited the production update prompt in lib/graph/prompt-builder.js and confirmed the user answer is explicitly identifiable to the model via a distinct named section (## User Answer). However, the requested proposal output schema has no provenance fields on nodes or edges, so supplied-versus-inferred origin cannot be preserved in the output. No production code changed. Branch: feature/user-workspace-ux-v0.7. First file to inspect when resuming: lib/graph/schema.js (graphUpdateSchema and situationNodeSchema) to evaluate minimal schema additions for per-node provenance fields.