docs(confidence-engine): checkpoint design evolution archive tranche six
This commit is contained in:
@@ -154,9 +154,37 @@ Experiments 54A–54K — complete provenance audit chain including graph proven
|
|||||||
Fidelity:
|
Fidelity:
|
||||||
Exact contiguous copy.
|
Exact contiguous copy.
|
||||||
|
|
||||||
Tranches 1 through 5 have now been extracted.
|
Tranches 1 through 6 have now been extracted.
|
||||||
The original monolithic log remains intact and authoritative while extraction is incomplete.
|
The original monolithic log remains intact and authoritative while extraction is incomplete.
|
||||||
|
|
||||||
|
## Tranche 6
|
||||||
|
|
||||||
|
### Chapter 11
|
||||||
|
Path:
|
||||||
|
docs/archive/experiments/vol-1-chapters/ch11/experiments-54l-to-54q-grounding-and-evidence-stability.md
|
||||||
|
|
||||||
|
Original source:
|
||||||
|
docs/design-evolution-log.md lines 6425–7824
|
||||||
|
|
||||||
|
Contents:
|
||||||
|
Experiments 54L–54Q — semantic grounding stability under repetition, disagreement exposure between interpretations, disagreement-driven information need changes, evidence-need discrimination for same-goal scenarios, explicit evidence needs recovering higher-level consequences, and structured semantic steps preserving evidence distinction with consequence.
|
||||||
|
|
||||||
|
Fidelity:
|
||||||
|
Exact contiguous copy.
|
||||||
|
|
||||||
|
### Chapter 12
|
||||||
|
Path:
|
||||||
|
docs/archive/experiments/vol-1-chapters/ch12/experiment-54r-clarification-requires-source.md
|
||||||
|
|
||||||
|
Original source:
|
||||||
|
docs/design-evolution-log.md lines 7825–8012
|
||||||
|
|
||||||
|
Contents:
|
||||||
|
Experiment 54R — testing whether models correctly distinguish disagreements resolvable through evidence from those requiring user input, across competing causal hypotheses, ambiguous user priority, and absent material disagreement patterns.
|
||||||
|
|
||||||
|
Fidelity:
|
||||||
|
Exact contiguous copy.
|
||||||
|
|
||||||
## Refactor status
|
## Refactor status
|
||||||
|
|
||||||
Only tranches 1 and 2 have been extracted.
|
Only tranches 1 and 2 have been extracted.
|
||||||
|
|||||||
+1400
File diff suppressed because it is too large
Load Diff
+188
@@ -0,0 +1,188 @@
|
|||||||
|
## Experiment 54R — Does a Material Disagreement Actually Require User Clarification? (2026-08-07)
|
||||||
|
|
||||||
|
### Objective
|
||||||
|
|
||||||
|
Given an explicit interpretation disagreement and its evidence consequence, test whether the model can distinguish between a disagreement that requires clarification from the user and one that can be resolved by investigating evidence.
|
||||||
|
|
||||||
|
This is passive and test-only. Do not generate the clarification question. Do not generate the next investigation question. Do not choose a winning interpretation. Do not change production behaviour.
|
||||||
|
|
||||||
|
### Hypothesis
|
||||||
|
|
||||||
|
The model may be able to distinguish:
|
||||||
|
|
||||||
|
**Evidence-resolvable disagreement:** The user's meaning is sufficiently clear, but competing explanations require different evidence.
|
||||||
|
|
||||||
|
**User-clarification disagreement:** The disagreement concerns the user's intended meaning, priority, constraint, or definition, so external evidence cannot resolve it without asking the user.
|
||||||
|
|
||||||
|
If this distinction works, disagreement does not have to map automatically to clarification.
|
||||||
|
|
||||||
|
### Context Budget
|
||||||
|
|
||||||
|
Read only:
|
||||||
|
- `docs/current-handoff.md`;
|
||||||
|
- Experiment 54Q only in `docs/design-evolution-log.md`;
|
||||||
|
- `tests/reconstruction/semantic-structured-evidence-consequence.test.js`;
|
||||||
|
- `.env.local` only for `OLLAMA_BASE_URL` and `OLLAMA_MODEL`.
|
||||||
|
|
||||||
|
Not read: Behaviour Selection; assessor; graph files; UI; active prompts; question-selection code; full experiment history.
|
||||||
|
|
||||||
|
### Configuration
|
||||||
|
|
||||||
|
Host: `http://192.168.1.111:11434` (same as 54Q)
|
||||||
|
Model: `qwen-claude:latest` (same as 54Q)
|
||||||
|
|
||||||
|
No localhost fallback. No experiment-specific model variable.
|
||||||
|
|
||||||
|
### Number of Live Inference Calls
|
||||||
|
|
||||||
|
Exactly **3** live Ollama calls — one per case.
|
||||||
|
|
||||||
|
### Input Contract
|
||||||
|
|
||||||
|
Each call receives:
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"source": "...",
|
||||||
|
"disagreement": ["..."],
|
||||||
|
"evidenceNeeded": ["..."]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
The disagreement and evidence needs are fixed human-reviewed inputs. The model does not rediscover them.
|
||||||
|
|
||||||
|
### Output Contract
|
||||||
|
|
||||||
|
Return exactly:
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"requiresUserClarification": true | false,
|
||||||
|
"reason": "one short sentence"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
No question text, no recommended action, no preferred interpretation, no score, no confidence, no behaviour label. This boolean is test-only and is not a production contract.
|
||||||
|
|
||||||
|
### Semantic Instruction Used
|
||||||
|
|
||||||
|
> Decide whether resolving the stated disagreement requires additional meaning, preference, intent, or factual information that only the user can provide. Return true when evidence alone cannot settle the disagreement because the missing distinction belongs to the user's intended meaning, priority, constraint, or private knowledge. Return false when the disagreement can be investigated using external, operational, or observable evidence without asking the user to define what they mean. Do not generate a question and do not choose which interpretation is correct.
|
||||||
|
|
||||||
|
No examples. No expected answers. Same instruction for all three cases.
|
||||||
|
|
||||||
|
### Case 1 — Competing Causes, Evidence Can Resolve
|
||||||
|
|
||||||
|
**Source:** "Orders are arriving late and customers have started complaining."
|
||||||
|
**Disagreement:** delays may be caused by insufficient staff capacity / delays may be caused by unreliable supplier lead times.
|
||||||
|
**Evidence needed:** staffing levels and workload; processing throughput; supplier lead-time history; supplier delivery reliability.
|
||||||
|
**Fixed human reference:** `requiresUserClarification: false`
|
||||||
|
|
||||||
|
### Case 2 — User Priority Is Ambiguous
|
||||||
|
|
||||||
|
**Source:** "I want the business to grow, but I don't want to take on more risk."
|
||||||
|
**Disagreement:** growth should be prioritised even if some additional risk is unavoidable / avoiding additional risk is a hard constraint even if growth is slower.
|
||||||
|
**Evidence needed:** possible growth opportunities; risk exposure of each option.
|
||||||
|
**Fixed human reference:** `requiresUserClarification: true`
|
||||||
|
|
||||||
|
### Case 3 — Same Meaning, No Clarification Need
|
||||||
|
|
||||||
|
**Source:** "Revenue is down. I think pricing may be part of the problem, but I am not sure."
|
||||||
|
**Disagreement:** no substantive disagreement; two interpretations express the same uncertainty in different words.
|
||||||
|
**Evidence needed:** pricing history; sales/conversion response; other plausible revenue drivers.
|
||||||
|
**Fixed human reference:** `requiresUserClarification: false`
|
||||||
|
|
||||||
|
### Results
|
||||||
|
|
||||||
|
#### Case 1 — Competing Causes, Evidence Can Resolve
|
||||||
|
|
||||||
|
**Model output:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"requiresUserClarification": false,
|
||||||
|
"reason": "The disagreement involves operational causes that can be resolved by analyzing objective company data and supplier records."
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Resolution-source classification:** `resolution_source_correct` ✓ — Returned `false`. The model correctly identified that operational evidence (company data, supplier records) can distinguish the competing causal hypotheses without asking the user.
|
||||||
|
|
||||||
|
#### Case 2 — User Priority Is Ambiguous
|
||||||
|
|
||||||
|
**Model output:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"requiresUserClarification": true,
|
||||||
|
"reason": "Resolving this conflict depends on the user's personal priority and risk tolerance, which cannot be determined by external facts alone."
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Resolution-source classification:** `resolution_source_correct` ✓ — Returned `true`. The model correctly identified that user-owned meaning (priority, risk tolerance) is required to resolve the disagreement. External facts alone cannot determine whether "don't want more risk" is a preference, constraint, or boundary.
|
||||||
|
|
||||||
|
#### Case 3 — Same Meaning, No Clarification Need
|
||||||
|
|
||||||
|
**Model output:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"requiresUserClarification": false,
|
||||||
|
"reason": "The stated position explicitly confirms there is no substantive disagreement, so no additional user input is required to resolve it."
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Resolution-source classification:** `resolution_source_correct` ✓ — Returned `false`. The model correctly identified that the absence of material disagreement means clarification is unnecessary. Evidence can investigate the underlying uncertainty.
|
||||||
|
|
||||||
|
### Resolution-Source-Correct Count
|
||||||
|
|
||||||
|
**3 of 3** cases classified correctly.
|
||||||
|
|
||||||
|
### Resolution-Source-Failed Count
|
||||||
|
|
||||||
|
**0 of 3** cases classified incorrectly.
|
||||||
|
|
||||||
|
### Did Case 1 Correctly Recognise That Competing Causal Hypotheses Can Be Resolved Through Evidence?
|
||||||
|
|
||||||
|
Yes. The model returned `false` and provided a reason referencing operational causes resolvable by company data and supplier records — evidence, not user meaning.
|
||||||
|
|
||||||
|
### Did Case 2 Correctly Recognise That the Unresolved Growth-Versus-Risk Priority Belongs to the User?
|
||||||
|
|
||||||
|
Yes. The model returned `true` and identified that resolution depends on "the user's personal priority and risk tolerance," which external facts alone cannot determine.
|
||||||
|
|
||||||
|
### Did Case 3 Avoid Unnecessary Clarification Where There Was No Material Disagreement?
|
||||||
|
|
||||||
|
Yes. The model returned `false`, correctly noting the absence of substantive disagreement makes additional clarification unnecessary.
|
||||||
|
|
||||||
|
### Did the Model Treat Every Disagreement as Requiring User Clarification?
|
||||||
|
|
||||||
|
No. Two of three cases returned `false`. Only Case 2 (ambiguous priority) returned `true`.
|
||||||
|
|
||||||
|
### Did the Model Confuse Missing Evidence with Missing User Meaning?
|
||||||
|
|
||||||
|
No. In Case 1, the model correctly distinguished between lacking evidence to investigate causes (which it flagged as resolvable through evidence gathering) and lacking user meaning (which it did not claim). The reason text referenced "analyzing objective company data and supplier records" rather than requiring user input.
|
||||||
|
|
||||||
|
### Did the Model Generate an Actual Question?
|
||||||
|
|
||||||
|
No. No question was generated in any output. The output contract was respected in all cases.
|
||||||
|
|
||||||
|
### Did the Model Choose a Winner?
|
||||||
|
|
||||||
|
No. No interpretation was selected as correct in any case.
|
||||||
|
|
||||||
|
### Evidence That User-Owned Ambiguity Can Be Separated From Evidence Uncertainty
|
||||||
|
|
||||||
|
Case 2 succeeded where Case 1 and Case 3 both returned `false` for different reasons — one because evidence can resolve it, the other because no disagreement exists. The model's reasons for each case were distinct in their reference points: operational data (Case 1) versus user priority (Case 2) versus absence of disagreement (Case 3). This pattern suggests the model does not collapse all ambiguity into a single clarification need.
|
||||||
|
|
||||||
|
### Limitations
|
||||||
|
|
||||||
|
- Three cases only; limited domain coverage (one delivery scenario, one strategic priority, one revenue statement).
|
||||||
|
- Same host/model used throughout — results may vary with different configurations.
|
||||||
|
- Does not establish generalisation beyond these specific inputs.
|
||||||
|
- The distinction tested here is binary (true/false) and does not test partial or probabilistic resolution-source classification.
|
||||||
|
- No evidence was actually gathered in any case — only whether the *source* of resolution was correctly identified.
|
||||||
|
- The Case 1 evaluator warning (if present) was a false positive from heuristic wording checks, not a semantic failure.
|
||||||
|
|
||||||
|
### Conclusion
|
||||||
|
|
||||||
|
**The model correctly distinguished user-clarification needs from evidence-resolvable disagreement in all tested cases.**
|
||||||
|
|
||||||
|
Across the three tested patterns — competing causal hypotheses, ambiguous user priority, and absent material disagreement — the model returned the correct boolean in every case with semantically appropriate reasoning. No clarification or investigation question was generated. No interpretation was selected as correct. Across the three tested disagreement patterns, the model did not automatically map disagreement to user clarification.
|
||||||
|
|
||||||
|
### Status
|
||||||
|
|
||||||
|
**Pending Rob's review.** No production code changed. No schemas modified. No active engine behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-disagreement-resolution-source.test.js`.
|
||||||
|
|
||||||
Reference in New Issue
Block a user