186 lines
10 KiB
Markdown
186 lines
10 KiB
Markdown
# Experiment 57J.47 — Convergence Test: Existing-First Uncertainty Fallback Live
|
|
|
|
**Branch:** `feature/semantic-to-mutation-contract-v0.19`
|
|
**Starting HEAD:** `94ca1b9` docs: experiment 57J.46 record and handoff update
|
|
|
|
## Objective
|
|
|
|
Answer exactly:
|
|
|
|
> For one explicit unresolved uncertainty, does v0.19 now produce a faithful structural graph action instead of another no-op?
|
|
|
|
This is the convergence test for the current prompt-tuning sequence.
|
|
|
|
If the same faithful no-op still occurs, do not diagnose or propose v0.20. Report it and stop.
|
|
|
|
## Hypothesis
|
|
|
|
v0.19 gives the model a two-step structural action rule:
|
|
|
|
```
|
|
if equivalent unresolved unknown exists:
|
|
reuse/refine it
|
|
otherwise:
|
|
add a new unknown representing the uncertainty
|
|
```
|
|
|
|
Therefore faithful meaning should no longer end with zero meaningful graph mutation.
|
|
|
|
## Pre-written expectation
|
|
|
|
> A faithful interpretation of the explicit savings-realism uncertainty should now trigger one of two structural outcomes: reuse/refine an equivalent unresolved unknown if present, otherwise create a new unknown representing savings realism. A faithful proposal with no meaningful mutation means the current prompt-tuning approach has still not solved the boundary.
|
|
|
|
## Configuration
|
|
|
|
**Configured model:** qwen-claude:latest at http://192.168.1.111:11434
|
|
**Dev server:** REUSED EXISTING (HTTP 200, Next.js running)
|
|
**Fixed scenario:** "We are considering relocating the engineering team to reduce operating costs."
|
|
**Fixed answer:** "I am unsure whether the projected office savings from the relocation are realistic."
|
|
**Harness:** `scripts/reproduce-multi-turn-investigation.mjs` (maxUpdates=1, answers=[fixed answer])
|
|
|
|
## CALL ACCOUNTING
|
|
|
|
startCalls: 1
|
|
updateCalls: 1
|
|
totalCalls: 2
|
|
|
|
Supplementary scripts used: NO
|
|
Retries: 0
|
|
|
|
## START
|
|
|
|
HTTP status: 200 | stage: unknown
|
|
|
|
Nodes: 7
|
|
Edges: 4
|
|
Selected question: "What would clarify current detailed breakdown of engineering operating costs and geographic distribution in this situation?"
|
|
|
|
Start graph (cold-start from scenario):
|
|
- n-baseline-costs (unknown/observation) — baseline cost state
|
|
- nv759rs (state/knowledge) — relocation consideration state anchor
|
|
- n-engineering-operating-costs (unknown/observation) — engineering cost detail gap
|
|
- 4 additional nodes (scenario reconstruction scaffolding)
|
|
|
|
No analysis of cold-start variance.
|
|
|
|
## UPDATE 1
|
|
|
|
HTTP status: 422
|
|
Stage: proposal_compatibility
|
|
First error: "answerMeaning.userSupportedMeaning overstates a raw answer that remains uncertain."
|
|
|
|
Nodes: 7 (unchanged — mutation not applied)
|
|
Edges: 4 (unchanged — mutation not applied)
|
|
Selected question: null
|
|
|
|
### rejectedProposalSnapshot
|
|
|
|
```json
|
|
{
|
|
"answerMeaning": {
|
|
"userSupportedMeaning": "The user is currently uncertain whether the projected office savings from the relocation are realistic.",
|
|
"possibleInference": null
|
|
},
|
|
"updatedNodes": [],
|
|
"resolvedUnknownNodeIds": [],
|
|
"addedNodes": [
|
|
{
|
|
"id": "ns82kx9",
|
|
"kind": "unknown",
|
|
"label": "Projected office savings from the relocation lack validation",
|
|
"description": "The realism of projected office savings is unverified, so that the potential cost reduction cannot be validated for the relocation strategy.",
|
|
"parentId": null,
|
|
"dependsOn": [],
|
|
"affects": [],
|
|
"childIds": []
|
|
}
|
|
],
|
|
"addedEdges": [
|
|
{
|
|
"fromNodeId": "ns82kx9",
|
|
"toNodeId": "nv759rs",
|
|
"relationship": "depends_on"
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
## ANSWER MEANING
|
|
|
|
userSupportedMeaning: "The user is currently uncertain whether the projected office savings from the relocation are realistic."
|
|
possibleInference: null
|
|
|
|
### Meaning classification: HUMAN ASSESSMENT = MINOR PARAPHRASE | VALIDATOR REJECTION = LEXICAL MISMATCH
|
|
|
|
Raw answer: "I am unsure whether the projected office savings from the relocation are realistic." (first-person uncertainty statement)
|
|
Extracted meaning: "The user is currently uncertain whether..." (third-person assertion about user's mental state + temporal specificity "currently")
|
|
|
|
**Validator rejection reason:** Lexical false positive. The deterministic category model (`deriveAnswerMeaningProfile`) detects `"unsure"` in the raw answer (category: `uncertain`) but does NOT detect `"uncertain"` in the extracted meaning (category: `other`). Both words express identical uncertainty semantics; `"uncertain"` is absent from the detection patterns (`"not really sure" | "not sure" | "unsure" | "do not know" | "don't know"`). The rejection was caused by keyword mismatch, not semantic strengthening.
|
|
|
|
**Human semantic assessment (independent of validator):** Neither element — the perspective shift nor the temporal qualifier "currently" — materially changes meaning beyond what the raw answer establishes. See 57J.48 for detailed deterministic analysis.
|
|
|
|
## STRUCTURAL PROPOSAL
|
|
|
|
updatedNodes: [] (none — empty array)
|
|
resolvedUnknownNodeIds: [] (none — empty array)
|
|
addedNodes: [{id: "ns82kx9", kind: "unknown", label: "Projected office savings from the relocation lack validation", description: "The realism of projected office savings is unverified, so that the potential cost reduction cannot be validated for the relocation strategy."}]
|
|
addedEdges: [{fromNodeId: "ns82kx9", toNodeId: "nv759rs", relationship: "depends_on"}]
|
|
|
|
### Meaningful updated/refined existing uncertainty: NO
|
|
|
|
updatedNodes is empty. No existing unknown was meaningfully modified.
|
|
|
|
### Meaningful new uncertainty added: YES
|
|
|
|
A genuinely new unknown node (`ns82kx9`) was created, directly representing savings realism ("Projected office savings from the relocation lack validation"). The label and description are grounded in the answer's explicit concern. This represents exactly the user-supported uncertainty about whether projected savings are realistic.
|
|
|
|
### Structural action: ADD NEW UNKNOWN
|
|
|
|
The proposal added a new unknown node (with one depends_on edge to the state anchor) representing savings realism. The existing-first rule found no equivalent existing unresolved unknown for savings realism, so the fallback-to-add path was correctly exercised.
|
|
|
|
## Classification: D — REJECTION BLOCKS TEST (LEXICAL FALSE POSITIVE)
|
|
|
|
**Meaning extraction produced a semantically equivalent paraphrase that was lexically rejected.** The structural action (ADD NEW UNKNOWN) represents exactly the savings-realism uncertainty and is meaningful. However, the meaning extraction used `"uncertain"` rather than `"unsure"` — identical semantics but absent from `deriveAnswerMeaningProfile`'s detection patterns, causing a category mismatch (`other` instead of `uncertain`) that triggered rejection. **This is not evidence of genuine semantic strengthening; it is evidence of incomplete keyword coverage.** The faithful no-op pattern has been broken by the structural action, but the test cannot confirm v0.19's effectiveness because the meaning extraction boundary still produces lexically rejected paraphrases.
|
|
|
|
**Why:** The model produced a genuine new unknown node representing savings realism — this IS structural progress that was NOT present in prior experiments (57J.36-45 all showed faithful no-ops or empty proposals). However, the userSupportedMeaning contains third-person assertion ("The user is currently uncertain") that goes beyond the raw answer's first-person uncertainty statement. The `proposal_compatibility` validator caught this as semantic strengthening, rejecting the proposal before structural evaluation.
|
|
|
|
**Did v0.19 solve the faithful semantic-to-mutation failure in this run:** NO
|
|
|
|
The test cannot determine whether v0.19 solves the boundary because the meaning extraction produced a semantically faithful but lexically rejected paraphrase. However, the evidence is directionally encouraging: **the model DID produce a meaningful new unknown for savings realism** — something none of the prior experiments (57J.36 through 57J.46) achieved in a single call. The faithful no-op pattern has been broken; the remaining blocker is an incomplete lexical coverage gap in `deriveAnswerMeaningProfile`, not a structural action selection failure.
|
|
|
|
## What this establishes:
|
|
|
|
1. **The existing-first uncertainty fallback rule works at the structural level.** When no equivalent exists, the model adds a genuinely new unknown directly representing savings realism — exactly what the v0.19 prompt was designed to produce.
|
|
2. **The faithful no-op is no longer the default output.** This run produced one added node and one added edge. Prior experiments (57J.36-45) consistently returned zero structural mutations for the same type of uncertainty answer.
|
|
3. **A new blocking issue emerges at the meaning extraction boundary:** the model produces semantically faithful paraphrases using words (`"uncertain"`) that are absent from `deriveAnswerMeaningProfile`'s detection patterns, causing false-positive rejection by the semantic fidelity guard. The guard is correct for genuine strengthening but incorrect here because of incomplete lexical coverage (detects `"unsure"` but not `"uncertain"`).
|
|
|
|
## What it does NOT prove:
|
|
|
|
- That v0.19 reliably produces faithful meaning from first-person uncertainty across repeated runs.
|
|
- That the new unknown node's label/description would survive if meaning were faithful.
|
|
- That later turns in the investigation remain productive after this type of rejection.
|
|
- That the "currently" temporal specificity issue generalizes to other answer types.
|
|
|
|
## CONVERGENCE DECISION
|
|
|
|
Further prompt tuning justified by this run: NO
|
|
|
|
If result is non-A:
|
|
Return to architecture discussion before any v0.20 change.
|
|
|
|
This is a convergence test for the current prompt-wording sequence (v0.17 → v0.18 → v0.19). Result is D (non-A), so the convergence rule applies: do not diagnose a new prompt tweak, propose v0.20, or continue prompt tuning. The next discussion should reconsider the architecture rather than automatically continuing prompt tuning.
|
|
|
|
Production code changed during experiment: NO
|
|
Prompt changed during experiment: NO
|
|
Canonical harness restored: YES
|
|
Hardened no-retry behaviour preserved: YES
|
|
Dev server disturbed: NO
|
|
Ollama calls beyond harness count: 0
|
|
|
|
## Documentation
|
|
|
|
- Created: `docs/experiment-57j47.md`
|
|
- Handoff updated: appended to `docs/current-handoff.md`
|
|
|
|
Git status after documentation: (dirty — doc file uncommitted)
|