experiment: convergence-test uncertainty action selection
This commit is contained in:
@@ -831,3 +831,33 @@ One explicit instruction-order rule in Additional Guidance: check existing unres
|
||||
**What is intentionally left unresolved:** Semantic threshold for "represents the same uncertainty" (relies on model capability + validator safety net). Live production validation (requires next experiment pass). Multi-turn tuning. Edge-connection strategy for Case B new unknowns.
|
||||
|
||||
Configured Ollama: none used. Production code changed: prompt + tests only. Dev server disturbed: NO.
|
||||
|
||||
### Experiment 57J.47 — Convergence Test: Existing-First Uncertainty Fallback Live
|
||||
|
||||
**Objective:** Does v0.19 produce a faithful structural graph action for one explicit unresolved uncertainty instead of another no-op?
|
||||
|
||||
**Fixed scenario:** "We are considering relocating the engineering team to reduce operating costs."
|
||||
**Fixed answer:** "I am unsure whether the projected office savings from the relocation are realistic."
|
||||
|
||||
**Live-call count:** start: 1, update: 1, total: 2
|
||||
|
||||
**START:** HTTP 200 | stage: unknown | nodes: 7 | edges: 4 | question: "What would clarify current detailed breakdown of engineering operating costs and geographic distribution in this situation?"
|
||||
|
||||
**UPDATE 1:** HTTP 422 | stage: proposal_compatibility | error: "answerMeaning.userSupportedMeaning overstates a raw answer that remains uncertain."
|
||||
|
||||
rejectedProposalSnapshot:
|
||||
- userSupportedMeaning: "The user is currently uncertain whether the projected office savings from the relocation are realistic." (STRENGTHENED — third-person assertion of mental state + temporal specificity "currently" beyond first-person "I am unsure")
|
||||
- possibleInference: null
|
||||
- addedNodes: [{id: "ns82kx9", kind: unknown, label: "Projected office savings from the relocation lack validation"}] (genuinely new — ADD NEW UNKNOWN structural action)
|
||||
- addedEdges: [{fromNodeId: "ns82kx9", toNodeId: "nv759rs", relationship: depends_on}]
|
||||
- updatedNodes: [] | resolvedUnknownNodeIds: []
|
||||
|
||||
**Meaning classification:** STRENGTHENED (third-person assertion of user mental state + temporal specificity overstates first-person uncertainty)
|
||||
|
||||
**Structural action:** ADD NEW UNKNOWN — a genuinely new unknown directly representing savings realism was created. This is the FIRST time in experiments 57J.36-47 that the model produced a meaningful structural mutation for this type of uncertainty answer.
|
||||
|
||||
**Classification: D — STRENGTHENING BLOCKS TEST.** Meaning is strengthened; the test cannot establish v0.19's effectiveness for the semantic-to-mutation boundary. However, structural progress was achieved (one new unknown + one edge) — the faithful no-op pattern has been broken. The remaining blocker is meaning extraction (first-person uncertainty → third-person assertion), not the existing-first rule.
|
||||
|
||||
**Key finding:** v0.19's existing-first fallback now produces meaningful graph mutation for explicit unresolved uncertainty. New blocking boundary: first-person uncertainty statements are being converted to third-person assertions, caught by proposal_compatibility as semantic strengthening. Convergence test result is non-A; prompt tuning sequence terminates here per convergence rule.
|
||||
|
||||
Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. 2 live calls total. No production code changed. No prompt change during experiment. Harness restored to original scenario/answers.
|
||||
|
||||
@@ -0,0 +1,187 @@
|
||||
# Experiment 57J.47 — Convergence Test: Existing-First Uncertainty Fallback Live
|
||||
|
||||
**Branch:** `feature/semantic-to-mutation-contract-v0.19`
|
||||
**Starting HEAD:** `94ca1b9` docs: experiment 57J.46 record and handoff update
|
||||
|
||||
## Objective
|
||||
|
||||
Answer exactly:
|
||||
|
||||
> For one explicit unresolved uncertainty, does v0.19 now produce a faithful structural graph action instead of another no-op?
|
||||
|
||||
This is the convergence test for the current prompt-tuning sequence.
|
||||
|
||||
If the same faithful no-op still occurs, do not diagnose or propose v0.20. Report it and stop.
|
||||
|
||||
## Hypothesis
|
||||
|
||||
v0.19 gives the model a two-step structural action rule:
|
||||
|
||||
```
|
||||
if equivalent unresolved unknown exists:
|
||||
reuse/refine it
|
||||
otherwise:
|
||||
add a new unknown representing the uncertainty
|
||||
```
|
||||
|
||||
Therefore faithful meaning should no longer end with zero meaningful graph mutation.
|
||||
|
||||
## Pre-written expectation
|
||||
|
||||
> A faithful interpretation of the explicit savings-realism uncertainty should now trigger one of two structural outcomes: reuse/refine an equivalent unresolved unknown if present, otherwise create a new unknown representing savings realism. A faithful proposal with no meaningful mutation means the current prompt-tuning approach has still not solved the boundary.
|
||||
|
||||
## Configuration
|
||||
|
||||
**Configured model:** qwen-claude:latest at http://192.168.1.111:11434
|
||||
**Dev server:** REUSED EXISTING (HTTP 200, Next.js running)
|
||||
**Fixed scenario:** "We are considering relocating the engineering team to reduce operating costs."
|
||||
**Fixed answer:** "I am unsure whether the projected office savings from the relocation are realistic."
|
||||
**Harness:** `scripts/reproduce-multi-turn-investigation.mjs` (maxUpdates=1, answers=[fixed answer])
|
||||
|
||||
## CALL ACCOUNTING
|
||||
|
||||
startCalls: 1
|
||||
updateCalls: 1
|
||||
totalCalls: 2
|
||||
|
||||
Supplementary scripts used: NO
|
||||
Retries: 0
|
||||
|
||||
## START
|
||||
|
||||
HTTP status: 200 | stage: unknown
|
||||
|
||||
Nodes: 7
|
||||
Edges: 4
|
||||
Selected question: "What would clarify current detailed breakdown of engineering operating costs and geographic distribution in this situation?"
|
||||
|
||||
Start graph (cold-start from scenario):
|
||||
- n-baseline-costs (unknown/observation) — baseline cost state
|
||||
- nv759rs (state/knowledge) — relocation consideration state anchor
|
||||
- n-engineering-operating-costs (unknown/observation) — engineering cost detail gap
|
||||
- 4 additional nodes (scenario reconstruction scaffolding)
|
||||
|
||||
No analysis of cold-start variance.
|
||||
|
||||
## UPDATE 1
|
||||
|
||||
HTTP status: 422
|
||||
Stage: proposal_compatibility
|
||||
First error: "answerMeaning.userSupportedMeaning overstates a raw answer that remains uncertain."
|
||||
|
||||
Nodes: 7 (unchanged — mutation not applied)
|
||||
Edges: 4 (unchanged — mutation not applied)
|
||||
Selected question: null
|
||||
|
||||
### rejectedProposalSnapshot
|
||||
|
||||
```json
|
||||
{
|
||||
"answerMeaning": {
|
||||
"userSupportedMeaning": "The user is currently uncertain whether the projected office savings from the relocation are realistic.",
|
||||
"possibleInference": null
|
||||
},
|
||||
"updatedNodes": [],
|
||||
"resolvedUnknownNodeIds": [],
|
||||
"addedNodes": [
|
||||
{
|
||||
"id": "ns82kx9",
|
||||
"kind": "unknown",
|
||||
"label": "Projected office savings from the relocation lack validation",
|
||||
"description": "The realism of projected office savings is unverified, so that the potential cost reduction cannot be validated for the relocation strategy.",
|
||||
"parentId": null,
|
||||
"dependsOn": [],
|
||||
"affects": [],
|
||||
"childIds": []
|
||||
}
|
||||
],
|
||||
"addedEdges": [
|
||||
{
|
||||
"fromNodeId": "ns82kx9",
|
||||
"toNodeId": "nv759rs",
|
||||
"relationship": "depends_on"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## ANSWER MEANING
|
||||
|
||||
userSupportedMeaning: "The user is currently uncertain whether the projected office savings from the relocation are realistic."
|
||||
possibleInference: null
|
||||
|
||||
### Meaning classification: STRENGTHENED
|
||||
|
||||
Raw answer: "I am unsure whether the projected office savings from the relocation are realistic." (first-person uncertainty statement)
|
||||
Extracted meaning: "The user is currently uncertain whether..." (third-person assertion about user's mental state + temporal specificity "currently")
|
||||
|
||||
The model converted first-person uncertainty ("I am unsure") into third-person assertion ("The user is currently uncertain"). This adds two elements beyond the raw answer:
|
||||
1. **Perspective shift:** Asserting what "the user" feels/thinks, rather than preserving the first-person expression.
|
||||
2. **Temporal specificity:** Adding "currently" — an explicit temporal qualifier not present in the raw answer.
|
||||
|
||||
This goes slightly beyond preserving "only uncertainty over whether projected office savings are realistic." The extracted meaning introduces a stronger claim (assertion about user mental state) that was not established by the raw answer.
|
||||
|
||||
## STRUCTURAL PROPOSAL
|
||||
|
||||
updatedNodes: [] (none — empty array)
|
||||
resolvedUnknownNodeIds: [] (none — empty array)
|
||||
addedNodes: [{id: "ns82kx9", kind: "unknown", label: "Projected office savings from the relocation lack validation", description: "The realism of projected office savings is unverified, so that the potential cost reduction cannot be validated for the relocation strategy."}]
|
||||
addedEdges: [{fromNodeId: "ns82kx9", toNodeId: "nv759rs", relationship: "depends_on"}]
|
||||
|
||||
### Meaningful updated/refined existing uncertainty: NO
|
||||
|
||||
updatedNodes is empty. No existing unknown was meaningfully modified.
|
||||
|
||||
### Meaningful new uncertainty added: YES
|
||||
|
||||
A genuinely new unknown node (`ns82kx9`) was created, directly representing savings realism ("Projected office savings from the relocation lack validation"). The label and description are grounded in the answer's explicit concern. This represents exactly the user-supported uncertainty about whether projected savings are realistic.
|
||||
|
||||
### Structural action: ADD NEW UNKNOWN
|
||||
|
||||
The proposal added a new unknown node (with one depends_on edge to the state anchor) representing savings realism. The existing-first rule found no equivalent existing unresolved unknown for savings realism, so the fallback-to-add path was correctly exercised.
|
||||
|
||||
## Classification: D — STRENGTHENING BLOCKS TEST
|
||||
|
||||
**Meaning is STRENGTHENED.** The structural action (ADD NEW UNKNOWN) represents exactly the savings-realism uncertainty and is meaningful. However, the meaning extraction is not purely faithful (third-person assertion + temporal specificity overstates the raw first-person uncertainty). Therefore this test does not establish v0.19's effectiveness for the semantic-to-mutation boundary.
|
||||
|
||||
**Why:** The model produced a genuine new unknown node representing savings realism — this IS structural progress that was NOT present in prior experiments (57J.36-45 all showed faithful no-ops or empty proposals). However, the userSupportedMeaning contains third-person assertion ("The user is currently uncertain") that goes beyond the raw answer's first-person uncertainty statement. The `proposal_compatibility` validator caught this as semantic strengthening, rejecting the proposal before structural evaluation.
|
||||
|
||||
**Did v0.19 solve the faithful semantic-to-mutation failure in this run:** NO
|
||||
|
||||
The test cannot determine whether v0.19 solves the boundary because meaning was strengthened, which blocks the test. However, the evidence is directionally encouraging: **the model DID produce a meaningful new unknown for savings realism** — something none of the prior experiments (57J.36 through 57J.46) achieved in a single call. The faithful no-op pattern has been broken; the remaining blocker is the meaning extraction boundary, not the structural action selection.
|
||||
|
||||
## What this establishes:
|
||||
|
||||
1. **The existing-first uncertainty fallback rule works at the structural level.** When no equivalent exists, the model adds a genuinely new unknown directly representing savings realism — exactly what the v0.19 prompt was designed to produce.
|
||||
2. **The faithful no-op is no longer the default output.** This run produced one added node and one added edge. Prior experiments (57J.36-45) consistently returned zero structural mutations for the same type of uncertainty answer.
|
||||
3. **A new blocking issue emerges at the meaning extraction boundary:** first-person uncertainty statements ("I am unsure") are being converted to third-person assertions ("The user is currently uncertain"), which the semantic fidelity guard correctly flags as strengthening.
|
||||
|
||||
## What it does NOT prove:
|
||||
|
||||
- That v0.19 reliably produces faithful meaning from first-person uncertainty across repeated runs.
|
||||
- That the new unknown node's label/description would survive if meaning were faithful.
|
||||
- That later turns in the investigation remain productive after this type of rejection.
|
||||
- That the "currently" temporal specificity issue generalizes to other answer types.
|
||||
|
||||
## CONVERGENCE DECISION
|
||||
|
||||
Further prompt tuning justified by this run: NO
|
||||
|
||||
If result is non-A:
|
||||
Return to architecture discussion before any v0.20 change.
|
||||
|
||||
This is a convergence test for the current prompt-wording sequence (v0.17 → v0.18 → v0.19). Result is D (non-A), so the convergence rule applies: do not diagnose a new prompt tweak, propose v0.20, or continue prompt tuning. The next discussion should reconsider the architecture rather than automatically continuing prompt tuning.
|
||||
|
||||
Production code changed during experiment: NO
|
||||
Prompt changed during experiment: NO
|
||||
Canonical harness restored: YES
|
||||
Hardened no-retry behaviour preserved: YES
|
||||
Dev server disturbed: NO
|
||||
Ollama calls beyond harness count: 0
|
||||
|
||||
## Documentation
|
||||
|
||||
- Created: `docs/experiment-57j47.md`
|
||||
- Handoff updated: appended to `docs/current-handoff.md`
|
||||
|
||||
Git status after documentation: (dirty — doc file uncommitted)
|
||||
Reference in New Issue
Block a user