experiment: confirm clean closure with direct metadata
This commit is contained in:
@@ -2994,3 +2994,7 @@ Experiment 60B.40 performed read-only post-mutation guard location diagnosis for
|
||||
Experiment 60B.41 was a read-only diagnosis of whether `selectActiveUnknownCandidate` must independently exclude terminal-status nodes (known/resolved/contradicted) for the 60B.37 closure path to be correct and fallback selection to remain semantically sound. **Classification: A — SELECTOR ITSELF MUST FILTER TERMINAL STATUS.** The selector at utils.js:593 filters only `kind === "unknown"` and `!resolvedNodeIds.includes(n.id)`. Zero status filtering exists. Key findings: (1) Scoring (`scoreUnknownCandidate`) is completely blind to node.status — it uses text classification, downstream count, and upstream unresolved dependency count, none of which inspect status. A known-status unknown-kind node scores identically to an unknown-status one. (2) In 60B.37's exact scenario: if no other genuine unresolved unknown remains, the selector would return the known decision node as the sole candidate, recreating the stale-target bug post-guard. (3) Adding terminal-status filtering does NOT affect genuine fallback candidates or prerequisite-first ordering because filtering happens before scoring and all genuine unknown-status nodes pass unchanged. (4) Five independent locations share the same gap pattern (`["resolved", "contradicted"]` exclusion without "known"): isSelectableUnresolvedUnknown, listUnresolvedUnknownCandidates, carriedActiveUnknownStillUnresolved, remainingUnknownExists inline check, and selectPatternCompatibleUnknownCandidate. This is PARTIAL overlap — different contracts serving different operational contexts but sharing the same systematic gap. **Critical distinction: D — SHARED ELIGIBILITY CONTRACT IS REQUIRED.** **Minimum implementation model: A — add terminal-status filter to selectActiveUnknownCandidate.** The selector must independently guard because it has zero caller-enforced input protection and known-status nodes bypass resolvedNodeIds entirely (they transition via updatedNodes.newStatus). Implementation boundary: one line change at utils.js:596 — add `!["known", "resolved", "contradicted"].includes(n.status)` to the existing filter. Ready for bounded implementation. No production code changed. 0 Ollama calls. Pure code inspection. Full trace in docs/experiment-60b41.md.
|
||||
|
||||
Experiment 60B.44 was a live regression of 60B.43's terminal post-mutation eligibility fix on the customer-signing closure case through the full production update path. **Classification: A — LIVE CLEAN CLOSURE CONFIRMED.** One bounded update call resolved `n_enterprise_customer_signing` in place (unknown → resolved) and closed `n_product_launch_decision` (unknown → resolved). Both option identities preserved. No new nodes or edges created. `activeUnknownNodeId = null` (cleared). Final `selectedQuestion = null`. Zero stale artifacts. This proves that 60B.43's terminal-status filtering across all six post-mutation eligibility paths eliminates the exact 60B.37-shaped stale-target persistence through live LLM-driven production code. One live Ollama call at qwen-claude:latest on http://192.168.1.111:11434. No production code changed. Status pending Rob's review.
|
||||
|
||||
---
|
||||
|
||||
Experiment 60B.46 reran the exact 60B.44 closure case through one bounded live update using 60B.45/46's explicit harness metadata capture fields (`finalActiveUnknownNodeId` and `finalSelectedQuestion`). **Classification: A — LIVE CLEAN CLOSURE DIRECTLY CONFIRMED.** The enterprise-customer unknown resolved in place (unknown → resolved) and the product-launch decision also resolved in place (unknown → resolved). Both option identities preserved. Zero new nodes or edges. `finalActiveUnknownNodeId = null` (directly observed, not inferred from omission). `finalSelectedQuestion = null` (directly observed, not inferred from omission). structuralActionRequired = null. This confirms that 60B.45's harness-level metadata exposure correctly captures the production runtime's clean closure state with explicit null values rather than field omission. One live Ollama call at qwen-claude:latest on http://192.168.1.111:11434. No production code changed.
|
||||
|
||||
@@ -0,0 +1,136 @@
|
||||
# Experiment 60B.46 — Direct closure metadata evidence in live harness
|
||||
|
||||
**Date:** 2026-08-14
|
||||
**Branch:** `feature/closure-metadata-capture-v0.39`
|
||||
|
||||
## Purpose
|
||||
|
||||
Confirm that when the final material customer-signing uncertainty is resolved, the production runtime directly returns:
|
||||
|
||||
- `finalActiveUnknownNodeId = null`
|
||||
- `finalSelectedQuestion = null`
|
||||
|
||||
using the explicit harness fields added in 60B.45/46 rather than inferring from field omission.
|
||||
|
||||
## Hypothesis
|
||||
|
||||
A successful result should show:
|
||||
|
||||
```text
|
||||
n_enterprise_customer_signing: resolved
|
||||
n_product_launch_decision: known or resolved
|
||||
addedNodes: []
|
||||
addedEdges: []
|
||||
finalActiveUnknownNodeId: null
|
||||
finalSelectedQuestion: null
|
||||
both existing options preserved (no duplication)
|
||||
```
|
||||
|
||||
No directional recommendation is required.
|
||||
|
||||
## Method
|
||||
|
||||
One bounded live update using the 60B.44 pre-anchored fixture:
|
||||
|
||||
- **Fixture:** `tests/fixtures/pre-anchored-product-launch-customer-signing.json`
|
||||
- **Mode:** `updateOnly` (single Update, no Start)
|
||||
- **Answer:** "Yes. The enterprise customer has now confirmed in writing that they will sign if we launch this year, so the £700,000 of expected annual revenue from them is confirmed. There are no other material uncertainties between launching this year and waiting twelve months."
|
||||
- **Model:** `qwen-claude:latest` via Ollama (`http://192.168.1.111:11434`)
|
||||
|
||||
## Call accounting
|
||||
|
||||
```
|
||||
startCalls: 0
|
||||
updateCalls: 1
|
||||
totalCalls: 1
|
||||
retries: 0
|
||||
```
|
||||
|
||||
## Results
|
||||
|
||||
### HTTP
|
||||
|
||||
- **Stage:** accepted (no rejection path)
|
||||
|
||||
### Structural mutation
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| `updatedNodes` | 2 nodes: `n_enterprise_customer_signing` (unknown→resolved), `n_product_launch_decision` (unknown→resolved) |
|
||||
| `resolvedUnknownNodeIds` | `["n_enterprise_customer_signing", "n_product_launch_decision"]` |
|
||||
| `addedNodes` | `[]` |
|
||||
| `addedEdges` | `[]` |
|
||||
|
||||
### Node final states
|
||||
|
||||
| Node ID | Kind | Status |
|
||||
|---------|------|--------|
|
||||
| `n_product_launch_state` | state | provisional |
|
||||
| `opt_launch_this_year` | option | known |
|
||||
| `opt_wait_twelve_months` | option | known |
|
||||
| `n_product_launch_decision` | unknown | **resolved** |
|
||||
| `n_enterprise_customer_signing` | unknown | **resolved** |
|
||||
|
||||
### Direct closure metadata (60B.46 harness fields)
|
||||
|
||||
```
|
||||
finalActiveUnknownNodeId: null
|
||||
finalSelectedQuestion: null
|
||||
```
|
||||
|
||||
### Identity preservation
|
||||
|
||||
- **Decision node (`n_product_launch_decision`):** PRESERVED — status changed to resolved, id unchanged
|
||||
- **Launch option (`opt_launch_this_year`):** PRESERVED — status known, id unchanged
|
||||
- **Wait option (`opt_wait_twelve_months`):** PRESERVED — status known, id unchanged
|
||||
|
||||
## Assessment
|
||||
|
||||
| Criterion | Result |
|
||||
|-----------|--------|
|
||||
| Customer factor | RESOLVED IN PLACE |
|
||||
| Decision state | RESOLVED |
|
||||
| Decision identity | PRESERVED |
|
||||
| Launch option | PRESERVED |
|
||||
| Wait option | PRESERVED |
|
||||
| Active lifecycle | **NULL — CLEARED** |
|
||||
| Final question | **NULL — DECISION COMPLETE** |
|
||||
| New uncertainty discipline | NONE |
|
||||
|
||||
## 60B.44 comparison
|
||||
|
||||
| Field | 60B.44 (inferred) | 60B.46 (direct) |
|
||||
|-------|--------------------|------------------|
|
||||
| activeUnknownNodeId | inferred from omission | **null — directly exposed** |
|
||||
| selectedQuestion | inferred from omission | **null — directly exposed** |
|
||||
|
||||
The important difference is measurement:
|
||||
- **60B.44:** active/final question inferred from omission
|
||||
- **60B.46:** active/final question directly exposed as raw values
|
||||
|
||||
## Classification: A — LIVE CLEAN CLOSURE DIRECTLY CONFIRMED
|
||||
|
||||
All criteria directly observed:
|
||||
|
||||
- ✅ customer factor resolves in place
|
||||
- ✅ decision closes in place
|
||||
- ✅ both options preserved
|
||||
- ✅ no added unknowns
|
||||
- ✅ `finalActiveUnknownNodeId = null` (direct)
|
||||
- ✅ `finalSelectedQuestion = null` (direct)
|
||||
|
||||
## What this proves
|
||||
|
||||
The production confidence engine correctly performs a **clean graph-level closure** when the final material uncertainty is resolved:
|
||||
|
||||
1. Both the customer-signing unknown and the central decision unknown are resolved in place (no duplication, no loss).
|
||||
2. The harness-observed `activeUnknownNodeId` is explicitly cleared to `null`, confirming the engine's internal active-target pointer is zeroed.
|
||||
3. The harness-observed `selectedQuestion` is explicitly `null`, confirming no follow-up question remains pending.
|
||||
4. No new uncertainties are introduced (zero addedNodes/edges).
|
||||
|
||||
## What remains weak or unproven
|
||||
|
||||
- Closure under contradictory/unexpected inputs (this test used a clean, expected-resolution path).
|
||||
- Multiple simultaneous uncertainty resolution in a single update.
|
||||
- Full-suite regression coverage for the closure metadata harness layer itself (60B.45 added focused unit tests: 67/67 pass).
|
||||
- Live closure verification on production hosts beyond localhost.
|
||||
Reference in New Issue
Block a user