Feature/product platform foundation v0.62 #1
@@ -377,3 +377,7 @@ Tested whether a single answer containing two independent evidence dimensions is
|
||||
**What remains unproven:** Repeated-run stability; cross-domain generalisation; downstream investigation viability past Update 2+; implicit conjunction admission; whether `too_broad` conversation health eventually blocks later turns.
|
||||
|
||||
Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No production code changed. Branch: `feature/user-supported-unknown-admission-v0.15`.
|
||||
|
||||
### Experiment 57J.28 — Live Node-Support Semantic Inputs Capture
|
||||
|
||||
**Classification: C — GATE BEHAVES AS EXPECTED.** On a fresh live run, both user-supported unknowns (savings realism + retention impact) were admitted with HTTP 200 at `update_applied`, zero validation errors. The answerMeaning correctly captured both dimensions. The semantic gate (`rawAnswerSupportsUnclassifiedMeaning`) passed for both via the token-count clause (5 shared content tokens >= 3), not overlap ratio. Structural fallback also valid for both nodes. Static helper evaluation agrees with actual gate behavior — live admission outcome fully explained by captured inputs. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. Start: 7 nodes, Update 1: 9 nodes (+2). What remains unproven: token-count adequacy for paraphrased/low-overlap answers, downstream investigation viability past Update 2+, run-to-run start graph stability. No production code changed.
|
||||
|
||||
@@ -0,0 +1,144 @@
|
||||
# Experiment 57J.28 — Live Node-Support Semantic Inputs Capture
|
||||
|
||||
**Objective:** On one fresh live run of the 57J.25 case, capture the exact semantic inputs that reach the v0.15 node-support gate and determine whether the savings/retention unknowns pass or fail.
|
||||
|
||||
## Pre-written expectation recorded: YES
|
||||
|
||||
> The raw answer explicitly contains both savings-realism and retention concerns. If a proposed unknown fails semantic admission, the captured userSupportedMeaning and node text should show whether the failure came from answerMeaning loss or from the existing grounding helper's overlap decision.
|
||||
|
||||
## Configured apparatus
|
||||
|
||||
- **Ollama base URL:** `http://192.168.1.111:11434`
|
||||
- **Model:** `qwen-claude:latest`
|
||||
- **Branch:** `feature/user-supported-unknown-admission-v0.15`
|
||||
- **HEAD:** current HEAD of branch at session start
|
||||
|
||||
## Fixed scenario and answer
|
||||
|
||||
**Scenario:** "We are considering relocating the engineering team to reduce operating costs."
|
||||
|
||||
**Answer:** "Before deciding, I need evidence that the projected office savings are realistic and evidence that the move will not materially increase loss of key engineers."
|
||||
|
||||
## Live-call count
|
||||
|
||||
```
|
||||
start: 1
|
||||
update 1: 1
|
||||
total: 2
|
||||
```
|
||||
|
||||
## START
|
||||
|
||||
- **HTTP status:** 200
|
||||
- **Stage:** unknown
|
||||
- **Node count:** 7
|
||||
- **Edge count:** 4
|
||||
- **Selected question:** "What would clarify current detailed breakdown of engineering operating costs by location and category in this situation?"
|
||||
|
||||
## UPDATE 1
|
||||
|
||||
- **HTTP status:** 200
|
||||
- **Stage:** `update_applied`
|
||||
- **Error/validation summary:** null (none)
|
||||
- **Node count:** 9 (+2 new unknowns)
|
||||
- **Edge count:** 6 (+2 new edges)
|
||||
|
||||
## ANSWER MEANING
|
||||
|
||||
- **userSupportedMeaning:** "The decision requires evidence that projected office savings are realistic and evidence that key engineers will not materially leave due to the move."
|
||||
- **possibleInference:** null
|
||||
|
||||
## SAVINGS UNKNOWN
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| id | `n-savings-est` |
|
||||
| label | "Projected office savings from relocation" |
|
||||
| description | "Financial estimate of reduced operational expenses due to the move, needed to decide if the primary goal of lowering operating costs is achievable so that cost reduction justifies the transition." |
|
||||
| combined node text | `projected office savings from relocation financial estimate of reduced operational expenses due to the move needed decide primary goal lowering operating costs achievable cost reduction justifies transition` |
|
||||
|
||||
### Semantic token analysis
|
||||
|
||||
| Metric | Value |
|
||||
|---|---|
|
||||
| userSupportedMeaning tokens (filtered) | decision, requires, evidence, projected, office, savings, are, realistic, key, engineers, will, not, materially, leave, due, move |
|
||||
| node tokens (filtered) | projected, office, savings, relocation, financial, estimate, reduced, operational, expenses, due, move, needed, decide, primary, goal, lowering, operating, costs, achievable, cost, reduction, justifies, transition |
|
||||
| shared tokens | projected, office, savings, due, move |
|
||||
| overlap ratio (shared / candidate) | 0.217 (5/23) |
|
||||
| absolute overlap count | 5 |
|
||||
| expected rawAnswerSupportsUnclassifiedMeaning result | PASS (overlapRatio=0.217 < 0.4 BUT overlappingTokens=5 >= 3) |
|
||||
|
||||
## RETENTION UNKNOWN
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| id | `n-retention-risk` |
|
||||
| label | "Risk of key engineer turnover due to relocation" |
|
||||
| description | "Potential increase in voluntary departure of critical staff following the move, so that workforce stability and project continuity are not compromised despite financial gains, matters because retaining engineering talent is prerequisite to sustaining output." |
|
||||
| combined node text | `risk of key engineer turnover due to relocation potential increase in voluntary departure of critical staff following the move so that workforce stability and project continuity are not compromised despite financial gains matters because retaining engineering talent prerequisite sustaining output` |
|
||||
|
||||
### Semantic token analysis
|
||||
|
||||
| Metric | Value |
|
||||
|---|---|
|
||||
| userSupportedMeaning tokens (filtered) | decision, requires, evidence, projected, office, savings, are, realistic, key, engineers, will, not, materially, leave, due, move |
|
||||
| node tokens (filtered) | risk, key, engineer, turnover, due, relocation, potential, increase, voluntary, departure, critical, staff, following, move, workforce, stability, project, continuity, are, not, compromised, despite, financial, gains, matters, retaining, engineering, talent, prerequisite, sustaining, output |
|
||||
| shared tokens | key, due, move, are, not |
|
||||
| overlap ratio (shared / candidate) | 0.161 (5/31) |
|
||||
| absolute overlap count | 5 |
|
||||
| expected rawAnswerSupportsUnclassifiedMeaning result | PASS (overlapRatio=0.161 < 0.4 BUT overlappingTokens=5 >= 3) |
|
||||
|
||||
## Structural fallback result
|
||||
|
||||
Both nodes have `depends_on` edges to the central state node (`n8g9g4v`). The structural fallback gate (`hasExplicitAnswerDerivedRelationship`) checks for explicit graph linkage between the new unknown and answer-derived nodes from startCase. Both nodes satisfy this via their depends_on wiring.
|
||||
|
||||
- **savings structural fallback:** PASS
|
||||
- **retention structural fallback:** PASS
|
||||
|
||||
## Gate behavior verification
|
||||
|
||||
The v0.15 gate is: `!hasNodeLevelUserSupport(unknownNode) && !hasExplicitAnswerDerivedRelationship(unknownNode)`. Both conditions must be true for rejection. In the live run, neither condition was triggered — both nodes passed at least one sub-gate (in fact both passed the semantic gate first).
|
||||
|
||||
## Classification: C — GATE BEHAVES AS EXPECTED
|
||||
|
||||
Both proposed unknowns were admitted with HTTP 200 at `update_applied`, zero validation errors. The live semantic inputs explain the outcome fully:
|
||||
|
||||
1. **userSupportedMeaning** contains both savings and retention targets semantically — no answer-meaning loss (rules out A).
|
||||
2. **Semantic gate passes for both nodes** via the token-count clause (5 shared tokens >= 3 threshold) despite overlap ratios below 0.4 (rules out B).
|
||||
3. **Live admission outcome matches expected helper result** for both nodes (PASS/PASS → admitted/admitted) (confirms C).
|
||||
4. No code-path mismatch observed: static helper evaluation and actual `hasNodeLevelUserSupport` agree (rules out D).
|
||||
5. **answerMeaning is populated** with a valid `userSupportedMeaning` string — semantic path is available, not unavailable (rules out E).
|
||||
|
||||
## What this experiment established
|
||||
|
||||
1. On a fresh live run through v0.15, the answer meaning gate correctly captures both savings-realism and retention-impact dimensions from a single conjunction-rich user answer.
|
||||
2. The token-count clause of `rawAnswerSupportsUnclassifiedMeaning` (>= 3 shared content tokens) is the operative mechanism for this case — overlap ratios alone (0.16–0.22) would not suffice, but absolute token matches do.
|
||||
3. Structural fallback edges (`depends_on` to the central state node) exist and are valid as a secondary admission path, confirming that both layers work correctly when activated.
|
||||
|
||||
## What this experiment does NOT prove
|
||||
|
||||
1. That the semantic gate passes for answers where shared tokens fall below 3 (e.g., paraphrased savings language).
|
||||
2. That downstream investigation (Update 2+) proceeds without new failures at a different boundary.
|
||||
3. That admission stability holds across repeated runs (7→9 start nodes variance was already observed in 57J.26).
|
||||
4. That the same token-count mechanism works for cross-domain answers with no vocabulary overlap.
|
||||
|
||||
## Production code changed: NO
|
||||
## Prompt changed: NO
|
||||
## Schema changed: NO
|
||||
|
||||
## Temporary instrumentation location
|
||||
|
||||
`scripts/reproduce-multi-turn-investigation.mjs` (temporary additions only, now removed)
|
||||
|
||||
## Temporary instrumentation removed: YES
|
||||
|
||||
## Retries: 0
|
||||
|
||||
## Ollama calls beyond budget: 0
|
||||
|
||||
## Documentation updated
|
||||
|
||||
- `docs/experiment-57j28.md` — created
|
||||
- `docs/current-handoff.md` — appended below
|
||||
|
||||
---
|
||||
Reference in New Issue
Block a user