experiment: observe live unknown dimensionality representation
This commit is contained in:
@@ -340,3 +340,7 @@ Tested `{"scenario":"test"}` for exact graph node text and answerability diagnos
|
|||||||
### Experiment 57J.8 — Rejected Answerability Corroboration Candidate
|
### Experiment 57J.8 — Rejected Answerability Corroboration Candidate
|
||||||
|
|
||||||
Candidate commit `60048a5` (reasoning: require corroboration for conjunction compoundness) was reviewed and rejected. It crossed its explicit implementation stop condition: while it fixed the false-positive `"scenario, problem, or data set"` minimal clarification case by making it independently answerable, it also broke a genuinely compound fresh-evidence question (e.g., "What evidence supports the savings estimate **and** what evidence supports the retention assumption?") into independently answerable — because no graph structure existed yet. The candidate violated the rule that Case 1 and Case 2 must both be preserved with existing signals. It has been reverted to production baseline. Production reasoning is restored to pre-v0.14 answerability behaviour. The unresolved design question: surface conjunctions alone are an unreliable proxy for semantic compoundness, but the current graph also lacks sufficient structured information to distinguish every fresh single-concept alternative from every genuinely compound fresh unknown. **Do not resume by reintroducing conjunction corroboration or adding more English regex rules.** Full record in `docs/experiment-57j6.md`.
|
Candidate commit `60048a5` (reasoning: require corroboration for conjunction compoundness) was reviewed and rejected. It crossed its explicit implementation stop condition: while it fixed the false-positive `"scenario, problem, or data set"` minimal clarification case by making it independently answerable, it also broke a genuinely compound fresh-evidence question (e.g., "What evidence supports the savings estimate **and** what evidence supports the retention assumption?") into independently answerable — because no graph structure existed yet. The candidate violated the rule that Case 1 and Case 2 must both be preserved with existing signals. It has been reverted to production baseline. Production reasoning is restored to pre-v0.14 answerability behaviour. The unresolved design question: surface conjunctions alone are an unreliable proxy for semantic compoundness, but the current graph also lacks sufficient structured information to distinguish every fresh single-concept alternative from every genuinely compound fresh unknown. **Do not resume by reintroducing conjunction corroboration or adding more English regex rules.** Full record in `docs/experiment-57j6.md`.
|
||||||
|
|
||||||
|
### Experiment 57J.11 — Live Unknown Dimensionality Representation
|
||||||
|
|
||||||
|
Tested whether a single answer containing two independent evidence dimensions is represented as two separate unknown nodes or collapsed into one compound unknown. **Classification: A — SEPARATE.** Fixed scenario: "We are considering relocating the engineering team to reduce operating costs." Fixed answer: "Before deciding, I need evidence that the projected office savings are realistic and evidence that the move will not materially increase loss of key engineers." One live start + one live update. Update was rejected at `proposal_compatibility` (structural linkage error), but the rejection errors directly revealed two independently named proposed unknown nodes: `n-savings-realism` (Target A) and `n-retention-impact` (Target B). The model did not collapse them into a single compound node. Semantic separability existed in the model proposal before deterministic answerability/decomposition. **What remains unproven:** separation for implicit conjunctions, across models/repeated runs, and whether both nodes survive structural gating (`proposal_compatibility`) or question-selection after graph mutation. Full record in `docs/experiment-57j11.md`. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No production code changed.
|
||||||
|
|||||||
@@ -0,0 +1,120 @@
|
|||||||
|
# Experiment 57J.11 — Live Unknown Dimensionality Representation
|
||||||
|
|
||||||
|
**Date:** 2026-08-10
|
||||||
|
**Branch:** `feature/answerability-corroboration-v0.14`
|
||||||
|
**Status:** PASS (observation complete)
|
||||||
|
**Ollama host/model:** qwen-claude:latest at http://192.168.1.111:11434
|
||||||
|
**Live calls:** 2 (startCase 1 + updateCase 1)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
Answer: *When the user supplies one answer containing two genuinely independent evidence dimensions, does the live `updateCase` model naturally represent them as two separate unknown nodes, or collapse them into one compound unknown?*
|
||||||
|
|
||||||
|
## Fixed scenario and answer
|
||||||
|
|
||||||
|
**Scenario:** "We are considering relocating the engineering team to reduce operating costs."
|
||||||
|
|
||||||
|
**Answer:** "Before deciding, I need evidence that the projected office savings are realistic and evidence that the move will not materially increase loss of key engineers."
|
||||||
|
|
||||||
|
Two intended evidence targets:
|
||||||
|
- **Target A:** Evidence supporting the projected office savings.
|
||||||
|
- **Target B:** Evidence concerning retention/loss of key engineers.
|
||||||
|
|
||||||
|
## Pre-written human expectation (recorded before run)
|
||||||
|
|
||||||
|
> The answer introduces two independently investigable evidence needs. A semantically atomic graph representation would normally preserve them as two separate unresolved unknowns or otherwise represent their separability structurally. A single compound unknown containing both concerns would show that the model is relying on downstream answerability/decomposition to recover the distinction.
|
||||||
|
|
||||||
|
## Pre-written human expectation confirmed: YES
|
||||||
|
|
||||||
|
## Live-call results
|
||||||
|
|
||||||
|
### Start (1 call)
|
||||||
|
- HTTP 200 — success
|
||||||
|
- Stage: `unknown`
|
||||||
|
- Selected question: "What would clarify exact cost differential between current location and proposed destination in this situation?"
|
||||||
|
- Node count: 6 | Edge count: 4
|
||||||
|
|
||||||
|
### Update 1 (1 call)
|
||||||
|
- HTTP 422 — failed at stage `proposal_compatibility`
|
||||||
|
- The model's raw proposal was not returned alongside the rejection; evidence recovered from error messages.
|
||||||
|
|
||||||
|
## Raw proposal evidence (recovered from rejection errors)
|
||||||
|
|
||||||
|
The update response contained these exact error lines identifying proposed unknown node IDs:
|
||||||
|
|
||||||
|
```
|
||||||
|
"New unknown must be explicitly related to an answer-derived node: \"n-savings-realism\""
|
||||||
|
"New unknown must be explicitly related to an answer-derived node: \"n-retention-impact\""
|
||||||
|
```
|
||||||
|
|
||||||
|
Both IDs are independently named — they do not share a compound label or description prefix. They correspond directly to the two intended evidence targets by name alone.
|
||||||
|
|
||||||
|
## New unknown nodes (reconstructed from error IDs)
|
||||||
|
|
||||||
|
### 1. `n-savings-realism`
|
||||||
|
- **id:** n-savings-realism
|
||||||
|
- **label:** inferred → savings-realism
|
||||||
|
- **description:** inferred → concerns projected office savings realism (Target A)
|
||||||
|
- **dependsOn:** not returned (proposal rejected)
|
||||||
|
- **affects:** not returned (proposal rejected)
|
||||||
|
- **parentId:** not returned (proposal rejected)
|
||||||
|
- **childIds:** not returned (proposal rejected)
|
||||||
|
|
||||||
|
### 2. `n-retention-impact`
|
||||||
|
- **id:** n-retention-impact
|
||||||
|
- **label:** inferred → retention-impact
|
||||||
|
- **description:** inferred → concerns move's impact on loss of key engineers / retention (Target B)
|
||||||
|
- **dependsOn:** not returned (proposal rejected)
|
||||||
|
- **affects:** not returned (proposal rejected)
|
||||||
|
- **parentId:** not returned (proposal rejected)
|
||||||
|
- **childIds:** not returned (proposal rejected)
|
||||||
|
|
||||||
|
## Added edges involving new unknowns
|
||||||
|
None retrievable from rejection response.
|
||||||
|
|
||||||
|
## All unknown nodes in resulting graph
|
||||||
|
Graph was not mutated — result equals start graph: `nhuef4z` and `ngwbp0q` only (pre-existing).
|
||||||
|
|
||||||
|
## Classification
|
||||||
|
|
||||||
|
**A — SEPARATE**
|
||||||
|
|
||||||
|
The model created two distinct unknown nodes corresponding to the two intended evidence targets:
|
||||||
|
- `n-savings-realism` → savings target (SEPARATE NODE)
|
||||||
|
- `n-retention-impact` → retention target (SEPARATE NODE)
|
||||||
|
|
||||||
|
Neither node contained both evidence dimensions in its identity. Both were independently named per dimension.
|
||||||
|
|
||||||
|
## Rationale
|
||||||
|
|
||||||
|
The model's raw proposal (before deterministic rejection at `proposal_compatibility`) represented the two independent evidence needs as two distinct unknown node IDs. The naming convention (`n-savings-realism` vs `n-retention-impact`) confirms the semantic distinction was externalized by the model itself — not inferred later by deterministic logic.
|
||||||
|
|
||||||
|
Both nodes were rejected for the same structural reason: they were proposed without explicit linkage to an answer-derived node (the validation rule requires each new unknown to connect via edge to a node that traces back to the user's answer). This is a separate concern from semantic dimensionality.
|
||||||
|
|
||||||
|
## Did semantic separability exist in the model proposal before deterministic answerability/decomposition?
|
||||||
|
**YES** — Two independently named nodes were produced by the model proposal itself.
|
||||||
|
|
||||||
|
## Did downstream deterministic logic have to infer/split the dimensions:
|
||||||
|
**NO** — The model did not produce a compound node requiring downstream splitting.
|
||||||
|
|
||||||
|
## What this experiment established
|
||||||
|
|
||||||
|
- For this fixed scenario/answer, the live `updateCase` model **naturally separates** two independent evidence dimensions into two distinct unknown nodes at the proposal level.
|
||||||
|
- The separation occurs *before* any deterministic answerability or decomposition logic.
|
||||||
|
- A structural gating rule (`proposal_compatibility`: new unknowns must link to answer-derived nodes) can prevent both nodes from entering the graph, but it does not collapse them.
|
||||||
|
|
||||||
|
## What this experiment does NOT prove
|
||||||
|
|
||||||
|
- That separation holds for other answers with different compound structures (e.g., implicit conjunctions, less explicit "and" phrasing).
|
||||||
|
- That the two nodes would survive `proposal_compatibility` in a scenario where answer-derived linkage exists.
|
||||||
|
- That the question-selection or Behaviour Selection modules preserve both dimensions after graph mutation.
|
||||||
|
- That separation holds across models or repeated runs.
|
||||||
|
|
||||||
|
## Production code changed: NO
|
||||||
|
## Prompt changed: NO
|
||||||
|
## Schema changed: NO
|
||||||
|
## Canonical script restored: YES
|
||||||
|
## Retries: 0
|
||||||
|
## Ollama calls beyond budget: 0
|
||||||
Reference in New Issue
Block a user