experiment: capture minimal clarification answerability

This commit is contained in:
2026-08-10 14:26:07 +01:00
parent 90e662397f
commit dcefb36f4d
2 changed files with 114 additions and 0 deletions
+4
View File
@@ -332,3 +332,7 @@ Tested whether the cost-reduction answer (unclassified "other") now passes propo
### Experiment 57I — No-Structure Relationship Fallback Live Validation
**Objective:** Validate that commit `4c5666d` (reasoning: suppress explanation question without relationship structure) no longer creates the `Explanation for why...` parent when no meaningful relationship structure exists. **Classification: PASS.** Fixed scenario: "Should I relocate my engineering team from London to Manchester?" with answer "We're looking at this mainly for cost reduction — roughly £2M annual savings on office overhead." Update 1 returned HTTP 200 at stage `update_applied`. Nodes containing "Explanation for why": None. Nodes containing "why these observations differ": None. The reasoning pattern correctly shifted from "explanation" to "decision". The selected next question ("What would clarify team size, seniority levels, and willingness to relocate in this situation?") is grounded in a genuine unresolved relocation issue. No production code changed. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. Branch: `feature/relationship-fallback-v0.13`. Status: PASS.
### Experiment 57J.2 — Minimal Clarification Answerability Diagnostics
Tested `{"scenario":"test"}` for exact graph node text and answerability diagnostics. **Classification: B — inconsistent diagnostics.** Single start produced `centralStatement="test"` with one unknown (`nlgonjv`): "The actual scenario, problem description, or data set intended for analysis." The API reported `prerequisiteConceptCount=3`, but only 1 of 7 prerequisite regex signals actually matched (rule 1: `\bproblem\b`). The count of 3 comes from conjunction-based amplification: two instances of "or" in the label+description → `conjunctionCount=2` → formula `Math.max(regex_matches=1, unresolved_deps=0, conjunctions+1=3)` = 3. The diagnostic name is misleading — it reports a maximum across three different amplification strategies (prerequisite regex, unresolved dependencies, conjunctions), not just prerequisite concept signals. No graph-backed question produced. One live Ollama call at qwen-claude:latest on http://192.168.1.111:11434 (27,109 ms). Full results in `docs/experiment-57j2.md`.
+110
View File
@@ -0,0 +1,110 @@
# Experiment 57J.2 — Minimal Clarification Answerability Diagnostics
**Date:** 2026-08-10
**Branch:** `feature/relationship-fallback-v0.13`
**HEAD at start:** `90e6623` (experiment: validate relationship fallback live)
## Objective
Capture the exact graph node text and answerability diagnostics for `{"scenario":"test"}` — determine what produces the reported `prerequisiteConceptCount`, and which prerequisite regex signals actually match.
## Fixed Input
```json
{"scenario":"test"}
```
## Live Call Result
**HTTP status:** 200
**Live Ollama calls:** 1 (qwen-claude:latest at http://192.168.1.111:11434, duration: 27,109 ms)
### Graph
- **centralStatement:** `"test"`
- **activeUnknownNodeId:** `nlgonjv`
### Exact Active Unknown
- **id:** `nlgonjv`
- **label:** `"The actual scenario, problem description, or data set intended for analysis."`
- **description:** `"The actual scenario, problem description, or data set intended for analysis."`
- **kind:** `unknown`
- **status:** `unknown`
### Question Diagnostics
- **reconstructionQuestion:** `"What specific situation, problem, or scenario would you like me to reconstruct and analyze?"`
- **reconstructionQuestionAccepted:** `false`
- **rejectionReasons:** `["reconstruction_question_not_authoritative", "graph_backed_pipeline_required"]`
- **finalGraphBackedQuestion:** `null`
- **selectedUnknownNodeId:** `null`
- **noQuestionReason:** `"Compatible unresolved candidates remain, but none produced a valid graph-backed question."`
### Answerability Diagnostics
- **independentlyAnswerable:** `false`
- **prerequisiteConceptCount:** `3`
- **decompositionRequired:** `true`
- **selectedContainerUnknown:** `nlgonjv`
- **selectedChildUnknown:** `null`
- **decompositionReason:** `null`
## Prerequisite Regex Signal Matching
The active unknown text (label + description) normalised by the code (lowercase, non-alphanumeric → space):
> `the actual scenario problem description or data set intended for analysis the actual scenario problem description or data set intended for analysis`
| # | Rule pattern | Result | Matched text |
|---|-------------|--------|-------------|
| 1 | `\bproblem\b` | **MATCH** | `problem` |
| 2 | `\b(audience\|customer\|user\|buyer\|stakeholder\|recipient)\b` | NO MATCH | — |
| 3 | `\b(demand\|seek help\|actively look for help)\b` | NO MATCH | — |
| 4 | `\b(pay\|willingness to pay\|price\|pricing)\b` | NO MATCH | — |
| 5 | `\b(compare\|comparison\|different from\|alternatives\|alternative\|existing alternatives\|existing tools\|better than)\b` | NO MATCH | — |
| 6 | `\b(value\|viability\|justified\|business case\|commercial)\b` | NO MATCH | — |
| 7 | `\b(feasibility\|technical)\b` | NO MATCH | — |
**Prerequisite regex matches: 1 of 7** (only rule 1: `problem`)
## Count Discrepancy Analysis
The API reports `prerequisiteConceptCount: 3`. The prerequisite regex only matches once.
However, `countIndependentAnswerDimensions()` computes the final count as:
```js
Math.max(prerequisiteConceptCount, unresolvedDependencies, conjunctionCount + 1)
```
For this node:
- `prerequisiteConceptCount` (regex): **1**
- `unresolvedDependencies`: **0** (single unknown with no dependsOn/affects edges)
- `conjunctionCount`: **2** (`"or"` appears twice in the normalised label+description)
- Final: `Math.max(1, 0, 2+1)` = **3**
The count of 3 is driven by **conjunction detection**, not prerequisite concept signals. The node's description contains "scenario, problem description, **or** data set" — two instances of "or", yielding conjunctionCount=2, then `+1` per the formula gives 3.
## Consistency Classification: B — Inconsistent diagnostics
The reported `prerequisiteConceptCount=3` does not correspond to seven prerequisite concept matches. It is a composite count including conjunction-based amplification. Only 1 of 7 prerequisite regex patterns actually matched; the remaining 2 units come from conjunction counting (`or × 2 → +1`).
## What This Experiment Established
- The `{"scenario":"test"}` input produces a minimal graph with `centralStatement="test"` and one unknown node (`nlgonjv`) about the missing scenario context itself.
- The active unknown label/description contains "problem" (prerequisite signal) and two instances of "or" (conjunction).
- `prerequisiteConceptCount` is computed as `Math.max(regex_matches, unresolved_deps, conjunctions + 1)` — meaning the name is misleading; it reports a maximum across three different amplification strategies, not just prerequisite concept signals.
- Reconstruction question was generated but rejected (not authoritative per pipeline design). No graph-backed question produced.
## What This Experiment Does NOT Prove
- Whether other scenarios produce different decomposition paths.
- Whether conjunction-based amplification is appropriate for this node type (the unknown is about missing context, not a compound inquiry).
- Stability of the initial graph across runs.
- Whether `prerequisiteConceptCount` as reported should be disaggregated into its constituent signals (regex count vs conjunction count vs unresolved deps).
## Production code changed: NO
## Tests changed: NO
## Retries: 0
## Ollama calls beyond budget: 0