experiment: capture minimal clarification answerability

2026-08-10 14:26:07 +01:00
parent 90e662397f
commit dcefb36f4d
2 changed files with 114 additions and 0 deletions
@@ -0,0 +1,110 @@
+# Experiment 57J.2 — Minimal Clarification Answerability Diagnostics
+
+**Date:** 2026-08-10
+**Branch:** `feature/relationship-fallback-v0.13`
+**HEAD at start:** `90e6623` (experiment: validate relationship fallback live)
+
+## Objective
+
+Capture the exact graph node text and answerability diagnostics for `{"scenario":"test"}` — determine what produces the reported `prerequisiteConceptCount`, and which prerequisite regex signals actually match.
+
+## Fixed Input
+
+```json
+{"scenario":"test"}
+```
+
+## Live Call Result
+
+**HTTP status:** 200
+**Live Ollama calls:** 1 (qwen-claude:latest at http://192.168.1.111:11434, duration: 27,109 ms)
+
+### Graph
+
+- **centralStatement:** `"test"`
+- **activeUnknownNodeId:** `nlgonjv`
+
+### Exact Active Unknown
+
+- **id:** `nlgonjv`
+- **label:** `"The actual scenario, problem description, or data set intended for analysis."`
+- **description:** `"The actual scenario, problem description, or data set intended for analysis."`
+- **kind:** `unknown`
+- **status:** `unknown`
+
+### Question Diagnostics
+
+- **reconstructionQuestion:** `"What specific situation, problem, or scenario would you like me to reconstruct and analyze?"`
+- **reconstructionQuestionAccepted:** `false`
+- **rejectionReasons:** `["reconstruction_question_not_authoritative", "graph_backed_pipeline_required"]`
+- **finalGraphBackedQuestion:** `null`
+- **selectedUnknownNodeId:** `null`
+- **noQuestionReason:** `"Compatible unresolved candidates remain, but none produced a valid graph-backed question."`
+
+### Answerability Diagnostics
+
+- **independentlyAnswerable:** `false`
+- **prerequisiteConceptCount:** `3`
+- **decompositionRequired:** `true`
+- **selectedContainerUnknown:** `nlgonjv`
+- **selectedChildUnknown:** `null`
+- **decompositionReason:** `null`
+
+## Prerequisite Regex Signal Matching
+
+The active unknown text (label + description) normalised by the code (lowercase, non-alphanumeric → space):
+
+> `the actual scenario problem description or data set intended for analysis the actual scenario problem description or data set intended for analysis`
+
+| # | Rule pattern | Result | Matched text |
+|---|-------------|--------|-------------|
+| 1 | `\bproblem\b` | **MATCH** | `problem` |
+| 2 | `\b(audience\|customer\|user\|buyer\|stakeholder\|recipient)\b` | NO MATCH | — |
+| 3 | `\b(demand\|seek help\|actively look for help)\b` | NO MATCH | — |
+| 4 | `\b(pay\|willingness to pay\|price\|pricing)\b` | NO MATCH | — |
+| 5 | `\b(compare\|comparison\|different from\|alternatives\|alternative\|existing alternatives\|existing tools\|better than)\b` | NO MATCH | — |
+| 6 | `\b(value\|viability\|justified\|business case\|commercial)\b` | NO MATCH | — |
+| 7 | `\b(feasibility\|technical)\b` | NO MATCH | — |
+
+**Prerequisite regex matches: 1 of 7** (only rule 1: `problem`)
+
+## Count Discrepancy Analysis
+
+The API reports `prerequisiteConceptCount: 3`. The prerequisite regex only matches once.
+
+However, `countIndependentAnswerDimensions()` computes the final count as:
+```js
+Math.max(prerequisiteConceptCount, unresolvedDependencies, conjunctionCount + 1)
+```
+
+For this node:
+- `prerequisiteConceptCount` (regex): **1**
+- `unresolvedDependencies`: **0** (single unknown with no dependsOn/affects edges)
+- `conjunctionCount`: **2** (`"or"` appears twice in the normalised label+description)
+- Final: `Math.max(1, 0, 2+1)` = **3**
+
+The count of 3 is driven by **conjunction detection**, not prerequisite concept signals. The node's description contains "scenario, problem description, **or** data set" — two instances of "or", yielding conjunctionCount=2, then `+1` per the formula gives 3.
+
+## Consistency Classification: B — Inconsistent diagnostics
+
+The reported `prerequisiteConceptCount=3` does not correspond to seven prerequisite concept matches. It is a composite count including conjunction-based amplification. Only 1 of 7 prerequisite regex patterns actually matched; the remaining 2 units come from conjunction counting (`or × 2 → +1`).
+
+## What This Experiment Established
+
+- The `{"scenario":"test"}` input produces a minimal graph with `centralStatement="test"` and one unknown node (`nlgonjv`) about the missing scenario context itself.
+- The active unknown label/description contains "problem" (prerequisite signal) and two instances of "or" (conjunction).
+- `prerequisiteConceptCount` is computed as `Math.max(regex_matches, unresolved_deps, conjunctions + 1)` — meaning the name is misleading; it reports a maximum across three different amplification strategies, not just prerequisite concept signals.
+- Reconstruction question was generated but rejected (not authoritative per pipeline design). No graph-backed question produced.
+
+## What This Experiment Does NOT Prove
+
+- Whether other scenarios produce different decomposition paths.
+- Whether conjunction-based amplification is appropriate for this node type (the unknown is about missing context, not a compound inquiry).
+- Stability of the initial graph across runs.
+- Whether `prerequisiteConceptCount` as reported should be disaggregated into its constituent signals (regex count vs conjunction count vs unresolved deps).
+
+## Production code changed: NO
+## Tests changed: NO
+## Retries: 0
+## Ollama calls beyond budget: 0
+