111 lines
5.1 KiB
Markdown
111 lines
5.1 KiB
Markdown
# Experiment 57J.2 — Minimal Clarification Answerability Diagnostics
|
||
|
||
**Date:** 2026-08-10
|
||
**Branch:** `feature/relationship-fallback-v0.13`
|
||
**HEAD at start:** `90e6623` (experiment: validate relationship fallback live)
|
||
|
||
## Objective
|
||
|
||
Capture the exact graph node text and answerability diagnostics for `{"scenario":"test"}` — determine what produces the reported `prerequisiteConceptCount`, and which prerequisite regex signals actually match.
|
||
|
||
## Fixed Input
|
||
|
||
```json
|
||
{"scenario":"test"}
|
||
```
|
||
|
||
## Live Call Result
|
||
|
||
**HTTP status:** 200
|
||
**Live Ollama calls:** 1 (qwen-claude:latest at http://192.168.1.111:11434, duration: 27,109 ms)
|
||
|
||
### Graph
|
||
|
||
- **centralStatement:** `"test"`
|
||
- **activeUnknownNodeId:** `nlgonjv`
|
||
|
||
### Exact Active Unknown
|
||
|
||
- **id:** `nlgonjv`
|
||
- **label:** `"The actual scenario, problem description, or data set intended for analysis."`
|
||
- **description:** `"The actual scenario, problem description, or data set intended for analysis."`
|
||
- **kind:** `unknown`
|
||
- **status:** `unknown`
|
||
|
||
### Question Diagnostics
|
||
|
||
- **reconstructionQuestion:** `"What specific situation, problem, or scenario would you like me to reconstruct and analyze?"`
|
||
- **reconstructionQuestionAccepted:** `false`
|
||
- **rejectionReasons:** `["reconstruction_question_not_authoritative", "graph_backed_pipeline_required"]`
|
||
- **finalGraphBackedQuestion:** `null`
|
||
- **selectedUnknownNodeId:** `null`
|
||
- **noQuestionReason:** `"Compatible unresolved candidates remain, but none produced a valid graph-backed question."`
|
||
|
||
### Answerability Diagnostics
|
||
|
||
- **independentlyAnswerable:** `false`
|
||
- **prerequisiteConceptCount:** `3`
|
||
- **decompositionRequired:** `true`
|
||
- **selectedContainerUnknown:** `nlgonjv`
|
||
- **selectedChildUnknown:** `null`
|
||
- **decompositionReason:** `null`
|
||
|
||
## Prerequisite Regex Signal Matching
|
||
|
||
The active unknown text (label + description) normalised by the code (lowercase, non-alphanumeric → space):
|
||
|
||
> `the actual scenario problem description or data set intended for analysis the actual scenario problem description or data set intended for analysis`
|
||
|
||
| # | Rule pattern | Result | Matched text |
|
||
|---|-------------|--------|-------------|
|
||
| 1 | `\bproblem\b` | **MATCH** | `problem` |
|
||
| 2 | `\b(audience\|customer\|user\|buyer\|stakeholder\|recipient)\b` | NO MATCH | — |
|
||
| 3 | `\b(demand\|seek help\|actively look for help)\b` | NO MATCH | — |
|
||
| 4 | `\b(pay\|willingness to pay\|price\|pricing)\b` | NO MATCH | — |
|
||
| 5 | `\b(compare\|comparison\|different from\|alternatives\|alternative\|existing alternatives\|existing tools\|better than)\b` | NO MATCH | — |
|
||
| 6 | `\b(value\|viability\|justified\|business case\|commercial)\b` | NO MATCH | — |
|
||
| 7 | `\b(feasibility\|technical)\b` | NO MATCH | — |
|
||
|
||
**Prerequisite regex matches: 1 of 7** (only rule 1: `problem`)
|
||
|
||
## Count Discrepancy Analysis
|
||
|
||
The API reports `prerequisiteConceptCount: 3`. The prerequisite regex only matches once.
|
||
|
||
However, `countIndependentAnswerDimensions()` computes the final count as:
|
||
```js
|
||
Math.max(prerequisiteConceptCount, unresolvedDependencies, conjunctionCount + 1)
|
||
```
|
||
|
||
For this node:
|
||
- `prerequisiteConceptCount` (regex): **1**
|
||
- `unresolvedDependencies`: **0** (single unknown with no dependsOn/affects edges)
|
||
- `conjunctionCount`: **2** (`"or"` appears twice in the normalised label+description)
|
||
- Final: `Math.max(1, 0, 2+1)` = **3**
|
||
|
||
The count of 3 is driven by **conjunction detection**, not prerequisite concept signals. The node's description contains "scenario, problem description, **or** data set" — two instances of "or", yielding conjunctionCount=2, then `+1` per the formula gives 3.
|
||
|
||
## Consistency Classification: B — Inconsistent diagnostics
|
||
|
||
The reported `prerequisiteConceptCount=3` does not correspond to seven prerequisite concept matches. It is a composite count including conjunction-based amplification. Only 1 of 7 prerequisite regex patterns actually matched; the remaining 2 units come from conjunction counting (`or × 2 → +1`).
|
||
|
||
## What This Experiment Established
|
||
|
||
- The `{"scenario":"test"}` input produces a minimal graph with `centralStatement="test"` and one unknown node (`nlgonjv`) about the missing scenario context itself.
|
||
- The active unknown label/description contains "problem" (prerequisite signal) and two instances of "or" (conjunction).
|
||
- `prerequisiteConceptCount` is computed as `Math.max(regex_matches, unresolved_deps, conjunctions + 1)` — meaning the name is misleading; it reports a maximum across three different amplification strategies, not just prerequisite concept signals.
|
||
- Reconstruction question was generated but rejected (not authoritative per pipeline design). No graph-backed question produced.
|
||
|
||
## What This Experiment Does NOT Prove
|
||
|
||
- Whether other scenarios produce different decomposition paths.
|
||
- Whether conjunction-based amplification is appropriate for this node type (the unknown is about missing context, not a compound inquiry).
|
||
- Stability of the initial graph across runs.
|
||
- Whether `prerequisiteConceptCount` as reported should be disaggregated into its constituent signals (regex count vs conjunction count vs unresolved deps).
|
||
|
||
## Production code changed: NO
|
||
## Tests changed: NO
|
||
## Retries: 0
|
||
## Ollama calls beyond budget: 0
|
||
|