experiment: capture minimal clarification answerability
This commit is contained in:
@@ -332,3 +332,7 @@ Tested whether the cost-reduction answer (unclassified "other") now passes propo
|
||||
### Experiment 57I — No-Structure Relationship Fallback Live Validation
|
||||
|
||||
**Objective:** Validate that commit `4c5666d` (reasoning: suppress explanation question without relationship structure) no longer creates the `Explanation for why...` parent when no meaningful relationship structure exists. **Classification: PASS.** Fixed scenario: "Should I relocate my engineering team from London to Manchester?" with answer "We're looking at this mainly for cost reduction — roughly £2M annual savings on office overhead." Update 1 returned HTTP 200 at stage `update_applied`. Nodes containing "Explanation for why": None. Nodes containing "why these observations differ": None. The reasoning pattern correctly shifted from "explanation" to "decision". The selected next question ("What would clarify team size, seniority levels, and willingness to relocate in this situation?") is grounded in a genuine unresolved relocation issue. No production code changed. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. Branch: `feature/relationship-fallback-v0.13`. Status: PASS.
|
||||
|
||||
### Experiment 57J.2 — Minimal Clarification Answerability Diagnostics
|
||||
|
||||
Tested `{"scenario":"test"}` for exact graph node text and answerability diagnostics. **Classification: B — inconsistent diagnostics.** Single start produced `centralStatement="test"` with one unknown (`nlgonjv`): "The actual scenario, problem description, or data set intended for analysis." The API reported `prerequisiteConceptCount=3`, but only 1 of 7 prerequisite regex signals actually matched (rule 1: `\bproblem\b`). The count of 3 comes from conjunction-based amplification: two instances of "or" in the label+description → `conjunctionCount=2` → formula `Math.max(regex_matches=1, unresolved_deps=0, conjunctions+1=3)` = 3. The diagnostic name is misleading — it reports a maximum across three different amplification strategies (prerequisite regex, unresolved dependencies, conjunctions), not just prerequisite concept signals. No graph-backed question produced. One live Ollama call at qwen-claude:latest on http://192.168.1.111:11434 (27,109 ms). Full results in `docs/experiment-57j2.md`.
|
||||
|
||||
@@ -0,0 +1,110 @@
|
||||
# Experiment 57J.2 — Minimal Clarification Answerability Diagnostics
|
||||
|
||||
**Date:** 2026-08-10
|
||||
**Branch:** `feature/relationship-fallback-v0.13`
|
||||
**HEAD at start:** `90e6623` (experiment: validate relationship fallback live)
|
||||
|
||||
## Objective
|
||||
|
||||
Capture the exact graph node text and answerability diagnostics for `{"scenario":"test"}` — determine what produces the reported `prerequisiteConceptCount`, and which prerequisite regex signals actually match.
|
||||
|
||||
## Fixed Input
|
||||
|
||||
```json
|
||||
{"scenario":"test"}
|
||||
```
|
||||
|
||||
## Live Call Result
|
||||
|
||||
**HTTP status:** 200
|
||||
**Live Ollama calls:** 1 (qwen-claude:latest at http://192.168.1.111:11434, duration: 27,109 ms)
|
||||
|
||||
### Graph
|
||||
|
||||
- **centralStatement:** `"test"`
|
||||
- **activeUnknownNodeId:** `nlgonjv`
|
||||
|
||||
### Exact Active Unknown
|
||||
|
||||
- **id:** `nlgonjv`
|
||||
- **label:** `"The actual scenario, problem description, or data set intended for analysis."`
|
||||
- **description:** `"The actual scenario, problem description, or data set intended for analysis."`
|
||||
- **kind:** `unknown`
|
||||
- **status:** `unknown`
|
||||
|
||||
### Question Diagnostics
|
||||
|
||||
- **reconstructionQuestion:** `"What specific situation, problem, or scenario would you like me to reconstruct and analyze?"`
|
||||
- **reconstructionQuestionAccepted:** `false`
|
||||
- **rejectionReasons:** `["reconstruction_question_not_authoritative", "graph_backed_pipeline_required"]`
|
||||
- **finalGraphBackedQuestion:** `null`
|
||||
- **selectedUnknownNodeId:** `null`
|
||||
- **noQuestionReason:** `"Compatible unresolved candidates remain, but none produced a valid graph-backed question."`
|
||||
|
||||
### Answerability Diagnostics
|
||||
|
||||
- **independentlyAnswerable:** `false`
|
||||
- **prerequisiteConceptCount:** `3`
|
||||
- **decompositionRequired:** `true`
|
||||
- **selectedContainerUnknown:** `nlgonjv`
|
||||
- **selectedChildUnknown:** `null`
|
||||
- **decompositionReason:** `null`
|
||||
|
||||
## Prerequisite Regex Signal Matching
|
||||
|
||||
The active unknown text (label + description) normalised by the code (lowercase, non-alphanumeric → space):
|
||||
|
||||
> `the actual scenario problem description or data set intended for analysis the actual scenario problem description or data set intended for analysis`
|
||||
|
||||
| # | Rule pattern | Result | Matched text |
|
||||
|---|-------------|--------|-------------|
|
||||
| 1 | `\bproblem\b` | **MATCH** | `problem` |
|
||||
| 2 | `\b(audience\|customer\|user\|buyer\|stakeholder\|recipient)\b` | NO MATCH | — |
|
||||
| 3 | `\b(demand\|seek help\|actively look for help)\b` | NO MATCH | — |
|
||||
| 4 | `\b(pay\|willingness to pay\|price\|pricing)\b` | NO MATCH | — |
|
||||
| 5 | `\b(compare\|comparison\|different from\|alternatives\|alternative\|existing alternatives\|existing tools\|better than)\b` | NO MATCH | — |
|
||||
| 6 | `\b(value\|viability\|justified\|business case\|commercial)\b` | NO MATCH | — |
|
||||
| 7 | `\b(feasibility\|technical)\b` | NO MATCH | — |
|
||||
|
||||
**Prerequisite regex matches: 1 of 7** (only rule 1: `problem`)
|
||||
|
||||
## Count Discrepancy Analysis
|
||||
|
||||
The API reports `prerequisiteConceptCount: 3`. The prerequisite regex only matches once.
|
||||
|
||||
However, `countIndependentAnswerDimensions()` computes the final count as:
|
||||
```js
|
||||
Math.max(prerequisiteConceptCount, unresolvedDependencies, conjunctionCount + 1)
|
||||
```
|
||||
|
||||
For this node:
|
||||
- `prerequisiteConceptCount` (regex): **1**
|
||||
- `unresolvedDependencies`: **0** (single unknown with no dependsOn/affects edges)
|
||||
- `conjunctionCount`: **2** (`"or"` appears twice in the normalised label+description)
|
||||
- Final: `Math.max(1, 0, 2+1)` = **3**
|
||||
|
||||
The count of 3 is driven by **conjunction detection**, not prerequisite concept signals. The node's description contains "scenario, problem description, **or** data set" — two instances of "or", yielding conjunctionCount=2, then `+1` per the formula gives 3.
|
||||
|
||||
## Consistency Classification: B — Inconsistent diagnostics
|
||||
|
||||
The reported `prerequisiteConceptCount=3` does not correspond to seven prerequisite concept matches. It is a composite count including conjunction-based amplification. Only 1 of 7 prerequisite regex patterns actually matched; the remaining 2 units come from conjunction counting (`or × 2 → +1`).
|
||||
|
||||
## What This Experiment Established
|
||||
|
||||
- The `{"scenario":"test"}` input produces a minimal graph with `centralStatement="test"` and one unknown node (`nlgonjv`) about the missing scenario context itself.
|
||||
- The active unknown label/description contains "problem" (prerequisite signal) and two instances of "or" (conjunction).
|
||||
- `prerequisiteConceptCount` is computed as `Math.max(regex_matches, unresolved_deps, conjunctions + 1)` — meaning the name is misleading; it reports a maximum across three different amplification strategies, not just prerequisite concept signals.
|
||||
- Reconstruction question was generated but rejected (not authoritative per pipeline design). No graph-backed question produced.
|
||||
|
||||
## What This Experiment Does NOT Prove
|
||||
|
||||
- Whether other scenarios produce different decomposition paths.
|
||||
- Whether conjunction-based amplification is appropriate for this node type (the unknown is about missing context, not a compound inquiry).
|
||||
- Stability of the initial graph across runs.
|
||||
- Whether `prerequisiteConceptCount` as reported should be disaggregated into its constituent signals (regex count vs conjunction count vs unresolved deps).
|
||||
|
||||
## Production code changed: NO
|
||||
## Tests changed: NO
|
||||
## Retries: 0
|
||||
## Ollama calls beyond budget: 0
|
||||
|
||||
Reference in New Issue
Block a user