diff --git a/docs/current-handoff.md b/docs/current-handoff.md index 36fa560..bd41c1f 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -332,3 +332,7 @@ Tested whether the cost-reduction answer (unclassified "other") now passes propo ### Experiment 57I — No-Structure Relationship Fallback Live Validation **Objective:** Validate that commit `4c5666d` (reasoning: suppress explanation question without relationship structure) no longer creates the `Explanation for why...` parent when no meaningful relationship structure exists. **Classification: PASS.** Fixed scenario: "Should I relocate my engineering team from London to Manchester?" with answer "We're looking at this mainly for cost reduction — roughly £2M annual savings on office overhead." Update 1 returned HTTP 200 at stage `update_applied`. Nodes containing "Explanation for why": None. Nodes containing "why these observations differ": None. The reasoning pattern correctly shifted from "explanation" to "decision". The selected next question ("What would clarify team size, seniority levels, and willingness to relocate in this situation?") is grounded in a genuine unresolved relocation issue. No production code changed. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. Branch: `feature/relationship-fallback-v0.13`. Status: PASS. + +### Experiment 57J.2 — Minimal Clarification Answerability Diagnostics + +Tested `{"scenario":"test"}` for exact graph node text and answerability diagnostics. **Classification: B — inconsistent diagnostics.** Single start produced `centralStatement="test"` with one unknown (`nlgonjv`): "The actual scenario, problem description, or data set intended for analysis." The API reported `prerequisiteConceptCount=3`, but only 1 of 7 prerequisite regex signals actually matched (rule 1: `\bproblem\b`). The count of 3 comes from conjunction-based amplification: two instances of "or" in the label+description → `conjunctionCount=2` → formula `Math.max(regex_matches=1, unresolved_deps=0, conjunctions+1=3)` = 3. The diagnostic name is misleading — it reports a maximum across three different amplification strategies (prerequisite regex, unresolved dependencies, conjunctions), not just prerequisite concept signals. No graph-backed question produced. One live Ollama call at qwen-claude:latest on http://192.168.1.111:11434 (27,109 ms). Full results in `docs/experiment-57j2.md`. diff --git a/docs/experiment-57j2.md b/docs/experiment-57j2.md new file mode 100644 index 0000000..b444ee9 --- /dev/null +++ b/docs/experiment-57j2.md @@ -0,0 +1,110 @@ +# Experiment 57J.2 — Minimal Clarification Answerability Diagnostics + +**Date:** 2026-08-10 +**Branch:** `feature/relationship-fallback-v0.13` +**HEAD at start:** `90e6623` (experiment: validate relationship fallback live) + +## Objective + +Capture the exact graph node text and answerability diagnostics for `{"scenario":"test"}` — determine what produces the reported `prerequisiteConceptCount`, and which prerequisite regex signals actually match. + +## Fixed Input + +```json +{"scenario":"test"} +``` + +## Live Call Result + +**HTTP status:** 200 +**Live Ollama calls:** 1 (qwen-claude:latest at http://192.168.1.111:11434, duration: 27,109 ms) + +### Graph + +- **centralStatement:** `"test"` +- **activeUnknownNodeId:** `nlgonjv` + +### Exact Active Unknown + +- **id:** `nlgonjv` +- **label:** `"The actual scenario, problem description, or data set intended for analysis."` +- **description:** `"The actual scenario, problem description, or data set intended for analysis."` +- **kind:** `unknown` +- **status:** `unknown` + +### Question Diagnostics + +- **reconstructionQuestion:** `"What specific situation, problem, or scenario would you like me to reconstruct and analyze?"` +- **reconstructionQuestionAccepted:** `false` +- **rejectionReasons:** `["reconstruction_question_not_authoritative", "graph_backed_pipeline_required"]` +- **finalGraphBackedQuestion:** `null` +- **selectedUnknownNodeId:** `null` +- **noQuestionReason:** `"Compatible unresolved candidates remain, but none produced a valid graph-backed question."` + +### Answerability Diagnostics + +- **independentlyAnswerable:** `false` +- **prerequisiteConceptCount:** `3` +- **decompositionRequired:** `true` +- **selectedContainerUnknown:** `nlgonjv` +- **selectedChildUnknown:** `null` +- **decompositionReason:** `null` + +## Prerequisite Regex Signal Matching + +The active unknown text (label + description) normalised by the code (lowercase, non-alphanumeric → space): + +> `the actual scenario problem description or data set intended for analysis the actual scenario problem description or data set intended for analysis` + +| # | Rule pattern | Result | Matched text | +|---|-------------|--------|-------------| +| 1 | `\bproblem\b` | **MATCH** | `problem` | +| 2 | `\b(audience\|customer\|user\|buyer\|stakeholder\|recipient)\b` | NO MATCH | — | +| 3 | `\b(demand\|seek help\|actively look for help)\b` | NO MATCH | — | +| 4 | `\b(pay\|willingness to pay\|price\|pricing)\b` | NO MATCH | — | +| 5 | `\b(compare\|comparison\|different from\|alternatives\|alternative\|existing alternatives\|existing tools\|better than)\b` | NO MATCH | — | +| 6 | `\b(value\|viability\|justified\|business case\|commercial)\b` | NO MATCH | — | +| 7 | `\b(feasibility\|technical)\b` | NO MATCH | — | + +**Prerequisite regex matches: 1 of 7** (only rule 1: `problem`) + +## Count Discrepancy Analysis + +The API reports `prerequisiteConceptCount: 3`. The prerequisite regex only matches once. + +However, `countIndependentAnswerDimensions()` computes the final count as: +```js +Math.max(prerequisiteConceptCount, unresolvedDependencies, conjunctionCount + 1) +``` + +For this node: +- `prerequisiteConceptCount` (regex): **1** +- `unresolvedDependencies`: **0** (single unknown with no dependsOn/affects edges) +- `conjunctionCount`: **2** (`"or"` appears twice in the normalised label+description) +- Final: `Math.max(1, 0, 2+1)` = **3** + +The count of 3 is driven by **conjunction detection**, not prerequisite concept signals. The node's description contains "scenario, problem description, **or** data set" — two instances of "or", yielding conjunctionCount=2, then `+1` per the formula gives 3. + +## Consistency Classification: B — Inconsistent diagnostics + +The reported `prerequisiteConceptCount=3` does not correspond to seven prerequisite concept matches. It is a composite count including conjunction-based amplification. Only 1 of 7 prerequisite regex patterns actually matched; the remaining 2 units come from conjunction counting (`or × 2 → +1`). + +## What This Experiment Established + +- The `{"scenario":"test"}` input produces a minimal graph with `centralStatement="test"` and one unknown node (`nlgonjv`) about the missing scenario context itself. +- The active unknown label/description contains "problem" (prerequisite signal) and two instances of "or" (conjunction). +- `prerequisiteConceptCount` is computed as `Math.max(regex_matches, unresolved_deps, conjunctions + 1)` — meaning the name is misleading; it reports a maximum across three different amplification strategies, not just prerequisite concept signals. +- Reconstruction question was generated but rejected (not authoritative per pipeline design). No graph-backed question produced. + +## What This Experiment Does NOT Prove + +- Whether other scenarios produce different decomposition paths. +- Whether conjunction-based amplification is appropriate for this node type (the unknown is about missing context, not a compound inquiry). +- Stability of the initial graph across runs. +- Whether `prerequisiteConceptCount` as reported should be disaggregated into its constituent signals (regex count vs conjunction count vs unresolved deps). + +## Production code changed: NO +## Tests changed: NO +## Retries: 0 +## Ollama calls beyond budget: 0 +