docs: record rejected answerability corroboration candidate
This commit is contained in:
@@ -336,3 +336,7 @@ Tested whether the cost-reduction answer (unclassified "other") now passes propo
|
||||
### Experiment 57J.2 — Minimal Clarification Answerability Diagnostics
|
||||
|
||||
Tested `{"scenario":"test"}` for exact graph node text and answerability diagnostics. **Classification: B — inconsistent diagnostics.** Single start produced `centralStatement="test"` with one unknown (`nlgonjv`): "The actual scenario, problem description, or data set intended for analysis." The API reported `prerequisiteConceptCount=3`, but only 1 of 7 prerequisite regex signals actually matched (rule 1: `\bproblem\b`). The count of 3 comes from conjunction-based amplification: two instances of "or" in the label+description → `conjunctionCount=2` → formula `Math.max(regex_matches=1, unresolved_deps=0, conjunctions+1=3)` = 3. The diagnostic name is misleading — it reports a maximum across three different amplification strategies (prerequisite regex, unresolved dependencies, conjunctions), not just prerequisite concept signals. No graph-backed question produced. One live Ollama call at qwen-claude:latest on http://192.168.1.111:11434 (27,109 ms). Full results in `docs/experiment-57j2.md`.
|
||||
|
||||
### Experiment 57J.8 — Rejected Answerability Corroboration Candidate
|
||||
|
||||
Candidate commit `60048a5` (reasoning: require corroboration for conjunction compoundness) was reviewed and rejected. It crossed its explicit implementation stop condition: while it fixed the false-positive `"scenario, problem, or data set"` minimal clarification case by making it independently answerable, it also broke a genuinely compound fresh-evidence question (e.g., "What evidence supports the savings estimate **and** what evidence supports the retention assumption?") into independently answerable — because no graph structure existed yet. The candidate violated the rule that Case 1 and Case 2 must both be preserved with existing signals. It has been reverted to production baseline. Production reasoning is restored to pre-v0.14 answerability behaviour. The unresolved design question: surface conjunctions alone are an unreliable proxy for semantic compoundness, but the current graph also lacks sufficient structured information to distinguish every fresh single-concept alternative from every genuinely compound fresh unknown. **Do not resume by reintroducing conjunction corroboration or adding more English regex rules.** Full record in `docs/experiment-57j6.md`.
|
||||
|
||||
@@ -0,0 +1,37 @@
|
||||
# Experiment 57J.6 — Rejected Answerability Corroboration Candidate
|
||||
|
||||
**Status:** REJECTED (not production)
|
||||
|
||||
## Established Defect
|
||||
|
||||
`conjunctionCount + 1` can mistake alternative wording for independent answer dimensions. A label/description pair containing "scenario, problem, or data set" has `conjunctionCount=2`, yielding a minimum score of 3 that inflates its perceived compoundness beyond what the surface grammar warrants.
|
||||
|
||||
## Candidate Attempted
|
||||
|
||||
Commit `60048a5` made conjunction-based evidence require structural corroboration from existing graph edges before treating a question as decomposable. Fresh-evidence questions with "and" needed at least one established relationship edge to trigger decomposition.
|
||||
|
||||
## What Candidate Improved
|
||||
|
||||
The minimal clarification case:
|
||||
|
||||
```
|
||||
The actual scenario, problem description, or data set intended for analysis.
|
||||
```
|
||||
|
||||
Now correctly scored as independently answerable (score=1) rather than falsely flagged as compound (score=3).
|
||||
|
||||
## Why Candidate Was Rejected
|
||||
|
||||
A genuinely compound fresh unknown such as:
|
||||
|
||||
```
|
||||
What evidence supports the savings estimate and what evidence supports the retention assumption?
|
||||
```
|
||||
|
||||
Also became independently answerable because no graph structure existed yet. Without any established relationship edges, the conjunction corroboration gate blocked decomposition of a legitimately compound question.
|
||||
|
||||
The implementation prompt explicitly required stopping if Case 1 (false positive minimal clarification) and Case 2 (true compound preserved) could not both be preserved with existing signals. The candidate crossed that stop condition by sacrificing Case 2 to fix Case 1.
|
||||
|
||||
## Durable Finding
|
||||
|
||||
> Surface grammar is not sufficient evidence of semantic compoundness, but fresh graph state may also be too sparse to establish compoundness structurally. A future refinement must resolve that distinction rather than choosing one failure mode by weakening the other.
|
||||
Reference in New Issue
Block a user