fix(confidence-engine): define focused relationship contract
This commit is contained in:
@@ -40,7 +40,8 @@ If YES, the next live experiment is one timed/costed OpenAI UI investigation mea
|
||||
- The latest Terra request then exposed an inconsistent root `properties`/`required` contract. The projector now derives `required` after projection from the surviving property keys, and recursive tests verify `properties`, `required`, and `additionalProperties` consistency. Property-less object strictness and initial-reconstruction transport behavior remain preserved.
|
||||
- Current deterministic final-fetch schema remains internally valid, yet the live rejection contradicts it. `CONFIDENCE_ENGINE_EXPERIMENT_TRACE_OPENAI_SCHEMA=1` now emits one safe, server-side structural summary immediately before the OpenAI fetch—no prompt, answer, request body, secret, or model output.
|
||||
- The focused-deconstruction route now emits complementary safe server diagnostics for start, provider success/failure, focused validation, and end status; the OpenAI schema trace remains provider-owned. No user content or secrets are logged.
|
||||
- Focused route suite passes with zero live calls. Next boundary: one fresh focused UI submission with the OpenAI provider and schema-trace flags enabled; capture the single trace, route diagnostics, and OpenAI response, with no Retry.
|
||||
- Canonical focused relationship items are now strict `{ from, to, type }`: all required non-empty strings, free-text `type`, and no `rationale`; `relationships` remains required and may be `[]`. Schema, prompt field names, and validator align; the stale rationale-bearing test fixture was corrected.
|
||||
- Focused and provider deterministic suites pass; the OpenAI projector is unchanged. Zero live calls occurred. Next evidence boundary: live provider compatibility of this same canonical focused contract.
|
||||
|
||||
## Repository checkpoint
|
||||
|
||||
|
||||
@@ -16,7 +16,19 @@ export const focusedDeconstructJsonSchema = {
|
||||
observations: { type: "array", items: { type: "string" } },
|
||||
uncertainties: { type: "array", items: { type: "string" } },
|
||||
assumptions: { type: "array", items: { type: "string" } },
|
||||
relationships: { type: "array", items: { type: "object" } },
|
||||
relationships: {
|
||||
type: "array",
|
||||
items: {
|
||||
type: "object",
|
||||
properties: {
|
||||
from: { type: "string" },
|
||||
to: { type: "string" },
|
||||
type: { type: "string" },
|
||||
},
|
||||
required: ["from", "to", "type"],
|
||||
additionalProperties: false,
|
||||
},
|
||||
},
|
||||
possibleFollowUpQuestions: { type: "array", items: { type: "string" } },
|
||||
},
|
||||
required: FOCUSED_ANSWER_SCHEMA_FIELDS,
|
||||
@@ -115,7 +127,7 @@ Field rules (semantic contract):
|
||||
- assumptions: what unstated proposition does the user's answer itself rely upon for it to make sense? Include only when such a proposition is genuinely attributable to the user's reasoning. The boundary is narrow: attribute only propositions that the user's answer would cease to make sense if they were false. Do NOT import plausible interpretations from the wider investigation context, scenario framing, domain relevance, strategic implications, or model-generated analysis into this field — those belong in uncertainties, relationships (where permitted), or possibleFollowUpQuestions. Do NOT connect a factual statement the user makes to a broader capability or constraint concept unless the user explicitly links them. Example: answering "I only have bank account access" to a question about delegation constraints does NOT assume that "delegation feasibility is contingent upon banking access" — it only states a fact about access, and connecting that fact to delegation feasibility is your own scenario-level inference, not a user-held assumption. If the user's answer does not contain or rely upon an identifiable assumption, return assumptions: []. Do NOT require verbatim copying from the user's answer; paraphrasing is allowed only when the reasoning genuinely relies on it.
|
||||
|
||||
Answer-dependence test: Only attribute an assumption if the user's answer needs that proposition to make sense. If the proposition could be false and the user's answer would still make complete sense, do not attribute it. One observed success in a single concrete example does NOT by itself establish a general rule about competence, readiness, training, safety, transferability, or similar tasks across other work. Do not generalise from one successful example into a broader capability/readiness rule unless the user explicitly or implicitly relies on that broader proposition.
|
||||
- relationships: must connect two distinct propositions that the user's answer itself links. Do not create a relationship by merely restating, reformatting, or relabelling an observation. Co-mentioned facts do not themselves create a relationship. Tentative, speculative, or conditional language must not be promoted into an established relationship. If the answer does not directly establish a relationship, return relationships: [].
|
||||
- relationships: each non-empty item must be { "from": "first proposition", "to": "second distinct proposition", "type": "concise free-text relationship label" }. It must connect two distinct propositions that the user's answer itself links. Do not create a relationship by merely restating, reformatting, or relabelling an observation. Co-mentioned facts do not themselves create a relationship. Tentative, speculative, or conditional language must not be promoted into an established relationship. If the answer does not directly establish a relationship, return relationships: [].
|
||||
- possibleFollowUpQuestions: must be a JSON array containing exactly one string — your single best follow-up question. Example shape: ["one question"]. This question must directly investigate the single uncertainty returned in uncertainties (uncertainties[0] → possibleFollowUpQuestions[0]): one unresolved proposition mapped to one question designed to clarify it. The question must not introduce a second unresolved issue, must not broaden beyond the uncertainty it is meant to resolve, and must not contain more than one investigative step. Do not provide alternatives, a roadmap, or questions that belong after this one has been answered. A later question must be generated only after the current question has been answered and deconstructed. Do not ask about consequences, expansion, requirements, interventions, or other branches until the immediate unresolved relationship has been clarified. Those may become later questions after new evidence is obtained. Ask only what the Engine has earned the right to ask now. Each epistemic step waits its turn — do not combine steps that should happen in sequence across multiple turns: one question that investigates one thing only, never a bundle of future reasoning joined together. Before formulating, check whether the question tests a proposition against the current epistemic state: if an explanation, deficit, dependency, cause, intervention, recommendation, or solution has not been established by prior evidence, phrase the question so it tests whether that proposition is true rather than assuming it — verify the unresolved fact before seeking remedy. Prefer questions that identify what remains unknown, distinguish competing explanations, test whether a suspected factor actually matters, clarify scope, or identify what evidence would change the investigation. Do not jump to implementation details unless the answer has already established that intervention as the relevant next issue. Use ordinary language that a capable person with no specialist vocabulary can understand immediately. If the question needs abstract phrases, management jargon, specialist terminology, or several concepts joined together to express it, break the reasoning down again before returning it. Simple wording of an over-composed idea is still a failure: first ask "what is the smallest thing we actually do not know yet?" then express that one thing simply.
|
||||
- cross-field ownership: preserve who or what owns each proposition. When a statement expresses the user's comfort, willingness, threshold, belief, uncertainty, preference, or judgement, keep it attached to that stance — do not elevate it into an objective requirement, capability fact, or situational constraint.
|
||||
|
||||
@@ -150,6 +162,30 @@ export function validateFocusedDeconstructSchema(result) {
|
||||
}
|
||||
}
|
||||
|
||||
if (!Array.isArray(result.relationships)) {
|
||||
errors.push("relationships must be an array");
|
||||
} else {
|
||||
result.relationships.forEach((relationship, index) => {
|
||||
if (!relationship || typeof relationship !== "object" || Array.isArray(relationship)) {
|
||||
errors.push(`relationships[${index}] must be an object`);
|
||||
return;
|
||||
}
|
||||
const keys = Object.keys(relationship);
|
||||
for (const field of ["from", "to", "type"]) {
|
||||
if (!(field in relationship)) {
|
||||
errors.push(`relationships[${index}] missing required field: ${field}`);
|
||||
} else if (typeof relationship[field] !== "string" || relationship[field].trim().length === 0) {
|
||||
errors.push(`relationships[${index}].${field} must be a non-empty string`);
|
||||
}
|
||||
}
|
||||
for (const field of keys) {
|
||||
if (!["from", "to", "type"].includes(field)) {
|
||||
errors.push(`relationships[${index}] contains unknown field: ${field}`);
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
return errors;
|
||||
}
|
||||
|
||||
|
||||
@@ -7,7 +7,7 @@
|
||||
*/
|
||||
|
||||
import { afterEach, beforeEach, describe, it, expect, vi } from "vitest";
|
||||
import { focusedDeconstructJsonSchema } from "@/lib/graph/focused-investigation";
|
||||
import { focusedDeconstructJsonSchema, validateFocusedDeconstructSchema } from "@/lib/graph/focused-investigation";
|
||||
|
||||
// ── helpers ──────────────────────────────────────────────────────────────
|
||||
|
||||
@@ -20,8 +20,8 @@ function makeMockProvider(inventedTargetNodeId) {
|
||||
uncertainties: ["whether formal docs can capture tacit knowledge"],
|
||||
assumptions: ["documentation is primary mechanism for knowledge transfer"],
|
||||
relationships: [
|
||||
{ from: "founder", to: "processes", type: "holds", rationale: "tacit" },
|
||||
{ from: "ops-context", to: "docs-infra", type: "depends_on", rationale: "formal docs required" },
|
||||
{ from: "founder", to: "processes", type: "holds" },
|
||||
{ from: "ops-context", to: "docs-infra", type: "depends_on" },
|
||||
],
|
||||
possibleFollowUpQuestions: [
|
||||
"What processes does the founder hold tacitly?",
|
||||
@@ -45,6 +45,33 @@ describe("focused-deconstruct targetNodeId identity boundary", () => {
|
||||
vi.doUnmock("@/lib/llm/provider");
|
||||
});
|
||||
|
||||
it("enforces the canonical focused relationship structure", () => {
|
||||
const base = {
|
||||
targetNodeId: "node-id", observations: [], uncertainties: [], assumptions: [],
|
||||
possibleFollowUpQuestions: [],
|
||||
};
|
||||
expect(validateFocusedDeconstructSchema({ ...base, relationships: [] })).toEqual([]);
|
||||
expect(validateFocusedDeconstructSchema({
|
||||
...base,
|
||||
relationships: [{ from: "supplier changed", to: "defect rate increased", type: "associated with" }],
|
||||
})).toEqual([]);
|
||||
|
||||
const invalidRelationships = [
|
||||
"not an array",
|
||||
[{}],
|
||||
[{ from: "a", type: "links" }],
|
||||
[{ from: "a", to: "b" }],
|
||||
[{ from: "", to: "b", type: "links" }],
|
||||
[{ from: "a", to: "", type: "links" }],
|
||||
[{ from: "a", to: "b", type: "" }],
|
||||
[{ from: "a", to: "b", type: "links", rationale: "extra" }],
|
||||
[{ from: "a", to: "b", type: "links", extra: "extra" }],
|
||||
];
|
||||
invalidRelationships.forEach((relationships) => {
|
||||
expect(validateFocusedDeconstructSchema({ ...base, relationships }).length).toBeGreaterThan(0);
|
||||
});
|
||||
});
|
||||
|
||||
it("request targetNodeId overrides model-invented targetNodeId", async () => {
|
||||
const requestTargetNodeId = "nk04xvk"; // original graph node ID
|
||||
const inventedModelId = "invented-model-id";
|
||||
@@ -128,8 +155,8 @@ describe("focused-deconstruct targetNodeId identity boundary", () => {
|
||||
const mockUnc = ["whether formal docs can capture tacit knowledge"];
|
||||
const mockAssm = ["documentation is primary mechanism for knowledge transfer"];
|
||||
const mockRel = [
|
||||
{ from: "founder", to: "processes", type: "holds", rationale: "tacit" },
|
||||
{ from: "ops-context", to: "docs-infra", type: "depends_on", rationale: "formal docs required" },
|
||||
{ from: "founder", to: "processes", type: "holds" },
|
||||
{ from: "ops-context", to: "docs-infra", type: "depends_on" },
|
||||
];
|
||||
const mockFuq = [
|
||||
"What processes does the founder hold tacitly?",
|
||||
|
||||
Reference in New Issue
Block a user