robbond
956fc2e31e
Two bugs were causing the model to return {"status":"ok"} / {"status":"ready"}
instead of structured reconstruction data, resulting in POST /api/analyse 500:
1. DOUBLE-WRAPPING BUG (lib/llm/provider.js):
generateReconstruction() called buildPrompt(scenario) on input that was
already a fully-built prompt string from analyseScenario(). This wrapped the
v0.1 prompt (~5000+ chars) in another template layer, producing incomprehensible
output that the model could not parse as structured JSON.
Fix: Pass scenario through directly (it is ALREADY a built prompt).
2. MISSING JSON SPEC (prompts/reconstruct-v0.2.md):
The v0.2 prompt template said 'matching the structure exactly' but never
defined what that structure was. The model invented its own field names
(input_classification, reasoning_mode, anchors) with snake_case instead of
camelCase, which failed Zod validation -> 500 errors.
Fix: Added explicit JSON schema section with exact key names, enum values,
and nested structure matching the Zod validation layer.
Additionally:
- Refactored route to use analyseScenario from lib/analysis (centralized)
- Added lib/analysis.js with shared analysis logic
- Updated components to display promptVersion and validation errors
- Added lib/reconstruction/prompt.js v0.1/v0.2 versioning
- Added lib/reconstruction/schema.js v0.2 Zod schemas
- Added debug tool scripts, evaluation results, and comparison findings
Confidence Engine
An experimental prototype that tests whether an LLM can build and maintain an evidence-based reconstruction of a situation over multiple turns.
Purpose
This is Milestone 1 — a technical vertical slice only. It demonstrates:
- Sending a scenario to a local Ollama model via a Next.js server route
- Receiving structured JSON output
- Validating the result with Zod schemas
- Displaying the reconstruction and diagnostic information in a plain UI
Prerequisites
- Node.js 18+ (LTS recommended)
- npm (or equivalent package manager)
- Ollama installed and running on your local network, with a model pulled (e.g.,
ollama pull llama3)
Installation
cd confidence-engine
npm install
cp .env.example .env.local
Edit .env.local and set:
OLLAMA_BASE_URL— your Ollama server address (e.g.,http://192.168.1.100:11434)OLLAMA_MODEL— the model name (e.g.,llama3)
Development Commands
npm run dev # Start development server on localhost:3000
npm run build # Production build
npm run start # Run production server
npm run lint # ESLint check
Testing Commands
npm test # Run all tests (one-shot)
npm run test:watch # Run tests in watch mode
Tests mock the Ollama network request. No real Ollama server is needed to run them.
Verifying Ollama Connectivity
- Start the dev server:
npm run dev - Open http://localhost:3000/api/health
- You should see JSON with
"reachable": trueand your model name
Current Limitations
- Single-turn only — no conversation memory or multi-turn dialogue
- No persistence — results are not saved between requests
- Ollama only — the provider abstraction exists but only Ollama is implemented
- JSON mode reliance — output quality depends on the model's ability to produce valid structured JSON
- No question generation — no follow-up questions or uncertainty resolution yet
- No real-time streaming — waits for full response before displaying results
- Plain UI — no animations, theming, or responsive polish beyond basic layout
Deliberately Not Implemented
- Authentication / user accounts
- Billing / rate limiting
- Database or file storage
- Vector databases or embeddings
- Deployment configuration (Docker, Vercel, etc.)
- External cloud LLM providers (OpenAI, Anthropic, etc.)
- Agent frameworks or tool use
- Complex state management (Zustand, Redux, etc.)
- Multi-turn conversation history
- Question generation or ranking
Architecture Notes
The application uses a provider abstraction in lib/llm/provider.js. The server routes call the provider, which hides the Ollama-specific implementation. To add a new provider, implement the same interface:
{
generateReconstruction(scenario, modelName): Promise<Reconstruction>
}
```READMEEOF