From 2dc1cb0fe64dd0c87ca894fbdd2c94b2f0a62edf Mon Sep 17 00:00:00 2001 From: robbond Date: Fri, 11 Sep 2026 10:34:53 +0100 Subject: [PATCH] docs(confidence-engine): record tester readiness findings --- docs/current-handoff.md | 99 +++++++++++++++++++++++++++++++++++ docs/current-project-state.md | 36 +++++++++++-- 2 files changed, 132 insertions(+), 3 deletions(-) diff --git a/docs/current-handoff.md b/docs/current-handoff.md index 9897939..4b372f0 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -409,3 +409,102 @@ Previous focused-deconstruction semantic runs before the plumbing fix remain **i 2. Initial-reconstruction schema was supplied to focused-deconstruction call instead of its own contract These are recorded as known contamination in the v0.61 archive chapter (`docs/design-evolution/ch19/initial-decomposition-v0.61.md`). The plumbing fix is complete and verified (48/48 tests). + +## Tester-Readiness Findings — Recorded for Session Continuity + +### Tester-readiness position + +``` +controlled external testing: GO + +known pre-tester security blockers: none identified + +next uncertainty worth reducing: real first-time-user product value, trust and independent usability +``` + +Do not claim commercial launch readiness, general production scalability, security certification or proven product-market fit. + +### First external-user experiment + +**Primary question:** Can a first-time user, without Rob guiding the investigation, use Confidence Engine to reach a Current Understanding that they regard as materially better than the way they framed the situation at the start? + +**Tester evidence should prioritise:** +- did understanding materially improve? +- did they trust the changed understanding? +- could they reach it without Rob's help? +- was the experience usable? + +**Post-use discovery questions (do not seed categories or mention a mental-health interpretation):** +- Who do you think this would be useful for? +- What kinds of situations would you use this for? +- Is there a situation in your own life or work where you can imagine coming back and using this? + +### Landing-page positioning + +- Current landing wording is deliberately unchanged for the first cohort. +- It may naturally be interpreted broadly, potentially including mental-health/anxiety-adjacent situations. +- This is an observation to learn from, not currently a defect. +- Use unprompted tester responses to learn what category/audience/use cases users believe Confidence Engine belongs to. +- Do not reposition before this evidence exists. + +### Synthetic-user testing + +- AI-driven first-time-user journeys may be useful as a pre-human stress test. +- Preferred conceptual separation: **synthetic first-time user → real Confidence Engine journey → independent evaluator**. +- Synthetic testing can expose reasoning/UX/systematic failures. +- It does NOT replace human evidence about genuine trust, changed understanding, usefulness, repeat use or willingness to return. +- No synthetic-testing implementation is currently required before human testing. + +### Reasoning concurrency + +- Current CE application has **no** server-side per-user reasoning concurrency protection. +- No CE-level global reasoning concurrency limit. +- Multiple tabs/users can reach provider concurrently. +- Ollama/provider concurrency and queue behaviour are not controlled by CE repository code. +- Do not invent queue/latency behaviour from the 300-second request timeout. +- Controlled cohort of roughly 5–10 invited testers: **not currently a release blocker**. +- Observe actual behaviour before designing queue/semaphore infrastructure. +- Before wider/public access, reasoning-resource concurrency protection should be reconsidered. + +### Rate limiting + +- No application-level per-user reasoning rate limit currently exists. +- Generic API rate limiting is not required for the controlled tester cohort. +- Wider/public access should revisit reasoning-specific abuse/resource protection. +- Future protection should target scarce reasoning/provider capacity rather than indiscriminately throttling cheap authenticated persistence/read operations. +- `withAuthenticatedApi` establishes authenticated identity but should not automatically become a generic reasoning-throttling owner. + +### Accidental duplicate submission (deferred small product/UX item) + +- Trace indicates initial Analyse action is **not disabled** by `status === "loading"`. +- Rapid same-page duplicate submission may therefore be possible. +- This is distinct from server-side rate/concurrency protection. +- Retain for a future bounded correction; do not change production code now. + +### Future operational evidence + +If concurrency instrumentation becomes necessary, prefer metadata only: +- reasoning endpoint/action +- request start/end or duration +- number of concurrent active reasoning calls +- result/timeout classification + +Do not log investigation content merely for capacity measurement. + +### Investigation growth + +Preserve the existing decision: +- No arbitrary whole-investigation snapshot ceiling before real-user evidence. +- Legitimate turn depth and mature investigation size are unknown. +- Later observation may use serialized snapshot bytes, revision, contribution count and finding count without recording content. + +### Existing deferred hardening/cleanup (not tester blockers) + +Ensure these remain visible and are not accidentally promoted to tester blockers: +- malformed persistence-envelope/runtime validation +- investigation overview unexpected-extra-field validation +- provider JSON recovery robustness +- orchestrator update-flow control/indentation clarity +- whole-snapshot size decision pending real-user evidence +- reasoning concurrency/rate protection before wider access +- initial Analyse duplicate-submit prevention diff --git a/docs/current-project-state.md b/docs/current-project-state.md index 013025d..72c0ef5 100644 --- a/docs/current-project-state.md +++ b/docs/current-project-state.md @@ -303,16 +303,46 @@ The following material learnings are carried forward as durable context for safe ### Focused input bound - 10,000-character server-side + UI boundary on focused investigation (commit `a6796c6`). -### Investigation snapshot size envelope -- Authenticated persistence lacks a whole-snapshot size ceiling. Not a pre-tester blocker — legitimate size/depth is unknown; monitor via metadata later without logging content. +### Investigation snapshot size envelope / investigation growth +- Authenticated persistence lacks a whole-snapshot size ceiling. +- No arbitrary whole-investigation snapshot ceiling before real-user evidence. +- Legitimate turn depth and mature investigation size are unknown. +- Later observation may use serialized snapshot bytes, revision, contribution count and finding count without recording content. +- Not a pre-tester blocker. ### Prompt injection - **Low risk under the current architecture.** Untrusted text can influence model reasoning but no evidence it gains application authority. No model-accessible arbitrary tools, DB targeting, auth control, or privileged side effects found. Output passes structured validation before application mutation. Not a pre-tester blocker. -### Deferred non-security observations +### Deferred non-security observations (backlog only — no code change) - Orchestrator update flow indentation/control-flow clarity deferred. - Investigation overview validation may accept unexpected extra fields. - Provider JSON recovery permissiveness deferred as future robustness review. +- Malformed persistence-envelope/runtime validation deferred. +- Initial Analyse duplicate-submit prevention deferred (action not disabled by `status === "loading"`). + +### Tester-readiness position + +``` +controlled external testing: GO + +known pre-tester security blockers: none identified + +next uncertainty worth reducing: real first-time-user product value, trust and independent usability +``` + +Do not claim commercial launch readiness, general production scalability, security certification or proven product-market fit. + +### Reasoning concurrency (deferred) +- No server-side per-user reasoning concurrency protection; no CE-level global limit. +- Multiple tabs/users can reach provider concurrently — Ollama/provider behaviour not controlled by CE repository code. +- Controlled cohort of ~5–10 invited testers: **not a release blocker**. +- Before wider/public access, reasoning-resource concurrency protection should be reconsidered. +- If instrumentation needed later: prefer metadata only (endpoint/action, request start/end or duration, concurrent active call count, result/timeout classification). Do not log investigation content for capacity measurement. + +### Rate limiting (deferred) +- No application-level per-user reasoning rate limit currently exists. +- Not required for controlled tester cohort; revisit before wider/public access. +- Future protection should target scarce reasoning/provider capacity, not indiscriminately throttle cheap authenticated persistence/read operations. ---