docs(confidence-engine): record tester readiness findings

This commit is contained in:
2026-09-11 10:34:53 +01:00
parent 2349ef9d59
commit 2dc1cb0fe6
2 changed files with 132 additions and 3 deletions
+99
View File
@@ -409,3 +409,102 @@ Previous focused-deconstruction semantic runs before the plumbing fix remain **i
2. Initial-reconstruction schema was supplied to focused-deconstruction call instead of its own contract 2. Initial-reconstruction schema was supplied to focused-deconstruction call instead of its own contract
These are recorded as known contamination in the v0.61 archive chapter (`docs/design-evolution/ch19/initial-decomposition-v0.61.md`). The plumbing fix is complete and verified (48/48 tests). These are recorded as known contamination in the v0.61 archive chapter (`docs/design-evolution/ch19/initial-decomposition-v0.61.md`). The plumbing fix is complete and verified (48/48 tests).
## Tester-Readiness Findings — Recorded for Session Continuity
### Tester-readiness position
```
controlled external testing: GO
known pre-tester security blockers: none identified
next uncertainty worth reducing: real first-time-user product value, trust and independent usability
```
Do not claim commercial launch readiness, general production scalability, security certification or proven product-market fit.
### First external-user experiment
**Primary question:** Can a first-time user, without Rob guiding the investigation, use Confidence Engine to reach a Current Understanding that they regard as materially better than the way they framed the situation at the start?
**Tester evidence should prioritise:**
- did understanding materially improve?
- did they trust the changed understanding?
- could they reach it without Rob's help?
- was the experience usable?
**Post-use discovery questions (do not seed categories or mention a mental-health interpretation):**
- Who do you think this would be useful for?
- What kinds of situations would you use this for?
- Is there a situation in your own life or work where you can imagine coming back and using this?
### Landing-page positioning
- Current landing wording is deliberately unchanged for the first cohort.
- It may naturally be interpreted broadly, potentially including mental-health/anxiety-adjacent situations.
- This is an observation to learn from, not currently a defect.
- Use unprompted tester responses to learn what category/audience/use cases users believe Confidence Engine belongs to.
- Do not reposition before this evidence exists.
### Synthetic-user testing
- AI-driven first-time-user journeys may be useful as a pre-human stress test.
- Preferred conceptual separation: **synthetic first-time user → real Confidence Engine journey → independent evaluator**.
- Synthetic testing can expose reasoning/UX/systematic failures.
- It does NOT replace human evidence about genuine trust, changed understanding, usefulness, repeat use or willingness to return.
- No synthetic-testing implementation is currently required before human testing.
### Reasoning concurrency
- Current CE application has **no** server-side per-user reasoning concurrency protection.
- No CE-level global reasoning concurrency limit.
- Multiple tabs/users can reach provider concurrently.
- Ollama/provider concurrency and queue behaviour are not controlled by CE repository code.
- Do not invent queue/latency behaviour from the 300-second request timeout.
- Controlled cohort of roughly 510 invited testers: **not currently a release blocker**.
- Observe actual behaviour before designing queue/semaphore infrastructure.
- Before wider/public access, reasoning-resource concurrency protection should be reconsidered.
### Rate limiting
- No application-level per-user reasoning rate limit currently exists.
- Generic API rate limiting is not required for the controlled tester cohort.
- Wider/public access should revisit reasoning-specific abuse/resource protection.
- Future protection should target scarce reasoning/provider capacity rather than indiscriminately throttling cheap authenticated persistence/read operations.
- `withAuthenticatedApi` establishes authenticated identity but should not automatically become a generic reasoning-throttling owner.
### Accidental duplicate submission (deferred small product/UX item)
- Trace indicates initial Analyse action is **not disabled** by `status === "loading"`.
- Rapid same-page duplicate submission may therefore be possible.
- This is distinct from server-side rate/concurrency protection.
- Retain for a future bounded correction; do not change production code now.
### Future operational evidence
If concurrency instrumentation becomes necessary, prefer metadata only:
- reasoning endpoint/action
- request start/end or duration
- number of concurrent active reasoning calls
- result/timeout classification
Do not log investigation content merely for capacity measurement.
### Investigation growth
Preserve the existing decision:
- No arbitrary whole-investigation snapshot ceiling before real-user evidence.
- Legitimate turn depth and mature investigation size are unknown.
- Later observation may use serialized snapshot bytes, revision, contribution count and finding count without recording content.
### Existing deferred hardening/cleanup (not tester blockers)
Ensure these remain visible and are not accidentally promoted to tester blockers:
- malformed persistence-envelope/runtime validation
- investigation overview unexpected-extra-field validation
- provider JSON recovery robustness
- orchestrator update-flow control/indentation clarity
- whole-snapshot size decision pending real-user evidence
- reasoning concurrency/rate protection before wider access
- initial Analyse duplicate-submit prevention
+33 -3
View File
@@ -303,16 +303,46 @@ The following material learnings are carried forward as durable context for safe
### Focused input bound ### Focused input bound
- 10,000-character server-side + UI boundary on focused investigation (commit `a6796c6`). - 10,000-character server-side + UI boundary on focused investigation (commit `a6796c6`).
### Investigation snapshot size envelope ### Investigation snapshot size envelope / investigation growth
- Authenticated persistence lacks a whole-snapshot size ceiling. Not a pre-tester blocker — legitimate size/depth is unknown; monitor via metadata later without logging content. - Authenticated persistence lacks a whole-snapshot size ceiling.
- No arbitrary whole-investigation snapshot ceiling before real-user evidence.
- Legitimate turn depth and mature investigation size are unknown.
- Later observation may use serialized snapshot bytes, revision, contribution count and finding count without recording content.
- Not a pre-tester blocker.
### Prompt injection ### Prompt injection
- **Low risk under the current architecture.** Untrusted text can influence model reasoning but no evidence it gains application authority. No model-accessible arbitrary tools, DB targeting, auth control, or privileged side effects found. Output passes structured validation before application mutation. Not a pre-tester blocker. - **Low risk under the current architecture.** Untrusted text can influence model reasoning but no evidence it gains application authority. No model-accessible arbitrary tools, DB targeting, auth control, or privileged side effects found. Output passes structured validation before application mutation. Not a pre-tester blocker.
### Deferred non-security observations ### Deferred non-security observations (backlog only — no code change)
- Orchestrator update flow indentation/control-flow clarity deferred. - Orchestrator update flow indentation/control-flow clarity deferred.
- Investigation overview validation may accept unexpected extra fields. - Investigation overview validation may accept unexpected extra fields.
- Provider JSON recovery permissiveness deferred as future robustness review. - Provider JSON recovery permissiveness deferred as future robustness review.
- Malformed persistence-envelope/runtime validation deferred.
- Initial Analyse duplicate-submit prevention deferred (action not disabled by `status === "loading"`).
### Tester-readiness position
```
controlled external testing: GO
known pre-tester security blockers: none identified
next uncertainty worth reducing: real first-time-user product value, trust and independent usability
```
Do not claim commercial launch readiness, general production scalability, security certification or proven product-market fit.
### Reasoning concurrency (deferred)
- No server-side per-user reasoning concurrency protection; no CE-level global limit.
- Multiple tabs/users can reach provider concurrently — Ollama/provider behaviour not controlled by CE repository code.
- Controlled cohort of ~510 invited testers: **not a release blocker**.
- Before wider/public access, reasoning-resource concurrency protection should be reconsidered.
- If instrumentation needed later: prefer metadata only (endpoint/action, request start/end or duration, concurrent active call count, result/timeout classification). Do not log investigation content for capacity measurement.
### Rate limiting (deferred)
- No application-level per-user reasoning rate limit currently exists.
- Not required for controlled tester cohort; revisit before wider/public access.
- Future protection should target scarce reasoning/provider capacity, not indiscriminately throttle cheap authenticated persistence/read operations.
--- ---