docs(confidence-engine): record tester readiness findings
This commit is contained in:
@@ -409,3 +409,102 @@ Previous focused-deconstruction semantic runs before the plumbing fix remain **i
|
|||||||
2. Initial-reconstruction schema was supplied to focused-deconstruction call instead of its own contract
|
2. Initial-reconstruction schema was supplied to focused-deconstruction call instead of its own contract
|
||||||
|
|
||||||
These are recorded as known contamination in the v0.61 archive chapter (`docs/design-evolution/ch19/initial-decomposition-v0.61.md`). The plumbing fix is complete and verified (48/48 tests).
|
These are recorded as known contamination in the v0.61 archive chapter (`docs/design-evolution/ch19/initial-decomposition-v0.61.md`). The plumbing fix is complete and verified (48/48 tests).
|
||||||
|
|
||||||
|
## Tester-Readiness Findings — Recorded for Session Continuity
|
||||||
|
|
||||||
|
### Tester-readiness position
|
||||||
|
|
||||||
|
```
|
||||||
|
controlled external testing: GO
|
||||||
|
|
||||||
|
known pre-tester security blockers: none identified
|
||||||
|
|
||||||
|
next uncertainty worth reducing: real first-time-user product value, trust and independent usability
|
||||||
|
```
|
||||||
|
|
||||||
|
Do not claim commercial launch readiness, general production scalability, security certification or proven product-market fit.
|
||||||
|
|
||||||
|
### First external-user experiment
|
||||||
|
|
||||||
|
**Primary question:** Can a first-time user, without Rob guiding the investigation, use Confidence Engine to reach a Current Understanding that they regard as materially better than the way they framed the situation at the start?
|
||||||
|
|
||||||
|
**Tester evidence should prioritise:**
|
||||||
|
- did understanding materially improve?
|
||||||
|
- did they trust the changed understanding?
|
||||||
|
- could they reach it without Rob's help?
|
||||||
|
- was the experience usable?
|
||||||
|
|
||||||
|
**Post-use discovery questions (do not seed categories or mention a mental-health interpretation):**
|
||||||
|
- Who do you think this would be useful for?
|
||||||
|
- What kinds of situations would you use this for?
|
||||||
|
- Is there a situation in your own life or work where you can imagine coming back and using this?
|
||||||
|
|
||||||
|
### Landing-page positioning
|
||||||
|
|
||||||
|
- Current landing wording is deliberately unchanged for the first cohort.
|
||||||
|
- It may naturally be interpreted broadly, potentially including mental-health/anxiety-adjacent situations.
|
||||||
|
- This is an observation to learn from, not currently a defect.
|
||||||
|
- Use unprompted tester responses to learn what category/audience/use cases users believe Confidence Engine belongs to.
|
||||||
|
- Do not reposition before this evidence exists.
|
||||||
|
|
||||||
|
### Synthetic-user testing
|
||||||
|
|
||||||
|
- AI-driven first-time-user journeys may be useful as a pre-human stress test.
|
||||||
|
- Preferred conceptual separation: **synthetic first-time user → real Confidence Engine journey → independent evaluator**.
|
||||||
|
- Synthetic testing can expose reasoning/UX/systematic failures.
|
||||||
|
- It does NOT replace human evidence about genuine trust, changed understanding, usefulness, repeat use or willingness to return.
|
||||||
|
- No synthetic-testing implementation is currently required before human testing.
|
||||||
|
|
||||||
|
### Reasoning concurrency
|
||||||
|
|
||||||
|
- Current CE application has **no** server-side per-user reasoning concurrency protection.
|
||||||
|
- No CE-level global reasoning concurrency limit.
|
||||||
|
- Multiple tabs/users can reach provider concurrently.
|
||||||
|
- Ollama/provider concurrency and queue behaviour are not controlled by CE repository code.
|
||||||
|
- Do not invent queue/latency behaviour from the 300-second request timeout.
|
||||||
|
- Controlled cohort of roughly 5–10 invited testers: **not currently a release blocker**.
|
||||||
|
- Observe actual behaviour before designing queue/semaphore infrastructure.
|
||||||
|
- Before wider/public access, reasoning-resource concurrency protection should be reconsidered.
|
||||||
|
|
||||||
|
### Rate limiting
|
||||||
|
|
||||||
|
- No application-level per-user reasoning rate limit currently exists.
|
||||||
|
- Generic API rate limiting is not required for the controlled tester cohort.
|
||||||
|
- Wider/public access should revisit reasoning-specific abuse/resource protection.
|
||||||
|
- Future protection should target scarce reasoning/provider capacity rather than indiscriminately throttling cheap authenticated persistence/read operations.
|
||||||
|
- `withAuthenticatedApi` establishes authenticated identity but should not automatically become a generic reasoning-throttling owner.
|
||||||
|
|
||||||
|
### Accidental duplicate submission (deferred small product/UX item)
|
||||||
|
|
||||||
|
- Trace indicates initial Analyse action is **not disabled** by `status === "loading"`.
|
||||||
|
- Rapid same-page duplicate submission may therefore be possible.
|
||||||
|
- This is distinct from server-side rate/concurrency protection.
|
||||||
|
- Retain for a future bounded correction; do not change production code now.
|
||||||
|
|
||||||
|
### Future operational evidence
|
||||||
|
|
||||||
|
If concurrency instrumentation becomes necessary, prefer metadata only:
|
||||||
|
- reasoning endpoint/action
|
||||||
|
- request start/end or duration
|
||||||
|
- number of concurrent active reasoning calls
|
||||||
|
- result/timeout classification
|
||||||
|
|
||||||
|
Do not log investigation content merely for capacity measurement.
|
||||||
|
|
||||||
|
### Investigation growth
|
||||||
|
|
||||||
|
Preserve the existing decision:
|
||||||
|
- No arbitrary whole-investigation snapshot ceiling before real-user evidence.
|
||||||
|
- Legitimate turn depth and mature investigation size are unknown.
|
||||||
|
- Later observation may use serialized snapshot bytes, revision, contribution count and finding count without recording content.
|
||||||
|
|
||||||
|
### Existing deferred hardening/cleanup (not tester blockers)
|
||||||
|
|
||||||
|
Ensure these remain visible and are not accidentally promoted to tester blockers:
|
||||||
|
- malformed persistence-envelope/runtime validation
|
||||||
|
- investigation overview unexpected-extra-field validation
|
||||||
|
- provider JSON recovery robustness
|
||||||
|
- orchestrator update-flow control/indentation clarity
|
||||||
|
- whole-snapshot size decision pending real-user evidence
|
||||||
|
- reasoning concurrency/rate protection before wider access
|
||||||
|
- initial Analyse duplicate-submit prevention
|
||||||
|
|||||||
@@ -303,16 +303,46 @@ The following material learnings are carried forward as durable context for safe
|
|||||||
### Focused input bound
|
### Focused input bound
|
||||||
- 10,000-character server-side + UI boundary on focused investigation (commit `a6796c6`).
|
- 10,000-character server-side + UI boundary on focused investigation (commit `a6796c6`).
|
||||||
|
|
||||||
### Investigation snapshot size envelope
|
### Investigation snapshot size envelope / investigation growth
|
||||||
- Authenticated persistence lacks a whole-snapshot size ceiling. Not a pre-tester blocker — legitimate size/depth is unknown; monitor via metadata later without logging content.
|
- Authenticated persistence lacks a whole-snapshot size ceiling.
|
||||||
|
- No arbitrary whole-investigation snapshot ceiling before real-user evidence.
|
||||||
|
- Legitimate turn depth and mature investigation size are unknown.
|
||||||
|
- Later observation may use serialized snapshot bytes, revision, contribution count and finding count without recording content.
|
||||||
|
- Not a pre-tester blocker.
|
||||||
|
|
||||||
### Prompt injection
|
### Prompt injection
|
||||||
- **Low risk under the current architecture.** Untrusted text can influence model reasoning but no evidence it gains application authority. No model-accessible arbitrary tools, DB targeting, auth control, or privileged side effects found. Output passes structured validation before application mutation. Not a pre-tester blocker.
|
- **Low risk under the current architecture.** Untrusted text can influence model reasoning but no evidence it gains application authority. No model-accessible arbitrary tools, DB targeting, auth control, or privileged side effects found. Output passes structured validation before application mutation. Not a pre-tester blocker.
|
||||||
|
|
||||||
### Deferred non-security observations
|
### Deferred non-security observations (backlog only — no code change)
|
||||||
- Orchestrator update flow indentation/control-flow clarity deferred.
|
- Orchestrator update flow indentation/control-flow clarity deferred.
|
||||||
- Investigation overview validation may accept unexpected extra fields.
|
- Investigation overview validation may accept unexpected extra fields.
|
||||||
- Provider JSON recovery permissiveness deferred as future robustness review.
|
- Provider JSON recovery permissiveness deferred as future robustness review.
|
||||||
|
- Malformed persistence-envelope/runtime validation deferred.
|
||||||
|
- Initial Analyse duplicate-submit prevention deferred (action not disabled by `status === "loading"`).
|
||||||
|
|
||||||
|
### Tester-readiness position
|
||||||
|
|
||||||
|
```
|
||||||
|
controlled external testing: GO
|
||||||
|
|
||||||
|
known pre-tester security blockers: none identified
|
||||||
|
|
||||||
|
next uncertainty worth reducing: real first-time-user product value, trust and independent usability
|
||||||
|
```
|
||||||
|
|
||||||
|
Do not claim commercial launch readiness, general production scalability, security certification or proven product-market fit.
|
||||||
|
|
||||||
|
### Reasoning concurrency (deferred)
|
||||||
|
- No server-side per-user reasoning concurrency protection; no CE-level global limit.
|
||||||
|
- Multiple tabs/users can reach provider concurrently — Ollama/provider behaviour not controlled by CE repository code.
|
||||||
|
- Controlled cohort of ~5–10 invited testers: **not a release blocker**.
|
||||||
|
- Before wider/public access, reasoning-resource concurrency protection should be reconsidered.
|
||||||
|
- If instrumentation needed later: prefer metadata only (endpoint/action, request start/end or duration, concurrent active call count, result/timeout classification). Do not log investigation content for capacity measurement.
|
||||||
|
|
||||||
|
### Rate limiting (deferred)
|
||||||
|
- No application-level per-user reasoning rate limit currently exists.
|
||||||
|
- Not required for controlled tester cohort; revisit before wider/public access.
|
||||||
|
- Future protection should target scarce reasoning/provider capacity, not indiscriminately throttle cheap authenticated persistence/read operations.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user