mirror of
https://github.com/leonvanzyl/autocoder.git
synced 2026-01-31 14:43:35 +00:00
fix: Prevent mock data implementations with infrastructure features
Problem: The coding agent can implement in-memory storage (e.g., `dev-store.ts` with `globalThis`) instead of a real database. These implementations pass all tests because data persists during runtime, but data is lost on server restart. This is a root cause for #68 - agent "passes" features that don't actually work. Solution: 1. Add 5 mandatory Infrastructure Features (indices 0-4) that run first: - Feature 0: Database connection established - Feature 1: Database schema applied correctly - Feature 2: Data persists across server restart (CRITICAL) - Feature 3: No mock data patterns in codebase - Feature 4: Backend API queries real database 2. Add STEP 5.7: Server Restart Persistence Test to coding prompt: - Create test data, stop server, restart, verify data still exists 3. Extend grep patterns for mock detection in STEP 5.6: - globalThis., devStore, dev-store, mockData, fakeData - TODO.*real, STUB, MOCK, new Map() as data stores Changes: - .claude/templates/initializer_prompt.template.md - Infrastructure features - .claude/templates/coding_prompt.template.md - STEP 5.6/5.7 enhancements - .claude/commands/create-spec.md - Phase 3b database question Backwards Compatible: - Works with YOLO mode (uses bash/grep, not browser automation) - Stateless apps can skip database features via create-spec question Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
@@ -156,6 +156,9 @@ Use browser automation tools:
|
||||
- [ ] Deleted the test data - verified it's gone everywhere
|
||||
- [ ] NO unexplained data appeared (would indicate mock data)
|
||||
- [ ] Dashboard/counts reflect real numbers after my changes
|
||||
- [ ] **Ran extended mock data grep (STEP 5.6) - no hits in src/ (excluding tests)**
|
||||
- [ ] **Verified no globalThis, devStore, or dev-store patterns**
|
||||
- [ ] **Server restart test passed (STEP 5.7) - data persists across restart**
|
||||
|
||||
#### Navigation Verification
|
||||
|
||||
@@ -174,10 +177,72 @@ Use browser automation tools:
|
||||
|
||||
### STEP 5.6: MOCK DATA DETECTION (Before marking passing)
|
||||
|
||||
1. **Search code:** `grep -r "mockData\|fakeData\|TODO\|STUB" --include="*.ts" --include="*.tsx"`
|
||||
2. **Runtime test:** Create unique data (e.g., "TEST_12345") → verify in UI → delete → verify gone
|
||||
3. **Check database:** All displayed data must come from real DB queries
|
||||
4. If unexplained data appears, it's mock data - fix before marking passing.
|
||||
**Run ALL these grep checks. Any hits in src/ (excluding test files) require investigation:**
|
||||
|
||||
```bash
|
||||
# 1. In-memory storage patterns (CRITICAL - catches dev-store)
|
||||
grep -r "globalThis\." --include="*.ts" --include="*.tsx" --include="*.js" src/
|
||||
grep -r "dev-store\|devStore\|DevStore\|mock-db\|mockDb" --include="*.ts" --include="*.tsx" src/
|
||||
|
||||
# 2. Mock data variables
|
||||
grep -r "mockData\|fakeData\|sampleData\|dummyData\|testData" --include="*.ts" --include="*.tsx" src/
|
||||
|
||||
# 3. TODO/incomplete markers
|
||||
grep -r "TODO.*real\|TODO.*database\|TODO.*API\|STUB\|MOCK" --include="*.ts" --include="*.tsx" src/
|
||||
|
||||
# 4. Development-only conditionals
|
||||
grep -r "isDevelopment\|isDev\|process\.env\.NODE_ENV.*development" --include="*.ts" --include="*.tsx" src/
|
||||
|
||||
# 5. In-memory collections as data stores (check lib/store/data directories)
|
||||
grep -r "new Map()\|new Set()" --include="*.ts" --include="*.tsx" src/lib/ src/store/ src/data/ 2>/dev/null
|
||||
```
|
||||
|
||||
**Rule:** If ANY grep returns results in production code → investigate → FIX before marking passing.
|
||||
|
||||
**Runtime verification:**
|
||||
1. Create unique data (e.g., "TEST_12345") → verify in UI → delete → verify gone
|
||||
2. Check database directly - all displayed data must come from real DB queries
|
||||
3. If unexplained data appears, it's mock data - fix before marking passing.
|
||||
|
||||
### STEP 5.7: SERVER RESTART PERSISTENCE TEST (MANDATORY for data features)
|
||||
|
||||
**When required:** Any feature involving CRUD operations or data persistence.
|
||||
|
||||
**This test is NON-NEGOTIABLE. It catches in-memory storage implementations that pass all other tests.**
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. Create unique test data via UI or API (e.g., item named "RESTART_TEST_12345")
|
||||
2. Verify data appears in UI and API response
|
||||
|
||||
3. **STOP the server completely:**
|
||||
```bash
|
||||
pkill -f "node" || pkill -f "npm" || pkill -f "next"
|
||||
sleep 5
|
||||
# Verify server is stopped
|
||||
pgrep -f "node" && echo "ERROR: Server still running!" && exit 1
|
||||
```
|
||||
|
||||
4. **RESTART the server:**
|
||||
```bash
|
||||
./init.sh &
|
||||
sleep 15 # Allow server to fully start
|
||||
```
|
||||
|
||||
5. **Query for test data - it MUST still exist**
|
||||
- Via UI: Navigate to data location, verify data appears
|
||||
- Via API: `curl http://localhost:PORT/api/items` - verify data in response
|
||||
|
||||
6. **If data is GONE:** Implementation uses in-memory storage → CRITICAL FAIL
|
||||
- Search for: `grep -r "globalThis\|devStore\|dev-store" src/`
|
||||
- You MUST fix the mock data implementation before proceeding
|
||||
- Replace in-memory storage with real database queries
|
||||
|
||||
7. **Clean up test data** after successful verification
|
||||
|
||||
**Why this test exists:** In-memory stores like `globalThis.devStore` pass all other tests because data persists during a single server run. Only a full server restart reveals this bug. Skipping this step WILL allow dev-store implementations to slip through.
|
||||
|
||||
**YOLO Mode Note:** Even in YOLO mode, this verification is MANDATORY for data features. Use curl instead of browser automation.
|
||||
|
||||
### STEP 6: UPDATE FEATURE STATUS (CAREFULLY!)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user