diff --git a/bmad-core/data/test-levels-framework.md b/bmad-core/data/test-levels-framework.md new file mode 100644 index 00000000..b31f5b7b --- /dev/null +++ b/bmad-core/data/test-levels-framework.md @@ -0,0 +1,146 @@ +# Test Levels Framework + +Comprehensive guide for determining appropriate test levels (unit, integration, E2E) for different scenarios. + +## Test Level Decision Matrix + +### Unit Tests + +**When to use:** + +- Testing pure functions and business logic +- Algorithm correctness +- Input validation and data transformation +- Error handling in isolated components +- Complex calculations or state machines + +**Characteristics:** + +- Fast execution (immediate feedback) +- No external dependencies (DB, API, file system) +- Highly maintainable and stable +- Easy to debug failures + +**Example scenarios:** + +```yaml +unit_test: + component: "PriceCalculator" + scenario: "Calculate discount with multiple rules" + justification: "Complex business logic with multiple branches" + mock_requirements: "None - pure function" +``` + +### Integration Tests + +**When to use:** + +- Component interaction verification +- Database operations and transactions +- API endpoint contracts +- Service-to-service communication +- Middleware and interceptor behavior + +**Characteristics:** + +- Moderate execution time +- Tests component boundaries +- May use test databases or containers +- Validates system integration points + +**Example scenarios:** + +```yaml +integration_test: + components: ["UserService", "AuthRepository"] + scenario: "Create user with role assignment" + justification: "Critical data flow between service and persistence" + test_environment: "In-memory database" +``` + +### End-to-End Tests + +**When to use:** + +- Critical user journeys +- Cross-system workflows +- Visual regression testing +- Compliance and regulatory requirements +- Final validation before release + +**Characteristics:** + +- Slower execution +- Tests complete workflows +- Requires full environment setup +- Most realistic but most brittle + +**Example scenarios:** + +```yaml +e2e_test: + journey: "Complete checkout process" + scenario: "User purchases with saved payment method" + justification: "Revenue-critical path requiring full validation" + environment: "Staging with test payment gateway" +``` + +## Test Level Selection Rules + +### Favor Unit Tests When: + +- Logic can be isolated +- No side effects involved +- Fast feedback needed +- High cyclomatic complexity + +### Favor Integration Tests When: + +- Testing persistence layer +- Validating service contracts +- Testing middleware/interceptors +- Component boundaries critical + +### Favor E2E Tests When: + +- User-facing critical paths +- Multi-system interactions +- Regulatory compliance scenarios +- Visual regression important + +## Anti-patterns to Avoid + +- E2E testing for business logic validation +- Unit testing framework behavior +- Integration testing third-party libraries +- Duplicate coverage across levels + +## Duplicate Coverage Guard + +**Before adding any test, check:** + +1. Is this already tested at a lower level? +2. Can a unit test cover this instead of integration? +3. Can an integration test cover this instead of E2E? + +**Coverage overlap is only acceptable when:** + +- Testing different aspects (unit: logic, integration: interaction, e2e: user experience) +- Critical paths requiring defense in depth +- Regression prevention for previously broken functionality + +## Test Naming Conventions + +- Unit: `test_{component}_{scenario}` +- Integration: `test_{flow}_{interaction}` +- E2E: `test_{journey}_{outcome}` + +## Test ID Format + +`{EPIC}.{STORY}-{LEVEL}-{SEQ}` + +Examples: + +- `1.3-UNIT-001` +- `1.3-INT-002` +- `1.3-E2E-001` diff --git a/bmad-core/data/test-priorities-matrix.md b/bmad-core/data/test-priorities-matrix.md new file mode 100644 index 00000000..7463ef9b --- /dev/null +++ b/bmad-core/data/test-priorities-matrix.md @@ -0,0 +1,172 @@ +# Test Priorities Matrix + +Guide for prioritizing test scenarios based on risk, criticality, and business impact. + +## Priority Levels + +### P0 - Critical (Must Test) + +**Criteria:** + +- Revenue-impacting functionality +- Security-critical paths +- Data integrity operations +- Regulatory compliance requirements +- Previously broken functionality (regression prevention) + +**Examples:** + +- Payment processing +- Authentication/authorization +- User data creation/deletion +- Financial calculations +- GDPR/privacy compliance + +**Testing Requirements:** + +- Comprehensive coverage at all levels +- Both happy and unhappy paths +- Edge cases and error scenarios +- Performance under load + +### P1 - High (Should Test) + +**Criteria:** + +- Core user journeys +- Frequently used features +- Features with complex logic +- Integration points between systems +- Features affecting user experience + +**Examples:** + +- User registration flow +- Search functionality +- Data import/export +- Notification systems +- Dashboard displays + +**Testing Requirements:** + +- Primary happy paths required +- Key error scenarios +- Critical edge cases +- Basic performance validation + +### P2 - Medium (Nice to Test) + +**Criteria:** + +- Secondary features +- Admin functionality +- Reporting features +- Configuration options +- UI polish and aesthetics + +**Examples:** + +- Admin settings panels +- Report generation +- Theme customization +- Help documentation +- Analytics tracking + +**Testing Requirements:** + +- Happy path coverage +- Basic error handling +- Can defer edge cases + +### P3 - Low (Test if Time Permits) + +**Criteria:** + +- Rarely used features +- Nice-to-have functionality +- Cosmetic issues +- Non-critical optimizations + +**Examples:** + +- Advanced preferences +- Legacy feature support +- Experimental features +- Debug utilities + +**Testing Requirements:** + +- Smoke tests only +- Can rely on manual testing +- Document known limitations + +## Risk-Based Priority Adjustments + +### Increase Priority When: + +- High user impact (affects >50% of users) +- High financial impact (>$10K potential loss) +- Security vulnerability potential +- Compliance/legal requirements +- Customer-reported issues +- Complex implementation (>500 LOC) +- Multiple system dependencies + +### Decrease Priority When: + +- Feature flag protected +- Gradual rollout planned +- Strong monitoring in place +- Easy rollback capability +- Low usage metrics +- Simple implementation +- Well-isolated component + +## Test Coverage by Priority + +| Priority | Unit Coverage | Integration Coverage | E2E Coverage | +| -------- | ------------- | -------------------- | ------------------ | +| P0 | >90% | >80% | All critical paths | +| P1 | >80% | >60% | Main happy paths | +| P2 | >60% | >40% | Smoke tests | +| P3 | Best effort | Best effort | Manual only | + +## Priority Assignment Rules + +1. **Start with business impact** - What happens if this fails? +2. **Consider probability** - How likely is failure? +3. **Factor in detectability** - Would we know if it failed? +4. **Account for recoverability** - Can we fix it quickly? + +## Priority Decision Tree + +``` +Is it revenue-critical? +├─ YES → P0 +└─ NO → Does it affect core user journey? + ├─ YES → Is it high-risk? + │ ├─ YES → P0 + │ └─ NO → P1 + └─ NO → Is it frequently used? + ├─ YES → P1 + └─ NO → Is it customer-facing? + ├─ YES → P2 + └─ NO → P3 +``` + +## Test Execution Order + +1. Execute P0 tests first (fail fast on critical issues) +2. Execute P1 tests second (core functionality) +3. Execute P2 tests if time permits +4. P3 tests only in full regression cycles + +## Continuous Adjustment + +Review and adjust priorities based on: + +- Production incident patterns +- User feedback and complaints +- Usage analytics +- Test failure history +- Business priority changes diff --git a/bmad-core/tasks/nfr-assess.md b/bmad-core/tasks/nfr-assess.md index 2dd3b1cd..6b77526c 100644 --- a/bmad-core/tasks/nfr-assess.md +++ b/bmad-core/tasks/nfr-assess.md @@ -6,9 +6,9 @@ Quick NFR validation focused on the core four: security, performance, reliabilit ```yaml required: - - story_id: "{epic}.{story}" # e.g., "1.3" + - story_id: "{epic}.{story}" # e.g., "1.3" - story_path: "docs/stories/{epic}.{story}.*.md" - + optional: - architecture_refs: "docs/architecture/*.md" - technical_preferences: "docs/technical-preferences.md" @@ -18,7 +18,6 @@ optional: ## Purpose Assess non-functional requirements for a story and generate: - 1. YAML block for the gate file's `nfr_validation` section 2. Brief markdown assessment saved to `docs/qa/assessments/{epic}.{story}-nfr-{YYYYMMDD}.md` @@ -27,7 +26,6 @@ Assess non-functional requirements for a story and generate: ### 0. Fail-safe for Missing Inputs If story_path or story file can't be found: - - Still create assessment file with note: "Source story not found" - Set all selected NFRs to CONCERNS with notes: "Target unknown / evidence missing" - Continue with assessment to provide value @@ -40,7 +38,7 @@ If story_path or story file can't be found: ```text Which NFRs should I assess? (Enter numbers or press Enter for default) [1] Security (default) -[2] Performance (default) +[2] Performance (default) [3] Reliability (default) [4] Maintainability (default) [5] Usability @@ -54,7 +52,6 @@ Which NFRs should I assess? (Enter numbers or press Enter for default) ### 2. Check for Thresholds Look for NFR requirements in: - - Story acceptance criteria - `docs/architecture/*.md` files - `docs/technical-preferences.md` @@ -75,7 +72,6 @@ No security requirements found. Required auth method? ### 3. Quick Assessment For each selected NFR, check: - - Is there evidence it's implemented? - Can we validate it? - Are there obvious gaps? @@ -90,7 +86,7 @@ Generate ONLY for NFRs actually assessed (no placeholders): # Gate YAML (copy/paste): nfr_validation: _assessed: [security, performance, reliability, maintainability] - security: + security: status: CONCERNS notes: "No rate limiting on auth endpoints" performance: @@ -107,7 +103,7 @@ nfr_validation: ## Deterministic Status Rules - **FAIL**: Any selected NFR has critical gap or target clearly not met -- **CONCERNS**: No FAILs, but any NFR is unknown/partial/missing evidence +- **CONCERNS**: No FAILs, but any NFR is unknown/partial/missing evidence - **PASS**: All selected NFRs meet targets with evidence ## Quality Score Calculation @@ -127,21 +123,18 @@ If `technical-preferences.md` defines custom weights, use those instead. ```markdown # NFR Assessment: {epic}.{story} - Date: {date} Reviewer: Quinn ## Summary - - Security: CONCERNS - Missing rate limiting - Performance: PASS - Meets <200ms requirement - Reliability: PASS - Proper error handling - Maintainability: CONCERNS - Test coverage below target ## Critical Issues - 1. **No rate limiting** (Security) - Risk: Brute force attacks possible - Fix: Add rate limiting middleware to auth endpoints @@ -151,7 +144,6 @@ Reviewer: Quinn - Fix: Add tests for uncovered branches ## Quick Wins - - Add rate limiting: ~2 hours - Increase test coverage: ~4 hours - Add performance monitoring: ~1 hour @@ -160,7 +152,6 @@ Reviewer: Quinn ## Output 3: Story Update Line **End with this line for the review task to quote:** - ``` NFR assessment: docs/qa/assessments/{epic}.{story}-nfr-{YYYYMMDD}.md ``` @@ -168,7 +159,6 @@ NFR assessment: docs/qa/assessments/{epic}.{story}-nfr-{YYYYMMDD}.md ## Output 4: Gate Integration Line **Always print at the end:** - ``` Gate NFR block ready → paste into docs/qa/gates/{epic}.{story}-{slug}.yml under nfr_validation ``` @@ -176,82 +166,66 @@ Gate NFR block ready → paste into docs/qa/gates/{epic}.{story}-{slug}.yml unde ## Assessment Criteria ### Security - **PASS if:** - - Authentication implemented - Authorization enforced - Input validation present - No hardcoded secrets **CONCERNS if:** - - Missing rate limiting - Weak encryption - Incomplete authorization **FAIL if:** - - No authentication - Hardcoded credentials - SQL injection vulnerabilities ### Performance - **PASS if:** - - Meets response time targets - No obvious bottlenecks - Reasonable resource usage **CONCERNS if:** - - Close to limits - Missing indexes - No caching strategy **FAIL if:** - - Exceeds response time limits - Memory leaks - Unoptimized queries ### Reliability - **PASS if:** - - Error handling present - Graceful degradation - Retry logic where needed **CONCERNS if:** - - Some error cases unhandled - No circuit breakers - Missing health checks **FAIL if:** - - No error handling - Crashes on errors - No recovery mechanisms ### Maintainability - **PASS if:** - - Test coverage meets target - Code well-structured - Documentation present **CONCERNS if:** - - Test coverage below target - Some code duplication - Missing documentation **FAIL if:** - - No tests - Highly coupled code - No documentation @@ -309,7 +283,7 @@ maintainability: 1. **Functional Suitability**: Completeness, correctness, appropriateness 2. **Performance Efficiency**: Time behavior, resource use, capacity -3. **Compatibility**: Co-existence, interoperability +3. **Compatibility**: Co-existence, interoperability 4. **Usability**: Learnability, operability, accessibility 5. **Reliability**: Maturity, availability, fault tolerance 6. **Security**: Confidentiality, integrity, authenticity @@ -317,7 +291,6 @@ maintainability: 8. **Portability**: Adaptability, installability Use these when assessing beyond the core four. -
@@ -339,5 +312,4 @@ performance_deep_dive: max_rps: 150 breaking_point: 200 rps ``` - -
+ \ No newline at end of file diff --git a/bmad-core/tasks/review-story.md b/bmad-core/tasks/review-story.md index 64c96d71..869a58af 100644 --- a/bmad-core/tasks/review-story.md +++ b/bmad-core/tasks/review-story.md @@ -7,7 +7,7 @@ Perform a comprehensive test architecture review with quality gate decision. Thi ```yaml required: - story_id: "{epic}.{story}" # e.g., "1.3" - - story_path: "docs/stories/{epic}.{story}.*.md" + - story_path: "{devStoryLocation}/{epic}.{story}.*.md" # Path from core-config.yaml - story_title: "{title}" # If missing, derive from story file H1 - story_slug: "{slug}" # If missing, derive from title (lowercase, hyphenated) ``` @@ -171,6 +171,8 @@ Gate: {STATUS} → docs/qa/gates/{epic}.{story}-{slug}.yml Risk profile: docs/qa/assessments/{epic}.{story}-risk-{YYYYMMDD}.md NFR assessment: docs/qa/assessments/{epic}.{story}-nfr-{YYYYMMDD}.md +# Note: Paths should reference core-config.yaml for custom configurations + ### Recommended Status [✓ Ready for Done] / [✗ Changes Required - See unchecked items above] @@ -182,7 +184,7 @@ NFR assessment: docs/qa/assessments/{epic}.{story}-nfr-{YYYYMMDD}.md **Template and Directory:** - Render from `templates/qa-gate-tmpl.yaml` -- Create `docs/qa/gates/` directory if missing +- Create `docs/qa/gates/` directory if missing (or configure in core-config.yaml) - Save to: `docs/qa/gates/{epic}.{story}-{slug}.yml` Gate file structure: diff --git a/bmad-core/tasks/test-design.md b/bmad-core/tasks/test-design.md index 81eda507..ec0798fd 100644 --- a/bmad-core/tasks/test-design.md +++ b/bmad-core/tasks/test-design.md @@ -7,7 +7,7 @@ Create comprehensive test scenarios with appropriate test level recommendations ```yaml required: - story_id: "{epic}.{story}" # e.g., "1.3" - - story_path: "docs/stories/{epic}.{story}.*.md" + - story_path: "{devStoryLocation}/{epic}.{story}.*.md" # Path from core-config.yaml - story_title: "{title}" # If missing, derive from story file H1 - story_slug: "{slug}" # If missing, derive from title (lowercase, hyphenated) ``` @@ -16,289 +16,69 @@ required: Design a complete test strategy that identifies what to test, at which level (unit/integration/e2e), and why. This ensures efficient test coverage without redundancy while maintaining appropriate test boundaries. -## Test Level Decision Framework - -### Unit Tests - -**When to use:** - -- Testing pure functions and business logic -- Algorithm correctness -- Input validation and data transformation -- Error handling in isolated components -- Complex calculations or state machines - -**Characteristics:** - -- Fast execution (immediate feedback) -- No external dependencies (DB, API, file system) -- Highly maintainable and stable -- Easy to debug failures - -**Example scenarios:** +## Dependencies ```yaml -unit_test: - component: "PriceCalculator" - scenario: "Calculate discount with multiple rules" - justification: "Complex business logic with multiple branches" - mock_requirements: "None - pure function" +data: + - test-levels-framework.md # Unit/Integration/E2E decision criteria + - test-priorities-matrix.md # P0/P1/P2/P3 classification system ``` -### Integration Tests - -**When to use:** - -- Testing component interactions -- Database operations and queries -- API endpoint behavior -- Service layer orchestration -- External service integration (with test doubles) - -**Characteristics:** - -- Moderate execution time -- May use test databases or containers -- Tests multiple components together -- Validates contracts between components - -**Example scenarios:** - -```yaml -integration_test: - components: ["UserService", "UserRepository", "Database"] - scenario: "Create user with duplicate email check" - justification: "Tests transaction boundaries and constraint handling" - test_doubles: "Mock email service, real test database" -``` - -### End-to-End Tests - -**When to use:** - -- Critical user journeys -- Cross-system workflows -- UI interaction flows -- Full stack validation -- Production-like scenario testing - -**Characteristics:** - -- Keep under 90 seconds per test -- Tests complete user scenarios -- Uses real or production-like environment -- Higher maintenance cost -- More prone to flakiness - -**Example scenarios:** - -```yaml -e2e_test: - flow: "Complete purchase flow" - scenario: "User browses, adds to cart, and completes checkout" - justification: "Critical business flow requiring full stack validation" - environment: "Staging with test payment gateway" -``` - -## Test Design Process +## Process ### 1. Analyze Story Requirements -Break down each acceptance criterion into testable scenarios: +Break down each acceptance criterion into testable scenarios. For each AC: -```yaml -acceptance_criterion: "User can reset password via email" -test_scenarios: - - level: unit - what: "Password validation rules" - why: "Complex regex and business rules" +- Identify the core functionality to test +- Determine data variations needed +- Consider error conditions +- Note edge cases - - level: integration - what: "Password reset token generation and storage" - why: "Database interaction with expiry logic" +### 2. Apply Test Level Framework - - level: integration - what: "Email service integration" - why: "External service with retry logic" +**Reference:** Load `test-levels-framework.md` for detailed criteria - - level: e2e - what: "Complete password reset flow" - why: "Critical security flow needing full validation" -``` +Quick rules: -### 2. Apply Test Level Heuristics +- **Unit**: Pure logic, algorithms, calculations +- **Integration**: Component interactions, DB operations +- **E2E**: Critical user journeys, compliance -Use these rules to determine appropriate test levels: +### 3. Assign Priorities -```markdown -## Test Level Selection Rules +**Reference:** Load `test-priorities-matrix.md` for classification -### Favor Unit Tests When: +Quick priority assignment: -- Logic can be isolated -- No side effects involved -- Fast feedback needed -- High cyclomatic complexity +- **P0**: Revenue-critical, security, compliance +- **P1**: Core user journeys, frequently used +- **P2**: Secondary features, admin functions +- **P3**: Nice-to-have, rarely used -### Favor Integration Tests When: +### 4. Design Test Scenarios -- Testing persistence layer -- Validating service contracts -- Testing middleware/interceptors -- Component boundaries critical - -### Favor E2E Tests When: - -- User-facing critical paths -- Multi-system interactions -- Regulatory compliance scenarios -- Visual regression important - -### Anti-patterns to Avoid: - -- E2E testing for business logic validation -- Unit testing framework behavior -- Integration testing third-party libraries -- Duplicate coverage across levels - -### Duplicate Coverage Guard - -**Before adding any test, check:** - -1. Is this already tested at a lower level? -2. Can a unit test cover this instead of integration? -3. Can an integration test cover this instead of E2E? - -**Coverage overlap is only acceptable when:** - -- Testing different aspects (unit: logic, integration: interaction, e2e: user experience) -- Critical paths requiring defense in depth -- Regression prevention for previously broken functionality -``` - -### 3. Design Test Scenarios - -**Test ID Format:** `{EPIC}.{STORY}-{LEVEL}-{SEQ}` - -- Example: `1.3-UNIT-001`, `1.3-INT-002`, `1.3-E2E-001` -- Ensures traceability across all artifacts - -**Naming Convention:** - -- Unit: `test_{component}_{scenario}` -- Integration: `test_{flow}_{interaction}` -- E2E: `test_{journey}_{outcome}` - -**Risk Linkage:** - -- Tag tests with risk IDs they mitigate -- Prioritize tests for high-risk areas (P0) -- Link to risk profile when available - -For each identified test need: +For each identified test need, create: ```yaml test_scenario: - id: "1.3-INT-002" - requirement: "AC2: Rate limiting on login attempts" - mitigates_risks: ["SEC-001", "PERF-003"] # Links to risk profile - priority: P0 # Based on risk score - - unit_tests: - - name: "RateLimiter calculates window correctly" - input: "Timestamp array" - expected: "Correct window calculation" - - integration_tests: - - name: "Login endpoint enforces rate limit" - setup: "5 failed attempts" - action: "6th attempt" - expected: "429 response with retry-after header" - - e2e_tests: - - name: "User sees rate limit message" - setup: "Trigger rate limit" - validation: "Error message displayed, retry timer shown" + id: "{epic}.{story}-{LEVEL}-{SEQ}" + requirement: "AC reference" + priority: P0|P1|P2|P3 + level: unit|integration|e2e + description: "What is being tested" + justification: "Why this level was chosen" + mitigates_risks: ["RISK-001"] # If risk profile exists ``` -## Deterministic Test Level Minimums +### 5. Validate Coverage -**Per Acceptance Criterion:** +Ensure: -- At least 1 unit test for business logic -- At least 1 integration test if multiple components interact -- At least 1 E2E test if it's a user-facing feature - -**Exceptions:** - -- Pure UI changes: May skip unit tests -- Pure logic changes: May skip E2E tests -- Infrastructure changes: May focus on integration tests - -**When in doubt:** Start with unit tests, add integration for interactions, E2E for critical paths only. - -## Test Quality Standards - -### Core Testing Principles - -**No Flaky Tests:** Ensure reliability through proper async handling, explicit waits, and atomic test design. - -**No Hard Waits/Sleeps:** Use dynamic waiting strategies (e.g., polling, event-based triggers). - -**Stateless & Parallel-Safe:** Tests run independently; use cron jobs or semaphores only if unavoidable. - -**No Order Dependency:** Every it/describe/context block works in isolation (supports .only execution). - -**Self-Cleaning Tests:** Test sets up its own data and automatically deletes/deactivates entities created during testing. - -**Tests Live Near Source Code:** Co-locate test files with the code they validate (e.g., `*.spec.js` alongside components). - -### Execution Strategy - -**Shifted Left:** - -- Start with local environments or ephemeral stacks -- Validate functionality across all deployment stages (local → dev → stage) - -**Low Maintenance:** Minimize manual upkeep (avoid brittle selectors, do not repeat UI actions, leverage APIs). - -**CI Execution Evidence:** Integrate into pipelines with clear logs/artifacts. - -**Visibility:** Generate test reports (e.g., JUnit XML, HTML) for failures and trends. - -### Coverage Requirements - -**Release Confidence:** - -- Happy Path: Core user journeys are prioritized -- Edge Cases: Critical error/validation scenarios are covered -- Feature Flags: Test both enabled and disabled states where applicable - -### Test Design Rules - -**Assertions:** Keep them explicit in tests; avoid abstraction into helpers. Use parametrized tests for soft assertions. - -**Naming:** Follow conventions (e.g., `describe('Component')`, `it('should do X when Y')`). - -**Size:** Aim for files ≤200 lines; split/chunk large tests logically. - -**Speed:** Target individual tests ≤90 seconds; optimize slow setups (e.g., shared fixtures). - -**Careful Abstractions:** Favor readability over DRY when balancing helper reuse (page objects are okay, assertion logic is not). - -**Test Cleanup:** Ensure tests clean up resources they create (e.g., closing browser, deleting test data). - -**Deterministic Flow:** Tests should refrain from using conditionals (e.g., if/else) to control flow or try/catch blocks where possible. - -### API Testing Standards - -- Tests must not depend on hardcoded data → use factories and per-test setup -- Always test both happy path and negative/error cases -- API tests should run parallel safely (no global state shared) -- Test idempotency where applicable (e.g., duplicate requests) -- Tests should clean up their data -- Response logs should only be printed in case of failure -- Auth tests must validate token expiration and renewal +- Every AC has at least one test +- No duplicate coverage across levels +- Critical paths have multiple levels +- Risk mitigations are addressed ## Outputs @@ -306,13 +86,11 @@ test_scenario: **Save to:** `docs/qa/assessments/{epic}.{story}-test-design-{YYYYMMDD}.md` -Generate a comprehensive test design document: - ```markdown # Test Design: Story {epic}.{story} Date: {date} -Reviewer: Quinn (Test Architect) +Designer: Quinn (Test Architect) ## Test Strategy Overview @@ -320,209 +98,77 @@ Reviewer: Quinn (Test Architect) - Unit tests: Y (A%) - Integration tests: Z (B%) - E2E tests: W (C%) +- Priority distribution: P0: X, P1: Y, P2: Z -## Test Level Rationale +## Test Scenarios by Acceptance Criteria -[Explain why this distribution was chosen] +### AC1: {description} -## Detailed Test Scenarios +#### Scenarios -### Requirement: AC1 - {description} +| ID | Level | Priority | Test | Justification | +| ------------ | ----------- | -------- | ------------------------- | ------------------------ | +| 1.3-UNIT-001 | Unit | P0 | Validate input format | Pure validation logic | +| 1.3-INT-001 | Integration | P0 | Service processes request | Multi-component flow | +| 1.3-E2E-001 | E2E | P1 | User completes journey | Critical path validation | -#### Unit Tests (3 scenarios) +[Continue for all ACs...] -1. **ID**: 1.3-UNIT-001 - **Test**: Validate input format - - **Why Unit**: Pure validation logic - - **Coverage**: Input edge cases - - **Mocks**: None needed - - **Mitigates**: DATA-001 (if applicable) +## Risk Coverage -#### Integration Tests (2 scenarios) +[Map test scenarios to identified risks if risk profile exists] -1. **ID**: 1.3-INT-001 - **Test**: Service processes valid request - - **Why Integration**: Multiple components involved - - **Coverage**: Happy path + error handling - - **Test Doubles**: Mock external API - - **Mitigates**: TECH-002 +## Recommended Execution Order -#### E2E Tests (1 scenario) - -1. **ID**: 1.3-E2E-001 - **Test**: Complete user workflow - - **Why E2E**: Critical user journey - - **Coverage**: Full stack validation - - **Environment**: Staging - - **Max Duration**: 90 seconds - - **Mitigates**: BUS-001 - -[Continue for all requirements...] - -## Test Data Requirements - -### Unit Test Data - -- Static fixtures for calculations -- Edge case values arrays - -### Integration Test Data - -- Test database seeds -- API response fixtures - -### E2E Test Data - -- Test user accounts -- Sandbox environment data - -## Mock/Stub Strategy - -### What to Mock - -- External services (payment, email) -- Time-dependent functions -- Random number generators - -### What NOT to Mock - -- Core business logic -- Database in integration tests -- Critical security functions - -## Test Execution Implementation - -### Parallel Execution - -- All unit tests: Fully parallel (stateless requirement) -- Integration tests: Parallel with isolated databases -- E2E tests: Sequential or limited parallelism - -### Execution Order - -1. Unit tests first (fail fast) -2. Integration tests second -3. E2E tests last (expensive, max 90 seconds each) - -## Risk-Based Test Priority - -### P0 - Must Have (Linked to Critical/High Risks) - -- Security-related tests (SEC-\* risks) -- Data integrity tests (DATA-\* risks) -- Critical business flow tests (BUS-\* risks) -- Tests for risks scored ≥6 in risk profile - -### P1 - Should Have (Medium Risks) - -- Edge case coverage -- Performance tests (PERF-\* risks) -- Error recovery tests -- Tests for risks scored 4-5 - -### P2 - Nice to Have (Low Risks) - -- UI polish tests -- Minor validation tests -- Tests for risks scored ≤3 - -## Test Maintenance Considerations - -### High Maintenance Tests - -[List tests that may need frequent updates] - -### Stability Measures - -- No retry strategies (tests must be deterministic) -- Dynamic waits only (no hard sleeps) -- Environment isolation -- Self-cleaning test data - -## Coverage Goals - -### Unit Test Coverage - -- Target: 80% line coverage -- Focus: Business logic, calculations - -### Integration Coverage - -- Target: All API endpoints -- Focus: Contract validation - -### E2E Coverage - -- Target: Critical paths only -- Focus: User value delivery +1. P0 Unit tests (fail fast) +2. P0 Integration tests +3. P0 E2E tests +4. P1 tests in order +5. P2+ as time permits ``` -## Test Level Smells to Flag +### Output 2: Gate YAML Block -### Over-testing Smells +Generate for inclusion in quality gate: -- Same logic tested at multiple levels -- E2E tests for calculations -- Integration tests for framework features +```yaml +test_design: + scenarios_total: X + by_level: + unit: Y + integration: Z + e2e: W + by_priority: + p0: A + p1: B + p2: C + coverage_gaps: [] # List any ACs without tests +``` -### Under-testing Smells +### Output 3: Trace References -- No unit tests for complex logic -- Missing integration tests for data operations -- No E2E tests for critical user paths +Print for use by trace-requirements task: -### Wrong Level Smells +```text +Test design matrix: docs/qa/assessments/{epic}.{story}-test-design-{YYYYMMDD}.md +P0 tests identified: {count} +``` -- Unit tests with real database -- E2E tests checking calculation results -- Integration tests mocking everything +## Quality Checklist -## Quality Indicators +Before finalizing, verify: -Good test design shows: - -- Clear level separation -- No redundant coverage -- Fast feedback from unit tests -- Reliable integration tests -- Focused e2e tests +- [ ] Every AC has test coverage +- [ ] Test levels are appropriate (not over-testing) +- [ ] No duplicate coverage across levels +- [ ] Priorities align with business risk +- [ ] Test IDs follow naming convention +- [ ] Scenarios are atomic and independent ## Key Principles -- Test at the lowest appropriate level -- One clear owner per test -- Fast tests run first -- Mock at boundaries, not internals -- E2E for user value, not implementation -- Maintain test/production parity where critical -- Tests must be atomic and self-contained -- No shared state between tests -- Explicit assertions in test files (not helpers) - -### Output 2: Story Hook Line - -**Print this line for review task to quote:** - -```text -Test design: docs/qa/assessments/{epic}.{story}-test-design-{YYYYMMDD}.md -``` - -**For traceability:** This planning document will be referenced by trace-requirements task. - -### Output 3: Test Count Summary - -**Print summary for quick reference:** - -```yaml -test_summary: - total: { total_count } - by_level: - unit: { unit_count } - integration: { int_count } - e2e: { e2e_count } - by_priority: - P0: { p0_count } - P1: { p1_count } - P2: { p2_count } - coverage_gaps: [] # List any ACs without tests -``` +- **Shift left**: Prefer unit over integration, integration over E2E +- **Risk-based**: Focus on what could go wrong +- **Efficient coverage**: Test once at the right level +- **Maintainability**: Consider long-term test maintenance +- **Fast feedback**: Quick tests run first diff --git a/bmad-core/templates/qa-gate-tmpl.yaml b/bmad-core/templates/qa-gate-tmpl.yaml index 2632ef03..ae8b8c79 100644 --- a/bmad-core/templates/qa-gate-tmpl.yaml +++ b/bmad-core/templates/qa-gate-tmpl.yaml @@ -29,65 +29,74 @@ risk_summary: must_fix: [] monitor: [] -# Example with issues: -# top_issues: -# - id: "SEC-001" -# severity: high # ONLY: low|medium|high -# finding: "No rate limiting on login endpoint" -# suggested_action: "Add rate limiting middleware before production" -# - id: "TEST-001" -# severity: medium -# finding: "Missing integration tests for auth flow" -# suggested_action: "Add test coverage for critical paths" - -# Example when waived: -# waiver: -# active: true -# reason: "Accepted for MVP release - will address in next sprint" -# approved_by: "Product Owner" +# Examples section using block scalars for clarity +examples: + with_issues: | + top_issues: + - id: "SEC-001" + severity: high # ONLY: low|medium|high + finding: "No rate limiting on login endpoint" + suggested_action: "Add rate limiting middleware before production" + - id: "TEST-001" + severity: medium + finding: "Missing integration tests for auth flow" + suggested_action: "Add test coverage for critical paths" + + when_waived: | + waiver: + active: true + reason: "Accepted for MVP release - will address in next sprint" + approved_by: "Product Owner" # ============ Optional Extended Fields ============ # Uncomment and use if your team wants more detail -# quality_score: 75 # 0-100 (optional scoring) -# expires: "2025-01-26T00:00:00Z" # Optional gate freshness window - -# evidence: -# tests_reviewed: 15 -# risks_identified: 3 -# trace: -# ac_covered: [1, 2, 3] # AC numbers with test coverage -# ac_gaps: [4] # AC numbers lacking coverage - -# nfr_validation: -# security: { status: CONCERNS, notes: "Rate limiting missing" } -# performance: { status: PASS, notes: "" } -# reliability: { status: PASS, notes: "" } -# maintainability: { status: PASS, notes: "" } - -# history: # Append-only audit trail -# - at: "2025-01-12T10:00:00Z" -# gate: FAIL -# note: "Initial review - missing tests" -# - at: "2025-01-12T15:00:00Z" -# gate: CONCERNS -# note: "Tests added but rate limiting still missing" - -# risk_summary: # From risk-profile task -# totals: -# critical: 0 -# high: 0 -# medium: 0 -# low: 0 -# # 'highest' is emitted only when risks exist -# recommendations: -# must_fix: [] -# monitor: [] - -# recommendations: -# immediate: # Must fix before production -# - action: "Add rate limiting to auth endpoints" -# refs: ["api/auth/login.ts:42-68"] -# future: # Can be addressed later -# - action: "Consider caching for better performance" -# refs: ["services/data.service.ts"] \ No newline at end of file +optional_fields_examples: + quality_and_expiry: | + quality_score: 75 # 0-100 (optional scoring) + expires: "2025-01-26T00:00:00Z" # Optional gate freshness window + + evidence: | + evidence: + tests_reviewed: 15 + risks_identified: 3 + trace: + ac_covered: [1, 2, 3] # AC numbers with test coverage + ac_gaps: [4] # AC numbers lacking coverage + + nfr_validation: | + nfr_validation: + security: { status: CONCERNS, notes: "Rate limiting missing" } + performance: { status: PASS, notes: "" } + reliability: { status: PASS, notes: "" } + maintainability: { status: PASS, notes: "" } + + history: | + history: # Append-only audit trail + - at: "2025-01-12T10:00:00Z" + gate: FAIL + note: "Initial review - missing tests" + - at: "2025-01-12T15:00:00Z" + gate: CONCERNS + note: "Tests added but rate limiting still missing" + + risk_summary: | + risk_summary: # From risk-profile task + totals: + critical: 0 + high: 0 + medium: 0 + low: 0 + # 'highest' is emitted only when risks exist + recommendations: + must_fix: [] + monitor: [] + + recommendations: | + recommendations: + immediate: # Must fix before production + - action: "Add rate limiting to auth endpoints" + refs: ["api/auth/login.ts:42-68"] + future: # Can be addressed later + - action: "Consider caching for better performance" + refs: ["services/data.service.ts"] \ No newline at end of file