diff --git a/bmad-core/agents/qa.md b/bmad-core/agents/qa.md index 892f3da6..3de910a0 100644 --- a/bmad-core/agents/qa.md +++ b/bmad-core/agents/qa.md @@ -30,26 +30,30 @@ activation-instructions: agent: name: Quinn id: qa - title: Senior Developer & QA Architect + title: Test Architect & Quality Advisor icon: ๐Ÿงช - whenToUse: Use for senior code review, refactoring, test planning, quality assurance, and mentoring through code improvements + whenToUse: | + Use for comprehensive test architecture review, quality gate decisions, + and code improvement. Provides thorough analysis including requirements + traceability, risk assessment, and test strategy. + Advisory only - teams choose their quality bar. customization: null persona: - role: Senior Developer & Test Architect - style: Methodical, detail-oriented, quality-focused, mentoring, strategic - identity: Senior developer with deep expertise in code quality, architecture, and test automation - focus: Code excellence through review, refactoring, and comprehensive testing strategies + role: Test Architect with Quality Advisory Authority + style: Comprehensive, systematic, advisory, educational, pragmatic + identity: Test architect who provides thorough quality assessment and actionable recommendations without blocking progress + focus: Comprehensive quality analysis through test architecture, risk assessment, and advisory gates core_principles: - - Senior Developer Mindset - Review and improve code as a senior mentoring juniors - - Active Refactoring - Don't just identify issues, fix them with clear explanations - - Test Strategy & Architecture - Design holistic testing strategies across all levels - - Code Quality Excellence - Enforce best practices, patterns, and clean code principles - - Shift-Left Testing - Integrate testing early in development lifecycle - - Performance & Security - Proactively identify and fix performance/security issues - - Mentorship Through Action - Explain WHY and HOW when making improvements - - Risk-Based Testing - Prioritize testing based on risk and critical areas - - Continuous Improvement - Balance perfection with pragmatism - - Architecture & Design Patterns - Ensure proper patterns and maintainable code structure + - Depth As Needed - Go deep based on risk signals, stay concise when low risk + - Requirements Traceability - Map all stories to tests using Given-When-Then patterns + - Risk-Based Testing - Assess and prioritize by probability ร— impact + - Quality Attributes - Validate NFRs (security, performance, reliability) via scenarios + - Testability Assessment - Evaluate controllability, observability, debuggability + - Gate Governance - Provide clear PASS/CONCERNS/FAIL/WAIVED decisions with rationale + - Advisory Excellence - Educate through documentation, never block arbitrarily + - Technical Debt Awareness - Identify and quantify debt with improvement suggestions + - LLM Acceleration - Use LLMs to accelerate thorough yet focused analysis + - Pragmatic Balance - Distinguish must-fix from nice-to-have improvements story-file-permissions: - CRITICAL: When reviewing stories, you are ONLY authorized to update the "QA Results" section of story files - CRITICAL: DO NOT modify any other sections including Status, Story, Acceptance Criteria, Tasks/Subtasks, Dev Notes, Testing, Dev Agent Record, Change Log, or any other sections @@ -57,13 +61,28 @@ story-file-permissions: # All commands require * prefix when used (e.g., *help) commands: - help: Show numbered list of the following commands to allow selection - - review {story}: execute the task review-story for the highest sequence story in docs/stories unless another is specified - keep any specified technical-preferences in mind as needed - - exit: Say goodbye as the QA Engineer, and then abandon inhabiting this persona + - review {story}: | + Adaptive, risk-aware comprehensive review. + Produces: QA Results update in story file + gate file (PASS/CONCERNS/FAIL/WAIVED). + Gate file location: docs/qa/gates/{epic}.{story}-{slug}.yml + Executes review-story task which includes all analysis and creates gate decision. + - gate {story}: Execute qa-gate task to write/update quality gate decision in docs/qa/gates/ + - trace {story}: Execute trace-requirements task to map requirements to tests using Given-When-Then + - risk-profile {story}: Execute risk-profile task to generate risk assessment matrix + - test-design {story}: Execute test-design task to create comprehensive test scenarios + - nfr-assess {story}: Execute nfr-assess task to validate non-functional requirements + - exit: Say goodbye as the Test Architect, and then abandon inhabiting this persona dependencies: tasks: - review-story.md + - qa-gate.md + - trace-requirements.md + - risk-profile.md + - test-design.md + - nfr-assess.md data: - technical-preferences.md templates: - story-tmpl.yaml + - qa-gate-tmpl.yaml ``` diff --git a/bmad-core/tasks/nfr-assess.md b/bmad-core/tasks/nfr-assess.md new file mode 100644 index 00000000..6b77526c --- /dev/null +++ b/bmad-core/tasks/nfr-assess.md @@ -0,0 +1,315 @@ +# nfr-assess + +Quick NFR validation focused on the core four: security, performance, reliability, maintainability. + +## Inputs + +```yaml +required: + - story_id: "{epic}.{story}" # e.g., "1.3" + - story_path: "docs/stories/{epic}.{story}.*.md" + +optional: + - architecture_refs: "docs/architecture/*.md" + - technical_preferences: "docs/technical-preferences.md" + - acceptance_criteria: From story file +``` + +## Purpose + +Assess non-functional requirements for a story and generate: +1. YAML block for the gate file's `nfr_validation` section +2. Brief markdown assessment saved to `docs/qa/assessments/{epic}.{story}-nfr-{YYYYMMDD}.md` + +## Process + +### 0. Fail-safe for Missing Inputs + +If story_path or story file can't be found: +- Still create assessment file with note: "Source story not found" +- Set all selected NFRs to CONCERNS with notes: "Target unknown / evidence missing" +- Continue with assessment to provide value + +### 1. Elicit Scope + +**Interactive mode:** Ask which NFRs to assess +**Non-interactive mode:** Default to core four (security, performance, reliability, maintainability) + +```text +Which NFRs should I assess? (Enter numbers or press Enter for default) +[1] Security (default) +[2] Performance (default) +[3] Reliability (default) +[4] Maintainability (default) +[5] Usability +[6] Compatibility +[7] Portability +[8] Functional Suitability + +> [Enter for 1-4] +``` + +### 2. Check for Thresholds + +Look for NFR requirements in: +- Story acceptance criteria +- `docs/architecture/*.md` files +- `docs/technical-preferences.md` + +**Interactive mode:** Ask for missing thresholds +**Non-interactive mode:** Mark as CONCERNS with "Target unknown" + +```text +No performance requirements found. What's your target response time? +> 200ms for API calls + +No security requirements found. Required auth method? +> JWT with refresh tokens +``` + +**Unknown targets policy:** If a target is missing and not provided, mark status as CONCERNS with notes: "Target unknown" + +### 3. Quick Assessment + +For each selected NFR, check: +- Is there evidence it's implemented? +- Can we validate it? +- Are there obvious gaps? + +### 4. Generate Outputs + +## Output 1: Gate YAML Block + +Generate ONLY for NFRs actually assessed (no placeholders): + +```yaml +# Gate YAML (copy/paste): +nfr_validation: + _assessed: [security, performance, reliability, maintainability] + security: + status: CONCERNS + notes: "No rate limiting on auth endpoints" + performance: + status: PASS + notes: "Response times < 200ms verified" + reliability: + status: PASS + notes: "Error handling and retries implemented" + maintainability: + status: CONCERNS + notes: "Test coverage at 65%, target is 80%" +``` + +## Deterministic Status Rules + +- **FAIL**: Any selected NFR has critical gap or target clearly not met +- **CONCERNS**: No FAILs, but any NFR is unknown/partial/missing evidence +- **PASS**: All selected NFRs meet targets with evidence + +## Quality Score Calculation + +``` +quality_score = 100 +- 20 for each FAIL attribute +- 10 for each CONCERNS attribute +Floor at 0, ceiling at 100 +``` + +If `technical-preferences.md` defines custom weights, use those instead. + +## Output 2: Brief Assessment Report + +**ALWAYS save to:** `docs/qa/assessments/{epic}.{story}-nfr-{YYYYMMDD}.md` + +```markdown +# NFR Assessment: {epic}.{story} +Date: {date} +Reviewer: Quinn + + + +## Summary +- Security: CONCERNS - Missing rate limiting +- Performance: PASS - Meets <200ms requirement +- Reliability: PASS - Proper error handling +- Maintainability: CONCERNS - Test coverage below target + +## Critical Issues +1. **No rate limiting** (Security) + - Risk: Brute force attacks possible + - Fix: Add rate limiting middleware to auth endpoints + +2. **Test coverage 65%** (Maintainability) + - Risk: Untested code paths + - Fix: Add tests for uncovered branches + +## Quick Wins +- Add rate limiting: ~2 hours +- Increase test coverage: ~4 hours +- Add performance monitoring: ~1 hour +``` + +## Output 3: Story Update Line + +**End with this line for the review task to quote:** +``` +NFR assessment: docs/qa/assessments/{epic}.{story}-nfr-{YYYYMMDD}.md +``` + +## Output 4: Gate Integration Line + +**Always print at the end:** +``` +Gate NFR block ready โ†’ paste into docs/qa/gates/{epic}.{story}-{slug}.yml under nfr_validation +``` + +## Assessment Criteria + +### Security +**PASS if:** +- Authentication implemented +- Authorization enforced +- Input validation present +- No hardcoded secrets + +**CONCERNS if:** +- Missing rate limiting +- Weak encryption +- Incomplete authorization + +**FAIL if:** +- No authentication +- Hardcoded credentials +- SQL injection vulnerabilities + +### Performance +**PASS if:** +- Meets response time targets +- No obvious bottlenecks +- Reasonable resource usage + +**CONCERNS if:** +- Close to limits +- Missing indexes +- No caching strategy + +**FAIL if:** +- Exceeds response time limits +- Memory leaks +- Unoptimized queries + +### Reliability +**PASS if:** +- Error handling present +- Graceful degradation +- Retry logic where needed + +**CONCERNS if:** +- Some error cases unhandled +- No circuit breakers +- Missing health checks + +**FAIL if:** +- No error handling +- Crashes on errors +- No recovery mechanisms + +### Maintainability +**PASS if:** +- Test coverage meets target +- Code well-structured +- Documentation present + +**CONCERNS if:** +- Test coverage below target +- Some code duplication +- Missing documentation + +**FAIL if:** +- No tests +- Highly coupled code +- No documentation + +## Quick Reference + +### What to Check + +```yaml +security: + - Authentication mechanism + - Authorization checks + - Input validation + - Secret management + - Rate limiting + +performance: + - Response times + - Database queries + - Caching usage + - Resource consumption + +reliability: + - Error handling + - Retry logic + - Circuit breakers + - Health checks + - Logging + +maintainability: + - Test coverage + - Code structure + - Documentation + - Dependencies +``` + +## Key Principles + +- Focus on the core four NFRs by default +- Quick assessment, not deep analysis +- Gate-ready output format +- Brief, actionable findings +- Skip what doesn't apply +- Deterministic status rules for consistency +- Unknown targets โ†’ CONCERNS, not guesses + +--- + +## Appendix: ISO 25010 Reference + +
+Full ISO 25010 Quality Model (click to expand) + +### All 8 Quality Characteristics + +1. **Functional Suitability**: Completeness, correctness, appropriateness +2. **Performance Efficiency**: Time behavior, resource use, capacity +3. **Compatibility**: Co-existence, interoperability +4. **Usability**: Learnability, operability, accessibility +5. **Reliability**: Maturity, availability, fault tolerance +6. **Security**: Confidentiality, integrity, authenticity +7. **Maintainability**: Modularity, reusability, testability +8. **Portability**: Adaptability, installability + +Use these when assessing beyond the core four. +
+ +
+Example: Deep Performance Analysis (click to expand) + +```yaml +performance_deep_dive: + response_times: + p50: 45ms + p95: 180ms + p99: 350ms + database: + slow_queries: 2 + missing_indexes: ["users.email", "orders.user_id"] + caching: + hit_rate: 0% + recommendation: "Add Redis for session data" + load_test: + max_rps: 150 + breaking_point: 200 rps +``` +
\ No newline at end of file diff --git a/bmad-core/tasks/qa-gate.md b/bmad-core/tasks/qa-gate.md new file mode 100644 index 00000000..0448aa3b --- /dev/null +++ b/bmad-core/tasks/qa-gate.md @@ -0,0 +1,151 @@ +# qa-gate + +Create or update a quality gate decision file for a story based on review findings. + +## Purpose + +Generate a standalone quality gate file that provides a clear pass/fail decision with actionable feedback. This gate serves as an advisory checkpoint for teams to understand quality status. + +## Prerequisites + +- Story has been reviewed (manually or via review-story task) +- Review findings are available +- Understanding of story requirements and implementation + +## Gate File Location + +**ALWAYS** create file at: `docs/qa/gates/{epic}.{story}-{slug}.yml` + +Slug rules: +- Convert to lowercase +- Replace spaces with hyphens +- Strip punctuation +- Example: "User Auth - Login!" becomes "user-auth-login" + +## Minimal Required Schema + +```yaml +schema: 1 +story: "{epic}.{story}" +gate: PASS|CONCERNS|FAIL|WAIVED +status_reason: "1-2 sentence explanation of gate decision" +reviewer: "Quinn" +updated: "{ISO-8601 timestamp}" +top_issues: [] # Empty array if no issues +waiver: { active: false } # Only set active: true if WAIVED +``` + +## Schema with Issues + +```yaml +schema: 1 +story: "1.3" +gate: CONCERNS +status_reason: "Missing rate limiting on auth endpoints poses security risk." +reviewer: "Quinn" +updated: "2025-01-12T10:15:00Z" +top_issues: + - id: "SEC-001" + severity: high # ONLY: low|medium|high + finding: "No rate limiting on login endpoint" + suggested_action: "Add rate limiting middleware before production" + - id: "TEST-001" + severity: medium + finding: "No integration tests for auth flow" + suggested_action: "Add integration test coverage" +waiver: { active: false } +``` + +## Schema when Waived + +```yaml +schema: 1 +story: "1.3" +gate: WAIVED +status_reason: "Known issues accepted for MVP release." +reviewer: "Quinn" +updated: "2025-01-12T10:15:00Z" +top_issues: + - id: "PERF-001" + severity: low + finding: "Dashboard loads slowly with 1000+ items" + suggested_action: "Implement pagination in next sprint" +waiver: + active: true + reason: "MVP release - performance optimization deferred" + approved_by: "Product Owner" +``` + +## Gate Decision Criteria + +### PASS +- All acceptance criteria met +- No high-severity issues +- Test coverage meets project standards + +### CONCERNS +- Non-blocking issues present +- Should be tracked and scheduled +- Can proceed with awareness + +### FAIL +- Acceptance criteria not met +- High-severity issues present +- Recommend return to InProgress + +### WAIVED +- Issues explicitly accepted +- Requires approval and reason +- Proceed despite known issues + +## Severity Scale + +**FIXED VALUES - NO VARIATIONS:** +- `low`: Minor issues, cosmetic problems +- `medium`: Should fix soon, not blocking +- `high`: Critical issues, should block release + +## Issue ID Prefixes + +- `SEC-`: Security issues +- `PERF-`: Performance issues +- `REL-`: Reliability issues +- `TEST-`: Testing gaps +- `MNT-`: Maintainability concerns +- `ARCH-`: Architecture issues +- `DOC-`: Documentation gaps +- `REQ-`: Requirements issues + +## Output Requirements + +1. **ALWAYS** create gate file at: `docs/qa/gates/{epic}.{story}-{slug}.yml` +2. **ALWAYS** append this exact format to story's QA Results section: + ``` + Gate: {STATUS} โ†’ docs/qa/gates/{epic}.{story}-{slug}.yml + ``` +3. Keep status_reason to 1-2 sentences maximum +4. Use severity values exactly: `low`, `medium`, or `high` + +## Example Story Update + +After creating gate file, append to story's QA Results section: + +```markdown +## QA Results + +### Review Date: 2025-01-12 +### Reviewed By: Quinn (Test Architect) + +[... existing review content ...] + +### Gate Status +Gate: CONCERNS โ†’ docs/qa/gates/1.3-user-auth-login.yml +``` + +## Key Principles + +- Keep it minimal and predictable +- Fixed severity scale (low/medium/high) +- Always write to standard path +- Always update story with gate reference +- Clear, actionable findings \ No newline at end of file diff --git a/bmad-core/tasks/review-story.md b/bmad-core/tasks/review-story.md index 16ff8ad4..9f2abc19 100644 --- a/bmad-core/tasks/review-story.md +++ b/bmad-core/tasks/review-story.md @@ -1,6 +1,16 @@ # review-story -When a developer agent marks a story as "Ready for Review", perform a comprehensive senior developer code review with the ability to refactor and improve code directly. +Perform a comprehensive test architecture review with quality gate decision. This adaptive, risk-aware review creates both a story update and a detailed gate file. + +## Inputs + +```yaml +required: + - story_id: "{epic}.{story}" # e.g., "1.3" + - story_path: "docs/stories/{epic}.{story}.*.md" + - story_title: "{title}" # If missing, derive from story file H1 + - story_slug: "{slug}" # If missing, derive from title (lowercase, hyphenated) +``` ## Prerequisites @@ -8,80 +18,102 @@ When a developer agent marks a story as "Ready for Review", perform a comprehens - Developer has completed all tasks and updated the File List - All automated tests are passing -## Review Process +## Review Process - Adaptive Test Architecture -1. **Read the Complete Story** - - Review all acceptance criteria - - Understand the dev notes and requirements - - Note any completion notes from the developer +### 1. Risk Assessment (Determines Review Depth) -2. **Verify Implementation Against Dev Notes Guidance** - - Review the "Dev Notes" section for specific technical guidance provided to the developer - - Verify the developer's implementation follows the architectural patterns specified in Dev Notes - - Check that file locations match the project structure guidance in Dev Notes - - Confirm any specified libraries, frameworks, or technical approaches were used correctly - - Validate that security considerations mentioned in Dev Notes were implemented +**Auto-escalate to deep review when:** +- Auth/payment/security files touched +- No tests added to story +- Diff > 500 lines +- Previous gate was FAIL/CONCERNS +- Story has > 5 acceptance criteria -3. **Focus on the File List** - - Verify all files listed were actually created/modified - - Check for any missing files that should have been updated - - Ensure file locations align with the project structure guidance from Dev Notes +### 2. Comprehensive Analysis -4. **Senior Developer Code Review** - - Review code with the eye of a senior developer - - If changes form a cohesive whole, review them together - - If changes are independent, review incrementally file by file - - Focus on: - - Code architecture and design patterns - - Refactoring opportunities - - Code duplication or inefficiencies - - Performance optimizations - - Security concerns - - Best practices and patterns +**A. Requirements Traceability** -5. **Active Refactoring** - - As a senior developer, you CAN and SHOULD refactor code where improvements are needed - - When refactoring: - - Make the changes directly in the files - - Explain WHY you're making the change - - Describe HOW the change improves the code - - Ensure all tests still pass after refactoring - - Update the File List if you modify additional files +- Map each acceptance criteria to its validating tests (document mapping with Given-When-Then, not test code) +- Identify coverage gaps +- Verify all requirements have corresponding test cases -6. **Standards Compliance Check** - - Verify adherence to `docs/coding-standards.md` - - Check compliance with `docs/unified-project-structure.md` - - Validate testing approach against `docs/testing-strategy.md` - - Ensure all guidelines mentioned in the story are followed +**B. Code Quality Review** +- Architecture and design patterns +- Refactoring opportunities (and perform them) +- Code duplication or inefficiencies +- Performance optimizations +- Security vulnerabilities +- Best practices adherence -7. **Acceptance Criteria Validation** - - Verify each AC is fully implemented - - Check for any missing functionality - - Validate edge cases are handled +**C. Test Architecture Assessment** +- Test coverage adequacy at appropriate levels +- Test level appropriateness (what should be unit vs integration vs e2e) +- Test design quality and maintainability +- Test data management strategy +- Mock/stub usage appropriateness +- Edge case and error scenario coverage +- Test execution time and reliability -8. **Test Coverage Review** - - Ensure unit tests cover edge cases - - Add missing tests if critical coverage is lacking - - Verify integration tests (if required) are comprehensive - - Check that test assertions are meaningful - - Look for missing test scenarios +**D. Non-Functional Requirements (NFRs)** +- Security: Authentication, authorization, data protection +- Performance: Response times, resource usage +- Reliability: Error handling, recovery mechanisms +- Maintainability: Code clarity, documentation -9. **Documentation and Comments** - - Verify code is self-documenting where possible - - Add comments for complex logic if missing - - Ensure any API changes are documented +**E. Testability Evaluation** +- Controllability: Can we control the inputs? +- Observability: Can we observe the outputs? +- Debuggability: Can we debug failures easily? -## Update Story File - QA Results Section ONLY +**F. Technical Debt Identification** +- Accumulated shortcuts +- Missing tests +- Outdated dependencies +- Architecture violations + +### 3. Active Refactoring + +- Refactor code where safe and appropriate +- Run tests to ensure changes don't break functionality +- Document all changes in QA Results section with clear WHY and HOW +- Do NOT alter story content beyond QA Results section +- Do NOT change story Status or File List; recommend next status only + +### 4. Standards Compliance Check + +- Verify adherence to `docs/coding-standards.md` +- Check compliance with `docs/unified-project-structure.md` +- Validate testing approach against `docs/testing-strategy.md` +- Ensure all guidelines mentioned in the story are followed + +### 5. Acceptance Criteria Validation + +- Verify each AC is fully implemented +- Check for any missing functionality +- Validate edge cases are handled + +### 6. Documentation and Comments + +- Verify code is self-documenting where possible +- Add comments for complex logic if missing +- Ensure any API changes are documented + +## Output 1: Update Story File - QA Results Section ONLY **CRITICAL**: You are ONLY authorized to update the "QA Results" section of the story file. DO NOT modify any other sections. +**QA Results Anchor Rule:** +- If `## QA Results` doesn't exist, append it at end of file +- If it exists, append a new dated entry below existing entries +- Never edit other sections + After review and any refactoring, append your results to the story file in the QA Results section: ```markdown ## QA Results ### Review Date: [Date] -### Reviewed By: Quinn (Senior Developer QA) +### Reviewed By: Quinn (Test Architect) ### Code Quality Assessment [Overall assessment of implementation quality] @@ -114,17 +146,130 @@ After review and any refactoring, append your results to the story file in the Q ### Performance Considerations [Any performance issues found and whether addressed] -### Final Status -[โœ“ Approved - Ready for Done] / [โœ— Changes Required - See unchecked items above] +### Files Modified During Review +[If you modified files, list them here - ask Dev to update File List] + +### Gate Status +Gate: {STATUS} โ†’ docs/qa/gates/{epic}.{story}-{slug}.yml +Risk profile: docs/qa/assessments/{epic}.{story}-risk-{YYYYMMDD}.md +NFR assessment: docs/qa/assessments/{epic}.{story}-nfr-{YYYYMMDD}.md + +### Recommended Status +[โœ“ Ready for Done] / [โœ— Changes Required - See unchecked items above] +(Story owner decides final status) ``` +## Output 2: Create Quality Gate File + +**Template and Directory:** +- Render from `templates/qa-gate-tmpl.yaml` +- Create `docs/qa/gates/` directory if missing +- Save to: `docs/qa/gates/{epic}.{story}-{slug}.yml` + +Gate file structure: + +```yaml +schema: 1 +story: "{epic}.{story}" +story_title: "{story title}" +gate: PASS|CONCERNS|FAIL|WAIVED +status_reason: "1-2 sentence explanation of gate decision" +reviewer: "Quinn (Test Architect)" +updated: "{ISO-8601 timestamp}" + +top_issues: [] # Empty if no issues +waiver: { active: false } # Set active: true only if WAIVED + +# Extended fields (optional but recommended): +quality_score: 0-100 # 100 - (20*FAILs) - (10*CONCERNS) or use technical-preferences.md weights +expires: "{ISO-8601 timestamp}" # Typically 2 weeks from review + +evidence: + tests_reviewed: {count} + risks_identified: {count} + trace: + ac_covered: [1, 2, 3] # AC numbers with test coverage + ac_gaps: [4] # AC numbers lacking coverage + +nfr_validation: + security: + status: PASS|CONCERNS|FAIL + notes: "Specific findings" + performance: + status: PASS|CONCERNS|FAIL + notes: "Specific findings" + reliability: + status: PASS|CONCERNS|FAIL + notes: "Specific findings" + maintainability: + status: PASS|CONCERNS|FAIL + notes: "Specific findings" + +recommendations: + immediate: # Must fix before production + - action: "Add rate limiting" + refs: ["api/auth/login.ts"] + future: # Can be addressed later + - action: "Consider caching" + refs: ["services/data.ts"] +``` + +### Gate Decision Criteria + +**Deterministic rule (apply in order):** + +If risk_summary exists, apply its thresholds first (โ‰ฅ9 โ†’ FAIL, โ‰ฅ6 โ†’ CONCERNS), then NFR statuses, then top_issues severity. + +1. **Risk thresholds (if risk_summary present):** + - If any risk score โ‰ฅ 9 โ†’ Gate = FAIL (unless waived) + - Else if any score โ‰ฅ 6 โ†’ Gate = CONCERNS + +2. **Test coverage gaps (if trace available):** + - If any P0 test from test-design is missing โ†’ Gate = CONCERNS + - If security/data-loss P0 test missing โ†’ Gate = FAIL + +3. **Issue severity:** + - If any `top_issues.severity == high` โ†’ Gate = FAIL (unless waived) + - Else if any `severity == medium` โ†’ Gate = CONCERNS + +4. **NFR statuses:** + - If any NFR status is FAIL โ†’ Gate = FAIL + - Else if any NFR status is CONCERNS โ†’ Gate = CONCERNS + - Else โ†’ Gate = PASS + +- WAIVED only when waiver.active: true with reason/approver + +Detailed criteria: +- **PASS**: All critical requirements met, no blocking issues +- **CONCERNS**: Non-critical issues found, team should review +- **FAIL**: Critical issues that should be addressed +- **WAIVED**: Issues acknowledged but explicitly waived by team + +### Quality Score Calculation + +```text +quality_score = 100 - (20 ร— number of FAILs) - (10 ร— number of CONCERNS) +Bounded between 0 and 100 +``` + +If `technical-preferences.md` defines custom weights, use those instead. + +### Suggested Owner Convention + +For each issue in `top_issues`, include a `suggested_owner`: + +- `dev`: Code changes needed +- `sm`: Requirements clarification needed +- `po`: Business decision needed + ## Key Principles -- You are a SENIOR developer reviewing junior/mid-level work -- You have the authority and responsibility to improve code directly +- You are a Test Architect providing comprehensive quality assessment +- You have the authority to improve code directly when appropriate - Always explain your changes for learning purposes - Balance between perfection and pragmatism -- Focus on significant improvements, not nitpicks +- Focus on risk-based prioritization +- Provide actionable recommendations with clear ownership ## Blocking Conditions @@ -140,6 +285,8 @@ Stop the review and request clarification if: After review: -1. If all items are checked and approved: Update story status to "Done" -2. If unchecked items remain: Keep status as "Review" for dev to address -3. Always provide constructive feedback and explanations for learning \ No newline at end of file +1. Update the QA Results section in the story file +2. Create the gate file in `docs/qa/gates/` +3. Recommend status: "Ready for Done" or "Changes Required" (owner decides) +4. If files were modified, list them in QA Results and ask Dev to update File List +5. Always provide constructive feedback and actionable recommendations diff --git a/bmad-core/tasks/risk-profile.md b/bmad-core/tasks/risk-profile.md new file mode 100644 index 00000000..5882c849 --- /dev/null +++ b/bmad-core/tasks/risk-profile.md @@ -0,0 +1,353 @@ +# risk-profile + +Generate a comprehensive risk assessment matrix for a story implementation using probability ร— impact analysis. + +## Inputs + +```yaml +required: + - story_id: "{epic}.{story}" # e.g., "1.3" + - story_path: "docs/stories/{epic}.{story}.*.md" + - story_title: "{title}" # If missing, derive from story file H1 + - story_slug: "{slug}" # If missing, derive from title (lowercase, hyphenated) +``` + +## Purpose + +Identify, assess, and prioritize risks in the story implementation. Provide risk mitigation strategies and testing focus areas based on risk levels. + +## Risk Assessment Framework + +### Risk Categories + +**Category Prefixes:** + +- `TECH`: Technical Risks +- `SEC`: Security Risks +- `PERF`: Performance Risks +- `DATA`: Data Risks +- `BUS`: Business Risks +- `OPS`: Operational Risks + +1. **Technical Risks (TECH)** + - Architecture complexity + - Integration challenges + - Technical debt + - Scalability concerns + - System dependencies + +2. **Security Risks (SEC)** + - Authentication/authorization flaws + - Data exposure vulnerabilities + - Injection attacks + - Session management issues + - Cryptographic weaknesses + +3. **Performance Risks (PERF)** + - Response time degradation + - Throughput bottlenecks + - Resource exhaustion + - Database query optimization + - Caching failures + +4. **Data Risks (DATA)** + - Data loss potential + - Data corruption + - Privacy violations + - Compliance issues + - Backup/recovery gaps + +5. **Business Risks (BUS)** + - Feature doesn't meet user needs + - Revenue impact + - Reputation damage + - Regulatory non-compliance + - Market timing + +6. **Operational Risks (OPS)** + - Deployment failures + - Monitoring gaps + - Incident response readiness + - Documentation inadequacy + - Knowledge transfer issues + +## Risk Analysis Process + +### 1. Risk Identification + +For each category, identify specific risks: + +```yaml +risk: + id: "SEC-001" # Use prefixes: SEC, PERF, DATA, BUS, OPS, TECH + category: security + title: "Insufficient input validation on user forms" + description: "Form inputs not properly sanitized could lead to XSS attacks" + affected_components: + - "UserRegistrationForm" + - "ProfileUpdateForm" + detection_method: "Code review revealed missing validation" +``` + +### 2. Risk Assessment + +Evaluate each risk using probability ร— impact: + +**Probability Levels:** + +- `High (3)`: Likely to occur (>70% chance) +- `Medium (2)`: Possible occurrence (30-70% chance) +- `Low (1)`: Unlikely to occur (<30% chance) + +**Impact Levels:** + +- `High (3)`: Severe consequences (data breach, system down, major financial loss) +- `Medium (2)`: Moderate consequences (degraded performance, minor data issues) +- `Low (1)`: Minor consequences (cosmetic issues, slight inconvenience) + +**Risk Score = Probability ร— Impact** + +- 9: Critical Risk (Red) +- 6: High Risk (Orange) +- 4: Medium Risk (Yellow) +- 2-3: Low Risk (Green) +- 1: Minimal Risk (Blue) + +### 3. Risk Prioritization + +Create risk matrix: + +```markdown +## Risk Matrix + +| Risk ID | Description | Probability | Impact | Score | Priority | +| -------- | ----------------------- | ----------- | ---------- | ----- | -------- | +| SEC-001 | XSS vulnerability | High (3) | High (3) | 9 | Critical | +| PERF-001 | Slow query on dashboard | Medium (2) | Medium (2) | 4 | Medium | +| DATA-001 | Backup failure | Low (1) | High (3) | 3 | Low | +``` + +### 4. Risk Mitigation Strategies + +For each identified risk, provide mitigation: + +```yaml +mitigation: + risk_id: "SEC-001" + strategy: "preventive" # preventive|detective|corrective + actions: + - "Implement input validation library (e.g., validator.js)" + - "Add CSP headers to prevent XSS execution" + - "Sanitize all user inputs before storage" + - "Escape all outputs in templates" + testing_requirements: + - "Security testing with OWASP ZAP" + - "Manual penetration testing of forms" + - "Unit tests for validation functions" + residual_risk: "Low - Some zero-day vulnerabilities may remain" + owner: "dev" + timeline: "Before deployment" +``` + +## Outputs + +### Output 1: Gate YAML Block + +Generate for pasting into gate file under `risk_summary`: + +**Output rules:** + +- Only include assessed risks; do not emit placeholders +- Sort risks by score (desc) when emitting highest and any tabular lists +- If no risks: totals all zeros, omit highest, keep recommendations arrays empty + +```yaml +# risk_summary (paste into gate file): +risk_summary: + totals: + critical: X # score 9 + high: Y # score 6 + medium: Z # score 4 + low: W # score 2-3 + highest: + id: SEC-001 + score: 9 + title: "XSS on profile form" + recommendations: + must_fix: + - "Add input sanitization & CSP" + monitor: + - "Add security alerts for auth endpoints" +``` + +### Output 2: Markdown Report + +**Save to:** `docs/qa/assessments/{epic}.{story}-risk-{YYYYMMDD}.md` + +```markdown +# Risk Profile: Story {epic}.{story} + +Date: {date} +Reviewer: Quinn (Test Architect) + +## Executive Summary + +- Total Risks Identified: X +- Critical Risks: Y +- High Risks: Z +- Risk Score: XX/100 (calculated) + +## Critical Risks Requiring Immediate Attention + +### 1. [ID]: Risk Title + +**Score: 9 (Critical)** +**Probability**: High - Detailed reasoning +**Impact**: High - Potential consequences +**Mitigation**: + +- Immediate action required +- Specific steps to take + **Testing Focus**: Specific test scenarios needed + +## Risk Distribution + +### By Category + +- Security: X risks (Y critical) +- Performance: X risks (Y critical) +- Data: X risks (Y critical) +- Business: X risks (Y critical) +- Operational: X risks (Y critical) + +### By Component + +- Frontend: X risks +- Backend: X risks +- Database: X risks +- Infrastructure: X risks + +## Detailed Risk Register + +[Full table of all risks with scores and mitigations] + +## Risk-Based Testing Strategy + +### Priority 1: Critical Risk Tests + +- Test scenarios for critical risks +- Required test types (security, load, chaos) +- Test data requirements + +### Priority 2: High Risk Tests + +- Integration test scenarios +- Edge case coverage + +### Priority 3: Medium/Low Risk Tests + +- Standard functional tests +- Regression test suite + +## Risk Acceptance Criteria + +### Must Fix Before Production + +- All critical risks (score 9) +- High risks affecting security/data + +### Can Deploy with Mitigation + +- Medium risks with compensating controls +- Low risks with monitoring in place + +### Accepted Risks + +- Document any risks team accepts +- Include sign-off from appropriate authority + +## Monitoring Requirements + +Post-deployment monitoring for: + +- Performance metrics for PERF risks +- Security alerts for SEC risks +- Error rates for operational risks +- Business KPIs for business risks + +## Risk Review Triggers + +Review and update risk profile when: + +- Architecture changes significantly +- New integrations added +- Security vulnerabilities discovered +- Performance issues reported +- Regulatory requirements change +``` + +## Risk Scoring Algorithm + +Calculate overall story risk score: + +``` +Base Score = 100 +For each risk: + - Critical (9): Deduct 20 points + - High (6): Deduct 10 points + - Medium (4): Deduct 5 points + - Low (2-3): Deduct 2 points + +Minimum score = 0 (extremely risky) +Maximum score = 100 (minimal risk) +``` + +## Risk-Based Recommendations + +Based on risk profile, recommend: + +1. **Testing Priority** + - Which tests to run first + - Additional test types needed + - Test environment requirements + +2. **Development Focus** + - Code review emphasis areas + - Additional validation needed + - Security controls to implement + +3. **Deployment Strategy** + - Phased rollout for high-risk changes + - Feature flags for risky features + - Rollback procedures + +4. **Monitoring Setup** + - Metrics to track + - Alerts to configure + - Dashboard requirements + +## Integration with Quality Gates + +**Deterministic gate mapping:** + +- Any risk with score โ‰ฅ 9 โ†’ Gate = FAIL (unless waived) +- Else if any score โ‰ฅ 6 โ†’ Gate = CONCERNS +- Else โ†’ Gate = PASS +- Unmitigated risks โ†’ Document in gate + +### Output 3: Story Hook Line + +**Print this line for review task to quote:** + +``` +Risk profile: docs/qa/assessments/{epic}.{story}-risk-{YYYYMMDD}.md +``` + +## Key Principles + +- Identify risks early and systematically +- Use consistent probability ร— impact scoring +- Provide actionable mitigation strategies +- Link risks to specific test requirements +- Track residual risk after mitigation +- Update risk profile as story evolves diff --git a/bmad-core/tasks/test-design.md b/bmad-core/tasks/test-design.md new file mode 100644 index 00000000..81eda507 --- /dev/null +++ b/bmad-core/tasks/test-design.md @@ -0,0 +1,528 @@ +# test-design + +Create comprehensive test scenarios with appropriate test level recommendations for story implementation. + +## Inputs + +```yaml +required: + - story_id: "{epic}.{story}" # e.g., "1.3" + - story_path: "docs/stories/{epic}.{story}.*.md" + - story_title: "{title}" # If missing, derive from story file H1 + - story_slug: "{slug}" # If missing, derive from title (lowercase, hyphenated) +``` + +## Purpose + +Design a complete test strategy that identifies what to test, at which level (unit/integration/e2e), and why. This ensures efficient test coverage without redundancy while maintaining appropriate test boundaries. + +## Test Level Decision Framework + +### Unit Tests + +**When to use:** + +- Testing pure functions and business logic +- Algorithm correctness +- Input validation and data transformation +- Error handling in isolated components +- Complex calculations or state machines + +**Characteristics:** + +- Fast execution (immediate feedback) +- No external dependencies (DB, API, file system) +- Highly maintainable and stable +- Easy to debug failures + +**Example scenarios:** + +```yaml +unit_test: + component: "PriceCalculator" + scenario: "Calculate discount with multiple rules" + justification: "Complex business logic with multiple branches" + mock_requirements: "None - pure function" +``` + +### Integration Tests + +**When to use:** + +- Testing component interactions +- Database operations and queries +- API endpoint behavior +- Service layer orchestration +- External service integration (with test doubles) + +**Characteristics:** + +- Moderate execution time +- May use test databases or containers +- Tests multiple components together +- Validates contracts between components + +**Example scenarios:** + +```yaml +integration_test: + components: ["UserService", "UserRepository", "Database"] + scenario: "Create user with duplicate email check" + justification: "Tests transaction boundaries and constraint handling" + test_doubles: "Mock email service, real test database" +``` + +### End-to-End Tests + +**When to use:** + +- Critical user journeys +- Cross-system workflows +- UI interaction flows +- Full stack validation +- Production-like scenario testing + +**Characteristics:** + +- Keep under 90 seconds per test +- Tests complete user scenarios +- Uses real or production-like environment +- Higher maintenance cost +- More prone to flakiness + +**Example scenarios:** + +```yaml +e2e_test: + flow: "Complete purchase flow" + scenario: "User browses, adds to cart, and completes checkout" + justification: "Critical business flow requiring full stack validation" + environment: "Staging with test payment gateway" +``` + +## Test Design Process + +### 1. Analyze Story Requirements + +Break down each acceptance criterion into testable scenarios: + +```yaml +acceptance_criterion: "User can reset password via email" +test_scenarios: + - level: unit + what: "Password validation rules" + why: "Complex regex and business rules" + + - level: integration + what: "Password reset token generation and storage" + why: "Database interaction with expiry logic" + + - level: integration + what: "Email service integration" + why: "External service with retry logic" + + - level: e2e + what: "Complete password reset flow" + why: "Critical security flow needing full validation" +``` + +### 2. Apply Test Level Heuristics + +Use these rules to determine appropriate test levels: + +```markdown +## Test Level Selection Rules + +### Favor Unit Tests When: + +- Logic can be isolated +- No side effects involved +- Fast feedback needed +- High cyclomatic complexity + +### Favor Integration Tests When: + +- Testing persistence layer +- Validating service contracts +- Testing middleware/interceptors +- Component boundaries critical + +### Favor E2E Tests When: + +- User-facing critical paths +- Multi-system interactions +- Regulatory compliance scenarios +- Visual regression important + +### Anti-patterns to Avoid: + +- E2E testing for business logic validation +- Unit testing framework behavior +- Integration testing third-party libraries +- Duplicate coverage across levels + +### Duplicate Coverage Guard + +**Before adding any test, check:** + +1. Is this already tested at a lower level? +2. Can a unit test cover this instead of integration? +3. Can an integration test cover this instead of E2E? + +**Coverage overlap is only acceptable when:** + +- Testing different aspects (unit: logic, integration: interaction, e2e: user experience) +- Critical paths requiring defense in depth +- Regression prevention for previously broken functionality +``` + +### 3. Design Test Scenarios + +**Test ID Format:** `{EPIC}.{STORY}-{LEVEL}-{SEQ}` + +- Example: `1.3-UNIT-001`, `1.3-INT-002`, `1.3-E2E-001` +- Ensures traceability across all artifacts + +**Naming Convention:** + +- Unit: `test_{component}_{scenario}` +- Integration: `test_{flow}_{interaction}` +- E2E: `test_{journey}_{outcome}` + +**Risk Linkage:** + +- Tag tests with risk IDs they mitigate +- Prioritize tests for high-risk areas (P0) +- Link to risk profile when available + +For each identified test need: + +```yaml +test_scenario: + id: "1.3-INT-002" + requirement: "AC2: Rate limiting on login attempts" + mitigates_risks: ["SEC-001", "PERF-003"] # Links to risk profile + priority: P0 # Based on risk score + + unit_tests: + - name: "RateLimiter calculates window correctly" + input: "Timestamp array" + expected: "Correct window calculation" + + integration_tests: + - name: "Login endpoint enforces rate limit" + setup: "5 failed attempts" + action: "6th attempt" + expected: "429 response with retry-after header" + + e2e_tests: + - name: "User sees rate limit message" + setup: "Trigger rate limit" + validation: "Error message displayed, retry timer shown" +``` + +## Deterministic Test Level Minimums + +**Per Acceptance Criterion:** + +- At least 1 unit test for business logic +- At least 1 integration test if multiple components interact +- At least 1 E2E test if it's a user-facing feature + +**Exceptions:** + +- Pure UI changes: May skip unit tests +- Pure logic changes: May skip E2E tests +- Infrastructure changes: May focus on integration tests + +**When in doubt:** Start with unit tests, add integration for interactions, E2E for critical paths only. + +## Test Quality Standards + +### Core Testing Principles + +**No Flaky Tests:** Ensure reliability through proper async handling, explicit waits, and atomic test design. + +**No Hard Waits/Sleeps:** Use dynamic waiting strategies (e.g., polling, event-based triggers). + +**Stateless & Parallel-Safe:** Tests run independently; use cron jobs or semaphores only if unavoidable. + +**No Order Dependency:** Every it/describe/context block works in isolation (supports .only execution). + +**Self-Cleaning Tests:** Test sets up its own data and automatically deletes/deactivates entities created during testing. + +**Tests Live Near Source Code:** Co-locate test files with the code they validate (e.g., `*.spec.js` alongside components). + +### Execution Strategy + +**Shifted Left:** + +- Start with local environments or ephemeral stacks +- Validate functionality across all deployment stages (local โ†’ dev โ†’ stage) + +**Low Maintenance:** Minimize manual upkeep (avoid brittle selectors, do not repeat UI actions, leverage APIs). + +**CI Execution Evidence:** Integrate into pipelines with clear logs/artifacts. + +**Visibility:** Generate test reports (e.g., JUnit XML, HTML) for failures and trends. + +### Coverage Requirements + +**Release Confidence:** + +- Happy Path: Core user journeys are prioritized +- Edge Cases: Critical error/validation scenarios are covered +- Feature Flags: Test both enabled and disabled states where applicable + +### Test Design Rules + +**Assertions:** Keep them explicit in tests; avoid abstraction into helpers. Use parametrized tests for soft assertions. + +**Naming:** Follow conventions (e.g., `describe('Component')`, `it('should do X when Y')`). + +**Size:** Aim for files โ‰ค200 lines; split/chunk large tests logically. + +**Speed:** Target individual tests โ‰ค90 seconds; optimize slow setups (e.g., shared fixtures). + +**Careful Abstractions:** Favor readability over DRY when balancing helper reuse (page objects are okay, assertion logic is not). + +**Test Cleanup:** Ensure tests clean up resources they create (e.g., closing browser, deleting test data). + +**Deterministic Flow:** Tests should refrain from using conditionals (e.g., if/else) to control flow or try/catch blocks where possible. + +### API Testing Standards + +- Tests must not depend on hardcoded data โ†’ use factories and per-test setup +- Always test both happy path and negative/error cases +- API tests should run parallel safely (no global state shared) +- Test idempotency where applicable (e.g., duplicate requests) +- Tests should clean up their data +- Response logs should only be printed in case of failure +- Auth tests must validate token expiration and renewal + +## Outputs + +### Output 1: Test Design Document + +**Save to:** `docs/qa/assessments/{epic}.{story}-test-design-{YYYYMMDD}.md` + +Generate a comprehensive test design document: + +```markdown +# Test Design: Story {epic}.{story} + +Date: {date} +Reviewer: Quinn (Test Architect) + +## Test Strategy Overview + +- Total test scenarios: X +- Unit tests: Y (A%) +- Integration tests: Z (B%) +- E2E tests: W (C%) + +## Test Level Rationale + +[Explain why this distribution was chosen] + +## Detailed Test Scenarios + +### Requirement: AC1 - {description} + +#### Unit Tests (3 scenarios) + +1. **ID**: 1.3-UNIT-001 + **Test**: Validate input format + - **Why Unit**: Pure validation logic + - **Coverage**: Input edge cases + - **Mocks**: None needed + - **Mitigates**: DATA-001 (if applicable) + +#### Integration Tests (2 scenarios) + +1. **ID**: 1.3-INT-001 + **Test**: Service processes valid request + - **Why Integration**: Multiple components involved + - **Coverage**: Happy path + error handling + - **Test Doubles**: Mock external API + - **Mitigates**: TECH-002 + +#### E2E Tests (1 scenario) + +1. **ID**: 1.3-E2E-001 + **Test**: Complete user workflow + - **Why E2E**: Critical user journey + - **Coverage**: Full stack validation + - **Environment**: Staging + - **Max Duration**: 90 seconds + - **Mitigates**: BUS-001 + +[Continue for all requirements...] + +## Test Data Requirements + +### Unit Test Data + +- Static fixtures for calculations +- Edge case values arrays + +### Integration Test Data + +- Test database seeds +- API response fixtures + +### E2E Test Data + +- Test user accounts +- Sandbox environment data + +## Mock/Stub Strategy + +### What to Mock + +- External services (payment, email) +- Time-dependent functions +- Random number generators + +### What NOT to Mock + +- Core business logic +- Database in integration tests +- Critical security functions + +## Test Execution Implementation + +### Parallel Execution + +- All unit tests: Fully parallel (stateless requirement) +- Integration tests: Parallel with isolated databases +- E2E tests: Sequential or limited parallelism + +### Execution Order + +1. Unit tests first (fail fast) +2. Integration tests second +3. E2E tests last (expensive, max 90 seconds each) + +## Risk-Based Test Priority + +### P0 - Must Have (Linked to Critical/High Risks) + +- Security-related tests (SEC-\* risks) +- Data integrity tests (DATA-\* risks) +- Critical business flow tests (BUS-\* risks) +- Tests for risks scored โ‰ฅ6 in risk profile + +### P1 - Should Have (Medium Risks) + +- Edge case coverage +- Performance tests (PERF-\* risks) +- Error recovery tests +- Tests for risks scored 4-5 + +### P2 - Nice to Have (Low Risks) + +- UI polish tests +- Minor validation tests +- Tests for risks scored โ‰ค3 + +## Test Maintenance Considerations + +### High Maintenance Tests + +[List tests that may need frequent updates] + +### Stability Measures + +- No retry strategies (tests must be deterministic) +- Dynamic waits only (no hard sleeps) +- Environment isolation +- Self-cleaning test data + +## Coverage Goals + +### Unit Test Coverage + +- Target: 80% line coverage +- Focus: Business logic, calculations + +### Integration Coverage + +- Target: All API endpoints +- Focus: Contract validation + +### E2E Coverage + +- Target: Critical paths only +- Focus: User value delivery +``` + +## Test Level Smells to Flag + +### Over-testing Smells + +- Same logic tested at multiple levels +- E2E tests for calculations +- Integration tests for framework features + +### Under-testing Smells + +- No unit tests for complex logic +- Missing integration tests for data operations +- No E2E tests for critical user paths + +### Wrong Level Smells + +- Unit tests with real database +- E2E tests checking calculation results +- Integration tests mocking everything + +## Quality Indicators + +Good test design shows: + +- Clear level separation +- No redundant coverage +- Fast feedback from unit tests +- Reliable integration tests +- Focused e2e tests + +## Key Principles + +- Test at the lowest appropriate level +- One clear owner per test +- Fast tests run first +- Mock at boundaries, not internals +- E2E for user value, not implementation +- Maintain test/production parity where critical +- Tests must be atomic and self-contained +- No shared state between tests +- Explicit assertions in test files (not helpers) + +### Output 2: Story Hook Line + +**Print this line for review task to quote:** + +```text +Test design: docs/qa/assessments/{epic}.{story}-test-design-{YYYYMMDD}.md +``` + +**For traceability:** This planning document will be referenced by trace-requirements task. + +### Output 3: Test Count Summary + +**Print summary for quick reference:** + +```yaml +test_summary: + total: { total_count } + by_level: + unit: { unit_count } + integration: { int_count } + e2e: { e2e_count } + by_priority: + P0: { p0_count } + P1: { p1_count } + P2: { p2_count } + coverage_gaps: [] # List any ACs without tests +``` diff --git a/bmad-core/tasks/trace-requirements.md b/bmad-core/tasks/trace-requirements.md new file mode 100644 index 00000000..b41ffdcb --- /dev/null +++ b/bmad-core/tasks/trace-requirements.md @@ -0,0 +1,264 @@ +# trace-requirements + +Map story requirements to test cases using Given-When-Then patterns for comprehensive traceability. + +## Purpose + +Create a requirements traceability matrix that ensures every acceptance criterion has corresponding test coverage. This task helps identify gaps in testing and ensures all requirements are validated. + +**IMPORTANT**: Given-When-Then is used here for documenting the mapping between requirements and tests, NOT for writing the actual test code. Tests should follow your project's testing standards (no BDD syntax in test code). + +## Prerequisites + +- Story file with clear acceptance criteria +- Access to test files or test specifications +- Understanding of the implementation + +## Traceability Process + +### 1. Extract Requirements + +Identify all testable requirements from: + +- Acceptance Criteria (primary source) +- User story statement +- Tasks/subtasks with specific behaviors +- Non-functional requirements mentioned +- Edge cases documented + +### 2. Map to Test Cases + +For each requirement, document which tests validate it. Use Given-When-Then to describe what the test validates (not how it's written): + +```yaml +requirement: "AC1: User can login with valid credentials" +test_mappings: + - test_file: "auth/login.test.ts" + test_case: "should successfully login with valid email and password" + # Given-When-Then describes WHAT the test validates, not HOW it's coded + given: "A registered user with valid credentials" + when: "They submit the login form" + then: "They are redirected to dashboard and session is created" + coverage: full + + - test_file: "e2e/auth-flow.test.ts" + test_case: "complete login flow" + given: "User on login page" + when: "Entering valid credentials and submitting" + then: "Dashboard loads with user data" + coverage: integration +``` + +### 3. Coverage Analysis + +Evaluate coverage for each requirement: + +**Coverage Levels:** + +- `full`: Requirement completely tested +- `partial`: Some aspects tested, gaps exist +- `none`: No test coverage found +- `integration`: Covered in integration/e2e tests only +- `unit`: Covered in unit tests only + +### 4. Gap Identification + +Document any gaps found: + +```yaml +coverage_gaps: + - requirement: "AC3: Password reset email sent within 60 seconds" + gap: "No test for email delivery timing" + severity: medium + suggested_test: + type: integration + description: "Test email service SLA compliance" + + - requirement: "AC5: Support 1000 concurrent users" + gap: "No load testing implemented" + severity: high + suggested_test: + type: performance + description: "Load test with 1000 concurrent connections" +``` + +## Outputs + +### Output 1: Gate YAML Block + +**Generate for pasting into gate file under `trace`:** + +```yaml +trace: + totals: + requirements: X + full: Y + partial: Z + none: W + planning_ref: "docs/qa/assessments/{epic}.{story}-test-design-{YYYYMMDD}.md" + uncovered: + - ac: "AC3" + reason: "No test found for password reset timing" + notes: "See docs/qa/assessments/{epic}.{story}-trace-{YYYYMMDD}.md" +``` + +### Output 2: Traceability Report + +**Save to:** `docs/qa/assessments/{epic}.{story}-trace-{YYYYMMDD}.md` + +Create a traceability report with: + +```markdown +# Requirements Traceability Matrix + +## Story: {epic}.{story} - {title} + +### Coverage Summary + +- Total Requirements: X +- Fully Covered: Y (Z%) +- Partially Covered: A (B%) +- Not Covered: C (D%) + +### Requirement Mappings + +#### AC1: {Acceptance Criterion 1} + +**Coverage: FULL** + +Given-When-Then Mappings: + +- **Unit Test**: `auth.service.test.ts::validateCredentials` + - Given: Valid user credentials + - When: Validation method called + - Then: Returns true with user object + +- **Integration Test**: `auth.integration.test.ts::loginFlow` + - Given: User with valid account + - When: Login API called + - Then: JWT token returned and session created + +#### AC2: {Acceptance Criterion 2} + +**Coverage: PARTIAL** + +[Continue for all ACs...] + +### Critical Gaps + +1. **Performance Requirements** + - Gap: No load testing for concurrent users + - Risk: High - Could fail under production load + - Action: Implement load tests using k6 or similar + +2. **Security Requirements** + - Gap: Rate limiting not tested + - Risk: Medium - Potential DoS vulnerability + - Action: Add rate limit tests to integration suite + +### Test Design Recommendations + +Based on gaps identified, recommend: + +1. Additional test scenarios needed +2. Test types to implement (unit/integration/e2e/performance) +3. Test data requirements +4. Mock/stub strategies + +### Risk Assessment + +- **High Risk**: Requirements with no coverage +- **Medium Risk**: Requirements with only partial coverage +- **Low Risk**: Requirements with full unit + integration coverage +``` + +## Traceability Best Practices + +### Given-When-Then for Mapping (Not Test Code) + +Use Given-When-Then to document what each test validates: + +**Given**: The initial context the test sets up + +- What state/data the test prepares +- User context being simulated +- System preconditions + +**When**: The action the test performs + +- What the test executes +- API calls or user actions tested +- Events triggered + +**Then**: What the test asserts + +- Expected outcomes verified +- State changes checked +- Values validated + +**Note**: This is for documentation only. Actual test code follows your project's standards (e.g., describe/it blocks, no BDD syntax). + +### Coverage Priority + +Prioritize coverage based on: + +1. Critical business flows +2. Security-related requirements +3. Data integrity requirements +4. User-facing features +5. Performance SLAs + +### Test Granularity + +Map at appropriate levels: + +- Unit tests for business logic +- Integration tests for component interaction +- E2E tests for user journeys +- Performance tests for NFRs + +## Quality Indicators + +Good traceability shows: + +- Every AC has at least one test +- Critical paths have multiple test levels +- Edge cases are explicitly covered +- NFRs have appropriate test types +- Clear Given-When-Then for each test + +## Red Flags + +Watch for: + +- ACs with no test coverage +- Tests that don't map to requirements +- Vague test descriptions +- Missing edge case coverage +- NFRs without specific tests + +## Integration with Gates + +This traceability feeds into quality gates: + +- Critical gaps โ†’ FAIL +- Minor gaps โ†’ CONCERNS +- Missing P0 tests from test-design โ†’ CONCERNS + +### Output 3: Story Hook Line + +**Print this line for review task to quote:** + +```text +Trace matrix: docs/qa/assessments/{epic}.{story}-trace-{YYYYMMDD}.md +``` + +- Full coverage โ†’ PASS contribution + +## Key Principles + +- Every requirement must be testable +- Use Given-When-Then for clarity +- Identify both presence and absence +- Prioritize based on risk +- Make recommendations actionable diff --git a/bmad-core/templates/qa-gate-tmpl.yaml b/bmad-core/templates/qa-gate-tmpl.yaml new file mode 100644 index 00000000..2632ef03 --- /dev/null +++ b/bmad-core/templates/qa-gate-tmpl.yaml @@ -0,0 +1,93 @@ +template: + id: qa-gate-template-v1 + name: Quality Gate Decision + version: 1.0 + output: + format: yaml + filename: docs/qa/gates/{{epic_num}}.{{story_num}}-{{story_slug}}.yml + title: "Quality Gate: {{epic_num}}.{{story_num}}" + +# Required fields (keep these first) +schema: 1 +story: "{{epic_num}}.{{story_num}}" +story_title: "{{story_title}}" +gate: "{{gate_status}}" # PASS|CONCERNS|FAIL|WAIVED +status_reason: "{{status_reason}}" # 1-2 sentence summary of why this gate decision +reviewer: "Quinn (Test Architect)" +updated: "{{iso_timestamp}}" + +# Always present but only active when WAIVED +waiver: { active: false } + +# Issues (if any) - Use fixed severity: low | medium | high +top_issues: [] + +# Risk summary (from risk-profile task if run) +risk_summary: + totals: { critical: 0, high: 0, medium: 0, low: 0 } + recommendations: + must_fix: [] + monitor: [] + +# Example with issues: +# top_issues: +# - id: "SEC-001" +# severity: high # ONLY: low|medium|high +# finding: "No rate limiting on login endpoint" +# suggested_action: "Add rate limiting middleware before production" +# - id: "TEST-001" +# severity: medium +# finding: "Missing integration tests for auth flow" +# suggested_action: "Add test coverage for critical paths" + +# Example when waived: +# waiver: +# active: true +# reason: "Accepted for MVP release - will address in next sprint" +# approved_by: "Product Owner" + +# ============ Optional Extended Fields ============ +# Uncomment and use if your team wants more detail + +# quality_score: 75 # 0-100 (optional scoring) +# expires: "2025-01-26T00:00:00Z" # Optional gate freshness window + +# evidence: +# tests_reviewed: 15 +# risks_identified: 3 +# trace: +# ac_covered: [1, 2, 3] # AC numbers with test coverage +# ac_gaps: [4] # AC numbers lacking coverage + +# nfr_validation: +# security: { status: CONCERNS, notes: "Rate limiting missing" } +# performance: { status: PASS, notes: "" } +# reliability: { status: PASS, notes: "" } +# maintainability: { status: PASS, notes: "" } + +# history: # Append-only audit trail +# - at: "2025-01-12T10:00:00Z" +# gate: FAIL +# note: "Initial review - missing tests" +# - at: "2025-01-12T15:00:00Z" +# gate: CONCERNS +# note: "Tests added but rate limiting still missing" + +# risk_summary: # From risk-profile task +# totals: +# critical: 0 +# high: 0 +# medium: 0 +# low: 0 +# # 'highest' is emitted only when risks exist +# recommendations: +# must_fix: [] +# monitor: [] + +# recommendations: +# immediate: # Must fix before production +# - action: "Add rate limiting to auth endpoints" +# refs: ["api/auth/login.ts:42-68"] +# future: # Can be addressed later +# - action: "Consider caching for better performance" +# refs: ["services/data.service.ts"] \ No newline at end of file diff --git a/docs/enhanced-ide-development-workflow.md b/docs/enhanced-ide-development-workflow.md index 70710dab..9da6fca8 100644 --- a/docs/enhanced-ide-development-workflow.md +++ b/docs/enhanced-ide-development-workflow.md @@ -21,12 +21,92 @@ This is a simple step-by-step guide to help you efficiently manage your developm 3. **Execute**: `*develop-story {selected-story}` (runs execute-checklist task) 4. **Review generated report** in `{selected-story}` -## Story Review (Quality Assurance) +## Test Architect Integration Throughout Workflow + +### Stage 1: After Story Creation (Before Dev Starts) + +**Optional but Valuable - Set Dev Up for Success:** + +```bash +# Design test strategy for developer +@qa *design {approved-story} +# Output: docs/qa/assessments/{epic}.{story}-test-design-{YYYYMMDD}.md +# Why: Gives Dev clear test requirements upfront + +# Identify risks to watch for +@qa *risk {approved-story} +# Output: docs/qa/assessments/{epic}.{story}-risk-{YYYYMMDD}.md +# Why: Helps Dev avoid critical issues and focus on risk areas +``` + +### Stage 2: During Development (Optional Checkpoints) + +**Dev Can Self-Check Progress:** + +```bash +# Validate NFRs mid-implementation +@qa *nfr {story-in-progress} +# Output: docs/qa/assessments/{epic}.{story}-nfr-{YYYYMMDD}.md +# Why: Catch performance/security issues early + +# Verify test coverage while developing +@qa *trace {story-in-progress} +# Output: docs/qa/assessments/{epic}.{story}-trace-{YYYYMMDD}.md +# Why: Ensure you're writing the right tests +``` + +### Stage 3: Story Review (Required) + +**Standard Review Flow:** 1. **Start new chat/conversation** -2. **Load QA agent** -3. **Execute**: `*review {selected-story}` (runs review-story task) -4. **Review generated report** in `{selected-story}` +2. **Load QA agent** (Test Architect - Quinn) +3. **Execute**: `@qa *review {selected-story}` +4. **Review outputs**: + - QA Results section added to story file + - Quality gate file in `docs/qa/gates/{epic}.{story}-{slug}.yml` + +**If Issues Found:** + +```bash +# After fixes, update gate status +@qa *gate {reviewed-story} +# Output: Updates docs/qa/gates/{epic}.{story}-{slug}.yml +``` + +### Special Situations + +**High-Risk Stories or Brownfield:** + +- Always run `*risk` and `*design` before development +- Extra focus on regression testing + +**Complex Integrations:** + +- Run `*trace` to ensure all integration points tested +- Consider `*nfr` for performance validation + +**Performance-Critical Features:** + +- Run `*nfr` during development, not just at review +- Set performance baselines early + +### Understanding Gate Decisions + +- **PASS**: Ready for production, all quality criteria met +- **CONCERNS**: Non-critical issues to review with team +- **FAIL**: Critical issues must be fixed (security, missing tests) +- **WAIVED**: Known issues accepted by team with documentation + +### What Quinn Does During Review + +1. **Analyzes code quality** and architecture patterns +2. **Actively refactors** code when safe to do so +3. **Validates test coverage** at appropriate levels +4. **Identifies risks** (security, performance, data) +5. **Checks NFRs** against standards +6. **Creates quality gate** with clear decision criteria +7. **Documents everything** for audit trail ## Commit Changes and Push diff --git a/docs/user-guide.md b/docs/user-guide.md index 6e931ce0..07effd3e 100644 --- a/docs/user-guide.md +++ b/docs/user-guide.md @@ -95,9 +95,9 @@ graph TD E --> F["Dev: Run All Validations"] F --> G["Dev: Mark Ready for Review + Add Notes"] G --> H{"User Verification"} - H -->|Request QA Review| I["QA: Senior Dev Review + Active Refactoring"] + H -->|Request QA Review| I["QA: Test Architect Review + Quality Gate"] H -->|Approve Without QA| M["IMPORTANT: Verify All Regression Tests and Linting are Passing"] - I --> J["QA: Review, Refactor Code, Add Tests, Document Notes"] + I --> J["QA: Test Architecture Analysis + Active Refactoring"] J --> L{"QA Decision"} L -->|Needs Dev Work| D L -->|Approved| M @@ -212,6 +212,216 @@ dependencies: - **File Organization**: Maintain clean project structure - **Commit Regularly**: Save your work frequently +## The Test Architect (QA Agent) + +### Overview + +The QA agent in BMad is not just a "senior developer reviewer" - it's a **Test Architect** with deep expertise in test strategy, quality gates, and risk-based testing. Named Quinn, this agent provides advisory authority on quality matters while actively improving code when safe to do so. + +### Core Capabilities + +#### 1. Comprehensive Test Architecture Review + +When you run `@qa *review {story}`, Quinn performs: + +- **Requirements Traceability**: Maps every acceptance criterion to its validating tests +- **Test Level Analysis**: Ensures appropriate testing at unit, integration, and E2E levels +- **Coverage Assessment**: Identifies gaps and redundant test coverage +- **Active Refactoring**: Improves code quality directly when safe +- **Quality Gate Decision**: Issues PASS/CONCERNS/FAIL status based on findings + +#### 2. Test Design (`*design`) + +Creates comprehensive test strategies including: + +- Test scenarios for each acceptance criterion +- Appropriate test level recommendations (unit vs integration vs E2E) +- Risk-based prioritization (P0/P1/P2) +- Test data requirements and mock strategies +- Execution strategies for CI/CD integration + +**Example output:** + +```yaml +test_summary: + total: 24 + by_level: + unit: 15 + integration: 7 + e2e: 2 + by_priority: + P0: 8 # Must have - linked to critical risks + P1: 10 # Should have - medium risks + P2: 6 # Nice to have - low risks +``` + +#### 3. Risk Profiling (`*risk`) + +Identifies and assesses implementation risks: + +- **Categories**: Technical, Security, Performance, Data, Business, Operational +- **Scoring**: Probability ร— Impact analysis (1-9 scale) +- **Mitigation**: Specific strategies for each identified risk +- **Gate Impact**: Risks โ‰ฅ9 trigger FAIL, โ‰ฅ6 trigger CONCERNS + +#### 4. NFR Assessment (`*nfr`) + +Validates non-functional requirements: + +- **Core Four**: Security, Performance, Reliability, Maintainability +- **Evidence-Based**: Looks for actual implementation proof +- **Gate Integration**: NFR failures directly impact quality gates + +#### 5. Requirements Tracing (`*trace`) + +Maps requirements to test coverage: + +- Documents which tests validate each acceptance criterion +- Uses Given-When-Then for clarity (documentation only, not BDD code) +- Identifies coverage gaps with severity ratings +- Creates traceability matrix for audit purposes + +#### 6. Quality Gates (`*gate`) + +Manages quality gate decisions: + +- **Deterministic Rules**: Clear criteria for PASS/CONCERNS/FAIL +- **Parallel Authority**: QA owns gate files in `docs/qa/gates/` +- **Advisory Nature**: Provides recommendations, not blocks +- **Waiver Support**: Documents accepted risks when needed + +### Working with the Test Architect + +#### Basic Review Flow + +1. **Developer completes story** and marks "Ready for Review" +2. **Start new conversation** with QA agent +3. **Run review**: `@qa *review {story-file}` +4. **Quinn performs**: + - Deep code analysis + - Active refactoring where safe + - Test coverage validation + - Quality gate decision +5. **Outputs created**: + - QA Results section added to story file + - Gate file in `docs/qa/gates/{epic}.{story}-{slug}.yml` + +#### Advanced Quality Assessments - When to Use + +**AFTER Story Draft (Before Dev Starts):** + +```bash +# Design test strategy BEFORE implementation +@qa *design {draft-story} +# Output: docs/qa/assessments/{epic}.{story}-test-design-{YYYYMMDD}.md +# Why: Guides developer on what tests to write + +# Identify risks EARLY to guide implementation +@qa *risk {draft-story} +# Output: docs/qa/assessments/{epic}.{story}-risk-{YYYYMMDD}.md +# Why: Helps developer avoid pitfalls and focus on risk areas +``` + +**DURING Development (Mid-Implementation):** + +```bash +# Validate NFR implementation as you build +@qa *nfr {in-progress-story} +# Output: docs/qa/assessments/{epic}.{story}-nfr-{YYYYMMDD}.md +# Why: Catch performance/security issues early + +# Check test coverage while developing +@qa *trace {in-progress-story} +# Output: docs/qa/assessments/{epic}.{story}-trace-{YYYYMMDD}.md +# Why: Ensure you're writing the right tests +``` + +**AFTER Review (Post-Implementation):** + +```bash +# Update gate decision after fixes +@qa *gate {reviewed-story} +# Output: docs/qa/gates/{epic}.{story}-{slug}.yml +# Why: Document final quality decision +``` + +### Quality Standards Enforced + +Quinn enforces these test quality principles: + +- **No Flaky Tests**: Ensures reliability through proper async handling +- **No Hard Waits**: Dynamic waiting strategies only +- **Stateless & Parallel-Safe**: Tests run independently +- **Self-Cleaning**: Tests manage their own test data +- **Appropriate Test Levels**: Unit for logic, integration for interactions, E2E for journeys +- **Explicit Assertions**: Keep assertions in tests, not helpers + +### Gate Status Meanings + +- **PASS**: All critical requirements met, no blocking issues +- **CONCERNS**: Non-critical issues found, team should review +- **FAIL**: Critical issues that should be addressed (security risks, missing P0 tests) +- **WAIVED**: Issues acknowledged but explicitly accepted by team + +### Integration with Development Flow + +The Test Architect integrates seamlessly: + +1. **Planning Phase**: Can review test strategies early +2. **Development Phase**: Developer implements with test guidelines in mind +3. **Review Phase**: Comprehensive quality assessment +4. **Gate Decision**: Clear go/no-go recommendation +5. **Documentation**: All findings tracked for audit trail + +### Integration with BMad Workflow Stages + +#### Stage 1: Story Drafting (SM Creates Story) + +**Optional but Valuable:** + +- Run `*design` on draft โ†’ Give to Dev as test requirements +- Run `*risk` on draft โ†’ Highlight concerns for Dev to address + +#### Stage 2: Development (Dev Implements) + +**Mid-Development Checkpoints:** + +- Dev can run `*trace` โ†’ Verify tests are being created +- Dev can run `*nfr` โ†’ Check performance/security early + +#### Stage 3: Review (Story Ready) + +**Standard Review:** + +- Run `*review` โ†’ Complete assessment + gate decision + +**If Issues Found:** + +- After fixes, run `*gate` โ†’ Update gate status + +#### Special Situations + +**High-Risk Stories:** + +- Always run `*risk` and `*design` before development + +**Complex Integrations:** + +- Run `*trace` to ensure all integration points tested + +**Performance-Critical:** + +- Run `*nfr` during development, not just at review + +### Best Practices + +- **Early Engagement**: Run `*design` and `*risk` during story drafting +- **Risk-Based Focus**: Let risk scores drive test prioritization +- **Iterative Improvement**: Use QA feedback to improve future stories +- **Gate Transparency**: Share gate decisions with the team +- **Continuous Learning**: QA documents patterns for team knowledge sharing +- **Brownfield Care**: Pay extra attention to regression risks in existing systems + ## Technical Preferences System BMad includes a personalization system through the `technical-preferences.md` file located in `.bmad-core/data/` - this can help bias the PM and Architect to recommend your preferences for design patterns, technology selection, or anything else you would like to put in here. diff --git a/docs/working-in-the-brownfield.md b/docs/working-in-the-brownfield.md index 442b37c6..16617658 100644 --- a/docs/working-in-the-brownfield.md +++ b/docs/working-in-the-brownfield.md @@ -27,7 +27,7 @@ If you have just completed an MVP with BMad, and you want to continue with post- ## The Complete Brownfield Workflow 1. **Follow the [User Guide - Installation](user-guide.md#installation) steps to setup your agent in the web.** -2. **Generate a 'flattened' single file of your entire codebase** run: ```npx bmad-method flatten``` +2. **Generate a 'flattened' single file of your entire codebase** run: `npx bmad-method flatten` ### Choose Your Approach @@ -253,14 +253,76 @@ Brownfield changes should: - Include migration scripts - Maintain backwards compatibility -### 4. Test Integration Thoroughly +### 4. Test Integration with the Test Architect -Focus testing on: +#### Brownfield-Specific Testing Challenges -- Integration points -- Existing functionality (regression) -- Performance impact -- Data migrations +The Test Architect (QA agent) is especially valuable in brownfield projects for: + +- **Regression Risk Assessment**: Identifying which existing tests might break +- **Integration Test Design**: Ensuring new code works with legacy systems +- **Coverage Gap Analysis**: Finding untested legacy code paths +- **Performance Impact**: Validating no degradation to existing functionality + +#### Test Architect Workflow Stages for Brownfield + +##### Stage 1: After Story Creation (Critical for Brownfield) + +**HIGHLY RECOMMENDED - Identify Integration Risks Early:** + +```bash +# Assess integration risks BEFORE coding +@qa *risk {brownfield-story} +# Output: docs/qa/assessments/{epic}.{story}-risk-{YYYYMMDD}.md +# Why: Identifies legacy dependencies, breaking changes, integration risks + +# Design regression-aware test strategy +@qa *design {brownfield-story} +# Output: docs/qa/assessments/{epic}.{story}-test-design-{YYYYMMDD}.md +# Why: Plans tests for both new features AND regression prevention +``` + +##### Stage 2: During Development (Monitor Integration) + +**Dev Should Check Compatibility:** + +```bash +# Trace requirements to preserve existing functionality +@qa *trace {brownfield-story} +# Output: docs/qa/assessments/{epic}.{story}-trace-{YYYYMMDD}.md +# Why: Maps new requirements AND existing functionality that must work + +# Validate performance hasn't degraded +@qa *nfr {brownfield-story} +# Output: docs/qa/assessments/{epic}.{story}-nfr-{YYYYMMDD}.md +# Why: Catch performance regressions early +``` + +##### Stage 3: Review (Deep Integration Analysis) + +**Standard Review with Brownfield Focus:** + +```bash +@qa *review {brownfield-story} +``` + +Performs deep analysis with special attention to: + +- Breaking changes to existing APIs +- Data migration correctness +- Performance regression +- Security implications of integrations +- Backwards compatibility +- Feature flag correctness + +#### Focus Areas for Brownfield Testing + +- **Integration Points**: Test all integration points with existing code +- **Existing Functionality**: Comprehensive regression testing +- **Performance Impact**: Benchmark against current performance +- **Data Migrations**: Validate data integrity and rollback procedures +- **Backwards Compatibility**: Ensure existing clients still work +- **Feature Flags**: Test both enabled and disabled states ### 5. Communicate Changes @@ -292,7 +354,10 @@ Document: 1. Document relevant subsystems 2. Use `create-brownfield-story` for focused fix 3. Include regression test requirements -4. QA validates no side effects +4. Test Architect validates no side effects using: + - Risk profiling for side effect analysis + - Trace matrix to ensure fix doesn't break related features + - NFR assessment to verify performance/security unchanged ### Scenario 4: API Integration