feat: transform QA agent into Test Architect with advanced quality capabilities

- Add 6 specialized quality assessment commands - Implement risk-based testing with scoring - Create quality gate system with deterministic decisions - Add comprehensive test design and NFR validation - Update documentation with stage-based workflow integration
2025-08-12 12:59:22 -05:00
parent ffcb4d4bf2
commit b7a34b4fc6
11 changed files with 2321 additions and 96 deletions
--- a/docs/enhanced-ide-development-workflow.md
+++ b/docs/enhanced-ide-development-workflow.md
@@ -21,12 +21,92 @@ This is a simple step-by-step guide to help you efficiently manage your developm
 3. **Execute**: `*develop-story {selected-story}` (runs execute-checklist task)
 4. **Review generated report** in `{selected-story}`

-## Story Review (Quality Assurance)
+## Test Architect Integration Throughout Workflow
+
+### Stage 1: After Story Creation (Before Dev Starts)
+
+**Optional but Valuable - Set Dev Up for Success:**
+
+```bash
+# Design test strategy for developer
+@qa *design {approved-story}
+# Output: docs/qa/assessments/{epic}.{story}-test-design-{YYYYMMDD}.md
+# Why: Gives Dev clear test requirements upfront
+
+# Identify risks to watch for
+@qa *risk {approved-story}
+# Output: docs/qa/assessments/{epic}.{story}-risk-{YYYYMMDD}.md
+# Why: Helps Dev avoid critical issues and focus on risk areas
+```
+
+### Stage 2: During Development (Optional Checkpoints)
+
+**Dev Can Self-Check Progress:**
+
+```bash
+# Validate NFRs mid-implementation
+@qa *nfr {story-in-progress}
+# Output: docs/qa/assessments/{epic}.{story}-nfr-{YYYYMMDD}.md
+# Why: Catch performance/security issues early
+
+# Verify test coverage while developing
+@qa *trace {story-in-progress}
+# Output: docs/qa/assessments/{epic}.{story}-trace-{YYYYMMDD}.md
+# Why: Ensure you're writing the right tests
+```
+
+### Stage 3: Story Review (Required)
+
+**Standard Review Flow:**

 1. **Start new chat/conversation**
-2. **Load QA agent**
-3. **Execute**: `*review {selected-story}` (runs review-story task)
-4. **Review generated report** in `{selected-story}`
+2. **Load QA agent** (Test Architect - Quinn)
+3. **Execute**: `@qa *review {selected-story}`
+4. **Review outputs**:
+   - QA Results section added to story file
+   - Quality gate file in `docs/qa/gates/{epic}.{story}-{slug}.yml`
+
+**If Issues Found:**
+
+```bash
+# After fixes, update gate status
+@qa *gate {reviewed-story}
+# Output: Updates docs/qa/gates/{epic}.{story}-{slug}.yml
+```
+
+### Special Situations
+
+**High-Risk Stories or Brownfield:**
+
+- Always run `*risk` and `*design` before development
+- Extra focus on regression testing
+
+**Complex Integrations:**
+
+- Run `*trace` to ensure all integration points tested
+- Consider `*nfr` for performance validation
+
+**Performance-Critical Features:**
+
+- Run `*nfr` during development, not just at review
+- Set performance baselines early
+
+### Understanding Gate Decisions
+
+- **PASS**: Ready for production, all quality criteria met
+- **CONCERNS**: Non-critical issues to review with team
+- **FAIL**: Critical issues must be fixed (security, missing tests)
+- **WAIVED**: Known issues accepted by team with documentation
+
+### What Quinn Does During Review
+
+1. **Analyzes code quality** and architecture patterns
+2. **Actively refactors** code when safe to do so
+3. **Validates test coverage** at appropriate levels
+4. **Identifies risks** (security, performance, data)
+5. **Checks NFRs** against standards
+6. **Creates quality gate** with clear decision criteria
+7. **Documents everything** for audit trail

 ## Commit Changes and Push

--- a/docs/user-guide.md
+++ b/docs/user-guide.md
@@ -95,9 +95,9 @@ graph TD
    E --> F["Dev: Run All Validations"]
    F --> G["Dev: Mark Ready for Review + Add Notes"]
    G --> H{"User Verification"}
-    H -->|Request QA Review| I["QA: Senior Dev Review + Active Refactoring"]
+    H -->|Request QA Review| I["QA: Test Architect Review + Quality Gate"]
    H -->|Approve Without QA| M["IMPORTANT: Verify All Regression Tests and Linting are Passing"]
-    I --> J["QA: Review, Refactor Code, Add Tests, Document Notes"]
+    I --> J["QA: Test Architecture Analysis + Active Refactoring"]
    J --> L{"QA Decision"}
    L -->|Needs Dev Work| D
    L -->|Approved| M
@@ -212,6 +212,216 @@ dependencies:
 - **File Organization**: Maintain clean project structure
 - **Commit Regularly**: Save your work frequently

+## The Test Architect (QA Agent)
+
+### Overview
+
+The QA agent in BMad is not just a "senior developer reviewer" - it's a **Test Architect** with deep expertise in test strategy, quality gates, and risk-based testing. Named Quinn, this agent provides advisory authority on quality matters while actively improving code when safe to do so.
+
+### Core Capabilities
+
+#### 1. Comprehensive Test Architecture Review
+
+When you run `@qa *review {story}`, Quinn performs:
+
+- **Requirements Traceability**: Maps every acceptance criterion to its validating tests
+- **Test Level Analysis**: Ensures appropriate testing at unit, integration, and E2E levels
+- **Coverage Assessment**: Identifies gaps and redundant test coverage
+- **Active Refactoring**: Improves code quality directly when safe
+- **Quality Gate Decision**: Issues PASS/CONCERNS/FAIL status based on findings
+
+#### 2. Test Design (`*design`)
+
+Creates comprehensive test strategies including:
+
+- Test scenarios for each acceptance criterion
+- Appropriate test level recommendations (unit vs integration vs E2E)
+- Risk-based prioritization (P0/P1/P2)
+- Test data requirements and mock strategies
+- Execution strategies for CI/CD integration
+
+**Example output:**
+
+```yaml
+test_summary:
+  total: 24
+  by_level:
+    unit: 15
+    integration: 7
+    e2e: 2
+  by_priority:
+    P0: 8 # Must have - linked to critical risks
+    P1: 10 # Should have - medium risks
+    P2: 6 # Nice to have - low risks
+```
+
+#### 3. Risk Profiling (`*risk`)
+
+Identifies and assesses implementation risks:
+
+- **Categories**: Technical, Security, Performance, Data, Business, Operational
+- **Scoring**: Probability × Impact analysis (1-9 scale)
+- **Mitigation**: Specific strategies for each identified risk
+- **Gate Impact**: Risks ≥9 trigger FAIL, ≥6 trigger CONCERNS
+
+#### 4. NFR Assessment (`*nfr`)
+
+Validates non-functional requirements:
+
+- **Core Four**: Security, Performance, Reliability, Maintainability
+- **Evidence-Based**: Looks for actual implementation proof
+- **Gate Integration**: NFR failures directly impact quality gates
+
+#### 5. Requirements Tracing (`*trace`)
+
+Maps requirements to test coverage:
+
+- Documents which tests validate each acceptance criterion
+- Uses Given-When-Then for clarity (documentation only, not BDD code)
+- Identifies coverage gaps with severity ratings
+- Creates traceability matrix for audit purposes
+
+#### 6. Quality Gates (`*gate`)
+
+Manages quality gate decisions:
+
+- **Deterministic Rules**: Clear criteria for PASS/CONCERNS/FAIL
+- **Parallel Authority**: QA owns gate files in `docs/qa/gates/`
+- **Advisory Nature**: Provides recommendations, not blocks
+- **Waiver Support**: Documents accepted risks when needed
+
+### Working with the Test Architect
+
+#### Basic Review Flow
+
+1. **Developer completes story** and marks "Ready for Review"
+2. **Start new conversation** with QA agent
+3. **Run review**: `@qa *review {story-file}`
+4. **Quinn performs**:
+   - Deep code analysis
+   - Active refactoring where safe
+   - Test coverage validation
+   - Quality gate decision
+5. **Outputs created**:
+   - QA Results section added to story file
+   - Gate file in `docs/qa/gates/{epic}.{story}-{slug}.yml`
+
+#### Advanced Quality Assessments - When to Use
+
+**AFTER Story Draft (Before Dev Starts):**
+
+```bash
+# Design test strategy BEFORE implementation
+@qa *design {draft-story}
+# Output: docs/qa/assessments/{epic}.{story}-test-design-{YYYYMMDD}.md
+# Why: Guides developer on what tests to write
+
+# Identify risks EARLY to guide implementation
+@qa *risk {draft-story}
+# Output: docs/qa/assessments/{epic}.{story}-risk-{YYYYMMDD}.md
+# Why: Helps developer avoid pitfalls and focus on risk areas
+```
+
+**DURING Development (Mid-Implementation):**
+
+```bash
+# Validate NFR implementation as you build
+@qa *nfr {in-progress-story}
+# Output: docs/qa/assessments/{epic}.{story}-nfr-{YYYYMMDD}.md
+# Why: Catch performance/security issues early
+
+# Check test coverage while developing
+@qa *trace {in-progress-story}
+# Output: docs/qa/assessments/{epic}.{story}-trace-{YYYYMMDD}.md
+# Why: Ensure you're writing the right tests
+```
+
+**AFTER Review (Post-Implementation):**
+
+```bash
+# Update gate decision after fixes
+@qa *gate {reviewed-story}
+# Output: docs/qa/gates/{epic}.{story}-{slug}.yml
+# Why: Document final quality decision
+```
+
+### Quality Standards Enforced
+
+Quinn enforces these test quality principles:
+
+- **No Flaky Tests**: Ensures reliability through proper async handling
+- **No Hard Waits**: Dynamic waiting strategies only
+- **Stateless & Parallel-Safe**: Tests run independently
+- **Self-Cleaning**: Tests manage their own test data
+- **Appropriate Test Levels**: Unit for logic, integration for interactions, E2E for journeys
+- **Explicit Assertions**: Keep assertions in tests, not helpers
+
+### Gate Status Meanings
+
+- **PASS**: All critical requirements met, no blocking issues
+- **CONCERNS**: Non-critical issues found, team should review
+- **FAIL**: Critical issues that should be addressed (security risks, missing P0 tests)
+- **WAIVED**: Issues acknowledged but explicitly accepted by team
+
+### Integration with Development Flow
+
+The Test Architect integrates seamlessly:
+
+1. **Planning Phase**: Can review test strategies early
+2. **Development Phase**: Developer implements with test guidelines in mind
+3. **Review Phase**: Comprehensive quality assessment
+4. **Gate Decision**: Clear go/no-go recommendation
+5. **Documentation**: All findings tracked for audit trail
+
+### Integration with BMad Workflow Stages
+
+#### Stage 1: Story Drafting (SM Creates Story)
+
+**Optional but Valuable:**
+
+- Run `*design` on draft → Give to Dev as test requirements
+- Run `*risk` on draft → Highlight concerns for Dev to address
+
+#### Stage 2: Development (Dev Implements)
+
+**Mid-Development Checkpoints:**
+
+- Dev can run `*trace` → Verify tests are being created
+- Dev can run `*nfr` → Check performance/security early
+
+#### Stage 3: Review (Story Ready)
+
+**Standard Review:**
+
+- Run `*review` → Complete assessment + gate decision
+
+**If Issues Found:**
+
+- After fixes, run `*gate` → Update gate status
+
+#### Special Situations
+
+**High-Risk Stories:**
+
+- Always run `*risk` and `*design` before development
+
+**Complex Integrations:**
+
+- Run `*trace` to ensure all integration points tested
+
+**Performance-Critical:**
+
+- Run `*nfr` during development, not just at review
+
+### Best Practices
+
+- **Early Engagement**: Run `*design` and `*risk` during story drafting
+- **Risk-Based Focus**: Let risk scores drive test prioritization
+- **Iterative Improvement**: Use QA feedback to improve future stories
+- **Gate Transparency**: Share gate decisions with the team
+- **Continuous Learning**: QA documents patterns for team knowledge sharing
+- **Brownfield Care**: Pay extra attention to regression risks in existing systems
+
 ## Technical Preferences System

 BMad includes a personalization system through the `technical-preferences.md` file located in `.bmad-core/data/` - this can help bias the PM and Architect to recommend your preferences for design patterns, technology selection, or anything else you would like to put in here.
--- a/docs/working-in-the-brownfield.md
+++ b/docs/working-in-the-brownfield.md
@@ -27,7 +27,7 @@ If you have just completed an MVP with BMad, and you want to continue with post-
 ## The Complete Brownfield Workflow

 1. **Follow the [<ins>User Guide - Installation</ins>](user-guide.md#installation) steps to setup your agent in the web.**
-2. **Generate a 'flattened' single file of your entire codebase** run: ```npx bmad-method flatten```
+2. **Generate a 'flattened' single file of your entire codebase** run: `npx bmad-method flatten`

 ### Choose Your Approach

@@ -253,14 +253,76 @@ Brownfield changes should:
 - Include migration scripts
 - Maintain backwards compatibility

-### 4. Test Integration Thoroughly
+### 4. Test Integration with the Test Architect

-Focus testing on:
+#### Brownfield-Specific Testing Challenges

- Integration points
- Existing functionality (regression)
- Performance impact
- Data migrations
+The Test Architect (QA agent) is especially valuable in brownfield projects for:
+
+- **Regression Risk Assessment**: Identifying which existing tests might break
+- **Integration Test Design**: Ensuring new code works with legacy systems
+- **Coverage Gap Analysis**: Finding untested legacy code paths
+- **Performance Impact**: Validating no degradation to existing functionality
+
+#### Test Architect Workflow Stages for Brownfield
+
+##### Stage 1: After Story Creation (Critical for Brownfield)
+
+**HIGHLY RECOMMENDED - Identify Integration Risks Early:**
+
+```bash
+# Assess integration risks BEFORE coding
+@qa *risk {brownfield-story}
+# Output: docs/qa/assessments/{epic}.{story}-risk-{YYYYMMDD}.md
+# Why: Identifies legacy dependencies, breaking changes, integration risks
+
+# Design regression-aware test strategy
+@qa *design {brownfield-story}
+# Output: docs/qa/assessments/{epic}.{story}-test-design-{YYYYMMDD}.md
+# Why: Plans tests for both new features AND regression prevention
+```
+
+##### Stage 2: During Development (Monitor Integration)
+
+**Dev Should Check Compatibility:**
+
+```bash
+# Trace requirements to preserve existing functionality
+@qa *trace {brownfield-story}
+# Output: docs/qa/assessments/{epic}.{story}-trace-{YYYYMMDD}.md
+# Why: Maps new requirements AND existing functionality that must work
+
+# Validate performance hasn't degraded
+@qa *nfr {brownfield-story}
+# Output: docs/qa/assessments/{epic}.{story}-nfr-{YYYYMMDD}.md
+# Why: Catch performance regressions early
+```
+
+##### Stage 3: Review (Deep Integration Analysis)
+
+**Standard Review with Brownfield Focus:**
+
+```bash
+@qa *review {brownfield-story}
+```
+
+Performs deep analysis with special attention to:
+
+- Breaking changes to existing APIs
+- Data migration correctness
+- Performance regression
+- Security implications of integrations
+- Backwards compatibility
+- Feature flag correctness
+
+#### Focus Areas for Brownfield Testing
+
+- **Integration Points**: Test all integration points with existing code
+- **Existing Functionality**: Comprehensive regression testing
+- **Performance Impact**: Benchmark against current performance
+- **Data Migrations**: Validate data integrity and rollback procedures
+- **Backwards Compatibility**: Ensure existing clients still work
+- **Feature Flags**: Test both enabled and disabled states

 ### 5. Communicate Changes

@@ -292,7 +354,10 @@ Document:
 1. Document relevant subsystems
 2. Use `create-brownfield-story` for focused fix
 3. Include regression test requirements
-4. QA validates no side effects
+4. Test Architect validates no side effects using:
+   - Risk profiling for side effect analysis
+   - Trace matrix to ensure fix doesn't break related features
+   - NFR assessment to verify performance/security unchanged

 ### Scenario 4: API Integration