mirror of
https://github.com/bmad-code-org/BMAD-METHOD.git
synced 2026-01-30 04:32:02 +00:00
docs: tea in 4; Diátaxis (#1320)
* docs: tea in 4; Diátaxis * docs: addressed review comments * docs: refined the docs
This commit is contained in:
Murat K Ozcan
@@ -23,11 +23,16 @@ BMad does not mandate TEA. There are five valid ways to use it (or skip it). Pic
|
||||
1. **No TEA**
|
||||
- Skip all TEA workflows. Use your existing team testing approach.
|
||||
|
||||
2. **TEA-only (Standalone)**
|
||||
2. **TEA Solo (Standalone)**
|
||||
- Use TEA on a non-BMad project. Bring your own requirements, acceptance criteria, and environments.
|
||||
- Typical sequence: `*test-design` (system or epic) -> `*atdd` and/or `*automate` -> optional `*test-review` -> `*trace` for coverage and gate decisions.
|
||||
- Run `*framework` or `*ci` only if you want TEA to scaffold the harness or pipeline; they work best after you decide the stack/architecture.
|
||||
|
||||
**TEA Lite (Beginner Approach):**
|
||||
- Simplest way to use TEA - just use `*automate` to test existing features.
|
||||
- Perfect for learning TEA fundamentals in 30 minutes.
|
||||
- See [TEA Lite Quickstart Tutorial](/docs/tutorials/getting-started/tea-lite-quickstart.md).
|
||||
|
||||
3. **Integrated: Greenfield - BMad Method (Simple/Standard Work)**
|
||||
- Phase 3: system-level `*test-design`, then `*framework` and `*ci`.
|
||||
- Phase 4: per-epic `*test-design`, optional `*atdd`, then `*automate` and optional `*test-review`.
|
||||
@@ -55,8 +60,8 @@ If you are unsure, default to the integrated path for your track and adjust late
|
||||
| `*framework` | Playwright/Cypress scaffold, `.env.example`, `.nvmrc`, sample specs | Use when no production-ready harness exists | - |
|
||||
| `*ci` | CI workflow, selective test scripts, secrets checklist | Platform-aware (GitHub Actions default) | - |
|
||||
| `*test-design` | Combined risk assessment, mitigation plan, and coverage strategy | Risk scoring + optional exploratory mode | **+ Exploratory**: Interactive UI discovery with browser automation (uncover actual functionality) |
|
||||
| `*atdd` | Failing acceptance tests + implementation checklist | TDD red phase + optional recording mode | **+ Recording**: AI generation verified with live browser (accurate selectors from real DOM) |
|
||||
| `*automate` | Prioritized specs, fixtures, README/script updates, DoD summary | Optional healing/recording, avoid duplicate coverage | **+ Healing**: Pattern fixes enhanced with visual debugging + **+ Recording**: AI verified with live browser |
|
||||
| `*atdd` | Failing acceptance tests + implementation checklist | TDD red phase + optional recording mode | **+ Recording**: UI selectors verified with live browser; API tests benefit from trace analysis |
|
||||
| `*automate` | Prioritized specs, fixtures, README/script updates, DoD summary | Optional healing/recording, avoid duplicate coverage | **+ Healing**: Visual debugging + trace analysis for test fixes; **+ Recording**: Verified selectors (UI) + network inspection (API) |
|
||||
| `*test-review` | Test quality review report with 0-100 score, violations, fixes | Reviews tests against knowledge base patterns | - |
|
||||
| `*nfr-assess` | NFR assessment report with actions | Focus on security/performance/reliability | - |
|
||||
| `*trace` | Phase 1: Coverage matrix, recommendations. Phase 2: Gate decision (PASS/CONCERNS/FAIL/WAIVED) | Two-phase workflow: traceability + gate decision | - |
|
||||
@@ -279,6 +284,31 @@ These cheat sheets map TEA workflows to the **BMad Method and Enterprise tracks*
|
||||
**Related how-to guides:**
|
||||
- [How to Run Test Design](/docs/how-to/workflows/run-test-design.md)
|
||||
- [How to Set Up a Test Framework](/docs/how-to/workflows/setup-test-framework.md)
|
||||
- [How to Run ATDD](/docs/how-to/workflows/run-atdd.md)
|
||||
- [How to Run Automate](/docs/how-to/workflows/run-automate.md)
|
||||
- [How to Run Test Review](/docs/how-to/workflows/run-test-review.md)
|
||||
- [How to Set Up CI Pipeline](/docs/how-to/workflows/setup-ci.md)
|
||||
- [How to Run NFR Assessment](/docs/how-to/workflows/run-nfr-assess.md)
|
||||
- [How to Run Trace](/docs/how-to/workflows/run-trace.md)
|
||||
|
||||
## Deep Dive Concepts
|
||||
|
||||
Want to understand TEA principles and patterns in depth?
|
||||
|
||||
**Core Principles:**
|
||||
- [Risk-Based Testing](/docs/explanation/tea/risk-based-testing.md) - Probability × impact scoring, P0-P3 priorities
|
||||
- [Test Quality Standards](/docs/explanation/tea/test-quality-standards.md) - Definition of Done, determinism, isolation
|
||||
- [Knowledge Base System](/docs/explanation/tea/knowledge-base-system.md) - Context engineering with tea-index.csv
|
||||
|
||||
**Technical Patterns:**
|
||||
- [Fixture Architecture](/docs/explanation/tea/fixture-architecture.md) - Pure function → fixture → composition
|
||||
- [Network-First Patterns](/docs/explanation/tea/network-first-patterns.md) - Eliminating flakiness with intercept-before-navigate
|
||||
|
||||
**Engagement & Strategy:**
|
||||
- [Engagement Models](/docs/explanation/tea/engagement-models.md) - TEA Lite, TEA Solo, TEA Integrated (5 models explained)
|
||||
|
||||
**Philosophy:**
|
||||
- [Testing as Engineering](/docs/explanation/philosophy/testing-as-engineering.md) - **Start here to understand WHY TEA exists** - The problem with AI-generated tests and TEA's three-part solution
|
||||
|
||||
## Optional Integrations
|
||||
|
||||
|
||||
710
docs/explanation/tea/engagement-models.md
Normal file
710
docs/explanation/tea/engagement-models.md
Normal file
@@ -0,0 +1,710 @@
|
||||
---
|
||||
title: "TEA Engagement Models Explained"
|
||||
description: Understanding the five ways to use TEA - from standalone to full BMad Method integration
|
||||
---
|
||||
|
||||
# TEA Engagement Models Explained
|
||||
|
||||
TEA is optional and flexible. There are five valid ways to engage with TEA - choose intentionally based on your project needs and methodology.
|
||||
|
||||
## Overview
|
||||
|
||||
**TEA is not mandatory.** Pick the engagement model that fits your context:
|
||||
|
||||
1. **No TEA** - Skip all TEA workflows, use existing testing approach
|
||||
2. **TEA Solo** - Use TEA standalone without BMad Method
|
||||
3. **TEA Lite** - Beginner approach using just `*automate`
|
||||
4. **TEA Integrated (Greenfield)** - Full BMad Method integration from scratch
|
||||
5. **TEA Integrated (Brownfield)** - Full BMad Method integration with existing code
|
||||
|
||||
## The Problem
|
||||
|
||||
### One-Size-Fits-All Doesn't Work
|
||||
|
||||
**Traditional testing tools force one approach:**
|
||||
- Must use entire framework
|
||||
- All-or-nothing adoption
|
||||
- No flexibility for different project types
|
||||
- Teams abandon tool if it doesn't fit
|
||||
|
||||
**TEA recognizes:**
|
||||
- Different projects have different needs
|
||||
- Different teams have different maturity levels
|
||||
- Different contexts require different approaches
|
||||
- Flexibility increases adoption
|
||||
|
||||
## The Five Engagement Models
|
||||
|
||||
### Model 1: No TEA
|
||||
|
||||
**What:** Skip all TEA workflows, use your existing testing approach.
|
||||
|
||||
**When to Use:**
|
||||
- Team has established testing practices
|
||||
- Quality is already high
|
||||
- Testing tools already in place
|
||||
- TEA doesn't add value
|
||||
|
||||
**What You Miss:**
|
||||
- Risk-based test planning
|
||||
- Systematic quality review
|
||||
- Gate decisions with evidence
|
||||
- Knowledge base patterns
|
||||
|
||||
**What You Keep:**
|
||||
- Full control
|
||||
- Existing tools
|
||||
- Team expertise
|
||||
- No learning curve
|
||||
|
||||
**Example:**
|
||||
```
|
||||
Your team:
|
||||
- 10-year veteran QA team
|
||||
- Established testing practices
|
||||
- High-quality test suite
|
||||
- No problems to solve
|
||||
|
||||
Decision: Skip TEA, keep what works
|
||||
```
|
||||
|
||||
**Verdict:** Valid choice if existing approach works.
|
||||
|
||||
---
|
||||
|
||||
### Model 2: TEA Solo
|
||||
|
||||
**What:** Use TEA workflows standalone without full BMad Method integration.
|
||||
|
||||
**When to Use:**
|
||||
- Non-BMad projects
|
||||
- Want TEA's quality operating model only
|
||||
- Don't need full planning workflow
|
||||
- Bring your own requirements
|
||||
|
||||
**Typical Sequence:**
|
||||
```
|
||||
1. *test-design (system or epic)
|
||||
2. *atdd or *automate
|
||||
3. *test-review (optional)
|
||||
4. *trace (coverage + gate decision)
|
||||
```
|
||||
|
||||
**You Bring:**
|
||||
- Requirements (user stories, acceptance criteria)
|
||||
- Development environment
|
||||
- Project context
|
||||
|
||||
**TEA Provides:**
|
||||
- Risk-based test planning (`*test-design`)
|
||||
- Test generation (`*atdd`, `*automate`)
|
||||
- Quality review (`*test-review`)
|
||||
- Coverage traceability (`*trace`)
|
||||
|
||||
**Optional:**
|
||||
- Framework setup (`*framework`) if needed
|
||||
- CI configuration (`*ci`) if needed
|
||||
|
||||
**Example:**
|
||||
```
|
||||
Your project:
|
||||
- Using Scrum (not BMad Method)
|
||||
- Jira for story management
|
||||
- Need better test strategy
|
||||
|
||||
Workflow:
|
||||
1. Export stories from Jira
|
||||
2. Run *test-design on epic
|
||||
3. Run *atdd for each story
|
||||
4. Implement features
|
||||
5. Run *trace for coverage
|
||||
```
|
||||
|
||||
**Verdict:** Best for teams wanting TEA benefits without BMad Method commitment.
|
||||
|
||||
---
|
||||
|
||||
### Model 3: TEA Lite
|
||||
|
||||
**What:** Beginner approach using just `*automate` to test existing features.
|
||||
|
||||
**When to Use:**
|
||||
- Learning TEA fundamentals
|
||||
- Want quick results
|
||||
- Testing existing application
|
||||
- No time for full methodology
|
||||
|
||||
**Workflow:**
|
||||
```
|
||||
1. *framework (setup test infrastructure)
|
||||
2. *test-design (optional, risk assessment)
|
||||
3. *automate (generate tests for existing features)
|
||||
4. Run tests (they pass immediately)
|
||||
```
|
||||
|
||||
**Example:**
|
||||
```
|
||||
Beginner developer:
|
||||
- Never used TEA before
|
||||
- Want to add tests to existing app
|
||||
- 30 minutes available
|
||||
|
||||
Steps:
|
||||
1. Run *framework
|
||||
2. Run *automate on TodoMVC demo
|
||||
3. Tests generated and passing
|
||||
4. Learn TEA basics
|
||||
```
|
||||
|
||||
**What You Get:**
|
||||
- Working test framework
|
||||
- Passing tests for existing features
|
||||
- Learning experience
|
||||
- Foundation to expand
|
||||
|
||||
**What You Miss:**
|
||||
- TDD workflow (ATDD)
|
||||
- Risk-based planning (test-design depth)
|
||||
- Quality gates (trace Phase 2)
|
||||
- Full TEA capabilities
|
||||
|
||||
**Verdict:** Perfect entry point for beginners.
|
||||
|
||||
---
|
||||
|
||||
### Model 4: TEA Integrated (Greenfield)
|
||||
|
||||
**What:** Full BMad Method integration with TEA workflows across all phases.
|
||||
|
||||
**When to Use:**
|
||||
- New projects starting from scratch
|
||||
- Using BMad Method or Enterprise track
|
||||
- Want complete quality operating model
|
||||
- Testing is critical to success
|
||||
|
||||
**Lifecycle:**
|
||||
|
||||
**Phase 2: Planning**
|
||||
- PM creates PRD with NFRs
|
||||
- (Optional) TEA runs `*nfr-assess` (Enterprise only)
|
||||
|
||||
**Phase 3: Solutioning**
|
||||
- Architect creates architecture
|
||||
- TEA runs `*test-design` (system-level) → testability review
|
||||
- TEA runs `*framework` → test infrastructure
|
||||
- TEA runs `*ci` → CI/CD pipeline
|
||||
- Architect runs `*implementation-readiness` (fed by test design)
|
||||
|
||||
**Phase 4: Implementation (Per Epic)**
|
||||
- SM runs `*sprint-planning`
|
||||
- TEA runs `*test-design` (epic-level) → risk assessment for THIS epic
|
||||
- SM creates stories
|
||||
- (Optional) TEA runs `*atdd` → failing tests before dev
|
||||
- DEV implements story
|
||||
- TEA runs `*automate` → expand coverage
|
||||
- (Optional) TEA runs `*test-review` → quality audit
|
||||
- TEA runs `*trace` Phase 1 → refresh coverage
|
||||
|
||||
**Release Gate:**
|
||||
- (Optional) TEA runs `*test-review` → final audit
|
||||
- (Optional) TEA runs `*nfr-assess` → validate NFRs
|
||||
- TEA runs `*trace` Phase 2 → gate decision (PASS/CONCERNS/FAIL/WAIVED)
|
||||
|
||||
**What You Get:**
|
||||
- Complete quality operating model
|
||||
- Systematic test planning
|
||||
- Risk-based prioritization
|
||||
- Evidence-based gate decisions
|
||||
- Consistent patterns across epics
|
||||
|
||||
**Example:**
|
||||
```
|
||||
New SaaS product:
|
||||
- 50 stories across 8 epics
|
||||
- Security critical
|
||||
- Need quality gates
|
||||
|
||||
Workflow:
|
||||
- Phase 2: Define NFRs in PRD
|
||||
- Phase 3: Architecture → test design → framework → CI
|
||||
- Phase 4: Per epic: test design → ATDD → dev → automate → review → trace
|
||||
- Gate: NFR assess → trace Phase 2 → decision
|
||||
```
|
||||
|
||||
**Verdict:** Most comprehensive TEA usage, best for structured teams.
|
||||
|
||||
---
|
||||
|
||||
### Model 5: TEA Integrated (Brownfield)
|
||||
|
||||
**What:** Full BMad Method integration with TEA for existing codebases.
|
||||
|
||||
**When to Use:**
|
||||
- Existing codebase with legacy tests
|
||||
- Want to improve test quality incrementally
|
||||
- Adding features to existing application
|
||||
- Need to establish coverage baseline
|
||||
|
||||
**Differences from Greenfield:**
|
||||
|
||||
**Phase 0: Documentation (if needed)**
|
||||
```
|
||||
- Run *document-project
|
||||
- Create baseline documentation
|
||||
```
|
||||
|
||||
**Phase 2: Planning**
|
||||
```
|
||||
- TEA runs *trace Phase 1 → establish coverage baseline
|
||||
- PM creates PRD (with existing system context)
|
||||
```
|
||||
|
||||
**Phase 3: Solutioning**
|
||||
```
|
||||
- Architect creates architecture (with brownfield constraints)
|
||||
- TEA runs *test-design (system-level) → testability review
|
||||
- TEA runs *framework (only if modernizing test infra)
|
||||
- TEA runs *ci (update existing CI or create new)
|
||||
```
|
||||
|
||||
**Phase 4: Implementation**
|
||||
```
|
||||
- TEA runs *test-design (epic-level) → focus on REGRESSION HOTSPOTS
|
||||
- Per story: ATDD → dev → automate
|
||||
- TEA runs *test-review → improve legacy test quality
|
||||
- TEA runs *trace Phase 1 → track coverage improvement
|
||||
```
|
||||
|
||||
**Brownfield-Specific:**
|
||||
- Baseline coverage BEFORE planning
|
||||
- Focus on regression hotspots (bug-prone areas)
|
||||
- Incremental quality improvement
|
||||
- Compare coverage to baseline (trending up?)
|
||||
|
||||
**Example:**
|
||||
```
|
||||
Legacy e-commerce platform:
|
||||
- 200 existing tests (30% passing, 70% flaky)
|
||||
- Adding new checkout flow
|
||||
- Want to improve quality
|
||||
|
||||
Workflow:
|
||||
1. Phase 2: *trace baseline → 30% coverage
|
||||
2. Phase 3: *test-design → identify regression risks
|
||||
3. Phase 4: Fix top 20 flaky tests + add tests for new checkout
|
||||
4. Gate: *trace → 60% coverage (2x improvement)
|
||||
```
|
||||
|
||||
**Verdict:** Best for incrementally improving legacy systems.
|
||||
|
||||
---
|
||||
|
||||
## Decision Guide: Which Model?
|
||||
|
||||
### Quick Decision Tree
|
||||
|
||||
```mermaid
|
||||
%%{init: {'theme':'base', 'themeVariables': { 'fontSize':'14px'}}}%%
|
||||
flowchart TD
|
||||
Start([Choose TEA Model]) --> BMad{Using<br/>BMad Method?}
|
||||
|
||||
BMad -->|No| NonBMad{Project Type?}
|
||||
NonBMad -->|Learning| Lite[TEA Lite<br/>Just *automate<br/>30 min tutorial]
|
||||
NonBMad -->|Serious Project| Solo[TEA Solo<br/>Standalone workflows<br/>Full capabilities]
|
||||
|
||||
BMad -->|Yes| WantTEA{Want TEA?}
|
||||
WantTEA -->|No| None[No TEA<br/>Use existing approach<br/>Valid choice]
|
||||
WantTEA -->|Yes| ProjectType{New or<br/>Existing?}
|
||||
|
||||
ProjectType -->|New Project| Green[TEA Integrated<br/>Greenfield<br/>Full lifecycle]
|
||||
ProjectType -->|Existing Code| Brown[TEA Integrated<br/>Brownfield<br/>Baseline + improve]
|
||||
|
||||
Green --> Compliance{Compliance<br/>Needs?}
|
||||
Compliance -->|Yes| Enterprise[Enterprise Track<br/>NFR + audit trails]
|
||||
Compliance -->|No| Method[BMad Method Track<br/>Standard quality]
|
||||
|
||||
style Lite fill:#bbdefb,stroke:#1565c0,stroke-width:2px
|
||||
style Solo fill:#c5cae9,stroke:#283593,stroke-width:2px
|
||||
style None fill:#e0e0e0,stroke:#616161,stroke-width:1px
|
||||
style Green fill:#c8e6c9,stroke:#2e7d32,stroke-width:2px
|
||||
style Brown fill:#fff9c4,stroke:#f57f17,stroke-width:2px
|
||||
style Enterprise fill:#f3e5f5,stroke:#6a1b9a,stroke-width:2px
|
||||
style Method fill:#e1f5fe,stroke:#01579b,stroke-width:2px
|
||||
```
|
||||
|
||||
**Decision Path Examples:**
|
||||
- Learning TEA → TEA Lite (blue)
|
||||
- Non-BMad project → TEA Solo (purple)
|
||||
- BMad + new project + compliance → Enterprise (purple)
|
||||
- BMad + existing code → Brownfield (yellow)
|
||||
- Don't want TEA → No TEA (gray)
|
||||
|
||||
### By Project Type
|
||||
|
||||
| Project Type | Recommended Model | Why |
|
||||
|--------------|------------------|-----|
|
||||
| **New SaaS product** | TEA Integrated (Greenfield) | Full quality operating model from day one |
|
||||
| **Existing app + new feature** | TEA Integrated (Brownfield) | Improve incrementally while adding features |
|
||||
| **Bug fix** | TEA Lite or No TEA | Quick flow, minimal overhead |
|
||||
| **Learning project** | TEA Lite | Learn basics with immediate results |
|
||||
| **Non-BMad enterprise** | TEA Solo | Quality model without full methodology |
|
||||
| **High-quality existing tests** | No TEA | Keep what works |
|
||||
|
||||
### By Team Maturity
|
||||
|
||||
| Team Maturity | Recommended Model | Why |
|
||||
|---------------|------------------|-----|
|
||||
| **Beginners** | TEA Lite → TEA Solo | Learn basics, then expand |
|
||||
| **Intermediate** | TEA Solo or Integrated | Depends on methodology |
|
||||
| **Advanced** | TEA Integrated or No TEA | Full model or existing expertise |
|
||||
|
||||
### By Compliance Needs
|
||||
|
||||
| Compliance | Recommended Model | Why |
|
||||
|------------|------------------|-----|
|
||||
| **None** | Any model | Choose based on project needs |
|
||||
| **Light** (internal audit) | TEA Solo or Integrated | Gate decisions helpful |
|
||||
| **Heavy** (SOC 2, HIPAA) | TEA Integrated (Enterprise) | NFR assessment mandatory |
|
||||
|
||||
## Switching Between Models
|
||||
|
||||
### Can Change Models Mid-Project
|
||||
|
||||
**Scenario:** Start with TEA Lite, expand to TEA Solo
|
||||
|
||||
```
|
||||
Week 1: TEA Lite
|
||||
- Run *framework
|
||||
- Run *automate
|
||||
- Learn basics
|
||||
|
||||
Week 2: Expand to TEA Solo
|
||||
- Add *test-design
|
||||
- Use *atdd for new features
|
||||
- Add *test-review
|
||||
|
||||
Week 3: Continue expanding
|
||||
- Add *trace for coverage
|
||||
- Setup *ci
|
||||
- Full TEA Solo workflow
|
||||
```
|
||||
|
||||
**Benefit:** Start small, expand as comfortable.
|
||||
|
||||
### Can Mix Models
|
||||
|
||||
**Scenario:** TEA Integrated for main features, No TEA for bug fixes
|
||||
|
||||
```
|
||||
Main features (epics):
|
||||
- Use full TEA workflow
|
||||
- Risk assessment, ATDD, quality gates
|
||||
|
||||
Bug fixes:
|
||||
- Skip TEA
|
||||
- Quick Flow + manual testing
|
||||
- Move fast
|
||||
|
||||
Result: TEA where it adds value, skip where it doesn't
|
||||
```
|
||||
|
||||
**Benefit:** Flexible, pragmatic, not dogmatic.
|
||||
|
||||
## Comparison Table
|
||||
|
||||
| Aspect | No TEA | TEA Lite | TEA Solo | Integrated (Green) | Integrated (Brown) |
|
||||
|--------|--------|----------|----------|-------------------|-------------------|
|
||||
| **BMad Required** | No | No | No | Yes | Yes |
|
||||
| **Learning Curve** | None | Low | Medium | High | High |
|
||||
| **Setup Time** | 0 | 30 min | 2 hours | 1 day | 2 days |
|
||||
| **Workflows Used** | 0 | 2-3 | 4-6 | 8 | 8 |
|
||||
| **Test Planning** | Manual | Optional | Yes | Systematic | + Regression focus |
|
||||
| **Quality Gates** | No | No | Optional | Yes | Yes + baseline |
|
||||
| **NFR Assessment** | No | No | No | Optional | Recommended |
|
||||
| **Coverage Tracking** | Manual | No | Optional | Yes | Yes + trending |
|
||||
| **Best For** | Experts | Beginners | Standalone | New projects | Legacy code |
|
||||
|
||||
## Real-World Examples
|
||||
|
||||
### Example 1: Startup (TEA Lite → TEA Integrated)
|
||||
|
||||
**Month 1:** TEA Lite
|
||||
```
|
||||
Team: 3 developers, no QA
|
||||
Testing: Manual only
|
||||
Decision: Start with TEA Lite
|
||||
|
||||
Result:
|
||||
- Run *framework (Playwright setup)
|
||||
- Run *automate (20 tests generated)
|
||||
- Learning TEA basics
|
||||
```
|
||||
|
||||
**Month 3:** TEA Solo
|
||||
```
|
||||
Team: Growing to 5 developers
|
||||
Testing: Automated tests exist
|
||||
Decision: Expand to TEA Solo
|
||||
|
||||
Result:
|
||||
- Add *test-design (risk assessment)
|
||||
- Add *atdd (TDD workflow)
|
||||
- Add *test-review (quality audits)
|
||||
```
|
||||
|
||||
**Month 6:** TEA Integrated
|
||||
```
|
||||
Team: 8 developers, 1 QA
|
||||
Testing: Critical to business
|
||||
Decision: Full BMad Method + TEA Integrated
|
||||
|
||||
Result:
|
||||
- Full lifecycle integration
|
||||
- Quality gates before releases
|
||||
- NFR assessment for enterprise customers
|
||||
```
|
||||
|
||||
### Example 2: Enterprise (TEA Integrated - Brownfield)
|
||||
|
||||
**Project:** Legacy banking application
|
||||
|
||||
**Challenge:**
|
||||
- 500 existing tests (50% flaky)
|
||||
- Adding new features
|
||||
- SOC 2 compliance required
|
||||
|
||||
**Model:** TEA Integrated (Brownfield)
|
||||
|
||||
**Phase 2:**
|
||||
```
|
||||
- *trace baseline → 45% coverage (lots of gaps)
|
||||
- Document current state
|
||||
```
|
||||
|
||||
**Phase 3:**
|
||||
```
|
||||
- *test-design (system) → identify regression hotspots
|
||||
- *framework → modernize test infrastructure
|
||||
- *ci → add selective testing
|
||||
```
|
||||
|
||||
**Phase 4:**
|
||||
```
|
||||
Per epic:
|
||||
- *test-design → focus on regression + new features
|
||||
- Fix top 10 flaky tests
|
||||
- *atdd for new features
|
||||
- *automate for coverage expansion
|
||||
- *test-review → track quality improvement
|
||||
- *trace → compare to baseline
|
||||
```
|
||||
|
||||
**Result after 6 months:**
|
||||
- Coverage: 45% → 85%
|
||||
- Quality score: 52 → 82
|
||||
- Flakiness: 50% → 2%
|
||||
- SOC 2 compliant (traceability + NFR evidence)
|
||||
|
||||
### Example 3: Consultancy (TEA Solo)
|
||||
|
||||
**Context:** Testing consultancy working with multiple clients
|
||||
|
||||
**Challenge:**
|
||||
- Different clients use different methodologies
|
||||
- Need consistent testing approach
|
||||
- Not always using BMad Method
|
||||
|
||||
**Model:** TEA Solo (bring to any client project)
|
||||
|
||||
**Workflow:**
|
||||
```
|
||||
Client project 1 (Scrum):
|
||||
- Import Jira stories
|
||||
- Run *test-design
|
||||
- Generate tests with *atdd/*automate
|
||||
- Deliver quality report with *test-review
|
||||
|
||||
Client project 2 (Kanban):
|
||||
- Import requirements from Notion
|
||||
- Same TEA workflow
|
||||
- Consistent quality across clients
|
||||
|
||||
Client project 3 (Ad-hoc):
|
||||
- Document requirements manually
|
||||
- Same TEA workflow
|
||||
- Same patterns, different context
|
||||
```
|
||||
|
||||
**Benefit:** Consistent testing approach regardless of client methodology.
|
||||
|
||||
## Choosing Your Model
|
||||
|
||||
### Start Here Questions
|
||||
|
||||
**Question 1:** Are you using BMad Method?
|
||||
- **No** → TEA Solo or TEA Lite or No TEA
|
||||
- **Yes** → TEA Integrated or No TEA
|
||||
|
||||
**Question 2:** Is this a new project?
|
||||
- **Yes** → TEA Integrated (Greenfield) or TEA Lite
|
||||
- **No** → TEA Integrated (Brownfield) or TEA Solo
|
||||
|
||||
**Question 3:** What's your testing maturity?
|
||||
- **Beginner** → TEA Lite
|
||||
- **Intermediate** → TEA Solo or Integrated
|
||||
- **Advanced** → TEA Integrated or No TEA (already expert)
|
||||
|
||||
**Question 4:** Do you need compliance/quality gates?
|
||||
- **Yes** → TEA Integrated (Enterprise)
|
||||
- **No** → Any model
|
||||
|
||||
**Question 5:** How much time can you invest?
|
||||
- **30 minutes** → TEA Lite
|
||||
- **Few hours** → TEA Solo
|
||||
- **Multiple days** → TEA Integrated
|
||||
|
||||
### Recommendation Matrix
|
||||
|
||||
| Your Context | Recommended Model | Alternative |
|
||||
|--------------|------------------|-------------|
|
||||
| BMad Method + new project | TEA Integrated (Greenfield) | TEA Lite (learning) |
|
||||
| BMad Method + existing code | TEA Integrated (Brownfield) | TEA Solo |
|
||||
| Non-BMad + need quality | TEA Solo | TEA Lite |
|
||||
| Just learning testing | TEA Lite | No TEA (learn basics first) |
|
||||
| Enterprise + compliance | TEA Integrated (Enterprise) | TEA Solo |
|
||||
| Established QA team | No TEA | TEA Solo (supplement) |
|
||||
|
||||
## Transitioning Between Models
|
||||
|
||||
### TEA Lite → TEA Solo
|
||||
|
||||
**When:** Outgrow beginner approach, need more workflows.
|
||||
|
||||
**Steps:**
|
||||
1. Continue using `*framework` and `*automate`
|
||||
2. Add `*test-design` for planning
|
||||
3. Add `*atdd` for TDD workflow
|
||||
4. Add `*test-review` for quality audits
|
||||
5. Add `*trace` for coverage tracking
|
||||
|
||||
**Timeline:** 2-4 weeks of gradual expansion
|
||||
|
||||
### TEA Solo → TEA Integrated
|
||||
|
||||
**When:** Adopt BMad Method, want full integration.
|
||||
|
||||
**Steps:**
|
||||
1. Install BMad Method (see installation guide)
|
||||
2. Run planning workflows (PRD, architecture)
|
||||
3. Integrate TEA into Phase 3 (system-level test design)
|
||||
4. Follow integrated lifecycle (per epic workflows)
|
||||
5. Add release gates (trace Phase 2)
|
||||
|
||||
**Timeline:** 1-2 sprints of transition
|
||||
|
||||
### TEA Integrated → TEA Solo
|
||||
|
||||
**When:** Moving away from BMad Method, keep TEA.
|
||||
|
||||
**Steps:**
|
||||
1. Export BMad artifacts (PRD, architecture, stories)
|
||||
2. Continue using TEA workflows standalone
|
||||
3. Skip BMad-specific integration
|
||||
4. Bring your own requirements to TEA
|
||||
|
||||
**Timeline:** Immediate (just skip BMad workflows)
|
||||
|
||||
## Common Patterns
|
||||
|
||||
### Pattern 1: TEA Lite for Learning, Then Choose
|
||||
|
||||
```
|
||||
Phase 1 (Week 1-2): TEA Lite
|
||||
- Learn with *automate on demo app
|
||||
- Understand TEA fundamentals
|
||||
- Low commitment
|
||||
|
||||
Phase 2 (Week 3-4): Evaluate
|
||||
- Try *test-design (planning)
|
||||
- Try *atdd (TDD)
|
||||
- See if value justifies investment
|
||||
|
||||
Phase 3 (Month 2+): Decide
|
||||
- Valuable → Expand to TEA Solo or Integrated
|
||||
- Not valuable → Stay with TEA Lite or No TEA
|
||||
```
|
||||
|
||||
### Pattern 2: TEA Solo for Quality, Skip Full Method
|
||||
|
||||
```
|
||||
Team decision:
|
||||
- Don't want full BMad Method (too heavyweight)
|
||||
- Want systematic testing (TEA benefits)
|
||||
|
||||
Approach: TEA Solo only
|
||||
- Use existing project management (Jira, Linear)
|
||||
- Use TEA for testing only
|
||||
- Get quality without methodology commitment
|
||||
```
|
||||
|
||||
### Pattern 3: Integrated for Critical, Lite for Non-Critical
|
||||
|
||||
```
|
||||
Critical features (payment, auth):
|
||||
- Full TEA Integrated workflow
|
||||
- Risk assessment, ATDD, quality gates
|
||||
- High confidence required
|
||||
|
||||
Non-critical features (UI tweaks):
|
||||
- TEA Lite or No TEA
|
||||
- Quick tests, minimal overhead
|
||||
- Move fast
|
||||
```
|
||||
|
||||
## Technical Implementation
|
||||
|
||||
Each model uses different TEA workflows. See:
|
||||
- [TEA Overview](/docs/explanation/features/tea-overview.md) - Model details
|
||||
- [TEA Command Reference](/docs/reference/tea/commands.md) - Workflow reference
|
||||
- [TEA Configuration](/docs/reference/tea/configuration.md) - Setup options
|
||||
|
||||
## Related Concepts
|
||||
|
||||
**Core TEA Concepts:**
|
||||
- [Risk-Based Testing](/docs/explanation/tea/risk-based-testing.md) - Risk assessment in different models
|
||||
- [Test Quality Standards](/docs/explanation/tea/test-quality-standards.md) - Quality across all models
|
||||
- [Knowledge Base System](/docs/explanation/tea/knowledge-base-system.md) - Consistent patterns across models
|
||||
|
||||
**Technical Patterns:**
|
||||
- [Fixture Architecture](/docs/explanation/tea/fixture-architecture.md) - Infrastructure in different models
|
||||
- [Network-First Patterns](/docs/explanation/tea/network-first-patterns.md) - Reliability in all models
|
||||
|
||||
**Overview:**
|
||||
- [TEA Overview](/docs/explanation/features/tea-overview.md) - 5 engagement models with cheat sheets
|
||||
- [Testing as Engineering](/docs/explanation/philosophy/testing-as-engineering.md) - Design philosophy
|
||||
|
||||
## Practical Guides
|
||||
|
||||
**Getting Started:**
|
||||
- [TEA Lite Quickstart Tutorial](/docs/tutorials/getting-started/tea-lite-quickstart.md) - Model 3: TEA Lite
|
||||
|
||||
**Use-Case Guides:**
|
||||
- [Using TEA with Existing Tests](/docs/how-to/brownfield/use-tea-with-existing-tests.md) - Model 5: Brownfield
|
||||
- [Running TEA for Enterprise](/docs/how-to/enterprise/use-tea-for-enterprise.md) - Enterprise integration
|
||||
|
||||
**All Workflow Guides:**
|
||||
- [How to Run Test Design](/docs/how-to/workflows/run-test-design.md) - Used in TEA Solo and Integrated
|
||||
- [How to Run ATDD](/docs/how-to/workflows/run-atdd.md)
|
||||
- [How to Run Automate](/docs/how-to/workflows/run-automate.md)
|
||||
- [How to Run Test Review](/docs/how-to/workflows/run-test-review.md)
|
||||
- [How to Run Trace](/docs/how-to/workflows/run-trace.md)
|
||||
|
||||
## Reference
|
||||
|
||||
- [TEA Command Reference](/docs/reference/tea/commands.md) - All workflows explained
|
||||
- [TEA Configuration](/docs/reference/tea/configuration.md) - Config per model
|
||||
- [Glossary](/docs/reference/glossary/index.md#test-architect-tea-concepts) - TEA Lite, TEA Solo, TEA Integrated terms
|
||||
|
||||
---
|
||||
|
||||
Generated with [BMad Method](https://bmad-method.org) - TEA (Test Architect)
|
||||
457
docs/explanation/tea/fixture-architecture.md
Normal file
457
docs/explanation/tea/fixture-architecture.md
Normal file
@@ -0,0 +1,457 @@
|
||||
---
|
||||
title: "Fixture Architecture Explained"
|
||||
description: Understanding TEA's pure function → fixture → composition pattern for reusable test utilities
|
||||
---
|
||||
|
||||
# Fixture Architecture Explained
|
||||
|
||||
Fixture architecture is TEA's pattern for building reusable, testable, and composable test utilities. The core principle: build pure functions first, wrap in framework fixtures second.
|
||||
|
||||
## Overview
|
||||
|
||||
**The Pattern:**
|
||||
1. Write utility as pure function (unit-testable)
|
||||
2. Wrap in framework fixture (Playwright, Cypress)
|
||||
3. Compose fixtures with mergeTests (combine capabilities)
|
||||
4. Package for reuse across projects
|
||||
|
||||
**Why this order?**
|
||||
- Pure functions are easier to test
|
||||
- Fixtures depend on framework (less portable)
|
||||
- Composition happens at fixture level
|
||||
- Reusability maximized
|
||||
|
||||
### Fixture Architecture Flow
|
||||
|
||||
```mermaid
|
||||
%%{init: {'theme':'base', 'themeVariables': { 'fontSize':'14px'}}}%%
|
||||
flowchart TD
|
||||
Start([Testing Need]) --> Pure[Step 1: Pure Function<br/>helpers/api-request.ts]
|
||||
Pure -->|Unit testable<br/>Framework agnostic| Fixture[Step 2: Fixture Wrapper<br/>fixtures/api-request.ts]
|
||||
Fixture -->|Injects framework<br/>dependencies| Compose[Step 3: Composition<br/>fixtures/index.ts]
|
||||
Compose -->|mergeTests| Use[Step 4: Use in Tests<br/>tests/**.spec.ts]
|
||||
|
||||
Pure -.->|Can test in isolation| UnitTest[Unit Tests<br/>No framework needed]
|
||||
Fixture -.->|Reusable pattern| Other[Other Projects<br/>Package export]
|
||||
Compose -.->|Combine utilities| Multi[Multiple Fixtures<br/>One test]
|
||||
|
||||
style Pure fill:#e3f2fd,stroke:#1565c0,stroke-width:2px
|
||||
style Fixture fill:#fff3e0,stroke:#e65100,stroke-width:2px
|
||||
style Compose fill:#f3e5f5,stroke:#6a1b9a,stroke-width:2px
|
||||
style Use fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
|
||||
style UnitTest fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px
|
||||
style Other fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px
|
||||
style Multi fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px
|
||||
```
|
||||
|
||||
**Benefits at Each Step:**
|
||||
1. **Pure Function:** Testable, portable, reusable
|
||||
2. **Fixture:** Framework integration, clean API
|
||||
3. **Composition:** Combine capabilities, flexible
|
||||
4. **Usage:** Simple imports, type-safe
|
||||
|
||||
## The Problem
|
||||
|
||||
### Framework-First Approach (Common Anti-Pattern)
|
||||
|
||||
```typescript
|
||||
// ❌ Bad: Built as fixture from the start
|
||||
export const test = base.extend({
|
||||
apiRequest: async ({ request }, use) => {
|
||||
await use(async (options) => {
|
||||
const response = await request.fetch(options.url, {
|
||||
method: options.method,
|
||||
data: options.data
|
||||
});
|
||||
|
||||
if (!response.ok()) {
|
||||
throw new Error(`API request failed: ${response.status()}`);
|
||||
}
|
||||
|
||||
return response.json();
|
||||
});
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
**Problems:**
|
||||
- Cannot unit test (requires Playwright context)
|
||||
- Tied to framework (not reusable in other tools)
|
||||
- Hard to compose with other fixtures
|
||||
- Difficult to mock for testing the utility itself
|
||||
|
||||
### Copy-Paste Utilities
|
||||
|
||||
```typescript
|
||||
// test-1.spec.ts
|
||||
test('test 1', async ({ request }) => {
|
||||
const response = await request.post('/api/users', { data: {...} });
|
||||
const body = await response.json();
|
||||
if (!response.ok()) throw new Error('Failed');
|
||||
// ... repeated in every test
|
||||
});
|
||||
|
||||
// test-2.spec.ts
|
||||
test('test 2', async ({ request }) => {
|
||||
const response = await request.post('/api/users', { data: {...} });
|
||||
const body = await response.json();
|
||||
if (!response.ok()) throw new Error('Failed');
|
||||
// ... same code repeated
|
||||
});
|
||||
```
|
||||
|
||||
**Problems:**
|
||||
- Code duplication (violates DRY)
|
||||
- Inconsistent error handling
|
||||
- Hard to update (change 50 tests)
|
||||
- No shared behavior
|
||||
|
||||
## The Solution: Three-Step Pattern
|
||||
|
||||
### Step 1: Pure Function
|
||||
|
||||
```typescript
|
||||
// helpers/api-request.ts
|
||||
|
||||
/**
|
||||
* Make API request with automatic error handling
|
||||
* Pure function - no framework dependencies
|
||||
*/
|
||||
export async function apiRequest({
|
||||
request, // Passed in (dependency injection)
|
||||
method,
|
||||
url,
|
||||
data,
|
||||
headers = {}
|
||||
}: ApiRequestParams): Promise<ApiResponse> {
|
||||
const response = await request.fetch(url, {
|
||||
method,
|
||||
data,
|
||||
headers
|
||||
});
|
||||
|
||||
if (!response.ok()) {
|
||||
throw new Error(`API request failed: ${response.status()}`);
|
||||
}
|
||||
|
||||
return {
|
||||
status: response.status(),
|
||||
body: await response.json()
|
||||
};
|
||||
}
|
||||
|
||||
// ✅ Can unit test this function!
|
||||
describe('apiRequest', () => {
|
||||
it('should throw on non-OK response', async () => {
|
||||
const mockRequest = {
|
||||
fetch: vi.fn().mockResolvedValue({ ok: () => false, status: () => 500 })
|
||||
};
|
||||
|
||||
await expect(apiRequest({
|
||||
request: mockRequest,
|
||||
method: 'GET',
|
||||
url: '/api/test'
|
||||
})).rejects.toThrow('API request failed: 500');
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
**Benefits:**
|
||||
- Unit testable (mock dependencies)
|
||||
- Framework-agnostic (works with any HTTP client)
|
||||
- Easy to reason about (pure function)
|
||||
- Portable (can use in Node scripts, CLI tools)
|
||||
|
||||
### Step 2: Fixture Wrapper
|
||||
|
||||
```typescript
|
||||
// fixtures/api-request.ts
|
||||
import { test as base } from '@playwright/test';
|
||||
import { apiRequest as apiRequestFn } from '../helpers/api-request';
|
||||
|
||||
/**
|
||||
* Playwright fixture wrapping the pure function
|
||||
*/
|
||||
export const test = base.extend<{ apiRequest: typeof apiRequestFn }>({
|
||||
apiRequest: async ({ request }, use) => {
|
||||
// Inject framework dependency (request)
|
||||
await use((params) => apiRequestFn({ request, ...params }));
|
||||
}
|
||||
});
|
||||
|
||||
export { expect } from '@playwright/test';
|
||||
```
|
||||
|
||||
**Benefits:**
|
||||
- Fixture provides framework context (request)
|
||||
- Pure function handles logic
|
||||
- Clean separation of concerns
|
||||
- Can swap frameworks (Cypress, etc.) by changing wrapper only
|
||||
|
||||
### Step 3: Composition with mergeTests
|
||||
|
||||
```typescript
|
||||
// fixtures/index.ts
|
||||
import { mergeTests } from '@playwright/test';
|
||||
import { test as apiRequestTest } from './api-request';
|
||||
import { test as authSessionTest } from './auth-session';
|
||||
import { test as logTest } from './log';
|
||||
|
||||
/**
|
||||
* Compose all fixtures into one test
|
||||
*/
|
||||
export const test = mergeTests(
|
||||
apiRequestTest,
|
||||
authSessionTest,
|
||||
logTest
|
||||
);
|
||||
|
||||
export { expect } from '@playwright/test';
|
||||
```
|
||||
|
||||
**Usage:**
|
||||
```typescript
|
||||
// tests/profile.spec.ts
|
||||
import { test, expect } from '../support/fixtures';
|
||||
|
||||
test('should update profile', async ({ apiRequest, authToken, log }) => {
|
||||
log.info('Starting profile update test');
|
||||
|
||||
// Use API request fixture (matches pure function signature)
|
||||
const { status, body } = await apiRequest({
|
||||
method: 'PATCH',
|
||||
url: '/api/profile',
|
||||
data: { name: 'New Name' },
|
||||
headers: { Authorization: `Bearer ${authToken}` }
|
||||
});
|
||||
|
||||
expect(status).toBe(200);
|
||||
expect(body.name).toBe('New Name');
|
||||
|
||||
log.info('Profile updated successfully');
|
||||
});
|
||||
```
|
||||
|
||||
**Note:** This example uses the vanilla pure function signature (`url`, `data`). Playwright Utils uses different parameter names (`path`, `body`). See [Integrate Playwright Utils](/docs/how-to/customization/integrate-playwright-utils.md) for the utilities API.
|
||||
|
||||
**Note:** `authToken` requires auth-session fixture setup with provider configuration. See [auth-session documentation](https://seontechnologies.github.io/playwright-utils/auth-session.html).
|
||||
|
||||
**Benefits:**
|
||||
- Use multiple fixtures in one test
|
||||
- No manual composition needed
|
||||
- Type-safe (TypeScript knows all fixture types)
|
||||
- Clean imports
|
||||
|
||||
## How It Works in TEA
|
||||
|
||||
### TEA Generates This Pattern
|
||||
|
||||
When you run `*framework` with `tea_use_playwright_utils: true`:
|
||||
|
||||
**TEA scaffolds:**
|
||||
```
|
||||
tests/
|
||||
├── support/
|
||||
│ ├── helpers/ # Pure functions
|
||||
│ │ ├── api-request.ts
|
||||
│ │ └── auth-session.ts
|
||||
│ └── fixtures/ # Framework wrappers
|
||||
│ ├── api-request.ts
|
||||
│ ├── auth-session.ts
|
||||
│ └── index.ts # Composition
|
||||
└── e2e/
|
||||
└── example.spec.ts # Uses composed fixtures
|
||||
```
|
||||
|
||||
### TEA Reviews Against This Pattern
|
||||
|
||||
When you run `*test-review`:
|
||||
|
||||
**TEA checks:**
|
||||
- Are utilities pure functions? ✓
|
||||
- Are fixtures minimal wrappers? ✓
|
||||
- Is composition used? ✓
|
||||
- Can utilities be unit tested? ✓
|
||||
|
||||
## Package Export Pattern
|
||||
|
||||
### Make Fixtures Reusable Across Projects
|
||||
|
||||
**Option 1: Build Your Own (Vanilla)**
|
||||
```json
|
||||
// package.json
|
||||
{
|
||||
"name": "@company/test-utils",
|
||||
"exports": {
|
||||
"./api-request": "./fixtures/api-request.ts",
|
||||
"./auth-session": "./fixtures/auth-session.ts",
|
||||
"./log": "./fixtures/log.ts"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Usage:**
|
||||
```typescript
|
||||
import { test as apiTest } from '@company/test-utils/api-request';
|
||||
import { test as authTest } from '@company/test-utils/auth-session';
|
||||
import { mergeTests } from '@playwright/test';
|
||||
|
||||
export const test = mergeTests(apiTest, authTest);
|
||||
```
|
||||
|
||||
**Option 2: Use Playwright Utils (Recommended)**
|
||||
```bash
|
||||
npm install -D @seontechnologies/playwright-utils
|
||||
```
|
||||
|
||||
**Usage:**
|
||||
```typescript
|
||||
import { test as base } from '@playwright/test';
|
||||
import { mergeTests } from '@playwright/test';
|
||||
import { test as apiRequestFixture } from '@seontechnologies/playwright-utils/api-request/fixtures';
|
||||
import { createAuthFixtures } from '@seontechnologies/playwright-utils/auth-session';
|
||||
|
||||
const authFixtureTest = base.extend(createAuthFixtures());
|
||||
export const test = mergeTests(apiRequestFixture, authFixtureTest);
|
||||
// Production-ready utilities, battle-tested!
|
||||
```
|
||||
|
||||
**Note:** Auth-session requires provider configuration. See [auth-session setup guide](https://seontechnologies.github.io/playwright-utils/auth-session.html).
|
||||
|
||||
**Why Playwright Utils:**
|
||||
- Already built, tested, and maintained
|
||||
- Consistent patterns across projects
|
||||
- 11 utilities available (API, auth, network, logging, files)
|
||||
- Community support and documentation
|
||||
- Regular updates and improvements
|
||||
|
||||
**When to Build Your Own:**
|
||||
- Company-specific patterns
|
||||
- Custom authentication systems
|
||||
- Unique requirements not covered by utilities
|
||||
|
||||
## Comparison: Good vs Bad Patterns
|
||||
|
||||
### Anti-Pattern: God Fixture
|
||||
|
||||
```typescript
|
||||
// ❌ Bad: Everything in one fixture
|
||||
export const test = base.extend({
|
||||
testUtils: async ({ page, request, context }, use) => {
|
||||
await use({
|
||||
// 50 different methods crammed into one fixture
|
||||
apiRequest: async (...) => { },
|
||||
login: async (...) => { },
|
||||
createUser: async (...) => { },
|
||||
deleteUser: async (...) => { },
|
||||
uploadFile: async (...) => { },
|
||||
// ... 45 more methods
|
||||
});
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
**Problems:**
|
||||
- Cannot test individual utilities
|
||||
- Cannot compose (all-or-nothing)
|
||||
- Cannot reuse specific utilities
|
||||
- Hard to maintain (1000+ line file)
|
||||
|
||||
### Good Pattern: Single-Concern Fixtures
|
||||
|
||||
```typescript
|
||||
// ✅ Good: One concern per fixture
|
||||
|
||||
// api-request.ts
|
||||
export const test = base.extend({ apiRequest });
|
||||
|
||||
// auth-session.ts
|
||||
export const test = base.extend({ authSession });
|
||||
|
||||
// log.ts
|
||||
export const test = base.extend({ log });
|
||||
|
||||
// Compose as needed
|
||||
import { mergeTests } from '@playwright/test';
|
||||
export const test = mergeTests(apiRequestTest, authSessionTest, logTest);
|
||||
```
|
||||
|
||||
**Benefits:**
|
||||
- Each fixture is unit-testable
|
||||
- Compose only what you need
|
||||
- Reuse individual fixtures
|
||||
- Easy to maintain (small files)
|
||||
|
||||
## Technical Implementation
|
||||
|
||||
For detailed fixture architecture patterns, see the knowledge base:
|
||||
- [Knowledge Base Index - Architecture & Fixtures](/docs/reference/tea/knowledge-base.md)
|
||||
- [Complete Knowledge Base Index](/docs/reference/tea/knowledge-base.md)
|
||||
|
||||
## When to Use This Pattern
|
||||
|
||||
### Always Use For:
|
||||
|
||||
**Reusable utilities:**
|
||||
- API request helpers
|
||||
- Authentication handlers
|
||||
- File operations
|
||||
- Network mocking
|
||||
|
||||
**Test infrastructure:**
|
||||
- Shared fixtures across teams
|
||||
- Packaged utilities (playwright-utils)
|
||||
- Company-wide test standards
|
||||
|
||||
### Consider Skipping For:
|
||||
|
||||
**One-off test setup:**
|
||||
```typescript
|
||||
// Simple one-time setup - inline is fine
|
||||
test.beforeEach(async ({ page }) => {
|
||||
await page.goto('/');
|
||||
await page.click('#accept-cookies');
|
||||
});
|
||||
```
|
||||
|
||||
**Test-specific helpers:**
|
||||
```typescript
|
||||
// Used in one test file only - keep local
|
||||
function createTestUser(name: string) {
|
||||
return { name, email: `${name}@test.com` };
|
||||
}
|
||||
```
|
||||
|
||||
## Related Concepts
|
||||
|
||||
**Core TEA Concepts:**
|
||||
- [Test Quality Standards](/docs/explanation/tea/test-quality-standards.md) - Quality standards fixtures enforce
|
||||
- [Knowledge Base System](/docs/explanation/tea/knowledge-base-system.md) - Fixture patterns in knowledge base
|
||||
|
||||
**Technical Patterns:**
|
||||
- [Network-First Patterns](/docs/explanation/tea/network-first-patterns.md) - Network fixtures explained
|
||||
- [Risk-Based Testing](/docs/explanation/tea/risk-based-testing.md) - Fixture complexity matches risk
|
||||
|
||||
**Overview:**
|
||||
- [TEA Overview](/docs/explanation/features/tea-overview.md) - Fixture architecture in workflows
|
||||
- [Testing as Engineering](/docs/explanation/philosophy/testing-as-engineering.md) - Why fixtures matter
|
||||
|
||||
## Practical Guides
|
||||
|
||||
**Setup Guides:**
|
||||
- [How to Set Up Test Framework](/docs/how-to/workflows/setup-test-framework.md) - TEA scaffolds fixtures
|
||||
- [Integrate Playwright Utils](/docs/how-to/customization/integrate-playwright-utils.md) - Production-ready fixtures
|
||||
|
||||
**Workflow Guides:**
|
||||
- [How to Run ATDD](/docs/how-to/workflows/run-atdd.md) - Using fixtures in tests
|
||||
- [How to Run Automate](/docs/how-to/workflows/run-automate.md) - Fixture composition examples
|
||||
|
||||
## Reference
|
||||
|
||||
- [TEA Command Reference](/docs/reference/tea/commands.md) - *framework command
|
||||
- [Knowledge Base Index](/docs/reference/tea/knowledge-base.md) - Fixture architecture fragments
|
||||
- [Glossary](/docs/reference/glossary/index.md#test-architect-tea-concepts) - Fixture architecture term
|
||||
|
||||
---
|
||||
|
||||
Generated with [BMad Method](https://bmad-method.org) - TEA (Test Architect)
|
||||
554
docs/explanation/tea/knowledge-base-system.md
Normal file
554
docs/explanation/tea/knowledge-base-system.md
Normal file
@@ -0,0 +1,554 @@
|
||||
---
|
||||
title: "Knowledge Base System Explained"
|
||||
description: Understanding how TEA uses tea-index.csv for context engineering and consistent test quality
|
||||
---
|
||||
|
||||
# Knowledge Base System Explained
|
||||
|
||||
TEA's knowledge base system is how context engineering works - automatically loading domain-specific standards into AI context so tests are consistently high-quality regardless of prompt variation.
|
||||
|
||||
## Overview
|
||||
|
||||
**The Problem:** AI without context produces inconsistent results.
|
||||
|
||||
**Traditional approach:**
|
||||
```
|
||||
User: "Write tests for login"
|
||||
AI: [Generates tests with random quality]
|
||||
- Sometimes uses hard waits
|
||||
- Sometimes uses good patterns
|
||||
- Inconsistent across sessions
|
||||
- Quality depends on prompt
|
||||
```
|
||||
|
||||
**TEA with knowledge base:**
|
||||
```
|
||||
User: "Write tests for login"
|
||||
TEA: [Loads test-quality.md, network-first.md, auth-session.md]
|
||||
TEA: [Generates tests following established patterns]
|
||||
- Always uses network-first patterns
|
||||
- Always uses proper fixtures
|
||||
- Consistent across all sessions
|
||||
- Quality independent of prompt
|
||||
```
|
||||
|
||||
**Result:** Systematic quality, not random chance.
|
||||
|
||||
## The Problem
|
||||
|
||||
### Prompt-Driven Testing = Inconsistency
|
||||
|
||||
**Session 1:**
|
||||
```
|
||||
User: "Write tests for profile editing"
|
||||
|
||||
AI: [No context loaded]
|
||||
// Generates test with hard waits
|
||||
await page.waitForTimeout(3000);
|
||||
```
|
||||
|
||||
**Session 2:**
|
||||
```
|
||||
User: "Write comprehensive tests for profile editing with best practices"
|
||||
|
||||
AI: [Still no systematic context]
|
||||
// Generates test with some improvements, but still issues
|
||||
await page.waitForSelector('.success', { timeout: 10000 });
|
||||
```
|
||||
|
||||
**Session 3:**
|
||||
```
|
||||
User: "Write tests using network-first patterns and proper fixtures"
|
||||
|
||||
AI: [Better prompt, but still reinventing patterns]
|
||||
// Generates test with network-first, but inconsistent with other tests
|
||||
```
|
||||
|
||||
**Problem:** Quality depends on prompt engineering skill, no consistency.
|
||||
|
||||
### Knowledge Drift
|
||||
|
||||
Without a knowledge base:
|
||||
- Team A uses pattern X
|
||||
- Team B uses pattern Y
|
||||
- Both work, but inconsistent
|
||||
- No single source of truth
|
||||
- Patterns drift over time
|
||||
|
||||
## The Solution: tea-index.csv Manifest
|
||||
|
||||
### How It Works
|
||||
|
||||
**1. Manifest Defines Fragments**
|
||||
|
||||
`src/modules/bmm/testarch/tea-index.csv`:
|
||||
```csv
|
||||
id,name,description,tags,fragment_file
|
||||
test-quality,Test Quality,Execution limits and isolation rules,quality;standards,knowledge/test-quality.md
|
||||
network-first,Network-First Safeguards,Intercept-before-navigate workflow,network;stability,knowledge/network-first.md
|
||||
fixture-architecture,Fixture Architecture,Composable fixture patterns,fixtures;architecture,knowledge/fixture-architecture.md
|
||||
```
|
||||
|
||||
**2. Workflow Loads Relevant Fragments**
|
||||
|
||||
When user runs `*atdd`:
|
||||
```
|
||||
TEA reads tea-index.csv
|
||||
Identifies fragments needed for ATDD:
|
||||
- test-quality.md (quality standards)
|
||||
- network-first.md (avoid flakiness)
|
||||
- component-tdd.md (TDD patterns)
|
||||
- fixture-architecture.md (reusable fixtures)
|
||||
- data-factories.md (test data)
|
||||
|
||||
Loads only these 5 fragments (not all 33)
|
||||
Generates tests following these patterns
|
||||
```
|
||||
|
||||
**3. Consistent Output**
|
||||
|
||||
Every time `*atdd` runs:
|
||||
- Same fragments loaded
|
||||
- Same patterns applied
|
||||
- Same quality standards
|
||||
- Consistent test structure
|
||||
|
||||
**Result:** Tests look like they were written by the same expert, every time.
|
||||
|
||||
### Knowledge Base Loading Diagram
|
||||
|
||||
```mermaid
|
||||
%%{init: {'theme':'base', 'themeVariables': { 'fontSize':'14px'}}}%%
|
||||
flowchart TD
|
||||
User([User: *atdd]) --> Workflow[TEA Workflow<br/>Triggered]
|
||||
Workflow --> Read[Read Manifest<br/>tea-index.csv]
|
||||
|
||||
Read --> Identify{Identify Relevant<br/>Fragments for ATDD}
|
||||
|
||||
Identify -->|Needed| L1[✓ test-quality.md]
|
||||
Identify -->|Needed| L2[✓ network-first.md]
|
||||
Identify -->|Needed| L3[✓ component-tdd.md]
|
||||
Identify -->|Needed| L4[✓ data-factories.md]
|
||||
Identify -->|Needed| L5[✓ fixture-architecture.md]
|
||||
|
||||
Identify -.->|Skip| S1[✗ contract-testing.md]
|
||||
Identify -.->|Skip| S2[✗ burn-in.md]
|
||||
Identify -.->|Skip| S3[+ 26 other fragments]
|
||||
|
||||
L1 --> Context[AI Context<br/>5 fragments loaded]
|
||||
L2 --> Context
|
||||
L3 --> Context
|
||||
L4 --> Context
|
||||
L5 --> Context
|
||||
|
||||
Context --> Gen[Generate Tests<br/>Following patterns]
|
||||
Gen --> Out([Consistent Output<br/>Same quality every time])
|
||||
|
||||
style User fill:#e3f2fd,stroke:#1565c0,stroke-width:2px
|
||||
style Read fill:#fff3e0,stroke:#e65100,stroke-width:2px
|
||||
style L1 fill:#c8e6c9,stroke:#2e7d32,stroke-width:2px
|
||||
style L2 fill:#c8e6c9,stroke:#2e7d32,stroke-width:2px
|
||||
style L3 fill:#c8e6c9,stroke:#2e7d32,stroke-width:2px
|
||||
style L4 fill:#c8e6c9,stroke:#2e7d32,stroke-width:2px
|
||||
style L5 fill:#c8e6c9,stroke:#2e7d32,stroke-width:2px
|
||||
style S1 fill:#e0e0e0,stroke:#616161,stroke-width:1px
|
||||
style S2 fill:#e0e0e0,stroke:#616161,stroke-width:1px
|
||||
style S3 fill:#e0e0e0,stroke:#616161,stroke-width:1px
|
||||
style Context fill:#f3e5f5,stroke:#6a1b9a,stroke-width:3px
|
||||
style Out fill:#4caf50,stroke:#1b5e20,stroke-width:3px,color:#fff
|
||||
```
|
||||
|
||||
## Fragment Structure
|
||||
|
||||
### Anatomy of a Fragment
|
||||
|
||||
Each fragment follows this structure:
|
||||
|
||||
```markdown
|
||||
# Fragment Name
|
||||
|
||||
## Principle
|
||||
[One sentence - what is this pattern?]
|
||||
|
||||
## Rationale
|
||||
[Why use this instead of alternatives?]
|
||||
Why this pattern exists
|
||||
Problems it solves
|
||||
Benefits it provides
|
||||
|
||||
## Pattern Examples
|
||||
|
||||
### Example 1: Basic Usage
|
||||
```code
|
||||
[Runnable code example]
|
||||
```
|
||||
[Explanation of example]
|
||||
|
||||
### Example 2: Advanced Pattern
|
||||
```code
|
||||
[More complex example]
|
||||
```
|
||||
[Explanation]
|
||||
|
||||
## Anti-Patterns
|
||||
|
||||
### Don't Do This
|
||||
```code
|
||||
[Bad code example]
|
||||
```
|
||||
[Why it's bad]
|
||||
[What breaks]
|
||||
|
||||
## Related Patterns
|
||||
- [Link to related fragment]
|
||||
```
|
||||
|
||||
<!-- markdownlint-disable MD024 -->
|
||||
### Example: test-quality.md Fragment
|
||||
|
||||
```markdown
|
||||
# Test Quality
|
||||
|
||||
## Principle
|
||||
Tests must be deterministic, isolated, explicit, focused, and fast.
|
||||
|
||||
## Rationale
|
||||
Tests that fail randomly, depend on each other, or take too long lose team trust.
|
||||
[... detailed explanation ...]
|
||||
|
||||
## Pattern Examples
|
||||
|
||||
### Example 1: Deterministic Test
|
||||
```typescript
|
||||
// ✅ Wait for actual response, not timeout
|
||||
const promise = page.waitForResponse(matcher);
|
||||
await page.click('button');
|
||||
await promise;
|
||||
```
|
||||
|
||||
### Example 2: Isolated Test
|
||||
```typescript
|
||||
// ✅ Self-cleaning test
|
||||
test('test', async ({ page }) => {
|
||||
const userId = await createTestUser();
|
||||
// ... test logic ...
|
||||
await deleteTestUser(userId); // Cleanup
|
||||
});
|
||||
```
|
||||
|
||||
## Anti-Patterns
|
||||
|
||||
### Hard Waits
|
||||
```typescript
|
||||
// ❌ Non-deterministic
|
||||
await page.waitForTimeout(3000);
|
||||
```
|
||||
[Why this causes flakiness]
|
||||
```
|
||||
|
||||
**Total:** 24.5 KB, 12 code examples
|
||||
<!-- markdownlint-enable MD024 -->
|
||||
|
||||
## How TEA Uses the Knowledge Base
|
||||
|
||||
### Workflow-Specific Loading
|
||||
|
||||
**Different workflows load different fragments:**
|
||||
|
||||
| Workflow | Fragments Loaded | Purpose |
|
||||
|----------|-----------------|---------|
|
||||
| `*framework` | fixture-architecture, playwright-config, fixtures-composition | Infrastructure patterns |
|
||||
| `*test-design` | test-quality, test-priorities-matrix, risk-governance | Planning standards |
|
||||
| `*atdd` | test-quality, component-tdd, network-first, data-factories | TDD patterns |
|
||||
| `*automate` | test-quality, test-levels-framework, selector-resilience | Comprehensive generation |
|
||||
| `*test-review` | All quality/resilience/debugging fragments | Full audit patterns |
|
||||
| `*ci` | ci-burn-in, burn-in, selective-testing | CI/CD optimization |
|
||||
|
||||
**Benefit:** Only load what's needed (focused context, no bloat).
|
||||
|
||||
### Dynamic Fragment Selection
|
||||
|
||||
TEA doesn't load all 33 fragments at once:
|
||||
|
||||
```
|
||||
User runs: *atdd for authentication feature
|
||||
|
||||
TEA analyzes context:
|
||||
- Feature type: Authentication
|
||||
- Relevant fragments:
|
||||
- test-quality.md (always loaded)
|
||||
- auth-session.md (auth patterns)
|
||||
- network-first.md (avoid flakiness)
|
||||
- email-auth.md (if email-based auth)
|
||||
- data-factories.md (test users)
|
||||
|
||||
Skips:
|
||||
- contract-testing.md (not relevant)
|
||||
- feature-flags.md (not relevant)
|
||||
- file-utils.md (not relevant)
|
||||
|
||||
Result: 5 relevant fragments loaded, 28 skipped
|
||||
```
|
||||
|
||||
**Benefit:** Focused context = better results, lower token usage.
|
||||
|
||||
## Context Engineering in Practice
|
||||
|
||||
### Example: Consistent Test Generation
|
||||
|
||||
**Without Knowledge Base (Vanilla Playwright, Random Quality):**
|
||||
```
|
||||
Session 1: User runs *atdd
|
||||
AI: [Guesses patterns from general knowledge]
|
||||
|
||||
Generated:
|
||||
test('api test', async ({ request }) => {
|
||||
const response = await request.get('/api/users');
|
||||
await page.waitForTimeout(2000); // Hard wait
|
||||
const users = await response.json();
|
||||
// Random quality
|
||||
});
|
||||
|
||||
Session 2: User runs *atdd (different day)
|
||||
AI: [Different random patterns]
|
||||
|
||||
Generated:
|
||||
test('api test', async ({ request }) => {
|
||||
const response = await request.get('/api/users');
|
||||
const users = await response.json();
|
||||
// Better but inconsistent
|
||||
});
|
||||
|
||||
Result: Inconsistent quality, random patterns
|
||||
```
|
||||
|
||||
**With Knowledge Base (TEA + Playwright Utils):**
|
||||
```
|
||||
Session 1: User runs *atdd
|
||||
TEA: [Loads test-quality.md, network-first.md, api-request.md from tea-index.csv]
|
||||
|
||||
Generated:
|
||||
import { test } from '@seontechnologies/playwright-utils/api-request/fixtures';
|
||||
|
||||
test('should fetch users', async ({ apiRequest }) => {
|
||||
const { status, body } = await apiRequest({
|
||||
method: 'GET',
|
||||
path: '/api/users'
|
||||
}).validateSchema(UsersSchema); // Chained validation
|
||||
|
||||
expect(status).toBe(200);
|
||||
expect(body).toBeInstanceOf(Array);
|
||||
});
|
||||
|
||||
Session 2: User runs *atdd (different day)
|
||||
TEA: [Loads same fragments from tea-index.csv]
|
||||
|
||||
Generated: Identical pattern, same quality
|
||||
|
||||
Result: Systematic quality, established patterns (ALWAYS uses apiRequest utility when playwright-utils enabled)
|
||||
```
|
||||
|
||||
**Key Difference:**
|
||||
- **Without KB:** Random patterns, inconsistent APIs
|
||||
- **With KB:** Always uses `apiRequest` utility, always validates schemas, always returns `{ status, body }`
|
||||
|
||||
### Example: Test Review Consistency
|
||||
|
||||
**Without Knowledge Base:**
|
||||
```
|
||||
*test-review session 1:
|
||||
"This test looks okay" [50 issues missed]
|
||||
|
||||
*test-review session 2:
|
||||
"This test has some issues" [Different issues flagged]
|
||||
|
||||
Result: Inconsistent feedback
|
||||
```
|
||||
|
||||
**With Knowledge Base:**
|
||||
```
|
||||
*test-review session 1:
|
||||
[Loads all quality fragments]
|
||||
Flags: 12 hard waits, 5 conditionals (based on test-quality.md)
|
||||
|
||||
*test-review session 2:
|
||||
[Loads same fragments]
|
||||
Flags: Same issues with same explanations
|
||||
|
||||
Result: Consistent, reliable feedback
|
||||
```
|
||||
|
||||
## Maintaining the Knowledge Base
|
||||
|
||||
### When to Add a Fragment
|
||||
|
||||
**Good reasons:**
|
||||
- Pattern is used across multiple workflows
|
||||
- Standard is non-obvious (needs documentation)
|
||||
- Team asks "how should we handle X?" repeatedly
|
||||
- New tool integration (e.g., new testing library)
|
||||
|
||||
**Bad reasons:**
|
||||
- One-off pattern (document in test file instead)
|
||||
- Obvious pattern (everyone knows this)
|
||||
- Experimental (not proven yet)
|
||||
|
||||
### Fragment Quality Standards
|
||||
|
||||
**Good fragment:**
|
||||
- Principle stated in one sentence
|
||||
- Rationale explains why clearly
|
||||
- 3+ pattern examples with code
|
||||
- Anti-patterns shown (what not to do)
|
||||
- Self-contained (minimal dependencies)
|
||||
|
||||
**Example size:** 10-30 KB optimal
|
||||
|
||||
### Updating Existing Fragments
|
||||
|
||||
**When to update:**
|
||||
- Pattern evolved (better approach discovered)
|
||||
- Tool updated (new Playwright API)
|
||||
- Team feedback (pattern unclear)
|
||||
- Bug in example code
|
||||
|
||||
**How to update:**
|
||||
1. Edit fragment markdown file
|
||||
2. Update examples
|
||||
3. Test with affected workflows
|
||||
4. Ensure no breaking changes
|
||||
|
||||
**No need to update tea-index.csv** unless description/tags change.
|
||||
|
||||
## Benefits of Knowledge Base System
|
||||
|
||||
### 1. Consistency
|
||||
|
||||
**Before:** Test quality varies by who wrote it
|
||||
**After:** All tests follow same patterns (TEA-generated or reviewed)
|
||||
|
||||
### 2. Onboarding
|
||||
|
||||
**Before:** New team member reads 20 documents, asks 50 questions
|
||||
**After:** New team member runs `*atdd`, sees patterns in generated code, learns by example
|
||||
|
||||
### 3. Quality Gates
|
||||
|
||||
**Before:** "Is this test good?" → subjective opinion
|
||||
**After:** "*test-review" → objective score against knowledge base
|
||||
|
||||
### 4. Pattern Evolution
|
||||
|
||||
**Before:** Update tests manually across 100 files
|
||||
**After:** Update fragment once, all new tests use new pattern
|
||||
|
||||
### 5. Cross-Project Reuse
|
||||
|
||||
**Before:** Reinvent patterns for each project
|
||||
**After:** Same fragments across all BMad projects (consistency at scale)
|
||||
|
||||
## Comparison: With vs Without Knowledge Base
|
||||
|
||||
### Scenario: Testing Async Background Job
|
||||
|
||||
**Without Knowledge Base:**
|
||||
|
||||
Developer 1:
|
||||
```typescript
|
||||
// Uses hard wait
|
||||
await page.click('button');
|
||||
await page.waitForTimeout(10000); // Hope job finishes
|
||||
```
|
||||
|
||||
Developer 2:
|
||||
```typescript
|
||||
// Uses polling
|
||||
await page.click('button');
|
||||
for (let i = 0; i < 10; i++) {
|
||||
const status = await page.locator('.status').textContent();
|
||||
if (status === 'complete') break;
|
||||
await page.waitForTimeout(1000);
|
||||
}
|
||||
```
|
||||
|
||||
Developer 3:
|
||||
```typescript
|
||||
// Uses waitForSelector
|
||||
await page.click('button');
|
||||
await page.waitForSelector('.success', { timeout: 30000 });
|
||||
```
|
||||
|
||||
**Result:** 3 different patterns, all suboptimal.
|
||||
|
||||
**With Knowledge Base (recurse.md fragment):**
|
||||
|
||||
All developers:
|
||||
```typescript
|
||||
import { test } from '@seontechnologies/playwright-utils/fixtures';
|
||||
|
||||
test('job completion', async ({ apiRequest, recurse }) => {
|
||||
// Start async job
|
||||
const { body: job } = await apiRequest({
|
||||
method: 'POST',
|
||||
path: '/api/jobs'
|
||||
});
|
||||
|
||||
// Poll until complete (correct API: command, predicate, options)
|
||||
const result = await recurse(
|
||||
() => apiRequest({ method: 'GET', path: `/api/jobs/${job.id}` }),
|
||||
(response) => response.body.status === 'completed', // response.body from apiRequest
|
||||
{
|
||||
timeout: 30000,
|
||||
interval: 2000,
|
||||
log: 'Waiting for job to complete'
|
||||
}
|
||||
);
|
||||
|
||||
expect(result.body.status).toBe('completed');
|
||||
});
|
||||
```
|
||||
|
||||
**Result:** Consistent pattern using correct playwright-utils API (command, predicate, options).
|
||||
|
||||
## Technical Implementation
|
||||
|
||||
For details on the knowledge base index, see:
|
||||
- [Knowledge Base Index](/docs/reference/tea/knowledge-base.md)
|
||||
- [TEA Configuration](/docs/reference/tea/configuration.md)
|
||||
|
||||
## Related Concepts
|
||||
|
||||
**Core TEA Concepts:**
|
||||
- [Test Quality Standards](/docs/explanation/tea/test-quality-standards.md) - Standards in knowledge base
|
||||
- [Risk-Based Testing](/docs/explanation/tea/risk-based-testing.md) - Risk patterns in knowledge base
|
||||
- [Engagement Models](/docs/explanation/tea/engagement-models.md) - Knowledge base across all models
|
||||
|
||||
**Technical Patterns:**
|
||||
- [Fixture Architecture](/docs/explanation/tea/fixture-architecture.md) - Fixture patterns in knowledge base
|
||||
- [Network-First Patterns](/docs/explanation/tea/network-first-patterns.md) - Network patterns in knowledge base
|
||||
|
||||
**Overview:**
|
||||
- [TEA Overview](/docs/explanation/features/tea-overview.md) - Knowledge base in workflows
|
||||
- [Testing as Engineering](/docs/explanation/philosophy/testing-as-engineering.md) - **Foundation: Context engineering philosophy** (why knowledge base solves AI test problems)
|
||||
|
||||
## Practical Guides
|
||||
|
||||
**All Workflow Guides Use Knowledge Base:**
|
||||
- [How to Run Test Design](/docs/how-to/workflows/run-test-design.md)
|
||||
- [How to Run ATDD](/docs/how-to/workflows/run-atdd.md)
|
||||
- [How to Run Automate](/docs/how-to/workflows/run-automate.md)
|
||||
- [How to Run Test Review](/docs/how-to/workflows/run-test-review.md)
|
||||
|
||||
**Integration:**
|
||||
- [Integrate Playwright Utils](/docs/how-to/customization/integrate-playwright-utils.md) - PW-Utils in knowledge base
|
||||
|
||||
## Reference
|
||||
|
||||
- [Knowledge Base Index](/docs/reference/tea/knowledge-base.md) - Complete fragment index
|
||||
- [TEA Command Reference](/docs/reference/tea/commands.md) - Which workflows load which fragments
|
||||
- [TEA Configuration](/docs/reference/tea/configuration.md) - Config affects fragment loading
|
||||
- [Glossary](/docs/reference/glossary/index.md#test-architect-tea-concepts) - Context engineering, knowledge fragment terms
|
||||
|
||||
---
|
||||
|
||||
Generated with [BMad Method](https://bmad-method.org) - TEA (Test Architect)
|
||||
853
docs/explanation/tea/network-first-patterns.md
Normal file
853
docs/explanation/tea/network-first-patterns.md
Normal file
@@ -0,0 +1,853 @@
|
||||
---
|
||||
title: "Network-First Patterns Explained"
|
||||
description: Understanding how TEA eliminates test flakiness by waiting for actual network responses
|
||||
---
|
||||
|
||||
# Network-First Patterns Explained
|
||||
|
||||
Network-first patterns are TEA's solution to test flakiness. Instead of guessing how long to wait with fixed timeouts, wait for the actual network event that causes UI changes.
|
||||
|
||||
## Overview
|
||||
|
||||
**The Core Principle:**
|
||||
UI changes because APIs respond. Wait for the API response, not an arbitrary timeout.
|
||||
|
||||
**Traditional approach:**
|
||||
```typescript
|
||||
await page.click('button');
|
||||
await page.waitForTimeout(3000); // Hope 3 seconds is enough
|
||||
await expect(page.locator('.success')).toBeVisible();
|
||||
```
|
||||
|
||||
**Network-first approach:**
|
||||
```typescript
|
||||
const responsePromise = page.waitForResponse(
|
||||
resp => resp.url().includes('/api/submit') && resp.ok()
|
||||
);
|
||||
await page.click('button');
|
||||
await responsePromise; // Wait for actual response
|
||||
await expect(page.locator('.success')).toBeVisible();
|
||||
```
|
||||
|
||||
**Result:** Deterministic tests that wait exactly as long as needed.
|
||||
|
||||
## The Problem
|
||||
|
||||
### Hard Waits Create Flakiness
|
||||
|
||||
```typescript
|
||||
// ❌ The flaky test pattern
|
||||
test('should submit form', async ({ page }) => {
|
||||
await page.fill('#name', 'Test User');
|
||||
await page.click('button[type="submit"]');
|
||||
|
||||
await page.waitForTimeout(2000); // Wait 2 seconds
|
||||
|
||||
await expect(page.locator('.success')).toBeVisible();
|
||||
});
|
||||
```
|
||||
|
||||
**Why this fails:**
|
||||
- **Fast network:** Wastes 1.5 seconds waiting
|
||||
- **Slow network:** Not enough time, test fails
|
||||
- **CI environment:** Slower than local, fails randomly
|
||||
- **Under load:** API takes 3 seconds, test fails
|
||||
|
||||
**Result:** "Works on my machine" syndrome, flaky CI.
|
||||
|
||||
### The Timeout Escalation Trap
|
||||
|
||||
```typescript
|
||||
// Developer sees flaky test
|
||||
await page.waitForTimeout(2000); // Failed in CI
|
||||
|
||||
// Increases timeout
|
||||
await page.waitForTimeout(5000); // Still fails sometimes
|
||||
|
||||
// Increases again
|
||||
await page.waitForTimeout(10000); // Now it passes... slowly
|
||||
|
||||
// Problem: Now EVERY test waits 10 seconds
|
||||
// Suite that took 5 minutes now takes 30 minutes
|
||||
```
|
||||
|
||||
**Result:** Slow, still-flaky tests.
|
||||
|
||||
### Race Conditions
|
||||
|
||||
```typescript
|
||||
// ❌ Navigate-then-wait race condition
|
||||
test('should load dashboard data', async ({ page }) => {
|
||||
await page.goto('/dashboard'); // Navigation starts
|
||||
|
||||
// Race condition! API might not have responded yet
|
||||
await expect(page.locator('.data-table')).toBeVisible();
|
||||
});
|
||||
```
|
||||
|
||||
**What happens:**
|
||||
1. `goto()` starts navigation
|
||||
2. Page loads HTML
|
||||
3. JavaScript requests `/api/dashboard`
|
||||
4. Test checks for `.data-table` BEFORE API responds
|
||||
5. Test fails intermittently
|
||||
|
||||
**Result:** "Sometimes it works, sometimes it doesn't."
|
||||
|
||||
## The Solution: Intercept-Before-Navigate
|
||||
|
||||
### Wait for Response Before Asserting
|
||||
|
||||
```typescript
|
||||
// ✅ Good: Network-first pattern
|
||||
test('should load dashboard data', async ({ page }) => {
|
||||
// Set up promise BEFORE navigation
|
||||
const dashboardPromise = page.waitForResponse(
|
||||
resp => resp.url().includes('/api/dashboard') && resp.ok()
|
||||
);
|
||||
|
||||
// Navigate
|
||||
await page.goto('/dashboard');
|
||||
|
||||
// Wait for API response
|
||||
const response = await dashboardPromise;
|
||||
const data = await response.json();
|
||||
|
||||
// Now assert UI
|
||||
await expect(page.locator('.data-table')).toBeVisible();
|
||||
await expect(page.locator('.data-table tr')).toHaveCount(data.items.length);
|
||||
});
|
||||
```
|
||||
|
||||
**Why this works:**
|
||||
- Wait set up BEFORE navigation (no race)
|
||||
- Wait for actual API response (deterministic)
|
||||
- No fixed timeout (fast when API is fast)
|
||||
- Validates API response (catch backend errors)
|
||||
|
||||
**With Playwright Utils (Even Cleaner):**
|
||||
```typescript
|
||||
import { test } from '@seontechnologies/playwright-utils/fixtures';
|
||||
import { expect } from '@playwright/test';
|
||||
|
||||
test('should load dashboard data', async ({ page, interceptNetworkCall }) => {
|
||||
// Set up interception BEFORE navigation
|
||||
const dashboardCall = interceptNetworkCall({
|
||||
method: 'GET',
|
||||
url: '**/api/dashboard'
|
||||
});
|
||||
|
||||
// Navigate
|
||||
await page.goto('/dashboard');
|
||||
|
||||
// Wait for API response (automatic JSON parsing)
|
||||
const { status, responseJson: data } = await dashboardCall;
|
||||
|
||||
// Validate API response
|
||||
expect(status).toBe(200);
|
||||
expect(data.items).toBeDefined();
|
||||
|
||||
// Assert UI matches API data
|
||||
await expect(page.locator('.data-table')).toBeVisible();
|
||||
await expect(page.locator('.data-table tr')).toHaveCount(data.items.length);
|
||||
});
|
||||
```
|
||||
|
||||
**Playwright Utils Benefits:**
|
||||
- Automatic JSON parsing (no `await response.json()`)
|
||||
- Returns `{ status, responseJson, requestJson }` structure
|
||||
- Cleaner API (no need to check `resp.ok()`)
|
||||
- Same intercept-before-navigate pattern
|
||||
|
||||
### Intercept-Before-Navigate Pattern
|
||||
|
||||
**Key insight:** Set up wait BEFORE triggering the action.
|
||||
|
||||
```typescript
|
||||
// ✅ Pattern: Intercept → Action → Await
|
||||
|
||||
// 1. Intercept (set up wait)
|
||||
const promise = page.waitForResponse(matcher);
|
||||
|
||||
// 2. Action (trigger request)
|
||||
await page.click('button');
|
||||
|
||||
// 3. Await (wait for actual response)
|
||||
await promise;
|
||||
```
|
||||
|
||||
**Why this order:**
|
||||
- `waitForResponse()` starts listening immediately
|
||||
- Then trigger the action that makes the request
|
||||
- Then wait for the promise to resolve
|
||||
- No race condition possible
|
||||
|
||||
#### Intercept-Before-Navigate Flow
|
||||
|
||||
```mermaid
|
||||
%%{init: {'theme':'base', 'themeVariables': { 'fontSize':'14px'}}}%%
|
||||
sequenceDiagram
|
||||
participant Test
|
||||
participant Playwright
|
||||
participant Browser
|
||||
participant API
|
||||
|
||||
rect rgb(200, 230, 201)
|
||||
Note over Test,Playwright: ✅ CORRECT: Intercept First
|
||||
Test->>Playwright: 1. waitForResponse(matcher)
|
||||
Note over Playwright: Starts listening for response
|
||||
Test->>Browser: 2. click('button')
|
||||
Browser->>API: 3. POST /api/submit
|
||||
API-->>Browser: 4. 200 OK {success: true}
|
||||
Browser-->>Playwright: 5. Response captured
|
||||
Test->>Playwright: 6. await promise
|
||||
Playwright-->>Test: 7. Returns response
|
||||
Note over Test: No race condition!
|
||||
end
|
||||
|
||||
rect rgb(255, 205, 210)
|
||||
Note over Test,API: ❌ WRONG: Action First
|
||||
Test->>Browser: 1. click('button')
|
||||
Browser->>API: 2. POST /api/submit
|
||||
API-->>Browser: 3. 200 OK (already happened!)
|
||||
Test->>Playwright: 4. waitForResponse(matcher)
|
||||
Note over Test,Playwright: Too late - response already occurred
|
||||
Note over Test: Race condition! Test hangs or fails
|
||||
end
|
||||
```
|
||||
|
||||
**Correct Order (Green):**
|
||||
1. Set up listener (`waitForResponse`)
|
||||
2. Trigger action (`click`)
|
||||
3. Wait for response (`await promise`)
|
||||
|
||||
**Wrong Order (Red):**
|
||||
1. Trigger action first
|
||||
2. Set up listener too late
|
||||
3. Response already happened - missed!
|
||||
|
||||
## How It Works in TEA
|
||||
|
||||
### TEA Generates Network-First Tests
|
||||
|
||||
**Vanilla Playwright:**
|
||||
```typescript
|
||||
// When you run *atdd or *automate, TEA generates:
|
||||
|
||||
test('should create user', async ({ page }) => {
|
||||
// TEA automatically includes network wait
|
||||
const createUserPromise = page.waitForResponse(
|
||||
resp => resp.url().includes('/api/users') &&
|
||||
resp.request().method() === 'POST' &&
|
||||
resp.ok()
|
||||
);
|
||||
|
||||
await page.fill('#name', 'Test User');
|
||||
await page.click('button[type="submit"]');
|
||||
|
||||
const response = await createUserPromise;
|
||||
const user = await response.json();
|
||||
|
||||
// Validate both API and UI
|
||||
expect(user.id).toBeDefined();
|
||||
await expect(page.locator('.success')).toContainText(user.name);
|
||||
});
|
||||
```
|
||||
|
||||
**With Playwright Utils (if `tea_use_playwright_utils: true`):**
|
||||
```typescript
|
||||
import { test } from '@seontechnologies/playwright-utils/fixtures';
|
||||
import { expect } from '@playwright/test';
|
||||
|
||||
test('should create user', async ({ page, interceptNetworkCall }) => {
|
||||
// TEA uses interceptNetworkCall for cleaner interception
|
||||
const createUserCall = interceptNetworkCall({
|
||||
method: 'POST',
|
||||
url: '**/api/users'
|
||||
});
|
||||
|
||||
await page.getByLabel('Name').fill('Test User');
|
||||
await page.getByRole('button', { name: 'Submit' }).click();
|
||||
|
||||
// Wait for response (automatic JSON parsing)
|
||||
const { status, responseJson: user } = await createUserCall;
|
||||
|
||||
// Validate both API and UI
|
||||
expect(status).toBe(201);
|
||||
expect(user.id).toBeDefined();
|
||||
await expect(page.locator('.success')).toContainText(user.name);
|
||||
});
|
||||
```
|
||||
|
||||
**Playwright Utils Benefits:**
|
||||
- Automatic JSON parsing (`responseJson` ready to use)
|
||||
- No manual `await response.json()`
|
||||
- Returns `{ status, responseJson }` structure
|
||||
- Cleaner, more readable code
|
||||
|
||||
### TEA Reviews for Hard Waits
|
||||
|
||||
When you run `*test-review`:
|
||||
|
||||
```markdown
|
||||
## Critical Issue: Hard Wait Detected
|
||||
|
||||
**File:** tests/e2e/submit.spec.ts:45
|
||||
**Issue:** Using `page.waitForTimeout(3000)`
|
||||
**Severity:** Critical (causes flakiness)
|
||||
|
||||
**Current Code:**
|
||||
```typescript
|
||||
await page.click('button');
|
||||
await page.waitForTimeout(3000); // ❌
|
||||
```
|
||||
|
||||
**Fix:**
|
||||
```typescript
|
||||
const responsePromise = page.waitForResponse(
|
||||
resp => resp.url().includes('/api/submit') && resp.ok()
|
||||
);
|
||||
await page.click('button');
|
||||
await responsePromise; // ✅
|
||||
```
|
||||
|
||||
**Why:** Hard waits are non-deterministic. Use network-first patterns.
|
||||
```
|
||||
|
||||
## Pattern Variations
|
||||
|
||||
### Basic Response Wait
|
||||
|
||||
**Vanilla Playwright:**
|
||||
```typescript
|
||||
// Wait for any successful response
|
||||
const promise = page.waitForResponse(resp => resp.ok());
|
||||
await page.click('button');
|
||||
await promise;
|
||||
```
|
||||
|
||||
**With Playwright Utils:**
|
||||
```typescript
|
||||
import { test } from '@seontechnologies/playwright-utils/fixtures';
|
||||
|
||||
test('basic wait', async ({ page, interceptNetworkCall }) => {
|
||||
const responseCall = interceptNetworkCall({ url: '**' }); // Match any
|
||||
await page.click('button');
|
||||
const { status } = await responseCall;
|
||||
expect(status).toBe(200);
|
||||
});
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Specific URL Match
|
||||
|
||||
**Vanilla Playwright:**
|
||||
```typescript
|
||||
// Wait for specific endpoint
|
||||
const promise = page.waitForResponse(
|
||||
resp => resp.url().includes('/api/users/123')
|
||||
);
|
||||
await page.goto('/user/123');
|
||||
await promise;
|
||||
```
|
||||
|
||||
**With Playwright Utils:**
|
||||
```typescript
|
||||
test('specific URL', async ({ page, interceptNetworkCall }) => {
|
||||
const userCall = interceptNetworkCall({ url: '**/api/users/123' });
|
||||
await page.goto('/user/123');
|
||||
const { status, responseJson } = await userCall;
|
||||
expect(status).toBe(200);
|
||||
});
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Method + Status Match
|
||||
|
||||
**Vanilla Playwright:**
|
||||
```typescript
|
||||
// Wait for POST that returns 201
|
||||
const promise = page.waitForResponse(
|
||||
resp =>
|
||||
resp.url().includes('/api/users') &&
|
||||
resp.request().method() === 'POST' &&
|
||||
resp.status() === 201
|
||||
);
|
||||
await page.click('button[type="submit"]');
|
||||
await promise;
|
||||
```
|
||||
|
||||
**With Playwright Utils:**
|
||||
```typescript
|
||||
test('method and status', async ({ page, interceptNetworkCall }) => {
|
||||
const createCall = interceptNetworkCall({
|
||||
method: 'POST',
|
||||
url: '**/api/users'
|
||||
});
|
||||
await page.click('button[type="submit"]');
|
||||
const { status, responseJson } = await createCall;
|
||||
expect(status).toBe(201); // Explicit status check
|
||||
});
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Multiple Responses
|
||||
|
||||
**Vanilla Playwright:**
|
||||
```typescript
|
||||
// Wait for multiple API calls
|
||||
const [usersResp, postsResp] = await Promise.all([
|
||||
page.waitForResponse(resp => resp.url().includes('/api/users')),
|
||||
page.waitForResponse(resp => resp.url().includes('/api/posts')),
|
||||
page.goto('/dashboard') // Triggers both requests
|
||||
]);
|
||||
|
||||
const users = await usersResp.json();
|
||||
const posts = await postsResp.json();
|
||||
```
|
||||
|
||||
**With Playwright Utils:**
|
||||
```typescript
|
||||
test('multiple responses', async ({ page, interceptNetworkCall }) => {
|
||||
const usersCall = interceptNetworkCall({ url: '**/api/users' });
|
||||
const postsCall = interceptNetworkCall({ url: '**/api/posts' });
|
||||
|
||||
await page.goto('/dashboard'); // Triggers both
|
||||
|
||||
const [{ responseJson: users }, { responseJson: posts }] = await Promise.all([
|
||||
usersCall,
|
||||
postsCall
|
||||
]);
|
||||
|
||||
expect(users).toBeInstanceOf(Array);
|
||||
expect(posts).toBeInstanceOf(Array);
|
||||
});
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Validate Response Data
|
||||
|
||||
**Vanilla Playwright:**
|
||||
```typescript
|
||||
// Verify API response before asserting UI
|
||||
const promise = page.waitForResponse(
|
||||
resp => resp.url().includes('/api/checkout') && resp.ok()
|
||||
);
|
||||
|
||||
await page.click('button:has-text("Complete Order")');
|
||||
|
||||
const response = await promise;
|
||||
const order = await response.json();
|
||||
|
||||
// Response validation
|
||||
expect(order.status).toBe('confirmed');
|
||||
expect(order.total).toBeGreaterThan(0);
|
||||
|
||||
// UI validation
|
||||
await expect(page.locator('.order-confirmation')).toContainText(order.id);
|
||||
```
|
||||
|
||||
**With Playwright Utils:**
|
||||
```typescript
|
||||
test('validate response data', async ({ page, interceptNetworkCall }) => {
|
||||
const checkoutCall = interceptNetworkCall({
|
||||
method: 'POST',
|
||||
url: '**/api/checkout'
|
||||
});
|
||||
|
||||
await page.click('button:has-text("Complete Order")');
|
||||
|
||||
const { status, responseJson: order } = await checkoutCall;
|
||||
|
||||
// Response validation (automatic JSON parsing)
|
||||
expect(status).toBe(200);
|
||||
expect(order.status).toBe('confirmed');
|
||||
expect(order.total).toBeGreaterThan(0);
|
||||
|
||||
// UI validation
|
||||
await expect(page.locator('.order-confirmation')).toContainText(order.id);
|
||||
});
|
||||
```
|
||||
|
||||
## Advanced Patterns
|
||||
|
||||
### HAR Recording for Offline Testing
|
||||
|
||||
**Vanilla Playwright (Manual HAR Handling):**
|
||||
|
||||
```typescript
|
||||
// First run: Record mode (saves HAR file)
|
||||
test('offline testing - RECORD', async ({ page, context }) => {
|
||||
// Record mode: Save network traffic to HAR
|
||||
await context.routeFromHAR('./hars/dashboard.har', {
|
||||
url: '**/api/**',
|
||||
update: true // Update HAR file
|
||||
});
|
||||
|
||||
await page.goto('/dashboard');
|
||||
// All network traffic saved to dashboard.har
|
||||
});
|
||||
|
||||
// Subsequent runs: Playback mode (uses saved HAR)
|
||||
test('offline testing - PLAYBACK', async ({ page, context }) => {
|
||||
// Playback mode: Use saved network traffic
|
||||
await context.routeFromHAR('./hars/dashboard.har', {
|
||||
url: '**/api/**',
|
||||
update: false // Use existing HAR, no network calls
|
||||
});
|
||||
|
||||
await page.goto('/dashboard');
|
||||
// Uses recorded responses, no backend needed
|
||||
});
|
||||
```
|
||||
|
||||
**With Playwright Utils (Automatic HAR Management):**
|
||||
```typescript
|
||||
import { test } from '@seontechnologies/playwright-utils/network-recorder/fixtures';
|
||||
|
||||
// Record mode: Set environment variable
|
||||
process.env.PW_NET_MODE = 'record';
|
||||
|
||||
test('should work offline', async ({ page, context, networkRecorder }) => {
|
||||
await networkRecorder.setup(context); // Handles HAR automatically
|
||||
|
||||
await page.goto('/dashboard');
|
||||
await page.click('#add-item');
|
||||
// All network traffic recorded, CRUD operations detected
|
||||
});
|
||||
```
|
||||
|
||||
**Switch to playback:**
|
||||
```bash
|
||||
# Playback mode (offline)
|
||||
PW_NET_MODE=playback npx playwright test
|
||||
# Uses HAR file, no backend needed!
|
||||
```
|
||||
|
||||
**Playwright Utils Benefits:**
|
||||
- Automatic HAR file management (naming, paths)
|
||||
- CRUD operation detection (stateful mocking)
|
||||
- Environment variable control (easy switching)
|
||||
- Works for complex interactions (create, update, delete)
|
||||
- No manual route configuration
|
||||
|
||||
### Network Request Interception
|
||||
|
||||
**Vanilla Playwright:**
|
||||
```typescript
|
||||
test('should handle API error', async ({ page }) => {
|
||||
// Manual route setup
|
||||
await page.route('**/api/users', (route) => {
|
||||
route.fulfill({
|
||||
status: 500,
|
||||
body: JSON.stringify({ error: 'Internal server error' })
|
||||
});
|
||||
});
|
||||
|
||||
await page.goto('/users');
|
||||
|
||||
const response = await page.waitForResponse('**/api/users');
|
||||
const error = await response.json();
|
||||
|
||||
expect(error.error).toContain('Internal server');
|
||||
await expect(page.locator('.error-message')).toContainText('Server error');
|
||||
});
|
||||
```
|
||||
|
||||
**With Playwright Utils:**
|
||||
```typescript
|
||||
import { test } from '@seontechnologies/playwright-utils/fixtures';
|
||||
|
||||
test('should handle API error', async ({ page, interceptNetworkCall }) => {
|
||||
// Stub API to return error (set up BEFORE navigation)
|
||||
const usersCall = interceptNetworkCall({
|
||||
method: 'GET',
|
||||
url: '**/api/users',
|
||||
fulfillResponse: {
|
||||
status: 500,
|
||||
body: { error: 'Internal server error' }
|
||||
}
|
||||
});
|
||||
|
||||
await page.goto('/users');
|
||||
|
||||
// Wait for mocked response and access parsed data
|
||||
const { status, responseJson } = await usersCall;
|
||||
|
||||
expect(status).toBe(500);
|
||||
expect(responseJson.error).toContain('Internal server');
|
||||
await expect(page.locator('.error-message')).toContainText('Server error');
|
||||
});
|
||||
```
|
||||
|
||||
**Playwright Utils Benefits:**
|
||||
- Automatic JSON parsing (`responseJson` ready to use)
|
||||
- Returns promise with `{ status, responseJson, requestJson }`
|
||||
- No need to pass `page` (auto-injected by fixture)
|
||||
- Glob pattern matching (simpler than regex)
|
||||
- Single declarative call (setup + wait in one)
|
||||
|
||||
## Comparison: Traditional vs Network-First
|
||||
|
||||
### Loading Dashboard Data
|
||||
|
||||
**Traditional (Flaky):**
|
||||
```typescript
|
||||
test('dashboard loads data', async ({ page }) => {
|
||||
await page.goto('/dashboard');
|
||||
await page.waitForTimeout(2000); // ❌ Magic number
|
||||
await expect(page.locator('table tr')).toHaveCount(5);
|
||||
});
|
||||
```
|
||||
|
||||
**Failure modes:**
|
||||
- API takes 2.5s → test fails
|
||||
- API returns 3 items not 5 → hard to debug (which issue?)
|
||||
- CI slower than local → fails in CI only
|
||||
|
||||
**Network-First (Deterministic):**
|
||||
```typescript
|
||||
test('dashboard loads data', async ({ page }) => {
|
||||
const apiPromise = page.waitForResponse(
|
||||
resp => resp.url().includes('/api/dashboard') && resp.ok()
|
||||
);
|
||||
|
||||
await page.goto('/dashboard');
|
||||
|
||||
const response = await apiPromise;
|
||||
const { items } = await response.json();
|
||||
|
||||
// Validate API response
|
||||
expect(items).toHaveLength(5);
|
||||
|
||||
// Validate UI matches API
|
||||
await expect(page.locator('table tr')).toHaveCount(items.length);
|
||||
});
|
||||
```
|
||||
|
||||
**Benefits:**
|
||||
- Waits exactly as long as needed (100ms or 5s, doesn't matter)
|
||||
- Validates API response (catch backend errors)
|
||||
- Validates UI matches API (catch frontend bugs)
|
||||
- Works in any environment (local, CI, staging)
|
||||
|
||||
**With Playwright Utils (Even Better):**
|
||||
```typescript
|
||||
import { test } from '@seontechnologies/playwright-utils/fixtures';
|
||||
|
||||
test('dashboard loads data', async ({ page, interceptNetworkCall }) => {
|
||||
const dashboardCall = interceptNetworkCall({
|
||||
method: 'GET',
|
||||
url: '**/api/dashboard'
|
||||
});
|
||||
|
||||
await page.goto('/dashboard');
|
||||
|
||||
const { status, responseJson: { items } } = await dashboardCall;
|
||||
|
||||
// Validate API response (automatic JSON parsing)
|
||||
expect(status).toBe(200);
|
||||
expect(items).toHaveLength(5);
|
||||
|
||||
// Validate UI matches API
|
||||
await expect(page.locator('table tr')).toHaveCount(items.length);
|
||||
});
|
||||
```
|
||||
|
||||
**Additional Benefits:**
|
||||
- No manual `await response.json()` (automatic parsing)
|
||||
- Cleaner destructuring of nested data
|
||||
- Consistent API across all network calls
|
||||
|
||||
---
|
||||
|
||||
### Form Submission
|
||||
|
||||
**Traditional (Flaky):**
|
||||
```typescript
|
||||
test('form submission', async ({ page }) => {
|
||||
await page.fill('#email', 'test@example.com');
|
||||
await page.click('button[type="submit"]');
|
||||
await page.waitForTimeout(3000); // ❌ Hope it's enough
|
||||
await expect(page.locator('.success')).toBeVisible();
|
||||
});
|
||||
```
|
||||
|
||||
**Network-First (Deterministic):**
|
||||
```typescript
|
||||
test('form submission', async ({ page }) => {
|
||||
const submitPromise = page.waitForResponse(
|
||||
resp => resp.url().includes('/api/submit') &&
|
||||
resp.request().method() === 'POST' &&
|
||||
resp.ok()
|
||||
);
|
||||
|
||||
await page.fill('#email', 'test@example.com');
|
||||
await page.click('button[type="submit"]');
|
||||
|
||||
const response = await submitPromise;
|
||||
const result = await response.json();
|
||||
|
||||
expect(result.success).toBe(true);
|
||||
await expect(page.locator('.success')).toBeVisible();
|
||||
});
|
||||
```
|
||||
|
||||
**With Playwright Utils:**
|
||||
```typescript
|
||||
import { test } from '@seontechnologies/playwright-utils/fixtures';
|
||||
|
||||
test('form submission', async ({ page, interceptNetworkCall }) => {
|
||||
const submitCall = interceptNetworkCall({
|
||||
method: 'POST',
|
||||
url: '**/api/submit'
|
||||
});
|
||||
|
||||
await page.getByLabel('Email').fill('test@example.com');
|
||||
await page.getByRole('button', { name: 'Submit' }).click();
|
||||
|
||||
const { status, responseJson: result } = await submitCall;
|
||||
|
||||
// Automatic JSON parsing, no manual await
|
||||
expect(status).toBe(200);
|
||||
expect(result.success).toBe(true);
|
||||
await expect(page.locator('.success')).toBeVisible();
|
||||
});
|
||||
```
|
||||
|
||||
**Progression:**
|
||||
- Traditional: Hard waits (flaky)
|
||||
- Network-First (Vanilla): waitForResponse (deterministic)
|
||||
- Network-First (PW-Utils): interceptNetworkCall (deterministic + cleaner API)
|
||||
|
||||
---
|
||||
|
||||
## Common Misconceptions
|
||||
|
||||
### "I Already Use waitForSelector"
|
||||
|
||||
```typescript
|
||||
// This is still a hard wait in disguise
|
||||
await page.click('button');
|
||||
await page.waitForSelector('.success', { timeout: 5000 });
|
||||
```
|
||||
|
||||
**Problem:** Waiting for DOM, not for the API that caused DOM change.
|
||||
|
||||
**Better:**
|
||||
```typescript
|
||||
await page.waitForResponse(matcher); // Wait for root cause
|
||||
await page.waitForSelector('.success'); // Then validate UI
|
||||
```
|
||||
|
||||
### "My Tests Are Fast, Why Add Complexity?"
|
||||
|
||||
**Short-term:** Tests are fast locally
|
||||
|
||||
**Long-term problems:**
|
||||
- Different environments (CI slower)
|
||||
- Under load (API slower)
|
||||
- Network variability (random)
|
||||
- Scaling test suite (100 → 1000 tests)
|
||||
|
||||
**Network-first prevents these issues before they appear.**
|
||||
|
||||
### "Too Much Boilerplate"
|
||||
|
||||
**Problem:** `waitForResponse` is verbose, repeated in every test.
|
||||
|
||||
**Solution:** Use Playwright Utils `interceptNetworkCall` - built-in fixture that reduces boilerplate.
|
||||
|
||||
**Vanilla Playwright (Repetitive):**
|
||||
```typescript
|
||||
test('test 1', async ({ page }) => {
|
||||
const promise = page.waitForResponse(
|
||||
resp => resp.url().includes('/api/submit') && resp.ok()
|
||||
);
|
||||
await page.click('button');
|
||||
await promise;
|
||||
});
|
||||
|
||||
test('test 2', async ({ page }) => {
|
||||
const promise = page.waitForResponse(
|
||||
resp => resp.url().includes('/api/load') && resp.ok()
|
||||
);
|
||||
await page.click('button');
|
||||
await promise;
|
||||
});
|
||||
// Repeated pattern in every test
|
||||
```
|
||||
|
||||
**With Playwright Utils (Cleaner):**
|
||||
```typescript
|
||||
import { test } from '@seontechnologies/playwright-utils/fixtures';
|
||||
|
||||
test('test 1', async ({ page, interceptNetworkCall }) => {
|
||||
const submitCall = interceptNetworkCall({ url: '**/api/submit' });
|
||||
await page.click('button');
|
||||
const { status, responseJson } = await submitCall;
|
||||
expect(status).toBe(200);
|
||||
});
|
||||
|
||||
test('test 2', async ({ page, interceptNetworkCall }) => {
|
||||
const loadCall = interceptNetworkCall({ url: '**/api/load' });
|
||||
await page.click('button');
|
||||
const { responseJson } = await loadCall;
|
||||
// Automatic JSON parsing, cleaner API
|
||||
});
|
||||
```
|
||||
|
||||
**Benefits:**
|
||||
- Less boilerplate (fixture handles complexity)
|
||||
- Automatic JSON parsing
|
||||
- Glob pattern matching (`**/api/**`)
|
||||
- Consistent API across all tests
|
||||
|
||||
See [Integrate Playwright Utils](/docs/how-to/customization/integrate-playwright-utils.md#intercept-network-call) for setup.
|
||||
|
||||
## Technical Implementation
|
||||
|
||||
For detailed network-first patterns, see the knowledge base:
|
||||
- [Knowledge Base Index - Network & Reliability](/docs/reference/tea/knowledge-base.md)
|
||||
- [Complete Knowledge Base Index](/docs/reference/tea/knowledge-base.md)
|
||||
|
||||
## Related Concepts
|
||||
|
||||
**Core TEA Concepts:**
|
||||
- [Test Quality Standards](/docs/explanation/tea/test-quality-standards.md) - Determinism requires network-first
|
||||
- [Risk-Based Testing](/docs/explanation/tea/risk-based-testing.md) - High-risk features need reliable tests
|
||||
|
||||
**Technical Patterns:**
|
||||
- [Fixture Architecture](/docs/explanation/tea/fixture-architecture.md) - Network utilities as fixtures
|
||||
- [Knowledge Base System](/docs/explanation/tea/knowledge-base-system.md) - Network patterns in knowledge base
|
||||
|
||||
**Overview:**
|
||||
- [TEA Overview](/docs/explanation/features/tea-overview.md) - Network-first in workflows
|
||||
- [Testing as Engineering](/docs/explanation/philosophy/testing-as-engineering.md) - Why flakiness matters
|
||||
|
||||
## Practical Guides
|
||||
|
||||
**Workflow Guides:**
|
||||
- [How to Run Test Review](/docs/how-to/workflows/run-test-review.md) - Review for hard waits
|
||||
- [How to Run ATDD](/docs/how-to/workflows/run-atdd.md) - Generate network-first tests
|
||||
- [How to Run Automate](/docs/how-to/workflows/run-automate.md) - Expand with network patterns
|
||||
|
||||
**Use-Case Guides:**
|
||||
- [Using TEA with Existing Tests](/docs/how-to/brownfield/use-tea-with-existing-tests.md) - Fix flaky legacy tests
|
||||
|
||||
**Customization:**
|
||||
- [Integrate Playwright Utils](/docs/how-to/customization/integrate-playwright-utils.md) - Network utilities (recorder, interceptor, error monitor)
|
||||
|
||||
## Reference
|
||||
|
||||
- [TEA Command Reference](/docs/reference/tea/commands.md) - All workflows use network-first
|
||||
- [Knowledge Base Index](/docs/reference/tea/knowledge-base.md) - Network-first fragment
|
||||
- [Glossary](/docs/reference/glossary/index.md#test-architect-tea-concepts) - Network-first pattern term
|
||||
|
||||
---
|
||||
|
||||
Generated with [BMad Method](https://bmad-method.org) - TEA (Test Architect)
|
||||
586
docs/explanation/tea/risk-based-testing.md
Normal file
586
docs/explanation/tea/risk-based-testing.md
Normal file
@@ -0,0 +1,586 @@
|
||||
---
|
||||
title: "Risk-Based Testing Explained"
|
||||
description: Understanding how TEA uses probability × impact scoring to prioritize testing effort
|
||||
---
|
||||
|
||||
# Risk-Based Testing Explained
|
||||
|
||||
Risk-based testing is TEA's core principle: testing depth scales with business impact. Instead of testing everything equally, focus effort where failures hurt most.
|
||||
|
||||
## Overview
|
||||
|
||||
Traditional testing approaches treat all features equally:
|
||||
- Every feature gets same test coverage
|
||||
- Same level of scrutiny regardless of impact
|
||||
- No systematic prioritization
|
||||
- Testing becomes checkbox exercise
|
||||
|
||||
**Risk-based testing asks:**
|
||||
- What's the probability this will fail?
|
||||
- What's the impact if it does fail?
|
||||
- How much testing is appropriate for this risk level?
|
||||
|
||||
**Result:** Testing effort matches business criticality.
|
||||
|
||||
## The Problem
|
||||
|
||||
### Equal Testing for Unequal Risk
|
||||
|
||||
```markdown
|
||||
Feature A: User login (critical path, millions of users)
|
||||
Feature B: Export to PDF (nice-to-have, rarely used)
|
||||
|
||||
Traditional approach:
|
||||
- Both get 10 tests
|
||||
- Both get same review scrutiny
|
||||
- Both take same development time
|
||||
|
||||
Problem: Wasting effort on low-impact features while under-testing critical paths.
|
||||
```
|
||||
|
||||
### No Objective Prioritization
|
||||
|
||||
```markdown
|
||||
PM: "We need more tests for checkout"
|
||||
QA: "How many tests?"
|
||||
PM: "I don't know... a lot?"
|
||||
QA: "How do we know when we have enough?"
|
||||
PM: "When it feels safe?"
|
||||
|
||||
Problem: Subjective decisions, no data, political debates.
|
||||
```
|
||||
|
||||
## The Solution: Probability × Impact Scoring
|
||||
|
||||
### Risk Score = Probability × Impact
|
||||
|
||||
**Probability** (How likely to fail?)
|
||||
- **1 (Low):** Stable, well-tested, simple logic
|
||||
- **2 (Medium):** Moderate complexity, some unknowns
|
||||
- **3 (High):** Complex, untested, many edge cases
|
||||
|
||||
**Impact** (How bad if it fails?)
|
||||
- **1 (Low):** Minor inconvenience, few users affected
|
||||
- **2 (Medium):** Degraded experience, workarounds exist
|
||||
- **3 (High):** Critical path broken, business impact
|
||||
|
||||
**Score Range:** 1-9
|
||||
|
||||
#### Risk Scoring Matrix
|
||||
|
||||
```mermaid
|
||||
%%{init: {'theme':'base', 'themeVariables': { 'fontSize':'14px'}}}%%
|
||||
graph TD
|
||||
subgraph Matrix[" "]
|
||||
direction TB
|
||||
subgraph Impact3["Impact: HIGH (3)"]
|
||||
P1I3["Score: 3<br/>Low Risk"]
|
||||
P2I3["Score: 6<br/>HIGH RISK<br/>Mitigation Required"]
|
||||
P3I3["Score: 9<br/>CRITICAL<br/>Blocks Release"]
|
||||
end
|
||||
subgraph Impact2["Impact: MEDIUM (2)"]
|
||||
P1I2["Score: 2<br/>Low Risk"]
|
||||
P2I2["Score: 4<br/>Medium Risk"]
|
||||
P3I2["Score: 6<br/>HIGH RISK<br/>Mitigation Required"]
|
||||
end
|
||||
subgraph Impact1["Impact: LOW (1)"]
|
||||
P1I1["Score: 1<br/>Low Risk"]
|
||||
P2I1["Score: 2<br/>Low Risk"]
|
||||
P3I1["Score: 3<br/>Low Risk"]
|
||||
end
|
||||
end
|
||||
|
||||
Prob1["Probability: LOW (1)"] -.-> P1I1
|
||||
Prob1 -.-> P1I2
|
||||
Prob1 -.-> P1I3
|
||||
|
||||
Prob2["Probability: MEDIUM (2)"] -.-> P2I1
|
||||
Prob2 -.-> P2I2
|
||||
Prob2 -.-> P2I3
|
||||
|
||||
Prob3["Probability: HIGH (3)"] -.-> P3I1
|
||||
Prob3 -.-> P3I2
|
||||
Prob3 -.-> P3I3
|
||||
|
||||
style P3I3 fill:#f44336,stroke:#b71c1c,stroke-width:3px,color:#fff
|
||||
style P2I3 fill:#ff9800,stroke:#e65100,stroke-width:2px,color:#000
|
||||
style P3I2 fill:#ff9800,stroke:#e65100,stroke-width:2px,color:#000
|
||||
style P2I2 fill:#fff9c4,stroke:#f57f17,stroke-width:1px,color:#000
|
||||
style P1I1 fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px,color:#000
|
||||
style P2I1 fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px,color:#000
|
||||
style P3I1 fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px,color:#000
|
||||
style P1I2 fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px,color:#000
|
||||
style P1I3 fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px,color:#000
|
||||
```
|
||||
|
||||
**Legend:**
|
||||
- 🔴 Red (Score 9): CRITICAL - Blocks release
|
||||
- 🟠 Orange (Score 6-8): HIGH RISK - Mitigation required
|
||||
- 🟡 Yellow (Score 4-5): MEDIUM - Mitigation recommended
|
||||
- 🟢 Green (Score 1-3): LOW - Optional mitigation
|
||||
|
||||
### Scoring Examples
|
||||
|
||||
**Score 9 (Critical):**
|
||||
```
|
||||
Feature: Payment processing
|
||||
Probability: 3 (complex third-party integration)
|
||||
Impact: 3 (broken payments = lost revenue)
|
||||
Score: 3 × 3 = 9
|
||||
|
||||
Action: Extensive testing required
|
||||
- E2E tests for all payment flows
|
||||
- API tests for all payment scenarios
|
||||
- Error handling for all failure modes
|
||||
- Security testing for payment data
|
||||
- Load testing for high traffic
|
||||
- Monitoring and alerts
|
||||
```
|
||||
|
||||
**Score 1 (Low):**
|
||||
```
|
||||
Feature: Change profile theme color
|
||||
Probability: 1 (simple UI toggle)
|
||||
Impact: 1 (cosmetic only)
|
||||
Score: 1 × 1 = 1
|
||||
|
||||
Action: Minimal testing
|
||||
- One E2E smoke test
|
||||
- Skip edge cases
|
||||
- No API tests needed
|
||||
```
|
||||
|
||||
**Score 6 (Medium-High):**
|
||||
```
|
||||
Feature: User profile editing
|
||||
Probability: 2 (moderate complexity)
|
||||
Impact: 3 (users can't update info)
|
||||
Score: 2 × 3 = 6
|
||||
|
||||
Action: Focused testing
|
||||
- E2E test for happy path
|
||||
- API tests for CRUD operations
|
||||
- Validation testing
|
||||
- Skip low-value edge cases
|
||||
```
|
||||
|
||||
## How It Works in TEA
|
||||
|
||||
### 1. Risk Categories
|
||||
|
||||
TEA assesses risk across 6 categories:
|
||||
|
||||
**TECH** - Technical debt, architecture fragility
|
||||
```
|
||||
Example: Migrating from REST to GraphQL
|
||||
Probability: 3 (major architectural change)
|
||||
Impact: 3 (affects all API consumers)
|
||||
Score: 9 - Extensive integration testing required
|
||||
```
|
||||
|
||||
**SEC** - Security vulnerabilities
|
||||
```
|
||||
Example: Adding OAuth integration
|
||||
Probability: 2 (third-party dependency)
|
||||
Impact: 3 (auth breach = data exposure)
|
||||
Score: 6 - Security testing mandatory
|
||||
```
|
||||
|
||||
**PERF** - Performance degradation
|
||||
```
|
||||
Example: Adding real-time notifications
|
||||
Probability: 2 (WebSocket complexity)
|
||||
Impact: 2 (slower experience)
|
||||
Score: 4 - Load testing recommended
|
||||
```
|
||||
|
||||
**DATA** - Data integrity, corruption
|
||||
```
|
||||
Example: Database migration
|
||||
Probability: 2 (schema changes)
|
||||
Impact: 3 (data loss unacceptable)
|
||||
Score: 6 - Data validation tests required
|
||||
```
|
||||
|
||||
**BUS** - Business logic errors
|
||||
```
|
||||
Example: Discount calculation
|
||||
Probability: 2 (business rules complex)
|
||||
Impact: 3 (wrong prices = revenue loss)
|
||||
Score: 6 - Business logic tests mandatory
|
||||
```
|
||||
|
||||
**OPS** - Operational issues
|
||||
```
|
||||
Example: Logging system update
|
||||
Probability: 1 (straightforward)
|
||||
Impact: 2 (debugging harder without logs)
|
||||
Score: 2 - Basic smoke test sufficient
|
||||
```
|
||||
|
||||
### 2. Test Priorities (P0-P3)
|
||||
|
||||
Risk scores inform test priorities (but aren't the only factor):
|
||||
|
||||
**P0 - Critical Path**
|
||||
- **Risk Scores:** Typically 6-9 (high risk)
|
||||
- **Other Factors:** Revenue impact, security-critical, regulatory compliance, frequent usage
|
||||
- **Coverage Target:** 100%
|
||||
- **Test Levels:** E2E + API
|
||||
- **Example:** Login, checkout, payment processing
|
||||
|
||||
**P1 - High Value**
|
||||
- **Risk Scores:** Typically 4-6 (medium-high risk)
|
||||
- **Other Factors:** Core user journeys, complex logic, integration points
|
||||
- **Coverage Target:** 90%
|
||||
- **Test Levels:** API + selective E2E
|
||||
- **Example:** Profile editing, search, filters
|
||||
|
||||
**P2 - Medium Value**
|
||||
- **Risk Scores:** Typically 2-4 (medium risk)
|
||||
- **Other Factors:** Secondary features, admin functionality, reporting
|
||||
- **Coverage Target:** 50%
|
||||
- **Test Levels:** API happy path only
|
||||
- **Example:** Export features, advanced settings
|
||||
|
||||
**P3 - Low Value**
|
||||
- **Risk Scores:** Typically 1-2 (low risk)
|
||||
- **Other Factors:** Rarely used, nice-to-have, cosmetic
|
||||
- **Coverage Target:** 20% (smoke test)
|
||||
- **Test Levels:** E2E smoke test only
|
||||
- **Example:** Theme customization, experimental features
|
||||
|
||||
**Note:** Priorities consider risk scores plus business context (usage frequency, user impact, etc.). See [Test Priorities Matrix](/docs/reference/tea/knowledge-base.md#test-priorities-matrix) for complete criteria.
|
||||
|
||||
### 3. Mitigation Plans
|
||||
|
||||
**Scores ≥6 require documented mitigation:**
|
||||
|
||||
```markdown
|
||||
## Risk Mitigation
|
||||
|
||||
**Risk:** Payment integration failure (Score: 9)
|
||||
|
||||
**Mitigation Plan:**
|
||||
- Create comprehensive test suite (20+ tests)
|
||||
- Add payment sandbox environment
|
||||
- Implement retry logic with idempotency
|
||||
- Add monitoring and alerts
|
||||
- Document rollback procedure
|
||||
|
||||
**Owner:** Backend team lead
|
||||
**Deadline:** Before production deployment
|
||||
**Status:** In progress
|
||||
```
|
||||
|
||||
**Gate Rules:**
|
||||
- **Score = 9** (Critical): Mandatory FAIL - blocks release without mitigation
|
||||
- **Score 6-8** (High): Requires mitigation plan, becomes CONCERNS if incomplete
|
||||
- **Score 4-5** (Medium): Mitigation recommended but not required
|
||||
- **Score 1-3** (Low): No mitigation needed
|
||||
|
||||
## Comparison: Traditional vs Risk-Based
|
||||
|
||||
### Traditional Approach
|
||||
|
||||
```typescript
|
||||
// Test everything equally
|
||||
describe('User profile', () => {
|
||||
test('should display name');
|
||||
test('should display email');
|
||||
test('should display phone');
|
||||
test('should display address');
|
||||
test('should display bio');
|
||||
test('should display avatar');
|
||||
test('should display join date');
|
||||
test('should display last login');
|
||||
test('should display theme preference');
|
||||
test('should display language preference');
|
||||
// 10 tests for profile display (all equal priority)
|
||||
});
|
||||
```
|
||||
|
||||
**Problems:**
|
||||
- Same effort for critical (name) vs trivial (theme)
|
||||
- No guidance on what matters
|
||||
- Wastes time on low-value tests
|
||||
|
||||
### Risk-Based Approach
|
||||
|
||||
```typescript
|
||||
// Test based on risk
|
||||
|
||||
describe('User profile - Critical (P0)', () => {
|
||||
test('should display name and email'); // Score: 9 (identity critical)
|
||||
test('should allow editing name and email');
|
||||
test('should validate email format');
|
||||
test('should prevent unauthorized edits');
|
||||
// 4 focused tests on high-risk areas
|
||||
});
|
||||
|
||||
describe('User profile - High Value (P1)', () => {
|
||||
test('should upload avatar'); // Score: 6 (users care about this)
|
||||
test('should update bio');
|
||||
// 2 tests for high-value features
|
||||
});
|
||||
|
||||
// P2: Theme preference - single smoke test
|
||||
// P3: Last login display - skip (read-only, low value)
|
||||
```
|
||||
|
||||
**Benefits:**
|
||||
- 6 focused tests vs 10 unfocused tests
|
||||
- Effort matches business impact
|
||||
- Clear priorities guide development
|
||||
- No wasted effort on trivial features
|
||||
|
||||
## When to Use Risk-Based Testing
|
||||
|
||||
### Always Use For:
|
||||
|
||||
**Enterprise projects:**
|
||||
- High stakes (revenue, compliance, security)
|
||||
- Many features competing for test effort
|
||||
- Need objective prioritization
|
||||
|
||||
**Large codebases:**
|
||||
- Can't test everything exhaustively
|
||||
- Need to focus limited QA resources
|
||||
- Want data-driven decisions
|
||||
|
||||
**Regulated industries:**
|
||||
- Must justify testing decisions
|
||||
- Auditors want risk assessments
|
||||
- Compliance requires evidence
|
||||
|
||||
### Consider Skipping For:
|
||||
|
||||
**Tiny projects:**
|
||||
- 5 features total
|
||||
- Can test everything thoroughly
|
||||
- Risk scoring is overhead
|
||||
|
||||
**Prototypes:**
|
||||
- Throw-away code
|
||||
- Speed over quality
|
||||
- Learning experiments
|
||||
|
||||
## Real-World Example
|
||||
|
||||
### Scenario: E-Commerce Checkout Redesign
|
||||
|
||||
**Feature:** Redesigning checkout flow from 5 steps to 3 steps
|
||||
|
||||
**Risk Assessment:**
|
||||
|
||||
| Component | Probability | Impact | Score | Priority | Testing |
|
||||
|-----------|-------------|--------|-------|----------|---------|
|
||||
| **Payment processing** | 3 | 3 | 9 | P0 | 15 E2E + 20 API tests |
|
||||
| **Order validation** | 2 | 3 | 6 | P1 | 5 E2E + 10 API tests |
|
||||
| **Shipping calculation** | 2 | 2 | 4 | P1 | 3 E2E + 8 API tests |
|
||||
| **Promo code validation** | 2 | 2 | 4 | P1 | 2 E2E + 5 API tests |
|
||||
| **Gift message** | 1 | 1 | 1 | P3 | 1 E2E smoke test |
|
||||
|
||||
**Test Budget:** 40 hours
|
||||
|
||||
**Allocation:**
|
||||
- Payment (Score 9): 20 hours (50%)
|
||||
- Order validation (Score 6): 8 hours (20%)
|
||||
- Shipping (Score 4): 6 hours (15%)
|
||||
- Promo codes (Score 4): 4 hours (10%)
|
||||
- Gift message (Score 1): 2 hours (5%)
|
||||
|
||||
**Result:** 50% of effort on highest-risk feature (payment), proportional allocation for others.
|
||||
|
||||
### Without Risk-Based Testing:
|
||||
|
||||
**Equal allocation:** 8 hours per component = wasted effort on gift message, under-testing payment.
|
||||
|
||||
**Result:** Payment bugs slip through (critical), perfect testing of gift message (trivial).
|
||||
|
||||
## Mitigation Strategies by Risk Level
|
||||
|
||||
### Score 9: Mandatory Mitigation (Blocks Release)
|
||||
|
||||
```markdown
|
||||
**Gate Impact:** FAIL - Cannot deploy without mitigation
|
||||
|
||||
**Actions:**
|
||||
- Comprehensive test suite (E2E, API, security)
|
||||
- Multiple test environments (dev, staging, prod-mirror)
|
||||
- Load testing and performance validation
|
||||
- Security audit and penetration testing
|
||||
- Monitoring and alerting
|
||||
- Rollback plan documented
|
||||
- On-call rotation assigned
|
||||
|
||||
**Cannot deploy until score is mitigated below 9.**
|
||||
```
|
||||
|
||||
### Score 6-8: Required Mitigation (Gate: CONCERNS)
|
||||
|
||||
```markdown
|
||||
**Gate Impact:** CONCERNS - Can deploy with documented mitigation plan
|
||||
|
||||
**Actions:**
|
||||
- Targeted test suite (happy path + critical errors)
|
||||
- Test environment setup
|
||||
- Monitoring plan
|
||||
- Document mitigation and owners
|
||||
|
||||
**Can deploy with approved mitigation plan.**
|
||||
```
|
||||
|
||||
### Score 4-5: Recommended Mitigation
|
||||
|
||||
```markdown
|
||||
**Gate Impact:** Advisory - Does not affect gate decision
|
||||
|
||||
**Actions:**
|
||||
- Basic test coverage
|
||||
- Standard monitoring
|
||||
- Document known limitations
|
||||
|
||||
**Can deploy, mitigation recommended but not required.**
|
||||
```
|
||||
|
||||
### Score 1-3: Optional Mitigation
|
||||
|
||||
```markdown
|
||||
**Gate Impact:** None
|
||||
|
||||
**Actions:**
|
||||
- Smoke test if desired
|
||||
- Feature flag for easy disable (optional)
|
||||
|
||||
**Can deploy without mitigation.**
|
||||
```
|
||||
|
||||
## Technical Implementation
|
||||
|
||||
For detailed risk governance patterns, see the knowledge base:
|
||||
- [Knowledge Base Index - Risk & Gates](/docs/reference/tea/knowledge-base.md)
|
||||
- [TEA Command Reference - *test-design](/docs/reference/tea/commands.md#test-design)
|
||||
|
||||
### Risk Scoring Matrix
|
||||
|
||||
TEA uses this framework in `*test-design`:
|
||||
|
||||
```
|
||||
Impact
|
||||
1 2 3
|
||||
┌────┬────┬────┐
|
||||
1 │ 1 │ 2 │ 3 │ Low risk
|
||||
P 2 │ 2 │ 4 │ 6 │ Medium risk
|
||||
r 3 │ 3 │ 6 │ 9 │ High risk
|
||||
o └────┴────┴────┘
|
||||
b Low Med High
|
||||
```
|
||||
|
||||
### Gate Decision Rules
|
||||
|
||||
| Score | Mitigation Required | Gate Impact |
|
||||
|-------|-------------------|-------------|
|
||||
| **9** | Mandatory, blocks release | FAIL if no mitigation |
|
||||
| **6-8** | Required, documented plan | CONCERNS if incomplete |
|
||||
| **4-5** | Recommended | Advisory only |
|
||||
| **1-3** | Optional | No impact |
|
||||
|
||||
#### Gate Decision Flow
|
||||
|
||||
```mermaid
|
||||
%%{init: {'theme':'base', 'themeVariables': { 'fontSize':'14px'}}}%%
|
||||
flowchart TD
|
||||
Start([Risk Assessment]) --> Score{Risk Score?}
|
||||
|
||||
Score -->|Score = 9| Critical[CRITICAL RISK<br/>Score: 9]
|
||||
Score -->|Score 6-8| High[HIGH RISK<br/>Score: 6-8]
|
||||
Score -->|Score 4-5| Medium[MEDIUM RISK<br/>Score: 4-5]
|
||||
Score -->|Score 1-3| Low[LOW RISK<br/>Score: 1-3]
|
||||
|
||||
Critical --> HasMit9{Mitigation<br/>Plan?}
|
||||
HasMit9 -->|Yes| Concerns9[CONCERNS ⚠️<br/>Can deploy with plan]
|
||||
HasMit9 -->|No| Fail[FAIL ❌<br/>Blocks release]
|
||||
|
||||
High --> HasMit6{Mitigation<br/>Plan?}
|
||||
HasMit6 -->|Yes| Pass6[PASS ✅<br/>or CONCERNS ⚠️]
|
||||
HasMit6 -->|No| Concerns6[CONCERNS ⚠️<br/>Document plan needed]
|
||||
|
||||
Medium --> Advisory[Advisory Only<br/>No gate impact]
|
||||
Low --> NoAction[No Action<br/>Proceed]
|
||||
|
||||
style Critical fill:#f44336,stroke:#b71c1c,stroke-width:3px,color:#fff
|
||||
style Fail fill:#d32f2f,stroke:#b71c1c,stroke-width:3px,color:#fff
|
||||
style High fill:#ff9800,stroke:#e65100,stroke-width:2px,color:#000
|
||||
style Concerns9 fill:#ffc107,stroke:#f57f17,stroke-width:2px,color:#000
|
||||
style Concerns6 fill:#ffc107,stroke:#f57f17,stroke-width:2px,color:#000
|
||||
style Pass6 fill:#4caf50,stroke:#1b5e20,stroke-width:2px,color:#fff
|
||||
style Medium fill:#fff9c4,stroke:#f57f17,stroke-width:1px,color:#000
|
||||
style Low fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px,color:#000
|
||||
style Advisory fill:#e8f5e9,stroke:#2e7d32,stroke-width:1px,color:#000
|
||||
style NoAction fill:#e8f5e9,stroke:#2e7d32,stroke-width:1px,color:#000
|
||||
```
|
||||
|
||||
## Common Misconceptions
|
||||
|
||||
### "Risk-based = Less Testing"
|
||||
|
||||
**Wrong:** Risk-based testing often means MORE testing where it matters.
|
||||
|
||||
**Example:**
|
||||
- Traditional: 50 tests spread equally
|
||||
- Risk-based: 70 tests focused on P0/P1 (more total, better allocated)
|
||||
|
||||
### "Low Priority = Skip Testing"
|
||||
|
||||
**Wrong:** P3 still gets smoke tests.
|
||||
|
||||
**Correct:**
|
||||
- P3: Smoke test (feature works at all)
|
||||
- P2: Happy path (feature works correctly)
|
||||
- P1: Happy path + errors
|
||||
- P0: Comprehensive (all scenarios)
|
||||
|
||||
### "Risk Scores Are Permanent"
|
||||
|
||||
**Wrong:** Risk changes over time.
|
||||
|
||||
**Correct:**
|
||||
- Initial launch: Payment is Score 9 (untested integration)
|
||||
- After 6 months: Payment is Score 6 (proven in production)
|
||||
- Re-assess risk quarterly
|
||||
|
||||
## Related Concepts
|
||||
|
||||
**Core TEA Concepts:**
|
||||
- [Test Quality Standards](/docs/explanation/tea/test-quality-standards.md) - Quality complements risk assessment
|
||||
- [Engagement Models](/docs/explanation/tea/engagement-models.md) - When risk-based testing matters most
|
||||
- [Knowledge Base System](/docs/explanation/tea/knowledge-base-system.md) - How risk patterns are loaded
|
||||
|
||||
**Technical Patterns:**
|
||||
- [Fixture Architecture](/docs/explanation/tea/fixture-architecture.md) - Building risk-appropriate test infrastructure
|
||||
- [Network-First Patterns](/docs/explanation/tea/network-first-patterns.md) - Quality patterns for high-risk features
|
||||
|
||||
**Overview:**
|
||||
- [TEA Overview](/docs/explanation/features/tea-overview.md) - Risk assessment in TEA lifecycle
|
||||
- [Testing as Engineering](/docs/explanation/philosophy/testing-as-engineering.md) - Design philosophy
|
||||
|
||||
## Practical Guides
|
||||
|
||||
**Workflow Guides:**
|
||||
- [How to Run Test Design](/docs/how-to/workflows/run-test-design.md) - Apply risk scoring
|
||||
- [How to Run Trace](/docs/how-to/workflows/run-trace.md) - Gate decisions based on risk
|
||||
- [How to Run NFR Assessment](/docs/how-to/workflows/run-nfr-assess.md) - NFR risk assessment
|
||||
|
||||
**Use-Case Guides:**
|
||||
- [Running TEA for Enterprise](/docs/how-to/enterprise/use-tea-for-enterprise.md) - Enterprise risk management
|
||||
|
||||
## Reference
|
||||
|
||||
- [TEA Command Reference](/docs/reference/tea/commands.md) - `*test-design`, `*nfr-assess`, `*trace`
|
||||
- [Knowledge Base Index](/docs/reference/tea/knowledge-base.md) - Risk governance fragments
|
||||
- [Glossary](/docs/reference/glossary/index.md#test-architect-tea-concepts) - Risk-based testing term
|
||||
|
||||
---
|
||||
|
||||
Generated with [BMad Method](https://bmad-method.org) - TEA (Test Architect)
|
||||
907
docs/explanation/tea/test-quality-standards.md
Normal file
907
docs/explanation/tea/test-quality-standards.md
Normal file
@@ -0,0 +1,907 @@
|
||||
---
|
||||
title: "Test Quality Standards Explained"
|
||||
description: Understanding TEA's Definition of Done for deterministic, isolated, and maintainable tests
|
||||
---
|
||||
|
||||
# Test Quality Standards Explained
|
||||
|
||||
Test quality standards define what makes a test "good" in TEA. These aren't suggestions - they're the Definition of Done that prevents tests from rotting in review.
|
||||
|
||||
## Overview
|
||||
|
||||
**TEA's Quality Principles:**
|
||||
- **Deterministic** - Same result every run
|
||||
- **Isolated** - No dependencies on other tests
|
||||
- **Explicit** - Assertions visible in test body
|
||||
- **Focused** - Single responsibility, appropriate size
|
||||
- **Fast** - Execute in reasonable time
|
||||
|
||||
**Why these matter:** Tests that violate these principles create maintenance burden, slow down development, and lose team trust.
|
||||
|
||||
## The Problem
|
||||
|
||||
### Tests That Rot in Review
|
||||
|
||||
```typescript
|
||||
// ❌ The anti-pattern: This test will rot
|
||||
test('user can do stuff', async ({ page }) => {
|
||||
await page.goto('/');
|
||||
await page.waitForTimeout(5000); // Non-deterministic
|
||||
|
||||
if (await page.locator('.banner').isVisible()) { // Conditional
|
||||
await page.click('.dismiss');
|
||||
}
|
||||
|
||||
try { // Try-catch for flow control
|
||||
await page.click('#load-more');
|
||||
} catch (e) {
|
||||
// Silently continue
|
||||
}
|
||||
|
||||
// ... 300 more lines of test logic
|
||||
// ... no clear assertions
|
||||
});
|
||||
```
|
||||
|
||||
**What's wrong:**
|
||||
- **Hard wait** - Flaky, wastes time
|
||||
- **Conditional** - Non-deterministic behavior
|
||||
- **Try-catch** - Hides failures
|
||||
- **Too large** - Hard to maintain
|
||||
- **Vague name** - Unclear purpose
|
||||
- **No explicit assertions** - What's being tested?
|
||||
|
||||
**Result:** PR review comments: "This test is flaky, please fix" → never merged → test deleted → coverage lost
|
||||
|
||||
### AI-Generated Tests Without Standards
|
||||
|
||||
AI-generated tests without quality guardrails:
|
||||
|
||||
```typescript
|
||||
// AI generates 50 tests like this:
|
||||
test('test1', async ({ page }) => {
|
||||
await page.goto('/');
|
||||
await page.waitForTimeout(3000);
|
||||
// ... flaky, vague, redundant
|
||||
});
|
||||
|
||||
test('test2', async ({ page }) => {
|
||||
await page.goto('/');
|
||||
await page.waitForTimeout(3000);
|
||||
// ... duplicates test1
|
||||
});
|
||||
|
||||
// ... 48 more similar tests
|
||||
```
|
||||
|
||||
**Result:** 50 tests, 80% redundant, 90% flaky, 0% trusted by team - low-quality outputs that create maintenance burden.
|
||||
|
||||
## The Solution: TEA's Quality Standards
|
||||
|
||||
### 1. Determinism (No Flakiness)
|
||||
|
||||
**Rule:** Test produces same result every run.
|
||||
|
||||
**Requirements:**
|
||||
- ❌ No hard waits (`waitForTimeout`)
|
||||
- ❌ No conditionals for flow control (`if/else`)
|
||||
- ❌ No try-catch for flow control
|
||||
- ✅ Use network-first patterns (wait for responses)
|
||||
- ✅ Use explicit waits (waitForSelector, waitForResponse)
|
||||
|
||||
**Bad Example:**
|
||||
```typescript
|
||||
test('flaky test', async ({ page }) => {
|
||||
await page.click('button');
|
||||
await page.waitForTimeout(2000); // ❌ Might be too short
|
||||
|
||||
if (await page.locator('.modal').isVisible()) { // ❌ Non-deterministic
|
||||
await page.click('.dismiss');
|
||||
}
|
||||
|
||||
try { // ❌ Silently handles errors
|
||||
await expect(page.locator('.success')).toBeVisible();
|
||||
} catch (e) {
|
||||
// Test passes even if assertion fails!
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
**Good Example (Vanilla Playwright):**
|
||||
```typescript
|
||||
test('deterministic test', async ({ page }) => {
|
||||
const responsePromise = page.waitForResponse(
|
||||
resp => resp.url().includes('/api/submit') && resp.ok()
|
||||
);
|
||||
|
||||
await page.click('button');
|
||||
await responsePromise; // ✅ Wait for actual response
|
||||
|
||||
// Modal should ALWAYS show (make it deterministic)
|
||||
await expect(page.locator('.modal')).toBeVisible();
|
||||
await page.click('.dismiss');
|
||||
|
||||
// Explicit assertion (fails if not visible)
|
||||
await expect(page.locator('.success')).toBeVisible();
|
||||
});
|
||||
```
|
||||
|
||||
**With Playwright Utils (Even Cleaner):**
|
||||
```typescript
|
||||
import { test } from '@seontechnologies/playwright-utils/fixtures';
|
||||
import { expect } from '@playwright/test';
|
||||
|
||||
test('deterministic test', async ({ page, interceptNetworkCall }) => {
|
||||
const submitCall = interceptNetworkCall({
|
||||
method: 'POST',
|
||||
url: '**/api/submit'
|
||||
});
|
||||
|
||||
await page.click('button');
|
||||
|
||||
// Wait for actual response (automatic JSON parsing)
|
||||
const { status, responseJson } = await submitCall;
|
||||
expect(status).toBe(200);
|
||||
|
||||
// Modal should ALWAYS show (make it deterministic)
|
||||
await expect(page.locator('.modal')).toBeVisible();
|
||||
await page.click('.dismiss');
|
||||
|
||||
// Explicit assertion (fails if not visible)
|
||||
await expect(page.locator('.success')).toBeVisible();
|
||||
});
|
||||
```
|
||||
|
||||
**Why both work:**
|
||||
- Waits for actual event (network response)
|
||||
- No conditionals (behavior is deterministic)
|
||||
- Assertions fail loudly (no silent failures)
|
||||
- Same result every run (deterministic)
|
||||
|
||||
**Playwright Utils additional benefits:**
|
||||
- Automatic JSON parsing
|
||||
- `{ status, responseJson }` structure (can validate response data)
|
||||
- No manual `await response.json()`
|
||||
|
||||
### 2. Isolation (No Dependencies)
|
||||
|
||||
**Rule:** Test runs independently, no shared state.
|
||||
|
||||
**Requirements:**
|
||||
- ✅ Self-cleaning (cleanup after test)
|
||||
- ✅ No global state dependencies
|
||||
- ✅ Can run in parallel
|
||||
- ✅ Can run in any order
|
||||
- ✅ Use unique test data
|
||||
|
||||
**Bad Example:**
|
||||
```typescript
|
||||
// ❌ Tests depend on execution order
|
||||
let userId: string; // Shared global state
|
||||
|
||||
test('create user', async ({ apiRequest }) => {
|
||||
const { body } = await apiRequest({
|
||||
method: 'POST',
|
||||
path: '/api/users',
|
||||
body: { email: 'test@example.com' } (hard-coded)
|
||||
});
|
||||
userId = body.id; // Store in global
|
||||
});
|
||||
|
||||
test('update user', async ({ apiRequest }) => {
|
||||
// Depends on previous test setting userId
|
||||
await apiRequest({
|
||||
method: 'PATCH',
|
||||
path: `/api/users/${userId}`,
|
||||
body: { name: 'Updated' }
|
||||
});
|
||||
// No cleanup - leaves user in database
|
||||
});
|
||||
```
|
||||
|
||||
**Problems:**
|
||||
- Tests must run in order (can't parallelize)
|
||||
- Second test fails if first skipped (`.only`)
|
||||
- Hard-coded data causes conflicts
|
||||
- No cleanup (database fills with test data)
|
||||
|
||||
**Good Example (Vanilla Playwright):**
|
||||
```typescript
|
||||
test('should update user profile', async ({ request }) => {
|
||||
// Create unique test data
|
||||
const testEmail = `test-${Date.now()}@example.com`;
|
||||
|
||||
// Setup: Create user
|
||||
const createResp = await request.post('/api/users', {
|
||||
data: { email: testEmail, name: 'Original' }
|
||||
});
|
||||
const user = await createResp.json();
|
||||
|
||||
// Test: Update user
|
||||
const updateResp = await request.patch(`/api/users/${user.id}`, {
|
||||
data: { name: 'Updated' }
|
||||
});
|
||||
const updated = await updateResp.json();
|
||||
|
||||
expect(updated.name).toBe('Updated');
|
||||
|
||||
// Cleanup: Delete user
|
||||
await request.delete(`/api/users/${user.id}`);
|
||||
});
|
||||
```
|
||||
|
||||
**Even Better (With Playwright Utils):**
|
||||
```typescript
|
||||
import { test } from '@seontechnologies/playwright-utils/api-request/fixtures';
|
||||
import { expect } from '@playwright/test';
|
||||
import { faker } from '@faker-js/faker';
|
||||
|
||||
test('should update user profile', async ({ apiRequest }) => {
|
||||
// Dynamic unique test data
|
||||
const testEmail = faker.internet.email();
|
||||
|
||||
// Setup: Create user
|
||||
const { status: createStatus, body: user } = await apiRequest({
|
||||
method: 'POST',
|
||||
path: '/api/users',
|
||||
body: { email: testEmail, name: faker.person.fullName() }
|
||||
});
|
||||
|
||||
expect(createStatus).toBe(201);
|
||||
|
||||
// Test: Update user
|
||||
const { status, body: updated } = await apiRequest({
|
||||
method: 'PATCH',
|
||||
path: `/api/users/${user.id}`,
|
||||
body: { name: 'Updated Name' }
|
||||
});
|
||||
|
||||
expect(status).toBe(200);
|
||||
expect(updated.name).toBe('Updated Name');
|
||||
|
||||
// Cleanup: Delete user
|
||||
await apiRequest({
|
||||
method: 'DELETE',
|
||||
path: `/api/users/${user.id}`
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
**Playwright Utils Benefits:**
|
||||
- `{ status, body }` destructuring (cleaner than `response.status()` + `await response.json()`)
|
||||
- No manual `await response.json()`
|
||||
- Automatic retry for 5xx errors
|
||||
- Optional schema validation with `.validateSchema()`
|
||||
|
||||
**Why it works:**
|
||||
- No global state
|
||||
- Unique test data (no conflicts)
|
||||
- Self-cleaning (deletes user)
|
||||
- Can run in parallel
|
||||
- Can run in any order
|
||||
|
||||
### 3. Explicit Assertions (No Hidden Validation)
|
||||
|
||||
**Rule:** Assertions visible in test body, not abstracted.
|
||||
|
||||
**Requirements:**
|
||||
- ✅ Assertions in test code (not helper functions)
|
||||
- ✅ Specific assertions (not generic `toBeTruthy`)
|
||||
- ✅ Meaningful expectations (test actual behavior)
|
||||
|
||||
**Bad Example:**
|
||||
```typescript
|
||||
// ❌ Assertions hidden in helper
|
||||
async function verifyProfilePage(page: Page) {
|
||||
// Assertions buried in helper (not visible in test)
|
||||
await expect(page.locator('h1')).toBeVisible();
|
||||
await expect(page.locator('.email')).toContainText('@');
|
||||
await expect(page.locator('.name')).not.toBeEmpty();
|
||||
}
|
||||
|
||||
test('profile page', async ({ page }) => {
|
||||
await page.goto('/profile');
|
||||
await verifyProfilePage(page); // What's being verified?
|
||||
});
|
||||
```
|
||||
|
||||
**Problems:**
|
||||
- Can't see what's tested (need to read helper)
|
||||
- Hard to debug failures (which assertion failed?)
|
||||
- Reduces test readability
|
||||
- Hides important validation
|
||||
|
||||
**Good Example:**
|
||||
```typescript
|
||||
// ✅ Assertions explicit in test
|
||||
test('should display profile with correct data', async ({ page }) => {
|
||||
await page.goto('/profile');
|
||||
|
||||
// Explicit assertions - clear what's tested
|
||||
await expect(page.locator('h1')).toContainText('Test User');
|
||||
await expect(page.locator('.email')).toContainText('test@example.com');
|
||||
await expect(page.locator('.bio')).toContainText('Software Engineer');
|
||||
await expect(page.locator('img[alt="Avatar"]')).toBeVisible();
|
||||
});
|
||||
```
|
||||
|
||||
**Why it works:**
|
||||
- See what's tested at a glance
|
||||
- Debug failures easily (know which assertion failed)
|
||||
- Test is self-documenting
|
||||
- No hidden behavior
|
||||
|
||||
**Exception:** Use helper for setup/cleanup, not assertions.
|
||||
|
||||
### 4. Focused Tests (Appropriate Size)
|
||||
|
||||
**Rule:** Test has single responsibility, reasonable size.
|
||||
|
||||
**Requirements:**
|
||||
- ✅ Test size < 300 lines
|
||||
- ✅ Single responsibility (test one thing well)
|
||||
- ✅ Clear describe/test names
|
||||
- ✅ Appropriate scope (not too granular, not too broad)
|
||||
|
||||
**Bad Example:**
|
||||
```typescript
|
||||
// ❌ 500-line test testing everything
|
||||
test('complete user flow', async ({ page }) => {
|
||||
// Registration (50 lines)
|
||||
await page.goto('/register');
|
||||
await page.fill('#email', 'test@example.com');
|
||||
// ... 48 more lines
|
||||
|
||||
// Profile setup (100 lines)
|
||||
await page.goto('/profile');
|
||||
// ... 98 more lines
|
||||
|
||||
// Settings configuration (150 lines)
|
||||
await page.goto('/settings');
|
||||
// ... 148 more lines
|
||||
|
||||
// Data export (200 lines)
|
||||
await page.goto('/export');
|
||||
// ... 198 more lines
|
||||
|
||||
// Total: 500 lines, testing 4 different features
|
||||
});
|
||||
```
|
||||
|
||||
**Problems:**
|
||||
- Failure in line 50 prevents testing lines 51-500
|
||||
- Hard to understand (what's being tested?)
|
||||
- Slow to execute (testing too much)
|
||||
- Hard to debug (which feature failed?)
|
||||
|
||||
**Good Example:**
|
||||
```typescript
|
||||
// ✅ Focused tests - one responsibility each
|
||||
|
||||
test('should register new user', async ({ page }) => {
|
||||
await page.goto('/register');
|
||||
await page.fill('#email', 'test@example.com');
|
||||
await page.fill('#password', 'password123');
|
||||
await page.click('button[type="submit"]');
|
||||
|
||||
await expect(page).toHaveURL('/welcome');
|
||||
await expect(page.locator('h1')).toContainText('Welcome');
|
||||
});
|
||||
|
||||
test('should configure user profile', async ({ page, authSession }) => {
|
||||
await authSession.login({ email: 'test@example.com', password: 'pass' });
|
||||
await page.goto('/profile');
|
||||
|
||||
await page.fill('#name', 'Test User');
|
||||
await page.fill('#bio', 'Software Engineer');
|
||||
await page.click('button:has-text("Save")');
|
||||
|
||||
await expect(page.locator('.success')).toBeVisible();
|
||||
});
|
||||
|
||||
// ... separate tests for settings, export (each < 50 lines)
|
||||
```
|
||||
|
||||
**Why it works:**
|
||||
- Each test has one responsibility
|
||||
- Failure is easy to diagnose
|
||||
- Can run tests independently
|
||||
- Test names describe exactly what's tested
|
||||
|
||||
### 5. Fast Execution (Performance Budget)
|
||||
|
||||
**Rule:** Individual test executes in < 1.5 minutes.
|
||||
|
||||
**Requirements:**
|
||||
- ✅ Test execution < 90 seconds
|
||||
- ✅ Efficient selectors (getByRole > XPath)
|
||||
- ✅ Minimal redundant actions
|
||||
- ✅ Parallel execution enabled
|
||||
|
||||
**Bad Example:**
|
||||
```typescript
|
||||
// ❌ Slow test (3+ minutes)
|
||||
test('slow test', async ({ page }) => {
|
||||
await page.goto('/');
|
||||
await page.waitForTimeout(10000); // 10s wasted
|
||||
|
||||
// Navigate through 10 pages (2 minutes)
|
||||
for (let i = 1; i <= 10; i++) {
|
||||
await page.click(`a[href="/page-${i}"]`);
|
||||
await page.waitForTimeout(5000); // 5s per page = 50s wasted
|
||||
}
|
||||
|
||||
// Complex XPath selector (slow)
|
||||
await page.locator('//div[@class="container"]/section[3]/div[2]/p').click();
|
||||
|
||||
// More waiting
|
||||
await page.waitForTimeout(30000); // 30s wasted
|
||||
|
||||
await expect(page.locator('.result')).toBeVisible();
|
||||
});
|
||||
```
|
||||
|
||||
**Total time:** 3+ minutes (95 seconds wasted on hard waits)
|
||||
|
||||
**Good Example (Vanilla Playwright):**
|
||||
```typescript
|
||||
// ✅ Fast test (< 10 seconds)
|
||||
test('fast test', async ({ page }) => {
|
||||
// Set up response wait
|
||||
const apiPromise = page.waitForResponse(
|
||||
resp => resp.url().includes('/api/result') && resp.ok()
|
||||
);
|
||||
|
||||
await page.goto('/');
|
||||
|
||||
// Direct navigation (skip intermediate pages)
|
||||
await page.goto('/page-10');
|
||||
|
||||
// Efficient selector
|
||||
await page.getByRole('button', { name: 'Submit' }).click();
|
||||
|
||||
// Wait for actual response (fast when API is fast)
|
||||
await apiPromise;
|
||||
|
||||
await expect(page.locator('.result')).toBeVisible();
|
||||
});
|
||||
```
|
||||
|
||||
**With Playwright Utils:**
|
||||
```typescript
|
||||
import { test } from '@seontechnologies/playwright-utils/fixtures';
|
||||
import { expect } from '@playwright/test';
|
||||
|
||||
test('fast test', async ({ page, interceptNetworkCall }) => {
|
||||
// Set up interception
|
||||
const resultCall = interceptNetworkCall({
|
||||
method: 'GET',
|
||||
url: '**/api/result'
|
||||
});
|
||||
|
||||
await page.goto('/');
|
||||
|
||||
// Direct navigation (skip intermediate pages)
|
||||
await page.goto('/page-10');
|
||||
|
||||
// Efficient selector
|
||||
await page.getByRole('button', { name: 'Submit' }).click();
|
||||
|
||||
// Wait for actual response (automatic JSON parsing)
|
||||
const { status, responseJson } = await resultCall;
|
||||
|
||||
expect(status).toBe(200);
|
||||
await expect(page.locator('.result')).toBeVisible();
|
||||
|
||||
// Can also validate response data if needed
|
||||
// expect(responseJson.data).toBeDefined();
|
||||
});
|
||||
```
|
||||
|
||||
**Total time:** < 10 seconds (no wasted waits)
|
||||
|
||||
**Both examples achieve:**
|
||||
- No hard waits (wait for actual events)
|
||||
- Direct navigation (skip unnecessary steps)
|
||||
- Efficient selectors (getByRole)
|
||||
- Fast execution
|
||||
|
||||
**Playwright Utils bonus:**
|
||||
- Can validate API response data easily
|
||||
- Automatic JSON parsing
|
||||
- Cleaner API
|
||||
|
||||
## TEA's Quality Scoring
|
||||
|
||||
TEA reviews tests against these standards in `*test-review`:
|
||||
|
||||
### Scoring Categories (100 points total)
|
||||
|
||||
**Determinism (35 points):**
|
||||
- No hard waits: 10 points
|
||||
- No conditionals: 10 points
|
||||
- No try-catch flow: 10 points
|
||||
- Network-first patterns: 5 points
|
||||
|
||||
**Isolation (25 points):**
|
||||
- Self-cleaning: 15 points
|
||||
- No global state: 5 points
|
||||
- Parallel-safe: 5 points
|
||||
|
||||
**Assertions (20 points):**
|
||||
- Explicit in test body: 10 points
|
||||
- Specific and meaningful: 10 points
|
||||
|
||||
**Structure (10 points):**
|
||||
- Test size < 300 lines: 5 points
|
||||
- Clear naming: 5 points
|
||||
|
||||
**Performance (10 points):**
|
||||
- Execution time < 1.5 min: 10 points
|
||||
|
||||
#### Quality Scoring Breakdown
|
||||
|
||||
```mermaid
|
||||
%%{init: {'theme':'base', 'themeVariables': { 'fontSize':'14px'}}}%%
|
||||
pie title Test Quality Score (100 points)
|
||||
"Determinism" : 35
|
||||
"Isolation" : 25
|
||||
"Assertions" : 20
|
||||
"Structure" : 10
|
||||
"Performance" : 10
|
||||
```
|
||||
|
||||
```mermaid
|
||||
%%{init: {'theme':'base', 'themeVariables': { 'fontSize':'13px'}}}%%
|
||||
flowchart LR
|
||||
subgraph Det[Determinism - 35 pts]
|
||||
D1[No hard waits<br/>10 pts]
|
||||
D2[No conditionals<br/>10 pts]
|
||||
D3[No try-catch flow<br/>10 pts]
|
||||
D4[Network-first<br/>5 pts]
|
||||
end
|
||||
|
||||
subgraph Iso[Isolation - 25 pts]
|
||||
I1[Self-cleaning<br/>15 pts]
|
||||
I2[No global state<br/>5 pts]
|
||||
I3[Parallel-safe<br/>5 pts]
|
||||
end
|
||||
|
||||
subgraph Assrt[Assertions - 20 pts]
|
||||
A1[Explicit in body<br/>10 pts]
|
||||
A2[Specific/meaningful<br/>10 pts]
|
||||
end
|
||||
|
||||
subgraph Struct[Structure - 10 pts]
|
||||
S1[Size < 300 lines<br/>5 pts]
|
||||
S2[Clear naming<br/>5 pts]
|
||||
end
|
||||
|
||||
subgraph Perf[Performance - 10 pts]
|
||||
P1[Time < 1.5 min<br/>10 pts]
|
||||
end
|
||||
|
||||
Det --> Total([Total: 100 points])
|
||||
Iso --> Total
|
||||
Assrt --> Total
|
||||
Struct --> Total
|
||||
Perf --> Total
|
||||
|
||||
style Det fill:#ffebee,stroke:#c62828,stroke-width:2px
|
||||
style Iso fill:#e3f2fd,stroke:#1565c0,stroke-width:2px
|
||||
style Assrt fill:#f3e5f5,stroke:#6a1b9a,stroke-width:2px
|
||||
style Struct fill:#fff9c4,stroke:#f57f17,stroke-width:2px
|
||||
style Perf fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
|
||||
style Total fill:#fff,stroke:#000,stroke-width:3px
|
||||
```
|
||||
|
||||
### Score Interpretation
|
||||
|
||||
| Score | Interpretation | Action |
|
||||
| ---------- | -------------- | -------------------------------------- |
|
||||
| **90-100** | Excellent | Production-ready, minimal changes |
|
||||
| **80-89** | Good | Minor improvements recommended |
|
||||
| **70-79** | Acceptable | Address recommendations before release |
|
||||
| **60-69** | Needs Work | Fix critical issues |
|
||||
| **< 60** | Critical | Significant refactoring needed |
|
||||
|
||||
## Comparison: Good vs Bad Tests
|
||||
|
||||
### Example: User Login
|
||||
|
||||
**Bad Test (Score: 45/100):**
|
||||
```typescript
|
||||
test('login test', async ({ page }) => { // Vague name
|
||||
await page.goto('/login');
|
||||
await page.waitForTimeout(3000); // -10 (hard wait)
|
||||
|
||||
await page.fill('[name="email"]', 'test@example.com');
|
||||
await page.fill('[name="password"]', 'password');
|
||||
|
||||
if (await page.locator('.remember-me').isVisible()) { // -10 (conditional)
|
||||
await page.click('.remember-me');
|
||||
}
|
||||
|
||||
await page.click('button');
|
||||
|
||||
try { // -10 (try-catch flow)
|
||||
await page.waitForURL('/dashboard', { timeout: 5000 });
|
||||
} catch (e) {
|
||||
// Ignore navigation failure
|
||||
}
|
||||
|
||||
// No assertions! -10
|
||||
// No cleanup! -10
|
||||
});
|
||||
```
|
||||
|
||||
**Issues:**
|
||||
- Determinism: 5/35 (hard wait, conditional, try-catch)
|
||||
- Isolation: 10/25 (no cleanup)
|
||||
- Assertions: 0/20 (no assertions!)
|
||||
- Structure: 15/10 (okay)
|
||||
- Performance: 5/10 (slow)
|
||||
- **Total: 45/100**
|
||||
|
||||
**Good Test (Score: 95/100):**
|
||||
```typescript
|
||||
test('should login with valid credentials and redirect to dashboard', async ({ page, authSession }) => {
|
||||
// Use fixture for deterministic auth
|
||||
const loginPromise = page.waitForResponse(
|
||||
resp => resp.url().includes('/api/auth/login') && resp.ok()
|
||||
);
|
||||
|
||||
await page.goto('/login');
|
||||
await page.getByLabel('Email').fill('test@example.com');
|
||||
await page.getByLabel('Password').fill('password123');
|
||||
await page.getByRole('button', { name: 'Sign in' }).click();
|
||||
|
||||
// Wait for actual API response
|
||||
const response = await loginPromise;
|
||||
const { token } = await response.json();
|
||||
|
||||
// Explicit assertions
|
||||
expect(token).toBeDefined();
|
||||
await expect(page).toHaveURL('/dashboard');
|
||||
await expect(page.getByText('Welcome back')).toBeVisible();
|
||||
|
||||
// Cleanup handled by authSession fixture
|
||||
});
|
||||
```
|
||||
|
||||
**Quality:**
|
||||
- Determinism: 35/35 (network-first, no conditionals)
|
||||
- Isolation: 25/25 (fixture handles cleanup)
|
||||
- Assertions: 20/20 (explicit and specific)
|
||||
- Structure: 10/10 (clear name, focused)
|
||||
- Performance: 5/10 (< 1 min)
|
||||
- **Total: 95/100**
|
||||
|
||||
### Example: API Testing
|
||||
|
||||
**Bad Test (Score: 50/100):**
|
||||
```typescript
|
||||
test('api test', async ({ request }) => {
|
||||
const response = await request.post('/api/users', {
|
||||
data: { email: 'test@example.com' } // Hard-coded (conflicts)
|
||||
});
|
||||
|
||||
if (response.ok()) { // Conditional
|
||||
const user = await response.json();
|
||||
// Weak assertion
|
||||
expect(user).toBeTruthy();
|
||||
}
|
||||
|
||||
// No cleanup - user left in database
|
||||
});
|
||||
```
|
||||
|
||||
**Good Test (Score: 92/100):**
|
||||
```typescript
|
||||
test('should create user with valid data', async ({ apiRequest }) => {
|
||||
// Unique test data
|
||||
const testEmail = `test-${Date.now()}@example.com`;
|
||||
|
||||
// Create user
|
||||
const { status, body } = await apiRequest({
|
||||
method: 'POST',
|
||||
path: '/api/users',
|
||||
body: { email: testEmail, name: 'Test User' }
|
||||
});
|
||||
|
||||
// Explicit assertions
|
||||
expect(status).toBe(201);
|
||||
expect(body.id).toBeDefined();
|
||||
expect(body.email).toBe(testEmail);
|
||||
expect(body.name).toBe('Test User');
|
||||
|
||||
// Cleanup
|
||||
await apiRequest({
|
||||
method: 'DELETE',
|
||||
path: `/api/users/${body.id}`
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
## How TEA Enforces Standards
|
||||
|
||||
### During Test Generation (`*atdd`, `*automate`)
|
||||
|
||||
TEA generates tests following standards by default:
|
||||
|
||||
```typescript
|
||||
// TEA-generated test (automatically follows standards)
|
||||
test('should submit contact form', async ({ page }) => {
|
||||
// Network-first pattern (no hard waits)
|
||||
const submitPromise = page.waitForResponse(
|
||||
resp => resp.url().includes('/api/contact') && resp.ok()
|
||||
);
|
||||
|
||||
// Accessible selectors (resilient)
|
||||
await page.getByLabel('Name').fill('Test User');
|
||||
await page.getByLabel('Email').fill('test@example.com');
|
||||
await page.getByLabel('Message').fill('Test message');
|
||||
await page.getByRole('button', { name: 'Send' }).click();
|
||||
|
||||
const response = await submitPromise;
|
||||
const result = await response.json();
|
||||
|
||||
// Explicit assertions
|
||||
expect(result.success).toBe(true);
|
||||
await expect(page.getByText('Message sent')).toBeVisible();
|
||||
|
||||
// Size: 15 lines (< 300 ✓)
|
||||
// Execution: ~2 seconds (< 90s ✓)
|
||||
});
|
||||
```
|
||||
|
||||
### During Test Review (*test-review)
|
||||
|
||||
TEA audits tests and flags violations:
|
||||
|
||||
```markdown
|
||||
## Critical Issues
|
||||
|
||||
### Hard Wait Detected (tests/login.spec.ts:23)
|
||||
**Issue:** `await page.waitForTimeout(3000)`
|
||||
**Score Impact:** -10 (Determinism)
|
||||
**Fix:** Use network-first pattern
|
||||
|
||||
### Conditional Flow Control (tests/profile.spec.ts:45)
|
||||
**Issue:** `if (await page.locator('.banner').isVisible())`
|
||||
**Score Impact:** -10 (Determinism)
|
||||
**Fix:** Make banner presence deterministic
|
||||
|
||||
## Recommendations
|
||||
|
||||
### Extract Fixture (tests/auth.spec.ts)
|
||||
**Issue:** Login code repeated 5 times
|
||||
**Score Impact:** -3 (Structure)
|
||||
**Fix:** Extract to authSession fixture
|
||||
```
|
||||
|
||||
## Definition of Done Checklist
|
||||
|
||||
When is a test "done"?
|
||||
|
||||
**Test Quality DoD:**
|
||||
- [ ] No hard waits (`waitForTimeout`)
|
||||
- [ ] No conditionals for flow control
|
||||
- [ ] No try-catch for flow control
|
||||
- [ ] Network-first patterns used
|
||||
- [ ] Assertions explicit in test body
|
||||
- [ ] Test size < 300 lines
|
||||
- [ ] Clear, descriptive test name
|
||||
- [ ] Self-cleaning (cleanup in afterEach or test)
|
||||
- [ ] Unique test data (no hard-coded values)
|
||||
- [ ] Execution time < 1.5 minutes
|
||||
- [ ] Can run in parallel
|
||||
- [ ] Can run in any order
|
||||
|
||||
**Code Review DoD:**
|
||||
- [ ] Test quality score > 80
|
||||
- [ ] No critical issues from `*test-review`
|
||||
- [ ] Follows project patterns (fixtures, selectors)
|
||||
- [ ] Test reviewed by team member
|
||||
|
||||
## Common Quality Issues
|
||||
|
||||
### Issue: "My test needs conditionals for optional elements"
|
||||
|
||||
**Wrong approach:**
|
||||
```typescript
|
||||
if (await page.locator('.banner').isVisible()) {
|
||||
await page.click('.dismiss');
|
||||
}
|
||||
```
|
||||
|
||||
**Right approach - Make it deterministic:**
|
||||
```typescript
|
||||
// Option 1: Always expect banner
|
||||
await expect(page.locator('.banner')).toBeVisible();
|
||||
await page.click('.dismiss');
|
||||
|
||||
// Option 2: Test both scenarios separately
|
||||
test('should show banner for new users', ...);
|
||||
test('should not show banner for returning users', ...);
|
||||
```
|
||||
|
||||
### Issue: "My test needs try-catch for error handling"
|
||||
|
||||
**Wrong approach:**
|
||||
```typescript
|
||||
try {
|
||||
await page.click('#optional-button');
|
||||
} catch (e) {
|
||||
// Silently continue
|
||||
}
|
||||
```
|
||||
|
||||
**Right approach - Make failures explicit:**
|
||||
```typescript
|
||||
// Option 1: Button should exist
|
||||
await page.click('#optional-button'); // Fails loudly if missing
|
||||
|
||||
// Option 2: Button might not exist (test both)
|
||||
test('should work with optional button', async ({ page }) => {
|
||||
const hasButton = await page.locator('#optional-button').count() > 0;
|
||||
if (hasButton) {
|
||||
await page.click('#optional-button');
|
||||
}
|
||||
// But now you're testing optional behavior explicitly
|
||||
});
|
||||
```
|
||||
|
||||
### Issue: "Hard waits are easier than network patterns"
|
||||
|
||||
**Short-term:** Hard waits seem simpler
|
||||
**Long-term:** Flaky tests waste more time than learning network patterns
|
||||
|
||||
**Investment:**
|
||||
- 30 minutes to learn network-first patterns
|
||||
- Prevents hundreds of hours debugging flaky tests
|
||||
- Tests run faster (no wasted waits)
|
||||
- Team trusts test suite
|
||||
|
||||
## Technical Implementation
|
||||
|
||||
For detailed test quality patterns, see:
|
||||
- [Test Quality Fragment](/docs/reference/tea/knowledge-base.md#test-quality)
|
||||
- [Test Levels Framework Fragment](/docs/reference/tea/knowledge-base.md#test-levels-framework)
|
||||
- [Complete Knowledge Base Index](/docs/reference/tea/knowledge-base.md)
|
||||
|
||||
## Related Concepts
|
||||
|
||||
**Core TEA Concepts:**
|
||||
- [Risk-Based Testing](/docs/explanation/tea/risk-based-testing.md) - Quality scales with risk
|
||||
- [Knowledge Base System](/docs/explanation/tea/knowledge-base-system.md) - How standards are enforced
|
||||
- [Engagement Models](/docs/explanation/tea/engagement-models.md) - Quality in different models
|
||||
|
||||
**Technical Patterns:**
|
||||
- [Network-First Patterns](/docs/explanation/tea/network-first-patterns.md) - Determinism explained
|
||||
- [Fixture Architecture](/docs/explanation/tea/fixture-architecture.md) - Isolation through fixtures
|
||||
|
||||
**Overview:**
|
||||
- [TEA Overview](/docs/explanation/features/tea-overview.md) - Quality standards in lifecycle
|
||||
- [Testing as Engineering](/docs/explanation/philosophy/testing-as-engineering.md) - Why quality matters
|
||||
|
||||
## Practical Guides
|
||||
|
||||
**Workflow Guides:**
|
||||
- [How to Run Test Review](/docs/how-to/workflows/run-test-review.md) - Audit against these standards
|
||||
- [How to Run ATDD](/docs/how-to/workflows/run-atdd.md) - Generate quality tests
|
||||
- [How to Run Automate](/docs/how-to/workflows/run-automate.md) - Expand with quality
|
||||
|
||||
**Use-Case Guides:**
|
||||
- [Using TEA with Existing Tests](/docs/how-to/brownfield/use-tea-with-existing-tests.md) - Improve legacy quality
|
||||
- [Running TEA for Enterprise](/docs/how-to/enterprise/use-tea-for-enterprise.md) - Enterprise quality thresholds
|
||||
|
||||
## Reference
|
||||
|
||||
- [TEA Command Reference](/docs/reference/tea/commands.md) - *test-review command
|
||||
- [Knowledge Base Index](/docs/reference/tea/knowledge-base.md) - Test quality fragment
|
||||
- [Glossary](/docs/reference/glossary/index.md#test-architect-tea-concepts) - TEA terminology
|
||||
|
||||
---
|
||||
|
||||
Generated with [BMad Method](https://bmad-method.org) - TEA (Test Architect)
|
||||
Reference in New Issue
Block a user