From 3e3c92ed3e80307674a8c38092aa19292db4e32b Mon Sep 17 00:00:00 2001 From: Murat K Ozcan <34237651+muratkeremozcan@users.noreply.github.com> Date: Fri, 9 Jan 2026 12:55:57 -0600 Subject: [PATCH] =?UTF-8?q?docs:=20expand=20TEA=20documentation=20with=20c?= =?UTF-8?q?heat=20sheets,=20MCP=20enhancements,=20a=E2=80=A6=20(#1289)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * docs: expand TEA documentation with cheat sheets, MCP enhancements, and API testing patterns * docs: update TEA fragment counts and fix playwright-utils code examples * docs: addressed PR review concerns * docs: update TEA MCP configuration link to point to documentation site --- docs/explanation/features/tea-overview.md | 295 +++++- src/modules/bmm/agents/tea.agent.yaml | 8 +- src/modules/bmm/module.yaml | 2 +- .../bmm/testarch/knowledge/api-request.md | 171 +++- .../knowledge/api-testing-patterns.md | 843 ++++++++++++++++++ .../bmm/testarch/knowledge/auth-session.md | 198 +++- .../bmm/testarch/knowledge/file-utils.md | 311 +++++-- .../knowledge/intercept-network-call.md | 168 +++- src/modules/bmm/testarch/knowledge/log.md | 213 ++++- .../knowledge/network-error-monitor.md | 321 +++++-- .../testarch/knowledge/network-recorder.md | 384 ++++++-- .../bmm/testarch/knowledge/overview.md | 27 +- src/modules/bmm/testarch/knowledge/recurse.md | 221 ++++- src/modules/bmm/testarch/tea-index.csv | 37 +- 14 files changed, 2833 insertions(+), 366 deletions(-) create mode 100644 src/modules/bmm/testarch/knowledge/api-testing-patterns.md diff --git a/docs/explanation/features/tea-overview.md b/docs/explanation/features/tea-overview.md index 11e4c8f3..cacd25e3 100644 --- a/docs/explanation/features/tea-overview.md +++ b/docs/explanation/features/tea-overview.md @@ -195,20 +195,295 @@ Epic/Release Gate β†’ TEA: *nfr-assess, *trace Phase 2 (release decision) **Note**: `*trace` is a two-phase workflow: Phase 1 (traceability) + Phase 2 (gate decision). This reduces cognitive load while maintaining natural workflow. +### Why TEA Requires Its Own Knowledge Base + +TEA uniquely requires: + +- **Extensive domain knowledge**: 30+ fragments covering test patterns, CI/CD, fixtures, quality practices, and optional playwright-utils integration +- **Cross-cutting concerns**: Domain-specific testing patterns that apply across all BMad projects (vs project-specific artifacts like PRDs/stories) +- **Optional integrations**: MCP capabilities (exploratory, verification) and playwright-utils support + +This architecture enables TEA to maintain consistent, production-ready testing patterns across all BMad projects while operating across multiple development phases. + +--- + + +## High-Level Cheat Sheets + +These cheat sheets map TEA workflows to the **BMad Method and Enterprise tracks** across the **4-Phase Methodology** (Phase 1: Analysis, Phase 2: Planning, Phase 3: Solutioning, Phase 4: Implementation). + +**Note:** Quick Flow projects typically don't require TEA (covered in Overview). These cheat sheets focus on BMad Method and Enterprise tracks where TEA adds value. + +**Legend for Track Deltas:** + +- βž• = New workflow or phase added (doesn't exist in baseline) +- πŸ”„ = Modified focus (same workflow, different emphasis or purpose) +- πŸ“¦ = Additional output or archival requirement + +### Greenfield - BMad Method (Simple/Standard Work) + +**Planning Track:** BMad Method (PRD + Architecture) +**Use Case:** New projects with standard complexity + +| Workflow Stage | Test Architect | Dev / Team | Outputs | +| -------------------------- | ----------------------------------------------------------------- | ----------------------------------------------------------------------------------- | ---------------------------------------------------------- | +| **Phase 1**: Discovery | - | Analyst `*product-brief` (optional) | `product-brief.md` | +| **Phase 2**: Planning | - | PM `*prd` (creates PRD with FRs/NFRs) | PRD with functional/non-functional requirements | +| **Phase 3**: Solutioning | Run `*framework`, `*ci` AFTER architecture and epic creation | Architect `*architecture`, `*create-epics-and-stories`, `*implementation-readiness` | Architecture, epics/stories, test scaffold, CI pipeline | +| **Phase 4**: Sprint Start | - | SM `*sprint-planning` | Sprint status file with all epics and stories | +| **Phase 4**: Epic Planning | Run `*test-design` for THIS epic (per-epic test plan) | Review epic scope | `test-design-epic-N.md` with risk assessment and test plan | +| **Phase 4**: Story Dev | (Optional) `*atdd` before dev, then `*automate` after | SM `*create-story`, DEV implements | Tests, story implementation | +| **Phase 4**: Story Review | Execute `*test-review` (optional), re-run `*trace` | Address recommendations, update code/tests | Quality report, refreshed coverage matrix | +| **Phase 4**: Release Gate | (Optional) `*test-review` for final audit, Run `*trace` (Phase 2) | Confirm Definition of Done, share release notes | Quality audit, Gate YAML + release summary | + +
+Execution Notes + +- Run `*framework` only once per repo or when modern harness support is missing. +- **Phase 3 (Solutioning)**: After architecture is complete, run `*framework` and `*ci` to setup test infrastructure based on architectural decisions. +- **Phase 4 starts**: After solutioning is complete, sprint planning loads all epics. +- **`*test-design` runs per-epic**: At the beginning of working on each epic, run `*test-design` to create a test plan for THAT specific epic/feature. Output: `test-design-epic-N.md`. +- Use `*atdd` before coding when the team can adopt ATDD; share its checklist with the dev agent. +- Post-implementation, keep `*trace` current, expand coverage with `*automate`, optionally review test quality with `*test-review`. For release gate, run `*trace` with Phase 2 enabled to get deployment decision. +- Use `*test-review` after `*atdd` to validate generated tests, after `*automate` to ensure regression quality, or before gate for final audit. +- Clarification: `*test-review` is optional and only audits existing tests; run it after `*atdd` or `*automate` when you want a quality review, not as a required step. +- Clarification: `*atdd` outputs are not auto-consumed; share the ATDD doc/tests with the dev workflow. `*trace` does not run `*atdd`β€”it evaluates existing artifacts for coverage and gate readiness. +- Clarification: `*ci` is a one-time setup; recommended early (Phase 3 or before feature work), but it can be done later if it was skipped. + +
+ +
+Worked Example – β€œNova CRM” Greenfield Feature + +1. **Planning (Phase 2):** Analyst runs `*product-brief`; PM executes `*prd` to produce PRD with FRs/NFRs. +2. **Solutioning (Phase 3):** Architect completes `*architecture` for the new module; `*create-epics-and-stories` generates epics/stories based on architecture; TEA sets up test infrastructure via `*framework` and `*ci` based on architectural decisions; gate check validates planning completeness. +3. **Sprint Start (Phase 4):** Scrum Master runs `*sprint-planning` to load all epics into sprint status. +4. **Epic 1 Planning (Phase 4):** TEA runs `*test-design` to create test plan for Epic 1, producing `test-design-epic-1.md` with risk assessment. +5. **Story Implementation (Phase 4):** For each story in Epic 1, SM generates story via `*create-story`; TEA optionally runs `*atdd`; Dev implements with guidance from failing tests. +6. **Post-Dev (Phase 4):** TEA runs `*automate`, optionally `*test-review` to audit test quality, re-runs `*trace` to refresh coverage. +7. **Release Gate:** TEA runs `*trace` with Phase 2 enabled to generate gate decision. + +
+ +### Brownfield - BMad Method or Enterprise (Simple or Complex) + +**Planning Tracks:** BMad Method or Enterprise Method +**Use Case:** Existing codebases - simple additions (BMad Method) or complex enterprise requirements (Enterprise Method) + +**πŸ”„ Brownfield Deltas from Greenfield:** + +- βž• Documentation (Prerequisite) - Document existing codebase if undocumented +- βž• Phase 2: `*trace` - Baseline existing test coverage before planning +- πŸ”„ Phase 4: `*test-design` - Focus on regression hotspots and brownfield risks +- πŸ”„ Phase 4: Story Review - May include `*nfr-assess` if not done earlier + +| Workflow Stage | Test Architect | Dev / Team | Outputs | +| --------------------------------- | --------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- | ---------------------------------------------------------------------- | +| **Documentation**: Prerequisite βž• | - | Analyst `*document-project` (if undocumented) | Comprehensive project documentation | +| **Phase 1**: Discovery | - | Analyst/PM/Architect rerun planning workflows | Updated planning artifacts in `{output_folder}` | +| **Phase 2**: Planning | Run βž• `*trace` (baseline coverage) | PM `*prd` (creates PRD with FRs/NFRs) | PRD with FRs/NFRs, βž• coverage baseline | +| **Phase 3**: Solutioning | Run `*framework`, `*ci` AFTER architecture and epic creation | Architect `*architecture`, `*create-epics-and-stories`, `*implementation-readiness` | Architecture, epics/stories, test framework, CI pipeline | +| **Phase 4**: Sprint Start | - | SM `*sprint-planning` | Sprint status file with all epics and stories | +| **Phase 4**: Epic Planning | Run `*test-design` for THIS epic πŸ”„ (regression hotspots) | Review epic scope and brownfield risks | `test-design-epic-N.md` with brownfield risk assessment and mitigation | +| **Phase 4**: Story Dev | (Optional) `*atdd` before dev, then `*automate` after | SM `*create-story`, DEV implements | Tests, story implementation | +| **Phase 4**: Story Review | Apply `*test-review` (optional), re-run `*trace`, βž• `*nfr-assess` if needed | Resolve gaps, update docs/tests | Quality report, refreshed coverage matrix, NFR report | +| **Phase 4**: Release Gate | (Optional) `*test-review` for final audit, Run `*trace` (Phase 2) | Capture sign-offs, share release notes | Quality audit, Gate YAML + release summary | + +
+Execution Notes + +- Lead with `*trace` during Planning (Phase 2) to baseline existing test coverage before architecture work begins. +- **Phase 3 (Solutioning)**: After architecture is complete, run `*framework` and `*ci` to modernize test infrastructure. For brownfield, framework may need to integrate with or replace existing test setup. +- **Phase 4 starts**: After solutioning is complete and sprint planning loads all epics. +- **`*test-design` runs per-epic**: At the beginning of working on each epic, run `*test-design` to identify regression hotspots, integration risks, and mitigation strategies for THAT specific epic/feature. Output: `test-design-epic-N.md`. +- Use `*atdd` when stories benefit from ATDD; otherwise proceed to implementation and rely on post-dev automation. +- After development, expand coverage with `*automate`, optionally review test quality with `*test-review`, re-run `*trace` (Phase 2 for gate decision). Run `*nfr-assess` now if non-functional risks weren't addressed earlier. +- Use `*test-review` to validate existing brownfield tests or audit new tests before gate. + +
+ +
+Worked Example – β€œAtlas Payments” Brownfield Story + +1. **Planning (Phase 2):** PM executes `*prd` to create PRD with FRs/NFRs; TEA runs `*trace` to baseline existing coverage. +2. **Solutioning (Phase 3):** Architect triggers `*architecture` capturing legacy payment flows and integration architecture; `*create-epics-and-stories` generates Epic 1 (Payment Processing) based on architecture; TEA sets up `*framework` and `*ci` based on architectural decisions; gate check validates planning. +3. **Sprint Start (Phase 4):** Scrum Master runs `*sprint-planning` to load Epic 1 into sprint status. +4. **Epic 1 Planning (Phase 4):** TEA runs `*test-design` for Epic 1 (Payment Processing), producing `test-design-epic-1.md` that flags settlement edge cases, regression hotspots, and mitigation plans. +5. **Story Implementation (Phase 4):** For each story in Epic 1, SM generates story via `*create-story`; TEA runs `*atdd` producing failing Playwright specs; Dev implements with guidance from tests and checklist. +6. **Post-Dev (Phase 4):** TEA applies `*automate`, optionally `*test-review` to audit test quality, re-runs `*trace` to refresh coverage. +7. **Release Gate:** TEA performs `*nfr-assess` to validate SLAs, runs `*trace` with Phase 2 enabled to generate gate decision (PASS/CONCERNS/FAIL). + +
+ +### Greenfield - Enterprise Method (Enterprise/Compliance Work) + +**Planning Track:** Enterprise Method (BMad Method + extended security/devops/test strategies) +**Use Case:** New enterprise projects with compliance, security, or complex regulatory requirements + +**🏒 Enterprise Deltas from BMad Method:** + +- βž• Phase 1: `*research` - Domain and compliance research (recommended) +- βž• Phase 2: `*nfr-assess` - Capture NFR requirements early (security/performance/reliability) +- πŸ”„ Phase 4: `*test-design` - Enterprise focus (compliance, security architecture alignment) +- πŸ“¦ Release Gate - Archive artifacts and compliance evidence for audits + +| Workflow Stage | Test Architect | Dev / Team | Outputs | +| -------------------------- | ----------------------------------------------------------------------- | ----------------------------------------------------------------------------------- | ------------------------------------------------------------------ | +| **Phase 1**: Discovery | - | Analyst βž• `*research`, `*product-brief` | Domain research, compliance analysis, product brief | +| **Phase 2**: Planning | Run βž• `*nfr-assess` | PM `*prd` (creates PRD with FRs/NFRs), UX `*create-ux-design` | Enterprise PRD with FRs/NFRs, UX design, βž• NFR documentation | +| **Phase 3**: Solutioning | Run `*framework`, `*ci` AFTER architecture and epic creation | Architect `*architecture`, `*create-epics-and-stories`, `*implementation-readiness` | Architecture, epics/stories, test framework, CI pipeline | +| **Phase 4**: Sprint Start | - | SM `*sprint-planning` | Sprint plan with all epics | +| **Phase 4**: Epic Planning | Run `*test-design` for THIS epic πŸ”„ (compliance focus) | Review epic scope and compliance requirements | `test-design-epic-N.md` with security/performance/compliance focus | +| **Phase 4**: Story Dev | (Optional) `*atdd`, `*automate`, `*test-review`, `*trace` per story | SM `*create-story`, DEV implements | Tests, fixtures, quality reports, coverage matrices | +| **Phase 4**: Release Gate | Final `*test-review` audit, Run `*trace` (Phase 2), πŸ“¦ archive artifacts | Capture sign-offs, πŸ“¦ compliance evidence | Quality audit, updated assessments, gate YAML, πŸ“¦ audit trail | + +
+Execution Notes + +- `*nfr-assess` runs early in Planning (Phase 2) to capture compliance, security, and performance requirements upfront. +- **Phase 3 (Solutioning)**: After architecture is complete, run `*framework` and `*ci` with enterprise-grade configurations (selective testing, burn-in jobs, caching, notifications). +- **Phase 4 starts**: After solutioning is complete and sprint planning loads all epics. +- **`*test-design` runs per-epic**: At the beginning of working on each epic, run `*test-design` to create an enterprise-focused test plan for THAT specific epic, ensuring alignment with security architecture, performance targets, and compliance requirements. Output: `test-design-epic-N.md`. +- Use `*atdd` for stories when feasible so acceptance tests can lead implementation. +- Use `*test-review` per story or sprint to maintain quality standards and ensure compliance with testing best practices. +- Prior to release, rerun coverage (`*trace`, `*automate`), perform final quality audit with `*test-review`, and formalize the decision with `*trace` Phase 2 (gate decision); archive artifacts for compliance audits. + +
+ +
+Worked Example – β€œHelios Ledger” Enterprise Release + +1. **Planning (Phase 2):** Analyst runs `*research` and `*product-brief`; PM completes `*prd` creating PRD with FRs/NFRs; TEA runs `*nfr-assess` to establish NFR targets. +2. **Solutioning (Phase 3):** Architect completes `*architecture` with enterprise considerations; `*create-epics-and-stories` generates epics/stories based on architecture; TEA sets up `*framework` and `*ci` with enterprise-grade configurations based on architectural decisions; gate check validates planning completeness. +3. **Sprint Start (Phase 4):** Scrum Master runs `*sprint-planning` to load all epics into sprint status. +4. **Per-Epic (Phase 4):** For each epic, TEA runs `*test-design` to create epic-specific test plan (e.g., `test-design-epic-1.md`, `test-design-epic-2.md`) with compliance-focused risk assessment. +5. **Per-Story (Phase 4):** For each story, TEA uses `*atdd`, `*automate`, `*test-review`, and `*trace`; Dev teams iterate on the findings. +6. **Release Gate:** TEA re-checks coverage, performs final quality audit with `*test-review`, and logs the final gate decision via `*trace` Phase 2, archiving artifacts for compliance. + +
+ --- ## TEA Command Catalog -| Command | Primary Outputs | Notes | -| -------------- | --------------------------------------------------------------------------------------------- | ---------------------------------------------------- | -| `*framework` | Playwright/Cypress scaffold, `.env.example`, `.nvmrc`, sample specs | Use when no production-ready harness exists | -| `*ci` | CI workflow, selective test scripts, secrets checklist | Platform-aware (GitHub Actions default) | -| `*test-design` | Combined risk assessment, mitigation plan, and coverage strategy | Risk scoring + optional exploratory mode | -| `*atdd` | Failing acceptance tests + implementation checklist | TDD red phase + optional recording mode | -| `*automate` | Prioritized specs, fixtures, README/script updates, DoD summary | Optional healing/recording, avoid duplicate coverage | -| `*test-review` | Test quality review report with 0-100 score, violations, fixes | Reviews tests against knowledge base patterns | -| `*nfr-assess` | NFR assessment report with actions | Focus on security/performance/reliability | -| `*trace` | Phase 1: Coverage matrix, recommendations. Phase 2: Gate decision (PASS/CONCERNS/FAIL/WAIVED) | Two-phase workflow: traceability + gate decision | +| Command | Primary Outputs | Notes | With Playwright MCP Enhancements | +| -------------- | --------------------------------------------------------------------------------------------- | ---------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | +| `*framework` | Playwright/Cypress scaffold, `.env.example`, `.nvmrc`, sample specs | Use when no production-ready harness exists | - | +| `*ci` | CI workflow, selective test scripts, secrets checklist | Platform-aware (GitHub Actions default) | - | +| `*test-design` | Combined risk assessment, mitigation plan, and coverage strategy | Risk scoring + optional exploratory mode | **+ Exploratory**: Interactive UI discovery with browser automation (uncover actual functionality) | +| `*atdd` | Failing acceptance tests + implementation checklist | TDD red phase + optional recording mode | **+ Recording**: AI generation verified with live browser (accurate selectors from real DOM) | +| `*automate` | Prioritized specs, fixtures, README/script updates, DoD summary | Optional healing/recording, avoid duplicate coverage | **+ Healing**: Pattern fixes enhanced with visual debugging + **+ Recording**: AI verified with live browser | +| `*test-review` | Test quality review report with 0-100 score, violations, fixes | Reviews tests against knowledge base patterns | - | +| `*nfr-assess` | NFR assessment report with actions | Focus on security/performance/reliability | - | +| `*trace` | Phase 1: Coverage matrix, recommendations. Phase 2: Gate decision (PASS/CONCERNS/FAIL/WAIVED) | Two-phase workflow: traceability + gate decision | - | + +--- + +## Playwright Utils Integration + +TEA optionally integrates with `@seontechnologies/playwright-utils`, an open-source library providing fixture-based utilities for Playwright tests. This integration enhances TEA's test generation and review workflows with production-ready patterns. + +
+Installation & Configuration + +**Package**: `@seontechnologies/playwright-utils` ([npm](https://www.npmjs.com/package/@seontechnologies/playwright-utils) | [GitHub](https://github.com/seontechnologies/playwright-utils)) + +**Install**: `npm install -D @seontechnologies/playwright-utils` + +**Enable during BMAD installation** by answering "Yes" when prompted, or manually set `tea_use_playwright_utils: true` in `_bmad/bmm/config.yaml`. + +**To disable**: Set `tea_use_playwright_utils: false` in `_bmad/bmm/config.yaml`. + +
+ +
+How Playwright Utils Enhances TEA Workflows + +1. `*framework`: + - Default: Basic Playwright scaffold + - **+ playwright-utils**: Scaffold with api-request, network-recorder, auth-session, burn-in, network-error-monitor fixtures pre-configured + + Benefit: Production-ready patterns from day one + +2. `*automate`, `*atdd`: + - Default: Standard test patterns + - **+ playwright-utils**: Tests using api-request (schema validation), intercept-network-call (mocking), recurse (polling), log (structured logging), file-utils (CSV/PDF) + + Benefit: Advanced patterns without boilerplate + +3. `*test-review`: + - Default: Reviews against core knowledge base (22 fragments) + - **+ playwright-utils**: Reviews against expanded knowledge base (33 fragments: 22 core + 11 playwright-utils) + + Benefit: Reviews include fixture composition, auth patterns, network recording best practices + +4. `*ci`: + - Default: Standard CI workflow + - **+ playwright-utils**: CI workflow with burn-in script (smart test selection) and network-error-monitor integration + + Benefit: Faster CI feedback, HTTP error detection + +**Utilities available** (10 total): api-request, network-recorder, auth-session, intercept-network-call, recurse, log, file-utils, burn-in, network-error-monitor, fixtures-composition + +
+ +--- + +## Playwright MCP Enhancements + +TEA can leverage Playwright MCP servers to enhance test generation with live browser verification. MCP provides interactive capabilities on top of TEA's default AI-based approach. + +
+MCP Server Configuration + +**Two Playwright MCP servers** (actively maintained, continuously updated): + +- `playwright` - Browser automation (`npx @playwright/mcp@latest`) +- `playwright-test` - Test runner with failure analysis (`npx playwright run-test-mcp-server`) + +**Config example**: + +```json +{ + "mcpServers": { + "playwright": { + "command": "npx", + "args": ["@playwright/mcp@latest"] + }, + "playwright-test": { + "command": "npx", + "args": ["playwright", "run-test-mcp-server"] + } + } +} +``` + +**To disable**: Set `tea_use_mcp_enhancements: false` in `_bmad/bmm/config.yaml` OR remove MCPs from IDE config. + +
+ +
+How MCP Enhances TEA Workflows + +1. `*test-design`: + - Default: Analysis + documentation + - **+ MCP**: Interactive UI discovery with `browser_navigate`, `browser_click`, `browser_snapshot`, behavior observation + + Benefit: Discover actual functionality, edge cases, undocumented features + +2. `*atdd`, `*automate`: + - Default: Infers selectors and interactions from requirements and knowledge fragments + - **+ MCP**: Generates tests **then** verifies with `generator_setup_page`, `browser_*` tools, validates against live app + + Benefit: Accurate selectors from real DOM, verified behavior, refined test code + +3. `*automate` (healing mode): + - Default: Pattern-based fixes from error messages + knowledge fragments + - **+ MCP**: Pattern fixes **enhanced with** `browser_snapshot`, `browser_console_messages`, `browser_network_requests`, `browser_generate_locator` + + Benefit: Visual failure context, live DOM inspection, root cause discovery + +
--- diff --git a/src/modules/bmm/agents/tea.agent.yaml b/src/modules/bmm/agents/tea.agent.yaml index 2ff25637..fa42b41b 100644 --- a/src/modules/bmm/agents/tea.agent.yaml +++ b/src/modules/bmm/agents/tea.agent.yaml @@ -12,15 +12,17 @@ agent: persona: role: Master Test Architect - identity: Test architect specializing in CI/CD, automated frameworks, and scalable quality gates. + identity: Test architect specializing in API testing, backend services, UI automation, CI/CD pipelines, and scalable quality gates. Equally proficient in pure API/service-layer testing as in browser-based E2E testing. communication_style: "Blends data with gut instinct. 'Strong opinions, weakly held' is their mantra. Speaks in risk calculations and impact assessments." principles: | - Risk-based testing - depth scales with impact - Quality gates backed by data - - Tests mirror usage patterns + - Tests mirror usage patterns (API, UI, or both) - Flakiness is critical technical debt - Tests first AI implements suite validates - Calculate risk vs value for every testing decision + - Prefer lower test levels (unit > integration > E2E) when possible + - API tests are first-class citizens, not just UI support critical_actions: - "Consult {project-root}/_bmad/bmm/testarch/tea-index.csv to select knowledge fragments under knowledge/ and load only the files needed for the current task" @@ -39,7 +41,7 @@ agent: - trigger: AT or fuzzy match on atdd workflow: "{project-root}/_bmad/bmm/workflows/testarch/atdd/workflow.yaml" - description: "[AT] Generate E2E tests first, before starting implementation" + description: "[AT] Generate API and/or E2E tests first, before starting implementation" - trigger: TA or fuzzy match on test-automate workflow: "{project-root}/_bmad/bmm/workflows/testarch/automate/workflow.yaml" diff --git a/src/modules/bmm/module.yaml b/src/modules/bmm/module.yaml index e6b8dee1..b5aaf889 100644 --- a/src/modules/bmm/module.yaml +++ b/src/modules/bmm/module.yaml @@ -45,7 +45,7 @@ project_knowledge: # Artifacts from research, document-project output, other lon result: "{project-root}/{value}" tea_use_mcp_enhancements: - prompt: "Test Architect Playwright MCP capabilities (healing, exploratory, verification) are optionally available.\nYou will have to setup your MCPs yourself; refer to test-architecture.md for hints.\nWould you like to enable MCP enhancements in Test Architect?" + prompt: "Test Architect Playwright MCP capabilities (healing, exploratory, verification) are optionally available.\nYou will have to setup your MCPs yourself; refer to https://docs.bmad-method.org/explanation/features/tea-overview for configuration examples.\nWould you like to enable MCP enhancements in Test Architect?" default: false result: "{value}" diff --git a/src/modules/bmm/testarch/knowledge/api-request.md b/src/modules/bmm/testarch/knowledge/api-request.md index b47bfc4f..d2b36cde 100644 --- a/src/modules/bmm/testarch/knowledge/api-request.md +++ b/src/modules/bmm/testarch/knowledge/api-request.md @@ -2,7 +2,7 @@ ## Principle -Use typed HTTP client with built-in schema validation and automatic retry for server errors. The utility handles URL resolution, header management, response parsing, and single-line response validation with proper TypeScript support. +Use typed HTTP client with built-in schema validation and automatic retry for server errors. The utility handles URL resolution, header management, response parsing, and single-line response validation with proper TypeScript support. **Works without a browser** - ideal for pure API/service testing. ## Rationale @@ -21,6 +21,7 @@ The `apiRequest` utility provides: - **Schema validation**: Single-line validation (JSON Schema, Zod, OpenAPI) - **URL resolution**: Four-tier strategy (explicit > config > Playwright > direct) - **TypeScript generics**: Type-safe response bodies +- **No browser required**: Pure API testing without browser overhead ## Pattern Examples @@ -60,10 +61,11 @@ test('should fetch user data', async ({ apiRequest }) => { ```typescript import { test } from '@seontechnologies/playwright-utils/api-request/fixtures'; +import { z } from 'zod'; -test('should validate response schema', async ({ apiRequest }) => { - // JSON Schema validation - const response = await apiRequest({ +// JSON Schema validation +test('should validate response schema (JSON Schema)', async ({ apiRequest }) => { + const { status, body } = await apiRequest({ method: 'GET', path: '/api/users/123', validateSchema: { @@ -77,22 +79,25 @@ test('should validate response schema', async ({ apiRequest }) => { }, }); // Throws if schema validation fails + expect(status).toBe(200); +}); - // Zod schema validation - import { z } from 'zod'; +// Zod schema validation +const UserSchema = z.object({ + id: z.string(), + name: z.string(), + email: z.string().email(), +}); - const UserSchema = z.object({ - id: z.string(), - name: z.string(), - email: z.string().email(), - }); - - const response = await apiRequest({ +test('should validate response schema (Zod)', async ({ apiRequest }) => { + const { status, body } = await apiRequest({ method: 'GET', path: '/api/users/123', validateSchema: UserSchema, }); // Response body is type-safe AND validated + expect(status).toBe(200); + expect(body.email).toContain('@'); }); ``` @@ -236,6 +241,136 @@ test('should poll until job completes', async ({ apiRequest, recurse }) => { - `recurse` polls until predicate returns true - Composable utilities work together seamlessly +### Example 6: Microservice Testing (Multiple Services) + +**Context**: Test interactions between microservices without a browser. + +**Implementation**: + +```typescript +import { test, expect } from '@seontechnologies/playwright-utils/fixtures'; + +const USER_SERVICE = process.env.USER_SERVICE_URL || 'http://localhost:3001'; +const ORDER_SERVICE = process.env.ORDER_SERVICE_URL || 'http://localhost:3002'; + +test.describe('Microservice Integration', () => { + test('should validate cross-service user lookup', async ({ apiRequest }) => { + // Create user in user-service + const { body: user } = await apiRequest({ + method: 'POST', + path: '/api/users', + baseUrl: USER_SERVICE, + body: { name: 'Test User', email: 'test@example.com' }, + }); + + // Create order in order-service (validates user via user-service) + const { status, body: order } = await apiRequest({ + method: 'POST', + path: '/api/orders', + baseUrl: ORDER_SERVICE, + body: { + userId: user.id, + items: [{ productId: 'prod-1', quantity: 2 }], + }, + }); + + expect(status).toBe(201); + expect(order.userId).toBe(user.id); + }); + + test('should reject order for invalid user', async ({ apiRequest }) => { + const { status, body } = await apiRequest({ + method: 'POST', + path: '/api/orders', + baseUrl: ORDER_SERVICE, + body: { + userId: 'non-existent-user', + items: [{ productId: 'prod-1', quantity: 1 }], + }, + }); + + expect(status).toBe(400); + expect(body.code).toBe('INVALID_USER'); + }); +}); +``` + +**Key Points**: + +- Test multiple services without browser +- Use `baseUrl` to target different services +- Validate cross-service communication +- Pure API testing - fast and reliable + +### Example 7: GraphQL API Testing + +**Context**: Test GraphQL endpoints with queries and mutations. + +**Implementation**: + +```typescript +test.describe('GraphQL API', () => { + const GRAPHQL_ENDPOINT = '/graphql'; + + test('should query users via GraphQL', async ({ apiRequest }) => { + const query = ` + query GetUsers($limit: Int) { + users(limit: $limit) { + id + name + email + } + } + `; + + const { status, body } = await apiRequest({ + method: 'POST', + path: GRAPHQL_ENDPOINT, + body: { + query, + variables: { limit: 10 }, + }, + }); + + expect(status).toBe(200); + expect(body.errors).toBeUndefined(); + expect(body.data.users).toHaveLength(10); + }); + + test('should create user via mutation', async ({ apiRequest }) => { + const mutation = ` + mutation CreateUser($input: CreateUserInput!) { + createUser(input: $input) { + id + name + } + } + `; + + const { status, body } = await apiRequest({ + method: 'POST', + path: GRAPHQL_ENDPOINT, + body: { + query: mutation, + variables: { + input: { name: 'GraphQL User', email: 'gql@example.com' }, + }, + }, + }); + + expect(status).toBe(200); + expect(body.data.createUser.id).toBeDefined(); + }); +}); +``` + +**Key Points**: + +- GraphQL via POST request +- Variables in request body +- Check `body.errors` for GraphQL errors (not status code) +- Works for queries and mutations + ## Comparison with Vanilla Playwright | Vanilla Playwright | playwright-utils apiRequest | @@ -251,11 +386,13 @@ test('should poll until job completes', async ({ apiRequest, recurse }) => { **Use apiRequest for:** -- βœ… API endpoint testing -- βœ… Background API calls in UI tests +- βœ… Pure API/service testing (no browser needed) +- βœ… Microservice integration testing +- βœ… GraphQL API testing - βœ… Schema validation needs - βœ… Tests requiring retry logic -- βœ… Typed API responses +- βœ… Background API calls in UI tests +- βœ… Contract testing support **Stick with vanilla Playwright for:** @@ -265,11 +402,13 @@ test('should poll until job completes', async ({ apiRequest, recurse }) => { ## Related Fragments +- `api-testing-patterns.md` - Comprehensive pure API testing patterns - `overview.md` - Installation and design principles - `auth-session.md` - Authentication token management - `recurse.md` - Polling for async operations - `fixtures-composition.md` - Combining utilities with mergeTests - `log.md` - Logging API requests +- `contract-testing.md` - Pact contract testing ## Anti-Patterns diff --git a/src/modules/bmm/testarch/knowledge/api-testing-patterns.md b/src/modules/bmm/testarch/knowledge/api-testing-patterns.md new file mode 100644 index 00000000..65c81d7a --- /dev/null +++ b/src/modules/bmm/testarch/knowledge/api-testing-patterns.md @@ -0,0 +1,843 @@ +# API Testing Patterns + +## Principle + +Test APIs and backend services directly without browser overhead. Use Playwright's `request` context for HTTP operations, `apiRequest` utility for enhanced features, and `recurse` for async operations. Pure API tests run faster, are more stable, and provide better coverage for service-layer logic. + +## Rationale + +Many teams over-rely on E2E/browser tests when API tests would be more appropriate: + +- **Slower feedback**: Browser tests take seconds, API tests take milliseconds +- **More brittle**: UI changes break tests even when API works correctly +- **Wrong abstraction**: Testing business logic through UI layers adds noise +- **Resource heavy**: Browsers consume memory and CPU + +API-first testing provides: + +- **Fast execution**: No browser startup, no rendering, no JavaScript execution +- **Direct validation**: Test exactly what the service returns +- **Better isolation**: Test service logic independent of UI +- **Easier debugging**: Clear request/response without DOM noise +- **Contract validation**: Verify API contracts explicitly + +## When to Use API Tests vs E2E Tests + +| Scenario | API Test | E2E Test | +|----------|----------|----------| +| CRUD operations | βœ… Primary | ❌ Overkill | +| Business logic validation | βœ… Primary | ❌ Overkill | +| Error handling (4xx, 5xx) | βœ… Primary | ⚠️ Supplement | +| Authentication flows | βœ… Primary | ⚠️ Supplement | +| Data transformation | βœ… Primary | ❌ Overkill | +| User journeys | ❌ Can't test | βœ… Primary | +| Visual regression | ❌ Can't test | βœ… Primary | +| Cross-browser issues | ❌ Can't test | βœ… Primary | + +**Rule of thumb**: If you're testing what the server returns (not how it looks), use API tests. + +## Pattern Examples + +### Example 1: Pure API Test (No Browser) + +**Context**: Test REST API endpoints directly without any browser context. + +**Implementation**: + +```typescript +// tests/api/users.spec.ts +import { test, expect } from '@playwright/test'; + +// No page, no browser - just API +test.describe('Users API', () => { + test('should create user', async ({ request }) => { + const response = await request.post('/api/users', { + data: { + name: 'John Doe', + email: 'john@example.com', + role: 'user', + }, + }); + + expect(response.status()).toBe(201); + + const user = await response.json(); + expect(user.id).toBeDefined(); + expect(user.name).toBe('John Doe'); + expect(user.email).toBe('john@example.com'); + }); + + test('should get user by ID', async ({ request }) => { + // Create user first + const createResponse = await request.post('/api/users', { + data: { name: 'Jane Doe', email: 'jane@example.com' }, + }); + const { id } = await createResponse.json(); + + // Get user + const getResponse = await request.get(`/api/users/${id}`); + expect(getResponse.status()).toBe(200); + + const user = await getResponse.json(); + expect(user.id).toBe(id); + expect(user.name).toBe('Jane Doe'); + }); + + test('should return 404 for non-existent user', async ({ request }) => { + const response = await request.get('/api/users/non-existent-id'); + expect(response.status()).toBe(404); + + const error = await response.json(); + expect(error.code).toBe('USER_NOT_FOUND'); + }); + + test('should validate required fields', async ({ request }) => { + const response = await request.post('/api/users', { + data: { name: 'Missing Email' }, // email is required + }); + + expect(response.status()).toBe(400); + + const error = await response.json(); + expect(error.code).toBe('VALIDATION_ERROR'); + expect(error.details).toContainEqual( + expect.objectContaining({ field: 'email', message: expect.any(String) }) + ); + }); +}); +``` + +**Key Points**: + +- No `page` fixture needed - only `request` +- Tests run without browser overhead +- Direct HTTP assertions +- Clear error handling tests + +### Example 2: API Test with apiRequest Utility + +**Context**: Use enhanced apiRequest for schema validation, retry, and type safety. + +**Implementation**: + +```typescript +// tests/api/orders.spec.ts +import { test, expect } from '@seontechnologies/playwright-utils/api-request/fixtures'; +import { z } from 'zod'; + +// Define schema for type safety and validation +const OrderSchema = z.object({ + id: z.string().uuid(), + userId: z.string(), + items: z.array( + z.object({ + productId: z.string(), + quantity: z.number().positive(), + price: z.number().positive(), + }) + ), + total: z.number().positive(), + status: z.enum(['pending', 'processing', 'shipped', 'delivered']), + createdAt: z.string().datetime(), +}); + +type Order = z.infer; + +test.describe('Orders API', () => { + test('should create order with schema validation', async ({ apiRequest }) => { + const { status, body } = await apiRequest({ + method: 'POST', + path: '/api/orders', + body: { + userId: 'user-123', + items: [ + { productId: 'prod-1', quantity: 2, price: 29.99 }, + { productId: 'prod-2', quantity: 1, price: 49.99 }, + ], + }, + validateSchema: OrderSchema, // Validates response matches schema + }); + + expect(status).toBe(201); + expect(body.id).toBeDefined(); + expect(body.status).toBe('pending'); + expect(body.total).toBe(109.97); // 2*29.99 + 49.99 + }); + + test('should handle server errors with retry', async ({ apiRequest }) => { + // apiRequest retries 5xx errors by default + const { status, body } = await apiRequest({ + method: 'GET', + path: '/api/orders/order-123', + retryConfig: { + maxRetries: 3, + retryDelay: 1000, + }, + }); + + expect(status).toBe(200); + }); + + test('should list orders with pagination', async ({ apiRequest }) => { + const { status, body } = await apiRequest<{ orders: Order[]; total: number; page: number }>({ + method: 'GET', + path: '/api/orders', + params: { page: 1, limit: 10, status: 'pending' }, + }); + + expect(status).toBe(200); + expect(body.orders).toHaveLength(10); + expect(body.total).toBeGreaterThan(10); + expect(body.page).toBe(1); + }); +}); +``` + +**Key Points**: + +- Zod schema for runtime validation AND TypeScript types +- `validateSchema` throws if response doesn't match +- Built-in retry for transient failures +- Type-safe `body` access + +### Example 3: Microservice-to-Microservice Testing + +**Context**: Test service interactions without browser - validate API contracts between services. + +**Implementation**: + +```typescript +// tests/api/service-integration.spec.ts +import { test, expect } from '@seontechnologies/playwright-utils/fixtures'; + +test.describe('Service Integration', () => { + const USER_SERVICE_URL = process.env.USER_SERVICE_URL || 'http://localhost:3001'; + const ORDER_SERVICE_URL = process.env.ORDER_SERVICE_URL || 'http://localhost:3002'; + const INVENTORY_SERVICE_URL = process.env.INVENTORY_SERVICE_URL || 'http://localhost:3003'; + + test('order service should validate user exists', async ({ apiRequest }) => { + // Create user in user-service + const { body: user } = await apiRequest({ + method: 'POST', + path: '/api/users', + baseUrl: USER_SERVICE_URL, + body: { name: 'Test User', email: 'test@example.com' }, + }); + + // Create order in order-service (should validate user via user-service) + const { status, body: order } = await apiRequest({ + method: 'POST', + path: '/api/orders', + baseUrl: ORDER_SERVICE_URL, + body: { + userId: user.id, + items: [{ productId: 'prod-1', quantity: 1 }], + }, + }); + + expect(status).toBe(201); + expect(order.userId).toBe(user.id); + }); + + test('order service should reject invalid user', async ({ apiRequest }) => { + const { status, body } = await apiRequest({ + method: 'POST', + path: '/api/orders', + baseUrl: ORDER_SERVICE_URL, + body: { + userId: 'non-existent-user', + items: [{ productId: 'prod-1', quantity: 1 }], + }, + }); + + expect(status).toBe(400); + expect(body.code).toBe('INVALID_USER'); + }); + + test('order should decrease inventory', async ({ apiRequest, recurse }) => { + // Get initial inventory + const { body: initialInventory } = await apiRequest({ + method: 'GET', + path: '/api/inventory/prod-1', + baseUrl: INVENTORY_SERVICE_URL, + }); + + // Create order + await apiRequest({ + method: 'POST', + path: '/api/orders', + baseUrl: ORDER_SERVICE_URL, + body: { + userId: 'user-123', + items: [{ productId: 'prod-1', quantity: 2 }], + }, + }); + + // Poll for inventory update (eventual consistency) + const { body: updatedInventory } = await recurse( + () => + apiRequest({ + method: 'GET', + path: '/api/inventory/prod-1', + baseUrl: INVENTORY_SERVICE_URL, + }), + (response) => response.body.quantity === initialInventory.quantity - 2, + { timeout: 10000, interval: 500 } + ); + + expect(updatedInventory.quantity).toBe(initialInventory.quantity - 2); + }); +}); +``` + +**Key Points**: + +- Multiple service URLs for microservice testing +- Tests service-to-service communication +- Uses `recurse` for eventual consistency +- No browser needed for full integration testing + +### Example 4: GraphQL API Testing + +**Context**: Test GraphQL endpoints with queries and mutations. + +**Implementation**: + +```typescript +// tests/api/graphql.spec.ts +import { test, expect } from '@seontechnologies/playwright-utils/api-request/fixtures'; + +const GRAPHQL_ENDPOINT = '/graphql'; + +test.describe('GraphQL API', () => { + test('should query users', async ({ apiRequest }) => { + const query = ` + query GetUsers($limit: Int) { + users(limit: $limit) { + id + name + email + role + } + } + `; + + const { status, body } = await apiRequest({ + method: 'POST', + path: GRAPHQL_ENDPOINT, + body: { + query, + variables: { limit: 10 }, + }, + }); + + expect(status).toBe(200); + expect(body.errors).toBeUndefined(); + expect(body.data.users).toHaveLength(10); + expect(body.data.users[0]).toHaveProperty('id'); + expect(body.data.users[0]).toHaveProperty('name'); + }); + + test('should create user via mutation', async ({ apiRequest }) => { + const mutation = ` + mutation CreateUser($input: CreateUserInput!) { + createUser(input: $input) { + id + name + email + } + } + `; + + const { status, body } = await apiRequest({ + method: 'POST', + path: GRAPHQL_ENDPOINT, + body: { + query: mutation, + variables: { + input: { + name: 'GraphQL User', + email: 'graphql@example.com', + }, + }, + }, + }); + + expect(status).toBe(200); + expect(body.errors).toBeUndefined(); + expect(body.data.createUser.id).toBeDefined(); + expect(body.data.createUser.name).toBe('GraphQL User'); + }); + + test('should handle GraphQL errors', async ({ apiRequest }) => { + const query = ` + query GetUser($id: ID!) { + user(id: $id) { + id + name + } + } + `; + + const { status, body } = await apiRequest({ + method: 'POST', + path: GRAPHQL_ENDPOINT, + body: { + query, + variables: { id: 'non-existent' }, + }, + }); + + expect(status).toBe(200); // GraphQL returns 200 even for errors + expect(body.errors).toBeDefined(); + expect(body.errors[0].message).toContain('not found'); + expect(body.data.user).toBeNull(); + }); + + test('should handle validation errors', async ({ apiRequest }) => { + const mutation = ` + mutation CreateUser($input: CreateUserInput!) { + createUser(input: $input) { + id + } + } + `; + + const { status, body } = await apiRequest({ + method: 'POST', + path: GRAPHQL_ENDPOINT, + body: { + query: mutation, + variables: { + input: { + name: '', // Invalid: empty name + email: 'invalid-email', // Invalid: bad format + }, + }, + }, + }); + + expect(status).toBe(200); + expect(body.errors).toBeDefined(); + expect(body.errors[0].extensions.code).toBe('BAD_USER_INPUT'); + }); +}); +``` + +**Key Points**: + +- GraphQL queries and mutations via POST +- Variables passed in request body +- GraphQL returns 200 even for errors (check `body.errors`) +- Test validation and business logic errors + +### Example 5: Database Seeding and Cleanup via API + +**Context**: Use API calls to set up and tear down test data without direct database access. + +**Implementation**: + +```typescript +// tests/api/with-data-setup.spec.ts +import { test, expect } from '@seontechnologies/playwright-utils/fixtures'; + +test.describe('Orders with Data Setup', () => { + let testUser: { id: string; email: string }; + let testProducts: Array<{ id: string; name: string; price: number }>; + + test.beforeAll(async ({ request }) => { + // Seed user via API + const userResponse = await request.post('/api/users', { + data: { + name: 'Test User', + email: `test-${Date.now()}@example.com`, + }, + }); + testUser = await userResponse.json(); + + // Seed products via API + testProducts = []; + for (const product of [ + { name: 'Widget A', price: 29.99 }, + { name: 'Widget B', price: 49.99 }, + { name: 'Widget C', price: 99.99 }, + ]) { + const productResponse = await request.post('/api/products', { + data: product, + }); + testProducts.push(await productResponse.json()); + } + }); + + test.afterAll(async ({ request }) => { + // Cleanup via API + if (testUser?.id) { + await request.delete(`/api/users/${testUser.id}`); + } + for (const product of testProducts) { + await request.delete(`/api/products/${product.id}`); + } + }); + + test('should create order with seeded data', async ({ apiRequest }) => { + const { status, body } = await apiRequest({ + method: 'POST', + path: '/api/orders', + body: { + userId: testUser.id, + items: [ + { productId: testProducts[0].id, quantity: 2 }, + { productId: testProducts[1].id, quantity: 1 }, + ], + }, + }); + + expect(status).toBe(201); + expect(body.userId).toBe(testUser.id); + expect(body.items).toHaveLength(2); + expect(body.total).toBe(2 * 29.99 + 49.99); + }); + + test('should list user orders', async ({ apiRequest }) => { + // Create an order first + await apiRequest({ + method: 'POST', + path: '/api/orders', + body: { + userId: testUser.id, + items: [{ productId: testProducts[2].id, quantity: 1 }], + }, + }); + + // List orders for user + const { status, body } = await apiRequest({ + method: 'GET', + path: '/api/orders', + params: { userId: testUser.id }, + }); + + expect(status).toBe(200); + expect(body.orders.length).toBeGreaterThanOrEqual(1); + expect(body.orders.every((o: any) => o.userId === testUser.id)).toBe(true); + }); +}); +``` + +**Key Points**: + +- `beforeAll`/`afterAll` for test data setup/cleanup +- API-based seeding (no direct DB access needed) +- Unique emails to prevent conflicts in parallel runs +- Cleanup after all tests complete + +### Example 6: Background Job Testing with Recurse + +**Context**: Test async operations like background jobs, webhooks, and eventual consistency. + +**Implementation**: + +```typescript +// tests/api/background-jobs.spec.ts +import { test, expect } from '@seontechnologies/playwright-utils/fixtures'; + +test.describe('Background Jobs', () => { + test('should process export job', async ({ apiRequest, recurse }) => { + // Trigger export job + const { body: job } = await apiRequest({ + method: 'POST', + path: '/api/exports', + body: { + type: 'users', + format: 'csv', + filters: { createdAfter: '2024-01-01' }, + }, + }); + + expect(job.id).toBeDefined(); + expect(job.status).toBe('pending'); + + // Poll until job completes + const { body: completedJob } = await recurse( + () => apiRequest({ method: 'GET', path: `/api/exports/${job.id}` }), + (response) => response.body.status === 'completed', + { + timeout: 60000, + interval: 2000, + log: `Waiting for export job ${job.id} to complete`, + } + ); + + expect(completedJob.status).toBe('completed'); + expect(completedJob.downloadUrl).toBeDefined(); + expect(completedJob.recordCount).toBeGreaterThan(0); + }); + + test('should handle job failure gracefully', async ({ apiRequest, recurse }) => { + // Trigger job that will fail + const { body: job } = await apiRequest({ + method: 'POST', + path: '/api/exports', + body: { + type: 'invalid-type', // This will cause failure + format: 'csv', + }, + }); + + // Poll until job fails + const { body: failedJob } = await recurse( + () => apiRequest({ method: 'GET', path: `/api/exports/${job.id}` }), + (response) => ['completed', 'failed'].includes(response.body.status), + { timeout: 30000 } + ); + + expect(failedJob.status).toBe('failed'); + expect(failedJob.error).toBeDefined(); + expect(failedJob.error.code).toBe('INVALID_EXPORT_TYPE'); + }); + + test('should process webhook delivery', async ({ apiRequest, recurse }) => { + // Trigger action that sends webhook + const { body: order } = await apiRequest({ + method: 'POST', + path: '/api/orders', + body: { + userId: 'user-123', + items: [{ productId: 'prod-1', quantity: 1 }], + webhookUrl: 'https://webhook.site/test-endpoint', + }, + }); + + // Poll for webhook delivery status + const { body: webhookStatus } = await recurse( + () => apiRequest({ method: 'GET', path: `/api/webhooks/order/${order.id}` }), + (response) => response.body.delivered === true, + { timeout: 30000, interval: 1000 } + ); + + expect(webhookStatus.delivered).toBe(true); + expect(webhookStatus.deliveredAt).toBeDefined(); + expect(webhookStatus.responseStatus).toBe(200); + }); +}); +``` + +**Key Points**: + +- `recurse` for polling async operations +- Test both success and failure scenarios +- Configurable timeout and interval +- Log messages for debugging + +### Example 7: Service Authentication (No Browser) + +**Context**: Test authenticated API endpoints using tokens directly - no browser login needed. + +**Implementation**: + +```typescript +// tests/api/authenticated.spec.ts +import { test, expect } from '@seontechnologies/playwright-utils/fixtures'; + +test.describe('Authenticated API Tests', () => { + let authToken: string; + + test.beforeAll(async ({ request }) => { + // Get token via API (no browser!) + const response = await request.post('/api/auth/login', { + data: { + email: process.env.TEST_USER_EMAIL, + password: process.env.TEST_USER_PASSWORD, + }, + }); + + const { token } = await response.json(); + authToken = token; + }); + + test('should access protected endpoint with token', async ({ apiRequest }) => { + const { status, body } = await apiRequest({ + method: 'GET', + path: '/api/me', + headers: { + Authorization: `Bearer ${authToken}`, + }, + }); + + expect(status).toBe(200); + expect(body.email).toBe(process.env.TEST_USER_EMAIL); + }); + + test('should reject request without token', async ({ apiRequest }) => { + const { status, body } = await apiRequest({ + method: 'GET', + path: '/api/me', + // No Authorization header + }); + + expect(status).toBe(401); + expect(body.code).toBe('UNAUTHORIZED'); + }); + + test('should reject expired token', async ({ apiRequest }) => { + const expiredToken = 'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...'; // Expired token + + const { status, body } = await apiRequest({ + method: 'GET', + path: '/api/me', + headers: { + Authorization: `Bearer ${expiredToken}`, + }, + }); + + expect(status).toBe(401); + expect(body.code).toBe('TOKEN_EXPIRED'); + }); + + test('should handle role-based access', async ({ apiRequest }) => { + // User token (non-admin) + const { status } = await apiRequest({ + method: 'GET', + path: '/api/admin/users', + headers: { + Authorization: `Bearer ${authToken}`, + }, + }); + + expect(status).toBe(403); // Forbidden for non-admin + }); +}); +``` + +**Key Points**: + +- Token obtained via API login (no browser) +- Token reused across all tests in describe block +- Test auth, expired tokens, and RBAC +- Pure API testing without UI + +## API Test Configuration + +### Playwright Config for API-Only Tests + +```typescript +// playwright.config.ts +import { defineConfig } from '@playwright/test'; + +export default defineConfig({ + testDir: './tests/api', + + // No browser needed for API tests + use: { + baseURL: process.env.API_URL || 'http://localhost:3000', + extraHTTPHeaders: { + 'Accept': 'application/json', + 'Content-Type': 'application/json', + }, + }, + + // Faster without browser overhead + timeout: 30000, + + // Run API tests in parallel + workers: 4, + fullyParallel: true, + + // No screenshots/traces needed for API tests + reporter: [['html'], ['json', { outputFile: 'api-test-results.json' }]], +}); +``` + +### Separate API Test Project + +```typescript +// playwright.config.ts +export default defineConfig({ + projects: [ + { + name: 'api', + testDir: './tests/api', + use: { + baseURL: process.env.API_URL, + }, + }, + { + name: 'e2e', + testDir: './tests/e2e', + use: { + baseURL: process.env.APP_URL, + ...devices['Desktop Chrome'], + }, + }, + ], +}); +``` + +## Comparison: API Tests vs E2E Tests + +| Aspect | API Test | E2E Test | +|--------|----------|----------| +| **Speed** | ~50-100ms per test | ~2-10s per test | +| **Stability** | Very stable | More flaky (UI timing) | +| **Setup** | Minimal | Browser, context, page | +| **Debugging** | Clear request/response | DOM, screenshots, traces | +| **Coverage** | Service logic | User experience | +| **Parallelization** | Easy (stateless) | Complex (browser resources) | +| **CI Cost** | Low (no browser) | High (browser containers) | + +## Related Fragments + +- `api-request.md` - apiRequest utility details +- `recurse.md` - Polling patterns for async operations +- `auth-session.md` - Token management +- `contract-testing.md` - Pact contract testing +- `test-levels-framework.md` - When to use which test level +- `data-factories.md` - Test data setup patterns + +## Anti-Patterns + +**DON'T use E2E for API validation:** + +```typescript +// Bad: Testing API through UI +test('validate user creation', async ({ page }) => { + await page.goto('/admin/users'); + await page.fill('#name', 'John'); + await page.click('#submit'); + await expect(page.getByText('User created')).toBeVisible(); +}); +``` + +**DO test APIs directly:** + +```typescript +// Good: Direct API test +test('validate user creation', async ({ apiRequest }) => { + const { status, body } = await apiRequest({ + method: 'POST', + path: '/api/users', + body: { name: 'John' }, + }); + expect(status).toBe(201); + expect(body.id).toBeDefined(); +}); +``` + +**DON'T ignore API tests because "E2E covers it":** + +```typescript +// Bad thinking: "Our E2E tests create users, so API is tested" +// Reality: E2E tests one happy path; API tests cover edge cases +``` + +**DO have dedicated API test coverage:** + +```typescript +// Good: Explicit API test suite +test.describe('Users API', () => { + test('creates user', async ({ apiRequest }) => { /* ... */ }); + test('handles duplicate email', async ({ apiRequest }) => { /* ... */ }); + test('validates required fields', async ({ apiRequest }) => { /* ... */ }); + test('handles malformed JSON', async ({ apiRequest }) => { /* ... */ }); + test('rate limits requests', async ({ apiRequest }) => { /* ... */ }); +}); +``` diff --git a/src/modules/bmm/testarch/knowledge/auth-session.md b/src/modules/bmm/testarch/knowledge/auth-session.md index 3aa456af..e290476b 100644 --- a/src/modules/bmm/testarch/knowledge/auth-session.md +++ b/src/modules/bmm/testarch/knowledge/auth-session.md @@ -2,7 +2,7 @@ ## Principle -Persist authentication tokens to disk and reuse across test runs. Support multiple user identifiers, ephemeral authentication, and worker-specific accounts for parallel execution. Fetch tokens once, use everywhere. +Persist authentication tokens to disk and reuse across test runs. Support multiple user identifiers, ephemeral authentication, and worker-specific accounts for parallel execution. Fetch tokens once, use everywhere. **Works for both API-only tests and browser tests.** ## Rationale @@ -22,6 +22,7 @@ The `auth-session` utility provides: - **Worker-specific accounts**: Parallel execution with isolated user accounts - **Automatic token management**: Checks validity, renews if expired - **Flexible provider pattern**: Adapt to any auth system (OAuth2, JWT, custom) +- **API-first design**: Get tokens for API tests without browser overhead ## Pattern Examples @@ -244,6 +245,200 @@ test('parallel test 2', async ({ page }) => { - Token management automatic per worker - Scales to any number of workers +### Example 6: Pure API Authentication (No Browser) + +**Context**: Get auth tokens for API-only tests using auth-session disk persistence. + +**Implementation**: + +```typescript +// Step 1: Create API-only auth provider (no browser needed) +// playwright/support/api-auth-provider.ts +import { type AuthProvider } from '@seontechnologies/playwright-utils/auth-session'; + +const apiAuthProvider: AuthProvider = { + getEnvironment: (options) => options.environment || 'local', + getUserIdentifier: (options) => options.userIdentifier || 'api-user', + + extractToken: (storageState) => { + // Token stored in localStorage format for disk persistence + const tokenEntry = storageState.origins?.[0]?.localStorage?.find( + (item) => item.name === 'auth_token' + ); + return tokenEntry?.value; + }, + + isTokenExpired: (storageState) => { + const expiryEntry = storageState.origins?.[0]?.localStorage?.find( + (item) => item.name === 'token_expiry' + ); + if (!expiryEntry) return true; + return Date.now() > parseInt(expiryEntry.value, 10); + }, + + manageAuthToken: async (request, options) => { + const email = process.env.TEST_USER_EMAIL; + const password = process.env.TEST_USER_PASSWORD; + + if (!email || !password) { + throw new Error('TEST_USER_EMAIL and TEST_USER_PASSWORD must be set'); + } + + // Pure API login - no browser! + const response = await request.post('/api/auth/login', { + data: { email, password }, + }); + + if (!response.ok()) { + throw new Error(`Auth failed: ${response.status()}`); + } + + const { token, expiresIn } = await response.json(); + const expiryTime = Date.now() + expiresIn * 1000; + + // Return storage state format for disk persistence + return { + cookies: [], + origins: [ + { + origin: process.env.API_BASE_URL || 'http://localhost:3000', + localStorage: [ + { name: 'auth_token', value: token }, + { name: 'token_expiry', value: String(expiryTime) }, + ], + }, + ], + }; + }, +}; + +export default apiAuthProvider; + +// Step 2: Create auth fixture +// playwright/support/fixtures.ts +import { test as base } from '@playwright/test'; +import { createAuthFixtures, setAuthProvider } from '@seontechnologies/playwright-utils/auth-session'; +import apiAuthProvider from './api-auth-provider'; + +setAuthProvider(apiAuthProvider); + +export const test = base.extend(createAuthFixtures()); + +// Step 3: Use in tests - token persisted to disk! +// tests/api/authenticated-api.spec.ts +import { test } from '../support/fixtures'; +import { expect } from '@playwright/test'; + +test('should access protected endpoint', async ({ authToken, apiRequest }) => { + // authToken is automatically loaded from disk or fetched if expired + const { status, body } = await apiRequest({ + method: 'GET', + path: '/api/me', + headers: { Authorization: `Bearer ${authToken}` }, + }); + + expect(status).toBe(200); +}); + +test('should create resource with auth', async ({ authToken, apiRequest }) => { + const { status, body } = await apiRequest({ + method: 'POST', + path: '/api/orders', + headers: { Authorization: `Bearer ${authToken}` }, + body: { items: [{ productId: 'prod-1', quantity: 2 }] }, + }); + + expect(status).toBe(201); + expect(body.id).toBeDefined(); +}); +``` + +**Key Points**: + +- Token persisted to disk (not in-memory) - survives test reruns +- Provider fetches token once, reuses until expired +- Pure API authentication - no browser context needed +- `authToken` fixture handles disk read/write automatically +- Environment variables validated with clear error message + +### Example 7: Service-to-Service Authentication + +**Context**: Test microservice authentication patterns (API keys, service tokens) with proper environment validation. + +**Implementation**: + +```typescript +// tests/api/service-auth.spec.ts +import { test as base, expect } from '@playwright/test'; +import { test as apiFixture } from '@seontechnologies/playwright-utils/api-request/fixtures'; +import { mergeTests } from '@playwright/test'; + +// Validate environment variables at module load +const SERVICE_API_KEY = process.env.SERVICE_API_KEY; +const INTERNAL_SERVICE_URL = process.env.INTERNAL_SERVICE_URL; + +if (!SERVICE_API_KEY) { + throw new Error('SERVICE_API_KEY environment variable is required'); +} +if (!INTERNAL_SERVICE_URL) { + throw new Error('INTERNAL_SERVICE_URL environment variable is required'); +} + +const test = mergeTests(base, apiFixture); + +test.describe('Service-to-Service Auth', () => { + test('should authenticate with API key', async ({ apiRequest }) => { + const { status, body } = await apiRequest({ + method: 'GET', + path: '/internal/health', + baseUrl: INTERNAL_SERVICE_URL, + headers: { 'X-API-Key': SERVICE_API_KEY }, + }); + + expect(status).toBe(200); + expect(body.status).toBe('healthy'); + }); + + test('should reject invalid API key', async ({ apiRequest }) => { + const { status, body } = await apiRequest({ + method: 'GET', + path: '/internal/health', + baseUrl: INTERNAL_SERVICE_URL, + headers: { 'X-API-Key': 'invalid-key' }, + }); + + expect(status).toBe(401); + expect(body.code).toBe('INVALID_API_KEY'); + }); + + test('should call downstream service with propagated auth', async ({ apiRequest }) => { + const { status, body } = await apiRequest({ + method: 'POST', + path: '/internal/aggregate-data', + baseUrl: INTERNAL_SERVICE_URL, + headers: { + 'X-API-Key': SERVICE_API_KEY, + 'X-Request-ID': `test-${Date.now()}`, + }, + body: { sources: ['users', 'orders', 'inventory'] }, + }); + + expect(status).toBe(200); + expect(body.aggregatedFrom).toHaveLength(3); + }); +}); +``` + +**Key Points**: + +- Environment variables validated at module load with clear errors +- API key authentication (simpler than OAuth - no disk persistence needed) +- Test internal/service endpoints +- Validate auth rejection scenarios +- Correlation ID for request tracing + +> **Note**: API keys are typically static secrets that don't expire, so disk persistence (auth-session) isn't needed. For rotating service tokens, use the auth-session provider pattern from Example 6. + ## Custom Auth Provider Pattern **Context**: Adapt auth-session to your authentication system (OAuth2, JWT, SAML, custom). @@ -310,6 +505,7 @@ test('authenticated API call', async ({ apiRequest, authToken }) => { ## Related Fragments +- `api-testing-patterns.md` - Pure API testing patterns (no browser) - `overview.md` - Installation and fixture composition - `api-request.md` - Authenticated API requests - `fixtures-composition.md` - Merging auth with other utilities diff --git a/src/modules/bmm/testarch/knowledge/file-utils.md b/src/modules/bmm/testarch/knowledge/file-utils.md index 1fa02397..014ac81c 100644 --- a/src/modules/bmm/testarch/knowledge/file-utils.md +++ b/src/modules/bmm/testarch/knowledge/file-utils.md @@ -22,6 +22,16 @@ The `file-utils` module provides: - **Validation helpers**: Row count, header checks, content validation - **Format support**: Multiple sheet support (XLSX), text extraction (PDF), archive extraction (ZIP) +## Why Use This Instead of Vanilla Playwright? + +| Vanilla Playwright | File Utils | +| ------------------------------------------- | ------------------------------------------------ | +| ~80 lines per CSV flow (download + parse) | ~10 lines end-to-end | +| Manual event orchestration for downloads | Encapsulated in `handleDownload()` | +| Manual path handling and `saveAs` | Returns a ready-to-use file path | +| Manual existence checks and error handling | Centralized in one place via utility patterns | +| Manual CSV parsing config (headers, typing) | `readCSV()` returns `{ data, headers }` directly | + ## Pattern Examples ### Example 1: UI-Triggered CSV Download @@ -40,20 +50,18 @@ test('should download and validate CSV', async ({ page }) => { const downloadPath = await handleDownload({ page, downloadDir: DOWNLOAD_DIR, - trigger: () => page.click('[data-testid="export-csv"]'), + trigger: () => page.getByTestId('download-button-text/csv').click(), }); - const { content } = await readCSV({ filePath: downloadPath }); + const csvResult = await readCSV({ filePath: downloadPath }); - // Validate headers - expect(content.headers).toEqual(['ID', 'Name', 'Email', 'Role']); - - // Validate data - expect(content.data).toHaveLength(10); - expect(content.data[0]).toMatchObject({ + // Access parsed data and headers + const { data, headers } = csvResult.content; + expect(headers).toEqual(['ID', 'Name', 'Email']); + expect(data[0]).toMatchObject({ ID: expect.any(String), Name: expect.any(String), - Email: expect.stringMatching(/@/), + Email: expect.any(String), }); }); ``` @@ -81,25 +89,27 @@ test('should read multi-sheet XLSX', async () => { trigger: () => page.click('[data-testid="export-xlsx"]'), }); - const { content } = await readXLSX({ filePath: downloadPath }); + const xlsxResult = await readXLSX({ filePath: downloadPath }); - // Access specific sheets - const summarySheet = content.sheets.find((s) => s.name === 'Summary'); - const detailsSheet = content.sheets.find((s) => s.name === 'Details'); + // Verify worksheet structure + expect(xlsxResult.content.worksheets.length).toBeGreaterThan(0); + const worksheet = xlsxResult.content.worksheets[0]; + expect(worksheet).toBeDefined(); + expect(worksheet).toHaveProperty('name'); - // Validate summary - expect(summarySheet.data).toHaveLength(1); - expect(summarySheet.data[0].TotalRecords).toBe('150'); + // Access sheet data + const sheetData = worksheet?.data; + expect(Array.isArray(sheetData)).toBe(true); - // Validate details - expect(detailsSheet.data).toHaveLength(150); - expect(detailsSheet.headers).toContain('TransactionID'); + // Use type assertion for type safety + const firstRow = sheetData![0] as Record; + expect(firstRow).toHaveProperty('id'); }); ``` **Key Points**: -- `sheets` array with `name` and `data` properties +- `worksheets` array with `name` and `data` properties - Access sheets by name - Each sheet has its own headers and data - Type-safe sheet iteration @@ -117,26 +127,48 @@ test('should validate PDF report', async () => { const downloadPath = await handleDownload({ page, downloadDir: DOWNLOAD_DIR, - trigger: () => page.click('[data-testid="download-report"]'), + trigger: () => page.getByTestId('download-button-Text-based PDF Document').click(), }); - const { content } = await readPDF({ filePath: downloadPath }); + const pdfResult = await readPDF({ filePath: downloadPath }); - // content.text is extracted text from all pages - expect(content.text).toContain('Financial Report Q4 2024'); - expect(content.text).toContain('Total Revenue:'); - - // Validate page count - expect(content.numpages).toBeGreaterThan(10); + // content is extracted text from all pages + expect(pdfResult.pagesCount).toBe(1); + expect(pdfResult.fileName).toContain('.pdf'); + expect(pdfResult.content).toContain('All you need is the free Adobe Acrobat Reader'); }); ``` -**Key Points**: +**PDF Reader Options:** -- `content.text` contains all extracted text -- `content.numpages` for page count -- PDF parsing handles multi-page documents -- Search for specific phrases +```typescript +const result = await readPDF({ + filePath: '/path/to/document.pdf', + mergePages: false, // Keep pages separate (default: true) + debug: true, // Enable debug logging + maxPages: 10, // Limit processing to first 10 pages +}); +``` + +**Important Limitation - Vector-based PDFs:** + +Text extraction may fail for PDFs that store text as vector graphics (e.g., those generated by jsPDF): + +```typescript +// Vector-based PDF example (extraction fails gracefully) +const pdfResult = await readPDF({ filePath: downloadPath }); + +expect(pdfResult.pagesCount).toBe(1); +expect(pdfResult.info.extractionNotes).toContain( + 'Text extraction from vector-based PDFs is not supported.' +); +``` + +Such PDFs will have: + +- `textExtractionSuccess: false` +- `isVectorBased: true` +- Explanatory message in `extractionNotes` ### Example 4: ZIP Archive Validation @@ -154,25 +186,33 @@ test('should validate ZIP archive', async () => { trigger: () => page.click('[data-testid="download-backup"]'), }); - const { content } = await readZIP({ filePath: downloadPath }); + const zipResult = await readZIP({ filePath: downloadPath }); // Check file list - expect(content.files).toContain('data.csv'); - expect(content.files).toContain('config.json'); - expect(content.files).toContain('readme.txt'); + expect(Array.isArray(zipResult.content.entries)).toBe(true); + expect(zipResult.content.entries).toContain( + 'Case_53125_10-19-22_AM/Case_53125_10-19-22_AM_case_data.csv' + ); - // Read specific file from archive - const configContent = content.zip.readAsText('config.json'); - const config = JSON.parse(configContent); + // Extract specific file + const targetFile = 'Case_53125_10-19-22_AM/Case_53125_10-19-22_AM_case_data.csv'; + const zipWithExtraction = await readZIP({ + filePath: downloadPath, + fileToExtract: targetFile, + }); - expect(config.version).toBe('2.0'); + // Access extracted file buffer + const extractedFiles = zipWithExtraction.content.extractedFiles || {}; + const fileBuffer = extractedFiles[targetFile]; + expect(fileBuffer).toBeInstanceOf(Buffer); + expect(fileBuffer?.length).toBeGreaterThan(0); }); ``` **Key Points**: -- `content.files` lists all files in archive -- `content.zip.readAsText()` extracts specific files +- `content.entries` lists all files in archive +- `fileToExtract` extracts specific files to Buffer - Validate archive structure - Read and parse individual files from ZIP @@ -185,7 +225,7 @@ test('should validate ZIP archive', async () => { ```typescript test('should download via API', async ({ page, request }) => { const downloadPath = await handleDownload({ - page, + page, // Still need page for download events downloadDir: DOWNLOAD_DIR, trigger: async () => { const response = await request.get('/api/export/csv', { @@ -211,20 +251,123 @@ test('should download via API', async ({ page, request }) => { - Still need `page` for download events - Works with authenticated endpoints -## Validation Helpers +### Example 6: Reading CSV from Buffer (ZIP extraction) + +**Context**: Read CSV content directly from a Buffer (e.g., extracted from ZIP). + +**Implementation**: ```typescript -// CSV validation -const { isValid, errors } = await validateCSV({ - filePath: downloadPath, - expectedRowCount: 10, - requiredHeaders: ['ID', 'Name', 'Email'], +// Read from a Buffer (e.g., extracted from a ZIP) +const zipResult = await readZIP({ + filePath: 'archive.zip', + fileToExtract: 'data.csv', }); +const fileBuffer = zipResult.content.extractedFiles?.['data.csv']; +const csvFromBuffer = await readCSV({ content: fileBuffer }); -expect(isValid).toBe(true); -expect(errors).toHaveLength(0); +// Read from a string +const csvString = 'name,age\nJohn,30\nJane,25'; +const csvFromString = await readCSV({ content: csvString }); + +const { data, headers } = csvFromString.content; +expect(headers).toContain('name'); +expect(headers).toContain('age'); ``` +## API Reference + +### CSV Reader Options + +| Option | Type | Default | Description | +| -------------- | ------------------ | -------- | -------------------------------------- | +| `filePath` | `string` | - | Path to CSV file (mutually exclusive) | +| `content` | `string \| Buffer` | - | Direct content (mutually exclusive) | +| `delimiter` | `string \| 'auto'` | `','` | Value separator, auto-detect if 'auto' | +| `encoding` | `string` | `'utf8'` | File encoding | +| `parseHeaders` | `boolean` | `true` | Use first row as headers | +| `trim` | `boolean` | `true` | Trim whitespace from values | + +### XLSX Reader Options + +| Option | Type | Description | +| ----------- | -------- | ------------------------------ | +| `filePath` | `string` | Path to XLSX file | +| `sheetName` | `string` | Name of sheet to set as active | + +### PDF Reader Options + +| Option | Type | Default | Description | +| ------------ | --------- | ------- | --------------------------- | +| `filePath` | `string` | - | Path to PDF file (required) | +| `mergePages` | `boolean` | `true` | Merge text from all pages | +| `maxPages` | `number` | - | Maximum pages to extract | +| `debug` | `boolean` | `false` | Enable debug logging | + +### ZIP Reader Options + +| Option | Type | Description | +| --------------- | -------- | ---------------------------------- | +| `filePath` | `string` | Path to ZIP file | +| `fileToExtract` | `string` | Specific file to extract to Buffer | + +### Return Values + +#### CSV Reader Return Value + +```typescript +{ + content: { + data: Array>, // Parsed rows (excludes header row if parseHeaders: true) + headers: string[] | null // Column headers (null if parseHeaders: false) + } +} +``` + +#### XLSX Reader Return Value + +```typescript +{ + content: { + worksheets: Array<{ + name: string, // Sheet name + rows: Array>, // All rows including headers + headers?: string[] // First row as headers (if present) + }> + } +} +``` + +#### PDF Reader Return Value + +```typescript +{ + content: string, // Extracted text (merged or per-page based on mergePages) + pagesCount: number, // Total pages in PDF + fileName?: string, // Original filename if available + info?: Record // PDF metadata (author, title, etc.) +} +``` + +> **Note**: When `mergePages: false`, `content` is an array of strings (one per page). When `maxPages` is set, only that many pages are extracted. + +#### ZIP Reader Return Value + +```typescript +{ + content: { + entries: Array<{ + name: string, // File/directory path within ZIP + size: number, // Uncompressed size in bytes + isDirectory: boolean // True for directories + }>, + extractedFiles: Record // Extracted file contents by path + } +} +``` + +> **Note**: When `fileToExtract` is specified, only that file appears in `extractedFiles`. + ## Download Cleanup Pattern ```typescript @@ -234,6 +377,66 @@ test.afterEach(async () => { }); ``` +## Comparison with Vanilla Playwright + +Vanilla Playwright (real test) snippet: + +```typescript +// ~80 lines of boilerplate! +const [download] = await Promise.all([ + page.waitForEvent('download'), + page.getByTestId('download-button-CSV Export').click(), +]); + +const failure = await download.failure(); +expect(failure).toBeNull(); + +const filePath = testInfo.outputPath(download.suggestedFilename()); +await download.saveAs(filePath); + +await expect + .poll( + async () => { + try { + await fs.access(filePath); + return true; + } catch { + return false; + } + }, + { timeout: 5000, intervals: [100, 200, 500] } + ) + .toBe(true); + +const csvContent = await fs.readFile(filePath, 'utf-8'); + +const parseResult = parse(csvContent, { + header: true, + skipEmptyLines: true, + dynamicTyping: true, + transformHeader: (header: string) => header.trim(), +}); + +if (parseResult.errors.length > 0) { + throw new Error(`CSV parsing errors: ${JSON.stringify(parseResult.errors)}`); +} + +const data = parseResult.data as Array>; +const headers = parseResult.meta.fields || []; +``` + +With File Utils, the same flow becomes: + +```typescript +const downloadPath = await handleDownload({ + page, + downloadDir: DOWNLOAD_DIR, + trigger: () => page.getByTestId('download-button-text/csv').click(), +}); + +const { data, headers } = (await readCSV({ filePath: downloadPath })).content; +``` + ## Related Fragments - `overview.md` - Installation and imports @@ -242,7 +445,7 @@ test.afterEach(async () => { ## Anti-Patterns -**❌ Not cleaning up downloads:** +**DON'T leave downloads in place:** ```typescript test('creates file', async () => { @@ -251,7 +454,7 @@ test('creates file', async () => { }) ``` -**βœ… Clean up after tests:** +**DO clean up after tests:** ```typescript test.afterEach(async () => { diff --git a/src/modules/bmm/testarch/knowledge/intercept-network-call.md b/src/modules/bmm/testarch/knowledge/intercept-network-call.md index a175d559..e47419da 100644 --- a/src/modules/bmm/testarch/knowledge/intercept-network-call.md +++ b/src/modules/bmm/testarch/knowledge/intercept-network-call.md @@ -183,7 +183,31 @@ test('should handle timeout', async ({ page, interceptNetworkCall }) => { - Validate error UI states - No real failures needed -### Example 5: Multiple Intercepts (Order Matters!) +### Example 5: Order Matters - Intercept Before Navigate + +**Context**: The interceptor must be set up before the network request occurs. + +**Implementation**: + +```typescript +// INCORRECT - interceptor set up too late +await page.goto('https://example.com'); // Request already happened +const networkCall = interceptNetworkCall({ url: '**/api/data' }); +await networkCall; // Will hang indefinitely! + +// CORRECT - Set up interception first +const networkCall = interceptNetworkCall({ url: '**/api/data' }); +await page.goto('https://example.com'); +const result = await networkCall; +``` + +This pattern follows the classic test spy/stub pattern: + +1. Define the spy/stub (set up interception) +2. Perform the action (trigger the network request) +3. Assert on the spy/stub (await and verify the response) + +### Example 6: Multiple Intercepts **Context**: Intercepting different endpoints in same test - setup order is critical. @@ -191,7 +215,7 @@ test('should handle timeout', async ({ page, interceptNetworkCall }) => { ```typescript test('multiple intercepts', async ({ page, interceptNetworkCall }) => { - // βœ… CORRECT: Setup all intercepts BEFORE navigation + // Setup all intercepts BEFORE navigation const usersCall = interceptNetworkCall({ url: '**/api/users' }); const productsCall = interceptNetworkCall({ url: '**/api/products' }); const ordersCall = interceptNetworkCall({ url: '**/api/orders' }); @@ -211,11 +235,85 @@ test('multiple intercepts', async ({ page, interceptNetworkCall }) => { - Setup all intercepts before triggering actions - Use `Promise.all()` to wait for multiple calls -- Order: intercept β†’ navigate β†’ await +- Order: intercept -> navigate -> await - Prevents race conditions +### Example 7: Capturing Multiple Requests to the Same Endpoint + +**Context**: Each `interceptNetworkCall` captures only the first matching request. + +**Implementation**: + +```typescript +// Capturing a known number of requests +const firstRequest = interceptNetworkCall({ url: '/api/data' }); +const secondRequest = interceptNetworkCall({ url: '/api/data' }); + +await page.click('#load-data-button'); + +const firstResponse = await firstRequest; +const secondResponse = await secondRequest; + +expect(firstResponse.status).toBe(200); +expect(secondResponse.status).toBe(200); + +// Handling an unknown number of requests +const getDataRequestInterceptor = () => + interceptNetworkCall({ + url: '/api/data', + timeout: 1000, // Short timeout to detect when no more requests are coming + }); + +let currentInterceptor = getDataRequestInterceptor(); +const allResponses = []; + +await page.click('#load-multiple-data-button'); + +while (true) { + try { + const response = await currentInterceptor; + allResponses.push(response); + currentInterceptor = getDataRequestInterceptor(); + } catch (error) { + // No more requests (timeout) + break; + } +} + +console.log(`Captured ${allResponses.length} requests to /api/data`); +``` + +### Example 8: Using Timeout + +**Context**: Set a timeout for waiting on a network request. + +**Implementation**: + +```typescript +const dataCall = interceptNetworkCall({ + method: 'GET', + url: '/api/data-that-might-be-slow', + timeout: 5000, // 5 seconds timeout +}); + +await page.goto('/data-page'); + +try { + const { responseJson } = await dataCall; + console.log('Data loaded successfully:', responseJson); +} catch (error) { + if (error.message.includes('timeout')) { + console.log('Request timed out as expected'); + } else { + throw error; + } +} +``` + ## URL Pattern Matching +The utility uses [picomatch](https://github.com/micromatch/picomatch) for powerful glob pattern matching, dramatically simplifying URL targeting: + **Supported glob patterns:** ```typescript @@ -226,7 +324,59 @@ test('multiple intercepts', async ({ page, interceptNetworkCall }) => { '**/api/users?id=*'; // With query params ``` -**Uses picomatch library** - same pattern syntax as Playwright's `page.route()` but cleaner API. +**Comparison with vanilla Playwright:** + +```typescript +// Vanilla Playwright - complex predicate +const predicate = (response) => { + const url = response.url(); + return ( + url.endsWith('/api/users') || + url.match(/\/api\/users\/\d+/) || + (url.includes('/api/users/') && url.includes('/profile')) + ); +}; +page.waitForResponse(predicate); + +// With interceptNetworkCall - simple glob patterns +interceptNetworkCall({ url: '/api/users' }); // Exact endpoint +interceptNetworkCall({ url: '/api/users/*' }); // User by ID pattern +interceptNetworkCall({ url: '/api/users/*/profile' }); // Specific sub-paths +interceptNetworkCall({ url: '/api/users/**' }); // Match all +``` + +## API Reference + +### `interceptNetworkCall(options)` + +| Parameter | Type | Description | +| ----------------- | ---------- | --------------------------------------------------------------------- | +| `page` | `Page` | Required when using direct import (not needed with fixture) | +| `method` | `string` | Optional: HTTP method to match (e.g., 'GET', 'POST') | +| `url` | `string` | Optional: URL pattern to match (supports glob patterns via picomatch) | +| `fulfillResponse` | `object` | Optional: Response to use when mocking | +| `handler` | `function` | Optional: Custom handler function for the route | +| `timeout` | `number` | Optional: Timeout in milliseconds for the network request | + +### `fulfillResponse` Object + +| Property | Type | Description | +| --------- | ------------------------ | ----------------------------------------------------- | +| `status` | `number` | HTTP status code (default: 200) | +| `headers` | `Record` | Response headers | +| `body` | `any` | Response body (will be JSON.stringified if an object) | + +### Return Value + +Returns a `Promise` with: + +| Property | Type | Description | +| -------------- | ---------- | --------------------------------------- | +| `request` | `Request` | The intercepted request | +| `response` | `Response` | The response (null if mocked) | +| `responseJson` | `any` | Parsed JSON response (if available) | +| `status` | `number` | HTTP status code | +| `requestJson` | `any` | Parsed JSON request body (if available) | ## Comparison with Vanilla Playwright @@ -238,7 +388,7 @@ test('multiple intercepts', async ({ page, interceptNetworkCall }) => { | `const status = resp.status()` | `const { status } = await call` | | Complex filter predicates | Simple glob patterns | -**Reduction:** ~5-7 lines β†’ ~2-3 lines per interception +**Reduction:** ~5-7 lines -> ~2-3 lines per interception ## Related Fragments @@ -248,14 +398,14 @@ test('multiple intercepts', async ({ page, interceptNetworkCall }) => { ## Anti-Patterns -**❌ Intercepting after navigation:** +**DON'T intercept after navigation:** ```typescript await page.goto('/dashboard'); // Navigation starts const usersCall = interceptNetworkCall({ url: '**/api/users' }); // Too late! ``` -**βœ… Intercept before navigate:** +**DO intercept before navigate:** ```typescript const usersCall = interceptNetworkCall({ url: '**/api/users' }); // First @@ -263,7 +413,7 @@ await page.goto('/dashboard'); // Then navigate const { responseJson } = await usersCall; // Then await ``` -**❌ Ignoring the returned Promise:** +**DON'T ignore the returned Promise:** ```typescript interceptNetworkCall({ url: '**/api/users' }); // Not awaited! @@ -271,7 +421,7 @@ await page.goto('/dashboard'); // No deterministic wait - race condition ``` -**βœ… Always await the intercept:** +**DO always await the intercept:** ```typescript const usersCall = interceptNetworkCall({ url: '**/api/users' }); diff --git a/src/modules/bmm/testarch/knowledge/log.md b/src/modules/bmm/testarch/knowledge/log.md index 42ddc228..c222a0c6 100644 --- a/src/modules/bmm/testarch/knowledge/log.md +++ b/src/modules/bmm/testarch/knowledge/log.md @@ -21,6 +21,20 @@ The `log` utility provides: - **Multiple levels**: info, step, success, warning, error, debug - **Optional console**: Can disable console output but keep report logs +## Quick Start + +```typescript +import { log } from '@seontechnologies/playwright-utils'; + +// Basic logging +await log.info('Starting test'); +await log.step('Test step shown in Playwright UI'); +await log.success('Operation completed'); +await log.warning('Something to note'); +await log.error('Something went wrong'); +await log.debug('Debug information'); +``` + ## Pattern Examples ### Example 1: Basic Logging Levels @@ -143,41 +157,105 @@ test('organized with steps', async ({ page, apiRequest }) => { - Steps visible in Playwright trace viewer - Better debugging when tests fail -### Example 4: Conditional Logging +### Example 4: Test Step Decorators -**Context**: Log different messages based on environment or test conditions. +**Context**: Create collapsible test steps in Playwright UI using decorators. + +**Page Object Methods with @methodTestStep:** + +```typescript +import { methodTestStep } from '@seontechnologies/playwright-utils'; + +class TodoPage { + constructor(private page: Page) { + this.name = 'TodoPage'; + } + + readonly name: string; + + @methodTestStep('Add todo item') + async addTodo(text: string) { + await log.info(`Adding todo: ${text}`); + const newTodo = this.page.getByPlaceholder('What needs to be done?'); + await newTodo.fill(text); + await newTodo.press('Enter'); + await log.step('step within a decorator'); + await log.success(`Added todo: ${text}`); + } + + @methodTestStep('Get all todos') + async getTodos() { + await log.info('Getting all todos'); + return this.page.getByTestId('todo-title'); + } +} +``` + +**Function Helpers with functionTestStep:** + +```typescript +import { functionTestStep } from '@seontechnologies/playwright-utils'; + +// Define todo items for the test +const TODO_ITEMS = ['buy groceries', 'pay bills', 'schedule meeting']; + +const createDefaultTodos = functionTestStep('Create default todos', async (page: Page) => { + await log.info('Creating default todos'); + await log.step('step within a functionWrapper'); + const todoPage = new TodoPage(page); + + for (const item of TODO_ITEMS) { + await todoPage.addTodo(item); + } + + await log.success('Created all default todos'); +}); + +const checkNumberOfTodosInLocalStorage = functionTestStep( + 'Check total todos count fn-step', + async (page: Page, expected: number) => { + await log.info(`Verifying todo count: ${expected}`); + const result = await page.waitForFunction( + (e) => JSON.parse(localStorage['react-todos']).length === e, + expected + ); + await log.success(`Verified todo count: ${expected}`); + return result; + } +); +``` + +### Example 5: File Logging + +**Context**: Enable file logging for persistent logs. **Implementation**: ```typescript -test('conditional logging', async ({ page }) => { - const isCI = process.env.CI === 'true'; +// playwright/support/fixtures.ts +import { test as base } from '@playwright/test'; +import { log, captureTestContext } from '@seontechnologies/playwright-utils'; - if (isCI) { - await log.info('Running in CI environment'); - } else { - await log.debug('Running locally'); - } +// Configure file logging globally +log.configure({ + fileLogging: { + enabled: true, + outputDir: 'playwright-logs/organized-logs', + forceConsolidated: false, // One file per test + }, +}); - const isKafkaWorking = await checkKafkaHealth(); - - if (!isKafkaWorking) { - await log.warning('Kafka unavailable - skipping event checks'); - } else { - await log.step('Verifying Kafka events'); - // ... event verification - } +// Extend base test with file logging context capture +export const test = base.extend({ + // Auto-capture test context for file logging + autoTestContext: [async ({}, use, testInfo) => { + captureTestContext(testInfo); + await use(undefined); + }, { auto: true }], }); ``` -**Key Points**: - -- Log based on environment -- Skip logging with conditionals -- Use appropriate log levels -- Debug info for local, minimal for CI - -### Example 5: Integration with Auth and API +### Example 6: Integration with Auth and API **Context**: Log authenticated API requests with tokens (safely). @@ -221,16 +299,73 @@ test('should log auth flow', async ({ authToken, apiRequest }) => { - Combine with auth and API utilities - Log at appropriate detail level +## Configuration + +**Defaults:** console logging enabled, file logging disabled. + +```typescript +// Enable file logging in config +log.configure({ + console: true, // default + fileLogging: { + enabled: true, + outputDir: 'playwright-logs', + forceConsolidated: false, // One file per test + }, +}); + +// Per-test override +await log.info('Message', { + console: { enabled: false }, + fileLogging: { enabled: true }, +}); +``` + +### Environment Variables + +```bash +# Disable all logging +SILENT=true + +# Disable only file logging +DISABLE_FILE_LOGS=true + +# Disable only console logging +DISABLE_CONSOLE_LOGS=true +``` + +### Level Filtering + +```typescript +log.configure({ + level: 'warning', // Only warning, error levels will show +}); + +// Available levels (in priority order): +// debug < info < step < success < warning < error +``` + +### Sync Methods + +For non-test contexts (global setup, utility functions): + +```typescript +// Use sync methods when async/await isn't available +log.infoSync('Initializing configuration'); +log.successSync('Environment configured'); +log.errorSync('Setup failed'); +``` + ## Log Levels Guide -| Level | When to Use | Shows in Report | Shows in Console | -| --------- | ----------------------------------- | -------------------- | ---------------- | -| `step` | Test organization, major actions | βœ… Collapsible steps | βœ… Yes | -| `info` | General information, state changes | βœ… Yes | βœ… Yes | -| `success` | Successful operations | βœ… Yes | βœ… Yes | -| `warning` | Non-critical issues, skipped checks | βœ… Yes | βœ… Yes | -| `error` | Failures, exceptions | βœ… Yes | βœ… Configurable | -| `debug` | Detailed data, objects | βœ… Yes (attached) | βœ… Configurable | +| Level | When to Use | Shows in Report | Shows in Console | +| --------- | ----------------------------------- | ----------------- | ---------------- | +| `step` | Test organization, major actions | Collapsible steps | Yes | +| `info` | General information, state changes | Yes | Yes | +| `success` | Successful operations | Yes | Yes | +| `warning` | Non-critical issues, skipped checks | Yes | Yes | +| `error` | Failures, exceptions | Yes | Configurable | +| `debug` | Detailed data, objects | Yes (attached) | Configurable | ## Comparison with console.log @@ -251,34 +386,34 @@ test('should log auth flow', async ({ authToken, apiRequest }) => { ## Anti-Patterns -**❌ Logging objects in steps:** +**DON'T log objects in steps:** ```typescript await log.step({ user: 'test', action: 'create' }); // Shows empty in UI ``` -**βœ… Use strings for steps, objects for debug:** +**DO use strings for steps, objects for debug:** ```typescript await log.step('Creating user: test'); // Readable in UI await log.debug({ user: 'test', action: 'create' }); // Detailed data ``` -**❌ Logging sensitive data:** +**DON'T log sensitive data:** ```typescript await log.info(`Password: ${password}`); // Security risk! await log.info(`Token: ${authToken}`); // Full token exposed! ``` -**βœ… Use previews or omit sensitive data:** +**DO use previews or omit sensitive data:** ```typescript await log.info('User authenticated successfully'); // No sensitive data await log.debug({ tokenPreview: token.slice(0, 6) + '...' }); ``` -**❌ Excessive logging in loops:** +**DON'T log excessively in loops:** ```typescript for (const item of items) { @@ -286,7 +421,7 @@ for (const item of items) { } ``` -**βœ… Log summary or use debug level:** +**DO log summary or use debug level:** ```typescript await log.step(`Processing ${items.length} items`); diff --git a/src/modules/bmm/testarch/knowledge/network-error-monitor.md b/src/modules/bmm/testarch/knowledge/network-error-monitor.md index 0a2321bd..a5ee9069 100644 --- a/src/modules/bmm/testarch/knowledge/network-error-monitor.md +++ b/src/modules/bmm/testarch/knowledge/network-error-monitor.md @@ -21,6 +21,19 @@ The `network-error-monitor` provides: - **Smart opt-out**: Disable for validation tests expecting errors - **Deduplication**: Group repeated errors by pattern - **Domino effect prevention**: Limit test failures per error pattern +- **Respects test status**: Won't suppress actual test failures + +## Quick Start + +```typescript +import { test } from '@seontechnologies/playwright-utils/network-error-monitor/fixtures'; + +// That's it! Network monitoring is automatically enabled +test('my test', async ({ page }) => { + await page.goto('/dashboard'); + // If any HTTP 4xx/5xx errors occur, the test will fail +}); +``` ## Pattern Examples @@ -38,8 +51,8 @@ test('should load dashboard', async ({ page }) => { await page.goto('/dashboard'); await expect(page.locator('h1')).toContainText('Dashboard'); - // βœ… Passes if no HTTP errors - // ❌ Fails if any 4xx/5xx errors detected with clear message: + // Passes if no HTTP errors + // Fails if any 4xx/5xx errors detected with clear message: // "Network errors detected: 2 request(s) failed" // Failed requests: // GET 500 https://api.example.com/users @@ -64,13 +77,17 @@ test('should load dashboard', async ({ page }) => { import { test } from '@seontechnologies/playwright-utils/network-error-monitor/fixtures'; // Opt-out with annotation -test('should show error on invalid input', { annotation: [{ type: 'skipNetworkMonitoring' }] }, async ({ page }) => { - await page.goto('/form'); - await page.click('#submit'); // Triggers 400 error +test( + 'should show error on invalid input', + { annotation: [{ type: 'skipNetworkMonitoring' }] }, + async ({ page }) => { + await page.goto('/form'); + await page.click('#submit'); // Triggers 400 error - // Monitoring disabled - test won't fail on 400 - await expect(page.getByText('Invalid input')).toBeVisible(); -}); + // Monitoring disabled - test won't fail on 400 + await expect(page.getByText('Invalid input')).toBeVisible(); + } +); // Or opt-out entire describe block test.describe('error handling', { annotation: [{ type: 'skipNetworkMonitoring' }] }, () => { @@ -91,7 +108,139 @@ test.describe('error handling', { annotation: [{ type: 'skipNetworkMonitoring' } - Monitoring still active for other tests - Perfect for intentional error scenarios -### Example 3: Integration with Merged Fixtures +### Example 3: Respects Test Status + +**Context**: The monitor respects final test statuses to avoid suppressing important test outcomes. + +**Behavior by test status:** + +- **`failed`**: Network errors logged as additional context, not thrown +- **`timedOut`**: Network errors logged as additional context +- **`skipped`**: Network errors logged, skip status preserved +- **`interrupted`**: Network errors logged, interrupted status preserved +- **`passed`**: Network errors throw and fail the test + +**Example with test.skip():** + +```typescript +test('feature gated test', async ({ page }) => { + const featureEnabled = await checkFeatureFlag(); + test.skip(!featureEnabled, 'Feature not enabled'); + // If skipped, network errors won't turn this into a failure + await page.goto('/new-feature'); +}); +``` + +### Example 4: Excluding Legitimate Errors + +**Context**: Some endpoints legitimately return 4xx/5xx responses. + +**Implementation**: + +```typescript +import { test as base } from '@playwright/test'; +import { createNetworkErrorMonitorFixture } from '@seontechnologies/playwright-utils/network-error-monitor/fixtures'; + +export const test = base.extend( + createNetworkErrorMonitorFixture({ + excludePatterns: [ + /email-cluster\/ml-app\/has-active-run/, // ML service returns 404 when no active run + /idv\/session-templates\/list/, // IDV service returns 404 when not configured + /sentry\.io\/api/, // External Sentry errors should not fail tests + ], + }) +); +``` + +**For merged fixtures:** + +```typescript +import { test as base, mergeTests } from '@playwright/test'; +import { createNetworkErrorMonitorFixture } from '@seontechnologies/playwright-utils/network-error-monitor/fixtures'; + +const networkErrorMonitor = base.extend( + createNetworkErrorMonitorFixture({ + excludePatterns: [/analytics\.google\.com/, /cdn\.example\.com/], + }) +); + +export const test = mergeTests(authFixture, networkErrorMonitor); +``` + +### Example 5: Preventing Domino Effect + +**Context**: One failing endpoint shouldn't fail all tests. + +**Implementation**: + +```typescript +import { test as base } from '@playwright/test'; +import { createNetworkErrorMonitorFixture } from '@seontechnologies/playwright-utils/network-error-monitor/fixtures'; + +const networkErrorMonitor = base.extend( + createNetworkErrorMonitorFixture({ + excludePatterns: [], // Required when using maxTestsPerError + maxTestsPerError: 1, // Only first test fails per error pattern, rest just log + }) +); +``` + +**How it works:** + +When `/api/v2/case-management/cases` returns 500: + +- **First test** encountering this error: **FAILS** with clear error message +- **Subsequent tests** encountering same error: **PASSES** but logs warning + +Error patterns are grouped by `method + status + base path`: + +- `GET /api/v2/case-management/cases/123` -> Pattern: `GET:500:/api/v2/case-management` +- `GET /api/v2/case-management/quota` -> Pattern: `GET:500:/api/v2/case-management` (same group!) +- `POST /api/v2/case-management/cases` -> Pattern: `POST:500:/api/v2/case-management` (different group!) + +**Why include HTTP method?** A GET 404 vs POST 404 might represent different issues: + +- `GET 404 /api/users/123` -> User not found (expected in some tests) +- `POST 404 /api/users` -> Endpoint doesn't exist (critical error) + +**Output for subsequent tests:** + +``` +Warning: Network errors detected but not failing test (maxTestsPerError limit reached): + GET 500 https://api.example.com/api/v2/case-management/cases +``` + +**Recommended configuration:** + +```typescript +createNetworkErrorMonitorFixture({ + excludePatterns: [...], // Required - known broken endpoints (can be empty []) + maxTestsPerError: 1 // Stop domino effect (requires excludePatterns) +}) +``` + +**Understanding worker-level state:** + +Error pattern counts are stored in worker-level global state: + +```typescript +// test-file-1.spec.ts (runs in Worker 1) +test('test A', () => { + /* triggers GET:500:/api/v2/cases */ +}); // FAILS + +// test-file-2.spec.ts (runs later in Worker 1) +test('test B', () => { + /* triggers GET:500:/api/v2/cases */ +}); // PASSES (limit reached) + +// test-file-3.spec.ts (runs in Worker 2 - different worker) +test('test C', () => { + /* triggers GET:500:/api/v2/cases */ +}); // FAILS (fresh worker) +``` + +### Example 6: Integration with Merged Fixtures **Context**: Combine network-error-monitor with other utilities. @@ -105,7 +254,7 @@ import { test as networkErrorMonitorFixture } from '@seontechnologies/playwright export const test = mergeTests( authFixture, - networkErrorMonitorFixture, + networkErrorMonitorFixture // Add other fixtures ); @@ -127,110 +276,94 @@ test('authenticated with monitoring', async ({ page, authToken }) => { - Monitoring active automatically - No extra setup needed -### Example 4: Domino Effect Prevention - -**Context**: One failing endpoint shouldn't fail all tests. - -**Implementation**: - -```typescript -// Configuration (internal to utility) -const config = { - maxTestsPerError: 3, // Max 3 tests fail per unique error pattern -}; - -// Scenario: -// Test 1: GET /api/broken β†’ 500 error β†’ Test fails ❌ -// Test 2: GET /api/broken β†’ 500 error β†’ Test fails ❌ -// Test 3: GET /api/broken β†’ 500 error β†’ Test fails ❌ -// Test 4: GET /api/broken β†’ 500 error β†’ Test passes ⚠️ (limit reached, warning logged) -// Test 5: Different error pattern β†’ Test fails ❌ (new pattern, counter resets) -``` - -**Key Points**: - -- Limits cascading failures -- Groups errors by URL + status code pattern -- Warns when limit reached -- Prevents flaky backend from failing entire suite - -### Example 5: Artifact Structure +### Example 7: Artifact Structure **Context**: Debugging failed tests with network error artifacts. -**Implementation**: - When test fails due to network errors, artifact attached: ```json -// test-results/my-test/network-errors.json -{ - "errors": [ - { - "url": "https://api.example.com/users", - "method": "GET", - "status": 500, - "statusText": "Internal Server Error", - "timestamp": "2024-08-13T10:30:45.123Z" - }, - { - "url": "https://api.example.com/metrics", - "method": "POST", - "status": 503, - "statusText": "Service Unavailable", - "timestamp": "2024-08-13T10:30:46.456Z" - } - ], - "summary": { - "totalErrors": 2, - "uniquePatterns": 2 +[ + { + "url": "https://api.example.com/users", + "status": 500, + "method": "GET", + "timestamp": "2025-11-10T12:34:56.789Z" + }, + { + "url": "https://api.example.com/metrics", + "status": 503, + "method": "POST", + "timestamp": "2025-11-10T12:34:57.123Z" } -} +] ``` -**Key Points**: +## Implementation Details -- JSON artifact per failed test -- Full error details (URL, method, status, timestamp) -- Summary statistics -- Easy debugging with structured data +### How It Works -## Comparison with Manual Error Checks +1. **Fixture Extension**: Uses Playwright's `base.extend()` with `auto: true` +2. **Response Listener**: Attaches `page.on('response')` listener at test start +3. **Multi-Page Monitoring**: Automatically monitors popups and new tabs via `context.on('page')` +4. **Error Collection**: Captures 4xx/5xx responses, checking exclusion patterns +5. **Try/Finally**: Ensures error processing runs even if test fails early +6. **Status Check**: Only throws errors if test hasn't already reached final status +7. **Artifact**: Attaches JSON file to test report for debugging -| Manual Approach | network-error-monitor | -| ------------------------------------------------------ | -------------------------- | -| `page.on('response', resp => { if (!resp.ok()) ... })` | Auto-enabled, zero setup | -| Check each response manually | Automatic for all requests | -| Custom error tracking logic | Built-in deduplication | -| No structured artifacts | JSON artifacts attached | -| Easy to forget | Never miss a backend error | +### Performance + +The monitor has minimal performance impact: + +- Event listener overhead: ~0.1ms per response +- Memory: ~200 bytes per unique error +- No network delay (observes responses, doesn't intercept them) + +## Comparison with Alternatives + +| Approach | Network Error Monitor | Manual afterEach | +| --------------------------- | --------------------- | --------------------- | +| **Setup Required** | Zero (auto-enabled) | Every test file | +| **Catches Silent Failures** | Yes | Yes (if configured) | +| **Structured Artifacts** | JSON attached | Custom impl | +| **Test Failure Safety** | Try/finally | afterEach may not run | +| **Opt-Out Mechanism** | Annotation | Custom logic | +| **Status Aware** | Respects skip/failed | No | ## When to Use **Auto-enabled for:** -- βœ… All E2E tests -- βœ… Integration tests -- βœ… Any test hitting real APIs +- All E2E tests +- Integration tests +- Any test hitting real APIs **Opt-out for:** -- ❌ Validation tests (expecting 4xx) -- ❌ Error handling tests (expecting 5xx) -- ❌ Offline tests (network-recorder playback) +- Validation tests (expecting 4xx) +- Error handling tests (expecting 5xx) +- Offline tests (network-recorder playback) -## Integration with Framework Setup +## Troubleshooting -In `*framework` workflow, mention network-error-monitor: +### Test fails with network errors but I don't see them in my app + +The errors might be happening during page load or in background polling. Check the `network-errors.json` artifact in your test report for full details including timestamps. + +### False positives from external services + +Configure exclusion patterns as shown in the "Excluding Legitimate Errors" section above. + +### Network errors not being caught + +Ensure you're importing the test from the correct fixture: ```typescript -// Add to merged-fixtures.ts -import { test as networkErrorMonitorFixture } from '@seontechnologies/playwright-utils/network-error-monitor/fixtures'; +// Correct +import { test } from '@seontechnologies/playwright-utils/network-error-monitor/fixtures'; -export const test = mergeTests( - // ... other fixtures - networkErrorMonitorFixture, -); +// Wrong - this won't have network monitoring +import { test } from '@playwright/test'; ``` ## Related Fragments @@ -241,14 +374,14 @@ export const test = mergeTests( ## Anti-Patterns -**❌ Opting out of monitoring globally:** +**DON'T opt out of monitoring globally:** ```typescript // Every test skips monitoring test.use({ annotation: [{ type: 'skipNetworkMonitoring' }] }); ``` -**βœ… Opt-out only for specific error tests:** +**DO opt-out only for specific error tests:** ```typescript test.describe('error scenarios', { annotation: [{ type: 'skipNetworkMonitoring' }] }, () => { @@ -256,17 +389,17 @@ test.describe('error scenarios', { annotation: [{ type: 'skipNetworkMonitoring' }); ``` -**❌ Ignoring network error artifacts:** +**DON'T ignore network error artifacts:** ```typescript // Test fails, artifact shows 500 errors // Developer: "Works on my machine" Β―\_(ツ)_/Β― ``` -**βœ… Check artifacts for root cause:** +**DO check artifacts for root cause:** ```typescript // Read network-errors.json artifact -// Identify failing endpoint: GET /api/users β†’ 500 +// Identify failing endpoint: GET /api/users -> 500 // Fix backend issue before merging ``` diff --git a/src/modules/bmm/testarch/knowledge/network-recorder.md b/src/modules/bmm/testarch/knowledge/network-recorder.md index ff24cb4e..7b9c6e26 100644 --- a/src/modules/bmm/testarch/knowledge/network-recorder.md +++ b/src/modules/bmm/testarch/knowledge/network-recorder.md @@ -21,6 +21,46 @@ HAR-based recording/playback provides: - **Stateful mocking**: CRUD operations work naturally (not just read-only) - **Environment flexibility**: Map URLs for any environment +## Quick Start + +### 1. Record Network Traffic + +```typescript +// Set mode to 'record' to capture network traffic +process.env.PW_NET_MODE = 'record'; + +test('should add, edit and delete a movie', async ({ page, context, networkRecorder }) => { + // Setup network recorder - it will record all network traffic + await networkRecorder.setup(context); + + // Your normal test code + await page.goto('/'); + await page.fill('#movie-name', 'Inception'); + await page.click('#add-movie'); + + // Network traffic is automatically saved to HAR file +}); +``` + +### 2. Playback Network Traffic + +```typescript +// Set mode to 'playback' to use recorded traffic +process.env.PW_NET_MODE = 'playback'; + +test('should add, edit and delete a movie', async ({ page, context, networkRecorder }) => { + // Setup network recorder - it will replay from HAR file + await networkRecorder.setup(context); + + // Same test code runs without hitting real backend! + await page.goto('/'); + await page.fill('#movie-name', 'Inception'); + await page.click('#add-movie'); +}); +``` + +That's it! Your tests now run completely offline using recorded network traffic. + ## Pattern Examples ### Example 1: Basic Record and Playback @@ -115,74 +155,173 @@ test.describe('Movie CRUD - offline with network recorder', () => { - Combine with `interceptNetworkCall` for deterministic waits - First run records, subsequent runs replay -### Example 3: Environment Switching +### Example 3: Common Patterns + +**Recording Only API Calls**: + +```typescript +await networkRecorder.setup(context, { + recording: { + urlFilter: /\/api\// // Only record API calls, ignore static assets + } +}); +``` + +**Playback with Fallback**: + +```typescript +await networkRecorder.setup(context, { + playback: { + fallback: true // Fall back to live requests if HAR entry missing + } +}); +``` + +**Custom HAR File Location**: + +```typescript +await networkRecorder.setup(context, { + harFile: { + harDir: 'recordings/api-calls', + baseName: 'user-journey', + organizeByTestFile: false // Optional: flatten directory structure + } +}); +``` + +**Directory Organization:** + +- `organizeByTestFile: true` (default): `har-files/test-file-name/baseName-test-title.har` +- `organizeByTestFile: false`: `har-files/baseName-test-title.har` + +### Example 4: Response Content Storage - Embed vs Attach + +**Context**: Choose how response content is stored in HAR files. + +**`embed` (Default - Recommended):** + +```typescript +await networkRecorder.setup(context, { + recording: { + content: 'embed' // Store content inline (default) + } +}); +``` + +**Pros:** + +- Single self-contained file - Easy to share, version control +- Better for small-medium responses (API JSON, HTML pages) +- HAR specification compliant + +**Cons:** + +- Larger HAR files +- Not ideal for large binary content (images, videos) + +**`attach` (Alternative):** + +```typescript +await networkRecorder.setup(context, { + recording: { + content: 'attach' // Store content separately + } +}); +``` + +**Pros:** + +- Smaller HAR files +- Better for large responses (images, videos, documents) + +**Cons:** + +- Multiple files to manage +- Harder to share + +**When to Use Each:** + +| Use `embed` (default) when | Use `attach` when | +|---------------------------|-------------------| +| Recording API responses (JSON, XML) | Recording large images, videos | +| Small to medium HTML pages | HAR file size >50MB | +| You want a single, portable file | Maximum disk efficiency needed | +| Sharing HAR files with team | Working with ZIP archive output | + +### Example 5: Cross-Environment Compatibility (URL Mapping) **Context**: Record in dev environment, play back in CI with different base URLs. -**Implementation**: +**The Problem**: HAR files contain URLs for the recording environment (e.g., `dev.example.com`). Playing back on a different environment fails. + +**Simple Hostname Mapping:** ```typescript -// playwright.config.ts - Map URLs for different environments -export default defineConfig({ - use: { - baseURL: process.env.CI ? 'https://app.ci.example.com' : 'http://localhost:3000', - }, -}); - -// Test works in both environments -test('cross-environment playback', async ({ page, context, networkRecorder }) => { - await networkRecorder.setup(context); - - // In dev: hits http://localhost:3000/api/movies - // In CI: HAR replays with https://app.ci.example.com/api/movies - await page.goto('/movies'); - - // Network recorder auto-maps URLs - await expect(page.getByTestId('movie-list')).toBeVisible(); +await networkRecorder.setup(context, { + playback: { + urlMapping: { + hostMapping: { + 'preview.example.com': 'dev.example.com', + 'staging.example.com': 'dev.example.com', + 'localhost:3000': 'dev.example.com' + } + } + } }); ``` -**Key Points**: - -- HAR files record absolute URLs -- Playback maps to current baseURL -- Same HAR works across environments -- No manual URL rewriting needed - -### Example 4: Automatic vs Manual Mode Control - -**Context**: Choose between environment-based switching or in-test mode control. - -**Implementation**: +**Pattern-Based Mapping (Recommended):** ```typescript -// Option 1: Environment variable (recommended for CI) -PW_NET_MODE=record npm run test:pw # Record traffic -PW_NET_MODE=playback npm run test:pw # Playback traffic - -// Option 2: In-test control (recommended for development) -process.env.PW_NET_MODE = 'record' // Set at top of test file - -test('my test', async ({ page, context, networkRecorder }) => { - await networkRecorder.setup(context) - // ... -}) - -// Option 3: Auto-fallback (record if HAR missing, else playback) -// This is the default behavior when PW_NET_MODE not set -test('auto mode', async ({ page, context, networkRecorder }) => { - await networkRecorder.setup(context) - // First run: auto-records - // Subsequent runs: auto-plays back -}) +await networkRecorder.setup(context, { + playback: { + urlMapping: { + patterns: [ + // Map any preview-XXXX subdomain to dev + { match: /preview-\d+\.example\.com/, replace: 'dev.example.com' } + ] + } + } +}); ``` -**Key Points**: +**Custom Function:** -- Three mode options: record, playback, auto -- `PW_NET_MODE` environment variable -- In-test `process.env.PW_NET_MODE` assignment -- Auto-fallback when no mode specified +```typescript +await networkRecorder.setup(context, { + playback: { + urlMapping: { + mapUrl: (url) => url.replace('staging.example.com', 'dev.example.com') + } + } +}); +``` + +**Complex Multi-Environment Example:** + +```typescript +await networkRecorder.setup(context, { + playback: { + urlMapping: { + hostMapping: { + 'localhost:3000': 'admin.seondev.space', + 'admin-staging.seon.io': 'admin.seondev.space', + 'admin.seon.io': 'admin.seondev.space', + }, + patterns: [ + { match: /admin-\d+\.seondev\.space/, replace: 'admin.seondev.space' }, + { match: /admin-staging-pr-\w+-\d\.seon\.io/, replace: 'admin.seondev.space' } + ] + } + } +}); +``` + +**Benefits:** + +- Record once on dev, all environments map back to recordings +- CORS headers automatically updated based on request origin +- Debug with: `LOG_LEVEL=debug npm run test` ## Why Use This Instead of Native Playwright? @@ -191,7 +330,7 @@ test('auto mode', async ({ page, context, networkRecorder }) => { | ~80 lines setup boilerplate | ~5 lines total | | Manual HAR file management | Automatic file organization | | Complex setup/teardown | Automatic cleanup via fixtures | -| **Read-only tests** | **Full CRUD support** | +| **Read-only tests only** | **Full CRUD support** | | **Stateless** | **Stateful mocking** | | Manual URL mapping | Automatic environment mapping | @@ -199,9 +338,132 @@ test('auto mode', async ({ page, context, networkRecorder }) => { Native Playwright HAR playback is stateless - a POST create followed by GET list won't show the created item. This utility intelligently tracks CRUD operations in memory to reflect state changes, making offline tests behave like real APIs. +## How Stateful CRUD Detection Works + +When in playback mode, the Network Recorder automatically analyzes your HAR file to detect CRUD patterns. If it finds: + +- Multiple GET requests to the same resource endpoint (e.g., `/movies`) +- Mutation operations (POST, PUT, DELETE) to those resources +- Evidence of state changes between identical requests + +It automatically switches from static HAR playback to an intelligent stateful mock that: + +- Maintains state across requests +- Auto-generates IDs for new resources +- Returns proper 404s for deleted resources +- Supports polling scenarios where state changes over time + +**This happens automatically - no configuration needed!** + +## API Reference + +### NetworkRecorder Methods + +| Method | Return Type | Description | +| -------------------- | ------------------------ | ----------------------------------------------------- | +| `setup(context)` | `Promise` | Sets up recording/playback on browser context | +| `cleanup()` | `Promise` | Flushes data to disk and cleans up memory | +| `getContext()` | `NetworkRecorderContext` | Gets current recorder context information | +| `getStatusMessage()` | `string` | Gets human-readable status message | +| `getHarStats()` | `Promise` | Gets HAR file statistics and metadata | + +### Understanding `cleanup()` + +The `cleanup()` method performs memory and resource cleanup - **it does NOT delete HAR files**: + +**What it does:** + +- Flushes recorded data to disk (writes HAR file in recording mode) +- Releases file locks +- Clears in-memory data +- Resets internal state + +**What it does NOT do:** + +- Delete HAR files from disk +- Remove recorded network traffic +- Clear browser context or cookies + +### Configuration Options + +```typescript +type NetworkRecorderConfig = { + harFile?: { + harDir?: string // Directory for HAR files (default: 'har-files') + baseName?: string // Base name for HAR files (default: 'network-traffic') + organizeByTestFile?: boolean // Organize by test file (default: true) + } + + recording?: { + content?: 'embed' | 'attach' // Response content handling (default: 'embed') + urlFilter?: string | RegExp // URL filter for recording + update?: boolean // Update existing HAR files (default: false) + } + + playback?: { + fallback?: boolean // Fall back to live requests (default: false) + urlFilter?: string | RegExp // URL filter for playback + updateMode?: boolean // Update mode during playback (default: false) + } + + forceMode?: 'record' | 'playback' | 'disabled' +} +``` + +## Environment Configuration + +Control the recording mode using the `PW_NET_MODE` environment variable: + +```bash +# Record mode - captures network traffic to HAR files +PW_NET_MODE=record npm run test:pw + +# Playback mode - replays network traffic from HAR files +PW_NET_MODE=playback npm run test:pw + +# Disabled mode - no network recording/playback +PW_NET_MODE=disabled npm run test:pw + +# Default behavior (when PW_NET_MODE is empty/unset) - same as disabled +npm run test:pw +``` + +**Tip**: We recommend setting `process.env.PW_NET_MODE` directly in your test file for better control. + +## Troubleshooting + +### HAR File Not Found + +If you see "HAR file not found" errors during playback: + +1. Ensure you've recorded the test first with `PW_NET_MODE=record` +2. Check the HAR file exists in the expected location (usually `har-files/`) +3. Enable fallback mode: `playback: { fallback: true }` + +### Authentication and Network Recording + +The network recorder works seamlessly with authentication: + +```typescript +test('Authenticated recording', async ({ page, context, authSession, networkRecorder }) => { + // First authenticate + await authSession.login('testuser', 'password'); + + // Then setup network recording with authenticated context + await networkRecorder.setup(context); + + // Test authenticated flows + await page.goto('/dashboard'); +}); +``` + +### Concurrent Test Issues + +The recorder includes built-in file locking for safe parallel execution. Each test gets its own HAR file based on the test name. + ## Integration with Other Utilities -**With interceptNetworkCall** (deterministic waits): +**With interceptNetworkCall (deterministic waits):** ```typescript test('use both utilities', async ({ page, context, networkRecorder, interceptNetworkCall }) => { @@ -228,7 +490,7 @@ test('use both utilities', async ({ page, context, networkRecorder, interceptNet ## Anti-Patterns -**❌ Mixing record and playback in same test:** +**DON'T mix record and playback in same test:** ```typescript process.env.PW_NET_MODE = 'record'; @@ -236,7 +498,7 @@ process.env.PW_NET_MODE = 'record'; process.env.PW_NET_MODE = 'playback'; // Don't switch mid-test ``` -**βœ… One mode per test:** +**DO use one mode per test:** ```typescript process.env.PW_NET_MODE = 'playback'; // Set once at top @@ -247,7 +509,7 @@ test('my test', async ({ page, context, networkRecorder }) => { }); ``` -**❌ Forgetting to call setup:** +**DON'T forget to call setup:** ```typescript test('broken', async ({ page, networkRecorder }) => { @@ -255,7 +517,7 @@ test('broken', async ({ page, networkRecorder }) => { }); ``` -**βœ… Always call setup before navigation:** +**DO always call setup before navigation:** ```typescript test('correct', async ({ page, context, networkRecorder }) => { diff --git a/src/modules/bmm/testarch/knowledge/overview.md b/src/modules/bmm/testarch/knowledge/overview.md index 8155d55f..a539f03f 100644 --- a/src/modules/bmm/testarch/knowledge/overview.md +++ b/src/modules/bmm/testarch/knowledge/overview.md @@ -2,7 +2,7 @@ ## Principle -Use production-ready, fixture-based utilities from `@seontechnologies/playwright-utils` for common Playwright testing patterns. Build test helpers as pure functions first, then wrap in framework-specific fixtures for composability and reuse. +Use production-ready, fixture-based utilities from `@seontechnologies/playwright-utils` for common Playwright testing patterns. Build test helpers as pure functions first, then wrap in framework-specific fixtures for composability and reuse. **Works equally well for pure API testing (no browser) and UI testing.** ## Rationale @@ -20,6 +20,7 @@ Writing Playwright utilities from scratch for every project leads to: - **Composable fixtures**: Use `mergeTests` to combine utilities - **TypeScript support**: Full type safety with generic types - **Comprehensive coverage**: API requests, auth, network, logging, file handling, burn-in +- **Backend-first mentality**: Most utilities work without a browser - pure API/service testing is a first-class use case ## Installation @@ -37,17 +38,19 @@ npm install -D @seontechnologies/playwright-utils ### Core Testing Utilities -| Utility | Purpose | Test Context | -| -------------------------- | ------------------------------------------ | ------------- | -| **api-request** | Typed HTTP client with schema validation | API tests | -| **network-recorder** | HAR record/playback for offline testing | UI tests | -| **auth-session** | Token persistence, multi-user auth | Both UI & API | -| **recurse** | Cypress-style polling for async conditions | Both UI & API | -| **intercept-network-call** | Network spy/stub with auto JSON parsing | UI tests | -| **log** | Playwright report-integrated logging | Both UI & API | -| **file-utils** | CSV/XLSX/PDF/ZIP reading & validation | Both UI & API | -| **burn-in** | Smart test selection with git diff | CI/CD | -| **network-error-monitor** | Automatic HTTP 4xx/5xx detection | UI tests | +| Utility | Purpose | Test Context | +| -------------------------- | ---------------------------------------------------- | ------------------ | +| **api-request** | Typed HTTP client with schema validation and retry | **API/Backend** | +| **recurse** | Polling for async operations, background jobs | **API/Backend** | +| **auth-session** | Token persistence, multi-user, service-to-service | **API/Backend/UI** | +| **log** | Playwright report-integrated logging | **API/Backend/UI** | +| **file-utils** | CSV/XLSX/PDF/ZIP reading & validation | **API/Backend/UI** | +| **burn-in** | Smart test selection with git diff | **CI/CD** | +| **network-recorder** | HAR record/playback for offline testing | UI only | +| **intercept-network-call** | Network spy/stub with auto JSON parsing | UI only | +| **network-error-monitor** | Automatic HTTP 4xx/5xx detection | UI only | + +**Note**: 6 of 9 utilities work without a browser. Only 3 are UI-specific (network-recorder, intercept-network-call, network-error-monitor). ## Design Patterns diff --git a/src/modules/bmm/testarch/knowledge/recurse.md b/src/modules/bmm/testarch/knowledge/recurse.md index aec553a1..d9536400 100644 --- a/src/modules/bmm/testarch/knowledge/recurse.md +++ b/src/modules/bmm/testarch/knowledge/recurse.md @@ -2,7 +2,7 @@ ## Principle -Use Cypress-style polling with Playwright's `expect.poll` to wait for asynchronous conditions. Provides configurable timeout, interval, logging, and post-polling callbacks with enhanced error categorization. +Use Cypress-style polling with Playwright's `expect.poll` to wait for asynchronous conditions. Provides configurable timeout, interval, logging, and post-polling callbacks with enhanced error categorization. **Ideal for backend testing**: polling API endpoints for job completion, database eventual consistency, message queue processing, and cache propagation. ## Rationale @@ -21,6 +21,29 @@ The `recurse` utility provides: - **Post-poll callbacks**: Process results after success - **Type-safe**: Full TypeScript generic support +## Quick Start + +```typescript +import { test } from '@seontechnologies/playwright-utils/recurse/fixtures'; + +test('wait for job completion', async ({ recurse, apiRequest }) => { + const { body } = await apiRequest({ + method: 'POST', + path: '/api/jobs', + body: { type: 'export' }, + }); + + // Poll until job completes + const result = await recurse( + () => apiRequest({ method: 'GET', path: `/api/jobs/${body.id}` }), + (response) => response.body.status === 'completed', + { timeout: 60000 } + ); + + expect(result.body.downloadUrl).toBeDefined(); +}); +``` + ## Pattern Examples ### Example 1: Basic Polling @@ -48,7 +71,7 @@ test('should wait for job completion', async ({ recurse, apiRequest }) => { timeout: 60000, // 60 seconds max interval: 2000, // Check every 2 seconds log: 'Waiting for export job to complete', - }, + } ); expect(result.body.downloadUrl).toBeDefined(); @@ -62,7 +85,7 @@ test('should wait for job completion', async ({ recurse, apiRequest }) => { - Options: timeout, interval, log message - Returns the value when predicate returns true -### Example 2: Polling with Assertions +### Example 2: Working with Assertions **Context**: Use assertions directly in predicate for more expressive tests. @@ -76,35 +99,76 @@ test('should poll with assertions', async ({ recurse, apiRequest }) => { body: { type: 'user-created', userId: '123' }, }); - // Poll with assertions in predicate + // Poll with assertions in predicate - no return true needed! await recurse( async () => { const { body } = await apiRequest({ method: 'GET', path: '/api/events/123' }); return body; }, (event) => { - // Use assertions instead of boolean returns + // If all assertions pass, predicate succeeds expect(event.processed).toBe(true); expect(event.timestamp).toBeDefined(); - // If assertions pass, predicate succeeds + // No need to return true - just let assertions pass }, - { timeout: 30000 }, + { timeout: 30000 } ); }); ``` -**Key Points**: +**Why no `return true` needed?** -- Predicate can use `expect()` assertions -- If assertions throw, polling continues -- If assertions pass, polling succeeds -- More expressive than boolean returns +The predicate checks for "truthiness" of the return value. But there's a catch - in JavaScript, an empty `return` (or no return) returns `undefined`, which is falsy! -### Example 3: Custom Error Messages +The utility handles this by checking if: -**Context**: Provide context-specific error messages for timeout failures. +1. The predicate didn't throw (assertions passed) +2. The return value was either `undefined` (implicit return) or truthy -**Implementation**: +So you can: + +```typescript +// Option 1: Use assertions only (recommended) +(event) => { + expect(event.processed).toBe(true); +}; + +// Option 2: Return boolean (also works) +(event) => event.processed === true; + +// Option 3: Mixed (assertions + explicit return) +(event) => { + expect(event.processed).toBe(true); + return true; +}; +``` + +### Example 3: Error Handling + +**Context**: Understanding the different error types. + +**Error Types:** + +```typescript +// RecurseTimeoutError - Predicate never returned true within timeout +// Contains last command value and predicate error +try { + await recurse(/* ... */); +} catch (error) { + if (error instanceof RecurseTimeoutError) { + console.log('Timed out. Last value:', error.lastCommandValue); + console.log('Last predicate error:', error.lastPredicateError); + } +} + +// RecurseCommandError - Command function threw an error +// The command itself failed (e.g., network error, API error) + +// RecursePredicateError - Predicate function threw (not from assertions failing) +// Logic error in your predicate code +``` + +**Custom Error Messages:** ```typescript test('custom error on timeout', async ({ recurse, apiRequest }) => { @@ -115,7 +179,7 @@ test('custom error on timeout', async ({ recurse, apiRequest }) => { { timeout: 10000, error: 'System failed to become ready within 10 seconds - check background workers', - }, + } ); } catch (error) { // Error message includes custom context @@ -125,13 +189,6 @@ test('custom error on timeout', async ({ recurse, apiRequest }) => { }); ``` -**Key Points**: - -- `error` option provides custom message -- Replaces default "Timed out after X ms" -- Include debugging hints in error message -- Helps diagnose failures faster - ### Example 4: Post-Polling Callback **Context**: Process or log results after successful polling. @@ -151,7 +208,7 @@ test('post-poll processing', async ({ recurse, apiRequest }) => { console.log(`Processed ${result.body.itemsProcessed} items`); return result.body; }, - }, + } ); expect(finalResult.itemsProcessed).toBeGreaterThan(0); @@ -165,7 +222,67 @@ test('post-poll processing', async ({ recurse, apiRequest }) => { - Can transform or log results - Return value becomes final `recurse` result -### Example 5: Integration with API Request (Common Pattern) +### Example 5: UI Testing Scenarios + +**Context**: Wait for UI elements to reach a specific state through polling. + +**Implementation**: + +```typescript +test('table data loads', async ({ page, recurse }) => { + await page.goto('/reports'); + + // Poll for table rows to appear + await recurse( + async () => page.locator('table tbody tr').count(), + (count) => count >= 10, // Wait for at least 10 rows + { + timeout: 15000, + interval: 500, + log: 'Waiting for table data to load', + } + ); + + // Now safe to interact with table + await page.locator('table tbody tr').first().click(); +}); +``` + +### Example 6: Event-Based Systems (Kafka/Message Queues) + +**Context**: Testing eventual consistency with message queue processing. + +**Implementation**: + +```typescript +test('kafka event processed', async ({ recurse, apiRequest }) => { + // Trigger action that publishes Kafka event + await apiRequest({ + method: 'POST', + path: '/api/orders', + body: { productId: 'ABC123', quantity: 2 }, + }); + + // Poll for downstream effect of Kafka consumer processing + const inventoryResult = await recurse( + () => apiRequest({ method: 'GET', path: '/api/inventory/ABC123' }), + (res) => { + // Assumes test fixture seeds inventory at 100; in production tests, + // fetch baseline first and assert: expect(res.body.available).toBe(baseline - 2) + expect(res.body.available).toBeLessThanOrEqual(98); + }, + { + timeout: 30000, // Kafka processing may take time + interval: 1000, + log: 'Waiting for Kafka event to be processed', + } + ); + + expect(inventoryResult.body.lastOrderId).toBeDefined(); +}); +``` + +### Example 7: Integration with API Request (Common Pattern) **Context**: Most common use case - polling API endpoints for state changes. @@ -193,7 +310,7 @@ test('end-to-end polling', async ({ apiRequest, recurse }) => { timeout: 120000, // 2 minutes for large imports interval: 5000, // Check every 5 seconds log: `Polling import ${createResp.importId}`, - }, + } ); expect(importResult.body.rowsImported).toBeGreaterThan(1000); @@ -208,20 +325,26 @@ test('end-to-end polling', async ({ apiRequest, recurse }) => { - Complex predicates with multiple conditions - Logging shows polling progress in test reports -## Enhanced Error Types +## API Reference -The utility categorizes errors for easier debugging: +### RecurseOptions -```typescript -// TimeoutError - Predicate never returned true -Error: Polling timed out after 30000ms: Job never completed +| Option | Type | Default | Description | +| ---------- | ------------------ | ----------- | ------------------------------------ | +| `timeout` | `number` | `30000` | Maximum time to wait (ms) | +| `interval` | `number` | `1000` | Time between polls (ms) | +| `log` | `string` | `undefined` | Message logged on each poll | +| `error` | `string` | `undefined` | Custom error message for timeout | +| `post` | `(result: T) => R` | `undefined` | Callback after successful poll | +| `delay` | `number` | `0` | Initial delay before first poll (ms) | -// CommandError - Command function threw -Error: Command failed: Request failed with status 500 +### Error Types -// PredicateError - Predicate function threw (not from assertions) -Error: Predicate failed: Cannot read property 'status' of undefined -``` +| Error Type | When Thrown | Properties | +| ----------------------- | --------------------------------------- | ---------------------------------------- | +| `RecurseTimeoutError` | Predicate never passed within timeout | `lastCommandValue`, `lastPredicateError` | +| `RecurseCommandError` | Command function threw an error | `cause` (original error) | +| `RecursePredicateError` | Predicate threw (not assertion failure) | `cause` (original error) | ## Comparison with Vanilla Playwright @@ -236,11 +359,11 @@ Error: Predicate failed: Cannot read property 'status' of undefined **Use recurse for:** -- βœ… Background job completion -- βœ… Webhook/event processing -- βœ… Database eventual consistency -- βœ… Cache propagation -- βœ… State machine transitions +- Background job completion +- Webhook/event processing +- Database eventual consistency +- Cache propagation +- State machine transitions **Stick with vanilla expect.poll for:** @@ -250,13 +373,15 @@ Error: Predicate failed: Cannot read property 'status' of undefined ## Related Fragments +- `api-testing-patterns.md` - Comprehensive pure API testing patterns - `api-request.md` - Combine for API endpoint polling - `overview.md` - Fixture composition patterns - `fixtures-composition.md` - Using with mergeTests +- `contract-testing.md` - Contract testing with async verification ## Anti-Patterns -**❌ Using hard waits instead of polling:** +**DON'T use hard waits instead of polling:** ```typescript await page.click('#export'); @@ -264,33 +389,33 @@ await page.waitForTimeout(5000); // Arbitrary wait expect(await page.textContent('#status')).toBe('Ready'); ``` -**βœ… Poll for actual condition:** +**DO poll for actual condition:** ```typescript await page.click('#export'); await recurse( () => page.textContent('#status'), (status) => status === 'Ready', - { timeout: 10000 }, + { timeout: 10000 } ); ``` -**❌ Polling too frequently:** +**DON'T poll too frequently:** ```typescript await recurse( () => apiRequest({ method: 'GET', path: '/status' }), (res) => res.body.ready, - { interval: 100 }, // Hammers API every 100ms! + { interval: 100 } // Hammers API every 100ms! ); ``` -**βœ… Reasonable interval for API calls:** +**DO use reasonable interval for API calls:** ```typescript await recurse( () => apiRequest({ method: 'GET', path: '/status' }), (res) => res.body.ready, - { interval: 2000 }, // Check every 2 seconds (reasonable) + { interval: 2000 } // Check every 2 seconds (reasonable) ); ``` diff --git a/src/modules/bmm/testarch/tea-index.csv b/src/modules/bmm/testarch/tea-index.csv index cf1efd67..4546d351 100644 --- a/src/modules/bmm/testarch/tea-index.csv +++ b/src/modules/bmm/testarch/tea-index.csv @@ -1,33 +1,34 @@ id,name,description,tags,fragment_file fixture-architecture,Fixture Architecture,"Composable fixture patterns (pure function β†’ fixture β†’ merge) and reuse rules","fixtures,architecture,playwright,cypress",knowledge/fixture-architecture.md -network-first,Network-First Safeguards,"Intercept-before-navigate workflow, HAR capture, deterministic waits, edge mocking","network,stability,playwright,cypress",knowledge/network-first.md -data-factories,Data Factories and API Setup,"Factories with overrides, API seeding, cleanup discipline","data,factories,setup,api",knowledge/data-factories.md +network-first,Network-First Safeguards,"Intercept-before-navigate workflow, HAR capture, deterministic waits, edge mocking","network,stability,playwright,cypress,ui",knowledge/network-first.md +data-factories,Data Factories and API Setup,"Factories with overrides, API seeding, cleanup discipline","data,factories,setup,api,backend,seeding",knowledge/data-factories.md component-tdd,Component TDD Loop,"Redβ†’greenβ†’refactor workflow, provider isolation, accessibility assertions","component-testing,tdd,ui",knowledge/component-tdd.md playwright-config,Playwright Config Guardrails,"Environment switching, timeout standards, artifact outputs","playwright,config,env",knowledge/playwright-config.md ci-burn-in,CI and Burn-In Strategy,"Staged jobs, shard orchestration, burn-in loops, artifact policy","ci,automation,flakiness",knowledge/ci-burn-in.md selective-testing,Selective Test Execution,"Tag/grep usage, spec filters, diff-based runs, promotion rules","risk-based,selection,strategy",knowledge/selective-testing.md feature-flags,Feature Flag Governance,"Enum management, targeting helpers, cleanup, release checklists","feature-flags,governance,launchdarkly",knowledge/feature-flags.md -contract-testing,Contract Testing Essentials,"Pact publishing, provider verification, resilience coverage","contract-testing,pact,api",knowledge/contract-testing.md +contract-testing,Contract Testing Essentials,"Pact publishing, provider verification, resilience coverage","contract-testing,pact,api,backend,microservices,service-contract",knowledge/contract-testing.md email-auth,Email Authentication Testing,"Magic link extraction, state preservation, caching, negative flows","email-authentication,security,workflow",knowledge/email-auth.md -error-handling,Error Handling Checks,"Scoped exception handling, retry validation, telemetry logging","resilience,error-handling,stability",knowledge/error-handling.md -visual-debugging,Visual Debugging Toolkit,"Trace viewer usage, artifact expectations, accessibility integration","debugging,dx,tooling",knowledge/visual-debugging.md +error-handling,Error Handling Checks,"Scoped exception handling, retry validation, telemetry logging","resilience,error-handling,stability,api,backend",knowledge/error-handling.md +visual-debugging,Visual Debugging Toolkit,"Trace viewer usage, artifact expectations, accessibility integration","debugging,dx,tooling,ui",knowledge/visual-debugging.md risk-governance,Risk Governance,"Scoring matrix, category ownership, gate decision rules","risk,governance,gates",knowledge/risk-governance.md probability-impact,Probability and Impact Scale,"Shared definitions for scoring matrix and gate thresholds","risk,scoring,scale",knowledge/probability-impact.md test-quality,Test Quality Definition of Done,"Execution limits, isolation rules, green criteria","quality,definition-of-done,tests",knowledge/test-quality.md nfr-criteria,NFR Review Criteria,"Security, performance, reliability, maintainability status definitions","nfr,assessment,quality",knowledge/nfr-criteria.md -test-levels,Test Levels Framework,"Guidelines for choosing unit, integration, or end-to-end coverage","testing,levels,selection",knowledge/test-levels-framework.md +test-levels,Test Levels Framework,"Guidelines for choosing unit, integration, or end-to-end coverage","testing,levels,selection,api,backend,ui",knowledge/test-levels-framework.md test-priorities,Test Priorities Matrix,"P0–P3 criteria, coverage targets, execution ordering","testing,prioritization,risk",knowledge/test-priorities-matrix.md test-healing-patterns,Test Healing Patterns,"Common failure patterns and automated fixes","healing,debugging,patterns",knowledge/test-healing-patterns.md -selector-resilience,Selector Resilience,"Robust selector strategies and debugging techniques","selectors,locators,debugging",knowledge/selector-resilience.md +selector-resilience,Selector Resilience,"Robust selector strategies and debugging techniques","selectors,locators,debugging,ui",knowledge/selector-resilience.md timing-debugging,Timing Debugging,"Race condition identification and deterministic wait fixes","timing,async,debugging",knowledge/timing-debugging.md -overview,Playwright Utils Overview,"Installation, design principles, fixture patterns","playwright-utils,fixtures",knowledge/overview.md -api-request,API Request,"Typed HTTP client, schema validation","api,playwright-utils",knowledge/api-request.md -network-recorder,Network Recorder,"HAR record/playback, CRUD detection","network,playwright-utils",knowledge/network-recorder.md -auth-session,Auth Session,"Token persistence, multi-user","auth,playwright-utils",knowledge/auth-session.md -intercept-network-call,Intercept Network Call,"Network spy/stub, JSON parsing","network,playwright-utils",knowledge/intercept-network-call.md -recurse,Recurse Polling,"Async polling, condition waiting","polling,playwright-utils",knowledge/recurse.md -log,Log Utility,"Report logging, structured output","logging,playwright-utils",knowledge/log.md -file-utils,File Utilities,"CSV/XLSX/PDF/ZIP validation","files,playwright-utils",knowledge/file-utils.md -burn-in,Burn-in Runner,"Smart test selection, git diff","ci,playwright-utils",knowledge/burn-in.md -network-error-monitor,Network Error Monitor,"HTTP 4xx/5xx detection","monitoring,playwright-utils",knowledge/network-error-monitor.md -fixtures-composition,Fixtures Composition,"mergeTests composition patterns","fixtures,playwright-utils",knowledge/fixtures-composition.md +overview,Playwright Utils Overview,"Installation, design principles, fixture patterns for API and UI testing","playwright-utils,fixtures,api,backend,ui",knowledge/overview.md +api-request,API Request,"Typed HTTP client, schema validation, retry logic for API and service testing","api,backend,service-testing,api-testing,playwright-utils",knowledge/api-request.md +network-recorder,Network Recorder,"HAR record/playback, CRUD detection for offline UI testing","network,playwright-utils,ui,har",knowledge/network-recorder.md +auth-session,Auth Session,"Token persistence, multi-user, API and browser authentication","auth,playwright-utils,api,backend,jwt,token",knowledge/auth-session.md +intercept-network-call,Intercept Network Call,"Network spy/stub, JSON parsing for UI tests","network,playwright-utils,ui",knowledge/intercept-network-call.md +recurse,Recurse Polling,"Async polling for API responses, background jobs, eventual consistency","polling,playwright-utils,api,backend,async,eventual-consistency",knowledge/recurse.md +log,Log Utility,"Report logging, structured output for API and UI tests","logging,playwright-utils,api,ui",knowledge/log.md +file-utils,File Utilities,"CSV/XLSX/PDF/ZIP validation for API exports and UI downloads","files,playwright-utils,api,backend,ui",knowledge/file-utils.md +burn-in,Burn-in Runner,"Smart test selection, git diff for CI optimization","ci,playwright-utils",knowledge/burn-in.md +network-error-monitor,Network Error Monitor,"HTTP 4xx/5xx detection for UI tests","monitoring,playwright-utils,ui",knowledge/network-error-monitor.md +fixtures-composition,Fixtures Composition,"mergeTests composition patterns for combining utilities","fixtures,playwright-utils",knowledge/fixtures-composition.md +api-testing-patterns,API Testing Patterns,"Pure API test patterns without browser: service testing, microservices, GraphQL","api,backend,service-testing,api-testing,microservices,graphql,no-browser",knowledge/api-testing-patterns.md