automaker/apps/ui/tests/e2e-testing-guide.md

E2E Testing Guide

Best practices and patterns for writing reliable, non-flaky Playwright e2e tests in this codebase.

Core Principles

1. No arbitrary timeouts - Never use page.waitForTimeout(). Always wait for specific conditions.
2. Use data-testid attributes - Prefer [data-testid="..."] selectors over CSS classes or text content.
3. Clean up after tests - Use unique temp directories and clean them up in afterAll.
4. Test isolation - Each test should be independent and not rely on state from other tests.

Setting Up Test State

Use Setup Utilities (Recommended)

Use the provided utility functions to set up localStorage state. These utilities hide the internal store structure and version details, making tests more maintainable.

import { setupWelcomeView, setupRealProject } from './utils';

// Show welcome view with workspace directory configured
await setupWelcomeView(page, { workspaceDir: TEST_TEMP_DIR });

// Show welcome view with recent projects
await setupWelcomeView(page, {
  workspaceDir: TEST_TEMP_DIR,
  recentProjects: [
    {
      id: 'project-123',
      name: 'My Project',
      path: '/path/to/project',
      lastOpened: new Date().toISOString(),
    },
  ],
});

// Set up a real project on the filesystem
await setupRealProject(page, projectPath, projectName, {
  setAsCurrent: true, // Opens board view (default)
});

Why Use Utilities Instead of Raw localStorage

1. Version management - Store versions are centralized in one place
2. Less brittle - If store structure changes, update one file instead of every test
3. Cleaner tests - Focus on test logic, not setup boilerplate
4. Type safety - Utilities provide typed interfaces for test data

Manual LocalStorage Setup (Advanced)

If you need custom setup not covered by utilities, use page.addInitScript(). Store versions are defined in tests/utils/project/setup.ts:

• APP_STORE: version 2 (matches app-store.ts)
• SETUP_STORE: version 0 (matches setup-store.ts default)

Temp Directory Management

Create unique temp directories for test isolation:

import { createTempDirPath, cleanupTempDir } from './utils';

const TEST_TEMP_DIR = createTempDirPath('my-test-name');

test.describe('My Tests', () => {
  test.beforeAll(async () => {
    if (!fs.existsSync(TEST_TEMP_DIR)) {
      fs.mkdirSync(TEST_TEMP_DIR, { recursive: true });
    }
  });

  test.afterAll(async () => {
    cleanupTempDir(TEST_TEMP_DIR);
  });
});

Waiting for Elements

Prefer toBeVisible() over waitForSelector()

// Good - uses Playwright's auto-waiting with expect
await expect(page.locator('[data-testid="welcome-view"]')).toBeVisible({ timeout: 10000 });

// Avoid - manual waiting
await page.waitForSelector('[data-testid="welcome-view"]');

Wait for page load after navigation

Important: Use load state, NOT networkidle. This app has persistent connections (websockets, polling) that prevent the network from ever becoming "idle", causing networkidle to timeout.

await page.goto('/');
await page.waitForLoadState('load');

// Then wait for specific elements to verify the page is ready
await expect(page.locator('[data-testid="board-view"]')).toBeVisible({ timeout: 10000 });

Why not networkidle?

• networkidle requires no network activity for 500ms
• Modern SPAs with real-time features (websockets, polling, SSE) never reach this state
• Using networkidle causes 30+ second timeouts and flaky tests
• The load state fires when the page finishes loading, which is sufficient
• Always follow up with element visibility checks for reliability

Use appropriate timeouts

• Quick UI updates: 5000ms (default)
• Page loads/navigation: 10000ms
• Async operations (API calls, file system): 15000ms

// Fast UI element
await expect(button).toBeVisible();

// Page load
await expect(page.locator('[data-testid="board-view"]')).toBeVisible({ timeout: 10000 });

// Async operation completion
await expect(page.locator('[data-testid="board-view"]')).toBeVisible({ timeout: 15000 });

Element Selection

Use data-testid attributes

// Good - stable selector
const button = page.locator('[data-testid="create-new-project"]');

// Avoid - brittle selectors
const button = page.locator('.btn-primary');
const button = page.getByText('Create');

Scope selectors when needed

When text appears in multiple places, scope to a parent:

// Bad - might match multiple elements
await expect(page.getByText(projectName)).toBeVisible();

// Good - scoped to specific container
await expect(page.locator('[data-testid="project-selector"]').getByText(projectName)).toBeVisible();

Handle strict mode violations

If a selector matches multiple elements:

// Use .first() if you need the first match
await page.locator('[data-testid="item"]').first().click();

// Or scope to a unique parent
await page.locator('[data-testid="sidebar"]').locator('[data-testid="item"]').click();

Clicking Elements

Always verify visibility before clicking

const button = page.locator('[data-testid="submit"]');
await expect(button).toBeVisible();
await button.click();

Handle dialogs that may close quickly

Some dialogs may appear briefly or auto-close. Don't rely on clicking them:

// Instead of trying to close a dialog that might disappear:
// await expect(dialog).toBeVisible();
// await closeButton.click();  // May fail if dialog closes first

// Just verify the end state:
await expect(page.locator('[data-testid="board-view"]')).toBeVisible({ timeout: 15000 });

Filesystem Verification

Verify files were created after async operations:

// Wait for UI to confirm operation completed first
await expect(page.locator('[data-testid="board-view"]')).toBeVisible({ timeout: 15000 });

// Then verify filesystem
const projectPath = path.join(TEST_TEMP_DIR, projectName);
expect(fs.existsSync(projectPath)).toBe(true);

const appSpecPath = path.join(projectPath, '.automaker', 'app_spec.txt');
expect(fs.existsSync(appSpecPath)).toBe(true);

const content = fs.readFileSync(appSpecPath, 'utf-8');
expect(content).toContain(projectName);

Test Structure

Use descriptive test names

test('should create a new blank project from welcome view', async ({ page }) => {
  // ...
});

Group related tests with describe blocks

test.describe('Project Creation', () => {
  test('should create a new blank project from welcome view', ...);
  test('should create a project from template', ...);
});

Use serial mode when tests depend on each other

test.describe.configure({ mode: 'serial' });

Common Patterns

Waiting for either of two outcomes

When multiple outcomes are possible (e.g., dialog or direct navigation):

// Wait for either the dialog or the board view
await Promise.race([
  initDialog.waitFor({ state: 'visible', timeout: 10000 }).catch(() => {}),
  boardView.waitFor({ state: 'visible', timeout: 10000 }).catch(() => {}),
]);

// Then handle whichever appeared
if (await initDialog.isVisible()) {
  await closeButton.click();
}

await expect(boardView).toBeVisible();

Generating unique test data

const projectName = `test-project-${Date.now()}`;

Running Tests

# Run all tests
npm run test

# Run specific test file
npm run test -- project-creation.spec.ts

# Run with headed browser (see what's happening)
npm run test:headed -- project-creation.spec.ts

# Run multiple times to check for flakiness
npm run test -- project-creation.spec.ts --repeat-each=5

Debugging Failed Tests

1. Check the screenshot in test-results/
2. Read the error context markdown file in test-results/
3. Run with --headed to watch the test
4. Add await page.pause() to pause execution at a specific point

Common Pitfalls

Timeout on waitForLoadState('networkidle')

If tests timeout waiting for network idle, the app likely has persistent connections. Use load state instead:

// Bad - will timeout with persistent connections
await page.waitForLoadState('networkidle');

// Good - completes when page loads
await page.waitForLoadState('load');
await expect(page.locator('[data-testid="my-element"]')).toBeVisible();

Port conflicts

If you see "Port 3008 is already in use", kill the process:

lsof -ti:3008 | xargs kill -9

Available Test Utilities

Import from ./utils:

State Setup Utilities

• setupWelcomeView(page, options?) - Set up empty state showing welcome view
  • options.workspaceDir - Pre-configure workspace directory
  • options.recentProjects - Add projects to recent list (not current)
• setupRealProject(page, path, name, options?) - Set up state with a real filesystem project
  • options.setAsCurrent - Open board view (default: true)
  • options.additionalProjects - Add more projects to list
• setupMockProject(page) - Set up mock project for unit-style tests
• setupComplete(page) - Mark setup wizard as complete

Filesystem Utilities

• createTempDirPath(prefix) - Create unique temp directory path
• cleanupTempDir(path) - Remove temp directory
• createTestGitRepo(tempDir) - Create a git repo for testing

Waiting Utilities

• waitForNetworkIdle(page) - Wait for page to load (uses load state, not networkidle)
• waitForElement(page, testId) - Wait for element by test ID
• waitForBoardView(page) - Navigate to board and wait for it to be visible

Async File Verification

Use expect().toPass() for polling filesystem operations:

await expect(async () => {
  expect(fs.existsSync(filePath)).toBe(true);
}).toPass({ timeout: 10000 });

See tests/utils/index.ts for the full list of available utilities.