Commit Graph

5 Commits

Author SHA1 Message Date
Romuald Członkowski
de2abaf89d refactor: streamline test suite — 33 fewer files, 11.9x faster (#670)
* refactor: streamline test suite - cut 33 files, enable parallel execution (11.9x speedup)

Remove duplicate, low-value, and fragmented test files while preserving
all meaningful coverage. Enable parallel test execution and remove
the entire benchmark infrastructure.

Key changes:
- Consolidate workflow-validator tests (13 files -> 3)
- Consolidate config-validator tests (9 files -> 3)
- Consolidate telemetry tests (11 files -> 6)
- Merge AI validator tests (2 files -> 1)
- Remove example/demo test files, mock-testing files, and already-skipped tests
- Remove benchmark infrastructure (10 files, CI workflow, 4 npm scripts)
- Enable parallel test execution (remove singleThread: true)
- Remove retry:2 that was masking flaky tests
- Slim CI publish-results job

Results: 224 -> 191 test files, 4690 -> 4303 tests, 121K -> 106K lines
Local runtime: 319s -> 27s (11.9x speedup)

Conceived by Romuald Członkowski - www.aiadvisors.pl/en

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor: absorb config-validator satellite tests into consolidated file

The previous commit deleted 4 config-validator satellite files. This
properly merges their unique tests into the consolidated config-validator.test.ts,
recovering 89 tests that were dropped during the bulk deletion.

Deduplicates 5 tests that existed in both the satellite files and the
security test file.

Conceived by Romuald Członkowski - www.aiadvisors.pl/en

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: delete missed benchmark-pr.yml workflow, fix flaky session test

- Remove benchmark-pr.yml that referenced deleted benchmark:ci script
- Fix session-persistence round-trip test using timestamps closer to
  now to avoid edge cases exposed by removing retry:2

Conceived by Romuald Członkowski - www.aiadvisors.pl/en

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: rebuild FTS5 index after database rebuild to prevent stale rowid refs

The FTS5 content-synced index could retain phantom rowid references from
previous rebuild cycles, causing 'missing row N from content table'
errors on MATCH queries.

- Add explicit FTS5 rebuild command in rebuild script after all nodes saved
- Add FTS5 rebuild in test beforeAll as defense-in-depth
- Rebuild nodes.db with consistent FTS5 index

Conceived by Romuald Członkowski - www.aiadvisors.pl/en

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: use recent timestamps in all session persistence tests

Session round-trip tests used timestamps 5-10 minutes in the past which
could fail under CI load when combined with session timeout validation.
Use timestamps 30 seconds in the past for all valid-session test data.

Conceived by Romuald Członkowski - www.aiadvisors.pl/en

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-27 14:22:22 +01:00
czlonkowski
caf9383ba1 test: Add comprehensive edge case coverage for telemetry quick wins
Added 20 edge case tests based on code review recommendations:

**Duplicate ID Validation (4 tests)**:
- Multiple duplicate IDs (3+ nodes with same ID)
- Duplicate IDs with same node type
- Duplicate IDs with empty/null node names
- Duplicate IDs with missing node properties

**AI Agent Validator (16 tests)**:

maxIterations edge cases (7 tests):
- Boundary values: 0 (reject), 1 (accept), 51 (warn), MAX_SAFE_INTEGER (warn)
- Invalid types: NaN (reject), negative decimal (reject)
- Threshold testing: 50 vs 51

promptType validation (4 tests):
- Whitespace-only text (reject)
- Very long text 3200+ chars (accept)
- undefined/null text (reject)

System message validation (5 tests):
- Empty/whitespace messages (suggest adding)
- Very long messages >1000 chars (accept)
- Special characters, emojis, unicode (accept)
- Multi-line formatting (accept)
- Boundary: 19 chars (warn), 20 chars (accept)

**Test Quality Improvements**:
- Fixed flaky system message test (changed from expect.stringContaining to .some())
- All tests are deterministic
- Comprehensive inline comments
- Follows existing test patterns

All 20 new tests passing. Zero regressions.

Concieved by Romuald Członkowski - www.aiadvisors.pl/en
2025-11-08 18:49:59 +01:00
czlonkowski
60ab66d64d feat: telemetry-driven quick wins to reduce AI agent validation errors by 30-40%
Enhanced tools documentation, duplicate ID errors, and AI Agent validator based on telemetry analysis of 593 validation errors across 3 categories:
- 378 errors: Duplicate node IDs (64%)
- 179 errors: AI Agent configuration (30%)
- 36 errors: Other validations (6%)

Quick Win #1: Enhanced tools documentation (src/mcp/tools-documentation.ts)
- Added prominent warnings to call get_node_essentials() FIRST before configuring nodes
- Emphasized 5KB vs 100KB+ size difference between essentials and full info
- Updated workflow patterns to prioritize essentials over get_node_info

Quick Win #2: Improved duplicate ID error messages (src/services/workflow-validator.ts)
- Added crypto import for UUID generation examples
- Enhanced error messages with node indices, names, and types
- Included crypto.randomUUID() example in error messages
- Helps AI agents understand EXACTLY which nodes conflict and how to fix

Quick Win #3: Added AI Agent node-specific validator (src/services/node-specific-validators.ts)
- Validates prompt configuration (promptType + text requirement)
- Checks maxIterations bounds (1-50 recommended)
- Suggests error handling (onError + retryOnFail)
- Warns about high iteration limits (cost/performance impact)
- Integrated into enhanced-config-validator.ts

Test Coverage:
- Added duplicate ID validation tests (workflow-validator.test.ts)
- Added AI Agent validator tests (node-specific-validators.test.ts:2312-2491)
- All new tests passing (3527 total passing)

Version: 2.22.12 → 2.22.13

Expected Impact: 30-40% reduction in AI agent validation errors

Technical Details:
- Telemetry analysis: 593 validation errors (Dec 2024 - Jan 2025)
- 100% error recovery rate maintained (validation working correctly)
- Root cause: Documentation/guidance gaps, not validation logic failures
- Solution: Proactive guidance at decision points

References:
- Telemetry analysis findings
- Issue #392 (helpful error messages pattern)
- Existing Slack validator pattern (node-specific-validators.ts:98-230)

Concieved by Romuald Członkowski - www.aiadvisors.pl/en
2025-11-08 18:07:26 +01:00
czlonkowski
6699a1d34c test: implement comprehensive testing improvements from PR #104 review
Major improvements based on comprehensive test suite review:

Test Fixes:
- Fix all 78 failing tests across logger, MSW, and validator tests
- Fix console spy management in logger tests with proper DEBUG env handling
- Fix MSW test environment restoration in session-management.test.ts
- Fix workflow validator tests by adding proper node connections
- Fix mock setup issues in edge case tests

Test Organization:
- Split large config-validator.test.ts (1,075 lines) into 4 focused files
- Rename 63+ tests to follow "should X when Y" naming convention
- Add comprehensive edge case test files for all major validators
- Create tests/README.md with testing guidelines and best practices

New Features:
- Add ConfigValidator.validateBatch() method for bulk validation
- Add edge case coverage for null/undefined, boundaries, invalid data
- Add CI-aware performance test timeouts
- Add JSDoc comments to test utilities and factories
- Add workflow duplicate node name validation tests

Results:
- All tests passing: 1,356 passed, 19 skipped
- Test coverage: 85.34% statements, 85.3% branches
- From 78 failures to 0 failures

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-07-30 13:44:35 +02:00
czlonkowski
b49043171e test: Phase 3 - Create comprehensive unit tests for services
- Add unit tests for ConfigValidator with 44 test cases (95.21% coverage)
- Create test templates for 7 major services:
  - PropertyFilter (23 tests)
  - ExampleGenerator (35 tests)
  - TaskTemplates (36 tests)
  - PropertyDependencies (21 tests)
  - EnhancedConfigValidator (8 tests)
  - ExpressionValidator (11 tests)
  - WorkflowValidator (9 tests)
- Fix service implementations to handle edge cases discovered during testing
- Add comprehensive testing documentation:
  - Phase 3 testing plan with priorities and timeline
  - Context documentation for quick implementation
  - Mocking strategy for complex dependencies
- All 262 tests now passing (up from 75)

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-07-28 14:15:09 +02:00