feat: add comprehensive performance benchmark tracking system
- Create benchmark test suites for critical operations: - Node loading performance - Database query performance - Search operations performance - Validation performance - MCP tool execution performance - Add GitHub Actions workflow for benchmark tracking: - Runs on push to main and PRs - Uses github-action-benchmark for historical tracking - Comments on PRs with performance results - Alerts on >10% performance regressions - Stores results in GitHub Pages - Create benchmark infrastructure: - Custom Vitest benchmark configuration - JSON reporter for CI results - Result formatter for github-action-benchmark - Performance threshold documentation - Add supporting utilities: - SQLiteStorageService for benchmark database setup - MCPEngine wrapper for testing MCP tools - Test factories for generating benchmark data - Enhanced NodeRepository with benchmark methods - Document benchmark system: - Comprehensive benchmark guide in docs/BENCHMARKS.md - Performance thresholds in .github/BENCHMARK_THRESHOLDS.md - README for benchmarks directory - Integration with existing test suite The benchmark system will help monitor performance over time and catch regressions before they reach production. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
czlonkowski
185
docs/BENCHMARKS.md
Normal file
185
docs/BENCHMARKS.md
Normal file
@@ -0,0 +1,185 @@
|
||||
# n8n-mcp Performance Benchmarks
|
||||
|
||||
## Overview
|
||||
|
||||
The n8n-mcp project includes comprehensive performance benchmarks to ensure optimal performance across all critical operations. These benchmarks help identify performance regressions and guide optimization efforts.
|
||||
|
||||
## Running Benchmarks
|
||||
|
||||
### Local Development
|
||||
|
||||
```bash
|
||||
# Run all benchmarks
|
||||
npm run benchmark
|
||||
|
||||
# Run in watch mode
|
||||
npm run benchmark:watch
|
||||
|
||||
# Run with UI
|
||||
npm run benchmark:ui
|
||||
|
||||
# Run specific benchmark suite
|
||||
npm run benchmark tests/benchmarks/node-loading.bench.ts
|
||||
```
|
||||
|
||||
### Continuous Integration
|
||||
|
||||
Benchmarks run automatically on:
|
||||
- Every push to `main` branch
|
||||
- Every pull request
|
||||
- Manual workflow dispatch
|
||||
|
||||
Results are:
|
||||
- Tracked over time using GitHub Actions
|
||||
- Displayed in PR comments
|
||||
- Available at: https://czlonkowski.github.io/n8n-mcp/benchmarks/
|
||||
|
||||
## Benchmark Suites
|
||||
|
||||
### 1. Node Loading Performance
|
||||
Tests the performance of loading n8n node packages and parsing their metadata.
|
||||
|
||||
**Key Metrics:**
|
||||
- Package loading time (< 100ms target)
|
||||
- Individual node file loading (< 5ms target)
|
||||
- Package.json parsing (< 1ms target)
|
||||
|
||||
### 2. Database Query Performance
|
||||
Measures database operation performance including queries, inserts, and updates.
|
||||
|
||||
**Key Metrics:**
|
||||
- Node retrieval by type (< 5ms target)
|
||||
- Search operations (< 50ms target)
|
||||
- Bulk operations (< 100ms target)
|
||||
|
||||
### 3. Search Operations
|
||||
Tests various search modes and their performance characteristics.
|
||||
|
||||
**Key Metrics:**
|
||||
- Simple word search (< 10ms target)
|
||||
- Multi-word OR search (< 20ms target)
|
||||
- Fuzzy search (< 50ms target)
|
||||
|
||||
### 4. Validation Performance
|
||||
Measures configuration and workflow validation speed.
|
||||
|
||||
**Key Metrics:**
|
||||
- Simple config validation (< 1ms target)
|
||||
- Complex config validation (< 10ms target)
|
||||
- Workflow validation (< 50ms target)
|
||||
|
||||
### 5. MCP Tool Execution
|
||||
Tests the overhead of MCP tool execution.
|
||||
|
||||
**Key Metrics:**
|
||||
- Tool invocation overhead (< 5ms target)
|
||||
- Complex tool operations (< 50ms target)
|
||||
|
||||
## Performance Targets
|
||||
|
||||
| Operation Category | Target | Warning | Critical |
|
||||
|-------------------|--------|---------|----------|
|
||||
| Node Loading | < 100ms | > 150ms | > 200ms |
|
||||
| Database Query | < 5ms | > 10ms | > 20ms |
|
||||
| Search (simple) | < 10ms | > 20ms | > 50ms |
|
||||
| Search (complex) | < 50ms | > 100ms | > 200ms |
|
||||
| Validation | < 10ms | > 20ms | > 50ms |
|
||||
| MCP Tools | < 50ms | > 100ms | > 200ms |
|
||||
|
||||
## Optimization Guidelines
|
||||
|
||||
### Current Optimizations
|
||||
|
||||
1. **In-memory caching**: Frequently accessed nodes are cached
|
||||
2. **Indexed database**: Key fields are indexed for fast lookups
|
||||
3. **Lazy loading**: Large properties are loaded on demand
|
||||
4. **Batch operations**: Multiple operations are batched when possible
|
||||
|
||||
### Future Optimizations
|
||||
|
||||
1. **FTS5 Search**: Implement SQLite FTS5 for faster full-text search
|
||||
2. **Connection pooling**: Reuse database connections
|
||||
3. **Query optimization**: Analyze and optimize slow queries
|
||||
4. **Parallel loading**: Load multiple packages concurrently
|
||||
|
||||
## Benchmark Implementation
|
||||
|
||||
### Writing New Benchmarks
|
||||
|
||||
```typescript
|
||||
import { bench, describe } from 'vitest';
|
||||
|
||||
describe('My Performance Suite', () => {
|
||||
bench('operation name', async () => {
|
||||
// Code to benchmark
|
||||
}, {
|
||||
iterations: 100,
|
||||
warmupIterations: 10,
|
||||
warmupTime: 500,
|
||||
time: 3000
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
### Best Practices
|
||||
|
||||
1. **Isolate operations**: Benchmark specific operations, not entire workflows
|
||||
2. **Use realistic data**: Load actual n8n nodes for accurate measurements
|
||||
3. **Include warmup**: Allow JIT compilation to stabilize
|
||||
4. **Consider memory**: Monitor memory usage for memory-intensive operations
|
||||
5. **Statistical significance**: Run enough iterations for reliable results
|
||||
|
||||
## Interpreting Results
|
||||
|
||||
### Key Metrics
|
||||
|
||||
- **hz**: Operations per second (higher is better)
|
||||
- **mean**: Average time per operation (lower is better)
|
||||
- **p99**: 99th percentile (worst-case performance)
|
||||
- **rme**: Relative margin of error (lower is more reliable)
|
||||
|
||||
### Performance Regression Detection
|
||||
|
||||
A performance regression is flagged when:
|
||||
1. Operation time increases by >10% from baseline
|
||||
2. Multiple related operations show degradation
|
||||
3. P99 latency exceeds critical thresholds
|
||||
|
||||
### Analyzing Trends
|
||||
|
||||
1. **Gradual degradation**: Often indicates growing technical debt
|
||||
2. **Sudden spikes**: Usually from specific code changes
|
||||
3. **Seasonal patterns**: May indicate cache effectiveness
|
||||
4. **Outliers**: Check p99 vs mean for consistency
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **Inconsistent results**: Increase warmup iterations
|
||||
2. **High variance**: Check for background processes
|
||||
3. **Memory issues**: Reduce iteration count
|
||||
4. **CI failures**: Verify runner resources
|
||||
|
||||
### Performance Debugging
|
||||
|
||||
1. Use `--reporter=verbose` for detailed output
|
||||
2. Profile with `node --inspect` for bottlenecks
|
||||
3. Check database query plans
|
||||
4. Monitor memory allocation patterns
|
||||
|
||||
## Contributing
|
||||
|
||||
When submitting performance improvements:
|
||||
|
||||
1. Run benchmarks before and after changes
|
||||
2. Include benchmark results in PR description
|
||||
3. Explain optimization approach
|
||||
4. Consider trade-offs (memory vs speed)
|
||||
5. Add new benchmarks for new features
|
||||
|
||||
## References
|
||||
|
||||
- [Vitest Benchmark Documentation](https://vitest.dev/guide/features.html#benchmarking)
|
||||
- [GitHub Action Benchmark](https://github.com/benchmark-action/github-action-benchmark)
|
||||
- [SQLite Performance Tuning](https://www.sqlite.org/optoverview.html)
|
||||
113
docs/CODECOV_SETUP.md
Normal file
113
docs/CODECOV_SETUP.md
Normal file
@@ -0,0 +1,113 @@
|
||||
# Codecov Setup Guide
|
||||
|
||||
This guide explains how to set up and configure Codecov for the n8n-MCP project.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
1. A Codecov account (sign up at https://codecov.io)
|
||||
2. Repository admin access to add the CODECOV_TOKEN secret
|
||||
|
||||
## Setup Steps
|
||||
|
||||
### 1. Get Your Codecov Token
|
||||
|
||||
1. Sign in to [Codecov](https://codecov.io)
|
||||
2. Add your repository: `czlonkowski/n8n-mcp`
|
||||
3. Copy the upload token from the repository settings
|
||||
|
||||
### 2. Add Token to GitHub Secrets
|
||||
|
||||
1. Go to your GitHub repository settings
|
||||
2. Navigate to `Settings` → `Secrets and variables` → `Actions`
|
||||
3. Click "New repository secret"
|
||||
4. Name: `CODECOV_TOKEN`
|
||||
5. Value: Paste your Codecov token
|
||||
6. Click "Add secret"
|
||||
|
||||
### 3. Update the Badge Token
|
||||
|
||||
Edit the README.md file and replace `YOUR_TOKEN` in the Codecov badge with your actual token:
|
||||
|
||||
```markdown
|
||||
[](https://codecov.io/gh/czlonkowski/n8n-mcp)
|
||||
```
|
||||
|
||||
Note: The token in the badge URL is a read-only token and safe to commit.
|
||||
|
||||
## Configuration Details
|
||||
|
||||
### codecov.yml
|
||||
|
||||
The configuration file sets:
|
||||
- **Target coverage**: 80% for both project and patch
|
||||
- **Coverage precision**: 2 decimal places
|
||||
- **Comment behavior**: Comments on all PRs with coverage changes
|
||||
- **Ignored files**: Test files, scripts, node_modules, and build outputs
|
||||
|
||||
### GitHub Actions
|
||||
|
||||
The workflow:
|
||||
1. Runs tests with coverage using `npm run test:coverage`
|
||||
2. Generates LCOV format coverage report
|
||||
3. Uploads to Codecov using the official action
|
||||
4. Fails the build if upload fails
|
||||
|
||||
### Vitest Configuration
|
||||
|
||||
Coverage settings in `vitest.config.ts`:
|
||||
- **Provider**: V8 (fast and accurate)
|
||||
- **Reporters**: text, json, html, and lcov
|
||||
- **Thresholds**: 80% lines, 80% functions, 75% branches, 80% statements
|
||||
|
||||
## Viewing Coverage
|
||||
|
||||
### Local Coverage
|
||||
|
||||
```bash
|
||||
# Generate coverage report
|
||||
npm run test:coverage
|
||||
|
||||
# View HTML report
|
||||
open coverage/index.html
|
||||
```
|
||||
|
||||
### Online Coverage
|
||||
|
||||
1. Visit https://codecov.io/gh/czlonkowski/n8n-mcp
|
||||
2. View detailed reports, graphs, and file-by-file coverage
|
||||
3. Check PR comments for coverage changes
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Coverage Not Uploading
|
||||
|
||||
1. Verify CODECOV_TOKEN is set in GitHub secrets
|
||||
2. Check GitHub Actions logs for errors
|
||||
3. Ensure coverage/lcov.info is generated
|
||||
|
||||
### Badge Not Showing
|
||||
|
||||
1. Wait a few minutes after first upload
|
||||
2. Verify the token in the badge URL is correct
|
||||
3. Check if the repository is public/private settings match
|
||||
|
||||
### Low Coverage Areas
|
||||
|
||||
Current areas with lower coverage that could be improved:
|
||||
- HTTP server implementations
|
||||
- MCP index files
|
||||
- Some edge cases in validators
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Write tests first**: Aim for TDD when adding features
|
||||
2. **Focus on critical paths**: Prioritize testing core functionality
|
||||
3. **Mock external dependencies**: Use MSW for HTTP, mock for databases
|
||||
4. **Keep coverage realistic**: 80% is good, 100% isn't always practical
|
||||
5. **Monitor trends**: Watch coverage over time, not just absolute numbers
|
||||
|
||||
## Resources
|
||||
|
||||
- [Codecov Documentation](https://docs.codecov.io/)
|
||||
- [Vitest Coverage](https://vitest.dev/guide/coverage.html)
|
||||
- [GitHub Actions + Codecov](https://github.com/codecov/codecov-action)
|
||||
Reference in New Issue
Block a user