feat: add comprehensive performance benchmark tracking system
- Create benchmark test suites for critical operations: - Node loading performance - Database query performance - Search operations performance - Validation performance - MCP tool execution performance - Add GitHub Actions workflow for benchmark tracking: - Runs on push to main and PRs - Uses github-action-benchmark for historical tracking - Comments on PRs with performance results - Alerts on >10% performance regressions - Stores results in GitHub Pages - Create benchmark infrastructure: - Custom Vitest benchmark configuration - JSON reporter for CI results - Result formatter for github-action-benchmark - Performance threshold documentation - Add supporting utilities: - SQLiteStorageService for benchmark database setup - MCPEngine wrapper for testing MCP tools - Test factories for generating benchmark data - Enhanced NodeRepository with benchmark methods - Document benchmark system: - Comprehensive benchmark guide in docs/BENCHMARKS.md - Performance thresholds in .github/BENCHMARK_THRESHOLDS.md - README for benchmarks directory - Integration with existing test suite The benchmark system will help monitor performance over time and catch regressions before they reach production. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
121
tests/benchmarks/README.md
Normal file
121
tests/benchmarks/README.md
Normal file
@@ -0,0 +1,121 @@
|
||||
# Performance Benchmarks
|
||||
|
||||
This directory contains performance benchmarks for critical operations in the n8n-mcp project.
|
||||
|
||||
## Running Benchmarks
|
||||
|
||||
### Local Development
|
||||
|
||||
```bash
|
||||
# Run all benchmarks
|
||||
npm run benchmark
|
||||
|
||||
# Watch mode for development
|
||||
npm run benchmark:watch
|
||||
|
||||
# Interactive UI
|
||||
npm run benchmark:ui
|
||||
|
||||
# Run specific benchmark file
|
||||
npx vitest bench tests/benchmarks/node-loading.bench.ts
|
||||
```
|
||||
|
||||
### CI/CD
|
||||
|
||||
Benchmarks run automatically on:
|
||||
- Every push to `main` branch
|
||||
- Every pull request
|
||||
- Manual workflow dispatch
|
||||
|
||||
## Benchmark Suites
|
||||
|
||||
### 1. Node Loading Performance (`node-loading.bench.ts`)
|
||||
- Package loading (n8n-nodes-base, @n8n/n8n-nodes-langchain)
|
||||
- Individual node file loading
|
||||
- Package.json parsing
|
||||
|
||||
### 2. Database Query Performance (`database-queries.bench.ts`)
|
||||
- Node retrieval by type
|
||||
- Category filtering
|
||||
- Search operations (OR, AND, FUZZY modes)
|
||||
- Node counting and statistics
|
||||
- Insert/update operations
|
||||
|
||||
### 3. Search Operations (`search-operations.bench.ts`)
|
||||
- Single and multi-word searches
|
||||
- Exact phrase matching
|
||||
- Fuzzy search performance
|
||||
- Property search within nodes
|
||||
- Complex filtering operations
|
||||
|
||||
### 4. Validation Performance (`validation-performance.bench.ts`)
|
||||
- Node configuration validation (minimal, strict, ai-friendly)
|
||||
- Expression validation
|
||||
- Workflow validation
|
||||
- Property dependency resolution
|
||||
|
||||
### 5. MCP Tool Execution (`mcp-tools.bench.ts`)
|
||||
- Tool execution overhead
|
||||
- Response formatting
|
||||
- Complex query handling
|
||||
|
||||
## Performance Targets
|
||||
|
||||
| Operation | Target | Alert Threshold |
|
||||
|-----------|--------|-----------------|
|
||||
| Node loading | <100ms per package | >150ms |
|
||||
| Database query | <5ms per query | >10ms |
|
||||
| Search (simple) | <10ms | >20ms |
|
||||
| Search (complex) | <50ms | >100ms |
|
||||
| Validation (simple) | <1ms | >2ms |
|
||||
| Validation (complex) | <10ms | >20ms |
|
||||
| MCP tool execution | <50ms | >100ms |
|
||||
|
||||
## Benchmark Results
|
||||
|
||||
- Results are tracked over time using GitHub Actions
|
||||
- Historical data available at: https://czlonkowski.github.io/n8n-mcp/benchmarks/
|
||||
- Performance regressions >10% trigger automatic alerts
|
||||
- PR comments show benchmark comparisons
|
||||
|
||||
## Writing New Benchmarks
|
||||
|
||||
```typescript
|
||||
import { bench, describe } from 'vitest';
|
||||
|
||||
describe('My Performance Suite', () => {
|
||||
bench('operation name', async () => {
|
||||
// Code to benchmark
|
||||
}, {
|
||||
iterations: 100, // Number of times to run
|
||||
warmupIterations: 10, // Warmup runs (not measured)
|
||||
warmupTime: 500, // Warmup duration in ms
|
||||
time: 3000 // Total benchmark duration in ms
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Isolate Operations**: Benchmark specific operations, not entire workflows
|
||||
2. **Use Realistic Data**: Load actual n8n nodes for realistic measurements
|
||||
3. **Warmup**: Always include warmup iterations to avoid JIT compilation effects
|
||||
4. **Memory**: Use in-memory databases for consistent results
|
||||
5. **Iterations**: Balance between accuracy and execution time
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Inconsistent Results
|
||||
- Increase `warmupIterations` and `warmupTime`
|
||||
- Run benchmarks in isolation
|
||||
- Check for background processes
|
||||
|
||||
### Memory Issues
|
||||
- Reduce `iterations` for memory-intensive operations
|
||||
- Add cleanup in `afterEach` hooks
|
||||
- Monitor memory usage during benchmarks
|
||||
|
||||
### CI Failures
|
||||
- Check benchmark timeout settings
|
||||
- Verify GitHub Actions runner resources
|
||||
- Review alert thresholds for false positives
|
||||
149
tests/benchmarks/database-queries.bench.ts
Normal file
149
tests/benchmarks/database-queries.bench.ts
Normal file
@@ -0,0 +1,149 @@
|
||||
import { bench, describe } from 'vitest';
|
||||
import { NodeRepository } from '../../src/database/node-repository';
|
||||
import { SQLiteStorageService } from '../../src/services/sqlite-storage-service';
|
||||
import { NodeFactory } from '../factories/node-factory';
|
||||
import { PropertyDefinitionFactory } from '../factories/property-definition-factory';
|
||||
|
||||
describe('Database Query Performance', () => {
|
||||
let repository: NodeRepository;
|
||||
let storage: SQLiteStorageService;
|
||||
const testNodeCount = 500;
|
||||
|
||||
beforeAll(async () => {
|
||||
storage = new SQLiteStorageService(':memory:');
|
||||
repository = new NodeRepository(storage);
|
||||
|
||||
// Seed database with test data
|
||||
for (let i = 0; i < testNodeCount; i++) {
|
||||
const node = NodeFactory.build({
|
||||
name: `TestNode${i}`,
|
||||
type: `nodes-base.testNode${i}`,
|
||||
category: i % 2 === 0 ? 'transform' : 'trigger',
|
||||
package: 'n8n-nodes-base',
|
||||
documentation: `Test documentation for node ${i}`,
|
||||
properties: PropertyDefinitionFactory.buildList(5)
|
||||
});
|
||||
await repository.upsertNode(node);
|
||||
}
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
storage.close();
|
||||
});
|
||||
|
||||
bench('getNodeByType - existing node', async () => {
|
||||
await repository.getNodeByType('nodes-base.testNode100');
|
||||
}, {
|
||||
iterations: 1000,
|
||||
warmupIterations: 100,
|
||||
warmupTime: 500,
|
||||
time: 3000
|
||||
});
|
||||
|
||||
bench('getNodeByType - non-existing node', async () => {
|
||||
await repository.getNodeByType('nodes-base.nonExistentNode');
|
||||
}, {
|
||||
iterations: 1000,
|
||||
warmupIterations: 100,
|
||||
warmupTime: 500,
|
||||
time: 3000
|
||||
});
|
||||
|
||||
bench('getNodesByCategory - transform', async () => {
|
||||
await repository.getNodesByCategory('transform');
|
||||
}, {
|
||||
iterations: 100,
|
||||
warmupIterations: 10,
|
||||
warmupTime: 500,
|
||||
time: 3000
|
||||
});
|
||||
|
||||
bench('searchNodes - OR mode', async () => {
|
||||
await repository.searchNodes('test node data', 'OR', 20);
|
||||
}, {
|
||||
iterations: 100,
|
||||
warmupIterations: 10,
|
||||
warmupTime: 500,
|
||||
time: 3000
|
||||
});
|
||||
|
||||
bench('searchNodes - AND mode', async () => {
|
||||
await repository.searchNodes('test node', 'AND', 20);
|
||||
}, {
|
||||
iterations: 100,
|
||||
warmupIterations: 10,
|
||||
warmupTime: 500,
|
||||
time: 3000
|
||||
});
|
||||
|
||||
bench('searchNodes - FUZZY mode', async () => {
|
||||
await repository.searchNodes('tst nde', 'FUZZY', 20);
|
||||
}, {
|
||||
iterations: 100,
|
||||
warmupIterations: 10,
|
||||
warmupTime: 500,
|
||||
time: 3000
|
||||
});
|
||||
|
||||
bench('getAllNodes - no limit', async () => {
|
||||
await repository.getAllNodes();
|
||||
}, {
|
||||
iterations: 50,
|
||||
warmupIterations: 5,
|
||||
warmupTime: 500,
|
||||
time: 3000
|
||||
});
|
||||
|
||||
bench('getAllNodes - with limit', async () => {
|
||||
await repository.getAllNodes(50);
|
||||
}, {
|
||||
iterations: 100,
|
||||
warmupIterations: 10,
|
||||
warmupTime: 500,
|
||||
time: 3000
|
||||
});
|
||||
|
||||
bench('getNodeCount', async () => {
|
||||
await repository.getNodeCount();
|
||||
}, {
|
||||
iterations: 1000,
|
||||
warmupIterations: 100,
|
||||
warmupTime: 100,
|
||||
time: 2000
|
||||
});
|
||||
|
||||
bench('getAIToolNodes', async () => {
|
||||
await repository.getAIToolNodes();
|
||||
}, {
|
||||
iterations: 100,
|
||||
warmupIterations: 10,
|
||||
warmupTime: 500,
|
||||
time: 3000
|
||||
});
|
||||
|
||||
bench('upsertNode - new node', async () => {
|
||||
const node = NodeFactory.build({
|
||||
name: `BenchNode${Date.now()}`,
|
||||
type: `nodes-base.benchNode${Date.now()}`
|
||||
});
|
||||
await repository.upsertNode(node);
|
||||
}, {
|
||||
iterations: 100,
|
||||
warmupIterations: 10,
|
||||
warmupTime: 500,
|
||||
time: 3000
|
||||
});
|
||||
|
||||
bench('upsertNode - existing node update', async () => {
|
||||
const existingNode = await repository.getNodeByType('nodes-base.testNode0');
|
||||
if (existingNode) {
|
||||
existingNode.description = `Updated description ${Date.now()}`;
|
||||
await repository.upsertNode(existingNode);
|
||||
}
|
||||
}, {
|
||||
iterations: 100,
|
||||
warmupIterations: 10,
|
||||
warmupTime: 500,
|
||||
time: 3000
|
||||
});
|
||||
});
|
||||
6
tests/benchmarks/index.ts
Normal file
6
tests/benchmarks/index.ts
Normal file
@@ -0,0 +1,6 @@
|
||||
// Export all benchmark suites
|
||||
export * from './node-loading.bench';
|
||||
export * from './database-queries.bench';
|
||||
export * from './search-operations.bench';
|
||||
export * from './validation-performance.bench';
|
||||
export * from './mcp-tools.bench';
|
||||
204
tests/benchmarks/mcp-tools.bench.ts
Normal file
204
tests/benchmarks/mcp-tools.bench.ts
Normal file
@@ -0,0 +1,204 @@
|
||||
import { bench, describe } from 'vitest';
|
||||
import { MCPEngine } from '../../src/mcp-tools-engine';
|
||||
import { NodeRepository } from '../../src/database/node-repository';
|
||||
import { SQLiteStorageService } from '../../src/services/sqlite-storage-service';
|
||||
import { NodeLoader } from '../../src/loaders/node-loader';
|
||||
|
||||
describe('MCP Tool Execution Performance', () => {
|
||||
let engine: MCPEngine;
|
||||
let storage: SQLiteStorageService;
|
||||
|
||||
beforeAll(async () => {
|
||||
storage = new SQLiteStorageService(':memory:');
|
||||
const repository = new NodeRepository(storage);
|
||||
const loader = new NodeLoader(repository);
|
||||
await loader.loadPackage('n8n-nodes-base');
|
||||
|
||||
engine = new MCPEngine(repository);
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
storage.close();
|
||||
});
|
||||
|
||||
bench('list_nodes - default limit', async () => {
|
||||
await engine.listNodes({});
|
||||
}, {
|
||||
iterations: 100,
|
||||
warmupIterations: 10,
|
||||
warmupTime: 500,
|
||||
time: 3000
|
||||
});
|
||||
|
||||
bench('list_nodes - large limit', async () => {
|
||||
await engine.listNodes({ limit: 200 });
|
||||
}, {
|
||||
iterations: 50,
|
||||
warmupIterations: 5,
|
||||
warmupTime: 500,
|
||||
time: 3000
|
||||
});
|
||||
|
||||
bench('list_nodes - filtered by category', async () => {
|
||||
await engine.listNodes({ category: 'transform', limit: 100 });
|
||||
}, {
|
||||
iterations: 100,
|
||||
warmupIterations: 10,
|
||||
warmupTime: 500,
|
||||
time: 3000
|
||||
});
|
||||
|
||||
bench('search_nodes - single word', async () => {
|
||||
await engine.searchNodes({ query: 'http' });
|
||||
}, {
|
||||
iterations: 100,
|
||||
warmupIterations: 10,
|
||||
warmupTime: 500,
|
||||
time: 3000
|
||||
});
|
||||
|
||||
bench('search_nodes - multiple words', async () => {
|
||||
await engine.searchNodes({ query: 'http request webhook', mode: 'OR' });
|
||||
}, {
|
||||
iterations: 100,
|
||||
warmupIterations: 10,
|
||||
warmupTime: 500,
|
||||
time: 3000
|
||||
});
|
||||
|
||||
bench('get_node_info', async () => {
|
||||
await engine.getNodeInfo({ nodeType: 'n8n-nodes-base.httpRequest' });
|
||||
}, {
|
||||
iterations: 500,
|
||||
warmupIterations: 50,
|
||||
warmupTime: 500,
|
||||
time: 3000
|
||||
});
|
||||
|
||||
bench('get_node_essentials', async () => {
|
||||
await engine.getNodeEssentials({ nodeType: 'n8n-nodes-base.httpRequest' });
|
||||
}, {
|
||||
iterations: 1000,
|
||||
warmupIterations: 100,
|
||||
warmupTime: 500,
|
||||
time: 3000
|
||||
});
|
||||
|
||||
bench('get_node_documentation', async () => {
|
||||
await engine.getNodeDocumentation({ nodeType: 'n8n-nodes-base.httpRequest' });
|
||||
}, {
|
||||
iterations: 500,
|
||||
warmupIterations: 50,
|
||||
warmupTime: 500,
|
||||
time: 3000
|
||||
});
|
||||
|
||||
bench('validate_node_operation - simple', async () => {
|
||||
await engine.validateNodeOperation({
|
||||
nodeType: 'n8n-nodes-base.httpRequest',
|
||||
config: {
|
||||
url: 'https://api.example.com',
|
||||
method: 'GET'
|
||||
},
|
||||
profile: 'minimal'
|
||||
});
|
||||
}, {
|
||||
iterations: 1000,
|
||||
warmupIterations: 100,
|
||||
warmupTime: 500,
|
||||
time: 3000
|
||||
});
|
||||
|
||||
bench('validate_node_operation - complex', async () => {
|
||||
await engine.validateNodeOperation({
|
||||
nodeType: 'n8n-nodes-base.slack',
|
||||
config: {
|
||||
resource: 'message',
|
||||
operation: 'send',
|
||||
channel: 'C1234567890',
|
||||
text: 'Hello from benchmark'
|
||||
},
|
||||
profile: 'strict'
|
||||
});
|
||||
}, {
|
||||
iterations: 500,
|
||||
warmupIterations: 50,
|
||||
warmupTime: 500,
|
||||
time: 3000
|
||||
});
|
||||
|
||||
bench('validate_node_minimal', async () => {
|
||||
await engine.validateNodeMinimal({
|
||||
nodeType: 'n8n-nodes-base.httpRequest',
|
||||
config: {}
|
||||
});
|
||||
}, {
|
||||
iterations: 2000,
|
||||
warmupIterations: 200,
|
||||
warmupTime: 500,
|
||||
time: 3000
|
||||
});
|
||||
|
||||
bench('search_node_properties', async () => {
|
||||
await engine.searchNodeProperties({
|
||||
nodeType: 'n8n-nodes-base.httpRequest',
|
||||
query: 'authentication'
|
||||
});
|
||||
}, {
|
||||
iterations: 500,
|
||||
warmupIterations: 50,
|
||||
warmupTime: 500,
|
||||
time: 3000
|
||||
});
|
||||
|
||||
bench('get_node_for_task', async () => {
|
||||
await engine.getNodeForTask({ task: 'post_json_request' });
|
||||
}, {
|
||||
iterations: 1000,
|
||||
warmupIterations: 100,
|
||||
warmupTime: 500,
|
||||
time: 3000
|
||||
});
|
||||
|
||||
bench('list_ai_tools', async () => {
|
||||
await engine.listAITools({});
|
||||
}, {
|
||||
iterations: 100,
|
||||
warmupIterations: 10,
|
||||
warmupTime: 500,
|
||||
time: 3000
|
||||
});
|
||||
|
||||
bench('get_database_statistics', async () => {
|
||||
await engine.getDatabaseStatistics({});
|
||||
}, {
|
||||
iterations: 1000,
|
||||
warmupIterations: 100,
|
||||
warmupTime: 500,
|
||||
time: 3000
|
||||
});
|
||||
|
||||
bench('validate_workflow - simple', async () => {
|
||||
await engine.validateWorkflow({
|
||||
workflow: {
|
||||
name: 'Test',
|
||||
nodes: [
|
||||
{
|
||||
id: '1',
|
||||
name: 'Manual',
|
||||
type: 'n8n-nodes-base.manualTrigger',
|
||||
typeVersion: 1,
|
||||
position: [250, 300],
|
||||
parameters: {}
|
||||
}
|
||||
],
|
||||
connections: {}
|
||||
}
|
||||
});
|
||||
}, {
|
||||
iterations: 500,
|
||||
warmupIterations: 50,
|
||||
warmupTime: 500,
|
||||
time: 3000
|
||||
});
|
||||
});
|
||||
59
tests/benchmarks/node-loading.bench.ts
Normal file
59
tests/benchmarks/node-loading.bench.ts
Normal file
@@ -0,0 +1,59 @@
|
||||
import { bench, describe } from 'vitest';
|
||||
import { NodeLoader } from '../../src/loaders/node-loader';
|
||||
import { NodeRepository } from '../../src/database/node-repository';
|
||||
import { SQLiteStorageService } from '../../src/services/sqlite-storage-service';
|
||||
import path from 'path';
|
||||
|
||||
describe('Node Loading Performance', () => {
|
||||
let loader: NodeLoader;
|
||||
let repository: NodeRepository;
|
||||
let storage: SQLiteStorageService;
|
||||
|
||||
beforeAll(() => {
|
||||
storage = new SQLiteStorageService(':memory:');
|
||||
repository = new NodeRepository(storage);
|
||||
loader = new NodeLoader(repository);
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
storage.close();
|
||||
});
|
||||
|
||||
bench('loadPackage - n8n-nodes-base', async () => {
|
||||
await loader.loadPackage('n8n-nodes-base');
|
||||
}, {
|
||||
iterations: 5,
|
||||
warmupIterations: 2,
|
||||
warmupTime: 1000,
|
||||
time: 5000
|
||||
});
|
||||
|
||||
bench('loadPackage - @n8n/n8n-nodes-langchain', async () => {
|
||||
await loader.loadPackage('@n8n/n8n-nodes-langchain');
|
||||
}, {
|
||||
iterations: 5,
|
||||
warmupIterations: 2,
|
||||
warmupTime: 1000,
|
||||
time: 5000
|
||||
});
|
||||
|
||||
bench('loadNodesFromPath - single file', async () => {
|
||||
const testPath = path.join(process.cwd(), 'node_modules/n8n-nodes-base/dist/nodes/HttpRequest');
|
||||
await loader.loadNodesFromPath(testPath, 'n8n-nodes-base');
|
||||
}, {
|
||||
iterations: 100,
|
||||
warmupIterations: 10,
|
||||
warmupTime: 500,
|
||||
time: 3000
|
||||
});
|
||||
|
||||
bench('parsePackageJson', async () => {
|
||||
const packageJsonPath = path.join(process.cwd(), 'node_modules/n8n-nodes-base/package.json');
|
||||
await loader['parsePackageJson'](packageJsonPath);
|
||||
}, {
|
||||
iterations: 1000,
|
||||
warmupIterations: 100,
|
||||
warmupTime: 100,
|
||||
time: 2000
|
||||
});
|
||||
});
|
||||
47
tests/benchmarks/sample.bench.ts
Normal file
47
tests/benchmarks/sample.bench.ts
Normal file
@@ -0,0 +1,47 @@
|
||||
import { bench, describe } from 'vitest';
|
||||
|
||||
/**
|
||||
* Sample benchmark to verify the setup works correctly
|
||||
*/
|
||||
describe('Sample Benchmarks', () => {
|
||||
bench('array sorting - small', () => {
|
||||
const arr = Array.from({ length: 100 }, () => Math.random());
|
||||
arr.sort((a, b) => a - b);
|
||||
}, {
|
||||
iterations: 1000,
|
||||
warmupIterations: 100
|
||||
});
|
||||
|
||||
bench('array sorting - large', () => {
|
||||
const arr = Array.from({ length: 10000 }, () => Math.random());
|
||||
arr.sort((a, b) => a - b);
|
||||
}, {
|
||||
iterations: 100,
|
||||
warmupIterations: 10
|
||||
});
|
||||
|
||||
bench('string concatenation', () => {
|
||||
let str = '';
|
||||
for (let i = 0; i < 1000; i++) {
|
||||
str += 'a';
|
||||
}
|
||||
}, {
|
||||
iterations: 1000,
|
||||
warmupIterations: 100
|
||||
});
|
||||
|
||||
bench('object creation', () => {
|
||||
const objects = [];
|
||||
for (let i = 0; i < 1000; i++) {
|
||||
objects.push({
|
||||
id: i,
|
||||
name: `Object ${i}`,
|
||||
value: Math.random(),
|
||||
timestamp: Date.now()
|
||||
});
|
||||
}
|
||||
}, {
|
||||
iterations: 1000,
|
||||
warmupIterations: 100
|
||||
});
|
||||
});
|
||||
143
tests/benchmarks/search-operations.bench.ts
Normal file
143
tests/benchmarks/search-operations.bench.ts
Normal file
@@ -0,0 +1,143 @@
|
||||
import { bench, describe } from 'vitest';
|
||||
import { NodeRepository } from '../../src/database/node-repository';
|
||||
import { SQLiteStorageService } from '../../src/services/sqlite-storage-service';
|
||||
import { NodeLoader } from '../../src/loaders/node-loader';
|
||||
|
||||
describe('Search Operations Performance', () => {
|
||||
let repository: NodeRepository;
|
||||
let storage: SQLiteStorageService;
|
||||
|
||||
beforeAll(async () => {
|
||||
storage = new SQLiteStorageService(':memory:');
|
||||
repository = new NodeRepository(storage);
|
||||
const loader = new NodeLoader(repository);
|
||||
|
||||
// Load real nodes for realistic benchmarking
|
||||
await loader.loadPackage('n8n-nodes-base');
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
storage.close();
|
||||
});
|
||||
|
||||
bench('searchNodes - single word', async () => {
|
||||
await repository.searchNodes('http', 'OR', 20);
|
||||
}, {
|
||||
iterations: 100,
|
||||
warmupIterations: 10,
|
||||
warmupTime: 500,
|
||||
time: 3000
|
||||
});
|
||||
|
||||
bench('searchNodes - multiple words OR', async () => {
|
||||
await repository.searchNodes('http request webhook', 'OR', 20);
|
||||
}, {
|
||||
iterations: 100,
|
||||
warmupIterations: 10,
|
||||
warmupTime: 500,
|
||||
time: 3000
|
||||
});
|
||||
|
||||
bench('searchNodes - multiple words AND', async () => {
|
||||
await repository.searchNodes('http request', 'AND', 20);
|
||||
}, {
|
||||
iterations: 100,
|
||||
warmupIterations: 10,
|
||||
warmupTime: 500,
|
||||
time: 3000
|
||||
});
|
||||
|
||||
bench('searchNodes - fuzzy search', async () => {
|
||||
await repository.searchNodes('htpp requst', 'FUZZY', 20);
|
||||
}, {
|
||||
iterations: 100,
|
||||
warmupIterations: 10,
|
||||
warmupTime: 500,
|
||||
time: 3000
|
||||
});
|
||||
|
||||
bench('searchNodes - exact phrase', async () => {
|
||||
await repository.searchNodes('"HTTP Request"', 'OR', 20);
|
||||
}, {
|
||||
iterations: 100,
|
||||
warmupIterations: 10,
|
||||
warmupTime: 500,
|
||||
time: 3000
|
||||
});
|
||||
|
||||
bench('searchNodes - large result set', async () => {
|
||||
await repository.searchNodes('data', 'OR', 100);
|
||||
}, {
|
||||
iterations: 50,
|
||||
warmupIterations: 5,
|
||||
warmupTime: 500,
|
||||
time: 3000
|
||||
});
|
||||
|
||||
bench('searchNodes - no results', async () => {
|
||||
await repository.searchNodes('xyznonexistentquery123', 'OR', 20);
|
||||
}, {
|
||||
iterations: 200,
|
||||
warmupIterations: 20,
|
||||
warmupTime: 500,
|
||||
time: 3000
|
||||
});
|
||||
|
||||
bench('searchNodeProperties - common property', async () => {
|
||||
const node = await repository.getNodeByType('n8n-nodes-base.httpRequest');
|
||||
if (node) {
|
||||
await repository.searchNodeProperties(node.type, 'url', 20);
|
||||
}
|
||||
}, {
|
||||
iterations: 100,
|
||||
warmupIterations: 10,
|
||||
warmupTime: 500,
|
||||
time: 3000
|
||||
});
|
||||
|
||||
bench('searchNodeProperties - nested property', async () => {
|
||||
const node = await repository.getNodeByType('n8n-nodes-base.httpRequest');
|
||||
if (node) {
|
||||
await repository.searchNodeProperties(node.type, 'authentication', 20);
|
||||
}
|
||||
}, {
|
||||
iterations: 100,
|
||||
warmupIterations: 10,
|
||||
warmupTime: 500,
|
||||
time: 3000
|
||||
});
|
||||
|
||||
bench('getNodesByCategory - all categories', async () => {
|
||||
const categories = ['trigger', 'transform', 'output', 'input'];
|
||||
for (const category of categories) {
|
||||
await repository.getNodesByCategory(category);
|
||||
}
|
||||
}, {
|
||||
iterations: 50,
|
||||
warmupIterations: 5,
|
||||
warmupTime: 500,
|
||||
time: 3000
|
||||
});
|
||||
|
||||
bench('getNodesByPackage', async () => {
|
||||
await repository.getNodesByPackage('n8n-nodes-base');
|
||||
}, {
|
||||
iterations: 50,
|
||||
warmupIterations: 5,
|
||||
warmupTime: 500,
|
||||
time: 3000
|
||||
});
|
||||
|
||||
bench('complex filter - AI tools in transform category', async () => {
|
||||
const allNodes = await repository.getAllNodes();
|
||||
const filtered = allNodes.filter(node =>
|
||||
node.category === 'transform' &&
|
||||
node.isAITool
|
||||
);
|
||||
}, {
|
||||
iterations: 100,
|
||||
warmupIterations: 10,
|
||||
warmupTime: 500,
|
||||
time: 3000
|
||||
});
|
||||
});
|
||||
238
tests/benchmarks/validation-performance.bench.ts
Normal file
238
tests/benchmarks/validation-performance.bench.ts
Normal file
@@ -0,0 +1,238 @@
|
||||
import { bench, describe } from 'vitest';
|
||||
import { ConfigValidator } from '../../src/services/config-validator';
|
||||
import { EnhancedConfigValidator } from '../../src/services/enhanced-config-validator';
|
||||
import { ExpressionValidator } from '../../src/services/expression-validator';
|
||||
import { WorkflowValidator } from '../../src/services/workflow-validator';
|
||||
import { NodeRepository } from '../../src/database/node-repository';
|
||||
import { SQLiteStorageService } from '../../src/services/sqlite-storage-service';
|
||||
import { NodeLoader } from '../../src/loaders/node-loader';
|
||||
|
||||
describe('Validation Performance', () => {
|
||||
let validator: ConfigValidator;
|
||||
let enhancedValidator: EnhancedConfigValidator;
|
||||
let expressionValidator: ExpressionValidator;
|
||||
let workflowValidator: WorkflowValidator;
|
||||
let repository: NodeRepository;
|
||||
let storage: SQLiteStorageService;
|
||||
|
||||
const simpleConfig = {
|
||||
url: 'https://api.example.com',
|
||||
method: 'GET',
|
||||
authentication: 'none'
|
||||
};
|
||||
|
||||
const complexConfig = {
|
||||
resource: 'message',
|
||||
operation: 'send',
|
||||
channel: 'C1234567890',
|
||||
text: 'Hello from benchmark',
|
||||
authentication: {
|
||||
type: 'oAuth2',
|
||||
credentials: {
|
||||
oauthTokenData: {
|
||||
access_token: 'xoxb-test-token'
|
||||
}
|
||||
}
|
||||
},
|
||||
options: {
|
||||
as_user: true,
|
||||
link_names: true,
|
||||
parse: 'full',
|
||||
reply_broadcast: false,
|
||||
thread_ts: '',
|
||||
unfurl_links: true,
|
||||
unfurl_media: true
|
||||
}
|
||||
};
|
||||
|
||||
const simpleWorkflow = {
|
||||
name: 'Simple Workflow',
|
||||
nodes: [
|
||||
{
|
||||
id: '1',
|
||||
name: 'Manual Trigger',
|
||||
type: 'n8n-nodes-base.manualTrigger',
|
||||
typeVersion: 1,
|
||||
position: [250, 300],
|
||||
parameters: {}
|
||||
},
|
||||
{
|
||||
id: '2',
|
||||
name: 'HTTP Request',
|
||||
type: 'n8n-nodes-base.httpRequest',
|
||||
typeVersion: 4.2,
|
||||
position: [450, 300],
|
||||
parameters: {
|
||||
url: 'https://api.example.com',
|
||||
method: 'GET'
|
||||
}
|
||||
}
|
||||
],
|
||||
connections: {
|
||||
'1': {
|
||||
main: [
|
||||
[
|
||||
{
|
||||
node: '2',
|
||||
type: 'main',
|
||||
index: 0
|
||||
}
|
||||
]
|
||||
]
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
const complexWorkflow = {
|
||||
name: 'Complex Workflow',
|
||||
nodes: Array.from({ length: 20 }, (_, i) => ({
|
||||
id: `${i + 1}`,
|
||||
name: `Node ${i + 1}`,
|
||||
type: i % 3 === 0 ? 'n8n-nodes-base.httpRequest' :
|
||||
i % 3 === 1 ? 'n8n-nodes-base.slack' :
|
||||
'n8n-nodes-base.code',
|
||||
typeVersion: 1,
|
||||
position: [250 + (i % 5) * 200, 300 + Math.floor(i / 5) * 150],
|
||||
parameters: {
|
||||
url: '={{ $json.url }}',
|
||||
method: 'POST',
|
||||
body: '={{ JSON.stringify($json) }}',
|
||||
headers: {
|
||||
'Content-Type': 'application/json'
|
||||
}
|
||||
}
|
||||
})),
|
||||
connections: Object.fromEntries(
|
||||
Array.from({ length: 19 }, (_, i) => [
|
||||
`${i + 1}`,
|
||||
{
|
||||
main: [[{ node: `${i + 2}`, type: 'main', index: 0 }]]
|
||||
}
|
||||
])
|
||||
)
|
||||
};
|
||||
|
||||
beforeAll(async () => {
|
||||
storage = new SQLiteStorageService(':memory:');
|
||||
repository = new NodeRepository(storage);
|
||||
const loader = new NodeLoader(repository);
|
||||
await loader.loadPackage('n8n-nodes-base');
|
||||
|
||||
validator = new ConfigValidator(repository);
|
||||
enhancedValidator = new EnhancedConfigValidator(repository);
|
||||
expressionValidator = new ExpressionValidator();
|
||||
workflowValidator = new WorkflowValidator(repository);
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
storage.close();
|
||||
});
|
||||
|
||||
bench('validateNode - simple config minimal', async () => {
|
||||
await validator.validateNode('n8n-nodes-base.httpRequest', simpleConfig, 'minimal');
|
||||
}, {
|
||||
iterations: 1000,
|
||||
warmupIterations: 100,
|
||||
warmupTime: 500,
|
||||
time: 3000
|
||||
});
|
||||
|
||||
bench('validateNode - simple config strict', async () => {
|
||||
await validator.validateNode('n8n-nodes-base.httpRequest', simpleConfig, 'strict');
|
||||
}, {
|
||||
iterations: 500,
|
||||
warmupIterations: 50,
|
||||
warmupTime: 500,
|
||||
time: 3000
|
||||
});
|
||||
|
||||
bench('validateNode - complex config', async () => {
|
||||
await enhancedValidator.validateNode('n8n-nodes-base.slack', complexConfig, 'ai-friendly');
|
||||
}, {
|
||||
iterations: 500,
|
||||
warmupIterations: 50,
|
||||
warmupTime: 500,
|
||||
time: 3000
|
||||
});
|
||||
|
||||
bench('validateMinimal - missing fields check', async () => {
|
||||
await validator.validateMinimal('n8n-nodes-base.httpRequest', {});
|
||||
}, {
|
||||
iterations: 2000,
|
||||
warmupIterations: 200,
|
||||
warmupTime: 500,
|
||||
time: 3000
|
||||
});
|
||||
|
||||
bench('validateExpression - simple expression', async () => {
|
||||
expressionValidator.validateExpression('{{ $json.data }}');
|
||||
}, {
|
||||
iterations: 5000,
|
||||
warmupIterations: 500,
|
||||
warmupTime: 500,
|
||||
time: 3000
|
||||
});
|
||||
|
||||
bench('validateExpression - complex expression', async () => {
|
||||
expressionValidator.validateExpression('{{ $node["HTTP Request"].json.items.map(item => item.id).join(",") }}');
|
||||
}, {
|
||||
iterations: 2000,
|
||||
warmupIterations: 200,
|
||||
warmupTime: 500,
|
||||
time: 3000
|
||||
});
|
||||
|
||||
bench('validateWorkflow - simple workflow', async () => {
|
||||
await workflowValidator.validateWorkflow(simpleWorkflow);
|
||||
}, {
|
||||
iterations: 500,
|
||||
warmupIterations: 50,
|
||||
warmupTime: 500,
|
||||
time: 3000
|
||||
});
|
||||
|
||||
bench('validateWorkflow - complex workflow', async () => {
|
||||
await workflowValidator.validateWorkflow(complexWorkflow);
|
||||
}, {
|
||||
iterations: 100,
|
||||
warmupIterations: 10,
|
||||
warmupTime: 500,
|
||||
time: 3000
|
||||
});
|
||||
|
||||
bench('validateConnections - simple', async () => {
|
||||
workflowValidator.validateConnections(simpleWorkflow);
|
||||
}, {
|
||||
iterations: 2000,
|
||||
warmupIterations: 200,
|
||||
warmupTime: 500,
|
||||
time: 3000
|
||||
});
|
||||
|
||||
bench('validateConnections - complex', async () => {
|
||||
workflowValidator.validateConnections(complexWorkflow);
|
||||
}, {
|
||||
iterations: 500,
|
||||
warmupIterations: 50,
|
||||
warmupTime: 500,
|
||||
time: 3000
|
||||
});
|
||||
|
||||
bench('validateExpressions - workflow with many expressions', async () => {
|
||||
workflowValidator.validateExpressions(complexWorkflow);
|
||||
}, {
|
||||
iterations: 200,
|
||||
warmupIterations: 20,
|
||||
warmupTime: 500,
|
||||
time: 3000
|
||||
});
|
||||
|
||||
bench('getPropertyDependencies', async () => {
|
||||
await enhancedValidator.getPropertyDependencies('n8n-nodes-base.httpRequest');
|
||||
}, {
|
||||
iterations: 1000,
|
||||
warmupIterations: 100,
|
||||
warmupTime: 500,
|
||||
time: 3000
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user