feat: telemetry-driven quick wins to reduce AI agent validation errors by 30-40%

Enhanced tools documentation, duplicate ID errors, and AI Agent validator based on telemetry analysis of 593 validation errors across 3 categories:
- 378 errors: Duplicate node IDs (64%)
- 179 errors: AI Agent configuration (30%)
- 36 errors: Other validations (6%)

Quick Win #1: Enhanced tools documentation (src/mcp/tools-documentation.ts)
- Added prominent warnings to call get_node_essentials() FIRST before configuring nodes
- Emphasized 5KB vs 100KB+ size difference between essentials and full info
- Updated workflow patterns to prioritize essentials over get_node_info

Quick Win #2: Improved duplicate ID error messages (src/services/workflow-validator.ts)
- Added crypto import for UUID generation examples
- Enhanced error messages with node indices, names, and types
- Included crypto.randomUUID() example in error messages
- Helps AI agents understand EXACTLY which nodes conflict and how to fix

Quick Win #3: Added AI Agent node-specific validator (src/services/node-specific-validators.ts)
- Validates prompt configuration (promptType + text requirement)
- Checks maxIterations bounds (1-50 recommended)
- Suggests error handling (onError + retryOnFail)
- Warns about high iteration limits (cost/performance impact)
- Integrated into enhanced-config-validator.ts

Test Coverage:
- Added duplicate ID validation tests (workflow-validator.test.ts)
- Added AI Agent validator tests (node-specific-validators.test.ts:2312-2491)
- All new tests passing (3527 total passing)

Version: 2.22.12 → 2.22.13

Expected Impact: 30-40% reduction in AI agent validation errors

Technical Details:
- Telemetry analysis: 593 validation errors (Dec 2024 - Jan 2025)
- 100% error recovery rate maintained (validation working correctly)
- Root cause: Documentation/guidance gaps, not validation logic failures
- Solution: Proactive guidance at decision points

References:
- Telemetry analysis findings
- Issue #392 (helpful error messages pattern)
- Existing Slack validator pattern (node-specific-validators.ts:98-230)

Concieved by Romuald Członkowski - www.aiadvisors.pl/en
This commit is contained in:
czlonkowski
2025-11-08 18:07:26 +01:00
parent eee52a7f53
commit 60ab66d64d
20 changed files with 4918 additions and 15 deletions

View File

@@ -3,6 +3,7 @@
* Validates complete workflow structure, connections, and node configurations
*/
import crypto from 'crypto';
import { NodeRepository } from '../database/node-repository';
import { EnhancedConfigValidator } from './enhanced-config-validator';
import { ExpressionValidator } from './expression-validator';
@@ -297,8 +298,11 @@ export class WorkflowValidator {
// Check for duplicate node names
const nodeNames = new Set<string>();
const nodeIds = new Set<string>();
for (const node of workflow.nodes) {
const nodeIdToIndex = new Map<string, number>(); // Track which node index has which ID
for (let i = 0; i < workflow.nodes.length; i++) {
const node = workflow.nodes[i];
if (nodeNames.has(node.name)) {
result.errors.push({
type: 'error',
@@ -310,13 +314,18 @@ export class WorkflowValidator {
nodeNames.add(node.name);
if (nodeIds.has(node.id)) {
const firstNodeIndex = nodeIdToIndex.get(node.id);
const firstNode = firstNodeIndex !== undefined ? workflow.nodes[firstNodeIndex] : undefined;
result.errors.push({
type: 'error',
nodeId: node.id,
message: `Duplicate node ID: "${node.id}"`
message: `Duplicate node ID: "${node.id}". Node at index ${i} (name: "${node.name}", type: "${node.type}") conflicts with node at index ${firstNodeIndex} (name: "${firstNode?.name || 'unknown'}", type: "${firstNode?.type || 'unknown'}"). Each node must have a unique ID. Generate a new UUID using crypto.randomUUID() - Example: {id: "${crypto.randomUUID()}", name: "${node.name}", type: "${node.type}", ...}`
});
} else {
nodeIds.add(node.id);
nodeIdToIndex.set(node.id, i);
}
nodeIds.add(node.id);
}
// Count trigger nodes using shared trigger detection