mirror of
https://github.com/czlonkowski/n8n-mcp.git
synced 2026-01-29 22:12:05 +00:00
60ab66d64d
Enhanced tools documentation, duplicate ID errors, and AI Agent validator based on telemetry analysis of 593 validation errors across 3 categories: - 378 errors: Duplicate node IDs (64%) - 179 errors: AI Agent configuration (30%) - 36 errors: Other validations (6%) Quick Win #1: Enhanced tools documentation (src/mcp/tools-documentation.ts) - Added prominent warnings to call get_node_essentials() FIRST before configuring nodes - Emphasized 5KB vs 100KB+ size difference between essentials and full info - Updated workflow patterns to prioritize essentials over get_node_info Quick Win #2: Improved duplicate ID error messages (src/services/workflow-validator.ts) - Added crypto import for UUID generation examples - Enhanced error messages with node indices, names, and types - Included crypto.randomUUID() example in error messages - Helps AI agents understand EXACTLY which nodes conflict and how to fix Quick Win #3: Added AI Agent node-specific validator (src/services/node-specific-validators.ts) - Validates prompt configuration (promptType + text requirement) - Checks maxIterations bounds (1-50 recommended) - Suggests error handling (onError + retryOnFail) - Warns about high iteration limits (cost/performance impact) - Integrated into enhanced-config-validator.ts Test Coverage: - Added duplicate ID validation tests (workflow-validator.test.ts) - Added AI Agent validator tests (node-specific-validators.test.ts:2312-2491) - All new tests passing (3527 total passing) Version: 2.22.12 → 2.22.13 Expected Impact: 30-40% reduction in AI agent validation errors Technical Details: - Telemetry analysis: 593 validation errors (Dec 2024 - Jan 2025) - 100% error recovery rate maintained (validation working correctly) - Root cause: Documentation/guidance gaps, not validation logic failures - Solution: Proactive guidance at decision points References: - Telemetry analysis findings - Issue #392 (helpful error messages pattern) - Existing Slack validator pattern (node-specific-validators.ts:98-230) Concieved by Romuald Członkowski - www.aiadvisors.pl/en
210 lines
5.8 KiB
Markdown
210 lines
5.8 KiB
Markdown
# N8N-MCP Validation Analysis: Quick Reference
|
|
|
|
**Analysis Date**: November 8, 2025 | **Data Period**: 90 days | **Sample Size**: 29,218 events
|
|
|
|
---
|
|
|
|
## The Core Finding
|
|
|
|
**Validation is working perfectly. Guidance is the problem.**
|
|
|
|
- 29,218 validation events successfully prevented bad deployments
|
|
- 100% of agents fix errors same-day (proving feedback works)
|
|
- 12.6% error rate for advanced users (who attempt complex workflows)
|
|
- High error volume = high usage, not broken system
|
|
|
|
---
|
|
|
|
## Top 3 Problem Areas (75% of errors)
|
|
|
|
| Area | Errors | Root Cause | Quick Fix |
|
|
|------|--------|-----------|-----------|
|
|
| **Workflow Structure** | 1,268 (26%) | JSON malformation | Better error messages with examples |
|
|
| **Connections** | 676 (14%) | Syntax unintuitive | Create connections guide with diagrams |
|
|
| **Required Fields** | 378 (8%) | Not marked upfront | Add "⚠️ REQUIRED" to tool responses |
|
|
|
|
---
|
|
|
|
## Problem Nodes (By Frequency)
|
|
|
|
```
|
|
Webhook/Trigger ......... 127 failures (40 users)
|
|
Slack .................. 73 failures (2 users)
|
|
AI Agent ............... 36 failures (20 users)
|
|
HTTP Request ........... 31 failures (13 users)
|
|
OpenAI ................. 35 failures (8 users)
|
|
```
|
|
|
|
---
|
|
|
|
## Top 5 Validation Errors
|
|
|
|
1. **"Duplicate node ID: undefined"** (179)
|
|
- Fix: Point to exact location + show example format
|
|
|
|
2. **"Single-node workflows only valid for webhooks"** (58)
|
|
- Fix: Create webhook guide explaining rule
|
|
|
|
3. **"responseNode requires onError: continueRegularOutput"** (57)
|
|
- Fix: Same guide + inline error context
|
|
|
|
4. **"Required property X cannot be empty"** (25)
|
|
- Fix: Mark required fields before validation
|
|
|
|
5. **"Duplicate node name: undefined"** (61)
|
|
- Fix: Related to structural issues, same solution as #1
|
|
|
|
---
|
|
|
|
## Success Indicators
|
|
|
|
✓ **Agents learn from errors**: 100% same-day correction rate
|
|
✓ **Validation catches issues**: Prevents bad deployments
|
|
✓ **Feedback is clear**: Quick fixes show error messages work
|
|
✓ **No systemic failures**: No "unfixable" errors
|
|
|
|
---
|
|
|
|
## What Works Well
|
|
|
|
- Error messages lead to immediate corrections
|
|
- Agents retry and succeed same-day
|
|
- Validation prevents broken workflows
|
|
- 9,021 users actively using system
|
|
|
|
---
|
|
|
|
## What Needs Improvement
|
|
|
|
1. Required fields not marked in tool responses
|
|
2. Error messages don't show valid options for enums
|
|
3. Workflow structure documentation lacks examples
|
|
4. Connection syntax unintuitive/undocumented
|
|
5. Some error messages too generic
|
|
|
|
---
|
|
|
|
## Implementation Plan
|
|
|
|
### Phase 1 (2 weeks): Quick Wins
|
|
- Enhanced error messages (location + example)
|
|
- Required field markers in tools
|
|
- Webhook configuration guide
|
|
- **Expected Impact**: 25-30% failure reduction
|
|
|
|
### Phase 2 (2 weeks): Documentation
|
|
- Enum value suggestions in validation
|
|
- Workflow connections guide
|
|
- Error handler configuration guide
|
|
- AI Agent validation improvements
|
|
- **Expected Impact**: Additional 15-20% reduction
|
|
|
|
### Phase 3 (2 weeks): Advanced Features
|
|
- Improved search with config hints
|
|
- Node type fuzzy matching
|
|
- KPI tracking setup
|
|
- Test coverage
|
|
- **Expected Impact**: Additional 10-15% reduction
|
|
|
|
**Total Impact**: 50-65% failure reduction (target: 6-7% error rate)
|
|
|
|
---
|
|
|
|
## Key Metrics
|
|
|
|
| Metric | Current | Target | Timeline |
|
|
|--------|---------|--------|----------|
|
|
| Validation failure rate | 12.6% | 6-7% | 6 weeks |
|
|
| First-attempt success | ~77% | 85%+ | 6 weeks |
|
|
| Retry success | 100% | 100% | N/A |
|
|
| Webhook failures | 127 | <30 | Week 2 |
|
|
| Connection errors | 676 | <270 | Week 4 |
|
|
|
|
---
|
|
|
|
## Files Delivered
|
|
|
|
1. **VALIDATION_ANALYSIS_REPORT.md** (27KB)
|
|
- Complete analysis with 16 SQL queries
|
|
- Detailed findings by category
|
|
- 8 actionable recommendations
|
|
|
|
2. **VALIDATION_ANALYSIS_SUMMARY.md** (13KB)
|
|
- Executive summary (one-page)
|
|
- Key metrics scorecard
|
|
- Top recommendations with ROI
|
|
|
|
3. **IMPLEMENTATION_ROADMAP.md** (4.3KB)
|
|
- 6-week implementation plan
|
|
- Phase-by-phase breakdown
|
|
- Code locations and effort estimates
|
|
|
|
4. **ANALYSIS_QUICK_REFERENCE.md** (this file)
|
|
- Quick lookup reference
|
|
- Top problems at a glance
|
|
- Decision-making summary
|
|
|
|
---
|
|
|
|
## Next Steps
|
|
|
|
1. **Week 1**: Review analysis + get team approval
|
|
2. **Week 2**: Start Phase 1 (error messages + markers)
|
|
3. **Week 4**: Deploy Phase 1 + start Phase 2
|
|
4. **Week 6**: Deploy Phase 2 + start Phase 3
|
|
5. **Week 8**: Deploy Phase 3 + measure impact
|
|
6. **Week 9+**: Monitor KPIs + iterate
|
|
|
|
---
|
|
|
|
## Key Recommendations Priority
|
|
|
|
### HIGH (Do First - Week 1-2)
|
|
1. Enhance structure error messages
|
|
2. Add required field markers to tools
|
|
3. Create webhook configuration guide
|
|
|
|
### MEDIUM (Do Next - Week 3-4)
|
|
4. Add enum suggestions to validation responses
|
|
5. Create workflow connections guide
|
|
6. Add AI Agent node validation
|
|
|
|
### LOW (Do Later - Week 5-6)
|
|
7. Enhance search with config hints
|
|
8. Build fuzzy node matcher
|
|
9. Setup KPI tracking
|
|
|
|
---
|
|
|
|
## Discussion Points
|
|
|
|
**Q: Why don't we just weaken validation?**
|
|
A: Validation prevents 29,218 bad deployments. That's its job. We improve guidance instead.
|
|
|
|
**Q: Are agents really learning from errors?**
|
|
A: Yes, 100% same-day recovery across 661 user-date pairs with errors.
|
|
|
|
**Q: Why do documentation readers have higher error rates?**
|
|
A: They attempt more complex workflows (6.8x more attempts). Success rate is still 87.4%.
|
|
|
|
**Q: Which node needs the most help?**
|
|
A: Webhook/Trigger configuration (127 failures). Most urgent fix.
|
|
|
|
**Q: Can we hit 50% reduction in 6 weeks?**
|
|
A: Yes, analysis shows 50-65% reduction is achievable with these changes.
|
|
|
|
---
|
|
|
|
## Contact & Questions
|
|
|
|
For detailed information:
|
|
- Full analysis: `VALIDATION_ANALYSIS_REPORT.md`
|
|
- Executive summary: `VALIDATION_ANALYSIS_SUMMARY.md`
|
|
- Implementation plan: `IMPLEMENTATION_ROADMAP.md`
|
|
|
|
---
|
|
|
|
**Report Status**: Complete and Ready for Action
|
|
**Confidence Level**: High (9,021 users, 29,218 events, comprehensive analysis)
|
|
**Generated**: November 8, 2025
|