Files
n8n-mcp/src/scripts/test-telemetry-mutations-verbose.ts
Romuald Członkowski 597bd290b6 fix: critical telemetry improvements for data quality and security (#421)
* fix: critical telemetry improvements for data quality and security

Fixed three critical issues in workflow mutation telemetry:

1. Fixed Inconsistent Sanitization (Security Critical)
   - Problem: 30% of workflows unsanitized, exposing credentials/tokens
   - Solution: Use robust WorkflowSanitizer.sanitizeWorkflowRaw()
   - Impact: 100% sanitization with 17 sensitive patterns redacted
   - Files: workflow-sanitizer.ts, mutation-tracker.ts

2. Enabled Validation Data Capture (Data Quality)
   - Problem: Zero validation metrics captured (all NULL)
   - Solution: Add pre/post mutation validation with WorkflowValidator
   - Impact: Measure mutation quality, track error resolution
   - Non-blocking validation that captures errors/warnings
   - Files: handlers-workflow-diff.ts

3. Improved Intent Capture (Data Quality)
   - Problem: 92.62% generic "Partial workflow update" intents
   - Solution: Enhanced docs + automatic intent inference
   - Impact: Meaningful intents auto-generated from operations
   - Files: n8n-update-partial-workflow.ts, handlers-workflow-diff.ts

Expected Results:
- 100% sanitization coverage (up from 70%)
- 100% validation capture (up from 0%)
- 50%+ meaningful intents (up from 7.33%)

Version bumped to 2.22.17

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Conceived by Romuald Członkowski - https://www.aiadvisors.pl/en

Co-Authored-By: Claude <noreply@anthropic.com>

* perf: implement validator instance caching to avoid redundant initialization

- Add module-level cached WorkflowValidator instance
- Create getValidator() helper to reuse validator across mutations
- Update pre/post mutation validation to use cached instance
- Avoids redundant NodeSimilarityService initialization on every mutation

Conceived by Romuald Członkowski - https://www.aiadvisors.pl/en

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: restore backward-compatible sanitization with context preservation

Fixed CI test failures by updating WorkflowSanitizer to use pattern-specific
placeholders while maintaining backward compatibility:

Changes:
- Convert SENSITIVE_PATTERNS to PatternDefinition objects with specific placeholders
- Update sanitizeString() to preserve context (Bearer prefix, URL paths)
- Refactor sanitizeObject() to handle sensitive fields vs URL fields differently
- Remove overly greedy field patterns that conflicted with token patterns

Pattern-specific placeholders:
- [REDACTED_URL_WITH_AUTH] for URLs with credentials
- [REDACTED_TOKEN] for long tokens (32+ chars)
- [REDACTED_APIKEY] for OpenAI-style keys
- Bearer [REDACTED] for Bearer tokens (preserves "Bearer " prefix)
- [REDACTED] for generic sensitive fields

Test Results:
- All 13 mutation-tracker tests passing
- URL with auth: preserves path after credentials
- Long tokens: properly detected and marked
- OpenAI keys: correctly identified
- Bearer tokens: prefix preserved
- Sensitive field names: generic redaction for non-URL fields

Fixes #421 CI failures

Conceived by Romuald Członkowski - https://www.aiadvisors.pl/en

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: prevent double-redaction in workflow sanitizer

Added safeguard to stop pattern matching once a placeholder is detected,
preventing token patterns from matching text inside placeholders like
[REDACTED_URL_WITH_AUTH].

Also expanded database URL pattern to match full URLs including port and
path, and updated test expectations to match context-preserving sanitization.

Fixes:
- Database URLs now properly sanitized to [REDACTED_URL_WITH_AUTH]
- Prevents [[REDACTED]] double-redaction issue
- All 25 workflow-sanitizer tests passing
- No regression in mutation-tracker tests

Conceived by Romuald Członkowski - www.aiadvisors.pl/en

---------

Co-authored-by: Claude <noreply@anthropic.com>
2025-11-13 22:13:31 +01:00

152 lines
4.5 KiB
TypeScript

/**
* Test telemetry mutations with enhanced logging
* Verifies that mutations are properly tracked and persisted
*/
import { telemetry } from '../telemetry/telemetry-manager.js';
import { TelemetryConfigManager } from '../telemetry/config-manager.js';
import { logger } from '../utils/logger.js';
async function testMutations() {
console.log('Starting verbose telemetry mutation test...\n');
const configManager = TelemetryConfigManager.getInstance();
console.log('Telemetry config is enabled:', configManager.isEnabled());
console.log('Telemetry config file:', configManager['configPath']);
// Test data with valid workflow structure
const testMutation = {
sessionId: 'test_session_' + Date.now(),
toolName: 'n8n_update_partial_workflow',
userIntent: 'Add a Merge node for data consolidation',
operations: [
{
type: 'addNode',
nodeId: 'Merge1',
node: {
id: 'Merge1',
type: 'n8n-nodes-base.merge',
name: 'Merge',
position: [600, 200],
parameters: {}
}
},
{
type: 'addConnection',
source: 'previous_node',
target: 'Merge1'
}
],
workflowBefore: {
id: 'test-workflow',
name: 'Test Workflow',
active: true,
nodes: [
{
id: 'previous_node',
type: 'n8n-nodes-base.manualTrigger',
name: 'When called',
position: [300, 200],
parameters: {}
}
],
connections: {},
nodeIds: []
},
workflowAfter: {
id: 'test-workflow',
name: 'Test Workflow',
active: true,
nodes: [
{
id: 'previous_node',
type: 'n8n-nodes-base.manualTrigger',
name: 'When called',
position: [300, 200],
parameters: {}
},
{
id: 'Merge1',
type: 'n8n-nodes-base.merge',
name: 'Merge',
position: [600, 200],
parameters: {}
}
],
connections: {
'previous_node': [
{
node: 'Merge1',
type: 'main',
index: 0,
source: 0,
destination: 0
}
]
},
nodeIds: []
},
mutationSuccess: true,
durationMs: 125
};
console.log('\nTest Mutation Data:');
console.log('==================');
console.log(JSON.stringify({
intent: testMutation.userIntent,
tool: testMutation.toolName,
operationCount: testMutation.operations.length,
sessionId: testMutation.sessionId
}, null, 2));
console.log('\n');
// Call trackWorkflowMutation
console.log('Calling telemetry.trackWorkflowMutation...');
try {
await telemetry.trackWorkflowMutation(testMutation);
console.log('✓ trackWorkflowMutation completed successfully\n');
} catch (error) {
console.error('✗ trackWorkflowMutation failed:', error);
console.error('\n');
}
// Check queue size before flush
const metricsBeforeFlush = telemetry.getMetrics();
console.log('Metrics before flush:');
console.log('- mutationQueueSize:', metricsBeforeFlush.tracking.mutationQueueSize);
console.log('- eventsTracked:', metricsBeforeFlush.processing.eventsTracked);
console.log('- eventsFailed:', metricsBeforeFlush.processing.eventsFailed);
console.log('\n');
// Flush telemetry with 10-second wait for Supabase
console.log('Flushing telemetry (waiting 10 seconds for Supabase)...');
try {
await telemetry.flush();
console.log('✓ Telemetry flush completed\n');
} catch (error) {
console.error('✗ Flush failed:', error);
console.error('\n');
}
// Wait a bit for async operations
await new Promise(resolve => setTimeout(resolve, 2000));
// Get final metrics
const metricsAfterFlush = telemetry.getMetrics();
console.log('Metrics after flush:');
console.log('- mutationQueueSize:', metricsAfterFlush.tracking.mutationQueueSize);
console.log('- eventsTracked:', metricsAfterFlush.processing.eventsTracked);
console.log('- eventsFailed:', metricsAfterFlush.processing.eventsFailed);
console.log('- batchesSent:', metricsAfterFlush.processing.batchesSent);
console.log('- batchesFailed:', metricsAfterFlush.processing.batchesFailed);
console.log('- circuitBreakerState:', metricsAfterFlush.processing.circuitBreakerState);
console.log('\n');
console.log('Test completed. Check workflow_mutations table in Supabase.');
}
testMutations().catch(error => {
console.error('Test failed:', error);
process.exit(1);
});