Commit Graph

8 Commits

Author SHA1 Message Date
Romuald Członkowski
c8c76e435d fix: critical memory leak from per-session database connections (#554)
* fix: critical memory leak from per-session database connections (#542)

Each MCP session was creating its own database connection (~900MB),
causing OOM kills every ~20 minutes with 3-4 concurrent sessions.

Changes:
- Add SharedDatabase singleton pattern - all sessions share ONE connection
- Reduce session timeout from 30 min to 5 min (configurable)
- Add eager cleanup for reconnecting instances
- Fix telemetry event listener leak

Memory impact: ~900MB/session → ~68MB shared + ~5MB/session overhead

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Conceived by Romuald Czlonkowski - https://www.aiadvisors.pl/en

* fix: resolve test failures from shared database race conditions

- Fix `shutdown()` to respect shared database pattern (was directly closing)
- Add `await this.initialized` in both `close()` and `shutdown()` to prevent
  race condition where cleanup runs while initialization is in progress
- Add double-shutdown protection with `isShutdown` flag
- Export `SharedDatabaseState` type for proper typing
- Include error details in debug logs
- Add MCP server close to `shutdown()` for consistency with `close()`
- Null out `earlyLogger` in `shutdown()` for consistency

The CI test failure "The database connection is not open" was caused by:
1. `shutdown()` directly calling `this.db.close()` which closed the SHARED
   database connection, breaking subsequent tests
2. Race condition where `shutdown()` ran before initialization completed

Conceived by Romuald Członkowski - www.aiadvisors.pl/en

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* test: add unit tests for shared-database module

Add comprehensive unit tests covering:
- getSharedDatabase: initialization, reuse, different path error, concurrent requests
- releaseSharedDatabase: refCount decrement, double-release guard
- closeSharedDatabase: state clearing, error handling, re-initialization
- Helper functions: isSharedDatabaseInitialized, getSharedDatabaseRefCount

21 tests covering the singleton database connection pattern used to prevent
~900MB memory leaks per session.

Conceived by Romuald Członkowski - www.aiadvisors.pl/en

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-23 19:51:22 +01:00
Romuald Członkowski
0d2d9bdd52 fix: Critical memory leak in sql.js adapter (fixes #330) (#335)
* fix: Critical memory leak in sql.js adapter (fixes #330)

Resolves critical memory leak causing growth from 100Mi to 2.2GB over 72 hours in Docker/Kubernetes deployments.

Problem Analysis:
- Environment: Kubernetes/Docker using sql.js fallback
- Growth rate: ~23 MB/hour (444Mi after 19 hours)
- Pattern: Linear accumulation, garbage collection couldn't keep pace
- Impact: OOM kills every 24-48 hours in memory-limited pods

Root Causes:
1. Over-aggressive save triggering: prepare() called scheduleSave() on reads
2. Too frequent saves: 100ms debounce = 3-5 saves/second under load
3. Double allocation: Buffer.from() copied Uint8Array (4-10MB per save)
4. No cleanup: Relied solely on GC which couldn't keep pace
5. Docker limitation: Missing build tools forced sql.js instead of better-sqlite3

Code-Level Fixes (sql.js optimization):
 Removed scheduleSave() from prepare() (read operations don't modify DB)
 Increased debounce: 100ms → 5000ms (98% reduction in save frequency)
 Removed Buffer.from() copy (50% reduction in temporary allocations)
 Made save interval configurable via SQLJS_SAVE_INTERVAL_MS env var
 Added input validation (minimum 100ms, falls back to 5000ms default)

Infrastructure Fix (Dockerfile):
 Added build tools (python3, make, g++) to main Dockerfile
 Compile better-sqlite3 during npm install, then remove build tools
 Image size increase: ~5-10MB (acceptable for eliminating memory leak)
 Railway Dockerfile already had build tools (added explanatory comment)

Impact:
With better-sqlite3 (now default in Docker):
- Memory: Stable at ~100-120 MB (native SQLite)
- Performance: Better than sql.js (no WASM overhead)
- No periodic saves needed (writes directly to disk)
- Eliminates memory leak entirely

With sql.js (fallback only):
- Memory: Stable at 150-200 MB (vs 2.2GB after 3 days)
- No OOM kills in long-running Kubernetes pods
- Reduced CPU usage (98% fewer disk writes)
- Same data safety (5-second save window acceptable)

Configuration:
- New env var: SQLJS_SAVE_INTERVAL_MS (default: 5000)
- Only relevant when sql.js fallback is used
- Minimum: 100ms, invalid values fall back to default

Testing:
 All unit tests passing
 New integration tests for memory leak prevention
 TypeScript compilation successful
 Docker builds verified (build tools working)

Files Modified:
- src/database/database-adapter.ts: SQLJSAdapter optimization
- Dockerfile: Added build tools for better-sqlite3
- Dockerfile.railway: Added documentation comment
- tests/unit/database/database-adapter-unit.test.ts: New test suites
- tests/integration/database/sqljs-memory-leak.test.ts: Integration tests
- package.json: Version bump to 2.20.2
- package.runtime.json: Version bump to 2.20.2
- CHANGELOG.md: Comprehensive v2.20.2 entry
- README.md: Database & Memory Configuration section

Closes #330

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: Address code review findings for memory leak fix (#330)

## Code Review Fixes

1. **Test Assertion Error (line 292)** - CRITICAL
   - Fixed incorrect assertion in sqljs-memory-leak test
   - Changed from `expect(saveCallback).toBeLessThan(10)`
   - To: `expect(saveCallback.mock.calls.length).toBeLessThan(10)`
   -  Test now passes (12/12 tests passing)

2. **Upper Bound Validation**
   - Added maximum value validation for SQLJS_SAVE_INTERVAL_MS
   - Valid range: 100ms - 60000ms (1 minute)
   - Falls back to default 5000ms if out of range
   - Location: database-adapter.ts:255

3. **Railway Dockerfile Optimization**
   - Removed build tools after installing dependencies
   - Reduces image size by ~50-100MB
   - Pattern: install → build native modules → remove tools
   - Location: Dockerfile.railway:38-41

4. **Defensive Programming**
   - Added `closed` flag to prevent double-close issues
   - Early return if already closed
   - Location: database-adapter.ts:236, 283-286

5. **Documentation Improvements**
   - Added comprehensive comments for DEFAULT_SAVE_INTERVAL_MS
   - Documented data loss window trade-off (5 seconds)
   - Explained constructor optimization (no initial save)
   - Clarified scheduleSave() debouncing under load

6. **CHANGELOG Accuracy**
   - Fixed discrepancy about explicit cleanup
   - Updated to reflect automatic cleanup via function scope
   - Removed misleading `data = null` reference

## Verification

-  Build: Success
-  Lint: No errors
-  Critical test: sqljs-memory-leak (12/12 passing)
-  All code review findings addressed

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
2025-10-18 22:11:27 +02:00
czlonkowski
b65a2f8f3d chore: update n8n dependencies to latest versions
- Updated n8n-nodes-base to 1.106.3
- Updated @n8n/n8n-nodes-langchain to 1.106.3
- Enhanced SQL.js compatibility in database adapter
- Fixed parameter binding and state management in SQLJSStatement
- Rebuilt node database with 535 nodes
- All tests passing with Node.js v22.17.0 LTS
2025-09-04 10:24:33 +02:00
czlonkowski
1170ad27a6 fix: resolve WASM file loading issue for npx execution (closes #31)
- Enhanced database adapter to support multiple WASM file resolution strategies
- Added require.resolve() for reliable package location in npm environments
- Made better-sqlite3 an optional dependency
- Improved error handling with clear messages
- Updated version to 2.7.13
- Updated CHANGELOG and README badges
2025-07-11 08:48:37 +02:00
czlonkowski
e8f6b684f0 fix: make FTS5 optional for template search (fixes Claude Desktop compatibility)
- Added runtime FTS5 detection in database adapters
- Removed FTS5 from required schema to prevent "no such module" errors
- FTS5 tables/triggers created conditionally only if supported
- Template search automatically falls back to LIKE when FTS5 unavailable
- Works in ALL SQLite environments (Claude Desktop, restricted envs, etc.)

This ensures search_templates() works correctly regardless of SQLite build,
while still providing optimal performance when FTS5 is available.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-07-10 12:13:08 +02:00
czlonkowski
19b9b5ca2d fix: webhook and 4 other nodes incorrectly marked as non-triggers
Fixed issue where Docker images using sql.js adapter returned boolean fields
as strings, causing is_trigger=0 to evaluate as true instead of false.

Changes:
- Added convertIntegerColumns() to sql.js adapter to convert SQLite integers
- Updated server.ts and node-repository.ts to use Number() conversion as backup
- Added test script to verify fix works with sql.js adapter

This fixes webhook, cron, interval, and emailReadImap nodes showing
isTrigger: false in Docker deployments.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-07-06 17:01:05 +02:00
czlonkowski
a688ad3d14 fix: resolve Docker stdio initialization timeout issue
- Add InitializeRequestSchema handler to MCP server
- Implement stdout flushing for Docker environments
- Create stdio-wrapper for clean JSON-RPC communication
- Update docker-entrypoint.sh to prevent stdout pollution
- Fix logger to check MCP_MODE before level check

These changes ensure the MCP server responds to initialization requests
within Claude Desktop's 60-second timeout when running in Docker.
2025-06-17 09:12:01 +02:00
czlonkowski
b476d36275 feat: implement universal Node.js compatibility with automatic database adapter fallback 2025-06-12 23:51:47 +02:00