automaker/docs/server/debug-api.md

Debug API Documentation

The Debug API provides endpoints for monitoring server performance, memory usage, CPU metrics, and process tracking. These endpoints are only available in development mode or when ENABLE_DEBUG_PANEL=true.

Table of Contents

• Overview
• Authentication
• Metrics Endpoints
  • GET /api/debug/metrics
  • POST /api/debug/metrics/start
  • POST /api/debug/metrics/stop
  • POST /api/debug/metrics/gc
  • POST /api/debug/metrics/clear
• Process Endpoints
  • GET /api/debug/processes
  • GET /api/debug/processes/summary
  • GET /api/debug/processes/:id
• Agent Resource Metrics Endpoints
  • GET /api/debug/agents
  • GET /api/debug/agents/summary
  • GET /api/debug/agents/:id/metrics
• Types
• Events

---

Overview

The Debug API is designed for development and debugging purposes. It provides:

• Memory Monitoring: Track heap usage, RSS, and detect memory leaks
• CPU Monitoring: Track CPU usage percentage and event loop lag
• Process Tracking: Monitor agents, terminals, CLIs, and worker processes
• Trend Analysis: Detect memory leaks using linear regression

Enabling the Debug API

The Debug API is enabled when:

• NODE_ENV !== 'production' (development mode), OR
• ENABLE_DEBUG_PANEL=true environment variable is set

---

Authentication

All debug endpoints require authentication. Requests must include a valid session token or use the standard Automaker authentication mechanism.

---

Metrics Endpoints

GET /api/debug/metrics

Returns the current metrics snapshot including memory, CPU, and process information.

Response

{
  "active": true,
  "config": {
    "memoryEnabled": true,
    "cpuEnabled": true,
    "processTrackingEnabled": true,
    "collectionInterval": 1000,
    "maxDataPoints": 60,
    "leakThreshold": 1048576
  },
  "snapshot": {
    "timestamp": 1704067200000,
    "memory": {
      "timestamp": 1704067200000,
      "server": {
        "heapTotal": 104857600,
        "heapUsed": 52428800,
        "external": 5242880,
        "rss": 157286400,
        "arrayBuffers": 1048576
      }
    },
    "cpu": {
      "timestamp": 1704067200000,
      "server": {
        "percentage": 25.5,
        "user": 1000000,
        "system": 500000
      },
      "eventLoopLag": 5
    },
    "processes": [],
    "processSummary": {
      "total": 0,
      "running": 0,
      "idle": 0,
      "stopped": 0,
      "errored": 0,
      "byType": {
        "agent": 0,
        "cli": 0,
        "terminal": 0,
        "worker": 0
      }
    },
    "memoryTrend": {
      "growthRate": 1024,
      "isLeaking": false,
      "confidence": 0.85,
      "sampleCount": 30,
      "windowDuration": 30000
    }
  }
}

---

POST /api/debug/metrics/start

Starts metrics collection with optional configuration overrides.

Request Body (optional)

{
  "config": {
    "collectionInterval": 2000,
    "maxDataPoints": 100,
    "memoryEnabled": true,
    "cpuEnabled": true,
    "leakThreshold": 2097152
  }
}

Configuration Limits (enforced server-side)

Field	Min	Max	Default
collectionInterval	100ms	60000ms	1000ms
maxDataPoints	10	10000	60
leakThreshold	1KB	100MB	1MB

Response

{
  "active": true,
  "config": {
    "memoryEnabled": true,
    "cpuEnabled": true,
    "processTrackingEnabled": true,
    "collectionInterval": 2000,
    "maxDataPoints": 100,
    "leakThreshold": 2097152
  }
}

---

POST /api/debug/metrics/stop

Stops metrics collection.

Response

{
  "active": false,
  "config": {
    "memoryEnabled": true,
    "cpuEnabled": true,
    "processTrackingEnabled": true,
    "collectionInterval": 1000,
    "maxDataPoints": 60,
    "leakThreshold": 1048576
  }
}

---

POST /api/debug/metrics/gc

Forces garbage collection if Node.js was started with --expose-gc flag.

Response (success)

{
  "success": true,
  "message": "Garbage collection triggered"
}

Response (not available)

{
  "success": false,
  "message": "Garbage collection not available (start Node.js with --expose-gc flag)"
}

---

POST /api/debug/metrics/clear

Clears the metrics history buffer.

Response

{
  "success": true,
  "message": "Metrics history cleared"
}

---

Process Endpoints

GET /api/debug/processes

Returns a list of tracked processes with optional filtering.

Query Parameters

Parameter	Type	Description
type	string	Filter by process type: agent, cli, terminal, worker
status	string	Filter by status: starting, running, idle, stopping, stopped, error
includeStopped	string	Set to "true" to include stopped processes
sessionId	string	Filter by session ID
featureId	string	Filter by feature ID

Example Request

GET /api/debug/processes?type=agent&status=running&includeStopped=true

Response

{
  "processes": [
    {
      "id": "agent-12345",
      "pid": 1234,
      "type": "agent",
      "name": "Feature Agent",
      "status": "running",
      "startedAt": 1704067200000,
      "memoryUsage": 52428800,
      "cpuUsage": 15.5,
      "featureId": "feature-123",
      "sessionId": "session-456"
    }
  ],
  "summary": {
    "total": 5,
    "running": 2,
    "idle": 1,
    "stopped": 1,
    "errored": 1,
    "byType": {
      "agent": 2,
      "cli": 1,
      "terminal": 2,
      "worker": 0
    }
  }
}

---

GET /api/debug/processes/summary

Returns summary statistics for all tracked processes.

Response

{
  "total": 5,
  "running": 2,
  "idle": 1,
  "stopped": 1,
  "errored": 1,
  "byType": {
    "agent": 2,
    "cli": 1,
    "terminal": 2,
    "worker": 0
  }
}

---

GET /api/debug/processes/:id

Returns details for a specific process.

Path Parameters

Parameter	Type	Description
id	string	Process ID (max 256 characters)

Response (success)

{
  "id": "agent-12345",
  "pid": 1234,
  "type": "agent",
  "name": "Feature Agent",
  "status": "running",
  "startedAt": 1704067200000,
  "memoryUsage": 52428800,
  "cpuUsage": 15.5,
  "featureId": "feature-123",
  "sessionId": "session-456",
  "command": "node agent.js",
  "cwd": "/path/to/project"
}

Response (not found)

{
  "error": "Process not found",
  "id": "non-existent-id"
}

Response (invalid ID)

{
  "error": "Invalid process ID format"
}

---

Agent Resource Metrics Endpoints

These endpoints provide detailed resource usage metrics for agent processes, including file I/O, tool usage, bash commands, and memory tracking.

GET /api/debug/agents

Returns all agent processes with their detailed resource metrics.

Response

{
  "agents": [
    {
      "id": "agent-feature-123",
      "pid": -1,
      "type": "agent",
      "name": "Feature Agent",
      "status": "running",
      "startedAt": 1704067200000,
      "featureId": "feature-123",
      "resourceMetrics": {
        "agentId": "agent-feature-123",
        "featureId": "feature-123",
        "startedAt": 1704067200000,
        "lastUpdatedAt": 1704067260000,
        "duration": 60000,
        "isRunning": true,
        "memory": {
          "startHeapUsed": 52428800,
          "currentHeapUsed": 57671680,
          "peakHeapUsed": 58720256,
          "deltaHeapUsed": 5242880,
          "samples": [...]
        },
        "fileIO": {
          "reads": 25,
          "bytesRead": 524288,
          "writes": 5,
          "bytesWritten": 10240,
          "edits": 3,
          "globs": 10,
          "greps": 8,
          "filesAccessed": ["src/index.ts", "src/utils.ts", ...]
        },
        "tools": {
          "totalInvocations": 51,
          "byTool": {
            "Read": 25,
            "Glob": 10,
            "Grep": 8,
            "Write": 5,
            "Edit": 3
          },
          "avgExecutionTime": 150,
          "totalExecutionTime": 7650,
          "failedInvocations": 1
        },
        "bash": {
          "commandCount": 5,
          "totalExecutionTime": 2500,
          "failedCommands": 0,
          "commands": [...]
        },
        "api": {
          "turns": 12,
          "totalDuration": 45000,
          "errors": 0
        }
      }
    }
  ],
  "summary": {
    "totalAgents": 3,
    "runningAgents": 1,
    "totalFileReads": 75,
    "totalFileWrites": 15,
    "totalBytesRead": 1572864,
    "totalBytesWritten": 30720,
    "totalToolInvocations": 153,
    "totalBashCommands": 12,
    "totalAPITurns": 36,
    "peakMemoryUsage": 58720256,
    "totalDuration": 180000
  }
}

---

GET /api/debug/agents/summary

Returns aggregate resource usage statistics across all agent processes.

Response

{
  "totalAgents": 3,
  "runningAgents": 1,
  "totalFileReads": 75,
  "totalFileWrites": 15,
  "totalBytesRead": 1572864,
  "totalBytesWritten": 30720,
  "totalToolInvocations": 153,
  "totalBashCommands": 12,
  "totalAPITurns": 36,
  "peakMemoryUsage": 58720256,
  "totalDuration": 180000
}

---

GET /api/debug/agents/:id/metrics

Returns detailed resource metrics for a specific agent.

Path Parameters

Parameter	Type	Description
id	string	Agent process ID (e.g., agent-feature-123 or chat-session-456)

Response (success)

{
  "agentId": "agent-feature-123",
  "featureId": "feature-123",
  "startedAt": 1704067200000,
  "lastUpdatedAt": 1704067260000,
  "duration": 60000,
  "isRunning": true,
  "memory": {
    "startHeapUsed": 52428800,
    "currentHeapUsed": 57671680,
    "peakHeapUsed": 58720256,
    "deltaHeapUsed": 5242880,
    "samples": [
      { "timestamp": 1704067200000, "heapUsed": 52428800 },
      { "timestamp": 1704067201000, "heapUsed": 53477376 }
    ]
  },
  "fileIO": {
    "reads": 25,
    "bytesRead": 524288,
    "writes": 5,
    "bytesWritten": 10240,
    "edits": 3,
    "globs": 10,
    "greps": 8,
    "filesAccessed": ["src/index.ts", "src/utils.ts", "package.json"]
  },
  "tools": {
    "totalInvocations": 51,
    "byTool": {
      "Read": 25,
      "Glob": 10,
      "Grep": 8,
      "Write": 5,
      "Edit": 3
    },
    "avgExecutionTime": 150,
    "totalExecutionTime": 7650,
    "failedInvocations": 1
  },
  "bash": {
    "commandCount": 5,
    "totalExecutionTime": 2500,
    "failedCommands": 0,
    "commands": [
      {
        "command": "npm test",
        "exitCode": 0,
        "duration": 1500,
        "timestamp": 1704067230000
      }
    ]
  },
  "api": {
    "turns": 12,
    "inputTokens": 15000,
    "outputTokens": 8000,
    "thinkingTokens": 5000,
    "totalDuration": 45000,
    "errors": 0
  }
}

Response (not found)

{
  "error": "Agent metrics not found",
  "id": "non-existent-id"
}

---

Types

TrackedProcess

interface TrackedProcess {
  id: string; // Unique identifier
  pid?: number; // OS process ID
  type: ProcessType; // 'agent' | 'cli' | 'terminal' | 'worker'
  name: string; // Human-readable name
  status: ProcessStatus; // Current status
  startedAt: number; // Start timestamp (ms)
  stoppedAt?: number; // Stop timestamp (ms)
  memoryUsage?: number; // Memory in bytes
  cpuUsage?: number; // CPU percentage
  featureId?: string; // Associated feature
  sessionId?: string; // Associated session
  command?: string; // Command executed
  cwd?: string; // Working directory
  exitCode?: number; // Exit code (if stopped)
  error?: string; // Error message (if failed)
  resourceMetrics?: AgentResourceMetrics; // Detailed metrics for agents
}

AgentResourceMetrics

interface AgentResourceMetrics {
  agentId: string; // Agent/process ID
  sessionId?: string; // Session ID if available
  featureId?: string; // Feature ID if running a feature
  startedAt: number; // When metrics collection started
  lastUpdatedAt: number; // When metrics were last updated
  duration: number; // Duration of agent execution (ms)
  isRunning: boolean; // Whether the agent is still running
  memory: AgentMemoryMetrics;
  fileIO: FileIOMetrics;
  tools: ToolUsageMetrics;
  bash: BashMetrics;
  api: APIMetrics;
}

interface AgentMemoryMetrics {
  startHeapUsed: number; // Memory at agent start (bytes)
  currentHeapUsed: number; // Current memory (bytes)
  peakHeapUsed: number; // Peak memory during execution (bytes)
  deltaHeapUsed: number; // Memory change since start
  samples: Array<{ timestamp: number; heapUsed: number }>;
}

interface FileIOMetrics {
  reads: number; // Number of file reads
  bytesRead: number; // Total bytes read
  writes: number; // Number of file writes
  bytesWritten: number; // Total bytes written
  edits: number; // Number of file edits
  globs: number; // Number of glob operations
  greps: number; // Number of grep operations
  filesAccessed: string[]; // Unique files accessed (max 100)
}

interface ToolUsageMetrics {
  totalInvocations: number;
  byTool: Record<string, number>; // Invocations per tool name
  avgExecutionTime: number; // Average tool execution time (ms)
  totalExecutionTime: number; // Total tool execution time (ms)
  failedInvocations: number;
}

interface BashMetrics {
  commandCount: number;
  totalExecutionTime: number;
  failedCommands: number;
  commands: Array<{
    command: string;
    exitCode: number | null;
    duration: number;
    timestamp: number;
  }>;
}

interface APIMetrics {
  turns: number; // Number of API turns/iterations
  inputTokens?: number; // Input tokens used
  outputTokens?: number; // Output tokens generated
  thinkingTokens?: number; // Thinking tokens used
  totalDuration: number; // Total API call duration (ms)
  errors: number; // Number of API errors
}

ProcessStatus

• starting - Process is starting up
• running - Process is actively running
• idle - Process is idle/waiting
• stopping - Process is shutting down
• stopped - Process has stopped normally
• error - Process encountered an error

MemoryTrend

interface MemoryTrend {
  growthRate: number; // Bytes per second
  isLeaking: boolean; // Leak detected flag
  confidence: number; // R² value (0-1)
  sampleCount: number; // Data points analyzed
  windowDuration: number; // Analysis window (ms)
}

---

Events

The debug system emits the following WebSocket events:

Event	Description
debug:metrics	Periodic metrics snapshot (at collectionInterval)
debug:memory-warning	Memory usage exceeds 70% of heap limit
debug:memory-critical	Memory usage exceeds 90% of heap limit
debug:leak-detected	Memory leak pattern detected
debug:process-spawned	New process registered
debug:process-updated	Process status changed
debug:process-stopped	Process stopped normally
debug:process-error	Process encountered an error
debug:high-cpu	CPU usage exceeds 80%
debug:event-loop-blocked	Event loop lag exceeds 100ms

---

Usage Example

Starting metrics collection with custom config

// Start with 500ms interval and 120 data points
await fetch('/api/debug/metrics/start', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({
    config: {
      collectionInterval: 500,
      maxDataPoints: 120,
    },
  }),
});

// Poll for metrics
const response = await fetch('/api/debug/metrics');
const { snapshot } = await response.json();

console.log(`Heap used: ${(snapshot.memory.server.heapUsed / 1024 / 1024).toFixed(1)} MB`);
console.log(`CPU: ${snapshot.cpu.server.percentage.toFixed(1)}%`);

Monitoring for memory leaks

const response = await fetch('/api/debug/metrics');
const { snapshot } = await response.json();

if (snapshot.memoryTrend?.isLeaking) {
  console.warn(`Memory leak detected!`);
  console.warn(`Growth rate: ${snapshot.memoryTrend.growthRate} bytes/s`);
  console.warn(`Confidence: ${(snapshot.memoryTrend.confidence * 100).toFixed(0)}%`);
}