595 lines
30 KiB
Plaintext
595 lines
30 KiB
Plaintext
# Task ID: 77
|
|
# Title: Implement AI Usage Telemetry for Taskmaster (with external analytics endpoint)
|
|
# Status: done
|
|
# Dependencies: None
|
|
# Priority: medium
|
|
# Description: Capture detailed AI usage data (tokens, costs, models, commands) within Taskmaster and send this telemetry to an external, closed-source analytics backend for usage analysis, profitability measurement, and pricing optimization.
|
|
# Details:
|
|
* Add a telemetry utility (`logAiUsage`) within `ai-services.js` to track AI usage.
|
|
* Collected telemetry data fields must include:
|
|
* `timestamp`: Current date/time in ISO 8601.
|
|
* `userId`: Unique user identifier generated at setup (stored in `.taskmasterconfig`).
|
|
* `commandName`: Taskmaster command invoked (`expand`, `parse-prd`, `research`, etc.).
|
|
* `modelUsed`: Name/ID of the AI model invoked.
|
|
* `inputTokens`: Count of input tokens used.
|
|
* `outputTokens`: Count of output tokens generated.
|
|
* `totalTokens`: Sum of input and output tokens.
|
|
* `totalCost`: Monetary cost calculated using pricing from `supported_models.json`.
|
|
* Send telemetry payload securely via HTTPS POST request from user's Taskmaster installation directly to the closed-source analytics API (Express/Supabase backend).
|
|
* Introduce a privacy notice and explicit user consent prompt upon initial installation/setup to enable telemetry.
|
|
* Provide a graceful fallback if telemetry request fails (e.g., no internet connectivity).
|
|
* Optionally display a usage summary directly in Taskmaster CLI output for user transparency.
|
|
|
|
# Test Strategy:
|
|
|
|
|
|
# Subtasks:
|
|
## 1. Implement telemetry utility and data collection [done]
|
|
### Dependencies: None
|
|
### Description: Create the logAiUsage utility in ai-services.js that captures all required telemetry data fields
|
|
### Details:
|
|
Develop the logAiUsage function that collects timestamp, userId, commandName, modelUsed, inputTokens, outputTokens, totalTokens, and totalCost. Implement token counting logic and cost calculation using pricing from supported_models.json. Ensure proper error handling and data validation.
|
|
<info added on 2025-05-05T21:08:51.413Z>
|
|
Develop the logAiUsage function that collects timestamp, userId, commandName, modelUsed, inputTokens, outputTokens, totalTokens, and totalCost. Implement token counting logic and cost calculation using pricing from supported_models.json. Ensure proper error handling and data validation.
|
|
|
|
Implementation Plan:
|
|
1. Define `logAiUsage` function in `ai-services-unified.js` that accepts parameters: userId, commandName, providerName, modelId, inputTokens, and outputTokens.
|
|
|
|
2. Implement data collection and calculation logic:
|
|
- Generate timestamp using `new Date().toISOString()`
|
|
- Calculate totalTokens by adding inputTokens and outputTokens
|
|
- Create a helper function `_getCostForModel(providerName, modelId)` that:
|
|
- Loads pricing data from supported-models.json
|
|
- Finds the appropriate provider/model entry
|
|
- Returns inputCost and outputCost rates or defaults if not found
|
|
- Calculate totalCost using the formula: ((inputTokens/1,000,000) * inputCost) + ((outputTokens/1,000,000) * outputCost)
|
|
- Assemble complete telemetryData object with all required fields
|
|
|
|
3. Add initial logging functionality:
|
|
- Use existing log utility to record telemetry data at 'info' level
|
|
- Implement proper error handling with try/catch blocks
|
|
|
|
4. Integrate with `_unifiedServiceRunner`:
|
|
- Modify to accept commandName and userId parameters
|
|
- After successful API calls, extract usage data from results
|
|
- Call logAiUsage with the appropriate parameters
|
|
|
|
5. Update provider functions in src/ai-providers/*.js:
|
|
- Ensure all provider functions return both the primary result and usage statistics
|
|
- Standardize the return format to include a usage object with inputTokens and outputTokens
|
|
</info added on 2025-05-05T21:08:51.413Z>
|
|
<info added on 2025-05-07T17:28:57.361Z>
|
|
To implement the AI usage telemetry effectively, we need to update each command across our different stacks. Let's create a structured approach for this implementation:
|
|
|
|
Command Integration Plan:
|
|
1. Core Function Commands:
|
|
- Identify all AI-utilizing commands in the core function library
|
|
- For each command, modify to pass commandName and userId to _unifiedServiceRunner
|
|
- Update return handling to process and forward usage statistics
|
|
|
|
2. Direct Function Commands:
|
|
- Map all direct function commands that leverage AI capabilities
|
|
- Implement telemetry collection at the appropriate execution points
|
|
- Ensure consistent error handling and telemetry reporting
|
|
|
|
3. MCP Tool Stack Commands:
|
|
- Inventory all MCP commands with AI dependencies
|
|
- Standardize the telemetry collection approach across the tool stack
|
|
- Add telemetry hooks that maintain backward compatibility
|
|
|
|
For each command category, we'll need to:
|
|
- Document current implementation details
|
|
- Define specific code changes required
|
|
- Create tests to verify telemetry is being properly collected
|
|
- Establish validation procedures to ensure data accuracy
|
|
</info added on 2025-05-07T17:28:57.361Z>
|
|
|
|
## 2. Implement secure telemetry transmission [deferred]
|
|
### Dependencies: 77.1
|
|
### Description: Create a secure mechanism to transmit telemetry data to the external analytics endpoint
|
|
### Details:
|
|
Implement HTTPS POST request functionality to securely send the telemetry payload to the closed-source analytics API. Include proper encryption in transit using TLS. Implement retry logic and graceful fallback mechanisms for handling transmission failures due to connectivity issues.
|
|
<info added on 2025-05-14T17:52:40.647Z>
|
|
To securely send structured JSON telemetry payloads from a Node.js CLI tool to an external analytics backend, follow these steps:
|
|
|
|
1. Use the Axios library for HTTPS POST requests. Install it with: npm install axios.
|
|
2. Store sensitive configuration such as the analytics endpoint URL and any secret keys in environment variables (e.g., process.env.ANALYTICS_URL, process.env.ANALYTICS_KEY). Use dotenv or a similar library to load these securely.
|
|
3. Construct the telemetry payload as a JSON object with the required fields: userId, commandName, modelUsed, inputTokens, outputTokens, totalTokens, totalCost, and timestamp (ISO 8601).
|
|
4. Implement robust retry logic using the axios-retry package (npm install axios-retry). Configure exponential backoff with a recommended maximum of 3 retries and a base delay (e.g., 500ms).
|
|
5. Ensure all requests use HTTPS to guarantee TLS encryption in transit. Axios automatically uses HTTPS when the endpoint URL starts with https://.
|
|
6. Handle errors gracefully: catch all transmission errors, log them for diagnostics, and ensure failures do not interrupt or degrade the CLI user experience. Optionally, queue failed payloads for later retry if persistent connectivity issues occur.
|
|
7. Example code snippet:
|
|
|
|
require('dotenv').config();
|
|
const axios = require('axios');
|
|
const axiosRetry = require('axios-retry');
|
|
|
|
axiosRetry(axios, {
|
|
retries: 3,
|
|
retryDelay: axiosRetry.exponentialDelay,
|
|
retryCondition: (error) => axiosRetry.isNetworkOrIdempotentRequestError(error),
|
|
});
|
|
|
|
async function sendTelemetry(payload) {
|
|
try {
|
|
await axios.post(process.env.ANALYTICS_URL, payload, {
|
|
headers: {
|
|
'Content-Type': 'application/json',
|
|
'Authorization': `Bearer ${process.env.ANALYTICS_KEY}`,
|
|
},
|
|
timeout: 5000,
|
|
});
|
|
} catch (error) {
|
|
// Log error, do not throw to avoid impacting CLI UX
|
|
console.error('Telemetry transmission failed:', error.message);
|
|
// Optionally, queue payload for later retry
|
|
}
|
|
}
|
|
|
|
const telemetryPayload = {
|
|
userId: 'user-123',
|
|
commandName: 'expand',
|
|
modelUsed: 'gpt-4',
|
|
inputTokens: 100,
|
|
outputTokens: 200,
|
|
totalTokens: 300,
|
|
totalCost: 0.0123,
|
|
timestamp: new Date().toISOString(),
|
|
};
|
|
|
|
sendTelemetry(telemetryPayload);
|
|
|
|
8. Best practices:
|
|
- Never hardcode secrets or endpoint URLs in source code.
|
|
- Use environment variables and restrict access permissions.
|
|
- Validate all payload fields before transmission.
|
|
- Ensure the CLI continues to function even if telemetry transmission fails.
|
|
|
|
References: [1][2][3][5]
|
|
</info added on 2025-05-14T17:52:40.647Z>
|
|
<info added on 2025-05-14T17:57:18.218Z>
|
|
User ID Retrieval and Generation:
|
|
|
|
The telemetry system must securely retrieve the user ID from the .taskmasterconfig globals, where it should have been generated during the initialization phase. Implementation should:
|
|
|
|
1. Check for an existing user ID in the .taskmasterconfig file before sending any telemetry data.
|
|
2. If no user ID exists (for users who run AI commands without prior initialization or during upgrades), automatically generate a new UUID v4 and persist it to the .taskmasterconfig file.
|
|
3. Implement a getOrCreateUserId() function that:
|
|
- Reads from the global configuration file
|
|
- Returns the existing ID if present
|
|
- Generates a cryptographically secure UUID v4 if not present
|
|
- Saves the newly generated ID to the configuration file
|
|
- Handles file access errors gracefully
|
|
|
|
4. Example implementation:
|
|
```javascript
|
|
const fs = require('fs');
|
|
const path = require('path');
|
|
const { v4: uuidv4 } = require('uuid');
|
|
|
|
function getOrCreateUserId() {
|
|
const configPath = path.join(os.homedir(), '.taskmasterconfig');
|
|
|
|
try {
|
|
// Try to read existing config
|
|
const config = JSON.parse(fs.readFileSync(configPath, 'utf8'));
|
|
|
|
if (config.userId) {
|
|
return config.userId;
|
|
}
|
|
|
|
// No user ID found, generate and save
|
|
config.userId = uuidv4();
|
|
fs.writeFileSync(configPath, JSON.stringify(config, null, 2));
|
|
return config.userId;
|
|
} catch (error) {
|
|
// Handle case where config doesn't exist or is invalid
|
|
const userId = uuidv4();
|
|
const newConfig = { userId };
|
|
|
|
try {
|
|
fs.writeFileSync(configPath, JSON.stringify(newConfig, null, 2));
|
|
} catch (writeError) {
|
|
console.error('Failed to save user ID to config:', writeError.message);
|
|
}
|
|
|
|
return userId;
|
|
}
|
|
}
|
|
```
|
|
|
|
5. Ensure this function is called before constructing any telemetry payload to guarantee a consistent user ID across all telemetry events.
|
|
</info added on 2025-05-14T17:57:18.218Z>
|
|
<info added on 2025-05-15T18:45:32.123Z>
|
|
**Invocation Point for Sending Telemetry:**
|
|
* The primary invocation for sending the telemetry payload should occur in `scripts/modules/ai-services-unified.js`.
|
|
* This should happen *after* the `telemetryData` object is fully constructed and *after* user consent (from subtask 77.3) has been confirmed.
|
|
|
|
**Dedicated Module for Transmission Logic:**
|
|
* The actual HTTPS POST request mechanism, including TLS encryption, retry logic, and graceful fallbacks, should be implemented in a new, separate module (e.g., `scripts/modules/telemetry-sender.js` or `scripts/utils/telemetry-client.js`).
|
|
* This module will be imported and utilized by `scripts/modules/ai-services-unified.js`.
|
|
|
|
**Key Considerations:**
|
|
* Robust error handling must be in place for the telemetry transmission process; failures should be logged locally and must not disrupt core application functionality.
|
|
* The entire telemetry sending process is contingent upon explicit user consent as outlined in subtask 77.3.
|
|
|
|
**Implementation Plan:**
|
|
1. Create a new module `scripts/utils/telemetry-client.js` with the following functions:
|
|
- `sendTelemetryData(telemetryPayload)`: Main function that handles the HTTPS POST request
|
|
- `isUserConsentGiven()`: Helper function to check if user has consented to telemetry
|
|
- `logTelemetryError(error)`: Helper function for consistent error logging
|
|
|
|
2. In `ai-services-unified.js`, after constructing the telemetryData object:
|
|
```javascript
|
|
const telemetryClient = require('../utils/telemetry-client');
|
|
|
|
// After telemetryData is constructed
|
|
if (telemetryClient.isUserConsentGiven()) {
|
|
// Non-blocking telemetry submission
|
|
telemetryClient.sendTelemetryData(telemetryData)
|
|
.catch(error => telemetryClient.logTelemetryError(error));
|
|
}
|
|
```
|
|
|
|
3. Ensure the telemetry-client module implements:
|
|
- Axios with retry logic for robust HTTP requests
|
|
- Proper TLS encryption via HTTPS
|
|
- Comprehensive error handling
|
|
- Configuration loading from environment variables
|
|
- Validation of payload data before transmission
|
|
</info added on 2025-05-15T18:45:32.123Z>
|
|
|
|
## 3. Develop user consent and privacy notice system [deferred]
|
|
### Dependencies: None
|
|
### Description: Create a privacy notice and explicit consent mechanism during Taskmaster setup
|
|
### Details:
|
|
Design and implement a clear privacy notice explaining what data is collected and how it's used. Create a user consent prompt during initial installation/setup that requires explicit opt-in. Store the consent status in the .taskmasterconfig file and respect this setting throughout the application.
|
|
|
|
## 4. Integrate telemetry into Taskmaster commands [done]
|
|
### Dependencies: 77.1, 77.3
|
|
### Description: Integrate the telemetry utility across all relevant Taskmaster commands
|
|
### Details:
|
|
Modify each Taskmaster command (expand, parse-prd, research, etc.) to call the logAiUsage utility after AI interactions. Ensure telemetry is only sent if user has provided consent. Implement the integration in a way that doesn't impact command performance or user experience.
|
|
<info added on 2025-05-06T17:57:13.980Z>
|
|
Modify each Taskmaster command (expand, parse-prd, research, etc.) to call the logAiUsage utility after AI interactions. Ensure telemetry is only sent if user has provided consent. Implement the integration in a way that doesn't impact command performance or user experience.
|
|
|
|
Successfully integrated telemetry calls into `addTask` (core) and `addTaskDirect` (MCP) functions by passing `commandName` and `outputType` parameters to the telemetry system. The `ai-services-unified.js` module now logs basic telemetry data, including calculated cost information, whenever the `add-task` command or tool is invoked. This integration respects user consent settings and maintains performance standards.
|
|
</info added on 2025-05-06T17:57:13.980Z>
|
|
|
|
## 5. Implement usage summary display [done]
|
|
### Dependencies: 77.1, 77.4
|
|
### Description: Create an optional feature to display AI usage summary in the CLI output
|
|
### Details:
|
|
Develop functionality to display a concise summary of AI usage (tokens used, estimated cost) directly in the CLI output after command execution. Make this feature configurable through Taskmaster settings. Ensure the display is formatted clearly and doesn't clutter the main command output.
|
|
|
|
## 6. Telemetry Integration for parse-prd [done]
|
|
### Dependencies: None
|
|
### Description: Integrate AI usage telemetry capture and propagation for the parse-prd functionality.
|
|
### Details:
|
|
\
|
|
Apply telemetry pattern from telemetry.mdc:
|
|
|
|
1. **Core (`scripts/modules/task-manager/parse-prd.js`):**
|
|
* Modify AI service call to include `commandName: \'parse-prd\'` and `outputType`.
|
|
* Receive `{ mainResult, telemetryData }`.
|
|
* Return object including `telemetryData`.
|
|
* Handle CLI display via `displayAiUsageSummary` if applicable.
|
|
|
|
2. **Direct (`mcp-server/src/core/direct-functions/parse-prd.js`):**
|
|
* Pass `commandName`, `outputType: \'mcp\'` to core.
|
|
* Pass `outputFormat: \'json\'` if applicable.
|
|
* Receive `{ ..., telemetryData }` from core.
|
|
* Return `{ success: true, data: { ..., telemetryData } }`.
|
|
|
|
3. **Tool (`mcp-server/src/tools/parse-prd.js`):**
|
|
* Verify `handleApiResult` correctly passes `data.telemetryData` through.
|
|
|
|
|
|
## 7. Telemetry Integration for expand-task [done]
|
|
### Dependencies: None
|
|
### Description: Integrate AI usage telemetry capture and propagation for the expand-task functionality.
|
|
### Details:
|
|
\
|
|
Apply telemetry pattern from telemetry.mdc:
|
|
|
|
1. **Core (`scripts/modules/task-manager/expand-task.js`):**
|
|
* Modify AI service call to include `commandName: \'expand-task\'` and `outputType`.
|
|
* Receive `{ mainResult, telemetryData }`.
|
|
* Return object including `telemetryData`.
|
|
* Handle CLI display via `displayAiUsageSummary` if applicable.
|
|
|
|
2. **Direct (`mcp-server/src/core/direct-functions/expand-task.js`):**
|
|
* Pass `commandName`, `outputType: \'mcp\'` to core.
|
|
* Pass `outputFormat: \'json\'` if applicable.
|
|
* Receive `{ ..., telemetryData }` from core.
|
|
* Return `{ success: true, data: { ..., telemetryData } }`.
|
|
|
|
3. **Tool (`mcp-server/src/tools/expand-task.js`):**
|
|
* Verify `handleApiResult` correctly passes `data.telemetryData` through.
|
|
|
|
|
|
## 8. Telemetry Integration for expand-all-tasks [done]
|
|
### Dependencies: None
|
|
### Description: Integrate AI usage telemetry capture and propagation for the expand-all-tasks functionality.
|
|
### Details:
|
|
\
|
|
Apply telemetry pattern from telemetry.mdc:
|
|
|
|
1. **Core (`scripts/modules/task-manager/expand-all-tasks.js`):**
|
|
* Modify AI service call (likely within a loop or called by a helper) to include `commandName: \'expand-all-tasks\'` and `outputType`.
|
|
* Receive `{ mainResult, telemetryData }`.
|
|
* Aggregate or handle `telemetryData` appropriately if multiple AI calls are made.
|
|
* Return object including aggregated/relevant `telemetryData`.
|
|
* Handle CLI display via `displayAiUsageSummary` if applicable.
|
|
|
|
2. **Direct (`mcp-server/src/core/direct-functions/expand-all-tasks.js`):**
|
|
* Pass `commandName`, `outputType: \'mcp\'` to core.
|
|
* Pass `outputFormat: \'json\'` if applicable.
|
|
* Receive `{ ..., telemetryData }` from core.
|
|
* Return `{ success: true, data: { ..., telemetryData } }`.
|
|
|
|
3. **Tool (`mcp-server/src/tools/expand-all.js`):**
|
|
* Verify `handleApiResult` correctly passes `data.telemetryData` through.
|
|
|
|
|
|
## 9. Telemetry Integration for update-tasks [done]
|
|
### Dependencies: None
|
|
### Description: Integrate AI usage telemetry capture and propagation for the update-tasks (bulk update) functionality.
|
|
### Details:
|
|
\
|
|
Apply telemetry pattern from telemetry.mdc:
|
|
|
|
1. **Core (`scripts/modules/task-manager/update-tasks.js`):**
|
|
* Modify AI service call (likely within a loop) to include `commandName: \'update-tasks\'` and `outputType`.
|
|
* Receive `{ mainResult, telemetryData }` for each AI call.
|
|
* Aggregate or handle `telemetryData` appropriately for multiple calls.
|
|
* Return object including aggregated/relevant `telemetryData`.
|
|
* Handle CLI display via `displayAiUsageSummary` if applicable.
|
|
|
|
2. **Direct (`mcp-server/src/core/direct-functions/update-tasks.js`):**
|
|
* Pass `commandName`, `outputType: \'mcp\'` to core.
|
|
* Pass `outputFormat: \'json\'` if applicable.
|
|
* Receive `{ ..., telemetryData }` from core.
|
|
* Return `{ success: true, data: { ..., telemetryData } }`.
|
|
|
|
3. **Tool (`mcp-server/src/tools/update.js`):**
|
|
* Verify `handleApiResult` correctly passes `data.telemetryData` through.
|
|
|
|
|
|
## 10. Telemetry Integration for update-task-by-id [done]
|
|
### Dependencies: None
|
|
### Description: Integrate AI usage telemetry capture and propagation for the update-task-by-id functionality.
|
|
### Details:
|
|
\
|
|
Apply telemetry pattern from telemetry.mdc:
|
|
|
|
1. **Core (`scripts/modules/task-manager/update-task-by-id.js`):**
|
|
* Modify AI service call to include `commandName: \'update-task\'` and `outputType`.
|
|
* Receive `{ mainResult, telemetryData }`.
|
|
* Return object including `telemetryData`.
|
|
* Handle CLI display via `displayAiUsageSummary` if applicable.
|
|
|
|
2. **Direct (`mcp-server/src/core/direct-functions/update-task-by-id.js`):**
|
|
* Pass `commandName`, `outputType: \'mcp\'` to core.
|
|
* Pass `outputFormat: \'json\'` if applicable.
|
|
* Receive `{ ..., telemetryData }` from core.
|
|
* Return `{ success: true, data: { ..., telemetryData } }`.
|
|
|
|
3. **Tool (`mcp-server/src/tools/update-task.js`):**
|
|
* Verify `handleApiResult` correctly passes `data.telemetryData` through.
|
|
|
|
|
|
## 11. Telemetry Integration for update-subtask-by-id [done]
|
|
### Dependencies: None
|
|
### Description: Integrate AI usage telemetry capture and propagation for the update-subtask-by-id functionality.
|
|
### Details:
|
|
\
|
|
Apply telemetry pattern from telemetry.mdc:
|
|
|
|
1. **Core (`scripts/modules/task-manager/update-subtask-by-id.js`):**
|
|
* Verify if this function *actually* calls an AI service. If it only appends text, telemetry integration might not apply directly here, but ensure its callers handle telemetry if they use AI.
|
|
* *If it calls AI:* Modify AI service call to include `commandName: \'update-subtask\'` and `outputType`.
|
|
* *If it calls AI:* Receive `{ mainResult, telemetryData }`.
|
|
* *If it calls AI:* Return object including `telemetryData`.
|
|
* *If it calls AI:* Handle CLI display via `displayAiUsageSummary` if applicable.
|
|
|
|
2. **Direct (`mcp-server/src/core/direct-functions/update-subtask-by-id.js`):**
|
|
* *If core calls AI:* Pass `commandName`, `outputType: \'mcp\'` to core.
|
|
* *If core calls AI:* Pass `outputFormat: \'json\'` if applicable.
|
|
* *If core calls AI:* Receive `{ ..., telemetryData }` from core.
|
|
* *If core calls AI:* Return `{ success: true, data: { ..., telemetryData } }`.
|
|
|
|
3. **Tool (`mcp-server/src/tools/update-subtask.js`):**
|
|
* Verify `handleApiResult` correctly passes `data.telemetryData` through (if present).
|
|
|
|
|
|
## 12. Telemetry Integration for analyze-task-complexity [done]
|
|
### Dependencies: None
|
|
### Description: Integrate AI usage telemetry capture and propagation for the analyze-task-complexity functionality. [Updated: 5/9/2025]
|
|
### Details:
|
|
\
|
|
Apply telemetry pattern from telemetry.mdc:
|
|
|
|
1. **Core (`scripts/modules/task-manager/analyze-task-complexity.js`):**
|
|
* Modify AI service call to include `commandName: \'analyze-complexity\'` and `outputType`.
|
|
* Receive `{ mainResult, telemetryData }`.
|
|
* Return object including `telemetryData` (perhaps alongside the complexity report data).
|
|
* Handle CLI display via `displayAiUsageSummary` if applicable.
|
|
|
|
2. **Direct (`mcp-server/src/core/direct-functions/analyze-task-complexity.js`):**
|
|
* Pass `commandName`, `outputType: \'mcp\'` to core.
|
|
* Pass `outputFormat: \'json\'` if applicable.
|
|
* Receive `{ ..., telemetryData }` from core.
|
|
* Return `{ success: true, data: { ..., telemetryData } }`.
|
|
|
|
3. **Tool (`mcp-server/src/tools/analyze.js`):**
|
|
* Verify `handleApiResult` correctly passes `data.telemetryData` through.
|
|
|
|
<info added on 2025-05-09T04:02:44.847Z>
|
|
## Implementation Details for Telemetry Integration
|
|
|
|
### Best Practices for Implementation
|
|
|
|
1. **Use Structured Telemetry Objects:**
|
|
- Create a standardized `TelemetryEvent` object with fields:
|
|
```javascript
|
|
{
|
|
commandName: string, // e.g., 'analyze-complexity'
|
|
timestamp: ISO8601 string,
|
|
duration: number, // in milliseconds
|
|
inputTokens: number,
|
|
outputTokens: number,
|
|
model: string, // e.g., 'gpt-4'
|
|
success: boolean,
|
|
errorType?: string, // if applicable
|
|
metadata: object // command-specific context
|
|
}
|
|
```
|
|
|
|
2. **Asynchronous Telemetry Processing:**
|
|
- Use non-blocking telemetry submission to avoid impacting performance
|
|
- Implement queue-based processing for reliability during network issues
|
|
|
|
3. **Error Handling:**
|
|
- Implement robust try/catch blocks around telemetry operations
|
|
- Ensure telemetry failures don't affect core functionality
|
|
- Log telemetry failures locally for debugging
|
|
|
|
4. **Privacy Considerations:**
|
|
- Never include PII or sensitive data in telemetry
|
|
- Implement data minimization principles
|
|
- Add sanitization functions for metadata fields
|
|
|
|
5. **Testing Strategy:**
|
|
- Create mock telemetry endpoints for testing
|
|
- Add unit tests verifying correct telemetry data structure
|
|
- Implement integration tests for end-to-end telemetry flow
|
|
|
|
### Code Implementation Examples
|
|
|
|
```javascript
|
|
// Example telemetry submission function
|
|
async function submitTelemetry(telemetryData, endpoint) {
|
|
try {
|
|
// Non-blocking submission
|
|
fetch(endpoint, {
|
|
method: 'POST',
|
|
headers: { 'Content-Type': 'application/json' },
|
|
body: JSON.stringify(telemetryData)
|
|
}).catch(err => console.error('Telemetry submission failed:', err));
|
|
} catch (error) {
|
|
// Log locally but don't disrupt main flow
|
|
console.error('Telemetry error:', error);
|
|
}
|
|
}
|
|
|
|
// Example integration in AI service call
|
|
async function callAiService(params) {
|
|
const startTime = Date.now();
|
|
try {
|
|
const result = await aiService.call({
|
|
...params,
|
|
commandName: 'analyze-complexity',
|
|
outputType: 'mcp'
|
|
});
|
|
|
|
// Construct telemetry object
|
|
const telemetryData = {
|
|
commandName: 'analyze-complexity',
|
|
timestamp: new Date().toISOString(),
|
|
duration: Date.now() - startTime,
|
|
inputTokens: result.usage?.prompt_tokens || 0,
|
|
outputTokens: result.usage?.completion_tokens || 0,
|
|
model: result.model || 'unknown',
|
|
success: true,
|
|
metadata: {
|
|
taskId: params.taskId,
|
|
// Add other relevant non-sensitive metadata
|
|
}
|
|
};
|
|
|
|
return { mainResult: result.data, telemetryData };
|
|
} catch (error) {
|
|
// Error telemetry
|
|
const telemetryData = {
|
|
commandName: 'analyze-complexity',
|
|
timestamp: new Date().toISOString(),
|
|
duration: Date.now() - startTime,
|
|
success: false,
|
|
errorType: error.name,
|
|
metadata: {
|
|
taskId: params.taskId,
|
|
errorMessage: sanitizeErrorMessage(error.message)
|
|
}
|
|
};
|
|
|
|
// Re-throw the original error after capturing telemetry
|
|
throw error;
|
|
}
|
|
}
|
|
```
|
|
</info added on 2025-05-09T04:02:44.847Z>
|
|
|
|
## 13. Update google.js for Telemetry Compatibility [done]
|
|
### Dependencies: None
|
|
### Description: Modify src/ai-providers/google.js functions to return usage data.
|
|
### Details:
|
|
Update the provider functions in `src/ai-providers/google.js` to ensure they return telemetry-compatible results:\n\n1. **`generateGoogleText`**: Return `{ text: ..., usage: { inputTokens: ..., outputTokens: ... } }`. Extract token counts from the Vercel AI SDK result.\n2. **`generateGoogleObject`**: Return `{ object: ..., usage: { inputTokens: ..., outputTokens: ... } }`. Extract token counts.\n3. **`streamGoogleText`**: Return the *full stream result object* returned by the Vercel AI SDK's `streamText`, not just the `textStream` property. The full object contains usage information.\n\nReference `anthropic.js` for the pattern.
|
|
|
|
## 14. Update openai.js for Telemetry Compatibility [done]
|
|
### Dependencies: None
|
|
### Description: Modify src/ai-providers/openai.js functions to return usage data.
|
|
### Details:
|
|
Update the provider functions in `src/ai-providers/openai.js` to ensure they return telemetry-compatible results:\n\n1. **`generateOpenAIText`**: Return `{ text: ..., usage: { inputTokens: ..., outputTokens: ... } }`. Extract token counts from the Vercel AI SDK result.\n2. **`generateOpenAIObject`**: Return `{ object: ..., usage: { inputTokens: ..., outputTokens: ... } }`. Extract token counts.\n3. **`streamOpenAIText`**: Return the *full stream result object* returned by the Vercel AI SDK's `streamText`, not just the `textStream` property. The full object contains usage information.\n\nReference `anthropic.js` for the pattern.
|
|
|
|
## 15. Update openrouter.js for Telemetry Compatibility [done]
|
|
### Dependencies: None
|
|
### Description: Modify src/ai-providers/openrouter.js functions to return usage data.
|
|
### Details:
|
|
Update the provider functions in `src/ai-providers/openrouter.js` to ensure they return telemetry-compatible results:\n\n1. **`generateOpenRouterText`**: Return `{ text: ..., usage: { inputTokens: ..., outputTokens: ... } }`. Extract token counts from the Vercel AI SDK result.\n2. **`generateOpenRouterObject`**: Return `{ object: ..., usage: { inputTokens: ..., outputTokens: ... } }`. Extract token counts.\n3. **`streamOpenRouterText`**: Return the *full stream result object* returned by the Vercel AI SDK's `streamText`, not just the `textStream` property. The full object contains usage information.\n\nReference `anthropic.js` for the pattern.
|
|
|
|
## 16. Update perplexity.js for Telemetry Compatibility [done]
|
|
### Dependencies: None
|
|
### Description: Modify src/ai-providers/perplexity.js functions to return usage data.
|
|
### Details:
|
|
Update the provider functions in `src/ai-providers/perplexity.js` to ensure they return telemetry-compatible results:\n\n1. **`generatePerplexityText`**: Return `{ text: ..., usage: { inputTokens: ..., outputTokens: ... } }`. Extract token counts from the Vercel AI SDK result.\n2. **`generatePerplexityObject`**: Return `{ object: ..., usage: { inputTokens: ..., outputTokens: ... } }`. Extract token counts.\n3. **`streamPerplexityText`**: Return the *full stream result object* returned by the Vercel AI SDK's `streamText`, not just the `textStream` property. The full object contains usage information.\n\nReference `anthropic.js` for the pattern.
|
|
|
|
## 17. Update xai.js for Telemetry Compatibility [done]
|
|
### Dependencies: None
|
|
### Description: Modify src/ai-providers/xai.js functions to return usage data.
|
|
### Details:
|
|
Update the provider functions in `src/ai-providers/xai.js` to ensure they return telemetry-compatible results:\n\n1. **`generateXaiText`**: Return `{ text: ..., usage: { inputTokens: ..., outputTokens: ... } }`. Extract token counts from the Vercel AI SDK result.\n2. **`generateXaiObject`**: Return `{ object: ..., usage: { inputTokens: ..., outputTokens: ... } }`. Extract token counts.\n3. **`streamXaiText`**: Return the *full stream result object* returned by the Vercel AI SDK's `streamText`, not just the `textStream` property. The full object contains usage information.\n\nReference `anthropic.js` for the pattern.
|
|
|
|
## 18. Create dedicated telemetry transmission module [done]
|
|
### Dependencies: 77.1, 77.3
|
|
### Description: Implement a separate module for handling telemetry transmission logic
|
|
### Details:
|
|
Create a new module (e.g., `scripts/utils/telemetry-client.js`) that encapsulates all telemetry transmission functionality:
|
|
|
|
1. Implement core functions:
|
|
- `sendTelemetryData(telemetryPayload)`: Main function to handle HTTPS POST requests
|
|
- `isUserConsentGiven()`: Helper to check if user has consented to telemetry
|
|
- `logTelemetryError(error)`: Helper for consistent error logging
|
|
|
|
2. Use Axios with retry logic:
|
|
- Configure with exponential backoff (max 3 retries, 500ms base delay)
|
|
- Implement proper TLS encryption via HTTPS
|
|
- Set appropriate timeouts (5000ms recommended)
|
|
|
|
3. Implement robust error handling:
|
|
- Catch all transmission errors
|
|
- Log failures locally without disrupting application flow
|
|
- Ensure failures are transparent to users
|
|
|
|
4. Configure securely:
|
|
- Load endpoint URL and authentication from environment variables
|
|
- Never hardcode secrets in source code
|
|
- Validate payload data before transmission
|
|
|
|
5. Integration with ai-services-unified.js:
|
|
- Import the telemetry-client module
|
|
- Call after telemetryData object is constructed
|
|
- Only send if user consent is confirmed
|
|
- Use non-blocking approach to avoid performance impact
|
|
|