fix merge conflicts to prep for merge with branch next
- Enhance E2E testing and LLM analysis report and: - Add --analyze-log flag to run_e2e.sh to re-run LLM analysis on existing logs. - Add test:e2e and analyze-log scripts to package.json for easier execution. - Correct display errors and dependency validation output: - Update chalk usage in add-task.js to use bracket notation (chalk[color]) compatible with v5, resolving 'chalk.keyword is not a function' error. - Modify fix-dependencies command output to show red failure box with issue count instead of green success box when validation fails. - Refactor interactive model setup: - Verify inclusion of 'No change' option during interactive model setup flow (task-master models --setup). - Update model definitions: - Add max_tokens field for gpt-4o in supported-models.json. - Remove unused scripts: - Delete prepare-package.js and rule-transformer.test.js. Release candidate
This commit is contained in:
162
tests/e2e/e2e_helpers.sh
Normal file
162
tests/e2e/e2e_helpers.sh
Normal file
@@ -0,0 +1,162 @@
|
||||
#!/bin/bash
|
||||
|
||||
# --- LLM Analysis Helper Function ---
|
||||
# This function should be sourced by the main E2E script or test scripts.
|
||||
# It requires curl and jq to be installed.
|
||||
# It expects the project root path to be passed as the second argument.
|
||||
|
||||
analyze_log_with_llm() {
|
||||
local log_file="$1"
|
||||
local project_root="$2" # Expect project root as the second argument
|
||||
|
||||
if [ -z "$project_root" ]; then
|
||||
echo "[HELPER_ERROR] Project root argument is missing. Skipping LLM analysis." >&2
|
||||
return 1
|
||||
fi
|
||||
|
||||
local env_file="${project_root}/.env" # Path to .env in project root
|
||||
|
||||
local provider_summary_log="provider_add_task_summary.log" # File summarizing provider test outcomes
|
||||
local api_key=""
|
||||
# !!! IMPORTANT: Replace with your actual Claude API endpoint if different !!!
|
||||
local api_endpoint="https://api.anthropic.com/v1/messages"
|
||||
# !!! IMPORTANT: Ensure this matches the variable name in your .env file !!!
|
||||
local api_key_name="ANTHROPIC_API_KEY"
|
||||
|
||||
echo "" # Add a newline before analysis starts
|
||||
|
||||
# Check for jq and curl
|
||||
if ! command -v jq &> /dev/null; then
|
||||
echo "[HELPER_ERROR] LLM Analysis requires 'jq'. Skipping analysis." >&2
|
||||
return 1
|
||||
fi
|
||||
if ! command -v curl &> /dev/null; then
|
||||
echo "[HELPER_ERROR] LLM Analysis requires 'curl'. Skipping analysis." >&2
|
||||
return 1
|
||||
fi
|
||||
|
||||
# Check for API Key in the PROJECT ROOT's .env file
|
||||
if [ -f "$env_file" ]; then
|
||||
# Original assignment - Reading from project root .env
|
||||
api_key=$(grep "^${api_key_name}=" "$env_file" | sed -e "s/^${api_key_name}=//" -e 's/^[[:space:]"]*//' -e 's/[[:space:]"]*$//')
|
||||
fi
|
||||
|
||||
if [ -z "$api_key" ]; then
|
||||
echo "[HELPER_ERROR] ${api_key_name} not found or empty in project root .env file ($env_file). Skipping LLM analysis." >&2 # Updated error message
|
||||
return 1
|
||||
fi
|
||||
|
||||
# Log file path is passed as argument, need to ensure it exists relative to where the script *calling* this function is, OR use absolute path.
|
||||
# Assuming absolute path or path relative to the initial PWD for simplicity here.
|
||||
# The calling script passes the correct path relative to the original PWD.
|
||||
if [ ! -f "$log_file" ]; then
|
||||
echo "[HELPER_ERROR] Log file not found: $log_file (PWD: $(pwd)). Check path passed to function. Skipping LLM analysis." >&2 # Updated error
|
||||
return 1
|
||||
fi
|
||||
|
||||
local log_content
|
||||
# Read entire file, handle potential errors
|
||||
log_content=$(cat "$log_file") || {
|
||||
echo "[HELPER_ERROR] Failed to read log file: $log_file. Skipping LLM analysis." >&2
|
||||
return 1
|
||||
}
|
||||
|
||||
# Prepare the prompt using a quoted heredoc for literal interpretation
|
||||
read -r -d '' prompt_template <<'EOF'
|
||||
Analyze the following E2E test log for the task-master tool. The log contains output from various 'task-master' commands executed sequentially.
|
||||
|
||||
Your goal is to:
|
||||
1. Verify if the key E2E steps completed successfully based on the log messages (e.g., init, parse PRD, list tasks, analyze complexity, expand task, set status, manage models, add/remove dependencies, add/update/remove tasks/subtasks, generate files).
|
||||
2. **Specifically analyze the Multi-Provider Add-Task Test Sequence:**
|
||||
a. Identify which providers were tested for `add-task`. Look for log steps like "Testing Add-Task with Provider: ..." and the summary log 'provider_add_task_summary.log'.
|
||||
b. For each tested provider, determine if `add-task` succeeded or failed. Note the created task ID if successful.
|
||||
c. Review the corresponding `add_task_show_output_<provider>_id_<id>.log` file (if created) for each successful `add-task` execution.
|
||||
d. **Compare the quality and completeness** of the task generated by each successful provider based on their `show` output. Assign a score (e.g., 1-10, 10 being best) based on relevance to the prompt, detail level, and correctness.
|
||||
e. Note any providers where `add-task` failed or where the task ID could not be extracted.
|
||||
3. Identify any general explicit "[ERROR]" messages or stack traces throughout the *entire* log.
|
||||
4. Identify any potential warnings or unusual output that might indicate a problem even if not marked as an explicit error.
|
||||
5. Provide an overall assessment of the test run's health based *only* on the log content.
|
||||
|
||||
Return your analysis **strictly** in the following JSON format. Do not include any text outside of the JSON structure:
|
||||
|
||||
{
|
||||
"overall_status": "Success|Failure|Warning",
|
||||
"verified_steps": [ "Initialization", "PRD Parsing", /* ...other general steps observed... */ ],
|
||||
"provider_add_task_comparison": {
|
||||
"prompt_used": "... (extract from log if possible or state 'standard auth prompt') ...",
|
||||
"provider_results": {
|
||||
"anthropic": { "status": "Success|Failure|ID_Extraction_Failed|Set_Model_Failed", "task_id": "...", "score": "X/10 | N/A", "notes": "..." },
|
||||
"openai": { "status": "Success|Failure|...", "task_id": "...", "score": "X/10 | N/A", "notes": "..." },
|
||||
/* ... include all tested providers ... */
|
||||
},
|
||||
"comparison_summary": "Brief overall comparison of generated tasks..."
|
||||
},
|
||||
"detected_issues": [ { "severity": "Error|Warning|Anomaly", "description": "...", "log_context": "[Optional, short snippet from log near the issue]" } ],
|
||||
"llm_summary_points": [ "Overall summary point 1", "Provider comparison highlight", "Any major issues noted" ]
|
||||
}
|
||||
|
||||
Here is the main log content:
|
||||
|
||||
%s
|
||||
EOF
|
||||
# Note: The final %s is a placeholder for printf later
|
||||
|
||||
local full_prompt
|
||||
# Use printf to substitute the log content into the %s placeholder
|
||||
if ! printf -v full_prompt "$prompt_template" "$log_content"; then
|
||||
echo "[HELPER_ERROR] Failed to format prompt using printf." >&2
|
||||
# It's unlikely printf itself fails, but good practice
|
||||
return 1
|
||||
fi
|
||||
|
||||
# Construct the JSON payload for Claude Messages API
|
||||
local payload
|
||||
payload=$(jq -n --arg prompt "$full_prompt" '{
|
||||
"model": "claude-3-haiku-20240307", # Using Haiku for faster/cheaper testing
|
||||
"max_tokens": 3072, # Increased slightly
|
||||
"messages": [
|
||||
{"role": "user", "content": $prompt}
|
||||
]
|
||||
# "temperature": 0.0 # Optional: Lower temperature for more deterministic JSON output
|
||||
}') || {
|
||||
echo "[HELPER_ERROR] Failed to create JSON payload using jq." >&2
|
||||
return 1
|
||||
}
|
||||
|
||||
local response_raw response_http_code response_body
|
||||
# Capture body and HTTP status code separately
|
||||
response_raw=$(curl -s -w "\nHTTP_STATUS_CODE:%{http_code}" -X POST "$api_endpoint" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "x-api-key: $api_key" \
|
||||
-H "anthropic-version: 2023-06-01" \
|
||||
--data "$payload")
|
||||
|
||||
# Extract status code and body
|
||||
response_http_code=$(echo "$response_raw" | grep '^HTTP_STATUS_CODE:' | sed 's/HTTP_STATUS_CODE://')
|
||||
response_body=$(echo "$response_raw" | sed '$d') # Remove last line (status code)
|
||||
|
||||
if [ "$response_http_code" != "200" ]; then
|
||||
echo "[HELPER_ERROR] LLM API call failed with HTTP status $response_http_code." >&2
|
||||
echo "[HELPER_ERROR] Response Body: $response_body" >&2
|
||||
return 1
|
||||
fi
|
||||
|
||||
if [ -z "$response_body" ]; then
|
||||
echo "[HELPER_ERROR] LLM API call returned empty response body." >&2
|
||||
return 1
|
||||
fi
|
||||
|
||||
# Pipe the raw response body directly to the Node.js parser script
|
||||
if echo "$response_body" | node "${project_root}/tests/e2e/parse_llm_output.cjs" "$log_file"; then
|
||||
echo "[HELPER_SUCCESS] LLM analysis parsed and printed successfully by Node.js script."
|
||||
return 0 # Success
|
||||
else
|
||||
local node_exit_code=$?
|
||||
echo "[HELPER_ERROR] Node.js parsing script failed with exit code ${node_exit_code}."
|
||||
echo "[HELPER_ERROR] Raw API response body (first 500 chars): $(echo "$response_body" | head -c 500)"
|
||||
return 1 # Failure
|
||||
fi
|
||||
}
|
||||
|
||||
# Export the function so it might be available to subshells if sourced
|
||||
export -f analyze_log_with_llm
|
||||
266
tests/e2e/parse_llm_output.cjs
Normal file
266
tests/e2e/parse_llm_output.cjs
Normal file
@@ -0,0 +1,266 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
// Note: We will use dynamic import() inside the async callback due to project being type: module
|
||||
|
||||
const readline = require('readline');
|
||||
const path = require('path'); // Import path module
|
||||
|
||||
let inputData = '';
|
||||
|
||||
const rl = readline.createInterface({
|
||||
input: process.stdin,
|
||||
output: process.stdout,
|
||||
terminal: false
|
||||
});
|
||||
|
||||
rl.on('line', (line) => {
|
||||
inputData += line;
|
||||
});
|
||||
|
||||
// Make the callback async to allow await for dynamic imports
|
||||
rl.on('close', async () => {
|
||||
let chalk, boxen, Table;
|
||||
try {
|
||||
// Dynamically import libraries
|
||||
chalk = (await import('chalk')).default;
|
||||
boxen = (await import('boxen')).default;
|
||||
Table = (await import('cli-table3')).default;
|
||||
|
||||
// 1. Parse the initial API response body
|
||||
const apiResponse = JSON.parse(inputData);
|
||||
|
||||
// 2. Extract the text content containing the nested JSON
|
||||
// Robust check for content structure
|
||||
const textContent = apiResponse?.content?.[0]?.text;
|
||||
if (!textContent) {
|
||||
console.error(
|
||||
chalk.red(
|
||||
"Error: Could not find '.content[0].text' in the API response JSON."
|
||||
)
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// 3. Find the start of the actual JSON block
|
||||
const jsonStart = textContent.indexOf('{');
|
||||
const jsonEnd = textContent.lastIndexOf('}');
|
||||
|
||||
if (jsonStart === -1 || jsonEnd === -1 || jsonEnd < jsonStart) {
|
||||
console.error(
|
||||
chalk.red(
|
||||
'Error: Could not find JSON block starting with { and ending with } in the extracted text content.'
|
||||
)
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
const jsonString = textContent.substring(jsonStart, jsonEnd + 1);
|
||||
|
||||
// 4. Parse the extracted JSON string
|
||||
let reportData;
|
||||
try {
|
||||
reportData = JSON.parse(jsonString);
|
||||
} catch (parseError) {
|
||||
console.error(
|
||||
chalk.red('Error: Failed to parse the extracted JSON block.')
|
||||
);
|
||||
console.error(chalk.red('Parse Error:'), parseError.message);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// Ensure reportData is an object
|
||||
if (typeof reportData !== 'object' || reportData === null) {
|
||||
console.error(
|
||||
chalk.red('Error: Parsed report data is not a valid object.')
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// --- Get Log File Path and Format Timestamp ---
|
||||
const logFilePath = process.argv[2]; // Get the log file path argument
|
||||
let formattedTime = 'Unknown';
|
||||
if (logFilePath) {
|
||||
const logBasename = path.basename(logFilePath);
|
||||
const timestampMatch = logBasename.match(/e2e_run_(\d{8}_\d{6})\.log$/);
|
||||
if (timestampMatch && timestampMatch[1]) {
|
||||
const ts = timestampMatch[1]; // YYYYMMDD_HHMMSS
|
||||
// Format into YYYY-MM-DD HH:MM:SS
|
||||
formattedTime = `${ts.substring(0, 4)}-${ts.substring(4, 6)}-${ts.substring(6, 8)} ${ts.substring(9, 11)}:${ts.substring(11, 13)}:${ts.substring(13, 15)}`;
|
||||
}
|
||||
}
|
||||
// --------------------------------------------
|
||||
|
||||
// 5. Generate CLI Report (with defensive checks)
|
||||
console.log(
|
||||
'\n' +
|
||||
chalk.cyan.bold(
|
||||
boxen(
|
||||
`TASKMASTER E2E Log Analysis Report\nRun Time: ${chalk.yellow(formattedTime)}`, // Display formatted time
|
||||
{
|
||||
padding: 1,
|
||||
borderStyle: 'double',
|
||||
borderColor: 'cyan',
|
||||
textAlign: 'center' // Center align title
|
||||
}
|
||||
)
|
||||
) +
|
||||
'\n'
|
||||
);
|
||||
|
||||
// Overall Status
|
||||
let statusColor = chalk.white;
|
||||
const overallStatus = reportData.overall_status || 'Unknown'; // Default if missing
|
||||
if (overallStatus === 'Success') statusColor = chalk.green.bold;
|
||||
if (overallStatus === 'Warning') statusColor = chalk.yellow.bold;
|
||||
if (overallStatus === 'Failure') statusColor = chalk.red.bold;
|
||||
console.log(
|
||||
boxen(`Overall Status: ${statusColor(overallStatus)}`, {
|
||||
padding: { left: 1, right: 1 },
|
||||
margin: { bottom: 1 },
|
||||
borderColor: 'blue'
|
||||
})
|
||||
);
|
||||
|
||||
// LLM Summary Points
|
||||
console.log(chalk.blue.bold('📋 Summary Points:'));
|
||||
if (
|
||||
Array.isArray(reportData.llm_summary_points) &&
|
||||
reportData.llm_summary_points.length > 0
|
||||
) {
|
||||
reportData.llm_summary_points.forEach((point) => {
|
||||
console.log(chalk.white(` - ${point || 'N/A'}`)); // Handle null/undefined points
|
||||
});
|
||||
} else {
|
||||
console.log(chalk.gray(' No summary points provided.'));
|
||||
}
|
||||
console.log();
|
||||
|
||||
// Verified Steps
|
||||
console.log(chalk.green.bold('✅ Verified Steps:'));
|
||||
if (
|
||||
Array.isArray(reportData.verified_steps) &&
|
||||
reportData.verified_steps.length > 0
|
||||
) {
|
||||
reportData.verified_steps.forEach((step) => {
|
||||
console.log(chalk.green(` - ${step || 'N/A'}`)); // Handle null/undefined steps
|
||||
});
|
||||
} else {
|
||||
console.log(chalk.gray(' No verified steps listed.'));
|
||||
}
|
||||
console.log();
|
||||
|
||||
// Provider Add-Task Comparison
|
||||
console.log(chalk.magenta.bold('🔄 Provider Add-Task Comparison:'));
|
||||
const comp = reportData.provider_add_task_comparison;
|
||||
if (typeof comp === 'object' && comp !== null) {
|
||||
console.log(
|
||||
chalk.white(` Prompt Used: ${comp.prompt_used || 'Not specified'}`)
|
||||
);
|
||||
console.log();
|
||||
|
||||
if (
|
||||
typeof comp.provider_results === 'object' &&
|
||||
comp.provider_results !== null &&
|
||||
Object.keys(comp.provider_results).length > 0
|
||||
) {
|
||||
const providerTable = new Table({
|
||||
head: ['Provider', 'Status', 'Task ID', 'Score', 'Notes'].map((h) =>
|
||||
chalk.magenta.bold(h)
|
||||
),
|
||||
colWidths: [15, 18, 10, 12, 45],
|
||||
style: { head: [], border: [] },
|
||||
wordWrap: true
|
||||
});
|
||||
|
||||
for (const provider in comp.provider_results) {
|
||||
const result = comp.provider_results[provider] || {}; // Default to empty object if provider result is null/undefined
|
||||
const status = result.status || 'Unknown';
|
||||
const isSuccess = status === 'Success';
|
||||
const statusIcon = isSuccess ? chalk.green('✅') : chalk.red('❌');
|
||||
const statusText = isSuccess
|
||||
? chalk.green(status)
|
||||
: chalk.red(status);
|
||||
providerTable.push([
|
||||
chalk.white(provider),
|
||||
`${statusIcon} ${statusText}`,
|
||||
chalk.white(result.task_id || 'N/A'),
|
||||
chalk.white(result.score || 'N/A'),
|
||||
chalk.dim(result.notes || 'N/A')
|
||||
]);
|
||||
}
|
||||
console.log(providerTable.toString());
|
||||
console.log();
|
||||
} else {
|
||||
console.log(chalk.gray(' No provider results available.'));
|
||||
console.log();
|
||||
}
|
||||
console.log(chalk.white.bold(` Comparison Summary:`));
|
||||
console.log(chalk.white(` ${comp.comparison_summary || 'N/A'}`));
|
||||
} else {
|
||||
console.log(chalk.gray(' Provider comparison data not found.'));
|
||||
}
|
||||
console.log();
|
||||
|
||||
// Detected Issues
|
||||
console.log(chalk.red.bold('🚨 Detected Issues:'));
|
||||
if (
|
||||
Array.isArray(reportData.detected_issues) &&
|
||||
reportData.detected_issues.length > 0
|
||||
) {
|
||||
reportData.detected_issues.forEach((issue, index) => {
|
||||
if (typeof issue !== 'object' || issue === null) return; // Skip invalid issue entries
|
||||
|
||||
const severity = issue.severity || 'Unknown';
|
||||
let boxColor = 'blue';
|
||||
let icon = 'ℹ️';
|
||||
if (severity === 'Error') {
|
||||
boxColor = 'red';
|
||||
icon = '❌';
|
||||
}
|
||||
if (severity === 'Warning') {
|
||||
boxColor = 'yellow';
|
||||
icon = '⚠️';
|
||||
}
|
||||
|
||||
let issueContent = `${chalk.bold('Description:')} ${chalk.white(issue.description || 'N/A')}`;
|
||||
// Only add log context if it exists and is not empty
|
||||
if (issue.log_context && String(issue.log_context).trim()) {
|
||||
issueContent += `\n${chalk.bold('Log Context:')} \n${chalk.dim(String(issue.log_context).trim())}`;
|
||||
}
|
||||
|
||||
console.log(
|
||||
boxen(issueContent, {
|
||||
title: `${icon} Issue ${index + 1}: [${severity}]`,
|
||||
padding: 1,
|
||||
margin: { top: 1, bottom: 0 },
|
||||
borderColor: boxColor,
|
||||
borderStyle: 'round'
|
||||
})
|
||||
);
|
||||
});
|
||||
console.log(); // Add final newline if issues exist
|
||||
} else {
|
||||
console.log(chalk.green(' No specific issues detected by the LLM.'));
|
||||
}
|
||||
console.log();
|
||||
|
||||
console.log(chalk.cyan.bold('========================================'));
|
||||
console.log(chalk.cyan.bold(' End of LLM Report'));
|
||||
console.log(chalk.cyan.bold('========================================\n'));
|
||||
} catch (error) {
|
||||
// Ensure chalk is available for error reporting, provide fallback
|
||||
const errorChalk = chalk || { red: (t) => t, yellow: (t) => t };
|
||||
console.error(
|
||||
errorChalk.red('Error processing LLM response:'),
|
||||
error.message
|
||||
);
|
||||
// Avoid printing potentially huge inputData here unless necessary for debugging
|
||||
// console.error(errorChalk.yellow('Raw input data (first 500 chars):'), inputData.substring(0, 500));
|
||||
process.exit(1);
|
||||
}
|
||||
});
|
||||
|
||||
// Handle potential errors during stdin reading
|
||||
process.stdin.on('error', (err) => {
|
||||
console.error('Error reading standard input:', err);
|
||||
process.exit(1);
|
||||
});
|
||||
@@ -18,6 +18,58 @@ SAMPLE_PRD_SOURCE="$TASKMASTER_SOURCE_DIR/tests/fixtures/sample-prd.txt"
|
||||
MAIN_ENV_FILE="$TASKMASTER_SOURCE_DIR/.env"
|
||||
# ---
|
||||
|
||||
# <<< Source the helper script >>>
|
||||
source "$TASKMASTER_SOURCE_DIR/tests/e2e/e2e_helpers.sh"
|
||||
|
||||
# --- Argument Parsing for Analysis-Only Mode ---
|
||||
if [ "$#" -ge 2 ] && [ "$1" == "--analyze-log" ]; then
|
||||
LOG_TO_ANALYZE="$2"
|
||||
# Ensure the log path is absolute
|
||||
if [[ "$LOG_TO_ANALYZE" != /* ]]; then
|
||||
LOG_TO_ANALYZE="$(pwd)/$LOG_TO_ANALYZE"
|
||||
fi
|
||||
echo "[INFO] Running in analysis-only mode for log: $LOG_TO_ANALYZE"
|
||||
|
||||
# --- Derive TEST_RUN_DIR from log file path ---
|
||||
# Extract timestamp like YYYYMMDD_HHMMSS from e2e_run_YYYYMMDD_HHMMSS.log
|
||||
log_basename=$(basename "$LOG_TO_ANALYZE")
|
||||
timestamp_match=$(echo "$log_basename" | sed -n 's/^e2e_run_\([0-9]\{8\}_[0-9]\{6\}\).log$/\1/p')
|
||||
|
||||
if [ -z "$timestamp_match" ]; then
|
||||
echo "[ERROR] Could not extract timestamp from log file name: $log_basename" >&2
|
||||
echo "[ERROR] Expected format: e2e_run_YYYYMMDD_HHMMSS.log" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Construct the expected run directory path relative to project root
|
||||
EXPECTED_RUN_DIR="$TASKMASTER_SOURCE_DIR/tests/e2e/_runs/run_$timestamp_match"
|
||||
# Make it absolute
|
||||
EXPECTED_RUN_DIR_ABS="$(cd "$TASKMASTER_SOURCE_DIR" && pwd)/tests/e2e/_runs/run_$timestamp_match"
|
||||
|
||||
if [ ! -d "$EXPECTED_RUN_DIR_ABS" ]; then
|
||||
echo "[ERROR] Corresponding test run directory not found: $EXPECTED_RUN_DIR_ABS" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Save original dir before changing
|
||||
ORIGINAL_DIR=$(pwd)
|
||||
|
||||
echo "[INFO] Changing directory to $EXPECTED_RUN_DIR_ABS for analysis context..."
|
||||
cd "$EXPECTED_RUN_DIR_ABS"
|
||||
|
||||
# Call the analysis function (sourced from helpers)
|
||||
echo "[INFO] Calling analyze_log_with_llm function..."
|
||||
analyze_log_with_llm "$LOG_TO_ANALYZE" "$(cd "$ORIGINAL_DIR/$TASKMASTER_SOURCE_DIR" && pwd)" # Pass absolute project root
|
||||
ANALYSIS_EXIT_CODE=$?
|
||||
|
||||
# Return to original directory
|
||||
cd "$ORIGINAL_DIR"
|
||||
exit $ANALYSIS_EXIT_CODE
|
||||
fi
|
||||
# --- End Analysis-Only Mode Logic ---
|
||||
|
||||
# --- Normal Execution Starts Here (if not in analysis-only mode) ---
|
||||
|
||||
# --- Test State Variables ---
|
||||
# Note: These are mainly for step numbering within the log now, not for final summary
|
||||
test_step_count=0
|
||||
@@ -29,7 +81,8 @@ start_time_for_helpers=0 # Separate start time for helper functions inside the p
|
||||
mkdir -p "$LOG_DIR"
|
||||
# Define timestamped log file path
|
||||
TIMESTAMP=$(date +"%Y%m%d_%H%M%S")
|
||||
LOG_FILE="$LOG_DIR/e2e_run_$TIMESTAMP.log"
|
||||
# <<< Use pwd to create an absolute path >>>
|
||||
LOG_FILE="$(pwd)/$LOG_DIR/e2e_run_$TIMESTAMP"
|
||||
|
||||
# Define and create the test run directory *before* the main pipe
|
||||
mkdir -p "$BASE_TEST_DIR" # Ensure base exists first
|
||||
@@ -44,167 +97,53 @@ echo "--- Starting E2E Run ---" # Separator before piped output starts
|
||||
# Record start time for overall duration *before* the pipe
|
||||
overall_start_time=$(date +%s)
|
||||
|
||||
# ==========================================
|
||||
# >>> MOVE FUNCTION DEFINITION HERE <<<
|
||||
# --- Helper Functions (Define globally) ---
|
||||
_format_duration() {
|
||||
local total_seconds=$1
|
||||
local minutes=$((total_seconds / 60))
|
||||
local seconds=$((total_seconds % 60))
|
||||
printf "%dm%02ds" "$minutes" "$seconds"
|
||||
}
|
||||
|
||||
# Note: This relies on 'overall_start_time' being set globally before the function is called
|
||||
_get_elapsed_time_for_log() {
|
||||
local current_time=$(date +%s)
|
||||
# Use overall_start_time here, as start_time_for_helpers might not be relevant globally
|
||||
local elapsed_seconds=$((current_time - overall_start_time))
|
||||
_format_duration "$elapsed_seconds"
|
||||
}
|
||||
|
||||
log_info() {
|
||||
echo "[INFO] [$(_get_elapsed_time_for_log)] $(date +"%Y-%m-%d %H:%M:%S") $1"
|
||||
}
|
||||
|
||||
log_success() {
|
||||
echo "[SUCCESS] [$(_get_elapsed_time_for_log)] $(date +"%Y-%m-%d %H:%M:%S") $1"
|
||||
}
|
||||
|
||||
log_error() {
|
||||
echo "[ERROR] [$(_get_elapsed_time_for_log)] $(date +"%Y-%m-%d %H:%M:%S") $1" >&2
|
||||
}
|
||||
|
||||
log_step() {
|
||||
test_step_count=$((test_step_count + 1))
|
||||
echo ""
|
||||
echo "============================================="
|
||||
echo " STEP ${test_step_count}: [$(_get_elapsed_time_for_log)] $(date +"%Y-%m-%d %H:%M:%S") $1"
|
||||
echo "============================================="
|
||||
}
|
||||
|
||||
# ==========================================
|
||||
|
||||
# --- Main Execution Block (Piped to tee) ---
|
||||
# Wrap the main part of the script in braces and pipe its output (stdout and stderr) to tee
|
||||
{
|
||||
# Record start time for helper functions *inside* the pipe
|
||||
start_time_for_helpers=$(date +%s)
|
||||
|
||||
# --- Helper Functions (Output will now go to tee -> terminal & log file) ---
|
||||
_format_duration() {
|
||||
local total_seconds=$1
|
||||
local minutes=$((total_seconds / 60))
|
||||
local seconds=$((total_seconds % 60))
|
||||
printf "%dm%02ds" "$minutes" "$seconds"
|
||||
}
|
||||
|
||||
_get_elapsed_time_for_log() {
|
||||
local current_time=$(date +%s)
|
||||
local elapsed_seconds=$((current_time - start_time_for_helpers))
|
||||
_format_duration "$elapsed_seconds"
|
||||
}
|
||||
|
||||
log_info() {
|
||||
echo "[INFO] [$(_get_elapsed_time_for_log)] $(date +"%Y-%m-%d %H:%M:%S") $1"
|
||||
}
|
||||
|
||||
log_success() {
|
||||
# We no longer increment success_step_count here for the final summary
|
||||
echo "[SUCCESS] [$(_get_elapsed_time_for_log)] $(date +"%Y-%m-%d %H:%M:%S") $1"
|
||||
}
|
||||
|
||||
log_error() {
|
||||
# Output errors to stderr, which gets merged and sent to tee
|
||||
echo "[ERROR] [$(_get_elapsed_time_for_log)] $(date +"%Y-%m-%d %H:%M:%S") $1" >&2
|
||||
}
|
||||
|
||||
log_step() {
|
||||
test_step_count=$((test_step_count + 1))
|
||||
echo ""
|
||||
echo "============================================="
|
||||
echo " STEP ${test_step_count}: [$(_get_elapsed_time_for_log)] $(date +"%Y-%m-%d %H:%M:%S") $1"
|
||||
echo "============================================="
|
||||
}
|
||||
|
||||
analyze_log_with_llm() {
|
||||
local log_file="$1"
|
||||
local provider_summary_log="provider_add_task_summary.log" # File summarizing provider test outcomes
|
||||
local api_key=""
|
||||
local api_endpoint="https://api.anthropic.com/v1/messages"
|
||||
local api_key_name="CLAUDE_API_KEY"
|
||||
|
||||
echo "" # Add a newline before analysis starts
|
||||
log_info "Attempting LLM analysis of log: $log_file"
|
||||
|
||||
# Check for jq and curl
|
||||
if ! command -v jq &> /dev/null; then
|
||||
log_error "LLM Analysis requires 'jq'. Skipping analysis."
|
||||
return 1
|
||||
fi
|
||||
if ! command -v curl &> /dev/null; then
|
||||
log_error "LLM Analysis requires 'curl'. Skipping analysis."
|
||||
return 1
|
||||
fi
|
||||
|
||||
# Check for API Key in the TEST_RUN_DIR/.env (copied earlier)
|
||||
if [ -f ".env" ]; then
|
||||
# Using grep and sed for better handling of potential quotes/spaces
|
||||
api_key=$(grep "^${api_key_name}=" .env | sed -e "s/^${api_key_name}=//" -e 's/^[[:space:]"]*//' -e 's/[[:space:]"]*$//')
|
||||
fi
|
||||
|
||||
if [ -z "$api_key" ]; then
|
||||
log_error "${api_key_name} not found or empty in .env file in the test run directory ($(pwd)/.env). Skipping LLM analysis."
|
||||
return 1
|
||||
fi
|
||||
|
||||
if [ ! -f "$log_file" ]; then
|
||||
log_error "Log file not found: $log_file. Skipping LLM analysis."
|
||||
return 1
|
||||
fi
|
||||
|
||||
log_info "Reading log file content..."
|
||||
local log_content
|
||||
# Read entire file, handle potential errors
|
||||
log_content=$(cat "$log_file") || {
|
||||
log_error "Failed to read log file: $log_file. Skipping LLM analysis."
|
||||
return 1
|
||||
}
|
||||
|
||||
# Prepare the prompt
|
||||
# Using printf with %s for the log content is generally safer than direct variable expansion
|
||||
local prompt_template='Analyze the following E2E test log for the task-master tool. The log contains output from various '\''task-master'\'' commands executed sequentially.\n\nYour goal is to:\n1. Verify if the key E2E steps completed successfully based on the log messages (e.g., init, parse PRD, list tasks, analyze complexity, expand task, set status, manage models, add/remove dependencies, add/update/remove tasks/subtasks, generate files).\n2. **Specifically analyze the Multi-Provider Add-Task Test Sequence:**\n a. Identify which providers were tested for `add-task`. Look for log steps like "Testing Add-Task with Provider: ..." and the summary log `'"$provider_summary_log"'`.\n b. For each tested provider, determine if `add-task` succeeded or failed. Note the created task ID if successful.\n c. Review the corresponding `add_task_show_output_<provider>_id_<id>.log` file (if created) for each successful `add-task` execution.\n d. **Compare the quality and completeness** of the task generated by each successful provider based on their `show` output. Assign a score (e.g., 1-10, 10 being best) based on relevance to the prompt, detail level, and correctness.\n e. Note any providers where `add-task` failed or where the task ID could not be extracted.\n3. Identify any general explicit "[ERROR]" messages or stack traces throughout the *entire* log.\n4. Identify any potential warnings or unusual output that might indicate a problem even if not marked as an explicit error.\n5. Provide an overall assessment of the test run'\''s health based *only* on the log content.\n\nReturn your analysis **strictly** in the following JSON format. Do not include any text outside of the JSON structure:\n\n{\n "overall_status": "Success|Failure|Warning",\n "verified_steps": [ "Initialization", "PRD Parsing", /* ...other general steps observed... */ ],\n "provider_add_task_comparison": {\n "prompt_used": "... (extract from log if possible or state 'standard auth prompt') ...",\n "provider_results": {\n "anthropic": { "status": "Success|Failure|ID_Extraction_Failed|Set_Model_Failed", "task_id": "...", "score": "X/10 | N/A", "notes": "..." },\n "openai": { "status": "Success|Failure|...", "task_id": "...", "score": "X/10 | N/A", "notes": "..." },\n /* ... include all tested providers ... */\n },\n "comparison_summary": "Brief overall comparison of generated tasks..."\n },\n "detected_issues": [ { "severity": "Error|Warning|Anomaly", "description": "...", "log_context": "[Optional, short snippet from log near the issue]" } ],\n "llm_summary_points": [ "Overall summary point 1", "Provider comparison highlight", "Any major issues noted" ]\n}\n\nHere is the main log content:\n\n%s'
|
||||
|
||||
local full_prompt
|
||||
printf -v full_prompt "$prompt_template" "$log_content"
|
||||
|
||||
# Construct the JSON payload for Claude Messages API
|
||||
# Using jq for robust JSON construction
|
||||
local payload
|
||||
payload=$(jq -n --arg prompt "$full_prompt" '{
|
||||
"model": "claude-3-7-sonnet-20250219",
|
||||
"max_tokens": 10000,
|
||||
"messages": [
|
||||
{"role": "user", "content": $prompt}
|
||||
],
|
||||
"temperature": 0.0
|
||||
}') || {
|
||||
log_error "Failed to create JSON payload using jq."
|
||||
return 1
|
||||
}
|
||||
|
||||
log_info "Sending request to LLM API endpoint: $api_endpoint ..."
|
||||
local response_raw response_http_code response_body
|
||||
# Capture body and HTTP status code separately
|
||||
response_raw=$(curl -s -w "\nHTTP_STATUS_CODE:%{http_code}" -X POST "$api_endpoint" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "x-api-key: $api_key" \
|
||||
-H "anthropic-version: 2023-06-01" \
|
||||
--data "$payload")
|
||||
|
||||
# Extract status code and body
|
||||
response_http_code=$(echo "$response_raw" | grep '^HTTP_STATUS_CODE:' | sed 's/HTTP_STATUS_CODE://')
|
||||
response_body=$(echo "$response_raw" | sed '$d') # Remove last line (status code)
|
||||
|
||||
if [ "$response_http_code" != "200" ]; then
|
||||
log_error "LLM API call failed with HTTP status $response_http_code."
|
||||
log_error "Response Body: $response_body"
|
||||
return 1
|
||||
fi
|
||||
|
||||
if [ -z "$response_body" ]; then
|
||||
log_error "LLM API call returned empty response body."
|
||||
return 1
|
||||
fi
|
||||
|
||||
log_info "Received LLM response (HTTP 200). Parsing analysis JSON..."
|
||||
|
||||
# Extract the analysis JSON string from the API response (adjust jq path if needed)
|
||||
local analysis_json_string
|
||||
analysis_json_string=$(echo "$response_body" | jq -r '.content[0].text' 2>/dev/null) # Assumes Messages API structure
|
||||
|
||||
if [ -z "$analysis_json_string" ]; then
|
||||
log_error "Failed to extract 'content[0].text' from LLM response JSON."
|
||||
log_error "Full API response body: $response_body"
|
||||
return 1
|
||||
fi
|
||||
|
||||
# Validate and pretty-print the extracted JSON
|
||||
if ! echo "$analysis_json_string" | jq -e . > /dev/null 2>&1; then
|
||||
log_error "Extracted content from LLM is not valid JSON."
|
||||
log_error "Raw extracted content: $analysis_json_string"
|
||||
return 1
|
||||
fi
|
||||
|
||||
log_success "LLM analysis completed successfully."
|
||||
echo ""
|
||||
echo "--- LLM Analysis ---"
|
||||
# Pretty print the JSON analysis
|
||||
echo "$analysis_json_string" | jq '.'
|
||||
echo "--------------------"
|
||||
|
||||
return 0
|
||||
}
|
||||
# ---
|
||||
# Note: Helper functions are now defined globally above,
|
||||
# but we still need start_time_for_helpers if any logging functions
|
||||
# called *inside* this block depend on it. If not, it can be removed.
|
||||
start_time_for_helpers=$(date +%s) # Keep if needed by helpers called inside this block
|
||||
|
||||
# --- Test Setup (Output to tee) ---
|
||||
log_step "Setting up test environment"
|
||||
@@ -264,8 +203,8 @@ overall_start_time=$(date +%s)
|
||||
|
||||
log_step "Initializing Task Master project (non-interactive)"
|
||||
task-master init -y --name="E2E Test $TIMESTAMP" --description="Automated E2E test run"
|
||||
if [ ! -f ".taskmasterconfig" ] || [ ! -f "package.json" ]; then
|
||||
log_error "Initialization failed: .taskmasterconfig or package.json not found."
|
||||
if [ ! -f ".taskmasterconfig" ]; then
|
||||
log_error "Initialization failed: .taskmasterconfig not found."
|
||||
exit 1
|
||||
fi
|
||||
log_success "Project initialized."
|
||||
@@ -385,9 +324,9 @@ overall_start_time=$(date +%s)
|
||||
|
||||
# 3. Check for success and extract task ID
|
||||
new_task_id=""
|
||||
if [ $add_task_exit_code -eq 0 ] && echo "$add_task_cmd_output" | grep -q "Successfully added task with ID:"; then
|
||||
if [ $add_task_exit_code -eq 0 ] && echo "$add_task_cmd_output" | grep -q "✓ Added new task #"; then
|
||||
# Attempt to extract the ID (adjust grep/sed/awk as needed based on actual output format)
|
||||
new_task_id=$(echo "$add_task_cmd_output" | grep "Successfully added task with ID:" | sed 's/.*Successfully added task with ID: \([0-9.]\+\).*/\1/')
|
||||
new_task_id=$(echo "$add_task_cmd_output" | grep "✓ Added new task #" | sed 's/.*✓ Added new task #\([0-9.]\+\).*/\1/')
|
||||
if [ -n "$new_task_id" ]; then
|
||||
log_success "Add-task succeeded for $provider. New task ID: $new_task_id"
|
||||
echo "Provider $provider add-task SUCCESS (ID: $new_task_id)" >> provider_add_task_summary.log
|
||||
@@ -458,6 +397,68 @@ overall_start_time=$(date +%s)
|
||||
task-master fix-dependencies > fix_dependencies_output.log
|
||||
log_success "Fix dependencies attempted."
|
||||
|
||||
# === Start New Test Section: Validate/Fix Bad Dependencies ===
|
||||
|
||||
log_step "Intentionally adding non-existent dependency (1 -> 999)"
|
||||
task-master add-dependency --id=1 --depends-on=999 || log_error "Failed to add non-existent dependency (unexpected)"
|
||||
# Don't exit even if the above fails, the goal is to test validation
|
||||
log_success "Attempted to add dependency 1 -> 999."
|
||||
|
||||
log_step "Validating dependencies (expecting non-existent error)"
|
||||
task-master validate-dependencies > validate_deps_non_existent.log 2>&1 || true # Allow command to fail without exiting script
|
||||
if grep -q "Non-existent dependency ID: 999" validate_deps_non_existent.log; then
|
||||
log_success "Validation correctly identified non-existent dependency 999."
|
||||
else
|
||||
log_error "Validation DID NOT report non-existent dependency 999 as expected. Check validate_deps_non_existent.log"
|
||||
# Consider exiting here if this check fails, as it indicates a validation logic problem
|
||||
# exit 1
|
||||
fi
|
||||
|
||||
log_step "Fixing dependencies (should remove 1 -> 999)"
|
||||
task-master fix-dependencies > fix_deps_after_non_existent.log
|
||||
log_success "Attempted to fix dependencies."
|
||||
|
||||
log_step "Validating dependencies (after fix)"
|
||||
task-master validate-dependencies > validate_deps_after_fix_non_existent.log 2>&1 || true # Allow potential failure
|
||||
if grep -q "Non-existent dependency ID: 999" validate_deps_after_fix_non_existent.log; then
|
||||
log_error "Validation STILL reports non-existent dependency 999 after fix. Check logs."
|
||||
# exit 1
|
||||
else
|
||||
log_success "Validation shows non-existent dependency 999 was removed."
|
||||
fi
|
||||
|
||||
|
||||
log_step "Intentionally adding circular dependency (4 -> 5 -> 4)"
|
||||
task-master add-dependency --id=4 --depends-on=5 || log_error "Failed to add dependency 4->5"
|
||||
task-master add-dependency --id=5 --depends-on=4 || log_error "Failed to add dependency 5->4"
|
||||
log_success "Attempted to add dependencies 4 -> 5 and 5 -> 4."
|
||||
|
||||
|
||||
log_step "Validating dependencies (expecting circular error)"
|
||||
task-master validate-dependencies > validate_deps_circular.log 2>&1 || true # Allow command to fail
|
||||
# Note: Adjust the grep pattern based on the EXACT error message from validate-dependencies
|
||||
if grep -q -E "Circular dependency detected involving task IDs: (4, 5|5, 4)" validate_deps_circular.log; then
|
||||
log_success "Validation correctly identified circular dependency between 4 and 5."
|
||||
else
|
||||
log_error "Validation DID NOT report circular dependency 4<->5 as expected. Check validate_deps_circular.log"
|
||||
# exit 1
|
||||
fi
|
||||
|
||||
log_step "Fixing dependencies (should remove one side of 4 <-> 5)"
|
||||
task-master fix-dependencies > fix_deps_after_circular.log
|
||||
log_success "Attempted to fix dependencies."
|
||||
|
||||
log_step "Validating dependencies (after fix circular)"
|
||||
task-master validate-dependencies > validate_deps_after_fix_circular.log 2>&1 || true # Allow potential failure
|
||||
if grep -q -E "Circular dependency detected involving task IDs: (4, 5|5, 4)" validate_deps_after_fix_circular.log; then
|
||||
log_error "Validation STILL reports circular dependency 4<->5 after fix. Check logs."
|
||||
# exit 1
|
||||
else
|
||||
log_success "Validation shows circular dependency 4<->5 was resolved."
|
||||
fi
|
||||
|
||||
# === End New Test Section ===
|
||||
|
||||
log_step "Adding Task 11 (Manual)"
|
||||
task-master add-task --title="Manual E2E Task" --description="Add basic health check endpoint" --priority=low --dependencies=3 # Depends on backend setup
|
||||
# Assuming the new task gets ID 11 (adjust if PRD parsing changes)
|
||||
@@ -485,7 +486,7 @@ overall_start_time=$(date +%s)
|
||||
log_success "Attempted update for Subtask 8.1."
|
||||
|
||||
# Add a couple more subtasks for multi-remove test
|
||||
log_step "Adding subtasks to Task 2 (for multi-remove test)"
|
||||
log_step 'Adding subtasks to Task 2 (for multi-remove test)'
|
||||
task-master add-subtask --parent=2 --title="Subtask 2.1 for removal"
|
||||
task-master add-subtask --parent=2 --title="Subtask 2.2 for removal"
|
||||
log_success "Added subtasks 2.1 and 2.2."
|
||||
@@ -502,6 +503,18 @@ overall_start_time=$(date +%s)
|
||||
task-master next > next_task_after_change.log
|
||||
log_success "Next task after change saved to next_task_after_change.log"
|
||||
|
||||
# === Start New Test Section: List Filtering ===
|
||||
log_step "Listing tasks filtered by status 'done'"
|
||||
task-master list --status=done > task_list_status_done.log
|
||||
log_success "Filtered list saved to task_list_status_done.log (Manual/LLM check recommended)"
|
||||
# Optional assertion: Check if Task 1 ID exists and Task 2 ID does NOT
|
||||
# if grep -q "^1\." task_list_status_done.log && ! grep -q "^2\." task_list_status_done.log; then
|
||||
# log_success "Basic check passed: Task 1 found, Task 2 not found in 'done' list."
|
||||
# else
|
||||
# log_error "Basic check failed for list --status=done."
|
||||
# fi
|
||||
# === End New Test Section ===
|
||||
|
||||
log_step "Clearing subtasks from Task 8"
|
||||
task-master clear-subtasks --id=8
|
||||
log_success "Attempted to clear subtasks from Task 8."
|
||||
@@ -511,6 +524,46 @@ overall_start_time=$(date +%s)
|
||||
task-master remove-task --id=11,12 -y
|
||||
log_success "Removed tasks 11 and 12."
|
||||
|
||||
# === Start New Test Section: Subtasks & Dependencies ===
|
||||
|
||||
log_step "Expanding Task 2 (to ensure multiple tasks have subtasks)"
|
||||
task-master expand --id=2 # Expand task 2: Backend setup
|
||||
log_success "Attempted to expand Task 2."
|
||||
|
||||
log_step "Listing tasks with subtasks (Before Clear All)"
|
||||
task-master list --with-subtasks > task_list_before_clear_all.log
|
||||
log_success "Task list before clear-all saved."
|
||||
|
||||
log_step "Clearing ALL subtasks"
|
||||
task-master clear-subtasks --all
|
||||
log_success "Attempted to clear all subtasks."
|
||||
|
||||
log_step "Listing tasks with subtasks (After Clear All)"
|
||||
task-master list --with-subtasks > task_list_after_clear_all.log
|
||||
log_success "Task list after clear-all saved. (Manual/LLM check recommended to verify subtasks removed)"
|
||||
|
||||
log_step "Expanding Task 1 again (to have subtasks for next test)"
|
||||
task-master expand --id=1
|
||||
log_success "Attempted to expand Task 1 again."
|
||||
|
||||
log_step "Adding dependency: Task 3 depends on Subtask 1.1"
|
||||
task-master add-dependency --id=3 --depends-on=1.1
|
||||
log_success "Added dependency 3 -> 1.1."
|
||||
|
||||
log_step "Showing Task 3 details (after adding subtask dependency)"
|
||||
task-master show 3 > task_3_details_after_dep_add.log
|
||||
log_success "Task 3 details saved. (Manual/LLM check recommended for dependency [1.1])"
|
||||
|
||||
log_step "Removing dependency: Task 3 depends on Subtask 1.1"
|
||||
task-master remove-dependency --id=3 --depends-on=1.1
|
||||
log_success "Removed dependency 3 -> 1.1."
|
||||
|
||||
log_step "Showing Task 3 details (after removing subtask dependency)"
|
||||
task-master show 3 > task_3_details_after_dep_remove.log
|
||||
log_success "Task 3 details saved. (Manual/LLM check recommended to verify dependency removed)"
|
||||
|
||||
# === End New Test Section ===
|
||||
|
||||
log_step "Generating task files (final)"
|
||||
task-master generate
|
||||
log_success "Generated task files."
|
||||
@@ -601,25 +654,16 @@ fi
|
||||
echo "-------------------------"
|
||||
|
||||
# --- Attempt LLM Analysis ---
|
||||
echo "DEBUG: Entering LLM Analysis section..."
|
||||
# Run this *after* the main execution block and tee pipe finish writing the log file
|
||||
# It will read the completed log file and append its output to the terminal (and the log via subsequent writes if tee is still active, though it shouldn't be)
|
||||
# Change directory back into the test run dir where .env is located
|
||||
if [ -d "$TEST_RUN_DIR" ]; then
|
||||
echo "DEBUG: Found TEST_RUN_DIR: $TEST_RUN_DIR. Attempting cd..."
|
||||
cd "$TEST_RUN_DIR"
|
||||
echo "DEBUG: Changed directory to $(pwd). Calling analyze_log_with_llm..."
|
||||
analyze_log_with_llm "$LOG_FILE"
|
||||
echo "DEBUG: analyze_log_with_llm function call finished."
|
||||
# Optional: cd back again if needed, though script is ending
|
||||
analyze_log_with_llm "$LOG_FILE" "$TASKMASTER_SOURCE_DIR"
|
||||
ANALYSIS_EXIT_CODE=$? # Capture the exit code of the analysis function
|
||||
# Optional: cd back again if needed
|
||||
# cd "$ORIGINAL_DIR"
|
||||
else
|
||||
# Use log_error format even outside the pipe for consistency
|
||||
current_time_for_error=$(date +%s)
|
||||
elapsed_seconds_for_error=$((current_time_for_error - overall_start_time)) # Use overall start time
|
||||
formatted_duration_for_error=$(_format_duration "$elapsed_seconds_for_error")
|
||||
formatted_duration_for_error=$(_format_duration "$total_elapsed_seconds")
|
||||
echo "[ERROR] [$formatted_duration_for_error] $(date +"%Y-%m-%d %H:%M:%S") Test run directory $TEST_RUN_DIR not found. Cannot perform LLM analysis." >&2
|
||||
fi
|
||||
|
||||
echo "DEBUG: Reached end of script before final exit."
|
||||
exit $EXIT_CODE # Exit with the status of the main script block
|
||||
exit $EXIT_CODE
|
||||
71
tests/e2e/test_llm_analysis.sh
Executable file
71
tests/e2e/test_llm_analysis.sh
Executable file
@@ -0,0 +1,71 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Script to test the LLM analysis function independently
|
||||
|
||||
# Exit on error
|
||||
set -u
|
||||
set -o pipefail
|
||||
|
||||
# Source the helper functions
|
||||
HELPER_SCRIPT="tests/e2e/e2e_helpers.sh"
|
||||
if [ -f "$HELPER_SCRIPT" ]; then
|
||||
source "$HELPER_SCRIPT"
|
||||
echo "[INFO] Sourced helper script: $HELPER_SCRIPT"
|
||||
else
|
||||
echo "[ERROR] Helper script not found at $HELPER_SCRIPT. Exiting." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# --- Configuration ---
|
||||
# Get the absolute path to the project root (assuming this script is run from the root)
|
||||
PROJECT_ROOT="$(pwd)"
|
||||
|
||||
# --- Argument Parsing ---
|
||||
if [ "$#" -ne 2 ]; then
|
||||
echo "Usage: $0 <path_to_log_file> <path_to_test_run_directory>" >&2
|
||||
echo "Example: $0 tests/e2e/log/e2e_run_YYYYMMDD_HHMMSS.log tests/e2e/_runs/run_YYYYMMDD_HHMMSS" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
LOG_FILE_REL="$1" # Relative path from project root
|
||||
TEST_RUN_DIR_REL="$2" # Relative path from project root
|
||||
|
||||
# Construct absolute paths
|
||||
LOG_FILE_ABS="$PROJECT_ROOT/$LOG_FILE_REL"
|
||||
TEST_RUN_DIR_ABS="$PROJECT_ROOT/$TEST_RUN_DIR_REL"
|
||||
|
||||
# --- Validation ---
|
||||
if [ ! -f "$LOG_FILE_ABS" ]; then
|
||||
echo "[ERROR] Log file not found: $LOG_FILE_ABS" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ ! -d "$TEST_RUN_DIR_ABS" ]; then
|
||||
echo "[ERROR] Test run directory not found: $TEST_RUN_DIR_ABS" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ ! -f "$TEST_RUN_DIR_ABS/.env" ]; then
|
||||
echo "[ERROR] .env file not found in test run directory: $TEST_RUN_DIR_ABS/.env" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
|
||||
# --- Execution ---
|
||||
echo "[INFO] Changing directory to test run directory: $TEST_RUN_DIR_ABS"
|
||||
cd "$TEST_RUN_DIR_ABS" || { echo "[ERROR] Failed to cd into $TEST_RUN_DIR_ABS"; exit 1; }
|
||||
|
||||
echo "[INFO] Current directory: $(pwd)"
|
||||
echo "[INFO] Calling analyze_log_with_llm function with log file: $LOG_FILE_ABS"
|
||||
|
||||
# Call the function (sourced earlier)
|
||||
analyze_log_with_llm "$LOG_FILE_ABS"
|
||||
ANALYSIS_EXIT_CODE=$?
|
||||
|
||||
echo "[INFO] analyze_log_with_llm finished with exit code: $ANALYSIS_EXIT_CODE"
|
||||
|
||||
# Optional: cd back to original directory
|
||||
# echo "[INFO] Changing back to project root: $PROJECT_ROOT"
|
||||
# cd "$PROJECT_ROOT"
|
||||
|
||||
exit $ANALYSIS_EXIT_CODE
|
||||
74
tests/integration/roo-files-inclusion.test.js
Normal file
74
tests/integration/roo-files-inclusion.test.js
Normal file
@@ -0,0 +1,74 @@
|
||||
import { jest } from '@jest/globals';
|
||||
import fs from 'fs';
|
||||
import path from 'path';
|
||||
import os from 'os';
|
||||
import { execSync } from 'child_process';
|
||||
|
||||
describe('Roo Files Inclusion in Package', () => {
|
||||
// This test verifies that the required Roo files are included in the final package
|
||||
|
||||
test('package.json includes assets/** in the "files" array for Roo source files', () => {
|
||||
// Read the package.json file
|
||||
const packageJsonPath = path.join(process.cwd(), 'package.json');
|
||||
const packageJson = JSON.parse(fs.readFileSync(packageJsonPath, 'utf8'));
|
||||
|
||||
// Check if assets/** is included in the files array (which contains Roo files)
|
||||
expect(packageJson.files).toContain('assets/**');
|
||||
});
|
||||
|
||||
test('prepare-package.js verifies required Roo files', () => {
|
||||
// Read the prepare-package.js file
|
||||
const preparePackagePath = path.join(
|
||||
process.cwd(),
|
||||
'scripts',
|
||||
'prepare-package.js'
|
||||
);
|
||||
const preparePackageContent = fs.readFileSync(preparePackagePath, 'utf8');
|
||||
|
||||
// Check if prepare-package.js includes verification for Roo files
|
||||
expect(preparePackageContent).toContain('.roo/rules/');
|
||||
expect(preparePackageContent).toContain('.roomodes');
|
||||
expect(preparePackageContent).toContain('assets/roocode/');
|
||||
});
|
||||
|
||||
test('init.js creates Roo directories and copies files', () => {
|
||||
// Read the init.js file
|
||||
const initJsPath = path.join(process.cwd(), 'scripts', 'init.js');
|
||||
const initJsContent = fs.readFileSync(initJsPath, 'utf8');
|
||||
|
||||
// Check for Roo directory creation (using more flexible pattern matching)
|
||||
const hasRooDir = initJsContent.includes(
|
||||
"ensureDirectoryExists(path.join(targetDir, '.roo"
|
||||
);
|
||||
expect(hasRooDir).toBe(true);
|
||||
|
||||
// Check for .roomodes file copying
|
||||
const hasRoomodes = initJsContent.includes("copyTemplateFile('.roomodes'");
|
||||
expect(hasRoomodes).toBe(true);
|
||||
|
||||
// Check for mode-specific patterns (using more flexible pattern matching)
|
||||
const hasArchitect = initJsContent.includes('architect');
|
||||
const hasAsk = initJsContent.includes('ask');
|
||||
const hasBoomerang = initJsContent.includes('boomerang');
|
||||
const hasCode = initJsContent.includes('code');
|
||||
const hasDebug = initJsContent.includes('debug');
|
||||
const hasTest = initJsContent.includes('test');
|
||||
|
||||
expect(hasArchitect).toBe(true);
|
||||
expect(hasAsk).toBe(true);
|
||||
expect(hasBoomerang).toBe(true);
|
||||
expect(hasCode).toBe(true);
|
||||
expect(hasDebug).toBe(true);
|
||||
expect(hasTest).toBe(true);
|
||||
});
|
||||
|
||||
test('source Roo files exist in assets directory', () => {
|
||||
// Verify that the source files for Roo integration exist
|
||||
expect(
|
||||
fs.existsSync(path.join(process.cwd(), 'assets', 'roocode', '.roo'))
|
||||
).toBe(true);
|
||||
expect(
|
||||
fs.existsSync(path.join(process.cwd(), 'assets', 'roocode', '.roomodes'))
|
||||
).toBe(true);
|
||||
});
|
||||
});
|
||||
69
tests/integration/roo-init-functionality.test.js
Normal file
69
tests/integration/roo-init-functionality.test.js
Normal file
@@ -0,0 +1,69 @@
|
||||
import { jest } from '@jest/globals';
|
||||
import fs from 'fs';
|
||||
import path from 'path';
|
||||
|
||||
describe('Roo Initialization Functionality', () => {
|
||||
let initJsContent;
|
||||
|
||||
beforeAll(() => {
|
||||
// Read the init.js file content once for all tests
|
||||
const initJsPath = path.join(process.cwd(), 'scripts', 'init.js');
|
||||
initJsContent = fs.readFileSync(initJsPath, 'utf8');
|
||||
});
|
||||
|
||||
test('init.js creates Roo directories in createProjectStructure function', () => {
|
||||
// Check if createProjectStructure function exists
|
||||
expect(initJsContent).toContain('function createProjectStructure');
|
||||
|
||||
// Check for the line that creates the .roo directory
|
||||
const hasRooDir = initJsContent.includes(
|
||||
"ensureDirectoryExists(path.join(targetDir, '.roo'))"
|
||||
);
|
||||
expect(hasRooDir).toBe(true);
|
||||
|
||||
// Check for the line that creates .roo/rules directory
|
||||
const hasRooRulesDir = initJsContent.includes(
|
||||
"ensureDirectoryExists(path.join(targetDir, '.roo', 'rules'))"
|
||||
);
|
||||
expect(hasRooRulesDir).toBe(true);
|
||||
|
||||
// Check for the for loop that creates mode-specific directories
|
||||
const hasRooModeLoop =
|
||||
initJsContent.includes(
|
||||
"for (const mode of ['architect', 'ask', 'boomerang', 'code', 'debug', 'test'])"
|
||||
) ||
|
||||
(initJsContent.includes('for (const mode of [') &&
|
||||
initJsContent.includes('architect') &&
|
||||
initJsContent.includes('ask') &&
|
||||
initJsContent.includes('boomerang') &&
|
||||
initJsContent.includes('code') &&
|
||||
initJsContent.includes('debug') &&
|
||||
initJsContent.includes('test'));
|
||||
expect(hasRooModeLoop).toBe(true);
|
||||
});
|
||||
|
||||
test('init.js copies Roo files from assets/roocode directory', () => {
|
||||
// Check for the .roomodes case in the copyTemplateFile function
|
||||
const casesRoomodes = initJsContent.includes("case '.roomodes':");
|
||||
expect(casesRoomodes).toBe(true);
|
||||
|
||||
// Check that assets/roocode appears somewhere in the file
|
||||
const hasRoocodePath = initJsContent.includes("'assets', 'roocode'");
|
||||
expect(hasRoocodePath).toBe(true);
|
||||
|
||||
// Check that roomodes file is copied
|
||||
const copiesRoomodes = initJsContent.includes(
|
||||
"copyTemplateFile('.roomodes'"
|
||||
);
|
||||
expect(copiesRoomodes).toBe(true);
|
||||
});
|
||||
|
||||
test('init.js has code to copy rule files for each mode', () => {
|
||||
// Look for template copying for rule files
|
||||
const hasModeRulesCopying =
|
||||
initJsContent.includes('copyTemplateFile(') &&
|
||||
initJsContent.includes('rules-') &&
|
||||
initJsContent.includes('-rules');
|
||||
expect(hasModeRulesCopying).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -199,16 +199,35 @@ describe('Commands Module', () => {
|
||||
// Use input option if file argument not provided
|
||||
const inputFile = file || options.input;
|
||||
const defaultPrdPath = 'scripts/prd.txt';
|
||||
const append = options.append || false;
|
||||
const force = options.force || false;
|
||||
const outputPath = options.output || 'tasks/tasks.json';
|
||||
|
||||
// Mock confirmOverwriteIfNeeded function to test overwrite behavior
|
||||
const mockConfirmOverwrite = jest.fn().mockResolvedValue(true);
|
||||
|
||||
// Helper function to check if tasks.json exists and confirm overwrite
|
||||
async function confirmOverwriteIfNeeded() {
|
||||
if (fs.existsSync(outputPath) && !force && !append) {
|
||||
return mockConfirmOverwrite();
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
// If no input file specified, check for default PRD location
|
||||
if (!inputFile) {
|
||||
if (fs.existsSync(defaultPrdPath)) {
|
||||
console.log(chalk.blue(`Using default PRD file: ${defaultPrdPath}`));
|
||||
const numTasks = parseInt(options.numTasks, 10);
|
||||
const outputPath = options.output;
|
||||
|
||||
// Check if we need to confirm overwrite
|
||||
if (!(await confirmOverwriteIfNeeded())) return;
|
||||
|
||||
console.log(chalk.blue(`Generating ${numTasks} tasks...`));
|
||||
await mockParsePRD(defaultPrdPath, outputPath, numTasks);
|
||||
if (append) {
|
||||
console.log(chalk.blue('Appending to existing tasks...'));
|
||||
}
|
||||
await mockParsePRD(defaultPrdPath, outputPath, numTasks, { append });
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -221,12 +240,20 @@ describe('Commands Module', () => {
|
||||
}
|
||||
|
||||
const numTasks = parseInt(options.numTasks, 10);
|
||||
const outputPath = options.output;
|
||||
|
||||
// Check if we need to confirm overwrite
|
||||
if (!(await confirmOverwriteIfNeeded())) return;
|
||||
|
||||
console.log(chalk.blue(`Parsing PRD file: ${inputFile}`));
|
||||
console.log(chalk.blue(`Generating ${numTasks} tasks...`));
|
||||
if (append) {
|
||||
console.log(chalk.blue('Appending to existing tasks...'));
|
||||
}
|
||||
|
||||
await mockParsePRD(inputFile, outputPath, numTasks);
|
||||
await mockParsePRD(inputFile, outputPath, numTasks, { append });
|
||||
|
||||
// Return mock for testing
|
||||
return { mockConfirmOverwrite };
|
||||
}
|
||||
|
||||
beforeEach(() => {
|
||||
@@ -252,7 +279,8 @@ describe('Commands Module', () => {
|
||||
expect(mockParsePRD).toHaveBeenCalledWith(
|
||||
'scripts/prd.txt',
|
||||
'tasks/tasks.json',
|
||||
10 // Default value from command definition
|
||||
10, // Default value from command definition
|
||||
{ append: false }
|
||||
);
|
||||
});
|
||||
|
||||
@@ -290,7 +318,8 @@ describe('Commands Module', () => {
|
||||
expect(mockParsePRD).toHaveBeenCalledWith(
|
||||
testFile,
|
||||
'tasks/tasks.json',
|
||||
10
|
||||
10,
|
||||
{ append: false }
|
||||
);
|
||||
expect(mockExistsSync).not.toHaveBeenCalledWith('scripts/prd.txt');
|
||||
});
|
||||
@@ -313,7 +342,8 @@ describe('Commands Module', () => {
|
||||
expect(mockParsePRD).toHaveBeenCalledWith(
|
||||
testFile,
|
||||
'tasks/tasks.json',
|
||||
10
|
||||
10,
|
||||
{ append: false }
|
||||
);
|
||||
expect(mockExistsSync).not.toHaveBeenCalledWith('scripts/prd.txt');
|
||||
});
|
||||
@@ -331,7 +361,126 @@ describe('Commands Module', () => {
|
||||
});
|
||||
|
||||
// Assert
|
||||
expect(mockParsePRD).toHaveBeenCalledWith(testFile, outputFile, numTasks);
|
||||
expect(mockParsePRD).toHaveBeenCalledWith(
|
||||
testFile,
|
||||
outputFile,
|
||||
numTasks,
|
||||
{ append: false }
|
||||
);
|
||||
});
|
||||
|
||||
test('should pass append flag to parsePRD when provided', async () => {
|
||||
// Arrange
|
||||
const testFile = 'test/prd.txt';
|
||||
|
||||
// Act - call the handler directly with append flag
|
||||
await parsePrdAction(testFile, {
|
||||
numTasks: '10',
|
||||
output: 'tasks/tasks.json',
|
||||
append: true
|
||||
});
|
||||
|
||||
// Assert
|
||||
expect(mockConsoleLog).toHaveBeenCalledWith(
|
||||
expect.stringContaining('Appending to existing tasks')
|
||||
);
|
||||
expect(mockParsePRD).toHaveBeenCalledWith(
|
||||
testFile,
|
||||
'tasks/tasks.json',
|
||||
10,
|
||||
{ append: true }
|
||||
);
|
||||
});
|
||||
|
||||
test('should bypass confirmation when append flag is true and tasks.json exists', async () => {
|
||||
// Arrange
|
||||
const testFile = 'test/prd.txt';
|
||||
const outputFile = 'tasks/tasks.json';
|
||||
|
||||
// Mock that tasks.json exists
|
||||
mockExistsSync.mockImplementation((path) => {
|
||||
if (path === outputFile) return true;
|
||||
if (path === testFile) return true;
|
||||
return false;
|
||||
});
|
||||
|
||||
// Act - call the handler with append flag
|
||||
const { mockConfirmOverwrite } =
|
||||
(await parsePrdAction(testFile, {
|
||||
numTasks: '10',
|
||||
output: outputFile,
|
||||
append: true
|
||||
})) || {};
|
||||
|
||||
// Assert - confirm overwrite should not be called with append flag
|
||||
expect(mockConfirmOverwrite).not.toHaveBeenCalled();
|
||||
expect(mockParsePRD).toHaveBeenCalledWith(testFile, outputFile, 10, {
|
||||
append: true
|
||||
});
|
||||
|
||||
// Reset mock implementation
|
||||
mockExistsSync.mockReset();
|
||||
});
|
||||
|
||||
test('should prompt for confirmation when append flag is false and tasks.json exists', async () => {
|
||||
// Arrange
|
||||
const testFile = 'test/prd.txt';
|
||||
const outputFile = 'tasks/tasks.json';
|
||||
|
||||
// Mock that tasks.json exists
|
||||
mockExistsSync.mockImplementation((path) => {
|
||||
if (path === outputFile) return true;
|
||||
if (path === testFile) return true;
|
||||
return false;
|
||||
});
|
||||
|
||||
// Act - call the handler without append flag
|
||||
const { mockConfirmOverwrite } =
|
||||
(await parsePrdAction(testFile, {
|
||||
numTasks: '10',
|
||||
output: outputFile
|
||||
// append: false (default)
|
||||
})) || {};
|
||||
|
||||
// Assert - confirm overwrite should be called without append flag
|
||||
expect(mockConfirmOverwrite).toHaveBeenCalled();
|
||||
expect(mockParsePRD).toHaveBeenCalledWith(testFile, outputFile, 10, {
|
||||
append: false
|
||||
});
|
||||
|
||||
// Reset mock implementation
|
||||
mockExistsSync.mockReset();
|
||||
});
|
||||
|
||||
test('should bypass confirmation when force flag is true, regardless of append flag', async () => {
|
||||
// Arrange
|
||||
const testFile = 'test/prd.txt';
|
||||
const outputFile = 'tasks/tasks.json';
|
||||
|
||||
// Mock that tasks.json exists
|
||||
mockExistsSync.mockImplementation((path) => {
|
||||
if (path === outputFile) return true;
|
||||
if (path === testFile) return true;
|
||||
return false;
|
||||
});
|
||||
|
||||
// Act - call the handler with force flag
|
||||
const { mockConfirmOverwrite } =
|
||||
(await parsePrdAction(testFile, {
|
||||
numTasks: '10',
|
||||
output: outputFile,
|
||||
force: true,
|
||||
append: false
|
||||
})) || {};
|
||||
|
||||
// Assert - confirm overwrite should not be called with force flag
|
||||
expect(mockConfirmOverwrite).not.toHaveBeenCalled();
|
||||
expect(mockParsePRD).toHaveBeenCalledWith(testFile, outputFile, 10, {
|
||||
append: false
|
||||
});
|
||||
|
||||
// Reset mock implementation
|
||||
mockExistsSync.mockReset();
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
182
tests/unit/roo-integration.test.js
Normal file
182
tests/unit/roo-integration.test.js
Normal file
@@ -0,0 +1,182 @@
|
||||
import { jest } from '@jest/globals';
|
||||
import fs from 'fs';
|
||||
import path from 'path';
|
||||
import os from 'os';
|
||||
|
||||
// Mock external modules
|
||||
jest.mock('child_process', () => ({
|
||||
execSync: jest.fn()
|
||||
}));
|
||||
|
||||
// Mock console methods
|
||||
jest.mock('console', () => ({
|
||||
log: jest.fn(),
|
||||
info: jest.fn(),
|
||||
warn: jest.fn(),
|
||||
error: jest.fn(),
|
||||
clear: jest.fn()
|
||||
}));
|
||||
|
||||
describe('Roo Integration', () => {
|
||||
let tempDir;
|
||||
|
||||
beforeEach(() => {
|
||||
jest.clearAllMocks();
|
||||
|
||||
// Create a temporary directory for testing
|
||||
tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'task-master-test-'));
|
||||
|
||||
// Spy on fs methods
|
||||
jest.spyOn(fs, 'writeFileSync').mockImplementation(() => {});
|
||||
jest.spyOn(fs, 'readFileSync').mockImplementation((filePath) => {
|
||||
if (filePath.toString().includes('.roomodes')) {
|
||||
return 'Existing roomodes content';
|
||||
}
|
||||
if (filePath.toString().includes('-rules')) {
|
||||
return 'Existing mode rules content';
|
||||
}
|
||||
return '{}';
|
||||
});
|
||||
jest.spyOn(fs, 'existsSync').mockImplementation(() => false);
|
||||
jest.spyOn(fs, 'mkdirSync').mockImplementation(() => {});
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
// Clean up the temporary directory
|
||||
try {
|
||||
fs.rmSync(tempDir, { recursive: true, force: true });
|
||||
} catch (err) {
|
||||
console.error(`Error cleaning up: ${err.message}`);
|
||||
}
|
||||
});
|
||||
|
||||
// Test function that simulates the createProjectStructure behavior for Roo files
|
||||
function mockCreateRooStructure() {
|
||||
// Create main .roo directory
|
||||
fs.mkdirSync(path.join(tempDir, '.roo'), { recursive: true });
|
||||
|
||||
// Create rules directory
|
||||
fs.mkdirSync(path.join(tempDir, '.roo', 'rules'), { recursive: true });
|
||||
|
||||
// Create mode-specific rule directories
|
||||
const rooModes = ['architect', 'ask', 'boomerang', 'code', 'debug', 'test'];
|
||||
for (const mode of rooModes) {
|
||||
fs.mkdirSync(path.join(tempDir, '.roo', `rules-${mode}`), {
|
||||
recursive: true
|
||||
});
|
||||
fs.writeFileSync(
|
||||
path.join(tempDir, '.roo', `rules-${mode}`, `${mode}-rules`),
|
||||
`Content for ${mode} rules`
|
||||
);
|
||||
}
|
||||
|
||||
// Create additional directories
|
||||
fs.mkdirSync(path.join(tempDir, '.roo', 'config'), { recursive: true });
|
||||
fs.mkdirSync(path.join(tempDir, '.roo', 'templates'), { recursive: true });
|
||||
fs.mkdirSync(path.join(tempDir, '.roo', 'logs'), { recursive: true });
|
||||
|
||||
// Copy .roomodes file
|
||||
fs.writeFileSync(path.join(tempDir, '.roomodes'), 'Roomodes file content');
|
||||
}
|
||||
|
||||
test('creates all required .roo directories', () => {
|
||||
// Act
|
||||
mockCreateRooStructure();
|
||||
|
||||
// Assert
|
||||
expect(fs.mkdirSync).toHaveBeenCalledWith(path.join(tempDir, '.roo'), {
|
||||
recursive: true
|
||||
});
|
||||
expect(fs.mkdirSync).toHaveBeenCalledWith(
|
||||
path.join(tempDir, '.roo', 'rules'),
|
||||
{ recursive: true }
|
||||
);
|
||||
|
||||
// Verify all mode directories are created
|
||||
expect(fs.mkdirSync).toHaveBeenCalledWith(
|
||||
path.join(tempDir, '.roo', 'rules-architect'),
|
||||
{ recursive: true }
|
||||
);
|
||||
expect(fs.mkdirSync).toHaveBeenCalledWith(
|
||||
path.join(tempDir, '.roo', 'rules-ask'),
|
||||
{ recursive: true }
|
||||
);
|
||||
expect(fs.mkdirSync).toHaveBeenCalledWith(
|
||||
path.join(tempDir, '.roo', 'rules-boomerang'),
|
||||
{ recursive: true }
|
||||
);
|
||||
expect(fs.mkdirSync).toHaveBeenCalledWith(
|
||||
path.join(tempDir, '.roo', 'rules-code'),
|
||||
{ recursive: true }
|
||||
);
|
||||
expect(fs.mkdirSync).toHaveBeenCalledWith(
|
||||
path.join(tempDir, '.roo', 'rules-debug'),
|
||||
{ recursive: true }
|
||||
);
|
||||
expect(fs.mkdirSync).toHaveBeenCalledWith(
|
||||
path.join(tempDir, '.roo', 'rules-test'),
|
||||
{ recursive: true }
|
||||
);
|
||||
});
|
||||
|
||||
test('creates rule files for all modes', () => {
|
||||
// Act
|
||||
mockCreateRooStructure();
|
||||
|
||||
// Assert - check all rule files are created
|
||||
expect(fs.writeFileSync).toHaveBeenCalledWith(
|
||||
path.join(tempDir, '.roo', 'rules-architect', 'architect-rules'),
|
||||
expect.any(String)
|
||||
);
|
||||
expect(fs.writeFileSync).toHaveBeenCalledWith(
|
||||
path.join(tempDir, '.roo', 'rules-ask', 'ask-rules'),
|
||||
expect.any(String)
|
||||
);
|
||||
expect(fs.writeFileSync).toHaveBeenCalledWith(
|
||||
path.join(tempDir, '.roo', 'rules-boomerang', 'boomerang-rules'),
|
||||
expect.any(String)
|
||||
);
|
||||
expect(fs.writeFileSync).toHaveBeenCalledWith(
|
||||
path.join(tempDir, '.roo', 'rules-code', 'code-rules'),
|
||||
expect.any(String)
|
||||
);
|
||||
expect(fs.writeFileSync).toHaveBeenCalledWith(
|
||||
path.join(tempDir, '.roo', 'rules-debug', 'debug-rules'),
|
||||
expect.any(String)
|
||||
);
|
||||
expect(fs.writeFileSync).toHaveBeenCalledWith(
|
||||
path.join(tempDir, '.roo', 'rules-test', 'test-rules'),
|
||||
expect.any(String)
|
||||
);
|
||||
});
|
||||
|
||||
test('creates .roomodes file in project root', () => {
|
||||
// Act
|
||||
mockCreateRooStructure();
|
||||
|
||||
// Assert
|
||||
expect(fs.writeFileSync).toHaveBeenCalledWith(
|
||||
path.join(tempDir, '.roomodes'),
|
||||
expect.any(String)
|
||||
);
|
||||
});
|
||||
|
||||
test('creates additional required Roo directories', () => {
|
||||
// Act
|
||||
mockCreateRooStructure();
|
||||
|
||||
// Assert
|
||||
expect(fs.mkdirSync).toHaveBeenCalledWith(
|
||||
path.join(tempDir, '.roo', 'config'),
|
||||
{ recursive: true }
|
||||
);
|
||||
expect(fs.mkdirSync).toHaveBeenCalledWith(
|
||||
path.join(tempDir, '.roo', 'templates'),
|
||||
{ recursive: true }
|
||||
);
|
||||
expect(fs.mkdirSync).toHaveBeenCalledWith(
|
||||
path.join(tempDir, '.roo', 'logs'),
|
||||
{ recursive: true }
|
||||
);
|
||||
});
|
||||
});
|
||||
113
tests/unit/rule-transformer.test.js
Normal file
113
tests/unit/rule-transformer.test.js
Normal file
@@ -0,0 +1,113 @@
|
||||
import { expect } from 'chai';
|
||||
import fs from 'fs';
|
||||
import path from 'path';
|
||||
import { fileURLToPath } from 'url';
|
||||
import { dirname } from 'path';
|
||||
import { convertCursorRuleToRooRule } from '../modules/rule-transformer.js';
|
||||
|
||||
const __filename = fileURLToPath(import.meta.url);
|
||||
const __dirname = dirname(__filename);
|
||||
|
||||
describe('Rule Transformer', () => {
|
||||
const testDir = path.join(__dirname, 'temp-test-dir');
|
||||
|
||||
before(() => {
|
||||
// Create test directory
|
||||
if (!fs.existsSync(testDir)) {
|
||||
fs.mkdirSync(testDir, { recursive: true });
|
||||
}
|
||||
});
|
||||
|
||||
after(() => {
|
||||
// Clean up test directory
|
||||
if (fs.existsSync(testDir)) {
|
||||
fs.rmSync(testDir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it('should correctly convert basic terms', () => {
|
||||
// Create a test Cursor rule file with basic terms
|
||||
const testCursorRule = path.join(testDir, 'basic-terms.mdc');
|
||||
const testContent = `---
|
||||
description: Test Cursor rule for basic terms
|
||||
globs: **/*
|
||||
alwaysApply: true
|
||||
---
|
||||
|
||||
This is a Cursor rule that references cursor.so and uses the word Cursor multiple times.
|
||||
Also has references to .mdc files.`;
|
||||
|
||||
fs.writeFileSync(testCursorRule, testContent);
|
||||
|
||||
// Convert it
|
||||
const testRooRule = path.join(testDir, 'basic-terms.md');
|
||||
convertCursorRuleToRooRule(testCursorRule, testRooRule);
|
||||
|
||||
// Read the converted file
|
||||
const convertedContent = fs.readFileSync(testRooRule, 'utf8');
|
||||
|
||||
// Verify transformations
|
||||
expect(convertedContent).to.include('Roo Code');
|
||||
expect(convertedContent).to.include('roocode.com');
|
||||
expect(convertedContent).to.include('.md');
|
||||
expect(convertedContent).to.not.include('cursor.so');
|
||||
expect(convertedContent).to.not.include('Cursor rule');
|
||||
});
|
||||
|
||||
it('should correctly convert tool references', () => {
|
||||
// Create a test Cursor rule file with tool references
|
||||
const testCursorRule = path.join(testDir, 'tool-refs.mdc');
|
||||
const testContent = `---
|
||||
description: Test Cursor rule for tool references
|
||||
globs: **/*
|
||||
alwaysApply: true
|
||||
---
|
||||
|
||||
- Use the search tool to find code
|
||||
- The edit_file tool lets you modify files
|
||||
- run_command executes terminal commands
|
||||
- use_mcp connects to external services`;
|
||||
|
||||
fs.writeFileSync(testCursorRule, testContent);
|
||||
|
||||
// Convert it
|
||||
const testRooRule = path.join(testDir, 'tool-refs.md');
|
||||
convertCursorRuleToRooRule(testCursorRule, testRooRule);
|
||||
|
||||
// Read the converted file
|
||||
const convertedContent = fs.readFileSync(testRooRule, 'utf8');
|
||||
|
||||
// Verify transformations
|
||||
expect(convertedContent).to.include('search_files tool');
|
||||
expect(convertedContent).to.include('apply_diff tool');
|
||||
expect(convertedContent).to.include('execute_command');
|
||||
expect(convertedContent).to.include('use_mcp_tool');
|
||||
});
|
||||
|
||||
it('should correctly update file references', () => {
|
||||
// Create a test Cursor rule file with file references
|
||||
const testCursorRule = path.join(testDir, 'file-refs.mdc');
|
||||
const testContent = `---
|
||||
description: Test Cursor rule for file references
|
||||
globs: **/*
|
||||
alwaysApply: true
|
||||
---
|
||||
|
||||
This references [dev_workflow.mdc](mdc:.cursor/rules/dev_workflow.mdc) and
|
||||
[taskmaster.mdc](mdc:.cursor/rules/taskmaster.mdc).`;
|
||||
|
||||
fs.writeFileSync(testCursorRule, testContent);
|
||||
|
||||
// Convert it
|
||||
const testRooRule = path.join(testDir, 'file-refs.md');
|
||||
convertCursorRuleToRooRule(testCursorRule, testRooRule);
|
||||
|
||||
// Read the converted file
|
||||
const convertedContent = fs.readFileSync(testRooRule, 'utf8');
|
||||
|
||||
// Verify transformations
|
||||
expect(convertedContent).to.include('(mdc:.roo/rules/dev_workflow.md)');
|
||||
expect(convertedContent).to.include('(mdc:.roo/rules/taskmaster.md)');
|
||||
expect(convertedContent).to.not.include('(mdc:.cursor/rules/');
|
||||
});
|
||||
});
|
||||
@@ -134,33 +134,59 @@ jest.mock('../../scripts/modules/task-manager.js', () => {
|
||||
});
|
||||
|
||||
// Create a simplified version of parsePRD for testing
|
||||
const testParsePRD = async (prdPath, outputPath, numTasks) => {
|
||||
const testParsePRD = async (prdPath, outputPath, numTasks, options = {}) => {
|
||||
const { append = false } = options;
|
||||
try {
|
||||
// Handle existing tasks when append flag is true
|
||||
let existingTasks = { tasks: [] };
|
||||
let lastTaskId = 0;
|
||||
|
||||
// Check if the output file already exists
|
||||
if (mockExistsSync(outputPath)) {
|
||||
const confirmOverwrite = await mockPromptYesNo(
|
||||
`Warning: ${outputPath} already exists. Overwrite?`,
|
||||
false
|
||||
);
|
||||
if (append) {
|
||||
// Simulate reading existing tasks.json
|
||||
existingTasks = {
|
||||
tasks: [
|
||||
{ id: 1, title: 'Existing Task 1', status: 'done' },
|
||||
{ id: 2, title: 'Existing Task 2', status: 'pending' }
|
||||
]
|
||||
};
|
||||
lastTaskId = 2; // Highest existing ID
|
||||
} else {
|
||||
const confirmOverwrite = await mockPromptYesNo(
|
||||
`Warning: ${outputPath} already exists. Overwrite?`,
|
||||
false
|
||||
);
|
||||
|
||||
if (!confirmOverwrite) {
|
||||
console.log(`Operation cancelled. ${outputPath} was not modified.`);
|
||||
return null;
|
||||
if (!confirmOverwrite) {
|
||||
console.log(`Operation cancelled. ${outputPath} was not modified.`);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const prdContent = mockReadFileSync(prdPath, 'utf8');
|
||||
const tasks = await mockCallClaude(prdContent, prdPath, numTasks);
|
||||
// Modify mockCallClaude to accept lastTaskId parameter
|
||||
let newTasks = await mockCallClaude(prdContent, prdPath, numTasks);
|
||||
|
||||
// Merge tasks if appending
|
||||
const tasksData = append
|
||||
? {
|
||||
...existingTasks,
|
||||
tasks: [...existingTasks.tasks, ...newTasks.tasks]
|
||||
}
|
||||
: newTasks;
|
||||
|
||||
const dir = mockDirname(outputPath);
|
||||
|
||||
if (!mockExistsSync(dir)) {
|
||||
mockMkdirSync(dir, { recursive: true });
|
||||
}
|
||||
|
||||
mockWriteJSON(outputPath, tasks);
|
||||
mockWriteJSON(outputPath, tasksData);
|
||||
await mockGenerateTaskFiles(outputPath, dir);
|
||||
|
||||
return tasks;
|
||||
return tasksData;
|
||||
} catch (error) {
|
||||
console.error(`Error parsing PRD: ${error.message}`);
|
||||
process.exit(1);
|
||||
@@ -628,6 +654,27 @@ describe('Task Manager Module', () => {
|
||||
// Mock the sample PRD content
|
||||
const samplePRDContent = '# Sample PRD for Testing';
|
||||
|
||||
// Mock existing tasks for append test
|
||||
const existingTasks = {
|
||||
tasks: [
|
||||
{ id: 1, title: 'Existing Task 1', status: 'done' },
|
||||
{ id: 2, title: 'Existing Task 2', status: 'pending' }
|
||||
]
|
||||
};
|
||||
|
||||
// Mock new tasks with continuing IDs for append test
|
||||
const newTasksWithContinuedIds = {
|
||||
tasks: [
|
||||
{ id: 3, title: 'New Task 3' },
|
||||
{ id: 4, title: 'New Task 4' }
|
||||
]
|
||||
};
|
||||
|
||||
// Mock merged tasks for append test
|
||||
const mergedTasks = {
|
||||
tasks: [...existingTasks.tasks, ...newTasksWithContinuedIds.tasks]
|
||||
};
|
||||
|
||||
beforeEach(() => {
|
||||
// Reset all mocks
|
||||
jest.clearAllMocks();
|
||||
@@ -811,6 +858,66 @@ describe('Task Manager Module', () => {
|
||||
sampleClaudeResponse
|
||||
);
|
||||
});
|
||||
|
||||
test('should append new tasks when append option is true', async () => {
|
||||
// Setup mocks to simulate tasks.json already exists
|
||||
mockExistsSync.mockImplementation((path) => {
|
||||
if (path === 'tasks/tasks.json') return true; // Output file exists
|
||||
if (path === 'tasks') return true; // Directory exists
|
||||
return false;
|
||||
});
|
||||
|
||||
// Mock for reading existing tasks
|
||||
mockReadJSON.mockReturnValue(existingTasks);
|
||||
// mockReadJSON = jest.fn().mockReturnValue(existingTasks);
|
||||
|
||||
// Mock callClaude to return new tasks with continuing IDs
|
||||
mockCallClaude.mockResolvedValueOnce(newTasksWithContinuedIds);
|
||||
|
||||
// Call the function with append option
|
||||
const result = await testParsePRD(
|
||||
'path/to/prd.txt',
|
||||
'tasks/tasks.json',
|
||||
2,
|
||||
{ append: true }
|
||||
);
|
||||
|
||||
// Verify prompt was NOT called (no confirmation needed for append)
|
||||
expect(mockPromptYesNo).not.toHaveBeenCalled();
|
||||
|
||||
// Verify the file was written with merged tasks
|
||||
expect(mockWriteJSON).toHaveBeenCalledWith(
|
||||
'tasks/tasks.json',
|
||||
expect.objectContaining({
|
||||
tasks: expect.arrayContaining([
|
||||
expect.objectContaining({ id: 1 }),
|
||||
expect.objectContaining({ id: 2 }),
|
||||
expect.objectContaining({ id: 3 }),
|
||||
expect.objectContaining({ id: 4 })
|
||||
])
|
||||
})
|
||||
);
|
||||
|
||||
// Verify the result contains merged tasks
|
||||
expect(result.tasks.length).toBe(4);
|
||||
});
|
||||
|
||||
test('should skip prompt and not overwrite when append is true', async () => {
|
||||
// Setup mocks to simulate tasks.json already exists
|
||||
mockExistsSync.mockImplementation((path) => {
|
||||
if (path === 'tasks/tasks.json') return true; // Output file exists
|
||||
if (path === 'tasks') return true; // Directory exists
|
||||
return false;
|
||||
});
|
||||
|
||||
// Call the function with append option
|
||||
await testParsePRD('path/to/prd.txt', 'tasks/tasks.json', 3, {
|
||||
append: true
|
||||
});
|
||||
|
||||
// Verify prompt was NOT called with append flag
|
||||
expect(mockPromptYesNo).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
|
||||
describe.skip('updateTasks function', () => {
|
||||
|
||||
Reference in New Issue
Block a user