fix: enhance task expansion with multiple improvements

This commit resolves several issues with the task expansion system to
ensure higher quality subtasks and better synchronization:

1. Task File Generation
- Add automatic regeneration of task files after expanding tasks
- Ensure individual task text files stay in sync with tasks.json
- Avoids manual regeneration steps after task expansion

2. Perplexity API Integration
- Fix 'researchPrompt is not defined' error in Perplexity integration
- Add specialized research-oriented prompt template
- Improve system message for better context and instruction
- Better fallback to Claude when Perplexity unavailable

3. Subtask Parsing Improvements
- Enhance regex pattern to handle more formatting variations
- Implement multiple parsing strategies for different response formats:
  * Improved section detection with flexible headings
  * Added support for numbered and bulleted lists
  * Implemented heuristic-based title and description extraction
- Create more meaningful dummy subtasks with relevant titles and descriptions
  instead of generic placeholders
- Ensure minimal descriptions are always provided

4. Quality Verification and Retry System
- Add post-expansion verification to identify low-quality subtask sets
- Detect tasks with too many generic/placeholder subtasks
- Implement interactive retry mechanism with enhanced prompts
- Use adjusted settings for retries (research mode, subtask count)
- Clear existing subtasks before retry to prevent duplicates
- Provide detailed reporting of verification and retry process

These changes significantly improve the quality of generated subtasks
and reduce the need for manual intervention when subtask generation
produces suboptimal results.

scripts/README.md

@@ -178,4 +178,80 @@ The script supports different logging levels controlled by the `LOG_LEVEL` envir
 - `warn`: Warning messages that don't prevent execution
 - `error`: Error messages that might prevent execution
 
-When `DEBUG=true` is set, debug logs are also written to a `dev-debug.log` file in the project root.
+When `DEBUG=true` is set, debug logs are also written to a `dev-debug.log` file in the project root.
+
+## Analyzing Task Complexity
+
+The `analyze-complexity` command allows you to automatically assess task complexity and generate expansion recommendations:
+
+```bash
+# Analyze all tasks and generate expansion recommendations
+node scripts/dev.js analyze-complexity
+
+# Specify a custom output file
+node scripts/dev.js analyze-complexity --output=custom-report.json
+
+# Override the model used for analysis
+node scripts/dev.js analyze-complexity --model=claude-3-opus-20240229
+
+# Set a custom complexity threshold (1-10)
+node scripts/dev.js analyze-complexity --threshold=6
+
+# Use Perplexity AI for research-backed complexity analysis
+node scripts/dev.js analyze-complexity --research
+```
+
+Notes:
+- The command uses Claude to analyze each task's complexity (or Perplexity with --research flag)
+- Tasks are scored on a scale of 1-10
+- Each task receives a recommended number of subtasks based on DEFAULT_SUBTASKS configuration
+- The default output path is `scripts/task-complexity-report.json`
+- Each task in the analysis includes a ready-to-use `expansionCommand` that can be copied directly to the terminal or executed programmatically
+- Tasks with complexity scores below the threshold (default: 5) may not need expansion
+- The research flag provides more contextual and informed complexity assessments
+
+### Integration with Expand Command
+
+The `expand` command automatically checks for and uses complexity analysis if available:
+
+```bash
+# Expand a task, using complexity report recommendations if available
+node scripts/dev.js expand --id=8
+
+# Expand all tasks, prioritizing by complexity score if a report exists
+node scripts/dev.js expand --all
+
+# Override recommendations with explicit values
+node scripts/dev.js expand --id=8 --num=5 --prompt="Custom prompt"
+```
+
+When a complexity report exists:
+- The `expand` command will use the recommended subtask count from the report (unless overridden)
+- It will use the tailored expansion prompt from the report (unless a custom prompt is provided)
+- When using `--all`, tasks are sorted by complexity score (highest first)
+- The `--research` flag is preserved from the complexity analysis to expansion
+
+The output report structure is:
+```json
+{
+  "meta": {
+    "generatedAt": "2023-06-15T12:34:56.789Z",
+    "tasksAnalyzed": 20,
+    "thresholdScore": 5,
+    "projectName": "Your Project Name",
+    "usedResearch": true
+  },
+  "complexityAnalysis": [
+    {
+      "taskId": 8,
+      "taskTitle": "Develop Implementation Drift Handling",
+      "complexityScore": 9.5,
+      "recommendedSubtasks": 6,
+      "expansionPrompt": "Create subtasks that handle detecting...",
+      "reasoning": "This task requires sophisticated logic...",
+      "expansionCommand": "node scripts/dev.js expand --id=8 --num=6 --prompt=\"Create subtasks...\" --research"
+    },
+    // More tasks sorted by complexity score (highest first)
+  ]
+}
+```