fix: enhance task expansion with multiple improvements

This commit resolves several issues with the task expansion system to
ensure higher quality subtasks and better synchronization:

1. Task File Generation
- Add automatic regeneration of task files after expanding tasks
- Ensure individual task text files stay in sync with tasks.json
- Avoids manual regeneration steps after task expansion

2. Perplexity API Integration
- Fix 'researchPrompt is not defined' error in Perplexity integration
- Add specialized research-oriented prompt template
- Improve system message for better context and instruction
- Better fallback to Claude when Perplexity unavailable

3. Subtask Parsing Improvements
- Enhance regex pattern to handle more formatting variations
- Implement multiple parsing strategies for different response formats:
  * Improved section detection with flexible headings
  * Added support for numbered and bulleted lists
  * Implemented heuristic-based title and description extraction
- Create more meaningful dummy subtasks with relevant titles and descriptions
  instead of generic placeholders
- Ensure minimal descriptions are always provided

4. Quality Verification and Retry System
- Add post-expansion verification to identify low-quality subtask sets
- Detect tasks with too many generic/placeholder subtasks
- Implement interactive retry mechanism with enhanced prompts
- Use adjusted settings for retries (research mode, subtask count)
- Clear existing subtasks before retry to prevent duplicates
- Provide detailed reporting of verification and retry process

These changes significantly improve the quality of generated subtasks
and reduce the need for manual intervention when subtask generation
produces suboptimal results.

README-task-master.md

@@ -361,8 +361,80 @@ Please mark it as complete and tell me what I should work on next.
 
 ## Documentation
 
-For more detailed documentation on the scripts, see the [scripts/README.md](scripts/README.md) file in your initialized project.
+For more detailed documentation on the scripts and command-line options, see the [scripts/README.md](scripts/README.md) file in your initialized project.
 
 ## License
 
-MIT 
+MIT 
+
+### Analyzing Task Complexity
+
+To analyze the complexity of tasks and automatically generate expansion recommendations:
+
+```bash
+npm run dev -- analyze-complexity
+```
+
+This command:
+- Analyzes each task using AI to assess its complexity
+- Recommends optimal number of subtasks based on configured DEFAULT_SUBTASKS
+- Generates tailored prompts for expanding each task
+- Creates a comprehensive JSON report with ready-to-use commands
+- Saves the report to scripts/task-complexity-report.json by default
+
+Options:
+```bash
+# Save report to a custom location
+npm run dev -- analyze-complexity --output=my-report.json
+
+# Use a specific LLM model
+npm run dev -- analyze-complexity --model=claude-3-opus-20240229
+
+# Set a custom complexity threshold (1-10)
+npm run dev -- analyze-complexity --threshold=6
+
+# Use an alternative tasks file
+npm run dev -- analyze-complexity --file=custom-tasks.json
+
+# Use Perplexity AI for research-backed complexity analysis
+npm run dev -- analyze-complexity --research
+```
+
+The generated report contains:
+- Complexity analysis for each task (scored 1-10)
+- Recommended number of subtasks based on complexity
+- AI-generated expansion prompts customized for each task
+- Ready-to-run expansion commands directly within each task analysis
+
+### Smart Task Expansion
+
+The `expand` command now automatically checks for and uses the complexity report:
+
+```bash
+# Expand a task, using complexity report recommendations if available
+npm run dev -- expand --id=8
+
+# Expand all tasks, prioritizing by complexity score if a report exists
+npm run dev -- expand --all
+```
+
+When a complexity report exists:
+- Tasks are automatically expanded using the recommended subtask count and prompts
+- When expanding all tasks, they're processed in order of complexity (highest first)
+- Research-backed generation is preserved from the complexity analysis
+- You can still override recommendations with explicit command-line options
+
+Example workflow:
+```bash
+# Generate the complexity analysis report with research capabilities
+npm run dev -- analyze-complexity --research
+
+# Review the report in scripts/task-complexity-report.json
+
+# Expand tasks using the optimized recommendations
+npm run dev -- expand --id=8
+# or expand all tasks
+npm run dev -- expand --all
+```
+
+This integration ensures that task expansion is informed by thorough complexity analysis, resulting in better subtask organization and more efficient development.