agent updates

2025-10-02 21:45:59 -05:00
parent c6704b4b6e
commit 3f40ef4756
69 changed files with 2596 additions and 55160 deletions
--- a/issues-report.md
+++ b/issues-report.md
@@ -0,0 +1,437 @@
+# BMAD Agent PM.md - LLM Compatibility Issues Report
+
+**Date**: 2025-10-02
+**Analyzed File**: `/z9/bmad/bmm/agents/pm.md`
+**Purpose**: Deep analysis of potential LLM interpretation issues
+
+---
+
+## Executive Summary
+
+Analysis of the Product Manager agent reveals **10 distinct issues** ranging from critical functionality breaks to documentation inconsistencies. The most severe issues involve undefined variable references, handler mismatches, and ambiguous execution instructions that could cause LLMs to behave unpredictably.
+
+---
+
+## CRITICAL Issues (Breaks Functionality)
+
+### Issue #8: Validate-Workflow Handler Mismatch
+
+**Severity**: 🔴 CRITICAL
+**Location**: Lines 27-33 vs Line 73
+
+**Problem**:
+
+```xml
+<!-- Handler expects this attribute: -->
+<handler type="validate-workflow">
+  When command has: validate-workflow="path/to/workflow.yaml"
+  ...
+</handler>
+
+<!-- But actual command uses this: -->
+<c cmd="*validate" exec="{project-root}/bmad/core/tasks/validate-workflow.md">
+```
+
+The handler is configured to trigger on `validate-workflow` attribute, but no command in the agent actually uses that attribute. The `*validate` command uses `exec` instead.
+
+**Impact**: Handler will never execute; validation functionality is broken or unclear.
+
+**Suggested Fix**:
+
+```xml
+<!-- Option 1: Change command to match handler -->
+<c cmd="*validate" validate-workflow="{output_folder}/document.md">
+  Validate any document against its workflow checklist
+</c>
+
+<!-- Option 2: Change handler to match command -->
+<handler type="exec">
+  When command has: exec="{project-root}/bmad/core/tasks/validate-workflow.md"
+  Load and execute the validation task file
+  Prompt user for document path if not provided
+</handler>
+```
+
+---
+
+### Issue #5: Undefined Fuzzy Match Algorithm
+
+**Severity**: 🔴 CRITICAL
+**Location**: Line 16
+
+**Problem**:
+
+```xml
+<input>Number → cmd[n] | Text → fuzzy match *commands</input>
+```
+
+"Fuzzy match" is completely undefined. Different LLMs may implement:
+
+- Exact substring matching
+- Levenshtein distance
+- Semantic similarity
+- Partial token matching
+- Case-sensitive vs case-insensitive
+
+**Impact**: Inconsistent command matching behavior across different LLM providers. User experience varies significantly.
+
+**Suggested Fix**:
+
+```xml
+<input>
+  Number → Execute cmd[n] directly
+  Text → Case-insensitive substring match against command triggers
+  If multiple matches found, list matches and ask user to clarify
+  If no matches found, show "Command not recognized. Type number or *help"
+</input>
+```
+
+---
+
+## HIGH Priority Issues (Causes Confusion)
+
+### Issue #1: Activation Step 5 Ambiguity
+
+**Severity**: 🟠 HIGH
+**Location**: Line 13
+
+**Problem**:
+
+```xml
+<step n="5">CRITICAL HALT. AWAIT user input. NEVER continue without it.</step>
+```
+
+"CRITICAL HALT" and "NEVER continue" are extreme imperatives that may confuse LLMs designed to be responsive. Some LLMs may:
+
+- Ignore the halt entirely
+- Become stuck in a loop
+- Over-apply the restriction to subsequent interactions
+- Not understand this specifically means "wait for command selection"
+
+**Impact**: Agent may continue without waiting, or may become unresponsive.
+
+**Suggested Fix**:
+
+```xml
+<step n="5">
+  STOP. Display the menu and WAIT for user to select a command.
+  Accept either: (a) number corresponding to command index, or (b) command text for fuzzy matching.
+  DO NOT proceed until user provides input. DO NOT improvise or suggest actions.
+</step>
+```
+
+---
+
+### Issue #3: Override Path Variable Undefined
+
+**Severity**: 🟠 HIGH
+**Location**: Line 10
+
+**Problem**:
+
+```xml
+<step n="2">Override with {project-root}/bmad/_cfg/agents/{agent-filename} if exists (replace, not merge)</step>
+```
+
+Issues:
+
+- `{agent-filename}` is never defined (should it be `pm`, `pm.md`, or something else?)
+- Doesn't specify WHICH sections get replaced
+- No explicit handling for "file doesn't exist" case
+
+**Impact**: LLM may construct wrong path, may not find config, or may merge instead of replace.
+
+**Suggested Fix**:
+
+```xml
+<step n="2">
+  Check if agent config exists at: {project-root}/bmad/_cfg/agents/pm.md
+  If config file exists:
+    - REPLACE entire persona section with config persona (do NOT merge fields)
+    - Config persona overrides completely; ignore original persona
+  If config file does NOT exist:
+    - Continue with persona from current agent file
+    - No error needed; this is expected behavior
+</step>
+```
+
+---
+
+### Issue #6: Variable Storage Mechanism Unclear
+
+**Severity**: 🟠 HIGH
+**Location**: Line 65
+
+**Problem**:
+
+```xml
+<i>Load into memory {project-root}/bmad/bmm/config.yaml and set variable project_name, output_folder, user_name, communication_language</i>
+```
+
+Ambiguities:
+
+- "set variable" - where are these stored?
+- How are they accessed later (as `{project_name}` or `$project_name` or something else?)
+- What's the scope (session? agent-only? global?)
+- What if config.yaml doesn't exist or is malformed?
+
+**Impact**: Variables might not be properly initialized or accessible in subsequent operations.
+
+**Suggested Fix**:
+
+```xml
+<i>
+  Load configuration from {project-root}/bmad/bmm/config.yaml
+  Parse YAML and extract these fields: project_name, output_folder, user_name, communication_language
+  Store these in persistent session memory with syntax: {project_name}, {output_folder}, {user_name}, {communication_language}
+  These variables remain accessible throughout the entire agent session
+  If any field is missing from config, prompt user to provide it
+</i>
+```
+
+---
+
+## MEDIUM Priority Issues (Best Practices)
+
+### Issue #4: Handler Priority Order Not Specified
+
+**Severity**: 🟡 MEDIUM
+**Location**: Lines 18-50
+
+**Problem**:
+Multiple command handlers are defined, but there's no explicit priority when a command has multiple attributes simultaneously (e.g., `exec` + `tmpl` + `data`).
+
+**Impact**: LLM might execute handlers in wrong order, skip handlers, or be confused about precedence.
+
+**Suggested Fix**:
+
+```xml
+<handlers>
+  <!-- PROCESSING ORDER: Execute in this sequence when multiple attributes present -->
+  <!-- 1. data (load supplementary data first) -->
+  <!-- 2. tmpl (load template second) -->
+  <!-- 3. action/exec/run-workflow/validate-workflow (execute main handler last) -->
+
+  <handler type="data" priority="1">
+    When command has: data="path/to/file.json|yaml|yml|csv|xml"
+    Load the file first, parse according to extension
+    Make available as {data} variable to subsequent handler operations
+  </handler>
+
+  <handler type="tmpl" priority="2">
+    When command has: tmpl="path/to/template.md"
+    Load template file, parse {{mustache}} style variables
+    Make available as {template} to action/exec/workflow handlers
+  </handler>
+
+  <handler type="run-workflow" priority="3">
+    <!-- existing run-workflow handler -->
+  </handler>
+
+  <!-- etc -->
+</handlers>
+```
+
+---
+
+### Issue #7: Workflow Handler Typo and Rigidity
+
+**Severity**: 🟡 MEDIUM
+**Location**: Lines 20-24
+
+**Problem**:
+
+```xml
+2. READ its entire contents - the is the CORE OS for EXECUTING modules
+```
+
+Typo: "the is" should be "this is"
+
+Also: "Follow workflow.md instructions EXACTLY as written" may be too rigid and conflict with need for judgment calls.
+
+**Impact**:
+
+- Typo could confuse some LLMs
+- "EXACTLY" might prevent necessary adaptations
+
+**Suggested Fix**:
+
+```xml
+<handler type="run-workflow">
+  When command has run-workflow="path/to/workflow.yaml":
+  1. Load the workflow execution engine: {project-root}/bmad/core/tasks/workflow.md
+  2. Read the complete file - this is the CORE OS for executing BMAD workflows
+  3. Pass the workflow.yaml path as the 'workflow-config' parameter
+  4. Execute workflow.md instructions precisely following all steps
+  5. Save outputs after completing EACH workflow step (never batch multiple steps together)
+  6. If workflow.yaml path is "todo", inform user the workflow hasn't been implemented yet
+</handler>
+```
+
+---
+
+### Issue #10: Persona Principles Formatting Inconsistency
+
+**Severity**: 🟡 MEDIUM
+**Location**: Line 62
+
+**Problem**:
+Architecture documentation specifies "2-5 sentences" for principles (updated guidance), but the actual principles section is ONE very long run-on sentence:
+
+```xml
+<principles>I operate with an investigative mindset that seeks to uncover the deeper "why" behind every requirement while maintaining relentless focus on delivering value to target users. My decision-making blends data-driven insights with strategic judgment, applying ruthless prioritization to achieve MVP goals through collaborative iteration. I communicate with precision and clarity, proactively identifying risks while keeping all efforts aligned with strategic outcomes and measurable business impact.</principles>
+```
+
+**Impact**:
+
+- Harder to parse individual principles
+- Violates stated guidelines
+- Reduces readability
+- LLMs copying this pattern may create similarly compressed principles
+
+**Suggested Fix**:
+
+```xml
+<principles>
+I operate with an investigative mindset that seeks to uncover the deeper "why" behind every requirement.
+I maintain relentless focus on delivering value to target users through data-driven insights and strategic judgment.
+I apply ruthless prioritization to achieve MVP goals through collaborative iteration.
+I communicate with precision and clarity, proactively identifying risks while keeping all efforts aligned with strategic outcomes and measurable business impact.
+</principles>
+```
+
+---
+
+## LOW Priority Issues (Documentation/Polish)
+
+### Issue #2: Self-Referential Verbosity
+
+**Severity**: 🟢 LOW
+**Location**: Line 9
+
+**Problem**:
+
+```xml
+<step n="1">Load persona from this current file containing this activation you are reading now</step>
+```
+
+"this current file containing this activation you are reading now" is unnecessarily verbose and potentially confusing.
+
+**Impact**: Minor - may cause some LLMs to attempt redundant file loading operations.
+
+**Suggested Fix**:
+
+```xml
+<step n="1">Load the persona section from the current agent file (already loaded in context)</step>
+```
+
+---
+
+### Issue #9: Unused Action Handler Pattern
+
+**Severity**: 🟢 LOW
+**Location**: Lines 34-37
+
+**Problem**:
+
+```xml
+<handler type="action">
+  When command has: action="#id" → Find prompt with id="id" in current agent XML
+  When command has: action="text" → Execute the text directly as a critical action prompt
+</handler>
+```
+
+This agent has NO `<prompts>` section and no commands using `action="#id"` pattern. The handler documents a pattern that isn't used in this agent.
+
+**Impact**: Minor confusion - documentation exists for unused functionality.
+
+**Suggested Fix**:
+
+**Option 1** - Remove unused documentation:
+
+```xml
+<handler type="action">
+  When command has: action="text"
+  Execute the text directly as an inline instruction
+</handler>
+```
+
+**Option 2** - Keep for reference but add comment:
+
+```xml
+<handler type="action">
+  <!-- Pattern: action="#id" finds <prompt id="id"> in agent (not used in this agent) -->
+  When command has: action="text" → Execute the text directly as an inline instruction
+</handler>
+```
+
+---
+
+## Additional Observations
+
+### Positive Patterns
+
+✅ Persona uses first-person voice consistently
+✅ Critical actions properly load config
+✅ Commands follow standard *help, *exit pattern
+✅ Icon and basic metadata well-formed
+
+### Missing Elements
+
+- No `<prompts>` section (which is fine for this agent type)
+- No `yolo` mode toggle (optional but common)
+- No validation of whether workflow paths actually exist
+
+---
+
+## Recommendations Priority
+
+### Immediate Action Required
+
+1. **Fix Issue #8**: Resolve validate-workflow handler mismatch
+2. **Fix Issue #5**: Define explicit fuzzy match algorithm
+3. **Fix Issue #3**: Define {agent-filename} variable
+
+### High Priority
+
+4. **Fix Issue #1**: Clarify HALT instruction
+5. **Fix Issue #6**: Document variable storage mechanism
+
+### Medium Priority
+
+6. **Fix Issue #4**: Add handler priority order
+7. **Fix Issue #7**: Fix typo and clarify "EXACTLY"
+8. **Fix Issue #10**: Break principles into proper sentences
+
+### Low Priority (Polish)
+
+9. **Fix Issue #2**: Simplify self-referential language
+10. **Fix Issue #9**: Clean up unused handler documentation
+
+---
+
+## Testing Recommendations
+
+After implementing fixes, test with multiple LLM providers:
+
+- Claude (Anthropic)
+- GPT-4 (OpenAI)
+- Gemini (Google)
+- Llama (Meta)
+
+Verify:
+
+- [ ] Agent loads without errors
+- [ ] Menu displays correctly
+- [ ] Agent waits for input after menu
+- [ ] Command matching works consistently
+- [ ] Config override works (test with and without config file)
+- [ ] Variables load and are accessible
+- [ ] Workflow execution triggers correctly
+- [ ] Validation command works as expected
+
+---
+
+**Report Generated**: 2025-10-02
+**Analyst**: Claude (Sonnet 4.5)
+**Confidence**: High (based on deep structural analysis)