BMAD-METHOD/issues-report.md

BMAD Agent PM.md - LLM Compatibility Issues Report

Date: 2025-10-02 Analyzed File: /z9/bmad/bmm/agents/pm.md Purpose: Deep analysis of potential LLM interpretation issues

---

Executive Summary

Analysis of the Product Manager agent reveals 10 distinct issues ranging from critical functionality breaks to documentation inconsistencies. The most severe issues involve undefined variable references, handler mismatches, and ambiguous execution instructions that could cause LLMs to behave unpredictably.

---

CRITICAL Issues (Breaks Functionality)

Issue #8: Validate-Workflow Handler Mismatch

Severity: 🔴 CRITICAL Location: Lines 27-33 vs Line 73

Problem:

<!-- Handler expects this attribute: -->
<handler type="validate-workflow">
  When command has: validate-workflow="path/to/workflow.yaml"
  ...
</handler>

<!-- But actual command uses this: -->
<c cmd="*validate" exec="{project-root}/bmad/core/tasks/validate-workflow.md">

The handler is configured to trigger on validate-workflow attribute, but no command in the agent actually uses that attribute. The *validate command uses exec instead.

Impact: Handler will never execute; validation functionality is broken or unclear.

Suggested Fix:

<!-- Option 1: Change command to match handler -->
<c cmd="*validate" validate-workflow="{output_folder}/document.md">
  Validate any document against its workflow checklist
</c>

<!-- Option 2: Change handler to match command -->
<handler type="exec">
  When command has: exec="{project-root}/bmad/core/tasks/validate-workflow.md"
  Load and execute the validation task file
  Prompt user for document path if not provided
</handler>

---

Issue #5: Undefined Fuzzy Match Algorithm

Severity: 🔴 CRITICAL Location: Line 16

Problem:

<input>Number → cmd[n] | Text → fuzzy match *commands</input>

"Fuzzy match" is completely undefined. Different LLMs may implement:

• Exact substring matching
• Levenshtein distance
• Semantic similarity
• Partial token matching
• Case-sensitive vs case-insensitive

Impact: Inconsistent command matching behavior across different LLM providers. User experience varies significantly.

Suggested Fix:

<input>
  Number → Execute cmd[n] directly
  Text → Case-insensitive substring match against command triggers
  If multiple matches found, list matches and ask user to clarify
  If no matches found, show "Command not recognized. Type number or *help"
</input>

---

HIGH Priority Issues (Causes Confusion)

Issue #1: Activation Step 5 Ambiguity

Severity: 🟠 HIGH Location: Line 13

Problem:

<step n="5">CRITICAL HALT. AWAIT user input. NEVER continue without it.</step>

"CRITICAL HALT" and "NEVER continue" are extreme imperatives that may confuse LLMs designed to be responsive. Some LLMs may:

• Ignore the halt entirely
• Become stuck in a loop
• Over-apply the restriction to subsequent interactions
• Not understand this specifically means "wait for command selection"

Impact: Agent may continue without waiting, or may become unresponsive.

Suggested Fix:

<step n="5">
  STOP. Display the menu and WAIT for user to select a command.
  Accept either: (a) number corresponding to command index, or (b) command text for fuzzy matching.
  DO NOT proceed until user provides input. DO NOT improvise or suggest actions.
</step>

---

Issue #3: Override Path Variable Undefined

Severity: 🟠 HIGH Location: Line 10

Problem:

<step n="2">Override with {project-root}/bmad/_cfg/agents/{agent-filename} if exists (replace, not merge)</step>

Issues:

• {agent-filename} is never defined (should it be pm, pm.md, or something else?)
• Doesn't specify WHICH sections get replaced
• No explicit handling for "file doesn't exist" case

Impact: LLM may construct wrong path, may not find config, or may merge instead of replace.

Suggested Fix:

<step n="2">
  Check if agent config exists at: {project-root}/bmad/_cfg/agents/pm.md
  If config file exists:
    - REPLACE entire persona section with config persona (do NOT merge fields)
    - Config persona overrides completely; ignore original persona
  If config file does NOT exist:
    - Continue with persona from current agent file
    - No error needed; this is expected behavior
</step>

---

Issue #6: Variable Storage Mechanism Unclear

Severity: 🟠 HIGH Location: Line 65

Problem:

<i>Load into memory {project-root}/bmad/bmm/config.yaml and set variable project_name, output_folder, user_name, communication_language</i>

Ambiguities:

• "set variable" - where are these stored?
• How are they accessed later (as {project_name} or $project_name or something else?)
• What's the scope (session? agent-only? global?)
• What if config.yaml doesn't exist or is malformed?

Impact: Variables might not be properly initialized or accessible in subsequent operations.

Suggested Fix:

<i>
  Load configuration from {project-root}/bmad/bmm/config.yaml
  Parse YAML and extract these fields: project_name, output_folder, user_name, communication_language
  Store these in persistent session memory with syntax: {project_name}, {output_folder}, {user_name}, {communication_language}
  These variables remain accessible throughout the entire agent session
  If any field is missing from config, prompt user to provide it
</i>

---

MEDIUM Priority Issues (Best Practices)

Issue #4: Handler Priority Order Not Specified

Severity: 🟡 MEDIUM Location: Lines 18-50

Problem: Multiple command handlers are defined, but there's no explicit priority when a command has multiple attributes simultaneously (e.g., exec + tmpl + data).

Impact: LLM might execute handlers in wrong order, skip handlers, or be confused about precedence.

Suggested Fix:

<handlers>
  <!-- PROCESSING ORDER: Execute in this sequence when multiple attributes present -->
  <!-- 1. data (load supplementary data first) -->
  <!-- 2. tmpl (load template second) -->
  <!-- 3. action/exec/run-workflow/validate-workflow (execute main handler last) -->

  <handler type="data" priority="1">
    When command has: data="path/to/file.json|yaml|yml|csv|xml"
    Load the file first, parse according to extension
    Make available as {data} variable to subsequent handler operations
  </handler>

  <handler type="tmpl" priority="2">
    When command has: tmpl="path/to/template.md"
    Load template file, parse {{mustache}} style variables
    Make available as {template} to action/exec/workflow handlers
  </handler>

  <handler type="run-workflow" priority="3">
    <!-- existing run-workflow handler -->
  </handler>

  <!-- etc -->
</handlers>

---

Issue #7: Workflow Handler Typo and Rigidity

Severity: 🟡 MEDIUM Location: Lines 20-24

Problem:

2. READ its entire contents - the is the CORE OS for EXECUTING modules

Typo: "the is" should be "this is"

Also: "Follow workflow.md instructions EXACTLY as written" may be too rigid and conflict with need for judgment calls.

Impact:

• Typo could confuse some LLMs
• "EXACTLY" might prevent necessary adaptations

Suggested Fix:

<handler type="run-workflow">
  When command has run-workflow="path/to/workflow.yaml":
  1. Load the workflow execution engine: {project-root}/bmad/core/tasks/workflow.md
  2. Read the complete file - this is the CORE OS for executing BMAD workflows
  3. Pass the workflow.yaml path as the 'workflow-config' parameter
  4. Execute workflow.md instructions precisely following all steps
  5. Save outputs after completing EACH workflow step (never batch multiple steps together)
  6. If workflow.yaml path is "todo", inform user the workflow hasn't been implemented yet
</handler>

---

Issue #10: Persona Principles Formatting Inconsistency

Severity: 🟡 MEDIUM Location: Line 62

Problem: Architecture documentation specifies "2-5 sentences" for principles (updated guidance), but the actual principles section is ONE very long run-on sentence:

<principles>I operate with an investigative mindset that seeks to uncover the deeper "why" behind every requirement while maintaining relentless focus on delivering value to target users. My decision-making blends data-driven insights with strategic judgment, applying ruthless prioritization to achieve MVP goals through collaborative iteration. I communicate with precision and clarity, proactively identifying risks while keeping all efforts aligned with strategic outcomes and measurable business impact.</principles>

Impact:

• Harder to parse individual principles
• Violates stated guidelines
• Reduces readability
• LLMs copying this pattern may create similarly compressed principles

Suggested Fix:

<principles>
I operate with an investigative mindset that seeks to uncover the deeper "why" behind every requirement.
I maintain relentless focus on delivering value to target users through data-driven insights and strategic judgment.
I apply ruthless prioritization to achieve MVP goals through collaborative iteration.
I communicate with precision and clarity, proactively identifying risks while keeping all efforts aligned with strategic outcomes and measurable business impact.
</principles>

---

LOW Priority Issues (Documentation/Polish)

Issue #2: Self-Referential Verbosity

Severity: 🟢 LOW Location: Line 9

Problem:

<step n="1">Load persona from this current file containing this activation you are reading now</step>

"this current file containing this activation you are reading now" is unnecessarily verbose and potentially confusing.

Impact: Minor - may cause some LLMs to attempt redundant file loading operations.

Suggested Fix:

<step n="1">Load the persona section from the current agent file (already loaded in context)</step>

---

Issue #9: Unused Action Handler Pattern

Severity: 🟢 LOW Location: Lines 34-37

Problem:

<handler type="action">
  When command has: action="#id" → Find prompt with id="id" in current agent XML
  When command has: action="text" → Execute the text directly as a critical action prompt
</handler>

This agent has NO <prompts> section and no commands using action="#id" pattern. The handler documents a pattern that isn't used in this agent.

Impact: Minor confusion - documentation exists for unused functionality.

Suggested Fix:

Option 1 - Remove unused documentation:

<handler type="action">
  When command has: action="text"
  Execute the text directly as an inline instruction
</handler>

Option 2 - Keep for reference but add comment:

<handler type="action">
  <!-- Pattern: action="#id" finds <prompt id="id"> in agent (not used in this agent) -->
  When command has: action="text" → Execute the text directly as an inline instruction
</handler>

---

Additional Observations

Positive Patterns

✅ Persona uses first-person voice consistently ✅ Critical actions properly load config ✅ Commands follow standard *help, *exit pattern ✅ Icon and basic metadata well-formed

Missing Elements

• No <prompts> section (which is fine for this agent type)
• No yolo mode toggle (optional but common)
• No validation of whether workflow paths actually exist

---

Recommendations Priority

Immediate Action Required

1. Fix Issue #8: Resolve validate-workflow handler mismatch
2. Fix Issue #5: Define explicit fuzzy match algorithm
3. Fix Issue #3: Define {agent-filename} variable

High Priority

4. Fix Issue #1: Clarify HALT instruction
5. Fix Issue #6: Document variable storage mechanism

Medium Priority

6. Fix Issue #4: Add handler priority order
7. Fix Issue #7: Fix typo and clarify "EXACTLY"
8. Fix Issue #10: Break principles into proper sentences

Low Priority (Polish)

9. Fix Issue #2: Simplify self-referential language
10. Fix Issue #9: Clean up unused handler documentation

---

Testing Recommendations

After implementing fixes, test with multiple LLM providers:

• Claude (Anthropic)
• GPT-4 (OpenAI)
• Gemini (Google)
• Llama (Meta)

Verify:

• Agent loads without errors
• Menu displays correctly
• Agent waits for input after menu
• Command matching works consistently
• Config override works (test with and without config file)
• Variables load and are accessible
• Workflow execution triggers correctly
• Validation command works as expected

---

Report Generated: 2025-10-02 Analyst: Claude (Sonnet 4.5) Confidence: High (based on deep structural analysis)