ros/BMAD-METHOD
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
v6-alpha-testarch
BMAD-METHOD/docs/testarch-file-review/audit-report-testarch-trace-2025-10-16.md
Brian Madison fbbda0067a testarch review
2025-10-16 18:58:44 -05:00

18 KiB
Raw Permalink Blame History

Workflow Audit Report

Workflow: testarch-trace Audit Date: 2025-10-16 Auditor: Audit Workflow (BMAD v6) Workflow Type: Document workflow (template-based) Workflow Path: /Users/brianmadison/dev/BMAD-METHOD/src/modules/bmm/workflows/testarch/trace

Executive Summary

Overall Status: ⚠️ CONCERNS - Significant bloat detected, missing web_bundle configuration

  • Critical Issues: 1
  • Important Issues: 2
  • Cleanup Recommendations: 8

Key Findings:

  1. ✅ Standard config block is correctly configured
  2. ⚠️ Massive bloat detected - 52 variables defined in workflow.yaml with extensive unused fields
  3. ❌ No web_bundle configuration (critical for web deployment)
  4. ⚠️ Config variable usage needs improvement in instructions

1. Standard Config Block Validation

Required Variables Check

✅ Config Source Check:

  • config_source is defined: "{project-root}/bmad/bmm/config.yaml"
  • Points to correct module config path (bmm)
  • Uses {project-root} variable

✅ Standard Variables Check:

  • output_folder pulls from config_source: "{config_source}:output_folder"
  • user_name pulls from config_source: "{config_source}:user_name"
  • communication_language pulls from config_source: "{config_source}:communication_language"
  • date is set to system-generated: "system-generated"

Status: ✅ PASS - All standard config variables present and correctly configured

2. YAML/Instruction/Template Alignment

Variables Analysis

Total YAML fields analyzed: 52 variables defined in workflow.yaml (excluding standard config block and metadata)

Categorization:

INSTRUCTION_USED (Variables referenced in instructions.md):

The following variables appear to be contextually referenced in the instructions but NOT with explicit {variable_name} syntax:

  • story_file (mentioned conceptually in "Read story file")
  • test_dir (mentioned as test directory)
  • source_dir (mentioned as source directory)
  • coverage_levels (conceptually referenced in coverage analysis)
  • output_file (used in deliverables)
  • gate_type (used in Phase 2)
  • decision_mode (used in decision rules)
  • test_results (required for Phase 2)
  • nfr_file (optional NFR loading)
  • Various threshold values (min_p0_coverage, etc.) used in decision rules

TEMPLATE_USED (Variables referenced in trace-template.md):

Template uses these placeholder patterns:

  • {STORY_ID}, {STORY_TITLE}, {DATE}
  • {P0_TOTAL}, {P0_FULL}, {P0_PCT}, {P0_STATUS}
  • {P1_TOTAL}, {P1_FULL}, {P1_PCT}, {P1_STATUS}
  • {P2_TOTAL}, {P2_FULL}, {P2_PCT}, {P2_STATUS}
  • {P3_TOTAL}, {P3_FULL}, {P3_PCT}, {P3_STATUS}
  • {TOTAL}, {FULL}, {PCT}, {STATUS}
  • {CRITERION_ID}, {CRITERION_DESCRIPTION}, {PRIORITY}
  • {COVERAGE_STATUS}, {STATUS_ICON}
  • {TEST_ID}, {TEST_FILE}, {LINE}
  • {GIVEN}, {WHEN}, {THEN}
  • Many more...

CRITICAL ISSUE: Template uses curly braces {VARIABLE} instead of double curly braces {{variable}}. This is inconsistent with BMAD v6 template variable standards which use {{variable_name}} notation.

UNUSED_BLOAT (Variables defined but NOT explicitly used in instructions or template):

This is where MASSIVE bloat exists. The workflow.yaml contains 52 variables, most of which are:

  1. Boolean configuration flags (auto_discover_tests, map_by_test_id, require_explicit_mapping, etc.)
  2. Threshold numbers (min_p0_coverage, min_p1_pass_rate, etc.)
  3. Path specifications that should be derived, not pre-configured

Bloat Items:

  • acceptance_criteria (empty string, should be elicited)
  • auto_discover_tests, map_by_test_id, map_by_describe, map_by_filename (behavior flags that should be workflow defaults, not variables)
  • require_explicit_mapping, flag_unit_only, flag_integration_only, flag_partial_coverage (all behavior configuration)
  • prioritize_by_risk, suggest_missing_tests, check_duplicate_coverage (more behavior flags)
  • use_test_design, use_tech_spec, use_prd (file loading flags - should be automatic)
  • generate_gate_yaml, generate_coverage_badge, update_story_file (output control flags)
  • min_p0_coverage, min_p1_coverage, min_overall_coverage (threshold config)
  • auto_load_knowledge, include_code_coverage, check_assertions (more behavior flags)
  • enable_gate_decision (Phase 2 control flag)
  • allow_waivers, require_evidence, check_all_workflows_complete, validate_evidence_freshness, require_sign_off (gate configuration)
  • min_p0_pass_rate, min_p1_pass_rate, min_overall_pass_rate (more thresholds)
  • max_critical_nfrs_fail, max_security_issues (threshold config)
  • allow_p2_failures, allow_p3_failures, escalate_p1_failures (risk tolerance flags)
  • gate_output_file, append_to_history, notify_stakeholders (output configuration)

Total Variables Analyzed: 52 variables Used in Instructions (explicitly): ~10-15 conceptually (but not with {variable_name} syntax) Used in Template: 0 (template uses {UPPERCASE} pattern, not {{yaml_variables}}) Unused (Bloat): ~35-40 variables (75%+ of defined variables!)

Status: ❌ FAIL - Severe bloat detected (75%+ unused variables)

3. Config Variable Usage

Communication Language Check:

  • ❌ MISSING - No "communicate in {communication_language}" pattern found in instructions
  • Instructions do not reference communication_language variable
  • Severity: IMPORTANT

User Name Check:

  • ❌ MISSING - No {user_name} usage found in instructions
  • No personalization or greeting patterns detected
  • Severity: MINOR (optional for this workflow type)

Output Folder Check:

  • ⚠️ PARTIAL - Output folder is referenced in workflow.yaml (output_file: "{output_folder}/..."), but instructions don't explicitly use {output_folder} variable pattern
  • File writes mention saving to output_folder conceptually
  • Severity: MINOR

Date Usage Check:

  • ✅ USED - Date is available in workflow.yaml and used in template as {DATE}
  • Severity: N/A

Status: ⚠️ IMPORTANT - Config variables not properly utilized in instructions

4. Web Bundle Validation

Web Bundle Present: ❌ NO

Status: ❌ CRITICAL - No web_bundle configuration found

The workflow.yaml contains:

web_bundle: false

This means the workflow is NOT configured for web deployment. For a production workflow, this is a critical omission.

Missing web_bundle requirements:

  • No web_bundle_files list
  • No existing_workflows mapping (critical since instructions reference knowledge fragments and other workflows)
  • No web deployment path configuration

Workflow Dependencies Detected in Instructions: The instructions reference loading knowledge fragments:

  • test-priorities-matrix.md
  • risk-governance.md
  • probability-impact.md
  • test-quality.md
  • selective-testing.md

These fragments should be declared in a web_bundle configuration.

Impact: This workflow cannot be bundled for web deployment without web_bundle configuration.

Severity: CRITICAL (if web deployment is intended)

5. Bloat Detection

Unused YAML Fields Analysis

Total YAML fields: 52 variables (excluding standard config and metadata) Used fields: ~15 (estimated, mostly conceptual usage, not explicit {variable} references) Unused fields: ~37 Bloat percentage: 71%

Bloat Categories:

Category 1: Boolean Behavior Flags (Should be workflow defaults, not variables)

These should be removed from variables and implemented as hardcoded workflow behavior:

  1. auto_discover_tests: true - This should be the default workflow behavior
  2. map_by_test_id: true - Mapping strategy should be built into the workflow
  3. map_by_describe: true - Same as above
  4. map_by_filename: true - Same as above
  5. require_explicit_mapping: true - Quality standard, not a variable
  6. flag_unit_only: true - Analysis behavior, not configurable
  7. flag_integration_only: true - Same
  8. flag_partial_coverage: true - Same
  9. prioritize_by_risk: true - Core workflow principle
  10. suggest_missing_tests: true - Core workflow output
  11. check_duplicate_coverage: true - Quality check, not configurable
  12. use_test_design: true - Should auto-detect file existence
  13. use_tech_spec: true - Same
  14. use_prd: true - Same
  15. generate_gate_yaml: true - Core workflow output
  16. generate_coverage_badge: true - Optional output, could be removed
  17. update_story_file: true - Optional behavior, could be removed
  18. auto_load_knowledge: true - Should be default behavior
  19. include_code_coverage: false - Optional feature flag, acceptable to keep
  20. check_assertions: true - Quality check, not configurable
  21. enable_gate_decision: true - Workflow mode, could be simplified
  22. allow_waivers: true - Gate policy, could be hardcoded or project-level config
  23. require_evidence: true - Gate policy, should be default
  24. check_all_workflows_complete: true - Validation behavior
  25. validate_evidence_freshness: true - Quality check
  26. require_sign_off: false - Optional gate requirement
  27. allow_p2_failures: true - Risk policy
  28. allow_p3_failures: true - Risk policy
  29. escalate_p1_failures: true - Risk policy
  30. append_to_history: true - Output behavior
  31. notify_stakeholders: true - Output behavior

Recommendation: Remove 25+ boolean flags. These should be workflow defaults or auto-detected behaviors, not user-configurable variables.

Category 2: Threshold Configuration (Should be project-level config, not workflow variables)

These belong in module config.yaml or project-specific configuration, not workflow.yaml:

  1. min_p0_coverage: 100
  2. min_p1_coverage: 90
  3. min_overall_coverage: 80
  4. min_p0_pass_rate: 100
  5. min_p1_pass_rate: 95
  6. min_overall_pass_rate: 90
  7. max_critical_nfrs_fail: 0
  8. max_security_issues: 0

Recommendation: Move thresholds to bmm/config.yaml as project-wide quality standards. Reference via {config_source}:threshold_name pattern.

Category 3: Empty/Placeholder Variables (Should be elicited, not pre-declared)

  1. story_file: "" - Should be elicited with tag
  2. acceptance_criteria: "" - Same
  3. nfr_file: "" - Optional, should be elicited if needed
  4. test_results: "" - Should be elicited for Phase 2

Recommendation: Remove empty variables. Use tags in instructions to elicit required inputs.

Category 4: Derived/Output Paths (Should be calculated in workflow, not pre-defined)

  1. test_dir: "{project-root}/tests" - Should be auto-detected or elicited
  2. source_dir: "{project-root}/src" - Same
  3. output_file: "{output_folder}/traceability-matrix.md" - Duplicates default_output_file
  4. gate_output_file: "{output_folder}/gate-decision-{gate_type}-{story_id}{epic_num}{release_version}.md" - Complex derivation

Recommendation: Remove redundant path variables. Use default_output_file or calculate paths in instructions.

Category 5: Acceptable Variables (Keep these)

These variables have legitimate workflow-specific purpose:

  1. coverage_levels: "e2e,api,component,unit" - Customizable test level scope
  2. gate_type: "story" - Determines gate scope (story/epic/release/hotfix)
  3. decision_mode: "deterministic" - Affects decision process
  4. installed_path, instructions, validation, template - Standard workflow paths (KEEP)

Total Bloat Items: 35+ variables that should be removed or relocated

Bloat Percentage: 71% Cleanup Potential: HIGH - Removing bloat would reduce workflow.yaml from 146 lines to ~30-40 lines

6. Template Variable Mapping

Template Variables Analysis

Template Variable Pattern: Template uses {UPPERCASE_VARIABLE} format (e.g., {STORY_ID}, {P0_TOTAL})

Critical Issue: ❌ BMAD v6 standard uses {{lowercase_variable}} format for template variables.

The trace-template.md uses 60+ template placeholders with {UPPERCASE} pattern instead of {{variable_name}} pattern.

Cross-Reference with Instructions

Instructions tags: None found in instructions.md

Critical Problem: The instructions.md file does NOT contain <template-output> tags to map workflow outputs to template variables.

Workflow Type: Document workflow (has template: "{installed_path}/trace-template.md")

Expected Behavior: Document workflows should have tags in instructions to populate template sections.

Actual Behavior: Instructions are written as pure procedural markdown without template-output integration.

Impact: This workflow appears to be a HYBRID workflow (action + document) but is configured as a document workflow. The template exists but isn't integrated with the workflow execution.

Recommendation:

  1. Either remove the template and set template: false (make it an action workflow)
  2. Or refactor instructions to use tags and update template to use {{variable}} pattern

Status: ❌ IMPORTANT - Template/instruction integration broken

Recommendations

Critical (Fix Immediately)

  1. Add web_bundle configuration
    • Add web_bundle block with all required files (instructions, template, checklist)
    • Map knowledge fragments (test-priorities-matrix.md, risk-governance.md, etc.)
    • Enable web deployment capability
    • Severity: CRITICAL
    • Impact: Workflow cannot be deployed to web without this

Important (Address Soon)

  1. Fix template integration

    • Either: Remove template, set template: false, make this an action workflow
    • Or: Add tags to instructions and convert template to {{variable}} format
    • Current hybrid state is broken
    • Severity: IMPORTANT
    • Impact: Template is not being populated by workflow execution

  2. Add config variable usage in instructions

    • Add communication_language support for multilingual workflows
    • Consider adding user_name for personalization (optional)
    • Ensure output_folder is explicitly used in all file write operations
    • Severity: IMPORTANT
    • Impact: Workflows not following BMAD v6 config standards

Cleanup (Nice to Have)

  1. Remove boolean behavior flags (25+ variables)

    • Delete: auto_discover_tests, map_by_test_id, require_explicit_mapping, flag_unit_only, prioritize_by_risk, suggest_missing_tests, check_duplicate_coverage, use_test_design, generate_gate_yaml, auto_load_knowledge, check_assertions, enable_gate_decision, require_evidence, check_all_workflows_complete, allow_p2_failures, escalate_p1_failures, append_to_history, notify_stakeholders, etc.
    • These should be workflow defaults, not user-configurable variables
    • Severity: CLEANUP
    • Impact: Reduces bloat from 71% to ~15%, improves maintainability

  2. Move threshold configuration to module config

    • Move to bmm/config.yaml: min_p0_coverage, min_p1_coverage, min_overall_coverage, min_p0_pass_rate, min_p1_pass_rate, min_overall_pass_rate, max_critical_nfrs_fail, max_security_issues
    • Reference via {config_source}:threshold_name
    • Severity: CLEANUP
    • Impact: Centralizes quality standards, enables project-wide consistency

  3. Remove empty placeholder variables

    • Delete: story_file: "", acceptance_criteria: "", nfr_file: "", test_results: ""
    • Use tags in instructions to elicit these inputs
    • Severity: CLEANUP
    • Impact: Cleaner workflow.yaml, more dynamic input gathering

  4. Remove redundant path variables

    • Delete: output_file (duplicates default_output_file)
    • Auto-detect or elicit: test_dir, source_dir
    • Simplify: gate_output_file (calculate in instructions)
    • Severity: CLEANUP
    • Impact: Reduces path configuration redundancy

  5. Simplify variables block

    • Keep only: coverage_levels, gate_type, decision_mode, include_code_coverage (optional feature flag)
    • Remove everything else per recommendations 4-7
    • Severity: CLEANUP
    • Impact: Workflow.yaml drops from 146 lines to ~35 lines (76% reduction)

  6. Fix template variable naming convention

    • Convert {UPPERCASE} to {{lowercase_variable}} throughout trace-template.md
    • Align with BMAD v6 standards
    • Severity: CLEANUP
    • Impact: Template consistency with framework standards

  7. Add tags for required inputs

    • Add for story_file path at start of workflow
    • Add for test_results path before Phase 2
    • Add conditional for nfr_file if release-level gate
    • Severity: CLEANUP
    • Impact: Makes workflow properly interactive

  8. Document workflow mode decision

    • Decide: Is this an action workflow or document workflow?
    • If action: Remove template, set template: false
    • If document: Add tags and integrate template properly
    • Severity: CLEANUP
    • Impact: Clarifies workflow purpose and execution model

Validation Checklist

Use this checklist to verify fixes:

  • All standard config variables present and correct ✅ (Already passing)
  • No unused yaml fields (bloat removed to <15%)
  • Config variables used appropriately in instructions
  • Web bundle includes all dependencies
  • Template variables properly mapped (or template removed)
  • File structure follows v6 conventions
  • Variables block reduced from 52 to ~4-5 essential variables
  • Threshold config moved to bmm/config.yaml
  • Boolean flags removed (workflow defaults)
  • Empty placeholders removed ( tags added)

Next Steps

  1. Review critical issues and fix web_bundle configuration immediately
  2. Address important issues in next iteration (template integration, config usage)
  3. Consider cleanup recommendations for optimization (bloat removal)
  4. Re-run audit after fixes to verify improvements

Estimated Cleanup Impact:

  • Workflow.yaml: 146 lines → ~35 lines (76% reduction)
  • Bloat: 71% → <15%
  • Maintainability: Significantly improved
  • Web deployment: Enabled (after web_bundle added)

Audit Complete - Generated by audit-workflow v1.0