feat: add agent schema validation with comprehensive testing (#774)

Introduce automated validation for agent YAML files using Zod to ensure
schema compliance across all agent definitions. This feature validates
17 agent files across core and module directories, catching structural
errors and maintaining consistency.

Schema Validation (tools/schema/agent.js):
- Zod-based schema validating metadata, persona, menu, prompts, and critical actions
- Module-aware validation: module field required for src/modules/**/agents/,
  optional for src/core/agents/
- Enforces kebab-case unique triggers and at least one command target per menu item
- Validates persona.principles as array (not string)
- Comprehensive refinements for data integrity

CLI Validator (tools/validate-agent-schema.js):
- Scans src/{core,modules/*}/agents/*.agent.yaml
- Parses with js-yaml and validates using Zod schema
- Reports detailed errors with file paths and field paths
- Exits 1 on failures, 0 on success
- Accepts optional project_root parameter for testing

Testing (679 lines across 3 test files):
- test/test-cli-integration.sh: CLI behavior and error handling tests
- test/unit-test-schema.js: Direct schema validation unit tests
- test/test-agent-schema.js: Comprehensive fixture-based tests
- 50 test fixtures covering valid and invalid scenarios
- ESLint configured to support CommonJS test files
- Prettier configured to ignore intentionally broken fixtures

CI Integration (.github/workflows/lint.yaml):
- Renamed from format-check.yaml to lint.yaml
- Added schema-validation job running npm run validate:schemas
- Runs in parallel with prettier and eslint jobs
- Validates on all pull requests

Data Cleanup:
- Fixed src/core/agents/bmad-master.agent.yaml: converted persona.principles
  from string to array format

Documentation:
- Updated schema-classification.md with validation section
- Documents validator usage, enforcement rules, and CI integration

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
Alex Verkhovsky
2025-10-20 05:14:50 -07:00
committed by GitHub
parent 2a6eb71612
commit 31666c1f0f
64 changed files with 2732 additions and 11 deletions

View File

@@ -0,0 +1,23 @@
# Test: Empty critical_actions array
# Expected: PASS - empty array is valid for optional field
agent:
metadata:
id: empty-critical-actions
name: Empty Critical Actions
title: Empty Critical Actions
icon: 🧪
persona:
role: Test agent with empty critical actions
identity: I am a test agent with empty critical actions array.
communication_style: Clear
principles:
- Test empty arrays
critical_actions: []
menu:
- trigger: help
description: Show help
action: display_help

View File

@@ -0,0 +1,21 @@
# Test: No critical_actions field (optional)
# Expected: PASS
agent:
metadata:
id: no-critical-actions
name: No Critical Actions
title: No Critical Actions
icon: 🧪
persona:
role: Test agent without critical actions
identity: I am a test agent without critical actions.
communication_style: Clear
principles:
- Test optional fields
menu:
- trigger: help
description: Show help
action: display_help

View File

@@ -0,0 +1,26 @@
# Test: critical_actions with valid strings
# Expected: PASS
agent:
metadata:
id: valid-critical-actions
name: Valid Critical Actions
title: Valid Critical Actions
icon: 🧪
persona:
role: Test agent with critical actions
identity: I am a test agent with valid critical actions.
communication_style: Clear
principles:
- Test valid arrays
critical_actions:
- Load configuration from disk
- Initialize user context
- Set communication preferences
menu:
- trigger: help
description: Show help
action: display_help

View File

@@ -0,0 +1,39 @@
# Test: Menu items with all valid command target types
# Expected: PASS
agent:
metadata:
id: all-commands
name: All Command Types
title: All Commands
icon: 🧪
persona:
role: Test agent with all command types
identity: I test all available command target types.
communication_style: Clear
principles:
- Test all command types
menu:
- trigger: workflow-test
description: Test workflow command
workflow: path/to/workflow
- trigger: validate-test
description: Test validate-workflow command
validate-workflow: path/to/validation
- trigger: exec-test
description: Test exec command
exec: npm test
- trigger: action-test
description: Test action command
action: perform_action
- trigger: tmpl-test
description: Test tmpl command
tmpl: path/to/template
- trigger: data-test
description: Test data command
data: path/to/data
- trigger: run-workflow-test
description: Test run-workflow command
run-workflow: path/to/workflow

View File

@@ -0,0 +1,23 @@
# Test: Menu item with multiple command targets
# Expected: PASS - multiple targets are allowed
agent:
metadata:
id: multiple-commands
name: Multiple Commands
title: Multiple Commands
icon: 🧪
persona:
role: Test agent with multiple command targets
identity: I test multiple command targets per menu item.
communication_style: Clear
principles:
- Test multiple targets
menu:
- trigger: multi-command
description: Menu item with multiple command targets
workflow: path/to/workflow
exec: npm test
action: perform_action

View File

@@ -0,0 +1,33 @@
# Test: Valid kebab-case triggers
# Expected: PASS
agent:
metadata:
id: kebab-triggers
name: Kebab Case Triggers
title: Kebab Triggers
icon: 🧪
persona:
role: Test agent with kebab-case triggers
identity: I test kebab-case trigger validation.
communication_style: Clear
principles:
- Test kebab-case format
menu:
- trigger: help
description: Single word trigger
action: display_help
- trigger: list-tasks
description: Two word trigger
action: list_tasks
- trigger: workflow-init-process
description: Three word trigger
action: init_workflow
- trigger: test123
description: Trigger with numbers
action: test
- trigger: multi-word-kebab-case-trigger
description: Long kebab-case trigger
action: long_action

View File

@@ -0,0 +1,30 @@
# Test: Menu with multiple valid items using different command types
# Expected: PASS
agent:
metadata:
id: multiple-menu
name: Multiple Menu Items
title: Multiple Menu
icon: 🧪
persona:
role: Test agent with multiple menu items
identity: I am a test agent with diverse menu commands.
communication_style: Clear
principles:
- Test multiple menu items
menu:
- trigger: help
description: Show help
action: display_help
- trigger: start-workflow
description: Start a workflow
workflow: path/to/workflow
- trigger: execute
description: Execute command
exec: npm test
- trigger: use-template
description: Use template
tmpl: path/to/template

View File

@@ -0,0 +1,21 @@
# Test: Menu with single valid item
# Expected: PASS
agent:
metadata:
id: single-menu
name: Single Menu Item
title: Single Menu
icon: 🧪
persona:
role: Test agent with single menu item
identity: I am a test agent.
communication_style: Clear
principles:
- Test minimal menu
menu:
- trigger: help
description: Show help information
action: display_help

View File

@@ -0,0 +1,23 @@
# Test: Empty module name in path (src/modules//agents/)
# Expected: PASS - treated as core agent (empty module normalizes to null)
# Path context: src/modules//agents/test.agent.yaml
agent:
metadata:
id: empty-module-path
name: Empty Module in Path
title: Empty Module Path
icon: 🧪
# No module field - path has empty module name, treated as core
persona:
role: Test agent for empty module name in path
identity: I test the edge case where module name in path is empty.
communication_style: Clear
principles:
- Test path parsing edge cases
menu:
- trigger: help
description: Show help
action: display_help

View File

@@ -0,0 +1,23 @@
# Test: Malformed module path (no slash after module name) treated as core
# Expected: PASS - malformed path returns null, treated as core agent
# Path context: src/modules/bmm
agent:
metadata:
id: malformed-path
name: Malformed Path Test
title: Malformed Path
icon: 🧪
# No module field - will be treated as core since path parsing returns null
persona:
role: Test agent for malformed path edge case
identity: I test edge cases in path parsing.
communication_style: Clear
principles:
- Test edge case handling
menu:
- trigger: help
description: Show help
action: display_help

View File

@@ -0,0 +1,23 @@
# Test: Valid module agent with correct module field
# Expected: PASS
# Path context: src/modules/bmm/agents/module-agent-correct.agent.yaml
agent:
metadata:
id: bmm-test
name: BMM Test Agent
title: BMM Test
icon: 🧪
module: bmm
persona:
role: Test module agent
identity: I am a module-scoped test agent.
communication_style: Professional
principles:
- Test module validation
menu:
- trigger: help
description: Show help
action: display_help

View File

@@ -0,0 +1,23 @@
# Test: All persona fields properly filled
# Expected: PASS
agent:
metadata:
id: complete-persona
name: Complete Persona Agent
title: Complete Persona
icon: 🧪
persona:
role: Comprehensive test agent with all persona fields
identity: I am a test agent designed to validate complete persona structure with multiple characteristics and attributes.
communication_style: Professional, clear, and thorough with attention to detail
principles:
- Validate all persona fields are present
- Ensure array fields work correctly
- Test comprehensive documentation
menu:
- trigger: help
description: Show help
action: display_help

View File

@@ -0,0 +1,23 @@
# Test: Empty prompts array
# Expected: PASS - empty array valid for optional field
agent:
metadata:
id: empty-prompts
name: Empty Prompts
title: Empty Prompts
icon: 🧪
persona:
role: Test agent with empty prompts
identity: I am a test agent with empty prompts array.
communication_style: Clear
principles:
- Test empty arrays
prompts: []
menu:
- trigger: help
description: Show help
action: display_help

View File

@@ -0,0 +1,21 @@
# Test: No prompts field (optional)
# Expected: PASS
agent:
metadata:
id: no-prompts
name: No Prompts
title: No Prompts
icon: 🧪
persona:
role: Test agent without prompts
identity: I am a test agent without prompts field.
communication_style: Clear
principles:
- Test optional fields
menu:
- trigger: help
description: Show help
action: display_help

View File

@@ -0,0 +1,27 @@
# Test: Prompts with required id and content only
# Expected: PASS
agent:
metadata:
id: valid-prompts-minimal
name: Valid Prompts Minimal
title: Valid Prompts
icon: 🧪
persona:
role: Test agent with minimal prompts
identity: I am a test agent with minimal prompt structure.
communication_style: Clear
principles:
- Test minimal prompts
prompts:
- id: prompt1
content: This is a valid prompt content
- id: prompt2
content: Another valid prompt
menu:
- trigger: help
description: Show help
action: display_help

View File

@@ -0,0 +1,29 @@
# Test: Prompts with optional description field
# Expected: PASS
agent:
metadata:
id: valid-prompts-description
name: Valid Prompts With Description
title: Valid Prompts Desc
icon: 🧪
persona:
role: Test agent with prompts including descriptions
identity: I am a test agent with complete prompt structure.
communication_style: Clear
principles:
- Test complete prompts
prompts:
- id: prompt1
content: This is a valid prompt content
description: This prompt does something useful
- id: prompt2
content: Another valid prompt
description: This prompt does something else
menu:
- trigger: help
description: Show help
action: display_help

View File

@@ -0,0 +1,23 @@
# Test: Valid core agent with only required fields
# Expected: PASS
# Path context: src/core/agents/minimal-core-agent.agent.yaml
agent:
metadata:
id: minimal-test
name: Minimal Test Agent
title: Minimal Test
icon: 🧪
persona:
role: Test agent with minimal configuration
identity: I am a minimal test agent used for schema validation testing.
communication_style: Clear and concise
principles:
- Validate schema requirements
- Demonstrate minimal valid structure
menu:
- trigger: help
description: Show help
action: display_help