feat: add agent schema validation with comprehensive testing

Introduce automated validation for agent YAML files using Zod to ensure
schema compliance across all agent definitions. This feature validates
17 agent files across core and module directories, catching structural
errors and maintaining consistency.

Schema Validation (tools/schema/agent.js):
- Zod-based schema validating metadata, persona, menu, prompts, and critical actions
- Module-aware validation: module field required for src/modules/**/agents/,
  optional for src/core/agents/
- Enforces kebab-case unique triggers and at least one command target per menu item
- Validates persona.principles as array (not string)
- Comprehensive refinements for data integrity

CLI Validator (tools/validate-agent-schema.js):
- Scans src/{core,modules/*}/agents/*.agent.yaml
- Parses with js-yaml and validates using Zod schema
- Reports detailed errors with file paths and field paths
- Exits 1 on failures, 0 on success
- Accepts optional project_root parameter for testing

Testing (679 lines across 3 test files):
- test/test-cli-integration.sh: CLI behavior and error handling tests
- test/unit-test-schema.js: Direct schema validation unit tests
- test/test-agent-schema.js: Comprehensive fixture-based tests
- 50 test fixtures covering valid and invalid scenarios
- ESLint configured to support CommonJS test files
- Prettier configured to ignore intentionally broken fixtures

CI Integration (.github/workflows/lint.yaml):
- Renamed from format-check.yaml to lint.yaml
- Added schema-validation job running npm run validate:schemas
- Runs in parallel with prettier and eslint jobs
- Validates on all pull requests

Data Cleanup:
- Fixed src/core/agents/bmad-master.agent.yaml: converted persona.principles
  from string to array format

Documentation:
- Updated schema-classification.md with validation section
- Documents validator usage, enforcement rules, and CI integration

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Alex Verkhovsky
2025-10-20 02:14:33 -07:00
parent 940cc15751
commit 6b9ab8b201
64 changed files with 2732 additions and 11 deletions

View File

@@ -0,0 +1,26 @@
# Test: critical_actions as non-array
# Expected: FAIL
# Error code: invalid_type
# Error path: agent.critical_actions
# Error expected: array
agent:
metadata:
id: actions-string
name: Actions String
title: Actions String
icon:
persona:
role: Test agent
identity: Test identity
communication_style: Test style
principles:
- Test principle
critical_actions: This should be an array
menu:
- trigger: help
description: Show help
action: display_help

View File

@@ -0,0 +1,29 @@
# Test: critical_actions with empty strings
# Expected: FAIL
# Error code: custom
# Error path: agent.critical_actions[1]
# Error message: agent.critical_actions[] must be a non-empty string
agent:
metadata:
id: empty-action-string
name: Empty Action String
title: Empty Action String
icon:
persona:
role: Test agent
identity: Test identity
communication_style: Test style
principles:
- Test principle
critical_actions:
- Valid action
- " "
- Another valid action
menu:
- trigger: help
description: Show help
action: display_help

View File

@@ -0,0 +1,24 @@
# Test: Menu item with empty string command target
# Expected: FAIL
# Error code: custom
# Error path: agent.menu[0].action
# Error message: agent.menu[].action must be a non-empty string
agent:
metadata:
id: empty-command
name: Empty Command Target
title: Empty Command
icon:
persona:
role: Test agent
identity: Test identity
communication_style: Test style
principles:
- Test principle
menu:
- trigger: help
description: Show help
action: " "

View File

@@ -0,0 +1,23 @@
# Test: Menu item with no command target fields
# Expected: FAIL
# Error code: custom
# Error path: agent.menu[0]
# Error message: agent.menu[] entries must include at least one command target field
agent:
metadata:
id: no-command
name: No Command Target
title: No Command
icon:
persona:
role: Test agent
identity: Test identity
communication_style: Test style
principles:
- Test principle
menu:
- trigger: help
description: Show help but no command target

View File

@@ -0,0 +1,24 @@
# Test: CamelCase trigger
# Expected: FAIL
# Error code: custom
# Error path: agent.menu[0].trigger
# Error message: agent.menu[].trigger must be kebab-case (lowercase words separated by hyphen)
agent:
metadata:
id: camel-case-trigger
name: CamelCase Trigger
title: CamelCase
icon:
persona:
role: Test agent
identity: Test identity
communication_style: Test style
principles:
- Test principle
menu:
- trigger: listTasks
description: Invalid CamelCase trigger
action: list_tasks

View File

@@ -0,0 +1,30 @@
# Test: Duplicate triggers within same agent
# Expected: FAIL
# Error code: custom
# Error path: agent.menu[2].trigger
# Error message: agent.menu[].trigger duplicates "help" within the same agent
agent:
metadata:
id: duplicate-triggers
name: Duplicate Triggers
title: Duplicate
icon:
persona:
role: Test agent
identity: Test identity
communication_style: Test style
principles:
- Test principle
menu:
- trigger: help
description: First help command
action: display_help
- trigger: list-tasks
description: List tasks
action: list_tasks
- trigger: help
description: Duplicate help command
action: show_help

View File

@@ -0,0 +1,24 @@
# Test: Empty trigger string
# Expected: FAIL
# Error code: custom
# Error path: agent.menu[0].trigger
# Error message: agent.menu[].trigger must be a non-empty string
agent:
metadata:
id: empty-trigger
name: Empty Trigger
title: Empty
icon:
persona:
role: Test agent
identity: Test identity
communication_style: Test style
principles:
- Test principle
menu:
- trigger: " "
description: Empty trigger
action: display_help

View File

@@ -0,0 +1,24 @@
# Test: Trigger with leading asterisk
# Expected: FAIL
# Error code: custom
# Error path: agent.menu[0].trigger
# Error message: agent.menu[].trigger must be kebab-case (lowercase words separated by hyphen)
agent:
metadata:
id: asterisk-trigger
name: Asterisk Trigger
title: Asterisk
icon:
persona:
role: Test agent
identity: Test identity
communication_style: Test style
principles:
- Test principle
menu:
- trigger: "*help"
description: Invalid trigger with asterisk
action: display_help

View File

@@ -0,0 +1,24 @@
# Test: Snake_case trigger
# Expected: FAIL
# Error code: custom
# Error path: agent.menu[0].trigger
# Error message: agent.menu[].trigger must be kebab-case (lowercase words separated by hyphen)
agent:
metadata:
id: snake-case-trigger
name: Snake Case Trigger
title: Snake Case
icon:
persona:
role: Test agent
identity: Test identity
communication_style: Test style
principles:
- Test principle
menu:
- trigger: list_tasks
description: Invalid snake_case trigger
action: list_tasks

View File

@@ -0,0 +1,24 @@
# Test: Trigger with spaces
# Expected: FAIL
# Error code: custom
# Error path: agent.menu[0].trigger
# Error message: agent.menu[].trigger must be kebab-case (lowercase words separated by hyphen)
agent:
metadata:
id: spaces-trigger
name: Spaces Trigger
title: Spaces
icon:
persona:
role: Test agent
identity: Test identity
communication_style: Test style
principles:
- Test principle
menu:
- trigger: list tasks
description: Invalid trigger with spaces
action: list_tasks

View File

@@ -0,0 +1,21 @@
# Test: Empty menu array
# Expected: FAIL
# Error code: too_small
# Error path: agent.menu
# Error minimum: 1
agent:
metadata:
id: empty-menu
name: Empty Menu
title: Empty Menu
icon:
persona:
role: Test agent
identity: Test identity
communication_style: Test style
principles:
- Test principle
menu: []

View File

@@ -0,0 +1,19 @@
# Test: Missing menu field
# Expected: FAIL
# Error code: invalid_type
# Error path: agent.menu
# Error expected: array
agent:
metadata:
id: missing-menu
name: Missing Menu
title: Missing Menu
icon:
persona:
role: Test agent
identity: Test identity
communication_style: Test style
principles:
- Test principle

View File

@@ -0,0 +1,26 @@
# Test: Core agent with unexpected module field
# Expected: FAIL
# Error code: custom
# Error path: agent.metadata.module
# Error message: core agents must not include agent.metadata.module
# Path context: src/core/agents/core-agent-with-module.agent.yaml
agent:
metadata:
id: core-with-module
name: Core With Module
title: Core Agent
icon:
module: bmm
persona:
role: Test agent
identity: Test identity
communication_style: Test style
principles:
- Test principle
menu:
- trigger: help
description: Show help
action: display_help

View File

@@ -0,0 +1,26 @@
# Test: Module field with whitespace only
# Expected: FAIL
# Error code: custom
# Error path: agent.metadata.module
# Error message: agent.metadata.module must be a non-empty string
# Path context: src/modules/bmm/agents/empty-module-string.agent.yaml
agent:
metadata:
id: empty-module
name: Empty Module String
title: Empty Module
icon:
module: " "
persona:
role: Test agent
identity: Test identity
communication_style: Test style
principles:
- Test principle
menu:
- trigger: help
description: Show help
action: display_help

View File

@@ -0,0 +1,24 @@
# Test: Empty string in metadata.name field
# Expected: FAIL
# Error code: custom
# Error path: agent.metadata.name
# Error message: agent.metadata.name must be a non-empty string
agent:
metadata:
id: empty-name-test
name: " "
title: Empty Name Test
icon:
persona:
role: Test agent
identity: Test identity
communication_style: Test style
principles:
- Test principle
menu:
- trigger: help
description: Show help
action: display_help

View File

@@ -0,0 +1,26 @@
# Test: Extra unknown fields in metadata
# Expected: FAIL
# Error code: unrecognized_keys
# Error path: agent.metadata
# Error keys: ["unknown_field", "another_extra"]
agent:
metadata:
id: extra-fields
name: Extra Fields
title: Extra Fields
icon:
unknown_field: This is not allowed
another_extra: Also invalid
persona:
role: Test agent
identity: Test identity
communication_style: Test style
principles:
- Test principle
menu:
- trigger: help
description: Show help
action: display_help

View File

@@ -0,0 +1,23 @@
# Test: Missing required metadata.id field
# Expected: FAIL
# Error code: invalid_type
# Error path: agent.metadata.id
# Error expected: string
agent:
metadata:
name: Missing ID Agent
title: Missing ID
icon:
persona:
role: Test agent
identity: Test identity
communication_style: Test style
principles:
- Test principle
menu:
- trigger: help
description: Show help
action: display_help

View File

@@ -0,0 +1,25 @@
# Test: Module agent missing required module field
# Expected: FAIL
# Error code: custom
# Error path: agent.metadata.module
# Error message: module-scoped agents must declare agent.metadata.module
# Path context: src/modules/bmm/agents/module-agent-missing-module.agent.yaml
agent:
metadata:
id: bmm-missing-module
name: BMM Missing Module
title: Missing Module
icon:
persona:
role: Test agent
identity: Test identity
communication_style: Test style
principles:
- Test principle
menu:
- trigger: help
description: Show help
action: display_help

View File

@@ -0,0 +1,26 @@
# Test: Module agent with wrong module value
# Expected: FAIL
# Error code: custom
# Error path: agent.metadata.module
# Error message: agent.metadata.module must equal "bmm"
# Path context: src/modules/bmm/agents/wrong-module-value.agent.yaml
agent:
metadata:
id: wrong-module
name: Wrong Module
title: Wrong Module
icon:
module: cis
persona:
role: Test agent
identity: Test identity
communication_style: Test style
principles:
- Test principle
menu:
- trigger: help
description: Show help
action: display_help

View File

@@ -0,0 +1,23 @@
# Test: Empty principles array
# Expected: FAIL
# Error code: too_small
# Error path: agent.persona.principles
# Error minimum: 1
agent:
metadata:
id: empty-principles
name: Empty Principles
title: Empty Principles
icon:
persona:
role: Test agent
identity: Test identity
communication_style: Test style
principles: []
menu:
- trigger: help
description: Show help
action: display_help

View File

@@ -0,0 +1,26 @@
# Test: Empty string in principles array
# Expected: FAIL
# Error code: custom
# Error path: agent.persona.principles[1]
# Error message: agent.persona.principles[] must be a non-empty string
agent:
metadata:
id: empty-principle-string
name: Empty Principle String
title: Empty Principle
icon:
persona:
role: Test agent
identity: Test identity
communication_style: Test style
principles:
- Valid principle
- " "
- Another valid principle
menu:
- trigger: help
description: Show help
action: display_help

View File

@@ -0,0 +1,26 @@
# Test: Extra unknown fields in persona
# Expected: FAIL
# Error code: unrecognized_keys
# Error path: agent.persona
# Error keys: ["extra_field", "another_extra"]
agent:
metadata:
id: extra-persona-fields
name: Extra Persona Fields
title: Extra Persona
icon:
persona:
role: Test agent
identity: Test identity
communication_style: Test style
principles:
- Test principle
extra_field: Not allowed
another_extra: Also invalid
menu:
- trigger: help
description: Show help
action: display_help

View File

@@ -0,0 +1,23 @@
# Test: Missing required persona.role field
# Expected: FAIL
# Error code: invalid_type
# Error path: agent.persona.role
# Error expected: string
agent:
metadata:
id: missing-role
name: Missing Role
title: Missing Role
icon:
persona:
identity: Test identity
communication_style: Test style
principles:
- Test principle
menu:
- trigger: help
description: Show help
action: display_help

View File

@@ -0,0 +1,23 @@
# Test: principles as string instead of array
# Expected: FAIL
# Error code: invalid_type
# Error path: agent.persona.principles
# Error expected: array
agent:
metadata:
id: principles-string
name: Principles String
title: Principles String
icon:
persona:
role: Test agent
identity: Test identity
communication_style: Test style
principles: This should be an array, not a string
menu:
- trigger: help
description: Show help
action: display_help

View File

@@ -0,0 +1,28 @@
# Test: Prompt with empty content string
# Expected: FAIL
# Error code: custom
# Error path: agent.prompts[0].content
# Error message: agent.prompts[].content must be a non-empty string
agent:
metadata:
id: empty-content
name: Empty Content
title: Empty Content
icon:
persona:
role: Test agent
identity: Test identity
communication_style: Test style
principles:
- Test principle
prompts:
- id: prompt1
content: " "
menu:
- trigger: help
description: Show help
action: display_help

View File

@@ -0,0 +1,30 @@
# Test: Extra unknown fields in prompts
# Expected: FAIL
# Error code: unrecognized_keys
# Error path: agent.prompts[0]
# Error keys: ["extra_field"]
agent:
metadata:
id: extra-prompt-fields
name: Extra Prompt Fields
title: Extra Fields
icon:
persona:
role: Test agent
identity: Test identity
communication_style: Test style
principles:
- Test principle
prompts:
- id: prompt1
content: Valid content
description: Valid description
extra_field: Not allowed
menu:
- trigger: help
description: Show help
action: display_help

View File

@@ -0,0 +1,27 @@
# Test: Prompt missing required content field
# Expected: FAIL
# Error code: invalid_type
# Error path: agent.prompts[0].content
# Error expected: string
agent:
metadata:
id: prompt-missing-content
name: Prompt Missing Content
title: Missing Content
icon:
persona:
role: Test agent
identity: Test identity
communication_style: Test style
principles:
- Test principle
prompts:
- id: prompt1
menu:
- trigger: help
description: Show help
action: display_help

View File

@@ -0,0 +1,27 @@
# Test: Prompt missing required id field
# Expected: FAIL
# Error code: invalid_type
# Error path: agent.prompts[0].id
# Error expected: string
agent:
metadata:
id: prompt-missing-id
name: Prompt Missing ID
title: Missing ID
icon:
persona:
role: Test agent
identity: Test identity
communication_style: Test style
principles:
- Test principle
prompts:
- content: Prompt without ID
menu:
- trigger: help
description: Show help
action: display_help

View File

@@ -0,0 +1,5 @@
# Test: Empty YAML file
# Expected: FAIL
# Error code: invalid_type
# Error path:
# Error expected: object

View File

@@ -0,0 +1,27 @@
# Test: Extra top-level keys beyond 'agent'
# Expected: FAIL
# Error code: unrecognized_keys
# Error path:
# Error keys: ["extra_key", "another_extra"]
agent:
metadata:
id: extra-test
name: Extra Test Agent
title: Extra Test
icon: 🧪
persona:
role: Test agent
identity: Test identity
communication_style: Test style
principles:
- Test principle
menu:
- trigger: help
description: Show help
action: display_help
extra_key: This should not be allowed
another_extra: Also invalid

View File

@@ -0,0 +1,11 @@
# Test: Missing required 'agent' top-level key
# Expected: FAIL
# Error code: invalid_type
# Error path: agent
# Error expected: object
metadata:
id: bad-test
name: Bad Test Agent
title: Bad Test
icon:

View File

@@ -0,0 +1,19 @@
# Test: Invalid YAML structure with inconsistent indentation
# Expected: FAIL - YAML parse error
agent:
metadata:
id: invalid-indent
name: Invalid Indentation
title: Invalid
icon:
persona:
role: Test
identity: Test
communication_style: Test
principles:
- Test
menu:
- trigger: help
description: Help
action: help

View File

@@ -0,0 +1,18 @@
# Test: Malformed YAML with syntax errors
# Expected: FAIL - YAML parse error
agent:
metadata:
id: malformed
name: Malformed YAML
title: [Malformed
icon: 🧪
persona:
role: Test
identity: Test
communication_style: Test
principles:
- Test
menu:
- trigger: help
description: Help