mirror of
https://github.com/github/spec-kit.git
synced 2026-04-01 10:13:08 +00:00
docs: ensure manual tests use local specify (#2020)
* docs: ensure manual tests use local specify * docs: mention venv activation before editable install * docs: clarify Windows venv activation commands
This commit is contained in:
66
TESTING.md
66
TESTING.md
@@ -1,8 +1,59 @@
|
||||
# Manual Testing Guide
|
||||
# Testing Guide
|
||||
|
||||
This document is the detailed testing companion to [`CONTRIBUTING.md`](./CONTRIBUTING.md).
|
||||
|
||||
Use it for three things:
|
||||
|
||||
1. running quick automated checks before manual testing,
|
||||
2. manually testing affected slash commands through an AI agent, and
|
||||
3. capturing the results in a PR-friendly format.
|
||||
|
||||
Any change that affects a slash command's behavior requires manually testing that command through an AI agent and submitting results with the PR.
|
||||
|
||||
## Process
|
||||
## Recommended order
|
||||
|
||||
1. **Sync your environment** — install the project and test dependencies.
|
||||
2. **Run focused automated checks** — especially for packaging, scaffolding, agent config, and generated-file changes.
|
||||
3. **Run manual agent tests** — for any affected slash commands.
|
||||
4. **Paste results into your PR** — include both command-selection reasoning and manual test results.
|
||||
|
||||
## Quick automated checks
|
||||
|
||||
Run these before manual testing when your change affects packaging, scaffolding, templates, release artifacts, or agent wiring.
|
||||
|
||||
### Environment setup
|
||||
|
||||
```bash
|
||||
cd <spec-kit-repo>
|
||||
uv sync --extra test
|
||||
source .venv/bin/activate # On Windows (CMD): .venv\Scripts\activate | (PowerShell): .venv\Scripts\Activate.ps1
|
||||
```
|
||||
|
||||
### Generated package structure and content
|
||||
|
||||
```bash
|
||||
uv run python -m pytest tests/test_core_pack_scaffold.py -q
|
||||
```
|
||||
|
||||
This validates the generated files that CI-style packaging depends on, including directory layout, file names, frontmatter/TOML validity, placeholder replacement, `.specify/` path rewrites, and parity with `create-release-packages.sh`.
|
||||
|
||||
### Agent configuration and release wiring consistency
|
||||
|
||||
```bash
|
||||
uv run python -m pytest tests/test_agent_config_consistency.py -q
|
||||
```
|
||||
|
||||
Run this when you change agent metadata, release scripts, context update scripts, or artifact naming.
|
||||
|
||||
### Optional single-agent packaging spot check
|
||||
|
||||
```bash
|
||||
AGENTS=copilot SCRIPTS=sh ./.github/workflows/scripts/create-release-packages.sh v1.0.0
|
||||
```
|
||||
|
||||
Inspect `.genreleases/sdd-copilot-package-sh/` and the matching ZIP in `.genreleases/` when you want to review the exact packaged output for one agent/script combination.
|
||||
|
||||
## Manual testing process
|
||||
|
||||
1. **Identify affected commands** — use the [prompt below](#determining-which-tests-to-run) to have your agent analyze your changed files and determine which commands need testing.
|
||||
2. **Set up a test project** — scaffold from your local branch (see [Setup](#setup)).
|
||||
@@ -13,19 +64,22 @@ Any change that affects a slash command's behavior requires manually testing tha
|
||||
## Setup
|
||||
|
||||
```bash
|
||||
# Install the CLI from your local branch
|
||||
# Install the project and test dependencies from your local branch
|
||||
cd <spec-kit-repo>
|
||||
uv venv .venv
|
||||
source .venv/bin/activate # On Windows: .venv\Scripts\activate
|
||||
uv sync --extra test
|
||||
source .venv/bin/activate # On Windows (CMD): .venv\Scripts\activate | (PowerShell): .venv\Scripts\Activate.ps1
|
||||
uv pip install -e .
|
||||
# Ensure the `specify` binary in this environment points at your working tree so the agent runs the branch you're testing.
|
||||
|
||||
# Initialize a test project using your local changes
|
||||
specify init /tmp/speckit-test --ai <agent> --offline
|
||||
uv run specify init /tmp/speckit-test --ai <agent> --offline
|
||||
cd /tmp/speckit-test
|
||||
|
||||
# Open in your agent
|
||||
```
|
||||
|
||||
If you are testing the packaged output rather than the live source tree, create a local release package first as described in [`CONTRIBUTING.md`](./CONTRIBUTING.md).
|
||||
|
||||
## Reporting results
|
||||
|
||||
Paste this into your PR:
|
||||
|
||||
Reference in New Issue
Block a user