docs: ensure manual tests use local specify (#2020)

* docs: ensure manual tests use local specify

* docs: mention venv activation before editable install

* docs: clarify Windows venv activation commands
This commit is contained in:
Hamilton Snow
2026-04-01 00:52:36 +08:00
committed by GitHub
parent 804cd10c71
commit cb16412f88
2 changed files with 74 additions and 7 deletions

View File

@@ -1,8 +1,59 @@
# Manual Testing Guide
# Testing Guide
This document is the detailed testing companion to [`CONTRIBUTING.md`](./CONTRIBUTING.md).
Use it for three things:
1. running quick automated checks before manual testing,
2. manually testing affected slash commands through an AI agent, and
3. capturing the results in a PR-friendly format.
Any change that affects a slash command's behavior requires manually testing that command through an AI agent and submitting results with the PR.
## Process
## Recommended order
1. **Sync your environment** — install the project and test dependencies.
2. **Run focused automated checks** — especially for packaging, scaffolding, agent config, and generated-file changes.
3. **Run manual agent tests** — for any affected slash commands.
4. **Paste results into your PR** — include both command-selection reasoning and manual test results.
## Quick automated checks
Run these before manual testing when your change affects packaging, scaffolding, templates, release artifacts, or agent wiring.
### Environment setup
```bash
cd <spec-kit-repo>
uv sync --extra test
source .venv/bin/activate # On Windows (CMD): .venv\Scripts\activate | (PowerShell): .venv\Scripts\Activate.ps1
```
### Generated package structure and content
```bash
uv run python -m pytest tests/test_core_pack_scaffold.py -q
```
This validates the generated files that CI-style packaging depends on, including directory layout, file names, frontmatter/TOML validity, placeholder replacement, `.specify/` path rewrites, and parity with `create-release-packages.sh`.
### Agent configuration and release wiring consistency
```bash
uv run python -m pytest tests/test_agent_config_consistency.py -q
```
Run this when you change agent metadata, release scripts, context update scripts, or artifact naming.
### Optional single-agent packaging spot check
```bash
AGENTS=copilot SCRIPTS=sh ./.github/workflows/scripts/create-release-packages.sh v1.0.0
```
Inspect `.genreleases/sdd-copilot-package-sh/` and the matching ZIP in `.genreleases/` when you want to review the exact packaged output for one agent/script combination.
## Manual testing process
1. **Identify affected commands** — use the [prompt below](#determining-which-tests-to-run) to have your agent analyze your changed files and determine which commands need testing.
2. **Set up a test project** — scaffold from your local branch (see [Setup](#setup)).
@@ -13,19 +64,22 @@ Any change that affects a slash command's behavior requires manually testing tha
## Setup
```bash
# Install the CLI from your local branch
# Install the project and test dependencies from your local branch
cd <spec-kit-repo>
uv venv .venv
source .venv/bin/activate # On Windows: .venv\Scripts\activate
uv sync --extra test
source .venv/bin/activate # On Windows (CMD): .venv\Scripts\activate | (PowerShell): .venv\Scripts\Activate.ps1
uv pip install -e .
# Ensure the `specify` binary in this environment points at your working tree so the agent runs the branch you're testing.
# Initialize a test project using your local changes
specify init /tmp/speckit-test --ai <agent> --offline
uv run specify init /tmp/speckit-test --ai <agent> --offline
cd /tmp/speckit-test
# Open in your agent
```
If you are testing the packaged output rather than the live source tree, create a local release package first as described in [`CONTRIBUTING.md`](./CONTRIBUTING.md).
## Reporting results
Paste this into your PR: