Add /selftest.extension core extension to test other extensions (#1758)

* test(commands): create extension-commands LLM playground sandbox

* update(tests): format LLM evaluation as an automated test runner

* test(commands): map extension-commands python script with timestamps

* test(commands): map extension-commands python script with timestamps

* test(commands): update TESTING.md to evaluate discovery, lint, and deploy explicitly

* test(commands): simplify execution expectations and add timestamp calculation

* fix(tests): address copilot review comments on prompt formatting and relative paths

* fix(tests): resolve copilot PR feedback regarding extension schema structure and argparse mutually exclusive groups

* feat(extensions): add core selftest utility and migrate away from manual tests sandbox

* fix(selftest): update command name array to match spec-kit validation schema

* fix(selftest): wrap arguments in quotes to support multi-word extension names

* update the command to be more meaningful

* fix: if the extension is discovery only, it should not be installable

* Address review comments for selftest documentation

* address review comments

* address review comments

* Update extensions/selftest/commands/selftest.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Manfred Riem <15701806+mnriem@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
This commit is contained in:
Dhilip
2026-03-13 08:26:01 -04:00
committed by GitHub
parent 58ce653908
commit d3fc056743
3 changed files with 103 additions and 3 deletions

View File

@@ -0,0 +1,69 @@
---
description: "Validate the lifecycle of an extension from the catalog."
---
# Extension Self-Test: `$ARGUMENTS`
This command drives a self-test simulating the developer experience with the `$ARGUMENTS` extension.
## Goal
Validate the end-to-end lifecycle (discovery, installation, registration) for the extension: `$ARGUMENTS`.
If `$ARGUMENTS` is empty, you must tell the user to provide an extension name, for example: `/speckit.selftest.extension linear`.
## Steps
### Step 1: Catalog Discovery Validation
Check if the extension exists in the Spec Kit catalog.
Execute this command and verify that it completes successfully and that the returned extension ID exactly matches `$ARGUMENTS`. If the command fails or the ID does not match `$ARGUMENTS`, fail the test.
```bash
specify extension info "$ARGUMENTS"
```
### Step 2: Simulate Installation
First, try to add the extension to the current workspace configuration directly. If the catalog provides the extension as `install_allowed: false` (discovery-only), this step is *expected* to fail.
```bash
specify extension add "$ARGUMENTS"
```
Then, simulate adding the extension by installing it from its catalog download URL, which should bypass the restriction.
Obtain the extension's `download_url` from the catalog metadata (for example, via a catalog info command or UI), then run:
```bash
specify extension add "$ARGUMENTS" --from "<download_url>"
```
### Step 3: Registration Verification
Once the `add` command completes, verify the installation by checking the project configuration.
Use terminal tools (like `cat`) to verify that the following file contains a record for `$ARGUMENTS`.
```bash
cat .specify/extensions/.registry/$ARGUMENTS.json
```
### Step 4: Verification Report
Analyze the standard output of the three steps.
Generate a terminal-style test output format detailing the results of discovery, installation, and registration. Return this directly to the user.
Example output format:
```text
============================= test session starts ==============================
collected 3 items
test_selftest_discovery.py::test_catalog_search [PASS/FAIL]
Details: [Provide execution result of specify extension search]
test_selftest_installation.py::test_extension_add [PASS/FAIL]
Details: [Provide execution result of specify extension add]
test_selftest_registration.py::test_config_verification [PASS/FAIL]
Details: [Provide execution result of registry record verification]
============================== [X] passed in ... ==============================
```

View File

@@ -0,0 +1,16 @@
schema_version: "1.0"
extension:
id: selftest
name: Spec Kit Self-Test Utility
version: 1.0.0
description: Verifies catalog extensions by programmatically walking through the discovery, installation, and registration lifecycle.
author: spec-kit-core
repository: https://github.com/github/spec-kit
license: MIT
requires:
speckit_version: ">=0.2.0"
provides:
commands:
- name: speckit.selftest.extension
file: commands/selftest.md
description: Validate the lifecycle of an extension from the catalog.