feat(extensions): support multiple active catalogs simultaneously (#1720)

* Initial plan * feat(extensions): implement multi-catalog stack support - Add CatalogEntry dataclass to represent catalog entries - Add get_active_catalogs() reading SPECKIT_CATALOG_URL, project config, user config, or built-in default stack (org-approved + community) - Add _load_catalog_config() to parse .specify/extension-catalogs.yml - Add _validate_catalog_url() HTTPS validation helper - Add _fetch_single_catalog() with per-URL caching, backward-compat for DEFAULT_CATALOG_URL - Add _get_merged_extensions() that merges all catalogs (priority wins on conflict) - Update search() and get_extension_info() to use merged results annotated with _catalog_name and _install_allowed - Update clear_cache() to also remove per-URL hash cache files - Add extension_catalogs CLI command to list active catalogs - Add catalog add/remove sub-commands for .specify/extension-catalogs.yml - Update extension_add to enforce install_allowed=false policy - Update extension_search to show source catalog per result - Update extension_info to show source catalog with install_allowed status - Add 13 new tests covering catalog stack, merge conflict resolution, install_allowed enforcement, and catalog metadata Co-authored-by: mnriem <15701806+mnriem@users.noreply.github.com> * docs: update RFC, user guide, and API reference for multi-catalog support - RFC: replace FUTURE FEATURE section with full implementation docs, add catalog stack resolution order, config file examples, merge conflict resolution, and install_allowed behavior - EXTENSION-USER-GUIDE.md: add multi-catalog section with CLI examples for catalogs/catalog-add/catalog-remove, update catalog config docs - EXTENSION-API-REFERENCE.md: add CatalogEntry class docs, update ExtensionCatalog docs with new methods and result annotations, add catalog CLI commands (catalogs, catalog add, catalog remove) Also fix extension_catalogs command to correctly show "Using built-in default catalog stack" when config file exists but has empty catalogs Co-authored-by: mnriem <15701806+mnriem@users.noreply.github.com> * Potential fix for pull request finding 'Empty except' Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com> * fix: remove extraneous f-string prefixes (ruff F541) Remove f-prefix from strings with no placeholders in catalog_remove and extension_search commands. Co-authored-by: mnriem <15701806+mnriem@users.noreply.github.com> * fix: address PR review feedback for multi-catalog support - Rename 'org-approved' catalog to 'default' - Move 'catalogs' command to 'catalog list' for consistency - Add 'description' field to CatalogEntry dataclass - Add --description option to 'catalog add' CLI command - Align install_allowed default to False in _load_catalog_config - Add user-level config detection in catalog list footer - Fix _load_catalog_config docstring (document ValidationError) - Fix test isolation for test_search_by_tag, test_search_by_query, test_search_verified_only, test_get_extension_info - Update version to 0.1.14 and CHANGELOG - Update all docs (RFC, User Guide, API Reference) * fix: wrap _load_catalog_config() calls in catalog_list with try/except - Check SPECKIT_CATALOG_URL first (matching get_active_catalogs() resolution order) - Wrap both _load_catalog_config() calls in try/except ValidationError so a malformed config file cannot crash `specify extension catalog list` after the active catalogs have already been printed successfully Co-authored-by: mnriem <15701806+mnriem@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: mnriem <15701806+mnriem@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
2026-03-16 18:33:07 +00:00 · 2026-03-09 14:30:27 -05:00
parent 1df24f1953
commit ee922cbde9
7 changed files with 1287 additions and 96 deletions
--- a/extensions/EXTENSION-API-REFERENCE.md
+++ b/extensions/EXTENSION-API-REFERENCE.md
@@ -243,6 +243,34 @@ manager.check_compatibility(
 )  # Raises: CompatibilityError if incompatible
 ```

+### CatalogEntry
+
+**Module**: `specify_cli.extensions`
+
+Represents a single catalog in the active catalog stack.
+
+```python
+from specify_cli.extensions import CatalogEntry
+
+entry = CatalogEntry(
+    url="https://example.com/catalog.json",
+    name="default",
+    priority=1,
+    install_allowed=True,
+    description="Built-in catalog of installable extensions",
+)
+```
+
+**Fields**:
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `url` | `str` | Catalog URL (must use HTTPS, or HTTP for localhost) |
+| `name` | `str` | Human-readable catalog name |
+| `priority` | `int` | Sort order (lower = higher priority, wins on conflicts) |
+| `install_allowed` | `bool` | Whether extensions from this catalog can be installed |
+| `description` | `str` | Optional human-readable description of the catalog (default: empty) |
+
 ### ExtensionCatalog

 **Module**: `specify_cli.extensions`
@@ -253,30 +281,67 @@ from specify_cli.extensions import ExtensionCatalog
 catalog = ExtensionCatalog(project_root)
 ```

+**Class attributes**:
+
+```python
+ExtensionCatalog.DEFAULT_CATALOG_URL    # default catalog URL
+ExtensionCatalog.COMMUNITY_CATALOG_URL  # community catalog URL
+```
+
 **Methods**:

 ```python
-# Fetch catalog
+# Get the ordered list of active catalogs
+entries = catalog.get_active_catalogs()  # List[CatalogEntry]
+
+# Fetch catalog (primary catalog, backward compat)
 catalog_data = catalog.fetch_catalog(force_refresh: bool = False)  # Dict

-# Search extensions
+# Search extensions across all active catalogs
+# Each result includes _catalog_name and _install_allowed
 results = catalog.search(
    query: Optional[str] = None,
    tag: Optional[str] = None,
    author: Optional[str] = None,
    verified_only: bool = False
-)  # Returns: List[Dict]
+)  # Returns: List[Dict]  — each dict includes _catalog_name, _install_allowed

-# Get extension info
+# Get extension info (searches all active catalogs)
+# Returns None if not found; includes _catalog_name and _install_allowed
 ext_info = catalog.get_extension_info(extension_id: str)  # Optional[Dict]

-# Check cache validity
+# Check cache validity (primary catalog)
 is_valid = catalog.is_cache_valid()  # bool

-# Clear cache
+# Clear all catalog caches
 catalog.clear_cache()
 ```

+**Result annotation fields**:
+
+Each extension dict returned by `search()` and `get_extension_info()` includes:
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `_catalog_name` | `str` | Name of the source catalog |
+| `_install_allowed` | `bool` | Whether installation is allowed from this catalog |
+
+**Catalog config file** (`.specify/extension-catalogs.yml`):
+
+```yaml
+catalogs:
+  - name: "default"
+    url: "https://raw.githubusercontent.com/github/spec-kit/main/extensions/catalog.json"
+    priority: 1
+    install_allowed: true
+    description: "Built-in catalog of installable extensions"
+  - name: "community"
+    url: "https://raw.githubusercontent.com/github/spec-kit/main/extensions/catalog.community.json"
+    priority: 2
+    install_allowed: false
+    description: "Community-contributed extensions (discovery only)"
+```
+
 ### HookExecutor

 **Module**: `specify_cli.extensions`
@@ -543,6 +608,39 @@ EXECUTE_COMMAND: {command}

 **Output**: List of installed extensions with metadata

+### extension catalog list
+
+**Usage**: `specify extension catalog list`
+
+Lists all active catalogs in the current catalog stack, showing name, description, URL, priority, and `install_allowed` status.
+
+### extension catalog add
+
+**Usage**: `specify extension catalog add URL [OPTIONS]`
+
+**Options**:
+
+- `--name NAME` - Catalog name (required)
+- `--priority INT` - Priority (lower = higher priority, default: 10)
+- `--install-allowed / --no-install-allowed` - Allow installs from this catalog (default: false)
+- `--description TEXT` - Optional description of the catalog
+
+**Arguments**:
+
+- `URL` - Catalog URL (must use HTTPS)
+
+Adds a catalog entry to `.specify/extension-catalogs.yml`.
+
+### extension catalog remove
+
+**Usage**: `specify extension catalog remove NAME`
+
+**Arguments**:
+
+- `NAME` - Catalog name to remove
+
+Removes a catalog entry from `.specify/extension-catalogs.yml`.
+
 ### extension add

 **Usage**: `specify extension add EXTENSION [OPTIONS]`
@@ -551,13 +649,13 @@ EXECUTE_COMMAND: {command}

 - `--from URL` - Install from custom URL
 - `--dev PATH` - Install from local directory
- `--version VERSION` - Install specific version
- `--no-register` - Skip command registration

 **Arguments**:

 - `EXTENSION` - Extension name or URL

+**Note**: Extensions from catalogs with `install_allowed: false` cannot be installed via this command.
+
 ### extension remove

 **Usage**: `specify extension remove EXTENSION [OPTIONS]`
@@ -575,6 +673,8 @@ EXECUTE_COMMAND: {command}

 **Usage**: `specify extension search [QUERY] [OPTIONS]`

+Searches all active catalogs simultaneously. Results include source catalog name and install_allowed status.
+
 **Options**:

 - `--tag TAG` - Filter by tag
@@ -589,6 +689,8 @@ EXECUTE_COMMAND: {command}

 **Usage**: `specify extension info EXTENSION`

+Shows source catalog and install_allowed status.
+
 **Arguments**:

 - `EXTENSION` - Extension ID
--- a/extensions/EXTENSION-USER-GUIDE.md
+++ b/extensions/EXTENSION-USER-GUIDE.md
@@ -76,7 +76,7 @@ vim .specify/extensions/jira/jira-config.yml

 ## Finding Extensions

-**Note**: By default, `specify extension search` uses your organization's catalog (`catalog.json`). If the catalog is empty, you won't see any results. See [Extension Catalogs](#extension-catalogs) to learn how to populate your catalog from the community reference catalog.
+`specify extension search` searches **all active catalogs** simultaneously, including the community catalog by default. Results are annotated with their source catalog and install status.

 ### Browse All Extensions

@@ -84,7 +84,7 @@ vim .specify/extensions/jira/jira-config.yml
 specify extension search
 ```

-Shows all extensions in your organization's catalog.
+Shows all extensions across all active catalogs (default and community by default).

 ### Search by Keyword

@@ -402,13 +402,13 @@ In addition to extension-specific environment variables (`SPECKIT_{EXT_ID}_*`),

 | Variable | Description | Default |
 |----------|-------------|---------|
-| `SPECKIT_CATALOG_URL`       | Override the extension catalog URL | GitHub-hosted catalog |
+| `SPECKIT_CATALOG_URL`       | Override the full catalog stack with a single URL (backward compat) | Built-in default stack |
 | `GH_TOKEN` / `GITHUB_TOKEN` | GitHub API token for downloads     | None                  |

 #### Example: Using a custom catalog for testing

 ```bash
-# Point to a local or alternative catalog
+# Point to a local or alternative catalog (replaces the full stack)
 export SPECKIT_CATALOG_URL="http://localhost:8000/catalog.json"

 # Or use a staging catalog
@@ -419,13 +419,76 @@ export SPECKIT_CATALOG_URL="https://example.com/staging/catalog.json"

 ## Extension Catalogs

-For information about how Spec Kit's dual-catalog system works (`catalog.json` vs `catalog.community.json`), see the main [Extensions README](README.md#extension-catalogs).
+Spec Kit uses a **catalog stack** — an ordered list of catalogs searched simultaneously. By default, two catalogs are active:
+
+| Priority | Catalog | Install Allowed | Purpose |
+|----------|---------|-----------------|---------|
+| 1 | `catalog.json` (default) | ✅ Yes | Curated extensions available for installation |
+| 2 | `catalog.community.json` (community) | ❌ No (discovery only) | Browse community extensions |
+
+### Listing Active Catalogs
+
+```bash
+specify extension catalog list
+```
+
+### Adding a Catalog (Project-scoped)
+
+```bash
+# Add an internal catalog that allows installs
+specify extension catalog add \
+  --name "internal" \
+  --priority 2 \
+  --install-allowed \
+  https://internal.company.com/spec-kit/catalog.json
+
+# Add a discovery-only catalog
+specify extension catalog add \
+  --name "partner" \
+  --priority 5 \
+  https://partner.example.com/spec-kit/catalog.json
+```
+
+This creates or updates `.specify/extension-catalogs.yml`.
+
+### Removing a Catalog
+
+```bash
+specify extension catalog remove internal
+```
+
+### Manual Config File
+
+You can also edit `.specify/extension-catalogs.yml` directly:
+
+```yaml
+catalogs:
+  - name: "default"
+    url: "https://raw.githubusercontent.com/github/spec-kit/main/extensions/catalog.json"
+    priority: 1
+    install_allowed: true
+    description: "Built-in catalog of installable extensions"
+
+  - name: "internal"
+    url: "https://internal.company.com/spec-kit/catalog.json"
+    priority: 2
+    install_allowed: true
+    description: "Internal company extensions"
+
+  - name: "community"
+    url: "https://raw.githubusercontent.com/github/spec-kit/main/extensions/catalog.community.json"
+    priority: 3
+    install_allowed: false
+    description: "Community-contributed extensions (discovery only)"
+```
+
+A user-level equivalent lives at `~/.specify/extension-catalogs.yml`. Project-level config takes full precedence when it contains one or more catalog entries. An empty `catalogs: []` list falls back to built-in defaults.

 ## Organization Catalog Customization

 ### Why Customize Your Catalog

-Organizations customize their `catalog.json` to:
+Organizations customize their catalogs to:

 - **Control available extensions** - Curate which extensions your team can install
 - **Host private extensions** - Internal tools that shouldn't be public
@@ -503,24 +566,40 @@ Options for hosting your catalog:

 #### 3. Configure Your Environment

-##### Option A: Environment variable (recommended for CI/CD)
+##### Option A: Catalog stack config file (recommended)
+
+Add to `.specify/extension-catalogs.yml` in your project:
+
+```yaml
+catalogs:
+  - name: "my-org"
+    url: "https://your-org.com/spec-kit/catalog.json"
+    priority: 1
+    install_allowed: true
+```
+
+Or use the CLI:
+
+```bash
+specify extension catalog add \
+  --name "my-org" \
+  --install-allowed \
+  https://your-org.com/spec-kit/catalog.json
+```
+
+##### Option B: Environment variable (recommended for CI/CD, single-catalog)

 ```bash
 # In ~/.bashrc, ~/.zshrc, or CI pipeline
 export SPECKIT_CATALOG_URL="https://your-org.com/spec-kit/catalog.json"
 ```

-##### Option B: Per-project configuration
-
-Create `.env` or set in your shell before running spec-kit commands:
-
-```bash
-SPECKIT_CATALOG_URL="https://your-org.com/spec-kit/catalog.json" specify extension search
-```
-
 #### 4. Verify Configuration

 ```bash
+# List active catalogs
+specify extension catalog list
+
 # Search should now show your catalog's extensions
 specify extension search

--- a/extensions/RFC-EXTENSION-SYSTEM.md
+++ b/extensions/RFC-EXTENSION-SYSTEM.md
@@ -868,7 +868,7 @@ Spec Kit uses two catalog files with different purposes:

 - **Purpose**: Organization's curated catalog of approved extensions
 - **Default State**: Empty by design - users populate with extensions they trust
- **Usage**: Default catalog used by `specify extension` CLI commands
+- **Usage**: Primary catalog (priority 1, `install_allowed: true`) in the default stack
 - **Control**: Organizations maintain their own fork/version for their teams

 #### Community Reference Catalog (`catalog.community.json`)
@@ -879,16 +879,16 @@ Spec Kit uses two catalog files with different purposes:
 - **Verification**: Community extensions may have `verified: false` initially
 - **Status**: Active - open for community contributions
 - **Submission**: Via Pull Request following the Extension Publishing Guide
- **Usage**: Browse to discover extensions, then copy to your `catalog.json`
+- **Usage**: Secondary catalog (priority 2, `install_allowed: false`) in the default stack — discovery only

-**How It Works:**
+**How It Works (default stack):**

-1. **Discover**: Browse `catalog.community.json` to find available extensions
-2. **Review**: Evaluate extensions for security, quality, and organizational fit
-3. **Curate**: Copy approved extension entries from community catalog to your `catalog.json`
-4. **Install**: Use `specify extension add <name>` (pulls from your curated catalog)
+1. **Discover**: `specify extension search` searches both catalogs — community extensions appear automatically
+2. **Review**: Evaluate community extensions for security, quality, and organizational fit
+3. **Curate**: Copy approved entries from community catalog to your `catalog.json`, or add to `.specify/extension-catalogs.yml` with `install_allowed: true`
+4. **Install**: Use `specify extension add <name>` — only allowed from `install_allowed: true` catalogs

-This approach gives organizations full control over which extensions are available to their teams while maintaining a shared community resource for discovery.
+This approach gives organizations full control over which extensions can be installed while still providing community discoverability out of the box.

 ### Catalog Format

@@ -961,30 +961,92 @@ specify extension info jira

 ### Custom Catalogs

-**⚠️ FUTURE FEATURE - NOT YET IMPLEMENTED**
+Spec Kit supports a **catalog stack** — an ordered list of catalogs that the CLI merges and searches across. This allows organizations to maintain their own org-approved extensions alongside an internal catalog and community discovery, all at once.

-The following catalog management commands are proposed design concepts but are not yet available in the current implementation:
+#### Catalog Stack Resolution

-```bash
-# Add custom catalog (FUTURE - NOT AVAILABLE)
-specify extension add-catalog https://internal.company.com/spec-kit/catalog.json
+The active catalog stack is resolved in this order (first match wins):

-# Set as default (FUTURE - NOT AVAILABLE)
-specify extension set-catalog --default https://internal.company.com/spec-kit/catalog.json
+1. **`SPECKIT_CATALOG_URL` environment variable** — single catalog replacing all defaults (backward compat)
+2. **Project-level `.specify/extension-catalogs.yml`** — full control for the project
+3. **User-level `~/.specify/extension-catalogs.yml`** — personal defaults
+4. **Built-in default stack** — `catalog.json` (install_allowed: true) + `catalog.community.json` (install_allowed: false)

-# List catalogs (FUTURE - NOT AVAILABLE)
-specify extension catalogs
+#### Default Built-in Stack
+
+When no config file exists, the CLI uses:
+
+| Priority | Catalog | install_allowed | Purpose |
+|----------|---------|-----------------|---------|
+| 1 | `catalog.json` (default) | `true` | Curated extensions available for installation |
+| 2 | `catalog.community.json` (community) | `false` | Discovery only — browse but not install |
+
+This means `specify extension search` surfaces community extensions out of the box, while `specify extension add` is still restricted to entries from catalogs with `install_allowed: true`.
+
+#### `.specify/extension-catalogs.yml` Config File
+
+```yaml
+catalogs:
+  - name: "default"
+    url: "https://raw.githubusercontent.com/github/spec-kit/main/extensions/catalog.json"
+    priority: 1          # Highest — only approved entries can be installed
+    install_allowed: true
+    description: "Built-in catalog of installable extensions"
+
+  - name: "internal"
+    url: "https://internal.company.com/spec-kit/catalog.json"
+    priority: 2
+    install_allowed: true
+    description: "Internal company extensions"
+
+  - name: "community"
+    url: "https://raw.githubusercontent.com/github/spec-kit/main/extensions/catalog.community.json"
+    priority: 3          # Lowest — discovery only, not installable
+    install_allowed: false
+    description: "Community-contributed extensions (discovery only)"
 ```

-**Proposed catalog priority** (future design):
+A user-level equivalent lives at `~/.specify/extension-catalogs.yml`. When a project-level config is present with one or more catalog entries, it takes full control and the built-in defaults are not applied. An empty `catalogs: []` list is treated the same as no config file, falling back to defaults.

-1. Project-specific catalog (`.specify/extension-catalogs.yml`) - *not implemented*
-2. User-level catalog (`~/.specify/extension-catalogs.yml`) - *not implemented*
-3. Default GitHub catalog
+#### Catalog CLI Commands

-#### Current Implementation: SPECKIT_CATALOG_URL
+```bash
+# List active catalogs with name, URL, priority, and install_allowed
+specify extension catalog list

-**The currently available method** for using custom catalogs is the `SPECKIT_CATALOG_URL` environment variable:
+# Add a catalog (project-scoped)
+specify extension catalog add --name "internal" --install-allowed \
+  https://internal.company.com/spec-kit/catalog.json
+
+# Add a discovery-only catalog
+specify extension catalog add --name "community" \
+  https://raw.githubusercontent.com/github/spec-kit/main/extensions/catalog.community.json
+
+# Remove a catalog
+specify extension catalog remove internal
+
+# Show which catalog an extension came from
+specify extension info jira
+# → Source catalog: default
+```
+
+#### Merge Conflict Resolution
+
+When the same extension `id` appears in multiple catalogs, the higher-priority (lower priority number) catalog wins. Extensions from lower-priority catalogs with the same `id` are ignored.
+
+#### `install_allowed: false` Behavior
+
+Extensions from discovery-only catalogs are shown in `specify extension search` results but cannot be installed directly:
+
+```
+⚠  'linear' is available in the 'community' catalog but installation is not allowed from that catalog.
+
+To enable installation, add 'linear' to an approved catalog (install_allowed: true) in .specify/extension-catalogs.yml.
+```
+
+#### `SPECKIT_CATALOG_URL` (Backward Compatibility)
+
+The `SPECKIT_CATALOG_URL` environment variable still works — it is treated as a single `install_allowed: true` catalog, **replacing both defaults** for full backward compatibility:

 ```bash
 # Point to your organization's catalog