Files
automaker/apps/server/tests/unit/providers/provider-factory.test.ts
Stefan de Vogelaere 0b92349890 feat: Add GitHub Copilot SDK provider integration (#661)
* feat: add GitHub Copilot SDK provider integration

Adds comprehensive GitHub Copilot SDK provider support including:
- CopilotProvider class with CLI detection and OAuth authentication check
- Copilot models definition with GPT-4o, Claude, and o1/o3 series models
- Settings UI integration with provider tab, model configuration, and navigation
- Onboarding flow integration with Copilot setup step
- Model selector integration for all phase-specific model dropdowns
- Persistence of enabled models and default model settings via API sync
- Server route for Copilot CLI status endpoint

https://claude.ai/code/session_01D26w7ZyEzP4H6Dor3ttk9d

* chore: update package-lock.json

https://claude.ai/code/session_01D26w7ZyEzP4H6Dor3ttk9d

* refactor: rename Copilot SDK to Copilot CLI and use GitHub icon

- Update all references from "GitHub Copilot SDK" to "GitHub Copilot CLI"
- Change install command from @github/copilot-sdk to @github/copilot
- Update CopilotIcon to use official GitHub Octocat logo
- Update error codes and comments throughout codebase

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: update Copilot model definitions and add dynamic model discovery

- Update COPILOT_MODEL_MAP with correct models from CLI (claude-sonnet-4.5,
  claude-haiku-4.5, claude-opus-4.5, claude-sonnet-4, gpt-5.x series, gpt-4.1,
  gemini-3-pro-preview)
- Change default Copilot model to copilot-claude-sonnet-4.5
- Add model caching methods to CopilotProvider (hasCachedModels,
  clearModelCache, refreshModels)
- Add API routes for dynamic model discovery:
  - GET /api/setup/copilot/models
  - POST /api/setup/copilot/models/refresh
  - POST /api/setup/copilot/cache/clear

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor: use @github/copilot-sdk instead of direct CLI calls

- Install @github/copilot-sdk package for proper SDK integration
- Rewrite CopilotProvider to use SDK's CopilotClient API
- Use client.createSession() for session management
- Handle SDK events (assistant.message, tool.execution_*, session.idle)
- Auto-approve permissions for autonomous agent operation
- Remove incorrect CLI flags (--mode, --output-format)
- Update default model to claude-sonnet-4.5

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: add Copilot and Gemini model support to model resolver

- Import isCopilotModel and isGeminiModel from types
- Add explicit checks for copilot- and gemini- prefixed models
- Pass through Copilot/Gemini models unchanged to their providers
- Update resolver documentation to list all supported providers

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: pass working directory to Copilot SDK and reduce event noise

- Create CopilotClient per execution with correct cwd from options.cwd
- This ensures the CLI operates in the correct project directory, not the
  server's current directory
- Skip assistant.message_delta events (they create excessive noise)
- Only yield the final assistant.message event which has complete content
- Clean up client on completion and error paths

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: simplify Copilot SDK execution with sendAndWait

- Use sendAndWait() instead of manual event polling for more reliable
  execution
- Disable streaming (streaming: false) to simplify response handling
- Increase timeout to 10 minutes for agentic operations
- Still capture tool execution events for UI display
- Add more debug logging for troubleshooting
- This should fix the "invalid_request_body" error on subsequent calls

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: allow Copilot model IDs with claude-, gemini-, gpt- prefixes

Copilot's bare model IDs legitimately contain prefixes like claude-,
gemini-, gpt- because those are the actual model names from the
Copilot CLI (e.g., claude-sonnet-4.5, gemini-3-pro-preview, gpt-5.1).

The generic validateBareModelId function was incorrectly rejecting
these valid model IDs. Now we only check that the copilot- prefix
has been stripped by the ProviderFactory.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat: enable real-time streaming of tool events for Copilot

- Switch back to streaming mode (streaming: true) for real-time events
- Use async queue pattern to bridge SDK callbacks to async generator
- Events are now yielded as they happen, not batched at the end
- Tool calls (Read, Write, Edit, Bash, TodoWrite, etc.) show in real-time
- Better progress visibility during agentic operations

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat: expand Copilot tool name and input normalization

Tool name mapping additions:
- view → Read (Copilot's file viewing tool)
- create_file → Write
- replace, patch → Edit
- run_shell_command, terminal → Bash
- search_file_content → Grep
- list_directory → Ls
- google_web_search → WebSearch
- report_intent → ReportIntent (Copilot-specific planning)
- think, plan → Think, Plan

Input normalization improvements:
- Read/Write/Edit: Map file, filename, filePath → file_path
- Bash: Map cmd, script → command
- Grep: Map query, search, regex → pattern

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: convert git+ssh to git+https in package-lock.json

The @electron/node-gyp dependency was resolved with a git+ssh URL
which fails in CI environments without SSH keys. Convert to HTTPS.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: address code review feedback for Copilot SDK provider

- Add guard for non-text prompts (vision not yet supported)
- Clear runtime model cache on fetch failure
- Fix race condition in async queue error handling
- Import CopilotAuthStatus from shared types
- Fix comment mismatch for default model constant
- Add auth-copilot and deauth-copilot routes
- Extract shared tool normalization utilities
- Create base model configuration UI component
- Add comprehensive unit tests for CopilotProvider
- Replace magic strings with constants
- Add debug logging for cleanup errors

* fix: address CodeRabbit review nitpicks

- Fix test mocks to include --version check for CLI detection
- Add aria-label for accessibility on refresh button
- Ensure default model checkbox always appears checked/enabled

* fix: address CodeRabbit review feedback

- Fix test mocks by creating fresh provider instances after mock setup
- Extract COPILOT_DISCONNECTED_MARKER_FILE constant to common.ts
- Add AUTONOMOUS MODE comment explaining auto-approval of permissions
- Improve tool-normalization with union types and null guards
- Handle 'canceled' (American spelling) status in todo normalization

* refactor: extract copilot connection logic to service and fix test mocks

- Create copilot-connection-service.ts with connect/disconnect logic
- Update auth-copilot and deauth-copilot routes to use service
- Fix test mocks for CLI detection:
  - Mock fs.existsSync for CLI path validation
  - Mock which/where command for CLI path detection

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-23 14:48:33 +01:00

332 lines
12 KiB
TypeScript

import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
import { ProviderFactory } from '@/providers/provider-factory.js';
import { ClaudeProvider } from '@/providers/claude-provider.js';
import { CursorProvider } from '@/providers/cursor-provider.js';
import { CodexProvider } from '@/providers/codex-provider.js';
import { OpencodeProvider } from '@/providers/opencode-provider.js';
import { GeminiProvider } from '@/providers/gemini-provider.js';
import { CopilotProvider } from '@/providers/copilot-provider.js';
describe('provider-factory.ts', () => {
let consoleSpy: any;
let detectClaudeSpy: any;
let detectCursorSpy: any;
let detectCodexSpy: any;
let detectOpencodeSpy: any;
let detectGeminiSpy: any;
let detectCopilotSpy: any;
beforeEach(() => {
consoleSpy = {
warn: vi.spyOn(console, 'warn').mockImplementation(() => {}),
};
// Avoid hitting real CLI / filesystem checks during unit tests
detectClaudeSpy = vi
.spyOn(ClaudeProvider.prototype, 'detectInstallation')
.mockResolvedValue({ installed: true });
detectCursorSpy = vi
.spyOn(CursorProvider.prototype, 'detectInstallation')
.mockResolvedValue({ installed: true });
detectCodexSpy = vi
.spyOn(CodexProvider.prototype, 'detectInstallation')
.mockResolvedValue({ installed: true });
detectOpencodeSpy = vi
.spyOn(OpencodeProvider.prototype, 'detectInstallation')
.mockResolvedValue({ installed: true });
detectGeminiSpy = vi
.spyOn(GeminiProvider.prototype, 'detectInstallation')
.mockResolvedValue({ installed: true });
detectCopilotSpy = vi
.spyOn(CopilotProvider.prototype, 'detectInstallation')
.mockResolvedValue({ installed: true });
});
afterEach(() => {
consoleSpy.warn.mockRestore();
detectClaudeSpy.mockRestore();
detectCursorSpy.mockRestore();
detectCodexSpy.mockRestore();
detectOpencodeSpy.mockRestore();
detectGeminiSpy.mockRestore();
detectCopilotSpy.mockRestore();
});
describe('getProviderForModel', () => {
describe('Claude models (claude-* prefix)', () => {
it('should return ClaudeProvider for claude-opus-4-5-20251101', () => {
const provider = ProviderFactory.getProviderForModel('claude-opus-4-5-20251101');
expect(provider).toBeInstanceOf(ClaudeProvider);
});
it('should return ClaudeProvider for claude-sonnet-4-20250514', () => {
const provider = ProviderFactory.getProviderForModel('claude-sonnet-4-20250514');
expect(provider).toBeInstanceOf(ClaudeProvider);
});
it('should return ClaudeProvider for claude-haiku-4-5', () => {
const provider = ProviderFactory.getProviderForModel('claude-haiku-4-5');
expect(provider).toBeInstanceOf(ClaudeProvider);
});
it('should be case-insensitive for claude models', () => {
const provider = ProviderFactory.getProviderForModel('CLAUDE-OPUS-4-5-20251101');
expect(provider).toBeInstanceOf(ClaudeProvider);
});
});
describe('Claude aliases', () => {
it("should return ClaudeProvider for 'haiku'", () => {
const provider = ProviderFactory.getProviderForModel('haiku');
expect(provider).toBeInstanceOf(ClaudeProvider);
});
it("should return ClaudeProvider for 'sonnet'", () => {
const provider = ProviderFactory.getProviderForModel('sonnet');
expect(provider).toBeInstanceOf(ClaudeProvider);
});
it("should return ClaudeProvider for 'opus'", () => {
const provider = ProviderFactory.getProviderForModel('opus');
expect(provider).toBeInstanceOf(ClaudeProvider);
});
it('should be case-insensitive for aliases', () => {
const provider1 = ProviderFactory.getProviderForModel('HAIKU');
const provider2 = ProviderFactory.getProviderForModel('Sonnet');
const provider3 = ProviderFactory.getProviderForModel('Opus');
expect(provider1).toBeInstanceOf(ClaudeProvider);
expect(provider2).toBeInstanceOf(ClaudeProvider);
expect(provider3).toBeInstanceOf(ClaudeProvider);
});
});
describe('Cursor models (cursor-* prefix)', () => {
it('should return CursorProvider for cursor-auto', () => {
const provider = ProviderFactory.getProviderForModel('cursor-auto');
expect(provider).toBeInstanceOf(CursorProvider);
});
it('should return CursorProvider for cursor-sonnet-4.5', () => {
const provider = ProviderFactory.getProviderForModel('cursor-sonnet-4.5');
expect(provider).toBeInstanceOf(CursorProvider);
});
it('should return CursorProvider for cursor-gpt-5.2', () => {
const provider = ProviderFactory.getProviderForModel('cursor-gpt-5.2');
expect(provider).toBeInstanceOf(CursorProvider);
});
it('should be case-insensitive for cursor models', () => {
const provider = ProviderFactory.getProviderForModel('CURSOR-AUTO');
expect(provider).toBeInstanceOf(CursorProvider);
});
it('should return CursorProvider for known cursor model ID without prefix', () => {
const provider = ProviderFactory.getProviderForModel('auto');
expect(provider).toBeInstanceOf(CursorProvider);
});
});
describe('Unknown models', () => {
it('should default to ClaudeProvider for unknown model', () => {
const provider = ProviderFactory.getProviderForModel('unknown-model-123');
expect(provider).toBeInstanceOf(ClaudeProvider);
});
it('should handle empty string by defaulting to ClaudeProvider', () => {
const provider = ProviderFactory.getProviderForModel('');
expect(provider).toBeInstanceOf(ClaudeProvider);
});
it('should default to ClaudeProvider for completely unknown prefixes', () => {
const provider = ProviderFactory.getProviderForModel('random-xyz-model');
expect(provider).toBeInstanceOf(ClaudeProvider);
});
});
describe('Cursor models via model ID lookup', () => {
it('should return CodexProvider for gpt-5.2 (Codex model, not Cursor)', () => {
// gpt-5.2 is in both CURSOR_MODEL_MAP and CODEX_MODEL_CONFIG_MAP
// It should route to Codex since Codex models take priority
const provider = ProviderFactory.getProviderForModel('gpt-5.2');
expect(provider).toBeInstanceOf(CodexProvider);
});
it('should return CursorProvider for grok (valid Cursor model)', () => {
const provider = ProviderFactory.getProviderForModel('grok');
expect(provider).toBeInstanceOf(CursorProvider);
});
it('should return CursorProvider for gemini-3-pro (valid Cursor model)', () => {
const provider = ProviderFactory.getProviderForModel('gemini-3-pro');
expect(provider).toBeInstanceOf(CursorProvider);
});
});
});
describe('getAllProviders', () => {
it('should return array of all providers', () => {
const providers = ProviderFactory.getAllProviders();
expect(Array.isArray(providers)).toBe(true);
});
it('should include ClaudeProvider', () => {
const providers = ProviderFactory.getAllProviders();
const hasClaudeProvider = providers.some((p) => p instanceof ClaudeProvider);
expect(hasClaudeProvider).toBe(true);
});
it('should return exactly 6 providers', () => {
const providers = ProviderFactory.getAllProviders();
expect(providers).toHaveLength(6);
});
it('should include CopilotProvider', () => {
const providers = ProviderFactory.getAllProviders();
const hasCopilotProvider = providers.some((p) => p instanceof CopilotProvider);
expect(hasCopilotProvider).toBe(true);
});
it('should include GeminiProvider', () => {
const providers = ProviderFactory.getAllProviders();
const hasGeminiProvider = providers.some((p) => p instanceof GeminiProvider);
expect(hasGeminiProvider).toBe(true);
});
it('should include CursorProvider', () => {
const providers = ProviderFactory.getAllProviders();
const hasCursorProvider = providers.some((p) => p instanceof CursorProvider);
expect(hasCursorProvider).toBe(true);
});
it('should create new instances each time', () => {
const providers1 = ProviderFactory.getAllProviders();
const providers2 = ProviderFactory.getAllProviders();
expect(providers1[0]).not.toBe(providers2[0]);
});
});
describe('checkAllProviders', () => {
it('should return installation status for all providers', async () => {
const statuses = await ProviderFactory.checkAllProviders();
expect(statuses).toHaveProperty('claude');
});
it('should call detectInstallation on each provider', async () => {
const statuses = await ProviderFactory.checkAllProviders();
expect(statuses.claude).toHaveProperty('installed');
});
it('should return correct provider names as keys', async () => {
const statuses = await ProviderFactory.checkAllProviders();
const keys = Object.keys(statuses);
expect(keys).toContain('claude');
expect(keys).toContain('cursor');
expect(keys).toContain('codex');
expect(keys).toContain('opencode');
expect(keys).toContain('gemini');
expect(keys).toContain('copilot');
expect(keys).toHaveLength(6);
});
it('should include cursor status', async () => {
const statuses = await ProviderFactory.checkAllProviders();
expect(statuses.cursor).toHaveProperty('installed');
});
});
describe('getProviderByName', () => {
it("should return ClaudeProvider for 'claude'", () => {
const provider = ProviderFactory.getProviderByName('claude');
expect(provider).toBeInstanceOf(ClaudeProvider);
});
it("should return ClaudeProvider for 'anthropic'", () => {
const provider = ProviderFactory.getProviderByName('anthropic');
expect(provider).toBeInstanceOf(ClaudeProvider);
});
it("should return CursorProvider for 'cursor'", () => {
const provider = ProviderFactory.getProviderByName('cursor');
expect(provider).toBeInstanceOf(CursorProvider);
});
it('should be case-insensitive', () => {
const provider1 = ProviderFactory.getProviderByName('CLAUDE');
const provider2 = ProviderFactory.getProviderByName('ANTHROPIC');
const provider3 = ProviderFactory.getProviderByName('CURSOR');
expect(provider1).toBeInstanceOf(ClaudeProvider);
expect(provider2).toBeInstanceOf(ClaudeProvider);
expect(provider3).toBeInstanceOf(CursorProvider);
});
it('should return null for unknown provider', () => {
const provider = ProviderFactory.getProviderByName('unknown');
expect(provider).toBeNull();
});
it('should return null for empty string', () => {
const provider = ProviderFactory.getProviderByName('');
expect(provider).toBeNull();
});
it('should create new instance each time', () => {
const provider1 = ProviderFactory.getProviderByName('claude');
const provider2 = ProviderFactory.getProviderByName('claude');
expect(provider1).not.toBe(provider2);
expect(provider1).toBeInstanceOf(ClaudeProvider);
expect(provider2).toBeInstanceOf(ClaudeProvider);
});
});
describe('getAllAvailableModels', () => {
it('should return array of models', () => {
const models = ProviderFactory.getAllAvailableModels();
expect(Array.isArray(models)).toBe(true);
});
it('should include models from all providers', () => {
const models = ProviderFactory.getAllAvailableModels();
expect(models.length).toBeGreaterThan(0);
});
it('should return models with required fields', () => {
const models = ProviderFactory.getAllAvailableModels();
models.forEach((model) => {
expect(model).toHaveProperty('id');
expect(model).toHaveProperty('name');
expect(typeof model.id).toBe('string');
expect(typeof model.name).toBe('string');
});
});
it('should include Claude models', () => {
const models = ProviderFactory.getAllAvailableModels();
// Claude models should include claude-* in their IDs
const hasClaudeModels = models.some((m) => m.id.toLowerCase().includes('claude'));
expect(hasClaudeModels).toBe(true);
});
it('should include Cursor models', () => {
const models = ProviderFactory.getAllAvailableModels();
// Cursor models should include cursor provider
const hasCursorModels = models.some((m) => m.provider === 'cursor');
expect(hasCursorModels).toBe(true);
});
});
});