ros/autocoder
1
0
Fork 0
mirror of https://github.com/leonvanzyl/autocoder.git synced 2026-02-02 15:23:37 +00:00
Code Issues Packages Projects Releases Wiki Activity

refactor: optimize token usage, deduplicate code, fix bugs across agents

Browse Source 
Token reduction (~40% per session, ~2.3M fewer tokens per 200-feature project):
- Agent-type-specific tool lists: coding 9, testing 5, init 5 (was 19 for all)
- Right-sized max_turns: coding 300, testing 100 (was 1000 for all)
- Trimmed coding prompt template (~150 lines removed)
- Streamlined testing prompt with batch support
- YOLO mode now strips browser testing instructions from prompt
- Added Grep, WebFetch, WebSearch to expand project session

Performance improvements:
- Rate limit retries start at ~15s with jitter (was fixed 60s)
- Post-spawn delay reduced to 0.5s (was 2s)
- Orchestrator consolidated to 1 DB query per loop (was 5-7)
- Testing agents batch 3 features per session (was 1)
- Smart context compaction preserves critical state, discards noise

Bug fixes:
- Removed ghost feature_release_testing MCP tool (wasted tokens every test session)
- Forward all 9 Vertex AI env vars to chat sessions (was missing 3)
- Fix DetachedInstanceError risk in test batch ORM access
- Prevent duplicate testing of same features in parallel mode

Code deduplication:
- _get_project_path(): 9 copies -> 1 shared utility (project_helpers.py)
- validate_project_name(): 9 copies -> 2 variants in 1 file (validation.py)
- ROOT_DIR: 10 copies -> 1 definition (chat_constants.py)
- API_ENV_VARS: 4 copies -> 1 source of truth (env_constants.py)

Security hardening:
- Unified sensitive directory blocklist (14 dirs, was two divergent lists)
- Cached get_blocked_paths() for O(1) directory listing checks
- Terminal security warning when ALLOW_REMOTE=1 exposes WebSocket
- 20 new security tests for EXTRA_READ_PATHS blocking
- Extracted _validate_command_list() and _validate_pkill_processes() helpers

Type safety:
- 87 mypy errors -> 0 across 58 source files
- Installed types-PyYAML for proper yaml stub types
- Fixed SQLAlchemy Column[T] coercions across all routers

Dead code removed:
- 13 files deleted (~2,679 lines): unused UI components, debug logs, outdated docs
- 7 unused npm packages removed (Radix UI components with 0 imports)
- AgentAvatar.tsx reduced from 615 -> 119 lines (SVGs extracted to mascotData.tsx)

New CLI options:
- --testing-batch-size (1-5) for parallel mode test batching
- --testing-feature-ids for direct multi-feature testing

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
Auto
2026-02-01 13:16:24 +02:00
parent dc5bcc4ae9
commit 94e0b05cb1
57 changed files with 1974 additions and 4300 deletions
Download Patch File Download Diff File Expand all files Collapse all files

28
server/routers/agent.py
View File

@@ -6,24 +6,15 @@ API endpoints for agent control (start/stop/pause/resume).
Uses project registry for path lookups.
"""
import re
from pathlib import Path
from fastapi import APIRouter, HTTPException
from ..schemas import AgentActionResponse, AgentStartRequest, AgentStatus
from ..services.chat_constants import ROOT_DIR
from ..services.process_manager import get_manager
def _get_project_path(project_name: str) -> Path:
"""Get project path from registry."""
import sys
root = Path(__file__).parent.parent.parent
if str(root) not in sys.path:
sys.path.insert(0, str(root))
from registry import get_project_path
return get_project_path(project_name)
from ..utils.project_helpers import get_project_path as _get_project_path
from ..utils.validation import validate_project_name
def _get_settings_defaults() -> tuple[bool, str, int]:
@@ -54,19 +45,6 @@ def _get_settings_defaults() -> tuple[bool, str, int]:
router = APIRouter(prefix="/api/projects/{project_name}/agent", tags=["agent"])
# Root directory for process manager
ROOT_DIR = Path(__file__).parent.parent.parent
def validate_project_name(name: str) -> str:
"""Validate and sanitize project name to prevent path traversal."""
if not re.match(r'^[a-zA-Z0-9_-]{1,50}$', name):
raise HTTPException(
status_code=400,
detail="Invalid project name"
)
return name
def get_project_manager(project_name: str):
"""Get the process manager for a project."""

29
server/routers/assistant_chat.py
View File

@@ -7,8 +7,6 @@ WebSocket and REST endpoints for the read-only project assistant.
import json
import logging
import re
from pathlib import Path
from typing import Optional
from fastapi import APIRouter, HTTPException, WebSocket, WebSocketDisconnect
@@ -27,30 +25,13 @@ from ..services.assistant_database import (
get_conversation,
get_conversations,
)
from ..utils.project_helpers import get_project_path as _get_project_path
from ..utils.validation import is_valid_project_name as validate_project_name
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/assistant", tags=["assistant-chat"])
# Root directory
ROOT_DIR = Path(__file__).parent.parent.parent
def _get_project_path(project_name: str) -> Optional[Path]:
"""Get project path from registry."""
import sys
root = Path(__file__).parent.parent.parent
if str(root) not in sys.path:
sys.path.insert(0, str(root))
from registry import get_project_path
return get_project_path(project_name)
def validate_project_name(name: str) -> bool:
"""Validate project name to prevent path traversal."""
return bool(re.match(r'^[a-zA-Z0-9_-]{1,50}$', name))
# ============================================================================
# Pydantic Models
@@ -145,9 +126,9 @@ async def create_project_conversation(project_name: str):
conversation = create_conversation(project_dir, project_name)
return ConversationSummary(
id=conversation.id,
project_name=conversation.project_name,
title=conversation.title,
id=int(conversation.id),
project_name=str(conversation.project_name),
title=str(conversation.title) if conversation.title else None,
created_at=conversation.created_at.isoformat() if conversation.created_at else None,
updated_at=conversation.updated_at.isoformat() if conversation.updated_at else None,
message_count=0,

26
server/routers/devserver.py
View File

@@ -7,7 +7,6 @@ Uses project registry for path lookups and project_config for command detection.
"""
import logging
import re
import sys
from pathlib import Path
@@ -27,41 +26,22 @@ from ..services.project_config import (
get_project_config,
set_dev_command,
)
from ..utils.project_helpers import get_project_path as _get_project_path
from ..utils.validation import validate_project_name
# Add root to path for registry import
# Add root to path for security module import
_root = Path(__file__).parent.parent.parent
if str(_root) not in sys.path:
sys.path.insert(0, str(_root))
from registry import get_project_path as registry_get_project_path
from security import extract_commands, get_effective_commands, is_command_allowed
logger = logging.getLogger(__name__)
def _get_project_path(project_name: str) -> Path | None:
"""Get project path from registry."""
return registry_get_project_path(project_name)
router = APIRouter(prefix="/api/projects/{project_name}/devserver", tags=["devserver"])
# ============================================================================
# Helper Functions
# ============================================================================
def validate_project_name(name: str) -> str:
"""Validate and sanitize project name to prevent path traversal."""
if not re.match(r'^[a-zA-Z0-9_-]{1,50}$', name):
raise HTTPException(
status_code=400,
detail="Invalid project name"
)
return name
def get_project_dir(project_name: str) -> Path:
"""
Get the validated project directory for a project name.

17
server/routers/expand_project.py
View File

@@ -8,7 +8,6 @@ Allows adding multiple features to existing projects via natural language.
import json
import logging
from pathlib import Path
from typing import Optional
from fastapi import APIRouter, HTTPException, WebSocket, WebSocketDisconnect
@@ -22,27 +21,13 @@ from ..services.expand_chat_session import (
list_expand_sessions,
remove_expand_session,
)
from ..utils.project_helpers import get_project_path as _get_project_path
from ..utils.validation import validate_project_name
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/expand", tags=["expand-project"])
# Root directory
ROOT_DIR = Path(__file__).parent.parent.parent
def _get_project_path(project_name: str) -> Path:
"""Get project path from registry."""
import sys
root = Path(__file__).parent.parent.parent
if str(root) not in sys.path:
sys.path.insert(0, str(root))
from registry import get_project_path
return get_project_path(project_name)
# ============================================================================

17
server/routers/features.py
View File

@@ -8,10 +8,12 @@ API endpoints for feature/test case management.
import logging
from contextlib import contextmanager
from pathlib import Path
from typing import Literal
from fastapi import APIRouter, HTTPException
from ..schemas import (
DependencyGraphEdge,
DependencyGraphNode,
DependencyGraphResponse,
DependencyUpdate,
@@ -22,6 +24,7 @@ from ..schemas import (
FeatureResponse,
FeatureUpdate,
)
from ..utils.project_helpers import get_project_path as _get_project_path
from ..utils.validation import validate_project_name
# Lazy imports to avoid circular dependencies
@@ -31,17 +34,6 @@ _Feature = None
logger = logging.getLogger(__name__)
def _get_project_path(project_name: str) -> Path:
"""Get project path from registry."""
import sys
root = Path(__file__).parent.parent.parent
if str(root) not in sys.path:
sys.path.insert(0, str(root))
from registry import get_project_path
return get_project_path(project_name)
def _get_db_classes():
"""Lazy import of database classes."""
global _create_database, _Feature
@@ -349,6 +341,7 @@ async def get_dependency_graph(project_name: str):
deps = f.dependencies or []
blocking = [d for d in deps if d not in passing_ids]
status: Literal["pending", "in_progress", "done", "blocked"]
if f.passes:
status = "done"
elif blocking:
@@ -368,7 +361,7 @@ async def get_dependency_graph(project_name: str):
))
for dep_id in deps:
edges.append({"source": dep_id, "target": f.id})
edges.append(DependencyGraphEdge(source=dep_id, target=f.id))
return DependencyGraphResponse(nodes=nodes, edges=edges)
except HTTPException:

30
server/routers/filesystem.py
View File

@@ -6,6 +6,7 @@ API endpoints for browsing the filesystem for project folder selection.
Provides cross-platform support for Windows, macOS, and Linux.
"""
import functools
import logging
import os
import re
@@ -14,6 +15,8 @@ from pathlib import Path
from fastapi import APIRouter, HTTPException, Query
from security import SENSITIVE_DIRECTORIES
# Module logger
logger = logging.getLogger(__name__)
@@ -77,17 +80,10 @@ LINUX_BLOCKED = {
"/opt",
}
# Universal blocked paths (relative to home directory)
UNIVERSAL_BLOCKED_RELATIVE = {
".ssh",
".aws",
".gnupg",
".config/gh",
".netrc",
".docker",
".kube",
".terraform",
}
# Universal blocked paths (relative to home directory).
# Delegates to the canonical SENSITIVE_DIRECTORIES set in security.py so that
# the filesystem browser and the EXTRA_READ_PATHS validator share one source of truth.
UNIVERSAL_BLOCKED_RELATIVE = SENSITIVE_DIRECTORIES
# Patterns for files that should not be shown
HIDDEN_PATTERNS = [
@@ -99,8 +95,14 @@ HIDDEN_PATTERNS = [
]
def get_blocked_paths() -> set[Path]:
"""Get the set of blocked paths for the current platform."""
@functools.lru_cache(maxsize=1)
def get_blocked_paths() -> frozenset[Path]:
"""
Get the set of blocked paths for the current platform.
Cached because the platform and home directory do not change at runtime,
and this function is called once per directory entry in list_directory().
"""
home = Path.home()
blocked = set()
@@ -119,7 +121,7 @@ def get_blocked_paths() -> set[Path]:
for rel in UNIVERSAL_BLOCKED_RELATIVE:
blocked.add((home / rel).resolve())
return blocked
return frozenset(blocked)
def is_path_blocked(path: Path) -> bool:

21
server/routers/projects.py
View File

@@ -10,6 +10,7 @@ import re
import shutil
import sys
from pathlib import Path
from typing import Any, Callable
from fastapi import APIRouter, HTTPException
@@ -24,11 +25,12 @@ from ..schemas import (
)
# Lazy imports to avoid circular dependencies
# These are initialized by _init_imports() before first use.
_imports_initialized = False
_check_spec_exists = None
_scaffold_project_prompts = None
_get_project_prompts_dir = None
_count_passing_tests = None
_check_spec_exists: Callable[..., Any] | None = None
_scaffold_project_prompts: Callable[..., Any] | None = None
_get_project_prompts_dir: Callable[..., Any] | None = None
_count_passing_tests: Callable[..., Any] | None = None
def _init_imports():
@@ -99,6 +101,7 @@ def validate_project_name(name: str) -> str:
def get_project_stats(project_dir: Path) -> ProjectStats:
"""Get statistics for a project."""
_init_imports()
assert _count_passing_tests is not None # guaranteed by _init_imports()
passing, in_progress, total = _count_passing_tests(project_dir)
percentage = (passing / total * 100) if total > 0 else 0.0
return ProjectStats(
@@ -113,6 +116,7 @@ def get_project_stats(project_dir: Path) -> ProjectStats:
async def list_projects():
"""List all registered projects."""
_init_imports()
assert _check_spec_exists is not None # guaranteed by _init_imports()
(_, _, _, list_registered_projects, validate_project_path,
get_project_concurrency, _) = _get_registry_functions()
@@ -145,6 +149,7 @@ async def list_projects():
async def create_project(project: ProjectCreate):
"""Create a new project at the specified path."""
_init_imports()
assert _scaffold_project_prompts is not None # guaranteed by _init_imports()
(register_project, _, get_project_path, list_registered_projects,
_, _, _) = _get_registry_functions()
@@ -225,6 +230,8 @@ async def create_project(project: ProjectCreate):
async def get_project(name: str):
"""Get detailed information about a project."""
_init_imports()
assert _check_spec_exists is not None # guaranteed by _init_imports()
assert _get_project_prompts_dir is not None # guaranteed by _init_imports()
(_, _, get_project_path, _, _, get_project_concurrency, _) = _get_registry_functions()
name = validate_project_name(name)
@@ -296,6 +303,7 @@ async def delete_project(name: str, delete_files: bool = False):
async def get_project_prompts(name: str):
"""Get the content of project prompt files."""
_init_imports()
assert _get_project_prompts_dir is not None # guaranteed by _init_imports()
(_, _, get_project_path, _, _, _, _) = _get_registry_functions()
name = validate_project_name(name)
@@ -307,7 +315,7 @@ async def get_project_prompts(name: str):
if not project_dir.exists():
raise HTTPException(status_code=404, detail="Project directory not found")
prompts_dir = _get_project_prompts_dir(project_dir)
prompts_dir: Path = _get_project_prompts_dir(project_dir)
def read_file(filename: str) -> str:
filepath = prompts_dir / filename
@@ -329,6 +337,7 @@ async def get_project_prompts(name: str):
async def update_project_prompts(name: str, prompts: ProjectPromptsUpdate):
"""Update project prompt files."""
_init_imports()
assert _get_project_prompts_dir is not None # guaranteed by _init_imports()
(_, _, get_project_path, _, _, _, _) = _get_registry_functions()
name = validate_project_name(name)
@@ -480,6 +489,8 @@ async def reset_project(name: str, full_reset: bool = False):
async def update_project_settings(name: str, settings: ProjectSettingsUpdate):
"""Update project-level settings (concurrency, etc.)."""
_init_imports()
assert _check_spec_exists is not None # guaranteed by _init_imports()
assert _get_project_prompts_dir is not None # guaranteed by _init_imports()
(_, _, get_project_path, _, _, get_project_concurrency,
set_project_concurrency) = _get_registry_functions()

93
server/routers/schedules.py
View File

@@ -6,12 +6,10 @@ API endpoints for managing agent schedules.
Provides CRUD operations for time-based schedule configuration.
"""
import re
import sys
from contextlib import contextmanager
from datetime import datetime, timedelta, timezone
from pathlib import Path
from typing import Generator, Tuple
from typing import TYPE_CHECKING, Generator, Tuple
from fastapi import APIRouter, HTTPException
from sqlalchemy.orm import Session
@@ -26,17 +24,21 @@ from ..schemas import (
ScheduleResponse,
ScheduleUpdate,
)
from ..utils.project_helpers import get_project_path as _get_project_path
from ..utils.validation import validate_project_name
if TYPE_CHECKING:
from api.database import Schedule as ScheduleModel
def _get_project_path(project_name: str) -> Path:
"""Get project path from registry."""
root = Path(__file__).parent.parent.parent
if str(root) not in sys.path:
sys.path.insert(0, str(root))
from registry import get_project_path
return get_project_path(project_name)
def _schedule_to_response(schedule: "ScheduleModel") -> ScheduleResponse:
"""Convert a Schedule ORM object to a ScheduleResponse Pydantic model.
SQLAlchemy Column descriptors resolve to Python types at instance access time,
but mypy sees the Column[T] descriptor type. Using model_validate with
from_attributes handles this conversion correctly.
"""
return ScheduleResponse.model_validate(schedule, from_attributes=True)
router = APIRouter(
prefix="/api/projects/{project_name}/schedules",
@@ -44,16 +46,6 @@ router = APIRouter(
)
def validate_project_name(name: str) -> str:
"""Validate and sanitize project name to prevent path traversal."""
if not re.match(r'^[a-zA-Z0-9_-]{1,50}$', name):
raise HTTPException(
status_code=400,
detail="Invalid project name"
)
return name
@contextmanager
def _get_db_session(project_name: str) -> Generator[Tuple[Session, Path], None, None]:
"""Get database session for a project as a context manager.
@@ -102,21 +94,7 @@ async def list_schedules(project_name: str):
).order_by(Schedule.start_time).all()
return ScheduleListResponse(
schedules=[
ScheduleResponse(
id=s.id,
project_name=s.project_name,
start_time=s.start_time,
duration_minutes=s.duration_minutes,
days_of_week=s.days_of_week,
enabled=s.enabled,
yolo_mode=s.yolo_mode,
model=s.model,
crash_count=s.crash_count,
created_at=s.created_at,
)
for s in schedules
]
schedules=[_schedule_to_response(s) for s in schedules]
)
@@ -190,18 +168,7 @@ async def create_schedule(project_name: str, data: ScheduleCreate):
except Exception as e:
logger.error(f"Failed to start agent for schedule {schedule.id}: {e}", exc_info=True)
return ScheduleResponse(
id=schedule.id,
project_name=schedule.project_name,
start_time=schedule.start_time,
duration_minutes=schedule.duration_minutes,
days_of_week=schedule.days_of_week,
enabled=schedule.enabled,
yolo_mode=schedule.yolo_mode,
model=schedule.model,
crash_count=schedule.crash_count,
created_at=schedule.created_at,
)
return _schedule_to_response(schedule)
@router.get("/next", response_model=NextRunResponse)
@@ -259,8 +226,8 @@ async def get_next_scheduled_run(project_name: str):
return NextRunResponse(
has_schedules=True,
next_start=next_start.isoformat() if (active_count == 0 and next_start) else None,
next_end=latest_end.isoformat() if latest_end else None,
next_start=next_start if active_count == 0 else None,
next_end=latest_end,
is_currently_running=active_count > 0,
active_schedule_count=active_count,
)
@@ -280,18 +247,7 @@ async def get_schedule(project_name: str, schedule_id: int):
if not schedule:
raise HTTPException(status_code=404, detail="Schedule not found")
return ScheduleResponse(
id=schedule.id,
project_name=schedule.project_name,
start_time=schedule.start_time,
duration_minutes=schedule.duration_minutes,
days_of_week=schedule.days_of_week,
enabled=schedule.enabled,
yolo_mode=schedule.yolo_mode,
model=schedule.model,
crash_count=schedule.crash_count,
created_at=schedule.created_at,
)
return _schedule_to_response(schedule)
@router.patch("/{schedule_id}", response_model=ScheduleResponse)
@@ -334,18 +290,7 @@ async def update_schedule(
# Was enabled, now disabled - remove jobs
scheduler.remove_schedule(schedule_id)
return ScheduleResponse(
id=schedule.id,
project_name=schedule.project_name,
start_time=schedule.start_time,
duration_minutes=schedule.duration_minutes,
days_of_week=schedule.days_of_week,
enabled=schedule.enabled,
yolo_mode=schedule.yolo_mode,
model=schedule.model,
crash_count=schedule.crash_count,
created_at=schedule.created_at,
)
return _schedule_to_response(schedule)
@router.delete("/{schedule_id}", status_code=204)

5
server/routers/settings.py
View File

@@ -9,17 +9,16 @@ Settings are stored in the registry database and shared across all projects.
import mimetypes
import os
import sys
from pathlib import Path
from fastapi import APIRouter
from ..schemas import ModelInfo, ModelsResponse, SettingsResponse, SettingsUpdate
from ..services.chat_constants import ROOT_DIR
# Mimetype fix for Windows - must run before StaticFiles is mounted
mimetypes.add_type("text/javascript", ".js", True)
# Add root to path for registry import
ROOT_DIR = Path(__file__).parent.parent.parent
# Ensure root is on sys.path for registry import
if str(ROOT_DIR) not in sys.path:
sys.path.insert(0, str(ROOT_DIR))

23
server/routers/spec_creation.py
View File

@@ -7,8 +7,6 @@ WebSocket and REST endpoints for interactive spec creation with Claude.
import json
import logging
import re
from pathlib import Path
from typing import Optional
from fastapi import APIRouter, HTTPException, WebSocket, WebSocketDisconnect
@@ -22,30 +20,13 @@ from ..services.spec_chat_session import (
list_sessions,
remove_session,
)
from ..utils.project_helpers import get_project_path as _get_project_path
from ..utils.validation import is_valid_project_name as validate_project_name
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/spec", tags=["spec-creation"])
# Root directory
ROOT_DIR = Path(__file__).parent.parent.parent
def _get_project_path(project_name: str) -> Path:
"""Get project path from registry."""
import sys
root = Path(__file__).parent.parent.parent
if str(root) not in sys.path:
sys.path.insert(0, str(root))
from registry import get_project_path
return get_project_path(project_name)
def validate_project_name(name: str) -> bool:
"""Validate project name to prevent path traversal."""
return bool(re.match(r'^[a-zA-Z0-9_-]{1,50}$', name))
# ============================================================================
# REST Endpoints

32
server/routers/terminal.py
View File

@@ -12,8 +12,6 @@ import base64
import json
import logging
import re
import sys
from pathlib import Path
from fastapi import APIRouter, HTTPException, WebSocket, WebSocketDisconnect
from pydantic import BaseModel
@@ -27,13 +25,8 @@ from ..services.terminal_manager import (
rename_terminal,
stop_terminal_session,
)
# Add project root to path for registry import
_root = Path(__file__).parent.parent.parent
if str(_root) not in sys.path:
sys.path.insert(0, str(_root))
from registry import get_project_path as registry_get_project_path
from ..utils.project_helpers import get_project_path as _get_project_path
from ..utils.validation import is_valid_project_name as validate_project_name
logger = logging.getLogger(__name__)
@@ -48,27 +41,6 @@ class TerminalCloseCode:
FAILED_TO_START = 4500
def _get_project_path(project_name: str) -> Path | None:
"""Get project path from registry."""
return registry_get_project_path(project_name)
def validate_project_name(name: str) -> bool:
"""
Validate project name to prevent path traversal attacks.
Allows only alphanumeric characters, underscores, and hyphens.
Maximum length of 50 characters.
Args:
name: The project name to validate
Returns:
True if valid, False otherwise
"""
return bool(re.match(r"^[a-zA-Z0-9_-]{1,50}$", name))
def validate_terminal_id(terminal_id: str) -> bool:
"""
Validate terminal ID format.