diff --git a/BETA-V3/docs/technical-preferences.md b/BETA-V3/docs/technical-preferences.md deleted file mode 100644 index 8726d25c..00000000 --- a/BETA-V3/docs/technical-preferences.md +++ /dev/null @@ -1,21 +0,0 @@ -# User-Defined Preferred Patterns and Preferences - -This document is intended for you, the user, to maintain a list of your preferred architectural patterns, design choices, technologies, or specific configurations - so hopefully you do not need to suggest them every time to the PM or architect. The Architect Agent will consult this file, when available, to understand your preferences and will aim to incorporate them into its recommendations, still seeking your confirmation. - -You can evolve this document over time as you discover new patterns, solidify your preferences, or wish to guide the Architect Agent more specifically for certain types of projects, platforms, languages, or architectural styles. - -## How to Use This Document - -Remove all content aside from the first line and add anything you learn or already prefer along the way for future project reference and preference. See the examples in this folder. - -No specific format required, but it is helpful to have sections to make it easier for the architect - possible sections include the following: - -## Favorite Design Patterns - -## Testing Standards - -## Code Style Specific Preferences - -## Tech Stack to Use - -## etc... diff --git a/BETA-V3/docs/technical-preferences.txt b/BETA-V3/docs/technical-preferences.txt new file mode 100644 index 00000000..d1c2f660 --- /dev/null +++ b/BETA-V3/docs/technical-preferences.txt @@ -0,0 +1,6 @@ +# User-Defined Preferred Patterns and Preferences + +See example files in this folder. +list out your technical preferences, patterns you like to follow, language framework or starter project preferences. + +nything you learn or prefer over time to drive future project choices. diff --git a/BETA-V3/v3-demos/project1/architecture.txt b/BETA-V3/v3-demos/project1/architecture.txt new file mode 100644 index 00000000..1e3623a1 --- /dev/null +++ b/BETA-V3/v3-demos/project1/architecture.txt @@ -0,0 +1,1034 @@ +# BMad DiCaster Architecture Document + +## Introduction / Preamble + +This document outlines the overall project architecture for BMad DiCaster, including backend systems, shared services, and non-UI specific concerns. Its primary goal is to serve as the guiding architectural blueprint for AI-driven development, ensuring consistency and adherence to chosen patterns and technologies. + +**Relationship to Frontend Architecture:** +This project includes a significant user interface. A separate Frontend Architecture Document (expected to be named `frontend-architecture.md` and linked in "Key Reference Documents" once created) will detail the frontend-specific design and MUST be used in conjunction with this document. Core technology stack choices documented herein (see "Definitive Tech Stack Selections") are definitive for the entire project, including any frontend components. + +## Table of Contents + +1. [Introduction / Preamble](https://www.google.com/search?q=%23introduction--preamble) +2. [Table of Contents](https://www.google.com/search?q=%23table-of-contents) +3. [Technical Summary](https://www.google.com/search?q=%23technical-summary) +4. [High-Level Overview](https://www.google.com/search?q=%23high-level-overview) +5. [Component View](https://www.google.com/search?q=%23component-view) + - [Architectural / Design Patterns Adopted](https://www.google.com/search?q=%23architectural--design-patterns-adopted) +6. [Workflow Orchestration and Status Management](https://www.google.com/search?q=%23workflow-orchestration-and-status-management) +7. [Project Structure](https://www.google.com/search?q=%23project-structure) + - [Key Directory Descriptions](https://www.google.com/search?q=%23key-directory-descriptions) + - [Monorepo Management](https://www.google.com/search?q=%23monorepo-management) + - [Notes](https://www.google.com/search?q=%23notes) +8. [API Reference](https://www.google.com/search?q=%23api-reference) + - [External APIs Consumed](https://www.google.com/search?q=%23external-apis-consumed) + - [Internal APIs Provided (by BMad DiCaster)](https://www.google.com/search?q=%23internal-apis-provided-by-bmad-dicaster) +9. [Data Models](https://www.google.com/search?q=%23data-models) + - [Core Application Entities / Domain Objects](https://www.google.com/search?q=%23core-application-entities--domain-objects) + - [Database Schemas (Supabase PostgreSQL)](https://www.google.com/search?q=%23database-schemas-supabase-postgresql) +10. [Core Workflow / Sequence Diagrams](https://www.google.com/search?q=%23core-workflow--sequence-diagrams) + - [1. Daily Workflow Initiation & HN Content Acquisition](https://www.google.com/search?q=%231-daily-workflow-initiation--hn-content-acquisition) + - [2. Article Scraping & Summarization Flow](https://www.google.com/search?q=%232-article-scraping--summarization-flow) + - [3. Newsletter, Podcast, and Delivery Flow](https://www.google.com/search?q=%233-newsletter-podcast-and-delivery-flow) +11. [Definitive Tech Stack Selections](https://www.google.com/search?q=%23definitive-tech-stack-selections) +12. [Infrastructure and Deployment Overview](https://www.google.com/search?q=%23infrastructure-and-deployment-overview) +13. [Error Handling Strategy](https://www.google.com/search?q=%23error-handling-strategy) +14. [Coding Standards](https://www.google.com/search?q=%23coding-standards) + - [Detailed Language & Framework Conventions](https://www.google.com/search?q=%23detailed-language--framework-conventions) +15. [Overall Testing Strategy](https://www.google.com/search?q=%23overall-testing-strategy) +16. [Security Best Practices](https://www.google.com/search?q=%23security-best-practices) +17. [Key Reference Documents](https://www.google.com/search?q=%23key-reference-documents) +18. [Change Log](https://www.google.com/search?q=%23change-log) +19. [Prompt for Design Architect: Frontend Architecture Definition](https://www.google.com/search?q=%23prompt-for-design-architect-frontend-architecture-definition) + +## Technical Summary + +BMad DiCaster is a web application designed to provide daily, concise summaries of top Hacker News (HN) posts, delivered as an HTML newsletter and an optional AI-generated podcast, accessible via a Next.js web interface. The system employs a serverless, event-driven architecture hosted on Vercel, with Supabase providing PostgreSQL database services and function hosting. Key components include services for HN content retrieval, article scraping (using Cheerio), AI-powered summarization (via a configurable LLM facade for Ollama/remote APIs), podcast generation (Play.ht), newsletter generation (Nodemailer), and workflow orchestration. The architecture emphasizes modularity, clear separation of concerns (pragmatic hexagonal approach for complex functions), and robust error handling, aiming for efficient development, particularly by AI developer agents. + +## High-Level Overview + +The BMad DiCaster application will adopt a **serverless, event-driven architecture** hosted entirely on Vercel, with Supabase providing backend services (database and functions). The project will be structured as a **monorepo**, containing both the Next.js frontend application and the backend Supabase functions. + +The core data processing flow is designed as an event-driven pipeline: + +1. A scheduled mechanism (Vercel Cron Job) or manual trigger (API/CLI) initiates the daily workflow, creating a `workflow_run` job. +2. Hacker News posts and comments are retrieved (HN Algolia API) and stored in Supabase. +3. This data insertion triggers a Supabase function (via database webhook) to scrape linked articles. +4. Successful article scraping and storage trigger further Supabase functions for AI-powered summarization of articles and comments. +5. The completion of summarization steps for a workflow run is tracked, and once all prerequisites are met, a newsletter generation service is triggered. +6. The newsletter content is sent to the Play.ht API to generate a podcast. +7. Play.ht calls a webhook to notify our system when the podcast is ready, providing the podcast URL. +8. The newsletter data in Supabase is updated with the podcast URL. +9. The newsletter is then delivered to subscribers via Nodemailer, after considering podcast availability (with delay/retry logic). +10. The Next.js frontend allows users to view current and past newsletters and listen to the podcasts. + +This event-driven approach, using Supabase Database Webhooks (via `pg_net` or native functionality) to trigger Vercel-hosted Supabase Functions, aims to create a resilient and scalable system. It mitigates potential timeout issues by breaking down long-running processes into smaller, asynchronously triggered units. + +Below is a system context diagram illustrating the primary services and user interactions: + +```mermaid +graph TD + User[Developer/Admin] -- "Triggers Daily Workflow (API/CLI/Cron)" --> BMadDiCasterBE[BMad DiCaster Backend Logic] + UserWeb[End User] -- "Accesses Web Interface" --> BMadDiCasterFE[BMad DiCaster Frontend (Next.js on Vercel)] + BMadDiCasterFE -- "Displays Data From" --> SupabaseDB[Supabase PostgreSQL] + BMadDiCasterFE -- "Interacts With for Data/Triggers" --> SupabaseFunctions[Supabase Functions on Vercel] + + subgraph "BMad DiCaster Backend Logic (Supabase Functions & Vercel)" + direction LR + SupabaseFunctions + HNAPI[Hacker News Algolia API] + ArticleScraper[Article Scraper Service] + Summarizer[Summarization Service (LLM Facade)] + PlayHTAPI[Play.ht API] + NewsletterService[Newsletter Generation & Delivery Service] + Nodemailer[Nodemailer Service] + end + + BMadDiCasterBE --> SupabaseDB + SupabaseFunctions -- "Fetches HN Data" --> HNAPI + SupabaseFunctions -- "Scrapes Articles" --> ArticleScraper + ArticleScraper -- "Gets URLs from" --> SupabaseDB + ArticleScraper -- "Stores Content" --> SupabaseDB + SupabaseFunctions -- "Summarizes Content" --> Summarizer + Summarizer -- "Uses Prompts from / Stores Summaries" --> SupabaseDB + SupabaseFunctions -- "Generates Podcast" --> PlayHTAPI + PlayHTAPI -- "Sends Webhook (Podcast URL)" --> SupabaseFunctions + SupabaseFunctions -- "Updates Podcast URL" --> SupabaseDB + SupabaseFunctions -- "Generates Newsletter" --> NewsletterService + NewsletterService -- "Uses Template/Data from" --> SupabaseDB + NewsletterService -- "Sends Emails Via" --> Nodemailer + SupabaseDB -- "Stores Subscriber List" --> NewsletterService + + classDef user fill:#9cf,stroke:#333,stroke-width:2px; + classDef fe fill:#f9f,stroke:#333,stroke-width:2px; + classDef be fill:#ccf,stroke:#333,stroke-width:2px; + classDef external fill:#ffc,stroke:#333,stroke-width:2px; + classDef db fill:#cfc,stroke:#333,stroke-width:2px; + + class User,UserWeb user; + class BMadDiCasterFE fe; + class BMadDiCasterBE,SupabaseFunctions,ArticleScraper,Summarizer,NewsletterService be; + class HNAPI,PlayHTAPI,Nodemailer external; + class SupabaseDB db; +``` + +## Component View + +The BMad DiCaster system is composed of several key logical components, primarily implemented as serverless functions (Supabase Functions deployed on Vercel) and a Next.js frontend application. These components work together in an event-driven manner. + +```mermaid +graph TD + subgraph FrontendApp [Frontend Application (Next.js)] + direction LR + WebAppUI["Web Application UI (React Components)"] + APIServiceFE["API Service (Frontend - Next.js Route Handlers)"] + end + + subgraph BackendServices [Backend Services (Supabase Functions & Core Logic)] + direction TB + WorkflowTriggerAPI["Workflow Trigger API (/api/system/trigger-workflow)"] + HNContentService["HN Content Service (Supabase Fn)"] + ArticleScrapingService["Article Scraping Service (Supabase Fn)"] + SummarizationService["Summarization Service (LLM Facade - Supabase Fn)"] + PodcastGenerationService["Podcast Generation Service (Supabase Fn)"] + NewsletterGenerationService["Newsletter Generation Service (Supabase Fn)"] + PlayHTWebhookHandlerAPI["Play.ht Webhook API (/api/webhooks/playht)"] + CheckWorkflowCompletionService["CheckWorkflowCompletionService (Supabase Cron Fn)"] + end + + subgraph ExternalIntegrations [External APIs & Services] + direction TB + HNAlgoliaAPI["Hacker News Algolia API"] + PlayHTAPI["Play.ht API"] + LLMProvider["LLM Provider (Ollama/Remote API)"] + NodemailerService["Nodemailer (Email Delivery)"] + end + + subgraph DataStorage [Data Storage (Supabase PostgreSQL)] + direction TB + DB_WorkflowRuns["workflow_runs Table"] + DB_Posts["hn_posts Table"] + DB_Comments["hn_comments Table"] + DB_Articles["scraped_articles Table"] + DB_Summaries["article_summaries / comment_summaries Tables"] + DB_Newsletters["newsletters Table"] + DB_Subscribers["subscribers Table"] + DB_Prompts["summarization_prompts Table"] + DB_NewsletterTemplates["newsletter_templates Table"] + end + + UserWeb[End User] --> WebAppUI + WebAppUI --> APIServiceFE + APIServiceFE --> WorkflowTriggerAPI + APIServiceFE --> DataStorage + + + DevAdmin[Developer/Admin/Cron] --> WorkflowTriggerAPI + + WorkflowTriggerAPI --> DB_WorkflowRuns + + DB_WorkflowRuns -- "Triggers (via CheckWorkflowCompletion or direct)" --> HNContentService + HNContentService --> HNAlgoliaAPI + HNContentService --> DB_Posts + HNContentService --> DB_Comments + HNContentService --> DB_WorkflowRuns + + + DB_Posts -- "Triggers (via DB Webhook)" --> ArticleScrapingService + ArticleScrapingService --> DB_Articles + ArticleScrapingService --> DB_WorkflowRuns + + DB_Articles -- "Triggers (via DB Webhook)" --> SummarizationService + SummarizationService --> LLMProvider + SummarizationService --> DB_Prompts + SummarizationService --> DB_Summaries + SummarizationService --> DB_WorkflowRuns + + CheckWorkflowCompletionService -- "Monitors & Triggers Next Steps Based On" --> DB_WorkflowRuns + CheckWorkflowCompletionService -- "Monitors & Triggers Next Steps Based On" --> DB_Summaries + CheckWorkflowCompletionService -- "Monitors & Triggers Next Steps Based On" --> DB_Newsletters + + + CheckWorkflowCompletionService --> NewsletterGenerationService + NewsletterGenerationService --> DB_NewsletterTemplates + NewsletterGenerationService --> DB_Summaries + NewsletterGenerationService --> DB_Newsletters + NewsletterGenerationService --> DB_WorkflowRuns + + + CheckWorkflowCompletionService --> PodcastGenerationService + PodcastGenerationService --> PlayHTAPI + PodcastGenerationService --> DB_Newsletters + PodcastGenerationService --> DB_WorkflowRuns + + PlayHTAPI -- "Webhook" --> PlayHTWebhookHandlerAPI + PlayHTWebhookHandlerAPI --> DB_Newsletters + PlayHTWebhookHandlerAPI --> DB_WorkflowRuns + + + CheckWorkflowCompletionService -- "Triggers Delivery" --> NewsletterGenerationService + NewsletterGenerationService -- "(For Delivery)" --> NodemailerService + NewsletterGenerationService -- "(For Delivery)" --> DB_Subscribers + NewsletterGenerationService -- "(For Delivery)" --> DB_Newsletters + NewsletterGenerationService -- "(For Delivery)" --> DB_WorkflowRuns + + + classDef user fill:#9cf,stroke:#333,stroke-width:2px; + classDef feapp fill:#f9d,stroke:#333,stroke-width:2px; + classDef beapp fill:#cdf,stroke:#333,stroke-width:2px; + classDef external fill:#ffc,stroke:#333,stroke-width:2px; + classDef db fill:#cfc,stroke:#333,stroke-width:2px; + + class UserWeb,DevAdmin user; + class FrontendApp,WebAppUI,APIServiceFE feapp; + class BackendServices,WorkflowTriggerAPI,HNContentService,ArticleScrapingService,SummarizationService,PodcastGenerationService,NewsletterGenerationService,PlayHTWebhookHandlerAPI,CheckWorkflowCompletionService beapp; + class ExternalIntegrations,HNAlgoliaAPI,PlayHTAPI,LLMProvider,NodemailerService external; + class DataStorage,DB_WorkflowRuns,DB_Posts,DB_Comments,DB_Articles,DB_Summaries,DB_Newsletters,DB_Subscribers,DB_Prompts,DB_NewsletterTemplates db; +``` + +- **Frontend Application (Next.js on Vercel):** + - **Web Application UI (React Components):** Renders UI, displays newsletters/podcasts, handles user interactions. + - **API Service (Frontend - Next.js Route Handlers):** Handles frontend-initiated API calls (e.g., for future admin functions) and receives incoming webhooks (Play.ht). +- **Backend Services (Supabase Functions & Core Logic):** + - **Workflow Trigger API (`/api/system/trigger-workflow`):** Secure Next.js API route to manually initiate the daily workflow. + - **HN Content Service (Supabase Fn):** Retrieves posts/comments from HN Algolia API, stores them. + - **Article Scraping Service (Supabase Fn):** Triggered by new HN posts, scrapes article content. + - **Summarization Service (LLM Facade - Supabase Fn):** Triggered by new articles/comments, generates summaries using LLM. + - **Podcast Generation Service (Supabase Fn):** Sends newsletter content to Play.ht API. + - **Newsletter Generation Service (Supabase Fn):** Compiles newsletter, handles podcast link logic, triggers email delivery. + - **Play.ht Webhook API (`/api/webhooks/playht`):** Next.js API route to receive podcast status from Play.ht. + - **CheckWorkflowCompletionService (Supabase Cron Fn):** Periodically monitors `workflow_runs` and related tables to orchestrate the progression between pipeline stages (e.g., from summarization to newsletter generation, then to delivery). +- **Data Storage (Supabase PostgreSQL):** Stores all application data including workflow state, content, summaries, newsletters, subscribers, prompts, and templates. +- **External APIs & Services:** HN Algolia API, Play.ht API, LLM Provider (Ollama/Remote), Nodemailer. + +### Architectural / Design Patterns Adopted + +- **Event-Driven Architecture:** Core backend processing is a series of steps triggered by database events (Supabase Database Webhooks calling Supabase Functions hosted on Vercel) and orchestrated via the `workflow_runs` table and the `CheckWorkflowCompletionService`. +- **Serverless Functions:** Backend logic is encapsulated in Supabase Functions (running on Vercel). +- **Monorepo:** All code resides in a single repository. +- **Facade Pattern:** Encapsulates interactions with external services (HN API, Play.ht API, LLM, Nodemailer) within `supabase/functions/_shared/`. +- **Factory Pattern (for LLM Service):** The `LLMFacade` will use a factory to instantiate the appropriate LLM client based on environment configuration. +- **Hexagonal Architecture (Pragmatic Application):** For complex Supabase Functions, core business logic will be separated from framework-specific handlers and data interaction code (adapters) to improve testability and maintainability. Simpler functions may have a more direct implementation. +- **Repository Pattern (for Data Access - Conceptual):** Data access logic within services will be organized, conceptually resembling repositories, even if not strictly implemented with separate repository classes for all entities in MVP Supabase Functions. +- **Configuration via Environment Variables:** All sensitive and environment-specific configurations managed via environment variables. + +## Workflow Orchestration and Status Management + +The BMad DiCaster application employs an event-driven pipeline for its daily content processing. To manage, monitor, and ensure the robust execution of this multi-step workflow, the following orchestration strategy is implemented: + +**1. Central Workflow Tracking (`workflow_runs` Table):** + +- A dedicated table, `public.workflow_runs` (defined in Data Models), serves as the single source of truth for the state and progress of each initiated daily workflow. +- Each workflow execution is identified by a unique `id` (jobId) in this table. +- Key fields include `status`, `current_step_details`, `error_message`, and a `details` JSONB column to store metadata and progress counters (e.g., `posts_fetched`, `articles_scraped_successfully`, `summaries_generated`, `podcast_playht_job_id`, `podcast_status`). + +**2. Workflow Initiation:** + +- A workflow is initiated via the `POST /api/system/trigger-workflow` API endpoint (callable manually, by CLI, or by a cron job). +- Upon successful trigger, a new record is created in `workflow_runs` with an initial status (e.g., 'pending' or 'fetching_hn'), and the `jobId` is returned to the caller. +- This initial record creation triggers the first service in the pipeline (`HNContentService`) via a database webhook or an initial direct call from the trigger API logic. + +**3. Service Function Responsibilities:** + +- Each backend Supabase Function (`HNContentService`, `ArticleScrapingService`, `SummarizationService`, `PodcastGenerationService`, `NewsletterGenerationService`) participating in the workflow **must**: + - Be aware of the `workflow_run_id` for the job it is processing. + - **Before starting its primary task:** Update the `workflow_runs` table for the current `workflow_run_id` to reflect its `current_step_details`. + - **Upon successful completion of its task:** Update relevant data tables and the `workflow_runs.details` JSONB field. + - **Upon failure:** Update the `workflow_runs` table to set `status` to 'failed', and populate `error_message` and `current_step_details`. + - Utilize the shared `WorkflowTrackerService` for consistent status updates. +- The `PlayHTWebhookHandlerAPI` updates the `newsletters` table and then the `workflow_runs.details` with podcast status. + +**4. Orchestration and Progression (`CheckWorkflowCompletionService`):** + +- A dedicated Supabase Function, `CheckWorkflowCompletionService`, will be scheduled to run periodically (e.g., every 5-10 minutes via Vercel Cron Jobs invoking a dedicated HTTP endpoint for this service, or Supabase's `pg_cron` if preferred for DB-centric scheduling). +- This service orchestrates progression between major stages by: + - Querying `workflow_runs` for jobs in intermediate statuses. + - Verifying if all prerequisites for the next stage are met (e.g., all summaries done before newsletter generation, podcast ready before delivery). + - If conditions are met, it updates `workflow_runs.status` and invokes the appropriate next service, passing the `workflow_run_id`. + +**5. Shared `WorkflowTrackerService`:** + +- A utility service in `supabase/functions/_shared/` will provide standardized methods for backend functions to interact with the `workflow_runs` table. + +**6. Podcast Link Before Email Delivery:** + +- The `NewsletterGenerationService` initiates podcast creation. +- The `CheckWorkflowCompletionService` monitors `newsletters.podcast_url` (populated by `PlayHTWebhookHandlerAPI`) or `newsletters.podcast_status`. +- Email delivery is triggered by `CheckWorkflowCompletionService` once the podcast URL is available, a timeout is reached, or podcast generation fails (as per PRD's delay/retry logic). + +## Project Structure + +The BMad DiCaster project is organized as a monorepo, leveraging the Vercel/Supabase Next.js App Router template as its foundation. + +```plaintext +{project-root}/ +├── app/ # Next.js App Router +│ ├── (api)/ # API route handlers +│ │ ├── system/ +│ │ │ ├── trigger-workflow/route.ts +│ │ │ └── workflow-status/[jobId]/route.ts +│ │ └── webhooks/ +│ │ └── playht/route.ts +│ ├── components/ # Application-specific UI react components +│ │ └── core/ # e.g., NewsletterCard, PodcastPlayer +│ ├── newsletters/ +│ │ ├── [newsletterId]/page.tsx +│ │ └── page.tsx +│ ├── auth/ # Auth-related pages and components (from template) +│ ├── login/page.tsx # Login page (from template) +│ ├── layout.tsx +│ └── page.tsx # Homepage +├── components/ # Shadcn UI components root (as configured by components.json) +│ ├── tutorial/ # Example/template components (can be removed) +│ ├── typography/ # Example/template components (can be removed) +│ └── ui/ # Base UI elements (button.tsx, card.tsx etc.) +├── docs/ # Project documentation +│ ├── prd.md +│ ├── architecture.md # This document +│ ├── ui-ux-spec.md +│ ├── ADR/ # Architecture Decision Records (to be created as needed) +│ └── environment-vars.md # (To be created) +├── lib/ # General utility functions for frontend (e.g., utils.ts from template) +│ └── utils.ts +├── supabase/ # Supabase specific project files (backend logic) +│ ├── functions/ # Supabase Edge Functions (for event-driven pipeline) +│ │ ├── hn-content-service/index.ts +│ │ ├── article-scraper-service/index.ts +│ │ ├── summarization-service/index.ts +│ │ ├── podcast-generation-service/index.ts +│ │ ├── newsletter-generation-service/index.ts +│ │ ├── check-workflow-completion-service/index.ts # Cron-triggered orchestrator +│ │ └── _shared/ # Shared utilities/facades FOR Supabase backend functions +│ │ ├── supabase-admin-client.ts +│ │ ├── llm-facade.ts +│ │ ├── playht-facade.ts +│ │ ├── nodemailer-facade.ts +│ │ └── workflow-tracker-service.ts # For updating workflow_runs table +│ ├── migrations/ # Database schema migrations +│ │ └── YYYYMMDDHHMMSS_initial_schema.sql +│ └── config.toml # Supabase project configuration (for CLI) +├── public/ # Static assets (images, favicon, etc.) +├── shared/ # Shared code/types between frontend and Supabase functions +│ └── types/ +│ ├── api-schemas.ts # Request/response types for app/(api) routes +│ ├── domain-models.ts # Core entity types (HNPost, ArticleSummary etc.) +│ └── index.ts # Barrel file for shared types +├── styles/ # Global styles (e.g., globals.css for Tailwind base) +├── tests/ # Automated tests +│ ├── e2e/ # Playwright E2E tests +│ │ ├── newsletter-view.spec.ts +│ │ └── playwright.config.ts +│ └── integration/ # Integration tests +│ └── api-trigger-workflow.integration.test.ts +│ # Unit tests are co-located with source files, e.g., app/components/core/MyComponent.test.tsx +├── utils/ # Root utilities (from template) +│ └── supabase/ # Supabase helper functions FOR FRONTEND (from template) +│ ├── client.ts # Client-side Supabase client +│ ├── middleware.ts # Logic for Next.js middleware +│ └── server.ts # Server-side Supabase client +├── .env.example +├── .gitignore +├── components.json # Shadcn UI configuration +├── middleware.ts # Next.js middleware (root, uses utils/supabase/middleware.ts) +├── next-env.d.ts +├── next.config.mjs +├── package.json +├── postcss.config.js +├── README.md +├── tailwind.config.ts +└── tsconfig.json +``` + +### Key Directory Descriptions: + +- **`app/`**: Next.js frontend (pages, UI components, Next.js API routes). + - **`app/(api)/`**: Backend API routes hosted on Vercel, including webhook receivers and system triggers. + - **`app/components/core/`**: Application-specific reusable React components. +- **`components/`**: Root for Shadcn UI components. +- **`docs/`**: All project documentation. +- **`lib/`**: Frontend-specific utility functions. +- **`supabase/functions/`**: Backend serverless functions (event-driven pipeline steps). + - **`supabase/functions/_shared/`**: Utilities and facades for these backend functions, including `WorkflowTrackerService`. +- **`supabase/migrations/`**: Database migrations managed by Supabase CLI. +- **`shared/types/`**: TypeScript types/interfaces shared between frontend and `supabase/functions/`. Path alias `@shared/*` to be configured in `tsconfig.json`. +- **`tests/`**: Contains E2E and integration tests. Unit tests are co-located with source files. +- **`utils/supabase/`**: Frontend-focused Supabase client helpers provided by the starter template. + +### Monorepo Management: + +- Standard `npm` (or `pnpm`/`yarn` workspaces if adopted later) for managing dependencies. +- The root `tsconfig.json` includes path aliases (`@/*`, `@shared/*`). + +### Notes: + +- Supabase functions in `supabase/functions/` are deployed to Vercel via Supabase CLI and Vercel integration. +- The `CheckWorkflowCompletionService` might be invoked via a Vercel Cron Job calling a simple HTTP trigger endpoint for that function, or via `pg_cron` if direct database scheduling is preferred. + +## API Reference + +### External APIs Consumed + +#### 1\. Hacker News (HN) Algolia API + +- **Purpose:** To retrieve top Hacker News posts and their associated comments. +- **Base URL(s):** Production: `http://hn.algolia.com/api/v1/` +- **Authentication:** None required. +- **Key Endpoints Used:** + - **`GET /search` (for top posts)** + - Description: Retrieves stories currently on the Hacker News front page. + - Request Parameters: `tags=front_page` + - Example Request: `curl "http://hn.algolia.com/api/v1/search?tags=front_page"` + - Post-processing: Application sorts fetched stories by `points` (descending), selects up to top 30. + - Success Response Schema (Code: `200 OK`): (Standard Algolia search response containing 'hits' array with story objects). + - **`GET /items/{objectID}` (for comments)** + - Description: Retrieves a story by `objectID` to get its full comment tree from the `children` field. Called for each selected top story. + - Success Response Schema (Code: `200 OK`): (Standard Algolia item response, `children` array contains comment tree). +- **Rate Limits:** Generous for public use; daily calls are fine. +- **Link to Official Docs:** [https://hn.algolia.com/api](https://hn.algolia.com/api) + +#### 2\. Play.ht API + +- **Purpose:** To generate AI-powered podcast versions of the newsletter content. +- **Base URL(s):** Production: `https://api.play.ai/api/v1` +- **Authentication:** API Key (`X-USER-ID` header) and Bearer Token (`Authorization` header). Stored as `PLAYHT_USER_ID` and `PLAYHT_API_KEY`. +- **Key Endpoints Used:** + - **`POST /playnotes`** + - Description: Initiates text-to-speech conversion. + - Request Body: `multipart/form-data` including `sourceFile` (HTML newsletter content), `synthesisStyle` ("podcast"), voice parameters, and `webHookUrl` (pointing to `/api/webhooks/playht` on our Vercel deployment). + - **Note on Content Delivery:** MVP uses `sourceFile` (direct upload). Fallback: upload content to Supabase Storage and provide `sourceFileUrl`. + - Success Response Schema (Code: `201 Created`): JSON object with `id` (PlayNote ID), `status`, etc. +- **Webhook Handling:** Our endpoint `/api/webhooks/playht` receives `POST` requests from Play.ht with `id`, `audioUrl`, and `status`. +- **Rate Limits:** Refer to official Play.ht documentation. +- **Link to Official Docs:** [https://docs.play.ai/api-reference/playnote/post](https://docs.play.ai/api-reference/playnote/post) + +#### 3\. LLM Provider (Facade for Summarization) + +- **Purpose:** To generate summaries for articles and comment threads. +- **Configuration:** Via environment variables (`LLM_PROVIDER_TYPE`, `OLLAMA_API_URL`, `REMOTE_LLM_API_KEY`, `REMOTE_LLM_API_URL`, `LLM_MODEL_NAME`). +- **Facade Interface (`LLMFacade` in `supabase/functions/_shared/llm-facade.ts`):** + ```typescript + export interface LLMSummarizationOptions { + /* ... */ + } + export interface LLMFacade { + generateSummary( + textToSummarize: string, + options?: LLMSummarizationOptions + ): Promise; + } + ``` +- **Implementations:** + - **Local Ollama Adapter:** HTTP requests to `OLLAMA_API_URL` (e.g., `POST /api/generate` or `/api/chat`). + - **Remote LLM API Adapter:** Authenticated HTTP requests to `REMOTE_LLM_API_URL`. +- **Rate Limits:** Provider-dependent. +- **Link to Official Docs:** Ollama: [https://github.com/ollama/ollama/blob/main/docs/api.md](https://www.google.com/search?q=https://github.com/ollama/ollama/blob/main/docs/api.md) + +#### 4\. Nodemailer (Email Delivery Service) + +- **Purpose:** To send generated HTML newsletters. +- **Interaction Type:** Library integration within `NewsletterGenerationService` (Supabase Function) via `NodemailerFacade` in `supabase/functions/_shared/nodemailer-facade.ts`. +- **Configuration:** Via SMTP environment variables (`SMTP_HOST`, `SMTP_PORT`, `SMTP_USER`, `SMTP_PASS`, etc.). +- **Key Operations:** Create transporter, construct email (From, To, Subject, HTML), send email. +- **Link to Official Docs:** [https://nodemailer.com/](https://nodemailer.com/) + +### Internal APIs Provided (by BMad DiCaster) + +#### 1\. Workflow Trigger API + +- **Purpose:** To manually initiate the daily content processing pipeline. +- **Endpoint Path:** `/api/system/trigger-workflow` (Next.js API Route Handler) +- **Method:** `POST` +- **Authentication:** API Key in `X-API-KEY` header (matches `WORKFLOW_TRIGGER_API_KEY` env var). +- **Request Body:** MVP: Empty or `{}`. +- **Success Response (`202 Accepted`):** `{"message": "Daily workflow triggered...", "jobId": ""}` +- **Action:** Creates a record in `workflow_runs` and initiates the pipeline. + +#### 2\. Workflow Status API + +- **Purpose:** Allow developers/admins to check the status of a workflow run. +- **Endpoint Path:** `/api/system/workflow-status/{jobId}` (Next.js API Route Handler) +- **Method:** `GET` +- **Authentication:** API Key in `X-API-KEY` header. +- **Success Response (`200 OK`):** JSON object with `jobId`, `status`, `currentStep`, `details`, etc. (from `workflow_runs` table). + +#### 3\. Play.ht Webhook Receiver + +- **Purpose:** To receive status updates and podcast audio URLs from Play.ht. +- **Endpoint Path:** `/api/webhooks/playht` (Next.js API Route Handler) +- **Method:** `POST` +- **Authentication:** Implement verification (shared secret or signature if Play.ht supports). +- **Request Body (from Play.ht):** JSON with `id`, `audioUrl`, `status`. +- **Action:** Updates `newsletters` and `workflow_runs` tables. + +## Data Models + +### Core Application Entities / Domain Objects + +(Conceptual types, typically defined in `shared/types/domain-models.ts`) + +#### 1\. `WorkflowRun` + +- **Description:** A single execution of the daily workflow. +- **Schema:** `id (string UUID)`, `createdAt (string ISO)`, `lastUpdatedAt (string ISO)`, `status (enum string)`, `currentStepDetails (string?)`, `errorMessage (string?)`, `details (object?)` + +#### 2\. `HNPost` + +- **Description:** A post from Hacker News. +- **Schema:** `id (string HN_objectID)`, `hnNumericId (number?)`, `title (string)`, `url (string?)`, `author (string)`, `points (number)`, `createdAt (string ISO)`, `retrievedAt (string ISO)`, `numComments (number?)`, `workflowRunId (string UUID?)` + +#### 3\. `HNComment` + +- **Description:** A comment on an HN post. +- **Schema:** `id (string HN_commentID)`, `hnPostId (string)`, `parentId (string?)`, `author (string?)`, `text (string HTML)`, `createdAt (string ISO)` + +#### 4\. `ScrapedArticle` + +- **Description:** Content scraped from an article URL. +- **Schema:** `id (string UUID)`, `hnPostId (string)`, `originalUrl (string)`, `title (string?)`, `mainTextContent (string?)`, `scrapedAt (string ISO)`, `scrapingStatus (enum string)`, `workflowRunId (string UUID?)` + +#### 5\. `ArticleSummary` + +- **Description:** AI-generated summary of a `ScrapedArticle`. +- **Schema:** `id (string UUID)`, `scrapedArticleId (string UUID)`, `summaryText (string)`, `generatedAt (string ISO)`, `llmModelUsed (string?)`, `workflowRunId (string UUID?)` + +#### 6\. `CommentSummary` + +- **Description:** AI-generated summary of comments for an `HNPost`. +- **Schema:** `id (string UUID)`, `hnPostId (string)`, `summaryText (string)`, `generatedAt (string ISO)`, `llmModelUsed (string?)`, `workflowRunId (string UUID?)` + +#### 7\. `Newsletter` + +- **Description:** The daily generated newsletter. +- **Schema:** `id (string UUID)`, `workflowRunId (string UUID)`, `title (string)`, `generatedAt (string ISO)`, `htmlContent (string)`, `podcastPlayhtJobId (string?)`, `podcastUrl (string?)`, `podcastStatus (enum string?)`, `deliveryStatus (enum string)`, `targetDate (string YYYY-MM-DD)` + +#### 8\. `Subscriber` + +- **Description:** An email subscriber. +- **Schema:** `id (string UUID)`, `email (string)`, `subscribedAt (string ISO)`, `isActive (boolean)` + +#### 9\. `SummarizationPrompt` + +- **Description:** Stores prompts for AI summarization. +- **Schema:** `id (string UUID)`, `promptName (string)`, `promptText (string)`, `version (string)`, `isDefaultArticlePrompt (boolean)`, `isDefaultCommentPrompt (boolean)` + +#### 10\. `NewsletterTemplate` + +- **Description:** HTML/MJML templates for newsletters. +- **Schema:** `id (string UUID)`, `templateName (string)`, `mjmlContent (string?)`, `htmlContent (string)`, `version (string)`, `isDefault (boolean)` + +### Database Schemas (Supabase PostgreSQL) + +#### 1\. `workflow_runs` + +```sql +CREATE TABLE public.workflow_runs ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + last_updated_at TIMESTAMPTZ NOT NULL DEFAULT now(), + status TEXT NOT NULL DEFAULT 'pending', -- pending, fetching_hn, scraping_articles, summarizing_content, generating_podcast, generating_newsletter, delivering_newsletter, completed, failed + current_step_details TEXT NULL, + error_message TEXT NULL, + details JSONB NULL -- {postsFetched, articlesAttempted, articlesScrapedSuccessfully, summariesGenerated, podcastJobId, podcastStatus, newsletterSentAt, subscribersNotified} +); +``` + +#### 2\. `hn_posts` + +```sql +CREATE TABLE public.hn_posts ( + id TEXT PRIMARY KEY, -- HN's objectID + hn_numeric_id BIGINT NULL UNIQUE, + title TEXT NOT NULL, + url TEXT NULL, + author TEXT NULL, + points INTEGER NOT NULL DEFAULT 0, + created_at TIMESTAMPTZ NOT NULL, + retrieved_at TIMESTAMPTZ NOT NULL DEFAULT now(), + hn_story_text TEXT NULL, + num_comments INTEGER NULL DEFAULT 0, + tags TEXT[] NULL, + workflow_run_id UUID NULL REFERENCES public.workflow_runs(id) ON DELETE SET NULL -- The run that fetched this instance of the post +); +``` + +#### 3\. `hn_comments` + +```sql +CREATE TABLE public.hn_comments ( + id TEXT PRIMARY KEY, -- HN's comment ID + hn_post_id TEXT NOT NULL REFERENCES public.hn_posts(id) ON DELETE CASCADE, + parent_comment_id TEXT NULL REFERENCES public.hn_comments(id) ON DELETE CASCADE, + author TEXT NULL, + comment_text TEXT NOT NULL, + created_at TIMESTAMPTZ NOT NULL, + retrieved_at TIMESTAMPTZ NOT NULL DEFAULT now() +); +CREATE INDEX idx_hn_comments_post_id ON public.hn_comments(hn_post_id); +``` + +#### 4\. `scraped_articles` + +```sql +CREATE TABLE public.scraped_articles ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + hn_post_id TEXT NOT NULL REFERENCES public.hn_posts(id) ON DELETE CASCADE, -- Should be unique if one article per post processing for a workflow run + original_url TEXT NOT NULL, + resolved_url TEXT NULL, + title TEXT NULL, + author TEXT NULL, + publication_date TIMESTAMPTZ NULL, + main_text_content TEXT NULL, + scraped_at TIMESTAMPTZ NOT NULL DEFAULT now(), + scraping_status TEXT NOT NULL DEFAULT 'pending', -- pending, success, failed_unreachable, failed_paywall, failed_parsing + error_message TEXT NULL, + workflow_run_id UUID NULL REFERENCES public.workflow_runs(id) ON DELETE SET NULL +); +CREATE UNIQUE INDEX idx_scraped_articles_hn_post_id_workflow_run_id ON public.scraped_articles(hn_post_id, workflow_run_id); +``` + +#### 5\. `article_summaries` + +```sql +CREATE TABLE public.article_summaries ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + scraped_article_id UUID NOT NULL REFERENCES public.scraped_articles(id) ON DELETE CASCADE, + summary_text TEXT NOT NULL, + generated_at TIMESTAMPTZ NOT NULL DEFAULT now(), + llm_prompt_version TEXT NULL, + llm_model_used TEXT NULL, + workflow_run_id UUID NOT NULL REFERENCES public.workflow_runs(id) ON DELETE CASCADE -- Summary is specific to a workflow run +); +CREATE UNIQUE INDEX idx_article_summaries_scraped_article_id_workflow_run_id ON public.article_summaries(scraped_article_id, workflow_run_id); +``` + +#### 6\. `comment_summaries` + +```sql +CREATE TABLE public.comment_summaries ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + hn_post_id TEXT NOT NULL REFERENCES public.hn_posts(id) ON DELETE CASCADE, + summary_text TEXT NOT NULL, + generated_at TIMESTAMPTZ NOT NULL DEFAULT now(), + llm_prompt_version TEXT NULL, + llm_model_used TEXT NULL, + workflow_run_id UUID NOT NULL REFERENCES public.workflow_runs(id) ON DELETE CASCADE -- Summary is specific to a workflow run +); +CREATE UNIQUE INDEX idx_comment_summaries_hn_post_id_workflow_run_id ON public.comment_summaries(hn_post_id, workflow_run_id); +``` + +#### 7\. `newsletters` + +```sql +CREATE TABLE public.newsletters ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + workflow_run_id UUID NOT NULL UNIQUE REFERENCES public.workflow_runs(id) ON DELETE CASCADE, + target_date DATE NOT NULL UNIQUE, + title TEXT NOT NULL, + generated_at TIMESTAMPTZ NOT NULL DEFAULT now(), + html_content TEXT NOT NULL, + mjml_template_version TEXT NULL, + podcast_playht_job_id TEXT NULL, + podcast_url TEXT NULL, + podcast_status TEXT NULL DEFAULT 'pending', -- pending, generating, completed, failed + delivery_status TEXT NOT NULL DEFAULT 'pending', -- pending, sending, sent, failed, partially_failed + scheduled_send_at TIMESTAMPTZ NULL, + sent_at TIMESTAMPTZ NULL +); +``` + +#### 8\. `subscribers` + +```sql +CREATE TABLE public.subscribers ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + email TEXT NOT NULL UNIQUE, + subscribed_at TIMESTAMPTZ NOT NULL DEFAULT now(), + is_active BOOLEAN NOT NULL DEFAULT TRUE, + unsubscribed_at TIMESTAMPTZ NULL +); +``` + +#### 9\. `summarization_prompts` + +```sql +CREATE TABLE public.summarization_prompts ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + prompt_name TEXT NOT NULL UNIQUE, + prompt_text TEXT NOT NULL, + version TEXT NOT NULL DEFAULT '1.0', + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + updated_at TIMESTAMPTZ NOT NULL DEFAULT now(), + is_default_article_prompt BOOLEAN NOT NULL DEFAULT FALSE, + is_default_comment_prompt BOOLEAN NOT NULL DEFAULT FALSE + -- Note: Logic to enforce single default will be in application layer or via more complex DB constraints/triggers. +); +``` + +#### 10\. `newsletter_templates` + +```sql +CREATE TABLE public.newsletter_templates ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + template_name TEXT NOT NULL UNIQUE, + mjml_content TEXT NULL, + html_content TEXT NOT NULL, + version TEXT NOT NULL DEFAULT '1.0', + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + updated_at TIMESTAMPTZ NOT NULL DEFAULT now(), + is_default BOOLEAN NOT NULL DEFAULT FALSE + -- Note: Logic to enforce single default will be in application layer. +); +``` + +## Core Workflow / Sequence Diagrams + +### 1\. Daily Workflow Initiation & HN Content Acquisition + +```mermaid +sequenceDiagram + actor Caller as Manual/API/CLI/Cron + participant TriggerAPI as POST /api/system/trigger-workflow + participant WorkflowRunsDB as workflow_runs (DB Table) + participant WorkflowTracker as WorkflowTrackerService + participant HNContentService as HNContentService (Supabase Fn) + participant HNAlgoliaAPI as HN Algolia API + participant HNPostsDB as hn_posts (DB Table) + participant HNCommentsDB as hn_comments (DB Table) + participant EventTrigger1 as DB Event/Webhook (on hn_posts insert) + + Caller->>+TriggerAPI: Request to start daily workflow + TriggerAPI->>+WorkflowTracker: initiateNewWorkflow() + WorkflowTracker->>+WorkflowRunsDB: INSERT new run (status='pending', details={}) + WorkflowRunsDB-->>-WorkflowTracker: new_workflow_run_id + WorkflowTracker-->>TriggerAPI: { jobId: new_workflow_run_id } + TriggerAPI-->>-Caller: HTTP 202 Accepted { jobId } + + alt Initial Trigger for HN Content Fetch + WorkflowTracker->>+HNContentService: triggerFetch(workflow_run_id) + else Alternative: Event from WorkflowRunsDB insert + WorkflowRunsDB-->>EventTrigger1: New workflow_run record + EventTrigger1->>+HNContentService: Invoke(workflow_run_id, event_payload) + end + + HNContentService->>+WorkflowTracker: updateWorkflowStep(workflow_run_id, 'fetching_hn_posts', 'fetching_hn') + WorkflowTracker->>+WorkflowRunsDB: UPDATE workflow_runs (status, current_step_details) + + HNContentService->>+HNAlgoliaAPI: GET /search?tags=front_page + HNAlgoliaAPI-->>-HNContentService: Front page story items + + loop For each story item (up to 30 after sorting by points) + HNContentService->>+HNPostsDB: INSERT story (hn_post_id, ..., workflow_run_id) + HNPostsDB-->>EventTrigger1: Notifies: New hn_post inserted + EventTrigger1-->>ArticleScrapingService: (Async) Trigger ArticleScrapingService(hn_post_id, workflow_run_id) + + HNContentService->>+HNAlgoliaAPI: GET /items/{story_objectID} (to fetch comments) + HNAlgoliaAPI-->>-HNContentService: Story details with comments + loop For each comment + HNContentService->>+HNCommentsDB: INSERT comment + end + end + HNContentService->>+WorkflowTracker: updateWorkflowDetails(workflow_run_id, {posts_fetched: X, comments_fetched: Y}) + WorkflowTracker->>WorkflowRunsDB: UPDATE workflow_runs (details) +``` + +### 2\. Article Scraping & Summarization Flow + +```mermaid +sequenceDiagram + participant EventTrigger1 as DB Event/Webhook (on hn_posts insert) + participant ArticleScrapingService as ArticleScrapingService (Supabase Fn) + participant ScrapedArticlesDB as scraped_articles (DB Table) + participant WorkflowTracker as WorkflowTrackerService + participant WorkflowRunsDB as workflow_runs (DB Table) + participant EventTrigger2 as DB Event/Webhook (on scraped_articles insert/update) + participant SummarizationService as SummarizationService (Supabase Fn) + participant LLMFacade as LLMFacade (shared function) + participant LLMProvider as LLM Provider (Ollama/Remote) + participant SummariesDB as article_summaries / comment_summaries (DB Tables) + participant PromptsDB as summarization_prompts (DB Table) + + EventTrigger1->>+ArticleScrapingService: Invoke(hn_post_id, workflow_run_id, article_url) + ArticleScrapingService->>+WorkflowTracker: updateWorkflowStep(workflow_run_id, 'scraping_article_for_post_' + hn_post_id, 'scraping_articles') + + ArticleScrapingService->>+ScrapedArticlesDB: INSERT new article (status='pending', workflow_run_id) + + opt Article URL is valid + ArticleScrapingService->>ArticleScrapingService: Fetch & Parse HTML with Cheerio + ArticleScrapingService->>+ScrapedArticlesDB: UPDATE scraped_articles SET content, status='success' + else Scraping fails + ArticleScrapingService->>+ScrapedArticlesDB: UPDATE scraped_articles SET status='failed_...' + end + ScrapedArticlesDB-->>EventTrigger2: Notifies: New/Updated scraped_article (status='success') + EventTrigger2-->>SummarizationService: (Async) Trigger SummarizationService(scraped_article_id, workflow_run_id, 'article') + + ArticleScrapingService->>+WorkflowTracker: updateWorkflowDetails(workflow_run_id, {articles_attempted_increment: 1, ...}) + + SummarizationService->>+WorkflowTracker: updateWorkflowStep(workflow_run_id, 'summarizing_content_for_post_' + hn_post_id, 'summarizing_content') + + alt Summarize Article + SummarizationService->>SummarizationService: Get article_text + SummarizationService->>+PromptsDB: Get article_prompt + SummarizationService->>+LLMFacade: generateSummary(article_text, article_prompt) + LLMFacade->>+LLMProvider: Request summary + LLMProvider-->>-LLMFacade: article_summary + SummarizationService->>+SummariesDB: INSERT into article_summaries + end + + alt Summarize Comments (for hn_post_id) + SummarizationService->>SummarizationService: Get comment_texts + SummarizationService->>+PromptsDB: Get comment_prompt + SummarizationService->>+LLMFacade: generateSummary(comment_texts, comment_prompt) + LLMFacade->>+LLMProvider: Request summary + LLMProvider-->>-LLMFacade: comment_summary + SummarizationService->>+SummariesDB: INSERT into comment_summaries + end + SummarizationService->>+WorkflowTracker: updateWorkflowDetails(workflow_run_id, {summaries_generated_increment: N}) +``` + +### 3\. Newsletter, Podcast, and Delivery Flow + +```mermaid +sequenceDiagram + participant CheckWorkflowService as CheckWorkflowCompletionService (Supabase Cron Fn) + participant WorkflowRunsDB as workflow_runs (DB Table) + participant WorkflowTracker as WorkflowTrackerService + participant NewsletterGenService as NewsletterGenerationService (Supabase Fn) + participant PodcastGenService as PodcastGenerationService (Supabase Fn) + participant PlayHTAPI as Play.ht API + participant NewsletterTemplatesDB as newsletter_templates (DB Table) + participant SummariesDB as article_summaries / comment_summaries (DB Tables) + participant NewslettersDB as newsletters (DB Table) + participant PlayHTWebhook as POST /api/webhooks/playht (Next.js API Route) + participant NodemailerService as NodemailerFacade (shared function) + participant SubscribersDB as subscribers (DB Table) + participant ExternalEmailService as Email Service (e.g., Gmail SMTP) + + CheckWorkflowService->>+WorkflowRunsDB: Query runs (status='summarizing_content', all summaries done?) + WorkflowRunsDB-->>-CheckWorkflowService: workflow_run_id (ready for newsletter) + + CheckWorkflowService->>+WorkflowTracker: updateWorkflowStep(workflow_run_id, 'starting_newsletter_generation', 'generating_newsletter') + CheckWorkflowService->>+NewsletterGenService: Invoke(workflow_run_id) + + NewsletterGenService->>+NewsletterTemplatesDB: Get default_template + NewsletterGenService->>+SummariesDB: Get summaries for workflow_run_id + NewsletterGenService->>NewsletterGenService: Compile HTML newsletter + NewsletterGenService->>+NewslettersDB: INSERT newsletter (html_content, podcast_status='pending') + + NewsletterGenService->>+PodcastGenService: initiatePodcast(newsletter_id, html_content) + PodcastGenService->>+PlayHTAPI: POST /playnotes (webHookUrl=...) + PlayHTAPI-->>-PodcastGenService: { playht_job_id, status: 'generating' } + PodcastGenService->>+NewslettersDB: UPDATE newsletters SET podcast_playht_job_id, podcast_status='generating' + PodcastGenService->>+WorkflowTracker: updateWorkflowStep(workflow_run_id, 'podcast_generation_initiated', 'generating_podcast') + + PlayHTAPI-->>+PlayHTWebhook: POST (status='completed', audioUrl='...') + PlayHTWebhook->>+NewslettersDB: UPDATE newsletters SET podcast_url, podcast_status='completed' + PlayHTWebhook->>+WorkflowTracker: updateWorkflowDetails(workflow_run_id, {podcast_status: 'completed'}) + PlayHTWebhook-->>-PlayHTAPI: HTTP 200 OK + + CheckWorkflowService->>+WorkflowRunsDB: Query runs (status='generating_podcast', podcast_status IN ('completed', 'failed') OR timeout) + WorkflowRunsDB-->>-CheckWorkflowService: workflow_run_id (ready for delivery) + + CheckWorkflowService->>+WorkflowTracker: updateWorkflowStep(workflow_run_id, 'starting_newsletter_delivery', 'delivering_newsletter') + CheckWorkflowService->>+NewsletterGenService: triggerDelivery(newsletter_id) + + NewsletterGenService->>+NewslettersDB: Get newsletter_data (html, podcast_url) + NewsletterGenService->>NewsletterGenService: (Embed podcast_url in HTML if available) + NewsletterGenService->>+SubscribersDB: Get active_subscribers + loop For each subscriber + NewsletterGenService->>+NodemailerService: sendEmail(to, subject, html) + NodemailerService->>+ExternalEmailService: SMTP send + end + NewsletterGenService->>+NewslettersDB: UPDATE newsletters SET delivery_status='sent', sent_at=now() + NewsletterGenService->>+WorkflowTracker: completeWorkflow(workflow_run_id, {delivery_status: 'sent'}) +``` + +## Definitive Tech Stack Selections + +This section outlines the definitive technology choices for the BMad DiCaster project. + +- **Preferred Starter Template Frontend & Backend:** Vercel/Supabase Next.js App Router Template ([https://vercel.com/templates/next.js/supabase](https://vercel.com/templates/next.js/supabase)) + +| Category | Technology | Version / Details | Description / Purpose | Justification (Optional, from PRD/User) | +| :------------------- | :-------------------------- | :-------------------------- | :------------------------------------------------------------------------ | :----------------------------------------------------------- | +| **Languages** | TypeScript | `5.7.2` | Primary language for backend/frontend | Strong typing, community support, aligns with Next.js/React | +| **Runtime** | Node.js | `22.10.2` | Server-side execution environment for Next.js & Supabase Functions | Compatible with Next.js, Vercel environment | +| **Frameworks** | Next.js | `latest` (e.g., 14.x.x) | Full-stack React framework | App Router, SSR, API routes, Vercel synergy | +| | React | `19.0.0` | Frontend UI library | Component-based, declarative | +| **UI Libraries** | Tailwind CSS | `3.4.17` | Utility-first CSS framework | Rapid UI development, consistent styling | +| | Shadcn UI | `latest` | React component library (via CLI) | Pre-styled, accessible components, built on Radix & Tailwind | +| **Databases** | PostgreSQL | (via Supabase) | Primary relational data store | Provided by Supabase, robust, scalable | +| **Cloud Platform** | Vercel | N/A | Hosting platform for Next.js app & Supabase Functions | Seamless Next.js/Supabase deployment, Edge Network | +| **Cloud Services** | Supabase Functions | N/A (via Vercel deploy) | Serverless compute for backend pipeline & APIs | Integrated with Supabase DB, event-driven capabilities | +| | Supabase Auth | N/A | User authentication and management | Integrated with Supabase, RLS | +| | Supabase Storage | N/A | File storage (e.g., for temporary newsletter files if needed for Play.ht) | Integrated with Supabase | +| **Infrastructure** | Supabase CLI | `latest` | Local development, migrations, function deployment | Official tool for Supabase development | +| | Docker | `latest` (via Supabase CLI) | Containerization for local Supabase services | Local development consistency | +| **State Management** | Zustand | `latest` | Frontend state management | Simple, unopinionated, performant for React | +| **Testing** | React Testing Library (RTL) | `latest` | Testing React components | User-centric testing, works well with Jest | +| | Jest | `latest` | Unit/Integration testing framework for JS/TS | Widely used, good support for Next.js/React | +| | Playwright | `latest` | End-to-end testing framework | Modern, reliable, cross-browser | +| **CI/CD** | GitHub Actions | N/A | Continuous Integration/Deployment | Integration with GitHub, automation of build/deploy/test | +| **Other Tools** | Cheerio | `latest` | HTML parsing/scraping for articles | Server-side HTML manipulation | +| | Nodemailer | `latest` | Email sending library for newsletters | Robust email sending from Node.js | +| | Zod | `latest` | TypeScript-first schema declaration and validation | Data validation for API inputs, environment variables etc. | +| | `tsx` / `ts-node` | `latest` (for scripts) | TypeScript execution for Node.js scripts (e.g. `scripts/`) | Running TS scripts directly | +| | Prettier | `3.3.3` | Code formatter | Consistent code style | +| | ESLint | `latest` | Linter for TypeScript/JavaScript | Code quality and error prevention | +| | Pino | `latest` | High-performance JSON logger for Node.js | Structured and efficient logging | + +## Infrastructure and Deployment Overview + +- **Cloud Provider(s):** Vercel (for hosting Next.js app and Supabase Functions) and Supabase (managed PostgreSQL, Auth, Storage; runs on underlying cloud like AWS). +- **Core Services Used:** Vercel (Next.js Hosting, Serverless/Edge Functions, CDN, CI/CD, Cron Jobs), Supabase (PostgreSQL, Auth, Storage, Functions, Database Webhooks). +- **Infrastructure as Code (IaC):** Supabase Migrations (`supabase/migrations/`) for database schema; Vercel project settings (`vercel.json` if needed). +- **Deployment Strategy:** GitHub Actions for CI/CD. Frontend (Next.js) via Vercel Git integration. Backend (Supabase Functions) via Supabase CLI within GitHub Actions. Database migrations via Supabase CLI. +- **Environments:** Local (Next.js dev server, Supabase CLI local stack), Development/Preview (Vercel preview deployments linked to dev Supabase instance), Production (Vercel production deployment linked to prod Supabase instance). +- **Environment Promotion:** Local -\> Dev/Preview (PR) -\> Production (merge to main). +- **Rollback Strategy:** Vercel dashboard/CLI for app/function rollbacks; Supabase migrations or Point-in-Time Recovery for database. + +## Error Handling Strategy + +- **General Approach:** Use standard `Error` objects or custom extensions. Supabase Functions catch errors, log via Pino, update `workflow_runs`, and avoid unhandled rejections. Next.js API routes return appropriate HTTP error responses with JSON payloads. +- **Logging (Pino):** + - Library: Pino (`pino`) for structured JSON logging in Supabase Functions and Next.js API routes. + - Configuration: Shared Pino logger instance (`supabase/functions/_shared/logger.ts`). + - Format: JSON. + - Levels: `trace`, `debug`, `info`, `warn`, `error`, `fatal`. + - Context: Logs include `timestamp`, `severity`, `workflowRunId`, `service`/`functionName`, `message`, and relevant `details`. **No sensitive data logged.** +- **Specific Handling Patterns:** + - **External API Calls:** Through facades with timeouts and limited retries (exponential backoff) for transient errors. Standardized custom errors thrown by facades. + - **Internal Errors/Business Logic:** Caught within functions; log details, update `workflow_runs` to 'failed'. API routes return generic errors to clients. + - **Database Operations:** Critical errors lead to 'failed' workflow status. + - **Scraping/Summarization Failures:** Individual item failures are logged and status updated (e.g., `scraped_articles.scraping_status`), but may not halt the entire workflow run if other items succeed. + - **Podcast/Delivery Failures:** Logged, status updated in `newsletters` and `workflow_runs`. Newsletter may be sent without podcast after timeout/failure. + - **`CheckWorkflowCompletionService`:** Designed for resilience; errors in processing one run should not prevent processing of others or future scheduled runs. + +## Coding Standards + +(As detailed previously, including TypeScript, Node.js, ESLint, Prettier, naming conventions, co-located unit tests `*.test.ts(x)`/`*.spec.ts(x)`, async/await, strict type safety, Pino logging, and specific framework/anti-pattern guidelines.) + +## Overall Testing Strategy + +(As detailed previously, covering Unit Tests with Jest/RTL, Integration Tests, E2E Tests with Playwright, 80% unit test coverage target, specific mocking strategies for facades and external dependencies, and test data management.) + +## Security Best Practices + +(As detailed previously, including Zod for input validation, output encoding, secrets management via environment variables, dependency security scanning, API key authentication for system APIs, Play.ht webhook verification, Supabase RLS, principle of least privilege, HTTPS, and secure error information disclosure.) + +## Key Reference Documents + +1. **Product Requirements Document (PRD):** `docs/prd-incremental-full-agile-mode.txt` +2. **UI/UX Specification:** `docs/ui-ux-spec.txt` +3. **Technical Preferences:** `docs/technical-preferences copy.txt` +4. **Environment Variables Documentation:** `docs/environment-vars.md` (To be created) +5. **(Optional) Frontend Architecture Document:** `docs/frontend-architecture.md` (To be created by Design Architect) +6. **Play.ht API Documentation:** [https://docs.play.ai/api-reference/playnote/post](https://docs.play.ai/api-reference/playnote/post) +7. **Hacker News Algolia API:** [https://hn.algolia.com/api](https://hn.algolia.com/api) +8. **Ollama API Documentation:** [https://github.com/ollama/ollama/blob/main/docs/api.md](https://www.google.com/search?q=https://github.com/ollama/ollama/blob/main/docs/api.md) +9. **Supabase Documentation:** [https://supabase.com/docs](https://supabase.com/docs) +10. **Next.js Documentation:** [https://nextjs.org/docs](https://nextjs.org/docs) +11. **Vercel Documentation:** [https://vercel.com/docs](https://vercel.com/docs) +12. **Pino Logging Documentation:** [https://getpino.io/](https://getpino.io/) + +## Change Log + +| Change | Date | Version | Description | Author | +| :----------------------------------------- | :--------- | :------ | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------- | +| Initial Draft based on PRD and discussions | 2025-05-13 | 0.1 | First complete draft covering project overview, components, data models, tech stack, deployment, error handling, coding standards, testing strategy, security, and workflow orchestration. | 3-arch (Agent) | + +--- + +## Prompt for Design Architect: Frontend Architecture Definition + +**To the Design Architect (Agent Specializing in Frontend Architecture):** + +You are now tasked with defining the detailed **Frontend Architecture** for the BMad DiCaster project. This main Architecture Document and the `docs/ui-ux-spec.txt` are your primary input artifacts. Your goal is to produce a dedicated `frontend-architecture.md` document. + +**Key Inputs & Constraints (from this Main Architecture Document & UI/UX Spec):** + +1. **Overall Project Architecture:** Familiarize yourself with the "High-Level Overview," "Component View," "Data Models" (especially any shared types in `shared/types/`), and "API Reference" (particularly internal APIs like `/api/system/trigger-workflow` and `/api/webhooks/playht` that the frontend might indirectly be aware of or need to interact with for admin purposes in the future, though MVP frontend primarily reads newsletter data). +2. **UI/UX Specification (`docs/ui-ux-spec.txt`):** This document contains user flows, wireframes, core screens (Newsletter List, Newsletter Detail), component inventory (NewsletterCard, PodcastPlayer, DownloadButton, BackButton), branding considerations (synthwave, minimalist), and accessibility aspirations. +3. **Definitive Technology Stack (Frontend Relevant):** + - Framework: Next.js (`latest`, App Router) + - Language: React (`19.0.0`) with TypeScript (`5.7.2`) + - UI Libraries: Tailwind CSS (`3.4.17`), Shadcn UI (`latest`) + - State Management: Zustand (`latest`) + - Testing: React Testing Library (RTL) (`latest`), Jest (`latest`) + - Starter Template: Vercel/Supabase Next.js App Router template ([https://vercel.com/templates/next.js/supabase](https://vercel.com/templates/next.js/supabase)). Leverage its existing structure for `app/`, `components/ui/` (from Shadcn), `lib/utils.ts`, and `utils/supabase/` (client, server, middleware helpers for Supabase). +4. **Project Structure (Frontend Relevant):** Refer to the "Project Structure" section in this document, particularly the `app/` directory, `components/` (for Shadcn `ui` and your `core` application components), `lib/`, and `utils/supabase/`. +5. **Existing Frontend Files (from template):** Be aware of `middleware.ts` (for Supabase auth) and any existing components or utility functions provided by the starter template. + +**Tasks for Frontend Architecture Document (`frontend-architecture.md`):** + +1. **Refine Frontend Project Structure:** + - Detail the specific folder structure within `app/`. Propose organization for pages (routes), layouts, application-specific components (`app/components/core/`), data fetching logic, context providers, and Zustand stores. + - How will Shadcn UI components (`components/ui/`) be used and potentially customized? +2. **Component Architecture:** + - For each core screen identified in the UI/UX spec (Newsletter List, Newsletter Detail), define the primary React component hierarchy. + - Specify responsibilities and key props for major reusable application components (e.g., `NewsletterCard`, `NewsletterDetailView`, `PodcastPlayerControls`). + - How will components fetch and display data from Supabase? (e.g., Server Components, Client Components using Supabase client from `utils/supabase/client.ts` or `utils/supabase/server.ts`). +3. **State Management (Zustand):** + - Identify global and local state needs. + - Define specific Zustand store(s): what data they will hold (e.g., current newsletter list, selected newsletter details, podcast player state), and what actions they will expose. + - How will components interact with these stores? +4. **Data Fetching & Caching (Frontend):** + - Specify patterns for fetching newsletter data (lists and individual items) and podcast information. + - How will Next.js data fetching capabilities (Server Components, Route Handlers, `Workspace` with caching options) be utilized with the Supabase client? + - Address loading and error states for data fetching in the UI. +5. **Routing:** + - Confirm Next.js App Router usage and define URL structure for the newsletter list and detail pages. +6. **Styling Approach:** + - Reiterate use of Tailwind CSS and Shadcn UI. + - Define any project-specific conventions for applying Tailwind classes or extending the theme (beyond what's in `tailwind.config.ts`). + - How will the "synthwave technical glowing purple vibes" be implemented using Tailwind? +7. **Error Handling (Frontend):** + - How will errors from API calls (to Supabase or internal Next.js API routes if any) be handled and displayed to the user? + - Strategy for UI error boundaries. +8. **Accessibility (AX):** + - Elaborate on how the WCAG 2.1 Level A requirements (keyboard navigation, semantic HTML, alt text, color contrast) will be met in component design and implementation, leveraging Next.js and Shadcn UI capabilities. +9. **Testing (Frontend):** + - Reiterate the use of Jest and RTL for unit/integration testing of React components. + - Provide examples or guidelines for writing effective frontend tests. +10. **Key Frontend Libraries & Versioning:** Confirm versions from the main tech stack and list any additional frontend-only libraries required. + +Your output should be a clean, well-formatted `frontend-architecture.md` document ready for AI developer agents to use for frontend implementation. Adhere to the output formatting guidelines. You are now operating in **Frontend Architecture Mode**. diff --git a/BETA-V3/v3-demos/project1/prd-incremental-full-agile-mode.txt b/BETA-V3/v3-demos/project1/prd.txt similarity index 99% rename from BETA-V3/v3-demos/project1/prd-incremental-full-agile-mode.txt rename to BETA-V3/v3-demos/project1/prd.txt index cba82b77..0e4d9eb9 100644 --- a/BETA-V3/v3-demos/project1/prd-incremental-full-agile-mode.txt +++ b/BETA-V3/v3-demos/project1/prd.txt @@ -341,7 +341,7 @@ _(This section will be created later, from the sections prior to this being carv | Change | Date | Version | Description | Author | | ------------- | ---------- | ------- | -------------------------------------------------- | ------ | -| Initial Draft | 2024-07-26 | 0.1 | Initial draft of the Product Requirements Document | 2-pm | +| Initial Draft | 2025-05-13 | 0.1 | Initial draft of the Product Requirements Document | 2-pm | ----- END PRD START CHECKLIST OUTPUT ------ diff --git a/BETA-V3/v3-demos/project1/simulated-prd-no-architect-later.txt b/BETA-V3/v3-demos/project1/simulated-prd-no-architect-later.txt deleted file mode 100644 index ff7076bb..00000000 --- a/BETA-V3/v3-demos/project1/simulated-prd-no-architect-later.txt +++ /dev/null @@ -1,508 +0,0 @@ -# BMad DiCaster Product Requirements Document (PRD) - -## Goal, Objective and Context - -**Goal:** To develop a web application that provides a daily, concise summary of top Hacker News (HN) posts, delivered as a newsletter and accessible via a web interface. - -**Objective:** To streamline the consumption of HN content by curating the top stories, providing AI-powered summaries, and offering an optional AI-generated podcast version. - -**Context:** Busy professionals and enthusiasts want to stay updated on HN but lack the time to sift through numerous posts and discussions. This application will address this problem by automating the delivery of summarized content. - -## Functional Requirements (MVP) - -- **HN Content Retrieval & Storage:** - - Daily retrieval of the top 30 Hacker News posts and associated comments using the HN Algolia API. - - Scraping and storage of up to 10 linked articles per day. - - Storage of all retrieved data (posts, comments, articles) with date association. -- **AI-Powered Summarization:** - - AI-powered summarization of the 10 selected articles (2-paragraph summaries). - - AI-powered summarization of comments for the 10 selected posts (2-paragraph summaries highlighting interesting interactions). - - Configuration for local or remote LLM usage via environment variables, including specifying the LLM API endpoint and any necessary authentication credentials. -- **Newsletter Generation & Delivery:** - - Generation of a daily newsletter in HTML format, including summaries, links to HN posts and articles, and original post dates/times. - - Automated delivery of the newsletter to a manually configured list of subscribers in Supabase. The list of emails will be manually populated in the "subscribers" table. Account information for the Nodemailer service (including SMTP server, port, username, and password) will be provided via environment variables. -- **Podcast Generation & Integration:** - - Integration with Play.ht's PlayNote API for AI-generated podcast creation from the newsletter content. - - Webhook handler to update the newsletter with the generated podcast link. -- **Web Interface (MVP):** - - Display of current and past newsletters. - - Functionality to read the newsletter content within the web page. - - Download option for newsletters (in HTML format). - - Web player for listening to generated podcasts. -- **API & Triggering:** - - Secure API endpoint (`/api/trigger-workflow`) to manually trigger the daily workflow, secured with API keys passed in the `X-API-Key` header. - - CLI command (`node trigger.js`) to manually trigger the daily workflow locally. - -## Non-Functional Requirements (MVP) - -- **Performance:** - - The system should retrieve HN posts and generate the newsletter within a reasonable timeframe (e.g., under 30 minutes) to ensure timely delivery. - - The web interface should load quickly (e.g., within 2 seconds) to provide a smooth user experience. -- **Scalability:** - - The system is designed for an initial MVP delivery to 3-5 email subscribers. Scalability beyond this will be considered post-MVP. -- **Security:** - - The API endpoint (`/api/trigger-workflow`) for triggering the daily workflow must be secure, using API keys passed in the `X-API-Key` header. - - User data (email addresses) should be stored securely in the Supabase database, with appropriate access controls. No other security measures are required for the MVP. -- **Reliability:** - - No specific uptime or availability requirements are defined for the MVP. - - The newsletter generation and delivery process should be robust and handle potential errors gracefully, with logging and retry mechanisms where appropriate. - - The system must be executable from a local development environment using Node.js and the Supabase CLI. -- **Maintainability:** - - The codebase should adhere to good quality coding standards, including separation of concerns and following SOLID principles. - - The system should employ design patterns like facades and factories to facilitate future expansion. - - The system should be built as an event-driven pipeline, leveraging Supabase to capture data at each stage and trigger subsequent functions asynchronously. This approach aims to mitigate potential timeout issues with Vercel hosting. - - Key parameters (e.g., API keys, email settings, LLM configuration) should be configurable via environment variables. - -## User Interaction and Design Goals - -This section captures the high-level vision and goals for the User Experience (UX) to guide the Design Architect. - -- **Overall Vision & Experience:** - - The desired look and feel is modern and minimalist, with synthwave technical glowing purple vibes. - - Users should have a clean and efficient experience when accessing and consuming newsletter content and podcasts. -- **Key Interaction Paradigms:** - - The web interface will primarily involve browsing lists and reading detailed content. - - Users will interact with the podcast player using standard playback controls. -- **Core Screens/Views (Conceptual):** - - The MVP will consist of two pages: - - A list page (`/newsletters`) to display current and past newsletters, with links to the detail page. - - A detail page (`/newsletters/{date}`) to display the selected newsletter content, including: - - Download option for the newsletter (in HTML format). - - Web player for listening to the generated podcast. - - The article laid out for viewing. -- **Accessibility Aspirations:** - - Basic semantic HTML should be used to ensure a degree of screen reader compatibility. -- **Branding Considerations (High-Level):** - - A logo for the application will be provided. - - The application will use the name "BMad DiCaster". -- **Target Devices/Platforms:** - - The application will be designed as a mobile-first responsive web app, ensuring it looks good on both mobile and desktop devices. The primary target is modern web browsers (Chrome, Firefox, Safari, Edge). - -## Technical Assumptions - -This section captures any existing technical information that will guide the development. - -- The application will be developed using the Next.js framework for the frontend and Vercel serverless functions for the backend, leveraging the Vercel/Supabase template as a starting point. -- This implies a monorepo structure, as the frontend (Next.js) and backend (Vercel functions) will reside within the same repository. -- Frontend development will be in Next.js with React, utilizing Tailwind CSS for styling and theming. -- Data storage will be handled by Supabase's PostgreSQL database. -- Separate Supabase instances will be used for development and production environments to ensure data isolation and stability. Supabase CLI will be used for local development. -- Testing will include unit tests (Jest), integration tests (testing API calls and Supabase interactions), and end-to-end tests (Cypress or Playwright). - -## Technology Choices - -| Technology | Version | Purpose | -| :----------------- | :------ | :------------------- | -| Next.js | Latest | Frontend Framework | -| React | Latest | UI Library | -| Tailwind CSS | 3.x | CSS Framework | -| Vercel | N/A | Hosting Platform | -| Supabase | N/A | Backend as a Service | -| Node.js | 18.x | Backend Runtime | -| HN Algolia API | N/A | HN Data Source | -| Play.ht API | N/A | Podcast Generation | -| Nodemailer | 6.x | Email Sending | -| Cheerio | 1.x | HTML Parsing | -| Jest | 29.x | Unit Testing | -| Cypress/Playwright | Latest | E2E Testing | - -## Proposed Directory Structure - -``` -├── app/ # Next.js app directory -│ ├── api/ # API routes -│ │ └── trigger-workflow.js -│ ├── newsletters/ # Newsletter list and detail pages -│ │ ├── [date]/ -│ │ │ └── page.jsx -│ │ └── page.jsx -│ ├── layout.jsx # Root layout -│ └── page.jsx # Home page (redirect to /newsletters) -├── components/ # React components -│ ├── NewsletterList.jsx -│ ├── NewsletterDetail.jsx -│ └── PodcastPlayer.jsx -├── lib/ # Utility functions -│ ├── hn.js # HN API interactions -│ ├── scraping.js # Article scraping -│ ├── summarization.js # LLM integration -│ ├── email.js # Nodemailer -│ └── db.js # Supabase client -├── public/ # Static assets (logo, etc.) -├── styles/ # Global CSS (if needed) -├── .env.local # Local environment variables -├── next.config.js # Next.js config -├── package.json -├── tailwind.config.js # Tailwind CSS config -└── trigger.js # CLI trigger script -``` - -## Data Models - -### Hacker News Post - -| Field | Type | Description | -| :--------- | :------- | :--------------------------------- | -| id | String | HN Post ID (objectID from Algolia) | -| title | String | Post Title | -| url | String | Post URL | -| created_at | DateTime | Post Creation Date/Time | -| author | String | Post Author | -| date | Date | Retrieval Date | - -### Hacker News Comment - -| Field | Type | Description | -| :--------- | :------- | :------------------------------------ | -| id | String | HN Comment ID (objectID from Algolia) | -| text | String | Comment Text | -| author | String | Comment Author | -| created_at | DateTime | Comment Creation Date/Time | -| parent_id | String | ID of the parent post/comment | -| date | Date | Retrieval Date | -| summary | String | AI-generated summary | - -### Article - -| Field | Type | Description | -| :--------------- | :----- | :----------------------------------- | -| id | UUID | Primary Key | -| post_id | String | HN Post ID (Foreign Key to hn_posts) | -| title | String | Article Title | -| author | String | Article Author | -| publication_date | Date | Article Publication Date | -| content | String | Article Content | -| summary | String | AI-generated summary | - -### Newsletter - -| Field | Type | Description | -| :---------- | :----- | :---------------------------- | -| date | Date | Newsletter Date (Primary Key) | -| content | String | HTML Newsletter Content | -| podcast_url | String | URL of the generated podcast | - -### Subscriber - -| Field | Type | String | -| :---- | :----- | :----------------------------- | -| email | String | Subscriber Email (Primary Key) | - -### Template - -| Field | Type | Description | -| :------ | :----- | :----------------------------------------------------------------------- | -| type | String | Template Type (e.g., 'newsletter', 'article_summary', 'comment_summary') | -| content | String | Template Content (HTML or Prompt) | - -## Database Schema - -```sql -CREATE TABLE hn_posts ( - id VARCHAR(255) PRIMARY KEY, - title TEXT, - url TEXT, - created_at TIMESTAMPTZ, - author VARCHAR(255), - date DATE NOT NULL -); - -CREATE TABLE hn_comments ( - id VARCHAR(255) PRIMARY KEY, - text TEXT, - author VARCHAR(255), - created_at TIMESTAMPTZ, - parent_id VARCHAR(255), - date DATE NOT NULL, - summary TEXT -); - -CREATE TABLE articles ( - id UUID PRIMARY KEY DEFAULT gen_random_uuid(), - post_id VARCHAR(255) NOT NULL REFERENCES hn_posts(id), - title TEXT, - author VARCHAR(255), - publication_date DATE, - content TEXT, - summary TEXT -); - -CREATE TABLE newsletters ( - date DATE PRIMARY KEY, - content TEXT NOT NULL, - podcast_url TEXT -); - -CREATE TABLE subscribers ( - email VARCHAR(255) PRIMARY KEY -); - -CREATE TABLE templates ( - type VARCHAR(255) PRIMARY KEY, - content TEXT NOT NULL -); -``` - -## Epic Overview - -- **Epic 1: Project Initialization, Setup, and HN Content Acquisition** - - Goal: Establish the foundational project structure, including the Next.js application, Supabase integration, and deployment pipeline, implement the initial API and CLI trigger mechanisms, and implement the functionality to retrieve, process, and store Hacker News posts and comments, providing the necessary data for newsletter generation. -- **Epic 2: Article Scraping** - - Goal: Implement the functionality to scrape and store linked articles from HN posts, enriching the data available for summarization and the newsletter, and ensure this functionality can be triggered via the API and CLI. -- **Epic 3: AI-Powered Content Summarization** - - Goal: Integrate AI summarization capabilities to generate concise summaries of articles and comments, retrieving prompts from the database for flexibility, and ensure this functionality can be triggered via the API and CLI. -- **Epic 4: Automated Newsletter Creation and Distribution** - - Goal: Automate the generation and delivery of the daily newsletter, including a placeholder for the podcast URL and conditional logic with configurable delay/retry parameters (configurable via environment variables) to handle podcast link availability. Ensure this functionality can be triggered via the API and CLI. -- **Epic 5: Podcast Generation Integration** - - Goal: Integrate with the Play.ht's PlayNote API (using webhooks) to generate and incorporate podcast versions of the newsletter, including a webhook to update the newsletter data with the podcast URL in Supabase. Ensure this functionality can be triggered via the API and CLI. -- **Epic 6: Web Interface Development** - - Goal: Develop a user-friendly and mobile-responsive web interface, starting with an initial structure generated by a tool like Vercel v0 and styled with Tailwind CSS using design tokens, to display newsletters and provide access to podcast content. - -## Epic and User Stories - -**Epic 1: Project Initialization, Setup, and HN Content Acquisition** - -- Goal: Establish the foundational project structure, including the Next.js application, Supabase integration, and deployment pipeline, implement the initial API and CLI trigger mechanisms, and implement the functionality to retrieve, process, and store Hacker News posts and comments, providing the necessary data for newsletter generation. -- Story 1.1: Project Setup - - As a developer, I want to set up the Next.js project with Supabase integration and initialize a Git repository, so that I have a functional foundation for building the application. - - Acceptance Criteria: - 1. The Next.js project is initialized using the Vercel/Supabase template (`npx create-next-app@latest --example with-supabase`). - 2. Supabase client libraries are installed (`npm install @supabase/supabase-js`). - 3. The Supabase client is initialized with the Supabase URL and Anon Key from environment variables (`NEXT_PUBLIC_SUPABASE_URL`, `NEXT_PUBLIC_SUPABASE_ANON_KEY`). - 4. The project codebase is initialized in a Git repository with a `.gitignore` file. -- Story 1.2: Deployment Pipeline - - As a developer, I want to configure the deployment pipeline to Vercel with separate development and production environments, so that I can easily deploy and update the application. - - Acceptance Criteria: - 1. The project is successfully linked to a Vercel project using the Vercel CLI (`vercel link`). - 2. Separate Vercel projects are set up for development and production environments. - 3. Automated deployments are configured for the `main` branch to the production environment. - 4. Environment variables are set up for local development (`.env.local`) and Vercel deployments (via the Vercel dashboard) for Supabase URL/Key, API key, and other sensitive configuration. -- Story 1.3: API and CLI Trigger Mechanisms - - As a developer, I want to implement a secure API endpoint and a CLI command to trigger the workflow, so that I can manually trigger the process during development and testing. - - Acceptance Criteria: - 1. A secure API endpoint (`/api/trigger-workflow`) is created as a Vercel API Route. - 2. The API endpoint requires authentication using an API key passed in the `X-API-Key` header. - 3. The API endpoint triggers the execution of the main workflow (HN data retrieval). - 4. The API endpoint returns a JSON response with a success/failure status code and message. - 5. A CLI command (`node trigger.js`) is created using Node.js and a library like `commander` or `yargs` to trigger the workflow locally. - 6. The CLI command reads necessary configuration (e.g., Supabase credentials) from environment variables. - 7. Both the API endpoint and CLI command log all requests with timestamps, source (API/CLI), and status. -- Story 1.4: HN Data Retrieval - - As a system, I want to retrieve the top 30 Hacker News posts and associated comments daily using the HN Algolia API, and store them in the Supabase database, so that the data is available for summarization and newsletter generation. - - Acceptance Criteria: - 1. The system retrieves the top 30 Hacker News posts daily using the HN Algolia API (`https://hn.algolia.com/api/v1/search_by_date?tags=front_page`). - 2. The system retrieves associated comments for the top 30 posts using the HN Algolia API (using the post's `objectID`). - 3. Retrieved data (posts and comments) is stored in the Supabase database in tables named `hn_posts` and `hn_comments`, with appropriate fields (e.g., `title`, `url`, `created_at`, `author` for posts; `text`, `author`, `created_at`, `parent_id` for comments) and a `date` field to associate the data with the retrieval date. - 4. This functionality can be triggered via the API endpoint (`/api/trigger-workflow`) and the CLI command (`node trigger.js`). - 5. The system logs the start and completion of the retrieval process, including the number of posts/comments retrieved and any errors encountered (e.g., API request failures, database connection errors). - -**Epic 2: Article Scraping** - -- Goal: Implement the functionality to scrape and store linked articles from HN posts, enriching the data available for summarization and the newsletter, and ensure this functionality can be triggered via the API and CLI. -- Story 2.1: URL Identification - - As a system, I want to identify URLs within the retrieved Hacker News posts, so that I can extract the content of linked articles. - - Acceptance Criteria: - 1. The system parses the `url` field of each post in the `hn_posts` table. - 2. The system filters out any URLs that are not relevant to article scraping (e.g., links to images, videos, or known non-article sites) using a basic filter or a library like `url-parse`. - 3. The system limits the number of URLs to be scraped per post to 1, prioritizing the first valid URL. -- Story 2.2: Article Content Scraping - - As a system, I want to scrape the content of the identified article URLs using Cheerio, so that I can provide summaries in the newsletter. - - Acceptance Criteria: - 1. The system scrapes the content from the identified article URLs using the Cheerio library. - 2. The system extracts the following relevant content: - - Article title (using `` tag or schema.org metadata) - - Article author (if available, using metadata) - - Article publication date (if available, using metadata) - - Main article text (by identifying and extracting content from `<p>` tags within the main article body, excluding navigation and ads). - 3. The system handles potential issues during scraping: - - Website errors (e.g., 404 status codes) are logged. - - Changes in website structure are handled gracefully (e.g., by providing alternative CSS selectors or logging a warning). - - A timeout is implemented for scraping requests. -- Story 2.3: Scraped Data Storage - - As a system, I want to store the scraped article content in the Supabase database, associated with the corresponding Hacker News post, so that it can be used for summarization and newsletter generation. - - Acceptance Criteria: - 1. The system stores the scraped article content in a table named `articles` in the Supabase database. - 2. The `articles` table includes fields for `post_id` (foreign key referencing `hn_posts`), `title`, `author`, `publication_date`, and `content`. - 3. The system ensures that the stored data includes all extracted information and is correctly associated with the corresponding Hacker News post. -- Story 2.4: API/CLI Triggering - - As a developer, I want to trigger the article scraping process via the API and CLI, so that I can manually initiate it for testing and debugging. - - Acceptance Criteria: - 1. The existing API endpoint (`/api/trigger-workflow`) can trigger the article scraping process. - 2. The existing CLI command (`node trigger.js`) can trigger the article scraping process locally. - 3. The system logs the start and completion of the scraping process, including the number of articles scraped, any errors encountered, and the execution time. - -**Epic 3: AI-Powered Content Summarization** - -- Goal: Integrate AI summarization capabilities to generate concise summaries of articles and comments, retrieving prompts from the database for flexibility, and ensure this functionality can be triggered via the API and CLI. -- Story 3.1: LLM Integration - - As a system, I want to integrate an AI summarization library or API, so that I can generate concise summaries of articles and comments. - - Acceptance Criteria: - 1. The system integrates with a suitable AI summarization library (e.g., `langchain` with a local LLM like `Ollama`) or a cloud-based API (e.g., OpenAI API). - 2. The LLM or API endpoint and any necessary authentication credentials (API keys) are configurable via environment variables (`LLM_API_URL`, `LLM_API_KEY`). - 3. The system implements error handling for LLM/API requests, including retries and fallback mechanisms. -- Story 3.2: Article Summarization - - As a system, I want to retrieve summarization prompts from the database, and then use them to generate 2-paragraph summaries of the scraped articles, so that users can quickly grasp the main content and the prompts can be easily updated. - - Acceptance Criteria: - 1. The system retrieves the appropriate summarization prompt from a table named `prompts` in the Supabase database, using a query to select the prompt based on `type = 'article_summary'`. The prompt should include placeholders for article title and content. - 2. The system generates a 2-paragraph summary (approximately 100-150 words) for each scraped article using the retrieved prompt and the LLM/API. - 3. The summaries are accurate, concise, and capture the key information from the article. - 4. The generated summaries are stored in the `articles` table in a field named `summary`. -- Story 3.3: Comment Summarization - - As a system, I want to retrieve summarization prompts from the database, and then use them to generate 2-paragraph summaries of the comments for the selected HN posts, so that users can understand the main discussions and the prompts can be easily updated. - - Acceptance Criteria: - 1. The system retrieves the appropriate summarization prompt from the `prompts` table, using a query to select the prompt based on `type = 'comment_summary'`. The prompt should include placeholders for comment text. - 2. The system generates a 2-paragraph summary (approximately 100-150 words) of the comments for each selected HN post using the retrieved prompt and the LLM/API. - 3. The summaries highlight interesting interactions and key points from the discussion, focusing on distinct viewpoints and arguments. - 4. The generated comment summaries are stored in the `hn_comments` table in a field named `summary`. -- Story 3.4: API/CLI Triggering and Logging - - As a developer, I want to trigger the AI summarization process via the API and CLI, and log all summarization requests, prompts used, and results, so that I can manually initiate it for testing/debugging and monitor the process. - - Acceptance Criteria: - 1. The existing API endpoint (`/api/trigger-workflow`) can trigger the AI summarization process. - 2. The existing CLI command (`node trigger.js`) can trigger the AI summarization process locally. - 3. The system logs the following information for each summarization: - - Timestamp - - Source (API/CLI) - - Article title or HN post ID - - Summarization prompt used - - Generated summary - - LLM/API response time - - Any errors encountered - 4. The system handles partial execution gracefully (i.e., if triggered before Epic 2 is complete, it logs a message and exits). - -**Epic 4: Automated Newsletter Creation and Distribution** - -- Goal: Automate the generation and delivery of the daily newsletter, including a placeholder for the podcast URL and conditional logic with configurable delay/retry parameters (configurable via environment variables) to handle podcast link availability. Ensure this functionality can be triggered via the API and CLI. -- Story 4.1: Newsletter Template Retrieval - - As a system, I want to retrieve the newsletter template from the database, so that the newsletter's design and structure can be updated without code changes. - - Acceptance Criteria: - 1. The system retrieves the newsletter template from the `templates` table in the Supabase database, using a query to select the template based on `type = 'newsletter'`. The template is an HTML string with placeholders for article summaries, comment summaries, HN post links, and the podcast URL placeholder (`{{podcast_url}}`). -- Story 4.2: Newsletter Generation - - As a system, I want to generate a daily newsletter in HTML format using the retrieved template and the summarized data, so that users can receive a concise summary of Hacker News content. - - Acceptance Criteria: - 1. The system generates a newsletter in HTML format using the template retrieved from the `templates` table. - 2. The newsletter includes: - - Summaries of the top 10 articles from the `articles` table. - - Summaries of comments for the top 10 HN posts from the `hn_comments` table. - - Links to the original HN posts and articles. - - Original post dates/times. - - A placeholder for the podcast URL (`{{podcast_url}}`). - 3. The generated newsletter is stored in the `newsletters` table with a `date` field. -- Story 4.3: Newsletter Delivery - - As a system, I want to send the generated newsletter to a list of subscribers using Nodemailer, with credentials securely provided via environment variables, so that users receive the daily summary in their inbox. - - Acceptance Criteria: - 1. The system retrieves the list of subscriber email addresses from the `subscribers` table in the Supabase database. - 2. The system sends the HTML newsletter to all active subscribers using Nodemailer. - 3. Nodemailer is configured using the following environment variables: `SMTP_HOST`, `SMTP_PORT`, `SMTP_USER`, `SMTP_PASS`. - 4. The system logs the delivery status (success/failure, timestamp) for each subscriber in a table named `newsletter_logs`. - 5. The system implements conditional logic and a configurable delay/retry mechanism (using `EMAIL_DELAY` in milliseconds and `EMAIL_RETRIES` as the number of retries, both from environment variables) to handle podcast link availability before sending the email. If the podcast URL is not available after the delay and retries, the email is sent without the podcast link. -- Story 4.4: API/CLI Triggering and Logging - - As a developer, I want to trigger the newsletter generation and distribution process via the API and CLI, and log all relevant actions, so that I can manually initiate it for testing and debugging, and monitor the process. - - Acceptance Criteria: - 1. The existing API endpoint (`/api/trigger-workflow`) can trigger the newsletter generation and distribution process. - 2. The existing CLI command (`node trigger.js`) can trigger the newsletter generation and distribution process locally. - 3. The system logs the following information: - - Timestamp - - Source (API/CLI) - - Newsletter generation start/end times - - Number of emails sent successfully/failed - - Podcast URL availability status - - Email delay/retry attempts - - Any errors encountered - 4. The system handles partial execution gracefully (i.e., if triggered before Epic 3 is complete, it logs a message and exits). - -**Epic 5: Podcast Generation Integration** - -- Goal: Integrate with the Play.ht's PlayNote API (using webhooks) to generate and incorporate podcast versions of the newsletter, including a webhook to update the newsletter data with the podcast URL in Supabase. Ensure this functionality can be triggered via the API and CLI. -- _(Note for Architect: The Play.ht API - [https://docs.play.ai/api-reference/playnote/post](https://docs.play.ai/api-reference/playnote/post) - supports both webhook and polling mechanisms for receiving the podcast URL. We will use the webhook approach for real-time updates and efficiency.)_ -- Story 5.1: Play.ht API Integration - - As a system, I want to integrate with the Play.ht's PlayNote API, so that I can generate AI-powered podcast versions of the newsletter content. - - Acceptance Criteria: - 1. The system integrates with the Play.ht's PlayNote API using webhooks for asynchronous updates. - 2. The Play.ht API key and user ID are configured via environment variables (`PLAYHT_API_KEY`, `PLAYHT_USER_ID`). - 3. The Play.ht webhook URL is configured via the `PLAYHT_WEBHOOK_URL` environment variable. -- Story 5.2: Podcast Generation Initiation - - As a system, I want to send the newsletter content to the Play.ht API to initiate podcast generation, and receive a job ID or initial response, so that I can track the podcast creation process. - - Acceptance Criteria: - 1. The system sends the HTML newsletter content to the Play.ht API's `POST /v1/convert` endpoint to generate a podcast. - 2. The system receives a JSON response from the Play.ht API containing a `jobId`. - 3. The system stores the `jobId` in the `newsletters` table along with the newsletter data. -- Story 5.3: Webhook Handling - - As a system, I want to implement a webhook handler to receive the podcast URL from Play.ht, and update the newsletter data, so that the podcast link can be included in the newsletter and web interface. - - Acceptance Criteria: - 1. The system implements a webhook handler at the URL specified in the `PLAYHT_WEBHOOK_URL` environment variable. - 2. The webhook handler receives a `POST` request from Play.ht with a JSON payload containing the `jobId` and the `audioUrl`. - 3. The system extracts the `audioUrl` from the webhook payload. - 4. The system updates the `podcast_url` field in the `newsletters` table for the corresponding `jobId`. -- Story 5.4: API/CLI Triggering and Logging - - As a developer, I want to trigger the podcast generation process via the API and CLI, and log all interactions with the Play.ht API, so that I can manually initiate it for testing and debugging, and monitor the process. - - Acceptance Criteria: - 1. The existing API endpoint (`/api/trigger-workflow`) can trigger the podcast generation process. - 2. The existing CLI command (`node trigger.js`) can trigger the podcast generation process locally. - 3. The system logs the following information: - - Timestamp - - Source (API/CLI) - - Play.ht API requests (endpoint, payload) and responses (status code, body) - - Webhook requests received from Play.ht (payload) - - Updates to the `newsletters` table - - Any errors encountered - 4. The system handles partial execution gracefully (i.e., if triggered before Epic 4 is complete, it logs a message and exits). - -**Epic 6: Web Interface Development** - -- Goal: Develop a user-friendly and mobile-responsive web interface, starting with an initial structure generated by a tool like Vercel v0 and styled with Tailwind CSS using design tokens, to display newsletters and provide access to podcast content. -- Story 6.1: Initial UI Structure Generation - - As a developer, I want to use a tool like Vercel v0 to generate the initial structure of the web interface, so that I have a basic layout to work with. - - Acceptance Criteria: - 1. A tool (e.g., Vercel v0) is used to generate the initial HTML/CSS/React component structure for the web interface. - 2. The generated structure includes basic layouts for the newsletter list page (`/newsletters`) and detail page (`/newsletters/{date}`), as well as a basic podcast player component. - 3. The generated structure incorporates the desired "modern and minimalist, with synthwave technical glowing purple vibes" aesthetic, using initial Tailwind CSS classes. -- Story 6.2: Newsletter List Page - - As a user, I want to see a list of current and past newsletters, so that I can easily browse available content. - - Acceptance Criteria: - 1. The `/newsletters` page displays a list of newsletters fetched from the `newsletters` table. - 2. Each item in the list displays the newsletter title (e.g., "Daily Digest - YYYY-MM-DD") and date. - 3. Each item is a link to the corresponding detail page (`/newsletters/{date}`). - 4. The list is paginated (e.g., 10 newsletters per page) or provides infinite scrolling. - 5. The page is styled using Tailwind CSS, adhering to the project's design tokens. -- Story 6.3: Newsletter - Detail Page - _ As a user, I want to be able to read the newsletter content and download it, so that I can conveniently consume the information and access it offline. - _ Acceptance Criteria: 1. The `/newsletters/{date}` page displays the full newsletter content from the `newsletters` table. 2. The newsletter content is displayed in a readable format, preserving HTML formatting. 3. A download option (e.g., a button) is provided to download the newsletter in HTML format. 4. The page is styled using Tailwind CSS, adhering to the project's design tokens. -- Story 6.4: Podcast Player Integration - - As a user, I want to listen to generated podcasts within the web interface, so that I can consume the content in audio format. - - Acceptance Criteria: - 1. The `/newsletters/{date}` page includes a web player component. - 2. The web player uses the `podcast_url` from the `newsletters` table as the audio source. - 3. The player includes standard playback controls (play, pause, volume, progress bar). - 4. The player is styled using Tailwind CSS, adhering to the project's design tokens. -- Story 6.5: Mobile Responsiveness - - As a developer, I want the web interface to be mobile-responsive, so that it is accessible and usable on various devices. - - Acceptance Criteria: - 1. The layout of both the `/newsletters` and `/newsletters/{date}` pages adapts to different screen sizes and orientations, using Tailwind CSS responsive modifiers. - 2. The interface is usable and readable on both desktop and mobile devices, with appropriate font sizes, spacing, and navigation. - -## Key Reference Documents - -_(This section will be created later, from the sections prior to this being carved up into smaller documents)_ - -## Out of Scope Ideas Post MVP - -- User Authentication and Management -- Subscription Management -- Admin Dashboard - - Viewing and updating daily podcast settings - - Prompt management for summarization - - UI for template modification -- Enhanced Newsletter Customization -- Additional Content Digests - - Configuration and creation of different digests - - Support for content sources beyond Hacker News -- Advanced scraping techniques (e.g., Playwright) - -## Change Log - -| Change | Date | Version | Description | Author | -| :------------ | :--------- | :------ | :------------------------------------------------------------------------------ | :----- | -| Initial Draft | 2024-07-26 | 0.1 | Initial draft of the Product Requirements Document (YOLO, Simplified PM-to-Dev) | 2-pm | - --- checklists still follow diff --git a/BETA-V3/v3-demos/project1/technical-preferences.txt b/BETA-V3/v3-demos/project1/technical-preferences.txt new file mode 100644 index 00000000..987dcff6 --- /dev/null +++ b/BETA-V3/v3-demos/project1/technical-preferences.txt @@ -0,0 +1,20 @@ +# User-Defined Preferred Patterns and Preferences + +Handlers deal with request and response logic only, pass everything to and from a business layer. +Hex architecture + +Wrap External APIs in Facades +Wrap 3rd party libraries in Facades and use factory pattern when there could be multiple choices to use for a similar type of thing. + +Mock the facade when unit testing a file that uses it. +Unit test the facade itself with the real library do not mock it - but beware of non determinism that will need to be accounted for with some libraries such as date libraries. +Use common sense with what should or should not be behind a facade. +for facades around APIs - intercept requests and mock responses for unit tests. + +use RTL for react testing + +if building ui with Next JS, host in vercel and favor starting projects with https://vercel.com/templates/next.js/supabase + +Tailwind and Shadcn are good choices for UI. + +80% unit test coverage. diff --git a/BETA-V3/v3-demos/project1/ux-ui-spec.txt b/BETA-V3/v3-demos/project1/ux-ui-spec.txt new file mode 100644 index 00000000..949b244a --- /dev/null +++ b/BETA-V3/v3-demos/project1/ux-ui-spec.txt @@ -0,0 +1,104 @@ +# BMad DiCaster UI/UX Specification + +## Introduction + +This document defines the user experience goals, information architecture, user flows, and visual design specifications for the BMad DiCaster's user interface. + +- **Link to Primary Design Files:** N/A (Low-fidelity wireframes described below; detailed mockups to be created during development) +- **Link to Deployed Storybook / Design System:** N/A (Basic component definitions below; full design system not in MVP scope) + +## Overall UX Goals & Principles + +- **Target User Personas:** Primary User: A developer and Hacker News enthusiast who wants a quick, daily overview of top HN posts. + + - The main need is to save time and efficiently consume HN content. + +- **Usability Goals:** Efficiency: + + - The UI should allow for quick scanning of daily summaries. + - Access to full articles and podcast versions should be straightforward. + +- **Design Principles:** + + - Minimalism: Clean layout with only essential information to avoid clutter. + - Clarity: Easy-to-read typography and clear visual hierarchy. + - Speed: Fast loading times and quick access to content. + - Synthwave Aesthetic: Incorporate the visual theme through color accents and typography. + +## Information Architecture (IA) + +- **Site Map / Screen Inventory:** + +```mermaid +graph TD A[Newsletter List] --> B(Newsletter Detail); +``` + +- **Navigation Structure:** + - Primary Navigation: A simple link from the "Newsletter List" page to the "Newsletter Detail" page for a selected newsletter. + - Secondary Navigation: + - On the "Newsletter Detail" page, include a "Back to List" link/button. + +## User Flows + +### Viewing a Newsletter + +- **Goal:** To read the content of a specific daily newsletter. +- **Steps / Diagram:** + +<!-- end list --> + +```mermaid +graph TD A[Newsletter List Page] --> B(Select Newsletter); B --> C[Newsletter Detail Page]; C --> D{Read Newsletter}; C --> E{Download Newsletter (Optional)}; C --> F{Listen to Podcast (Optional)}; C --> G{Back to List (Optional)}; +``` + +## Wireframes & Mockups + +- **Newsletter List Page:** + - Description: A simple list of newsletters, ordered by date (most recent first). Each list item includes the title/subject and date of the newsletter, with navigation to the detail page. + - Key Elements: Page Title ("Daily DiCaster" or similar), Newsletter list. +- **Newsletter Detail Page:** + - Description: Display the full content of the selected newsletter. Include options for playing the podcast, downloading the newsletter, and navigating back to the list page. + - Key Elements: Newsletter Title, Newsletter Content, Podcast Player, Download Button, Back to List Button/Link. + +## Component Library / Design System Reference + +- **NewsletterCard:** + - Used in the Newsletter List Page to display a brief summary of each newsletter. + - Includes the newsletter title and date. + - Provides a link/button to navigate to the Newsletter Detail Page. +- **PodcastPlayer:** + - Used in the Newsletter Detail Page to embed and control the podcast playback. +- **DownloadButton:** + - Used in the Newsletter Detail Page to allow users to download the newsletter. +- **BackButton:** + - Used in the Newsletter Detail Page to navigate back to the Newsletter List Page. + +## Branding & Style Guide Reference + +- **Color Palette:** Use Tailwind's default color palette, with `#800080` (purple) as a custom accent color if needed. +- **Typography:** Use Tailwind's default font family (sans-serif) and semantic classes for headings (e.g., `text-2xl font-bold`) and body text (e.g., `text-base`). +- **Iconography:** Use a minimalist icon set (e.g., Font Awesome), styled with Tailwind classes. +- **Spacing & Grid:** Use Tailwind's utility classes for spacing (e.g., `p-4`, `m-2`) and grid layout (e.g., `grid`, `grid-cols-2`). + +## Accessibility (AX) Requirements + +- **Target Compliance:** WCAG 2.1 Level A +- **Specific Requirements:** + - Keyboard Navigation: Ensure all interactive elements (links, buttons) are focusable and operable using the keyboard. + - Semantic HTML: Use appropriate HTML5 elements for structure and meaning (e.g., `<nav>`, `<article>`, `<button>`). + - Alternative Text: Provide descriptive alternative text for images (if any are used). + - Color Contrast: Ensure sufficient color contrast between text and background elements. + +## Responsiveness + +- **Breakpoints:** Use Tailwind's default breakpoints: `sm`, `md`, `lg`, `xl`. +- **Adaptation Strategy:** Use Tailwind's responsive prefixes (`sm:`, `md:`, `lg:`, `xl:`) to apply styles conditionally at different breakpoints. + - For example: + - `text-base md:text-lg` (base font size on small screens, larger on medium screens and up). + - `grid grid-cols-1 md:grid-cols-2` (single-column grid on small screens, two-column grid on medium screens and up). + +## Change Log + +| Change | Date | Version | Description | Author | +| :----------------------------------- | :--------- | :------ | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :----- | +| Initial Draft | 2025-05-13 | 0.1 | Initial draft of the UI/UX Specification | 4-arch | diff --git a/BETA-V3/web-agent-modes/3-architect.md b/BETA-V3/web-agent-modes/3-architect.md index fbc9796b..7d0a5219 100644 --- a/BETA-V3/web-agent-modes/3-architect.md +++ b/BETA-V3/web-agent-modes/3-architect.md @@ -172,6 +172,7 @@ To perform deep research effectively, please be aware: - Explain the importance of these technical stories for enabling the functional requirements and successful project execution. - Collaborate with the user to refine these stories (clear description, acceptance criteria) and suggest adding them to the project backlog or relevant epics. - Review existing epics/stories from the PRD and suggest technical considerations or acceptance criteria refinements to ensure they are implementable based on the chosen architecture. For example, specifying API endpoints to be called, data formats, or critical library versions. + - After collaboration, prepare a concise summary detailing all proposed additions, updates, or modifications to epics and user stories. If no changes are identified, explicitly state this. 6. **Validate Architecture Against Checklist & Finalize Output:** - Once the main architecture document components have been drafted and reviewed with the user, perform a comprehensive review using the `architect-checklist.txt`. @@ -202,6 +203,7 @@ To perform deep research effectively, please be aware: - A comprehensive Architecture Document, structured according to the `architecture-tmpl.txt` (which is all markdown) or an agreed-upon format, including all sections detailed above. - Clear Mermaid diagrams for architecture overview, data models, etc. - A list of new or refined technical user stories/tasks ready for backlog integration. +- A summary of any identified changes (additions, updates, modifications) required for existing epics or user stories, or an explicit confirmation if no such changes are needed. - A completed `architect-checklist.txt` (or a summary of its validation). - Optionally, if UI components are involved and the user agrees: A prompt for a "Design Architect" appended to the main architecture document, summarizing relevant UI considerations and outlining the Design Architect's next steps. diff --git a/BETA-V3/web-agent-modes/4-design-architect.md b/BETA-V3/web-agent-modes/4-design-architect.md index d56a603f..adf91748 100644 --- a/BETA-V3/web-agent-modes/4-design-architect.md +++ b/BETA-V3/web-agent-modes/4-design-architect.md @@ -119,10 +119,17 @@ To define the technical architecture for the frontend application. This includes ### Key Activities & Instructions -1. **Review Inputs & Establish Context:** +1. **Confirm Interaction Mode:** + + - Before proceeding with detailed architecture definition, explicitly ask the user if they prefer to proceed: + - **Incrementally (Default):** Work through each section of the `front-end-architecture.md` (Overall Philosophy, Directory Structure, Component Strategy, etc.) step-by-step. For each section: explain its purpose, present a draft, discuss it with the user, incorporate feedback, and seek their approval before moving to the next section. This is recommended for ensuring detailed alignment. + - **"YOLO" Mode:** Develop a comprehensive draft of the entire `front-end-architecture.md` document, covering all relevant sections, and present it for review once largely complete. Use this mode if the user expresses a desire for faster drafting of initial ideas. + - Confirm the chosen mode with the user. + +2. **Review Inputs & Establish Context:** - Thoroughly review the inputs, including the UI/UX Specification and the main Architecture Document (especially "Definitive Tech Stack Selections", API contracts, and the documented overall system structure like monorepo/polyrepo choices). - Ask clarifying questions to bridge any gaps between the UI/UX vision and the overall system architecture. -2. **Define Overall Frontend Philosophy & Patterns (for `front-end-architecture.md`):** +3. **Define Overall Frontend Philosophy & Patterns (for `front-end-architecture.md`):** - Based on the main architecture's tech stack and overall system structure (monorepo/polyrepo, backend service details), confirm and detail: - Framework & Core Libraries choices. - High-level Component Architecture strategy. @@ -130,41 +137,59 @@ To define the technical architecture for the frontend application. This includes - Data Flow principles. - Styling Approach. - Key Design Patterns to be employed. -3. **Specify Detailed Frontend Directory Structure (for `front-end-architecture.md`):** +4. **Specify Detailed Frontend Directory Structure (for `front-end-architecture.md`):** - Collaboratively define or refine the frontend-specific directory structure, ensuring it aligns with the chosen framework and promotes modularity and scalability. -4. **Outline Component Strategy & Conventions (for `front-end-architecture.md`):** +5. **Outline Component Strategy & Conventions (for `front-end-architecture.md`):** - Define Component Naming & Organization conventions. - Establish the "Template for Component Specification" (as per `front-end-architecture.md`), emphasizing that most components will be detailed emergently but must follow this template. - Optionally, specify a few absolutely foundational/shared UI components (e.g., a generic Button or Modal wrapper if the chosen UI library needs one, or if no UI library is used). -5. **Detail State Management Setup & Conventions (for `front-end-architecture.md`):** +6. **Detail State Management Setup & Conventions (for `front-end-architecture.md`):** - Based on the high-level strategy, detail: - Chosen Solution and core setup. - Conventions for Store Structure / Slices (e.g., "feature-based slices"). Define any genuinely global/core slices (e.g., session/auth). - Conventions for Selectors and Actions/Reducers/Thunks. Provide templates or examples. -6. **Plan API Interaction Layer (for `front-end-architecture.md`):** +7. **Plan API Interaction Layer (for `front-end-architecture.md`):** - Define the HTTP Client Setup. - Establish patterns for Service Definitions (how API calls will be encapsulated). - Outline frontend Error Handling & Retry strategies for API calls. -7. **Define Routing Strategy (for `front-end-architecture.md`):** +8. **Define Routing Strategy (for `front-end-architecture.md`):** - Confirm the Routing Library. - Collaboratively define the main Route Definitions and any Route Guards. -8. **Specify Build, Bundling, and Deployment Details (for `front-end-architecture.md`):** +9. **Specify Build, Bundling, and Deployment Details (for `front-end-architecture.md`):** - Outline the frontend-specific Build Process & Scripts. - Discuss and document Key Bundling Optimizations. - Confirm Deployment to CDN/Hosting details relevant to the frontend. -9. **Refine Frontend Testing Strategy (for `front-end-architecture.md`):** +10. **Refine Frontend Testing Strategy (for `front-end-architecture.md`):** - Elaborate on the main testing strategy with specifics for: Component Testing, UI Integration/Flow Testing, and E2E UI Testing scope and tools. -10. **Document Accessibility (AX) Implementation Plan (for `front-end-architecture.md`):** - - Translate AX requirements from the UI/UX spec into technical implementation guidelines. 11. **Outline Performance Considerations (for `front-end-architecture.md`):** - List key frontend-specific performance strategies to be employed. -12. **Output Generation:** +12. **Document Drafting & Confirmation (Guided by `front-end-architecture-tmpl.txt`):** - - Incrementally populate the sections of the `front-end-architecture-tmpl.txt` file. - - Present sections for user review and confirmation. + - **If "Incremental Mode" was selected:** + - For each relevant section of the `front-end-architecture.md` (as outlined in steps 3-11 above, covering topics from Overall Philosophy to Performance Considerations): + - Explain the purpose of the section. + - Present a draft for that section. + - Discuss the draft with the user, incorporate their feedback, and iterate as needed. + - Obtain explicit user approval for the section before proceeding to the next. + - Ensure all placeholder links and references are correctly noted within each section. + - Once all sections are individually approved, confirm with the user that the overall `front-end-architecture.md` document is populated and ready for the checklist review. + - **If "YOLO Mode" was selected:** + - Collaboratively populate all relevant sections of the `front-end-architecture-tmpl.txt` (as outlined in steps 3-11 above) to create a comprehensive first draft. + - Present the complete draft of `front-end-architecture.md` to the user for a holistic review. + - Discuss the draft, incorporate feedback, and iterate on the document as needed. + - Ensure all placeholder links and references are correctly noted throughout the document. + - Obtain explicit user approval for the entire `front-end-architecture.md` document before proceeding to the checklist review. -13. **Checklist Review and Finalization:** - - Once the `front-end-architecture.md` has been populated and reviewed with the user, use the `frontend-architecture-checklist.txt`. +13. **Identify & Summarize Epic/Story Impacts (Frontend Focus):** + + - After the `front-end-architecture.md` is confirmed, review it in context of existing epics and user stories (if provided or known). + - Identify any frontend-specific technical tasks that might need to be added as new stories or sub-tasks (e.g., "Implement responsive layout for product details page based on defined breakpoints," "Set up X state management slice for user profile," "Develop reusable Y component as per specification"). + - Identify if any existing user stories require refinement of their acceptance criteria due to frontend architectural decisions (e.g., specifying interaction details, component usage, or performance considerations for UI elements). + - Collaborate with the user to define these additions or refinements. + - Prepare a concise summary detailing all proposed additions, updates, or modifications to epics and user stories related to the frontend. If no changes are identified, explicitly state this (e.g., "No direct impacts on existing epics/stories were identified from the frontend architecture"). + +14. **Checklist Review and Finalization:** + - Once the `front-end-architecture.md` has been populated and reviewed with the user, and epic/story impacts have been summarized, use the `frontend-architecture-checklist.txt`. - Go through each item in the checklist to ensure the `front-end-architecture.md` is comprehensive and all sections are adequately addressed. - For each checklist item, confirm its status (e.g., [x] Completed, [ ] N/A, [!] Needs Attention). - If deficiencies or areas needing more detail are identified: