trentuna/candle-annotator
8
0
Fork 0
Code Issues Pull requests Releases Packages Activity Actions
candle-annotator/openspec/changes/ml-db-consolidation/proposal.md
Marko Djordjevic 5f70f13da3 feat: migrate from SQLite to PostgreSQL - complete schema and API updates 
- Remove better-sqlite3, add pg driver
- Convert schema to PostgreSQL types (serial, timestamp, boolean, jsonb)
- Generate fresh PostgreSQL migrations
- Update database connection layer with pg.Pool
- Fix all API routes: remove JSON.parse/stringify, use native timestamps and booleans
- Update drizzle.config.ts and .env.example for PostgreSQL
2026-02-17 13:43:06 +01:00

3.1 KiB
Raw Blame History

Why

The project currently runs two separate database servers: SQLite (via Drizzle ORM) for the Next.js frontend and PostgreSQL for the ML service. This creates unnecessary operational complexity — two different ORMs, two migration systems, two backup strategies, and no ability for the ML service to directly query annotation/candle data. Consolidating to PostgreSQL as the single database simplifies deployment, enables direct cross-service data access, and reduces the infrastructure footprint.

What Changes

  • BREAKING: Replace SQLite/better-sqlite3/Drizzle with PostgreSQL/Drizzle (pg driver) for the Next.js frontend
  • Remove the candle-data Docker volume (SQLite file storage) and DATABASE_PATH env var
  • Migrate all frontend tables (charts, candles, annotations, annotation_types, span_annotations, span_label_types) into the existing PostgreSQL instance
  • Update Drizzle schema and config to target PostgreSQL instead of SQLite
  • Regenerate Drizzle migrations for PostgreSQL dialect (column types change: integer → serial, real → double precision, timestamps as proper timestamp types, etc.)
  • Update the ML service to share the same PostgreSQL database (or a separate schema within it) so it can directly query candle/annotation data instead of relying on CSV/JSON exports
  • Update docker-compose.yml to remove SQLite volume dependency and point the frontend at PostgreSQL
  • Update environment variables: frontend gets DATABASE_URL pointing to PostgreSQL

Capabilities

New Capabilities

  • postgres-data-layer: Unified PostgreSQL data access layer for the Next.js frontend, replacing the SQLite/better-sqlite3 setup with Drizzle's PostgreSQL driver

Modified Capabilities

  • docker-deployment: Container configuration changes — remove SQLite volume, add PostgreSQL dependency for the frontend service, update environment variables
  • ml-training: ML service can now query annotations and candle data directly from PostgreSQL instead of requiring CSV/JSON file exports

Impact

  • Database schema: All 6 frontend tables move to PostgreSQL with type adaptations (SQLite integers → PostgreSQL serial/integer/timestamp)
  • ORM layer: src/lib/db/index.ts switches from better-sqlite3 to postgres driver; schema types in src/lib/db/schema.ts change to PostgreSQL equivalents
  • Dependencies: Remove better-sqlite3, add postgres (or pg) npm package for Drizzle's PostgreSQL adapter
  • Migrations: Existing SQLite migrations become obsolete; new PostgreSQL migrations needed
  • Docker: candle-annotator service gains depends_on: postgres, loses candle-data volume mount
  • Environment: .env and .env.example updated with PostgreSQL connection string for frontend
  • ML service: services/ml/app/db.py gains access to frontend tables (candles, annotations) for direct querying
  • Data migration: Existing SQLite data needs a one-time migration script to PostgreSQL
  • API routes: All Next.js API routes using db from src/lib/db continue working (Drizzle abstracts the driver change), but queries using SQLite-specific syntax may need adjustment