feat: migrate from SQLite to PostgreSQL - complete schema and API updates

- Remove better-sqlite3, add pg driver
- Convert schema to PostgreSQL types (serial, timestamp, boolean, jsonb)
- Generate fresh PostgreSQL migrations
- Update database connection layer with pg.Pool
- Fix all API routes: remove JSON.parse/stringify, use native timestamps and booleans
- Update drizzle.config.ts and .env.example for PostgreSQL

openspec/changes/ml-db-consolidation/specs/postgres-data-layer/spec.md

@@ -0,0 +1,80 @@
+## ADDED Requirements
+
+### Requirement: PostgreSQL connection via Drizzle ORM
+The Next.js application SHALL connect to PostgreSQL using Drizzle ORM with the `node-postgres` (`pg`) driver. The connection SHALL use a pool with a configurable maximum number of connections (default: 10). The connection string SHALL be read from the `DATABASE_URL` environment variable.
+
+#### Scenario: Successful connection
+- **WHEN** the application starts with a valid `DATABASE_URL` pointing to a running PostgreSQL instance
+- **THEN** Drizzle ORM establishes a connection pool and the `db` export is ready for queries
+
+#### Scenario: Missing DATABASE_URL
+- **WHEN** the `DATABASE_URL` environment variable is not set
+- **THEN** the application SHALL fail to start with an error message indicating the missing variable
+
+#### Scenario: Database unreachable
+- **WHEN** the PostgreSQL instance is not reachable at the configured URL
+- **THEN** the application SHALL fail to start with a connection error
+
+### Requirement: PostgreSQL schema definitions
+The Drizzle schema SHALL define all frontend tables using `pgTable` from `drizzle-orm/pg-core`. The following tables SHALL be defined: `charts`, `candles`, `annotation_types`, `annotations`, `span_label_types`, `span_annotations`.
+
+#### Scenario: Charts table schema
+- **WHEN** the schema is loaded
+- **THEN** the `charts` table has columns: `id` (serial, primary key), `name` (text, unique, not null), `created_at` (timestamp, not null, default now)
+
+#### Scenario: Candles table schema
+- **WHEN** the schema is loaded
+- **THEN** the `candles` table has columns: `id` (serial, primary key), `chart_id` (integer, foreign key to charts.id, not null), `time` (timestamp, not null), `open` (double precision, not null), `high` (double precision, not null), `low` (double precision, not null), `close` (double precision, not null), with a unique index on `(chart_id, time)`
+
+#### Scenario: Annotation types table schema
+- **WHEN** the schema is loaded
+- **THEN** the `annotation_types` table has columns: `id` (serial, primary key), `name` (text, unique, not null), `display_name` (text, not null), `color` (text, not null), `category` (text, not null), `icon` (text, nullable), `is_active` (boolean, not null, default true), `created_at` (timestamp, not null, default now)
+
+#### Scenario: Annotations table schema
+- **WHEN** the schema is loaded
+- **THEN** the `annotations` table has columns: `id` (serial, primary key), `chart_id` (integer, foreign key to charts.id, not null), `timestamp` (timestamp, not null), `label_type` (text, not null), `geometry` (jsonb, nullable), `color` (text, default '#3b82f6'), `created_at` (timestamp, not null, default now)
+
+#### Scenario: Span label types table schema
+- **WHEN** the schema is loaded
+- **THEN** the `span_label_types` table has columns: `id` (serial, primary key), `name` (text, unique, not null), `display_name` (text, not null), `color` (text, not null), `hotkey` (text, nullable), `is_active` (boolean, not null, default true), `sort_order` (integer, not null, default 0), `created_at` (timestamp, not null, default now)
+
+#### Scenario: Span annotations table schema
+- **WHEN** the schema is loaded
+- **THEN** the `span_annotations` table has columns: `id` (serial, primary key), `chart_id` (integer, foreign key to charts.id, not null), `start_time` (timestamp, not null), `end_time` (timestamp, not null), `label` (text, not null), `confidence` (integer, nullable), `outcome` (text, nullable), `notes` (text, nullable), `sub_spans` (jsonb, nullable), `color` (text, not null, default '#2196F3'), `source` (text, not null, default 'human'), `model_prediction` (jsonb, nullable), `created_at` (timestamp, not null, default now)
+
+### Requirement: PostgreSQL migrations via Drizzle Kit
+The project SHALL use Drizzle Kit to generate and apply PostgreSQL migrations. The `drizzle.config.ts` SHALL target the `postgresql` dialect. Existing SQLite migrations SHALL be removed.
+
+#### Scenario: Generate migrations
+- **WHEN** `drizzle-kit generate` is executed
+- **THEN** a new SQL migration file is created in the `drizzle/` directory with PostgreSQL-dialect DDL
+
+#### Scenario: Apply migrations at startup
+- **WHEN** the application starts (not during build phase)
+- **THEN** Drizzle runs pending migrations against the PostgreSQL database
+
+#### Scenario: Skip migrations during build
+- **WHEN** `NEXT_PHASE` is `phase-production-build` or `phase-development-build`
+- **THEN** migration execution is skipped
+
+### Requirement: npm dependency changes
+The project SHALL remove `better-sqlite3` and `@types/better-sqlite3` from dependencies and add `pg` and `@types/pg`.
+
+#### Scenario: Dependencies updated
+- **WHEN** `package.json` is inspected
+- **THEN** `better-sqlite3` and `@types/better-sqlite3` are absent, and `pg` and `@types/pg` are present in dependencies
+
+### Requirement: Data migration from SQLite to PostgreSQL
+The project SHALL include a one-time migration script at `scripts/migrate-sqlite-to-postgres.ts` that reads all data from the SQLite database and inserts it into PostgreSQL with appropriate type conversions.
+
+#### Scenario: Migrate all tables
+- **WHEN** the migration script is executed with both databases accessible
+- **THEN** all rows from charts, candles, annotation_types, annotations, span_label_types, and span_annotations are transferred to PostgreSQL
+
+#### Scenario: Type conversions applied
+- **WHEN** data is migrated
+- **THEN** SQLite integer timestamps are converted to PostgreSQL timestamps, integer booleans (0/1) are converted to PostgreSQL booleans, and text JSON fields are inserted as jsonb
+
+#### Scenario: Idempotent execution
+- **WHEN** the migration script is run a second time on an already-migrated database
+- **THEN** the script either skips existing data or clears and re-inserts (with a flag), without creating duplicates