38df874255
- Archived change to openspec/changes/archive/2026-02-17-ml-db-consolidation/ - Created new postgres-data-layer spec with PostgreSQL connection, schema definitions, Drizzle migrations, npm deps, and SQLite migration requirements - Updated docker-deployment spec: Docker Compose now PostgreSQL-based (postgres dependency, ml-data volume, DATABASE_URL); env vars updated (DATABASE_URL added, DATABASE_PATH removed); database persistence updated to PostgreSQL volumes; health check updated to PostgreSQL - Updated ml-training spec: added database name scenario (candle_annotator) and new direct annotation data access requirement Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
53 lines
3.4 KiB
Markdown
53 lines
3.4 KiB
Markdown
## 1. Dependencies and Configuration
|
|
|
|
- [x] 1.1 Remove `better-sqlite3` and `@types/better-sqlite3` from package.json
|
|
- [x] 1.2 Add `pg` and `@types/pg` to package.json dependencies
|
|
- [x] 1.3 Run `npm install` to update node_modules and lockfile
|
|
- [x] 1.4 Update `drizzle.config.ts` to target `postgresql` dialect with `DATABASE_URL` env var
|
|
- [x] 1.5 Update `.env.example` — replace `DATABASE_PATH` with `DATABASE_URL=postgresql://ml_user:ml_password@postgres:5432/candle_annotator`
|
|
|
|
## 2. Drizzle Schema Migration (SQLite → PostgreSQL)
|
|
|
|
- [x] 2.1 Rewrite `src/lib/db/schema.ts` — replace all `sqliteTable` with `pgTable`, apply type mappings (integer→serial, integer→timestamp, integer→boolean, real→doublePrecision, text JSON→jsonb)
|
|
- [x] 2.2 Delete all existing SQLite migration files in `drizzle/` directory
|
|
- [x] 2.3 Run `drizzle-kit generate` to produce fresh PostgreSQL migration SQL
|
|
- [x] 2.4 Review generated migration SQL for correctness
|
|
|
|
## 3. Database Connection Layer
|
|
|
|
- [x] 3.1 Rewrite `src/lib/db/index.ts` — replace `better-sqlite3` driver with `pg.Pool` (max: 10), read `DATABASE_URL` from env, fail if missing
|
|
- [x] 3.2 Update migration runner to use PostgreSQL-compatible execution (skip during build phase via `NEXT_PHASE` check)
|
|
- [x] 3.3 Update all imports if any changed (verify `db` export still works for API routes)
|
|
|
|
## 4. API Route Adjustments
|
|
|
|
- [x] 4.1 Audit all Next.js API routes using `db` for SQLite-specific syntax (e.g., integer booleans, raw SQL fragments)
|
|
- [x] 4.2 Fix any SQLite-specific query patterns to work with PostgreSQL (boolean handling, timestamp handling, jsonb operations)
|
|
- [x] 4.3 Update health check endpoint (`/api/health`) to verify PostgreSQL connectivity
|
|
|
|
## 5. Docker and Deployment
|
|
|
|
- [x] 5.1 Update `docker-compose.yml` — rename `POSTGRES_DB` to `candle_annotator`, add `DATABASE_URL` env to candle-annotator service, add `depends_on: postgres` with health check condition
|
|
- [x] 5.2 Remove `candle-data` volume from `docker-compose.yml` (SQLite volume)
|
|
- [x] 5.3 Update `Dockerfile` if it references SQLite or `DATABASE_PATH`
|
|
- [x] 5.4 Update ML service database connection — change database name from `ml_db` to `candle_annotator` in environment config
|
|
|
|
## 6. ML Service Direct Data Access
|
|
|
|
- [x] 6.1 Add SQLAlchemy table reflections or raw queries in the ML service for reading `candles`, `annotations`, `span_annotations`, `charts` tables
|
|
- [x] 6.2 Update ML training pipeline to query candle/annotation data from PostgreSQL instead of CSV/JSON exports
|
|
- [x] 6.3 Remove or deprecate any CSV/JSON export code paths that are no longer needed
|
|
|
|
## 7. Data Migration Script
|
|
|
|
- [x] 7.1 Create `scripts/migrate-sqlite-to-postgres.ts` — read all 6 tables from SQLite, apply type conversions (timestamps, booleans, JSON→jsonb), insert into PostgreSQL
|
|
- [x] 7.2 Make the script idempotent (skip or clear+re-insert with flag)
|
|
- [x] 7.3 Test migration script with existing SQLite data
|
|
|
|
## 8. Testing and Verification
|
|
|
|
- [x] 8.1 Run the full application locally with PostgreSQL — verify all API routes work
|
|
- [x] 8.2 Verify ML service can query candle/annotation data from shared database
|
|
- [x] 8.3 Run `docker compose up` and verify all services start correctly with new configuration
|
|
- [x] 8.4 Update `DEPLOYMENT.md` with new deployment steps (PostgreSQL migration, data migration script, rollback procedure)
|
|
- [x] 8.5 Update `README.md` and `CLAUDE_DESCRIPTION.md` with database architecture changes
|