candle-annotator/DEPLOYMENT.md

Deployment Guide

Prerequisites

• Node.js 18.x or higher
• npm 9.x or higher
• PostgreSQL 16 or higher

Local Development Setup

1. Install Dependencies

npm install

2. Database Setup

PostgreSQL Setup

The application uses PostgreSQL for all data storage. Set up the database:

# Create database
createdb candle_annotator

# Create user (if needed)
createuser -P ml_user
# Enter password: ml_password

# Grant privileges
psql -c "GRANT ALL PRIVILEGES ON DATABASE candle_annotator TO ml_user;"

Environment Configuration

Create a .env file in the project root based on .env.example:

cp .env.example .env

Edit .env to customize:

DATABASE_URL=postgresql://ml_user:ml_password@localhost:5432/candle_annotator
NODE_ENV=development
PORT=3000

# Authentication (Auth.js v5)
AUTH_SECRET=your_strong_random_secret_here
AUTH_TRUST_HOST=true

# Google OAuth
AUTH_GOOGLE_ID=your_google_oauth_client_id
AUTH_GOOGLE_SECRET=your_google_oauth_client_secret

# Default admin user
DEFAULT_ADMIN_EMAIL=admin@example.com
DEFAULT_ADMIN_PASSWORD=your_strong_password_here

Google OAuth Setup

To enable Google OAuth sign-in:

1. Create a Google OAuth Application:

   • Go to Google Cloud Console
   • Create a new project
   • Enable the "Google+ API"
   • Go to "Credentials" and click "Create Credentials" → "OAuth 2.0 Client IDs"
   • Select "Web application"
   • Add authorized redirect URIs:
     • http://localhost:3000/api/auth/callback/google (local development)
     • https://yourdomain.com/api/auth/callback/google (production)
   • Copy the Client ID and Client Secret

2. Set Environment Variables:

   AUTH_GOOGLE_ID=<your-client-id>
   AUTH_GOOGLE_SECRET=<your-client-secret>
   

3. Generate AUTH_SECRET:

   openssl rand -hex 32
   

Database Migrations

After setting up the database and environment variables, run the migrations:

1. Run Drizzle migrations (creates schema):

   npm run db:migrate
   

2. Run the user migration script (creates default admin user, backfills user_id):

   DEFAULT_ADMIN_EMAIL=admin@example.com \
   DEFAULT_ADMIN_PASSWORD=your_password \
   npx tsx scripts/migrate-users.ts
   

   This script:

   • Creates a default admin user with the specified email and password (hashed with bcryptjs)
   • Backfills user_id on all existing rows in: charts, annotations, annotation_types, span_annotations, span_label_types
   • Is idempotent (safe to run multiple times)

Run Migrations

Database migrations run automatically on application startup. To run manually:

npx drizzle-kit generate
npx drizzle-kit migrate

3. Start Development Server

npm run dev

The application will be available at:

• http://localhost:3000

4. Verify Setup

1. Open the application in your browser
2. Upload a sample CSV file with OHLC data (columns: time, open, high, low, close)
3. Verify the candlestick chart renders correctly
4. Test annotation tools (Break Up, Break Down, Draw Line, Delete)
5. Export annotations as CSV

CSV File Format

The application expects CSV files with the following format:

time,open,high,low,close
1700000000,1.0500,1.0520,1.0490,1.0510
1700000060,1.0510,1.0530,1.0505,1.0525

Time column formats:

• Unix timestamp (seconds): 1700000000
• Date string: 2024-01-15

4. Migrating from SQLite (if applicable)

If you have existing data in an SQLite database from a previous version, use the migration script:

# Run the migration script
npm run migrate:sqlite-to-postgres

# Or with TypeScript directly
npx ts-node scripts/migrate-sqlite-to-postgres.ts

This script will:

• Read all data from the SQLite database (data/candles.db)
• Convert data types (timestamps, booleans, JSON→jsonb)
• Insert data into PostgreSQL
• Skip if run multiple times (idempotent)

Building for Production

npm run build

Running Production Build

npm run build
npm start

The production server will run on port 3000 by default.

Troubleshooting

Database Connection Issues

If the application fails to connect to PostgreSQL:

1. Verify PostgreSQL is running:

   pg_isready -h localhost -p 5432
   

2. Check DATABASE_URL environment variable:

   echo $DATABASE_URL
   

3. Verify credentials:

   psql -U ml_user -d candle_annotator
   

Database Issues

If you want to reset the database:

1. Stop the application
2. Drop and recreate the database:

   dropdb candle_annotator
   createdb candle_annotator
   psql -c "GRANT ALL PRIVILEGES ON DATABASE candle_annotator TO ml_user;"
   

3. Restart the application (migrations will run automatically)

Port Already in Use

If port 3000 is already in use, you can specify a different port:

PORT=3001 npm run dev

Environment Variables

Required Variables

• DATABASE_URL - PostgreSQL connection string (e.g., postgresql://ml_user:ml_password@localhost:5432/candle_annotator)
• NODE_ENV - Environment (development or production)
• PORT - Server port (default: 3000)
• AUTH_SECRET - Random secret for Auth.js JWT signing (generate with openssl rand -hex 32)
• AUTH_GOOGLE_ID - Google OAuth Client ID
• AUTH_GOOGLE_SECRET - Google OAuth Client Secret
• DEFAULT_ADMIN_EMAIL - Default admin user email (used by migration script)
• DEFAULT_ADMIN_PASSWORD - Default admin user password (used by migration script)

Authentication Variables

• AUTH_TRUST_HOST - Set to true when using HTTP (localhost development), false for HTTPS production with proper domain
• AUTH_GOOGLE_ID - Google OAuth Client ID from Google Cloud Console
• AUTH_GOOGLE_SECRET - Google OAuth Client Secret from Google Cloud Console

Optional Variables for ML Inference

• INFERENCE_API_URL - ML service endpoint (default: http://localhost:8001)
• INFERENCE_API_TIMEOUT - Request timeout in ms (default: 30000)
• INFERENCE_BATCH_TIMEOUT - Batch processing timeout in ms (default: 120000)
• NEXT_PUBLIC_PREDICTIONS_ENABLED - Enable predictions UI (default: true)

API Key Variable

• API_KEY - Strong random key for authenticating requests between Next.js and ML service (generate with openssl rand -hex 32)

File Structure

.
├── src/
│   ├── app/              # Next.js app router
│   │   ├── api/          # API routes
│   │   ├── layout.tsx    # Root layout
│   │   └── page.tsx      # Main page
│   ├── components/       # React components
│   │   ├── CandleChart.tsx
│   │   ├── SvgOverlay.tsx
│   │   ├── Toolbox.tsx
│   │   └── FileUpload.tsx
│   └── lib/              # Utilities
│       └── db/           # Database configuration
├── data/                 # SQLite database directory
├── drizzle/              # Database migrations
└── public/               # Static assets

ML Service Setup (Optional)

The Candle Annotator includes an optional Python ML service for pattern recognition and prediction.

Prerequisites for ML Service

• Python 3.11+
• TA-Lib C library
• PostgreSQL 16

Local ML Service Setup

1. Install TA-Lib C Library

Linux (Debian/Ubuntu):

sudo apt-get update
sudo apt-get install libta-lib-dev

macOS:

brew install ta-lib

From Source:

wget http://prdownloads.sourceforge.net/ta-lib/ta-lib-0.4.0-src.tar.gz
tar -xzf ta-lib-0.4.0-src.tar.gz
cd ta-lib/
./configure --prefix=/usr
make
sudo make install

2. Install Python Dependencies

cd services/ml
uv sync
#pip install -r requirements.txt

3. Setup PostgreSQL

The ML service shares the same PostgreSQL database as the frontend (candle_annotator). If you've already set up the database in the main setup steps, you're all set. The ML service will use the same connection.

4. Initialize DVC

DVC is used for dataset versioning:

cd services/ml
dvc init #--subdir
dvc remote add -d local /path/to/dvc-storage

5. Run MLflow Tracking Server

MLflow tracks experiments and stores models:

mlflow server \
  --backend-store-uri ./mlruns \
  --default-artifact-root ./mlruns/artifacts \
  --host 0.0.0.0 \
  --port 5000

6. Configure Pipeline

Edit services/ml/config/pipeline.yaml to configure:

• Feature engineering settings
• Model hyperparameters
• Data paths
• MLflow experiment name

7. Start ML Service

cd services/ml
uvicorn app.main:app --host 0.0.0.0 --port 8001 --reload

The inference API will be available at http://localhost:8001

8. Configure Next.js App

Create .env.local in the project root:

INFERENCE_API_URL=http://localhost:8001
INFERENCE_API_TIMEOUT=30000
INFERENCE_BATCH_TIMEOUT=120000
NEXT_PUBLIC_PREDICTIONS_ENABLED=true

Running the ML Pipeline

The ML pipeline consists of:

1. Feature Engineering - Extract TA-Lib indicators from OHLCV data
2. Annotation Ingestion - Convert span annotations to labeled datasets
3. Training - Train models with MLflow tracking
4. Inference - Serve predictions via FastAPI

Train a Model

cd services/ml
python pipeline.py --config config/pipeline.yaml

This will:

• Load raw OHLCV data from data/raw/
• Compute features and save to data/enriched/
• Load annotations and create labeled dataset in data/labeled/
• Train the model with MLflow tracking
• Save model artifacts

Run Individual Stages

# Feature engineering only
python pipeline.py --config config/pipeline.yaml --stage feature_engineering

# Training only (requires labeled data)
python pipeline.py --config config/pipeline.yaml --stage training

View Experiments

Open MLflow UI at http://localhost:5000

Test Inference API

# Check health
curl http://localhost:8001/health

# Get model info
curl http://localhost:8001/model/info

# Predict (requires candles JSON)
curl -X POST http://localhost:8001/predict \
  -H "Content-Type: application/json" \
  -d '{"candles": [...]}'

Docker Deployment

Prerequisites

• Docker (20.10+)
• docker-compose (2.0+)

Build and Run with Docker Compose

The easiest way to deploy is with docker-compose:

docker compose up --build

This will:

1. Build the Next.js app and ML service Docker images
2. Start PostgreSQL (shared by frontend and ML service)
3. Start MLflow tracking server
4. Start the ML inference service (FastAPI)
5. Start the Next.js web application
6. Create named volumes for persistent storage:
   • ml-data - OHLCV data, features, labeled datasets
   • mlflow-data - MLflow experiments and model artifacts
   • postgres-data - PostgreSQL data (all application tables)
7. Enable automatic restart unless stopped

Services will be available at:

• Web UI: http://localhost:3000
• ML Inference API: http://localhost:8001
• MLflow UI: http://localhost:5000
• PostgreSQL: localhost:5432

Running in Detached Mode

docker-compose up -d --build

View logs:

docker-compose logs -f candle-annotator

Stop the service:

docker-compose down

Manual Docker Build and Run

If you prefer to build and run manually:

# Build image
docker build -t candle-annotator .

# Run container
docker run -d \
  -p 3000:3000 \
  -v candle-data:/app/data \
  --restart unless-stopped \
  candle-annotator

Environment Configuration for Docker

Create a .env file in the project root based on .env.example:

cp .env.example .env

Edit .env to customize for your deployment:

NODE_ENV=production
PORT=3000
DATABASE_URL=postgresql://ml_user:ml_password@postgres:5432/candle_annotator

# Authentication
AUTH_SECRET=your_strong_random_secret_here
AUTH_TRUST_HOST=false
AUTH_GOOGLE_ID=your_google_oauth_client_id
AUTH_GOOGLE_SECRET=your_google_oauth_client_secret

# Default admin user
DEFAULT_ADMIN_EMAIL=admin@example.com
DEFAULT_ADMIN_PASSWORD=your_strong_password_here

# API Key
API_KEY=your_strong_random_api_key_here

# ML Inference (optional)
INFERENCE_API_URL=http://ml-service:8001
INFERENCE_API_TIMEOUT=30000
INFERENCE_BATCH_TIMEOUT=120000
NEXT_PUBLIC_PREDICTIONS_ENABLED=true

Pass environment variables to docker-compose:

docker-compose --env-file .env up -d

After containers are running, the user migration will run automatically during startup.

Data Persistence

The application stores all data in PostgreSQL using the Docker named volume postgres-data. This ensures data persists across container restarts:

# View volumes
docker volume ls | grep postgres

# Backup database
docker exec candle_annotator-postgres-1 pg_dump -U ml_user candle_annotator > backup.sql

# Restore database
cat backup.sql | docker exec -i candle_annotator-postgres-1 psql -U ml_user -d candle_annotator

Data Migration from SQLite

If you're upgrading from a SQLite-based version, you need to migrate your data:

1. Before upgrading, backup your SQLite database:

   docker cp candle_annotator-candle-annotator-1:/app/data/candles.db ./backup-sqlite.db
   

2. Stop the old containers:

   docker compose down
   

3. Pull the new version and start services:

   git pull origin master
   docker compose up -d
   

4. Run the migration script from your host machine:

   # Copy SQLite database to a location accessible to the script
   cp backup-sqlite.db data/candles.db
   
   # Run migration (requires ts-node and dependencies)
   npm install
   DATABASE_URL=postgresql://ml_user:ml_password@localhost:5432/candle_annotator \
     npx ts-node scripts/migrate-sqlite-to-postgres.ts
   

Rollback Procedure (if migration fails):

1. Stop new containers:

   docker compose down
   

2. Restore SQLite-based docker-compose.yml from git history:

   git checkout HEAD~1 docker-compose.yml Dockerfile
   

3. Restore SQLite database:

   mkdir -p data
   cp backup-sqlite.db data/candles.db
   

4. Start old version:

   docker compose up -d
   

Accessing the Application

Once running, access the application at:

http://localhost:3000

Health check endpoint:

curl http://localhost:3000/api/health

With database check:

curl http://localhost:3000/api/health?check=db

Port Mapping

To run on a different port (e.g., 8080), modify docker-compose.yml:

services:
  candle-annotator:
    ports:
      - "8080:3000"

Or use environment variable in docker-compose:

services:
  candle-annotator:
    ports:
      - "${HOST_PORT:-3000}:3000"

Then run:

HOST_PORT=8080 docker-compose up -d

Container Health Checks

Docker automatically checks container health every 30 seconds using the /api/health endpoint. The container will restart if:

• Health check fails 3 times consecutively
• Takes longer than 3 seconds to respond

View health status:

docker ps

Look for the STATUS column - it should show healthy.

Troubleshooting

Port already in use:

docker-compose down  # Stop any existing containers
docker-compose up -d -p 8080:3000/tcp

Database connection errors:

# Check PostgreSQL logs
docker compose logs postgres

# Verify database exists
docker exec -it candle_annotator-postgres-1 psql -U ml_user -d candle_annotator -c "\dt"

# Recreate database if needed
docker compose down
docker volume rm candle_annotator_postgres-data
docker compose up --build

Rebuild without cache:

docker-compose build --no-cache
docker-compose up -d

View container logs:

docker-compose logs -f --tail=100

ML service healthcheck failing:

If the candle-annotator service fails to start with error "dependency failed to start: container candle_annotator-ml-service-1 is unhealthy", this is because the ml-service healthcheck requires curl to be installed in the container. This was fixed in commit ecb2385 by adding curl to the ml-service Dockerfile.

If you encounter this issue:

1. Rebuild the ml-service: docker compose build ml-service
2. Restart services: docker compose up -d

Migration errors during startup:

If you see Drizzle migration errors during container startup, check:

1. Ensure PostgreSQL is fully started and healthy:

   docker compose ps postgres
   

2. Check migration logs:

   docker compose logs candle-annotator | grep -i migration
   

3. If needed, run migrations manually:

   docker exec -it candle_annotator-candle-annotator-1 npm run db:migrate
   

Update Procedure

To update the application:

git pull origin master
docker-compose down
docker-compose up --build -d

Or with no-cache rebuild:

git pull
docker-compose down
docker-compose build --no-cache
docker-compose up -d

Production Deployment

For production deployments, consider:

1. Use a container registry (Docker Hub, ECR, GCR):

   docker tag candle-annotator myregistry/candle-annotator:v1.0.0
   docker push myregistry/candle-annotator:v1.0.0
   

2. Run on a remote server (AWS, DigitalOcean, etc.):

   # SSH into server, clone repo, then:
   docker-compose up -d
   

3. Add reverse proxy (nginx, traefik) for HTTPS:

   # docker-compose.yml
   services:
     nginx:
       image: nginx:alpine
       ports:
         - "443:443"
       volumes:
         - ./nginx.conf:/etc/nginx/nginx.conf
   

4. Enable Docker logging for production monitoring:

   docker-compose logs -f --tail=1000 > app.log &
   

Notes

• The application now includes user authentication and multi-user support
• Users can sign up with email/password or use Google OAuth
• A default admin user is created during the migration phase (see Database Migrations section)
• Each user's data (charts, annotations, etc.) is isolated and scoped by user_id
• PostgreSQL is used for all application data (frontend and ML service)
• The shared database enables the ML service to directly query candle and annotation data with user scoping
• Docker deployment provides lightweight containerization ideal for standalone instances
• The multi-stage Dockerfile keeps image size minimal (~100MB)
• Auth.js v5 is used for authentication with JWT strategy and 30-day session expiry