- Add build_dataset_from_db() that exports candles from DB, runs feature
engineering, and ingests span annotations into labeled CSV
- Call it automatically in _run_training_background before training starts
- Add POST /training/build-dataset endpoint for standalone use
- Add Next.js proxy route /api/training/build-dataset
- Update TrainingPanel: remove dataset-missing block on Start Training,
show informational message that dataset builds automatically
- Convert numpy.int64 to Python int before passing to SQLAlchemy queries
- Prevents psycopg2.ProgrammingError: can't adapt type 'numpy.int64'
- Applied to get_candles(), get_span_annotations(), and get_point_annotations()
- All ML service database access tests now passing successfully
- Created scripts/migrate-sqlite-to-postgres.py as alternative to TypeScript version
- Handles all type conversions: timestamps, booleans, and JSONB fields
- Successfully migrated all 2,836 rows from SQLite to PostgreSQL
- Verified data integrity: all 6 tables migrated correctly
- Charts: 1, Candles: 2,592, Annotations: 4, Span annotations: 223
The model info returned empty labels array because the pkl file has
no metadata dict. Now extracts labels from model.classes_ or
model.model.classes_ as fallback.
The model was trained on 94-candle sliding windows flattened to 2820
features (94 candles x 30 features). Inference was sending raw per-candle
features (27 columns).
Changes:
- Rewrite preprocessing to return (X, window_times) tuple
- Add sliding window creation with correct feature ordering
- Fill missing columns (average, barCount) with 0 for feature parity
- Fill NaN from indicator warmup with 0 instead of dropping rows
- Always compute all indicators (including MFI) for feature parity
- Update predict and batch predict endpoints for new signature
The Pydantic model sets volume=None when absent, creating an all-NaN
column rather than a missing column. Check isna().all() in addition
to column existence.
- Fill volume with 0 when column is absent from candle data
- Skip MFI/OBV/AD/ADOSC indicators when no real volume data available
- Fix pandas FutureWarning for inplace fillna in candle_features
- Remove temporary debug NaN logging
- Add GET handler to /api/charts/[id] route to fetch chart metadata
- Fix batch prediction to use regular /predict endpoint with database candles
- Remove /predict/batch usage (was designed for file-based predictions)
- Make volume field optional in CandleData model (database candles don't have volume)
- Convert timestamps to ISO dates for batch requests
Known issue: TA-Lib indicators failing with 'input array type is not double'
- May need to ensure candle data is float64/double type before processing
- Change pair and timeframe fields from required to optional
- Frontend only sends candles array, not pair/timeframe metadata
- These fields are only used for logging, not prediction logic
- Update logging to handle None values with 'unknown' fallback
- Fixes 422 validation error on /predict endpoint
- Fix CCI indicator to use HLC prices instead of close only
- Parse datetime column when loading enriched CSV
- Strip timezone from annotation timestamps
- Fix TA-Lib pattern names (CDL3WHITESOLDIERS, CDL3BLACKCROWS)
- Exclude programmatic label columns from training features
- Fix classification report to handle missing classes
- Update MLflow tracking to use localhost:5000
- Grant PostgreSQL permissions to ml_user
Pipeline now runs successfully end-to-end:
- Feature engineering: 2543 rows, 31 columns
- Annotation ingestion: 286 samples
- Training: 89.47% test accuracy with Random Forest
Add complete workflow for using TA-Lib to bootstrap training data:
- generate_talib_annotations.py: Python script to run TA-Lib CDL* functions
and output span annotations in UI-compatible format
- import_talib_annotations.ts: TypeScript script to import generated
annotations into the UI database with auto-label-type creation
- npm script 'import-annotations' for easy execution
- TALIB_WORKFLOW.md: Comprehensive guide covering the full cycle:
* Generate patterns with TA-Lib
* Import into UI
* Review and edit in browser
* Export and train model
* Compare predictions with TA-Lib detections
* Iterate for improvement
This enables the intended workflow: use TA-Lib for initial annotations,
manually refine them, then train a model that learns from corrections.
