# #sentiment # #Jake Bradley Bragan Culberson # #3/21/2026 1:43 AM """ ML Stock Trading System v9 — Alpaca WebSocket + Finnhub Multi-Source Edition ============================================================================= KEY IMPROVEMENTS OVER v6: BUG FIXES: - DB freeze fixed: dedup cache (5s buckets) replaces UNIQUE constraint so valid quotes are never silently dropped. - Training stall fixed: first train triggers as soon as 5+ symbols have enough data, not gated behind a 100-cycle counter. - Clone agents each get their own independent StockModel instance. - Quote polling is strict round-robin (not random weighted). - MetaLearner trains unconditionally at every retrain cycle. - Agent→agent knowledge sharing still requires profit (donor must be +ve). - Clone min_conf hard-capped at 0.62 after param nudging. DATA SOURCES: - Alpaca WebSocket (FREE): minute bars for ALL IEX symbols (~8,000-9,000). Subscribed with wildcard "*" — zero polling calls used. Pushes a bar every 60s per symbol automatically. No rate limit, no daily cap, 0 REST calls consumed. - Finnhub REST (FREE, 58/min): live quotes for your active 300-stock universe via strict round-robin. Now supplemental to Alpaca — used for sub-minute price freshness on symbols agents are holding or watching. - Alpha Vantage (FREE, 25/day): 5-min intraday bars, supplemental. - Polygon (FREE, 5/min): prev-day minute bars at market open. VELOCITY FEATURE (NEW): - Every bar arrival from Alpaca records the time since the last bar. - price_velocity = (close_now - close_prev) / seconds_elapsed - Added as "price_velocity" to FEATURE_COLS so models learn from it. DATA COVERAGE: - Alpaca free IEX wildcard: ~8,000-9,000 unique symbols per minute. - Finnhub 58/min round-robin: your 300-stock universe covered every ~5 min. - Combined: all 300 core stocks get sub-minute Alpaca bars PLUS Finnhub confirmation quotes, giving the densest free live dataset possible. BEST STOCK RANGE FOR PROFIT (see STOCK_UNIVERSE_PRIMARY comments): Sweet spot is $5-$200 mid-cap liquid stocks with tight spreads. Mega-caps (AAPL, MSFT) are too efficient — thin margins for ML edge. Penny stocks (<$5) are too volatile and often illiquid on IEX. Best performers empirically: $20-$80 range, daily volume >500k, sectors: Tech, Financials, Healthcare, Energy. ETFs excluded from trading. THOUGHT PROCESS / VISIBILITY: - All BUY/SELL decisions logged with full reasoning chain: own_conf, meta_conf, effective_conf, threshold, decision. - Hourly report includes WIN/LOSS summary: gross profit, gross loss, total P&L, win streaks, per-agent breakdown. - TRADE_LOG.csv written in real-time for external Excel/pandas analysis. - MetaLearner logs which symbols it's boosting and why. TRAINING: - Each agent has slightly different GBM hyperparams so models diverge. - Training is ONLINE: only live quotes collected during market hours. - Retrain every 15 min at :00/:15/:30/:45 during market hours. - Agents scale to 3x CPU cores, capped at 48 total. Install: pip install finnhub-python scikit-learn numpy pandas joblib pytz \ matplotlib requests websocket-client Usage: export FINNHUB_API_KEY=your_key_here export ALPACA_API_KEY=your_alpaca_key # free at alpaca.markets export ALPACA_SECRET_KEY=your_alpaca_secret export ALPHA_VANTAGE_KEY=your_key_here # optional export POLYGON_KEY=your_key_here # optional python StockTrading_v7.py """ import os, time, sqlite3, logging, threading, random, math, json, copy, csv import hashlib, xml.etree.ElementTree as ET from datetime import datetime, timedelta from collections import deque, defaultdict import multiprocessing import numpy as np import pandas as pd import finnhub import joblib import pytz import requests import matplotlib matplotlib.use("Agg") import matplotlib.pyplot as plt import matplotlib.gridspec as gridspec try: import websocket _WS_AVAILABLE = True except ImportError: _WS_AVAILABLE = False print("[WARNING] websocket-client not installed. Run: pip install websocket-client") print(" Alpaca stream disabled — falling back to Finnhub only.") try: from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer VADER_OK = True except ImportError: class SentimentIntensityAnalyzer: def polarity_scores(self, text): return {"compound": 0.0} VADER_OK = False try: import feedparser FEEDPARSER_OK = True except ImportError: feedparser = None FEEDPARSER_OK = False try: import ta TA_OK = True except ImportError: TA_OK = False # v9: LightGBM -- releases GIL during C++ predict, 5-10x faster, supports init_model try: import lightgbm as lgb LGB_OK = True except ImportError: lgb = None LGB_OK = False print("[WARNING] lightgbm not installed. Run: pip install lightgbm") # v9: uvloop -- 2x asyncio throughput try: import uvloop UVLOOP_OK = True except ImportError: uvloop = None UVLOOP_OK = False # v9: ThreadPoolExecutor for concurrent LightGBM inference (GIL released) from concurrent.futures import ThreadPoolExecutor, as_completed from sklearn.ensemble import GradientBoostingClassifier, GradientBoostingRegressor from sklearn.preprocessing import StandardScaler from sklearn.model_selection import train_test_split from sklearn.metrics import accuracy_score # ============================================================================= # AUTO-DETECT CORES # ============================================================================= PHYSICAL_CORES = multiprocessing.cpu_count() BASE_AGENTS = 50 CLONE_AGENTS = 50 NUM_AGENTS = BASE_AGENTS + CLONE_AGENTS # 100 # ============================================================================= # CONFIGURATION # ============================================================================= FINNHUB_API_KEY = os.environ.get("FINNHUB_API_KEY", "d6c79h9r01qsiik11kt0d6c79h9r01qsiik11ktg") ALPACA_API_KEY = os.environ.get("ALPACA_API_KEY", "PKLNUNIF76AZQBULGIKCRYLKQ2") ALPACA_SECRET_KEY = os.environ.get("ALPACA_SECRET_KEY", "FNqvcb93Vb7C5U1t6vuafNkjR61zhxqkPq7as8qYnSm8") ALPHA_VANTAGE_KEY = os.environ.get("ALPHA_VANTAGE_KEY", "CKDWGTR0PKRWHEBU") POLYGON_KEY = os.environ.get("POLYGON_KEY", "") STARTING_CAPITAL = 1_000.0 DRAWDOWN_RESET_PCT = 0.20 DEFAULT_STOP_LOSS_PCT = 0.07 DEFAULT_TAKE_PROFIT_PCT= 0.15 DEFAULT_MAX_POSITIONS = 5 SHORT_STOP_LOSS_PCT = 0.05 # cover short if price rises 5% against us SHORT_TAKE_PROFIT_PCT = 0.10 # cover short if price falls 10% SHORT_CAPITAL_RATIO = 0.50 # short positions use 50% of normal sizing BUY_COOLDOWN_SECS = 5 STOP_LOSS_RANGE = (0.02, 0.20) TAKE_PROFIT_RANGE = (0.03, 0.60) MAX_POSITIONS_RANGE = (1, 10) MIN_CONF_RANGE = (0.50, 0.90) MAX_POS_PCT_RANGE = (0.02, 0.40) PARAM_LEARNING_RATE = 0.15 # ============================================================================= # DAILY DATABASE SYSTEM # ============================================================================= # Each calendar day gets its own SQLite file: trading_data_YYYYMMDD.db # WRITES: always go to today's DB (small, fast, no fragmentation) # READS: span all daily DBs combined (full historical training data) # Old DBs are auto-pruned after DB_HISTORY_DAYS days. # # Benefits over a single growing DB: # - Today's inserts are always fast (small file, warm cache) # - Training sees all historical data via combined queries # - Easy to inspect or archive individual days # - DB corruption on any one day doesn't lose all history DB_FILE_PREFIX = "data/db/trading_data" DB_HISTORY_DAYS = 90 def _today_db_path() -> str: """Return the DB filename for today's date.""" return f"{DB_FILE_PREFIX}_{datetime.now().strftime('%Y%m%d')}.db" def _all_db_paths() -> list: """Return all daily DB paths sorted oldest-first. Auto-deletes files older than DB_HISTORY_DAYS.""" import glob cutoff = (datetime.now() - timedelta(days=DB_HISTORY_DAYS)).strftime("%Y%m%d") paths = [] for p in sorted(glob.glob(f"{DB_FILE_PREFIX}_????????.db")): date_str = p.replace(f"{DB_FILE_PREFIX}_", "").replace(".db", "") if date_str >= cutoff: paths.append(p) else: try: for suffix in ["", "-wal", "-shm"]: if os.path.exists(p + suffix): os.remove(p + suffix) except Exception: pass today = _today_db_path() if today not in paths: paths.append(today) return paths BACKUP_DIR = "backup" # v9: Lifetime learning -- permanent feature file storage FEATURES_DAILY_DIR = "features/daily" # one file per trading day, kept forever FEATURES_WEEKLY_DIR = "features/weekly" # one file per week (Mon-Fri combined) DAILY_TRAIN_TREES = 50 # trees added per EOD incremental train WEEKLY_TRAIN_TREES = 100 # trees added per weekend deep train DISTILL_THRESHOLD = 2500 # tree count that triggers monthly distillation DISTILL_BASE_TREES = 300 # fresh trees after distillation (compact base) SENTIMENT_DB_FILE = "data/db/sentiment_data.db" # separate DB for sentiment scores # --------------------------------------------------------------------------- # SENTIMENT PIPELINE CONFIGURATION # --------------------------------------------------------------------------- # How often each sentiment source is polled (seconds). # These are deliberately spaced to avoid hammering free APIs. SENTIMENT_NEWS_INTERVAL = 1800 # Google News RSS: every 30 min per ticker SENTIMENT_SEC_INTERVAL = 3600 # SEC EDGAR 8-K feed: every 60 min SENTIMENT_REDDIT_INTERVAL = 900 # Reddit mention scan: every 15 min SENTIMENT_CONGRESS_INTERVAL= 86400 # ProPublica Congress trades: once per day # How far back to look when scoring sentiment (seconds) SENTIMENT_LOOKBACK_NEWS = 86400 # 24 hours of news SENTIMENT_LOOKBACK_REDDIT = 3600 # 1 hour of Reddit mentions SENTIMENT_LOOKBACK_SEC = 172800 # 48 hours for SEC filings # Minimum articles/mentions needed before a sentiment score is trusted SENTIMENT_MIN_ARTICLES = 2 SENTIMENT_MIN_MENTIONS = 3 # Source credibility weights — higher = more trusted SENTIMENT_WEIGHT_NEWS = 1.0 # Google News (aggregates many sources) SENTIMENT_WEIGHT_SEC = 2.0 # SEC filing = highest weight (official) SENTIMENT_WEIGHT_REDDIT = 0.4 # Reddit = noisy but early signal SENTIMENT_WEIGHT_CONGRESS = 1.5 # Congress trades = historically predictive # Reddit app credentials (free at reddit.com/prefs/apps) # Leave blank to disable Reddit sentiment REDDIT_CLIENT_ID = os.environ.get("REDDIT_CLIENT_ID", "") REDDIT_CLIENT_SECRET = os.environ.get("REDDIT_CLIENT_SECRET", "") REDDIT_USER_AGENT = "StockTradingBot/9.0" # StockTwits: free financial social sentiment, no approval needed. # Optional token at stocktwits.com/developers doubles the rate limit. STOCKTWITS_TOKEN = os.environ.get("STOCKTWITS_TOKEN", "") SENTIMENT_STOCKTWITS_INTERVAL = 600 # poll each symbol every 10 minutes # Subreddits to scan for stock mentions SENTIMENT_SUBREDDITS = ["wallstreetbets", "stocks", "investing", "StockMarket"] # SEC EDGAR base URLs (free, no key needed) SEC_EDGAR_RSS = "https://efts.sec.gov/LATEST/search-index?q=%22{symbol}%22&dateRange=custom&startdt={start}&enddt={end}&forms=8-K" SEC_EDGAR_SEARCH = "https://efts.sec.gov/LATEST/search-index?q=%22{symbol}%22&forms=8-K&hits.hits._source=period_of_report" # ProPublica Congress API (free, no key needed) PROPUBLICA_TRADES = "https://financialmodelingprep.com/api/v4/senate-disclosure?page=0&apikey=demo" # Google News RSS template GOOGLE_NEWS_RSS = "https://news.google.com/rss/search?q={symbol}+stock&hl=en-US&gl=US&ceid=US:en" # 8-K keywords that indicate negative events (triggers sell-side alert) SEC_NEGATIVE_KEYWORDS = [ "restatement", "resignation", "termination", "default", "bankruptcy", "fraud", "investigation", "lawsuit", "delisting", "going concern", "material weakness", "cybersecurity incident", "data breach", "regulatory action", "sec investigation" ] # 8-K keywords that indicate positive events SEC_POSITIVE_KEYWORDS = [ "acquisition", "merger", "strategic partnership", "new contract", "buyback", "dividend increase", "guidance raised", "record revenue", "fda approval", "patent granted", "expansion" ] DATA_LOG_FILE = "logs/trading_data.log" REPORT_LOG_FILE = "logs/hourly_report.log" TRADE_LOG_FILE = "logs/TRADE_LOG.csv" TRAIN_CHECKPOINT_FILE = "logs/train_checkpoint.json" # v9: round-robin resume TRAIN_DONE_FILE = "logs/train_done.json" # v9: one-per-day guard # v9: RAM cap global -- None = uncapped (max efficiency), int = MB ceiling # Set at startup from --ram-cap arg. Read by ResourceGuard + feature rebuild. RAM_CAP_MB = None # int (MB) or None -- set from --ram-cap arg at startup MAX_WORKERS_CAP = None # int or None -- set from --max-workers arg at startup DEBUG_LOG_FILE = "logs/debug/debug.log" # detailed crash/thread diagnostics MODEL_SAVE_DIR = "models" AGENT_STATE_FILE = "logs/agent_state.json" CHARTS_DIR = "charts" MIN_ROWS_TO_TRAIN = 50 IC_MIN_VOTE_THRESHOLD = 0.015 # agents below this IC are excluded from trade voting MIN_ROWS_TO_TRADE = 60 MARKET_TZ = pytz.timezone("America/New_York") DISPLAY_TZ = pytz.timezone("America/Chicago") # CST — display only # Retrain every 15 min during market hours RETRAIN_MINUTES = [0, 15, 30, 45] # Finnhub rate: use 59 of 60 free calls/min FH_CALLS_PER_MIN = 59 # Alpaca IEX WebSocket URL (free tier) ALPACA_WS_URL = "wss://stream.data.alpaca.markets/v2/iex" # Ensure all required directories exist at startup for _d in [MODEL_SAVE_DIR, CHARTS_DIR, BACKUP_DIR, "data", "logs", "logs/debug", "ops", FEATURES_DAILY_DIR, FEATURES_WEEKLY_DIR]: os.makedirs(_d, exist_ok=True) os.makedirs(CHARTS_DIR, exist_ok=True) os.makedirs("data/db", exist_ok=True) # trading DBs + sentiment os.makedirs("logs", exist_ok=True) # all log files os.makedirs(BACKUP_DIR, exist_ok=True) # Limit OpenMP threads per job to prevent kernel SIGKILL. # sklearn GBM uses OpenMP internally — without this, each joblib job # spawns as many OpenMP threads as there are CPU cores, causing # 4 jobs x 16 OpenMP threads = 64 threads total -> kernel kills process. os.environ.setdefault("OMP_NUM_THREADS", "4") os.environ.setdefault("OPENBLAS_NUM_THREADS", "4") os.environ.setdefault("MKL_NUM_THREADS", "4") # ============================================================================= # BACKUP # ============================================================================= def run_backup(): """ Local backup of important files to backup/ WAL FIX: The trading_data DB WAL file grew to 2.9 GB. Copying 2.9 GB in a background thread saturated disk I/O for 20-30 seconds, delaying all threads including the WebSocket ping thread. Now: WAL files > WAL_BACKUP_CAP_MB are skipped -- they will be checkpointed at the next PRAGMA wal_checkpoint call anyway. """ _set_nice(19) # P4: lowest OS scheduling priority _resource_guard.yield_p4("local_backup_start") import shutil WAL_BACKUP_CAP_MB = 100 # skip WAL files larger than this try: ts = datetime.now(DISPLAY_TZ).strftime("%Y%m%d_%H%M%S") t_start = time.time() bytes_copied = 0 # Checkpoint WAL files before backup so they are as small as possible import glob as _glob for _db in _glob.glob(f"{DB_FILE_PREFIX}_????????.db"): try: with sqlite3.connect(_db, timeout=3) as _c: _c.execute("PRAGMA wal_checkpoint(PASSIVE)") debug_logger.debug(f"BACKUP_WAL_CHECKPOINT | {_db}") except Exception as _e: debug_logger.warning(f"BACKUP_WAL_CKPT_FAIL | {_db} | {_e}") # Single log files # Ensure backup subfolders exist (mirrors data/, logs/ layout) os.makedirs(os.path.join(BACKUP_DIR, "data"), exist_ok=True) os.makedirs(os.path.join(BACKUP_DIR, "logs"), exist_ok=True) os.makedirs(os.path.join(BACKUP_DIR, "logs", "debug"), exist_ok=True) # Clean ghost files from old backup paths (pre-reorganization artifacts) _ghost_files = [ os.path.join(BACKUP_DIR, "agent_state.json"), os.path.join(BACKUP_DIR, "debug.log"), os.path.join(BACKUP_DIR, "hourly_report.log"), os.path.join(BACKUP_DIR, "TRADE_LOG.csv"), os.path.join(BACKUP_DIR, "trading_data.log"), os.path.join(BACKUP_DIR, "sentiment_data.db"), os.path.join(BACKUP_DIR, "trading_data.db"), os.path.join(BACKUP_DIR, "trading_data.db-shm"), os.path.join(BACKUP_DIR, "trading_data.db-wal"), ] for _gf in _ghost_files: if os.path.exists(_gf): os.remove(_gf) debug_logger.info(f"BACKUP_GHOST_CLEAN | removed {_gf}") for fname in [ "logs/agent_state.json", "logs/hourly_report.log", "logs/TRADE_LOG.csv", "logs/trading_data.log", "logs/human_report.txt", "logs/claude_handoff.json", "logs/bot_dashboard.html","logs/train_done.json", SENTIMENT_DB_FILE, DATA_LOG_FILE, REPORT_LOG_FILE, DEBUG_LOG_FILE, ]: if os.path.exists(fname): t0 = time.time() # Preserve subfolder structure inside backup/ dest_dir = os.path.join(BACKUP_DIR, os.path.dirname(fname)) os.makedirs(dest_dir, exist_ok=True) shutil.copy2(fname, os.path.join(BACKUP_DIR, fname)) sz = os.path.getsize(fname) bytes_copied += sz elapsed = int((time.time() - t0) * 1000) if elapsed > 500: debug_logger.warning(f"BACKUP_SLOW_FILE | {fname} | {sz//1024}KB | {elapsed}ms") # Daily DB files -- skip oversized WAL files for _db in _glob.glob(f"{DB_FILE_PREFIX}_????????.db"): for _sfx in ["", "-wal", "-shm"]: _src = _db + _sfx if not os.path.exists(_src): continue sz = os.path.getsize(_src) if _sfx == "-wal" and sz > WAL_BACKUP_CAP_MB * 1024 * 1024: debug_logger.warning( f"BACKUP_WAL_SKIPPED | {_src} | " f"{sz//1024//1024}MB > {WAL_BACKUP_CAP_MB}MB cap -- " f"too large to copy safely during trading" ) continue t0 = time.time() shutil.copy2(_src, os.path.join(BACKUP_DIR, os.path.basename(_src))) bytes_copied += sz elapsed = int((time.time() - t0) * 1000) if elapsed > 1000: debug_logger.warning(f"BACKUP_SLOW_DB | {_src} | {sz//1024//1024}MB | {elapsed}ms") if os.path.exists("trading.log"): os.makedirs(os.path.join(BACKUP_DIR, "logs"), exist_ok=True) shutil.copy2("trading.log", os.path.join(BACKUP_DIR, "logs", "trading.log")) # Charts folder — copy current charts, then prune to last 10 per type backup_charts = os.path.join(BACKUP_DIR, "charts") os.makedirs(backup_charts, exist_ok=True) for f in os.listdir(CHARTS_DIR): shutil.copy2(os.path.join(CHARTS_DIR, f), os.path.join(backup_charts, f)) # Prune old charts: keep last 10 per chart type (accuracy / overview / resources) try: _chart_by_type: dict = {} for _cf in os.listdir(backup_charts): _ctype = _cf.split("_")[0] # e.g. "accuracy", "overview", "resources" _chart_by_type.setdefault(_ctype, []).append(_cf) for _ctype, _cfiles in _chart_by_type.items(): _sorted = sorted(_cfiles, reverse=True) # newest first (datetime in name) for _old_chart in _sorted[10:]: # delete beyond 10 os.remove(os.path.join(backup_charts, _old_chart)) except Exception as _ce3: debug_logger.warning(f"CHART_PRUNE_ERR | {_ce3}") # Saved models folder — only copy if any model was recently trained (within 3h) # Models change only after EOD or weekend train. Skipping otherwise saves 103MB I/O. backup_models = os.path.join(BACKUP_DIR, "saved_models") os.makedirs(backup_models, exist_ok=True) _model_cutoff = time.time() - 3 * 3600 # 3 hours _any_model_recent = any( os.path.getmtime(os.path.join(MODEL_SAVE_DIR, f)) > _model_cutoff for f in os.listdir(MODEL_SAVE_DIR) if f.endswith(".pkl") ) if os.path.isdir(MODEL_SAVE_DIR) else False if _any_model_recent: for f in os.listdir(MODEL_SAVE_DIR): shutil.copy2(os.path.join(MODEL_SAVE_DIR, f), os.path.join(backup_models, f)) debug_logger.info("BACKUP_MODELS | copied (recently trained)") else: debug_logger.debug("BACKUP_MODELS | skipped (no recent training)") bytes_copied -= 0 # no change, just note # Cleanup: remove debug session logs older than 30 days try: _debug_dir = os.path.join("logs", "debug") if os.path.isdir(_debug_dir): _cutoff = time.time() - 30 * 86400 for _f in os.listdir(_debug_dir): if _f.startswith("debug_") and _f.endswith(".log"): _fp = os.path.join(_debug_dir, _f) if os.path.getmtime(_fp) < _cutoff: os.remove(_fp) except Exception as _ce: debug_logger.warning(f"DEBUG_LOG_CLEANUP_ERR | {_ce}") # Cleanup: remove backup DB copies older than 7 days # Also catches legacy trading_data.db (no date pattern) try: import glob as _g2 _db_cutoff = time.time() - 7 * 86400 _db_patterns = [ os.path.join(BACKUP_DIR, "trading_data_????????.db"), # dated os.path.join(BACKUP_DIR, "trading_data.db"), # legacy (no date) ] for _pat in _db_patterns: for _old in _g2.glob(_pat): if os.path.getmtime(_old) < _db_cutoff: for _sfx in ["", "-wal", "-shm"]: _p = _old + _sfx if os.path.exists(_p): os.remove(_p) except Exception as _ce2: debug_logger.warning(f"BACKUP_DB_CLEANUP_ERR | {_ce2}") # Cleanup: feature file retention # Daily features: keep 14 days (EOD only reads today's file) # Weekly features: keep 26 weeks (6 months of weekly context) try: _feat_daily_cutoff = time.time() - 14 * 86400 _feat_weekly_cutoff = time.time() - 26 * 7 * 86400 for _fdir, _cutoff, _label in [ (FEATURES_DAILY_DIR, _feat_daily_cutoff, "DAILY_FEAT"), (FEATURES_WEEKLY_DIR, _feat_weekly_cutoff, "WEEKLY_FEAT"), ]: if not os.path.isdir(_fdir): continue for _ff in os.listdir(_fdir): _fp = os.path.join(_fdir, _ff) if os.path.getmtime(_fp) < _cutoff: os.remove(_fp) debug_logger.info(f"FEAT_RETENTION_CLEAN | {_label} | removed {_ff}") except Exception as _fe: debug_logger.warning(f"FEAT_RETENTION_ERR | {_fe}") # Cleanup: orphaned -shm/-wal files in data/db/ with no matching main DB try: for _sfx in ["-shm", "-wal"]: for _side in _g2.glob(f"{DB_FILE_PREFIX}_????????.db{_sfx}"): _main = _side[:-len(_sfx)] if not os.path.exists(_main): os.remove(_side) debug_logger.info(f"ORPHAN_SIDECAR_CLEAN | removed {_side}") except Exception as _oe: debug_logger.warning(f"ORPHAN_SIDECAR_ERR | {_oe}") total_ms = int((time.time() - t_start) * 1000) logger.info(f"Backup complete -> {BACKUP_DIR}/ ({ts} CST)") debug_logger.info( f"BACKUP_COMPLETE | ts={ts} | " f"bytes={bytes_copied//1024//1024}MB | elapsed={total_ms}ms" ) if total_ms > 5000: debug_logger.warning( f"BACKUP_SLOW_TOTAL | {total_ms}ms -- " f"disk I/O pressure may delay WebSocket thread!" ) except Exception as e: logger.warning(f"Backup error: {e}") debug_logger.error(f"BACKUP_ERROR | {e}", exc_info=True) # ============================================================================= # LOGGING # ============================================================================= # ============================================================================= # CST LOG FORMATTER # ============================================================================= # All log timestamps display in CST (America/Chicago) for readability. # This ONLY affects what you see in log files -- all internal market timing # (is_market_open, trade decisions, retrain schedule) still uses ET as before. # Overriding formatTime() is the clean way to inject a display timezone # without touching any other part of the program. class _CSTFormatter(logging.Formatter): """Log formatter that outputs timestamps in CST (America/Chicago).""" _CST = pytz.timezone("America/Chicago") def formatTime(self, record, datefmt=None): dt = datetime.fromtimestamp(record.created, tz=self._CST) return dt.strftime(datefmt or "%Y-%m-%d %H:%M:%S") class _CSTFormatterMs(_CSTFormatter): """CST formatter with milliseconds for debug.log.""" def formatTime(self, record, datefmt=None): dt = datetime.fromtimestamp(record.created, tz=self._CST) base = dt.strftime("%Y-%m-%d %H:%M:%S") return f"{base}.{int(record.msecs):03d}" _main_fmt = _CSTFormatter( "%(asctime)s [%(levelname)s] %(message)s", datefmt="%Y-%m-%d %H:%M:%S") _debug_fmt = _CSTFormatterMs( "%(asctime)s [%(levelname)s] [%(threadName)s] %(message)s", datefmt="%Y-%m-%d %H:%M:%S") # Main logger (trading.log + console) -- CST timestamps logging.basicConfig(level=logging.INFO, handlers=[]) logger = logging.getLogger(__name__) logger.setLevel(logging.INFO) _fh_main = logging.FileHandler(DATA_LOG_FILE) _fh_main.setFormatter(_main_fmt) _sh_main = logging.StreamHandler() _sh_main.setFormatter(_main_fmt) logger.addHandler(_fh_main) logger.addHandler(_sh_main) logger.propagate = False # Report logger (hourly_report.log) -- CST timestamps report_logger = logging.getLogger("report") report_logger.setLevel(logging.INFO) _rh = logging.FileHandler(REPORT_LOG_FILE) _rh.setFormatter(_main_fmt) report_logger.addHandler(_rh) report_logger.propagate = False # Debug logger -- per-session dated file so restarts never lose log context # Also maintains a "debug.log" symlink -> latest session file _debug_session_file = DEBUG_LOG_FILE.replace( ".log", f"_{datetime.now().strftime('%Y%m%d_%H%M%S')}.log") debug_logger = logging.getLogger("debug") debug_logger.setLevel(logging.DEBUG) debug_logger.handlers.clear() # Truncate fresh on each restart -- always current session only, no stale content _dh = logging.FileHandler(DEBUG_LOG_FILE, mode="w") _dh.setFormatter(_debug_fmt) debug_logger.addHandler(_dh) debug_logger.propagate = False _dh2 = logging.FileHandler(_debug_session_file, mode="w") _dh2.setFormatter(_debug_fmt) debug_logger.addHandler(_dh2) # ============================================================================= # TRADE LOG # ============================================================================= _trade_log_lock = threading.Lock() def _init_trade_log(): if not os.path.exists(TRADE_LOG_FILE): with open(TRADE_LOG_FILE, "w", newline="") as f: w = csv.writer(f) w.writerow([ "timestamp","agent","type","symbol", "shares","entry_price","exit_price", "pnl","pnl_pct","predicted_pct", "disc_pp","reason","held_secs", "own_conf","meta_conf","eff_conf", ]) def write_trade_log(row: dict): try: with _trade_log_lock: with open(TRADE_LOG_FILE, "a", newline="") as f: w = csv.writer(f) w.writerow([ row.get("timestamp",""), row.get("agent",""), row.get("type","SELL"), row.get("symbol",""), f"{row.get('shares',0):.4f}", f"{row.get('entry',0):.4f}", f"{row.get('exit',0):.4f}", f"{row.get('pnl',0):+.4f}", f"{row.get('pnl_pct',0):+.2f}", f"{row.get('predicted_pct',0):+.2f}", f"{row.get('disc_pp',0):.3f}", row.get("reason",""), row.get("held_secs",0), f"{row.get('own_conf',0):.3f}", f"{row.get('meta_conf',0):.3f}", f"{row.get('eff_conf',0):.3f}", ]) except Exception as _wtle: logger.error(f"TRADE_LOG_WRITE_ERR | {_wtle} | file={TRADE_LOG_FILE}") # ============================================================================= # STOCK UNIVERSE # Sweet spot for profit: $20-$80, daily volume >500k, tight spreads. # Mega-caps (AAPL/MSFT) are too efficient for ML edge. # Penny stocks (<$5) are too volatile and thin on IEX. # Best sectors: Tech, Financials, Healthcare, Energy. # Alpaca wildcard "*" captures ALL ~8,000-9,000 IEX symbols automatically — # the universe below drives Finnhub polling and agent trading decisions. # ============================================================================= STOCK_UNIVERSE_PRIMARY = [ # --- SWEET SPOT: Mid-cap tech $20-$150, high volume, good ML signal --- "AMD","INTC","CRM","ADBE","ORCL","QCOM","TXN","MU","AMAT","LRCX", "KLAC","MRVL","AVGO","CSCO","IBM","HPQ","HPE","DELL","WDC","SNPS", "CDNS","ANSS","DDOG","ZS","CRWD","PANW","FTNT","NET","OKTA","TEAM", "HUBS","NVDA","GOOGL","META","MSFT","AAPL", # --- FINANCIALS: Banks and exchanges have good intraday vol --- "JPM","BAC","GS","MS","WFC","C","V","MA","AXP","COF","DFS","BK", "STT","USB","PNC","TFC","FITB","KEY","CFG","HBAN","MTB","RF","ZION", "CMA","SCHW","RJF","NTRS","ICE","CME","CBOE","SPGI","MCO","BLK", "TROW","IVZ","BEN", # --- HEALTHCARE: Steady movers, good for momentum --- "JNJ","UNH","PFE","ABBV","MRK","CVS","MDT","ABT","TMO","DHR","ISRG", "BSX","EW","SYK","ZBH","BDX","BAX","HOLX","ALGN","IDXX","IQV","LH", "DGX","HCA","CNC","MOH","HUM","CI","ELV","MCK","ABC","CAH","WBA", # --- ENERGY: Volatile, reactive to news, good ML signal --- "XOM","CVX","COP","SLB","OXY","EOG","PXD","MPC","VLO","PSX","HES", "DVN","FANG","BKR","HAL","NOV","MRO","APA","CTRA","AR","EQT", "TPL","DINO","SM","MTDR", # --- CONSUMER DISCRETIONARY --- "AMZN","TSLA","HD","MCD","NKE","SBUX","LOW","TJX","ROST","BBY", "KSS","M","JWN","DRI","CMG","YUM","QSR","TXRH","EAT","DPZ","WEN", "BURL","GPS","ANF","AEO","URBN","PVH","HBI","VFC","LEVI","SKX", # --- CONSUMER STAPLES --- "WMT","PG","KO","PEP","COST","MDLZ","GIS","K","CPB","CAG","HRL", "MKC","CHD","CLX","CL","KMB","NWL","POST","LANC","BGS","INGR", "SJM","THS", # --- INDUSTRIALS --- "CAT","BA","GE","MMM","HON","UPS","FDX","LMT","RTX","NOC","GD", "ETN","EMR","ROK","PH","ITW","DOV","XYL","GNRC","CARR","OTIS","TT", "JCI","SWK","FAST","GWW","MSM","ALLE","RXO","CHRW","EXPD","XPO", "SAIA","ODFL","JBHT","KNX", # --- MATERIALS --- "LIN","APD","ECL","SHW","PPG","NEM","FCX","AA","NUE","STLD","RS", "CMC","ATI","CF","MOS","ALB","FMC","RPM","HUN","EMN","CE","OLN", "SEE","TREX","AVNT", # --- REAL ESTATE --- "AMT","PLD","CCI","EQIX","PSA","EXR","AVB","EQR","MAA","UDR","CPT", "ESS","NNN","O","WPC","KIM","REG","BRX","SPG","IRM","VICI","GLPI", # --- UTILITIES --- "NEE","DUK","SO","D","AEP","EXC","SRE","PEG","ED","FE","ETR","PPL", "CMS","NI","WEC","DTE","CNP","AES","ES","EIX","AWK","LNT","EVRG", # --- COMMUNICATION SERVICES --- "NFLX","DIS","CMCSA","T","VZ","TMUS","CHTR","SIRI","FOXA","NWS", "PARA","WBD", # --- SECTOR ETFs (market signal only — NOT traded) --- "SPY","QQQ","IWM","DIA","MDY", "XLF","XLE","XLK","XLV","XLI","XLU","XLP","XLB","XLRE","XLC", "GLD","SLV","TLT","IEF","HYG","LQD","USO","EEM","VWO","EFA", ] STOCK_UNIVERSE = list(dict.fromkeys(STOCK_UNIVERSE_PRIMARY))[:300] _ETF_SET = { "SPY","QQQ","IWM","DIA","MDY", "XLF","XLE","XLK","XLV","XLI","XLU","XLP","XLB","XLRE","XLC", "GLD","SLV","TLT","IEF","HYG","LQD","USO","EEM","VWO","EFA", } # ============================================================================= # AGENT CONFIGS # ============================================================================= # Tuple: (name, stop_loss_pct, take_profit_pct, min_conf, take_profit_mult, max_pos_pct) _BASE_PROFILES = [ # ── Original 16 (unchanged) ────────────────────────────────────────────── ("UltraConserv", 0.03, 0.06, 0.70, 1.0, 0.20), ("Conservative", 0.05, 0.09, 0.66, 1.1, 0.20), ("ModerateDefens", 0.07, 0.12, 0.63, 1.2, 0.20), ("Moderate", 0.09, 0.15, 0.60, 1.3, 0.20), ("MeanReversion", 0.06, 0.11, 0.62, 0.9, 0.20), ("Momentum", 0.11, 0.18, 0.57, 1.5, 0.20), ("ModerateAggr", 0.12, 0.20, 0.56, 1.4, 0.20), ("Aggressive", 0.14, 0.23, 0.55, 1.6, 0.20), ("HighConviction", 0.10, 0.30, 0.68, 2.0, 0.20), ("Contrarian", 0.08, 0.14, 0.59, 1.1, 0.20), ("Volatility", 0.13, 0.17, 0.58, 1.3, 0.20), ("UltraAggress", 0.18, 0.28, 0.53, 1.7, 0.20), ("SectorRotation", 0.09, 0.20, 0.61, 1.2, 0.20), ("LowFreq", 0.06, 0.10, 0.67, 1.8, 0.20), ("HighFreq", 0.12, 0.22, 0.54, 1.0, 0.20), ("Balanced", 0.08, 0.13, 0.61, 1.3, 0.20), # ── Sentiment Specialists (17-21) ──────────────────────────────────────── ("NewsSentiment", 0.06, 0.12, 0.65, 1.3, 0.20), ("SocialMomentum", 0.07, 0.15, 0.58, 1.3, 0.15), ("CongressTracker", 0.08, 0.18, 0.67, 1.4, 0.20), ("SentimentDecay", 0.06, 0.10, 0.62, 1.3, 0.20), ("SECWatcher", 0.07, 0.20, 0.70, 1.5, 0.20), # ── Oscillator Specialists (22-31) ─────────────────────────────────────── ("MACDPure", 0.06, 0.12, 0.62, 1.3, 0.20), ("RSIBounce", 0.05, 0.10, 0.63, 1.3, 0.20), ("BollingerBreakout", 0.07, 0.15, 0.65, 1.4, 0.20), ("BollingerSqueeze", 0.08, 0.18, 0.68, 1.4, 0.18), ("VWAPTrader", 0.04, 0.08, 0.62, 1.3, 0.20), ("StochCycles", 0.05, 0.10, 0.60, 1.3, 0.20), ("MFIFlow", 0.06, 0.13, 0.63, 1.3, 0.20), ("OBVFollower", 0.07, 0.14, 0.60, 1.3, 0.20), ("ADXTrend", 0.06, 0.16, 0.65, 1.4, 0.20), ("GapTrader", 0.05, 0.12, 0.67, 1.3, 0.20), # ── Volatility Regime Specialists (32-36) ──────────────────────────────── ("LowVolEnv", 0.04, 0.10, 0.65, 1.3, 0.20), ("HighVolEnv", 0.10, 0.22, 0.62, 1.5, 0.20), ("ATRScaled", 0.06, 0.12, 0.62, 1.4, 0.20), ("VolImbalance", 0.06, 0.13, 0.62, 1.3, 0.20), ("CorwinSchultz", 0.06, 0.15, 0.65, 1.3, 0.25), # ── Trend & Slope Specialists (37-41) ──────────────────────────────────── ("TrendLong", 0.08, 0.20, 0.64, 1.5, 0.20), ("TrendShort", 0.05, 0.10, 0.60, 1.3, 0.20), ("SlopeAccel", 0.07, 0.18, 0.68, 1.4, 0.20), ("MomentumAge", 0.06, 0.14, 0.65, 1.3, 0.20), ("ConsistencyTrader", 0.07, 0.16, 0.70, 1.4, 0.22), # ── Volume Specialists (42-44) ─────────────────────────────────────────── ("VolRatioBurst", 0.06, 0.13, 0.62, 1.3, 0.20), ("VolTrend", 0.06, 0.12, 0.61, 1.3, 0.20), ("MoneyFlow", 0.07, 0.16, 0.66, 1.4, 0.20), # ── Stop/Target Architecture Variants (45-47) ──────────────────────────── ("TightScalper", 0.025,0.05, 0.60, 1.3, 0.12), ("WideSwing", 0.12, 0.28, 0.68, 1.6, 0.20), ("AsymmetricRR", 0.04, 0.20, 0.66, 2.0, 0.15), # ── Composite Specialists (48-50) ──────────────────────────────────────── ("TechPure", 0.07, 0.15, 0.63, 1.3, 0.20), ("SentPure", 0.07, 0.15, 0.65, 1.3, 0.20), ("BreakoutConfirm", 0.07, 0.15, 0.70, 1.4, 0.20), ] def _mutate(val, lo, hi, scale=0.12): return float(np.clip(val + np.random.uniform(-scale, scale) * (hi - lo), lo, hi)) def generate_agent_configs(n_base, n_clones): """ Build configs for n_base base agents + n_clones clones (1-to-1 pairing). Clone names: Clone__ (N = 1-indexed position in _BASE_PROFILES). """ rng = random.Random(42) configs = [] n_base = min(n_base, len(_BASE_PROFILES)) for i in range(n_base): p = _BASE_PROFILES[i] # (name, stop, tp, conf, mult, max_pos) configs.append({ "name": p[0], "stop_loss_pct": p[1], "take_profit_pct": p[2], "min_confidence": p[3], "take_profit_mult": p[4], "max_position_pct": p[5], "risk_tolerance": p[1], "is_clone": False, "n_estimators": rng.randint(200, 400), "learning_rate": round(rng.uniform(0.02, 0.08), 3), "max_depth": rng.randint(3, 5), }) n_clones = min(n_clones, n_base) for i in range(n_clones): base = configs[i] # 1-to-1: clone i mirrors base i np.random.seed(i + 100) configs.append({ "name": f"Clone_{base['name']}_{i+1}", "stop_loss_pct": _mutate(base["stop_loss_pct"], *STOP_LOSS_RANGE), "take_profit_pct": _mutate(base["take_profit_pct"], *TAKE_PROFIT_RANGE), "min_confidence": min(_mutate(base["min_confidence"], *MIN_CONF_RANGE), 0.72), "take_profit_mult": base["take_profit_mult"] * np.random.uniform(0.8, 1.2), "max_position_pct": _mutate(base["max_position_pct"], *MAX_POS_PCT_RANGE), "risk_tolerance": base["risk_tolerance"], "is_clone": True, "n_estimators": rng.randint(150, 350), "learning_rate": round(rng.uniform(0.02, 0.10), 3), "max_depth": rng.randint(3, 5), }) return configs AGENT_CONFIGS = generate_agent_configs(BASE_AGENTS, CLONE_AGENTS) # ============================================================================= # MARKET HOURS # ============================================================================= def is_market_open() -> bool: now = datetime.now(MARKET_TZ) if now.weekday() >= 5: return False open_t = now.replace(hour=9, minute=30, second=0, microsecond=0) close_t = now.replace(hour=16, minute=0, second=0, microsecond=0) return open_t <= now < close_t def seconds_until_open() -> float: now = datetime.now(MARKET_TZ) nxt = now.replace(hour=9, minute=30, second=0, microsecond=0) if now >= nxt: nxt += timedelta(days=1) while nxt.weekday() >= 5: nxt += timedelta(days=1) return (nxt - now).total_seconds() def market_minutes_remaining() -> float: now = datetime.now(MARKET_TZ) close = now.replace(hour=16, minute=0, second=0, microsecond=0) return max(0.0, (close - now).total_seconds() / 60.0) # ============================================================================= # DATABASE # ============================================================================= class Database: def __init__(self, path: str): self._path = path self._local = threading.local() self._lock = threading.Lock() self._init_schema() self._dedup: dict = {} def _conn(self): if not hasattr(self._local, "conn") or self._local.conn is None: conn = sqlite3.connect(self._path, check_same_thread=False) # WAL mode: concurrent reads never block writes. # Fixes Alpaca ping/pong timeouts caused by feature-rebuild reads # competing with bar-insert writes for the same DB lock. conn.execute("PRAGMA journal_mode=WAL") conn.execute("PRAGMA synchronous=NORMAL") conn.execute("PRAGMA cache_size=-32768") conn.execute("PRAGMA mmap_size=268435456") conn.execute("PRAGMA temp_store=MEMORY") conn.execute("PRAGMA busy_timeout=5000") # NORMAL is safe with WAL and ~3x faster than FULL (default). conn.execute("PRAGMA synchronous=NORMAL") # 32 MB page cache per connection (negative = kilobytes). conn.execute("PRAGMA cache_size=-32000") # Wait up to 5s if DB is locked before raising an error. conn.execute("PRAGMA busy_timeout=5000") self._local.conn = conn return self._local.conn def _init_schema(self): with sqlite3.connect(self._path) as conn: conn.execute("PRAGMA journal_mode=WAL") conn.execute("PRAGMA synchronous=NORMAL") conn.execute("PRAGMA cache_size=-32768") conn.execute("PRAGMA mmap_size=268435456") conn.execute("PRAGMA temp_store=MEMORY") conn.execute("PRAGMA busy_timeout=5000") conn.execute("PRAGMA synchronous=NORMAL") conn.execute("PRAGMA cache_size=-32000") conn.execute("PRAGMA busy_timeout=5000") conn.execute(""" CREATE TABLE IF NOT EXISTS quotes ( id INTEGER PRIMARY KEY AUTOINCREMENT, symbol TEXT NOT NULL, ts INTEGER NOT NULL, ts_micro INTEGER NOT NULL DEFAULT 0, price REAL NOT NULL, open REAL, high REAL, low REAL, prev_close REAL, volume REAL, source TEXT DEFAULT 'finnhub' )""") conn.execute("CREATE INDEX IF NOT EXISTS idx_sym_ts ON quotes(symbol,ts)") conn.execute(""" CREATE TABLE IF NOT EXISTS accuracy_log ( id INTEGER PRIMARY KEY AUTOINCREMENT, ts INTEGER NOT NULL, agent TEXT NOT NULL, discrepancy REAL, profit_rate REAL, trades INTEGER, gross_profit REAL DEFAULT 0, gross_loss REAL DEFAULT 0 )""") conn.commit() def insert(self, symbol, ts, price, open_, high, low, prev_close, volume, source="finnhub"): bucket = (symbol, ts // 5) with self._lock: if bucket in self._dedup: return self._dedup[bucket] = True if len(self._dedup) > 50_000: oldest = list(self._dedup.keys())[:10_000] for k in oldest: del self._dedup[k] try: conn = self._conn() conn.execute( "INSERT INTO quotes(symbol,ts,ts_micro,price,open,high,low,prev_close,volume,source)" " VALUES(?,?,?,?,?,?,?,?,?,?)", (symbol, ts, int(time.time() * 1_000_000) % 1_000_000, price, open_, high, low, prev_close, volume, source) ) conn.commit() except Exception as e: logger.debug(f"DB insert error {symbol}: {e}") def bulk_insert(self, rows: list) -> int: if not rows: return 0 n = 0 with self._lock: try: conn = self._conn() for r in rows: sym, ts = r[0], r[1] bucket = (sym, ts // 5) if bucket in self._dedup: continue self._dedup[bucket] = True conn.execute( "INSERT OR IGNORE INTO quotes(symbol,ts,ts_micro,price,open,high,low,prev_close,volume,source)" " VALUES(?,?,?,?,?,?,?,?,?,?)", (r[0], r[1], 0, r[2], r[3], r[4], r[5], r[6], r[7], r[8] if len(r) > 8 else "external") ) n += 1 conn.commit() except Exception as e: logger.debug(f"DB bulk insert error: {e}") return n def get_series(self, symbol: str) -> pd.DataFrame: try: conn = self._conn() df = pd.read_sql_query( "SELECT ts,price,open,high,low,prev_close,volume FROM quotes" " WHERE symbol=? ORDER BY ts ASC", conn, params=(symbol,) ) return df except Exception: return pd.DataFrame() def row_counts(self) -> dict: try: conn = self._conn() rows = conn.execute( "SELECT symbol, COUNT(*) FROM quotes GROUP BY symbol" ).fetchall() return {r[0]: r[1] for r in rows} except Exception: return {} def log_accuracy(self, ts, agent, disc, profit_rate, trades, gross_profit=0.0, gross_loss=0.0): with self._lock: try: conn = self._conn() conn.execute( "INSERT INTO accuracy_log(ts,agent,discrepancy,profit_rate,trades,gross_profit,gross_loss)" " VALUES(?,?,?,?,?,?,?)", (ts, agent, disc, profit_rate, trades, gross_profit, gross_loss) ) conn.commit() except Exception as e: logger.debug(f"log_accuracy error: {e}") def get_accuracy_history(self, agent: str, limit: int = 100) -> pd.DataFrame: try: conn = self._conn() return pd.read_sql_query( "SELECT ts,discrepancy,profit_rate,trades,gross_profit,gross_loss" " FROM accuracy_log WHERE agent=? ORDER BY ts DESC LIMIT ?", conn, params=(agent, limit) ) except Exception: return pd.DataFrame() def size_mb(self) -> float: try: return os.path.getsize(self._path) / 1_048_576 except Exception: return 0.0 # ============================================================================= # FINNHUB CLIENT (58/min, strict round-robin, supplemental to Alpaca) # ============================================================================= class FinnhubClient: def __init__(self, api_key: str): self.client = finnhub.Client(api_key=api_key) self._times = deque() self._lock = threading.Lock() def _throttle(self): while True: with self._lock: now = time.time() while self._times and now - self._times[0] > 60.0: self._times.popleft() if len(self._times) < FH_CALLS_PER_MIN: self._times.append(time.time()) return wait = 61.0 - (now - self._times[0]) if wait > 0: time.sleep(wait) def calls_this_minute(self) -> int: now = time.time() with self._lock: return sum(1 for t in self._times if now - t <= 60.0) def quote(self, symbol: str): self._throttle() try: q = self.client.quote(symbol) return q if q and q.get("c", 0) > 0 else None except Exception as e: logger.debug(f"Quote error {symbol}: {e}") return None def profile(self, symbol: str): self._throttle() try: return self.client.company_profile2(symbol=symbol) except Exception as e: logger.debug(f"Profile error {symbol}: {e}") return None # ============================================================================= # ALPACA WEBSOCKET CLIENT (free IEX, minute bars, unlimited symbols) # # Coverage summary: # - Subscribes with wildcard "*" on bars channel # - Receives minute bars for ALL ~8,000-9,000 IEX-listed symbols # - Each bar: open, high, low, close, volume, timestamp, vwap, trade_count # - Bars arrive ~1-2 seconds after each minute closes # - Zero REST calls consumed — one persistent WebSocket connection # - Reconnects automatically on disconnect # - price_velocity computed per bar: (close-prev_close)/seconds_elapsed # ============================================================================= class AlpacaWebSocketClient: """ Connects to Alpaca's free IEX data stream and subscribes to minute bars for ALL symbols using the "*" wildcard. Received bars are written directly to the shared Database and prices dict. price_velocity is computed from the time delta between consecutive bars for each symbol and stored alongside the price for use as a feature. Runs in a background daemon thread. Reconnects automatically on dropout. """ def __init__(self, db: "Database", prices: dict, velocity: dict): self.db = db self.prices = prices # shared dict: symbol -> latest price self.velocity = velocity # shared dict: symbol -> price_velocity ($/s) self._running = False self._thread = None self._ws = None # Track last bar time per symbol for velocity calculation self._last_bar: dict = {} # symbol -> (ts, close) self._bars_received = 0 self._lock = threading.Lock() def _on_open(self, ws): logger.info("Alpaca WS: connected, authenticating...") # Do NOT reset _consec_fails here. Resetting on WS_OPEN caused a rapid # reconnect loop when Alpaca was rate-limiting us: connect -> immediate # drop -> wait 10s (because fails=0) -> repeat. The counter now only # resets after we have received a meaningful number of bars, confirming # the session was genuinely healthy and not just a momentary handshake. self._connect_time = time.time() self._bars_at_connect = self._bars_received # snapshot for health check debug_logger.info(f"WS_OPEN | bars_so_far={self._bars_received} | consec_fails={getattr(self,'_consec_fails',0)}") auth_msg = json.dumps({ "action": "auth", "key": ALPACA_API_KEY, "secret": ALPACA_SECRET_KEY, }) ws.send(auth_msg) def _on_message(self, ws, message): try: data = json.loads(message) if not isinstance(data, list): data = [data] for msg in data: msg_type = msg.get("T") if msg_type == "success": if msg.get("msg") == "authenticated": logger.info("Alpaca WS: authenticated — subscribing to all bars (*)") sub_msg = json.dumps({ "action": "subscribe", "bars": ["*"], # ALL IEX symbols, no symbol limit }) ws.send(sub_msg) elif msg.get("msg") == "connected": logger.info("Alpaca WS: connection established") elif msg_type == "subscription": bar_count = len(msg.get("bars", [])) logger.info(f"Alpaca WS: subscribed — bars={bar_count} (wildcard active)") elif msg_type == "b": # Minute bar received self._handle_bar(msg) elif msg_type == "error": logger.error(f"Alpaca WS error: {msg.get('msg')} (code={msg.get('code')})") except Exception as e: logger.debug(f"Alpaca WS message error: {e}") def _handle_bar(self, msg: dict): """ Process a single minute bar from Alpaca. Format: {T, S, o, h, l, c, v, t, n, vw} T = message type ("b") S = symbol o = open, h = high, l = low, c = close v = volume, t = bar timestamp (ISO), n = trade count, vw = vwap """ try: sym = msg.get("S", "") if not sym: return close = float(msg.get("c", 0)) if close <= 0: return open_ = float(msg.get("o", close)) high = float(msg.get("h", close)) low = float(msg.get("l", close)) vol = float(msg.get("v", 0)) # Parse ISO timestamp from Alpaca bar t_str = msg.get("t", "") try: bar_dt = datetime.fromisoformat(t_str.replace("Z", "+00:00")) ts = int(bar_dt.timestamp()) except Exception: ts = int(time.time()) # Compute price velocity: change per second since last bar with self._lock: prev = self._last_bar.get(sym) if prev is not None: prev_ts, prev_close = prev elapsed = max(ts - prev_ts, 1) # seconds between bars velocity = (close - prev_close) / elapsed else: velocity = 0.0 self._last_bar[sym] = (ts, close) # Write to DB (prev_close = last known close for this symbol) prev_close = self._last_bar.get(sym, (0, close))[1] if sym in self._last_bar else close self.db.insert(sym, ts, close, open_, high, low, prev_close, vol, source="alpaca") # Update shared price and velocity dicts self.prices[sym] = close self.velocity[sym] = velocity with self._lock: self._bars_received += 1 if self._bars_received % 1000 == 0: logger.info(f"Alpaca WS: {self._bars_received:,} bars received total") except Exception as e: logger.debug(f"Alpaca bar error: {e}") def _on_error(self, ws, error): logger.warning(f"Alpaca WS error: {error}") debug_logger.warning( f"WS_ERROR | {error} | bars={self._bars_received:,} | " f"resource={_resource_guard.status} | " f"load={_resource_guard.load_str} | " f"ram={_resource_guard._ram_used_mb}MB" ) def _on_close(self, ws, close_status_code, close_msg): logger.warning(f"Alpaca WS closed (code={close_status_code}): {close_msg}") debug_logger.warning( f"WS_CLOSE | code={close_status_code} | msg={close_msg} | " f"bars={self._bars_received:,} | consec_fails={self._consec_fails} | " f"resource={_resource_guard.status} | " f"load={_resource_guard.load_str} | " f"ram={_resource_guard._ram_used_mb}MB" ) def _run_forever(self): """Reconnect loop with smart backoff -- restarts the WebSocket if it drops. KEY CHANGES FROM v8: - Minimum reconnect wait is 30s regardless of fail count. Previously 10s * consec_fails with consec_fails resetting on WS_OPEN meant every reconnect after a rate-limit drop waited only 10s, causing Alpaca to ban the connection immediately on re-auth. Alpaca has a server-side rate limit on rapid reconnects from the same account. - _consec_fails only resets if we received 50+ bars in the session. A session that connects and drops within 90 seconds without bars is a rate-limit bounce -- we should keep backing off, not restart at 10s. - ping_interval=20, ping_timeout=15: pings more often (catches network stalls faster) but gives a longer window for the pong reply, reducing spurious timeouts under heavy DB load. - Max wait is 5 minutes (300s) instead of 2 minutes (120s). This gives Alpaca enough time to lift a temporary connection ban. """ # Minimum wait between reconnect attempts regardless of fail count. # Prevents rapid-reconnect bans from Alpaca's server-side rate limiter. MIN_RECONNECT_WAIT = 30 _set_nice(-5) # P0 CRITICAL: Alpaca WS gets highest CPU priority while self._running: if not ALPACA_API_KEY or not ALPACA_SECRET_KEY: logger.warning("Alpaca WS: no API keys set -- stream disabled.") return try: self._consec_fails = getattr(self, "_consec_fails", 0) self._bars_at_connect = self._bars_received debug_logger.info( f"WS_CONNECT_ATTEMPT | attempt={self._consec_fails+1} | " f"bars_so_far={self._bars_received}" ) logger.info(f"Alpaca WS: connecting to {ALPACA_WS_URL}") self._ws = websocket.WebSocketApp( ALPACA_WS_URL, on_open = self._on_open, on_message = self._on_message, on_error = self._on_error, on_close = self._on_close, ) # ping_interval=20: ping every 20s (catches stalls faster than 30s) # ping_timeout=15: allow 15s for pong (more headroom under DB load) self._ws.run_forever(ping_interval=30, ping_timeout=25) # run_forever returned -- assess session health before deciding backoff bars_this_session = self._bars_received - getattr(self, "_bars_at_connect", self._bars_received) if bars_this_session >= 50: # Healthy session: we received real data before dropping. # This is a normal network hiccup -- reset the fail counter. self._consec_fails = 0 debug_logger.info( f"WS_DROPPED_HEALTHY | bars_this_session={bars_this_session} | " f"consec_fails reset to 0" ) else: # Dropped before receiving meaningful data -- likely a rate-limit # or auth rejection. Keep backing off. self._consec_fails += 1 debug_logger.warning( f"WS_DROPPED | consec_fails={self._consec_fails} | " f"bars_this_session={bars_this_session} | " f"bars_so_far={self._bars_received}" ) except Exception as e: logger.error(f"Alpaca WS exception: {e}") debug_logger.error(f"WS_EXCEPTION | {type(e).__name__}: {e}", exc_info=True) self._consec_fails = getattr(self, "_consec_fails", 0) + 1 if self._running: # Exponential backoff: 30s minimum, grows by 30s per fail, caps at 5 min. # The large cap gives Alpaca time to lift a temporary connection ban. wait = max(MIN_RECONNECT_WAIT, min(30 * self._consec_fails, 300)) logger.info(f"Alpaca WS: reconnecting in {wait}s (consec_fails={self._consec_fails})...") debug_logger.info(f"WS_RECONNECT_WAIT | {wait}s | consec_fails={self._consec_fails}") time.sleep(wait) debug_logger.info("WS_LOOP_EXITED | _running=False") def start(self): if not _WS_AVAILABLE: logger.warning("Alpaca WS: websocket-client not installed, stream disabled.") return self._running = True self._thread = threading.Thread(target=self._run_forever, daemon=True, name="AlpacaWebSocket") self._thread.start() logger.info("Alpaca WebSocket thread started.") def stop(self): self._running = False if self._ws: try: self._ws.close() except Exception: pass def bars_received(self) -> int: return self._bars_received # ============================================================================= # SUPPLEMENTAL DATA SOURCES # ============================================================================= class AlphaVantageClient: BASE = "https://www.alphavantage.co/query" def __init__(self, api_key: str): self.key = api_key self._calls_today = 0 self._last_reset = datetime.now().date() self._lock = threading.Lock() def _can_call(self) -> bool: with self._lock: today = datetime.now().date() if today != self._last_reset: self._calls_today = 0 self._last_reset = today return bool(self.key) and self._calls_today < 23 def get_intraday(self, symbol: str, interval: str = "5min") -> list: if not self._can_call(): return [] try: r = requests.get(self.BASE, params={ "function": "TIME_SERIES_INTRADAY", "symbol": symbol, "interval": interval, "outputsize": "compact", "apikey": self.key, }, timeout=10) data = r.json() key = f"Time Series ({interval})" if key not in data: return [] rows = [] ts_data = data[key] prices_list = list(ts_data.items()) for i, (dt_str, bar) in enumerate(prices_list): ts = int(datetime.strptime(dt_str, "%Y-%m-%d %H:%M:%S").timestamp()) close = float(bar["4. close"]) prev = float(prices_list[i+1][1]["4. close"]) if i+1 < len(prices_list) else close rows.append(( symbol, ts, close, float(bar["1. open"]), float(bar["2. high"]), float(bar["3. low"]), prev, float(bar["5. volume"]), "alphavantage" )) with self._lock: self._calls_today += 1 logger.info(f" AlphaVantage: {symbol} {len(rows)} bars ({interval})") return rows except Exception as e: logger.debug(f"AlphaVantage error {symbol}: {e}") return [] class PolygonClient: BASE = "https://api.polygon.io/v2/aggs/ticker" def __init__(self, api_key: str): self.key = api_key self._fetched_today: set = set() self._last_reset = datetime.now().date() self._times = deque() self._lock = threading.Lock() def _throttle(self): while True: with self._lock: now = time.time() while self._times and now - self._times[0] > 60.0: self._times.popleft() if len(self._times) < 4: self._times.append(time.time()) return wait = 61.0 - (now - self._times[0]) if wait > 0: time.sleep(wait) def _reset_if_new_day(self): today = datetime.now().date() with self._lock: if today != self._last_reset: self._fetched_today.clear() self._last_reset = today def get_prev_day_minute_bars(self, symbol: str) -> list: self._reset_if_new_day() if not self.key or symbol in self._fetched_today: return [] self._throttle() try: today = datetime.now(MARKET_TZ).date() prev = today - timedelta(days=1) if prev.weekday() >= 5: prev -= timedelta(days=prev.weekday() - 4) date_str = prev.strftime("%Y-%m-%d") url = f"{self.BASE}/{symbol}/range/1/minute/{date_str}/{date_str}" r = requests.get(url, params={"adjusted": "true", "sort": "asc", "limit": 500, "apiKey": self.key}, timeout=15) data = r.json() if data.get("status") not in ("OK", "DELAYED") or "results" not in data: return [] rows = [] results = data["results"] for i, bar in enumerate(results): ts = int(bar["t"] / 1000) close = float(bar["c"]) prev_c = float(results[i-1]["c"]) if i > 0 else close rows.append(( symbol, ts, close, float(bar["o"]), float(bar["h"]), float(bar["l"]), prev_c, float(bar.get("v", 0)), "polygon" )) with self._lock: self._fetched_today.add(symbol) logger.info(f" Polygon: {symbol} {len(rows)} bars for {date_str}") return rows except Exception as e: logger.debug(f"Polygon error {symbol}: {e}") return [] # ============================================================================= # FEATURE ENGINEERING (includes price_velocity from Alpaca bars) # ============================================================================= FEATURE_COLS = [ # Price momentum "ret_1", "ret_3", "ret_5", "ret_10", "ret_20", # RSI "rsi", "rsi_diff", # MACD "macd", "macd_sig", "macd_hist", "macd_cross", # Bollinger Bands "bb_pct_b", "bb_bwidth", # Moving average position "vs_sma_5", "vs_sma_10", "vs_sma_20", "vs_sma_50", # Volume "vol_ratio", # Rolling volatility "vol5", "vol10", "vol20", # Candle structure "hl_range", "body_ratio", # Gap detection "gap_open", # Statistical "ret_skew", "autocorr_1", "vol_trend", "price_accel", # Bar velocity "price_velocity", # Session B: Technical Indicators (ADX, OBV, VWAP, ATR, Stochastic, MFI) "adx", # Average Directional Index -- trend strength [0,100] "adx_pos", # +DI positive directional indicator "obv_norm", # OBV z-score vs 20-bar rolling mean "vwap_dist", # % distance from 20-bar rolling VWAP "atr_pct", # ATR as % of price "stoch_k", # Stochastic %K normalized [0,1] "stoch_d", # Stochastic %D normalized [0,1] "mfi", # Money Flow Index normalized [0,1] # Sentiment (v8.1 -- 5 sources) "news_sentiment", "news_count", "sec_flag", "reddit_mentions", "reddit_sentiment", "congress_bought", "stocktwits_bullish", "stocktwits_mentions", "slope_8", "slope_20", "trend_cons_8", "trend_cons_20", "momentum_age", "slope_accel", # v9: Research-backed microstructure + volatility features "yang_zhang_vol", # Yang-Zhang: 14x more efficient than close-close vol "vol_imbalance", # (price-VWAP)/ATR: buy/sell pressure proxy "corwin_schultz", # bid-ask spread from OHLC (no L2 needed) "sentiment_decay", # exponentially decayed composite sentiment (30min half-life) "atr_z", # ATR z-score: volatility regime signal ] N_FEATURES = len(FEATURE_COLS) # 56 def build_features(df: pd.DataFrame, velocity_map: dict = None, sentiment_scores: dict = None) -> "pd.DataFrame | None": """ Builds the full feature matrix for a single symbol. Accepts optional sentiment_scores dict from SentimentEngine.get_scores(). All sentiment features default to 0 if no scores are available yet — the model will treat the stock as neutral until data arrives. """ if len(df) < 55: return None f = pd.DataFrame(index=df.index) c = df["price"].astype(float) hi = df["high"].fillna(c).astype(float) lo = df["low"].fillna(c).astype(float) op = df["open"].fillna(c).astype(float) vo = df["volume"].fillna(0).astype(float) for d, n in [(1,"ret_1"),(3,"ret_3"),(5,"ret_5"),(10,"ret_10"),(20,"ret_20")]: f[n] = c.pct_change(d) delta = c.diff() ag = delta.clip(lower=0).ewm(com=13, min_periods=14).mean() al = (-delta.clip(upper=0)).ewm(com=13, min_periods=14).mean() f["rsi"] = 100 - 100 / (1 + ag / (al + 1e-10)) f["rsi_diff"] = f["rsi"].diff() e12 = c.ewm(span=12, adjust=False).mean() e26 = c.ewm(span=26, adjust=False).mean() macd = e12 - e26 sig = macd.ewm(span=9, adjust=False).mean() f["macd"] = macd f["macd_sig"] = sig f["macd_hist"] = macd - sig f["macd_cross"] = (macd > sig).astype(int) sma20 = c.rolling(20).mean() std20 = c.rolling(20).std() f["bb_pct_b"] = (c - (sma20 - 2*std20)) / (4*std20 + 1e-10) f["bb_bwidth"] = (4*std20) / (sma20 + 1e-10) for w in [5, 10, 20, 50]: sma = c.rolling(w).mean() f[f"vs_sma_{w}"] = (c - sma) / (sma + 1e-10) vma = vo.rolling(20).mean() f["vol_ratio"] = vo / (vma + 1e-10) r1 = c.pct_change() for w, n in [(5,"vol5"),(10,"vol10"),(20,"vol20")]: f[n] = r1.rolling(w).std() f["hl_range"] = (hi - lo) / (c + 1e-10) f["body_ratio"] = (c - op).abs() / (hi - lo + 1e-10) f["gap_open"] = (op - c.shift(1)) / (c.shift(1) + 1e-10) f["ret_skew"] = r1.rolling(20).skew() f["autocorr_1"] = r1.rolling(20).apply( lambda x: x.autocorr(lag=1) if len(x) > 2 else 0.0, raw=False ) vol_short = r1.rolling(5).std() vol_long = r1.rolling(20).std() f["vol_trend"] = vol_short / (vol_long + 1e-10) f["price_accel"] = r1.diff().rolling(5).mean() # price_velocity: from Alpaca bar timing. If not available, derive from # price difference / assumed 60s bar interval as fallback. if "ts" in df.columns: ts_s = df["ts"].astype(float) dt = ts_s.diff().replace(0, 60).fillna(60) f["price_velocity"] = c.diff() / dt else: f["price_velocity"] = c.diff() / 60.0 # ----------------------------------------------------------------------- # SESSION B: TECHNICAL INDICATORS # ----------------------------------------------------------------------- if TA_OK: try: adx_ind = ta.trend.ADXIndicator(high=hi, low=lo, close=c, window=14, fillna=True) f["adx"] = adx_ind.adx() f["adx_pos"] = adx_ind.adx_pos() except Exception: f["adx"] = 25.0; f["adx_pos"] = 25.0 try: obv = ta.volume.OnBalanceVolumeIndicator( close=c, volume=vo, fillna=True).on_balance_volume() f["obv_norm"] = (obv - obv.rolling(20).mean()) / ( obv.rolling(20).std().replace(0, 1)) except Exception: f["obv_norm"] = 0.0 try: tp = (hi + lo + c) / 3 vwap = (tp * vo).rolling(20).sum() / vo.rolling(20).sum().replace(0, np.nan) f["vwap_dist"] = (c - vwap) / (vwap + 1e-8) except Exception: f["vwap_dist"] = 0.0 try: atr = ta.volatility.AverageTrueRange( high=hi, low=lo, close=c, window=14, fillna=True ).average_true_range() f["atr_pct"] = atr / (c + 1e-8) except Exception: f["atr_pct"] = 0.01 try: stoch = ta.momentum.StochasticOscillator( high=hi, low=lo, close=c, window=14, smooth_window=3, fillna=True) f["stoch_k"] = stoch.stoch() / 100.0 f["stoch_d"] = stoch.stoch_signal() / 100.0 except Exception: f["stoch_k"] = 0.5; f["stoch_d"] = 0.5 try: f["mfi"] = ta.volume.MFIIndicator( high=hi, low=lo, close=c, volume=vo, window=14, fillna=True ).money_flow_index() / 100.0 except Exception: f["mfi"] = 0.5 else: # Fallbacks when ta not available ret_s = c.pct_change() f["adx"] = (ret_s.rolling(14).std() * 100).clip(0, 100) f["adx_pos"] = 25.0 obv_s = (np.sign(c.diff()) * vo).cumsum() f["obv_norm"] = (obv_s - obv_s.rolling(20).mean()) / (obv_s.rolling(20).std() + 1) f["vwap_dist"]= (c - c.rolling(20).mean()) / (c.rolling(20).mean() + 1e-8) f["atr_pct"] = ((hi - lo) / (c + 1e-8)).rolling(14).mean() lo14, hi14 = lo.rolling(14).min(), hi.rolling(14).max() f["stoch_k"] = ((c - lo14) / (hi14 - lo14 + 1e-8)).rolling(3).mean() f["stoch_d"] = f["stoch_k"].rolling(3).mean() f["mfi"] = 0.5 # ----------------------------------------------------------------------- # SENTIMENT FEATURES (v8) # Sentiment scores come from SentimentEngine running in background threads. # They are constant across all rows in this symbol's feature matrix — # the model learns whether the CURRENT sentiment state correlates with # upcoming price direction. Scores update every 15-60 min depending on # the source. All values default to 0 (neutral) if not yet available. # ----------------------------------------------------------------------- s = sentiment_scores or {} n_rows = len(f) # Fill entire column with the current score (repeated for all bars) # The model sees sentiment as a "current state" feature, not historical f["news_sentiment"] = float(s.get("news_sentiment", 0.0)) f["news_count"] = float(s.get("news_count", 0)) f["sec_flag"] = float(s.get("sec_flag", 0)) f["reddit_mentions"] = float(s.get("reddit_mentions", 0)) f["reddit_sentiment"] = float(s.get("reddit_sentiment", 0.0)) f["congress_bought"] = float(s.get("congress_bought", 0)) f["stocktwits_bullish"] = float(s.get("stocktwits_bullish", 0.5)) f["stocktwits_mentions"] = float(s.get("stocktwits_mentions", 0)) # Ensure Session B columns exist even if TA_OK was False for _col, _def in [ ("adx",0.25),("adx_pos",0.25),("obv_norm",0.0),("vwap_dist",0.0), ("atr_pct",0.01),("stoch_k",0.5),("stoch_d",0.5),("mfi",0.5)]: if _col not in f.columns: f[_col] = _def f["target_ret"] = c.pct_change().shift(-1) f["target_dir"] = (f["target_ret"] > 0).astype(int) # Sequence / directional features if len(f) >= 21: _c = f["ret_1"].cumsum() # proxy close from cumulative returns _ret = f["ret_1"].fillna(0.0) def _slope(s, w): _x = np.arange(w, dtype=np.float32) _o = np.full(len(s), np.nan) for _j in range(w - 1, len(s)): _o[_j] = np.polyfit(_x, s.iloc[_j-w+1:_j+1].values, 1)[0] return pd.Series(_o, index=s.index) f["slope_8"] = _slope(_ret, 8).fillna(0.0) f["slope_20"] = _slope(_ret, 20).fillna(0.0) _up = (_ret > 0).astype(float) _dn = (_ret < 0).astype(float) f["trend_cons_8"] = np.where(f["slope_8"] >= 0, _up.rolling(8).mean(), _dn.rolling(8).mean()).clip(0, 1) f["trend_cons_20"] = np.where(f["slope_20"] >= 0, _up.rolling(20).mean(), _dn.rolling(20).mean()).clip(0, 1) f["trend_cons_8"] = f["trend_cons_8"].fillna(0.5) f["trend_cons_20"] = f["trend_cons_20"].fillna(0.5) _ema8 = _c.ewm(span=8, adjust=False).mean() _above = (_c > _ema8).astype(int) _age, _cnt = np.zeros(len(f)), 0 for _j in range(len(_above)): _cnt = (max(0, _cnt) + 1) if _above.iloc[_j] == 1 else (min(0, _cnt) - 1) _age[_j] = _cnt f["momentum_age"] = _age / 20.0 f["slope_accel"] = ( f["slope_8"] / f["slope_20"].replace(0, np.nan) ).clip(-3, 3).fillna(0.0) else: for _sf in ["slope_8","slope_20","trend_cons_8", "trend_cons_20","momentum_age","slope_accel"]: f[_sf] = 0.0 # ----------------------------------------------------------------------- # v9: YANG-ZHANG VOLATILITY (14x more efficient than close-to-close) # Unbiased estimator combining overnight, close-to-open, and Rogers-Satchell # ----------------------------------------------------------------------- try: _N = 14 _o = op.apply(np.log) _c2 = c.apply(np.log) _h = hi.apply(np.log) _l = lo.apply(np.log) _cprev = _c2.shift(1) _k = 0.34 / (1.34 + (_N + 1) / (_N - 1)) _ov = (_o - _cprev).rolling(_N).var() _cv = (_c2 - _o).rolling(_N).var() _rs = ((_h - _o) * (_h - _c2) + (_l - _o) * (_l - _c2)).rolling(_N).mean() f["yang_zhang_vol"] = np.sqrt((_ov + _k * _cv + (1 - _k) * _rs).clip(lower=0)) except Exception: f["yang_zhang_vol"] = f.get("atr_pct", pd.Series(0.01, index=f.index)) # ----------------------------------------------------------------------- # v9: VOLUME IMBALANCE -- (price - VWAP) / ATR # Proxies order book pressure without L2 data # ----------------------------------------------------------------------- try: _tp = (hi + lo + c) / 3 _vwap = (_tp * vo).rolling(20).sum() / vo.rolling(20).sum().replace(0, np.nan) _atr14 = ((hi - lo).rolling(14).mean()).replace(0, np.nan) f["vol_imbalance"] = ((c - _vwap) / (_atr14 + 1e-8)).clip(-3, 3) except Exception: f["vol_imbalance"] = 0.0 # ----------------------------------------------------------------------- # v9: CORWIN-SCHULTZ BID-ASK SPREAD from OHLC data # ----------------------------------------------------------------------- try: _lnh = hi.apply(np.log) _lnl = lo.apply(np.log) _beta = (_lnh - _lnl)**2 + (_lnh.shift(1) - _lnl.shift(1))**2 _gamma = (np.maximum(_lnh, _lnh.shift(1)) - np.minimum(_lnl, _lnl.shift(1)))**2 _k_cs = 3.0 - 2.0**1.5 _alpha = (2**0.5 - 1) * (_beta**0.5) / _k_cs - (_gamma / _k_cs)**0.5 _spread = 2 * (np.exp(_alpha.rolling(5).mean()) - 1) / (1 + np.exp(_alpha.rolling(5).mean())) f["corwin_schultz"] = _spread.clip(0, 0.05).fillna(0.002) except Exception: f["corwin_schultz"] = 0.002 # ----------------------------------------------------------------------- # v9: SENTIMENT DECAY -- 30-min exponential half-life # Research: sentiment rank drops from 5 -> 18 over 2 lags # ----------------------------------------------------------------------- try: _raw = ( float(s.get("news_sentiment", 0.0)) * 1.0 + float(s.get("reddit_sentiment", 0.0)) * 0.4 + (float(s.get("stocktwits_bullish", 0.5)) - 0.5) * 2.0 * 0.6 + float(s.get("sec_flag", 0.0)) * 2.0 + float(s.get("congress_bought", 0.0)) * 1.5 ) / 5.0 _last_ts = float(s.get("last_update_ts", time.time() - 1800)) _age_min = (time.time() - _last_ts) / 60.0 _decay = 2 ** (-_age_min / 30.0) f["sentiment_decay"] = float(_raw * _decay) except Exception: f["sentiment_decay"] = 0.0 # ----------------------------------------------------------------------- # v9: ATR Z-SCORE -- volatility regime signal # ----------------------------------------------------------------------- try: _atr_s = f["atr_pct"] if "atr_pct" in f.columns else (hi - lo) / (c + 1e-8) _mu = _atr_s.rolling(20).mean() _sd = _atr_s.rolling(20).std().replace(0, 1) f["atr_z"] = ((_atr_s - _mu) / _sd).clip(-3, 3) except Exception: f["atr_z"] = 0.0 f = f.replace([np.inf, -np.inf], np.nan).dropna() return f if len(f) >= 30 else None # ============================================================================= # SENTIMENT ENGINE # ============================================================================= # The SentimentEngine is the intelligence layer that reads the world outside # of pure price data. It runs four parallel pipelines: # # 1. Google News RSS — headline sentiment per ticker (free, no key) # 2. SEC EDGAR 8-K — official filings for material events (free, no key) # 3. Reddit — retail investor mention velocity + mood (free key) # 4. Congress Trades — STOCK Act disclosures via ProPublica (free, no key) # # Scores are stored in a separate SQLite database (sentiment_data.db) so # they persist across restarts. Each ticker gets a SentimentScore object # that the feature builder reads and adds to FEATURE_COLS as: # # news_sentiment — float in [-1, 1], negative=bearish, positive=bullish # news_count — int, number of articles in last 24h # sec_flag — int, 1 if negative 8-K filed in last 48h, -1 positive # reddit_mentions — int, mention count in last 1h # reddit_sentiment — float in [-1, 1] # congress_bought — int, 1 if Congress member bought in last 30 days # # The GBM models treat these as regular features and learn their predictive # weight automatically from historical price movements. No hardcoded rules. # ============================================================================= class SentimentDatabase: """ Lightweight SQLite store for sentiment scores. Separate from the price DB so sentiment writes never block trading writes. Schema: sentiment(symbol, source, score, count, flag, ts, raw_text) symbol — ticker e.g. "AAPL" source — "news" | "sec" | "reddit" | "congress" score — float [-1, 1] composite sentiment count — int number of items contributing to this score flag — int special signal flag (e.g. 1=negative 8-K, 2=positive) ts — int unix timestamp of when this score was computed raw_text— text most recent headline/title for debug inspection """ def __init__(self, db_path: str = SENTIMENT_DB_FILE): self.path = db_path self._lock = threading.Lock() # Persistent read connection — avoids opening/closing for every symbol # during rebuild_all_features (2000+ calls). check_same_thread=False # is safe because all reads use self._lock. self._conn = sqlite3.connect(self.path, check_same_thread=False) self._conn.execute("PRAGMA journal_mode=WAL") self._init_db() def _init_db(self): with sqlite3.connect(self.path) as conn: conn.execute(""" CREATE TABLE IF NOT EXISTS sentiment ( symbol TEXT NOT NULL, source TEXT NOT NULL, score REAL DEFAULT 0.0, count INTEGER DEFAULT 0, flag INTEGER DEFAULT 0, ts INTEGER NOT NULL, raw_text TEXT DEFAULT '', PRIMARY KEY (symbol, source) ) """) conn.execute("CREATE INDEX IF NOT EXISTS idx_sym ON sentiment(symbol)") conn.commit() def upsert(self, symbol: str, source: str, score: float, count: int, flag: int, raw_text: str = ""): """Insert or update a sentiment record for a symbol+source pair.""" ts = int(time.time()) with self._lock: with sqlite3.connect(self.path) as conn: conn.execute(""" INSERT INTO sentiment(symbol,source,score,count,flag,ts,raw_text) VALUES(?,?,?,?,?,?,?) ON CONFLICT(symbol,source) DO UPDATE SET score=excluded.score, count=excluded.count, flag=excluded.flag, ts=excluded.ts, raw_text=excluded.raw_text """, (symbol, source, round(score,4), count, flag, ts, raw_text[:500])) conn.commit() def get(self, symbol: str) -> dict: """ Returns a dict of all sentiment scores for a symbol. Keys: news_sentiment, news_count, sec_flag, reddit_mentions, reddit_sentiment, congress_bought Returns zeros/defaults if no data exists yet. """ with self._lock: try: rows = self._conn.execute( "SELECT source,score,count,flag FROM sentiment WHERE symbol=?", (symbol,) ).fetchall() except sqlite3.OperationalError: # Reconnect if connection dropped self._conn = sqlite3.connect(self.path, check_same_thread=False) self._conn.execute("PRAGMA journal_mode=WAL") rows = [] result = { "news_sentiment": 0.0, "news_count": 0, "sec_flag": 0, "reddit_mentions": 0, "reddit_sentiment": 0.0, "congress_bought": 0, "stocktwits_bullish": 0.5, "stocktwits_mentions": 0, } for source, score, count, flag in rows: if source == "news": result["news_sentiment"] = score result["news_count"] = count elif source == "sec": result["sec_flag"] = flag elif source == "reddit": result["reddit_mentions"] = count result["reddit_sentiment"] = score elif source == "congress": result["congress_bought"] = flag elif source == "stocktwits": result["stocktwits_bullish"] = score result["stocktwits_mentions"] = count return result def get_all_symbols(self) -> list: with self._lock: try: rows = self._conn.execute( "SELECT DISTINCT symbol FROM sentiment" ).fetchall() return [r[0] for r in rows] except sqlite3.OperationalError: self._conn = sqlite3.connect(self.path, check_same_thread=False) self._conn.execute("PRAGMA journal_mode=WAL") return [] class SentimentEngine: """ Orchestrates all four sentiment data pipelines. Each pipeline runs in its own background daemon thread so they never block the main trading loop. Results are written to SentimentDatabase and read by build_features() when constructing feature vectors. Usage: engine = SentimentEngine(symbols=["AAPL","MSFT",...]) engine.start() # launches all background threads # Later, from feature builder: scores = engine.db.get("AAPL") """ def __init__(self, symbols: list): self.symbols = symbols self.db = SentimentDatabase() self._running = False self._vader = SentimentIntensityAnalyzer() if VADER_OK else None self._last_news = {} # symbol -> timestamp of last fetch self._last_sec = 0.0 # timestamp of last full SEC scan self._last_reddit = 0.0 self._last_congress = 0.0 self._last_stocktwits = {} # symbol -> timestamp of last fetch debug_logger.info(f"SENTIMENT_ENGINE_INIT | vader={VADER_OK} | " f"feedparser={FEEDPARSER_OK} | symbols={len(symbols)}") def start(self): """Launch all sentiment pipeline threads.""" self._running = True # News pipeline — staggered per-symbol polling threading.Thread(target=self._news_loop, daemon=True, name="SentNews").start() # SEC EDGAR pipeline — scans 8-K filings threading.Thread(target=self._sec_loop, daemon=True, name="SentSEC").start() # Reddit pipeline — mention velocity + sentiment if REDDIT_CLIENT_ID and REDDIT_CLIENT_SECRET: threading.Thread(target=self._reddit_loop, daemon=True, name="SentReddit").start() else: debug_logger.info("SENTIMENT_REDDIT_DISABLED | no credentials set") # Congress trades pipeline threading.Thread(target=self._congress_loop, daemon=True, name="SentCongress").start() # StockTwits pipeline -- free financial social sentiment, no key required # Authenticated at 400 req/hr with STOCKTWITS_TOKEN, else 200 req/hr threading.Thread(target=self._stocktwits_loop, daemon=True, name="SentStockTwits").start() logger.info(f"Sentiment engine started — " f"news={'ON' if FEEDPARSER_OK else 'OFF(no feedparser)'} | " f"sec=ON | " f"reddit={'ON' if REDDIT_CLIENT_ID else 'OFF(no key)'} | " f"congress=ON | " f"stocktwits=ON{'(auth)' if STOCKTWITS_TOKEN else ''}") def stop(self): self._running = False # ------------------------------------------------------------------ # PIPELINE 1: GOOGLE NEWS RSS # ------------------------------------------------------------------ # Google provides a free RSS feed for any search query. We query # "{TICKER} stock" which returns recent headlines from major financial # outlets. Each headline is scored with VADER (Valence Aware Dictionary # and sEntiment Reasoner), a lexicon-based tool tuned for social media # and financial text. We average scores across all recent articles, # weighting more recent articles higher (exponential decay). # No API key needed. Rate: ~1 request per 30 min per symbol. # ------------------------------------------------------------------ def _news_loop(self): """Cycles through all symbols, fetching news RSS for each.""" idx = 0 while self._running: try: sym = self.symbols[idx % len(self.symbols)] idx += 1 last = self._last_news.get(sym, 0) # Space out per-symbol fetches — don't hammer Google if time.time() - last > SENTIMENT_NEWS_INTERVAL: self._fetch_news(sym) self._last_news[sym] = time.time() # Sleep between symbols to stay gentle on Google's servers time.sleep(2.0) except Exception as e: debug_logger.error(f"SENTIMENT_NEWS_LOOP_ERR | {e}") time.sleep(30) def _fetch_news(self, symbol: str): """ Fetches Google News RSS for a single symbol and scores it. Steps: 1. Request RSS feed for "{symbol} stock" 2. Parse up to 20 recent entries (feedparser handles XML) 3. Score each headline with VADER -> compound score [-1, 1] 4. Apply time decay: articles > 12h old weighted at 50% 5. Weighted average -> single score for the symbol 6. Upsert into SentimentDatabase """ if not FEEDPARSER_OK or not VADER_OK: return try: url = GOOGLE_NEWS_RSS.format(symbol=symbol) feed = feedparser.parse(url) if not feed.entries: return scores = [] weights = [] latest = "" now_ts = time.time() for entry in feed.entries[:20]: title = entry.get("title", "") # Parse publication time for decay weighting published = entry.get("published_parsed") if published: pub_ts = time.mktime(published) age_h = (now_ts - pub_ts) / 3600.0 # Exponential decay: half-weight at 12 hours weight = max(0.1, 2 ** (-age_h / 12.0)) else: weight = 0.5 # unknown age = half weight vs = self._vader.polarity_scores(title) scores.append(vs["compound"]) weights.append(weight) if not latest: latest = title[:120] if len(scores) >= SENTIMENT_MIN_ARTICLES: total_w = sum(weights) wgt_score = sum(s * w for s, w in zip(scores, weights)) / total_w self.db.upsert(symbol, "news", score=wgt_score, count=len(scores), flag=0, raw_text=latest) debug_logger.debug( f"SENT_NEWS | {symbol} | score={wgt_score:.3f} | " f"articles={len(scores)} | top={latest[:60]}" ) except Exception as e: debug_logger.warning(f"SENT_NEWS_ERR | {symbol} | {e}") # ------------------------------------------------------------------ # PIPELINE 2: SEC EDGAR 8-K FILINGS # ------------------------------------------------------------------ # SEC EDGAR is the US Securities and Exchange Commission's public # database of all company filings. Form 8-K is a "current report" # that companies MUST file within 4 business days of any material # event — earnings, executive departures, mergers, lawsuits, etc. # These are the highest-quality signals because they are: # - Official and verified (not rumor) # - Filed before news outlets pick them up # - Completely free and unlimited via EDGAR's full-text search API # # We scan for 8-Ks mentioning each ticker in the last 48 hours, # then keyword-match the filing title to classify as positive/negative. # Result stored as sec_flag: 1=negative event, -1=positive, 0=nothing # ------------------------------------------------------------------ def _sec_loop(self): """Polls SEC EDGAR full-text search for recent 8-K filings.""" while self._running: try: if time.time() - self._last_sec > SENTIMENT_SEC_INTERVAL: self._last_sec = time.time() # Scan in batches of 10 symbols to avoid hammering EDGAR for i in range(0, len(self.symbols), 10): batch = self.symbols[i:i+10] for sym in batch: self._fetch_sec(sym) time.sleep(1.0) # 1s between EDGAR calls time.sleep(60) except Exception as e: debug_logger.error(f"SENTIMENT_SEC_LOOP_ERR | {e}") time.sleep(120) def _fetch_sec(self, symbol: str): """ Queries SEC EDGAR full-text search for recent 8-K filings. The EDGAR full-text search API allows searching all filings by company name or ticker. We look for 8-K forms in the last 48h. Steps: 1. Build EDGAR search URL for the symbol, last 48h, form=8-K 2. Parse JSON response for filing titles and dates 3. Keyword match title against positive/negative word lists 4. Assign sec_flag: 1=negative, -1=positive, 0=neutral/none 5. Upsert into SentimentDatabase """ try: from datetime import datetime, timedelta now = datetime.utcnow() start_str = (now - timedelta(hours=48)).strftime("%Y-%m-%d") end_str = now.strftime("%Y-%m-%d") url = (f"https://efts.sec.gov/LATEST/search-index" f"?q=%22{symbol}%22&forms=8-K" f"&dateRange=custom&startdt={start_str}&enddt={end_str}" f"&hits.hits._source=period_of_report,file_date,display_names") resp = requests.get(url, timeout=10, headers={"User-Agent": "StockTradingBot admin@example.com"}) if resp.status_code != 200: return data = resp.json() hits = data.get("hits", {}).get("hits", []) if not hits: # No recent 8-K — clear any stale flag self.db.upsert(symbol, "sec", score=0.0, count=0, flag=0, raw_text="") return # Analyze the most recent filing latest_src = hits[0].get("_source", {}) filing_title = str(latest_src).lower() flag = 0 raw = str(latest_src)[:200] # Keyword matching — negative events for kw in SEC_NEGATIVE_KEYWORDS: if kw in filing_title: flag = 1 # 1 = negative 8-K break # Keyword matching — positive events (only if not already negative) if flag == 0: for kw in SEC_POSITIVE_KEYWORDS: if kw in filing_title: flag = -1 # -1 = positive 8-K break self.db.upsert(symbol, "sec", score=0.0, count=len(hits), flag=flag, raw_text=raw) if flag != 0: polarity = "NEGATIVE" if flag == 1 else "POSITIVE" debug_logger.info( f"SENT_SEC | {symbol} | {polarity} 8-K | " f"filings={len(hits)} | {raw[:80]}" ) except Exception as e: debug_logger.warning(f"SENT_SEC_ERR | {symbol} | {e}") # ------------------------------------------------------------------ # PIPELINE 3: REDDIT MENTION VELOCITY # ------------------------------------------------------------------ # Reddit's r/wallstreetbets, r/stocks, and r/investing are the most # influential retail investor communities. A sudden spike in mentions # of a ticker — especially with positive sentiment — often precedes # retail-driven price moves (as seen with GME, AMC, etc.). # # We track TWO signals: # - mention_count: how many times the ticker appeared in the last 1h # - sentiment: average VADER score of post titles mentioning it # # The velocity (rate of change) is more predictive than raw count. # Requires a free Reddit developer account (reddit.com/prefs/apps). # Rate limit: 100 requests/minute on free tier. # ------------------------------------------------------------------ def _reddit_loop(self): """Scans configured subreddits for ticker mentions.""" while self._running: try: if time.time() - self._last_reddit > SENTIMENT_REDDIT_INTERVAL: self._last_reddit = time.time() self._fetch_reddit_mentions() time.sleep(60) except Exception as e: debug_logger.error(f"SENTIMENT_REDDIT_LOOP_ERR | {e}") time.sleep(120) def _fetch_reddit_mentions(self): """ Uses Reddit's JSON API (no PRAW library needed) to fetch recent posts from each subreddit, then scans titles for ticker mentions. Reddit's public API allows basic requests with a user agent. Steps: 1. Fetch /r/{subreddit}/new.json (100 posts, last ~1-2 hours) 2. Scan each post title for any of our ticker symbols 3. Score each matching title with VADER 4. Aggregate per-symbol: count + average sentiment 5. Upsert all symbols into SentimentDatabase """ if not VADER_OK: return sym_set = set(self.symbols) mentions = defaultdict(int) sent_sums = defaultdict(float) now_ts = time.time() cutoff_ts = now_ts - SENTIMENT_LOOKBACK_REDDIT headers = {"User-Agent": REDDIT_USER_AGENT} for sub in SENTIMENT_SUBREDDITS: try: url = f"https://www.reddit.com/r/{sub}/new.json?limit=100" resp = requests.get(url, headers=headers, timeout=10) if resp.status_code != 200: continue posts = resp.json().get("data", {}).get("children", []) for post in posts: data = post.get("data", {}) created = float(data.get("created_utc", 0)) if created < cutoff_ts: continue title = data.get("title", "") title_u = " " + title.upper() + " " # Check for ticker mention (word boundary match) for sym in sym_set: if f" {sym} " in title_u or f"${sym}" in title_u.replace(" $", " "): mentions[sym] += 1 vs = self._vader.polarity_scores(title) sent_sums[sym] += vs["compound"] time.sleep(1.0) # gentle on Reddit servers except Exception as e: debug_logger.warning(f"SENT_REDDIT_ERR | r/{sub} | {e}") # Write all found mentions to sentiment DB for sym, count in mentions.items(): avg_sent = sent_sums[sym] / count if count > 0 else 0.0 if count >= SENTIMENT_MIN_MENTIONS: self.db.upsert(sym, "reddit", score=avg_sent, count=count, flag=0, raw_text=f"{count} mentions") debug_logger.debug( f"SENT_REDDIT | {sym} | mentions={count} | " f"sentiment={avg_sent:.3f}" ) # ------------------------------------------------------------------ # PIPELINE 4: CONGRESS TRADES (STOCK ACT DISCLOSURES) # ------------------------------------------------------------------ # The STOCK Act (Stop Trading on Congressional Knowledge Act) requires # members of Congress to publicly disclose stock trades within 45 days. # Academic research shows that following Congress member trades — # particularly from members who sit on industry-relevant committees — # has historically produced above-market returns. # # WHY IT WORKS: Committee members receive non-public briefings about # industries they regulate. While trading on this information is # technically illegal, enforcement is rare and the signal persists. # # We use the Financial Modeling Prep API (free demo tier) which # aggregates Senate disclosure filings. When a Congress member buys # a stock we hold or are considering, congress_bought = 1. # ------------------------------------------------------------------ def _congress_loop(self): """Fetches Senate stock disclosures from ProPublica/FMP once daily.""" while self._running: try: if time.time() - self._last_congress > SENTIMENT_CONGRESS_INTERVAL: self._last_congress = time.time() self._fetch_congress_trades() time.sleep(3600) # check hourly if it's time to refresh except Exception as e: debug_logger.error(f"SENTIMENT_CONGRESS_LOOP_ERR | {e}") time.sleep(3600) def _fetch_congress_trades(self): """ Fetches Senate stock trading disclosures. Uses Financial Modeling Prep's free demo endpoint which aggregates senate disclosure JSON. Looks for BUY transactions in the last 30 days on any of our tracked symbols. Steps: 1. GET senate-disclosure endpoint (no key needed for demo) 2. Filter for transactions in last 30 days, type=Purchase 3. Extract ticker symbols that were purchased 4. Set congress_bought=1 for each purchased symbol 5. Clear flag for symbols NOT purchased (expire old signals) """ try: resp = requests.get(PROPUBLICA_TRADES, timeout=15, headers={"User-Agent": REDDIT_USER_AGENT}) if resp.status_code != 200: debug_logger.warning(f"SENT_CONGRESS_HTTP | {resp.status_code}") return trades = resp.json() cutoff_dt = datetime.utcnow() - timedelta(days=30) bought = set() for trade in trades: try: ticker = str(trade.get("ticker", "")).upper().strip() tx_type = str(trade.get("type", "")).lower() tx_date = trade.get("transactionDate", "") if not ticker or not tx_date: continue # Parse date — format varies: "2026-01-15" or "01/15/2026" for fmt in ("%Y-%m-%d", "%m/%d/%Y"): try: dt = datetime.strptime(tx_date, fmt) break except ValueError: dt = None if dt and dt >= cutoff_dt and "purchase" in tx_type: bought.add(ticker) except Exception: continue # Update sentiment DB for all tracked symbols sym_set = set(self.symbols) for sym in sym_set: flag = 1 if sym in bought else 0 self.db.upsert(sym, "congress", score=0.0, count=0, flag=flag, raw_text=f"congress_buy={flag}") debug_logger.info( f"SENT_CONGRESS | bought={len(bought)} tickers | " f"sample={list(bought)[:5]}" ) except Exception as e: debug_logger.warning(f"SENT_CONGRESS_ERR | {e}") # ------------------------------------------------------------------ # PIPELINE 5: STOCKTWITS FINANCIAL SOCIAL SENTIMENT # ------------------------------------------------------------------ # StockTwits is a financial social network where users explicitly tag # posts as Bullish or Bearish. This gives a cleaner signal than NLP # because the users do their own labeling. # # API: api.stocktwits.com/api/2/streams/symbol/{TICKER}.json # Rate: 200 req/hr unauthenticated, 400 req/hr with free token. # Key: Optional. Set STOCKTWITS_TOKEN env var for higher rate limit. # Get a free token at stocktwits.com/developers (instant, no approval). # # Features: # stocktwits_bullish -- fraction of tagged posts that are Bullish [0,1] # 0.5 = neutral, 0.7 = strongly bullish # stocktwits_mentions -- message count (velocity signal, spikes = news) # ------------------------------------------------------------------ def _stocktwits_loop(self): """Cycles through all symbols fetching StockTwits sentiment.""" idx = 0 while self._running: try: sym = self.symbols[idx % len(self.symbols)] idx += 1 last = self._last_stocktwits.get(sym, 0) if time.time() - last > SENTIMENT_STOCKTWITS_INTERVAL: self._fetch_stocktwits(sym) self._last_stocktwits[sym] = time.time() time.sleep(2) # ~30 symbols/min, well within rate limit except Exception as e: debug_logger.error(f"SENTIMENT_STOCKTWITS_LOOP_ERR | {e}") time.sleep(10) def _fetch_stocktwits(self, symbol: str): """Fetch bullish/bearish ratio from StockTwits for one symbol.""" try: url = f"https://api.stocktwits.com/api/2/streams/symbol/{symbol}.json" headers = {"User-Agent": "StockTradingBot/8.1"} params = {} if STOCKTWITS_TOKEN: params["access_token"] = STOCKTWITS_TOKEN resp = requests.get(url, headers=headers, params=params, timeout=8) if resp.status_code == 429: debug_logger.warning(f"SENT_STOCKTWITS_RATE_LIMIT | {symbol}") time.sleep(30) return if resp.status_code in (404, 422): return # symbol not on StockTwits -- normal for some tickers if resp.status_code != 200: return data = resp.json() messages = data.get("messages", []) if not messages: return # Count explicit Bullish / Bearish tags bull = sum(1 for m in messages if (m.get("entities") or {}).get("sentiment") and m["entities"]["sentiment"].get("basic") == "Bullish") bear = sum(1 for m in messages if (m.get("entities") or {}).get("sentiment") and m["entities"]["sentiment"].get("basic") == "Bearish") total_tagged = bull + bear # Bullish fraction: neutral default 0.5 when fewer than 3 tagged posts bullish_pct = (bull / total_tagged) if total_tagged >= 3 else 0.5 msg_count = len(messages) self.db.upsert(symbol, "stocktwits", score=round(bullish_pct, 4), count=msg_count, flag=0, raw_text=f"bull={bull} bear={bear} total={msg_count}") debug_logger.debug( f"SENT_STOCKTWITS | {symbol} | bullish={bullish_pct:.2f} | " f"bull={bull} bear={bear} msgs={msg_count}" ) except Exception as e: debug_logger.warning(f"SENT_STOCKTWITS_ERR | {symbol} | {e}") def get_scores(self, symbol: str) -> dict: """ Public interface for the feature builder. Returns all sentiment scores for a symbol as a flat dict. Values are always valid floats/ints — never None. """ return self.db.get(symbol) # ============================================================================= # DATA MANAGER # ============================================================================= # ============================================================================= # FEATURE FILE I/O -- lifetime learning storage # ============================================================================= def _save_feature_file(df: 'pd.DataFrame', stem: str) -> str: """Save a feature DataFrame to disk. Tries parquet first, falls back to pickle.gz.""" for ext, writer in [('.parquet', lambda p: df.to_parquet(p, index=False)), ('.pkl.gz', lambda p: df.to_pickle(p))]: try: path = stem + ext writer(path) return path except Exception: continue return '' def _load_feature_file(stem: str) -> 'pd.DataFrame | None': """Load a feature file from disk. Tries parquet then pickle.gz.""" for ext in ['.parquet', '.pkl.gz']: path = stem + ext if os.path.exists(path): try: if ext == '.parquet': return pd.read_parquet(path) else: return pd.read_pickle(path) except Exception as e: debug_logger.warning(f"FEATURE_FILE_LOAD_ERR | {path} | {e}") return None class DataManager: def __init__(self): # Daily DB: writes go to today's file, reads span all history. self._today_db_path = _today_db_path() self.db = Database(self._today_db_path) self._history_dbs = [] # read-only handles to older daily DBs self._load_history_dbs() self.fh = FinnhubClient(FINNHUB_API_KEY) self.av = AlphaVantageClient(ALPHA_VANTAGE_KEY) self.poly = PolygonClient(POLYGON_KEY) self.features = {} self.prices = {} self.velocity = {} # symbol -> price_velocity ($/s) from Alpaca self.sector_map = {} self.sector_perf = {} self.bad_symbols = set() self._rr_queue = list(STOCK_UNIVERSE) self._rr_idx = 0 self._av_queue = list(STOCK_UNIVERSE[:50]) self._av_idx = 0 # Alpaca WebSocket client -- starts in TradingSystem.initialize() self.alpaca_ws = AlpacaWebSocketClient(self.db, self.prices, self.velocity) # Sentiment engine -- starts background pipelines in initialize() self.sentiment = SentimentEngine(symbols=list(STOCK_UNIVERSE)) # v9: Incremental feature cache -- tracks last processed timestamp per symbol # Eliminates full-history recompute on every rebuild. Cost stays O(new_rows). self._feature_last_ts: dict = {} # symbol -> last bar ts processed self._feature_day_cache: dict = {} # symbol -> previous full trading day's raw bars # One trading day = ~390 bars (6.5 hrs x 60 bars/hr). # Used as rolling-window context so RSI/SMA/etc compute correctly on new bars. # Context is exactly one day -- semantically clean, not an arbitrary row count. self._TRADING_DAY_BARS = 400 # 390 market bars + 10 buffer # v9: Row count cache -- avoids querying all DB files every cycle self._row_count_cache: dict = {} self._row_count_cache_ts: float = 0.0 self._ROW_COUNT_TTL: float = 60.0 # seconds # v9: Max feature history per symbol -- caps memory growth self._MAX_FEATURE_ROWS = 200 # ~20 min of bars per symbol for live decisions # Training draws from features/daily/ disk files -- NOT this store. # 200 rows x 9,000 symbols x 56 features x 8 bytes = ~800MB max # vs 800 rows = ~3.2GB -- the difference is the OOM crash margin. def _load_history_dbs(self): """Open read-only handles to all daily DBs except today. Handles one-time migration from legacy trading_data.db on first run. Called at startup and after midnight rotation.""" import glob as _glob, shutil as _shutil # One-time migration: if old non-dated trading_data.db exists and # no dated files exist yet, copy it so history reads can find it. legacy = "trading_data.db" today_path = self._today_db_path dated_others = [p for p in _glob.glob(f"{DB_FILE_PREFIX}_????????.db") if p != today_path] if os.path.exists(legacy): # Find any dated file that is suspiciously small (< 1 MB) # which indicates a failed/partial migration. Delete and redo. for bad in dated_others[:]: if os.path.getsize(bad) < 1_000_000: try: for sfx in ["", "-wal", "-shm"]: if os.path.exists(bad + sfx): os.remove(bad + sfx) dated_others.remove(bad) logger.info(f"DB migration: removed incomplete {bad}") except Exception: pass # Migrate if no valid dated history exists yet if not dated_others: yesterday = (datetime.now() - timedelta(days=1)).strftime("%Y%m%d") migrated = f"{DB_FILE_PREFIX}_{yesterday}.db" # Checkpoint WAL so the copy is a complete snapshot try: import sqlite3 as _sq _c = _sq.connect(legacy) _c.execute("PRAGMA wal_checkpoint(TRUNCATE)") _c.close() except Exception: pass _shutil.copy2(legacy, migrated) for sfx in ["-wal", "-shm"]: if os.path.exists(legacy + sfx): _shutil.copy2(legacy + sfx, migrated + sfx) logger.info(f"DB migration: {legacy} -> {migrated} ({os.path.getsize(migrated)//1024//1024}MB)") # Load all dated history DBs (excludes today) self._history_dbs = [] today = self._today_db_path for path in _all_db_paths(): if path != today and os.path.exists(path): self._history_dbs.append(Database(path)) logger.info( f"Daily DB: today={os.path.basename(today)} | " f"history={len(self._history_dbs)} older DB(s)" ) def _rotate_db_if_needed(self): """Switch to a new daily DB if the calendar date has changed. Called at every market open so each trading day starts fresh.""" new_path = _today_db_path() if new_path == self._today_db_path: return logger.info(f"Daily DB rotation: {self._today_db_path} -> {new_path}") self._history_dbs.append(self.db) # move old DB to history self._today_db_path = new_path self.db = Database(new_path) self.alpaca_ws.db = self.db # point WS writer at new DB _all_db_paths() # trigger pruning of old files logger.info(f"Daily DB rotation complete.") # v9: On DB rotation, keep the feature cache but force row count refresh # so new symbols in today's DB get picked up immediately if new_path != self._today_db_path: self._row_count_cache_ts = 0.0 # force refresh debug_logger.info("DB_ROTATED | feature_cache_preserved | row_count_cache_cleared") def get_combined_series(self, symbol: str) -> pd.DataFrame: """Full price series across ALL daily DBs. Used for first-time builds only.""" frames = [] for hdb in self._history_dbs: df = hdb.get_series(symbol) if not df.empty: frames.append(df) df_today = self.db.get_series(symbol) if not df_today.empty: frames.append(df_today) if not frames: return pd.DataFrame() combined = pd.concat(frames, ignore_index=True) combined = combined.drop_duplicates(subset=["ts"], keep="last") return combined.sort_values("ts").reset_index(drop=True) def get_new_series(self, symbol: str, since_ts: float) -> pd.DataFrame: """ v9: Incremental query -- only rows newer than since_ts. Only queries today's DB and yesterday's DB (new bars can't appear in older history files). Dramatically faster than full-history query. """ frames = [] # Only need to check the last 2 DBs -- today and potentially yesterday dbs_to_check = ([self._history_dbs[-1]] if self._history_dbs else []) + [self.db] for db in dbs_to_check: try: conn = db._conn() df = pd.read_sql_query( "SELECT ts,price,open,high,low,prev_close,volume FROM quotes" " WHERE symbol=? AND ts>? ORDER BY ts ASC", conn, params=(symbol, since_ts) ) if not df.empty: frames.append(df) except Exception: pass if not frames: return pd.DataFrame() combined = pd.concat(frames, ignore_index=True) combined = combined.drop_duplicates(subset=["ts"], keep="last") return combined.sort_values("ts").reset_index(drop=True) def get_combined_row_counts(self, force: bool = False) -> dict: """ Row counts across ALL daily DBs. v9: Cached with 60s TTL -- avoids querying all DB files every cycle. Was O(symbols x DB_files) every call; now O(1) within TTL window. """ now = time.time() if not force and (now - self._row_count_cache_ts) < self._ROW_COUNT_TTL: return self._row_count_cache counts: dict = {} for hdb in self._history_dbs: for sym, cnt in hdb.row_counts().items(): counts[sym] = counts.get(sym, 0) + cnt for sym, cnt in self.db.row_counts().items(): counts[sym] = counts.get(sym, 0) + cnt self._row_count_cache = counts self._row_count_cache_ts = now return counts def load_sectors(self): logger.info("Loading sector info (skipping ETFs)...") for sym in STOCK_UNIVERSE[:200]: if sym in _ETF_SET: continue p = self.fh.profile(sym) if p: self.sector_map[sym] = p.get("finnhubIndustry", "Unknown") logger.info(f" Sectors loaded: {len(self.sector_map)}") def on_market_open(self): # Rotate to a fresh daily DB if the calendar date has changed self._rotate_db_if_needed() if self.bad_symbols: logger.info(f"Market open — clearing {len(self.bad_symbols)} bad_symbols") self.bad_symbols.clear() threading.Thread(target=self._bulk_polygon_fetch, daemon=True).start() def _bulk_polygon_fetch(self): if not POLYGON_KEY: return logger.info("Polygon burst fetch at market open...") for sym in STOCK_UNIVERSE[:60]: if sym in _ETF_SET: continue rows = self.poly.get_prev_day_minute_bars(sym) if rows: n = self.db.bulk_insert(rows) if n: self.rebuild_features_for(sym) def fetch_next_quote(self) -> bool: """ Strict round-robin Finnhub polling — covers all 300 universe stocks. Supplemental to Alpaca: Alpaca gives minute bars on all symbols, Finnhub gives sub-minute confirmation quotes for the active universe. """ eligible = [s for s in self._rr_queue if s not in self.bad_symbols] if not eligible: return False self._rr_idx = self._rr_idx % len(eligible) sym = eligible[self._rr_idx] self._rr_idx += 1 q = self.fh.quote(sym) if q is None: return False price = float(q.get("c", 0)) if price <= 0: return False ts = int(time.time()) open_ = float(q.get("o", price)) high = float(q.get("h", price)) low = float(q.get("l", price)) prev_close = float(q.get("pc", price)) volume = float(q.get("v") or 0) self.db.insert(sym, ts, price, open_, high, low, prev_close, volume) self.prices[sym] = price logger.debug(f"FH Quote {sym}: ${price:.2f}") return True def fetch_alpha_vantage_burst(self): sym = self._av_queue[self._av_idx % len(self._av_queue)] self._av_idx += 1 rows = self.av.get_intraday(sym, "5min") if rows: n = self.db.bulk_insert(rows) if n: self.rebuild_features_for(sym) def rebuild_features_for(self, symbol: str) -> bool: """ v9: INCREMENTAL feature rebuild. First call per symbol: full history query (unavoidable). Subsequent calls: only queries new bars since last processed timestamp. Uses last 100 rows as rolling-window context so RSI/SMA/etc compute correctly. Cost: O(new_rows) not O(total_rows). Daily rebuild stays fast regardless of how many months of history accumulate. """ sentiment = self.sentiment.get_scores(symbol) last_ts = self._feature_last_ts.get(symbol, 0.0) cached_feat = self.features.get(symbol) day_ctx = self._feature_day_cache.get(symbol) if last_ts == 0.0 or cached_feat is None or day_ctx is None: # --------------------------------------------------------------- # FIRST BUILD: full history query -- runs once per symbol lifetime # After this, _feature_day_cache holds yesterday's bars and all # future rebuilds are incremental day-by-day. # --------------------------------------------------------------- df = self.get_combined_series(symbol) if df.empty: return False feat = build_features(df, self.velocity, sentiment_scores=sentiment) if feat is None: return False # Store the last full trading day's raw bars as tomorrow's context. # One trading day = ~390 bars. This is the context window for all # rolling indicators going forward -- semantically one day, not N rows. self._feature_day_cache[symbol] = df.tail(self._TRADING_DAY_BARS).copy() self._feature_last_ts[symbol] = float(df["ts"].iloc[-1]) if len(feat) > self._MAX_FEATURE_ROWS: feat = feat.iloc[-self._MAX_FEATURE_ROWS:] self.features[symbol] = feat return True # ----------------------------------------------------------------------- # DAILY INCREMENTAL BUILD: yesterday's bars as context, today's as data # # Context = previous day's bars (up to ~390, stored in _feature_day_cache) # New data = only today's new bars from DB (ts > last_ts) # # The context provides the rolling-window lookback that RSI/SMA/etc need. # After processing, today's bars become tomorrow's context. # DB query cost is O(today's bars only) -- constant regardless of history. # ----------------------------------------------------------------------- new_df = self.get_new_series(symbol, since_ts=last_ts) if new_df.empty: # No new bars today -- just refresh sentiment columns and return if cached_feat is not None and not cached_feat.empty: s = sentiment or {} cached_feat["news_sentiment"] = float(s.get("news_sentiment", 0.0)) cached_feat["reddit_sentiment"] = float(s.get("reddit_sentiment", 0.0)) cached_feat["stocktwits_bullish"] = float(s.get("stocktwits_bullish", 0.5)) cached_feat["sec_flag"] = float(s.get("sec_flag", 0.0)) cached_feat["sentiment_decay"] = float( (float(s.get("news_sentiment", 0.0)) + float(s.get("reddit_sentiment", 0.0)) * 0.4) / 1.4 ) return True # Combine: previous day's raw bars + today's new bars combined_df = pd.concat([day_ctx, new_df], ignore_index=True) combined_df = combined_df.drop_duplicates(subset=["ts"], keep="last") combined_df = combined_df.sort_values("ts").reset_index(drop=True) feat_combined = build_features(combined_df, self.velocity, sentiment_scores=sentiment) if feat_combined is None: return False # Slice to only today's new feature rows # context rows = len(day_ctx), so new rows start after that boundary n_ctx = len(day_ctx) if len(feat_combined) > n_ctx: new_feat = feat_combined.iloc[n_ctx:].copy() else: # build_features dropped some rows -- take the tail new_feat = feat_combined.tail(len(new_df)).copy() if new_feat.empty: return False # Append today's features to cache, cap at MAX_FEATURE_ROWS if cached_feat is not None and not cached_feat.empty: updated = pd.concat([cached_feat, new_feat], ignore_index=True) updated = updated.drop_duplicates(keep="last") else: updated = new_feat if len(updated) > self._MAX_FEATURE_ROWS: updated = updated.iloc[-self._MAX_FEATURE_ROWS:] self.features[symbol] = updated self._feature_last_ts[symbol] = float(new_df["ts"].iloc[-1]) # Today's bars become tomorrow's context window # Keep exactly one trading day's worth (newest _TRADING_DAY_BARS rows) new_day_ctx = pd.concat([day_ctx, new_df], ignore_index=True) self._feature_day_cache[symbol] = new_day_ctx.tail( self._TRADING_DAY_BARS ).copy() return True def _get_rss_mb(self) -> int: """Current process RSS in MB from /proc/self/status.""" try: for line in open('/proc/self/status'): if line.startswith('VmRSS:'): return int(line.split()[1]) // 1024 except Exception: pass return 0 def rebuild_all_features(self) -> int: """ v9: Incremental rebuild. First run builds all symbols from scratch. Subsequent runs only process new bars -- stays fast as history grows. RAM guard: pauses rebuild if process RSS exceeds 5.5GB to prevent OOM kills. The Linux kernel OOM limit on this VM is ~7.3GB. """ # v9: Use --ram-cap ceiling at 90% if set, else 80% of system RAM if RAM_CAP_MB is not None: RAM_CEIL_MB = int(RAM_CAP_MB * 0.90) else: try: _sys_total_mb = int( open('/proc/meminfo').read().split('MemTotal:')[1].split()[0] ) // 1024 RAM_CEIL_MB = int(_sys_total_mb * 0.80) except Exception: RAM_CEIL_MB = 5500 # safe fallback counts = self.get_combined_row_counts() ok = 0 full = 0 incremental = 0 skipped_ram = 0 for sym, cnt in counts.items(): if cnt >= 30: # RAM guard + progress log every 500 symbols if ok % 500 == 0: rss = self._get_rss_mb() debug_logger.info( f"REBUILD_PROGRESS | symbols_done={ok} | " f"rss={rss}MB | ceil={RAM_CEIL_MB}MB" ) if rss > RAM_CEIL_MB: debug_logger.warning( f"FEATURE_REBUILD_RAM_GUARD | rss={rss}MB > {RAM_CEIL_MB}MB | " f"pausing 30s to let GC free memory" ) import gc; gc.collect() time.sleep(30) rss2 = self._get_rss_mb() if rss2 > RAM_CEIL_MB: debug_logger.warning( f"FEATURE_REBUILD_RAM_BAIL | rss={rss2}MB still high | " f"stopping rebuild early to prevent OOM" ) break was_cached = (sym in self._feature_last_ts and self._feature_last_ts[sym] > 0.0) if self.rebuild_features_for(sym): ok += 1 if was_cached: incremental += 1 else: full += 1 else: skipped_ram += 1 debug_logger.info( f"FEATURE_REBUILD_DONE | ok={ok} | " f"full_builds={full} | incremental={incremental} | " f"skipped={skipped_ram} | total_symbols={len(counts)}" ) _log_ram(f"rebuild_all_features complete | ok={ok}/{len(counts)}") return ok def update_sector_perf(self): sr = {} for sym, sec in self.sector_map.items(): feat = self.features.get(sym) if feat is not None and len(feat) >= 5: recent = feat["target_ret"].iloc[-10:].sum() sr.setdefault(sec, []).append(float(recent)) self.sector_perf = {s: float(np.mean(v)) for s, v in sr.items()} def save_daily_features(self) -> str: """ v9: Save today's feature rows to disk as a permanent record. Called after each EOD rebuild. Each file covers one trading day. These files accumulate forever and are used for: - Daily incremental training (50 new trees from today's data) - Weekend deep training (Mon-Fri combined) - Monthly distillation (full historical rebuild of compact model) Format: flat DataFrame with 'symbol' column + all FEATURE_COLS + targets. """ today = datetime.now(MARKET_TZ).strftime("%Y%m%d") frames = [] for sym, feat_df in self.features.items(): if feat_df is None or feat_df.empty: continue # Save the most recent trading day's worth of rows per symbol daily = feat_df.tail(self._TRADING_DAY_BARS).copy() daily["symbol"] = sym frames.append(daily) if not frames: debug_logger.warning("DAILY_FEATURES_SAVE | no data") return "" combined = pd.concat(frames, ignore_index=True) stem = os.path.join(FEATURES_DAILY_DIR, f"features_{today}") path = _save_feature_file(combined, stem) if path: logger.info( f"Daily features saved -> {path} " f"({len(combined):,} rows, {len(frames)} symbols)" ) debug_logger.info( f"DAILY_FEATURES_SAVED | {path} | " f"rows={len(combined)} | symbols={len(frames)}" ) return path def load_features_for_training(self, date_str: str = None) -> dict: """ Load a daily feature file as a symbol->DataFrame dict for training. date_str: YYYYMMDD string. Defaults to today. """ if date_str is None: date_str = datetime.now(MARKET_TZ).strftime("%Y%m%d") stem = os.path.join(FEATURES_DAILY_DIR, f"features_{date_str}") df = _load_feature_file(stem) if df is None or df.empty: return {} result = {} for sym, group in df.groupby("symbol"): result[sym] = group.drop(columns=["symbol"]).reset_index(drop=True) debug_logger.info( f"FEATURES_LOADED | date={date_str} | " f"symbols={len(result)} | rows={len(df)}" ) return result def load_distill_dataset(self, max_days: int = 90) -> dict: """ Load all available daily feature files for model distillation. Combines up to max_days of history into one comprehensive training set. Deduplicates on (symbol, ts) so overlapping saves don't inflate the dataset. This is the 'complete memory' dataset -- everything the model has ever seen, used to create a compact fresh model before incremental growth resumes. """ import glob as _glob files = sorted( _glob.glob(os.path.join(FEATURES_DAILY_DIR, "features_????????.parquet")) + _glob.glob(os.path.join(FEATURES_DAILY_DIR, "features_????????.pkl.gz")) ) files = files[-max_days:] # most recent max_days only if not files: debug_logger.warning("DISTILL_DATASET | no daily feature files found") return {} frames = [] for f in files: df = _load_feature_file(f.replace(".parquet","").replace(".pkl.gz","")) if df is not None and not df.empty: frames.append(df) if not frames: return {} combined = pd.concat(frames, ignore_index=True) if "ts" in combined.columns and "symbol" in combined.columns: combined = combined.drop_duplicates( subset=["symbol", "ts"], keep="last" ) combined = combined.sort_values(["symbol", "ts"]).reset_index(drop=True) if "ts" in combined.columns else combined logger.info( f"Distillation dataset: {len(combined):,} rows " f"from {len(files)} daily files ({max_days} day window)" ) result = {} for sym, group in combined.groupby("symbol"): result[sym] = group.drop(columns=["symbol"]).reset_index(drop=True) return result def build_weekly_dataset(self) -> dict: """ Build this week's combined training dataset from Mon-Fri daily files. Loading the full week's bars gives models a 5-day context window. Rolling features like ret_5, ret_10, slope_20 naturally capture weekly patterns when computed over 5 days of data vs 1 day. Called on Saturday during weekend deep train. Returns symbol->DataFrame dict ready for init_model training. """ from datetime import timedelta as _td today = datetime.now(MARKET_TZ) # Find Monday of the most recently completed trading week dow = today.weekday() # Mon=0 ... Fri=4 ... Sat=5 ... Sun=6 if dow == 5: # Saturday -- use this week (Mon-Fri just ended) days_back = 5 elif dow == 6: # Sunday -- same days_back = 6 else: # Weekday -- use Mon through today days_back = dow monday = today - _td(days=days_back) frames = [] loaded_days = 0 for i in range(5): day = monday + _td(days=i) if day.date() > today.date(): break date_str = day.strftime("%Y%m%d") day_data = self.load_features_for_training(date_str) for sym, df in day_data.items(): df = df.copy() df["symbol"] = sym frames.append(df) if day_data: loaded_days += 1 if not frames: debug_logger.warning("WEEKLY_DATASET | no daily files found for this week") return {} combined = pd.concat(frames, ignore_index=True) if "ts" in combined.columns: combined = combined.sort_values(["symbol", "ts"]).reset_index(drop=True) logger.info( f"Weekly dataset: {len(combined):,} rows " f"from {loaded_days} trading days" ) # Save combined weekly file as a record week_stem = os.path.join( FEATURES_WEEKLY_DIR, f"features_week_{monday.strftime('%Y%m%d')}" ) _save_feature_file(combined, week_stem) result = {} for sym, group in combined.groupby("symbol"): result[sym] = group.drop(columns=["symbol"]).reset_index(drop=True) return result def status_line(self) -> str: counts = self.get_combined_row_counts() total = sum(counts.values()) trainable = sum(1 for n in counts.values() if n >= MIN_ROWS_TO_TRAIN) tradeable = sum(1 for n in counts.values() if n >= MIN_ROWS_TO_TRADE) fh_calls = self.fh.calls_this_minute() alpaca_bars = self.alpaca_ws.bars_received() hist_count = len(self._history_dbs) return ( f"DB {self.db.size_mb():.1f}MB (+{hist_count}hist) | total_quotes={total:,} | " f"symbols_in_db={len(counts)} | trainable={trainable} | " f"tradeable={tradeable} | fh_calls/min={fh_calls} | " f"alpaca_bars={alpaca_bars:,}" ) # ============================================================================= # ML MODEL # ============================================================================= class StockModel: """ v9: LightGBM-backed model replacing sklearn GBM. Key improvements: - LightGBM releases the Python GIL during C++ prediction -- safe for ThreadPoolExecutor - 5-10x faster training than sklearn GBM - Incremental training via init_model (appends 50 trees per cycle, no full retrain) - Native feature importance tracking (gain-based, per retrain cycle) - PSI-based drift detection - Rolling Information Coefficient (IC) for agent quality ranking and pruning """ def __init__(self, name: str, cfg: dict = None): self.name = name cfg = cfg or {} self.trained = False self.train_acc = 0.0 self.val_acc = 0.0 self.preds: list = [] # v9: IC tracking for agent pruning (Spearman rank correlation of predictions vs outcomes) self.ic_history: list = [] # v9: Feature gain importance tracked across retrains self.feat_importance: dict = {} # v9: PSI drift baseline -- prediction distribution at first full train self._pred_hist_baseline: np.ndarray = np.array([]) self._train_count = 0 # LightGBM booster objects (None until trained) self.lgb_clf = None self.lgb_reg = None # Hyperparams from agent config self.n_estimators = cfg.get("n_estimators", 300) self.learning_rate = cfg.get("learning_rate", 0.04) self.max_depth = cfg.get("max_depth", 4) self.num_leaves = cfg.get("num_leaves", 31) self.feature_fraction = cfg.get("feature_fraction", 0.8) # Sklearn GBM fallback when lightgbm not installed if not LGB_OK: ne, lr, md = self.n_estimators, self.learning_rate, self.max_depth self.clf = GradientBoostingClassifier( n_estimators=ne, learning_rate=lr, max_depth=md, subsample=0.8, min_samples_leaf=20, random_state=42, n_iter_no_change=20, validation_fraction=0.15, ) self.reg = GradientBoostingRegressor( n_estimators=ne, learning_rate=lr, max_depth=md, subsample=0.8, min_samples_leaf=20, random_state=42, n_iter_no_change=20, validation_fraction=0.15, ) else: self.clf = None self.reg = None self.scaler = StandardScaler() def _lgb_params_clf(self) -> dict: return { "objective": "binary", "metric": "binary_logloss", "num_leaves": self.num_leaves, "max_depth": self.max_depth, "learning_rate": self.learning_rate, "feature_fraction": self.feature_fraction, "bagging_fraction": 0.8, "bagging_freq": 5, "min_child_samples": 20, "verbosity": -1, "num_threads": 1, # CRITICAL: 1 per call; ThreadPool provides parallelism "random_state": 42, } def _lgb_params_reg(self) -> dict: p = self._lgb_params_clf() p["objective"] = "regression" p["metric"] = "rmse" return p def train(self, features: dict, weights: dict = None) -> bool: """ v9: Incremental LightGBM training. First call: full train (n_estimators trees). Subsequent calls: appends 50 new trees via init_model. """ Xp, Ydp, Yrp, Wp = [], [], [], [] for sym, feat in features.items(): if len(feat) < MIN_ROWS_TO_TRAIN: continue Xp.append(feat[FEATURE_COLS].values) Ydp.append(feat["target_dir"].values) Yrp.append(feat["target_ret"].values) w = float(weights.get(sym, 1.0)) if weights else 1.0 Wp.append(np.full(len(feat), w)) if not Xp: logger.warning(f" [{self.name}] No trainable features yet.") return False X = np.vstack(Xp).astype(np.float32) Yd = np.concatenate(Ydp) Yr = np.concatenate(Yrp) W = np.concatenate(Wp) Xs = self.scaler.fit_transform(X) Xtr, Xv, Ydt, Ydv, Yrt, Yrv, Wt, _ = train_test_split( Xs, Yd, Yr, W, test_size=0.2, random_state=42 ) if LGB_OK: is_incremental = (self.lgb_clf is not None and self._train_count > 0) params_clf = self._lgb_params_clf() params_clf["n_estimators"] = getattr(self, '_override_trees', 50 if is_incremental else self.n_estimators) params_reg = self._lgb_params_reg() params_reg["n_estimators"] = getattr(self, '_override_trees', 50 if is_incremental else self.n_estimators) # v9: free_raw_data=True -- releases numpy arrays after LightGBM # builds internal histograms. Saves ~200MB per agent during training. ds_tr_clf = lgb.Dataset(Xtr, label=Ydt.astype(int), weight=Wt, free_raw_data=True) ds_val_clf = lgb.Dataset(Xv, label=Ydv.astype(int), reference=ds_tr_clf, free_raw_data=True) ds_tr_reg = lgb.Dataset(Xtr, label=Yrt, weight=Wt, free_raw_data=True) ds_val_reg = lgb.Dataset(Xv, label=Yrv, reference=ds_tr_reg, free_raw_data=True) cbs = [lgb.early_stopping(20, verbose=False), lgb.log_evaluation(-1)] self.lgb_clf = lgb.train( params_clf, ds_tr_clf, valid_sets=[ds_val_clf], callbacks=cbs, init_model=self.lgb_clf if is_incremental else None, keep_training_booster=True, ) self.lgb_reg = lgb.train( params_reg, ds_tr_reg, valid_sets=[ds_val_reg], callbacks=cbs, init_model=self.lgb_reg if is_incremental else None, keep_training_booster=True, ) val_proba = self.lgb_clf.predict(Xv) val_preds = (val_proba > 0.5).astype(int) tr_proba = self.lgb_clf.predict(Xtr) tr_preds = (tr_proba > 0.5).astype(int) self.train_acc = float(accuracy_score(Ydt, tr_preds)) self.val_acc = float(accuracy_score(Ydv, val_preds)) # Feature importance (gain-based) raw_imp = self.lgb_clf.feature_importance(importance_type="gain") total = raw_imp.sum() + 1e-10 self.feat_importance = { FEATURE_COLS[i]: float(raw_imp[i] / total) for i in range(min(len(FEATURE_COLS), len(raw_imp))) } # Drift baseline on first full train if not is_incremental: self._pred_hist_baseline = val_proba # Rolling IC try: from scipy.stats import spearmanr as _sp ic_val, _ = _sp(val_proba, Yrv) if not np.isnan(ic_val): self.ic_history.append(float(ic_val)) if len(self.ic_history) > 100: self.ic_history.pop(0) except Exception: pass self._train_count += 1 mode = "INCREMENTAL" if is_incremental else "FULL" logger.info( f" [{self.name}] {mode} train={self.train_acc:.3f} " f"val={self.val_acc:.3f} n={len(X):,} IC={self.rolling_ic():.3f}" ) else: # Sklearn fallback self.clf.fit(Xtr, Ydt, sample_weight=Wt) self.reg.fit(Xtr, Yrt, sample_weight=Wt) self.train_acc = float(accuracy_score(Ydt, self.clf.predict(Xtr))) self.val_acc = float(accuracy_score(Ydv, self.clf.predict(Xv))) logger.info(f" [{self.name}] train={self.train_acc:.3f} val={self.val_acc:.3f} n={len(X):,}") self.trained = True return True def release_training_data(self): """ v9: Fully release booster memory after save() so the OS reclaims it. The problem: LightGBM allocates at the C level (glibc heap). Python's gc.collect() and even lgb.free_dataset() don't return those pages to the OS -- glibc holds them in its own pool for reuse. After training 32 agents sequentially this leaves 6+ GB of unreturned heap even though the Python objects are gone. The fix: 1. Null lgb_clf/lgb_reg entirely -- Python releases its reference 2. gc.collect() -- frees Python-level cyclic refs 3. malloc_trim(0) via ctypes -- tells glibc to return free heap pages back to the OS immediately. This is the key step. Without it the RSS stays high even after gc.collect(). The booster is reloaded from pkl on the next predict() call or the next training cycle's init_model. The pkl on disk is the source of truth -- nulling RAM is safe. """ import gc, ctypes # Step 1: null the boosters -- drop Python references self.lgb_clf = None self.lgb_reg = None # Step 2: Python GC -- clears cyclic refs gc.collect() # Step 3: Tell glibc to return freed heap pages to the OS # malloc_trim(0) = release all free pages above program break # This is what actually drops RSS. No-op on non-glibc systems. try: ctypes.CDLL("libc.so.6").malloc_trim(0) except Exception: pass @property def tree_count(self) -> int: """Total trees in the LightGBM model. Grows with each incremental update.""" if LGB_OK and self.lgb_clf is not None: return self.lgb_clf.num_trees() return 0 def distill(self, all_features: dict) -> bool: """ v9: Monthly distillation -- compact the accumulated trees. When tree_count exceeds DISTILL_THRESHOLD (~44 days of growth): 1. Train a fresh 300-tree model on ALL historical feature data 2. This compact model encodes everything seen so far, efficiently 3. Daily incremental updates resume on top of the compact base 4. The agent 'evolves' -- same knowledge, leaner structure, room to grow Over years this creates models with genuine multi-year market understanding. Each distillation cycle is smarter than the last because the training dataset grows richer with every passing month. """ old_count = self.tree_count logger.info( f" [{self.name}] DISTILLATION START | " f"current_trees={old_count} | threshold={DISTILL_THRESHOLD} | " f"training on {len(all_features)} symbols" ) # Temporarily clear model so train() does a full fresh build saved_clf = self.lgb_clf saved_reg = self.lgb_reg self.lgb_clf = None self.lgb_reg = None # Full retrain on all historical data with compact tree count success = self.train(all_features) if not success: # Restore backup -- don't lose existing model on distill failure self.lgb_clf = saved_clf self.lgb_reg = saved_reg logger.warning(f" [{self.name}] DISTILLATION FAILED -- kept old model") return False new_count = self.tree_count logger.info( f" [{self.name}] DISTILLATION COMPLETE | " f"{old_count} trees -> {new_count} (compact) | " f"val={self.val_acc:.3f} IC={self.rolling_ic():.3f}" ) debug_logger.info( f"DISTILL_DONE | {self.name} | " f"old_trees={old_count} | new_trees={new_count}" ) return True def rolling_ic(self, n: int = 20) -> float: """Rolling Information Coefficient -- key metric for agent pruning.""" if len(self.ic_history) < 3: return 0.0 return float(np.mean(self.ic_history[-n:])) def compute_psi(self, current_proba: np.ndarray, n_bins: int = 10) -> float: """ Population Stability Index -- detects prediction distribution drift. PSI < 0.10: stable | 0.10-0.25: investigate | > 0.25: retrain now """ if len(self._pred_hist_baseline) < 50 or len(current_proba) < 10: return 0.0 try: bins = np.linspace(0, 1, n_bins + 1) base = np.histogram(self._pred_hist_baseline, bins=bins)[0] / len(self._pred_hist_baseline) curr = np.histogram(current_proba, bins=bins)[0] / len(current_proba) base = np.clip(base, 1e-4, None) curr = np.clip(curr, 1e-4, None) return float(np.sum((curr - base) * np.log(curr / base))) except Exception: return 0.0 def predict(self, x: np.ndarray): if not self.trained: return None, None, None try: xs = self.scaler.transform(x.reshape(1, -1)).astype(np.float32) if LGB_OK and self.lgb_clf is not None: cf = float(self.lgb_clf.predict(xs)[0]) pct = float(self.lgb_reg.predict(xs)[0]) d = 1 if cf > 0.5 else 0 return d, pct, cf elif self.clf is not None: d = int(self.clf.predict(xs)[0]) pct = float(self.reg.predict(xs)[0]) cf = float(np.max(self.clf.predict_proba(xs)[0])) return d, pct, cf return None, None, None except ValueError as e: if "features" in str(e): n_exp = getattr(self.scaler, "n_features_in_", "?") debug_logger.warning( f"[{self.name}] Feature mismatch (model={n_exp} vs input={len(x)}) " f"-- neutral. Delete models/ to retrain.") return None, None, None raise def predict_batch(self, feat_matrix: np.ndarray): """ v9: Batch predict -- all symbols in one C call. LightGBM releases GIL during predict(), safe for ThreadPoolExecutor. """ n = len(feat_matrix) _zd = np.zeros(n, dtype=np.int8) _zc = np.full(n, 0.5, dtype=np.float32) if not self.trained: return _zd, _zc try: if feat_matrix.shape[1] != N_FEATURES: return _zd, _zc sc = self.scaler.transform(feat_matrix).astype(np.float32) if LGB_OK and self.lgb_clf is not None: cf = self.lgb_clf.predict(sc).astype(np.float32) return (cf > 0.5).astype(np.int8), cf elif self.clf is not None: pr = self.clf.predict_proba(sc) cf = pr[:, 1].astype(np.float32) return (cf > 0.5).astype(np.int8), cf return _zd, _zc except Exception as e: debug_logger.debug(f"PREDICT_BATCH_ERR | {e}") return _zd, _zc def record(self, predicted_pct: float, actual_pct: float): self.preds.append({"predicted_pct": predicted_pct, "actual_pct": actual_pct}) def avg_discrepancy(self): if not self.preds: return None diffs = [abs(p["predicted_pct"] - p["actual_pct"]) * 100.0 for p in self.preds] return float(np.mean(diffs)) def last_n_discrepancy(self, n: int = 20): if not self.preds: return None recent = self.preds[-n:] diffs = [abs(p["predicted_pct"] - p["actual_pct"]) * 100.0 for p in recent] return float(np.mean(diffs)) def save(self): path = os.path.join(MODEL_SAVE_DIR, f"{self.name}.pkl") joblib.dump({ "lgb_clf": self.lgb_clf, "lgb_reg": self.lgb_reg, "clf": self.clf, "reg": self.reg, "scaler": self.scaler, "trained": self.trained, "ta": self.train_acc, "va": self.val_acc, "preds": self.preds, "ic_history": self.ic_history, "feat_imp": self.feat_importance, "train_count": self._train_count, "pred_baseline": self._pred_hist_baseline, }, path) def load(self) -> bool: path = os.path.join(MODEL_SAVE_DIR, f"{self.name}.pkl") if not os.path.exists(path): return False try: d = joblib.load(path) self.lgb_clf = d.get("lgb_clf") self.lgb_reg = d.get("lgb_reg") self.clf = d.get("clf") self.reg = d.get("reg") self.scaler = d["scaler"] self.trained = d["trained"] self.train_acc = d["ta"] self.val_acc = d["va"] self.preds = d.get("preds", []) self.ic_history = d.get("ic_history", []) self.feat_importance = d.get("feat_imp", {}) self._train_count = d.get("train_count", 1) self._pred_hist_baseline = d.get("pred_baseline", np.array([])) logger.info( f" [{self.name}] Loaded (val={self.val_acc:.3f} " f"IC={self.rolling_ic():.3f} train_count={self._train_count})") return self.trained except Exception as e: logger.warning(f" [{self.name}] Load error: {e}") return False class MetaLearner(StockModel): def __init__(self): super().__init__("MetaLearner") self.profitable_symbols: dict = {} def absorb_agent_history(self, agents: list): counts: dict = {} for a in agents: for t in a.history: if t["pnl"] > 0: counts[t["symbol"]] = counts.get(t["symbol"], 0) + 1 self.profitable_symbols = counts def build_weights(self, features: dict) -> dict: weights = {} for sym in features: cnt = self.profitable_symbols.get(sym, 0) weights[sym] = 1.0 + min(cnt, 4) * 1.0 return weights def retrain_from_agents(self, agents: list, features: dict): self.absorb_agent_history(agents) w = self.build_weights(features) n_profitable = sum(1 for v in w.values() if v > 1.0) boosted = [(s, w[s]) for s in sorted(w, key=lambda x: -w[x]) if w[s] > 1.0][:10] logger.info(f" MetaLearner: {n_profitable} profitable symbols boosted") if boosted: logger.info(" Top boosts: " + " | ".join(f"{s}x{v:.0f}" for s, v in boosted)) self.train(features, weights=w) self.save() def predict_batch(self, feat_matrix: np.ndarray) -> np.ndarray: """Batch meta-confidence for all symbols. Returns shape (n_symbols,).""" if not self.trained or self.clf is None: return np.ones(len(feat_matrix), dtype=np.float32) try: scaled = self.scaler.transform(feat_matrix) proba = self.clf.predict_proba(scaled) return proba[:, 1].astype(np.float32) except Exception: return np.ones(len(feat_matrix), dtype=np.float32) def predict_batch(self, feat_matrix): if not self.trained or self.clf is None: return np.full(len(feat_matrix), 0.5, dtype=np.float32) try: sc = self.scaler.transform(feat_matrix) return self.clf.predict_proba(sc)[:, 1].astype(np.float32) except Exception: return np.ones(len(feat_matrix), dtype=np.float32) def meta_confidence(self, x: np.ndarray) -> float: if not self.trained: return 0.5 _, _, cf = self.predict(x) return cf if cf is not None else 0.5 # ============================================================================= # META SYSTEM (5 specialized MetaLearners + MetaSystem wrapper — 105 total agents) # ============================================================================= # Clone name map: base_name -> clone index (1-indexed, matches _BASE_PROFILES order) _CLONE_NAME_MAP = {_BASE_PROFILES[i][0]: i + 1 for i in range(len(_BASE_PROFILES))} def _clone_names(base_set): """Return clone names for a set of base agent names.""" return {f"Clone_{n}_{_CLONE_NAME_MAP[n]}" for n in base_set if n in _CLONE_NAME_MAP} # Base agent name sets per sub-MetaLearner _TECH_BASES = {"UltraConserv","Conservative","ModerateDefens","Moderate", "MeanReversion","Momentum","ModerateAggr","Aggressive", "HighConviction","Contrarian","Volatility","UltraAggress", "SectorRotation","LowFreq","HighFreq","Balanced"} _SENTIMENT_BASES = {"NewsSentiment","SocialMomentum","CongressTracker","SentimentDecay","SECWatcher"} _VOLATILITY_BASES = {"LowVolEnv","HighVolEnv","ATRScaled","VolImbalance","CorwinSchultz"} _TREND_BASES = {"TrendLong","TrendShort","SlopeAccel","MomentumAge", "ConsistencyTrader","VolRatioBurst","VolTrend","MoneyFlow"} class MetaTech(MetaLearner): """MetaLearner for the original 16 agents + their clones.""" AGENT_NAMES = _TECH_BASES | _clone_names(_TECH_BASES) def __init__(self): StockModel.__init__(self, "MetaTech") self.profitable_symbols: dict = {} class MetaSentiment(MetaLearner): """MetaLearner for sentiment agents 17-21 + their clones.""" AGENT_NAMES = _SENTIMENT_BASES | _clone_names(_SENTIMENT_BASES) def __init__(self): StockModel.__init__(self, "MetaSentiment") self.profitable_symbols: dict = {} class MetaVolatility(MetaLearner): """MetaLearner for volatility regime agents 32-36 + their clones.""" AGENT_NAMES = _VOLATILITY_BASES | _clone_names(_VOLATILITY_BASES) def __init__(self): StockModel.__init__(self, "MetaVolatility") self.profitable_symbols: dict = {} class MetaTrend(MetaLearner): """MetaLearner for trend + volume agents 37-44 + their clones.""" AGENT_NAMES = _TREND_BASES | _clone_names(_TREND_BASES) def __init__(self): StockModel.__init__(self, "MetaTrend") self.profitable_symbols: dict = {} class MetaComposite(MetaLearner): """MetaLearner that sees all 100 agents — highest authority.""" AGENT_NAMES = None # None = use all agents def __init__(self): StockModel.__init__(self, "MetaComposite") self.profitable_symbols: dict = {} class GrandMetaLearner: """ Sits on top of the 5 sub-MetaLearners. Input : 5-element confidence vector (one per sub-MetaLearner) Output : final go/no-go confidence score Before trained: equal-weight average of the 5 inputs. After trained : IC-proportional weighted average (updated each retrain cycle). """ MODEL_FILE = "models/GrandMetaLearner.pkl" def __init__(self): self.weights = np.array([0.20, 0.20, 0.20, 0.20, 0.20], dtype=np.float32) self.trained = False self.val_acc = 0.0 self.train_acc = 0.0 def predict_from_scores(self, scores: np.ndarray) -> np.ndarray: """scores: [N, 5] -> [N] weighted confidence.""" return (scores * self.weights).sum(axis=1).astype(np.float32) def update_weights(self, sub_metas: list): """Recompute weights proportional to each sub-MetaLearner's rolling IC.""" ics = [] for m in sub_metas: try: ic = m.rolling_ic() if (hasattr(m, "rolling_ic") and m.trained) else 0.0 except Exception: ic = 0.0 ics.append(max(ic, 0.0)) total = sum(ics) if total > 1e-9: self.weights = np.array([v / total for v in ics], dtype=np.float32) else: self.weights = np.array([0.20] * 5, dtype=np.float32) self.trained = True debug_logger.info( "GRAND_META_WEIGHTS | " + " | ".join(f"{m.__class__.__name__}={w:.3f}" for m, w in zip(sub_metas, self.weights)) ) def rolling_ic(self) -> float: return float(self.weights.max()) if self.trained else 0.0 def avg_discrepancy(self): return None def save(self): try: import pickle with open(self.MODEL_FILE, "wb") as f: pickle.dump({"weights": self.weights.tolist(), "trained": self.trained}, f) except Exception as e: debug_logger.warning(f"GRAND_META_SAVE_ERR | {e}") def load(self): try: import pickle with open(self.MODEL_FILE, "rb") as f: d = pickle.load(f) self.weights = np.array(d.get("weights", [0.20]*5), dtype=np.float32) self.trained = d.get("trained", False) except Exception: pass class MetaSystem: """ Wrapper around 5 specialized MetaLearners (no Grand layer). Exposes the same interface as the old single MetaLearner so the rest of the codebase needs minimal changes. Trade gate in _trading_pass(): Grand MetaLearner confidence >= 0.50 AND at least 2 of 5 sub-MetaLearners score > 0.50. Gate only activates once >= 2 sub-MetaLearners are trained. """ def __init__(self): self.tech = MetaTech() self.sentiment = MetaSentiment() self.volatility = MetaVolatility() self.trend = MetaTrend() self.composite = MetaComposite() self._sub_list = [self.tech, self.sentiment, self.volatility, self.trend, self.composite] # Backward-compat state (used in agent state save/load + reporting) self.preds: list = [] self.profitable_symbols: dict = {} # ── Backward-compat properties ────────────────────────────────────────── @property def trained(self) -> bool: return any(m.trained for m in self._sub_list) @property def val_acc(self) -> float: accs = [m.val_acc for m in self._sub_list if m.trained] return float(sum(accs) / len(accs)) if accs else 0.0 @property def train_acc(self) -> float: accs = [m.train_acc for m in self._sub_list if m.trained] return float(sum(accs) / len(accs)) if accs else 0.0 def rolling_ic(self) -> float: ics = [m.rolling_ic() for m in self._sub_list if m.trained] return float(sum(ics) / len(ics)) if ics else 0.0 def avg_discrepancy(self): vals = [m.avg_discrepancy() for m in self._sub_list if m.trained and m.avg_discrepancy() is not None] return float(sum(vals) / len(vals)) if vals else None @property def gate_active(self) -> bool: """Gate only enforced once >= 2 of the 5 sub-MetaLearners are trained.""" return sum(m.trained for m in self._sub_list) >= 2 # ── Core prediction ────────────────────────────────────────────────────── def _get_sub_scores(self, feat_matrix: np.ndarray) -> np.ndarray: """Returns [N, 5] — one confidence column per sub-MetaLearner.""" n = len(feat_matrix) scores = np.full((n, 5), 0.5, dtype=np.float32) for j, m in enumerate(self._sub_list): try: scores[:, j] = m.predict_batch(feat_matrix) except Exception: pass return scores def predict_batch(self, feat_matrix: np.ndarray) -> np.ndarray: """Returns mean confidence across the 5 sub-MetaLearners [N].""" return self._get_sub_scores(feat_matrix).mean(axis=1).astype(np.float32) def predict_batch_full(self, feat_matrix: np.ndarray): """ Returns: scores : [N, 5] per-sub-MetaLearner confidences mean_cf : [N] mean confidence across all 5 agree : [N] count of sub-MetaLearners scoring > 0.50 """ scores = self._get_sub_scores(feat_matrix) mean_cf = scores.mean(axis=1).astype(np.float32) agree = (scores > 0.50).sum(axis=1).astype(np.int32) return scores, mean_cf, agree def meta_confidence(self, x: np.ndarray) -> float: """Single-symbol confidence — mean of the 5 sub-MetaLearner scores.""" return float(self._get_sub_scores(x.reshape(1, -1)).mean()) # ── Training ───────────────────────────────────────────────────────────── def absorb_agent_history(self, agents: list): counts: dict = {} for a in agents: for t in a.history: if t["pnl"] > 0: counts[t["symbol"]] = counts.get(t["symbol"], 0) + 1 self.profitable_symbols = counts def retrain_from_agents(self, agents: list, features: dict): """Train all 5 sub-MetaLearners on the full agent pool (learn from each other).""" self.absorb_agent_history(agents) for m in self._sub_list: try: m.retrain_from_agents(agents, features) except Exception as e: debug_logger.error( f"META_RETRAIN_ERR | {m.__class__.__name__} | {e}" ) # ── Persistence ────────────────────────────────────────────────────────── def save(self): for m in self._sub_list: try: m.save() except Exception as e: debug_logger.warning(f"META_SAVE_ERR | {m.__class__.__name__} | {e}") def load(self): for m in self._sub_list: try: m.load() except Exception: pass def sub_meta_summary(self) -> list: return [ { "name": m.__class__.__name__, "trained": m.trained, "val_acc": round(m.val_acc, 4) if m.trained else None, "rolling_ic": round(m.rolling_ic(), 4) if m.trained else None, } for m in self._sub_list ] # ============================================================================= # PARALLEL TRAINING # ============================================================================= def _train_one(args): model, features, weights = args model.train(features, weights) model.save() return model def parallel_train(models: list, features: dict, weights: dict = None): _proc_alloc.refresh_avail_ram() n_safe = min(len(models), _proc_alloc.training_procs) logger.info(f"Parallel training {len(models)} models on {n_safe} processes...") debug_logger.info( f"PARALLEL_TRAIN_START | models={len(models)} | " f"n_procs={n_safe} | {_proc_alloc.status()}" ) args = [(m, features, weights) for m in models] # Use threading backend -- LightGBM releases the GIL during C-level # tree fitting so threads genuinely run in parallel for the heavy work. # loky (processes) requires full object pickling which fails on Oracle # Linux with some GBM configurations. Threading is safe and fast enough. # v9: ThreadPoolExecutor -- LightGBM releases GIL, threads work correctly # v9: Use dynamic worker count (updated by ResourceGuard based on live CPU/RAM/disk) # v9: honour --max-workers cap -- prevents OOM from too many concurrent # LightGBM training threads each holding large feature arrays in memory if MAX_WORKERS_CAP is not None: effective_workers = min(n_safe, _dynamic_workers, MAX_WORKERS_CAP) else: effective_workers = min(n_safe, _dynamic_workers) with ThreadPoolExecutor(max_workers=effective_workers) as pool: futures = {pool.submit(_train_one, a): i for i, a in enumerate(args)} trained_results = [None] * len(args) for fut in as_completed(futures): idx = futures[fut] try: trained_results[idx] = fut.result() except Exception as e: debug_logger.error(f"TRAIN_THREAD_ERR | idx={idx} | {e}") trained_results[idx] = None for orig, result in zip(models, trained_results): if result is None: continue if LGB_OK: orig.lgb_clf = result.lgb_clf orig.lgb_reg = result.lgb_reg orig.ic_history = result.ic_history orig.feat_importance = result.feat_importance orig._train_count = result._train_count orig._pred_hist_baseline = result._pred_hist_baseline else: orig.clf = result.clf; orig.reg = result.reg orig.scaler = result.scaler; orig.trained = result.trained orig.train_acc = result.train_acc; orig.val_acc = result.val_acc logger.info("Parallel training complete.") # ============================================================================= # TRADING AGENT # ============================================================================= class Position: __slots__ = ("symbol","shares","entry","ts","predicted_pct") def __init__(self, symbol, shares, entry, ts, predicted_pct=0.0): self.symbol = symbol self.shares = shares self.entry = entry self.ts = ts self.predicted_pct = predicted_pct def pnl(self, p): return (p - self.entry) * self.shares def pnl_pct(self, p): return (p / self.entry - 1) * 100 def value(self, p): return self.shares * p class ShortPosition: """A short (borrowed-and-sold) position. Profit = entry_price - current_price (inverse of long). Stored in agent.short_positions dict, parallel to agent.positions (longs). Capital locked = entry_price * shares (the proceeds from the short sale). """ __slots__ = ("symbol","shares","entry","ts","predicted_pct") def __init__(self, symbol, shares, entry, ts, predicted_pct=0.0): self.symbol = symbol self.shares = shares self.entry = entry self.ts = ts self.predicted_pct = predicted_pct def pnl(self, p): return (self.entry - p) * self.shares # profit when price falls def pnl_pct(self, p): return (self.entry / p - 1) * 100 # pct gain when price falls def value(self, p): return self.entry * self.shares # locked capital (constant) def risk_pct(self, p): return (p / self.entry - 1) * 100 # how much price has moved against us class _NeutralMeta: """ Dummy meta for the MetaLearner's own TradingAgent. The MetaLearner is its own model, so it doesn't need an external meta. Returns 0.5 (neutral) so the agent trades purely on its own signal. Effect: eff_cf = cf * 0.7 + 0.5 * 0.3 = cf * 0.7 + 0.15 """ trained = True preds = [] profitable_symbols = {} def meta_confidence(self, x) -> float: return 0.5 def predict_batch(self, feat_matrix): import numpy as _np return _np.full(len(feat_matrix), 0.5, dtype=_np.float32) class TradingAgent: def __init__(self, cfg: dict, model: StockModel, meta: MetaLearner): self.name = cfg["name"] self.risk_tol = cfg["risk_tolerance"] self.max_pos_pct = cfg["max_position_pct"] self.min_conf = cfg["min_confidence"] self.tp_mult = cfg.get("take_profit_mult", 1.3) self.is_clone = cfg.get("is_clone", False) self.model = model self.meta = meta self.capital = STARTING_CAPITAL self.positions: dict = {} # long positions (symbol -> Position) self.short_positions: dict = {} # short positions (symbol -> ShortPosition) self.history: list = [] self.resets = 0 self.trades = 0 self.wins = 0 self.gross_profit = 0.0 self.gross_loss = 0.0 self.short_gross_profit = 0.0 # separate P&L tracking for shorts self.short_gross_loss = 0.0 self.short_trades = 0 self.short_wins = 0 self.total_pnl = 0.0 self._last_sold: dict = {} self._pending_predicted: dict = {} self._pending_confs: dict = {} self.stop_loss_pct = DEFAULT_STOP_LOSS_PCT self.take_profit_pct = DEFAULT_TAKE_PROFIT_PCT self.max_positions = DEFAULT_MAX_POSITIONS # v9: Fractional Kelly tracking self._kelly_wins = 0 self._kelly_losses = 0 self._kelly_avg_win = 0.0 self._kelly_avg_loss = 0.0 # v9: ATR at entry per position for dynamic stops self._position_atr: dict = {} def params(self) -> dict: return { "stop_loss_pct": self.stop_loss_pct, "take_profit_pct": self.take_profit_pct, "max_positions": self.max_positions, "min_conf": self.min_conf, "max_pos_pct": self.max_pos_pct, } def adopt_params(self, donor: "TradingAgent"): lr = PARAM_LEARNING_RATE def nudge(cur, tgt, lo, hi): return max(lo, min(hi, cur + lr * (tgt - cur))) self.stop_loss_pct = nudge(self.stop_loss_pct, donor.stop_loss_pct, *STOP_LOSS_RANGE) self.take_profit_pct = nudge(self.take_profit_pct, donor.take_profit_pct, *TAKE_PROFIT_RANGE) self.max_positions = round(nudge(self.max_positions, donor.max_positions, *MAX_POSITIONS_RANGE)) self.min_conf = nudge(self.min_conf, donor.min_conf, *MIN_CONF_RANGE) self.max_pos_pct = nudge(self.max_pos_pct, donor.max_pos_pct, *MAX_POS_PCT_RANGE) # Bug 22 fix: clones hard-capped at 0.62 so param nudging never # pushes them above the tradeable confidence threshold if self.is_clone: self.min_conf = min(self.min_conf, 0.62) def total_equity(self, prices: dict) -> float: return self.capital + sum( p.value(prices.get(p.symbol, p.entry)) for p in self.positions.values() ) def drawdown(self, prices: dict) -> float: return max(0.0, 1.0 - self.total_equity(prices) / STARTING_CAPITAL) def decide(self, sym: str, feat: np.ndarray, price: float, allow_short: bool = False): """Decide action for a symbol. Returns: (action, predicted_pct, own_cf, meta_cf, eff_cf) Actions: BUY, SELL, SHORT, COVER, HOLD allow_short: if True, d==0 with no long position can trigger SHORT """ # IC gate: skip entirely if model's rolling IC is below minimum threshold. # Prevents noise from degraded models polluting the ensemble. # IC_MIN_VOTE_THRESHOLD is currently 0.015 -- below all current agents (0.028+) # but provides a safety net for future drift. Has no cost on healthy models. if hasattr(self.model, "rolling_ic") and len(self.model.ic_history) >= 3: if self.model.rolling_ic() < IC_MIN_VOTE_THRESHOLD: return "HOLD", 0.0, 0.0, 0.0, 0.0 d, pct, cf = self.model.predict(feat) if d is None: return "HOLD", 0.0, 0.0, 0.0, 0.0 meta_cf = self.meta.meta_confidence(feat) effective_cf = cf * 0.7 + meta_cf * 0.3 if effective_cf < self.min_conf: return "HOLD", 0.0, cf, meta_cf, effective_cf if d == 1: if sym in self.short_positions: # close short before going long return "COVER", pct, cf, meta_cf, effective_cf return "BUY", pct, cf, meta_cf, effective_cf if d == 0: if sym in self.positions: # sell existing long return "SELL", pct, cf, meta_cf, effective_cf if sym in self.short_positions: # already short, hold return "HOLD", 0.0, cf, meta_cf, effective_cf if allow_short: # no position, go short return "SHORT", pct, cf, meta_cf, effective_cf return "HOLD", 0.0, cf, meta_cf, effective_cf def _fractional_kelly(self) -> float: """Quarter-Kelly position sizing. Falls back to max_pos_pct if <10 trades.""" n = self._kelly_wins + self._kelly_losses if n < 10 or self._kelly_avg_win < 1e-6: return self.max_pos_pct win_rate = self._kelly_wins / n loss_rate = self._kelly_losses / n b = self._kelly_avg_win / max(self._kelly_avg_loss, 1e-6) f_star = max(0.0, min((b * win_rate - loss_rate) / b, 0.5)) f_quarter = f_star / 4.0 # Blend with max_pos_pct until 30 trades of history blend = min(1.0, n / 30.0) return blend * f_quarter + (1.0 - blend) * self.max_pos_pct def buy(self, sym: str, price: float, predicted_pct: float = 0.0, own_conf=0.0, meta_conf=0.0, eff_conf=0.0, atr_pct: float = 0.0): if sym in self.positions or price <= 0: return if len(self.positions) >= self.max_positions: return if time.time() - self._last_sold.get(sym, 0) < BUY_COOLDOWN_SECS: return # v9: Fractional Kelly position sizing kelly_frac = self._fractional_kelly() invest = min(self.capital * kelly_frac, self.capital * 0.90) if invest < 1.00: return shares = invest / price self.capital -= invest self.positions[sym] = Position(sym, shares, price, datetime.now(MARKET_TZ), predicted_pct) self._pending_predicted[sym] = predicted_pct self._pending_confs[sym] = (own_conf, meta_conf, eff_conf) self._position_atr[sym] = max(atr_pct, 0.005) logger.info( f" BUY [{self.name}] {sym} {shares:.4f}sh @${price:.2f} " f"pred={predicted_pct*100:+.2f}% own={own_conf:.3f} " f"meta={meta_conf:.3f} eff={eff_conf:.3f} kelly={kelly_frac:.3f}" ) def sell(self, sym: str, price: float, reason: str = "signal"): if sym not in self.positions: return pos = self.positions.pop(sym) pnl = pos.pnl(price) actual_pct = pos.pnl_pct(price) / 100.0 self.capital += pos.value(price) self.total_pnl += pnl self.trades += 1 if pnl > 0: self.wins += 1 self.gross_profit += pnl # v9: Kelly stat tracking self._kelly_wins += 1 nw = self._kelly_wins self._kelly_avg_win = (self._kelly_avg_win * (nw - 1) + abs(pnl)) / nw else: self.gross_loss += abs(pnl) self._kelly_losses += 1 nl = self._kelly_losses self._kelly_avg_loss = (self._kelly_avg_loss * (nl - 1) + abs(pnl)) / nl self._position_atr.pop(sym, None) self._last_sold[sym] = time.time() predicted_pct = self._pending_predicted.pop(sym, pos.predicted_pct) own_cf, meta_cf, eff_cf = self._pending_confs.pop(sym, (0.0, 0.0, 0.0)) self.model.record(predicted_pct, actual_pct) disc_pp = abs(predicted_pct - actual_pct) * 100.0 held_secs = int((datetime.now(MARKET_TZ) - pos.ts).total_seconds()) trade_rec = { "symbol": sym, "entry": pos.entry, "exit": price, "shares": pos.shares, "pnl": pnl, "pnl_pct": pos.pnl_pct(price), "predicted_pct": predicted_pct * 100.0, "disc_pp": disc_pp, "held_secs": held_secs, "agent": self.name, "reason": reason, "own_conf": own_cf, "meta_conf": meta_cf, "eff_conf": eff_cf, } self.history.append(trade_rec) logger.info( f" SELL [{self.name}] {sym} {reason} " f"pnl=${pnl:+.2f} ({pos.pnl_pct(price):+.1f}%) " f"disc={disc_pp:.2f}pp own={own_cf:.3f} meta={meta_cf:.3f}" ) write_trade_log({ "timestamp": datetime.now(MARKET_TZ).strftime("%Y-%m-%d %H:%M:%S"), "agent": self.name, "type": "SELL", "symbol": sym, "shares": pos.shares, "entry": pos.entry, "exit": price, "pnl": pnl, "pnl_pct": pos.pnl_pct(price), "predicted_pct": predicted_pct * 100.0, "disc_pp": disc_pp, "reason": reason, "held_secs": held_secs, "own_conf": own_cf, "meta_conf": meta_cf, "eff_conf": eff_cf, }) def short(self, sym: str, price: float, predicted_pct: float = 0.0, own_conf=0.0, meta_conf=0.0, eff_conf=0.0): """Open a short position. We borrow shares and sell them, hoping to buy them back cheaper. Profit = entry - exit price. Locked capital = entry_price * shares (SHORT_CAPITAL_RATIO of normal). STOP LOSS: cover if price rises SHORT_STOP_LOSS_PCT (5%) against us. """ if sym in self.short_positions or sym in self.positions or price <= 0: return # already have long or short on this symbol if len(self.positions) + len(self.short_positions) >= self.max_positions: return if time.time() - self._last_sold.get(sym + "_short", 0) < BUY_COOLDOWN_SECS: return # Use SHORT_CAPITAL_RATIO of what a long would invest invest = min(self.capital * self.max_pos_pct, self.capital * 0.90) invest *= SHORT_CAPITAL_RATIO if invest < 1.00: return shares = invest / price self.capital -= invest # lock the capital (returned on cover) self.short_positions[sym] = ShortPosition( sym, shares, price, datetime.now(MARKET_TZ), predicted_pct ) self._pending_predicted[sym + "_s"] = predicted_pct self._pending_confs[sym + "_s"] = (own_conf, meta_conf, eff_conf) logger.info( f" SHORT [{self.name}] {sym} {shares:.4f}sh @${price:.2f} " f"pred={predicted_pct*100:+.2f}% own={own_conf:.3f} " f"meta={meta_conf:.3f} eff={eff_conf:.3f}" ) def cover(self, sym: str, price: float, reason: str = "signal"): """Cover (close) a short position by buying back the borrowed shares. Profit if price fell, loss if price rose. """ if sym not in self.short_positions: return pos = self.short_positions.pop(sym) pnl = pos.pnl(price) # positive if price fell actual_pct = pos.pnl_pct(price) / 100.0 self.capital += pos.value(price) + pnl # return locked capital + profit/loss self.total_pnl += pnl self.short_trades += 1 if pnl > 0: self.short_wins += 1 self.short_gross_profit += pnl else: self.short_gross_loss += abs(pnl) self._last_sold[sym + "_short"] = time.time() predicted_pct = self._pending_predicted.pop(sym + "_s", pos.predicted_pct) own_cf, meta_cf, eff_cf = self._pending_confs.pop(sym + "_s", (0.0, 0.0, 0.0)) self.model.record(predicted_pct, actual_pct) disc_pp = abs(predicted_pct - actual_pct) * 100.0 held_secs = int((datetime.now(MARKET_TZ) - pos.ts).total_seconds()) trade_rec = { "symbol": sym, "entry": pos.entry, "exit": price, "shares": pos.shares, "pnl": pnl, "pnl_pct": pos.pnl_pct(price), "predicted_pct": predicted_pct * 100.0, "disc_pp": disc_pp, "held_secs": held_secs, "agent": self.name, "reason": reason, "own_conf": own_cf, "meta_conf": meta_cf, "eff_conf": eff_cf, } self.history.append(trade_rec) logger.info( f" COVER [{self.name}] {sym} {reason} " f"pnl=${pnl:+.2f} ({pos.pnl_pct(price):+.1f}%) " f"disc={disc_pp:.2f}pp own={own_cf:.3f} meta={meta_cf:.3f}" ) write_trade_log({ "timestamp": datetime.now(MARKET_TZ).strftime("%Y-%m-%d %H:%M:%S"), "agent": self.name, "type": "COVER", "symbol": sym, "shares": pos.shares, "entry": pos.entry, "exit": price, "pnl": pnl, "pnl_pct": pos.pnl_pct(price), "predicted_pct": predicted_pct * 100.0, "disc_pp": disc_pp, "reason": reason, "held_secs": held_secs, "own_conf": own_cf, "meta_conf": meta_cf, "eff_conf": eff_cf, }) def exit_checks(self, prices: dict): # Long position exits for sym in list(self.positions): p = prices.get(sym) if not p: continue pct = self.positions[sym].pnl_pct(p) # v9: ATR-based dynamic stop (2x ATR at entry, floored at 50% of agent stop) entry_atr = self._position_atr.get(sym, self.stop_loss_pct / 2.0) atr_stop_pct = min(entry_atr * 2.0 * 100, self.stop_loss_pct * 100) effective_stop = max(atr_stop_pct, self.stop_loss_pct * 50) # 50% floor if pct < -effective_stop: self.sell(sym, p, "stop_loss_atr") elif pct > self.take_profit_pct * 100 * self.tp_mult: self.sell(sym, p, "take_profit") # Short position exits (inverted: cover when price rises against us) for sym in list(self.short_positions): p = prices.get(sym) if not p: continue pos = self.short_positions[sym] risk = pos.risk_pct(p) # positive = price moved against us (up) gain = pos.pnl_pct(p) # positive = price fell (we profit) if risk > SHORT_STOP_LOSS_PCT * 100: # price rose 5% -- cover loss self.cover(sym, p, "short_stop_loss") elif gain > SHORT_TAKE_PROFIT_PCT * 100: # price fell 10% -- take profit self.cover(sym, p, "short_take_profit") def check_reset(self, prices: dict): dd = self.drawdown(prices) if dd >= DRAWDOWN_RESET_PCT: logger.warning(f"[{self.name}] DRAWDOWN RESET — {dd:.1%} lost.") self.capital = STARTING_CAPITAL for sym in list(self.positions): self.positions.pop(sym) self._last_sold[sym] = time.time() self._pending_predicted.pop(sym, None) self._pending_confs.pop(sym, None) self.resets += 1 def profit_rate(self) -> float: return self.wins / self.trades if self.trades else 0.0 def save_state(self, prices: dict = None) -> dict: """ Save agent state. Capital saved as TOTAL EQUITY (cash + mark-to-market value of open positions) so unrealized P&L is properly captured. On load, positions are NOT restored -- agent starts clean with correct equity. This prevents silent loss-forgiveness on bot restart. """ # Mark open positions to market and include in saved capital equity = self.capital if prices and self.positions: for sym, pos in self.positions.items(): cur = prices.get(sym, pos.entry) # fallback to entry if no price equity += pos.value(cur) # includes unrealized P&L elif self.positions: # No prices available -- use entry price (neutral, no loss forgiveness) for pos in self.positions.values(): equity += pos.value(pos.entry) # Same for short positions (return locked capital +/- unrealized P&L) if self.short_positions: for sym, pos in self.short_positions.items(): cur = prices.get(sym, pos.entry) if prices else pos.entry equity += pos.value(cur) + pos.pnl(cur) return { "capital": round(equity, 4), "total_pnl": self.total_pnl, "trades": self.trades, "wins": self.wins, "resets": self.resets, "gross_profit": self.gross_profit, "gross_loss": self.gross_loss, "last_sold": self._last_sold, "stop_loss_pct": self.stop_loss_pct, "take_profit_pct": self.take_profit_pct, "max_positions": self.max_positions, "min_conf": self.min_conf, "max_pos_pct": self.max_pos_pct, "model_preds": self.model.preds, # v9.1: positions no longer saved -- equity captured above "positions_closed_on_exit": [], } def load_state(self, state: dict): self.capital = state.get("capital", STARTING_CAPITAL) self.total_pnl = state.get("total_pnl", 0.0) self.trades = state.get("trades", 0) self.wins = state.get("wins", 0) self.resets = state.get("resets", 0) self.gross_profit = state.get("gross_profit", 0.0) self.gross_loss = state.get("gross_loss", 0.0) self._last_sold = {k: float(v) for k, v in state.get("last_sold", {}).items()} self.stop_loss_pct = state.get("stop_loss_pct", self.stop_loss_pct) self.take_profit_pct = state.get("take_profit_pct", self.take_profit_pct) self.max_positions = int(state.get("max_positions", self.max_positions)) self.min_conf = state.get("min_conf", self.min_conf) self.max_pos_pct = state.get("max_pos_pct", self.max_pos_pct) self.model.preds = state.get("model_preds", []) # v9.1: positions_closed_on_exit no longer used -- equity pre-captured in capital # Do NOT add back position capital here: that would double-count # the equity already baked into saved capital. self.capital = min(self.capital, STARTING_CAPITAL * 5) # sanity cap at 5x def summary(self, prices: dict) -> dict: disc = self.model.avg_discrepancy() roll = self.model.last_n_discrepancy(20) return { "agent": self.name, "is_clone": self.is_clone, "capital": round(self.capital, 2), "equity": round(self.total_equity(prices), 2), "pnl": round(self.total_pnl, 2), "trades": self.trades, "profit_rate": round(self.profit_rate(), 4), "avg_discrepancy": round(disc, 3) if disc is not None else None, "roll_discrepancy": round(roll, 3) if roll is not None else None, "resets": self.resets, "positions": len(self.positions), "drawdown": round(self.drawdown(prices), 4), "stop_loss": self.stop_loss_pct, "take_profit": self.take_profit_pct, "max_pos": self.max_positions, "min_conf": self.min_conf, "gross_profit": round(self.gross_profit, 2), "gross_loss": round(self.gross_loss, 2), } # ============================================================================= # KNOWLEDGE SHARING (Bug 21 fix: MetaLearner always trains) # ============================================================================= def share_knowledge(agents: list, meta: MetaLearner, features: dict): if len(agents) < 2: return # MetaLearner ALWAYS trains on all available data — Bug 21 fix. # It learns from every symbol regardless of profit status. meta.retrain_from_agents(agents, features) # Agent→agent sharing only when a profitable donor exists. ranked = sorted(agents, key=lambda a: a.total_pnl, reverse=True) if not ranked[0].history or ranked[0].total_pnl <= 0: logger.debug("Agent→agent share skipped — no profitable agents yet.") return n_top = max(1, len(ranked) // 4) top_agents = ranked[:n_top] bot_agents = ranked[n_top:] logger.info(f"Knowledge share: top {n_top} agents -> {len(bot_agents)} others") best, worst = ranked[0], ranked[-1] good = {t["symbol"] for t in best.history if t["pnl"] > 0} if good and features and best.wins > 0: w = {s: (3.0 if s in good else 1.0) for s in features} worst.model.train(features, weights=w) worst.model.save() logger.info(f" ML share: [{best.name}] -> [{worst.name}] {len(good)} symbols") profitable_top = [a for a in top_agents if a.total_pnl > 0] if not profitable_top: logger.debug(" Param share skipped — no profitable top agents.") return for agent in bot_agents: donor = random.choice(profitable_top) old_p = agent.params() agent.adopt_params(donor) new_p = agent.params() changed = [k for k in old_p if abs(float(old_p[k]) - float(new_p[k])) > 0.001] if changed: logger.info(f" Param share: [{donor.name}] -> [{agent.name}]: {', '.join(changed)}") # ============================================================================= # CHARTING # ============================================================================= def generate_accuracy_chart(agents: list, db: Database): _set_nice(10) # P3 LOW _resource_guard.yield_p3("chart_generation") """Two-page trading dashboard: Page 1 (overview.png): Portfolio summary, top/bottom agents by PnL, win rate bars. Page 2 (accuracy.png): Per-agent win rate over time + trade count (replaces the old 'discrepancy' grid which was an internal ML metric invisible to humans). What each chart shows: Win Rate: % of completed trades that were profitable (>50% = making money) Net PnL: total dollars gained/lost per agent since inception Profit Factor: gross_profit / gross_loss (>1.0 = system is net profitable) Trade count: how active each agent is (more trades = faster learning) """ import math as _math now_str = datetime.now(DISPLAY_TZ).strftime("%Y%m%d_%H%M") ts_now = int(time.time()) for a in agents: disc = a.model.avg_discrepancy() if disc is not None: db.log_accuracy(ts_now, a.name, disc, a.profit_rate(), a.trades, a.gross_profit, a.gross_loss) summaries = [{ "name": a.name, "is_clone": a.is_clone, "capital": a.capital, "pnl": a.total_pnl, "trades": a.trades, "win_rate": a.profit_rate() * 100, "gross_profit": a.gross_profit, "gross_loss": a.gross_loss, } for a in agents] if not summaries: return total_pnl = sum(s["pnl"] for s in summaries) total_trades = sum(s["trades"] for s in summaries) total_wins = sum(s["trades"] * s["win_rate"] / 100 for s in summaries) overall_wr = (total_wins / total_trades * 100) if total_trades > 0 else 0 total_capital = sum(s["capital"] for s in summaries) total_gross_p = sum(s["gross_profit"] for s in summaries) total_gross_l = sum(s["gross_loss"] for s in summaries) profit_factor = (total_gross_p / total_gross_l) if total_gross_l > 0 else float("inf") now_cst = datetime.now(DISPLAY_TZ).strftime("%Y-%m-%d %H:%M CST") BG, PANEL = "#0d1117", "#161b22" # ========================================================================= # PAGE 1: Overview dashboard # ========================================================================= fig1, axes1 = plt.subplots(2, 2, figsize=(16, 10), facecolor=BG) fig1.suptitle(f"Trading System Overview -- {now_cst}", color="white", fontsize=15, fontweight="bold", y=0.98) for ax in axes1.flat: ax.set_facecolor(PANEL) ax.tick_params(colors="gray", labelsize=9) for spine in ax.spines.values(): spine.set_edgecolor("#30363d") # Top-left: key metrics scoreboard ax = axes1[0, 0] ax.axis("off") pf_str = f"{profit_factor:.2f}x" if profit_factor != float("inf") else "inf" metrics = [ ("Total Capital", f"${total_capital:,.2f}", "#74c0fc"), ("Net PnL", f"${total_pnl:+,.2f}", "#51cf66" if total_pnl >= 0 else "#ff6b6b"), ("Win Rate", f"{overall_wr:.1f}%", "#51cf66" if overall_wr >= 50 else "#ff6b6b"), ("Profit Factor", pf_str, "#51cf66" if profit_factor >= 1.0 else "#ff6b6b"), ("Total Trades", f"{int(total_trades):,}", "#e9ecef"), ("Gross Profit", f"${total_gross_p:,.2f}", "#51cf66"), ("Gross Loss", f"${total_gross_l:,.2f}", "#ff6b6b"), ("Agents Running", f"{len(summaries)}", "#e9ecef"), ] for i, (label, value, color) in enumerate(metrics): y_pos = 0.92 - i * 0.12 ax.text(0.02, y_pos, label, transform=ax.transAxes, color="#8b949e", fontsize=11) ax.text(0.98, y_pos, value, transform=ax.transAxes, color=color, fontsize=13, fontweight="bold", ha="right") ax.set_title("Portfolio Summary", color="white", fontsize=11, pad=8) # Top-right: Top 8 agents by PnL ax = axes1[0, 1] sorted_agents = sorted(summaries, key=lambda x: x["pnl"], reverse=True) top8 = sorted_agents[:8] names8 = [s["name"].replace("Clone_", "C_") for s in top8] pnls8 = [s["pnl"] for s in top8] colors8 = ["#51cf66" if p >= 0 else "#ff6b6b" for p in pnls8] bars = ax.barh(range(len(names8)), pnls8, color=colors8, alpha=0.8) ax.set_yticks(range(len(names8))) ax.set_yticklabels(names8, fontsize=8, color="white") ax.axvline(0, color="#8b949e", linewidth=0.5, linestyle="--") ax.set_xlabel("Net PnL ($)", color="#8b949e", fontsize=9) ax.set_title("Top 8 Agents by Profit", color="white", fontsize=11, pad=8) for bar, pnl in zip(bars, pnls8): offset = 0.3 if pnl >= 0 else -0.3 ha = "left" if pnl >= 0 else "right" ax.text(bar.get_width() + offset, bar.get_y() + bar.get_height() / 2, f"${pnl:+.1f}", va="center", ha=ha, color="#e9ecef", fontsize=8) # Bottom-left: Win rate by agent (all agents with >= 5 trades) ax = axes1[1, 0] traded = sorted([s for s in summaries if s["trades"] >= 5], key=lambda x: x["win_rate"], reverse=True) names_wr = [s["name"].replace("Clone_", "C_") for s in traded] wrs = [s["win_rate"] for s in traded] colors_wr = ["#51cf66" if w >= 55 else "#74c0fc" if w >= 50 else "#ff6b6b" for w in wrs] ax.barh(range(len(names_wr)), wrs, color=colors_wr, alpha=0.8) ax.axvline(50, color="#ff6b6b", linewidth=1, linestyle="--", label="50% break-even") ax.axvline(55, color="#51cf66", linewidth=0.7, linestyle=":", label="55% target") ax.set_yticks(range(len(names_wr))) ax.set_yticklabels(names_wr, fontsize=7, color="white") ax.set_xlabel("Win Rate (%)", color="#8b949e", fontsize=9) ax.set_xlim(0, 100) ax.set_title("Win Rate by Agent (>= 5 trades)", color="white", fontsize=11, pad=8) ax.legend(fontsize=8, labelcolor="gray", facecolor=PANEL, edgecolor="#30363d") # Bottom-right: Bottom 8 agents by PnL ax = axes1[1, 1] bottom8 = sorted_agents[-8:][::-1] names_b = [s["name"].replace("Clone_", "C_") for s in bottom8] pnls_b = [s["pnl"] for s in bottom8] colors_b = ["#51cf66" if p >= 0 else "#ff6b6b" for p in pnls_b] bars_b = ax.barh(range(len(names_b)), pnls_b, color=colors_b, alpha=0.8) ax.set_yticks(range(len(names_b))) ax.set_yticklabels(names_b, fontsize=8, color="white") ax.axvline(0, color="#8b949e", linewidth=0.5, linestyle="--") ax.set_xlabel("Net PnL ($)", color="#8b949e", fontsize=9) ax.set_title("Bottom 8 Agents by PnL", color="white", fontsize=11, pad=8) for bar, pnl in zip(bars_b, pnls_b): offset = -0.3 if pnl < 0 else 0.3 ha = "right" if pnl < 0 else "left" ax.text(bar.get_width() + offset, bar.get_y() + bar.get_height() / 2, f"${pnl:+.1f}", va="center", ha=ha, color="#e9ecef", fontsize=8) plt.tight_layout(rect=[0, 0, 1, 0.96]) fname1 = os.path.join(CHARTS_DIR, f"overview_{now_str}.png") plt.savefig(fname1, dpi=120, bbox_inches="tight", facecolor=BG) plt.close(fig1) logger.info(f"Overview chart saved -> {fname1}") # ========================================================================= # PAGE 2: Per-agent win rate history (replaces discrepancy grid) # ========================================================================= agent_data = {} for a in agents: hist = db.get_accuracy_history(a.name, limit=60) if not hist.empty: agent_data[a.name] = hist.sort_values("ts") if not agent_data: logger.info("Detail chart skipped -- no history yet.") return n_agents = len(agent_data) cols = min(4, n_agents) rows = _math.ceil(n_agents / cols) fig2 = plt.figure(figsize=(cols * 5, rows * 3.5 + 2), facecolor=BG) fig2.suptitle(f"Agent Win Rate History -- {now_cst}", color="white", fontsize=14, fontweight="bold", y=0.99) gs2 = plt.matplotlib.gridspec.GridSpec( rows, cols, figure=fig2, hspace=0.7, wspace=0.4) axes2 = [fig2.add_subplot(gs2[i // cols, i % cols]) for i in range(n_agents)] agent_names = sorted(agent_data.keys()) for ax, name in zip(axes2, agent_names): hist = agent_data[name] times = pd.to_datetime(hist["ts"], unit="s") profit = hist["profit_rate"].values * 100 trades = hist["trades"].values ax.set_facecolor(PANEL) ax.tick_params(colors="gray", labelsize=7) for spine in ax.spines.values(): spine.set_edgecolor("#30363d") wr_color = "#51cf66" if (len(profit) == 0 or profit[-1] >= 50) else "#ff6b6b" ax.plot(times, profit, color=wr_color, linewidth=1.5) if len(profit) > 1: ax.fill_between(times, profit, 50, alpha=0.12, where=[p >= 50 for p in profit], color="#51cf66") ax.fill_between(times, profit, 50, alpha=0.12, where=[p < 50 for p in profit], color="#ff6b6b") ax.axhline(50, color="#8b949e", linewidth=0.5, linestyle="--") ax.set_ylim(0, 100) ax.set_ylabel("Win %", color=wr_color, fontsize=7) ax2 = ax.twinx() ax2.plot(times, trades, color="#74c0fc", linewidth=0.8, linestyle=":", alpha=0.5) ax2.set_ylabel("Trades", color="#74c0fc", fontsize=6) ax2.tick_params(colors="gray", labelsize=6) for spine in ax2.spines.values(): spine.set_edgecolor("#30363d") curr_wr = f"{profit[-1]:.0f}%" if len(profit) > 0 else "N/A" n_trades = int(trades[-1]) if len(trades) > 0 else 0 agent_pnl = next((s["pnl"] for s in summaries if s["name"] == name), 0) pnl_str = f"${agent_pnl:+.1f}" short = name.replace("Clone_", "C_") clone_m = "+" if any(a.name == name and a.is_clone for a in agents) else "" ax.set_title( f"{short}{clone_m}\nWin={curr_wr} PnL={pnl_str} n={n_trades}", color="white", fontsize=7.5, pad=3 ) ax.xaxis.set_major_formatter(plt.matplotlib.dates.DateFormatter("%H:%M")) ax.xaxis.set_tick_params(rotation=30) for i in range(len(agent_names), len(axes2)): axes2[i].set_visible(False) ranked = sorted( [(name, agent_data[name]["profit_rate"].iloc[-1] * 100) for name in agent_names if len(agent_data[name]) > 0], key=lambda x: x[1], reverse=True ) if ranked: fig2.text(0.01, 0.005, "Win rate ranking: " + " | ".join( f"{n.replace('Clone_','C_')} {r:.0f}%" for n, r in ranked[:10]), color="#8b949e", fontsize=7, va="bottom") fname2 = os.path.join(CHARTS_DIR, f"accuracy_{now_str}.png") plt.savefig(fname2, dpi=120, bbox_inches="tight", facecolor=BG) plt.close(fig2) logger.info(f"Detail chart saved -> {fname2}") # ============================================================================= # AGENT STATE PERSISTENCE # ============================================================================= def save_all_agent_state(agents: list, meta: MetaLearner, prices: dict = None, meta_agents=None): """Save agent state for all agents including MetaLearner agents.""" _set_nice(19) # P4: lowest OS scheduling priority _resource_guard.yield_p4("agent_state_save") state = {a.name: a.save_state(prices) for a in agents} if meta_agents: for _ma in meta_agents: state[_ma.name] = _ma.save_state(prices) state["__meta__"] = {"preds": meta.preds, "profitable_symbols": meta.profitable_symbols} with open(AGENT_STATE_FILE, "w") as f: json.dump(state, f, indent=2) logger.info(f"Agent state saved -> {AGENT_STATE_FILE}") def load_all_agent_state(agents: list, meta: MetaLearner, meta_agents=None): if not os.path.exists(AGENT_STATE_FILE): logger.info("No saved agent state — starting fresh.") return try: with open(AGENT_STATE_FILE) as f: all_state = json.load(f) for a in agents: if a.name in all_state: a.load_state(all_state[a.name]) logger.info(f" [{a.name}] Restored: capital=${a.capital:.2f} pnl=${a.total_pnl:+.2f}") if meta_agents: for _ma in meta_agents: if _ma.name in all_state: _ma.load_state(all_state[_ma.name]) logger.info(f" [{_ma.name}] Restored: capital=${_ma.capital:.2f} pnl=${_ma.total_pnl:+.2f}") meta_s = all_state.get("__meta__", {}) meta.preds = meta_s.get("preds", []) meta.profitable_symbols = meta_s.get("profitable_symbols", {}) logger.info("Agent state loaded.") except Exception as e: logger.warning(f"Could not load agent state: {e}") # ============================================================================= # REPORTING # ============================================================================= def _write_collect_only_report(dm, total_bars: int, n_syms: int, n_feat: int, ws_bars: int): """ v9: Human-readable report for --collect-only mode. Simpler than full trading report since no agents/trades to show. Written to logs/human_report.txt + logs/claude_handoff.json hourly. """ now_cst = datetime.now(DISPLAY_TZ).strftime("%Y-%m-%d %H:%M:%S CST") res = _get_vm_resources() mkt_open = is_market_open() mkt_str = "OPEN" if mkt_open else f"CLOSED (opens in {seconds_until_open()/3600:.1f}h)" W = 72 sep = "=" * W thin = "-" * W def bar(val, mx, w=20, f="█", e="░"): filled = int(round(val / max(mx, 1e-6) * w)) return f * filled + e * (w - filled) lines = [ sep, f" TRADING BOT --collect-only MODE | {now_cst}", f" Market: {mkt_str}", sep, "", " VM RESOURCES", thin, f" CPU {bar(res['cpu_pct'],100)} {res['cpu_pct']:5.1f}% " f"load={res['load_1m']:.2f}/{res['load_5m']:.2f}/{res['load_15m']:.2f}", f" RAM {bar(res['ram_pct'],100)} {res['ram_pct']:5.1f}% " f"{res['ram_used_mb']:,}MB / {res['ram_total_mb']:,}MB", f" DSK {bar(res['disk_pct'],100)} {res['disk_pct']:5.1f}% " f"{res['disk_used_gb']:.1f}GB / {res['disk_total_gb']:.1f}GB", f" Threads: {res['active_threads']} | Dynamic workers: {res['dynamic_workers']}", "", " DATA COLLECTION STATUS", thin, f" Bars collected (session) : {ws_bars:>12,}", f" Bars in DB (all time) : {total_bars:>12,}", f" Symbols tracked : {n_syms:>12,}", f" Symbols ready to trade : {n_feat:>12,}", f" DB bar rate (approx) : {total_bars // max(1, int(time.time() % 86400 or 3600)):>8,}/hr", "", " MODE: Data collection only -- no trades being made.", " Switch to --mode long when ready to begin trading.", "", sep, f" Generated: {now_cst} | Bot: v9", sep, ] try: with open("logs/human_report.txt", "w") as f: f.write("\n".join(lines)) logger.info("Human report (collect-only) -> logs/human_report.txt") except Exception as e: debug_logger.warning(f"COLLECT_HUMAN_REPORT_ERR | {e}") # Minimal handoff JSON for collect-only handoff = { "generated_at": now_cst, "generated_ts": time.time(), "bot_version": "v9", "mode": "collect_only", "market_open": mkt_open, "vm_resources": res, "dynamic_workers": _dynamic_workers, "n_features": N_FEATURES, "data": { "bars_session": ws_bars, "bars_total": total_bars, "n_symbols": n_syms, "n_tradeable": n_feat, }, } try: with open("logs/claude_handoff.json", "w") as f: json.dump(handoff, f, indent=2) logger.info("Claude handoff -> logs/claude_handoff.json") except Exception as e: debug_logger.warning(f"COLLECT_HANDOFF_ERR | {e}") def _get_vm_resources() -> dict: """Read live VM resource stats from /proc for reports and handoff.""" out = {} try: with open("/proc/loadavg") as f: parts = f.read().split() out["load_1m"] = float(parts[0]) out["load_5m"] = float(parts[1]) out["load_15m"] = float(parts[2]) out["cpu_pct"] = round(out["load_1m"] / max(PHYSICAL_CORES, 1) * 100, 1) except Exception: out.update({"load_1m": 0, "load_5m": 0, "load_15m": 0, "cpu_pct": 0}) try: with open("/proc/meminfo") as f: mi = dict(l.split(":") for l in f.read().splitlines() if ":" in l) out["ram_total_mb"] = int(mi["MemTotal"].split()[0]) // 1024 out["ram_avail_mb"] = int(mi["MemAvailable"].split()[0]) // 1024 out["ram_used_mb"] = out["ram_total_mb"] - out["ram_avail_mb"] out["ram_pct"] = round(out["ram_used_mb"] / max(out["ram_total_mb"], 1) * 100, 1) except Exception: out.update({"ram_total_mb": 0, "ram_avail_mb": 0, "ram_used_mb": 0, "ram_pct": 0}) try: import shutil total, used, free = shutil.disk_usage("/") out["disk_total_gb"] = round(total / 1e9, 1) out["disk_used_gb"] = round(used / 1e9, 1) out["disk_pct"] = round(used / total * 100, 1) except Exception: out.update({"disk_total_gb": 0, "disk_used_gb": 0, "disk_pct": 0}) out["active_threads"] = threading.active_count() out["dynamic_workers"] = _dynamic_workers return out def _write_human_report(agents: list, dm: DataManager, meta: MetaLearner, existing_lines: list = None): """ v9: Write logs/human_report.txt -- clean, phone-readable report. Covers: market status, VM health, agent rankings, active positions, recent trades, top signals, feature importance leaders. """ now_cst = datetime.now(DISPLAY_TZ).strftime("%Y-%m-%d %H:%M:%S CST") res = _get_vm_resources() mkt_open = is_market_open() mkt_str = "OPEN" if mkt_open else f"CLOSED (opens in {seconds_until_open()/3600:.1f}h)" prices = dm.prices W = 72 # column width for phone readability sep = "=" * W thin = "-" * W def bar_chart(val, max_val, width=20, fill="█", empty="░"): filled = int(round(val / max(max_val, 1e-6) * width)) return fill * filled + empty * (width - filled) lines = [] lines.append(sep) lines.append(f" TRADING BOT REPORT | {now_cst}") lines.append(f" Market: {mkt_str}") lines.append(sep) # VM Resources lines.append("") lines.append(" VM RESOURCES") lines.append(thin) cpu_bar = bar_chart(res["cpu_pct"], 100) ram_bar = bar_chart(res["ram_pct"], 100) dsk_bar = bar_chart(res["disk_pct"], 100) lines.append(f" CPU {cpu_bar} {res['cpu_pct']:5.1f}% load={res['load_1m']:.2f}/{res['load_5m']:.2f}/{res['load_15m']:.2f}") lines.append(f" RAM {ram_bar} {res['ram_pct']:5.1f}% {res['ram_used_mb']:,}MB / {res['ram_total_mb']:,}MB") lines.append(f" DSK {dsk_bar} {res['disk_pct']:5.1f}% {res['disk_used_gb']:.1f}GB / {res['disk_total_gb']:.1f}GB") lines.append(f" Threads: {res['active_threads']} | Dynamic workers: {res['dynamic_workers']}") # Portfolio summary total_pnl = sum(a.total_pnl for a in agents) total_start = STARTING_CAPITAL * len(agents) # True equity = starting + cumulative P&L (not inflated capital sum from restarts) total_equity = total_start + total_pnl total_trades = sum(a.trades for a in agents) total_wins = sum(a.wins for a in agents) win_rate = total_wins / max(total_trades, 1) * 100 lines.append("") lines.append(" PORTFOLIO SUMMARY") lines.append(thin) lines.append(f" Starting capital : ${total_start:>12,.2f}") lines.append(f" Current equity : ${total_equity:>12,.2f}") lines.append(f" Net P&L : ${total_pnl:>+12,.2f} ({total_pnl/total_start*100:+.2f}%)") lines.append(f" Total trades : {total_trades:>6} | Win rate: {win_rate:.1f}%") lines.append(f" Active positions : {sum(len(a.positions) for a in agents):>6}") # Agent leaderboard -- top 10 by total_pnl, show IC score from v9 lines.append("") lines.append(" AGENT LEADERBOARD (sorted by Net P&L)") lines.append(thin) lines.append(f" {'Agent':<22} {'Type':>4} {'NetP&L':>10} {'WinRate':>8} {'Trades':>7} {'IC':>6} {'Pos':>4}") lines.append(" " + "-" * 60) # Include MetaLearner trading agent in leaderboard (passed via agents list or kwarg) _all_agents = list(agents) ranked = sorted(_all_agents, key=lambda a: a.total_pnl, reverse=True) for ag in ranked[:16]: # +1 for MetaLearner wr = ag.wins / max(ag.trades, 1) * 100 ic = ag.model.rolling_ic() if hasattr(ag.model, "rolling_ic") else 0.0 typ = "Clone" if ag.is_clone else "Base " pnl_arrow = "▲" if ag.total_pnl > 0 else "▼" if ag.total_pnl < 0 else "─" lines.append( f" {ag.name:<22} {typ:>5} {pnl_arrow}${ag.total_pnl:>+9,.2f} " f"{wr:>7.1f}% {ag.trades:>7} {ic:>6.3f} {len(ag.positions):>4}" + (" ⚠️LOW-IC" if ic < IC_MIN_VOTE_THRESHOLD * 2 and ag.model.trained else "") ) # Bottom 5 agents (underperformers -- by IC since P&L is 0 early on) bottom = sorted(agents, key=lambda a: a.model.rolling_ic() if hasattr(a.model, "rolling_ic") else 0.0)[:5] lines.append("") lines.append(" UNDERPERFORMERS (pruning candidates)") lines.append(thin) for ag in bottom: ic = ag.model.rolling_ic() if hasattr(ag.model, "rolling_ic") else 0.0 wr = ag.wins / max(ag.trades, 1) * 100 lines.append(f" {ag.name:<22} P&L=${ag.total_pnl:>+9,.2f} WR={wr:.1f}% IC={ic:.3f}") # Active positions across all agents all_positions = [] for ag in agents: for sym, pos in ag.positions.items(): cur = prices.get(sym, pos.entry) pnl_pct = pos.pnl_pct(cur) all_positions.append((ag.name, sym, pos.entry, cur, pnl_pct, pos.shares)) if all_positions: lines.append("") lines.append(f" ACTIVE POSITIONS ({len(all_positions)} open)") lines.append(thin) lines.append(f" {'Agent':<22} {'Sym':<6} {'Entry':>8} {'Current':>8} {'P&L%':>8} {'Shares':>8}") lines.append(" " + "-" * 65) for ag_name, sym, entry, cur, pnl_pct, shares in sorted(all_positions, key=lambda x: x[4], reverse=True): arrow = "▲" if pnl_pct > 0 else "▼" lines.append(f" {ag_name:<22} {sym:<6} ${entry:>7.2f} ${cur:>7.2f} {arrow}{pnl_pct:>+6.1f}% {shares:>8.3f}") # Recent trades (last 24h across all agents) recent_trades = sorted( [t for a in agents for t in a.history[-10:]], key=lambda t: t.get("ts", 0), reverse=True )[:15] if recent_trades: lines.append("") lines.append(" RECENT TRADES") lines.append(thin) lines.append(f" {'Agent':<20} {'Sym':<6} {'P&L':>9} {'Act%':>7} {'Reason':<15}") lines.append(" " + "-" * 65) for t in recent_trades: arrow = "▲" if t["pnl"] > 0 else "▼" lines.append( f" {t['agent']:<20} {t['symbol']:<6} {arrow}${t['pnl']:>+8.2f} " f"{t['pnl_pct']:>+6.1f}% {t.get('reason',''):<15}" ) # Feature importance -- from highest IC agent (most statistically reliable signal) # Using IC (information coefficient) not P&L since P&L is 0 early on best_agent = max(agents, key=lambda a: a.model.rolling_ic() if hasattr(a.model, "rolling_ic") else 0.0) if hasattr(best_agent.model, "feat_importance") and best_agent.model.feat_importance: fi = sorted(best_agent.model.feat_importance.items(), key=lambda x: -x[1])[:10] lines.append("") lines.append(f" TOP FEATURES (from {best_agent.name})") lines.append(thin) for feat, imp in fi: bar = bar_chart(imp, fi[0][1], width=25) lines.append(f" {feat:<20} {bar} {imp:.4f}") # Data collection status lines.append("") lines.append(" DATA COLLECTION") lines.append(thin) lines.append(f" {dm.status_line()}") lines.append(f" Sentiment symbols tracked: {len(dm.sentiment_engine.scores) if hasattr(dm, 'sentiment_engine') and dm.sentiment_engine else 'N/A'}") lines.append("") lines.append(sep) lines.append(f" Generated: {now_cst} | Bot: v9") lines.append(sep) report_text = "\n".join(lines) report_path = "logs/human_report.txt" try: with open(report_path, "w") as f: f.write(report_text) logger.info(f"Human report -> {report_path}") except Exception as e: debug_logger.warning(f"HUMAN_REPORT_WRITE_ERR | {e}") # Generate HTML dashboard alongside the text report try: _write_html_dashboard(agents, dm, meta) except Exception as _he: debug_logger.warning(f"HTML_DASHBOARD_CALL_ERR | {_he}") def _write_html_dashboard(agents: list, dm: DataManager, meta: MetaLearner) -> None: """ Generate a comprehensive HTML dashboard saved to logs/bot_dashboard.html. Reads claude_handoff.json, TRADE_LOG.csv, train_done.json for all data. Pushed to GitHub daily via _do_github_push. Sections / Tabs: Overview -- market status, capital summary, VM gauges, quick stats Agents -- sortable leaderboard, IC bar chart, IC history line chart Training -- per-agent training details, IC trends, distillation status Features -- feature importance bar chart from highest-IC agent Trades -- trade log table + cumulative P&L chart (when trades exist) System -- VM resources, disk breakdown, data collection status """ import csv as _csv, base64 as _b64 # ── Load data files ───────────────────────────────────────────────── handoff = {} try: handoff = json.load(open("logs/claude_handoff.json")) except Exception: pass trade_rows = [] try: with open("logs/TRADE_LOG.csv", newline="") as f: trade_rows = list(_csv.DictReader(f)) except Exception: pass train_done = {} try: train_done = json.load(open("logs/train_done.json")) except Exception: pass train_ckpt = {} try: train_ckpt = json.load(open("logs/train_checkpoint.json")) except Exception: pass # ── Derive values ──────────────────────────────────────────────────── now_str = datetime.now(DISPLAY_TZ).strftime("%Y-%m-%d %H:%M:%S CST") mkt_open = is_market_open() secs_open = seconds_until_open() hours_open = f"{secs_open/3600:.1f}h" mkt_status = "OPEN" if mkt_open else f"CLOSED — opens in {hours_open}" mkt_color = "#3fb950" if mkt_open else "#f85149" portfolio = handoff.get("portfolio", {}) vm = handoff.get("vm_resources", {}) ag_list = handoff.get("agents", []) feature_cols= handoff.get("feature_cols", FEATURE_COLS) n_features = handoff.get("n_features", N_FEATURES) gen_at = handoff.get("generated_at", now_str) total_pnl = portfolio.get("total_pnl", 0.0) # Count actual agents including MetaLearner for accurate starting capital n_agents = len(ag_list) if ag_list else NUM_AGENTS total_start = STARTING_CAPITAL * n_agents # True equity = starting + cumulative P&L (not inflated equity sum) total_capital = total_start + total_pnl total_trades = portfolio.get("total_trades", 0) total_wins = portfolio.get("total_wins", 0) win_rate = (total_wins / max(total_trades, 1) * 100) pnl_pct = (total_pnl / total_start * 100) if total_start else 0 total_pos = portfolio.get("total_positions", 0) # IC stats ic_vals = [a.get("rolling_ic", 0) for a in ag_list if a.get("rolling_ic") is not None] avg_ic = (sum(ic_vals) / len(ic_vals)) if ic_vals else 0 best_ic = max(ic_vals) if ic_vals else 0 worst_ic = min(ic_vals) if ic_vals else 0 # Feature importance from highest IC agent best_ag = max(ag_list, key=lambda a: a.get("rolling_ic", 0)) if ag_list else {} feat_imp = best_ag.get("top_features", {}) # Training info total_train_count = sum(a.get("train_count", 0) for a in ag_list) last_eod = train_done.get("eod_date") or "Not yet recorded" last_eod = last_eod if last_eod else "Not yet recorded" last_wknd = train_done.get("weekend_date", "Never") last_ckpt = train_ckpt.get("date", "N/A") # Cumulative P&L from trade log cum_pnl_data = [] running = 0.0 for row in trade_rows: try: running += float(row.get("pnl", 0)) cum_pnl_data.append({ "ts": row.get("timestamp", ""), "pnl": round(running, 2), "symbol": row.get("symbol", ""), "agent": row.get("agent", ""), }) except Exception: pass # ── Build JS data objects ──────────────────────────────────────────── # Agent IC bar chart data ag_names = [a["name"] for a in ag_list] ag_ic = [round(a.get("rolling_ic", 0), 4) for a in ag_list] ag_colors = [] for ic in ag_ic: if ic >= 0.045: ag_colors.append("rgba(63,185,80,0.85)") elif ic >= 0.030: ag_colors.append("rgba(210,153,34,0.85)") else: ag_colors.append("rgba(248,81,73,0.85)") # IC history line chart — top 5 agents by IC top5 = sorted(ag_list, key=lambda a: a.get("rolling_ic", 0), reverse=True)[:5] line_colors = ["#3fb950","#58a6ff","#f0883e","#a371f7","#39d353"] ic_history_datasets = [] for i, ag in enumerate(top5): hist = ag.get("ic_history", []) ic_history_datasets.append({ "label": ag["name"], "data": [round(v,4) for v in hist], "borderColor": line_colors[i], "backgroundColor": line_colors[i] + "33", "tension": 0.3, "fill": False, "pointRadius": 3, }) max_hist_len = max((len(ag.get("ic_history",[])) for ag in top5), default=1) ic_history_labels = [f"Train {i+1}" for i in range(max_hist_len)] # Feature importance chart feat_labels = list(feat_imp.keys())[:15] feat_vals = [round(v, 4) for v in list(feat_imp.values())[:15]] # VM gauges cpu_pct = vm.get("cpu_pct", 0) ram_pct = vm.get("ram_pct", 0) ram_used = vm.get("ram_used_mb", 0) ram_tot = vm.get("ram_total_mb", 7476) dsk_pct = vm.get("disk_pct", 0) dsk_used = vm.get("disk_used_gb", 0) dsk_tot = vm.get("disk_total_gb", 31.6) threads = vm.get("active_threads", 0) load_1 = vm.get("load_1m", 0) load_5 = vm.get("load_5m", 0) load_15 = vm.get("load_15m", 0) # Serialize to JSON for embedding _json_agents = json.dumps(ag_list, default=str) _json_trades = json.dumps(trade_rows[:500], default=str) # cap at 500 rows _json_cum_pnl = json.dumps(cum_pnl_data[-200:], default=str) _json_ic_hist = json.dumps(ic_history_datasets, default=str) _json_ic_labels = json.dumps(ic_history_labels, default=str) _json_ag_names = json.dumps(ag_names, default=str) _json_ag_ic = json.dumps(ag_ic, default=str) _json_ag_colors = json.dumps(ag_colors, default=str) _json_feat_lbl = json.dumps(feat_labels, default=str) _json_feat_val = json.dumps(feat_vals, default=str) _json_feature_cols = json.dumps(feature_cols, default=str) pnl_color = "#3fb950" if total_pnl >= 0 else "#f85149" pnl_arrow = "▲" if total_pnl > 0 else "▼" if total_pnl < 0 else "─" refresh_tag = '' # always-on, 60s # ── Generate agent table rows ─────────────────────────────────────── def esc(s): return str(s).replace("&","&").replace("<","<").replace(">",">") agent_rows_html = "" for ag in sorted(ag_list, key=lambda a: a.get("rolling_ic",0), reverse=True): ic = ag.get("rolling_ic", 0) wr = ag.get("win_rate", 0) * 100 pnl = ag.get("total_pnl", 0) tc = ag.get("tree_count", ag.get("train_count", 0)) trnc = ag.get("train_count", 0) vacc = ag.get("val_acc", 0) trds = ag.get("trades", 0) typ = "Clone" if ag.get("type") == "clone" else "Base" ic_bar_w = int(ic / max(best_ic, 0.001) * 80) ic_color = "#3fb950" if ic >= 0.045 else "#d29922" if ic >= 0.030 else "#f85149" warn = " ⚠️" if ic < IC_MIN_VOTE_THRESHOLD * 2 else "" agent_rows_html += f""" {esc(ag['name'])}{warn} {typ}
{ic:.4f}
{trnc} {vacc:.3f} {ag.get('train_acc',0):.3f} {trds} {wr:.1f}% {pnl:+.2f} {ag.get('open_positions',0)} {ag.get('min_conf',0):.2f} {ag.get('stop_loss_pct',0)*100:.1f}% {ag.get('take_profit_pct',0)*100:.1f}% """ # ── Trade log rows ────────────────────────────────────────────────── trade_rows_html = "" if trade_rows: for row in reversed(trade_rows[-200:]): pnl_v = float(row.get("pnl", 0)) cls = "pos" if pnl_v > 0 else "neg" if pnl_v < 0 else "" trade_rows_html += f""" {esc(row.get('timestamp','')[:19])} {esc(row.get('agent',''))} {esc(row.get('type',''))} {esc(row.get('symbol',''))} {row.get('shares','')} ${float(row.get('entry_price',0)):.2f} ${float(row.get('exit_price',0)):.2f} {pnl_v:+.2f} {float(row.get('pnl_pct',0)):+.2f}% {float(row.get('meta_conf',0)):.3f} {esc(row.get('reason',''))} """ else: trade_rows_html = 'No trades yet — bot is in learning phase' # ── Feature rows ──────────────────────────────────────────────────── feat_rows_html = "" if feat_imp: max_imp = max(feat_imp.values()) if feat_imp else 1 for feat, imp in sorted(feat_imp.items(), key=lambda x: -x[1])[:20]: bar_w = int(imp / max_imp * 100) feat_rows_html += f""" {esc(feat)}
{imp:.4f} """ else: feat_rows_html = 'Feature importance loads after first training session' # ── Training detail table rows (precomputed to avoid nested f-string issues) ── _train_rows_html = "" for _a in sorted(ag_list, key=lambda x: x.get('rolling_ic', 0), reverse=True): _ic = _a.get('rolling_ic', 0) _ic_c = "var(--green)" if _ic >= 0.040 else "var(--yellow)" if _ic >= 0.025 else "var(--red)" _atype = "clone" if _a.get("type") == "clone" else "base" _badge = "badge-clone" if _atype == "clone" else "badge-base" _blabel= "Clone" if _atype == "clone" else "Base" _hist = _a.get("ic_history", []) _hist_str = " ".join(f"{v:.3f}" for v in _hist[-8:]) _trend = "↗" if len(_hist) >= 2 and _hist[-1] > _hist[0] else "↘" if len(_hist) >= 2 else "─" _train_rows_html += ( f'' f'{esc(_a["name"])}' f'{_blabel}' f'{_a.get("train_count", 0)}' f'{_ic:.4f}' f'{_hist_str}' f'{_a.get("val_acc", 0):.4f}' f'{_a.get("train_acc", 0):.4f}' f'{_trend}' f'\n' ) # ── HTML ───────────────────────────────────────────────────────────── html = f""" {refresh_tag} Trading Bot Dashboard
🤖 ML Trading Bot v9
Capital ${total_capital:,.2f}
P&L {pnl_arrow}${abs(total_pnl):.2f}
Avg IC {avg_ic:.4f}
Agents {len(ag_list)}
● {mkt_status}
Updated: {gen_at}
${total_capital:,.2f}
Total Equity
Started ${total_start:,.2f}
{pnl_arrow}${abs(total_pnl):.2f}
Net P&L
{pnl_pct:+.2f}%
{win_rate:.1f}%
Win Rate
{total_wins}W / {total_trades - total_wins}L of {total_trades}
{avg_ic:.4f}
Avg IC Score
Best: {best_ic:.4f} · Worst: {worst_ic:.4f}
{total_pos}
Open Positions
Across all agents
{total_train_count}
Total Trains
Last EOD: {last_eod}
📊 Overview
🤖 Agents
🎓 Training
🔬 Features
📈 Trades
⚙️ System
Portfolio Equity Curve
Agent IC Distribution
VM Resources
CPU Usage
{cpu_pct:.1f}%
0%Load: {load_1:.2f} / {load_5:.2f} / {load_15:.2f}100%
RAM Usage
{ram_pct:.1f}% {ram_used:,} MB / {ram_tot:,} MB
0 MB{ram_tot:,} MB
Disk Usage
{dsk_pct:.1f}% {dsk_used:.1f} GB / {dsk_tot:.1f} GB
0 GB{dsk_tot:.1f} GB
Data & Model Status
{n_features}
Feature Columns
{len(ag_list)}
Active Agents
{threads}
Threads Running
{last_eod}
Last EOD Train
{last_wknd}
Last Weekend Train
{IC_MIN_VOTE_THRESHOLD:.3f}
IC Vote Threshold
IC Score — All Agents (sorted by IC)
Agent Leaderboard
{agent_rows_html}
Name Type IC Score ▼ Trains Val Acc Train Acc Trades Win % P&L ($) Pos Min Conf Stop % TP %
IC Score History — Top 5 Agents
Training Timeline
Last Weekend Deep Train
{last_wknd} · +100 trees per agent · {len(ag_list)} agents
Last EOD Incremental Train
{last_eod} · +50 trees per agent
Last Checkpoint
Date: {last_ckpt} · Agent: {train_ckpt.get('last_completed','N/A')} · Idx: {train_ckpt.get('next_agent_idx','N/A')}

Training Parameters
Daily trees per agent{DAILY_TRAIN_TREES}
Weekend trees per agent{WEEKLY_TRAIN_TREES}
Distillation threshold{DISTILL_THRESHOLD} trees
Distillation base trees{DISTILL_BASE_TREES}
IC vote threshold{IC_MIN_VOTE_THRESHOLD:.3f}
Workers (training){MAX_WORKERS_CAP or 8}
Total trains all agents{total_train_count}
Per-Agent Training Details
_train_rows_html
AgentTypeTrainsIC ScoreIC History (last 8)Val AccTrain AccIC Trend
Feature Importance — {esc(best_ag.get('name','Highest IC Agent'))} (IC={best_ic:.4f})
Top Feature Rankings
{feat_rows_html}
FeatureRelative ImportanceGain
All {n_features} Feature Columns
{''.join(f'{esc(c)}' for c in feature_cols)}
Cumulative P&L Over Time
Trade Log ({len(trade_rows)} total trades)
{trade_rows_html}
TimestampAgentTypeSymbolSharesEntryExitP&L ($)P&L %Meta ConfReason
Market
Status{mkt_status}
Mode{'OPEN — collecting bars' if mkt_open else 'CLOSED — sentiment only'}
TradingPaper (Alpaca)
Starting capital${STARTING_CAPITAL:,.2f} / agent
Total capital${total_start:,.2f}
Bot Version
Versionv9 (Lifetime Learning)
Agents{len(ag_list)} ({NUM_AGENTS} base+clone + 5 MetaLearners = 105 total)
Features{n_features}
Min trade rows{MIN_ROWS_TO_TRADE}
Min train rows{MIN_ROWS_TO_TRAIN}
Generated{gen_at}
VM Resources
CPU{cpu_pct:.1f}%
RAM Used{ram_used:,} MB / {ram_tot:,} MB ({ram_pct:.1f}%)
Disk Used{dsk_used:.1f} GB / {dsk_tot:.1f} GB ({dsk_pct:.1f}%)
Threads{threads}
Load (1/5/15m){load_1:.2f} / {load_5:.2f} / {load_15:.2f}
Configuration
Training
Daily trees{DAILY_TRAIN_TREES}
Weekend trees{WEEKLY_TRAIN_TREES}
Distill threshold{DISTILL_THRESHOLD}
Distill base{DISTILL_BASE_TREES}
IC vote min{IC_MIN_VOTE_THRESHOLD}
Max workers{MAX_WORKERS_CAP or 8}
Scheduling
EOD trigger4:15 PM ET post-close
Weekend triggerSaturday 6AM–10PM ET
Backup (market hours)Every 30 min
Backup (off-hours)Every 2 hours
Backup (weekend)Every 6 hours
GitHub pushDaily
""" out = "logs/bot_dashboard.html" try: with open(out, "w", encoding="utf-8") as f: f.write(html) sz = os.path.getsize(out) logger.info(f"HTML dashboard -> {out} ({sz//1024}KB)") debug_logger.info(f"HTML_DASHBOARD | {sz//1024}KB | agents={len(ag_list)} | trades={len(trade_rows)}") except Exception as e: debug_logger.warning(f"HTML_DASHBOARD_ERR | {e}") def _write_claude_handoff(agents: list, dm: DataManager, meta: MetaLearner): """ v9: Write logs/claude_handoff.json -- structured data for Claude analysis. Jake can paste this file directly to Claude for deep analysis. Updated every hour alongside the human report. """ now_ts = time.time() now_str = datetime.now(DISPLAY_TZ).strftime("%Y-%m-%d %H:%M:%S CST") prices = dm.prices res = _get_vm_resources() def safe_float(x): try: return round(float(x), 6) except Exception: return None agent_data = [] for ag in agents: # Feature importance from LGB model fi = {} ic_hist = [] psi = None train_count = 0 if hasattr(ag.model, "feat_importance"): fi = ag.model.feat_importance if hasattr(ag.model, "ic_history"): ic_hist = ag.model.ic_history[-20:] if hasattr(ag.model, "rolling_ic"): rolling_ic = ag.model.rolling_ic() else: rolling_ic = 0.0 if hasattr(ag.model, "_train_count"): train_count = ag.model._train_count # PSI on recent predictions (estimate) recent_trades = ag.history[-50:] if ag.history else [] agent_data.append({ "name": ag.name, "type": "clone" if ag.is_clone else "base", "capital": safe_float(ag.capital), "total_pnl": safe_float(ag.total_pnl), "gross_profit": safe_float(ag.gross_profit), "gross_loss": safe_float(ag.gross_loss), "trades": ag.trades, "wins": ag.wins, "win_rate": safe_float(ag.wins / max(ag.trades, 1)), "rolling_ic": safe_float(rolling_ic), "ic_history": [safe_float(x) for x in ic_hist], "train_count": train_count, "ic_score": safe_float(rolling_ic), # alias for rolling_ic (consistent naming) "tree_count": ag.model.tree_count if hasattr(ag.model, "tree_count") else 0, "val_acc": safe_float(ag.model.val_acc), "train_acc": safe_float(ag.model.train_acc), "trained": ag.model.trained, "stop_loss_pct": safe_float(ag.stop_loss_pct), "take_profit_pct": safe_float(ag.take_profit_pct), "min_conf": safe_float(ag.min_conf), "max_pos_pct": safe_float(ag.max_pos_pct), "kelly_wins": ag._kelly_wins, "kelly_losses": ag._kelly_losses, "kelly_avg_win": safe_float(ag._kelly_avg_win), "kelly_avg_loss":safe_float(ag._kelly_avg_loss), "positions": list(ag.positions.keys()), "open_positions": len(ag.positions), "top_features": dict(sorted(fi.items(), key=lambda x: -x[1])[:15]), "recent_trades": [ { "symbol": t.get("symbol"), "pnl": safe_float(t.get("pnl", 0)), "pnl_pct": safe_float(t.get("pnl_pct", 0)), "predicted": safe_float(t.get("predicted_pct", 0)), "reason": t.get("reason", ""), "own_conf": safe_float(t.get("own_conf", 0)), "meta_conf": safe_float(t.get("meta_conf", 0)), } for t in recent_trades[-10:] ], }) # Sort by total_pnl descending agent_data.sort(key=lambda x: x["total_pnl"] or 0, reverse=True) # Symbol universe snapshot top_symbols = sorted(prices.items(), key=lambda x: x[1], reverse=True)[:20] handoff = { "generated_at": now_str, "generated_ts": now_ts, "bot_version": "v9", "market_open": is_market_open(), "vm_resources": res, "dynamic_workers": _dynamic_workers, "n_features": N_FEATURES, "feature_cols": FEATURE_COLS, "portfolio": { "n_agents": len(agents), "total_pnl": safe_float(sum(a.total_pnl for a in agents)), "total_equity": safe_float(sum(a.total_equity(prices) for a in agents)), "total_trades": sum(a.trades for a in agents), "total_wins": sum(a.wins for a in agents), "win_rate": safe_float(sum(a.wins for a in agents) / max(sum(a.trades for a in agents), 1)), "total_positions":sum(len(a.positions) for a in agents), }, "agents": agent_data, "meta_learner": { "trained": meta.trained, "val_acc": safe_float(meta.val_acc), "train_acc": safe_float(meta.train_acc), "rolling_ic": safe_float(meta.rolling_ic() if hasattr(meta, "rolling_ic") else 0), "gate_active": getattr(meta, "gate_active", False), "sub_metas": meta.sub_meta_summary() if hasattr(meta, "sub_meta_summary") else [], }, "data": { "n_prices": len(prices), "status_line": dm.status_line(), }, "top_prices": {sym: safe_float(p) for sym, p in top_symbols}, } handoff_path = "logs/claude_handoff.json" try: with open(handoff_path, "w") as f: json.dump(handoff, f, indent=2) logger.info(f"Claude handoff -> {handoff_path}") except Exception as e: debug_logger.warning(f"CLAUDE_HANDOFF_WRITE_ERR | {e}") def hourly_report(agents: list, dm: DataManager, meta: MetaLearner): _set_nice(10) # P3 LOW: reports are not time-critical _resource_guard.yield_p3("hourly_report") now_cst = datetime.now(DISPLAY_TZ).strftime("%Y-%m-%d %H:%M:%S") sep = "=" * 150 meta_disc = meta.avg_discrepancy() lines = [ sep, f"HOURLY REPORT {now_cst} CST", dm.status_line(), f"CPU cores: {PHYSICAL_CORES} | Agents: {NUM_AGENTS} ({BASE_AGENTS} base + {CLONE_AGENTS} clones)", f"MetaSystem: trained={meta.trained} gate={getattr(meta,'gate_active','N/A')} " f"discrepancy={f'{meta_disc:.2f}pp' if meta_disc else 'N/A'} " f"profitable_symbols={len(meta.profitable_symbols)}", sep, f"{'Agent':<24}{'T':>2}{'Equity':>10}{'Cash':>10}{'NetPnL':>10}" f"{'GrossWin':>10}{'GrossLoss':>10}{'Trades':>7}{'Win%':>7}" f"{'AvgDisc':>9}{'RollDisc':>9}{'SL%':>6}{'TP%':>6}" f"{'MaxP':>5}{'Conf':>6}{'Pos':>4}{'Rst':>4}", "-" * 150, ] for a in agents: s = a.summary(dm.prices) tag = "C" if s["is_clone"] else "B" d = f"{s['avg_discrepancy']:.2f}pp" if s["avg_discrepancy"] is not None else " N/A" rd = f"{s['roll_discrepancy']:.2f}pp" if s["roll_discrepancy"] is not None else " N/A" lines.append( f"{s['agent']:<24}{tag:>2}${s['equity']:>9,.2f}${s['capital']:>9,.2f}" f"${s['pnl']:>9,.2f}${s['gross_profit']:>9,.2f}${s['gross_loss']:>9,.2f}" f"{s['trades']:>7}{s['profit_rate']:>6.1%}" f"{d:>9}{rd:>9}" f"{s['stop_loss']*100:>5.1f}%{s['take_profit']*100:>5.1f}%" f"{s['max_pos']:>5}{s['min_conf']:>5.2f}{s['positions']:>4}{s['resets']:>4}" ) total_pnl = sum(a.total_pnl for a in agents) total_gp = sum(a.gross_profit for a in agents) total_gl = sum(a.gross_loss for a in agents) total_t = sum(a.trades for a in agents) total_w = sum(a.wins for a in agents) short_gp = sum(getattr(a,"short_gross_profit",0.0) for a in agents) short_gl = sum(getattr(a,"short_gross_loss",0.0) for a in agents) short_t = sum(getattr(a,"short_trades",0) for a in agents) short_w = sum(getattr(a,"short_wins",0) for a in agents) long_net = total_gp - total_gl short_net = short_gp - short_gl lines += [ "-" * 150, f" TOTALS: NetPnL=${total_pnl:>+.2f} GrossProfit=${total_gp:.2f} " f"GrossLoss=${total_gl:.2f} Trades={total_t} " f"WinRate={total_w/total_t*100 if total_t else 0:.1f}%", f" LONG: NetPnL=${long_net:>+.2f} GrossProfit=${total_gp:.2f} " f"GrossLoss=${total_gl:.2f} Trades={total_t} WinRate={total_w/max(total_t,1)*100:.1f}%\n" f" SHORT: NetPnL=${short_net:>+.2f} GrossProfit=${short_gp:.2f} " f"GrossLoss=${short_gl:.2f} Trades={short_t} WinRate={short_w/max(short_t,1)*100:.1f}%\n" "", " T=type(B=base,C=clone) AvgDisc=avg pp discrepancy(lower=better) " "RollDisc=last-20-trade discrepancy", ] if dm.sector_perf: lines += ["", "Sector Performance:"] for sec, perf in sorted(dm.sector_perf.items(), key=lambda x: -x[1]): lines.append(f" {sec:<38} {perf:>+.4f}") all_trades = sorted( [t for a in agents for t in a.history[-15:]], key=lambda t: abs(t["pnl"]), reverse=True )[:20] if all_trades: lines += ["", "Top Recent Trades (sorted by |PnL|):"] lines.append(f" {'Agent':<24} {'Sym':<6} {'PnL':>9} {'Actual%':>8} " f"{'Pred%':>8} {'Disc':>6} {'OwnCf':>6} {'MetaCf':>6} " f"{'EffCf':>6} Reason") lines.append(" " + "-" * 105) for t in all_trades: disc = t.get("disc_pp", abs(t["pnl_pct"]/100 - t["predicted_pct"]/100) * 100) lines.append( f" [{t['agent']:<22}] {t['symbol']:<6} " f"PnL={t['pnl']:>+8.2f} actual={t['pnl_pct']:>+6.1f}% " f"pred={t['predicted_pct']:>+6.1f}% {disc:>5.2f}pp " f"{t.get('own_conf',0):>5.3f} {t.get('meta_conf',0):>5.3f} " f"{t.get('eff_conf',0):>5.3f} {t['reason']}" ) lines.append(sep + "\n") report_logger.info("\n" + "\n".join(lines)) logger.info(f"Report written -> {REPORT_LOG_FILE}") # v9: Write enhanced human-readable report + Claude handoff JSON try: _write_human_report(agents, dm, meta, lines) except Exception as _e: debug_logger.warning(f"HUMAN_REPORT_ERR | {_e}") try: _write_claude_handoff(agents, dm, meta) except Exception as _e: debug_logger.warning(f"CLAUDE_HANDOFF_ERR | {_e}") # ============================================================================= # RETRAIN SCHEDULER # ============================================================================= class MarketAwareRetrainScheduler: """ v9: Simplified scheduler -- intraday retraining removed. Training fires ONCE per day, 15-30 min after market close. The _after_hours_train() method handles the full sequential cycle. The scheduler.tick() is kept for future use but does nothing during hours. """ def __init__(self, callback): self.callback = callback self._last_minute = -1 self._did_eod = False def tick(self): # v9: No intraday retraining. Training is EOD-only and triggered # directly in the main loop by the _after_hours_train_done check. # This tick() is a no-op kept for structural compatibility. pass # ============================================================================= # MAIN SYSTEM # ============================================================================= # Forensic timestamp helpers -- use in debug log entries for absolute timing def unix_ts() -> int: return int(time.time()) def ts_to_cst(ts: int) -> str: """Unix int -> HHMM CST string (no colon, Python 3.12 safe). CST = UTC minus 21600 seconds. Example: unix_ts() -> 1712345678, ts_to_cst(...) -> 1423 CST """ s = (ts - 21600) % 86400 return f"{s // 3600:02d}{(s % 3600) // 60:02d} CST" # ============================================================================= # COLLECT-ONLY MODE -- DATA COLLECTION HEALTH TRACKER # ============================================================================= COLLECT_ONLY_REPORT_INTERVAL = 60 VALID_SOURCES = {"alpaca","finnhub","news","sec","stocktwits","reddit","congress","all"} class DataCollectionHealth: """Per-source diagnostic metrics. Populated by _health.record() calls throughout the data pipelines. Read by _run_collect_only() health reports.""" def __init__(self): self._lock = threading.Lock() # Alpaca self.alpaca_bars = 0; self.alpaca_errors = 0 self.alpaca_connected = False; self.alpaca_last_bar = None self.alpaca_consec_fails = 0 # Finnhub self.fh_calls = 0; self.fh_errors = 0 self.fh_last_call = None; self.fh_last_symbol = "" # News self.news_fetches = 0; self.news_errors = 0 self.news_last_ts = None; self.news_last_symbol = "" # SEC self.sec_scans = 0; self.sec_errors = 0 self.sec_filings_found = 0; self.sec_last_ts = None # StockTwits self.st_fetches = 0; self.st_errors = 0 self.st_last_ts = None; self.st_last_symbol = "" # Reddit self.reddit_fetches = 0; self.reddit_errors = 0; self.reddit_last_ts = None # Congress self.congress_scans = 0; self.congress_errors = 0; self.congress_last_ts = None # DB self.db_bars_written = 0 def record(self, source: str, **kw): with self._lock: if source == "alpaca_bar": self.alpaca_bars += 1; self.alpaca_last_bar = time.time() self.alpaca_connected = True elif source == "alpaca_connect": self.alpaca_connected = kw.get("connected", True) self.alpaca_consec_fails = kw.get("consec_fails", 0) elif source == "alpaca_error": self.alpaca_errors += 1 self.alpaca_consec_fails = kw.get("consec_fails", self.alpaca_consec_fails + 1) elif source == "finnhub": self.fh_calls += 1; self.fh_last_call = time.time() self.fh_last_symbol = kw.get("symbol", "") if kw.get("error"): self.fh_errors += 1 elif source == "news": self.news_fetches += 1; self.news_last_ts = time.time() self.news_last_symbol = kw.get("symbol", "") if kw.get("error"): self.news_errors += 1 elif source == "sec": self.sec_scans += 1; self.sec_last_ts = time.time() self.sec_filings_found += kw.get("filings", 0) if kw.get("error"): self.sec_errors += 1 elif source == "stocktwits": self.st_fetches += 1; self.st_last_ts = time.time() self.st_last_symbol = kw.get("symbol", "") if kw.get("error"): self.st_errors += 1 elif source == "reddit": self.reddit_fetches += 1; self.reddit_last_ts = time.time() if kw.get("error"): self.reddit_errors += 1 elif source == "congress": self.congress_scans += 1; self.congress_last_ts = time.time() if kw.get("error"): self.congress_errors += 1 elif source == "db_write": self.db_bars_written += kw.get("count", 1) def _age(self, ts): if ts is None: return "never" a = time.time() - ts if a < 60: return f"{a:.0f}s ago" if a < 3600: return f"{a/60:.0f}m ago" return f"{a/3600:.1f}h ago" def report(self) -> str: with self._lock: ws = ("CONNECTED" if self.alpaca_connected else f"DISCONNECTED(fails={self.alpaca_consec_fails})") return "\n".join([ f"[DATA-HEALTH] {'='*58}", f"[DATA-HEALTH] Alpaca WS : {ws} | bars={self.alpaca_bars:,} | " f"last_bar={self._age(self.alpaca_last_bar)} | errors={self.alpaca_errors}", f"[DATA-HEALTH] Finnhub : calls={self.fh_calls:,} | " f"last={self._age(self.fh_last_call)} | sym={self.fh_last_symbol} | errors={self.fh_errors}", f"[DATA-HEALTH] News RSS : fetches={self.news_fetches:,} | " f"last={self._age(self.news_last_ts)} | sym={self.news_last_symbol} | errors={self.news_errors}", f"[DATA-HEALTH] SEC EDGAR : scans={self.sec_scans} | " f"filings={self.sec_filings_found} | last={self._age(self.sec_last_ts)} | errors={self.sec_errors}", f"[DATA-HEALTH] StockTwits : fetches={self.st_fetches:,} | " f"last={self._age(self.st_last_ts)} | sym={self.st_last_symbol} | errors={self.st_errors}", f"[DATA-HEALTH] Reddit : fetches={self.reddit_fetches} | " f"last={self._age(self.reddit_last_ts)} | errors={self.reddit_errors}", f"[DATA-HEALTH] Congress : scans={self.congress_scans} | " f"last={self._age(self.congress_last_ts)} | errors={self.congress_errors}", f"[DATA-HEALTH] DB writes : {self.db_bars_written:,} bars", f"[DATA-HEALTH] {'='*58}", ]) _health = DataCollectionHealth() # module-level singleton used by all pipelines # ============================================================================= # RESOURCE MONITOR -- logs CPU/memory/disk every 30s to debug.log # Used to correlate resource spikes with Alpaca disconnects. # ============================================================================= _RESOURCE_MONITOR_INTERVAL = 30 # seconds def _resource_monitor_loop(): """ Background thread: logs system resources every 30 seconds. Writes to debug.log with prefix RESOURCE so you can grep it. Helps correlate CPU/memory/I/O spikes with Alpaca ping timeouts. Run: grep RESOURCE ~/trading/debug.log | tail -50 """ import resource as _res while True: try: # Memory: RSS in MB rss_mb = _res.getrusage(_res.RUSAGE_SELF).ru_maxrss // 1024 # CPU times cpu = _res.getrusage(_res.RUSAGE_SELF) user_s = cpu.ru_utime sys_s = cpu.ru_stime # Thread count n_threads = threading.active_count() # Disk I/O (Linux only) try: with open("/proc/self/io") as _f: io_lines = dict(l.split(": ") for l in _f.read().splitlines() if ": " in l) read_mb = int(io_lines.get("rchar", "0")) // 1024 // 1024 write_mb = int(io_lines.get("wchar", "0")) // 1024 // 1024 except Exception: read_mb = write_mb = -1 # Load average (1/5/15 min) try: with open("/proc/loadavg") as _f: load = _f.read().split()[:3] load_str = "/".join(load) except Exception: load_str = "?" # Open file descriptors try: import os as _os n_fds = len(_os.listdir(f"/proc/{_os.getpid()}/fd")) except Exception: n_fds = -1 debug_logger.info( f"RESOURCE | rss={rss_mb}MB | threads={n_threads} | " f"load={load_str} | io_read={read_mb}MB | io_write={write_mb}MB | " f"cpu_user={user_s:.1f}s | cpu_sys={sys_s:.1f}s | fds={n_fds}" ) # Record for daily resource graph try: with open("/proc/loadavg") as _lf: _cpu_pct = float(_lf.read().split()[0]) / PHYSICAL_CORES * 100 _record_resource_snapshot(_cpu_pct, rss_mb, n_threads) except Exception: pass # v9: Read actual RAM from /proc/meminfo (more precise than loadavg alone) try: with open("/proc/meminfo") as _mf: _mi = dict(l.split(":") for l in _mf.read().splitlines() if ":" in l) _ram_total = int(_mi["MemTotal"].split()[0]) / 1024 # MB _ram_avail = int(_mi["MemAvailable"].split()[0]) / 1024 # MB _ram_used_pct = 1.0 - (_ram_avail / max(_ram_total, 1)) except Exception: _ram_used_pct = 0.5 # v9: Read disk I/O wait from /proc/stat (iowait %) try: with open("/proc/stat") as _sf: _cpu_line = _sf.readline().split() _iowait_pct = float(_cpu_line[5]) / (sum(float(x) for x in _cpu_line[1:]) + 1e-6) except Exception: _iowait_pct = 0.0 # v9: Dynamically compute optimal ThreadPool worker count # Based on CPU load, RAM pressure, and I/O wait global _dynamic_workers try: _load1 = float(load.split("/")[0]) if isinstance(load, str) else float(load[0]) except Exception: _load1 = 0.0 _cpu_pct_now = min(1.0, _load1 / max(PHYSICAL_CORES, 1)) _max_workers = PHYSICAL_CORES if _iowait_pct > 0.30: # heavy disk I/O -- back off _target = max(1, _max_workers // 4) elif _ram_used_pct > 0.88: # RAM critical _target = 1 elif _ram_used_pct > 0.78 or _cpu_pct_now > 0.90: _target = max(1, _max_workers // 4) elif _ram_used_pct > 0.65 or _cpu_pct_now > 0.75: _target = max(2, _max_workers // 2) elif _cpu_pct_now > 0.50: _target = max(4, _max_workers * 3 // 4) else: _target = _max_workers # full speed if _target != _dynamic_workers: debug_logger.info( f"DYNAMIC_WORKERS | {_dynamic_workers} -> {_target} | " f"cpu={_cpu_pct_now:.0%} ram={_ram_used_pct:.0%} iowait={_iowait_pct:.0%}" ) _dynamic_workers = _target except Exception as e: debug_logger.debug(f"RESOURCE_MONITOR_ERR | {e}") time.sleep(_RESOURCE_MONITOR_INTERVAL) def start_resource_monitor(): """Start the resource monitor background thread.""" t = threading.Thread(target=_resource_monitor_loop, daemon=True, name="ResourceMonitor") t.start() debug_logger.info("RESOURCE_MONITOR_STARTED | interval=30s") # ============================================================================= # THREAD PRIORITY & RESOURCE THROTTLING SYSTEM # ============================================================================= # # Priority tiers (P0 = highest): # P0 CRITICAL -- Alpaca WebSocket -- NEVER throttled # P1 HIGH -- Finnhub, all Sentiment pipelines # P2 NORMAL -- Feature rebuild, trading decisions # P3 LOW -- Charts, hourly reports # P4 BACKGROUND-- GitHub backup, local backup, state save # # How it works: # ResourceGuard monitors /proc/loadavg every 5 seconds. # When load exceeds LOAD_NOMINAL (90% of nCPU), P3/P4 threads are throttled. # When load exceeds LOAD_CAP (95% of nCPU), P4 pauses entirely. # P0/P1 threads ALWAYS run at full speed regardless of load. # # Thread niceness (CPU scheduling priority, set inside each thread): # P0/P1: nice = -5 (higher OS scheduling priority) # P2: nice = 0 (normal) # P3: nice = 10 (lower, yields to everything important) # P4: nice = 19 (absolute lowest -- OS schedules last) # # GitHub backup runs in a SUBPROCESS so the Python GIL is fully released. # Previously, base64-encoding 22 MB of debug.log held the GIL for 5-15 seconds, # starving the Alpaca WebSocket ping thread and triggering ping/pong timeouts. # ============================================================================= # Load thresholds as fraction of nCPU (matches /proc/loadavg semantics) LOAD_NOMINAL = 0.90 # run freely below 90% -- max sustained throughput LOAD_CAP = 0.95 # 5% headroom for OS + thread switching above this RESOURCE_CHECK_INTERVAL = 5.0 # seconds between load checks # Extra sleep injected into low-priority threads when load is high _THROTTLE_DELAY_P3 = 1.5 # P3 sleeps this many extra seconds per unit _THROTTLE_DELAY_P4 = 5.0 # P4 sleeps this many extra seconds per unit _PAUSE_DELAY_P4 = 20.0 # P4 pauses this long when at LOAD_CAP import os as _os def _set_nice(level: int): """Set CPU scheduling priority of the current thread. Negative = higher priority, positive = lower. Range: -20 to 19. Silently ignored if process lacks permission to increase priority.""" try: _os.nice(level) except (PermissionError, OSError): pass # Can't raise priority without root -- lowering always works RESOURCE_GUARD_RAM_WARN = 88 # % RAM at which throttling kicks in RESOURCE_GUARD_RAM_BAIL = 93 # % RAM at which we log critical warning def _rss_mb() -> int: """Current process RSS in MB. Zero-cost diagnostic helper.""" try: for line in open('/proc/self/status'): if line.startswith('VmRSS:'): return int(line.split()[1]) // 1024 except Exception: pass return 0 def _log_ram(label: str): """Log current RSS to both trading.log and debug.log with a clear tag.""" mb = _rss_mb() cap = RAM_CAP_MB or 0 pct = f"{mb/cap*100:.1f}% of cap" if cap else "uncapped" logger.info(f"[RAM] {label} | {mb:,} MB ({pct})") debug_logger.info(f"RAM_TRACE | {label} | rss={mb}MB | cap={cap}MB") class ResourceGuard: """ Central resource monitor and throttle controller. All background threads call yield_p3() or yield_p4() between work units to cooperatively yield when the system is under load. Usage in a background thread: _resource_guard.yield_p4() # will sleep if load is high do_expensive_backup_work() _resource_guard.yield_p4() # check again after work Grep debug.log for 'RESOURCE_GUARD' to see throttle events. Grep debug.log for 'RESOURCE |' to see load history. """ def __init__(self, ncpus: int): self.ncpus = max(ncpus, 1) self._lock = threading.Lock() self._load_1m = 0.0 self._load_5m = 0.0 self._load_15m = 0.0 self._throttle_p3 = False self._throttle_p4 = False self._pause_p4 = False self._running = False self._ram_used_mb = 0 self._ram_total_mb = 7400 # default 7.3 GB Oracle VM self._ram_pct = 0.0 def start(self): """Start the background resource monitoring loop.""" self._running = True threading.Thread(target=self._loop, daemon=True, name="ResourceGuard").start() debug_logger.info( f"RESOURCE_GUARD_START | ncpus={self.ncpus} | " f"nominal={LOAD_NOMINAL*100:.0f}% | cap={LOAD_CAP*100:.0f}%" ) def _loop(self): _set_nice(0) # ResourceGuard itself is normal priority while self._running: try: # CPU load with open("/proc/loadavg") as _f: parts = _f.read().split() load_1m = float(parts[0]) # RAM -- read MemAvailable from /proc/meminfo with open("/proc/meminfo") as _mf: _mi = dict(l.split(":") for l in _mf.read().splitlines() if ":" in l) _total_mb = int(_mi.get("MemTotal", "7400000 kB").split()[0]) // 1024 _avail_mb = int(_mi.get("MemAvailable","4000000 kB").split()[0]) // 1024 _used_mb = _total_mb - _avail_mb _ram_pct = _used_mb / max(_total_mb, 1) # v9: RAM cap -- honour --ram-cap ceiling if set, else % of system RAM if RAM_CAP_MB is not None: _ram_throttle = _used_mb > int(RAM_CAP_MB * 0.90) _ram_pause = _used_mb > int(RAM_CAP_MB * 0.97) else: _ram_throttle = _ram_pct > 0.85 _ram_pause = _ram_pct > 0.92 load_5m = float(parts[1]) load_15m = float(parts[2]) load_pct = load_1m / self.ncpus throttle_p3 = (load_pct > LOAD_NOMINAL) or _ram_throttle throttle_p4 = (load_pct > LOAD_NOMINAL) or _ram_throttle pause_p4 = (load_pct > LOAD_CAP) or _ram_pause with self._lock: prev_state = (self._throttle_p3, self._pause_p4) self._load_1m = load_1m self._ram_used_mb = _used_mb self._ram_total_mb= _total_mb self._ram_pct = _ram_pct self._load_5m = load_5m self._load_15m = load_15m self._throttle_p3 = throttle_p3 self._throttle_p4 = throttle_p4 self._pause_p4 = pause_p4 new_state = (throttle_p3, pause_p4) if new_state != prev_state: debug_logger.info( f"RESOURCE_GUARD_STATE_CHANGE | " f"load={load_1m:.2f}/{self.ncpus}cpu ({load_pct*100:.0f}%) | " f"ram={_used_mb}MB/{_total_mb}MB ({_ram_pct*100:.0f}%) | " f"throttle_p3={throttle_p3} | pause_p4={pause_p4} | " f"prev=({prev_state[0]},{prev_state[1]})" ) # Log current state to debug.log for correlation with WS drops debug_logger.debug( f"RESOURCE_GUARD | load={load_1m:.2f}/{load_5m:.2f}/{load_15m:.2f} | " f"cpu={load_pct*100:.0f}% | " f"ram={_used_mb}MB/{_total_mb}MB({_ram_pct*100:.0f}%) | " f"state={'PAUSE_P4' if pause_p4 else 'THROTTLE' if throttle_p3 else 'NOMINAL'}" ) except Exception as e: debug_logger.debug(f"RESOURCE_GUARD_ERR | {e}") time.sleep(RESOURCE_CHECK_INTERVAL) def yield_p3(self, label: str = ""): """Cooperative yield point for P3 (LOW) threads. Sleeps extra when system load is high. Always returns -- never blocks forever.""" with self._lock: throttle = self._throttle_p3 if throttle: debug_logger.debug(f"RESOURCE_GUARD_THROTTLE_P3 | {label} | sleep={_THROTTLE_DELAY_P3}s") time.sleep(_THROTTLE_DELAY_P3) def yield_p4(self, label: str = ""): """Cooperative yield point for P4 (BACKGROUND) threads. Pauses when load is at cap, sleeps extra when throttled.""" with self._lock: pause = self._pause_p4 throttle = self._throttle_p4 if pause: debug_logger.info( f"RESOURCE_GUARD_PAUSE_P4 | {label} | " f"load_cap={LOAD_CAP*100:.0f}% exceeded | pausing {_PAUSE_DELAY_P4}s" ) time.sleep(_PAUSE_DELAY_P4) elif throttle: debug_logger.debug(f"RESOURCE_GUARD_THROTTLE_P4 | {label} | sleep={_THROTTLE_DELAY_P4}s") time.sleep(_THROTTLE_DELAY_P4) @property def load_pct(self) -> float: with self._lock: return self._load_1m / self.ncpus @property def status(self) -> str: with self._lock: cpu_pct = self._load_1m / self.ncpus * 100 ram_pct = self._ram_pct * 100 ram_mb = self._ram_used_mb if self._pause_p4: return f"CRITICAL(cpu={cpu_pct:.0f}%,ram={ram_mb}MB/{ram_pct:.0f}%) P4-PAUSED" elif self._throttle_p3: return f"HIGH(cpu={cpu_pct:.0f}%,ram={ram_mb}MB/{ram_pct:.0f}%) P3-P4-THROTTLED" return f"NOMINAL(cpu={cpu_pct:.0f}%,ram={ram_mb}MB/{ram_pct:.0f}%)" @property def load_str(self) -> str: with self._lock: return f"{self._load_1m:.2f}/{self._load_5m:.2f}/{self._load_15m:.2f}" # Module-level singleton -- shared by all background threads # ============================================================================= # RESOURCE USAGE GRAPH -- generated at end of each trading day # ============================================================================= _resource_history = [] # list of (timestamp, cpu_pct, ram_mb, n_threads) _resource_history_lock = threading.Lock() def _record_resource_snapshot(cpu_pct: float, ram_mb: int, n_threads: int): """Called by resource monitor loop every 30s. Appends to history list.""" with _resource_history_lock: _resource_history.append((time.time(), cpu_pct, ram_mb, n_threads)) # Keep only last 24 hours (24*60*2 = 2880 samples at 30s interval) if len(_resource_history) > 2880: del _resource_history[:-2880] def generate_resource_graph(): """ Generate a resource usage chart (CPU%, RAM MB, thread count) and save to charts/. Called once after market close alongside the accuracy chart. Uses only stdlib if matplotlib is unavailable. """ try: import matplotlib matplotlib.use("Agg") import matplotlib.pyplot as plt import matplotlib.dates as mdates from datetime import datetime as dt with _resource_history_lock: history = list(_resource_history) if len(history) < 5: debug_logger.warning("RESOURCE_GRAPH | not enough data points yet") return timestamps = [dt.fromtimestamp(t) for t, *_ in history] cpu_pcts = [c for _, c, _, _ in history] ram_mbs = [r for _, _, r, _ in history] threads = [n for _, _, _, n in history] fig, (ax1, ax2, ax3) = plt.subplots(3, 1, figsize=(14, 10), sharex=True) fig.suptitle(f"Resource Usage -- {dt.now().strftime('%Y-%m-%d')}", fontsize=14, fontweight="bold") # CPU ax1.fill_between(timestamps, cpu_pcts, alpha=0.4, color="steelblue") ax1.plot(timestamps, cpu_pcts, color="steelblue", linewidth=0.8) ax1.axhline(LOAD_NOMINAL * 100, color="orange", linestyle="--", linewidth=1, label=f"Nominal ({LOAD_NOMINAL*100:.0f}%)") ax1.axhline(LOAD_CAP * 100, color="red", linestyle="--", linewidth=1, label=f"Cap ({LOAD_CAP*100:.0f}%)") ax1.set_ylabel("CPU Load %") ax1.set_ylim(0, max(max(cpu_pcts) * 1.1, 110)) ax1.legend(fontsize=8) ax1.grid(True, alpha=0.3) # RAM ax2.fill_between(timestamps, ram_mbs, alpha=0.4, color="coral") ax2.plot(timestamps, ram_mbs, color="coral", linewidth=0.8) _cap_mb = RAM_CAP_MB if RAM_CAP_MB is not None else 7400 _thr_mb = int(_cap_mb * 0.90) if RAM_CAP_MB else int(7400 * 0.85) _pau_mb = int(_cap_mb * 0.97) if RAM_CAP_MB else int(7400 * 0.92) ax2.axhline(_thr_mb, color="orange", linestyle="--", linewidth=1, label=f"Throttle ({_thr_mb}MB)") ax2.axhline(_pau_mb, color="red", linestyle="--", linewidth=1, label=f"Pause ({_pau_mb}MB)") if RAM_CAP_MB: ax2.axhline(RAM_CAP_MB, color="darkred", linestyle="-", linewidth=1.5, label=f"Hard cap ({RAM_CAP_MB/1024:.1f}GB)") ax2.set_ylabel("RAM Used (MB)") ax2.set_ylim(0, _cap_mb + 300) ax2.legend(fontsize=8) ax2.grid(True, alpha=0.3) # Thread count ax3.plot(timestamps, threads, color="mediumpurple", linewidth=1) ax3.fill_between(timestamps, threads, alpha=0.3, color="mediumpurple") ax3.set_ylabel("Active Threads") ax3.set_ylim(0, max(max(threads) + 2, 20)) ax3.grid(True, alpha=0.3) ax3.xaxis.set_major_formatter(mdates.DateFormatter("%H:%M")) ax3.xaxis.set_major_locator(mdates.HourLocator()) plt.xticks(rotation=30) plt.tight_layout() ts = dt.now().strftime("%Y%m%d_%H%M") out_path = os.path.join(CHARTS_DIR, f"resources_{ts}.png") plt.savefig(out_path, dpi=100, bbox_inches="tight") plt.close(fig) logger.info(f"Resource chart saved -> {out_path}") debug_logger.info(f"RESOURCE_GRAPH_SAVED | {out_path} | samples={len(history)}") except ImportError: debug_logger.warning("RESOURCE_GRAPH | matplotlib not available -- skipping chart") except Exception as e: debug_logger.error(f"RESOURCE_GRAPH_ERR | {e}", exc_info=True) class ProcessAllocator: """Dynamic process budget based on CPU count and available RAM. Uses processes (not threads) for CPU-bound work so each worker gets its own Python interpreter and GIL. """ RAM_PER_TRAIN_GB = 0.45 RAM_PER_FEATURE_GB = 0.30 RAM_HEADROOM_GB = 2.0 def __init__(self): import multiprocessing as _mp self.n_cpus = _mp.cpu_count() or 1 self.ram_total_gb, self.ram_avail_gb = self._read_ram() self.training_procs = self._calc(self.RAM_PER_TRAIN_GB, max_cpu_frac=1.0) self.feature_procs = self._calc(self.RAM_PER_FEATURE_GB, max_cpu_frac=0.5) def _read_ram(self): try: with open("/proc/meminfo") as _f: mi = dict(l.split(":") for l in _f.read().splitlines() if ":" in l) total = int(mi["MemTotal"].split()[0]) / 1024 / 1024 avail = int(mi["MemAvailable"].split()[0]) / 1024 / 1024 return total, avail except Exception: return 4.0, 2.0 def _calc(self, ram_per_proc: float, max_cpu_frac: float) -> int: usable_cpu = max(1, int((self.n_cpus - 2) * max_cpu_frac)) usable_ram = max(1, int( (self.ram_avail_gb - self.RAM_HEADROOM_GB) / ram_per_proc )) base = max(1, min(usable_cpu, usable_ram, 16)) # v9: honour --max-workers cap if set if MAX_WORKERS_CAP is not None: return min(base, MAX_WORKERS_CAP) return base def refresh_avail_ram(self): _, self.ram_avail_gb = self._read_ram() self.training_procs = self._calc(self.RAM_PER_TRAIN_GB, 1.0) self.feature_procs = self._calc(self.RAM_PER_FEATURE_GB, 0.5) def status(self) -> str: return ( f"ProcessAllocator | cpus={self.n_cpus} | " f"ram={self.ram_total_gb:.1f} GB / {self.ram_avail_gb:.1f} GB avail | " f"training_procs={self.training_procs} | feature_procs={self.feature_procs}" ) _proc_alloc = ProcessAllocator() # v9: Dynamic worker count -- ResourceGuard updates this based on live CPU+RAM+disk # ThreadPoolExecutor reads this every call so throttling takes effect immediately _dynamic_workers = max(1, min(PHYSICAL_CORES, 8)) _resource_guard = ResourceGuard(ncpus=PHYSICAL_CORES) class TradingSystem: def __init__(self, collect_only: bool = False, active_sources: set = None, trading_mode: str = "long"): self.collect_only = collect_only self.active_sources = active_sources or {"all"} self.health = _health # Trading mode: "long" (buy/sell only), "short" (short/cover only), "both" self.trading_mode = trading_mode self.allow_long = trading_mode in ("long", "both") self.allow_short = trading_mode in ("short", "both") self.dm = DataManager() self.meta = MetaSystem() # 5 specialized MetaLearners + GrandMetaLearner # Each agent gets its own independent model instance (Bug clone fix) self.models = {c["name"]: StockModel(c["name"], c) for c in AGENT_CONFIGS} self.agents = [TradingAgent(c, self.models[c["name"]], self.meta) for c in AGENT_CONFIGS] # Agents 101-105: each of the 5 MetaLearners trades independently with $1,000. # They use _NeutralMeta so they rely purely on their own signal. # share_knowledge passes ALL 105 agents so they learn from each other. _ma_cfgs = [ {"name": "MetaTech", "risk_tolerance": 0.5, "max_position_pct": 0.20, "min_confidence": 0.55, "is_clone": False, "take_profit_mult": 1.3}, {"name": "MetaSentiment", "risk_tolerance": 0.5, "max_position_pct": 0.20, "min_confidence": 0.55, "is_clone": False, "take_profit_mult": 1.3}, {"name": "MetaVolatility", "risk_tolerance": 0.5, "max_position_pct": 0.20, "min_confidence": 0.55, "is_clone": False, "take_profit_mult": 1.3}, {"name": "MetaTrend", "risk_tolerance": 0.5, "max_position_pct": 0.20, "min_confidence": 0.55, "is_clone": False, "take_profit_mult": 1.3}, {"name": "MetaComposite", "risk_tolerance": 0.5, "max_position_pct": 0.20, "min_confidence": 0.55, "is_clone": False, "take_profit_mult": 1.3}, ] _ma_models = [self.meta.tech, self.meta.sentiment, self.meta.volatility, self.meta.trend, self.meta.composite] self.meta_agents = [TradingAgent(c, m, _NeutralMeta()) for c, m in zip(_ma_cfgs, _ma_models)] self._last_watcher_check = 0.0 self.cycle = 0 self.feat_rebuild_every = 50 self._retrain_scheduler = MarketAwareRetrainScheduler(self._retrain) self._last_chart_hour = -1 self._last_av_fetch = 0.0 self._last_report_slot = -1 # tracks last 30-min slot reported (0..N) self._last_state_save = 0.0 self._initial_train_done = False self._fetch_thread_running = False self._train_checkpoint_idx = self._load_train_checkpoint() # v9: round-robin resume self._last_train_check_ts = 0.0 # v9: throttle trigger check to every 5 min self._last_weekend_check_ts = 0.0 # v9: throttle weekend check # v9: Persistent one-per-day guard -- survives bot restarts _done = self._load_train_done() _today = datetime.now(MARKET_TZ).strftime("%Y-%m-%d") self._after_hours_train_done = (_done.get("eod_date") == _today) self._weekend_train_done = (_done.get("weekend_date") == _today) if self._after_hours_train_done: logger.info(f"[TrainGuard] EOD already done today ({_today}) -- skipping") if self._weekend_train_done: logger.info(f"[TrainGuard] Weekend train already done today ({_today}) -- skipping") self._last_heartbeat = 0.0 # debug heartbeat every 5 min self._last_github_push_ts = 0.0 # daily GitHub report push self._last_github_backup = 0.0 # GitHub log backup every 30 min def _source_active(self, source: str) -> bool: """Return True if the given data source is enabled. In collect-only mode respects --sources flag. In full trading mode all sources are always active. """ if not self.collect_only: return True return "all" in self.active_sources or source in self.active_sources def initialize(self): logger.info("=" * 60) logger.info("ML Trading System v8 - Alpaca + Finnhub + Sentiment Intelligence") logger.info(f"CPU cores : {PHYSICAL_CORES}") logger.info(f"Base agents : {BASE_AGENTS}") logger.info(f"Clone agents : {CLONE_AGENTS}") logger.info(f"Total agents : {NUM_AGENTS}") logger.info(f"MetaLearners : 5 agents trading independently (101-105, $1,000 each)") logger.info(f"Retrain : every 15 min during market hours + EOD") logger.info(f"Data sources : Alpaca WS (all IEX bars) + Finnhub (58/min) + AV + Polygon") logger.info(f"Charts : {CHARTS_DIR}/ hourly accuracy PNG") logger.info(f"Trade CSV : {TRADE_LOG_FILE}") if ALPACA_API_KEY and ALPACA_SECRET_KEY: logger.info(f"Alpaca : keys set, WebSocket will connect on start") else: logger.warning("Alpaca : no keys set (ALPACA_API_KEY / ALPACA_SECRET_KEY)") logger.warning(" Sign up free at alpaca.markets to enable ~8,000 symbol stream") logger.info("=" * 60) _init_trade_log() _log_ram("startup baseline") self.dm.load_sectors() _log_ram("after load_sectors") if not self.collect_only: _log_ram("before model load") for a in self.agents: a.model.load() self.meta.load() # meta_agent uses same model object as self.meta (already loaded above) _log_ram("after model load (100+ PKL files in memory)") load_all_agent_state(self.agents, self.meta, self.meta_agents) _log_ram("after agent state restore") else: logger.info("[COLLECT-ONLY] Agent/model loading SKIPPED -- trading disabled") _log_ram("before rebuild_all_features") n = self.dm.rebuild_all_features() _log_ram(f"after rebuild_all_features ({n} symbols)") logger.info(f"Features built for {n} symbols from existing DB data") if n > 0: _log_ram("before _retrain") self._retrain() _log_ram("after _retrain") self._initial_train_done = True counts = self.dm.get_combined_row_counts() total = sum(counts.values()) hist = len(self.dm._history_dbs) logger.info(f"DB: {total:,} quotes across {len(counts)} symbols (+{hist} history DB(s))") if total < 100: logger.info(f">>> First run --- need {MIN_ROWS_TO_TRADE} quotes/symbol to trade.") _log_ram("before Alpaca WS start") if self._source_active("alpaca"): self.dm.alpaca_ws.start() else: logger.info("[COLLECT-ONLY] Alpaca WS: DISABLED by --sources flag") self.dm.sentiment.start() logger.info("Sentiment : engine started (news + SEC + reddit + congress)") _log_ram("after sentiment engine start") start_resource_monitor() _resource_guard.start() _log_ram("fully initialized") logger.info("Ready.\n") return True def _fetch_loop(self): """ Background thread: Finnhub round-robin polling at 58 calls/min. Supplements Alpaca bars with sub-minute quote confirmation for the active 300-stock universe. """ _set_nice(-5) # P1 HIGH while self._fetch_thread_running: if is_market_open(): self.dm.fetch_next_quote() else: time.sleep(5) def _retrain(self): """ During market hours: skipped entirely. Training now happens after market close to avoid competing with Alpaca WebSocket, DB writes, and trading decisions simultaneously. This was the cause of silent crashes — parallel training 32 models on five hundred thousand rows while Alpaca was also writing to SQLite caused thread collisions that killed the process with no Python error logged. """ if is_market_open(): debug_logger.debug("RETRAIN_SKIPPED_MARKET_OPEN") return # WAL auto-checkpoint: runs each after-hours _retrain cycle try: import glob as _g for _db in _g.glob("data/trading_data_????????.db"): _wal = _db + "-wal" if os.path.exists(_wal): _wal_mb = os.path.getsize(_wal) // 1024 // 1024 if _wal_mb > 50: with sqlite3.connect(_db, timeout=5) as _wc: _res = _wc.execute("PRAGMA wal_checkpoint(PASSIVE)").fetchone() debug_logger.info( f"WAL_CHECKPOINT | {os.path.basename(_db)} | " f"was={_wal_mb}MB | result={_res}" ) except Exception as _we: debug_logger.debug(f"WAL_CHECKPOINT_ERR | {_we}") # After hours: just rebuild features and save state # Full model training handled by _after_hours_train() self.dm.rebuild_all_features() self.dm.update_sector_perf() save_all_agent_state(self.agents, self.meta, self.dm.prices, self.meta_agents) debug_logger.info("RETRAIN_AFTER_HOURS_FEATURES_REBUILT") def _load_train_checkpoint(self) -> int: """v9: Load the last completed agent index for round-robin resume.""" try: if os.path.exists(TRAIN_CHECKPOINT_FILE): d = json.load(open(TRAIN_CHECKPOINT_FILE)) idx = int(d.get("next_agent_idx", 0)) date = d.get("date", "") today = datetime.now(MARKET_TZ).strftime("%Y-%m-%d") if date == today: logger.info(f"[CHECKPOINT] Resuming training from agent index {idx} (crashed at {d.get('last_completed','?')})") return idx except Exception as e: debug_logger.warning(f"CHECKPOINT_LOAD_ERR | {e}") return 0 def _save_train_checkpoint(self, next_idx: int, last_completed: str): try: json.dump({ "date": datetime.now(MARKET_TZ).strftime("%Y-%m-%d"), "next_agent_idx": next_idx, "last_completed": last_completed, "saved_at": datetime.now(DISPLAY_TZ).strftime("%Y-%m-%d %H:%M:%S CST"), }, open(TRAIN_CHECKPOINT_FILE, "w"), indent=2) except Exception as e: debug_logger.warning(f"CHECKPOINT_SAVE_ERR | {e}") def _clear_train_checkpoint(self): try: if os.path.exists(TRAIN_CHECKPOINT_FILE): os.remove(TRAIN_CHECKPOINT_FILE) except Exception: pass def _load_train_done(self) -> dict: """Load the persistent one-per-day training guard file.""" try: if os.path.exists(TRAIN_DONE_FILE): return json.load(open(TRAIN_DONE_FILE)) except Exception: pass return {} def _save_train_done(self, key: str): """ Write today's date under 'key' (eod_date or weekend_date) to TRAIN_DONE_FILE so that bot restarts on the same calendar day will not re-trigger training. """ try: today = datetime.now(MARKET_TZ).strftime("%Y-%m-%d") data = self._load_train_done() data[key] = today data["saved_at"] = datetime.now(DISPLAY_TZ).strftime("%Y-%m-%d %H:%M:%S CST") json.dump(data, open(TRAIN_DONE_FILE, "w"), indent=2) logger.info(f"[TrainGuard] {key} written → {today}") except Exception as e: debug_logger.warning(f"TRAIN_DONE_SAVE_ERR | {e}") def _after_hours_train(self): """ v9: LIFETIME LEARNING EOD training. Flow: 1. Write pre-train report 2. Rebuild features (incremental, fast) 3. Save today's feature rows to features/daily/features_YYYYMMDD.pkl.gz 4. Load today's feature file as training dataset 5. For each agent sequentially: a. Check tree_count -- if > DISTILL_THRESHOLD, distill first b. Add DAILY_TRAIN_TREES (50) new trees via init_model c. Checkpoint after each agent 6. Write post-train report Agents never forget. Each day adds knowledge on top of all previous days. Monday's trees are still in the model when it runs on Friday of next year. """ t_start = time.time() now_str = datetime.now(DISPLAY_TZ).strftime("%Y-%m-%d %H:%M:%S CST") logger.info(f"EOD training started at {now_str}") debug_logger.info( f"TRAIN_START | symbols={len(self.dm.features)} | " f"models={len(self.models)} | checkpoint_idx={self._train_checkpoint_idx}" ) # PRE-TRAIN REPORT _log_ram("EOD train start") logger.info("Writing pre-train report...") try: _write_human_report(self.agents + self.meta_agents, self.dm, self.meta) _write_claude_handoff(self.agents + self.meta_agents, self.dm, self.meta) except Exception as e: debug_logger.warning(f"PRE_TRAIN_REPORT_ERR | {e}") logger.info("Pre-train report written.") # FEATURE REBUILD (incremental -- processes only today's new bars) n = self.dm.rebuild_all_features() logger.info(f" Features rebuilt: {n} symbols") self.dm.update_sector_perf() if not self.dm.features: logger.warning(" No features -- skipping EOD train.") return total_rows = sum(len(f) for f in self.dm.features.values()) debug_logger.info( f"TRAIN_DATA_SIZE | total_rows={total_rows:,} | " f"symbols={len(self.dm.features)}" ) # SAVE TODAY'S FEATURE FILE (permanent record) logger.info("Saving today's feature file...") today_path = self.dm.save_daily_features() if not today_path: logger.warning(" Daily feature save failed -- using in-memory features as fallback") # LOAD TODAY'S TRAINING DATASET FROM FILE # This decouples training from the in-memory feature store. # Even if RAM feature store is capped, training sees today's full day. today_str = datetime.now(MARKET_TZ).strftime("%Y%m%d") training_data = self.dm.load_features_for_training(today_str) if not training_data: logger.warning(" Could not load daily file -- falling back to in-memory features") training_data = self.dm.features logger.info( f" Training dataset: {sum(len(v) for v in training_data.values()):,} rows " f"across {len(training_data)} symbols" ) # SEQUENTIAL AGENT TRAINING WITH LIFETIME ACCUMULATION ordered_models = sorted(self.models.items(), key=lambda x: x[0]) n_models = len(ordered_models) start_idx = self._train_checkpoint_idx % n_models trained_count = 0 failed_count = 0 distilled_count = 0 # v9: Worker-pool training -- N workers run simultaneously. # Each worker: pick model → train → save → NULL booster → malloc_trim → done. # Workers never hold more than 1 booster at a time. # Peak RAM = base + (n_workers × ~320MB peak per booster). # After each agent completes its worker fully releases before taking next. # --max-workers controls concurrency. Default 8 on 32GB, 4 on 7.3GB VM. N_WORKERS = MAX_WORKERS_CAP if MAX_WORKERS_CAP is not None else 8 logger.info( f"Training {n_models} agents | {N_WORKERS} workers | " f"each worker: train → save → release → next | " f"DAILY_TREES={DAILY_TRAIN_TREES}" ) import gc, ctypes def _train_save_release(job): """ Complete lifecycle for one agent in a worker thread: 1. Train -- builds booster, adds DAILY_TRAIN_TREES trees 2. Save -- writes pkl to disk 3. Null -- drops Python reference to booster 4. Trim -- malloc_trim(0) tells glibc to return freed pages to OS 5. Return -- worker is now free for next agent Peak RAM per worker = ~320MB during step 1. After step 4 that memory is fully returned to OS. N concurrent workers = N × 320MB peak, then drops back to base. """ idx, name, model = job t0 = time.time() # Distillation check if model.tree_count > DISTILL_THRESHOLD: logger.info( f" [{idx+1}/{n_models}] {name} | " f"tree_count={model.tree_count} > {DISTILL_THRESHOLD} | DISTILLING" ) distill_data = self.dm.load_distill_dataset(max_days=90) if distill_data: model.distill(distill_data) # Step 1: Train err = None success = False try: model._override_trees = DAILY_TRAIN_TREES success = model.train(training_data) model._override_trees = None except Exception as e: model._override_trees = None err = e elapsed = time.time() - t0 # Step 2: Save (regardless of success -- preserves existing model) if success: model.save() # Step 3+4: Null booster + force OS page return # This is the critical step -- without malloc_trim RSS stays high # even after gc.collect(), because glibc holds freed heap as a pool. model.lgb_clf = None model.lgb_reg = None gc.collect() try: ctypes.CDLL("libc.so.6").malloc_trim(0) except Exception: pass return (idx, name, success, elapsed, err) # Build full job list from checkpoint position all_jobs = [ ((start_idx + offset) % n_models, ordered_models[(start_idx + offset) % n_models][0], ordered_models[(start_idx + offset) % n_models][1]) for offset in range(n_models) ] _log_ram(f"before worker-pool training ({N_WORKERS} workers)") # Submit all jobs to the pool -- workers self-regulate via the pool queue. # Each worker completes its full lifecycle (train+save+release) before # picking up the next job. So at any moment exactly N_WORKERS boosters # exist in RAM simultaneously, then each drops as soon as it finishes. with ThreadPoolExecutor(max_workers=N_WORKERS) as pool: futures = {pool.submit(_train_save_release, job): job for job in all_jobs} for fut in as_completed(futures): idx, name, success, elapsed, err = fut.result() if err is not None: failed_count += 1 debug_logger.error( f"TRAIN_AGENT_ERR | {name} | {err}", exc_info=False) logger.error( f" [{idx+1}/{n_models}] {name} FAILED {elapsed:.1f}s: {err}") elif success: trained_count += 1 # Reload model from pkl for inference (booster was nulled) ordered_models[idx][1].load() logger.info( f" [{idx+1}/{n_models}] {name} | " f"+{DAILY_TRAIN_TREES} trees | " f"val={ordered_models[idx][1].val_acc:.3f} | " f"IC={ordered_models[idx][1].rolling_ic():.3f} | " f"{elapsed:.1f}s | released" ) else: failed_count += 1 logger.warning( f" [{idx+1}/{n_models}] {name} no data {elapsed:.1f}s") # Checkpoint after each completion next_idx = (idx + 1) % n_models self._train_checkpoint_idx = next_idx self._save_train_checkpoint( next_idx, name if not err else f"FAILED:{name}") _log_ram(f"after {name} released ({trained_count} done)") _log_ram("worker-pool training complete") # META-LEARNER try: logger.info("Training MetaLearner...") share_knowledge(self.agents + self.meta_agents, self.meta, self.dm.features) self.meta.save() logger.info("MetaLearner done.") except Exception as e: debug_logger.error(f"TRAIN_META_ERR | {e}", exc_info=True) save_all_agent_state(self.agents, self.meta, self.dm.prices, self.meta_agents) elapsed_total = time.time() - t_start logger.info( f"EOD training complete: {trained_count} trained, " f"{distilled_count} distilled, {failed_count} failed | " f"{elapsed_total:.1f}s total" ) debug_logger.info( f"TRAIN_COMPLETE | trained={trained_count}/{n_models} | " f"distilled={distilled_count} | failed={failed_count} | " f"elapsed={elapsed_total:.1f}s" ) if failed_count == 0: self._clear_train_checkpoint() logger.info("Training checkpoint cleared (full success).") # POST-TRAIN REPORT logger.info("Writing post-train report...") try: _write_human_report(self.agents + self.meta_agents, self.dm, self.meta) _write_claude_handoff(self.agents + self.meta_agents, self.dm, self.meta) except Exception as e: debug_logger.warning(f"POST_TRAIN_REPORT_ERR | {e}") logger.info("Post-train report written.") try: generate_resource_graph() except Exception as e: debug_logger.warning(f"RESOURCE_GRAPH_SKIP | {e}") run_backup() # Mark EOD train as done for today -- survives restarts self._save_train_done("eod_date") def _weekend_train(self): """ v9: WEEKEND DEEP TRAIN -- Saturday once per week. Combines Mon-Fri daily feature files into a single weekly dataset. The full-week context makes rolling features (ret_5, ret_10, slope_20, weekly momentum, etc.) far more informative than any single day. Adds WEEKLY_TRAIN_TREES (100) trees per agent -- double a daily update. Agents that ran EOD training all week will end Saturday with an extra 100 trees trained on the week's complete picture. Over 52 weekends per year that's 5,200 additional trees of weekly context on top of the 12,600 from daily updates -- agents develop genuine week-over-week pattern recognition. """ t_start = time.time() now_str = datetime.now(DISPLAY_TZ).strftime("%Y-%m-%d %H:%M:%S CST") logger.info(f"Weekend deep train started at {now_str}") # Build weekly dataset from Mon-Fri daily files weekly_data = self.dm.build_weekly_dataset() if not weekly_data: logger.warning("Weekend train: no weekly data available -- skipping") return weekly_rows = sum(len(v) for v in weekly_data.values()) logger.info( f" Weekly dataset: {weekly_rows:,} rows | " f"{len(weekly_data)} symbols | +{WEEKLY_TRAIN_TREES} trees/agent" ) ordered_models = sorted(self.models.items(), key=lambda x: x[0]) trained = 0 failed = 0 for i, (name, model) in enumerate(ordered_models, 1): try: model._override_trees = WEEKLY_TRAIN_TREES success = model.train(weekly_data) model._override_trees = None if success: model.save() trained += 1 logger.info( f" [{i}/{len(ordered_models)}] {name} | " f"+{WEEKLY_TRAIN_TREES} trees | total={model.tree_count} | " f"val={model.val_acc:.3f}" ) else: model._override_trees = None failed += 1 except Exception as e: model._override_trees = None failed += 1 debug_logger.error(f"WEEKEND_TRAIN_ERR | {name} | {e}", exc_info=True) logger.error(f" [{i}/{len(ordered_models)}] {name} FAILED: {e}") time.sleep(0.5) # Meta-learner on weekly data try: share_knowledge(self.agents + self.meta_agents, self.meta, self.dm.features) self.meta.save() except Exception as e: debug_logger.error(f"WEEKEND_META_ERR | {e}", exc_info=True) save_all_agent_state(self.agents, self.meta, self.dm.prices, self.meta_agents) elapsed = time.time() - t_start logger.info( f"Weekend deep train complete: {trained} trained, " f"{failed} failed | {elapsed:.1f}s total" ) # Weekend post-train report try: _write_human_report(self.agents + self.meta_agents, self.dm, self.meta) _write_claude_handoff(self.agents + self.meta_agents, self.dm, self.meta) except Exception as e: debug_logger.warning(f"WEEKEND_REPORT_ERR | {e}") # Mark weekend train as done for today -- survives restarts self._save_train_done("weekend_date") def _maybe_chart(self): """Generate accuracy chart every hour.""" now = time.time() if now - getattr(self, '_last_chart_ts', 0) < 3600: return self._last_chart_ts = now try: threading.Thread( target=lambda: generate_accuracy_chart(self.agents, self.dm.db), daemon=True, name="ChartGen" ).start() except Exception as e: debug_logger.warning(f"CHART_ERR | {e}") def _maybe_report(self): """Write hourly report every 30 minutes during market hours.""" now = time.time() if now - getattr(self, '_last_report_ts', 0) < 1800: return self._last_report_ts = now try: threading.Thread( target=lambda: hourly_report(self.agents, self.dm, self.meta), daemon=True, name="HourlyReport" ).start() except Exception as e: debug_logger.warning(f"REPORT_ERR | {e}") def _maybe_write_dashboard(self): """Write dashboard HTML every 60 s (lightweight: reads JSON, builds string).""" now = time.time() if now - getattr(self, "_last_dashboard_ts", 0) < 60: return self._last_dashboard_ts = now try: threading.Thread( target=lambda: _write_html_dashboard(self.agents, self.dm, self.meta), daemon=True, name="DashWriter" ).start() except Exception as e: debug_logger.warning(f"DASHBOARD_WRITE_ERR | {e}") def _maybe_av_fetch(self): """Trigger Alpha Vantage burst fetch periodically.""" now = time.time() if now - getattr(self, '_last_av_ts', 0) < 300: return self._last_av_ts = now try: self.dm.fetch_alpha_vantage_burst() except Exception as e: debug_logger.warning(f"AV_FETCH_ERR | {e}") def _maybe_save_state(self): """Save agent state every 15 minutes.""" now = time.time() if now - getattr(self, '_last_state_save_ts', 0) < 900: return self._last_state_save_ts = now threading.Thread( target=lambda: save_all_agent_state(self.agents, self.meta, self.dm.prices, self.meta_agents), daemon=True, name="StateSave" ).start() def _maybe_check_watcher(self): """Rotate DB if needed -- called each cycle.""" try: self.dm._rotate_db_if_needed() except Exception as e: debug_logger.warning(f"WATCHER_CHECK_ERR | {e}") def _maybe_local_backup(self): """ Smart local backup with frequency based on market schedule: - Weekday market hours : every 30 min (new data every minute) - Weekday off-hours : every 2 hours (EOD train done, just protecting state) - Weekend : every 6 hours (no new data, just safety net) Renamed from _maybe_github_backup (misleading -- this is local-only). """ now = time.time() now_et = datetime.now(MARKET_TZ) dow = now_et.weekday() # 0=Mon, 6=Sun is_weekend = dow >= 5 if is_weekend: interval = 21600 # 6 hours on weekends elif is_market_open(): interval = 1800 # 30 min during market hours else: interval = 7200 # 2 hours off-hours weekdays if now - getattr(self, '_last_backup_ts', 0) < interval: return self._last_backup_ts = now threading.Thread( target=run_backup, daemon=True, name="LocalBackup" ).start() def _maybe_github_push(self): """ Push key report files to GitHub Stock_Bot/backup/ once per day. This allows remote review of bot state without SSH access. Files pushed: human_report.txt, claude_handoff.json, TRADE_LOG.csv, agent_state.json, train_done.json, last 500 lines of trading.log """ now = time.time() if now - getattr(self, '_last_github_push_ts', 0) < 86400: # once per day return self._last_github_push_ts = now threading.Thread( target=self._do_github_push, daemon=True, name="GithubPush" ).start() def _do_github_push(self): """Upload key files to GitHub via REST API.""" import urllib.request, base64, json as _json _set_nice(19) PAT = "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" REPO = "Jake-Culberson/Claud-Code" API = f"https://api.github.com/repos/{REPO}/contents" hdrs = {"Authorization": f"Bearer {PAT}", "Content-Type": "application/json"} def _push(remote_path: str, content_bytes: bytes, msg: str): url = f"{API}/{remote_path}" # Get current SHA if file exists req = urllib.request.Request(url, headers={"Authorization": f"Bearer {PAT}"}) sha = "" try: with urllib.request.urlopen(req) as resp: sha = _json.loads(resp.read()).get("sha", "") except Exception: pass payload = {"message": msg, "content": base64.b64encode(content_bytes).decode()} if sha: payload["sha"] = sha req2 = urllib.request.Request(url, data=_json.dumps(payload).encode(), method="PUT", headers=hdrs) try: with urllib.request.urlopen(req2) as resp: return _json.loads(resp.read()).get("commit", {}).get("sha", "")[:10] except Exception as e: debug_logger.warning(f"GITHUB_PUSH_ERR | {remote_path} | {e}") return "" now_str = datetime.now(DISPLAY_TZ).strftime("%Y-%m-%d %H:%M CST") pushed = 0 files_to_push = [ ("logs/human_report.txt", "Stock_Bot/backup/human_report.txt"), ("logs/claude_handoff.json", "Stock_Bot/backup/claude_handoff.json"), ("logs/TRADE_LOG.csv", "Stock_Bot/backup/TRADE_LOG.csv"), ("logs/agent_state.json", "Stock_Bot/backup/agent_state.json"), ("logs/train_done.json", "Stock_Bot/backup/train_done.json"), ("logs/train_checkpoint.json", "Stock_Bot/backup/train_checkpoint.json"), ("logs/bot_dashboard.html", "Stock_Bot/backup/bot_dashboard.html"), ("logs/human_report.txt", "Stock_Bot/backup/human_report.txt"), ("logs/TRADE_LOG.csv", "Stock_Bot/backup/TRADE_LOG.csv"), ("logs/claude_handoff.json", "Stock_Bot/backup/claude_handoff.json"), ] for local, remote in files_to_push: if not os.path.exists(local): continue try: content = open(local, "rb").read() if len(content) > 20 * 1024 * 1024: # skip files > 20MB continue sha = _push(remote, content, f"bot: daily push {now_str}") if sha: pushed += 1 except Exception as e: debug_logger.warning(f"GITHUB_PUSH_FILE_ERR | {local} | {e}") # Last 500 lines of trading.log (trimmed to stay under size limit) if os.path.exists("trading.log"): try: with open("trading.log", "r") as f: lines = f.readlines() tail = "".join(lines[-500:]).encode() _push("Stock_Bot/backup/trading_log_tail.txt", tail, f"bot: daily push {now_str}") pushed += 1 except Exception as e: debug_logger.warning(f"GITHUB_PUSH_LOG_ERR | {e}") # Push model manifest (list of PKL files + sizes + timestamps) # Full PKL files are too large for GitHub but a manifest lets us # audit model state remotely. try: import glob as _gl manifest = [] for _pkl in sorted(_gl.glob(os.path.join(MODEL_SAVE_DIR, "*.pkl"))): _st = os.stat(_pkl) manifest.append({ "name": os.path.basename(_pkl), "size_kb": _st.st_size // 1024, "mtime": _st.st_mtime, }) manifest_bytes = _json.dumps({ "generated": now_str, "count": len(manifest), "models": manifest }, indent=2).encode() _push("Stock_Bot/backup/model_manifest.json", manifest_bytes, f"bot: model manifest {now_str}") pushed += 1 except Exception as _me: debug_logger.warning(f"GITHUB_MANIFEST_ERR | {_me}") logger.info(f"GitHub daily push complete: {pushed} files → Stock_Bot/backup/") debug_logger.info(f"GITHUB_PUSH_COMPLETE | files={pushed}") def _rebuild_features_background(self): """Background feature rebuild -- runs in FeatRebuild thread.""" try: _set_nice(0) self.dm.rebuild_all_features() except Exception as e: debug_logger.error(f"FEAT_REBUILD_BG_ERR | {e}", exc_info=True) def _trading_pass(self): """ v9: Core trading loop — runs every ~1 second during market hours. Flow: 1. Exit checks on all agents (ATR stop-loss, take-profit, drawdown reset) 2. Build tradeable symbol list + stacked feature matrix 3. Per-agent: batch-predict all symbols in one LGB call 4. Signals passing IC gate + min_conf: execute buy/sell/short/cover Uses predict_batch() so each agent makes ONE LightGBM call per cycle. Full predict() only called at actual trade execution to get pred_pct. """ prices = self.dm.prices # Snapshot features dict to prevent RuntimeError: dictionary changed size during iteration # New symbols added by WebSocket during this cycle are safely excluded until next cycle features = dict(self.dm.features) # ── Step 1: exit checks (stop-loss, take-profit, drawdown reset) ── for ag in self.agents: ag.exit_checks(prices) ag.check_reset(prices) # ── Step 2: build tradeable symbol universe ──────────────────────── syms: list = [] vecs: list = [] for sym, df in features.items(): if df is None or len(df) < MIN_ROWS_TO_TRADE: continue price = prices.get(sym, 0.0) if price <= 0: continue try: vec = df[FEATURE_COLS].values[-1].astype(np.float32) if not np.all(np.isfinite(vec)): continue except Exception: continue syms.append(sym) vecs.append(vec) if not syms: return feat_matrix = np.array(vecs, dtype=np.float32) # Batch meta-confidence: 5 sub-MetaLearners + GrandMetaLearner try: _meta_scores, meta_batch, meta_agree = self.meta.predict_batch_full(feat_matrix) except Exception: _meta_scores = np.full((len(syms), 5), 0.5, dtype=np.float32) meta_batch = np.full(len(syms), 0.5, dtype=np.float32) meta_agree = np.full(len(syms), 5, dtype=np.int32) # ATR index for dynamic stop sizing try: atr_idx = FEATURE_COLS.index("atr_pct") except ValueError: atr_idx = -1 # ── Step 3 & 4: per-agent batch predict + signal execution ──────── for ag in self.agents: if not ag.model.trained: continue # IC gate: agent must have meaningful signal quality if (hasattr(ag.model, "rolling_ic") and len(ag.model.ic_history) >= 3 and ag.model.rolling_ic() < IC_MIN_VOTE_THRESHOLD): continue # One LGB predict call for all symbols try: directions, confidences = ag.model.predict_batch(feat_matrix) except Exception as _pe: debug_logger.warning(f"BATCH_PREDICT_ERR | {ag.name} | {_pe}") continue for i, sym in enumerate(syms): price = prices.get(sym, 0.0) if price <= 0: continue d = int(directions[i]) cf = float(confidences[i]) meta_cf = float(meta_batch[i]) agree = int(meta_agree[i]) eff_cf = cf * 0.7 + meta_cf * 0.3 if eff_cf < ag.min_conf: continue # MetaLearner gate: >=2 of 5 sub-MetaLearners must agree if self.meta.gate_active and agree < 2: continue atr_pct = float(vecs[i][atr_idx]) if atr_idx >= 0 else 0.02 if d == 1: # bullish if sym in ag.short_positions: ag.cover(sym, price, reason="signal") elif sym not in ag.positions and self.allow_long: _, pred_pct, _ = ag.model.predict(vecs[i]) ag.buy(sym, price, pred_pct or 0.0, cf, meta_cf, eff_cf, atr_pct) elif d == 0: # bearish if sym in ag.positions: ag.sell(sym, price, reason="signal") elif sym not in ag.short_positions and self.allow_short: _, pred_pct, _ = ag.model.predict(vecs[i]) ag.short(sym, price, pred_pct or 0.0, cf, meta_cf, eff_cf) # ── 5 MetaLearner agents (101-105) trading pass ────────────────────────── for _ma in self.meta_agents: if not _ma.model.trained: continue _ma.exit_checks(prices) _ma.check_reset(prices) try: meta_dirs, meta_cfs = _ma.model.predict_batch(feat_matrix) for i, sym in enumerate(syms): price = prices.get(sym, 0.0) if price <= 0: continue md = int(meta_dirs[i]) mcf = float(meta_cfs[i]) eff = mcf * 0.7 + 0.5 * 0.3 atr_pct = float(vecs[i][atr_idx]) if atr_idx >= 0 else 0.02 if md == 1 and eff >= _ma.min_conf: if sym in _ma.short_positions: _ma.cover(sym, price, reason="signal") elif sym not in _ma.positions and self.allow_long: _, pred_pct, _ = _ma.model.predict(vecs[i]) _ma.buy(sym, price, pred_pct or 0.0, mcf, 0.5, eff, atr_pct) elif md == 0 and eff >= _ma.min_conf: if sym in _ma.positions: _ma.sell(sym, price, reason="signal") elif sym not in _ma.short_positions and self.allow_short: _, pred_pct, _ = _ma.model.predict(vecs[i]) _ma.short(sym, price, pred_pct or 0.0, mcf, 0.5, eff) except Exception as _mpe: debug_logger.warning(f"META_AGENT_TRADE_ERR | {_ma.name} | {_mpe}") debug_logger.debug( f"TRADING_PASS | syms={len(syms)} | " f"cycle={self.cycle}" ) def run(self): if not self.initialize(): return if self.collect_only: self._run_collect_only() return logger.info("Main loop running. Ctrl+C to stop.\n") debug_logger.info("MAIN_LOOP_START") _was_open = False # Start Finnhub background fetch thread (58/min, round-robin) self._fetch_thread_running = True fetch_thread = threading.Thread(target=self._fetch_loop, daemon=True, name="FinnhubFetch") fetch_thread.start() logger.info("Finnhub fetch thread started (58 calls/min round-robin).") debug_logger.info("FINNHUB_THREAD_STARTED") while True: try: open_now = is_market_open() now_et = datetime.now(MARKET_TZ) # --- Debug heartbeat every 5 minutes --- if time.time() - self._last_heartbeat > 300: self._last_heartbeat = time.time() active_threads = [t.name for t in threading.enumerate()] # Count symbols with non-zero sentiment data sent_syms = len(self.dm.sentiment.db.get_all_symbols()) debug_logger.info( f"HEARTBEAT | market_open={open_now} | " f"cycle={self.cycle} | " f"alpaca_bars={self.dm.alpaca_ws.bars_received():,} | " f"features={len(self.dm.features)} | " f"sentiment_symbols={sent_syms} | " f"threads={active_threads}" ) # --- Market just opened --- if open_now and not _was_open: logger.info("Market just opened.") debug_logger.info("MARKET_OPEN") self._after_hours_train_done = False # reset for next close # Clear persisted EOD flag so tonight's train fires try: _d = self._load_train_done() _d.pop("eod_date", None) json.dump(_d, open(TRAIN_DONE_FILE, "w"), indent=2) except Exception: pass self.dm.on_market_open() _was_open = open_now # --- Market hours: collect data and trade --- if open_now: self.cycle += 1 if self.cycle % self.feat_rebuild_every == 0: # v9: One-at-a-time guard -- only spawn a new FeatRebuild # thread if the previous one has finished. Without this, # threads stack (32 were observed simultaneously), each # holding thousands of symbol DataFrames, causing OOM. _active_rebuilds = sum( 1 for t in threading.enumerate() if t.name == "FeatRebuild" and t.is_alive() ) if _active_rebuilds == 0: threading.Thread( target=self._rebuild_features_background, daemon=True, name="FeatRebuild" ).start() else: debug_logger.debug( f"FEAT_REBUILD_SKIP | {_active_rebuilds} already running" ) _t_loop_start = time.time() if self.dm.features: self._trading_pass() self._maybe_chart() self._maybe_report() self._maybe_write_dashboard() self._maybe_av_fetch() self._maybe_save_state() self._maybe_check_watcher() self._maybe_local_backup() self._maybe_github_push() _t_loop_elapsed = time.time() - _t_loop_start if _t_loop_elapsed > 2.0: debug_logger.warning( f"GIL_STALL | main loop took {_t_loop_elapsed:.2f}s " f"(normal <0.2s) -- WebSocket ping thread was starved. " f"cycle={self.cycle} features={len(self.dm.features)} " f"agents_trained=" f"{sum(1 for a in self.agents if a.model.trained)} " f"resource={_resource_guard.status} " f"load={_resource_guard.load_str}" ) time.sleep(max(0, 1.0 - _t_loop_elapsed)) # --- After market close: trigger training once at sixteen-oh-one --- else: # v9: EOD training trigger -- fires once per trading day. # # WINDOWS (ET, weekdays only): # Post-close : hour >= 16, minute >= 15 (4:15 PM onwards) # -- Primary window: 15 min after close lets final bars settle # -- Catches any restart after 4:15 PM on a trading day # Pre-market : hour <= 8, minute <= 30 (up to 8:30 AM) # -- Catches restarts overnight/early morning if previous # day's training was missed (e.g. bot was restarted late) # # CHECK RATE: every 5 minutes max (time.sleep(30) x 10 = 5 min) # No CPU cost -- just a timestamp compare and two integer checks. _now_ts = time.time() _check_due = (_now_ts - self._last_train_check_ts) >= 300 # 5 min if _check_due and not self._after_hours_train_done and now_et.weekday() < 5: self._last_train_check_ts = _now_ts _h, _m = now_et.hour, now_et.minute # Post-close window: 4:15 PM ET or later _post_close = (_h > 16) or (_h == 16 and _m >= 15) # Pre-market window: up to 8:30 AM ET (catches overnight restart) _pre_market = (_h < 8) or (_h == 8 and _m <= 30) _in_window = _post_close or _pre_market if _in_window: self._after_hours_train_done = True logger.info( f"EOD training trigger: {now_et.strftime('%H:%M')} ET | " f"{'post-close' if _post_close else 'pre-market'} window" ) debug_logger.info( f"AFTER_HOURS_TRAIN_TRIGGER | window={'post_close' if _post_close else 'pre_market'} | " f"time={now_et.strftime('%H:%M')} ET" ) threading.Thread( target=self._after_hours_train, daemon=False, name="AfterHoursTrain" ).start() # v9: Weekend deep train -- Saturday OR Sunday, once per weekend # 6 AM - 10 PM ET window gives time to complete on either day # Sunday included so a missed Saturday still fires _is_weekend = (now_et.weekday() in (5, 6)) # 5=Sat, 6=Sun _sat_window = _is_weekend and 6 <= now_et.hour <= 22 _wt_check_due = (time.time() - self._last_weekend_check_ts) >= 300 if _sat_window and not self._weekend_train_done and _wt_check_due: self._last_weekend_check_ts = time.time() self._weekend_train_done = True logger.info( f"Weekend deep train trigger: " f"{now_et.strftime('%A %H:%M')} ET" ) threading.Thread( target=self._weekend_train, daemon=False, name="WeekendTrain" ).start() # Reset weekend flag on Monday if now_et.weekday() == 0 and self._weekend_train_done: self._weekend_train_done = False # Clear persisted weekend flag so next Saturday fires try: _d = self._load_train_done() _d.pop("weekend_date", None) json.dump(_d, open(TRAIN_DONE_FILE, "w"), indent=2) except Exception: pass self._maybe_local_backup() self._maybe_github_push() self._maybe_write_dashboard() secs = seconds_until_open() if int(secs) % 3600 < 31: logger.info(f"Market closed -- {secs/3600:.2f}h until open.") time.sleep(30) except KeyboardInterrupt: logger.info("\nShutting down...") debug_logger.info("SHUTDOWN_KEYBOARD_INTERRUPT") self._fetch_thread_running = False self.dm.alpaca_ws.stop() self.dm.sentiment.stop() for a in self.agents: a.model.save() self.meta.save() save_all_agent_state(self.agents, self.meta, self.dm.prices, self.meta_agents) hourly_report(self.agents, self.dm, self.meta) generate_accuracy_chart(self.agents, self.dm.db) logger.info("All models saved. Goodbye.") break except Exception as e: logger.error(f"Loop error: {e}", exc_info=True) debug_logger.error( f"MAIN_LOOP_EXCEPTION | {type(e).__name__}: {e}", exc_info=True ) time.sleep(5) # ============================================================================= if __name__ == "__main__": # v9: uvloop -- ~2x asyncio throughput via Cython + libuv if UVLOOP_OK: uvloop.install() print("[v9] uvloop installed") import argparse as _ap _p = _ap.ArgumentParser( prog="StockTrading.py", description="ML Stock Trading Bot v8 -- Alpaca + Finnhub + Sentiment", formatter_class=_ap.RawDescriptionHelpFormatter, epilog=( "Examples:\n" " python3 StockTrading.py # full trading mode\n" " python3 StockTrading.py --collect-only # all sources, no trading\n" " python3 StockTrading.py --collect-only --sources alpaca\n" " python3 StockTrading.py --collect-only --sources alpaca,finnhub\n" " python3 StockTrading.py --collect-only --sources alpaca,finnhub,news,sec\n" " python3 StockTrading.py --collect-only --sources all\n" "\nValid source names: alpaca finnhub news sec stocktwits reddit congress all" ), ) _p.add_argument("--collect-only", action="store_true", default=False, help="Run data collection pipelines only. No agents loaded, no trading. " "Use to test/debug data sources independently.") _p.add_argument("--sources", default="all", metavar="SRC[,SRC,...]", help="Comma-separated sources to enable (only relevant with --collect-only). " "Options: alpaca finnhub news sec stocktwits reddit congress all. Default: all") _p.add_argument("--mode", default="long", choices=["long", "short", "both"], help="Trading direction mode. " "long=buy/sell only (default), " "short=short/cover only, " "both=full bidirectional trading") _p.add_argument("--ram-cap", type=float, default=None, metavar="GB", help="Hard RAM ceiling in GB. Bot will never allocate beyond this. " "If not specified, runs uncapped at max efficiency. " "Example: --ram-cap 6.0 sets a 6 GB ceiling on a 7.3 GB VM.") _p.add_argument("--max-workers", type=int, default=None, metavar="N", help="Hard cap on concurrent threads/workers across feature rebuild, " "training, and all ThreadPoolExecutor usage. " "If not specified, the bot auto-scales to CPU count (max efficiency). " "Example: --max-workers 4 limits to 4 concurrent threads on any VM.") _args = _p.parse_args() _raw = {s.strip().lower() for s in _args.sources.split(",")} _bad = _raw - VALID_SOURCES if _bad: print(f"[ERROR] Unknown sources: {_bad}") print(f" Valid options : {sorted(VALID_SOURCES)}") exit(1) # v9: RAM cap + worker cap -- apply args to module-level globals # Written via sys.modules to avoid Python's global-in-main-block restriction. import sys as _sys _this_mod = _sys.modules.get(__name__, _sys.modules["__main__"]) if _args.ram_cap is not None: _this_mod.RAM_CAP_MB = int(_args.ram_cap * 1024) print( f"[RAM CAP] Hard ceiling: {_args.ram_cap:.1f} GB ({_this_mod.RAM_CAP_MB:,} MB) | " f"ResourceGuard throttle at: {int(_this_mod.RAM_CAP_MB*0.90):,} MB | " f"pause at: {int(_this_mod.RAM_CAP_MB*0.97):,} MB" ) else: _this_mod.RAM_CAP_MB = None print("[RAM CAP] Uncapped -- running at max efficiency") if _args.max_workers is not None: _this_mod.MAX_WORKERS_CAP = max(1, _args.max_workers) print( f"[MAX WORKERS] Hard cap: {_this_mod.MAX_WORKERS_CAP} concurrent threads | " f"Applies to: feature rebuild, training, ThreadPoolExecutor" ) else: _this_mod.MAX_WORKERS_CAP = None print("[MAX WORKERS] Uncapped -- auto-scales to CPU count") if _args.collect_only: print(f"\n[COLLECT-ONLY] Starting data collection mode") print(f"[COLLECT-ONLY] Sources : {', '.join(sorted(_raw))}") print(f"[COLLECT-ONLY] Trading : DISABLED") print(f"[COLLECT-ONLY] Log file : trading.log\n") elif not FINNHUB_API_KEY: print("\n[ERROR] Set your Finnhub API key first!") print(" export FINNHUB_API_KEY=your_key_here\n") exit(1) if not _args.collect_only: print(f"\n[MODE] Trading mode: {_args.mode.upper()}") if _args.mode == "short": print("[MODE] Short selling ONLY -- no long positions") elif _args.mode == "both": print("[MODE] Bidirectional -- longs AND shorts enabled") else: print("[MODE] Long only -- standard buy/sell (default)") print() TradingSystem(collect_only=_args.collect_only, active_sources=_raw, trading_mode=_args.mode).run() # ================================================================================ # ORACLE CLOUD TRADING BOT — MANAGEMENT GUIDE # jakec | VM IP: 193.122.158.156 # ================================================================================ # ================================================================================ # FULL SYNC SCRIPT (saves everything from VM to your PC) # Save as: C:\Users\jakec\Desktop\Oracle_trader\sync_trading.ps1 # Replaces the old version — now downloads ALL important files. # ================================================================================ # # $key = "C:\Users\jakec\Desktop\Oracle_trader\ssh-key-2026-03-07.key" # $vm = "opc@193.122.158.156" # $dst = "C:\Users\jakec\Desktop\Pythoncoding\Stocks\" # # # Create charts folder locally if it doesn't exist # New-Item -ItemType Directory -Force -Path "$dst\charts" | Out-Null # New-Item -ItemType Directory -Force -Path "$dst\saved_models" | Out-Null # # # Core logs and state files # scp -i $key "$vm`:~/trading/hourly_report.log" "$dst" # scp -i $key "$vm`:~/trading/trading_data.log" "$dst" # scp -i $key "$vm`:~/trading/agent_state.json" "$dst" # scp -i $key "$vm`:~/trading/TRADE_LOG.csv" "$dst" # scp -i $key "$vm`:~/trading/debug.log" "$dst" # # # Database (main data store — gets large over time) # scp -i $key "$vm`:~/trading/trading_data.db" "$dst" # # # Charts folder (all hourly PNGs) # scp -i $key -r "$vm`:~/trading/charts/" "$dst\charts\" # # # Saved models (trained ML models — important for recovery after crash) # scp -i $key -r "$vm`:~/trading/saved_models/" "$dst\saved_models\" # # Write-Host "Sync complete: $(Get-Date)" # # ================================================================================ # WHY SYNC saved_models/: # If the VM crashes and loses agent_state.json, the saved .pkl model files # in saved_models/ are the next best thing — they preserve the trained ML # weights so the bot doesn't have to learn from scratch on restart. # With auto-save after every retrain, agent_state.json should always be # fresh (max 15 min old), but having models locally is a good backup. # ================================================================================ # Install new dependency for v7: # cd ~/trading && source venv/bin/activate # pip install websocket-client # Set Alpaca keys (free signup at alpaca.markets — use paper trading account): # export ALPACA_API_KEY=your_key_here # export ALPACA_SECRET_KEY=your_secret_here # (add to ~/.bashrc to persist across reboots) # -------------------------------------------------------------------------------- # HOW TO CHECK ON THE BOT # -------------------------------------------------------------------------------- # ssh -i "C:\Users\jakec\Desktop\Oracle_trader\ssh-key-2026-03-07.key" opc@193.122.158.156 # tail -f ~/trading/trading.log # Press Ctrl+C to stop watching. Type exit to close SSH. # -------------------------------------------------------------------------------- # HOW TO CHECK IF THE BOT IS STILL RUNNING # -------------------------------------------------------------------------------- # ps aux | grep StockTrading # If you see "StockTrading_v7.py" it is running. # -------------------------------------------------------------------------------- # HOW TO START THE BOT # -------------------------------------------------------------------------------- # cd ~/trading # source venv/bin/activate # nohup python3.11 StockTrading_v7.py > trading.log 2>&1 & # tail -f ~/trading/trading.log # -------------------------------------------------------------------------------- # HOW TO STOP THE BOT # -------------------------------------------------------------------------------- # pkill -f StockTrading_v7.py # ps aux | grep StockTrading (confirm stopped) # -------------------------------------------------------------------------------- # HOW TO UPDATE THE SCRIPT # -------------------------------------------------------------------------------- # Stop the bot, then from PowerShell on your PC: # scp -i "C:\Users\jakec\Desktop\Oracle_trader\ssh-key-2026-03-07.key" "C:\Users\jakec\Desktop\Pythoncoding\Stocks\StockTrading_v7.py" opc@193.122.158.156:~/trading/ # -------------------------------------------------------------------------------- # HOW TO DOWNLOAD LOGS # -------------------------------------------------------------------------------- # scp -i "C:\Users\jakec\Desktop\Oracle_trader\ssh-key-2026-03-07.key" opc@193.122.158.156:~/trading/hourly_report.log "C:\Users\jakec\Desktop\Pythoncoding\Stocks\" # scp -i "C:\Users\jakec\Desktop\Oracle_trader\ssh-key-2026-03-07.key" opc@193.122.158.156:~/trading/agent_state.json "C:\Users\jakec\Desktop\Pythoncoding\Stocks\" # scp -i "C:\Users\jakec\Desktop\Oracle_trader\ssh-key-2026-03-07.key" opc@193.122.158.156:~/trading/trading_data.db "C:\Users\jakec\Desktop\Pythoncoding\Stocks\" # scp -i "C:\Users\jakec\Desktop\Oracle_trader\ssh-key-2026-03-07.key" opc@193.122.158.156:~/trading/TRADE_LOG.csv "C:\Users\jakec\Desktop\Pythoncoding\Stocks\" # -------------------------------------------------------------------------------- # AUTO-SYNC (Windows Task Scheduler — same setup as before, add TRADE_LOG.csv) # -------------------------------------------------------------------------------- # Update sync_trading.ps1 to also sync: # scp -i "C:\Users\jakec\Desktop\Oracle_trader\ssh-key-2026-03-07.key" opc@193.122.158.156:~/trading/TRADE_LOG.csv "C:\Users\jakec\Desktop\Pythoncoding\Stocks\" # ================================================================================ # KEY INFORMATION # VM IP: 193.122.158.156 # Username: opc # Script: ~/trading/StockTrading_v7.py # Log: ~/trading/trading.log # Report: ~/trading/hourly_report.log # Trade CSV: ~/trading/TRADE_LOG.csv # ================================================================================ # ================================================================================ # ML STOCK TRADING SYSTEM — VERSION 8 # SENTIMENT INTELLIGENCE LAYER: FULL EXPLANATION GUIDE # Written for: Jake | Date: March 2026 # ================================================================================ # This document explains every part of the sentiment system added in v8. # It covers what each component does, why it matters, how the data flows # through the system, what you need to set it up, and what to expect. # Read this top to bottom once. After that it serves as a reference. # -------------------------------------------------------------------------------- # WHAT CHANGED FROM v7 TO v8 # -------------------------------------------------------------------------------- # v7 was purely price-driven. Every decision the agents made was based on: # - Past price movements (momentum, RSI, MACD, Bollinger Bands, etc.) # - Volume patterns # - Alpaca WebSocket bar data (velocity) # The fundamental limitation of a pure price system is that it is always # REACTING. By the time a price signal forms, other traders have already seen # it. The edge a pure price model has is speed — it processes signals faster # than a human. But every other algorithmic trader has the same speed advantage. # v8 adds information that most retail traders do NOT monitor continuously: # - What are financial news outlets saying about a stock RIGHT NOW? # - Has the company filed an emergency disclosure with the SEC? # - Is Reddit suddenly talking about this stock more than usual? # - Did a member of Congress who oversees this company's industry just buy it? # These signals are PREDICTIVE rather than reactive. They capture intent and # events BEFORE price fully reflects them. The GBM models learn on their own # how much weight each signal deserves — you do not hardcode any rules. # -------------------------------------------------------------------------------- # THE FOUR SENTIMENT PIPELINES # -------------------------------------------------------------------------------- # Each pipeline runs in its own background thread. They never touch the main # trading loop, never block a trade, and never crash the bot if they fail. # All results are written to sentiment_data.db, a separate SQLite database. # ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ # PIPELINE 1: GOOGLE NEWS RSS # Thread name: SentNews | Update interval: every 30 minutes per ticker # API key required: NO | Rate limit: None enforced, we self-limit to be gentle # ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ # WHAT IT DOES: # Google provides a free RSS (news feed) for any search query. We query: # https://news.google.com/rss/search?q=AAPL+stock&hl=en-US&gl=US&ceid=US:en # This returns up to 20 recent headlines from Reuters, Bloomberg, CNBC, WSJ, # MarketWatch, Yahoo Finance, and hundreds of other outlets aggregated by Google. # Each headline is scored using VADER (Valence Aware Dictionary and sEntiment # Reasoner). VADER is a specialized NLP tool built specifically for social media # and financial text. It understands phrases like: # "AAPL crushes earnings expectations" → compound score: +0.82 (strongly positive) # "Apple faces antitrust investigation" → compound score: -0.71 (strongly negative) # "Apple stock moves sideways" → compound score: 0.00 (neutral) # HOW SCORING WORKS: # 1. Fetch RSS feed for "{TICKER} stock" # 2. Parse up to 20 recent entries using feedparser # 3. Score each headline title with VADER → compound score between -1 and +1 # 4. Apply time decay: articles older than 12 hours get 50% less weight # (news from 24 hours ago is barely relevant to an intraday trade) # 5. Compute weighted average across all headlines # 6. Store as news_sentiment [-1, 1] and news_count [0, ∞] # WHAT THE MODEL LEARNS: # The GBM model sees news_sentiment as a feature alongside price features. # Over many training cycles it learns: "when news_sentiment is above +0.5 # AND RSI is rising, the next bar tends to close higher." Or: "when # news_sentiment is below -0.5, stop-loss triggers become more likely." # The model discovers these relationships from your actual trading history — # no rule is hardcoded. # OUTPUT FEATURES ADDED TO MODEL: # news_sentiment — float, range [-1, +1] # news_count — int, number of recent articles (proxy for "buzz intensity") # DEPENDENCIES: # feedparser — pip install feedparser --break-system-packages # vaderSentiment — pip install vaderSentiment --break-system-packages # If these are not installed, the pipeline is disabled and all news features # default to 0.0 (neutral). The bot still runs normally. # ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ # PIPELINE 2: SEC EDGAR 8-K FILINGS # Thread name: SentSEC | Update interval: every 60 minutes, all symbols # API key required: NO | Rate limit: Self-throttled to 1 request/second # ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ # WHAT IT DOES: # SEC EDGAR is the US Securities and Exchange Commission's public database. # Every public company must file a Form 8-K (Current Report) within 4 business # days of any "material event" — meaning anything significant enough to affect # the stock price. This includes: # NEGATIVE EVENTS that trigger mandatory 8-K: # - Earnings restatement (company admitted prior financials were wrong) # - CEO/CFO resignation or termination # - Default on debt or missed loan payment # - Bankruptcy filing or going concern warning # - SEC investigation or regulatory enforcement action # - Major lawsuit or settlement # - Cybersecurity breach or data theft # - Material weakness in internal controls # - Delisting from exchange # POSITIVE EVENTS that trigger mandatory 8-K: # - Major acquisition or merger agreement # - New strategic partnership or large contract # - Share buyback authorization # - Dividend increase # - FDA drug approval (for pharma stocks) # - New patent grant # - Guidance raised above analyst expectations # WHY THIS IS POWERFUL: # These filings are official and verified — not rumors. Companies are legally # required to file them. They often arrive BEFORE news outlets pick up the story. # A company that files an 8-K about a CEO resignation at 6 PM Eastern will see # the Reuters article at 6:30 PM, the stock reaction at nine-thirty the next day. # Your bot sees the 8-K at 7 PM during the hourly scan. # HOW IT WORKS: # 1. Query EDGAR full-text search API for 8-K filings mentioning the ticker # in the last 48 hours. The API endpoint is completely free: # https://efts.sec.gov/LATEST/search-index?q="AAPL"&forms=8-K&dateRange=... # 2. Parse the JSON response for filing titles and metadata # 3. Keyword-match the filing against two lists (negative/positive keywords) # 4. Assign sec_flag: 1 = negative event, -1 = positive event, 0 = nothing # NEGATIVE KEYWORDS CHECKED: # restatement, resignation, termination, default, bankruptcy, fraud, # investigation, lawsuit, delisting, going concern, material weakness, # cybersecurity incident, data breach, regulatory action, sec investigation # POSITIVE KEYWORDS CHECKED: # acquisition, merger, strategic partnership, new contract, buyback, # dividend increase, guidance raised, record revenue, fda approval, # patent granted, expansion # OUTPUT FEATURES ADDED TO MODEL: # sec_flag — int: 0 (nothing), 1 (negative 8-K detected), -1 (positive 8-K) # IMPORTANT NOTE ON NEGATIVE FLAGS: # If sec_flag = 1 (negative 8-K) is detected for a stock the bot currently # holds, the exit_checks logic in TradingAgent will see a model prediction # heavily influenced by this signal and is more likely to trigger a sell. # The model learns from history that sec_flag=1 correlates with sharp drops. # This is how WNW/LGVN-style disasters would be partially mitigated. # DEPENDENCIES: requests (already installed), standard library only. # ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ # PIPELINE 3: REDDIT MENTION VELOCITY # Thread name: SentReddit | Update interval: every 15 minutes # API key required: YES (free) | Rate limit: 100 requests/minute (free tier) # ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ # WHAT IT DOES: # Reddit's financial communities — especially r/wallstreetbets (14M members), # r/stocks (5M members), and r/investing (3M members) — are the largest retail # investor forums in the world. Coordinated buying from these communities moved # GameStop from $4 to $480 in January 2021. AMC, BlackBerry, and hundreds of # others saw similar retail-driven spikes. # The signal is not the sentiment of individual posts — it is the VELOCITY of # mentions. A stock going from 2 mentions per hour to 50 mentions per hour is # a strong leading indicator of a retail-driven price move, regardless of whether # the posts are bullish or bearish (though sentiment provides additional signal). # WHAT WE TRACK: # - Subreddits scanned: wallstreetbets, stocks, investing, StockMarket # - Lookback window: last 60 minutes of posts only # - Per post: detect ticker mention → score title with VADER # - Per symbol: aggregate mention count + average sentiment # HOW MENTION DETECTION WORKS: # The post title is searched for the ticker symbol as a word boundary match: # " AAPL " — detects AAPL as a standalone word (not inside "FAANG" etc.) # "$AAPL" — detects cashtag format common on Reddit/StockTwits # This avoids false positives from words that contain ticker symbols by accident # (e.g., "TEAM" in "my team won" vs "TEAM" the software company). # HOW TO GET A FREE REDDIT API KEY: # 1. Go to https://www.reddit.com/prefs/apps # 2. Click "create another app..." # 3. Select "script" type # 4. Give it any name (e.g. "StockBot") # 5. Set redirect URI to: http://localhost # 6. Click "create app" # 7. You will see: # Client ID: (the string under "personal use script") # Secret: (labeled "secret") # 8. Add to your ~/.bashrc on the Oracle VM: # export REDDIT_CLIENT_ID=your_client_id_here # export REDDIT_CLIENT_SECRET=your_secret_here # If you do not set these environment variables, the Reddit pipeline is # silently disabled. All reddit features default to 0. The bot runs normally. # OUTPUT FEATURES ADDED TO MODEL: # reddit_mentions — int, how many times ticker was mentioned in last 1h # reddit_sentiment — float [-1, +1], average VADER score of mentioning posts # DEPENDENCIES: requests (already installed), no additional libraries. # ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ # PIPELINE 4: CONGRESS TRADING DISCLOSURES # Thread name: SentCongress | Update interval: once per day # API key required: NO | Data source: Financial Modeling Prep (free demo) # ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ # WHAT IT DOES: # The STOCK Act (Stop Trading on Congressional Knowledge Act), passed in 2012, # requires members of Congress and their spouses to disclose stock trades within # 45 days of the transaction. These disclosures are public record. # WHY CONGRESS MEMBERS OUTPERFORM THE MARKET: # Members of Congress who sit on committees that regulate specific industries # receive regular confidential briefings about those industries. They know about # regulatory decisions, government contracts, and policy changes before the # public does. Trading on material non-public information is technically illegal, # but enforcement has been effectively zero. A 2022 study found that members # of Congress who trade stocks tied to their committee assignments outperform # the market by 12% annually. # WHAT WE TRACK: # When a Congress member files a "Purchase" disclosure for a stock in our # universe in the last 30 days, congress_bought is set to 1 for that stock. # This is not a buy signal by itself — it is one feature among 35+ that the # model uses. But the model will learn from history that congress_bought=1 # combined with strong momentum features correlates with better outcomes. # HOW IT WORKS: # 1. Fetch senate disclosure data from Financial Modeling Prep's free API # 2. Filter for "Purchase" type transactions in the last 30 days # 3. Extract all ticker symbols that were purchased # 4. Set congress_bought=1 for each purchased ticker in our universe # 5. Update once per day (data does not change more frequently than that) # IMPORTANT LIMITATION: # Congress members have up to 45 days to file disclosures. By the time we see # the disclosure, the price may have already moved. This signal is most useful # as a CONFIRMATION signal — it adds weight to a trade the model was already # considering based on price signals, rather than being the sole trigger. # OUTPUT FEATURES ADDED TO MODEL: # congress_bought — int: 1 if purchased by Congress member in last 30 days, # 0 if not purchased or no data # DEPENDENCIES: requests (already installed), standard library only. # -------------------------------------------------------------------------------- # HOW SENTIMENT DATA FLOWS INTO THE MODEL # -------------------------------------------------------------------------------- # Here is the complete data flow from raw information to trading decision: # BACKGROUND THREADS (running 24/7): # ┌──────────────────┐ ┌──────────────────┐ # │ Google News RSS │ │ SEC EDGAR 8-K │ # │ (every 30 min) │ │ (every 60 min) │ # └────────┬─────────┘ └────────┬─────────┘ # │ │ # ┌──────────────────┐ ┌──────────────────┐ # │ Reddit mentions │ │ Congress trades │ # │ (every 15 min) │ │ (once per day) │ # └────────┬─────────┘ └────────┬─────────┘ # │ │ # └──────────┬───────────┘ # ▼ # sentiment_data.db # (SQLite, separate # from price DB) # MAIN LOOP (runs every second during market hours): # ┌────────────────────────────────────────────┐ # │ rebuild_features_for(symbol) │ # │ 1. Get price history from trading_data.db│ # │ 2. Get sentiment scores from sent DB │ # │ 3. build_features(df, sentiment=scores) │ # │ 4. Sentiment columns filled into matrix │ # └────────────────────┬───────────────────────┘ # ▼ # ┌────────────────────────────────────────────┐ # │ Feature matrix (35 columns per symbol): │ # │ ret_1, ret_3, ..., price_velocity, │ # │ news_sentiment, news_count, sec_flag, │ # │ reddit_mentions, reddit_sentiment, │ # │ congress_bought │ # └────────────────────┬───────────────────────┘ # ▼ # ┌────────────────────────────────────────────┐ # │ GBM model predicts: │ # │ direction (BUY=1 / SELL=0) │ # │ predicted return % │ # │ confidence score [0, 1] │ # └────────────────────┬───────────────────────┘ # ▼ # ┌────────────────────────────────────────────┐ # │ TradingAgent.decide(): │ # │ If confidence >= min_conf → BUY or SELL │ # │ Otherwise → HOLD │ # └────────────────────────────────────────────┘ # -------------------------------------------------------------------------------- # WHY NO HARDCODED RULES # -------------------------------------------------------------------------------- # A naive approach would be: "if sec_flag = 1, always sell immediately." # We deliberately do NOT do this. Here is why: # 1. Not all negative 8-Ks crash stocks. A lawsuit announcement for a large # company with strong cash flow might barely move the price. The model # learns this from history. # 2. Context matters. A negative 8-K on a stock that is already deeply # oversold (RSI < 20) might actually be a buy signal because the bad # news is already priced in. The model can discover this pattern. # A hardcoded rule would sell into an already-beaten-down stock. # 3. The GBM model performs feature importance ranking automatically. # After enough training data, you can inspect which features drive # the most decisions. If news_sentiment turns out to have low importance # for your specific stock universe, the model will naturally down-weight it. # 4. Compounding signals. A stock with: # - news_sentiment = +0.8 (very positive news) # - sec_flag = -1 (positive 8-K filed) # - reddit_mentions = 47 (unusual spike) # - congress_bought = 1 # ...is a very different opportunity than a stock with just one of these. # The GBM model captures interactions between features automatically. # Hardcoded rules would need you to enumerate every combination manually. # -------------------------------------------------------------------------------- # THE SENTIMENT DATABASE SCHEMA # -------------------------------------------------------------------------------- # File location: ~/trading/sentiment_data.db (backed up to backup/ folder) # Table: sentiment # symbol TEXT — stock ticker, e.g. "AAPL" # source TEXT — one of: "news", "sec", "reddit", "congress" # score REAL — sentiment score, range [-1.0, +1.0], 0.0 = neutral # count INT — number of items contributing (articles, mentions, filings) # flag INT — special signal: 1=negative event, -1=positive, 0=nothing # ts INT — Unix timestamp of when this record was last updated # raw_text TEXT — most recent headline/title for human inspection # Primary key: (symbol, source) — one row per symbol per source, always upserted. # This means there is no unbounded growth. 300 symbols × 4 sources = 1,200 rows max. # To inspect sentiment data manually from the Oracle VM: # sqlite3 ~/trading/sentiment_data.db # SELECT symbol, source, score, count, flag, raw_text # FROM sentiment # ORDER BY symbol, source; # -------------------------------------------------------------------------------- # WHAT YOU NEED TO INSTALL ON THE ORACLE VM # -------------------------------------------------------------------------------- # Connect to the VM and run: # cd ~/trading # source venv/bin/activate # pip install vaderSentiment feedparser --break-system-packages # That is ALL that is required for: # - Google News sentiment (feedparser + vaderSentiment) # - SEC EDGAR 8-K scanning (requests, already installed) # - Congress trades (requests, already installed) # - Reddit WITHOUT authentication (requests, already installed) # For Reddit WITH authentication (more reliable, higher rate limit): # # No additional library needed — we use raw HTTP requests # # Just set environment variables in ~/.bashrc: # echo 'export REDDIT_CLIENT_ID=your_id_here' >> ~/.bashrc # echo 'export REDDIT_CLIENT_SECRET=your_secret_here' >> ~/.bashrc # source ~/.bashrc # If you skip the Reddit credentials, the pipeline runs unauthenticated, # which works but may get rate-limited more aggressively by Reddit. # The bot handles this gracefully — it just logs a warning and tries again # at the next 15-minute interval. # -------------------------------------------------------------------------------- # WHAT TO EXPECT WHEN YOU DEPLOY v8 # -------------------------------------------------------------------------------- # FIRST HOUR: # - Sentiment database starts empty. All features default to 0 (neutral). # - The news pipeline starts cycling through your 300 symbols. # - At 2 seconds per symbol, it takes ~10 minutes to complete one full pass. # - SEC pipeline starts scanning 8-Ks for all symbols in batches. # - Congress trades load once at startup. # AFTER FIRST DAY: # - Every stock in your universe has news sentiment scores. # - Any 8-K filings from the last 48 hours are flagged. # - Reddit mentions are tracking for the last hour. # - Congress purchases from the last 30 days are loaded. # - The model now has 6 extra columns of information per trade decision. # AFTER FIRST WEEK OF TRAINING: # - The after-hours training cycle (sixteen-oh-one ET) has trained all 32 models # on feature matrices that include sentiment columns. # - Feature importance in the GBM models now reflects the actual predictive # value of each sentiment signal for YOUR specific stock universe. # - You will see in the logs which signals the model is responding to most. # AFTER ONE MONTH: # - Enough historical data exists for the model to learn seasonal patterns # in sentiment (e.g., earnings season causes news_count to spike for all # stocks — the model learns this and adjusts its weighting accordingly). # -------------------------------------------------------------------------------- # HOW TO SEE SENTIMENT WORKING IN THE LOGS # -------------------------------------------------------------------------------- # In debug.log you will see entries like: # SENT_NEWS | AAPL | score=0.412 | articles=14 | top=Apple beats Q1 earnings... # SENT_NEWS | XOM | score=-0.681 | articles=8 | top=Exxon faces climate lawsuit... # SENT_SEC | PARA | NEGATIVE 8-K | filings=1 | ... # SENT_REDDIT | GME | mentions=47 | sentiment=0.234 # SENT_CONGRESS | bought=3 tickers | sample=['NVDA', 'MSFT', 'CRM'] # In trading_data.log you will see the heartbeat now include: # HEARTBEAT | ... | sentiment_symbols=147 | ... # This tells you how many symbols currently have sentiment data loaded. # -------------------------------------------------------------------------------- # WHAT DATA MAKES SENTIMENT MOST ACCURATE # -------------------------------------------------------------------------------- # The quality of sentiment analysis depends heavily on: # BEST CASE — High news volume stocks (AAPL, MSFT, TSLA, NVDA): # - 10-20 articles per day per stock # - Rich Reddit discussion # - Regular 8-K filings # - High model accuracy improvement expected: 2-4% # MODERATE CASE — Mid-cap stocks with regular coverage (MCK, CAH, FAST): # - 2-5 articles per day # - Occasional Reddit mentions # - 8-Ks a few times per year # - Modest improvement expected: 0.5-1.5% # LOWEST SIGNAL — Quiet utility stocks (NI, CMS, LNT): # - Often 0-1 articles per day # - Rarely on Reddit # - 8-Ks mainly for routine earnings # - Minimal sentiment improvement, model will learn to down-weight these # The system handles all three cases automatically. Stocks with sparse # sentiment data just have more 0.0 values in their feature matrices, # which the model correctly learns means "no strong signal from sentiment." # ================================================================================ # v9+ DEVELOPMENT ROADMAP -- FULL DETAIL # ================================================================================ # This section documents every planned improvement, why it matters, how it fits # into the existing architecture, and what it will take to implement. # Sessions are ordered by expected impact per implementation hour. # ================================================================================ # -------------------------------------------------------------------------------- # SESSION B: ENHANCED TECHNICAL FEATURES (Next priority after v8 stabilizes) # -------------------------------------------------------------------------------- # # WHY THESE MATTER: # RSI=30 means "oversold, buy" in a mean-reverting stock. # RSI=30 means "momentum continuing down, sell" in a trending stock. # Without regime detection the model learns a muddled average of both cases. # ADX resolves this: it tells the model whether any momentum signal is valid. # # FEATURES TO ADD: # # ADX (Average Directional Index) -- trend strength, 0-100 scale # adx = ADX(14). 0-20=no trend, 20-40=developing, 40+=strong # adx_plus_di = +DI line (bullish directional pressure) # adx_minus_di = -DI line (bearish directional pressure) # adx_regime = 0 (choppy ADX<20), 1 (ranging 20-25), 2 (trending ADX>25) # WHY: Regime classification is the single biggest gap in current features. # The model learns that RSI signals only matter in regime 0-1, while # momentum signals only matter in regime 2. # # OBV (On-Balance Volume) -- cumulative buying/selling pressure # obv = running total: +volume on up bars, -volume on down bars # obv_slope_5 = linear regression slope of OBV over last 5 bars # obv_slope_20 = slope of OBV over last 20 bars # WHY: OBV divergence from price is one of the most reliable leading signals. # Price making new highs while OBV falls = distribution (sell). # Price making new lows while OBV rises = accumulation (buy). # # Stochastic Oscillator # stoch_k = %K: (close - low_14) / (high_14 - low_14) * 100 # stoch_d = %D: 3-period SMA of %K (the signal line) # WHY: More sensitive than RSI for short-term overbought/oversold in # ranging (regime 0-1) markets. # # MFI (Money Flow Index) -- volume-weighted RSI # mfi = RSI applied to (typical_price * volume) instead of price # WHY: RSI=70 with low volume = weak signal. MFI=70 with high volume = # institutional buying = real signal. Volume confirmation matters. # # VWAP (Volume-Weighted Average Price) -- intraday fair value # vwap = cumulative(price*volume)/cumulative(volume), resets daily # vwap_gap = (price - vwap) / vwap # WHY: Institutional desks benchmark to VWAP. Extreme VWAP gaps (>1.5%) # tend to revert -- strong intraday mean-reversion signal. # # Volume Anomaly Detection # vol_zscore = (volume - vol_sma_20) / vol_std_20 # vol_ratio = volume / vol_sma_20 # WHY: High volume on a price move = confirmation. High volume with no # price move = absorption, often precedes reversal. z-score > 2.0 # is a statistically significant event. # # Structural / Positional Features # pct_52w_high = (price - low_52w) / (high_52w - low_52w) range 0-1 # golden_cross = 1 if SMA50 > SMA200 else 0 # sma_stack = 1 if price > SMA20 > SMA50 > SMA200 (fully bullish) # WHY: 52-week position tells the model where in the stock's cycle it is. # Golden cross is self-fulfilling -- many traders act on it simultaneously. # # ATR (Average True Range) -- volatility normalization # atr_14 = 14-bar average true range # atr_ratio = atr_14 / price (normalized) # WHY: Use to dynamically size stop losses. # stop_loss_pct = max(0.05, atr_ratio * 2.5) # A 10% ATR stock needs a 12% stop; a 0.8% ATR stock needs a 3% stop. # Flat 7% is wrong for both. This single change should reduce # premature stop-outs significantly. # # Hurst Exponent (compute nightly, not per-bar) # hurst = 0.5=random, >0.5=trending, <0.5=mean-reverting # WHY: Mathematically confirms regime from the price series itself. # Combined with ADX: if both say trending, momentum signals get a boost. # Computed from 200-bar window nightly during after-hours training. # # IMPLEMENTATION: Add compute_enhanced_features(df) called inside # rebuild_all_features(). VWAP requires a daily reset dict per symbol. # pip requirements: None new -- all math is numpy/pandas. # -------------------------------------------------------------------------------- # SESSION C: LSTM MODEL ADDITION (hybrid ensemble per agent) # -------------------------------------------------------------------------------- # # WHAT GBM CANNOT DO: # GBM treats each bar as independent. It cannot learn: # - "The last 8 bars showed a slow grind up before a breakout" # - "Volume has been accumulating for 3 weeks" # - "This is the pre-earnings drift pattern" # LSTM (Long Short-Term Memory) networks are designed specifically to learn # from sequences -- they remember context across 60 bars of lookback. # # ARCHITECTURE PER AGENT (after SESSION C): # StockModel (GBM) -- existing, trains on tabular features # LSTMModel (new) -- trains on 60-bar price/feature sequences # EnsembleCombiner -- weights by recent accuracy, dynamically adjusted # # If GBM has been outperforming LSTM in the last 20 trades, GBM gets a # larger weight. If LSTM is outperforming, it gets more weight. Automatic. # # LSTM INPUT: # Shape: (batch, 60, n_features) -- 60 bars of lookback # Architecture: 2-layer LSTM (64 units each) + Dense(1) + threshold # Training: Adam optimizer, lr=0.001, 20 epochs, early stopping patience=3 # Minimum data: 120 bars per symbol (60 lookback + 60 validation) # # pip requirements: pip install torch (PyTorch ~2.5 GB) # IMPLEMENTATION COMPLEXITY: HIGH (~500 lines). Sequence construction is # the hard part -- sliding window over price history for (X, y) pairs. # -------------------------------------------------------------------------------- # SESSION D: PAIRS TRADING MODULE (statistical arbitrage) # -------------------------------------------------------------------------------- # # WHAT IT IS: # Find two stocks that historically move together. When they diverge, # buy the underperformer and wait for convergence. Market-neutral -- # works regardless of market direction. # # PROVEN: 8-9% annualized across 34 countries per academic research. # As a SUPPLEMENT to directional ML, it adds uncorrelated alpha. # # NIGHTLY (during after-hours training): # 1. Pull 90-day history for all symbols in universe # 2. For correlated pairs (correlation > 0.85): run ADF test on spread # 3. If ADF p-value < 0.05: cointegrated pair -- store with hedge ratio # 4. Keep top 50 pairs by cointegration strength # # MARKET HOURS (each cycle): # z_score = (spread - spread_mean) / spread_std # If z_score > 2.0: stock A overpriced vs B -- flag B as "pairs_long" # If z_score < -2.0: B overpriced vs A -- flag A as "pairs_long" # Close when |z_score| < 0.5 (convergence) # # INTEGRATION: pairs_signal feature in the feature matrix (0 or 1). # Agents learn to weight it from actual outcomes -- no hardcoding. # # pip requirements: pip install statsmodels (ADF test) # IMPLEMENTATION COMPLEXITY: MEDIUM (~350 lines) # -------------------------------------------------------------------------------- # SESSION E: REINFORCEMENT LEARNING LAYER (PPO position sizing) # -------------------------------------------------------------------------------- # # WHAT IS MISSING: # Current agents make binary BUY/SELL decisions. They never learn: # - When NOT to trade (most important trading skill) # - How much capital to deploy (position sizing) # - How to behave differently when in drawdown vs when profitable # A GBM at 55% accuracy can lose money if it trades too frequently. # RL learns the optimal FREQUENCY and SIZE of trades, not just direction. # # ALGORITHM: PPO (Proximal Policy Optimization) # Published research: PPO achieves 15% annual return and 83% cumulative # over 4 years on Dow Jones stocks. Outperforms DQN and A2C in bull markets. # # STATE SPACE (what RL observes): # - Current 80-feature vector for symbol # - GBM + LSTM prediction confidence # - Current position (held / not held) # - Portfolio: cash%, drawdown%, positions, win_rate_last_20 # - Market regime: VIX proxy, time of day, day of week # # ACTION SPACE: # HOLD, BUY_25%, BUY_50%, BUY_100%, SELL # # REWARD: # Sharpe ratio contribution = pnl/capital - 0.5*(pnl_std/capital) # Rewards consistent profit, penalizes volatility. An agent making # 10 small consistent gains scores higher than one making one big # gain and one big loss. # # PPO sits ON TOP of GBM/LSTM -- does not replace them. # GBM/LSTM generate signals. PPO decides size and timing. # # pip requirements: pip install stable-baselines3 (requires PyTorch) # IMPLEMENTATION COMPLEXITY: HIGH (~600 lines + gym environment wrapper) # -------------------------------------------------------------------------------- # SESSION F: MACRO REGIME FEATURES (free from FRED API) # -------------------------------------------------------------------------------- # # FEATURES TO ADD: # # Time-of-day session (proven in academic literature): # intraday_session = 0 (first 30 min), 1 (midday), 2 (last 30 min) # minutes_since_open = raw minutes since nine-thirty ET # WHY: First 30 min = high volatility (gap resolution). Midday = low # volatility, high whipsaw risk. Last 30 min = rebalancing volume. # Model should learn to reduce size during midday. # # Day of week: # day_of_week = 0 (Monday) through 4 (Friday) # WHY: Monday gaps from weekend news. Friday sees early de-risking. # # Market calendar awareness: # fomc_days_away = days until next Fed meeting (from public schedule) # options_exp_week = 1 if current week has third Friday of month # earnings_season = 1 if weeks 2-6 after quarter end # WHY: FOMC meetings compress implied vol beforehand, explode after. # Options expiration week has max-pain market-maker effects. # # VIX proxy (computed from your own bar data -- no external API): # vix_proxy = average ATR ratio across all 300 core symbols # vix_regime = 0 (low fear <15), 1 (normal 15-25), 2 (high fear >25) # WHY: High fear (VIX>25) = momentum fails, mean-reversion dominates. # Model shifts behavior based on fear regime automatically. # # FRED API (optional, free key at fred.stlouisfed.org): # yield_curve_spread = 10Y Treasury yield - 2Y Treasury yield # yield_curve_regime = 1 if inverted (<0), 0 if normal (>0) # WHY: Yield curve inversion preceded every US recession in 50 years. # # IMPLEMENTATION COMPLEXITY: LOW (~200 lines) # -------------------------------------------------------------------------------- # SESSION G: STOCKTWITS SOCIAL SENTIMENT # -------------------------------------------------------------------------------- # # Real-time retail trader sentiment missing from current pipeline. # StockTwits API: completely free, no credit card. # # stocktwits_bullish_pct = % of posts tagged "Bullish" by poster # stocktwits_bearish_pct = % of posts tagged "Bearish" # stocktwits_message_vol = messages per hour (velocity signal) # stocktwits_trending = 1 if on the trending list # # Endpoint: api.stocktwits.com/api/2/streams/symbol/{TICKER}.json # Rate limit: 400 requests/hour authenticated (free account) # Add SentStockTwits thread to SentimentEngine alongside existing threads. # Poll every 10 minutes per symbol (fits within 400/hour for 300 stocks). # # IMPLEMENTATION COMPLEXITY: LOW (~100 lines) # -------------------------------------------------------------------------------- # SESSION H: EARNINGS CALENDAR INTEGRATION # -------------------------------------------------------------------------------- # # PROBLEM: Agents buy 2 days before earnings on a bullish signal. # Earnings miss -> 15% drop regardless of pre-earnings signal. # This is the largest source of unexpected large losses. # # FEATURES: # days_to_earnings = calendar days until next earnings # days_since_earnings = days since last earnings # earnings_week = 1 if earnings within 5 trading days # # AUTOMATIC RISK REDUCTION (soft rules while model learns): # days_to_earnings <= 3: multiply position size by 0.5 # days_to_earnings <= 1: refuse all BUY signals for this symbol # # DATA SOURCE: Alpha Vantage EARNINGS endpoint (already have API key). # One call per symbol per week during after-hours training. # # IMPLEMENTATION COMPLEXITY: LOW-MEDIUM (~150 lines) # -------------------------------------------------------------------------------- # QUICK-WIN IMPROVEMENTS (add to next release, no new session needed) # -------------------------------------------------------------------------------- # # 1. DYNAMIC STOP-LOSS FROM ATR (SESSION B prerequisite) # stop_loss_pct = max(0.05, atr_ratio * 2.5) # Eliminates premature stop-outs on volatile stocks. Single biggest # improvement to loss reduction per line of code. # # 2. SECTOR-LEVEL SIGNAL AGGREGATION # sector_bullish_pct = mean confidence of peers in same sector # If 80% of healthcare is showing SELL, down-weight healthcare BUY signals. # Zero new data sources -- purely from existing agent confidence values. # # 3. MARKET OPEN GAP DETECTION # gap_pct = (open - prev_close) / prev_close at nine-thirty ET # opening_gap_direction = 1 (up gap) or -1 (down gap) # Large gaps continue; small gaps reverse. Model learns the threshold. # # 4. TRADE SIZE SCALING BY CONVICTION # invest *= (eff_conf / min_conf), capped at 2x normal size # High-conviction trades get larger positions automatically. # 5-line change to the buy() method. # # 5. AFTER-HOURS TRAINING DEADLINE # training_deadline = market_close + 5 hours # If training takes longer, save progress and stop. # Prevents running past midnight into next market open. # # 6. AGENT SPECIALIZATION BY SECTOR (once 3+ months of history exists) # 4 agents per sector (8 sectors * 4 = 32 agents). # Each trains only on its sector's stocks. # Sector-specific patterns outweigh general patterns once data is rich enough. # -------------------------------------------------------------------------------- # CURRENT FEATURE REFERENCE (all features in FEATURE_COLS as of v8) # -------------------------------------------------------------------------------- # # Price and returns: # returns_1, returns_5, returns_10, returns_20 # # Moving averages: # sma_ratio_5, sma_ratio_20, sma_ratio_50, sma_cross # # RSI: # rsi_14 # # MACD: # macd, macd_signal, macd_hist # # Bollinger Bands: # bb_pos, bb_width # # Volatility: # volatility_10, volatility_20 # # Momentum: # momentum_10, momentum_20 # # Volume: # volume_ratio # # Price velocity (Alpaca-specific): # price_velocity = (close - prev_close) / seconds_since_last_bar # # Sentiment (v8): # news_sentiment [-1.0, +1.0] VADER score of recent headlines # news_count [0, inf] article count in last 24h # sec_flag {-1, 0, 1} 8-K sentiment flag # reddit_mentions [0, inf] mention count in last hour # reddit_sentiment [-1.0, +1.0] VADER score of Reddit posts # congress_bought {0, 1} Congress purchase in last 30 days