Documentation

Metrics

MetricsCollector

Collects intraday performance data for EOD reporting and algorithm fine-tuning. Now backed by SQLite, providing durable persistence and efficient querying of trade history and execution outcomes. Before every record or checkpoint write, the collector verifies the current IST date and resets daily counters when the date changes so a long-running bot does not copy prior-day executions and closes into later metrics rows.

Funnel Tracking

Counts how many candidates are seen vs filtered vs executed each day, broken down by filter reason. Stored as JSON within the metrics_days table.

Stage	Filter reason
Box raw pairs	—
Box filtered	low profit / low volume / margin / cant-trade
Box executed	—
Calendar signals	—
Calendar filtered	low profit / cant-trade
Calendar executed	—
DC / DCS signals	—
DC / DCS filtered	DTE window / empty chain / missing strike or leg ID / IV low or high / far-IV spread / premium / duplicate / max concurrent / margin
DC / DCS executed	executed or simulated, depending on go-live state

Normalized Records

Executions and closed trades are normalized into their own tables for the Dashboard and Strategy Analytics.

Table	Fields captured
`executions`	tag, timestamp, strategy, outcome, date, payload_json. Records both filled and aborted attempts.
`closed_trades`	tag, close_time, strategy, date, actual_profit, dry_run, shadow, payload_json. Captured on trade completion.
`trade_journal`	tag, close_time, strategy, exit reason, realized P&L, entry/exit spot, entry/exit IV, skew, MFE/MAE, slippage proxies, MDC context, rejected strategies, AI verdict, operator notes, AI debrief, structured AI debrief JSON, rejected-strategy delta, verdict outcome, event risk, payload_json. Written at close and idempotently backfilled when requested.
Shadow snapshots	Marked with `shadow=1` in the database. Excluded from normal live/dry-run totals by Dashboard APIs.
Safety	Safety alerts and event counters are stored within the daily `session_json` column.

Trade Journal

Close Snapshot

ExecutionEngine._finalize_close() now writes a trade_journal row after the open-trade save succeeds. The row reuses the closed trade payload and enriches it with the nearest same-day IV snapshot, latest same-day strategy battle row before close, and that day's AI market verdict when present. Slice 2 also invokes TradeDebriefGenerator after the row is saved. The generator uses AI_DEBRIEF_MODEL with low reasoning, no web search, and strict JSON output so the debrief is replayable and cost-bounded. The generated payload includes schema_version, root_cause, regime_correct, exit_timing, theta/vega/spot attribution, suggested parameter deltas, and confidence.

Operator Views

The dashboard exposes /journal backed by GET /api/journal with strategy, date, and exit-reason filters. Expanded rows show operator notes, narrative debriefs, structured root-cause fields, and suggested parameter deltas; notes are edited through PATCH /api/journal/{tag}/notes. Telegram /journal <tag> returns the compact journal entry and /note <tag> <text> appends mobile notes.

Backtesting Prep Audit

Replay-Ready Data

The current SQLite tables can support attribution and comparison, but not a full options replay engine yet. strategy_battle_log stores scan-cycle selection context, iv_snapshots stores ATM IV surface summaries, and trade_journal stores close-time trade outcomes enriched with IV, MDC, and AI context.

Source	Backtesting value	Gap
`strategy_battle_log`	Eligibility, rejection, and MDC selection context by scan cycle.	No full option-chain snapshot, candidate leg bid/ask, Greeks, margin inputs, or executable price trail.
`iv_snapshots`	Spot, ATM IV, term structure, bounded smile, skew, straddles, and realized range over time. The dashboard falls back to the latest captured session when today has no snapshots yet and shows near/far ATM IV history when skew is unavailable.	ATM summaries plus bounded smile only; no full-chain bid/ask/open-interest history or intraday leg price paths.
`trade_journal`	Closed-trade outcomes, exit reason, realized P&L, MFE/MAE, slippage proxies, and context snapshots.	Close-time record only; cannot reconstruct skipped candidates, partial fills, exact entry/exit quotes, or mark-to-market P&L between entry and close.

Fields Needed Next

The full backtesting sprint should add scan-cycle option-chain snapshots for strategy-relevant expiries and strikes, candidate-level decision rows for accepted and rejected trades, per-leg entry/exit/mark snapshots, and stable replay metadata such as config version, strategy code version, slippage model inputs, and no-trade guard state.

Telegram P&L Charts

Chart Commands

arb_bot/charts.py renders compact Matplotlib PNGs from existing SQLite metrics and current open-trade P&L estimates. No new persistence table is required.

Command	Chart
`/chart_today`	Intraday cumulative P&L, including live open-trade estimates when available.
`/chart_week`	7-day daily P&L bar chart.
`/chart_month`	30-day cumulative equity curve.
`/chart_strategy`	30-day per-strategy P&L contribution.

Report Attachments

Telegram text reports remain the primary source of detail. The bot now sends a PNG via Telegram sendPhoto after the 15:15 performance snapshot and after the EOD report. Chart failures are logged and do not block the text report.

Report	Attachment
15:15 hourly report	`/chart_today` equivalent.
EOD report	`/chart_today` equivalent sent after the final text summary.

Dashboard & Strategy Analytics

Unified Reader

The Dashboard API (dashboard_server.py) and Backtest Recovery script now read directly from arb_bot_records.sqlite3. Dashboard trade history is served newest-first; the paged history endpoint supports strategy and dry-run filters so the React table does not load the complete history for every page.

Function	SQL Source
`get_metrics_files()`	`SELECT date FROM metrics_days ORDER BY date`
`get_daily_summary()`	`SELECT payload_json FROM metrics_days WHERE date=?`
`get_trades_by_date()`	`SELECT payload_json FROM closed_trades WHERE date=?`
`get_open_trades()`	`SELECT payload_json FROM open_trades`
`GET /api/trades/history/page`	Deduplicated newest-first history with `page`, `page_size`, `strategy`, and `dry_run` filters.

Strategy Analytics

Queries the SQLite database to aggregate P&L, win rates, and profit factors across all trading days. Closed trades are deduplicated by tag before aggregation so historical rollover defects cannot multiply a trade's P&L.

View	Mechanism
Cumulative P&L	Iterates through `metrics_days`; aggregates strategy-specific totals.
Shadow P&L	Aggregates trades where `shadow=1`.
Exit distribution	Parses `close_reason` from the `payload_json` column.

Durable Persistence

MetricsCollector.save_json() ← replaces JSON file writing │ └── Write to arb_bot_records.sqlite3 → atomic upsert of daily metrics, executions, and closed trades _save_metrics_checkpoint() ← runs every 30 min via scheduler │ └── MetricsCollector.save_json() → ensures intraday durability _install_shutdown_handlers() ← Ctrl+C + SIGTERM + SIGHUP → KeyboardInterrupt │ └── On shutdown: _save_metrics_checkpoint() + executor._save_trades() All state is committed to SQLite before process exit. MetricsCollector.__init__() ← on bot startup │ └── _load_checkpoint() → SELECT payload_json FROM metrics_days WHERE date=today Restores scan_cycles, funnel counts, executions, and closed trades back into memory.

Fine-tuning with legacy data

Legacy JSON Import To use historical data with the new analytics, download your legacy JSON files using make fetch-metrics, then import them into the database using make import-json-records. Once imported, the Dashboard and Strategy Analytics will reflect the historical performance.