Persistence Files

Runtime SQLite Records

The bot maintains all runtime state in a single SQLite database. Legacy JSON files are now only used as inputs for the migration script.

Resource	Table / File	Purpose
Unified Store	arb_bot_records.sqlite3	Source of truth for all bot runtime state: open trades, metrics, go-live status, and scanner history. Managed via SQLiteStore in arb_bot/storage.py. Gitignored. Sync excludes *.sqlite3*.
Open Trades	open_trades table	Active positions — survives bot restarts. Holds dry-run, shadow, and live positions for all managed strategies. Replaced transactionally on every save.
Metrics	metrics_days table	Daily performance snapshots. Funnel counts and session stats are stored in JSON columns; executions and closed_trades are normalized into their own tables for query efficiency. Closed-trade records are deduplicated by tag - record_close ignores a tag already recorded that day, and scripts/dedupe_closed_trades.py repairs historical duplicates. Restored on restart.
Go-Live State	go_live_state table	Runtime paper-to-live state for all strategies. Config *_DRY_RUN flags only seed the DB on first run; thereafter, Telegram commands and the GoLiveManager own the state.
Dhan Token State	runtime_state row dhan_token	Latest Dhan access-token expiry, refresh source, and update time written by token_refresher.py. The raw token remains runtime state and must not be committed.
Strategy Slot State	runtime_state row strategy_slots	Current explicit slot occupancy, including the dc_family used/limit counts and occupying trade tags. Recomputed from persisted open_trades whenever trades are loaded or saved.
Risk Envelope State	risk_envelope_state table	Singleton row for daily, weekly, monthly, and lifetime P&L counters, loss-streak count, pause reason/until, and last close date. Updated after trade close and read before new-entry scanners.
AI Market Analysis	ai_market_analysis table	One row per AI_MARKET analysis run (Flow 1/2). OTP-confirmed /rerun_ai_market can add multiple same-day copies. Stores call time, raw response, parsed JSON, verdict, confidence, Codex token usage, error text, and downstream execution status.
AI Market Trades	ai_market_trades table	Mirror record for each simulated AI_MARKET option trade. Links to the analysis row and stores open tag, strike, option type, entry/exit prices, target, stop, and realised simulated P&L.
AI Shared Predictions	ai_predictions table	One row per market_view or breakout_risk Codex run (Flow 2 and Flow 3). Columns: date, task, payload_json (full parsed output), verdict, confidence_pct, input_tokens, output_tokens, created_at. Queried at the start of each scan cycle to apply AI hints to strategy signals. Limited to AI_PREDICT_DAILY_RUN_CAP rows per day.
AI Exit Advisories	ai_exit_advisories table	One row per breakout-risk exit event from the overlay (Flow 3). Columns: date, trade_tag, strategy, reason, confidence_pct, action_taken (AUTO_CLOSE or ADVISORY_ONLY), created_at. Exposed via GET /api/ai/exit-advisories and the AIPredictionsPanel dashboard component. The sentiment-risk overlay (Flow 4) reuses this table with reason="AI_SENTIMENT_RISK".
AI Market Sentiment	ai_market_sentiment table	One row per deterministic ~10-min sentiment run (Flow 4, SentimentService — no LLM). Columns: date, created_at, score (−100..100), label (STRONGLY_BEARISH..STRONGLY_BULLISH), strength_pct, coverage_pct, spot, prev_close, pcr, iv, components_json (per-component normalized value + weight), note. Read by the sentiment-risk exit overlay during position monitoring.
NSE Holidays	nse_holidays table	Cached exchange holidays used by the AI trading-day gate. Weekends are still rejected even when the holiday table is empty.
Market Events	market_events table	Seeded and fetched market-event risk calendar used by the option-strategy event guard, Telegram /events//why_blocked, and dashboard event APIs. Updated by make fetch-events.
Reconciliation Audit	reconciliation_audit table	Append-only audit rows for auto-heal actions: confirmed-flat removals, orphan adoptions, pending/confirmed quantity mismatches, and orphan alerts. The dashboard reads it through /api/reconciliation/log.
Strategy Battle Log	strategy_battle_log table	One row per scan cycle where scanners run. Stores per-strategy eligibility, rejection reasons, selected MDC strategy, spot, IV, realized range, and near straddle. MDC early-return reasons and IC fallback skips are recorded explicitly so an ineligible cycle explains why no new options trade was taken. Auto-pruned after 60 days.
IV Surface Snapshots	iv_snapshots table	One row per successful IV sample: near/far ATM IV, ATM call/put IV, skew, near/far straddles, realized range, expiries, and DTEs. Auto-pruned after 180 days; JSON payload.term_structure stores near/mid/far ATM IV and payload.smile stores bounded strike-level CE/PE IV around ATM. When Dhan omits per-leg IV, smile rows estimate CE/PE IV from option LTP using Black-Scholes inversion and mark iv_source="estimated". Snapshots also store payload.far_smile (far-expiry strikes + LTPs + IVs), which the backtest uses to reprice far-expiry legs from real recorded data instead of synthesizing them with Black-Scholes.
Trade Journal	trade_journal table	One row per closed trade, keyed by tag. Retains the original trade payload and captures realized P&L, exit reason, entry/exit spot and IV, skew, MFE/MAE, slippage proxies, MDC selection context, rejected strategies, the day's AI verdict, operator notes, low-cost narrative AI debrief, structured AI debrief JSON, rejected-strategy delta, AI verdict outcome, and event-risk label. Rows are written at close and can be backfilled from closed_trades, iv_snapshots, strategy_battle_log, and ai_market_analysis.
Manual Event Seed	data/market_events_seed.json	Operator-maintained official/seeded events such as Budget, RBI MPC, FOMC, and known CPI dates. The fetch job reloads it before source adapters.
Event Source Adapters	arb_bot/event_sources/	Fetch and parse official NSE holiday, MoSPI macro, and OEA WPI source payloads into market-event rows. Parsers are tested with static payloads; runtime source failures do not abort the whole job.
Calendar Spread	calendar_spread_history table	Rolling z-score history (last 100 samples). Cleared on contract roll.
Security IDs	futures_id_cache table	Cached NIFTY near/far futures security IDs. Refreshed when near contract expires.
Runtime Config Overrides	bot_settings table	One row per active runtime override for a tunable Config knob. Only keys registered in arb_bot/settings/registry.py are stored here; absent row = code default. Columns: key (PK), value (raw text, coerced on read), updated_by (telegram user or "dashboard"), updated_at. The in-process cache has a 5-minute TTL and is busted instantly on write via Postgres LISTEN/NOTIFY. Edit via the dashboard /settings page or /set KEY VALUE Telegram command.
Config Change History	settings_audit table	Append-only log of every runtime config change. Columns: id (serial PK), key, old_value (NULL if no previous override), new_value (NULL on reset), updated_by, updated_at. Rows are never deleted — provides a full history of who changed what and when.
Logs	arb_bot.log	Size-capped rotating log, 50 MB per file with 7 backups. IST-aware via ISTFormatter. Telegram /logs streams only the requested tail instead of loading the whole file.

Migration from JSON

To import legacy JSON records into the SQLite database, run the importer script. It is idempotent and can be re-run safely if new files are fetched from the server.

python scripts/migrate_json_to_sqlite.py --root . --db arb_bot_records.sqlite3

Market Event Refresh

To refresh the event guard calendar from the seed file and official source adapters, run:

make fetch-events

Backtesting Records and API

The backtesting engine (arb_bot/backtest/) adds three tables to the same arb_bot_records.sqlite3 database — no new database file. They are created idempotently inside SQLiteStore._create_schema(), so reopening an existing production database migrates it automatically.

Table	Purpose
bhavcopy_ohlc	End-of-day NIFTY/BANKNIFTY option OHLC parsed from NSE UDiFF F&O common bhavcopy. Primary key (date, symbol, expiry, option_type, strike); upserts are idempotent.
backtest_runs	Single backtest runs. Stores config_snapshot, trades_json, metrics_json, and status (pending → running → complete/error).
backtest_grid_jobs	Grid-search jobs. Stores the param_space and ranked results_json.

CLI

python -m arb_bot.backtest download --date 2026-06-03 --inspect-header
python -m arb_bot.backtest download --start 2026-01-01 --end today
python -m arb_bot.backtest download --status
python -m arb_bot.backtest run --strategy DC --start 2026-05-18 --end 2026-06-07 --param DC_PROFIT_TARGET_PCT=0.28
python -m arb_bot.backtest grid-search --strategy MDC --start 2026-05-18 --end 2026-06-07 --use-defaults

API + UI

The React /backtest tab drives these endpoints (all require the dashboard session cookie). Runs and grids execute asynchronously in a ProcessPoolExecutor worker so temporary Config overrides cannot leak between jobs or into the live bot.

Method	Path	Description
POST	/api/backtest/run	Start a single run; returns run_id.
GET	/api/backtest/runs	List past runs with summary metrics.
GET	/api/backtest/runs/{run_id}	Full trades + metrics for one run.
POST	/api/backtest/grid-search	Start a grid search; returns job_id.
GET	/api/backtest/grid-search/{job_id}	Ranked results; UI polls until complete.