5. Data Sources and Input Files

5.1 IBD Big Cap 20 / Big Cap 50 (IBD 50 screen)¶

• Format: Browser Export to Excel (.xls) or CSV.

• Parsing: ingest_ibd_lists._xls_to_df uses LibreOffice headless: soffice --convert-to csv (Windows path C:/Program Files/LibreOffice/program/soffice.exe), then pd.read_csv(..., header=8) — row 9 in 1-based terms is the header.

• Column map: IBD_EXPORT_COL_MAP in ingest_ibd_lists.py (e.g. Symbol → Ticker, Rank → IBD_Rank, Composite Rating → Composite_Rating, RS Rating, EPS Rating, ACC/DIS Rating, Annual ROE, etc.).

• Cleaning: Rows containing footer text (“Screen results”, “Data provided”, etc.) are removed.

5.2 IBD RS Master¶

• Same header=8 convention and IBD_EXPORT_COL_MAP; RS Master may use symbol column naming consistent with other screens after rename.

5.3 E*TRADE portfolio export¶

• Raw file: User saves from E*TRADE (e.g. PortfolioDownload*.csv) into Downloads; watch_downloads.py may rename to holdings/etrade_ira_raw_YYYY_MM_DD.csv / etrade_brokerage_raw_YYYY_MM_DD.csv.

• Account identification (etrade_identify.py): Reads first lines of the file: phrases like “Rollover IRA”, “Traditional IRA”, or standalone “IRA” → IRA; “Individual Brokerage”, “Brokerage”, or “Individual” → brokerage; filename may also hint IBD vs E*TRADE.

• Normalization (normalize_etrade.py): Finds the positions header row where column 0 is Symbol and columns include Quantity and Price Paid $. Maps to Ticker, Shares, Cost_Basis (per-share). Drops CASH, TOTAL, MARGIN DEBIT, rows with shares < 1.

5.4 yfinance¶

• Per ticker: info for current price, beta, marketCap, totalDebt, interestExpense, etc.

• Fallbacks: Beta → 1.0; WACC components guarded with NaN on failure.

• Rate limits: Serial calls per holding and per ingest row—SaaS needs caching, batching, or a paid API.

5.5 ^TNX (risk-free rate)¶

• capm_wacc.get_risk_free_rate(): yf.Ticker("^TNX").history(period="5d"), last Close treated as percent → divide by 100 for decimal (R_f). On failure → DEFAULT_RF (0.043).

5.4 MarketSurge (Optional Premium Upgrade)¶

What it is: IBD’s professional screening and charting platform. Formerly known as MarketSmith.

Cost: $49.95 for 6-week trial |$1,499/year annual

Key features not in current system:

• Breaking Out Today list — real-time breakout confirmation

• Near Pivot list — stocks within 5% of buy point

• AI pattern recognition — automatic base identification

• Color-coded buy zones on charts

• Buy point alerts via email/push notification

Current workaround (V1):

• Read Big Picture article daily for specific buy point mentions

• Check IBD 50 charts manually for buy point proximity

• Use Stock of the Day on investors.com for one daily idea

V2 plan — three-tier approach:

Tier 1 (Now — free): Systematic daily Big Picture reading Add buy point mentions from Big Picture to weekly journal Cross-reference against conviction list manually

Tier 2 (Week 12 — $49.95 trial): Test MarketSurge Evaluate Breaking Out Today vs system conviction list overlap Measure whether Near Pivot list improves entry timing Decision: subscribe annually if adds measurable value

Tier 3 (V2 code — free): Build proximity detection Add 52-week high proximity score to ingest_ibd_lists.py Flag stocks within 5% of 52-week high as NEAR_PIVOT This approximates Near Pivot list without MarketSurge cost

ROI threshold for annual subscription: $1,499/year =$125/month Must prevent 2-3 bad entries per year OR Must identify 1-2 breakouts missed without it Evaluate after 6-week trial at Week 12

Note on data independence: MarketSurge uses IBD proprietary data — same IP constraints as current IBD Digital subscription. If moving toward data independence (Path 2), MarketSurge is not on that path. Own RS calculation (Enhancement 4) + yfinance 52-week high proximity is the data-independent alternative.