An HFT firm trains an ML-based execution model on two years of historical equity data sourced from the Securities Information Processor (SIP) consolidated feed, which aggregates best-bid/best-offer quotes across all exchanges with approximately 2–5 ms latency. The firm then deploys this model in production against a direct proprietary exchange feed with sub-100 microsecond latency. After deployment, the quant team observes that the model's input feature distributions at inference time diverge substantially from those observed during training, and realized alpha is 60% lower than backtest projections. What is the PRIMARY cause of this performance degradation?

Training-serving skew is one of the most common and costly failure modes in production ML systems, and it is especially acute in market microstructure applications where data sources have meaningfully different latency profiles, normalization conventions, and temporal resolution. The SIP consolidated feed aggregates quotes across all registered exchanges and applies a normalization layer; quotes are timestamped at SIP receipt, not exchange origination, introducing a systematic 2–5 ms delay. Direct proprietary feeds deliver raw exchange-native messages with sub-millisecond timestamps and exchange-specific quote formats. Features computed on SIP data — such as mid-price, bid-ask spread, order book imbalance, or short-term momentum — will differ from the same features computed on direct feed data even at the same wall-clock instant, because the SIP 'sees' a later, consolidated view of the market. When the model was trained on SIP data, it implicitly learned to interpret feature values that correspond to that latency profile. In production, the same feature pipeline now produces systematically different numerical outputs from direct feed inputs, placing the model in an out-of-distribution inference regime. That distribution shift explains the observed feature divergence and the 60% alpha shortfall. Why the other explanations fail: The UDP multicast versus TCP transport explanation is a real microstructure consideration, but the question identifies feature distribution divergence — this is a training data mismatch problem, not a transport-layer reliability problem. The overfitting-to-mean-reversion explanation is plausible in isolation, but the question specifically attributes the problem to input feature distributions diverging, which is definitional of training-serving skew, not model complexity. The NYSE-throttling explanation is not a standard industry practice for proprietary direct feeds, which are sold explicitly for their reliability advantage over consolidated feeds. The remedy is to re-train the model using direct feed historical data (available from exchange colocation vendors or consolidated direct-feed replay providers) to ensure feature distributions at training time match those seen in production.