A proprietary trading firm needs to propagate trade execution events — occurring at sub-millisecond frequency during market hours — to downstream systems including a real-time P&L calculator, a risk aggregator, a regulatory reporting engine, and an ML-based market impact estimator. The current batch-based ETL pipeline introduces 15-minute latency that is incompatible with intraday risk limits. Which architectural pattern most effectively decouples event producers from consumers while supporting the required throughput and latency profile?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Todd Lammle: Think of an event stream like a financial newswire — it publishes every headline once, and any subscriber who wants it can tune in at any time, even replay missed editions. That's exactly what Kafka provides for trade events. Option B wins because a durable partitioned log decouples producers from consumers completely, lets each system consume at its own pace, and provides replay capability for disaster recovery and ML backfill — all at the microsecond-to-millisecond latency the firm needs.
Full explanation below image
Full Explanation
Event-driven architecture (EDA) using a durable log-based streaming platform is the standard infrastructure pattern for high-frequency financial event propagation. Apache Kafka and Redpanda implement the log-structured broker model where events are written once to a partitioned, replicated log and can be consumed independently by any number of consumer groups. This provides three critical properties for the use case described: (1) producer-consumer decoupling — the execution management system publishes events without knowing or caring about downstream systems; (2) independent consumption — P&L, risk, regulatory reporting, and ML systems consume the same events at their own rates and processing guarantees without interfering with each other; and (3) replay capability — the immutable event log allows downstream systems to reprocess historical events for backtesting, audit, or disaster recovery.
Option A (polling via REST API) trades latency for simplicity but creates a polling anti-pattern. With 30-second polling intervals, the firm still has substantial latency. More critically, N consumers each polling a production execution system multiplies load on that system and creates thundering-herd problems at market open. REST polling cannot match the push-based latency of streaming.
Option C (message queue like RabbitMQ) is architecturally distinct from a log-structured streaming platform. Traditional message queues deliver each message to exactly one consumer and delete it after acknowledgment. For a use case where four downstream systems all need every trade event, a queue requires fan-out exchange configurations that become complex to manage and lack the replay semantics of a durable log. Queues excel at task distribution; logs excel at event propagation to multiple independent consumers.
Option D (database triggers to push updates) couples the execution management system's database directly to all downstream systems. This creates a synchronous dependency chain where a slow regulatory reporting engine can back-pressure and destabilize the trading system's core database. It violates the fundamental principle that operational systems should not be blocked by analytical consumers — a principle especially critical in latency-sensitive trading environments.