A high-frequency trading desk needs to store and query 10 years of tick-by-tick trade and quote data for 8,000 US equities, averaging 2.5 billion ticks per trading day. The primary access pattern is time-series range queries by symbol and timestamp for backtesting and signal research. Which storage architecture best satisfies throughput, compression, and query performance requirements for this workload?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Tick data is the most columnar workload in finance — millions of rows with the same five or six fields repeated, sorted by time. kdb+ was built for exactly this: it stores data column-by-column on disk, applies delta-of-delta compression (because consecutive prices change by tiny amounts), and responds to range queries in microseconds because it reads only the columns you requested. A relational database trying to handle 2.5 billion ticks per day would be spending most of its compute on row storage overhead it didn't need.
Full explanation below image
Full Explanation
Tick-by-tick market data has a structure that is fundamentally different from transactional workloads: it is append-only, temporally ordered, extremely high cardinality in the time dimension, and almost always queried as a range (all quotes for AAPL between 09:30 and 10:00 on a given date) or an aggregation (VWAP over a rolling window). This access pattern is the canonical use case for columnar time-series databases.
kdb+ (Q language) has been the industry standard for tick data storage in institutional trading for two decades because its columnar on-disk format (splayed tables) stores each field — bid, ask, size, exchange, timestamp — as a separate contiguous binary file. For a range query on timestamp and symbol, the database reads only the timestamp and symbol columns to locate the matching row indices, then reads the requested data columns. This IO pattern is 10-100x more efficient than a row-oriented database that reads entire rows including unneeded fields. Delta-of-delta compression is applied at the column level: consecutive timestamps differ by small increments, and consecutive prices differ by fractional cents, allowing ratios of 20:1 or better on raw tick data.
Arctic (from Man AHL, now open-source) provides similar columnar partitioning on top of MongoDB's GridFS or Parquet files in object storage, offering a Python-native alternative for firms not running kdb+ licenses. Apache Parquet files in an object store (S3 + Athena or Redshift Spectrum) are also viable for firms with batch-oriented backtesting patterns, though query latency is higher than native kdb+.
PostgreSQL (option A) can handle tick data for smaller universes but fails at this scale — 2.5 billion ticks per day creates hundreds of millions of rows per table partition, and even with optimized indexes, MVCC overhead and row storage format make it unsuitable for microsecond-latency analytical queries. MongoDB (option C) and Redis (option D) are designed for different workloads entirely: MongoDB's BSON document storage wastes space on field-name repetition per tick, and Redis is an in-memory cache — ten years of 8,000-symbol tick data would require petabytes of RAM.
Firms such as Two Sigma, Citadel, and Jump Trading all operate kdb+ or kdb+-compatible columnar tick stores as their foundational market data layer, reflecting the technology's dominance in the quantitative finance data infrastructure space.