During a post-stress-event regulatory examination, a buy-side risk manager is asked by examiners to provide full provenance for a VaR figure that appeared in a stress report filed 18 months earlier. The examiners specifically request documentation of: (1) the exact raw market data inputs used on that date, (2) each data cleaning and transformation step applied, (3) which model version produced the VaR output, and (4) the compute environment in which the calculation ran. The firm's infrastructure logs model version tags and compute job IDs but does not track how raw market data was transformed before entering the risk engine. What critical data infrastructure capability is the firm MISSING?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Having the model version is like knowing which chef cooked the dish — helpful, but if you can't trace which farm the ingredients came from or which prep cook trimmed them, you still can't answer 'why does this taste wrong?' Data lineage is the end-to-end ingredient provenance: from the raw market data tick, through every filter, join, and normalization step, all the way to the number that ended up in the regulatory report. Without it, you can't reconstruct the calculation, and regulators don't accept 'trust us.'
Full explanation below image
Full Explanation
Data lineage is the documented record of where data originates, how it moves through a system, and how each transformation modifies it along the way. In the context of financial risk reporting, end-to-end lineage serves multiple critical functions: regulatory auditability (demonstrating to examiners exactly how a reported figure was derived), error investigation (identifying which transformation step introduced a pricing error), and model risk management (showing that model inputs were clean and correctly processed).
The gap described in this scenario is specifically in the transformation layer: raw market data is ingested (from a price feed, for example), cleaned (outlier removal, curve interpolation, missing value imputation), transformed (returns computation, volatility estimation, correlation matrix construction), and then fed into the VaR model. None of these intermediate steps are being logged in a recoverable, queryable form.
A data lineage system would record, for each calculation: the source dataset version (e.g., Bloomberg pricing snapshot ID, timestamp), each transformation operation applied (with its parameter configuration), the intermediate dataset states, and the final input vector presented to the model. Modern data lineage tools (Apache Atlas, OpenLineage/Marquez, dbt's lineage DAG, Alation, Collibra) capture this at the column level, enabling regulators to trace a single VaR input back through the pipeline to its raw source record.
Why the distractors fail: (A) A model registry tracks model artifacts, hyperparameters, and version history — valuable for model governance, but the firm already has model version logging. The gap is in data transformation provenance, not model metadata. (C) A feature store with point-in-time correct lookup addresses the temporal consistency of features during training and inference (preventing future-data leakage), but does not inherently track the transformation logic that produced those features from raw inputs. (D) A data catalog provides a searchable inventory of datasets, their business definitions, and ownership — valuable for data discovery and governance, but not a lineage system. Lineage tracks the dynamic flow of data through computations, not just the static descriptions of datasets.
Regulatory frameworks requiring demonstrable data lineage in buy-side risk include BCBS 239 (principles for risk data aggregation), SEC Rule 18f-4 (derivatives risk management for funds), and ESMA's AIFMD reporting requirements.