A data engineer needs to run a custom deduplication and text-normalization routine across millions of OCR-extracted oral-history transcript records, using a third-party Python NLP library and iterative logic that doesn't map cleanly onto point-and-click transformations. Which tool should the engineer use?
Select an answer to reveal the explanation.
Short Explanation
Picture handing a stack of worn oral-history transcripts to a specialist with their own toolkit, instead of a general clerk following a checklist. A PySpark notebook is that specialist's toolkit — it runs custom Python libraries and intricate logic across millions of records that a visual tool just can't express.
Full Explanation
A Spark notebook running PySpark gives a data engineer a full code-first environment: arbitrary Python libraries can be installed and imported, and custom logic — however iterative or library-dependent — can be expressed directly, then distributed across the Spark cluster to handle millions of records efficiently. That combination of library support and scale is what this transcript-cleanup job actually needs. Dataflow Gen2's Power Query is intentionally low-code; it exposes a curated set of transformation steps and connectors, not an arbitrary Python runtime, so a third-party NLP library and custom iterative logic fall outside what it can express. T-SQL window functions are powerful for set-based relational logic, but they aren't a substitute for a general-purpose programming language and its library ecosystem — “approximate” is the tell that the tool doesn't actually fit. KQL's function support is aimed at querying and reshaping data already inside an Eventhouse, not at running general-purpose NLP processing over a batch of transcripts. Before scaling this out, confirm the chosen NLP library is compatible with the notebook's Spark runtime version to avoid a dependency mismatch mid-job.