Two competing options are on the table for a new nightly ingestion step against the ticketing SaaS system: a curator-maintained Dataflow Gen2, or a PySpark notebook written and owned by the data engineering team. Governance policy says citizen-developer-maintained logic must stay in low-code tools that non-engineers can actually read and edit. Given that policy, which factor should decide between the two?
Select an answer to reveal the explanation.
Short Explanation
Speed isn't the question here — ownership is. If a governance policy says curator-maintained logic has to stay somewhere non-engineers can actually open and edit, that points straight at the low-code Dataflow Gen2, because that's the whole reason the choice-between-tools skill matters in the first place: who's going to touch this next.
Full Explanation
Choosing between Dataflow Gen2, a pipeline, and a notebook is meant to weigh factors like who maintains the logic, what skill level that requires, and how the resulting artifact fits governance expectations — not just raw technical capability. Given a stated policy that citizen-developer-owned logic must remain in a low-code tool a non-engineer can read and edit, the Dataflow Gen2 is the correct fit specifically because its visual, Power Query-style interface matches that maintenance requirement, while a PySpark notebook requires Python and Spark fluency the policy explicitly wants to avoid depending on. Performance in a side-by-side test might matter for some decisions, but it's beside the point when the deciding constraint is who can maintain the logic afterward, not how many milliseconds a run takes. Which tool was used most recently on an unrelated pipeline says nothing about this specific governance requirement and is essentially irrelevant to the decision. Alphabetical ordering of tool names has no bearing on technical or governance fit whatsoever and isn't a real decision criterion in any orchestration context. The caveat: if the transformation logic later grows too complex for a low-code tool to express cleanly, that's a signal to revisit the decision and possibly hand ownership to the engineering team rather than stretching the Dataflow Gen2 past its comfort zone. A concrete check: confirm with the curator that she can open the Dataflow Gen2 and describe what each step does without engineering help.