A data engineer is designing a nightly flow that ingests visitor-ticketing data, checks whether the branch is open that day, and then either loads the data normally or skips loading entirely on days the branch was closed for a private event. Multiple branching decisions and a multi-step sequence are involved. Where should this control-flow logic live?
Select an answer to reveal the explanation.
Short Explanation
Branching logic like this — check a condition, then go one way or the other — is exactly what a pipeline's control-flow activities are built for. Put it in the pipeline using something like an If Condition activity, and the decision and the multi-step sequence live in one visible, monitorable place.
Full Explanation
Fabric pipelines provide dedicated control-flow activities, including If Condition, precisely for scenarios where a run needs to branch based on an evaluated condition and then proceed down one of several paths, which matches checking a branch's open/closed status before deciding whether to load. Keeping that logic in the pipeline means the decision, the loading path, and the skip path are all visible together in one run's history, easy to audit later. Putting the same logic entirely inside a PySpark notebook with Python if-statements works technically, but it buries orchestration-level decisions inside transformation code, makes the branching invisible to anyone looking at the pipeline's monitoring view, and conflates two different concerns that should stay separate. Splitting the decision across several independent Dataflows Gen2 fragments the logic in a way that makes it hard to see the whole flow at a glance, and Dataflow Gen2 isn't designed as a control-flow orchestrator in the first place. Relying on a person to check a calendar and decide by hand reintroduces exactly the manual dependency and human-error risk that orchestration tooling exists to remove. The caveat: If Condition activities evaluate a single expression per branch, so genuinely complex multi-factor logic may need to be pre-computed into a simpler flag beforehand. A concrete check: run the pipeline on a known closed day and confirm the Monitoring hub shows the skip branch executed, not the load branch.