How can a trained Spark MLlib PipelineModel be applied to a Structured Streaming DataFrame for real-time batch scoring?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Spark MLlib's transform() is stream-agnostic — whether the DataFrame is static or streaming doesn't matter. The pipeline processes each micro-batch exactly like a static DataFrame.
Full explanation below image
Full Explanation
Spark's Structured Streaming is built on the same DataFrame/Dataset API as batch processing. MLlib Transformers (fitted models) implement transform(df) which works on any DataFrame, streaming or not. Pattern: stream_df = spark.readStream.format('kafka')...; predictions_stream = pipeline_model.transform(stream_df); query = predictions_stream.writeStream.format('delta').start(output_path). Each micro-batch is processed by pipeline_model.transform() as if it were a static DataFrame. Estimators (fit()) cannot be called on streaming DataFrames — but Transformers (transform()) can. The pipeline_model (result of pipeline.fit(static_df)) is already fitted and only uses transform() at inference, making it streaming-compatible. TensorFlow conversion is unnecessary. model.predict_stream() doesn't exist.