During pretraining a transformer can score many token positions of…

Question

During pretraining a transformer can score many token positions of a packed sequence in one pass, but at serving time a decoder still emits tokens one after another. How do those time-shapes differ?

Accepted Answer

Training can be highly parallel across positions; generative inference is still sequential in the output. — Training can grade a packed sequence across many positions in one pass. Serving a decoder is still one token after another, because the next word depends on the last. Flip those shapes, or call them job schedules, and the time picture is backwards.

Answer

Inference writes every future token in one parallel shot, and training is strictly one position at a time.

Answer

Both phases are sequential on one token, because transformers cannot parallelize training.

Answer

Both phases are datacenter job schedules, not model time-shapes.

During pretraining a transformer can score many token positions of a packed sequence in one pass, but at serving time a decoder still emits tokens one after another. How do those time-shapes differ?

Short Explanation

Full Explanation