A municipal kite-maker loft wants the language stack to look at sail-cloth stills while it writes repair notes. Which transformer mechanism hooks another modality into the text generation path?
Select an answer to reveal the explanation.
Short Explanation
Cross-attention is the hook: the writing stream asks questions, and the image stream answers with keys and values. That is how sail-cloth patches can steer the note without dropping vision.
Full Explanation
When a municipal kite-maker loft wants the language stack to look at sail-cloth stills while writing repair notes, cross-attention is the transformer mechanism that hooks another modality into the text generation path: queries come from one stream (often the decoder/text side) while keys and values come from the other (vision features). Early dropout of all image tokens prevents the decoder from seeing sail-cloth features. Fixed dictionary lookup from patch id to word id is not cross-attention. Running only an encoder-only text classifier never generates notes and never hooks vision into generation. Exam caveat: name cross-attention’s Q-from-one / KV-from-other pattern for multimodal conditioning. Operational check: in the loft’s generator diagram, locate cross-attention blocks that consume visual keys/values during note generation; confirm image tokens are not dropped before those layers. Probe cross-attention maps on sail-cloth features while generating notes to confirm vision conditioning is active. Cross-attention feeds sail-cloth keys/values into note generation; dropping image tokens or using classifiers alone removes that hook. Probe attention to vision features during repair-note decoding.