A housing authority needs embeddings from a decoder-only municipal assistant for similarity search over caseworker chat turns. Which practice yields usable embeddings?
Select an answer to reveal the explanation.
Short Explanation
Decoder-only models still grow rich hidden states as they read a chat turn—the last token’s state is a common “summary vector” for that turn. Grab those hidden states (or a pool of them) for similarity search. Token IDs alone, beam width, or a random vision tower are not chat-turn embeddings.
Full Explanation
A housing authority that needs embeddings from a decoder-only municipal assistant for similarity search over caseworker chat turns must derive vectors from the model’s contextual hidden states after the decoder actually processes the text. Taking the last-token hidden state, or a pooled combination of token states, from each chat turn is a standard practice for obtaining a usable turn embedding for retrieval and near-duplicate detection across casework dialogues. Averaging only tokenizer integer IDs never runs the model, so the result is not a semantic embedding of the civic conversation—just an ID statistic. Reading embeddings from an unrelated frozen vision backbone ignores the chat text modality and the municipal assistant’s language representations. Using beam width as a numeric “embedding” confuses a decoding hyperparameter with a vector representation of content. Exam caveat: which layer and pooling recipe work best is empirical; mean pooling, last-token, or learned pooling may differ for short caseworker turns, but the principle is hidden-state-derived embeddings. Operational check: build a small labeled similarity set of caseworker turns, compare last-token versus mean pooling retrieval metrics, and ensure PII handling before indexing housing-case embeddings.