A tide-table caption is split into pieces that may be whole words or fragments before anything is embedded. What are those pieces?
Select an answer to reveal the explanation.
Short Explanation
Before anything gets a meaning vector, the caption is chopped into pieces, whole words or fragments. Those pieces are tokens. Embeddings come later, the context window is the budget, and CUDA kernels do not split English.
Full Explanation
Tokens are the discrete units the model consumes after tokenization. Embeddings are the later numeric vectors, and the context window is the token budget. CUDA kernels are not tokenization.