Meridian's generative-AI knowledge assistant needs to process thousands of pages of maintenance manuals and dispatch procedures so a language model can search and reason over them. A vendor proposes 'tokenization and vectorization' as the first processing step. What should the PM understand this step accomplishes?

CPMAI Task 3 requires applying tokenization and vectorization to text data, and this is precisely the scenario: before a generative-AI knowledge assistant can search or reason over Meridian's maintenance manuals and dispatch procedures, the raw text must be tokenized (split into words, subwords, or other processable units) and then vectorized (converted into numerical representations, such as embeddings, that a model can mathematically compare and process). This is a foundational, necessary text-processing step, not an optional add-on. The 'security/encryption' distractor confuses a machine-learning text-processing technique with an unrelated data-security concept; tokenization/vectorization does not protect confidentiality and is not a substitute for access controls. The 'convert PDFs to printed paper for a filing cabinet' distractor describes a purely physical, non-technical process with no relationship to how a language model actually processes text. The 'only applies to image data, not text' distractor inverts the facts — tokenization and vectorization are specifically named in the outline under text data, and while images have their own separate representation techniques, this enabler is squarely about text like the maintenance manuals in this scenario. Understanding this step helps the PM ask the right technical questions when scoping the knowledge-assistant vendor's proposal.