Within a Natural Language Processing (NLP) development pipeline, what is the primary function of gathering and preparing a text corpus?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Check this out: a machine learning model doesn't start with any built-in knowledge of how humans speak. It's basically a blank slate. If you want it to learn how words relate to each other, you have to feed it data—lots of it. That's what the corpus is for. It's the dataset that the model reads over and over to learn statistical patterns and vocabulary. Think of it as the textbook for the model. That's why Option C is correct. It's not a rulebook (A), a database cache (B), or a list of stop words (D).
Full explanation below image
Full Explanation
In Natural Language Processing (NLP), a corpus serves as the fundamental training dataset. Machine learning algorithms, particularly deep learning models like transformers, learn by discovering statistical associations and structural patterns within text. A corpus provides the large-scale linguistic representation needed to teach these models how words are distributed, how they form sentences, and how context influences meaning.
Without a robust, representative corpus, an NLP model cannot generalize to real-world language. For example, to train a medical chat assistant, developers compile a corpus of clinical notes and medical textbooks so the model learns domain-specific jargon and phrasing.
Let's examine why the other choices are incorrect: - Option A describes a rule-based system. Modern NLP models are statistical, not rule-based; they learn patterns from data rather than following hardcoded grammatical rules. - Option B describes a database or memory store for a production system, which is a software architecture component, not a training dataset. - Option D refers to a stop word list, which is a preprocessing tool used to filter out common words (like 'and', 'the', 'is'), not the corpus itself.
Remember for the exam: the corpus is the raw material (training dataset) from which models extract patterns and build their linguistic capabilities.