A data scientist applies PROC TEXTMINE to a corpus of 50,000 customer emails. After parsing, the term-document matrix has 200,000 unique terms. Most terms appear in only 1-2 documents. Which PARSE statement option most effectively removes these rare terms?
Select an answer to reveal the explanation.
Short Explanation
Here's the deal — you use MINFREQ= in the PARSE statement because it sets a minimum total frequency a term must have across all documents. If you set MINFREQ=3, terms that appear in only 1 or 2 documents are dropped from the term-document matrix.
Full Explanation
The PARSE statement option that most effectively removes rare terms is MINFREQ=, because it sets a minimum total term frequency threshold: terms appearing fewer times than this threshold across the entire corpus are excluded from the term-document matrix. Setting MINFREQ=3 would remove all terms appearing in only 1-2 documents. The maximum-total-frequency option, MAXFREQ=, does the opposite: it removes terms that are too common by setting an upper frequency limit, so it does not target rare terms. The stop-list file option, STOPLIST=, specifies predefined stop words (like 'the', 'and'), not frequency-based filtering, and manually listing all rare terms is impractical. The no-weighting option, WEIGHT=NONE, affects term weighting for existing terms, not removal of rare terms. A minimum-cells option such as MINCELLS=, which would set the minimum number of non-zero cells in the term matrix, is not a standard PROC TEXTMINE PARSE option.