A data scientist applies PROC TEXTMINE to a corpus of 50,000 customer emails. After parsing, the term-document matrix has 200,000 unique terms. Most terms appear in only 1-2 documents. Which PARSE statement option most effectively removes these rare terms?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Here's the deal — a is correct because the MINFREQ= option in the PARSE statement of PROC TEXTMINE sets a minimum total term frequency threshold — terms appearing fewer times than this threshold across the entire corpus are excluded from the term-document matrix. Setting MINFREQ=3 would remove all terms appearing in only 1-2 documents.
Full explanation below image
Full Explanation
A is correct because the MINFREQ= option in the PARSE statement of PROC TEXTMINE sets a minimum total term frequency threshold — terms appearing fewer times than this threshold across the entire corpus are excluded from the term-document matrix. Setting MINFREQ=3 would remove all terms appearing in only 1-2 documents. B (MINCELLS=) is not a standard PROC TEXTMINE PARSE option. C (STOPLIST=) specifies predefined stop words (like 'the', 'and'), not frequency-based filtering. D (WEIGHT=NONE) affects term weighting for existing terms, not removal of rare terms.