An analyst working in Python must clean a large tabular CSV file before it is loaded into the database: remove duplicate rows, fill in missing values, join it to a lookup table and pivot several columns. Which library is designed for this work?
Select an answer to reveal the explanation.
Short Explanation
If your data looks like a spreadsheet and you're in Python, you reach for pandas. Its DataFrame is basically a programmable spreadsheet that drops duplicates, fills gaps, joins and pivots in a line or two. Tidyverse does similar work, but it lives in R, and the analyst here is working in Python.
Full Explanation
pandas is the standard Python library for tabular data manipulation. Its DataFrame structure holds labeled columns of mixed types, and it provides built-in operations for reading CSV and other formats, removing duplicates, handling missing values by filling or dropping, merging with other tables, grouping, pivoting and reshaping, all of which match the cleaning steps in the scenario. The cleaned DataFrame can then be written to a database through a driver or SQLAlchemy connection. scikit-learn is a Python machine learning library for training, evaluating and applying models such as classifiers and regressors; it includes some preprocessing utilities, but it is not designed for general cleaning, joining and pivoting of tabular data. MATLAB is a commercial numeric computing environment with its own programming language; it can manipulate tables, but it is not a Python library, so it does not fit an analyst working in Python. Tidyverse is a collection of R packages, such as dplyr and tidyr, that provide very similar data wrangling capabilities, but it is used from R, not Python. Exam caveat: pandas is built on NumPy, so numeric columns are stored in efficient arrays, but very large files may need chunked reading or a database-side transformation. Operational check: after cleaning, compare row counts, null counts and duplicate counts with the original file before loading into the target table.