betterwithdata_cleaning_4/README.md

MEPS Data Scraping

A product of Get Better With Data Team 4

Get Better With Data Hackathon - Team 4 Cleaning - Medical Expenditure Panel Survey Scraping

---

Using the data

All data is located in the /data folder as CSV files. There are two types:

• Non-Enhanced: Raw data converted from MEPS into a plain CSV
• Enhanced: Values for flags added to the dataset

Using the scripts

To fetch more data from MEPS, you need R. The important functions are located in /scripts/puf_download.R. You can download data for any dataset if you have the PUF (ex: HC-175E). An example usage is as follows:

short_puf <- shorten_puf("HC-175E")
download_puf(short_puf)
# The file h175e.csv will be created in the current directory

All PUF files, regardless of what dataset they come from, can be downloaded through this command.

At this stage the enrich_dataset.py script can be used to add categorical labels and convert to better variable names.

# Usage:
enrich_dataset.py --input-file h94e.csv --column-dictionary FYCCodebook_2013.csv

The script will extract information about categorical variables in the input file using import.io API to parse codebook tables from the MEPS site and add columns with labels, rather than numeric IDs.

The column dictionary is one time construction from the codebooks on the MEPS website, mapping 8-character variable names to more descriptive ones.