MEPS scraping example
Add a full example of how to scrape the MEPS site
|3 years ago|
|data||3 years ago|
|exploration||3 years ago|
|scripts||3 years ago|
|shiny-app||3 years ago|
|FYCCodebook_2013.csv||3 years ago|
|README.md||3 years ago|
Get Better With Data Hackathon - Team 4 Cleaning - Medical Expenditure Panel Survey Scraping
All data is located in the
/data folder as CSV files. There are two types:
To fetch more data from MEPS, you need R. The important functions are located in
/scripts/puf_download.R. You can download data for any dataset if you have the PUF (ex:
HC-175E). An example usage is as follows:
short_puf <- shorten_puf("HC-175E") download_puf(short_puf) # The file h175e.csv will be created in the current directory
All PUF files, regardless of what dataset they come from, can be downloaded through this command.
At this stage the
enrich_dataset.py script can be used to add categorical labels and convert to better variable names.
# Usage: enrich_dataset.py --input-file h94e.csv --column-dictionary FYCCodebook_2013.csv
The script will extract information about categorical variables in the input file using import.io API to parse codebook tables from the MEPS site and add columns with labels, rather than numeric IDs.
The column dictionary is one time construction from the codebooks on the MEPS website, mapping 8-character variable names to more descriptive ones.