Make Work for RSocrata

[stuagano]

Get and clean Atlanta Police Department Data get.apd.data <- function(selector.text, save.location){ require(rvest) require(dplyr) temp <- tempfile() download.file( read_html("http://www.atlantapd.org/crimedatadownloads.aspx") %>% html_nodes(selector.text) %>% html_attr("href") %>% paste0("http://www.atlantapd.org/", .), destfile = temp) # Save filename of csv crime data raw file, for later use crime.data.filename <- unzip(temp, list = TRUE)$Name # Unzip crime data to "data" folder and close temp file placeholder unzip(temp, exdir = save.location) unlink(temp) crime.data.filename

Was there a reason why you did it this way instead of using SODA? Other PD's could definitely take advantage of this if it was using RSocrata from Chicago.

[bbrewington]

Good point. That will probably be the best route; I think we need to clean up the data in Socrata, though. Looks like there's a difference starting in 2015:

Socrata Data:

COBRA051916.csv file:

Possible Cause: import tool is duplicating rows

[bbrewington]

@stuagano would it make more sense to do the ETL via https://dev.socrata.com/connectors/pentaho-kettle.html instead of the R script in this repo? Maybe if there was a way to schedule an R ETL job it could work. What do you think?

Here's the ETL current process (it would be nice to automate the last step of publishing cleaned data to Socrata):

edit: note to self - https://github.com/jwijffels/taskscheduleR