rewire.Rmd

---
title: "Rewire Legislative Tracker"
author: "Maddy Kluesner"
date: "Last edited `r format(Sys.time(), '%b %d %Y')`"
output: html_document
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(fig.align = 'center', out.width = '80%', echo = TRUE)
```
## Summary

This document represents an attempt to import and manipulate data from [Rewire's Legislative Tracker](https://rewirenewsgroup.com/legislative-tracker/), which represents legislation on abortion and sexual and reproductive health from 2000 - Present.

## Libraries

```{r}
require(XML)
library(tidyverse)
library(rvest)
library(knitr)
```

## Data

```{r Data loading}
html_coercion <- read_html("https://rewirenewsgroup.com/legislative-tracker/law-topic/coercion-tests/")

coercion_tests <- read_html("https://rewirenewsgroup.com/legislative-tracker/law-topic/coercion-tests/") %>% 
  html_table() %>% 
  data.frame() %>% 
  mutate(type = "Coercion Tests")
twenty_week_bans <- read_html("https://rewirenewsgroup.com/legislative-tracker/law-topic/20-week-bans/") %>% 
  html_table() %>% 
  data.frame() %>% 
  mutate(type = "20-Week Bans")
admitting_privileges <- read_html("https://rewirenewsgroup.com/legislative-tracker/law-topic/admitting-privileges/") %>% 
  html_table() %>% 
  data.frame() %>% 
  mutate(type = "Admitting Privileges")
anti_transgender <- read_html("https://rewirenewsgroup.com/legislative-tracker/law-topic/anti-transgender-2/") %>% 
  html_table() %>% 
  data.frame() %>% 
  mutate(type = "Anti-Transgender")
buffer_zones_bubble_zones <- read_html("https://rewirenewsgroup.com/legislative-tracker/law-topic/buffer-zones-bubble-zones-and-clinic-access/") %>% 
  html_table() %>% 
  data.frame() %>%
  mutate(type = "Buffer Zones, Bubble Zones, and Clinic Access")
conscience_and_refusal_clauses <- read_html("https://rewirenewsgroup.com/legislative-tracker/law-topic/conscience-and-refusal-clauses/") %>% 
  html_table() %>% 
  data.frame() %>% 
  mutate(type = "Conscience and Refusal Clauses")
crisis_pregnancy_centers <- read_html("https://rewirenewsgroup.com/legislative-tracker/law-topic/crisis-pregnancy-centers/") %>% 
  html_table() %>%
  data.frame() %>% 
  mutate(type = "Crisis Pregnancy Centers")
dilation_and_evacuation_bans <- read_html("https://rewirenewsgroup.com/legislative-tracker/law-topic/dilation-and-evacuation-bans/") %>% 
  html_table() %>% 
  data.frame() %>% 
  mutate(type = "Dilation and Evacuation Bans")
fetal_homicide <- read_html("https://rewirenewsgroup.com/legislative-tracker/law-topic/fetal-homicide/") %>% 
  html_table() %>% 
  data.frame() %>% 
  mutate(type = "Fetal Homicide")
fetal_tissue <- read_html("https://rewirenewsgroup.com/legislative-tracker/law-topic/care-of-aborted-fetuses/") %>% 
  html_table() %>% 
  data.frame() %>% 
  mutate(type = "Fetal Tissue")
forced_ultrasound <- read_html("https://rewirenewsgroup.com/legislative-tracker/law-topic/forced-ultrasound/") %>% 
  html_table() %>% 
  data.frame() %>% 
  mutate(type = "Forced Ultrasound")
funding_restrictions_for_family_planning <- read_html("https://rewirenewsgroup.com/legislative-tracker/law-topic/funding-restrictions-for-family-planning/") %>% 
  html_table() %>% 
  data.frame() %>% 
  mutate(type = "Funding Restrictions for Family Planning")
genetic_anomalies <- read_html("https://rewirenewsgroup.com/legislative-tracker/law-topic/genetic-anomalies-abortion-ban/") %>% 
  html_table() %>% 
  data.frame() %>% 
  mutate(type = "Genetic Anomalies")
heartbeat_bans <- read_html("https://rewirenewsgroup.com/legislative-tracker/law-topic/heartbeat-bans/") %>% 
  html_table() %>% 
  data.frame() %>% 
  mutate(type = "Heartbeat Bans")
human_embryo_and_fetal_research <- read_html("https://rewirenewsgroup.com/legislative-tracker/law-topic/human-embryo-and-fetal-research/") %>% 
  html_table() %>% 
  data.frame() %>% 
  mutate(type = "Human Embryo and Fetal Research")
informed_consent <- read_html("https://rewirenewsgroup.com/legislative-tracker/law-topic/informed-consent/") %>% 
  html_table() %>% 
  data.frame() %>% 
  mutate(type = "Informed Consent")
insurance_coverage <- read_html("https://rewirenewsgroup.com/legislative-tracker/law-topic/insurance-coverage/") %>%
  html_table() %>% 
  data.frame() %>% 
  mutate(type = "Insurance Coverage")
later_abortion <- read_html("https://rewirenewsgroup.com/legislative-tracker/law-topic/later-abortion/") %>% 
  html_table() %>% 
  data.frame() %>% 
  mutate(type = "Later Abortion")
lgbtq <- read_html("https://rewirenewsgroup.com/legislative-tracker/law-topic/lgbt/") %>%
  html_table() %>% 
  data.frame() %>% 
  mutate(type = "LGBTQ")
medication_abortion <- read_html("https://rewirenewsgroup.com/legislative-tracker/law-topic/medication-abortion/") %>% 
  html_table() %>% 
  data.frame() %>% 
  mutate(type = "Medication Abortion")
omnibus_multiple_topics <- read_html("https://rewirenewsgroup.com/legislative-tracker/law-topic/omnibus-multiple-topics/") %>% 
  html_table() %>% 
  data.frame() %>% 
  mutate(type = "Omnibus (multiple topics)")
parental_involvement <- read_html("https://rewirenewsgroup.com/legislative-tracker/law-topic/parental-involvement/") %>% 
  html_table() %>% 
  data.frame() %>% 
  mutate(type = "Parental Involvement")
partial_birth_abortion_bans <- read_html("https://rewirenewsgroup.com/legislative-tracker/law-topic/partial-birth-abortion-bans/") %>% 
  html_table() %>% 
  data.frame() %>% 
  mutate(type = "Partial Birth Abortion Bans")
personhood <- read_html("https://rewirenewsgroup.com/legislative-tracker/law-topic/personhood/") %>% 
  html_table() %>% 
  data.frame() %>% 
  mutate(type = "Personhood")
physicians_reporting_requirements <- read_html("https://rewirenewsgroup.com/legislative-tracker/law-topic/physicians-reporting-requirements/") %>%
  html_table() %>% 
  data.frame() %>% 
  mutate(type = "Physicians Reporting Requirements")
pregnancy_exclusion <- read_html("https://rewirenewsgroup.com/legislative-tracker/law-topic/pregnancy-exclusion/") %>% 
  html_table() %>% 
  data.frame() %>% 
  mutate(type = "Pregnancy Exclusion")
religious_freedom <- read_html("https://rewirenewsgroup.com/legislative-tracker/law-topic/religious-freedom/") %>%
  html_table() %>% 
  data.frame() %>% 
  mutate(type = "Religious Freedom")
religious_freedom_restoration_act <- read_html("https://rewirenewsgroup.com/legislative-tracker/law-topic/religious-freedom-restoration-act/") %>% 
  html_table() %>% 
  data.frame() %>% 
  mutate(type = "Religious Freedom Restoration Act (RFRA)")
religious_imposition <- read_html("https://rewirenewsgroup.com/legislative-tracker/law-topic/religious-imposition-laws/") %>% 
  html_table() %>% 
  data.frame() %>% 
  mutate(type = "Religious Imposition") # this table is the only with zero values. I think it is a new one they will begin tracking with.
reporting_requirements <- read_html("https://rewirenewsgroup.com/legislative-tracker/law-topic/reporting-requirements/") %>% 
  html_table() %>% 
  data.frame() %>% 
  mutate(type = "Reporting Requirements")
reproductive_health_awareness <- read_html("https://rewirenewsgroup.com/legislative-tracker/law-topic/reproductive-health-awareness/") %>%
  html_table() %>% 
  data.frame() %>% 
  mutate(type = "Reproductive Health Awareness")
restrictions_funding_surgery <- read_html("https://rewirenewsgroup.com/legislative-tracker/law-topic/restrictions-on-funding-for-hormone-therapy-and-gender-reassignment-surgery/") %>% html_table() %>% data.frame() %>% mutate(type = "Restrictions on Funding for Hormone Therapy and Gender Reassignment Surgery")
sex_ed_and_abstinence_only <- read_html("https://rewirenewsgroup.com/legislative-tracker/law-topic/sex-ed-and-abstinence-only/") %>% 
  html_table() %>% 
  data.frame() %>% 
  mutate(type = "Sex Ed and Abstinence Only")
sex_or_race_selective_bans <- read_html("https://rewirenewsgroup.com/legislative-tracker/law-topic/sex-or-race-selective-bans/") %>% 
  html_table() %>% 
  data.frame() %>% 
  mutate(type = "Sex- or Race-Selective Bans")
substance_abuse_during_pregnancy <-read_html("https://rewirenewsgroup.com/legislative-tracker/law-topic/substance-abuse-during-pregnancy/") %>% 
  html_table() %>% 
  data.frame() %>% 
  mutate(type = "Substance Abuse During Pregnancy")
targeted_regulation_of_abortion_providers <- read_html("https://rewirenewsgroup.com/legislative-tracker/law-topic/targeted-regulation-of-abortion-providers/") %>% html_table() %>% data.frame() %>% 
  mutate(type = "Targeted Regulation of Abortion Providers")
telemedicine_abortion_bans <- read_html("https://rewirenewsgroup.com/legislative-tracker/law-topic/telemedicine-abortion-bans/") %>% 
  html_table() %>% 
  data.frame() %>% 
  mutate(type = "Telemedicine Abortion Bans")
waiting_periods_and_forced_counseling <- read_html("https://rewirenewsgroup.com/legislative-tracker/law-topic/waiting-periods-and-forced-counseling/") %>% 
  html_table() %>% 
  data.frame() %>% 
  mutate(type = "Waiting Periods and Forced Counseling")
wrongful_birth <- read_html("https://rewirenewsgroup.com/legislative-tracker/law-topic/wrongful-birth/") %>% 
  html_table() %>% 
  data.frame() %>% 
  mutate(type = "Wrongful Birth")
```

## Merging Data

```{r}
rewire_leg <- rbind(coercion_tests, 
                    twenty_week_bans,
                    admitting_privileges,
                    anti_transgender, 
                    buffer_zones_bubble_zones, 
                    conscience_and_refusal_clauses, 
                    crisis_pregnancy_centers, 
                    dilation_and_evacuation_bans, 
                    fetal_homicide, fetal_tissue, 
                    forced_ultrasound, 
                    funding_restrictions_for_family_planning,
                    genetic_anomalies, heartbeat_bans,
                    human_embryo_and_fetal_research, 
                    informed_consent,
                    insurance_coverage, 
                    later_abortion,
                    lgbtq, 
                    medication_abortion, 
                    omnibus_multiple_topics, 
                    parental_involvement, 
                    partial_birth_abortion_bans, 
                    personhood, 
                    physicians_reporting_requirements,
                    pregnancy_exclusion, 
                    religious_freedom, 
                    religious_freedom_restoration_act,
                    religious_imposition, 
                    reporting_requirements, 
                    reproductive_health_awareness, 
                    restrictions_funding_surgery,
                    sex_ed_and_abstinence_only, 
                    sex_or_race_selective_bans,
                    substance_abuse_during_pregnancy, 
                    targeted_regulation_of_abortion_providers, 
                    telemedicine_abortion_bans, 
                    waiting_periods_and_forced_counseling, 
                    wrongful_birth)
```

## Cleaning Data

```{r}
rewire_leg2 <- rewire_leg %>%
  # Fix Dates
  mutate(date = as.Date(Proposed, format = "%b %d, %Y")) %>%
  #Rename
  rename(number = Number) %>%
  rename(state = State) %>%
  rename(status = Status) %>%
  rename(title = Title) %>%
  rename(proposed = Proposed) %>%
  #Rename Washington DC
  mutate(state = recode(state, "District of Columbia" = "Washington, D.C."))%>%
  #Create indicator variables
  mutate(n = 1) %>%
  #Add divisions and regions
  mutate(division = NA) %>%
  #Merge duplicate records
  pivot_wider(names_from = type, values_from = n, values_fill = 0) %>%
  mutate(division = case_when(state == "New Jersey"| 
                              state == "New York"| 
                              state == "Pennsylvania" 
                              ~ "New England",
                              state == "Connecticut"|
                              state == "Maine"|
                              state == "Massachusetts"|
                              state == "New Hampshire"|
                              state == "Rhode Island"|
                              state == "Vermont"
                              ~ "Mid-Atlantic",
                              state == "Illinois"|
                              state == "Indiana"|
                              state == "Michigan"|
                              state == "Ohio"|
                              state == "Wisconsin"
                              ~ "East North Central",
                              state == "Iowa"|
                              state == "Kansas"|
                              state == "Minnesota"|
                              state == "Missouri"|
                              state == "Nebraska"|
                              state == "North Dakota"|
                              state == "South Dakota"
                              ~ "West North Central",
                              state == "Delaware"|
                              state == "Florida"|
                              state == "Georgia"|
                              state == "Maryland"|
                              state == "North Carolina"|
                              state == "South Carolina"|
                              state == "Virginia"|
                              state == "Washington, D.C."|
                              state == "West Virginia"
                              ~ "South Atlantic",
                              state == "Alabama"|
                              state == "Kentucky"|
                              state == "Mississippi"|
                              state == "Tennessee"
                              ~ "East South Central",
                              state == "Arkansas"|
                              state == "Louisiana"|
                              state == "Oklahoma"|
                              state == "Texas"
                              ~"West South Central",
                              state == "Arizona"|
                              state == "Colorado"|
                              state == "Idaho"|
                              state == "Montana"|
                              state == "Nevada"|
                              state == "New Mexico"|
                              state == "Utah"|
                              state == "Wyoming"
                              ~ "Mountain",
                              state == "Alaska"|
                              state == "California"|
                              state == "Hawaii"|
                              state == "Oregon"|
                              state == "Washington"
                              ~"Pacific")) %>%
  mutate(region = case_when(division == "New England"|
                            division == "Mid-Atlantic"
                            ~ "Northeast",
                            division == "East North Central"| 
                            division == "West North Central"
                            ~ "Midwest",
                            division == "South Atlantic"|
                            division == "East South Central"| 
                            division == "West South Central"
                            ~ "South",
                            division == "Mountain"|
                            division ==   "Pacific" 
                            ~"West"
))

```

## Identify Missing Data
  
```{r}
#MISSING DATES
rewire_leg_prop_miss <- rewire_leg2 %>%
  filter(proposed == "")

view(rewire_leg_prop_miss)
```

## Exploring Data

```{r}
#View state counts
state_counts <- rewire_leg2 %>%
  filter(date > "2010-01-01" & date < "2020-12-31") %>%
  group_by(state) %>%
  count(state) %>%
  arrange(desc(n))

```

```{r state count table 1}
kable(state_counts, 
      col.names = c("state", "n"), 
      align = "lc",
      caption = "Table 1: State Counts of All Introduced Legislation, 2010 - 2020")
```

## Rough Histogram
```{r}
states_legdates <- rewire_leg2 %>%
  filter(date > "2010-01-01" & date < "2020-12-31")
```

```{r fig.cap = "Abortion Legislation Introduced 2010 - 2020"}
ggplot(states_legdates, aes(x = date, fill = region)) +
geom_histogram(binwidth=7) +
  labs(x = "",  y="")
```
```

## Processing Steps
- ~~**Rename** columns to *lower* case and update~~
- ~~Correctly classify `date`~~
- Clean up the `state` so federal legislation is tracked differently for the 330 bills.
- Classify which bills are anti and pro choice politically.
- Make plan for classifying missing values. 
  - Email rewire? 
  - Visualize where the gaps are.
- Graph each region's activity over time.
- ~~Classify states by region~~
- Link to census data?
- "https://rewirenewsgroup.com/legislative-tracker/legal-case/" + title of the bill (spaces and punct replaced with "-")
- ~~Merge duplicate values, preserve different "type" data.~~

## Questions
- Why are we missing so much data for 2000 - 2010 and 2010?
- Why is there a jump in 2013? Is this database even well-maintained enough to use?

## Potential Inquiries
- [Open States](https://docs.openstates.org/api-v3/) shows state legislation and has an open API.
- [ProPublica](https://projects.propublica.org/api-docs/congress-api/) has a federal congressional open API.
- [Govtrack](https://www.govtrack.us/congress/bills/subjects/sex_and_reproductive_health/6186#track=) would need to be scraped, but is more reliable for federal legislation than Rewire.