Entity_Resolution_Writeup.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Q: Describe your entity resolution technique, as well as its precision, recall, and F1 score. What were the most important features that powered your technique? How did you avoid pairwise comparison of all movies across both datasets?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A: I started out by cleaning the Amazon and Rotten Tomatoes datasets. For the time attributes in each dataset, I constructed a regular expression that would parse the hours and/or minutes as capturing groups, and then convert them to a single measure, in minutes. In cases where one of the time attributes had a value of NaN or a date, I changed that value to 0.\n",
    "\n",
    "Next, I looked at the star attributes. If any of those fields had a null value, I replaced it with an empty string.\n",
    "\n",
    "I chose to disregard the cost attribute in the Amazon dataset, as well as the ratings, reviews, and year attributes in the Rotten Tomatoes dataset. I did so because these features were not found in both files. As such, I considered the time, director, and star features to be the most important ones for my entity resolution technique, as they are all common to both datasets.\n",
    "\n",
    "To avoid pairwise comparison of all movies across both datasets, I used the training data file (*train.csv*) and joined it to the Amazon and Rotten Tomatoes files by using their corresponding IDs. Each row of the training dataset had a pair of IDs, one for a movie in the Amazon file, and another for a movie in the Rotten Tomatoes file. All of my comparisons were only made between these provided ID pairs.\n",
    "\n",
    "For this joined dataset, I computed three new features:\n",
    "The first feature calculates the absolute difference in the two time attributes (*time_diff*).\n",
    "\n",
    "The second feature computes the normalized Damerau-Levenshtein Distance between the two director attributes. I chose this metric because it allows for typos when considering the similarities between two strings, including insertions, deletions, substitutions, and transpositions of proximate characters. Normalizing this score produces a value between 0 and 1, making it easy to compare scores for different pairs to each other.\n",
    "\n",
    "The third feature computes the Jaccard Distance between the star feature of the Amazon dataset and the concatenation of all six star attributes in the Rotten Tomatoes dataset. I used this measure because Amazon has a comma-separated list of one or more actors/actresses in its single star feature, while Rotten Tomatoes has up to six actors/actresses, with each in their own star attribute. Because there is no clear, unified ordering of these names (main actor/actress, supporting actors/actresses, etc.), and because some names in an entity pair may appear in one dataset and not another, it would not be ideal to compare them using the Levenshtein Distance. Instead, the names can be broken up into individual tokens (words) to compare the proportion of names that are present between both entities.\n",
    "\n",
    "Once these three new features were engineered, I dropped all others except for the IDs and the gold feature. I randomly partitioned the dataset into 90% training and 10% holdout subsets. I developed a random forest classifier by doing the following:\n",
    "1. Using random search to optimize two of the hyperparameters of the random forest: \n",
    "\n",
    "    a. the number of trees (a integer selected uniformly at random between 5 and 500)\n",
    "    \n",
    "    b. the class weights to be learned (balanced with respect to class frequencies, with or without bootstrapping)\n",
    "    \n",
    "2. Applying stratified 10-fold cross-validation\n",
    "3. Optimizing with respect to the F1 score\n",
    "4. Training this model on the new 90% training subset.\n",
    "\n",
    "This random search for optimial hyperparameters was done over 30 iterations. Afterwards, the hyperparameters that maximized the F1 score were determined to be 187 trees and balanced class weights with bootstrapping. The model with these hyperparameters was chosen, and evaluated on the 10% holdout subset by examining the resulting precision-recall curve. On the 10% holdout subset, the entity resolution method achieved 100% precision, 66.7% recall, and an F1 score of 80%.\n",
    "\n",
    "This model was retrained on the full training set. On the full set, the method produced 100% precision, 96.4% recall, and an F1 score of 98.2%.\n",
    "\n",
    "The holdout and test sets were processed in the same manner as the training set (in terms of features engineered and data cleaning). The model was applied to the holdout and test sets in order to produce predictions of whether the entity pairs within each dataset corresponded to the same movie. According to Instabase, the method produced 91.25% precision, recall, and F1 on the test file (*test.csv*).\n",
    "\n",
    "\n",
    "I used the following external packages in my Python script:\n",
    "1. pandas\n",
    "2. numpy\n",
    "3. matplotlib\n",
    "4. re\n",
    "5. dateutil\n",
    "6. pyxdameraulevenshtein\n",
    "7. scipy\n",
    "8. sklearn\n",
    "9. sys\n",
    "10. \\__future__\n",
    "\n",
    "My script assumes that it is being executed in the same directory where the training and testing sets are located. Further, it also assumes that the name of the dataset to be tested is a parameter. The script would then be executed as such:\n",
    "\n",
    "~~~~\n",
    "python entity_resolution.py TESTFILE\n",
    "~~~~"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [Root]",
   "language": "python",
   "name": "Python [Root]"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}