RNN_LSTM_using_Glove_vectors.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Book Genre Classification using RNNs and LSTM\n",
    "\n",
    "In this notebook, we'll be looking at how to apply deep learning techniques to classify books using just their titles. Text Classification can be thought of as the exercise of taking a sentence, paragraph, document and determining what kind it belongs it like in case of sentiment analysis it could be sad, happy or positive, negative. \n",
    "\n",
    "This notebook will go through numerous topics like word vectors, recurrent neural networks, and long short-term memory units (LSTMs). After getting a good understanding of these terms, we’ll walk through concrete code examples and a full Tensorflow sentiment classifier at the end."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Using an RNN rather than a feedfoward network is more accurate since we can include information about the sequence of words.\n",
    "\n",
    "In order to understand how deep learning can be applied, think about all the different forms of data that are used as inputs into machine learning or deep learning models. Convolutional neural networks use arrays of pixel values, logistic regression uses quantifiable features, and reinforcement learning models use reward signals. The common theme is that the inputs need to be scalar values, or matrices of scalar values. When you think of NLP tasks, however, a data pipeline like this may come to mind. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "We want these vectors to be created in such a way that they somehow represent the word and its context, meaning, and semantics. For example, we’d like the vectors for the words “love” and “adore” to reside in relatively the same area in the vector space since they both have similar definitions and are both used in similar contexts. The vector representation of a word is also known as a word embedding.\n",
    "\n",
    "![caption](assets/vectors.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Loading our dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import re\n",
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "To create vectors for book titles, we're going to be using a pretrained model. \n",
    "\n",
    "For this task, we'll be using a much more manageable matrix that is trained using [GloVe](http://nlp.stanford.edu/projects/glove/), a similar word vector generation model. The matrix will contain 400,000 word vectors, each with a dimensionality of 50. \n",
    "\n",
    "We're going to be importing two different data structures, one will be a Python list with the 400,000 words, and one will be a 400,000 x 50 dimensional embedding matrix that holds all of the word vector values. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Here, we'll pass in words to an embedding layer. We need an embedding layer because we have tens of thousands of words, so we'll need a more efficient representation for our input data than one-hot encoded vectors.\n",
    "\n",
    "From the embedding layer, the new representations will be passed to LSTM cells. These will add recurrent connections to the network so we can include information about the sequence of words in the data. Finally, the LSTM cells will go to a sigmoid output layer here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded the word list!\n",
      "Loaded the word vectors!\n"
     ]
    }
   ],
   "source": [
    "wordsList = np.load('wordsList.npy')\n",
    "print('Loaded the word list!')\n",
    "wordsList = wordsList.tolist() #Originally loaded as numpy array\n",
    "wordsList = [word.decode('UTF-8') for word in wordsList] #Encode words as UTF-8\n",
    "wordVectors = np.load('wordVectors.npy')\n",
    "print ('Loaded the word vectors!')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['0', ',', '.', 'of', 'to', 'and', 'in', 'a', '\"', \"'s\", 'for', '-', 'that', 'on', 'is', 'was', 'said', 'with', 'he', 'as', 'it', 'by', 'at', '(', ')', 'from', 'his', \"''\", '``', 'an', 'be', 'has', 'are', 'have', 'but', 'were', 'not', 'this', 'who', 'they', 'had', 'i', 'which', 'will', 'their', ':', 'or', 'its', 'one', 'after', 'new', 'been', 'also', 'we', 'would', 'two', 'more', \"'\", 'first', 'about', 'up', 'when', 'year', 'there', 'all', '--', 'out', 'she', 'other', 'people', \"n't\", 'her', 'percent', 'than', 'over', 'into', 'last', 'some', 'government', 'time', '$', 'you', 'years', 'if', 'no', 'world', 'can', 'three', 'do', ';', 'president', 'only', 'state', 'million', 'could', 'us', 'most', '_', 'against', 'u.s.']\n",
      "400000\n"
     ]
    }
   ],
   "source": [
    "print (wordsList[:100])\n",
    "print (len(wordsList))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[  2.96050012e-01  -1.38410002e-01   4.37740013e-02  -3.87439996e-01\n",
      "   1.22620001e-01  -6.51799977e-01  -2.82400012e-01   9.03119966e-02\n",
      "  -5.51859975e-01   3.20600003e-01   3.74220009e-03   9.32290018e-01\n",
      "  -2.20339999e-01  -2.19219998e-01   9.21700001e-01   7.57239997e-01\n",
      "   8.48919988e-01  -4.21970012e-03   5.36260009e-01  -1.26670003e+00\n",
      "  -6.10279977e-01   1.66999996e-01   8.27530026e-01   6.57649994e-01\n",
      "   4.89589989e-01  -1.97440004e+00  -1.14900005e+00  -2.14609995e-01\n",
      "   8.05390000e-01  -1.47449994e+00   3.74900007e+00   1.01409996e+00\n",
      "  -1.12930000e+00  -5.26610017e-01  -1.20290004e-01  -2.79309988e-01\n",
      "   6.50919974e-02  -4.36390005e-02   6.04260027e-01  -2.08920002e-01\n",
      "  -4.57390010e-01   1.04409996e-02   4.14579988e-01   6.89000010e-01\n",
      "   1.44679993e-01  -3.19730006e-02  -4.80730012e-02  -1.12790003e-04\n",
      "   1.38540000e-01   9.69540000e-01]\n",
      "(400000, 50)\n"
     ]
    }
   ],
   "source": [
    "print (wordVectors[88])\n",
    "print (wordVectors.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Loading the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "data = pd.read_csv('data/book32-listing.csv',encoding = \"ISO-8859-1\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "columns = ['Id', 'Image', 'Image_link', 'Title', 'Author', 'Class', 'Genre']\n",
    "data.columns = columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "books = pd.DataFrame(data['Title'])\n",
    "author = pd.DataFrame(data['Author'])\n",
    "genre = pd.DataFrame(data['Genre'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Doug the Pug 2016 Wall Calendar</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Moleskine 2016 Weekly Notebook, 12M, Large, Bl...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>365 Cats Color Page-A-Day Calendar 2016</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Sierra Club Engagement Calendar 2016</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Sierra Club Wilderness Calendar 2016</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               Title\n",
       "0                    Doug the Pug 2016 Wall Calendar\n",
       "1  Moleskine 2016 Weekly Notebook, 12M, Large, Bl...\n",
       "2            365 Cats Color Page-A-Day Calendar 2016\n",
       "3               Sierra Club Engagement Calendar 2016\n",
       "4               Sierra Club Wilderness Calendar 2016"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "books.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Genre</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Calendars</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Calendars</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Calendars</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Calendars</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Calendars</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Genre\n",
       "0  Calendars\n",
       "1  Calendars\n",
       "2  Calendars\n",
       "3  Calendars\n",
       "4  Calendars"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "genre.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "32\n",
      "['Calendars' 'Comics & Graphic Novels' 'Test Preparation'\n",
      " 'Mystery, Thriller & Suspense' 'Science Fiction & Fantasy' 'Romance'\n",
      " 'Humor & Entertainment' 'Literature & Fiction' 'Gay & Lesbian'\n",
      " 'Engineering & Transportation' 'Cookbooks, Food & Wine'\n",
      " 'Crafts, Hobbies & Home' 'Arts & Photography' 'Education & Teaching'\n",
      " 'Parenting & Relationships' 'Self-Help' 'Computers & Technology'\n",
      " 'Medical Books' 'Science & Math' 'Health, Fitness & Dieting'\n",
      " 'Business & Money' 'Law' 'Biographies & Memoirs' 'History'\n",
      " 'Politics & Social Sciences' 'Reference' 'Christian Books & Bibles'\n",
      " 'Religion & Spirituality' 'Sports & Outdoors' 'Teen & Young Adult'\n",
      " \"Children's Books\" 'Travel']\n"
     ]
    }
   ],
   "source": [
    "print (len((genre['Genre'].unique())))\n",
    "print ((genre['Genre'].unique()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "207571\n",
      "207571\n"
     ]
    }
   ],
   "source": [
    "numBooks =  (len(books))\n",
    "numGenres =  (len(genre))\n",
    "print (numBooks)\n",
    "print (numGenres)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "19087\n",
      "[ 0.24808     0.032891   -0.062788   -0.45602    -0.15155999 -0.66189998\n",
      " -0.51319999 -0.092383   -0.67546999  1.94749999  1.13080001  0.94336998\n",
      " -0.90491003  0.18035001  1.11570001 -0.32481     0.44242001 -0.46054\n",
      " -1.19659996 -0.17732    -1.7263      0.5043     -0.28464001 -0.45795\n",
      "  0.33803999 -0.13706     1.20309997  0.34261    -0.58967    -0.70613003\n",
      "  0.62695998  0.76880997 -0.46966001 -0.081293    0.39987001  0.89332002\n",
      "  0.40821001  0.73082     0.077319   -0.81061     0.61938     0.95126998\n",
      " -0.20062999 -0.19441    -0.70710999  0.093774    0.99932998  0.05538\n",
      "  0.20422    -1.36650002]\n",
      "(50,)\n"
     ]
    }
   ],
   "source": [
    "aWordIndex = wordsList.index('messi')\n",
    "print (aWordIndex)\n",
    "print (wordVectors[aWordIndex])\n",
    "print (wordVectors[aWordIndex].shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now that we have our vectors, our first step is taking an input sentence and then constructing its vector representation. Let's say that we have the input sentence \"I thought the movie was incredible and inspiring\". In order to get the word vectors, we can use Tensorflow's embedding lookup function.\n",
    "\n",
    "This function takes in two arguments, one for the embedding matrix (the wordVectors matrix in our case), and one for the ids of each of the words. The ids vector can be thought of as the integerized representation of the training set. This is basically just the row index of each of the words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(10,)\n",
      "[ 41 317  40   7 835 523   0   0   0   0]\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "maxSeqLength = 10 #Maximum length of sentence\n",
    "numDimensions = 300 #Dimensions for each word vector\n",
    "firstSentence = np.zeros((maxSeqLength), dtype='int32')\n",
    "firstSentence[0] = wordsList.index(\"i\")\n",
    "firstSentence[1] = wordsList.index(\"too\")\n",
    "firstSentence[2] = wordsList.index(\"had\")\n",
    "firstSentence[3] = wordsList.index(\"a\")\n",
    "firstSentence[4] = wordsList.index(\"love\")\n",
    "firstSentence[5] = wordsList.index(\"story\")\n",
    "#firstSentence[6] = wordsList.index(\"k\")\n",
    "#firstSentence[7] = wordsList.index(\"ki\")\n",
    "print(firstSentence.shape)\n",
    "print(firstSentence) #Shows the row index for each word"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Word vector for a single word [  1.18910000e-01   1.52549997e-01  -8.20730031e-02  -7.41439998e-01\n",
      "   7.59169996e-01  -4.83280003e-01  -3.10090005e-01   5.14760017e-01\n",
      "  -9.87079978e-01   6.17570011e-04  -1.50429994e-01   8.37700009e-01\n",
      "  -1.07969999e+00  -5.14599979e-01   1.31879997e+00   6.20069981e-01\n",
      "   1.37789994e-01   4.71080005e-01  -7.28740022e-02  -7.26750016e-01\n",
      "  -7.41159976e-01   7.52629995e-01   8.81799996e-01   2.95610011e-01\n",
      "   1.35479999e+00  -2.57010007e+00  -1.35230005e+00   4.58799988e-01\n",
      "   1.00680006e+00  -1.18560004e+00   3.47370005e+00   7.78980017e-01\n",
      "  -7.29290009e-01   2.51020014e-01  -2.61559993e-01  -3.46839994e-01\n",
      "   5.58409989e-01   7.50980020e-01   4.98299986e-01  -2.68229991e-01\n",
      "  -2.74430006e-03  -1.82980001e-02  -2.80959994e-01   5.53179979e-01\n",
      "   3.77059989e-02   1.85550004e-01  -1.50250003e-01  -5.75119972e-01\n",
      "  -2.66710013e-01   9.21209991e-01]\n",
      "\n",
      "\n",
      "Dimension for a single word (50,)\n",
      "\n",
      "\n",
      "Shape of the vector for this sentence (10, 50)\n",
      "\n",
      "\n",
      "Vector for this sentence [[  1.18910000e-01   1.52549997e-01  -8.20730031e-02  -7.41439998e-01\n",
      "    7.59169996e-01  -4.83280003e-01  -3.10090005e-01   5.14760017e-01\n",
      "   -9.87079978e-01   6.17570011e-04  -1.50429994e-01   8.37700009e-01\n",
      "   -1.07969999e+00  -5.14599979e-01   1.31879997e+00   6.20069981e-01\n",
      "    1.37789994e-01   4.71080005e-01  -7.28740022e-02  -7.26750016e-01\n",
      "   -7.41159976e-01   7.52629995e-01   8.81799996e-01   2.95610011e-01\n",
      "    1.35479999e+00  -2.57010007e+00  -1.35230005e+00   4.58799988e-01\n",
      "    1.00680006e+00  -1.18560004e+00   3.47370005e+00   7.78980017e-01\n",
      "   -7.29290009e-01   2.51020014e-01  -2.61559993e-01  -3.46839994e-01\n",
      "    5.58409989e-01   7.50980020e-01   4.98299986e-01  -2.68229991e-01\n",
      "   -2.74430006e-03  -1.82980001e-02  -2.80959994e-01   5.53179979e-01\n",
      "    3.77059989e-02   1.85550004e-01  -1.50250003e-01  -5.75119972e-01\n",
      "   -2.66710013e-01   9.21209991e-01]\n",
      " [  2.86669999e-01  -3.80199999e-01  -4.18169983e-02  -8.58460009e-01\n",
      "    6.24799989e-02   2.59860009e-02  -3.24589998e-01   1.80820003e-01\n",
      "   -6.56059980e-01   5.78800023e-01  -4.66969997e-01   4.79079992e-01\n",
      "   -3.97249997e-01   2.36399993e-01   5.29340029e-01   5.40769994e-01\n",
      "    5.60329974e-01  -9.96870026e-02   1.86130002e-01  -1.06729996e+00\n",
      "   -5.57240009e-01   5.07780015e-01   5.60959995e-01   2.22900003e-01\n",
      "    6.41600013e-01  -2.13310003e+00  -7.09439993e-01   8.67299974e-01\n",
      "    9.85189974e-01  -2.61889994e-01   3.43400002e+00   4.33250010e-01\n",
      "    2.89270014e-01  -2.30430007e-01   1.49240000e-02  -8.17599967e-02\n",
      "   -2.57099986e-01   6.59169972e-01  -1.13669999e-01  -5.97159982e-01\n",
      "   -4.59520012e-01   5.84209979e-01   4.07099992e-01   8.60480011e-01\n",
      "    2.11099997e-01  -4.43519987e-02  -2.72309989e-01  -1.30640000e-01\n",
      "    1.83489993e-01   5.86989999e-01]\n",
      " [  6.03479981e-01  -5.20959973e-01   4.08510000e-01  -3.72170001e-01\n",
      "    3.69780004e-01   6.10819995e-01  -1.32280004e+00   2.43750006e-01\n",
      "   -5.94200015e-01  -3.57080013e-01   3.99419993e-01   3.19110006e-02\n",
      "   -1.06429994e+00  -5.23270011e-01   7.14529991e-01   6.33839965e-02\n",
      "   -4.63829994e-01  -3.46410006e-01  -7.24449992e-01  -1.37140006e-01\n",
      "   -1.91790000e-01   7.22249985e-01   6.29499972e-01  -8.08600008e-01\n",
      "   -3.76939997e-02  -2.03550005e+00   1.05659999e-01  -3.85910012e-02\n",
      "   -2.32010007e-01  -2.96270013e-01   3.32150006e+00   3.24429981e-02\n",
      "    8.53680000e-02  -4.07709986e-01   4.53410000e-01  -9.96740013e-02\n",
      "    4.47039992e-01   5.42200029e-01   1.81850001e-01   1.75040007e-01\n",
      "   -3.38330001e-01   3.16969991e-01  -2.52679996e-02   9.57949981e-02\n",
      "   -2.50710011e-01  -4.75639999e-01  -1.04069996e+00  -1.51380002e-01\n",
      "   -2.20569998e-01  -5.96329987e-01]\n",
      " [  2.17050001e-01   4.65149999e-01  -4.67570007e-01   1.00819997e-01\n",
      "    1.01349998e+00   7.48449981e-01  -5.31040013e-01  -2.62560010e-01\n",
      "    1.68119997e-01   1.31819993e-01  -2.49090001e-01  -4.41850007e-01\n",
      "   -2.17390001e-01   5.10039985e-01   1.34480000e-01  -4.31410015e-01\n",
      "   -3.12300008e-02   2.06740007e-01  -7.81379998e-01  -2.01480001e-01\n",
      "   -9.74010006e-02   1.60879999e-01  -6.18359983e-01  -1.85039997e-01\n",
      "   -1.24609999e-01  -2.25259995e+00  -2.23210007e-01   5.04299998e-01\n",
      "    3.22569996e-01   1.53129995e-01   3.96359992e+00  -7.13649988e-01\n",
      "   -6.70120001e-01   2.83879995e-01   2.17380002e-01   1.44329995e-01\n",
      "    2.59259999e-01   2.34339997e-01   4.27399993e-01  -4.44510013e-01\n",
      "    1.38129994e-01   3.69729996e-01  -6.42889977e-01   2.41420008e-02\n",
      "   -3.93150002e-02  -2.60369986e-01   1.20169997e-01  -4.37819995e-02\n",
      "    4.10129994e-01   1.79600000e-01]\n",
      " [ -1.38860002e-01   1.14010000e+00  -8.52119982e-01  -2.92120010e-01\n",
      "    7.55339980e-01   8.27620029e-01  -3.18100005e-01   7.22040003e-03\n",
      "   -3.47620010e-01   1.07309997e+00  -2.46649995e-01   9.77649987e-01\n",
      "   -5.58350027e-01  -9.03180018e-02   8.31820011e-01  -3.33169997e-01\n",
      "    2.26480007e-01   3.09130013e-01   2.69290004e-02  -8.67390037e-02\n",
      "   -1.47029996e-01   1.35430002e+00   5.36949992e-01   4.37350005e-01\n",
      "    1.27489996e+00  -1.43820000e+00  -1.28149998e+00  -1.51960000e-01\n",
      "    1.05060005e+00  -9.36439991e-01   2.75609994e+00   5.89670002e-01\n",
      "   -2.94730008e-01   2.75739998e-01  -3.29279989e-01  -2.01000005e-01\n",
      "   -2.85470009e-01  -4.59870011e-01  -1.46029994e-01  -6.93719983e-01\n",
      "    7.07610026e-02  -1.93259999e-01  -1.85499996e-01  -1.60950005e-01\n",
      "    2.42679998e-01   2.07839996e-01   3.09239998e-02  -1.37109995e+00\n",
      "   -2.86060005e-01   2.89799988e-01]\n",
      " [  4.82510000e-01   8.77460003e-01  -2.34549999e-01   2.62000002e-02\n",
      "    7.96909988e-01   4.31019992e-01  -6.09019995e-01  -6.07640028e-01\n",
      "   -4.28119987e-01  -1.25230001e-02  -1.28939998e+00   5.26560009e-01\n",
      "   -8.27629983e-01   3.06890011e-01   1.19719994e+00  -4.76740003e-01\n",
      "   -4.68849987e-01  -1.95240006e-01  -2.84029990e-01   3.52369994e-01\n",
      "    4.55359995e-01   7.68530011e-01   6.21569995e-03   5.54210007e-01\n",
      "    1.00059998e+00  -1.39730000e+00  -1.68939996e+00   3.00029993e-01\n",
      "    6.06779993e-01  -4.60440010e-01   2.59610009e+00  -1.21780002e+00\n",
      "    2.87470013e-01  -4.61750001e-01  -2.59429991e-01   3.82090002e-01\n",
      "   -2.83120006e-01  -4.76419985e-01  -5.94439991e-02  -5.92019975e-01\n",
      "    2.56130010e-01   2.13060006e-01  -1.61290001e-02  -2.98729986e-01\n",
      "   -1.94680005e-01   5.36109984e-01   7.54589975e-01  -4.11199987e-01\n",
      "    2.36249998e-01   2.64510006e-01]\n",
      " [  0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00\n",
      "    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00\n",
      "    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00\n",
      "    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00\n",
      "    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00\n",
      "    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00\n",
      "    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00\n",
      "    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00\n",
      "    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00\n",
      "    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00\n",
      "    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00\n",
      "    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00\n",
      "    0.00000000e+00   0.00000000e+00]\n",
      " [  0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00\n",
      "    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00\n",
      "    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00\n",
      "    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00\n",
      "    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00\n",
      "    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00\n",
      "    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00\n",
      "    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00\n",
      "    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00\n",
      "    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00\n",
      "    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00\n",
      "    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00\n",
      "    0.00000000e+00   0.00000000e+00]\n",
      " [  0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00\n",
      "    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00\n",
      "    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00\n",
      "    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00\n",
      "    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00\n",
      "    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00\n",
      "    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00\n",
      "    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00\n",
      "    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00\n",
      "    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00\n",
      "    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00\n",
      "    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00\n",
      "    0.00000000e+00   0.00000000e+00]\n",
      " [  0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00\n",
      "    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00\n",
      "    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00\n",
      "    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00\n",
      "    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00\n",
      "    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00\n",
      "    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00\n",
      "    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00\n",
      "    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00\n",
      "    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00\n",
      "    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00\n",
      "    0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00\n",
      "    0.00000000e+00   0.00000000e+00]]\n"
     ]
    }
   ],
   "source": [
    "with tf.Session() as sess:\n",
    "    print(\"Word vector for a single word\", tf.nn.embedding_lookup(wordVectors, firstSentence).eval()[0])\n",
    "    print(\"\\n\")\n",
    "    print(\"Dimension for a single word\", tf.nn.embedding_lookup(wordVectors, firstSentence).eval()[0].shape)\n",
    "    print ('\\n')\n",
    "    print(\"Shape of the vector for this sentence\", tf.nn.embedding_lookup(wordVectors, firstSentence).eval().shape)\n",
    "    print (\"\\n\")\n",
    "    print(\"Vector for this sentence\", tf.nn.embedding_lookup(wordVectors, firstSentence).eval())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Data Pipeline\n",
    "\n",
    "![caption](assets/pipeline.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Basic dataset Exploration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "numWords = []\n",
    "for i,j in enumerate(books['Title']):\n",
    "    counter = len(j.split())\n",
    "    numWords.append(counter)       "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The total number of words in all books is 1900083\n",
      "The average number of words in the each book title is 9.153894330132822\n"
     ]
    }
   ],
   "source": [
    "print('The total number of words in all books is', sum(numWords))\n",
    "print('The average number of words in the each book title is', sum(numWords)/len(numWords))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAEKCAYAAAAvlUMdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHbNJREFUeJzt3X2wXVWd5vHvY0CIvBkgZtJJmOAQZSCtkcQ0Peg0SisZ\nsQW7AUONErtThCoYG8ueaYPjtDg1qYZpFZseoQyChPgCadQmw0treFHLLkhMMBASpLklock1kPAi\ngR6JJjzzx1lHTq733pyb3JVzbu7zqTp11vntvfZZe+vNj7XXOmvLNhEREcPtNZ1uQERE7J+SYCIi\noookmIiIqCIJJiIiqkiCiYiIKpJgIiKiiuoJRtIYST+RdFv5fKSkFZIeK+/jWva9VFKPpEclnd4S\nnylpXdl2lSSV+EGSbi7xlZKm1j6fiIhoz77owVwCPNLyeSFwt+1pwN3lM5JOAOYCJwJzgKsljSl1\nrgEuAKaV15wSnw88b/s44ErgirqnEhER7aqaYCRNBs4AvtISPhNYUspLgLNa4jfZ3m77caAHmC1p\nInC47fvd+FXojX3qNI91C3Bas3cTERGddUDl438R+EvgsJbYBNubS/kpYEIpTwLub9lvU4n9upT7\nxpt1ngSwvUPSC8BRwDOtjZC0AFgAcMghh8w8/vjj9+6sIiJGmTVr1jxje/xQ6lRLMJLeD2yxvUbS\nqf3tY9uSqq9VY3sxsBhg1qxZXr16de2vjIjYr0h6Yqh1avZgTgE+IOl9wMHA4ZK+BjwtaaLtzeX2\n15ayfy8wpaX+5BLrLeW+8dY6myQdABwBPFvrhCIion3VxmBsX2p7su2pNAbv77H9YWA5MK/sNg+4\ntZSXA3PLzLBjaQzmryq307ZJOrmMr5zfp07zWGeX78jqnRERXaD2GEx/LgeWSZoPPAGcC2B7vaRl\nwAZgB3Cx7Z2lzkXADcBY4M7yArgOWCqpB3iORiKLiIguoNH2H/wZg4mIGDpJa2zPGkqd/JI/IiKq\nSIKJiIgqkmAiIqKKJJiIiKgiCSYiIqpIgomIiCqSYCIioookmIiIqCIJJiIiqkiCiYiIKpJgIiKi\niiSYiIioIgkmIiKq6MRy/TGAqQtv3+O6Gy8/YxhbEhGx99KDiYiIKpJgIiKiiiSYiIioIgkmIiKq\nqJZgJB0saZWkByWtl/TZEr9MUq+kteX1vpY6l0rqkfSopNNb4jMlrSvbrpKkEj9I0s0lvlLS1Frn\nExERQ1NzFtl24N22X5J0IPAjSXeWbVfa/lzrzpJOAOYCJwK/A9wl6U22dwLXABcAK4E7gDnAncB8\n4Hnbx0maC1wBfKjiOXWtvZmBBpmFFhHDr1oPxg0vlY8HlpcHqXImcJPt7bYfB3qA2ZImAofbvt+2\ngRuBs1rqLCnlW4DTmr2biIjorKpjMJLGSFoLbAFW2F5ZNn1M0kOSrpc0rsQmAU+2VN9UYpNKuW98\nlzq2dwAvAEdVOZmIiBiSqgnG9k7bM4DJNHoj02nc7nojMAPYDHy+ZhsAJC2QtFrS6q1bt9b+uoiI\nYB/NIrP9C+BeYI7tp0vieQW4FphddusFprRUm1xivaXcN75LHUkHAEcAz/bz/Yttz7I9a/z48cN3\nYhERMaCas8jGS3p9KY8F3gP8tIypNH0QeLiUlwNzy8ywY4FpwCrbm4Ftkk4u4yvnA7e21JlXymcD\n95RxmoiI6LCas8gmAkskjaGRyJbZvk3SUkkzaAz4bwQuBLC9XtIyYAOwA7i4zCADuAi4ARhLY/ZY\nczbadcBSST3AczRmoUVERBeolmBsPwS8rZ/4RwapswhY1E98NTC9n/jLwDl719KIiKghv+SPiIgq\nkmAiIqKKJJiIiKgiCSYiIqpIgomIiCqSYCIioookmIiIqCIJJiIiqkiCiYiIKpJgIiKiiiSYiIio\nIgkmIiKqSIKJiIgqkmAiIqKKJJiIiKgiCSYiIqpIgomIiCpqPjI5RpCpC2/f47obLz9jGFsSEfuL\n9GAiIqKKaglG0sGSVkl6UNJ6SZ8t8SMlrZD0WHkf11LnUkk9kh6VdHpLfKakdWXbVZJU4gdJurnE\nV0qaWut8IiJiaGr2YLYD77b9VmAGMEfSycBC4G7b04C7y2cknQDMBU4E5gBXSxpTjnUNcAEwrbzm\nlPh84HnbxwFXAldUPJ+IiBiCagnGDS+VjweWl4EzgSUlvgQ4q5TPBG6yvd3240APMFvSROBw2/fb\nNnBjnzrNY90CnNbs3URERGdVHYORNEbSWmALsML2SmCC7c1ll6eACaU8CXiypfqmEptUyn3ju9Sx\nvQN4ATiqn3YskLRa0uqtW7cOy7lFRMTgqiYY2zttzwAm0+iNTO+z3TR6NVXZXmx7lu1Z48ePr/11\nERHBPppFZvsXwL00xk6eLre9KO9bym69wJSWapNLrLeU+8Z3qSPpAOAI4Nk6ZxEREUNRcxbZeEmv\nL+WxwHuAnwLLgXllt3nAraW8HJhbZoYdS2Mwf1W5nbZN0sllfOX8PnWaxzobuKf0iiIiosNq/tBy\nIrCkzAR7DbDM9m2S7gOWSZoPPAGcC2B7vaRlwAZgB3Cx7Z3lWBcBNwBjgTvLC+A6YKmkHuA5GrPQ\nIiKiC1RLMLYfAt7WT/xZ4LQB6iwCFvUTXw1M7yf+MnDOXjc2IiKGXX7JHxERVSTBREREFUkwERFR\nRRJMRERUkQQTERFVJMFEREQVSTAREVFFEkxERFSRRyYPs7159HBExP4kPZiIiKgiCSYiIqpIgomI\niCqSYCIioookmIiIqCIJJiIiqkiCiYiIKpJgIiKiiiSYiIioolqCkTRF0r2SNkhaL+mSEr9MUq+k\nteX1vpY6l0rqkfSopNNb4jMlrSvbrpKkEj9I0s0lvlLS1FrnExERQ1OzB7MD+AvbJwAnAxdLOqFs\nu9L2jPK6A6BsmwucCMwBrpY0pux/DXABMK285pT4fOB528cBVwJXVDyfiIgYgmoJxvZm2w+U8ovA\nI8CkQaqcCdxke7vtx4EeYLakicDhtu+3beBG4KyWOktK+RbgtGbvJiIiOmufjMGUW1dvA1aW0Mck\nPSTpeknjSmwS8GRLtU0lNqmU+8Z3qWN7B/ACcFQ/379A0mpJq7du3Tos5xQREYOrnmAkHQp8C/i4\n7W00bne9EZgBbAY+X7sNthfbnmV71vjx42t/XUREUDnBSDqQRnL5uu1vA9h+2vZO268A1wKzy+69\nwJSW6pNLrLeU+8Z3qSPpAOAI4Nk6ZxMREUPRVoKR9LtDPXAZC7kOeMT2F1riE1t2+yDwcCkvB+aW\nmWHH0hjMX2V7M7BN0snlmOcDt7bUmVfKZwP3lHGaiIjosHYfOHa1pIOAG2j0Rl5oo84pwEeAdZLW\nltingPMkzQAMbAQuBLC9XtIyYAONGWgX295Z6l1UvnsscGd5QSOBLZXUAzxHYxZaRER0gbYSjO13\nSpoG/BmwRtIq4Ku2VwxS50dAfzO67hikziJgUT/x1cD0fuIvA+fs/gwiImJfa3sMxvZjwKeBTwJ/\nAFwl6aeS/rhW4yIiYuRqdwzmLZKupPFblncDf2T735fylRXbFxERI1S7YzB/B3wF+JTtXzaDtn8u\n6dNVWhYRESNauwnmDOCXzUF3Sa8BDrb9/2wvrda6iIgYsdodg7mLxgyupteVWERERL/aTTAH236p\n+aGUX1enSRERsT9oN8H8q6STmh8kzQR+Ocj+ERExyrU7BvNx4O8l/ZzGb1v+DfChaq2KiIgRr90f\nWv5Y0vHAm0voUdu/rtesiIgY6drtwQC8HZha6pwkCds3VmlVRESMeG0lGElLgX8HrAWa64M1H/4V\nERHxW9rtwcwCTshKxRER0a52Z5E9TGNgPyIioi3t9mCOBjaUVZS3N4O2P1ClVRERMeK1m2Auq9mI\niIjY/7Q7TfkHkv4tMM32XZJeB4yp27SIiBjJ2l2u/wLgFuDLJTQJ+IdajYqIiJGv3UH+i2k8Ankb\n/ObhY2+o1aiIiBj52k0w223/qvlB0gE0fgczIElTJN0raYOk9ZIuKfEjJa2Q9Fh5H9dS51JJPZIe\nlXR6S3ympHVl21WSVOIHSbq5xFdKmtr+qUdERE3tDvL/QNKngLGS3gNcBPzf3dTZAfyF7QckHQas\nkbQC+Chwt+3LJS0EFgKflHQCMBc4Efgd4C5JbyrPoLkGuABYCdwBzAHuBOYDz9s+TtJc4AqyRtqI\nMnXh7Xtcd+PlZwxjSyJiuLXbg1kIbAXWARfS+Ed+0CdZ2t5s+4FSfpHG45YnAWcCS8puS4CzSvlM\n4Cbb220/DvQAsyVNBA63fX/5oeeNfeo0j3ULcFqzdxMREZ3V7iyyV4Bry2vIyq2rt9HogUywvbls\negqYUMqTgPtbqm0qsV+Xct94s86TpY07JL0AHAU80+f7FwALAI455pg9OYWIiBiidtcie5x+xlxs\nv7GNuocC3wI+bntbawfDtiVVX37G9mJgMcCsWbOy3E1ExD4wlLXImg4GzgGO3F0lSQfSSC5ft/3t\nEn5a0kTbm8vtry0l3gtMaak+ucR6S7lvvLXOpjLx4Ajg2TbPKSIiKmprDMb2sy2vXttfBAYdYS1j\nIdcBj9j+Qsum5cC8Up4H3NoSn1tmhh0LTANWldtp2ySdXI55fp86zWOdDdyTBTkjIrpDu7fITmr5\n+BoaPZrd1T0F+AiwTtLaEvsUcDmwTNJ84AngXADb6yUtAzbQmIF2cZlBBo1ZazcAY2nMHruzxK8D\nlkrqAZ6jMQstIiK6QLu3yD7fUt4BbKQkhoHY/hGNxyv357QB6iwCFvUTXw1M7yf+Mo3bdRER0WXa\nnUX2rtoNiYiI/Uu7t8g+Mdj2PmMsERERQ5pF9nYag+oAfwSsAh6r0aiIiBj52k0wk4GTyi/ykXQZ\ncLvtD9dqWEREjGztLhUzAfhVy+df8eov8CMiIn5Luz2YG4FVkr5TPp/Fq2uARURE/JZ2Z5EtknQn\n8M4S+lPbP6nXrIiIGOnavUUG8Dpgm+2/pbE0y7GV2hQREfuBdh+Z/Bngk8ClJXQg8LVajYqIiJGv\n3R7MB4EPAP8KYPvnwGG1GhURESNfuwnmV2URSQNIOqRekyIiYn/QboJZJunLwOslXQDcxR4+fCwi\nIkaHdmeRfU7Se4BtwJuBv7K9omrLIiJiRNttgpE0BrirLHiZpBIREW3Z7S2y8kyWVyQdsQ/aExER\n+4l2f8n/Eo0Hh62gzCQDsP3nVVoVEREjXrsJ5tvlFRER0ZZBE4ykY2z/i+2sOxYREUOyuzGYf2gW\nJH1rKAeWdL2kLZIeboldJqlX0tryel/Ltksl9Uh6VNLpLfGZktaVbVdJUokfJOnmEl8paepQ2hcR\nEXXtLsGopfzGIR77BmBOP/Erbc8orzsAJJ0AzAVOLHWuLrPXAK4BLgCmlVfzmPOB520fB1wJXDHE\n9kVEREW7SzAeoLxbtn8IPNfm7mcCN9nebvtxoAeYLWkicLjt+8tKAjfSeFRAs07z1t0twGnN3k1E\nRHTe7hLMWyVtk/Qi8JZS3ibpRUnb9vA7PybpoXILbVyJTQKebNlnU4lNKuW+8V3q2N4BvAAc1d8X\nSlogabWk1Vu3bt3DZkdExFAMmmBsj7F9uO3DbB9Qys3Ph+/B911D41bbDGAz8Pk9OMaQ2V5se5bt\nWePHj98XXxkRMeoN5Xkwe83207Z32n6Fxlpms8umXmBKy66TS6y3lPvGd6kj6QDgCODZeq2PiIih\n2KcJpoypNH0QaM4wWw7MLTPDjqUxmL/K9mZgm6STy/jK+cCtLXXmlfLZwD1lnCYiIrpAuz+0HDJJ\n3wROBY6WtAn4DHCqpBk0JgxsBC4EsL1e0jJgA7ADuLgsUQNwEY0ZaWOBO8sL4DpgqaQeGpMJ5tY6\nl4iIGLpqCcb2ef2Erxtk/0XAon7iq4Hp/cRfBs7ZmzZGREQ9+/QWWUREjB5JMBERUUUSTEREVFFt\nDCaitqkLb9+r+hsvP2OYWhIR/UkPJiIiqkgPJvba3vYkImL/lB5MRERUkQQTERFVJMFEREQVSTAR\nEVFFEkxERFSRBBMREVUkwURERBVJMBERUUUSTEREVJEEExERVSTBREREFUkwERFRRbUEI+l6SVsk\nPdwSO1LSCkmPlfdxLdsuldQj6VFJp7fEZ0paV7ZdJUklfpCkm0t8paSptc4lIiKGrmYP5gZgTp/Y\nQuBu29OAu8tnJJ0AzAVOLHWuljSm1LkGuACYVl7NY84Hnrd9HHAlcEW1M4mIiCGrlmBs/xB4rk/4\nTGBJKS8BzmqJ32R7u+3HgR5gtqSJwOG277dt4MY+dZrHugU4rdm7iYiIztvXYzATbG8u5aeACaU8\nCXiyZb9NJTaplPvGd6ljewfwAnBUf18qaYGk1ZJWb926dTjOIyIidqNjg/ylR+J99F2Lbc+yPWv8\n+PH74isjIka9fZ1gni63vSjvW0q8F5jSst/kEust5b7xXepIOgA4Ani2WssjImJI9nWCWQ7MK+V5\nwK0t8bllZtixNAbzV5XbadsknVzGV87vU6d5rLOBe0qvKCIiusABtQ4s6ZvAqcDRkjYBnwEuB5ZJ\nmg88AZwLYHu9pGXABmAHcLHtneVQF9GYkTYWuLO8AK4DlkrqoTGZYG6tc4mIiKGrlmBsnzfAptMG\n2H8RsKif+Gpgej/xl4Fz9qaNERFRT37JHxERVSTBREREFUkwERFRRbUxmJFq6sLbO92E2Ef25n/r\njZefMYwtidg/pQcTERFVJMFEREQVSTAREVFFEkxERFSRBBMREVUkwURERBVJMBERUUUSTEREVJEE\nExERVSTBREREFUkwERFRRRJMRERUkQQTERFVJMFEREQVHUkwkjZKWidpraTVJXakpBWSHivv41r2\nv1RSj6RHJZ3eEp9ZjtMj6SpJ6sT5RETEb+tkD+ZdtmfYnlU+LwTutj0NuLt8RtIJwFzgRGAOcLWk\nMaXONcAFwLTymrMP2x8REYPopgeOnQmcWspLgO8Dnyzxm2xvBx6X1APMlrQRONz2/QCSbgTOAu7c\nt82O0SgPK4vYvU71YAzcJWmNpAUlNsH25lJ+CphQypOAJ1vqbiqxSaXcNx4REV2gUz2Yd9julfQG\nYIWkn7ZutG1JHq4vK0lsAcAxxxwzXIeNiIhBdKQHY7u3vG8BvgPMBp6WNBGgvG8pu/cCU1qqTy6x\n3lLuG+/v+xbbnmV71vjx44fzVCIiYgD7PMFIOkTSYc0y8F7gYWA5MK/sNg+4tZSXA3MlHSTpWBqD\n+avK7bRtkk4us8fOb6kTEREd1olbZBOA75QZxQcA37D9j5J+DCyTNB94AjgXwPZ6ScuADcAO4GLb\nO8uxLgJuAMbSGNzPAH9ERJfY5wnG9s+At/YTfxY4bYA6i4BF/cRXA9OHu40REbH38kv+iIioIgkm\nIiKqSIKJiIgqkmAiIqKKbloqJmJUyDIzMVqkBxMREVUkwURERBVJMBERUUUSTEREVJEEExERVSTB\nREREFUkwERFRRRJMRERUkR9aRowi+ZFn7EvpwURERBVJMBERUUVukUWMIHtzi6uT353ba6NTejAR\nEVFFejARUd3e9rzSAxqZRnyCkTQH+FtgDPAV25d3uEkRMcxye25kGtEJRtIY4EvAe4BNwI8lLbe9\nobMti4hukeTUOSM6wQCzgR7bPwOQdBNwJpAEExF7rZOTKvZGtyTGkZ5gJgFPtnzeBPxe350kLQAW\nlI/bJT28D9q2t44Gnul0I9qQdg6fkdBGSDuH27C3U1cM59F+481DrTDSE0xbbC8GFgNIWm17Voeb\ntFtp5/AaCe0cCW2EtHO4jaR2DrXOSJ+m3AtMafk8ucQiIqLDRnqC+TEwTdKxkl4LzAWWd7hNERHB\nCL9FZnuHpP8CfJfGNOXrba/fTbXF9Vs2LNLO4TUS2jkS2ghp53Dbb9sp2zUaEhERo9xIv0UWERFd\nKgkmIiKqGFUJRtIcSY9K6pG0sNPtGYikjZLWSVq7J1MDa5F0vaQtrb8jknSkpBWSHivv47qwjZdJ\n6i3Xc62k93WyjaVNUyTdK2mDpPWSLinxbrueA7Wzq66ppIMlrZL0YGnnZ0u8a67nIG3sqmvZJGmM\npJ9Iuq18HvK1HDVjMGVZmX+mZVkZ4LxuXFZG0kZglu2u+pGYpP8IvATcaHt6if1v4Dnbl5ekPc72\nJ7usjZcBL9n+XKfa1ZekicBE2w9IOgxYA5wFfJTuup4DtfNcuuiaShJwiO2XJB0I/Ai4BPhjuuR6\nDtLGOXTRtWyS9AlgFnC47ffvyd/6aOrB/GZZGdu/AprLykSbbP8QeK5P+ExgSSkvofGPT8cM0Mau\nY3uz7QdK+UXgERorU3Tb9RyonV3FDS+VjweWl+mi6zlIG7uOpMnAGcBXWsJDvpajKcH0t6xM1/2h\nFAbukrSmLHPTzSbY3lzKTwETOtmYQXxM0kPlFlpHbzv1JWkq8DZgJV18Pfu0E7rsmpZbOmuBLcAK\n2113PQdoI3TZtQS+CPwl8EpLbMjXcjQlmJHkHbZnAP8JuLjc9ul6btxv7cb/IrsGeCMwA9gMfL6z\nzXmVpEOBbwEft72tdVs3Xc9+2tl119T2zvJ3MxmYLWl6n+0dv54DtLGrrqWk9wNbbK8ZaJ92r+Vo\nSjAjZlkZ273lfQvwHRq397rV0+U+ffN+/ZYOt+e32H66/GG/AlxLl1zPch/+W8DXbX+7hLvuevbX\nzm69pgC2fwHcS2Nso+uuJ+zaxi68lqcAHyhjwTcB75b0NfbgWo6mBDMilpWRdEgZTEXSIcB7gW5e\n/Xk5MK+U5wG3drAt/Wr+URQfpAuuZxnwvQ54xPYXWjZ11fUcqJ3ddk0ljZf0+lIeS2Myz0/pous5\nUBu77VravtT2ZNtTafw7eY/tD7MH13JELxUzFHu4rEwnTAC+0/i75gDgG7b/sbNNapD0TeBU4GhJ\nm4DPAJcDyyTNB56gMbuoYwZo46mSZtDo0m8ELuxYA191CvARYF25Jw/wKbrsejJwO8/rsms6EVhS\nZou+Blhm+zZJ99E913OgNi7tsms5kCH/f3PUTFOOiIh9azTdIouIiH0oCSYiIqpIgomIiCqSYCIi\noookmIiIqCIJJvZ7kv57Wb32obJa7e91uk17Q9INks6uePwZrSv6ltV+/2ut74v916j5HUyMTpJ+\nH3g/cJLt7ZKOBl7b4WZ1uxk0VtG9o9MNiZEtPZjY300EnrG9HcD2M7Z/DiBppqQflEVFv9uyDMZM\nNZ7Z8aCkv1F5toykj0r6P80DS7pN0qml/F5J90l6QNLfl7W7ms/2+WyJr5N0fIkfKumrJfaQpD8Z\n7DjtkPTfJP24HK/5rJGpkh6RdG3pxX2v/IocSW9v6dX9jaSHyyoX/xP4UIl/qBz+BEnfl/QzSX++\nx/9rxKiSBBP7u+8BUyT9s6SrJf0B/GZ9rb8DzrY9E7geWFTqfBX4mO23tvMFpVf0aeAPbZ8ErAY+\n0bLLMyV+DdC81fQ/gBds/67ttwD3tHGcwdrwXmAajXWsZgAz9eoiqdOAL9k+EfgF8Cct53lhWXxx\nJ0B5lMVfATfbnmH75rLv8cDp5fifKdcvYlC5RRb7tfJwp5nAO4F3ATer8bCk1cB0YEVZlmcMsLms\nFfX68lwZgKU0VrUezMnACcA/lWO9FrivZXtzIcs1NB6ABfCHNNZ5arbzeTVWsR3sOIN5b3n9pHw+\nlEZi+RfgcdvNZV7WAFPLeR5mu3n8b9C4lTiQ20svcLukLTSWNNrUZttilEqCif2e7Z3A94HvS1pH\nY6G+NcB627/fum9zMcIB7GDXXv/BzWo0nu1x3gD1tpf3nQz+N7e74wxGwF/b/vIuwcYzXLa3hHYC\nY/fg+H2PkX87Yrdyiyz2a5LeLGlaS2gGjYX6HgXGl0kASDpQ0ollGfVfSHpH2f8/t9TdCMyQ9BpJ\nU3h1WfX7gVMkHVeOdYikN+2maSuAi1vaOW4Pj9P0XeDPWsZ+Jkl6w0A7l/N8sWVG3dyWzS8Ch7X5\nvREDSoKJ/d2hNFaw3SDpIRq3oC4rYw1nA1dIehBYC/yHUudPgS+V1YPVcqx/Ah4HNgBXAc1HCW8F\nPgp8s3zHfTTGLAbzv4BxZWD9QeBdQzzOlyVtKq/7bH+Pxm2u+0ov7RZ2nyTmA9eW8zwEeKHE76Ux\nqN86yB8xZFlNOWIQ5RbTbban72bXEUfSoc1nxJdxqYm2L+lws2I/kvuoEaPXGZIupfHvwBM0ek8R\nwyY9mIiIqCJjMBERUUUSTEREVJEEExERVSTBREREFUkwERFRxf8HrTEmjqOvXSwAAAAASUVORK5C\nYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10a932b38>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "plt.hist(numWords, 40)\n",
    "plt.xlabel('Sequence Length')\n",
    "plt.ylabel('Frequency')\n",
    "plt.axis([0, 40, 0, 40000])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "From the histogram as well as the average number of words, book title are 10 words, which is less than max sequence length value we will set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Removes punctuation, parentheses, question marks, etc., and leaves only alphanumeric characters\n",
    "import re\n",
    "strip_special_chars = re.compile(\"[^A-Za-z0-9 ]+\")\n",
    "\n",
    "def cleanSentences(string):\n",
    "    string = string.lower().replace(\"<br />\", \" \")\n",
    "    return re.sub(strip_special_chars, \"\", string.lower())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Doug the Pug 2016 Wall Calendar\n",
      "doug the pug 2016 wall calendar\n",
      "\n",
      "\n",
      "Face Reading in Chinese Medicine, 2e\n",
      "face reading in chinese medicine 2e\n",
      "\n",
      "\n",
      "Bare-Bones Meditation: Waking Up from the Story of My Life\n",
      "barebones meditation waking up from the story of my life\n",
      "\n",
      "\n",
      "The Last Taxi Ride: A Ranjit Singh Novel\n",
      "the last taxi ride a ranjit singh novel\n",
      "\n",
      "\n",
      "CCNA Routing and Switching ICND2 200-101 Official Cert Guide\n",
      "ccna routing and switching icnd2 200101 official cert guide\n",
      "\n",
      "\n",
      "City Secrets Paris: The Essential Insider's Guide\n",
      "city secrets paris the essential insiders guide\n"
     ]
    }
   ],
   "source": [
    "print (books['Title'][0])\n",
    "print (cleanSentences(books['Title'][0]))\n",
    "print ('\\n')\n",
    "print (books['Title'][100000])\n",
    "print (cleanSentences(books['Title'][100000]))\n",
    "print ('\\n')\n",
    "print (books['Title'][33330])\n",
    "print (cleanSentences(books['Title'][33330]))\n",
    "print ('\\n')\n",
    "print (books['Title'][8889])\n",
    "print (cleanSentences(books['Title'][8889]))\n",
    "print ('\\n')\n",
    "print (books['Title'][70000])\n",
    "print (cleanSentences(books['Title'][70000]))\n",
    "print ('\\n')\n",
    "print (books['Title'][200000])\n",
    "print (cleanSentences(books['Title'][200000]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "We will keep our vector sequence length equal to 20."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "maxSeqLength = 20"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 18min 5s, sys: 6.11 s, total: 18min 11s\n",
      "Wall time: 18min 25s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "features = np.zeros((numBooks, maxSeqLength), dtype='int32')\n",
    "\n",
    "fileCounter = 0\n",
    "\n",
    "for i,j in enumerate(books['Title']): \n",
    "    indexCounter = 0\n",
    "    cleanedLine = cleanSentences(j)\n",
    "    split = cleanedLine.split()\n",
    "    for word in split:\n",
    "        try:\n",
    "            features[fileCounter][indexCounter] = wordsList.index(word)\n",
    "        except ValueError:\n",
    "            features[fileCounter][indexCounter] = 399999 #Vector for unkown words\n",
    "        indexCounter = indexCounter + 1\n",
    "        if indexCounter >= maxSeqLength:\n",
    "            break\n",
    "    fileCounter = fileCounter + 1 \n",
    "\n",
    "np.save('features/idsMatrix20', features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "We have already created idMatrices for sequences 10,20 and 50. You can choose different matrix for experiemtnal purpose"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "features = np.load('features/idsMatrix20.npy')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(207571, 20)"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "features.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "First Book title from our dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[  6406 201534  65579  15463   1015   5657      0      0      0      0\n",
      "      0      0      0      0      0      0      0      0      0      0]\n"
     ]
    }
   ],
   "source": [
    "firstSentence = np.zeros((maxSeqLength), dtype='int32')\n",
    "firstSentence[0] = wordsList.index(\"doug\")\n",
    "firstSentence[1] = wordsList.index(\"the\")\n",
    "firstSentence[2] = wordsList.index(\"pug\")\n",
    "firstSentence[3] = wordsList.index(\"2016\")\n",
    "firstSentence[4] = wordsList.index(\"wall\")\n",
    "firstSentence[5] = wordsList.index(\"calendar\")\n",
    "print(firstSentence) #Shows the row index for each word"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Doug the Pug 2016 Wall Calendar\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([  6406, 201534,  65579,  15463,   1015,   5657,      0,      0,\n",
       "            0,      0,      0,      0,      0,      0,      0,      0,\n",
       "            0,      0,      0,      0], dtype=int32)"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print (books['Title'][0])\n",
    "features[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Converting genres(labels) using LabelEncoder and keras' to_categorical function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import LabelEncoder\n",
    "\n",
    "feat = ['Genre']\n",
    "for x in feat:\n",
    "    le = LabelEncoder()\n",
    "    le.fit(list(genre[x].values))\n",
    "    genre[x] = le.transform(list(genre[x]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "from keras.utils.np_utils import to_categorical\n",
    "target = to_categorical(genre['Genre'], 32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(207571, 32)"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "target.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Splitting into training and test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/anaconda/lib/python3.6/site-packages/sklearn/cross_validation.py:44: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.\n",
      "  \"This module will be removed in 0.20.\", DeprecationWarning)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.cross_validation import train_test_split "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(features, target, test_size=0.10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(186813, 20)\n",
      "(186813, 32)\n",
      "(20758, 20)\n",
      "(20758, 32)\n"
     ]
    }
   ],
   "source": [
    "print (X_train.shape)\n",
    "print (y_train.shape)\n",
    "print (X_test.shape)\n",
    "print (y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Defining out hyperparameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Choosing the right values for your hyperparameters is a crucial part of training deep neural networks effectively. You'll find that your training loss curves can vary with your choice of optimizer (Adam, Adadelta, SGD, etc), learning rate, and network architecture. With RNNs and LSTMs in particular, some other important factors include the number of LSTM units and the size of the word vectors.\n",
    "\n",
    "* Learning Rate: RNNs are infamous for being diffult to train because of the large number of time steps they have. Learning rate becomes extremely important since we don't want our weight values to fluctuate wildly as a result of a large learning rate, nor do we want a slow training process due to a low learning rate. The default value of 0.001 is a good place to start. You should increase this value if the training loss is changing very slowly, and decrease if the loss is unstable.  \n",
    "* Optimizer: There isn't a consensus choice among researchers, but Adam has been widely popular due to having the adaptive learning rate property (Keep in mind that optimal learning rates can differ with the choice of optimizer).\n",
    "* Number of LSTM units: This value is largely dependent on the average length of your input texts. While a greater number of units provides more expressibility for the model and allows the model to store more information for longer texts, the network will take longer to train and will be computationally expensive. \n",
    "* Word Vector Size: Dimensions for word vectors generally range from 50 to 300. A larger size means that the vector is able to encapsulate more information about the word, but you should also expect a more computationally expensive model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "batchSize = 64\n",
    "lstmUnits = 2\n",
    "numClasses = 32\n",
    "iterations = 10\n",
    "numDimensions = 20"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Helper Function to get batches"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def get_batches(x, y, batch_size=100):\n",
    "    \n",
    "    n_batches = len(x)//batch_size\n",
    "    x, y = x[:n_batches*batch_size], y[:n_batches*batch_size]\n",
    "    for ii in range(0, len(x), batch_size):\n",
    "        yield x[ii:ii+batch_size], y[ii:ii+batch_size]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Defining Placeholders for input and labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "tf.reset_default_graph()\n",
    "\n",
    "labels = tf.placeholder(tf.float32, [batchSize, numClasses])\n",
    "input_data = tf.placeholder(tf.int32, [batchSize, maxSeqLength])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "\n",
    "#### Embedding\n",
    "Now we'll add an embedding layer. We need to do this because there are 1900083 words in our vocabulary. It is massively inefficient to one-hot encode our classes here. Instead of one-hot encoding, we can have an embedding layer and use that layer as a lookup table.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "data = tf.Variable(tf.zeros([batchSize, maxSeqLength, numDimensions]),dtype=tf.float32)\n",
    "data = tf.nn.embedding_lookup(wordVectors, input_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Next, we'll create our LSTM cells to use in the recurrent network (TensorFlow documentation). Here we are just defining what the cells look like. This isn't actually building the graph, just defining the type of cells we want in our graph.\n",
    "\n",
    "The tf.nn.rnn_cell.BasicLSTMCell  function takes in an integer for the number of LSTM units that we want. This is one of the hyperparameters that will take some tuning to figure out the optimal value. We’ll then wrap that LSTM cell in a dropout layer to help prevent the network from overfitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "lstmCell = tf.contrib.rnn.BasicLSTMCell(lstmUnits)\n",
    "lstmCell = tf.contrib.rnn.DropoutWrapper(cell=lstmCell, output_keep_prob=0.50)\n",
    "value, _ = tf.nn.dynamic_rnn(lstmCell, data, dtype=tf.float32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The first output of the dynamic RNN function can be thought of as the last hidden state vector. This vector will be reshaped and then multiplied by a final weight matrix and a bias term to obtain the final output values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "weight = tf.Variable(tf.truncated_normal([lstmUnits, numClasses]))\n",
    "bias = tf.Variable(tf.constant(0.1, shape=[numClasses]))\n",
    "value = tf.transpose(value, [1, 0, 2])\n",
    "last = tf.gather(value, int(value.get_shape()[0]) - 1)\n",
    "prediction = (tf.matmul(last, weight) + bias)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Next, we’ll define correct prediction and accuracy metrics to track how the network is doing. The correct prediction formulation works by looking at the index of the maximum value of the 2 output values, and then seeing whether it matches with the training labels. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "correctPred = tf.equal(tf.argmax(prediction,1), tf.argmax(labels,1))\n",
    "accuracy = tf.reduce_mean(tf.cast(correctPred, tf.float32))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Calculating loss and defining optimizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=prediction, labels=labels))\n",
    "optimizer = tf.train.AdamOptimizer(0.001).minimize(loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Training "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loss for 0 epoch: 3.114346504211426\n",
      "Accuracy for 0 epoch: 10.9375\n",
      "CPU times: user 1min 35s, sys: 1min 18s, total: 2min 54s\n",
      "Wall time: 1min 37s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "sess = tf.InteractiveSession()\n",
    "saver = tf.train.Saver()\n",
    "sess.run(tf.global_variables_initializer())\n",
    "\n",
    "for i in range(iterations):\n",
    "    #Next Batch of reviews\n",
    "    for ii, (x, y) in enumerate(get_batches(X_train, y_train, batch_size = batchSize)):\n",
    "        sess.run(optimizer, {input_data: x, labels: y})\n",
    "        #acc,loss = sess.run([accuracy,loss], {input_data: x, labels: y})\n",
    "    print(\"Loss for {} epoch: {}\".format(i, (sess.run(loss, {input_data: x, labels: y}))))    \n",
    "    print(\"Accuracy for {} epoch: {}\".format(i, (sess.run(accuracy, {input_data: x, labels: y})) * 100))\n",
    "    \n",
    "save_path = saver.save(sess, \"models/test.ckpt\", global_step=i)    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "sess = tf.InteractiveSession()\n",
    "saver = tf.train.Saver()\n",
    "saver.restore(sess, tf.train.latest_checkpoint('models'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Testing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy for 0 epoch: 56.00000023841858\n",
      "Accuracy for 1 epoch: 58.99999737739563\n",
      "Accuracy for 2 epoch: 60.00000238418579\n",
      "Accuracy for 3 epoch: 58.99999737739563\n",
      "Accuracy for 4 epoch: 62.99999952316284\n",
      "Accuracy for 5 epoch: 60.00000238418579\n",
      "Accuracy for 6 epoch: 57.999998331069946\n",
      "Accuracy for 7 epoch: 60.00000238418579\n",
      "Accuracy for 8 epoch: 57.999998331069946\n",
      "Accuracy for 9 epoch: 55.000001192092896\n",
      "Accuracy for 10 epoch: 60.00000238418579\n",
      "Accuracy for 11 epoch: 56.00000023841858\n",
      "Accuracy for 12 epoch: 57.999998331069946\n",
      "Accuracy for 13 epoch: 58.99999737739563\n",
      "Accuracy for 14 epoch: 61.000001430511475\n",
      "Accuracy for 15 epoch: 62.00000047683716\n",
      "Accuracy for 16 epoch: 61.000001430511475\n",
      "Accuracy for 17 epoch: 58.99999737739563\n",
      "Accuracy for 18 epoch: 60.00000238418579\n",
      "Accuracy for 19 epoch: 57.999998331069946\n"
     ]
    }
   ],
   "source": [
    "for i in range(iterations):\n",
    "    for ii, (x, y) in enumerate(get_batches(X_test, y_test, batch_size = batchSize)):\n",
    "        if (ii % 30000 ==0):\n",
    "            print (\"Accuracy for {} epoch: {}\".format(i, (sess.run(accuracy, {input_data: x, labels: y})) * 100))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}