1. Proof of Concept - Sentiment Analysis on Airline Tweets.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Mood Detection of Online Tweets\n",
    "\n",
    "**Project Name:** Mood Detection of Online Tweets\n",
    "\n",
    "**Author:** Christopher Goh Zhen Fung\n",
    "- **Email:** chrisgzf@gmail.com\n",
    "- **LinkedIn:** https://www.linkedin.com/in/chrisgzf/\n",
    "- **GitHub:** https://github.com/chrisgzf\n",
    "\n",
    "This project is about sentiment analysis of online tweets. I would like to train a neural network to be able to recognise if tweets (or text in general) sound happy or sad.\n",
    "\n",
    "**Why did I decide to work on this project:**\n",
    "In recent years, there has been an unsettling trend of deteriorating mental health. If you browse social networks like Twitter, Reddit, Tumblr, you will realise that many depressed individuals have been going online to talk about their depression. In these cases, it is very common for other netizens to show concern to these individuals, often suggesting means to get help. With this tool, perhaps social workers can identify these at-risk individuals sooner and provide these individuals with care and companionship before they go over the edge.\n",
    "\n",
    "**Parts to this Project**\n",
    "1. Trying out sentiment analysis on an already available airline twitter sentiment dataset\n",
    "2. Mining & data wrangling of tweets with Twitter Search API\n",
    "3. Sentiment analysis on these \"mood tweets\"\n",
    "\n",
    "**Credits:** I would like to thank Sam & Martin from Red Dragon AI for giving a very comprehensive and well-run deep learning jump start workshop, where I learnt the skills required to work on this project.\n",
    "\n",
    "Please do feel free to point out any mistakes or provide me with any suggestions!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Testing for CUDA-enabled TF with GPU Support\n",
    "\n",
    "**What this means:**\n",
    "Deep Learning requires a lot of matrix calculations and your CPU is not really meant for that sort of task. However, graphics cards (the hardware that your computer uses to make games run well) are able to perform these sort of tasks very well, so we need to check if the Jupyter Notebook has access to your (Nvidia) GPU, in order to train the model much more efficiently."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Default GPU Device: /device:GPU:0\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "if tf.test.gpu_device_name():\n",
    "    print('Default GPU Device: {}'.format(tf.test.gpu_device_name()))\n",
    "else:\n",
    "    print(\"Please install GPU version of TF\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n",
      "2.2.4-tf\n",
      "1.13.1\n"
     ]
    }
   ],
   "source": [
    "print(tf.test.is_built_with_cuda())\n",
    "print(tf.keras.__version__)\n",
    "print(tf.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Trying out the CrowdFlower Airline Twitter Sentiment Dataset\n",
    "\n",
    "Dataset obtained from: https://www.kaggle.com/crowdflower/twitter-airline-sentiment#Tweets.csv\n",
    "\n",
    "I found this dataset on Kaggle and it is pretty well organised. So I thought I would try out my sentiment analysis task on this dataset first, for two reasons:\n",
    "1. As a proof of concept\n",
    "2. When I generate my own \"mood tweets\" dataset, I can reference something that works"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reading the CSV dataset with pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tweet_id</th>\n",
       "      <th>airline_sentiment</th>\n",
       "      <th>airline_sentiment_confidence</th>\n",
       "      <th>negativereason</th>\n",
       "      <th>negativereason_confidence</th>\n",
       "      <th>airline</th>\n",
       "      <th>airline_sentiment_gold</th>\n",
       "      <th>name</th>\n",
       "      <th>negativereason_gold</th>\n",
       "      <th>retweet_count</th>\n",
       "      <th>text</th>\n",
       "      <th>tweet_coord</th>\n",
       "      <th>tweet_created</th>\n",
       "      <th>tweet_location</th>\n",
       "      <th>user_timezone</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>570306133677760513</td>\n",
       "      <td>neutral</td>\n",
       "      <td>1.0000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Virgin America</td>\n",
       "      <td>NaN</td>\n",
       "      <td>cairdin</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>@VirginAmerica What @dhepburn said.</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2015-02-24 11:35:52 -0800</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Eastern Time (US &amp; Canada)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>570301130888122368</td>\n",
       "      <td>positive</td>\n",
       "      <td>0.3486</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>Virgin America</td>\n",
       "      <td>NaN</td>\n",
       "      <td>jnardino</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>@VirginAmerica plus you've added commercials t...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2015-02-24 11:15:59 -0800</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Pacific Time (US &amp; Canada)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>570301083672813571</td>\n",
       "      <td>neutral</td>\n",
       "      <td>0.6837</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Virgin America</td>\n",
       "      <td>NaN</td>\n",
       "      <td>yvonnalynn</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>@VirginAmerica I didn't today... Must mean I n...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2015-02-24 11:15:48 -0800</td>\n",
       "      <td>Lets Play</td>\n",
       "      <td>Central Time (US &amp; Canada)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>570301031407624196</td>\n",
       "      <td>negative</td>\n",
       "      <td>1.0000</td>\n",
       "      <td>Bad Flight</td>\n",
       "      <td>0.7033</td>\n",
       "      <td>Virgin America</td>\n",
       "      <td>NaN</td>\n",
       "      <td>jnardino</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>@VirginAmerica it's really aggressive to blast...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2015-02-24 11:15:36 -0800</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Pacific Time (US &amp; Canada)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>570300817074462722</td>\n",
       "      <td>negative</td>\n",
       "      <td>1.0000</td>\n",
       "      <td>Can't Tell</td>\n",
       "      <td>1.0000</td>\n",
       "      <td>Virgin America</td>\n",
       "      <td>NaN</td>\n",
       "      <td>jnardino</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>@VirginAmerica and it's a really big bad thing...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2015-02-24 11:14:45 -0800</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Pacific Time (US &amp; Canada)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             tweet_id airline_sentiment  airline_sentiment_confidence  \\\n",
       "0  570306133677760513           neutral                        1.0000   \n",
       "1  570301130888122368          positive                        0.3486   \n",
       "2  570301083672813571           neutral                        0.6837   \n",
       "3  570301031407624196          negative                        1.0000   \n",
       "4  570300817074462722          negative                        1.0000   \n",
       "\n",
       "  negativereason  negativereason_confidence         airline  \\\n",
       "0            NaN                        NaN  Virgin America   \n",
       "1            NaN                     0.0000  Virgin America   \n",
       "2            NaN                        NaN  Virgin America   \n",
       "3     Bad Flight                     0.7033  Virgin America   \n",
       "4     Can't Tell                     1.0000  Virgin America   \n",
       "\n",
       "  airline_sentiment_gold        name negativereason_gold  retweet_count  \\\n",
       "0                    NaN     cairdin                 NaN              0   \n",
       "1                    NaN    jnardino                 NaN              0   \n",
       "2                    NaN  yvonnalynn                 NaN              0   \n",
       "3                    NaN    jnardino                 NaN              0   \n",
       "4                    NaN    jnardino                 NaN              0   \n",
       "\n",
       "                                                text tweet_coord  \\\n",
       "0                @VirginAmerica What @dhepburn said.         NaN   \n",
       "1  @VirginAmerica plus you've added commercials t...         NaN   \n",
       "2  @VirginAmerica I didn't today... Must mean I n...         NaN   \n",
       "3  @VirginAmerica it's really aggressive to blast...         NaN   \n",
       "4  @VirginAmerica and it's a really big bad thing...         NaN   \n",
       "\n",
       "               tweet_created tweet_location               user_timezone  \n",
       "0  2015-02-24 11:35:52 -0800            NaN  Eastern Time (US & Canada)  \n",
       "1  2015-02-24 11:15:59 -0800            NaN  Pacific Time (US & Canada)  \n",
       "2  2015-02-24 11:15:48 -0800      Lets Play  Central Time (US & Canada)  \n",
       "3  2015-02-24 11:15:36 -0800            NaN  Pacific Time (US & Canada)  \n",
       "4  2015-02-24 11:14:45 -0800            NaN  Pacific Time (US & Canada)  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv(\"datasets/airline-twitter-sentiment.csv\")\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tweet_id</th>\n",
       "      <th>airline_sentiment_confidence</th>\n",
       "      <th>negativereason_confidence</th>\n",
       "      <th>retweet_count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>1.464000e+04</td>\n",
       "      <td>14640.000000</td>\n",
       "      <td>10522.000000</td>\n",
       "      <td>14640.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>5.692184e+17</td>\n",
       "      <td>0.900169</td>\n",
       "      <td>0.638298</td>\n",
       "      <td>0.082650</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>7.791112e+14</td>\n",
       "      <td>0.162830</td>\n",
       "      <td>0.330440</td>\n",
       "      <td>0.745778</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>5.675883e+17</td>\n",
       "      <td>0.335000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>5.685592e+17</td>\n",
       "      <td>0.692300</td>\n",
       "      <td>0.360600</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>5.694779e+17</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.670600</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>5.698905e+17</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>5.703106e+17</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>44.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           tweet_id  airline_sentiment_confidence  negativereason_confidence  \\\n",
       "count  1.464000e+04                  14640.000000               10522.000000   \n",
       "mean   5.692184e+17                      0.900169                   0.638298   \n",
       "std    7.791112e+14                      0.162830                   0.330440   \n",
       "min    5.675883e+17                      0.335000                   0.000000   \n",
       "25%    5.685592e+17                      0.692300                   0.360600   \n",
       "50%    5.694779e+17                      1.000000                   0.670600   \n",
       "75%    5.698905e+17                      1.000000                   1.000000   \n",
       "max    5.703106e+17                      1.000000                   1.000000   \n",
       "\n",
       "       retweet_count  \n",
       "count   14640.000000  \n",
       "mean        0.082650  \n",
       "std         0.745778  \n",
       "min         0.000000  \n",
       "25%         0.000000  \n",
       "50%         0.000000  \n",
       "75%         0.000000  \n",
       "max        44.000000  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dropping Irrelevant Information\n",
    "I'm only concerned about the sentiment & text. So I will drop the rest of the information"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df[[\"airline_sentiment\", \"text\"]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's analyse how the dataset is lain out and roughly how the data looks like"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>airline_sentiment</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>7319</th>\n",
       "      <td>negative</td>\n",
       "      <td>@JetBlue how do we watch the oscars though right now?! No abc on flight :( #missingtheoscars</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11625</th>\n",
       "      <td>negative</td>\n",
       "      <td>@USAirways I've been sitting in the Charlotte airport for 4 hours. Waited for crew and now maintenance.  #NeverAgain</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6512</th>\n",
       "      <td>negative</td>\n",
       "      <td>@SouthwestAir I do need help Cancelled Flighting my flight back. Your calls times are ridiculous.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11624</th>\n",
       "      <td>negative</td>\n",
       "      <td>@USAirways No, they won't because after 5 hours of holding I had to give up because I couldn't borrow the phone any longer. 5 hours...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6514</th>\n",
       "      <td>negative</td>\n",
       "      <td>@SouthwestAir your customer service is terrible, you're terrible, thought you should know.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      airline_sentiment  \\\n",
       "7319   negative           \n",
       "11625  negative           \n",
       "6512   negative           \n",
       "11624  negative           \n",
       "6514   negative           \n",
       "\n",
       "                                                                                                                                         text  \n",
       "7319   @JetBlue how do we watch the oscars though right now?! No abc on flight :( #missingtheoscars                                            \n",
       "11625  @USAirways I've been sitting in the Charlotte airport for 4 hours. Waited for crew and now maintenance.  #NeverAgain                    \n",
       "6512   @SouthwestAir I do need help Cancelled Flighting my flight back. Your calls times are ridiculous.                                       \n",
       "11624  @USAirways No, they won't because after 5 hours of holding I had to give up because I couldn't borrow the phone any longer. 5 hours...  \n",
       "6514   @SouthwestAir your customer service is terrible, you're terrible, thought you should know.                                              "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>airline_sentiment</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4822</th>\n",
       "      <td>positive</td>\n",
       "      <td>@SouthwestAir kudos to the crew of flight 1050 to GRR for making a very special memory for a sweet young passenger, and her Momma. Well done</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7735</th>\n",
       "      <td>positive</td>\n",
       "      <td>@JetBlue your employee Charles cave at the gate at MSY went above and beyond to help try to help me find my glasses. Thought u should know</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4818</th>\n",
       "      <td>positive</td>\n",
       "      <td>@SouthwestAir Thanks for helping my mom after @allegiantair wouldn't let her get on her plane in Orlando! You're the best! #customerservice</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10895</th>\n",
       "      <td>positive</td>\n",
       "      <td>@USAirways made it!!! Send Bloody Mary's to row 27!!!</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4078</th>\n",
       "      <td>positive</td>\n",
       "      <td>@united thanks for all the help! Totally appreciate it and you made it super easy too</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      airline_sentiment  \\\n",
       "4822   positive           \n",
       "7735   positive           \n",
       "4818   positive           \n",
       "10895  positive           \n",
       "4078   positive           \n",
       "\n",
       "                                                                                                                                               text  \n",
       "4822   @SouthwestAir kudos to the crew of flight 1050 to GRR for making a very special memory for a sweet young passenger, and her Momma. Well done  \n",
       "7735   @JetBlue your employee Charles cave at the gate at MSY went above and beyond to help try to help me find my glasses. Thought u should know    \n",
       "4818   @SouthwestAir Thanks for helping my mom after @allegiantair wouldn't let her get on her plane in Orlando! You're the best! #customerservice   \n",
       "10895  @USAirways made it!!! Send Bloody Mary's to row 27!!!                                                                                         \n",
       "4078   @united thanks for all the help! Totally appreciate it and you made it super easy too                                                         "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pd.set_option('display.max_colwidth', -1) # option to be set so that the tweet's texts won't be truncated\n",
    "df.sort_values(\"airline_sentiment\", inplace=True)\n",
    "display(df.head())\n",
    "display(df.tail())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now change the sentiment into a numerical representation to make it easier for deep learning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>airline_sentiment</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>7319</th>\n",
       "      <td>0</td>\n",
       "      <td>@JetBlue how do we watch the oscars though right now?! No abc on flight :( #missingtheoscars</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11625</th>\n",
       "      <td>0</td>\n",
       "      <td>@USAirways I've been sitting in the Charlotte airport for 4 hours. Waited for crew and now maintenance.  #NeverAgain</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6512</th>\n",
       "      <td>0</td>\n",
       "      <td>@SouthwestAir I do need help Cancelled Flighting my flight back. Your calls times are ridiculous.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11624</th>\n",
       "      <td>0</td>\n",
       "      <td>@USAirways No, they won't because after 5 hours of holding I had to give up because I couldn't borrow the phone any longer. 5 hours...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6514</th>\n",
       "      <td>0</td>\n",
       "      <td>@SouthwestAir your customer service is terrible, you're terrible, thought you should know.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      airline_sentiment  \\\n",
       "7319   0                  \n",
       "11625  0                  \n",
       "6512   0                  \n",
       "11624  0                  \n",
       "6514   0                  \n",
       "\n",
       "                                                                                                                                         text  \n",
       "7319   @JetBlue how do we watch the oscars though right now?! No abc on flight :( #missingtheoscars                                            \n",
       "11625  @USAirways I've been sitting in the Charlotte airport for 4 hours. Waited for crew and now maintenance.  #NeverAgain                    \n",
       "6512   @SouthwestAir I do need help Cancelled Flighting my flight back. Your calls times are ridiculous.                                       \n",
       "11624  @USAirways No, they won't because after 5 hours of holding I had to give up because I couldn't borrow the phone any longer. 5 hours...  \n",
       "6514   @SouthwestAir your customer service is terrible, you're terrible, thought you should know.                                              "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>airline_sentiment</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4822</th>\n",
       "      <td>2</td>\n",
       "      <td>@SouthwestAir kudos to the crew of flight 1050 to GRR for making a very special memory for a sweet young passenger, and her Momma. Well done</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7735</th>\n",
       "      <td>2</td>\n",
       "      <td>@JetBlue your employee Charles cave at the gate at MSY went above and beyond to help try to help me find my glasses. Thought u should know</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4818</th>\n",
       "      <td>2</td>\n",
       "      <td>@SouthwestAir Thanks for helping my mom after @allegiantair wouldn't let her get on her plane in Orlando! You're the best! #customerservice</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10895</th>\n",
       "      <td>2</td>\n",
       "      <td>@USAirways made it!!! Send Bloody Mary's to row 27!!!</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4078</th>\n",
       "      <td>2</td>\n",
       "      <td>@united thanks for all the help! Totally appreciate it and you made it super easy too</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      airline_sentiment  \\\n",
       "4822   2                  \n",
       "7735   2                  \n",
       "4818   2                  \n",
       "10895  2                  \n",
       "4078   2                  \n",
       "\n",
       "                                                                                                                                               text  \n",
       "4822   @SouthwestAir kudos to the crew of flight 1050 to GRR for making a very special memory for a sweet young passenger, and her Momma. Well done  \n",
       "7735   @JetBlue your employee Charles cave at the gate at MSY went above and beyond to help try to help me find my glasses. Thought u should know    \n",
       "4818   @SouthwestAir Thanks for helping my mom after @allegiantair wouldn't let her get on her plane in Orlando! You're the best! #customerservice   \n",
       "10895  @USAirways made it!!! Send Bloody Mary's to row 27!!!                                                                                         \n",
       "4078   @united thanks for all the help! Totally appreciate it and you made it super easy too                                                         "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>airline_sentiment</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>14640</td>\n",
       "      <td>14640</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unique</th>\n",
       "      <td>3</td>\n",
       "      <td>14427</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>top</th>\n",
       "      <td>0</td>\n",
       "      <td>@united thanks</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>freq</th>\n",
       "      <td>9178</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        airline_sentiment            text\n",
       "count   14640              14640         \n",
       "unique  3                  14427         \n",
       "top     0                  @united thanks\n",
       "freq    9178               6             "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "df.loc[(df[\"airline_sentiment\"] == \"negative\"), \"airline_sentiment\"] = 0 # replacing all \"negative\" with 0\n",
    "df.loc[(df[\"airline_sentiment\"] == \"neutral\"), \"airline_sentiment\"] = 1 # \"neutral\" with 1\n",
    "df.loc[(df[\"airline_sentiment\"] == \"positive\"), \"airline_sentiment\"] = 2 # \"positive\" with 2\n",
    "\n",
    "display(df.head())\n",
    "display(df.tail())\n",
    "display(df.describe())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading Glove word embeddings.\n",
    "\n",
    "Right now, we have our tweets in text form. In order for our neural network to take them in as inputs, they have to be numerical. Therefore, we will be performing 2 tasks:\n",
    "1. Tokenising the text\n",
    "2. Adding a second layer in the neural network to map the text to their respective word embeddings\n",
    "\n",
    "Word embeddings are vector representations of distinct words in a vector space of many dimensions (in this case we are using 100 dimensions). This gives every word a distinct vector, and we can perform some very interesting vector operations on them.\n",
    "\n",
    "A lot of the code here is referenced from: https://github.com/keras-team/keras/blob/master/examples/pretrained_word_embeddings.py\n",
    "\n",
    "Note: Please run the 2 cells below in a Linux environment / Google Colab so that the commands work properly. Windows might not have the proper binaries to perform these commands."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget -qq http://nlp.stanford.edu/data/glove.6B.zip\n",
    "\n",
    "!unzip glove.6B.zip"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Glove word embeddings should be downloaded and unzipped. Check the output of the cell below to see if they are reflected correctly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " datasets\t     glove.6B.300d.txt\t LICENSE\r\n",
      " glove.6B.100d.txt   glove.6B.50d.txt\t README.md\r\n",
      " glove.6B.200d.txt   glove.6B.zip\t'Sentiment Analysis of Tweets.ipynb'\r\n"
     ]
    }
   ],
   "source": [
    "!ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a new dictionary to map the words in the embeddings to their 100-dimension embedding vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 400000 word vectors.\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "# NumPy is a very widely-used library in Python for mathematical operations.\n",
    "# It supports many complex mathematical functions and can work on large arrays and matrices.\n",
    "\n",
    "embeddings_index = {}\n",
    "with open(os.path.join('glove.6B.100d.txt')) as f:\n",
    "    for line in f:\n",
    "        values = line.split()\n",
    "        word = values[0]\n",
    "        coefs = np.asarray(values[1:], dtype='float32')\n",
    "        embeddings_index[word] = coefs\n",
    "\n",
    "print(\"Found {} word vectors.\".format(len(embeddings_index)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.37058 , -0.67807 ,  0.68365 , -1.8257  ,  0.033376, -0.19271 ,\n",
       "        0.23817 ,  0.20734 , -0.59292 , -0.68866 ,  0.056996, -0.45526 ,\n",
       "        0.050337,  0.38388 ,  0.5716  , -0.58939 , -0.065345, -0.15682 ,\n",
       "       -0.71076 ,  0.80948 ,  0.28472 ,  0.33693 ,  0.16452 , -0.10708 ,\n",
       "        0.93074 ,  0.63066 ,  0.31136 , -0.47565 ,  0.26755 ,  0.43212 ,\n",
       "       -0.54809 ,  0.45665 , -0.13095 ,  0.41672 ,  0.07025 , -1.0302  ,\n",
       "       -0.21576 ,  0.34737 , -0.52304 ,  0.43099 , -0.39802 , -0.25244 ,\n",
       "        0.39093 ,  0.58856 , -1.1839  ,  0.31923 , -1.0152  , -0.32431 ,\n",
       "       -0.099747, -0.12774 ,  0.20104 ,  0.16763 , -0.59854 , -0.25134 ,\n",
       "        0.49871 , -2.4691  , -0.88555 , -0.14335 , -0.037509,  0.87851 ,\n",
       "        0.086941, -0.097006, -0.59662 ,  0.63888 ,  0.72839 , -0.87794 ,\n",
       "        0.71174 ,  1.1118  , -0.26572 ,  0.3617  , -0.43472 ,  0.17808 ,\n",
       "       -0.57865 , -0.20282 ,  0.3917  , -0.37085 , -0.37463 ,  0.024523,\n",
       "        0.017991, -0.43885 ,  0.093244, -0.019675,  0.20245 , -0.0862  ,\n",
       "       -0.30843 , -0.20882 , -0.15978 , -0.37059 , -0.3803  , -0.33614 ,\n",
       "        0.24756 , -0.06655 , -0.015687, -0.18252 , -0.70065 ,  0.29659 ,\n",
       "        0.077268, -0.016513, -0.027181, -0.13    ], dtype=float32)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# to see what word embeddings look like\n",
    "embeddings_index[\"chris\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This array of numbers make up the representation of the word \"chris\" in 100d-space. This is beneficial as it gives every trained word a distinct vector representation, and we can calculate the cosine similarity between 2 different words to determine how similar / different the words are, in terms of contextual information."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Processing them for TensorFlow\n",
    "\n",
    "TensorFlow (& Keras) is the library/framework which we will use to train our neural network.\n",
    "\n",
    "We will process our dataframe above into TF-ready numpy arrays."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "\n",
    "from tensorflow.keras.preprocessing.text import Tokenizer\n",
    "from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
    "from tensorflow.keras.utils import to_categorical\n",
    "from tensorflow.keras.layers import Dense, Input, LSTM, Embedding\n",
    "from tensorflow.keras.models import Model\n",
    "from tensorflow.keras.initializers import Constant"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Setting some parameters\n",
    "\n",
    "MAX_SEQUENCE_LENGTH = 40 # determines how many words of every tweet will be processed\n",
    "# if the tweet length is smaller than this, the tweet will be padded until it has a length equivalent to this\n",
    "\n",
    "MAX_NUM_WORDS = 20000 # how many unique words in the training text to be tokenised\n",
    "EMBEDDING_DIM = 100 # number of dimensions in the word embeddings\n",
    "TRAINING_SPLIT = 0.8 # proportion of data to be used to training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Converting the dataframe into lists\n",
    "\n",
    "texts = df[\"text\"].tolist()\n",
    "labels = df[\"airline_sentiment\"].tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokenising the Tweets into a 2D integer tensor\n",
    "\n",
    "Tokenising tweets will break up the text into individual words, and replace them with a numeric representation.\n",
    "\n",
    "e.g. \n",
    "- \"I am a human\" > \"i\", \"am\", \"a\", \"human\" > 1, 2, 3, 4\n",
    "- \"I am a robot\" > \"i\", \"am\", \"a\", \"robot\" > 1, 2, 3, 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "15768 distinct words in original text\n"
     ]
    }
   ],
   "source": [
    "tokenizer = Tokenizer(num_words=MAX_NUM_WORDS)\n",
    "tokenizer.fit_on_texts(texts)\n",
    "sequences = tokenizer.texts_to_sequences(texts)\n",
    "word_index = tokenizer.word_index\n",
    "\n",
    "print(\"{} distinct words in original text\".format(len(word_index)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All our tensors have to be of the same length, so we need to set a maximum length, and if a certain tweet has fewer than the maximum length of words, it will be padded until it is the same size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "36"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD8CAYAAABn919SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl4FFX6NuDnJQu7ECDsYAARBGQNi4IogoriuCs6fMqMOuiIzug4oyg/HRxnRlAHFHVUVBQZXFBUFHBhU1YDCRAIW8ISQiAkYUsIgZDlfH90dae7U52u7q7eiue+rlzprq7l7dPVb1Wfc+qUKKVARETRr064AyAiInMwoRMRWQQTOhGRRTChExFZBBM6EZFFMKETEVkEEzoRkUUwoRMRWQQTOhGRRcSGcmMtWrRQSUlJodwkEVHUS0tLO6qUSvQ2X0gTelJSElJTU0O5SSKiqCciB4zMxyoXIiKLYEInIrIIJnQiIotgQicisggmdCIii2BCJyKyCCZ0IiKLYEInj7bmnsTW3JPhDoOIDArphUUUXW56cy0AIHvqmDBHQkRG8AydiMgivCZ0EaknIhtEJF1EtovIC9r0j0Rkv4hs0f76Bj9cIiLyxEiVSxmAq5VSJSISB2CNiHyvvfY3pdSXwQuPiIiM8prQlVIKQIn2NE77U8EMioiIfGeoDl1EYkRkC4ACAEuVUinaS/8Ska0iMkNE6gYtSiIi8spQQldKVSql+gJoD2CQiPQC8AyA7gAGAmgG4Gm9ZUVkgoikikhqYWGhSWETEZE7n3q5KKVOAvgZwGilVJ6yKQPwIYBBHpaZpZRKVkolJyZ6HZ+diIj8ZKSXS6KINNUe1wcwCsAuEWmjTRMAtwDICGagRERUOyO9XNoAmCMiMbAdAOYrpRaJyAoRSQQgALYAeDiIcRIRkRdGerlsBdBPZ/rVQYmIiIj8witFiYgsggmdiMgimNCJiCyCCZ2IyCKY0ImILIIJnYjIIpjQiaLQyt0FSJq0GPsKS7zPfB4oPFWGpEmLsSAtN9yhhBUTOlEU+i79MABgUw5vEQjAcWD7fOPBMEcSXkzoREQWwYRORGQRTOhERBZhqYReVaWQe6LU8PznKqpwpOhsECMKzLGSMpwuqwhoHYdOnkFVlfEbTJ0uq8CxkjLD85+rqELageOw3djKPIdPnkF5ZZXLtNwTpT69FwA4eNz4/hBOh0+eQYXb+zWiorIKZRWVSDtwwvTPIFAHj/v+eQUrjvOFpRL6Gyv2YNi0lcg+etrQ/E9+kY4hLy2vkTgixYB/LsP1r6/2e/mcY6UYOnUFXl+eZXiZ619fjQH/XGZ4/lveWovb316PN1fs8SdEXSdLz+HyqSvwwnfbHdP2FJRg2LSVePuXvYbXszqrEFe8vBLfag2IkepoSRkun7oC/16yy/AypWWVAIDJ32Rg1PRfcPvb6/D+6v3BCtFnWfmncMXLKzFr9b6wxrFwyyFc8fJKrMk6GtY4QsVSCX3tXtuHdqTY2Fn3T9uPAAAqI+AswpOcAM4u7OWwbq/xndnX7e3IKwYArNlj3hfm1Fnbr5Kfd1ff4erQyTMAgF/3HTO8nl15pwAAWw9Gdk+Qk6XnAAC/ZBYYXub0OVsZVVYpHDxuK5uU/cfND85PB7Vfyik+fF7BkH6wCACw60hxWOMIFUsldLsI++VJREHGr7yNpRK6hDsAInIR8kR7nicBSyV0O8XjtSXwl1b0kvM9s4ZJVCR0pRR+2n7EY1332j1HUXy2HKLtQ59tOIgNEVSfGKgVu/Jdnm/OOYE567IBAOv2HEXRmXKkZh9H4Slb75SDx0uRcajIMf/G7BPIKzqDVZm2OumqKoUftx8x3CtCKYW/L8xA0ZlyLN+Zj3MVNRuRxYfv78+7C3DmnK1R73RZBX7JLHR5vayisub6ja9eV1WVbR/y9J697WO+qNTZ1q4jxdhvsLHervRczbLxh3N5A7YG5qz8UwGvN+dYKbYftu1nv2QW4uvNuVBKYemOfGzItn3/svJLsDfChyfwtm9Ek6hI6D9kHMGEuWmYtapmi/mxkjKMez8FE+dtckz7Nv0w7np3fShDNJ1zQ+b9H6ViU84Jx/Nb/7sOf/92O7YfLsJv30/BH+ak4o531uOmN9cAAK54eSVufGONy/oue2kF7pu9AQePl+LTjTl4aG4a5qcau0z6gzX7MWf9AfR54Sc8MCcVL/9gvDeGuz0FJfjdhxvxzFdbAQBPLdiK8bM34MCx6mT3929tvVvsDaFm+HBdNibMTcPCLfo9Xn7cno8Jc9Pwjg+9aDx5f/U+TJibhsXb8hzTRr+2GiNe/dmn9Ty9YBvGz97g84HAmb28n/16m2PaqOm/4JoZq/xep93wV1ZizEzbfjZ+9gY88Xk6Fm45jD98nIq3f7aV46GTZzDyP78EvK1g8rZvRJOoSOiFWr/owzpf8DLtbHFPQYmlfuadOF3u8rzoTHmNeYpKbdMyC2xnW3kG+tSfKa909L3PLzbW39y954teTxijJzclWr96e5LaW2A7eyt1OoPMPmpuv2EFIE/bdwpO6ZeRfR8z4yBi30+PnjLen1+PvWwCuRbh1FnbPrIvgIOCLw4XmXcQDhX7vlEY4OcVCbwmdBGpJyIbRCRdRLaLyAva9E4ikiIiWSLyuYjEBytIo8nCl5/9luDH+7XAr0rDnPcH+2NP798+a3SXT1QHTyYwcoZeBuBqpVQfAH0BjBaRIQCmAZihlOoK4ASAB4IXps15l7C9sNIvkmATbefxlPKq963QJ8XoPohQJPGa0JWNvVUjTvtTAK4G8KU2fQ6AW4ISoRfO3wUrJfxI6qljJOH4Wvb2VYYqmXk7A7cfHMOZXCXgHdhCXwAf8aBoY6gOXURiRGQLgAIASwHsBXBSKWWv3MsF0C44IVZf8VhSVoFR039xaSC0yys6i7V79K9KO3OuEn/7Ih13vbMe7/6yF5dO+RFJkxa7pMz0gyfxwnfbHeNMG1VVpfDG8ixHHffB46X4cO1+nDh9Dm+t3AOlFL7enOvS6wQAlu3Ix/ur9+HZr7fh6ld/xvSlmRg/ewPGvrseq7MKkbLPey8de6t8yVnjday1HSiceyOszqruXTH31wPG1q0Uxr3/K75My8WZc5VImrQYD3y0EYu2VpepPeWcOVeJ15ZlorzK1gZSVlGFIf9ejqe+TK+1Htv+xf1w7X50eXZJjStHP9uQgynfbsfewhJHI1dq9nF8pPUK2n2kGO/8srdGj4bvM/Jc1v/5xhxkaj1BVmUW4ufdBUjNPo6x767H+6v3YWP2cVRUVuH1ZVmOdgG7Oett5bVoax6mfr8LC7cccrz2/MIMTPthF5RSeG/VPox+bRU+3VDdOF16rgKvLcv0a1wX918XJWUVmOnDsA/uyioqcf3rq132hUAt3ZGP9XvNv3rU3s7gaeyY1T5c+r+3sATzUozt8/Z9I1LEGplJKVUJoK+INAXwNYBL9GbTW1ZEJgCYAAAdO3b0K0h7i/lXm2xfjNv+uw7ZU8cYXn7miix8od3JxN6dCoBL97ub31oLAPhwbTZ+06et4XWv2FWA/yzNxP5jpzH9rr4Y934Kco6XYumOfKzbewz9OjTFE5+nA4BLzA9+nOoao9MXL+WDDYa2vURLQhVOO/FRHwbWcneD07gx9xqMwVlWQQnW7jmGtXuOOQ4Oy3cVYPmuAtzYu22NeV9bVv2e3/l5L44Un8X8VP07zrifvL7w3Q4AwN2zfnUp10lf2XpzzE896GhoTc+tPph+s+UwsOUwrujaAj3bNnFMt3/h7Qe8pxfY1pM9dQzum+1aFvZL7F+/uy9mLMvE0ZIyvHhLrxoxpx44gdQDricfH2vJfmxyB/xryU4AwK4j1V0IZyzNxHur96Ntk/q65eCLad/vwsrd/ifjuesPYGdeMe79YINP37fa/EHb781an92rP+0GgBrlbVdb90/3xHXjzDU4U16JcYMv9Lpd+75h9vvxl0+9XJRSJwH8DGAIgKYiYj8gtAege2qrlJqllEpWSiUnJiYGEqvfnPvgmu2cdiZl30ax1qvAPh5JuVnjxOispqy85llcIKPblen0L/dFRWX1tn0t8zPltc/v60/qUi/br/LwVn3Zjr28vMWup9LDhuxxl5kwYJx9vBd/nfXjfYWLt8/bCPtJgz+fZ6Qw0sslUTszh4jUBzAKwE4AKwHcoc02HsDCYAVZGyMXA1ipbj1QwaprDLSB1mhYkfhZRk79bQQWDoWUkSqXNgDmiEgMbAeA+UqpRSKyA8BnIvJPAJsBfBDEOANyXvUGCeNbNZJsA03IkZM8IzF9hr9wIunzOR95TehKqa0A+ulM3wdgUDCCiiYh24FNyh5mxBuMt+zt7YXqzDzKPk4iF4YaRSPRfbM3YGvuSZwsrXkFpbPjp89h9lr/Bv4vr6xC18nf4/6hndCobgxmOt3EoU2TesgrOosZY/sAAL7POII//i/NUYe9TevVUuWUQZMmLcZnE4ZgSOfmPseyclcBmjWIdzTeAnA09Dob9K/lta7nhpnVDZ/Tl2Zi+tJMAMC4wcYbrJfuyMdHa/fjd0M7Oaat33fM5WYcqQdce+kkTVqMm/u2xdZc194+dp4araZ8ux1lFVX4dEMOANuViEmTFrvM88i8NCzZdgTfPTrM8Hu48911OFtehcGdmjk+KwD4Mi0X/To2dVm3J3/70jZ8wVebc7FgUy7euy8ZP2QcMbT9q3Uuh99TUII92tWhz32T4Zj+1y/ScX2vNnhv9T4kNIzD6qeu1u21sWxnAZImLcb1vVqj+Gy5x15fdvM3HsRdAzu4TKuqUpgwNw2pB45jSKfq/fSxTzdj5t19dbtWXjrlR8fjpTvya7zur7QDJ3D72+vw6IiL8NfruuGDNfvxRepBDOncHFNu6gkAeH1ZFpo1jHPpvZU0aTFeuaM37kzu4GnVDn/6dLMpN0B56st0XNujNUb1aBXwugIRFZf+61mVWeg1mQPAJwa7H+mxX54+e+1+l2QOVF9mn3GoeuD87zOOoNitC6F7l8K7Z/3qVywfrz+AP3+22a9ljZiXkuPT/FO0XiaeOJeL3cIth30el+SjddmOZA4A+wprLr9kmy2JPvnFFsPrPas1KKfsP16jQW3y19XJ1L7u2tiP2X/4OBULNun30gnEriOnMGNZJkrKKhw3s6jN9xlHaiZznZ9mTy3YWmPa0ZIyLNuZj5Ol5fhhe/V7/y79MDy1t59y2se3mHgzkYf/ZzuYvrnS9t17cdEO7DpyytEFFQBmLMvEcwu313h79oOtN2bdzWp+am6NnmvhELUJnYiIXFk+oQd+9V3tvNVJm7l5tjfVjg1ytTC4I7IIo5vlEzoRRSc2HPuOCb0WZpzxhbPLpBUG7Cci46K2l4sR7r0hPOn+3A9+LQfA600i9HrY+LJ+ZweO+TZO+Fg/G2CN8vd9BEtWQWTfGccsXxi8MYm7K19Z6XUfqu0coMuzS/DuvQPw0FzPPX88cd9XCk+VYeC/ljmePz26O6b5cOOUpEmL0SA+xut8zt+/37yxBt89ZusJtSnnBG7777oa8y9w6jl2+OQZtG1aPQRDVv4pXDNjFb57dBgubd+kxrL2uPa/dEPQq3o94Rl6gNwHZnKX5mFsCSJ/2cct8ZWvJwR6/r5we8DrAICMw67dV31J5na+Xu7v3D31602HdOd5Y0X1+EIbs1273i7baRuEa9G22nvGhCuZA0zoRFHHjPueeuJt2OYqVuNFNCZ0oihTEcSE7k2kpnMeZ2yY0ImiTDDP0L1hQzsi96gGizeKElnRKR9uaOJYRucm44AtQZ8tr8LRkjLEx9bxeqZ7tOScz9v2tF1flAY4FLBd7olSNKobi7KKmvXv2cdOu5RtRaVy3LjmXEWVY5lTZRUoKi1Ho3qe02dRaTnqx8egvLIK9eJikFd0Bu0TGpjyHmrDhE50HtjnYciFP322xeUuXT8+Pjwk8by0xLdG0B7P/1jr63p3udK709KwaSs9ruN/v7oOf/HkF+m6832SkoNPUnIw1sNYMRuzj+POd9Y7nic0iMOJ0nJ8PmEIBvsxjpMvWOVCdB5zv+VikYczebOFootpsHuYeRq3x327J7Qxp9JzzRvnxhMmdCJyYB15dGNCJyKyCCZ0InKw0vl5pP3YCEU8TOhEZEn+3tjGKE/XA8xd7/89GALFhE5EDqlul7tHM3+6d5pBr8dNqDChE5HDkeKz4Q6BAsCETkQUAqGo0vea0EWkg4isFJGdIrJdRP6sTZ8iIodEZIv2d0PwwyUiIk+MXClaAeBJpdQmEWkMIE1ElmqvzVBKvRq88IgolCKtZwj5xmtCV0rlAcjTHp8SkZ0A2gU7MCIKvXkpOd5nIr+EYpR0n+rQRSQJQD8AKdqkR0Vkq4jMFpEED8tMEJFUEUktLKw5tgIREZnDcEIXkUYAFgB4XClVDOBtAF0A9IXtDP4/essppWYppZKVUsmJiYkmhExERHoMJXQRiYMtmc9TSn0FAEqpfKVUpVKqCsB7AAYFL0wiIvLGSC8XAfABgJ1KqelO09s4zXYrgAzzwyMiIqOM9HIZCuBeANtEZIs27VkA94hIX9i6V2YDeCgoERIRWUAoOhAZ6eWyBvoNtEvMD4eIiPzFK0WJiCyCCZ2IKAQ4fC4RkUWoENSiM6ETEVkEEzoRUQhICC7+Z0InIgoBCcFgLkzoREQhwEZRIiIyjAmdiMgimNCJiCyCCZ2IKATYD52IiAxjQicisggmdCIii2BCJyIKAfZDJyIiw5jQiYhCgJf+ExGRYUzoREQhwDp0IiIyzGtCF5EOIrJSRHaKyHYR+bM2vZmILBWRLO1/QvDDJSIiT4ycoVcAeFIpdQmAIQAmikgPAJMALFdKdQWwXHtORERh4jWhK6XylFKbtMenAOwE0A7AzQDmaLPNAXBLsIIkIiLvfKpDF5EkAP0ApABopZTKA2xJH0BLD8tMEJFUEUktLCwMLFoioiilQtAqajihi0gjAAsAPK6UKja6nFJqllIqWSmVnJiY6E+MRERkgKGELiJxsCXzeUqpr7TJ+SLSRnu9DYCC4IRIRERGGOnlIgA+ALBTKTXd6aVvAYzXHo8HsND88IiIyKhYA/MMBXAvgG0iskWb9iyAqQDmi8gDAHIA3BmcEImIyAivCV0ptQaAp1EIRpobDhER+YtXihIRWQQTOhGRRTChExGFAAfnIiIiw5jQiYgsggmdiMgimNCJiEIgBFXoTOhERFbBhE5EZBFM6EREFsGETkQUAuyHTkREhjGhExFZBBM6EZFFMKETEYWACkFPdCZ0IiKLYEInIrIIJnQiIotgQiciCoFL2zUJ+jaY0ImIQiChYXzQt+E1oYvIbBEpEJEMp2lTROSQiGzR/m4IbphERNFNQrANI2foHwEYrTN9hlKqr/a3xNywiIjIV14TulJqFYDjIYiFiIgCEEgd+qMislWrkknwNJOITBCRVBFJLSwsDGBzRERUG38T+tsAugDoCyAPwH88zaiUmqWUSlZKJScmJvq5OSIi8savhK6UyldKVSqlqgC8B2CQuWEREZGv/EroItLG6emtADI8zUtERKER620GEfkUwFUAWohILoC/A7hKRPrCdt/TbAAPBTFGIqKoF4qbRHtN6Eqpe3QmfxCEWIiIKAC8UpSIKAQKisuCvg0mdCKiEFAhuKkoEzoRUQhICK79Z0InIrIIJnQiIotgQicisggmdCKikAh+JToTOhGRRTChExFZBBM6EZFFMKETEVkEEzoRkUVERUL/7tFh4Q6BiCggvFJU07pJvXCHQEQU8aIioRMRkXdM6EREIRCCGhcmdCKiUAjFHYuiIqHXi4uKMImIPIqL4aX/AIDG9eLCHQIRUUDqxcUEfRtRkdCJiMg7rwldRGaLSIGIZDhNayYiS0UkS/ufENwwiYiiXAgq0Y2coX8EYLTbtEkAliulugJYrj0nIqIw8prQlVKrABx3m3wzgDna4zkAbjE5LiIi8pG/deitlFJ5AKD9b+lpRhGZICKpIpJaWFjo5+aIiKJbem5R0LcR9EZRpdQspVSyUio5MTEx2JsjIopI+cVng74NfxN6voi0AQDtf4F5IRERkT/8TejfAhivPR4PYKE54RARkb+MdFv8FMB6AN1EJFdEHgAwFcA1IpIF4BrtORERhVGstxmUUvd4eGmkybEQEVmWUsHviM4rRYmIQoCDc0WQPh2aep1nw2T+aCGi8GFCJyKyCCZ0IqIQCEEVOhO6mSQk9yQhomikQlCLzoRORGQRTOhUw7TbL8Xt/duHOwwiS7muZ+ugb4MJnWoYO7Aj/nNXH4zoFjlj73RObGh43p5tLzA8b5P6+nfDGtO7jeF1EBlRn3csItL4UP0obMqg8xQTulGhaKImIgoAEzp5FK2HMDOOvTzJp2jEhG6Ugd/xdSyWBWKj9A3VjTW+W3saXyMU9Z1EZmNCN1HzRnW9zpN8YeD3027XtH7A6zAioUF8wOtoGG9SYtQ5toxN7qA768192wW8ud8P7eTxtcGdmgW8fqJgsHRCv65nK5/mj48xVhz3DOro8bUuXnpjfPnHyz2+1qONsd4ZA5MCPygEQ4tGNQ8A2/8xGgsnDq11uX/e0svnbf3u8iRMu6O37mt1TPhlERfjeR1mNLq2aVIv8JUQubF0QveZwS9quHtRSLgDIKKIxIQehUKVz4PVKOpX/L50W/Rj9URWYOmEHqyxVYKVMIzmLKuNGROtvWmIIo21E7qPee/Sdk0C3mb31savUvRXMM/Qjdbj67kkgGXDpYcPV5WapWmDOFManIncRU1C79isAQBgzv2DXKYvemwYfnx8uO4ytfVHHnpR8xrTPvz9QMx/6DK88//647Wxff2K85U7e+Oj3w/EJ38Y7JjWv6P3m2MY8dyNPbDsL1fqnp8bvTT+N33a4sFhnbDosWGY8psejumfTRiCeQ8OdonbF188fBn+O66/1/mWPqH/WdXmh8ev8Cckhz7t9Q/U/3tgMGbdl4wL6nm9E6NPHhzm2kOm9QWuDaBLn7jSlJMHInfm7slBdEF9W6gJDVzH3ujVrglOnD6nu4yn4SrbNa2vO4bHBfXiMMipS9rjn2/xOc4G8bG4qltLl2kjL2mFTTknfV6Xu3GDO6JeXIzuGXodg6ftb9zTz/G4V7smmPLdDgDAkM41D3C+GJhkrCtf58RGEPhWzaL3q8eXXym92zdFem5RjenDurYAALRtWh/FR075EFHtEhq6nn2PvKQl5qXkOJ4nNvbevZXIHwEldBHJBnAKQCWACqVUshlB1bpNnfPTUHf6CHedr14ZBOMGtCEb7eA8HFaBHZUoGMw4Qx+hlDpqwnpMZ7XGQzu9ZBAtKdHfTySY7y/Y3UCj5bOh6Bc1dei1scoJntGz7Gjvhx7t8ZvBKvssRZZAE7oC8JOIpInIBL0ZRGSCiKSKSGphYaHfGxqh1UsnNq6L5g0D6yFwTQ/friB1F4x0ZOSq1hjtCki9fDjaz8Hzw3HForeraY0wWmcPAMl+XFlb2zFn+MW1jxPv3ggajb1/KDoFmtCHKqX6A7gewEQRqdGFQSk1SymVrJRKTkz0/4YJT4y6GBueHYnWTeph9dMjPM635ukReOzqi2zb1vmxu2HySPzfmEv8jsMMW56/psa0N+7R7yHSTDt4fTNxKOK0oQnsuebp0d3x0xPDsXHyKPz12m61bnPeg4N1t7v8ySuR/vy1ussYPZFO+79RxmbU9Gzrfw+Pn54YjkWPDcMNl+rfgKKtzgHq5r7t8OszI3GVdsOO1+/ui83PVZeF/W3OfWAQOjTzPk7Ow8O7OB6P6JaIlGdHOp4v+8twtE9wXcfgTs3w6zMjQRRsASV0pdRh7X8BgK8BDKp9Cf/VqSNoqZ35NIjXr/pv2iAO7RMa1HoG37JxPcQaHLPFk0BrDJrq9EGO9zBCoH1YEuf3ZN9+w7oxuLhVYyQ2rut1/JJ6cTG6220QH4smDfTv2mO0WsDIoGRmia0j6OVHl7/WTeo5xuqpGxvj0hPFXp4JDeINjefjXNYiglZOZ+QXtWysW6XU2u1Aw1onCga/M5uINBSRxvbHAK4FkGFWYOSZvYsi62FrYv08nc8C6eXSCsDX2hcoFsAnSqkfTImKHPSStj1lVVkko1vjXRCFn98JXSm1D0AfE2MxXaTkO7P7iEsUn6H7ewLtSxl634b+uqKxPImcWaLbYsO6tuPSxKtsjaFDL7JdAfjbwZ7HLb9zgP7NETwZf9mFjse39mtf4/U/XtWlxjRnN1za2usl5ve7XTIOABOGdwYANHcaa9x+R3r7lY7uhl+c6LgS9slrLgYAdGoRWM+S2/rr3zTC/YYdRi6jv7Wf67q6tmzscV73S+R9rVJxbiT1tKin6X071Byy4U8ju7oso3egsffisQ8vYa8/v2dQ9T53U9+2tQdO5IeoufS/NvGxdZA9dYzjeddWjZE9dQwqq2xftjoCVLl970Z0b4kd/7gOPZ7/0ev67ev+y/x0AMAApyT2vwcGe0ysE0d0wVsr9wIA/jtugNft3JXcAXdpd+Hp/tz3OFtehfsuS8JDV7oeLAYmNXN5v+4+dhvv5jEtCfnK3kvolTt6487kDph+l218m6RJiwFAN4YJwzvj1Z8ya13v8IsTXZbdW1iiO98dA9rj1TuN/Qhc/dQIXPHySlvc2md9z6COeOm2Sw0tr+ebiUOx/XARxsxcA8D1/X4wPhn3f5Squ1zzRnV1y+al23rjpdtsN+W4vEsLZE8d4yhLd/blPb3uSdMGcThZWu7TMu5evr03nlqw1ePrt/Vvh682HQpoG85G92yNH7YfMW195zNLnKF7EozL4d2dD21w0drQGKVhn3f4OZnH0gndzlNCYp2p9fiTHOxDROhdt8B9hKLJeZHQPeF3lQDn+vDwxmGGaDzZ5Rm6ec7rhE7WoZeMrZCgfXUevmVyErWNoi/c1BPbDtUc49pZTB3B5V2a4/6hnbAw/TCGud3UokFcDIZ0boZf9x3XXf7FW3phc84Jx/OXb++NVVm28WjuH9oJs9fuR79abl4xbvCFWLojH3cm1+xRM2NsH/yQcQQN42PR78KaY428PW4A3l21F3Wu5ZItAAAKI0lEQVQ9XEGq55GruhgeF92IiSMuwvZDxRh1iev47ld1S/TYS+iu5A74atMh7Dt62vB2OiQ0qDGtW6vGeMSp59C023tj6g+70K6p62X1Yy5tgyu7VQ8pIeL5jO/Ja7vhwLFSXH6RfiM2YGu4/PeSnejYrPZeQYM6NcclbS7A367rDsDWi2mE2zj4Rtw9sAO6tmqMFxftcEy7qY//PWDM+vRbNIrH0RL9+ww8OuIi7DhcjMoqhawC/QZtX9SLjQl4HZGmf8emNe6BEMiQF0ZJKBoO7ZKTk1Vqqn7PgHCqrdcG+ce5TNMPnsTNb61F7/ZN8O2jwwwv54ucY6UY/spKdGhWHw9f2QWTv84w3MvlpjfXYGtuEb6ZOLRGV8WMQ0W48Y2avVxCxWgvl4QGcThRWo4WjeriaEkZ/jSyK2Yuz3KZp7ZeNc5evbMP7hjQ3mX7nt77Va+sRPaxUkMxOuvR5gLcP6wT/vpFOm7r3w7T7+rrsq0H56Ri2c58l2V2/mM0Lnm+9msXVz81Au0T6qPTM0sA2O5C9vsPN/ocn55rerTC0h353meE7T0Uny1H7yk/uUzzl4ikGbnfBKtcyBI83Z3KCCtU4dq76MZ6GdPHiACHOrKsaKjC40dHlmLVm5p4Y0/oMSYkdDOr7cxk5KAd3KQb+RmdCZ3IAirOg4Qebu4XJ0YiJnQKOnuS8aWB11f2JFQvrg7i6ti2Ex9jLDHVjbM1yunNbUaCDLaYOoL68bb3YB8GI5CyjvOhzqVenH8NmnXj6jiqh/SGLK4bZ86+EmPiwcnI0MrOwrHnRG0vFzMtemwY0g6c8D4jGTZjbB/HOOE9216AP43sit8O8jy2jt1zN/bA5V2ae53PXfuE+vjLNRfj1n7t0OqCeth7tASPjrjI0LIz7+6HT1IOoHf7mr0QurdujIFJCRh1SWB3ufLXd48Owx3vrMPwixPRrml9jBvcEYu25uH15VkY2b0lBiQlYGT3VoipA6zYVYAbe7fF/NSD+OOVXTB9aSYGd2qGOwa0dxyYFj02DKuzjuLn3QXYnHMSS/58BUa/tgrdWjdGRaXCwROlLnf0evGWXuijUy52792XjMnfZKB768aYtWofera9AHlFZ/HXa7uh+Gw5ftldiLjYOvh17zGcq6zC7N8lI/1gEcYO7IDExnWx80gxHtHGYJp17wDHRYAv3twLHZs1wHU9W+OWt9YCsI3dP+/BwVi45RAu79ICzy/MwAs390RmfgnuHtgBC7ccRodm9SEiePjKLthXWIJhF7XAq3f2QfuE+liVWYgl2/Jw6OQZlFcqLJw4FLe/vQ6v390Pu44U48O12WjaIA5JzRtizZ6jiI+tg2YN4qGgkF9chn/d2gsFp85iU85JxMfWQdeWjZCZfwrllQpv3NMPeUVn8O8lu7DiySsBAI3rxeHp0d1RR4AuiY2Csn+4Yy8XIqIIx14uRETnGSZ0IiKLYEInIrIIJnQiIotgQicisggmdCIii2BCJyKyCCZ0IiKLCOmFRSJSCOCAn4u3AHDUxHCCJRrijIYYgeiIMxpiBBinmcIR44VKqURvM4U0oQdCRFKNXCkVbtEQZzTECERHnNEQI8A4zRTJMbLKhYjIIpjQiYgsIpoS+qxwB2BQNMQZDTEC0RFnNMQIME4zRWyMUVOHTkREtYumM3QiIqpFVCR0ERktIrtFZI+ITArxtjuIyEoR2Ski20Xkz9r0ZiKyVESytP8J2nQRkZlarFtFpL/TusZr82eJyPggxBojIptFZJH2vJOIpGjb+1xE4rXpdbXne7TXk5zW8Yw2fbeIXBeEGJuKyJcisksr08sitCyf0D7vDBH5VETqRUJ5ishsESkQkQynaaaVn4gMEJFt2jIzRXy/5Y+HGF/RPvOtIvK1iDR1ek23jDx97z19DmbE6fTaX0VEiUgL7XlYytJnSqmI/gMQA2AvgM4A4gGkA+gRwu23AdBfe9wYQCaAHgBeBjBJmz4JwDTt8Q0AvoftDlRDAKRo05sB2Kf9T9AeJ5gc618AfAJgkfZ8PoC7tcfvAPij9vgRAO9oj+8G8Ln2uIdWvnUBdNLKPcbkGOcAeFB7HA+gaaSVJYB2APYDqO9Ujr+LhPIEMBxAfwAZTtNMKz8AGwBcpi3zPYDrTYrxWgCx2uNpTjHqlhFq+d57+hzMiFOb3gHAj7BdM9MinGXp83sK9gZM+HJdBuBHp+fPAHgmjPEsBHANgN0A2mjT2gDYrT1+F8A9TvPv1l6/B8C7TtNd5jMhrvYAlgO4GsAibSc66vQlcpSjtrNepj2O1eYT97J1ns+kGC+ALVGK2/RIK8t2AA5qX9JYrTyvi5TyBJAE12RpSvlpr+1ymu4yXyAxur12K4B52mPdMoKH731t+7VZcQL4EkAfANmoTuhhK0tf/qKhysX+5bLL1aaFnPZTuh+AFACtlFJ5AKD9b6nN5ineYL+P1wA8BaBKe94cwEmlVIXO9hyxaK8XafMHO8bOAAoBfCi2qqH3RaQhIqwslVKHALwKIAdAHmzlk4bIK087s8qvnfY42PHeD9sZqz8x1rZfB0xEbgJwSCmV7vZSpJali2hI6Hr1TiHvmiMijQAsAPC4Uqq4tll1pqlappsR240ACpRSaQbiqO21YJd1LGw/cd9WSvUDcBq2KgJPwhKnVgd9M2xVAG0BNARwfS3bDFd5euNrXEGPV0QmA6gAMM8+ycdYgvk9agBgMoDn9V72MZ6wfPbRkNBzYavTsmsP4HAoAxCRONiS+Tyl1Ffa5HwRaaO93gZAgTbdU7zBfB9DAdwkItkAPoOt2uU1AE1FJFZne45YtNebADge5Bjt281VSqVoz7+ELcFHUlkCwCgA+5VShUqpcgBfAbgckVeedmaVX672OCjxag2GNwIYp7R6CD9iPArPn0OgusB2EE/XvkvtAWwSkdZ+xBnUsvQo2HU6gf7Bdla3D7aCtjeO9Azh9gXAxwBec5v+Clwbol7WHo+Ba+PJBm16M9jqjxO0v/0AmgUh3qtQ3Sj6BVwbjx7RHk+EayPefO1xT7g2UO2D+Y2iqwF00x5P0coxosoSwGAA2wE00LY9B8BjkVKeqFmHblr5AdiozWtvyLvBpBhHA9gBINFtPt0yQi3fe0+fgxlxur2Wjeo69LCVpU/vJ9gbMCVIWwtzJmyt3pNDvO1hsP1U2gpgi/Z3A2x1ecsBZGn/7R+iAHhLi3UbgGSndd0PYI/29/sgxXsVqhN6Z9ha2vdoX4K62vR62vM92uudnZafrMW+G0FolQfQF0CqVp7faF+CiCtLAC8A2AUgA8BcLeGEvTwBfApbvX45bGeBD5hZfgCStfe8F8CbcGvADiDGPbDVNdu/Q+94KyN4+N57+hzMiNPt9WxUJ/SwlKWvf7xSlIjIIqKhDp2IiAxgQicisggmdCIii2BCJyKyCCZ0IiKLYEInIrIIJnQiIotgQicisoj/D41yYXXP5LIxAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "seq_lengths = [len(x) for x in sequences]\n",
    "plt.plot(seq_lengths)\n",
    "max(seq_lengths)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As seen from the graph, the tweets vary quite a lot in length, and maxes out at 36. We will now pad all of them to 36."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "MAX_SEQUENCE_LENGTH = max(seq_lengths)\n",
    "\n",
    "data = pad_sequences(sequences, maxlen=MAX_SEQUENCE_LENGTH)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Take note that if you set the maximum sequence length to be 36 during training, after your model is trained, it can only predict text up to 36 words as well. You may wish to increase the number above **before training** to give yourself more flexibility."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating One Hot Encoding Labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of data tensor: (14640, 36)\n",
      "Shape of label tensor: (14640, 3)\n"
     ]
    }
   ],
   "source": [
    "labels = to_categorical(np.asarray(labels))\n",
    "print('Shape of data tensor:', data.shape)\n",
    "print('Shape of label tensor:', labels.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Splitting the data into a train test split\n",
    "\n",
    "What this means is that we will allocate 80% of our cleaned up dataset for training the neural network. After every epoch (cycle) of training, the model will be \"examined\" against the other 20%, named the validation set, to see how well our model generalises (whether our model is 'memorising' its training set, which is undesirable)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we shuffle our sorted data, to make it random first\n",
    "indices = np.arange(data.shape[0])\n",
    "np.random.shuffle(indices)\n",
    "data = data[indices]\n",
    "labels = labels[indices]\n",
    "\n",
    "# before we begin to split them\n",
    "num_training_samples = int(TRAINING_SPLIT * data.shape[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train = data[:num_training_samples]\n",
    "y_train = labels[:num_training_samples]\n",
    "x_val = data[num_training_samples:]\n",
    "y_val = labels[num_training_samples:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparing Embedding Matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_words = min(MAX_NUM_WORDS, len(word_index)) + 1\n",
    "embedding_matrix = np.zeros((num_words, EMBEDDING_DIM))\n",
    "for word, i in word_index.items():\n",
    "    if i > MAX_NUM_WORDS:\n",
    "        continue\n",
    "    embedding_vector = embeddings_index.get(word)\n",
    "    if embedding_vector is not None:\n",
    "        # words not found in embedding index will be all-zeros.\n",
    "        embedding_matrix[i] = embedding_vector\n",
    "\n",
    "# load pre-trained word embeddings into an Embedding layer\n",
    "# note that we set trainable = False so as to keep the embeddings fixed\n",
    "embedding_layer = Embedding(num_words,\n",
    "                            EMBEDDING_DIM,\n",
    "                            embeddings_initializer=Constant(embedding_matrix),\n",
    "                            input_length=MAX_SEQUENCE_LENGTH,\n",
    "                            trainable=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defining and creating our model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /opt/anaconda/lib/python3.7/site-packages/tensorflow/python/ops/resource_variable_ops.py:435: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Colocations handled automatically by placer.\n"
     ]
    }
   ],
   "source": [
    "sequence_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='int32')\n",
    "embedded_sequences = embedding_layer(sequence_input)\n",
    "x = LSTM(128)(embedded_sequences)\n",
    "\n",
    "preds = Dense(labels.shape[1], activation='softmax')(x)\n",
    "model = Model(sequence_input, preds)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_1 (InputLayer)         (None, 36)                0         \n",
      "_________________________________________________________________\n",
      "embedding (Embedding)        (None, 36, 100)           1576900   \n",
      "_________________________________________________________________\n",
      "lstm (LSTM)                  (None, 128)               117248    \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 3)                 387       \n",
      "=================================================================\n",
      "Total params: 1,694,535\n",
      "Trainable params: 117,635\n",
      "Non-trainable params: 1,576,900\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='rmsprop',\n",
    "              metrics=['acc'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 11712 samples, validate on 2928 samples\n",
      "WARNING:tensorflow:From /opt/anaconda/lib/python3.7/site-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.cast instead.\n",
      "Epoch 1/10\n",
      "11712/11712 [==============================] - 7s 576us/sample - loss: 0.7311 - acc: 0.6997 - val_loss: 0.7483 - val_acc: 0.7240\n",
      "Epoch 2/10\n",
      "11712/11712 [==============================] - 6s 503us/sample - loss: 0.6280 - acc: 0.7450 - val_loss: 0.5924 - val_acc: 0.7527\n",
      "Epoch 3/10\n",
      "11712/11712 [==============================] - 6s 504us/sample - loss: 0.5875 - acc: 0.7610 - val_loss: 0.5686 - val_acc: 0.7688\n",
      "Epoch 4/10\n",
      "11712/11712 [==============================] - 6s 492us/sample - loss: 0.5578 - acc: 0.7719 - val_loss: 0.5784 - val_acc: 0.7657\n",
      "Epoch 5/10\n",
      "11712/11712 [==============================] - 6s 501us/sample - loss: 0.5368 - acc: 0.7812 - val_loss: 0.6173 - val_acc: 0.7555\n",
      "Epoch 6/10\n",
      "11712/11712 [==============================] - 6s 499us/sample - loss: 0.5126 - acc: 0.7935 - val_loss: 0.5788 - val_acc: 0.7630\n",
      "Epoch 7/10\n",
      "11712/11712 [==============================] - 6s 504us/sample - loss: 0.4901 - acc: 0.8044 - val_loss: 0.5330 - val_acc: 0.7886\n",
      "Epoch 8/10\n",
      "11712/11712 [==============================] - 6s 500us/sample - loss: 0.4743 - acc: 0.8095 - val_loss: 0.5688 - val_acc: 0.7688\n",
      "Epoch 9/10\n",
      "11712/11712 [==============================] - 6s 502us/sample - loss: 0.4507 - acc: 0.8177 - val_loss: 0.5204 - val_acc: 0.7862\n",
      "Epoch 10/10\n",
      "11712/11712 [==============================] - 6s 495us/sample - loss: 0.4283 - acc: 0.8295 - val_loss: 0.5547 - val_acc: 0.7794\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7efc4ba1e588>"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(x_train, y_train,\n",
    "          batch_size=128,\n",
    "          epochs=10,\n",
    "          validation_data=(x_val, y_val))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Making predictions with the model\n",
    "\n",
    "Our model is now trained! We have come to the exciting part where we try to make predictions with our model and see how well it predicts whether a tweet to an airline is negative or neutral or positive. Give it a try with your own input!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict_tweet(text_to_predict):\n",
    "    test_sequence = tokenizer.texts_to_sequences([text_to_predict])\n",
    "    test_data = pad_sequences(test_sequence, maxlen=MAX_SEQUENCE_LENGTH)\n",
    "    prediction = np.argmax(model.predict(test_data))\n",
    "    results = {\n",
    "    0: \"The tweet is negative about the airline.\",\n",
    "    1: \"The tweet is neutral about the airline.\",\n",
    "    2: \"The tweet is positive about the airline. Well done!\"\n",
    "    }\n",
    "    print(results[prediction])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The tweet is positive about the airline. Well done!\n"
     ]
    }
   ],
   "source": [
    "predict_tweet(\"Good job! Everyone that served me was friendly.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The tweet is negative about the airline.\n"
     ]
    }
   ],
   "source": [
    "predict_tweet(\"Food was horrible\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The tweet is positive about the airline. Well done!\n"
     ]
    }
   ],
   "source": [
    "predict_tweet(\"Food was horrible but the air-stewardesses were fantastic to me\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The tweet is neutral about the airline.\n"
     ]
    }
   ],
   "source": [
    "predict_tweet(\"Any chance there will be a discount on the tickets this week?\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The tweet is positive about the airline. Well done!\n"
     ]
    }
   ],
   "source": [
    "predict_tweet(\"Checking in was smooth and efficient.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The tweet is negative about the airline.\n"
     ]
    }
   ],
   "source": [
    "predict_tweet(\"In-flight entertainment was faulty\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are done! Now please open `2. Creating Mood Tweets Dataset.ipynb` for the next notebook on obtaining our mood tweets dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}