text_summarization_with_LSTM_Attention.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Jjvz_Vh3QJje"
   },
   "source": [
    "# Text Summarization\n",
    "\n",
    "Text summarization refers to the technique of shortening long pieces of text. The intention is to create a coherent and fluent summary having only the main points outlined in the document.\n",
    "Automatic text summarization is a common problem in machine learning and natural language processing (NLP)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "chqE5xQqJlCF"
   },
   "source": [
    "There are broadly two different approaches that are used for text summarization:\n",
    "\n",
    "* Extractive Summarization\n",
    "* Abstractive Summarization\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Y1ocpYWLMvK4"
   },
   "source": [
    "In this notebook, we will build an abstractive based text summarizer using deep learning from the scratch in python using keras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "snegcNtiJwmk"
   },
   "source": [
    "### Abstractive Summarization\n",
    "This is a very interesting approach. Here, we generate new sentences from the original text. This is in contrast to the extractive approach where we used only the sentences that were present. The sentences generated through abstractive summarization might not be present in the original text:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "q5UhiclEMFU4"
   },
   "source": [
    "![Abstractive_Summarization.png](images/Abstractive_Summarization.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "F5dSoP8lGMZi"
   },
   "source": [
    "# Problem Statement\n",
    "\n",
    "* Generating short length descriptions(headlines) from text(news articles).\n",
    "* Summarizing large amount of information which can be represented in compressed space\n",
    "\n",
    "The objective here is to generate a summary for the News Articles using the abstraction-based approach. You can download the dataset from[ here ](https://www.kaggle.com/sunnysai12345/news-summary)\n",
    "\n",
    "## Custom Attention Layer\n",
    "\n",
    "Keras does not officially support attention layer. So, we can either implement our own attention layer or use a third-party implementation. We will go with the latter option for this article. You can download the attention layer from [here](https://github.com/thushv89/attention_keras/blob/master/layers/attention.py) and copy it in a different file called attention.py.\n",
    "\n",
    "Let’s import it into our environment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Fi64aA0FFxcS"
   },
   "outputs": [],
   "source": [
    "# Get the attention layer\n",
    "from attention import AttentionLayer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
    "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5",
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 80
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 3799,
     "status": "ok",
     "timestamp": 1576665019347,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "_Jpu8qLEFxcY",
    "outputId": "0037372e-ecbb-45f7-f622-d3a276482e31"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<p style=\"color: red;\">\n",
       "The default version of TensorFlow in Colab will soon switch to TensorFlow 2.x.<br>\n",
       "We recommend you <a href=\"https://www.tensorflow.org/guide/migrate\" target=\"_blank\">upgrade</a> now \n",
       "or ensure your notebook will continue to use TensorFlow 1.x via the <code>%tensorflow_version 1.x</code> magic:\n",
       "<a href=\"https://colab.research.google.com/notebooks/tensorflow_version.ipynb\" target=\"_blank\">more info</a>.</p>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Import the Libraries\n",
    "import numpy as np\n",
    "import pandas as pd \n",
    "import re\n",
    "from bs4 import BeautifulSoup\n",
    "from keras.preprocessing.text import Tokenizer \n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "from nltk.corpus import stopwords\n",
    "from tensorflow.keras.layers import Input, LSTM, Embedding, Dense, Concatenate, TimeDistributed\n",
    "from tensorflow.keras.models import Model\n",
    "from tensorflow.keras.callbacks import EarlyStopping\n",
    "import warnings\n",
    "pd.set_option(\"display.max_colwidth\", 200)\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 68
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 3598,
     "status": "ok",
     "timestamp": 1576665019349,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "qsHhvN1aupnt",
    "outputId": "2fa8efa0-a5f7-4662-cc55-aa70da95bc4b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
      "[nltk_data]   Unzipping corpora/stopwords.zip.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 7,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import nltk\n",
    "nltk.download('stopwords')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "UVakjZ3oICgx"
   },
   "source": [
    "# Read the dataset\n",
    "\n",
    "The dataset consists of 4515 examples and contains Author_name, Headlines, Url of Article, Short text, Complete Article. I gathered the summarized news from Inshorts and only scraped the news articles from Hindu, Indian times and Guardian. Time period ranges from febrauary to august 2017."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "lbjHTtQBuSWu"
   },
   "outputs": [],
   "source": [
    "project_path = \"/text_summarization/\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "OHYJJ2G6jYWg"
   },
   "outputs": [],
   "source": [
    "news_summary=pd.read_csv(project_path+\"news_summary.csv\", encoding='iso-8859-1')\n",
    "news_summary_more=pd.read_csv(project_path+\"news_summary_more.csv\", encoding='iso-8859-1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 281
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1586,
     "status": "ok",
     "timestamp": 1576665569465,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "KjDrrPnmlZs3",
    "outputId": "19549188-720b-4a0b-99d9-704f94f0ab39"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>author</th>\n",
       "      <th>date</th>\n",
       "      <th>headlines</th>\n",
       "      <th>read_more</th>\n",
       "      <th>text</th>\n",
       "      <th>ctext</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Chhavi Tyagi</td>\n",
       "      <td>03 Aug 2017,Thursday</td>\n",
       "      <td>Daman &amp; Diu revokes mandatory Rakshabandhan in offices order</td>\n",
       "      <td>http://www.hindustantimes.com/india-news/rakshabandhan-compulsory-in-daman-and-diu-women-employees-to-tie-rakhis-to-male-colleagues/story-E5h5U1ZDJii5zFpLXWRkhJ.html?utm_source=inshorts&amp;utm_medium...</td>\n",
       "      <td>The Administration of Union Territory Daman and Diu has revoked its order that made it compulsory for women to tie rakhis to their male colleagues on the occasion of Rakshabandhan on August 7. The...</td>\n",
       "      <td>The Daman and Diu administration on Wednesday withdrew a circular that asked women staff to tie rakhis on male colleagues after the order triggered a backlash from employees and was ripped apart o...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Daisy Mowke</td>\n",
       "      <td>03 Aug 2017,Thursday</td>\n",
       "      <td>Malaika slams user who trolled her for 'divorcing rich man'</td>\n",
       "      <td>http://www.hindustantimes.com/bollywood/malaika-arora-khan-was-trolled-for-divorcing-a-rich-man-her-reply-is-dignity-itself/story-oIPZNI9deImCmabLMMwR2H.html?utm_source=inshorts&amp;utm_medium=referra...</td>\n",
       "      <td>Malaika Arora slammed an Instagram user who trolled her for \"divorcing a rich man\" and \"having fun with the alimony\". \"Her life now is all about wearing short clothes, going to gym or salon, enjoy...</td>\n",
       "      <td>From her special numbers to TV?appearances, Bollywood actor Malaika Arora Khan has managed to carve her own identity. The actor, who made her debut in the Hindi film industry with the blockbuster ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         author  ...                                                                                                                                                                                                    ctext\n",
       "0  Chhavi Tyagi  ...  The Daman and Diu administration on Wednesday withdrew a circular that asked women staff to tie rakhis on male colleagues after the order triggered a backlash from employees and was ripped apart o...\n",
       "1   Daisy Mowke  ...  From her special numbers to TV?appearances, Bollywood actor Malaika Arora Khan has managed to carve her own identity. The actor, who made her debut in the Hindi film industry with the blockbuster ...\n",
       "\n",
       "[2 rows x 6 columns]"
      ]
     },
     "execution_count": 14,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "news_summary.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 145
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1387,
     "status": "ok",
     "timestamp": 1576665569468,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "MFF-z1EalY2d",
    "outputId": "523b6a22-b244-499d-897b-6b349bb5bffc"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>headlines</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>upGrad learner switches to career in ML &amp; Al with 90% salary hike</td>\n",
       "      <td>Saurav Kant, an alumnus of upGrad and IIIT-B's PG Program in Machine learning and Artificial Intelligence, was a Sr Systems Engineer at Infosys with almost 5 years of work experience. The program ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Delhi techie wins free food from Swiggy for one year on CRED</td>\n",
       "      <td>Kunal Shah's credit card bill payment platform, CRED, gave users a chance to win free food from Swiggy for one year. Pranav Kaushik, a Delhi techie, bagged this reward after spending 2000 CRED coi...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                           headlines                                                                                                                                                                                                     text\n",
       "0  upGrad learner switches to career in ML & Al with 90% salary hike  Saurav Kant, an alumnus of upGrad and IIIT-B's PG Program in Machine learning and Artificial Intelligence, was a Sr Systems Engineer at Infosys with almost 5 years of work experience. The program ...\n",
       "1       Delhi techie wins free food from Swiggy for one year on CRED  Kunal Shah's credit card bill payment platform, CRED, gave users a chance to win free food from Swiggy for one year. Pranav Kaushik, a Delhi techie, bagged this reward after spending 2000 CRED coi..."
      ]
     },
     "execution_count": 15,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "news_summary_more.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "vNAiOdzvkdEd"
   },
   "outputs": [],
   "source": [
    "temp_df = news_summary.copy()\n",
    "temp_df[\"text\"] = temp_df[\"text\"].str.cat(temp_df['ctext'], sep = \" \")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "W_OX2ITQlb6W"
   },
   "outputs": [],
   "source": [
    "data = pd.DataFrame()\n",
    "data['Text'] = pd.concat([news_summary_more['text'], temp_df['text']], ignore_index=True)\n",
    "data['Summary'] = pd.concat([news_summary_more['headlines'],temp_df['headlines']],ignore_index = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "wnK5o4Z1Fxcj"
   },
   "outputs": [],
   "source": [
    "# data=pd.read_csv(project_path+\"Reviews.csv\",nrows=100000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kGNQKvCaISIn"
   },
   "source": [
    "# Drop Duplicates and NA values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Cjul88oOFxcr"
   },
   "outputs": [],
   "source": [
    "data.drop_duplicates(subset=['Text'],inplace=True)#dropping duplicates\n",
    "data.dropna(axis=0,inplace=True)#dropping na"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qi0xD6BkIWAm"
   },
   "source": [
    "# Information about dataset\n",
    "\n",
    "Let us look at datatypes and shape of the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 136
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1456,
     "status": "ok",
     "timestamp": 1576665606458,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "__fy-JxTFxc9",
    "outputId": "7813e4b9-bf97-4983-8d66-e5172854c180"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 102756 entries, 0 to 102914\n",
      "Data columns (total 2 columns):\n",
      "Text       102756 non-null object\n",
      "Summary    102756 non-null object\n",
      "dtypes: object(2)\n",
      "memory usage: 2.4+ MB\n"
     ]
    }
   ],
   "source": [
    "data.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "r0xLYACiFxdJ"
   },
   "source": [
    "# Preprocessing\n",
    "\n",
    "Performing basic preprocessing steps is very important before we get to the model building part. Using messy and uncleaned text data is a potentially disastrous move. So in this step, we will drop all the unwanted symbols, characters, etc. from the text that do not affect the objective of our problem.\n",
    "\n",
    "Here is the dictionary that we will use for expanding the contractions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "0s6IY-x2FxdL"
   },
   "outputs": [],
   "source": [
    "contraction_mapping = {\"ain't\": \"is not\", \"aren't\": \"are not\",\"can't\": \"cannot\", \"'cause\": \"because\", \"could've\": \"could have\", \"couldn't\": \"could not\",\n",
    "                           \"didn't\": \"did not\",  \"doesn't\": \"does not\", \"don't\": \"do not\", \"hadn't\": \"had not\", \"hasn't\": \"has not\", \"haven't\": \"have not\",\n",
    "                           \"he'd\": \"he would\",\"he'll\": \"he will\", \"he's\": \"he is\", \"how'd\": \"how did\", \"how'd'y\": \"how do you\", \"how'll\": \"how will\", \"how's\": \"how is\",\n",
    "                           \"I'd\": \"I would\", \"I'd've\": \"I would have\", \"I'll\": \"I will\", \"I'll've\": \"I will have\",\"I'm\": \"I am\", \"I've\": \"I have\", \"i'd\": \"i would\",\n",
    "                           \"i'd've\": \"i would have\", \"i'll\": \"i will\",  \"i'll've\": \"i will have\",\"i'm\": \"i am\", \"i've\": \"i have\", \"isn't\": \"is not\", \"it'd\": \"it would\",\n",
    "                           \"it'd've\": \"it would have\", \"it'll\": \"it will\", \"it'll've\": \"it will have\",\"it's\": \"it is\", \"let's\": \"let us\", \"ma'am\": \"madam\",\n",
    "                           \"mayn't\": \"may not\", \"might've\": \"might have\",\"mightn't\": \"might not\",\"mightn't've\": \"might not have\", \"must've\": \"must have\",\n",
    "                           \"mustn't\": \"must not\", \"mustn't've\": \"must not have\", \"needn't\": \"need not\", \"needn't've\": \"need not have\",\"o'clock\": \"of the clock\",\n",
    "                           \"oughtn't\": \"ought not\", \"oughtn't've\": \"ought not have\", \"shan't\": \"shall not\", \"sha'n't\": \"shall not\", \"shan't've\": \"shall not have\",\n",
    "                           \"she'd\": \"she would\", \"she'd've\": \"she would have\", \"she'll\": \"she will\", \"she'll've\": \"she will have\", \"she's\": \"she is\",\n",
    "                           \"should've\": \"should have\", \"shouldn't\": \"should not\", \"shouldn't've\": \"should not have\", \"so've\": \"so have\",\"so's\": \"so as\",\n",
    "                           \"this's\": \"this is\",\"that'd\": \"that would\", \"that'd've\": \"that would have\", \"that's\": \"that is\", \"there'd\": \"there would\",\n",
    "                           \"there'd've\": \"there would have\", \"there's\": \"there is\", \"here's\": \"here is\",\"they'd\": \"they would\", \"they'd've\": \"they would have\",\n",
    "                           \"they'll\": \"they will\", \"they'll've\": \"they will have\", \"they're\": \"they are\", \"they've\": \"they have\", \"to've\": \"to have\",\n",
    "                           \"wasn't\": \"was not\", \"we'd\": \"we would\", \"we'd've\": \"we would have\", \"we'll\": \"we will\", \"we'll've\": \"we will have\", \"we're\": \"we are\",\n",
    "                           \"we've\": \"we have\", \"weren't\": \"were not\", \"what'll\": \"what will\", \"what'll've\": \"what will have\", \"what're\": \"what are\",\n",
    "                           \"what's\": \"what is\", \"what've\": \"what have\", \"when's\": \"when is\", \"when've\": \"when have\", \"where'd\": \"where did\", \"where's\": \"where is\",\n",
    "                           \"where've\": \"where have\", \"who'll\": \"who will\", \"who'll've\": \"who will have\", \"who's\": \"who is\", \"who've\": \"who have\",\n",
    "                           \"why's\": \"why is\", \"why've\": \"why have\", \"will've\": \"will have\", \"won't\": \"will not\", \"won't've\": \"will not have\",\n",
    "                           \"would've\": \"would have\", \"wouldn't\": \"would not\", \"wouldn't've\": \"would not have\", \"y'all\": \"you all\",\n",
    "                           \"y'all'd\": \"you all would\",\"y'all'd've\": \"you all would have\",\"y'all're\": \"you all are\",\"y'all've\": \"you all have\",\n",
    "                           \"you'd\": \"you would\", \"you'd've\": \"you would have\", \"you'll\": \"you will\", \"you'll've\": \"you will have\",\n",
    "                           \"you're\": \"you are\", \"you've\": \"you have\"}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "2JFRXFHmI7Mj"
   },
   "source": [
    "We will perform the below preprocessing tasks for our data:\n",
    "\n",
    "1.Convert everything to lowercase\n",
    "\n",
    "2.Remove HTML tags\n",
    "\n",
    "3.Contraction mapping\n",
    "\n",
    "4.Remove (‘s)\n",
    "\n",
    "5.Remove any text inside the parenthesis ( )\n",
    "\n",
    "6.Eliminate punctuations and special characters\n",
    "\n",
    "7.Remove stopwords\n",
    "\n",
    "8.Remove short words\n",
    "\n",
    "Let’s define the function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "XZr-u3OEFxdT"
   },
   "outputs": [],
   "source": [
    "stop_words = set(stopwords.words('english')) \n",
    "\n",
    "def text_cleaner(text,num):\n",
    "    newString = text.lower()\n",
    "    newString = BeautifulSoup(newString, \"lxml\").text\n",
    "    newString = re.sub(r'\\([^)]*\\)', '', newString)\n",
    "    newString = re.sub('\"','', newString)\n",
    "    newString = ' '.join([contraction_mapping[t] if t in contraction_mapping else t for t in newString.split(\" \")])    \n",
    "    newString = re.sub(r\"'s\\b\",\"\",newString)\n",
    "    newString = re.sub(\"[^a-zA-Z]\", \" \", newString) \n",
    "    newString = re.sub('[m]{2,}', 'mm', newString)\n",
    "    if(num==0):\n",
    "        tokens = [w for w in newString.split() if not w in stop_words]\n",
    "    else:\n",
    "        tokens=newString.split()\n",
    "    long_words=[]\n",
    "    for i in tokens:\n",
    "        if len(i)>1: #removing short word                                                \n",
    "            long_words.append(i)   \n",
    "    return (\" \".join(long_words)).strip()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "A2QAeCHWFxdY"
   },
   "outputs": [],
   "source": [
    "#call the function\n",
    "cleaned_text = []\n",
    "for t in data['Text']:\n",
    "    cleaned_text.append(text_cleaner(t,0)) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "snRZY8wjLao2"
   },
   "source": [
    "Let us look at the first five preprocessed reviews"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 122
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1306,
     "status": "ok",
     "timestamp": 1576665662360,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "NCAIkhWbFxdh",
    "outputId": "1a2d0042-e49c-469d-bd30-5e069f9d94bd"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['saurav kant alumnus upgrad iiit pg program machine learning artificial intelligence sr systems engineer infosys almost years work experience program upgrad degree career support helped transition data scientist tech mahindra salary hike upgrad online power learning powered lakh careers',\n",
       " 'kunal shah credit card bill payment platform cred gave users chance win free food swiggy one year pranav kaushik delhi techie bagged reward spending cred coins users get one cred coin per rupee bill paid used avail rewards brands like ixigo bookmyshow ubereats cult fit',\n",
       " 'new zealand defeated india wickets fourth odi hamilton thursday win first match five match odi series india lost international match rohit sharma captaincy consecutive victories dating back march match witnessed india getting seventh lowest total odi cricket history',\n",
       " 'aegon life iterm insurance plan customers enjoy tax benefits premiums paid save taxes plan provides life cover age years also customers options insure critical illnesses disability accidental death benefit rider life cover age years',\n",
       " 'speaking sexual harassment allegations rajkumar hirani sonam kapoor said known hirani many years true metoo movement get derailed metoo movement always believe woman case need reserve judgment added hirani accused assistant worked sanju']"
      ]
     },
     "execution_count": 23,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cleaned_text[:5]  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "GsRXocxoFxd-"
   },
   "outputs": [],
   "source": [
    "#call the function\n",
    "cleaned_summary = []\n",
    "for t in data['Summary']:\n",
    "    cleaned_summary.append(text_cleaner(t,1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "oZeD0gs6Lnb-"
   },
   "source": [
    "Let us look at the first 10 preprocessed summaries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 187
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 34121,
     "status": "ok",
     "timestamp": 1576389567187,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "jQJdZcAzFxee",
    "outputId": "afd9bbab-7252-4893-9bc4-1919c4097202"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['good quality dog food',\n",
       " 'not as advertised',\n",
       " 'delight says it all',\n",
       " 'cough medicine',\n",
       " 'great taffy',\n",
       " 'nice taffy',\n",
       " 'great just as good as the expensive brands',\n",
       " 'wonderful tasty taffy',\n",
       " 'yay barley',\n",
       " 'healthy dog food']"
      ]
     },
     "execution_count": 14,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cleaned_summary[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "L1zLpnqsFxey"
   },
   "outputs": [],
   "source": [
    "data['cleaned_text']=cleaned_text\n",
    "data['cleaned_summary']=cleaned_summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "KT_D2cLiLy77"
   },
   "source": [
    "#Drop empty rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "sYK390unFxfA"
   },
   "outputs": [],
   "source": [
    "data.replace('', np.nan, inplace=True)\n",
    "data.dropna(axis=0,inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Vm8Fk2TCL7Sp"
   },
   "source": [
    "# Understanding the distribution of the sequences\n",
    "\n",
    "Here, we will analyze the length of the reviews and the summary to get an overall idea about the distribution of length of the text. This will help us fix the maximum length of the sequence:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 281
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1967,
     "status": "ok",
     "timestamp": 1576665755678,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "MdF76AHHFxgw",
    "outputId": "3a80028d-9221-43ee-8bc0-2d3cc0421adc"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAEICAYAAAC9E5gJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAfbklEQVR4nO3df5BV5Z3n8fcnoMZoFIyZDgFnIJHK\nFNGNP1glZSrTKwkgZgZTZbK4bkDDhOyKs2bHTcTMVJmJukN2V426xiyJjOgQ0RATGcUhjNKVTe2C\nghIRiWsHSYBCifLDYKIG890/zrf1cLm3+3bTfe9t+LyqTvU5z3nOud97+3R/73nOc86jiMDMzA5v\n72h2AGZm1nxOBmZm5mRgZmZOBmZmhpOBmZnhZGBmZjgZmJkZTgZm1uIkbZb0iX7Yz52SruuPmA5F\nTgZWN0lDmx2DmQ0MJ4MGk3SVpG2SfiPpWUkTK7+xSGqXtLW0vFnSlyU9JelVSXdIapP0cO7nXyQN\nz7qjJYWkSyVtkbRL0n+Q9K9z+92S/mdp3x+U9KiklyW9JGmRpGEVr32VpKeAVzOOH1S8p1sk3Tyg\nH5wdliTdDfwx8E+S9kr6iqQJkv5PHss/k9SedU+QtFXSn+fysZI6Jc2QNBu4GPhK7uefmvamWlVE\neGrQBHwI2AK8P5dHAx8E7gSuK9VrB7aWljcDq4A2YCSwA3gCOB14J/AocE1pnwF8O9dNAl4DfgT8\nUWn7P8v6JwOfBI4C3gv8BPhmxWuvA04CjgZGAK8Cw3L90Nzfmc3+fD0dmlMeg5/I+ZHAy8BUii+z\nn8zl9+b6ScALeax/B1hS2s9+f2ee9p98ZtBYb1L80x0n6YiI2BwRv6hz21sj4sWI2Ab8b2B1RDwZ\nEa8BP6RIDGXXRsRrEfFjin/e90TEjtL2pwNERGdErIiI1yPi18CNwJ9V7OuWiNgSEb+LiO0UCeMz\nuW4K8FJErO3VJ2HWN/8eWBYRyyLiDxGxAlhDkRzI4/37wCNZ9sWmRTrIOBk0UER0Al8CvgbskLRY\n0vvr3PzF0vzvqiwf25f62dy0OJuuXgH+ETixYl9bKpYXUvxRkj/vrvM9mB2sPwE+k01EuyXtBj5G\nccbaZT5wCnBnRLzcjCAHIyeDBouI70XExygO6gC+QfHN/V2lau9rYEj/NeM4NSKOo/jnroo6lY+2\n/RHwrySdAnwKWDTgUdrhrHz8bQHujohhpemYiJgHIGkIRTK4C7hM0sk19mMVnAwaSNKHJJ0r6SiK\ndvzfAX+gaJOfmhfA3kdx9tAo7wb2AnskjQS+3NMG2TS1BPge8FhE/GpgQ7TD3IvAB3L+H4E/lzRZ\n0hBJ78wOF6Ny/Vcp/ul/HvjvwF2ZICr3YxWcDBrrKGAe8BJvX+S6mqKZ5WcUF8p+DNzbwJj+DjgD\n2AM8BNxf53YLgVNxE5ENvL8H/jabhP4tMI3in/6vKc4Uvgy8Q9KZwF8DMyLiTYqz7gDm5n7uoLhe\nt1vSjxr8Hlqe8iq7Wa9I+mPg58D7IuKVZsdjZgfHZwbWa5LeQfENbLETgdmhwXeUWq9IOoai7fWX\nFN1KzewQ4GYiMzNzM5GZmQ3iZqITTzwxRo8e3ewwevTqq69yzDHHNDuMuh1O8a5du/aliHhvP4c0\nYLo75lv59+bY+m4g4qt53Df7eRh9nc4888wYDFauXNnsEHrlcIoXWBMtcCzXO3V3zLfy782x9d1A\nxFfruHczkZmZORmYmZmTgZmZ4WRgZmY4GZiZGXUkg3wq4GM5vNwGSX+X5WMkrc5h5e6VdGSWH5XL\nnbl+dGlfV2f5s5Iml8qnZFmnpLmVMZiZ2cCq58zgdeDciPgIcBowRdIEiicC3hQRJwO7gFlZfxaw\nK8tvynpIGgdMBz5M8RiDb+UjaIcAtwHnAeOAi7KumZk1SI/JILum7s3FI3IK4FyKZ9pD8TjjC3J+\nWi6T6ydKUpYvjmJ4xeeBTuCsnDojYlNEvAEszrpmZtYgdd2BnN/e11IMnn4b8Atgd0TsyypbKQaq\nJn9uAYiIfZL2AO/J8lWl3Za32VJRfnaNOGYDswHa2tro6OioJ/ym2rt376CIs8thFu9oSTuAHRFx\nCoCkEyjGkxhNMb7EZyNiV36huZliXN3fApdExBO5zUzgb3Of10XEwiw/k2IQ9qOBZcAVERG1XqOv\nb8KsP9SVDKIYKOI0ScMoBl//0wGNqnYc8ymGtGP8+PHR3t7ejDB6paOjg1aJc/Tch/Zb3jzv/APq\ntFK89TjIeF8C/h3FEIld5gKPRMS8vH41F7iKohlzbE5nA7cDZ+c/9muA8RRnzGslLc1/7rcDXwBW\nUySDKcDD3bxGn63ftodLSr/far9bs+70qjdRROwGVgIfBYZJ6komo4BtOb8NOAkg1x8PvFwur9im\nVrnZQNsL7KwoKzdzVjZ/3pXNpqsojv8RwGRgRUTszASwguK62gjguIhYlY8AuIvqTanl1zBrmh7P\nDCS9F/h9ROyWdDTwSYqLwiuBCyna+GcCD+QmS3P5/+b6R/PUeCnwPUk3Au+n+Ib1GMXg62MljaFI\nAtMpvq2ZNUNbRGzP+ReAtpx/q/kzdTVzdle+tUp5d69xgHqbRtuOhitP3ffWcis19bVy02MrxwaN\nja+eZqIRwMK8bvAO4L6IeFDSM8BiSdcBT1KML0r+vFtSJ8W3rukAEbFB0n3AM8A+YE42PyHpcmA5\nMARYEBEb+u0dmvVRfokZ0AE/enqNeptGb130ADesf/vPefPF1es1Qys3PbZybNDY+HpMBhHxFHB6\nlfJNFD2BKstfAz5TY1/XA9dXKV9G0aZq1mwvShoREduzqWdHlnfXzNleUd6R5aOq1O/uNcyaxncg\nm+2vq5kTDmz+nKHCBGBPNvUsByZJGi5pODAJWJ7rXpE0IXsizeDAptTK1zBrmkE7uI1ZPxhDcW3r\nRElbKXoFzQPukzSLYpznz2bdZRTdSjspupZeChAROyVdCzye9b4eEV0XpS/j7a6lD+dEN69h1jRO\nBnY4ez4ixlcpn1hZkD2C5lTbSUQsABZUKV8DnFKl/OVqr2HWTG4mMjMzJwMzM3MyMDMznAzMzAwn\nAzMzw8nAzMxwMjAzM5wMzMwMJwMzM8PJwMzMcDIwMzOcDMzMDCcDMzPDycDMzHAyMDMznAzMzAwn\nAzMzw8nAzMxwMjAzM5wMzMwMJwMzMwOGNjsAay3rt+3hkrkPvbW8ed75TYzGzBrFZwZmZtZzMpB0\nkqSVkp6RtEHSFVn+NUnbJK3LaWppm6sldUp6VtLkUvmULOuUNLdUPkbS6iy/V9KR/f1GzcystnrO\nDPYBV0bEOGACMEfSuFx3U0ScltMygFw3HfgwMAX4lqQhkoYAtwHnAeOAi0r7+Ubu62RgFzCrn96f\nmZnVocdkEBHbI+KJnP8NsBEY2c0m04DFEfF6RDwPdAJn5dQZEZsi4g1gMTBNkoBzgSW5/ULggr6+\nITMz671eXUCWNBo4HVgNnANcLmkGsIbi7GEXRaJYVdpsK28njy0V5WcD7wF2R8S+KvUrX382MBug\nra2Njo6O3oTfFHv37m2ZOK88dd9+y9Xiajt6/3qtEnstrfT5mg1mdScDSccCPwC+FBGvSLoduBaI\n/HkD8PkBiTJFxHxgPsD48eOjvb19IF+uX3R0dNAqcZZ7CQFsvrj9gDq3LnqAG9YP7bZOK2mlz9ds\nMKsrGUg6giIRLIqI+wEi4sXS+u8AD+biNuCk0uajsowa5S8DwyQNzbODcn0zM2uAenoTCbgD2BgR\nN5bKR5SqfRp4OueXAtMlHSVpDDAWeAx4HBibPYeOpLjIvDQiAlgJXJjbzwQeOLi3ZWZmvVHPmcE5\nwOeA9ZLWZdlXKXoDnUbRTLQZ+CJARGyQdB/wDEVPpDkR8SaApMuB5cAQYEFEbMj9XQUslnQd8CRF\n8jEzswbpMRlExE8BVVm1rJttrgeur1K+rNp2EbGJoreRmZk1ge9ANqtC0n/OmyyflnSPpHfWujky\nm0TvzfLV2euuaz+9ugHTrFmcDMwqSBoJ/CdgfEScQtGsOZ3aN0fOAnZl+U1Zr683YJo1hZOBWXVD\ngaMlDQXeBWyn9s2R03KZXD8xO1706gbMBrwns5r81FKzChGxTdL/AH4F/A74MbCW2jdHjiRvqIyI\nfZL2UNxM2dsbMA9Q742WrXyzYCvfGNjKsUFj43MyMKsgaTjFN/UxwG7g+xTNPA1X742WrXyzYCvf\nGNjKsUFj43MzkdmBPgE8HxG/jojfA/dTdLEels1GsP/NkW/daJnrj6e4mbLWDZjd3Zhp1hROBmYH\n+hUwQdK7su1/IsV9M7Vujlyay+T6R/Nmyl7dgNmA92VWk5uJzCpExGpJS4AnKG6cfJKiqeYhqt8c\neQdwt6ROYCfFP/e+3oBp1hROBmZVRMQ1wDUVxVVvjoyI14DP1NhPr27ANGsWNxOZmZmTgZmZORmY\nmRlOBmZmhpOBmZnhZGBmZjgZmJkZTgZmZoaTgZmZ4WRgZmY4GZiZGU4GZmaGk4GZmeFkYGZmOBmY\nmRlOBmZmRh3JQNJJklZKekbSBklXZPkJklZIei5/Ds9ySbpFUqekpySdUdrXzKz/nKSZpfIzJa3P\nbW7JoQbNzKxB6jkz2AdcGRHjgAnAHEnjgLnAIxExFngklwHOoxjrdSwwG7gdiuRBMXLU2RSjRV3T\nlUCyzhdK2005+LdmZmb16jEZRMT2iHgi538DbARGAtOAhVltIXBBzk8D7orCKmCYpBHAZGBFROyM\niF3ACmBKrjsuIlblIOJ3lfZlZmYN0KsxkCWNBk4HVgNtEbE9V70AtOX8SGBLabOtWdZd+dYq5dVe\nfzbF2QZtbW10dHT0Jvym2Lt3b8vEeeWp+/ZbrhZX29H712uV2Gtppc/XbDCrOxlIOhb4AfCliHil\n3KwfESEpBiC+/UTEfGA+wPjx46O9vX2gX/KgdXR00CpxXjL3of2WN1/cfkCdWxc9wA3rh3Zbp5W0\n0udrNpjV1ZtI0hEUiWBRRNyfxS9mEw/5c0eWbwNOKm0+Ksu6Kx9VpdzMzBqknt5EAu4ANkbEjaVV\nS4GuHkEzgQdK5TOyV9EEYE82Jy0HJkkanheOJwHLc90rkibka80o7cvMzBqgnmaic4DPAeslrcuy\nrwLzgPskzQJ+CXw21y0DpgKdwG+BSwEiYqeka4HHs97XI2Jnzl8G3AkcDTyck5mZNUiPySAifgrU\n6vc/sUr9AObU2NcCYEGV8jXAKT3FYrWNrrweMO/8JkViZoOR70A2MzMnAzMzczIwMzOcDMzMDCcD\nMzPDycDMzHAyMDMznAzMqpI0TNISST+XtFHSRz2Ghx3KnAzMqrsZ+OeI+FPgIxSPbvcYHnbIcjIw\nqyDpeODjFM/kIiLeiIjdeAwPO4T1ajwDs8PEGODXwD9I+giwFriCFh7Do5XHoWjlMSdaOTZobHxO\nBmYHGgqcAfxVRKyWdDNvNwkBrTeGRyuPQ9HKY060cmzQ2PjcTGR2oK3A1ohYnctLKJKDx/CwQ5aT\ngVmFiHgB2CLpQ1k0EXgGj+FhhzA3E5lV91fAIklHApsoxuV4Bx7Dww5RTgZmVUTEOmB8lVUew8MO\nSW4mMjMzJwMzM3MyMDMznAzMzAwnAzMzw8nAzMxwMjAzM5wMzMwMJwMzM8PJwMzMqCMZSFogaYek\np0tlX5O0TdK6nKaW1l2dQ/k9K2lyqXxKlnVKmlsqHyNpdZbfm8+CMTOzBqrnzOBOqg/Jd1NEnJbT\nMgBJ44DpwIdzm29JGiJpCHAbxfCA44CLsi7AN3JfJwO7gFkH84bMzKz3ekwGEfETYGdP9dI0YHFE\nvB4Rz1M8xfGsnDojYlNEvAEsBqbl43vPpXhePOw/lKCZmTXIwTy19HJJM4A1wJU5xutIYFWpTnk4\nv8rh/84G3gPsjoh9VeofoN4hAFtJo4atKw95CNWHPaynTisPn1hNqw9baDZY9DUZ3A5cC0T+vAH4\nfH8FVUu9QwC2kkYNW3fJ3If2W6427GE9dVp5+MRqWn3YQrPBok/JICJe7JqX9B3gwVysNcwfNcpf\nBoZJGppnBx7+z8ysCfrUtbRrHNj0aaCrp9FSYLqkoySNAcYCj1GM9DQ2ew4dSXGReWkOCrISuDC3\nLw8laGZmDdLjmYGke4B24ERJW4FrgHZJp1E0E20GvggQERsk3UcxXuw+YE5EvJn7uZxiTNghwIKI\n2JAvcRWwWNJ1wJPAHf327szMrC49JoOIuKhKcc1/2BFxPXB9lfJlFGPFVpZvouhtZGZmTeI7kM3M\nzMnAzMycDMzMDCcDMzPDycDMzHAyMDMznAzMzIyDe1CdHaZGVz7jaN75TYrEzPqLzwzMzMzJwMzM\nnAzMzAwnAzMzw8nAzMxwMjCrSdIQSU9KejCXx0haLalT0r05Ngc5fse9Wb5a0ujSPq7O8mclTS6V\nT8myTklzG/3ezCo5GZjVdgWwsbT8DeCmiDgZ2AXMyvJZwK4svynrIWkcxUBOHwamAN/KBDMEuA04\nDxgHXJR1zZrGycCsCkmjgPOB7+aygHOBJVllIXBBzk/LZXL9xKw/DVgcEa9HxPNAJ8XYHWcBnRGx\nKSLeABZnXbOm8U1nZtV9E/gK8O5cfg+wO8fqBtgKjMz5kcAWgIjYJ2lP1h8JrCrts7zNlorys6sF\nIWk2MBugra2Njo6OqsG2HQ1XnrrvreVa9Zph7969LRVPWSvHBo2Nz8nArIKkTwE7ImKtpPZmxhIR\n84H5AOPHj4/29urh3LroAW5Y//af8+aLq9drho6ODmrF3WytHBs0Nj4nA7MDnQP8haSpwDuB44Cb\ngWGShubZwShgW9bfBpwEbJU0FDgeeLlU3qW8Ta1ys6bwNQOzChFxdUSMiojRFBeAH42Ii4GVwIVZ\nbSbwQM4vzWVy/aMREVk+PXsbjQHGAo8BjwNjs3fSkfkaSxvw1sxq8pmBWf2uAhZLug54Ergjy+8A\n7pbUCeyk+OdORGyQdB/wDLAPmBMRbwJIuhxYDgwBFkTEhoa+E7MKTgZm3YiIDqAj5zdR9ASqrPMa\n8Jka218PXF+lfBmwrB9DNTsobiYyMzMnAzMzczIwMzPqSAaSFkjaIenpUtkJklZIei5/Ds9ySbol\nn7fylKQzStvMzPrPSZpZKj9T0vrc5pa8c9PMzBqonjODOymeq1I2F3gkIsYCj+QyFM9aGZvTbOB2\nKJIHcA3FXZZnAdd0JZCs84XSdpWvZWZmA6zHZBARP6HoLldWfhZL5TNa7orCKoqbdEYAk4EVEbEz\nInYBK4Apue64iFiV/bLvKu3LzMwapK/XDNoiYnvOvwC05fxbz2hJXc9i6a58a5VyMzNroIO+zyAi\nQlL0RzA9qfehXa2kUQ+aKj+kDKo/qKyeOvU88Kye/TRKqz9ozGyw6GsyeFHSiIjYnk09O7K81rNY\ntgHtFeUdWT6qSv2q6n1oVytp1IOmLpn70H7L1R5UVk+deh54Vs9+GqXVHzRmNlj0tZmo/CyWyme0\nzMheRROAPdmctByYJGl4XjieBCzPda9ImpC9iGaU9mVmZg3S45mBpHsovtWfKGkrRa+gecB9kmYB\nvwQ+m9WXAVMpBvH4LXApQETslHQtxQO6AL4eEV0XpS+j6LF0NPBwTmZm1kA9JoOIuKjGqolV6gYw\np8Z+FgALqpSvAU7pKQ4zMxs4vgPZzMycDMzMzMnAzMxwMjAzM5wMzMwMJwMzM8PJwMzMcDIwMzOc\nDMzMDCcDMzPDycDMzHAyMDMznAzMzAwnAzMzox+GvbSBN7piZDGAzfPOb0IkZnao8pmBmZk5GZiZ\nmZOBmZnhZGBmZjgZmJkZTgZmB5B0kqSVkp6RtEHSFVl+gqQVkp7Ln8OzXJJukdQp6SlJZ5T2NTPr\nPydpZqn8TEnrc5tbJKnx79TsbU4GZgfaB1wZEeOACcAcSeOAucAjETEWeCSXAc4DxuY0G7gdiuQB\nXAOcDZwFXNOVQLLOF0rbTWnA+zKrycnArEJEbI+IJ3L+N8BGYCQwDViY1RYCF+T8NOCuKKwChkka\nAUwGVkTEzojYBawApuS64yJiVUQEcFdpX2ZN4ZvOzLohaTRwOrAaaIuI7bnqBaAt50cCW0qbbc2y\n7sq3Vimv9vqzKc42aGtro6Ojo2qcbUfDlafue2u5Vr1m2Lt3b0vFU9bKsUFj43MyMKtB0rHAD4Av\nRcQr5Wb9iAhJMdAxRMR8YD7A+PHjo729vWq9Wxc9wA3r3/5z3nxx9XrN0NHRQa24m62VY4PGxudm\nIrMqJB1BkQgWRcT9WfxiNvGQP3dk+TbgpNLmo7Ksu/JRVcrNmuagkoGkzdkjYp2kNVnWbz0uzJoh\ne/bcAWyMiBtLq5YCXcfnTOCBUvmMPMYnAHuyOWk5MEnS8Pw7mAQsz3WvSJqQrzWjtC+zpuiPM4N/\nExGnRcT4XO7PHhdmzXAO8Dng3Pyis07SVGAe8ElJzwGfyGWAZcAmoBP4DnAZQETsBK4FHs/p61lG\n1vlubvML4OFGvDGzWgbimsE0oD3nFwIdwFWUelwAqyR19bhoJ3tcAEhaQdHN7p4BiM2sRxHxU6BW\nv/+JVeoHMKfGvhYAC6qUrwFOOYgwzfrVwSaDAH6cF9L+V17s6q8eFweot2dFK+mP3gDlXiJdKvdZ\nWafaa9ZTp55eKfXsp1FavTeI2WBxsMngYxGxTdIfASsk/by8sr97XNTbs6KV9EdvgEuqjWdQ0Vuk\nsk613iT11KmnV0o9+2mUVu8NYjZYHNQ1g4jYlj93AD+kaPPvrx4XZmbWIH1OBpKOkfTurnmKnhJP\n0089Lvoal5mZ9d7BNBO1AT/MG3GGAt+LiH+W9Dhwn6RZwC+Bz2b9ZcBUit4TvwUuhaLHhaSuHhew\nf48LMzNrgD4ng4jYBHykSvnL9FOPCxucPGaz2eDjO5DNzMzJwMzMnAzMzAwnAzMzw8nAzMxwMjAz\nM5wMzMwMJwMzM8PJwMzMcDIwMzOcDMzMDCcDMzPDycDMzHAyMDMznAzMzAwnAzMz4+BGOrN+4IFg\nzKwV+MzAzMycDMzMzMnAzMxwMjAzM5wMzMwM9yayJnEvKrPW4jMDMzNzMjAzMycDMzOjha4ZSJoC\n3AwMAb4bEfOaHJLZgBrIY77ymoyvx1hPWiIZSBoC3AZ8EtgKPC5paUQ809zIDt76bXu4pPSH6T9K\ng0P7mLfBqSWSAXAW0BkRmwAkLQamAf7DOIwd4t9uG3rMu/eW9UQR0ewYkHQhMCUi/jKXPwecHRGX\nV9SbDczOxQ8BzzY00L45EXip2UH0wuEU759ExHv7M5h6DcAx38q/N8fWdwMRX9XjvlXODOoSEfOB\n+c2OozckrYmI8c2Oo16Ot7XUe8y38ufg2PqukfG1Sm+ibcBJpeVRWWZ2qPIxby2lVZLB48BYSWMk\nHQlMB5Y2OSazgeRj3lpKSzQTRcQ+SZcDyym62S2IiA1NDqu/DKpmLRxvQwzAMd/Kn4Nj67uGxdcS\nF5DNzKy5WqWZyMzMmsjJwMzMnAwGiqTNktZLWidpTbPjqUbSAkk7JD1dKjtB0gpJz+XP4c2MsaxG\nvF+TtC0/53WSpjYzxkaTNEXSs5I6Jc1t0GueJGmlpGckbZB0RZZXPXZUuCVjfErSGaV9zcz6z0ma\n2Y8xDpH0pKQHc3mMpNUZw7150R5JR+VyZ64fXdrH1Vn+rKTJ/RjbMElLJP1c0kZJH22Jzy4iPA3A\nBGwGTmx2HD3E+HHgDODpUtl/A+bm/FzgG82Os4d4vwb8l2bH1qTPYwjwC+ADwJHAz4BxDXjdEcAZ\nOf9u4P8B42odO8BU4GFAwARgdZafAGzKn8Nzfng/xfjXwPeAB3P5PmB6zn8b+I85fxnw7ZyfDtyb\n8+Py8zwKGJOf85B+im0h8Jc5fyQwrBU+O58ZHMYi4ifAzoriaRQHK/nzgoYG1Y0a8R7O3nqkRUS8\nAXQ90mJARcT2iHgi538DbARGUvvYmQbcFYVVwDBJI4DJwIqI2BkRu4AVwJSDjU/SKOB84Lu5LOBc\nYEmN2LpiXgJMzPrTgMUR8XpEPA90UnzeBxvb8RRfau4AiIg3ImI3LfDZORkMnAB+LGltPlJgsGiL\niO05/wLQ1sxg6nR5nkIvaKVmrQYYCWwpLW/NsobJZpXTgdXUPnZqxTlQ8X8T+Arwh1x+D7A7IvZV\neZ23Ysj1e7L+QMU2Bvg18A/ZjPVdScfQAp+dk8HA+VhEnAGcB8yR9PFmB9RbUZyPtnrf49uBDwKn\nAduBG5obzuFD0rHAD4AvRcQr5XXNOnYkfQrYERFrG/3adRpK0dR5e0ScDrxK0Sz0lmZ9dk4GAyQi\ntuXPHcAP6YdTzAZ5MU9DyZ87mhxPtyLixYh4MyL+AHyHwfM594emPdJC0hEUiWBRRNyfxbWOnVpx\nDkT85wB/IWkzRbPZuRRjRgyT1HWTbfl13ooh1x8PvDxAsUHxDX5rRKzO5SUUyaHpn52TwQCQdIyk\nd3fNA5OAp7vfqmUsBbp6JswEHmhiLD3q+gNKn2bwfM79oSmPtMg29TuAjRFxY2lVrWNnKTAje8ZM\nAPZkk8hyYJKk4dm8NynL+iwiro6IURExmuLzeDQiLgZWAhfWiK0r5guzfmT59OxtNAYYCzx2MLFl\nfC8AWyR9KIsmUjy2vOmfXdN7RByKE0Xvjp/ltAH4m2bHVCPOeyiaVn5P8Y1lFkV76SPAc8C/ACc0\nO84e4r0bWA88lX84I5odZ4M/k6kUvXl+0ajjDPgYRTPGU8C6nKbWOnYoesLcljGuB8aX9vV5iouz\nncCl/RxnO2/3JvoAxT/zTuD7wFFZ/s5c7sz1Hyht/zcZ87PAef0Y12nAmvz8fkTRG6jpn50fR2Fm\nZm4mMjMzJwMzM8PJwMzMcDIwMzOcDMzMDCcDMzPDycDMzID/DxuGRVvV6UkoAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "text_word_count = []\n",
    "summary_word_count = []\n",
    "\n",
    "# populate the lists with sentence lengths\n",
    "for i in data['cleaned_text']:\n",
    "      text_word_count.append(len(i.split()))\n",
    "\n",
    "for i in data['cleaned_summary']:\n",
    "      summary_word_count.append(len(i.split()))\n",
    "\n",
    "length_df = pd.DataFrame({'text':text_word_count, 'summary':summary_word_count})\n",
    "\n",
    "length_df.hist(bins = 30)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "QwdSGIhGMEbz"
   },
   "source": [
    "Interesting. We can fix the maximum length of the summary to 16 since that seems to be the majority summary length.\n",
    "\n",
    "Let us understand the proportion of the length of summaries below 16"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1298,
     "status": "ok",
     "timestamp": 1576665822992,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "7JRjwdIOFxg3",
    "outputId": "e651bccc-db4d-403b-e541-ba2b15f2031f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.0\n"
     ]
    }
   ],
   "source": [
    "cnt=0\n",
    "for i in data['cleaned_summary']:\n",
    "    if(len(i.split())<=16):\n",
    "        cnt=cnt+1\n",
    "print(cnt/len(data['cleaned_summary']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "yYB4Ga9KMjEu"
   },
   "source": [
    "We observe that 100% of the summaries have length below 16. So, we can fix maximum length of summary to 16.\n",
    "\n",
    "Let us fix the maximum length of review to 100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ZKD5VOWqFxhC"
   },
   "outputs": [],
   "source": [
    "max_text_len=100\n",
    "max_summary_len=16"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "E6d48E-8M4VO"
   },
   "source": [
    "Let us select the reviews and summaries whose length falls below or equal to **max_text_len** and **max_summary_len**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "yY0tEJP0FxhI"
   },
   "outputs": [],
   "source": [
    "cleaned_text =np.array(data['cleaned_text'])\n",
    "cleaned_summary=np.array(data['cleaned_summary'])\n",
    "\n",
    "short_text=[]\n",
    "short_summary=[]\n",
    "\n",
    "for i in range(len(cleaned_text)):\n",
    "    if(len(cleaned_summary[i].split())<=max_summary_len and len(cleaned_text[i].split())<=max_text_len):\n",
    "        short_text.append(cleaned_text[i])\n",
    "        short_summary.append(cleaned_summary[i])\n",
    "        \n",
    "df=pd.DataFrame({'text':short_text,'summary':short_summary})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "tR1uh8xSNUma"
   },
   "source": [
    "Remember to add the **START** and **END** special tokens at the beginning and end of the summary. Here, I have chosen **sostok** and **eostok** as START and END tokens\n",
    "\n",
    "**Note:** Be sure that the chosen special tokens never appear in the summary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "EwLUH78CFxhg"
   },
   "outputs": [],
   "source": [
    "df['summary'] = df['summary'].apply(lambda x : 'sostok '+ x + ' eostok')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1GlcX4RFOh13"
   },
   "source": [
    "We are getting closer to the model building part. Before that, we need to split our dataset into a training and validation set. We’ll use 90% of the dataset as the training data and evaluate the performance on the remaining 10% (holdout set):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "RakakKHcFxhl"
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "x_tr,x_val,y_tr,y_val=train_test_split(np.array(df['text']),np.array(df['summary']),test_size=0.1,random_state=0,shuffle=True) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Vq1mqyOHOtIl"
   },
   "source": [
    "# Preparing the Tokenizer\n",
    "\n",
    "A tokenizer builds the vocabulary and converts a word sequence to an integer sequence. Go ahead and build tokenizers for text and summary:\n",
    "\n",
    "# Text Tokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "oRHTgX6hFxhq"
   },
   "outputs": [],
   "source": [
    "from keras.preprocessing.text import Tokenizer \n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "\n",
    "#prepare a tokenizer for reviews on training data\n",
    "x_tokenizer = Tokenizer() \n",
    "x_tokenizer.fit_on_texts(list(x_tr))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "RzvLwYL_PDcx"
   },
   "source": [
    "# Rarewords and its Coverage\n",
    "\n",
    "Let us look at the proportion rare words and its total coverage in the entire text\n",
    "\n",
    "Here, I am defining the threshold to be 4 which means word whose count is below 4 is considered as a rare word"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 858,
     "status": "ok",
     "timestamp": 1576666059438,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "y8KronV2Fxhx",
    "outputId": "23bba19e-a365-47c4-f3ab-af6210c8d85d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "% of rare words in vocabulary: 57.214851943665046\n",
      "Total Coverage of rare words: 2.0147421871063895\n"
     ]
    }
   ],
   "source": [
    "thresh=4\n",
    "\n",
    "cnt=0\n",
    "tot_cnt=0\n",
    "freq=0\n",
    "tot_freq=0\n",
    "\n",
    "for key,value in x_tokenizer.word_counts.items():\n",
    "    tot_cnt=tot_cnt+1\n",
    "    tot_freq=tot_freq+value\n",
    "    if(value<thresh):\n",
    "        cnt=cnt+1\n",
    "        freq=freq+value\n",
    "    \n",
    "print(\"% of rare words in vocabulary:\",(cnt/tot_cnt)*100)\n",
    "print(\"Total Coverage of rare words:\",(freq/tot_freq)*100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "So-J-5kzQIeO"
   },
   "source": [
    "**Remember**:\n",
    "\n",
    "\n",
    "* **tot_cnt** gives the size of vocabulary (which means every unique words in the text)\n",
    " \n",
    "*   **cnt** gives me the no. of rare words whose count falls below threshold\n",
    "\n",
    "*  **tot_cnt - cnt** gives me the top most common words \n",
    "\n",
    "Let us define the tokenizer with top most common words for reviews."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "J2giEsF3Fxh3"
   },
   "outputs": [],
   "source": [
    "#prepare a tokenizer for reviews on training data\n",
    "x_tokenizer = Tokenizer(num_words=tot_cnt-cnt) \n",
    "x_tokenizer.fit_on_texts(list(x_tr))\n",
    "\n",
    "#convert text sequences into integer sequences\n",
    "x_tr_seq    =   x_tokenizer.texts_to_sequences(x_tr) \n",
    "x_val_seq   =   x_tokenizer.texts_to_sequences(x_val)\n",
    "\n",
    "#padding zero upto maximum length\n",
    "x_tr    =   pad_sequences(x_tr_seq,  maxlen=max_text_len, padding='post')\n",
    "x_val   =   pad_sequences(x_val_seq, maxlen=max_text_len, padding='post')\n",
    "\n",
    "#size of vocabulary ( +1 for padding token)\n",
    "x_voc   =  x_tokenizer.num_words + 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 6544,
     "status": "ok",
     "timestamp": 1576666088192,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "DCbGMsm4FxiA",
    "outputId": "db3f4c86-8f32-424e-c07f-1dd6d997dacc"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "31413"
      ]
     },
     "execution_count": 40,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_voc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "uQfKP3sqRxi9"
   },
   "source": [
    "# Summary Tokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "eRHqyBkBFxiJ"
   },
   "outputs": [],
   "source": [
    "#prepare a tokenizer for reviews on training data\n",
    "y_tokenizer = Tokenizer()   \n",
    "y_tokenizer.fit_on_texts(list(y_tr))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "KInA6O6ZSkJz"
   },
   "source": [
    "#Rarewords and its Coverage\n",
    "\n",
    "Let us look at the proportion rare words and its total coverage in the entire summary\n",
    "\n",
    "Here, I am defining the threshold to be 6 which means word whose count is below 6 is considered as a rare word"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 936,
     "status": "ok",
     "timestamp": 1576666092119,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "yzE5OiRLFxiM",
    "outputId": "649a3ca3-f169-434c-e129-8359043eb9a8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "% of rare words in vocabulary: 64.6257852610096\n",
      "Total Coverage of rare words: 3.87308024004095\n"
     ]
    }
   ],
   "source": [
    "thresh=6\n",
    "\n",
    "cnt=0\n",
    "tot_cnt=0\n",
    "freq=0\n",
    "tot_freq=0\n",
    "\n",
    "for key,value in y_tokenizer.word_counts.items():\n",
    "    tot_cnt=tot_cnt+1\n",
    "    tot_freq=tot_freq+value\n",
    "    if(value<thresh):\n",
    "        cnt=cnt+1\n",
    "        freq=freq+value\n",
    "    \n",
    "print(\"% of rare words in vocabulary:\",(cnt/tot_cnt)*100)\n",
    "print(\"Total Coverage of rare words:\",(freq/tot_freq)*100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0PBhzKuRSw_9"
   },
   "source": [
    "Let us define the tokenizer with top most common words for summary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "-fswLvIgFxiR"
   },
   "outputs": [],
   "source": [
    "#prepare a tokenizer for reviews on training data\n",
    "y_tokenizer = Tokenizer(num_words=tot_cnt-cnt) \n",
    "y_tokenizer.fit_on_texts(list(y_tr))\n",
    "\n",
    "#convert text sequences into integer sequences\n",
    "y_tr_seq    =   y_tokenizer.texts_to_sequences(y_tr) \n",
    "y_val_seq   =   y_tokenizer.texts_to_sequences(y_val) \n",
    "\n",
    "#padding zero upto maximum length\n",
    "y_tr    =   pad_sequences(y_tr_seq, maxlen=max_summary_len, padding='post')\n",
    "y_val   =   pad_sequences(y_val_seq, maxlen=max_summary_len, padding='post')\n",
    "\n",
    "#size of vocabulary\n",
    "y_voc  =   y_tokenizer.num_words +1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qqwDUT5oTFmn"
   },
   "source": [
    "Let us check whether word count of start token is equal to length of the training data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 3843,
     "status": "ok",
     "timestamp": 1576666098370,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "pR8IX9FRFxiY",
    "outputId": "7cd69898-1072-4124-c31e-7ceca43e081d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(88886, 88886)"
      ]
     },
     "execution_count": 44,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_tokenizer.word_counts['sostok'],len(y_tr)   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "LVFhFVguTTtw"
   },
   "source": [
    "Here, I am deleting the rows that contain only **START** and **END** tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "kZ-vW82sFxih"
   },
   "outputs": [],
   "source": [
    "ind=[]\n",
    "for i in range(len(y_tr)):\n",
    "    cnt=0\n",
    "    for j in y_tr[i]:\n",
    "        if j!=0:\n",
    "            cnt=cnt+1\n",
    "    if(cnt==2):\n",
    "        ind.append(i)\n",
    "\n",
    "y_tr=np.delete(y_tr,ind, axis=0)\n",
    "x_tr=np.delete(x_tr,ind, axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "cx5NISuMFxik"
   },
   "outputs": [],
   "source": [
    "ind=[]\n",
    "for i in range(len(y_val)):\n",
    "    cnt=0\n",
    "    for j in y_val[i]:\n",
    "        if j!=0:\n",
    "            cnt=cnt+1\n",
    "    if(cnt==2):\n",
    "        ind.append(i)\n",
    "\n",
    "y_val=np.delete(y_val,ind, axis=0)\n",
    "x_val=np.delete(x_val,ind, axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "wOtlDcthFxip"
   },
   "source": [
    "# Model building\n",
    "\n",
    "We are finally at the model building part. But before we do that, we need to familiarize ourselves with a few terms which are required prior to building the model.\n",
    "\n",
    "**Return Sequences = True**: When the return sequences parameter is set to True, LSTM produces the hidden state and cell state for every timestep\n",
    "\n",
    "**Return State = True**: When return state = True, LSTM produces the hidden state and cell state of the last timestep only\n",
    "\n",
    "**Initial State**: This is used to initialize the internal states of the LSTM for the first timestep\n",
    "\n",
    "**Stacked LSTM**: Stacked LSTM has multiple layers of LSTM stacked on top of each other. \n",
    "This leads to a better representation of the sequence. I encourage you to experiment with the multiple layers of the LSTM stacked on top of each other (it’s a great way to learn this)\n",
    "\n",
    "Here, we are building a 3 stacked LSTM for the encoder:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![topic_modeling](images/final.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![topic_modeling](images/enc_dec.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 595
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 3441,
     "status": "ok",
     "timestamp": 1576666286470,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "zXef38nBFxir",
    "outputId": "561265de-1349-4b7a-c6a3-1342494bda0f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model\"\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "input_1 (InputLayer)            [(None, 100)]        0                                            \n",
      "__________________________________________________________________________________________________\n",
      "embedding (Embedding)           (None, 100, 100)     3141300     input_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "lstm (LSTM)                     [(None, 100, 300), ( 481200      embedding[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "input_2 (InputLayer)            [(None, None)]       0                                            \n",
      "__________________________________________________________________________________________________\n",
      "lstm_1 (LSTM)                   [(None, 100, 300), ( 721200      lstm[0][0]                       \n",
      "__________________________________________________________________________________________________\n",
      "embedding_1 (Embedding)         (None, None, 100)    1109400     input_2[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "lstm_2 (LSTM)                   [(None, 100, 300), ( 721200      lstm_1[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "lstm_3 (LSTM)                   [(None, None, 300),  481200      embedding_1[0][0]                \n",
      "                                                                 lstm_2[0][1]                     \n",
      "                                                                 lstm_2[0][2]                     \n",
      "__________________________________________________________________________________________________\n",
      "attention_layer (AttentionLayer ((None, None, 300),  180300      lstm_2[0][0]                     \n",
      "                                                                 lstm_3[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "concat_layer (Concatenate)      (None, None, 600)    0           lstm_3[0][0]                     \n",
      "                                                                 attention_layer[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "time_distributed (TimeDistribut (None, None, 11094)  6667494     concat_layer[0][0]               \n",
      "==================================================================================================\n",
      "Total params: 13,503,294\n",
      "Trainable params: 13,503,294\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "from keras import backend as K \n",
    "K.clear_session()\n",
    "\n",
    "latent_dim = 300\n",
    "embedding_dim=100\n",
    "\n",
    "# Encoder\n",
    "encoder_inputs = Input(shape=(max_text_len,))\n",
    "\n",
    "#embedding layer\n",
    "enc_emb =  Embedding(x_voc, embedding_dim,trainable=True)(encoder_inputs)\n",
    "\n",
    "#encoder lstm 1\n",
    "encoder_lstm1 = LSTM(latent_dim,return_sequences=True,return_state=True,dropout=0.4,recurrent_dropout=0.4)\n",
    "encoder_output1, state_h1, state_c1 = encoder_lstm1(enc_emb)\n",
    "\n",
    "#encoder lstm 2\n",
    "encoder_lstm2 = LSTM(latent_dim,return_sequences=True,return_state=True,dropout=0.4,recurrent_dropout=0.4)\n",
    "encoder_output2, state_h2, state_c2 = encoder_lstm2(encoder_output1)\n",
    "\n",
    "#encoder lstm 3\n",
    "encoder_lstm3=LSTM(latent_dim, return_state=True, return_sequences=True,dropout=0.4,recurrent_dropout=0.4)\n",
    "encoder_outputs, state_h, state_c= encoder_lstm3(encoder_output2)\n",
    "\n",
    "# Set up the decoder, using `encoder_states` as initial state.\n",
    "decoder_inputs = Input(shape=(None,))\n",
    "\n",
    "#embedding layer\n",
    "dec_emb_layer = Embedding(y_voc, embedding_dim,trainable=True)\n",
    "dec_emb = dec_emb_layer(decoder_inputs)\n",
    "\n",
    "decoder_lstm = LSTM(latent_dim, return_sequences=True, return_state=True,dropout=0.4,recurrent_dropout=0.2)\n",
    "decoder_outputs,decoder_fwd_state, decoder_back_state = decoder_lstm(dec_emb,initial_state=[state_h, state_c])\n",
    "\n",
    "# Attention layer\n",
    "attn_layer = AttentionLayer(name='attention_layer')\n",
    "attn_out, attn_states = attn_layer([encoder_outputs, decoder_outputs])\n",
    "\n",
    "# Concat attention input and decoder LSTM output\n",
    "decoder_concat_input = Concatenate(axis=-1, name='concat_layer')([decoder_outputs, attn_out])\n",
    "\n",
    "#dense layer\n",
    "decoder_dense =  TimeDistributed(Dense(y_voc, activation='softmax'))\n",
    "decoder_outputs = decoder_dense(decoder_concat_input)\n",
    "\n",
    "# Define the model \n",
    "model = Model([encoder_inputs, decoder_inputs], decoder_outputs)\n",
    "\n",
    "model.summary() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0ZVlfRuMUcoP"
   },
   "source": [
    "I am using sparse categorical cross-entropy as the loss function since it converts the integer sequence to a one-hot vector on the fly. This overcomes any memory issues."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Lwfi1Fm8Fxiz"
   },
   "outputs": [],
   "source": [
    "model.compile(optimizer='rmsprop', loss='sparse_categorical_crossentropy')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "p0ykDbxfUhyw"
   },
   "source": [
    "Remember the concept of early stopping? It is used to stop training the neural network at the right time by monitoring a user-specified metric. Here, I am monitoring the validation loss (val_loss). Our model will stop training once the validation loss increases:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "s-A3J92MUljB"
   },
   "outputs": [],
   "source": [
    "es = EarlyStopping(monitor='val_loss', mode='min', verbose=1,patience=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Mw6CVECaUq5b"
   },
   "source": [
    "We’ll train the model on a batch size of 128 and validate it on the holdout set (which is 10% of our dataset):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 3895632,
     "status": "ok",
     "timestamp": 1576677091481,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "ETnPzA4OFxi3",
    "outputId": "2ccef67e-87c3-45ed-c27b-f4d578ce4628"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 88886 samples, validate on 9877 samples\n",
      "Epoch 1/30\n",
      "88886/88886 [==============================] - 346s 4ms/sample - loss: 4.0532 - val_loss: 3.8773\n",
      "Epoch 2/30\n",
      "88886/88886 [==============================] - 347s 4ms/sample - loss: 3.8026 - val_loss: 3.6762\n",
      "Epoch 3/30\n",
      "88886/88886 [==============================] - 350s 4ms/sample - loss: 3.6253 - val_loss: 3.5310\n",
      "Epoch 4/30\n",
      "88886/88886 [==============================] - 350s 4ms/sample - loss: 3.4900 - val_loss: 3.4157\n",
      "Epoch 5/30\n",
      "88886/88886 [==============================] - 355s 4ms/sample - loss: 3.3755 - val_loss: 3.3281\n",
      "Epoch 6/30\n",
      "88886/88886 [==============================] - 356s 4ms/sample - loss: 3.2833 - val_loss: 3.2552\n",
      "Epoch 7/30\n",
      "88886/88886 [==============================] - 359s 4ms/sample - loss: 3.2018 - val_loss: 3.1989\n",
      "Epoch 8/30\n",
      "88886/88886 [==============================] - 361s 4ms/sample - loss: 3.1368 - val_loss: 3.1499\n",
      "Epoch 9/30\n",
      "88886/88886 [==============================] - 367s 4ms/sample - loss: 3.0776 - val_loss: 3.1046\n",
      "Epoch 10/30\n",
      "88886/88886 [==============================] - 363s 4ms/sample - loss: 3.0265 - val_loss: 3.0740\n",
      "Epoch 11/30\n",
      "88886/88886 [==============================] - 355s 4ms/sample - loss: 2.9824 - val_loss: 3.0559\n",
      "Epoch 12/30\n",
      "88886/88886 [==============================] - 347s 4ms/sample - loss: 2.9406 - val_loss: 3.0200\n",
      "Epoch 13/30\n",
      "88886/88886 [==============================] - 345s 4ms/sample - loss: 2.9037 - val_loss: 2.9937\n",
      "Epoch 14/30\n",
      "88886/88886 [==============================] - 346s 4ms/sample - loss: 2.8708 - val_loss: 2.9738\n",
      "Epoch 15/30\n",
      "88886/88886 [==============================] - 347s 4ms/sample - loss: 2.8391 - val_loss: 2.9727\n",
      "Epoch 16/30\n",
      "88886/88886 [==============================] - 346s 4ms/sample - loss: 2.8125 - val_loss: 2.9371\n",
      "Epoch 17/30\n",
      "88886/88886 [==============================] - 346s 4ms/sample - loss: 2.7855 - val_loss: 2.9303\n",
      "Epoch 18/30\n",
      "88886/88886 [==============================] - 346s 4ms/sample - loss: 2.7583 - val_loss: 2.9096\n",
      "Epoch 19/30\n",
      "88886/88886 [==============================] - 355s 4ms/sample - loss: 2.7284 - val_loss: 2.8948\n",
      "Epoch 20/30\n",
      "88886/88886 [==============================] - 353s 4ms/sample - loss: 2.6997 - val_loss: 2.8768\n",
      "Epoch 21/30\n",
      "88886/88886 [==============================] - 348s 4ms/sample - loss: 2.6741 - val_loss: 2.8650\n",
      "Epoch 22/30\n",
      "88886/88886 [==============================] - 347s 4ms/sample - loss: 2.6556 - val_loss: 2.8557\n",
      "Epoch 23/30\n",
      "88886/88886 [==============================] - 345s 4ms/sample - loss: 2.6342 - val_loss: 2.8413\n",
      "Epoch 24/30\n",
      "88886/88886 [==============================] - 345s 4ms/sample - loss: 2.6075 - val_loss: 2.8426\n",
      "Epoch 25/30\n",
      "88886/88886 [==============================] - 346s 4ms/sample - loss: 2.5861 - val_loss: 2.8162\n",
      "Epoch 26/30\n",
      "88886/88886 [==============================] - 344s 4ms/sample - loss: 2.5687 - val_loss: 2.7975\n",
      "Epoch 27/30\n",
      "88886/88886 [==============================] - 345s 4ms/sample - loss: 2.5479 - val_loss: 2.7985\n",
      "Epoch 28/30\n",
      "88886/88886 [==============================] - 345s 4ms/sample - loss: 2.5304 - val_loss: 2.7853\n",
      "Epoch 29/30\n",
      "88886/88886 [==============================] - 347s 4ms/sample - loss: 2.5115 - val_loss: 2.7912\n",
      "Epoch 30/30\n",
      "88886/88886 [==============================] - 349s 4ms/sample - loss: 2.4951 - val_loss: 2.7817\n"
     ]
    }
   ],
   "source": [
    "history=model.fit([x_tr,y_tr[:,:-1]], y_tr.reshape(y_tr.shape[0],y_tr.shape[1], 1)[:,1:] ,epochs=30,callbacks=[es],batch_size=128, validation_data=([x_val,y_val[:,:-1]], y_val.reshape(y_val.shape[0],y_val.shape[1], 1)[:,1:]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "2Twl3sGv44Be"
   },
   "outputs": [],
   "source": [
    "model.save(project_path+\"summary.hdf5\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0ezKYOp2UxG5"
   },
   "source": [
    "# Understanding the Diagnostic plot\n",
    "\n",
    "Now, we will plot a few diagnostic plots to understand the behavior of the model over time:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 265
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1445,
     "status": "ok",
     "timestamp": 1576677136905,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "tDTNLAURFxjE",
    "outputId": "99a54a76-e1e0-4923-ced5-6e62ae71e3d5"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXUAAAD4CAYAAAATpHZ6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3deXhV1b3/8fc3JzNknslAIEwJCWOC\nKKIMogyKWi3S1lZbFf31Xqu1etXWarW1V9vqY3tttThUrXXAWREEakFEEAhzmGcykHki87R+f+wD\nJCGBACecnJPv63ny5GSvfXa+m/PwYbP22muJMQallFLuwcPZBSillHIcDXWllHIjGupKKeVGNNSV\nUsqNaKgrpZQb8XTWLw4PDzeJiYnO+vVKKeWSNmzYUGyMieis3WmhnpiYSGZmprN+vVJKuSQROXy6\ndu1+UUopN6KhrpRSbkRDXSml3IjT+tSVUupcNDY2kpOTQ11dnbNL6Va+vr7ExcXh5eV1Vu/TUFdK\nuZScnBwCAgJITExERJxdTrcwxlBSUkJOTg4DBgw4q/dq94tSyqXU1dURFhbmtoEOICKEhYWd0/9G\nNNSVUi7HnQP9uHM9R5cL9b0Fx3jisx3UNzU7uxSllOpxXC7Uc8pqefWbg6zeV+LsUpRSvVB5eTl/\n+9vfzvp9M2fOpLy8vBsqasvlQn3CoHACfD35fNtRZ5eilOqFOgv1pqam075v0aJFBAcHd1dZJ7hc\nqHt7ejAtJYql2/NpaGpxdjlKqV7moYceYv/+/YwaNYqMjAwmTpzI7NmzSUlJAeC6665j7NixDB8+\nnPnz5594X2JiIsXFxRw6dIjk5GTuuOMOhg8fzpVXXkltba3D6nPJIY2z0mL4cGMuq/cXM2lopLPL\nUUo5yeOfbWdHXqVDj5nSL5DHrhneaftTTz1FVlYWmzdvZsWKFcyaNYusrKwTQw9fffVVQkNDqa2t\nJSMjgxtuuIGwsLA2x9i7dy9vv/02L730EnPmzOGDDz7g5ptvdkj9Xb5SFxGbiGwSkYUdtPmIyLsi\nsk9E1opIokOq68Slg8MJ8PFkkXbBKKWcbNy4cW3Gkv/lL39h5MiRjB8/nuzsbPbu3XvKewYMGMCo\nUaMAGDt2LIcOHXJYPWdzpX4PsBMI7KDtNqDMGDNIROYCTwM3OaC+Dvl42rgiJYqlOwp4srkFL5vL\n9SIppRzgdFfUF0qfPn1OvF6xYgX//ve/WbNmDf7+/kyaNKnDseY+Pj4nXttsNod2v3QpDUUkDpgF\nvNzJLtcCr9tfvw9MlW4eSDojNZrymkbW7NdRMEqpCycgIIBjx4512FZRUUFISAj+/v7s2rWLb7/9\n9gJX1/Ur9eeA/wECOmmPBbIBjDFNIlIBhAHFrXcSkXnAPICEhIRzqfeEy4ZE0MfbxuKso1w2pNP5\n4pVSyqHCwsKYMGECqamp+Pn5ERUVdaJt+vTpvPjiiyQnJzN06FDGjx9/wes7Y6iLyNVAoTFmg4hM\nOp9fZoyZD8wHSE9PN+dzLF8vG1OTo1iyvYDfXtuCp3bBKKUukLfeeqvD7T4+PixevLjDtuP95uHh\n4WRlZZ3Yfv/99zu0tq4k4QRgtogcAt4BpojIm+32yQXiAUTEEwgCur1fZGZaDKXVDaw9WNrdv0op\npVzCGUPdGPOwMSbOGJMIzAX+Y4xpP/bmU+AW++sb7fuc15V4V0waGoG/t00fRFJKKbtz7rMQkSdE\nZLb9x1eAMBHZB9wHPOSI4s7E18vGlGGRLMnKp7ml2/8NUUqpHu+sHj4yxqwAVthfP9pqex3wXUcW\n1lUz02JYuPUoaw+WcElSuDNKUEqpHsPl7y5OHhqJn5eNxdvynV2KUko5ncuHup+3jcnDIlisXTBK\nKeX6oQ5WF0xxVT2Zh3QUjFKqe53r1LsAzz33HDU1NQ6uqC23CPXJQyPx9fLQuWCUUt2up4e6S87S\n2F4fH08mDYlkcVY+j10zHA8P91/qSinlHK2n3p02bRqRkZEsWLCA+vp6rr/+eh5//HGqq6uZM2cO\nOTk5NDc38+tf/5qCggLy8vKYPHky4eHhLF++vFvqc4tQB5g5IoYvtuez4UgZGYmhzi5HKXUhLH4I\n8rc59pjRaTDjqU6bW0+9u3TpUt5//33WrVuHMYbZs2ezcuVKioqK6NevH59//jlgzQkTFBTEs88+\ny/LlywkP776Rem7R/QIwZVgk3p4efL5Vu2CUUhfG0qVLWbp0KaNHj2bMmDHs2rWLvXv3kpaWxrJl\ny3jwwQf5+uuvCQoKumA1uc2Vel8fTyYNieCLrHwevTpFu2CU6g1Oc0V9IRhjePjhh7nzzjtPadu4\ncSOLFi3ikUceYerUqTz66KMdHMHxXO9KvaUZDn7dYdPMtBjyK+vYlF12gYtSSvUWrafeveqqq3j1\n1VepqqoCIDc3l8LCQvLy8vD39+fmm2/mgQceYOPGjae8t7u43pX6pjfhs5/BHf+B2LFtmqYmR+Jt\n82DRtnzG9td+daWU47WeenfGjBl8//vf5+KLLwagb9++vPnmm+zbt48HHngADw8PvLy8eOGFFwCY\nN28e06dPp1+/ft12o1QuwLxbHUpPTzeZmZln/8a6CnhmGAz/Dlz311Oab399PTvyKln14BTtglHK\nDe3cuZPk5GRnl3FBdHSuIrLBGJPe2Xtcr/vFNwhGzIGs96Hm1IeNZqbFkFdRx5accicUp5RSzuV6\noQ6Qfhs01cGWt09pmpochZdN9EEkpVSv5JqhHjMC4jIg81Vo130U5OfFxMERLNqWj7O6lpRS3as3\n/N0+13N0zVAHyLgdSvbBwa9OaZqRGk1ueS1bcyqcUJhSqjv5+vpSUlLi1sFujKGkpARfX9+zfq/r\njX45LuU6+OJhWP8yDJzUpunKlGge9tjGoqyjjIwPdkp5SqnuERcXR05ODkVFRc4upVv5+voSFxd3\n1u/rysLTvsBKwMe+//vGmMfa7ZMAvA4EAzbgIWPMorOu5mx4+cLoH8Cav0FlHgT2O9EU5O/FhEHh\nLNp2lIemD0NER8Eo5S68vLwYMGCAs8vosbrS/VIPTDHGjARGAdNFZHy7fR4BFhhjRmOtY3puU5id\nrfSfgGmGjW+c0jQrLYbs0lq251VekFKUUqon6MrC08YYU2X/0cv+1b4zywCB9tdBQJ7DKjyd0IGQ\nNBU2vAbNjW2apqVEYfMQFupcMEqpXqRLN0pFxCYim4FCYJkxZm27XX4D3CwiOcAi4O5OjjNPRDJF\nJNNh/WEZt8Gxo7B7cZvNIX28mTw0knfXH6Gmockxv0sppXq4LoW6MabZGDMKiAPGiUhqu12+B7xm\njIkDZgL/FJFTjm2MmW+MSTfGpEdERJxv7ZbBV0FgLGS+ckrTnZcPpKymkQXrsx3zu5RSqoc7qyGN\nxphyYDkwvV3TbcAC+z5rAF+g+yYMbs3mCWN/DAdWQPG+Nk0ZiaGk9w/hpa8P0tjcckHKUUopZzpj\nqItIhIgE21/7AdOAXe12OwJMte+TjBXqF2680ZgfgYen9TBSO3ddnkRueS0Lt16Ybn6llHKmrlyp\nxwDLRWQrsB6rT32hiDwhIrPt+/wCuENEtgBvA7eaC/lkQEAUDLsaNv8LGmvbNE0ZFsmQqL78/asD\nbv2wglJKQddGv2w1xow2xowwxqQaY56wb3/UGPOp/fUOY8wEY8xIY8woY8zS7i78FBm3Q105ZH3Y\nZrOHh3DnZUnsyj/Git3u/bCCUkq57jQB7SVeCuFDrSdM25k9qh/9gnx5YcV+JxSmlFIXjvuEuoj1\nMFLeRsjb1KbJy+bB7RMHsu5QKRsO66pISin35T6hDjByLnj5w/pThzfOHRdPsL8XL36lV+tKKffl\nXqHuFwxpN8K296G27RW5v7cnt1ycyLIdBewt6N41ApVSylncK9TBvoBGLWx555SmWy5JxNfLg7+v\nPOCEwpRSqvu5X6j3G2UtSN3BAhqhfbyZm5HAJ5tzySuv7eQASinlutwv1MEa3li8Bw59fUrT7RMH\n0GLglVUHnVCYUkp1L/cM9eHXg29wh8Mb40L8mT2yH2+vO0J5TYMTilNKqe7jnqHu5Qejb4Zdn8Ox\n/FOa77x8IDUNzbyx5rATilNKqe7jnqEO1pj1liZrrvV2hkUHMmVYJK+tPkRtQ/OFr00ppbqJ+4Z6\nWBIMmQ5r/grVxac033V5EqXVDby3QaflVUq5D/cNdYBpT0BDNSx/8pSmjMQQxvYPYf7KAzTptLxK\nKTfh3qEeMdQaCbPhNSjY3qZJRLjr8iRyymr5fJsueaeUcg/uHeoAkx4Cn0BY8stTxq1PHRbJ4Mi+\nvLBiv07Lq5RyC+4f6v6hMOlha2WkPUvaNHl4CHdebp+Wd49Oy6uUcn3uH+pgLU4dNhiW/gqa2o5N\nnz2yHzFBvryo0/IqpdxA7wh1mxdc9SSU7DvlgSRvT2ta3rUHS1m199RRMkop5Uq6skapr4isE5Et\nIrJdRB7vZL85IrLDvs9bji/1PA2+EpKmwFdPQXVJm6YfXJRAYpg/j36SRX2TjltXSrmurlyp1wNT\njDEjgVHAdBEZ33oHERkMPAxMMMYMB+51eKXnSwSu+j3UH4MV/9umydfLxhPXpnKguJr5X+kMjkop\n19WVNUqNMabK/qOX/av9UJE7gL8aY8rs7yl0aJWOEplsPWma+SoU7mrTdNmQCGalxfD88n0cKalx\nUoFKKXV+utSnLiI2EdkMFALLjDFr2+0yBBgiIt+IyLciMr2T48wTkUwRySwqctJok0m/BO++1k3T\ndn59dQqeHsJjn2bpEEellEvqUqgbY5qNMaOAOGCciKS228UTGAxMAr4HvCQiwR0cZ74xJt0Ykx4R\nEXF+lZ+rPmEw6UHY92/Yu6xNU3SQLz+fNoTlu4tYsv3UicCUUqqnO6vRL8aYcmA50P5KPAf41BjT\naIw5COzBCvmeKeMOCE2yHkhqbmzTdOsliQyLDuDxz3ZQXd/kpAKVUurcdGX0S8Txq24R8QOmAbva\n7fYx1lU6IhKO1R3Tc+84enrDlb+zFtLIfLVtk82DJ69P5WhFHX/+cq+TClRKqXPTlSv1GGC5iGwF\n1mP1qS8UkSdEZLZ9nyVAiYjswLqSf8AYU9LJ8XqGoTNgwOWw/PdQU9qmaWz/UOZmxPPKqoPsztdF\nqpVSrkOcdUMwPT3dZGZmOuV3n5CfBX+fCOPuhBlPtWkqq25gyjMrGBTZl3fnXYyHhzipSKWUOklE\nNhhj0jtr7x1PlHYmOhXG3ALrX4KiPW2aQvp48/CMZNYfKuODjTlOKlAppc5O7w51gMm/Ai9/WPrI\nKU03jo1jbP8Q/nfxLsqqdT1TpVTPp6HeNwIuux/2LoGsD9s0eXgIv7sulYraRv6wZLeTClRKqa7T\nUAcY/1OITYfP7oXyI22akmMC+fEliby97ggbj5Q5qUCllOoaDXWwZnG84SUwzfDhndDSdlKve6cN\nITrQl0c+ytKl75RSPZqG+nGhA2HWM3BkNXz9bJumvj6ePHpNCjuOVvLGmsNOKlAppc5MQ721ETdB\n2netWRyz17VpmpEazeVDInh22R4KKuucVKBSSp2ehnprItbVelAsfHA71FW2ahIenz2chuYWfv2x\nTvillOqZNNTb8w2C77wMFTmw6P42TYnhfXjgyqEs3VHAW+uOdHIApZRyHg31jiRcBJc/CFvfhS3v\ntmm67dIBTBwczm8X7mBvgU4hoJTqWTTUOzPxF5BwMXz+Cyg9eGKzh4fwzJyR9PH25O63N1HXqMvf\nKaV6Dg31ztg84TvzQTzgwzvaTNEbGeDLn747kl35x3hqcfsJK5VSynk01E8nOAGufhZy1sNXf2jT\nNHlYJD+ZMIDXVh/iy50FTipQKaXa0lA/k7QbYdQP4Os/waFv2jQ9OGMoKTGBPPD+Vgp1mKNSqgfQ\nUO+KGU9DcH/4cB7UnpwqwMfTxl++N5rahmZ+vmAzLS06zFEp5Vwa6l3hEwA3vAJV+bDw59BqjPqg\nyL48dk0K3+wr4e8re+5iT0qp3qEry9n5isg6EdkiIttF5PHT7HuDiBgR6XQCd5cVN9aapnf7R7Dx\n9TZNN2XEMzMtmmeW7mZzdrmTClRKqa5dqdcDU4wxI4FRwHQRGd9+JxEJAO4B1jq2xB5kwj2QNAUW\n3ge7Pj+xWUT43+tHEBXoyz3vbKJKF6xWSjnJGUPdWKrsP3rZvzrqPP4t8DTgvncMPWww5w3oNxre\nuxX2Lz/RFOTvxXNzR5FdWsOjH2c5r0alVK/WpT51EbGJyGagEGvh6bXt2scA8caYzzs8wMn95olI\npohkFhUVnXPRTuUTAD94D8KHwDvfhyMn/ygyEkO5e8pgPtyUy8ebcp1YpFKqt+pSqBtjmo0xo4A4\nYJyIpB5vExEP4FngF104znxjTLoxJj0iIuJca3Y+/1D44UcQEAP/+i4c3XKi6e4pg0jvH8IjH2dx\npKTGiUUqpXqjsxr9YowpB5YD01ttDgBSgRUicggYD3zqljdLW+sbCT/6xLpy/+d3Tixc7Wnz4Lm5\no/AQuPudTTQ06aIaSqkLpyujXyJEJNj+2g+YBpx4Nt4YU2GMCTfGJBpjEoFvgdnGmMxuqrnnCI6H\nWz61phJ441oosxbQiAvx5+kbRrAlu5z739ui49eVUhdMV67UY4DlIrIVWI/Vp75QRJ4QkdndW54L\nCEuyumIaq61gP5YPwIy0GP5n+lA+3ZLH7xftdHKRSqnewvNMOxhjtgKjO9j+aCf7Tzr/slxMdCrc\n/CG8PhveuA5+vAj8Q/l/lydRWFnPy6sOEhXoyx2XDXR2pUopN6dPlDpKXDp8/x0oPQBvfgfqKhER\nfn11CrPSYnhy0U4+2awjYpRS3UtD3ZEGXGaNY8/fBm/PhYYabPb518cPDOX+97bw9V4XHcqplHIJ\nGuqONnS6NQ/74dWw4EfQWIevl435P0onKaIvd/1zA1m5Fc6uUinlpjTUu0PqDXDNn2HfMvjndVBT\nSqCvF6//ZBzB/t7c+o91HC6pdnaVSik3pKHeXcbeAje+Crkb4JUroewQUYG+vP6TcTS1GG55dR3F\nVfXOrlIp5WY01LtT6g3ww4+huhBengZ5mxgU2ZdXbskgv7KOn7y2nmqd/Esp5UAa6t0tcQLctgw8\nfeEfs2DvMsb2D+H5740hK7eCn/5rI43N+tSpUsoxNNQvhIihcPsy60Glt26CDa9zRUoUv78+ja/2\nFPHgB1sxRp86VUqdvzM+fKQcJCDaeihpwS3w2c+gMpe5kx6m8Fg9zy7bQ6i/N7+alYyIOLtSpZQL\n01C/kHwC4PvvwsJ74aunoSKHu69+jtLqBl5edRADPKLBrpQ6DxrqF5rNC2Y/D0HxsOJ/kco8Hptj\nLY/3yqqDNLcYHrsmRYNdKXVONNSdQQQmPQRBcfDpz5B/zOKx77+LzWPAiWB/fPZwPDw02JVSZ0dD\n3ZlG32z1tS+4BXnxUh6Z9SyeHsP4+8oDNBvD765N1WBXSp0VHf3ibIOugDv+AyH9kfdv5aGaP3Hf\npeG8tfYID3+4TediV0qdFb1S7wkihsJt/4ZVzyJfPc3d/quIH/0gP8+EZmN4+oYR2PSKXSnVBXql\n3lPYPOHy/4E7/oP4h3L9zp/zWf93WbxhLw+8t4VmvWJXSnVBV5az8xWRdSKyRUS2i8jjHexzn4js\nEJGtIvKliPTvnnJ7gZiRMG8FTLiXtMLPWB30a/K2LOO+BZtp0idPlVJn0JUr9XpgijFmJDAKmC4i\n49vtswlIN8aMAN4H/uDYMnsZTx+Y9jj8+AuC+vjyjvfvGJn1FPe/vVanFFBKndYZQ91Yquw/etm/\nTLt9lhtjauw/fgvEObTK3irhIrhrFYybx088v+DuPT/mmX+8RV1js7MrU0r1UF3qUxcRm4hsBgqx\nFp5ee5rdbwMWd3KceSKSKSKZRUW6AlCXePeBmX+EH35MlJ/hgez/5stnfkhefp6zK1NK9UBdCnVj\nTLMxZhTWFfg4EUntaD8RuRlIB/7YyXHmG2PSjTHpERER51pz75Q0mb4/X0fO4JuZXrsIvxfHsXfx\n89CiV+1KqZPOavSLMaYcWA5Mb98mIlcAvwJmG2N09Yfu4BtE/5uf5+jcJeTY4hm89lcUPjsBk73O\n2ZUppXqIrox+iRCRYPtrP2AasKvdPqOBv2MFemF3FKpOiku+iIEPrOTV6EdoOZaPvDKNxg/ugir9\no1eqt+vKlXoMsFxEtgLrsfrUF4rIEyIy277PH4G+wHsisllEPu2mepVdH18vfnzn/Sy+fCEvNs+G\nbe/R/JcxsOZv0Nzo7PKUUk4izlqcIT093WRmZjrld7ubNftL+OO/PuPnzf9gomyGiGSY8TQMvNzZ\npSmlHExENhhj0jtr1ydK3cDFSWH89Z6b+FPEk9ze8AvKKivgjdnw7s1QvM/Z5SmlLiANdTcRE+TH\ngrsuJjLjesZX/J73An6E2fcl/O0i+Px+qNIhpEr1BhrqbsTH08bvr0/jtzem86uymcww/8fRpJsg\n81X4y2hY+UdoqDnzgZRSLktD3Q3NSY/no59eQqNfOBdvm8ULw/9F84DL4T+/g/8bAxvf0PHtSrkp\nDXU3NbxfEAvvnsj3L0rg6cwWZhfdRc71H1vL6H16N7wwAfYsASfdKFdKdQ8NdTfm5211x/z9h2PJ\nK6/livfr+FfqS5jvvg7N9fDWHHj9Gsjd4OxSlVIOoqHeC1w1PJov7r2M9P6h/Orj7dy5IY6yW1fB\nzD9B4U54aQrMnwRr50N1ibPLVUqdBx2n3ou0tBheWXWQPyzZRWgfb56dM4oJcd6w6U3Y8hbkbwMP\nLxhyFYz8Hgy+Ejy9nV22UqqVM41T11DvhbJyK7jnnU0cKK5m3sSB/OLKoXh7ekB+Fmx5G7YugOpC\n8AuFtButgO83GkSX1FPK2TTUVYdqG5r57ec7eGvtEYb3C+TpG0aQGhtkNTY3wf7/WFfvuxZZ/e8R\nw6xwHzoTwgaBh/bcKeUMGurqtJZsz+dXH22jtLqBWy8ZwH1XDqGvT6v1yGvLYPvH1hV8tn0afd9g\niEuHuAzre+xY8Atxzgko1ctoqKszqqhp5A9LdvHWuiNEBfjy2DUpTE+NRtp3t5QegEPfQM56yMmE\nwh2cWAQrfMjJkI/LsOafsXme8ruUUudHQ1112cYjZfzqoyx2Hq1k8tAInrg2lfhQ/87fUH8Mcjee\nDPmc9VBTbLX5BMHom2HcHRA64MKcgFK9gIa6OitNzS28tvoQzy7bQ4sx/GzqYG6/dKB1I/VMjIGy\nQ1bA71kMOz6xnlwdOhPG3wWJE/Vmq1LnSUNdnZO88lqe+GwHX2zPZ3BkX568Po1xA0LP7iCVebD+\nFWvumdpSiEqFi+6EtO+Cl1/3FK6Um9NQV+fly50FPPrJdnLLa/nu2DgemjGMsL4+Z3eQxlrY9h58\n+yIUbreGSqb/GDJuh8B+3VO4Um5KQ12dt5qGJv7y5T5e/voAvl425l02kNsuHUAfn7O8EWoMHPra\nCvfdi8DDBinXwpgfQf8JYPPqnhNQyo2cd6iLiC+wEvABPIH3jTGPtdvHB3gDGAuUADcZYw6d7rga\n6q5nX+Ex/vDFbpbuKCC8rw8/mzqIuRkJXetvb6/0AKx7yXqatb7SGhI5ZAYkXw1JU7R7RqlOOCLU\nBehjjKkSES9gFXCPMebbVvv8FBhhjLlLROYC1xtjbjrdcTXUXdfGI2U8vXgXaw+WkhDqzy+uHMI1\nI/rh4XEON0EbamD/l7BzoXVzta4CvPxh0BWQfI01VYFfsONPQikX5dDuFxHxxwr1/2eMWdtq+xLg\nN8aYNSLiCeQDEeY0B9dQd23GGL7aU8TTX+xm59FKkmMC+Z/pQ5k0JOLU8e1d1dxodc/sXAi7Poeq\nfGsumgGXWVfwQ2dBQJRjT0QpF+OQUBcRG7ABGAT81RjzYLv2LGC6MSbH/vN+4CJjTHG7/eYB8wAS\nEhLGHj58+CxPR/U0LS2Gz7bm8czSPRwprWHcgFAemjGMMQnn+YRpS4s1JfDOT2HXQqu7BqyHnPpf\nAgmXWN+D48//JJRyIY6+Ug8GPgLuNsZktdrepVBvTa/U3UtDUwvvrD/CX77cS3FVA9NSorhv2hCS\nYwLP/+DGWFME710Ch9fAkW+hvsJqC4q3h/zF1s3W8ME6Fl65NYePfhGRR4EaY8yfWm3T7hcFQHV9\nE6+uOsj8lQc4Vt/EzLRo7pk6hKHRAY77JS3NULAdjqyBw99YQV9daLX5h0PCeCvsPWz2L08Q28mf\nxb7Nw2bNY5N8Dfg64B8fpS4AR9wojQAajTHlIuIHLAWeNsYsbLXPfwFprW6UfscYM+d0x9VQd28V\nNY28vOoA//jmENUNTcxMi+HeqYMZHOXAcD/OGCjZD0dWw+HVVtjXlFrh39IExv69M959YdQPYNw8\nCB/k+PqUciBHhPoI4HXAhrVS0gJjzBMi8gSQaYz51D7s8Z/AaKAUmGuMOXC642qo9w5l1Q28vOoA\nr31ziJrGZq4Z0Y+fTR3MoMi+F76YlpaTAd/SbL0u3gfr5kPWB9DSCIOmwUV3WcMqdXph1QPpw0eq\nRyitbmD+ygO8vvoQ9U3NXDsqlp9NHcyA8D7OLs1yrAA2vAaZr0BVAYQNtqY0GDkXfLrhfxdKnSMN\nddWjFFfVM3/lAd5Yc4iGphauGx3Lf08exMAIJ1y5d6SpwZqIbO0L1ugbn8BWs00OdHZ1Smmoq56p\n6Fg9f/9qP//89jANzS1MHRbJ7RMHctGA0HMf5+5oOZmw9kXY/pHVXRPS37oR28f+5d/u+/HXfSPB\n8yznx1GqizTUVY9WdKyef357mDe/PUxpdQOpsYHcMXEgM9Ni8LL1kD7tyqPWdAbFu6G62PqqsX9v\naTx1fw9Pa03XxEut6YbjLwKfHvI/EeXyNNSVS6hrbObDjbm8suoA+4uqiQ705dYJiXwvI4Eg/x46\n0Zcx1rw1bYK+yJpT/tA3kLfRuinr4Qn9xthD/lJryKV3D7mXoFyOhrpyKS0t1vQDL319gNX7S/D3\ntjEnPZ6fTBhAQthpVmHqieqrrHVdD62yvlqHfOxYK+Bj0yEqBYL760NTqks01JXL2p5XwSurDvLZ\nljyaWwzTUqK4eXx/JiSFnxIQHjMAABHJSURBVNvkYc7WWcgDeAdY4R41HCJTrAVFolLAN8i5Nase\nR0NdubyCyjreWHOIt9YeoaymkfhQP+ZmJPDdsXFEBvo6u7xz11BtTX9QkGU9IVuw3XpdV3Fyn6AE\nK+ijhluLesemQ98I59WsnE5DXbmN+qZmlmwv4O21R1hzoASbh3BFciRzxyVw2eAIbK549d6eMVCZ\nCwU72oZ98R7rYSmwumriMqyQj8uA6DQdbdOLaKgrt3SgqIp312fz/oYcSqobiA3246aMeOakxxMd\n5MJX751pqIGjm61hljnrrTH0lblWm80bokecDPnIFAgdoAuNuCkNdeXWGppaWLajgLfXHWHVvmI8\nBKYMi2RuRgKThkbg2VOGRXaHyry2IZ+3CRprTrYHxloPTIUOhLAk++skDXwXp6Gueo3DJdW8uz6b\nBZk5FFfVExHgw3fGxDInPZ6knvLEandqboLCHVZXTekB66tkv/W9pt0s2IGxEDIAgmKt10Gx1syW\nx1/7ButonB5KQ131Oo3NLSzfVciCzByW7y6kucWQ3j+EOenxzBwRQ9+zXTDbHdSWQ9nBkyFfegBK\nD1pdOJV5J/vrj/Pq0zbwwwZZ3TqRyVb4a+A7jYa66tUKj9Xx0cZc3s3M5kBRNf7eNmalxTAnI570\n/iE9Z0oCZ2pptiYxq8iFyhz791yoyDn5varg5P7eAVa4RyafDPqo4dY0CarbaagrhbWm6sYjZSxY\nn8PCrXlUNzQzMLwPN4yNY1ZaDIk9ZbbInqq2HIp2Wd07hTut0TmF26G27OQ+fSIgJNF63dJ0corj\nE6+brOmPj7+OSoHk2TBsFgT2c8ppuSINdaXaqa5vYtG2oyzIzGb9ISuUhkUHMD01mump0QyNCtAr\n+K4wBqoKrXAv3GkFfnn2ydWmjq8uJa1/tm/DWCtWley1jhWbbq1AlXyNdVNXdUpDXanTyC6tYcn2\nfJZszyfzcBnGQGKYP1elRjMjNYYRsUGu+fSqqyjabS0uvnOhNWQTrC6d4wEflar99+04YuWjeOAN\nIAowwHxjzJ/b7RMEvAkkAJ7An4wx/zjdcTXUVU9TeKyOZTsK+CIrnzX7S2hqMcQE+XLV8GiuGh5N\nRmKIew+RdLbyI1a471poLUuIsR60GjbLWlg8fhwERDu7SqdzRKjHADHGmI0iEgBsAK4zxuxotc8v\ngSBjzIP2NU13A9HGmIbOjquhrnqyippG/r2zgC+257NyTxH1TS2E9fHmyuFRzEiN4eKksJ4zNbA7\nqiqC3Ytg52dwcCU011vbg/tbUxnHj7O+IoeDrXeNZnJ494uIfAI8b4xZ1mrbw0A88F9AIrAMGGKM\naensOBrqylVU1zexYncRi7OOsnxXIdUNzQT5eTEtJYqZadFMGBSOj6fN2WW6r6Z6OLrVmgwtey1k\nr4OqfKvNqw/EjbWCPm4c+IdZwzNNy8l1aE98bzn5s6eP9TBWSKK9j991ODTURSQRWAmkGmMqW20P\nAD4FhgEBwE3GmM87eP88YB5AQkLC2MOHD3f5dyvVE9Q1NrNyTxFfZOWzbGcBx+qaCPDxZGpyJDPS\nYrh8SAS+Xq4VEi7HGKjItsL9eNDnZ5061r4rbD7WGPyIIRA+1P59iLWts6duj8+jX1Nqjf6pLbVG\nBzXVW6N4guKs8f3e3TNVtMNCXUT6Al8BTxpjPmzXdiMwAbgPSMK6Uh/ZOvjb0yt15erqm5pZva+E\nxVlHWbqjgPKaRvy9bUweFsnM1BgmD4vA37t3dQ04TX2VdaO1vso+4saj1cib1t/t2xtroXivtZpV\n0R7re9lhrNuGAGItXxg+xHpP+wDvyj8gfqFWwB//Cow9+To06Zxn23RIqIuIF7AQWGKMebaD9s+B\np4wxX9t//g/wkDFmXWfH1FBX7qSxuYW1B0pZlHWUpdvzKa5qwNfLg0lDIpmRFs3U5Kje+SSrK2ms\nhZJ91jQLx4O+eB8IVkD7hYC//XtHP9s8raUPK3LsD3Ed/7I/wFXfakrlS+6GK393TmU64kapAK8D\npcaYezvZ5wWgwBjzGxGJAjZiXakXd7Q/aKgr99XcYlh3sJTFWUdZnJVP0bF6vD09uHxIBDPtAR/o\n20OX6FPdp67y5BO6gbHWw1fnwBGhfinwNbANOH7j85dYwxcxxrwoIv2A14AYrH/XnjLGvHm642qo\nq96gpcWw4UgZi7Yd5YusfI5W1OFlEyYOjmBGajRXDo8myE8DXnWdPnykVA/R0mLYnFPO4m1HWbQt\nn9zyWrxtHkxNjuS60bFMHhqJt6cOk1Snp6GuVA9kjGFrTgUfb87lsy15FFc1EOzvxdUjYrh+dCxj\nEnSyMdUxDXWlerim5ha+3lfMRxtzWbojn7rGFvqH+XPdqFiuGx3LAJ1sTLWioa6UCzlW18iS7QV8\ntCmH1ftLMAZGJwRz/ehYZqbFEN5X1yLt7TTUlXJRRytq+WRzHh9tzGV3wTFsHsIlSWHMHtmPq1Kj\ndQRNL6WhrpQb2JVfyaeb8/hsax7ZpbV4e3oweWgEs0fGMjU5Up9i7UU01JVyI8YYNmeX8+mWPBZu\nPUrRsXr6eNuYlhLF7FH9mDg4Qicac3Ma6kq5qeYWw9oDJXy2NY9F2/KpqG0k2N+LK1OimJYSzcTB\n4XoF74Y01JXqBRqaWvh6bxGfbcnjy12FHKtrws/LxsTB4UxLiWJqchShfbydXaZygDOFuk5GoZQb\n8Pb0YGqyFd4NTS2sO1jK0h35LNtRwNIdBXgIpCeG2q/io+gfpsMk3ZVeqSvlxowxbM+rZOmOApZu\nz2dX/jEAhkYFMCU5kozEEEbHhxCiV/EuQ7tflFInZJfW2K/e81l/qIzmFuvv/8DwPoxOCGFs/xDG\n9A9mcGQANl2btUfSUFdKdaimoYmtORVsPFLGxsPlbDpSRkm1tQJlXx9PRsUHMyYhmNH9rbDXcfE9\ng/apK6U65O/tyfiBYYwfGAZYXTVHSmvYcLjsRNA/v3wfLQY8BNLigpmQFMYlSeGkJ4boyJoeSq/U\nlVKdqq5vYkt2Od8eKGH1/hI2Z5fT1GLwtnkwpn8wlySFM2FQGCPignV8/AWi3S9KKYeprm9i3aFS\n1uwv4Zt9xew4Wokx4O9tY9yAUC4dFM5Vw6OJD+2e9TmVhrpSqhuVVTecuIpfvb+Y/UXVAAzvF8j0\n4dHMSItmUGSAk6t0L45Y+SgeeAOIwlqVdb4x5s8d7DcJeA7wAoqNMZef7rga6kq5n+zSGr7IyueL\n7flsOFwGQFJEH2akxjA9NZrh/QJ1nvjz5IhQjwFijDEbRSQA2ABcZ4zZ0WqfYGA1MN0Yc0REIo0x\nhac7roa6Uu6toLKOpdvzWZyVz9qDpTS3GOJC/Jg+PJrpqdGMTgjRYZPnwOHdLyLyCfC8MWZZq20/\nBfoZYx7p6nE01JXqPUqrG/j3jgIWZx1l1b5iGpsNQX5ejB8YyoRB4VySFEZSRF+9iu8Chw5pFJFE\nYDSwtl3TEMBLRFYAAcCfjTFvnFWlSim3FdrHmzkZ8czJiKeyrpHluwr5Zl8x3+wrYcn2AgAiAny4\nJCnM/hWuN1vPUZev1EWkL/AV8KQx5sN2bc8D6cBUwA9YA8wyxuxpt988YB5AQkLC2MOHD5/3CSil\nXFt2aQ3f7Cu232wtobiqHoD4UD8uGRjO+KRQRseH0D/MX6/kcVD3i4h4AQuBJcaYZztofwjwM8Y8\nZv/5FeALY8x7nR1Tu1+UUu0ZY9hXWMVq+5DJbw+UUFnXBECQnxcj4oIYFR/MyLhgRsYHExHQ+5b3\nc8SNUgFeB0qNMfd2sk8y8DxwFeANrAPmGmOyOjuuhrpS6kyaWwy784+xNaecLTnlbM6uYE/BsRNz\n1sQG+zEyPogRcceDPgh/b/d+UN4RfeoTgB8C20Rks33bL4EEAGPMi8aYnSLyBbAVaAFePl2gK6VU\nV9g8hJR+gaT0C2TuuAQAahua2Z5XwebscrbkVLAlu5xF2/IB8LIJI+OCGT8wjIuTwhiTEIKfd++a\nzkAfPlJKubzS6ga2ZJez9mAp3x4oYVtuBc326QxGxQczfmAo4weGMaa/689Zo0+UKqV6nWN1jWQe\nLuPb/SWsOVBCVm4FLQYr5BOCGT8glFEJVpdNWF/X6pfXWRqVUr1OgK8Xk4dGMnloJACVdY2st1/F\nf3ug9MTsk2CNshkZF2zdgI0PJrVfkEt32WioK6XcXqCv14nl/gCq6pvIyrX647fklLPpSDkLtx4F\nrH78IVEBjIoPYmRcMKMTQhgc2RcPF3n6VbtflFIKKDxWx9bsCvsom3K2ZJefGE4Z6OvJmP4hjE0I\nYWxiCKPig502yka7X5RSqgsiA3y5IsWXK1Ksq3ljDAeLq9l4pJwNh8vYcLiUFbuLAPuonJhAxtpX\nhRrbP4R+wX7OLP8EvVJXSqkuqqhpZGN2GRsOlbHhcBmbs8upbWwGICbIl9TYIFL7BZEWF0hqbBCR\nAb4Or0Gv1JVSykGC/NvegG1sbmHX0WNkHi5l05FysvIq+PfOAo5fK0cG+JAWG8Tw2CDS7F9RgT7d\nOt2BhrpSSp0jL5sHaXFBpMUF8eMJ1raq+iZ25FWyLbeCLPvX8t2FJ0bbhPf15s7LkrjjsoHdUpOG\nulJKOVBfH0/GDQhl3IDQE9tqGqygz8qtYFtuJZGB3Tc2XkNdKaW6mb+3J+mJoaQnhp555/Oky38r\npZQb0VBXSik3oqGulFJuRENdKaXciIa6Ukq5EQ11pZRyIxrqSinlRjTUlVLKjThtQi8RKQIOn+Pb\nw4FiB5bTE7jbObnb+YD7nZO7nQ+43zl1dD79jTERnb3BaaF+PkQk83SzlLkidzsndzsfcL9zcrfz\nAfc7p3M5H+1+UUopN6KhrpRSbsRVQ32+swvoBu52Tu52PuB+5+Ru5wPud05nfT4u2aeulFKqY656\npa6UUqoDGupKKeVGXC7URWS6iOwWkX0i8pCz63EEETkkIttEZLOIuNxq3CLyqogUikhWq22hIrJM\nRPbav4c4s8az1ck5/UZEcu2f02YRmenMGs+GiMSLyHIR2SEi20XkHvt2l/ycTnM+rvwZ+YrIOhHZ\nYj+nx+3bB4jIWnvmvSsi3qc9jiv1qYuIDdgDTANygPXA94wxO5xa2HkSkUNAujHGJR+aEJHLgCrg\nDWNMqn3bH4BSY8xT9n98Q4wxDzqzzrPRyTn9BqgyxvzJmbWdCxGJAWKMMRtFJADYAFwH3IoLfk6n\nOZ85uO5nJEAfY0yViHgBq4B7gPuAD40x74jIi8AWY8wLnR3H1a7UxwH7jDEHjDENwDvAtU6uqdcz\nxqwEStttvhZ43f76day/cC6jk3NyWcaYo8aYjfbXx4CdQCwu+jmd5nxclrFU2X/0sn8ZYArwvn37\nGT8jVwv1WCC71c85uPgHaWeApSKyQUTmObsYB4kyxhy1v84HopxZjAP9t4hstXfPuERXRXsikgiM\nBtbiBp9Tu/MBF/6MRMQmIpuBQmAZsB8oN8Y02Xc5Y+a5Wqi7q0uNMWOAGcB/2f/r7zaM1cfnOv18\nnXsBSAJGAUeBZ5xbztkTkb7AB8C9xpjK1m2u+Dl1cD4u/RkZY5qNMaOAOKyeiWFnewxXC/VcIL7V\nz3H2bS7NGJNr/14IfIT1Ybq6Anu/5/H+z0In13PejDEF9r90LcBLuNjnZO+n/QD4lzHmQ/tml/2c\nOjofV/+MjjPGlAPLgYuBYBHxtDedMfNcLdTXA4Ptd4O9gbnAp06u6byISB/7jR5EpA9wJZB1+ne5\nhE+BW+yvbwE+cWItDnE8/Oyux4U+J/tNuFeAncaYZ1s1ueTn1Nn5uPhnFCEiwfbXflgDQnZihfuN\n9t3O+Bm51OgXAPsQpecAG/CqMeZJJ5d0XkRkINbVOYAn8JarnZOIvA1MwpomtAB4DPgYWAAkYE2x\nPMcY4zI3Hjs5p0lY/603wCHgzlb90T2aiFwKfA1sA1rsm3+J1Q/tcp/Tac7ne7juZzQC60aoDeuC\ne4Ex5gl7RrwDhAKbgJuNMfWdHsfVQl0ppVTnXK37RSml1GloqCullBvRUFdKKTeioa6UUm5EQ10p\npdyIhrpSSrkRDXWllHIj/x/Gs20ZN0qtfwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from matplotlib import pyplot\n",
    "pyplot.plot(history.history['loss'], label='train')\n",
    "pyplot.plot(history.history['val_loss'], label='test')\n",
    "pyplot.legend()\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "HSyx-HvpUz2o"
   },
   "source": [
    "From the plot, we can infer that validation loss has increased after epoch 17 for 2 successive epochs. Hence, training is stopped at epoch 19.\n",
    "\n",
    "Next, let’s build the dictionary to convert the index to word for target and source vocabulary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "sBX0zZnOFxjW"
   },
   "outputs": [],
   "source": [
    "reverse_target_word_index=y_tokenizer.index_word\n",
    "reverse_source_word_index=x_tokenizer.index_word\n",
    "target_word_index=y_tokenizer.word_index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "eM_nU_VvFxjq"
   },
   "source": [
    "# Inference\n",
    "\n",
    "Set up the inference for the encoder and decoder:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![topic_modeling](images/inference.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "9QkrNV-4Fxjt"
   },
   "outputs": [],
   "source": [
    "# Encode the input sequence to get the feature vector\n",
    "encoder_model = Model(inputs=encoder_inputs,outputs=[encoder_outputs, state_h, state_c])\n",
    "\n",
    "# Decoder setup\n",
    "# Below tensors will hold the states of the previous time step\n",
    "decoder_state_input_h = Input(shape=(latent_dim,))\n",
    "decoder_state_input_c = Input(shape=(latent_dim,))\n",
    "decoder_hidden_state_input = Input(shape=(max_text_len,latent_dim))\n",
    "\n",
    "# Get the embeddings of the decoder sequence\n",
    "dec_emb2= dec_emb_layer(decoder_inputs) \n",
    "# To predict the next word in the sequence, set the initial states to the states from the previous time step\n",
    "decoder_outputs2, state_h2, state_c2 = decoder_lstm(dec_emb2, initial_state=[decoder_state_input_h, decoder_state_input_c])\n",
    "\n",
    "#attention inference\n",
    "attn_out_inf, attn_states_inf = attn_layer([decoder_hidden_state_input, decoder_outputs2])\n",
    "decoder_inf_concat = Concatenate(axis=-1, name='concat')([decoder_outputs2, attn_out_inf])\n",
    "\n",
    "# A dense softmax layer to generate prob dist. over the target vocabulary\n",
    "decoder_outputs2 = decoder_dense(decoder_inf_concat) \n",
    "\n",
    "# Final decoder model\n",
    "decoder_model = Model(\n",
    "    [decoder_inputs] + [decoder_hidden_state_input,decoder_state_input_h, decoder_state_input_c],\n",
    "    [decoder_outputs2] + [state_h2, state_c2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "zOiyk4ToWe74"
   },
   "source": [
    "We are defining a function below which is the implementation of the inference process"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "6f6TTFnBFxj6"
   },
   "outputs": [],
   "source": [
    "def decode_sequence(input_seq):\n",
    "    # Encode the input as state vectors.\n",
    "    e_out, e_h, e_c = encoder_model.predict(input_seq)\n",
    "    \n",
    "    # Generate empty target sequence of length 1.\n",
    "    target_seq = np.zeros((1,1))\n",
    "    \n",
    "    # Populate the first word of target sequence with the start word.\n",
    "    target_seq[0, 0] = target_word_index['sostok']\n",
    "\n",
    "    stop_condition = False\n",
    "    decoded_sentence = ''\n",
    "    while not stop_condition:\n",
    "      \n",
    "        output_tokens, h, c = decoder_model.predict([target_seq] + [e_out, e_h, e_c])\n",
    "\n",
    "        # Sample a token\n",
    "        sampled_token_index = np.argmax(output_tokens[0, -1, :])\n",
    "        sampled_token = reverse_target_word_index[sampled_token_index]\n",
    "        \n",
    "        if(sampled_token!='eostok'):\n",
    "            decoded_sentence += ' '+sampled_token\n",
    "\n",
    "        # Exit condition: either hit max length or find stop word.\n",
    "        if (sampled_token == 'eostok'  or len(decoded_sentence.split()) >= (max_summary_len-1)):\n",
    "            stop_condition = True\n",
    "\n",
    "        # Update the target sequence (of length 1).\n",
    "        target_seq = np.zeros((1,1))\n",
    "        target_seq[0, 0] = sampled_token_index\n",
    "\n",
    "        # Update internal states\n",
    "        e_h, e_c = h, c\n",
    "\n",
    "    return decoded_sentence"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "6GuDf4TPWt6_"
   },
   "source": [
    "Let us define the functions to convert an integer sequence to a word sequence for summary as well as the reviews:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "aAUntznIFxj9"
   },
   "outputs": [],
   "source": [
    "def seq2summary(input_seq):\n",
    "    newString=''\n",
    "    for i in input_seq:\n",
    "        if((i!=0 and i!=target_word_index['sostok']) and i!=target_word_index['eostok']):\n",
    "            newString=newString+reverse_target_word_index[i]+' '\n",
    "    return newString\n",
    "\n",
    "def seq2text(input_seq):\n",
    "    newString=''\n",
    "    for i in input_seq:\n",
    "        if(i!=0):\n",
    "            newString=newString+reverse_source_word_index[i]+' '\n",
    "    return newString"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9gM4ALyfWwA9"
   },
   "source": [
    "Here are a few summaries generated by the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 14427,
     "status": "ok",
     "timestamp": 1576677173440,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "BUtQmQTmFxkI",
    "outputId": "3faed8e9-f498-481c-bec0-53e627f13f96"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Review: ceat tyres released new range light hearted videos ignite consumers curiosity cost efficient range tyres videos funny take consumers save years instead making small savings daily tyres offer better fuel economy normal tyres longer life optimised handling \n",
      "Original summary: ceat new videos can help you save in years \n",
      "Predicted summary:  scientists develop new year for using camera\n",
      "\n",
      "\n",
      "Review: senior journalist kj singh found dead along mother home stabbed times chest lower abdomen right hand throat according post mortem report doctors said singh mother probably strangulated death notably motive behind murders unknown special investigation team probing case \n",
      "Original summary: journalist kj singh was stabbed times post mortem report \n",
      "Predicted summary:  journalist who stabbed mother times times found dead\n",
      "\n",
      "\n",
      "Review: senior leader former jammu kashmir minister javaid mustafa mir resigned peoples democratic party mir added forward formal resignation party sunday mir cite reason decision resign said consult followers future course action \n",
      "Original summary: former minister resigns from pdp \n",
      "Predicted summary:  ex minister resigns from pdp\n",
      "\n",
      "\n",
      "Review: original hollywood sign intended real estate advertisement sold ebay november crore owner stated bought sign man acquired city put new sign sign sold finance hollywood project \n",
      "Original summary: the original hollywood sign was sold on ebay for crore \n",
      "Predicted summary:  real madrid sell real estate sold for crore\n",
      "\n",
      "\n",
      "Review: saudi journalist critic jamal khashoggi death brutal planned murder turkish president recep tayyip erdo said tuesday called saudi arabia extradite suspects could tried turkish courts khashoggi killed inside saudi arabia consulate istanbul \n",
      "Original summary: saudi journalist death was planned murder turkey \n",
      "Predicted summary:  saudi journo death journo saudi journo journo saudi crown prince\n",
      "\n",
      "\n",
      "Review: patidar leader hardik patel dalit leader jignesh mevani alleged police protection provided gujarat government bjp tactic monitor movements ahead state assembly elections slamming police cover patel said police would better working people gujarat moving around \n",
      "Original summary: police protection bjp to monitor my moves hardik \n",
      "Predicted summary:  police to gujarat gujarat hardik patel\n",
      "\n",
      "\n",
      "Review: bahujan samaj party decided extend support samajwadi party upcoming phulpur gorakhpur bypolls uttar pradesh workers want eliminate bharatiya janata party bsp zonal coordinator ashok gautam said bsp supremo mayawati later clarified alliance extend general elections \n",
      "Original summary: bsp extends support to sp for bypolls in up \n",
      "Predicted summary:  bsp bsp alliance mayawati yadav in up\n",
      "\n",
      "\n",
      "Review: government decided sell four air india subsidiaries include regional airline alliance air hotel corporation india owns two hotels delhi srinagar subsidiaries air india engineering services involved maintenance repair overhaul engines air india air transport services passenger cargo handling unit \n",
      "Original summary: government to sell air india alliance air hotel business \n",
      "Predicted summary:  air india sale sale to air india sale for air india sale\n",
      "\n",
      "\n",
      "Review: police saturday arrested three kashmiri students hyderabad standing national anthem cinema hall police said management hall registered complaint following three booked relevant sections prevention insults national honour act however granted bail hours detention \n",
      "Original summary: kashmiris held for not standing during national anthem \n",
      "Predicted summary:  arrested for national anthem in hyderabad\n",
      "\n",
      "\n",
      "Review: filmmaker satish kaushik said always used address late actress sridevi madam adding persona could call sridevi added like family relations never changed even marriage acted kaushik directorial debut roop ki rani choron ka raja \n",
      "Original summary: always used to address sridevi as ma am satish kaushik \n",
      "Predicted summary:  sridevi was not an sridevi in sridevi says writer\n",
      "\n",
      "\n",
      "Review: karnataka minister ahmed khan left tip chef mohammed lunch seafood restaurant mangaluru minister also made chef sit next fed plate chef later revealed minister told never eaten course fish \n",
      "Original summary: taka minister feeds chef at hotel tips him \n",
      "Predicted summary:  taka minister throws food at restaurant in karnataka\n",
      "\n",
      "\n",
      "Review: weeks quitting trinamool congress joining bjp mukul roy friday said tmc biswa bangla initiative private company owned run mamata banerjee nephew abhishek banerjee people dying dengue chief minister busy celebrating film festivals stars celebrities added \n",
      "Original summary: trinamool congress is private company mukul roy \n",
      "Predicted summary:  bjp tmc workers quit over non bailable warrant\n",
      "\n",
      "\n",
      "Review: former un secretary general kofi annan receive state burial home country ghana september ghanaian president nana said annan given burial status global icon diplomat statesman added annan nobel peace laureate passed away switzerland earlier month age \n",
      "Original summary: ex un chief annan to get state burial in home country ghana \n",
      "Predicted summary:  un chief calls for not work in brazil\n",
      "\n",
      "\n",
      "Review: bengaluru man charged water bottle costing got compensation year filing complaint overcharged reportedly filed complaint vendor well manufacturer coca cola substantiate argument court also submitted another bottle invoice purchased \n",
      "Original summary: man charged extra on water bottle gets compensation \n",
      "Predicted summary:  man fined for cheating man of water\n",
      "\n",
      "\n",
      "Review: kangana ranaut sister rangoli sharing picture hrithik roshan kangana claimed photoshopped asked prove claim picture surfaced earlier taken party hrithik ex wife sussanne khan also present could wife stop multiple affairs nose rangoli tweeted \n",
      "Original summary: prove it is photoshopped kangana sister to hrithik on pic \n",
      "Predicted summary:  kangana sister hrithik sister kangana hrithik\n",
      "\n",
      "\n",
      "Review: world trade organization agreed hear india complaint us steel aluminium tariffs wto dispute settlement body agreed create separate panel hear complaints india filed second request us blocked india first request panel automatically established complainant makes second request \n",
      "Original summary: wto agrees to hear india complaint against us steel tariffs \n",
      "Predicted summary:  india to us against india us firms in us\n",
      "\n",
      "\n",
      "Review: congress president sonia gandhi returned india late thursday night accompanied son rahul gandhi left country earlier month undisclosed destination routine medical check party vice president rahul gandhi left march join mother soon attending swearing ceremony punjab cm amarinder singh \n",
      "Original summary: sonia gandhi returns from abroad after medical check up \n",
      "Predicted summary:  sonia gandhi sonia gandhi joins rahul gandhi\n",
      "\n",
      "\n",
      "Review: comedian kapil sharma speaking negative reviews film firangi said everyone film critic days maybe someone likes watching gandi films wondering sort film hero even hold hands heroine added however kapil said things intimidate \n",
      "Original summary: everyone is film critic these days kapil sharma \n",
      "Predicted summary:  kapil sharma is the only of the film industry kapil\n",
      "\n",
      "\n",
      "Review: myanmar parliament elected win myint country new president comes week myanmar first democratically elected president kyaw resigned position due health issues myint considered close aide state counsellor aung san suu kyi speaker lower house since \n",
      "Original summary: myanmar elects win as new president \n",
      "Predicted summary:  myanmar prez elected as president of president\n",
      "\n",
      "\n",
      "Review: heirs right access facebook accounts deceased relatives german court ruled thursday court hearing case mother year old deceased could access daughter facebook account company locked citing privacy reasons social media account inherited like letters court said \n",
      "Original summary: parents to access dead daughter fb german court \n",
      "Predicted summary:  court rules court court account for facebook court\n",
      "\n",
      "\n",
      "Review: year old man beaten iron rods shot leg neighbours outside house water dispute delhi reports said family members also beaten trying protect rushed hospital said critical condition \n",
      "Original summary: delhi man thrashed with shot at over water dispute \n",
      "Predicted summary:  man shot dead by delhi metro in noida\n",
      "\n",
      "\n",
      "Review: kolkata court wednesday sentenced bangladeshi man convicted raping year old nun robbery convent life imprisonment till death court also sentenced four others accused case years imprisonment robbery sixth accused given seven year term providing shelter others \n",
      "Original summary: man convicted of raping kolkata nun given life sentence \n",
      "Predicted summary:  man sentenced to life term for raping minor in life\n",
      "\n",
      "\n",
      "Review: karan johar occasion international coffee day hinted sixth season talk show koffee karan tweeted may changed spelling love remains season anyone twinkle khanna sania mirza kapil sharma appeared first time show fifth season aired november last year \n",
      "Original summary: karan johar hints at season of koffee with karan \n",
      "Predicted summary:  johar karan johar to appear together on koffee with karan\n",
      "\n",
      "\n",
      "Review: six french soldiers injured wednesday bmw car deliberately drove paris suburb according reports without doubt deliberate act vehicle waiting officials said authorities launched investigation incident searching vehicle driver involved \n",
      "Original summary: car deliberately drives into french soldiers injures six \n",
      "Predicted summary:  french soldiers car crashes into car bomb in paris\n",
      "\n",
      "\n",
      "Review: indigo flight headed towards bengaluru made emergency landing netaji subhas chandra bose international airport kolkata sunday pilot discovered cracks windshield flight departed landed passengers safe aircraft repaired \n",
      "Original summary: indigo flight forced to land due to cracks in \n",
      "Predicted summary:  indigo flight makes emergency landing after flight\n",
      "\n",
      "\n",
      "Review: year old first year btech student iit guwahati sc allegedly committed suicide hanging wednesday resident karnataka left behind suicide note wrote engineering sucks said wanted become teacher also wrote better die fail live expectation parents \n",
      "Original summary: iit student commits suicide note says engineering \n",
      "Predicted summary:  iit student commits suicide after being commits suicide\n",
      "\n",
      "\n",
      "Review: oscar winning hollywood actor robert de niro said us president donald trump either impeached arrested cannot imagine going another three years adding us administration helpful slammed trump policies climate change saying president know \n",
      "Original summary: trump will either be impeached or arrested robert de niro \n",
      "Predicted summary:  trump cannot be arrested for calling us president de niro\n",
      "\n",
      "\n",
      "Review: us based geologists found million year old fossil fragments trees antarctica time predating first dinosaurs plants must survived polar extremes perpetual light total darkness said researchers however massive increase greenhouse conditions years caused extinction species including polar forests added \n",
      "Original summary: uncover remains of forests in antarctica \n",
      "Predicted summary:  scientists find million year old dinosaur era\n",
      "\n",
      "\n",
      "Review: per reports hrithik roshan perform opening ceremony upcoming edition indian premier league comes ranveer singh reportedly opted performance twenty cricket league opening ceremony due shoulder injury th season ipl set begin april \n",
      "Original summary: hrithik replaces ranveer for ipl opening performance report \n",
      "Predicted summary:  hrithik hrithik to perform in ipl report\n",
      "\n",
      "\n",
      "Review: demonetisation banks received lakh crore high value deposits customers without providing permanent account numbers indicating possible collusion officials hide suspicious cash according hindustan times half transactions six states like gujarat rajasthan pan details mandatory deposits help detect tax evasion \n",
      "Original summary: banks accepted over lakh crore without pan details \n",
      "Predicted summary:  banks to get crore from pan accounts in\n",
      "\n",
      "\n",
      "Review: union sports minister vijay goel ruled chances bilateral series india pakistan saying cricket terrorism cannot go hand hand comments come ahead meeting bcci pakistan cricket board officials scheduled monday evening dubai added bcci send proposal consulting government \n",
      "Original summary: no chance of cricket series with pakistan govt \n",
      "Predicted summary:  will not let pak cricket team india to be held for terror\n",
      "\n",
      "\n",
      "Review: tanveer hussain year old indian athlete kashmir indicted us charges sexually abusing minor girl visit us competition essex county jury charged hussain one count first degree sexual abuse two counts endangering welfare child \n",
      "Original summary: indian athlete indicted on sex abuse charge in us \n",
      "Predicted summary:  indian teen charged with sexual abuse in us\n",
      "\n",
      "\n",
      "Review: people across several locations australia looked moon time week help australian national university break world record number people involved five times previous guinness world record might inspired next nobel prize winner australia said anu astronomer \n",
      "Original summary: aussies look at moon at same time break world record \n",
      "Predicted summary:  australia to get world record for th time in aus\n",
      "\n",
      "\n",
      "Review: bombay high court stayed income tax department notice issued several tata trusts including ratan tata trust notice issued alleged non compliance rules governing use trust funds tata trusts moved court challenge notice issued cancel trust registration would result withdrawal tax exemption \n",
      "Original summary: bombay hc stays dept notices to several tata trusts \n",
      "Predicted summary:  bombay hc issues notice to tata tata sons notice over tata\n",
      "\n",
      "\n",
      "Review: former congress mp priya dutt monday announced contest lok sabha polls struggled keep balance personal political life took toll many aspects life said priya entered electoral politics fought three general assembly elections last one \n",
      "Original summary: ex cong mp priya dutt to not contest lok sabha polls \n",
      "Predicted summary:  ex mp congress mp joins lok sabha elections\n",
      "\n",
      "\n",
      "Review: migration venezuelans nearing moment similar refugee crisis mediterranean un international organization migration said venezuelans fleeing ongoing political economic crisis country caused shortage food medicine basic goods million venezuelans fled country since \n",
      "Original summary: venezuelan migration nears refugee crisis un \n",
      "Predicted summary:  venezuela crisis un urges un to withdraw from un\n",
      "\n",
      "\n",
      "Review: upgraded version indigenous rocket successfully test fired proof experimental establishment odisha wednesday two rounds tests conducted rounds planned earlier version rocket new rocket includes guidance navigation control kit \n",
      "Original summary: indigenous rocket test fired successfully \n",
      "Predicted summary:  rocket test successfully test fires test rocket\n",
      "\n",
      "\n",
      "Review: two women including district congress leader found dead sunday house uttar pradesh ballia women hit head iron rods evidence theft police said bodies sent postmortem motive behind murder clear soon police added \n",
      "Original summary: two women including congress leader found dead in up \n",
      "Predicted summary:  women congress leader found dead in up\n",
      "\n",
      "\n",
      "Review: technology giant google acqui hired former workers startup enhance ai powered assistant ability accurately answer questions acquisition took place reported amount nearly million founded former google employees sophia yang william li developed question answer app users queries responded experts \n",
      "Original summary: google hires ex workers startup to improve assistant \n",
      "Predicted summary:  google hires ex startup exec as startup ai startup\n",
      "\n",
      "\n",
      "Review: former pakistani president pervez musharraf thursday said hoped pakistan tehreek insaf chairman imran khan govern pakistan better way imran khan problem areas honest man wants good good qualities added musharraf barred pakistan supreme court contesting elections \n",
      "Original summary: hope imran pakistan in better way ex prez musharraf \n",
      "Predicted summary:  imran khan will not be imran khan musharraf musharraf\n",
      "\n",
      "\n",
      "Review: president ram nath kovind wednesday said eradicating corruption foremost condition creating new india kovind emphasised role technology internet regard addressing event central vigilance commission digital systems managed address loopholes misuse public money added \n",
      "Original summary: corruption for creating new india prez \n",
      "Predicted summary:  india should be more than new other vp kovind\n",
      "\n",
      "\n",
      "Review: describing law order situation haryana worrisome congress leader bhupinder singh hooda demanded cm manohar lal khattar resign moral grounds mentioning violence jat reservation riots ram rahim conviction recent rape incidents former cm said state crime graph increasing three years bjp rule \n",
      "Original summary: congress demands haryana cm resignation over rise in crime \n",
      "Predicted summary:  haryana cm seeks probe into dera riots case\n",
      "\n",
      "\n",
      "Review: centre saturday dismissed reports suggested agreed carve larger nagaland state reports erroneous clarified agreement decision government home ministry spokesperson said comes im said centre accepted territorial integration naga inhabited areas \n",
      "Original summary: govt dismisses reports of out larger nagaland state \n",
      "Predicted summary:  govt denies reports of govt employees in nagaland\n",
      "\n",
      "\n",
      "Review: square kilometre iceberg separated antarctica pine island glacier late september disintegrated smaller icebergs within two months pig fastest melting glacier antarctica responsible quarter frozen continent ice loss scientists expressed concern events resulting rise sea levels \n",
      "Original summary: antarctic iceberg two months after separation \n",
      "Predicted summary:  antarctic iceberg finds km long ice\n",
      "\n",
      "\n",
      "Review: scientists new zealand performed first ever colour ray human body using imaging technology developed large hadron collider cern physics lab scientists claim new imaging technology help providing information tissues mass surrounding bones something possible traditional ray imaging \n",
      "Original summary: world first ever colour ray performed on human body \n",
      "Predicted summary:  new zealand first human human cells\n",
      "\n",
      "\n",
      "Review: russia denied nato allegation may secretly sending supplies taliban afghanistan saying designed justify failure us military afghan campaign russia considers taliban terrorist organisation however said taliban interest coincides fight islamic state \n",
      "Original summary: russia denies supplying the taliban after nato claim \n",
      "Predicted summary:  russia denies russia claim of afghan afghanistan taliban\n",
      "\n",
      "\n",
      "Review: plane carrying otago volts cricket team struck lightning towards new zealand dunedin airport interesting flight home coming dunedin particularly nasty said otago volts co captain jacob added passengers feel anything witnessed white flash \n",
      "Original summary: plane carrying new zealand side struck by lightning \n",
      "Predicted summary:  plane carrying out of new zealand flight found in new zealand\n",
      "\n",
      "\n",
      "Review: sweden based brand management company partnered german carmaker make iphone cases metal salvaged car caused texting driving handmade iphone crashed cases made priced one crash since sweden ban using phones driving proceeds go charity \n",
      "Original summary: iphone cases being made using metal from car \n",
      "Predicted summary:  iphone iphone sales to be iphone issue\n",
      "\n",
      "\n",
      "Review: apple launched expensive smartphone iphone edge edge display face recognition technology gb gb variants respectively samsung launched galaxy note inch display google launched android oreo powered smartphones pixel pixel xl respectively gb variants \n",
      "Original summary: which were the costliest phones launched in india in \n",
      "Predicted summary:  apple watch iphone sales in iphone iphone\n",
      "\n",
      "\n",
      "Review: attacking congress president rahul gandhi post titled clown prince union minister arun jaitley said rahul living state self stating rahul alternative facts suit jaitley said false narrative dozen times convince falsehood fact truth \n",
      "Original summary: is clown prince out himself jaitley post on rahul \n",
      "Predicted summary:  rahul gandhi is the only of the mother jaitley\n",
      "\n",
      "\n",
      "Review: weightlifter sanjita chanu india second gold medal commonwealth games friday said sad breaking cwg record failing lift kg last attempt probably god today probably push well jerk added chanu gold medal women kg category \n",
      "Original summary: sad could not break cwg record india gold medalist \n",
      "Predicted summary:  cwg gold is not times in cwg gold winner\n",
      "\n",
      "\n",
      "Review: rohit shetty said audience happy feels restless adding want enjoy films happy audience happy laughing cheering person goes watch film puts per cent earning see entertained said \n",
      "Original summary: when audience is not happy feel rohit shetty \n",
      "Predicted summary:  happy birthday my wife rohit shetty on rohit sharma\n",
      "\n",
      "\n",
      "Review: trends showed congress led alliance trailing behind telangana rashtra samithi telangana union home minister rajnath singh said mahagathbandhan failed terribly telangana alliance consists congress telugu desam party communist party india telangana jana samithi leading seats trs leading seats \n",
      "Original summary: mahagathbandhan failed in telangana rajnath singh \n",
      "Predicted summary:  telangana cm rajnath singh takes dig at telangana cm\n",
      "\n",
      "\n",
      "Review: turkey friday called international community establish fly safe zones syria supported us missile strikes targeted syrian military airfield suspected launching chemical attack turkey also spoke support removing syrian president bashar al assad government accused attack killed people injured another \n",
      "Original summary: turkey calls for no fly zone in syria after us strikes \n",
      "Predicted summary:  turkey to turkey to fly missile strikes in syria\n",
      "\n",
      "\n",
      "Review: filmmaker karan johar responded twitter user trolled nepotism troll tweeted karan johar win gold medal launching star kids replying johar wrote win bronze bullshit johar earlier slammed launching star kids like janhvi kapoor ishaan khatter bollywood \n",
      "Original summary: you can win bronze medal in bulls johar to troll \n",
      "Predicted summary:  johar tweets user on twitter on johar win\n",
      "\n",
      "\n",
      "Review: major markets delhi remained shut wednesday traders unions called strike protest sealing drive carried municipal corporations drive came based directives supreme court monitored committee violation delhi master plan traders associations estimate sealing affected lakh people including traders employees families \n",
      "Original summary: delhi markets remain shut as traders protest sealing drive \n",
      "Predicted summary:  delhi schools shut down after protest against strike\n",
      "\n",
      "\n",
      "Review: campaigning rajasthan wednesday bsp leader jagat singh said ka ak ke sath karta hoon main jagat former bjp leader added come ashok ji come modi ji come vasundhara ji send packing meanwhile bjp responded party strongly condemns statement given ego fury \n",
      "Original summary: respond to stones with ak will send pm bsp leader \n",
      "Predicted summary:  will not be part of the film if it is bjp leader\n",
      "\n",
      "\n",
      "Review: actress shivpuri worked rape accused actor alok nath various films revealed caught peeing open deplaned flight misbehaved said would behave normally daytime would change completely consuming liquor many actresses told tough time working \n",
      "Original summary: alok nath was once caught peeing in open \n",
      "Predicted summary:  rape victim is open to open in open says farah\n",
      "\n",
      "\n",
      "Review: man beaten death haryana sonipat allegedly sharing pictures clicked family members whatsapp group victim brother said accidentally shared pictures group angered one recipient accused invited man family home attacked rods bricks \n",
      "Original summary: man killed over sharing family pics on whatsapp group \n",
      "Predicted summary:  man kills himself after whatsapp posts pics with whatsapp\n",
      "\n",
      "\n",
      "Review: filmmaker mani ratnam son nandan robbed italy wife sought help twitter son requested help saying son needed go airport within hour tweeted help arrived nandan thanked people stepping help \n",
      "Original summary: mani son robbed in italy mother seeks help online \n",
      "Predicted summary:  son son seeks to son for not giving son son\n",
      "\n",
      "\n",
      "Review: supreme court issued contempt notice delhi bjp chief manoj tiwari allegedly breaking sealed house lock summoned appear court september operating dairy facility house sealed allegedly violating delhi master plan bench said unfortunate elected representative allegedly tried defy orders nn \n",
      "Original summary: sc summons delhi bjp chief for breaking lock of sealed house \n",
      "Predicted summary:  sc notice to delhi bjp chief for illegal construction\n",
      "\n",
      "\n",
      "Review: many maharashtra doctors associated indian medical association called strike inadequate security hospitals meeting cm devendra fadnavis friday fadnavis said security provided since thursday arrangements made next days earlier doctors warned join work take pay cut \n",
      "Original summary: maharashtra doctors call off their strike after meeting cm \n",
      "Predicted summary:  doctors strike in maharashtra cm amid cm shortage\n",
      "\n",
      "\n",
      "Review: singapore based wearable technology startup kaha raised million funding round led metals international jungle ventures spring seeds capital existing investor yournest also participated funding round founded pawan gandhi justin tang shantaram kaha offers products including smart watches smart belts smart wallets \n",
      "Original summary: wearable tech startup raises million \n",
      "Predicted summary:  singapore startup raises mn from app\n",
      "\n",
      "\n",
      "Review: according reports actress angelina jolie dating british entrepreneur reportedly two meeting angelina estate malibu also planning introduce children reports added angelina estranged husband brad pitt reportedly know caught guard learning dating entrepreneur \n",
      "Original summary: angelina jolie dating british entrepreneur reports \n",
      "Predicted summary:  dating husband report on reports of dating report\n",
      "\n",
      "\n",
      "Review: minister state human resource development satyapal singh recent education meeting said mantras laws motion much discovered newton asserting traditional knowledge must incorporated curriculum singh reportedly also suggested vaastu compliance educational buildings important learning \n",
      "Original summary: laws of motion before newton mos hrd \n",
      "Predicted summary:  no one can be allowed to be allowed to be in japan minister\n",
      "\n",
      "\n",
      "Review: nine year old girl allegedly gangraped killed along minor brother sunday bihar bodies found bank river search launched return home evening police said parents identified three persons village culprits \n",
      "Original summary: yr old girl gangraped killed with minor brother in bihar \n",
      "Predicted summary:  year old girl gangraped by brother in law in bihar\n",
      "\n",
      "\n",
      "Review: london based indian woman found dead inside bathtub star hotel faridabad friday body recovered police reached hotel ignored family calls day woman early reportedly asked hotel send staff room \n",
      "Original summary: nri woman found dead in bathtub at star hotel \n",
      "Predicted summary:  indian woman found dead in hotel room in hotel\n",
      "\n",
      "\n",
      "Review: archaeologists turkey discovered tomb beneath ruins ancient church believe contains remains saint nicholas also known santa claus grave detected using digital imaging tools obtained good results real work starts maybe find untouched body saint nicholas archaeologists said \n",
      "Original summary: claim to have found tomb of santa \n",
      "Predicted summary:  found in human rights to find human cells\n",
      "\n",
      "\n",
      "Review: uk based augmented reality app launched feature lets users turn face digital billboard lets users create profile saving face providing information like favourite music latest photos information shows bubbles around user face ar someone scans face app \n",
      "Original summary: startup makes feature to turn user face into billboard \n",
      "Predicted summary:  app lets users control their users in uk\n",
      "\n",
      "\n",
      "Review: russia help build new pipeline project deliver natural gas iran india russian energy minister aleksandr novak said russia iran prepare legal framework kilometre long pipeline project expected begin next year involve participation iranian pakistani indian firms novak added \n",
      "Original summary: russia to help build india iran natural gas pipeline \n",
      "Predicted summary:  russia to build new nuclear system in russia\n",
      "\n",
      "\n",
      "Review: kerala rounder saxena four bcci awards last four years said depressed even received india call giving award reward seems like insult added year old highest wicket taker ranji trophy last year \n",
      "Original summary: am depressed uncapped cricketer who won awards in yrs \n",
      "Predicted summary:  kerala award winning award was not getting kerala award\n",
      "\n",
      "\n",
      "Review: exit polls predicted karnataka face hung assembly party able secure seats required form government member assembly india today polls suggested bjp likely win seats congress winning seats times suggested bjp form government majority \n",
      "Original summary: exit polls predict hung assembly in karnataka \n",
      "Predicted summary:  karnataka assembly elections to be held in karnataka\n",
      "\n",
      "\n",
      "Review: bihar police district directed radicalised muslim youth clean police station ahead independence day celebrations alleged anti india pro pakistan activities alam son serving home guard jawan allegedly posted photos pakistan pm nawaz sharif anti india terrorists facebook \n",
      "Original summary: anti india bihar boy to clean police station as \n",
      "Predicted summary:  bihar police station to be held for independence day\n",
      "\n",
      "\n",
      "Review: acharya das chief priest ram janmabhoomi temple tuesday said bjp wanted come power general elections start constructing ram temple bjp way betrayed lord ram came power name ram forgot lord added \n",
      "Original summary: bjp must construct ram temple to win in ayodhya priest \n",
      "Predicted summary:  bjp will build ram temple in ayodhya case priest\n",
      "\n",
      "\n",
      "Review: india tuesday announced emergency aid crore caribbean countries worst affected recent hurricanes feel pain trying hour stand stand ready assist efforts rebuild broken lives external affairs minister sushma swaraj said announcing aid \n",
      "Original summary: india declares cr relief aid for hurricane hit countries \n",
      "Predicted summary:  india to emergency landing in india\n",
      "\n",
      "\n",
      "Review: stephanie adams former playboy model year old son reportedly jumped death th floor hotel new york us friday adams involved custody dispute son estranged husband adams claimed direct descendant us presidents john adams john adams \n",
      "Original summary: former playboy model jumps to death in us with yr old son \n",
      "Predicted summary:  ex boyfriend son jumps to death in son death in us\n",
      "\n",
      "\n",
      "Review: woman ias officer accused haryana additional chief secretary sunil gulati sexual harassment said fears life added incidents happened action taken officers complained meanwhile gulati denied allegations said ready face probe \n",
      "Original summary: fear for my life officer post accusing senior of harassment \n",
      "Predicted summary:  woman who asked me to resign for sexual harassment claims\n",
      "\n",
      "\n",
      "Review: national payments corporation india thursday launched upgraded version unified payments interface upi customers link overdraft account upi addition current savings account transaction limit doubled lakh upi also allows customers check invoice making payment merchant \n",
      "Original summary: what are the key features of upi launched on august \n",
      "Predicted summary:  india launches payments bank payments bank\n",
      "\n",
      "\n",
      "Review: russian president vladimir putin said us counterpart donald trump wants find culprit surge oil prices needs look mirror said contributing factors fluctuating oil prices include political problems venezuela collapse libya government \n",
      "Original summary: look in mirror to find oil price rise culprit putin to trump \n",
      "Predicted summary:  trump says he is not making oil diesel\n",
      "\n",
      "\n",
      "Review: uber sued woman named diamond claims rejected full time job uber enough lawsuit claiming gender discrimination diamond said allegedly replaced acting photo brand manager man less experience also claims upon confrontation uber supervisor said dude \n",
      "Original summary: denied job for not being uber enough claims woman \n",
      "Predicted summary:  uber sues woman for not paying woman for lawsuit\n",
      "\n",
      "\n",
      "Review: anand indian origin senior facebook executive named head workplace social media giant communications tool businesses anand btech graduate iiit hyderabad also studied kellogg school management us worked microsoft years joining facebook nearly four years ago \n",
      "Original summary: iiit hyderabad graduate anand to head facebook workplace \n",
      "Predicted summary:  facebook co founder facebook facebook new tech mahindra\n",
      "\n",
      "\n",
      "Review: social media platforms youtube facebook removing videos show people eating detergent part challenge platforms said policies allow promotion physical harm self injury comes tide pod challenge went viral social media part people ingested detergent pods \n",
      "Original summary: youtube facebook remove videos of people eating \n",
      "Predicted summary:  facebook videos of people in pictures of\n",
      "\n",
      "\n",
      "Review: former indian cricketer ajit agarkar scored one international hundred career came test england lord july batting number eight agarkar slammed unbeaten becoming seventh indian smash test ton lord agarkar turns today also hit fastest fifty indian odis \n",
      "Original summary: agarkar scored only ton in int cricket it was at lord \n",
      "Predicted summary:  indian cricketer scores runs in test for st time in yrs\n",
      "\n",
      "\n",
      "Review: rbi th governor urjit patel whose term end september resigned monday sir osborne smith first governor resigned differences government benegal rama rau quit due differences finance minister kr puri rn malhotra stepped respectively change governments \n",
      "Original summary: which rbi governors have resigned before their term ended \n",
      "Predicted summary:  rbi governor patel resigns as rbi governor\n",
      "\n",
      "\n",
      "Review: actress rajshri deshpande went topless scenes web series sacred games revealed getting messages saying porn star made saying hot indian actress mangalsutra scene made way porn sites added rajshri said ignore comments terribly cheap \n",
      "Original summary: got messages that am porn star sacred games actress on nude scene \n",
      "Predicted summary:  was not allowed to remove porn from porn film actress\n",
      "\n",
      "\n",
      "Review: haryana man mother arrested buying odisha based minor girl lakh keeping hostage two months said police accused forcefully marrying sexually assaulting girl tricked marriage uncle escaped jumping second floor accused house \n",
      "Original summary: haryana man buys minor for lakh for marriage rapes her \n",
      "Predicted summary:  haryana man arrested for molesting girl in haryana\n",
      "\n",
      "\n",
      "Review: new song diljit dosanjh sonakshi sinha starrer welcome new york released sung rahat fateh ali khan bhanushali composed tandon lyrics penned charan also starring karan johar lara dutta riteish deshmukh boman irani film scheduled release february \n",
      "Original summary: new song from diljit sonakshi welcome to new york out \n",
      "Predicted summary:  new song from diljit sonakshi starrer dhadak released\n",
      "\n",
      "\n",
      "Review: man uttar pradesh kaushambi forced carry body niece shoulder riding bicycle allegedly denied ambulance government hospital per reports hospital also asked man pay fuel cost ambulance wanted avail take body home \n",
      "Original summary: denied ambulance man cycles carrying dead niece on shoulder \n",
      "Predicted summary:  man beaten up for not ambulance on road hospital\n",
      "\n",
      "\n",
      "Review: finance minister arun jaitley monday said single standard gst rate could introduced instead two standard rates added lowering tax rate cement priority congress called new indirect tax four rates besides exempted items cess complex system \n",
      "Original summary: india is working towards single gst rate arun jaitley \n",
      "Predicted summary:  gst rate to be reduced to gst arun jaitley\n",
      "\n",
      "\n",
      "Review: veteran congress leader sm krishna joined bjp wednesday slammed party vice president rahul gandhi saying politics neither hit run job part time job added long sonia gandhi comfortable congress past three years leaders like made feel unwanted \n",
      "Original summary: rahul gandhi hit and run politics will not work krishna \n",
      "Predicted summary:  rahul gandhi is the job of politics rahul\n",
      "\n",
      "\n",
      "Review: isro thursday displayed prototype space suit crew module india first manned mission scheduled bengaluru space expo suit fitted one oxygen cylinder last minutes crew capsule fitted thermal shield three astronauts orbiting earth km altitude days \n",
      "Original summary: isro unveils prototype of space suit crew module for mission \n",
      "Predicted summary:  isro space station crew completes st space space\n",
      "\n",
      "\n",
      "Review: uk court denied bail underworld dawood ibrahim aide motiwala second time motiwala arrested hotel london last month described uk court proceedings top lieutenant company us filed extradition case money laundering extortion charges \n",
      "Original summary: uk court denies bail to dawood ibrahim aide \n",
      "Predicted summary:  uk court rejects bail to dawood ibrahim aide\n",
      "\n",
      "\n",
      "Review: delhi based ganesh mandal association announced plans symbolically immerse ganesh idols large tub instead yamuna river ahead ganesh chaturthi association also encouraged delhi residents immerse idols four water set durga pandals city reportedly plan follow similar eco friendly measures durga puja \n",
      "Original summary: delhi residents to not ganesh idols in yamuna \n",
      "Predicted summary:  delhi civic body to get into gorge in gorge\n",
      "\n",
      "\n",
      "Review: illegal shipment live crocodiles seized london heathrow airport friday authorities said crocodiles crammed five boxes packed accordance international regulations made illegal crocodiles supposed bred meat one died others rehabilitated \n",
      "Original summary: live crocodiles into boxes seized at uk airport \n",
      "Predicted summary:  uk airport finds cocaine in london airport\n",
      "\n",
      "\n",
      "Review: dentsu india said happy ceo kartik iyer md praveen das dasgupta india dinesh swamy stepped posts following sexual harassment allegations dentsu acquired advertising company happy creative services rebranded happy zero tolerance policy towards harassment dentsu said \n",
      "Original summary: removes popular amid sexual harassment claims \n",
      "Predicted summary:  india ceo kapoor responds to india ceo responds to india\n",
      "\n",
      "\n",
      "Review: supreme court monday refused quash pune police fir activist anand teltumbde alleged role bhima koregaon violence alleged maoist links noted investigation getting bigger bigger refused interfere court granted teltumbde four week extension interim protection arrest \n",
      "Original summary: sc refuses to fir against bhima koregaon accused \n",
      "Predicted summary:  sc denies fir against fir against fir against bhima koregaon violence\n",
      "\n",
      "\n",
      "Review: social media giant facebook testing resume feature lets users list contact details along professional experience education facebook accounts feature allows users add image qualifications combines information single tab feature reportedly expansion facebook work education section \n",
      "Original summary: facebook tests resume feature on users accounts \n",
      "Predicted summary:  facebook tests feature to let users share photos\n",
      "\n",
      "\n",
      "Review: american airlines group inc united continental holdings inc aviation defendants agreed pay crore developer world trade center new york developer accused airlines negligence failing prevent twin towers destruction hijacked planes around people died terror attack \n",
      "Original summary: us airlines to pay crore over terror attack claims \n",
      "Predicted summary:  united airlines ceo pay crore to fly world bank\n",
      "\n",
      "\n",
      "Review: whatsapp co founder jan koum selling porsche cars us auction march year unlimited time unlimited garage space would never sell said koum cars include trio yellow carrera rs models porsche covered kilometres \n",
      "Original summary: whatsapp co founder to sell does not have time to drive \n",
      "Predicted summary:  whatsapp co founder sells for lakh for auction\n",
      "\n",
      "\n",
      "Review: uncapped year old indian batsman mayank agarwal broken sachin tendulkar year old record runs indian list tournament agarwal slammed runs eight matches vijay hazare trophy surpassing tendulkar slammed runs world cup agarwal player slam runs indian domestic season \n",
      "Original summary: uncapped indian batsman breaks sachin year old record \n",
      "Predicted summary:  sachin tendulkar record most indian cricketer sachin\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for i in range(0,100):\n",
    "    print(\"Review:\",seq2text(x_tr[i]))\n",
    "    print(\"Original summary:\",seq2summary(y_tr[i]))\n",
    "    print(\"Predicted summary:\",decode_sequence(x_tr[i].reshape(1,max_text_len)))\n",
    "    print(\"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "OTkaYNjHW4lC"
   },
   "source": [
    "This is really cool stuff. Even though the actual summary and the summary generated by our model do not match in terms of words, both of them are conveying the same meaning. Our model is able to generate a legible summary based on the context present in the text.\n",
    "\n",
    "This is how we can perform text summarization using deep learning concepts in Python.\n",
    "\n",
    "#How can we Improve the Model’s Performance Even Further?\n",
    "\n",
    "\n",
    "* We can **increase the training dataset** size and build the model. The generalization capability of a deep learning model enhances with an increase in the training dataset size\n",
    "\n",
    "* We can try implementing **Bi-Directional LSTM** which is capable of capturing the context from both the directions and results in a better context vector\n",
    "\n",
    "* We can use the **beam search strategy** for decoding the test sequence instead of using the greedy approach (argmax)\n",
    "\n",
    "* We can evaluate the performance of the model based on the **BLEU score**\n",
    "\n",
    "* We can implement **pointer-generator networks** and **coverage mechanisms**\n",
    " \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "dC1koV7CFNto"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "text_summarization_with_LSTM_Attention.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}