Out of scope intent imbalanced classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Acknowledgement\n",
    "Dataset:https://www.kaggle.com/stefanlarson/outofscope-intent-classification-dataset\n",
    "\n",
    "Context:\n",
    "\n",
    "Most supervised machine learning tasks assume a dataset with a set of well-defined target label set. But what happens when a trained model meets the real world, where inputs to the trained model might not be from the well-defined target label set? This dataset offers a way to evaluate intent classification models on \"out-of-scope\" inputs.\n",
    "\n",
    "\"Out-of-scope\" inputs are those that do not belong to the set of \"in-scope\" target labels. You may have heard other ways of referring to out-of-scope, including \"out-of-domain\" or \"out-of-distribution\".\n",
    "Content\n",
    "\n",
    "    is_*.json: these files house the train/val/test sets for the in-scope data. There are 150 in-scope \"intents\" (aka classes), which include samples such as \"what is my balance\" (which belongs to the balance class).\n",
    "    oos_*.json: these files house the train/val/test sets for the out-of-scope data. There is one out-of-scope intent: oos. Note that you don't have to use the oos_train.json data. In other words, an ML solution to the out-of-scope problem need not be trained on out-of-scope data, but it might help!\n",
    "\n",
    "Evaluation Metrics:\n",
    "\n",
    "The task is intent classification, which generalizes to text classification (or categorization). This is a supervised ML problem. We use two metrics to evaluate:\n",
    "\n",
    "    In-scope accuracy is defined as #(correctly classified in-scope samples) / #(in-scope samples).\n",
    "    Out-of-scope recall is defined as #(correctly classified out-of-scope samples) / #(out-of-scope samples).\n",
    "\n",
    "Acknowledgements:\n",
    "\n",
    "This dataset is from An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction by Larson et al., which was published in EMNLP in 2019. The GitHub page for this dataset is linked here.\n",
    "Inspiration\n",
    "\n",
    "Most supervised machine learning tasks assume a dataset with a set of well-defined target label set. But what happens when a trained model meets the real world, where inputs to the trained model might not be from the well-defined target label set? This \"out-of-distribution\" problem has seen lots of recent development, as researchers and practitioners in both academia and industry are observing that many ML methods struggle on out-of-distribution data in a wide variety of tasks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Import dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
    "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/kaggle/input/outofscope-intent-classification-dataset/oos_test.json\n",
      "/kaggle/input/outofscope-intent-classification-dataset/is_train.json\n",
      "/kaggle/input/outofscope-intent-classification-dataset/oos_val.json\n",
      "/kaggle/input/outofscope-intent-classification-dataset/oos_train.json\n",
      "/kaggle/input/outofscope-intent-classification-dataset/is_val.json\n",
      "/kaggle/input/outofscope-intent-classification-dataset/is_test.json\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "import os\n",
    "import numpy as np \n",
    "import pandas as pd \n",
    "from sklearn.svm import SVC\n",
    "import plotly.express as px\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.model_selection import train_test_split\n",
    "from imblearn.under_sampling import RandomUnderSampler\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.linear_model import SGDClassifier, LogisticRegression\n",
    "from sklearn.metrics import recall_score,precision_score,make_scorer\n",
    "from sklearn.ensemble import RandomForestClassifier,AdaBoostClassifier\n",
    "from imblearn.ensemble import BalancedRandomForestClassifier,EasyEnsembleClassifier,RUSBoostClassifier\n",
    "\n",
    "pd.set_option('display.max_rows', 700)\n",
    "for dirname, _, filenames in os.walk('/kaggle/input'):\n",
    "    for filename in filenames:\n",
    "        print(os.path.join(dirname, filename))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Load and clean data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "_cell_guid": "79c7e3d0-c299-4dcb-8224-4455121ee9b0",
    "_uuid": "d629ff2d2480ee46fbb7e2d37f6b5fab8052498a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train shape:(15000, 2), train has 0 null values\n",
      "val shape:(3000, 2), val has 0 null values\n",
      "test shape:(4500, 2), test has 0 null values\n",
      "oos_train shape:(100, 2), oos_train has 0 null values\n",
      "oos_val shape:(100, 2), oos_val has 0 null values\n",
      "oos_test shape:(1000, 2), oos_test has 0 null values\n"
     ]
    }
   ],
   "source": [
    "train = pd.read_json('/kaggle/input/outofscope-intent-classification-dataset/is_train.json')\n",
    "val = pd.read_json('/kaggle/input/outofscope-intent-classification-dataset/is_val.json')\n",
    "test = pd.read_json('/kaggle/input/outofscope-intent-classification-dataset/is_test.json')\n",
    "oos_train = pd.read_json('/kaggle/input/outofscope-intent-classification-dataset/oos_train.json')\n",
    "oos_val = pd.read_json('/kaggle/input/outofscope-intent-classification-dataset/oos_val.json')\n",
    "oos_test = pd.read_json('/kaggle/input/outofscope-intent-classification-dataset/oos_test.json')\n",
    "files = [(train,'train'),(val,'val'),(test,'test'),(oos_train,'oos_train'),(oos_val,'oos_val'),(oos_test,'oos_test')]\n",
    "for file,name in files:\n",
    "    file.columns = ['text','intent']\n",
    "    print(f'{name} shape:{file.shape}, {name} has {train.isna().sum().sum()} null values')\n",
    "in_train = train.copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "gas                          100\n",
       "who_do_you_work_for          100\n",
       "pto_balance                  100\n",
       "user_name                    100\n",
       "timer                        100\n",
       "how_old_are_you              100\n",
       "credit_limit                 100\n",
       "meaning_of_life              100\n",
       "bill_due                     100\n",
       "greeting                     100\n",
       "travel_alert                 100\n",
       "restaurant_reservation       100\n",
       "pin_change                   100\n",
       "roll_dice                    100\n",
       "report_lost_card             100\n",
       "confirm_reservation          100\n",
       "recipe                       100\n",
       "reminder_update              100\n",
       "credit_limit_change          100\n",
       "who_made_you                 100\n",
       "w2                           100\n",
       "how_busy                     100\n",
       "lost_luggage                 100\n",
       "time                         100\n",
       "interest_rate                100\n",
       "maybe                        100\n",
       "update_playlist              100\n",
       "cancel_reservation           100\n",
       "weather                      100\n",
       "cancel                       100\n",
       "oil_change_when              100\n",
       "apr                          100\n",
       "carry_on                     100\n",
       "spelling                     100\n",
       "pto_used                     100\n",
       "restaurant_suggestion        100\n",
       "calendar_update              100\n",
       "exchange_rate                100\n",
       "whisper_mode                 100\n",
       "restaurant_reviews           100\n",
       "gas_type                     100\n",
       "schedule_meeting             100\n",
       "flight_status                100\n",
       "order_status                 100\n",
       "sync_device                  100\n",
       "balance                      100\n",
       "text                         100\n",
       "bill_balance                 100\n",
       "meeting_schedule             100\n",
       "tell_joke                    100\n",
       "smart_home                   100\n",
       "calendar                     100\n",
       "change_language              100\n",
       "thank_you                    100\n",
       "order_checks                 100\n",
       "shopping_list                100\n",
       "current_location             100\n",
       "fun_fact                     100\n",
       "account_blocked              100\n",
       "application_status           100\n",
       "timezone                     100\n",
       "no                           100\n",
       "food_last                    100\n",
       "next_song                    100\n",
       "what_are_your_hobbies        100\n",
       "reset_settings               100\n",
       "ingredients_list             100\n",
       "income                       100\n",
       "insurance_change             100\n",
       "replacement_card_duration    100\n",
       "international_visa           100\n",
       "transfer                     100\n",
       "uber                         100\n",
       "book_hotel                   100\n",
       "are_you_a_bot                100\n",
       "todo_list                    100\n",
       "book_flight                  100\n",
       "payday                       100\n",
       "oil_change_how               100\n",
       "next_holiday                 100\n",
       "definition                   100\n",
       "last_maintenance             100\n",
       "travel_notification          100\n",
       "tire_change                  100\n",
       "what_can_i_ask_you           100\n",
       "credit_score                 100\n",
       "min_payment                  100\n",
       "international_fees           100\n",
       "make_call                    100\n",
       "pay_bill                     100\n",
       "pto_request_status           100\n",
       "insurance                    100\n",
       "what_song                    100\n",
       "nutrition_info               100\n",
       "change_ai_name               100\n",
       "ingredient_substitution      100\n",
       "play_music                   100\n",
       "translate                    100\n",
       "traffic                      100\n",
       "vaccines                     100\n",
       "plug_type                    100\n",
       "pto_request                  100\n",
       "schedule_maintenance         100\n",
       "find_phone                   100\n",
       "distance                     100\n",
       "redeem_rewards               100\n",
       "cook_time                    100\n",
       "change_accent                100\n",
       "expiration_date              100\n",
       "what_is_your_name            100\n",
       "repeat                       100\n",
       "shopping_list_update         100\n",
       "damaged_card                 100\n",
       "tire_pressure                100\n",
       "order                        100\n",
       "date                         100\n",
       "taxes                        100\n",
       "accept_reservations          100\n",
       "rollover_401k                100\n",
       "new_card                     100\n",
       "change_speed                 100\n",
       "do_you_have_pets             100\n",
       "jump_start                   100\n",
       "meal_suggestion              100\n",
       "yes                          100\n",
       "directions                   100\n",
       "change_user_name             100\n",
       "reminder                     100\n",
       "where_are_you_from           100\n",
       "spending_history             100\n",
       "calories                     100\n",
       "measurement_conversion       100\n",
       "share_location               100\n",
       "freeze_account               100\n",
       "improve_credit_score         100\n",
       "todo_list_update             100\n",
       "change_volume                100\n",
       "rewards_balance              100\n",
       "alarm                        100\n",
       "flip_coin                    100\n",
       "travel_suggestion            100\n",
       "report_fraud                 100\n",
       "card_declined                100\n",
       "mpg                          100\n",
       "routing                      100\n",
       "calculator                   100\n",
       "car_rental                   100\n",
       "transactions                 100\n",
       "direct_deposit               100\n",
       "goodbye                      100\n",
       "Name: intent, dtype: int64"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "in_train.intent.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All in-scope intents are balanced, no sampling required"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check if any text sampe has email,url,mention and hashtags"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def ngrams_top(corpus,ngram_range,n=10):\n",
    "    vec = CountVectorizer(stop_words = 'english',ngram_range=ngram_range).fit(corpus)\n",
    "    bag_of_words = vec.transform(corpus)\n",
    "    sum_words = bag_of_words.sum(axis=0) \n",
    "    words_freq = [(word, sum_words[0, idx]) for word, idx in vec.vocabulary_.items()]\n",
    "    words_freq =sorted(words_freq, key = lambda x: x[1], reverse=True)\n",
    "    total_list=words_freq[:n]\n",
    "    df=pd.DataFrame(total_list,columns=['text','count'])\n",
    "    return df\n",
    "\n",
    "def get_emails(x):\n",
    "    email = re.findall(r'[\\w\\.-]+@[\\w-]+\\.[\\w]+',str(x))\n",
    "    return \" \".join(email)\n",
    "def get_urls(x):\n",
    "    url = re.findall('http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\\(\\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+\\.[\\w]+',str(x))\n",
    "    return \" \".join(url)\n",
    "def get_mentions(x):\n",
    "    mention = re.findall(r'(?<=@)\\w+',str(x))\n",
    "    return \" \".join(mention)\n",
    "def get_hashtags(x):\n",
    "    hashtag = re.findall(r'(?<=#)\\w+',str(x))\n",
    "    return \" \".join(hashtag)\n",
    "def text_at_a_glance(df):\n",
    "    res = df.apply(get_emails)\n",
    "    res = res[res.values!=\"\"]\n",
    "    print(\"Data has {} rows with emails\".format(len(res)))\n",
    "    res = df.apply(get_urls)\n",
    "    res = res[res.values!=\"\"]\n",
    "    print(\"Data has {} rows with urls\".format(len(res)))\n",
    "    res = df.apply(get_mentions)\n",
    "    res = res[res.values!=\"\"]\n",
    "    print(\"Data has {} rows with mentions\".format(len(res)))\n",
    "    res = df.apply(get_hashtags)\n",
    "    res = res[res.values!=\"\"]\n",
    "    print(\"Data has {} rows with hashtags\".format(len(res)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data has 0 rows with emails\n",
      "Data has 0 rows with urls\n",
      "Data has 0 rows with mentions\n",
      "Data has 1 rows with hashtags\n"
     ]
    }
   ],
   "source": [
    "text_at_a_glance(in_train.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "13644    put mambo #5 by lou bega on my playlist, please\n",
       "Name: text, dtype: object"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#check where hashtag has been found, punctuations will be taken care of by vectorizer()\n",
    "temp = in_train.text.apply(get_hashtags)\n",
    "in_train.iloc[temp[temp.values!=\"\"].index].text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ngrams"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "        <script type=\"text/javascript\">\n",
       "        window.PlotlyConfig = {MathJaxConfig: 'local'};\n",
       "        if (window.MathJax) {MathJax.Hub.Config({SVG: {font: \"STIX-Web\"}});}\n",
       "        if (typeof require !== 'undefined') {\n",
       "        require.undef(\"plotly\");\n",
       "        requirejs.config({\n",
       "            paths: {\n",
       "                'plotly': ['https://cdn.plot.ly/plotly-latest.min']\n",
       "            }\n",
       "        });\n",
       "        require(['plotly'], function(Plotly) {\n",
       "            window._Plotly = Plotly;\n",
       "        });\n",
       "        }\n",
       "        </script>\n",
       "        "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "        \n",
       "        \n",
       "            <div id=\"940b6818-6987-4010-9f3f-9c378cd9efe4\" class=\"plotly-graph-div\" style=\"height:500px; width:900px;\"></div>\n",
       "            <script type=\"text/javascript\">\n",
       "                require([\"plotly\"], function(Plotly) {\n",
       "                    window.PLOTLYENV=window.PLOTLYENV || {};\n",
       "                    \n",
       "                if (document.getElementById(\"940b6818-6987-4010-9f3f-9c378cd9efe4\")) {\n",
       "                    Plotly.newPlot(\n",
       "                        '940b6818-6987-4010-9f3f-9c378cd9efe4',\n",
       "                        [{\"alignmentgroup\": \"True\", \"hovertemplate\": \"count=%{x}<br>text=%{y}<extra></extra>\", \"legendgroup\": \"\", \"marker\": {\"color\": \"#dc3912\", \"opacity\": 0.8}, \"name\": \"\", \"offsetgroup\": \"\", \"orientation\": \"h\", \"showlegend\": false, \"textposition\": \"auto\", \"type\": \"bar\", \"x\": [93, 98, 99, 111, 114, 126, 152, 193, 205, 335], \"xaxis\": \"x\", \"y\": [\"change oil\", \"visa card\", \"routing number\", \"bank account\", \"let know\", \"credit limit\", \"credit score\", \"shopping list\", \"need know\", \"credit card\"], \"yaxis\": \"y\"}],\n",
       "                        {\"barmode\": \"relative\", \"height\": 500, \"legend\": {\"tracegroupgap\": 0}, \"margin\": {\"t\": 60}, \"template\": {\"data\": {\"bar\": [{\"error_x\": {\"color\": \"#2a3f5f\"}, \"error_y\": {\"color\": \"#2a3f5f\"}, \"marker\": {\"line\": {\"color\": \"#E5ECF6\", \"width\": 0.5}}, \"type\": \"bar\"}], \"barpolar\": [{\"marker\": {\"line\": {\"color\": \"#E5ECF6\", \"width\": 0.5}}, \"type\": \"barpolar\"}], \"carpet\": [{\"aaxis\": {\"endlinecolor\": \"#2a3f5f\", \"gridcolor\": \"white\", \"linecolor\": \"white\", \"minorgridcolor\": \"white\", \"startlinecolor\": \"#2a3f5f\"}, \"baxis\": {\"endlinecolor\": \"#2a3f5f\", \"gridcolor\": \"white\", \"linecolor\": \"white\", \"minorgridcolor\": \"white\", \"startlinecolor\": \"#2a3f5f\"}, \"type\": \"carpet\"}], \"choropleth\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"choropleth\"}], \"contour\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"contour\"}], \"contourcarpet\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"contourcarpet\"}], \"heatmap\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"heatmap\"}], \"heatmapgl\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"heatmapgl\"}], \"histogram\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"histogram\"}], \"histogram2d\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"histogram2d\"}], \"histogram2dcontour\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"histogram2dcontour\"}], \"mesh3d\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"mesh3d\"}], \"parcoords\": [{\"line\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"parcoords\"}], \"pie\": [{\"automargin\": true, \"type\": \"pie\"}], \"scatter\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatter\"}], \"scatter3d\": [{\"line\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatter3d\"}], \"scattercarpet\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattercarpet\"}], \"scattergeo\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattergeo\"}], \"scattergl\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattergl\"}], \"scattermapbox\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattermapbox\"}], \"scatterpolar\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterpolar\"}], \"scatterpolargl\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterpolargl\"}], \"scatterternary\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterternary\"}], \"surface\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"surface\"}], \"table\": [{\"cells\": {\"fill\": {\"color\": \"#EBF0F8\"}, \"line\": {\"color\": \"white\"}}, \"header\": {\"fill\": {\"color\": \"#C8D4E3\"}, \"line\": {\"color\": \"white\"}}, \"type\": \"table\"}]}, \"layout\": {\"annotationdefaults\": {\"arrowcolor\": \"#2a3f5f\", \"arrowhead\": 0, \"arrowwidth\": 1}, \"coloraxis\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"colorscale\": {\"diverging\": [[0, \"#8e0152\"], [0.1, \"#c51b7d\"], [0.2, \"#de77ae\"], [0.3, \"#f1b6da\"], [0.4, \"#fde0ef\"], [0.5, \"#f7f7f7\"], [0.6, \"#e6f5d0\"], [0.7, \"#b8e186\"], [0.8, \"#7fbc41\"], [0.9, \"#4d9221\"], [1, \"#276419\"]], \"sequential\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"sequentialminus\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]]}, \"colorway\": [\"#636efa\", \"#EF553B\", \"#00cc96\", \"#ab63fa\", \"#FFA15A\", \"#19d3f3\", \"#FF6692\", \"#B6E880\", \"#FF97FF\", \"#FECB52\"], \"font\": {\"color\": \"#2a3f5f\"}, \"geo\": {\"bgcolor\": \"white\", \"lakecolor\": \"white\", \"landcolor\": \"#E5ECF6\", \"showlakes\": true, \"showland\": true, \"subunitcolor\": \"white\"}, \"hoverlabel\": {\"align\": \"left\"}, \"hovermode\": \"closest\", \"mapbox\": {\"style\": \"light\"}, \"paper_bgcolor\": \"white\", \"plot_bgcolor\": \"#E5ECF6\", \"polar\": {\"angularaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"bgcolor\": \"#E5ECF6\", \"radialaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}}, \"scene\": {\"xaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}, \"yaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}, \"zaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}}, \"shapedefaults\": {\"line\": {\"color\": \"#2a3f5f\"}}, \"ternary\": {\"aaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"baxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"bgcolor\": \"#E5ECF6\", \"caxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}}, \"title\": {\"x\": 0.05}, \"xaxis\": {\"automargin\": true, \"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\", \"title\": {\"standoff\": 15}, \"zerolinecolor\": \"white\", \"zerolinewidth\": 2}, \"yaxis\": {\"automargin\": true, \"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\", \"title\": {\"standoff\": 15}, \"zerolinecolor\": \"white\", \"zerolinewidth\": 2}}}, \"width\": 900, \"xaxis\": {\"anchor\": \"y\", \"domain\": [0.0, 1.0], \"title\": {\"text\": \"count\"}}, \"yaxis\": {\"anchor\": \"x\", \"domain\": [0.0, 1.0], \"title\": {\"text\": \"text\"}}},\n",
       "                        {\"responsive\": true}\n",
       "                    ).then(function(){\n",
       "                            \n",
       "var gd = document.getElementById('940b6818-6987-4010-9f3f-9c378cd9efe4');\n",
       "var x = new MutationObserver(function (mutations, observer) {{\n",
       "        var display = window.getComputedStyle(gd).display;\n",
       "        if (!display || display === 'none') {{\n",
       "            console.log([gd, 'removed!']);\n",
       "            Plotly.purge(gd);\n",
       "            observer.disconnect();\n",
       "        }}\n",
       "}});\n",
       "\n",
       "// Listen for the removal of the full notebook cells\n",
       "var notebookContainer = gd.closest('#notebook-container');\n",
       "if (notebookContainer) {{\n",
       "    x.observe(notebookContainer, {childList: true});\n",
       "}}\n",
       "\n",
       "// Listen for the clearing of the current output cell\n",
       "var outputEl = gd.closest('.output');\n",
       "if (outputEl) {{\n",
       "    x.observe(outputEl, {childList: true});\n",
       "}}\n",
       "\n",
       "                        })\n",
       "                };\n",
       "                });\n",
       "            </script>\n",
       "        </div>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#top bigrams\n",
    "bigrams = ngrams_top(in_train.text,(2,2)).sort_values(by='count')\n",
    "px.bar(data_frame=bigrams,y='text',x='count',orientation='h',color_discrete_sequence=['#dc3912'],opacity=0.8,width=900,height=500)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "        \n",
       "        \n",
       "            <div id=\"7c0fe945-c3a8-42dd-8a7e-e6a1eabcd285\" class=\"plotly-graph-div\" style=\"height:500px; width:900px;\"></div>\n",
       "            <script type=\"text/javascript\">\n",
       "                require([\"plotly\"], function(Plotly) {\n",
       "                    window.PLOTLYENV=window.PLOTLYENV || {};\n",
       "                    \n",
       "                if (document.getElementById(\"7c0fe945-c3a8-42dd-8a7e-e6a1eabcd285\")) {\n",
       "                    Plotly.newPlot(\n",
       "                        '7c0fe945-c3a8-42dd-8a7e-e6a1eabcd285',\n",
       "                        [{\"alignmentgroup\": \"True\", \"hovertemplate\": \"count=%{x}<br>text=%{y}<extra></extra>\", \"legendgroup\": \"\", \"marker\": {\"color\": \"#f58518\", \"opacity\": 0.8}, \"name\": \"\", \"offsetgroup\": \"\", \"orientation\": \"h\", \"showlegend\": false, \"textposition\": \"auto\", \"type\": \"bar\", \"x\": [25, 25, 27, 29, 29, 38, 44, 45, 48, 54], \"xaxis\": \"x\", \"y\": [\"let bank know\", \"change pin number\", \"change oil car\", \"american express card\", \"jump start car\", \"increase credit limit\", \"credit card application\", \"set direct deposit\", \"new credit card\", \"credit card points\"], \"yaxis\": \"y\"}],\n",
       "                        {\"barmode\": \"relative\", \"height\": 500, \"legend\": {\"tracegroupgap\": 0}, \"margin\": {\"t\": 60}, \"template\": {\"data\": {\"bar\": [{\"error_x\": {\"color\": \"#2a3f5f\"}, \"error_y\": {\"color\": \"#2a3f5f\"}, \"marker\": {\"line\": {\"color\": \"#E5ECF6\", \"width\": 0.5}}, \"type\": \"bar\"}], \"barpolar\": [{\"marker\": {\"line\": {\"color\": \"#E5ECF6\", \"width\": 0.5}}, \"type\": \"barpolar\"}], \"carpet\": [{\"aaxis\": {\"endlinecolor\": \"#2a3f5f\", \"gridcolor\": \"white\", \"linecolor\": \"white\", \"minorgridcolor\": \"white\", \"startlinecolor\": \"#2a3f5f\"}, \"baxis\": {\"endlinecolor\": \"#2a3f5f\", \"gridcolor\": \"white\", \"linecolor\": \"white\", \"minorgridcolor\": \"white\", \"startlinecolor\": \"#2a3f5f\"}, \"type\": \"carpet\"}], \"choropleth\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"choropleth\"}], \"contour\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"contour\"}], \"contourcarpet\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"contourcarpet\"}], \"heatmap\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"heatmap\"}], \"heatmapgl\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"heatmapgl\"}], \"histogram\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"histogram\"}], \"histogram2d\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"histogram2d\"}], \"histogram2dcontour\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"histogram2dcontour\"}], \"mesh3d\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"mesh3d\"}], \"parcoords\": [{\"line\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"parcoords\"}], \"pie\": [{\"automargin\": true, \"type\": \"pie\"}], \"scatter\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatter\"}], \"scatter3d\": [{\"line\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatter3d\"}], \"scattercarpet\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattercarpet\"}], \"scattergeo\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattergeo\"}], \"scattergl\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattergl\"}], \"scattermapbox\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattermapbox\"}], \"scatterpolar\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterpolar\"}], \"scatterpolargl\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterpolargl\"}], \"scatterternary\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterternary\"}], \"surface\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"surface\"}], \"table\": [{\"cells\": {\"fill\": {\"color\": \"#EBF0F8\"}, \"line\": {\"color\": \"white\"}}, \"header\": {\"fill\": {\"color\": \"#C8D4E3\"}, \"line\": {\"color\": \"white\"}}, \"type\": \"table\"}]}, \"layout\": {\"annotationdefaults\": {\"arrowcolor\": \"#2a3f5f\", \"arrowhead\": 0, \"arrowwidth\": 1}, \"coloraxis\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"colorscale\": {\"diverging\": [[0, \"#8e0152\"], [0.1, \"#c51b7d\"], [0.2, \"#de77ae\"], [0.3, \"#f1b6da\"], [0.4, \"#fde0ef\"], [0.5, \"#f7f7f7\"], [0.6, \"#e6f5d0\"], [0.7, \"#b8e186\"], [0.8, \"#7fbc41\"], [0.9, \"#4d9221\"], [1, \"#276419\"]], \"sequential\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"sequentialminus\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]]}, \"colorway\": [\"#636efa\", \"#EF553B\", \"#00cc96\", \"#ab63fa\", \"#FFA15A\", \"#19d3f3\", \"#FF6692\", \"#B6E880\", \"#FF97FF\", \"#FECB52\"], \"font\": {\"color\": \"#2a3f5f\"}, \"geo\": {\"bgcolor\": \"white\", \"lakecolor\": \"white\", \"landcolor\": \"#E5ECF6\", \"showlakes\": true, \"showland\": true, \"subunitcolor\": \"white\"}, \"hoverlabel\": {\"align\": \"left\"}, \"hovermode\": \"closest\", \"mapbox\": {\"style\": \"light\"}, \"paper_bgcolor\": \"white\", \"plot_bgcolor\": \"#E5ECF6\", \"polar\": {\"angularaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"bgcolor\": \"#E5ECF6\", \"radialaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}}, \"scene\": {\"xaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}, \"yaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}, \"zaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}}, \"shapedefaults\": {\"line\": {\"color\": \"#2a3f5f\"}}, \"ternary\": {\"aaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"baxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"bgcolor\": \"#E5ECF6\", \"caxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}}, \"title\": {\"x\": 0.05}, \"xaxis\": {\"automargin\": true, \"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\", \"title\": {\"standoff\": 15}, \"zerolinecolor\": \"white\", \"zerolinewidth\": 2}, \"yaxis\": {\"automargin\": true, \"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\", \"title\": {\"standoff\": 15}, \"zerolinecolor\": \"white\", \"zerolinewidth\": 2}}}, \"width\": 900, \"xaxis\": {\"anchor\": \"y\", \"domain\": [0.0, 1.0], \"title\": {\"text\": \"count\"}}, \"yaxis\": {\"anchor\": \"x\", \"domain\": [0.0, 1.0], \"title\": {\"text\": \"text\"}}},\n",
       "                        {\"responsive\": true}\n",
       "                    ).then(function(){\n",
       "                            \n",
       "var gd = document.getElementById('7c0fe945-c3a8-42dd-8a7e-e6a1eabcd285');\n",
       "var x = new MutationObserver(function (mutations, observer) {{\n",
       "        var display = window.getComputedStyle(gd).display;\n",
       "        if (!display || display === 'none') {{\n",
       "            console.log([gd, 'removed!']);\n",
       "            Plotly.purge(gd);\n",
       "            observer.disconnect();\n",
       "        }}\n",
       "}});\n",
       "\n",
       "// Listen for the removal of the full notebook cells\n",
       "var notebookContainer = gd.closest('#notebook-container');\n",
       "if (notebookContainer) {{\n",
       "    x.observe(notebookContainer, {childList: true});\n",
       "}}\n",
       "\n",
       "// Listen for the clearing of the current output cell\n",
       "var outputEl = gd.closest('.output');\n",
       "if (outputEl) {{\n",
       "    x.observe(outputEl, {childList: true});\n",
       "}}\n",
       "\n",
       "                        })\n",
       "                };\n",
       "                });\n",
       "            </script>\n",
       "        </div>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#top trigrams\n",
    "trigrams = ngrams_top(in_train.text,(3,3)).sort_values(by='count')\n",
    "px.bar(data_frame=trigrams,y='text',x='count',orientation='h',color_discrete_sequence=['#f58518'],opacity=0.8,width=900,height=500)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# In-scope prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def binarize(df):\n",
    "    df.intent = np.where(df.intent!='oos',0,1)\n",
    "    return df\n",
    "\n",
    "def vectorizer(X):\n",
    "    cv = CountVectorizer(min_df=1,ngram_range=(1,2))\n",
    "    X_en = cv.fit_transform(X)\n",
    "    return cv,X_en\n",
    "\n",
    "def labelencoder(y):\n",
    "    le = LabelEncoder()\n",
    "    le.fit(y)\n",
    "    y_enc = le.transform(y)\n",
    "    return le,y_enc\n",
    "\n",
    "def preprocess(train):\n",
    "    X = train.text\n",
    "    y = train.intent\n",
    "    le,y = labelencoder(y)\n",
    "    cv,X = vectorizer(X)\n",
    "    return X,y,cv,le\n",
    "\n",
    "def process_non_train(df,cv,le):\n",
    "    X = df.text\n",
    "    y = df.intent\n",
    "    X = cv.transform(X)\n",
    "    y = le.transform(y)\n",
    "    return X,y\n",
    "\n",
    "def get_score(clf,binary=0):\n",
    "    clf.fit(X_train,y_train)\n",
    "    if binary==1:\n",
    "        y_pred = clf.predict(X_test)\n",
    "        return clf,clf.score(X_val,y_val),clf.score(X_test,y_test),recall_score(y_test,y_pred),precision_score(y_test,y_pred)\n",
    "    elif binary==0:\n",
    "        return clf,clf.score(X_val,y_val),clf.score(X_test,y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train,y_train,cv,le = preprocess(in_train)\n",
    "X_val,y_val = process_non_train(val,cv,le)\n",
    "X_test,y_test = process_non_train(test,cv,le)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluate data over models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style  type=\"text/css\" >\n",
       "    #T_f50b6a3a_ad61_11ea_a39c_0242ac130202row0_col0 {\n",
       "            background-color:  #fff7fb;\n",
       "            color:  #000000;\n",
       "        }    #T_f50b6a3a_ad61_11ea_a39c_0242ac130202row0_col1 {\n",
       "            background-color:  #fff7fb;\n",
       "            color:  #000000;\n",
       "        }    #T_f50b6a3a_ad61_11ea_a39c_0242ac130202row1_col0 {\n",
       "            background-color:  #023858;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_f50b6a3a_ad61_11ea_a39c_0242ac130202row1_col1 {\n",
       "            background-color:  #023858;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_f50b6a3a_ad61_11ea_a39c_0242ac130202row2_col0 {\n",
       "            background-color:  #045b8f;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_f50b6a3a_ad61_11ea_a39c_0242ac130202row2_col1 {\n",
       "            background-color:  #04588a;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_f50b6a3a_ad61_11ea_a39c_0242ac130202row3_col0 {\n",
       "            background-color:  #034f7d;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_f50b6a3a_ad61_11ea_a39c_0242ac130202row3_col1 {\n",
       "            background-color:  #034f7d;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_f50b6a3a_ad61_11ea_a39c_0242ac130202row4_col0 {\n",
       "            background-color:  #034871;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_f50b6a3a_ad61_11ea_a39c_0242ac130202row4_col1 {\n",
       "            background-color:  #034c78;\n",
       "            color:  #f1f1f1;\n",
       "        }</style><table id=\"T_f50b6a3a_ad61_11ea_a39c_0242ac130202\" ><thead>    <tr>        <th class=\"blank level0\" ></th>        <th class=\"col_heading level0 col0\" >val_score</th>        <th class=\"col_heading level0 col1\" >test_score</th>    </tr></thead><tbody>\n",
       "                <tr>\n",
       "                        <th id=\"T_f50b6a3a_ad61_11ea_a39c_0242ac130202level0_row0\" class=\"row_heading level0 row0\" >KNN</th>\n",
       "                        <td id=\"T_f50b6a3a_ad61_11ea_a39c_0242ac130202row0_col0\" class=\"data row0 col0\" >48.266667</td>\n",
       "                        <td id=\"T_f50b6a3a_ad61_11ea_a39c_0242ac130202row0_col1\" class=\"data row0 col1\" >47.533333</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_f50b6a3a_ad61_11ea_a39c_0242ac130202level0_row1\" class=\"row_heading level0 row1\" >SGD clf</th>\n",
       "                        <td id=\"T_f50b6a3a_ad61_11ea_a39c_0242ac130202row1_col0\" class=\"data row1 col0\" >90.533333</td>\n",
       "                        <td id=\"T_f50b6a3a_ad61_11ea_a39c_0242ac130202row1_col1\" class=\"data row1 col1\" >90.600000</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_f50b6a3a_ad61_11ea_a39c_0242ac130202level0_row2\" class=\"row_heading level0 row2\" >MultinomialNB</th>\n",
       "                        <td id=\"T_f50b6a3a_ad61_11ea_a39c_0242ac130202row2_col0\" class=\"data row2 col0\" >84.833333</td>\n",
       "                        <td id=\"T_f50b6a3a_ad61_11ea_a39c_0242ac130202row2_col1\" class=\"data row2 col1\" >85.488889</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_f50b6a3a_ad61_11ea_a39c_0242ac130202level0_row3\" class=\"row_heading level0 row3\" >Random Forest</th>\n",
       "                        <td id=\"T_f50b6a3a_ad61_11ea_a39c_0242ac130202row3_col0\" class=\"data row3 col0\" >86.800000</td>\n",
       "                        <td id=\"T_f50b6a3a_ad61_11ea_a39c_0242ac130202row3_col1\" class=\"data row3 col1\" >86.888889</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_f50b6a3a_ad61_11ea_a39c_0242ac130202level0_row4\" class=\"row_heading level0 row4\" >Linear SVC</th>\n",
       "                        <td id=\"T_f50b6a3a_ad61_11ea_a39c_0242ac130202row4_col0\" class=\"data row4 col0\" >87.933333</td>\n",
       "                        <td id=\"T_f50b6a3a_ad61_11ea_a39c_0242ac130202row4_col1\" class=\"data row4 col1\" >87.266667</td>\n",
       "            </tr>\n",
       "    </tbody></table>"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7f7a6852dfd0>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "val_scores = []\n",
    "test_scores = []\n",
    "names = []\n",
    "\n",
    "models = [(KNeighborsClassifier(n_neighbors=15),'KNN'),(SGDClassifier(),'SGD clf'),(MultinomialNB(),'MultinomialNB'),\n",
    "          (RandomForestClassifier(),'Random Forest'),(SVC(kernel='linear'),'Linear SVC')]\n",
    "\n",
    "for model,name in models:\n",
    "    clf,score,test_score = get_score(model,0)\n",
    "    names.append(name)\n",
    "    val_scores.append(score*100)\n",
    "    test_scores.append(test_score*100)\n",
    "pd.DataFrame(data=zip(val_scores,test_scores),index=names,columns=['val_score','test_score']).style.background_gradient()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "params = {\n",
    "    'loss':['squared_hinge','modified_huber'],\n",
    "    'alpha':[0.0001,0.001,0.01],\n",
    "    'max_iter':[250,500,1000],\n",
    "    'validation_fraction':[0.2]\n",
    "}\n",
    "cv = GridSearchCV(SGDClassifier(random_state=111),param_grid=params, cv=5,n_jobs=-1,verbose=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 18 candidates, totalling 90 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Using backend LokyBackend with 4 concurrent workers.\n",
      "[Parallel(n_jobs=-1)]: Done  33 tasks      | elapsed:   29.4s\n",
      "[Parallel(n_jobs=-1)]: Done  90 out of  90 | elapsed:  1.1min finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=5, error_score=nan,\n",
       "             estimator=SGDClassifier(alpha=0.0001, average=False,\n",
       "                                     class_weight=None, early_stopping=False,\n",
       "                                     epsilon=0.1, eta0=0.0, fit_intercept=True,\n",
       "                                     l1_ratio=0.15, learning_rate='optimal',\n",
       "                                     loss='hinge', max_iter=1000,\n",
       "                                     n_iter_no_change=5, n_jobs=None,\n",
       "                                     penalty='l2', power_t=0.5,\n",
       "                                     random_state=111, shuffle=True, tol=0.001,\n",
       "                                     validation_fraction=0.1, verbose=0,\n",
       "                                     warm_start=False),\n",
       "             iid='deprecated', n_jobs=-1,\n",
       "             param_grid={'alpha': [0.0001, 0.001, 0.01],\n",
       "                         'loss': ['squared_hinge', 'modified_huber'],\n",
       "                         'max_iter': [250, 500, 1000],\n",
       "                         'validation_fraction': [0.2]},\n",
       "             pre_dispatch='2*n_jobs', refit=True, return_train_score=False,\n",
       "             scoring=None, verbose=2)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cv.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'alpha': 0.001,\n",
       " 'loss': 'modified_huber',\n",
       " 'max_iter': 250,\n",
       " 'validation_fraction': 0.2}"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cv.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9207333333333333"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cv.best_score_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# In-Scope and Out-Scope classification (binary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To perform in and out scope classification we have to encode all samples with intent features other than 'oos' to 0 and with 'oos' to 1 which will be done by binarize()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "oos_plus_train = binarize(pd.concat([in_train,oos_train],axis=0).reset_index(drop=True))\n",
    "oos_plus_val = binarize(pd.concat([val,oos_val],axis=0).reset_index(drop=True))\n",
    "oos_plus_test = binarize(pd.concat([test,oos_test],axis=0).reset_index(drop=True))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    15000\n",
       "1      100\n",
       "Name: intent, dtype: int64"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "oos_count = oos_plus_train.intent.value_counts()\n",
    "oos_count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7f7a687d4210>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAD1CAYAAABQtIIDAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAARG0lEQVR4nO3cf6zV913H8edLcNhtYeuP28rupYIWN4Fott5UdIlZRAWzZfSPNbmNs2SSEJtON39kA/dH/yJpo7HaxJKQdZbqUkbqTMmWzjXUZTGystv9osBYr6ODO1i5c7WiZji6t3+cD/H0crjAOfReWp6P5OR8v+/P5/O975PQvu73x7mpKiRJ+om5bkCSdHkwECRJgIEgSWoMBEkSYCBIkhoDQZIEwPy5bqBf1113XS1ZsmSu25CkV5Wnn376+1U11GvsVRsIS5YsYXx8fK7bkKRXlSTfOdeYl4wkSYCBIElqDARJEmAgSJIaA0GSBBgIkqTGQJAkAQaCJKl51X4x7dViyabPznULrynP3fPuuW5Bes3yDEGSBBgIkqTGQJAkAQaCJKkxECRJgIEgSWoMBEkSYCBIkhoDQZIEXEAgJPlEkhNJnukx9qdJKsl1XbXNSSaSHEqypqt+c5J9bez+JGn1BUk+1epPJVlyaT6aJOliXMgZwkPA2unFJIuB3wSOdNWWA2PAirbmgSTz2vBWYCOwrL3OHHMD8EJV3QTcB9zbzweRJA3mvIFQVV8EftBj6D7gI0B11dYBO6rqVFUdBiaAW5IsAhZW1Z6qKuBh4NauNdvb9qPA6jNnD5Kk2dPXPYQk7wW+W1VfnzY0DBzt2p9steG2Pb3+sjVVdRp4Ebi2n74kSf276L92muT1wMeA3+o13KNWM9RnWtPrZ2+kc9mJG2+88by9SpIuXD9nCD8HLAW+nuQ5YAT4SpKfpvOb/+KuuSPAsVYf6VGne02S+cCb6H2JiqraVlWjVTU6NDTUR+uSpHO56ECoqn1VdX1VLamqJXT+h/6OqvoesAsYa08OLaVz83hvVR0HTiZZ1e4P3AE81g65C1jftt8HPNnuM0iSZtGFPHb6CLAHeGuSySQbzjW3qvYDO4EDwOeAu6rqpTZ8J/BxOjea/w14vNUfBK5NMgH8MbCpz88iSRrAee8hVNXt5xlfMm1/C7Clx7xxYGWP+g+B287XhyTpleU3lSVJgIEgSWoMBEkSYCBIkhoDQZIEGAiSpMZAkCQBBoIkqTEQJEmAgSBJagwESRJgIEiSGgNBkgQYCJKkxkCQJAEGgiSpMRAkSYCBIElqDARJEnABgZDkE0lOJHmmq/bnSb6Z5BtJ/jHJm7vGNieZSHIoyZqu+s1J9rWx+5Ok1Rck+VSrP5VkyaX9iJKkC3EhZwgPAWun1Z4AVlbVLwLfAjYDJFkOjAEr2poHksxra7YCG4Fl7XXmmBuAF6rqJuA+4N5+P4wkqX/nDYSq+iLwg2m1z1fV6bb7JWCkba8DdlTVqao6DEwAtyRZBCysqj1VVcDDwK1da7a37UeB1WfOHiRJs+dS3EP4PeDxtj0MHO0am2y14bY9vf6yNS1kXgSuvQR9SZIuwkCBkORjwGngk2dKPabVDPWZ1vT6eRuTjCcZn5qauth2JUkz6DsQkqwH3gP8TrsMBJ3f/Bd3TRsBjrX6SI/6y9YkmQ+8iWmXqM6oqm1VNVpVo0NDQ/22Lknqoa9ASLIW+Cjw3qr6n66hXcBYe3JoKZ2bx3ur6jhwMsmqdn/gDuCxrjXr2/b7gCe7AkaSNEvmn29CkkeAdwHXJZkE7qbzVNEC4Il2//dLVfX7VbU/yU7gAJ1LSXdV1UvtUHfSeWLpKjr3HM7cd3gQ+LskE3TODMYuzUeTJF2M8wZCVd3eo/zgDPO3AFt61MeBlT3qPwRuO18fkqRXlt9UliQBBoIkqTEQJEmAgSBJagwESRJgIEiSGgNBkgQYCJKkxkCQJAEGgiSpMRAkSYCBIElqDARJEmAgSJIaA0GSBBgIkqTGQJAkAQaCJKkxECRJwAUEQpJPJDmR5Jmu2jVJnkjybHu/umtsc5KJJIeSrOmq35xkXxu7P0lafUGST7X6U0mWXNqPKEm6EBdyhvAQsHZabROwu6qWAbvbPkmWA2PAirbmgSTz2pqtwEZgWXudOeYG4IWqugm4D7i33w8jSerfeQOhqr4I/GBaeR2wvW1vB27tqu+oqlNVdRiYAG5JsghYWFV7qqqAh6etOXOsR4HVZ84eJEmzp997CDdU1XGA9n59qw8DR7vmTbbacNueXn/Zmqo6DbwIXNvrhybZmGQ8yfjU1FSfrUuSernUN5V7/WZfM9RnWnN2sWpbVY1W1ejQ0FCfLUqSeuk3EJ5vl4Fo7ydafRJY3DVvBDjW6iM96i9bk2Q+8CbOvkQlSXqF9RsIu4D1bXs98FhXfaw9ObSUzs3jve2y0skkq9r9gTumrTlzrPcBT7b7DJKkWTT/fBOSPAK8C7guySRwN3APsDPJBuAIcBtAVe1PshM4AJwG7qqql9qh7qTzxNJVwOPtBfAg8HdJJuicGYxdkk8mSboo5w2Eqrr9HEOrzzF/C7ClR30cWNmj/kNaoEiS5o7fVJYkAQaCJKkxECRJgIEgSWoMBEkSYCBIkhoDQZIEGAiSpMZAkCQBBoIkqTEQJEmAgSBJagwESRJgIEiSGgNBkgQYCJKkxkCQJAEGgiSpGSgQkvxRkv1JnknySJKfSnJNkieSPNver+6avznJRJJDSdZ01W9Osq+N3Z8kg/QlSbp4fQdCkmHgD4HRqloJzAPGgE3A7qpaBuxu+yRZ3sZXAGuBB5LMa4fbCmwElrXX2n77kiT1Z9BLRvOBq5LMB14PHAPWAdvb+Hbg1ra9DthRVaeq6jAwAdySZBGwsKr2VFUBD3etkSTNkr4Doaq+C/wFcAQ4DrxYVZ8Hbqiq423OceD6tmQYONp1iMlWG27b0+uSpFk0yCWjq+n81r8UeAvwhiTvn2lJj1rNUO/1MzcmGU8yPjU1dbEtS5JmMMglo98ADlfVVFX9CPg08KvA8+0yEO39RJs/CSzuWj9C5xLTZNueXj9LVW2rqtGqGh0aGhqgdUnSdIMEwhFgVZLXt6eCVgMHgV3A+jZnPfBY294FjCVZkGQpnZvHe9tlpZNJVrXj3NG1RpI0S+b3u7CqnkryKPAV4DTwVWAb8EZgZ5INdELjtjZ/f5KdwIE2/66qeqkd7k7gIeAq4PH2kiTNor4DAaCq7gbunlY+Redsodf8LcCWHvVxYOUgvUiSBuM3lSVJgIEgSWoMBEkSYCBIkhoDQZIEGAiSpMZAkCQBBoIkqTEQJEmAgSBJagwESRJgIEiSGgNBkgQYCJKkxkCQJAEGgiSpMRAkSYCBIElqDARJEjBgICR5c5JHk3wzycEkv5LkmiRPJHm2vV/dNX9zkokkh5Ks6arfnGRfG7s/SQbpS5J08QY9Q/hr4HNV9Tbgl4CDwCZgd1UtA3a3fZIsB8aAFcBa4IEk89pxtgIbgWXttXbAviRJF6nvQEiyEPg14EGAqvrfqvoPYB2wvU3bDtzattcBO6rqVFUdBiaAW5IsAhZW1Z6qKuDhrjWSpFkyyBnCzwJTwN8m+WqSjyd5A3BDVR0HaO/Xt/nDwNGu9ZOtNty2p9clSbNokECYD7wD2FpVbwf+m3Z56Bx63ReoGepnHyDZmGQ8yfjU1NTF9itJmsEggTAJTFbVU23/UToB8Xy7DER7P9E1f3HX+hHgWKuP9Kifpaq2VdVoVY0ODQ0N0Lokabq+A6GqvgccTfLWVloNHAB2AetbbT3wWNveBYwlWZBkKZ2bx3vbZaWTSVa1p4vu6FojSZol8wdc/wfAJ5O8Dvg28AE6IbMzyQbgCHAbQFXtT7KTTmicBu6qqpface4EHgKuAh5vL0nSLBooEKrqa8Boj6HV55i/BdjSoz4OrBykF0nSYPymsiQJMBAkSY2BIEkCDARJUmMgSJIAA0GS1BgIkiTAQJAkNQaCJAkwECRJjYEgSQIMBElSYyBIkgADQZLUGAiSJMBAkCQ1BoIkCTAQJEnNwIGQZF6Sryb5TNu/JskTSZ5t71d3zd2cZCLJoSRruuo3J9nXxu5PkkH7kiRdnEtxhvAh4GDX/iZgd1UtA3a3fZIsB8aAFcBa4IEk89qarcBGYFl7rb0EfUmSLsJAgZBkBHg38PGu8jpge9veDtzaVd9RVaeq6jAwAdySZBGwsKr2VFUBD3etkSTNkkHPEP4K+Ajw467aDVV1HKC9X9/qw8DRrnmTrTbctqfXJUmzqO9ASPIe4ERVPX2hS3rUaoZ6r5+5Mcl4kvGpqakL/LGSpAsxyBnCO4H3JnkO2AH8epK/B55vl4Fo7yfa/Elgcdf6EeBYq4/0qJ+lqrZV1WhVjQ4NDQ3QuiRpur4Doao2V9VIVS2hc7P4yap6P7ALWN+mrQcea9u7gLEkC5IspXPzeG+7rHQyyar2dNEdXWskSbNk/itwzHuAnUk2AEeA2wCqan+SncAB4DRwV1W91NbcCTwEXAU83l6SpFl0SQKhqr4AfKFt/zuw+hzztgBbetTHgZWXohdJUn/8prIkCTAQJEmNgSBJAgwESVJjIEiSAANBktQYCJIkwECQJDUGgiQJMBAkSY2BIEkCDARJUmMgSJIAA0GS1BgIkiTAQJAkNQaCJAkwECRJjYEgSQIGCIQki5P8c5KDSfYn+VCrX5PkiSTPtveru9ZsTjKR5FCSNV31m5Psa2P3J8lgH0uSdLEGOUM4DfxJVf0CsAq4K8lyYBOwu6qWAbvbPm1sDFgBrAUeSDKvHWsrsBFY1l5rB+hLktSHvgOhqo5X1Vfa9kngIDAMrAO2t2nbgVvb9jpgR1WdqqrDwARwS5JFwMKq2lNVBTzctUaSNEsuyT2EJEuAtwNPATdU1XHohAZwfZs2DBztWjbZasNte3pdkjSLBg6EJG8E/gH4cFX950xTe9Rqhnqvn7UxyXiS8ampqYtvVpJ0TgMFQpKfpBMGn6yqT7fy8+0yEO39RKtPAou7lo8Ax1p9pEf9LFW1rapGq2p0aGhokNYlSdMM8pRRgAeBg1X1l11Du4D1bXs98FhXfSzJgiRL6dw83tsuK51Msqod846uNZKkWTJ/gLXvBH4X2Jfka632Z8A9wM4kG4AjwG0AVbU/yU7gAJ0nlO6qqpfaujuBh4CrgMfbS5I0i/oOhKr6F3pf/wdYfY41W4AtPerjwMp+e5EkDc5vKkuSAANBktQYCJIkwECQJDUGgiQJMBAkSY2BIEkCDARJUmMgSJIAA0GS1BgIkiTAQJAkNQaCJAkwECRJjYEgSQIMBElSYyBIkgADQZLUGAiSJOAyCoQka5McSjKRZNNc9yNJV5rLIhCSzAP+BvhtYDlwe5Llc9uVJF1Z5s91A80twERVfRsgyQ5gHXBgTruSXsOWbPrsXLfwmvLcPe+e6xYGdrkEwjBwtGt/Evjl6ZOSbAQ2tt3/SnJoFnq7UlwHfH+umzif3DvXHWgO+G/z0vqZcw1cLoGQHrU6q1C1Ddj2yrdz5UkyXlWjc92HNJ3/NmfPZXEPgc4ZweKu/RHg2Bz1IklXpMslEL4MLEuyNMnrgDFg1xz3JElXlMviklFVnU7yQeCfgHnAJ6pq/xy3daXxUpwuV/7bnCWpOutSvSTpCnS5XDKSJM0xA0GSBBgIkqTmsriprNmV5G10vgk+TOf7HseAXVV1cE4bkzSnPEO4wiT5KLCDzpcB99J55DfAI/5RQV3Oknxgrnt4rfMpoytMkm8BK6rqR9PqrwP2V9WyuelMmlmSI1V141z38VrmJaMrz4+BtwDfmVZf1MakOZPkG+caAm6YzV6uRAbClefDwO4kz/L/f1DwRuAm4INz1pXUcQOwBnhhWj3Av85+O1cWA+EKU1WfS/LzdP7k+DCd/9AmgS9X1Utz2pwEnwHeWFVfmz6Q5Auz386VxXsIkiTAp4wkSY2BIEkCDARJUmMgSJIAA0GS1Pwfbcuuh5Th5EUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "oos_count.plot(kind='bar')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The imbalance ratio between 0 and 1 is 150\n",
    "\n",
    "This imbalance can be dealt with by using sampling, here I will be using undersampling i.e the majority class will be undersampled to the size of minority class, coupling this with ensembling will give us best results.\n",
    "\n",
    "Resources:\n",
    "\n",
    "https://machinelearningmastery.com/random-oversampling-and-undersampling-for-imbalanced-classification/\n",
    "\n",
    "https://www.kaggle.com/residentmario/undersampling-and-oversampling-imbalanced-data/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train,y_train,cv,le = preprocess(oos_plus_train)\n",
    "X_val,y_val = process_non_train(oos_plus_val,cv,le)\n",
    "X_test,y_test = process_non_train(oos_plus_test,cv,le)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style  type=\"text/css\" >\n",
       "    #T_233c7778_ad62_11ea_a39c_0242ac130202row0_col0 {\n",
       "            background-color:  #fff7fb;\n",
       "            color:  #000000;\n",
       "        }    #T_233c7778_ad62_11ea_a39c_0242ac130202row0_col1 {\n",
       "            background-color:  #fff7fb;\n",
       "            color:  #000000;\n",
       "        }    #T_233c7778_ad62_11ea_a39c_0242ac130202row0_col2 {\n",
       "            background-color:  #023858;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_233c7778_ad62_11ea_a39c_0242ac130202row0_col3 {\n",
       "            background-color:  #fff7fb;\n",
       "            color:  #000000;\n",
       "        }    #T_233c7778_ad62_11ea_a39c_0242ac130202row1_col0 {\n",
       "            background-color:  #034369;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_233c7778_ad62_11ea_a39c_0242ac130202row1_col1 {\n",
       "            background-color:  #034a74;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_233c7778_ad62_11ea_a39c_0242ac130202row1_col2 {\n",
       "            background-color:  #fff7fb;\n",
       "            color:  #000000;\n",
       "        }    #T_233c7778_ad62_11ea_a39c_0242ac130202row1_col3 {\n",
       "            background-color:  #056ba7;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_233c7778_ad62_11ea_a39c_0242ac130202row2_col0 {\n",
       "            background-color:  #023858;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_233c7778_ad62_11ea_a39c_0242ac130202row2_col1 {\n",
       "            background-color:  #023858;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_233c7778_ad62_11ea_a39c_0242ac130202row2_col2 {\n",
       "            background-color:  #f1ebf5;\n",
       "            color:  #000000;\n",
       "        }    #T_233c7778_ad62_11ea_a39c_0242ac130202row2_col3 {\n",
       "            background-color:  #023858;\n",
       "            color:  #f1f1f1;\n",
       "        }</style><table id=\"T_233c7778_ad62_11ea_a39c_0242ac130202\" ><thead>    <tr>        <th class=\"blank level0\" ></th>        <th class=\"col_heading level0 col0\" >val_score</th>        <th class=\"col_heading level0 col1\" >test_score</th>        <th class=\"col_heading level0 col2\" >recall_score</th>        <th class=\"col_heading level0 col3\" >precision_score</th>    </tr></thead><tbody>\n",
       "                <tr>\n",
       "                        <th id=\"T_233c7778_ad62_11ea_a39c_0242ac130202level0_row0\" class=\"row_heading level0 row0\" >Balanced Random Forest</th>\n",
       "                        <td id=\"T_233c7778_ad62_11ea_a39c_0242ac130202row0_col0\" class=\"data row0 col0\" >53.225806</td>\n",
       "                        <td id=\"T_233c7778_ad62_11ea_a39c_0242ac130202row0_col1\" class=\"data row0 col1\" >54.618182</td>\n",
       "                        <td id=\"T_233c7778_ad62_11ea_a39c_0242ac130202row0_col2\" class=\"data row0 col2\" >71.700000</td>\n",
       "                        <td id=\"T_233c7778_ad62_11ea_a39c_0242ac130202row0_col3\" class=\"data row0 col3\" >24.470990</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_233c7778_ad62_11ea_a39c_0242ac130202level0_row1\" class=\"row_heading level0 row1\" >Random Undersampling + Adaboost</th>\n",
       "                        <td id=\"T_233c7778_ad62_11ea_a39c_0242ac130202row1_col0\" class=\"data row1 col0\" >76.612903</td>\n",
       "                        <td id=\"T_233c7778_ad62_11ea_a39c_0242ac130202row1_col1\" class=\"data row1 col1\" >71.472727</td>\n",
       "                        <td id=\"T_233c7778_ad62_11ea_a39c_0242ac130202row1_col2\" class=\"data row1 col2\" >47.600000</td>\n",
       "                        <td id=\"T_233c7778_ad62_11ea_a39c_0242ac130202row1_col3\" class=\"data row1 col3\" >31.295201</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_233c7778_ad62_11ea_a39c_0242ac130202level0_row2\" class=\"row_heading level0 row2\" >Easy Ensemble</th>\n",
       "                        <td id=\"T_233c7778_ad62_11ea_a39c_0242ac130202row2_col0\" class=\"data row2 col0\" >77.645161</td>\n",
       "                        <td id=\"T_233c7778_ad62_11ea_a39c_0242ac130202row2_col1\" class=\"data row2 col1\" >72.690909</td>\n",
       "                        <td id=\"T_233c7778_ad62_11ea_a39c_0242ac130202row2_col2\" class=\"data row2 col2\" >49.800000</td>\n",
       "                        <td id=\"T_233c7778_ad62_11ea_a39c_0242ac130202row2_col3\" class=\"data row2 col3\" >33.244326</td>\n",
       "            </tr>\n",
       "    </tbody></table>"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7f7a68eb6410>"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "val_scores = []\n",
    "test_scores = []\n",
    "recall = []\n",
    "names = []\n",
    "precision = []\n",
    "\n",
    "models = [(BalancedRandomForestClassifier(sampling_strategy='not minority',random_state=111),'Balanced Random Forest'),\n",
    "          (RUSBoostClassifier(base_estimator=LogisticRegression(),sampling_strategy='not minority',random_state=111),'Random Undersampling + Adaboost'),\n",
    "          (EasyEnsembleClassifier(n_estimators=30,base_estimator=LogisticRegression(),replacement=True,sampling_strategy='not minority',random_state=111),'Easy Ensemble')]\n",
    "\n",
    "for model,name in models:\n",
    "    _,score,test_score,recall_sc,precision_sc = get_score(model,1)\n",
    "    names.append(name)\n",
    "    val_scores.append(score*100)\n",
    "    test_scores.append(test_score*100)\n",
    "    recall.append(recall_sc*100)\n",
    "    precision.append(precision_sc*100)\n",
    "pd.DataFrame(data=zip(val_scores,test_scores,recall,precision),index=names,\n",
    "             columns=['val_score','test_score','recall_score','precision_score']).style.background_gradient()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Balanced Random Forest:\n",
    "\n",
    "Accuracy 😪\n",
    "\n",
    "recall 🤩\n",
    "\n",
    "precision 😑"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Easy ensemble has provided the best performance, next step is to explore easyensemble more\n",
    "\n",
    "Learn more about easyensemble:\n",
    "\n",
    "X. Y. Liu, J. Wu and Z. H. Zhou, “Exploratory Undersampling for Class-Imbalance Learning,” in IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), vol. 39, no. 2, pp. 539-550, April 2009."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tuning EasyEnsemble"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 12 candidates, totalling 60 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Using backend LokyBackend with 4 concurrent workers.\n",
      "/opt/conda/lib/python3.7/site-packages/joblib/externals/loky/process_executor.py:706: UserWarning:\n",
      "\n",
      "A worker stopped while some jobs were given to the executor. This can be caused by a too short worker timeout or by a memory leak.\n",
      "\n",
      "[Parallel(n_jobs=-1)]: Done  33 tasks      | elapsed:  3.6min\n",
      "[Parallel(n_jobs=-1)]: Done  60 out of  60 | elapsed:  4.7min finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=5, error_score=nan,\n",
       "             estimator=EasyEnsembleClassifier(base_estimator=None,\n",
       "                                              n_estimators=10, n_jobs=-1,\n",
       "                                              random_state=111,\n",
       "                                              replacement=False,\n",
       "                                              sampling_strategy='not minority',\n",
       "                                              verbose=0, warm_start=False),\n",
       "             iid='deprecated', n_jobs=-1,\n",
       "             param_grid={'base_estimator': [AdaBoostClassifier(algorithm='SAMME.R',\n",
       "                                                               base_estimator=None,\n",
       "                                                               learning_rate=1.0,\n",
       "                                                               n_estima...\n",
       "                                                               class_weight=None,\n",
       "                                                               dual=False,\n",
       "                                                               fit_intercept=True,\n",
       "                                                               intercept_scaling=1,\n",
       "                                                               l1_ratio=None,\n",
       "                                                               max_iter=100,\n",
       "                                                               multi_class='auto',\n",
       "                                                               n_jobs=None,\n",
       "                                                               penalty='l2',\n",
       "                                                               random_state=None,\n",
       "                                                               solver='lbfgs',\n",
       "                                                               tol=0.0001,\n",
       "                                                               verbose=0,\n",
       "                                                               warm_start=False)],\n",
       "                         'n_estimators': [30, 50, 70],\n",
       "                         'replacement': [False, True]},\n",
       "             pre_dispatch='2*n_jobs', refit=True, return_train_score=False,\n",
       "             scoring='recall', verbose=2)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params = {\n",
    "    'n_estimators':[30,50,70],\n",
    "    'base_estimator':[AdaBoostClassifier(),LogisticRegression()],\n",
    "    'replacement':[False,True]\n",
    "}\n",
    "cv = GridSearchCV(EasyEnsembleClassifier(n_jobs=-1,sampling_strategy='not minority',random_state=111),\n",
    "                  verbose=2,\n",
    "                  scoring='recall',\n",
    "                  param_grid=params, cv=5,\n",
    "                  n_jobs=-1)\n",
    "cv.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8400000000000001"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cv.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'base_estimator': LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "                    intercept_scaling=1, l1_ratio=None, max_iter=100,\n",
       "                    multi_class='auto', n_jobs=None, penalty='l2',\n",
       "                    random_state=None, solver='lbfgs', tol=0.0001, verbose=0,\n",
       "                    warm_start=False),\n",
       " 'n_estimators': 50,\n",
       " 'replacement': False}"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cv.best_params_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tune base estimator of EasyEnsemble"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_estimator = [25,50,100]\n",
    "learning_rate=[0.001,0.0001]\n",
    "result = []\n",
    "for nest in n_estimator:\n",
    "    for lr in learning_rate:\n",
    "        eec = EasyEnsembleClassifier(base_estimator=AdaBoostClassifier(n_estimators=nest,learning_rate=lr)\n",
    "                                     ,n_estimators=100,\n",
    "                                     replacement=True,n_jobs=-1,\n",
    "                                     sampling_strategy='not minority',random_state=111)\n",
    "        eec.fit(X_train,y_train)\n",
    "        recall = recall_score(y_test,eec.predict(X_test))\n",
    "        result.append([nest,lr,recall])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style  type=\"text/css\" >\n",
       "    #T_1381bda6_ad63_11ea_a39c_0242ac130202row0_col2 {\n",
       "            background-color:  #fff7fb;\n",
       "            color:  #000000;\n",
       "        }    #T_1381bda6_ad63_11ea_a39c_0242ac130202row1_col2 {\n",
       "            background-color:  #fff7fb;\n",
       "            color:  #000000;\n",
       "        }    #T_1381bda6_ad63_11ea_a39c_0242ac130202row2_col2 {\n",
       "            background-color:  #fff7fb;\n",
       "            color:  #000000;\n",
       "        }    #T_1381bda6_ad63_11ea_a39c_0242ac130202row3_col2 {\n",
       "            background-color:  #fff7fb;\n",
       "            color:  #000000;\n",
       "        }    #T_1381bda6_ad63_11ea_a39c_0242ac130202row4_col2 {\n",
       "            background-color:  #fff7fb;\n",
       "            color:  #000000;\n",
       "        }    #T_1381bda6_ad63_11ea_a39c_0242ac130202row5_col2 {\n",
       "            background-color:  #fff7fb;\n",
       "            color:  #000000;\n",
       "        }</style><table id=\"T_1381bda6_ad63_11ea_a39c_0242ac130202\" ><thead>    <tr>        <th class=\"blank level0\" ></th>        <th class=\"col_heading level0 col0\" >n_estimators</th>        <th class=\"col_heading level0 col1\" >learning_rate</th>        <th class=\"col_heading level0 col2\" >recall_score</th>    </tr></thead><tbody>\n",
       "                <tr>\n",
       "                        <th id=\"T_1381bda6_ad63_11ea_a39c_0242ac130202level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
       "                        <td id=\"T_1381bda6_ad63_11ea_a39c_0242ac130202row0_col0\" class=\"data row0 col0\" >25</td>\n",
       "                        <td id=\"T_1381bda6_ad63_11ea_a39c_0242ac130202row0_col1\" class=\"data row0 col1\" >0.001000</td>\n",
       "                        <td id=\"T_1381bda6_ad63_11ea_a39c_0242ac130202row0_col2\" class=\"data row0 col2\" >0.801000</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_1381bda6_ad63_11ea_a39c_0242ac130202level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
       "                        <td id=\"T_1381bda6_ad63_11ea_a39c_0242ac130202row1_col0\" class=\"data row1 col0\" >25</td>\n",
       "                        <td id=\"T_1381bda6_ad63_11ea_a39c_0242ac130202row1_col1\" class=\"data row1 col1\" >0.000100</td>\n",
       "                        <td id=\"T_1381bda6_ad63_11ea_a39c_0242ac130202row1_col2\" class=\"data row1 col2\" >0.801000</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_1381bda6_ad63_11ea_a39c_0242ac130202level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
       "                        <td id=\"T_1381bda6_ad63_11ea_a39c_0242ac130202row2_col0\" class=\"data row2 col0\" >50</td>\n",
       "                        <td id=\"T_1381bda6_ad63_11ea_a39c_0242ac130202row2_col1\" class=\"data row2 col1\" >0.001000</td>\n",
       "                        <td id=\"T_1381bda6_ad63_11ea_a39c_0242ac130202row2_col2\" class=\"data row2 col2\" >0.801000</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_1381bda6_ad63_11ea_a39c_0242ac130202level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
       "                        <td id=\"T_1381bda6_ad63_11ea_a39c_0242ac130202row3_col0\" class=\"data row3 col0\" >50</td>\n",
       "                        <td id=\"T_1381bda6_ad63_11ea_a39c_0242ac130202row3_col1\" class=\"data row3 col1\" >0.000100</td>\n",
       "                        <td id=\"T_1381bda6_ad63_11ea_a39c_0242ac130202row3_col2\" class=\"data row3 col2\" >0.801000</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_1381bda6_ad63_11ea_a39c_0242ac130202level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
       "                        <td id=\"T_1381bda6_ad63_11ea_a39c_0242ac130202row4_col0\" class=\"data row4 col0\" >100</td>\n",
       "                        <td id=\"T_1381bda6_ad63_11ea_a39c_0242ac130202row4_col1\" class=\"data row4 col1\" >0.001000</td>\n",
       "                        <td id=\"T_1381bda6_ad63_11ea_a39c_0242ac130202row4_col2\" class=\"data row4 col2\" >0.801000</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_1381bda6_ad63_11ea_a39c_0242ac130202level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
       "                        <td id=\"T_1381bda6_ad63_11ea_a39c_0242ac130202row5_col0\" class=\"data row5 col0\" >100</td>\n",
       "                        <td id=\"T_1381bda6_ad63_11ea_a39c_0242ac130202row5_col1\" class=\"data row5 col1\" >0.000100</td>\n",
       "                        <td id=\"T_1381bda6_ad63_11ea_a39c_0242ac130202row5_col2\" class=\"data row5 col2\" >0.801000</td>\n",
       "            </tr>\n",
       "    </tbody></table>"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7f7a63f78f10>"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(result,columns=['n_estimators','learning_rate','recall_score']).style.background_gradient(subset=['recall_score'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Thank you for coming by, suggest modifications if any"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}