Day 02 Practice_CNN_on_texts_Class.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "Copy of week04_practice_CNN_for_texts.ipynb",
      "provenance": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/lagom-QB/M12/blob/master/week04_practice_CNN_for_texts.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "13pL--6rycN3"
      },
      "source": [
        "## Practice 02: Dealing with texts using CNN\n",
        "\n",
        "Today we're gonna apply the newly learned tools for the task of predicting job salary.\n",
        "\n",
        "<img src=\"https://storage.googleapis.com/kaggle-competitions/kaggle/3342/logos/front_page.png\" width=400px>\n",
        "\n",
        "Based on YSDA [materials](https://github.com/yandexdataschool/nlp_course/blob/master/week02_classification/seminar.ipynb). _Special thanks to [Oleg Vasilev](https://github.com/Omrigan/) for the core assignment idea._"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "P8zS7m-gycN5",
        "colab": {}
      },
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "%matplotlib inline"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "34x92vWQycN_"
      },
      "source": [
        "### About the challenge\n",
        "For starters, let's download and unpack the data from [here](https://www.dropbox.com/s/5msc5ix7ndyba10/Train_rev1.csv.tar.gz?dl=0). \n",
        "\n",
        "You can also get it from [yadisk url](https://yadi.sk/d/vVEOWPFY3NruT7) the competition [page](https://www.kaggle.com/c/job-salary-prediction/data) (pick `Train_rev1.*`)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "vwN72gd4ycOA",
        "outputId": "f1287dd5-1dde-4fda-bfcb-d9d9811e6ab2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 87
        }
      },
      "source": [
        "# Do this only once\n",
        "!curl -L https://www.dropbox.com/s/5msc5ix7ndyba10/Train_rev1.csv.tar.gz?dl=1 -o Train_rev1.csv.tar.gz\n",
        "!tar -xvzf ./Train_rev1.csv.tar.gz"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
            "                                 Dload  Upload   Total   Spent    Left  Speed\n",
            "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\n",
            "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\n",
            "100  119M  100  119M    0     0  10.8M      0  0:00:11  0:00:11 --:--:-- 12.5M\n",
            "Train_rev1.csv\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "outputId": "0e434fa7-5e56-4e36-9582-7ed007ae6446",
        "id": "OiudyVdHXudg",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 29
        }
      },
      "source": [
        "data = pd.read_csv(\"./Train_rev1.csv\", index_col=None)\n",
        "data.shape"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(244768, 12)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "z7kznuJfycOH"
      },
      "source": [
        "One problem with salary prediction is that it's oddly distributed: there are many people who are paid standard salaries and a few that get tons o money. The distribution is fat-tailed on the right side, which is inconvenient for MSE minimization.\n",
        "\n",
        "There are several techniques to combat this: using a different loss function, predicting log-target instead of raw target or even replacing targets with their percentiles among all salaries in the training set. We gonna use logarithm for now.\n",
        "\n",
        "_You can read more [in the official description](https://www.kaggle.com/c/job-salary-prediction#description)._"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "UuuKIKfrycOH",
        "outputId": "82ff2806-6a5a-449e-d37c-db659c1943f0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 265
        }
      },
      "source": [
        "data['Log1pSalary'] = np.log1p(data['SalaryNormalized']).astype('float32')\n",
        "\n",
        "plt.figure(figsize=[8, 4])\n",
        "plt.subplot(1, 2, 1)\n",
        "plt.hist(data[\"SalaryNormalized\"], bins=20);\n",
        "\n",
        "plt.subplot(1, 2, 2)\n",
        "plt.hist(data['Log1pSalary'], bins=20);"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfMAAAD4CAYAAAD4vw88AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAeDUlEQVR4nO3dfaxdV3nn8e+veSPlLQ5xIytOxmmxStNIhORO4gqGoYlInFDVYQQoTNW4TIQ7Q6hA7UwxbaVQIFKYUckQlaZ1GzdORQmZAMKCBOMJYRj+yIsDJq/QXEJQbJnYjfMCQg2T9Jk/9rrh4Nzre6593/Y93490dPZ59tr7rH18t5+z11l7rVQVkiSpv35hoSsgSZIOj8lckqSeM5lLktRzJnNJknrOZC5JUs8dudAVOFQnnHBCrVq1aqGrIS1q99xzzz9X1fKFrsfBeC5LwznY+dzbZL5q1Sp27Nix0NWQFrUkP1joOkzHc1kazsHOZ5vZJUnqOZO5JEk9ZzKXJKnnTOaSJPWcyVySpJ4zmUuS1HMmc0mSes5kLklSz5nMJUnqud6OADdbVm380rRlHr3qLfNQE0laPPy/sV9GPpkPwz9qSdJiZjO7JEk9ZzKXJKnnTOaSJPWcyVySpJ4zmUuS1HMmc0mSes5b06QRkeQlwNeBY+jO/Zur6ook1wP/Hni6Ff29qtqZJMAngIuAn7T4N9u+1gN/1sp/tKq2tPhZwPXAscAtwPuqqubh8DSkYW61Vf+YzKXR8SxwblX9OMlRwDeS3NrW/bequvmA8hcCq9vjHOBa4JwkxwNXAGNAAfck2VpVT7Yy7wbupEvma4FbkTSnbGaXRkR1ftxeHtUeB7tqXgfc0La7AzguyQrgAmB7Ve1vCXw7sLate0VV3dGuxm8ALp6zA5L0gqGSeZLjktyc5DtJHkryG0mOT7I9ycPteVkrmyTXJBlPcm+SMwf2s76Vf7g1003Ez0pyX9vmmta8J2mWJTkiyU5gL11CvrOturKdr1cnOabFTgIeG9h8V4sdLL5rkvhk9diQZEeSHfv27Tvs45JG3bBX5p8AvlxVrwFeCzwEbARuq6rVwG3tNfx809wGumY3BprmzgHOBq6Y+ALAz5rmJrZbe3iHJWkyVfV8VZ0BrATOTnI68EHgNcC/BY4HPjAP9dhUVWNVNbZ8+fK5fjtpyZs2mSd5JfBG4DqAqvppVT1F1wS3pRXbws+a02yakxa5dg7fDqytqj3tfH0W+Hu6L9sAu4GTBzZb2WIHi6+cJC5pjg1zZX4qsA/4+yTfSvJ3SV4KnFhVe1qZHwIntmWb5qRFKMnyJMe15WOBNwPfaV+oaT9vXQzc3zbZClzafjpbAzzdzvltwPlJlrXWtfOBbW3dM0nWtH1dCnxhPo9RGlXD9GY/EjgT+IOqujPJJ/hZkzrQdaxJMue3n1TVJmATwNjYmLe7SDOzAtiS5Ai6L/I3VdUXk3w1yXIgwE7gP7fyt9DdljZOd2vauwCqan+SjwB3t3Ifrqr9bfk9/OzWtFuxJ7s0L4ZJ5ruAXQMdZW6mS+aPJ1lRVXvaN/u9bf3BmuDedED8a9g0J82LqroXeN0k8XOnKF/A5VOs2wxsniS+Azj98GoqaaambWavqh8CjyX51RY6D3iQrgluokf6en7WnGbTnCRJ82jYQWP+APhUkqOBR+ia234BuCnJZcAPgHe0sjbNSZI0j4ZK5lW1k260pwOdN0lZm+YkSZpHjgAnSVLPOTa7JC0RTqIyurwylySp50zmkiT1nMlckqSeM5lLktRzJnNJknrOZC5JUs+ZzCVJ6jmTuSRJPWcylySp50zmkiT1nMlckqSeM5lLktRzJnNJknrOZC5JUs+ZzKURkuQlSe5K8u0kDyT58xY/NcmdScaTfCbJ0S1+THs93tavGtjXB1v8u0kuGIivbbHxJBvn+xilUWQyl0bLs8C5VfVa4AxgbZI1wMeAq6vq1cCTwGWt/GXAky1+dStHktOAS4BfB9YCf5XkiCRHAJ8ELgROA97ZykqaQyZzaYRU58ft5VHtUcC5wM0tvgW4uC2va69p689Lkha/saqerarvA+PA2e0xXlWPVNVPgRtbWUlzyGQujZh2Bb0T2AtsB74HPFVVz7Uiu4CT2vJJwGMAbf3TwKsG4wdsM1X8wDpsSLIjyY59+/bN1qFJI8tkLo2Yqnq+qs4AVtJdSb9mAeqwqarGqmps+fLl8/320pJjMpdGVFU9BdwO/AZwXJIj26qVwO62vBs4GaCtfyXwxGD8gG2mikuaQyZzaYQkWZ7kuLZ8LPBm4CG6pP62Vmw98IW2vLW9pq3/alVVi1/SerufCqwG7gLuBla33vFH03WS2zr3RyaNtqGSeZJHk9yXZGeSHS12fJLtSR5uz8taPEmuabel3JvkzIH9rG/lH06yfiB+Vtv/eNs2s32gkgBYAdye5F66xLu9qr4IfAD4wyTjdL+JX9fKXwe8qsX/ENgIUFUPADcBDwJfBi5vzffPAe8FttF9SbiplZU0h46cvsgLfrOq/nng9Ubgtqq6qt1LupHuP4QL6b6lrwbOAa4FzklyPHAFMEbXe/aeJFur6slW5t3AncAtdLe63HpYRybpRarqXuB1k8Qfofv9/MD4vwBvn2JfVwJXThK/he48ljRPDqeZffCWlQNvZbmh3QJzB91vcSuAC+iuAva3BL6d7h7XFcArquqO1nx3w8C+JEnSNIZN5gV8Jck9STa02IlVtact/xA4sS3P9JaVk9rygfEX8XYWSZJebNhm9jdU1e4kvwRsT/KdwZVVVUlq9qv386pqE7AJYGxsbM7fT5KkPhjqyryqdrfnvcDn6X5be7w1kdOe97biM71lZXdbPjAuSZKGMG0yT/LSJC+fWAbOB+7n529ZOfBWlktbr/Y1wNOtOX4bcH6SZa3n+/nAtrbumSRrWi/2Swf2JUmSpjFMM/uJwOfb3WJHAv9YVV9OcjdwU5LLgB8A72jlbwEuohur+SfAuwCqan+Sj9DdDgPw4ara35bfA1wPHEvXi92e7JK0yK3a+KVpyzx61VvmoSaaNpm3W1ZeO0n8CeC8SeIFXD7FvjYDmyeJ7wBOH6K+kiTpAI4AJ0lSz5nMJUnqOZO5JEk9ZzKXJKnnTOaSJPWcyVySpJ4zmUuS1HMmc0mSes5kLklSz5nMJUnqOZO5JEk9ZzKXJKnnTObSiEhycpLbkzyY5IEk72vxDyXZnWRne1w0sM0Hk4wn+W6SCwbia1tsPMnGgfipSe5s8c8kOXp+j1IaTSZzaXQ8B/xRVZ0GrAEuT3JaW3d1VZ3RHrcAtHWXAL8OrAX+KskRSY4APglcCJwGvHNgPx9r+3o18CRw2XwdnDTKTObSiKiqPVX1zbb8I+Ah4KSDbLIOuLGqnq2q7wPjwNntMV5Vj1TVT4EbgXVJApwL3Ny23wJcPDdHI2mQyVwaQUlWAa8D7myh9ya5N8nmJMta7CTgsYHNdrXYVPFXAU9V1XMHxCd7/w1JdiTZsW/fvlk4Imm0mcylEZPkZcBngfdX1TPAtcCvAGcAe4C/mOs6VNWmqhqrqrHly5fP9dtJS96RC10BSfMnyVF0ifxTVfU5gKp6fGD93wJfbC93AycPbL6yxZgi/gRwXJIj29X5YHlJc8grc2lEtN+0rwMeqqqPD8RXDBR7K3B/W94KXJLkmCSnAquBu4C7gdWt5/rRdJ3ktlZVAbcDb2vbrwe+MJfHJKnjlfksWbXxS0OVe/Sqt8xxTaQpvR74XeC+JDtb7E/oeqOfARTwKPD7AFX1QJKbgAfpesJfXlXPAyR5L7ANOALYXFUPtP19ALgxyUeBb9F9eZA0x0zm0oioqm8AmWTVLQfZ5krgyknit0y2XVU9QtfbXdI8spldkqSeM5lLktRzQyfzNvLTt5J8sb2edNjG1lnmMy1+Z7ufdWIfMxoaUpIkTW8mV+bvoxsxasJUwzZeBjzZ4le3coc6NKQkSZrGUMk8yUrgLcDftdcHG7ZxXXtNW39eKz+joSEP98AkSRoVw16Z/0/gj4F/ba8PNmzjC0M9tvVPt/IzHRryRRwCUpKkF5v21rQkvwXsrap7krxp7qs0taraBGwCGBsbq4WsiyTNp2HHstBoGuY+89cDv93mOH4J8ArgE0w9bOPEEJC7khwJvJJumMeZDg0pSZKGMG0ze1V9sKpWVtUqug5sX62q32HqYRu3tte09V9twzzOaGjIWTk6SZJGwOGMADfVsI3XAf+QZBzYT5ecD3VoSEmSNI0ZJfOq+hrwtbY86bCNVfUvwNun2H5GQ0NKkqTpOQKcJEk9ZzKXJKnnTOaSJPWcyVySpJ4zmUuS1HMmc0mSes5kLklSz5nMJUnqOZO5NCKSnJzk9iQPJnkgyfta/Pgk25M83J6XtXiSXJNkPMm9Sc4c2Nf6Vv7hJOsH4mclua9tc02b/ljSHDOZS6PjOeCPquo0YA1weZLTgI3AbVW1GritvQa4kG4OhdXABuBa6JI/cAVwDt0okFdMfAFoZd49sN3aeTguaeSZzKURUVV7quqbbflHwEPAScA6YEsrtgW4uC2vA26ozh10MyWuAC4AtlfV/qp6EtgOrG3rXlFVd7TJlW4Y2JekOWQyl0ZQklXA64A7gROrak9b9UPgxLZ8EvDYwGa7Wuxg8V2TxCd7/w1JdiTZsW/fvsM6Fkkmc2nkJHkZ8Fng/VX1zOC6dkVdc12HqtpUVWNVNbZ8+fK5fjtpyTucKVAl9UySo+gS+aeq6nMt/HiSFVW1pzWV723x3cDJA5uvbLHdwJsOiH+txVdOUl7TWLXxSwtdBfWcV+bSiGg9y68DHqqqjw+s2gpM9EhfD3xhIH5p69W+Bni6NcdvA85Psqx1fDsf2NbWPZNkTXuvSwf2JWkOeWUujY7XA78L3JdkZ4v9CXAVcFOSy4AfAO9o624BLgLGgZ8A7wKoqv1JPgLc3cp9uKr2t+X3ANcDxwK3toekOWYyl0ZEVX0DmOq+7/MmKV/A5VPsazOweZL4DuD0w6impENgM7skST1nMpckqedM5pIk9ZzJXJKknjOZS5LUcyZzSZJ6btpknuQlSe5K8u02beKft/ipSe5sUx1+JsnRLX5Mez3e1q8a2NcHW/y7SS4YiK9tsfEkGw+sgyRJmtowV+bPAudW1WuBM+hmR1oDfAy4uqpeDTwJXNbKXwY82eJXt3K0qRYvAX6dblrEv0pyRJIjgE/STbd4GvDOVlaSJA1h2kFj2sARP24vj2qPAs4F/mOLbwE+RDeX8bq2DHAz8JdtaMd1wI1V9Szw/STjdHMhA4xX1SMASW5sZR88nAOTJPXDMGPTP3rVW+ahJv011G/m7Qp6J90EDNuB7wFPVdVzrcjgVIcvTI/Y1j8NvIqZT6coSZKGMFQyr6rnq+oMulmQzgZeM6e1moJzIEuS9GIz6s1eVU8BtwO/ARyXZKKZfnCqwxemTWzrXwk8wcGnU5wsPtn7OweyJEkHGKY3+/Ikx7XlY4E3Aw/RJfW3tWIHTps4MZ3i24Cvtt/dtwKXtN7upwKrgbvoZl5a3XrHH03XSW7rbBycJEmjYJhZ01YAW1qv818AbqqqLyZ5ELgxyUeBb9HNk0x7/ofWwW0/XXKmqh5IchNdx7bngMur6nmAJO+lmyP5CGBzVT0wa0coSdISN0xv9nuB100Sf4Sf9UYfjP8L8PYp9nUlcOUk8Vvo5k6WJC0hw/RU1+FzBDhJknrOZC5JUs+ZzCVJ6jmTuSRJPWcyl0ZIks1J9ia5fyD2oSS7k+xsj4sG1s1ocqSpJmCSNLdM5tJouZ5uoqMDXV1VZ7THLXDIkyNNNQGTpDlkMpdGSFV9nW78h2G8MDlSVX0fmJgc6Wza5EhV9VPgRmBdm1DpXLoJlqCbgOniWT0ASZMymUsCeG+Se1sz/LIWm+nkSK9i6gmYfo7zLEizy2Qu6VrgV4AzgD3AX8z1GzrPgjS7hhnOVdISVlWPTywn+Vvgi+3lwSZBmiz+BG0CpnZ1PuWkSZJml1fm0ohLsmLg5VuBiZ7uM5ocqU2oNNUETJLmkFfm0ghJ8mngTcAJSXYBVwBvSnIGUMCjwO/DIU+O9AEmn4BJ0hwymUsjpKreOUl4yoQ708mRppqASdLcspldkqSe88p8ng0zHeCjV71lHmoiSVoqvDKXJKnnTOaSJPWcyVySpJ7zN3NJmiPD9JGRZoNX5pIk9ZzJXJKknjOZS5LUcyZzSZJ6btpknuTkJLcneTDJA0ne1+LHJ9me5OH2vKzFk+SaJONtfuQzB/a1vpV/OMn6gfhZSe5r21yTJHNxsJIkLUXDXJk/B/xRVZ0GrAEuT3IasBG4rapWA7e11wAX0s2utBrYQDdXMkmOp5vU4Ry6sZuvmPgC0Mq8e2C7tYd/aJIkjYZpk3lV7amqb7blHwEPAScB64AtrdgW4OK2vA64oTp30M1vvAK4ANheVfur6klgO7C2rXtFVd3RplC8YWBfkiRpGjP6zTzJKuB1wJ3AiVW1p636IXBiWz4JeGxgs10tdrD4rknikiRpCEMn8yQvAz4LvL+qnhlc166oa5brNlkdNiTZkWTHvn375vrtJEnqhaGSeZKj6BL5p6rqcy38eGsipz3vbfHdwMkDm69ssYPFV04Sf5Gq2lRVY1U1tnz58mGqLknSkjdMb/YA1wEPVdXHB1ZtBSZ6pK8HvjAQv7T1al8DPN2a47cB5ydZ1jq+nQ9sa+ueSbKmvdelA/uSJEnTGGZs9tcDvwvcl2Rni/0JcBVwU5LLgB8A72jrbgEuAsaBnwDvAqiq/Uk+Atzdyn24qva35fcA1wPHAre2hyRJGsK0ybyqvgFMdd/3eZOUL+DyKfa1Gdg8SXwHcPp0dZEkSS/mCHDSCEmyOcneJPcPxBwASuo5k7k0Wq7nxYMyOQCU1HMmc2mEVNXXgf0HhB0ASuq5YTrA9daqjV9a6CpIfTDvA0Al2UB3tc8pp5xymNWX5JW5pBfM1wBQjhkhzS6TuaR5HwBK0uwymUtyACip55b0b+aSfl6STwNvAk5IsouuV7oDQEk9ZzKXRkhVvXOKVQ4AJfWYzeySJPWcV+aL0DC31D161VvmoSaSpD7wylySpJ4zmUuS1HMmc0mSes5kLklSz5nMJUnqOZO5JEk9ZzKXJKnnTOaSJPWcg8ZIkha9YQbTgtEdUMsrc0mSes5kLklSz5nMJUnqOZO5JEk9N20yT7I5yd4k9w/Ejk+yPcnD7XlZiyfJNUnGk9yb5MyBbda38g8nWT8QPyvJfW2ba5Jktg9SkqSlbJgr8+uBtQfENgK3VdVq4Lb2GuBCYHV7bACuhS75A1cA5wBnA1dMfAFoZd49sN2B7yVJkg5i2mReVV8H9h8QXgdsactbgIsH4jdU5w7guCQrgAuA7VW1v6qeBLYDa9u6V1TVHVVVwA0D+5IkSUM41N/MT6yqPW35h8CJbfkk4LGBcrta7GDxXZPEJ5VkQ5IdSXbs27fvEKsuSdLSctiDxlRVJanZqMwQ77UJ2AQwNjY2L+8pjYokjwI/Ap4HnquqsfYT2WeAVcCjwDuq6snWt+UTwEXAT4Dfq6pvtv2sB/6s7fajVbWFJWjYQUw0v4b5d1mKA8sc6pX5462JnPa8t8V3AycPlFvZYgeLr5wkLmlh/GZVnVFVY+31bPaPkTRHDjWZbwUmeqSvB74wEL+09WpfAzzdmuO3AecnWdZO7POBbW3dM0nWtG/6lw7sS9LCm5X+MfNdaWnUTNvMnuTTwJuAE5LsovvWfRVwU5LLgB8A72jFb6Frdhuna3p7F0BV7U/yEeDuVu7DVTXRqe49dD3mjwVubQ9J86+Ar7Sfzf6m/aw1W/1jfk6SDXRX9JxyyimzeQzSSJo2mVfVO6dYdd4kZQu4fIr9bAY2TxLfAZw+XT0kzbk3VNXuJL8EbE/yncGVs9k/xv4v0uxyBDhJAFTV7va8F/g83W/es9U/RtIcMplLIslLk7x8YpmuX8v9zFL/mHk8FGkkOZ95T43q7ReaMycCn2+jKR8J/GNVfTnJ3cxe/xhJc8RkLomqegR47STxJ5il/jGS5o7N7JIk9ZzJXJKknjOZS5LUcyZzSZJ6zmQuSVLPmcwlSeo5k7kkST1nMpckqeccNGYJG2aUOHCkOEnqO6/MJUnqOZO5JEk9ZzO7nLRFknrOK3NJknrOZC5JUs/ZzC5JGilL8adFr8wlSeo5r8w1lKX4TVaSlgqvzCVJ6jmvzCXpAMOOnigtFosmmSdZC3wCOAL4u6q6aoGrJOkQzMW5PJtDE5uotRQtimSe5Ajgk8CbgV3A3Um2VtWDC1szSTOx0OeyiVqjalEkc+BsYLyqHgFIciOwDjCZS/3iuawlYTa/GM5H5+DFksxPAh4beL0LOOfAQkk2ABvayx8n+e7A6hOAf56zGs6tPtcdWv3zsYWuxiHp82c/TN3/zXxUZMBsnMsLoc9/B4M8jsVltv9vnPJ8XizJfChVtQnYNNm6JDuqamyeqzQr+lx36Hf9rfvCONi5vBD6/FkO8jgWl/k8jsVya9pu4OSB1ytbTFK/eC5LC2CxJPO7gdVJTk1yNHAJsHWB6yRp5jyXpQWwKJrZq+q5JO8FttHdzrK5qh6Y4W4WTZPdIehz3aHf9bfus2iWzuWFsOg+y0PkcSwu83Ycqar5ei9JkjQHFkszuyRJOkQmc0mSem5JJPMka5N8N8l4ko0LWI9Hk9yXZGeSHS12fJLtSR5uz8taPEmuaXW+N8mZA/tZ38o/nGT9QPystv/xtm0Os76bk+xNcv9AbM7rO9V7zELdP5Rkd/v8dya5aGDdB1s9vpvkgoH4pH87rQPXnS3+mdaZiyTHtNfjbf2qQ6j7yUluT/JgkgeSvO9gn8ti++yXmiTvS3J/+7d4/0LXZ1gzOX8XsymO4+3t3+Nfk/TiFrUpjuN/JPlOO28/n+S4OatAVfX6QdfJ5nvALwNHA98GTlugujwKnHBA7L8DG9vyRuBjbfki4FYgwBrgzhY/HnikPS9ry8vaurta2bRtLzzM+r4ROBO4fz7rO9V7zELdPwT810nKntb+Lo4BTm1/L0cc7G8HuAm4pC3/NfBf2vJ7gL9uy5cAnzmEuq8AzmzLLwf+qdWxF5/9UnoApwP3A79I1yH4fwOvXuh6DVn3oc/fxfyY4jh+DfhV4GvA2ELX8TCO43zgyLb8sbn891gKV+YvDB9ZVT8FJoaPXCzWAVva8hbg4oH4DdW5AzguyQrgAmB7Ve2vqieB7cDatu4VVXVHdX8ZNwzs65BU1deB/QtQ36ne43DrPpV1wI1V9WxVfR8Yp/u7mfRvp13FngvcPMXnMFH3m4HzZtpCUlV7quqbbflHwEN0I6f14rNfYn6N7svRT6rqOeD/AP9hges0lBmev4vWZMdRVQ9V1UKPCjgjUxzHV9rfFcAddOMuzImlkMwnGz7ypAWqSwFfSXJPuuEqAU6sqj1t+YfAiW15qnofLL5rkvhsm4/6TvUes+G9rUlr80AT40zr/irgqYGTcLDuL2zT1j/dyh+S1kz/OuBO+v/Z99H9wL9L8qokv0jXCnLyNNssZv77Ll7/ia6VbE4shWS+mLyhqs4ELgQuT/LGwZXtKqk39wLOR31n+T2uBX4FOAPYA/zFLO13TiR5GfBZ4P1V9czguh5+9r1UVQ/RNX9+BfgysBN4fkErNUv89108kvwp8Bzwqbl6j6WQzBfN8JFVtbs97wU+T9eM+3hr9qQ9723Fp6r3weIrJ4nPtvmo71TvcViq6vGqer6q/hX4W7rP/1Dq/gRdU/aRB8R/bl9t/Stb+RlJchRdIv9UVX2uhXv72fdZVV1XVWdV1RuBJ+n6MPSV/76LTJLfA34L+J32BWtOLIVkviiGj0zy0iQvn1im6/hwf6vLRC/j9cAX2vJW4NLWU3kN8HRrHtsGnJ9kWWsmPh/Y1tY9k2RN+4320oF9zab5qO9U73FYJv4Ta95K9/lPvN8l6XqinwqspusgNunfTjvhbgfeNsXnMFH3twFfnekJ2j6P64CHqurjA6t6+9n3WZJfas+n0P1e/o8LW6PD4r/vIpJkLfDHwG9X1U/m9M3mqmfdfD7ofuf6J7qeyX+6QHX4Zbre0N8GHpioB93vqbcBD9P1lD2+xQN8stX5PgZ6bNL9tjLeHu8aiI/RJajvAX9JG8HvMOr8abrm6P9H97vqZfNR36neYxbq/g+tbvfS/ae2YqD8n7Z6fJeBuwCm+ttp/553tWP6X8AxLf6S9nq8rf/lQ6j7G+iaP++la9bd2erRi89+qT2A/0s33/q3gfMWuj4zqPfQ5+9ifkxxHG9ty88Cj9N9SV3wuh7CcYzT9WuZOM//eq7e3+FcJUnquaXQzC5J0kgzmUuS1HMmc0mSes5kLklSz5nMJUnqOZO5JEk9ZzKXJKnn/j/IvdH797Ag2QAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 576x288 with 2 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Fcu-qmHRycOK"
      },
      "source": [
        "Our task is to predict one number, __Log1pSalary__.\n",
        "\n",
        "To do so, our model can access a number of features:\n",
        "* Free text: __`Title`__ and  __`FullDescription`__\n",
        "* Categorical: __`Category`__, __`Company`__, __`LocationNormalized`__, __`ContractType`__, and __`ContractTime`__."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "p9vyA_erycOK",
        "outputId": "ebd96d18-a077-4ae1-a6a3-a2fe2b4c7fbb",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 215
        }
      },
      "source": [
        "text_columns = [\"Title\", \"FullDescription\"]\n",
        "categorical_columns = [\"Category\", \"Company\", \"LocationNormalized\", \"ContractType\", \"ContractTime\"]\n",
        "target_column = \"Log1pSalary\"\n",
        "\n",
        "data[categorical_columns] = data[categorical_columns].fillna('NaN') # cast missing values to string \"NaN\"\n",
        "\n",
        "data.sample(3)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Id</th>\n",
              "      <th>Title</th>\n",
              "      <th>FullDescription</th>\n",
              "      <th>LocationRaw</th>\n",
              "      <th>LocationNormalized</th>\n",
              "      <th>ContractType</th>\n",
              "      <th>ContractTime</th>\n",
              "      <th>Company</th>\n",
              "      <th>Category</th>\n",
              "      <th>SalaryRaw</th>\n",
              "      <th>SalaryNormalized</th>\n",
              "      <th>SourceName</th>\n",
              "      <th>Log1pSalary</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>35543</th>\n",
              "      <td>68238244</td>\n",
              "      <td>Office Cleaner</td>\n",
              "      <td>Office cleaner required. Reliable person to cl...</td>\n",
              "      <td>Wallington</td>\n",
              "      <td>Wallington</td>\n",
              "      <td>part_time</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Maintenance and More Ltd</td>\n",
              "      <td>Maintenance Jobs</td>\n",
              "      <td>6.50 - 6.50 per hour</td>\n",
              "      <td>12480</td>\n",
              "      <td>Jobcentre Plus</td>\n",
              "      <td>9.431963</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>227519</th>\n",
              "      <td>72444504</td>\n",
              "      <td>Sheet metal worker C****WLA</td>\n",
              "      <td>Overview: A privately owned Sheet Metal fabric...</td>\n",
              "      <td>Greenford Middlesex South East</td>\n",
              "      <td>UK</td>\n",
              "      <td>NaN</td>\n",
              "      <td>contract</td>\n",
              "      <td>Pioneer Selection</td>\n",
              "      <td>Engineering Jobs</td>\n",
              "      <td>From 11 to 13 per hour</td>\n",
              "      <td>23040</td>\n",
              "      <td>totaljobs.com</td>\n",
              "      <td>10.045031</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>165962</th>\n",
              "      <td>71295290</td>\n",
              "      <td>Learning Support Assistant  Primary</td>\n",
              "      <td>Learning Support Assistant South East London P...</td>\n",
              "      <td>South East London, London</td>\n",
              "      <td>South East London</td>\n",
              "      <td>NaN</td>\n",
              "      <td>contract</td>\n",
              "      <td>Remedy Recruitment Group Ltd</td>\n",
              "      <td>Teaching Jobs</td>\n",
              "      <td>70 - 90/day</td>\n",
              "      <td>19200</td>\n",
              "      <td>cv-library.co.uk</td>\n",
              "      <td>9.862718</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "              Id  ... Log1pSalary\n",
              "35543   68238244  ...    9.431963\n",
              "227519  72444504  ...   10.045031\n",
              "165962  71295290  ...    9.862718\n",
              "\n",
              "[3 rows x 13 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "IUdclucmycON"
      },
      "source": [
        "### Preprocessing text data\n",
        "\n",
        "Just like last week, applying NLP to a problem begins from tokenization: splitting raw text into sequences of tokens (words, punctuation, etc).\n",
        "\n",
        "__Your task__ is to lowercase and tokenize all texts under `Title` and `FullDescription` columns. Store the tokenized data as a __space-separated__ string of tokens for performance reasons.\n",
        "\n",
        "It's okay to use nltk tokenizers. Assertions were designed for WordPunctTokenizer, slight deviations are okay."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "YzeOxD_aycOO",
        "outputId": "54b66786-dc61-45a8-eb68-c9bdccc606df",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 75
        }
      },
      "source": [
        "print(\"Raw text:\")\n",
        "print(data[\"FullDescription\"][2::100000])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Raw text:\n",
            "2         Mathematical Modeller / Simulation Analyst / O...\n",
            "100002    A successful and high achieving specialist sch...\n",
            "200002    Web Designer  HTML, CSS, JavaScript, Photoshop...\n",
            "Name: FullDescription, dtype: object\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "RUWkpd7PycOQ",
        "colab": {}
      },
      "source": [
        "import nltk\n",
        "tokenizer = nltk.tokenize.WordPunctTokenizer()\n",
        "\n",
        "# see task above\n",
        "def normalize(text):\n",
        "    text = str(text).lower()\n",
        "    tokens = tokenizer.tokenize(text)\n",
        "    return ' '.join(tokens)\n",
        "\n",
        "data[text_columns] = data[text_columns].applymap(normalize) "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "o3pQdHihycOT"
      },
      "source": [
        "Now we can assume that our text is a space-separated list of tokens:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "Gs-6lnS_ycOU",
        "outputId": "8672a1c7-9b58-4fc7-9c77-d403c9f3dcec",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 75
        }
      },
      "source": [
        "print(\"Tokenized:\")\n",
        "print(data[\"FullDescription\"][2::100000])\n",
        "assert data[\"FullDescription\"][2][:50] == 'mathematical modeller / simulation analyst / opera'\n",
        "assert data[\"Title\"][54321] == 'international digital account manager ( german )'"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Tokenized:\n",
            "2         mathematical modeller / simulation analyst / o...\n",
            "100002    a successful and high achieving specialist sch...\n",
            "200002    web designer html , css , javascript , photosh...\n",
            "Name: FullDescription, dtype: object\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "ouE3L2hyycOX"
      },
      "source": [
        "Not all words are equally useful. Some of them are typos or rare words that are only present a few times. \n",
        "\n",
        "Let's count how many times is each word present in the data so that we can build a \"white list\" of known words."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "iC7hBwwjycOX",
        "colab": {}
      },
      "source": [
        "# Count how many times does each token occur in both \"Title\" and \"FullDescription\" in total\n",
        "# build a dictionary { token -> it's count }\n",
        "from collections import Counter\n",
        "from tqdm import tqdm as tqdm\n",
        "\n",
        "token_counts = Counter()                                                                    # <YOUR CODE HERE>\n",
        "for row in data[text_columns].values.flatten():\n",
        "    token_counts.update(row.split(' ')) \n",
        "\n",
        "# hint: you may or may not want to use collections.Counter"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "GiOWbc15ycOb",
        "outputId": "cc51dfb5-b7cc-4a26-c83b-1da0592d4eff",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 145
        }
      },
      "source": [
        "print(\"Total unique tokens :\", len(token_counts))\n",
        "print('\\n'.join(map(str, token_counts.most_common(n=5))))\n",
        "print('...')\n",
        "print('\\n'.join(map(str, token_counts.most_common()[-3:])))\n",
        "\n",
        "assert token_counts.most_common(1)[0][1] in  range(2600000, 2700000)\n",
        "assert len(token_counts) in range(200000, 210000)\n",
        "print('Correct!')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Total unique tokens : 202704\n",
            "('and', 2657388)\n",
            "('.', 2523216)\n",
            "(',', 2318606)\n",
            "('the', 2080994)\n",
            "('to', 2019884)\n",
            "...\n",
            "('stephanietraveltraderecruitmnt', 1)\n",
            "('ruabon', 1)\n",
            "('lowehays', 1)\n",
            "Correct!\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "nd5v3BNfycOf",
        "outputId": "b73691e0-7b5b-4ff1-f442-b0ae338e0530",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 279
        }
      },
      "source": [
        "# Let's see how many words are there for each count\n",
        "plt.hist(list(token_counts.values()), range=[0, 10**4], bins=50, log=True)\n",
        "plt.xlabel(\"Word counts\");"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEGCAYAAACevtWaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAASJ0lEQVR4nO3dfaxlVXnH8e/PoUDVdgAhhgLjjA5Bp5r4coMvfQlVq4My0libMpr6UupUDbbWJhVik+ofTbDaF41EnCJF+wKiNZaBMdRaLUbQMvgGCKMjYhmqDkg7WtOq6NM/9h49Xu6dOfeec+bcs+73k9zM3mvvs/daZ9957jrPXmftVBWSpLY8aNoVkCSNn8FdkhpkcJekBhncJalBBndJatAR064AwPHHH1/r16+fdjUkaabcdNNN91bVCQttWxHBff369ezatWva1ZCkmZLkq4ttm2paJsmWJNv3798/zWpIUnOmGtyrakdVbVu7du00qyFJzfGGqiQ1yOAuSQ0yuEtSgwzuktQgg7skNcjgLkkNmuqXmJJsAbZs3Lhx2cdYf/41C5bfeeFzl31MSZp1jnOXpAaZlpGkBhncJalBBndJapDBXZIaZHCXpAYZ3CWpQQZ3SWqQwV2SGjT24J7kjCQfT3JxkjPGfXxJ0qENFdyTXJpkX5Jb5pVvTrI7yZ4k5/fFBfwPcDSwd7zVlSQNY9ie+2XA5sGCJGuAi4AzgU3A1iSbgI9X1ZnA64A3jq+qkqRhDRXcq+o64L55xacDe6rqjqr6HnAFcHZV/bDf/l/AUWOrqSRpaKPMCnkScNfA+l7gyUmeDzwbOAZ4+2IvTrIN2Aawbt26EaohSZpv7FP+VtUHgA8Msd92YDvA3NxcjbsekrSajTJa5m7glIH1k/uyoSXZkmT7/v37R6iGJGm+UYL7jcCpSTYkORI4B7hqKQdwPndJmoxhh0JeDtwAnJZkb5Jzq+p+4DzgWuA24MqqunUpJ7fnLkmTMVTOvaq2LlK+E9i53JNX1Q5gx9zc3MuXewxJ0gM5/YAkNWiqwd20jCRNhg/IlqQGmZaRpAaZlpGkBpmWkaQGmZaRpAaZlpGkBpmWkaQGmZaRpAYZ3CWpQQZ3SWqQN1QlqUHeUJWkBpmWkaQGGdwlqUEGd0lqkMFdkhrkaBlJapCjZSSpQaZlJKlBBndJapDBXZIaZHCXpAYZ3CWpQQZ3SWqQ49wlqUGOc5ekBpmWkaQGGdwlqUEGd0lqkMFdkhpkcJekBhncJalBBndJapDBXZIaNJHgnuQhSXYlOWsSx5ckHdxQwT3JpUn2JbllXvnmJLuT7Ely/sCm1wFXjrOikqThDdtzvwzYPFiQZA1wEXAmsAnYmmRTkl8FvgDsG2M9JUlLcMQwO1XVdUnWzys+HdhTVXcAJLkCOBt4KPAQuoD/v0l2VtUPx1ZjSdIhDRXcF3EScNfA+l7gyVV1HkCSlwL3LhbYk2wDtgGsW7duhGpIkuab2GiZqrqsqq4+yPbtVTVXVXMnnHDCpKohSavSKMH9buCUgfWT+7KhOZ+7JE3GKMH9RuDUJBuSHAmcA1y1lAM4n7skTcawQyEvB24ATkuyN8m5VXU/cB5wLXAbcGVV3bqUk9tzl6TJGHa0zNZFyncCO5d78qraAeyYm5t7+XKPIUl6IKcfkKQG+YBsSWqQD8iWpAaZlpGkBpmWkaQGmZaRpAaZlpGkBhncJalB5twlqUHm3CWpQaZlJKlBBndJapA5d0lqkDl3SWqQaRlJapDBXZIaZHCXpAYZ3CWpQY6WkaQGOVpGkhpkWkaSGmRwl6QGGdwlqUFHTLsCk7L+/GsWLL/zwuce5ppI0uFnz12SGmRwl6QGOc5dkhrkOHdJapBpGUlqkMFdkhpkcJekBhncJalBBndJapDBXZIaZHCXpAYZ3CWpQWMP7kkek+TiJO9P8spxH1+SdGhDzQqZ5FLgLGBfVT12oHwz8FZgDXBJVV1YVbcBr0jyIOA9wDvGX+3lW2y2SHDGSEntGLbnfhmwebAgyRrgIuBMYBOwNcmmftvzgGuAnWOrqSRpaEMF96q6DrhvXvHpwJ6quqOqvgdcAZzd739VVZ0JvGiclZUkDWeUh3WcBNw1sL4XeHKSM4DnA0dxkJ57km3ANoB169aNUA1J0nxjfxJTVX0M+NgQ+20HtgPMzc3VuOshSavZKKNl7gZOGVg/uS8bmvO5S9JkjBLcbwROTbIhyZHAOcBVSzmA87lL0mQMFdyTXA7cAJyWZG+Sc6vqfuA84FrgNuDKqrp1KSe35y5JkzFUzr2qti5SvpMRhjtW1Q5gx9zc3MuXewxJ0gM5/YAkNWjso2WWIskWYMvGjRunWY0fWezbq35zVdKs8QHZktQg0zKS1KCpBndHy0jSZJiWkaQGmZaRpAZNdbTMrHAUjaRZY85dkhpkzl2SGmTOXZIaZHCXpAaZc5ekBplzl6QGmZaRpAY5zn0Ejn+XtFLZc5ekBhncJalBjpaRpAY5WkaSGuQN1QnwRqukaTPnLkkNMrhLUoMM7pLUIHPuh5G5eEmHiz13SWqQ49wlqUFTTctU1Q5gx9zc3MunWY9pM10jadxMy0hSgwzuktQgR8usYKZrJC2XPXdJapDBXZIaZFpmBpmukXQo9twlqUH23Btij17SAfbcJalBE+m5J/k14LnAzwLvqqp/nsR5JEkLGzq4J7kUOAvYV1WPHSjfDLwVWANcUlUXVtUHgQ8mORZ4C2Bwn6LF0jWLMY0jzb6lpGUuAzYPFiRZA1wEnAlsArYm2TSwyx/32yVJh9HQwb2qrgPum1d8OrCnqu6oqu8BVwBnp/Mm4ENV9emFjpdkW5JdSXbdc889y62/JGkBo+bcTwLuGljfCzwZeDXwTGBtko1VdfH8F1bVdmA7wNzcXI1YD43RwdI4pmyk2TCRG6pV9TbgbYfaL8kWYMvGjRsnUQ1JWrVGHQp5N3DKwPrJfdlQqmpHVW1bu3btiNWQJA0ated+I3Bqkg10Qf0c4IXDvtie++zxi1LSbFjKUMjLgTOA45PsBf6kqt6V5DzgWrqhkJdW1a3DHtMnMbXDoC+tLEMH96raukj5TmDn2GokSRqZD8iWpAb5gGxNxVLTOKZ9pKVxVkhN1FKnPljq/pIWZlpGkho01eDuOHdJmgznc5ekBhncJalBU72h6jdUNSpH0UgLcyikmjTNoO/DUbQSOBRSYnlDMA3KWsnMuUtSg8y5SyuU9xM0CnPu0jKttG/T+sdAg0zLSFKDDO6S1CCDuyQ1yOAuSQ1ytIxWlZV2E1SaFEfLSDNmNf6BciTQ0vkNValxBsbVyZy7JDXI4C5JDTItI02ZOXRNgsFdWqXGNTXxwY5jXn96DO6SJmZaPXRvIk85555kS5Lt+/fvn2Y1JKk5Uw3uVbWjqratXbt2mtWQpOaYlpE0lBZugq6mdI3BXdKq12LQd5y7JDXI4C5JDTItI0lLNAtpHHvuktQgg7skNci0jKSZ1cLwzEkxuEvSIsb5x+Nw5+nHnpZJ8sgk70ry/nEfW5I0nKGCe5JLk+xLcsu88s1JdifZk+R8gKq6o6rOnURlJUnDGbbnfhmwebAgyRrgIuBMYBOwNcmmsdZOkrQsQ+Xcq+q6JOvnFZ8O7KmqOwCSXAGcDXxhmGMm2QZsA1i3bt2Q1ZWklWsl3eAdJed+EnDXwPpe4KQkD0tyMfCEJBcs9uKq2l5Vc1U1d8IJJ4xQDUnSfGMfLVNV3wReMcy+SbYAWzZu3DjuakjSqjZKz/1u4JSB9ZP7sqE5n7skTcYowf1G4NQkG5IcCZwDXLWUA/gkJkmajGGHQl4O3ACclmRvknOr6n7gPOBa4Dbgyqq6dSknt+cuSZMx7GiZrYuU7wR2jrVGkqSR+YBsSWqQD8iWpAY55a8kNShVNe06kOQe4KvLfPnxwL1jrM4ssM2rg21eHUZp8yOqasFvga6I4D6KJLuqam7a9TicbPPqYJtXh0m12bSMJDXI4C5JDWohuG+fdgWmwDavDrZ5dZhIm2c+5y5JeqAWeu6SpHkM7pLUoJkO7gs9w3UWJTklyUeTfCHJrUl+vy8/LsmHk3yp//fYvjxJ3ta3+/NJnjhwrJf0+38pyUum1aZhJVmT5DNJru7XNyT5VN+29/YzjpLkqH59T799/cAxLujLdyd59nRaMpwkxyR5f5Lbk9yW5KmtX+ckf9D/Xt+S5PIkR7d2nRd6zvQ4r2uSJyW5uX/N25LkkJWqqpn8AdYAXwYeCRwJfA7YNO16LbMtJwJP7Jd/Bvgi3XNp/ww4vy8/H3hTv/wc4ENAgKcAn+rLjwPu6P89tl8+dtrtO0TbXwv8A3B1v34lcE6/fDHwyn75VcDF/fI5wHv75U39tT8K2ND/TqyZdrsO0t53A7/TLx8JHNPydaZ7YttXgJ8euL4vbe06A78MPBG4ZaBsbNcV+Pd+3/SvPfOQdZr2mzLCm/lU4NqB9QuAC6ZdrzG17Z+AXwV2Ayf2ZScCu/vldwJbB/bf3W/fCrxzoPwn9ltpP3QPePkI8HTg6v4X917giPnXmG5q6af2y0f0+2X+dR/cb6X9AGv7QJd55c1eZ378OM7j+ut2NfDsFq8zsH5ecB/Lde233T5Q/hP7LfYzy2mZBZ/hOqW6jE3/MfQJwKeAh1fV1/pNXwce3i8v1vZZe0/+Cvgj4If9+sOA/67uWQHwk/X/Udv67fv7/WepzRuAe4C/6VNRlyR5CA1f56q6G3gL8B/A1+iu2020fZ0PGNd1Palfnl9+ULMc3JuT5KHAPwKvqapvDW6r7k92M+NWk5wF7Kuqm6Zdl8PoCLqP7u+oqicA36H7uP4jDV7nY4Gz6f6w/RzwEGDzVCs1BdO4rrMc3Ed+hutKkuSn6AL731fVB/ribyQ5sd9+IrCvL1+s7bP0nvwC8LwkdwJX0KVm3gock+TAQ2QG6/+jtvXb1wLfZLbavBfYW1Wf6tffTxfsW77OzwS+UlX3VNX3gQ/QXfuWr/MB47qud/fL88sPapaD+8jPcF0p+jvf7wJuq6q/GNh0FXDgjvlL6HLxB8pf3N91fwqwv//4dy3wrCTH9j2mZ/VlK05VXVBVJ1fVerpr969V9SLgo8AL+t3mt/nAe/GCfv/qy8/pR1lsAE6lu/m04lTV14G7kpzWFz0D+AINX2e6dMxTkjy4/z0/0OZmr/OAsVzXftu3kjylfw9fPHCsxU37JsSINzCeQzey5MvA66ddnxHa8Yt0H9k+D3y2/3kOXa7xI8CXgH8Bjuv3D3BR3+6bgbmBY/02sKf/edm02zZk+8/gx6NlHkn3n3YP8D7gqL786H59T7/9kQOvf33/XuxmiFEEU27r44Fd/bX+IN2oiKavM/BG4HbgFuBv6Ua8NHWdgcvp7il8n+4T2rnjvK7AXP/+fRl4O/Nuyi/04/QDktSgWU7LSJIWYXCXpAYZ3CWpQQZ3SWqQwV2SGmRw14qX5C+TvGZg/doklwys/3mS1y7z2Gekn5HycEo3O+SrDvd5tXoY3DULPgE8DSDJg4DjgZ8f2P404PphDpRkzdhrtzzH0M2AKE2EwV2z4Hq6mQOhC+q3AN/uv8l3FPAY4NNJntFPyHVzP7/2UQBJ7kzypiSfBn4j3XMAbu/Xn7/QCdPNM/+Wfg7yzyd5dV9+sHMc3y/PJflYv/yGfr+PJbkjye/1p7gQeFSSzyZ5c5ITk1zXr9+S5Jcm8D5qFTni0LtI01VV/5nk/iTr6HrpN9DNivdUulkDb6brqFwGPKOqvpjkPcAr6WaeBPhmVT0xydF03xh8Ot23AN+7yGm30U3h+viquj/dgxeOPsQ5FvNo4Ffo5urfneQddBOGPbaqHg+Q5A/pvmr+p/2niwcP/w5JD2TPXbPierrAfiC43zCw/gngNLoJqr7Y7/9uugcoHHAgiD+63+9L1X09++8WOd8z6ebMvh+gqu4b4hyLuaaqvltV99JNHvXwBfa5EXhZkjcAj6uqbw9xXGlRBnfNigN598fRpWU+SddzHzbf/p3JVQ2A+/nx/6ej52377sDyD1jgE3NVXUf3h+Ju4LIkL55EJbV6GNw1K64HzgLuq6of9D3pY+gC/PV0k0mtT7Kx3/+3gH9b4Di39/s9ql/fusj5Pgz87oFpaZMcd4hz3Ak8qV/+9SHa8226NA398R8BfKOq/hq4hG4qYGnZDO6aFTfTjZL55Lyy/VV1b1X9H/Ay4H1JbqZ7utPF8w/S77cNuKa/obpv/j69S+imq/18ks8BLzzEOd4IvDXJLrre+UFV1TeBT/Q3T99MNzPm55J8BvhNurntpWVzVkhJapA9d0lqkMFdkhpkcJekBhncJalBBndJapDBXZIaZHCXpAb9P1m9/ETw6anQAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "znuXxeghycOh"
      },
      "source": [
        "Now filter tokens a list of all tokens that occur at least 10 times."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "SeNFBWx5ycOh",
        "colab": {}
      },
      "source": [
        "min_count = 10\n",
        "\n",
        "# tokens from token_counts keys that had at least min_count occurrences throughout the dataset\n",
        "tokens = [token for token, counts in token_counts.items() if counts >= min_count]                          # <YOUR CODE HERE>"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "RATIRyPKycOk",
        "outputId": "a0da9e29-c927-45d4-f06e-f05426fe4ec2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 40
        }
      },
      "source": [
        "# Add a special tokens for unknown and empty words\n",
        "UNK, PAD = \"UNK\", \"PAD\"\n",
        "tokens = [UNK, PAD] + sorted(tokens)\n",
        "print(\"Vocabulary size:\", len(tokens))\n",
        "\n",
        "assert type(tokens) == list\n",
        "assert len(tokens) in range(32000, 35000)\n",
        "assert 'me' in tokens\n",
        "assert UNK in tokens\n",
        "print(\"Correct!\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Vocabulary size: 34158\n",
            "Correct!\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "cqEsgbjZycOo"
      },
      "source": [
        "Build an inverse token index: a dictionary from token(string) to it's index in `tokens` (int)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "L60lo1l_ycOq",
        "colab": {}
      },
      "source": [
        "# You have already done that ;)\n",
        "\n",
        "token_to_id = {token: idx for idx, token in enumerate(tokens)}# <YOUR CODE HERE>"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "DeAoVo4mycOr",
        "outputId": "c3291017-36b0-4cbe-f6c2-e4c6d765b05b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 29
        }
      },
      "source": [
        "assert isinstance(token_to_id, dict)\n",
        "assert len(token_to_id) == len(tokens)\n",
        "for tok in tokens:\n",
        "    assert tokens[token_to_id[tok]] == tok\n",
        "\n",
        "print(\"Correct!\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Correct!\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "cmJAkq3gycOv"
      },
      "source": [
        "And finally, let's use the vocabulary you've built to map text lines into neural network-digestible matrices."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "JEsLeBjVycOw",
        "colab": {}
      },
      "source": [
        "UNK_IX, PAD_IX = map(token_to_id.get, [UNK, PAD])\n",
        "\n",
        "def as_matrix(sequences, max_len=None):\n",
        "    \"\"\" Convert a list of tokens into a matrix with padding \"\"\"\n",
        "    if isinstance(sequences[0], str):\n",
        "        sequences = list(map(str.split, sequences))\n",
        "        \n",
        "    max_len = min(max(map(len, sequences)), max_len or float('inf'))\n",
        "    \n",
        "    matrix = np.full((len(sequences), max_len), np.int32(PAD_IX))\n",
        "    for i,seq in enumerate(sequences):\n",
        "        row_ix = [token_to_id.get(word, UNK_IX) for word in seq[:max_len]]\n",
        "        matrix[i, :len(row_ix)] = row_ix\n",
        "    \n",
        "    return matrix"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "JiBlPkdKycOy",
        "outputId": "31d79451-d5c5-4a78-eb39-a518216d5d3b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 122
        }
      },
      "source": [
        "print(\"Lines:\")\n",
        "print('\\n'.join(data[\"Title\"][::100000].values), end='\\n\\n')\n",
        "print(\"Matrix:\")\n",
        "print(as_matrix(data[\"Title\"][::100000]))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Lines:\n",
            "engineering systems analyst\n",
            "hr assistant\n",
            "senior ec & i engineer\n",
            "\n",
            "Matrix:\n",
            "[[10807 30161  2166     1     1]\n",
            " [15020  2844     1     1     1]\n",
            " [27645 10201    16 15215 10804]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "nGOdZ3-dycO4"
      },
      "source": [
        "Now let's  encode the categirical data we have.\n",
        "\n",
        "As usual, we shall use one-hot encoding for simplicity. Kudos if you implement more advanced encodings: tf-idf, pseudo-time-series, etc."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "DpOlBp7ZycO6",
        "outputId": "9ecafb5b-dfc0-4ca4-a0d8-edf89fac5acc",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 40
        }
      },
      "source": [
        "from sklearn.feature_extraction import DictVectorizer\n",
        "\n",
        "# we only consider top-1k most frequent companies to minimize memory usage\n",
        "top_companies, top_counts = zip(*Counter(data['Company']).most_common(1000))\n",
        "recognized_companies = set(top_companies)\n",
        "data[\"Company\"] = data[\"Company\"].apply(lambda comp: comp if comp in recognized_companies else \"Other\")\n",
        "\n",
        "categorical_vectorizer = DictVectorizer(dtype=np.float32, sparse=False)\n",
        "categorical_vectorizer.fit(data[categorical_columns].apply(dict, axis=1))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "DictVectorizer(dtype=<class 'numpy.float32'>, separator='=', sort=True,\n",
              "               sparse=False)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 23
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "yk4jmtAYycO8"
      },
      "source": [
        "### The deep learning part\n",
        "\n",
        "Once we've learned to tokenize the data, let's design a machine learning experiment.\n",
        "\n",
        "As before, we won't focus too much on validation, opting for a simple train-test split.\n",
        "\n",
        "__To be completely rigorous,__ we've comitted a small crime here: we used the whole data for tokenization and vocabulary building. A more strict way would be to do that part on training set only. You may want to do that and measure the magnitude of changes."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "TngLcWA0ycO_",
        "outputId": "880b9774-4857-4c64-ab26-d08d50b932d4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 40
        }
      },
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "data_train, data_val = train_test_split(data, test_size=0.2, random_state=42)\n",
        "data_train.index = range(len(data_train))\n",
        "data_val.index = range(len(data_val))\n",
        "\n",
        "print(\"Train size = \", len(data_train))\n",
        "print(\"Validation size = \", len(data_val))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train size =  195814\n",
            "Validation size =  48954\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "2PXuKgOSycPB",
        "colab": {}
      },
      "source": [
        "def make_batch(data, max_len=None, word_dropout=0):\n",
        "    \"\"\"\n",
        "    Creates a keras-friendly dict from the batch data.\n",
        "    :param word_dropout: replaces token index with UNK_IX with this probability\n",
        "    :returns: a dict with {'title' : int64[batch, title_max_len]\n",
        "    \"\"\"\n",
        "    batch = {}\n",
        "    batch[\"Title\"] = as_matrix(data[\"Title\"].values, max_len)\n",
        "    batch[\"FullDescription\"] = as_matrix(data[\"FullDescription\"].values, max_len)\n",
        "    batch['Categorical'] = categorical_vectorizer.transform(data[categorical_columns].apply(dict, axis=1))\n",
        "    \n",
        "    if word_dropout != 0:\n",
        "        batch[\"FullDescription\"] = apply_word_dropout(batch[\"FullDescription\"], 1. - word_dropout)\n",
        "    \n",
        "    if target_column in data.columns:\n",
        "        batch[target_column] = data[target_column].values\n",
        "    \n",
        "    return batch\n",
        "\n",
        "def apply_word_dropout(matrix, keep_prop, replace_with=UNK_IX, pad_ix=PAD_IX,):\n",
        "    dropout_mask = np.random.choice(2, np.shape(matrix), p=[keep_prop, 1 - keep_prop])\n",
        "    dropout_mask &= matrix != pad_ix\n",
        "    return np.choose(dropout_mask, [matrix, np.full_like(matrix, replace_with)])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "I6LpEQf0ycPD",
        "colab": {}
      },
      "source": [
        "a = make_batch(data_train[:3], max_len=10)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "0eI5h9UMycPF"
      },
      "source": [
        "#### Architecture\n",
        "\n",
        "Our main model consists of three branches:\n",
        "* Title encoder\n",
        "* Description encoder\n",
        "* Categorical features encoder\n",
        "\n",
        "We will then feed all 3 branches into one common network that predicts salary.\n",
        "\n",
        "<img src=\"https://github.com/yandexdataschool/nlp_course/raw/master/resources/w2_conv_arch.png\" width=600px>\n",
        "\n",
        "This clearly doesn't fit into PyTorch __Sequential__ interface. To build such a network, one will have to use [__PyTorch nn.Module API__](https://pytorch.org/docs/stable/nn.html#torch.nn.Module)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4YShw1xHXuiS",
        "colab_type": "text"
      },
      "source": [
        "But to start with let's build the simple model using only the part of the data. Let's create the baseline solution using only the description part (so it should definetely fit into the Sequential model)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EDENpmbFXuiU",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import torch\n",
        "from torch import nn\n",
        "import torch.nn.functional as F"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9osIfGo7Xuip",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# You will need these to make it simple\n",
        "\n",
        "class Flatten(nn.Module):\n",
        "    def forward(self, input):\n",
        "        return input.view(input.size(0), -1)\n",
        "\n",
        "class Reorder(nn.Module):\n",
        "    def forward(self, input):\n",
        "        return input.permute((0, 2, 1))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uYjl3RGMXuix",
        "colab_type": "text"
      },
      "source": [
        "To generate minibatches we will use simple pyton generator."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qfjCurg-Xui0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def iterate_minibatches(data, batch_size=256, shuffle=True, cycle=False, **kwargs):\n",
        "    \"\"\" iterates minibatches of data in random order \"\"\"\n",
        "    while True:\n",
        "        indices = np.arange(len(data))\n",
        "        if shuffle:\n",
        "            indices = np.random.permutation(indices)\n",
        "\n",
        "        for start in range(0, len(indices), batch_size):\n",
        "            batch = make_batch(data.iloc[indices[start : start + batch_size]], **kwargs)\n",
        "            target = batch.pop(target_column)\n",
        "            yield batch, target\n",
        "        \n",
        "        if not cycle: break"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kOPRluJIXui8",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "iterator = iterate_minibatches(data_train, 3)\n",
        "batch, target = next(iterator)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FIw7SnHeXujD",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Here is some startup code:\n",
        "n_tokens=len(tokens)\n",
        "n_cat_features=len(categorical_vectorizer.vocabulary_)\n",
        "hid_size=64\n",
        "n_max = 2\n",
        "simple_model = nn.Sequential()\n",
        "\n",
        "simple_model.add_module('emb', nn.Embedding(num_embeddings=n_tokens, embedding_dim=hid_size))\n",
        "simple_model.add_module('reorder', Reorder())\n",
        "# <YOUR CODE HERE>\n",
        "\n",
        "simple_model.add_module('conv1', nn.Conv1d(hid_size, hid_size*2, kernel_size = 2))\n",
        "simple_model.add_module('relu' , nn.ReLU(True))\n",
        "\n",
        "simple_model.add_module('conv2', nn.Conv1d(hid_size*2, hid_size*2, kernel_size = 3))\n",
        "simple_model.add_module('relu2' , nn.ReLU(True))\n",
        "\n",
        "simple_model.add_module('conv3', nn.Conv1d(hid_size*2, hid_size*2, kernel_size = 3))\n",
        "simple_model.add_module('bn1' , nn.BatchNorm1d(hid_size*2))\n",
        "simple_model.add_module('adaptive_pool', nn.AdaptiveAvgPool1d(n_max))\n",
        "simple_model.add_module('flatten', nn.Flatten())\n",
        "simple_model.add_module('out', nn.Linear(hid_size*2*n_max, 1))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZE3dweqzXujO",
        "colab_type": "text"
      },
      "source": [
        "__Remember!__ We are working with regression problem and predicting only one number."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9VdHTRWPXujP",
        "colab_type": "code",
        "outputId": "70f1efcc-323d-4399-8d05-a34b566b8229",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 139
        }
      },
      "source": [
        "# Try this to check your model. `torch.long` tensors are required for nn.Embedding layers.\n",
        "simple_model(torch.tensor(batch['FullDescription'], dtype=torch.long)).shape"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "error",
          "ename": "IndexError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mIndexError\u001b[0m                                Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-64-6ab0c28ecff1>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m# Try this to check your model. `torch.long` tensors are required for nn.Embedding layers.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0msimple_model\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtensor\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbatch\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'FullDescription'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlong\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;31mIndexError\u001b[0m: too many indices for tensor of dimension 2"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XHDFIyg6mfHI",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "device = torch.device(\"cuda:0\") if torch.cuda.is_available() else torch.device('cpu')\n",
        "simple_model = simple_model.to(device)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pe3Ssj3bXuja",
        "colab_type": "text"
      },
      "source": [
        "And now simple training pipeline:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_7q7yUwbXujd",
        "colab_type": "code",
        "outputId": "b5f39a80-5008-4c1e-bda5-8b98347613b9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 265
        }
      },
      "source": [
        "from IPython.display import clear_output\n",
        "from random import sample\n",
        "\n",
        "epochs = 1\n",
        "\n",
        "model = simple_model\n",
        "opt = torch.optim.Adam(model.parameters())\n",
        "loss_func = nn.MSELoss()                                                                  # <YOUR CODE HERE>\n",
        "\n",
        "history = []\n",
        "for epoch_num in range(epochs):\n",
        "    for idx, (batch, target) in enumerate(iterate_minibatches(data_train)):\n",
        "        # Preprocessing the batch data and target\n",
        "        batch = torch.tensor(batch['FullDescription'], dtype=torch.long).to(device)\n",
        "        target = torch.tensor(target).to(device)\n",
        "\n",
        "\n",
        "        predictions = model(batch)\n",
        "        predictions = predictions.view(predictions.size(0))\n",
        "\n",
        "        loss = loss_func(predictions, target)                                               # <YOUR CODE HERE>\n",
        "\n",
        "        # train with backprop\n",
        "        loss.backward()\n",
        "        opt.step()\n",
        "        opt.zero_grad()\n",
        "\n",
        "        history.append(loss.item())\n",
        "        if (idx+1)%10==0:\n",
        "            clear_output(True)\n",
        "            plt.plot(history,label='loss')\n",
        "            plt.legend()\n",
        "            plt.show()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD4CAYAAAAXUaZHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3deXiU5b3/8fc9k0kCSdhCCEvAgKKoIIiIogWr1oraulRPq7WK1uWcU7uf04r1d2ptbbVyWpduauuCp26t1WpFRUQE3NCwyr6ELUDIYshKksnM/ftjnpnMTCZAMgmZmXxe15UrM89s3yH6mXu+z/3cj7HWIiIiqcXV0wWIiEjXU7iLiKQghbuISApSuIuIpCCFu4hICkrr6QIABg8ebAsLC3u6DBGRpLJ8+fIKa21erNsSItwLCwspKirq6TJERJKKMWZne7epLSMikoIU7iIiKUjhLiKSghKi5y4i0hW8Xi8lJSU0Njb2dCldKjMzk4KCAjwezxE/RuEuIimjpKSEnJwcCgsLMcb0dDldwlpLZWUlJSUljB49+ogfp7aMiKSMxsZGcnNzUybYAYwx5ObmdvjbiMJdRFJKKgV7UGfeU1KH+yc7PuO3b22iucXf06WIiCSUpA73FTurePidrXh9CncR6XnZ2dk9XUJIUoe7y/mqotONiIhESupwD7ah/DqblIgkEGstP/rRjxg/fjwTJkzghRdeAGDfvn3MmDGDSZMmMX78eJYuXYrP5+OGG24I3feBBx7okhqSeipkaOSuroyIRLn7X+tYv7emS5/zpOH9uOvLJx/2fi+99BKrVq1i9erVVFRUcPrppzNjxgyeffZZLrzwQu688058Ph8NDQ2sWrWKPXv2sHbtWgAOHDjQJbUm9cjdpZG7iCSg9957j2uuuQa3201+fj7nnHMOn3zyCaeffjpPPvkkP/vZz/j000/JyclhzJgxFBcX853vfIc333yTfv36dUkNyT1yd9Jd4S4i0Y5khH20zZgxgyVLljBv3jxuuOEGfvjDH3L99dezevVq5s+fzyOPPMLf/vY3nnjiibhfK6lH7sG5n35lu4gkkOnTp/PCCy/g8/koLy9nyZIlTJ06lZ07d5Kfn88tt9zCzTffzIoVK6ioqMDv93PllVdyzz33sGLFii6p4bAjd2PME8CXgDJr7Xhn2yDgBaAQ2AF81VpbZQJp+xBwMdAA3GCt7ZpKYwi2ZaxG7iKSQK644go+/PBDJk6ciDGG+++/n6FDhzJ37lzmzJmDx+MhOzubp59+mj179nDjjTfi9wd2Ht57771dUsORtGWeAn4PPB22bTaw0Fp7nzFmtnP9duAiYKzzcwbwJ+d3t3Bp5C4iCaSurg4IdBXmzJnDnDlzIm6fNWsWs2bNavO4rhqthztsW8ZauwT4LGrzZcBc5/Jc4PKw7U/bgI+AAcaYYV1VbDTtUBURia2zPfd8a+0+53IpkO9cHgHsDrtfibOtDWPMrcaYImNMUXl5eaeKaO25K9xFRMLFvUPVBhreHU5Xa+1j1top1topeXkxz+96WKF57sp2EXGk4j64zrynzob7/mC7xfld5mzfA4wMu1+Bs61bqC0jIuEyMzOprKxMqYAPrueemZnZocd1dp77q8As4D7n9yth279tjHmewI7U6rD2TZfTDlURCVdQUEBJSQmdbfUmquCZmDriSKZCPgd8HhhsjCkB7iIQ6n8zxtwE7AS+6tz9dQLTILcSmAp5Y4eq6SCtLSMi4TweT4fOVpTKDhvu1tpr2rnp/Bj3tcBt8RZ1pFp77gp3EZFwSX2EqtoyIiKxJXm4B36rLSMiEimpwz00z11L/oqIREjqcNfIXUQktiQPdx3EJCISS3KHu1O9Ru4iIpGSOty1toyISGxJHe6aCikiEluSh3vgtw5iEhGJlOThrpG7iEgsSR3uWltGRCS2pA53l3aoiojElNTh7gzcNc9dRCRKUoe7y6WRu4hILMkd7qHZMj1bh4hIoknqcNdBTCIisSV1uGttGRGR2JI83AO/NXIXEYmU5OGug5hERGJJ6nDXQUwiIrEldbjrBNkiIrGlRLirLSMiEinJwz3wW20ZEZFISR3uRiN3EZGYkjrctZ67iEhsSR7uOkJVRCSW1Ah3fw8XIiKSYJI63DXPXUQktqQO9+CSv8p2EZFIcYW7MeYHxph1xpi1xpjnjDGZxpjRxphlxpitxpgXjDHpXVVsNE2FFBGJrdPhbowZAXwXmGKtHQ+4gauBXwMPWGuPA6qAm7qi0Fh0EJOISGzxtmXSgD7GmDSgL7APOA940bl9LnB5nK/RLvXcRURi63S4W2v3AP8L7CIQ6tXAcuCAtbbFuVsJMCLW440xtxpjiowxReXl5Z2qQWvLiIjEFk9bZiBwGTAaGA5kATOP9PHW2sestVOstVPy8vI6VYPaMiIiscXTlvkCsN1aW26t9QIvAWcDA5w2DUABsCfOGtulHaoiIrHFE+67gDONMX1NYJGX84H1wCLgKuc+s4BX4iuxfVpbRkQktnh67ssI7DhdAXzqPNdjwO3AD40xW4Fc4PEuqDOm0Mhd6S4iEiHt8Hdpn7X2LuCuqM3FwNR4nvdIpbkCn00+tWVERCIk9RGqbmfo7tPIXUQkQlKHe5oT7l6fVg4TEQmX1OHuchlcRiN3EZFoSR3uEOi7tyjcRUQiJH24u11GI3cRkShJH+5pLkOLT+EuIhIu6cPd7Ta06FRMIiIRkj7c01xGPXcRkSgpEO4ufGrLiIhESPpwd2vkLiLSRtKHe5rb4FPPXUQkQtKHu9tl8GrkLiISIenDPc1l1HMXEYmSAuGuI1RFRKIlf7ir5y4i0kbSh7tmy4iItJX04a7lB0RE2kr6cNfCYSIibSV9uHvcLq0tIyISJenDXSN3EZG2kj7ctXCYiEhbSR/uGrmLiLSV9OGe5nLpBNkiIlGSPtw9brVlRESipUC4u/C2aOQuIhIu+cM9zUWzDmISEYmQ9OGe7lbPXUQkWtKHu8dtFO4iIlHiCndjzABjzIvGmI3GmA3GmGnGmEHGmAXGmC3O74FdVWwsHo3cRUTaiHfk/hDwprV2HDAR2ADMBhZaa8cCC53r3SYQ7hZr1XcXEQnqdLgbY/oDM4DHAay1zdbaA8BlwFznbnOBy+Mt8lDS0wJvwaudqiIiIfGM3EcD5cCTxpiVxpi/GGOygHxr7T7nPqVAfqwHG2NuNcYUGWOKysvLO12Ex20A1JoREQkTT7inAZOBP1lrTwXqiWrB2ECvJOaQ2lr7mLV2irV2Sl5eXqeL8LiDI3eFu4hIUDzhXgKUWGuXOddfJBD2+40xwwCc32XxlXhowXBvVriLiIR0OtyttaXAbmPMCc6m84H1wKvALGfbLOCVuCo8jHS3eu4iItHS4nz8d4BnjDHpQDFwI4EPjL8ZY24CdgJfjfM1DsmTFui5N2sJAhGRkLjC3Vq7CpgS46bz43nejlDPXUSkrRQ4QtXpuWvkLiISkvThnq6Ru4hIG0kf7hq5i4i0lfThnukJvIVGhbuISEjSh3vf9MA+4YPNLT1ciYhI4kiBcHcD8N7WCuqbFPAiIpBC4f7Xj3bxvedX9XA1IiKJIfnDPaN1qv6new70YCUiIokj6cO9j8fd0yWIiCScpA93t8uELhvMIe4pItJ7JH24hzPKdhERIMXCXUREAlIq3K2F6oPeni5DRKTHpVS4l9Y0MvHut6jTfHcR6eVSKtyDGhTuItLLpUS4R+9INdqzKiK9XEqEe3DZ3yAb+5zcIiK9RkqEe0Za5Nvw+RXuItK7pUS4337RuIjrCncR6e1SItyvPeMYvjZlZOi6X0u7i0gvlxLhDpCXkxG63KJ0F5FeLiXD3W/VlhGR3i0lw71FPXcR6eVSJtxzs9JDl7VDVUR6u5QJ95xMT+jy34tKerASEZGel0Lh3npGpqc+2NFzhYiIJICUCffssNPtiYj0dikT7lkKdxGRkJQJ9/S0lHkrIiJxizsRjTFuY8xKY8xrzvXRxphlxpitxpgXjDHph3sOERHpWl0x3P0esCHs+q+BB6y1xwFVwE1d8BoiItIBcYW7MaYAuAT4i3PdAOcBLzp3mQtcHs9rdMSDX5t0tF5KRCShxTtyfxD4MRBczCUXOGCtDZ4KqQQYEeuBxphbjTFFxpii8vLyOMsIOG5Idpc8j4hIsut0uBtjvgSUWWuXd+bx1trHrLVTrLVT8vLyOltGhDR36xmYrNaXEZFeLJ75g2cDlxpjLgYygX7AQ8AAY0yaM3ovAPbEX+aRcYedXs/ntxFhLyLSm3R65G6tvcNaW2CtLQSuBt6x1l4LLAKucu42C3gl7iqPkNvVGuZaPExEerPumBx+O/BDY8xWAj34x7vhNWJSuIuIBHTJYZ3W2neBd53LxcDUrnjejgoPd59P4S4ivVdKHdYZHu5ev5+6ppZD3FtEJHWlbLgv3lTO+Lvm8+G2yh6sSESkZ6RWuIfNlpn36T4ANpbW9FQ5IiI9JqXCPXwf6kfFgRF7fr/MHqpGRKTnpFS4h3VlaGj2AeBxp9RbFBE5IimVfLnZGUwbkxuxrcXnb+feIiKpK6XCHWDWWcdEXNd8dxHpjVIu3JtaIkfqPoW7iPRCKRfuwV57kEbuItIbpVy410cduOTzq+cuIr1PyoX7yEF9I65r5C4ivVHKhfuFJw9l1rTWnarquYtIb5Ry4Q5w92XjWfXTCwBo0QJiItILpWS4A6Q5By9p5C4ivVHqhrtzuKp67iLSG6VsuAdXiPT5/SwrrmRTaW0PVyQicvR0yck6ElFwhUivz/K1xz4CYMd9l/RkSSIiR03KjtxdLoPLwO6qhp4uRUTkqEvZcIfATtVt5fUAZKSl9FsVEYmQ0onX3OJn9e4DAPRJd/dwNSIiR09Kh3u4zDSFu4j0Hr0n3D295q2KiPSmcNfIXUR6j14T7u7wc/CJiKS4XhPuWoZARHqTXhPu3hjnUo1e+11EJFX0inDv43G3WWNm+c4qTr5rPu9s3B/adrDZR6PXF/1wEZGkk9Lh/uzNZ/D6d6dz8YRhNHn9EaP35Ts/A+CDrZWhbSf+9E0+P+fdo12miEiXS+lwP+u4wZw0vB8et6G0ppGxd74Rui2Y89E7WktrGo9miSIi3aLT4W6MGWmMWWSMWW+MWWeM+Z6zfZAxZoExZovze2DXlds5ae62M2X8NtCmcWkWjYikoHhG7i3Af1lrTwLOBG4zxpwEzAYWWmvHAgud6z0qzRX5Nq21zJm/CWhdPVJEJJV0OtyttfustSucy7XABmAEcBkw17nbXODyeIuMlydq5N7obe29a+AuIqmoS3ruxphC4FRgGZBvrd3n3FQK5LfzmFuNMUXGmKLy8vKuKKNd7rCRu89vaWppnRFjNHIXkRQUd7gbY7KBfwDft9bWhN9mrbVAzKOHrLWPWWunWGun5OXlxVvGIYWP3JtafBEjd6/Pz1vrSjnuJ693aw0iIkdTXOFujPEQCPZnrLUvOZv3G2OGObcPA8riKzF+4T33Rq+fM+9dGHH9tws261yrIpJS4pktY4DHgQ3W2t+G3fQqMMu5PAt4pfPldY3w2TLRByk1tvjatGZG3zHvqNQlItJd4hm5nw1cB5xnjFnl/FwM3AdcYIzZAnzBud6jwtsy1Qe9Ebc1en1tdqraXjiI/9fqvUy4a37E/ggRSV6dPkG2tfY9oL29ked39nm7Q3hb5jdvbY64rcnr14qRwK9e30BtUwsVdc2MGNCnp8sRkTil9BGqQTmZrZ9hb2/YH3FbeV1TzBkzR2sVyeoGLxPums9HxZWHv3M38rgD/yl4W9ousCYiyadXhHv/Pp6Y28cNzWFHRX3Mue4b9tW03dgNVpccoLaphT8s2npUXq89wdZVrNUzRST59IpwH9A3PXT5G2eOCl0eNzSHstqmmEv/ful37x2V2vwJ0uAPjtzDp4mKSPLqFeEeHLmPHpzFPZdPCPWUp48NzK/fvL+ux2oLRntPH0yVnhb4T+FgJ5c89vstlXVNXVmSiMShV4R7sOfe0BwYoQenRk4aNYCBfWO3bI4W64zce3qXbnDk3tlwf3DhFk67523KaxXwIomgV4R7cOQ+dXQuAHdcdCLZGWmMGNCHCQUD2n1ccM337hTsyvT0hJ30YLg3dy7c31pXCqBwF0kQvSLcszLSWPhf5zDnqlMAmDl+KGvvvpBMj5uJBf3bfdyVf/ow4vrOynr2HDjYJTXtPXCQwtnzeHnlHuDotmWqG7z4o2YDeUJtmc6dejBYv4292oSIHGW9ItwBjs3LJtPjbrP9lEOM3AFunlvEks3lfO3RDzlnzrucfd87R/yaH2yroHD2vJi96I2lgdk4r60JrLHmMoHQ/e1bm2JOw/xgWwV3vPTpEb92e6rqm5n487d4aOGWiO3pTqvqYHPndqgGP5oSZP+wSK/Xa8K9PRNGtB25D8pqnV3z9ob9XP/Exyzb3tqisdayeHM5r67eC0Bto5fHlmyjcPY8fvDCqtD9Hl1cDMCakmqstTz94Q5qGwNHyLadcWi4+1/rePidrSze3HY5nq//eRnPfbwr7vn3OyrrAVi4MXK+f7Dnvq/6IP/zz7XMfHBJh543+MWjKcHnyV/92IfMuH9RT5ch0u06fYRqqsjvlxFx/f6rTqGyrplfv7mx3cd87/lVoWAfOySbix5aGrrt5ZV7eOBrkyLub7G8u7mcn76yjs37a7nn8gn4/JEh6DKtp/gLBm0sjV4fWRmd/7Ptd14jNyvyffdJD3yreXNtKVvKOj57KBTuCX6C8Y+Ku38/ikgi6PUjd2MMv7jsZMYMzgICvfDsjLbtm3DBYAditko+2fFZxAh7f00TZU6oNjQFwq+uKTIEK+ub+WBb4CjVQ50d6nCzWZZuKWfrIcK5pCqwzyD82wkQ6sGHz/mP7ssfinEaM41am0YkIfT6cAe4blohf/zGZCAw9318WKvmgpNinmskZNXuA222/dsjH/LI4m3s+qwBCHwAPLAg0OPu58zcqYlawGz5zqrQ5UMFZKPXx5+XFLc7K+W6xz/mC79d3O7jqxqagbZnpwoueRzeVvnqo5E7lMMt2lTGil2tNbeO3BO7LSPSW/T6tkzQuKH92HHfJUBgxHrhyfl848xjmD42j2N/8nqHe93PLtsVMbMm2HJ56oMdPPXBDi4aP7Tdx0YfJbp2T3Xo8opdB/jl6xtYurWCq08fyYEGL18/Y1T0U8T0wbYKnl22C4CDUa8RfH+V9c2hbUVhHzjRbnzyE4DQv1nwo0Ijd5HEoJF7DC6X4dHrpoSOYB2Sk9HmPh634crJBay+64tcP+2YNrcfbsrkG2tL271t3qf7eDPs9vClEII7ZD+rb+Jbz6zgJy+3toVs2FSVV1fv5dOS1g8FCOyUrWoIPP5gc+SUx7hPVuIM3bV8gUhi0Mj9CMw6q5D73tjIK7edzYiBfVi9+wDnHJ9HmrPj89KJw3n6w51d9nrz1uxj3pp9DOjr4ffXTI647YATzuFTDhdtKsPns0w/fnBo23efWwnAhp/PDO0sDdcQdbBSvLNwgiP3RN+hKtJbaOR+BP59xhhW3/VFJo4cwODsDM4/MT8U7AATRwbmyt9wVmG7z3HG6EFtth03JJsvTxxOYW7fmI850ODlG48vi9hW4cyZDw/3W+YW8es3N8Y8unTO/E2s3VNNcXnkTtbocO/IyD38g+DGJz9mf01jqOfeGGMq5Lubynjw7c1ttneH2kZvmxOyJKJnlu1kt7NPRqQ7KNyPgDGm3WWDITB1cesvL+KuL59ERpqLb597HBt/MTN0++dPyOOFf5/Giv+5gK9NGRnaPnvmOH53zakxD65qT3BH6vqwJYlb/JaqhuY2gQ2wcncVX/rde5z3m8idrNErYUZPzQzasr824vrCDftZuqU8dH3RpnLmfrAjdL2hqYXtFfW8t6UitO2GJz/hwbdbD5raVl5H4ex5LNp0ZKfXLS6vo8ZpRy3eXE7h7HmU1TbGvO/Eu99i4t1vHdHz9pSDzT7ufHktVz/2UU+XIilM4d5F0twujDFsuuci/vvCE8j0uPnD1yfzqysm8Oh1pwGB6Ye/vuoU/u+mqRgDp4wMzMr53TWncs/l4/nuecfFfO51d1/IUzeeDrQe0RqtqsEbWhgNYMbxgf0FK3e1nc0DsKWsLtTX/8Oirby/NfbJQr71zIqI6zfNLeIGZ2dq0MC+6bT4AqP5kqqDnPu/77b5xgGt+wQ+2BoI/rfWlWKtPeSUS2st5/1mMdc//jEAf3TWvd9UWhvz/kf6BcT24KG0zc4RbJ+F7bwW6WrquXejS04ZFnP79LF5bL/3ktD1sfk5jM3PAeDhdyJP2vHe7eeSlZHGmMHZh3wtn9/yn39tDeKxQ7JZsrk84j5fOHEIb28oI9PjotHr5z/+upx/O62Avy8vafd5t5TVYa3FGNPukr7r9laHRtIvOWvlQGDWkStsRbRGr58+6W7qnW8YWelpXPDAEtJchje/PyPmc58z512gdcppsOWSkRb4tlPT6KW6wcvIQbFbW+3x+izpaT2zWpvOUytHg0buCSi4RHFOZhoFAwOhNWxAJmcfF1jVcrRzwFW08CNLB2WlMyoq8CYfMxCA88e1zt0/VLAHPfvxLirqmjjtnrdj3v7PVXvZX9M2+GujWj91zvVgS6hvuputZXVsdEbhDyzY3GYlzl1Rfeng8QHBg7m+//wqpt+/iNtfXNOhkXB7Z5xqbvHz3Me7OnQAV0fpWAA5GjRyTzALfjCDITmZbNpfS8HA1hNVe9wunrn5zND1wtnzyMlM48czx1Fz0Muc+ZsinmdITgavffdzNLf4Ofu+d2hq8ZPpjHZPLxxIcUX9EZ9K8PH3tnPny2s7/F6qG7y8uba1jfTjF1dz+uhB/M75dhJ+wNSuygYeWriFhxZuCc2d3xs2nTT4BSD4gRGcyhk8+OuFot3UNR/5ipbthfuflxYzZ/4mPG4XV51WcMTP1xGJvv6OpAaFe4IJtmemxphdE+6ft51Nfr8MhvUPfABEh/u0Y3PplxnYCfz+7PNo9PoYkpNJmtvw9amjuOHs0VTVN3PqLxa0ee4rTh3BF07M57ZnA22e4vL6I6r9hrMKeSps5+oP/rYq4sjbRZvKWbSptVVU09gaxs9+vCviufx+y1lhK3D6Lfxr9V5qG4Oj/8DIPdPjotr5DAgurRCtqr6ZPulu0sJaRM3thHuVM/rvzrNKqS0jR4PaMklq0sgBoWAHOHNM4MNgwQ9m8PA1p4baOQCDszMoGNiX9DQX108rDE3jHJiVzozj8xg7JLKff9KwfjH3F9w+cxw77ruk3bbQsXmR25cf4ghXgOfCAv2RxdtClz/e/hmrS9ruCP6OM3cfoMHrw1pLVX3rtMfmsBFxcIeptZZTf7GAW54uijjidlNpLV/47eI2yzgE/23iPqjrEBJh5P7Gp/vaTI+V1KKRe4r407WnsbG0NmLn7JF4+ptTgUCbp4/Hza++Mp5LJ44AAuvqLFjfujTwaU7P/tVvn019k4/X1uwlJzONuR/sZP2+mojQumX6aPL7ZXLWsYO5+OGlhJtY0J/VUUfPhjvUmjZBB5tbqGtqodnn5/aZ41i1uypiWeamFj+ZHndon8LSLRUsDZueed8bG9laVsdb60u59ozWI4yDa+6017Y5Ej6/5dvPruCbnxvN6YVtv4F1tOfu91v2Vh+M+MCO138+swKP27Dllxd32XNKYlG4p4iBWelMOza3049f8qNzycpwk5vdutTCn6+fEpops7+mkfx+mQDkZHrIyfRw8/QxAEweNZAv//49zjk+j3vmbQDgzktOCj3P0h+fy/SwNdRPLxzEL6+YwL1vbKBoRxWDszM6fIarhmZfaPXLITkZjBrUl/nrWj+I/vrRTtbvrYmYvRNu3d7A/oadlQ3sqz5I/z4eKmqbef6T3UDs0fWLy0soGNiHM8fksqm0ls/qm5l2bC47K+s5Jrf1W0tFXRNvrC3lo+JKVv70i0Dg4KqDzT6G9Mts05ZZuauKSSMHtDkbV/VBLws37Gensz/imZvP4JjcvjFD3lrLOxvLyO+XGbHwXSzBg928zvTV2f9YQ0Ozj4evOfWQj5PkonAXAEa1c5RsMHCCwR7L2PwcNv7iIgDmfnMq2VHrzY8c1JcNP5/Jf/x1OYs3l1PX1ML4Ef15+ptn4PX5ueXponbD/cazCzluSDZLN1dwySnDQq2ZB9/eEjowqnBwX8qjeuTBD5nDeWxJMY8tKW6z/UBD68ybhuYWinZU8d9/Xw0Evrlc+vv3Afj6GaN4dtkuzh83hOKKer50yjCGD+jT5vku/f37bK+oZ8d9l4Q+OIyBdzbu55tPFXHvVyZwzdTIBeDueGkNr39aSrpzCsRr/xI4diC4wzncu5vKuWluUai+UwoGsKm0FpehzTe5AwcjZxUFP9COVrgH/20zPe4OHcAnHaOeu3Spc47PC7VvwvVJd/Obr07khPyc0Ijf7TJketx8ZfKINvc/paA//++SE7nryydz7RnH8Mh1p3HC0EBIjRrUN+Lct2MGZ7dZnz5c+PIO/TLTuGzS8MO+j8Wbyrn6sQ956v3tnPTT+Vz/xMeh2/79/5aHLgdX2Vy4sYztFfX87p2toTX+qxq8bNlfS6PXx/aKwE7p5Ts/i3h88NvHC5/s5h9OC6naWT9oR0VgGmhz1LeIRq+Pxc4xDH9ZWsyGfTWsCWtzXfr799lf08iFDy7hggfanlEruD7RoTz/8S7O+827EQd7PbNsJ39YtPUQjzq82kYvk36+gEk/X8C5//tuaHuj1xc69eSR2FZexy9eWx/3mkipTCN3OWoGZ2cw/wdtD1a64tQCLps4guKKeoYPyKS8timizRF0fH4O788+j+H9A98i/l5UwrubyxiYlc6Zo3OZPnYwLT7Lh8WVPD5rCvM+3cdLK/bw9TNG8avXN9IvM401P7sQgDsvPpG3N5RFrKoZbm91I3urG2OeuWlfdeylD2KJDtfwk643NPvYeyDwXKt2H2DV7gP89JW11Df7+Mv1U9oNu5+/tp5nl+3iZ18+iXvmbeDk4f04Pmp0/rNX14Uuf/e5lcwcP5QZx+fx2uq97R7wdcE7dbMAAApvSURBVP+bG/nT4m28d/t5zHY+oC5++D0mjezPLdPHhKbD3vS50W1G3BtLaxiUlc6QnEy2V9RT73w7ixY+q2tfdSMVdU0Mzs7gV69v4OkPd/LP285m0sjY5zVeu6eakqqDzBw/lJvnFrG9op6Lxg/lmNwsXl5ZwuRRA5kSYx9Hb2W64zBsY8xM4CHADfzFWnvfoe4/ZcoUW1RU1OV1SO/j91s2l9Uybmg/tuyv5e/LS7hl+hha/H68LbZN++mZZTu58+W1fPGkfCrrm7l95jj+uWpPaEQedNu5x9Iv08OAvh5u/0fsD4QHvzaJ74edQzfReNwGr89y9nG5oeUmin91MWN+8nrE/a6ZOipiJhPAlZML+MeKwDeLWdOOoaHZx87KBo4dks13zjsuYtpq0MUThnLWsYOZt2Yf3z7vOPqmu7nijx9E3Gf04Cz+eO1k/u2RD0MHub13+7kM79+HdXtrGJyTzrLiz3h55Z7Qt5URA/pEtPGG9c8MfeDe95UJlNc2MevsQvplemhu8bN4czkTR/ZnSE5ra7HF56e0pjG0/6LRG5xaG/mhtXZPNcbAycPbflA1NLdQVtNEYdTssa1ltby/tZLrpx3TZj9KVzPGLLfWTol5W1eHuzHGDWwGLgBKgE+Aa6y169t7jMJdepLX529z3tqy2kYG9k3n4YVbuHJyQeh/4OqD3tDCZLfOGMOFJ+fz1rr9PLqkmI2/mEm624XfWtburWHJ5nKaWnxsK6vnvBOHsKfqICMG9mH88P7MevLj0DTMITkZlNU2cf64IXz9jFHc68zkCXK7TMz2Q3vbO6IrnqM7DOzrCZ17oCt43IbvnT+W9DQXtY0t/HPVHnZ/dpDpYwdz64wxXOesXVQwsA9nHZtLmttFutsVOm7jiyfl8+6mckbl9qW20UthblZodtatM8YwJCeDc47P49ElxbzotNdunzmOdXuryUhzc+KwHMYN7ceGfTW8tb6UiQUDOOeEPE4a1i9iEkNHHe1wnwb8zFp7oXP9DgBr7b3tPUbhLsnEWkv1QS8D+qaHrltLxDo6R6K0upHX1uxl1lmFER8utY1e0lwu+qS7afT6qDnoZdP+Wh5bUsxHxZV857yx3OLst8hIc7GvppHh/TMpqTqIy2X4rK6Zr/zpfW763Bhys9LJ8LgY1r8PK3ZVsb+6kZdW7uGGswpZvLk8tC/gRxeeQGVdM8fk9sXr8/Pg21uoa2phwQ9msLWsju8+v5Ibzirk8lNH8K1nVrCzsoH7rzyFbRV1PLq47Q7pE/JzOGFoDg3NLZTXNsWc+jpuaA4bS2tJT3PR3OLn+Pxspo/N4/H3tjNuaA6ZHjel1Y1keFxMOWYQd1w8jsq6Zi58cAn9+3h4/XvT+fycRaFZP/37ePjCifls2l/D2j1H3r/viFGD+rZZEiOWAX09R7RvA+DuS09m1iGWCz+Uox3uVwEzrbU3O9evA86w1n476n63ArcCjBo16rSdO7vuZBcicmRqGr1s2V/HxIL+EecoiHaw2Rdx0pcWnz90f2ttaNnpLfvrcBnD1NGDcId92DW1+PD7wWctm0prGZSVHnEw3MpdVeRmZTCkXwZrSqo5vXBguy2NpVvKGT+8PwOdnehV9c1kZ6ZFfEAGp/DWNHp5+oMdZGWkhY5QnlAwgHFDc9h74CB+a/l4exXpaS7cLhg9OJu+6W7W763B6/OT5jZs2FfL8P6ZnHPCEEYPzmJjaQ07KxsYlJVO/z4ePiqu5MRh/dhWVofXb2nx+Zk5fii/eWszuVnpDOibzmf1TRw3JJuMNDcWSx9PGm6XYXtFHdPH5nHisH6d+OslaLiH08hdRKTjDhXu3TEVcg8wMux6gbNNRESOku4I90+AscaY0caYdOBq4NVueB0REWlHl89zt9a2GGO+DcwnMBXyCWvtusM8TEREulC3HMRkrX0deP2wdxQRkW6h5QdERFKQwl1EJAUp3EVEUpDCXUQkBXXLwmEdLsKYcqCzh6gOBioOe6+eo/ril+g1qr74qL7OO8ZamxfrhoQI93gYY4raO0IrEai++CV6jaovPqqve6gtIyKSghTuIiIpKBXC/bGeLuAwVF/8Er1G1Rcf1dcNkr7nLiIibaXCyF1ERKIo3EVEUlBSh7sxZqYxZpMxZqsxZnYP1fCEMabMGLM2bNsgY8wCY8wW5/dAZ7sxxjzs1LvGGDP5KNQ30hizyBiz3hizzhjzvUSq0RiTaYz52Biz2qnvbmf7aGPMMqeOF5zlozHGZDjXtzq3F3ZnfWF1uo0xK40xryVafcaYHcaYT40xq4wxRc62hPj7Oq85wBjzojFmozFmgzFmWqLUZ4w5wfl3C/7UGGO+nyj1xSVw/sfk+yGwnPA2YAyQDqwGTuqBOmYAk4G1YdvuB2Y7l2cDv3YuXwy8ARjgTGDZUahvGDDZuZxD4OTlJyVKjc7rZDuXPcAy53X/BlztbH8E+E/n8reAR5zLVwMvHKW/8w+BZ4HXnOsJUx+wAxgctS0h/r7Oa84FbnYupwMDEqm+sDrdQClwTCLW1+H309MFxPGHmAbMD7t+B3BHD9VSGBXum4BhzuVhwCbn8qPANbHudxRrfQW4IBFrBPoCK4AzCBwRmBb9tyZwnoBpzuU0536mm+sqABYC5wGvOf9jJ1J9scI9If6+QH9ge/S/QaLUF1XTF4H3E7W+jv4kc1tmBLA77HqJsy0R5Ftr9zmXS4F853KP1uy0CE4lMDpOmBqdlscqoAxYQOAb2QFrbUuMGkL1ObdXA7ndWR/wIPBjwO9cz02w+izwljFmuQmceB4S5+87GigHnnTaWn8xxmQlUH3hrgaecy4nYn0dkszhnhRs4OO9x+ebGmOygX8A37fW1oTf1tM1Wmt91tpJBEbIU4FxPVVLNGPMl4Aya+3ynq7lED5nrZ0MXATcZoyZEX5jD/990wi0Lf9krT0VqCfQ5gjp6f/+AJx9JpcCf4++LRHq64xkDvdEPhH3fmPMMADnd5mzvUdqNsZ4CAT7M9balxKxRgBr7QFgEYE2xwBjTPBMYeE1hOpzbu8PVHZjWWcDlxpjdgDPE2jNPJRA9WGt3eP8LgNeJvABmSh/3xKgxFq7zLn+IoGwT5T6gi4CVlhr9zvXE62+DkvmcE/kE3G/CsxyLs8i0OcObr/e2eN+JlAd9tWvWxhjDPA4sMFa+9tEq9EYk2eMGeBc7kNgf8AGAiF/VTv1Beu+CnjHGVl1C2vtHdbaAmttIYH/xt6x1l6bKPUZY7KMMTnBywT6xmtJkL+vtbYU2G2MOcHZdD6wPlHqC3MNrS2ZYB2JVF/H9XTTP54fAnuuNxPo0d7ZQzU8B+wDvARGKTcR6LEuBLYAbwODnPsa4A9OvZ8CU45CfZ8j8JVyDbDK+bk4UWoETgFWOvWtBX7qbB8DfAxsJfBVOcPZnulc3+rcPuYo/q0/T+tsmYSoz6ljtfOzLvj/QaL8fZ3XnAQUOX/jfwIDE6y+LALfrvqHbUuY+jr7o+UHRERSUDK3ZUREpB0KdxGRFKRwFxFJQQp3EZEUpHAXEUlBCncRkRSkcBcRSUH/H1s5w+TwNsj7AAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H1z1gzUEXuji",
        "colab_type": "text"
      },
      "source": [
        "To evaluate the model it can be switched to `eval` state."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "A76w5t1EXuji",
        "colab_type": "code",
        "outputId": "88c06a8b-d2f8-48e3-a2eb-6ae3339bdcba",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 168
        }
      },
      "source": [
        "simple_model.eval()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Sequential(\n",
              "  (emb): Embedding(34158, 64)\n",
              "  (reorder): Reorder()\n",
              "  (conv1): Conv1d(64, 128, kernel_size=(2,), stride=(1,))\n",
              "  (relu): ReLU(inplace=True)\n",
              "  (conv2): Conv1d(128, 128, kernel_size=(3,), stride=(1,))\n",
              "  (relu2): ReLU(inplace=True)\n",
              "  (conv3): Conv1d(128, 128, kernel_size=(4,), stride=(1,))\n",
              "  (bn1): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (adaptive_pool): AdaptiveAvgPool1d(output_size=2)\n",
              "  (flatten): Flatten()\n",
              "  (out): Linear(in_features=256, out_features=1, bias=True)\n",
              ")"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 53
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KnfWR0scXujs",
        "colab_type": "text"
      },
      "source": [
        "Let's check the model quality."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UUMqpgLSXujt",
        "colab_type": "code",
        "outputId": "0c3b2393-fb13-4e56-9dd7-5a64230f9d19",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 87
        }
      },
      "source": [
        "%timeit\n",
        "from tqdm import tqdm, tqdm_notebook\n",
        " \n",
        "def print_metrics(model, data, batch_size=batch_size, name=\"\", **kw):\n",
        "    squared_error = abs_error = num_samples = 0.0\n",
        "    for batch_x, batch_y in iterate_minibatches(data, batch_size=batch_size, shuffle=False, **kw):\n",
        "        batch = torch.tensor(batch_x['FullDescription'], dtype=torch.long).to(device)\n",
        "        batch_pred = simple_model(batch)[:, 0].detach().cpu().numpy()\n",
        "        squared_error += np.sum(np.square(batch_pred - batch_y))\n",
        "        abs_error += np.sum(np.abs(batch_pred - batch_y))\n",
        "        num_samples += len(batch_y)\n",
        "    print(\"%s results:\" % (name or \"\"))\n",
        "    print(\"Mean square error: %.5f\" % (squared_error / num_samples))\n",
        "    print(\"Mean absolute error: %.5f\" % (abs_error / num_samples))\n",
        "    return squared_error, abs_error\n",
        "   \n",
        "print_metrics(simple_model, data_train, name='Train')\n",
        "print_metrics(simple_model, data_val, name='Val');"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train results:\n",
            "Mean square error: 1.21098\n",
            "Mean absolute error: 0.98707\n",
            "Val results:\n",
            "Mean square error: 1.23193\n",
            "Mean absolute error: 0.99081\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yANvsvKFXuj1",
        "colab_type": "text"
      },
      "source": [
        "### Bonus area: three-headed network.\n",
        "\n",
        "Now you can try to implement the network we've discussed above. Use [__PyTorch nn.Module API__](https://pytorch.org/docs/stable/nn.html#torch.nn.Module)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_yWR2QD0Xuj4",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class ThreeInputsNet(nn.Module):\n",
        "    def __init__(self, n_tokens=len(tokens), n_cat_features=len(categorical_vectorizer.vocabulary_), hid_size=64):\n",
        "        super(TwoInputsNet, self).__init__()\n",
        "        self.title_emb = nn.Embedding(n_tokens, embedding_dim=hid_size)\n",
        "        # <YOUR CODE HERE>        \n",
        "        \n",
        "        self.full_emb = nn.Embedding(num_embeddings=n_tokens, embedding_dim=hid_size)\n",
        "        # <YOUR CODE HERE>\n",
        "        \n",
        "        self.category_out = # <YOUR CODE HERE>\n",
        "        \n",
        "\n",
        "    def forward(self, whole_input):\n",
        "        input1, input2, input3 = whole_input\n",
        "        title_beg = self.title_emb(input1).permute((0, 2, 1))\n",
        "        title = # <YOUR CODE HERE>\n",
        "        \n",
        "        full_beg = self.full_emb(input2).permute((0, 2, 1))\n",
        "        full = # <YOUR CODE HERE>        \n",
        "        \n",
        "        category = # <YOUR CODE HERE>        \n",
        "        \n",
        "        concatenated = torch.cat(\n",
        "            [\n",
        "            title.view(title.size(0), -1),\n",
        "            full.view(full.size(0), -1),\n",
        "            category.view(category.size(0), -1)\n",
        "            ],\n",
        "            dim=1)\n",
        "        \n",
        "        out = # <YOUR CODE HERE>\n",
        "        \n",
        "        return out"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PiwHqS_uXukB",
        "colab_type": "text"
      },
      "source": [
        "### Bonus area 2: comparing RNN to CNN\n",
        "Try implementing simple RNN (or LSTM) and applying it to this task. Compare the quality/performance of these networks. \n",
        "*Hint: try to build networks with ~same number of paremeters.*"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7ReWcDLiXukG",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# <YOUR CODE HERE>"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9LuO7LeDXukQ",
        "colab_type": "text"
      },
      "source": [
        "### Bonus area 3: fixing the data leaks\n",
        "Fix the data leak we ignored in the beginning of the __Deep Learning part__. Compare results with and without data leaks using same architectures and training time.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ubi1o5kQXukR",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# <YOUR CODE HERE>"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n2AFKsPGXukZ",
        "colab_type": "text"
      },
      "source": [
        "__Terrible start-up idea #1962:__ make a tool that automaticaly rephrases your job description (or CV) to meet salary expectations :)"
      ]
    }
  ]
}