IMDB.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# IMBA next purchase prediction\n",
    "\n",
    "## Using XGBoost in SageMaker\n",
    "\n",
    "---\n",
    "\n",
    "In this example of using Amazon's SageMaker service we will construct a random tree model to productionaze the predictive model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: Preprocess the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# read the data into dataframe\n",
    "import pandas as pd\n",
    "\n",
    "bucket='imba'\n",
    "\n",
    "# we load a smaller version of the full dataset, you can increase the instance size to load the full dataset instead\n",
    "data_key = 'output_small/data_small.csv'\n",
    "data_location = 's3://{}/{}'.format(bucket, data_key)\n",
    "\n",
    "data = pd.read_csv(data_location)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load order product train dataframe\n",
    "order_product_train_key = 'data/order_products/order_products__train.csv.gz'\n",
    "order_product_train_location = 's3://{}/{}'.format(bucket, order_product_train_key)\n",
    "\n",
    "order_product_train = pd.read_csv(order_product_train_location)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>order_id</th>\n",
       "      <th>product_id</th>\n",
       "      <th>add_to_cart_order</th>\n",
       "      <th>reordered</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>49302</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>11109</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>10246</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>49683</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>43633</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   order_id  product_id  add_to_cart_order  reordered\n",
       "0         1       49302                  1          1\n",
       "1         1       11109                  2          1\n",
       "2         1       10246                  3          0\n",
       "3         1       49683                  4          0\n",
       "4         1       43633                  5          1"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# have a look at the data\n",
    "order_product_train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load orders dataframe\n",
    "orders_key = 'data/orders/orders.csv'\n",
    "orders_location = 's3://{}/{}'.format(bucket, orders_key)\n",
    "\n",
    "orders = pd.read_csv(orders_location)\n",
    "# only select train and test orders\n",
    "#orders = orders[orders.eval_set != 'prior'][['user_id', 'order_id','eval_set']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# attach user_id to order_product_train\n",
    "order_product_train = order_product_train.merge(orders[['user_id', 'order_id']])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# attach eval_set to data\n",
    "#data = data.merge(orders[orders.eval_set != 'prior'][['user_id', 'order_id','eval_set']])\n",
    "data = data.merge(orders[orders.eval_set != 'prior'][['user_id','eval_set']])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# attach target variable: reordered\n",
    "data = data.merge(order_product_train[['user_id', 'product_id', 'reordered']], how = 'left')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# a few more feature engineering, refer to the R code\n",
    "data['prod_reorder_probability'] = data.prod_second_orders / data.prod_first_orders\n",
    "data['prod_reorder_times'] = 1 + data.prod_reorders / data.prod_first_orders\n",
    "data['prod_reorder_ratio'] = data.prod_reorders / data.prod_orders\n",
    "data.drop(['prod_reorders', 'prod_first_orders', 'prod_second_orders'], axis=1, inplace=True)\n",
    "data['user_average_basket'] = data.user_total_products / data.user_orders\n",
    "data['up_order_rate'] = data.up_orders / data.user_orders\n",
    "data['up_orders_since_last_order'] = data.user_orders - data.up_last_order\n",
    "data['up_order_rate_since_first_order'] = data.up_orders / (data.user_orders - data.up_first_order + 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>product_id</th>\n",
       "      <th>up_orders</th>\n",
       "      <th>user_mean_days_since_prior</th>\n",
       "      <th>user_period</th>\n",
       "      <th>user_distinct_products</th>\n",
       "      <th>user_reorder_ratio</th>\n",
       "      <th>user_total_products</th>\n",
       "      <th>up_average_cart_position</th>\n",
       "      <th>up_first_order</th>\n",
       "      <th>user_orders</th>\n",
       "      <th>...</th>\n",
       "      <th>user_id</th>\n",
       "      <th>eval_set</th>\n",
       "      <th>reordered</th>\n",
       "      <th>prod_reorder_probability</th>\n",
       "      <th>prod_reorder_times</th>\n",
       "      <th>prod_reorder_ratio</th>\n",
       "      <th>user_average_basket</th>\n",
       "      <th>up_order_rate</th>\n",
       "      <th>up_orders_since_last_order</th>\n",
       "      <th>up_order_rate_since_first_order</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>19508</td>\n",
       "      <td>6</td>\n",
       "      <td>5.969574</td>\n",
       "      <td>2943</td>\n",
       "      <td>200</td>\n",
       "      <td>0.602434</td>\n",
       "      <td>497</td>\n",
       "      <td>7.833333</td>\n",
       "      <td>11</td>\n",
       "      <td>61</td>\n",
       "      <td>...</td>\n",
       "      <td>144185</td>\n",
       "      <td>test</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.348857</td>\n",
       "      <td>1.812378</td>\n",
       "      <td>0.448239</td>\n",
       "      <td>8.147541</td>\n",
       "      <td>0.098361</td>\n",
       "      <td>32</td>\n",
       "      <td>0.117647</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>42307</td>\n",
       "      <td>1</td>\n",
       "      <td>5.969574</td>\n",
       "      <td>2943</td>\n",
       "      <td>200</td>\n",
       "      <td>0.602434</td>\n",
       "      <td>497</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>55</td>\n",
       "      <td>61</td>\n",
       "      <td>...</td>\n",
       "      <td>144185</td>\n",
       "      <td>test</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.546166</td>\n",
       "      <td>3.424100</td>\n",
       "      <td>0.707952</td>\n",
       "      <td>8.147541</td>\n",
       "      <td>0.016393</td>\n",
       "      <td>6</td>\n",
       "      <td>0.142857</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>35883</td>\n",
       "      <td>1</td>\n",
       "      <td>5.969574</td>\n",
       "      <td>2943</td>\n",
       "      <td>200</td>\n",
       "      <td>0.602434</td>\n",
       "      <td>497</td>\n",
       "      <td>8.000000</td>\n",
       "      <td>52</td>\n",
       "      <td>61</td>\n",
       "      <td>...</td>\n",
       "      <td>144185</td>\n",
       "      <td>test</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.357664</td>\n",
       "      <td>2.010219</td>\n",
       "      <td>0.502542</td>\n",
       "      <td>8.147541</td>\n",
       "      <td>0.016393</td>\n",
       "      <td>9</td>\n",
       "      <td>0.100000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>13539</td>\n",
       "      <td>1</td>\n",
       "      <td>5.969574</td>\n",
       "      <td>2943</td>\n",
       "      <td>200</td>\n",
       "      <td>0.602434</td>\n",
       "      <td>497</td>\n",
       "      <td>12.000000</td>\n",
       "      <td>50</td>\n",
       "      <td>61</td>\n",
       "      <td>...</td>\n",
       "      <td>144185</td>\n",
       "      <td>test</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.176471</td>\n",
       "      <td>1.270588</td>\n",
       "      <td>0.212963</td>\n",
       "      <td>8.147541</td>\n",
       "      <td>0.016393</td>\n",
       "      <td>11</td>\n",
       "      <td>0.083333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>27966</td>\n",
       "      <td>3</td>\n",
       "      <td>5.969574</td>\n",
       "      <td>2943</td>\n",
       "      <td>200</td>\n",
       "      <td>0.602434</td>\n",
       "      <td>497</td>\n",
       "      <td>7.000000</td>\n",
       "      <td>53</td>\n",
       "      <td>61</td>\n",
       "      <td>...</td>\n",
       "      <td>144185</td>\n",
       "      <td>test</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.611129</td>\n",
       "      <td>4.330669</td>\n",
       "      <td>0.769089</td>\n",
       "      <td>8.147541</td>\n",
       "      <td>0.049180</td>\n",
       "      <td>0</td>\n",
       "      <td>0.333333</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 22 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   product_id  up_orders  user_mean_days_since_prior  user_period  \\\n",
       "0       19508          6                    5.969574         2943   \n",
       "1       42307          1                    5.969574         2943   \n",
       "2       35883          1                    5.969574         2943   \n",
       "3       13539          1                    5.969574         2943   \n",
       "4       27966          3                    5.969574         2943   \n",
       "\n",
       "   user_distinct_products  user_reorder_ratio  user_total_products  \\\n",
       "0                     200            0.602434                  497   \n",
       "1                     200            0.602434                  497   \n",
       "2                     200            0.602434                  497   \n",
       "3                     200            0.602434                  497   \n",
       "4                     200            0.602434                  497   \n",
       "\n",
       "   up_average_cart_position  up_first_order  user_orders  ...  user_id  \\\n",
       "0                  7.833333              11           61  ...   144185   \n",
       "1                  3.000000              55           61  ...   144185   \n",
       "2                  8.000000              52           61  ...   144185   \n",
       "3                 12.000000              50           61  ...   144185   \n",
       "4                  7.000000              53           61  ...   144185   \n",
       "\n",
       "   eval_set  reordered prod_reorder_probability  prod_reorder_times  \\\n",
       "0      test        NaN                 0.348857            1.812378   \n",
       "1      test        NaN                 0.546166            3.424100   \n",
       "2      test        NaN                 0.357664            2.010219   \n",
       "3      test        NaN                 0.176471            1.270588   \n",
       "4      test        NaN                 0.611129            4.330669   \n",
       "\n",
       "   prod_reorder_ratio  user_average_basket  up_order_rate  \\\n",
       "0            0.448239             8.147541       0.098361   \n",
       "1            0.707952             8.147541       0.016393   \n",
       "2            0.502542             8.147541       0.016393   \n",
       "3            0.212963             8.147541       0.016393   \n",
       "4            0.769089             8.147541       0.049180   \n",
       "\n",
       "   up_orders_since_last_order  up_order_rate_since_first_order  \n",
       "0                          32                         0.117647  \n",
       "1                           6                         0.142857  \n",
       "2                           9                         0.100000  \n",
       "3                          11                         0.083333  \n",
       "4                           0                         0.333333  \n",
       "\n",
       "[5 rows x 22 columns]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# split into training and test set, test set does not have target variable\n",
    "train = data[data.eval_set == 'train'].copy()\n",
    "test = data[data.eval_set == 'test'].copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# id field won't be used in model, thus make a backup of them and remove from dataframe\n",
    "#test_id = test[['product_id','user_id', 'order_id', 'eval_set']]\n",
    "test_id = test[['product_id','user_id', 'eval_set']]\n",
    "#test.drop(['product_id','user_id', 'order_id', 'eval_set', 'reordered'], axis=1, inplace=True)\n",
    "test.drop(['product_id','user_id', 'eval_set', 'reordered'], axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# convert target variable to 1/0 for training dataframe\n",
    "train['reordered'] = train['reordered'].fillna(0)\n",
    "train['reordered'] = train.reordered.astype(int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# drop id columns as they won't be used in model\n",
    "#train.drop(['eval_set', 'user_id', 'product_id', 'order_id'], axis=1, inplace=True)\n",
    "train.drop(['eval_set', 'user_id', 'product_id'], axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# this is the target variable dataframe\n",
    "train_y = train[['reordered']]\n",
    "# this is the dataframe without target variable\n",
    "train_X = train.drop(['reordered'], axis = 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Classification\n",
    "\n",
    "Now that we have created the feature representation of our training (and testing) data, it is time to start setting up and using the XGBoost classifier provided by SageMaker.\n",
    "\n",
    "### Writing the dataset\n",
    "\n",
    "The XGBoost classifier that we will be using requires the dataset to be written to a file and stored using Amazon S3. To do this, we will start by splitting the training dataset into two parts, the data we will train the model with and a validation set. Then, we will write those datasets to a file and upload the files to S3."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we split the data into training and validation set. Training data is for training the model, validation data is for evaluating the model performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "val_X = train_X[:20000]\n",
    "train_X = train_X[20000:]\n",
    "\n",
    "val_y = train_y[:20000]\n",
    "train_y = train_y[20000:]\n",
    "\n",
    "#test_y = pd.DataFrame(test_y)\n",
    "#test_X = pd.DataFrame(test_X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For more information about this and other algorithms, the SageMaker developer documentation can be found on __[Amazon's website.](https://docs.aws.amazon.com/sagemaker/latest/dg/)__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First we make sure that the local directory in which we'd like to store the training and validation csv files exists.\n",
    "import os\n",
    "data_dir = 'data/xgboost'\n",
    "if not os.path.exists(data_dir):\n",
    "    os.makedirs(data_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First, save the test data to test.csv in the data_dir directory without label.\n",
    "pd.DataFrame(test).to_csv(os.path.join(data_dir, 'test.csv'), header=False, index=False)\n",
    "\n",
    "# Then we save the training and validation set into local disk as csv files\n",
    "pd.concat([val_y, val_X], axis=1).to_csv(os.path.join(data_dir, 'validation.csv'), header=False, index=False)\n",
    "pd.concat([train_y, train_X], axis=1).to_csv(os.path.join(data_dir, 'train.csv'), header=False, index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# To save a bit of memory we can set text_X, train_X, val_X, train_y and val_y to None.\n",
    "\n",
    "train_X = val_X = train_y = val_y = None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Uploading Training / Validation files to S3\n",
    "\n",
    "Amazon's S3 service allows us to store files that can be access by both the built-in training models such as the XGBoost model we will be using as well as custom models such as the one we will see a little later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "\n",
    "session = sagemaker.Session() # Store the current SageMaker session\n",
    "\n",
    "# S3 prefix (which folder will we use)\n",
    "prefix = 'imba-xgboost'\n",
    "\n",
    "test_location = session.upload_data(os.path.join(data_dir, 'test.csv'), key_prefix=prefix)\n",
    "val_location = session.upload_data(os.path.join(data_dir, 'validation.csv'), key_prefix=prefix)\n",
    "train_location = session.upload_data(os.path.join(data_dir, 'train.csv'), key_prefix=prefix)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating a tuned XGBoost model\n",
    "\n",
    "Now that the data has been uploaded it is time to create the XGBoost model. The first step is to create an estimator object which will be used as the *base* of your hyperparameter tuning job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker import get_execution_role\n",
    "\n",
    "# Our current execution role is require when creating the model as the training\n",
    "# and inference code will need to access the model artifacts.\n",
    "role = get_execution_role()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "'get_image_uri' method will be deprecated in favor of 'ImageURIProvider' class in SageMaker Python SDK v2.\n",
      "WARNING:root:There is a more up to date SageMaker XGBoost image. To use the newer image, please set 'repo_version'='1.0-1'. For example:\n",
      "\tget_image_uri(region, 'xgboost', '1.0-1').\n"
     ]
    }
   ],
   "source": [
    "# We need to retrieve the location of the container which is provided by Amazon for using XGBoost.\n",
    "# As a matter of convenience, the training and inference code both use the same container.\n",
    "from sagemaker.amazon.amazon_estimator import get_image_uri\n",
    "\n",
    "container = get_image_uri(session.boto_region_name, 'xgboost', '0.90-1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Parameter image_name will be renamed to image_uri in SageMaker Python SDK v2.\n"
     ]
    }
   ],
   "source": [
    "#       Create a SageMaker estimator using the container location determined in the previous cell.\n",
    "#       It is recommended that you use a single training instance of type ml.m4.xlarge. It is also\n",
    "#       recommended that you use 's3://{}/{}/output'.format(session.default_bucket(), prefix) as the\n",
    "#       output path.\n",
    "\n",
    "xgb = sagemaker.estimator.Estimator(container, # The location of the container we wish to use\n",
    "                                    role,                                    # What is our current IAM Role\n",
    "                                    train_instance_count=1,                  # How many compute instances\n",
    "                                    train_instance_type='ml.m4.xlarge',      # What kind of compute instances\n",
    "                                    output_path='s3://{}/{}/output'.format(session.default_bucket(), prefix),\n",
    "                                    sagemaker_session=session)\n",
    "\n",
    "#       Set the XGBoost hyperparameters in the xgb object. Don't forget that in this case we have a binary\n",
    "#       label so we should be using the 'binary:logistic' objective.\n",
    "\n",
    "# Solution:\n",
    "xgb.set_hyperparameters(max_depth=5,\n",
    "                        eta=0.1,\n",
    "                        gamma=4,\n",
    "                        min_child_weight=6,\n",
    "                        subsample=0.8,\n",
    "                        silent=0,\n",
    "                        objective='binary:logistic',\n",
    "                        early_stopping_rounds=10,\n",
    "                        num_round=500)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the hyperparameter tuner\n",
    "\n",
    "Now that the base estimator has been set up we need to construct a hyperparameter tuner object which we will use to request SageMaker construct a hyperparameter tuning job.\n",
    "\n",
    "**Note:** If you don't want the hyperparameter tuning job to take too long, make sure to not set the total number of models (jobs) too high."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First, make sure to import the relevant objects used to construct the tuner\n",
    "from sagemaker.tuner import IntegerParameter, ContinuousParameter, HyperparameterTuner\n",
    "\n",
    "\n",
    "# create the tuner object:\n",
    "\n",
    "xgb_hyperparameter_tuner = HyperparameterTuner(estimator = xgb, # The estimator object to use as the basis for the training jobs.\n",
    "                                               objective_metric_name = 'validation:rmse', # The metric used to compare trained models.\n",
    "                                               objective_type = 'Minimize', # Whether we wish to minimize or maximize the metric.\n",
    "                                               max_jobs = 4, # The total number of models to train\n",
    "                                               max_parallel_jobs = 3, # The number of models to train in parallel\n",
    "                                               hyperparameter_ranges = {\n",
    "                                                    'max_depth': IntegerParameter(3, 12),\n",
    "                                                    'eta'      : ContinuousParameter(0.05, 0.5),\n",
    "                                                    'min_child_weight': IntegerParameter(2, 8),\n",
    "                                                    'subsample': ContinuousParameter(0.5, 0.9),\n",
    "                                                    'gamma': ContinuousParameter(0, 10),\n",
    "                                               })"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fit the hyperparameter tuner\n",
    "\n",
    "Now that the hyperparameter tuner object has been constructed, it is time to fit the various models and find the best performing model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:sagemaker:'s3_input' class will be renamed to 'TrainingInput' in SageMaker Python SDK v2.\n",
      "WARNING:sagemaker:'s3_input' class will be renamed to 'TrainingInput' in SageMaker Python SDK v2.\n"
     ]
    }
   ],
   "source": [
    "s3_input_train = sagemaker.s3_input(s3_data=train_location, content_type='csv')\n",
    "s3_input_validation = sagemaker.s3_input(s3_data=val_location, content_type='csv')\n",
    "xgb_hyperparameter_tuner.fit({'train': s3_input_train, 'validation': s3_input_validation})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember that the tuning job is constructed and run in the background so if we want to see the progress of our training job we need to call the `wait()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "................................................................................................!\n"
     ]
    }
   ],
   "source": [
    "xgb_hyperparameter_tuner.wait()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testing the model\n",
    "\n",
    "Now that we've run our hyperparameter tuning job, it's time to see how well the best performing model actually performs. To do this we will use SageMaker's Batch Transform functionality. Batch Transform is a convenient way to perform inference on a large dataset in a way that is not realtime. That is, we don't necessarily need to use our model's results immediately and instead we can peform inference on a large number of samples. An example of this in industry might be peforming an end of month report. This method of inference can also be useful to us as it means to can perform inference on our entire test set. \n",
    "\n",
    "Remember that in order to create a transformer object to perform the batch transform job, we need a trained estimator object. We can do that using the `attach()` method, creating an estimator object which is attached to the best trained job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Parameter image_name will be renamed to image_uri in SageMaker Python SDK v2.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-07-14 13:02:26 Starting - Preparing the instances for training\n",
      "2020-07-14 13:02:26 Downloading - Downloading input data\n",
      "2020-07-14 13:02:26 Training - Training image download completed. Training in progress.\n",
      "2020-07-14 13:02:26 Uploading - Uploading generated training model\n",
      "2020-07-14 13:02:26 Completed - Training job completed\u001b[34mINFO:sagemaker-containers:Imported framework sagemaker_xgboost_container.training\u001b[0m\n",
      "\u001b[34mINFO:sagemaker-containers:Failed to parse hyperparameter _tuning_objective_metric value validation:rmse to Json.\u001b[0m\n",
      "\u001b[34mReturning the value itself\u001b[0m\n",
      "\u001b[34mINFO:sagemaker-containers:Failed to parse hyperparameter objective value binary:logistic to Json.\u001b[0m\n",
      "\u001b[34mReturning the value itself\u001b[0m\n",
      "\u001b[34mINFO:sagemaker-containers:No GPUs detected (normal if no gpus installed)\u001b[0m\n",
      "\u001b[34mINFO:sagemaker_xgboost_container.training:Running XGBoost Sagemaker in algorithm mode\u001b[0m\n",
      "\u001b[34mINFO:root:Determined delimiter of CSV input is ','\u001b[0m\n",
      "\u001b[34mINFO:root:Determined delimiter of CSV input is ','\u001b[0m\n",
      "\u001b[34mINFO:root:Determined delimiter of CSV input is ','\u001b[0m\n",
      "\u001b[34m[13:01:55] 826992x18 matrix with 14885856 entries loaded from /opt/ml/input/data/train?format=csv&label_column=0&delimiter=,\u001b[0m\n",
      "\u001b[34mINFO:root:Determined delimiter of CSV input is ','\u001b[0m\n",
      "\u001b[34m[13:01:55] 20000x18 matrix with 360000 entries loaded from /opt/ml/input/data/validation?format=csv&label_column=0&delimiter=,\u001b[0m\n",
      "\u001b[34mINFO:root:Single node training.\u001b[0m\n",
      "\u001b[34mINFO:root:Setting up HPO optimized metric to be : rmse\u001b[0m\n",
      "\u001b[34mINFO:root:Train matrix has 826992 rows\u001b[0m\n",
      "\u001b[34mINFO:root:Validation matrix has 20000 rows\u001b[0m\n",
      "\u001b[34m[0]#011train-rmse:0.401835#011validation-rmse:0.393083\u001b[0m\n",
      "\u001b[34m[1]#011train-rmse:0.344718#011validation-rmse:0.329465\u001b[0m\n",
      "\u001b[34m[2]#011train-rmse:0.311713#011validation-rmse:0.29133\u001b[0m\n",
      "\u001b[34m[3]#011train-rmse:0.292556#011validation-rmse:0.269264\u001b[0m\n",
      "\u001b[34m[4]#011train-rmse:0.282075#011validation-rmse:0.256851\u001b[0m\n",
      "\u001b[34m[5]#011train-rmse:0.276078#011validation-rmse:0.249945\u001b[0m\n",
      "\u001b[34m[6]#011train-rmse:0.272728#011validation-rmse:0.245936\u001b[0m\n",
      "\u001b[34m[7]#011train-rmse:0.270853#011validation-rmse:0.243575\u001b[0m\n",
      "\u001b[34m[8]#011train-rmse:0.2697#011validation-rmse:0.242284\u001b[0m\n",
      "\u001b[34m[9]#011train-rmse:0.268966#011validation-rmse:0.241327\u001b[0m\n",
      "\u001b[34m[10]#011train-rmse:0.268448#011validation-rmse:0.240733\u001b[0m\n",
      "\u001b[34m[11]#011train-rmse:0.268114#011validation-rmse:0.240383\u001b[0m\n",
      "\u001b[34m[12]#011train-rmse:0.267863#011validation-rmse:0.240127\u001b[0m\n",
      "\u001b[34m[13]#011train-rmse:0.26765#011validation-rmse:0.239859\u001b[0m\n",
      "\u001b[34m[14]#011train-rmse:0.267474#011validation-rmse:0.239694\u001b[0m\n",
      "\u001b[34m[15]#011train-rmse:0.267408#011validation-rmse:0.239621\u001b[0m\n",
      "\u001b[34m[16]#011train-rmse:0.26733#011validation-rmse:0.239501\u001b[0m\n",
      "\u001b[34m[17]#011train-rmse:0.26727#011validation-rmse:0.239471\u001b[0m\n",
      "\u001b[34m[18]#011train-rmse:0.267223#011validation-rmse:0.239398\u001b[0m\n",
      "\u001b[34m[19]#011train-rmse:0.26717#011validation-rmse:0.239363\u001b[0m\n",
      "\u001b[34m[20]#011train-rmse:0.267131#011validation-rmse:0.239317\u001b[0m\n",
      "\u001b[34m[21]#011train-rmse:0.267104#011validation-rmse:0.239286\u001b[0m\n",
      "\u001b[34m[22]#011train-rmse:0.267065#011validation-rmse:0.239232\u001b[0m\n",
      "\u001b[34m[23]#011train-rmse:0.267034#011validation-rmse:0.23916\u001b[0m\n",
      "\u001b[34m[24]#011train-rmse:0.267001#011validation-rmse:0.239152\u001b[0m\n",
      "\u001b[34m[25]#011train-rmse:0.266958#011validation-rmse:0.239129\u001b[0m\n",
      "\u001b[34m[26]#011train-rmse:0.266941#011validation-rmse:0.23913\u001b[0m\n",
      "\u001b[34m[27]#011train-rmse:0.266908#011validation-rmse:0.239105\u001b[0m\n",
      "\u001b[34m[28]#011train-rmse:0.266857#011validation-rmse:0.239028\u001b[0m\n",
      "\u001b[34m[29]#011train-rmse:0.266836#011validation-rmse:0.239033\u001b[0m\n",
      "\u001b[34m[30]#011train-rmse:0.26682#011validation-rmse:0.239014\u001b[0m\n",
      "\u001b[34m[31]#011train-rmse:0.266781#011validation-rmse:0.238956\u001b[0m\n",
      "\u001b[34m[32]#011train-rmse:0.26674#011validation-rmse:0.238908\u001b[0m\n",
      "\u001b[34m[33]#011train-rmse:0.266727#011validation-rmse:0.238918\u001b[0m\n",
      "\u001b[34m[34]#011train-rmse:0.266697#011validation-rmse:0.238892\u001b[0m\n",
      "\u001b[34m[35]#011train-rmse:0.266688#011validation-rmse:0.238928\u001b[0m\n",
      "\u001b[34m[36]#011train-rmse:0.266668#011validation-rmse:0.238932\u001b[0m\n",
      "\u001b[34m[37]#011train-rmse:0.266653#011validation-rmse:0.238933\u001b[0m\n",
      "\u001b[34m[38]#011train-rmse:0.266634#011validation-rmse:0.238914\u001b[0m\n",
      "\u001b[34m[39]#011train-rmse:0.26662#011validation-rmse:0.238908\u001b[0m\n",
      "\u001b[34m[40]#011train-rmse:0.266607#011validation-rmse:0.238903\u001b[0m\n",
      "\u001b[34m[41]#011train-rmse:0.266602#011validation-rmse:0.238905\u001b[0m\n",
      "\u001b[34m[42]#011train-rmse:0.266593#011validation-rmse:0.23892\u001b[0m\n",
      "\u001b[34m[43]#011train-rmse:0.266582#011validation-rmse:0.238944\u001b[0m\n",
      "\u001b[34m[44]#011train-rmse:0.266558#011validation-rmse:0.238899\u001b[0m\n",
      "Training seconds: 83\n",
      "Billable seconds: 83\n"
     ]
    }
   ],
   "source": [
    "# attach the model:\n",
    "\n",
    "xgb_attached = sagemaker.estimator.Estimator.attach(xgb_hyperparameter_tuner.best_training_job())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have an estimator object attached to the correct training job, we can proceed as we normally would and create a transformer object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:sagemaker:Parameter image will be renamed to image_uri in SageMaker Python SDK v2.\n"
     ]
    }
   ],
   "source": [
    "# Create a transformer object from the attached estimator. Using an instance count of 1 and an instance type of ml.m4.xlarge\n",
    "#       should be more than enough.:\n",
    "xgb_transformer = xgb_attached.transformer(instance_count = 1, instance_type = 'ml.m4.xlarge')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we actually perform the transform job. When doing so we need to make sure to specify the type of data we are sending so that it is serialized correctly in the background. In our case we are providing our model with csv data so we specify `text/csv`. Also, if the test data that we have provided is too large to process all at once then we need to specify how the data file should be split up. Since each line is a single entry in our data set we tell SageMaker that it can split the input on each line."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Start the transform job. Make sure to specify the content type and the split type of the test data.\n",
    "xgb_transformer.transform(test_location, content_type='text/csv', split_type='Line')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Currently the transform job is running but it is doing so in the background. Since we wish to wait until the transform job is done and we would like a bit of feedback we can run the `wait()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "........................\u001b[34m[2020-07-14 13:16:40 +0000] [15] [INFO] Starting gunicorn 19.10.0\u001b[0m\n",
      "\u001b[34m[2020-07-14 13:16:40 +0000] [15] [INFO] Listening at: unix:/tmp/gunicorn.sock (15)\u001b[0m\n",
      "\u001b[34m[2020-07-14 13:16:40 +0000] [15] [INFO] Using worker: gevent\u001b[0m\n",
      "\u001b[34m[2020-07-14 13:16:40 +0000] [22] [INFO] Booting worker with pid: 22\u001b[0m\n",
      "\u001b[34m[2020-07-14 13:16:40 +0000] [23] [INFO] Booting worker with pid: 23\u001b[0m\n",
      "\u001b[34m[2020-07-14 13:16:40 +0000] [30] [INFO] Booting worker with pid: 30\u001b[0m\n",
      "\u001b[34m[2020-07-14 13:16:40 +0000] [31] [INFO] Booting worker with pid: 31\u001b[0m\n",
      "\n",
      "\u001b[34m[2020-07-14:13:17:15:INFO] No GPUs detected (normal if no gpus installed)\u001b[0m\n",
      "\u001b[35m[2020-07-14:13:17:15:INFO] No GPUs detected (normal if no gpus installed)\u001b[0m\n",
      "\u001b[34m169.254.255.130 - - [14/Jul/2020:13:17:15 +0000] \"GET /ping HTTP/1.1\" 200 0 \"-\" \"Go-http-client/1.1\"\u001b[0m\n",
      "\u001b[34m[2020-07-14:13:17:15:INFO] No GPUs detected (normal if no gpus installed)\u001b[0m\n",
      "\u001b[34m169.254.255.130 - - [14/Jul/2020:13:17:15 +0000] \"GET /execution-parameters HTTP/1.1\" 200 84 \"-\" \"Go-http-client/1.1\"\u001b[0m\n",
      "\u001b[35m169.254.255.130 - - [14/Jul/2020:13:17:15 +0000] \"GET /ping HTTP/1.1\" 200 0 \"-\" \"Go-http-client/1.1\"\u001b[0m\n",
      "\u001b[35m[2020-07-14:13:17:15:INFO] No GPUs detected (normal if no gpus installed)\u001b[0m\n",
      "\u001b[35m169.254.255.130 - - [14/Jul/2020:13:17:15 +0000] \"GET /execution-parameters HTTP/1.1\" 200 84 \"-\" \"Go-http-client/1.1\"\u001b[0m\n",
      "\u001b[34m[2020-07-14:13:17:16:INFO] No GPUs detected (normal if no gpus installed)\u001b[0m\n",
      "\u001b[34m[2020-07-14:13:17:16:INFO] No GPUs detected (normal if no gpus installed)\u001b[0m\n",
      "\u001b[34m[2020-07-14:13:17:16:INFO] Determined delimiter of CSV input is ','\u001b[0m\n",
      "\u001b[34m[2020-07-14:13:17:16:INFO] Determined delimiter of CSV input is ','\u001b[0m\n",
      "\u001b[34m[2020-07-14:13:17:16:INFO] Determined delimiter of CSV input is ','\u001b[0m\n",
      "\u001b[34m[2020-07-14:13:17:16:INFO] Determined delimiter of CSV input is ','\u001b[0m\n",
      "\u001b[35m[2020-07-14:13:17:16:INFO] No GPUs detected (normal if no gpus installed)\u001b[0m\n",
      "\u001b[35m[2020-07-14:13:17:16:INFO] No GPUs detected (normal if no gpus installed)\u001b[0m\n",
      "\u001b[35m[2020-07-14:13:17:16:INFO] Determined delimiter of CSV input is ','\u001b[0m\n",
      "\u001b[35m[2020-07-14:13:17:16:INFO] Determined delimiter of CSV input is ','\u001b[0m\n",
      "\u001b[35m[2020-07-14:13:17:16:INFO] Determined delimiter of CSV input is ','\u001b[0m\n",
      "\u001b[35m[2020-07-14:13:17:16:INFO] Determined delimiter of CSV input is ','\u001b[0m\n",
      "\u001b[34m169.254.255.130 - - [14/Jul/2020:13:17:17 +0000] \"POST /invocations HTTP/1.1\" 200 733762 \"-\" \"Go-http-client/1.1\"\u001b[0m\n",
      "\u001b[34m169.254.255.130 - - [14/Jul/2020:13:17:17 +0000] \"POST /invocations HTTP/1.1\" 200 736054 \"-\" \"Go-http-client/1.1\"\u001b[0m\n",
      "\u001b[34m169.254.255.130 - - [14/Jul/2020:13:17:17 +0000] \"POST /invocations HTTP/1.1\" 200 736660 \"-\" \"Go-http-client/1.1\"\u001b[0m\n",
      "\u001b[34m169.254.255.130 - - [14/Jul/2020:13:17:18 +0000] \"POST /invocations HTTP/1.1\" 200 738716 \"-\" \"Go-http-client/1.1\"\u001b[0m\n",
      "\u001b[34m[2020-07-14:13:17:18:INFO] Determined delimiter of CSV input is ','\u001b[0m\n",
      "\u001b[34m[2020-07-14:13:17:18:INFO] Determined delimiter of CSV input is ','\u001b[0m\n",
      "\u001b[34m[2020-07-14:13:17:18:INFO] Determined delimiter of CSV input is ','\u001b[0m\n",
      "\u001b[35m169.254.255.130 - - [14/Jul/2020:13:17:17 +0000] \"POST /invocations HTTP/1.1\" 200 733762 \"-\" \"Go-http-client/1.1\"\u001b[0m\n",
      "\u001b[35m169.254.255.130 - - [14/Jul/2020:13:17:17 +0000] \"POST /invocations HTTP/1.1\" 200 736054 \"-\" \"Go-http-client/1.1\"\u001b[0m\n",
      "\u001b[35m169.254.255.130 - - [14/Jul/2020:13:17:17 +0000] \"POST /invocations HTTP/1.1\" 200 736660 \"-\" \"Go-http-client/1.1\"\u001b[0m\n",
      "\u001b[35m169.254.255.130 - - [14/Jul/2020:13:17:18 +0000] \"POST /invocations HTTP/1.1\" 200 738716 \"-\" \"Go-http-client/1.1\"\u001b[0m\n",
      "\u001b[35m[2020-07-14:13:17:18:INFO] Determined delimiter of CSV input is ','\u001b[0m\n",
      "\u001b[35m[2020-07-14:13:17:18:INFO] Determined delimiter of CSV input is ','\u001b[0m\n",
      "\u001b[35m[2020-07-14:13:17:18:INFO] Determined delimiter of CSV input is ','\u001b[0m\n",
      "\u001b[32m2020-07-14T13:17:15.199:[sagemaker logs]: MaxConcurrentTransforms=4, MaxPayloadInMB=6, BatchStrategy=MULTI_RECORD\u001b[0m\n",
      "\u001b[34m[2020-07-14:13:17:18:INFO] Determined delimiter of CSV input is ','\u001b[0m\n",
      "\u001b[35m[2020-07-14:13:17:18:INFO] Determined delimiter of CSV input is ','\u001b[0m\n",
      "\u001b[34m169.254.255.130 - - [14/Jul/2020:13:17:19 +0000] \"POST /invocations HTTP/1.1\" 200 740895 \"-\" \"Go-http-client/1.1\"\u001b[0m\n",
      "\u001b[34m169.254.255.130 - - [14/Jul/2020:13:17:19 +0000] \"POST /invocations HTTP/1.1\" 200 744490 \"-\" \"Go-http-client/1.1\"\u001b[0m\n",
      "\u001b[34m169.254.255.130 - - [14/Jul/2020:13:17:19 +0000] \"POST /invocations HTTP/1.1\" 200 747139 \"-\" \"Go-http-client/1.1\"\u001b[0m\n",
      "\u001b[34m[2020-07-14:13:17:19:INFO] Determined delimiter of CSV input is ','\u001b[0m\n",
      "\u001b[34m169.254.255.130 - - [14/Jul/2020:13:17:19 +0000] \"POST /invocations HTTP/1.1\" 200 750962 \"-\" \"Go-http-client/1.1\"\u001b[0m\n",
      "\u001b[34m[2020-07-14:13:17:19:INFO] Determined delimiter of CSV input is ','\u001b[0m\n",
      "\u001b[34m[2020-07-14:13:17:20:INFO] Determined delimiter of CSV input is ','\u001b[0m\n",
      "\u001b[34m[2020-07-14:13:17:20:INFO] Determined delimiter of CSV input is ','\u001b[0m\n",
      "\u001b[35m169.254.255.130 - - [14/Jul/2020:13:17:19 +0000] \"POST /invocations HTTP/1.1\" 200 740895 \"-\" \"Go-http-client/1.1\"\u001b[0m\n",
      "\u001b[35m169.254.255.130 - - [14/Jul/2020:13:17:19 +0000] \"POST /invocations HTTP/1.1\" 200 744490 \"-\" \"Go-http-client/1.1\"\u001b[0m\n",
      "\u001b[35m169.254.255.130 - - [14/Jul/2020:13:17:19 +0000] \"POST /invocations HTTP/1.1\" 200 747139 \"-\" \"Go-http-client/1.1\"\u001b[0m\n",
      "\u001b[35m[2020-07-14:13:17:19:INFO] Determined delimiter of CSV input is ','\u001b[0m\n",
      "\u001b[35m169.254.255.130 - - [14/Jul/2020:13:17:19 +0000] \"POST /invocations HTTP/1.1\" 200 750962 \"-\" \"Go-http-client/1.1\"\u001b[0m\n",
      "\u001b[35m[2020-07-14:13:17:19:INFO] Determined delimiter of CSV input is ','\u001b[0m\n",
      "\u001b[35m[2020-07-14:13:17:20:INFO] Determined delimiter of CSV input is ','\u001b[0m\n",
      "\u001b[35m[2020-07-14:13:17:20:INFO] Determined delimiter of CSV input is ','\u001b[0m\n",
      "\u001b[34m169.254.255.130 - - [14/Jul/2020:13:17:21 +0000] \"POST /invocations HTTP/1.1\" 200 755087 \"-\" \"Go-http-client/1.1\"\u001b[0m\n",
      "\u001b[34m169.254.255.130 - - [14/Jul/2020:13:17:21 +0000] \"POST /invocations HTTP/1.1\" 200 763015 \"-\" \"Go-http-client/1.1\"\u001b[0m\n",
      "\u001b[34m169.254.255.130 - - [14/Jul/2020:13:17:21 +0000] \"POST /invocations HTTP/1.1\" 200 769487 \"-\" \"Go-http-client/1.1\"\u001b[0m\n",
      "\u001b[34m[2020-07-14:13:17:21:INFO] Determined delimiter of CSV input is ','\u001b[0m\n",
      "\u001b[34m169.254.255.130 - - [14/Jul/2020:13:17:21 +0000] \"POST /invocations HTTP/1.1\" 200 781389 \"-\" \"Go-http-client/1.1\"\u001b[0m\n",
      "\u001b[35m169.254.255.130 - - [14/Jul/2020:13:17:21 +0000] \"POST /invocations HTTP/1.1\" 200 755087 \"-\" \"Go-http-client/1.1\"\u001b[0m\n",
      "\u001b[35m169.254.255.130 - - [14/Jul/2020:13:17:21 +0000] \"POST /invocations HTTP/1.1\" 200 763015 \"-\" \"Go-http-client/1.1\"\u001b[0m\n",
      "\u001b[35m169.254.255.130 - - [14/Jul/2020:13:17:21 +0000] \"POST /invocations HTTP/1.1\" 200 769487 \"-\" \"Go-http-client/1.1\"\u001b[0m\n",
      "\u001b[35m[2020-07-14:13:17:21:INFO] Determined delimiter of CSV input is ','\u001b[0m\n",
      "\u001b[35m169.254.255.130 - - [14/Jul/2020:13:17:21 +0000] \"POST /invocations HTTP/1.1\" 200 781389 \"-\" \"Go-http-client/1.1\"\u001b[0m\n",
      "\u001b[34m169.254.255.130 - - [14/Jul/2020:13:17:22 +0000] \"POST /invocations HTTP/1.1\" 200 769686 \"-\" \"Go-http-client/1.1\"\u001b[0m\n",
      "\u001b[35m169.254.255.130 - - [14/Jul/2020:13:17:22 +0000] \"POST /invocations HTTP/1.1\" 200 769686 \"-\" \"Go-http-client/1.1\"\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "xgb_transformer.wait()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now the transform job has executed and the result, the estimated sentiment of each review, has been saved on S3. Since we would rather work on this file locally we can perform a bit of notebook magic to copy the file to the `data_dir`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "download: s3://sagemaker-ap-southeast-2-480972076311/sagemaker-xgboost-200714-1258-003-f46ce-2020-07-14-13-12-50-264/test.csv.out to data/xgboost/test.csv.out\n"
     ]
    }
   ],
   "source": [
    "!aws s3 cp --recursive $xgb_transformer.output_path $data_dir"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Putting our model to work\n",
    "\n",
    "As we've mentioned a few times now, our goal is to have our model deployed and then access it using a very simple web app. The intent is for this web app to take some user submitted data (a review), send it off to our endpoint (the model) and then display the result.\n",
    "\n",
    "However, there is a small catch. Currently the only way we can access the endpoint to send it data is using the SageMaker API. We can, if we wish, expose the actual URL that our model's endpoint is receiving data from, however, if we just send it data ourselves we will not get anything in return. This is because the endpoint created by SageMaker requires the entity accessing it have the correct permissions. So, we would need to somehow authenticate our web app with AWS.\n",
    "\n",
    "Having a website that authenticates to AWS seems a bit beyond the scope of this course so we will opt for an alternative approach. Namely, we will create a new endpoint which does not require authentication and which acts as a proxy for the SageMaker endpoint.\n",
    "\n",
    "As an additional constraint, we will try to avoid doing any data processing in the web app itself. Remember that when we constructed and tested our model we started with a movie review, then we simplified it by removing any html formatting and punctuation, then we constructed a bag of words embedding and the resulting vector is what we sent to our model. All of this needs to be done to our user input as well.\n",
    "\n",
    "Fortunately we can do all of this data processing in the backend, using Amazon's Lambda service.\n",
    "\n",
    "<img src=\"Web App Diagram.svg\">\n",
    "\n",
    "The diagram above gives an overview of how the various services will work together. On the far right is the model which we trained above and which will be deployed using SageMaker. On the far left is our web app that collects a user's feature data, sends it off and expects a buy or not buy in return.\n",
    "\n",
    "In the middle is where some of the magic happens. We will construct a Lambda function, which you can think of as a straightforward Python function that can be executed whenever a specified event occurs. This Python function will do the data processing we need to perform on a user submitted review. In addition, we will give this function permission to send and recieve data from a SageMaker endpoint.\n",
    "\n",
    "Lastly, the method we will use to execute the Lambda function is a new endpoint that we will create using API Gateway. This endpoint will be a url that listens for data to be sent to it. Once it gets some data it will pass that data on to the Lambda function and then return whatever the Lambda function returns. Essentially it will act as an interface that lets our web app communicate with the Lambda function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "\n",
    "runtime = boto3.Session().client('sagemaker-runtime')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:sagemaker:Parameter image will be renamed to image_uri in SageMaker Python SDK v2.\n",
      "WARNING:sagemaker:Using already existing model: sagemaker-xgboost-200714-1258-003-f46ce743\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------------!"
     ]
    }
   ],
   "source": [
    "# now we deploy the model as an endpoint\n",
    "xgb_predictor = xgb_attached.deploy(initial_instance_count = 1, instance_type = 'ml.m4.xlarge')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'sagemaker-xgboost-200714-1258-003-f46ce743'"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# make a note of the model endpoint been created\n",
    "xgb_predictor.endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "# let's invoke the endpoint and see if it works\n",
    "response = runtime.invoke_endpoint(EndpointName = xgb_predictor.endpoint, # The name of the endpoint we created\n",
    "                                       ContentType = 'text/csv',                     # The data format that is expected\n",
    "                                       Body = '1,6.67578125,3418,209,0.595703125,514,10.0,11,57,11,1599,0.1498791297340854,1.2884770346494763,0.22388993120700437,9.017543859649123,0.017543859649122806,46,0.02127659574468085')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0.003530889982357621'"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# here is the response: the probablility of a user buy the product\n",
    "response['Body'].read().decode('utf-8')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The last step is now to read in the output from our model, convert the output to something a little more usable, in this case we want the result to be either `1` (buying) or `0` (not buying), and then compare to the ground truth labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = pd.read_csv(os.path.join(data_dir, 'test.csv.out'), header=None)\n",
    "predictions = [round(num) for num in predictions.squeeze().values]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we know how to process the incoming user data we can start setting up the infrastructure to make our simple web app work. To do this we will make use of two different services. Amazon's Lambda and API Gateway services.\n",
    "\n",
    "Lambda is a service which allows someone to write some relatively simple code and have it executed whenever a chosen trigger occurs. For example, you may want to update a database whenever new data is uploaded to a folder stored on S3.\n",
    "\n",
    "API Gateway is a service that allows you to create HTTP endpoints (url addresses) which are connected to other AWS services. One of the benefits to this is that you get to decide what credentials, if any, are required to access these endpoints.\n",
    "\n",
    "In our case we are going to set up an HTTP endpoint through API Gateway which is open to the public. Then, whenever anyone sends data to our public endpoint we will trigger a Lambda function which will send the input (in our case a review) to our model's endpoint and then return the result."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setting up a Lambda function\n",
    "\n",
    "The first thing we are going to do is set up a Lambda function. This Lambda function will be executed whenever our public API has data sent to it. When it is executed it will receive the data, perform any sort of processing that is required, send the data (the review) to the SageMaker endpoint we've created and then return the result.\n",
    "\n",
    "#### Part A: Create an IAM Role for the Lambda function\n",
    "\n",
    "Since we want the Lambda function to call a SageMaker endpoint, we need to make sure that it has permission to do so. To do this, we will construct a role that we can later give the Lambda function.\n",
    "\n",
    "Using the AWS Console, navigate to the **IAM** page and click on **Roles**. Then, click on **Create role**. Make sure that the **AWS service** is the type of trusted entity selected and choose **Lambda** as the service that will use this role, then click **Next: Permissions**.\n",
    "\n",
    "In the search box type `sagemaker` and select the check box next to the **AmazonSageMakerFullAccess** policy. Then, click on **Next: Review**.\n",
    "\n",
    "Lastly, give this role a name. Make sure you use a name that you will remember later on, for example `LambdaSageMakerRole`. Then, click on **Create role**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Part B: Create a Lambda function\n",
    "\n",
    "Now it is time to actually create the Lambda function. Remember from earlier that in order to process the user provided input and send it to our endpoint we need to gather two pieces of information:\n",
    "\n",
    " - The name of the endpoint, and\n",
    " - the feature object.\n",
    "\n",
    "We will copy these pieces of information to our Lambda function after we create it.\n",
    "\n",
    "To start, using the AWS Console, navigate to the AWS Lambda page and click on **Create a function**. When you get to the next page, make sure that **Author from scratch** is selected. Now, name your Lambda function, using a name that you will remember later on, for example `imba_xgboost_func`. Make sure that the **Python 3.8** runtime is selected and then choose the role that you created in the previous part. Then, click on **Create Function**.\n",
    "\n",
    "On the next page you will see some information about the Lambda function you've just created. If you scroll down you should see an editor in which you can write the code that will be executed when your Lambda function is triggered. Collecting the code we wrote above to process a single review and adding it to the provided example `lambda_handler` we arrive at the following.\n",
    "\n",
    "```python\n",
    "# We need to use the low-level library to interact with SageMaker since the SageMaker API\n",
    "# is not available natively through Lambda.\n",
    "import boto3\n",
    "\n",
    "def lambda_handler(event, context):\n",
    "\n",
    "    body = event['body']\n",
    "\n",
    "    # The SageMaker runtime is what allows us to invoke the endpoint that we've created.\n",
    "    runtime = boto3.Session().client('sagemaker-runtime')\n",
    "\n",
    "    # Now we use the SageMaker runtime to invoke our endpoint, sending the review we were given\n",
    "    response = runtime.invoke_endpoint(EndpointName = '***ENDPOINT NAME HERE***',# The name of the endpoint we created\n",
    "                                       ContentType = 'text/csv',                 # The data format that is expected\n",
    "                                       Body = body\n",
    "                                       )\n",
    "\n",
    "    # The response is an HTTP response whose body contains the result of our inference\n",
    "    result = response['Body'].read().decode('utf-8')\n",
    "\n",
    "    # Round the result so that our web app only gets '1' or '0' as a response.\n",
    "    result = float(result)\n",
    "\n",
    "    return {\n",
    "        'statusCode' : 200,\n",
    "        'headers' : { 'Content-Type' : 'text/plain', 'Access-Control-Allow-Origin' : '*' },\n",
    "        'body' : str(result)\n",
    "    }\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once you have copy and pasted the code above into the Lambda code editor, replace the `**ENDPOINT NAME HERE**` portion with the name of the endpoint that we deployed earlier. You can determine the name of the endpoint using the code cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'sagemaker-xgboost-200714-1258-003-f46ce743'"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xgb_predictor.endpoint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setting up API Gateway\n",
    "\n",
    "Now that our Lambda function is set up, it is time to create a new API using API Gateway that will trigger the Lambda function we have just created.\n",
    "\n",
    "Using AWS Console, navigate to **Amazon API Gateway** and then click on **Get started**.\n",
    "\n",
    "On the next page, make sure that **New API** is selected and give the new api a name, for example, `imba_web_app`. Then, click on **Create API**.\n",
    "\n",
    "Now we have created an API, however it doesn't currently do anything. What we want it to do is to trigger the Lambda function that we created earlier.\n",
    "\n",
    "Select the **Actions** dropdown menu and click **Create Method**. A new blank method will be created, select its dropdown menu and select **POST**, then click on the check mark beside it.\n",
    "\n",
    "For the integration point, make sure that **Lambda Function** is selected and click on the **Use Lambda Proxy integration**. This option makes sure that the data that is sent to the API is then sent directly to the Lambda function with no processing. It also means that the return value must be a proper response object as it will also not be processed by API Gateway.\n",
    "\n",
    "Type the name of the Lambda function you created earlier into the **Lambda Function** text entry box and then click on **Save**. Click on **OK** in the pop-up box that then appears, giving permission to API Gateway to invoke the Lambda function you created.\n",
    "\n",
    "The last step in creating the API Gateway is to select the **Actions** dropdown and click on **Deploy API**. You will need to create a new Deployment stage and name it anything you like, for example `prod`.\n",
    "\n",
    "You have now successfully set up a public API to access your SageMaker model. Make sure to copy or write down the URL provided to invoke your newly created public API as this will be needed in the next step. This URL can be found at the top of the page, highlighted in blue next to the text **Invoke URL**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Deploying our web app\n",
    "\n",
    "Now that we have a publicly available API, we can start using it in a web app. For our purposes, we have provided a simple static html file which can make use of the public api you created earlier.\n",
    "\n",
    "In the `website` folder there should be a file called `index.html`. Download the file to your computer and open that file up in a text editor of your choice. There should be a line which contains **\\*\\*REPLACE WITH PUBLIC API URL\\*\\***. Replace this string with the url that you wrote down in the last step and then save the file.\n",
    "\n",
    "Now, if you open `index.html` on your local computer, your browser will behave as a local web server and you can use the provided site to interact with your SageMaker model.\n",
    "\n",
    "If you'd like to go further, you can host this html file anywhere you'd like, for example using github or hosting a static site on Amazon's S3. Once you have done this you can share the link with anyone you'd like and have them play with it too!\n",
    "\n",
    "> **Important Note** In order for the web app to communicate with the SageMaker endpoint, the endpoint has to actually be deployed and running. This means that you are paying for it. Make sure that the endpoint is running when you want to use the web app but that you shut it down when you don't need it, otherwise you will end up with a surprisingly large AWS bill."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Delete the endpoint\n",
    "\n",
    "Remember to always shut down your endpoint if you are no longer using it. You are charged for the length of time that the endpoint is running so if you forget and leave it on you could end up with an unexpectedly large bill."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xgb_predictor.delete_endpoint()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}