otto-orders-predict-combine.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4523a29a",
   "metadata": {
    "papermill": {
     "duration": 0.004064,
     "end_time": "2023-03-23T11:28:47.451745",
     "exception": false,
     "start_time": "2023-03-23T11:28:47.447681",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Making and combining predictions for orders\n",
    "The prediction of aids user is going order is made in this notebook. The notebook uses input from \"Orders Model\" notebook, where the orders models are fitted, and two \"parallel\" notebooks that produce w2vec features for orders, each preparing features for one of the cross-validation datasets and one chunk of the test dataset.\n",
    "\n",
    "It was impossible to fit the model and make predictions in the same notebook, because of limitations of kaggle platform. On kaggle platform, notebooks with GPU have less memory available, and it was hard to fit all the required data into 13 GB of available RAM, so I had to move prediction to a different notebook without GPU support, but with 30Gb RAM available.\n",
    "\n",
    "Two models are used to predict the orders, LGBM and catboost, trained on different cross-validation datasets. Both models rank the candidates, then those ranks evaluations are scaled and combined to produce the final prediction.\n",
    "## Imports and definitions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "f160d5d3",
   "metadata": {
    "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
    "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5",
    "execution": {
     "iopub.execute_input": "2023-03-23T11:28:47.459787Z",
     "iopub.status.busy": "2023-03-23T11:28:47.459351Z",
     "iopub.status.idle": "2023-03-23T11:28:49.675002Z",
     "shell.execute_reply": "2023-03-23T11:28:49.673980Z"
    },
    "papermill": {
     "duration": 2.222289,
     "end_time": "2023-03-23T11:28:49.677319",
     "exception": false,
     "start_time": "2023-03-23T11:28:47.455030",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type='text/css'>\n",
       ".datatable table.frame { margin-bottom: 0; }\n",
       ".datatable table.frame thead { border-bottom: none; }\n",
       ".datatable table.frame tr.coltypes td {  color: #FFFFFF;  line-height: 6px;  padding: 0 0.5em;}\n",
       ".datatable .bool    { background: #DDDD99; }\n",
       ".datatable .object  { background: #565656; }\n",
       ".datatable .int     { background: #5D9E5D; }\n",
       ".datatable .float   { background: #4040CC; }\n",
       ".datatable .str     { background: #CC4040; }\n",
       ".datatable .time    { background: #40CC40; }\n",
       ".datatable .row_index {  background: var(--jp-border-color3);  border-right: 1px solid var(--jp-border-color0);  color: var(--jp-ui-font-color3);  font-size: 9px;}\n",
       ".datatable .frame tbody td { text-align: left; }\n",
       ".datatable .frame tr.coltypes .row_index {  background: var(--jp-border-color0);}\n",
       ".datatable th:nth-child(2) { padding-left: 12px; }\n",
       ".datatable .hellipsis {  color: var(--jp-cell-editor-border-color);}\n",
       ".datatable .vellipsis {  background: var(--jp-layout-color0);  color: var(--jp-cell-editor-border-color);}\n",
       ".datatable .na {  color: var(--jp-cell-editor-border-color);  font-size: 80%;}\n",
       ".datatable .sp {  opacity: 0.25;}\n",
       ".datatable .footer { font-size: 9px; }\n",
       ".datatable .frame_dimensions {  background: var(--jp-border-color3);  border-top: 1px solid var(--jp-border-color0);  color: var(--jp-ui-font-color3);  display: inline-block;  opacity: 0.6;  padding: 1px 10px 1px 5px;}\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np # linear algebra\n",
    "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n",
    "\n",
    "import gc\n",
    "from lightgbm.sklearn import LGBMRanker\n",
    "from catboost import CatBoostRanker, Pool    \n",
    "from sklearn.preprocessing import StandardScaler\n",
    "import joblib\n",
    "\n",
    "# functions and classes common for several notebooks of current project\n",
    "import otto_common"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "788dda16",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-23T11:28:49.685848Z",
     "iopub.status.busy": "2023-03-23T11:28:49.685348Z",
     "iopub.status.idle": "2023-03-23T11:28:49.689155Z",
     "shell.execute_reply": "2023-03-23T11:28:49.688334Z"
    },
    "papermill": {
     "duration": 0.010285,
     "end_time": "2023-03-23T11:28:49.691080",
     "exception": false,
     "start_time": "2023-03-23T11:28:49.680795",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# This function was used to test new features before adding them to the pipeline.\n",
    "# Now it only deletes the day_of_week column, which is used to construct some features.\n",
    "def prepare_df(df):\n",
    "    del df['day_of_week']\n",
    "    return df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a989fe8d",
   "metadata": {
    "papermill": {
     "duration": 0.003059,
     "end_time": "2023-03-23T11:28:49.697773",
     "exception": false,
     "start_time": "2023-03-23T11:28:49.694714",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Load the models and make predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "fafea812",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-23T11:28:49.706047Z",
     "iopub.status.busy": "2023-03-23T11:28:49.705675Z",
     "iopub.status.idle": "2023-03-23T11:28:49.845504Z",
     "shell.execute_reply": "2023-03-23T11:28:49.844760Z"
    },
    "papermill": {
     "duration": 0.146527,
     "end_time": "2023-03-23T11:28:49.847606",
     "exception": false,
     "start_time": "2023-03-23T11:28:49.701079",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Load the LGBM model.\n",
    "model = joblib.load('/kaggle/input/otto-model-orders/lgb.pkl')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "c25e970a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-23T11:28:49.856468Z",
     "iopub.status.busy": "2023-03-23T11:28:49.855363Z",
     "iopub.status.idle": "2023-03-23T11:49:40.666366Z",
     "shell.execute_reply": "2023-03-23T11:49:40.664725Z"
    },
    "papermill": {
     "duration": 1250.823282,
     "end_time": "2023-03-23T11:49:40.674399",
     "exception": false,
     "start_time": "2023-03-23T11:28:49.851117",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Start predicting 0\n",
      "Predictions made 0\n",
      "Start predicting 1\n",
      "Predictions made 1\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Make predictions using the LGBM model.\n",
    "file_path_part_0 = '/kaggle/input/otto-orders-w2vec/train_features_with_w2v_part_0.parquet'\n",
    "file_path_part_1 = '/kaggle/input/otto-orders-w2vec-part1/train_features_with_w2v_part_1.parquet'\n",
    "\n",
    "# Load and prepare the data.\n",
    "for i in range(2):\n",
    "    print('Start predicting '+ str(i))\n",
    "    j_max = 3\n",
    "    for j in range(j_max):\n",
    "        if i == 0:\n",
    "            df_test = pd.read_parquet(file_path_part_0)\n",
    "        else:\n",
    "            df_test = pd.read_parquet(file_path_part_1)\n",
    "        df_test = otto_common.divide_df_by_column(df_test, j_max, j, 'session')\n",
    "        df_test = prepare_df(df_test)\n",
    "        x_cols = list(df_test.columns[2:])\n",
    "        \n",
    "        # Prediction itself.\n",
    "        df_test['gbdt_prediction'] = model.predict(df_test[x_cols])\n",
    "        \n",
    "        # Remove the features and combine the predictions for chunks of test data into a single dataframe.\n",
    "        df_test = df_test[['session','order_predictions','gbdt_prediction']]\n",
    "        gc.collect()\n",
    "        if (i == 0) & (j == 0):\n",
    "            df_cv1 = df_test\n",
    "        else:\n",
    "            df_cv1 = pd.concat([df_cv1, df_test])\n",
    "    print('Predictions made '+ str(i))\n",
    "del df_test, model\n",
    "gc.collect()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "378a5bc9",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-23T11:49:40.684225Z",
     "iopub.status.busy": "2023-03-23T11:49:40.683832Z",
     "iopub.status.idle": "2023-03-23T11:49:40.756626Z",
     "shell.execute_reply": "2023-03-23T11:49:40.755394Z"
    },
    "papermill": {
     "duration": 0.08142,
     "end_time": "2023-03-23T11:49:40.759493",
     "exception": false,
     "start_time": "2023-03-23T11:49:40.678073",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<catboost.core.CatBoostRanker at 0x7fa1363032d0>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Load the catboost model.\n",
    "\n",
    "model_catboost = CatBoostRanker()\n",
    "\n",
    "model_catboost.load_model(\"/kaggle/input/otto-model-orders/model\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "453f4120",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-23T11:49:40.769362Z",
     "iopub.status.busy": "2023-03-23T11:49:40.768955Z",
     "iopub.status.idle": "2023-03-23T11:53:03.631170Z",
     "shell.execute_reply": "2023-03-23T11:53:03.630278Z"
    },
    "papermill": {
     "duration": 202.874337,
     "end_time": "2023-03-23T11:53:03.637952",
     "exception": false,
     "start_time": "2023-03-23T11:49:40.763615",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Start predicting 0\n",
      "Predictions made 0\n",
      "Start predicting 1\n",
      "Predictions made 1\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Make predictions using the catboost model.\n",
    "\n",
    "# Load and prepare the data.\n",
    "for i in range(2):\n",
    "    print('Start predicting '+ str(i))\n",
    "    j_max = 5\n",
    "    for j in range(j_max):\n",
    "        if i == 0:\n",
    "            df_test = pd.read_parquet(file_path_part_0)\n",
    "        else:\n",
    "            df_test = pd.read_parquet(file_path_part_1)\n",
    "        df_test = otto_common.divide_df_by_column(df_test, j_max, j, 'session')\n",
    "        df_test = prepare_df(df_test)\n",
    "        x_cols = list(df_test.columns[2:])\n",
    "        test_pool = Pool(\n",
    "            data=df_test[x_cols],\n",
    "            group_id=df_test['session']\n",
    "        )\n",
    "        gc.collect()\n",
    "        \n",
    "        # Prediction itself.\n",
    "        df_test['from_cv2_prediction'] = model_catboost.predict(test_pool)\n",
    "        \n",
    "        # Remove the features and combine the predictions for chunks of test data into a single dataframe.\n",
    "        df_test = df_test[['session','order_predictions','from_cv2_prediction']]\n",
    "        gc.collect()\n",
    "        if (i == 0) & (j == 0):\n",
    "            df_cv2 = df_test\n",
    "        else:\n",
    "            df_cv2 = pd.concat([df_cv2, df_test])\n",
    "    print('Predictions made '+ str(i))\n",
    "del df_test, test_pool, model_catboost\n",
    "gc.collect()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "18c7acb2",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-23T11:53:03.648468Z",
     "iopub.status.busy": "2023-03-23T11:53:03.648081Z",
     "iopub.status.idle": "2023-03-23T11:54:45.919103Z",
     "shell.execute_reply": "2023-03-23T11:54:45.918101Z"
    },
    "papermill": {
     "duration": 102.282114,
     "end_time": "2023-03-23T11:54:45.924632",
     "exception": false,
     "start_time": "2023-03-23T11:53:03.642518",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "21"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Merge the predictions made by both models into a single dataframe\n",
    "df_total = pd.merge(df_cv1, df_cv2, how='outer', on=['session', 'order_predictions'])\n",
    "\n",
    "del df_cv1, df_cv2\n",
    "gc.collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8898e8d8",
   "metadata": {
    "papermill": {
     "duration": 0.004053,
     "end_time": "2023-03-23T11:54:45.932732",
     "exception": false,
     "start_time": "2023-03-23T11:54:45.928679",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Scale the results and calculate the average prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "e8e8db7f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-23T11:54:45.942483Z",
     "iopub.status.busy": "2023-03-23T11:54:45.942151Z",
     "iopub.status.idle": "2023-03-23T11:54:50.893414Z",
     "shell.execute_reply": "2023-03-23T11:54:50.892437Z"
    },
    "papermill": {
     "duration": 4.958755,
     "end_time": "2023-03-23T11:54:50.895667",
     "exception": false,
     "start_time": "2023-03-23T11:54:45.936912",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Scale the results.\n",
    "scaler = StandardScaler()\n",
    "scaled_data = scaler.fit_transform(df_total[['gbdt_prediction', 'from_cv2_prediction']].values)\n",
    "scaled_results = pd.DataFrame(scaled_data)\n",
    "scaled_results = scaled_results.rename(columns={0:'lgbm', 1:'cat'})\n",
    "df_total = pd.concat([df_total, scaled_results], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "700dda01",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-23T11:54:50.908498Z",
     "iopub.status.busy": "2023-03-23T11:54:50.908173Z",
     "iopub.status.idle": "2023-03-23T11:54:54.475621Z",
     "shell.execute_reply": "2023-03-23T11:54:54.474279Z"
    },
    "papermill": {
     "duration": 3.577556,
     "end_time": "2023-03-23T11:54:54.478628",
     "exception": false,
     "start_time": "2023-03-23T11:54:50.901072",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Calculate the combined prediction using hand-picked coefficients.\n",
    "df_total['sum'] = 0.7 * df_total['lgbm'] + 0.3 * df_total['cat']\n",
    "df_total = df_total[['session','order_predictions','sum']]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae603b0d",
   "metadata": {
    "papermill": {
     "duration": 0.003997,
     "end_time": "2023-03-23T11:54:54.487285",
     "exception": false,
     "start_time": "2023-03-23T11:54:54.483288",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Final formatting and export to file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "17915516",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-23T11:54:54.498634Z",
     "iopub.status.busy": "2023-03-23T11:54:54.498087Z",
     "iopub.status.idle": "2023-03-23T11:58:21.715745Z",
     "shell.execute_reply": "2023-03-23T11:58:21.713838Z"
    },
    "papermill": {
     "duration": 207.226738,
     "end_time": "2023-03-23T11:58:21.718221",
     "exception": false,
     "start_time": "2023-03-23T11:54:54.491483",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Select top 20 candidates and format the prediction as required by organizers.\n",
    "df_total = otto_common.select_top_20_and_format(df_total, 'order_predictions','sum')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "fe2d6926",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-23T11:58:21.729090Z",
     "iopub.status.busy": "2023-03-23T11:58:21.728744Z",
     "iopub.status.idle": "2023-03-23T11:58:23.242708Z",
     "shell.execute_reply": "2023-03-23T11:58:23.241423Z"
    },
    "papermill": {
     "duration": 1.5224,
     "end_time": "2023-03-23T11:58:23.245528",
     "exception": false,
     "start_time": "2023-03-23T11:58:21.723128",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "df_total.to_parquet('gbdt_predictions_from_both_cvs.parquet')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 1785.017313,
   "end_time": "2023-03-23T11:58:24.282145",
   "environment_variables": {},
   "exception": null,
   "input_path": "__notebook__.ipynb",
   "output_path": "__notebook__.ipynb",
   "parameters": {},
   "start_time": "2023-03-23T11:28:39.264832",
   "version": "2.3.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}