03_custom_metrics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Custom metrics\n",
    "\n",
    "In an attempt to improve the [revamped](https://github.com/Gearlux/football-predictor/blob/master/02_revamped.ipynb) code,\n",
    "we are going to implement a custom metric which measures the profit. In order to do this, we need to supply the odds\n",
    "to the keras metrics. I have not found a way to easily do this, but we can encode the odds and implement our own loss\n",
    "function which can handle the encoded labels. \n",
    "\n",
    "It turns out that these encoded labels can also be used to write a custom loss function, which optimizes the return of our betting. \n",
    "\n",
    "In addition, one could also think of a way not to bet on certain matches. This is also modeled with a fourth class and a new custom loss function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the data and setup a keras environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import dataset\n",
    "import keras\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "\n",
    "from keras import regularizers\n",
    "from keras import metrics\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense\n",
    "from keras.optimizers import Adagrad\n",
    "from keras.utils import np_utils\n",
    "from keras_tqdm import TQDMNotebookCallback\n",
    "\n",
    "from sklearn.preprocessing import StandardScaler, LabelEncoder\n",
    "\n",
    "from keras import backend as K\n",
    "\n",
    "np.set_printoptions(suppress=True)\n",
    "\n",
    "book = dataset.Dataset('data/book.csv')\n",
    "df = pd.DataFrame(book.processed_results)\n",
    "\n",
    "def get_feables(df):\n",
    "    features = df.drop(columns=['result','date'])\n",
    "    encoder = LabelEncoder()\n",
    "    labels = np_utils.to_categorical(-encoder.fit_transform(df.result.copy()) + 2)\n",
    "    return features, labels\n",
    "\n",
    "TRAINING_SET_FRACTION = 0.95\n",
    "train_results_len = int(TRAINING_SET_FRACTION * df.shape[0])\n",
    "\n",
    "features, labels = get_feables(df)\n",
    "train_features = features[:train_results_len]\n",
    "test_features = features[train_results_len:]\n",
    "y_train = labels[:train_results_len]\n",
    "y_test = labels[train_results_len:]\n",
    "\n",
    "scaler = StandardScaler()\n",
    "X_train = scaler.fit_transform(train_features.astype(float))\n",
    "X_test = scaler.transform(test_features.astype(float))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Utilities\n",
    "\n",
    "To make our live easier, we are going to implement a function which constructs our model with a specified\n",
    "- loss function\n",
    "- a list of metrics\n",
    "\n",
    "We also want to have reproducible results. \n",
    "With the construction of each model we are setting the seed of some libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from numpy.random import seed\n",
    "from tensorflow import set_random_seed\n",
    "    \n",
    "def construct_model(loss='categorical_crossentropy', metrics=['accuracy']):\n",
    "    K.clear_session()\n",
    "\n",
    "    seed(42)\n",
    "    set_random_seed(42)\n",
    "\n",
    "    model = Sequential()\n",
    "    model.add(Dense(10, input_dim=X_train.shape[1],\n",
    "                    kernel_regularizer=regularizers.l1(0.001)\n",
    "                   ))\n",
    "    model.add(Dense(3, \n",
    "                    kernel_regularizer=regularizers.l1(0.001),\n",
    "                    activation = 'softmax'\n",
    "                   ))\n",
    "       \n",
    "    model.compile(loss=loss, \n",
    "                  optimizer=Adagrad(0.1),\n",
    "                  metrics=metrics\n",
    "                 )\n",
    "\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In our [previous code](https://github.com/Gearlux/football-predictor/blob/master/02_revamped.ipynb), \n",
    "we have implemented a performance measure, relative to the odds of the bookmakers.\n",
    "\n",
    "You could also think of an absolute metric which does not use the bookmakers odds for comparison,\n",
    "but use the absolute probability of the model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def performance(th=0.05, features=test_features, name=None):\n",
    "    def perf_metric(y_pred, y_true):\n",
    "        odds = features[['odds-home','odds-draw','odds-away']]\n",
    "        selection = (y_pred > (1./odds + th))\n",
    "        profit = ( selection * (odds * y_true - 1)).sum()\n",
    "        count = selection.sum()\n",
    "        if name:\n",
    "            profit.name = 'Profit ' + name\n",
    "            count.name = 'Count ' + name\n",
    "        return profit, count\n",
    "    return perf_metric\n",
    "\n",
    "def absolute_performance(th=0.9, features=test_features, name=None):\n",
    "    def abs_perf_metric(y_pred, y_true):\n",
    "        odds = features[['odds-home','odds-draw','odds-away']]\n",
    "        selection = (y_pred > th)\n",
    "        profit = ( selection * (odds * y_true - 1)).sum()\n",
    "        count = selection.sum(axis=0)\n",
    "        count = pd.Series(count, index=profit.index)\n",
    "        if name:\n",
    "            profit.name = '|Profit| ' + name\n",
    "            count.name = '|Count| ' + name\n",
    "        return profit, count\n",
    "    return abs_perf_metric"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Original results\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import shutil\n",
    "import os\n",
    "if os.path.exists('logs'):\n",
    "    shutil.rmtree('logs')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /Users/gert/GitHub/MakingMoneyML/venv/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Colocations handled automatically by placer.\n",
      "WARNING:tensorflow:From /Users/gert/GitHub/MakingMoneyML/venv/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.cast instead.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "39274b9974864ae3a5ac1184d5ddff83",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=200, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "model = construct_model()\n",
    "_ = model.fit(X_train, y_train,\n",
    "      epochs=200,\n",
    "      batch_size=500, verbose=0,\n",
    "      validation_data = [X_test, y_test],\n",
    "      callbacks=[keras.callbacks.TensorBoard(log_dir='./logs/original', write_graph=True), \n",
    "                 TQDMNotebookCallback(show_inner=False)]\n",
    " )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Profit original</th>\n",
       "      <th>Count original</th>\n",
       "      <th>|Profit| original</th>\n",
       "      <th>|Count| original</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>odds-home</th>\n",
       "      <td>8.00</td>\n",
       "      <td>11</td>\n",
       "      <td>0.62</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>odds-draw</th>\n",
       "      <td>-2.00</td>\n",
       "      <td>2</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>odds-away</th>\n",
       "      <td>-0.84</td>\n",
       "      <td>13</td>\n",
       "      <td>-0.32</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           Profit original  Count original  |Profit| original  \\\n",
       "odds-home             8.00              11               0.62   \n",
       "odds-draw            -2.00               2               0.00   \n",
       "odds-away            -0.84              13              -0.32   \n",
       "\n",
       "           |Count| original  \n",
       "odds-home                 5  \n",
       "odds-draw                 0  \n",
       "odds-away                 4  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "original = pd.concat([pd.concat(performance(0.05,name='original')(model.predict(X_test), y_test),axis=1),\n",
    "pd.concat(absolute_performance(0.9,name='original')(model.predict(X_test), y_test),axis=1)], axis=1)\n",
    "display(original)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Custom loss function\n",
    "\n",
    "This custom loss function does not use the categorical encoded labels, \n",
    "but labels to also include the odds associated with the winning label.\n",
    "We will call the labels with the encoded winning odds as `b_train` or `b_test`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "odds_train = train_features[['odds-home','odds-draw','odds-away']]\n",
    "odds_test = test_features[['odds-home','odds-draw','odds-away']]\n",
    "\n",
    "b_train = odds_train * (2 * y_train - 1)\n",
    "b_test = odds_test * (2 * y_test - 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The new labels new look like"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>odds-home</th>\n",
       "      <th>odds-draw</th>\n",
       "      <th>odds-away</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.17</td>\n",
       "      <td>-6.5</td>\n",
       "      <td>-21.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-2.00</td>\n",
       "      <td>3.3</td>\n",
       "      <td>-4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-1.91</td>\n",
       "      <td>-3.4</td>\n",
       "      <td>4.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>8.00</td>\n",
       "      <td>-4.5</td>\n",
       "      <td>-1.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3.50</td>\n",
       "      <td>-3.3</td>\n",
       "      <td>-2.1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   odds-home  odds-draw  odds-away\n",
       "0       1.17       -6.5      -21.0\n",
       "1      -2.00        3.3       -4.0\n",
       "2      -1.91       -3.4        4.2\n",
       "3       8.00       -4.5       -1.4\n",
       "4       3.50       -3.3       -2.1"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b_train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So the winning label has a positive odd and the losing labels have negative odds.\n",
    "\n",
    "Our loss function should be capable of dealing with our now labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "_EPSILON = 10e-8\n",
    "\n",
    "def cat_loss(b_true, y_pred):\n",
    "    prob_true = K.clip(b_true, 0., 1.)\n",
    "    prob = K.clip(y_pred, _EPSILON, 1. - _EPSILON)\n",
    "    res2 = K.sum(prob_true * -K.log(prob), axis=-1)\n",
    "    return res2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create a new model which optimizes our new loss function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1ef010e4392541dc8fc4aa08df4e035c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=200, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "if os.path.exists('logs/cat_loss'):\n",
    "    shutil.rmtree('logs/cat_loss')\n",
    "    \n",
    "model = construct_model(cat_loss)\n",
    "_ = model.fit(X_train, b_train,\n",
    "      epochs=200,\n",
    "      batch_size=500, verbose=0,\n",
    "      validation_data = [X_test, b_test],\n",
    "      callbacks=[keras.callbacks.TensorBoard(log_dir='./logs/cat_loss', write_graph=True), \n",
    "                 TQDMNotebookCallback(show_inner=False)]\n",
    " )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also need to modify our performance functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def b_performance(th=0.05, name=None):\n",
    "    def perf_metric(y_pred, y_true):\n",
    "        odds = np.abs(y_true)\n",
    "        selection = (y_pred > (1./odds + th))\n",
    "        profit = ( selection * (odds * np.clip(y_true, 0, 1) - 1)).sum(axis=0)\n",
    "        count = selection.sum(axis=0)\n",
    "        profit = pd.Series(profit)\n",
    "        profit.index= ['home','draw','away']\n",
    "        count = pd.Series(count, index=profit.index)\n",
    "        if name:\n",
    "            profit.name = 'Profit ' + name\n",
    "            count.name = 'Count ' + name\n",
    "        return profit, count\n",
    "    return perf_metric\n",
    "\n",
    "def b_absolute_performance(th=0.9, name=None):\n",
    "    def abs_perf_metric(y_pred, y_true):\n",
    "        odds = np.abs(y_true)\n",
    "\n",
    "        selection = (y_pred > th)\n",
    "        profit = ( selection * (odds * np.clip(y_true, 0, 1) - 1)).sum(axis=0)\n",
    "        count = selection.sum(axis=0)\n",
    "        profit = pd.Series(profit)\n",
    "        profit.index= ['home','draw','away']\n",
    "        count = pd.Series(count, index=profit.index)\n",
    "        if name:\n",
    "            profit.name = '|Profit| ' + name\n",
    "            count.name = '|Count| ' + name\n",
    "        return profit, count\n",
    "    return abs_perf_metric"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Profit b</th>\n",
       "      <th>Count b</th>\n",
       "      <th>|Profit| b</th>\n",
       "      <th>|Count| b</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>home</th>\n",
       "      <td>8.00</td>\n",
       "      <td>11</td>\n",
       "      <td>0.62</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>draw</th>\n",
       "      <td>-2.00</td>\n",
       "      <td>2</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>away</th>\n",
       "      <td>-0.84</td>\n",
       "      <td>13</td>\n",
       "      <td>-0.32</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Profit b  Count b  |Profit| b  |Count| b\n",
       "home      8.00       11        0.62          5\n",
       "draw     -2.00        2        0.00          0\n",
       "away     -0.84       13       -0.32          4"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "b = pd.concat([pd.concat(b_performance(0.05,name='b')(model.predict(X_test), b_test.values),axis=1),\n",
    "pd.concat(b_absolute_performance(0.9,name='b')(model.predict(X_test), b_test.values),axis=1)], axis=1)\n",
    "display(b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we have implemented the `cat_loss` function correctly, we should have similar results as the original profit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Profit original</th>\n",
       "      <th>Count original</th>\n",
       "      <th>|Profit| original</th>\n",
       "      <th>|Count| original</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>odds-home</th>\n",
       "      <td>8.00</td>\n",
       "      <td>11</td>\n",
       "      <td>0.62</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>odds-draw</th>\n",
       "      <td>-2.00</td>\n",
       "      <td>2</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>odds-away</th>\n",
       "      <td>-0.84</td>\n",
       "      <td>13</td>\n",
       "      <td>-0.32</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           Profit original  Count original  |Profit| original  \\\n",
       "odds-home             8.00              11               0.62   \n",
       "odds-draw            -2.00               2               0.00   \n",
       "odds-away            -0.84              13              -0.32   \n",
       "\n",
       "           |Count| original  \n",
       "odds-home                 5  \n",
       "odds-draw                 0  \n",
       "odds-away                 4  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(original)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Success! Because we have setup the seeds of both our libraries to the answer to the universe,\n",
    "we obtain exaclty the same result.\n",
    "\n",
    "## Better metrics\n",
    "\n",
    "Don't forget the reason why we want to implement something similar.\n",
    "\n",
    "The reason why we want to implement this new loss function, is to define a metric which can shows us the\n",
    "profit.\n",
    "In the previous section, we could only monitor the profit at the end of fitting process.\n",
    "It makes more sense to monitor the profit during the learning process.\n",
    "\n",
    "We implemented two performance metrics in the previous section.\n",
    "Likewise, we will also implement two metrics who will each implement both the profit and the margin which \n",
    "can be monitored in tensorboard."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def bet_metric(th=0.05):\n",
    "    def profit(y_true, y_pred):\n",
    "        true_odds = K.abs(y_true)\n",
    "        selection = K.cast((y_pred > (1./true_odds + th)),'float32')\n",
    "        odds_true = K.clip(y_true, 0., np.inf) - 1\n",
    "        return K.sum(K.sum( selection * odds_true))\n",
    "    def margin(y_true, y_pred):\n",
    "        true_odds = K.abs(y_true)\n",
    "        selection = K.cast((y_pred > (1./true_odds + th)),'float32')\n",
    "        odds_true = K.clip(y_true, 0., np.inf) - 1\n",
    "        return K.sum(K.sum( selection * odds_true)) / K.sum(K.sum(selection))\n",
    "    return profit, margin\n",
    "\n",
    "def abs_bet_metric(th=0.9):\n",
    "    def abs_profit(y_true, y_pred):\n",
    "        selection = K.cast((y_pred > th),'float32')\n",
    "        odds_true = K.clip(y_true, 0., np.inf) - 1\n",
    "        return K.sum(K.sum( selection * odds_true))\n",
    "    def abs_margin(y_true, y_pred):\n",
    "        selection = K.cast((y_pred > th),'float32')\n",
    "        odds_true = K.clip(y_true, 0., np.inf) - 1\n",
    "        return K.sum(K.sum( selection * odds_true)) / K.sum(K.sum(selection))\n",
    "    return abs_profit, abs_margin"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8dfe4ddf5dd54c32b2500b04c367b7f6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=200, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "if os.path.exists('logs/bet_metric'):\n",
    "    shutil.rmtree('logs/bet_metric')\n",
    "    \n",
    "model = construct_model(cat_loss, metrics=['accuracy'] + list(bet_metric(0.05)) + list(abs_bet_metric(0.9)))\n",
    "_ = model.fit(X_train, b_train,\n",
    "      epochs=200,\n",
    "      batch_size=500, verbose=0,\n",
    "      validation_data = [X_test, b_test],\n",
    "      callbacks=[keras.callbacks.TensorBoard(log_dir='./logs/bet_metric', write_graph=True), \n",
    "                 TQDMNotebookCallback(show_inner=False)]\n",
    " )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Profit b</th>\n",
       "      <th>Count b</th>\n",
       "      <th>|Profit| b</th>\n",
       "      <th>|Count| b</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>home</th>\n",
       "      <td>8.00</td>\n",
       "      <td>11</td>\n",
       "      <td>0.62</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>draw</th>\n",
       "      <td>-2.00</td>\n",
       "      <td>2</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>away</th>\n",
       "      <td>-0.84</td>\n",
       "      <td>13</td>\n",
       "      <td>-0.32</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Profit b  Count b  |Profit| b  |Count| b\n",
       "home      8.00       11        0.62          5\n",
       "draw     -2.00        2        0.00          0\n",
       "away     -0.84       13       -0.32          4"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "b = pd.concat([pd.concat(b_performance(0.05,name='b')(model.predict(X_test), b_test.values),axis=1),\n",
    "pd.concat(b_absolute_performance(0.9,name='b')(model.predict(X_test), b_test.values),axis=1)], axis=1)\n",
    "display(b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are the same results as in the previous run, and surprise, surprise !\n",
    "In tensorboard, we can find the same end result for the profits and derived margins.\n",
    "Not only do we have the metric for our training set, but it's also available for the validation set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<tr>\n",
    "    <td><img src=\"images/acc.png\" width=\"400px\"></td>\n",
    "    <td><img src=\"images/loss.png\" width=\"400px\"></td>\n",
    "</tr>\n",
    "<tr>\n",
    "    <td><img src=\"images/profit.png\" width=\"400px\"></td>\n",
    "    <td><img src=\"images/val_profit.png\" width=\"400px\"></td>\n",
    "</tr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But the perfomance and the metrics, are similar functions. Can we share the code ?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "def profit_(y_true, y_pred, threshold):\n",
    "    selection = K.cast((y_pred > threshold),'float32')\n",
    "    odds_true = K.clip(y_true, 0., np.inf) - 1\n",
    "    return K.sum( selection * odds_true, axis=0), K.sum(selection, axis=0)\n",
    "\n",
    "def bet_metric(th=0.05):\n",
    "    def profit(y_true, y_pred):\n",
    "        true_odds = K.abs(y_true)\n",
    "        condition = (1./true_odds + th)\n",
    "        return K.sum(profit_(y_true, y_pred, condition)[0])\n",
    "    def margin(y_true, y_pred):\n",
    "        true_odds = K.abs(y_true)\n",
    "        condition = (1./true_odds + th)\n",
    "        pr = profit_(y_true, y_pred, condition)\n",
    "        return K.sum(pr[0]) / K.sum(pr[1])\n",
    "    return profit, margin\n",
    "\n",
    "def abs_bet_metric(th=0.9):\n",
    "    def abs_profit(y_true, y_pred):\n",
    "        return K.sum(profit_(y_true, y_pred, th)[0])\n",
    "    def abs_margin(y_true, y_pred):\n",
    "        pr = profit_(y_true, y_pred, th)\n",
    "        return K.sum(pr[0]) / K.sum(pr[1])\n",
    "    return abs_profit, abs_margin\n",
    "\n",
    "def b_performance(th=0.05, name=None):\n",
    "    def perf_metric(y_pred, y_true):\n",
    "        ypred = tf.convert_to_tensor(y_pred, dtype=tf.float32)\n",
    "        ytrue = tf.convert_to_tensor(y_true, dtype=tf.float32)\n",
    "        true_odds = K.abs(ytrue)\n",
    "        condition = (1./true_odds + th)\n",
    "        pr, co = profit_(ytrue, ypred, condition)\n",
    "        idx = ['home','draw','away']\n",
    "        profit = pd.Series(K.eval(pr), index=idx)\n",
    "        count = pd.Series(K.eval(co), index=idx)\n",
    "        if name:\n",
    "            profit.name = 'Profit ' + name\n",
    "            count.name = 'Count ' + name\n",
    "        return profit, count\n",
    "    return perf_metric\n",
    "\n",
    "def b_absolute_performance(th=0.9, name=None):\n",
    "    def abs_perf_metric(y_pred, y_true):\n",
    "        ypred = tf.convert_to_tensor(y_pred, dtype=tf.float32)\n",
    "        ytrue = tf.convert_to_tensor(y_true, dtype=tf.float32)\n",
    "        pr, co = profit_(ytrue, ypred, th)\n",
    "        idx = ['home','draw','away']\n",
    "        profit = pd.Series(K.eval(pr), index=idx)\n",
    "        count = pd.Series(K.eval(co), index=idx)\n",
    "        if name:\n",
    "            profit.name = '|Profit| ' + name\n",
    "            count.name = '|Count| ' + name\n",
    "        return profit, count\n",
    "    return abs_perf_metric"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2c577a76321f480b97b09dbd55b0b6eb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=200, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "if os.path.exists('logs/keras_metric'):\n",
    "    try:\n",
    "        shutil.rmtree('logs/keras_metric')\n",
    "    except OSError:\n",
    "        pass\n",
    "    \n",
    "model = construct_model(cat_loss, metrics=['accuracy'] + list(bet_metric(0.05)) + list(abs_bet_metric(0.9)))\n",
    "_ = model.fit(X_train, b_train,\n",
    "      epochs=200,\n",
    "      batch_size=500, verbose=0,\n",
    "      validation_data = [X_test, b_test],\n",
    "      callbacks=[keras.callbacks.TensorBoard(log_dir='./logs/keras_metric', write_graph=True), \n",
    "                 TQDMNotebookCallback(show_inner=False)]\n",
    " )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Profit cm</th>\n",
       "      <th>Count cm</th>\n",
       "      <th>|Profit| cm</th>\n",
       "      <th>|Count| cm</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>home</th>\n",
       "      <td>8.00</td>\n",
       "      <td>11.0</td>\n",
       "      <td>0.62</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>draw</th>\n",
       "      <td>-2.00</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>away</th>\n",
       "      <td>-0.84</td>\n",
       "      <td>13.0</td>\n",
       "      <td>-0.32</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Profit cm  Count cm  |Profit| cm  |Count| cm\n",
       "home       8.00      11.0         0.62         5.0\n",
       "draw      -2.00       2.0         0.00         0.0\n",
       "away      -0.84      13.0        -0.32         4.0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "cm = pd.concat([pd.concat(b_performance(0.05,name='cm')(model.predict(X_test), b_test.values),axis=1),\n",
    "pd.concat(b_absolute_performance(0.9,name='cm')(model.predict(X_test), b_test.values),axis=1)], axis=1)\n",
    "display(cm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Intermission\n",
    "\n",
    "With our `cat_loss`, `bet_metric` and `abs_bet_metric` we now have tools to better monitor "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "def clean_folds():\n",
    "    for i in range(100):\n",
    "        if os.path.exists('logs/fold_%02d' % i):\n",
    "            try:\n",
    "                shutil.rmtree('logs/fold_%02d' % i)\n",
    "            except (PermissionError, OSError):\n",
    "                pass\n",
    "clean_folds()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import KFold\n",
    "kfold = KFold(n_splits=20, random_state=42)\n",
    "\n",
    "def cross_validate(epochs=200, loss=cat_loss):\n",
    "    results = []\n",
    "    models = []\n",
    "\n",
    "    for i, (trainidx, valididx) in enumerate(kfold.split(df)):\n",
    "        train = df.iloc[trainidx]\n",
    "        test = df.iloc[valididx]\n",
    "        train_features = train.drop(columns=['result'])\n",
    "        train_labels = train.result.copy()\n",
    "        test_features = test.drop(columns=['result'])\n",
    "        test_labels = test.result.copy()\n",
    "        scaler = StandardScaler()\n",
    "        X_train = scaler.fit_transform(train_features.astype(float))\n",
    "        X_test = scaler.transform(test_features.astype(float))\n",
    "        encoder = LabelEncoder()\n",
    "        Y_train = -encoder.fit_transform(train_labels) +2\n",
    "        Y_test = -encoder.transform(test_labels) +2\n",
    "        y_train = np_utils.to_categorical(Y_train)\n",
    "        y_test = np_utils.to_categorical(Y_test)    \n",
    "        odds_train = train_features[['odds-home','odds-draw','odds-away']]\n",
    "        odds_test = test_features[['odds-home','odds-draw','odds-away']]\n",
    "        c_train = odds_train * (2 * y_train - 1)\n",
    "        c_test = odds_test * (2 * y_test - 1)\n",
    "\n",
    "        model = construct_model(loss, metrics=['accuracy'] + list(bet_metric(0.05)) + list(abs_bet_metric(0.9)))\n",
    "\n",
    "        _ = model.fit(X_train, c_train,\n",
    "              epochs=epochs,\n",
    "              batch_size=500, verbose=0,\n",
    "              validation_data = [X_test, c_test],\n",
    "              callbacks=[keras.callbacks.TensorBoard(log_dir='./logs/fold_%02d' % i, write_graph=True), \n",
    "                         TQDMNotebookCallback(show_inner=False)]\n",
    "         )    \n",
    "\n",
    "        cm = pd.concat([pd.concat(b_performance(0.05,name='cm')(model.predict(X_test), c_test.values),axis=1),\n",
    "        pd.concat(b_absolute_performance(0.9,name='cm')(model.predict(X_test), c_test.values),axis=1)], axis=1)\n",
    "        results.append(cm)\n",
    "        models.append(model)\n",
    "\n",
    "        display(cm)\n",
    "        \n",
    "    return results, models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e199b7d8b64649df8580aa9ca520fc51",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=200, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Profit cm</th>\n",
       "      <th>Count cm</th>\n",
       "      <th>|Profit| cm</th>\n",
       "      <th>|Count| cm</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>home</th>\n",
       "      <td>5.54</td>\n",
       "      <td>17.0</td>\n",
       "      <td>0.59</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>draw</th>\n",
       "      <td>6.15</td>\n",
       "      <td>7.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>away</th>\n",
       "      <td>0.46</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Profit cm  Count cm  |Profit| cm  |Count| cm\n",
       "home       5.54      17.0         0.59         4.0\n",
       "draw       6.15       7.0         0.00         0.0\n",
       "away       0.46       5.0         0.00         0.0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2c5b00647d244ed5b8e2b940b854b7c0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=200, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Profit cm</th>\n",
       "      <th>Count cm</th>\n",
       "      <th>|Profit| cm</th>\n",
       "      <th>|Count| cm</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>home</th>\n",
       "      <td>2.43</td>\n",
       "      <td>7.0</td>\n",
       "      <td>0.14</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>draw</th>\n",
       "      <td>5.30</td>\n",
       "      <td>16.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>away</th>\n",
       "      <td>0.30</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Profit cm  Count cm  |Profit| cm  |Count| cm\n",
       "home       2.43       7.0         0.14         1.0\n",
       "draw       5.30      16.0         0.00         0.0\n",
       "away       0.30       1.0         0.00         0.0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "97452fc2ef5241e8afcb2ad0b8e3a48a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=200, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Profit cm</th>\n",
       "      <th>Count cm</th>\n",
       "      <th>|Profit| cm</th>\n",
       "      <th>|Count| cm</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>home</th>\n",
       "      <td>2.25</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.13</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>draw</th>\n",
       "      <td>0.40</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>away</th>\n",
       "      <td>-1.57</td>\n",
       "      <td>7.0</td>\n",
       "      <td>-0.71</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Profit cm  Count cm  |Profit| cm  |Count| cm\n",
       "home       2.25       6.0         0.13         1.0\n",
       "draw       0.40       6.0         0.00         0.0\n",
       "away      -1.57       7.0        -0.71         2.0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e857e602b7bf41d8b4ed7c6996bebcd9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=200, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Profit cm</th>\n",
       "      <th>Count cm</th>\n",
       "      <th>|Profit| cm</th>\n",
       "      <th>|Count| cm</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>home</th>\n",
       "      <td>0.75</td>\n",
       "      <td>10.0</td>\n",
       "      <td>0.34</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>draw</th>\n",
       "      <td>9.35</td>\n",
       "      <td>11.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>away</th>\n",
       "      <td>4.52</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.22</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Profit cm  Count cm  |Profit| cm  |Count| cm\n",
       "home       0.75      10.0         0.34         3.0\n",
       "draw       9.35      11.0         0.00         0.0\n",
       "away       4.52       3.0         0.22         1.0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "01e033b63658487a8987f24dda0743ea",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=200, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Profit cm</th>\n",
       "      <th>Count cm</th>\n",
       "      <th>|Profit| cm</th>\n",
       "      <th>|Count| cm</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>home</th>\n",
       "      <td>6.93</td>\n",
       "      <td>12.0</td>\n",
       "      <td>0.13</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>draw</th>\n",
       "      <td>1.60</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>away</th>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Profit cm  Count cm  |Profit| cm  |Count| cm\n",
       "home       6.93      12.0         0.13         1.0\n",
       "draw       1.60       2.0         0.00         0.0\n",
       "away       0.00       0.0         0.00         0.0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "51ee1ad5aaf04252bcbbccffd95eebc9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=200, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Profit cm</th>\n",
       "      <th>Count cm</th>\n",
       "      <th>|Profit| cm</th>\n",
       "      <th>|Count| cm</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>home</th>\n",
       "      <td>2.7</td>\n",
       "      <td>11.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>draw</th>\n",
       "      <td>-4.7</td>\n",
       "      <td>8.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>away</th>\n",
       "      <td>1.8</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Profit cm  Count cm  |Profit| cm  |Count| cm\n",
       "home        2.7      11.0          0.0         0.0\n",
       "draw       -4.7       8.0          0.0         0.0\n",
       "away        1.8       4.0          0.0         0.0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "70d1e2d2d38d47c8b5b1a9bdbe2db112",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=200, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Profit cm</th>\n",
       "      <th>Count cm</th>\n",
       "      <th>|Profit| cm</th>\n",
       "      <th>|Count| cm</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>home</th>\n",
       "      <td>-4.17</td>\n",
       "      <td>15.0</td>\n",
       "      <td>-0.83</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>draw</th>\n",
       "      <td>-2.60</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>away</th>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Profit cm  Count cm  |Profit| cm  |Count| cm\n",
       "home      -4.17      15.0        -0.83         2.0\n",
       "draw      -2.60       6.0         0.00         0.0\n",
       "away       0.00       0.0         0.00         0.0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f28c98068b0c4e39a8e2167173589881",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=200, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Profit cm</th>\n",
       "      <th>Count cm</th>\n",
       "      <th>|Profit| cm</th>\n",
       "      <th>|Count| cm</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>home</th>\n",
       "      <td>3.01</td>\n",
       "      <td>12.0</td>\n",
       "      <td>0.26</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>draw</th>\n",
       "      <td>1.60</td>\n",
       "      <td>9.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>away</th>\n",
       "      <td>-1.78</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Profit cm  Count cm  |Profit| cm  |Count| cm\n",
       "home       3.01      12.0         0.26         2.0\n",
       "draw       1.60       9.0         0.00         0.0\n",
       "away      -1.78       3.0         0.00         0.0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "27d59a389e6a4e96b681e8f80af4c5e2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=200, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Profit cm</th>\n",
       "      <th>Count cm</th>\n",
       "      <th>|Profit| cm</th>\n",
       "      <th>|Count| cm</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>home</th>\n",
       "      <td>-5.6</td>\n",
       "      <td>14.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>draw</th>\n",
       "      <td>-1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>away</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Profit cm  Count cm  |Profit| cm  |Count| cm\n",
       "home       -5.6      14.0          0.0         0.0\n",
       "draw       -1.0       1.0          0.0         0.0\n",
       "away        0.0       0.0          0.0         0.0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "39e25f66191e4901a9ce3dbfe07f1c85",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=200, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Profit cm</th>\n",
       "      <th>Count cm</th>\n",
       "      <th>|Profit| cm</th>\n",
       "      <th>|Count| cm</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>home</th>\n",
       "      <td>0.45</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>draw</th>\n",
       "      <td>6.85</td>\n",
       "      <td>11.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>away</th>\n",
       "      <td>-1.00</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Profit cm  Count cm  |Profit| cm  |Count| cm\n",
       "home       0.45      20.0          0.0         0.0\n",
       "draw       6.85      11.0          0.0         0.0\n",
       "away      -1.00       1.0          0.0         0.0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ace1ff9c42264fe08110f8d38f4743c5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=200, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Profit cm</th>\n",
       "      <th>Count cm</th>\n",
       "      <th>|Profit| cm</th>\n",
       "      <th>|Count| cm</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>home</th>\n",
       "      <td>-3.01</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0.13</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>draw</th>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>away</th>\n",
       "      <td>-0.97</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Profit cm  Count cm  |Profit| cm  |Count| cm\n",
       "home      -3.01      20.0         0.13         1.0\n",
       "draw       0.00       0.0         0.00         0.0\n",
       "away      -0.97       4.0         0.00         0.0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8480614485234b6fbc2ef189f2709341",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=200, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Profit cm</th>\n",
       "      <th>Count cm</th>\n",
       "      <th>|Profit| cm</th>\n",
       "      <th>|Count| cm</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>home</th>\n",
       "      <td>-6.76</td>\n",
       "      <td>14.0</td>\n",
       "      <td>-0.56</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>draw</th>\n",
       "      <td>0.70</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>away</th>\n",
       "      <td>0.54</td>\n",
       "      <td>9.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Profit cm  Count cm  |Profit| cm  |Count| cm\n",
       "home      -6.76      14.0        -0.56         4.0\n",
       "draw       0.70       6.0         0.00         0.0\n",
       "away       0.54       9.0         0.00         0.0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ddf148a072e54d0fa36a85edec9d950f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=200, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Profit cm</th>\n",
       "      <th>Count cm</th>\n",
       "      <th>|Profit| cm</th>\n",
       "      <th>|Count| cm</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>home</th>\n",
       "      <td>0.47</td>\n",
       "      <td>7.0</td>\n",
       "      <td>0.17</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>draw</th>\n",
       "      <td>3.20</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>away</th>\n",
       "      <td>0.84</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Profit cm  Count cm  |Profit| cm  |Count| cm\n",
       "home       0.47       7.0         0.17         1.0\n",
       "draw       3.20       1.0         0.00         0.0\n",
       "away       0.84       2.0         0.00         0.0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a7885feac65a40798743e3b22da75ae2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=200, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Profit cm</th>\n",
       "      <th>Count cm</th>\n",
       "      <th>|Profit| cm</th>\n",
       "      <th>|Count| cm</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>home</th>\n",
       "      <td>2.11</td>\n",
       "      <td>14.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>draw</th>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>away</th>\n",
       "      <td>7.10</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Profit cm  Count cm  |Profit| cm  |Count| cm\n",
       "home       2.11      14.0          0.0         0.0\n",
       "draw       0.00       0.0          0.0         0.0\n",
       "away       7.10       3.0          0.0         0.0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4ec008ca38fb440b97b9e85b7db603c7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=200, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Profit cm</th>\n",
       "      <th>Count cm</th>\n",
       "      <th>|Profit| cm</th>\n",
       "      <th>|Count| cm</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>home</th>\n",
       "      <td>6.899999</td>\n",
       "      <td>22.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>draw</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>away</th>\n",
       "      <td>3.470000</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Profit cm  Count cm  |Profit| cm  |Count| cm\n",
       "home   6.899999      22.0          0.0         0.0\n",
       "draw   0.000000       0.0          0.0         0.0\n",
       "away   3.470000       3.0          0.0         0.0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "762869ccb42d4fc8b9bbdbeefc2b5565",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=200, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Profit cm</th>\n",
       "      <th>Count cm</th>\n",
       "      <th>|Profit| cm</th>\n",
       "      <th>|Count| cm</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>home</th>\n",
       "      <td>0.76</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>draw</th>\n",
       "      <td>3.75</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>away</th>\n",
       "      <td>0.95</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Profit cm  Count cm  |Profit| cm  |Count| cm\n",
       "home       0.76      24.0          0.0         0.0\n",
       "draw       3.75       1.0          0.0         0.0\n",
       "away       0.95       2.0          0.0         0.0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "deb44b80478842878a5b0f140dd6a891",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=200, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Profit cm</th>\n",
       "      <th>Count cm</th>\n",
       "      <th>|Profit| cm</th>\n",
       "      <th>|Count| cm</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>home</th>\n",
       "      <td>0.3</td>\n",
       "      <td>18.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>draw</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>away</th>\n",
       "      <td>3.6</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Profit cm  Count cm  |Profit| cm  |Count| cm\n",
       "home        0.3      18.0          0.0         0.0\n",
       "draw        0.0       0.0          0.0         0.0\n",
       "away        3.6       6.0          0.0         0.0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "05eec944e30144e48f3018ff15edec3d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=200, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Profit cm</th>\n",
       "      <th>Count cm</th>\n",
       "      <th>|Profit| cm</th>\n",
       "      <th>|Count| cm</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>home</th>\n",
       "      <td>-1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>draw</th>\n",
       "      <td>-2.5</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>away</th>\n",
       "      <td>0.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Profit cm  Count cm  |Profit| cm  |Count| cm\n",
       "home       -1.0       1.0          0.0         0.0\n",
       "draw       -2.5       6.0          0.0         0.0\n",
       "away        0.0      10.0          0.0         0.0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "eef46c3e66e44d219b73621ebc8fd516",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=200, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Profit cm</th>\n",
       "      <th>Count cm</th>\n",
       "      <th>|Profit| cm</th>\n",
       "      <th>|Count| cm</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>home</th>\n",
       "      <td>-2.92</td>\n",
       "      <td>15.0</td>\n",
       "      <td>0.9</td>\n",
       "      <td>7.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>draw</th>\n",
       "      <td>-3.00</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>away</th>\n",
       "      <td>-0.68</td>\n",
       "      <td>6.0</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Profit cm  Count cm  |Profit| cm  |Count| cm\n",
       "home      -2.92      15.0          0.9         7.0\n",
       "draw      -3.00       3.0          0.0         0.0\n",
       "away      -0.68       6.0         -1.0         1.0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a7ec948cbbbe43238d84865fc7f2e75d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training', max=200, style=ProgressStyle(description_width='in…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Profit cm</th>\n",
       "      <th>Count cm</th>\n",
       "      <th>|Profit| cm</th>\n",
       "      <th>|Count| cm</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>home</th>\n",
       "      <td>3.50</td>\n",
       "      <td>10.0</td>\n",
       "      <td>0.62</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>draw</th>\n",
       "      <td>-2.00</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>away</th>\n",
       "      <td>-2.29</td>\n",
       "      <td>16.0</td>\n",
       "      <td>-0.32</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Profit cm  Count cm  |Profit| cm  |Count| cm\n",
       "home       3.50      10.0         0.62         5.0\n",
       "draw      -2.00       2.0         0.00         0.0\n",
       "away      -2.29      16.0        -0.32         4.0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "results, models = cross_validate()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Examine the profit of our folded model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Profit cm</th>\n",
       "      <th>Count cm</th>\n",
       "      <th>|Profit| cm</th>\n",
       "      <th>|Count| cm</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>home</th>\n",
       "      <td>14.639999</td>\n",
       "      <td>269.0</td>\n",
       "      <td>2.02</td>\n",
       "      <td>32.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>draw</th>\n",
       "      <td>23.100000</td>\n",
       "      <td>96.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>away</th>\n",
       "      <td>15.290000</td>\n",
       "      <td>85.0</td>\n",
       "      <td>-1.81</td>\n",
       "      <td>8.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Profit cm  Count cm  |Profit| cm  |Count| cm\n",
       "home  14.639999     269.0         2.02        32.0\n",
       "draw  23.100000      96.0         0.00         0.0\n",
       "away  15.290000      85.0        -1.81         8.0"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result = pd.concat({n: df for n, df in enumerate(results)},axis=0)\n",
    "result.sum(level=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is a small \\(positive\\) difference with our \n",
    "[revamped](https://github.com/Gearlux/football-predictor/blob/master/02_revamped.ipynb)\n",
    "model.\n",
    "We now have a 11% margin for our relative model and a 0.5% margin for our absolute threshold.\n",
    "The difference with the [revamped](https://github.com/Gearlux/football-predictor/blob/master/02_revamped.ipynb)\n",
    "model are the number of epochs and the random seed.\n",
    "\n",
    "Maybe the evolution of our metrics gain some insight."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<tr>\n",
    "<td>\n",
    "    <img src=\"images/margin_folds.png\" width=\"400px\"></td>\n",
    "    <td><img src=\"images/profit_folds.png\" width=\"400px\"></td>\n",
    "</tr>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that margin for all our folds is increasing, but the profit stays more ore less constant.\n",
    "\n",
    "Inspection of the validation folds gives a diffent view. \n",
    "This might be an indication that we are overfitting the model or that our model is not capable of modeling\n",
    "the complex interactions for betting.\n",
    "\n",
    "Optimization of the network structure, will be investigated in a seperate project."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<tr>\n",
    "<td>\n",
    "    <img src=\"images/val_margin_folds.png\" width=\"400px\"></td>\n",
    "    <td><img src=\"images/val_profit_folds.png\" width=\"400px\"></td>\n",
    "</tr>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}