poc-py-fs-wrapper-pima-indians-diabetes.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Gerardo de Miguel González"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Feature Selection Proof of Concept"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**::GMG::** I've used the following *introductory* references:\n",
    "\n",
    "  - [DataCamp](https://www.datacamp.com/community/tutorials/feature-selection-python) Beginner's Guide to Feature Selection in Python. *Learn about the basics of feature selection and how to implement and investigate various feature selection techniques in Python*. Sayak Paul. September 25th, 2018.\n",
    "  - [Analytics Vidhya](https://www.analyticsvidhya.com/blog/2016/12/introduction-to-feature-selection-methods-with-an-example-or-how-to-select-the-right-variables/) Introduction to Feature Selection methods with an example (or how to select the right variables?). Saurav Kaushik, december 1, 2016.  \n",
    "  \n",
    "**::GMG::** There is also a more *in-deep* reference, a two part article by [Lahiru Liyanapathirana](https://towardsdatascience.com/@lahiru.tjay):\n",
    "\n",
    "  - Machine Learning Workflow on Diabetes Data : [Part 01](https://towardsdatascience.com/machine-learning-workflow-on-diabetes-data-part-01-573864fcc6b8)\n",
    "  - Machine Learning Workflow on Diabetes Data : [Part 02](https://towardsdatascience.com/machine-learning-workflow-on-diabetes-data-part-02-11262b7f7a5c)\n",
    "  \n",
    "where *cross-validation* techniques are used for _model selection_ and _feature selection_ within the workflow. I appreciate the fact that [the code from the article](https://github.com/LahiruTjay/Machine-Learning-With-Python/blob/master/Machine%20Learning%20Workflow%20on%20Diabetes%20Data.ipynb) is *available* in Github and I can copycat the workflow to learn it by heart and apply it to my projects.\n",
    "\n",
    "**::GMG::** Scikit-learn has also a wealth of material to read and learn:\n",
    "\n",
    "  - [Feature Selection](https://scikit-learn.org/stable/modules/feature_selection.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "#::GMG::General\n",
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "#::GMG::Visualization\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "#::GMG::Models\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.svm import SVC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "#::GMG::Train/Test and (Stratified) K-fold Split\n",
    "# https://scikit-learn.org/stable/modules/classes.html#module-sklearn.model_selection\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.model_selection import StratifiedKFold"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "#::GMG::wrapper feature selection standard and with cross vañidation\n",
    "from sklearn.feature_selection import RFE \n",
    "from sklearn.feature_selection import RFECV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "#::GMG::evaluation metrics\n",
    "from sklearn.metrics import confusion_matrix, classification_report\n",
    "from sklearn.metrics import balanced_accuracy_score, precision_score, recall_score, f1_score\n",
    "from sklearn.metrics import precision_recall_curve, auc, average_precision_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "#::GMG::Helper for precision-recall curve\n",
    "from inspect import signature"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### load"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "**::GMG::** You may [download the data from kaggle](https://www.kaggle.com/uciml/pima-indians-diabetes-database/downloads/pima-indians-diabetes-database.zip/1) in zipped csv format (which *includes a header* with the column names by the way) using [the reference provided](https://www.kaggle.com/uciml/pima-indians-diabetes-database) in the Datacamp article *if you have an account in kaggle*. I haven't checked it but you should be able [to use the kaggle API](https://medium.com/@yvettewu.dw/tutorial-kaggle-api-google-colaboratory-1a054a382de0) too to automate the download from code *with an API key* created with your account. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "**::GMG::** I've already downloaded *manually from kaggle* (with my account) the csv dataset and placed it in a data folder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pima-indians-diabetes.csv\r\n"
     ]
    }
   ],
   "source": [
    "#::GMG::Location of the downloaded dataset csv file\n",
    "!ls data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "#::GMG::Dataframe\n",
    "data = pd.read_csv(\"data/pima-indians-diabetes.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Pregnancies</th>\n",
       "      <th>Glucose</th>\n",
       "      <th>BloodPressure</th>\n",
       "      <th>SkinThickness</th>\n",
       "      <th>Insulin</th>\n",
       "      <th>BMI</th>\n",
       "      <th>DiabetesPedigreeFunction</th>\n",
       "      <th>Age</th>\n",
       "      <th>Outcome</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>6</td>\n",
       "      <td>148</td>\n",
       "      <td>72</td>\n",
       "      <td>35</td>\n",
       "      <td>0</td>\n",
       "      <td>33.6</td>\n",
       "      <td>0.627</td>\n",
       "      <td>50</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>85</td>\n",
       "      <td>66</td>\n",
       "      <td>29</td>\n",
       "      <td>0</td>\n",
       "      <td>26.6</td>\n",
       "      <td>0.351</td>\n",
       "      <td>31</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8</td>\n",
       "      <td>183</td>\n",
       "      <td>64</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>23.3</td>\n",
       "      <td>0.672</td>\n",
       "      <td>32</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>89</td>\n",
       "      <td>66</td>\n",
       "      <td>23</td>\n",
       "      <td>94</td>\n",
       "      <td>28.1</td>\n",
       "      <td>0.167</td>\n",
       "      <td>21</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>137</td>\n",
       "      <td>40</td>\n",
       "      <td>35</td>\n",
       "      <td>168</td>\n",
       "      <td>43.1</td>\n",
       "      <td>2.288</td>\n",
       "      <td>33</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Pregnancies  Glucose  BloodPressure  SkinThickness  Insulin   BMI  \\\n",
       "0            6      148             72             35        0  33.6   \n",
       "1            1       85             66             29        0  26.6   \n",
       "2            8      183             64              0        0  23.3   \n",
       "3            1       89             66             23       94  28.1   \n",
       "4            0      137             40             35      168  43.1   \n",
       "\n",
       "   DiabetesPedigreeFunction  Age  Outcome  \n",
       "0                     0.627   50        1  \n",
       "1                     0.351   31        0  \n",
       "2                     0.672   32        1  \n",
       "3                     0.167   21        0  \n",
       "4                     2.288   33        1  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#::GMG::Show some samples\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Pregnancies</th>\n",
       "      <th>Glucose</th>\n",
       "      <th>BloodPressure</th>\n",
       "      <th>SkinThickness</th>\n",
       "      <th>Insulin</th>\n",
       "      <th>BMI</th>\n",
       "      <th>DiabetesPedigreeFunction</th>\n",
       "      <th>Age</th>\n",
       "      <th>Outcome</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>763</th>\n",
       "      <td>10</td>\n",
       "      <td>101</td>\n",
       "      <td>76</td>\n",
       "      <td>48</td>\n",
       "      <td>180</td>\n",
       "      <td>32.9</td>\n",
       "      <td>0.171</td>\n",
       "      <td>63</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>764</th>\n",
       "      <td>2</td>\n",
       "      <td>122</td>\n",
       "      <td>70</td>\n",
       "      <td>27</td>\n",
       "      <td>0</td>\n",
       "      <td>36.8</td>\n",
       "      <td>0.340</td>\n",
       "      <td>27</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>765</th>\n",
       "      <td>5</td>\n",
       "      <td>121</td>\n",
       "      <td>72</td>\n",
       "      <td>23</td>\n",
       "      <td>112</td>\n",
       "      <td>26.2</td>\n",
       "      <td>0.245</td>\n",
       "      <td>30</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>766</th>\n",
       "      <td>1</td>\n",
       "      <td>126</td>\n",
       "      <td>60</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>30.1</td>\n",
       "      <td>0.349</td>\n",
       "      <td>47</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>767</th>\n",
       "      <td>1</td>\n",
       "      <td>93</td>\n",
       "      <td>70</td>\n",
       "      <td>31</td>\n",
       "      <td>0</td>\n",
       "      <td>30.4</td>\n",
       "      <td>0.315</td>\n",
       "      <td>23</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     Pregnancies  Glucose  BloodPressure  SkinThickness  Insulin   BMI  \\\n",
       "763           10      101             76             48      180  32.9   \n",
       "764            2      122             70             27        0  36.8   \n",
       "765            5      121             72             23      112  26.2   \n",
       "766            1      126             60              0        0  30.1   \n",
       "767            1       93             70             31        0  30.4   \n",
       "\n",
       "     DiabetesPedigreeFunction  Age  Outcome  \n",
       "763                     0.171   63        0  \n",
       "764                     0.340   27        0  \n",
       "765                     0.245   30        0  \n",
       "766                     0.349   47        1  \n",
       "767                     0.315   23        0  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#::GMG::Show some samples\n",
    "data.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Pregnancies</th>\n",
       "      <th>Glucose</th>\n",
       "      <th>BloodPressure</th>\n",
       "      <th>SkinThickness</th>\n",
       "      <th>Insulin</th>\n",
       "      <th>BMI</th>\n",
       "      <th>DiabetesPedigreeFunction</th>\n",
       "      <th>Age</th>\n",
       "      <th>Outcome</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>3.845052</td>\n",
       "      <td>120.894531</td>\n",
       "      <td>69.105469</td>\n",
       "      <td>20.536458</td>\n",
       "      <td>79.799479</td>\n",
       "      <td>31.992578</td>\n",
       "      <td>0.471876</td>\n",
       "      <td>33.240885</td>\n",
       "      <td>0.348958</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>3.369578</td>\n",
       "      <td>31.972618</td>\n",
       "      <td>19.355807</td>\n",
       "      <td>15.952218</td>\n",
       "      <td>115.244002</td>\n",
       "      <td>7.884160</td>\n",
       "      <td>0.331329</td>\n",
       "      <td>11.760232</td>\n",
       "      <td>0.476951</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.078000</td>\n",
       "      <td>21.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>99.000000</td>\n",
       "      <td>62.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>27.300000</td>\n",
       "      <td>0.243750</td>\n",
       "      <td>24.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>3.000000</td>\n",
       "      <td>117.000000</td>\n",
       "      <td>72.000000</td>\n",
       "      <td>23.000000</td>\n",
       "      <td>30.500000</td>\n",
       "      <td>32.000000</td>\n",
       "      <td>0.372500</td>\n",
       "      <td>29.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>6.000000</td>\n",
       "      <td>140.250000</td>\n",
       "      <td>80.000000</td>\n",
       "      <td>32.000000</td>\n",
       "      <td>127.250000</td>\n",
       "      <td>36.600000</td>\n",
       "      <td>0.626250</td>\n",
       "      <td>41.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>17.000000</td>\n",
       "      <td>199.000000</td>\n",
       "      <td>122.000000</td>\n",
       "      <td>99.000000</td>\n",
       "      <td>846.000000</td>\n",
       "      <td>67.100000</td>\n",
       "      <td>2.420000</td>\n",
       "      <td>81.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Pregnancies     Glucose  BloodPressure  SkinThickness     Insulin  \\\n",
       "count   768.000000  768.000000     768.000000     768.000000  768.000000   \n",
       "mean      3.845052  120.894531      69.105469      20.536458   79.799479   \n",
       "std       3.369578   31.972618      19.355807      15.952218  115.244002   \n",
       "min       0.000000    0.000000       0.000000       0.000000    0.000000   \n",
       "25%       1.000000   99.000000      62.000000       0.000000    0.000000   \n",
       "50%       3.000000  117.000000      72.000000      23.000000   30.500000   \n",
       "75%       6.000000  140.250000      80.000000      32.000000  127.250000   \n",
       "max      17.000000  199.000000     122.000000      99.000000  846.000000   \n",
       "\n",
       "              BMI  DiabetesPedigreeFunction         Age     Outcome  \n",
       "count  768.000000                768.000000  768.000000  768.000000  \n",
       "mean    31.992578                  0.471876   33.240885    0.348958  \n",
       "std      7.884160                  0.331329   11.760232    0.476951  \n",
       "min      0.000000                  0.078000   21.000000    0.000000  \n",
       "25%     27.300000                  0.243750   24.000000    0.000000  \n",
       "50%     32.000000                  0.372500   29.000000    0.000000  \n",
       "75%     36.600000                  0.626250   41.000000    1.000000  \n",
       "max     67.100000                  2.420000   81.000000    1.000000  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#::GMG::Get some statistics of the features and classification variable\n",
    "data.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "**::GMG::** To let the dataset ready to work on, I must do some preparatory tasks first. These tasks are related to the fact that there are some *dirty* features, i.e `Glucose`, `Insulin`, `BloodPressure`, `SkinThickness`, and `BMI` which have *zeros* and those are not valid values (*outliers*)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Zero Plasma Glucose  Total :  5\n",
      "Zero Insulin level   Total :  374\n",
      "Zero Blood Pressure  Total :  35\n",
      "Zero Skin Thickness  Total :  227\n",
      "Zero Body Mass Index Total :  11\n"
     ]
    }
   ],
   "source": [
    "print(\"Zero Plasma Glucose  Total : \", data[data.Glucose == 0].shape[0])\n",
    "print(\"Zero Insulin level   Total : \", data[data.Insulin == 0].shape[0])\n",
    "print(\"Zero Blood Pressure  Total : \", data[data.BloodPressure == 0].shape[0])\n",
    "print(\"Zero Skin Thickness  Total : \", data[data.SkinThickness == 0].shape[0])\n",
    "print(\"Zero Body Mass Index Total : \", data[data.BMI == 0].shape[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "**::GMG::** Some of the outliers may somehow safely *removed*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data shape after removing outliers: (724, 9)\n"
     ]
    }
   ],
   "source": [
    "#::GMG::Hard choice to make ... remove the rows where the “BloodPressure”, \n",
    "#       “BMI” and “Glucose” are zero (small number of rows, low impact)\n",
    "data_mod = data[(data.BloodPressure != 0) & \n",
    "                (data.BMI != 0) & \n",
    "                (data.Glucose != 0)\n",
    "               ]\n",
    "print('Data shape after removing outliers: {}'.format(data_mod.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "**::GMG::** The problem with `Insulin` and `Skin Thickness` is that they are a *significant part* of the dataset. They cannot be removed. I may use an *imputation method* to fill in the zeros, i.e. put *average/mean* values, but in this case putting a mean value to the blood pressure column would send a *wrong signal* to the model. I may also avoid using the features involved if they are not *relevant* or are *redundant*. Feature selection by filtering might come in handy and help with that. But the filter is going to use the outliered feature ... so I'm in doubt here!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Outcome                     1.000000\n",
       "Glucose                     0.488384\n",
       "BMI                         0.299375\n",
       "Age                         0.245741\n",
       "Pregnancies                 0.224417\n",
       "DiabetesPedigreeFunction    0.184947\n",
       "BloodPressure               0.166703\n",
       "Insulin                     0.145488\n",
       "SkinThickness               0.092030\n",
       "Name: Outcome, dtype: float64"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#::GMG::Some basic (Pearson) correlations with Outcome\n",
    "# https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.corr.html\n",
    "correlation = data_mod.corr()\n",
    "correlation['Outcome'].sort_values(ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "**::GMG::** Based on this basic analysis I might avoid using `Skin Thickness` but not `Insulin`. But I'm going to *keep the features as they are* at the moment and let the wrapper decide."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Outcome\n",
       "0    475\n",
       "1    249\n",
       "dtype: int64"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#::GMG::Beware of the imbalance dataset ...\n",
    "#       https://www.quora.com/What-is-an-imbalanced-dataset\n",
    "#       https://www.datacamp.com/community/tutorials/diving-deep-imbalanced-data\n",
    "#       https://towardsdatascience.com/methods-for-dealing-with-imbalanced-data-5b761be45a18\n",
    "#       How do I check?\n",
    "# https://towardsdatascience.com/machine-learning-workflow-on-diabetes-data-part-01-573864fcc6b8\n",
    "data_mod.groupby('Outcome').size()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0, 0.5, 'Frequency %')"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAFPxJREFUeJzt3Xm0XnV97/H3R+YIBZEYaRACgnjRymAuLR28VeqtLa2gbakTppZlxKEOUAt1eaurq3rxXhFsxWKUIVhlFISKEyCKLqkSkBYEvSCGmggkDJFBZfzeP559ug5ZOec8CeznCef3fq31rLP3b0/f56zk+Zzfb+9n71QVkqR2PWXcBUiSxssgkKTGGQSS1DiDQJIaZxBIUuMMAklqnEGgjVqS9yf5l3HXIc1mBoHGLslrkixLcl+SW5N8Kclvj6mWSnJ/V8t9SdaMow5plAwCjVWSI4ETgA8C84CdgY8DB4+xrL2rauvutd26Vkiy6aiLkvpiEGhskmwL/D3w1qo6r6rur6qHqupfq+rdU2xzTpLbkvwsyeVJnjdp2R8muT7JvUlWJvnrrn2HJF9IsibJXUm+mWS9/u0n+d0kK5IcneQ24NSu/Y+SXNPt+9tJXjBpm32TXN3Vc1aSM5P8Q7fsL5J8a61jVJLdu+ktknw4yX8muT3JSUm2WquWo5Ks6npRb5i0n62SHJfklu739K2u7aIkf7XWMf8jySvW53eh2ccg0DgdAGwJnL8e23wJ2AN4BnA18JlJy04G3lRV2wDPB77WtR8FrADmMuh1vAfYkHurPBPYHtgFWJxkX+AU4E3A04FPABd2H+KbA58HPt1tcw7wJ+txrGOB5wD7ALsD84G/W6uWbbv2w4ETkzytW/Zh4IXAb3bH/hvgUWAp8LqJHSTZu9v+ovWoS7OQQaBxejpwR1U9POwGVXVKVd1bVQ8A7wf27noWAA8BeyX5laq6u6quntS+I7BL1+P4Zk1/k62ru7/w1yT5x0ntjwLvq6oHquoXwGLgE1X1nap6pKqWAg8Av9G9NgNO6I55LnDlMO8xSbp9v6uq7qqqexkMnb1q0moPAX/f7fuLwH3Anl1P5y+Bd1TVyq6ub3e/rwuB5yTZo9vHYcBZVfXgMHVp9jIINE53AjsMO96eZJMkxyb5UZJ7gOXdoh26n38C/CFwS5JvJDmga/+/wE3AV5PcnOSYGQ61X1Vt173ePql9dVX9ctL8LsBRk0JjDfAs4Fe718q1AueWYd4ng57LHOCqSfv9ctc+4c61AvTnwNYMfhdbAj9ae6dd7WcBr+sC49UMeixqnEGgcbqCwV/Qhwy5/msYnET+PQbDIgu69gBU1ZVVdTCDYaPPA2d37fdW1VFVtRvwcuDIJAduQL1r9yJ+AnxgUmhsV1VzquoM4FZgfvfX/YSdJ03fz+DDfvAGkmdOWnYH8AvgeZP2u21VbT1EjXcAvwSePcXypcBrgQOBn1fVFUPsU7OcQaCxqaqfMRj3PjHJIUnmJNksyR8k+T/r2GQbBsFxJ4MP0Q9OLEiyeZLXJtm2qh4C7mEwlDNxQnf37kP5Z8AjE8sep08CRyT59Qw8NclBSbZhEHIPA2/v3tMrgf0nbfvvwPOS7JNkSwbDXBO/l0e7fR+f5Bnde5if5PdnKqjb9hTgI0l+tetFHZBki275Fd17Pw57A+oYBBqrqjoOOBJ4L7CawV/Zb2PwF/3aTmcwvLISuB74t7WWHwYs74aNjmDwly8MTi5fwmAc/Qrg41V12RNQ+zLgjcDHgLsZDD/9RbfsQeCV3fxdwJ8D503a9v8xuGLqEuBG4DFXEAFHd/v7t+79XALsOWRpfw1cy+CcxF3Ah3js//XTgV8D/KKeAIgPppFGI8lpwIqqeu+Y63g9sLiqxvKlPW187BFIDUkyB3gLsGTctWjjYRBIjejOMawGbgc+O+ZytBFxaEiSGmePQJIa96S4cdYOO+xQCxYsGHcZkvSkctVVV91RVXNnWu9JEQQLFixg2bJl4y5Dkp5Ukgz1bXaHhiSpcQaBJDXOIJCkxhkEktQ4g0CSGmcQSFLjDAJJapxBIEmNMwgkqXFPim8WPx4Ljrlo3CVoI7X82IPGXYK0UbBHIEmNMwgkqXEGgSQ1ziCQpMYZBJLUOINAkhpnEEhS4wwCSWqcQSBJjTMIJKlxBoEkNc4gkKTGGQSS1LhegyDJdknOTfKDJDckOSDJ9kkuTnJj9/NpfdYgSZpe3z2CjwJfrqrnAnsDNwDHAJdW1R7Apd28JGlMeguCJNsCLwJOBqiqB6tqDXAwsLRbbSlwSF81SJJm1mePYFdgNXBqku8l+VSSpwLzqurWbp3bgHnr2jjJ4iTLkixbvXp1j2VKUtv6DIJNgf2Af66qfYH7WWsYqKoKqHVtXFVLqmphVS2cO3duj2VKUtv6DIIVwIqq+k43fy6DYLg9yY4A3c9VPdYgSZpBb0FQVbcBP0myZ9d0IHA9cCGwqGtbBFzQVw2SpJn1/fD6vwI+k2Rz4GbgDQzC5+wkhwO3AIf2XIMkaRq9BkFVXQMsXMeiA/s8riRpeH6zWJIaZxBIUuMMAklqnEEgSY0zCCSpcQaBJDXOIJCkxhkEktQ4g0CSGmcQSFLjDAJJapxBIEmNMwgkqXEGgSQ1ziCQpMYZBJLUOINAkhpnEEhS4wwCSWqcQSBJjTMIJKlxBoEkNc4gkKTGbdrnzpMsB+4FHgEerqqFSbYHzgIWAMuBQ6vq7j7rkCRNbRQ9ghdX1T5VtbCbPwa4tKr2AC7t5iVJYzKOoaGDgaXd9FLgkDHUIEnq9B0EBXw1yVVJFndt86rq1m76NmDeujZMsjjJsiTLVq9e3XOZktSuXs8RAL9dVSuTPAO4OMkPJi+sqkpS69qwqpYASwAWLly4znUkSY9frz2CqlrZ/VwFnA/sD9yeZEeA7ueqPmuQJE2vtyBI8tQk20xMA/8TuA64EFjUrbYIuKCvGiRJM+tzaGgecH6SieN8tqq+nORK4OwkhwO3AIf2WIMkaQa9BUFV3QzsvY72O4ED+zquJGn9+M1iSWqcQSBJjTMIJKlxBoEkNc4gkKTGGQSS1DiDQJIaZxBIUuMMAklqnEEgSY0zCCSpcQaBJDVu6JvOJZkLvAPYCjipqm7srSpJ0sisT4/gOOArDB4w89l+ypEkjdqUQZDkK0leNKlpc2B599qi37IkSaMyXY/gUOCPk5yR5NnA/wL+N/BR4C2jKE6S1L8pzxFU1c+AdyfZDfgA8FPgbVW1ZlTFSZL6N2UQdL2ANwMPAkcBzwbOSnIRcGJVPTKaEiVJfZpuaOgM4DzgMuDTVfXNqvp9YA3w1VEUJ0nq33SXj24B/BjYGpgz0VhVpyc5p+/CJEmjMV0QvBn4GIOhoSMmL6iqX/RZlCRpdKY7Wfxt4NsjrEWSNAa932IiySZJvpfkC938rkm+k+SmJGcl2bzvGiRJUxvFvYbeAdwwaf5DwPFVtTtwN3D4CGqQJE1hxiBI8msbuvMkOwEHAZ/q5gO8BDi3W2UpcMiG7l+S9PgN0yP4eJLvJnlLkm3Xc/8nAH8DPNrNPx1YU1UPd/MrgPnruU9J0hNoxiCoqt8BXgs8C7gqyWeTvHSm7ZL8EbCqqq7akMKSLE6yLMmy1atXb8guJElDGOocQXfL6fcCRwP/A/jHJD9I8sppNvst4OVJlgNnMhgS+iiwXZKJq5V2AlZOccwlVbWwqhbOnTt3qDcjSVp/w5wjeEGS4xmc8H0J8MdV9d+66eOn2q6q/raqdqqqBcCrgK9V1WsZfFP5T7vVFgEXPL63IEl6PIbpEfwTcDWwd1W9taquBqiqnzLoJayvo4Ejk9zE4JzByRuwD0nSE2SYJ5QdBPxi4iZzSZ4CbFlVP6+qTw9zkKr6OvD1bvpmYP8NqlaS9IQbpkdwCYPHU06Y07VJkmaBYYJgy6q6b2Kmm54zzfqSpCeRYYLg/iT7TcwkeSHgTeckaZYY5hzBO4FzkvwUCPBM4M97rUqSNDIzBkFVXZnkucCeXdMPq+qhfsuSJI3KMD0CgP8OLOjW3y8JVXV6b1VJkkZmxiBI8mkGzyu+Bph4TnEBBoEkzQLD9AgWAntVVfVdjCRp9Ia5aug6BieIJUmz0DA9gh2A65N8F3hgorGqXt5bVZKkkRkmCN7fdxGSpPEZ5vLRbyTZBdijqi5JMgfYpP/SJEmjMMxVQ28EFgPbM7h6aD5wEnBgv6VJbVhwzEXjLkEbqeXHHjSS4wxzsvitDB4ycw/810NqntFnUZKk0RkmCB6oqgcnZrqni3kpqSTNEsMEwTeSvAfYqntW8TnAv/ZbliRpVIYJgmOA1cC1wJuAL7JhTyaTJG2Ehrlq6FHgk91LkjTLDHPV0I9ZxzmBqtqtl4okSSM17L2GJmwJ/BmDS0klSbPAjOcIqurOSa+VVXUCgwfaS5JmgWGGhvabNPsUBj2EYZ9jIEnayA3zgX7cpOmHgeXAob1UI0kauWGuGnrxhuw4yZbA5cAW3XHOrar3JdkVOBN4OnAVcNjkL6xJkkZrmKGhI6dbXlUfmWLRA8BLquq+JJsB30ryJeBI4PiqOjPJScDhwD+vZ92SpCfIMF8oWwi8mcHN5uYDRwD7Adt0r3Wqgfu62c26VwEvAc7t2pcCh2xQ5ZKkJ8Qw5wh2AvarqnsBkrwfuKiqXjfThkk2YTD8sztwIvAjYE1VPdytsoJBuKxr28UM7nrKzjvvPESZkqQNMUyPYB4weQz/wa5tRlX1SFXtwyBM9geeO2xhVbWkqhZW1cK5c+cOu5kkaT0N0yM4HfhukvO7+UMYDOkMrarWJLkMOADYLsmmXa9gJ2Dl+uxLkvTEGuYLZR8A3gDc3b3eUFUfnGm7JHOTbNdNbwW8FLgBuAz40261RcAFG1a6JOmJMOwXw+YA91TVqd0H/K5V9eMZttkRWNqdJ3gKcHZVfSHJ9cCZSf4B+B5w8gZXL0l63Ia5fPR9DK4c2hM4lcHVP//C4KllU6qq/wD2XUf7zQzOF0iSNgLDnCx+BfBy4H6Aqvop01w2Kkl6chkmCB6sqqK7FXWSp/ZbkiRplIYJgrOTfILB1T5vBC7Bh9RI0qwxzL2GPtw9q/geBucJ/q6qLu69MknSSEwbBN0VP5d0N57zw1+SZqFph4aq6hHg0STbjqgeSdKIDfM9gvuAa5NcTHflEEBVvb23qiRJIzNMEJzXvSRJs9CUQZBk56r6z6par/sKSZKeXKY7R/D5iYkknxtBLZKkMZguCDJpere+C5Ekjcd0QVBTTEuSZpHpThbvneQeBj2Drbppuvmqql/pvTpJUu+mDIKq2mSUhUiSxmOYew1JkmYxg0CSGmcQSFLjDAJJapxBIEmNMwgkqXEGgSQ1ziCQpMYZBJLUuN6CIMmzklyW5Pok30/yjq59+yQXJ7mx+/m0vmqQJM2szx7Bw8BRVbUX8BvAW5PsBRwDXFpVewCXdvOSpDHpLQiq6taqurqbvhe4AZgPHAxMPOxmKXBIXzVIkmY2knMESRYA+wLfAeZV1a3dotuAeVNsszjJsiTLVq9ePYoyJalJvQdBkq2BzwHvrKp7Ji+rqmKKZx1U1ZKqWlhVC+fOndt3mZLUrF6DIMlmDELgM1V1Xtd8e5Idu+U7Aqv6rEGSNL0+rxoKcDJwQ1V9ZNKiC4FF3fQi4IK+apAkzWy6J5Q9Xr8FHAZcm+Saru09wLHA2UkOB24BDu2xBknSDHoLgqr6FoPHWq7LgX0dV5K0fvxmsSQ1ziCQpMYZBJLUOINAkhpnEEhS4wwCSWqcQSBJjTMIJKlxBoEkNc4gkKTGGQSS1DiDQJIaZxBIUuMMAklqnEEgSY0zCCSpcQaBJDXOIJCkxhkEktQ4g0CSGmcQSFLjDAJJalxvQZDklCSrklw3qW37JBcnubH7+bS+ji9JGk6fPYLTgJet1XYMcGlV7QFc2s1LksaotyCoqsuBu9ZqPhhY2k0vBQ7p6/iSpOGM+hzBvKq6tZu+DZg31YpJFidZlmTZ6tWrR1OdJDVobCeLq6qAmmb5kqpaWFUL586dO8LKJKktow6C25PsCND9XDXi40uS1jLqILgQWNRNLwIuGPHxJUlr6fPy0TOAK4A9k6xIcjhwLPDSJDcCv9fNS5LGaNO+dlxVr55i0YF9HVOStP78ZrEkNc4gkKTGGQSS1DiDQJIaZxBIUuMMAklqnEEgSY0zCCSpcQaBJDXOIJCkxhkEktQ4g0CSGmcQSFLjDAJJapxBIEmNMwgkqXEGgSQ1ziCQpMYZBJLUOINAkhpnEEhS4wwCSWqcQSBJjRtLECR5WZIfJrkpyTHjqEGSNDDyIEiyCXAi8AfAXsCrk+w16jokSQPj6BHsD9xUVTdX1YPAmcDBY6hDkgRsOoZjzgd+Mml+BfDra6+UZDGwuJu9L8kPR1BbC3YA7hh3ERuDfGjcFWgK/hvtPAH/RncZZqVxBMFQqmoJsGTcdcw2SZZV1cJx1yFNxX+jozeOoaGVwLMmze/UtUmSxmAcQXAlsEeSXZNsDrwKuHAMdUiSGMPQUFU9nORtwFeATYBTqur7o66jYQ63aWPnv9ERS1WNuwZJ0hj5zWJJapxBIEmNMwga4W09tLFLckqSVUmuG3ctrTEIGuBtPfQkcRrwsnEX0SKDoA3e1kMbvaq6HLhr3HW0yCBow7pu6zF/TLVI2sgYBJLUOIOgDd7WQ9KUDII2eFsPSVMyCBpQVQ8DE7f1uAE429t6aGOT5AzgCmDPJCuSHD7umlrhLSYkqXH2CCSpcQaBJDXOIJCkxhkEktQ4g0CSGmcQSGtJ8swkZyb5UZKrknwxyXO8K6Zmq5E/qlLamCUJcD6wtKpe1bXtDcwba2FSj+wRSI/1YuChqjppoqGq/p1JN+1LsiDJN5Nc3b1+s2vfMcnlSa5Jcl2S30mySZLTuvlrk7xr9G9Jmp49Aumxng9cNcM6q4CXVtUvk+wBnAEsBF4DfKWqPtA9A2IOsA8wv6qeD5Bku/5KlzaMQSCtv82AjyXZB3gEeE7XfiVwSpLNgM9X1TVJbgZ2S/JPwEXAV8dSsTQNh4akx/o+8MIZ1nkXcDuwN4OewObwXw9WeRGDO7ueluT1VXV3t97XgSOAT/VTtrThDALpsb4GbJFk8URDkhfw2Nt4bwvcWlWPAocBm3Tr7QLcXlWfZPCBv1+SHYCnVNXngPcC+43mbUjDc2hImqSqKskrgBOSHA38ElgOvHPSah8HPpfk9cCXgfu79t8F3p3kIeA+4PUMngR3apKJP7r+tvc3Ia0n7z4qSY1zaEiSGmcQSFLjDAJJapxBIEmNMwgkqXEGgSQ1ziCQpMb9f9TouUBlJZApAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#::GMG::Say, 0 healthy, 1 diabetes\n",
    "# https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.chararray.astype.html\n",
    "# https://matplotlib.org/gallery/lines_bars_and_markers/categorical_variables.html\n",
    "classes = data_mod.Outcome.values\n",
    "unique, counts = np.unique(classes, return_counts=True)\n",
    "plt.bar(unique.astype(np.character),100*counts/classes.size)\n",
    "plt.title('Class Frequency')\n",
    "plt.xlabel('Class')\n",
    "plt.ylabel('Frequency %')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Wrapper: Recursive Feature Elimination (RFE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "**::GMG::** The Recursive Feature Elimination (or RFE) works by *recursively removing attributes* and *building a model* on those attributes that remain. It uses the *model accuracy* to identify which attributes (and combination of attributes) *contribute the most* to predicting the target attribute. See [scikit-learn documentation](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.RFE.html). There's also a [user guide](https://scikit-learn.org/stable/modules/feature_selection.html#rfe) and two examples:\n",
    "\n",
    " - [RFE on pixels](https://scikit-learn.org/stable/auto_examples/feature_selection/plot_rfe_digits.html)\n",
    " - [Recursive feature elimination with cross-validation](https://scikit-learn.org/stable/auto_examples/feature_selection/plot_rfe_with_cross_validation.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### feature selection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "#::GMG::Feature extraction wrapper method from Datacamp reference\n",
    "#       Logistic Regression classifier as accuracy estimator to select the top 3 features\n",
    "#       The “balanced” mode uses the values of y to automatically adjust weights inversely \n",
    "#       proportional to class frequencies in the input data as \n",
    "#       n_samples / (n_classes * np.bincount(y))\n",
    "#\n",
    "# https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html\n",
    "#::NOTE::Why three?\n",
    "model = LogisticRegression(solver = 'liblinear', \n",
    "                           class_weight = 'balanced', \n",
    "                           random_state = 1)\n",
    "rfe = RFE(estimator = model, n_features_to_select = 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "#::GMG::70/30 split, this is not done in the datacamp tutorial\n",
    "#::NOTE::I know the dataset is imbalanced so I use stratify\n",
    "train, test = train_test_split(data_mod, \n",
    "                               test_size = 0.3,\n",
    "                               stratify = data_mod.Outcome,\n",
    "                               random_state = 27)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "#::GMG::Split features/target in train and test\n",
    "train_features = train.iloc[:,:train.shape[1] - 1]\n",
    "train_target = train['Outcome']\n",
    "test_features = test.iloc[:,:test.shape[1] - 1]\n",
    "test_target = test['Outcome']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(506, 8)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_features.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(218, 8)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_features.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RFE(estimator=LogisticRegression(C=1.0, class_weight='balanced', dual=False,\n",
       "          fit_intercept=True, intercept_scaling=1, max_iter=100,\n",
       "          multi_class='warn', n_jobs=None, penalty='l2', random_state=1,\n",
       "          solver='liblinear', tol=0.0001, verbose=0, warm_start=False),\n",
       "  n_features_to_select=3, step=1, verbose=0)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#::GMG::Fit the RFE model and then the underlying estimator on the selectedvfeatures\n",
    "rfe.fit(X = train_features, y = train_target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Num Features: 3\n"
     ]
    }
   ],
   "source": [
    "#::GMG::no suprise in the number of selected features :)\n",
    "print(\"Num Features: %s\" % (rfe.n_features_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ True, False, False, False, False,  True,  True, False])"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#::GMG::Getting to kow the ropes ...\n",
    "rfe.get_support()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Features: \n",
      "[('Pregnancies', True, 1), ('Glucose', False, 2), ('BloodPressure', False, 3), ('SkinThickness', False, 5), ('Insulin', False, 6), ('BMI', True, 1), ('DiabetesPedigreeFunction', True, 1), ('Age', False, 4)] \n",
      "\n"
     ]
    }
   ],
   "source": [
    "#::GMG::Understanding how selection is made using support and ranking\n",
    "print(\"Features: \\n%s \\n\" % \n",
    "      [values for values in zip(data.columns, \n",
    "                                rfe.support_, \n",
    "                                rfe.ranking_)\n",
    "      ]\n",
    "     )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Selected Features: ['Pregnancies', 'BMI', 'DiabetesPedigreeFunction']\n"
     ]
    }
   ],
   "source": [
    "#::GMG::How to get the selected features in a pythonic way ...\n",
    "print(\"Selected Features: %s\" % \n",
    "      [value for value, supported in zip(data.columns, rfe.support_) if supported]\n",
    "     )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "**::GMG::** RFE chose the top 3 features as `Pregnancies`, `BMI`, and `DiabetesPedigreeFunction`. These are *marked True* in the support array and *marked with a choice “1”* in the ranking array. This, in turn, indicates *the strength* of these features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight='balanced', dual=False,\n",
       "          fit_intercept=True, intercept_scaling=1, max_iter=100,\n",
       "          multi_class='warn', n_jobs=None, penalty='l2', random_state=1,\n",
       "          solver='liblinear', tol=0.0001, verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#::GMG::This is the estimator/model trained (fit) with selected features\n",
    "# ::NOTE::Should I optimize its hyperparameters like C, penalty, ... over CV k-folds?\n",
    "rfe.estimator_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'estimator__C': 1.0,\n",
       " 'estimator__class_weight': 'balanced',\n",
       " 'estimator__dual': False,\n",
       " 'estimator__fit_intercept': True,\n",
       " 'estimator__intercept_scaling': 1,\n",
       " 'estimator__max_iter': 100,\n",
       " 'estimator__multi_class': 'warn',\n",
       " 'estimator__n_jobs': None,\n",
       " 'estimator__penalty': 'l2',\n",
       " 'estimator__random_state': 1,\n",
       " 'estimator__solver': 'liblinear',\n",
       " 'estimator__tol': 0.0001,\n",
       " 'estimator__verbose': 0,\n",
       " 'estimator__warm_start': False,\n",
       " 'estimator': LogisticRegression(C=1.0, class_weight='balanced', dual=False,\n",
       "           fit_intercept=True, intercept_scaling=1, max_iter=100,\n",
       "           multi_class='warn', n_jobs=None, penalty='l2', random_state=1,\n",
       "           solver='liblinear', tol=0.0001, verbose=0, warm_start=False),\n",
       " 'n_features_to_select': 3,\n",
       " 'step': 1,\n",
       " 'verbose': 0}"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#::GMG::All the parameters used\n",
    "# https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.RFE.html#sklearn.feature_selection.RFE.get_params\n",
    "rfe.get_params()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### prediction and metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "#::GMG::Reduce X to the selected features and then predict using the underlying estimator\n",
    "# https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.RFE.html#sklearn.feature_selection.RFE.predict\n",
    "pred = rfe.predict(X = test_features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Score Train: 0.658 \n",
      "Score Test: 0.706\n"
     ]
    }
   ],
   "source": [
    "#::GMG::Basic Scores\n",
    "print('Score Train: {:4.3f} \\nScore Test: {:4.3f}'.format(\n",
    "    rfe.score(X = train_features, y = train_target),\n",
    "    rfe.score(X = test_features, y = test_target)\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[107  36]\n",
      " [ 28  47]]\n"
     ]
    }
   ],
   "source": [
    "#::GMG::Get the confusion matrix\n",
    "# https://en.wikipedia.org/wiki/Confusion_matrix\n",
    "# # https://scikit-learn.org/stable/modules/model_evaluation.html#confusion-matrix\n",
    "cm_rfe = confusion_matrix(y_true = test_target, y_pred = pred)\n",
    "print(cm_rfe)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(107, 36, 28, 47)"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#::GMG::Flatten the confusion matrix and get its parts\n",
    "# https://www.w3resource.com/numpy/manipulation/ravel.php\n",
    "tn, fp, fn, tp = cm_rfe.ravel()\n",
    "tn, fp, fn, tp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "**::GMG::** *Sensitivity* and *specificity* are statistical measures of the performance of a binary classification test, also known in statistics as a classification function, that are widely used in medicin. ([wikipedia](https://en.wikipedia.org/wiki/Sensitivity_and_specificity)). I can compute them from values in the confusion matrix!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Specificity: 0.748 \n",
      "Sensitivity: 0.627\n"
     ]
    }
   ],
   "source": [
    "#::GMG::specificity, selectivity or true negative rate (TNR)\n",
    "# https://en.wikipedia.org/wiki/Sensitivity_and_specificity\n",
    "specificity = tn / (tn + fp)\n",
    "#::GMG::sensitivity, recall, hit rate, or true positive rate (TPR)\n",
    "# https://en.wikipedia.org/wiki/Sensitivity_and_specificitysensitivity = tp / (tp + fn)\n",
    "sensitivity = tp / (tp + fn)\n",
    "print('Specificity: {:4.3f} \\nSensitivity: {:4.3f}'.format(\n",
    "    specificity, sensitivity\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Balanced Accuracy Score: 0.687\n"
     ]
    }
   ],
   "source": [
    "# https://scikit-learn.org/stable/modules/generated/sklearn.metrics.balanced_accuracy_score.html\n",
    "print('Balanced Accuracy Score: {:4.3f}'.format(\n",
    "    balanced_accuracy_score(y_true = test_target.values, y_pred = pred)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "     healthy      0.793     0.748     0.770       143\n",
      "    diabetes      0.566     0.627     0.595        75\n",
      "\n",
      "   micro avg      0.706     0.706     0.706       218\n",
      "   macro avg      0.679     0.687     0.682       218\n",
      "weighted avg      0.715     0.706     0.710       218\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#::GMG::The classification report shows all the metrics ...\n",
    "target_names = ['healthy', 'diabetes']\n",
    "print(classification_report(y_true = test_target.values, y_pred = pred, \n",
    "                            target_names=target_names, digits = 3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "#::GMG::I use prediction as probabilities for AUC and precision-recall curve\n",
    "prob = rfe.predict_proba(X = test_features)[:, 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "#::GMG::Precision-Recall Curve\n",
    "precision, recall, _ = precision_recall_curve(\n",
    "    y_true = test_target, \n",
    "    probas_pred = prob, \n",
    "    pos_label = 1\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AUC Precision/Recall: 0.640\n"
     ]
    }
   ],
   "source": [
    "#::GMG::AUC (precision, recall)\n",
    "auc_pr = auc(x = recall, y = precision)\n",
    "print('AUC Precision/Recall: {:4.3f}'.format(auc_pr))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Precision (Weighted): 0.643\n"
     ]
    }
   ],
   "source": [
    "#::GMG::average precision \n",
    "apc = average_precision_score(y_true = test_target, \n",
    "                              y_score = prob, \n",
    "                              pos_label = 1, \n",
    "                              average = 'weighted')\n",
    "print('Average Precision (Weighted): {:4.3f}'.format(apc))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5, 1.0, '2-class Precision-Recall curve: AP=0.64')"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmkAAAImCAYAAAAIfx1rAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3X+cXXV95/HXhwwkkJDYEEEhBAwSlYglEvHXqlh/FFyFrr+jqHQttLu61dba1a1VpGtt7WrXrtoVq1XxB6J1bVQotYpGi1hioVSgpAQJBFAJgwkkmZCBz/5xzshkftx7MzPn3u+deT0fj3lw7/mee+/n3nOHeef7/Z7zjcxEkiRJZTmg1wVIkiRpPEOaJElSgQxpkiRJBTKkSZIkFciQJkmSVCBDmiRJUoEMaVKPRMTZEfG9Xtcx0yLiuog4tc0+KyLivoiY16WyGhcRt0TEc+vb50XEZ3pdk6T+ZkiT9kNEzI+Ij0fEloi4NyKuiYjTe11XJ+oQsbsORz+NiE9GxKKZfp3MXJ2Z326zz62ZuSgzH5jp168D0t76ff48Iq6IiKfO9OvMFfX3ZDgiHjlm+4x8zhHxqvr3aWdEfCUilrbYd15E/M+IuKP+/bs6Ih42wX7fjIiMiIH9rUcqiSFN2j8DwG3As4AlwDuAiyPi2B7WtD9elJmLgCcCa6nq30dU+v3/DV+o3+cy4HLgiz2uZ8Z1I4BExELgJcB24KwJdhn5nB8OfA/4ckTEfjz/auCjwGuAI4BdwEdaPOTdwNOApwKL68cNjXnOVwMHdlqDVLJ+/x+x1FWZuTMzz8vMWzLzwcz8GvBj4OTJHhMRR0fElyPiroi4OyI+NMl+H4yI2yJiR0T8MCKeMartlIjYWLf9NCI+UG9fEBGfqZ/35xFxVUQc0cH7uB24FHh8/Tzfjoj3RMQ/Uv2hXBkRS+pewzsj4va6B+MXw5MRcU5E3FD3aFwfEU+st48e9pus7mNH93RExJERsT4iBiPipog4Z9TrnBcRF0fEp+vXui4i1rZ7j/X7HAY+CxwVEQ8f9ZwvrHtBR3qAnjCqbcLjFRHHRcS36m3bIuKzE/XidCIizqxff0dEbI6I08Z+dqPe+2fGfGavj4hbgW9FxKUR8cYxz/0vEfHi+vZjI+Ib9ed6Y0S8fD9LfQnwc+B84HWT7ZSZe4FPAY8ADtuP53818NXM3JCZ9wF/CLw4Ig4du2NE/BLwZuCczNySlR9l5tCofZYA7wJ+fz9qkIplSJOmoQ5Eq4DrJmmfB3wN2AIcCxwFXDTJ010FnAQsBT4HfDEiFtRtHwQ+mJmLgeOAi+vtr6Pq0Tua6o/jbwG7O6j7aOAFwNWjNr8GOBc4tK73k8Aw8GhgDfB84Dfqx78MOA94LVWPxhnA3RO81GR1j3URsBU4Engp8McR8Suj2s+o93kYsB6YMOhO8D4Pqmu8G7in3rYG+ATwm1Sf2UeB9VENZbc6XgG8t67xcVSf+Xmd1DGmplOATwNvrd/PM4Fb9uMpnlW//q8CnwfWjXruE4BjgK/XvWDfoPouHQ68EvhIvc/IMOO1bV7rdfVrXAQ8NiIm/MdIRMwHzgZuy8xtEfEf6gA82c9/qB+6GviXkefJzM3A/VS/U2OdSPV9fGlE/CQiNkXEG8bs88fAXwI/afO+pL5gSJOmKCIOpOql+VRm/tsku51C9Uf9rXUv3FBmTniyQGZ+JjPvzszhzHw/MB94TN28F3h0RCzLzPsy88pR2w8DHp2ZD2TmDzNzR4uyvxIRP6camvoO1R+1EZ/MzOvq3qelVCHuzXXdPwP+nOoPPVRh7X2ZeVXdo3FTZm6Z4PUmq/sX6sD4dOC/15/PNcBfUYWrEd/LzEvqOWwXAr/c4j0CvLx+n7uBc4CX1u8LqiD60cz8Qf2ZfQrYAzyFFserfo/fyMw9mXkX8AGqwLS/Xg98on6uBzPz9hbfn4mcV9e2G/h/wEkRcUzd9mrgy5m5B3ghcEtm/nX9nboa+BvgZfX7+VxmPmGiF4Dq5A7g2cDnMvOnwDfZ95jAQ5/zbVS9yf+pfu7vZebDWvyM/A4sohpKHW071T8UxlpO9Q+SVcCjqML8eRHxvLretVTfo/8z+Ucn9RdDmjQFUc3ZupDqX/1vHLX90qgmUt8X1dyYo4EtowJCq+f8vXr4cHv9h28J1ZwqqP6wrwL+LaohzRfW2y8ELgMuimoy9fvq8DiZX6v/SB6Tmf+1/kM/4rZRt4+hmtdz50jvB1WP0+F1+9HA5nbvqUXdox0JDGbmvaO2baHqxRoxumdkF7AgIgYi4tWjPu9LR+1zcWY+jGqe04/Ydzj6GOAto3t26vdzJC2OV0QcEREXRTX0uwP4DA8dn/3R6Wc3mV8cp/oz+zoPhed1VP9wgOp9PnnM+3w11ZBkJ14D3FCHZurnfdWY79fF9ffp8Mz8lcz84X6+l/uoemJHWwzcO8G+I9/V8zNzd2ZeS9XD94L69/EjwJs6+V2T+oVnvkj7KSIC+DhVAHhBPR8HgMw8fcy+TwVWRMRAqz8eUc0/+33gOcB1mflgRNxDNcRGZv47sK7+Y/Ri4EsRcVhm7qSaTP3uqE5euAS4sa5vf+Wo27dR9S4tm6Tu26iGL1s/4SR1j9ntDmBpRBw6KqitAG7v4Pk/y0OhZKL2bRFxLrAxIj6XmXfWtb8nM98zdv82x+uPqT6jEzNzMCJ+jQ6HXcdo9dntBA4ZdX+iQJVj7n8eeFdEbAAWUJ0oMfI638nM502hRqh6zVZExEhAHqDqtX0B8LetHlh/ny9tscvpmfldqmkCv+gVjYiVVD3ImyZ4zMjQ7Oj3P3J7MdWJMF+ofj0ZmTu5NSJeVr+W1HfsSZP2319SzQl60ZieqIn8E3An8CcRsTCqif5Pn2C/Q6nm29wFDETEOxnVwxARZ0XEwzPzQaqJ3AAPRsSzI+LEei7VDqrhxQen9e6AOsz8PfD+iFgcEQdENXF+ZHjvr4Dfi4iTo/LoUUNuvzBZ3WNe6zbgCuC99efzBKoeuBm5zlhm3kjV2zgymfxjwG9FxJPr2hdGxH+MarJ6q+N1KFXPz/aIOIpqTtlUfBz49Yh4Tv25HhURj63brgFeGREH1sN3L+3g+S6h6jU7n+psy5HP92vAqoh4Tf18B0bEkyLice2esA6rx1EN/55U/zyean7b2CHPcTLzu1ldYmWyn5HQ9FngRRHxjKjm0J1PNVw7rietnq/2XeAPopo/+DiqHsSvUQ2RHjmq1hfUDzsZ+EG7eqVSGdKk/VAHkd+k+kPwkzFDm+PUc6heRDX5/laqyfGvmGDXy4C/o+pB2EJ1WYHRw4+nAddFxH1Uk/FfWQfERwBfogpoN1DNM7twuu+z9lrgIOB6qkn3XwIeWb+vLwLvofqjfS/wFap5bGNNVvdY66gm6t9BNc/qXZn5DzP0PgD+DDg3Ig7PzI1U89Q+VL+vm6gmvbc7Xu+munTJdqohxi9PpZDM/Cfg16nm+G2nOmYjAfcPqcLRPfXrfa6D59tT1/Lc0fvXQef5VEHmDqoh4z+l6qmiHiqe8IQXqhMG/jYz/zUzfzLyQ3UMXxgtrmW2PzLzOqqTXT4L/IwqCP/XkfZ6+sD/GPWQdVSf1d1Ux+APM/Ob9bzI0XXeVe//08y8fyZqlXohMsf2nEuSJKnX7EmTJEkqkCFNkiSpQIY0SZKkAhnSJEmSCmRIkyRJKlDfXcx22bJleeyxx/a6DEmSpLZ++MMfbsvMh0/lsX0X0o499lg2btzY6zIkSZLaioiJ1jXuiMOdkiRJBTKkSZIkFciQJkmSVCBDmiRJUoEMaZIkSQUypEmSJBXIkCZJklQgQ5okSVKBDGmSJEkFMqRJkiQVyJAmSZJUIEOaJElSgQxpkiRJBTKkSZIkFciQJkmSVCBDmiRJUoEMaZIkSQVqLKRFxCci4mcR8aNJ2iMi/iIiboqIayPiiU3VIkmS1G+a7En7JHBai/bTgePrn3OBv2ywFkmSpL7SWEjLzA3AYItdzgQ+nZUrgYdFxCObqkeSJKmfDPTwtY8Cbht1f2u97c5WDxoagk2bmixr5i1dCsuW9boKSZLUT3oZ0joWEedSDYmy7Iij2LBlQ48r6tye3fNYungB6049udelSJKkPtLLkHY7cPSo+8vrbeNk5gXABQCrTlyVa1YvbL66GbJl83wGh1qN+kqSJI3Xy0twrAdeW5/l+RRge2a2HOqUJEmaKxrrSYuIzwOnAssiYivwLuBAgMz8v8AlwAuAm4BdwK83VYskSVK/aSykZea6Nu0JvKGp15ckSepnrjggSZJUIEOaJElSgQxpkiRJBTKkSZIkFciQJkmSVCBDmiRJUoEMaZIkSQUypEmSJBXIkCZJklQgQ5okSVKBDGmSJEkFMqRJkiQVyJAmSZJUIEOaJElSgQxpkiRJBTKkSZIkFciQJkmSVCBDmiRJUoEMaZIkSQUypEmSJBXIkCZJklQgQ5okSVKBDGmSJEkFMqRJkiQVyJAmSZJUIEOaJElSgQxpkiRJBTKkSZIkFciQJkmSVCBDmiRJUoEMaZIkSQUypEmSJBXIkCZJklQgQ5okSVKBDGmSJEkFMqRJkiQVyJAmSZJUIEOaJElSgQxpkiRJBTKkSZIkFciQJkmSVCBDmiRJUoEMaZIkSQUypEmSJBXIkCZJklQgQ5okSVKBBnpdwFy2bRsMDrbeZ+lSWLasO/VIkqRyGNJ6aHAQrth8DcMDOyZs37N7HksXL2DdqSd3uTJJktRrhrQeGx7YwZrVCyds27J5PoNDbbraJEnSrOScNEmSpAIZ0iRJkgpkSJMkSSqQc9IatmvnPHYPDbBp0wRtu7pfjyRJ6g+GtC4YGh5iw5YNE7bNX7QTmPjEAUmSNHcZ0rpgaO8Qa1YfMUmrAU2SJI3nnDRJkqQCGdIkSZIKZEiTJEkqkCFNkiSpQIY0SZKkAhnSJEmSCuQlOBq2cNEDHHz/3l6XIUmS+owhrWErVu5hxcrlvS5DkiT1GYc7JUmSCmRIkyRJKpAhTZIkqUDOSZuFtm2DwcHW+yxdCsuWdaceSZK0/wxps9DgIFyx+RqGB3ZM2L5n9zyWLl7AulNP7nJlkiSpU4a0WWp4YAdrVi+csG3L5vkMDrXpapMkST3lnDRJkqQCGdIkSZIKZEiTJEkqkHPS+lSrMzh37epuLZIkaeYZ0vpUuzM45y/aCUx84oAkSSqfIa2PtTqD04AmSVJ/c06aJElSgQxpkiRJBTKkSZIkFciQJkmSVCBPHCjYrp3z2D00wKZNE7R5mQ1JkmY1Q1rhhoaH2LBlw4RtXmZDkqTZy5BWuKG9Q6xZfcQkrQY0SZJmK+ekSZIkFciQJkmSVCBDmiRJUoEaDWkRcVpE3BgRN0XE2yZoXxERl0fE1RFxbUS8oMl6JEmS+kVjIS0i5gEfBk4HTgDWRcQJY3Z7B3BxZq4BXgl8pKl6JEmS+kmTZ3eeAtyUmTcDRMRFwJnA9aP2SWBxfXsJcEeD9ajHtm2DwcHJ25cuhWXLuvuaTb2uJEnT1WRIOwq4bdT9rcCTx+xzHvD3EfHfqK4n8dyJnigizgXOBTj8yMNnvNBSLVz0AAffv7fXZcyYwUG4YvM1DA/sGNe2Z/c8li5ewLpTT+7aazb5upIkTVevr5O2DvhkZr4/Ip4KXBgRj8/MB0fvlJkXABcArDpxVfagzp5YsXIPK1Yu73UZM2p4YAdrVo+/vtuWzfMZHGrT5TXDr9n060qSNB1NnjhwO3D0qPvL622jvR64GCAzvw8sABx4kiRJc16TPWlXAcdHxKOowtkrgVeN2edW4DnAJyPicVQh7a4Ga9Is1GremWucSpL6VWMhLTOHI+KNwGXAPOATmXldRJwPbMzM9cBbgI9FxO9QnURwdmbOmeHM2abdJP2mAlO7eWeucSpJ6keNzknLzEuAS8Zse+eo29cDT2+yBnVPu7AEzQWmVvPODGiSpH7U6xMHNMu0DktgYJIkqTOGNPUF551JkuYaQ5r6gvPOJElzjSFN+6WXPVpNzDvbtXMeu4cG2LRp4nZXI5Ak9YohTftlNvZoDQ0PsWHLhnHbXY1AktRLhjTtt9l2JuXQ3iHWrD5i3HZXI5Ak9ZIhTUVoN+zoyQGSpLnGkKZiTDbsOKIfh1IlSZoqQ5qKMdmw40NmPqAtXPQAB9+/d8afV5Kk6TKkaR+9WtqpV1as3MOKlcun9Nh2n5VnhkqSpsOQpn30cmmnftPqs/LMUEnSdBnSNI5LO3Vuss/KM0MlSdNlSFMRSpwb5hmnkqReMqSpCNOZG9YkzziVJPWKIU1qoRdnnEqSBHBArwuQJEnSePakzUGt5lo5z0qSpDIY0uaoVnOtnGdVKfFkBknS3GFIm6Naz7UyoEG5JzNIkuYG56RJkiQVyJAmSZJUIEOaJElSgQxpkiRJBTKkSZIkFciQJkmSVCAvwTEHef0vSZLKZ0ibg7z+lyRJ5XO4U5IkqUCGNEmSpAIZ0iRJkgpkSJMkSSqQIU2SJKlAhjRJkqQCGdIkSZIK5HXSJM1K27bB4ODk7UuXwrJl3atHkvaXIU3SrDQ4CFdsvobhgR3j2vbsnsfSxQtYd+rJPahMkjpjSJM0aw0P7GDN6oXjtm/ZPJ/BoRbdbJJUAEOa1IBdO+exe2iATZsm38fhtulpN5y5a1f3apGkJhjSpIYMDQ+xYcuGCdscbpu+VsOZI+Yv2gmM70mTpH5gSJMaMrR3iDWrj5iwzeG2mTHZcOZDDGiS+pchTVKxWg1pOpwpabYzpEkqVrshTYczJc1mhjSpAQsXPcDB9+/tdRl9oV1vWeshTQOapNnLkCY1YMXKPaxYubzXZfQFe8skaWKGNEk9Z2+ZJI1nSJPUKK9nJklTY0iT1CivZyZJU2NIk9Q4r2cmSfvPkCbNIu2GFlstRTWdx0qSZp4hTZpFWg0ttluKajqP9aKzkjTzDGnSLDPZ0GInS1FN9bFeRkOSZp4hTeozpfZaeRkNSZpZhjSpz0y112rXznnsHhpg06aJn7dVwJvOYyVJU2NIk/rQVHuthoaH2LBlw6TtrYYlp/NYSdL+M6RJc8jQ3iHWrD6ixR4tAt40HitJ2n+GNKkwXqFfkgSGNKk4TV2hf+GiBzj4/r1Tqmk6j5UkTY0hTSpQE1foX7FyDytWLp9SPdN5rCRpag7odQGSJEkaz540qQdaXdLCOWeSJDCkST3T6pIWXs5CkmRIk3qk9SUtDGiSNNc5J02SJKlAhjRJkqQCGdIkSZIK5Jw0qQe8OKwkqR1DmtQDXhxWktSOw52SJEkFMqRJkiQVyJAmSZJUIEOaJElSgTxxQNKc02rt1BFLl8KyZd2rSZLGMqRJmpNarZ26Z/c8li5ewLpTT+5yVZL0EEOapDmp1dqpN1x7CNt+fs+kPW32sknqBkOaJE1gsp42e9kkdYshTdKc08mKD5P1tLXrZQN72iTNDEOapDlnuis+OJ9NUjcY0iRpjHY9ba3ms23ZPJ/BocGmSpM0hxjSJGmMVj1tnQyVStJMMKRJ0n6Y7lCpJHXKFQckSZIKZEiTJEkqkCFNkiSpQIY0SZKkAjUa0iLitIi4MSJuioi3TbLPyyPi+oi4LiI+12Q9kiRJ/aKxszsjYh7wYeB5wFbgqohYn5nXj9rneODtwNMz856IOLypeiRJkvpJkz1ppwA3ZebNmXk/cBFw5ph9zgE+nJn3AGTmzxqsR5IkqW80GdKOAm4bdX9rvW20VcCqiPjHiLgyIk6b6Iki4tyI2BgRG7cPbm+oXEmSpHL0+sSBAeB44FRgHfCxiHjY2J0y84LMXJuZa5csXdLlEiVJkrqvyRUHbgeOHnV/eb1ttK3ADzJzL/DjiNhEFdquarAuSWrMrp3z2D00wKZNE7cvXQrLlnW3Jkn9qcmQdhVwfEQ8iiqcvRJ41Zh9vkLVg/bXEbGMavjz5gZrkqTGDQ0PsWHLhnHb9+yex9LFC1h36sk9qEpSv2kspGXmcES8EbgMmAd8IjOvi4jzgY2Zub5ue35EXA88ALw1M+9uqiZJ6oahvUOsWX3EuO1bNs9ncGiwBxVJ6keNLrCemZcAl4zZ9s5RtxP43fpHkiRJtUZDmiTNNQsXPcDB9+/tdRmSZgFDmiTNoBUr97Bi5fIJ29qdVACeWCDpIYY0SeqiyU4qANi9c4BDDjyEZxx/0oTtBjhpbjGkSVIXTXZSAcAN1x7CnffdOWGIM8BJc48hTZIKMlmIm06Ag2ZC3LZtMNjmZFXDozR1hjRJ6pLpnlQwlQAHzV2fbXAQrth8DcMDO7r6utJcYUiTpC5pdVIBTC/EtRpGbfL6bMMDO1izemHXX1eaCwxpklSIViGuV5f2aDWkuWtXd2uR5hpDmiT1gV4FuHZDmvMX7QQm7klzHVNpejoOaRFxFHDM6Mdk5sQTICRJXdNuGLWVdpP/d+1qPaQ5WUAb4Tqm0tR1FNIi4k+BVwAja2wCJGBIk6Q+1q6nDFr3lrXjOqbS1HXak/ZrwGMyc0+TxUiSZl6rYcf2PWUw1YDmElnS9HQa0m4GDgQMaZLUh1qtdDCdnrJWpjMMK6nzkLYLuCYivsmooJaZv91IVZKkGdXqEh1NBDRJ09dpSFtf/0iSJKkLOgppmfmpiDgIWFVvujEznWggSX3AuWFSf+r07M5TgU8BtwABHB0Rr/MSHJJUvrk0N8z1RDWbdDrc+X7g+Zl5I0BErAI+D3iBG0lSMdpdUqTdYvQGOJWk05B24EhAA8jMTRFxYEM1SZI0qXZLVbW6pEirxei9wK5K02lI2xgRfwV8pr7/amBjMyVJkjS56SxVBV5gV/2j05D2X4A3ACOX3Pgu8JFGKpIkzXrt1vW8997qv4ceOsFjp7FUlSdRqJ90enbnHuAD9Y8kSdPW6gK7u3dWf54OXjg8YftUL8A7l06iUP9rGdIi4uLMfHlE/CvVWp37yMwnNFaZJGlWa3WB3RuuPYQf3/NjnnaKF+DV3NWuJ+1N9X9f2HQhkqS5w2FHqb2WIS0z76xvbgN2Z+aD9eU3Hgtc2nRxkqTZqd2woyFOggM63G8DsCAijgL+HngN8MmmipIkzW0rVu7h2WudO6a5rdOQFpm5C3gx8JHMfBmwurmyJEmS5raOQ1pEPJXq+mhfr7fNa6YkSZIkdRrS3gy8Hfh/mXldRKwELm+uLEmSpLmt0+ukfQf4zqj7N/PQhW0lSZI0w9pdJ+1/Z+abI+KrTHydtDMaq0ySJGkOa9eTdmH93//VdCGSJEl6SLvrpP2wvrmR+jppABExD5jfcG2SJElzVqcLrH8TeC5wX33/YKrrpT2tiaIkSeq2dou+AyxdCsuWda8mzW2dhrQFmTkS0MjM+yLikIZqkiSpJ1ot+r5n9zyWLl7AulNP7nJVmqs6DWk7I+KJmfnPABFxMrC7ubIkSeq+Vou+b9k8n8GhwS5XpLms05D2ZuCLEXEHEMAjgFc0VpUkSdIc1+l10q6KiMcCj6k33ZiZrnwrSZo1XNRdpekopNXzz34XOCYzz4mI4yPiMZn5tWbLkySpO1as3MOKlS7qrnJ0uizUXwP3A0+t798O/M9GKpIkSVLHc9KOy8xXRMQ6gMzcFRHRYF2TOmTHHaz66u/ss+2elady1+ozieEhjr/07eMec/eqX+Xux5zGvKHtHPeN88a133XCGdxz3LM58L6f8ajL3zuu/adPeBnbj3ka839+K8d898/Htd+55izuXX4yB2+7iaO//+Fx7bc/6fXsfMTjWfiTH3HUVR8f137bU9/A7mWP5tCtP+SRV39mXPuWZ/wOex62giVbruCIa784rv3Hz347excdzi9tvpyHX79+XPvm553HAwuWcNiNf8dhmy4b1/7vp7+XHFjAw6/7W37p5m+Pa9/0ouo9H/EvX2DJrVfu0/bgwHxuOv1PAHjEP1/I4tv/eZ/24fmLufn57wbgyH/6GIt+ev0+7fcvfDi3/Mr/AGD5FR/ikLs379M+tGQ5tz7zLQCs2PB+Fmzfuk/7rsOOY+vT3gjAsd/6Yw7aedc+7fcdcQJ3nHIOACv//l0M7NmxT/uOo57IT574GgAefenbOGB4zz7t21c8hZ/+cjX9cuz3Dvzu+d3zuzeXvnvx0+Usf2AYfvpBOOtvqh2/8z64+Tv7PviQX4JX1O/pH86D267at33xkfCSj1W3L30b/ORf920/7Dg44y+q2+t/G8Z8N3nEiVB/9/mbc2DHHfu2H/0keO551e0vnAW77tm3feWz4Fm/X93+zEtg79C+7at+FZ5er/z41/+RcVb/GpxyDty/Cz77svHtJ70K1rwadt4NF792fPuT/jM8/iWwfSt8+TfHtz/tjfCY02Hbv8NX3zy+/Zm/B8c9G+68Fv5u/Hef57wTVjwZbv0BfPP88e2nvRce+QTYfDlsmOBa/S/637DseLjxUrjiQ+PbX/xRWLIcfvQ3cNUnxre//NOw8DC4+rNwzefGt++HTnvS7o+Ig6mXhoqI44A9rR8iSZKkqYrMcUtyjt8p4nnAO4ATqC5i+3Tg7Mz8dqPVTWDViavy85d9vtsvK0ma40YuwfEbz3tmr0tRH4mIH2bm2qk8tu1wZz2s+W/Ai4GnUF2C402ZuW0qLyhJkqT22oa0zMyIuCQzTwS+3oWaJEmS5rxO56T9c0Q8qdFKJEmS9Audnt35ZOCsiLgF2Ek15JmZ+YSmCpMkSZrLOg1pv9poFZIkSdpHy5AWEQuA3wIeDfwr8PHMHO5GYZIkSXNZuzlpnwLWUgW004H3N16RJEmS2g53nlCf1UlEfBz4p+ZLkiRJUruetL0jNxzmlCRJ6p52PWm/HBEjC84FcHB9f+TszsWNVidJkjRHtQxpmTmvW4VIkiTpIZ1ezFaSJEld1Ol10iRJmtN27ZzH7qEBNm2auH3pUli2rLs1aXYzpEmS1KGh4SE2bNkwbvue3fNYungB6049uQdVabYypEmS1KGhvUOsWX3EuO1bNs9ncGiwBxVpNjOkSZLUgYWLHuDg+/dO2NZuKBQcDtX+M6RJktSBFSv3sGLl8knbJxsKBYdDNTWGNEmSZsBkQ6HgcKimxpAmSdI0tRoKBc8M1dQY0iRJmqZ2Q6HgmaHaf4ayz6k+AAAOsElEQVQ0SZK6wDNDtb8MaZIkNazdcKg0EUOaJEkN62Q4VBrLkCZJUg95UoEmY0iTJKnHPKlAEzGkSZLUY02cVLBtGwy2eai9dGUzpEmS1EPTOamgVRC791645ifXkAftmLDdXrryGdIkSeqh6ZxUMDgIV2y+huGBiYPY/KU7OeGYwydsa6qX7t57q/8eeujkj7cHrzOGNEmSCtXupIJdu2B4YAdrVi+c5Bkm2z49rcLh7p1VtDh44fCEj7UHr3OGNEmSCtZq4XaA+Yt2MtNhrN18tlbh8IZrD+HH9/yYp53iOqbTZUiTJKlgrRZur0wtoLXqpWs3nw0mD4deuHfmGNIkSSpU04Fnsl663TsHGBoe4lfWLGlV3YRbvXDvzDGkSZJUqCYDz8JFDxD338ua1eOf/9ab57N58C6gVUhT0wxpkiTNQa0CoL1hZTig1wVIkiRpPEOaJElSgQxpkiRJBXJOmiRJKkK767M1tVJBr163HUOaJEnqmqlen206KxW0C2GtXnf3zgEOOfAQnnH8SZM+vqkQZ0iTJEldNZXrs91w7SFs+/k9ky6R1WrN0I4uzjvJOqc3XHsId95356SrPjS5zJUhTZIkdc10rs/WaomsVmuGTufivNB61Ycml7lqNKRFxGnAB4F5wF9l5p9Mst9LgC8BT8rMjU3WJEmSemc612drFZaqgLeZp83wxXl7ucxVYyEtIuYBHwaeB2wFroqI9Zl5/Zj9DgXeBPygqVokSVJ/axeWmro4b7vHtppjN11N9qSdAtyUmTcDRMRFwJnA9WP2+yPgT4G3NliLJEnqYyWvgtBqGJaDWoyjttFkSDsKuG3U/a3Ak0fvEBFPBI7OzK9HxKQhLSLOBc4FOPzI8ZP6JEmSeqHVHDsADmDeVJ+7ZycORMQBwAeAs9vtm5kXABcArDpxVTZbmSRJUmea7OFrcsWB24GjR91fXm8bcSjweODbEXEL8BRgfUSsbbAmSZKkvtBkSLsKOD4iHhURBwGvBNaPNGbm9sxclpnHZuaxwJXAGZ7dKUmS1GBIy8xh4I3AZcANwMWZeV1EnB8RZzT1upIkSbNBo3PSMvMS4JIx2945yb6nNlmLJElSP2lyuFOSJElTZEiTJEkqkCFNkiSpQIY0SZKkAhnSJEmSCmRIkyRJKpAhTZIkqUCGNEmSpAIZ0iRJkgpkSJMkSSqQIU2SJKlAhjRJkqQCGdIkSZIKZEiTJEkqkCFNkiSpQIY0SZKkAhnSJEmSCmRIkyRJKpAhTZIkqUCGNEmSpAIZ0iRJkgpkSJMkSSqQIU2SJKlAhjRJkqQCGdIkSZIKZEiTJEkqkCFNkiSpQIY0SZKkAhnSJEmSCmRIkyRJKpAhTZIkqUCGNEmSpAIZ0iRJkgpkSJMkSSqQIU2SJKlAhjRJkqQCGdIkSZIKZEiTJEkqkCFNkiSpQIY0SZKkAhnSJEmSCmRIkyRJKpAhTZIkqUCGNEmSpAIZ0iRJkgpkSJMkSSqQIU2SJKlAhjRJkqQCGdIkSZIKZEiTJEkqkCFNkiSpQIY0SZKkAhnSJEmSCmRIkyRJKpAhTZIkqUCGNEmSpAIZ0iRJkgpkSJMkSSqQIU2SJKlAhjRJkqQCGdIkSZIKZEiTJEkqkCFNkiSpQIY0SZKkAhnSJEmSCmRIkyRJKpAhTZIkqUCGNEmSpAIZ0iRJkgpkSJMkSSqQIU2SJKlAhjRJkqQCGdIkSZIKZEiTJEkqkCFNkiSpQIY0SZKkAhnSJEmSCmRIkyRJKpAhTZIkqUCGNEmSpAIZ0iRJkgpkSJMkSSqQIU2SJKlAhjRJkqQCNRrSIuK0iLgxIm6KiLdN0P67EXF9RFwbEd+MiGOarEeSJKlfNBbSImIe8GHgdOAEYF1EnDBmt6uBtZn5BOBLwPuaqkeSJKmfNNmTdgpwU2benJn3AxcBZ47eITMvz8xd9d0rgeUN1iNJktQ3mgxpRwG3jbq/td42mdcDlzZYjyRJUt8Y6HUBABFxFrAWeNYk7ecC5wIcfuThXaxMkiSpN5rsSbsdOHrU/eX1tn1ExHOBPwDOyMw9Ez1RZl6QmWszc+2SpUsaKVaSJKkkTYa0q4DjI+JREXEQ8Epg/egdImIN8FGqgPazBmuRJEnqK42FtMwcBt4IXAbcAFycmddFxPkRcUa9258Bi4AvRsQ1EbF+kqeTJEmaUxqdk5aZlwCXjNn2zlG3n9vk60uSJPUrVxyQJEkqkCFNkiSpQIY0SZKkAhnSJEmSCmRIkyRJKpAhTZIkqUCGNEmSpAIZ0iRJkgpkSJMkSSqQIU2SJKlAhjRJkqQCGdIkSZIKZEiTJEkqkCFNkiSpQIY0SZKkAhnSJEmSCmRIkyRJKpAhTZIkqUCGNEmSpAIZ0iRJkgpkSJMkSSqQIU2SJKlAhjRJkqQCGdIkSZIKZEiTJEkqkCFNkiSpQIY0SZKkAhnSJEmSCmRIkyRJKpAhTZIkqUCGNEmSpAIZ0iRJkgpkSJMkSSqQIU2SJKlAhjRJkqQCGdIkSZIKZEiTJEkqkCFNkiSpQIY0SZKkAhnSJEmSCmRIkyRJKpAhTZIkqUCGNEmSpAIZ0iRJkgpkSJMkSSqQIU2SJKlAhjRJkqQCGdIkSZIKZEiTJEkqkCFNkiSpQIY0SZKkAhnSJEmSCmRIkyRJKpAhTZIkqUCGNEmSpAIZ0iRJkgpkSJMkSSqQIU2SJKlAhjRJkqQCGdIkSZIKZEiTJEkqkCFNkiSpQIY0SZKkAhnSJEmSCmRIkyRJKpAhTZIkqUCGNEmSpAIZ0iRJkgpkSJMkSSqQIU2SJKlAhjRJkqQCGdIkSZIKZEiTJEkqkCFNkiSpQIY0SZKkAhnSJEmSCmRIkyRJKpAhTZIkqUCGNEmSpAIZ0iRJkgpkSJMkSSqQIU2SJKlAhjRJkqQCGdIkSZIK1GhIi4jTIuLGiLgpIt42Qfv8iPhC3f6DiDi2yXokSZL6RWMhLSLmAR8GTgdOANZFxAljdns9cE9mPhr4c+BPm6pHkiSpnzTZk3YKcFNm3pyZ9wMXAWeO2edM4FP17S8Bz4mIaLAmSZKkvtBkSDsKuG3U/a31tgn3ycxhYDtwWIM1SZIk9YWBXhfQiYg4Fzi3vrt37XFrb+lhOZqOYZYwwPZel6Ep8Nj1N49f//LY9bc9LJ/qQ5sMabcDR4+6v7zeNtE+WyNiAFgC3D32iTLzAuACgIjYmLtzbSMVq3ERsTH3evz6kceuv3n8+pfHrr9FxMapPrbJ4c6rgOMj4lERcRDwSmD9mH3WA6+rb78U+FZmZoM1SZIk9YXGetIyczgi3ghcBswDPpGZ10XE+cDGzFwPfBy4MCJuAgapgpwkSdKc1+ictMy8BLhkzLZ3jro9BLxsP5/2ghkoTb3j8etfHrv+5vHrXx67/jbl4xeOLkqSJJXHZaEkSZIKVGxIc0mp/tXBsfvdiLg+Iq6NiG9GxDG9qFMTa3f8Ru33kojIiPCss4J0cvwi4uX17+B1EfG5bteoiXXw/84VEXF5RFxd///zBb2oU+NFxCci4mcR8aNJ2iMi/qI+ttdGxBM7ed4iQ5pLSvWvDo/d1cDazHwC1UoT7+tulZpMh8ePiDgUeBPwg+5WqFY6OX4RcTzwduDpmbkaeHPXC9U4Hf7uvQO4ODPXUJ1o95HuVqkWPgmc1qL9dOD4+udc4C87edIiQxouKdXP2h67zLw8M3fVd6+EqV/oTzOuk989gD+i+ofRUDeLU1udHL9zgA9n5j0AmfmzLteoiXVy7BJYXN9eAtzRxfrUQmZuoLpKxWTOBD6dlSuBh0XEI9s9b6khzSWl+lcnx2601wOXNlqR9kfb41d30x+dmV/vZmHqSCe/f6uAVRHxjxFxZUS0+te/uqeTY3cecFZEbKW6csJ/605pmgH7+7cR6JNloTQ7RcRZwFrgWb2uRZ2JiAOADwBn97gUTd0A1ZDLqVS92Bsi4sTM/HlPq1In1gGfzMz3R8RTqa4z+vjMfLDXhakZpfak7c+SUrRaUkpd18mxIyKeC/wBcEZm7ulSbWqv3fE7FHg88O2IuAV4CrDekweK0cnv31ZgfWbuzcwfA5uoQpt6q5Nj93rgYoDM/D6wAFjWleo0XR39bRyr1JDmklL9q+2xi4g1wEepAprzYcrS8vhl5vbMXJaZx2bmsVRzCs/IzCmvTacZ1cn/O79C1YtGRCyjGv68uZtFakKdHLtbgecARMTjqELaXV2tUlO1HnhtfZbnU4DtmXlnuwcVOdzpklL9q8Nj92fAIuCL9bket2bmGT0rWr/Q4fFToTo8fpcBz4+I64EHgLdmpqMQPdbhsXsL8LGI+B2qkwjOtnOiDBHxeap//Cyr5wy+CzgQIDP/L9UcwhcANwG7gF/v6Hk9vpIkSeUpdbhTkiRpTjOkSZIkFciQJkmSVCBDmiRJUoEMaZIkSQUypEmaVSLigYi4JiJ+FBFfjYiHzfDznx0RH6pvnxcRvzeTzy9JIwxpkmab3Zl5UmY+nuoaim/odUGSNBWGNEmz2fcZtYhxRLw1Iq6KiGsj4t2jtr+23vYvEXFhve1FEfGDiLg6Iv4hIo7oQf2S5rAiVxyQpOmKiHlUS+h8vL7/fKo1Kk8BgmrN0WdSrfn7DuBpmbktIpbWT/E94CmZmRHxG8DvU13xXZK6wpAmabY5OCKuoepBuwH4Rr39+fXP1fX9RVSh7ZeBL2bmNoDMHKzblwNfiIhHAgcBP+5O+ZJUcbhT0myzOzNPAo6h6jEbmZMWwHvr+WonZeajM/PjLZ7n/wAfyswTgd+kWsxakrrGkCZpVsrMXcBvA2+JiAGqhav/c0QsAoiIoyLicOBbwMsi4rB6+8hw5xLg9vr267pavCThcKekWSwzr46Ia4F1mXlhRDwO+H5EANwHnJWZ10XEe4DvRMQDVMOhZwPnAV+MiHuogtyjevEeJM1dkZm9rkGSJEljONwpSZJUIEOaJElSgQxpkiRJBTKkSZIkFciQJkmSVCBDmiRJUoEMaZIkSQUypEmSJBXo/wP2w/2p28lbgAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x648 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#::GMG::Plot Recall-Precision Curve\n",
    "step_kwargs = ({'step': 'post'}\n",
    "               if 'step' in signature(plt.fill_between).parameters\n",
    "               else {})\n",
    "plt.figure(figsize = (10,9))\n",
    "# plot no skill\n",
    "plt.plot([0, 1], [0.5, 0.5], linestyle='--', color = 'C1')\n",
    "plt.step(recall, precision, color='b', alpha=0.2, where='post')\n",
    "plt.fill_between(recall, precision, alpha=0.2, color='g', **step_kwargs)\n",
    "plt.xlabel('Recall')\n",
    "plt.ylabel('Precision')\n",
    "plt.ylim([0.0, 1.05])\n",
    "plt.xlim([0.0, 1.0])\n",
    "plt.title('2-class Precision-Recall curve: AP={0:0.2f}'.format(apc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "**::GMG::** where should I integrate feature selection in my machine learning pipeline? *Datacamp* reference author states:\n",
    "\n",
    "> you should include the feature selection step before feeding the data to the model for training especially when you are using accuracy estimation methods such as cross-validation. This ensures that feature selection is performed on the data fold right before the model is trained. But if you perform feature selection first to prepare your data, then perform model selection and training on the selected features then it would be a blunder. If you perform feature selection on all of the data and then cross-validate, then the test data in each fold of the cross-validation procedure was also used to choose the features, and this tends to bias the performance of your machine learning model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Wrapper: Recursive Feature Elimination with Cross-Validation (RFECV)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "**::GMG::** In the [second part](https://towardsdatascience.com/machine-learning-workflow-on-diabetes-data-part-02-11262b7f7a5c) of article by *Lahiru Liyanapathirana* (see also [the code](https://github.com/LahiruTjay/Machine-Learning-With-Python/blob/master/Machine%20Learning%20Workflow%20on%20Diabetes%20Data.ipynb) in Github) the feature selection is done with the *CV variant of RFE*, i.e. [`RFECV`](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.RFECV.html). As a bonus `RFECV` also allows let you know the optimum number of features to use. See also the [example](https://scikit-learn.org/stable/auto_examples/feature_selection/plot_rfe_with_cross_validation.html) from scikit-learn.  \n",
    "\n",
    "**::GMG::** *Lahiru Liyanapathirana* and the example from scikit-learn don't train/test split the data before doing `RFECV`, I'm not sure why. I'm going to *keep* the train/test split and do the cross-validation on the train set. Later I'm going to de prediction and evualtion on the test set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### feature selection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train features: (506, 8), target: (506,)\n",
      "test  features: (218, 8), target: (218,)\n"
     ]
    }
   ],
   "source": [
    "#::GMG::Split features/target in train and test is already done\n",
    "#::NOTE::\n",
    "print ('train features: {}, target: {}'.format(train_features.shape, train_target.shape))\n",
    "print ('test  features: {}, target: {}'.format(test_features.shape, test_target.shape))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "#::GMG::Create the RFE object and compute a cross-validated score.\n",
    "#::NOTE::the example from scikit learn uses as estimator SVC(kernel=\"linear\") but\n",
    "#        I'm going to keep my Logistic Regresor\n",
    "model_cv = LogisticRegression(solver = 'liblinear', \n",
    "                              class_weight = 'balanced', \n",
    "                              random_state = 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "#::GMG::This cross-validation object is a variation of KFold that returns stratified folds. \n",
    "#       The folds are made by preserving the percentage of samples for each class.\n",
    "# https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedKFold.html\n",
    "strat_k_fold = StratifiedKFold(n_splits=10, random_state=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "#::GMG::I use \"balanced accuracy\" as scoring\n",
    "# https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.RFECV.html\n",
    "# https://scikit-learn.org/stable/modules/model_evaluation.html#common-cases-predefined-values\n",
    "rfecv = RFECV(estimator = model_cv, \n",
    "              step = 1, \n",
    "              cv = strat_k_fold,\n",
    "              scoring='balanced_accuracy')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RFECV(cv=StratifiedKFold(n_splits=10, random_state=10, shuffle=False),\n",
       "   estimator=LogisticRegression(C=1.0, class_weight='balanced', dual=False,\n",
       "          fit_intercept=True, intercept_scaling=1, max_iter=100,\n",
       "          multi_class='warn', n_jobs=None, penalty='l2', random_state=2,\n",
       "          solver='liblinear', tol=0.0001, verbose=0, warm_start=False),\n",
       "   min_features_to_select=1, n_jobs=None, scoring='balanced_accuracy',\n",
       "   step=1, verbose=0)"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#::GMG::\n",
    "rfecv.fit(train_features, train_target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f76820d77f0>]"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEKCAYAAAA4t9PUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3Xd81eX5//HXm7D3SJApGxEUGRFBrLOuVnFWwa2to277q7W2/dZKta22aqtSFa2zKtaNW1uGA1QCIiMKhLCHhL0h4/r98fkEjzE5+UBycnKS6/l4nAfnfNa5ToBc577vz33dMjOcc865fVUn2QE455xLbZ5InHPOVYgnEueccxXiicQ551yFeCJxzjlXIZ5InHPOVYgnEueccxXiicQ551yFeCJxzjlXIXWjHiipCbDTzAoTGE9CpKenW9euXZMdhnPOpZTp06evNbOM8o4rM5FIqgOMBM4HDgV2AQ0krQXeAh4xs5xKijehunbtSlZWVrLDcM65lCJpSZTj4nVtTQR6ALcC7cyss5m1BY4APgXuknRBhSN1zjmX0uJ1bf3QzPJLbjSz9cDLwMuS6iUsMueccymhzBZJcRKR1ENSg/D50ZKul9Qy9hjnnHO1V5S7tl4GCiX1BMYCnYHnEhqVc865lBElkRSZWQFwBvCAmd0MtE9sWM4551JFlESSL2kUcDHwZrjNx0acc84B0RLJpcAw4E4zWySpG/BMYsNyzjmXKsqdkGhm2cD1AJJaAc3M7K5EB+ac2zdTFq4lZ81W+rZvTp/2zWnaIPK8Y+f2Sbn/wiRNAkaEx04H1kj6xMx+keDYnHN76aXpy7n5pS8x+3Zb1zaN6duhOX3bNw//bMF+zRsgKXmBuholyleVFma2WdLPgKfN7DZJsxIdmHNu7/wnaxm3vDyL4T3SufOMg1iYt5W5KzaTvWozc1du5u3Zq/cc26ZJ/RLJpTnd0ptQN83L7xXbuquARXnbyF27ldy8bSxaGzwA2rdoSIeWjWjXoiHtWzSkfYtGtG/RkHYtGlKvFv4MoySSupLaA+cAv01wPM65ffCfacu45ZVZHNEznUcvyqRhvTS6tGnCsX3223PMlp35fL16C9krNwePVZt54pPF7C4sAqBB3Tr0adfsOwmmT7vmNKnBXWP5hUUsW799T6KITRprtuzac5wEnVo1olt6UwQsWbedqbnr2LKz4DvXkyC9aQM6hMmlXYuGdGjZkHYtGgXbWjaibbMGNS7ZRPkXMhp4D/jYzKZJ6g4sSGxYzrmoXpi2lFtens2RvTMYe+FgGtZLK/W4Zg3rcWjX1hzatfWebfmFReTmbWPuyk17kss7c1bz/OfLgOAXY7c2TTgwJrn0a9+cjGap0zVmZqzZsovcsHWxaE/S2MbS9dspLPq2H7B1k/p0T2/CUb0z6J7RlG7pTeie0YT9Wzcu9ee6dVcBqzftYOXGnazatINVm3ayauNOVm7awcK8rXycs5atu76bbOoIMpo12NOK2fNny2+ft23WIKVah7LYztQaKjMz07xoo6uJnv98Kbe+MpujemfwSJwksjfMjFWbdu5JLNkrNzN31SaWrd+x55j0pvU5sH1z+nVo8Z2usbQ6yUsuW3bms3jt9j2tity121gUJo5tu78tWt6wXh26tgkSRPf0b5NFt/QmtGxcv9Lj2rwzn9WbdrJyY5hoNu1k1cYdrN787bbtu79bVL2OYL/mQVdZhxbfdqEVd6d1aNGIjGYNEv7zljTdzDLLPa68RCKpIfBToB/QsHi7mV1W0SCriicSVxM999lSfvPqbI4+IIOHL6icJBLPph35fL3q2+SSvWoz87/ZQn5h8DukYb069GnXvETXWDMa16+8rrH8wiKWrt++Z+xi0dpte5JGXildUcWJokdGE7qlN6VbRhPaN29InSQmvJLMjM07C4IWzcYw0YStnNWbd+xp4ezML/rOeWl1xH7NGtC+ZXHLJrZ1E3SlpTdtUKHPWpmJ5EXga+A8gm6u84GvzOyGfY6uinkicTXNvz9dwu9em8MxB2Tw8IWDaVA3sUmkLLsLiliYt3VPYinuItscjh1I0C29SdByiRnYz2jWoMxrFndFLcwLEsWiPa2L73dFtWlSP6ZF8W3S6FxGV1SqMjM27cjfk1xKdqUVP99V8N1kU7eOeOv6H3BAu2b79L6VmUi+MLOBkmaZWf+w4u9HZjZ0nyJLAk8kriZ55tMl/N9rczi2T1seumBQ0pJIWcyMFRt3fKdrLHvVZpZv+LZrLKNZgz2JpWubxqzYuDO8K6r0rqhu6U3pnt7kO91Q3dOb0qKxF9koZmZs2J7Pyo07WL3p2+Ry5VE9aNFo335OURNJlDZncYXfjZIOAlYDbfcpKudchTw9dTG/f30uPzywLWPOr35JBEASnVo1plOrxpzQr92e7Zu25/PV6nDMJUwun3yYS0GRUUfQqVVjuqU34dCurcOk0ZTuGU1oV826oqorSbRuUp/WTepzUMcWVfreURLJ2HBG+/8B44GmwO8TGpVz7nuemrKY28bP5YcH7seY8wdWyyQST4vG9RjavQ1Du7fZs21XQSHfbNrFfi0apNzncd+KUiLlsfDpZKD73lxc0knAP4A04DEz+0uJ/fcBx4QvGwNtzaxluK8QmB3uW2pmI8Lt3YBxQBuCmfYXmtnuvYnLuVTzxCeLuP2NbI7vux9jzhtE/bqpc2toPA3qprF/m8bJDsNVULw12+OWQDGze+Ptl5QGjAGOB5YD0ySND2t3FV/jppjjrwMGxlxih5kNKOXSdwH3mdk4SQ8T3FH2ULxYnEtlj3+8iNFvZnNiv/14YFTNSSKu5oj3L7JZOY/yDAFyzCw3bDGMA06Lc/wo4Pl4F1QwA+pY4KVw01PA6RFicS4lPfZRLqPfzOakfu14sAa1RFzNUmaLxMxur+C1OwLLYl4vBw4r7UBJXYBuwISYzQ0lZQEFwF/M7DWC7qyN4UJbxdfsWME4nauWHvsolzve+oqTD2rH/aMG1riyGq7mKPdfpqSnitdoD1+3kvR4JccxEnjJzGKnd3YJbzs7D/i7pB57c0FJV0jKkpSVl5dXmbE6l3CPfhgkkR8f3N6TiKv2ovzr7G9mG4tfmNkGvjuWUZYVBOu7F+sUbivNSEp0a5nZivDPXGBS+J7rgJaSiltSZV7TzMaaWaaZZWZkZEQI17nq4ZHJC7nz7a/4cf/2/H3kAE8irtqL8i+0Tnj7LwCSWhPttuFpQC9J3STVJ0gW40seJKkP0AqYGrOtlaQG4fN0YDiQbcHsyYnA2eGhFwOvR4jFuZTw0KSF/Pmdrzn1kA7841xPIi41REkI9wBTw1IpAD8B7izvJDMrkHQtQeXgNOBxM5sraTSQZWbFSWUkMM6+O8X+QOARSUUEye4vMXd73QKMk3QH8AXwrwifwblq75+Tcrj73XmMOKQD955zSEpVf3W1W6Tqv5L6EtwtBTAh9hbeVOAlUlx1N2ZiDn99bx6nDejAPT/xJOKqh8oskVK8bntKJQ/nUsWDExbwt/fnc/qADtxzzoCklmJ3bl/U3KXPnEsB9/9vAfd+MJ8zB3bkrz85xJOIS0meSJxLkn/8dwH3/Xc+Zw7qyF/P9iTiUpcnEueS4L4P5vOP/y3g7MGduOus/p5EXEqLMiHxTEkLJG2StFnSFkmbqyI452oaM+PeMIn8xJOIqyGitEjuBk41s68SHYxzNZmZcd8H87l/Qg7nZnbmz2ce7OtsuBohSiL5xpOIcxVjZtzz/nwenJjDyEM786czPIm4miNeGfkzw6dZkl4AXgN2Fe83s1cSHJtzNYKZ8bf35zFm4kJGDenMnad7EnE1S7wWyakxz7cDJ8S8NsATiXPlMDPufm8eD01ayHmH7c8dpx3kScTVOPHKyF9alYE4V9OYGX9592semZzLBUP3Z/QITyKuZqouZeSdq1HMjL+8EySRC4d24Y/eEnE1WJTB9u+VkZcUpYy8c7WSmfGnt7/i0Y8WcdGwLtw+oh/B4p7O1UyJLCPvXK1jZtz5VpBELjm8qycRVyvsbRl5EawFUm4ZeedqGzPjj29+xeOfLOLS4V35/Sl9PYm4WqHcRGJmT0uaDhwTbjoz1crIO5doZsboN7N54pPFXDa8G/93yoGeRFytEbWM/FxJeUBDAEn7m9nShEbmXIowM25/I5snpyzmp0d043c/9iTiapcod22NkLQAWARMBhYD7yQ4LudSgpnxh/FzeXLKYi7/gScRVztFGWz/IzAUmG9m3YDjgE8TGpVzKcDM+P3rc3lq6hKuPLI7v/mRJxFXO0VJJPlmto7g7q06ZjYRKHfpRQBJJ0maJylH0q9L2X+fpJnhY76kjeH2AZKmSporaZakc2POeVLSopjzBkT8rM5VmqIi4/9en8Mzny7hyqO68+uT+3gScbVWlDGSjZKaAh8Bz0paA2wr7yRJacAY4HhgOTBN0vjYgXozuynm+OuA4vkp24GLzGyBpA7AdEnvxcxnudnMXooQu3OVrqjI+N3rc3jus6X8/Oge/OrEAzyJuFotSovkNIJf7DcC7wIL+W4drrIMAXLMLNfMdgPjwmuVZRTwPICZzTezBeHzlcAaICPCezqXUEVFxm9fC5LINcd4EnEOIiQSM9sGdAaONrOngMeA3RGu3RFYFvN6ebjteyR1AboBE0rZNwSoT5DAit0ZdnndJ6lBhFicq7CiIuM3r87m+c+Xcu0xPfnlCZ5EnINod21dDrwEPBJu6khQUr4yjQReMrPCEu/dHngGuNTMisLNtwJ9gEOB1sAtZcR9haQsSVl5eXmVHK6rbYqKjFtfmc24acu4/tie/L8TensScS4UpWvrGmA4sBkg7HJqG+G8FQQtmWKdwm2lGUnYrVVMUnPgLeC3ZrbnLjEzW2WBXcATBF1o32NmY80s08wyMzK8V8ztu6Ii49evzOKFrGVcf1wvbjrek4hzsaIkkl3hGAcAkuoSrEdSnmlAL0ndJNUnSBbjSx4kqQ/QCpgas60+8CrwdMlB9bCVgoL/yacDcyLE4tw+e3nGcv6TtZzrj+vFLzyJOPc9URLJZEm/ARpJOh54EXijvJPMrAC4FngP+Ar4TzhDfrSkETGHjgTGmVlscjoHOBK4pJTbfJ+VNBuYDaQDd0T4DM7tk8IiY8zEHPp1aM5NP+yV7HCcq5b03d/fpRwg1QF+SrBCoggSw2NW3onVSGZmpmVlZSU7DJeCXp+5ghvGzeThCwZx0kHtkx2Oc1VK0nQzK3feYJSijUXAo+HDuVqjqMh4cEIOvfdrygl92yU7HOeqrShdW87VSu/NXc2CNVu55pievrqhc3F4InGuFGbGAxNy6J7ehFP6d0h2OM5Va55InCvFhK/XkL1qM1cf05M0b404F1eZYySS3iDObb5mNqKsfc6lMjPj/gk5dGrViNMGeGvEufLEG2z/W5VF4Vw18nHOWr5ctpE/nXEw9dK80e5cecpMJGY2uSoDca66eOB/ObRv0ZCzBpdaGs45V0K8rq3ZxO/a6p+QiJxLok9z1/H54vX84dS+NKibluxwnEsJ8bq2TqmyKJyrJh6ckEN60waMHLJ/skNxLmXE69paUpWBOJdsM5Zu4OOctfzmR31oWM9bI85FFaWM/FBJ0yRtlbRbUqGkzVURnHNV6YH/LaBV43qcf1iXZIfiXEqJckvKgwSrFy4AGgE/I1hC17kaY86KTUycl8fPftCdJg2irEDtnCsW6d5GM8sB0sys0MyeAE5KbFjOVa0HJiygecO6XDjMWyPO7a0oX722h+uDzJR0N7AKnxHvapB5q7fw3txvuP64XjRvWC/Z4TiXcqIkhAvD464FitdvPyuRQTlXlR6cmEOT+mlcNrxrskNxLiVFaZGsBXab2U7gdklpQIPEhuVc1ViYt5U3Z63kyiN70LJx/WSH41xKitIi+R/QOOZ1I+C/iQnHuar1z4kLaVC3Dj/7Qbdkh+JcyoqSSBqa2dbiF+HzxnGOdy4lLF23nddmruC8IV1Ib+qNbOf2VZREsk3SoOIXkgYDOxIXknNV46HJOaTVEVce1T3ZoTiX0qIkkhuBFyV9JOlj4AWCgfdySTpJ0jxJOZJ+Xcr++yTNDB/zJW2M2XexpAXh4+KY7YMlzQ6veb8kXyzC7bWVG3fw0vTlnJvZmf2aN0x2OM6ltChrtk+T1Ac4INw0z8zyyzsvHJQfAxwPLAemSRpvZtkx174p5vjrgIHh89bAbUAmQeHI6eG5G4CHgMuBz4C3Cea0vBPhszq3xyOTF2KGt0acqwRRSqQ0Bm4BbjCzOUBXSVEKOg4Bcsws18x2A+OA0+IcPwp4Pnx+IvCBma0Pk8cHwEmS2gPNzexTMzPgaeD0CLE4t8eaLTt5ftoyzhrUiU6tfLjPuYqK0rX1BLAbGBa+XgHcEeG8jsCymNfLw23fI6kL0A2YUM65HcPnUa55haQsSVl5eXkRwnW1xaMf5lJQWMTPj+6R7FCcqxGiJJIeZnY3kA9gZtuByh6XGAm8ZGaFlXVBMxtrZplmlpmRkVFZl3Upbt3WXfz706WcNqAjXdObJDsc52qEKIlkt6RGhItcSeoB7Ipw3gqCWfDFOoXbSjOSb7u14p27Inwe5ZrOfc/jnyxiZ0Eh1xzjrRHnKkuURHIb8C7QWdKzBBMUfxXhvGlAL0ndwlpdI4HxJQ8KB/JbAVNjNr8HnCCplaRWwAnAe2a2CtgclrYXcBHweoRYnGPT9nyemrKEHx3Unp5tmyU7HOdqjLh3bYW/rL8GzgSGEnRp3WBma8u7sJkVSLqWICmkAY+b2VxJo4EsMytOKiOBceHgefG56yX9kSAZAYw2s/Xh86uBJwlm2L+D37HlInpyymK27irg2mN7JjsU52oUxfz+Lv0AabaZHVxF8SREZmamZWVlJTsMl0RbduZzxF0TObRrax67ODPZ4TiXEiRNN7Ny/8NE6dqaIenQSojJuaR55tMlbNqRz/XHeWvEucoWpfrvYcD5kpYQlJEXYGbWP6GROVdJtu8u4LGPFnFU7wz6d2qZ7HCcq3GiJJITEx6Fcwn03GdLWb9tN9f52IhzCRGlRMqSqgjEuUTYmV/I2A9zGda9DZldWyc7HOdqJF8y19VoL2YtY82WXVznYyPOJUyZiUSSL9DgUtrugiIemrSQwV1aMax7m2SH41yNFa9FMhVA0jNVFItzlerVL5azctNOrju2J77agHOJE2+MpL6k84DDJZ1ZcqeZvZK4sJyrmILCIsZMXEj/Ti04qrfXWnMukeIlkquA84GWwKkl9hngicRVW+O/XMnS9dv53Y8He2vEuQQrM5GY2cfAx5KyzOxfVRiTcxVSWGSMmZhDn3bN+OGB+yU7HOdqvCjzSJ6RdD1wZPh6MvBwlFUSnUuGd+asYmHeNh48byB16nhrxLlEi5JI/gnUC/8EuJBgudufJSoo5/ZVUZHx4IQcemQ04eSD2ic7HOdqhSiJ5FAzOyTm9QRJXyYqIOcq4r9ffcPXq7dw7zmHkOatEeeqRJQJiYXhYlYASOoOVNpKhs5VFjPjgQk5dGnTmBGHdEh2OM7VGlFaJDcDEyXlEhRs7AJcmtConNsHk+fnMXvFJu4662DqpnnRBueqSpRaW/+T1As4INw0z8yiLLXrXJUpbo10bNmIMwZ2Kv8E51ylidIiIUwcsxIci3P7bOrCdUxfsoE/ntaP+nW9NeJcVfL/ca5GeGBCDm2bNeAnmZ2THYpztU5CE4mkkyTNk5Qj6ddlHHOOpGxJcyU9F247RtLMmMdOSaeH+56UtChm34BEfgZX/WUtXs/U3HVccWR3GtZLS3Y4ztU65XZtSRpUyuZNwBIzK4hzXhowBjgeWA5MkzTezLJjjukF3AoMN7MNktoCmNlEYEB4TGsgB3g/5vI3m9lL5cXuaocHJuTQpkl9zjts/2SH4lytFHVC4iCCMRIBBwFzgRaSfm5m75dx3hAgx8xyASSNA04DsmOOuRwYY2YbAMxsTSnXORt4x8y2R4jV1TJfLtvI5Pl5/OqkA2hcP9KQn3OukkXp2loJDDSzTDMbDAwEcglaGnfHOa8jsCzm9fJwW6zeQG9Jn0j6VNJJpVxnJPB8iW13Spol6T5fN6V2e2BCDi0a1eOiYV2THYpztVaURNLbzOYWvwi7pvoUtzQqqC7QCzgaGAU8Kqll8U5J7YGDgfdizrkV6AMcCrQGbintwpKukJQlKSsvL68SQnXVTfbKzfz3q2+4bHg3mjbw1ohzyRIlkcyV9JCko8LHP4HssCUQr3DjCiD2FppO4bZYy4HxZpZvZouA+QSJpdg5wKuxBSLNbJUFdgFPEHShfY+ZjQ1bUZkZGb4eRU00ZmIOTRvU5ZLDuyY7FOdqtSiJ5BKCwe4bw0duuC0fOCbOedOAXpK6SapP0EU1vsQxrxG0RpCUTtDVFdvSGUWJbq2wlYKCRSZOB+ZE+AyuhslZs4W356zi4sO70KJxvWSH41ytFqU/4GTgQTO7p5R9W8s6ycwKJF1L0C2VBjxuZnMljQayzGx8uO8ESdkE9btuNrN1AJK6ErRoJpe49LOSMggG/mcSLMDlapkxExfSsG4alw3vluxQnKv1oiSSU4H7JH0IvAC8G++231hm9jbwdoltv495bsAvwkfJcxfz/cF5zOzYKO/taq7Fa7fx+swV/PSIbrRp6vdaOJds5XZtmdmlQE/gRYKupoWSHkt0YM6V5aFJC6mbVofLj+ye7FCcc0SvtZUv6R2CtdobEYxN+MJWrsot37Cdl2cs5/zD9qdts4bJDsc5R4QWiaSTJT0JLADOAh4D2iU4LudK9fDkhUhw5VE9yj/YOVclorRILiIYG7nSy8e7ZPpm807+M205Zw/uTIeWjZIdjnMuFGU9klFVEYhz5Xlkci6FZvzcWyPOVStRuraGSpomaauk3ZIKJW2uiuCcK7Z26y6e+3wJpw/oyP5tGic7HOdcjCgTEh8kuFtrAcFA+88Iqvo6V2Ue+2gRuwqKuPoYb404V91EWo/EzHKANDMrNLMngNKKKzqXEBu27eaZqYs5pX8HemQ0TXY4zrkSogy2bw9LnMyUdDewCl9Z0VWhJ6YsZtvuQq49pmeyQ3HOlSJKQrgwPO5aYBtB2ZKzEhmUc8U278zniU8WcWK//TigXbNkh+OcK0WUu7aWhE93ArcnNhznvuvpKYvZsrOA647tVf7BzrmkKLNFIukNSadK+l5pVUndJY2WdFliw3O12bZdBfzr40Ucc0AGB3VskexwnHNliNciuZygmOLfJa0H8oCGQDeCsvIPmtnriQ/R1VbPfraEDdvzue44b404V52VmUjMbDXwK+BXYUn39sAOYL6vn+4SbWd+IWM/XMQRPdMZtH+rZIfjnIsjatHGxcDihEbiXIxxny9l7dZdXHvswGSH4pwrh9/G66qdXQWFPDw5lyFdWzO0e5tkh+OcK4cnElftvDx9Bas37+TaY33eiHOpIFIikdRI0gGJDsa5/MIi/jkph0M6t+QHvdKTHY5zLoIoRRtPJVgb/d3w9QBJ46NcXNJJkuZJypH06zKOOUdStqS5kp6L2V4oaWb4GB+zvZukz8JrvhDOunc1xGtfrGD5hh1cf2xPJCU7HOdcBFFaJH8AhgAbAcxsJsEtwHFJSiMo7ngy0BcYJalviWN6AbcCw82sH3BjzO4dZjYgfIyI2X4XcJ+Z9QQ2AD+N8BlcCigsMv45aSF92zfn2D5tkx2Ocy6iKIkk38w2ldhmEc4bAuSYWa6Z7QbGAaeVOOZyYIyZbQAwszXxLqjgK+qxwEvhpqcIlv11NcCbs1ayaO02rvPWiHMpJUoimSvpPCBNUi9JDwBTIpzXEVgW83p5uC1Wb6C3pE8kfSoptqpwQ0lZ4fbiZNEG2GhmBXGu6VJQUZExZmIOvdo25cR+vpKzc6kkSiK5DugH7AKeAzbx3S6oiqgL9AKOJljz5FFJLcN9XcwsEziPYHb9Xi1EIemKMBFl5eXlVVK4LlHez17N/G+2cu2xPalTx1sjzqWSuIkkHOcYbWa/NbNDw8fvzGxnhGuvIKgUXKxTuC3WcmC8meWb2SJgPkFiwcxWhH/mApOAgcA6oKWkunGuSXjeWDPLNLPMjIyMCOG6ZDEzHpiQQ7f0JpzSv0Oyw3HO7aW4icTMCoEj9vHa04Be4V1W9YGRQMm7vV4jaI0gKZ2gqytXUitJDWK2DweyzcyAicDZ4fkXA17vK8VNnLeGuSs38/Oje5DmrRHnUk6UEilfhLffvkiwHgkAZvZKvJPMrEDStcB7QBrwuJnNlTQayDKz8eG+EyRlA4XAzWa2TtLhwCOSigiS3V/MLDu89C3AOEl3AF8A/9qbD+yqFzPj/v/l0KlVI84Y6MNdzqWiKImkIUGX0rEx2wyIm0gAzOxt4O0S234f89wIKgz/osQxU4CDy7hmLsEdYa4G+CRnHTOXbeTOMw6iXpoXWnAuFUVZ2OrSqgjE1U73T1hAu+YNOXtwp2SH4pzbR1FmtneS9KqkNeHjZUn+v95V2Ge56/h80XquPKo7DeqmJTsc59w+itKX8ATBIHmH8PFGuM25CnlwYg7pTesz8tD9kx2Kc64CoiSSDDN7wswKwseTgN9P6yrki6Ub+GjBWi7/QXca1ffWiHOpLMpg+zpJFwDPh69HEQy+O7dXVm/aydTctUzJWcfEeXm0bFyPC4Z2SXZYzrkKipJILgMeAO4juFtrCuAD8K5c67ft5tPcdUxZuJYpC9eRmxfcPd6ycT2GdW/DRcO60qRBpEU6nXPVWJS7tpYAI8o7zrktO/OZtng9U3LWMWXhOrJXbQagSf00DuvehvOG7M+wHm04sF1zL4PiXA1SbiKR9BRwg5ltDF+3Au4xs8sSHZyr3nbmFzJ9yYY9LY5ZyzdRWGTUr1uHzC6t+OUJvRnWI53+nVr4HBHnarAo/Qr9i5MIgJltkDQwgTG5aiq/sIgvl21kysKgu2rGko3sLiyibh1xSOeWXH10D4b1aMOg/VvRsJ4PoDtXW0RJJHUktSpeM0RS64jnuRRXWGRkr9y8p8UxbfF6tu8uRIJ+HZpzyfCuDOvRhkO7tqapj3U4V2tF+d9/DzBV0ouACAr0Sy6mAAAXNklEQVQm3pnQqFxSmBkL1mxlSk6QOD7NXcfmncHSLz3bNuXswZ04vEc6Q7u3pmVjX+HYOReIMtj+tKQsvq21dWZMAUWXwsyMZet37GlxTFm4jrVbdwHQuXUjTj6oPYf3bMOw7m1o27xhkqN1zlVXUQbbewALzSxb0tHADyWtjB03cakjdi7HlIXrWLFxBwBtmzXgiJ5tOLxHOsN6tKFz68ZJjtQ5lyqidG29DGRK6gk8QlAu5TngR4kMzFWO8uZyXHVUd4b1SKdHRhNfJ905t0+iJJKicG2RM4EHzewBSV8kOjC3b7bszOfzRev3dFV9FTOXY0i31ow6NJjL0be9z+VwzlWOKIkkX9Io4CLg1HBbvcSF5PbFsvXb+X//+ZLpSzf4XA7nXJWKkkguBa4C7jSzRZK6Ac8kNiy3N7bvLuCKZ6azfMN2n8vhnKtyUe7aygauj3m9CLgrkUG56MyMX700i69Xb+aJSw7l6APaJjsk51wt430dKe7hybm8OWsVvzqxjycR51xSJDSRSDpJ0jxJOZJ+XcYx50jKljRX0nPhtgGSpobbZkk6N+b4JyUtkjQzfAxI5GeozibNW8Pd733NKf3bc9VR3ZMdjnOulkpYXQtJacAY4HhgOTBN0vjYyYySegG3AsPDGl7FX6m3AxeZ2QJJHYDpkt6Lmbtys5m9lKjYU8Gitdu47vkv6NOuOXef3d9v3XXOJU2ZiUTSGwTrj5TKzMorLT8EyDGz3PB644DTgNhZ8ZcDY4rreJnZmvDP+THvs1LSGoJVGX0SJLB1VwGXP51F3Tpi7IWDaVzf61w555InXtfW3wjqbC0CdgCPho+twMII1+4ILIt5vTzcFqs30FvSJ5I+lXRSyYtIGgLUL/Ged4ZdXvdJalDam0u6QlKWpKy8vLwI4aaGoiLjphdmsmjtNsacN8hnoDvnkq7Mr7JmNhlA0j1mlhmz642w9lZlvX8v4GigE/ChpINj1j5pT3Cr8cVmVhSecyuwmiC5jAVuAUaXEv/YcD+ZmZlltqxSzf0TFvBB9jf8/pS+HN4zPdnhOOdcpMH2JpL2jOSG80iaRDhvBdA55nWncFus5cB4M8sPbyueT5BYkNQceAv4rZl9WnyCma2ywC7gCYIutFrh/bmr+ft/F3DWoE5cOrxrssNxzjkgWiK5CZgkaZKkycBE4MYI500DeknqJqk+MJKgTles1whaI0hKJ+jqyg2PfxV4uuSgethKQcHo8unAnAixpLwF32zhphdmckinFtx5xkE+uO6cqzaiTEh8N7y7qk+46euwNVDeeQWSrgXeA9KAx81srqTRQJaZjQ/3nSApGygkuBtrnaQLgCOBNpIuCS95iZnNBJ6VlEGwNspMgln3NdqmHflc/nQWjerX5eELB/uMdedctSKz+MMHkhoDvwC6mNnlYVI5wMzerIoAK0NmZqZlZVXWsE7VKiwyLntyGlMWruX5y4eS2bV1skNyztUSkqaXGCMvVZSurSeA3cCw8PUK4I4KxOb2wt/en8fk+Xn8YUQ/TyLOuWopSiLpYWZ3A/kAZradoFvJJdibs1by0KSFjBqyP+cf1iXZ4TjnXKmiJJLdkhoRTk4MV0wsd4zEVUz2ys3c/OIsMru04vYR/ZIdjnPOlSnKlOjbgHeBzpKeBYYDlyQyqNpu/bbdXPFMFi0a1eOfFwyifl2vremcq76i3LX1gaQZwFCCLq0bzGxtwiOrpQoKi7j2uRms2bKLF68cRttmDZMdknPOxVXuV11JZwAFZvZWeKdWgaTTEx9a7fSnt79mysJ1/OmMgzmkc8tkh+Occ+WK0mdym5ltKn4Rli+5LXEh1V4vT1/O458s4tLhXTl7cKdkh+Occ5FESSSlHePlZivZl8s2cuursxnWvQ2/+dGByQ7HOecii5JIsiTdK6lH+LgXmJ7owGqTvC27uPKZ6WQ0bcCY8wdRL80H151zqSPKb6zrCCYkvhA+dgHXJDKo2mR3QRE///d0Nu7YzdiLBtO6Sf1kh+Scc3slyl1b24BSl8l1FXf7G3PJWrKBB0YNpF+HFskOxznn9lq5iURSb+CXQNfY483s2MSFVTs899lSnv1sKVcd1YNTD+mQ7HCcc26fRBk0fxF4GHiMoEKvqwRZi9dz2/g5HNU7g5tPPCDZ4Tjn3D6LkkgKzOyhhEdSi6zetJOr/j2Dji0bcf/IgaTV8dJlzrnUFWWw/Q1JV0tqL6l18SPhkdVQO/MLufKZLHbsLmDsRZm0aFwv2SE551yFRGmRXBz+eXPMNgO6l3Ksi8PM+O2rc/hy+SYeuXAwvfdrluyQnHOuwqLctdWtKgKpDZ6cspiXZyznhuN6cWK/dskOxznnKkWkGeqSDgL6AnsqCJrZ04kKqiaasnAtd7z1Fcf33Y8bjuuV7HCcc67SRCnaeBvwQPg4BrgbGBHl4pJOkjRPUo6kUueiSDpHUrakuZKei9l+saQF4ePimO2DJc0Or3m/pGo/Ur1s/XaueXYG3dKbcO85h1DHB9edczVIlMH2s4HjgNVmdilwCFDuzDlJacAY4GSC1swoSX1LHNMLuBUYbmb9gBvD7a0JCkMeBgwBbpPUKjztIeByoFf4OCnCZ0iaHbsLueKZ6RQUGWMvHEyzhj647pyrWaIkkh1mVkRQPr45sAboHOG8IUCOmeWa2W5gHHBaiWMuB8aY2QYAM1sTbj8R+MDM1of7PgBOktQeaG5mn5qZAU8D1bakvZnxq5dn8fXqzdw/ciDdM5omOyTnnKt0UYs2tgQeJSjWOAOYGuG8jsCymNfLw22xegO9JX0i6VNJJ5VzbsfwebxrAiDpCklZkrLy8vIihFv5Hvkwlze+XMnNJx7AMX3aJiUG55xLtCh3bV0dPn1Y0rsELYJZlfj+vYCjgU7Ah5IOrowLm9lYYCxAZmamVcY198akeWu4692v+XH/9vz8qB5V/fbOOVdlykwkkgbF22dmM8q59gq+2wXWKdwWaznwmZnlA4skzSdILCsIkkvsuZPC7Z1KbC95zaRbvHYb1z//BQfs14y/nt2fFLgfwDnn9lm8Fsk9cfYZUF7RxmlAL0ndCH7ZjwTOK3HMa8Ao4AlJ6QRdXbnAQuBPMQPsJwC3mtl6SZslDQU+Ay4iuJus2ti6q4DLn84irY549KJMGtf3NcCcczVbmb/lzOyYilzYzAokXQu8B6QBj5vZXEmjgSwzGx/uO0FSNkFByJvNbB2ApD8SJCOA0Wa2Pnx+NfAk0Ah4J3xUC0VFxi9emEnu2m08fdkQOrdunOyQnHMu4RTc/FTOQSk+ITEzM9OysrIS/j7/+O8C7vvvfP7vlL789AgvCOCcS22SpptZZnnHRVmP5DaC8Yq+wNsE80I+Jrj11oU+yP6G+/47nzMHduSy4V2THY5zzlWZhE1IrE1y1mzhphdm0r9TC/505sE+uO6cq1USOSGxVti0I5/Ln55Ow3p1ePiCwTSsl5bskJxzrkpFuaWo5ITErUSbkFjjFRYZN4z7gmXrt/P8FUPp0LJRskNyzrkql+wJiSntnvfnMWleHnecfhCHdvW1vpxztVPUMvJnAkcQzB/5GKj1ieTNWSv556SFjBqyPxcM7ZLscJxzLmmilJH/J3AVMBuYA1wpaUyiA6vOsldu5uYXZzG4Syv+MKJv+Sc451wNFqVFcixwYFhtF0lPAXMTGlU1tmHbbq54Jovmjery0PmDaFDXB9edc7VblLu2coD9Y153DrfVOgWFRVzz3AzWbN7FIxdm0rZ5w/JPcs65Gi5e0cY3CMZEmgFfSfo8fH0Y8HnVhFe9/Pmdr5mycB1/Pbs/Azq3THY4zjlXLcTr2vpblUWRAl6ZsZx/fbyISw7vyk8yfRqNc84Vi1e0cXJVBlKdzVq+kV+/Mpuh3Vvz2x8fmOxwnHOuWokyRlKr5W3ZxZXPTCejaQPGnDeIemn+I3POuVi+WEYcuwuKuPrZ6WzYvpuXrjqcNk0bJDsk55yrdsr8ei3pZkmdytpfG4x+cy7TFm/grrP6c1BHr1PpnHOliddP0wGYKukjSVdLyqiqoKoDM6NrmyZcfXQPThvQMdnhOOdctRV3YSsF9dCPJFgm93TgS+B54BUz21IlEVaCqlrYyjnnapKoC1vFHTm2wGQz+znQCbgPuBH4pnLCdM45l+oi3YIk6WBgNDAG2AXcGvG8kyTNk5Qj6del7L9EUp6kmeHjZ+H2Y2K2zZS0U9Lp4b4nJS2K2Tcg6od1zjlX+eLNbO8FjALOBQqBccAJZpYb5cKS0ggSz/HAcmCapPFmll3i0BfM7NrYDWY2ERgQXqc1QUmW92MOudnMXooSh3POucSKd/vvuwTjIeea2Zx9uPYQIKc48UgaB5wGlEwk5TkbeMfMtu9DDM455xIsXtfWScC7JZOIpOGSekS4dkdgWczr5eG2ks6SNEvSS5JKqz0ykiChxbozPOc+ST65wznnkiheIrkP2FTK9s3A3yvp/d8AuppZf+AD4KnYnZLaAwcD78VsvhXoAxwKtAZuKe3Ckq6QlCUpKy8vr5LCdc45V1K8RLKfmc0uuTHc1jXCtVcQlJwv1incFnutdWa2K3z5GDC4xDXOAV41s/yYc1aFd5PtAp4g6EL7HjMba2aZZpaZkVGrpsA451yVipdI4tVJbxTh2tOAXpK6SapP0EU1PvaAsMVRbATwVYlrjKJEt1bxOeEcl9MJVm10zjmXJPEG27MkXW5mj8ZuDG/RnV7ehc2sQNK1BN1SacDjZjZX0mggy8zGA9dLGgEUAOuBS2LepytBi6ZkFeJnw1n2AmYSLAMc1/Tp09dKWlLecWVIB9bu47nJkErxeqyJk0rxplKskFrxVjTWLlEOKnNmu6T9gFeB3XybODKB+sAZZra6AsGlDElZUWZ2VhepFK/HmjipFG8qxQqpFW9VxRpvPZJvgMMlHQMcFG5+y8wmJDoo55xzqaPcMvLh5MCJVRCLc865FOSrNJVvbLID2EupFK/HmjipFG8qxQqpFW+VxBq3+q9zzjlXHm+ROOecqxBPJGWQ9LikNZKq/TwVSZ0lTZSULWmupBuSHVM8khpK+lzSl2G8tyc7pvJISpP0haQ3kx1LeSQtljQ7rI5drRfikdQyLI/0taSvJA1LdkxlkXRAiarkmyXdmOy4yiLppvD/1xxJz0tqmLD38q6t0kk6EtgKPG1mB5V3fDKFkzTbm9kMSc0Ibtc+vZRKy9VCOJm0iZltlVQP+Bi4wcw+TXJoZZL0C4Lb35ub2SnJjiceSYuBTDOr9nMdJD0FfGRmj4UTlxub2cZkx1WesLr5CuAwM9vXOWoJI6kjwf+rvma2Q9J/gLfN7MlEvJ+3SMpgZh8STJKs9sKyMTPC51sIKgRU2/WBwxI3W8OX9cJHtf1GI6kT8GOCMj6ukkhqQbAC678AzGx3KiSR0HHAwuqYRGLUBRpJqgs0BlYm6o08kdQwYUWAgcBnyY0kvrCraCawBvjAzKpzvH8HfgUUJTuQiAx4X9J0SVckO5g4ugF5wBNht+FjkpokO6iISqtKXm2Y2Qrgb8BSYBWwyczej3/WvvNEUoNIagq8DNxoZpuTHU88ZlZoZgMIinkOkVQtuw8lnQKsMbNyywJVI0eY2SDgZOCasJu2OqoLDAIeMrOBwDbgeyupVjdhF9wI4MVkx1IWSa0I1n/qBnQAmki6IFHv54mkhgjHGl4GnjWzV5IdT1RhV8ZEgvVvqqPhwIhw3GEccKykfyc3pPjCb6OY2RqCMkelVsiuBpYDy2Naoy8RJJbq7mRgRlj9o7r6IbDIzPLC6umvAIcn6s08kdQA4eD1v4CvzOzeZMdTHkkZklqGzxsRLMf8dXKjKp2Z3WpmncysK0F3xgQzS9g3u4qS1CS84YKwm+gEqmmF7LBe3zJJB4SbjmPvV1BNhu9VJa+GlgJDJTUOfz8cx/erq1caTyRlkPQ8MBU4QNJyST9NdkxxDAcuJPi2XHxr4o+SHVQc7YGJkmYRLDfwgZlV+9tqU8R+wMeSvgQ+J6iP926SY4rnOoKK3rOAAcCfkhxPXGFyPp7gG361FbbyXgJmALMJftcnbJa73/7rnHOuQrxF4pxzrkI8kTjnnKsQTyTOOecqxBOJc865CvFE4pxzrkI8kbgqIckk3RPz+peS/lBJ135S0tmVca1y3ucnYYXa760YKumvYaXVv+7DdQdU89u1kbS1/KNKPe90SX2r6v1ccngicVVlF3CmpPRkBxIrLGgX1U+By83smFL2XQH0N7Ob9yGMAcBeJRIFUuH/7+nAXicSl1pS4R+iqxkKCCZE3VRyR8kWRfG3UUlHS5os6XVJuZL+Iun8cC2T2ZJ6xFzmh5KyJM0P62MVF4b8q6RpkmZJujLmuh9JGk8pM6kljQqvP0fSXeG23wNHAP8q2eoIr9MUmC7p3HDm/svh+06TNDw8boikqWGBwikK1reoD4wGzg0nkp4r6Q+Sfhlz/TmSuoaPeZKeJpit3lnSCeE1Z0h6May3Rvizyg4/999K+YxHxUxe/SJmNvzNMT+vUteJKesYSReF276U9IykwwlqUv01fJ8e4eNdBQUlP5LUJzy3W/g5Zku6o7T3ddWYmfnDHwl/EKzt0hxYDLQAfgn8Idz3JHB27LHhn0cDGwlmwjcgWP/h9nDfDcDfY85/l+CLUS+CGk4NCVoJvwuPaQBkERSxO5qgQGC3UuLsQFBeIoOgqOAEgrVdACYRrPNR6ueLef4cQeFEgP0JStcQfv664fMfAi+Hzy8BHow5/w/AL2NezwG6ho8iYGi4PR34kGBtF4BbgN8DbYB5fDvhuGUp8b4BDA+fNw0/6wkEyV7hz/JN4MgSfyelHgP0A+YD6eFxrcv4u/0f0Ct8fhhByRmA8cBF4fNrYn+e/qj+j71p1jtXIWa2Ofw2fT2wI+Jp08xsFYCkhUBxKezZQGwX03/MrAhYICkX6EPwS69/TGunBUGi2Q18bmaLSnm/Q4FJZpYXvuezBL8oX4sYLwRJoq+k4tfNw5ZCC+ApSb0ISr3X24trFlti3y4ANpSg2+iT8L3qE5T12QTsJGg9vUnwy76kT4B7w8/3ipktl3QCwc/si/CYpgQ/rw9jzivrmEOAFy1cTMvMvreWT/gzOBx4MeZn0yD8czhwVvj8GeCucn8SrtrwROKq2t8J6v88EbOtgLCbNez3rx+zb1fM86KY10V8999vyVo/RvCt+Tozey92h6SjCVokiVKHoNWws8T7PghMNLMzFKwbM6mM8/f8PEKxS6TGxi2COmWjSl5A0hCCQn1nA9cCx8buN7O/SHqLYGzmE0knhtf7s5k9EuezlXqMpOvinFOsDrDRguUDSuP1mlKUj5G4KhV+U/0PwcB1scXA4PD5CPbtm/pPJNUJx026E3TtvAf8XEGJfST1VvkLJ30OHCUpXcFyqqOAyXsZy/sExQgJ37f4F2cLgu45CLqzim0BmsW8XkxYTl3SIILuuNJ8CgyX1DM8tkn4GZsCLczsbYIxqUNKniiph5nNNrO7CApn9iH4eV0WM87SUVLbEqeWdcwEgr+DNuH21iU/mwVr5CyS9JPwGEkqju0TgurKAOeX8XldNeWJxCXDPQT9+8UeJfjl/SUwjH1rLSwlSALvAFeFrYHHCAbTZ0iaAzxCOa3wsBvt1wRrpHwJTDez1/cyluuBzHDgORu4Ktx+N/BnSV+UiGMiQVfYTEnnEqwr01rSXILWxPwyYs0jSEjPK6ieO5UgITQD3gy3fQz8opTTbwwH8WcB+cA7Fqyg9xwwVdJsguqxsQmOso4xs7nAncDk8O+xeDmDccDN4YB+D4Ik8dPwmLkEiy9BMOZ1TXjNartMtCudV/91zjlXId4icc45VyGeSJxzzlWIJxLnnHMV4onEOedchXgicc45VyGeSJxzzlWIJxLnnHMV4onEOedchfx/ScrPWeCmwvQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot number of features VS. cross-validation scores\n",
    "plt.figure()\n",
    "plt.xlabel(\"Number of features selected\")\n",
    "plt.ylabel(\"CV balanced score (avg of recall each class)\")\n",
    "plt.plot(range(1, len(rfecv.grid_scores_) + 1), rfecv.grid_scores_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimal number of features : 5\n"
     ]
    }
   ],
   "source": [
    "print(\"Optimal number of features : %d\" % rfecv.n_features_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Features: [('Pregnancies', True, 1), ('Glucose', True, 1), ('BloodPressure', True, 1), ('SkinThickness', False, 3), ('Insulin', False, 4), ('BMI', True, 1), ('DiabetesPedigreeFunction', True, 1), ('Age', False, 2)]\n"
     ]
    }
   ],
   "source": [
    "#::GMG::Understanding how selection is made using support and ranking\n",
    "print(\"Features: %s\" % \n",
    "      [values for values in zip(train_features.columns, \n",
    "                                rfecv.support_, \n",
    "                                rfecv.ranking_)\n",
    "      ]\n",
    "     )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Selected Features: ['Pregnancies', 'Glucose', 'BloodPressure', 'BMI', 'DiabetesPedigreeFunction']\n"
     ]
    }
   ],
   "source": [
    "#::GMG::How to get the selected features in a pythonic way ...\n",
    "print(\"Selected Features: %s\" % \n",
    "      [value for value, supported in zip(train_features.columns, rfecv.support_) if supported]\n",
    "     )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### prediction and metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "#::GMG::Reduce X to the selected features and then predict using the underlying estimator\n",
    "# https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.RFECV.html#sklearn.feature_selection.RFECV.predict\n",
    "#::NOTE::Lahiru Liyanapathirana uses accuracy cross_val_score() for evaluation and comparison\n",
    "pred_cv = rfecv.predict(X = test_features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[114  29]\n",
      " [ 24  51]]\n"
     ]
    }
   ],
   "source": [
    "#::GMG::Confusion matrix (baseline performance analysis)\n",
    "cm_rfe_cv = confusion_matrix(y_true = test_target, y_pred = pred_cv)\n",
    "tn, fp, fn, tp = cm_rfe_cv.ravel()\n",
    "print(cm_rfe_cv)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Specificity: 0.797 \n",
      "Sensitivity: 0.680\n"
     ]
    }
   ],
   "source": [
    "#::GMG::specificity, selectivity or true negative rate (TNR)\n",
    "# https://en.wikipedia.org/wiki/Sensitivity_and_specificity\n",
    "specificity = tn / (tn + fp)\n",
    "#::GMG::sensitivity, recall, hit rate, or true positive rate (TPR)\n",
    "# https://en.wikipedia.org/wiki/Sensitivity_and_specificity\n",
    "sensitivity = tp / (tp + fn)\n",
    "print('Specificity: {:4.3f} \\nSensitivity: {:4.3f}'.format(specificity, sensitivity))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Balanced Accuracy Score: 0.739\n"
     ]
    }
   ],
   "source": [
    "#::GMG::The balanced accuracy in binary and multiclass classification problems to deal \n",
    "#       with imbalanced datasets. It is defined as the average of recall obtained on each class\n",
    "print('Balanced Accuracy Score: {:4.3f}'.format(\n",
    "    balanced_accuracy_score(y_true = test_target, y_pred = pred_cv)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classification Report:\n",
      "\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "     healthy      0.826     0.797     0.811       143\n",
      "    diabetes      0.637     0.680     0.658        75\n",
      "\n",
      "   micro avg      0.757     0.757     0.757       218\n",
      "   macro avg      0.732     0.739     0.735       218\n",
      "weighted avg      0.761     0.757     0.759       218\n",
      "\n"
     ]
    }
   ],
   "source": [
    "target_names = ['healthy', 'diabetes']\n",
    "print('Classification Report:\\n')\n",
    "print(classification_report(y_true = test_target, \n",
    "                            y_pred = pred_cv, \n",
    "                            target_names = target_names, \n",
    "                            digits = 3)\n",
    "     )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "#::GMG::I use prediction as probabilities for AUC and precision-recall curve\n",
    "prob_cv = rfecv.predict_proba(X = test_features)[:, 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "#::GMG::Precision-Recall Curve\n",
    "precision_cv, recall_cv, _ = precision_recall_curve(\n",
    "    y_true = test_target, \n",
    "    probas_pred = prob_cv, \n",
    "    pos_label = 1\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AUC Precision/Recall: 0.730\n"
     ]
    }
   ],
   "source": [
    "#::GMG::AUC (precision, recall)\n",
    "auc_pr_cv = auc(x = recall_cv, y = precision_cv)\n",
    "print('AUC Precision/Recall: {:4.3f}'.format(auc_pr_cv))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Precision (Weighted): 0.732\n"
     ]
    }
   ],
   "source": [
    "#::GMG::average precision \n",
    "apc_cv = average_precision_score(y_true = test_target, \n",
    "                                 y_score = prob_cv, \n",
    "                                 pos_label = 1, \n",
    "                                 average = 'weighted')\n",
    "print('Average Precision (Weighted): {:4.3f}'.format(apc_cv))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5, 1.0, '2-class Precision-Recall curve: AP=0.73')"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmkAAAImCAYAAAAIfx1rAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3X+cXXV97/vXxwwkkJDYEEEhBAw/rETQSIq/qo31R8Gr0OtPUn9gL5X2nnJOba29eo5H0XOsp/Zq23PVHmmxKooUPB4bK0g9ikaLWqAgFShpAgQCKIaBBDKZJBM+94+1ApOZ2T9mkjX7u2dez8djHpm9vmuv/dmzdjLvfL9rfb+RmUiSJKksT+p1AZIkSRrPkCZJklQgQ5okSVKBDGmSJEkFMqRJkiQVyJAmSZJUIEOa1CMR8faI+H6v6zjQIuKWiFjdYZ9lEfFoRMyZprIaFxF3RcTL6+8vjIgv9LomSf3NkCZNQkTMjYiLI2JTRDwSETdFxJm9rqsbdYjYUYejn0XEZyNiwYF+ncxckZnf6bDP3Zm5IDP3HOjXrwPS7vp9PhwR10bECw7068wW9edkJCKeNmb7Afk5R8Rv1H+ftkfEVyNicYv9Xly/1uivjIjX1e3nRMTtEbE1Ih6IiM9FxMKpvWupDIY0aXIGgHuAXwEWAe8DLo+I43pY02S8JjMXAM8FVlHVv4+o9Pu/DX9bv88lwDXAFT2u54CLiIFpeI35wOuArcBbJthl78/5KcD3ga9EREzi+CuATwNvBY4EhoBPTbRvZn6vDvYL6td8NfAo8I16l38EXpSZi4DlVH9X/2u3tUgl6vd/iKVplZnbM/PCzLwrMx/LzL8H7gROa/WciDgmIr4SET+PiAcj4hMt9vuLiLgnIrZFxA0R8eJRbadHxPV1288i4uP19nkR8YX6uA9HxHURcWQX7+Ne4CrgWfVxvhMRH46If6T6Rbk8IhbVvYb3R8S9EfFfRw9PRsQ7IuK2ukfx1oh4br199LBfq7qPq3tBBurHR0XE2ogYjIgNEfGOUa9zYURcHhGfr1/rlohY1ek91u9zBPgicHREPGXUMV9d94Lu7QE6dVTbhOcrIo6PiG/X27ZExBcj4snd1DFWRJxdv/62iNgYEWeM/dmNeu9fGPMzOy8i7ga+HRFXRcQFY47944h4bf39L0bEN+uf6+0R8cZJlvo64GHgQ8C5rXbKzN3A54CnAodP4vhvBr6Wmesy81HgPwOvjYjDunjuucCXM3N7XcM9mbllVPse4IRJ1CIVx5Am7Yc6EJ0E3NKifQ7w98Am4DjgaOCyFoe7DngOsBi4FLgiIubVbX8B/EVmLgSOBy6vt59L1aN3DNUvx98BdnRR9zHAq4AbR21+K3A+cFhd72eBEapfdCuBVwK/VT//DcCFwNuAhcBZwIMTvFSruse6DNgMHAW8HvjjiPjVUe1n1fs8GVgLTBh0J3ifB9c1Pgg8VG9bCXwG+G2qn9mngbVRDWW3O18BfKSu8ZlUP/MLu6ljTE2nA58H3l2/n5cAd03iEL9Sv/6vAV8C1ow69snAscDX616wb1J9lo4AzgE+Ve+zd5jx5g6vdW79GpcBvxgRE/5nJCLmAm8H7snMLRHxy3UAbvX1y/VTVwA/3nuczNwI7KL6O9VS/d5eTxUMR2//5YjYCjxCFTD/vMP7k4pmSJOmKCIOouql+Vxm/muL3U6n+qX+7roXbjgzJ7xZIDO/kJkPZuZIZn4MmAs8o27eDZwQEUsy89HM/OGo7YcDJ2Tmnsy8ITO3tSn7qxHxMNXQ1HeBPx7V9tnMvKXufVpMFeLeWdf9APBnVL/ooQprH83M67KyITM3TfB6rep+XB0YXwT8P/XP5ybgr6nC1V7fz8wr62vYLgGe3eY9Aryxfp87gHcAr6/fF1RB9NOZ+aP6Z/Y5YCfwfNqcr/o9fjMzd2bmz4GPUwWmyToP+Ex9rMcy8942n5+JXFjXtgP4X8BzIuLYuu3NwFcycyfVcOBdmfk39WfqRuB/Am+o38+lmXnqRC8A1c0dwEuBSzPzZ8C32PecwBM/53uoepP/z/rY38/MJ7f52vt3YAHVUOpoW6n+o9DOa4EtVJ/hx9WvuwhYCvwpkwu/UnEMadIURHXN1iVU/+u/YNT2q+KJi5rfTNXbsmlUQGh3zD+shw+31r/4FlFdUwXVL/aTgH+thzRfXW+/BLgauCwi7ouIj9bhsZVfr39JHpuZ/67+Rb/XPaO+PxY4CLh/b+8HVY/TEXX7McDGTu+pTd2jHQUMZuYjo7ZtourF2uuno74fAuZFxEBEvHnUz/uqUftcnplPprrO6SfsOxx9LPCu0T079fs5ijbnKyKOjIjLohr63QZ8gSfOz2R0+7Nr5fHzVP/Mvs4T4XkN1X8coHqfzxvzPt9MNSTZjbcCt9Whmfq4vzHm83V5/Xk6IjN/NTNvmOR7eZSqJ3a0hVQ9Ye2cC3w+M3Oixno4/xu07rWW+kLjF55KM01EBHAxVQB4VX09DgCZeeaYfV8ALIuIgXZBLarrz/4IeBlwS2Y+FhEPUQ2xkZn/Bqypw+FrgS9HxOH19TgfBD4Y1c0LVwK31/VN1uhfePdQ9S4taVH3PVTDl+0P2KLuMbvdByyOiMNGBbVlwL1dHP+LPBFKJmrfEhHnA9dHxKWZeX9d+4cz88Nj9+9wvv6Y6md0SmYORsSv0+Ww6xjtfnbbgUNHPZ4oUI0NJl8CPhAR64B5VDdK7H2d72bmK6ZQI1S9ZssiYm9AHqDqtX0V8Hftnlh/nq9qs8uZmfk9qssEHu8VjYjlVD3I69sc+xhgNdVwdTsDdPEZlUpmT5o0eX9JdU3Qa8b0RE3kn4D7gf8WEfOjutD/RRPsdxjV9V8/BwYi4v2M6mGIiLdExFMy8zGqC7kBHouIl0bEKfW1VNuohhcf2693B9Rh5h+Aj0XEwoh4UlQXzu8d3vtr4A8j4rSonDBqyO1xreoe81r3ANcCH6l/PqdS9cAdkHnGMvN2qt7GP6o3/RXwOxHxvLr2+RHxf0R1sXq783UYVc/P1og4muqasqm4GPjNiHhZ/XM9OiJ+sW67CTgnIg6K6uaI13dxvCupes0+RHW35d6f798DJ0XEW+vjHRQRvxQRz+x0wDqsHk81/Puc+utZVNe3jR3yHCfH3Ik5wdf36l2/CLwmquk15tfv4StjelXHeitwbX392uia31wP0VJ/Fj9MNUQr9S1DmjQJ9T/+v031S+unY4Y2x6mvoXoN1cX3d1NdHP+mCXa9mmp4Zj3VUN8w+w4/ngHcEhGPUl2Mf04dEJ8KfJkqoN1GdY3OJfv7PmtvAw4GbqW66P7LwNPq93UF1S/BS6mGpr5KdR3bWK3qHmsN1YX691FdZ/WBzPzfB+h9QHV90vkRcURmXk91ndon6ve1geqi907n64NUU5dspRpi/MpUCsnMfwJ+k+oav61U52xvwP3PVOHoofr1Lu3ieDvrWl4+ev866LySaij0Pqoh4z+h6qnaG2omvOGFajjx7zLzXzLzp3u/qM7hq6PFXGaTlZm3UN3s8kXgAaog/O/2tteXD/zHMU97G2NuGKidDFwbEduppuO4neo8S30rWgzpS5IkqYfsSZMkSSqQIU2SJKlAhjRJkqQCGdIkSZIKZEiTJEkqUN9NZrtkyZI87rjjel2GJElSRzfccMOWzHzKVJ7bdyHtuOOO4/rrr+91GZIkSR1FxETrGnfF4U5JkqQCGdIkSZIKZEiTJEkqkCFNkiSpQIY0SZKkAhnSJEmSCmRIkyRJKpAhTZIkqUCGNEmSpAIZ0iRJkgpkSJMkSSqQIU2SJKlAhjRJkqQCGdIkSZIKZEiTJEkqkCFNkiSpQIY0SZKkAjUW0iLiMxHxQET8pEV7RMR/j4gNEXFzRDy3qVokSZL6TZM9aZ8FzmjTfiZwYv11PvCXDdYiSZLUVxoLaZm5Dhhss8vZwOez8kPgyRHxtKbqkSRJ6icDPXzto4F7Rj3eXG+7v92Thodh/fomy9LixbBkSa+rkCRpdutlSOtaRJxPNSTKkiOPZt2mdT2uaObauWMOixfOY83q03pdiiRJs1ovQ9q9wDGjHi+tt42TmRcBFwGcdMpJuXLF/Oarm6U2bZzL4HC7UWpJkjQdejkFx1rgbfVdns8HtmZm26FOSZKk2aKxnrSI+BKwGlgSEZuBDwAHAWTm/wCuBF4FbACGgN9sqhZJkqR+01hIy8w1HdoT+N2mXl+SJKmfueKAJElSgQxpkiRJBTKkSZIkFciQJkmSVCBDmiRJUoEMaZIkSQUypEmSJBXIkCZJklQgQ5okSVKBDGmSJEkFMqRJkiQVyJAmSZJUIEOaJElSgQxpkiRJBTKkSZIkFciQJkmSVCBDmiRJUoEMaZIkSQUypEmSJBXIkCZJklQgQ5okSVKBDGmSJEkFMqRJkiQVaKDXBWjm2LIFBgfb77N4MSxZMj31SJLUzwxpOmAGB+HajTcxMrBtwvadO+aweOE81qw+bZorkySp/xjSNCntesuGhmBkYBsrV8yfsH3TxrkMDnfoapMkSYAhTZPUqbds7oLtwMQhTZIkdc+Qpn0MbZ/DjuEB1q9v0d6ht8yAJknSgWFI0zjDI8Os27SuZbu9ZZIkNc+QpnGGdw+zcsWRbfYwoEmS1DTnSZMkSSqQPWnax/wFezhk1+5elyFJ0qxnSNM+li3fybLlS3tdhiRJs57DnZIkSQWyJ02zWqelrFzGSpLUK4Y0zWrtJud1GStJUi8Z0jSjdeopazc5r8tYSZJ6yZCmGa3TMlbg5LySpDIZ0tT39mfR94oBTZJUHkOa+p6LvkuSZiJDmvrC/vWWGdAkSf3HkKa+YG+ZJGm2MaSpb9hbJkmaTQxpKkI3U2VIkjSbGNJUBKfKkCRpX4Y0FcOpMiRJeoIhTdNmaPscdgwPsH79BG0OZ0qStA9DmqbV8Mgw6zatm7DN4UxJkp5gSNO0Gt49zMoVR7Zo7a+A1ulmh8WLYcmS6atHkjSzGNKkKWp3s8POHXNYvHAea1af1oPKJEkzgSFN02b+gj0csmt3r8voWrtr6KD9SgebNs5lcLhNN1sPdOr5A3v/JKkkhjRNm2XLd7Js+dJelzEp7a6hg/66jq7TNCf2/klSWQxpUhvtr6GD0gLa/qxxWmLvnyTNZoY0qYV+G54F1ziVpJnEkCa1UOrw7P70lhnQJKl/GNKkPmNvmSTNDoY0qTDdLDZvb5kkzXyGNKkwLjYvSQJDmtQT+3ddGRjQJGnmM6RJPeB1ZZKkTgxpUgP2Z7WCyvQHtE41uxqBJE0vQ5rUkH5craBVza5GIEnTz5AmNaTfViuA1jW7GoEkTT9DmiSgP1dYkKSZzJAmNaAfA0+pKyxI0mxlSJMaYOCRJO2vJ/W6AEmSJI1nSJMkSSqQIU2SJKlAhjRJkqQCGdIkSZIKZEiTJEkqkCFNkiSpQIY0SZKkAhnSJEmSCmRIkyRJKpAhTZIkqUCGNEmSpAIZ0iRJkgpkSJMkSSqQIU2SJKlAhjRJkqQCGdIkSZIKZEiTJEkqkCFNkiSpQI2GtIg4IyJuj4gNEfGeCdqXRcQ1EXFjRNwcEa9qsh5JkqR+0VhIi4g5wCeBM4GTgTURcfKY3d4HXJ6ZK4FzgE81VY8kSVI/abIn7XRgQ2bekZm7gMuAs8fsk8DC+vtFwH0N1iNJktQ3Bho89tHAPaMebwaeN2afC4F/iIh/D8wHXj7RgSLifOB8gCOOOuKAFypJklSaXt84sAb4bGYuBV4FXBIR42rKzIsyc1Vmrlq0eNG0FylJkjTdmgxp9wLHjHq8tN422nnA5QCZ+QNgHrCkwZokSZL6QpMh7TrgxIh4ekQcTHVjwNox+9wNvAwgIp5JFdJ+3mBNkiRJfaGxkJaZI8AFwNXAbVR3cd4SER+KiLPq3d4FvCMifgx8CXh7ZmZTNUmSJPWLJm8cIDOvBK4cs+39o76/FXhRkzVIkiT1o17fOCBJkqQJGNIkSZIKZEiTJEkqkCFNkiSpQIY0SZKkAhnSJEmSCmRIkyRJKlCj86RJUq9s2QKDg63bFy+GJS5CJ6lghjRJM9LgIFy78SZGBraNa9u5Yw6LF85jzerTelCZJHXHkCapL3XqKRsagpGBbaxcMX9c26aNcxkcbvNkSSqAIU1SX2rXU7bX3AXbgfEhTZL6gSFN0n5r6vqvdsdt11P2BAOapP5lSJO035q6/qtTb5k9ZZJmMkOapAOiqeu/2veWGdAkzVyGNEkdDW2fw47hAdavb9E+NPVjdxrSlKTZypAmqSvDI8Os27SuZftUhx4d0pSkiRnSJHVlePcwK1cc2WaPiYNUN71wDmlK0niGNEkdzV+wh0N27Z7y85vqhZOkmcyQJqmjZct3smz50ik/f6q9cJI0m7nAuiRJUoHsSZPUqP0dKpWk2cqQJqlR+ztUKkmzlcOdkiRJBTKkSZIkFciQJkmSVCBDmiRJUoEMaZIkSQUypEmSJBXIkCZJklQgQ5okSVKBDGmSJEkFMqRJkiQVyGWhJM06Q9vnsGN4gPXrW++zeDEsWTJ9NUnSWIY0SbPS8Mgw6zatm7Bt5445LF44jzWrT5vmqiTpCYY0SbPS8O5hVq44csK2TRvnMjg8OM0VSdK+DGmSZp35C/ZwyK7dLds7DYc6FCppOhjSJM06y5bvZNnypW33aTUc6lCopOliSJOkCbQaDnUoVNJ0cQoOSZKkAtmTJkljdLpmTZKmgyFNksbo5po1SWqaw52SJEkFsidNkg6gLVtgsM19BU7fIalbhjRJOoAGB+HajTcxMrBtXJvTd0iaDEOaJE1Cp4luh4ZgZGAbK1fMH9fm9B2SJsOQJkmT1G7dT4C5C7YD40OaC7tLmgxDmiRNUrt1PyvjA9rjz3Vhd0ldMqRJ0jRyYXdJ3TKkSdIk7M9Et06SK2kyDGmSNAn7M9Gtk+RKmgwns5UkSSqQIU2SJKlAhjRJkqQCGdIkSZIKZEiTJEkqkCFNkiSpQIY0SZKkAhnSJEmSCmRIkyRJKpAhTZIkqUCGNEmSpAIZ0iRJkgrkAuuSVIih7XPYMTzA+vUTty9eDEuWTG9NknrHkCZJBRkeGWbdpnXjtu/cMYfFC+exZvVpPahKUi8Y0iSpIMO7h1m54shx22+7+VC2PPyQvWzSLGJIk6Q+YS+bNLsY0iSpEPMX7OGQXbtbtk+1lw1a97Rt2QKDg+3rspdO6g1DmiQVYtnynSxbvnRKz23Vywbte9oGB+HajTcxMrBt0s/tpFMANPxJ7RnSJKkPTLWXDdr3tA0NwcjANlaumD/hczdtnMvgcIeuthbaBUCHaKXODGmS1Af2p5cN2ve0zV2wHZg4pLXTqaesXQDcnyFaabYwpElSn+vUywbte9qmEtCg81AptA+AUx2ilWYLQ5ok9blOvWzdhLipajdUWr96y5Z2wXF/hlmlmcKQJkkz3P4MlbZbBWFoaOo17U9w9IYEzRaGNElSW01cz7Y/wdEbEjRbGNIkSW01cT1bJ5168FoNszpMqpnEkCZJaqnJ69k6aaIHT+onhjRJUkv7O/XH/uhFD55UEkOaJKk4vezBk0phSJMkFaeXPXhSKZ7U6wIkSZI0nj1pkqQZo91doXs5j5r6hSFNkjSjuNyUZgpDmiRpRnG5Kc0UhjRJ0ozhXaGaSRq9cSAizoiI2yNiQ0S8p8U+b4yIWyPiloi4tMl6JEkz27LlO3npKu8K1czQWE9aRMwBPgm8AtgMXBcRazPz1lH7nAi8F3hRZj4UEUc0VY8kSVI/aXK483RgQ2beARARlwFnA7eO2ucdwCcz8yGAzHygwXokSZqyLVuqxd1b8a5RHWhNhrSjgXtGPd4MPG/MPicBRMQ/AnOACzPzG2MPFBHnA+cDHHGUnW2SpKnpNEVHu6A1OAjXbryJkYFt49q8a1RN6PWNAwPAicBqYCmwLiJOycyHR++UmRcBFwGcdMpJOd1FSpJmjlZTdOzYPsChBx3Ki098zoTPGxqCkYFtrFwxft1Q7xpVE5oMafcCx4x6vLTeNtpm4EeZuRu4MyLWU4W26xqsS5I0i7WaouO2mw/l/kfvbznHGsDcBdtxcXdNlyZD2nXAiRHxdKpwdg7wG2P2+SqwBvibiFhCNfx5R4M1SZJmsU5TdLSbY60+woEvSmqhsZCWmSMRcQFwNdX1Zp/JzFsi4kPA9Zm5tm57ZUTcCuwB3p2ZDzZVkyRpdmu3cLtzrKk0jV6TlplXAleO2fb+Ud8n8Af1lyRJPdMuwEm90OhktpIkSZoaQ5okSVKBDGmSJEkFMqRJkiQVyJAmSZJUIEOaJElSgQxpkiRJBTKkSZIkFciQJkmSVKCuVxyIiKOBY0c/JzNbr0IrSZKkKesqpEXEnwBvAvausQmQgCFNkiSpAd32pP068IzM3NlkMZIkSap0e03aHcBBTRYiSZKkJ3TbkzYE3BQR3wIe703LzP/QSFWSJPWRoe1z2DE8wPr1E7c/8kj152GHTdy+eDEsWdJMbepf3Ya0tfWXJEmawPDIMOs2TXyp9o7t1a/bQ+aPjGvbuWMOixfOY83q0xqtT/2nq5CWmZ+LiIOBk+pNt2fm7ubKkiSpvwzvHmbliiMnbLvt5kO586E7eeHp49s3bZzL4PBg0+WpD3V7d+dq4HPAXUAAx0TEuU7BIUkSzF+wh0N2te676NQuTaTb4c6PAa/MzNsBIuIk4EuAfbOSpFlv2fKdLFu+dMrt0kS6vbvzoL0BDSAz1+PdnpIkSY3ptift+oj4a+AL9eM3A9c3U5IkSZK6DWn/N/C7wN4pN74HfKqRiiRJktT13Z07gY/XX5IkSWpY25AWEZdn5hsj4l+o1urcR2ae2lhlkiRJs1innrTfq/98ddOFSJIk6Qlt7+7MzPvrb7cA92TmJmAu8GzgvoZrkyRJmrW6vXFgHfDiiPgF4B+A64A3Ud3lKUmSpqjTup+u6zl7dRvSIjOHIuI84FOZ+dGIuKnJwiRJmi1arfvpup6zW9chLSJeQNVzdl69bU4zJUmSNLu0WvfTdT1nt25D2juB9wL/KzNviYjlwDXNlSVJ0uzQ1LqeW7bAYJt898gj1Z+HHTZxu8OsvdftPGnfBb476vEdPDGxrSRJmqKm1vUcHIRrN97EyMC2Cdt3bK8iwCHzR8a1Ocxahk7zpP15Zr4zIr7GxPOkndVYZZIkzXKdbipo1xs2NAQjA9tYuWL+hM+97eZDufOhO3nh6Q6zlqpTT9ol9Z//b9OFSJKk8VrdVADte8MA5i7YDkwc0toNs3YKh+Bw6HRoG9Iy84b62+uBHZn5GEBEzKGaL02SJDWo1U0F0L43rDJxQIPOw6ztwqHDodOj2xsHvgW8HHi0fnwI1XxpL2yiKEmS1PmmgqZuOpi/YA+x6xFWrpg4xDkcOj26DWnzMnNvQCMzH42IQxuqSZIk0bm3q6mbDpo6rian7bJQo2yPiOfufRARpwE7milJkiRJk5kn7YqIuA8I4KlUy0JJkiQ9rtP8bOBNB93qdp606yLiF4Fn1Jtuz8wDPwguSZL6Wqf52bzpoHtdhbT6+rM/AI7NzHdExIkR8YzM/Ptmy5MkSaVpN0VHp/nZvOmge90Od/4NcAPwgvrxvcAVgCFNkqRZqN0UHe3mZ1P3ug1px2fmmyJiDUBmDkVENFhXS4duu4+Tvvb7+2x7aPlqfr7ibGJkmBOveu+45zx40q/x4DPOYM7wVo7/5oXj2n9+8lk8dPxLOejRB3j6NR8Z1/6zU9/A1mNfyNyH7+bY7/3ZuPb7V76FR5aexiFbNnDMDz45rv3eXzqP7U99FvN/+hOOvu7ice33vOB32bHkBA7bfANPu/EL49o3vfj32fnkZSzadC1H3nzFuPY7X/pedi84gl/YeA1PuXXtuPaNr7iQPfMWcfjt3+Dw9VePa/+3Mz9CDszjKbf8Hb9wx3fGta9/TfWej/zx37Lo7h/u0/bYwFw2nPnfAHjqP1/Cwnv/eZ/2kbkLueOVHwTgqH/6Kxb87NZ92nfNfwp3/ep/BGDptZ/g0Ac37tM+vGgpd7/kXQAsW/cx5m3dvE/70OHHs/mFFwBw3Lf/mIO3/3yf9kePPJn7Tn8HAMv/4QMM7Ny3+33b0c/lp899KwAnXPUenjSyc5/2rcuez8+eXV1+OfZzB372/Oz52fOzNzs/e7exhl3Dj/CmBz82rv3Bk36NB49t/dl7+LBzGXzKibB1M3zlt8e188IL4BlnwpZ/g6+9c3z7S/4Qjn8p3H8zfGP8Z5+XvR+WPQ/u/hF860Pj28/4CDztVNh4DaybYK7+1/w5LDkRbr8Krv3E+PbXfhoWLYWf/E+47jPj29/4eZh/ONz4Rbjp0vHtk9Dt3Z27IuIQ6qWhIuJ4YGf7p0iSpJlo/oI9HDLfS9ObFpnjluQcv1PEK4D3ASdTTWL7IuDtmfmdRqubwEmnnJRfuvpL0/2ykiTpANh7TdpvveIlvS5lWkTEDZm5airP7TjcWQ9r/ivwWuD5VFNw/F5mbpnKC0qSJKmzjiEtMzMirszMU4CvT0NNkiRJs16316T9c0T8UqOVSJIk6XHd3t35POAtEXEXsJ1qyDMz89SmCpMkSZrNug1pv9ZoFZIkSdpH25AWEfOA3wFOAP4FuDgzR6ajMEmSpNms0zVpnwNWUQW0M4Hxs9ZJkiTpgOs03HlyfVcnEXEx8E/NlyRJkqROPWmPTyfsMKckSdL06dST9uyI2LvoVwCH1I/33t25sNHqJEmSZqm2IS0z50xXIZIkSXpCt5PZSpIkaRoZ0iRJkgpkSJMkSSpQtysOSJIk7beh7XPYMTzA+vUTty9eDEuWTG9NpTKkSZKkaTU8Msy6TevGbd+5Yw6LF85jzerTelBVeQxpkiRpWg3vHmbliiPHbd+0cS6Dw4M9qKhMhjRJkjRt5i/YwyG7dnfeUYY0SZI0fZYt38my5Usn/bwtW2CwQyfbTLuezZAmSZKKNzgI1268iZGBbRO2z8Tr2QxpkiSpCO3u/BwagpGBbaxcMX/C587E69kMaZIkqRit7vwEmLtgOzBxSJuJDGmSJKkI8xfsIXY9wsoVra5Zmz0BDQxpkiSpEFO9qWB0/TVxAAAN7UlEQVSmclkoSZKkAhnSJEmSCmRIkyRJKpAhTZIkqUCGNEmSpAIZ0iRJkgpkSJMkSSqQIU2SJKlAhjRJkqQCGdIkSZIKZEiTJEkqkGt3SpKkvje0fQ47hgdYv37i9sWLYcmS6a1pfxnSJEnSjDA8Msy6TevGbd+5Yw6LF85jzerTelDV1BnSJEnSjDC8e5iVK44ct33TxrkMDg/2oKL9Y0iTJEl9b/6CPRyya3evyzigDGmSJKnvLVu+k2XLl/a6jAPKuzslSZIK1GhIi4gzIuL2iNgQEe9ps9/rIiIjYlWT9UiSJPWLxoY7I2IO8EngFcBm4LqIWJuZt47Z7zDg94AfNVWLJEmavTpNzwFlTtHR5DVppwMbMvMOgIi4DDgbuHXMfv8F+BPg3Q3WIkmSZrFW03NAuVN0NBnSjgbuGfV4M/C80TtExHOBYzLz6xHRMqRFxPnA+QBHHHVEA6VKkqSZrNX0HFDuFB09u3EgIp4EfBx4V6d9M/OizFyVmasWLV7UfHGSJGnGmL9gD4fM77/pOZrsSbsXOGbU46X1tr0OA54FfCciAJ4KrI2IszLz+gbrkiRJs0i/Ts/RZE/adcCJEfH0iDgYOAdYu7cxM7dm5pLMPC4zjwN+CBjQJEmSaDCkZeYIcAFwNXAbcHlm3hIRH4qIs5p6XUmSpJmg0RUHMvNK4Mox297fYt/VTdYiSZLUT1wWSpIkzWqd5lHr1RxqhjRJkjTrtZpHrZdzqBnSJEnSrDZ/wR5i1yOsXDH+DtBezqFmSJMkSbPa/kzRsWULDDaU4QxpkiRJUzQ4CNduvImRgW0T73Aw86d6bEOaJElSC51uKhgagpGBbaxc0SKLPYk5U31tQ5okSVIb7RZnB5i7YDtMvcOsJUOaJElSC+1uKhi1VyOvbUiTJElqoZfrfja5dqckSZKmyJAmSZJUIEOaJElSgQxpkiRJBTKkSZIkFciQJkmSVCBDmiRJUoEMaZIkSQUypEmSJBXIkCZJklQgQ5okSVKBDGmSJEkFMqRJkiQVyJAmSZJUIEOaJElSgQxpkiRJBTKkSZIkFciQJkmSVCBDmiRJUoEMaZIkSQUypEmSJBXIkCZJklQgQ5okSVKBDGmSJEkFMqRJkiQVyJAmSZJUIEOaJElSgQxpkiRJBTKkSZIkFciQJkmSVCBDmiRJUoEMaZIkSQUypEmSJBXIkCZJklQgQ5okSVKBDGmSJEkFMqRJkiQVyJAmSZJUIEOaJElSgQxpkiRJBTKkSZIkFciQJkmSVCBDmiRJUoEMaZIkSQUypEmSJBXIkCZJklQgQ5okSVKBDGmSJEkFMqRJkiQVyJAmSZJUIEOaJElSgQxpkiRJBTKkSZIkFciQJkmSVCBDmiRJUoEMaZIkSQUypEmSJBXIkCZJklQgQ5okSVKBDGmSJEkFMqRJkiQVyJAmSZJUIEOaJElSgQxpkiRJBTKkSZIkFciQJkmSVCBDmiRJUoEMaZIkSQUypEmSJBXIkCZJklQgQ5okSVKBDGmSJEkFMqRJkiQVyJAmSZJUIEOaJElSgRoNaRFxRkTcHhEbIuI9E7T/QUTcGhE3R8S3IuLYJuuRJEnqF42FtIiYA3wSOBM4GVgTESeP2e1GYFVmngp8GfhoU/VIkiT1kyZ70k4HNmTmHZm5C7gMOHv0Dpl5TWYO1Q9/CCxtsB5JkqS+0WRIOxq4Z9TjzfW2Vs4DrmqwHkmSpL4x0OsCACLiLcAq4FdatJ8PnA9wxFFHTGNlkiRJvdFkT9q9wDGjHi+tt+0jIl4O/CfgrMzcOdGBMvOizFyVmasWLV7USLGSJEklaTKkXQecGBFPj4iDgXOAtaN3iIiVwKepAtoDDdYiSZLUVxoLaZk5AlwAXA3cBlyembdExIci4qx6tz8FFgBXRMRNEbG2xeEkSZJmlUavScvMK4Erx2x7/6jvX97k60uSJPUrVxyQJEkqkCFNkiSpQIY0SZKkAhnSJEmSCmRIkyRJKpAhTZIkqUCGNEmSpAIZ0iRJkgpkSJMkSSqQIU2SJKlAhjRJkqQCGdIkSZIKZEiTJEkqkCFNkiSpQIY0SZKkAhnSJEmSCmRIkyRJKpAhTZIkqUCGNEmSpAIZ0iRJkgpkSJMkSSqQIU2SJKlAhjRJkqQCGdIkSZIKZEiTJEkqkCFNkiSpQIY0SZKkAhnSJEmSCmRIkyRJKpAhTZIkqUCGNEmSpAIZ0iRJkgpkSJMkSSqQIU2SJKlAhjRJkqQCGdIkSZIKZEiTJEkqkCFNkiSpQIY0SZKkAhnSJEmSCmRIkyRJKpAhTZIkqUCGNEmSpAIZ0iRJkgpkSJMkSSqQIU2SJKlAhjRJkqQCGdIkSZIKZEiTJEkqkCFNkiSpQIY0SZKkAhnSJEmSCmRIkyRJKpAhTZIkqUCGNEmSpAIZ0iRJkgpkSJMkSSqQIU2SJKlAhjRJkqQCGdIkSZIKZEiTJEkqkCFNkiSpQIY0SZKkAhnSJEmSCmRIkyRJKpAhTZIkqUCGNEmSpAIZ0iRJkgpkSJMkSSqQIU2SJKlAhjRJkqQCGdIkSZIKZEiTJEkqkCFNkiSpQIY0SZKkAhnSJEmSCmRIkyRJKpAhTZIkqUCGNEmSpAIZ0iRJkgpkSJMkSSqQIU2SJKlAhjRJkqQCGdIkSZIK1GhIi4gzIuL2iNgQEe+ZoH1uRPxt3f6jiDiuyXokSZL6RWMhLSLmAJ8EzgROBtZExMljdjsPeCgzTwD+DPiTpuqRJEnqJ032pJ0ObMjMOzJzF3AZcPaYfc4GPld//2XgZRERDdYkSZLUF5oMaUcD94x6vLneNuE+mTkCbAUOb7AmSZKkvjDQ6wK6ERHnA+fXD3evOn7VXT0sR/tjhEUMsLXXZWgKPHf9zfPXvzx3/W0nS6f61CZD2r3AMaMeL623TbTP5ogYABYBD449UGZeBFwEEBHX545c1UjFalxEXJ+7PX/9yHPX3zx//ctz198i4vqpPrfJ4c7rgBMj4ukRcTBwDrB2zD5rgXPr718PfDszs8GaJEmS+kJjPWmZORIRFwBXA3OAz2TmLRHxIeD6zFwLXAxcEhEbgEGqICdJkjTrNXpNWmZeCVw5Ztv7R30/DLxhkoe96ACUpt7x/PUvz11/8/z1L89df5vy+QtHFyVJksrjslCSJEkFKjakuaRU/+ri3P1BRNwaETdHxLci4the1KmJdTp/o/Z7XURkRHjXWUG6OX8R8cb67+AtEXHpdNeoiXXxb+eyiLgmIm6s//18VS/q1HgR8ZmIeCAiftKiPSLiv9fn9uaIeG43xy0ypLmkVP/q8tzdCKzKzFOpVpr46PRWqVa6PH9ExGHA7wE/mt4K1U435y8iTgTeC7woM1cA75z2QjVOl3/33gdcnpkrqW60+9T0Vqk2Pguc0ab9TODE+ut84C+7OWiRIQ2XlOpnHc9dZl6TmUP1wx/C1Cf60wHXzd89gP9C9R+j4eksTh11c/7eAXwyMx8CyMwHprlGTaybc5fAwvr7RcB901if2sjMdVSzVLRyNvD5rPwQeHJEPK3TcUsNaS4p1b+6OXejnQdc1WhFmoyO56/upj8mM78+nYWpK938/TsJOCki/jEifhgR7f73r+nTzbm7EHhLRGymmjnh309PaToAJvu7EeiTZaE0M0XEW4BVwK/0uhZ1JyKeBHwceHuPS9HUDVANuaym6sVeFxGnZObDPa1K3VgDfDYzPxYRL6CaZ/RZmflYrwtTM0rtSZvMklK0W1JK066bc0dEvBz4T8BZmblzmmpTZ53O32HAs4DvRMRdwPOBtd48UIxu/v5tBtZm5u7MvBNYTxXa1FvdnLvzgMsBMvMHwDxgybRUp/3V1e/GsUoNaS4p1b86nruIWAl8miqgeT1MWdqev8zcmplLMvO4zDyO6prCszJzymvT6YDq5t/Or1L1ohERS6iGP++YziI1oW7O3d3AywAi4plUIe3n01qlpmot8Lb6Ls/nA1sz8/5OTypyuNMlpfpXl+fuT4EFwBX1vR53Z+ZZPStaj+vy/KlQXZ6/q4FXRsStwB7g3ZnpKESPdXnu3gX8VUT8PtVNBG+3c6IMEfElqv/8LKmvGfwAcBBAZv4PqmsIXwVsAIaA3+zquJ5fSZKk8pQ63ClJkjSrGdIkSZIKZEiTJEkqkCFNkiSpQIY0SZKkAhnSJM0oEbEnIm6KiJ9ExNci4skH+Phvj4hP1N9fGBF/eCCPL0l7GdIkzTQ7MvM5mfksqjkUf7fXBUnSVBjSJM1kP2DUIsYR8e6IuC4ibo6ID47a/rZ6248j4pJ622si4kcRcWNE/O+IOLIH9UuaxYpccUCS9ldEzKFaQufi+vErqdaoPB0IqjVHX0K15u/7gBdm5paIWFwf4vvA8zMzI+K3gD+imvFdkqaFIU3STHNIRNxE1YN2G/DNevsr668b68cLqELbs4ErMnMLQGYO1u1Lgb+NiKcBBwN3Tk/5klRxuFPSTLMjM58DHEvVY7b3mrQAPlJfr/aczDwhMy9uc5z/D/hEZp4C/DbVYtaSNG0MaZJmpMwcAv4D8K6IGKBauPr/iogFABFxdEQcAXwbeENEHF5v3zvcuQi4t/7+3GktXpJwuFPSDJaZN0bEzcCazLwkIp4J/CAiAB4F3pKZt0TEh4HvRsQequHQtwMXAldExENUQe7pvXgPkmavyMxe1yBJkqQxHO6UJEkqkCFNkiSpQIY0SZKkAhnSJEmSCmRIkyRJKpAhTZIkqUCGNEmSpAIZ0iRJkgr0/wMXqaB7kBe2yAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x648 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#::GMG::Plot Recall-Precision Curve\n",
    "step_kwargs = ({'step': 'post'}\n",
    "               if 'step' in signature(plt.fill_between).parameters\n",
    "               else {})\n",
    "plt.figure(figsize = (10,9))\n",
    "# plot no skill\n",
    "plt.plot([0, 1], [0.5, 0.5], linestyle='--', color = 'C1')\n",
    "plt.step(recall_cv, precision_cv, color='b', alpha=0.2, where='post')\n",
    "plt.fill_between(recall_cv, precision_cv, alpha=0.2, color='g', **step_kwargs)\n",
    "plt.xlabel('Recall')\n",
    "plt.ylabel('Precision')\n",
    "plt.ylim([0.0, 1.05])\n",
    "plt.xlim([0.0, 1.0])\n",
    "plt.title('2-class Precision-Recall curve: AP={0:0.2f}'.format(apc_cv))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Notes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Score (train): 0.658 \n",
      "Score (test): 0.706\n"
     ]
    }
   ],
   "source": [
    "#::GMG::Scores comparison (?)\n",
    "# https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.RFE.html#sklearn.feature_selection.RFE.score\n",
    "print('Score (train): {:4.3f} \\nScore (test): {:4.3f}'.format(\n",
    "    rfe.score(train_features, train_target),\n",
    "    rfe.score(test_features, test_target)\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Score CV (train): 0.773 \n",
      "Score CV (test): 0.757\n"
     ]
    }
   ],
   "source": [
    "#::GMG::Scores comparison (?)\n",
    "# https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.RFE.html#sklearn.feature_selection.RFE.score\n",
    "print('Score CV (train): {:4.3f} \\nScore CV (test): {:4.3f}'.format(\n",
    "    rfecv.score(train_features, train_target),\n",
    "    rfecv.score(test_features, test_target)\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "**::GMG::** If I want to use the (split) dataset with the selected features with other classification algorithms I can get them from `rfe` object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "#::GMG::Perform the transformation of dataset with the selected features\n",
    "# https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.RFE.html#sklearn.feature_selection.RFE.transform\n",
    "train_features_fs = rfe.transform(X = train_features)\n",
    "test_features_fs = rfe.transform(X = test_features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(506, 3)"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_features_fs.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(218, 3)"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_features_fs.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "#::GMG::Get the 'new' dataset pandas dataframe with the selected features\n",
    "#       I'm not sure if this is necessary at all ... maybe if you're going to use it\n",
    "#       further for ML task ...\n",
    "train_fs = pd.DataFrame(\n",
    "    data = train_features_fs, \n",
    "    columns = [value for value, supported in zip(train_features.columns, rfe.support_) if supported]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "#::GMG::Adding new column to existing DataFrame in Python pandas\n",
    "# https://stackoverflow.com/questions/12555323/adding-new-column-to-existing-dataframe-in-python-pandas\n",
    "train_fs = train_fs.assign(Outcome = train_target.values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Pregnancies</th>\n",
       "      <th>BMI</th>\n",
       "      <th>DiabetesPedigreeFunction</th>\n",
       "      <th>Outcome</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>4.0</td>\n",
       "      <td>31.2</td>\n",
       "      <td>1.182</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4.0</td>\n",
       "      <td>31.2</td>\n",
       "      <td>0.539</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2.0</td>\n",
       "      <td>29.8</td>\n",
       "      <td>0.717</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>8.0</td>\n",
       "      <td>28.7</td>\n",
       "      <td>0.687</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3.0</td>\n",
       "      <td>31.9</td>\n",
       "      <td>0.591</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Pregnancies   BMI  DiabetesPedigreeFunction  Outcome\n",
       "0          4.0  31.2                     1.182        1\n",
       "1          4.0  31.2                     0.539        1\n",
       "2          2.0  29.8                     0.717        0\n",
       "3          8.0  28.7                     0.687        1\n",
       "4          3.0  31.9                     0.591        0"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_fs.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(506, 4)"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_fs.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "test_fs = pd.DataFrame(\n",
    "    data = test_features_fs, \n",
    "    columns = [value for value, supported in zip(test_features.columns, rfe.support_) if supported]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "test_fs = test_fs.assign(Outcome = test_target.values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Pregnancies</th>\n",
       "      <th>BMI</th>\n",
       "      <th>DiabetesPedigreeFunction</th>\n",
       "      <th>Outcome</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3.0</td>\n",
       "      <td>34.0</td>\n",
       "      <td>0.271</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>7.0</td>\n",
       "      <td>30.4</td>\n",
       "      <td>0.383</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.0</td>\n",
       "      <td>29.9</td>\n",
       "      <td>0.422</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.0</td>\n",
       "      <td>44.1</td>\n",
       "      <td>0.686</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3.0</td>\n",
       "      <td>31.6</td>\n",
       "      <td>0.268</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Pregnancies   BMI  DiabetesPedigreeFunction  Outcome\n",
       "0          3.0  34.0                     0.271        0\n",
       "1          7.0  30.4                     0.383        1\n",
       "2          1.0  29.9                     0.422        0\n",
       "3          0.0  44.1                     0.686        1\n",
       "4          3.0  31.6                     0.268        1"
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#::GMG::Outcome yields NaN and I don't know why!!!\n",
    "test_fs.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(218, 4)"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_fs.shape"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}