7.3-Getting_the_most_out_of_your_models.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Companion Notebook - 7.3 Getting the most out of your models \n",
    "## updated for TensorFlow & Keras 2.x\n",
    "## Chap 7 « Advanced Deep-learning best practices » \n",
    "## « Deep Learning with Python » book by François Chollet \n",
    "\n",
    "This notebook contains the code samples found in Chapter 7 of «Deep Learning with Python». Note that the original text features far more content, in particular further explanations and figures. In this companion Notebook, you will find source code along with small corrections and some additions by Claude COULOMBE - PhD - Montréal."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Keras version: 2.3.1\n",
      "TensorFlow version: 2.2.0\n"
     ]
    }
   ],
   "source": [
    "# sudo pip3 install --ignore-installed --upgrade tensorflow\n",
    "import keras\n",
    "print(\"Keras version:\",keras.__version__)\n",
    "import tensorflow as tf\n",
    "print(\"TensorFlow version:\",tf.__version__)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## 7.3. Getting the most out of your models\n",
    "\n",
    "In this section, we’ll go beyond “works okay” to “works great and wins machine-learning competitions” by offering you a quick guide to a set of must-know techniques for building state-of-the-art deep-learning models. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7.3.1. Advanced architecture patterns \n",
    "\n",
    "We covered one important design pattern in detail in the previous section: residual connections. There are two more design patterns you should know about: normalization and depthwise separable convolution. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Normalization\n",
    "\n",
    "<i>Normalization</i> is a broad category of methods that seek to make different samples seen by a machine-learning model more similar to each other, which helps the model learn and generalize well to new data. The most common form of data normalization is one you’ve seen several times in this book already: centering the data on 0 by subtracting the mean from the data, and giving the data a unit standard deviation by dividing the data by its standard deviation based on the assumption that the data follows a normal (or Gaussian) distribution:\n",
    "\n",
    "```Python\n",
    "    normalized_data = (data - np.mean(data, axis=...)) / np.std(data, axis=...)\n",
    "```\n",
    "\n",
    "#### Batch normalization \n",
    "\n",
    "Data normalization should be a concern after every transformation operated by the network: even if the data entering a Dense or Conv2D network has a 0 mean and unit variance, there’s no reason to expect a priori that this will be the case for the data coming out. \n",
    "\n",
    "<i>Batch normalization</i> is a type of layer (`BatchNormalization` in Keras) introduced in 2015 by Ioffe and Szegedy that can adaptively normalize data even as the mean and variance change over time during training. It works by internally maintaining an exponential moving average of the batch-wise mean and variance of the data seen during training. The main effect of batch normalization is that it helps with gradient propagation and allows deeper networks. For instance, BatchNormalization is used liberally in many of the advanced convnet architectures that come packaged with Keras, such as ResNet50, Inception V3, and Xception.  \n",
    "\n",
    "The `BatchNormalization` layer is typically used after a convolutional or densely connected layer:\n",
    "\n",
    "```Python\n",
    "    conv_model.add(layers.Conv2D(32, 3, activation='relu'))\n",
    "    # Batch normalization used after a Conv layer\n",
    "    conv_model.add(layers.BatchNormalization())\n",
    "\n",
    "    dense_model.add(layers.Dense(32, activation='relu'))\n",
    "    # Batch normalization used after a Dense layer\n",
    "    dense_model.add(layers.BatchNormalization())\n",
    "```\n",
    "The `BatchNormalization` layer takes an axis argument, which specifies the feature axis that should be normalized. This argument defaults to -1, the last axis in the input tensor. This is the correct value when using Dense layers, Conv1D layers, RNN layers, and Conv2D layers with data_format set to \"channels_last\". But in the niche use case of Conv2D layers with data_format set to \"channels_first\", the features axis is axis 1; the axis argument in BatchNormalization should accordingly be set to 1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Batch renormalization \n",
    "A recent improvement over regular batch normalization is batch renormalization, introduced by Ioffe in 2017. It offers clears benefits over batch normalization, at no apparent cost. At the time of writing, it’s too early to tell whether it will supplant batch normalization. Even more recently, Klambauer et al. introduced self-normalizing neural networks,which manage to keep data normalized after going through any Dense layer by using a specific activation function (`selu`) and a specific initializer (`lecun_normal`). This scheme, although highly interesting, is limited to densely connected networks for now, and its usefulness hasn’t yet been broadly replicated."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Depthwise separable convolution \n",
    "<i>Depthwise separable convolution layer</i> can make a model lighter (fewer trainable weight parameters) and faster (fewer floating-point operations) and cause it to perform a few percentage points better on its task. The For example, SeparableConv2D layer performs a spatial convolution on each channel of its input, independently, before mixing output channels via a pointwise convolution (a 1 × 1 convolution). This is equivalent to separating the learning of spatial features and the learning of channel-wise features. It requires significantly fewer parameters and involves fewer computations, thus resulting in smaller, speedier models. And because it’s a more representationally efficient way to perform convolution, it tends to learn better representations using less data, resulting in better-performing models. \n",
    "\n",
    "These advantages become especially important when you’re training small models from scratch on limited data. For instance, here’s how you can build a lightweight, depthwise separable convnet for an image-classification task (softmax categorical classification) on a small dataset:\n",
    "\n",
    "When it comes to larger-scale models, depthwise separable convolutions are the basis of the Xception architecture, a high-performing convnet that comes packaged with Keras. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training a Depthwise separable convolution\n",
    "#####  on the CIFAR10 dataset\n",
    "\n",
    "The original code above requires an imges dataset in the format 64 height x 64 width x 3 channels. Furthermore, in order to work, depthwise separable convolution needs multichannel data, so MNIST dataset is not appropriate (28x28x1) since that has only one channel. Fortunately, KERAS has the CIFAR10 dataset which is in the format (32x32x3), so 3 channels. \n",
    "\n",
    "Therefore, we will adapt a code example from the KERAS GitHub repo: https://github.com/keras-team/keras/blob/master/examples/mnist_cnn.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x_train shape: (50000, 32, 32, 3)\n",
      "50000 train samples\n",
      "10000 test samples\n"
     ]
    }
   ],
   "source": [
    "# https://github.com/keras-team/keras/blob/master/examples/mnist_cnn.py\n",
    "\n",
    "'''Trains a Depthwise separable convolution on the CIFAR10 dataset.\n",
    "'''\n",
    "\n",
    "from __future__ import print_function\n",
    "import keras\n",
    "from keras.datasets import cifar10\n",
    "from keras.models import Sequential\n",
    "from keras import layers\n",
    "from keras.layers import Dense, Dropout, Flatten\n",
    "from keras.layers import Conv2D, MaxPooling2D\n",
    "from keras import backend as K\n",
    "\n",
    "batch_size = 128\n",
    "num_classes = 10\n",
    "epochs = 12\n",
    "\n",
    "# input image dimensions\n",
    "img_height = 32\n",
    "img_width = 32\n",
    "channels = 3\n",
    "\n",
    "# the data, split between train and test sets\n",
    "(x_train, y_train), (x_test, y_test) = cifar10.load_data()\n",
    "\n",
    "if K.image_data_format() == 'channels_first':\n",
    "    x_train = x_train.reshape(x_train.shape[0], channels, img_height, img_width)\n",
    "    x_test = x_test.reshape(x_test.shape[0], channels, img_height, img_width)\n",
    "    input_shape = (channels, img_height, img_width)\n",
    "else:\n",
    "    x_train = x_train.reshape(x_train.shape[0], img_height, img_width, channels)\n",
    "    x_test = x_test.reshape(x_test.shape[0], img_height, img_width, channels)\n",
    "    input_shape = (img_height, img_width, channels)\n",
    "\n",
    "x_train = x_train.astype('float32')\n",
    "x_test = x_test.astype('float32')\n",
    "x_train /= 255\n",
    "x_test /= 255\n",
    "print('x_train shape:', x_train.shape)\n",
    "print(x_train.shape[0], 'train samples')\n",
    "print(x_test.shape[0], 'test samples')\n",
    "\n",
    "# convert class vectors to binary class matrices\n",
    "y_train = keras.utils.to_categorical(y_train, num_classes)\n",
    "y_test = keras.utils.to_categorical(y_test, num_classes)\n",
    "\n",
    "model_a = Sequential()\n",
    "model_a.add(layers.SeparableConv2D(32, 3,\n",
    "                                 activation='relu',\n",
    "                                 input_shape=(img_height, img_width, channels,))) \n",
    "model_a.add(layers.SeparableConv2D(64, 3, activation='relu'))\n",
    "model_a.add(layers.MaxPooling2D(2))\n",
    "model_a.add(layers.SeparableConv2D(64, 3, activation='relu'))\n",
    "model_a.add(layers.SeparableConv2D(128, 3, activation='relu'))\n",
    "model_a.add(layers.MaxPooling2D(2)) \n",
    "model_a.add(layers.SeparableConv2D(64, 3, activation='relu')) \n",
    "model_a.add(layers.SeparableConv2D(128, 3, activation='relu')) \n",
    "model_a.add(layers.GlobalAveragePooling2D())\n",
    "model_a.add(layers.Dense(32, activation='relu'))\n",
    "model_a.add(layers.Dense(num_classes, activation='softmax')) \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<svg height=\"807pt\" viewBox=\"0.00 0.00 497.00 968.00\" width=\"414pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g class=\"graph\" id=\"graph0\" transform=\"scale(0.83 0.83) rotate(0) translate(4 964)\">\n",
       "<title>G</title>\n",
       "<polygon fill=\"white\" points=\"-4,4 -4,-964 493,-964 493,4 -4,4\" stroke=\"transparent\"/>\n",
       "<!-- 5374952168 -->\n",
       "<g class=\"node\" id=\"node1\">\n",
       "<title>5374952168</title>\n",
       "<polygon fill=\"none\" points=\"46.5,-913.5 46.5,-959.5 442.5,-959.5 442.5,-913.5 46.5,-913.5\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"161.5\" y=\"-932.8\">separable_conv2d_1_input: InputLayer</text>\n",
       "<polyline fill=\"none\" points=\"276.5,-913.5 276.5,-959.5 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"304\" y=\"-944.3\">input:</text>\n",
       "<polyline fill=\"none\" points=\"276.5,-936.5 331.5,-936.5 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"304\" y=\"-921.3\">output:</text>\n",
       "<polyline fill=\"none\" points=\"331.5,-913.5 331.5,-959.5 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"387\" y=\"-944.3\">(None, 32, 32, 3)</text>\n",
       "<polyline fill=\"none\" points=\"331.5,-936.5 442.5,-936.5 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"387\" y=\"-921.3\">(None, 32, 32, 3)</text>\n",
       "</g>\n",
       "<!-- 5374844376 -->\n",
       "<g class=\"node\" id=\"node2\">\n",
       "<title>5374844376</title>\n",
       "<polygon fill=\"none\" points=\"41,-830.5 41,-876.5 448,-876.5 448,-830.5 41,-830.5\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"158\" y=\"-849.8\">separable_conv2d_1: SeparableConv2D</text>\n",
       "<polyline fill=\"none\" points=\"275,-830.5 275,-876.5 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"302.5\" y=\"-861.3\">input:</text>\n",
       "<polyline fill=\"none\" points=\"275,-853.5 330,-853.5 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"302.5\" y=\"-838.3\">output:</text>\n",
       "<polyline fill=\"none\" points=\"330,-830.5 330,-876.5 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"389\" y=\"-861.3\">(None, 32, 32, 3)</text>\n",
       "<polyline fill=\"none\" points=\"330,-853.5 448,-853.5 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"389\" y=\"-838.3\">(None, 30, 30, 32)</text>\n",
       "</g>\n",
       "<!-- 5374952168&#45;&gt;5374844376 -->\n",
       "<g class=\"edge\" id=\"edge1\">\n",
       "<title>5374952168-&gt;5374844376</title>\n",
       "<path d=\"M244.5,-913.37C244.5,-905.15 244.5,-895.66 244.5,-886.73\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"248,-886.61 244.5,-876.61 241,-886.61 248,-886.61\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 5374951832 -->\n",
       "<g class=\"node\" id=\"node3\">\n",
       "<title>5374951832</title>\n",
       "<polygon fill=\"none\" points=\"41,-747.5 41,-793.5 448,-793.5 448,-747.5 41,-747.5\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"158\" y=\"-766.8\">separable_conv2d_2: SeparableConv2D</text>\n",
       "<polyline fill=\"none\" points=\"275,-747.5 275,-793.5 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"302.5\" y=\"-778.3\">input:</text>\n",
       "<polyline fill=\"none\" points=\"275,-770.5 330,-770.5 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"302.5\" y=\"-755.3\">output:</text>\n",
       "<polyline fill=\"none\" points=\"330,-747.5 330,-793.5 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"389\" y=\"-778.3\">(None, 30, 30, 32)</text>\n",
       "<polyline fill=\"none\" points=\"330,-770.5 448,-770.5 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"389\" y=\"-755.3\">(None, 28, 28, 64)</text>\n",
       "</g>\n",
       "<!-- 5374844376&#45;&gt;5374951832 -->\n",
       "<g class=\"edge\" id=\"edge2\">\n",
       "<title>5374844376-&gt;5374951832</title>\n",
       "<path d=\"M244.5,-830.37C244.5,-822.15 244.5,-812.66 244.5,-803.73\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"248,-803.61 244.5,-793.61 241,-803.61 248,-803.61\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 5374952728 -->\n",
       "<g class=\"node\" id=\"node4\">\n",
       "<title>5374952728</title>\n",
       "<polygon fill=\"none\" points=\"54.5,-664.5 54.5,-710.5 434.5,-710.5 434.5,-664.5 54.5,-664.5\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"158\" y=\"-683.8\">max_pooling2d_1: MaxPooling2D</text>\n",
       "<polyline fill=\"none\" points=\"261.5,-664.5 261.5,-710.5 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"289\" y=\"-695.3\">input:</text>\n",
       "<polyline fill=\"none\" points=\"261.5,-687.5 316.5,-687.5 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"289\" y=\"-672.3\">output:</text>\n",
       "<polyline fill=\"none\" points=\"316.5,-664.5 316.5,-710.5 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"375.5\" y=\"-695.3\">(None, 28, 28, 64)</text>\n",
       "<polyline fill=\"none\" points=\"316.5,-687.5 434.5,-687.5 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"375.5\" y=\"-672.3\">(None, 14, 14, 64)</text>\n",
       "</g>\n",
       "<!-- 5374951832&#45;&gt;5374952728 -->\n",
       "<g class=\"edge\" id=\"edge3\">\n",
       "<title>5374951832-&gt;5374952728</title>\n",
       "<path d=\"M244.5,-747.37C244.5,-739.15 244.5,-729.66 244.5,-720.73\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"248,-720.61 244.5,-710.61 241,-720.61 248,-720.61\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 5374951776 -->\n",
       "<g class=\"node\" id=\"node5\">\n",
       "<title>5374951776</title>\n",
       "<polygon fill=\"none\" points=\"41,-581.5 41,-627.5 448,-627.5 448,-581.5 41,-581.5\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"158\" y=\"-600.8\">separable_conv2d_3: SeparableConv2D</text>\n",
       "<polyline fill=\"none\" points=\"275,-581.5 275,-627.5 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"302.5\" y=\"-612.3\">input:</text>\n",
       "<polyline fill=\"none\" points=\"275,-604.5 330,-604.5 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"302.5\" y=\"-589.3\">output:</text>\n",
       "<polyline fill=\"none\" points=\"330,-581.5 330,-627.5 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"389\" y=\"-612.3\">(None, 14, 14, 64)</text>\n",
       "<polyline fill=\"none\" points=\"330,-604.5 448,-604.5 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"389\" y=\"-589.3\">(None, 12, 12, 64)</text>\n",
       "</g>\n",
       "<!-- 5374952728&#45;&gt;5374951776 -->\n",
       "<g class=\"edge\" id=\"edge4\">\n",
       "<title>5374952728-&gt;5374951776</title>\n",
       "<path d=\"M244.5,-664.37C244.5,-656.15 244.5,-646.66 244.5,-637.73\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"248,-637.61 244.5,-627.61 241,-637.61 248,-637.61\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 5590560216 -->\n",
       "<g class=\"node\" id=\"node6\">\n",
       "<title>5590560216</title>\n",
       "<polygon fill=\"none\" points=\"37.5,-498.5 37.5,-544.5 451.5,-544.5 451.5,-498.5 37.5,-498.5\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"154.5\" y=\"-517.8\">separable_conv2d_4: SeparableConv2D</text>\n",
       "<polyline fill=\"none\" points=\"271.5,-498.5 271.5,-544.5 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"299\" y=\"-529.3\">input:</text>\n",
       "<polyline fill=\"none\" points=\"271.5,-521.5 326.5,-521.5 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"299\" y=\"-506.3\">output:</text>\n",
       "<polyline fill=\"none\" points=\"326.5,-498.5 326.5,-544.5 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"389\" y=\"-529.3\">(None, 12, 12, 64)</text>\n",
       "<polyline fill=\"none\" points=\"326.5,-521.5 451.5,-521.5 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"389\" y=\"-506.3\">(None, 10, 10, 128)</text>\n",
       "</g>\n",
       "<!-- 5374951776&#45;&gt;5590560216 -->\n",
       "<g class=\"edge\" id=\"edge5\">\n",
       "<title>5374951776-&gt;5590560216</title>\n",
       "<path d=\"M244.5,-581.37C244.5,-573.15 244.5,-563.66 244.5,-554.73\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"248,-554.61 244.5,-544.61 241,-554.61 248,-554.61\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 5590559824 -->\n",
       "<g class=\"node\" id=\"node7\">\n",
       "<title>5590559824</title>\n",
       "<polygon fill=\"none\" points=\"51,-415.5 51,-461.5 438,-461.5 438,-415.5 51,-415.5\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"154.5\" y=\"-434.8\">max_pooling2d_2: MaxPooling2D</text>\n",
       "<polyline fill=\"none\" points=\"258,-415.5 258,-461.5 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"285.5\" y=\"-446.3\">input:</text>\n",
       "<polyline fill=\"none\" points=\"258,-438.5 313,-438.5 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"285.5\" y=\"-423.3\">output:</text>\n",
       "<polyline fill=\"none\" points=\"313,-415.5 313,-461.5 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"375.5\" y=\"-446.3\">(None, 10, 10, 128)</text>\n",
       "<polyline fill=\"none\" points=\"313,-438.5 438,-438.5 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"375.5\" y=\"-423.3\">(None, 5, 5, 128)</text>\n",
       "</g>\n",
       "<!-- 5590560216&#45;&gt;5590559824 -->\n",
       "<g class=\"edge\" id=\"edge6\">\n",
       "<title>5590560216-&gt;5590559824</title>\n",
       "<path d=\"M244.5,-498.37C244.5,-490.15 244.5,-480.66 244.5,-471.73\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"248,-471.61 244.5,-461.61 241,-471.61 248,-471.61\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 5590591920 -->\n",
       "<g class=\"node\" id=\"node8\">\n",
       "<title>5590591920</title>\n",
       "<polygon fill=\"none\" points=\"44.5,-332.5 44.5,-378.5 444.5,-378.5 444.5,-332.5 44.5,-332.5\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"161.5\" y=\"-351.8\">separable_conv2d_5: SeparableConv2D</text>\n",
       "<polyline fill=\"none\" points=\"278.5,-332.5 278.5,-378.5 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"306\" y=\"-363.3\">input:</text>\n",
       "<polyline fill=\"none\" points=\"278.5,-355.5 333.5,-355.5 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"306\" y=\"-340.3\">output:</text>\n",
       "<polyline fill=\"none\" points=\"333.5,-332.5 333.5,-378.5 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"389\" y=\"-363.3\">(None, 5, 5, 128)</text>\n",
       "<polyline fill=\"none\" points=\"333.5,-355.5 444.5,-355.5 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"389\" y=\"-340.3\">(None, 3, 3, 64)</text>\n",
       "</g>\n",
       "<!-- 5590559824&#45;&gt;5590591920 -->\n",
       "<g class=\"edge\" id=\"edge7\">\n",
       "<title>5590559824-&gt;5590591920</title>\n",
       "<path d=\"M244.5,-415.37C244.5,-407.15 244.5,-397.66 244.5,-388.73\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"248,-388.61 244.5,-378.61 241,-388.61 248,-388.61\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 5590630640 -->\n",
       "<g class=\"node\" id=\"node9\">\n",
       "<title>5590630640</title>\n",
       "<polygon fill=\"none\" points=\"44.5,-249.5 44.5,-295.5 444.5,-295.5 444.5,-249.5 44.5,-249.5\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"161.5\" y=\"-268.8\">separable_conv2d_6: SeparableConv2D</text>\n",
       "<polyline fill=\"none\" points=\"278.5,-249.5 278.5,-295.5 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"306\" y=\"-280.3\">input:</text>\n",
       "<polyline fill=\"none\" points=\"278.5,-272.5 333.5,-272.5 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"306\" y=\"-257.3\">output:</text>\n",
       "<polyline fill=\"none\" points=\"333.5,-249.5 333.5,-295.5 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"389\" y=\"-280.3\">(None, 3, 3, 64)</text>\n",
       "<polyline fill=\"none\" points=\"333.5,-272.5 444.5,-272.5 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"389\" y=\"-257.3\">(None, 1, 1, 128)</text>\n",
       "</g>\n",
       "<!-- 5590591920&#45;&gt;5590630640 -->\n",
       "<g class=\"edge\" id=\"edge8\">\n",
       "<title>5590591920-&gt;5590630640</title>\n",
       "<path d=\"M244.5,-332.37C244.5,-324.15 244.5,-314.66 244.5,-305.73\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"248,-305.61 244.5,-295.61 241,-305.61 248,-305.61\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 5590630920 -->\n",
       "<g class=\"node\" id=\"node10\">\n",
       "<title>5590630920</title>\n",
       "<polygon fill=\"none\" points=\"0,-166.5 0,-212.5 489,-212.5 489,-166.5 0,-166.5\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"161.5\" y=\"-185.8\">global_average_pooling2d_1: GlobalAveragePooling2D</text>\n",
       "<polyline fill=\"none\" points=\"323,-166.5 323,-212.5 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"350.5\" y=\"-197.3\">input:</text>\n",
       "<polyline fill=\"none\" points=\"323,-189.5 378,-189.5 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"350.5\" y=\"-174.3\">output:</text>\n",
       "<polyline fill=\"none\" points=\"378,-166.5 378,-212.5 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"433.5\" y=\"-197.3\">(None, 1, 1, 128)</text>\n",
       "<polyline fill=\"none\" points=\"378,-189.5 489,-189.5 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"433.5\" y=\"-174.3\">(None, 128)</text>\n",
       "</g>\n",
       "<!-- 5590630640&#45;&gt;5590630920 -->\n",
       "<g class=\"edge\" id=\"edge9\">\n",
       "<title>5590630640-&gt;5590630920</title>\n",
       "<path d=\"M244.5,-249.37C244.5,-241.15 244.5,-231.66 244.5,-222.73\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"248,-222.61 244.5,-212.61 241,-222.61 248,-222.61\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 5590650776 -->\n",
       "<g class=\"node\" id=\"node11\">\n",
       "<title>5590650776</title>\n",
       "<polygon fill=\"none\" points=\"124.5,-83.5 124.5,-129.5 364.5,-129.5 364.5,-83.5 124.5,-83.5\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"175.5\" y=\"-102.8\">dense_1: Dense</text>\n",
       "<polyline fill=\"none\" points=\"226.5,-83.5 226.5,-129.5 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"254\" y=\"-114.3\">input:</text>\n",
       "<polyline fill=\"none\" points=\"226.5,-106.5 281.5,-106.5 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"254\" y=\"-91.3\">output:</text>\n",
       "<polyline fill=\"none\" points=\"281.5,-83.5 281.5,-129.5 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"323\" y=\"-114.3\">(None, 128)</text>\n",
       "<polyline fill=\"none\" points=\"281.5,-106.5 364.5,-106.5 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"323\" y=\"-91.3\">(None, 32)</text>\n",
       "</g>\n",
       "<!-- 5590630920&#45;&gt;5590650776 -->\n",
       "<g class=\"edge\" id=\"edge10\">\n",
       "<title>5590630920-&gt;5590650776</title>\n",
       "<path d=\"M244.5,-166.37C244.5,-158.15 244.5,-148.66 244.5,-139.73\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"248,-139.61 244.5,-129.61 241,-139.61 248,-139.61\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 5590681136 -->\n",
       "<g class=\"node\" id=\"node12\">\n",
       "<title>5590681136</title>\n",
       "<polygon fill=\"none\" points=\"128,-0.5 128,-46.5 361,-46.5 361,-0.5 128,-0.5\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"179\" y=\"-19.8\">dense_2: Dense</text>\n",
       "<polyline fill=\"none\" points=\"230,-0.5 230,-46.5 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"257.5\" y=\"-31.3\">input:</text>\n",
       "<polyline fill=\"none\" points=\"230,-23.5 285,-23.5 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"257.5\" y=\"-8.3\">output:</text>\n",
       "<polyline fill=\"none\" points=\"285,-0.5 285,-46.5 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"323\" y=\"-31.3\">(None, 32)</text>\n",
       "<polyline fill=\"none\" points=\"285,-23.5 361,-23.5 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"323\" y=\"-8.3\">(None, 10)</text>\n",
       "</g>\n",
       "<!-- 5590650776&#45;&gt;5590681136 -->\n",
       "<g class=\"edge\" id=\"edge11\">\n",
       "<title>5590650776-&gt;5590681136</title>\n",
       "<path d=\"M244.5,-83.37C244.5,-75.15 244.5,-65.66 244.5,-56.73\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"248,-56.61 244.5,-46.61 241,-56.61 248,-56.61\" stroke=\"black\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from keras.utils import plot_model \n",
    "plot_model(model_a, to_file='model.png')\n",
    "\n",
    "# https://stackoverflow.com/questions/51452569/how-to-resize-rescale-a-svg-graphic-in-an-ipython-jupyter-notebook/51452570#51452570\n",
    "from IPython.display import SVG\n",
    "from keras.utils import model_to_dot\n",
    "SVG(model_to_dot(model_a, show_shapes= True, show_layer_names=True, dpi=60).create(prog='dot', format='svg'))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_a.compile(loss='categorical_crossentropy',\n",
    "              optimizer='rmsprop',\n",
    "              # Adding accuracy metrics \n",
    "              metrics=['accuracy'])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 50000 samples, validate on 10000 samples\n",
      "Epoch 1/12\n",
      "50000/50000 [==============================] - 254s 5ms/step - loss: 2.3028 - accuracy: 0.0989 - val_loss: 2.3026 - val_accuracy: 0.1000\n",
      "Epoch 2/12\n",
      "50000/50000 [==============================] - 221s 4ms/step - loss: 2.3027 - accuracy: 0.0978 - val_loss: 2.3026 - val_accuracy: 0.1000\n",
      "Epoch 3/12\n",
      "50000/50000 [==============================] - 210s 4ms/step - loss: 2.3027 - accuracy: 0.0973 - val_loss: 2.3026 - val_accuracy: 0.1000\n",
      "Epoch 4/12\n",
      "50000/50000 [==============================] - 206s 4ms/step - loss: 2.3027 - accuracy: 0.0979 - val_loss: 2.3026 - val_accuracy: 0.1000\n",
      "Epoch 5/12\n",
      "50000/50000 [==============================] - 201s 4ms/step - loss: 2.3027 - accuracy: 0.0966 - val_loss: 2.3026 - val_accuracy: 0.1000\n",
      "Epoch 6/12\n",
      "50000/50000 [==============================] - 201s 4ms/step - loss: 2.3027 - accuracy: 0.0983 - val_loss: 2.3026 - val_accuracy: 0.1000\n",
      "Epoch 7/12\n",
      "50000/50000 [==============================] - 201s 4ms/step - loss: 2.3027 - accuracy: 0.0967 - val_loss: 2.3026 - val_accuracy: 0.1000\n",
      "Epoch 8/12\n",
      "50000/50000 [==============================] - 197s 4ms/step - loss: 2.3027 - accuracy: 0.0979 - val_loss: 2.3026 - val_accuracy: 0.1000\n",
      "Epoch 9/12\n",
      "50000/50000 [==============================] - 187s 4ms/step - loss: 2.3027 - accuracy: 0.0979 - val_loss: 2.3026 - val_accuracy: 0.1000\n",
      "Epoch 10/12\n",
      "50000/50000 [==============================] - 190s 4ms/step - loss: 2.3027 - accuracy: 0.0989 - val_loss: 2.3026 - val_accuracy: 0.1000\n",
      "Epoch 11/12\n",
      "50000/50000 [==============================] - 190s 4ms/step - loss: 2.3027 - accuracy: 0.0988 - val_loss: 2.3026 - val_accuracy: 0.1000\n",
      "Epoch 12/12\n",
      "50000/50000 [==============================] - 183s 4ms/step - loss: 2.3027 - accuracy: 0.0981 - val_loss: 2.3026 - val_accuracy: 0.1000\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.callbacks.History at 0x1482176a0>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_a.fit(x_train, y_train,\n",
    "          batch_size=batch_size,\n",
    "          epochs=epochs,\n",
    "          verbose=1,\n",
    "          validation_data=(x_test, y_test))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test loss: 2.3025950916290285\n",
      "Test accuracy: 0.10000000149011612\n"
     ]
    }
   ],
   "source": [
    "score = model_a.evaluate(x_test, y_test, verbose=0)\n",
    "print('Test loss:', score[0])\n",
    "print('Test accuracy:', score[1])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Regular Training with a simple ConvNet \n",
    "#####  on the CIFAR10 dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x_train shape: (50000, 32, 32, 3)\n",
      "50000 train samples\n",
      "10000 test samples\n",
      "Train on 50000 samples, validate on 10000 samples\n",
      "Epoch 1/12\n",
      "50000/50000 [==============================] - 235s 5ms/step - loss: 1.6998 - accuracy: 0.3898 - val_loss: 1.4155 - val_accuracy: 0.5259\n",
      "Epoch 2/12\n",
      "50000/50000 [==============================] - 198s 4ms/step - loss: 1.3263 - accuracy: 0.5328 - val_loss: 1.0801 - val_accuracy: 0.6237\n",
      "Epoch 3/12\n",
      "50000/50000 [==============================] - 132s 3ms/step - loss: 1.1752 - accuracy: 0.5878 - val_loss: 1.0560 - val_accuracy: 0.6326\n",
      "Epoch 4/12\n",
      "50000/50000 [==============================] - 127s 3ms/step - loss: 1.0811 - accuracy: 0.6225 - val_loss: 0.9751 - val_accuracy: 0.6583\n",
      "Epoch 5/12\n",
      "50000/50000 [==============================] - 127s 3ms/step - loss: 1.0200 - accuracy: 0.6457 - val_loss: 0.9886 - val_accuracy: 0.6582\n",
      "Epoch 6/12\n",
      "50000/50000 [==============================] - 128s 3ms/step - loss: 0.9657 - accuracy: 0.6658 - val_loss: 0.9338 - val_accuracy: 0.6780\n",
      "Epoch 7/12\n",
      "50000/50000 [==============================] - 126s 3ms/step - loss: 0.9279 - accuracy: 0.6796 - val_loss: 0.9331 - val_accuracy: 0.6767\n",
      "Epoch 8/12\n",
      "50000/50000 [==============================] - 126s 3ms/step - loss: 0.8930 - accuracy: 0.6919 - val_loss: 0.9366 - val_accuracy: 0.6812\n",
      "Epoch 9/12\n",
      "50000/50000 [==============================] - 127s 3ms/step - loss: 0.8666 - accuracy: 0.7032 - val_loss: 0.9405 - val_accuracy: 0.6919\n",
      "Epoch 10/12\n",
      "50000/50000 [==============================] - 126s 3ms/step - loss: 0.8511 - accuracy: 0.7102 - val_loss: 0.9454 - val_accuracy: 0.6980\n",
      "Epoch 11/12\n",
      "50000/50000 [==============================] - 126s 3ms/step - loss: 0.8356 - accuracy: 0.7174 - val_loss: 0.9237 - val_accuracy: 0.6992\n",
      "Epoch 12/12\n",
      "50000/50000 [==============================] - 126s 3ms/step - loss: 0.8244 - accuracy: 0.7201 - val_loss: 0.9858 - val_accuracy: 0.6706\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.callbacks.History at 0x149421c88>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'''\n",
    "Trains a simple convnet on the CIFAR10 dataset.\n",
    "with rmsprop optimizer\n",
    "'''\n",
    "\n",
    "from __future__ import print_function\n",
    "import keras\n",
    "from keras.datasets import cifar10\n",
    "from keras.models import Sequential\n",
    "from keras import layers\n",
    "from keras.layers import Dense, Dropout, Flatten\n",
    "from keras.layers import Conv2D, MaxPooling2D\n",
    "from keras import backend as K\n",
    "\n",
    "batch_size = 128\n",
    "num_classes = 10\n",
    "epochs = 12\n",
    "\n",
    "# input image dimensions\n",
    "img_height = 32\n",
    "img_width = 32\n",
    "channels = 3\n",
    "\n",
    "# the data, split between train and test sets\n",
    "(x_train, y_train), (x_test, y_test) = cifar10.load_data()\n",
    "\n",
    "if K.image_data_format() == 'channels_first':\n",
    "    x_train = x_train.reshape(x_train.shape[0], channels, img_height, img_width)\n",
    "    x_test = x_test.reshape(x_test.shape[0], channels, img_height, img_width)\n",
    "    input_shape = (channels, img_height, img_width)\n",
    "else:\n",
    "    x_train = x_train.reshape(x_train.shape[0], img_height, img_width, channels)\n",
    "    x_test = x_test.reshape(x_test.shape[0], img_height, img_width, channels)\n",
    "    input_shape = (img_height, img_width, channels)\n",
    "\n",
    "x_train = x_train.astype('float32')\n",
    "x_test = x_test.astype('float32')\n",
    "x_train /= 255\n",
    "x_test /= 255\n",
    "print('x_train shape:', x_train.shape)\n",
    "print(x_train.shape[0], 'train samples')\n",
    "print(x_test.shape[0], 'test samples')\n",
    "\n",
    "# convert class vectors to binary class matrices\n",
    "y_train = keras.utils.to_categorical(y_train, num_classes)\n",
    "y_test = keras.utils.to_categorical(y_test, num_classes)\n",
    "\n",
    "model_b = Sequential()\n",
    "model_b.add(Conv2D(32, kernel_size=(3, 3),\n",
    "                 activation='relu',\n",
    "                 input_shape=input_shape))\n",
    "model_b.add(Conv2D(64, (3, 3), activation='relu'))\n",
    "model_b.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "model_b.add(Dropout(0.25))\n",
    "model_b.add(Flatten())\n",
    "model_b.add(Dense(128, activation='relu'))\n",
    "model_b.add(Dropout(0.5))\n",
    "model_b.add(Dense(num_classes, activation='softmax'))\n",
    "\n",
    "model_b.compile(loss='categorical_crossentropy',\n",
    "              # replace by rmsprop for comparison\n",
    "              optimizer='rmsprop',\n",
    "              # Adding accuracy metrics \n",
    "              metrics=['accuracy'])\n",
    "\n",
    "model_b.fit(x_train, y_train,\n",
    "          batch_size=batch_size,\n",
    "          epochs=epochs,\n",
    "          verbose=1,\n",
    "          validation_data=(x_test, y_test))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test loss: 0.985818069267273\n",
      "Test accuracy: 0.6705999970436096\n"
     ]
    }
   ],
   "source": [
    "score = model_b.evaluate(x_test, y_test, verbose=0)\n",
    "print('Test loss:', score[0])\n",
    "print('Test accuracy:', score[1])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The results are not very convincing. Depthwise separable convolution gives an accuracy of 55% that is inferior to the 70% of the simple ConvNet model. Probably that the model is not big /deep enough. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x_train shape: (50000, 32, 32, 3)\n",
      "50000 train samples\n",
      "10000 test samples\n",
      "Train on 50000 samples, validate on 10000 samples\n",
      "Epoch 1/12\n",
      "50000/50000 [==============================] - 133s 3ms/step - loss: 1.7925 - accuracy: 0.3539 - val_loss: 1.3721 - val_accuracy: 0.5182\n",
      "Epoch 2/12\n",
      "50000/50000 [==============================] - 131s 3ms/step - loss: 1.3711 - accuracy: 0.5128 - val_loss: 1.1768 - val_accuracy: 0.5819\n",
      "Epoch 3/12\n",
      "50000/50000 [==============================] - 130s 3ms/step - loss: 1.1903 - accuracy: 0.5804 - val_loss: 1.1260 - val_accuracy: 0.6163\n",
      "Epoch 4/12\n",
      "50000/50000 [==============================] - 132s 3ms/step - loss: 1.0664 - accuracy: 0.6247 - val_loss: 0.9715 - val_accuracy: 0.6576\n",
      "Epoch 5/12\n",
      "50000/50000 [==============================] - 129s 3ms/step - loss: 0.9876 - accuracy: 0.6549 - val_loss: 0.9215 - val_accuracy: 0.6790\n",
      "Epoch 6/12\n",
      "50000/50000 [==============================] - 129s 3ms/step - loss: 0.9250 - accuracy: 0.6783 - val_loss: 0.9120 - val_accuracy: 0.6852\n",
      "Epoch 7/12\n",
      "50000/50000 [==============================] - 133s 3ms/step - loss: 0.8646 - accuracy: 0.6999 - val_loss: 0.8852 - val_accuracy: 0.6888\n",
      "Epoch 8/12\n",
      "50000/50000 [==============================] - 131s 3ms/step - loss: 0.8195 - accuracy: 0.7132 - val_loss: 0.8614 - val_accuracy: 0.7019\n",
      "Epoch 9/12\n",
      "50000/50000 [==============================] - 130s 3ms/step - loss: 0.7753 - accuracy: 0.7287 - val_loss: 0.8723 - val_accuracy: 0.6995\n",
      "Epoch 10/12\n",
      "50000/50000 [==============================] - 129s 3ms/step - loss: 0.7348 - accuracy: 0.7433 - val_loss: 0.8574 - val_accuracy: 0.7057\n",
      "Epoch 11/12\n",
      "50000/50000 [==============================] - 131s 3ms/step - loss: 0.6928 - accuracy: 0.7561 - val_loss: 0.8550 - val_accuracy: 0.7071\n",
      "Epoch 12/12\n",
      "50000/50000 [==============================] - 130s 3ms/step - loss: 0.6610 - accuracy: 0.7678 - val_loss: 0.8615 - val_accuracy: 0.7065\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.callbacks.History at 0x149532da0>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'''\n",
    "Trains a simple convnet on the CIFAR10 dataset.\n",
    "with Adadelta optimizer\n",
    "'''\n",
    "\n",
    "from __future__ import print_function\n",
    "import keras\n",
    "from keras.datasets import cifar10\n",
    "from keras.models import Sequential\n",
    "from keras import layers\n",
    "from keras.layers import Dense, Dropout, Flatten\n",
    "from keras.layers import Conv2D, MaxPooling2D\n",
    "from keras import backend as K\n",
    "\n",
    "batch_size = 128\n",
    "num_classes = 10\n",
    "epochs = 12\n",
    "\n",
    "# input image dimensions\n",
    "img_height = 32\n",
    "img_width = 32\n",
    "channels = 3\n",
    "\n",
    "# the data, split between train and test sets\n",
    "(x_train, y_train), (x_test, y_test) = cifar10.load_data()\n",
    "\n",
    "if K.image_data_format() == 'channels_first':\n",
    "    x_train = x_train.reshape(x_train.shape[0], channels, img_height, img_width)\n",
    "    x_test = x_test.reshape(x_test.shape[0], channels, img_height, img_width)\n",
    "    input_shape = (channels, img_height, img_width)\n",
    "else:\n",
    "    x_train = x_train.reshape(x_train.shape[0], img_height, img_width, channels)\n",
    "    x_test = x_test.reshape(x_test.shape[0], img_height, img_width, channels)\n",
    "    input_shape = (img_height, img_width, channels)\n",
    "\n",
    "x_train = x_train.astype('float32')\n",
    "x_test = x_test.astype('float32')\n",
    "x_train /= 255\n",
    "x_test /= 255\n",
    "print('x_train shape:', x_train.shape)\n",
    "print(x_train.shape[0], 'train samples')\n",
    "print(x_test.shape[0], 'test samples')\n",
    "\n",
    "# convert class vectors to binary class matrices\n",
    "y_train = keras.utils.to_categorical(y_train, num_classes)\n",
    "y_test = keras.utils.to_categorical(y_test, num_classes)\n",
    "\n",
    "model_c = Sequential()\n",
    "model_c.add(Conv2D(32, kernel_size=(3, 3),\n",
    "                 activation='relu',\n",
    "                 input_shape=input_shape))\n",
    "model_c.add(Conv2D(64, (3, 3), activation='relu'))\n",
    "model_c.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "model_c.add(Dropout(0.25))\n",
    "model_c.add(Flatten())\n",
    "model_c.add(Dense(128, activation='relu'))\n",
    "model_c.add(Dropout(0.5))\n",
    "model_c.add(Dense(num_classes, activation='softmax'))\n",
    "\n",
    "model_c.compile(loss='categorical_crossentropy',\n",
    "              # The model_c is optimized with the Adadelta optimizer\n",
    "              optimizer='Adadelta',\n",
    "              # Adding accuracy metrics \n",
    "              metrics=['accuracy'])\n",
    "\n",
    "model_c.fit(x_train, y_train,\n",
    "          batch_size=batch_size,\n",
    "          epochs=epochs,\n",
    "          verbose=1,\n",
    "          validation_data=(x_test, y_test))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test loss: 0.8615231519699097\n",
      "Test accuracy: 0.7064999938011169\n"
     ]
    }
   ],
   "source": [
    "score = model_c.evaluate(x_test, y_test, verbose=0)\n",
    "print('Test loss:', score[0])\n",
    "print('Test accuracy:', score[1])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(50000, 32, 32, 3)\n",
      "(10000, 32, 32, 3)\n",
      "50000 train samples\n",
      "10000 test samples\n",
      "Model: \"sequential_4\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_7 (Dense)              (None, 512)               1573376   \n",
      "_________________________________________________________________\n",
      "dropout_5 (Dropout)          (None, 512)               0         \n",
      "_________________________________________________________________\n",
      "dense_8 (Dense)              (None, 512)               262656    \n",
      "_________________________________________________________________\n",
      "dropout_6 (Dropout)          (None, 512)               0         \n",
      "_________________________________________________________________\n",
      "dense_9 (Dense)              (None, 10)                5130      \n",
      "=================================================================\n",
      "Total params: 1,841,162\n",
      "Trainable params: 1,841,162\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "'''Trains a simple deep NN (MLP: Multi-Layer Perceptron) \n",
    "   on the CIFAR10 dataset.\n",
    "'''\n",
    "from __future__ import print_function\n",
    "\n",
    "import keras\n",
    "from keras.datasets import mnist\n",
    "from keras.models import Sequential\n",
    "from keras import layers\n",
    "from keras.layers import Dense, Dropout\n",
    "from keras.optimizers import RMSprop\n",
    "\n",
    "batch_size = 128\n",
    "num_classes = 10\n",
    "epochs = 20\n",
    "# epochs = 12\n",
    "\n",
    "# the data, split between train and test sets\n",
    "# (x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
    "(x_train, y_train), (x_test, y_test) = cifar10.load_data()\n",
    "\n",
    "print(x_train.shape)\n",
    "print(x_test.shape)\n",
    "\n",
    "# x_train = x_train.reshape(60000, 784)\n",
    "x_train = x_train.reshape(50000, 3072)\n",
    "# x_test = x_test.reshape(10000, 784)\n",
    "x_test = x_test.reshape(10000, 3072)\n",
    "x_train = x_train.astype('float32')\n",
    "x_test = x_test.astype('float32')\n",
    "x_train /= 255\n",
    "x_test /= 255\n",
    "print(x_train.shape[0], 'train samples')\n",
    "print(x_test.shape[0], 'test samples')\n",
    "\n",
    "# convert class vectors to binary class matrices\n",
    "y_train = keras.utils.to_categorical(y_train, num_classes)\n",
    "y_test = keras.utils.to_categorical(y_test, num_classes)\n",
    "\n",
    "model_d = Sequential()\n",
    "# model_d.add(Dense(512, activation='relu', input_shape=(784,)))\n",
    "model_d.add(Dense(512, activation='relu', input_shape=(3072,)))\n",
    "model_d.add(Dropout(0.2))\n",
    "model_d.add(Dense(512, activation='relu'))\n",
    "model_d.add(Dropout(0.2))\n",
    "model_d.add(Dense(num_classes, activation='softmax'))\n",
    "\n",
    "model_d.summary()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 50000 samples, validate on 10000 samples\n",
      "Epoch 1/20\n",
      "50000/50000 [==============================] - 15s 290us/step - loss: 2.1510 - accuracy: 0.2481 - val_loss: 1.9004 - val_accuracy: 0.2986\n",
      "Epoch 2/20\n",
      "50000/50000 [==============================] - 14s 282us/step - loss: 1.8581 - accuracy: 0.3290 - val_loss: 1.8088 - val_accuracy: 0.3489\n",
      "Epoch 3/20\n",
      "50000/50000 [==============================] - 15s 293us/step - loss: 1.7757 - accuracy: 0.3607 - val_loss: 1.6859 - val_accuracy: 0.4073\n",
      "Epoch 4/20\n",
      "50000/50000 [==============================] - 14s 287us/step - loss: 1.7298 - accuracy: 0.3810 - val_loss: 1.6326 - val_accuracy: 0.4289\n",
      "Epoch 5/20\n",
      "50000/50000 [==============================] - 14s 278us/step - loss: 1.6898 - accuracy: 0.3945 - val_loss: 1.6298 - val_accuracy: 0.4160\n",
      "Epoch 6/20\n",
      "50000/50000 [==============================] - 14s 290us/step - loss: 1.6643 - accuracy: 0.4069 - val_loss: 1.6093 - val_accuracy: 0.4272\n",
      "Epoch 7/20\n",
      "50000/50000 [==============================] - 14s 272us/step - loss: 1.6480 - accuracy: 0.4111 - val_loss: 1.5698 - val_accuracy: 0.4416\n",
      "Epoch 8/20\n",
      "50000/50000 [==============================] - 14s 288us/step - loss: 1.6283 - accuracy: 0.4179 - val_loss: 1.5584 - val_accuracy: 0.4546\n",
      "Epoch 9/20\n",
      "50000/50000 [==============================] - 14s 274us/step - loss: 1.6095 - accuracy: 0.4248 - val_loss: 1.6247 - val_accuracy: 0.4128\n",
      "Epoch 10/20\n",
      "50000/50000 [==============================] - 14s 288us/step - loss: 1.5976 - accuracy: 0.4290 - val_loss: 1.5439 - val_accuracy: 0.4572\n",
      "Epoch 11/20\n",
      "50000/50000 [==============================] - 14s 289us/step - loss: 1.5835 - accuracy: 0.4340 - val_loss: 1.5424 - val_accuracy: 0.4485\n",
      "Epoch 12/20\n",
      "50000/50000 [==============================] - 14s 280us/step - loss: 1.5692 - accuracy: 0.4406 - val_loss: 1.6547 - val_accuracy: 0.4262\n",
      "Epoch 13/20\n",
      "50000/50000 [==============================] - 14s 288us/step - loss: 1.5622 - accuracy: 0.4454 - val_loss: 1.5261 - val_accuracy: 0.4668\n",
      "Epoch 14/20\n",
      "50000/50000 [==============================] - 14s 275us/step - loss: 1.5501 - accuracy: 0.4450 - val_loss: 1.5217 - val_accuracy: 0.4705\n",
      "Epoch 15/20\n",
      "50000/50000 [==============================] - 14s 286us/step - loss: 1.5412 - accuracy: 0.4508 - val_loss: 1.5655 - val_accuracy: 0.4488\n",
      "Epoch 16/20\n",
      "50000/50000 [==============================] - 14s 283us/step - loss: 1.5380 - accuracy: 0.4516 - val_loss: 1.5259 - val_accuracy: 0.4669\n",
      "Epoch 17/20\n",
      "50000/50000 [==============================] - 14s 290us/step - loss: 1.5256 - accuracy: 0.4563 - val_loss: 1.5092 - val_accuracy: 0.4714\n",
      "Epoch 18/20\n",
      "50000/50000 [==============================] - 14s 289us/step - loss: 1.5235 - accuracy: 0.4591 - val_loss: 1.5365 - val_accuracy: 0.4607\n",
      "Epoch 19/20\n",
      "50000/50000 [==============================] - 14s 274us/step - loss: 1.5229 - accuracy: 0.4588 - val_loss: 1.4739 - val_accuracy: 0.4855\n",
      "Epoch 20/20\n",
      "50000/50000 [==============================] - 14s 285us/step - loss: 1.5114 - accuracy: 0.4632 - val_loss: 1.5280 - val_accuracy: 0.4663\n"
     ]
    }
   ],
   "source": [
    "model_d.compile(loss='categorical_crossentropy',\n",
    "              optimizer=RMSprop(),\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "history = model_d.fit(x_train, y_train,\n",
    "                    batch_size=batch_size,\n",
    "                    epochs=epochs,\n",
    "                    verbose=1,\n",
    "                    validation_data=(x_test, y_test))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test loss: 1.527995453453064\n",
      "Test accuracy: 0.46630001068115234\n"
     ]
    }
   ],
   "source": [
    "score = model_d.evaluate(x_test, y_test, verbose=0)\n",
    "print('Test loss:', score[0])\n",
    "print('Test accuracy:', score[1])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7.3.2. Hyperparameter optimization \n",
    "\n",
    "When building a deep-learning model, you have to make many seemingly arbitrary decisions: How many layers should you stack? How many units or filters should go in each layer? Should you use `relu` as activation, or a different function? Should you use `BatchNormalization` after a given layer? How much dropout should you use? And so on. These architecture-level parameters are called <i>hyperparameters</i> to distinguish them from the parameters of a model, which are trained via backpropagation. \n",
    "\n",
    "In practice, experienced machine-learning engineers and researchers build intuition over time as to what works and what doesn’t when it comes to these choices—they develop hyperparameter-tuning skills. But there are no formal rules. If you want to get to the very limit of what can be achieved on a given task, you can’t be content with arbitrary choices made by a fallible human. Your initial decisions are almost always suboptimal, even if you have good intuition. You can refine your choices by tweaking them by hand and retraining the model repeatedly—that’s what machine-learning engineers and researchers spend most of their time doing. But it shouldn’t be your job as a human to fiddle with hyperparameters all day—that is better left to a machine. \n",
    "\n",
    "Thus you need to explore the space of possible decisions automatically, systematically, in a principled way. You need to search the architecture space and find the best-performing ones empirically. That’s what the field of automatic hyperparameter optimization is about: it’s an entire field of research, and an important one.\n",
    "\n",
    "The process of optimizing hyperparameters typically looks like this: \n",
    "\n",
    "* Choose a set of hyperparameters (automatically).\n",
    "* Build the corresponding model.\n",
    "* Fit it to your training data, and measure the final performance on the validation data. \n",
    "* Choose the next set of hyperparameters to try (automatically). \n",
    "* Repeat. \n",
    "* Eventually, measure performance on your test data. \n",
    "\n",
    "The key to this process is the algorithm that uses this history of validation performance, given various sets of hyperparameters, to choose the next set of hyperparameters to evaluate. Many different techniques are possible: Bayesian optimization (https://en.wikipedia.org/wiki/Bayesian_optimization), genetic algorithms (https://en.wikipedia.org/wiki/Genetic_algorithm), simple random search (https://en.wikipedia.org/wiki/Random_search), and so on.\n",
    "\n",
    "The key to this process is the algorithm that uses this history of validation performance, given various sets of hyperparameters, to choose the next set of hyperparameters to evaluate. Many different techniques are possible: Bayesian optimization, genetic algorithms, simple random search, and so on. \n",
    "\n",
    "Training the weights of a model is relatively easy: you compute a loss function on a mini-batch of data and then use the Backpropagation algorithm to move the weights in the right direction. Updating hyperparameters, on the other hand, is extremely challenging. Consider the following: \n",
    "\n",
    "* Computing the feedback signal (does this set of hyperparameters lead to a high-performing model on this task?) can be extremely expensive: it requires creating and training a new model from scratch on your dataset. \n",
    "* The hyperparameter space is typically made of discrete decisions and thus isn’t continuous or differentiable. Hence, you typically can’t do gradient descent in hyperparameter space. Instead, you must rely on gradient-free optimization techniques, which naturally are far less efficient than gradient descent.\n",
    "\n",
    "Because these challenges are difficult and the field is still young, we currently only have access to very limited tools to optimize models. Often, it turns out that random search (choosing hyperparameters to evaluate at random, repeatedly) is the best solution, despite being the most naive one. But one tool I have found reliably better than random search is Hyperopt (https://github.com/hyperopt/hyperopt), a Python library for hyperparameter optimization that internally uses trees of Parzen estimators to predict sets of hyperparameters that are likely to work well. Another library called Hyperas (https://github.com/maxpumperla/hyperas) integrates Hyperopt for use with Keras models. Do check it out.\n",
    "\n",
    "** Note ** \n",
    "> One important issue to keep in mind when doing automatic hyperparameter optimization at scale is validation-set overfitting. Because you’re updating hyperparameters based on a signal that is computed using your validation data, you’re effectively training them on the validation data, and thus they will quickly overfit to the validation data. Always keep this in mind.\n",
    "\n",
    "Overall, hyperparameter optimization is a powerful technique that is an absolute requirement to get to state-of-the-art models on any task or to win machine-learning competitions. Think about it: once upon a time, people handcrafted the features that went into shallow machine-learning models. That was very much suboptimal. Now, deep learning automates the task of hierarchical feature engineering—features are learned using a feedback signal, not hand-tuned, and that’s the way it should be. In the same way, you shouldn’t handcraft your model architectures; you should optimize them in a principled way. At the time of writing, the field of automatic hyperparameter optimization is very young and immature, as deep learning was some years ago, but I expect it to boom in the next few years."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7.3.3. Model ensembling \n",
    "Another powerful technique for obtaining the best possible results on a task is model ensembling. Ensembling consists of pooling together the predictions of a set of different models, to produce better predictions. If you look at machine-learning competitions, in particular on Kaggle, you’ll see that the winners use very large ensembles of models that inevitably beat any single model, no matter how good. \n",
    "\n",
    "Ensembling relies on the assumption that different good models trained independently are likely to be good for different reasons: each model looks at slightly different aspects of the data to make its predictions, getting part of the “truth” but not all of it. You may be familiar with the ancient parable of the blind men and the elephant: a group of blind men come across an elephant for the first time and try to understand what the elephant is by touching it. Each man touches a different part of the elephant’s body—just one part, such as the trunk or a leg. Then the men describe to each other what an elephant is: “It’s like a snake,” “Like a pillar or a tree,” and so on. The blind men are essentially machine-learning models trying to understand the manifold of the training data, each from its own perspective, using its own assumptions (provided by the unique architecture of the model and the unique random weight initialization). Each of them gets part of the truth of the data, but not the whole truth. By pooling their perspectives together, you can get a far more accurate description of the data. The elephant is a combination of parts: not any single blind man gets it quite right, but, interviewed together, they can tell a fairly accurate story. \n",
    "\n",
    "Let’s use classification as an example. The easiest way to pool the predictions of a set of classifiers (to ensemble the classifiers) is to average their predictions at inference time:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10000, 3072)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_test.shape\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7164"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_val = x_test.reshape(10000, 32, 32, 3)\n",
    "# Use four different models to compute initial predictions.\n",
    "preds_a = model_a.predict(x_val)\n",
    "preds_b = model_b.predict(x_val)\n",
    "preds_c = model_c.predict(x_val)\n",
    "preds_d = model_d.predict(x_test)\n",
    "\n",
    "# This new prediction array should be more accurate than any of the initial ones.\n",
    "final_preds = 0.25 * (preds_a + preds_b + preds_c + preds_d)\n",
    "\n",
    "import numpy as np\n",
    "final_preds_one_hot = np.zeros_like(final_preds)\n",
    "final_preds_one_hot[np.arange(len(final_preds)), final_preds.argmax(1)] = 1\n",
    "\n",
    "from sklearn.metrics import accuracy_score\n",
    "accuracy_score(y_test, final_preds_one_hot)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will work only if the classifiers are more or less equally good. If one of them is significantly worse than the others, the final predictions may not be as good as the best classifier of the group. A smarter way to ensemble classifiers is to do a weighted average, where the weights are learned on the validation data—typically, the better classifiers are given a higher weight, and the worse classifiers are given a lower weight. To search for a good set of ensembling weights, you can use random search or a simple optimization algorithm such as Nelder-Mead.\n",
    "\n",
    "With a homemade one_hot_encoder and accuracy_score from Scikit Learn:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7174"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "# These weights (0.2, 0.40, 0.35, 0.05) are assumed to be learned empirically.\n",
    "final_preds = 0.2 * preds_a + 0.40 * preds_b + 0.35 * preds_c + 0.05 * preds_d \n",
    "final_preds_one_hot = np.zeros_like(final_preds)\n",
    "final_preds_one_hot[np.arange(len(final_preds)), final_preds.argmax(1)] = 1\n",
    "\n",
    "from sklearn.metrics import accuracy_score\n",
    "accuracy_score(y_test, final_preds_one_hot)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With a Keras one_hot encoder and accuracy_score from Scikit Learn:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7174"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from keras import backend as K\n",
    "final_preds_one_hot = K.one_hot(K.argmax(final_preds,axis=1),10)\n",
    "\n",
    "from sklearn.metrics import accuracy_score\n",
    "# accuracy_score(y_test,tf.Session().run(final_preds_one_hot))\n",
    "accuracy_score(y_test,final_preds_one_hot)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are many possible variants: you can do an average of an exponential of the predictions, for instance. In general, a simple weighted average with weights optimized on the validation data provides a very strong baseline. \n",
    "\n",
    "The key to making ensembling work is the <i>diversity</i> of the set of classifiers. Diversity is strength. If your models are <i>biased in different ways</i>, the biases will cancel each other out, and the ensemble will be more robust and more accurate. \n",
    "\n",
    "For this reason, you should ensemble models that are <i>as good as possible</i> while being as different as possible. This typically means using very different architectures or even different brands of machine-learning approaches. One thing that is largely not worth doing is ensembling the same network trained several times independently, from different random initializations. If the only difference between your models is their random initialization and the order in which they were exposed to the training data, then your ensemble will be low-diversity and will provide only a tiny improvement over any single model. The point of ensembling. It’s not so much about how good your best model is; it’s about the diversity of your set of candidate models. \n",
    "\n",
    "In recent times, one style of basic ensemble that has been very successful in practice is the wide and deep category of models, blending deep learning with shallow learning. Such models consist of jointly training a deep neural network with a large linear model. The joint training of a family of diverse models is yet another option to achieve model ensembling. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "IPython Notebook executed\n"
     ]
    }
   ],
   "source": [
    "print(\"IPython Notebook executed\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}