PY580_Notebook11_DNNs_Keras.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Notebook 11: Introduction to Deep Neural Networks with Keras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning Goals\n",
    "The goal of this notebook is to introduce deep neural networks (DNNs) using the high-level Keras package. The reader will become familiar with how to choose an architecture, cost function, and optimizer in Keras. We will also learn how to train neural networks.\n",
    "\n",
    "\n",
    "# MNIST with Keras\n",
    "\n",
    "We will once again work with the MNIST dataset of hand written digits introduced in *Notebook 7: Logistic Regression (MNIST)*. The goal is to find a statistical model which recognizes and distinguishes between the ten handwritten digits (0-9).\n",
    "\n",
    "The MNIST dataset comprises $70000$ handwritten digits, each of which comes in a square image, divided into a $28\\times 28$ pixel grid. Every pixel can take on $256$ nuances of the gray color, interpolating between white and black, and hence each data point assumes any value in the set $\\{0,1,\\dots,255\\}$. Since there are $10$ categories in the problem, corresponding to the ten digits, this problem represents a generic classification task. \n",
    "\n",
    "In this Notebook, we show how to use the Keras python package to tackle the MNIST problem with the help of deep neural networks.\n",
    "\n",
    "The following code is a slight modification of a Keras tutorial, see [https://github.com/fchollet/keras/blob/master/examples/mnist_cnn.py](https://github.com/fchollet/keras/blob/master/examples/mnist_cnn.py). We invite the reader to read Sec. IX of the review to acquire a broad understanding of what the separate parts of the code do."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "from __future__ import print_function\n",
    "import keras,sklearn\n",
    "# suppress tensorflow compilation warnings\n",
    "import os\n",
    "os.environ['KMP_DUPLICATE_LIB_OK']='True'\n",
    "import tensorflow as tf\n",
    "os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'\n",
    "import numpy as np\n",
    "seed=0\n",
    "np.random.seed(seed) # fix random seed\n",
    "tf.set_random_seed(seed)\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Structure of the Procedure\n",
    "\n",
    "Constructing a Deep Neural Network to solve ML problems is a multiple-stage process. Quite generally, one can identify the key steps as follows:\n",
    "\n",
    "* ***step 1:*** Load and process the data\n",
    "* ***step 2:*** Define the model and its architecture\n",
    "* ***step 3:*** Choose the optimizer and the cost function\n",
    "* ***step 4:*** Train the model \n",
    "* ***step 5:*** Evaluate the model performance on the *unseen* test data\n",
    "* ***step 6:*** Modify the hyperparameters to optimize performance for the specific data set\n",
    "\n",
    "We would like to emphasize that, while it is always possible to view steps 1-5 as independent of the particular task we are trying to solve, it is only when they are put together in ***step 6*** that the real gain of using Deep Learning is revealed, compared to less sophisticated methods such as the regression models or bagging, described in Secs. VII and VIII of the review. With this remark in mind, we shall focus predominantly on steps 1-5 below. We show how one can use grid search methods to find optimal hyperparameters in ***step 6***."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1: Load and Process the Data\n",
    "\n",
    "Keras can conveniently download the MNIST data from the web. All we need to do is import the `mnist` module and use the `load_data()` class, and it will create the training and test data sets or us.\n",
    "\n",
    "The MNIST set has pre-defined test and training sets, in order to facilitate the comparison of the performance of different models on the data.\n",
    "\n",
    "Once we have loaded the data, we need to format it in the correct shape. This differs from one package to the other and, as we see in the case of Keras, it can even be different depending on the backend used.\n",
    "\n",
    "While choosing the correct `datatype` can help improve the computational speed, we emphasize the rescaling step, which is necessary to avoid large variations in the minimal and maximal possible values of each feature. In other words, we want to make sure a feature is not being over-represented just because it is \"large\".\n",
    "\n",
    "Last, we cast the label vectors $y$ to binary class matrices (a.k.a. one-hot format), as explained in Sec. VII on SoftMax regression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://s3.amazonaws.com/img-datasets/mnist.npz\n",
      "11493376/11490434 [==============================] - 2s 0us/step\n",
      "an example of a data point with label 4\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQQAAAECCAYAAAAYUakXAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAADlpJREFUeJzt3V+MVfV6xvHnqdgLUQFhJBOLRZELmppi3WqjJ9XmpCfWaJALTTE2mGjQeIx/okmFC4WYRq3CaS+MEZQcmiANRq1ekCoxRs+Jimf7J4illaOhikyGIZoAV0R9ezGbt3N05rdnZv9Ze4bvJyGzZ717Zj0umSdr7f2bhSNCACBJf1R1AAC9g0IAkCgEAIlCAJAoBACJQgCQKikE21fb/h/bv7f9YBUZSmzvt/2J7Y9t13sgz2bbh2zvGbHtLNs7be9rfJzTY/nW2v66cQw/tn1NhfkW2H7T9l7bn9q+p7G9J45hIV/Xj6G7vQ7B9imSPpP0t5IOSPqdpBUR8V9dDVJge7+kWkQcrjqLJNn+a0nHJP1bRPx5Y9s/S/omIh5rlOqciPjHHsq3VtKxiHiyikwj2e6X1B8RH9o+Q9IHkq6XdIt64BgW8t2oLh/DKs4QLpX0+4j4IiKOS/p3ScsqyDFlRMTbkr750eZlkrY0Hm/R8F+gSoyRr2dExEBEfNh4fFTSXknnqEeOYSFf11VRCOdI+mrE5wdU0X98QUh63fYHtldVHWYM8yNiQBr+CyXp7IrzjOYu27sblxSVXdKMZHuhpIsk7VIPHsMf5ZO6fAyrKASPsq3X1k9fERF/KenvJP2ycUqMiXla0iJJSyUNSFpfbRzJ9umSXpR0b0QcqTrPj42Sr+vHsIpCOCBpwYjP/0TSwQpyjCkiDjY+HpL0soYvc3rNYOPa88Q16KGK8/yBiBiMiO8j4gdJm1TxMbR9qoZ/2LZGxEuNzT1zDEfLV8UxrKIQfidpse3zbP+xpL+X9GoFOUZle2bjhR3ZninpF5L2lL+qEq9KWtl4vFLSKxVm+YkTP2gNy1XhMbRtSc9J2hsRG0aMeuIYjpWvimPY9XcZJKnx9sm/SDpF0uaI+KeuhxiD7fM1fFYgSTMkPV91PtvbJF0laZ6kQUkPS/oPSdslnSvpS0k3REQlL+yNke8qDZ/qhqT9km4/cb1eQb6fSfqNpE8k/dDYvEbD1+mVH8NCvhXq8jGspBAA9CZWKgJIFAKARCEASBQCgEQhAEiVFkIPLwuWRL5W9XK+Xs4mVZev6jOEnv6fIvK1qpfz9XI2qaJ8VRcCgB7S0sIk21dL+lcNrzh8NiIeKz1/3rx5sXDhwvx8aGhIfX19k95/p5GvNb2cr5ezSe3Pt3//fh0+fHi0Xyz8AzMmu4PGjU6e0ogbndh+tXSjk4ULF6per/wGRMBJp1arjet5rVwycKMTYJpppRCmwo1OAExAK4Uwrhud2F5lu267PjQ01MLuAHRaK4UwrhudRMTGiKhFRK2XX8QB0Foh9PSNTgBM3KTfZYiI72zfJek1/f+NTj5tWzIAXTfpQpCkiNghaUebsgCoGCsVASQKAUCiEAAkCgFAohAAJAoBQKIQACQKAUCiEAAkCgFAohAAJAoBQKIQACQKAUCiEAAkCgFAohAAJAoBQKIQACQKAUCiEAAkCgFAauk27EA7ffbZZ8X5HXfcUZxv3bq1OO/v759wppMNZwgAEoUAIFEIABKFACBRCAAShQAgUQgA0rRah3D06NHi/NixY8X5rFmzivPTTjttwpkwfjt27CjO33rrreL82WefLc5Xr15dnM+YMa1+HCalpSNge7+ko5K+l/RdRNTaEQpANdpRiX8TEYfb8H0AVIzXEACkVgshJL1u+wPbq9oRCEB1Wr1kuCIiDto+W9JO2/8dEW+PfEKjKFZJ0rnnntvi7gB0UktnCBFxsPHxkKSXJV06ynM2RkQtImp9fX2t7A5Ah026EGzPtH3GiceSfiFpT7uCAei+Vi4Z5kt62faJ7/N8RPxnW1JN0uOPP16cP/roo8X5k08+WZzfd999E86E8bv44otb+vq1a9cW5ytWrCjOL7jggpb2Px1MuhAi4gtJf9HGLAAqxtuOABKFACBRCAAShQAgUQgAEoUAIPEL4COsW7euOD///POL82XLlrUzzklncHCw6ggnPc4QACQKAUCiEAAkCgFAohAAJAoBQKIQACTWIYzQ7N91uOWWW4rznTt3Fue12sl9l/pm/y7G+vXrO7r/7du3F+dr1qzp6P6nAs4QACQKAUCiEAAkCgFAohAAJAoBQKIQAKRptQ7hvPPO6+j3P3LkSHH+0EMPFedbt24tzufMmTPhTFPJvn37ivP333+/S0kwFs4QACQKAUCiEAAkCgFAohAAJAoBQKIQAKRptQ6h2f0KDh48WJyvXbu2pf2/9tprxfmLL75YnN92220t7b/XzZ8/vzhftGhRcf7555+3tP8bb7yxpa8/GTQ9Q7C92fYh23tGbDvL9k7b+xofp/eKGuAkMZ5Lhl9LuvpH2x6U9EZELJb0RuNzAFNc00KIiLclffOjzcskbWk83iLp+jbnAlCByb6oOD8iBiSp8fHs9kUCUJWOv8tge5Xtuu360NBQp3cHoAWTLYRB2/2S1Ph4aKwnRsTGiKhFRK2vr2+SuwPQDZMthFclrWw8XinplfbEAVClpusQbG+TdJWkebYPSHpY0mOSttu+VdKXkm7oZMjxOuWUU4rzu+++uzhvdr+CZr/P38xTTz1VnC9fvrw4nzt3bkv7r9rg4GBx3uo6A7SuaSFExIoxRj9vcxYAFWPpMoBEIQBIFAKARCEASBQCgEQhAEjT6n4IzcyaNas4v/zyy4vzVtch7N69uzj/6quvivNOr0M4fvx4cf7MM8+09P1feOGFlr4enccZAoBEIQBIFAKARCEASBQCgEQhAEgUAoB0Uq1DaKbZOoQtW7YU56169913i/OlS5cW5++8805L82PHjhXnjzzySHFetSVLlhTnc+bwrwU0wxkCgEQhAEgUAoBEIQBIFAKARCEASBQCgOSI6NrOarVa1Ov1ru2v3W6++ebi/Pnnn+9Sks5o9nfBdpeSdMamTZuK81tvvbVLSbqvVqupXq83/R/IGQKARCEASBQCgEQhAEgUAoBEIQBIFAKAxP0QJuD+++8vzrdt29alJNWY6usQ3nvvveJ8Oq9DGK+mZwi2N9s+ZHvPiG1rbX9t++PGn2s6GxNAN4znkuHXkq4eZfuvImJp48+O9sYCUIWmhRARb0v6pgtZAFSslRcV77K9u3FJwc3qgGlgsoXwtKRFkpZKGpC0fqwn2l5lu267PjQ0NMndAeiGSRVCRAxGxPcR8YOkTZIuLTx3Y0TUIqLW19c32ZwAumBShWC7f8SnyyXtGeu5AKaOpusQbG+TdJWkebYPSHpY0lW2l0oKSfsl3d7BjOiSxYsXF+fN1iFcc0353efZs2cX5+vWrSvO0XlNCyEiVoyy+bkOZAFQMZYuA0gUAoBEIQBIFAKARCEASBQCgMT9EKaQuXPnFucLFiwozh944IHifMWK0d5hbp+PPvqoOGcdQvU4QwCQKAQAiUIAkCgEAIlCAJAoBACJQgCQWIcwAYsWLSrOV65cWZx/8cUXxfmSJUuK8zvvvLM4v/DCC4vzk93rr79enH/77bfF+Zw50//WoZwhAEgUAoBEIQBIFAKARCEASBQCgEQhAEisQ5iAM888szjfvHlzl5JgMg4cOFCcHz9+vEtJehdnCAAShQAgUQgAEoUAIFEIABKFACBRCAAS6xDQNbNnzy7O+/v7i/OBgYF2xvmJ1atXF+cbN24szmfMmPo/Tk3PEGwvsP2m7b22P7V9T2P7WbZ32t7X+Dj97x4BTHPjuWT4TtL9EbFE0l9J+qXtP5P0oKQ3ImKxpDcanwOYwpoWQkQMRMSHjcdHJe2VdI6kZZK2NJ62RdL1nQoJoDsm9KKi7YWSLpK0S9L8iBiQhktD0tntDgegu8ZdCLZPl/SipHsj4sgEvm6V7brt+tDQ0GQyAuiScRWC7VM1XAZbI+KlxuZB2/2Neb+kQ6N9bURsjIhaRNT6+vrakRlAh4znXQZLek7S3ojYMGL0qqQT9x1fKemV9scD0E2OiPIT7J9J+o2kTyT90Ni8RsOvI2yXdK6kLyXdEBHflL5XrVaLer3eamZMU7t27SrOly9fXpwPDg62M85PHDlSvlKeOXNmR/ffilqtpnq97mbPa7qSIiJ+K2msb/TziQYD0LtYugwgUQgAEoUAIFEIABKFACBRCADS1P8Fbkwbl112WXH+yivltW/XXXddcd7q0vlma2iuvPLKlr5/L+AMAUCiEAAkCgFAohAAJAoBQKIQACQKAUBiHQKmjEsuuaQ437BhQ3H+xBNPFOfXXnttcV6r1Yrz6YAzBACJQgCQKAQAiUIAkCgEAIlCAJAoBACJdQiYNm666aaW5uAMAcAIFAKARCEASBQCgEQhAEgUAoBEIQBITQvB9gLbb9rea/tT2/c0tq+1/bXtjxt/rul8XACdNJ6FSd9Juj8iPrR9hqQPbO9szH4VEU92Lh6AbmpaCBExIGmg8fio7b2Szul0MADdN6HXEGwvlHSRpF2NTXfZ3m17s+05bc4GoMvGXQi2T5f0oqR7I+KIpKclLZK0VMNnEOvH+LpVtuu2663+23oAOmtchWD7VA2XwdaIeEmSImIwIr6PiB8kbZJ06WhfGxEbI6IWEbW+vr525QbQAeN5l8GSnpO0NyI2jNjeP+JpyyXtaX88AN00nncZrpD0D5I+sf1xY9saSStsL5UUkvZLur0jCQF0zXjeZfitJI8y2tH+OACqxEpFAIlCAJAoBACJQgCQKAQAiUIAkCgEAIlCAJAoBACJQgCQKAQAiUIAkCgEAIlCAJAoBADJEdG9ndlDkv53xKZ5kg53LcDEka81vZyvl7NJ7c/3pxHR9B6GXS2En+zcrkdErbIATZCvNb2cr5ezSdXl45IBQKIQAKSqC2Fjxftvhnyt6eV8vZxNqihfpa8hAOgtVZ8hAOghFAKARCEASBQCgEQhAEj/B2LoAC5+9sIHAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 288x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X_train shape: (60000, 784)\n",
      "Y_train shape: (60000, 10)\n",
      "\n",
      "60000 train samples\n",
      "10000 test samples\n"
     ]
    }
   ],
   "source": [
    "from keras.datasets import mnist\n",
    "\n",
    "# input image dimensions\n",
    "num_classes = 10 # 10 digits\n",
    "\n",
    "img_rows, img_cols = 28, 28 # number of pixels \n",
    "\n",
    "# the data, shuffled and split between train and test sets\n",
    "(X_train, Y_train), (X_test, Y_test) = mnist.load_data()\n",
    "\n",
    "# reshape data, depending on Keras backend\n",
    "X_train = X_train.reshape(X_train.shape[0], img_rows*img_cols)\n",
    "X_test = X_test.reshape(X_test.shape[0], img_rows*img_cols)\n",
    "    \n",
    "# cast floats to single precesion\n",
    "X_train = X_train.astype('float32')\n",
    "X_test = X_test.astype('float32')\n",
    "\n",
    "# rescale data in interval [0,1]\n",
    "X_train /= 255\n",
    "X_test /= 255\n",
    "\n",
    "# look at an example of data point\n",
    "print('an example of a data point with label', Y_train[20])\n",
    "plt.matshow(X_train[20,:].reshape(28,28),cmap='binary')\n",
    "plt.show()\n",
    "\n",
    "# convert class vectors to binary class matrices\n",
    "Y_train = keras.utils.to_categorical(Y_train, num_classes)\n",
    "Y_test = keras.utils.to_categorical(Y_test, num_classes)\n",
    "\n",
    "print('X_train shape:', X_train.shape)\n",
    "print('Y_train shape:', Y_train.shape)\n",
    "print()\n",
    "print(X_train.shape[0], 'train samples')\n",
    "print(X_test.shape[0], 'test samples')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2: Define the Neural Net and its Architecture\n",
    "\n",
    "We can now move on to construct our deep neural net. We shall use Keras's `Sequential()` class to instantiate a model, and will add different deep layers one by one.\n",
    "\n",
    "At this stage, we refrain from using convolutional layers. This is done further below.\n",
    "\n",
    "Let us create an instance of Keras' `Sequential()` class, called `model`. As the name suggests, this class allows us to build DNNs layer by layer. We use the `add()` method to attach layers to our model. For the purposes of our introductory example, it suffices to focus on `Dense` layers for simplicity. Every `Dense()` layer accepts as its first required argument an integer which specifies the number of neurons. The type of activation function for the layer is defined using the `activation` optional argument, the input of which is the name of the activation function in `string` format. Examples include `relu`, `tanh`, `elu`, `sigmoid`, `softmax`. \n",
    "\n",
    "In order for our DNN to work properly, we have to make sure that the numbers of input and output neurons for each layer match. Therefore, we specify the shape of the input in the first layer of the model explicitly using the optional argument `input_shape=(N_features,)`. The sequential construction of the model then allows Keras to infer the correct input/output dimensions of all hidden layers automatically. Hence, we only need to specify the size of the softmax output layer to match the number of categories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model architecture created successfully!\n"
     ]
    }
   ],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Dropout, Flatten\n",
    "from keras.layers import Conv2D, MaxPooling2D\n",
    "\n",
    "\n",
    "def create_DNN():\n",
    "    # instantiate model\n",
    "    model = Sequential()\n",
    "    # add a dense all-to-all relu layer\n",
    "    model.add(Dense(400,input_shape=(img_rows*img_cols,), activation='relu'))\n",
    "    # add a dense all-to-all relu layer\n",
    "    model.add(Dense(100, activation='relu'))\n",
    "    # apply dropout with rate 0.5\n",
    "    model.add(Dropout(0.5))\n",
    "    # soft-max layer\n",
    "    model.add(Dense(num_classes, activation='softmax'))\n",
    "    \n",
    "    return model\n",
    "\n",
    "print('Model architecture created successfully!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3: Choose the Optimizer and the Cost Function\n",
    "\n",
    "Next, we choose the loss function according to which to train the DNN. For classification problems, this is the cross entropy, and since the output data was cast in categorical form, we choose the `categorical_crossentropy` defined in Keras' `losses` module. Depending on the problem of interest one can pick any other suitable loss function. To optimize the weights of the net, we choose SGD. This algorithm is already available to use under Keras' `optimizers` module, but we could use `Adam()` or any other built-in one as well. The parameters for the optimizer, such as `lr` (learning rate) or `momentum` are passed using the corresponding optional arguments of the `SGD()` function. All available arguments can be found in Keras' online documentation at [https://keras.io/](https://keras.io/). While the loss function and the optimizer are essential for the training procedure, to test the performance of the model one may want to look at a particular `metric` of performance. For instance, in categorical tasks one typically looks at their `accuracy`, which is defined as the percentage of correctly classified data points. To complete the definition of our model, we use the `compile()` method, with optional arguments for the `optimizer`, `loss`, and the validation `metric` as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model compiled successfully and ready to be trained.\n"
     ]
    }
   ],
   "source": [
    "def compile_model(optimizer=keras.optimizers.Adam()):\n",
    "    # create the mode\n",
    "    model=create_DNN()\n",
    "    # compile the model\n",
    "    model.compile(loss=keras.losses.categorical_crossentropy,\n",
    "                  optimizer=optimizer,\n",
    "                  metrics=['accuracy'])\n",
    "    return model\n",
    "\n",
    "print('Model compiled successfully and ready to be trained.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 4: Train the model\n",
    "\n",
    "We train our DNN in minibatches, the advantages of which were explained in Sec. IV. \n",
    "\n",
    "Shuffling the training data during training improves stability of the model. Thus, we train over a number of training epochs. \n",
    "\n",
    "Training the DNN is a one-liner using the `fit()` method of the `Sequential` class. The first two required arguments are the training input and output data. As optional arguments, we specify the mini-`batch_size`, the number of training `epochs`, and the test or `validation_data`. To monitor the training procedure for every epoch, we set `verbose=True`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 60000 samples, validate on 10000 samples\n",
      "Epoch 1/10\n",
      "60000/60000 [==============================] - 12s 207us/step - loss: 0.3138 - acc: 0.9072 - val_loss: 0.1330 - val_acc: 0.9578\n",
      "Epoch 2/10\n",
      "60000/60000 [==============================] - 12s 192us/step - loss: 0.1285 - acc: 0.9627 - val_loss: 0.0942 - val_acc: 0.9711\n",
      "Epoch 3/10\n",
      "60000/60000 [==============================] - 12s 193us/step - loss: 0.0889 - acc: 0.9738 - val_loss: 0.0772 - val_acc: 0.9765\n",
      "Epoch 4/10\n",
      "60000/60000 [==============================] - 12s 194us/step - loss: 0.0696 - acc: 0.9793 - val_loss: 0.0737 - val_acc: 0.9787\n",
      "Epoch 5/10\n",
      "60000/60000 [==============================] - 12s 195us/step - loss: 0.0571 - acc: 0.9828 - val_loss: 0.0745 - val_acc: 0.9776\n",
      "Epoch 6/10\n",
      "60000/60000 [==============================] - 12s 198us/step - loss: 0.0452 - acc: 0.9858 - val_loss: 0.0799 - val_acc: 0.9782\n",
      "Epoch 7/10\n",
      "60000/60000 [==============================] - 12s 195us/step - loss: 0.0382 - acc: 0.9878 - val_loss: 0.0839 - val_acc: 0.9779\n",
      "Epoch 8/10\n",
      "60000/60000 [==============================] - 12s 197us/step - loss: 0.0337 - acc: 0.9892 - val_loss: 0.0781 - val_acc: 0.9792\n",
      "Epoch 9/10\n",
      "60000/60000 [==============================] - 12s 201us/step - loss: 0.0295 - acc: 0.9909 - val_loss: 0.0755 - val_acc: 0.9818\n",
      "Epoch 10/10\n",
      "60000/60000 [==============================] - 12s 199us/step - loss: 0.0257 - acc: 0.9921 - val_loss: 0.0863 - val_acc: 0.9792\n"
     ]
    }
   ],
   "source": [
    "# training parameters\n",
    "batch_size = 64\n",
    "epochs = 10\n",
    "\n",
    "# create the deep neural net\n",
    "model_DNN=compile_model()\n",
    "\n",
    "# train DNN and store training info in history\n",
    "history=model_DNN.fit(X_train, Y_train,\n",
    "          batch_size=batch_size,\n",
    "          epochs=epochs,\n",
    "          verbose=1,\n",
    "          validation_data=(X_test, Y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 5: Evaluate the Model Performance on the *Unseen* Test Data\n",
    "\n",
    "Next, we evaluate the model and read of the loss on the test data, and its accuracy using the `evaluate()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10000/10000 [==============================] - 0s 36us/step\n",
      "\n",
      "Test loss: 0.07659573335081113\n",
      "Test accuracy: 0.9816\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEKCAYAAADjDHn2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3Xl8XXW97//XJ/PcpEmHpAPpgIUUSgul0FZkEqWoIOLAUFQ4HlT0iJ6LCseD5wf3evUehx/H44CCA4jAQQTFYxEUGS4NU0sLnShNaGnTdMrcJE0z7M/9Y+0ku+mwd9vs7CT7/Xw88th7r732Wp9synrn+/2u9V3m7oiIiBxJSqILEBGR4U9hISIiUSksREQkKoWFiIhEpbAQEZGoFBYiIhKVwkJERKJSWIiISFQKCxERiSot0QUMlpKSEi8vL090GSIiI8rKlSvr3H1ctPVGTViUl5ezYsWKRJchIjKimNk7saynbigREYlKYSEiIlEpLEREJKpRM2ZxKF1dXdTU1NDR0ZHoUuIuKyuLyZMnk56enuhSRGQUGtVhUVNTQ35+PuXl5ZhZosuJG3envr6empoapk2bluhyRGQUGtXdUB0dHRQXF4/qoAAwM4qLi5OiBSUiiTGqwwIY9UHRK1l+TxFJjFHdDSUiMlJ194Ro7+qhfX8PbZ3d/Y+d3bTt7+l7bNvfTXFeJlefNTWu9Sgs4qypqYkHHniAG2+88ag+d8kll/DAAw9QWFgYp8pEZDC4O509oYiDeXAAP+DxgIN9bO/v7w7FXMPpUwsVFiNdU1MTP/nJTw4Ki56eHlJTUw/7uWXLlsW7NBEZoCfkNO/roqGtk8b2zuCxrZP68GNDe+9jF41tnTS1d9Le2UN3yGPeR3Z6KrmZqeRkpJGTkUpuZhp5mWlMyM8iJzOV3Iy0/sfw+zkZBy7PzUw7YBvpqfEfUVBYxNktt9xCdXU1c+fOJT09nby8PEpLS1m9ejXr16/nwx/+MNu2baOjo4ObbrqJG264AeifvqS1tZUlS5bw7ne/m8rKSiZNmsQf//hHsrOzE/ybiQxv7k7r/m4a2jojDv5dBx70w8t7w6BpXxd+mON+TkYqRTkZjM3NoCg3g2nFOYzJTg8fuPsP7Ic62OdmpJKTmUZ2eiqpKSNzfDFpwuL2P61jfW3LoG6zoqyAf/vQ7COu853vfIe1a9eyevVqnn32WT7wgQ+wdu3avlNcf/nLXzJ27Fj27dvHmWeeyRVXXEFxcfEB29i0aRMPPvggd999Nx//+Mf5/e9/z9KlSwf1dxEZCdo7u6lt2seO5o6+v/ob+g7+A1oE7Z109Rz6yJ+eav0H/pwMTp5YQFFuOmMjwqD3vd7H7IzD9wQkg6QJi+FiwYIFB1wL8cMf/pDHHnsMgG3btrFp06aDwmLatGnMnTsXgDPOOIMtW7YMWb0iQ8XdqWvtZHvTPmrDPzWNwWPvssb2roM+ZwaF2enBAT4ngyljczhtciFFuRkU9x340w9oFeRnpukMwqOUNGERrQUwVHJzc/ueP/vss/ztb3/jxRdfJCcnh/POO++Q10pkZmb2PU9NTWXfvn1DUqvIYNrf3cOOpo4gBMIH/+2N+6ht3kdtUwfbm/bROWBQNycjlUmF2Uwqyua0KYXB88JsJo7JoiQv+Iu/MCdjxHbtjCRJExaJkp+fz969ew/5XnNzM0VFReTk5PDmm2/y0ksvDXF1IoPD3Wlq7+prAUQ+bg8HxJ69+w/63Pj8TMoKs6koK+CiiglMKsymrDCbssIsJhfmUJCtFsBwobCIs+LiYhYvXswpp5xCdnY2EyZM6Hvv4osv5q677mLOnDnMmjWLs88+O4GVihxZS0cXG3fupaaxne2N/SHQGwztnT0HrJ+ZltJ38L9g1njKwi2EssKsvtZBZlpyjwOMJOaHG/ofYebPn+8Db360YcMGTj755ARVNPSS7feV+HB3aps7WF/bEvzsaGb9jha2NRzY/Vmcm9HXCphUmBO0Bop6WwbZFOdmqFUwApjZSnefH209tSxEklhXT4jqPa0RwRD8NIUHks1gWnEucyYXcuWZU6koLWBqcQ5lY7KT/uygZKOwEEkSezu62LBjL+trm/tC4a2drXT2BIPKmWkpnFRawJJTSqkoK6CitICTJuaTm6nDhCgsREYdd2dnS3830rpwi2FrQ3vfOmNzM5hdVsB1i8upKCtgdlkB5cW5pA3BlcAyMiksREaw7p4Q1XvagnGF3m6k2pYDrkeYVpLLqZPG8Ikzp1BRWkBFWQHj8zM1niBHRWEhMkK07u/mzR39gbCutoWNu/b2XZuQkZbCSRPzef/sif3dSKUF5KkbSQaB/hWJDEN1rftZV9vCutpm1m0PHrfU93cjFeWkM7tsDJ9eVN7XWpheom4kiR+FRZwd6xTlAHfeeSc33HADOTk5cahMhgN3Z3vTviAYtjeHA6KFnS39V/JPGZvN7NIxXHH6ZGZPKqCidAwTCtSNJENLYRFnh5uiPBZ33nknS5cuVViMEj0hZ3NdW9Ba6G011PafpppiMHN8HgtnFDO7rIDZZWOoKCtgTHZ6gisXUVjEXeQU5RdddBHjx4/n4YcfZv/+/Vx++eXcfvvttLW18fGPf5yamhp6enq47bbb2LVrF7W1tZx//vmUlJTwzDPPJPpXkaOwv7uHTbta+wJh7fZmNuzYy76u4Crn3vGFJadMZHbZGGaXFXDSxAJduyDDVvKExRO3wM41g7vNiafCku8ccZXIKcqfeuopHnnkEV555RXcnUsvvZTnn3+ePXv2UFZWxp///GcgmDNqzJgx/OAHP+CZZ56hpKRkcOuWQdW2v5sNO1r6QmFdbQubdu/tmx47LzONitICrlwwpS8YZo7PG5Ib1ogMluQJi2Hgqaee4qmnnmLevHkAtLa2smnTJs455xxuvvlmvv71r/PBD36Qc845J8GVyuE0tnUGoRDRlbS5rq3vhjnFuRlUlBVw7qzpzC4r4JSyMUwdm0OKZkWVES55wiJKC2AouDu33norn/3sZw96b+XKlSxbtoxbb72V973vfXzzm99MQIUSqb51P6u3NbGmd+B5ezO1zf0Dz5MKs5ldVsBlp00KgmGSBp5l9EqesEiQyCnK3//+93PbbbdxzTXXkJeXx/bt20lPT6e7u5uxY8eydOlS8vLy+PWvf33AZ9UNFX+d3SHW72hh9dZGVm1rYvW2Jt4Jn6pqBtNLcplfPpZTJoUHnksLKMrNSHDVIkNHYRFnkVOUL1myhKuvvpqFCxcCkJeXx/33309VVRVf/epXSUlJIT09nZ/+9KcA3HDDDSxZsoTS0lINcA8id6emcR+rtzWxamsTq7c1sra2pe/itgkFmcybUsTVC6Yyb2oRs8sKND+SJD1NUT6KJNvvG6vW/d28UdMbDMFjXWtwI56s9BROnTSGeVOLmDulkHlTCykdk53gikUGCIVgXwO07oa23cFj625o3QVteyB/Irz3/zumTQ+LKcrN7GLgP4BU4B53/86A908AfgmMAxqApe5eE37v34EPACnAX4GbfLQkm8RNKORU7Wll1dbGvmB4a9deQuF/OdNLcnnPu0qYN6WQeVOLmDUxX2cljRT7W6G+Cuo2Qct2yMiFzPzwT0HwmFUQfl4AacO8m9Ad9jUePgBad/Uva9sD3nPwNlIzIG8CTDkr7uXGLSzMLBX4MXARUAO8amaPu/v6iNW+B9zn7vea2QXAt4FrzWwRsBiYE17vBeBc4Nl41SsjU13rflb3thi2NfLGtmb27u8GoCArjblTi3j/7InMm1rI3CmFFOYM8wNIsnOHllqoeysIhfpN/c9bth/dtlIzIwIkvz9E+l5HhsyYA1/3fi4jH1KP4jDpDh3NEQGwC1rDB/7IQGjbEzyGug7eRko65I2H3HFBi6F0DuSOD0Ihb1zwmDs+WCdrTDCoNgTi2bJYAFS5+9sAZvYQcBkQGRYVwFfCz58B/hB+7kAWkAEYkA7sOpYi3D0pzk5JhkbX/u4e1te2RIw1NPVNu52aYpw0MZ/L5pUxd0oR86YWMq04V6esDldd+6C+OgiC+qpwILwFdVXQ1da/XmYBlJwI5ecEjyXvCh7HTIHujuDAvH9v+KcleOxoCT9v6X+vI/y86Z1gee/rQ/21PlB67oDQiQielJSIMAg/9nQevA1L7Q+AvPEwYXb4+YTgdd74/gDILhqyADga8QyLScC2iNc1wMC20uvAFQRdVZcD+WZW7O4vmtkzwA6CsPiRu2842gKysrKor6+nuLh4VAeGu1NfX09WVlaiSxk0vYPQq7Y19XUprdve0nejngkFmZw+tYhrzgoGoU+dNEZXPw837sFfz71BEBkKTdsI/iYEMCicEgTB1EUHhkLehMMfODPzIPc4zhR0h672AYHSMiB09h562d6dwWOoO3yQHwfjZh0mACYEAZAysrs74xkWh/ovPPDP35uBH5nZp4Hnge1At5nNBE4GJofX+6uZvcfdnz9gB2Y3ADcATJ069aCdTZ48mZqaGvbs2XM8v8eIkJWVxeTJk6OvOIy5O2u2N7NszU6eWLuj79TV3kHoTy8uZ96UQuZqEHp46d4PDZsPEQqbgoNsr/ScIAAmL4C5S/tDoXgGpCfgv6dZMO6RkRt098gRxTMsaoApEa8nA7WRK7h7LfARADPLA65w9+ZwCLzk7q3h954AziYIlMjP/xz4OQRnQw0sID09nWnTpg3aLySDz91Zva2JJ9buZNmaHdQ07iM1xVg0o5jrF0/jjBM0CD0suENbXX8Q1G8KwqDuLWjcAh7qX7dgUhAEcz7R30IoORHyy0b8X9fJLJ5h8SpwoplNI2gxXAlcHbmCmZUADe4eAm4lODMKYCvwj2b2bYIWyrnAnXGsVYZQKOS8trWRZWt28pe1O6ht7iA91Xj3zBK+dOGJXHTyBIqy04I+6ZRUMA8OVqO4K3HY6GiBhupgPKG+OgiH+qrg+f7m/vXSsqB4JpSeBqd8tD8UimcG3UMy6sQtLNy928y+CDxJcOrsL919nZndAaxw98eB84Bvm5kTtBq+EP74I8AFwBqCrqu/uPuf4lWrxF9PyFmxpYEn19Tw8tqNWOsuylKb+eyEbs4s72RGdiuZ+/bAazvgufCZI6HuAzdiqZCSFgRIShpYSvh177LUiHWiLEtJi1gesZ1DLcvMh7yJQVdF/sSgDzp/IqRlJubLPF7d+4PWQF8QVPUHQ2vkeSThsYTimTDn48Fj8UwomRkMMKdojCiZjOqL8mQI9XQFB5q9O4Of1p2EWnayZ8c7NO3ehrfsZKw3UkwzqXaIf3M5JQceiPMnBgfpUE/QxRHqDp6HuoOfg5b1BGe29L4f6olxWShimz0HLwt1H/6smeyicIhMgPzS/trzwq/zJwTvZyTgfiShHmiuOTAIen+atx3YbZQ7LhwEM/oDoXgmFE2D9NFz0oQc2rC4KE9Gga4OaN0Je3eFH3vD4MBgoL3+oI86KZgX0EMhodwJdIw/g55J5aQWloYPruEDbe744X0BVagn+P327oj4HnYFr3u/h/oXgsdDnTefOSYcHBFBmDfx4GDJzD+6utyD0zUPaiFUQ8Pb0LO/f92M/CAMJp8Jp10VEQ4zgnP1RaJQWEi/uk2w7g/wzgv9QdDRdPB6ltp/4Cs6gZ7JZ7K1s4CVDZk8tyOVzR157E0vZu5JJ3LxqZM5d9Y4cjJG8D+1lNT+UyFLj7BeKBS+InfngUEaGbTbXgkCprvj4M+n5x6+lZI3DtobDg6GyLONUjNg7PQgCE686MBWQt54jfnIcVE3VLLbsxHW/zEIid3rgmWlpwV90pFdKZF/EecUsz/kvLCpjmVrdvLX9Ttp6egmLzON9548nktOLeU97xpHVrr6tA+p9yrfQ4XJwFZb5AVqwAHjCH0/4e4jjSPIMVA3lBze7jdh/R+CgNizATCYejZc/H+g4lIoKDvkxzq6enj+rT08sfYN/rZ+F3v3d1OQlcZFFRO55NSJvPvEEjLTdLCKygyyC4Of8Scded39e8NhsgtyxmocQRJGYZEM3GH3hqAFsf4PsOdNwOCERbDku3Dyh6Dg0P0r+zp7eHbjbpat3cnfN+yirbOHwpx0lpw6kSWnlrJ4RgkZaTp3Pm56p5comZnoSiTJKSxGK3fYvT5oPaz/Q3DxFAYnLIZLvhcExGGuWm3b380zG3fzxJqd/P3N3ezr6mFsbgaXzp3EJadO5OzpxbpITiTJKCxGE3fYtTYcEH8MrrK1lCAgzvosnPShYAzisB937v6/b/ODv75FR1eIkrxMrjhjEpecUsqCaWNJU0CIJC2FxUjnDjvf6B+kbqgOAqL8HDj780ELIm981M10dof4xmNr+N3KGi6qmMBn3j2N+eVjSdWsrSKCwmJkcocdrwfdS+v/GJxTb6kw7RxY9E9BQBzFbJwNbZ187jcreWVLAzddeCJffu+Jo3qWXhE5egqLkcIdalf1D1I3bgkCYvq5sPjLcNIHIbf4qDe7adderr/3VXa37OeHV83j0tMOfSaUiCQ3hcVw5g61r/WPQTS9E8xVNP08OOd/BAGRM/aYN//sxt380wOryExP5aEbzmbe1KJBK11ERheFxXDjDttXwrrHYP3j0Lw1uM3i9PPg3K/BrEuOKyCCXTj3Vm7hjv9ez6yJBfziU/MpK9T9IUTk8BQWw0VHCzz/70EronlbEBAzLoDzb4VZS4JJ6wZBV0+I2/+0jvtf2spFFRO48xNzyc3UPwMROTIdJYaD/a3w249BzavBnD7nfyMcEIWDupvm9i5ufGAly6vq+dy5M/ja+2fpHtUiEhOFRaJ1tsODVwZB8bFfQcVlcdnN5ro2/uHXr7KtsZ3vfnQOH5s/JfqHRETCFBaJ1NUBD10NW16AK+6JW1BUVtfx+ftfI8Xgt585mwXTjm/MQ0SSj8IiUbo74eFPwtvPwGU/gVM/GpfdPPjKVm77w1qmleTyi0+dydTiBNyIR0RGPIVFIvR0wSPXwaYn4YN3wrxrBn8XIed/L9vAL17YzLnvGsd/Xj2Pgqz0Qd+PiCQHhcVQ6+mGR2+AN/87mPF1/nWDvou9HV186cFVPLNxD9ctLucbl5yseZ1E5LgoLIZSqAf++AVY9yi873/BWTcM+i62NbTzmXtXULWnlW9dfgrXnHXCoO9DRJKPwmKohELwp5vgjYfggn8N5nAaZCu2NPDZ36ykqyfEfdcvYPHM2OeHEhE5EoXFUHCHZTfDqt/Ae74G7/nqoO/i0ddquOX3a5hUlM09n5rPjHF5g74PEUleCot4c4cn/wVW/AIW3wTn/8ugbj4Ucr731EZ+8mw1C6cX89Olp1OYkzGo+xARUVjEkzs8fTu89BM46/Pw3tuD+y8PkvbObr7yX6t5ct0urlowlTsum6072IlIXCgs4unZ78AL/z/Mvx4u/vagBsWO5n185t4VbNjRwm0frOD6xeW6B4WIxI3CIl7+7/fhue/AvKVwyfcHNShe39bEP963gvbOHn7xqTM5/6Tod8ITETkeCot4qPwRPH0HzPkEfOiHkDJ4XUN/fmMH//zwasblZ/KbfziLWRPzB23bIiKHo7AYbK/cDU99Ayo+HEzjkZI6KJt1d/7z71X84K9vMf+EIn527RkU52UOyrZFRKKJGhZmtgL4FfCAuzfGv6QRbOWvg1NkZ30gmBgwdXCyuKOrh6898gaPv17LR06fxLc/ciqZaYMTQiIisYilf+RKoAx41cweMrP3m0ZSD7b6AfjTl+HE9wVTjacOzjxMu/d2cOXPX+Lx12v52sWz+P7HTlNQiMiQixoW7l7l7t8A3gU8APwS2Gpmt5uZ5roGWPNIMI3H9HPh47+BtMHpHlpf28KHf7ScjTv3ctfSM7jxvJk640lEEiKmkVczmwN8H/gu8Hvgo0AL8Pf4lTZCrH88mBhw6kK48kFIzxqUzf51/S4+elclDvzucwu5+JSJg7JdEZFjEcuYxUqgCfgFcIu77w+/9bKZLY5nccPexr/AI9fDpDPg6v+CjOO/V4S787Pn3+b//OVN5kwaw92fnM/4gsEJIBGRYxXLCOzH3P3tQ73h7h8Z5HpGjqq/wcPXwsRTYekjkHn8p7B2dof4xmNr+N3KGj4wp5Tvf+w0stI1PiEiiRdLN9RnzKyw94WZFZnZ/4pjTcPf28/BQ9fAuFlw7aOQNea4N9ne2c3SX7zM71bWcNOFJ/Kjq+YpKERk2IglLJa4e1Pvi/Dps5fEr6Rh7p1KePBKKJoG1/4RsosGZbNPrNnJK5sb+O5H5/CVi96lgWwRGVZiCYtUM+s7vcfMsoHkvBps26vw249BwST41OOQWzxom66srmdsbgZXnD550LYpIjJYYhmzuB942sx+BThwPXBvXKsajmpXwf1XQO64ICjyBm8+JnensrqOhdOLSUlRi0JEhp+oYeHu/25ma4ALAQP+p7s/GffKhpOda+A3l0P2GPjUn6CgbFA3v7mujR3NHSyaOXgtFRGRwRTTfBTu/gTwxNFu3MwuBv4DSAXucffvDHj/BIKL/MYBDcBSd68JvzcVuAeYQtCiucTdtxxtDcdt95tw32WQnhMEReGUQd/F8up6ABbN0G1QRWR4ijpmYWZnm9mrZtZqZp1m1mNmLTF8LhX4MbAEqACuMrOKAat9D7jP3ecAdwDfjnjvPuC77n4ysADYHduvNIjqquC+SyElLQiKovK47ObF6jrKxmRRXnz812mIiMRDLAPcPwKuAjYB2cBngP+M4XMLgCp3f9vdO4GHgMsGrFMBPB1+/kzv++FQSXP3vwK4e6u7t8ewz8HTsBnu/RCEeuCTj0PxjLjsJhRyXqyuZ9HMEp0BJSLDVkzTfbh7FZDq7j3u/ivg/Bg+NgnYFvG6Jrws0uvAFeHnlwP5ZlZMMA9Vk5k9amarzOy74ZbK0GjaCvdeCt374JN/hPEnxW1X63e00NjexWKNV4jIMBZLWLSbWQaw2sz+3cy+AuTG8LlD/ZnsA17fDJxrZquAc4HtQDfBWMo54ffPBKYDnz5oB2Y3mNkKM1uxZ8+eGEqKQUtt0KLoaIZr/wATTxmc7R5GZXUdoPEKERneYgmLa8PrfRFoIxhwvuKInwjUhNftNRmojVzB3Wvd/SPuPg/4RnhZc/izq8JdWN3AH4DTB+7A3X/u7vPdff64ceNiKCmKvbuCoGirD67MLpt7/NuMorK6nhnjcpmg+Z9EZBg7YliEu36+5e4d7t7i7re7+z+Hu6WieRU40cymhVsmVwKPD9h+iZn11nArwZlRvZ8tMrPeBLgAWB/j73Rs2uqCweyWHXDN72Dy/LjuDoK5oF7Z3MDimWpViMjwdsSwcPceYFz4YH9Uwi2CLwJPAhuAh919nZndYWaXhlc7D9hoZm8BE4BvRez3ZoKLAdcQdGndfbQ1xKy9Ae77MDRugasfghMWxm1XkV6vaaK9s0ddUCIy7MVyncUWYLmZPU7QDQWAu/8g2gfdfRmwbMCyb0Y8fwR45DCf/SswJ4b6jk9HM9z/EajbCFc9BNPeE/dd9lpeVYcZnD1d95ASkeEtlrCoDf+kAMc/D/dw07UPerqCO9zNvHBId11ZXc8pZWMozDnqhpuIyJCKZbqP24eikITJnwiffR5ShnY68PbOblZtbeT6d08b0v2KiByLWO6U9wwHn/KKu18Ql4oSYYiDAuDVLY109TiLNV4hIiNALN1QN0c8zyI4bbY7PuUkj8qqOtJTjfnlg3M/DBGReIqlG2rlgEXLzey5ONWTNCqr65k3tYicjJjmchQRSahYJhIcG/FTYmbvByYOQW2jVlN7J2trm9UFJSIjRix/1q4kGLMwgu6nzcA/xLOo0e6lt+txR/NBiciIEUs3lE7XGWTLq+rJyUhlzuTCRJciIhKTWLqhvmBmhRGvi8zsxviWNbpVVtexYNpYMtJimvRXRCThYjla/aO7N/W+cPdG4B/jV9LotrO5g+o9bRqvEJERJZawSLGIu/KEJxfUJcfHqG9Kco1XiMgIEssA95PAw2Z2F8FA9+eAv8S1qlFseVU9RTnpnDyxINGliIjELJaw+DpwA/B5gjOingLuiWdRo5W782J1HQtnFJOSoluoisjIEUtYZAN3u/td0NcNlQkM7T2xR4Et9e3UNndwo8YrRGSEiWXM4mmCwOiVDfwtPuWMbsurgvEK3exIREaaWMIiy91be1+En+fEr6TRq7K6jtIxWZQX6+sTkZEllrBoM7O++1+b2RnAvviVNDqFQs6L1fUsmlFCxMllIiIjQixjFl8GfmdmteHXpcAn4lfS6LRhZwuN7V2a4kNERqRYpvt41cxOAmYRnA31prt3xb2yUaayqh5A99sWkREp1vmxZwEVBPezmGdmuPt98Str9FleXcf0cblMHJOV6FJERI5aLHfK+zfgPIKwWAYsAV4AFBYx6uoJ8crmBq44fXKiSxEROSaxDHB/FLgQ2Onu1wGnEVxnITF6fVsT7Z09Gq8QkRErlrDY5+4hoNvMCoDdwPT4ljW6LK+qxwzOnq6wEJGRKZYxixXhKcrvJrgRUivwSlyrGmWWV9cxu6yAwhzNvygiI1MsZ0P13rviLjP7C1Dg7m/Et6zRY19nD6u2NnL9Yt1DSkRGrljPhgLA3bfEqY5R69UtDXT1OIs0xYeIjGC6VVucLa+uIz3VOLO8KNGliIgcM4VFnFVW1TNvShE5GUfViBMRGVYOewQzs7FH+qC7Nwx+OaNLc3sXa2ubuenCExNdiojIcTnSn7srCe6Md6hZ7xydPhvVi2/X464pyUVk5DtsWLi7Tt85TpXVdeRkpHLa5MJElyIiclyijllYYKmZ3RZ+PdXMFsS/tJFveVUdZ5aPJSNNQ0MiMrLFchT7CbAQuDr8ei/w47hVNErsaumgek+bpvgQkVEhllN0znL3081sFYC7N5qZLkWOorI6uIWqpiQXkdEglpZFl5mlEgxqY2bjgFBcqxoFllfVU5iTTkVpQaJLERE5brGExQ+Bx4DxZvYtgunJ/3dcqxrh3J3KqjoWTi8mJUW3UBWRkS+WuaF+a2YrCaYpN+DD7r4h7pWNYO/Ut1Pb3MHnz1cXlIiMDrFelLcbeDDyPV2Ud3hYKXF9AAAOEUlEQVTLw+MVi2docFtERodYL8qbCjSGnxcCWwFdh3EYlVX1lI7JYlpJbqJLEREZFIcds3D3ae4+HXgS+JC7l7h7MfBB4NFYNm5mF5vZRjOrMrNbDvH+CWb2tJm9YWbPmtnkAe8XmNl2M/vR0f1aiRMKOZXVdSycUYyZxitEZHSIZYD7THdf1vvC3Z8Azo32ofAZVD8muGd3BXCVmVUMWO17wH3uPge4A/j2gPf/J/BcDDUOG2/u3EtjexeLdcqsiIwisYRFnZn9q5mVh1sC3wDqY/jcAqDK3d92907gIeCyAetUAE+Hnz8T+b6ZnQFMAJ6KYV/DRu/1FZoPSkRGk1jC4ipgHMHps38AxoeXRTMJ2Bbxuia8LNLrwBXh55cD+WZWbGYpwPeBrx5pB2Z2g5mtMLMVe/bsiaGk+FteVcf0cblMHJOV6FJERAZNLKfONgA3mVkBEHL31hi3fbjZaiPdDPzIzD4NPA9sB7qBG4Fl7r7tSP3+7v5z4OcA8+fPH7jtIdfVE+KVzQ1cfvrATBQRGdmihoWZnQrcB4wNv64DPuXua6N8tAaYEvF6MlAbuYK71wIfCW83D7jC3ZvNbCFwjpndCOQBGWbW6u4HDZIPJ2/UNNHW2aPxChEZdWKZG+pnwD+7+zMAZnYewV/zi6J87lXgRDObRtBiuJL+yQgJb6sEaHD3EHAr8EsAd78mYp1PA/OHe1BAMMWHGSzU9RUiMsrEMmaR2xsUAO7+LBD1AgJ37wa+SHDq7QbgYXdfZ2Z3mNml4dXOAzaa2VsEg9nfOrryh5flVXXMLiugMEfzLIrI6BJLy+Lt8L0sfhN+vRTYHMvGw6fcLhuw7JsRzx8BHomyjV8Dv45lf4m0r7OHVVub+PTi8kSXIiIy6GJpWVxPcDbUowRnRI0DrotnUSPRinca6OwJsUhdUCIyCsVyNlQj8KUhqGVEW15VT3qqsWDa2Ogri4iMMLGcDTUf+BegPHL98FXXElZZXce8KUXkZMTSsyciMrLEcmT7LcHFcWvQTY8Oqbm9izXbm/nSBScmuhQRkbiIJSz2uPvjca9kBHtpcz3umuJDREavWMLi38zsHoI5nPb3LnT3mGaeTQaVVXVkp6cyd0phoksREYmLWMLiOuAkIJ3+bignxmnKk8Hy6noWTBtLRlosJ5eJiIw8sYTFae5+atwrGaF2tXRQtbuVj50xOfrKIiIjVCx/Cr90iPtQSNiL1cFs7RqvEJHRLJaWxbuBT5nZZoIxCwNcp84GllfVUZiTTkVpQaJLERGJm1jC4uK4VzFCuTuV1fUsnF5MSopuoSoio1csV3C/MxSFjETv1LezvWkfnzt3eqJLERGJK52+cxwqw+MVizReISKjnMLiOCyvrmNiQRbTS6LO2C4iMqIpLI5RKOS8WF3PopnFHOnWryIio4HC4hi9uXMvDW2dLNItVEUkCSgsjlFldR0Ai2fq/hUiMvopLI5RZXU900tyKR2TnehSRETiTmFxDLp6Qrz8djBeISKSDBQWx+CNmibaOns0XiEiSUNhcQwqq+oxg4XT1bIQkeSgsDgGy6vrqCgtoCg3I9GliIgMCYXFUdrX2cNr7zRpllkRSSoKi6O04p0GOntCLJyhLigRSR4Ki6NUWV1PWoqxoHxsoksRERkyCoujVFlVx7ypheRmxjK7u4jI6KCwOArN+7pYs71Zp8yKSNJRWByFl96uJ+SwSOMVIpJkFBZH4cXqerLTU5k3tSjRpYiIDCmFxVFYXlXHmdPGkpGmr01EkouOejHa3dLBpt2tLFYXlIgkIYVFjPpuoarBbRFJQgqLGFVW1zEmO52KsoJElyIiMuQUFjFwd5ZX1bNwejGpKbqFqogkH4VFDLY2tLO9aZ/uiiciSUthEYPlVcF4xUKNV4hIklJYxKCyuo4JBZnMGJeb6FJERBJCYRFFKOS8WF3P4hklmGm8QkSSU1zDwswuNrONZlZlZrcc4v0TzOxpM3vDzJ41s8nh5XPN7EUzWxd+7xPxrPNINu7aS31bJ4t0/woRSWJxCwszSwV+DCwBKoCrzKxiwGrfA+5z9znAHcC3w8vbgU+6+2zgYuBOMyuMV61HsryqDtB8UCKS3OLZslgAVLn72+7eCTwEXDZgnQrg6fDzZ3rfd/e33H1T+HktsBsYF8daD+vF6nqmleRSVpidiN2LiAwL8QyLScC2iNc14WWRXgeuCD+/HMg3swP+hDezBUAGUB2nOg+ruyfEy5sb1KoQkaQXz7A41GiwD3h9M3Cuma0CzgW2A919GzArBX4DXOfuoYN2YHaDma0wsxV79uwZvMrDXq9ppnV/t+63LSJJL55hUQNMiXg9GaiNXMHda939I+4+D/hGeFkzgJkVAH8G/tXdXzrUDtz95+4+393njxs3+L1UleHxirOnq2UhIsktnmHxKnCimU0zswzgSuDxyBXMrMTMemu4FfhleHkG8BjB4Pfv4ljjEVVW11NRWsDY3IxElSAiMizELSzcvRv4IvAksAF42N3XmdkdZnZpeLXzgI1m9hYwAfhWePnHgfcAnzaz1eGfufGq9VA6unpYubVRU3yIiABp8dy4uy8Dlg1Y9s2I548Ajxzic/cD98eztmhWbGmkszuk6ytERNAV3Ie1vLqOtBRjQfnYRJciIpJwCovDqKyuZ+6UQnIz49r4EhEZERQWh9C8r4s1NU3qghIRCVNYHMLLb9cTcnS/bRGRMIXFIVRW15OVnsLcqQmZjkpEZNhRWBxCZXUdZ5aPJTMtNdGliIgMCwqLAXbv7eCtXa2a4kNEJILCYoAXq4NbqC7WLVRFRPooLAZYXlVHQVYaFWUFiS5FRGTYUFgMUFldz8IZxaSm6BaqIiK9FBYRtta3U9O4T+MVIiIDKCwiLK/uvYWqwkJEJJLCIsLyqjrG52cyY1xuoksRERlWFBZh7s6L1fUsnlmCmcYrREQiKSzCNu7aS31bp+63LSJyCAqLsOVVwfUVmjxQRORgCouwyqo6yotzmFSYnehSRESGHYUF0N0T4uXNDWpViIgchsICeGN7M637uzXFh4jIYSgsCLqgABZqcFtE5JAUFgSD2yeXFjA2NyPRpYiIDEtJHxYdXT2s3Nqou+KJiBxB0odFS0cXF8+eyAUnj090KSIiw1ZaogtItPH5WfzwqnmJLkNEZFhL+paFiIhEp7AQEZGoFBYiIhKVwkJERKJSWIiISFQKCxERiUphISIiUSksREQkKnP3RNcwKMxsD/DOcWyiBKgbpHJGOn0XB9L3cSB9H/1Gw3dxgruPi7bSqAmL42VmK9x9fqLrGA70XRxI38eB9H30S6bvQt1QIiISlcJCRESiUlj0+3miCxhG9F0cSN/HgfR99Eua70JjFiIiEpVaFiIiElXSh4WZXWxmG82sysxuSXQ9iWRmU8zsGTPbYGbrzOymRNeUaGaWamarzOy/E11LoplZoZk9YmZvhv+NLEx0TYlkZl8J/3+y1sweNLOsRNcUT0kdFmaWCvwYWAJUAFeZWUViq0qobuB/uPvJwNnAF5L8+wC4CdiQ6CKGif8A/uLuJwGnkcTfi5lNAr4EzHf3U4BU4MrEVhVfSR0WwAKgyt3fdvdO4CHgsgTXlDDuvsPdXws/30twMJiU2KoSx8wmAx8A7kl0LYlmZgXAe4BfALh7p7s3JbaqhEsDss0sDcgBahNcT1wle1hMArZFvK4hiQ+OkcysHJgHvJzYShLqTuBrQCjRhQwD04E9wK/C3XL3mFluootKFHffDnwP2ArsAJrd/anEVhVfyR4WdohlSX96mJnlAb8HvuzuLYmuJxHM7IPAbndfmehahok04HTgp+4+D2gDknaMz8yKCHohpgFlQK6ZLU1sVfGV7GFRA0yJeD2ZUd6UjMbM0gmC4rfu/mii60mgxcClZraFoHvyAjO7P7ElJVQNUOPuvS3NRwjCI1m9F9js7nvcvQt4FFiU4JriKtnD4lXgRDObZmYZBANUjye4poQxMyPok97g7j9IdD2J5O63uvtkdy8n+Hfxd3cf1X85Hom77wS2mdms8KILgfUJLCnRtgJnm1lO+P+bCxnlA/5piS4gkdy928y+CDxJcDbDL919XYLLSqTFwLXAGjNbHV72L+6+LIE1yfDxT8Bvw39YvQ1cl+B6EsbdXzazR4DXCM4iXMUov5pbV3CLiEhUyd4NJSIiMVBYiIhIVAoLERGJSmEhIiJRKSxERCQqhYXIMGBm52lmWxnOFBYiIhKVwkLkKJjZUjN7xcxWm9nPwve7aDWz75vZa2b2tJmNC68718xeMrM3zOyx8HxCmNlMM/ubmb0e/syM8ObzIu4X8dvwlcEiw4LCQiRGZnYy8AlgsbvPBXqAa4Bc4DV3Px14Dvi38EfuA77u7nOANRHLfwv82N1PI5hPaEd4+TzgywT3VplOcEW9yLCQ1NN9iBylC4EzgFfDf/RnA7sJpjD/r/A69wOPmtkYoNDdnwsvvxf4nZnlA5Pc/TEAd+8ACG/vFXevCb9eDZQDL8T/1xKJTmEhEjsD7nX3Ww9YaHbbgPWONIfOkbqW9kc870H/f8owom4okdg9DXzUzMYDmNlYMzuB4P+jj4bXuRp4wd2bgUYzOye8/FrgufD9QWrM7MPhbWSaWc6Q/hYix0B/uYjEyN3Xm9m/Ak+ZWQrQBXyB4EZAs81sJdBMMK4B8CngrnAYRM7Sei3wMzO7I7yNjw3hryFyTDTrrMhxMrNWd89LdB0i8aRuKBERiUotCxERiUotCxERiUphISIiUSksREQkKoWFiIhEpbAQEZGoFBYiIhLV/wNzSG4lzf9OZgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEKCAYAAADjDHn2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3Xt81PWd7/HXJ/cruQ1BIEACQQuCgtzUcFpdq2IvXuplrbVrrVvbrZ7tOXvaXT3by9bd7qNnt9vT7UPbqq3dbm31KPZCt1gvLbYCKjcBuYiEmwwoCSQBQu7J5/zxm8AQAxlCJpNM3s/HYx4z85vfb36fzCOZd37f7/f3/Zm7IyIicjopiS5ARESGPoWFiIj0SWEhIiJ9UliIiEifFBYiItInhYWIiPRJYSEiIn1SWIiISJ8UFiIi0qe0RBcwUEKhkJeXlye6DBGRYWXt2rUH3X10X+slTViUl5ezZs2aRJchIjKsmNmeWNZTM5SIiPRJYSEiIn1SWIiISJ+Sps9CRKQ/2tvbCYfDtLS0JLqUuMrKyqKsrIz09PR+ba+wEJERLRwOk5+fT3l5OWaW6HLiwt05dOgQ4XCYioqKfr2HmqFEZERraWmhpKQkaYMCwMwoKSk5q6MnhYWIjHjJHBTdzvZnHPFh0dDUxnd/v51N+w4nuhQRkSFrxIdFSorxnRff4rnN7ya6FBEZgRoaGvje9753xtt96EMfoqGhIQ4V9W7Eh8WorHQunFDI8uqDiS5FREagU4VFZ2fnabdbunQphYWF8SrrPUZ8WABUTQmxMXyYIy3tiS5FREaY++67jx07djBr1izmzZvH5Zdfzm233cbMmTMBuP7665kzZw7nn38+jzzyyPHtysvLOXjwILt372batGl85jOf4fzzz+eqq66iubl5wOvU0FmgqjLEg8uqeW1nHVdOH5PockQkQb7+m81s2X9kQN9z+rhRfO2j55/y9W9+85ts2rSJ9evX89JLL/HhD3+YTZs2HR/i+thjj1FcXExzczPz5s3jxhtvpKSk5KT32L59O0888QSPPvoot9xyC8888wy33377gP4ccT2yMLNFZrbNzKrN7L5eXv+cmb1hZuvNbLmZTY967f7IdtvM7Op41nnRpEKy0lNYoaYoEUmw+fPnn3QuxHe/+10uvPBCLr74Yvbu3cv27dvfs01FRQWzZs0CYM6cOezevXvA64rbkYWZpQIPAVcCYWC1mS1x9y1Rq/3c3X8QWf9a4NvAokho3AqcD4wDXjSzc9399I14/ZSZlsq88mL1W4iMcKc7Ahgsubm5xx+/9NJLvPjii7zyyivk5ORw2WWX9XquRGZm5vHHqampcWmGiueRxXyg2t13unsb8CRwXfQK7h59vJcLeOTxdcCT7t7q7ruA6sj7xc3CyhDVNY0cOJLcp/yLyNCSn5/P0aNHe33t8OHDFBUVkZOTw5tvvsmrr746yNWdEM8+i/HA3qjnYWBBz5XM7B7gb4AM4M+ito3+VMKRZXFTVRkCYEX1QT52UVk8dyUiclxJSQlVVVXMmDGD7Oxsxow50W+6aNEifvCDH3DBBRdw3nnncfHFFyeszniGRW+nC/p7Frg/BDxkZrcBXwbuiHVbM7sbuBtg4sSJZ1Xs9LGjKMpJZ7nCQkQG2c9//vNel2dmZvLss8/2+lp3v0QoFGLTpk3Hl3/xi18c8Pogvs1QYWBC1PMyYP9p1n8SuP5MtnX3R9x9rrvPHT26z6sCnlZKinHplBArqw/h/p5cEhEZ0eIZFquBqWZWYWYZBB3WS6JXMLOpUU8/DHR38y8BbjWzTDOrAKYCq+JYKxA0Rb17pIUdtcfivSsRkWElbs1Q7t5hZvcCzwGpwGPuvtnMHgDWuPsS4F4z+yDQDtQTNEERWe8pYAvQAdwTr5FQ0RZG9VtUlubFe3ciIsNGXE/Kc/elwNIey74a9fgLp9n2G8A34lfde00syaGsKJsV1Qe549Lywdy1iMiQpuk+elhYGeKVnYfo6OxKdCkiIkOGwqKHqsoQR1s6eENTlouIHKew6OHSKcGcK5r6Q0QGQ3+nKAf4zne+Q1NT0wBX1DuFRQ8leZlMGzuKFdWHEl2KiIwAwyUsNOtsLxZWlvCTlXtobuskOyM10eWISBKLnqL8yiuvpLS0lKeeeorW1lZuuOEGvv71r3Ps2DFuueUWwuEwnZ2dfOUrX+HAgQPs37+fyy+/nFAoxLJly+Jap8KiF1WVIR59eRerd9fx/nPP7mQ/ERlGnr0P3n1jYN/znJlwzTdP+XL0FOXPP/88ixcvZtWqVbg71157LX/605+ora1l3Lhx/Pa3vwWCOaMKCgr49re/zbJlywiFQgNbcy/UDNWL+RXFpKcaK3ao30JEBs/zzz/P888/z+zZs7nooot488032b59OzNnzuTFF1/k7/7u73j55ZcpKCgY9Np0ZNGLnIw0Zk8sUie3yEhzmiOAweDu3H///Xz2s599z2tr165l6dKl3H///Vx11VV89atf7eUd4kdHFqewsDLE5v1HqD/WluhSRCSJRU9RfvXVV/PYY4/R2NgIwL59+6ipqWH//v3k5ORw++2388UvfpF169a9Z9t405HFKVRVhvj2C2/xys5DfGjm2ESXIyJJKnqK8muuuYbbbruNSy65BIC8vDwef/xxqqur+dKXvkRKSgrp6el8//vfB+Duu+/mmmuuYezYsXHv4LZkmWF17ty5vmbNmgF7v47OLmY98ALXzhrHP98wc8DeV0SGlq1btzJt2rRElzEoevtZzWytu8/ta1s1Q51CWmoKF08uVr+FiAgKi9Oqqgyx51ATe+sG56QXEZGhSmFxGt2XWl2pIbQiSS1ZmuNP52x/RoXFaUwtzWN0fibLNfWHSNLKysri0KHkvkKmu3Po0CGysrL6/R4aDXUaZsbCyhB/equWri4nJaW3S4OLyHBWVlZGOBymtrY20aXEVVZWFmVlZf3eXmHRh6rKEL98fR/bDhxl2thRiS5HRAZYeno6FRUViS5jyFMzVB+qKjVluYiIwqIPYwuymTw6l+UKCxEZwRQWMVhYGeK1nXW0dehSqyIyMiksYnDplBDN7Z2s39uQ6FJERBJCYRGDSyaXkGKoKUpERiyFRQwKctKZWVaoTm4RGbEUFjFaWFnC+r0NHG1pT3QpIiKDTmERo6opITq7nFW76hJdiojIoFNYxOiiSUVkpqWo30JERqS4hoWZLTKzbWZWbWb39fL635jZFjPbaGa/N7NJUa91mtn6yG1JPOuMRVZ6KvMrilmpeaJEZASKW1iYWSrwEHANMB34uJlN77Ha68Bcd78AWAz8S9Rrze4+K3K7Nl51nolLp4TYduAoNUdbEl2KiMigiueRxXyg2t13unsb8CRwXfQK7r7M3bsvFvEq0P9ZrgbBwu4py3V0ISIjTDzDYjywN+p5OLLsVO4Cno16nmVma8zsVTO7vrcNzOzuyDprBmPGyOnjRlGYk65+CxEZceI562xv83n3OmG8md0OzAU+ELV4orvvN7PJwB/M7A1333HSm7k/AjwCwTW4B6bsU0tNMS6dUsLK6oO4O2aaslxERoZ4HlmEgQlRz8uA/T1XMrMPAn8PXOvurd3L3X1/5H4n8BIwO461xuzSKSH2H25h18FjiS5FRGTQxDMsVgNTzazCzDKAW4GTRjWZ2WzgYYKgqIlaXmRmmZHHIaAK2BLHWmPW3W+hs7lFZCSJW1i4ewdwL/AcsBV4yt03m9kDZtY9uulfgTzg6R5DZKcBa8xsA7AM+Ka7D4mwmFSSw/jCbFaok1tERpC4XinP3ZcCS3ss+2rU4w+eYruVwMx41tZfZkZVZQm/2/QunV1Oqi61KiIjgM7g7oeqyhBHWjrYtO9woksRERkUCot+uHRK0G+hIbQiMlIoLPphdH4m7zsnn5U7FBYiMjIoLPqpqjLE6t31tLR3JroUEZG4U1j008LKEG0dXazZXZ/oUkRE4k5h0U/zK4pJSzFWqClKREYAhUU/5WamMXuiLrUqIiODwuIsVFWGeGPfYRqa2hJdiohIXCkszsLCyhDu8OpOnc0tIslNYXEWLpxQSG5Gqs63EJGkp7A4C+mpKSyYXKJ5okQk6SkszlJVZYhdB4+xr6E50aWIiMSNwuIsacpyERkJFBZn6dwxeYTyMhUWIpLUFBZnqXvK8hXVh3CP+5VdRUQSQmExAKoqQxxsbOWtA42JLkVEJC4UFgOgqlJTlotIclNYDIDxhdlUhHLVbyEiSUthMUCqKkt4bech2ju7El2KiMiAU1gMkIWVIY61dbJhb0OiSxERGXAKiwFy8eQSzNRvISLJSWExQApzMpg5vkD9FiKSlBQWA6iqMsTrbzdwrLUj0aWIiAwohcUAWlgZoqPLWbWrLtGliIgMKIXFAJozqYiMtBT1W4hI0olrWJjZIjPbZmbVZnZfL6//jZltMbONZvZ7M5sU9dodZrY9crsjnnUOlKz0VOaVF6nfQkSSTp9hYWb/YmajzCw98oV+0Mxuj2G7VOAh4BpgOvBxM5veY7XXgbnufgGwGPiXyLbFwNeABcB84GtmVnQmP1iiVFWGePPdo9QebU10KSIiAyaWI4ur3P0I8BEgDJwLfCmG7eYD1e6+093bgCeB66JXcPdl7t4UefoqUBZ5fDXwgrvXuXs98AKwKIZ9JlzVlGDqj5U7dHQhIskjlrBIj9x/CHjC3WPtvR0P7I16Ho4sO5W7gGf7ue2QMWN8AaOy0tQUJSJJJS2GdX5jZm8CzcDnzWw00BLDdtbLsl7n8I40a80FPnAm25rZ3cDdABMnToyhpPhLTTEunRI6PmW5WW8/iojI8NLnkYW73wdcQtC30A4co0dz0imEgQlRz8uA/T1XMrMPAn8PXOvurWeyrbs/4u5z3X3u6NGjYyhpcFRNDbGvoZk9h5r6XllEZBiIpYP7ZqDD3TvN7MvA48C4GN57NTDVzCrMLAO4FVjS471nAw8TBEVN1EvPAVeZWVGkY/uqyLJhoWpKCaCpP0QkecTSZ/EVdz9qZgsJOp5/Any/r43cvQO4l+BLfivwlLtvNrMHzOzayGr/CuQBT5vZejNbEtm2DvhHgsBZDTxwBn0lCVcRymVcQZY6uUUkacTSZ9EZuf8w8H13/7WZ/UMsb+7uS4GlPZZ9NerxB0+z7WPAY7HsZ6gJLrUa4oWtB+jsclJT1G8hIsNbLEcW+8zsYeAWYKmZZca43YhWVRmioamdLfuPJLoUEZGzFsuX/i0ETUmL3L0BKCa28yxGtEsrg36LFWqKEpEkEMtoqCZgB3C1md0LlLr783GvbJgrzc/ivDH5Ot9CRJJCLKOhvgD8DCiN3B43s/8e78KSQVVliFW76mhp7+x7ZRGRISyWZqi7gAXu/tVI5/TFwGfiW1ZyqKosobWji3V76hNdiojIWYklLIwTI6KIPNbwnhgsmFxCaoqp30JEhr1YwuLHwGtm9g+RIbOvAj+Ka1VJIi8zjdkTCllefSjRpYiInJVYOri/DdwJ1AH1wJ3u/p14F5YsLq0M8Ua4gcPN7YkuRUSk304ZFmZW3H0DdhNM8/FTYE9kmcRgYWWILodXd+roQkSGr9Odwb2WYKbX7v6J7llfLfJ4chzrShqzJhSSk5HKiuqDXH3+OYkuR0SkX04ZFu5eMZiFJKuMtBQWVBRrUkERGdY0bccgqKoMsbP2GO8cbk50KSIi/aKwGARVlcGlVldoVJSIDFMKi0Fw3ph8QnkZmvpDRIatU/ZZ9DXiaThdXyLRUlKMS6aEWF59UJdaFZFh6UxGQ0XTaKgztLCyhN9s2E91TSNTx+QnuhwRkTOi0VCDpLvfYnn1QYWFiAw7scw6a2Z2u5l9JfJ8opnNj39pyaWsKIfykhz1W4jIsBRLB/f3gEuA2yLPjwIPxa2iJHZpZYhXd9bR0dmV6FJERM5ILGGxwN3vAVoA3L0eyIhrVUlqYWWIxtYONoQPJ7oUEZEzEktYtJtZKpHpPsxsNKB/jfvhksklmKGmKBEZdmIJi+8CvwRKzewbwHLgn+NaVZIqys3g/HGjNPWHiAw7pxs6C4C7/8zM1gJXEAyjvd7dt8a9siRVVRniseW7aGrrICejz49fRGRIiHWK8hrgCeDnwAFNUd5/CytDtHc6q3bpnEYRGT5iPSlvIsGFjwwoBN4GdB5GP8wrLyYjLYUV1Qe57LzSRJcjIhKTUx5ZuHuFu08GngM+6u4hdy8BPgL8YrAKTDZZ6anMmVikSQVFZFiJpYN7nrsv7X7i7s8CH4jlzc1skZltM7NqM7uvl9ffb2brzKzDzG7q8Vqnma2P3JbEsr/hYuHUEFveOcKhxtZElyIiEpNYwuKgmX3ZzMrNbJKZ/T3Q57/FkeG2DwHXANOBj5vZ9B6rvQ18iqAvpKdmd58VuV0bQ53DRvfUHyt36OhCRIaHWMLi48BoguGzvwJKI8v6Mh+odved7t4GPAlcF72Cu+92942MsPM2Zo4vID8rTedbiMiwEcvQ2TrgC2Y2Cuhy98YY33s8sDfqeRhYcAa1ZZnZGqAD+Ka7/+oMth3SUlOMSyaXsGKHwkJEhodYJhKcaWavA28Am81srZnNiOG9TzW1eawmuvtcgjmpvmNmU3qp7W4zW2Nma2pra8/grRNv4dQQe+uaeftQU6JLERHpUyzNUA8Df+Puk9x9EvC/gEdi2C4MTIh6Xgbsj7Uwd98fud8JvATM7mWdR9x9rrvPHT16dKxvPSRET1kuIjLUxRIWue6+rPuJu78E5Maw3WpgqplVmFkGcCsQ06gmMysys8zI4xBQBWyJZdvhYnIol3NGZakpSkSGhVjCYqeZfSUyGqrczL4M7OprI3fvAO4lOE9jK/CUu282swfM7FoAM5tnZmHgZuBhM9sc2XwasMbMNgDLCPoskioszIyqyhArqw/S1XUmrXMiIoMvlsmJPg18neBEPAP+BNwZy5tHzs9Y2mPZV6MeryZonuq53UpgZiz7GM4WTi3hmXVhtrxzhBnjCxJdjojIKcUyGqoe+OtBqGXEqZrSfb7FQYWFiAxpsYyGmmtmv4icab2x+zYYxSW70lFZTC3NY7mm/hCRIS6WZqifAV8iGDo7ok6eGwxVlSGeXP02rR2dZKalJrocEZFexdLBXevuS9x9l7vv6b7FvbIRYmFliJb2LtbtaUh0KSIipxTLkcXXzOyHwO+B4zPfubtmnh0ACyYXk5pirNxxkEumlCS6HBGRXsVyZHEnMAtYBHw0cvtIPIsaSfKz0rmwrEAn54nIkBbLkcWF7p7cw1g72iAtI2G7X1gZ4sFl1RxpaWdUVnrC6hAROZVYjixe7WVq8eTRWAvfuxjW/TRhJVRVhuhyeG2nLrUqIkNTLGGxEFgfuYjRRjN7I6mGzqakQuEEWHIv/PaL0Nk+6CXMnlhEdnqqpiwXkSErlmaoRXGvIpFyiuETz8CLX4NXHoSaLXDzTyBv8CYmzEhLYX5FsfotRGTI6vPIInq4bNIOnU1Ng6u/AR/7IexbC49cBvtfH9QSFlaGqK5pZEdtrJcLEREZPLE0Q40cF9wMn34OzOCxRbDhyUHb9aIZ55CflcYtP3iFNbvVdyEiQ4vCoqdxs+Dul6BsHvzys/C7/w2dHXHf7YTiHH51TxWjstP5+KOv8vSavX1vJCIySBQWvckNwSd/CQs+B68+BI/fAMfiP3/TlNF5/OrzVSyoKOFLizfyjd9uoVPTl4vIEKCwOJXUdLjm/8B134O3X4NHL4N334j7bgty0vmPO+fxqUvLefTlXdz1k9UcaRn8EVoiItEUFn2Z/Qm489mgKeqHV8KmZ+K+y7TUFP7h2vP55xtmsnz7QT72vZXsPngs7vsVETkVhUUsyuYE/RjjZsHiT8MLX4Wuzrjv9rYFE/npXQs42NjK9d9bwUoNrRWRBFFYxCp/DPzFEph7F6z4d/jZzdBcH/fdXjKlhCX3LKQ0P5NPPraKn76aXKOWRWR4UFicibQM+Mi34aP/Drv+BI9cDjVb477biSU5PPNXl3LZuaP5yq828ZVfbaK9U5cWEZHBo7Dojzmfgk/9Ftqb4NErYMuSuO8yPyudR/5iLp/9wGR++uoe7nhsFQ1NbXHfr4gIKCz6b+ICuPuPUDoNnvok/OGfoCu+/+2nphj3XzONf7v5Qtbsrue6h1ZQXXM0rvsUEQGFxdkZNRbuXAqzb4c//Ss8+XFoORz33d44p4wn7r6YY62d3PDQSpZtq4n7PkVkZFNYnK20TLj2QfjQt6D6xaBZqvatuO92zqQifn1vFROKc7jrP1bzw5d34q4T+EQkPhQWA8EM5n8mGC3V0gCP/hlsezbuux1fmM3iv7qEq88/h3/67Vb+dvFGWjviP6RXREYehcVAKq8KzscomQJP3Ap//Je492PkZKTx0G0X8YUrpvL02jCfePQ1Dja29r2hiMgZUFgMtIIy+PTv4IJbYdk3gs7v1vh2QqekGP/zynN58LbZbNp/mOseXMGW/Ufiuk8RGVniGhZmtihyhb1qM7uvl9ffb2brzKzDzG7q8dodZrY9crsjnnUOuPRsuOEHsOibQXPUDz8Ih3bEfbcfuWAcT3/2Ujq7nJt+sJLnNr8b932KyMgQt7Aws1TgIeAaYDrw8V6u5f028Cng5z22LQa+BiwA5gNfM7OieNUaF2Zw8V8Fs9c21gQn8G1/Ie67nVlWwJJ7qzh3TD6f/elaHvzDdnV8i8hZi+eRxXyg2t13unsb8CRwXfQK7r7b3TcCPRv2rwZecPc6d68HXmC4Xt518geCfozCicEUIS9/G+L85V06Kosn776Y62eN41vPv8VfP7melnZ1fItI/8UzLMYD0VfwCUeWxXvboadoEtz1PMz4GPz+67D4TmiL7yyyWemp/N8/n8XfLjqP/9q4n1sefoUDR1riuk8RSV7xDAvrZVms/1LHtK2Z3W1ma8xsTW1t7RkVN+gycuDGH8GV/whbfg0/ugrqdsV1l2bG5y+r5JFPzmVHTSPXPricDXsb4rpPEUlO8QyLMDAh6nkZsH8gt3X3R9x9rrvPHT16dL8LHTRmUPXX8InFcDgMj14OO5bFfbdXTh/DM5+/lPTUFG55+BV+vX5f3PcpIsklnmGxGphqZhVmlgHcCsQ6495zwFVmVhTp2L4qsiw5VF4Bdy+D/LHw+Mdg5YNx78d43zmj+PU9VVxYVsgXnlzPt57bRpcu2SoiMYpbWLh7B3AvwZf8VuApd99sZg+Y2bUAZjbPzMLAzcDDZrY5sm0d8I8EgbMaeCCyLHkUT4a7XoD3fQSe/3v4xd3Q1hTXXZbkZfL4Xy7g1nkTeHBZNZ97fC3HWjviuk8RSQ6WLMMq586d62vWrEl0GWfOHV7+t2DW2nNmwPu/BFOvhvSsOO7S+fGK3fzTb7dw7ph8fnjHXMqKcuK2PxEZusxsrbvP7Ws9ncGdaGbw/i/Cbf8PGmvhqb+Ab02FX90DO1+Ky+VbzYxPL6zgx3fOZ19DM9c9uII1u5PrwE1EBpaOLIaSzg7Y/SfY+DRs/Q20HYW8c2DGjXDBzTB2VhAuA2hHbSN/+ZM1hOub+MYNM7ll7oS+NxKRpBHrkYXCYqhqb4a3fhcEx/bnoasdSqbCBbfAzJuCPo8BcripnXt+vo7l1Qf5y4UV3P+haaSmDGwoicjQpLBIJs31wbkZG5+GPcuDZePnBsFx/g2QV3rWu+jo7OKffruV/1i5m6rKEj5/WSWXTC4hRaEhcrLODqjfBTVbgklCx8yA0umQlpHoyvpFYZGsDodh0zNBcBx4AywVJl8WBMf7PgyZ+Wf19k+sept/XrqVoy0djC/M5saLxnPjnDImleQOSPkiw0ZXFxzeCzVbg2Co2Qq1W4OLm3X2uAxAakYQGONmwbjZQZPxMAkQhcVIULMV3ng6uDW8DWnZcN41QXBMuaLfv6gt7Z08v+UAT6/Zy/Lqg7jD/IpibppTxodnjiU3M22AfxCRBHKHxgORQHgzKhjehLbGE+uNGg+l0yK36cF9Rj68uwH2r4d31sP+DdAaubRydICMnRXcl04Prq45hCgsRhJ32PsabHwKNv8Smusguyhoopp5M0y4GFL6N/DtncPN/GLdPhavDbPr4DFyMlL50Myx3DSnjPnlxWqmkuGlqS4Ige5A6D5qaK4/sU5OSSQMpp8IhtHnQXZh3+/vHjRR7X/9RIC8swFaIgGSkg5jpp8Ij7GzYMz5CQ0QhcVI1dkOO/4QBMe2pdDeBAUTgk7xmbcEv6j94O6s3VPP4rVh/mvjOzS2djCxOIcbLyrjYxeNZ0KxztOQIaS1EWq3RYVC5L4x6hovmaNOPlIY/b7gPm+Apw46HiDdRx+R++gAKZ128hHImBmDFiAKCwn+YLYtDYJjxx/AO6H0/GAY7oyboLB/w2Sb2jp4bvO7PL0mzModhwC4dEoJN80p45oZY8nOSB3In0IGQmd70BnbejRoWmltDIZmtx6NPI4saz0S9Xrje7dpb4LUdEjPCS7ylZ4d9fg0y9KyelnvFOunpsc+RLyjFQ6+dXIg1GyFhj0n1knLDo4MSqdD6ftOHDGMGj/gQ9Fj5g71u08Oj/3roSUy0WdKWiRAZkc1YZ0fl5N1FRZyssZa2PKrIDjCq4Jlk6qCI47p10NOcb/eNlzfxDNr97F43V721jWTl5nGh2eO5ea5ZcyZVITF+sfoDh0twZQnbY3BFO4dzcGXXGdb5L77cRt0dZx4fKp1OtuDIcenXOd07xe1jlnwx5uaEbmlR26RxylRj8/49YzY37ujJepL/MjJX/J9ffH37JA9ldRMyMwLBkpk5AePMyLPM/MgPTf4XNqbg+DoaAnu25tPLIu+72w7818qS42ER9apw8U9CIlDO4J/giD4HEPnBl+yo6edOGooKoeUYfAPjHsQcj2PQLqbyLoD5HgT1uygCessA0RhIadWtws2LQ5GVB3cFnwhTb0yCI5zrwmmUz8V9+ALoO1Y5Eu9CdqO0dV6lLfCNazZvpe33n6X9M4mxuU6s8ak8b7iFHKt7UQIRAdC27Hgi6WtEbznNbD66fiXb+SLNqXHl3Nqjy/n93yZR62Tkh68Z2dbJHh6CZ2uniHVcfoQ6mpj7/Q2AAAO2klEQVQfmJ+zW0aPL/Tuxycty48KgbxTLMsPPoOB1NkRhH5vQfKeZX0ET/Rj74LQ1JObkYqnDIvRR2fEPRi8sv/1UwfI6Gkw5XK46h/7tQuFhfTNHd7dGBxtbHoGjr4TfGmU/7fgv7UegXD8ucc+BUmrp9FEFp1pOWTm5JGbV0hKZi5kRN3SezzvvqVln/5LvtcgOIMmjERx7+Wop8fRTm/BlJb53i/59Nx+D16QYao7QKLDIycENz7ar7dTWMiZ6eqEPSsizVSrg0P9jLzg0L+3L/JTfcH3WL6noY1n1oZ5Zt0+9jU0k5+VxkcvHMfNc8qYNaEw9mYqEYkLhYUMKV1dzqs7D/H02jDPbnqHlvYuKkvzuGlOGR+bPZ7SUfGbZVdETk1hIUPWkZZ2lm58h6fXhlm7p54Ugw+cO5qb507gimmlZKYNg85IkSShsJBhYWdtI4vXhvnFun28e6SFwpx0rrtwHDfNmcCM8aPUTCUSZwoLGVY6u5zl1QdZvDbMc5vfpa2ji4pQLgsqiplXXsz8imLKirIVHiIDTGEhw9bhpnaWbNzPsjdrWLO7jiMtwaVfxxZkMa+8mHkVxcwvL2ZqaZ6mGxE5SwoLSQpdXc62A0dZvbuO13bVsXpXHTVHgxPMCnPSmTupmPkVRcwrL2bG+ALSUzWMVORMxBoWmj5UhrSUFGPa2FFMGzuKv7ikHHfn7bomVu2qY9WuOlbvruPFrQcAyE5P5aJJhUGzVXkxsycWaeoRkQGiIwsZ9mqOtLB6d/3xo4833z2CO6SlGDPLCphfHvR7zC0vojAnyc7wFTlLaoaSEetwczvr9tSzandw9LEx3EB7Z/B7ft6YfOZXnOj3OKdA53fIyKawEIloae9k/d4GVu+qY9XuOtbtqedYWzBlyYTibOaXlxzv96gI5WrElYwo6rMQichKT+XiySVcPLkECK43vuWdI8f7PJZtq+GZdWEAQnmZx4NjXnkx08aOIlUjrkR0ZCHi7uyobWTVrqDfY9WuOvY1NAOQn5nGrImFXFBWwAVlhVxYVqimK0kqaoYSOQv7GpqPN1utf7uBbQeO0tkV/K2U5mdy4YRCLowEyAVlBeo4l2FrSDRDmdki4N+BVOCH7v7NHq9nAv8JzAEOAX/u7rvNrBzYCmyLrPqqu38unrWKRBtfmM342eO5fvZ4AJrbOtnyzmE27D3MxnADG8OHeWHLgePrTyrJiRx5BAEyY/wocjLUyivJI26/zWaWCjwEXAmEgdVmtsTdt0StdhdQ7+6VZnYr8H+AP4+8tsPdZ8WrPpEzkZ2RypxJxcyZdOKKgoeb29m07zAbwg1s3HuYtbvr+M2G/QCkGEwtzefCCSear847J5+MNJ00KMNTPP/1mQ9Uu/tOADN7ErgOiA6L64B/iDxeDDxoGooiw0RBdjpVlSGqKkPHl9UebWVjuIEN4eAI5MWtNTy1Jug8z0hNYdq4UcePPi4sK2Dy6Dx1oMuwEM+wGA/sjXoeBhacah137zCzw0BJ5LUKM3sdOAJ82d1fjmOtIgNidH4mV0wbwxXTxgBB53m4vjk4+ggfZsPeBp5ZG+Y/X9kDQG5GKjPGFzBrQuHx/g9NmChDUTzDorff9p696ada5x1gorsfMrM5wK/M7Hx3P3LSxmZ3A3cDTJw4cQBKFhlYZsaE4hwmFOfwkQvGAcEMuztrG48ffWwIH+bHK3bT1hlcg7w4NyNq9FVwPzo/M5E/hkhcwyIMTIh6XgbsP8U6YTNLAwqAOg+GaLUCuPtaM9sBnAucNNzJ3R8BHoFgNFQ8fgiRgZaaYkwdk8/UMfncNKcMgLaOLra9ezRyBBIchfzpre1EBmAxriCLaWNHUVmaR2VpHlPH5FNZmkdepjrRZXDE8zdtNTDVzCqAfcCtwG091lkC3AG8AtwE/MHd3cxGE4RGp5lNBqYCO+NYq0hCZaSlMLOsgJllBcAkAJraOti8/wgb9gbh8daBo7y8/eDxIxAIpm2vLM1jaml+JETymFqap6G8MuDiFhaRPoh7gecIhs4+5u6bzewBYI27LwF+BPzUzKqBOoJAAXg/8ICZdQCdwOfcvS5etYoMRTkZacfPJO/W0dnF3vpmth84SnVtI9UHGtle08gTq96mub3z+HqhvEwqS3OZWprP1DF5x49IRudlqj9E+kUn5Ykkga4uZ//hZrbXdAfIUaprgiA5Grl4FAQjuKZGjkCmjA6as6aW5jG2IEshMkINiZPyRGRwpKQYZUU5lBXlcPl5pceXuzs1R1vZfqCR6pqjbI8EyHObD1B37MRgxdyM1MjRR+RIZHQQKGVFORraK4DCQiSpmRljRmUxZlQWC6eGTnrtUGPr8aOP6shteXXt8UkVATLTUpg8OugHmRppyppYEozuGpWVPtg/jiSQwkJkhCrJy6QkL5MFk0tOWn64uZ3qmkZ21ATNWdtrGlm7p54lG04ezFiYk86EohwmFGdH7iO3omzGF2WTmaarFCYThYWInKQgO505k4qYM6nopOVNbR3srD3G23VN7K1rCu7rm3nznaO8uKXmpFFaZnDOqCwmFOVQFgmTid1hUpzNmPwsUtS8NawoLEQkJjkZacwYX8CM8QXvea2ryzlwtIW9dc3srWtib30QJuG6Zl7ZcYhfHtlH9FiajNQUxhdlHz8SCe67AyWbgux0dbgPMQoLETlrKSnG2IJsxhZkM7+i+D2vt3Z0sq++mb31J8IkXNfM23VNbAw30NDUftL6+ZlplBXnMPGkJq5sJhYHnfhZ6WriGmwKCxGJu8y0VCaPzmPy6LxeXz/a0s7eSHiE65sigdLMjtpj/PGtWlrau05aP5SXQVFOcCvISacoJ53CnAwKstMpysmgMCc9uGVnUJQb3GdnKGDOhsJCRBIuPyud6ePSmT5u1Htec3dqG1vZW9dMuL6Jtw81sf9wM/XH2mlobmNvXRNvhIPHPUMlWmZayvEA6Q6TE2GTQWF2EDgnvZadrqOYCIWFiAxpZkZpfhal+Vnv6XTvqaW9k4amduqb2mhoaqehqY2G5uD54ZOWt7Pr4DFebwqawKI753vKTk+NBEh3oJwIlYLsdHIz08jNSCUnI428zDRyMlOD+4xUcjPSyM1MS4rrmCgsRCRpZKWnck5B6hldJ93daW7vpL47XCJhUt/UxuHmduqPBYHT/dr2msbjjzu6YpsBIz3VIqESCZHMNHIzowImoztgguW5p1iWmxGEUW5G2qCfLKmwEJERzczIyQi+lMcXZse8nbvT1NbJsbYOjrV2cqy1I3je2sGxtg6aWjtpbO2gqa2DxtZOmqLWO9YWrHuosemk7Vs7Tn2E01NWesrxMLmgrIAHb7uoPz9+zBQWIiL9YBY5WshMg/yBec+Ozi6Otb03WI71EjbBfSdNrR2MO4OQ6y+FhYjIEJGWmkJBdgoF2UNvKpXh3+siIiJxp7AQEZE+KSxERKRPCgsREemTwkJERPqksBARkT4pLEREpE8KCxER6ZO5xza3yVBnZrXAnrN4ixBwcIDKGe70WZxMn8fJ9HmckAyfxSR3H93XSkkTFmfLzNa4+9xE1zEU6LM4mT6Pk+nzOGEkfRZqhhIRkT4pLEREpE8KixMeSXQBQ4g+i5Pp8ziZPo8TRsxnoT4LERHpk44sRESkTyM+LMxskZltM7NqM7sv0fUkkplNMLNlZrbVzDab2RcSXVOimVmqmb1uZv+V6FoSzcwKzWyxmb0Z+R25JNE1JZKZ/c/I38kmM3vCzGK/luswNKLDwsxSgYeAa4DpwMfNbHpiq0qoDuB/ufs04GLgnhH+eQB8Adia6CKGiH8Hfufu7wMuZAR/LmY2HvhrYK67zwBSgVsTW1V8jeiwAOYD1e6+093bgCeB6xJcU8K4+zvuvi7y+CjBl8H4xFaVOGZWBnwY+GGia0k0MxsFvB/4EYC7t7l7Q2KrSrg0INvM0oAcYH+C64mrkR4W44G9Uc/DjOAvx2hmVg7MBl5LbCUJ9R3gb4GuRBcyBEwGaoEfR5rlfmhmuYkuKlHcfR/wLeBt4B3gsLs/n9iq4mukh4X1smzEDw8zszzgGeB/uPuRRNeTCGb2EaDG3dcmupYhIg24CPi+u88GjgEjto/PzIoIWiEqgHFArpndntiq4mukh0UYmBD1vIwkP5Tsi5mlEwTFz9z9F4muJ4GqgGvNbDdB8+SfmdnjiS0pocJA2N27jzQXE4THSPVBYJe717p7O/AL4NIE1xRXIz0sVgNTzazCzDIIOqiWJLimhDEzI2iT3uru3050PYnk7ve7e5m7lxP8XvzB3ZP6P8fTcfd3gb1mdl5k0RXAlgSWlGhvAxebWU7k7+YKkrzDPy3RBSSSu3eY2b3AcwSjGR5z980JLiuRqoBPAm+Y2frIsv/t7ksTWJMMHf8d+FnkH6udwJ0Jridh3P01M1sMrCMYRfg6SX42t87gFhGRPo30ZigREYmBwkJERPqksBARkT4pLEREpE8KCxER6ZPCQmQIMLPLNLOtDGUKCxER6ZPCQuQMmNntZrbKzNab2cOR6100mtm/mdk6M/u9mY2OrDvLzF41s41m9svIfEKYWaWZvWhmGyLbTIm8fV7U9SJ+FjkzWGRIUFiIxMjMpgF/DlS5+yygE/gEkAusc/eLgD8CX4ts8p/A37n7BcAbUct/Bjzk7hcSzCf0TmT5bOB/EFxbZTLBGfUiQ8KInu5D5AxdAcwBVkf+6c8GagimMP9/kXUeB35hZgVAobv/MbL8J8DTZpYPjHf3XwK4ewtA5P1WuXs48nw9UA4sj/+PJdI3hYVI7Az4ibvff9JCs6/0WO90c+icrmmpNepxJ/r7lCFEzVAisfs9cJOZlQKYWbGZTSL4O7opss5twHJ3PwzUm9l/iyz/JPDHyPVBwmZ2feQ9Ms0sZ1B/CpF+0H8uIjFy9y1m9mXgeTNLAdqBewguBHS+ma0FDhP0awDcAfwgEgbRs7R+EnjYzB6IvMfNg/hjiPSLZp0VOUtm1ujueYmuQySe1AwlIiJ90pGFiIj0SUcWIiLSJ4WFiIj0SWEhIiJ9UliIiEifFBYiItInhYWIiPTp/wP5BLJ0Oy5OZQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# evaluate model\n",
    "score = model_DNN.evaluate(X_test, Y_test, verbose=1)\n",
    "\n",
    "# print performance\n",
    "print()\n",
    "print('Test loss:', score[0])\n",
    "print('Test accuracy:', score[1])\n",
    "\n",
    "# look into training history\n",
    "\n",
    "# summarize history for accuracy\n",
    "plt.plot(history.history['acc'])\n",
    "plt.plot(history.history['val_acc'])\n",
    "plt.ylabel('model accuracy')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'test'], loc='best')\n",
    "plt.show()\n",
    "\n",
    "# summarize history for loss\n",
    "plt.plot(history.history['loss'])\n",
    "plt.plot(history.history['val_loss'])\n",
    "plt.ylabel('model loss')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'test'], loc='best')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Step 6: Modify the Hyperparameters to Optimize Performance of the Model\n",
    "\n",
    "Last, we show how to use the grid search option of scikit-learn to optimize the \n",
    "hyperparameters of our model. An excellent blog on this by Jason Brownlee can be found on [https://machinelearningmastery.com/grid-search-hyperparameters-deep-learning-models-python-keras/](https://machinelearningmastery.com/grid-search-hyperparameters-deep-learning-models-python-keras/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/1\n",
      "45000/45000 [==============================] - 5s 118us/step - loss: 1.1397 - acc: 0.6571\n",
      "15000/15000 [==============================] - 1s 38us/step\n",
      "45000/45000 [==============================] - 1s 23us/step\n",
      "Epoch 1/1\n",
      "45000/45000 [==============================] - 5s 117us/step - loss: 1.1162 - acc: 0.6668\n",
      "15000/15000 [==============================] - 1s 39us/step\n",
      "45000/45000 [==============================] - 1s 26us/step\n",
      "Epoch 1/1\n",
      "45000/45000 [==============================] - 5s 117us/step - loss: 1.1128 - acc: 0.6707\n",
      "15000/15000 [==============================] - 1s 38us/step\n",
      "45000/45000 [==============================] - 1s 26us/step\n",
      "Epoch 1/1\n",
      "45000/45000 [==============================] - 5s 120us/step - loss: 1.1454 - acc: 0.6564\n",
      "15000/15000 [==============================] - 1s 42us/step\n",
      "45000/45000 [==============================] - 1s 24us/step\n",
      "Epoch 1/1\n",
      "45000/45000 [==============================] - 8s 177us/step - loss: 0.3399 - acc: 0.8988\n",
      "15000/15000 [==============================] - 1s 42us/step\n",
      "45000/45000 [==============================] - 1s 25us/step\n",
      "Epoch 1/1\n",
      "45000/45000 [==============================] - 8s 177us/step - loss: 0.3430 - acc: 0.8996\n",
      "15000/15000 [==============================] - 1s 44us/step\n",
      "45000/45000 [==============================] - 1s 26us/step\n",
      "Epoch 1/1\n",
      "45000/45000 [==============================] - 8s 183us/step - loss: 0.3414 - acc: 0.8996\n",
      "15000/15000 [==============================] - 1s 45us/step\n",
      "45000/45000 [==============================] - 1s 26us/step \n",
      "Epoch 1/1\n",
      "45000/45000 [==============================] - 8s 180us/step - loss: 0.3402 - acc: 0.8990\n",
      "15000/15000 [==============================] - 1s 45us/step\n",
      "45000/45000 [==============================] - 1s 28us/step\n",
      "Epoch 1/1\n",
      "45000/45000 [==============================] - 8s 169us/step - loss: 0.3175 - acc: 0.9076\n",
      "15000/15000 [==============================] - 1s 47us/step\n",
      "45000/45000 [==============================] - 1s 26us/step\n",
      "Epoch 1/1\n",
      "45000/45000 [==============================] - 8s 169us/step - loss: 0.3346 - acc: 0.9031\n",
      "15000/15000 [==============================] - 1s 51us/step\n",
      "45000/45000 [==============================] - 1s 27us/step\n",
      "Epoch 1/1\n",
      "45000/45000 [==============================] - 8s 170us/step - loss: 0.3227 - acc: 0.9070\n",
      "15000/15000 [==============================] - 1s 47us/step\n",
      "45000/45000 [==============================] - 1s 27us/step\n",
      "Epoch 1/1\n",
      "45000/45000 [==============================] - 8s 171us/step - loss: 0.3497 - acc: 0.8977\n",
      "15000/15000 [==============================] - 1s 52us/step\n",
      "45000/45000 [==============================] - 1s 26us/step\n",
      "Epoch 1/1\n",
      "45000/45000 [==============================] - 10s 231us/step - loss: 0.3711 - acc: 0.8901\n",
      "15000/15000 [==============================] - 1s 52us/step\n",
      "45000/45000 [==============================] - 1s 26us/step\n",
      "Epoch 1/1\n",
      "45000/45000 [==============================] - 10s 225us/step - loss: 0.3731 - acc: 0.8888\n",
      "15000/15000 [==============================] - 1s 45us/step\n",
      "45000/45000 [==============================] - 1s 22us/step\n",
      "Epoch 1/1\n",
      "45000/45000 [==============================] - 10s 223us/step - loss: 0.3653 - acc: 0.8930\n",
      "15000/15000 [==============================] - 1s 48us/step\n",
      "45000/45000 [==============================] - 1s 23us/step\n",
      "Epoch 1/1\n",
      "45000/45000 [==============================] - 9s 203us/step - loss: 0.3721 - acc: 0.8891\n",
      "15000/15000 [==============================] - 1s 44us/step\n",
      "45000/45000 [==============================] - 1s 20us/step\n",
      "Epoch 1/1\n",
      "45000/45000 [==============================] - 8s 167us/step - loss: 0.3551 - acc: 0.8951\n",
      "15000/15000 [==============================] - 1s 48us/step\n",
      "45000/45000 [==============================] - 1s 20us/step\n",
      "Epoch 1/1\n",
      "45000/45000 [==============================] - 7s 166us/step - loss: 0.3561 - acc: 0.8954\n",
      "15000/15000 [==============================] - 1s 48us/step\n",
      "45000/45000 [==============================] - 1s 19us/step\n",
      "Epoch 1/1\n",
      "45000/45000 [==============================] - 7s 166us/step - loss: 0.3496 - acc: 0.8962\n",
      "15000/15000 [==============================] - 1s 50us/step\n",
      "45000/45000 [==============================] - 1s 20us/step\n",
      "Epoch 1/1\n",
      "45000/45000 [==============================] - 8s 171us/step - loss: 0.3621 - acc: 0.8937\n",
      "15000/15000 [==============================] - 1s 52us/step\n",
      "45000/45000 [==============================] - 1s 20us/step\n",
      "Epoch 1/1\n",
      "45000/45000 [==============================] - 6s 140us/step - loss: 0.4106 - acc: 0.8811\n",
      "15000/15000 [==============================] - 1s 55us/step\n",
      "45000/45000 [==============================] - 1s 22us/step\n",
      "Epoch 1/1\n",
      "45000/45000 [==============================] - 6s 143us/step - loss: 0.4125 - acc: 0.8780\n",
      "15000/15000 [==============================] - 1s 56us/step\n",
      "45000/45000 [==============================] - 1s 20us/step\n",
      "Epoch 1/1\n",
      "45000/45000 [==============================] - 6s 142us/step - loss: 0.3998 - acc: 0.8839\n",
      "15000/15000 [==============================] - 1s 56us/step\n",
      "45000/45000 [==============================] - 1s 21us/step\n",
      "Epoch 1/1\n",
      "45000/45000 [==============================] - 6s 142us/step - loss: 0.4139 - acc: 0.8779\n",
      "15000/15000 [==============================] - 1s 62us/step\n",
      "45000/45000 [==============================] - 1s 22us/step\n",
      "Epoch 1/1\n",
      "45000/45000 [==============================] - 9s 206us/step - loss: 0.2943 - acc: 0.9140\n",
      "15000/15000 [==============================] - 1s 55us/step\n",
      "45000/45000 [==============================] - 1s 20us/step\n",
      "Epoch 1/1\n",
      "45000/45000 [==============================] - 9s 206us/step - loss: 0.3047 - acc: 0.9111\n",
      "15000/15000 [==============================] - 1s 61us/step\n",
      "45000/45000 [==============================] - 1s 20us/step\n",
      "Epoch 1/1\n",
      "45000/45000 [==============================] - 9s 208us/step - loss: 0.2920 - acc: 0.9137\n",
      "15000/15000 [==============================] - 1s 60us/step\n",
      "45000/45000 [==============================] - 1s 21us/step\n",
      "Epoch 1/1\n",
      "45000/45000 [==============================] - 9s 209us/step - loss: 0.3015 - acc: 0.9112\n",
      "15000/15000 [==============================] - 1s 62us/step\n",
      "45000/45000 [==============================] - 1s 22us/step\n",
      "Epoch 1/1\n",
      "60000/60000 [==============================] - 12s 199us/step - loss: 0.2694 - acc: 0.9210\n",
      "Best: 0.957033 using {'optimizer': 'Nadam'}\n",
      "0.874133 (0.005897) with: {'optimizer': 'SGD'}\n",
      "0.953867 (0.003900) with: {'optimizer': 'RMSprop'}\n",
      "0.951767 (0.001012) with: {'optimizer': 'Adagrad'}\n",
      "0.950500 (0.003029) with: {'optimizer': 'Adadelta'}\n",
      "0.955000 (0.002867) with: {'optimizer': 'Adam'}\n",
      "0.945917 (0.002984) with: {'optimizer': 'Adamax'}\n",
      "0.957033 (0.001874) with: {'optimizer': 'Nadam'}\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "from keras.wrappers.scikit_learn import KerasClassifier\n",
    "\n",
    "# call Keras scikit wrapper\n",
    "model_gridsearch = KerasClassifier(build_fn=compile_model, \n",
    "                        epochs=1, \n",
    "                        batch_size=batch_size, \n",
    "                        verbose=1)\n",
    "\n",
    "# list of allowed optional arguments for the optimizer, see `compile_model()`\n",
    "optimizer = ['SGD', 'RMSprop', 'Adagrad', 'Adadelta', 'Adam', 'Adamax', 'Nadam']\n",
    "# define parameter dictionary\n",
    "param_grid = dict(optimizer=optimizer)\n",
    "# call scikit grid search module\n",
    "grid = GridSearchCV(estimator=model_gridsearch, param_grid=param_grid, n_jobs=1, cv=4)\n",
    "grid_result = grid.fit(X_train,Y_train)\n",
    "\n",
    "# summarize results\n",
    "print(\"Best: %f using %s\" % (grid_result.best_score_, grid_result.best_params_))\n",
    "means = grid_result.cv_results_['mean_test_score']\n",
    "stds = grid_result.cv_results_['std_test_score']\n",
    "params = grid_result.cv_results_['params']\n",
    "for mean, stdev, param in zip(means, stds, params):\n",
    "    print(\"%f (%f) with: %r\" % (mean, stdev, param))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating Convolutional Neural Nets with Keras\n",
    "\n",
    "We have so far considered each MNIST data sample as a $(28\\times 28,)$-long 1d vector. This approach neglects any spatial structure in the image. On the other hand, we do know that in every one of the hand-written digits there are *local* spatial correlations between the pixels, which we would like to take advantage of to improve the accuracy of our classification model. To this end, we first need to reshape the training and test input data as follows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X_train shape: (60000, 28, 28, 1)\n",
      "Y_train shape: (60000, 10)\n",
      "\n",
      "60000 train samples\n",
      "10000 test samples\n"
     ]
    }
   ],
   "source": [
    "# reshape data, depending on Keras backend\n",
    "if keras.backend.image_data_format() == 'channels_first':\n",
    "    X_train = X_train.reshape(X_train.shape[0], 1, img_rows, img_cols)\n",
    "    X_test = X_test.reshape(X_test.shape[0], 1, img_rows, img_cols)\n",
    "    input_shape = (1, img_rows, img_cols)\n",
    "else:\n",
    "    X_train = X_train.reshape(X_train.shape[0], img_rows, img_cols, 1)\n",
    "    X_test = X_test.reshape(X_test.shape[0], img_rows, img_cols, 1)\n",
    "    input_shape = (img_rows, img_cols, 1)\n",
    "    \n",
    "print('X_train shape:', X_train.shape)\n",
    "print('Y_train shape:', Y_train.shape)\n",
    "print()\n",
    "print(X_train.shape[0], 'train samples')\n",
    "print(X_test.shape[0], 'test samples')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One can ask the question of whether a neural net can learn to recognize such local patterns. As we saw in Sec. X of the review, this can be achieved by using convolutional layers. Luckily, all we need to do is change the architecture of our DNN, i.e. introduce small changes to the function `create_model()`. We can also merge **Step 2** and **Step 3** for convenience: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_CNN():\n",
    "    # instantiate model\n",
    "    model = Sequential()\n",
    "    # add first convolutional layer with 10 filters (dimensionality of output space)\n",
    "    model.add(Conv2D(10, kernel_size=(5, 5),\n",
    "                     activation='relu',\n",
    "                     input_shape=input_shape))\n",
    "    # add 2D pooling layer\n",
    "    model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "    # add second convolutional layer with 20 filters\n",
    "    model.add(Conv2D(20, (5, 5), activation='relu'))\n",
    "    # apply dropout with rate 0.5\n",
    "    model.add(Dropout(0.5))\n",
    "    # add 2D pooling layer\n",
    "    model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "    # flatten data\n",
    "    model.add(Flatten())\n",
    "    # add a dense all-to-all relu layer\n",
    "    model.add(Dense(20*4*4, activation='relu'))\n",
    "    # apply dropout with rate 0.5\n",
    "    model.add(Dropout(0.5))\n",
    "    # soft-max layer\n",
    "    model.add(Dense(num_classes, activation='softmax'))\n",
    "    \n",
    "    # compile the model\n",
    "    model.compile(loss=keras.losses.categorical_crossentropy,\n",
    "                  optimizer='Adam',\n",
    "                  metrics=['accuracy'])\n",
    "    \n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training the deep conv net (**Step 4**) and evaluating its performance (**Step 6**) proceeds exactly as before:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 60000 samples, validate on 10000 samples\n",
      "Epoch 1/10\n",
      "60000/60000 [==============================] - 39s 646us/step - loss: 0.2610 - acc: 0.9183 - val_loss: 0.0750 - val_acc: 0.9825\n",
      "Epoch 2/10\n",
      "60000/60000 [==============================] - 32s 533us/step - loss: 0.0932 - acc: 0.9709 - val_loss: 0.0551 - val_acc: 0.9876\n",
      "Epoch 3/10\n",
      "60000/60000 [==============================] - 32s 533us/step - loss: 0.0689 - acc: 0.9789 - val_loss: 0.0408 - val_acc: 0.9893\n",
      "Epoch 4/10\n",
      "60000/60000 [==============================] - 32s 533us/step - loss: 0.0622 - acc: 0.9808 - val_loss: 0.0378 - val_acc: 0.9907\n",
      "Epoch 5/10\n",
      "60000/60000 [==============================] - 32s 537us/step - loss: 0.0531 - acc: 0.9837 - val_loss: 0.0333 - val_acc: 0.9915\n",
      "Epoch 6/10\n",
      "60000/60000 [==============================] - 33s 554us/step - loss: 0.0495 - acc: 0.9854 - val_loss: 0.0339 - val_acc: 0.9909\n",
      "Epoch 7/10\n",
      "60000/60000 [==============================] - 39s 649us/step - loss: 0.0475 - acc: 0.9851 - val_loss: 0.0330 - val_acc: 0.9929\n",
      "Epoch 8/10\n",
      "60000/60000 [==============================] - 39s 648us/step - loss: 0.0423 - acc: 0.9870 - val_loss: 0.0289 - val_acc: 0.9931\n",
      "Epoch 9/10\n",
      "60000/60000 [==============================] - 38s 633us/step - loss: 0.0421 - acc: 0.9874 - val_loss: 0.0290 - val_acc: 0.9928\n",
      "Epoch 10/10\n",
      "60000/60000 [==============================] - 39s 646us/step - loss: 0.0374 - acc: 0.9884 - val_loss: 0.0253 - val_acc: 0.9922\n",
      "10000/10000 [==============================] - 3s 288us/step\n",
      "\n",
      "Test loss: 0.025273755778139458\n",
      "Test accuracy: 0.9922\n"
     ]
    }
   ],
   "source": [
    "# training parameters\n",
    "batch_size = 64\n",
    "epochs = 10\n",
    "\n",
    "# create the deep conv net\n",
    "model_CNN=create_CNN()\n",
    "\n",
    "# train CNN\n",
    "model_CNN.fit(X_train, Y_train,\n",
    "          batch_size=batch_size,\n",
    "          epochs=epochs,\n",
    "          verbose=1,\n",
    "          validation_data=(X_test, Y_test))\n",
    "\n",
    "# evaliate model\n",
    "score = model_CNN.evaluate(X_test, Y_test, verbose=1)\n",
    "\n",
    "# print performance\n",
    "print()\n",
    "print('Test loss:', score[0])\n",
    "print('Test accuracy:', score[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}