image-classification-mnist-emist.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Image Classification on MNIST and EMNIST Letters\n",
    "\n",
    "#### Summary of Techniques\n",
    "- inspired by what Nelsen mentioned the paper in 2010 by Ciresan, Meier, Gambardella, and Schmidhuber they used \n",
    "only fully-connected layers (no convolutions). Here we used 2 convnet plus added more than 2 fully connected layers \n",
    "with many epoches (196 epoches) of training time.\n",
    "- 0.5 dropout in between each fully connected layers;\n",
    "- did not apply dropout to convolution layers since according to Nelsen, the convolutional layers have consideralbe\n",
    "inbuilt resistance to overfitting.\n",
    "- applied Batch normalization for each layer in between to solve the potential vanishing gradient problem and\n",
    "potentially dead ReLu problem. \n",
    "- employed early stopping, it shows the network was keep learning so it did not reach to the point of early stopping,\n",
    "so it stops when all data are consumed.\n",
    "- Used optimizer=keras.optimizers.Adadelta() rather than SGD in Keras. The advantage for Adadelta is it can \n",
    "automaticaly update learning rate adaptively of its learning rate according to loss function. \n",
    "- We tested on SGD with eta=0.03 which produced very low test accuracy.\n",
    "- Expanded training dataset using Keras ImageDataGenerator() in MNIST dataset (see iteration 12). \n",
    "It did not help with model accuracy improvement. It might help in theory with the right parameters for data augmentation.\n",
    "- The fully connected layers with dropout helps gradually reduce over-fitting (better performance than fewer layers high \n",
    "droput)\n",
    "- added save the model to local disk to load in the future to re-produce the result\n",
    "- Used Relu activation, but it might have dead ReLu problem according to Standard CNN class. It worthy to try Leaky Relu.\n",
    "- displayed random chosen mis-classified hand writing letters\n",
    "- displayed accuracy and loss in TensorBoard\n",
    "- relatively larger batch size helps with model performance.\n",
    "\n",
    "#### Summary of Accuracy on MNIST\n",
    "- training accuracy: 99.92%\n",
    "- validation accuracy: 99.60%\n",
    "- test accuracy: 99.48%\n",
    "\n",
    "#### Summary of Accuracy on EMNIST\n",
    "- training accuracy: 98.7%\n",
    "- validation accuracy: 95.1%\n",
    "- test accuracy: 94.9%"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "from __future__ import print_function\n",
    "import keras\n",
    "from keras import backend as K\n",
    "from keras.datasets import mnist\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Dropout, Flatten\n",
    "from keras.layers import Conv2D, MaxPooling2D\n",
    "from keras.layers.normalization import BatchNormalization\n",
    "#from keras.preprocessing.image import ImageDataGenerator\n",
    "\n",
    "from keras.callbacks import EarlyStopping\n",
    "from matplotlib import pyplot\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.utils import shuffle\n",
    "\n",
    "from scipy import io as sio\n",
    "import numpy as np\n",
    "import gzip\n",
    "from time import time\n",
    "from keras.callbacks import TensorBoard\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import cv2\n",
    "import random"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "channels_last\n"
     ]
    }
   ],
   "source": [
    "# different backend has different requirement to input dimensions\n",
    "# which is 4 dimensional tensor for Tensorflow with channels_last\n",
    "# (samples, rows, cols, channels) \n",
    "print(K.image_data_format()) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Warm Up"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. Try running Niesel's mnist_cnn.py and verify that you can match the accuracy of 99.25% claimed in the comments\n",
    "Summary of my running result: \n",
    "- training accuracy: 0.9922\n",
    "- test accuracy (on test dataset): 0.9912"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(60000, 28, 28) (60000,) (10000, 28, 28) (10000,)\n"
     ]
    }
   ],
   "source": [
    "batch_size = 128\n",
    "num_classes = 10\n",
    "epochs = 12\n",
    "\n",
    "# input image dimensions\n",
    "img_rows, img_cols = 28, 28\n",
    "\n",
    "# the data, split between train and test sets\n",
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
    "print(x_train.shape,y_train.shape,x_test.shape,y_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x_train shape: (60000, 28, 28, 1)\n",
      "60000 train samples\n",
      "10000 test samples\n",
      "WARNING:tensorflow:From C:\\Users\\CampusUser\\Anaconda3\\envs\\Keras\\lib\\site-packages\\tensorflow\\python\\framework\\op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Colocations handled automatically by placer.\n",
      "WARNING:tensorflow:From C:\\Users\\CampusUser\\Anaconda3\\envs\\Keras\\lib\\site-packages\\keras\\backend\\tensorflow_backend.py:3445: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.\n",
      "WARNING:tensorflow:From C:\\Users\\CampusUser\\Anaconda3\\envs\\Keras\\lib\\site-packages\\tensorflow\\python\\ops\\math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.cast instead.\n",
      "Train on 60000 samples, validate on 10000 samples\n",
      "Epoch 1/12\n",
      "60000/60000 [==============================] - 120s 2ms/step - loss: 0.2729 - acc: 0.9150 - val_loss: 0.0632 - val_acc: 0.9800\n",
      "Epoch 2/12\n",
      "60000/60000 [==============================] - 119s 2ms/step - loss: 0.0891 - acc: 0.9743 - val_loss: 0.0427 - val_acc: 0.9865\n",
      "Epoch 3/12\n",
      "60000/60000 [==============================] - 121s 2ms/step - loss: 0.0672 - acc: 0.9795 - val_loss: 0.0329 - val_acc: 0.9890\n",
      "Epoch 4/12\n",
      "60000/60000 [==============================] - 122s 2ms/step - loss: 0.0532 - acc: 0.9844 - val_loss: 0.0319 - val_acc: 0.9897\n",
      "Epoch 5/12\n",
      "60000/60000 [==============================] - 121s 2ms/step - loss: 0.0459 - acc: 0.9860 - val_loss: 0.0302 - val_acc: 0.9898\n",
      "Epoch 6/12\n",
      "60000/60000 [==============================] - 122s 2ms/step - loss: 0.0405 - acc: 0.9875 - val_loss: 0.0281 - val_acc: 0.9913\n",
      "Epoch 7/12\n",
      "60000/60000 [==============================] - 122s 2ms/step - loss: 0.0384 - acc: 0.9884 - val_loss: 0.0327 - val_acc: 0.9895\n",
      "Epoch 8/12\n",
      "60000/60000 [==============================] - 122s 2ms/step - loss: 0.0326 - acc: 0.9901 - val_loss: 0.0264 - val_acc: 0.9916\n",
      "Epoch 9/12\n",
      "60000/60000 [==============================] - 122s 2ms/step - loss: 0.0311 - acc: 0.9902 - val_loss: 0.0296 - val_acc: 0.9899\n",
      "Epoch 10/12\n",
      "60000/60000 [==============================] - 122s 2ms/step - loss: 0.0280 - acc: 0.9914 - val_loss: 0.0285 - val_acc: 0.9913\n",
      "Epoch 11/12\n",
      "60000/60000 [==============================] - 122s 2ms/step - loss: 0.0268 - acc: 0.9916 - val_loss: 0.0276 - val_acc: 0.9908\n",
      "Epoch 12/12\n",
      "60000/60000 [==============================] - 122s 2ms/step - loss: 0.0251 - acc: 0.9922 - val_loss: 0.0282 - val_acc: 0.9912\n",
      "Test loss: 0.028169668871362866\n",
      "Test accuracy: 0.9912\n"
     ]
    }
   ],
   "source": [
    "#  TensorFlow's tensor expect the input shape (samples, rows, cols, channels) \n",
    "\n",
    "if K.image_data_format() == 'channels_first':\n",
    "    x_train = x_train.reshape(x_train.shape[0], 1, img_rows, img_cols)\n",
    "    x_test = x_test.reshape(x_test.shape[0], 1, img_rows, img_cols)\n",
    "    input_shape = (1, img_rows, img_cols)\n",
    "else:\n",
    "    x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)\n",
    "    x_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)\n",
    "    input_shape = (img_rows, img_cols, 1)\n",
    "\n",
    "x_train = x_train.astype('float32')\n",
    "x_test = x_test.astype('float32')\n",
    "x_train /= 255\n",
    "x_test /= 255\n",
    "print('x_train shape:', x_train.shape)\n",
    "print(x_train.shape[0], 'train samples')\n",
    "print(x_test.shape[0], 'test samples')\n",
    "\n",
    "# convert class vectors to binary class matrices\n",
    "y_train = keras.utils.to_categorical(y_train, num_classes)\n",
    "y_test = keras.utils.to_categorical(y_test, num_classes)\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Conv2D(32, kernel_size=(3, 3),\n",
    "                 activation='relu',\n",
    "                 input_shape=input_shape))\n",
    "model.add(Conv2D(64, (3, 3), activation='relu'))\n",
    "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "model.add(Dropout(0.25))\n",
    "model.add(Flatten())\n",
    "model.add(Dense(128, activation='relu'))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(num_classes, activation='softmax'))\n",
    "\n",
    "model.compile(loss=keras.losses.categorical_crossentropy,\n",
    "              optimizer=keras.optimizers.Adadelta(),\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "model.fit(x_train, y_train,\n",
    "          batch_size=batch_size,\n",
    "          epochs=epochs,\n",
    "          verbose=1,\n",
    "          validation_data=(x_test, y_test))\n",
    "score = model.evaluate(x_test, y_test, verbose=0)\n",
    "print('Test loss:', score[0])\n",
    "print('Test accuracy:', score[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Nielsen claims an accuracy of 99.60% for the last model in Chapter 6 (see Inserting an extra fully-connected layer). Modify mnist_cnn.py to match the parameters of that model, and compare your results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### add one extra layer based on Nielsen's network3 pipeline - iteration 1\n",
    "Summary of result:\n",
    "- training accuracy: 0.9971\n",
    "- test accuracy: 0.9912\n",
    "- added extra fully connected layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(60000, 28, 28) (60000,) (10000, 28, 28) (10000,)\n"
     ]
    }
   ],
   "source": [
    "batch_size = 128\n",
    "num_classes = 10\n",
    "epochs = 12\n",
    "\n",
    "# input image dimensions\n",
    "img_rows, img_cols = 28, 28\n",
    "\n",
    "# the data, split between train and test sets\n",
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
    "print(x_train.shape,y_train.shape,x_test.shape,y_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x_train shape: (60000, 28, 28, 1)\n",
      "60000 train samples\n",
      "10000 test samples\n",
      "WARNING:tensorflow:From C:\\Users\\CampusUser\\Anaconda3\\envs\\Keras\\lib\\site-packages\\tensorflow\\python\\framework\\op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Colocations handled automatically by placer.\n",
      "WARNING:tensorflow:From C:\\Users\\CampusUser\\Anaconda3\\envs\\Keras\\lib\\site-packages\\keras\\backend\\tensorflow_backend.py:3445: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.\n",
      "WARNING:tensorflow:From C:\\Users\\CampusUser\\Anaconda3\\envs\\Keras\\lib\\site-packages\\tensorflow\\python\\ops\\math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.cast instead.\n",
      "Train on 60000 samples, validate on 10000 samples\n",
      "Epoch 1/12\n",
      "60000/60000 [==============================] - 132s 2ms/step - loss: 0.2777 - acc: 0.9147 - val_loss: 0.0568 - val_acc: 0.9819\n",
      "Epoch 2/12\n",
      "60000/60000 [==============================] - 127s 2ms/step - loss: 0.0720 - acc: 0.9796 - val_loss: 0.0488 - val_acc: 0.9846\n",
      "Epoch 3/12\n",
      "60000/60000 [==============================] - 128s 2ms/step - loss: 0.0503 - acc: 0.9859 - val_loss: 0.0368 - val_acc: 0.9889\n",
      "Epoch 4/12\n",
      "60000/60000 [==============================] - 126s 2ms/step - loss: 0.0404 - acc: 0.9881 - val_loss: 0.0311 - val_acc: 0.9902\n",
      "Epoch 5/12\n",
      "60000/60000 [==============================] - 129s 2ms/step - loss: 0.0312 - acc: 0.9910 - val_loss: 0.0279 - val_acc: 0.9910\n",
      "Epoch 6/12\n",
      "60000/60000 [==============================] - 128s 2ms/step - loss: 0.0256 - acc: 0.9922 - val_loss: 0.0302 - val_acc: 0.9906\n",
      "Epoch 7/12\n",
      "60000/60000 [==============================] - 129s 2ms/step - loss: 0.0213 - acc: 0.9940 - val_loss: 0.0289 - val_acc: 0.9908\n",
      "Epoch 8/12\n",
      "60000/60000 [==============================] - 134s 2ms/step - loss: 0.0166 - acc: 0.9948 - val_loss: 0.0313 - val_acc: 0.9912\n",
      "Epoch 9/12\n",
      "60000/60000 [==============================] - 131s 2ms/step - loss: 0.0146 - acc: 0.9953 - val_loss: 0.0327 - val_acc: 0.9910\n",
      "Epoch 10/12\n",
      "60000/60000 [==============================] - 130s 2ms/step - loss: 0.0135 - acc: 0.9961 - val_loss: 0.0359 - val_acc: 0.9914\n",
      "Epoch 11/12\n",
      "60000/60000 [==============================] - 130s 2ms/step - loss: 0.0114 - acc: 0.9963 - val_loss: 0.0386 - val_acc: 0.9907\n",
      "Epoch 12/12\n",
      "60000/60000 [==============================] - 136s 2ms/step - loss: 0.0104 - acc: 0.9971 - val_loss: 0.0343 - val_acc: 0.9912\n",
      "Test loss: 0.03430560187711494\n",
      "Test accuracy: 0.9912\n"
     ]
    }
   ],
   "source": [
    "#  TensorFlow's tensor expect the input shape (samples, rows, cols, channels) \n",
    "\n",
    "if K.image_data_format() == 'channels_first':\n",
    "    x_train = x_train.reshape(x_train.shape[0], 1, img_rows, img_cols)\n",
    "    x_test = x_test.reshape(x_test.shape[0], 1, img_rows, img_cols)\n",
    "    input_shape = (1, img_rows, img_cols)\n",
    "else:\n",
    "    x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)\n",
    "    x_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)\n",
    "    input_shape = (img_rows, img_cols, 1)\n",
    "\n",
    "x_train = x_train.astype('float32')\n",
    "x_test = x_test.astype('float32')\n",
    "x_train /= 255\n",
    "x_test /= 255\n",
    "print('x_train shape:', x_train.shape)\n",
    "print(x_train.shape[0], 'train samples')\n",
    "print(x_test.shape[0], 'test samples')\n",
    "\n",
    "# convert class vectors to binary class matrices (one hot encoding)\n",
    "y_train = keras.utils.to_categorical(y_train, num_classes)\n",
    "y_test = keras.utils.to_categorical(y_test, num_classes)\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Conv2D(32, kernel_size=(3, 3),\n",
    "                 activation='relu',\n",
    "                 input_shape=input_shape))\n",
    "model.add(Conv2D(64, (3, 3), activation='relu'))\n",
    "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "model.add(Dropout(0.25))\n",
    "model.add(Flatten())\n",
    "model.add(Dense(128, activation='relu')) \n",
    "# added one extra fully connected layer below\n",
    "model.add(Dense(128, activation='relu'))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(num_classes, activation='softmax'))\n",
    "\n",
    "model.compile(loss=keras.losses.categorical_crossentropy,\n",
    "              optimizer=keras.optimizers.Adadelta(),\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "model.fit(x_train, y_train,\n",
    "          batch_size=batch_size,\n",
    "          epochs=epochs,\n",
    "          verbose=1,\n",
    "          validation_data=(x_test, y_test))\n",
    "score = model.evaluate(x_test, y_test, verbose=0)\n",
    "print('Test loss:', score[0])\n",
    "print('Test accuracy:', score[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### add one extra layer based on Nielsen's network3 pipeline- iteration 2 \n",
    "Summary of Result:\n",
    "- optimizer=keras.optimizers.Adadelta(), Nieselsen used SGD for optimizer but SGD in Keras did not perform well\n",
    "- training accuracy: 0.9993\n",
    "- test accuracy: 0.9941"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(60000, 28, 28) (60000,) (10000, 28, 28) (10000,)\n"
     ]
    }
   ],
   "source": [
    "batch_size = 10\n",
    "num_classes = 10\n",
    "epochs = 40\n",
    "\n",
    "# input image dimensions\n",
    "img_rows, img_cols = 28, 28\n",
    "\n",
    "# the data, split between train and test sets\n",
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
    "print(x_train.shape,y_train.shape,x_test.shape,y_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x_train shape: (60000, 28, 28, 1)\n",
      "60000 train samples\n",
      "10000 test samples\n",
      "Train on 60000 samples, validate on 10000 samples\n",
      "Epoch 1/40\n",
      "60000/60000 [==============================] - 221s 4ms/step - loss: 0.1323 - acc: 0.9603 - val_loss: 0.0610 - val_acc: 0.9817\n",
      "Epoch 2/40\n",
      "60000/60000 [==============================] - 218s 4ms/step - loss: 0.0545 - acc: 0.9849 - val_loss: 0.0337 - val_acc: 0.9908\n",
      "Epoch 3/40\n",
      "60000/60000 [==============================] - 211s 4ms/step - loss: 0.0450 - acc: 0.9885 - val_loss: 0.0348 - val_acc: 0.9899\n",
      "Epoch 4/40\n",
      "60000/60000 [==============================] - 207s 3ms/step - loss: 0.0365 - acc: 0.9902 - val_loss: 0.0354 - val_acc: 0.9914\n",
      "Epoch 5/40\n",
      "60000/60000 [==============================] - 208s 3ms/step - loss: 0.0321 - acc: 0.9919 - val_loss: 0.0468 - val_acc: 0.9895\n",
      "Epoch 6/40\n",
      "60000/60000 [==============================] - 209s 3ms/step - loss: 0.0286 - acc: 0.9929 - val_loss: 0.0336 - val_acc: 0.9924\n",
      "Epoch 7/40\n",
      "60000/60000 [==============================] - 209s 3ms/step - loss: 0.0249 - acc: 0.9940 - val_loss: 0.0390 - val_acc: 0.9917\n",
      "Epoch 8/40\n",
      "60000/60000 [==============================] - 209s 3ms/step - loss: 0.0218 - acc: 0.9948 - val_loss: 0.0364 - val_acc: 0.9923\n",
      "Epoch 9/40\n",
      "60000/60000 [==============================] - 213s 4ms/step - loss: 0.0221 - acc: 0.9950 - val_loss: 0.0454 - val_acc: 0.9911\n",
      "Epoch 10/40\n",
      "60000/60000 [==============================] - 210s 3ms/step - loss: 0.0187 - acc: 0.9958 - val_loss: 0.0428 - val_acc: 0.9928\n",
      "Epoch 11/40\n",
      "60000/60000 [==============================] - 210s 3ms/step - loss: 0.0175 - acc: 0.9961 - val_loss: 0.0344 - val_acc: 0.9936\n",
      "Epoch 12/40\n",
      "60000/60000 [==============================] - 210s 3ms/step - loss: 0.0167 - acc: 0.9963 - val_loss: 0.0420 - val_acc: 0.9921\n",
      "Epoch 13/40\n",
      "60000/60000 [==============================] - 210s 4ms/step - loss: 0.0142 - acc: 0.9968 - val_loss: 0.0443 - val_acc: 0.9930\n",
      "Epoch 14/40\n",
      "60000/60000 [==============================] - 210s 4ms/step - loss: 0.0147 - acc: 0.9972 - val_loss: 0.0491 - val_acc: 0.9929\n",
      "Epoch 15/40\n",
      "60000/60000 [==============================] - 218s 4ms/step - loss: 0.0118 - acc: 0.9973 - val_loss: 0.0439 - val_acc: 0.9931\n",
      "Epoch 16/40\n",
      "60000/60000 [==============================] - 212s 4ms/step - loss: 0.0126 - acc: 0.9974 - val_loss: 0.0513 - val_acc: 0.9921\n",
      "Epoch 17/40\n",
      "60000/60000 [==============================] - 213s 4ms/step - loss: 0.0114 - acc: 0.9979 - val_loss: 0.0451 - val_acc: 0.9932\n",
      "Epoch 18/40\n",
      "60000/60000 [==============================] - 213s 4ms/step - loss: 0.0105 - acc: 0.9979 - val_loss: 0.0466 - val_acc: 0.9930\n",
      "Epoch 19/40\n",
      "60000/60000 [==============================] - 213s 4ms/step - loss: 0.0084 - acc: 0.9985 - val_loss: 0.0491 - val_acc: 0.9928\n",
      "Epoch 20/40\n",
      "60000/60000 [==============================] - 213s 4ms/step - loss: 0.0104 - acc: 0.9982 - val_loss: 0.0543 - val_acc: 0.9930\n",
      "Epoch 21/40\n",
      "60000/60000 [==============================] - 218s 4ms/step - loss: 0.0072 - acc: 0.9985 - val_loss: 0.0543 - val_acc: 0.9927\n",
      "Epoch 22/40\n",
      "60000/60000 [==============================] - 212s 4ms/step - loss: 0.0082 - acc: 0.9985 - val_loss: 0.0505 - val_acc: 0.9933\n",
      "Epoch 23/40\n",
      "60000/60000 [==============================] - 196s 3ms/step - loss: 0.0076 - acc: 0.9987 - val_loss: 0.0544 - val_acc: 0.9931\n",
      "Epoch 24/40\n",
      "60000/60000 [==============================] - 197s 3ms/step - loss: 0.0078 - acc: 0.9985 - val_loss: 0.0525 - val_acc: 0.9936\n",
      "Epoch 25/40\n",
      "60000/60000 [==============================] - 197s 3ms/step - loss: 0.0085 - acc: 0.9985 - val_loss: 0.0568 - val_acc: 0.9933\n",
      "Epoch 26/40\n",
      "60000/60000 [==============================] - 215s 4ms/step - loss: 0.0068 - acc: 0.9990 - val_loss: 0.0483 - val_acc: 0.9936\n",
      "Epoch 27/40\n",
      "60000/60000 [==============================] - 220s 4ms/step - loss: 0.0062 - acc: 0.9988 - val_loss: 0.0532 - val_acc: 0.9939\n",
      "Epoch 28/40\n",
      "60000/60000 [==============================] - 201s 3ms/step - loss: 0.0058 - acc: 0.9992 - val_loss: 0.0509 - val_acc: 0.9937\n",
      "Epoch 29/40\n",
      "60000/60000 [==============================] - 221s 4ms/step - loss: 0.0063 - acc: 0.9987 - val_loss: 0.0534 - val_acc: 0.9936\n",
      "Epoch 30/40\n",
      "60000/60000 [==============================] - 264s 4ms/step - loss: 0.0059 - acc: 0.9990 - val_loss: 0.0583 - val_acc: 0.9933\n",
      "Epoch 31/40\n",
      "60000/60000 [==============================] - 221s 4ms/step - loss: 0.0048 - acc: 0.9991 - val_loss: 0.0537 - val_acc: 0.9931\n",
      "Epoch 32/40\n",
      "60000/60000 [==============================] - 222s 4ms/step - loss: 0.0051 - acc: 0.9991 - val_loss: 0.0607 - val_acc: 0.9932\n",
      "Epoch 33/40\n",
      "60000/60000 [==============================] - 225s 4ms/step - loss: 0.0051 - acc: 0.9993 - val_loss: 0.0600 - val_acc: 0.9937\n",
      "Epoch 34/40\n",
      "60000/60000 [==============================] - 221s 4ms/step - loss: 0.0054 - acc: 0.9991 - val_loss: 0.0504 - val_acc: 0.9937\n",
      "Epoch 35/40\n",
      "60000/60000 [==============================] - 222s 4ms/step - loss: 0.0053 - acc: 0.9991 - val_loss: 0.0530 - val_acc: 0.9938\n",
      "Epoch 36/40\n",
      "60000/60000 [==============================] - 223s 4ms/step - loss: 0.0052 - acc: 0.9992 - val_loss: 0.0531 - val_acc: 0.9939\n",
      "Epoch 37/40\n",
      "60000/60000 [==============================] - 223s 4ms/step - loss: 0.0033 - acc: 0.9994 - val_loss: 0.0629 - val_acc: 0.9936\n",
      "Epoch 38/40\n",
      "60000/60000 [==============================] - 231s 4ms/step - loss: 0.0039 - acc: 0.9995 - val_loss: 0.0594 - val_acc: 0.9940\n",
      "Epoch 39/40\n",
      "60000/60000 [==============================] - 223s 4ms/step - loss: 0.0045 - acc: 0.9993 - val_loss: 0.0619 - val_acc: 0.9931\n",
      "Epoch 40/40\n",
      "60000/60000 [==============================] - 224s 4ms/step - loss: 0.0045 - acc: 0.9993 - val_loss: 0.0532 - val_acc: 0.9941\n",
      "Test loss: 0.053241416100556944\n",
      "Test accuracy: 0.9941\n"
     ]
    }
   ],
   "source": [
    "#  TensorFlow's tensor expect the input shape (samples, rows, cols, channels) \n",
    "\n",
    "if K.image_data_format() == 'channels_first':\n",
    "    x_train = x_train.reshape(x_train.shape[0], 1, img_rows, img_cols)\n",
    "    x_test = x_test.reshape(x_test.shape[0], 1, img_rows, img_cols)\n",
    "    input_shape = (1, img_rows, img_cols)\n",
    "else:\n",
    "    x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)\n",
    "    x_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)\n",
    "    input_shape = (img_rows, img_cols, 1)\n",
    "\n",
    "x_train = x_train.astype('float32')\n",
    "x_test = x_test.astype('float32')\n",
    "x_train /= 255\n",
    "x_test /= 255\n",
    "print('x_train shape:', x_train.shape)\n",
    "print(x_train.shape[0], 'train samples')\n",
    "print(x_test.shape[0], 'test samples')\n",
    "\n",
    "# convert class vectors to binary class matrices\n",
    "y_train = keras.utils.to_categorical(y_train, num_classes)\n",
    "y_test = keras.utils.to_categorical(y_test, num_classes)\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Conv2D(20, kernel_size=(5, 5),\n",
    "                 activation='relu',\n",
    "                 input_shape=input_shape))\n",
    "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "model.add(Conv2D(40, (5, 5), activation='relu'))\n",
    "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "#model.add(Dropout(0.25))\n",
    "model.add(Flatten())\n",
    "model.add(Dense(1000, activation='relu')) \n",
    "model.add(Dropout(0.5))\n",
    "# added one extra fully connected layer below\n",
    "model.add(Dense(1000, activation='relu'))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(num_classes, activation='softmax'))\n",
    "\n",
    "model.compile(loss=keras.losses.categorical_crossentropy,\n",
    "              optimizer=keras.optimizers.Adadelta(),\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "model.fit(x_train, y_train,\n",
    "          batch_size=batch_size,\n",
    "          epochs=epochs,\n",
    "          verbose=1,\n",
    "          validation_data=(x_test, y_test))\n",
    "score = model.evaluate(x_test, y_test, verbose=0)\n",
    "print('Test loss:', score[0])\n",
    "print('Test accuracy:', score[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### add one extra layer based on Nielsen's network3 pipeline- iteration 13\n",
    "- Summary of result:\n",
    "    - training accuracy: 99.92%\n",
    "    - validation accuracy: 99.60%\n",
    "    - test accuracy: 99.48%\n",
    "- Summary of parameters and hyper-parameters\n",
    "    - inspired by Nelsen's mentioned a paper, we used 4 fully connected layers with many epoches (196) of training time.\n",
    "     - 0.5 dropout for fully connected layers\n",
    "     - Batch normalization for each layer in between to solve the potential vanishing gradient problem.\n",
    "     - no preprocessing, as it did not help in improving the accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(49980, 28, 28) (49980,) (10020, 28, 28) (10020,) (10000, 28, 28) (10000,)\n"
     ]
    }
   ],
   "source": [
    "batch_size = 249 #784\n",
    "num_classes = 10\n",
    "epochs = 196\n",
    "\n",
    "# input image dimensions\n",
    "img_rows, img_cols = 28, 28\n",
    "\n",
    "# the data, split between train and test sets\n",
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
    "# the data, split between train and validation sets\n",
    "x_train, x_valid, y_train, y_valid = train_test_split(x_train, y_train, test_size = 0.167, random_state = 0, shuffle=True)\n",
    "# x_train, y_train = shuffle(x_train, y_train)\n",
    "print(x_train.shape,y_train.shape,x_valid.shape,y_valid.shape,x_test.shape,y_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x_train shape: (49980, 28, 28, 1)\n",
      "49980 train samples\n",
      "49980 train samples\n",
      "10000 test samples\n",
      "WARNING:tensorflow:From C:\\Users\\CampusUser\\Anaconda3\\envs\\Keras\\lib\\site-packages\\tensorflow\\python\\framework\\op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Colocations handled automatically by placer.\n",
      "WARNING:tensorflow:From C:\\Users\\CampusUser\\Anaconda3\\envs\\Keras\\lib\\site-packages\\keras\\backend\\tensorflow_backend.py:3445: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.\n",
      "WARNING:tensorflow:From C:\\Users\\CampusUser\\Anaconda3\\envs\\Keras\\lib\\site-packages\\tensorflow\\python\\ops\\math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.cast instead.\n",
      "Train on 49980 samples, validate on 10020 samples\n",
      "Epoch 1/196\n",
      "49980/49980 [==============================] - 80s 2ms/step - loss: 0.4506 - acc: 0.8595 - val_loss: 0.0840 - val_acc: 0.9778\n",
      "Epoch 2/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.1232 - acc: 0.9629 - val_loss: 0.0550 - val_acc: 0.9845\n",
      "Epoch 3/196\n",
      "49980/49980 [==============================] - 77s 2ms/step - loss: 0.0868 - acc: 0.9740 - val_loss: 0.0381 - val_acc: 0.9875\n",
      "Epoch 4/196\n",
      "49980/49980 [==============================] - 75s 2ms/step - loss: 0.0725 - acc: 0.9784 - val_loss: 0.0316 - val_acc: 0.9911\n",
      "Epoch 5/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0619 - acc: 0.9814 - val_loss: 0.0325 - val_acc: 0.9912\n",
      "Epoch 6/196\n",
      "49980/49980 [==============================] - 72s 1ms/step - loss: 0.0570 - acc: 0.9831 - val_loss: 0.0287 - val_acc: 0.9910\n",
      "Epoch 7/196\n",
      "49980/49980 [==============================] - 72s 1ms/step - loss: 0.0524 - acc: 0.9845 - val_loss: 0.0294 - val_acc: 0.9922\n",
      "Epoch 8/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0435 - acc: 0.9866 - val_loss: 0.0292 - val_acc: 0.9908\n",
      "Epoch 9/196\n",
      "49980/49980 [==============================] - 72s 1ms/step - loss: 0.0416 - acc: 0.9870 - val_loss: 0.0296 - val_acc: 0.9914\n",
      "Epoch 10/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0403 - acc: 0.9883 - val_loss: 0.0245 - val_acc: 0.9927\n",
      "Epoch 11/196\n",
      "49980/49980 [==============================] - 72s 1ms/step - loss: 0.0378 - acc: 0.9886 - val_loss: 0.0233 - val_acc: 0.9934\n",
      "Epoch 12/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0336 - acc: 0.9896 - val_loss: 0.0221 - val_acc: 0.9936\n",
      "Epoch 13/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0329 - acc: 0.9897 - val_loss: 0.0276 - val_acc: 0.9924\n",
      "Epoch 14/196\n",
      "49980/49980 [==============================] - 72s 1ms/step - loss: 0.0302 - acc: 0.9907 - val_loss: 0.0254 - val_acc: 0.9936\n",
      "Epoch 15/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0307 - acc: 0.9908 - val_loss: 0.0222 - val_acc: 0.9939\n",
      "Epoch 16/196\n",
      "49980/49980 [==============================] - 72s 1ms/step - loss: 0.0267 - acc: 0.9916 - val_loss: 0.0261 - val_acc: 0.9931\n",
      "Epoch 17/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0254 - acc: 0.9919 - val_loss: 0.0251 - val_acc: 0.9931\n",
      "Epoch 18/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0256 - acc: 0.9925 - val_loss: 0.0260 - val_acc: 0.9934\n",
      "Epoch 19/196\n",
      "49980/49980 [==============================] - 72s 1ms/step - loss: 0.0241 - acc: 0.9924 - val_loss: 0.0204 - val_acc: 0.9946\n",
      "Epoch 20/196\n",
      "49980/49980 [==============================] - 74s 1ms/step - loss: 0.0231 - acc: 0.9927 - val_loss: 0.0222 - val_acc: 0.9939\n",
      "Epoch 21/196\n",
      "49980/49980 [==============================] - 75s 2ms/step - loss: 0.0207 - acc: 0.9936 - val_loss: 0.0256 - val_acc: 0.9927\n",
      "Epoch 22/196\n",
      "49980/49980 [==============================] - 72s 1ms/step - loss: 0.0223 - acc: 0.9932 - val_loss: 0.0258 - val_acc: 0.9933\n",
      "Epoch 23/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0220 - acc: 0.9933 - val_loss: 0.0223 - val_acc: 0.9944\n",
      "Epoch 24/196\n",
      "49980/49980 [==============================] - 72s 1ms/step - loss: 0.0171 - acc: 0.9946 - val_loss: 0.0238 - val_acc: 0.9941\n",
      "Epoch 25/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0196 - acc: 0.9940 - val_loss: 0.0252 - val_acc: 0.9935\n",
      "Epoch 26/196\n",
      "49980/49980 [==============================] - 72s 1ms/step - loss: 0.0186 - acc: 0.9943 - val_loss: 0.0234 - val_acc: 0.9941\n",
      "Epoch 27/196\n",
      "49980/49980 [==============================] - 72s 1ms/step - loss: 0.0186 - acc: 0.9942 - val_loss: 0.0228 - val_acc: 0.9935\n",
      "Epoch 28/196\n",
      "49980/49980 [==============================] - 72s 1ms/step - loss: 0.0180 - acc: 0.9944 - val_loss: 0.0239 - val_acc: 0.9944\n",
      "Epoch 29/196\n",
      "49980/49980 [==============================] - 72s 1ms/step - loss: 0.0169 - acc: 0.9942 - val_loss: 0.0236 - val_acc: 0.9943\n",
      "Epoch 30/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0163 - acc: 0.9948 - val_loss: 0.0243 - val_acc: 0.9940\n",
      "Epoch 31/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0160 - acc: 0.9949 - val_loss: 0.0253 - val_acc: 0.9939\n",
      "Epoch 32/196\n",
      "49980/49980 [==============================] - 72s 1ms/step - loss: 0.0150 - acc: 0.9954 - val_loss: 0.0263 - val_acc: 0.9937\n",
      "Epoch 33/196\n",
      "49980/49980 [==============================] - 72s 1ms/step - loss: 0.0146 - acc: 0.9954 - val_loss: 0.0235 - val_acc: 0.9946\n",
      "Epoch 34/196\n",
      "49980/49980 [==============================] - 72s 1ms/step - loss: 0.0149 - acc: 0.9954 - val_loss: 0.0260 - val_acc: 0.9938\n",
      "Epoch 35/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0154 - acc: 0.9955 - val_loss: 0.0226 - val_acc: 0.9951\n",
      "Epoch 36/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0149 - acc: 0.9951 - val_loss: 0.0230 - val_acc: 0.9943\n",
      "Epoch 37/196\n",
      "49980/49980 [==============================] - 72s 1ms/step - loss: 0.0138 - acc: 0.9955 - val_loss: 0.0255 - val_acc: 0.9935\n",
      "Epoch 38/196\n",
      "49980/49980 [==============================] - 76s 2ms/step - loss: 0.0127 - acc: 0.9962 - val_loss: 0.0248 - val_acc: 0.9946\n",
      "Epoch 39/196\n",
      "49980/49980 [==============================] - 77s 2ms/step - loss: 0.0126 - acc: 0.9961 - val_loss: 0.0220 - val_acc: 0.9944\n",
      "Epoch 40/196\n",
      "49980/49980 [==============================] - 76s 2ms/step - loss: 0.0122 - acc: 0.9963 - val_loss: 0.0318 - val_acc: 0.9928\n",
      "Epoch 41/196\n",
      "49980/49980 [==============================] - 77s 2ms/step - loss: 0.0120 - acc: 0.9959 - val_loss: 0.0249 - val_acc: 0.9937\n",
      "Epoch 42/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0129 - acc: 0.9957 - val_loss: 0.0241 - val_acc: 0.9942\n",
      "Epoch 43/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0113 - acc: 0.9961 - val_loss: 0.0255 - val_acc: 0.9947\n",
      "Epoch 44/196\n",
      "49980/49980 [==============================] - 75s 2ms/step - loss: 0.0121 - acc: 0.9962 - val_loss: 0.0421 - val_acc: 0.9902\n",
      "Epoch 45/196\n",
      "49980/49980 [==============================] - 78s 2ms/step - loss: 0.0109 - acc: 0.9966 - val_loss: 0.0250 - val_acc: 0.9947\n",
      "Epoch 46/196\n",
      "49980/49980 [==============================] - 78s 2ms/step - loss: 0.0118 - acc: 0.9965 - val_loss: 0.0247 - val_acc: 0.9945\n",
      "Epoch 47/196\n",
      "49980/49980 [==============================] - 79s 2ms/step - loss: 0.0118 - acc: 0.9963 - val_loss: 0.0263 - val_acc: 0.9940\n",
      "Epoch 48/196\n",
      "49980/49980 [==============================] - 78s 2ms/step - loss: 0.0106 - acc: 0.9965 - val_loss: 0.0271 - val_acc: 0.9938\n",
      "Epoch 49/196\n",
      "49980/49980 [==============================] - 77s 2ms/step - loss: 0.0095 - acc: 0.9971 - val_loss: 0.0243 - val_acc: 0.9946\n",
      "Epoch 50/196\n",
      "49980/49980 [==============================] - 78s 2ms/step - loss: 0.0111 - acc: 0.9964 - val_loss: 0.0240 - val_acc: 0.9944\n",
      "Epoch 51/196\n",
      "49980/49980 [==============================] - 78s 2ms/step - loss: 0.0100 - acc: 0.9966 - val_loss: 0.0271 - val_acc: 0.9940\n",
      "Epoch 52/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0103 - acc: 0.9967 - val_loss: 0.0272 - val_acc: 0.9943\n",
      "Epoch 53/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0087 - acc: 0.9974 - val_loss: 0.0240 - val_acc: 0.9946\n",
      "Epoch 54/196\n",
      "49980/49980 [==============================] - 77s 2ms/step - loss: 0.0084 - acc: 0.9975 - val_loss: 0.0257 - val_acc: 0.9944\n",
      "Epoch 55/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0098 - acc: 0.9969 - val_loss: 0.0239 - val_acc: 0.9950\n",
      "Epoch 56/196\n",
      "49980/49980 [==============================] - 72s 1ms/step - loss: 0.0085 - acc: 0.9972 - val_loss: 0.0242 - val_acc: 0.9945\n",
      "Epoch 57/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0101 - acc: 0.9971 - val_loss: 0.0240 - val_acc: 0.9951\n",
      "Epoch 58/196\n",
      "49980/49980 [==============================] - 72s 1ms/step - loss: 0.0073 - acc: 0.9976 - val_loss: 0.0251 - val_acc: 0.9948\n",
      "Epoch 59/196\n",
      "49980/49980 [==============================] - 74s 1ms/step - loss: 0.0084 - acc: 0.9975 - val_loss: 0.0246 - val_acc: 0.9950\n",
      "Epoch 60/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0078 - acc: 0.9975 - val_loss: 0.0238 - val_acc: 0.9950\n",
      "Epoch 61/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0077 - acc: 0.9975 - val_loss: 0.0244 - val_acc: 0.9953\n",
      "Epoch 62/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0092 - acc: 0.9971 - val_loss: 0.0238 - val_acc: 0.9952\n",
      "Epoch 63/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0076 - acc: 0.9976 - val_loss: 0.0231 - val_acc: 0.9952\n",
      "Epoch 64/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0081 - acc: 0.9974 - val_loss: 0.0248 - val_acc: 0.9946\n",
      "Epoch 65/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0072 - acc: 0.9977 - val_loss: 0.0242 - val_acc: 0.9952\n",
      "Epoch 66/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0078 - acc: 0.9976 - val_loss: 0.0238 - val_acc: 0.9949\n",
      "Epoch 67/196\n",
      "49980/49980 [==============================] - 74s 1ms/step - loss: 0.0080 - acc: 0.9974 - val_loss: 0.0229 - val_acc: 0.9950\n",
      "Epoch 68/196\n",
      "49980/49980 [==============================] - 74s 1ms/step - loss: 0.0052 - acc: 0.9981 - val_loss: 0.0238 - val_acc: 0.9953\n",
      "Epoch 69/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0086 - acc: 0.9973 - val_loss: 0.0228 - val_acc: 0.9951\n",
      "Epoch 70/196\n",
      "49980/49980 [==============================] - 74s 1ms/step - loss: 0.0077 - acc: 0.9975 - val_loss: 0.0247 - val_acc: 0.9955\n",
      "Epoch 71/196\n",
      "49980/49980 [==============================] - 76s 2ms/step - loss: 0.0065 - acc: 0.9979 - val_loss: 0.0267 - val_acc: 0.9952\n",
      "Epoch 72/196\n",
      "49980/49980 [==============================] - 74s 1ms/step - loss: 0.0078 - acc: 0.9978 - val_loss: 0.0263 - val_acc: 0.9949\n",
      "Epoch 73/196\n",
      "49980/49980 [==============================] - 74s 1ms/step - loss: 0.0073 - acc: 0.9976 - val_loss: 0.0255 - val_acc: 0.9951\n",
      "Epoch 74/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0062 - acc: 0.9980 - val_loss: 0.0275 - val_acc: 0.9952\n",
      "Epoch 75/196\n",
      "49980/49980 [==============================] - 74s 1ms/step - loss: 0.0070 - acc: 0.9980 - val_loss: 0.0253 - val_acc: 0.9951\n",
      "Epoch 76/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0070 - acc: 0.9979 - val_loss: 0.0242 - val_acc: 0.9951\n",
      "Epoch 77/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0066 - acc: 0.9978 - val_loss: 0.0253 - val_acc: 0.9947\n",
      "Epoch 78/196\n",
      "49980/49980 [==============================] - 76s 2ms/step - loss: 0.0063 - acc: 0.9983 - val_loss: 0.0247 - val_acc: 0.9953\n",
      "Epoch 79/196\n",
      "49980/49980 [==============================] - 72s 1ms/step - loss: 0.0072 - acc: 0.9978 - val_loss: 0.0252 - val_acc: 0.9954\n",
      "Epoch 80/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0065 - acc: 0.9979 - val_loss: 0.0243 - val_acc: 0.9950\n",
      "Epoch 81/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0064 - acc: 0.9978 - val_loss: 0.0251 - val_acc: 0.9953\n",
      "Epoch 82/196\n",
      "49980/49980 [==============================] - 72s 1ms/step - loss: 0.0078 - acc: 0.9975 - val_loss: 0.0280 - val_acc: 0.9940\n",
      "Epoch 83/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0055 - acc: 0.9981 - val_loss: 0.0259 - val_acc: 0.9950\n",
      "Epoch 84/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0063 - acc: 0.9979 - val_loss: 0.0268 - val_acc: 0.9945\n",
      "Epoch 85/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0061 - acc: 0.9981 - val_loss: 0.0259 - val_acc: 0.9950\n",
      "Epoch 86/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0060 - acc: 0.9980 - val_loss: 0.0263 - val_acc: 0.9950\n",
      "Epoch 87/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0065 - acc: 0.9978 - val_loss: 0.0268 - val_acc: 0.9947\n",
      "Epoch 88/196\n",
      "49980/49980 [==============================] - 78s 2ms/step - loss: 0.0051 - acc: 0.9984 - val_loss: 0.0222 - val_acc: 0.9954\n",
      "Epoch 89/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0047 - acc: 0.9984 - val_loss: 0.0240 - val_acc: 0.9957\n",
      "Epoch 90/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0055 - acc: 0.9981 - val_loss: 0.0255 - val_acc: 0.9953\n",
      "Epoch 91/196\n",
      "49980/49980 [==============================] - 74s 1ms/step - loss: 0.0053 - acc: 0.9984 - val_loss: 0.0233 - val_acc: 0.9949\n",
      "Epoch 92/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0049 - acc: 0.9985 - val_loss: 0.0250 - val_acc: 0.9954\n",
      "Epoch 93/196\n",
      "49980/49980 [==============================] - 74s 1ms/step - loss: 0.0053 - acc: 0.9983 - val_loss: 0.0257 - val_acc: 0.9948\n",
      "Epoch 94/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0058 - acc: 0.9983 - val_loss: 0.0260 - val_acc: 0.9950\n",
      "Epoch 95/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0057 - acc: 0.9984 - val_loss: 0.0237 - val_acc: 0.9951\n",
      "Epoch 96/196\n",
      "49980/49980 [==============================] - 74s 1ms/step - loss: 0.0058 - acc: 0.9983 - val_loss: 0.0249 - val_acc: 0.9954\n",
      "Epoch 97/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0058 - acc: 0.9980 - val_loss: 0.0255 - val_acc: 0.9950\n",
      "Epoch 98/196\n",
      "49980/49980 [==============================] - 74s 1ms/step - loss: 0.0051 - acc: 0.9984 - val_loss: 0.0271 - val_acc: 0.9954\n",
      "Epoch 99/196\n",
      "49980/49980 [==============================] - 74s 1ms/step - loss: 0.0055 - acc: 0.9982 - val_loss: 0.0249 - val_acc: 0.9950\n",
      "Epoch 100/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0050 - acc: 0.9985 - val_loss: 0.0261 - val_acc: 0.9949\n",
      "Epoch 101/196\n",
      "49980/49980 [==============================] - 74s 1ms/step - loss: 0.0054 - acc: 0.9984 - val_loss: 0.0251 - val_acc: 0.9953\n",
      "Epoch 102/196\n",
      "49980/49980 [==============================] - 74s 1ms/step - loss: 0.0057 - acc: 0.9983 - val_loss: 0.0258 - val_acc: 0.9953\n",
      "Epoch 103/196\n",
      "49980/49980 [==============================] - 74s 1ms/step - loss: 0.0047 - acc: 0.9986 - val_loss: 0.0258 - val_acc: 0.9955\n",
      "Epoch 104/196\n",
      "49980/49980 [==============================] - 74s 1ms/step - loss: 0.0049 - acc: 0.9983 - val_loss: 0.0250 - val_acc: 0.9953\n",
      "Epoch 105/196\n",
      "49980/49980 [==============================] - 78s 2ms/step - loss: 0.0052 - acc: 0.9983 - val_loss: 0.0255 - val_acc: 0.9955\n",
      "Epoch 106/196\n",
      "49980/49980 [==============================] - 75s 1ms/step - loss: 0.0047 - acc: 0.9984 - val_loss: 0.0253 - val_acc: 0.9953\n",
      "Epoch 107/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0047 - acc: 0.9986 - val_loss: 0.0254 - val_acc: 0.9947\n",
      "Epoch 108/196\n",
      "49980/49980 [==============================] - 72s 1ms/step - loss: 0.0045 - acc: 0.9984 - val_loss: 0.0272 - val_acc: 0.9954\n",
      "Epoch 109/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0048 - acc: 0.9986 - val_loss: 0.0256 - val_acc: 0.9951\n",
      "Epoch 110/196\n",
      "49980/49980 [==============================] - 72s 1ms/step - loss: 0.0041 - acc: 0.9988 - val_loss: 0.0266 - val_acc: 0.9952\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 111/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0046 - acc: 0.9985 - val_loss: 0.0254 - val_acc: 0.9956\n",
      "Epoch 112/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0044 - acc: 0.9987 - val_loss: 0.0275 - val_acc: 0.9955\n",
      "Epoch 113/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0043 - acc: 0.9986 - val_loss: 0.0259 - val_acc: 0.9957\n",
      "Epoch 114/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0041 - acc: 0.9985 - val_loss: 0.0263 - val_acc: 0.9953\n",
      "Epoch 115/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0042 - acc: 0.9987 - val_loss: 0.0260 - val_acc: 0.9955\n",
      "Epoch 116/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0056 - acc: 0.9982 - val_loss: 0.0263 - val_acc: 0.9955\n",
      "Epoch 117/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0044 - acc: 0.9986 - val_loss: 0.0249 - val_acc: 0.9958\n",
      "Epoch 118/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0045 - acc: 0.9985 - val_loss: 0.0236 - val_acc: 0.9953\n",
      "Epoch 119/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0042 - acc: 0.9987 - val_loss: 0.0239 - val_acc: 0.9955\n",
      "Epoch 120/196\n",
      "49980/49980 [==============================] - 72s 1ms/step - loss: 0.0039 - acc: 0.9987 - val_loss: 0.0254 - val_acc: 0.9949\n",
      "Epoch 121/196\n",
      "49980/49980 [==============================] - 74s 1ms/step - loss: 0.0044 - acc: 0.9987 - val_loss: 0.0267 - val_acc: 0.9953\n",
      "Epoch 122/196\n",
      "49980/49980 [==============================] - 80s 2ms/step - loss: 0.0052 - acc: 0.9985 - val_loss: 0.0246 - val_acc: 0.9948\n",
      "Epoch 123/196\n",
      "49980/49980 [==============================] - 79s 2ms/step - loss: 0.0037 - acc: 0.9986 - val_loss: 0.0258 - val_acc: 0.9952\n",
      "Epoch 124/196\n",
      "49980/49980 [==============================] - 80s 2ms/step - loss: 0.0043 - acc: 0.9988 - val_loss: 0.0274 - val_acc: 0.9949\n",
      "Epoch 125/196\n",
      "49980/49980 [==============================] - 76s 2ms/step - loss: 0.0039 - acc: 0.9989 - val_loss: 0.0273 - val_acc: 0.9951\n",
      "Epoch 126/196\n",
      "49980/49980 [==============================] - 72s 1ms/step - loss: 0.0039 - acc: 0.9987 - val_loss: 0.0276 - val_acc: 0.9948\n",
      "Epoch 127/196\n",
      "49980/49980 [==============================] - 75s 1ms/step - loss: 0.0041 - acc: 0.9986 - val_loss: 0.0264 - val_acc: 0.9952\n",
      "Epoch 128/196\n",
      "49980/49980 [==============================] - 77s 2ms/step - loss: 0.0032 - acc: 0.9990 - val_loss: 0.0256 - val_acc: 0.9957\n",
      "Epoch 129/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0036 - acc: 0.9988 - val_loss: 0.0276 - val_acc: 0.9949\n",
      "Epoch 130/196\n",
      "49980/49980 [==============================] - 75s 1ms/step - loss: 0.0040 - acc: 0.9986 - val_loss: 0.0268 - val_acc: 0.9950\n",
      "Epoch 131/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0037 - acc: 0.9987 - val_loss: 0.0256 - val_acc: 0.9955\n",
      "Epoch 132/196\n",
      "49980/49980 [==============================] - 74s 1ms/step - loss: 0.0045 - acc: 0.9985 - val_loss: 0.0265 - val_acc: 0.9957\n",
      "Epoch 133/196\n",
      "49980/49980 [==============================] - 74s 1ms/step - loss: 0.0032 - acc: 0.9990 - val_loss: 0.0253 - val_acc: 0.9958\n",
      "Epoch 134/196\n",
      "49980/49980 [==============================] - 79s 2ms/step - loss: 0.0043 - acc: 0.9986 - val_loss: 0.0292 - val_acc: 0.9948\n",
      "Epoch 135/196\n",
      "49980/49980 [==============================] - 77s 2ms/step - loss: 0.0046 - acc: 0.9986 - val_loss: 0.0251 - val_acc: 0.9955\n",
      "Epoch 136/196\n",
      "49980/49980 [==============================] - 79s 2ms/step - loss: 0.0039 - acc: 0.9989 - val_loss: 0.0289 - val_acc: 0.9947\n",
      "Epoch 137/196\n",
      "49980/49980 [==============================] - 75s 2ms/step - loss: 0.0030 - acc: 0.9990 - val_loss: 0.0264 - val_acc: 0.9955\n",
      "Epoch 138/196\n",
      "49980/49980 [==============================] - 80s 2ms/step - loss: 0.0037 - acc: 0.9988 - val_loss: 0.0256 - val_acc: 0.9952\n",
      "Epoch 139/196\n",
      "49980/49980 [==============================] - 76s 2ms/step - loss: 0.0030 - acc: 0.9990 - val_loss: 0.0275 - val_acc: 0.9951\n",
      "Epoch 140/196\n",
      "49980/49980 [==============================] - 75s 2ms/step - loss: 0.0031 - acc: 0.9991 - val_loss: 0.0252 - val_acc: 0.9956\n",
      "Epoch 141/196\n",
      "49980/49980 [==============================] - 76s 2ms/step - loss: 0.0034 - acc: 0.9991 - val_loss: 0.0271 - val_acc: 0.9948\n",
      "Epoch 142/196\n",
      "49980/49980 [==============================] - 77s 2ms/step - loss: 0.0041 - acc: 0.9987 - val_loss: 0.0270 - val_acc: 0.9949\n",
      "Epoch 143/196\n",
      "49980/49980 [==============================] - 77s 2ms/step - loss: 0.0040 - acc: 0.9988 - val_loss: 0.0266 - val_acc: 0.9959\n",
      "Epoch 144/196\n",
      "49980/49980 [==============================] - 75s 1ms/step - loss: 0.0033 - acc: 0.9990 - val_loss: 0.0276 - val_acc: 0.9953\n",
      "Epoch 145/196\n",
      "49980/49980 [==============================] - 76s 2ms/step - loss: 0.0032 - acc: 0.9990 - val_loss: 0.0270 - val_acc: 0.9954\n",
      "Epoch 146/196\n",
      "49980/49980 [==============================] - 74s 1ms/step - loss: 0.0030 - acc: 0.9989 - val_loss: 0.0253 - val_acc: 0.9953\n",
      "Epoch 147/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0034 - acc: 0.9989 - val_loss: 0.0240 - val_acc: 0.9957\n",
      "Epoch 148/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0030 - acc: 0.9989 - val_loss: 0.0256 - val_acc: 0.9956\n",
      "Epoch 149/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0036 - acc: 0.9988 - val_loss: 0.0269 - val_acc: 0.9951\n",
      "Epoch 150/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0034 - acc: 0.9989 - val_loss: 0.0253 - val_acc: 0.9957\n",
      "Epoch 151/196\n",
      "49980/49980 [==============================] - 74s 1ms/step - loss: 0.0038 - acc: 0.9989 - val_loss: 0.0287 - val_acc: 0.9950\n",
      "Epoch 152/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0030 - acc: 0.9991 - val_loss: 0.0267 - val_acc: 0.9956\n",
      "Epoch 153/196\n",
      "49980/49980 [==============================] - 72s 1ms/step - loss: 0.0036 - acc: 0.9988 - val_loss: 0.0256 - val_acc: 0.9956\n",
      "Epoch 154/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0034 - acc: 0.9989 - val_loss: 0.0272 - val_acc: 0.9958\n",
      "Epoch 155/196\n",
      "49980/49980 [==============================] - 76s 2ms/step - loss: 0.0034 - acc: 0.9988 - val_loss: 0.0253 - val_acc: 0.9955\n",
      "Epoch 156/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0032 - acc: 0.9990 - val_loss: 0.0263 - val_acc: 0.9951\n",
      "Epoch 157/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0038 - acc: 0.9988 - val_loss: 0.0239 - val_acc: 0.9953\n",
      "Epoch 158/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0039 - acc: 0.9986 - val_loss: 0.0248 - val_acc: 0.9959\n",
      "Epoch 159/196\n",
      "49980/49980 [==============================] - 78s 2ms/step - loss: 0.0037 - acc: 0.9990 - val_loss: 0.0245 - val_acc: 0.9957\n",
      "Epoch 160/196\n",
      "49980/49980 [==============================] - 77s 2ms/step - loss: 0.0029 - acc: 0.9991 - val_loss: 0.0261 - val_acc: 0.9952\n",
      "Epoch 161/196\n",
      "49980/49980 [==============================] - 77s 2ms/step - loss: 0.0024 - acc: 0.9993 - val_loss: 0.0263 - val_acc: 0.9958\n",
      "Epoch 162/196\n",
      "49980/49980 [==============================] - 77s 2ms/step - loss: 0.0024 - acc: 0.9992 - val_loss: 0.0268 - val_acc: 0.9958\n",
      "Epoch 163/196\n",
      "49980/49980 [==============================] - 76s 2ms/step - loss: 0.0036 - acc: 0.9989 - val_loss: 0.0275 - val_acc: 0.9955\n",
      "Epoch 164/196\n",
      "49980/49980 [==============================] - 77s 2ms/step - loss: 0.0025 - acc: 0.9992 - val_loss: 0.0279 - val_acc: 0.9956\n",
      "Epoch 165/196\n",
      "49980/49980 [==============================] - 74s 1ms/step - loss: 0.0026 - acc: 0.9993 - val_loss: 0.0286 - val_acc: 0.9956\n",
      "Epoch 166/196\n",
      "49980/49980 [==============================] - 74s 1ms/step - loss: 0.0029 - acc: 0.9990 - val_loss: 0.0277 - val_acc: 0.9958\n",
      "Epoch 167/196\n",
      "49980/49980 [==============================] - 74s 1ms/step - loss: 0.0035 - acc: 0.9990 - val_loss: 0.0270 - val_acc: 0.9954\n",
      "Epoch 168/196\n",
      "49980/49980 [==============================] - 74s 1ms/step - loss: 0.0025 - acc: 0.9992 - val_loss: 0.0282 - val_acc: 0.9955\n",
      "Epoch 169/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0029 - acc: 0.9991 - val_loss: 0.0277 - val_acc: 0.9955\n",
      "Epoch 170/196\n",
      "49980/49980 [==============================] - 74s 1ms/step - loss: 0.0024 - acc: 0.9992 - val_loss: 0.0286 - val_acc: 0.9952\n",
      "Epoch 171/196\n",
      "49980/49980 [==============================] - 75s 2ms/step - loss: 0.0030 - acc: 0.9991 - val_loss: 0.0286 - val_acc: 0.9954\n",
      "Epoch 172/196\n",
      "49980/49980 [==============================] - 75s 1ms/step - loss: 0.0027 - acc: 0.9992 - val_loss: 0.0276 - val_acc: 0.9955\n",
      "Epoch 173/196\n",
      "49980/49980 [==============================] - 74s 1ms/step - loss: 0.0035 - acc: 0.9989 - val_loss: 0.0278 - val_acc: 0.9957\n",
      "Epoch 174/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0025 - acc: 0.9992 - val_loss: 0.0257 - val_acc: 0.9963\n",
      "Epoch 175/196\n",
      "49980/49980 [==============================] - 74s 1ms/step - loss: 0.0029 - acc: 0.9990 - val_loss: 0.0247 - val_acc: 0.9957\n",
      "Epoch 176/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0028 - acc: 0.9991 - val_loss: 0.0267 - val_acc: 0.9956\n",
      "Epoch 177/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0028 - acc: 0.9991 - val_loss: 0.0277 - val_acc: 0.9958\n",
      "Epoch 178/196\n",
      "49980/49980 [==============================] - 74s 1ms/step - loss: 0.0027 - acc: 0.9991 - val_loss: 0.0254 - val_acc: 0.9960\n",
      "Epoch 179/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0025 - acc: 0.9993 - val_loss: 0.0250 - val_acc: 0.9959\n",
      "Epoch 180/196\n",
      "49980/49980 [==============================] - 74s 1ms/step - loss: 0.0024 - acc: 0.9992 - val_loss: 0.0270 - val_acc: 0.9954\n",
      "Epoch 181/196\n",
      "49980/49980 [==============================] - 74s 1ms/step - loss: 0.0024 - acc: 0.9992 - val_loss: 0.0265 - val_acc: 0.9957\n",
      "Epoch 182/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0027 - acc: 0.9991 - val_loss: 0.0251 - val_acc: 0.9958\n",
      "Epoch 183/196\n",
      "49980/49980 [==============================] - 74s 1ms/step - loss: 0.0024 - acc: 0.9992 - val_loss: 0.0274 - val_acc: 0.9959\n",
      "Epoch 184/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0030 - acc: 0.9989 - val_loss: 0.0286 - val_acc: 0.9959\n",
      "Epoch 185/196\n",
      "49980/49980 [==============================] - 74s 1ms/step - loss: 0.0026 - acc: 0.9991 - val_loss: 0.0284 - val_acc: 0.9958\n",
      "Epoch 186/196\n",
      "49980/49980 [==============================] - 74s 1ms/step - loss: 0.0022 - acc: 0.9995 - val_loss: 0.0282 - val_acc: 0.9963\n",
      "Epoch 187/196\n",
      "49980/49980 [==============================] - 75s 1ms/step - loss: 0.0029 - acc: 0.9992 - val_loss: 0.0275 - val_acc: 0.9960\n",
      "Epoch 188/196\n",
      "49980/49980 [==============================] - 81s 2ms/step - loss: 0.0025 - acc: 0.9992 - val_loss: 0.0295 - val_acc: 0.9958\n",
      "Epoch 189/196\n",
      "49980/49980 [==============================] - 74s 1ms/step - loss: 0.0020 - acc: 0.9994 - val_loss: 0.0307 - val_acc: 0.9953\n",
      "Epoch 190/196\n",
      "49980/49980 [==============================] - 74s 1ms/step - loss: 0.0025 - acc: 0.9994 - val_loss: 0.0296 - val_acc: 0.9959\n",
      "Epoch 191/196\n",
      "49980/49980 [==============================] - 74s 1ms/step - loss: 0.0026 - acc: 0.9991 - val_loss: 0.0281 - val_acc: 0.9961\n",
      "Epoch 192/196\n",
      "49980/49980 [==============================] - 73s 1ms/step - loss: 0.0027 - acc: 0.9990 - val_loss: 0.0279 - val_acc: 0.9958\n",
      "Epoch 193/196\n",
      "49980/49980 [==============================] - 75s 2ms/step - loss: 0.0023 - acc: 0.9992 - val_loss: 0.0286 - val_acc: 0.9956\n",
      "Epoch 194/196\n",
      "49980/49980 [==============================] - 76s 2ms/step - loss: 0.0023 - acc: 0.9992 - val_loss: 0.0263 - val_acc: 0.9956\n",
      "Epoch 195/196\n",
      "49980/49980 [==============================] - 74s 1ms/step - loss: 0.0029 - acc: 0.9991 - val_loss: 0.0272 - val_acc: 0.9960\n",
      "Epoch 196/196\n",
      "49980/49980 [==============================] - 76s 2ms/step - loss: 0.0026 - acc: 0.9992 - val_loss: 0.0291 - val_acc: 0.9960\n",
      "Test loss: 2.7507066126763036 %\n",
      "Test accuracy: 99.48 %\n",
      "saved model to disk\n"
     ]
    }
   ],
   "source": [
    "# TensorFlow's tensor expect the input shape (samples, rows, cols, channels) \n",
    "if K.image_data_format() == 'channels_first':\n",
    "    x_train = x_train.reshape(x_train.shape[0], 1, img_rows, img_cols)\n",
    "    x_valid = x_valid.reshape(x_valid.shape[0], 1, img_rows, img_cols)\n",
    "    x_test = x_test.reshape(x_test.shape[0], 1, img_rows, img_cols)\n",
    "    input_shape = (1, img_rows, img_cols)\n",
    "else:\n",
    "    x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)\n",
    "    x_valid = x_valid.reshape(x_valid.shape[0], img_rows, img_cols, 1)\n",
    "    x_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)\n",
    "    input_shape = (img_rows, img_cols, 1)\n",
    "\n",
    "# normalize the input data for reducing over-fitting\n",
    "x_train = x_train.astype('float32')\n",
    "x_valid = x_valid.astype('float32')\n",
    "x_test = x_test.astype('float32')\n",
    "x_train /= 255\n",
    "x_valid /= 255\n",
    "x_test /= 255\n",
    "print('x_train shape:', x_train.shape)\n",
    "print(x_train.shape[0], 'train samples')\n",
    "print(x_train.shape[0], 'train samples')\n",
    "print(x_test.shape[0], 'test samples')\n",
    "\n",
    "\n",
    "# convert class vectors to ont hot encoding\n",
    "y_train = keras.utils.to_categorical(y_train, num_classes,dtype='float32')\n",
    "y_valid = keras.utils.to_categorical(y_valid, num_classes,dtype='float32')\n",
    "y_test = keras.utils.to_categorical(y_test, num_classes,dtype='float32')\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Conv2D(32, kernel_size=(3, 3),\n",
    "                 activation='relu',\n",
    "                 input_shape=input_shape))\n",
    "model.add(BatchNormalization(axis=-1))\n",
    "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "model.add(Conv2D(64, (3, 3), activation='relu'))\n",
    "model.add(BatchNormalization(axis=-1))\n",
    "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Flatten())\n",
    "# model.add(Dense(128, activation='relu'))\n",
    "model.add(Dense(1568, activation='relu'))\n",
    "model.add(BatchNormalization(axis=-1))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(1000, activation='relu'))\n",
    "model.add(BatchNormalization(axis=-1))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(700, activation='relu'))\n",
    "model.add(BatchNormalization(axis=-1))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(300, activation='relu'))\n",
    "model.add(BatchNormalization(axis=-1))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(100, activation='relu'))\n",
    "model.add(BatchNormalization(axis=-1))\n",
    "model.add(Dense(num_classes, activation='softmax'))\n",
    "\n",
    "model.compile(loss=keras.losses.categorical_crossentropy,\n",
    "              optimizer=keras.optimizers.Adadelta(),\n",
    "              metrics=['accuracy'])\n",
    "tensorboard = TensorBoard(log_dir=\"logs_mnist_13/{}\".format(time()))\n",
    "# simple early stopping\n",
    "es = EarlyStopping(monitor='val_acc', mode='max', verbose=1)\n",
    "history= model.fit(x_train, y_train,\n",
    "          batch_size=batch_size,\n",
    "          epochs=epochs,\n",
    "          verbose=1,\n",
    "          callbacks=[tensorboard],\n",
    "          validation_data=(x_valid, y_valid))\n",
    "\n",
    "score = model.evaluate(x_test, y_test, verbose=0)\n",
    "print('Test loss:', score[0]*100,'%')\n",
    "print('Test accuracy:', score[1]*100,'%')\n",
    "model.save('mnist_v13.h5')\n",
    "print('saved model to disk')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train: 1.000, Test: 0.996\n",
      "Test: 0.995\n"
     ]
    }
   ],
   "source": [
    "# evaluate the model\n",
    "_, train_acc = model.evaluate(x_train, y_train, verbose=0)\n",
    "_, valid_acc = model.evaluate(x_valid, y_valid, verbose=0)\n",
    "print('Train: %.3f, Test: %.3f' % (train_acc, valid_acc))\n",
    "\n",
    "_, test_acc = model.evaluate(x_test, y_test, verbose=0)\n",
    "print('Test: %.3f' % (test_acc))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEWCAYAAACXGLsWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XmcXFWd9/HPr7qr933J2tlZTIAQIAQQZHFBAgoCMwwIKjrPwIjM4KP4CKOiw+jjLOrMOCKKPqiICMiIIEbZhkVlTSBAIAmEkKWzdjrpfa2q3/PHuZ1UOl1dTUils3zfr1e/uuoudX91q+r87jnn3nPN3RERERlObLQDEBGRfZ+ShYiIZKVkISIiWSlZiIhIVkoWIiKSlZKFiIhkpWQhApjZT83s6yNcdpWZvT/XMYnsS5QsREQkKyULkQOImeWPdgxyYFKykP1G1PzzBTN72cw6zez/mdlYM/u9mbWb2SNmVp22/Llm9qqZtZjZ42Y2M23eMWb2QrTeXUDRoG19yMwWR+s+ZWazRxjjOWb2opm1mdlaM/vaoPmnRK/XEs2/PJpebGbfNrPVZtZqZn+Kpp1uZo1D7If3R4+/Zmb3mNntZtYGXG5m88zs6WgbG8zse2ZWkLb+EWb2sJltNbNNZvYPZjbOzLrMrDZtuePMrMnM4iN573JgU7KQ/c2FwAeAw4APA78H/gGoI3yf/x7AzA4Dfgl8FqgHFgC/NbOCqOD8DfBzoAb4VfS6ROseC9wKXAnUAj8E7jezwhHE1wl8HKgCzgE+bWYfiV53chTvf0UxzQEWR+t9CzgOeHcU0/8BUiPcJ+cB90Tb/AWQBP53tE9OAt4HXBXFUA48AvwBmAAcAjzq7huBx4GL0l73MuBOd+8fYRxyAFOykP3Nf7n7JndfB/wReNbdX3T3XuBe4Jhoub8CfufuD0eF3beAYkJhfCIQB/7D3fvd/R7g+bRt/A3wQ3d/1t2T7v4zoDdab1ju/ri7v+LuKXd/mZCwTotmXwo84u6/jLbb7O6LzSwGfAq4xt3XRdt8KnpPI/G0u/8m2ma3uy9y92fcPeHuqwjJbiCGDwEb3f3b7t7j7u3u/mw072eEBIGZ5QGXEBKqiJKF7Hc2pT3uHuJ5WfR4ArB6YIa7p4C1wMRo3jrfeRTN1WmPpwCfj5pxWsysBZgUrTcsMzvBzB6Lmm9agb8lHOETvcabQ6xWR2gGG2reSKwdFMNhZvaAmW2Mmqb+7whiALgPmGVm0wm1t1Z3f243Y5IDjJKFHKjWEwp9AMzMCAXlOmADMDGaNmBy2uO1wDfcvSrtr8TdfzmC7d4B3A9McvdK4AfAwHbWAjOGWGcL0JNhXidQkvY+8ghNWOkGDx19M7AMONTdKwjNdNliwN17gLsJNaCPoVqFpFGykAPV3cA5Zva+qIP284SmpKeAp4EE8Pdmlm9mFwDz0tb9EfC3US3BzKw06rguH8F2y4Gt7t5jZvOAj6bN+wXwfjO7KNpurZnNiWo9twLfMbMJZpZnZidFfSSvA0XR9uPAl4FsfSflQBvQYWbvAj6dNu8BYJyZfdbMCs2s3MxOSJt/G3A5cC5w+wjerxwklCzkgOTuywnt7/9FOHL/MPBhd+9z9z7gAkKhuI3Qv/HrtHUXEvotvhfNXxEtOxJXATeaWTtwAyFpDbzuGuBsQuLaSujcPjqafS3wCqHvZCvwL0DM3Vuj1/wxoVbUCex0dtQQriUkqXZC4rsrLYZ2QhPTh4GNwBvAGWnz/0zoWH8h6u8QAcB08yMRSWdm/wPc4e4/Hu1YZN+hZCEi25nZ8cDDhD6X9tGOR/YdaoYSEQDM7GeEazA+q0Qhg6lmISIiWalmISIiWR0wg47V1dX51KlTRzsMEZH9yqJFi7a4++Brd3ZxwCSLqVOnsnDhwtEOQ0Rkv2Jmq7MvpWYoEREZASULERHJSslCRESyOmD6LIbS399PY2MjPT09ox1KzhUVFdHQ0EA8rvvUiMiel7NkYWa3EsbO3+zuRw4x34D/JIyV0wVc7u4vRPM+QRgwDeDr0f0E3rbGxkbKy8uZOnUqOw8wemBxd5qbm2lsbGTatGmjHY6IHIBy2Qz1U+CsYebPBw6N/q4gDKuMmdUAXwVOIIwE+tX0W2W+HT09PdTW1h7QiQLAzKitrT0oalAiMjpylizc/UnC6JmZnAfc5sEzQJWZjQc+CDzs7lvdfRthnJrhks6wDvREMeBgeZ8iMjpGs89iIjvf4asxmpZpuojIbtna2Ufjti7KCvOZUFVMUTyPVMpZ19LNupZuDhlTRl1ZuE1IKuVs6eylpCCfWHQMVhzPoz/pbGrrYUJVMXkxo62nn5J4Hvl5Ox9zuzub23tZu7WLkoJ8qkriVJcUUBQPy3X2JSktyMPMSKWcRWu28cLqbbxrfAXT60oBeG1DGxtbe4gZHD6ugjmTqijIj9GXSLFmaycph5KCPKpLCkgknZQ71aUFOd2Ho5kshjoU9mGm7/oCZlcQmrCYPHnyUIuMupaWFu644w6uuuqqt7Xe2WefzR133EFVVVWOIpO3I5lyDIjFDHensy9Jb3+SyuL4ToVFIpnapfAYkEo5Xf1JSuJ5xGJGMuW82dRBeVE+4yuLty/X3tPPyqZO+pIpDqkvY1tXHyubOlnV3MmEqmLeNa6cZRvbSaacCVVFFObn0dzZx5rmTk45tJ7JNSUsXrsNgPxYjLe2dNLRm9j++tUlBUypLaE3kWR1cxdLN7TR1p0gFoP6skLqK4pIpZwnXm+iL5FiUk0JsxsqqSktoKWrj7EVRWxu6+WBVzYwqbqYkw+po6a0gJcbW3hm5VbWt3RTWpjP1NpSWrr6WNfSzca2Hnr6k1QVF/C+mWMYX1lEc2cfL6xpobWrj7yYMb6ymOKCPLr6EhTH8ygrzKe4IJ+Wrj46ehNUFMVp3NbFhtYeZo6voLYsFJTlRfl09CZ4a0snY8oLOWRMOZNrSnhxzTYWrd7G1q4+Wrr6t7//oniM2ROreH1z+07Tp9aWUFqYz+rmrp32F0BhfoxEykmmnLqyAmpLC1m+qZ2C/Bhjygtp6eqnN5EkmXJSGYbbK8iPkWdGd3+S6pI4U2pLWdXcuVMMmcTzjEnVJWxo7aG7P7nL/GMmV3HvVSdnfZ13IqcDCZrZVOCBDB3cPwQeH7hVpZktB04f+HP3K4daLpO5c+f64Cu4ly5dysyZM9/x+3gnVq1axYc+9CGWLFmy0/RkMkleXt4e3da+8H73BHffqVntzyu2sGj1Nq44dTpF8bDP1m7torW7n8riOBOqimnv6efpN5tp701QkBdjTEUhpQX5rGvp5rcvraeurJBzZo+nuz+JAROripleX8amth7ufG4N5UVxqksLWLWlk5Q7tWWFzJlUxRPLN/OLZ9fQ3NlHZXGcOZOqWLaxjU1tvQDkxYza0gLieTFau/vp6E1QX17I4WPLOWRMGS+s2caW9l5qygpYvaWL9qgQKivMJ5ny7T/8sRWFVBTFae3uZ3N7727vOzOoKo6zbQQF0IDC/BhVJXGSKdja2bu9sJtSW0J1SQFvbemktXvX15tcU8KWjl66+nYUXoeMKWNKTQntvQlWbemktqyQiVVFjKssoqQgn7Vbu3hs+WZ6+lMU5sc4elIV4yuL6EukWN/STW8iRUlBHj39KTp6E3T1JagsjlNWFKe9u5+xFUWMryzitQ1ttPckyM8z2nsSFOXHmF5fxub2Ht7a0kl/0ikvzOekGbWMqShkSk0pk2tL6OxN8NLaFhavbeGwseUcM7ma8VVFvLa+jdc2tNHZm6ChuphDx5TTm0hu3xdbO/soyIsxtrKIZ1c209rdz7ypNbT1hM+ruqSAkoI8YmbEDGrLCplcW0JPX5KW7n62dfXR2tVPIuXUlRWysqmDxm3dTK0r5cTpNZw0o5bXN3awvrWbRNI5fFwZU2tLSaScxVG8K5s6GF9ZzNGTKonnxejsTbCtq594XowJlUXMP2r8bn5nbJG7z8263Cgmi3OAqwlnQ50AfNfd50Ud3IuAY6NFXwCOc/fh+j/22WRx8cUXc99993H44YcTj8cpKytj/PjxLF68mNdee42PfOQjrF27lp6eHq655hquuOIKYMfwJR0dHcyfP59TTjmFp556iokTJ3LfffdRXFy8y7b2xvtt7e6noih/p8I8kUzxq0WNLFq9ja6+BJ86eRrT68t4YfU26soLeW19G3ctXEtRVCh19SWJmVFbVsAHjxjHis0dPPzaJurKCmju7GPJulbqygqZXl9KaUE+D722CYDZDZUcNbGSP63Ywurmru3bL8iPkYyO+oZSX15Ia3c/fYnUTtMrivLpSaToT6YY+BnkRe0O6a915qyxvGt8BRtaunlxbQuHjy3nqIZKiuN5bOnoZXNbL/2pFJXFcSqK4mxs7eGVda28vqmdoydVMaWmhC2dfUyuKaahuoSuviQdPQkc58gJlbR29/Pq+ja6+hKUFuYzo76M6fWlxPOMFZs7qCkN+2JqbSlvbelgxeYO3jWugsJ4jA2tPfQnUpQXxRlbUcivX1hH47Yu3j9rLGWF+fQmUkyvK6WyJDql2qGpo5c1zV0UF+QxoaqYGfVl2993IpmiubOPvkSKhupizEJNauBou7I4zsa2HuJ5MY5uqKQ3keKNTR20dPcxtbaUSTUlZJNIpkg55MeMWGzP97X1J0PiGVcZal0yvFFPFmb2S0ItoQ7YRDjDKQ7g7j+ITp39HqHzugv4ZHQ7S8zsU4SbzAN8w91/km172ZLFP/72VV5b3/bO31iaWRMq+OqHjxh2mfSaxeOPP84555zDkiVLtp/iunXrVmpqauju7ub444/niSeeoLa2dqdkccghh7Bw4ULmzJnDRRddxLnnnstll122y7beTrJIpXzIH6q78+LaFhau2kpDdQnT60sZU17Eq+tb+cmfV/E/yzZz8iG1XHnqDLZFTQy/fWkDSze0UV9eSDLlbO3sI2bsVB2fOb6CssI8Wrr6KS3Mx91Zs7Vr+xHwnElV9PQnKS3MZ86kKrZ19vFmUwfrWno49+gJHDuliuv/+xVS7pw0o5ZTDqljXGUxrd2hiaYwP8Zph49hTHkhfckUm1p76EkkKSnI5/ipNbT39PP8qm3UlIZCc3VzF8+9tZV4XowrT5tOcTyPlu5+JteUkB8zmtp7Wbh6G5NrSjhyYuWI9ulQ+1InHsi+bqTJImd9Fu5+SZb5Dnwmw7xbCTewP+DMmzdvp2shvvvd73LvvfcCsHbtWt544w1qa2t3WmfatGnMmTMHgOOOO45Vq1a9rW0mkil++dwa7nlhHcXxGE3tvby1pZO5U2o484ixHDWxkmdWbuWZlc2s2drFupbuIV+noiify06czH0vrufjtz63ffqkmmJ+cNmxfPCIcXT3J/n506vp6kvy7hm1tHb3U1VSwPFTq3cpOPuTKZ5Z2cyY8iIOH1ee9X18YNZYYmbEM/QJpJtRX7bT86qSAj4wa+z258dNqeGCYxt2WqY26uAEGFNRxNnp1fquraGNp3jkZ3HnLFEkE5CXD/3d0L4RqqeG2Ab0dsD6F6B+JpRlHUw0s542yC+C/EEdp53NsPXNMH/KSVBQ+vZfu209rHwcjjgf4mm1ZHfYujJMKxsLsTxI9of3WlQRlunvAU9CayO0rIUJc6C0bsdr9PfAq/eGuCfOheop0NcJK5+AVD9UNsD48Hsi0RumNa+ARB9Mmhe2OZyOJlj0E3julvD6p14bPoP2jbDpVdi0JMRcORGOuAAqJsCWN6CtMazb2RS2GYtDcRUs/z2seCTENOUkGDcbDvtg2NbiO2DdIuhth6nvgUnHQ+0hYDF464/w5qOQ7IOqKSGOHDqgr+BOl60GsLeUlu74YT3++OM88sgjPP3005SUlHD66acPea1EYeGOQiwvL4/u7lCYuzu9iRRbO/vo7k/S1N7LV25+ir5kioldS6mdNJNJE8bzy+fWsKq5iyMmVJCIGVNrSzn98DE8+XoTX//dUiCUNbMnVjJnchVXnTGDM2eNY1NbDyu3dLKptYfDxpVz3JRqygrzueZ9h7F0QxvjK4uYUFVMaeGOr1FJQT5XnlAHWPhxb30L2t+C5HHQ0wpt6yCVgHWLiDct4z3FNaFgWFUKs/8qvMhrv4HaQ8OPon0DrH4KWtZQWDsj/HDWvRB+1NPPgIa5odDYtgq6t0JhORRVhh/TxLlQWAbP/hA2vhIKhsM+CAVl4fnJfw/l46B1XSiY8vLDa3VuCQVg7SHwxkPw+Ddh/eJQiBzyAZh9EcRL4JW7oeF4OO7y8BptjaGge+vJEMf0M+CtJ0JBd8QFMGYmdGwO76+7BXrboGlZeP+Vk8K6GIw7Cub9TZj+5L+F2MYeCZ2bYdkCWLcQCsqhvxM8FQqqmhmhQEr2QVdzKIzyCmHqKeE9jZ8Ncz8Fq/4c3tvsi2DtsyEJTnl3WKevAxrmhQJv0U/glXtCQfeR70NpPbx2Hyz6WXifA8YdBe/9Cvzx2zD+aDjhb+GhL4cCcdqpUDUZGhfC0t/CUX8BZ349fFa3Xxi+C4/+E0w8NuyL4mrYuCQkIgjbPOoiWPZAWPbEq8I+WvbArj+siXNh1nnhO/byXdCadkLljPfCptegY+OOafESSPSE/ZeubFwowLtbwvPqqTDxOGhZHf7aN4b3BjD9dFj9Z/jx73d+jbyC8NfXAY98LbyvgXWGUlwdPo+NS+Cp/wq/j+Lq8Pl1bAwx5RcO/b4LysN3fPzRmV9/DzloksVoKS8vp7190B0q2zdCvJjW1laqq6spKSlh2bJlPPPMM6T6utjU2o0TJYP+JO7Qm0iS6Omkv7eL/r5e3mrqoLMvSSpq6iiPQ2mqg8N9NRNYz2e6vs7GZTV89pWrqJr4bn5+aimnbP01VjEBZp0bfgTAhtZuXl+xgmM33k259cDZ/xYK0vs+R331NI485P1w1JkQi0HbBnj8e9QXVVE/6Xiofw/0d8HyP4WjnPwCSCXh+R+H9zn1FFjxaDgKjOWHH0G6oqpwxORRB+lj3wz/e1t33ZHxkrCtospQQC9fAIt/kf0DiJeG9eoPD7EtSDv6Wv3n8CN98EuhsK4/LBzhpcfX0xIS1xn/EGJ95VfwelQ4FFbAkv+GP1zPTifslU8Ihd8Lt4Xt5xeEAmxAXmEoCONFUP+ucNTbtCwU+ABvPAgv3wmWFxJbsg8SUW1vzBFwyufCeyosh9IxYV90bobJJ4T9VFIT9tHrD4aEUD0Flj4QYh/w4PUZdpiF9xIvhWM/Bm88Aj89Z8fsQz4AJ10VEmlPK/z2s3DHRVBSG7b13C1h3TEz4U//Hgrj/KLwXXj+xyHZpPpDAvzIzeHIufnNcGCxeWk4Gj/x0+Ho5fUH4ZmbYOxRMOt4eOq7oXB8999BSV14jcqJsObZkIAf/krYZw1z4dzvhmWW/x4W/RRqpoWkVzY27OvG58O+LSwL69RMC7WBpfeH/wM1yE1L4LkfQvU0qJ0BE46BusPD+5kwJyTblY+FGkNpHYw9IuybvHhIis/9KCSKqe+Bmulh+6V1ofBP9IYkXTFhR+0q0QeNz4UDnL5O+MufhtoGQMuacNCy7a1QAxszC2acEba1Fxwwt1Ud1Q5uT4UPOb8gHNEm+8MPKdEDJXV89BOf4uWXX6a4uJixNRU88JNvAdBrxZz3yc+yfv16Dj90Bk0b1vK1z13Je046nmknfogFv/8d+R2bOP/yq3ni0T8wxlr51g9uo6Oziy9//iqSsSI8FideWExez1aWrmxk5sOXhhjGHUmquwXbuhI78dPw6m+gY1MomMvHw6efCgXhk98OBVkqOtvlmpfh2R+EL2u8BPraQyE26QR4/Q+hEBwo9Etqw/tMJUKBkEqEvyMvDM9f/0M4op52avhxlo8L1eVYXihMqqdCKhXWaX4DnvhXwOHka8IReNu6UBhOPC78oNrW7/ihpZKw4SXYsDi8Zu0hoZDsbQ/NI4meUCvYtioc7Y6fHWLe9FrYBy1r4a5Lw2c3472h6WLbW3Dsx0NzQNeWUKOpPxxO/MyOpphUElb9KeyHw84KNYe3/gh1h4b3UzEhFCz9XeGHPWFOaG5Y/edwtJtXCIefFZJeJr0d8OzN4ej2PZ8Pn0P7hvDeC7M31w2pbUNo9pl2aii8lj0QPtPKhlDIl44J73H106FQnPnhsK3ullAQx0vD0Wv9YTu/7vrFIcHOuyJ8xi/fBad9MbxGojd8jkUV4f2ufBzeeDi8j6P+Mmw7m+5tUFgZDla2rAiFeGnt0Mu2rgvfyXjR7u2jTNx3buY7wIx6B/feNirJIpUIP8KuZsLRWAlUTQpHStuPomPRl9tCwdvfHY4uLIa3b6CdEtq9hPG2lQR5bExVM6agh6JE6IxPxArJS/VhOP2F1fQVVEOylxLvxgbaW5N9EIuzdHM/M5f9e9j+x+8LBfYfvggv3h5+RJ/4bSgUbz0zam5YEuKcc2moUt91aTjae+bm8OP+2L2h6eHF28Oy1VPD/LKx4Whq2e9C4TjjvaHgwUIhmt5+vC976a7QVn3aF0MTlMhBaNQ7uA84qURoaRgoVBJ9sOX1UFiX1IY2yvYN0LQ8VGvrDgvVw5Y10LkFBxJ5RbTn19OVqKInkaLYa5lozVRYF10UsSlvHKVlxRRVFIWkkkqQv70Zood4YQXxoY5wUslw5LNtOVx0287zzrspHMVVTQlVbYDTr4P/+Xpob774jtCunEqFo7blC0Iz1OnXhfiP+ovwN9is88LfYPn7SaIAOPqvRjsCkf2GksVIDCQGCM0SFgudcJ4KSWHgbJC8gujslCnbp/VWTmNbZz+t3X309qcoyIuRSiQoyI+RXz6GVH4ZMYtRUlTJtPREkH6GSH5h+Msk29kb00/f+fkpnwvtwNNOhYLovPhYDCa/O7Rt4zAlt1eDisj+RcliKP09ob082R+dutcXnTXhIUmkUpDsDW356acNltRASQ0pd/r6k+HqzrZe3J2SwjCkQ/mgC9pgD7evjkQsL7SbDzb1ZFj+u5D0GrLWSkXkIKJkMZTOptBRWlge9T1YOJMh2RdOn8svColi4LzvNL2JJKu2dNGbCGf4VBTFmVhVTDx/P7gp4UBtYuLcnWs2InLQU7IYzD2c4VNYAbXTd51fUBqOvNNqB+6+fVygtu4wjENDdTHF8TyK4nn7z1W8444KndgzPzTakYjIPkbJYrD+7tBpXZzh1MZBfQfuzvrWHpo7esmPGUXxPCZWFVMY3w/HpInlwTUvjXYUIrIP2g/aRvaynuiCsMJdm5jSpdxZt62bpRvaaO7opb68kJnjK5heX7ZTomhpaeH73//+boXyH//xH3R1dWVfUEQkx5QsButpDRcgDXNVpEeJormzl7KiOFNrSxlfWTxkc5OShYgcCNQMla6/JwyrUJH5xnwdvQmaO3pp7e5nTEUR4yqGP5vpuuuu480332TOnDl84AMfYMyYMdx999309vZy/vnn84//+I90dnZy0UUX0djYSDKZ5Ctf+QqbNm1i/fr1nHHGGdTV1fHYY4/t6XcrIjJiB0+y+P114WKz4SR7wxlPBaUMVelKplLQn6LOYFyeUTDxaJj/L8O+5D//8z+zZMkSFi9ezEMPPcQ999zDc889h7tz7rnn8uSTT9LU1MSECRP43e9+B0BrayuVlZV85zvf4bHHHqOubj+60E1EDkhqhtrOw9DPsXyG2i2O05d0zMK9bwvz8rAh7wCb2UMPPcRDDz3EMcccw7HHHsuyZct44403OOqoo3jkkUf44he/yB//+EcqK3fv/gkiIrly8NQs5v/z8PN728M4QdVTh7xnQVt3P6ubO2moLqF0N2+M7u5cf/31XHnllbvMW7RoEQsWLOD666/nzDPP5IYbbtitbYiI5IJqFgN6WoFYGOFykEQyxYaWbgrz86gueXvDAacPUf7BD36QW2+9lY6ODgDWrVvH5s2bWb9+PSUlJVx22WVce+21vPDCC7usKyIymg6emkU2iR6IF4YxktIM3P6zP+XMqC952xfY1dbWcvLJJ3PkkUcyf/58PvrRj3LSSWF8+rKyMm6//XZWrFjBF77wBWKxGPF4nJtvvhmAK664gvnz5zN+/Hh1cIvIqNIQ5QM2LgnDe1RP2Wny5vYeNrb20FBdQs1uNj/tLXvt/h0icsAY6RDlaoaC6KY9/btcnd3Tn2RTWy+VxfF9PlGIiORSTpOFmZ1lZsvNbIWZXTfE/Clm9qiZvWxmj5tZQ9q8fzWzV81sqZl913I5wFKiN/wfNHje+pZuYgYTqjSonogc3HKWLMwsD7gJmA/MAi4xs1mDFvsWcJu7zwZuBL4Zrftu4GRgNnAkcDxw2u7EMaJmtv7o/sb5Oy6w6+lP0tGboL68kHjevl8BO1CaE0Vk35TLUnAesMLdV7p7H3AnMPjWarOAR6PHj6XNd8KNHgqAQiAObHq7ARQVFdHc3Jy9IE30AhZGk41s6+rDMKpL9v3mJ3enubmZoqJRuDeGiBwUcnk21ERgbdrzRuCEQcu8BFwI/CdwPlBuZrXu/rSZPQZsAAz4nrsvHbwBM7sCuAJg8uTJuwTQ0NBAY2MjTU1Nw0fa0QSehNZlQBilfGNbDwV5xor2Ye5Qtw8pKiqioaEh+4IiIrshl8liqD6GwYf41wLfM7PLgSeBdUDCzA4BZgIDpd/DZnaquz+504u53wLcAuFsqMEbi8fjTJs2LXuk3/kLmPJuuPBHADz5ehOf+s1z3HzpsZwyc3z29UVEDnC5TBaNwKS05w3A+vQF3H09cAGAmZUBF7p7a1RjeMbdO6J5vwdOJCSUPaunDdoaw721I48vb6IwP8YZ7xqzxzcnIrI/ymWfxfPAoWY2zcwKgIuB+9MXMLM6MxuI4Xrg1ujxGuA0M8s3szihc3uXZqg9IpWAUz4H00/fPumZlc0cN6Waov3xBkYiIjmQs2Th7gngauBBQkF/t7u/amY3mtm50WKnA8vN7HVgLPCNaPo9wJvAK4R+jZfc/bc5CbSkBt7/VWgI16S0dPWxdGMbJ06vzcnmRET2Rzkd7sPdFwALBk27Ie3xPYTEMHi9JLDraHt7wXNvbcUdJQsRkTT7/gUEe9kzK7dSmB90yyWeAAAYaklEQVTj6EkaJlxEZICSxSAD/RWF+eqvEBEZoGSRJplylm9qZ86kqtEORURkn6Jkkaa5o5dkyhlfqSuhRUTSKVmk2dweBhSsL1eyEBFJp2SRZlNbDwBjK/aPIT5ERPYWJYs0AzWLMRWqWYiIpFOySDNQs6gvU81CRCSdkkWaze291JYWUJCv3SIikk6lYprNbT3Ul6tWISIymJJFms3tvYxVf4WIyC6ULNJsauthjGoWIiK7ULKIJFPOlo4+1SxERIagZBFp7gxXb4/RNRYiIrtQsohsbouusdDV2yIiu1CyiGxuD9dYqGYhIrIrJYvIQM1CfRYiIrtSsohsH0RQV2+LiOxCySLS1ZekIC+mq7dFRIaQ05LRzM4ys+VmtsLMrhti/hQze9TMXjazx82sIW3eZDN7yMyWmtlrZjY1l7G6OzHlCRGRIeWseDSzPOAmYD4wC7jEzGYNWuxbwG3uPhu4Efhm2rzbgH9z95nAPGBzrmKFcJ1FzCyXmxAR2W/l8lh6HrDC3Ve6ex9wJ3DeoGVmAY9Gjx8bmB8llXx3fxjA3TvcvSuHsZJ0J0/JQkRkSLlMFhOBtWnPG6Np6V4CLowenw+Um1ktcBjQYma/NrMXzezfoprKTszsCjNbaGYLm5qa3lGwqZQTiylZiIgMJZfJYqiS1wc9vxY4zcxeBE4D1gEJIB94TzT/eGA6cPkuL+Z+i7vPdfe59fX17yjYlINyhYjI0HKZLBqBSWnPG4D16Qu4+3p3v8DdjwG+FE1rjdZ9MWrCSgC/AY7NYayhGUrZQkRkSLlMFs8Dh5rZNDMrAC4G7k9fwMzqzGwghuuBW9PWrTazgerCe4HXchhraIZSn4WIyJByliyiGsHVwIPAUuBud3/VzG40s3OjxU4HlpvZ68BY4BvRuklCE9SjZvYKoUnrR7mKFcLZUKpZiIgMLT+XL+7uC4AFg6bdkPb4HuCeDOs+DMzOZXzpQp+FkoWIyFB0GVokpYvyREQyUvEYSaZ0nYWISCZKFpGk6zoLEZFMlCwi7jobSkQkEyWLiJqhREQyU7KIJFOoGUpEJAMli4i7k6e9ISIyJBWPkaT6LEREMlKyiOh+FiIimSlZRFIaSFBEJCMli0gqhc6GEhHJQMkiknRHuUJEZGhKFpGURp0VEclIySKimx+JiGSmZBHREOUiIpkpWUTCnfJGOwoRkX2TkkVEd8oTEclsRMnCzP7bzM5Ju1/2ASelK7hFRDIaaeF/M/BR4A0z+2cze1cOYxoVShYiIpmNKFm4+yPufilwLLAKeNjMnjKzT5pZPNN6ZnaWmS03sxVmdt0Q86eY2aNm9rKZPW5mDYPmV5jZOjP73tt7W2+fmqFERDIbcbOSmdUClwP/C3gR+E9C8ng4w/J5wE3AfGAWcImZzRq02LeA29x9NnAj8M1B8/8JeGKkMb4TKdcQ5SIimYy0z+LXwB+BEuDD7n6uu9/l7n8HlGVYbR6wwt1XunsfcCdw3qBlZgGPRo8fS59vZscBY4GHRvpm3olw86O9sSURkf3PSGsW33P3We7+TXffkD7D3edmWGcisDbteWM0Ld1LwIXR4/OBcjOrjTrSvw18YbigzOwKM1toZgubmppG+FaGpj4LEZHMRposZppZ1cATM6s2s6uyrDNUyeuDnl8LnGZmLwKnAeuABHAVsMDd1zIMd7/F3ee6+9z6+vqsb2I4qZSrGUpEJIP8ES73N+5+08ATd99mZn8DfH+YdRqBSWnPG4D16Qu4+3rgAgAzKwMudPdWMzsJeE+UkMqAAjPrcPddOsn3lKTrHtwiIpmMNFnEzMzc3WF753VBlnWeBw41s2mEGsPFhNNvtzOzOmCru6eA64FbAaIzrwaWuRyYm8tEAergFhEZzkiboR4E7jaz95nZe4FfAn8YbgV3TwBXR+suBe5291fN7EYzOzda7HRguZm9TujM/sZuvIc9QsN9iIhkNtKaxReBK4FPE/oiHgJ+nG0ld18ALBg07Ya0x/cA92R5jZ8CPx1hnLtNo86KiGQ2omQRNRPdHP0dkHQPbhGRzEaULMzsUMIFc7OAooHp7j49R3Htde6oZiEiksFI+yx+QqhVJIAzgNuAn+cqqNGQVJ+FiEhGI00Wxe7+KGDuvtrdvwa8N3dh7X1J13UWIiKZjLSDuye6qvoNM7uacCrsmNyFtfelUrrOQkQkk5HWLD5LGBfq74HjgMuAT+QqqNGQ0tlQIiIZZa1ZRBfgXeTuXwA6gE/mPKq9zN1JOZhqFiIiQ8pas3D3JHCcHcAlaSoasUrNUCIiQxtpn8WLwH1m9iugc2Ciu/86J1HtZckoW+QdsDeNFRF5Z0aaLGqAZnY+A8qBAyJZpMKQVzobSkQkg5FewX3A9VOk254s1AwlIjKkkV7B/RN2vRcF7v6pPR7RKNjeDKVkISIypJE2Qz2Q9riIcFe79RmW3e+kUuG/mqFERIY20mao/05/bma/BB7JSUSjYEcz1CgHIiKyj9rd838OBSbvyUBGU9IHzoZSthARGcpI+yza2bnPYiPhHhcHhFRKHdwiIsMZaTNUea4DGU2qWYiIDG9EzVBmdr6ZVaY9rzKzj+QurL1r4Apu5QoRkaGNtM/iq+7eOvDE3VuAr+YmpL1PzVAiIsMbabIYarmRDEJ4lpktN7MVZnbdEPOnmNmjZvaymT1uZg3R9Dlm9rSZvRrN+6sRxrlbdgz3oWQhIjKUkSaLhWb2HTObYWbTzezfgUXDrRCNVnsTMJ9wO9ZLzGzWoMW+Bdzm7rOBGwm3bgXoAj7u7kcAZwH/YWZVI4z1bUupz0JEZFgjTRZ/B/QBdwF3A93AZ7KsMw9Y4e4r3b0PuBM4b9Ays4BHo8ePDcx399fd/Y3o8XpgM1A/wljftoFkcQAPrCsi8o6M9GyoTmCXZqQsJgJr0543AicMWuYl4ELgPwlXhZebWa27Nw8sYGbzgALgzcEbMLMrgCsAJk/e/cs+ktEV3BruQ0RkaCM9G+rh9GYgM6s2swezrTbEtMHjS10LnGZmLwKnEW7Xmkjbznjg58An3T21y4u53+Luc919bn397lc8NES5iMjwRjo2VF10BhQA7r7NzLLdg7sRmJT2vIFB40lFTUwXAJhZGXDhwFlXZlYB/A74srs/M8I4d4tGnRURGd5Ij6VTZra9ncfMpjLEKLSDPA8cambTzKwAuBi4P30BM6szs4EYrgdujaYXAPcSOr9/NcIYd5uShYjI8EZas/gS8CczeyJ6fipRX0Em7p4ws6uBB4E84FZ3f9XMbgQWuvv9wOnAN83MgSfZ0Wl+UbSNWjO7PJp2ubsvHmG8b4tOnRURGd5IO7j/YGZzCQliMXAf4YyobOstABYMmnZD2uN7gHuGWO924PaRxLYn6E55IiLDG+lAgv8LuIbQ77AYOBF4mp1vs7rfGhjuQ2dDiYgMbaR9FtcAxwOr3f0M4BigKWdR7WXJlO5nISIynJEmix537wEws0J3XwYcnruw9q7tY0MpW4iIDGmkHdyN0XUWvwEeNrNtHEC3VdUQ5SIiwxtpB/f50cOvmdljQCXwh5xFtZdpiHIRkeGNtGaxnbs/kX2p/YuGKBcRGZ4GuEDXWYiIZKNkwY4+C9UsRESGpmQBuJKFiMiwlCxIG6JczVAiIkNSsiD91NlRDkREZB+l4hE1Q4mIZKNkQfpwH0oWIiJDUbJAp86KiGSjZIGGKBcRyUbJAg1RLiKSjZIFGqJcRCQbJQvUDCUiko2SBWkd3GqGEhEZkpIFaUOUq2YhIjKknCYLMzvLzJab2Qozu26I+VPM7FEze9nMHjezhrR5nzCzN6K/T+QyzpT6LEREhpWzZGFmecBNwHxgFnCJmc0atNi3gNvcfTZwI/DNaN0a4KvACcA84KtmVp2rWHWnPBGR4eWyZjEPWOHuK929D7gTOG/QMrOAR6PHj6XN/yDwsLtvdfdtwMPAWbkKVFdwi4gML5fJYiKwNu15YzQt3UvAhdHj84FyM6sd4bqY2RVmttDMFjY1Ne12oK6ahYjIsHKZLIYqeX3Q82uB08zsReA0YB2QGOG6uPst7j7X3efW19fvdqADQ5SrZiEiMrS3fQ/ut6ERmJT2vAFYn76Au68HLgAwszLgQndvNbNG4PRB6z6eq0B33CkvV1sQEdm/5bJm8TxwqJlNM7MC4GLg/vQFzKzOzAZiuB64NXr8IHCmmVVHHdtnRtNyIpVyYgammoWIyJBylizcPQFcTSjklwJ3u/urZnajmZ0bLXY6sNzMXgfGAt+I1t0K/BMh4TwP3BhNy4mUu5qgRESGkctmKNx9AbBg0LQb0h7fA9yTYd1b2VHTyKmkuy7IExEZhq7gJjRDaagPEZHMlCwIZ0PptFkRkcyULAh9FqpYiIhkpmRBSBaqWYiIZKZkQRjuQ30WIiKZKVkQhijX2VAiIpkpWbDjojwRERmakgXhOgs1Q4mIZKZkQVSzUNVCRCQjJQt0NpSISDZKFkDSNTy5iMhwlCxQB7eISDZKFkTXWShbiIhkpGSBhigXEclGyQIlCxGRbJQsUDOUiEg2ShZEZ0MpWYiIZKRkAbg7ecoVIiIZKVkQmqHUZyEikllOk4WZnWVmy81shZldN8T8yWb2mJm9aGYvm9nZ0fS4mf3MzF4xs6Vmdn0u40xquA8RkWHlLFmYWR5wEzAfmAVcYmazBi32ZeBudz8GuBj4fjT9L4FCdz8KOA640sym5irWlAYSFBEZVi5rFvOAFe6+0t37gDuB8wYt40BF9LgSWJ82vdTM8oFioA9oy1Wg4X4WuXp1EZH9Xy6LyInA2rTnjdG0dF8DLjOzRmAB8HfR9HuATmADsAb4lrtvHbwBM7vCzBaa2cKmpqbdDlR9FiIiw8tlshiq9PVBzy8BfuruDcDZwM/NLEaolSSBCcA04PNmNn2XF3O/xd3nuvvc+vr63Q5Uo86KiAwvl8miEZiU9ryBHc1MA/4auBvA3Z8GioA64KPAH9y93903A38G5uYqUPVZiIgML5fJ4nngUDObZmYFhA7s+wctswZ4H4CZzSQki6Zo+nstKAVOBJblKtBkCkzJQkQko5wlC3dPAFcDDwJLCWc9vWpmN5rZudFinwf+xsxeAn4JXO7uTjiLqgxYQkg6P3H3l3MVayrl5KmDW0Qko/xcvri7LyB0XKdPuyHt8WvAyUOs10E4fXavSKrPQkRkWDqeRqPOiohko2TBwJ3ylCxERDJRskDNUCIi2ShZAKkUqlmIiAxDyYKBi/JGOwoRkX2Xikg03IeISDZKFkRnQ6nPQkQkIyULontwq2YhIpKRkgVhiHKdDSUikpmSBeE6C1UsREQyU7Igus5C2UJEJCMlC6I+CzVDiYhkpGQBuKOzoUREhqFkQWiGUq4QEclMyQKdOisiks1BnyxSqXBbcDVDiYhkpmThUbJQzUJEJKODPlkko2Shs6FERDI76JNFKhX+q2YhIpJZTpOFmZ1lZsvNbIWZXTfE/Mlm9piZvWhmL5vZ2WnzZpvZ02b2qpm9YmZFuYgxtb1mkYtXFxE5MOTn6oXNLA+4CfgA0Ag8b2b3u/traYt9Gbjb3W82s1nAAmCqmeUDtwMfc/eXzKwW6M9FnEn1WYiIZJXL4+l5wAp3X+nufcCdwHmDlnGgInpcCayPHp8JvOzuLwG4e7O7J3MR5PazoZQsREQyymWymAisTXveGE1L9zXgMjNrJNQq/i6afhjgZvagmb1gZv9nqA2Y2RVmttDMFjY1Ne1WkMmUOrhFRLLJZbIYqvT1Qc8vAX7q7g3A2cDPzSxGaB47Bbg0+n++mb1vlxdzv8Xd57r73Pr6+t0KMp4f45yjxjO1rnS31hcRORjkrM+CUJOYlPa8gR3NTAP+GjgLwN2fjjqx66J1n3D3LQBmtgA4Fnh0TwdZURTnpkuP3dMvKyJyQMllzeJ54FAzm2ZmBcDFwP2DllkDvA/AzGYCRUAT8CAw28xKos7u04DXEBGRUZGzmoW7J8zsakLBnwfc6u6vmtmNwEJ3vx/4PPAjM/vfhCaqy93dgW1m9h1CwnFggbv/LlexiojI8Mx9cDfC/mnu3Lm+cOHC0Q5DRGS/YmaL3H1utuV0KZqIiGSlZCEiIlkpWYiISFZKFiIikpWShYiIZHXAnA1lZk3A6nfwEnXAlj0Uzp6m2HaPYts9im337K+xTXH3rENgHDDJ4p0ys4UjOX1sNCi23aPYdo9i2z0HemxqhhIRkayULEREJCslix1uGe0AhqHYdo9i2z2Kbfcc0LGpz0JERLJSzUJERLJSshARkawO+mRhZmeZ2XIzW2Fm141yLJPM7DEzW2pmr5rZNdH0r5nZOjNbHP2dPUrxrTKzV6IYFkbTaszsYTN7I/pfPQpxHZ62bxabWZuZfXa09puZ3Wpmm81sSdq0IfeTBd+Nvn8vm1lO78SVIbZ/M7Nl0fbvNbOqaPpUM+tO238/GIXYMn6GZnZ9tN+Wm9kHRyG2u9LiWmVmi6Ppe3u/ZSo39ux3zt0P2j/CfTbeBKYDBcBLwKxRjGc8cGz0uBx4HZhFuFf5tfvA/loF1A2a9q/AddHj64B/2Qc+043AlNHab8CphDs7Lsm2nwi3E/494TbEJwLPjkJsZwL50eN/SYttavpyo7TfhvwMo9/FS0AhMC36HeftzdgGzf82cMMo7bdM5cYe/c4d7DWLecAKd1/p7n3AncB5oxWMu29w9xeix+3AUmDiaMUzQucBP4se/wz4yCjGAuHOi2+6+zu5mv8dcfcnga2DJmfaT+cBt3nwDFBlZuP3Zmzu/pC7J6KnzxBugbzXZdhvmZwH3Onuve7+FrCC8Hve67GZmQEXAb/M1faHM0y5sUe/cwd7spgIrE173sg+Ujib2VTgGODZaNLVUZXx1tFo6ok48JCZLTKzK6JpY919A4QvLTBmlGIbcDE7/2j3hf0GmffTvvYd/BThqHPANDN70cyeMLP3jFJMQ32G+9J+ew+wyd3fSJs2KvttULmxR79zB3uysCGmjfq5xGZWBvw38Fl3bwNuBmYAc4ANhCrvaDjZ3Y8F5gOfMbNTRymOIVm41/u5wK+iSfvKfhvOPvMdNLMvAQngF9GkDcBkdz8G+Bxwh5lV7OWwMn2G+8x+Ay5h5wOUUdlvQ5QbGRcdYlrWfXewJ4tGYFLa8wZg/SjFAoCZxQkf+C/c/dcA7r7J3ZPungJ+RA6r28Nx9/XR/83AvVEcmwaqsNH/zaMRW2Q+8IK7b4J9Z79FMu2nfeI7aGafAD4EXOpRw3bUxNMcPV5E6Bc4bG/GNcxnuK/st3zgAuCugWmjsd+GKjfYw9+5gz1ZPA8cambToqPSi4H7RyuYqO3z/wFL3f07adPT2xPPB5YMXncvxFZqZuUDjwmdoksI++sT0WKfAO7b27Gl2ekIb1/Yb2ky7af7gY9HZ6icCLQONB3sLWZ2FvBF4Fx370qbXm9medHj6cChwMq9HFumz/B+4GIzKzSzaVFsz+3N2CLvB5a5e+PAhL293zKVG+zp79ze6rHfV/8IZwa8Tsj+XxrlWE4hVAdfBhZHf2cDPwdeiabfD4wfhdimE84+eQl4dWBfAbXAo8Ab0f+aUdp3JUAzUJk2bVT2GyFhbQD6CUdxf51pPxGaBG6Kvn+vAHNHIbYVhDbsge/cD6JlL4w+65eAF4APj0JsGT9D4EvRflsOzN/bsUXTfwr87aBl9/Z+y1Ru7NHvnIb7EBGRrA72ZigRERkBJQsREclKyUJERLJSshARkayULEREJCslC5F9gJmdbmYPjHYcIpkoWYiISFZKFiJvg5ldZmbPRfcp+KGZ5ZlZh5l928xeMLNHzaw+WnaOmT1jO+4TMXA/gUPM7BEzeylaZ0b08mVmdo+Fe0v8IroyV2SfoGQhMkJmNhP4K8KAinOAJHApUEoYk+pY4Angq9EqtwFfdPfZhCtlB6b/ArjJ3Y8G3k24MhjCaKGfJdyLYDpwcs7flMgI5Y92ACL7kfcBxwHPRwf9xYTB2VLsGEjuduDXZlYJVLn7E9H0nwG/isbXmuju9wK4ew9A9HrPeTTGkIW7rk0F/pT7tyWSnZKFyMgZ8DN3v36niWZfGbTccGPoDNe01Jv2OIl+n7IPUTOUyMg9CvyFmY2B7fc4nkL4Hf1FtMxHgT+5eyuwLe3GNx8DnvBwn4FGM/tI9BqFZlayV9+FyG7QkYvICLn7a2b2ZcLdAmOEEUg/A3QCR5jZIqCV0K8BYVjoH0TJYCXwyWj6x4AfmtmN0Wv85V58GyK7RaPOirxDZtbh7mWjHYdILqkZSkREslLNQkREslLNQkREslKyEBGRrJQsREQkKyULERHJSslCRESy+v9IEB+V9bPwHgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# summarize history for accuracy\n",
    "plt.plot(history.history['acc'])\n",
    "plt.plot(history.history['val_acc'])\n",
    "plt.title('model accuracy')\n",
    "plt.ylabel('accuracy')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'test'], loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XmYXGWd9//3t5be9yX71mEJCSGEEAKICypLAGVRBETcxhl0HnHw5zLCzLj+ZvHRGRUdRsERVxBRZIwaBkRZlDUhBEhIQnbS2brTSe9LdVXdzx/36U6nU13pNKmuTtfndV19ddWpU1XfOnXqfM59n82cc4iIiACEsl2AiIiMHQoFERHpp1AQEZF+CgUREemnUBARkX4KBRER6adQEBkmM/uRmf3zMMfdZmYXvN7XERltCgUREemnUBARkX4KBRlXgm6bz5rZS2bWYWY/MLOJZvagmbWZ2SNmVjlg/MvNbK2ZNZvZY2Y2d8BjZ5jZquB5vwAKBr3XO8xsdfDcp8xswQhr/hsz22Rm+81smZlNCYabmX3TzBrMrCX4TPODxy41s1eC2naa2WdGNMFEBlEoyHj0buBC4GTgncCDwD8ANfh5/u8AzOxk4OfAJ4FaYDnwWzPLM7M84H+AnwJVwC+D1yV47iLgLuCjQDVwB7DMzPKPplAzexvwb8A1wGRgO3Bv8PBFwJuDz1EBXAs0BY/9APioc64UmA/86WjeV2QoCgUZj77jnNvrnNsJ/Bl41jn3gnOuB3gAOCMY71rg9865PzjneoF/BwqBNwDnAFHgW865Xufcr4AVA97jb4A7nHPPOucSzrkfAz3B847G+4C7nHOrgvpuBc41s1lAL1AKnAKYc26dc2538LxeYJ6ZlTnnDjjnVh3l+4qkpFCQ8WjvgNtdKe6XBLen4NfMAXDOJYEdwNTgsZ3u0DNGbh9weybw6aDrqNnMmoHpwfOOxuAa2vGtganOuT8B/wncDuw1szvNrCwY9d3ApcB2M3vczM49yvcVSUmhILlsF37hDvg+fPyCfSewG5gaDOszY8DtHcC/OOcqBvwVOed+/jprKMZ3R+0EcM592zl3JnAqvhvps8HwFc65K4AJ+G6u+47yfUVSUihILrsPuMzM3m5mUeDT+C6gp4CngTjwd2YWMbN3AUsGPPf7wMfM7Oxgg3CxmV1mZqVHWcM9wIfNbGGwPeJf8d1d28zsrOD1o0AH0A0kgm0e7zOz8qDbqxVIvI7pINJPoSA5yzm3AbgB+A6wD79R+p3OuZhzLga8C/gQcAC//eHXA567Er9d4T+DxzcF4x5tDX8EPg/cj2+dnABcFzxchg+fA/gupib8dg+A9wPbzKwV+FjwOUReN9NFdkREpI9aCiIi0k+hICIi/RQKIiLST6EgIiL9Itku4GjV1NS4WbNmZbsMEZHjyvPPP7/POVd7pPGOu1CYNWsWK1euzHYZIiLHFTPbfuSx1H0kIiIDKBRERKSfQkFERPodd9sUUunt7aW+vp7u7u5sl5JRBQUFTJs2jWg0mu1SRGScGhehUF9fT2lpKbNmzeLQk1qOH845mpqaqK+vp66uLtvliMg4NS66j7q7u6murh63gQBgZlRXV4/71pCIZNe4CAVgXAdCn1z4jCKSXeMmFI6koyfOnpZukjorrIjIkHImFDpjcRrauslEJjQ3N/Nf//VfR/28Sy+9lObm5mNfkIjICOVMKEBf18uxT4WhQiGRSH8xrOXLl1NRUXHM6xERGalxsffR0chES+GWW25h8+bNLFy4kGg0SklJCZMnT2b16tW88sorXHnllezYsYPu7m5uvvlmbrzxRuDgKTva29u55JJLeOMb38hTTz3F1KlT+c1vfkNhYeGxL1ZEJI1xFwpf/u1aXtnVetjw3kSSWDxJUX6Eo91cO29KGV9856lDPv7Vr36VNWvWsHr1ah577DEuu+wy1qxZ07/r6F133UVVVRVdXV2cddZZvPvd76a6uvqQ19i4cSM///nP+f73v88111zD/fffzw036AqLIjK6xl0oDGU099tZsmTJIccSfPvb3+aBBx4AYMeOHWzcuPGwUKirq2PhwoUAnHnmmWzbtm3U6hUR6TPuQmGoNfqm9h52Nncxd3IZ0XBmN6UUFxf3337sscd45JFHePrppykqKuL8889PeaxBfn5+/+1wOExXV1dGaxQRSSVnNjT37eKfiW0KpaWltLW1pXyspaWFyspKioqKWL9+Pc8888yxL0BE5BgZdy2FoWVu76Pq6mrOO+885s+fT2FhIRMnTux/bOnSpXzve99jwYIFzJkzh3POOeeYv7+IyLFi7jg7mGvx4sVu8EV21q1bx9y5c9M+b39HjPoDncyZVEp+JJzJEjNqOJ9VRGQwM3veObf4SOPlXPdRBhoKIiLjRu6EQvBfmSAiMrScCQURETmynAkFtRRERI4sZ0Iho/ukioiMEzkTCmopiIgcWc6EQiaN9NTZAN/61rfo7Ow8xhWJiIxM7oRCBnuPFAoiMl7kzBHNmTwh3sBTZ1944YVMmDCB++67j56eHq666iq+/OUv09HRwTXXXEN9fT2JRILPf/7z7N27l127dvHWt76VmpoaHn300QxWKSJyZOMvFB68Bfa8fNjgomSS2b1JCvLCA45kG6ZJp8ElXx3y4YGnzn744Yf51a9+xXPPPYdzjssvv5wnnniCxsZGpkyZwu9//3vAnxOpvLycb3zjGzz66KPU1NQcXU0iIhmQ0e4jM1tqZhvMbJOZ3ZJmvKvNzJnZEQ/Bfh3V+H8Z3tL88MMP8/DDD3PGGWewaNEi1q9fz8aNGznttNN45JFH+NznPsef//xnysvLM1uIiMgIZKylYGZh4HbgQqAeWGFmy5xzrwwarxT4O+DZY/LGQ6zRd/XE2dLYzuyaYkoKosfkrVJxznHrrbfy0Y9+9LDHnn/+eZYvX86tt97KRRddxBe+8IWM1SEiMhKZbCksATY557Y452LAvcAVKcb7/4GvAYdfZOAYyuQuqQNPnX3xxRdz11130d7eDsDOnTtpaGhg165dFBUVccMNN/CZz3yGVatWHfZcEZFsy+Q2hanAjgH364GzB45gZmcA051zvzOzz2Swln6ZCIWBp86+5JJLuP766zn33HMBKCkp4Wc/+xmbNm3is5/9LKFQiGg0yne/+10AbrzxRi655BImT56sDc0iknWZDIVUW3P7l8lmFgK+CXzoiC9kdiNwI8CMGTNGVkyGj1675557Drl/8803H3L/hBNO4OKLLz7seZ/4xCf4xCc+kZmiRESOUia7j+qB6QPuTwN2DbhfCswHHjOzbcA5wLJUG5udc3c65xY75xbX1tZmsGQRkdyWyVBYAZxkZnVmlgdcByzre9A51+Kcq3HOzXLOzQKeAS53zq1M/XKvj05zISJyZBkLBedcHLgJeAhYB9znnFtrZl8xs8sz8H7pRxgHJ8Q73q6SJyLHn4wevOacWw4sHzQs5X6YzrnzR/o+BQUFNDU1UV1djQ1xYNrx3lJwztHU1ERBQUG2SxGRcWxcHNE8bdo06uvraWxsHHKc3kSSva099DZFKco7Pj92QUEB06ZNy3YZIjKOHZ9Lx0Gi0Sh1dXVpx9m6r4PLf/YY37z2dK6aqwWriEgqOXOW1HDQrZRIZrkQEZExLGdCIRR80mTyeN2qICKSeTkTCuGQbykktQePiMiQciYUQn3dRwoFEZEh5VwoqPtIRGRoORMKfd1HCYWCiMiQcicU+ruPslyIiMgYljOhYMEn1akiRESGljOhcPA4BYWCiMhQcicUQtr7SETkSHImFLT3kYjIkeVMKBw8eC3LhYiIjGE5EwpBJmibgohIGjkTCmaGmU5zISKSTs6EAvg9kNRSEBEZWk6FQihk2qYgIpJGboWCuo9ERNLKqVBQ95GISHo5FQqhkEJBRCSdnAqFcMjUfSQikkZOhULIFAoiIunkXCgkktmuQkRk7MqpUAiHdO4jEZF0cisUzHSWVBGRNHIqFELa0CwiklZuhYKZuo9ERNLIqVAIh0zXaBYRSSOnQiFk2tAsIpJOToVCWEc0i4iklVOhoIPXRETSUyiIiEi/nAoFdR+JiKSXU6EQ0t5HIiJpZTQUzGypmW0ws01mdkuKxz9mZi+b2Woz+4uZzctkPSEDp+4jEZEhZSwUzCwM3A5cAswD3ptioX+Pc+4059xC4GvANzJVD+giOyIiR5LJlsISYJNzbotzLgbcC1wxcATnXOuAu8VARpfYusiOiEh6kQy+9lRgx4D79cDZg0cys48DnwLygLdlsB7CZsSTOne2iMhQMtlSsBTDDltNd87d7pw7Afgc8E8pX8jsRjNbaWYrGxsbR1yQv/LaiJ8uIjLuZTIU6oHpA+5PA3alGf9e4MpUDzjn7nTOLXbOLa6trR1xQWao+0hEJI1MhsIK4CQzqzOzPOA6YNnAEczspAF3LwM2ZrAeXaNZROQIMrZNwTkXN7ObgIeAMHCXc26tmX0FWOmcWwbcZGYXAL3AAeCDmaoHtPeRiMiRZHJDM8655cDyQcO+MOD2zZl8/8G095GISHq5dUSzgXqPRESGllOh4C+yo1QQERlKToWCLscpIpJeToWCWgoiIunlVijoegoiImnlVCiYGTrLhYjI0HIqFMIhHdEsIpJOjoWCtimIiKSTU6GgvY9ERNLLvVBQS0FEZEg5FQphneZCRCStnAoF31LIdhUiImNXToWC9j4SEUkvp0JB2xRERNLLrVDQRXZERNLKqVDQRXZERNLLqVDwLQVwai2IiKSUU6EQNgN0oR0RkaHkVCiEfCboVBciIkPIrVAIUkHbFUREUsupUAgHoaA9kEREUhtWKJjZzWZWZt4PzGyVmV2U6eKOtb5tCmopiIikNtyWwl8551qBi4Ba4MPAVzNWVYYEmaBTXYiIDGG4oRAsTrkU+KFz7sUBw44b/d1HSgURkZSGGwrPm9nD+FB4yMxKgePuwpZ9oaC9j0REUosMc7yPAAuBLc65TjOrwnchHVdCppaCiEg6w20pnAtscM41m9kNwD8BLZkrKzP6Q0GZICKS0nBD4btAp5mdDvw9sB34ScaqypBw8GnVfSQiktpwQyHu/AmDrgBuc87dBpRmrqzMUPeRiEh6w92m0GZmtwLvB95kZmEgmrmyMiOsI5pFRNIabkvhWqAHf7zCHmAq8PWMVZUh2vtIRCS9YYVCEAR3A+Vm9g6g2zl33G1TsP6zpCoURERSGe5pLq4BngPeA1wDPGtmV2eysEw4eJqLLBciIjJGDXebwj8CZznnGgDMrBZ4BPhVpgrLhP69j7RNQUQkpeFuUwj1BUKg6SieO2YcPE5BoSAikspwWwr/a2YPAT8P7l8LLM9MSZmjUBARSW+4G5o/C9wJLABOB+50zn3uSM8zs6VmtsHMNpnZLSke/5SZvWJmL5nZH81s5tF+gKOhXVJFRNIbbksB59z9wP3DHT84luF24EKgHlhhZsucc68MGO0FYHFwPqW/Bb6Gb4VkREgX2RERSSttKJhZG5BqCWqAc86VpXn6EmCTc25L8Fr34o+I7g8F59yjA8Z/BrhhmHWPiPY+EhFJL20oOOdez6kspgI7BtyvB85OM/5HgAdTPWBmNwI3AsyYMWPEBYWCzjK1FEREUsvkHkSpLsKTcmkcnHl1MUMcJe2cu9M5t9g5t7i2tnbEBencRyIi6Q17m8II1APTB9yfBuwaPJKZXYA/DuItzrmeDNaj01yIiBxBJlsKK4CTzKzOzPKA64BlA0cwszOAO4DLBx0HkREh095HIiLpZCwUnHNx4CbgIWAdcJ9zbq2ZfcXMLg9G+zpQAvzSzFab2bIhXu6YCGvvIxGRtDLZfYRzbjmDDnJzzn1hwO0LMvn+gwWZQFJ7H4mIpHTcnari9ejvPlJLQUQkpZwKhf7uI21TEBFJKSdDQS0FEZHUcioU+rcpKBNERFLKsVBQ95GISDo5FQo6S6qISHo5FQra+0hEJL2cCgXtfSQikl5OhcLBK69luRARkTEqt0Ih+LTqPhIRSS2nQiGsvY9ERNLKrVDQ3kciImnlVCiY6SypIiLp5FQo6NTZIiLp5VYo9F9kJ8uFiIiMUTkVCn17H6mlICKSWm6FgvY+EhFJK6dCIazTXIiIpJVToRDSaS5ERNLKqVAAvweSWgoiIqnlXiiYae8jEZEh5FwomIFTS0FEJKWcC4VwyHSaCxGRIeReKJi2KYiIDCXnQiEUMu19JCIyhNwLBdNFdkREhpJzoaBdUkVEhpZzoRAJhYjFtU+qiEgqORcK1SV5NLX3ZLsMEZExKedCYUJpPg1tCgURkVRyMBQKFAoiIkPIvVAoy6epvUcHsImIpJBzoVBbmk/SQVOHWgsiIoPlXChMKM0HoKFVoSAiMljOhUJtaQEAjdquICJymIyGgpktNbMNZrbJzG5J8fibzWyVmcXN7OpM1tKnv6XQ1j0abyciclzJWCiYWRi4HbgEmAe818zmDRrtNeBDwD2ZqmOwWnUfiYgMKZLB114CbHLObQEws3uBK4BX+kZwzm0LHhu1Q4wLomHKC6PaLVVEJIVMdh9NBXYMuF8fDDtqZnajma00s5WNjY2vuzB/AJu6j0REBstkKFiKYSM6OMA5d6dzbrFzbnFtbe3rLMsfq6CWgojI4TIZCvXA9AH3pwG7Mvh+wzahtEDbFEREUshkKKwATjKzOjPLA64DlmXw/YZtQmk+jW09ulaziMggGQsF51wcuAl4CFgH3OecW2tmXzGzywHM7CwzqwfeA9xhZmszVc9AtaX5xBJJWrvio/F2IiLHjUzufYRzbjmwfNCwLwy4vQLfrTQ64jGI5DGp3B/AtuNAJ+VF5aP29iIiY13uHNH85G3wL5MgHmPh9AoAVm7bn+WiRETGltwJhaIacAlorWdaZRFTKwp5dqtCQURkoNwJhYpgR6hmf+jE2bOreHbrfm1sFhEZIHdCoTwIhRYfCufUVbO/I8bGhvYsFiUiMrbkTiiUTQUMml8DfEsB4NktTVksSkRkbMmdUIjkQenk/u6jGVVFTC4v4GmFgohIv9wJBYCKGf3dR2bG+XNqeXxDI929iSwXJiIyNuRYKEzv7z4CuPS0yXTEEjy24fWfZE9EZDzIrVAonw6tOyHpWwbnzq6msijK8pd3Z7kwEZGxIbdCoWI6JOPQ5kMgEg6xdP4kHlm3V11IIiLkXCjM8P+bD17m4R0LptAZS/DDJ7dlpyYRkTEkt0KhPAiFloOh8IYTqrn0tEn8+8MbeH67jnAWkdyWY6EQnHtvwMZmM+Or717AtMpCbrrnBfZ3xLJUnIhI9uVWKOQVQXEtHNh6yOCygii3X7+IpvYYn75vNcmkTn0hIrkpt0IBYNIC2PXiYYPnTy3n8++Yy6MbGvnVqvosFCYikn25FwpTz4SGVyDWcdhDN5wzkzkTS/nZM9uzUJiISPblZii4BOx+6bCHzIz3LpnOS/UtrNnZkoXiRESyKwdDYZH/v/P5lA9fdcY08iMh7l3xWsrHRUTGs9wLhZIJftfUIUKhvCjKZQsm84sVO/jn371CQ1v3KBc4CmId8MLdoGtJiMggGb1G85g19YwhQwHgny6bR9iMHzy5lR8+tY23zpnAF985j+lVRaNYZAa9+HP4/aeh+kSYcXa2qxGRMST3Wgrgtys0b4e2PSkfrirO4+vvOZ0/fuotfPTNs3l2SxNLv/UED46XcyTVr/T/d63Kbh0iMubkZiicfAlYCJ76TtrRZteW8PdLT+HBT76JkyeVcvO9q1mxbRwc9dzXStqpUBCRQ+VmKNSeDAuvh+fuhD1rYO9aSCaHHH1aZRE/+tASplUW8tc/Xsmtv36JJ149Tk+33dUM+171t9VSEJFBcjMUAM7/B99a+N558N03wIOfTbvhtbwoyg8/fBYLp1fw+5d284G7nuPLv13LT5/exqMbGkav7qPxl2/C998Gid6Dw/qCYNaboGmTDwkRGX0t9bDyh/DqwxDrPPL4o7RjSG5uaAYonwrvuhMa1vtTaa/4b3/sQvteCIX9OB374M2fgfNuBmBmdTE//qsl9HR38q/L1x1yZtVvXHM671o0LQsfZAjJBDx7h/9sL94Li97vh/d1HZ31Edj2Z9i9Gmafn60qR8eeNVA5E/JLs13J0UsmIXSM1t1iHRApgN5OiMeguPrgY4le6O2CgrLDnxfvgd99CibMhTfcBM/eCT0tcN4nIRw98vs6B51NUFh15M+yfwus+AGc+WGoOfHoPl8qnfv9ys/uF6GjESYvhOlLoKActj4OPW2+rl2roGUnRPJh8ukw6TT/G6o52V/Kd/BrRgr8aXPSaX4NXvmNX44UlPtpWzoZiqr9Ctur/3tw3GgxLHyvX1kLhaFxA9ScBHMvBzPY/Cd45Mtw4Vdg9lte/3RJI3dDAWDeFf7POb/A2PSI3wgdCoNLQutueORLMP1smHGOf04yQf7dV/Ll/Vv5+BVfJjbnCj77y5d46tf/iW2ey8nzzqBg7ypq576RssnBTO0c7HkJ2vb6Gb1q9sEanPNfejrxHnjtaahfAae8E2rn+FonzD14kj/n4MA2P9NFC/wCv2035JXCE1+DBdf6mbv+eT+j1wUz1s5VMPM8P/Ou+y00rINFH4BzP+7revYOeOLf/QLllMvg8u9A137/fuVTD691OJ9nuLY96cM6kg8VM6H6BN/lt+9VeMstsORv/EKpby0rrwh6u/37R/L9AvWxf/Ofv2QiLHyfP5q9oML/8Nv3+mnW2wkX/bOfruA/a6wTSmr9/Z42v8LQUu+vx3HqVf69nIP2Blj7ALz0C6h7Eyz+iP+uCipg8gJfa+d+X+ek06Cw0r9npACiRdDdDM9812/8X3ANhCK+O7PuTT7AV/3ELyhOvMDXGy2CaCF0t/jXn3OZ/7779HbB2v+BjQ/DvMvh5KWw4UF4+nbYufLQ6TtlEcy5xNf0l29C6y6YOB9OvtjPIztXQuUs2PGsnz/Af7b1v/O31/0OzvxQsCD/b5j1Rj9tnv0etDdCXjGUTfHnGmt+zdc+8zwfLHVv8d9T46vw/I+gcT3MfQc8/nVo2+XnuwXXwLTF/vP0dsEbPuG/m3i3/y4HLqwPbId1y/y0btvtg6Bps59XU4kW+e99oIIK/9rxAbuhV9bBBV/02yE3/xGevA12PAdFVXDp1+Gki2HbX+DJb/mehwlzYf67YfU98MJP/WuEopDsPfS98svg/Fth3pX+wl8v/8p/1yv++9Dxpi3x81/jOr8rfbwn9ec5hswdZ/uqL1682K1cufLIIx4L3a3wvTf6BcF5n/QBsuZX8NA/+B/LgW1w5ofpjpZR8Mxthzx1nZvJj079IReeOoXzt32TyIo7/AORQrj+Fz6EVv4A1vzaLyze+Cn/A2jaDPXP+YVG536/BrP1z9AbnJYjWgzTz4Itj0HJJPjA//gZ7MG/9z/WUMQvCAC2PgFX/hf84ga/hlRYAVse962Gy78Dty30e2BFC/2Pp2yqX3juWgV1b/aBsuH3/nb5DFj9Mx9oB4LTgJx2tV+g9Hb5IN3+tF8Ilk/zC5fak/21K8JRmDAPJs7zofPCz/zz+oZNnO/v16/0/8umwGNfhVcf9GtVkUK/oHBJH3rVJ/rQK6yCiaf6H2ki5k922NEI4Ty/0G+p98877T2wf6tfyFWfCF0H/JprOM9fY6OjEYonwA33w+q7ffB0t8KSG/3z1/3Wv3efyjqYca5f8Hbu88NqT/ELtpEIB2un9c/5+4VV/vsIRfxCY/uTfkEXKQgWCg4s7I/ML6jw30PNHD/tX77PB0ZeKcTa/GdMxHzNC671C+JokR+2bllwZL/z88fJF/sgfu0p/3kHLjjf/kU/P2151Nd06lXwv7cEF6wyP89tfcLPpzUn+7XxnjYfNMW1fqWqdZef3zsa/HxUMtGHTCji5+XWeiiqgavv8kG75n7oafVXTAxFDj2R5cTT/HzW8ArklfiFpksGrzXRr0BUneC/7+oTgkCu8i3j+hU+pE68wL9Ge4N/vGSCbx3secn/DhMxHwKN6/3rJuP+Nee/Gzb9AXa9cLCevs+z+0U/zSwM5/wtnPXXUFXnV1a6W/xp+/dvhRPeCsU1h84HXc0+IBIx/3ov3QdP/6e/PedSv7IWyR/ZPAaY2fPOucVHHE+hcAS7XoD/+T9+5rOQ/7JPeBtcdw/86St+pgFY9EGapl9Aw/YNREKOk1b9Cz9ylzEh2cil4efYOvt6HrE3sHT715kaf40Qzv/oTnmH/9G37kzx5uZn6Lq3wEkX+h/bAx/zNZ33d/4AtPZgt9pwng+u3k6/UEvE4Ixg4f/kbbBhuV8Qzr/az6yFFbDpj34tMtbuf+gnXeQXGk/e5tdaulv8BvkLvuQX+i/d55uwp17pfzyrfux/LNFCv7CavMAvXFrq/Y/vwDb/o4vHDtYJfoHqnJ+mPa2pp3t+Gbzx//O1Rgv9AqZhvQ+BaKFvKb10X7BW/Wb/eVp2+AVIT5sPmIrp/jOd9h7/mj1tvgnvnG/SF1X5z7X1CfjJFQcX/HMu8+G0+me+jkUf8O9RfaLflfn3n/HPP/li37KcvsQfKb/jOdj+lO+O627xAVg7B0on+eDc+byf1lWzg+6aTv+91b3ZB+H+LT4gyqb4BVN+mV+gJHqDbo5KX2Mi5p+39QkfYut+69duw3kw952+62XGufDCT/w0m3OJf4++btGBulv8AnLCqQe7dtob/cJ+wjz/eZu3+3m+pw3W/x5OfZdfS3fOLzxDIf+ZWup9t0fdWyA8RCdEbzes/bVfk+7YB6df69f6i6p9F0n1if4zg59vmjb6wMPBxj/41mnzDnjwc35lY8oZfjrWnuIXwOXTjl1LFfy03/KY/6s+0f+mwhFIxP1K2IGtvvbT3+vr6W7xv6mJp/qgGUMUCsdawzq/lrNrFbzztoPdNqt+6tfQLvjSoT+6e98H639HIpzPD0LX8K9tS8mLhDlvsuPyhjvYVXQK1//NZ7n3pRZKI3HeFF1PUetmiqumUnTK2w+uqeWXHFpHMuHXKIqr/RrH6rv9GtCJF/g1c/ALnz/9i+8SmTgvc9PkSF1FA/vDO5qgYe3BbpW+57fsgL0NY6/3AAAQP0lEQVSv+MCaeqbvImnc4NfGBvZ5Z9qqn/jpdvbfwoRT/LD9W/2CuLDi0HGdC9ZKUyxks6Gn3YdOYeXQC2PJeQqFbOto8kcOn3Y1Xfm1PP5qI4tnVVJTks+jGxr48A9XUJQXpjN2+LWhJ5cX8JaTazn3hGrmTCplUlkB5YVR7FiuAYlITlEojHH//tAGfvvSLv7tqtOYUlHI2l2tJJxjd3MXL9Y388Sr+2jvifePX1EU5Q0nVHNibQllhVFau3rZcaCLfe09TKss5PRpFbx97kRqS0fe5ygi45dC4TgXiyfZ3NjOxoZ2Glq7Wb+njac3N7G7pYtk0GszuayAmtJ86g90sb8jhhmcOaOShdMrKMwLU1GUR01JHuWFUdbuaqWxrYdrFk+nqjiPnc1dnDa1nHDI2LqvA3DUlvoWiYiMPwqFcSqeSNIRS1CSHyEc8t1JzjnW72nj4bV7efiVPWxp7KA7njjsWJe8SIhY/OBeNGUFEUIho7nT7y4XDhnnzq5m6fxJlORH+Plzr1FdkseSWVV0xBKcNKGEt8+dSDhk9CaSvLyzhTkTSwmZ8ci6vZw9u4oJpQWIyNijUMhxyaSjpauXfe097O+IMbu2hLxwiPuDS41OKi/gsQ0NJB2cXVdFfjTM+t2t/O+aPWzZ53d/rasppqMnTkPbwX2jp1YUMru2mHW729jX3kNhNExeJERLVy8TSvP59EUns253G5GQMaummEjI+PPGffx5YyNvO2UCS+qq2d3SxYkTSpheVcRrTZ00dcQozY/wrkVTWbOrlTse38wJtSWcP6eWRTMqebG+mfoDXcyoKmLu5DLyIiG6exP09CYJh43ivHD/9pauWIJ4MklpgW/xdPcmeGrzPuZPKWdCmQJLcpdCQUbEOcere9tp6ujhnDq/909jew+lBRH+tL6B36zeRUNbDxNL81k6fxLPbz9Ae0+ct8+dyH88vIHtTZ0URsMknaMnaJVUFEU574QaHtvQQEeKDet9zpxZyfrdrYRCRmcsQSLpDtsYX1oQYVZ1Met2txJP+nm3KC/MxLICivLCbNjTRjzpmF5VSEVhHtubOmjtjlNaEOH6s2ewtbGDrl7f0irKi9DcGWNPazdvO2UCly2YTG1JPpsbO9jd0kVlUR5VxXkURMPsbumi/kAXe1u7iSccDkfSQdiMiqIocyaV0tObpLM3wZTyAnoTjn3tPexr76G6JJ/FMytJJB3Nnb20dfcyuaKQyeU+pPIjfg+t1u44RXlhouH0R/12xuI8sq6BSWUFLJ5ZSSh0+A4Izjm2N3WScI4TaktSvMrIOOe0w8NxakyEgpktBW4DwsB/O+e+OujxfOAnwJlAE3Ctc25butdUKIxd7T1xXt3bxqlTyoiEQjS29ZBwjpqSPPIjYdp74hzoiDGpvID1u9vY29rNrJoiaksK+MO6vfzjAy8zvaqIu//6bIrywvxxXQNPbd7H4plVzJ9azramDh7b0MBr+zs5Y0YltSX5xBJJGtt62NPaTWtXL/OnllOcF2b9njbae+JUFeVx0amT+OGTW3l2637qaoqpKIrS0ROno8eHQ3lRlBXb9g/71DIhg5AZZpB0kEi+vt9QOGREQkZPPEl+JMSs6mIa2rrpjCXIi4TIj4TJj4SC2yF2NnfR1u13QigriOCA/EiYmpI8akry6Ykn2NTQzoGgW3DRjAoioRD72nuYWFZASYHfbbWtu5eaknzOnFlJR0+c3S3dNLT1UBAN0xVLsHZXCydOKOGMGZU8tWkfmxvbae+Jc8qkMs6cWcm8KWXsaemmqzfBnImlhENGZyxOW3ec9Xva2NLYTl4kxJSKQuZOKqOsMEJB1IfevvYeunsTTCwrYF97jIa2boqiEVq7e2nu7GX+1DIKo2G27+8kLxwiPxoimXQU50dIOti4t43u3gSRcIhIyJheVcSiGZXU1RRTWRyluzfJn9bvpak9xnkn1lBdkkdPb5KGtm62NHawrz3GkrpKJpQWsD+YJwE2N7T3d7O+sruVaZWFLJhWwf6OGLFEkryw/x6i/f+NvHCov45E0tHY1kN+NERBJBy0uh2TywuZUOrn1y2NHZQXRplcXkAkHMI5R2t3nJ54glg8iXNQW5pPQTRMLJ5ke1MH7T1xqovzmVDmh49E1kPBzMLAq8CFQD2wAnivc+6VAeP8H2CBc+5jZnYdcJVz7tp0r6tQGL/2tHRTVujX4I815xydsQTF+alfe8f+Tla9doB97TFmVRcxo6qI5q5emtpjdPcmmFxewLSqIiaW5hMZtCbf1N7Dhr1tFOdFKMwLs6u5i7xwiJrSfKqL86g/4PcoK4z6jf/FeWHqm7toDLrlOmNxYvEkE0oL2NvazdZ9HUwqL6AkP0JPPEkskaSnt+9/gvLCKO9aNI2Gtm6e3bqfvHCInniCfe0x9rX3EA2FqKsp5vTpFXTG4ty/amfQmspnb2sPnbEEzjlKCyK8tr+Tva2+jqriPCaU5tMTTxIymDu5jBfrm9mx3++UcPr0cgqjYdbsbGX1jma6ehOY+dZSfFAwVhZFOWVSGYmkY2tTR/9nHUo0bPQmHIXRMMX5Efa1+/EjocNfu6/WkvwI8USSWNAqG+vCIcM538Lsuz+xNJ+Wrt6ULehQsNIx0FeuOJUPnDtrRO8/FkLhXOBLzrmLg/u3Ajjn/m3AOA8F4zxtZhFgD1Dr0hSlUBA5dpzza7ZlhdGUa6B9a7GD90qLJ5K8tr+TiWUFRMMhtjV1YEBRfoTivPBhx9Uc6IjREYvT3ZskFk/61mM0TENrN5XFeVQX55FIOsIhw8zY1dxFPOGYVlmIA3oTSUJmtPfEcc5RXXLortcHOmK8tLOF1/Z30trlW0jnzK5iSkUhT21qorM3QX4kRG1JPjOri6goyuPpzU109MSpLM7ze/UlHSdNLCWedIQMTplUxtZ9HWzY20ZtST4FUd+C6E04ehP+c8QSSXoTSeIJRyyRJBwyakt8sHbG4tTVFBMK+c+zq7mLSCjEnEmltHb1Un/ADysvijK1opD8aLi/K7GxrYeuWIJoOMTM6iLKCiM0tcc4Y0YlJ04YWXfgWAiFq4Glzrm/Du6/HzjbOXfTgHHWBOPUB/c3B+PsG/RaNwI3AsyYMePM7du3Z6RmEZHxarihkMnrKaTaGjU4gYYzDs65O51zi51zi2tra49JcSIicrhMhkI9MH3A/WnArqHGCbqPyoFxcL1LEZHjUyZDYQVwkpnVmVkecB2wbNA4y4APBrevBv6UbnuCiIhkVsZOqeici5vZTcBD+F1S73LOrTWzrwArnXPLgB8APzWzTfgWwnWZqkdERI4so+fZdc4tB5YPGvaFAbe7gfdksgYRERm+THYfiYjIcUahICIi/RQKIiLS77g7IZ6ZNQIjPXqtBth3xLGyQ7WNjGobGdU2MsdzbTOdc0c80Ou4C4XXw8xWDueIvmxQbSOj2kZGtY1MLtSm7iMREemnUBARkX65Fgp3ZruANFTbyKi2kVFtIzPua8upbQoiIpJerrUUREQkDYWCiIj0y5lQMLOlZrbBzDaZ2S1ZrmW6mT1qZuvMbK2Z3RwM/5KZ7TSz1cHfpVmqb5uZvRzUsDIYVmVmfzCzjcH/yizUNWfAtFltZq1m9slsTTczu8vMGoKLRfUNSzmdzPt2MP+9ZGaLslDb181sffD+D5hZRTB8lpl1DZh+38tCbUN+h2Z2azDdNpjZxVmo7RcD6tpmZquD4aM93YZabhzbec45N+7/8Gdp3QzMBvKAF4F5WaxnMrAouF2Kv5b1POBLwGfGwPTaBtQMGvY14Jbg9i3A/x0D3+keYGa2phvwZmARsOZI0wm4FHgQf2Gpc4Bns1DbRUAkuP1/B9Q2a+B4WZpuKb/D4HfxIpAP1AW/4/Bo1jbo8f8AvpCl6TbUcuOYznO50lJYAmxyzm1xzsWAe4ErslWMc263c25VcLsNWAdMzVY9w3QF8OPg9o+BK7NYC8Dbgc3Ouaxdm9U59wSHXxRqqOl0BfAT5z0DVJjZ5NGszTn3sHMuHtx9Bn/hq1E3xHQbyhXAvc65HufcVmAT/vc86rWZmQHXAD/P1Punk2a5cUznuVwJhanAjgH36xkjC2EzmwWcATwbDLopaOrdlY0umoADHjaz581fHxtgonNuN/iZE5iQpdr6XMehP86xMN1g6Ok01ubBv8KvRfapM7MXzOxxM3tTlmpK9R2Open2JmCvc27jgGFZmW6DlhvHdJ7LlVAY1rWgR5uZlQD3A590zrUC3wVOABYCu/FN1Ww4zzm3CLgE+LiZvTlLdaRk/kp+lwO/DAaNlemWzpiZB83sH4E4cHcwaDcwwzl3BvAp4B4zKxvlsob6DsfMdAPey6ErIlmZbimWG0OOmmLYEaddroTCcK4XParMLIr/Yu92zv0awDm31zmXcM4lge+TwWZyOs65XcH/BuCBoI69fU3P4H9DNmoLXAKscs7thbEz3QJDTacxMQ+a2QeBdwDvc0HHc9A10xTcfh7fb3/yaNaV5jscK9MtArwL+EXfsGxMt1TLDY7xPJcroTCc60WPmqBv8gfAOufcNwYMH9jfdxWwZvBzR6G2YjMr7buN3zi5hkOvp/1B4DejXdsAh6yxjYXpNsBQ02kZ8IFgj5BzgJa+Jv9oMbOlwOeAy51znQOG15pZOLg9GzgJ2DLKtQ31HS4DrjOzfDOrC2p7bjRrC1wArHfO1fcNGO3pNtRyg2M9z43WlvNs/+G3xL+KT/N/zHItb8Q3414CVgd/lwI/BV4Ohi8DJmehttn4vT1eBNb2TSugGvgjsDH4X5WlaVcENAHlA4ZlZbrhg2k30ItfK/vIUNMJ35S/PZj/XgYWZ6G2Tfg+5r557nvBuO8OvusXgVXAO7NQ25DfIfCPwXTbAFwy2rUFw38EfGzQuKM93YZabhzTeU6nuRARkX650n0kIiLDoFAQEZF+CgUREemnUBARkX4KBRER6adQEBlFZna+mf0u23WIDEWhICIi/RQKIimY2Q1m9lxwnvw7zCxsZu1m9h9mtsrM/mhmtcG4C83sGTt4nYK+89mfaGaPmNmLwXNOCF6+xMx+Zf7aBncHR6qKjAkKBZFBzGwucC3+xIALgQTwPqAYf86lRcDjwBeDp/wE+JxzbgH+yNG+4XcDtzvnTgfegD9SFvzZLT+JPxf+bOC8jH8okWGKZLsAkTHo7cCZwIpgJb4Qf5KxJAdPiPYz4NdmVg5UOOceD4b/GPhlcP6oqc65BwCcc90Awes954Jz6Ji/itcs4C+Z/1giR6ZQEDmcAT92zt16yECzzw8aL905YtJ1CfUMuJ1Av0MZQ9R9JHK4PwJXm9kE6L8G7kz87+XqYJzrgb8451qAAwMusPJ+4HHnz3Nfb2ZXBq+Rb2ZFo/opREZAaygigzjnXjGzf8JffS6EP2Pmx4EO4FQzex5owW93AH+64u8FC/0twIeD4e8H7jCzrwSv8Z5R/BgiI6KzpIoMk5m1O+dKsl2HSCap+0hERPqppSAiIv3UUhARkX4KBRER6adQEBGRfgoFERHpp1AQEZF+/w/ynxVy2coIggAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# summarize history for loss\n",
    "plt.plot(history.history['loss'])\n",
    "plt.plot(history.history['val_loss'])\n",
    "plt.title('model loss')\n",
    "plt.ylabel('loss')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'test'], loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preprocess letters and mergere with training set, retrain (iteration 12)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(60000, 28, 28) (60000,)\n",
      "(60000, 1, 28, 28)\n"
     ]
    }
   ],
   "source": [
    "# the data, split between train and test sets\n",
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
    "print(x_train.shape, y_train.shape)\n",
    "# input image dimensions\n",
    "img_rows, img_cols = 28, 28\n",
    "x_train = x_train.reshape(x_train.shape[0], 1, img_rows, img_cols)\n",
    "print(x_train.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\CampusUser\\Anaconda3\\envs\\Keras\\lib\\site-packages\\keras_preprocessing\\image\\numpy_array_iterator.py:127: UserWarning: NumpyArrayIterator is set to use the data format convention \"channels_last\" (channels on axis 3), i.e. expected either 1, 3, or 4 channels on axis 3. However, it was passed an array with shape (60000, 1, 28, 28) (28 channels).\n",
      "  str(self.x.shape[channels_axis]) + ' channels).')\n"
     ]
    }
   ],
   "source": [
    "from keras.preprocessing.image import ImageDataGenerator, array_to_img, img_to_array, load_img\n",
    "\n",
    "x_preprocessed = []\n",
    "y_preprocessed = []\n",
    "datagen = ImageDataGenerator(\n",
    "        rotation_range=40,\n",
    "        width_shift_range=0.2,\n",
    "        height_shift_range=0.2,\n",
    "        shear_range=0.2,\n",
    "        zoom_range=0.2,\n",
    "        horizontal_flip=True,\n",
    "        fill_mode='nearest'\n",
    "        )\n",
    "\n",
    "# img = load_img('data/train/cats/cat.0.jpg')  # this is a PIL image\n",
    "# x = img_to_array(img)  # this is a Numpy array with shape (3, 150, 150)\n",
    "# x = x.reshape((1,) + x.shape)  # this is a Numpy array with shape (1, 3, 150, 150)\n",
    "\n",
    "# the .flow() command below generates batches of randomly transformed images\n",
    "# and saves the results to the `preprocessed/` directory\n",
    "i = 0\n",
    "for x,y in datagen.flow(x_train,y=y_train, batch_size=1):\n",
    "    x_preprocessed.append(x)\n",
    "    y_preprocessed.append(y)\n",
    "    i += 1\n",
    "    if i > 10000:\n",
    "        break  # otherwise the generator would loop indefinitely"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(10001, 1, 1, 28, 28) (10001, 1)\n",
      "(10001, 28, 28) (10001,)\n",
      "(70001, 28, 28) (70001,)\n"
     ]
    }
   ],
   "source": [
    "x_preprocessed = np.array(x_preprocessed)\n",
    "y_preprocessed = np.array(y_preprocessed)\n",
    "print(x_preprocessed.shape,y_preprocessed.shape)\n",
    "\n",
    "x_preprocessed = x_preprocessed.reshape(x_preprocessed.shape[0], img_rows, img_cols)\n",
    "y_preprocessed = y_preprocessed.reshape(y_preprocessed.shape[0])\n",
    "print(x_preprocessed.shape,y_preprocessed.shape)\n",
    "\n",
    "x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols)\n",
    "\n",
    "x_train = np.concatenate((x_train,x_preprocessed),axis=0)\n",
    "y_train = np.concatenate((y_train,y_preprocessed),axis=0)\n",
    "print(x_train.shape,y_train.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(60200, 28, 28) (60200,) (9801, 28, 28) (9801,) (10000, 28, 28) (10000,)\n"
     ]
    }
   ],
   "source": [
    "# the data, split between train and validation sets\n",
    "x_train, x_valid, y_train, y_valid = train_test_split(x_train, y_train, test_size = 0.14, random_state = 0, shuffle=True)\n",
    "# x_train, y_train = shuffle(x_train, y_train)\n",
    "print(x_train.shape,y_train.shape,x_valid.shape,y_valid.shape,x_test.shape,y_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 249 #784\n",
    "num_classes = 10\n",
    "epochs = 281"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x_train shape: (60200, 28, 28, 1)\n",
      "60200 train samples\n",
      "60200 train samples\n",
      "10000 test samples\n",
      "Train on 60200 samples, validate on 9801 samples\n",
      "Epoch 1/281\n",
      "60200/60200 [==============================] - 136s 2ms/step - loss: 0.4738 - acc: 0.8534 - val_loss: 0.1954 - val_acc: 0.9431\n",
      "Epoch 2/281\n",
      "60200/60200 [==============================] - 129s 2ms/step - loss: 0.1999 - acc: 0.9360 - val_loss: 0.1212 - val_acc: 0.9626\n",
      "Epoch 3/281\n",
      "60200/60200 [==============================] - 97s 2ms/step - loss: 0.1486 - acc: 0.9525 - val_loss: 0.0890 - val_acc: 0.9727\n",
      "Epoch 4/281\n",
      "60200/60200 [==============================] - 92s 2ms/step - loss: 0.1217 - acc: 0.9615 - val_loss: 0.0758 - val_acc: 0.9765\n",
      "Epoch 5/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.1058 - acc: 0.9665 - val_loss: 0.0622 - val_acc: 0.9802\n",
      "Epoch 6/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0905 - acc: 0.9714 - val_loss: 0.0672 - val_acc: 0.9780\n",
      "Epoch 7/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0828 - acc: 0.9735 - val_loss: 0.0738 - val_acc: 0.9784\n",
      "Epoch 8/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0766 - acc: 0.9760 - val_loss: 0.0542 - val_acc: 0.9828\n",
      "Epoch 9/281\n",
      "60200/60200 [==============================] - 86s 1ms/step - loss: 0.0708 - acc: 0.9779 - val_loss: 0.0553 - val_acc: 0.9833\n",
      "Epoch 10/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0650 - acc: 0.9791 - val_loss: 0.0462 - val_acc: 0.9848\n",
      "Epoch 11/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0637 - acc: 0.9791 - val_loss: 0.0440 - val_acc: 0.9865\n",
      "Epoch 12/281\n",
      "60200/60200 [==============================] - 90s 1ms/step - loss: 0.0577 - acc: 0.9814 - val_loss: 0.0447 - val_acc: 0.9860\n",
      "Epoch 13/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0573 - acc: 0.9819 - val_loss: 0.0465 - val_acc: 0.9852\n",
      "Epoch 14/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0511 - acc: 0.9828 - val_loss: 0.0467 - val_acc: 0.9875\n",
      "Epoch 15/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0510 - acc: 0.9837 - val_loss: 0.0463 - val_acc: 0.9859\n",
      "Epoch 16/281\n",
      "60200/60200 [==============================] - 86s 1ms/step - loss: 0.0465 - acc: 0.9849 - val_loss: 0.0468 - val_acc: 0.9857\n",
      "Epoch 17/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0459 - acc: 0.9852 - val_loss: 0.0478 - val_acc: 0.9862\n",
      "Epoch 18/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0419 - acc: 0.9862 - val_loss: 0.0413 - val_acc: 0.9881\n",
      "Epoch 19/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0421 - acc: 0.9862 - val_loss: 0.0431 - val_acc: 0.9862\n",
      "Epoch 20/281\n",
      "60200/60200 [==============================] - 86s 1ms/step - loss: 0.0391 - acc: 0.9869 - val_loss: 0.0419 - val_acc: 0.9866\n",
      "Epoch 21/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0402 - acc: 0.9872 - val_loss: 0.0430 - val_acc: 0.9882\n",
      "Epoch 22/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0381 - acc: 0.9875 - val_loss: 0.0443 - val_acc: 0.9857\n",
      "Epoch 23/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0356 - acc: 0.9882 - val_loss: 0.0392 - val_acc: 0.9883\n",
      "Epoch 24/281\n",
      "60200/60200 [==============================] - 86s 1ms/step - loss: 0.0341 - acc: 0.9890 - val_loss: 0.0562 - val_acc: 0.9846\n",
      "Epoch 25/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0347 - acc: 0.9885 - val_loss: 0.0415 - val_acc: 0.9876\n",
      "Epoch 26/281\n",
      "60200/60200 [==============================] - 90s 1ms/step - loss: 0.0325 - acc: 0.9895 - val_loss: 0.0363 - val_acc: 0.9897\n",
      "Epoch 27/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0325 - acc: 0.9895 - val_loss: 0.0389 - val_acc: 0.9891\n",
      "Epoch 28/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0308 - acc: 0.9898 - val_loss: 0.0443 - val_acc: 0.9878\n",
      "Epoch 29/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0299 - acc: 0.9900 - val_loss: 0.0406 - val_acc: 0.9886\n",
      "Epoch 30/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0293 - acc: 0.9905 - val_loss: 0.0464 - val_acc: 0.9872\n",
      "Epoch 31/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0287 - acc: 0.9902 - val_loss: 0.0376 - val_acc: 0.9898\n",
      "Epoch 32/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0268 - acc: 0.9911 - val_loss: 0.0380 - val_acc: 0.9897\n",
      "Epoch 33/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0273 - acc: 0.9910 - val_loss: 0.0466 - val_acc: 0.9879\n",
      "Epoch 34/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0264 - acc: 0.9912 - val_loss: 0.0467 - val_acc: 0.9881\n",
      "Epoch 35/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0251 - acc: 0.9917 - val_loss: 0.0450 - val_acc: 0.9878\n",
      "Epoch 36/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0236 - acc: 0.9916 - val_loss: 0.0451 - val_acc: 0.9883\n",
      "Epoch 37/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0237 - acc: 0.9916 - val_loss: 0.0430 - val_acc: 0.9882\n",
      "Epoch 38/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0233 - acc: 0.9925 - val_loss: 0.0464 - val_acc: 0.9881\n",
      "Epoch 39/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0230 - acc: 0.9919 - val_loss: 0.0418 - val_acc: 0.9895\n",
      "Epoch 40/281\n",
      "60200/60200 [==============================] - 89s 1ms/step - loss: 0.0228 - acc: 0.9924 - val_loss: 0.0403 - val_acc: 0.9881\n",
      "Epoch 41/281\n",
      "60200/60200 [==============================] - 89s 1ms/step - loss: 0.0215 - acc: 0.9928 - val_loss: 0.0412 - val_acc: 0.9884\n",
      "Epoch 42/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0202 - acc: 0.9935 - val_loss: 0.0409 - val_acc: 0.9889\n",
      "Epoch 43/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0201 - acc: 0.9930 - val_loss: 0.0405 - val_acc: 0.9886\n",
      "Epoch 44/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0208 - acc: 0.9934 - val_loss: 0.0529 - val_acc: 0.9852\n",
      "Epoch 45/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0198 - acc: 0.9931 - val_loss: 0.0439 - val_acc: 0.9889\n",
      "Epoch 46/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0203 - acc: 0.9933 - val_loss: 0.0417 - val_acc: 0.9894\n",
      "Epoch 47/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0195 - acc: 0.9930 - val_loss: 0.0418 - val_acc: 0.9893\n",
      "Epoch 48/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0212 - acc: 0.9928 - val_loss: 0.0405 - val_acc: 0.9895\n",
      "Epoch 49/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0184 - acc: 0.9937 - val_loss: 0.0431 - val_acc: 0.9896\n",
      "Epoch 50/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0178 - acc: 0.9941 - val_loss: 0.0449 - val_acc: 0.9890\n",
      "Epoch 51/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0177 - acc: 0.9944 - val_loss: 0.0420 - val_acc: 0.9896\n",
      "Epoch 52/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0192 - acc: 0.9939 - val_loss: 0.0365 - val_acc: 0.9903\n",
      "Epoch 53/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0171 - acc: 0.9943 - val_loss: 0.0422 - val_acc: 0.9890\n",
      "Epoch 54/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0163 - acc: 0.9950 - val_loss: 0.0425 - val_acc: 0.9891\n",
      "Epoch 55/281\n",
      "60200/60200 [==============================] - 91s 2ms/step - loss: 0.0152 - acc: 0.9952 - val_loss: 0.0405 - val_acc: 0.9893\n",
      "Epoch 56/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0173 - acc: 0.9944 - val_loss: 0.0423 - val_acc: 0.9902\n",
      "Epoch 57/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0159 - acc: 0.9948 - val_loss: 0.0429 - val_acc: 0.9893\n",
      "Epoch 58/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0183 - acc: 0.9942 - val_loss: 0.0492 - val_acc: 0.9882\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 59/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0144 - acc: 0.9950 - val_loss: 0.0409 - val_acc: 0.9898\n",
      "Epoch 60/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0162 - acc: 0.9949 - val_loss: 0.0469 - val_acc: 0.9889\n",
      "Epoch 61/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0153 - acc: 0.9948 - val_loss: 0.0394 - val_acc: 0.9900\n",
      "Epoch 62/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0149 - acc: 0.9951 - val_loss: 0.0428 - val_acc: 0.9890\n",
      "Epoch 63/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0155 - acc: 0.9947 - val_loss: 0.0488 - val_acc: 0.9883\n",
      "Epoch 64/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0145 - acc: 0.9950 - val_loss: 0.0428 - val_acc: 0.9894\n",
      "Epoch 65/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0131 - acc: 0.9955 - val_loss: 0.0401 - val_acc: 0.9900\n",
      "Epoch 66/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0147 - acc: 0.9953 - val_loss: 0.0435 - val_acc: 0.9898\n",
      "Epoch 67/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0140 - acc: 0.9953 - val_loss: 0.0428 - val_acc: 0.9898\n",
      "Epoch 68/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0145 - acc: 0.9952 - val_loss: 0.0391 - val_acc: 0.9904\n",
      "Epoch 69/281\n",
      "60200/60200 [==============================] - 91s 2ms/step - loss: 0.0142 - acc: 0.9950 - val_loss: 0.0493 - val_acc: 0.9886\n",
      "Epoch 70/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0137 - acc: 0.9955 - val_loss: 0.0394 - val_acc: 0.9909\n",
      "Epoch 71/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0135 - acc: 0.9953 - val_loss: 0.0451 - val_acc: 0.9892\n",
      "Epoch 72/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0140 - acc: 0.9950 - val_loss: 0.0416 - val_acc: 0.9901\n",
      "Epoch 73/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0126 - acc: 0.9960 - val_loss: 0.0430 - val_acc: 0.9895\n",
      "Epoch 74/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0127 - acc: 0.9958 - val_loss: 0.0483 - val_acc: 0.9882\n",
      "Epoch 75/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0120 - acc: 0.9960 - val_loss: 0.0458 - val_acc: 0.9893\n",
      "Epoch 76/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0142 - acc: 0.9953 - val_loss: 0.0431 - val_acc: 0.9897\n",
      "Epoch 77/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0111 - acc: 0.9961 - val_loss: 0.0428 - val_acc: 0.9899\n",
      "Epoch 78/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0113 - acc: 0.9959 - val_loss: 0.0436 - val_acc: 0.9891\n",
      "Epoch 79/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0110 - acc: 0.9962 - val_loss: 0.0440 - val_acc: 0.9898\n",
      "Epoch 80/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0132 - acc: 0.9956 - val_loss: 0.0405 - val_acc: 0.9909\n",
      "Epoch 81/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0107 - acc: 0.9963 - val_loss: 0.0414 - val_acc: 0.9907\n",
      "Epoch 82/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0123 - acc: 0.9959 - val_loss: 0.0436 - val_acc: 0.9896\n",
      "Epoch 83/281\n",
      "60200/60200 [==============================] - 90s 1ms/step - loss: 0.0112 - acc: 0.9963 - val_loss: 0.0435 - val_acc: 0.9906\n",
      "Epoch 84/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0113 - acc: 0.9962 - val_loss: 0.0485 - val_acc: 0.9893\n",
      "Epoch 85/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0108 - acc: 0.9962 - val_loss: 0.0441 - val_acc: 0.9901\n",
      "Epoch 86/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0112 - acc: 0.9963 - val_loss: 0.0433 - val_acc: 0.9903\n",
      "Epoch 87/281\n",
      "60200/60200 [==============================] - 86s 1ms/step - loss: 0.0103 - acc: 0.9968 - val_loss: 0.0483 - val_acc: 0.9899\n",
      "Epoch 88/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0104 - acc: 0.9964 - val_loss: 0.0476 - val_acc: 0.9897\n",
      "Epoch 89/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0106 - acc: 0.9963 - val_loss: 0.0437 - val_acc: 0.9909\n",
      "Epoch 90/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0106 - acc: 0.9965 - val_loss: 0.0507 - val_acc: 0.9889\n",
      "Epoch 91/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0109 - acc: 0.9963 - val_loss: 0.0518 - val_acc: 0.9889\n",
      "Epoch 92/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0105 - acc: 0.9963 - val_loss: 0.0425 - val_acc: 0.9902\n",
      "Epoch 93/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0103 - acc: 0.9963 - val_loss: 0.0431 - val_acc: 0.9902\n",
      "Epoch 94/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0094 - acc: 0.9968 - val_loss: 0.0478 - val_acc: 0.9896\n",
      "Epoch 95/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0104 - acc: 0.9966 - val_loss: 0.0459 - val_acc: 0.9900\n",
      "Epoch 96/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0097 - acc: 0.9967 - val_loss: 0.0446 - val_acc: 0.9901\n",
      "Epoch 97/281\n",
      "60200/60200 [==============================] - 90s 1ms/step - loss: 0.0096 - acc: 0.9967 - val_loss: 0.0406 - val_acc: 0.9909\n",
      "Epoch 98/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0088 - acc: 0.9970 - val_loss: 0.0417 - val_acc: 0.9900\n",
      "Epoch 99/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0091 - acc: 0.9969 - val_loss: 0.0414 - val_acc: 0.9895\n",
      "Epoch 100/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0098 - acc: 0.9966 - val_loss: 0.0448 - val_acc: 0.9903\n",
      "Epoch 101/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0107 - acc: 0.9965 - val_loss: 0.0438 - val_acc: 0.9904\n",
      "Epoch 102/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0107 - acc: 0.9965 - val_loss: 0.0422 - val_acc: 0.9898\n",
      "Epoch 103/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0095 - acc: 0.9965 - val_loss: 0.0440 - val_acc: 0.9901\n",
      "Epoch 104/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0093 - acc: 0.9972 - val_loss: 0.0532 - val_acc: 0.9892\n",
      "Epoch 105/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0094 - acc: 0.9968 - val_loss: 0.0392 - val_acc: 0.9902\n",
      "Epoch 106/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0079 - acc: 0.9972 - val_loss: 0.0440 - val_acc: 0.9899\n",
      "Epoch 107/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0105 - acc: 0.9966 - val_loss: 0.0490 - val_acc: 0.9889\n",
      "Epoch 108/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0089 - acc: 0.9972 - val_loss: 0.0418 - val_acc: 0.9904\n",
      "Epoch 109/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0082 - acc: 0.9972 - val_loss: 0.0496 - val_acc: 0.9899\n",
      "Epoch 110/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0098 - acc: 0.9965 - val_loss: 0.0445 - val_acc: 0.9895\n",
      "Epoch 111/281\n",
      "60200/60200 [==============================] - 89s 1ms/step - loss: 0.0097 - acc: 0.9968 - val_loss: 0.0470 - val_acc: 0.9903\n",
      "Epoch 112/281\n",
      "60200/60200 [==============================] - 89s 1ms/step - loss: 0.0085 - acc: 0.9972 - val_loss: 0.0442 - val_acc: 0.9896\n",
      "Epoch 113/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0083 - acc: 0.9972 - val_loss: 0.0454 - val_acc: 0.9903\n",
      "Epoch 114/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0079 - acc: 0.9973 - val_loss: 0.0445 - val_acc: 0.9903\n",
      "Epoch 115/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0084 - acc: 0.9971 - val_loss: 0.0529 - val_acc: 0.9897\n",
      "Epoch 116/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0082 - acc: 0.9973 - val_loss: 0.0433 - val_acc: 0.9905\n",
      "Epoch 117/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0099 - acc: 0.9968 - val_loss: 0.0418 - val_acc: 0.9905\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 118/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0094 - acc: 0.9969 - val_loss: 0.0452 - val_acc: 0.9901\n",
      "Epoch 119/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0078 - acc: 0.9974 - val_loss: 0.0459 - val_acc: 0.9901\n",
      "Epoch 120/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0096 - acc: 0.9966 - val_loss: 0.0533 - val_acc: 0.9892\n",
      "Epoch 121/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0078 - acc: 0.9975 - val_loss: 0.0421 - val_acc: 0.9905\n",
      "Epoch 122/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0083 - acc: 0.9971 - val_loss: 0.0451 - val_acc: 0.9899\n",
      "Epoch 123/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0076 - acc: 0.9975 - val_loss: 0.0409 - val_acc: 0.9900\n",
      "Epoch 124/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0082 - acc: 0.9973 - val_loss: 0.0454 - val_acc: 0.9903\n",
      "Epoch 125/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0090 - acc: 0.9970 - val_loss: 0.0433 - val_acc: 0.9902\n",
      "Epoch 126/281\n",
      "60200/60200 [==============================] - 91s 2ms/step - loss: 0.0079 - acc: 0.9974 - val_loss: 0.0448 - val_acc: 0.9903\n",
      "Epoch 127/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0074 - acc: 0.9976 - val_loss: 0.0432 - val_acc: 0.9901\n",
      "Epoch 128/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0070 - acc: 0.9976 - val_loss: 0.0481 - val_acc: 0.9902\n",
      "Epoch 129/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0087 - acc: 0.9971 - val_loss: 0.0472 - val_acc: 0.9899\n",
      "Epoch 130/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0069 - acc: 0.9976 - val_loss: 0.0439 - val_acc: 0.9909\n",
      "Epoch 131/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0067 - acc: 0.9978 - val_loss: 0.0418 - val_acc: 0.9897\n",
      "Epoch 132/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0072 - acc: 0.9974 - val_loss: 0.0456 - val_acc: 0.9902\n",
      "Epoch 133/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0074 - acc: 0.9976 - val_loss: 0.0429 - val_acc: 0.9901\n",
      "Epoch 134/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0077 - acc: 0.9974 - val_loss: 0.0453 - val_acc: 0.9905\n",
      "Epoch 135/281\n",
      "60200/60200 [==============================] - 86s 1ms/step - loss: 0.0063 - acc: 0.9980 - val_loss: 0.0475 - val_acc: 0.9904\n",
      "Epoch 136/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0074 - acc: 0.9977 - val_loss: 0.0509 - val_acc: 0.9897\n",
      "Epoch 137/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0070 - acc: 0.9977 - val_loss: 0.0485 - val_acc: 0.9899\n",
      "Epoch 138/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0065 - acc: 0.9977 - val_loss: 0.0475 - val_acc: 0.9897\n",
      "Epoch 139/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0082 - acc: 0.9971 - val_loss: 0.0435 - val_acc: 0.9901\n",
      "Epoch 140/281\n",
      "60200/60200 [==============================] - 91s 2ms/step - loss: 0.0075 - acc: 0.9976 - val_loss: 0.0461 - val_acc: 0.9900\n",
      "Epoch 141/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0065 - acc: 0.9976 - val_loss: 0.0458 - val_acc: 0.9900\n",
      "Epoch 142/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0073 - acc: 0.9975 - val_loss: 0.0434 - val_acc: 0.9904\n",
      "Epoch 143/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0078 - acc: 0.9976 - val_loss: 0.0439 - val_acc: 0.9902\n",
      "Epoch 144/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0066 - acc: 0.9980 - val_loss: 0.0455 - val_acc: 0.9904\n",
      "Epoch 145/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0076 - acc: 0.9975 - val_loss: 0.0501 - val_acc: 0.9898\n",
      "Epoch 146/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0059 - acc: 0.9979 - val_loss: 0.0452 - val_acc: 0.9905\n",
      "Epoch 147/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0068 - acc: 0.9979 - val_loss: 0.0431 - val_acc: 0.9906\n",
      "Epoch 148/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0066 - acc: 0.9978 - val_loss: 0.0445 - val_acc: 0.9912\n",
      "Epoch 149/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0068 - acc: 0.9977 - val_loss: 0.0455 - val_acc: 0.9903\n",
      "Epoch 150/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0069 - acc: 0.9977 - val_loss: 0.0501 - val_acc: 0.9902\n",
      "Epoch 151/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0069 - acc: 0.9978 - val_loss: 0.0505 - val_acc: 0.9894\n",
      "Epoch 152/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0063 - acc: 0.9978 - val_loss: 0.0427 - val_acc: 0.9906\n",
      "Epoch 153/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0064 - acc: 0.9980 - val_loss: 0.0446 - val_acc: 0.9904\n",
      "Epoch 154/281\n",
      "60200/60200 [==============================] - 91s 2ms/step - loss: 0.0064 - acc: 0.9978 - val_loss: 0.0445 - val_acc: 0.9908\n",
      "Epoch 155/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0057 - acc: 0.9983 - val_loss: 0.0513 - val_acc: 0.9905\n",
      "Epoch 156/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0059 - acc: 0.9980 - val_loss: 0.0416 - val_acc: 0.9906\n",
      "Epoch 157/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0061 - acc: 0.9980 - val_loss: 0.0441 - val_acc: 0.9902\n",
      "Epoch 158/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0063 - acc: 0.9978 - val_loss: 0.0495 - val_acc: 0.9895\n",
      "Epoch 159/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0061 - acc: 0.9978 - val_loss: 0.0459 - val_acc: 0.9903\n",
      "Epoch 160/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0060 - acc: 0.9980 - val_loss: 0.0479 - val_acc: 0.9904\n",
      "Epoch 161/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0061 - acc: 0.9978 - val_loss: 0.0450 - val_acc: 0.9907\n",
      "Epoch 162/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0058 - acc: 0.9982 - val_loss: 0.0510 - val_acc: 0.9901\n",
      "Epoch 163/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0071 - acc: 0.9978 - val_loss: 0.0437 - val_acc: 0.9916\n",
      "Epoch 164/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0056 - acc: 0.9982 - val_loss: 0.0429 - val_acc: 0.9908\n",
      "Epoch 165/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0065 - acc: 0.9979 - val_loss: 0.0456 - val_acc: 0.9908\n",
      "Epoch 166/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0059 - acc: 0.9980 - val_loss: 0.0492 - val_acc: 0.9904\n",
      "Epoch 167/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0062 - acc: 0.9978 - val_loss: 0.0489 - val_acc: 0.9908\n",
      "Epoch 168/281\n",
      "60200/60200 [==============================] - 90s 1ms/step - loss: 0.0054 - acc: 0.9982 - val_loss: 0.0439 - val_acc: 0.9911\n",
      "Epoch 169/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0062 - acc: 0.9980 - val_loss: 0.0480 - val_acc: 0.9901\n",
      "Epoch 170/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0063 - acc: 0.9980 - val_loss: 0.0511 - val_acc: 0.9903\n",
      "Epoch 171/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0060 - acc: 0.9983 - val_loss: 0.0395 - val_acc: 0.9916\n",
      "Epoch 172/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0057 - acc: 0.9981 - val_loss: 0.0462 - val_acc: 0.9908\n",
      "Epoch 173/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0057 - acc: 0.9982 - val_loss: 0.0440 - val_acc: 0.9909\n",
      "Epoch 174/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0056 - acc: 0.9980 - val_loss: 0.0493 - val_acc: 0.9901\n",
      "Epoch 175/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0055 - acc: 0.9980 - val_loss: 0.0497 - val_acc: 0.9899\n",
      "Epoch 176/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0054 - acc: 0.9982 - val_loss: 0.0401 - val_acc: 0.9904\n",
      "Epoch 177/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0050 - acc: 0.9981 - val_loss: 0.0496 - val_acc: 0.9896\n",
      "Epoch 178/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0063 - acc: 0.9981 - val_loss: 0.0469 - val_acc: 0.9902\n",
      "Epoch 179/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0059 - acc: 0.9980 - val_loss: 0.0509 - val_acc: 0.9895\n",
      "Epoch 180/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0058 - acc: 0.9981 - val_loss: 0.0509 - val_acc: 0.9896\n",
      "Epoch 181/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0062 - acc: 0.9979 - val_loss: 0.0488 - val_acc: 0.9898\n",
      "Epoch 182/281\n",
      "60200/60200 [==============================] - 89s 1ms/step - loss: 0.0060 - acc: 0.9979 - val_loss: 0.0437 - val_acc: 0.9904\n",
      "Epoch 183/281\n",
      "60200/60200 [==============================] - 89s 1ms/step - loss: 0.0048 - acc: 0.9985 - val_loss: 0.0427 - val_acc: 0.9905\n",
      "Epoch 184/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0050 - acc: 0.9982 - val_loss: 0.0421 - val_acc: 0.9908\n",
      "Epoch 185/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0059 - acc: 0.9981 - val_loss: 0.0453 - val_acc: 0.9910\n",
      "Epoch 186/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0057 - acc: 0.9980 - val_loss: 0.0488 - val_acc: 0.9906\n",
      "Epoch 187/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0061 - acc: 0.9979 - val_loss: 0.0506 - val_acc: 0.9904\n",
      "Epoch 188/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0054 - acc: 0.9984 - val_loss: 0.0475 - val_acc: 0.9899\n",
      "Epoch 189/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0050 - acc: 0.9984 - val_loss: 0.0432 - val_acc: 0.9908\n",
      "Epoch 190/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0054 - acc: 0.9983 - val_loss: 0.0521 - val_acc: 0.9903\n",
      "Epoch 191/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0048 - acc: 0.9983 - val_loss: 0.0511 - val_acc: 0.9904\n",
      "Epoch 192/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0063 - acc: 0.9980 - val_loss: 0.0443 - val_acc: 0.9907\n",
      "Epoch 193/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0052 - acc: 0.9982 - val_loss: 0.0467 - val_acc: 0.9904\n",
      "Epoch 194/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0044 - acc: 0.9985 - val_loss: 0.0499 - val_acc: 0.9900\n",
      "Epoch 195/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0053 - acc: 0.9983 - val_loss: 0.0534 - val_acc: 0.9898\n",
      "Epoch 196/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0069 - acc: 0.9978 - val_loss: 0.0486 - val_acc: 0.9910\n",
      "Epoch 197/281\n",
      "60200/60200 [==============================] - 90s 2ms/step - loss: 0.0050 - acc: 0.9983 - val_loss: 0.0469 - val_acc: 0.9907\n",
      "Epoch 198/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0057 - acc: 0.9981 - val_loss: 0.0479 - val_acc: 0.9907\n",
      "Epoch 199/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0051 - acc: 0.9985 - val_loss: 0.0464 - val_acc: 0.9903\n",
      "Epoch 200/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0053 - acc: 0.9983 - val_loss: 0.0487 - val_acc: 0.9896\n",
      "Epoch 201/281\n",
      "60200/60200 [==============================] - 86s 1ms/step - loss: 0.0061 - acc: 0.9981 - val_loss: 0.0459 - val_acc: 0.9906\n",
      "Epoch 202/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0058 - acc: 0.9981 - val_loss: 0.0442 - val_acc: 0.9913\n",
      "Epoch 203/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0055 - acc: 0.9979 - val_loss: 0.0501 - val_acc: 0.9894\n",
      "Epoch 204/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0047 - acc: 0.9985 - val_loss: 0.0447 - val_acc: 0.9910\n",
      "Epoch 205/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0054 - acc: 0.9982 - val_loss: 0.0433 - val_acc: 0.9908\n",
      "Epoch 206/281\n",
      "60200/60200 [==============================] - 89s 1ms/step - loss: 0.0045 - acc: 0.9986 - val_loss: 0.0440 - val_acc: 0.9902\n",
      "Epoch 207/281\n",
      "60200/60200 [==============================] - 89s 1ms/step - loss: 0.0054 - acc: 0.9982 - val_loss: 0.0474 - val_acc: 0.9901\n",
      "Epoch 208/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0062 - acc: 0.9982 - val_loss: 0.0477 - val_acc: 0.9906\n",
      "Epoch 209/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0043 - acc: 0.9985 - val_loss: 0.0443 - val_acc: 0.9902\n",
      "Epoch 210/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0048 - acc: 0.9987 - val_loss: 0.0449 - val_acc: 0.9907\n",
      "Epoch 211/281\n",
      "60200/60200 [==============================] - 91s 2ms/step - loss: 0.0050 - acc: 0.9983 - val_loss: 0.0490 - val_acc: 0.9898\n",
      "Epoch 212/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0055 - acc: 0.9982 - val_loss: 0.0488 - val_acc: 0.9903\n",
      "Epoch 213/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0043 - acc: 0.9984 - val_loss: 0.0429 - val_acc: 0.9907\n",
      "Epoch 214/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0046 - acc: 0.9984 - val_loss: 0.0482 - val_acc: 0.9905\n",
      "Epoch 215/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0048 - acc: 0.9985 - val_loss: 0.0440 - val_acc: 0.9903\n",
      "Epoch 216/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0048 - acc: 0.9984 - val_loss: 0.0488 - val_acc: 0.9906\n",
      "Epoch 217/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0045 - acc: 0.9986 - val_loss: 0.0418 - val_acc: 0.9912\n",
      "Epoch 218/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0049 - acc: 0.9985 - val_loss: 0.0460 - val_acc: 0.9906\n",
      "Epoch 219/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0046 - acc: 0.9985 - val_loss: 0.0503 - val_acc: 0.9901\n",
      "Epoch 220/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0038 - acc: 0.9986 - val_loss: 0.0467 - val_acc: 0.9898\n",
      "Epoch 221/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0042 - acc: 0.9987 - val_loss: 0.0517 - val_acc: 0.9896\n",
      "Epoch 222/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0043 - acc: 0.9985 - val_loss: 0.0500 - val_acc: 0.9899\n",
      "Epoch 223/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0042 - acc: 0.9985 - val_loss: 0.0479 - val_acc: 0.9903\n",
      "Epoch 224/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0045 - acc: 0.9986 - val_loss: 0.0435 - val_acc: 0.9906\n",
      "Epoch 225/281\n",
      "60200/60200 [==============================] - 91s 2ms/step - loss: 0.0046 - acc: 0.9984 - val_loss: 0.0477 - val_acc: 0.9903\n",
      "Epoch 226/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0042 - acc: 0.9986 - val_loss: 0.0453 - val_acc: 0.9910\n",
      "Epoch 227/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0048 - acc: 0.9985 - val_loss: 0.0449 - val_acc: 0.9903\n",
      "Epoch 228/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0041 - acc: 0.9987 - val_loss: 0.0490 - val_acc: 0.9902\n",
      "Epoch 229/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0042 - acc: 0.9986 - val_loss: 0.0483 - val_acc: 0.9907\n",
      "Epoch 230/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0047 - acc: 0.9984 - val_loss: 0.0477 - val_acc: 0.9904\n",
      "Epoch 231/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0045 - acc: 0.9987 - val_loss: 0.0463 - val_acc: 0.9913\n",
      "Epoch 232/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0046 - acc: 0.9985 - val_loss: 0.0436 - val_acc: 0.9908\n",
      "Epoch 233/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0042 - acc: 0.9986 - val_loss: 0.0467 - val_acc: 0.9906\n",
      "Epoch 234/281\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0050 - acc: 0.9983 - val_loss: 0.0487 - val_acc: 0.9906\n",
      "Epoch 235/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0046 - acc: 0.9983 - val_loss: 0.0483 - val_acc: 0.9905\n",
      "Epoch 236/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0047 - acc: 0.9985 - val_loss: 0.0495 - val_acc: 0.9909\n",
      "Epoch 237/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0047 - acc: 0.9984 - val_loss: 0.0501 - val_acc: 0.9904\n",
      "Epoch 238/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0036 - acc: 0.9987 - val_loss: 0.0454 - val_acc: 0.9911\n",
      "Epoch 239/281\n",
      "60200/60200 [==============================] - 91s 2ms/step - loss: 0.0051 - acc: 0.9984 - val_loss: 0.0508 - val_acc: 0.9907\n",
      "Epoch 240/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0040 - acc: 0.9986 - val_loss: 0.0481 - val_acc: 0.9911\n",
      "Epoch 241/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0048 - acc: 0.9986 - val_loss: 0.0447 - val_acc: 0.9907\n",
      "Epoch 242/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0046 - acc: 0.9984 - val_loss: 0.0533 - val_acc: 0.9910\n",
      "Epoch 243/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0040 - acc: 0.9988 - val_loss: 0.0510 - val_acc: 0.9913\n",
      "Epoch 244/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0050 - acc: 0.9983 - val_loss: 0.0488 - val_acc: 0.9899\n",
      "Epoch 245/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0042 - acc: 0.9986 - val_loss: 0.0473 - val_acc: 0.9906\n",
      "Epoch 246/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0052 - acc: 0.9983 - val_loss: 0.0488 - val_acc: 0.9905\n",
      "Epoch 247/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0039 - acc: 0.9989 - val_loss: 0.0522 - val_acc: 0.9899\n",
      "Epoch 248/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0038 - acc: 0.9988 - val_loss: 0.0511 - val_acc: 0.9906\n",
      "Epoch 249/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0041 - acc: 0.9986 - val_loss: 0.0435 - val_acc: 0.9908\n",
      "Epoch 250/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0043 - acc: 0.9987 - val_loss: 0.0509 - val_acc: 0.9902\n",
      "Epoch 251/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0036 - acc: 0.9989 - val_loss: 0.0478 - val_acc: 0.9901\n",
      "Epoch 252/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0044 - acc: 0.9984 - val_loss: 0.0446 - val_acc: 0.9910\n",
      "Epoch 253/281\n",
      "60200/60200 [==============================] - 89s 1ms/step - loss: 0.0044 - acc: 0.9985 - val_loss: 0.0519 - val_acc: 0.9908\n",
      "Epoch 254/281\n",
      "60200/60200 [==============================] - 89s 1ms/step - loss: 0.0040 - acc: 0.9987 - val_loss: 0.0500 - val_acc: 0.9914\n",
      "Epoch 255/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0034 - acc: 0.9989 - val_loss: 0.0494 - val_acc: 0.9914\n",
      "Epoch 256/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0048 - acc: 0.9984 - val_loss: 0.0491 - val_acc: 0.9907\n",
      "Epoch 257/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0038 - acc: 0.9988 - val_loss: 0.0563 - val_acc: 0.9898\n",
      "Epoch 258/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0037 - acc: 0.9989 - val_loss: 0.0480 - val_acc: 0.9907\n",
      "Epoch 259/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0039 - acc: 0.9988 - val_loss: 0.0499 - val_acc: 0.9895\n",
      "Epoch 260/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0038 - acc: 0.9988 - val_loss: 0.0535 - val_acc: 0.9903\n",
      "Epoch 261/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0044 - acc: 0.9986 - val_loss: 0.0470 - val_acc: 0.9908\n",
      "Epoch 262/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0042 - acc: 0.9984 - val_loss: 0.0469 - val_acc: 0.9907\n",
      "Epoch 263/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0042 - acc: 0.9988 - val_loss: 0.0475 - val_acc: 0.9909\n",
      "Epoch 264/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0039 - acc: 0.9988 - val_loss: 0.0456 - val_acc: 0.9906\n",
      "Epoch 265/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0037 - acc: 0.9987 - val_loss: 0.0484 - val_acc: 0.9904\n",
      "Epoch 266/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0038 - acc: 0.9987 - val_loss: 0.0477 - val_acc: 0.9913\n",
      "Epoch 267/281\n",
      "60200/60200 [==============================] - 89s 1ms/step - loss: 0.0036 - acc: 0.9990 - val_loss: 0.0501 - val_acc: 0.9912\n",
      "Epoch 268/281\n",
      "60200/60200 [==============================] - 90s 1ms/step - loss: 0.0038 - acc: 0.9988 - val_loss: 0.0509 - val_acc: 0.9900\n",
      "Epoch 269/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0037 - acc: 0.9988 - val_loss: 0.0478 - val_acc: 0.9905\n",
      "Epoch 270/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0037 - acc: 0.9986 - val_loss: 0.0496 - val_acc: 0.9909\n",
      "Epoch 271/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0043 - acc: 0.9987 - val_loss: 0.0546 - val_acc: 0.9910\n",
      "Epoch 272/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0039 - acc: 0.9987 - val_loss: 0.0550 - val_acc: 0.9904\n",
      "Epoch 273/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0044 - acc: 0.9983 - val_loss: 0.0456 - val_acc: 0.9906\n",
      "Epoch 274/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0036 - acc: 0.9988 - val_loss: 0.0502 - val_acc: 0.9903\n",
      "Epoch 275/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0043 - acc: 0.9986 - val_loss: 0.0515 - val_acc: 0.9907\n",
      "Epoch 276/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0036 - acc: 0.9987 - val_loss: 0.0456 - val_acc: 0.9906\n",
      "Epoch 277/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0045 - acc: 0.9986 - val_loss: 0.0458 - val_acc: 0.9915\n",
      "Epoch 278/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0037 - acc: 0.9987 - val_loss: 0.0494 - val_acc: 0.9915\n",
      "Epoch 279/281\n",
      "60200/60200 [==============================] - 87s 1ms/step - loss: 0.0034 - acc: 0.9989 - val_loss: 0.0518 - val_acc: 0.9900\n",
      "Epoch 280/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0036 - acc: 0.9989 - val_loss: 0.0468 - val_acc: 0.9907\n",
      "Epoch 281/281\n",
      "60200/60200 [==============================] - 88s 1ms/step - loss: 0.0038 - acc: 0.9988 - val_loss: 0.0460 - val_acc: 0.9905\n",
      "Test loss: 2.6041944881089556 %\n",
      "Test accuracy: 99.42999999999999 %\n",
      "saved model to disk\n"
     ]
    }
   ],
   "source": [
    "# TensorFlow's tensor expect the input shape (samples, rows, cols, channels) \n",
    "if K.image_data_format() == 'channels_first':\n",
    "    x_train = x_train.reshape(x_train.shape[0], 1, img_rows, img_cols)\n",
    "    x_valid = x_valid.reshape(x_valid.shape[0], 1, img_rows, img_cols)\n",
    "    x_test = x_test.reshape(x_test.shape[0], 1, img_rows, img_cols)\n",
    "    input_shape = (1, img_rows, img_cols)\n",
    "else:\n",
    "    x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)\n",
    "    x_valid = x_valid.reshape(x_valid.shape[0], img_rows, img_cols, 1)\n",
    "    x_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)\n",
    "    input_shape = (img_rows, img_cols, 1)\n",
    "\n",
    "# normalize the input data for reducing over-fitting\n",
    "x_train = x_train.astype('float32')\n",
    "x_valid = x_valid.astype('float32')\n",
    "x_test = x_test.astype('float32')\n",
    "x_train /= 255\n",
    "x_valid /= 255\n",
    "x_test /= 255\n",
    "print('x_train shape:', x_train.shape)\n",
    "print(x_train.shape[0], 'train samples')\n",
    "print(x_train.shape[0], 'train samples')\n",
    "print(x_test.shape[0], 'test samples')\n",
    "\n",
    "\n",
    "# convert class vectors to ont hot encoding\n",
    "y_train = keras.utils.to_categorical(y_train, num_classes,dtype='float32')\n",
    "y_valid = keras.utils.to_categorical(y_valid, num_classes,dtype='float32')\n",
    "y_test = keras.utils.to_categorical(y_test, num_classes,dtype='float32')\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Conv2D(32, kernel_size=(3, 3),\n",
    "                 activation='relu',\n",
    "                 input_shape=input_shape))\n",
    "model.add(BatchNormalization(axis=-1))\n",
    "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "model.add(Conv2D(64, (3, 3), activation='relu'))\n",
    "model.add(BatchNormalization(axis=-1))\n",
    "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Flatten())\n",
    "# model.add(Dense(128, activation='relu'))\n",
    "model.add(Dense(1568, activation='relu'))\n",
    "model.add(BatchNormalization(axis=-1))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(1000, activation='relu'))\n",
    "model.add(BatchNormalization(axis=-1))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(700, activation='relu'))\n",
    "model.add(BatchNormalization(axis=-1))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(300, activation='relu'))\n",
    "model.add(BatchNormalization(axis=-1))\n",
    "model.add(Dense(num_classes, activation='softmax'))\n",
    "\n",
    "model.compile(loss=keras.losses.categorical_crossentropy,\n",
    "              optimizer=keras.optimizers.Adadelta(),\n",
    "              metrics=['accuracy'])\n",
    "tensorboard = TensorBoard(log_dir=\"logs_mnist_11/{}\".format(time()))\n",
    "# simple early stopping\n",
    "es = EarlyStopping(monitor='val_acc', mode='max', verbose=1)\n",
    "history= model.fit(x_train, y_train,\n",
    "          batch_size=batch_size,\n",
    "          epochs=epochs,\n",
    "          verbose=1,\n",
    "          callbacks=[tensorboard],\n",
    "          validation_data=(x_valid, y_valid))\n",
    "\n",
    "score = model.evaluate(x_test, y_test, verbose=0)\n",
    "print('Test loss:', score[0]*100,'%')\n",
    "print('Test accuracy:', score[1]*100,'%')\n",
    "model.save('mnist_v11.h5')\n",
    "print('saved model to disk')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train: 1.000, Evaluate: 0.991\n"
     ]
    }
   ],
   "source": [
    "# evaluate the model\n",
    "_, train_acc = model.evaluate(x_train, y_train, verbose=0)\n",
    "_, valid_acc = model.evaluate(x_valid, y_valid, verbose=0)\n",
    "print('Train: %.3f, Evaluate: %.3f' % (train_acc, valid_acc))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test dataset accuracy: 0.994\n"
     ]
    }
   ],
   "source": [
    "_, test_acc = model.evaluate(x_test, y_test, verbose=0)\n",
    "print('Test dataset accuracy: %.3f' % (test_acc))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEWCAYAAACXGLsWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl8XGW9+PHPd5ZksrdZuu9QoKUbtBSUVZF930FRUK+oiNtPvBeuCFzUn/cq+nPjgiiVHayIgAgiIItSQFq60IXShZYmXZImzZ5MZvn+/njOJNN0JjNtmaZJv+/XK6/MnPU5c2ae73mW8xxRVYwxxpi++Po7AcYYY/Z/FiyMMcZkZMHCGGNMRhYsjDHGZGTBwhhjTEYWLIwxxmRkwcIYQETuFZHvZ7nsBhH5RK7TZMz+xIKFMcaYjCxYGDOIiEigv9NgBicLFmbA8Kp/vi0iy0SkTUTuEZHhIvKsiLSIyAsiMjRp+XNFZIWINIrIyyIyJWneESLytrfe74FQr32dLSJLvHUXiMiMLNN4logsFpFmEdkkIrf2mn+ct71Gb/7V3vQCEfmJiGwUkSYR+ac37SQRqU7xOXzCe32riDwmIg+KSDNwtYjMFZHXvX1sEZFfiUhe0vqHi8jzItIgIttE5D9FZISItItIRdJys0WkTkSC2Ry7GdwsWJiB5iLgFOAQ4BzgWeA/gUrc9/lrACJyCPAI8A2gCngG+LOI5HkZ5xPAA0A58Advu3jrHgnMA74IVAC/Bp4Skfws0tcGfAYYApwFfFlEzve2O85L7y+9NM0Clnjr3Q7MBj7qpenfgXiWn8l5wGPePh8CYsA3vc/kI8DJwLVeGkqAF4C/AqOAg4EXVXUr8DJwadJ2rwQeVdVIlukwg5gFCzPQ/FJVt6lqDfAP4E1VXayqYeBPwBHecpcBf1HV573M7nagAJcZHwMEgZ+pakRVHwPeStrHF4Bfq+qbqhpT1fuAsLden1T1ZVV9R1XjqroMF7BO9GZ/CnhBVR/x9luvqktExAd8Dvi6qtZ4+1zgHVM2XlfVJ7x9dqjqIlV9Q1WjqroBF+wSaTgb2KqqP1HVTlVtUdU3vXn34QIEIuIHrsAFVGMsWJgBZ1vS644U74u916OAjYkZqhoHNgGjvXk1uvMomhuTXo8HvuVV4zSKSCMw1luvTyJytIi85FXfNAFfwl3h421jXYrVKnHVYKnmZWNTrzQcIiJPi8hWr2rq/2aRBoAngakiMglXemtS1X/tYZrMIGPBwgxWm3GZPgAiIriMsgbYAoz2piWMS3q9CfiBqg5J+itU1Uey2O/DwFPAWFUtA+4CEvvZBByUYp3tQGeaeW1AYdJx+HFVWMl6Dx19J/AuMFlVS3HVdJnSgKp2AvNxJaBPY6UKk8SChRms5gNnicjJXgPtt3BVSQuA14Eo8DURCYjIhcDcpHV/A3zJKyWIiBR5DdclWey3BGhQ1U4RmQt8MmneQ8AnRORSb78VIjLLK/XMA34qIqNExC8iH/HaSN4DQt7+g8BNQKa2kxKgGWgVkcOALyfNexoYISLfEJF8ESkRkaOT5t8PXA2cCzyYxfGaA4QFCzMoqepqXP37L3FX7ucA56hql6p2ARfiMsUduPaNx5PWXYhrt/iVN3+tt2w2rgVuE5EW4GZc0Eps9wPgTFzgasA1bs/0Zl8PvINrO2kA/gfwqWqTt83f4kpFbcBOvaNSuB4XpFpwge/3SWlowVUxnQNsBdYAH0ua/xquYf1tr73DGADEHn5kjEkmIn8HHlbV3/Z3Wsz+w4KFMaabiBwFPI9rc2np7/SY/YdVQxljABCR+3D3YHzDAoXpzUoWxhhjMrKShTHGmIwGzaBjlZWVOmHChP5OhjHGDCiLFi3arqq9793ZxaAJFhMmTGDhwoX9nQxjjBlQRGRj5qWsGsoYY0wWLFgYY4zJyIKFMcaYjAZNm0UqkUiE6upqOjs7+zspA04oFGLMmDEEg/bcG2NMDoOFiMzDjZ1fq6rTUswX4Oe4sXLagatV9W1v3lW4AdMAvu89T2C3VVdXU1JSwoQJE9h5gFHTF1Wlvr6e6upqJk6c2N/JMcbsB3JZDXUvcHof888AJnt/1+CGVUZEyoFbgKNxI4HekvyozN3R2dlJRUWFBYrdJCJUVFRYicwY0y1nwUJVX8WNnpnOecD96rwBDBGRkcBpwPOq2qCqO3Dj1PQVdPpkgWLP2OdmjEnWn20Wo9n5CV/V3rR003chItfgSiWMGzcu1SLGmAGksb2LwrwAeYHU17GxuOL3uQuZzkiM7a1hKovzCQX9qCqq4PMJiWGM6lrClBYECQX9tIajFAT93esni8TiNHdEqCh2jwpRVUSETQ3tVJW47QOs3NzM8s1NHDysmMnDitlY387a2lZOPXw4oYCfmsYOFm3cwYwxZcTiyqqtLcTicfIDfsqL8pg5ZghBv/BOTROdkThF+X7yA37efL+eUMDPmKEFVBTnU16URzQepzAvQH7Ax9sbdzCuopCOrhhlBUFawlFWbWlmS2MnZYVB5owfyqSq4l2O68PUn8Ei1aWr9jF914mqdwN3A8yZM2e/HOSqsbGRhx9+mGuvvXa31jvzzDN5+OGHGTJkSI5SZj5M8bjSEo5SVtB3h4CWzggdXTGGlYaIxOLEVckP+HdapjUcZcP2NoaV5vPPNdvZ1NDBcZMrqCoOMba8AFXY2txJYZ6fp5dt4bARJayva6MoP8DUUaWUhAJsbepkXV0rhwwvobYlzFvvN1BZnEdHJM7Y8gJKQkHW1rYS9AvDSvJRhbhCcSjAtFGl/HPtdua9toHDR5UyfXQZEyqKKCsI8sSSGtZsa6EwP0BpKEhRnp9jJ1fy6nt1vPpeHdNHl3HJnLG8vq6esoIgb21oIBKLc8ykCsYMLSQv4GNtbStPL9vM6CEFLKtu4oRDKhk9pJC/vLOZ97a1UlWSz1nTR7KjvYsN9e3dn1vQ5+O92haK8wKUFQbZ3NhBXCEv4KOqOJ+Gti78PqGsIEhdS5hQ0EdzZxQROKiqmPV1rQT8PgT3fvroMjY2tFHbHGZHexc72iMcNqKEjfXthKMxRg8tYFNDB1Ul+URiccoKgtTs6CAa3zWrKSsI0hWN0xGJARDwScrlAj6htCBIQ1tXVt+rgE+oKM5jW3Pfj2M/fFQpf/na8Vltc0/ldCBBEZkAPJ2mgfvXwMuJR1WKyGrgpMSfqn4x1XLpzJkzR3vfwb1q1SqmTJmy18exNzZs2MDZZ5/N8uXLd5oei8Xw+/1p1to/7A+fXzYSV4F1LWH3o43Fqd7RztihhRTlB4jFlTW1LUSiis8HEyqKeHdrM4eNKKUoP8DymiZKQ0HqWjt5d2sLkyqLWVrdyIwxZUwdWcrTy7awvKaJ1nCUM6ePZPEHO3j1ve2MKAsxdVQp9a1h3t/exlsbdnBQVREKlIaCfPbYCaza0kJtSyfiXQO9+O42OrpizBw7hH+972ppC/PcVWdZQZBtzWG2t6bPGAq8K9xEppRLEyoK2dYc3mlffp8wZWQJLZ1R2sJRmjuidMXiBP3C0RMrWFbd2J1Bq0JFUR5lhUHW17XttO3DRpSwo72Lw0eVsWDddjojcWaOKePUw0fw1oYG3lhfT0HQz7TRZQCUhAK0hWNMHVVKR1eMHe1djC8vZOSQAt7f3kZ9a5d37mM0dUQZUZpPazjGocOLqW/rYsmmRqaNLiMeV2JxZdEHO9jc6ALB+PIi8gM+Rg4JsWjjDg4bUUoo6GfNthZmjR3Ckk2NFOUH2N4aZnhpiC+fdBBra1vZWN/OiLJ8KoryeWJxDUOL8phQWcTUkaU8+84WhhQGOfXwEQT9PsLRGJsbO3hrww5qdnRwytThVBTn0R6O0RqOMm10GT6B2pYw25o7ae6IEPT7eG9bK+9ubeayo8bS3BmlNBSguSNCUX6AycNKGF9ZSG1zJ00dEWaPL9+j8ywii1R1Tsbl+jFYnAVch+sNdTTwC1Wd6zVwLwKO9BZ9G5itqn21f+y3weLyyy/nySef5NBDDyUYDFJcXMzIkSNZsmQJK1eu5Pzzz2fTpk10dnby9a9/nWuuuQboGb6ktbWVM844g+OOO44FCxYwevRonnzySQoKCnKe9lx9fqrKP9dup2ZHB+fMHEXQ73OZe3khQb+PtnCUxxZVEwr6iMVdVcJ721rY2tzJ+IpCTp06nJFlBZQWBHl9XT0/fGYVo4cWsKa2lSkjS9jWHKauJUxVST7XnnQQD7/5AWtqW3dJx9jyAg4ZVsKL79YypDBIJBqnrSt1JlxelIff5wKSCJwwuYoVm5vZ0e4yKVXlkjljWVfbSijPz+qtLaytbcXvE0aWhVB1x33IiBICPmHl5mbOnTWa4nw/O9ojNLR10djeRVVJPhMrixlfUcj6ulaGlYY48ZAqVmxuYmtTmHV1rcRVmVBRRF1LmJMOrWJdXSsTKoqIqbKtuZOGtgihoI+ZY4bw/vY2SkIBjplUQXNnhMK8ABu2t9ERiTG+vJBoXGlsj+DzgeCOb/W2FqpK8jlj2gh8Imxu7GBjfTsN7V3MGF3GhMqi7s+ltqWTVVtcplpWEKSpI8Kz72xh7sRy8gI+KoryKcjzU9vcSX1bF80dETqjcU6YXNndLhaLKx2RGEV5/p2mCa5KyeRWvwcLEXkEV0qoBLbhejgFAVT1Lq/r7K9wjdftwGe9x1kiIp/DPWQe4Aeq+rtM+8sULP7rzytYubl57w8sydRRpdxyzuF9LpNcsnj55Zc566yzWL58eXeX1IaGBsrLy+no6OCoo47ilVdeoaKiYqdgcfDBB7Nw4UJmzZrFpZdeyrnnnsuVV175oR5LKpmCxXMrtlLbEqY0FKCqOJ+SUJA/L9tMWYG7kvzoQRX8+tV1zB4/lNVbW7qvjBraeqoXAj7B7xPC0Th+n5Dn9xGLK12x+E77GlEaYmJlESu3NNPUEdlp3pHjhhBXmFRZxF9XbKW8KI+vnTyZ//f8e2xp6uSgqiKuOWES5UX5dERirNnWwpihBdy3YCMdkRgfO3QYz63Yiqpy67mH09wZ5YTJlbz9wQ4Wb2rkrOkjmT66jEhMefCNjRw+qpSjJ1XQFY0TicUpzPN315UnhKMxHv3XJj5yUAWHDM/m0d3G9I9sg0XO2ixU9YoM8xX4Spp583APsB905s6du9O9C7/4xS/405/+BMCmTZtYs2YNFRUVO60zceJEZs2aBcDs2bPZsGHDbu9XVWkLR4mrK9InruAisTiCq17ojMTwiRDwC9uaw9Q2d/Kb3y+hJBRgQmUR48oLufvV9QwpDDJlZCm/eHENydWyeX5fdyYfCvr449vVVBbnM39hNeMrCjlsRAnNHVGGlYT4t+MnceiIEl5eXUs4EufgYcVs2tFOVzSOiHDa4SMYWhgkL+BjRGmIgN81eIajMVZubqa+tYvmzghDC/M4fnJl9/z/8OryS0JBPnboMD5oaOPIcUNT9u667KieThH/59RDiMV1pzaH06eN5PRpI3uOLyB87riJSe993Q2xvTefH/Bz1Ucn7PZ5MmZ/Najv4E6WqQSwrxQV9RThX375ZV544QVef/11CgsLOemkk1Le25Cfn9/92u/309HRkXE/qko4GneZv0/Y2NBOS6e7Ii8JBYnHlcJ8P9tbulAUnwhxVQTBJxDz2gFeX19Pa2eUlnAUgOGl+fgbhOdWbOOwESX87PJZxOPwx7ereaemiZ9fPoug34dfhD++Xc0FR4ymIM9PKOBPWaVw1ITdq2fND/g5Ylz6226Gl4a6X1eV5FNVkp922WTF+QfMT8GYPWK/kBwrKSmhpSX1EyqbmpoYOnQohYWFvPvuu7zxxhtZbzfRqJtKWzjK5sYOOiIxRISgT+iKxRlZVkA0HqeuJezaBlqiFOYFKC0IEI0p+QEf4WicWFypLM5jQ0s+r994MqpKTWMHiz9o5ITJVZQWBFha3cS48kLKi/IAmDpq6i7p+LfjJ2V9PMaY/ZsFixyrqKjg2GOPZdq0aRQUFDB8+PDueaeffjp33XUXM2bM4NBDD+WYY47JuD1VpakjwsotzRTnB+iIxAgF/ERicYpDrk92TWMnQZ8wekgB7V0xIrE4o4YWUBpyVSyVxfkEfEJLZ5Si/EDKfufJRIQxQwsZM7Swe9qssdal15gDyaB5Bvf+2htqb6kq25rDBPwuQ69rCXuNqgHCkRiF+QG6onECPqG9K4oCoaCfSZVF3fX4e2owfH7GmL71ewO32XPhaIzaZtdFMxpTmjt7ev8U5QcYM9TdVNVbJBYnHIlRkOfH77PR5wcM1V1byHfH2w/A6CNheJp2uY5GWPwAHPkZCJXt+X72VjwO0Q7IK8q8bDp781l1NEJ+KezOb6O9AUJDsl+n5m23zrhjID/LO6o7m0HjUJBFaV0VVv0ZWrbClLOhdFR2+/gQWLDYT7R2RtjaHO5umAbwiaAow0pC5AV8+MTdKZqurSLo9xHcy9JEn+JxWPBzmHYRDMkwvEq4Bf7xU6g4GKZf4mUSJbv+6BrehxWPw7HfTP+DXPVniEfh8As+nOMAiEXcX15P1RqJUnamzGjpozBkPIz/SM+0eAx8u3mTZXsDLJwHb9wJF/0WDvqYm95WD4E82PSmS+OI6RAIQVFlz7qRTtj6DoSb4KnrXBA49usw5ijwBeBfd0O4FS69H/52kwsWy/8IE0+A2VdDeRbtSU017txUv+XWm3mFy+hr3nbnd9KJ0LINGta7/foDbvpfrodhh7lzP+YoKBkBbdvh4cugrRYu/h289nMYezQc9W8QDHkZeQnUroKSkVBUAU3V0LgJRh0B/iC89jN47Rdw/LfgI9ft+n0Jt7jPa8M/Ye0LcOr3IFgENQvd9ud/GkYdCcd9E8rGQKgUNi5w68Vj8M4fXCb/wRsw6SSo/he8/yqc8G332XY2QckoiEdg/csQDcOUc6BxIzx/s8vwV/3ZpaXqMLjyj+5cxKNufw3vu3T5Au4zO/hk8AXhoYsh0g4n/Lvb19JHYP1LUFgJJ90A0U4oqoKNr7lj+8dP3D7W/A3O/im018OIGbv//dtNVg3VT1SV9q4Y25o76YzEUY3j9/nID/rJD/ioLM4jL7CXJ1/V/TjjsZ4rkFiX++EWD3Nf2sSVWqTTfdn8XoklFmXVineYMnEUbPgHvP4rmPM5eOqrLlhMu9j9AEbO6NlfZ7PbflEl/OlL7ksPcOhZ7ks+5RyYfjEseQiCBTD3i257m9+GTz/hMo9X/sd9+Wd/1m1r7fPej0Pg4nkQLIRX/ttlCsd/yy07/WJ44y6XnumXuMyxZQuc9n9hyxJY/SxMPc9lyj4/HHY2/P170LEDxn3UZdKTT4FF98L7/4CTvwvb17iMJFTqMonKQ10mESx0n0VoCHzsP6F+LWxdDh+87jKIC+6EQIH7DFY+4TIDn98FmEiHe51f4jLExQ+4zye/1J2HQ8+AIz4F8z/j9pnMF4SjPu+O44X/gtqVEG4G8UHxcJeuhnU9yxcMdRnkqCPc5zvxRKhZ5DKl4hFw8s2w9GEYdrgrlXQ2uYwb3Ofw0g+grc69Lx4OrdtcGqacDe8957ZTPgl2bASNuUz02K/BtuWw+MGedARCLgNc9ZQLbhp3aQu3uEx0+HQYPhWWzXff0eYaqDwEJn0MFt7jlgmE3GfWVgflB7njnHA8fOJWGD0bdmyAf/4Ulv3BfWc0DigMneDORd0ql5ahE6Gz0Z33VAqGunkVB7vzOmQ8FFbAthXuoqJjh8uURWDLUrfOxBNdUGuucd/Roz4PY+bAU19zv7FO796uE66H529xF03gzpt69xKVT4JhU+Hdv7g073gfysa5bWqs5zsS9rY1/RL3Gb30A/DnQywMI2fCF19NfVwZ9PtNefva/h4sEp9zV+1aIvipiVcQjrlB0Yb7W6mM1aL+fKRyck+Gnay9wX3BUhVV41GXaSLuKiQadj+uSIf7cYD7oQTyYMcH3lV+MQwdD3XvuWU7G13wqDrUrd/wPqver2HKc5f27Mef536MCKAu8zj/f92P4/lb3JddxAWSZY+6K6VoJyz4Rc864ofCcnfVm/jh+IIukJRPdIEhv9RlJokhwQ453f1QN73p1q842GXcDevd/KKqnoxtxHSXKSXSG4/1/OBKx7gfaMtml+lMv9hV4bRsdplKtKPnxwcuI0t8VnXvuu2Fm90+6la7zyJY6DKZaRfCO4+5IJVK6WgoG+vSUvsudLXAUV+Aw86CysnuanzjAldSCBTAMV920wu9K+yt78Ci3/UcxyGnugzjtZ/DKd9z++9qheWPu+/DrE/Bv34Nr97urpIvuMtdZdeuhAcucBcRwSKIJA3DMepIFxSaa1xmPPlUFxyGTnRXtSufchl4ySg44krYuswFyGFTXAlp42tuO0d/GT7yFbet137Wc7V92UPw3H+6K/HTfugyxr99B5o3uwuQxg/cd3LZfHcxMPsqOPgTsPF197lOu8h9XosfhL/e6D7Dwgr33ffnw6wr3Hcn2gkHney+d+EWt52mGpeRF1a6kkZ7vftNlY1xaat7F47+kguaRZWumqdomPtu/HKOC2RzPgf/uN39Ps79pbvoeu1nLq2XPQDjPtJTKt3wGjxwvjuXXa3uvE84Hs76qSuBFY9wF0LVC912iyrhzmPdd/6MH8Mhp8Gmf7nSQ36xO/+HX+AuUsYd4/Z5x9FuvaO/5NabeXnq714GFizYf4JFR1eU9+vbKfWFGROrBqBdiogUj6I4T/A317iMJx5xV5wlI3beQLTTZTCIK953NEBXm/uh5BW5jCvu7oNA/C5TS2TEJSPd8tHEeEPivmBtdS6Tjkd61tO42140DCKs2tbJlI5FLphs+IcrEYw/1lVDjJnjMuKty7wfbAPMudr9sGsWwvRL4fw73TZf/C/35X/6/7iM/VPzXSB792mIRd2V4lu/demYfokrEbz0A3c1P/1ilxlHOlypo36tC1Dig7Uvukzkrze4ksjww+GZ612G8Nln4Y07XJCZdSW89yx8/Lvux7xwnqtSKR3lruibN8M9p7of8ef+5vZRPglKe27I666i2vqOy+Q2LnCZwLSLvM9PXKa+9BGXmTZudMdcVOU+zyHjXbAGVx3RsM5lhMnqVsOjn4KPXueqinpbNt9t/9xf9mRyeyLaBdvecce4+EF3HuMR78p2oqsemfnJ1NWCOza4IFNctfN0VVeq2LrcZWrBUM/0Bb9wmficz8Ki++DNX8O/vdBTBdi7HWLTW+6CadSs9MfQ2Qwrn3Ql1vKJrj0mV/X3Oza473h+CbTWuu/i0PE989O1o9Qschca619xr8/5mStRp9PV5n67qS4W0y0fKNi9NpgULFjQD8Gis8l9kUpGQFc7NNfQUTgSf+P71DCcUm1mqLQixcOR1q1eURQg7jL1rla3fl4xBPJdBpwYiFdjXoblna9ERp+oIqg42BXXfV4zVKKkkFzC8Oe7qxR/nqsLbt/urlQSxfxYl8vkEKg8hFXrNvZ8fnWr4dcnwuUPQtUUlwk218D/HuPWu/oZV4cfbnEB4+BP7Poljobdvnv/sBo3wQu3uszj5Ft2rpvPRsN6GDLBbfeNO6HqkF0z4kw6Gl1gK9yzwdiMGaisN1SuqborxIIyd9Wh6q4sY11oqIxY4yYC0XYIbyRPoowtjODvaIf8IUjJCFfdEIv2VJGEytxVff1aFzQ6G10VRzAE0S4mzD6Zha/+jcqSkGv88wVc0bltu3udV7xzJlzQc5fzE395jkMOOYSpU5NunCsd7dYrrOi54gWXmWt854ZfcNVTN27a+aqnfKJrmI2Gexp780tcFUkqgTR3Uw8ZCxffk+ED70NyY+1Hdm8o+G7Z9EQx5gBmwWJ3xGPQutVlxPG4y/C7Wt3VebjFq8+HyPb15GkXChSImxbobHCBoWCIy9QrJrttduxwRepAyE0f6RW94xFXekgEAPG5K+5y76pbvfp/jUFeaZ89eJ544gnOPvvsnYOFz7dzNUtCX939UhWPp5yTfnljzKBhnfF3R1udq7OsW+1KEYnqoYZ1aOMHRAhQryXkaRfhQLGrqgEe/OMzzD3jCmadcjlf/Pq3ueOOO/j3G250PWOKKrn3z6/y1a99DYDzL7iA2XPmcPiMI7j7N7/ZJQkbNmxg2rRpLjjkFXH7Xfdz6+13AvCb3/yGo446ipkzZ3LRRRfR3t7OggULeOqpp/j2t7/NrFmzWLduHevWreP0009n9uzZHH/88bz77rv77CM0xgxMB07J4tkbenrJ7BF1DUricxl1PArDp8HJtxBv206YPD6IV1BRVkw4oOQXFEKkk1XL3ub3z7zMa0/MI1hYxrW3/Jzi4mIef/xxfvSjHwHw+9//nu985zsAzJs3b6chyy+66KJdRqHtlueVArwr/gsvvJAvfOELANx0003cc889fPWrX+Xcc8/l7LPP5uKLLwbg5JNP5q677mLy5Mm8+eabXHvttfz973/fi8/GGDPYHTjBYm/FI4C6+n3xd3eb7Cqo4r3mED4RKkvzXJtCQjDEi0s/YNHS5Rx15qfBH6AjHGXYsGFMmjSJN954g8mTJ7N69WqOPfZYILshy7sVlruAEXEFxOXLl3PTTTfR2NhIa2srp5122i6rtLa2smDBAi655JLuaeFw349sNMaYAydYnPHfe75uPOb6pwdCrteR1z6gqtR4D/E5eFhRypvoVJWrrrqaH3732677oN995Pfccw/z58/nsMMO44ILLkBEshqyPBAIEI97N/P4g3SSD+K6zV599dU88cQTzJw5k3vvvZeXX35510OJxxkyZAhLlizZ88/DGHPAsTaLTKJdrodSPOq6tyYFis1NnbR0RhhZFkp7t/XJJ5/MY489Rm1rFPwBGhoa2LhxIxdeeCFPPPEEjzzyCJdddhmQ3ZDlw4cPp7a2lvr6esLhME8//XT3vJaWFkaOHEkkEuGhhx7qnp48THppaSkTJ07kD3/4Q/dxLF269MP5rIwxg5YFi0za67ybcCZ09xRSVepaw9S3uuc8VxSnf8DO1KlT+f73v8+pp57KjBkzOOWUU9iyZQtDhw5l6tSpbNy4kblz5wJuyPJoNMqMGTP47ne/m3LI8mAwyM0338zRRx8EOiyiAAAcRElEQVTN2WefzWGHHdY973vf+x5HH300p5xyyk7TL7/8cn784x9zxBFHsG7dOh566CHuueceZs6cyeGHH86TTz75IX1YxpjBKqc35YnI6cDPAT/wW1X9717zx+Men1oFNABXqmq1N+9HwFm4gPY88HXtI7E5uymvfq27H2KYy3zjcWVDfRut4ShlBUHGlRemHdhvoNtf7oA3xuROtjfl5axkISJ+4A7gDGAqcIWI9H6c2u3A/ao6A7gN+KG37keBY4EZwDTgKODEXKW1T5GOnW7Rb+6M0BqOMqI0xNhBHCiMMSZZLquh5gJrVXW9qnYBjwLn9VpmKvCi9/qlpPkKhIA8IB8IAttymNad7djohpCIRVxbRVKwaOqIEPT7qCrJx2eBwhhzgMhlsBgNbEp6X+1NS7YU8EZi4wKgREQqVPV1XPDY4v09p6qreu9ARK4RkYUisrCuri5lIna7mk3VjfHU2dQzJHCwgFhcqWvppLkz2uczJQaLwTJmmDHmw5HLYJEqN+2dA10PnCgii3HVTDVAVEQOBqYAY3AB5uMicsIuG1O9W1XnqOqcqqqq3rMJhULU19fvXsYX6+oZr6llq/sfCLG5sYMtTZ0EfEJ5UV769QcBVaW+vp5QKJR5YWPMASGX91lUA2OT3o8BNicvoKqbgQsBRKQYuEhVm0TkGuANVW315j0LHAPs1tM9xowZQ3V1NelKHSl1tbvRWBPjLgXy6dy+mu2tXZQWBAiGgrzfuDupGJhCoRBjxuzFMNjGmEEll8HiLWCyiEzElRguBz6ZvICIVAINqhoHbsT1jAL4APiCiPwQV0I5EfjZ7iYgGAwyceLE3VvprzfCwt/B559zT0ub+nE+OW8RG+vbeeXbJxHI5WNLjTFmP5WznE9Vo8B1wHPAKmC+qq4QkdtE5FxvsZOA1SLyHjAc+IE3/TFgHfAOrl1jqar+OVdp3UnNIvfQlZEzYfrFrK0Ps2BdPZ88epwFCmPMASunw32o6jPAM72m3Zz0+jFcYOi9Xgz4Yi7TllIs4p6tm3gWMfDYomoCPuHSOWP7WNEYYwY3u1ROtm2Fe4Tp6CMB19D77PItfPTgSqpK0t+lbYwxg50Fi2Q1i9z/0bMBWLmlmY317Zw5bUQfKxljzOBnwSJZzdtQWAlD3MPY/7JsC36fcOrhFiyMMQc2CxbJaha5UoUIsbjyp8U1HHdw5aC/r8IYYzKxYJGscSNUumdjL1i3nS1NnVwyx+41MMYYCxYJXe0QaYdC91S6Py/dTGkowCemDO/nhBljTP+zYJHQ0eD+e8Fi1ZYWZo4dQiiY+qFGxhhzILFgkdC23f0vrCAeV9bWtnLwsOL+TZMxxuwnLFgktNe7/4UVbG7qoCMSs2BhjDEeCxYJ7V41VFEla2tbAZg8rKQfE2SMMfsPCxYJSSWLRLCwkoUxxjgWLBLat4P4IDSEtbWtlBfl2f0VxhjjsWCR0F4PBeXg87HGGreNMWYnFiwS2uuhsAJV6wlljDG9WbBIaG+AwgrqWsM0dUSYbMHCGGO6WbBIaNsOheXWuG2MMSlYsEhor4eiStZZsDDGmF1YsABQdcN9FJSzpraV4vwAI0pD/Z0qY4zZb+Q0WIjI6SKyWkTWisgNKeaPF5EXRWSZiLwsImOS5o0Tkb+JyCoRWSkiE3KW0GgY4lHIL2ZtbSsHDStGRHK2O2OMGWhyFixExA/cAZwBTAWuEJGpvRa7HbhfVWcAtwE/TJp3P/BjVZ0CzAVqc5VWIu3uf7CI6h0djC8vzNmujDFmIMplyWIusFZV16tqF/AocF6vZaYCL3qvX0rM94JKQFWfB1DVVlVtz1lKu4NFAdtbw1QW2/O2jTEmWS6DxWhgU9L7am9asqXARd7rC4ASEakADgEaReRxEVksIj/2Sio7EZFrRGShiCysq6vb85R2uWAR9oVo74pRWWJ3bhtjTLJcBotUlf7a6/31wIkishg4EagBokAAON6bfxQwCbh6l42p3q2qc1R1TlVV1Z6n1CtZNMdckKgsspKFMcYky2WwqAbGJr0fA2xOXkBVN6vqhap6BPAdb1qTt+5irworCjwBHJmzlHrBojESALCShTHG9JLLYPEWMFlEJopIHnA58FTyAiJSKSKJNNwIzEtad6iIJIoLHwdW5iylXrDYEQkCWJuFMcb0krNg4ZUIrgOeA1YB81V1hYjcJiLneoudBKwWkfeA4cAPvHVjuCqoF0XkHVyV1m9yldZEm0V9l2sWsWBhjDE7C+Ry46r6DPBMr2k3J71+DHgszbrPAzNymb5ukQ4A6jpdsLChyY0xZmd2BzdApA2Auk4fJaEAoeAuHa+MMeaAZsECuksWmzt8VFkVlDHG7MKCBXS3WWxtFyqKrQrKGGN6s2ABrjeUL8i2trg1bhtjTAoWLMAFi2AhO9q6rHHbGGNSsGAB0NUGeYW0hqMUh3LaQcwYYwYkCxYAkQ40WEA4Gqcoz4KFMcb0ZsECINJOPOCGJS/Ms26zxhjTmwULgEg7Mb97Ml5xvpUsjDGmNwsWAF3tRP0FABRasDDGmF1YsACIdBDxJUoWVg1ljDG9WbAAiLTR5QWLQmvgNsaYXViwAIh00CXuZjxrszDGmF1ZsADoaqdTEiULq4YyxpjeLFgARNrpxEoWxhiTjgWLWATiEdrVBQvrDWWMMbuyYOE9UrVd3ZhQhfYsC2OM2YUFi2gXlI2lkRIK8/z4fNLfKTLGmP1OToOFiJwuIqtFZK2I3JBi/ngReVFElonIyyIyptf8UhGpEZFf5SyRxVXwzeW8UXqadZs1xpg0chYsRMQP3AGcAUwFrhCRqb0Wux24X1VnALcBP+w1/3vAK7lKY7K2cNRuyDPGmDSyChYi8kcROUtEdie4zAXWqup6Ve0CHgXO67XMVOBF7/VLyfNFZDYwHPjbbuxzj7V3Ra1kYYwxaWSb+d8JfBJYIyL/LSKHZbHOaGBT0vtqb1qypcBF3usLgBIRqfCC0k+Ab/e1AxG5RkQWisjCurq6bI4jrbZwzLrNGmNMGlkFC1V9QVU/BRwJbACeF5EFIvJZEQmmWS1VS7H2en89cKKILAZOBGqAKHAt8IyqbqIPqnq3qs5R1TlVVVXZHEpabV1RCq0ayhhjUsr6UlpEKoArgU8Di4GHgOOAq4CTUqxSDYxNej8G2Jy8gKpuBi70tl8MXKSqTSLyEeB4EbkWKAbyRKRVVXdpJP+wtIWjjB1amKvNG2PMgJZVsBCRx4HDgAeAc1R1izfr9yKyMM1qbwGTRWQirsRwOa4qK3m7lUCDqsaBG4F5AF4pJrHM1cCcXAYKcNVQRVayMMaYlLItWfxKVf+eaoaqzkkzPSoi1wHPAX5gnqquEJHbgIWq+hSuRPJDEVHgVeAru3sAH5Y2a+A2xpi0ss0dp4jI26raCCAiQ4ErVPV/+1pJVZ8Bnuk17eak148Bj2XYxr3AvVmmc4+1d1nJwhhj0sm2N9QXEoECQFV3AF/ITZL2vXhcicWVoN9uaDfGmFSyzR19ItLdu8m74S4vN0na9+LqOmn5xYb6MMaYVLKthnoOmC8id+G6v34J+GvOUrWPxbxgYeNCGWNMatkGi/8Avgh8GXf/xN+A3+YqUfuaFyuwgoUxxqSWVbDwurbe6f0NOrG4VUMZY0xfsr3PYjJukL+pQCgxXVUn5Shd+1SizcJnwcIYY1LKtoH7d7hSRRT4GHA/7ga9QSEed/+tzcIYY1LLNlgUqOqLgKjqRlW9Ffh47pK1b/WULPo5IcYYs5/KtoG70xsJdo13V3YNMCx3ydq3urvOWrQwxpiUsi1ZfAMoBL4GzMYNKHhVrhK1ryW6zoq1WRhjTEoZSxbeDXiXquq3gVbgszlP1T6W6DprvaGMMSa1jCULVY0Bs2UQX3Ynus5aLZQxxqSWbZvFYuBJEfkD0JaYqKqP5yRV+1jc7uA2xpg+ZRssyoF6du4BpcDgCBaJrrODt/BkjDF7Jds7uAddO0Wynt5Q/ZwQY4zZT2V7B/fv2PX52ajq5z70FPWDmN3BbYwxfcq2GurppNch4AJ6PU97IFMLFsYY06dsq6H+mPxeRB4BXshJivpBzNosjDGmT3taSz8ZGJdpIRE5XURWi8haEbkhxfzxIvKiiCwTkZdFZIw3fZaIvC4iK7x5l+1hOrNibRbGGNO3bNssWti5zWIr7hkXfa3jB+4ATgGqgbdE5ClVXZm02O3A/ap6n4h8HDey7aeBduAzqrpGREYBi0TkueRHu36YEvdZDOJbSYwxZq9kWw1VsgfbngusVdX1ACLyKHAekBwspgLf9F6/BDzh7e+9pH1vFpFaoArISbCwO7iNMaZvWVW8iMgFIlKW9H6IiJyfYbXRwKak99XetGRLgYu81xcAJSJS0Wvfc3HP+16XIl3XiMhCEVlYV1eXzaGk1PNY1T3ehDHGDGrZZo+3qGpT4o1XHXRLhnVSXab37n57PXCiiCwGTsSNZhvt3oDISNxzMz7rPa1v542p3q2qc1R1TlVVVXZHkoI9/MgYY/qWbdfZVEEl07rVwNik92Po1d1WVTcDFwKISDFwUSIoiUgp8BfgJlV9I8t07pF43IKFMcb0JduSxUIR+amIHCQik0Tk/wGLMqzzFjBZRCaKSB5wOfBU8gIiUuk9JwPgRmCeNz0P+BOu8fsP2R7Mnoon2ixsbChjjEkp22DxVaAL+D0wH+gAvtLXCqoaBa4DngNWAfNVdYWI3CYi53qLnQSsFpH3gOHAD7zplwInAFeLyBLvb1b2h7V7enpD5WoPxhgzsGXbG6oN2OU+iSzWewZ4pte0m5NePwY8lmK9B4EHd3d/eypxB7f1hjLGmNSy7Q31vIgMSXo/VESey12y9q2YDVFujDF9yrYaqjL5hjhV3cGgega3+2+xwhhjUss2WMRFpHt4DxGZQIpRaAcq6w1ljDF9y7br7HeAf4rIK977E4BrcpOkfc/uszDGmL5l28D9VxGZgwsQS4AncT2iBgXrOmuMMX3LdiDBfwO+jruxbglwDPA6Oz9mdcCyrrPGGNO3bNssvg4cBWxU1Y8BRwB7PhjTfqa766yVLIwxJqVsg0WnqnYCiEi+qr4LHJq7ZO1b9lhVY4zpW7YN3NXefRZPAM+LyA4G0WNVe7rOWrAwxphUsm3gvsB7eauIvASUAX/NWar2sZ6us/2cEGOM2U9lW7LopqqvZF5qYIlbm4UxxvTJHvdDT28oq4YyxpjULFjQ81hVGxvKGGNSs2BBcm+ofk6IMcbspyxYkNRmYdVQxhiTkgULenpDiQULY4xJyYIFNjaUMcZkYsGC5N5Q/ZwQY4zZT+U0WIjI6SKyWkTWisguj2UVkfEi8qKILBORl0VkTNK8q0Rkjfd3VS7TGbcn5RljTJ9yFixExA/cAZwBTAWuEJGpvRa7HbhfVWcAtwE/9NYtB24BjgbmAreIyNBcpdWeZ2GMMX3LZcliLrBWVderahfwKHBer2WmAi96r19Kmn8a8LyqNniPcH0eOD1XCe1us7BgYYwxKeUyWIwGNiW9r/amJVsKXOS9vgAoEZGKLNf90NjzLIwxpm+5DBapst7ez+2+HjhRRBYDJwI1QDTLdRGRa0RkoYgsrKvb88dr2PMsjDGmb7kMFtXA2KT3Y+g1rLmqblbVC1X1CNxzvlHVpmzW9Za9W1XnqOqcqqqqPU5oLO7+W5uFMcaklstg8RYwWUQmikgecDnwVPICIlIpIok03AjM814/B5wqIkO9hu1TvWk5EbfhPowxpk85CxaqGgWuw2Xyq4D5qrpCRG4TkXO9xU4CVovIe8Bw4Afeug3A93AB5y3gNm9aTsRVEbE7uI0xJp3dfp7F7lDVZ4Bnek27Oen1Y8BjadadR09JI6fiqlYFZYwxfbA7uHFtFtZt1hhj0rNggesNZbHCGGPSs2CBq4aybrPGGJOeBQtcNZS1WRhjTHoWLEg0cPd3KowxZv9lwQIvWFi0MMaYtCxY4LVZWDWUMcakZcEC12ZhN+QZY0x6FixwXWf99kkYY0xalkXihii33lDGGJOeBQvcw48sWBhjTHoWLEj0hurvVBhjzP7LskisN5QxxmRiwQJrszDGmEwsWACq2E15xhjTBwsWJEoW/Z0KY4zZf1mwwB5+ZIwxmViwwIKFMcZkYsECd5+FPc/CGGPSy2mwEJHTRWS1iKwVkRtSzB8nIi+JyGIRWSYiZ3rTgyJyn4i8IyKrROTGXKbT2iyMMaZvOQsWIuIH7gDOAKYCV4jI1F6L3QTMV9UjgMuB//WmXwLkq+p0YDbwRRGZkKu02hDlxhjTt1yWLOYCa1V1vap2AY8C5/VaRoFS73UZsDlpepGIBIACoAtozlVCrc3CGGP6lstgMRrYlPS+2puW7FbgShGpBp4BvupNfwxoA7YAHwC3q2pD7x2IyDUislBEFtbV1e1xQuNx7A5uY4zpQy6DRarcV3u9vwK4V1XHAGcCD4iID1cqiQGjgInAt0Rk0i4bU71bVeeo6pyqqqo9TmhMFYsVxhiTXi6DRTUwNun9GHqqmRI+D8wHUNXXgRBQCXwS+KuqRlS1FngNmJOrhLrnWVi0MMaYdHIZLN4CJovIRBHJwzVgP9VrmQ+AkwFEZAouWNR50z8uThFwDPBurhJqY0MZY0zfchYsVDUKXAc8B6zC9XpaISK3ici53mLfAr4gIkuBR4CrVVVxvaiKgeW4oPM7VV2Wq7TGFauGMsaYPgRyuXFVfQbXcJ087eak1yuBY1Os14rrPrtPWDWUMcb0ze7gxjVwWzWUMcakZ8EC13XWgoUxxqRnwYLETXn9nQpjjNl/WbDAe6yqRQtjjEnLggXWddYYYzKxYIE9VtUYYzKxYEGiN1R/p8IYY/ZfFizw2iysGsoYY9KyYIHrOisWLIwxJi0LFiR6Q/V3KowxZv9lWSTWG8oYYzKxYIEbSNB6QxljTHoWLLA7uI0xJhMLFlhvKGOMycSCBa7NwnpDGWNMehYscHdw29hQxhiTngULEr2h+jsVxhiz/7JggdfAbdHCGGPSymmwEJHTRWS1iKwVkRtSzB8nIi+JyGIRWSYiZybNmyEir4vIChF5R0RCuUpn3J6UZ4wxfcrZM7hFxA/cAZwCVANvichT3nO3E24C5qvqnSIyFfe87gkiEgAeBD6tqktFpAKI5CqtccV6QxljTB9yWbKYC6xV1fWq2gU8CpzXaxkFSr3XZcBm7/WpwDJVXQqgqvWqGstVQq3Nwhhj+pbLYDEa2JT0vtqbluxW4EoRqcaVKr7qTT8EUBF5TkTeFpF/T7UDEblGRBaKyMK6uro9SqSqAnYHtzHG9CWXwSJV7qu93l8B3KuqY4AzgQdExIerHjsO+JT3/wIROXmXjanerapzVHVOVVXVHiUyFveChVVDGWNMWrkMFtXA2KT3Y+ipZkr4PDAfQFVfB0JApbfuK6q6XVXbcaWOI3ORSC9WWDWUMcb0IZfB4i1gsohMFJE84HLgqV7LfACcDCAiU3DBog54DpghIoVeY/eJwEpyIG7VUMYYk1HOekOpalRErsNl/H5gnqquEJHbgIWq+hTwLeA3IvJNXBXV1eoaEXaIyE9xAUeBZ1T1L7lIZ3ewsGooY4xJK2fBAkBVn8FVISVPuznp9Urg2DTrPojrPptTiWoo6zprjDHpHfB3cCcauC1WGGNMegd8sEh0nbWBBI0xJr0DPlhY11ljjMnsgA8WwYCPs6aPZEJlUX8nxRhj9ls5beAeCEpDQe74VE5u4TDGmEHjgC9ZGGOMycyChTHGmIwsWBhjjMnIgoUxxpiMLFgYY4zJyIKFMcaYjCxYGGOMyciChTHGmIwkMTbSQCcidcDGvdhEJbD9Q0rO/sSOa+AZrMdmx7V/Gq+qGR81OmiCxd4SkYWqOqe/0/Fhs+MaeAbrsdlxDWxWDWWMMSYjCxbGGGMysmDR4+7+TkCO2HENPIP12Oy4BjBrszDGGJORlSyMMcZkZMHCGGNMRgd8sBCR00VktYisFZEb+js9e0tENojIOyKyREQWetPKReR5EVnj/R/a3+nMRETmiUitiCxPmpbyOMT5hXcOl4nIfvs0qzTHdauI1HjnbImInJk070bvuFaLyGn9k+rMRGSsiLwkIqtEZIWIfN2bPhjOWbpjG/Dnbbeo6gH7B/iBdcAkIA9YCkzt73Tt5TFtACp7TfsRcIP3+gbgf/o7nVkcxwnAkcDyTMcBnAk8CwhwDPBmf6d/N4/rVuD6FMtO9b6T+cBE77vq7+9jSHNcI4EjvdclwHte+gfDOUt3bAP+vO3O34FespgLrFXV9araBTwKnNfPacqF84D7vNf3Aef3Y1qyoqqvAg29Jqc7jvOA+9V5AxgiIiP3TUp3T5rjSuc84FFVDavq+8Ba3Hd2v6OqW1T1be91C7AKGM3gOGfpji2dAXPedseBHixGA5uS3lfT95dgIFDgbyKySESu8aYNV9Ut4L74wLB+S93eSXccg+E8XudVx8xLqiYckMclIhOAI4A3GWTnrNexwSA6b5kc6MFCUkwb6H2Jj1XVI4EzgK+IyAn9naB9YKCfxzuBg4BZwBbgJ970AXdcIlIM/BH4hqo297VoimkD7dgGzXnLxoEeLKqBsUnvxwCb+yktHwpV3ez9rwX+hCv+bksU8b3/tf2Xwr2S7jgG9HlU1W2qGlPVOPAbeqosBtRxiUgQl5k+pKqPe5MHxTlLdWyD5bxl60APFm8Bk0VkoojkAZcDT/VzmvaYiBSJSEniNXAqsBx3TFd5i10FPNk/Kdxr6Y7jKeAzXg+bY4CmRNXHQNCrrv4C3DkDd1yXi0i+iEwEJgP/2tfpy4aICHAPsEpVf5o0a8Cfs3THNhjO227p7xb2/v7D9cp4D9dj4Tv9nZ69PJZJuF4YS4EVieMBKoAXgTXe//L+TmsWx/IIrmgfwV2pfT7dceCK/Xd45/AdYE5/p383j+sBL93LcBnNyKTlv+Md12rgjP5Ofx/HdRyuqmUZsMT7O3OQnLN0xzbgz9vu/NlwH8YYYzI60KuhjDHGZMGChTHGmIwsWBhjjMnIgoUxxpiMLFgYY4zJyIKFMfsBETlJRJ7u73QYk44FC2OMMRlZsDBmN4jIlSLyL+/5Bb8WEb+ItIrIT0TkbRF5UUSqvGVnicgb3kBzf0p6lsPBIvKCiCz11jnI23yxiDwmIu+KyEPencPG7BcsWBiTJRGZAlyGG6xxFhADPgUUAW+rG8DxFeAWb5X7gf9Q1Rm4O30T0x8C7lDVmcBHcXd0gxvN9Bu45yFMAo7N+UEZk6VAfyfAmAHkZGA28JZ30V+AGxgvDvzeW+ZB4HERKQOGqOor3vT7gD94Y3eNVtU/AahqJ4C3vX+parX3fgkwAfhn7g/LmMwsWBiTPQHuU9Ubd5oo8t1ey/U1hk5fVUvhpNcx7Pdp9iNWDWVM9l4ELhaRYdD9fOnxuN/Rxd4ynwT+qapNwA4ROd6b/mngFXXPQagWkfO9beSLSOE+PQpj9oBduRiTJVVdKSI34Z5E6MONHPsVoA04XEQWAU24dg1wQ3Lf5QWD9cBnvemfBn4tIrd527hkHx6GMXvERp01Zi+JSKuqFvd3OozJJauGMsYYk5GVLIwxxmRkJQtjjDEZWbAwxhiTkQULY4wxGVmwMMYYk5EFC2OMMRn9fyRt9/zsLYKvAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# summarize history for accuracy\n",
    "plt.plot(history.history['acc'])\n",
    "plt.plot(history.history['val_acc'])\n",
    "plt.title('model accuracy')\n",
    "plt.ylabel('accuracy')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'evaluate'], loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xd8VfX9x/HX547kZkISwgogYQgGZMhQ616IaN0Dq1b7ax0/q13WVqu1/rq02mlrq1Lce4HUglacdaAMWbJXIISQkL3u/v7++N5cQkhuwriE5H6ej0ceuffcc8/9fu8597zP93uWGGNQSimlABydXQCllFKHDw0FpZRSURoKSimlojQUlFJKRWkoKKWUitJQUEopFaWhoFQHiciTIvLrDo67RUTOPNDpKHWoaSgopZSK0lBQSikVpaGgupVIt83tIrJcROpFZKaI9BGReSJSKyLzRSSr2fjni8hXIlIlIh+IyFHNXhsvIksi73sJ8LT4rPNEZGnkvZ+KyJj9LPP1IrJBRCpEZI6I9I8MFxH5k4iUikh1pE6jI69NE5FVkbJtF5Ef79cXplQLGgqqO7oEOAs4Evg6MA/4GdALu8x/D0BEjgReAH4A5AJzgX+JSJKIJAGzgWeAbOCVyHSJvPcY4HHgRiAHeBSYIyLJ+1JQETkduA+4HOgHFAIvRl6eApwcqUdP4AqgPPLaTOBGY0wGMBp4b18+V6m2aCio7uivxpidxpjtwH+Bz40xXxpjfMAsYHxkvCuAfxtj3jHGBIDfAynA14DjADfwZ2NMwBjzKrCw2WdcDzxqjPncGBMyxjwF+CLv2xdXAY8bY5ZEyncncLyIDAYCQAYwEhBjzGpjzI7I+wJAgYhkGmMqjTFL9vFzlWqVhoLqjnY2e9zYyvP0yOP+2C1zAIwxYWAbkBd5bbvZ84qRhc0eHwHcFuk6qhKRKmBg5H37omUZ6rCtgTxjzHvA34CHgZ0i8piIZEZGvQSYBhSKyIcicvw+fq5SrdJQUImsGLtyB2wfPnbFvh3YAeRFhjUZ1OzxNuA3xpiezf5SjTEvHGAZ0rDdUdsBjDEPGWMmAKOw3Ui3R4YvNMZcAPTGdnO9vI+fq1SrNBRUInsZOFdEzhARN3AbtgvoU+AzIAh8T0RcInIxMLnZe2cAN4nIsZEdwmkicq6IZOxjGZ4HviUi4yL7I36L7e7aIiKTItN3A/WAFwhF9nlcJSI9It1eNUDoAL4HpaI0FFTCMsasBa4G/grswu6U/roxxm+M8QMXA9cBldj9D683e+8i7H6Fv0Ve3xAZd1/L8C7wc+A1bOtkKDA98nImNnwqsV1M5dj9HgDXAFtEpAa4KVIPpQ6Y6E12lFJKNdGWglJKqSgNBaWUUlEaCkoppaI0FJRSSkW5OrsA+6pXr15m8ODBnV0MpZTqUhYvXrzLGJPb3nhdLhQGDx7MokWLOrsYSinVpYhIYftjafeRUkqpZjQUlFJKRWkoKKWUiupy+xRaEwgEKCoqwuv1dnZRuhyPx8OAAQNwu92dXRSl1GGgW4RCUVERGRkZDB48mD0vaqliMcZQXl5OUVER+fn5nV0cpdRhoFt0H3m9XnJycjQQ9pGIkJOToy0spVRUtwgFQANhP+n3ppRqrtuEQnvqfUFKqr2E9aqwSinVpoQJhQZ/kNJaL/HIhKqqKv7+97/v8/umTZtGVVXVwS+QUkrtp4QJBWjqJjn4qdBWKIRCsW+GNXfuXHr27HnQy6OUUvurWxx9tC/i0Xl0xx13sHHjRsaNG4fb7SY9PZ1+/fqxdOlSVq1axYUXXsi2bdvwer18//vf54YbbgB2X7Kjrq6Oc845hxNPPJFPP/2UvLw83njjDVJSUuJQWqWUalu3C4X/+9dXrCqu2Wt4IBTGHwyTmuxiX3etFvTP5BdfH9Xm6/fffz8rV65k6dKlfPDBB5x77rmsXLkyepjn448/TnZ2No2NjUyaNIlLLrmEnJycPaaxfv16XnjhBWbMmMHll1/Oa6+9xtVX6x0WlVKHVrcLhcPB5MmT9zju/6GHHmLWrFkAbNu2jfXr1+8VCvn5+YwbNw6ACRMmsGXLlkNWXqWUatLtQqGtLfryOh/bqxo5ql8mbmd8d6WkpaVFH3/wwQfMnz+fzz77jNTUVE499dRWzwtITk6OPnY6nTQ2Nsa1jEop1ZoE2tEcPxkZGdTW1rb6WnV1NVlZWaSmprJmzRoWLFhwiEunlFId1+1aCm1pOkcrHoek5uTkcMIJJzB69GhSUlLo06dP9LWpU6fyyCOPMGbMGEaMGMFxxx138AuglFIHiZgudjLXxIkTTcub7KxevZqjjjoq5vsq6v0UVTYwsm8mSS5tIDXXke9PKdW1ichiY8zE9sZLwLVj1wpBpZQ6lBImFOJ36ppSSnUfCRMKmgpKKdW+hAkFzQSllGpfwoSCUkqp9iVMKGhLQSml2pcwobD7RIXOLUZ7Bg8ezK5du/brvbNnz2bVqlUHuURKqUSSOKEQdZinwgHQUFBKHaiECYVD0X307LPPMnnyZMaNG8eNN97Iww8/zE9+8pPo608++SS33norABdeeCETJkxg1KhRPPbYY3tNa8uWLYwePTr6/Pe//z333nsvADNmzGDSpEmMHTuWSy65hIaGBj799FPmzJnD7bffzrhx49i4cSMbN25k6tSpTJgwgZNOOok1a9bEsfZKqe6g+13mYt4dULJir8Gp4TBDAmGSk5y7u5I6qu/RcM79MUdZvXo1L730Ep988glut5ubb76Z9PR0Xn/9dR544AEAXnrpJe666y6gY5fTbsvFF1/M9ddfD8Ddd9/NzJkzufXWWzn//PM577zzuPTSSwE444wzeOSRRxg+fDiff/45N998M++9996+1V0plVC6Xyh0knfffZfFixczadIkABobG+nduzdDhgxhwYIFDB8+nLVr13LCCScAHbucdltWrlzJ3XffTVVVFXV1dZx99tl7jVNXV8enn37KZZddFh3m8/kOtJpKqW6u+4VCG1v0jd4Am3fVMzQ3nbTkg19tYwzXXnst99133x7DZ86cycsvv8zIkSO56KKLEJEOXU7b5XIRDoejz5u/ft111zF79mzGjh3Lk08+yQcffLBXecLhMD179mTp0qUHt6JKqW5N9ykcJGeccQavvvoqpaWlAFRUVFBYWMjFF1/M7NmzeeGFF7jiiiuAjl1Ou0+fPpSWllJeXo7P5+PNN9+MvlZbW0u/fv0IBAI899xz0eHNL+GdmZlJfn4+r7zyiq23MSxbtixOtVdKdRcJEwrRWIhTKhQUFPDrX/+aKVOmMGbMGM466yx27NhBVlYWBQUFFBYWMnnyZMBeTjsYDDJmzBh+/vOft3o5bbfbzT333MOxxx7Leeedx8iRI6Ov/epXv+LYY4/lrLPO2mP49OnTefDBBxk/fjwbN27kueeeY+bMmYwdO5ZRo0bxxhtvxKfySqluI2EunV3nC7KprI4hvdJI97jjWcQuRy+drVT3p5fObkHPaFZKqfYlTCgopZRqX7cJhfa6weJ5O86urKt1Hyql4qtbhILH46G8vFxXcPvIGEN5eTkej6ezi6KUOkzE9TwFEZkK/AVwAv80xrR6EoGIXAq8AkwyxixqbZxYBgwYQFFREWVlZW2O4w+GKa31EaxIIsXt3NeP6LY8Hg8DBgzo7GIopQ4TcQsFEXECDwNnAUXAQhGZY4xZ1WK8DOB7wOf7+1lut5v8/PyY46wqruH65/7LI1dPYOpRfff3o5RSqluLZ/fRZGCDMWaTMcYPvAhc0Mp4vwIeALytvHbQOCI1DWsXk1JKtSmeoZAHbGv2vCgyLEpExgMDjTFvEoOI3CAii0RkUawuolickT3NGgpKKdW2eIZCa5cija6RRcQB/Am4rb0JGWMeM8ZMNMZMzM3N3b/CREIhFNZQUEqptsQzFIqAgc2eDwCKmz3PAEYDH4jIFuA4YI6ItHvG3f5w6CGpSinVrniGwkJguIjki0gSMB2Y0/SiMabaGNPLGDPYGDMYWACcvz9HH3WE06HdR0op1Z64hYIxJgjcArwNrAZeNsZ8JSK/FJHz4/W5bXFo95FSSrUrrucpGGPmAnNbDLunjXFPjWdZHJGWgjYUlFKqbd3ijOaOaNqnENJUUEqpNiVMKOghqUop1b6ECQWJhkInF0QppQ5jCRMKTd1HYU0FpZRqU8KEgh6SqpRS7UuYUNAzmpVSqn0JEwpOPSRVKaXalTChoIekKqVU+xIoFHSfglJKtSfhQkEzQSml2pZAoWD/645mpZRqW8KEgh6SqpRS7UuYUIie0awtBaWUalPChALY1oJmglJKtS2hQsEh2n2klFKxJFQoiIiep6CUUjEkVCg4RfSQVKWUiiGhQsEhekiqUkrFklih4BDdp6CUUjEkViiI6CGpSikVQ4KFgt55TSmlYkmoUHBq95FSSsWUUKEgoqGglFKxJFQoOEUIhzu7FEopdfhKqFBwiN5kRymlYkmsUNB9CkopFVNihYKe0ayUUjElWCjoGc1KKRVLYoWCdh8ppVRMiRUKekiqUkrFlFChoIekKqVUbAkVCqKHpCqlVEwJFQr26CMNBaWUaktcQ0FEporIWhHZICJ3tPL6TSKyQkSWisjHIlIQz/LoPZqVUiq2uIWCiDiBh4FzgALgylZW+s8bY442xowDHgD+GK/ygB6SqpRS7YlnS2EysMEYs8kY4wdeBC5oPoIxpqbZ0zQgrmtsPSRVKaVic8Vx2nnAtmbPi4BjW44kIt8FfgQkAae3NiERuQG4AWDQoEH7XSA9JFUppWKLZ0tBWhm21xrZGPOwMWYo8FPg7tYmZIx5zBgz0RgzMTc3d78LpIekKqVUbPEMhSJgYLPnA4DiGOO/CFwYx/IggrYUlFIqhniGwkJguIjki0gSMB2Y03wEERne7Om5wPo4lke7j5RSqh1x26dgjAmKyC3A24ATeNwY85WI/BJYZIyZA9wiImcCAaASuDZe5QF7SKo/FM9PUEqpri2eO5oxxswF5rYYdk+zx9+P5+e3JHpIqlJKxZRQZzQ7HXpGs1JKxZJQoeAQ0WsfKaVUDAkWCughqUopFUOChYIefaSUUrFoKCillIpKqFDQq6QqpVRsCRUKIhDWVFBKqTYlVCg49SqpSikVU0KFgt2n0NmlUEqpw1dChYKe0ayUUrElVCg49R7NSikVU0KFgp7RrJRSsSVWKOghqUopFVNihYIekqqUUjF1KBRE5PsikinWTBFZIiJT4l24g03PaFZKqdg62lL4H2NMDTAFyAW+Bdwft1LFiZ7RrJRSsXU0FCTyfxrwhDFmWbNhXYae0ayUUrF1NBQWi8h/sKHwtohkAF3uItRO7T5SSqmYOno7zm8D44BNxpgGEcnGdiF1KQ6HHpKqlFKxdLSlcDyw1hhTJSJXA3cD1fErVnzoZS6UUiq2jobCP4AGERkL/AQoBJ6OW6nixCHoGc1KKRVDR0MhaOza9ALgL8aYvwAZ8StWfDhE9NpHSikVQ0f3KdSKyJ3ANcBJIuIE3PErVnzoGc1KKRVbR1sKVwA+7PkKJUAe8GDcShUnjshBtHpYqlJKta5DoRAJgueAHiJyHuA1xnS5fQpOsamgh6UqpVTrOnqZi8uBL4DLgMuBz0Xk0ngWLB4ckaaCHpaqlFKt6+g+hbuAScaYUgARyQXmA6/Gq2DxEGkooJmglFKt6+g+BUdTIESU78N7DxvafaSUUrF1tKXwloi8DbwQeX4FMDc+RYofRyQU9LBUpZRqXYdCwRhzu4hcApyAvRDeY8aYWXEtWRw07VPQTFBKqdZ1tKWAMeY14LU4liXu9JBUpZSKLWYoiEgt0NoaVABjjMmMS6nixOnQfQpKKRVLzFAwxnS5S1nEIqLdR0opFUtcjyASkakislZENojIHa28/iMRWSUiy0XkXRE5Ip7liXYfaUtBKaVaFbdQiFwf6WHgHKAAuFJEClqM9iUw0RgzBnvOwwPxKg/oIalKKdWeeLYUJgMbjDGbjDF+4EXsVVajjDHvG2MaIk8XAAPiWB49JFUppdoRz1DIA7Y1e14UGdaWbwPzWntBRG4QkUUisqisrGy/C+Ry2lAIhjQUlFKqNfEMBWllWKtr48jd3CbSxpVXjTGPGWMmGmMm5ubm7l9pCj9j7Lq/4iREvT+4f9NQSqluLp6hUAQMbPZ8AFDcciQRORN7baXzjTG++JVmIUPXPIIHP/W+UNw+RimlurJ4hsJCYLiI5ItIEjAdmNN8BBEZDzyKDYTSVqZx8LhTACKhoC0FpZRqTdxCwRgTBG4B3gZWAy8bY74SkV+KyPmR0R4E0oFXRGSpiMxpY3IHzuUBbCjUaSgopVSrOnyZi/1hjJlLiwvnGWPuafb4zHh+/h6aWgqiLQWllGpLl7v89X6LthQC2lJQSqk2JE4ouG0oJOuOZqWUalPihILLdh9luAI06CGpSinVqgQKBdtSyHKHtPtIKaXakDihEOk+ynSFdEezUkq1IXFCwdUUCkHqdJ+CUkq1KnFCwd20TyGoLQWllGpD4oRCpKWQ7gzqtY+UUqoNiRMKkZZCmjOoO5qVUqoNiRMKkZZCmgS0+0gppdqQOKEgAi4PqY6AnrymlFJtSJxQABsK4qfeH8ToLTmVUmoviRUK7hQ8EsAYaAxoa0EppVpKrFBwefDgB9CdzUop1YrECgV3CsmRUND9CkoptbfECgWXh6TIHT/1CCSllNpbYoWCO4UkY1sKFfX+Ti6MUkodfhIrFFyeaPdRaa2vkwujlFKHn8QKBXcK7rANg9JabycXRimlDj+JFQouD46Ql/RkF2XaUlBKqb0kVii4PRDwkpuRrN1HSinVisQKBVcKBG0oaEtBKaX2llih4PZA0EtvDQWllGpVYoWCywOBRnLTkyit0R3NSinVUuKFAoa+6U7q/XqvZqWUaimxQiFyo52+qfYKqdqFpJRSe0qsUIjcaKdPig0FPQJJKaX2lFihkJQGQL9UezG8osqGziyNUkoddhIrFFKyAchLasTlEDaU1nVygZRS6vCSWKGQlgOAy1tJfq801u3UUFBKqeYSKxRSbSjQsIsj+2SwobS2c8ujlFKHmQQLhV72f0M5w3qnU1jRgFdvy6mUUlGJFQpJaeBMhnrbUjAGNpZpF5JSSjWJayiIyFQRWSsiG0TkjlZeP1lElohIUEQujWdZIh8Iab2goYLhfdIBWK/7FZRSKipuoSAiTuBh4BygALhSRApajLYVuA54Pl7l2EtqNjTsYnBOGi6HsF73KyilVJQrjtOeDGwwxmwCEJEXgQuAVU0jGGO2RF4Lx7Ece0rtBfW7SHI5GKxHICml1B7i2X2UB2xr9rwoMmyficgNIrJIRBaVlZUdWKlSc6ChHIAj+6TruQpKKdVMPENBWhlm9mdCxpjHjDETjTETc3NzD6xUab2ioTCsdwaF5fV6BJJSSkXEMxSKgIHNng8AiuP4eR2TmgO+Ggj6ObJPOmEDm8rqO7tUSil1WIhnKCwEhotIvogkAdOBOXH8vI6JnsBWzvDeGQC6s1kppSLiFgrGmCBwC/A2sBp42RjzlYj8UkTOBxCRSSJSBFwGPCoiX8WrPFFpTSew7SK/Vxoet4NFWyrj/rFKKdUVxPU8BWPMXGPMkcaYocaY30SG3WOMmRN5vNAYM8AYk2aMyTHGjIpneQDoeYT9X76BJJeD00b05q2vSgiF92t3h1JKdSuJdUYzQO+jwOGCHcsAOHdMP8pqfSzcUtHJBVNKqc6XeKHgSrbBsGM5AKeP7I3H7WDWku2dXDCllOp8iRcKAH3H2paCMaQmubhwXB5vLNtOZb2/s0umlFKdKjFDod9YaNgFtTsAuO6EwXgDYV5cuK2dNyqlVPeWoKEwxv6PdCGN9K9m2mB45rMtBEOH7oobSil1uEnMUOgzGhDbhVS/Cx6fwm8CD1Jc7eU/q3Z2dumUUqrTJGYoJKdDzjAoWQ7LXgSgp38ng7JTefzjzZ1cOKWU6jyJGQpgu5CKl8KXzwAgbg/XfW0wiwor+XKrnsymlEpMCRwKY6GmCMrWQHpfqNrG5RP6k+Fxcf+8NZRUezu7hEopdcglbij0jexszhkGp9wO4QDpvlJ+MnUkS7ZWcuHDn+AL6tVTlVKJJXFDIW8CZA+FKb+2wQBQuYVrjjuCGd+cSEmNl38t29G5ZVRKqUMsnndeO7x5MuF7S+zjisjO5cotkH8SpxyZy5F90nno3fU4HXDhuDxEWrs9hFIq4YXD4Og+29fdpyYHoscA+//fP4K18xARfjbtKBoDIX740jIWdvQqqmv+DX8/HoK++JVVKXX42LUB7suDzf/dv/fvWA6f/AVKVh7cch0ADQUApxuy8iHkh7k/AWM4dURvPrr9NDI8Lp5ZUNix6Wx8D0pXwa51B7d8696Gla8f3Gl2lqAP/PtxU6NtC+Gtn4Hp4NVsi7+Ezx7ec5i/AV7+JpSs2PfPT1Slq+25PM0tfhJm/a/dQj5UfHG6be62L2DWTVDX4ja/2xZCbSvnLIUCe4679DkINMCKlyHoh1e/DSte7dhnVxbCP8+Ad+6B/9zd9njxqnsbNBSaXP0anPgjqN4aXamnJDm5bMJA5q7YwcV//4TNu9pZme1ab//vPMi3hXj3lzD3x/v3IzRm394XDtsFPx6WvwJ/GAmPndbxlXuTz/4GCx6OXpokqmorhFs5IOCD38HbP4Pakt3D1s2DVW/A8pf2vezNffxnG1ChYOzxNv8XalrcbLCuDN6/z87TpnL76uCxU2Hdf+xzf/2e82DzR/D8dLvSaRL0736/MfDmj+D93+75WeUboWiRfVyzw05zzq0w43T4+E+tLxclK+DfP4a6UmiogEdPtvNs6Qu7y/LmD2HZ87BqVtt13/Kx/fxY2lsGqovsMvPVLLhvAMy9HQKNsd/TlvKNsPR520X8xQxbf2PsVvqyF+zKuen3u/pfMPNM+FOBXcF//Cf45CH72nu/gj8eBR89CNuXwIpX7PC182D+L2DlqzD//1pfJhur7Hx/+kIbth/+DhAouBAKP7Xf7YzT4Z9n2nljDLz3a7h/EGx8f//qvR8Sd59CSzlDYeK34OM/wob5kDsCgBtPGUKNN8BbK0v426z3+cPYHVBwAaT33nsa5Rvs/46EQtMPsr2+SH+9XYBMyG79DpiwD5UCFvwDPn0IvrfUXjLc6bIroecug5Nvg2Fn7jn+J3+GRU/A95eCw2mHBX12YR59CeQdE/vzanbYcU+7E3oMtFtAIjDhW/D2nXZau9baH2B6b1g9B46+DNwpbU8zFIRNkR9FyQq7wnr+cjj/b/DSVXDC9+Gk28CZbL9Pf8Pu8Td9CGOvsI9XvGb/b/287c+q2mq3hI+cCgMn7/36V7Psjx/sGfGn/ASGnLL3eDtXwdPn2wMavv2O/Q4AZt0IG9+1j/NPhiGnwqrZdt4uewGGnQEzp9jvfsqv7Yp5wT9g2wK7XI6cZr+DJ8+zW6jn/82WedFMO81T77Sftfgp+Nf37LCLHoM3boaswXYZ7Xs0zL/XrnTP/cPuMq97G1662raYdyyFY661jzP6wYf323q++j/2AA2HC9651/6vKbZXHv7gd3DiD2HARHj2Ejvs+vdteXx1sP5tSEqH4VNg50p4Yhq4U+HSx6FsNfQbZ08mdbphym/gleugaCEkZUBKFnzxmA25q161J57mDIOeA+G162Hzh/b7PHIqjLrIfn9bPrYraBOC7YttHXuPgvL1tl4NFXZlm3+KbeH/8ww45pv2u+s3zgbQRw9CxSY7fnKGDRZXsl1Zv/drO83hU2D9f2DB36HP0bBzhW3ZZ/S15a4phiVP2emUrQFXCjxxDjRWwvG37F4GnrkYXB7w18KGd+zG6UcPgsNt59eQU3cvR3EkZl+32DrZxIkTzaJFi+L3AX+bDEmpcNVrdmW68J8QaMTrysATqAKg/KhryLnib3u+z18Pv+1vHw8707Y8wKa9MXblddTXYdw37Az++M/2x3bSjyB7iL1N6Nzb4et/gT4Fu6db+Bk8MdU+PvVOOPWO3a8ZYwOod0Hr4RIKwl/GQM12mHQ9fPks3PSxXWHO/THkTYTr393zPTPOgO2L4Nvz7Q+r/3i7wvn3bZCcCdfMsj/6tsy5FZY8DUeeA8dcAy9+A8RhVwa+Gpj6O3jrp3D2fXZFsORpe9mRs3+758q1tsSucNJ6wdYF8PjZdvjpd9sf+6YPICUbGivAnWbnWe5IOP67UPgJfPpXQGDsdDj/r7bu835ih2Hgurl2Q6Bys92C7jvGrgTm3AJr59rPuugxG1iLH4ehp9stw1k3Qf9xdrof/A4aymHaA7D2LcjsB6fdBZ4e8PwVdssPA9NfgOFn2RMl3/yhrcN//2incdpd9jva9rldBqb9Hl79Vuvf7aiLbF1mnAHeKkjvYwPBV2tXfGC/x5IV9rNTs21fdUbf3S2so86Hy5+283PxE3DLIvs9VGyGR0+B7MEw6Tt2PrrTbFhP+RXM/l/oMcheSPL698BbDS9eZZ835/LY5X/Nm/b5NbNg8Enw1Ndh62d22DkPwPKX7Va7021XspVb7Aq0sdKuBE+/y/5OMvPs8jv9BTBh+930Hx/5vnrBhOvgv7+HAZPtvKwvs8vTxP+xv6ceefZ3NvR0W7b5v7DL44hpu8t41avQazjM/i4UfmyndelMuwHwzj12nNyj7PIKcMWz9nnpKvt5oy6CZy6ygXTiD+BPo5t9L2Lr6EwGVxJcPMMuH89dCqMvtcFvQvC7wTZ4vjUPXvuOLWvVVjjybBhxDrzxXfs7P/1uGHlu27+/GERksTEmxo83Mp6GQgtLn4fZNwOR72XUxZB1BKH6Cj4q74Fj22dMMF/BlF+Rlt0fCXptE7RsDQS9kNwDktLgttVQvR2enAYDj4PlL0L/Y+A78+H3w+1WW9BvtyrAbjEFGuxWyzff2L1F8NnDthskK98uXN9+B1J62qborJtsl8ik79iVib/ebgVlD7UL/uIn9u6rPOnHtgulqtAuhN951/7AqrbZhfuJc+xCmjnAntx39m/tlmpKll2pN1TYrfLKLXZrsu8YOO9PdiX0+SOw/h37Q67eahf+1F4w9X60QHU6AAAWMklEQVR4/jL7+LY1dme8OGxZh5xmf1y1O+CCv8P4q+wW7CMn2ZXduCvtZ66dC2m5dgVStRU8Pe2KMaO/fW9qjl1BN803Tw+7MipaaLfkvnzGfi8Tv7X7O8mbYLsVvNW76xcO2vpt/dxuXY65zAZX0wpr0Nfgyud3P//bZKgvhbTetrwpWXbatcV25ffFDFu+tF72O+s7xn7nr19vuxxCkYMS+h5tV+ZpubbsY6bblWHVVtuKGH6WnW+9C2wL5erX7DLz+BQbDpf80654m0IP7Ap5/r12/JzhdoOj31h7mZe6UvjzGBtwJ/7QbvVWFcKNH9ll87XrbT/5mOl2/v5hhF1eLn/arqjAzpfti23dvpplV3JzbrXLxREn2i3jYKO92+GOpfD1h+Cr122gg11B1pXCf+6yGw3++sjvINJNO+hrMP05G/Ijz7O/ifn/Z1vzGf1tq7dqq936v/FDuxGxaja8cQv46+z8vWa2PdIQbJfO41PtRtc5D8DMs6B8E9y+Adweu5FVu8OGiIid9p+Ptt/H9e/blkRjJdy21i6HbSlZab/ztF62JVy2Fq58yYZ00+/amD23+t/4LjiT7He96Ak7P3JHwhXP2GV90Uw7/0/4AQw/s/XPbYeGwoHY+L7dGh12BhzxtT1eWvrxXMbNv3KPYaHU3jgbSu2TsVfaboAfr7d9kQv+vue0r3jOthoumWn7Eis3262mLx6FEefavtoJ10HAa7fUS5bbH9+0B+HlayNbz7k2eMo32K3rDfPh3D/arp+qrZGF2mlX6rlHQa9htp/UmRTZxxCwK4h3fmG3SOpKALELqQnb8UJ+oisYcdgfV85Q2+1Uusr+iHsXQNEXduVb+KntDsoabFdQX8yw5TrjHrsS+fhPduU17ht2pfzpX23w3PiRrcuzF+/eqq3baVfOR19qt/DDITjtZ/a7WP0v+76pv7U7jU+7GwafCNn59ocYaLBnqLuS7Q7SF6bbkDvuu3D2b2xwPDgsslKvsHUbeZ4NmMw8G+7Xvmmn8/QFtq6DjrfhMfgEuPARuwJpsn6+nXfn/9WuxN+604bfsTfaeVNTbLtB/A1w6k9tC8rpsvsPnr/cbtEOmASDjoWHxtv5ceWLMPQ0O/1w2IZVY6XdSi5ZARf+A8Zcbl//Yobt6ug/3vY9B+rh2P+FnoPguP+1ofDJn2HyDXYZam7R47YLyFdtn1/5EoyItEprS+zW79T7bLfF1s/tfOo7OvZvJ+iHpc/ajZuQ367E60pg/NV246WhwnYR5R0Dg46zgfzXCbYbJaMvZPaHeT+1K9KbPt6z1Qw2OGbdBJO+DUecYMfL7G9XuE22LrBdTWffBxl92i5rQ4UNgT4x7gI876c2SMd9w47fWGl/B12QhkK8GMO2525hqa8fb2304cfF566JLHd8w75+/fsw4zT7I1j5ug2V0jVw5BT7I+w5yK4obt9ot/ibhMN25TX3dtv/mJJlm5y1xXZaFzxsF/aVr9sV1bYv4PKnbJP10ZPtsHAQzvqV7RMHuxI44gTbH/3OL2xXyH/u2t2FsPUz24/Z+yi48O92J1c4COOusq2MSx+3XSZjr9y9MjDG/jCS0mx4vHGL3fobdbFdgTRtlcUS8Nqt5pyhtvUDdh/DjDPsDzSzv63z0NOgbJ3tY82bAB8+AO//xn4XY6+039PoS+yWdVvK1tot04n/s/uzNr5nuxjm/cSG5qk/bf29/nobSqMvtZ/hPMi74Hy1tp+6yeKnbNAOnNT2e4J+2w3RmifPs62K29ba1gDYsH7iHNsd2toWZqDRLlcmbDeCOkMoaDeAmract31hQ6ng/M4pTzeloXAI/PO/m6huDPDRujIo/pJ7x9Uy7rI7kVeutU09Tw8bEln5doH/02i79T7sLLg6xmFrjZV255rTZXfMOpP23sEU9O1uwq583W5F5k203VPhkB2/aUdx9D1+24oZddHulXd1kQ2gpDR7nkXlFtuC2TDf7nRrb8eWMfbzDsYKs2WTuqXqIlv+E3548FfQ3cGO5bYve+jpew4vWwe5R3ZOmdRhQ0PhEGrwB/nOU4v4dGM5A7JS6Bfczj9yXiJj2i9JHtTsaJ2KyI6wvmP27II4UOGQ3XE45go44viDN12lVLehoXCI+YNhnl1QyCcbdlFU2cjanbU4BK457gi+e/owemccxBBQSql9pKHQiep9QWZ9uZ2V26t5adE2BCjon8kxg7K4+JgBjBvYs91pKKXUwaShcJjYUFrHv5YVs3BLBcu2VeENhhmd1wOPy8FF4/O4ZMIA3E49sVwpFV8aCoehGm+A37y5ms3l9VQ1+Fm3s44hvdK4/uQhpCY5GZCVSn6vNLJS3XpVVqXUQaWhcJgzxvDu6lJ+99Ya1pfuecErt1M4Z3Q/vnfGcIb1TscbsGeretz2aCJvIMS6nbWMGaDdUEqpjuloKOhxfZ1ERDizoA+njshlU+RCe9sqGti8q56tFQ28uriIN5cXMyg7leIqL06HMCg7FV8whAEKyxv46dSR3HTKEAIhw7+WFTM5P5uB2al7fVYgFNYuKqVUh2hL4TBVXufjqU+3sLm8gb6ZydT7Q+yoakREKKv1kZ2WxIfryuiZ6sYpQnm9n56pbu6YOpJzx/Rj3soStlc2snlXPZ9s2MW/bj0Rhwh9MpO1a0qpBKTdR91cIBRm9pfbWbK1Em8gzInDejHz482s2lGD2ykEQrvna5LTgcftoMYbpH8PD/m5aRSWNzB2YE++c2I+a0pqqWkMkJnipqohwICsFAZmp9IzxU2NN8Do/j1wOGyQLC6sYFB2GrkZMa79opQ67GgoJCBjDF9uq2LO0mL69/Rw4rBcKur9bK1o4L55q7nmuCPYUl7P9spG+vbwsGBTBdWN7d874cg+6fTrkUKPFDdzlhUzJDeNf35zIqlJLv7y7npWFVfzoykjWFdSyykjcklLdtE300N1Y4CeKe5ooCilOo+GgtqDMWavbqNab4CXFm5jQFYq4wf1pNEfIic9ie1VjWyraKSywU8wZHh18Ta8gTAbSus4cXgvPlpXRjC8e7lJdjnwBfe8YYvTIYTChryeKfTt4cEbCNEYCFHTGMAYGJCdSr9MD8P7pLN0WxUF/TLJzUhm4uBsXA7ho/VlvP3VTnZUNZLfK43zx/UnKzWJtGQXOWlJ3DdvNdMnDeKsgj7M/nI7O2t8TBycxdeG5uxRzw2ldfx7+Q6OyEnl7FF9eX9tKcfmZ5OTnrzH92CATI87OiwUNjg1zFQ3oqGgDrpw2OBwCEu2VrK2pBZ/MEx+rzSy05L4YG0pZ4/qy5KtlYTCsLWigZ6pbpYUVlLvD5LiduJxO8nwuAmHDcXVjWwsraO42svQSHdW86ABGDugByP6ZvDZpnK2Vey+45ZD7I76UNjs1VWWk5bEwOxUDFBS3UhFvT/6emqSkwZ/iF7pSfRKTyY3I5l6X5BlRdU4RTh1RC4F/TMpqfYye+l2bj51GJMGZ1NYXk9xVSOltT7GDuxJhsdFaY2PI/tksLK4mvdWl3L80ByOG5LDhrI6VhXXcMygnpRUeynon8mIvhlU1PtxOoTqxgCN/hAet5ORfTMQEcLGkJOWFHNfT2W9n5XF1Rw3JIfqxgDZqUnRFpgxBl8wjMshuGIcUBAKm+h31/S739/9S3W+IMVVjRzZJ6P9kdVh4bAIBRGZCvwFcAL/NMbc3+L1ZOBpYAJQDlxhjNkSa5oaCt2HMYaqhgBZaUkEQ2GqGgN8vH4XbqeD8YN60r+nvRtbKGwormqkzhdk5fZq5q0s4e5zj2LF9mqWFFZyyohcvja0F3OWFrNkayVFlY2EwoYBWSlkpri54eQhrCiqZsZ/NzF1dF/eW1OKQ4SSai+eJCcnDetFjTfAh+vK2FrRgABjBvRk6baqaFkdQnSfS0tDctPYvKs+enfJFLeTxkArt2OMoV8PD+nJLhr8tkXV6LfvT0t2kpbsYletj3p/iEyPixpvkIxkF75QmGG56ZTV+Sir9eEQ6JPpoVd6Mh63A4/bSbLLSWqSkySXg38v30Gy20F2WhJlNT58oTCj+2cyZVRf1pXUsnx7NS6HUFrrY1huOgX9M6lpDNArI5m3VpZgMFw4Lo8Mj4sZ/91MWa2Ps0f1oc4XZFB2KvW+EJ9vLifJ5aB/jxSOzutByBicIvTOTGbeyhJKa3zcevowvIEQhRUN7KzxctLwXPr28LCz2os3ECIzxU3YQHVjgKG5aYwflEV5nY8GfwhvIIQ3EMYXDJHhcbOrzkeyy8Gw3umUVHspqfGS6XETDBvSk12s2lFDdpobp8NBMBRmzIAerNtZR4M/xKDsVLaU15ObnkxWWhJLt1YyKCeVpVurOGVELscMyqLOF6S01kdWahJhY9hQWkeyy8HIvpn4giE+XFfGsfk59O3hIRQ2CLBpVz31viBpyS7Sk12ke1ykup17daOGw4bN5fV43E7qfUHSk10kuxzU+0JkprgoLG+goH9m9MjBcNgQMma/jyTs9FAQESewDjgLKAIWAlcaY1Y1G+dmYIwx5iYRmQ5cZIy5ItZ0NRRUPDX6QzT4g2SnJbF2Zy1ltT4GZaeS1zMFp0P4ZEM5BsOIvhls2FmH0yEcOySHslofy7ZVcWSfDAZkpbBkayV9e3h4d3UpvmCI/F7phMJhMlPcpCW5qPEGWLezDqdAyMCXWysJhgypSU5SkuyKHKDeH6LeZ1taE47I4oN1ZRT0y2RHdSNJTicri6vplZ7EqP498AVCbK/yUlHvwxsI4w2G8AXC1PuDlNf5OX1kbzI8LqoaAmSnJZGS5OStlSVsrWigR4qbSYPtPQly0pJYV1rL6h019EhxU1ZrW0W5Gcl8smEXYQOTB2czJDeNf6/YwaDsVHZUe/G4HBxzRBZOh1BU2ciKomocDntVeH8oTEG/TMLGsKakFrAtt54pboqrvZ02v2MR2fM20s2fOwSSXA68Adtt2t6GgAikJdmVfq03iMFgDHu1jlt+lsftIDcjmTpvkKrGAPdddDTTJw/az/p0figcD9xrjDk78vxOAGPMfc3GeTsyzmci4gJKgFwTo1AaCkodPOGwodYXJNPjarMryRsIkexyICKU1njxBsIMytn7fJjW3icCXn+YOn+QvJ4p+IIhVhRVMzA7ld6RI9iWbqsibAx9Mj2kJrmobPADkJWaxPKiKr4qrqFvpod0jyvaDZnsclDVGCAr1U2dL8iOKi/ZaUkMzE6h1htERKis91PQP5M6XzDaXfb+mjJyM5LJy0qhvM5PQb9Miqsb2VHdyPiBWRRWNHBU3wzeXrWTndVeMjwuemcms72ykbCBCUdkUe8LsrK4hl11PqaN7seK7dWU1/lITbanfeX1tK21Ol+QOl+Qel+QOl+IOm8QbzBEhseFUwQD5PdKi24M7Krz4QuGyfC42FXnZ2huGiuKqimr85HhcZGdmsSZBX32+6TVwyEULgWmGmO+E3l+DXCsMeaWZuOsjIxTFHm+MTLOrhbTugG4AWDQoEETCgsL41JmpZTqrjoaCvE8zbW1zY6WCdSRcTDGPGaMmWiMmZibm3tQCqeUUmpv8QyFImBgs+cDgOK2xol0H/UAKuJYJqWUUjHEMxQWAsNFJF9EkoDpwJwW48wBro08vhR4L9b+BKWUUvEVtwviGWOCInIL8Db2kNTHjTFficgvgUXGmDnATOAZEdmAbSFMj1d5lFJKtS+uV0k1xswF5rYYdk+zx17gsniWQSmlVMfp9ZSVUkpFaSgopZSK0lBQSikV1eUuiCciZcD+nr3WC9jV7lhdU3etm9ar6+mudevq9TrCGNPuiV5dLhQOhIgs6sgZfV1Rd62b1qvr6a516671akm7j5RSSkVpKCillIpKtFB4rLMLEEfdtW5ar66nu9atu9ZrDwm1T0EppVRsidZSUEopFYOGglJKqaiECQURmSoia0Vkg4jc0dnlORAiskVEVojIUhFZFBmWLSLviMj6yP+szi5nR4jI4yJSGrnhUtOwVusi1kORebhcRI7pvJLH1ka97hWR7ZH5tlREpjV77c5IvdaKyNmdU+r2ichAEXlfRFaLyFci8v3I8C49z2LUq8vPs31mjOn2f9irtG4EhgBJwDKgoLPLdQD12QL0ajHsAeCOyOM7gN91djk7WJeTgWOAle3VBZgGzMPenOk44PPOLv8+1ute4MetjFsQWSaTgfzIsurs7Dq0Ua9+wDGRxxnY+7AXdPV5FqNeXX6e7etforQUJgMbjDGbjDF+4EXggk4u08F2AfBU5PFTwIWdWJYOM8Z8xN43VmqrLhcATxtrAdBTRPodmpLumzbq1ZYLgBeNMT5jzGZgA3aZPewYY3YYY5ZEHtcCq4E8uvg8i1GvtnSZebavEiUU8oBtzZ4XEXuGH+4M8B8RWRy5fzVAH2PMDrALONC700p34NqqS3eYj7dEulEeb9bF1yXrJSKDgfHA53SjedaiXtCN5llHJEoodOhe0F3ICcaYY4BzgO+KyMmdXaBDpKvPx38AQ4FxwA7gD5HhXa5eIpIOvAb8wBhTE2vUVoYdtnVrpV7dZp51VKKEQkfuF91lGGOKI/9LgVnYZuvOpmZ55H9p55XwgLVVly49H40xO40xIWNMGJjB7u6GLlUvEXFjV5zPGWNejwzu8vOstXp1l3m2LxIlFDpyv+guQUTSRCSj6TEwBVjJnve7vhZ4o3NKeFC0VZc5wDcjR7QcB1Q3dVl0BS360i/Czjew9ZouIskikg8MB7441OXrCBER7G10Vxtj/tjspS49z9qqV3eYZ/uss/d0H6o/7FEQ67BHCdzV2eU5gHoMwR71sAz4qqkuQA7wLrA+8j+7s8vawfq8gG2WB7BbX99uqy7YJvvDkXm4ApjY2eXfx3o9Eyn3cuxKpV+z8e+K1GstcE5nlz9GvU7EdpMsB5ZG/qZ19XkWo15dfp7t659e5kIppVRUonQfKaWU6gANBaWUUlEaCkoppaI0FJRSSkVpKCillIrSUFDqEBKRU0Xkzc4uh1Jt0VBQSikVpaGgVCtE5GoR+SJyDf1HRcQpInUi8gcRWSIi74pIbmTccSKyIHLRtFnN7iUwTETmi8iyyHuGRiafLiKvisgaEXkucjatUocFDQWlWhCRo4ArsBceHAeEgKuANGCJsRcj/BD4ReQtTwM/NcaMwZ792jT8OeBhY8xY4GvYM5zBXoHzB9hr8g8BToh7pZTqIFdnF0Cpw9AZwARgYWQjPgV7gbcw8FJknGeB10WkB9DTGPNhZPhTwCuR61PlGWNmARhjvACR6X1hjCmKPF8KDAY+jn+1lGqfhoJSexPgKWPMnXsMFPl5i/FiXSMmVpeQr9njEPo7VIcR7T5Sam/vApeKSG+I3n/4COzv5dLION8APjbGVAOVInJSZPg1wIfGXou/SEQujEwjWURSD2ktlNoPuoWiVAvGmFUicjf27nYO7JVOvwvUA6NEZDFQjd3vAPZS0Y9EVvqbgG9Fhl8DPCoiv4xM47JDWA2l9oteJVWpDhKROmNMemeXQ6l40u4jpZRSUdpSUEopFaUtBaWUUlEaCkoppaI0FJRSSkVpKCillIrSUFBKKRX1/+aFPvPVgquGAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# summarize history for loss\n",
    "plt.plot(history.history['loss'])\n",
    "plt.plot(history.history['val_loss'])\n",
    "plt.title('model loss')\n",
    "plt.ylabel('loss')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'evaluate'], loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### add one extra layer based on Nielsen's network3 pipeline- iteration 4\n",
    "add one extra layer - iteration 3, batch_size = 20\n",
    "epochs = 60 fully conncted layer=1000 nodes for 2 layers eta = 0.03 added regulatizer \n",
    "###### training accuracy: 0.9950, test accuracy: 0.9916"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(60000, 28, 28) (60000,) (10000, 28, 28) (10000,)\n"
     ]
    }
   ],
   "source": [
    "batch_size = 100\n",
    "num_classes = 20\n",
    "epochs = 60\n",
    "\n",
    "# input image dimensions\n",
    "img_rows, img_cols = 28, 28\n",
    "\n",
    "# the data, split between train and test sets\n",
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
    "print(x_train.shape,y_train.shape,x_test.shape,y_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x_train shape: (60000, 28, 28, 1)\n",
      "60000 train samples\n",
      "10000 test samples\n",
      "Train on 60000 samples, validate on 10000 samples\n",
      "Epoch 1/60\n",
      "60000/60000 [==============================] - 89s 1ms/step - loss: 1.5290 - acc: 0.9096 - val_loss: 0.2771 - val_acc: 0.9471\n",
      "Epoch 2/60\n",
      "60000/60000 [==============================] - 88s 1ms/step - loss: 0.1892 - acc: 0.9698 - val_loss: 0.1249 - val_acc: 0.9836\n",
      "Epoch 3/60\n",
      "60000/60000 [==============================] - 87s 1ms/step - loss: 0.1333 - acc: 0.9781 - val_loss: 0.1110 - val_acc: 0.9794\n",
      "Epoch 4/60\n",
      "60000/60000 [==============================] - 91s 2ms/step - loss: 0.1096 - acc: 0.9816 - val_loss: 0.0886 - val_acc: 0.9863\n",
      "Epoch 5/60\n",
      "60000/60000 [==============================] - 87s 1ms/step - loss: 0.0951 - acc: 0.9843 - val_loss: 0.0892 - val_acc: 0.9841\n",
      "Epoch 6/60\n",
      "60000/60000 [==============================] - 88s 1ms/step - loss: 0.0872 - acc: 0.9855 - val_loss: 0.0680 - val_acc: 0.9895\n",
      "Epoch 7/60\n",
      "60000/60000 [==============================] - 87s 1ms/step - loss: 0.0792 - acc: 0.9866 - val_loss: 0.0674 - val_acc: 0.9889\n",
      "Epoch 8/60\n",
      "60000/60000 [==============================] - 87s 1ms/step - loss: 0.0737 - acc: 0.9876 - val_loss: 0.0780 - val_acc: 0.9854\n",
      "Epoch 9/60\n",
      "60000/60000 [==============================] - 87s 1ms/step - loss: 0.0708 - acc: 0.9879 - val_loss: 0.0668 - val_acc: 0.9895\n",
      "Epoch 10/60\n",
      "60000/60000 [==============================] - 87s 1ms/step - loss: 0.0678 - acc: 0.9884 - val_loss: 0.0613 - val_acc: 0.9905\n",
      "Epoch 11/60\n",
      "60000/60000 [==============================] - 87s 1ms/step - loss: 0.0648 - acc: 0.9891 - val_loss: 0.0621 - val_acc: 0.9889\n",
      "Epoch 12/60\n",
      "60000/60000 [==============================] - 87s 1ms/step - loss: 0.0620 - acc: 0.9898 - val_loss: 0.0566 - val_acc: 0.9907\n",
      "Epoch 13/60\n",
      "60000/60000 [==============================] - 88s 1ms/step - loss: 0.0602 - acc: 0.9898 - val_loss: 0.0648 - val_acc: 0.9874\n",
      "Epoch 14/60\n",
      "60000/60000 [==============================] - 88s 1ms/step - loss: 0.0580 - acc: 0.9908 - val_loss: 0.0637 - val_acc: 0.9870\n",
      "Epoch 15/60\n",
      "60000/60000 [==============================] - 87s 1ms/step - loss: 0.0561 - acc: 0.9905 - val_loss: 0.0616 - val_acc: 0.9880\n",
      "Epoch 16/60\n",
      "60000/60000 [==============================] - 89s 1ms/step - loss: 0.0555 - acc: 0.9906 - val_loss: 0.0594 - val_acc: 0.9895\n",
      "Epoch 17/60\n",
      "60000/60000 [==============================] - 88s 1ms/step - loss: 0.0535 - acc: 0.9909 - val_loss: 0.0590 - val_acc: 0.9884\n",
      "Epoch 18/60\n",
      "60000/60000 [==============================] - 91s 2ms/step - loss: 0.0529 - acc: 0.9910 - val_loss: 0.0655 - val_acc: 0.9872\n",
      "Epoch 19/60\n",
      "60000/60000 [==============================] - 87s 1ms/step - loss: 0.0513 - acc: 0.9915 - val_loss: 0.0481 - val_acc: 0.9922\n",
      "Epoch 20/60\n",
      "60000/60000 [==============================] - 88s 1ms/step - loss: 0.0502 - acc: 0.9917 - val_loss: 0.0568 - val_acc: 0.9891\n",
      "Epoch 21/60\n",
      "60000/60000 [==============================] - 88s 1ms/step - loss: 0.0493 - acc: 0.9920 - val_loss: 0.0521 - val_acc: 0.9910\n",
      "Epoch 22/60\n",
      "60000/60000 [==============================] - 87s 1ms/step - loss: 0.0486 - acc: 0.9921 - val_loss: 0.0470 - val_acc: 0.9907\n",
      "Epoch 23/60\n",
      "60000/60000 [==============================] - 88s 1ms/step - loss: 0.0466 - acc: 0.9924 - val_loss: 0.0549 - val_acc: 0.9893\n",
      "Epoch 24/60\n",
      "60000/60000 [==============================] - 88s 1ms/step - loss: 0.0455 - acc: 0.9926 - val_loss: 0.0546 - val_acc: 0.9882\n",
      "Epoch 25/60\n",
      "60000/60000 [==============================] - 88s 1ms/step - loss: 0.0450 - acc: 0.9925 - val_loss: 0.0477 - val_acc: 0.9923\n",
      "Epoch 26/60\n",
      "60000/60000 [==============================] - 88s 1ms/step - loss: 0.0443 - acc: 0.9926 - val_loss: 0.0461 - val_acc: 0.9920\n",
      "Epoch 27/60\n",
      "60000/60000 [==============================] - 88s 1ms/step - loss: 0.0441 - acc: 0.9928 - val_loss: 0.0444 - val_acc: 0.9926\n",
      "Epoch 28/60\n",
      "60000/60000 [==============================] - 88s 1ms/step - loss: 0.0430 - acc: 0.9929 - val_loss: 0.0455 - val_acc: 0.9911\n",
      "Epoch 29/60\n",
      "60000/60000 [==============================] - 88s 1ms/step - loss: 0.0418 - acc: 0.9932 - val_loss: 0.0453 - val_acc: 0.9913\n",
      "Epoch 30/60\n",
      "60000/60000 [==============================] - 87s 1ms/step - loss: 0.0414 - acc: 0.9931 - val_loss: 0.0693 - val_acc: 0.9831\n",
      "Epoch 31/60\n",
      "60000/60000 [==============================] - 88s 1ms/step - loss: 0.0408 - acc: 0.9932 - val_loss: 0.0423 - val_acc: 0.9927\n",
      "Epoch 32/60\n",
      "60000/60000 [==============================] - 91s 2ms/step - loss: 0.0406 - acc: 0.9934 - val_loss: 0.0428 - val_acc: 0.9918\n",
      "Epoch 33/60\n",
      "60000/60000 [==============================] - 88s 1ms/step - loss: 0.0401 - acc: 0.9935 - val_loss: 0.0443 - val_acc: 0.9912\n",
      "Epoch 34/60\n",
      "60000/60000 [==============================] - 88s 1ms/step - loss: 0.0392 - acc: 0.9937 - val_loss: 0.0461 - val_acc: 0.9913\n",
      "Epoch 35/60\n",
      "60000/60000 [==============================] - 88s 1ms/step - loss: 0.0381 - acc: 0.9942 - val_loss: 0.0472 - val_acc: 0.9908\n",
      "Epoch 36/60\n",
      "60000/60000 [==============================] - 89s 1ms/step - loss: 0.0387 - acc: 0.9936 - val_loss: 0.0466 - val_acc: 0.9913\n",
      "Epoch 37/60\n",
      "60000/60000 [==============================] - 88s 1ms/step - loss: 0.0369 - acc: 0.9943 - val_loss: 0.0531 - val_acc: 0.9896\n",
      "Epoch 38/60\n",
      "60000/60000 [==============================] - 88s 1ms/step - loss: 0.0373 - acc: 0.9940 - val_loss: 0.0496 - val_acc: 0.9897\n",
      "Epoch 39/60\n",
      "60000/60000 [==============================] - 88s 1ms/step - loss: 0.0366 - acc: 0.9942 - val_loss: 0.0461 - val_acc: 0.9903\n",
      "Epoch 40/60\n",
      "60000/60000 [==============================] - 88s 1ms/step - loss: 0.0368 - acc: 0.9942 - val_loss: 0.0442 - val_acc: 0.9918\n",
      "Epoch 41/60\n",
      "60000/60000 [==============================] - 88s 1ms/step - loss: 0.0359 - acc: 0.9946 - val_loss: 0.0509 - val_acc: 0.9889\n",
      "Epoch 42/60\n",
      "60000/60000 [==============================] - 88s 1ms/step - loss: 0.0355 - acc: 0.9946 - val_loss: 0.0441 - val_acc: 0.9921\n",
      "Epoch 43/60\n",
      "60000/60000 [==============================] - 89s 1ms/step - loss: 0.0350 - acc: 0.9946 - val_loss: 0.0481 - val_acc: 0.9904\n",
      "Epoch 44/60\n",
      "60000/60000 [==============================] - 88s 1ms/step - loss: 0.0346 - acc: 0.9947 - val_loss: 0.0403 - val_acc: 0.9925\n",
      "Epoch 45/60\n",
      "60000/60000 [==============================] - 88s 1ms/step - loss: 0.0347 - acc: 0.9948 - val_loss: 0.0430 - val_acc: 0.9915\n",
      "Epoch 46/60\n",
      "60000/60000 [==============================] - 92s 2ms/step - loss: 0.0339 - acc: 0.9947 - val_loss: 0.0415 - val_acc: 0.9917\n",
      "Epoch 47/60\n",
      "60000/60000 [==============================] - 88s 1ms/step - loss: 0.0339 - acc: 0.9945 - val_loss: 0.0446 - val_acc: 0.9903\n",
      "Epoch 48/60\n",
      "60000/60000 [==============================] - 88s 1ms/step - loss: 0.0325 - acc: 0.9950 - val_loss: 0.0414 - val_acc: 0.9919\n",
      "Epoch 49/60\n",
      "60000/60000 [==============================] - 88s 1ms/step - loss: 0.0326 - acc: 0.9947 - val_loss: 0.0444 - val_acc: 0.9907\n",
      "Epoch 50/60\n",
      "60000/60000 [==============================] - 89s 1ms/step - loss: 0.0325 - acc: 0.9953 - val_loss: 0.0389 - val_acc: 0.9926\n",
      "Epoch 51/60\n",
      "60000/60000 [==============================] - 88s 1ms/step - loss: 0.0322 - acc: 0.9951 - val_loss: 0.0413 - val_acc: 0.9916\n",
      "Epoch 52/60\n",
      "60000/60000 [==============================] - 88s 1ms/step - loss: 0.0322 - acc: 0.9951 - val_loss: 0.0395 - val_acc: 0.9930\n",
      "Epoch 53/60\n",
      "60000/60000 [==============================] - 88s 1ms/step - loss: 0.0323 - acc: 0.9949 - val_loss: 0.0463 - val_acc: 0.9899\n",
      "Epoch 54/60\n",
      "60000/60000 [==============================] - 88s 1ms/step - loss: 0.0311 - acc: 0.9954 - val_loss: 0.0450 - val_acc: 0.9918\n",
      "Epoch 55/60\n",
      "60000/60000 [==============================] - 88s 1ms/step - loss: 0.0312 - acc: 0.9951 - val_loss: 0.0450 - val_acc: 0.9916\n",
      "Epoch 56/60\n",
      "60000/60000 [==============================] - 88s 1ms/step - loss: 0.0302 - acc: 0.9953 - val_loss: 0.0423 - val_acc: 0.9914\n",
      "Epoch 57/60\n",
      "60000/60000 [==============================] - 89s 1ms/step - loss: 0.0311 - acc: 0.9952 - val_loss: 0.0408 - val_acc: 0.9912\n",
      "Epoch 58/60\n",
      "60000/60000 [==============================] - 89s 1ms/step - loss: 0.0306 - acc: 0.9952 - val_loss: 0.0484 - val_acc: 0.9909\n",
      "Epoch 59/60\n",
      "60000/60000 [==============================] - 88s 1ms/step - loss: 0.0296 - acc: 0.9955 - val_loss: 0.0440 - val_acc: 0.9907\n",
      "Epoch 60/60\n",
      "60000/60000 [==============================] - 91s 2ms/step - loss: 0.0301 - acc: 0.9950 - val_loss: 0.0412 - val_acc: 0.9916\n",
      "Test loss: 0.04117969057112932\n",
      "Test accuracy: 0.9916\n"
     ]
    }
   ],
   "source": [
    "#  TensorFlow's tensor expect the input shape (samples, rows, cols, channels) \n",
    "from keras import regularizers\n",
    "\n",
    "if K.image_data_format() == 'channels_first':\n",
    "    x_train = x_train.reshape(x_train.shape[0], 1, img_rows, img_cols)\n",
    "    x_test = x_test.reshape(x_test.shape[0], 1, img_rows, img_cols)\n",
    "    input_shape = (1, img_rows, img_cols)\n",
    "else:\n",
    "    x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)\n",
    "    x_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)\n",
    "    input_shape = (img_rows, img_cols, 1)\n",
    "\n",
    "x_train = x_train.astype('float32')\n",
    "x_test = x_test.astype('float32')\n",
    "x_train /= 255\n",
    "x_test /= 255\n",
    "print('x_train shape:', x_train.shape)\n",
    "print(x_train.shape[0], 'train samples')\n",
    "print(x_test.shape[0], 'test samples')\n",
    "\n",
    "# convert class vectors to binary class matrices\n",
    "y_train = keras.utils.to_categorical(y_train, num_classes)\n",
    "y_test = keras.utils.to_categorical(y_test, num_classes)\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Conv2D(20, kernel_size=(5, 5),\n",
    "                 activation='relu',\n",
    "                 input_shape=input_shape))\n",
    "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "model.add(Conv2D(40, (5, 5), activation='relu'))\n",
    "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "#model.add(Dropout(0.25))\n",
    "model.add(Flatten())\n",
    "model.add(Dense(1000, kernel_regularizer=regularizers.l2(0.01), activation='relu')) \n",
    "#model.add(Dropout(0.5))\n",
    "# added one extra fully connected layer below\n",
    "model.add(Dense(1000, kernel_regularizer=regularizers.l2(0.01), activation='relu'))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(num_classes, activation='softmax'))\n",
    "\n",
    "model.compile(loss=keras.losses.categorical_crossentropy,\n",
    "              optimizer=keras.optimizers.Adadelta(),\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "model.fit(x_train, y_train,\n",
    "          batch_size=batch_size,\n",
    "          epochs=epochs,\n",
    "          verbose=1,\n",
    "          validation_data=(x_test, y_test))\n",
    "score = model.evaluate(x_test, y_test, verbose=0)\n",
    "print('Test loss:', score[0])\n",
    "print('Test accuracy:', score[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### add one extra layer based on Nielsen's network3 pipeline- iteration 5\n",
    "- add BatchNormalization for every computation layer\n",
    "- add one extra layer - iteration 3, batch_size = 20\n",
    "- epochs = 60, fully conncted layer=1000 nodes for 2 layers eta = 0.03 added regulatizer \n",
    "- training accuracy: 0.9977,\n",
    "- validation accuracy: 0.9924\n",
    "- test accuracy: 0.9928"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(49980, 28, 28) (49980,) (10020, 28, 28) (10020,) (10000, 28, 28) (10000,)\n"
     ]
    }
   ],
   "source": [
    "batch_size = 10\n",
    "num_classes = 10\n",
    "epochs = 40\n",
    "\n",
    "# input image dimensions\n",
    "img_rows, img_cols = 28, 28\n",
    "\n",
    "# the data, split between train and test sets\n",
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
    "# # the data, split between train and validation sets\n",
    "x_train, x_valid, y_train, y_valid = train_test_split(x_train, y_train, test_size = 0.167, random_state = 0)\n",
    "print(x_train.shape,y_train.shape,x_valid.shape,y_valid.shape,x_test.shape,y_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x_train shape: (49980, 28, 28, 1)\n",
      "49980 train samples\n",
      "10020 valid samples\n",
      "10000 test samples\n",
      "WARNING:tensorflow:From C:\\Users\\CampusUser\\Anaconda3\\envs\\Keras\\lib\\site-packages\\tensorflow\\python\\framework\\op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Colocations handled automatically by placer.\n",
      "WARNING:tensorflow:From C:\\Users\\CampusUser\\Anaconda3\\envs\\Keras\\lib\\site-packages\\keras\\backend\\tensorflow_backend.py:3445: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.\n",
      "WARNING:tensorflow:From C:\\Users\\CampusUser\\Anaconda3\\envs\\Keras\\lib\\site-packages\\tensorflow\\python\\ops\\math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.cast instead.\n",
      "Train on 49980 samples, validate on 10020 samples\n",
      "Epoch 1/40\n",
      "49980/49980 [==============================] - 324s 6ms/step - loss: 0.2364 - acc: 0.9370 - val_loss: 0.0660 - val_acc: 0.9800\n",
      "Epoch 2/40\n",
      "49980/49980 [==============================] - 373s 7ms/step - loss: 0.1004 - acc: 0.9732 - val_loss: 0.0366 - val_acc: 0.9888\n",
      "Epoch 3/40\n",
      "49980/49980 [==============================] - 309s 6ms/step - loss: 0.0728 - acc: 0.9801 - val_loss: 0.0374 - val_acc: 0.9883\n",
      "Epoch 4/40\n",
      "49980/49980 [==============================] - 318s 6ms/step - loss: 0.0591 - acc: 0.9841 - val_loss: 0.0375 - val_acc: 0.9903\n",
      "Epoch 5/40\n",
      "49980/49980 [==============================] - 309s 6ms/step - loss: 0.0473 - acc: 0.9868 - val_loss: 0.0328 - val_acc: 0.9907oss: 0.0\n",
      "Epoch 6/40\n",
      "49980/49980 [==============================] - 323s 6ms/step - loss: 0.0405 - acc: 0.9893 - val_loss: 0.0357 - val_acc: 0.9916\n",
      "Epoch 7/40\n",
      "49980/49980 [==============================] - 307s 6ms/step - loss: 0.0362 - acc: 0.9898 - val_loss: 0.0371 - val_acc: 0.9903\n",
      "Epoch 8/40\n",
      "49980/49980 [==============================] - 307s 6ms/step - loss: 0.0353 - acc: 0.9906 - val_loss: 0.0344 - val_acc: 0.9919\n",
      "Epoch 9/40\n",
      "49980/49980 [==============================] - 306s 6ms/step - loss: 0.0264 - acc: 0.9924 - val_loss: 0.0386 - val_acc: 0.9906\n",
      "Epoch 10/40\n",
      "49980/49980 [==============================] - 302s 6ms/step - loss: 0.0259 - acc: 0.9925 - val_loss: 0.0345 - val_acc: 0.9917\n",
      "Epoch 11/40\n",
      "49980/49980 [==============================] - 303s 6ms/step - loss: 0.0266 - acc: 0.9931 - val_loss: 0.0327 - val_acc: 0.9920\n",
      "Epoch 12/40\n",
      "49980/49980 [==============================] - 303s 6ms/step - loss: 0.0246 - acc: 0.9930 - val_loss: 0.0448 - val_acc: 0.9902\n",
      "Epoch 13/40\n",
      "49980/49980 [==============================] - 306s 6ms/step - loss: 0.0218 - acc: 0.9939 - val_loss: 0.0329 - val_acc: 0.9921\n",
      "Epoch 14/40\n",
      "49980/49980 [==============================] - 304s 6ms/step - loss: 0.0202 - acc: 0.9948 - val_loss: 0.0372 - val_acc: 0.9914\n",
      "Epoch 15/40\n",
      "49980/49980 [==============================] - 305s 6ms/step - loss: 0.0185 - acc: 0.9949 - val_loss: 0.0403 - val_acc: 0.9915\n",
      "Epoch 16/40\n",
      "49980/49980 [==============================] - 307s 6ms/step - loss: 0.0176 - acc: 0.9954 - val_loss: 0.0364 - val_acc: 0.9917\n",
      "Epoch 17/40\n",
      "49980/49980 [==============================] - 308s 6ms/step - loss: 0.0175 - acc: 0.9954 - val_loss: 0.0408 - val_acc: 0.9915\n",
      "Epoch 18/40\n",
      "49980/49980 [==============================] - 309s 6ms/step - loss: 0.0182 - acc: 0.9949 - val_loss: 0.0410 - val_acc: 0.9915s - loss: 0.0182 -\n",
      "Epoch 19/40\n",
      "49980/49980 [==============================] - 338s 7ms/step - loss: 0.0196 - acc: 0.9948 - val_loss: 0.0395 - val_acc: 0.9912\n",
      "Epoch 20/40\n",
      "49980/49980 [==============================] - 379s 8ms/step - loss: 0.0180 - acc: 0.9953 - val_loss: 0.0354 - val_acc: 0.9923\n",
      "Epoch 21/40\n",
      "49980/49980 [==============================] - 378s 8ms/step - loss: 0.0143 - acc: 0.9959 - val_loss: 0.0336 - val_acc: 0.9926\n",
      "Epoch 22/40\n",
      "49980/49980 [==============================] - 378s 8ms/step - loss: 0.0136 - acc: 0.9964 - val_loss: 0.0395 - val_acc: 0.9923\n",
      "Epoch 23/40\n",
      "49980/49980 [==============================] - 377s 8ms/step - loss: 0.0130 - acc: 0.9963 - val_loss: 0.0386 - val_acc: 0.9929\n",
      "Epoch 24/40\n",
      "49980/49980 [==============================] - 379s 8ms/step - loss: 0.0138 - acc: 0.9960 - val_loss: 0.0444 - val_acc: 0.9916\n",
      "Epoch 25/40\n",
      "49980/49980 [==============================] - 378s 8ms/step - loss: 0.0130 - acc: 0.9965 - val_loss: 0.0352 - val_acc: 0.9923\n",
      "Epoch 26/40\n",
      "49980/49980 [==============================] - 379s 8ms/step - loss: 0.0125 - acc: 0.9967 - val_loss: 0.0331 - val_acc: 0.9938\n",
      "Epoch 27/40\n",
      "49980/49980 [==============================] - 378s 8ms/step - loss: 0.0122 - acc: 0.9967 - val_loss: 0.0382 - val_acc: 0.9915\n",
      "Epoch 28/40\n",
      "49980/49980 [==============================] - 384s 8ms/step - loss: 0.0117 - acc: 0.9969 - val_loss: 0.0393 - val_acc: 0.9920\n",
      "Epoch 29/40\n",
      "49980/49980 [==============================] - 382s 8ms/step - loss: 0.0122 - acc: 0.9965 - val_loss: 0.0379 - val_acc: 0.9924\n",
      "Epoch 30/40\n",
      "49980/49980 [==============================] - 379s 8ms/step - loss: 0.0118 - acc: 0.9968 - val_loss: 0.0365 - val_acc: 0.9928\n",
      "Epoch 31/40\n",
      "49980/49980 [==============================] - 380s 8ms/step - loss: 0.0101 - acc: 0.9973 - val_loss: 0.0372 - val_acc: 0.9925\n",
      "Epoch 32/40\n",
      "49980/49980 [==============================] - 390s 8ms/step - loss: 0.0083 - acc: 0.9976 - val_loss: 0.0455 - val_acc: 0.9910\n",
      "Epoch 33/40\n",
      "49980/49980 [==============================] - 396s 8ms/step - loss: 0.0096 - acc: 0.9974 - val_loss: 0.0383 - val_acc: 0.9926\n",
      "Epoch 34/40\n",
      "49980/49980 [==============================] - 386s 8ms/step - loss: 0.0086 - acc: 0.9977 - val_loss: 0.0403 - val_acc: 0.9929\n",
      "Epoch 35/40\n",
      "49980/49980 [==============================] - 389s 8ms/step - loss: 0.0075 - acc: 0.9979 - val_loss: 0.0321 - val_acc: 0.9932\n",
      "Epoch 36/40\n",
      "49980/49980 [==============================] - 388s 8ms/step - loss: 0.0087 - acc: 0.9977 - val_loss: 0.0365 - val_acc: 0.9930\n",
      "Epoch 37/40\n",
      "49980/49980 [==============================] - 385s 8ms/step - loss: 0.0095 - acc: 0.9975 - val_loss: 0.0446 - val_acc: 0.9918\n",
      "Epoch 38/40\n",
      "49980/49980 [==============================] - 384s 8ms/step - loss: 0.0089 - acc: 0.9976 - val_loss: 0.0422 - val_acc: 0.9924\n",
      "Epoch 39/40\n",
      "49980/49980 [==============================] - 383s 8ms/step - loss: 0.0081 - acc: 0.9976 - val_loss: 0.0363 - val_acc: 0.9930\n",
      "Epoch 40/40\n",
      "49980/49980 [==============================] - 386s 8ms/step - loss: 0.0084 - acc: 0.9977 - val_loss: 0.0353 - val_acc: 0.9924\n",
      "Test loss: 0.038336180942842066\n",
      "Test accuracy: 0.9928\n"
     ]
    }
   ],
   "source": [
    "# use data augmentation to reduce over-fitting\n",
    "# gen = ImageDataGenerator(rotation_range=8, width_shift_range=0.08,shear_range=0.3,height_shift_range=0.08,zoom_range=0.08)\n",
    "# valid_gen = ImageDataGenerator()\n",
    "\n",
    "#  TensorFlow's tensor expect the input shape (samples, rows, cols, channels) \n",
    "\n",
    "if K.image_data_format() == 'channels_first':\n",
    "    x_train = x_train.reshape(x_train.shape[0], 1, img_rows, img_cols)\n",
    "    x_valid = x_valid.reshape(x_valid.shape[0], 1, img_rows, img_cols)\n",
    "    x_test = x_test.reshape(x_test.shape[0], 1, img_rows, img_cols)\n",
    "    input_shape = (1, img_rows, img_cols)\n",
    "else:\n",
    "    x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)\n",
    "    x_valid = x_valid.reshape(x_valid.shape[0], img_rows, img_cols, 1)\n",
    "    x_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)\n",
    "    input_shape = (img_rows, img_cols, 1)\n",
    "\n",
    "x_train = x_train.astype('float32')\n",
    "x_valid = x_valid.astype('float32')\n",
    "x_test = x_test.astype('float32')\n",
    "x_train /= 255\n",
    "x_valid /= 255\n",
    "x_test /= 255\n",
    "print('x_train shape:', x_train.shape)\n",
    "print(x_train.shape[0], 'train samples')\n",
    "print(x_valid.shape[0], 'valid samples')\n",
    "print(x_test.shape[0], 'test samples')\n",
    "\n",
    "# convert class vectors to binary class matrices\n",
    "y_train = keras.utils.to_categorical(y_train, num_classes)\n",
    "y_valid = keras.utils.to_categorical(y_valid, num_classes)\n",
    "y_test = keras.utils.to_categorical(y_test, num_classes)\n",
    "\n",
    "# train_generator = gen.flow(x_train,y_train,batch_size=batch_size)\n",
    "# valid_generator = valid_gen.flow(x_valid,y_valid,batch_size = batch_size)\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Conv2D(20, kernel_size=(5, 5),\n",
    "                 activation='relu',\n",
    "                 input_shape=input_shape))\n",
    "model.add(BatchNormalization(axis=-1))\n",
    "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "model.add(Conv2D(40, (5, 5), activation='relu'))\n",
    "model.add(BatchNormalization(axis=-1))\n",
    "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "#model.add(Dropout(0.25))\n",
    "model.add(Flatten())\n",
    "model.add(Dense(2000, activation='relu')) \n",
    "model.add(BatchNormalization(axis=-1))\n",
    "model.add(Dropout(0.3))\n",
    "# added one extra fully connected layer below\n",
    "model.add(Dense(1000, activation='relu'))\n",
    "model.add(BatchNormalization(axis=-1))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(num_classes, activation='softmax'))\n",
    "\n",
    "model.compile(loss=keras.losses.categorical_crossentropy,\n",
    "              optimizer=keras.optimizers.Adadelta(),\n",
    "              #optimizer=keras.optimizers.Adam(),\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "model.fit(x_train, y_train,\n",
    "          batch_size=batch_size,\n",
    "          epochs=epochs,\n",
    "          verbose=1,\n",
    "          validation_data=(x_valid, y_valid))\n",
    "# model.fit_generator(train_generator,steps_per_epoch=49980//batch_size,\n",
    "#                    validation_data=valid_generator,validation_steps=10020//batch_size)\n",
    "score = model.evaluate(x_test, y_test, verbose=0)\n",
    "print('Test loss:', score[0])\n",
    "print('Test accuracy:', score[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training on EMNIST Letters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 1. Baseline Training on EMNIST letters (iteration 1)\n",
    "- training accuracy: 93.72%\n",
    "- test accuracy: 93.9%\n",
    "- no train test split, validate directly on test dataset\n",
    "- The following further iterations will do the train, valid, and test datasets split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy import io as sio\n",
    "import numpy as np\n",
    "import gzip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "#load dataset\n",
    "mat = sio.loadmat('data/emnist-letters.mat')\n",
    "data = mat['dataset']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# StackOverflow reference: \n",
    "# https://stackoverflow.com/questions/51125969/loading-emnist-letters-dataset/53547262#53547262\n",
    "# There is an additional field 'writers' \n",
    "# (e.g. data['train'][0,0]['writers'][0,0]) that distinguishes the original sample writer. \n",
    "# The data['mapping'] field maps from class numbers to ASCII codes of the corresponding letters. \n",
    "# For example, class 1 maps to ASCII codes 65 and 97 ('A' and 'a'). \n",
    "X_train = data['train'][0,0]['images'][0,0]\n",
    "y_train = data['train'][0,0]['labels'][0,0]\n",
    "X_test = data['test'][0,0]['images'][0,0]\n",
    "y_test = data['test'][0,0]['labels'][0,0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "124800 124800\n"
     ]
    }
   ],
   "source": [
    "print(X_train.shape[0],y_train.shape[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# reshape the data into 2D, 28x28 sized images instead of a 1D 784 array\n",
    "# to get the correct image orientation you'll need to do a numpy reshape using Fortran ordering\n",
    "# (Matlab uses column-major ordering, just like Fortran\n",
    "# https://stackoverflow.com/questions/51125969/loading-emnist-letters-dataset/53547262#53547262\n",
    "x_train = X_train.reshape((X_train.shape[0], 28, 28), order='F')\n",
    "y_train = y_train.reshape(-1)\n",
    "x_test = X_test.reshape((X_test.shape[0], 28, 28), order = 'F')   \n",
    "y_test = y_test.reshape(-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(124800, 28, 28) (124800,) (20800, 28, 28) (20800,)\n"
     ]
    }
   ],
   "source": [
    "print(x_train.shape,y_train.shape,x_test.shape,y_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# input image dimensions\n",
    "img_rows, img_cols = 28, 28"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x_train shape: (124800, 28, 28, 1)\n",
      "124800 train samples\n",
      "20800 test samples\n",
      "WARNING:tensorflow:From C:\\Users\\CampusUser\\Anaconda3\\envs\\Keras\\lib\\site-packages\\tensorflow\\python\\framework\\op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Colocations handled automatically by placer.\n",
      "WARNING:tensorflow:From C:\\Users\\CampusUser\\Anaconda3\\envs\\Keras\\lib\\site-packages\\keras\\backend\\tensorflow_backend.py:3445: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.\n",
      "WARNING:tensorflow:From C:\\Users\\CampusUser\\Anaconda3\\envs\\Keras\\lib\\site-packages\\tensorflow\\python\\ops\\math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.cast instead.\n",
      "Train on 124800 samples, validate on 20800 samples\n",
      "Epoch 1/30\n",
      "124800/124800 [==============================] - 206s 2ms/step - loss: 1.2596 - acc: 0.6273 - val_loss: 0.4831 - val_acc: 0.8510\n",
      "Epoch 2/30\n",
      "124800/124800 [==============================] - 159s 1ms/step - loss: 0.5711 - acc: 0.8243 - val_loss: 0.3168 - val_acc: 0.8998\n",
      "Epoch 3/30\n",
      "124800/124800 [==============================] - 148s 1ms/step - loss: 0.4574 - acc: 0.8556 - val_loss: 0.2838 - val_acc: 0.9083\n",
      "Epoch 4/30\n",
      "124800/124800 [==============================] - 144s 1ms/step - loss: 0.4006 - acc: 0.8738 - val_loss: 0.2524 - val_acc: 0.9194\n",
      "Epoch 5/30\n",
      "124800/124800 [==============================] - 141s 1ms/step - loss: 0.3641 - acc: 0.8844 - val_loss: 0.2470 - val_acc: 0.9212\n",
      "Epoch 6/30\n",
      "124800/124800 [==============================] - 141s 1ms/step - loss: 0.3382 - acc: 0.8921 - val_loss: 0.2272 - val_acc: 0.9259\n",
      "Epoch 7/30\n",
      "124800/124800 [==============================] - 141s 1ms/step - loss: 0.3215 - acc: 0.8969 - val_loss: 0.2301 - val_acc: 0.9282\n",
      "Epoch 8/30\n",
      "124800/124800 [==============================] - 140s 1ms/step - loss: 0.3038 - acc: 0.9026 - val_loss: 0.2222 - val_acc: 0.9278\n",
      "Epoch 9/30\n",
      "124800/124800 [==============================] - 141s 1ms/step - loss: 0.2908 - acc: 0.9053 - val_loss: 0.2201 - val_acc: 0.9284\n",
      "Epoch 10/30\n",
      "124800/124800 [==============================] - 143s 1ms/step - loss: 0.2790 - acc: 0.9091 - val_loss: 0.2109 - val_acc: 0.9305\n",
      "Epoch 11/30\n",
      "124800/124800 [==============================] - 144s 1ms/step - loss: 0.2692 - acc: 0.9124 - val_loss: 0.2122 - val_acc: 0.9298\n",
      "Epoch 12/30\n",
      "124800/124800 [==============================] - 141s 1ms/step - loss: 0.2609 - acc: 0.9141 - val_loss: 0.2045 - val_acc: 0.9345\n",
      "Epoch 13/30\n",
      "124800/124800 [==============================] - 141s 1ms/step - loss: 0.2503 - acc: 0.9181 - val_loss: 0.2071 - val_acc: 0.9326\n",
      "Epoch 14/30\n",
      "124800/124800 [==============================] - 140s 1ms/step - loss: 0.2425 - acc: 0.9190 - val_loss: 0.2049 - val_acc: 0.9345\n",
      "Epoch 15/30\n",
      "124800/124800 [==============================] - 142s 1ms/step - loss: 0.2387 - acc: 0.9205 - val_loss: 0.2022 - val_acc: 0.9328\n",
      "Epoch 16/30\n",
      "124800/124800 [==============================] - 140s 1ms/step - loss: 0.2316 - acc: 0.9226 - val_loss: 0.2033 - val_acc: 0.9341\n",
      "Epoch 17/30\n",
      "124800/124800 [==============================] - 142s 1ms/step - loss: 0.2261 - acc: 0.9245 - val_loss: 0.1968 - val_acc: 0.9363\n",
      "Epoch 18/30\n",
      "124800/124800 [==============================] - 140s 1ms/step - loss: 0.2207 - acc: 0.9250 - val_loss: 0.1999 - val_acc: 0.9367\n",
      "Epoch 19/30\n",
      "124800/124800 [==============================] - 147s 1ms/step - loss: 0.2191 - acc: 0.9260 - val_loss: 0.1965 - val_acc: 0.9363\n",
      "Epoch 20/30\n",
      "124800/124800 [==============================] - 141s 1ms/step - loss: 0.2138 - acc: 0.9281 - val_loss: 0.1968 - val_acc: 0.9366\n",
      "Epoch 21/30\n",
      "124800/124800 [==============================] - 141s 1ms/step - loss: 0.2097 - acc: 0.9284 - val_loss: 0.1961 - val_acc: 0.9369\n",
      "Epoch 22/30\n",
      "124800/124800 [==============================] - 140s 1ms/step - loss: 0.2031 - acc: 0.9310 - val_loss: 0.1957 - val_acc: 0.9376\n",
      "Epoch 23/30\n",
      "124800/124800 [==============================] - 140s 1ms/step - loss: 0.2004 - acc: 0.9314 - val_loss: 0.1954 - val_acc: 0.9381\n",
      "Epoch 24/30\n",
      "124800/124800 [==============================] - 141s 1ms/step - loss: 0.1971 - acc: 0.9322 - val_loss: 0.1960 - val_acc: 0.9366\n",
      "Epoch 25/30\n",
      "124800/124800 [==============================] - 140s 1ms/step - loss: 0.1932 - acc: 0.9335 - val_loss: 0.1899 - val_acc: 0.9387\n",
      "Epoch 26/30\n",
      "124800/124800 [==============================] - 142s 1ms/step - loss: 0.1898 - acc: 0.9342 - val_loss: 0.1975 - val_acc: 0.9365\n",
      "Epoch 27/30\n",
      "124800/124800 [==============================] - 141s 1ms/step - loss: 0.1864 - acc: 0.9354 - val_loss: 0.1935 - val_acc: 0.9386\n",
      "Epoch 28/30\n",
      "124800/124800 [==============================] - 145s 1ms/step - loss: 0.1845 - acc: 0.9368 - val_loss: 0.1955 - val_acc: 0.9384\n",
      "Epoch 29/30\n",
      "124800/124800 [==============================] - 140s 1ms/step - loss: 0.1809 - acc: 0.9367 - val_loss: 0.2008 - val_acc: 0.9384\n",
      "Epoch 30/30\n",
      "124800/124800 [==============================] - 141s 1ms/step - loss: 0.1789 - acc: 0.9372 - val_loss: 0.2007 - val_acc: 0.9391\n",
      "Test loss: 0.20069139864895708\n",
      "Test accuracy: 0.9390865384615384\n"
     ]
    }
   ],
   "source": [
    "# TensorFlow's tensor expect the input shape (samples, rows, cols, channels) \n",
    "if K.image_data_format() == 'channels_first':\n",
    "    x_train = x_train.reshape(x_train.shape[0], 1, img_rows, img_cols)\n",
    "    x_test = x_test.reshape(x_test.shape[0], 1, img_rows, img_cols)\n",
    "    input_shape = (1, img_rows, img_cols)\n",
    "else:\n",
    "    x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)\n",
    "    x_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)\n",
    "    input_shape = (img_rows, img_cols, 1)\n",
    "    \n",
    "batch_size = 654\n",
    "num_classes = 27\n",
    "epochs = 30\n",
    "\n",
    "x_train = x_train.astype('float32')\n",
    "x_test = x_test.astype('float32')\n",
    "x_train /= 255\n",
    "x_test /= 255\n",
    "print('x_train shape:', x_train.shape)\n",
    "print(x_train.shape[0], 'train samples')\n",
    "print(x_test.shape[0], 'test samples')\n",
    "\n",
    "# convert class vectors to binary class matrices\n",
    "y_train = keras.utils.to_categorical(y_train, num_classes,dtype='float32')\n",
    "y_test = keras.utils.to_categorical(y_test, num_classes,dtype='float32')\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Conv2D(32, kernel_size=(3, 3),\n",
    "                 activation='relu',\n",
    "                 input_shape=input_shape))\n",
    "model.add(Conv2D(64, (3, 3), activation='relu'))\n",
    "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "model.add(Dropout(0.25))\n",
    "model.add(Flatten())\n",
    "model.add(Dense(128, activation='relu'))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(num_classes, activation='softmax'))\n",
    "\n",
    "model.compile(loss=keras.losses.categorical_crossentropy,\n",
    "              optimizer=keras.optimizers.Adadelta(),\n",
    "              metrics=['accuracy'])\n",
    "tensorboard = TensorBoard(log_dir=\"logs/{}\".format(time()))\n",
    "model.fit(x_train, y_train,\n",
    "          batch_size=batch_size,\n",
    "          epochs=epochs,\n",
    "          verbose=1,\n",
    "          callbacks=[tensorboard],\n",
    "          validation_data=(x_test, y_test))\n",
    "score = model.evaluate(x_test, y_test, verbose=0)\n",
    "print('Test loss:', score[0])\n",
    "print('Test accuracy:', score[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. Training on EMNIST letters (iteration 5)\n",
    "Split training dataset to training and validation datasets\n",
    "test on test dataset\n",
    "- training accuracy: 0.987\n",
    "- validation accuracy: 0.951\n",
    "- test accuracy: 0.949"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy import io as sio\n",
    "import numpy as np\n",
    "import gzip\n",
    "from time import time\n",
    "from keras.callbacks import TensorBoard"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(103958, 28, 28) (103958,) (20842, 28, 28) (20842,) (20800, 28, 28) (20800,)\n"
     ]
    }
   ],
   "source": [
    "#load dataset\n",
    "mat = sio.loadmat('data/emnist-letters.mat')\n",
    "data = mat['dataset']\n",
    "# https://stackoverflow.com/questions/51125969/loading-emnist-letters-dataset/53547262#53547262\n",
    "X_train = data['train'][0,0]['images'][0,0]\n",
    "y_train = data['train'][0,0]['labels'][0,0]\n",
    "X_test = data['test'][0,0]['images'][0,0]\n",
    "y_test = data['test'][0,0]['labels'][0,0]\n",
    "# reshape the data into 2D, 28x28 sized images instead of a 1D 784 array\n",
    "# to get the correct image orientation, need to do a numpy reshape using Fortran ordering\n",
    "# (Matlab uses column-major ordering, just like Fortran\n",
    "# https://stackoverflow.com/questions/51125969/loading-emnist-letters-dataset/53547262#53547262\n",
    "x_train = X_train.reshape((X_train.shape[0], 28, 28), order='F')\n",
    "y_train = y_train.reshape(-1)\n",
    "x_test = X_test.reshape((X_test.shape[0], 28, 28), order = 'F')   \n",
    "y_test = y_test.reshape(-1)\n",
    "# the data, split between train and validation sets\n",
    "x_train, x_valid, y_train, y_valid = train_test_split(x_train, y_train, test_size = 0.167, random_state = 0, shuffle=True)\n",
    "# we want to have the same size for valid and test\n",
    "print(x_train.shape,y_train.shape, x_valid.shape,y_valid.shape,x_test.shape,y_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 249\n",
    "num_classes = 27\n",
    "epochs = 200\n",
    "# input image dimensions\n",
    "img_rows, img_cols = 28, 28"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x_train shape: (103958, 28, 28, 1)\n",
      "103958 train samples\n",
      "20842 valid samples\n",
      "20800 test samples\n",
      "Train on 103958 samples, validate on 20842 samples\n",
      "Epoch 1/200\n",
      "103958/103958 [==============================] - 164s 2ms/step - loss: 0.9631 - acc: 0.7058 - val_loss: 0.3359 - val_acc: 0.8931\n",
      "Epoch 2/200\n",
      "103958/103958 [==============================] - 158s 2ms/step - loss: 0.3938 - acc: 0.8718 - val_loss: 0.2416 - val_acc: 0.9208\n",
      "Epoch 3/200\n",
      "103958/103958 [==============================] - 153s 1ms/step - loss: 0.3132 - acc: 0.8967 - val_loss: 0.2147 - val_acc: 0.9294\n",
      "Epoch 4/200\n",
      "103958/103958 [==============================] - 152s 1ms/step - loss: 0.2825 - acc: 0.9061 - val_loss: 0.1981 - val_acc: 0.9337\n",
      "Epoch 5/200\n",
      "103958/103958 [==============================] - 153s 1ms/step - loss: 0.2595 - acc: 0.9144 - val_loss: 0.2144 - val_acc: 0.9293\n",
      "Epoch 6/200\n",
      "103958/103958 [==============================] - 153s 1ms/step - loss: 0.2402 - acc: 0.9191 - val_loss: 0.1909 - val_acc: 0.9373\n",
      "Epoch 7/200\n",
      "103958/103958 [==============================] - 152s 1ms/step - loss: 0.2276 - acc: 0.9241 - val_loss: 0.1784 - val_acc: 0.9397\n",
      "Epoch 8/200\n",
      "103958/103958 [==============================] - 153s 1ms/step - loss: 0.2187 - acc: 0.9260 - val_loss: 0.1783 - val_acc: 0.9409\n",
      "Epoch 9/200\n",
      "103958/103958 [==============================] - 153s 1ms/step - loss: 0.2097 - acc: 0.9283 - val_loss: 0.1802 - val_acc: 0.9396\n",
      "Epoch 10/200\n",
      "103958/103958 [==============================] - 156s 1ms/step - loss: 0.2020 - acc: 0.9309 - val_loss: 0.1739 - val_acc: 0.9424\n",
      "Epoch 11/200\n",
      "103958/103958 [==============================] - 152s 1ms/step - loss: 0.1969 - acc: 0.9332 - val_loss: 0.1723 - val_acc: 0.9419\n",
      "Epoch 12/200\n",
      "103958/103958 [==============================] - 152s 1ms/step - loss: 0.1919 - acc: 0.9346 - val_loss: 0.1662 - val_acc: 0.9433\n",
      "Epoch 13/200\n",
      "103958/103958 [==============================] - 153s 1ms/step - loss: 0.1862 - acc: 0.9367 - val_loss: 0.1759 - val_acc: 0.9422\n",
      "Epoch 14/200\n",
      "103958/103958 [==============================] - 153s 1ms/step - loss: 0.1792 - acc: 0.9387 - val_loss: 0.1650 - val_acc: 0.9460\n",
      "Epoch 15/200\n",
      "103958/103958 [==============================] - 152s 1ms/step - loss: 0.1761 - acc: 0.9401 - val_loss: 0.1656 - val_acc: 0.9447\n",
      "Epoch 16/200\n",
      "103958/103958 [==============================] - 153s 1ms/step - loss: 0.1735 - acc: 0.9402 - val_loss: 0.1671 - val_acc: 0.9428\n",
      "Epoch 17/200\n",
      "103958/103958 [==============================] - 152s 1ms/step - loss: 0.1688 - acc: 0.9420 - val_loss: 0.1662 - val_acc: 0.9453\n",
      "Epoch 18/200\n",
      "103958/103958 [==============================] - 156s 1ms/step - loss: 0.1685 - acc: 0.9417 - val_loss: 0.1619 - val_acc: 0.9451\n",
      "Epoch 19/200\n",
      "103958/103958 [==============================] - 152s 1ms/step - loss: 0.1660 - acc: 0.9424 - val_loss: 0.1630 - val_acc: 0.9460\n",
      "Epoch 20/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.1623 - acc: 0.9433 - val_loss: 0.1634 - val_acc: 0.9463\n",
      "Epoch 21/200\n",
      "103958/103958 [==============================] - 152s 1ms/step - loss: 0.1576 - acc: 0.9449 - val_loss: 0.1596 - val_acc: 0.9468\n",
      "Epoch 22/200\n",
      "103958/103958 [==============================] - 153s 1ms/step - loss: 0.1548 - acc: 0.9455 - val_loss: 0.1579 - val_acc: 0.9481\n",
      "Epoch 23/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.1542 - acc: 0.9464 - val_loss: 0.1612 - val_acc: 0.9476\n",
      "Epoch 24/200\n",
      "103958/103958 [==============================] - 152s 1ms/step - loss: 0.1500 - acc: 0.9470 - val_loss: 0.1676 - val_acc: 0.9465\n",
      "Epoch 25/200\n",
      "103958/103958 [==============================] - 153s 1ms/step - loss: 0.1499 - acc: 0.9470 - val_loss: 0.1625 - val_acc: 0.9469\n",
      "Epoch 26/200\n",
      "103958/103958 [==============================] - 157s 2ms/step - loss: 0.1496 - acc: 0.9471 - val_loss: 0.1641 - val_acc: 0.9460\n",
      "Epoch 27/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.1476 - acc: 0.9475 - val_loss: 0.1556 - val_acc: 0.9482\n",
      "Epoch 28/200\n",
      "103958/103958 [==============================] - 153s 1ms/step - loss: 0.1429 - acc: 0.9492 - val_loss: 0.1600 - val_acc: 0.9486\n",
      "Epoch 29/200\n",
      "103958/103958 [==============================] - 152s 1ms/step - loss: 0.1416 - acc: 0.9493 - val_loss: 0.1594 - val_acc: 0.9483\n",
      "Epoch 30/200\n",
      "103958/103958 [==============================] - 153s 1ms/step - loss: 0.1408 - acc: 0.9498 - val_loss: 0.1612 - val_acc: 0.9487\n",
      "Epoch 31/200\n",
      "103958/103958 [==============================] - 153s 1ms/step - loss: 0.1389 - acc: 0.9506 - val_loss: 0.1630 - val_acc: 0.9464\n",
      "Epoch 32/200\n",
      "103958/103958 [==============================] - 153s 1ms/step - loss: 0.1370 - acc: 0.9513 - val_loss: 0.1572 - val_acc: 0.9489\n",
      "Epoch 33/200\n",
      "103958/103958 [==============================] - 153s 1ms/step - loss: 0.1343 - acc: 0.9518 - val_loss: 0.1606 - val_acc: 0.9492\n",
      "Epoch 34/200\n",
      "103958/103958 [==============================] - 155s 1ms/step - loss: 0.1357 - acc: 0.9514 - val_loss: 0.1596 - val_acc: 0.9486\n",
      "Epoch 35/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.1319 - acc: 0.9527 - val_loss: 0.1602 - val_acc: 0.9481\n",
      "Epoch 36/200\n",
      "103958/103958 [==============================] - 153s 1ms/step - loss: 0.1322 - acc: 0.9525 - val_loss: 0.1559 - val_acc: 0.9500\n",
      "Epoch 37/200\n",
      "103958/103958 [==============================] - 152s 1ms/step - loss: 0.1304 - acc: 0.9530 - val_loss: 0.1583 - val_acc: 0.9505\n",
      "Epoch 38/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.1314 - acc: 0.9523 - val_loss: 0.1547 - val_acc: 0.9497\n",
      "Epoch 39/200\n",
      "103958/103958 [==============================] - 153s 1ms/step - loss: 0.1266 - acc: 0.9542 - val_loss: 0.1619 - val_acc: 0.9475\n",
      "Epoch 40/200\n",
      "103958/103958 [==============================] - 153s 1ms/step - loss: 0.1260 - acc: 0.9541 - val_loss: 0.1610 - val_acc: 0.9491\n",
      "Epoch 41/200\n",
      "103958/103958 [==============================] - 153s 1ms/step - loss: 0.1265 - acc: 0.9542 - val_loss: 0.1673 - val_acc: 0.9467\n",
      "Epoch 42/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.1232 - acc: 0.9554 - val_loss: 0.1574 - val_acc: 0.9504\n",
      "Epoch 43/200\n",
      "103958/103958 [==============================] - 156s 2ms/step - loss: 0.1229 - acc: 0.9547 - val_loss: 0.1618 - val_acc: 0.9489\n",
      "Epoch 44/200\n",
      "103958/103958 [==============================] - 153s 1ms/step - loss: 0.1197 - acc: 0.9559 - val_loss: 0.1594 - val_acc: 0.9501\n",
      "Epoch 45/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.1196 - acc: 0.9560 - val_loss: 0.1569 - val_acc: 0.9512\n",
      "Epoch 46/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.1200 - acc: 0.9558 - val_loss: 0.1589 - val_acc: 0.9508\n",
      "Epoch 47/200\n",
      "103958/103958 [==============================] - 153s 1ms/step - loss: 0.1206 - acc: 0.9560 - val_loss: 0.1646 - val_acc: 0.9474\n",
      "Epoch 48/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.1191 - acc: 0.9569 - val_loss: 0.1577 - val_acc: 0.9506\n",
      "Epoch 49/200\n",
      "103958/103958 [==============================] - 153s 1ms/step - loss: 0.1180 - acc: 0.9568 - val_loss: 0.1619 - val_acc: 0.9489\n",
      "Epoch 50/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.1164 - acc: 0.9576 - val_loss: 0.1566 - val_acc: 0.9504\n",
      "Epoch 51/200\n",
      "103958/103958 [==============================] - 157s 2ms/step - loss: 0.1147 - acc: 0.9581 - val_loss: 0.1602 - val_acc: 0.9497\n",
      "Epoch 52/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.1143 - acc: 0.9579 - val_loss: 0.1614 - val_acc: 0.9507\n",
      "Epoch 53/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.1146 - acc: 0.9575 - val_loss: 0.1589 - val_acc: 0.9501\n",
      "Epoch 54/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.1133 - acc: 0.9579 - val_loss: 0.1668 - val_acc: 0.9489\n",
      "Epoch 55/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.1124 - acc: 0.9584 - val_loss: 0.1607 - val_acc: 0.9498\n",
      "Epoch 56/200\n",
      "103958/103958 [==============================] - 153s 1ms/step - loss: 0.1108 - acc: 0.9591 - val_loss: 0.1602 - val_acc: 0.9500\n",
      "Epoch 57/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.1099 - acc: 0.9595 - val_loss: 0.1588 - val_acc: 0.9516\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 58/200\n",
      "103958/103958 [==============================] - 153s 1ms/step - loss: 0.1101 - acc: 0.9588 - val_loss: 0.1597 - val_acc: 0.9498\n",
      "Epoch 59/200\n",
      "103958/103958 [==============================] - 158s 2ms/step - loss: 0.1095 - acc: 0.9596 - val_loss: 0.1619 - val_acc: 0.9489\n",
      "Epoch 60/200\n",
      "103958/103958 [==============================] - 153s 1ms/step - loss: 0.1097 - acc: 0.9593 - val_loss: 0.1656 - val_acc: 0.9494\n",
      "Epoch 61/200\n",
      "103958/103958 [==============================] - 153s 1ms/step - loss: 0.1076 - acc: 0.9604 - val_loss: 0.1652 - val_acc: 0.9490\n",
      "Epoch 62/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.1083 - acc: 0.9593 - val_loss: 0.1605 - val_acc: 0.9492\n",
      "Epoch 63/200\n",
      "103958/103958 [==============================] - 153s 1ms/step - loss: 0.1058 - acc: 0.9603 - val_loss: 0.1615 - val_acc: 0.9502\n",
      "Epoch 64/200\n",
      "103958/103958 [==============================] - 153s 1ms/step - loss: 0.1060 - acc: 0.9604 - val_loss: 0.1636 - val_acc: 0.9495\n",
      "Epoch 65/200\n",
      "103958/103958 [==============================] - 153s 1ms/step - loss: 0.1061 - acc: 0.9607 - val_loss: 0.1646 - val_acc: 0.9495\n",
      "Epoch 66/200\n",
      "103958/103958 [==============================] - 155s 1ms/step - loss: 0.1034 - acc: 0.9616 - val_loss: 0.1686 - val_acc: 0.9488\n",
      "Epoch 67/200\n",
      "103958/103958 [==============================] - 157s 2ms/step - loss: 0.1039 - acc: 0.9612 - val_loss: 0.1640 - val_acc: 0.9508\n",
      "Epoch 68/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.1032 - acc: 0.9617 - val_loss: 0.1647 - val_acc: 0.9498\n",
      "Epoch 69/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.1042 - acc: 0.9613 - val_loss: 0.1699 - val_acc: 0.9491\n",
      "Epoch 70/200\n",
      "103958/103958 [==============================] - 155s 1ms/step - loss: 0.1019 - acc: 0.9616 - val_loss: 0.1632 - val_acc: 0.9510\n",
      "Epoch 71/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.1018 - acc: 0.9616 - val_loss: 0.1654 - val_acc: 0.9499\n",
      "Epoch 72/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.1019 - acc: 0.9619 - val_loss: 0.1669 - val_acc: 0.9503\n",
      "Epoch 73/200\n",
      "103958/103958 [==============================] - 155s 1ms/step - loss: 0.0999 - acc: 0.9627 - val_loss: 0.1646 - val_acc: 0.9493\n",
      "Epoch 74/200\n",
      "103958/103958 [==============================] - 153s 1ms/step - loss: 0.0987 - acc: 0.9622 - val_loss: 0.1671 - val_acc: 0.9500\n",
      "Epoch 75/200\n",
      "103958/103958 [==============================] - 156s 2ms/step - loss: 0.1009 - acc: 0.9622 - val_loss: 0.1634 - val_acc: 0.9497\n",
      "Epoch 76/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.0967 - acc: 0.9634 - val_loss: 0.1674 - val_acc: 0.9495\n",
      "Epoch 77/200\n",
      "103958/103958 [==============================] - 153s 1ms/step - loss: 0.0990 - acc: 0.9629 - val_loss: 0.1668 - val_acc: 0.9508\n",
      "Epoch 78/200\n",
      "103958/103958 [==============================] - 153s 1ms/step - loss: 0.0965 - acc: 0.9631 - val_loss: 0.1672 - val_acc: 0.9491\n",
      "Epoch 79/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.0980 - acc: 0.9630 - val_loss: 0.1657 - val_acc: 0.9497\n",
      "Epoch 80/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.0961 - acc: 0.9641 - val_loss: 0.1669 - val_acc: 0.9504\n",
      "Epoch 81/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.0954 - acc: 0.9637 - val_loss: 0.1653 - val_acc: 0.9509\n",
      "Epoch 82/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.0949 - acc: 0.9648 - val_loss: 0.1689 - val_acc: 0.9500\n",
      "Epoch 83/200\n",
      "103958/103958 [==============================] - 158s 2ms/step - loss: 0.0937 - acc: 0.9647 - val_loss: 0.1680 - val_acc: 0.9510\n",
      "Epoch 84/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.0933 - acc: 0.9645 - val_loss: 0.1646 - val_acc: 0.9507\n",
      "Epoch 85/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.0939 - acc: 0.9645 - val_loss: 0.1685 - val_acc: 0.9511\n",
      "Epoch 86/200\n",
      "103958/103958 [==============================] - 153s 1ms/step - loss: 0.0928 - acc: 0.9645 - val_loss: 0.1667 - val_acc: 0.9504\n",
      "Epoch 87/200\n",
      "103958/103958 [==============================] - 155s 1ms/step - loss: 0.0928 - acc: 0.9651 - val_loss: 0.1673 - val_acc: 0.9501\n",
      "Epoch 88/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.0915 - acc: 0.9647 - val_loss: 0.1666 - val_acc: 0.9519\n",
      "Epoch 89/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.0916 - acc: 0.9656 - val_loss: 0.1701 - val_acc: 0.9510\n",
      "Epoch 90/200\n",
      "103958/103958 [==============================] - 155s 1ms/step - loss: 0.0912 - acc: 0.9648 - val_loss: 0.1705 - val_acc: 0.9512\n",
      "Epoch 91/200\n",
      "103958/103958 [==============================] - 157s 2ms/step - loss: 0.0929 - acc: 0.9653 - val_loss: 0.1653 - val_acc: 0.9496\n",
      "Epoch 92/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.0903 - acc: 0.9653 - val_loss: 0.1685 - val_acc: 0.9497\n",
      "Epoch 93/200\n",
      "103958/103958 [==============================] - 155s 1ms/step - loss: 0.0895 - acc: 0.9658 - val_loss: 0.1691 - val_acc: 0.9505\n",
      "Epoch 94/200\n",
      "103958/103958 [==============================] - 155s 1ms/step - loss: 0.0915 - acc: 0.9651 - val_loss: 0.1720 - val_acc: 0.9501\n",
      "Epoch 95/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.0899 - acc: 0.9662 - val_loss: 0.1747 - val_acc: 0.9492\n",
      "Epoch 96/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.0893 - acc: 0.9658 - val_loss: 0.1702 - val_acc: 0.9502\n",
      "Epoch 97/200\n",
      "103958/103958 [==============================] - 155s 1ms/step - loss: 0.0906 - acc: 0.9657 - val_loss: 0.1728 - val_acc: 0.9497\n",
      "Epoch 98/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.0880 - acc: 0.9658 - val_loss: 0.1684 - val_acc: 0.9501\n",
      "Epoch 99/200\n",
      "103958/103958 [==============================] - 162s 2ms/step - loss: 0.0866 - acc: 0.9664 - val_loss: 0.1741 - val_acc: 0.9491\n",
      "Epoch 100/200\n",
      "103958/103958 [==============================] - 153s 1ms/step - loss: 0.0888 - acc: 0.9658 - val_loss: 0.1736 - val_acc: 0.9500\n",
      "Epoch 101/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.0890 - acc: 0.9661 - val_loss: 0.1730 - val_acc: 0.9503\n",
      "Epoch 102/200\n",
      "103958/103958 [==============================] - 153s 1ms/step - loss: 0.0882 - acc: 0.9664 - val_loss: 0.1743 - val_acc: 0.9503\n",
      "Epoch 103/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.0865 - acc: 0.9671 - val_loss: 0.1718 - val_acc: 0.9498\n",
      "Epoch 104/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.0859 - acc: 0.9669 - val_loss: 0.1729 - val_acc: 0.9495\n",
      "Epoch 105/200\n",
      "103958/103958 [==============================] - 153s 1ms/step - loss: 0.0861 - acc: 0.9665 - val_loss: 0.1773 - val_acc: 0.9511\n",
      "Epoch 106/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.0864 - acc: 0.9665 - val_loss: 0.1793 - val_acc: 0.9488\n",
      "Epoch 107/200\n",
      "103958/103958 [==============================] - 157s 2ms/step - loss: 0.0852 - acc: 0.9672 - val_loss: 0.1747 - val_acc: 0.9494\n",
      "Epoch 108/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.0850 - acc: 0.9669 - val_loss: 0.1720 - val_acc: 0.9495\n",
      "Epoch 109/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.0839 - acc: 0.9674 - val_loss: 0.1751 - val_acc: 0.9506\n",
      "Epoch 110/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.0842 - acc: 0.9674 - val_loss: 0.1747 - val_acc: 0.9501\n",
      "Epoch 111/200\n",
      "103958/103958 [==============================] - 155s 1ms/step - loss: 0.0843 - acc: 0.9672 - val_loss: 0.1719 - val_acc: 0.9512\n",
      "Epoch 112/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.0847 - acc: 0.9675 - val_loss: 0.1734 - val_acc: 0.9505\n",
      "Epoch 113/200\n",
      "103958/103958 [==============================] - 153s 1ms/step - loss: 0.0824 - acc: 0.9683 - val_loss: 0.1778 - val_acc: 0.9494\n",
      "Epoch 114/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.0809 - acc: 0.9684 - val_loss: 0.1787 - val_acc: 0.9487\n",
      "Epoch 115/200\n",
      "103958/103958 [==============================] - 158s 2ms/step - loss: 0.0823 - acc: 0.9678 - val_loss: 0.1734 - val_acc: 0.9508\n",
      "Epoch 116/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.0827 - acc: 0.9683 - val_loss: 0.1733 - val_acc: 0.9506\n",
      "Epoch 117/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.0796 - acc: 0.9688 - val_loss: 0.1710 - val_acc: 0.9505\n",
      "Epoch 118/200\n",
      "103958/103958 [==============================] - 155s 1ms/step - loss: 0.0827 - acc: 0.9681 - val_loss: 0.1735 - val_acc: 0.9505\n",
      "Epoch 119/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.0810 - acc: 0.9691 - val_loss: 0.1789 - val_acc: 0.9508\n",
      "Epoch 120/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.0801 - acc: 0.9691 - val_loss: 0.1755 - val_acc: 0.9502\n",
      "Epoch 121/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.0807 - acc: 0.9683 - val_loss: 0.1810 - val_acc: 0.9490\n",
      "Epoch 122/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.0803 - acc: 0.9690 - val_loss: 0.1743 - val_acc: 0.9503\n",
      "Epoch 123/200\n",
      "103958/103958 [==============================] - 158s 2ms/step - loss: 0.0814 - acc: 0.9687 - val_loss: 0.1710 - val_acc: 0.9504\n",
      "Epoch 124/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.0793 - acc: 0.9689 - val_loss: 0.1759 - val_acc: 0.9498\n",
      "Epoch 125/200\n",
      "103958/103958 [==============================] - 155s 1ms/step - loss: 0.0786 - acc: 0.9690 - val_loss: 0.1754 - val_acc: 0.9505\n",
      "Epoch 126/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.0793 - acc: 0.9690 - val_loss: 0.1794 - val_acc: 0.9500\n",
      "Epoch 127/200\n",
      "103958/103958 [==============================] - 155s 1ms/step - loss: 0.0778 - acc: 0.9697 - val_loss: 0.1831 - val_acc: 0.9501\n",
      "Epoch 128/200\n",
      "103958/103958 [==============================] - 155s 1ms/step - loss: 0.0783 - acc: 0.9695 - val_loss: 0.1800 - val_acc: 0.9500\n",
      "Epoch 129/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.0804 - acc: 0.9692 - val_loss: 0.1778 - val_acc: 0.9498\n",
      "Epoch 130/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.0778 - acc: 0.9694 - val_loss: 0.1787 - val_acc: 0.9508\n",
      "Epoch 131/200\n",
      "103958/103958 [==============================] - 158s 2ms/step - loss: 0.0779 - acc: 0.9698 - val_loss: 0.1817 - val_acc: 0.9496\n",
      "Epoch 132/200\n",
      "103958/103958 [==============================] - 155s 1ms/step - loss: 0.0788 - acc: 0.9692 - val_loss: 0.1784 - val_acc: 0.9502\n",
      "Epoch 133/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.0774 - acc: 0.9699 - val_loss: 0.1780 - val_acc: 0.9510\n",
      "Epoch 134/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.0767 - acc: 0.9703 - val_loss: 0.1836 - val_acc: 0.9498\n",
      "Epoch 135/200\n",
      "103958/103958 [==============================] - 156s 1ms/step - loss: 0.0760 - acc: 0.9702 - val_loss: 0.1787 - val_acc: 0.9495\n",
      "Epoch 136/200\n",
      "103958/103958 [==============================] - 155s 1ms/step - loss: 0.0771 - acc: 0.9702 - val_loss: 0.1807 - val_acc: 0.9506\n",
      "Epoch 137/200\n",
      "103958/103958 [==============================] - 155s 1ms/step - loss: 0.0766 - acc: 0.9699 - val_loss: 0.1830 - val_acc: 0.9488\n",
      "Epoch 138/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.0768 - acc: 0.9707 - val_loss: 0.1776 - val_acc: 0.9506\n",
      "Epoch 139/200\n",
      "103958/103958 [==============================] - 159s 2ms/step - loss: 0.0742 - acc: 0.9708 - val_loss: 0.1830 - val_acc: 0.9496\n",
      "Epoch 140/200\n",
      "103958/103958 [==============================] - 155s 1ms/step - loss: 0.0760 - acc: 0.9700 - val_loss: 0.1780 - val_acc: 0.9498\n",
      "Epoch 141/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.0747 - acc: 0.9704 - val_loss: 0.1823 - val_acc: 0.9501\n",
      "Epoch 142/200\n",
      "103958/103958 [==============================] - 155s 1ms/step - loss: 0.0754 - acc: 0.9704 - val_loss: 0.1806 - val_acc: 0.9509\n",
      "Epoch 143/200\n",
      "103958/103958 [==============================] - 155s 1ms/step - loss: 0.0735 - acc: 0.9710 - val_loss: 0.1843 - val_acc: 0.9505\n",
      "Epoch 144/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.0736 - acc: 0.9713 - val_loss: 0.1839 - val_acc: 0.9497\n",
      "Epoch 145/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.0736 - acc: 0.9708 - val_loss: 0.1827 - val_acc: 0.9500\n",
      "Epoch 146/200\n",
      "103958/103958 [==============================] - 155s 1ms/step - loss: 0.0747 - acc: 0.9707 - val_loss: 0.1818 - val_acc: 0.9502\n",
      "Epoch 147/200\n",
      "103958/103958 [==============================] - 158s 2ms/step - loss: 0.0744 - acc: 0.9708 - val_loss: 0.1817 - val_acc: 0.9504\n",
      "Epoch 148/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.0738 - acc: 0.9710 - val_loss: 0.1830 - val_acc: 0.9494\n",
      "Epoch 149/200\n",
      "103958/103958 [==============================] - 155s 1ms/step - loss: 0.0733 - acc: 0.9712 - val_loss: 0.1838 - val_acc: 0.9504\n",
      "Epoch 150/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.0723 - acc: 0.9720 - val_loss: 0.1795 - val_acc: 0.9508\n",
      "Epoch 151/200\n",
      "103958/103958 [==============================] - 153s 1ms/step - loss: 0.0725 - acc: 0.9713 - val_loss: 0.1861 - val_acc: 0.9495\n",
      "Epoch 152/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.0713 - acc: 0.9716 - val_loss: 0.1814 - val_acc: 0.9503\n",
      "Epoch 153/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.0725 - acc: 0.9717 - val_loss: 0.1858 - val_acc: 0.9484\n",
      "Epoch 154/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.0714 - acc: 0.9721 - val_loss: 0.1866 - val_acc: 0.9503\n",
      "Epoch 155/200\n",
      "103958/103958 [==============================] - 158s 2ms/step - loss: 0.0719 - acc: 0.9719 - val_loss: 0.1903 - val_acc: 0.9499\n",
      "Epoch 156/200\n",
      "103958/103958 [==============================] - 155s 1ms/step - loss: 0.0719 - acc: 0.9719 - val_loss: 0.1851 - val_acc: 0.9500\n",
      "Epoch 157/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.0734 - acc: 0.9709 - val_loss: 0.1887 - val_acc: 0.9499\n",
      "Epoch 158/200\n",
      "103958/103958 [==============================] - 156s 1ms/step - loss: 0.0723 - acc: 0.9712 - val_loss: 0.1813 - val_acc: 0.9500\n",
      "Epoch 159/200\n",
      "103958/103958 [==============================] - 158s 2ms/step - loss: 0.0720 - acc: 0.9718 - val_loss: 0.1813 - val_acc: 0.9506\n",
      "Epoch 160/200\n",
      "103958/103958 [==============================] - 156s 2ms/step - loss: 0.0709 - acc: 0.9722 - val_loss: 0.1851 - val_acc: 0.9502\n",
      "Epoch 161/200\n",
      "103958/103958 [==============================] - 157s 2ms/step - loss: 0.0702 - acc: 0.9726 - val_loss: 0.1856 - val_acc: 0.9515\n",
      "Epoch 162/200\n",
      "103958/103958 [==============================] - 155s 1ms/step - loss: 0.0722 - acc: 0.9719 - val_loss: 0.1864 - val_acc: 0.9501\n",
      "Epoch 163/200\n",
      "103958/103958 [==============================] - 160s 2ms/step - loss: 0.0709 - acc: 0.9719 - val_loss: 0.1878 - val_acc: 0.9501\n",
      "Epoch 164/200\n",
      "103958/103958 [==============================] - 155s 1ms/step - loss: 0.0701 - acc: 0.9721 - val_loss: 0.1789 - val_acc: 0.9507\n",
      "Epoch 165/200\n",
      "103958/103958 [==============================] - 156s 1ms/step - loss: 0.0699 - acc: 0.9726 - val_loss: 0.1878 - val_acc: 0.9504\n",
      "Epoch 166/200\n",
      "103958/103958 [==============================] - 156s 2ms/step - loss: 0.0702 - acc: 0.9723 - val_loss: 0.1872 - val_acc: 0.9508\n",
      "Epoch 167/200\n",
      "103958/103958 [==============================] - 157s 2ms/step - loss: 0.0700 - acc: 0.9726 - val_loss: 0.1847 - val_acc: 0.9503\n",
      "Epoch 168/200\n",
      "103958/103958 [==============================] - 156s 1ms/step - loss: 0.0704 - acc: 0.9728 - val_loss: 0.1847 - val_acc: 0.9505\n",
      "Epoch 169/200\n",
      "103958/103958 [==============================] - 158s 2ms/step - loss: 0.0699 - acc: 0.9724 - val_loss: 0.1820 - val_acc: 0.9496\n",
      "Epoch 170/200\n",
      "103958/103958 [==============================] - 156s 1ms/step - loss: 0.0693 - acc: 0.9726 - val_loss: 0.1844 - val_acc: 0.9512\n",
      "Epoch 171/200\n",
      "103958/103958 [==============================] - 165s 2ms/step - loss: 0.0682 - acc: 0.9727 - val_loss: 0.1885 - val_acc: 0.9498\n",
      "Epoch 172/200\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "103958/103958 [==============================] - 155s 1ms/step - loss: 0.0688 - acc: 0.9731 - val_loss: 0.1846 - val_acc: 0.9509\n",
      "Epoch 173/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.0700 - acc: 0.9726 - val_loss: 0.1858 - val_acc: 0.9506\n",
      "Epoch 174/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.0699 - acc: 0.9722 - val_loss: 0.1866 - val_acc: 0.9502\n",
      "Epoch 175/200\n",
      "103958/103958 [==============================] - 155s 1ms/step - loss: 0.0686 - acc: 0.9728 - val_loss: 0.1840 - val_acc: 0.9514\n",
      "Epoch 176/200\n",
      "103958/103958 [==============================] - 155s 1ms/step - loss: 0.0673 - acc: 0.9734 - val_loss: 0.1825 - val_acc: 0.9498\n",
      "Epoch 177/200\n",
      "103958/103958 [==============================] - 155s 1ms/step - loss: 0.0671 - acc: 0.9729 - val_loss: 0.1866 - val_acc: 0.9507\n",
      "Epoch 178/200\n",
      "103958/103958 [==============================] - 155s 1ms/step - loss: 0.0673 - acc: 0.9736 - val_loss: 0.1854 - val_acc: 0.9506\n",
      "Epoch 179/200\n",
      "103958/103958 [==============================] - 159s 2ms/step - loss: 0.0678 - acc: 0.9732 - val_loss: 0.1892 - val_acc: 0.9506\n",
      "Epoch 180/200\n",
      "103958/103958 [==============================] - 155s 1ms/step - loss: 0.0678 - acc: 0.9726 - val_loss: 0.1863 - val_acc: 0.9515\n",
      "Epoch 181/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.0666 - acc: 0.9734 - val_loss: 0.1837 - val_acc: 0.9512\n",
      "Epoch 182/200\n",
      "103958/103958 [==============================] - 155s 1ms/step - loss: 0.0666 - acc: 0.9736 - val_loss: 0.1888 - val_acc: 0.9513\n",
      "Epoch 183/200\n",
      "103958/103958 [==============================] - 156s 2ms/step - loss: 0.0667 - acc: 0.9741 - val_loss: 0.1832 - val_acc: 0.9515\n",
      "Epoch 184/200\n",
      "103958/103958 [==============================] - 155s 1ms/step - loss: 0.0669 - acc: 0.9737 - val_loss: 0.1920 - val_acc: 0.9508\n",
      "Epoch 185/200\n",
      "103958/103958 [==============================] - 155s 1ms/step - loss: 0.0677 - acc: 0.9733 - val_loss: 0.1851 - val_acc: 0.9503\n",
      "Epoch 186/200\n",
      "103958/103958 [==============================] - 157s 2ms/step - loss: 0.0639 - acc: 0.9747 - val_loss: 0.1917 - val_acc: 0.9501\n",
      "Epoch 187/200\n",
      "103958/103958 [==============================] - 159s 2ms/step - loss: 0.0678 - acc: 0.9731 - val_loss: 0.1919 - val_acc: 0.9497\n",
      "Epoch 188/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.0665 - acc: 0.9736 - val_loss: 0.1908 - val_acc: 0.9503\n",
      "Epoch 189/200\n",
      "103958/103958 [==============================] - 155s 1ms/step - loss: 0.0660 - acc: 0.9738 - val_loss: 0.1854 - val_acc: 0.9510\n",
      "Epoch 190/200\n",
      "103958/103958 [==============================] - 155s 1ms/step - loss: 0.0644 - acc: 0.9742 - val_loss: 0.1918 - val_acc: 0.9502\n",
      "Epoch 191/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.0652 - acc: 0.9740 - val_loss: 0.1939 - val_acc: 0.9515\n",
      "Epoch 192/200\n",
      "103958/103958 [==============================] - 155s 1ms/step - loss: 0.0653 - acc: 0.9740 - val_loss: 0.1975 - val_acc: 0.9506\n",
      "Epoch 193/200\n",
      "103958/103958 [==============================] - 155s 1ms/step - loss: 0.0658 - acc: 0.9737 - val_loss: 0.1915 - val_acc: 0.9499\n",
      "Epoch 194/200\n",
      "103958/103958 [==============================] - 155s 1ms/step - loss: 0.0646 - acc: 0.9742 - val_loss: 0.1935 - val_acc: 0.9500\n",
      "Epoch 195/200\n",
      "103958/103958 [==============================] - 158s 2ms/step - loss: 0.0646 - acc: 0.9738 - val_loss: 0.1925 - val_acc: 0.9504\n",
      "Epoch 196/200\n",
      "103958/103958 [==============================] - 156s 1ms/step - loss: 0.0643 - acc: 0.9743 - val_loss: 0.1968 - val_acc: 0.9498\n",
      "Epoch 197/200\n",
      "103958/103958 [==============================] - 154s 1ms/step - loss: 0.0649 - acc: 0.9746 - val_loss: 0.1865 - val_acc: 0.9504\n",
      "Epoch 198/200\n",
      "103958/103958 [==============================] - 155s 1ms/step - loss: 0.0633 - acc: 0.9748 - val_loss: 0.1983 - val_acc: 0.9498\n",
      "Epoch 199/200\n",
      "103958/103958 [==============================] - 156s 1ms/step - loss: 0.0653 - acc: 0.9746 - val_loss: 0.1927 - val_acc: 0.9497\n",
      "Epoch 200/200\n",
      "103958/103958 [==============================] - 155s 1ms/step - loss: 0.0628 - acc: 0.9743 - val_loss: 0.1913 - val_acc: 0.9505\n",
      "Test loss: 19.93071638126316 %\n",
      "Test accuracy: 94.90865384615384 %\n",
      "saved model to disk\n",
      "Train: 0.987, Test: 0.951\n"
     ]
    },
    {
     "ename": "KeyError",
     "evalue": "'accuracy'",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mKeyError\u001b[0m                                  Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-15-961bd9c4afb2>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m     81\u001b[0m \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'Train: %.3f, Test: %.3f'\u001b[0m \u001b[1;33m%\u001b[0m \u001b[1;33m(\u001b[0m\u001b[0mtrain_acc\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mvalid_acc\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     82\u001b[0m \u001b[1;31m# plot training history\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 83\u001b[1;33m \u001b[0mpyplot\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mplot\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mhistory\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mhistory\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'accuracy'\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mlabel\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;34m'train'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     84\u001b[0m \u001b[0mpyplot\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mplot\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mhistory\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mhistory\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'accuracy'\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mlabel\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;34m'validation'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     85\u001b[0m \u001b[0mpyplot\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mlegend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mKeyError\u001b[0m: 'accuracy'"
     ]
    }
   ],
   "source": [
    "# TensorFlow's tensor expect the input shape (samples, rows, cols, channels) \n",
    "if K.image_data_format() == 'channels_first':\n",
    "    x_train = x_train.reshape(x_train.shape[0], 1, img_rows, img_cols)\n",
    "    x_valid = x_valid.reshape(x_valid.shape[0], 1, img_rows, img_cols)\n",
    "    x_test = x_test.reshape(x_test.shape[0], 1, img_rows, img_cols)\n",
    "    input_shape = (1, img_rows, img_cols)\n",
    "else:\n",
    "    x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)\n",
    "    x_valid = x_valid.reshape(x_valid.shape[0], img_rows, img_cols, 1)\n",
    "    x_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)\n",
    "    input_shape = (img_rows, img_cols, 1)\n",
    "\n",
    "# normalize the input data for reducing over-fitting\n",
    "x_train = x_train.astype('float32')\n",
    "x_valid = x_valid.astype('float32')\n",
    "x_test = x_test.astype('float32')\n",
    "x_train /= 255\n",
    "x_valid /= 255\n",
    "x_test /= 255\n",
    "print('x_train shape:', x_train.shape)\n",
    "print(x_train.shape[0], 'train samples')\n",
    "print(x_valid.shape[0], 'valid samples')\n",
    "print(x_test.shape[0], 'test samples')\n",
    "\n",
    "\n",
    "# convert class vectors to ont hot encoding\n",
    "y_train = keras.utils.to_categorical(y_train, num_classes,dtype='float32')\n",
    "y_valid = keras.utils.to_categorical(y_valid, num_classes,dtype='float32')\n",
    "y_test = keras.utils.to_categorical(y_test, num_classes,dtype='float32')\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Conv2D(32, kernel_size=(3, 3),\n",
    "                 activation='relu',\n",
    "                 input_shape=input_shape))\n",
    "model.add(BatchNormalization(axis=-1))\n",
    "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "model.add(Conv2D(64, (3, 3), activation='relu'))\n",
    "model.add(BatchNormalization(axis=-1))\n",
    "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Flatten())\n",
    "# model.add(Dense(128, activation='relu'))\n",
    "model.add(Dense(1568, activation='relu'))\n",
    "model.add(BatchNormalization(axis=-1))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(1000, activation='relu'))\n",
    "model.add(BatchNormalization(axis=-1))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(700, activation='relu'))\n",
    "model.add(BatchNormalization(axis=-1))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(300, activation='relu'))\n",
    "model.add(BatchNormalization(axis=-1))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(100, activation='relu'))\n",
    "model.add(BatchNormalization(axis=-1))\n",
    "model.add(Dense(num_classes, activation='softmax'))\n",
    "\n",
    "model.compile(loss=keras.losses.categorical_crossentropy,\n",
    "              optimizer=keras.optimizers.Adadelta(),\n",
    "              metrics=['accuracy'])\n",
    "tensorboard = TensorBoard(log_dir=\"logs_emnist_5/{}\".format(time()))\n",
    "# simple early stopping\n",
    "es = EarlyStopping(monitor='val_acc', mode='max', verbose=1)\n",
    "history= model.fit(x_train, y_train,\n",
    "          batch_size=batch_size,\n",
    "          epochs=epochs,\n",
    "          verbose=1,\n",
    "          callbacks=[tensorboard],\n",
    "          validation_data=(x_valid, y_valid))\n",
    "\n",
    "score = model.evaluate(x_test, y_test, verbose=0)\n",
    "print('Test loss:', score[0]*100,'%')\n",
    "print('Test accuracy:', score[1]*100,'%')\n",
    "model.save('emnist_v5.h5')\n",
    "print('saved model to disk')\n",
    "\n",
    "# below part caused key errors, so moved down a cless to re-run the plotting\n",
    "# it does not impact the training and evaluation parts\n",
    "# evaluate the model\n",
    "_, train_acc = model.evaluate(x_train, y_train, verbose=0)\n",
    "_, valid_acc = model.evaluate(x_valid, y_valid, verbose=0)\n",
    "print('Train: %.3f, Test: %.3f' % (train_acc, valid_acc))\n",
    "# plot training history\n",
    "pyplot.plot(history.history['accuracy'], label='train')\n",
    "pyplot.plot(history.history['accuracy'], label='validation')\n",
    "pyplot.legend()\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Rerun evaluate the model part as they key shall be \"acc\" for training accuracy (not accuracy), key for validation is \"val_acc\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train: 0.987, evaluate: 0.951\n",
      "Test: 0.949\n"
     ]
    }
   ],
   "source": [
    "# evaluate the model\n",
    "_, train_acc = model.evaluate(x_train, y_train, verbose=0)\n",
    "_, valid_acc = model.evaluate(x_valid, y_valid, verbose=0)\n",
    "print('Train: %.3f, evaluate: %.3f' % (train_acc, valid_acc))\n",
    "\n",
    "_, test_acc = model.evaluate(x_test, y_test, verbose=0)\n",
    "print('Test: %.3f' % (test_acc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Q: When finished, evaluate your results on the test set. Compare the performance on the test set with your previous networks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As in project 3, we employed deep neural networks. It is much faster to train to reach the 90%+ test accuracy with just several epochs. Compared with the project 2 where we employed MLP that we used many epochs and long time training, our model's final test accuracy is 85.98%.\n",
    "In project 3 our final model test accuracy is above 95% and the speed of training is much faster than the one in project 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEWCAYAAACXGLsWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XmcXFWZ//HP0/u+d0L2dCCQBIQAIUFAAQFld3dAUBh1GBfcRseB0VEGxxlHHR1x+ImoDC4gAoJERBl2VLYkEJZsJIEsnYSk01t6766q5/fHuZ1UOl3dlZjq7iTf9+vVr666S9VTt6rOU+ece841d0dERGQoWaMdgIiIjH1KFiIiMiwlCxERGZaShYiIDEvJQkREhqVkISIiw1KyEAHM7FYz+7c0t11nZmdnOiaRsUTJQkREhqVkIXIQMbOc0Y5BDk5KFnLAiJp//tHMXjKzDjP7qZmNN7M/mFmbmT1sZpVJ219sZsvMrMXMHjez2Unrjjez56P9fg0UDHiuC81sabTvU2Z2bJoxXmBmL5jZDjPbaGbXDVh/WvR4LdH6K6PlhWb2X2a23sxazezP0bIzzKx+kONwdnT7OjO728x+aWY7gCvNbL6ZPR09xxYz+x8zy0va/2gze8jMmsxsq5n9s5kdZmadZladtN2JZtZgZrnpvHY5uClZyIHmvcA5wJHARcAfgH8Gagif588AmNmRwK+AzwG1wAPA78wsLyo4fwv8AqgC7ooel2jfE4BbgL8HqoEfAQvNLD+N+DqADwMVwAXAJ8zsXdHjTo3i/UEU01xgabTfd4ATgVOimL4EJNI8Ju8E7o6e8zYgDnw+OiZvBs4CPhnFUAo8DPwRmAgcATzi7m8AjwMfSHrcy4E73L0vzTjkIKZkIQeaH7j7VnffBPwJeNbdX3D3HuBe4Phou78Bfu/uD0WF3XeAQkJhfDKQC/y3u/e5+93AoqTn+DvgR+7+rLvH3f1nQE+035Dc/XF3f9ndE+7+EiFhnR6tvgx42N1/FT1vo7svNbMs4CPAZ919U/ScT0WvKR1Pu/tvo+fscvcl7v6Mu8fcfR0h2fXHcCHwhrv/l7t3u3ubuz8brfsZIUFgZtnApYSEKqJkIQecrUm3uwa5XxLdngis71/h7glgIzApWrfJd59Fc33S7WnAF6JmnBYzawGmRPsNycwWmNljUfNNK/Bxwi98osdYO8huNYRmsMHWpWPjgBiONLP7zeyNqGnq39OIAeA+YI6ZzSDU3lrd/bl9jEkOMkoWcrDaTCj0ATAzIxSUm4AtwKRoWb+pSbc3At9w94qkvyJ3/1Uaz3s7sBCY4u7lwE1A//NsBA4fZJ/tQHeKdR1AUdLryCY0YSUbOHX0D4GVwEx3LyM00w0XA+7eDdxJqAF9CNUqJImShRys7gQuMLOzog7aLxCakp4CngZiwGfMLMfM3gPMT9r3x8DHo1qCmVlx1HFdmsbzlgJN7t5tZvOBDyatuw0428w+ED1vtZnNjWo9twDfNbOJZpZtZm+O+kheBQqi588FvgIM13dSCuwA2s1sFvCJpHX3A4eZ2efMLN/MSs1sQdL6nwNXAhcDv0zj9cohQslCDkruvorQ/v4Dwi/3i4CL3L3X3XuB9xAKxWZC/8Y9SfsuJvRb/E+0fk20bTo+CVxvZm3AVwlJq/9xNwDnExJXE6Fz+7ho9ReBlwl9J03AfwJZ7t4aPeZPCLWiDmC3s6MG8UVCkmojJL5fJ8XQRmhiugh4A1gNnJm0/i+EjvXno/4OEQBMFz8SkWRm9ihwu7v/ZLRjkbFDyUJEdjKzk4CHCH0ubaMdj4wdaoYSEQDM7GeEMRifU6KQgVSzEBGRYalmISIiwzpoJh2rqanx6dOnj3YYIiIHlCVLlmx394Fjd/Zw0CSL6dOns3jx4tEOQ0TkgGJm64ffSs1QIiKSBiULEREZlpKFiIgM66DpsxhMX18f9fX1dHd3j3YoB5yCggImT55Mbq6ueyMiB3myqK+vp7S0lOnTp7P7BKMyFHensbGR+vp66urqRjscERkDDupmqO7ubqqrq5Uo9pKZUV1drRqZiOx0UCcLQIliH+m4iUiyg7oZSkRktPRPpdT/w6u7L859Szdx5qxxjCst2G3bls5ecrKzKMnfvUiOJ5wsC4/R2tnH2u3tbG7pYlNzF7GEU1mUR11NMTPHl1BTks4l4vedkkWGtbS0cPvtt/PJT35yr/Y7//zzuf3226moqMhQZCKSSDgOZGftqknH4gnWNXbS1t3H7AllFORmA9DVG+fPa7aztqGduVMq2NDYyUubWpgzoZz5dZVMKC/k4RVbWbGljU0tXTy9djtdvXGOnlTOmyaV8+SrDaze1s5hZQV8/PQZrNjSxpYd3axv7GB9YycAeTlZ5GdnMb68gLKCHJZt3kFudhY1JXmsi7YZzNETy/j9Z96S0WN10EwkOG/ePB84gnvFihXMnj17lCIK1q1bx4UXXsgrr7yy2/J4PE52dvYoRZWesXD85NBS39zJU2sbOWZiOTNqi9nS2k17d4xNLZ28vKmVvOxsygtzyM4y3jKzluk1xQBsauniz6sbWLGljVVvtNHY0cNbZtZSV1NMZ2+Mzt44K7e0sWxLK4eVFZCXk0V9cxdbWrrB4IjaEmKJBE0dfbR09hJLhHIxN9uYM7Gc8sJcnn2tkZ5YYrd4C3Kz6O4Ly8zAPexTW5LP/LoqSgtyeXlTK8u37KCqKI/PnDWT//f4Guqbu6goymVaVRETygs5bkoFZtDc2UtPX4LNLV00d/ZyzKRy4gln645u3jSpnNkTyphUWcikikJys7No7OjltYZ24gnnjKPG7dMxN7Ml7j5vuO1Us8iwa665hrVr1zJ37lxyc3MpKSlhwoQJLF26lOXLl/Oud72LjRs30t3dzWc/+1muuuoqYNf0Je3t7Zx33nmcdtppPPXUU0yaNIn77ruPwsLCUX5lcqhxd+qbu2js6KWsIIeywlzKCsKp1Y+u3MbzG5rpjSWYN72St8ys5bWGdpo7e9nU0s2fV4df1W3dMeZMKGN6dRFF+Tn0xRI0dfbyRms3W1q7eX17R8rnz84y4gnf7f4ph1ezbUcPq7aGGdULc7M58rBSxpcV8Iun19Mb31W4T6oo5Lgp5TS09dDZG+dNk8o575gJxOIJVm9rpyA3ixOn5VFVnEddTQkl+dks3djKCxua2dzSxaXzp3L27PHMmlDKCxtaGFeaz7GTy1nX2MmidU28vr2DM46s5aTpVWRl7d7n1xdPkGVGdpbxruMnsrmlmxk1xXtst7cmVYTEMRIOmZrFv/5uGcs379ivzzlnYhlfu+joIbdJrlk8/vjjXHDBBbzyyis7T0ltamqiqqqKrq4uTjrpJJ544gmqq6t3SxZHHHEEixcvZu7cuXzgAx/g4osv5vLLL9+vr2UwqlkcOGLxBAkPzRjrGzt4YUMLAAtmVDGutIClG1sozs+mtiSfN3Z009kbpy+eoCeWYPG6Jl7etIPaknwmVRZSVZRLfXMXr2/vYGtbN2+aVEE8keCxVQ00tPXs8dz9hXheThbZZnT1xffYZlJFIcdOLqcoL4dlm1vZ0tpNR0+MvJwsKgpzmVBRyGFlBcyZWMYZR9XyyqZWtu7oYVJFIWWFudSW5jN7QimG0d4To6Mnxk///DpPr21kYkUBJ9VV8fY545lRU7KzAO7oidHRG6MoL4fC3OzdmppkF9Usxqj58+fvNnbhhhtu4N577wVg48aNrF69murq6t32qaurY+7cuQCceOKJrFu3bsTildHR0ROjN5agpCCH3Ows2ntiPL++mXuer2dNQzuxuHPyjGoqi/JYtK6J5zc0k3DnmInl0e3wOFkG5YW5NHf2pXyu7CzjyPGlvPpGG1vbunGH/Jws6mqKqSnJ5/4XNwNw+lG1LJhRzcTyAtq6Y+zo7mNHVx/tPXHm11Xy1pm1mBlPvtrAss2tHDm+lHFlBVQW5TK1qmivzrA7emJ5ynVVOeHX/3UXD/1DrTg/h+J8FXH7yyFzJIerAYyU4uLinbcff/xxHn74YZ5++mmKioo444wzBh3bkJ+/6yyH7Oxsurq6RiRWGV4i4axtaGdKVdHOjlAIbegvbmzh2MnlTCwv5I0d3axtaOeZ1xp5bGUDWVlQU5K/86+rN8aTq7eTnWUU5mazbHPrzgI/LyeL3qitvKwghxOnVRJLOLc/t4G+eIKjxpfyvhMnA/Dc60187C0zeO8Jk0m48/uXtlDf3MlZs8fjwPa2HiaUF1BakEtOtpGbbcyoKaGyOA+A3liClq5eaorzd/5CjyccdycnO70z7c+cNY4zZ+1b+7mMXYdMshgtpaWltLUNfoXK1tZWKisrKSoqYuXKlTzzzDMjHN2haUtrFz19CfJzs6gozKMgNwszI5FwXtrUysamTqZUFfHYym2s2dbOWbPH0d2XYPW28D6ubeigvrmTt86sZdG6JpZt3kFOljGuNJ/cnCzau2M0dvTufL7kTtAsg/l1VRTkZrO9vYeVW0JnbJYZpx1RQ0620drVx6fOPIKq4jzau2O09cQoL8xl1mGlnHpEzc6k1NkbI5bwnf0Gg5k9oWyvjk1eTtYep3WG5hs14RzqlCwyrLq6mlNPPZVjjjmGwsJCxo8fv3Pdueeey0033cSxxx7LUUcdxcknnzyKkR743J2G9h5qS/J3a/JYsr6ZhUs3UVWcz5INzTz5asNu++XlZFGUl01fLEFH7672djOoLs7j9y9vAaAoLxsDJlcWMaWyiNuf3UBtaT5fu2gO29t72LqjZ2fTUV11MSdMq2DRuma27ehhRm0xh9eWMOuw0p2/4pPjjifS/+XeryhPX18ZOYdMB7fsvbF8/Fa+sYP7X9xCSUEOR08so7svwY+eWMvi9c3UlORTUZRLW3cfWWZsae0mPyeLnliC6uI8rjxlOpOrCunqTdDa1UdLVy9dvXGyzDhuSjkzx5WyoamToyeWMaWyiJc3tVJWmMv06t3b3bt64+Rm214X8iJjiTq45YDSG0vw8qZWbntmPfUtXdRVF/PqtjZ2dPUxa0IZrZ19bGvrJuGwbUc3O7pje5xKWVOSx+fPPpL1jR109cUpK8gllnBmTyjl0vlTyTKL2umHL9yPmbSrg/W4KYMPjCzMG9vjZET2JyUL2e/iCae+uZPWrj7mTCijqbOXFza0UFGYywMvb+HxVxv4wLwpZJlx1+KNtPXEaO4IA6FK8nOYOb6E/1v+BofXljCjtoSX6luoiqY1yDLjzTOqOfKwUi540wTiCWfNtnZyso3ZE8r2mC5BRPYPfbNkr4UBVF1UF4fz8rMM3tjRzYsbW3lydQMPvLyFluhUzbKCHNp7YjvP7MnJMo6ZVM63H1wFwCmHV7Oguoiq4jyOHF/K22aNo3SIDtvB1JZmdk4cEVGykGF09sZ48tXtuDurtrbxuxc3s7Zh1yjbwtxsivNz2N4eBmsV5Gbx9jmHceoR1RTm5fCX1dupLc3nzFm17OiOcURtCVOqinhxYwtZZrxpcurz6UVk7FCyOIQl3OmNJcjPyaK7L05jRy+xuGMGOdlZNHX0csl/PEprV6glmMHJddVcOn8qdTXFNLb3svKNNnZ093HMxDKOnVLBnKSJ1wAuPm7ioM+dqh9ARMYmJYtDSHt3LJqjBuLuNLT10hOLk5OVRSyRINuM3Jws3CEWjSA+7YgaLlswlcriPKqL8xhXVjD8E4nIQSejycLMzgW+D2QDP3H3bw5YPw24BagFmoDL3b0+WhcHXo423eDuF2cy1rGuf66ompqaYbftisYKJNzp6I3x8AP3M6XucMon7n6J1LycLCaUF9LZGyMvJ5dxpflkZ+06UyirtYAbLxubp86KyMjKWLIws2zgRuAcoB5YZGYL3X150mbfAX7u7j8zs7cB/wF8KFrX5e5zMxXfwaI3lqC9py865z+Lrr74zmajfr+++x7eevY7eP8RRzG+NJ8EYTxuXk4WWWZAUgdxIh7mWc7+Kz8arZtg/V/gmPdB1n4ah5BI7L/HcgdPQFY2xGPgcciJjkP3Dnjs36GqDmZfDIk+KJsUtt3f4n2QPUyHflcLWBYU7N1o7ENWZxOsfwqOPHfvPsexHuhph+LqwdfXLwmfiaKqPde1bYXXn4TySZBbBJ3boaMRimug7q2p3+P+cW5msGMLbF0Wts/JG3z7UZTJmsV8YI27vwZgZncA7wSSk8Uc4PPR7ceA32YwnlHzy1/+khtuuIHe3l4WLFjAsccey/r16/nWt74FwK233sqSJUv4wQ9+kHLK8n79s9g++ezztHT2cuP3v0dnZwdXf+Fa7rztVn5z+8/weIzDZ8zglh9+j5UvvcCfHv4DSxc9xS9++D1+85vfAPCpT32KhoZtFOXn8uMf/5RZs+dARwO0bw0FZ3YexOPQvg3+eC2c9DGoPhyeuxlqZ4WCc9FPoWIaHP1uOOKskGgaVsL2VfDAP0JnI6z4HbznZsgdZBrl1k2w+kGYdirUHrVr+dbl8LvPwLRTYMHHoagafvtJWPsonPoZyC+F3k44/vJQ4L/xEpQcFr6oOzbD//0LVE6Dt38jPN6Gp2DNI/Da4+F1HXUevHw3NK0N+3VsA8sOr2H8MbD8t+F1APzhS+H/pBPhfbdAbjGsWAgbn4XxR8PMd0DNkbD2EejZAePmQOMaKKwK8bdtgdf/FB4v3hsKmhlnQO1sWHk/3P8PMPdSOO9bocAAaHodnroh3M4rhud+AokYHP42OP1LMHleSHAPXguvPQGFFVBYGZJdrCc8p2WF4zrvI1A2EZ7/ObRsCIXW1FOg6TXYthwmHBfW98svg3gPPHtzeO7TPhcKsGX3woZn4E3vg7d8MbwHz94EHdvh7K/BK/eEY/Dmq6F0Qnj8xtWw6g+w5UWorIMZp8Psi0Lya1kf3v+uZqiZGT5Dr/wmvE+tG+CEK2DGmfDqH+GY90JxLbzwixB/5fTwvidi4TMX64H6ReG4lx4WYulqgsnz4cQrw2f69SehrxNKxkPj2nB8Js6FWDcU1YT3/sF/htZ6OP2fYONzsPEZqJoBNUfBjk3w2mNQOhHOvBZW/RHaNofPDYTXmEgxWWNhFUw/FQ47NsRXc1SIfdUDsPL34Rge+XZ49UHobQ8xHncpzDwnvL6ty8LxmnZKeK1bXwmfx/HHQF8XvHRHeJ63/9sQpdBfL2MjuM3sfcC57v6x6P6HgAXufnXSNrcDz7r7983sPcBvgBp3bzSzGLAUiAHfdPc9EomZXQVcBTB16tQT169fv9v63UYg/+EaeOPlgQ/x1znsTXDeN4fcZMWKFXzpS1/innvuITc3l09+8pMsWLCAr3/966xZswaA8847jy9/+cucdtppg05ZXlVZybTp07n/wYdpbe/iY5f/Dfc98mfIyePXN3+feEcT//qlq9ne1k3VhOlkZ+fwlX/8DOOrK/j0Rz/IlZ/9Fy4853Ted+UnwJ2zzjmXm378E2bW5PLsU3/m2m/eyKN3/zR82PPLIL8E2rayor6J2a/fAsvvg6ycsK6radeLq5wevuzdrTDlZGjdGL5UEBLK7Ivgye+EwmPW+SEJWFb4dda+LRT+/V+wwqpQkE87JXwpPQE90ZxaZZPCY088ATY/v+v5c4vDl90HTImdVwq9bVA+JSTAWHd47Kknhy/mtuUw7miYeTa0vRG+wL2d4cvaujEUvO+/NRSIG58Nv/6f/HZIBv2KqkMyhFDYdG7f882vOhyaX49qMDmQUxAKg2Rlk8Ixm30RNK+D9oYQc3Zu2Ke3PRSWZRPhxV+HxHb428K61f8XbidioQCO9YRfpCWHhUKk/rnwy7W4NhRsucUhYfUf8/yy3V9TsuLacEz6ojPfyibDpONh5QPheFv2ruNeMTUkoqycEMvA1zft1HAc6hft+Tw5hRDrItR1PRyz/FLYsnTXNuVTQkJZ++jgsUJ47qoZ0LIxJNNZF8Lj/x4+mxCSc1E1tL8Rtov3hR8ZecXhMxDvheJx4UfLuj+Fz9DsC0PyaFgVXteCv4elt4eCu3hcSLQeDwX6+GPgTe8NtZp4b/hMFFXB9ldh+ULY8HTYL1l2HtSdHrZb+XuY+mY4/jJY+itY8/Dun+vs/JDEYdexSn7tx7wX3v2jXT849sJYGME9WNQDM9MXgf8xsyuBJ4FNhOQAMNXdN5vZDOBRM3vZ3dfu9mDuNwM3Q5juY38Gv7888sgjLFmyhJNOOgmArq4uxo0bx4wZM3jmmWeYOXUCq5a/wqlHjYO2LXz/v/8f9963EDxMWf7KC8/xljkTyPIYE+P1lCW6yLUYR2VvgvKpFMdaaI91YbEeli95mq9863JadrTT3tHJO95xbvg1U1AWPkTb19De3s5Tzy3i/e95185mmJ6enugX27TwRYVQGMS2hERx6udg++pQ4Fx5fygke9rhyHeEx1hyK/z5v8OX8KyvhoJt8kmhNlH3VvjTd8M2E44LyeLVP4Yv24lXwPEfCk0GjWtCwbjm4VBQXXZXKOhe/FX4VXjml8Mv8G0rQjW/twOe/WH4Us44PXxJW+vDF3XeR8KX85kfwqwL4PCzwi+7vOLwmK31UbPSgCatC74DsWgCwP5mgEknhP+zLoBl90BeCUyZHxJX+1Z4+a7wi/vod4ea17aV4f/WV+Dl34TlR787JM/snJAM1v0pFJ75ZeGX7+8+By/eHgqLiSeE1z//70Lh1tUCJbUhhtOvgadvhKW3hYLn7OvgtM+TUttWeOwb4Zf0e34U3ovejlBol00OcTavg+5w7Qvcw+3eDjji7PDcy38Lk+aFX7JZWaHgXPNIqL3Mvjj8v/fj4XWc9bVQYzMLPwgqo7/+49y8LrzXJePDD42ySaE2tO5PsOL+UDhPjy4Nuvy+kLjHzYZ7PxFqhRd8NxSuO+rD5zMrJ/rLhuojwufcfVeBOfeD4bNaUD5401G/9m3hc3f420Jsax+BCXNDE1K//sdd8PFQi5h68q5my6HUzAyfHYC+7pCstq0IyerwM3d935LjPvrdoQlr8wvhOaoPD5+J+kXh/vhjYNPzIUEnYuG9Kh0/+PPvT+6ekT/gzcCDSfevBa4dYvsSoD7FuluB9w31fCeeeKIPtHz58j2WZVwi4d7V6h6Pubv7DTfc4Ndcc82u9X1d7m1b/Sc3fMs///dX+I/+88v++b//sPe9scIfvetmP+Wkud60+lmP1b/gp5483x+982Zvr1/mU6dM8YYNa3zjqhd99qyj3Le87L7pef/6l672r/3Ll93dffr06b706SfcWzf5//7kR37FFVe4u/sVV1zhd91xm/sbr3jr6y/6YePHuW96PjxGPBZiHuR1LH/ucfcfne4e690/xyUd8Xj62x4sEgn37h17t33btszFs7f2x+djKK2bw+dVMgJY7GmU6ZmcAW0RMNPM6swsD7gEWJi8gZnVmFl/DNcSzozCzCrNLL9/G+BUdu/rGLu6mkJbeMNK6GnjrLPO4u67fs225U/D9tU0rXqG9csX8Z6zT+a3f3iEX9z3KPMv/DDL+w5jdXsB5RWVFJVVs2x9A4tfWEoiJ5+Cw47EsrKgsJzxdbPZ1rCdRiro8Tzuf/zZ8OsKaGtrY8KMWfQV1nLbHXftDKm0tJS2zh4YN4ey6cdSN+Nw7nrkeaicjlsWL7700p6vwyz8mvnoQ8N3wKYj3epxVtY+VaUPaGa7fmGmu31/bWMs2B+fj6GUTYCJx2f2OWRYGUsW7h4DrgYeBFYAd7r7MjO73sz6T4M9A1hlZq8C44GoR5LZwGIze5HQ8f1N3/0sqrElEQ9to/G+cEZDTgFg0LiGORNL+LcvXMXbP/C3HPvWCznnss+wqbeU+OT5TJl5NOvqN3PaqW9mRm0JH77k/Xh2Pied/W6+/r2bOfnkk8mumEx20hkdubm5fPWrX2XBaWdw4ZWfCx3Tka9//essWLCAc845h1mzZu1cfskll/Dtb3+b4084gbVr13Lbbbfx09vu4rj5p3L00Udz3333Df66zDJfEIjIAUFTlO+rWHfoGMvOC2278R52djxVzwzt9U2vQ28bidwiWoumk52dRWdPjObOPvriCUoLcplSWThmp7gey1OUi8j+MRY6uA8+Pe2hE7awIiSCWA/gIWFUTA1JIzsP8ktwdxKVdfS2N7K+PZfennApVMMoKchhUmUhpfk5e3VdYhGR0aJkkQ73cPZJV3O43xaunEbV4eEMG4va2YvCYJ7uvjj1zV109saAPApyszmispBEIky0N1ZrEiIiqRz0ycLd/7pf7/2nWnY1h9PqCivDKYl5RYOOqN3R1ceGpk7MYHxZATnZRkVh7m7TaBwIDpbmSRHZPw7qZFFQUEBjYyPV1dX7njC6W8KAq5Lxu0a6Vk0fdNPWKFEU5GYxvbo4rSuyjUXuTmNjIwUFmjRQRIKDOllMnjyZ+vp6Ghoa9v1BOraHgV6l+WCtKTfr6o3T1NFLbk4WNSV5rGk8sPsiCgoKmDx58miHISJjxEGdLHJzc6mrqxt+w4G6d4R5bSaeAN8+Iswb856bB900nnBuemIt333odU6YWsEtV56011d6ExEZ6w7qZLHP/vBPYXKu9/8sNEHVnT7oZj2xOFf9fAlPvNrABcdO4FvvPZZiXQNaRA5CKtkGal4HL/06zHm08NNhWd1b99gskXD+4c4XeeLVBv7tXcdw2YKpOg1WRA5aB2YPbCb95fthYrJZF4bO7aoZUDFlj83+66FV/P6lLVx73iwuP3maEoWIHNSULJL1tMELvwyzVb7jG2Fmy0GaoP7w8hZufGwtl86fwt+ffvgoBCoiMrLUDJVs43PhzKfZF4cplD/yx1CzSLJkfROfv3Mpx0+t4LqLjx6dOEVERpiSRbINT4faxJT54X7//8grm1r52/9dxMTyQn784Xnk52TgMpsiImOQmqGSrX8KJhw76HTRi9Y1cenNz1CSn8PPPzqfmpI0LnwiInKQULLoF+uB+sXhEpADdPbG+LufL6a2LJ+7P3EKkyuLRiFAEZHRo2aofpueD9OMTztlj1X3vrCJls4+fvzheUysKByF4ERERpdqFv02PBX+T33zbovdnVv/so6jJ5Yxb1rlKAQmIjL6lCz6Na+H4nF7XNhO5H3zAAAW/ElEQVT9qbWNrN7WzpWnTNdYChE5ZClZ9Otq2iNRAPzvX9ZRXZzHRcdNHIWgRETGBiWLfp1NULh7stjQ2MkjK7fywQVTKcjVabIicuhSsujXuWfN4udPryPbjMsWTBudmERExggli34DmqFi8QR3Lt7IucccxmHlugiQiBzalCwgXDp1QDPUK5t3sKM7xrnHHDaKgYmIjA1KFgC97ZDo261m8fTaRgAW1FWPVlQiImOGkgVAZ0gMFO1KDM+81sgR40qoLdW0HiIiShYQmqBgZzNUXzzBonVNvHmGahUiIqBkEXRFySJqhnp5UyudvXFOVrIQEQGULILO5vA/qlk881rUXzFjz0F6IiKHIiUL2KPP4vn1LcyoKdY05CIikYwmCzM718xWmdkaM7tmkPXTzOwRM3vJzB43s8lJ664ws9XR3xWZjDM0QxkUVuDuvLChmeOnatJAEZF+GUsWZpYN3AicB8wBLjWzOQM2+w7wc3c/Frge+I9o3yrga8ACYD7wNTPLXOnd2QQF5ZCVzcamLho7ejlhWkXGnk5E5ECTyZrFfGCNu7/m7r3AHcA7B2wzB3gkuv1Y0vp3AA+5e5O7NwMPAedmLNKk0dvPbwj9F8dPUc1CRKRfJpPFJGBj0v36aFmyF4H3RrffDZSaWXWa+2JmV5nZYjNb3NDQsO+RJo3efmFDM0V52Rx12J6XVhUROVRlMlkMdvEHH3D/i8DpZvYCcDqwCYiluS/ufrO7z3P3ebW1tfseaWfjrs7tDS0cN7mC7Cxdu0JEpF8mk0U9MCXp/mRgc/IG7r7Z3d/j7scDX46Wtaaz737V1QxFVfTFE6zYsoO5U9VfISKSLJPJYhEw08zqzCwPuARYmLyBmdWYWX8M1wK3RLcfBN5uZpVRx/bbo2WZETVDNXX0Eks4kyt1nW0RkWQZSxbuHgOuJhTyK4A73X2ZmV1vZhdHm50BrDKzV4HxwDeifZuArxMSziLg+mjZ/tfXDX0dUFRJY3svANXFeRl5KhGRA1VOJh/c3R8AHhiw7KtJt+8G7k6x7y3sqmlkTlc0eruomsaOHgCqNRhPRGQ3GU0WB4SyCfDPW8CMpmWh8lKlmoWIyG6ULADyigDYrmYoEZFBaW6oJE0dPeRkGWUFuaMdiojImKJkkaSxvZfK4jyyNMZCRGQ3ShZJGjt61QQlIjIIJYskTR29VJcoWYiIDKRkkaSxvYfqYp02KyIykJJFksaOXp02KyIyCCWLSE8sTlt3jBo1Q4mI7EHJItLc0QdAlZqhRET2oGQR2d4epvpQM5SIyJ6ULCJNHWH0tpqhRET2pGQR6U8WqlmIiOxJySLS3wylGWdFRPakZBFp6uiN5oXS3IoiIgMpWUS6+uIU5mVjpnmhREQGUrKIJBJOtiYQFBEZlJJFJO5OtmoVIiKDSitZmNlvzOwCMztok0tcNQsRkZTSLfx/CHwQWG1m3zSzWRmMaVQoWYiIpJZWsnD3h939MuAEYB3wkJk9ZWZ/a2YHxWXl4gnIUjOUiMig0m5WMrNq4ErgY8ALwPcJyeOhjEQ2whKumoWISCppDSows3uAWcAvgIvcfUu06tdmtjhTwY0kNUOJiKSW7gi0/3H3Rwdb4e7z9mM8o0bJQkQktXSboWabWUX/HTOrNLNPZiimURFP6NRZEZFU0k0Wf+fuLf133L0Z+LvMhDQ64u5kqWYhIjKodJNFliXNg2Fm2cBBNT1rGME92lGIiIxN6fZZPAjcaWY3AQ58HPhjxqIaBRrBLSKSWrq/pf8JeBT4BPAp4BHgS8PtZGbnmtkqM1tjZtcMsn6qmT1mZi+Y2Utmdn60fLqZdZnZ0ujvpvRf0r5RB7eISGpp1SzcPUEYxf3DdB84aqq6ETgHqAcWmdlCd1+etNlXgDvd/YdmNgd4AJgerVvr7nPTfb6/lpKFiEhq6c4NNdPM7jaz5Wb2Wv/fMLvNB9a4+2vu3gvcAbxzwDYOlEW3y4HNexP8/hRPuEZwi4ikkG4z1P8SahUx4Ezg54QBekOZBGxMul8fLUt2HXC5mdUTahWfTlpXFzVPPWFmbxnsCczsKjNbbGaLGxoa0nwpg9MIbhGR1NJNFoXu/ghg7r7e3a8D3jbMPoOVvD7g/qXAre4+GTgf+EU0s+0WYKq7Hw/8A3C7mZUN2Bd3v9nd57n7vNra2jRfyuBiaoYSEUkp3bOhuqNCfLWZXQ1sAsYNs089MCXp/mT2bGb6KHAugLs/bWYFQI27bwN6ouVLzGwtcCSQsalFdPEjEZHU0q1ZfA4oAj4DnAhcDlwxzD6LgJlmVmdmecAlwMIB22wAzgIws9lAAdBgZrVRBzlmNgOYCQzXR/JX0amzIiKpDVuziArtD7j7PwLtwN+m88DuHotqIQ8C2cAt7r7MzK4HFrv7QuALwI/N7POEJqor3d3N7K3A9WYWA+LAx929aV9eYLriCTSCW0QkhWGThbvHzexEMzN3H9jnMNy+DxA6rpOXfTXp9nLg1EH2+w3wm715rr9WQnNDiYiklG6fxQvAfWZ2F9DRv9Dd78lIVKMglkioz0JEJIV0k0UV0MjuZ0A5cNAki4SjZCEikkK6I7jT6qc4kGkEt4hIauleKe9/2XOMBO7+kf0e0SjRCG4RkdTSbYa6P+l2AfBuRnFqjkwII7hHOwoRkbEp3Wao3c5MMrNfAQ9nJKJRohHcIiKp7etv6ZnA1P0ZyGjTCG4RkdTS7bNoY/c+izcI17g4aGgEt4hIauk2Q5VmOpDRFk/oGtwiIqmkez2Ld5tZedL9CjN7V+bCGnkawS0iklq6fRZfc/fW/jvu3gJ8LTMhjQ51cIuIpJZushhsu3RPuz0g6OJHIiKppZssFpvZd83scDObYWbfA5ZkMrCRphHcIiKppZssPg30Ar8G7gS6gE9lKqiR5u4kHI3gFhFJId2zoTqAazIcy6hJRCcFq2YhIjK4dM+GesjMKpLuV5rZg5kLa2TFEglAyUJEJJV0m6FqojOgAHD3Zoa/BvcBI8oVShYiIimkmywSZrZzeg8zm84gs9AeqOLRBQA1zkJEZHDpnv76ZeDPZvZEdP+twFWZCWnkxaNOC43gFhEZXLod3H80s3mEBLEUuI9wRtRBoT9ZZCtXiIgMKt2JBD8GfBaYTEgWJwNPs/tlVg9YO5OFahYiIoNKt8/is8BJwHp3PxM4HmjIWFQjLNHfZ5Glqx+JiAwm3dKx2927Acws391XAkdlLqyRtatmMcqBiIiMUel2cNdH4yx+CzxkZs0cRJdV3dnBrbOhREQGlW4H97ujm9eZ2WNAOfDHjEU1wtRnISIytL2eOdbdnxh+qwPLznEWShYiIoNSKz3hwkegZCEikoqSBRrBLSIynIwmCzM718xWmdkaM9tj1lozm2pmj5nZC2b2kpmdn7Tu2mi/VWb2jkzGqRHcIiJDy9jV7swsG7gROAeoBxaZ2UJ3X5602VeAO939h2Y2B3gAmB7dvgQ4GpgIPGxmR7p7PBOx7hrBrWQhIjKYTNYs5gNr3P01d+8F7gDeOWAbB8qi2+XsOh33ncAd7t7j7q8Da6LHywidDSUiMrRMJotJwMak+/XRsmTXAZebWT2hVvHpvdgXM7vKzBab2eKGhn0fUJ7Q2VAiIkPKZLIYrOQdOK35pcCt7j4ZOB/4hZllpbkv7n6zu89z93m1tbX7HGhc17MQERlSxvosCLWBKUn3J7PnqO+PAucCuPvTZlYA1KS5736jEdwiIkPLZM1iETDTzOrMLI/QYb1wwDYbgLMAzGw2UECYoHAhcImZ5ZtZHTATeC5TgarPQkRkaBmrWbh7zMyuBh4EsoFb3H2ZmV0PLHb3hcAXgB+b2ecJzUxXursDy8zsTmA5EAM+lakzoSB5BHemnkFE5MCWyWYo3P0BQsd18rKvJt1eDpyaYt9vAN/IZHz9do3gVrYQERmMSkc0zkJEZDhKFuxqhlLFQkRkcCoeUQe3iMhwlCxQM5SIyHCULNAIbhGR4ShZoGYoEZHhKFkAMY3gFhEZkpIFulKeiMhwlCzQNbhFRIajZIFqFiIiw1GyQKfOiogMR8mCpA5u1SxERAalZIHGWYiIDEfJgqQr5akZSkRkUEoWqGYhIjIcJQs0gltEZDhKFiSP4B7lQERExiglC8I4iywDU5+FiMiglCwII7jVBCUikpqSBaFmoWQhIpKakgWhg1unzYqIpKZkQejg1uhtEZHUlCwI4yzUDCUikpqSBWqGEhEZjpIFqlmIiAxHyYKoZqFkISKSkpIFUQe3mqFERFLKaLIws3PNbJWZrTGzawZZ/z0zWxr9vWpmLUnr4knrFmYyTo2zEBEZWk6mHtjMsoEbgXOAemCRmS109+X927j755O2/zRwfNJDdLn73EzFlyzumkRQRGQomaxZzAfWuPtr7t4L3AG8c4jtLwV+lcF4UlLNQkRkaJlMFpOAjUn366NlezCzaUAd8GjS4gIzW2xmz5jZu1Lsd1W0zeKGhoZ9DjSWSOjUWRGRIWQyWQxW+nqKbS8B7nb3eNKyqe4+D/gg8N9mdvgeD+Z+s7vPc/d5tbW1+xxoPKHrb4uIDCWTyaIemJJ0fzKwOcW2lzCgCcrdN0f/XwMeZ/f+jP0qjLPI1KOLiBz4MllELgJmmlmdmeUREsIeZzWZ2VFAJfB00rJKM8uPbtcApwLLB+67v2gEt4jI0DJ2NpS7x8zsauBBIBu4xd2Xmdn1wGJ3708clwJ3uHtyE9Vs4EdmliAktG8mn0W1v2kEt4jI0DKWLADc/QHggQHLvjrg/nWD7PcU8KZMxpYsFleyEBEZilrqCVfK0whuEZHUlCzQOAsRkeEoWaBrcIuIDEfJAtUsRESGo2RBmHVWp86KiKSmZEEYZ6ER3CIiqSlZEI2zUM1CRCQlJQuiEdzZShYiIqkoWQAJRzULEZEhKFkQTVGuPgsRkZSULIBEAo3gFhEZgpIFUZ+FjoSISEoqIukfwa1DISKSikpI+kdwj3YUIiJjl4pINIJbRGQ4ShaEmoVGcIuIpKZkQdRnoZqFiEhKShZoBLeIyHCULIiShWoWIiIpKVmgix+JiAznkE8W7o67RnCLiAzlkE8W8YQDqGYhIjIEJQtXshARGY6ShWoWIiLDUrLoTxbqsxARSemQTxaJRPivEdwiIqkd8sliZ5+FcoWISEqHfLLIyTYueNMEptcUj3YoIiJjVkaThZmda2arzGyNmV0zyPrvmdnS6O9VM2tJWneFma2O/q7IVIxlBbnceNkJnHHUuEw9hYjIAS8nUw9sZtnAjcA5QD2wyMwWuvvy/m3c/fNJ238aOD66XQV8DZgHOLAk2rc5U/GKiEhqmaxZzAfWuPtr7t4L3AG8c4jtLwV+Fd1+B/CQuzdFCeIh4NwMxioiIkPIZLKYBGxMul8fLduDmU0D6oBH92ZfM7vKzBab2eKGhob9ErSIiOwpk8lisPOLPMW2lwB3u3t8b/Z195vdfZ67z6utrd3HMEVEZDiZTBb1wJSk+5OBzSm2vYRdTVB7u6+IiGRYJpPFImCmmdWZWR4hISwcuJGZHQVUAk8nLX4QeLuZVZpZJfD2aJmIiIyCjJ0N5e4xM7uaUMhnA7e4+zIzux5Y7O79ieNS4A5396R9m8zs64SEA3C9uzdlKlYRERmaJZXRB7R58+b54sWLRzsMEZEDipktcfd5w253sCQLM2sA1v8VD1EDbN9P4exPimvvjNW4YOzGprj2zliNC/YttmnuPuwZQgdNsvhrmdnidLLrSFNce2esxgVjNzbFtXfGalyQ2dgO+bmhRERkeEoWIiIyLCWLXW4e7QBSUFx7Z6zGBWM3NsW1d8ZqXJDB2NRnISIiw1LNQkREhqVkISIiwzrkk8VwF2gawTimmNljZrbCzJaZ2Wej5deZ2aaki0SdP0rxrTOzl6MYFkfLqszsoegCVQ9FU7OMZExHJR2XpWa2w8w+NxrHzMxuMbNtZvZK0rJBj48FN0SfuZfM7IQRjuvbZrYyeu57zawiWj7dzLqSjttNmYpriNhSvndmdm10zFaZ2TtGOK5fJ8W0zsyWRstH7JgNUUaMzOfM3Q/ZP8I0JGuBGUAe8CIwZ5RimQCcEN0uBV4F5gDXAV8cA8dqHVAzYNm3gGui29cA/znK7+UbwLTROGbAW4ETgFeGOz7A+cAfCLMrnww8O8JxvR3IiW7/Z1Jc05O3G6VjNuh7F30XXgTyCZczWAtkj1RcA9b/F/DVkT5mQ5QRI/I5O9RrFnt7gaaMcfct7v58dLsNWEGK63+MIe8Efhbd/hnwrlGM5Sxgrbv/NaP495m7PwkMnL8s1fF5J/BzD54BKsxswkjF5e7/5+6x6O4zhFmdR1yKY5bKOwlzyPW4++vAGsL3d0TjMjMDPsDus2SPiCHKiBH5nB3qySLtCzSNJDObTrjE7LPRoqujauQtI93Uk8SB/zOzJWZ2VbRsvLtvgfBBBkbzQuYDp7kfC8cs1fEZS5+7jxB+ffarM7MXzOwJM3vLKMU02Hs3Vo7ZW4Ct7r46admIH7MBZcSIfM4O9WSxNxdoGhFmVgL8Bvicu+8AfggcDswFthCqwKPhVHc/ATgP+JSZvXWU4tiDhSnwLwbuihaNlWOWypj43JnZl4EYcFu0aAsw1d2PB/4BuN3MykY4rFTv3Zg4Zux++WcYhWM2SBmRctNBlu3zMTvUk8WYusiSmeUSPgS3ufs9AO6+1d3j7p4AfkyGqt7DcffN0f9twL1RHFv7q7XR/22jERshgT3v7lujGMfEMSP18Rn1z52ZXQFcCFzmUQN31MTTGN1eQugXOHIk4xrivRsLxywHeA/w6/5lI33MBisjGKHP2aGeLNK6QNNIiNpCfwqscPfvJi1PbmN8N/DKwH1HILZiMyvtv03oIH2FcKyuiDa7ArhvpGOL7PZrbywcs0iq47MQ+HB0tsrJQGt/M8JIMLNzgX8CLnb3zqTltWaWHd2eAcwEXhupuKLnTfXeLQQuMbN8M6uLYntuJGMDzgZWunt9/4KRPGapyghG6nM2Er34Y/mPcMbAq4RfBF8exThOI1QRXwKWRn/nA78AXo6WLwQmjEJsMwhnorwILOs/TkA18AiwOvpfNQqxFQGNQHnSshE/ZoRktQXoI/yi+2iq40NoHrgx+sy9DMwb4bjWENqy+z9nN0Xbvjd6f18EngcuGoVjlvK9A74cHbNVwHkjGVe0/Fbg4wO2HbFjNkQZMSKfM033ISIiwzrUm6FERCQNShYiIjIsJQsRERmWkoWIiAxLyUJERIalZCEyBpjZGWZ2/2jHIZKKkoWIiAxLyUJkL5jZ5Wb2XHTtgh+ZWbaZtZvZf5nZ82b2iJnVRtvONbNnbNd1I/qvM3CEmT1sZi9G+xwePXyJmd1t4VoTt0UjdkXGBCULkTSZ2WzgbwiTKs4F4sBlQDFhbqoTgCeAr0W7/Bz4J3c/ljCCtn/5bcCN7n4ccAphtDCEWUQ/R7hGwQzg1Iy/KJE05Yx2ACIHkLOAE4FF0Y/+QsKkbQl2TS73S+AeMysHKtz9iWj5z4C7ojm2Jrn7vQDu3g0QPd5zHs07ZOFKbNOBP2f+ZYkMT8lCJH0G/Mzdr91todm/DNhuqDl0hmpa6km6HUffTxlD1Awlkr5HgPeZ2TjYee3jaYTv0fuibT4I/NndW4HmpIvhfAh4wsP1B+rN7F3RY+SbWdGIvgqRfaBfLiJpcvflZvYVwhUDswizkn4K6ACONrMlQCuhXwPCdNE3RcngNeBvo+UfAn5kZtdHj/H+EXwZIvtEs86K/JXMrN3dS0Y7DpFMUjOUiIgMSzULEREZlmoWIiIyLCULEREZlpKFiIgMS8lCRESGpWQhIiLD+v9qxpXC05hQigAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# summarize history for accuracy\n",
    "plt.plot(history.history['acc'])\n",
    "plt.plot(history.history['val_acc'])\n",
    "plt.title('model accuracy')\n",
    "plt.ylabel('accuracy')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'evaluate'], loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XmcXFWd9/HPr5be9yWdkD0hISQhewKyCA4CISC7gCMOruioqI/b4CMq4/LoqDM6jCjCgLggiyAhozgqIKiEAAkkkISE7OnO2ul9X6rO88ep7nQn1ZXOUt2dru/79epXV99769avblWf3z3n3HOuOecQEREBCAx2ACIiMnQoKYiISDclBRER6aakICIi3ZQURESkm5KCiIh0U1IQ6Scze8DMvtnPbbeb2TuPdz8iA01JQUREuikpiIhINyUFGVZizTZfMLPXzazJzO4zszIz+4OZNZjZ02ZW2GP7K8xsnZnVmtlzZnZ6j3VzzezV2PMeATIOea3LzWx17LnLzWzWMcb8ETPbbGbVZrbMzE6JLTcz+4GZ7Tezuth7mhlbt8TM1sdi22Vmnz+mAyZyCCUFGY6uBS4CpgLvAv4A/F+gBP+d/xSAmU0FHgI+A5QCTwH/Y2ZpZpYGLAV+CRQBv4ntl9hz5wH3Ax8FioGfAsvMLP1oAjWzfwC+DVwPjAJ2AA/HVl8MvD32PgqAG4Cq2Lr7gI8653KBmcCzR/O6In1RUpDh6L+cc/ucc7uAvwEvOedec861AU8Ac2Pb3QD83jn3Z+dcB/B9IBM4GzgLCAM/dM51OOceA17p8RofAX7qnHvJORdxzv0caIs972i8F7jfOfdqLL4vAW8zswlAB5ALTAPMOfemc25P7HkdwHQzy3PO1TjnXj3K1xWJS0lBhqN9PR63xPk7J/b4FPyZOQDOuShQDoyOrdvles8YuaPH4/HA52JNR7VmVguMjT3vaBwaQyO+NjDaOfcs8CPgLmCfmd1jZnmxTa8FlgA7zOx5M3vbUb6uSFxKCpLKduMLd8C34eML9l3AHmB0bFmXcT0elwPfcs4V9PjJcs49dJwxZOObo3YBOOfudM7NB2bgm5G+EFv+inPuSmAEvpnr0aN8XZG4lBQklT0KXGZmF5pZGPgcvgloOfAi0Al8ysxCZnYNsKjHc+8FPmZmZ8Y6hLPN7DIzyz3KGH4NfMDM5sT6I/4fvrlru5ktjO0/DDQBrUAk1ufxXjPLjzV71QOR4zgOIt2UFCRlOec2AjcB/wUcwHdKv8s51+6caweuAd4P1OD7H37b47kr8f0KP4qt3xzb9mhjeAb4CvA4vnYyGbgxtjoPn3xq8E1MVfh+D4D3AdvNrB74WOx9iBw30012RESki2oKIiLSTUlBRES6KSmIiEi3pCUFM7s/Njx/bR/rzczujA3vfz02QlRERAZRKIn7fgB/ZcYv+lh/KTAl9nMm8JPY74RKSkrchAkTTkyEIiIpYtWqVQecc6VH2i5pScE599fYUP2+XAn8IjZidIWZFZjZqB7D+OOaMGECK1euPIGRiogMf2a248hbDW6fwmj8qNAuFbFlhzGzW8xspZmtrKysHJDgRERS0WAmBYuzLO6gCefcPc65Bc65BaWlR6z9iIjIMRrMpFCBn2emyxj8PDAiIjJIktnRfCTLgE+a2cP4Dua6I/Un9KWjo4OKigpaW1tPaICpICMjgzFjxhAOhwc7FBEZApKWFMzsIeACoMTMKoCv4eenxzl3N/6GJkvwc8Y0Ax841teqqKggNzeXCRMm0HtSS0nEOUdVVRUVFRVMnDhxsMMRkSEgmVcfvecI6x3wiRPxWq2trUoIx8DMKC4uRp33ItJl2IxoVkI4NjpuItLTsEkKR9LU1sneulaimhVWRKRPKZMUmts72d/QSjJyQm1tLT/+8Y+P+nlLliyhtrb2xAckInKMUiYpHBwWceKzQl9JIRJJfDOsp556ioKCghMej4jIsRrMS1IHVHdKSEJN4bbbbmPLli3MmTOHcDhMTk4Oo0aNYvXq1axfv56rrrqK8vJyWltb+fSnP80tt9wCHJyyo7GxkUsvvZRzzz2X5cuXM3r0aJ588kkyMzNPfLAiIgkMu6Twr/+zjvW76w9b3hGJ0t4ZJSs9FHcodSLTT8nja++a0ef673znO6xdu5bVq1fz3HPPcdlll7F27druyzzvv/9+ioqKaGlpYeHChVx77bUUFxf32semTZt46KGHuPfee7n++ut5/PHHuekm3WFRRAbWsEsKfRnIa2wWLVrU67r/O++8kyeeeAKA8vJyNm3adFhSmDhxInPmzAFg/vz5bN++fcDiFRHpMuySQl9n9FVNbeyqaWHayDzSQsntSsnOzu5+/Nxzz/H000/z4osvkpWVxQUXXBB35HV6enr342AwSEtLS1JjFBGJJ2U6mi2JHc25ubk0NDTEXVdXV0dhYSFZWVls2LCBFStWnPDXFxE5UYZdTeFIkjFKobi4mHPOOYeZM2eSmZlJWVlZ97rFixdz9913M2vWLE477TTOOuusJEQgInJimDvJBnMtWLDAHXqTnTfffJPTTz894fNqmtspr27mtLJc0sPBZIZ40unP8RORk5uZrXLOLTjSdinUfOSdXClQRGRgpUxSEBGRI0uZpNA179tJ1lomIjKgUiYpqAFJROTIUiYpKCWIiBxZyiQF1HwkInJEKZMUTpZbyUyYMIEDBw4c03OXLl3K+vXrT3BEIpJKUi4pDOeKgpKCiByvlEkKA9F+9Ktf/YpFixYxZ84cPvrRj3LXXXfxxS9+sXv9Aw88wK233grAVVddxfz585kxYwb33HPPYfvavn07M2fO7P77+9//PnfccQcA9957LwsXLmT27Nlce+21NDc3s3z5cpYtW8YXvvAF5syZw5YtW9iyZQuLFy9m/vz5nHfeeWzYsCFp711EhofhN83FH26DvW8ctjjTOSa1R8gIByBwlLlw5Blw6XcSbvLmm2/yyCOP8MILLxAOh/n4xz9OTk4Ov/3tb/nud78LwCOPPMKXv/xloH/Tafflmmuu4SMf+QgAt99+O/fddx+33norV1xxBZdffjnXXXcdABdeeCF33303U6ZM4aWXXuLjH/84zz777NG9dxFJKcMvKQySZ555hlWrVrFw4UIAWlpaGDFiBJMmTWLFihVMmTKFjRs3cs455wD9m067L2vXruX222+ntraWxsZGLrnkksO2aWxsZPny5bz73e/uXtbW1na8b1NEhrnhlxT6OKNva+9k6/5GJhRnk5cZPuEv65zj5ptv5tvf/nav5ffddx+PPvoo06ZN4+qrr8bM+jWddigUIhqNdv/dc/373/9+li5dyuzZs3nggQd47rnnDosnGo1SUFDA6tWrT+wbFZFhLYX6FJLrwgsv5LHHHmP//v0AVFdXs2PHDq655hqWLl3KQw89xA033AD0bzrtsrIy9u/fT1VVFW1tbfzud7/rXtfQ0MCoUaPo6OjgwQcf7F7ecwrvvLw8Jk6cyG9+8xvAJ601a9Yk7f2LyPCQMkkh2VcfTZ8+nW9+85tcfPHFzJo1i4suuog9e/ZQWFjI9OnT2bFjB4sWLQL8dNqdnZ3MmjWLr3zlK3Gn0w6Hw3z1q1/lzDPP5PLLL2fatGnd677xjW9w5plnctFFF/VafuONN/K9732PuXPnsmXLFh588EHuu+8+Zs+ezYwZM3jyySeT9O5FZLhImamzWzoibNrXwPiiLPKz0pIZ4klHU2eLDH+aOvsQqTBOQUTkeKVMUhARkSMbNknhSM1gmjo7vpOt+VBEkmtYJIWMjAyqqqoSFnBqPjqcc46qqioyMjIGOxQRGSKGxTiFMWPGUFFRQWVlZZ/bRKKOfXWttB8Isy99WLztEyIjI4MxY8YMdhgiMkQMi9IxHA4zceLEhNvsq2/l8l8+w7eunsl754wfoMhERE4uw6L5qD8CsU6FaFQNSCIifUmZpBAM+KQQUVIQEelT6iSFWE0hopwgItKnpCYFM1tsZhvNbLOZ3RZn/Tgz+4uZvWZmr5vZkmTFEgyq+UhE5EiSlhTMLAjcBVwKTAfeY2bTD9nsduBR59xc4Ebgx8mK52BNQUlBRKQvyawpLAI2O+e2OufagYeBKw/ZxgF5scf5wO5kBdN1Xx31KYiI9C2ZSWE0UN7j74rYsp7uAG4yswrgKeDWeDsys1vMbKWZrUw0FiGR7pqCkoKISJ+SmRQszrJDS+T3AA8458YAS4BfmtlhMTnn7nHOLXDOLSgtLT2mYHT1kYjIkSUzKVQAY3v8PYbDm4c+BDwK4Jx7EcgASpIRjJlhBlH1KYiI9CmZSeEVYIqZTTSzNHxH8rJDttkJXAhgZqfjk8KxtQ/1Q9BMNQURkQSSlhScc53AJ4E/Am/irzJaZ2ZfN7MrYpt9DviIma0BHgLe75I4bWcgYLr6SEQkgaTOfeScewrfgdxz2Vd7PF4PnJPMGHoKBUzjFEREEkiZEc3Q1Xw02FGIiAxdKZUUAgEjElVWEBHpS0olhaD6FEREEkqppBBQ85GISEIplRSCAU2IJyKSSGolBVPzkYhIIqmVFIK6JFVEJJHUSgqqKYiIJJRSScFfkqqkICLSl5RKCpr7SEQksdRKCqopiIgklFJJIWCmqbNFRBJIqaSgmoKISGIplRT81NmDHYWIyNCVUklBU2eLiCSWUklBVx+JiCSWUkkhEECD10REEkippKCOZhGRxFIqKQTUfCQiklBKJYVgQOMUREQSSa2koJqCiEhCqZUU1KcgIpJQyiUFNR+JiPQtpZKCps4WEUkspZJC0AzlBBGRvqVWUggYndHoYIchIjJkpVRSCJihnCAi0reUSgrBAOpTEBFJIMWSgmnuIxGRBFIuKWjqbBGRvqVWUjDVFEREEkmppKBxCiIiiaVUUgiamo9ERBJJraQQMDqVFERE+pRSSSGguY9ERBJKalIws8VmttHMNpvZbX1sc72ZrTezdWb262TGo6mzRUQSCyVrx2YWBO4CLgIqgFfMbJlzbn2PbaYAXwLOcc7VmNmIZMUDXTUFcM5hZsl8KRGRk1IyawqLgM3Oua3OuXbgYeDKQ7b5CHCXc64GwDm3P4nxEAr4RKDKgohIfMlMCqOB8h5/V8SW9TQVmGpmL5jZCjNbHG9HZnaLma00s5WVlZXHHFAwlhTUhCQiEl8yk0K89plDS+MQMAW4AHgP8N9mVnDYk5y7xzm3wDm3oLS09JgDClhXTUFJQUQknmQmhQpgbI+/xwC742zzpHOuwzm3DdiITxJJEYy9W9UURETiS2ZSeAWYYmYTzSwNuBFYdsg2S4F3AJhZCb45aWuyAuqqKWisgohIfElLCs65TuCTwB+BN4FHnXPrzOzrZnZFbLM/AlVmth74C/AF51xVsmLq6lPQqGYRkfiSdkkqgHPuKeCpQ5Z9tcdjB3w29pN03R3N6lMQEYkrpUY0q6YgIpJYaiUFU01BRCSRlEoKAY1TEBFJKKWSQldNIRod5EBERIao1EoK6mgWEUkopZLCweYjVRVEROJJqaTQ3dGsnCAiEle/koKZfdrM8sy7z8xeNbOLkx3ciaZpLkREEutvTeGDzrl64GKgFPgA8J2kRZUkwYB/u5oQT0Qkvv4mha4ZT5cAP3POrSH+LKhDmmoKIiKJ9TcprDKzP+GTwh/NLBc46VrmAxq8JiKSUH/nPvoQMAfY6pxrNrMifBPSSUXTXIiIJNbfmsLbgI3OuVozuwm4HahLXljJcfDqIyUFEZF4+psUfgI0m9ls4IvADuAXSYsqSTTNhYhIYv1NCp2xaa6vBP7TOfefQG7ywkoOjWgWEUmsv30KDWb2JeB9wHlmFgTCyQsrOYKqKYiIJNTfmsINQBt+vMJeYDTwvaRFlSTdE+KppiAiEle/kkIsETwI5JvZ5UCrc+6k61M4WFMY5EBERIao/k5zcT3wMvBu4HrgJTO7LpmBJUNAVx+JiCTU3z6FLwMLnXP7AcysFHgaeCxZgSVD9zgFNR+JiMTV3z6FQFdCiKk6iucOGZrmQkQksf7WFP7XzP4IPBT7+wbgqeSElDxqPhIRSaxfScE59wUzuxY4Bz8R3j3OuSeSGlkS6JJUEZHE+ltTwDn3OPB4EmNJOg1eExFJLGFSMLMGIF4JaoBzzuUlJaok0YR4IiKJJUwKzrmTbiqLRIKaOltEJKGT7gqi4xFQTUFEJKGUSgqaOltEJLGUSgrdU2crJ4iIxJVSSeHgJama/EhEJJ6USgohTYgnIpJQSiWFgKbOFhFJKKWSgkY0i4gkllJJIZYTlBRERPqQUknBzAiYmo9ERPqS1KRgZovNbKOZbTaz2xJsd52ZOTNbkMx4wDchqaYgIhJf0pKCmQWBu4BLgenAe8xsepztcoFPAS8lK5aeAmaa5kJEpA/JrCksAjY757Y659qBh4Er42z3DeC7QGsSY+kWDBgRjV4TEYkrmUlhNFDe4++K2LJuZjYXGOuc+12iHZnZLWa20sxWVlZWHldQwYBqCiIifUlmUrA4y7pLYzMLAD8APnekHTnn7nHOLXDOLSgtLT2uoNJDQVo7NHpNRCSeZCaFCmBsj7/HALt7/J0LzASeM7PtwFnAsmR3Nhdnp1Hd1JbMlxAROWklMym8Akwxs4lmlgbcCCzrWumcq3POlTjnJjjnJgArgCuccyuTGBPFOWlUNbYn8yVERE5aSUsKzrlO4JPAH4E3gUedc+vM7OtmdkWyXvdIinPSqWpSUhARiaff92g+Fs65p4CnDln21T62vSCZsXQpyUnjQKOaj0RE4kmpEc0AJTnpNLR20toRGexQRESGnJRLCsXZaQBUqwlJROQwqZcUctIB1NksIhJHCiYFX1M4oMtSRUQOk3JJoSRbNQURkb6kXFLoqilU6QokEZHDpFxSyEoLkhEOaKyCiEgcqZMU1jwMPz0fi0YoyUnnQINqCiIih0qdpNBaB3tWQ2stxTnpHFBNQUTkMKmTFDKL/O/makqy09SnICISR+okhaxC/7ulWpPiiYj0IXWSQo+agp8Urw2nm+2IiPSSOkkhK5YUWqopzk6jI+Koa+kY3JhERIaY1EkKPWoKk0qzAdiwt2EQAxIRGXpSJymk50IgBC3VzB3r+xdW7agZ5KBERIaW1EkKZr620FxNYXYak0uzeVVJQUSkl9RJCuD7FVqqAZg/vpBVO2vU2Swi0kNqJYXMImj2tYP54wupbe5g64GmQQ5KRGToSLGkUNirpgDqVxAR6Sm1kkJWITT7pDCpJIeCrDCvbKse5KBERIaO1EoKmbE+BecIBIxzTi3hLxsriUbVryAiAqmWFLKKINIO7b4f4aLTyzjQ2Mbru+oGOTARkaEhtZJC1wC2Ft+PcMFppQQDxtPr9w1iUCIiQ0dqJYUeU10AFGSlsWB8IU+/qaQgIgKplhR6THXR5Z2nl7FhbwM7q5oHKSgRkaEjtZLCITUFgMUzRwLw5OpdgxGRiMiQklpJIU5NYWxRFosmFvHE6l0a3SwiKS/FkkLXjXZ6D1i7Zu5otlY28XqFrkISkdSWWkkhlAbpedDYu2P50jNGkRYK8NiqikEKTERkaEitpABwyhzY+VKvRfmZYa6cfQqPrCynvFodziKSulIvKUx8O+x7A5qqei3+7MVTMeD7f9o4OHGJiAwBKZgULvC/t/+11+JR+Zl8+LyJPLl6N8s3Hxj4uEREhoDUSwqnzIW0XNj218NWffyCUzl1RA63PvQae+paBiE4EZHBlXpJIRiC8Wf3TgodPgFkp4e4+6Z5tHZE+PiDr9LeGR2kIEVEBkfqJQWASedD1WY4sBne/B/4znio3grAqSNy+e51s3ltZy3f+v36QQ5URGRgJTUpmNliM9toZpvN7LY46z9rZuvN7HUze8bMxicznm4zr4VACFbeDy/8J0TaYMNT3asvmzWKD587kZ+/uINfv7RzQEISERkKkpYUzCwI3AVcCkwH3mNm0w/Z7DVggXNuFvAY8N1kxdNL7kiYfiWsvA8qXgEM3vrfXpv8y6XTuOC0Um5f+gZ/XLd3QMISERlsyawpLAI2O+e2OufagYeBK3tu4Jz7i3Oua2DACmBMEuM5JLpboLMV0nJg4Ydh54vQUtu9OhwM8OP3zuOMMQX8869Wceczm4joZjwiciI0HYB1SyHS2Xu5c3Bgk//d2QZ//2F30/ZACSVx36OB8h5/VwBnJtj+Q8Af4q0ws1uAWwDGjRt3YqIbeyZMuQTGLPBjF165F7Y8CzOv6d4kKy3Erz98JrcvXct//PktXtlezQ9umENJTvqJiUFEhodIJ7TV+0k3nYO6cmhvhtyyg9PrOAcVK+HNJ2Hlz6C90ZdBi26B3a9CzghY+1vY9jxc8m1/0vrMv8LffwCX/bsvp3JGJP2tWLImgTOzdwOXOOc+HPv7fcAi59ytcba9CfgkcL5zri3RfhcsWOBWrlx5YoONRuB7k2Hi+XD9zw9b7ZzjkVfK+eqydRRmhbnzxrmcOan4xMYgIl7FSnj+3+CUeXDe53zTbvFkKJsRf/tIB1RuhIx8KBgLbY2wZw1kl0DxFAjEaRCpLYe9b8Cp7/TT3xxJ1RZ4/VEY/zaY8Ha/z12roH43nLYEHnw37FwBNz4Ir/w3bPjdwecWTYYrfwSv/QpWPwgWhNMvh1Gz4dlvgutxlWN6HhROgANvgQVgzEJfq9i/zq9f/G9w1sf6fSh7MrNVzrkFR9oumTWFCmBsj7/HALsP3cjM3gl8mX4khKQJBGHezfDCD2H3aj8VRu8YuXHROGaNKeATv36V99y7gm9fcwY3LDxBtRaRgdBc7Qug7JIjb7vjRRgx7eBZblujPxPOO8WfRNXvgoLY9z8a8f9DteWw9jEIhCGnzBfkxadCRl7s7LkCqjb5Qm7yhX5/T3wUJpwHp13qz4r3rvVT26flwqY/wYof++3S8+H9v4NRs2D73+Hle/3+GvdBw16IdvhZkD/4R3jsg37WAoC80TD5HRDK8Jeij5wFb/wGXrgTOlt8gV02Haq3Qc12/57OuA7Gne0L9lU/92f7NdshGmvqKZrk97X6IXARKJsJ+9ZC9gj45VV+m7d/AUqn+eO08n742aV++Xmfg7NvPXhcJ18Ijfv9/pr2+/fQ2Qp3nekvlX/Xf/pjXrHSJ7qJ552Ib0JCyawphIC3gAuBXcArwD8659b12GYuvoN5sXNuU3/2m5SaAkBrHdw513+Ql3zLt+tVboQFH4T80d2bNbR28Ilfv8bfNlVyx7tm8N4zxxEKpuaVvXIUWut8QVt8KpSceuTtW2qhZhtkFR8sfA+1e7Uv9AJB+NPtvkl0/s2Hb9fW4AvB5f/lC88ZV8MZ1/tCe/9635TRWgsLP+IHd778U1+Q5Y6CMz/q4976nL+/+ds/D+Uv+XE+i27x+379EX9GXrPNb3OocLb/3dF0cFlmIQTToLXeF87gC9XTL4eSqTD3Jtjwe39mPfM6eP67PsacEb6NPacMRkz3F43kjoTCif4YdLb5GC77vk8E65/0x6mz1SeXLqdf4S82WX6nL3yLJkHBeNiz2r8/AMwfq0i7P85v+4Rft/JnsHM5zLjGfzYv/BDmvx8u+L/wP5/yz5l9Y4/Psgb+/DXfVD3vn4782QPsfs1/ZyZd0L/t+6G/NYWkJYVYEEuAHwJB4H7n3LfM7OvASufcMjN7GjgD2BN7yk7n3BWJ9pm0pAD+7OOpz/deljcabvilr8qaQUcrnc9/j9u2nMFj28KMzMvgxkVjuXHhOEbmZyQnLhlczkH5yxAMw+h5vde1Nfiz28xCyC71yyo3+MI/t8yvf+YbvknBRfxZ9Ns/7wvb1jpf8AVCMOJ0GH+uLyD/+r1YAd7pC86Lv+ULsIx8CGf49utnv+ELo1AmZBZAQ+xf6B++Auf+H9j7un/NQNi/RtN+X4jllPmCtmcBmTvK77tyw8FlCz7kz8gPbPSF5bTL/Bnt2sd8YTv1El/gBsK+AGzYA/lj4NzP+njq9/ixQF01Axf1Be+I0/1znv6aP/u+6bf+jL98Re8z6ENVbYG//YdPLKfMg0UfgXBm7202/B4euQku+BKc/8Xe66IRn9iqt/rY+0q0AA37YNdK/75Hzoy/TXszpGX5xzU7IH9s/GaqIWRIJIVkSGpScC52FVKNr7Jh8OvrfRU1ZyQs+S7sWA4v3Y0bvYA/v+0X/OrlXfz1rUoywgG+eMk0rp0/hpz0EMGAJSfG4SLS6UeXd2mtg8q3YOzCE7P/aMR/jlnFPpn3xTl/1lu5wRfOc/7RFzYdrdC4Fzb+L6z62cECc8bVvmCJRv2Z4/Pf8YXboSzgz3hrd/oz0fk3+4L9tV/B2schmO7P2nu2J+eUQXOVTwaz/9E3qbz2K9j0R78+lAETzvVJqHGvP+uMdPqzyst/4JPA2sf8WXNdOYSzfBwjz4B3/iuMme/309kG2/7m91EwHsad5d/7zhXQsBtyT/Ft551tvrAvGO+PoXN+sGfxqb7JpfwVX4j3p+bT12cUCB7bc/vSUtN3YklxSgonSuN+2PiUb1vc/RrgYPR838l09q1QfCoN219l0+ZNrG7I44nIuWwOTeHiGWX809smMH98nC/o9hd8YXLuZ317Z2d7/zq7jtauVf6M8+Jv+rO4vjRX+3+kRIUnwP4N/izzvM/5s8F42hr98cJ8R1rp1IPrnPNnja/+3J/1TX4HXPR1X4Atu9UXoBd+1TcXbHnWH+/sUr+fLc9AVolv733pbj8aPaswdsaW7Qu+hj2+ql80GV77pS+8w9kw4yrfLLj8Tr9t2UyfhMKZ0FQZ+1xjymZC6Wn+ckEX8ctOmQcLPuDPVpf/ly8Q2xp8MsgeAYu/7bdrOuAL9JIpfvzLnjU+ljOu800HXfasgTUP+1jm/ZMvvLc8Cxv/4DsZpy6GcbEL9aJR2Ph7f2JS+RZsftq/l/k3+zPenqJRWL8UXvopFE30camAlBglhROto8UXXPV74KbH4dH3+Y4wgPQ8XN4pRKt3EIi0siH3bXQ0VFLkagiF08k4/zMUnP1Bf1a09nF48hO+8HLOV2PryuHqe2DWu/3+AyF/xrPteV/lDYRgyff98/e+4Qv4aMQnksIJveOsq4BX7vPPefH75EVuAAASG0lEQVRH0NHsO/Ku+okviEbP9TPFBkP+9ZffCU/fAbNuhIu/4duSp1zk25artvgCs63Bd9A98l5f8JZOg4u+ATmlUHq6b+Ko2uQ75lb8xBe04GM49//4Nt+dK2DTn327MPiYyl/2o8kB8sf5gn3j7w++l4x8/9ou6seTtDcBzr/eKfP8vtKy/dl1zXa/vQV9R2XZGf54Vm32V410tvrXzMj3V3ZkFvmmmmjEj1OZdrlPokv/2Rfsc2/ybdajZvvOzS6RDt+M1N4Mrz/sLyns0eckMlQpKSRbe5NvUsguhbwxvj2xtR7+8v9gw++JFIxjU2s+rXvfYo5toi2UQyinhGDtdhi9wF/6uuInvjCr3wX71sO0JbDuid6vkz8udlVCod9/z846gLnv8wVd1WY4493+Erea7YDzBdqMq32hHwgdvHoiewRMvRj2rfOF/sgzfLIJhH2TRjDNN1Nsebb3a6XnwUX/Cn++A9pity4NhH0TRVfhPuE836abVQR/+3d/pQf4Zpwpl/iYRs+DsYt8Z/6WZ/0xPPVCX/C/fI9PmKct8c0UzdX+crwxC/2Z/8Y/+LEkRZN6x9bW6BME+KtRcsoOtvE27vfJbOSsI9eGOtt8ouhqLxYZJpQUhoidB5p49KH7OGXfXxhn+/h98B3UT76CL11+BmMKYwVP0wG49x3+muczP+bbcEPpfrBK0UR/BvvEx3yBOusGX8gFw74ZYsVPfKGcVezbiNNy4H1L/eWEaTl+/7/7jH/OO+/wZ8mvP+oL49JpvpNw4Yf9frY9D2d/ytcetv0NzvpnXwBbwDcJTTzfN4M0V/sCvXGfTypdTSaT/+HwZqrKjT6O3FFDviNOZDhTUhhiNu5t4M/r97KrtpWlr+2irTNCdlqItFCArPQgF4yGy0/P58x5849uxw37fDNSWo4fDVk61Tf9HA/nfDNJMvo5RGRQKCkMYbtqW3jk5Z00tHXS3hmltrmDF7dWUd3UzjVzR3PdgjFMLculODsNO1Jzh4hIPygpnGTaO6P86NlN3PXclu6J99JDAeaOK+C8KaWcN6WEmafkE9ClriJyDJQUTlLVTe28sauOrZWNlFe3sGJrFev3+IFG44qyWDihiG0HGlk4sYh/Pn8yaaEAmeGgahQikpCSwjBS2dDG829V8tiqct7a18jYoixer6il66MryUnn9FG57KlrZd64Am6/fDp5GeHBDVpEhhQlhWFu3e46nnlzP6GgsXFvA1sqGynJSedvmw5QmBVmyohcHI72zihXzx3NkjNGkZsRJi2kK4BEUpGSQop6bWcN//33beyra8UMmtoi3c1PwYCxeOZILp5e5u/hEXU0t3dS29zB+OIsZo8pYHxxlpqiRIYhJQUB/L0gXtlew7rddeysbuaxlRU0tHX2uX1BVpjZYwqYPbaAOWPzGZWfSVooQHtnlJKcdEpydEWUyMlISUHiamzrZHdtC6GAEQoEyEgLkJcRZktlI2vK61hTXsuailre2tdAvLuPFmaFmVKWy9SyHKaW5TJlRC75mWEiUUd+5sHmqbK8dCUPkSFESUGOS1NbJ+t211PV2EZ7JEo4GGBvXSub9jfw1r5G3trXQENr3zWOcUVZTC7Npq6lg4UTilg0sYhI1NEZdRRmpXHmxCJdXisygIbCndfkJJadHmLRxKI+1zvn2Fffxlv7GmhujxAMGLXN7XREHG2dEZ5/q5L9DW1kpQW57+/b+Olfe998fHRBJkXZaXREfLNUdVM7Te2d/MO0EUwqye7ebmJJDosmFpEWChCJOvY3tDIqP/PQcETkBFFSkGNiZozMz+jzxkIfOGdi9+Oapna2VTWRFgx0Xy21bPVuos4RCgaobGhjRF46AcvgwZd20t4Z7bWvjHCAU0fksKe2laqmdi6cNoK54wpYuaOGhROKGFeUxca9DZw2MpcFEwopzUnX3fBEjpGaj2RIaWmP0NTum6WizvFGRR0vbK5i0/4GCrPSGFuUyQMvbKepPcL44ix2VDUftg8zKM5OpywvnRG56YzIzSArPUhtcwdF2WmMLcwkGDBG5mdy6ogcMsIBwsEALe0RXt1ZQ0FWGudMLiYUDOCco6a5g8KssPpI5KSmPgUZtmqb22nrjFKWl0F5dTO1zR1MKcth/Z561u+uZ39DG5UNreyrb2N/7HdzWycFWWlUNbXR2hE94msUZacxuTSbvfWtlFe3UJgV5rSRvlN9a2UTZnDlnNGcPbmYUCDAXzdVUtXYTsBgZH4GM07JZ+64AjLCB+8s1vW/puQig0FJQSSOaNRR09xOxDnKq5vZfqCZ9kiUzkiUQMCYPaaAipoW/rx+H+U1zeRlhJk3voBtlU1sPdBEbXM7E4qzaWjt5OXt1b32nZMeojMa7ZV0umoYLe0RWjoiFGWnMakkm+z0EKW56YwryiISdbR2RGjt8NuU5qZz0fSRnDE6n2DAqGpso6a5g9yMEGV5GbR2RNhf38bYokwlGOk3JQWRJNtb18rq8lpaOyKcN6WE4pz07uam13bW8MauOiob2jCDzHCQzHCQysY2th1ooqU9wt56X4sBP/lhRjhIRjhAVWM7nVFHdlqQrPQQlQ1+GzM499QS1u+up6qpneLsNKaW5VKUk0ZdcweBgBE02FHdTGlOOm+fWsr4Yp906lo6mDIil9LcNPbV+2lTolHHZbNGUZKTTm1zB7tqWxhTmMnUslyNfB+GlBRETgLtnVFCAet1eW5tczvPbazk1Z01NLZ1Mn1UHqW56Wzc28DS13YxbVQeF5xWypryOrYeaKS2uYP8zDDOOToijnFFWWyvamLD3oY+XzctGADjsE79LsXZaYzIyyAtFKC8uhkDMtOCdESizDgln7MnF7N2Vx3BQIBJpdnUNLVTmO0vNe5qMmvrjLKjqonGtk7yM8PkZ4YpyEqjIDPMqIIM0kPBuK99qJqmdoJB03xex0lJQSTF1TV3sLe+lWAActLDvLm3nvoW39k+d1whkYjjubf2094ZJTcjzKj8DHZWN7OlspH9DW3sr2+ltSPK2KIszPxFAAEzXth8gL31rZTmpuMcHGhsIyMc6FdfTZeAQX5mmPbOKKMKMinMCrO9qpncdN9E1hGJ0toZoabJ12DAJ6qcjBBF2WmMKcxibGEm+xvaeG1nDaMLs5hUkt2dfNJCAVo7/KXSBjS0dpKZFqQ4J43ibN9sNzI/g9rmDqqa2qhuaqe6qZ28zDAzRuUxIi/+VXUnMyUFEUmKSNSxr76VUfkZ3f0lGeEAVU3trCmvpTPqCJgRChhji7LIzwxT19JBXUs7dS0dVDd1sLO6meqmNsLBALtqWqiJ9dU0tXeyv76NtFhzWm5GiOmj8nDAjqpmmto6OdDYRkVNC7trW8hOD7FgfCG761qpqGlOOKDyaJTkpMeSni8fOyK+r6itM0JWWogpI3JICwXojDqisUGZjbHYzp9ayrxxhdz9/BaCAWNqWS4FWWEml+Zw2shcqhrbqW/toLmtk6b2CPmZYSaVZhM0I+Kcv/Fh1FGUncapI3JYt7uO5vYIZ00qJnwcl1orKYjIsBaJOgx6Nb1Foo76lg46IlEy0oJEIg6HvwigpT1CVVMbBxrb2X6gicrGNgqywhRnp1GUnU5RdpgDje2s313P+j311LV0AGBAKGixPp8gdc0dbKlsJBJ1BANGKGgEAwEywwFy0kP8ZWMlkahjalkOZXkZbK30Fyg0tUeO6/0WZoW544oZXDln9DE9XyOaRWRYC8aZJiUYMAqz499bPC0UID8rzKRS+hytf+oIOGtS8XHFtbWykW0Hmjh/amn3IErnHDurm9la2URJTjqF2WGy00JkpgWpavJJyjkIBCBoRjBg7KlrZePeBqaNyiUtGGDZmt2MLkj+aH7VFEREUkB/awq67kxERLopKYiISDclBRER6aakICIi3ZQURESkm5KCiIh0U1IQEZFuSgoiItLtpBu8ZmaVwI5jfHoJcOAEhnMiDdXYFNfRUVxHb6jGNtziGu+cKz3SRiddUjgeZrayPyP6BsNQjU1xHR3FdfSGamypGpeaj0REpJuSgoiIdEu1pHDPYAeQwFCNTXEdHcV19IZqbCkZV0r1KYiISGKpVlMQEZEElBRERKRbyiQFM1tsZhvNbLOZ3TaIcYw1s7+Y2Ztmts7MPh1bfoeZ7TKz1bGfJYMQ23YzeyP2+itjy4rM7M9mtin2u3CAYzqtxzFZbWb1ZvaZwTpeZna/me03s7U9lsU9RubdGfvOvW5m8wY4ru+Z2YbYaz9hZgWx5RPMrKXHsbt7gOPq87Mzsy/FjtdGM7skWXEliO2RHnFtN7PVseUDcswSlA8D9x1zzg37HyAIbAEmAWnAGmD6IMUyCpgXe5wLvAVMB+4APj/Ix2k7UHLIsu8Ct8Ue3wb82yB/jnuB8YN1vIC3A/OAtUc6RsAS4A/42/yeBbw0wHFdDIRij/+tR1wTem43CMcr7mcX+z9YA6QDE2P/s8GBjO2Q9f8OfHUgj1mC8mHAvmOpUlNYBGx2zm11zrUDDwNXDkYgzrk9zrlXY48bgDeBY7sT98C4Evh57PHPgasGMZYLgS3OuWMd0X7cnHN/BaoPWdzXMboS+IXzVgAFZjZqoOJyzv3JOdcZ+3MFMCYZr320cSVwJfCwc67NObcN2Iz/3x3w2MzMgOuBh5L1+n3E1Ff5MGDfsVRJCqOB8h5/VzAECmIzmwDMBV6KLfpkrAp4/0A308Q44E9mtsrMboktK3PO7QH/hQVGDEJcXW6k9z/pYB+vLn0do6H0vfsg/oyyy0Qze83Mnjez8wYhnnif3VA6XucB+5xzm3osG9Bjdkj5MGDfsVRJChZn2aBei2tmOcDjwGecc/XAT4DJwBxgD77qOtDOcc7NAy4FPmFmbx+EGOIyszTgCuA3sUVD4XgdyZD43pnZl4FO4MHYoj3AOOfcXOCzwK/NLG8AQ+rrsxsSxyvmPfQ+ARnQYxanfOhz0zjLjuuYpUpSqADG9vh7DLB7kGLBzML4D/xB59xvAZxz+5xzEedcFLiXJFab++Kc2x37vR94IhbDvq7qaOz3/oGOK+ZS4FXn3L5YjIN+vHro6xgN+vfOzG4GLgfe62KN0LHmmarY41X4tvupAxVTgs9u0I8XgJmFgGuAR7qWDeQxi1c+MIDfsVRJCq8AU8xsYuyM80Zg2WAEEmurvA940zn3Hz2W92wHvBpYe+hzkxxXtpnldj3Gd1KuxR+nm2Ob3Qw8OZBx9dDrzG2wj9ch+jpGy4B/il0hchZQ19UEMBDMbDHwL8AVzrnmHstLzSwYezwJmAJsHcC4+vrslgE3mlm6mU2MxfXyQMXVwzuBDc65iq4FA3XM+iofGMjvWLJ704fKD76X/i18hv/yIMZxLr569zqwOvazBPgl8EZs+TJg1ADHNQl/5ccaYF3XMQKKgWeATbHfRYNwzLKAKiC/x7JBOV74xLQH6MCfpX2or2OEr9rfFfvOvQEsGOC4NuPbm7u+Z3fHtr029hmvAV4F3jXAcfX52QFfjh2vjcClA/1ZxpY/AHzskG0H5JglKB8G7DumaS5ERKRbqjQfiYhIPygpiIhINyUFERHppqQgIiLdlBRERKSbkoLIADKzC8zsd4Mdh0hflBRERKSbkoJIHGZ2k5m9HJs7/6dmFjSzRjP7dzN71cyeMbPS2LZzzGyFHbxvQddc96ea2dNmtib2nMmx3eeY2WPm73XwYGwUq8iQoKQgcggzOx24AT9B4BwgArwXyMbPvzQPeB74WuwpvwD+xTk3Cz+qtGv5g8BdzrnZwNn40bPgZ778DH6e/EnAOUl/UyL9FBrsAESGoAuB+cArsZP4TPwEZFEOTpL2K+C3ZpYPFDjnno8t/znwm9g8UqOdc08AOOdaAWL7e9nF5tUxf2evCcDfk/+2RI5MSUHkcAb83Dn3pV4Lzb5yyHaJ5ohJ1CTU1uNxBP0fyhCi5iORwz0DXGdmI6D7/rjj8f8v18W2+Ufg7865OqCmx01X3gc87/wc+BVmdlVsH+lmljWg70LkGOgMReQQzrn1ZnY7/i50Afwsmp8AmoAZZrYKqMP3O4CfyvjuWKG/FfhAbPn7gJ+a2ddj+3j3AL4NkWOiWVJF+snMGp1zOYMdh0gyqflIRES6qaYgIiLdVFMQEZFuSgoiItJNSUFERLopKYiISDclBRER6fb/AaFBJ+jSw6ijAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# summarize history for loss\n",
    "plt.plot(history.history['loss'])\n",
    "plt.plot(history.history['val_loss'])\n",
    "plt.title('model loss')\n",
    "plt.ylabel('loss')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'evaluate'], loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### display some mis-classified images randomly (part 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "number of misclassified is:  1059\n"
     ]
    }
   ],
   "source": [
    "# get the mis-classified images\n",
    "miscla_index = []\n",
    "y_prob = model.predict(x_test, verbose=0)\n",
    "y_pred = y_prob.argmax(axis=-1)\n",
    "y_real_label = y_test.argmax(axis=-1)\n",
    "for i in range(len(y_real_label)):\n",
    "    if y_pred[i] != y_real_label[i]:\n",
    "        miscla_index.append(i)\n",
    "print('number of misclassified is: ',len(miscla_index))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# used capatalized letter to represent both capatalized and small letters\n",
    "y_classes = [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26]  \n",
    "y_labels = ['A','B','C','D','E','F','G','H','I','J','K','L','M','N','O','P','Q','R','S','T','U','V','W','X','Y','Z']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "displaying indexes are:  [7042, 6782, 12956, 5515, 16971, 8343, 2668, 9272]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAv8AAAaOCAYAAADFylJVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzs3XuMZ2ddP/D3Z2dvrd22ge7SbmlL6QUoprfYAgran4gYURPwAorRSlCKBiJCQzQqakAgGkVstIixIhGtYKNQogJBkhpbqRUqglQqvdKLu91ets12LzPP74/vaTJsz3e6M9nd2Znn9UomO/M55zzPMzPf7573eb7f80y11gIAAKx+a5Z7AAAAwOEh/AMAQCeEfwAA6ITwDwAAnRD+AQCgE8I/AAB0QvhfJlXVqurMKds+V1WvP8B2bq+q71niGKYeW1VXVtWvLaVdAGBlqKo/r6p3Dp+/pKpuOUz9ykHLZO1yD4AjU2vtsqUcV1UtyVmttVsP8pAAgEOotXZdkuc81X5VdWmS17fWXnzIB7VMVnMOMvO/BFXlogkAOKLIJxwI4f8ADS8Nvb2q/jPJY1W1tqq2VtXfVtW2qrqtqt48b/+Lq+r6qnqoqu6tqiuqav0S+j2jqj5bVQ9U1faq+suqOn6/3S6qqq9U1YNVdVVVbZx3/A9U1ReHcfxrVZ17gP3OfxnwhKq6dmhjR1VdV1UeOwBwiA3545fHzvNVdUlV3T3kk/uSXDXUp577q+qCqvqPqtpZVVcnmZ8ZLqmqu+d9fUpVXTPknAeGLPO8JFcmeVFVPVpVDw37bqiq362qO6vq/uFtM0fNa+vyIQ/dU1WvW8T3LwcdZCt24Mvkx5O8IsnxSeaSfCLJzUlOTvLSJL9YVS8f9p1N8pYkJyR50bD955fQZyV5d5KtSZ6X5JQkv7HfPq9N8vIkZyQ5O8mvJklVXZjkz5K8IcnTk3wgycerasMix/DWJHcn2ZzkGUl+JUlb/LcCACzB6Hl+cGKSpyU5LcnPLXTuHyYh/y7Jh4djPprkh8c6rKqZJNcmuSPJszLJOn/dWvvvJJclub61dkxr7Ykg/t5hbOcnOXPY/9eHtr4vyduSvCzJWUkW8x59OeggE/4X5/2ttbtaa7uSXJRkc2vtt1pre1prX0/ywSSvSZLW2k2ttRtaa/taa7dn8oD7rsV22Fq7tbX26dba7tbatiS/N9LOFcO4diR5VyYXKUnys0k+0Fr7t9babGvtQ0l2J3nhIoexN8lJSU5rre1trV3XWluxD3oAWGGmneeTyWTkO4acsCsLn/tfmGRdkvcN5/OPJblxSp8XZxK4L2+tPdZae7y19i9jO1ZVDf2+pbW2o7W2M8lvZ8hESX4syVWttf9qrT2WJ4f3qeSgg897wxbnrnmfn5Zk6xMvdw1mklyXJFV1diYP0G9LcnQmP+ubFtthVW1J8v4kL0myKZMLtgcXGNcdmTxZnxjjT1fVm+ZtXz9v+4H6nUyeqJ+aPL/zJ6219yyyDQBgaaad55NkW2vt8XlfL3Tub0m+sV9wvWNKn6ckuaO1tu8Axrc5k6xz05ATksmM/czw+dZ8cwaa1ueTyEEHn5n/xZn/ZLkryW2ttePnfWxqrX3/sP2Pk3w1kzu+j83kJaLK4r176PfcoZ2fHGnnlHmfn5rknnljfNd+Yzy6tfZXixlAa21na+2trbVnJ/nBJL9UVS9dwvcCACzetPN88uS3nyx07r83yck1L6EP7Y25K8mpU24i3r/P7Ul2JXn+vD6Pa60dM2y/d+R7OFBy0EEm/C/d55M8Mtxkc1RVzVTVt1bVRcP2TUkeSfJoVT03yRuX2M+mJI8meaiqTk5y+cg+v1BVz6yqp2VykXH1UP9gksuq6gU18S1V9Yqq2rSYAQw3y5w5/GfxSCb3M8wucMj6qto472NmgX0BgIVNO8+PWejcf32SfUneXJOFS16Vydt7xnw+k9D+nqGNjVX1HcO2+5M8c7iHIK21uaHf3x9m6lNVJ8+7D/JvklxaVedU1dFJ3rGI730l5qAjmvC/RK212Uyu/s5PclsmV71/muS4YZe3JfmJJDszefAt9ERdyG8muTDJw0k+meSakX0+kuRTSb4+fLxzGOO/Z/J+tysyeYns1iSXLmEMZyX5TCZPvuuT/FFr7XML7P/lTGYAnvj4mSX0CQBMjJ7nxyx07m+t7UnyquHrB5O8OuO5Yn7OOTPJnZnc8PrqYfNnMznX31dV24fa24e+bqiqRzLJDc8Z2vqHJO8bjrt1+PdArcQcdESrFXy/AgDAqlZVt2fyB7U+s9xjYXUw8w8AAJ0Q/gEAoBPe9gMAAJ0w8w8AAJ0Q/gEAoBOH9S/8vmzNj3qPESvOp+c+upQ/zgbAKiC7sBItlF3M/AMAQCeEfwAA6ITwDwAAnRD+AQCgE8I/AAB0QvgHAIBOCP8AANAJ4R8AADoh/AMAQCeEfwAA6MTa5R4Ah9aao48erdfJJ049pt1z/2h97rHHDsqYAACmkV0OLTP/AADQCeEfAAA6IfwDAEAnhH8AAOiE8A8AAJ0Q/gEAoBOW+lwt1syMlnf8yHmj9dPfcMvUpm667tzR+tl/cNtofd+99z3F4AAA9iO7LAsz/wAA0AnhHwAAOiH8AwBAJ4R/AADohPAPAACdsNrPKlEz43fMP3BuG63/46nXTm3rIz/0tdH6J67+zvEDOr5jHgBYGtlleZj5BwCATgj/AADQCeEfAAA6IfwDAEAnhH8AAOiE1X5WuTbl8m5dTb/u21h7D9FoAAAWJrscWmb+AQCgE8I/AAB0QvgHAIBOCP8AANAJ4R8AADphtZ9VojZuGK3PHTM7Wl/jug8AWEayy/LwUwQAgE4I/wAA0AnhHwAAOiH8AwBAJ4R/AADohNV+Vphat368furW0fprL75htL6h/OoBgENPdjmymPkHAIBOCP8AANAJ4R8AADoh/AMAQCeEfwAA6ITwDwAAnbBm0ioxt378V3n6hm2HeSQAAE9NdlkeZv4BAKATwj8AAHRC+AcAgE4I/wAA0AnhHwAAOmG1n5VmTY3XZ8brMzV3CAcDAPAUZJcjipl/AADohPAPAACdEP4BAKATwj8AAHRC+AcAgE5Y7WeFmdmyebR+/3mbRuvnbPjGtJam9rG3Td8GALAYssuRxcw/AAB0QvgHAIBOCP8AANAJ4R8AADoh/AMAQCes9rPCzG45brT+4PPbaP2cdbOj9X0L9PHhO18wWj9++8OLbgsA6JvscmQx8w8AAJ0Q/gEAoBPCPwAAdEL4BwCATgj/AADQCeEfAAA6YanPI1StHf/VbL/w2NH6SefcP1pfVzOj9R2zu6f2vf3GZ4zWN91309RjAIC+yS4rg5l/AADohPAPAACdEP4BAKATwj8AAHRC+AcAgE5Y7ecINXPSiaP1ta/cNlp/79kfG98/43fM/8XDF0zt+9R/2jVab3v3TD0GAOib7LIymPkHAIBOCP8AANAJ4R8AADoh/AMAQCeEfwAA6ITVfo5Qs1uOG62/7vR/Hq2ft378bvbdbfz67srrL5na93Nv/upofW7qEQBA72SXlcHMPwAAdEL4BwCATgj/AADQCeEfAAA6IfwDAEAnrPazjGrt9B//9guPHa2fv/GO0fq6mhmt722zo/WZneP7J0l7fPfUbQBAv2SXlc/MPwAAdEL4BwCATgj/AADQCeEfAAA6IfwDAEAnrPazjGZOOnHqtrWv3DZaP2/9+P4bat1o/Qu7x6/vjv9KTe27zY7fZQ8A9E12WfnM/AMAQCeEfwAA6ITwDwAAnRD+AQCgE8I/AAB0QvgHAIBOWOpzGe0+c8vUbW864+9H62szM1rf28aXuPr4wxeM1jff+ODUvufmLJcFADyZ7LLymfkHAIBOCP8AANAJ4R8AADoh/AMAQCeEfwAA6ITVfg6DWrd+tH7XSzdMPea7j7pjtL4v48fcvGe8nU9+6MWj9a23fnFq3wBA32SX1cvMPwAAdEL4BwCATgj/AADQCeEfAAA6IfwDAEAnrPZzGNTG8bvc92zeN/WYTWvGfzV72+xo/Wt7Th6tn/Cl3aP1uV27pvYNAPRNdlm9zPwDAEAnhH8AAOiE8A8AAJ0Q/gEAoBPCPwAAdMJqP4fBvvPOGK1f9u2fm3rMUbV+tH7D+A3w+cP//X+j9af/z/3jY2ptat8AQN9kl9XLzD8AAHRC+AcAgE4I/wAA0AnhHwAAOiH8AwBAJ6z2cxDVuvG73O/63qNG6z913BemtrUvG0brb7vlNeP7X7N5tD57741T+wAA+ia79MfMPwAAdEL4BwCATgj/AADQCeEfAAA6IfwDAEAnhH8AAOiEpT4Pg73HtNH6pjXTf/w375nS1jVbRuvP+PTdo/V9+/YtPDgAgP3ILquXmX8AAOiE8A8AAJ0Q/gEAoBPCPwAAdEL4BwCATljt5zBYt3P8GmvH3PS72b/0+LNG6yd8Yedoffa+/1v0uAAAxsguq5eZfwAA6ITwDwAAnRD+AQCgE8I/AAB0QvgHAIBOWO3nIGp794zWT//ojtH6y89549S2ztqybbS+Ztfe0frcXHuK0QEAfDPZpT9m/gEAoBPCPwAAdEL4BwCATgj/AADQCeEfAAA6YbWfw2Duy7eM1p/9lq1Tj9l37LHjbf3v7aP1aXfrAwAsluyyepn5BwCATgj/AADQCeEfAAA6IfwDAEAnhH8AAOiE8A8AAJ2w1Ofh0Npoed/d3zjMAwEAOACyy6pl5h8AADoh/AMAQCeEfwAA6ITwDwAAnRD+AQCgE8I/AAB0QvgHAIBOCP8AANAJ4R8AADoh/AMAQCeEfwAA6ES11pZ7DAAAwGFg5h8AADoh/AMAQCeEfwAA6ITwDwAAnRD+AQCgE8I/AAB0QvgHAIBOCP8AANAJ4R8AADoh/AMAQCeEfwAA6ITwDwAAnRD+AQCgE8I/AAB0QvgHAIBOCP8AANAJ4R8AADoh/AMAQCeE/2VSVa2qzpyy7XNV9foDbOf2qvqeJY5h6rFVdWVV/dpS2gUAVoaq+vOqeufw+Uuq6pbD1K8ctEzWLvcAODK11i5bynFV1ZKc1Vq79SAPCQA4hFpr1yV5zlPtV1WXJnl9a+3Fh3xQy2Q15yAz/0tQVS6aAIAjinzCgRD+D9Dw0tDbq+o/kzxWVWuramtV/W1Vbauq26rqzfP2v7iqrq+qh6rq3qq6oqrWL6HfM6rqs1X1QFVtr6q/rKrj99vtoqr6SlU9WFVXVdXGecf/QFV9cRjHv1bVuQfY7/yXAU+oqmuHNnZU1XVV5bEDAIfYkD9+eew8X1WXVNXdQz65L8lVQ33qub+qLqiq/6iqnVV1dZL5meGSqrp73tenVNU1Q855YMgyz0tyZZIXVdWjVfXQsO+Gqvrdqrqzqu4f3jZz1Ly2Lh/y0D1V9bpFfP9y0EG2Yge+TH48ySuSHJ9kLsknktyc5OQkL03yi1X18mHf2SRvSXJCkhcN239+CX1Wkncn2ZrkeUlOSfIb++3z2iQvT3JGkrOT/GqSVNWFSf4syRuSPD3JB5J8vKo2LHIMb01yd5LNSZ6R5FeStMV/KwDAEoye5wcnJnlaktOS/NxC5/5hEvLvknx4OOajSX54rMOqmklybZI7kjwrk6zz1621/05yWZLrW2vHtNaeCOLvHcZ2fpIzh/1/fWjr+5K8LcnLkpyVZDHv0ZeDDjLhf3He31q7q7W2K8lFSTa31n6rtbantfb1JB9M8pokaa3d1Fq7obW2r7V2eyYPuO9abIettVtba59ure1urW1L8nsj7VwxjGtHkndlcpGSJD+b5AOttX9rrc221j6UZHeSFy5yGHuTnJTktNba3tbada21FfugB4AVZtp5PplMRr5jyAm7svC5/4VJ1iV533A+/1iSG6f0eXEmgfvy1tpjrbXHW2v/MrZjVdXQ71taaztaazuT/HaGTJTkx5Jc1Vr7r9baY3lyeJ9KDjr4vDdsce6a9/lpSbY+8XLXYCbJdUlSVWdn8gD9tiRHZ/KzvmmxHVbVliTvT/KSJJsyuWB7cIFx3ZHJk/WJMf50Vb1p3vb187YfqN/J5In6qcnzO3/SWnvPItsAAJZm2nk+Sba11h6f9/VC5/6W5Bv7Bdc7pvR5SpI7Wmv7DmB8mzPJOjcNOSGZzNjPDJ9vzTdnoGl9PokcdPCZ+V+c+U+Wu5Lc1lo7ft7Hptba9w/b/zjJVzO54/vYTF4iqizeu4d+zx3a+cmRdk6Z9/mpSe6ZN8Z37TfGo1trf7WYAbTWdrbW3tpae3aSH0zyS1X10iV8LwDA4k07zydPfvvJQuf+e5OcXPMS+tDemLuSnDrlJuL9+9yeZFeS58/r87jW2jHD9ntHvocDJQcdZML/0n0+ySPDTTZHVdVMVX1rVV00bN+U5JEkj1bVc5O8cYn9bEryaJKHqurkJJeP7PMLVfXMqnpaJhcZVw/1Dya5rKpeUBPfUlWvqKpNixnAcLPMmcN/Fo9kcj/D7AKHrK+qjfM+ZhbYFwBY2LTz/JiFzv3XJ9mX5M01WbjkVZm8vWfM5zMJ7e8Z2thYVd8xbLs/yTOHewjSWpsb+v39YaY+VXXyvPsg/ybJpVV1TlUdneQdi/jeV2IOOqIJ/0vUWpvN5Orv/CS3ZXLV+6dJjht2eVuSn0iyM5MH30JP1IX8ZpILkzyc5JNJrhnZ5yNJPpXk68PHO4cx/nsm73e7IpOXyG5NcukSxnBWks9k8uS7PskftdY+t8D+X85kBuCJj59ZQp8AwMToeX7MQuf+1tqeJK8avn4wyasznivm55wzk9yZyQ2vrx42fzaTc/19VbV9qL196OuGqnokk9zwnKGtf0jyvuG4W4d/D9RKzEFHtFrB9ysAAKxqVXV7Jn9Q6zPLPRZWBzP/AADQCeEfAAA64W0/AADQCTP/AADQCeEfAAA6cVj/wu/L1vyo9xix4nx67qNL+eNsAKwCsgsr0ULZxcw/AAB0QvgHAIBOCP8AANAJ4R8AADoh/AMAQCeEfwAA6ITwDwAAnRD+AQCgE8I/AAB0QvgHAIBOCP8AANCJtcs9AA6tNUcfPVqvk0+ceky75/7R+txjjx2UMQEATCO7HFpm/gEAoBPCPwAAdEL4BwCATgj/AADQCeEfAAA6IfwDAEAnLPW5WqyZGS3v+JHzRuunv+GWqU3ddN25o/Wz/+C20fq+e+97isEBAOxHdlkWZv4BAKATwj8AAHRC+AcAgE4I/wAA0AnhHwAAOmG1n1WiZsbvmH/g3DZa/8dTr53a1kd+6Guj9U9c/Z3jB3R8xzwAsDSyy/Iw8w8AAJ0Q/gEAoBPCPwAAdEL4BwCATgj/AADQCav9rHJtyuXdupp+3bex9h6i0QAALEx2ObTM/AMAQCeEfwAA6ITwDwAAnRD+AQCgE8I/AAB0wmo/q0Rt3DBanztmdrS+xnUfALCMZJfl4acIAACdEP4BAKATwj8AAHRC+AcAgE4I/wAA0Amr/awwtW79eP3UraP11158w2h9Q/nVAwCHnuxyZDHzDwAAnRD+AQCgE8I/AAB0QvgHAIBOCP8AANAJ4R8AADphzaRVYm79+K/y9A3bDvNIAACemuyyPMz8AwBAJ4R/AADohPAPAACdEP4BAKATwj8AAHTCaj8rzZoar8+M12dq7hAOBgDgKcguRxQz/wAA0AnhHwAAOiH8AwBAJ4R/AADohPAPAACdsNrPCjOzZfNo/f7zNo3Wz9nwjWktTe1jb5u+DQBgMWSXI4uZfwAA6ITwDwAAnRD+AQCgE8I/AAB0QvgHAIBOWO1nhZndctxo/cHnt9H6OetmR+v7Fujjw3e+YLR+/PaHF90WANA32eXIYuYfAAA6IfwDAEAnhH8AAOiE8A8AAJ0Q/gEAoBPCPwAAdMJSn0eoWjv+q9l+4bGj9ZPOuX+0vq5mRus7ZndP7Xv7jc8YrW+676apxwAAfZNdVgYz/wAA0AnhHwAAOiH8AwBAJ4R/AADohPAPAACdsNrPEWrmpBNH62tfuW20/t6zPza+f8bvmP+Lhy+Y2vep/7RrtN727pl6DADQN9llZTDzDwAAnRD+AQCgE8I/AAB0QvgHAIBOCP8AANAJq/0coWa3HDdaf93p/zxaP2/9+N3su9v49d2V118yte/n3vzV0frc1CMAgN7JLiuDmX8AAOiE8A8AAJ0Q/gEAoBPCPwAAdEL4BwCATljtZxnV2uk//u0XHjtaP3/jHaP1dTUzWt/bZkfrMzvH90+S9vjuqdsAgH7JLiufmX8AAOiE8A8AAJ0Q/gEAoBPCPwAAdEL4BwCATljtZxnNnHTi1G1rX7lttH7e+vH9N9S60foXdo9f3x3/lZrad5sdv8seAOib7LLymfkHAIBOCP8AANAJ4R8AADoh/AMAQCeEfwAA6ITwDwAAnbDU5zLafeaWqdvedMbfj9bXZma0vreNL3H18YcvGK1vvvHBqX3PzVkuCwB4Mtll5TPzDwAAnRD+AQCgE8I/AAB0QvgHAIBOCP8AANAJq/0cBrVu/Wj9rpdumHrMdx91x2h9X8aPuXnPeDuf/NCLR+tbb/3i1L4BgL7JLquXmX8AAOiE8A8AAJ0Q/gEAoBPCPwAAdEL4BwCATljt5zCojeN3ue/ZvG/qMZvWjP9q9rbZ0frX9pw8Wj/hS7tH63O7dk3tGwDom+yyepn5BwCATgj/AADQCeEfAAA6IfwDAEAnhH8AAOiE1X4Og33nnTFav+zbPzf1mKNq/Wj9hvEb4POH//v/RutP/5/7x8fU2tS+AYC+yS6rl5l/AADohPAPAACdEP4BAKATwj8AAHRC+AcAgE5Y7ecgqnXjd7nf9b1HjdZ/6rgvTG1rXzaM1t92y2vG979m82h99t4bp/YBAPRNdumPmX8AAOiE8A8AAJ0Q/gEAoBPCPwAAdEL4BwCATgj/AADQCUt9HgZ7j2mj9U1rpv/4b94zpa1rtozWn/Hpu0fr+/btW3hwAAD7kV1WLzP/AADQCeEfAAA6IfwDAEAnhH8AAOiE8A8AAJ2w2s9hsG7n+DXWjrnpd7N/6fFnjdZP+MLO0frsff+36HEBAIyRXVYvM/8AANAJ4R8AADoh/AMAQCeEfwAA6ITwDwAAnbDaz0HU9u4ZrZ/+0R2j9Zef88apbZ21Zdtofc2uvaP1ubn2FKMDAPhmskt/zPwDAEAnhH8AAOiE8A8AAJ0Q/gEAoBPCPwAAdMJqP4fB3JdvGa0/+y1bpx6z79hjx9v639tH69Pu1gcAWCzZZfUy8w8AAJ0Q/gEAoBPCPwAAdEL4BwCATgj/AADQCeEfAAA6YanPw6G10fK+u79xmAcCAHAAZJdVy8w/AAB0QvgHAIBOCP8AANAJ4R8AADoh/AMAQCeEfwAA6ITwDwAAnRD+AQCgE8I/AAB0QvgHAIBOCP8AANCJaq0t9xgAAIDDwMw/AAB0QvgHAIBOCP8AANAJ4R8AADoh/AMAQCeEfwAA6ITwDwAAnRD+AQCgE8I/AAB0QvgHAIBOCP8AANAJ4R8AADoh/AMAQCeEfwAA6ITwDwAAnRD+AQCgE8I/AAB0QvgHAIBOCP/LpKpaVZ05Zdvnqur1B9jO7VX1PUscw9Rjq+rKqvq1pbQLAKwMVfXnVfXO4fOXVNUth6lfOWiZrF3uAXBkaq1dtpTjqqolOau1dutBHhIAcAi11q5L8pyn2q+qLk3y+tbaiw/5oJbJas5BZv6XoKpcNAEARxT5hAMh/B+g4aWht1fVfyZ5rKrWVtXWqvrbqtpWVbdV1Zvn7X9xVV1fVQ9V1b1VdUVVrV9Cv2dU1Wer6oGq2l5Vf1lVx++320VV9ZWqerCqrqqqjfOO/4Gq+uIwjn+tqnMPsN/5LwOeUFXXDm3sqKrrqspjBwAOsSF//PLYeb6qLqmqu4d8cl+Sq4b61HN/VV1QVf9RVTur6uok8zPDJVV197yvT6mqa4ac88CQZZ6X5MokL6qqR6vqoWHfDVX1u1V1Z1XdP7xt5qh5bV0+5KF7qup1i/j+5aCDbMUOfJn8eJJXJDk+yVySTyS5OcnJSV6a5Ber6uXDvrNJ3pLkhCQvGrb//BL6rCTvTrI1yfOSnJLkN/bb57VJXp7kjCRnJ/nVJKmqC5P8WZI3JHl6kg8k+XhVbVjkGN6a5O4km5M8I8mvJGmL/1YAgCUYPc8PTkzytCSnJfm5hc79wyTk3yX58HDMR5P88FiHVTWT5NokdyR5ViZZ569ba/+d5LIk17fWjmmtPRHE3zuM7fwkZw77//rQ1vcleVuSlyU5K8li3qMvBx1kwv/ivL+1dldrbVeSi5Jsbq39VmttT2vt60k+mOQ1SdJau6m1dkNrbV9r7fZMHnDftdgOW2u3ttY+3Vrb3VrbluT3Rtq5YhjXjiTvyuQiJUl+NskHWmv/1lqbba19KMnuJC9c5DD2JjkpyWmttb2ttetaayv2QQ8AK8y083wymYx8x5ATdmXhc/8Lk6xL8r7hfP6xJDdO6fPiTAL35a21x1prj7fW/mVsx6qqod+3tNZ2tNZ2JvntDJkoyY8luaq19l+ttcfy5PA+lRx08Hlv2OLcNe/z05JsfeLlrsFMkuuSpKrOzuQB+m1Jjs7kZ33TYjusqi1J3p/kJUk2ZXLB9uAC47ojkyfrE2P86ap607zt6+dtP1C/k8kT9VOT53f+pLX2nkW2AQAszbTzfJJsa609Pu/rhc79Lck39guud0zp85Qkd7TW9h3A+DZnknVuGnJCMpmxnxk+35pvzkDT+nwSOejgM/O/OPOfLHclua21dvy8j02tte8ftv9xkq9mcsf3sZm8RFRZvHcP/Z47tPOTI+2cMu/zU5PcM2+M79pvjEe31v5qMQNore1srb21tfbsJD+Y5Jeq6qVL+F4AgMWbdp5Pnvz2k4XO/fcmObnmJfShvTF3JTl1yk3E+/e5PcmuJM+f1+dxrbVjhu33jnwPB0oOOsiE/6X7fJJHhptsjqqqmar61qq6aNi+KckjSR6tqucmeeMS+9mU5NEkD1XVyUkuH9nnF6rqmVX1tEwuMq4e6h9McllVvaAmvqWqXlFVmxYzgOFmmTOH/yweyeR+htkFDllfVRvnfcwssC8AsLBp5/kxC537r0+yL8ljyO28AAAgAElEQVSba7JwyasyeXvPmM9nEtrfM7Sxsaq+Y9h2f5JnDvcQpLU2N/T7+8NMfarq5Hn3Qf5Nkkur6pyqOjrJOxbxva/EHHREE/6XqLU2m8nV3/lJbsvkqvdPkxw37PK2JD+RZGcmD76FnqgL+c0kFyZ5OMknk1wzss9HknwqydeHj3cOY/z3TN7vdkUmL5HdmuTSJYzhrCSfyeTJd32SP2qtfW6B/b+cyQzAEx8/s4Q+AYCJ0fP8mIXO/a21PUleNXz9YJJXZzxXzM85Zya5M5MbXl89bP5sJuf6+6pq+1B7+9DXDVX1SCa54TlDW/+Q5H3DcbcO/x6olZiDjmi1gu9XAABY1arq9kz+oNZnlnssrA5m/gEAoBPCPwAAdMLbfgAAoBNm/gEAoBOH9Y98vWzNj3qZgRXn03MfXcrfZwBgFZBdWIkWyi5m/gEAoBPCPwAAdEL4BwCATgj/AADQCeEfAAA6IfwDAEAnhH8AAOiE8A8AAJ0Q/gEAoBPCPwAAdEL4BwCATqxd7gFwaK05+ujRep184tRj2j33j9bnHnvsoIwJAGAa2eXQMvMPAACdEP4BAKATwj8AAHRC+AcAgE4I/wAA0AnhHwAAOmGpz9VizcxoecePnDdaP/0Nt0xt6qbrzh2tn/0Ht43W991731MMDgBgP7LLsjDzDwAAnRD+AQCgE8I/AAB0QvgHAIBOCP8AANAJq/2sEjUzfsf8A+e20fo/nnrt1LY+8kNfG61/4urvHD+g4zvmAYClkV2Wh5l/AADohPAPAACdEP4BAKATwj8AAHRC+AcAgE5Y7WeVa1Mu79bV9Ou+jbX3EI0GAGBhssuhZeYfAAA6IfwDAEAnhH8AAOiE8A8AAJ0Q/gEAoBNW+1klauOG0frcMbOj9TWu+wCAZSS7LA8/RQAA6ITwDwAAnRD+AQCgE8I/AAB0QvgHAIBOWO1nhal168frp24drb/24htG6xvKrx4AOPRklyOLmX8AAOiE8A8AAJ0Q/gEAoBPCPwAAdEL4BwCATgj/AADQCWsmrRJz68d/ladv2HaYRwIA8NRkl+Vh5h8AADoh/AMAQCeEfwAA6ITwDwAAnRD+AQCgE1b7WWnW1Hh9Zrw+U3OHcDAAAE9BdjmimPkHAIBOCP8AANAJ4R8AADoh/AMAQCeEfwAA6ITVflaYmS2bR+v3n7dptH7Ohm9Ma2lqH3vb9G0AAIshuxxZzPwDAEAnhH8AAOiE8A8AAJ0Q/gEAoBPCPwAAdMJqPyvM7JbjRusPPr+N1s9ZNzta37dAHx++8wWj9eO3P7zotgCAvskuRxYz/wAA0AnhHwAAOiH8AwBAJ4R/AADohPAPAACdEP4BAKATlvo8QtXa8V/N9guPHa2fdM79o/V1NTNa3zG7e2rf2298xmh90303TT0GAOib7LIymPkHAIBOCP8AANAJ4R8AADoh/AMAQCeEfwAA6ITVfo5QMyedOFpf+8pto/X3nv2x8f0zfsf8Xzx8wdS+T/2nXaP1tnfP1GMAgL7JLiuDmX8AAOiE8A8AAJ0Q/gEAoBPCPwAAdEL4BwCATljt5wg1u+W40frrTv/n0fp568fvZt/dxq/vrrz+kql9P/fmr47W56YeAQD0TnZZGcz8AwBAJ4R/AADohPAPAACdEP4BAKATwj8AAHTCaj/LqNZO//Fvv/DY0fr5G+8Yra+rmdH63jY7Wp/ZOb5/krTHd0/dBgD0S3ZZ+cz8AwBAJ4R/AADohPAPAACdEP4BAKATwj8AAHTCaj/LaOakE6duW/vKbaP189aP77+h1o3Wv7B7/Pru+K/U1L7b7Phd9gBA32SXlc/MPwAAdEL4BwCATgj/AADQCeEfAAA6IfwDAEAnhH8AAOiEpT6X0e4zt0zd9qYz/n60vjYzo/W9bXyJq48/fMFoffOND07te27OclkAwJPJLiufmX8AAOiE8A8AAJ0Q/gEAoBPCPwAAdEL4BwCATljt5zCodetH63e9dMPUY777qDtG6/syfszNe8bb+eSHXjxa33rrF6f2DQD0TXZZvcz8AwBAJ4R/AADohPAPAACdEP4BAKATwj8AAHTCaj+HQW0cv8t9z+Z9U4/ZtGb8V7O3zY7Wv7bn5NH6CV/aPVqf27Vrat8AQN9kl9XLzD8AAHRC+AcAgE4I/wAA0AnhHwAAOiH8AwBAJ6z2cxjsO++M0fpl3/65qcccVetH6zeM3wCfP/zf/zdaf/r/3D8+ptam9g0A9E12Wb3M/AMAQCeEfwAA6ITwDwAAnRD+AQCgE8I/AAB0wmo/B1GtG7/L/a7vPWq0/lPHfWFqW/uyYbT+tlteM77/NZtH67P33ji1DwCgb7JLf8z8AwBAJ4R/AADohPAPAACdEP4BAKATwj8AAHRC+AcAgE5Y6vMw2HtMG61vWjP9x3/zniltXbNltP6MT989Wt+3b9/CgwMA2I/ssnqZ+QcAgE4I/wAA0AnhHwAAOiH8AwBAJ4R/AADohNV+DoN1O8evsXbMTb+b/UuPP2u0fsIXdo7WZ+/7v0WPCwBgjOyyepn5BwCATgj/AADQCeEfAAA6IfwDAEAnhH8AAOiE1X4OorZ3z2j99I/uGK2//Jw3Tm3rrC3bRutrdu0drc/NtacYHQDAN5Nd+mPmHwAAOiH8AwBAJ4R/AADohPAPAACdEP4BAKATVvs5DOa+fMto/dlv2Tr1mH3HHjve1v/ePlqfdrc+AMBiyS6rl5l/AADohPAPAACdEP4BAKATwj8AAHRC+AcAgE4I/wAA0AlLfR4OrY2W9939jcM8EACAAyC7rFpm/gEAoBPCPwAAdEL4BwCATgj/AADQCeEfAAA6IfwDAEAnhH8AAOiE8A8AAJ0Q/gEAoBPCPwAAdEL4BwCATlRrbbnHAAAAHAZm/gEAoBPCPwAAdEL4BwCATgj/AADQCeEfAAA6IfwDAEAnhH8AAOiE8A8AAJ0Q/gEAoBPCPwAAdEL4BwCATgj/AADQCeEfAAA6IfwDAEAnhH8AAOiE8A8AAJ0Q/gEAoBPCPwAAdEL4XyZV1arqzCnbPldVrz/Adm6vqu9Z4himHltVV1bVry2lXQBgZaiqP6+qdw6fv6SqbjlM/cpBy2Ttcg+AI1Nr7bKlHFdVLclZrbVbD/KQAIBDqLV2XZLnPNV+VXVpkte31l58yAe1TFZzDjLzvwRV5aIJADiiyCccCOH/AA0vDb29qv4zyWNVtbaqtlbV31bVtqq6rarePG//i6vq+qp6qKruraorqmr9Evo9o6o+W1UPVNX2qvrLqjp+v90uqqqvVNWDVXVVVW2cd/wPVNUXh3H8a1Wde4D9zn8Z8ISqunZoY0dVXVdVHjsAcIgN+eOXx87zVXVJVd095JP7klw11Kee+6vqgqr6j6raWVVXJ5mfGS6pqrvnfX1KVV0z5JwHhizzvCRXJnlRVT1aVQ8N+26oqt+tqjur6v7hbTNHzWvr8iEP3VNVr1vE9y8HHWQrduDL5MeTvCLJ8Unmknwiyc1JTk7y0iS/WFUvH/adTfKWJCckedGw/eeX0GcleXeSrUmel+SUJL+x3z6vTfLyJGckOTvJryZJVV2Y5M+SvCHJ05N8IMnHq2rDIsfw1iR3J9mc5BlJfiVJW/y3AgAsweh5fnBikqclOS3Jzy107h8mIf8uyYeHYz6a5IfHOqyqmSTXJrkjybMyyTp/3Vr77ySXJbm+tXZMa+2JIP7eYWznJzlz2P/Xh7a+L8nbkrwsyVlJFvMefTnoIBP+F+f9rbW7Wmu7klyUZHNr7bdaa3taa19P8sEkr0mS1tpNrbUbWmv7Wmu3Z/KA+67Fdthau7W19unW2u7W2rYkvzfSzhXDuHYkeVcmFylJ8rNJPtBa+7fW2mxr7UNJdid54SKHsTfJSUlOa63tba1d11pbsQ96AFhhpp3nk8lk5DuGnLArC5/7X5hkXZL3DefzjyW5cUqfF2cSuC9vrT3WWnu8tfYvYztWVQ39vqW1tqO1tjPJb2fIREl+LMlVrbX/aq09lieH96nkoIPPe8MW5655n5+WZOsTL3cNZpJclyRVdXYmD9BvS3J0Jj/rmxbbYVVtSfL+JC9JsimTC7YHFxjXHZk8WZ8Y409X1ZvmbV8/b/uB+p1Mnqifmjy/8yettfcssg0AYGmmneeTZFtr7fF5Xy907m9JvrFfcL1jSp+nJLmjtbbvAMa3OZOsc9OQE5LJjP3M8PnWfHMGmtbnk8hBB5+Z/8WZ/2S5K8ltrbXj531saq19/7D9j5N8NZM7vo/N5CWiyuK9e+j33KGdnxxp55R5n5+a5J55Y3zXfmM8urX2V4sZQGttZ2vtra21Zyf5wSS/VFUvXcL3AgAs3rTzfPLkt58sdO6/N8nJNS+hD+2NuSvJqVNuIt6/z+1JdiV5/rw+j2utHTNsv3fkezhQctBBJvwv3eeTPDLcZHNUVc1U1bdW1UXD9k1JHknyaFU9N8kbl9jPpiSPJnmoqk5OcvnIPr9QVc+sqqdlcpFx9VD/YJLLquoFNfEtVfWKqtq0mAEMN8ucOfxn8Ugm9zPMLnDI+qraOO9jZoF9AYCFTTvPj1no3H99kn1J3lyThUtelcnbe8Z8PpPQ/p6hjY1V9R3DtvuTPHO4hyCttbmh398fZupTVSfPuw/yb5JcWlXnVNXRSd6xiO99JeagI5rwv0SttdlMrv7OT3JbJle9f5rkuGGXtyX5iSQ7M3nwLfREXchvJrkwycNJPpnkmpF9PpLkU0m+Pny8cxjjv2fyfrcrMnmJ7NYkly5hDGcl+UwmT77rk/xRa+1zC+z/5UxmAJ74+Jkl9AkATIye58csdO5vre1J8qrh6weTvDrjuWJ+zjkzyZ2Z3PD66mHzZzM5199XVduH2tuHvm6oqkcyyQ3PGdr6hyTvG467dfj3QK3EHHREqxV8vwIAwKpWVbdn8ge1PrPcY2F1MPMPAACdEP4BAKAT3vYDAACdMPMPAACdEP4BAKATh/Uv/L5szY96jxErzqfnPrqUP84GwCogu7ASLZRdzPwDAEAnhH8AAOiE8A8AAJ0Q/gEAoBPCPwAAdEL4BwCATgj/AADQCeEfAAA6IfwDAEAnhH8AAOjE2uUeAIfWmqOPHq3XySdOPabdc/9ofe6xxw7KmAAAppFdDi0z/wAA0AnhHwAAOiH8AwBAJ4R/AADohPAPAACdEP4BAKATlvpcLdbMjJZ3/Mh5o/XT33DL1KZuuu7c0frZf3DbaH3fvfc9xeAAAPYjuywLM/8AANAJ4R8AADoh/AMAQCeEfwAA6ITwDwAAnbDazypRM+N3zD9wbhut/+Op105t6yM/9LXR+ieu/s7xAzq+Yx4AWBrZZXmY+QcAgE4I/wAA0AnhHwAAOiH8AwBAJ4R/AADohNV+Vrk25fJuXU2/7ttYew/RaAAAFia7HFpm/gEAoBPCPwAAdEL4BwCATgj/AADQCeEfAAA6YbWfVaI2bhitzx0zO1pf47oPAFhGssvy8FMEAIBOCP8AANAJ4R8AADoh/AMAQCeEfwAA6ITVflaYWrd+vH7q1tH6ay++YbS+ofzqAYBDT3Y5spj5BwCATgj/AADQCeEfAAA6IfwDAEAnhH8AAOiE8A8AAJ2wZtIqMbd+/Fd5+oZth3kkAABPTXZZHmb+AQCgE8I/AAB0QvgHAIBOCP8AANAJ4R8AADphtZ+VZk2N12fG6zM1dwgHAwDwFGSXI4qZfwAA6ITwDwAAnRD+AQCgE8I/AAB0QvgHAIBOWO1nhZnZsnm0fv95m0br52z4xrSWpvaxt03fBgCwGLLLkcXMPwAAdEL4BwCATgj/AADQCeEfAAA6IfwDAEAnrPazwsxuOW60/uDz22j9nHWzo/V9C/Tx4TtfMFo/fvvDi24LAOib7HJkMfMPAACdEP4BAKATwj8AAHRC+AcAgE4I/wAA0AnhHwAAOmGpzyNUrR3/1Wy/8NjR+knn3D9aX1czo/Uds7un9r39xmeM1jfdd9PUYwCAvskuK4OZfwAA6ITwDwAAnRD+AQCgE8I/AAB0QvgHAIBOWO3nCDVz0omj9bWv3DZaf+/ZHxvfP+N3zP/FwxdM7fvUf9o1Wm9790w9BgDom+yyMpj5BwCATgj/AADQCeEfAAA6IfwDAEAnhH8AAOiE1X6OULNbjhutv+70fx6tn7d+/G723W38+u7K6y+Z2vdzb/7qaH1u6hEAQO9kl5XBzD8AAHRC+AcAgE4I/wAA0AnhHwAAOiH8AwBAJ6z2s4xq7fQf//YLjx2tn7/xjtH6upoZre9ts6P1mZ3j+ydJe3z31G0AQL9kl5XPzD8AAHRC+AcAgE4I/wAA0AnhHwAAOiH8AwBAJ6z2s4xmTjpx6ra1r9w2Wj9v/fj+G2rdaP0Lu8ev747/Sk3tu82O32UPAPRNdln5zPwDAEAnhH8AAOiE8A8AAJ0Q/gEAoBPCPwAAdEL4BwCATljqcxntPnPL1G1vOuPvR+trMzNa39vGl7j6+MMXjNY33/jg1L7n5iyXBQA8meyy8pn5BwCATgj/AADQCeEf+P/s3X2MZ1ddP/D3Z2efWrttA93Sbmmh9AEopoXG8qCgaMUaURPxARSjlaAUDcRKG6JRUQMC0ShioyDGikS0FhuFEpUSfk1qbKVWqAhSKbSlpQ9uu33YNtvdnZnz++N7mwzb+53uTHZ3dua8XslkZz733nPOzHy/e9/3fL/3DADQCeEfAAA6IfwDAEAnrPZzCNSGjaP1O8/fNPWY7znijtH6bMaPuXnPeDuf/PDLR+vbbv381L4BgL7JLmuXmX8AAOiE8A8AAJ0Q/gEAoBPCPwAAdEL4BwCATljt5xCozeN3ue/ZOjv1mC3rxn81e9vcaP0re04arR/3hd2j9fldu6b2DQD0TXZZu8z8AwBAJ4R/AADohPAPAACdEP4BAKATwj8AAHTCaj+HwOw5p43WL/r2a6cec0RtHK3fMH4DfP74q989Wn/6/943PqbWpvYNAPRNdlm7zPwDAEAnhH8AAOiE8A8AAJ0Q/gEAoBPCPwAAdMJqPwdQbRi/y/3O7ztitP4zx3xualuz2TRav+SW143vf9XW0frcPTdO7QMA6Jvs0h8z/wAA0AnhHwAAOiH8AwBAJ4R/AADohPAPAACdEP4BAKATlvo8BPYe1UbrW9ZN//HfvGdKW1cdP1p/xjV3jdZnZ2cXHxwAwD5kl7XLzD8AAHRC+AcAgE4I/wAA0AnhHwAAOiH8AwBAJ6z2cwhs2Dl+jbVjfvrd7F94/Nmj9eM+t3O0Pnfv/y15XAAAY2SXtcvMPwAAdEL4BwCATgj/AADQCeEfAAA6IfwDAEAnrPZzALW9e0brp165Y7R+wVlvntrWGcdvH62v27V3tD4/355idAAA30x26Y+ZfwAA6ITwDwAAnRD+AQCgE8I/AAB0QvgHAIBOWO3nEJj/4i2j9edcvG3qMbNHHz3e1ldvH61Pu1sfAGCpZJe1y8w/AAB0QvgHAIBOCP8AANAJ4R8AADoh/AMAQCeEfwAA6ISlPg+F1kbLs3d94xAPBABgP8gua5aZfwAA6ITwDwAAnRD+AQCgE8I/AAB0QvgHAIBOCP8AANAJ4R8AADoh/AMAQCeEfwAA6ITwDwAAnRD+AQCgE9VaW+kxAAAAh4CZfwAA6ITwDwAAnRD+AQCgE8I/AAB0QvgHAIBOCP8AANAJ4R8AADoh/AMAQCeEfwAA6ITwDwAAnRD+AQCgE8I/AAB0QvgHAIBOCP8AANAJ4R8AADoh/AMAQCeEfwAA6ITwDwAAnRD+V0hVtao6fcq2a6vqjfvZzu1V9b3LHMPUY6vqA1X1G8tpFwBYHarqL6vqncPnr6iqWw5Rv3LQClm/0gPg8NRau2g5x1VVS3JGa+3WAzwkAOAgaq1dl+S5T7VfVV2Y5I2ttZcf9EGtkLWcg8z8L0NVuWgCAA4r8gn7Q/jfT8NLQ2+vqv9K8lhVra+qbVX191W1vapuq6q3Ltj/xVV1fVU9VFX3VNVlVbVxGf2eVlWfqaoHqur+qvrrqjp2n93Oq6ovVdWDVXV5VW1ecPwPVtXnh3H8W1WdvZ/9LnwZ8LiqunpoY0dVXVdVHjsAcJAN+eNXx87zVfXKqrpryCf3Jrl8qE8991fVi6rqP6tqZ1VdkWRhZnhlVd214OuTq+qqIec8MGSZ5yf5QJKXVdWjVfXQsO+mqvr9qvp6Vd03vG3miAVtXTrkobur6g1L+P7loANs1Q58hfxkklcnOTbJfJJPJLk5yUlJzk/yy1V1wbDvXJKLkxyX5GXD9l9cRp+V5N1JtiV5fpKTk/zWPvu8PskFSU5LcmaSX0+Sqjo3yV8keVOSpyf5YJKPV9WmJY7hbUnuSrI1yTOS/FqStvRvBQBYhtHz/OCEJE9L8qwkv7DYuX+YhPyHJB8ZjrkyyY+OdVhVM0muTnJHkmdnknX+trX2P0kuSnJ9a+2o1toTQfy9w9hemOT0Yf/fHNr6/iSXJHlVkjOSLOU9+nLQASb8L837W2t3ttZ2JTkvydbW2u+01va01r6W5ENJXpckrbWbWms3tNZmW2u3Z/KA+66ldthau7W1dk1rbXdrbXuSPxhp57JhXDuSvCuTi5Qk+fkkH2yt/Xtrba619uEku5O8dInD2JvkxCTPaq3tba1d11pbtQ96AFhlpp3nk8lk5DuGnLAri5/7X5pkQ5L3DefzjyW5cUqfL84kcF/aWnustfZ4a+1fx3asqhr6vbi1tqO1tjPJ72bIREl+IsnlrbX/bq09lieH96nkoAPPe8OW5s4Fnz8rybYnXu4azCS5Lkmq6sxMHqDfluTITH7WNy21w6o6Psn7k7wiyZZMLtgeXGRcd2TyZH1ijD9bVW9ZsH3jgu376/cyeaJ+avL8zp+11t6zxDYAgOWZdp5Pku2ttccXfL3Yub8l+cY+wfWOKX2enOSO1trsfoxvayZZ56YhJySTGfuZ4fNt+eYMNK3PJ5GDDjwz/0uz8MlyZ5LbWmvHLvjY0lr7gWH7nyb5ciZ3fB+dyUtElaV799Dv2UM7Pz3SzskLPj8lyd0LxviufcZ4ZGvtb5YygNbaztba21prz0nyQ0l+parOX8b3AgAs3bTzfPLkt58sdu6/J8lJtSChD+2NuTPJKVNuIt63z/uT7EryggV9HtNaO2rYfs/I97C/5KADTPhfvs8meWS4yeaIqpqpqm+tqvOG7VuSPJLk0ap6XpI3L7OfLUkeTfJQVZ2U5NKRfX6pqp5ZVU/L5CLjiqH+oSQXVdVLauJbqurVVbVlKQMYbpY5ffjP4pFM7meYW+SQjVW1ecHHzCL7AgCLm3aeH7PYuf/6JLNJ3lqThUtek8nbe8Z8NpPQ/p6hjc1V9R3DtvuSPHO4hyCttfmh3z8cZupTVSctuA/y75JcWFVnVdWRSd6xhO99Neagw5rwv0yttblMrv5emOS2TK56/zzJMcMulyT5qSQ7M3nwLfZEXcxvJzk3ycNJPpnkqpF9PprkU0m+Nny8cxjjf2TyfrfLMnmJ7NYkFy5jDGck+XQmT77rk/xJa+3aRfb/YiYzAE98/Nwy+gQAJkbP82MWO/e31vYkec3w9YNJXpvxXLEw55ye5OuZ3PD62mHzZzI5199bVfcPtbcPfd1QVY9kkhueO7T1T0neNxx36/Dv/lqNOeiwVqv4fgUAgDWtqm7P5A9qfXqlx8LaYOYfAAA6IfwDAEAnvO0HAAA6YeYfAAA6IfwDAEAnDulf+H3Vuh/3HiNWnWvmr1zOH2cDYA2QXViNFssuZv4BAKATwj8AAHRC+AcAgE4I/wAA0AnhHwAAOiH8AwBAJ4R/AADohPAPAACdEP4BAKATwj8AAHRi/UoPgINr3ZFHjtbrpBOmHtPuvm+0Pv/YYwdkTAAA08guB5eZfwAA6ITwDwAAnRD+AQCgE8I/AAB0QvgHAIBOCP8AANAJS32uFetmRss7fuyc0fqpb7plalM3XXf2aP3MP7pttD57z71PMTgAgH3ILivCzD8AAHRC+AcAgE4I/wAA0AnhHwAAOiH8AwBAJ6z2s0bUzPgd8w+c3Ubr/3zK1VPb+ugPf2W0/okrvnP8gI7vmAcAlkd2WRlm/gEAoBPCPwAAdEL4BwCATgj/AADQCeEfAAA6YbWfNa5NubzbUNOv+zbX3oM0GgCAxckuB5eZfwAA6ITwDwAAnRD+AQCgE8I/AAB0QvgHAIBOWO1njajNm0br80fNjdbXue4DAFaQ7LIy/BQBAKATwj8AAHRC+AcAgE4I/wAA0AnhHwAAOmG1n1WmNmwcr5+ybbT++hffMFrfVH71AMDBJ7scXsz8AwBAJ4R/AADohPAPAACdEP4BAKATwj8AAHRC+AcAgE5YM2mNmN84/qs8ddP2QzwSAICnJrusDDP/AADQCeEfAAA6IfwDAEAnhH8AAOiE8A8AAJ2w2s9qs67G6zPj9ZmaP4iDAQB4CrLLYcXMPwAAdEL4BwCATgj/AADQCeEfAAA6IfwDAEAnrPazyswcv3W0ft85W0brZ236xrSWpvaxt03fBgCwFLLL4cXMPwAAdEL4BwCATgj/AADQCeEfAAA6IfwDAEAnrPazyswdf8xo/cEXtNH6WRvmRuuzi/Txka+/ZLR+7P0PL7ktAKBvssvhxcw/AAB0QvgHAIBOCP8AANAJ4R8AADoh/AMAQCeEfwAA6ISlPg9TtX78V3P/uUeP1k88677R+oaaGa3vmNs9te/7b3zGaH3LvTdNPQYA6JvssjqY+QcAgE4I/wAA0AnhHwAAOiH8AwBAJ4R/AADohNV+DlMzJ54wWl//I9tH6+8982Pj+2f8jvm/evhFU/s+5V92jdbb3j1TjwEA+ia7rA5m/gEAoBPCPwAAdEL4BwCATgj/AADQCeEfAAA6YbWfw9Tc8ceM1t9w6v8brZ+zcfxu9t1t/PruA9e/cmrfz7v5y6P1+alHAAC9kzlsvrcAACAASURBVF1WBzP/AADQCeEfAAA6IfwDAEAnhH8AAOiE8A8AAJ2w2s8KqvXTf/z3n3v0aP2Fm+8YrW+omdH63jY3Wp/ZOb5/krTHd0/dBgD0S3ZZ/cz8AwBAJ4R/AADohPAPAACdEP4BAKATwj8AAHTCaj8raObEE6ZuW/8j20fr52wc339TbRitf273+PXdsV+qqX23ufG77AGAvskuq5+ZfwAA6ITwDwAAnRD+AQCgE8I/AAB0QvgHAIBOCP8AANAJS32uoN2nHz9121tO+8fR+vrMjNb3tvElrj7+8ItG61tvfHBq3/PzlssCAJ5Mdln9zPwDAEAnhH8AAOiE8A8AAJ0Q/gEAoBPCPwAAdMJqP4dAbdg4Wr/z/E1Tj/meI+4Yrc9m/Jib94y388kPv3y0vu3Wz0/tGwDom+yydpn5BwCATgj/AADQCeEfAAA6IfwDAEAnhH8AAOiE1X4Ogdo8fpf7nq2zU4/Zsm78V7O3zY3Wv7LnpNH6cV/YPVqf37Vrat8AQN9kl7XLzD8AAHRC+AcAgE4I/wAA0AnhHwAAOiH8AwBAJ6z2cwjMnnPaaP2ib7926jFH1MbR+g3jN8Dnj7/63aP1p//vfeNjam1q3wBA32SXtcvMPwAAdEL4BwCATgj/AADQCeEfAAA6IfwDAEAnrPZzANWG8bvc7/y+I0brP3PM56a2NZtNo/VLbnnd+P5XbR2tz91z49Q+AIC+yS79MfMPAACdEP4BAKATwj8AAHRC+AcAgE4I/wAA0AnhHwAAOmGpz0Ng71FttL5l3fQf/817prR11fGj9Wdcc9dofXZ2dvHBAQDsQ3ZZu8z8AwBAJ4R/AADohPAPAACdEP4BAKATwj8AAHTCaj+HwIad49dYO+an383+hcefPVo/7nM7R+tz9/7fkscFADBGdlm7zPwDAEAnhH8AAOiE8A8AAJ0Q/gEAoBPCPwAAdMJqPwdQ27tntH7qlTtG6xec9eapbZ1x/PbR+rpde0fr8/PtKUYHAPDNZJf+mPkHAIBOCP8AANAJ4R8AADoh/AMAQCeEfwAA6ITVfg6B+S/eMlp/zsXbph4ze/TR42199fbR+rS79QEAlkp2WbvM/AMAQCeEfwAA6ITwDwAAnRD+AQCgE8I/AAB0QvgHAIBOWOrzUGhttDx71zcO8UAAAPaD7LJmmfkHAIBOCP8AANAJ4R8AADoh/AMAQCeEfwAA6ITwDwAAnRD+AQCgE8I/AAB0QvgHAIBOCP8AANAJ4R8AADpRrbWVHgMAAHAImPkHAIBOCP8AANAJ4R8AADoh/AMAQCeEfwAA6ITwDwAAnRD+AQCgE8I/AAB0QvgHAIBOCP8AANAJ4R8AADoh/AMAQCeEfwAA6ITwDwAAnRD+AQCgE8I/AAB0QvgHAIBOCP8AANAJ4X+FVFWrqtOnbLu2qt64n+3cXlXfu8wxTD22qj5QVb+xnHYBgNWhqv6yqt45fP6KqrrlEPUrB62Q9Ss9AA5PrbWLlnNcVbUkZ7TWbj3AQwIADqLW2nVJnvtU+1XVhUne2Fp7+UEf1ApZyznIzP8yVJWLJgDgsCKfsD+E//00vDT09qr6rySPVdX6qtpWVX9fVdur6raqeuuC/V9cVddX1UNVdU9VXVZVG5fR72lV9ZmqeqCq7q+qv66qY/fZ7byq+lJVPVhVl1fV5gXH/2BVfX4Yx79V1dn72e/ClwGPq6qrhzZ2VNV1VeWxAwAH2ZA/fnXsPF9Vr6yqu4Z8cm+Sy4f61HN/Vb2oqv6zqnZW1RVJFmaGV1bVXQu+PrmqrhpyzgNDlnl+kg8keVlVPVpVDw37bqqq36+qr1fVfcPbZo5Y0NalQx66u6resITvXw46wFbtwFfITyZ5dZJjk8wn+USSm5OclOT8JL9cVRcM+84luTjJcUleNmz/xWX0WUnenWRbkucnOTnJb+2zz+uTXJDktCRnJvn1JKmqc5P8RZI3JXl6kg8m+XhVbVriGN6W5K4kW5M8I8mvJWlL/1YAgGUYPc8PTkjytCTPSvILi537h0nIf0jykeGYK5P86FiHVTWT5OokdyR5diZZ529ba/+T5KIk17fWjmqtPRHE3zuM7YVJTh/2/82hre9PckmSVyU5I8lS3qMvBx1gwv/SvL+1dmdrbVeS85Jsba39TmttT2vta0k+lOR1SdJau6m1dkNrbba1dnsmD7jvWmqHrbVbW2vXtNZ2t9a2J/mDkXYuG8a1I8m7MrlISZKfT/LB1tq/t9bmWmsfTrI7yUuXOIy9SU5M8qzW2t7W2nWttVX7oAeAVWbaeT6ZTEa+Y8gJu7L4uf+lSTYked9wPv9Ykhun9PniTAL3pa21x1prj7fW/nVsx6qqod+LW2s7Wms7k/xuhkyU5CeSXN5a++/W2mN5cnifSg468Lw3bGnuXPD5s5Jse+LlrsFMkuuSpKrOzOQB+m1JjszkZ33TUjusquOTvD/JK5JsyeSC7cFFxnVHJk/WJ8b4s1X1lgXbNy7Yvr9+L5Mn6qcmz+/8WWvtPUtsAwBYnmnn+STZ3lp7fMHXi537W5Jv7BNc75jS58lJ7mitze7H+LZmknVuGnJCMpmxnxk+35ZvzkDT+nwSOejAM/O/NAufLHcmua21duyCjy2ttR8Ytv9pki9ncsf30Zm8RFRZuncP/Z49tPPTI+2cvODzU5LcvWCM79pnjEe21v5mKQNore1srb2ttfacJD+U5Feq6vxlfC8AwNJNO88nT377yWLn/nuSnFQLEvrQ3pg7k5wy5Sbiffu8P8muJC9Y0OcxrbWjhu33jHwP+0sOOsCE/+X7bJJHhptsjqiqmar61qo6b9i+JckjSR6tquclefMy+9mS5NEkD1XVSUkuHdnnl6rqmVX1tEwuMq4Y6h9KclFVvaQmvqWqXl1VW5YygOFmmdOH/yweyeR+hrlFDtlYVZsXfMwssi8AsLhp5/kxi537r08ym+StNVm45DWZvL1nzGczCe3vGdrYXFXfMWy7L8kzh3sI0lqbH/r9w2GmPlV10oL7IP8uyYVVdVZVHZnkHUv43ldjDjqsCf/L1Fqby+Tq74VJbsvkqvfPkxwz7HJJkp9KsjOTB99iT9TF/HaSc5M8nOSTSa4a2eejST6V5GvDxzuHMf5HJu93uyyTl8huTXLhMsZwRpJPZ/Lkuz7Jn7TWrl1k/y9mMgPwxMfPLaNPAGBi9Dw/ZrFzf2ttT5LXDF8/mOS1Gc8VC3PO6Um+nskNr68dNn8mk3P9vVV1/1B7+9DXDVX1SCa54blDW/+U5H3DcbcO/+6v1ZiDDmu1iu9XAABY06rq9kz+oNanV3osrA1m/gEAoBPCPwAAdMLbfgAAoBNm/gEAoBPCPwAAdOKQ/oXfV637ce8xYtW5Zv7K5fxxNgDWANmF1Wix7GLmHwAAOiH8AwBAJ4R/AADohPAPAACdEP4BAKATwj8AAHRC+AcAgE4I/wAA0AnhHwAAOiH8AwBAJ9av9AA4uNYdeeRovU46Yeox7e77Ruvzjz12QMYEADCN7HJwmfkHAIBOCP8AANAJ4R8AADoh/AMAQCeEfwAA6ITwDwAAnbDU51qxbma0vOPHzhmtn/qmW6Y2ddN1Z4/Wz/yj20brs/fc+xSDAwDYh+yyIsz8AwBAJ4R/AADohPAPAACdEP4BAKATwj8AAHTCaj9rRM2M3zH/wNlttP7Pp1w9ta2P/vBXRuufuOI7xw/o+I55AGB5ZJeVYeYfAAA6IfwDAEAnhH8AAOiE8A8AAJ0Q/gEAoBNW+1nj2pTLuw01/bpvc+09SKMBAFic7HJwmfkHAIBOCP8AANAJ4R8AADoh/AMAQCeEfwAA6ITVftaI2rxptD5/1NxofZ3rPgBgBckuK8NPEQAAOiH8AwBAJ4R/AADohPAPAACdEP4BAKATVvtZZWrDxvH6KdtG669/8Q2j9U3lVw8AHHyyy+HFzD8AAHRC+AcAgE4I/wAA0AnhHwAAOiH8AwBAJ4R/AADohDWT1oj5jeO/ylM3bT/EIwEAeGqyy8ow8w8AAJ0Q/gEAoBPCPwAAdEL4BwCATgj/AADQCav9rDbrarw+M16fqfmDOBgAgKcguxxWzPwDAEAnhH8AAOiE8A8AAJ0Q/gEAoBPCPwAAdMJqP6vMzPFbR+v3nbNltH7Wpm9Ma2lqH3vb9G0AAEshuxxezPwDAEAnhH8AAOiE8A8AAJ0Q/gEAoBPCPwAAdMJqP6vM3PHHjNYffEEbrZ+1YW60PrtIHx/5+ktG68fe//CS2wIA+ia7HF7M/AMAQCeEfwAA6ITwDwAAnRD+AQCgE8I/AAB0QvgHAIBOWOrzMFXrx38195979Gj9xLPuG61vqJnR+o653VP7vv/GZ4zWt9x709RjAIC+yS6rg5l/AADohPAPAACdEP4BAKATwj8AAHRC+AcAgE5Y7ecwNXPiCaP19T+yfbT+3jM/Nr5/xu+Y/6uHXzS171P+Zddove3dM/UYAKBvssvqYOYfAAA6IfwDAEAnhH8AAOiE8A8AAJ0Q/gEAoBNW+zlMzR1/zGj9Daf+v9H6ORvH72bf3cav7z5w/Sun9v28m788Wp+fegQA0DvZZXUw8w8AAJ0Q/gEAoBPCPwAAdEL4BwCATgj/AADQCav9rKBaP/3Hf/+5R4/WX7j5jtH6hpoZre9tc6P1mZ3j+ydJe3z31G0AQL9kl9XPzD8AAHRC+AcAgE4I/wAA0AnhHwAAOiH8AwBAJ6z2s4JmTjxh6rb1P7J9tH7OxvH9N9WG0frndo9f3x37pZrad5sbv8seAOib7LL6mfkHAIBOCP8AANAJ4R8AADoh/AMAQCeEfwAA6ITwDwAAnbDU5wraffrxU7e95bR/HK2vz8xofW8bX+Lq4w+/aLS+9cYHp/Y9P2+5LADgyWSX1c/MPwAAdEL4BwCATgj/AADQCeEfAAA6IfwDAEAnrPZzCNSGjaP1O8/fNPWY7znijtH6bMaPuXnPeDuf/PDLR+vbbv381L4BgL7JLmuXmX8AAOiE8A8AAJ0Q/gEAoBPCPwAAdEL4BwCATljt5xCozeN3ue/ZOjv1mC3rxn81e9vcaP0re04arR/3hd2j9fldu6b2DQD0TXZZu8z8AwBAJ4R/AADohPAPAACdEP4BAKATwj8AAHTCaj+HwOw5p43WL/r2a6cec0RtHK3fMH4DfP74q989Wn/6/943PqbWpvYNAPRNdlm7zPwDAEAnhH8AAOiE8A8AAJ0Q/gEAoBPCPwAAdMJqPwdQbRi/y/3O7ztitP4zx3xualuz2TRav+SW143vf9XW0frcPTdO7QMA6Jvs0h8z/wAA0AnhHwAAOiH8AwBAJ4R/AADohPAPAACdEP4BAKATlvo8BPYe1UbrW9ZN//HfvGdKW1cdP1p/xjV3jdZnZ2cXHxwAwD5kl7XLzD8AAHRC+AcAgE4I/wAA0AnhHwAAOiH8AwBAJ6z2cwhs2Dl+jbVjfvrd7F94/Nmj9eM+t3O0Pnfv/y15XAAAY2SXtcvMPwAAdEL4BwCATgj/AADQCeEfAAA6IfwDAEAnrPZzALW9e0brp165Y7R+wVlvntrWGcdvH62v27V3tD4/355idAAA30x26Y+ZfwAA6ITwDwAAnRD+AQCgE8I/AAB0QvgHAIBOWO3nEJj/4i2j9edcvG3qMbNHHz3e1ldvH61Pu1sfAGCpZJe1y8w/AAB0QvgHAIBOCP8AANAJ4R8AADoh/AMAQCeEfwAA6ISlPg+F1kbLs3d94xAPBABgP8gua5aZfwAA6ITwDwAAnRD+AQCgE8I/AAB0QvgHAIBOCP8AANAJ4R8AADoh/AMAQCeEfwAA6ITwDwAAnRD+AQCgE9VaW+kxAAAAh4CZfwAA6ITwDwAAnRD+AQCgE8I/AAB0QvgHAIBOCP8AANAJ4R8AADoh/AMAQCeEfwAA6ITwDwAAnRD+AQCgE8I/AAB0QvgHAIBOCP8AANAJ4R8AADoh/AMAQCeEfwAA6ITwDwAAnRD+V0hVtao6fcq2a6vqjfvZzu1V9b3LHMPUY6vqA1X1G8tpFwBYHarqL6vqncPnr6iqWw5Rv3LQClm/0gPg8NRau2g5x1VVS3JGa+3WAzwkAOAgaq1dl+S5T7VfVV2Y5I2ttZcf9EGtkLWcg8z8L0NVuWgCAA4r8gn7Q/jfT8NLQ2+vqv9K8lhVra+qbVX191W1vapuq6q3Ltj/xVV1fVU9VFX3VNVlVbVxGf2eVlWfqaoHqur+qvrrqjp2n93Oq6ovVdWDVXV5VW1ecPwPVtXnh3H8W1WdvZ/9LnwZ8LiqunpoY0dVXVdVHjsAcJAN+eNXx87zVfXKqrpryCf3Jrl8qE8991fVi6rqP6tqZ1VdkWRhZnhlVd214OuTq+qqIec8MGSZ5yf5QJKXVdWjVfXQsO+mqvr9qvp6Vd03vG3miAVtXTrkobur6g1L+P7loANs1Q58hfxkklcnOTbJfJJPJLk5yUlJzk/yy1V1wbDvXJKLkxyX5GXD9l9cRp+V5N1JtiV5fpKTk/zWPvu8PskFSU5LcmaSX0+Sqjo3yV8keVOSpyf5YJKPV9WmJY7hbUnuSrI1yTOS/FqStvRvBQBYhtHz/OCEJE9L8qwkv7DYuX+YhPyHJB8ZjrkyyY+OdVhVM0muTnJHkmdnknX+trX2P0kuSnJ9a+2o1toTQfy9w9hemOT0Yf/fHNr6/iSXJHlVkjOSLOU9+nLQASb8L837W2t3ttZ2JTkvydbW2u+01va01r6W5ENJXpckrbWbWms3tNZmW2u3Z/KA+66ldthau7W1dk1rbXdrbXuSPxhp57JhXDuSvCuTi5Qk+fkkH2yt/Xtrba619uEku5O8dInD2JvkxCTPaq3tba1d11pbtQ96AFhlpp3nk8lk5DuGnLAri5/7X5pkQ5L3DefzjyW5cUqfL84kcF/aWnustfZ4a+1fx3asqhr6vbi1tqO1tjPJ72bIREl+IsnlrbX/bq09lieH96nkoAPPe8OW5s4Fnz8rybYnXu4azCS5Lkmq6sxMHqDfluTITH7WNy21w6o6Psn7k7wiyZZMLtgeXGRcd2TyZH1ijD9bVW9ZsH3jgu376/cyeaJ+avL8zp+11t6zxDYAgOWZdp5Pku2ttccXfL3Yub8l+cY+wfWOKX2enOSO1trsfoxvayZZ56YhJySTGfuZ4fNt+eYMNK3PJ5GDDjwz/0uz8MlyZ5LbWmvHLvjY0lr7gWH7nyb5ciZ3fB+dyUtElaV799Dv2UM7Pz3SzskLPj8lyd0LxviufcZ4ZGvtb5YygNbaztba21prz0nyQ0l+parOX8b3AgAs3bTzfPLkt58sdu6/J8lJtSChD+2NuTPJKVNuIt63z/uT7EryggV9HtNaO2rYfs/I97C/5KADTPhfvs8meWS4yeaIqpqpqm+tqvOG7VuSPJLk0ap6XpI3L7OfLUkeTfJQVZ2U5NKRfX6pqp5ZVU/L5CLjiqH+oSQXVdVLauJbqurVVbVlKQMYbpY5ffjP4pFM7meYW+SQjVW1ecHHzCL7AgCLm3aeH7PYuf/6JLNJ3lqThUtek8nbe8Z8NpPQ/p6hjc1V9R3DtvuSPHO4hyCttfmh3z8cZupTVSctuA/y75JcWFVnVdWRSd6xhO99Neagw5rwv0yttblMrv5emOS2TK56/zzJMcMulyT5qSQ7M3nwLfZEXcxvJzk3ycNJPpnkqpF9PprkU0m+Nny8cxjjf2TyfrfLMnmJ7NYkFy5jDGck+XQmT77rk/xJa+3aRfb/YiYzAE98/Nwy+gQAJkbP82MWO/e31vYkec3w9YNJXpvxXLEw55ye5OuZ3PD62mHzZzI5199bVfcPtbcPfd1QVY9kkhueO7T1T0neNxx36/Dv/lqNOeiwVqv4fgUAgDWtqm7P5A9qfXqlx8LaYOYfAAA6IfwDAEAnvO0HAAA6YeYfAAA6IfwDAEAnDulf+H3Vuh/3HiNWnWvmr1zOH2cDYA2QXViNFssuZv4BAKATwj8AAHRC+AcAgE4I/wAA0AnhHwAAOiH8AwBAJ4R/AADohPAPAACdEP4BAKATwj8AAHRC+AcAgE6sX+kBcHCtO/LI0XqddMLUY9rd943W5x977ICMCQBgGtnl4DLzDwAAnRD+AQCgE8I/AAB0QvgHAIBOCP8AANAJ4R8AADphqc+1Yt3MaHnHj50zWj/1TbdMbeqm684erZ/5R7eN1mfvufcpBgcAsA/ZZUWY+QcAgE4I/wAA0AnhHwAAOiH8AwBAJ4R/AADohNV+1oiaGb9j/oGz22j9n0+5empbH/3hr4zWP3HFd44f0PEd8wDA8sguK8PMPwAAdEL4BwCATgj/AADQCeEfAAA6IfwDAEAnrPazxrUpl3cbavp13+bae5BGAwCwONnl4DLzDwAAnRD+AQCgE8I/AAB0QvgHAIBOCP8AANAJq/2sEbV502h9/qi50fo6130AwAqSXVaGnyIAAHRC+AcAgE4I/wAA0AnhHwAAOiH8AwBAJ6z2s8rUho3j9VO2jdZf/+IbRuubyq8eADj4ZJfDi5l/AADohPAPAACdEP4BAKATwj8AAHRC+AcAgE4I/wAA0AlrJq0R8xvHf5Wnbtp+iEcCAPDUZJeVYeYfAAA6IfwDAEAnhH8AAOiE8A8AAJ0Q/gEAoBNW+1lt1tV4fWa8PlPzB3EwAABPQXY5rJj5BwCATgj/AADQCeEfAAA6IfwDAEAnhH8AAOiE1X5WmZnjt47W7ztny2j9rE3fmNbS1D72tunbAACWQnY5vJj5BwCATgj/AADQCeEfAAA6IfwDAEAnhH8AAOiE1X5WmbnjjxmtP/iCNlo/a8PcaH12kT4+8vWXjNaPvf/hJbcFAPRNdjm8mPkHAIBOCP8AANAJ4R8AADoh/AMAQCeEfwAA6ITwDwAAnbDU52Gq1o//au4/9+jR+oln3Tda31Azo/Udc7un9n3/jc8YrW+596apxwAAfZNdVgcz/wAA0AnhHwAAOiH8AwBAJ4R/AADohPAPAACdsNrPYWrmxBNG6+t/ZPto/b1nfmx8/4zfMf9XD79oat+n/Muu0Xrbu2fqMQBA32SX1cHMPwAAdEL4BwCATgj/AADQCeEfAAA6IfwDAEAnrPZzmJo7/pjR+htO/X+j9XM2jt/NvruNX9994PpXTu37eTd/ebQ+P/UIAKB3ssvqYOYfAAA6IfwDAEAnhH8AAOiE8A8AAJ0Q/gEAoBNW+1lBtX76j//+c48erb9w8x2j9Q01M1rf2+ZG6zM7x/dPkvb47qnbAIB+yS6rn5l/AADohPAPAACdEP4BAKATwj8AAHRC+AcAgE5Y7WcFzZx4wtRt639k+2j9nI3j+2+qDaP1z+0ev7479ks1te82N36XPQDQN9ll9TPzDwAAnRD+AQCgE8I/AAB0QvgHAIBOCP8AANAJ4R8AADphqc8VtPv046due8tp/zhaX5+Z0freNr7E1ccfftFofeuND07te37eclkAwJPJLqufmX8AAOiE8A8AAJ0Q/gEAoBPCPwAAdEL4BwCATljt5xCoDRtH63eev2nqMd9zxB2j9dmMH3PznvF2Pvnhl4/Wt936+al9AwB9k13WLjP/AADQCeEfAAA6IfwDAEAnhH8AAOiE8A8AAJ2w2s8hUJvH73Lfs3V26jFb1o3/ava2udH6V/acNFo/7gu7R+vzu3ZN7RsA6JvssnaZ+QcAgE4I/wAA0AnhHwAAOiH8AwBAJ4R/AADohNV+DoHZc04brV/07ddOPeaI2jhav2H8Bvj88Ve/e7T+9P+9b3xMrU3tGwDom+yydpn5BwCATgj/AADQCeEfAAA6IfwDAEAnhH8AAOiE1X4OoNowfpf7nd93xGj9Z4753NS2ZrNptH7JLa8b3/+qraP1uXtunNoHANA32aU/Zv4BAKATwj8AAHRC+AcAgE4I/wAA0AnhHwAAOiH8AwBAJyz1eQjsPaqN1resm/7jv3nPlLauOn60/oxr7hqtz87OLj44AIB9yC5rl5l/AADohPAPAACdEP4BAKATwj8AAHRC+AcAgE5Y7ecQ2LBz/Bprx/z0u9m/8PizR+vHfW7naH3u3v9b8rgAAMbILmuXmX8AAOiE8A8AAJ0Q/gEAoBPCPwAAdEL4BwCATljt5wBqe/eM1k+9csdo/YKz3jy1rTOO3z5aX7dr72h9fr49xegAAL6Z7NIfM/8AANAJ4R8AADoh/AMAQCeEfwAA6ITwDwAAnbDazyEw/8VbRuvPuXjb1GNmjz56vK2v3j5an3a3PgDAUskua5eZfwAA6ITwDwAAnRD+AQCgE8I/AAB0QvgHAIBOCP8AANAJS30eCq2Nlmfv+sYhHggAwH6QXdYsM/8AANAJ4R8AADoh/AMAQCeEfwAA6ITwDwAAnRD+AQCgE8I/AAB0QvgHAIBOCP8AANAJ4R8AADoh/AMAQCeqtbbSYwAAAA4BM/8AANAJ4R8AADoh/AMAQCeEfwAA6ITwDwAAnRD+AQCgE8I/AAB0QvgHAIBOCP8AANAJ4R8AADoh/AMAQCeEfwAA6ITwDwAAnRD+AQCgE8I/AAB0QvgHAIBOCP8AANAJ4R8AADoh/K+QqmpVdfqUbddW1Rv3s53bq+p7lzmGqcdW1Qeq6jeW0y4AsDpU1V9W1TuHz19RVbccon7loBWyfqUHwOGptXbRco6rqpbkjNbarQd4SADAQdRauy7Jc59qv6q6MMkbW2svP+iDWiFrOQeZ+V+GqnLRBAAcVuQT9ofwv5+Gl4beXlX/leSxqlpfVduq6u+rantV3VZVb12w/4ur6vqqeqiq7qmqy6pq4zL6Pa2qPlNVD1TV/VX111V17D67nVdVX6qqB6vq8qravOD4H6yqzw/j+LeqOns/+134MuBxVXX10MaOqrquqjx2AOAgG/LHr46d56vqlVV115BP7k1y+VCfeu6vqhdV1X9W1c6quiLJwszwyqq6/UWgFAAADGNJREFUa8HXJ1fVVUPOeWDIMs9P8oEkL6uqR6vqoWHfTVX1+1X19aq6b3jbzBEL2rp0yEN3V9UblvD9y0EH2Kod+Ar5ySSvTnJskvkkn0hyc5KTkpyf5Jer6oJh37kkFyc5LsnLhu2/uIw+K8m7k2xL8vwkJyf5rX32eX2SC5KcluTMJL+eJFV1bpK/SPKmJE9P8sEkH6+qTUscw9uS3JVka5JnJPm1JG3p3woAsAyj5/nBCUmeluRZSX5hsXP/MAn5D0k+MhxzZZIfHeuwqmaSXJ3kjiTPziTr/G1r7X+SXJTk+tbaUa21J4L4e4exvTDJ6cP+vzm09f1JLknyqiRnJFnKe/TloANM+F+a97fW7myt7UpyXpKtrbXfaa3taa19LcmHkrwuSVprN7XWbmitzbbWbs/kAfddS+2wtXZra+2a1tru1tr2JH8w0s5lw7h2JHlXJhcpSfLzST7YWvv31tpca+3DSXYneekSh7E3yYlJntVa29tau661tmof9ACwykw7zyeTych3DDlhVxY/9780yYYk7xvO5x9LcuOUPl+cSeC+tLX2WGvt8dbav47tWFU19Htxa21Ha21nkt/NkImS/ESSy1tr/91aeyxPDu9TyUEHnveGLc2dCz5/VpJtT7zcNZhJcl2SVNWZmTxAvy3JkZn8rG9aaodVdXyS9yd5RZItmVywPbjIuO7I5Mn6xBh/tqresmD7xgXb99fvZfJE/dTk+Z0/a629Z4ltAADLM+08nyTbW2uPL/h6sXN/S/KNfYLrHVP6PDnJHa212f0Y39ZMss5NQ05IJjP2M8Pn2/LNGWhan08iBx14Zv6XZuGT5c4kt7XWjl3wsaW19gPD9j9N8uVM7vg+OpOXiCpL9+6h37OHdn56pJ2TF3x+SpK7F4zxXfuM8cjW2t8sZQCttZ2ttbe11p6T5IeS/EpVnb+M7wUAWLpp5/nkyW8/Wezcf0+Sk2pBQh/aG3NnklOm3ES8b5/3J9mV5AUL+jymtXbUsP2eke9hf8lBB5jwv3yfTfLIcJPNEVU1U1XfWlXnDdu3JHkkyaNV9bwkb15mP1uSPJrkoao6KcmlI/v8UlU9s6qelslFxhVD/UNJLqqql9TEt1TVq6tqy1IGMNwsc/rwn8UjmdzPMLfIIRuravOCj5lF9gUAFjftPD9msXP/9Ulmk7y1JguXvCaTt/eM+Wwmof09Qxubq+o7hm33JXnmcA9BWmvzQ79/OMzUp6pOWnAf5N8lubCqzqqqI5O8Ywnf+2rMQYc14X+ZWmtzmVz9vTDJbZlc9f55kmOGXS5J8lNJdmby4FvsibqY305ybpKHk3wyyVUj+3w0yaeSfG34eOcwxv/I5P1ul2XyEtmtSS5cxhjOSPLpTJ581yf5k9batYvs/8VMZgCe+Pi5ZfQJAEyMnufHLHbub63tSfKa4esHk7w247liYc45PcnXM7nh9bXD5s9kcq6/t6ruH2pvH/q6oaoeySQ3PHdo65+SvG847tbh3/21GnPQYa1W8f0KAABrWlXdnskf1Pr0So+FtcHMPwAAdEL4BwCATnjbDwAAdMLMPwAAdOKQ/pGvV637cS8zsOpcM3/lcv4+AwBrgOzCarRYdjHzDwAAnRD+AQCgE8I/AAB0QvgHAIBOCP8AANAJ4R8AADoh/AMAQCeEfwAA6ITwDwAAnRD+AQCgE8I/AAB0Yv1KD4CDa92RR47W66QTph7T7r5vtD7/2GMHZEwAANPILgeXmX8AAOiE8A8AAJ0Q/gEAoBPCPwAAdEL4BwCATgj/AADQCUt9rhXrZkbLO37snNH6qW+6ZWpTN1139mj9zD+6bbQ+e8+9TzE4AIB9yC4rwsw/AAB0QvgHAIBOCP8AANAJ4R8AADoh/AMAQCes9rNG1Mz4HfMPnN1G6/98ytVT2/roD39ltP6JK75z/ICO75gHAJZHdlkZZv4BAKATwj8AAHRC+AcAgE4I/wAA0AnhHwAAOmG1nzWuTbm821DTr/s2196DNBoAgMXJLgeXmX8AAOiE8A8AAJ0Q/gEAoBPCPwAAdEL4B+D/t3M/L1LXcRzHZ3b2h4K7Crnrj0AyK8IOpoeC6NAPqLO3IOjQqQ4dgv6Jzl26VsdCKOgQHeqWIGEiRVZCppKmKOJh3Z0f387V+6st6OzOvB6P43u+v5i5POcD3w8AIez2MyW62xbK+WjHsJzP+N8HAGwi7bI5fIsAABBC/AMAQAjxDwAAIcQ/AACEEP8AABDCbj8Tpjs3X88P7C/nrz9zspwvdP30AMCDp122Fiv/AAAQQvwDAEAI8Q8AACHEPwAAhBD/AAAQQvwDAEAIeyZNidF8/VMeXLg25icBALg37bI5rPwDAEAI8Q8AACHEPwAAhBD/AAAQQvwDAEAIu/1MmpluPe/V81539AAfBgDgHrTLlmLlHwAAQoh/AAAIIf4BACCE+AcAgBDiHwAAQtjtZ8L0VpbL+dUji+X88MLltiu13qPftH8GALAR2mVrsfIPAAAhxD8AAIQQ/wAAEEL8AwBACPEPAAAh7PYzYYYrO8v5zaeacn54bljOB3e5xyd/PFvOd12/teFrAQDZtMvWYuUfAABCiH8AAAgh/gEAIIT4BwCAEOIfAABCiH8AAAhhq88tqjtb/zTXjy2V832Hr5bzuW6vnN8YrrXe+/qpPeV88cr3recAANm0y2Sw8g8AACHEPwAAhBD/AAAQQvwDAEAI8Q8AACHs9rNF9fbtLeezx6+V8/ef+Kw+vlO/Mf/xraOt9z7w1Wo5b/rrrecAANm0y2Sw8g8AACHEPwAAhBD/AAAQQvwDAEAI8Q8AACHs9rNFDVd2lvM3D35Tzo/M12+zrzX1/7sPv3uh9d5Pnvm5nI9azwAA0mmXyWDlHwAAQoh/AAAIIf4BACCE+AcAgBDiHwAAQtjtZxN1Z9u//uvHlsr509sulPO5bq+c95thOe/dro/vdDqd5s5a62cAQC7tMvms/AMAQAjxDwAAIcQ/AACEEP8AABBC/AMAQAi7/Wyi3r69rZ/NHr9Wzo/M18cvdOfK+em1+v/drp+6rfduhvVb9gBANu0y+az8AwBACPEPAAAhxD8AAIQQ/wAAEEL8AwBACPEPAAAhbPW5idYeW2n97J1Dn5fz2U6vnPebeourL24dLefLp2623ns0sl0WAPBf2mXyWfkHAIAQ4h8AAEKIfwAACCH+AQAghPgHAIAQdvsZg+7cfDm/+PJC6zkvbb9Qzged+pwz6/V1vvzo+XK+/7cfWu8NAGTTLtPLyj8AAIQQ/wAAEEL8AwBACPEPAAAhxD8AAISw288YdLfVb7mvLw9az1mcqX+afjMs57+uP1zOd59dK+ej1dXWewMA2bTL9LLyDwAAIcQ/AACEEP8AABBC/AMAQAjxDwAAIez2MwaDI4fK+VvPfdt6zvbufDk/Wb8A3/ng/Ivl/KFfrtbP1DSt9wYAsmmX6WXlHwAAQoh/AAAIIf4BACCE+AcAgBDiHwAAQtjt5z7qztVvuV98ZXs5f2Pn6dZrDToL5fy9c6/Vx59YLufDP0+13gMAyKZd8lj5BwCAEOIfAABCiH8AAAgh/gEAIIT4BwCAEOIfAABC2OpzDPo7mnK+ONP+9Z9Zb7nWiZVyvufrS+V8MBjc/eEAAP5Fu0wvK/8AABBC/AMAQAjxDwAAIcQ/AACEEP8AABDCbj9jMHe7/o91Y9T+NvvZO4+U892nb5fz4ZW/NvxcAAAV7TK9rPwDAEAI8Q8AACHEPwAAhBD/AAAQQvwDAEAIu/3cR01/vZwf/PRGOX/18Nut13p85Vo5n1ntl/PRqLnH0wEA/JN2yWPlHwAAQoh/AAAIIf4BACCE+AcAgBDiHwAAQtjtZwxGP54r54++u7/1nMHSUn2t87+X87a39QEANkq7TC8r/wAAEEL8AwBACPEPAAAhxD8AAIQQ/wAAEEL8AwBACFt9jkPTlOPBpctjfhAAgP9Bu0wtK/8AABBC/AMAQAjxDwAAIcQ/AACEEP8AABBC/AMAQAjxDwAAIcQ/AACEEP8AABBC/AMAQAjxDwAAIbpN02z2MwAAAGNg5R8AAEKIfwAACCH+AQAghPgHAIAQ4h8AAEKIfwAACCH+AQAghPgHAIAQ4h8AAEKIfwAACCH+AQAghPgHAIAQ4h8AAEKIfwAACCH+AQAghPgHAIAQ4h8AAEKIfwAACCH+AQAghPgHAIAQ4h8AAEKIfwAACCH+AQAgxN9vaexII4lk2gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 1440x2160 with 16 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# get 8 random mis-classified index to display\n",
    "display_index =[]\n",
    "for i in range(8):\n",
    "    n = random.randint(0,len(miscla_index))\n",
    "    index = miscla_index[n]\n",
    "    display_index.append(index)\n",
    "print(\"displaying indexes are: \", display_index)\n",
    "\n",
    "# display misclassification letters\n",
    "fig, axs = plt.subplots(8,2, figsize=(20, 30))\n",
    "fig.subplots_adjust(hspace = 0.20, wspace=.005)\n",
    "axs = axs.ravel()\n",
    "for i in range(8):\n",
    "    for j,value in enumerate(display_index):\n",
    "        image = x_test[value].squeeze()\n",
    "        axs[2*i].axis('off')\n",
    "        axs[2*i].imshow(image)\n",
    "        axs[2*i].set_title('real label is {}'.format(y_labels[y_real_label[value]-1]))\n",
    "\n",
    "        axs[2*i+1].axis('off')\n",
    "        axs[2*i+1].imshow(image)\n",
    "        axs[2*i+1].set_title('predicted label is {}'.format(y_labels[y_pred[value]-1]))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### display some mis-classified images randomly (part 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "displaying indexes are:  [8810, 5290, 19682, 9047, 5855, 5136, 8905, 6807]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAv8AAAaOCAYAAADFylJVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzs3XuMZ2V9P/D3h2GXlbJCEZSyIl5AES8/K8VL1JTGEpvSxnrF1iZSY1taU60VYtrYWhqsNm2qsaZKtaLpxVKsNa1NUzWEBC1oobVUqbZUwAW5rYAsiuzszPP743tIZpfznZ0Zdncuz+uVTJj5nHOe5/l+d76c93m+3/NMtdYCAABsfIes9gAAAICDQ/gHAIBOCP8AANAJ4R8AADoh/AMAQCeEfwAA6ITwv8qqqlXVSVO2XV5Vb1hiOzdW1Y+vcAxTj62qD1bVb6+kXQBg7aqqj1bVhcP3L6qqrx+kfmWfVXToag+Ata21du5KjquqluTk1tr1+3lIAMB+1lq7IslT9rVfVZ2T5A2ttRce8EGtko2efcz8PwxV5eIJAFh1MglLJfwv0/A20duq6tok362qQ6vq+Kr6u6q6s6puqKo3Ldj/OVV1ZVXdU1W3VtX7q2rzCvp9UlVdVlXfrqodVfVXVXXUXrudXlXXVdXdVXVxVW1ZcPxPVdWXh3H8a1U9c4n9LnxL8Jiq+vTQxl1VdUVV+R0CgANgyBy/OXZur6ozqurmIZPcluTioT71fF9VP1xV/15VO6vqkiQLc8IZVXXzgp9PqKpPDtnm20N+eWqSDyZ5flXdV1X3DPseVlV/VFXfrKrbh4/NPGJBW+cPGehbVfX6ZTx+2ecAWNeDX0U/m+SsJEclmU/yj0n+M8m2JC9O8utV9ZJh37kkb0lyTJLnD9t/dQV9VpJ3JTk+yVOTnJDkd/fa57VJXpLkSUmenOTtSVJVz07ykSS/nORRSS5K8g9Vddgyx/DWJDcnOTbJY5L8VpK2/IcCACzR6Ll9cFySo5OcmOSXFjvfDxOPn0ryF8MxlyZ5xViHVTWT5NNJbkry+Ezyzd+01v47yblJrmytHdFaezCI/8EwtmclOWnY/3eGtn4iyXlJzkxycpLlfEZf9jkAhP+VeV9rbXtr7f4kpyc5trX2e621Xa21byT5UJLXJElr7ZrW2lWttd2ttRsz+eX70eV22Fq7vrX22dbaA621O5P88Ug77x/GdVeSd2ZykZIkv5jkotbaF1trc621jyV5IMnzljmM2SQ/lOTE1tpsa+2K1tq6fgEAwBo37dyeTCYg3zFkg/uz+Pn+eUk2JXnvcA7/RJJ/m9LnczIJ3Oe31r7bWvt+a+3zYztWVQ39vqW1dldrbWeS38+Qg5K8OsnFrbWvtNa+m4eG96lknwPD58NWZvuC709McvyDb30NZpJckSRV9eRMfll/JMnhmTzn1yy3w6p6dJL3JXlRkq2ZXLjdvci4bsrkhfvgGF9XVb+2YPvmBduX6g8zedF+ZvJaz5+11t69zDYAgKWbdm5Pkjtba99f8PNi5/uW5Ja9gutNU/o8IclNrbXdSxjfsZnkm2uGbJBMZuxnhu+Pz565Z1qfDyH7HBhm/ldm4Qtne5IbWmtHLfja2lr7yWH7B5J8LZO7vx+ZydtFleV719DvM4d2fn6knRMWfP+4JN9aMMZ37jXGw1trH1/OAFprO1trb22tPTHJTyf5jap68QoeCwCwNNPO7clDP36y2Pn+1iTbakFCH9obsz3J46bcRLx3nzuS3J/kaQv6PLK1dsSw/daRx7BUss8BIPw/fF9Kcu9ww80jqmqmqp5eVacP27cmuTfJfVV1SpJfWWE/W5Pcl+SeqtqW5PyRfd5YVY+tqqMzuci4ZKh/KMm5VfXcmviBqjqrqrYuZwDDjTMnDf/juDeT+xnmVvh4AIB9m3ZuH7PY+f7KJLuTvKkmi5W8PJOP94z5Uiah/d1DG1uq6gXDttuTPHa4hyCttfmh3/cMM/Wpqm0L7n382yTnVNWpVXV4kncs47Gvx+yzeXi+HvyaWWTfVSH8P0yttblMrgSfleSGTK6AP5zkyGGX85L8XJKdmfwiLvaiXcwFSZ6d5DtJ/inJJ0f2+eskn0nyjeHrwmGMV2fy2bf3Z/J22fVJzlnBGE5O8rlMXohXJvnT1trlK2gHAFia0XP7mMXO9621XUlePvx8d5KzM54lFmabk5J8M5MbXs8eNl+W5KtJbquqHUPtbUNfV1XVvZlkhacMbf1zkvcOx10//Hep1mP2+Wom74Q8+PULK+jzgKp1fs8CAMCGVFU3ZvIHtT632mNh4zDzDwAAnRD+AQCgEz72AwAAnTDzDwAAnRD+AQCgEwf1L/yeecirfMaIdeez85eu5I+yAbAByC6sR4tlFzP/AADQCeEfAAA6IfwDAEAnhH8AAOjEQb3hlz3VodOf/rZ790EcCQDAvsku65+ZfwAA6ITwDwAAnRD+AQCgE8I/AAB0QvgHAIBOWO3nIDj0sdtG6zv/fPPUYw678KjR+iFX/Md+GRMAwDSyy8Zl5h8AADoh/AMAQCeEfwAA6ITwDwAAnRD+AQCgE8I/AAB0wlKfB8HcseNLX33slIumHnPWm88drZ/whZnxA+bnlj0uAIAxssvGZeYfAAA6IfwDAEAnhH8AAOiE8A8AAJ0Q/gEAoBNW+1lFx85Mf/o/etrFo/ULtr1stL57+837ZUwAANPILuufmX8AAOiE8A8AAJ0Q/gEAoBPCPwAAdEL4BwCATljt5yCY2fGd0fp1szNTjzl109xo/YGTHj3ehzvmAYD9RHbZuMz8AwBAJ4R/AADohPAPAACdEP4BAKATwj8AAHTCaj8Hwe5bbh2tv+XrZ0895rJnXDJa/7/XjN9lf8rVW0fr8zt37mN0AAB7kl02LjP/AADQCeEfAAA6IfwDAEAnhH8AAOiE8A8AAJ2w2s/BMD83Wt5x9WOmHnLXqQ+M1l/73KtG69c84WnjDV37tcXHBgCwN9llwzLzDwAAnRD+AQCgE8I/AAB0QvgHAIBOCP8AANAJ4R8AADphqc9V9IRP7Zy67V9e+cTR+huPvnK0/mOvfOFo/cRrlz8uAIAxssv6Z+YfAAA6IfwDAEAnhH8AAOiE8A8AAJ0Q/gEAoBNW+1lNc22RTePXZVsPGf8nmz1ielsAAPuF7LLumfkHAIBOCP8AANAJ4R8AADoh/AMAQCeEfwAA6ITVftaoueVel7mMAwBWkeyyPnjaAQCgE8I/AAB0QvgHAIBOCP8AANAJ4R8AADphtZ9VNLPjO1O3feym543WX/eMm0brx51yx3hDh8yM1+fnFh0bAMDeZJf1z8w/AAB0QvgHAIBOCP8AANAJ4R8AADoh/AMAQCeEfwAA6ISlPlfR3G1TlrhK8u0vnTZaP/QZ48tfvecpl4zWL9j2stH67u0372N0AAB7kl3WPzP/AADQCeEfAAA6IfwDAEAnhH8AAOiE8A8AAJ2w2s8qarO7pm7bdvn4tgdev3u0fuqm+dH63DFHjnfgjnkAYJlkl/XPzD8AAHRC+AcAgE4I/wAA0AnhHwAAOiH8AwBAJ6z2s0Zt+d/bR+tfma3R+tM3jV/Hzf7gltG6f3gAYH+SXdYHM/8AANAJ4R8AADoh/AMAQCeEfwAA6ITwDwAAnXDj9Bo1d8edo/XrHtg2Wj9t822j9VvO2Dxaf/wV4/UkabO79jE6AIA9yS7rg5l/AADohPAPAACdEP4BAKATwj8AAHRC+AcAgE5Y7Wetmm+j5bk2fr02U+P12SPG2wEA2K9kl3XBzD8AAHRC+AcAgE4I/wAA0AnhHwAAOiH8AwBAJ4R/AADohKU+15n/+f5xo/XZR94yWp/fOjdary2HTe2jze5a/sAAAEbILmuLmX8AAOiE8A8AAJ0Q/gEAoBPCPwAAdEL4BwCATljtZ42adtf61eedNlr/+w/cMFr/yxdfNFp/89lvnNr3oz585T5GBwCwJ9llfTDzDwAAnRD+AQCgE8I/AAB0QvgHAIBOCP8AANAJq/2sM4fddt9o/drvPW60/tJjd4zWN73ijumdfGRmvD4/t+jYAAD2JrusLWb+AQCgE8I/AAB0QvgHAIBOCP8AANAJ4R8AADphtZ91Zv76G0frH7/8BaP1C1/9X6P1Nzz+C1P7uHRm/O775o55AGCZZJe1xcw/AAB0QvgHAIBOCP8AANAJ4R8AADoh/AMAQCes9rPezLfR8qZ7a7Q+1+YP5GgAABYnu6wpZv4BAKATwj8AAHRC+AcAgE4I/wAA0AnhHwAAOiH8AwBAJyz1uc602V2j9Sd84u7R+h3nfG+0vnXm/ql91JbDltU3AMA0ssvaYuYfAAA6IfwDAEAnhH8AAOiE8A8AAJ0Q/gEAoBNW+9kobrhltHzZ/SeO1s88/JtTm/rQ/3vZaP2Qz395+eMCABgju6wKM/8AANAJ4R8AADoh/AMAQCeEfwAA6ITwDwAAnbDazwYxv3PnaP3tl798tP7Ss/5kals3/MyW0fpJX9w8Wm+zu/YxOgCAPckuq8PMPwAAdEL4BwCATgj/AADQCeEfAAA6IfwDAEAnrPazwT36C+P/xBec/vypx2y+xzUhALA6ZJcDyzMFAACdEP4BAKATwj8AAHRC+AcAgE4I/wAA0AnhHwAAOlGttYPW2ZmHvOrgdcaiatPmqdva3Nz4hvkp9Q3us/OX1mqPAYDVIbusHbLL0i2WXcz8AwBAJ4R/AADohPAPAACdEP4BAKATwj8AAHTi0NUeAKujze5a7SEAACyZ7LJ/mPkHAIBOCP8AANAJ4R8AADoh/AMAQCeEfwAA6ES11lZ7DAAAwEFg5h8AADoh/AMAQCeEfwAA6ITwDwAAnRD+AQCgE8I/AAB0QvgHAIBOCP8AANAJ4R8AADoh/AMAQCeEfwAA6ITwDwAAnRD+AQCgE8I/AAB0QvgHAIBOCP8AANAJ4R8AADoh/AMAQCeE/1VWVa2qTpqy7fKqesMS27mxqn58hWOYemxVfbCqfnsl7QIAa1dVfbSqLhy+f1FVff0g9Sv7rKJDV3sArG2ttXNXclxVtSQnt9au389DAgD2s9baFUmesq/9quqcJG9orb3wgA9qlWz07GPm/2GoKhdPAMCqk0lYKuF/mYa3id5WVdcm+W5VHVpVx1fV31XVnVV1Q1W9acH+z6mqK6vqnqq6tareX1WbV9Dvk6rqsqr6dlXtqKq/qqqj9trt9Kq6rqrurqqLq2rLguN/qqq+PIzjX6vqmUvsd+FbgsdU1aeHNu6qqiuqyu8QABwAQ+b4zbFze1WdUVU3D5nktiQXD/Wp5/uq+uGq+veq2llVlyRZmBPOqKqbF/x8QlV9csg23x7yy1OTfDDJ86vqvqq6Z9j3sKr6o6r6ZlXdPnxs5hEL2jp/yEDfqqrXL+Pxyz4HwLoe/Cr62SRnJTkqyXySf0zyn0m2JXlxkl+vqpcM+84leUuSY5I8f9j+qyvos5K8K8nxSZ6a5IQkv7vXPq9N8pIkT0ry5CRvT5KqenaSjyT55SSPSnJRkn+oqsOWOYa3Jrk5ybFJHpPkt5K05T8UAGCJRs/tg+OSHJ3kxCS/tNj5fph4/FSSvxiOuTTJK8Y6rKqZJJ9OclOSx2eSb/6mtfbfSc5NcmVr7YjW2oNB/A+GsT0ryUnD/r8ztPUTSc5LcmaSk5Ms5zP6ss8BIPyvzPtaa9tba/cnOT3Jsa2132ut7WqtfSPJh5K8Jklaa9e01q5qre1urd2YyS/fjy63w9ba9a21z7bWHmit3Znkj0faef8wrruSvDOTi5Qk+cUkF7XWvtham2utfSzJA0met8xhzCb5oSQnttZmW2tXtNbW9QsAANa4aef2ZDIB+Y4hG9yfxc/3z0uyKcl7h3P4J5L825Q+n5NJ4D6/tfbd1tr3W2ufH9uxqmro9y2ttbtaazuT/H6GHJTk1Ukubq19pbX23Tw0vE8l+xwYPh+2MtsXfH9ikuMffOtrMJPkiiSpqidn8sv6I0kOz+Q5v2a5HVbVo5O8L8mLkmzN5MLt7kXGdVMmL9wHx/i6qvq1Bds3L9i+VH+YyYv2M5PXev6stfbuZbYBACzdtHN7ktzZWvv+gp8XO9+3JLfsFVxvmtLnCUluaq3tXsL4js0k31wzZINkMmM/M3x/fPbMPdP6fAjZ58Aw878yC18425Pc0Fo7asHX1tbaTw7bP5Dka5nc/f3ITN4uqizfu4Z+nzm08/Mj7Zyw4PvHJfnWgjG+c68xHt5a+/hyBtBa29lae2tr7YlJfjrJb1TVi1fwWACApZl2bk8e+vGTxc73tybZVgsS+tDemO1JHjflJuK9+9yR5P4kT1vQ55GttSOG7beOPIalkn0OAOH/4ftSknuHG24eUVUzVfX0qjp92L41yb1J7quqU5L8ygr72ZrkviT3VNW2JOeP7PPGqnpsVR2dyUXGJUP9Q0nOrarn1sQPVNVZVbV1OQMYbpw5afgfx72Z3M8wt8LHAwDs27Rz+5jFzvdXJtmd5E01Wazk5Zl8vGfMlzIJ7e8e2thSVS8Ytt2e5LHDPQRprc0P/b5nmKlPVW1bcO/j3yY5p6pOrarDk7xjGY99PWafzcPz9eDXzCL7rgrh/2Fqrc1lciX4rCQ3ZHIF/OEkRw67nJfk55LszOQXcbEX7WIuSPLsJN9J8k9JPjmyz18n+UySbwxfFw5jvDqTz769P5O3y65Pcs4KxnByks9l8kK8MsmfttYuX0E7AMDSjJ7bxyx2vm+t7Ury8uHnu5OcnfEssTDbnJTkm5nc8Hr2sPmyJF9NcltV7Rhqbxv6uqqq7s0kKzxlaOufk7x3OO764b9LtR6zz1czeSfkwa9fWEGfB1St83sWAAA2pKq6MZM/qPW51R4LG4eZfwAA6ITwDwAAnfCxHwAA6ISZfwAA6ITwDwAAnTiof+H3zENe5TNGrDufnb90JX+UDYANQHZhPVosu5j5BwCATgj/AADQCeEfAAA6IfwDAEAnhH8AAOjEQV3thz3VodOf/rZ790EcCQDAvsku65+ZfwAA6ITwDwAAnRD+AQCgE8I/AAB0QvgHAIBOWO3nIDj0sdtG6zv/fPPUYw678KjR+iFX/Md+GRMAwDSyy8Zl5h8AADoh/AMAQCeEfwAA6ITwDwAAnRD+AQCgE8I/AAB0wlKfB8HcseNLX33slIumHnPWm88drZ/whZnxA+bnlj0uAIAxssvGZeYfAAA6IfwDAEAnhH8AAOiE8A8AAJ0Q/gEAoBNW+1lFx85Mf/o/etrFo/ULtr1stL57+837ZUwAANPILuufmX8AAOiE8A8AAJ0Q/gEAoBPCPwAAdEL4BwCATljt5yCY2fGd0fp1szNTjzl109xo/YGTHj3ehzvmAYD9RHbZuMz8AwBAJ4R/AADohPAPAACdEP4BAKATwj8AAHTCaj8Hwe5bbh2tv+XrZ0895rJnXDJa/7/XjN9lf8rVW0fr8zt37mN0AAB7kl02LjP/AADQCeEfAAA6IfwDAEAnhH8AAOiE8A8AAJ2w2s/BMD83Wt5x9WOmHnLXqQ+M1l/73KtG69c84WnjDV37tcXHBgCwN9llwzLzDwAAnRD+AQCgE8I/AAB0QvgHAIBOCP8AANAJ4R8AADphqc9V9IRP7Zy67V9e+cTR+huPvnK0/mOvfOFo/cRrlz8uAIAxssv6Z+YfAAA6IfwDAEAnhH8AAOiE8A8AAJ0Q/gEAoBNW+1lNc22RTePXZVsPGf8nmz1ielsAAPuF7LLumfkHAIBOCP8AANAJ4R8AADoh/AMAQCeEfwAA6ITVftaoueVel7mMAwBWkeyyPnjaAQCgE8I/AAB0QvgHAIBOCP8AANAJ4R8AADphtZ9VNLPjO1O3feym543WX/eMm0brx51yx3hDh8yM1+fnFh0bAMDeZJf1z8w/AAB0QvgHAIBOCP8AANAJ4R8AADoh/AMAQCeEfwAA6ISlPlfR3G1TlrhK8u0vnTZaP/QZ48tfvecpl4zWL9j2stH67u0372N0AAB7kl3WPzP/AADQCeEfAAA6IfwDAEAnhH8AAOiE8A8AAJ2w2s8qarO7pm7bdvn4tgdev3u0fuqm+dH63DFHjnfgjnkAYJlkl/XPzD8AAHRC+AcAgE4I/wAA0AnhHwAAOiH8AwBAJ6z2s0Zt+d/bR+tfma3R+tM3jV/Hzf7gltG6f3gAYH+SXdYHM/8AANAJ4R8AADoh/AMAQCeEfwAA6ITwDwAAnXDj9Bo1d8edo/XrHtg2Wj9t822j9VvO2Dxaf/wV4/UkabO79jE6AIA9yS7rg5l/AADohPAPAACdEP4BAKATwj8AAHRC+AcAgE5Y7Wetmm+j5bk2fr02U+P12SPG2wEA2K9kl3XBzD8AAHRC+AcAgE4I/wAA0AnhHwAAOiH8AwBAJ4R/AADohKU+15n/+f5xo/XZR94yWp/fOjdary2HTe2jze5a/sAAAEbILmuLmX8AAOiE8A8AAJ0Q/gEAoBPCPwAAdEL4BwCATljtZ42adtf61eedNlr/+w/cMFr/yxdfNFp/89lvnNr3oz585T5GBwCwJ9llfTDzDwAAnRD+AQCgE8I/AAB0QvgHAIBOCP8AANAJq/2sM4fddt9o/drvPW60/tJjd4zWN73ijumdfGRmvD4/t+jYAAD2JrusLWb+AQCgE8I/AAB0QvgHAIBOCP8AANAJ4R8AADphtZ91Zv76G0frH7/8BaP1C1/9X6P1Nzz+C1P7uHRm/O775o55AGCZZJe1xcw/AAB0QvgHAIBOCP8AANAJ4R8AADoh/AMAQCes9rPezLfR8qZ7a7Q+1+YP5GgAABYnu6wpZv4BAKATwj8AAHRC+AcAgE4I/wAA0AnhHwAAOiH8AwBAJyz1uc602V2j9Sd84u7R+h3nfG+0vnXm/ql91JbDltU3AMA0ssvaYuYfAAA6IfwDAEAnhH8AAOiE8A8AAJ0Q/gEAoBNW+9kobrhltHzZ/SeO1s88/JtTm/rQ/3vZaP2Qz395+eMCABgju6wKM/8AANAJ4R8AADoh/AMAQCeEfwAA6ITwDwAAnbDazwYxv3PnaP3tl798tP7Ss/5kals3/MyW0fpJX9w8Wm+zu/YxOgCAPckuq8PMPwAAdEL4BwCATgj/AADQCeEfAAA6IfwDAEAnrPazwT36C+P/xBec/vypx2y+xzUhALA6ZJcDyzMFAACdEP4BAKATwj8AAHRC+AcAgE4I/wAA0AnhHwAAOlGttYPW2ZmHvOrgdcaiatPmqdva3Nz4hvkp9Q3us/OX1mqPAYDVIbusHbLL0i2WXcz8AwBAJ4R/AADohPAPAACdEP4BAKATwj8AAHTi0NUeAKujze5a7SEAACyZ7LJ/mPkHAIBOCP8AANAJ4R8AADoh/AMAQCeEfwAA6ES11lZ7DAAAwEFg5h8AADoh/AMAQCeEfwAA6ITwDwAAnRD+AQCgE8I/AAB0QvgHAIBOCP8AANAJ4R8AADoh/AMAQCeEfwAA6ITwDwAAnRD+AQCgE8I/AAB0QvgHAIBOCP8AANAJ4R8AADoh/AMAQCeE/1VWVa2qTpqy7fKqesMS27mxqn58hWOYemxVfbCqfnsl7QIAa1dVfbSqLhy+f1FVff0g9Sv7rKJDV3sArG2ttXNXclxVtSQnt9au389DAgD2s9baFUmesq/9quqcJG9orb3wgA9qlWz07GPm/2GoKhdPAMCqk0lYKuF/mYa3id5WVdcm+W5VHVpVx1fV31XVnVV1Q1W9acH+z6mqK6vqnqq6tareX1WbV9Dvk6rqsqr6dlXtqKq/qqqj9trt9Kq6rqrurqqLq2rLguN/qqq+PIzjX6vqmUvsd+FbgsdU1aeHNu6qqiuqyu8QABwAQ+b4zbFze1WdUVU3D5nktiQXD/Wp5/uq+uGq+veq2llVlyRZmBPOqKqbF/x8QlV9csg23x7yy1OTfDDJ86vqvqq6Z9j3sKr6o6r6ZlXdPnxs5hEL2jp/yEDfqqrXL+Pxyz4HwLoe/Cr62SRnJTkqyXySf0zyn0m2JXlxkl+vqpcM+84leUuSY5I8f9j+qyvos5K8K8nxSZ6a5IQkv7vXPq9N8pIkT0ry5CRvT5KqenaSjyT55SSPSnJRkn+oqsOWOYa3Jrk5ybFJHpPkt5K05T8UAGCJRs/tg+OSHJ3kxCS/tNj5fph4/FSSvxiOuTTJK8Y6rKqZJJ9OclOSx2eSb/6mtfbfSc5NcmVr7YjW2oNB/A+GsT0ryUnD/r8ztPUTSc5LcmaSk5Ms5zP6ss8BIPyvzPtaa9tba/cnOT3Jsa2132ut7WqtfSPJh5K8Jklaa9e01q5qre1urd2YyS/fjy63w9ba9a21z7bWHmit3Znkj0faef8wrruSvDOTi5Qk+cUkF7XWvtham2utfSzJA0met8xhzCb5oSQnttZmW2tXtNbW9QsAANa4aef2ZDIB+Y4hG9yfxc/3z0uyKcl7h3P4J5L825Q+n5NJ4D6/tfbd1tr3W2ufH9uxqmro9y2ttbtaazuT/H6GHJTk1Ukubq19pbX23Tw0vE8l+xwYPh+2MtsXfH9ikuMffOtrMJPkiiSpqidn8sv6I0kOz+Q5v2a5HVbVo5O8L8mLkmzN5MLt7kXGdVMmL9wHx/i6qvq1Bds3L9i+VH+YyYv2M5PXev6stfbuZbYBACzdtHN7ktzZWvv+gp8XO9+3JLfsFVxvmtLnCUluaq3tXsL4js0k31wzZINkMmM/M3x/fPbMPdP6fAjZ58Aw878yC18425Pc0Fo7asHX1tbaTw7bP5Dka5nc/f3ITN4uqizfu4Z+nzm08/Mj7Zyw4PvHJfnWgjG+c68xHt5a+/hyBtBa29lae2tr7YlJfjrJb1TVi1fwWACApZl2bk8e+vGTxc73tybZVgsS+tDemO1JHjflJuK9+9yR5P4kT1vQ55GttSOG7beOPIalkn0OAOH/4ftSknuHG24eUVUzVfX0qjp92L41yb1J7quqU5L8ygr72ZrkviT3VNW2JOeP7PPGqnpsVR2dyUXGJUP9Q0nOrarn1sQPVNVZVbV1OQMYbpw5afgfx72Z3M8wt8LHAwDs27Rz+5jFzvdXJtmd5E01Wazk5Zl8vGfMlzLHWu/QAAAgAElEQVQJ7e8e2thSVS8Ytt2e5LHDPQRprc0P/b5nmKlPVW1bcO/j3yY5p6pOrarDk7xjGY99PWafzcPz9eDXzCL7rgrh/2Fqrc1lciX4rCQ3ZHIF/OEkRw67nJfk55LszOQXcbEX7WIuSPLsJN9J8k9JPjmyz18n+UySbwxfFw5jvDqTz769P5O3y65Pcs4KxnByks9l8kK8MsmfttYuX0E7AMDSjJ7bxyx2vm+t7Ury8uHnu5OcnfEssTDbnJTkm5nc8Hr2sPmyJF9NcltV7Rhqbxv6uqqq7s0kKzxlaOufk7x3OO764b9LtR6zz1czeSfkwa9fWEGfB1St83sWAAA2pKq6MZM/qPW51R4LG4eZfwAA6ITwDwAAnfCxHwAA6ISZfwAA6MRB/SNfZx7yKm8zsO58dv7SlfxdBgA2ANmF9Wix7GLmHwAAOiH8AwBAJ4R/AADohPAPAACdEP4BAKATB3W1H/ZUh05/+tvu3QdxJAAA+ya7rH9m/gEAoBPCPwAAdEL4BwCATgj/AADQCeEfAAA6YbWfg+DQx24bre/8881TjznswqNG64dc8R/7ZUwAANPILhuXmX8AAOiE8A8AAJ0Q/gEAoBPCPwAAdEL4BwCATgj/AADQCUt9HgRzx44vffWxUy6aesxZbz53tH7CF2bGD5ifW/a4AADGyC4bl5l/AADohPAPAACdEP4BAKATwj8AAHRC+AcAgE5Y7WcVHTsz/en/6GkXj9Yv2Pay0fru7TfvlzEBAEwju6x/Zv4BAKATwj8AAHRC+AcAgE4I/wAA0AnhHwAAOmG1n4NgZsd3RuvXzc5MPebUTXOj9QdOevR4H+6YBwD2E9ll4zLzDwAAnRD+AQCgE8I/AAB0QvgHAIBOCP8AANAJq/0cBLtvuXW0/pavnz31mMuecclo/f9eM36X/SlXbx2tz+/cuY/RAQDsSXbZuMz8AwBAJ4R/AADohPAPAACdEP4BAKATwj8AAHTCaj8Hw/zcaHnH1Y+Zeshdpz4wWn/tc68arV/zhKeNN3Tt1xYfGwDA3mSXDcvMPwAAdEL4BwCATgj/AADQCeEfAAA6IfwDAEAnhH8AAOiEpT5X0RM+tXPqtn955RNH6288+srR+o+98oWj9ROvXf64AADGyC7rn5l/AADohPAPAACdEP4BAKATwj8AAHRC+AcAgE5Y7Wc1zbVFNo1fl209ZPyfbPaI6W0BAOwXssu6Z+YfAAA6IfwDAEAnhH8AAOiE8A8AAJ0Q/gEAoBNW+1mj5pZ7XeYyDgBYRbLL+uBpBwCATgj/AADQCeEfAAA6IfwDAEAnhH8AAOiE1X5W0cyO70zd9rGbnjdaf90zbhqtH3fKHeMNHTIzXp+fW3RsAAB7k13WPzP/AADQCeEfAAA6IfwDAEAnhH8AAOiE8A8AAJ0Q/gEAoBOW+lxFc7dNWeIqybe/dNpo/dBnjC9/9Z6nXDJav2Dby0bru7ffvI/RAQDsSXZZ/8z8AwBAJ4R/AADohPAPAACdEP4BAKATwj8AAHTCaj+rqM3umrpt2+Xj2x54/e7R+qmb5kfrc8ccOd6BO+YBgGWSXdY/M/8AANAJ4R8AADoh/AMAQCeEfwAA6ITwDwAAnbDazxq15X9vH61/ZbZG60/fNH4dN/uDW0br/uEBgP1JdlkfzPwDAEAnhH8AAOiE8A8AAJ0Q/gEAoBPCPwAAdMKN02vU3B13jtave2DbaP20zbeN1m85Y/No/fFXjNeTpM3u2sfoAAD2JLusD2b+AQCgE8I/AAB0QvgHAIBOCP8AANAJ4R8AADphtZ+1ar6Nlufa+PXaTI3XZ48YbwcAYL+SXdYFM/8AANAJ4R8AADoh/AMAQCeEfwAA6ITwDwAAnRD+AQCgE5b6XGf+5/vHjdZnH3nLaH1+69xovbYcNrWPNrtr+QMDABghu6wtZv4BAKATwj8AAHRC+AcAgE4I/wAA0AnhHwAAOmG1nzVq2l3rV5932mj97z9ww2j9L1980Wj9zWe/cWrfj/rwlfsYHQDAnmSX9cHMPwAAdEL4BwCATgj/AADQCeEfAAA6IfwDAEAnrPazzhx2232j9Wu/97jR+kuP3TFa3/SKO6Z38pGZ8fr83KJjAwDYm+yytpj5BwCATgj/AADQCeEfAAA6IfwDAEAnhH8AAOiE1X7Wmfnrbxytf/zyF4zWL3z1f43W3/D4L0zt49KZ8bvvmzvmAYBlkl3WFjP/AADQCeEfAAA6IfwDAEAnhH8AAOiE8A8AAJ2w2s96M99Gy5vurdH6XJs/kKMBAFic7LKmmPkHAIBOCP8AANAJ4R8AADoh/AMAQCeEfwAA6ITwDwAAnbDU5zrTZneN1p/wibtH63ec873R+taZ+6f2UVsOW1bfAADTyC5ri5l/AADohPAPAACdEP4BAKATwj8AAHRC+AcAgE5Y7WejuOGW0fJl9584Wj/z8G9ObepD/+9lo/VDPv/l5Y8LAGCM7LIqzPwDAEAnhH8AAOiE8A8AAJ0Q/gEAoBPCPwAAdMJqPxvE/M6do/W3X/7y0fpLz/qTqW3d8DNbRusnfXHzaL3N7trH6AAA9iS7rA4z/wAA0AnhHwAAOiH8AwBAJ4R/AADohPAPAACdsNrPBvfoL4z/E19w+vOnHrP5HteEAMDqkF0OLM8UAAB0QvgHAIBOCP8AANAJ4R8AADoh/AMAQCeEfwAA6ES11g5aZ2ce8qqD1xmLqk2bp25rc3PjG+an1De4z85fWqs9BgBWh+yydsguS7dYdjHzDwAAnRD+AQCgE8I/AAB0QvgHAIBOCP8AANCJQ1d7AKyONrtrtYcAALBkssv+YeYfAAA6IfwDAEAnhH8AAOiE8A8AAJ0Q/gEAoBPVWlvtMQAAAAeBmX8AAOiE8A8AAJ0Q/gEAoBPCPwAAdEL4BwCATgj/AADQCeEfAAA6IfwDAEAnhH8AAOiE8A8AAJ0Q/gEAoBPCPwAAdEL4BwCATgj/AADQCeEfAAA6IfwDAEAnhH8AAOiE8A8AAJ0Q/ldZVbWqOmnKtsur6g1LbOfGqvrxFY5h6rFV9cGq+u2VtAsArF1V9dGqunD4/kVV9fWD1K/ss4oOXe0BsLa11s5dyXFV1ZKc3Fq7fj8PCQDYz1prVyR5yr72q6pzkryhtfbCAz6oVbLRs4+Z/4ehqlw8AQCrTiZhqYT/ZRreJnpbVV2b5LtVdWhVHV9Vf1dVd1bVDVX1pgX7P6eqrqyqe6rq1qp6f1VtXkG/T6qqy6rq21W1o6r+qqqO2mu306vquqq6u6ourqotC47/qar68jCOf62qZy6x34VvCR5TVZ8e2rirqq6oKr9DAHAADJnjN8fO7VV1RlXdPGSS25JcPNSnnu+r6oer6t+ramdVXZJkYU44o6puXvDzCVX1ySHbfHvIL09N8sEkz6+q+6rqnmHfw6rqj6rqm1V1+/CxmUcsaOv8IQN9q6pev4zHL/scAOt68KvoZ5OcleSoJPNJ/jHJfybZluTFSX69ql4y7DuX5C1Jjkny/GH7r66gz0ryriTHJ3lqkhOS/O5e+7w2yUuSPCnJk5O8PUmq6tlJPpLkl5M8KslFSf6hqg5b5hjemuTmJMcmeUyS30rSlv9QAIAlGj23D45LcnSSE5P80mLn+2Hi8VNJ/mI45tIkrxjrsKpmknw6yU1JHp9Jvvmb1tp/Jzk3yZWttSNaaw8G8T8YxvasJCcN+//O0NZPJDkvyZlJTk6ynM/oyz4HgPC/Mu9rrW1vrd2f5PQkx7bWfq+1tqu19o0kH0rymiRprV3TWruqtba7tXZjJr98P7rcDltr17fWPttae6C1dmeSPx5p5/3DuO5K8s5MLlKS5BeTXNRa+2Jrba619rEkDyR53jKHMZvkh5Kc2Fqbba1d0Vpb1y8AAFjjpp3bk8kE5DuGbHB/Fj/fPy/JpiTvHc7hn0jyb1P6fE4mgfv81tp3W2vfb619fmzHqqqh37e01u5qre1M8vsZclCSVye5uLX2ldbad/PQ8D6V7HNg+HzYymxf8P2JSY5/8K2vwUySK5Kkqp6cyS/rjyQ5PJPn/JrldlhVj07yviQvSrI1kwu3uxcZ102ZvHAfHOPrqurXFmzfvGD7Uv1hJi/az0xe6/mz1tq7l9kGALB0087tSXJna+37C35e7HzfktyyV3C9aUqfJyS5qbW2ewnjOzaTfHPNkA2SyYz9zPD98dkz90zr8yFknwPDzP/KLHzhbE9yQ2vtqAVfW1trPzls/0CSr2Vy9/cjM3m7qLJ87xr6febQzs+PtHPCgu8fl+RbC8b4zr3GeHhr7ePLGUBrbWdr7a2ttScm+ekkv1FVL17BYwEAlmbauT156MdPFjvf35pkWy1I6EN7Y7YnedyUm4j37nNHkvuTPG1Bn0e21o4Ytt868hiWSvY5AIT/h+9LSe4dbrh5RFXNVNXTq+r0YfvWJPcmua+qTknyKyvsZ2uS+5LcU1Xbkpw/ss8bq+qxVXV0JhcZlwz1DyU5t6qeWxM/UFVnVdXW5QxguHHmpOF/HPdmcj/D3AofDwCwb9PO7WMWO99fmWR3kjfVZLGSl2fy8Z4xX8oktL97aGNLVb1g2HZ7kscO9xCktTY/9PueYaY+VbVtwb2Pf5vknKo6taoOT/KOZTz29Zh9Ng/P14NfM4vsuyqE/4eptTaXyZXgs5LckMkV8IeTHDnscl6Sn0uyM5NfxMVetIu5IMmzk3wnyT8l+eTIPn+d5DNJvjF8XTiM8epMPvv2/kzeLrs+yTkrGMPJST6XyQvxyiR/2lq7fAXtAABLM3puH7PY+b61tivJy4ef705ydsazxMJsc1KSb2Zyw+vZw+bLknw1yW1VtWOovW3o66qqujeTrPCUoa1/TvLe4bjrh/8u1XrMPl/N5J2QB79+YQV9HlC1zu9ZAADYkKrqxkz+oNbnVnssbBxm/gEAoBPCPwAAdMLHfgAAoBNm/gEAoBPCPwAAdOKg/oXfMw95lc8Yse58dv7SlfxRNgA2ANmF9Wix7GLmHwAAOiH8AwBAJ4R/AADohPAPAACdOKg3/LKnOnT609927z6IIwEA2DfZZf0z8w8AAJ0Q/gEAoBPCPwAAdEL4BwCATgj/AADQCav9HASHPnbbaH3nn2+eesxhFx41Wj/kiv/YL2MCAJhGdtm4zPwDAEAnhH8AAOiE8A8AAJ0Q/gEAoBPCPwAAdEL4BwCATljq8yCYO3Z86auPnXLR1GPOevO5o/UTvjAzfsD83LLHBQAwRnbZuMz8AwBAJ4R/AADohPAPAACdEP4BAKATwj8AAHTCaj+r6NiZ6U//R0+7eLR+wbaXjdZ3b795v4wJAGAa2WX9M/MPAACdEP4BAKATwj8AAHRC+AcAgE4I/wAA0Amr/RwEMzu+M1q/bnZm6jGnbpobrT9w0qPH+3DHPACwn8guG5eZfwAA6ITwDwAAnRD+AQCgE8I/AAB0QvgHAIBOWO3nINh9y62j9bd8/eypx1z2jEtG6//3mvG77E+5eutofX7nzn2MDgBgT7LLxmXmHwAAOiH8AwBAJ4R/AADohPAPAACdEP4BAKATVvs5GObnRss7rn7M1EPuOvWB0fprn3vVaP2aJzxtvKFrv7b42AAA9ia7bFhm/gEAoBPCPwAAdEL4BwCATgj/AADQCeEfAAA6IfwDAEAnLPW5ip7wqZ1Tt/3LK584Wn/j0VeO1n/slS8crZ947fLHBQAwRnZZ/8z8AwBAJ4R/AADohPAPAACdEP4BAKATwj8AAHTCaj+raa4tsmn8umzrIeP/ZLNHTG8LAGC/kF3WPTP/AADQCeEfAAA6IfwDAEAnhH8AAOiE8A8AAJ2w2s8aNbfc6zKXcQDAKpJd1gdPOwAAdEL4BwCATgj/AADQCeEfAAA6IfwDAEAnrPazimZ2fGfqto/d9LzR+uuecdNo/bhT7hhv6JCZ8fr83KJjAwDYm+yy/pn5BwCATgj/AADQCeEfAAA6IfwDAEAnhH8AAOiE8A8AAJ2w1OcqmrttyhJXSb79pdNG64c+Y3z5q/c85ZLR+gXbXjZa37395n2MDgBgT7LL+mfmHwAAOiH8AwBAJ4R/AADohPAPAACdEP4BAKATVvtZRW1219Rt2y4f3/bA63eP1k/dND9anzvmyPEO3DEPACyT7LL+mfkHAIBOCP8AANAJ4R8AADoh/AMAQCeEfwAA6ITVftaoLf97+2j9K7M1Wn/6pvHruNkf3DJa9w8PAOxPssv6YOYfAAA6IfwDAEAnhH8AAOiE8A8AAJ0Q/gEAoBNunF6j5u64c7R+3QPbRuunbb5ttH7LGZtH64+/YryeJG121z5GBwCwJ9llfTDzDwAAnRD+AQCgE8I/AAB0QvgHAIBOCP8AANAJq/2sVfNttDzXxq/XZmq8PnvEeDsAAPuV7LIumPkHAIBOCP8AANAJ4R8AADoh/AMAQCeEfwAA6ITwDwAAnbDU5zrzP98/brQ++8hbRuvzW+dG67XlsKl9tNldyx8YAMAI2WVtMfMPAACdEP4BAKATwj8AAHRC+AcAgE4I/wAA0Amr/axR0+5av/q800brf///2bv/GM2q+n7g7w/DLitllaqgZUH8sSjij1opKlFTG0psShsr/sDWJlJDW1pTLVVi2thaGqw2baqxpEqxoukPS7HWtDZNxRASpKiF1lK12i+V34KwArIgsrsz5/vHc0lml/vMzgy7Oz/O65VMmPnce885z7PzcN/3PM8986EbRut/dcqFo/W3nfGWqX0/4SNX72V0AAC7k13WBjP/AADQCeEfAAA6IfwDAEAnhH8AAOiE8A8AAJ2w2s8ac8gd94/Wr/veU0brrzpi22h9w2vunN7JR2fG63OzC44NAGBPssvqYuYfAAA6IfwDAEAnhH8AAOiE8A8AAJ0Q/gEAoBNW+1lj5q6/cbT+iSteOlo///X/PVo/66lXTe3j0pnxu++bO+YBgCWSXVYXM/8AANAJ4R8AADoh/AMAQCeEfwAA6ITwDwAAnbDaz1oz10bLG+6r0fpsm9ufowEAWJjssqqY+QcAgE4I/wAA0AnhHwAAOiH8AwBAJ4R/AADohPAPAACdsNTnGtN27hitP+2T94zW7zzze6P1zTMPTu2jNh2ypL4BAKaRXVYXM/8AANAJ4R8AADoh/AMAQCeEfwAA6ITwDwAAnbDaz3pxw22j5csfPHa0fuqhN09t6qIffvVo/aDPf3np4wIAGCO7rAgz/wAA0AnhHwAAOiH8AwBAJ4R/AADohPAPAACdsNrPOjG3ffto/V1XnD5af9Vpfzq1rRt+dtNofesXN47W284dexkdAMDuZJeVYeYfAAA6IfwDAEAnhH8AAOiE8A8AAJ0Q/gEAoBNW+1nnjrxq/J/4vJNOnnrMxntdEwIAK0N22b88UwAA0AnhHwAAOiH8AwBAJ4R/AADohPAPAACdEP4BAKAT1Vo7YJ2detDrDlxnLKg2bJy6rc3Ojm+Ym1Jf5y6bu7RWegwArAzZZfWQXRZvoexi5h8AADoh/AMAQCeEfwAA6ITwDwAAnRD+AQCgEwev9ABYGW3njpUeAgDAosku+4aZfwAA6ITwDwAAnRD+AQCgE8I/AAB0QvgHAIBOVGttpccAAAAcAGb+AQCgE8I/AAB0QvgHAIBOCP8AANAJ4R8AADoh/AMAQCeEfwAA6ITwDwAAnRD+AQCgE8I/AAB0QvgHAIBOCP8AANAJ4R8AADoh/AMAQCeEfwAA6ITwDwAAnRD+AQCgE8I/AAB0QvhfYVXVqmrrlG1XVNVZi2znxqr6iWWOYeqxVfXhqvqd5bQLAKxeVfWxqjp/+P7lVfWNA9Sv7LOCDl7pAbC6tdbOXs5xVdWSHNdau34fDwkA2Mdaa1cmedbe9quqM5Oc1Vp72X4f1ApZ79nHzP+jUFUungCAFSeTsFjC/xINbxO9s6quS/JAVR1cVUdV1d9X1V1VdUNVvXXe/i+qqqur6t6qur2qLqiqjcvo9xlVdXlVfaeqtlXVX1fV4XvsdlJVfa2q7qmqi6tq07zjf7qqvjyM49+q6vmL7Hf+W4JPrKrPDG3cXVVXVpXfIQDYD4bM8Vtj5/aqekVV3TpkkjuSXDzUp57vq+pHquo/qmp7VV2SZH5OeEVV3Trv52Oq6lNDtvnOkF+eneTDSU6uqvur6t5h30Oq6o+r6uaq+vbwsZnHzGvr3CEDfauq3ryExy/77AdrevAr6OeSnJbk8CRzSf4pyX8l2ZLklCS/UVWvHPadTXJOkicmOXnY/mvL6LOSvDfJUUmeneSYJL+3xz5vTPLKJM9I8swk70qSqnphko8m+ZUkT0hyYZJ/rKpDljiGtye5NckRSZ6U5LeTtKU/FABgkUbP7YMnJ3l8kmOT/PJC5/th4vHTSf5yOObSJK8Z67CqZpJ8JslNSZ6aSb7529ba/yQ5O8nVrbXDWmsPB/E/HMb2giRbh/1/d2jrJ5O8I8mpSY5LspTP6Ms++4HwvzwfbK3d0lp7MMlJSY5orf1+a21Ha+2bSS5K8oYkaa1d21r7QmttV2vtxkx++X5sqR221q5vrV3WWnuotXZXkj8ZaeeCYVx3J3lPJhcpSfJLSS5srX2xtTbbWvt4koeSvGSJw9iZ5IeSHNta29lau7K1tqZfAACwyk07tyeTCch3D9ngwSx8vn9Jkg1JPjCcwz+Z5N+n9PmiTAL3ua21B1pr32+tfX5sx6qqod9zWmt3t9a2J/mDDDkoyeuTXNxa+0pr7YE8MrxPJfvsHz4ftjy3zPv+2CRHPfzW12AmyZVJUlXPzOSX9UeTHJrJc37tUjusqiOTfDDJy5NszuTC7Z4FxnVTJi/ch8f4pqr69XnbN87bvlh/lMmL9rOT13r+vLX2viW2AQAs3rRze5Lc1Vr7/ryfFzrftyS37RFcb5rS5zFJbmqt7VrE+I7IJN9cO2SDZDJjPzN8f1R2zz3T+nwE2Wf/MPO/PPNfOLckuaG1dvi8r82ttZ8atn8oydczufv7sZm8XVRZuvcO/T5/aOcXRto5Zt73T0nyrXljfM8eYzy0tfaJpQygtba9tfb21trTk/xMkt+sqlOW8VgAgMWZdm5PHvnxk4XO97cn2VLzEvrQ3phbkjxlyk3Ee/a5LcmDSZ4zr8/HtdYOG7bfPvIYFkv22Q+E/0fvS0nuG264eUxVzVTVc6vqpGH75iT3Jbm/qo5P8qvL7GdzkvuT3FtVW5KcO7LPW6rq6Kp6fCYXGZcM9YuSnF1VL66JH6iq06pq81IGMNw4s3X4H8d9mdzPMLvMxwMA7N20c/uYhc73VyfZleStNVms5PRMPt4z5kuZhPb3DW1sqqqXDtu+neTo4R6CtNbmhn7fP8zUp6q2zLv38e+SnFlVJ1TVoUnevYTHvhazz8bh+Xr4a2aBfVeE8P8otdZmM7kSfEGSGzK5Av5IkscNu7wjyc8n2Z7JL+JCL9qFnJfkhUm+m+Sfk3xqZJ+/SfLZJN8cvs4fxnhNJp99uyCTt8uuT3LmMsZwXJLPZfJCvDrJn7XWrlhGOwDA4oye28csdL5vre1Icvrw8z1Jzsh4lpifbbYmuTmTG17PGDZfnuSrSe6oqm1D7Z1DX1+oqvsyyQrPGtr6lyQfGI67fvjvYq3F7PPVTN4JefjrF5fR535Va/yeBQCAdamqbszkD2p9bqXHwvph5h8AADoh/AMAQCd87AcAADph5h8AADoh/AMAQCcO6F/4PfWg1/mMEWvOZXOXLuePsgGwDsgurEULZRcz/wAA0AnhHwAAOiH8AwBAJ4R/AADoxAG94Zfd1cHTn/62a9cBHAkAwN7JLmufmX8AAOiE8A8AAJ0Q/gEAoBPCPwAAdEL4BwCATljt5wA4+Ogto/Xtf7Fx6jGHnH/4aP2gK/9zn4wJAGAa2WX9MvMPAACdEP4BAKATwj8AAHRC+AcAgE4I/wAA0AnhHwAAOmGpzwNg9ojxpa8+fvyFU4857W1nj9aPuWpm/IC52SWPCwBgjOyyfpn5BwCATgj/AADQCeEfAAA6IfwDAEAnhH8AAOiE1X5W0BEz05/+j5148Wj9vC2vHq3vuuXWfTImAIBpZJe1z8w/AAB0QvgHAIBOCP8AANAJ4R8AADoh/AMAQCes9nMAzGz77mj9aztnph5zwobZ0fpDW48c78Md8wDAPiK7rF9m/gEAoBPCPwAAdEL4BwCATgj/AADQCeEfAAA6YbWfA2DXbbeP1s/5xhlTj7n8eZeM1v/vDeN32R9/zebR+tz27XsZHQDA7mSX9cvMPwAAdEL4BwCATgj/AADQCeEfAAA6IfwDAEAnrPZzIMzNjpa3XfOkqYfcfcJDo/U3vvgLo/Vrn/ac8Yau+/rCYwMA2JPssm6Z+QcAgE4I/wAA0AnhHwAAOiH8AwBAJ4R/AADohPAPAACdsNTnCnrap7dP3favr336aP0tj796tP7jr33ZaP3Y65Y+LgCAMbLL2mfmHwAAOiH8AwBAJ4R/AADohPAPAACdEP4BAKATVvtZSbNtgU3j12WbDxr/J9t52PS2AAD2CdllzTPzDwAAnRD+AQCgE8I/AAB0QvgHAIBOCP8AANAJq/2sUrNLvS5zGQcArCDZZW3wtAMAQCeEfwAA6ITwDwAAnRD+AQCgE8I/AAB0wmo/K2hm23enbvv4TS8Zrb/peTeN1p98/J3jDR00M16fm11wbAAAe5Jd1j4z/wAA0AnhHwAAOiH8AwBAJ4R/AADohPAPAACdEP4BAKATlvpcQbN3TFniKsl3vnTiaP3g540vf/X+Z10yWj9vy6tH67tuuXUvowMA2J3ssvaZ+QcAgE4I/wAA0AnhHwAAOiH8AwBAJ4R/AADohNV+VlDbuWPqti1XjG976M27RusnbJgbrc8+8XHjHbhjHgBYItll7TPzDwAAnRD+AQCgE8I/AAB0QvgHAIBOCP8AANAJq/2sUpv+37dH61/ZWctg+/YAACAASURBVKP1524Yv47b+YObRuv+4QGAfUl2WRvM/AMAQCeEfwAA6ITwDwAAnRD+AQCgE8I/AAB0wo3Tq9TsnXeN1r/20JbR+okb7xit3/aKjaP1p145Xk+StnPHXkYHALA72WVtMPMPAACdEP4BAKATwj8AAHRC+AcAgE4I/wAA0Amr/axWc220PNvGr9dmary+87DxdgAA9inZZU0w8w8AAJ0Q/gEAoBPCPwAAdEL4BwCATgj/AADQCeEfAAA6YanPNeZ/v//k0frOx942Wp/bPDtar02HTO2j7dyx9IEBAIyQXVYXM/8AANAJ4R8AADoh/AMAQCeEfwAA6ITwDwAAnbDazyo17a71a95x4mj9Hz50w2j9r065cLT+tjPeMrXvJ3zk6r2MDgBgd7LL2mDmHwAAOiH8AwBAJ4R/AADohPAPAACdEP4BAKATVvtZYw654/7R+nXfe8po/VVHbButb3jNndM7+ejMeH1udsGxAQDsSXZZXcz8AwBAJ4R/AADohPAPAACdEP4BAKATwj8AAHTCaj9rzNz1N47WP3HFS0fr57/+v0frZz31qql9XDozfvd9c8c8ALBEssvqYuYfAAA6IfwDAEAnhH8AAOiE8A8AAJ0Q/gEAoBNW+1lr5tpoecN9NVqfbXP7czQAAAuTXVYVM/8AANAJ4R8AADoh/AMAQCeEfwAA6ITwDwAAnRD+AQCgE5b6XGPazh2j9ad98p7R+p1nfm+0vnnmwal91KZDltQ3AMA0ssvqYuYfAAA6IfwDAEAnhH8AAOiE8A8AAJ0Q/gEAoBNW+1kvbrhttHz5g8eO1k899OapTV30w68erR/0+S8vfVwAAGNklxVh5h8AADoh/AMAQCeEfwAA6ITwDwAAnRD+AQCgE1b7WSfmtm8frb/ritNH66867U+ntnXDz24arW/94sbRetu5Yy+jAwDYneyyMsz8AwBAJ4R/AADohPAPAACdEP4BAKATwj8AAHTCaj/r3JFXjf8Tn3fSyVOP2Xiva0IAYGXILvuXZwoAADoh/AMAQCeEfwAA6ITwDwAAnRD+AQCgE8I/AAB0olprB6yzUw963YHrjAXVho1Tt7XZ2fENc1Pq69xlc5fWSo8BgJUhu6wessviLZRdzPwDAEAnhH8AAOiE8A8AAJ0Q/gEAoBPCPwAAdOLglR4AK6Pt3LHSQwAAWDTZZd8w8w8AAJ0Q/gEAoBPCPwAAdEL4BwCATgj/AADQiWqtrfQYAACAA8DMPwAAdEL4BwCATgj/AADQCeEfAAA6IfwDAEAnhH8AAOiE8A8AAJ0Q/gEAoBPCPwAAdEL4BwCATgj/AADQCeEfAAA6IfwDAEAnhH8AAOiE8A8AAJ0Q/gEAoBPCPwAAdEL4BwCATgj/K6yqWlVtnbLtiqo6a5Ht3FhVP7HMMUw9tqo+XFW/s5x2AYDVq6o+VlXnD9+/vKq+cYD6lX1W0MErPQBWt9ba2cs5rqpakuNaa9fv4yEBAPtYa+3KJM/a235VdWaSs1prL9vvg1oh6z37mPl/FKrKxRMAsOJkEhZL+F+i4W2id1bVdUkeqKqDq+qoqvr7qrqrqm6oqrfO2/9FVXV1Vd1bVbdX1QVVtXEZ/T6jqi6vqu9U1baq+uuqOnyP3U6qqq9V1T1VdXFVbZp3/E9X1ZeHcfxbVT1/kf3Of0vwiVX1maGNu6vqyqryOwQA+8GQOX5r7NxeVa+oqluHTHJHkouH+tTzfVX9SFX9R1Vtr6pLkszPCa+oqlvn/XxMVX1qyDbfGfLLs5N8OMnJVXV/Vd077HtIVf1xVd1cVd8ePjbzmHltnTtkoG9V1ZuX8Phln/1gTQ9+Bf1cktOSHJ5kLsk/JfmvJFuSnJLkN6rqlcO+s0nOSfLEJCcP239tGX1WkvcmOSrJs5Mck+T39tjnjUlemeQZSZ6Z5F1JUlUvTPLRJL+S5AlJLkzyj1V1yBLH8PYktyY5IsmTkvx2krb0hwIALNLouX3w5CSPT3Jskl9e6Hw/TDx+OslfDsdcmuQ1Yx1W1UySzyS5KclTM8k3f9ta+58kZye5urV2WGvt4SD+h8PYXpBk67D/7w5t/WSSdyQ5NclxSZbyGX3ZZz8Q/pfng621W1prDyY5KckRrbXfb63taK19M8lFSd6QJK21a1trX2it7Wqt3ZjJL9+PLbXD1tr1rbXLWmsPtdbuSvInI+1cMIzr7iTvyeQiJUl+KcmFrbUvttZmW2sfT/JQkpcscRg7k/xQkmNbaztba1e21tb0CwAAVrlp5/ZkMgH57iEbPJiFz/cvSbIhyQeGc/gnk/z7lD5flEngPre19kBr7futtc+P7VhVNfR7Tmvt7tba9iR/kCEHJXl9kotba19prT2QR4b3qWSf/cPnw5bnlnnfH5vkqIff+hrMJLkySarqmZn8sv5okkMzec6vXWqHVXVkkg8meXmSzZlcuN2zwLhuyuSF+/AY31RVvz5v+8Z52xfrjzJ50X528lrPn7fW3rfENgCAxZt2bk+Su1pr35/380Ln+5bktj2C601T+jwmyU2ttV2LGN8RmeSba4dskExm7GeG74/K7rlnWp+PIPvsH2b+l2f+C+eWJDe01g6f97W5tfZTw/YPJfl6Jnd/PzaTt4sqS/feod/nD+38wkg7x8z7/ilJvjVvjO/ZY4yHttY+sZQBtNa2t9be3lp7epKfSfKbVXXKMh4LALA4087tySM/frLQ+f72JFtqXkIf2htzS5KnTLmJeM8+tyV5MMlz5vX5uNbaYcP220cew2LJPvuB8P/ofSnJfcMNN4+pqpmqem5VnTRs35zkviT3V9XxSX51mf1sTnJ/knurakuSc0f2eUtVHV1Vj8/kIuOSoX5RkrOr6sU18QNVdVpVbV7KAIYbZ7YO/+O4L5P7GWaX+XgAgL2bdm4fs9D5/uoku5K8tSaLlZyeycd7xnwpk9D+vqGNTVX10mHbt5McPdxDkNba3NDv+4eZ+lTVlnn3Pv5dkjOr6oSqOjTJu5fw2Ndi9tk4PF8Pf80ssO+KEP4fpdbabCZXgi9IckMmV8AfSfK4YZd3JPn5JNsz+UVc6EW7kPOSvDDJd5P8c5JPjezzN0k+m+Sbw9f5wxivyeSzbxdk8nbZ9UnOXMYYjkvyuUxeiFcn+bPW2hXLaAcAWJzRc/uYhc73rbUdSU4ffr4nyRkZzxLzs83WJDdncsPrGcPmy5N8NckdVbVtqL1z6OsLVXVfJlnhWUNb/5LkA8Nx1w//Xay1mH2+msk7IQ9//eIy+tyvao3fswAAsC5V1Y2Z/EGtz630WFg/zPwDAEAnhH8AAOiEj/0AAEAnzPwDAEAnhH8AAOjEAf0Lv6ce9DqfMWLNuWzu0uX8UTYA1gHZhbVooexi5h8AADoh/AMAQCeEfwAA6ITwDwAAnTigN/yyuzp4+tPfdu06gCMBANg72WXtM/MPAACdEP4BAKATwj8AAHRC+AcAgE4I/wAA0Amr/RwABx+9ZbS+/S82Tj3mkPMPH60fdOV/7pMxAQBMI7usX2b+AQCgE8I/AAB0QvgHAIBOCP8AANAJ4R8AADoh/AMAQCcs9XkAzB4xvvTVx4+/cOoxp73t7NH6MVfNjB8wN7vkcQEAjJFd1i8z/wAA0AnhHwAAOiH8AwBAJ4R/AADohPAPAACdsNrPCjpiZvrT/7ETLx6tn7fl1aP1Xbfcuk/GBAAwjeyy9pn5BwCATgj/AADQCeEfAAA6IfwDAEAnhH8AAOiE1X4OgJlt3x2tf23nzNRjTtgwO1p/aOuR4324Yx4A2Edkl/XLzD8AAHRC+AcAgE4I/wAA0AnhHwAAOiH8AwBAJ6z2cwDsuu320fo53zhj6jGXP++S0fr/vWH8Lvvjr9k8Wp/bvn0vowMA2J3ssn6Z+QcAgE4I/wAA0AnhHwAAOiH8AwBAJ4R/AADohNV+DoS52dHytmueNPWQu094aLT+xhd/YbR+7dOeM97QdV9feGwAAHuSXdYtM/8AANAJ4R8AADoh/AMAQCeEfwAA6ITwDwAAnRD+AQCgE5b6XEFP+/T2qdv+9bVPH62/5fFXj9Z//LUvG60fe93SxwUAMEZ2WfvM/AMAQCeEfwAA6ITwDwAAnRD+AQCgE8I/AAB0wmo/K2m2LbBp/Lps80Hj/2Q7D5veFgDAPiG7rHlm/gEAoBPCPwAAdEL4BwCATgj/AADQCeEfAAA6YbWfVWp2qddlLuMAgBUku6wNnnYAAOiE8A8AAJ0Q/gEAoBPCPwAAdEL4BwCATljtZwXNbPvu1G0fv+klo/U3Pe+m0fqTj79zvKGDZsbrc7MLjg0AYE+yy9pn5h8AADoh/AMAQCeEfwAA6ITwDwAAnRD+AQCgE8I/AAB0wlKfK2j2jilLXCX5zpdOHK0f/Lzx5a/e/6xLRuvnbXn1aH3XLbfuZXQAALuTXdY+M/8AANAJ4R8AADoh/AMAQCeEfwAA6ITwDwAAnbDazwpqO3dM3bblivFtD71512j9hA1zo/XZJz5uvAN3zAMASyS7rH1m/gEAoBPCPwAAdEL4BwCATgj/AADQCeEfAAA6YbWfVWrT//v2aP0rO2u0/twN49dxO39w02jdPzwAsC/JLmuDmX8AAOiE8A8AAJ0Q/gEAoBPCPwAAdEL4BwCATrhxepWavfOu0frXHtoyWj9x4x2j9dtesXG0/tQrx+tJ0nbu2MvoAAB2J7usDWb+AQCgE8I/AAB0QvgHAIBOCP8AANAJ4R8AADphtZ/Vaq6Nlmfb+PXaTI3Xdx423g4AwD4lu6wJZv4BAKATwj8AAHRC+AcAgE4I/wAA0AnhHwAAOiH8AwBAJyz1ucb87/efPFrf+djbRutzm2dH67XpkKl9tJ07lj4wAIARssvqYuYfAAA6IfwDAEAnhH8AAOiE8A8AAJ0Q/gEAoBNW+1mlpt21fs07Thyt/8OHbhit/9UpF47W33bGW6b2/YSPXL2X0QEA7E52WRvM/AMAQCeEfwAA6ITwDwAAnRD+AQCgE8I/AAB0wmo/a8whd9w/Wr/ue08Zrb/qiG2j9Q2vuXN6Jx+dGa/PzS44NgCAPckuq4uZfwAA6ITwDwAAnRD+AQCgE8I/AAB0QvgHAIBOWO1njZm7/sbR+ieueOlo/fzX//do/aynXjW1j0tnxu++b+6YBwCWSHZZXcz8AwBAJ4R/AADohPAPAACdEP4BAKATwj8AAHTCaj9rzVwbLW+4r0brs21uf44GAGBhssuqYuYfAAA6IfwDAEAnhH8AAOiE8A8AAJ0Q/gEAoBPCPwAAdMJSn2tM27ljtP60T94zWr/zzO+N1jfPPDi1j9p0yJL6BgCYRnZZXcz8AwBAJ4R/AADohPAPAACdEP4BAKATwj8AAHTCaj/rxQ23jZYvf/DY0fqph948tamLfvjVo/WDPv/lpY8LAGCM7LIizPwDAEAnhH8AAOiE8A8AAJ0Q/gEAoBPCPwAAdMJqP+vE3Pbto/V3XXH6aP1Vp/3p1LZu+NlNo/WtX9w4Wm87d+xldAAAu5NdVoaZfwAA6ITwDwAAnRD+AQCgE8I/AAB0QvgHAIBOWO1nnTvyqvF/4vNOOnnqMRvvdU0IAKwM2WX/8kwBAEAnhH8AAOiE8A8AAJ0Q/gEAoBPCPwAAdEL4BwCATlRr7YB1dupBrztwnbGg2rBx6rY2Ozu+YW5KfZ27bO7SWukxALAyZJfVQ3ZZvIWyi5l/AADohPAPAACdEP4BAKATwj8AAHRC+AcAgE4cvNIDYGW0nTtWeggAAIsmu+wbZv4BAKATwj8AAHRC+AcAgE4I/wAA0AnhHwAAOlGttZUeAwAAcACY+QcAgE4I/wAA0AnhHwAAOiH8AwBAJ4R/AADohPAPAACdEP4BAKATwj8AAHRC+AcAgE4I/wAA0AnhHwAAOiH8AwBAJ4R/AADohPAPAACdEP4BAKATwj8AAHRC+AcAgE4I/wAA0Anhf4VVVauqrVO2XVFVZy2ynRur6ieWOYapx1bVh6vqd5bTLgCwelXVx6rq/OH7l1fVNw5Qv7LPCjp4pQfA6tZaO3s5x1VVS3Jca+36fTwkAGAfa61dmeRZe9uvqs5MclZr7WX7fVArZL1nHzP/j0JVuXgCAFacTMJiCf9LNLxN9M6qui7JA1V1cFUdVVV/X1V3VdUNVfXWefu/qKqurqp7q+r2qrqgqjYuo99nVNXlVfWdqtpWVX9dVYfvsdtJVfW1qrqnqi6uqk3zjv/pqvryMI5/q6rnL7Lf+W8JPrGqPjO0cXdVXVlVfocAYD8YMsdvjZ3bq+oVVXXrkEnuSHLxUJ96vq+qH6mq/6iq7VV1SZL5OeEVVXXrvJ+PqapPDdnmO0N+eXaSDyc5uarur6p7h30Pqao/rqqbq+rbw8dmHjOvrXOHDPStqnrzEh6/7LMfrOnBr6CfS3JaksOTzCX5pyT/lWRLklOS/EZVvXLYdzbJOUmemOTkYfuvLaPPSvLeJEcleXaSY5L83h77vDHJK5M8I8kzk7wrSarqhUk+muRXkjwhyYVJ/rGqDlniGN6e5NYkRyR5UpLfTtKW/lAAgEUaPbcPnpzk8UmOTfLLC53vh4nHTyf5y+GYS5O8ZqzDqppJ8pkkNyV5aib55m9ba/+T5OwkV7fWDmutPRzE/3AY2wuSbB32/92hrZ9M8o4kpyY5LslSPqMv++wHwv/yfLC1dktr7cEkJyU5orX2+621Ha21bya5KMkbkqS1dm1r7QuttV2ttRsz+eX7saV22Fq7vrV2WWvtodbaXUn+ZKSdC4Zx3Z3kPZlcpCTJLyW5sLX2xdbabGvt40keSvKSJQ5jZ5IfSnJsa21na+3K1tqafgEAwCo37dyeTCYg3z1kgwez8Pn+JUk2JPnAcA7/ZJJ/n9LnizIJ3Oe21h5orX2/tfb5sR2rqoZ+z2mt3d1a257kDzLkoCSvT3Jxa+0rrbUH8sjwPpXss3/4fNjy3DLv+2OTHPXwW1+DmSRXJklVPTOTX9YfTXJoJs/5tUvtsKqOTPLBJC9PsjmTC7d7FhjXTZm8cB8e45uq6tfnbd84b/ti/VEmL9rPTl7r+fPW2vuW2AYAsHjTzu1Jcldr7fvzfl7ofN+S3LZHcL1pSp/HJLmptbZrEeM7IpN8c+2QDZLJjP3M8P1R2T33TOvzEWSf/cPM//LMf+HckuSG1trh8742t9Z+atj+oSRfz+Tu78dm8nZRZeneO/T7/KGdXxhp55h53z8lybfmjfE9e4zx0NbaJ5YygNba9tba21trT0/yM0l+s6pOWcZjAQAWZ9q5PXnkx08WOt/fnmRLzUvoQ3tjbknylCk3Ee/Z57YkDyZ5zrw+H9daO2zYfvvIY1gs2Wc/EP4fvS8luW+44eYxVTVTVc+tqpOG7ZuT3Jfk/qo6PsmvLrOfzUnuT3JvVW1Jcu7IPm+pqqOr6vGZXGRcMtQvSnJ2Vb24Jn6gqk6rqs1LGcBw48zW4X8c92VyP8PsMh8PALB3087tYxY631+dZFeSt9ZksZLTM/l4z5gvZRLa3ze0samqXjps+3aSo4d7CNJamxv6ff8wU5+q2jLv3se/S3JmVZ1QVYcmefcSHvtazD4bh+fr4a+ZBfZdEcL/o9Ram83kSvAFSW7I5Ar4I0keN+zyjiQ/n2R7Jr+IC71oF3Jekhcm+W6Sf07yqZF9/ibJZ5N8c/g6fxjjNZl89u2CTN4uuz7JmcsYw3FJPpfJC/HqJH/WWrtiGe0AAIszem4fs9D5vrW2I8npw8/3JDkj41lifrbZmuTmTG54PWPYfHmSrya5o6q2DbV3Dn19oaruyyQrPGto61+SfGA47vrhv4u1FrPPVzN5J+Thr19cRp/7Va3xexYAANalqroxkz+o9bmVHgvrh5l/AADohPAPAACd8LEfAADohJl/AADohPAPAACdOKB/4ffUg17nM0asOZfNXbqcP8oGwDogu7AWLZRdzPwDAEAnhH8AAOiE8A8AAJ0Q/gEAoBPCPwAAdOKArvbD7urg6U9/27XrAI4EAGDvZJe1z8w/AAB0QvgHAIBOCP8AANAJ4R8AADoh/AMAQCes9nMAHHz0ltH69r/YOPWYQ84/fLR+0JX/uU/GBAAwjeyyfpn5BwCATgj/AADQCeEfAAA6IfwDAEAnhH8AAOiE8A8AAJ2w1OcBMHvE+NJXHz/+wqnHnPa2s0frx1w1M37A3OySxwUAMEZ2Wb/M/AMAQCeEfwAA6ITwDwAAnRD+AQCgE8I/AAB0wmo/K+iImelP/8dOvHi0ft6WV4/Wd91y6z4ZEwDANLLL2mfmHwAAOiH8AwBAJ4R/AADohPAPAACdEP4BAKATVvs5AGa2fXe0/rWdM1OPOWHD7Gj9oa1HjvfhjnkAYB+RXdYvM/8AANAJ4R8AADoh/AMAQCeEfwAA6ITwDwAAnbDazwGw67bbR+vnfOOMqcdc/rxLRuv/94bxu+yPv2bzaH1u+/a9jA4AYHeyy/pl5h8AADoh/AMAQCeEfwAA6ITwDwAAnRD+AQCgE1b7ORDmZkfL26550tRD7j7hodH6G1/8hdH6tU97znhD13194bEBAOxJdlm3zPwDAEAnhH8AAOiE8A8AAJ0Q/gEAoBPCPwAAdEL4BwCATljqcwU97dPbp27719c+fbT+lsdfPVr/8de+bLR+7HVLHxcAwBjZZe0z8w8AAJ0Q/gEAoBPCPwAAdEL4BwCATgj/AADQCav9rKTZtsCm8euyzQeN/5PtPGx6WwAA+4TssuaZ+QcAgE4I/wAA0AnhHwAAOiH8AwBAJ4R/AADohNV+VqnZpV6XuYwDAFaQ7LI2eNoBAKATwj8AAHRC+AcAgE4I/wAA0AnhHwAAOmG1nxU0s+27U7d9/KaXjNbf9LybRutPPv7O8YYOmhmvz80uODYAgD3JLmufmX8AAOiE8A8AAJ0Q/gEAoBPCPwAAdEL4BwCATgj/AADQCUt9rqDZO6YscZXkO186cbR+8PPGl796/7MuGa2ft+XVo/Vdt9y6l9EBAOxOdln7zPwDAEAnhH8AAOiE8A8AAJ0Q/gEAoBPCPwAAdMJqPyuo7dwxdduWK8a3PfTmXaP1EzbMjdZnn/i48Q7cMQ8ALJHssvaZ+QcAgE4I/wAA0AnhHwAAOiH8AwBAJ4R/AADohNV+VqlN/+/bo/Wv7KzR+nM3jF/H7fzBTaN1//AAwL4ku6wNZv4BAKATwj8AAHRC+AcAgE4I/wAA0AnhHwAAOuHG6VVq9s67Rutfe2jLaP3EjXeM1m97xcbR+lOvHK8nSdu5Yy+jAwDYneyyNpj5BwCATgj/AADQCeEfAAA6IfwDAEAnhH8AAOiE1X5Wq7k2Wp5t49drMzVe33nYeDsAAPuU7LImmPkHAIBOCP8AANAJ4R8AADoh/AMAQCeEfwAA6ITwDwAAnbDU5xrzv99/8mh952NvG63PbZ4drdemQ6b20XbuWPrAAABGyC6ri5l/AADohPAPAACdEP4BAKATwj8AAHRC+AcAgE5Y7WeVmnbX+jXvOHG0/g8fumG0/lenXDhaf9sZb5na9xM+cvVeRgcAsDvZZW0w8w8AAJ0Q/gEAoBPCPwAAdEL4BwCATgj/AADQCav9rDGH3HH/aP267z1ltP6qI7aN1je85s7pnXx0Zrw+N7vg2AAA9iS7rC5m/gEAoBPCPwAAdEL4BwCATgj/AADQCeEfAAA6YbWfNWbu+htH65+44qWj9fNf/9+j9bOeetXUPi6dGb/7vrljHgBYItlldTHzDwAAnRD+AQCgE8I/AAB0QvgHAIBOCP8AANAJq/2sNXNttLzhvhqtz7a5/TkaAICFyS6ripl/AADohPAPAACdEP4BAKATwj8AAHRC+AcAgE4I/wAA0AlLfa4xbeeO0frTPnnPaP3OM783Wt888+DUPmrTIUvqGwBgGtlldTHzDwAAnRD+AQCgE8I/AAB0QvgHAIBOCP8AANAJq/2sFzfcNlq+/MFjR+unHnrz1KYu+uFXj9YP+vyXlz4uAIAxssuKMPMPAACdEP4BAKATwj8AAHRC+AcAgE4I/wAA0Amr/awTc9u3j9bfdcXpo/VXnfanU9u64Wc3jda3fnHjaL3t3LGX0QEA7E52WRlm/gEAoBPCPwAAdEL4BwCATgj/AADQCeEfAAA6YbWfde7Iq8b/ic876eSpx2y81zUhALAyZJf9yzMFAACdEP4BAKATwj8AAHRC+AcAgE4I/wAA0AnhHwAAOlGttQPW2akHve7AdcaCasPGqdva7Oz4hrkp9XXusrlLa6XHAMDKkF1WD9ll8RbKLmb+AQCgE8I/AAB0QvgHAIBOCP8AANAJ4R8AADpx8EoPgJXRdu5Y6SEAACya7LJvmPkHAIBOCP8AANAJ4R8AADoh/AMAQCeEfwAA6ES11lZ6DAAAwAFg5h8AADoh/AMAQCeEfwAA6ITwDwAAnRD+AQCgE8I/AAB0QvgHAIBOCP8AANAJ4R8AADoh/AMAQCeEfwAA6ITwDwAAnRD+AQCgE8I/AAB0QvgHAIBOCP8AANAJ4R8AADoh/AMAQCeE/xVWVa2qtk7ZdkVVnbXIdm6sqp9Y5himHltVH66q31lOuwDA6lVVH6uq84fvX15V3zhA/co+K+jglR4Aq1tr7ezlHFdVLclxrbXr9/GQAIB9rLV2ZZJn7W2/qjozyVmttZft90GtkPWefcz8PwpV5eIJAFhxMgmLJfwv0fA20Tur6rokD1TVwVV1VFX9fVXdVVU3VNVb5+3/oqq6uqrurarbq+qCqtq4jH6fUVWXV9V3qmpbVf11VR2+x24nVdXXquqeqrq4qjbNO/6nq+rLwzj+raqev8h+578l+MSq+szQxt1VdWVV+R0CgP1gyBy/NXZur6pXVNWtQya5I8nFQ33q+b6qfqSq/qOqtlfVJUnm54RXVNWt834+pqo+NWSb7wz55dlJPpzk5Kq6v6ruHfY91Kec+QAADFBJREFUpKr+uKpurqpvDx+becy8ts4dMtC3qurNS3j8ss9+sKYHv4J+LslpSQ5PMpfkn5L8V5ItSU5J8htV9cph39kk5yR5YpKTh+2/tow+K8l7kxyV5NlJjknye3vs88Ykr0zyjCTPTPKuJKmqFyb5aJJfSfKEJBcm+ceqOmSJY3h7kluTHJHkSUl+O0lb+kMBABZp9Nw+eHKSxyc5NskvL3S+HyYeP53kL4djLk3ymrEOq2omyWeS3JTkqZnkm79trf1PkrOTXN1aO6y19nAQ/8NhbC9IsnXY/3eHtn4yyTuSnJrkuCRL+Yy+7LMfCP/L88HW2i2ttQeTnJTkiNba77fWdrTWvpnkoiRvSJLW2rWttS+01na11m7M5Jfvx5baYWvt+tbaZa21h1prdyX5k5F2LhjGdXeS92RykZIkv5TkwtbaF1trs621jyd5KMlLljiMnUl+KMmxrbWdrbUrW2tr+gUAAKvctHN7MpmAfPeQDR7Mwuf7lyTZkOQDwzn8k0n+fUqfL8okcJ/bWnugtfb91trnx3asqhr6Pae1dndrbXuSP8iQg5K8PsnFrbWvtNYeyCPD+1Syz/7h82HLc8u8749NctTDb30NZpJcmSRV9cxMfll/NMmhmTzn1y61w6o6MskHk7w8yeZMLtzuWWBcN2Xywn14jG+qql+ft33jvO2L9UeZvGg/O3mt589ba+9bYhsAwOJNO7cnyV2tte/P+3mh831LctsewfWmKX0ek+Sm1tquRYzviEzyzbVDNkgmM/Yzw/dHZffcM63PR5B99g8z/8sz/4VzS5IbWmuHz/va3Fr7qWH7h5J8PZO7vx+bydtFlaV779Dv84d2fmGknWPmff+UJN+aN8b37DHGQ1trn1jKAFpr21trb2+tPT3JzyT5zao6ZRmPBQBYnGnn9uSRHz9Z6Hx/e5ItNS+hD+2NuSXJU6bcRLxnn9uSPJjkOfP6fFxr7bBh++0jj2GxZJ/9QPh/9L6U5L7hhpvHVNVMVT23qk4atm9Ocl+S+6vq+CS/usx+Nie5P8m9VbUlybkj+7ylqo6uqsdncpFxyVC/KMnZVfXimviBqjqtqjYvZQDDjTNbh/9x3JfJ/Qyzy3w8AMDeTTu3j1nofH91kl1J3lqTxUpOz+TjPWO+lElof9/Qxqaqeumw7dtJjh7uIUhrbW7o9/3DTH2qasu8ex//LsmZVXVCVR2a5N1LeOxrMftsHJ6vh79mFth3RQj/j1JrbTaTK8EXJLkhkyvgjyR53LDLO5L8fJLtmfwiLvSiXch5SV6Y5LtJ/jnJp0b2+Zskn03yzeHr/GGM12Ty2bcLMnm77PokZy5jDMcl+VwmL8Srk/xZa+2KZbQDACzO6Ll9zELn+9bajiSnDz/fk+SMjGeJ+dlma5KbM7nh9Yxh8+VJvprkjqraNtTeOfT1haq6L5Os8KyhrX9J8oHhuOuH/y7WWsw+X83knZCHv35xGX3uV7XG71kAAFiXqurGTP6g1udWeiysH2b+AQCgE8I/AAB0wsd+AACgE2b+AQCgEwf0j3ydetDrvM3AmnPZ3KXL+bsMAKwDsgtr0ULZxcw/AAB0QvgHAIBOCP8AANAJ4R8AADoh/AMAQCcO6Go/7K4Onv70t127DuBIAAD2TnZZ+8z8AwBAJ4R/AADohPAPAACdEP4BAKATwj8AAHTCaj8HwMFHbxmtb/+LjVOPOeT8w0frB135n/tkTAAA08gu65eZfwAA6ITwDwAAnRD+AQCgE8I/AAB0QvgHAIBOCP8AANAJS30eALNHjC999fHjL5x6zGlvO3u0fsxVM+MHzM0ueVwAAGNkl/XLzD8AAHRC+AcAgE4I/wAA0AnhHwAAOiH8AwBAJ6z2s4KOmJn+9H/sxItH6+dtefVofdctt+6TMQEATCO7rH1m/gEAoBPCPwAAdEL4BwCATgj/AADQCeEfAAA6YbWfA2Bm23dH61/bOTP1mBM2zI7WH9p65Hgf7pgHAPYR2WX9MvMPAACdEP4BAKATwj8AAHRC+AcAgE4I/wAA0Amr/RwAu267fbR+zjfOmHrM5c+7ZLT+f28Yv8v++Gs2j9bntm/fy+gAAHYnu6xfZv4BAKATwj8AAHRC+AcAgP/fzv2EWFXGcRyeO+OMQ2jRf0FIjaCwpIVERS2KcNXCLKtFiyLauXDTso3QvkWLiCIIggiD2kYQQokVBmERUYFZCTbZX0Nzrvee1iO/M+PAOOfe+T7P8neYc17mbj7nhfOGEP8AABBC/AMAQAin/ayG4aAcnz56Y+uf/LH9fDl/6u5Py/kX226vb3Ts28XXBgBwMe2yZtn5BwCAEOIfAABCiH8AAAgh/gEAIIT4BwCAEOIfAABCOOqzQ9veP9N67YO9N5fzfdccKecP7r2/nG85tvx1AQBUtMv4s/MPAAAhxD8AAIQQ/wAAEEL8AwBACPEPAAAhnPbTpUGzyKX6vWzjZP2T9Te03wsAYEVol7Fn5x8AAEKIfwAACCH+AQAghPgHAIAQ4h8AAEI47WdEDZb7XuY1DgDokHYZD/7tAAAQQvwDAEAI8Q8AACHEPwAAhBD/AAAQwmk/HZo6/XfrtTdP3FPOn95xopxvum2uvtHkVD0fDhZdGwDAxbTL+LPzDwAAIcQ/AACEEP8AABBC/AMAQAjxDwAAIcQ/AACEcNRnhwanWo64mpiY+P3zneV83Y76+KuXbn2nnB/YvKecX/j5lyVWBwCwkHYZf3b+AQAghPgHAIAQ4h8AAEKIfwAACCH+AQAghNN+OtT051uvbT5UXzv/7IVyvn16WM4H111VP8AX8wDAMmmX8WfnHwAAQoh/AAAIIf4BACCE+AcAgBDiHwAAQjjtZ0TNfv9rOf+63yvnd0zX73H9q2fLuR8eAFhJ2mU82PkHAIAQ4h8AAEKIfwAACCH+AQAghPgHAIAQPpweUYO538r5N+c3l/OdM6fK+ckHZsr51o/r+cTExETTn19idQAAC2mX8WDnHwAAQoh/AAAIIf4BACCE+AcAgBDiHwAAQjjtZ1QNm3I8aOr3talePe9vqO8DALCitMtYsPMPAAAhxD8AAIQQ/wAAEEL8AwBACPEPAAAhxD8AAIRw1OeY+e6/TeW8f+XJcj7cOCjnvdn1rc9o+vPLXxgAQEG7jBY7/wAAEEL8AwBACPEPAAAhxD8AAIQQ/wAAEMJpPyOq7av1o8/vLOfvvXK8nL/10KvlfP+T+1qffe3rR5ZYHQDAQtplPNj5BwCAEOIfAABCiH8AAAgh/gEAIIT4BwCAEE77GTPrT/1bzo+dvamc777+dDmffmyu/SFvTNXz4WDRtQEAXEy7jBY7/wAAEEL8AwBACPEPAAAhxD8AAIQQ/wAAEMJpP2Nm+MOP5fztQ/eV8xef+KqcP7f1cOszDk7VX983vpgHAJZJu4wWO/8AABBC/AMAQAjxDwAAIcQ/AACEEP8AABDCaT/jZtiU4+l/euV80Awv52oAABanXUaKnX8AAAgh/gEAIIT4BwCAEOIfAABCiH8AAAgh/gEAIISjPsdM058v59ve/bOczz1ztpxvnDrX+oze7PplPRsAoI12GS12/gEAIIT4BwCAEOIfAABCiH8AAAgh/gEAIITTftaK4yfL8UfntpTzXVf81Hqr1+7cU84nP/ly+esCAKhol07Y+QcAgBDiHwAAQoh/AAAIIf4BACCE+AcAgBBO+1kjhmfOlPMXDj1aznc//HLrvY4/MlvOb/lsppw3/fklVgcAsJB26YadfwAACCH+AQAghPgHAIAQ4h8AAEKIfwAACOG0nzXuhsP1T3zgrntb/2bmL++EAEA3tMvl5T8FAAAhxD8AAIQQ/wAAEEL8AwBACPEPAAAhxD8AAIToNU2zag/bNfn46j2MRfWmZ1qvNYNBfWHYMl/jPhwe7HW9BgC6oV1Gh3a5dIu1i51/AAAIIf4BACCE+AcAgBDiHwAAQoh/AAAIsa7rBdCNpj/f9RIAAC6ZdlkZdv4BACCE+AcAgBDiHwAAQoh/AAAIIf4BACBEr2martcAAACsAjv/AAAQQvwDAEAI8Q8AACHEPwAAhBD/AAAQQvwDAEAI8Q8AACHEPwAAhBD/AAAQQvwDAEAI8Q8AACHEPwAAhBD/AAAQQvwDAEAI8Q8AACHEPwAAhBD/AAAQQvwDAEAI8Q8AACHEPwAAhBD/AAAQQvwDAEAI8Q8AACH+B4T8lifdadTTAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 1440x2160 with 16 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# get 8 random mis-classified index to display\n",
    "display_index =[]\n",
    "for i in range(8):\n",
    "    n = random.randint(0,len(miscla_index))\n",
    "    index = miscla_index[n]\n",
    "    display_index.append(index)\n",
    "print(\"displaying indexes are: \", display_index)\n",
    "\n",
    "# display misclassification letters\n",
    "fig, axs = plt.subplots(8,2, figsize=(20, 30))\n",
    "fig.subplots_adjust(hspace = 0.20, wspace=.005)\n",
    "axs = axs.ravel()\n",
    "for i in range(8):\n",
    "    for j,value in enumerate(display_index):\n",
    "        image = x_test[value].squeeze()\n",
    "        axs[2*i].axis('off')\n",
    "        axs[2*i].imshow(image)\n",
    "        axs[2*i].set_title('real label is {}'.format(y_labels[y_real_label[value]-1]))\n",
    "\n",
    "        axs[2*i+1].axis('off')\n",
    "        axs[2*i+1].imshow(image)\n",
    "        axs[2*i+1].set_title('predicted label is {}'.format(y_labels[y_pred[value]-1]))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Q: Recall that EMNIST Letters merges upper- and lowercase letters into the same class. How might you design a network that accounts for that difference?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Since there are upper and lower cases of letters, the features are doubled compared to just single type of letter (e.g. lower case letters). When we designed the network, we considered to employ twice of the convolutional layers as well as twice of the fully connected layers so that there are enough neorons to hold the features for both types of the letters.\n",
    "- In our experiment below, we tested the theory of using a pair of convnets (2 convolutions layers) and pairs of fully connected layers (2 or 4 fully connected layers respectively) with twice nodes of the training on MNIST. It shows that with only several epochs, the training works very well. The test accuracy reached above 90% without applying too much regularizations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(103958, 28, 28) (103958,) (20842, 28, 28) (20842,) (20800, 28, 28) (20800,)\n",
      "x_train shape: (103958, 28, 28, 1)\n",
      "103958 train samples\n",
      "103958 train samples\n",
      "20800 test samples\n",
      "Train on 103958 samples, validate on 20842 samples\n",
      "Epoch 1/5\n",
      "103958/103958 [==============================] - 125s 1ms/step - loss: 0.5276 - acc: 0.8379 - val_loss: 0.2799 - val_acc: 0.9094\n",
      "Epoch 2/5\n",
      "103958/103958 [==============================] - 118s 1ms/step - loss: 0.2705 - acc: 0.9092 - val_loss: 0.2179 - val_acc: 0.9288\n",
      "Epoch 3/5\n",
      "103958/103958 [==============================] - 122s 1ms/step - loss: 0.2221 - acc: 0.9234 - val_loss: 0.2018 - val_acc: 0.9328\n",
      "Epoch 4/5\n",
      "103958/103958 [==============================] - 121s 1ms/step - loss: 0.1969 - acc: 0.9308 - val_loss: 0.2054 - val_acc: 0.9311\n",
      "Epoch 5/5\n",
      "103958/103958 [==============================] - 121s 1ms/step - loss: 0.1774 - acc: 0.9370 - val_loss: 0.1822 - val_acc: 0.9386\n",
      "Test loss: 18.268253769555415 %\n",
      "Test accuracy: 93.96634615384616 %\n",
      "saved model to disk\n",
      "Train: 0.956, evaluate: 0.939\n",
      "Test: 0.940\n"
     ]
    }
   ],
   "source": [
    "#load dataset\n",
    "mat = sio.loadmat('data/emnist-letters.mat')\n",
    "data = mat['dataset']\n",
    "# https://stackoverflow.com/questions/51125969/loading-emnist-letters-dataset/53547262#53547262\n",
    "X_train = data['train'][0,0]['images'][0,0]\n",
    "y_train = data['train'][0,0]['labels'][0,0]\n",
    "X_test = data['test'][0,0]['images'][0,0]\n",
    "y_test = data['test'][0,0]['labels'][0,0]\n",
    "# reshape the data into 2D, 28x28 sized images instead of a 1D 784 array\n",
    "# to get the correct image orientation, need to do a numpy reshape using Fortran ordering\n",
    "# (Matlab uses column-major ordering, just like Fortran\n",
    "# https://stackoverflow.com/questions/51125969/loading-emnist-letters-dataset/53547262#53547262\n",
    "x_train = X_train.reshape((X_train.shape[0], 28, 28), order='F')\n",
    "y_train = y_train.reshape(-1)\n",
    "x_test = X_test.reshape((X_test.shape[0], 28, 28), order = 'F')   \n",
    "y_test = y_test.reshape(-1)\n",
    "# the data, split between train and validation sets\n",
    "x_train, x_valid, y_train, y_valid = train_test_split(x_train, y_train, test_size = 0.167, random_state = 0, shuffle=True)\n",
    "# we want to have the same size for valid and test\n",
    "print(x_train.shape,y_train.shape, x_valid.shape,y_valid.shape,x_test.shape,y_test.shape)\n",
    "\n",
    "batch_size = 654\n",
    "num_classes = 27 # the class label starts from 1-26, don't want to modify the label code, so apply a trick here\n",
    "epochs = 5 #100\n",
    "# input image dimensions\n",
    "img_rows, img_cols = 28, 28\n",
    "\n",
    "# TensorFlow's tensor expect the input shape (samples, rows, cols, channels) \n",
    "if K.image_data_format() == 'channels_first':\n",
    "    x_train = x_train.reshape(x_train.shape[0], 1, img_rows, img_cols)\n",
    "    x_valid = x_valid.reshape(x_valid.shape[0], 1, img_rows, img_cols)\n",
    "    x_test = x_test.reshape(x_test.shape[0], 1, img_rows, img_cols)\n",
    "    input_shape = (1, img_rows, img_cols)\n",
    "else:\n",
    "    x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)\n",
    "    x_valid = x_valid.reshape(x_valid.shape[0], img_rows, img_cols, 1)\n",
    "    x_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)\n",
    "    input_shape = (img_rows, img_cols, 1)\n",
    "\n",
    "# normalize the input data for reducing over-fitting\n",
    "x_train = x_train.astype('float32')\n",
    "x_valid = x_valid.astype('float32')\n",
    "x_test = x_test.astype('float32')\n",
    "x_train /= 255\n",
    "x_valid /= 255\n",
    "x_test /= 255\n",
    "print('x_train shape:', x_train.shape)\n",
    "print(x_train.shape[0], 'train samples')\n",
    "print(x_train.shape[0], 'train samples')\n",
    "print(x_test.shape[0], 'test samples')\n",
    "\n",
    "\n",
    "# convert class vectors to ont hot encoding\n",
    "y_train = keras.utils.to_categorical(y_train, num_classes,dtype='float32')\n",
    "y_valid = keras.utils.to_categorical(y_valid, num_classes,dtype='float32')\n",
    "y_test = keras.utils.to_categorical(y_test, num_classes,dtype='float32')\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Conv2D(32, kernel_size=(3, 3),\n",
    "                 activation='relu',\n",
    "                 input_shape=input_shape))\n",
    "model.add(BatchNormalization(axis=-1))\n",
    "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "model.add(Conv2D(64, (3, 3), activation='relu'))\n",
    "model.add(BatchNormalization(axis=-1))\n",
    "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "model.add(Dropout(0.25))\n",
    "model.add(Flatten())\n",
    "# model.add(Dense(128, activation='relu'))\n",
    "model.add(Dense(1568, activation='relu'))\n",
    "model.add(BatchNormalization(axis=-1))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(784, activation='relu'))\n",
    "model.add(BatchNormalization(axis=-1))\n",
    "model.add(Dense(num_classes, activation='softmax'))\n",
    "\n",
    "model.compile(loss=keras.losses.categorical_crossentropy,\n",
    "              optimizer=keras.optimizers.Adadelta(),\n",
    "              metrics=['accuracy'])\n",
    "tensorboard = TensorBoard(log_dir=\"logs_emnist_7/{}\".format(time()))\n",
    "# simple early stopping\n",
    "es = EarlyStopping(monitor='accuracy', mode='max', verbose=1)\n",
    "history= model.fit(x_train, y_train,\n",
    "          batch_size=batch_size,\n",
    "          epochs=epochs,\n",
    "          verbose=1,\n",
    "          callbacks=[tensorboard],\n",
    "          validation_data=(x_valid, y_valid))\n",
    "\n",
    "score = model.evaluate(x_test, y_test, verbose=0)\n",
    "print('Test loss:', score[0]*100,'%')\n",
    "print('Test accuracy:', score[1]*100,'%')\n",
    "model.save('emnist_v7.h5')\n",
    "print('saved model to disk')\n",
    "\n",
    "# evaluate the model\n",
    "_, train_acc = model.evaluate(x_train, y_train, verbose=0)\n",
    "_, valid_acc = model.evaluate(x_valid, y_valid, verbose=0)\n",
    "print('Train: %.3f, evaluate: %.3f' % (train_acc, valid_acc))\n",
    "\n",
    "_, test_acc = model.evaluate(x_test, y_test, verbose=0)\n",
    "print('Test: %.3f' % (test_acc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary of Results and Solutions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#### Summary of parameters and hyper-parameters\n",
    "- inspired by what Nelsen mentioned the paper in 2010 by Ciresan, Meier, Gambardella, and Schmidhuber they used \n",
    "only fully-connected layers (no convolutions). Here we used 2 convnet plus added more than 2 fully connected layers \n",
    "with many epoches (196) of training time.\n",
    "- 0.5 dropout in between each fully connected layers;\n",
    "- did not apply dropout to convolution layers since according to Nelsen, the convolutional layers have consideralbe\n",
    "inbuilt resistance to overfitting.\n",
    "- applied Batch normalization for each layer in between to solve the potential vanishing gradient problem and\n",
    "potentially dead ReLu problem. \n",
    "- employed early stopping, it shows the network was keep learning so it did not reach to early stop settings,\n",
    "but the number of datasets has been fully consumed (it stopped then)\n",
    "- Used optimizer=keras.optimizers.Adadelta() rather than SGD in Keras. The advantage for Adadelta is it can \n",
    "automaticaly update learning rate adaptively of its learning rate according to loss function. \n",
    "- We tested on SGD with eta=0.03 which produced very low test accuracy.\n",
    "- Expanded training dataset using Keras ImageDataGenerator() in MNIST dataset (see iteration 12). \n",
    "It did not help with model accuracy improvement. It might help in theory with the right parameters for data augmentation.\n",
    "- The fully connected layers with dropout helps gradually reduce over-fitting (better performance than fewer layers high \n",
    "droput)\n",
    "- added save the model to local disk to load in the future to re-produce the result\n",
    "- Used Relu activation, but it might have dead ReLu problem according to Standard CNN class. It worthy to try Leaky Relu.\n",
    "- displayed randome chosen mis-classified hand writing letters\n",
    "- displayed accuracy and loss in TensorBoard\n",
    "- relatively larger batch size helps with model performance."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}