TF_Convolutional_NN.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib\n",
    "import seaborn as sns\n",
    "matplotlib.rcParams['savefig.dpi'] = 144"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import time\n",
    "from datetime import datetime, timedelta\n",
    "\n",
    "import pylib.conv_widget as cw\n",
    "from pylib.tensorboardcmd import tensorboard_cmd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "sess = None\n",
    "\n",
    "def reset_vars():\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "\n",
    "def reset_tf():\n",
    "    global sess\n",
    "    if sess:\n",
    "        sess.close()\n",
    "    tf.reset_default_graph()\n",
    "    sess = tf.Session()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<!-- requirement: pylib/__init__.py -->\n",
    "<!-- requirement: pylib/conv_widget.py -->\n",
    "<!-- requirement: images/conv_movie.gif -->\n",
    "<!-- requirement: images/portal-v.png -->\n",
    "<!-- requirement: images/portal-h.png -->\n",
    "<!-- requirement: images/conv_weights.png -->\n",
    "<!-- requirement: images/conv_layer.png -->\n",
    "<!-- requirement: images/CNN_schematic.png -->\n",
    "\n",
    "# Convolutional Neural Networks\n",
    "\n",
    "In the previous exercise, we flattened the MNIST images into vectors before feeding them into our model. Doing so, however, destroys information about shapes that are present in clusters of neighboring pixels. We can preserve this two dimensional information by using Convolutional Neural Networks (CNNs). \n",
    "\n",
    "CNNs are inspired by the [visual cortex](https://en.wikipedia.org/wiki/Visual_cortex), which is responsible for image processing in animals. Neurons in the visual cortex fire in response to stimuli that activate small sub-regions of an animal's visual field. We can therefore think of these neurons as filters that detect patterns across the animal's visual field. \n",
    "\n",
    "We will train a CNN to find a set of filters that can detect repeated shapes in MNIST images. Convolving the image with a filter produces another image with highlighted features called a feature map. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Convolutions\n",
    "\n",
    "A convolution is a mathematical operation that is performed on two functions. For continuous functions, it is defined as:\n",
    "\n",
    "$$ (f * g)(t) = \\int_{-\\infty}^{\\infty} f(\\tau)g(t-\\tau) d\\tau $$\n",
    "\n",
    "For discrete functions, it is:\n",
    "\n",
    "$$ (f * g)(t) = \\sum_{m=-\\infty}^{\\infty} f[m] g[n-m] $$\n",
    "\n",
    "A function that is a convolution of two other functions can be thought of as the area between the two original functions as one function is translated. \n",
    "\n",
    "![convolution](images/conv_movie.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Convolving images and filters\n",
    "\n",
    "Play around with the widget below, which demonstrates the process of convolving a 7x7 pixel image with a 3x3 pixel filter. At each iteration, the value of one pixel in the image (denoted by the red square) is replaced by the average of the pixel values within the 3x3 filter. In convolutional neural networks, a weight is assigned to each pixel in the filter, and consequently, the \"red pixel\" is replaced by weighted sum of the surrounded pixels. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "image = np.zeros((7, 7), dtype=float)\n",
    "for i in xrange(1, 6):\n",
    "    for j in xrange(6-i, 6):\n",
    "        image[i][j] = 1\n",
    "\n",
    "img_filter = np.ones((3, 3), dtype=float)\n",
    "img_filter /= img_filter.size\n",
    "\n",
    "titles = ('Image and kernel', 'Filtered image')\n",
    "convwidget = cw.ConvWidget()\n",
    "convwidget.interactive_convolution_demo(image, img_filter, vmax=1, titles=titles)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "#### Strides\n",
    "\n",
    "In the interactive widget, we perform the convolution at each pixel. However, we can choose to skip pixels. In TensorFlow, we can set the number of pixels our filter moves at each step, or the **stride**. How does the stride effect the dimension of the filtered image? Why would we want to change the stride?\n",
    "\n",
    "In TensorFlow's `conv2d` function, we set the stride with `strides=[batch, stride_width, stride_height, channels]`. For the MNIST dataset, `strides=[1, k, k, 1]`. Since we are dealing with grayscale images, the number of channels is 1. If we had colored images, this value would be set to 3 (corresponding red, green, and blue values). \n",
    "\n",
    "#### Padding\n",
    "\n",
    "Notice how the filter extends past the edge of the image while performing convolutions on some pixels. To deal with these edge cases, we typically **pad** the image or add more pixels to the perimeter of the image. This process ensures that the dimensions of the input image are the same as the dimensions of the output image. In the case of the MNIST dataset, the image is padded with zeros (or white space). When might we run into problems when padding images?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Example: Applying a simple edge filter on a handwritten digit\n",
    "\n",
    "The magic behind the convolutional neural network lies in the fact that each filter learns to recognize certain patterns repeated throughout images. To build our intuition for why this is the case, we will consider two very simple filters $(1,-1)$ and $(1,-1)^T$ applied to an image consisting of only black and white pixels (with values of 1 or 0). The image consists of a black, horizontal line connected to a black, vertical line and roughly resembles a \"7.\" See the cell below. We see from the two resulting convolved images that the filters picked out vertical and horizontal edges, respectively. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from scipy import signal as sg\n",
    "ex_image = [[0, 0, 0, 0, 0],\n",
    "            [1, 1, 1, 1, 0], # <---horizontal line of ones\n",
    "            [0, 0, 0, 1, 0],\n",
    "            [0, 0, 0, 1, 0],\n",
    "            [0, 0, 0, 1, 0]]\n",
    "                    # ^ vertical line of ones\n",
    "\n",
    "print(sg.convolve(ex_image, [[1, -1]], \"same\")) #vertical edge\n",
    "print(sg.convolve(ex_image, [[1], [-1]], \"same\")) #horizontal edge\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "If we apply the vertical and horizontal edge filters a handwritten 7, we get the following feature maps:\n",
    "<table>\n",
    "    <tr>\n",
    "        <td> <img src=\"files/images/portal-v.png\" style=\"width: 250px;\"/> </td>\n",
    "        <td> <img src=\"files/images/portal-h.png\" style=\"width: 250px;\"/> </td>\n",
    "    </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Convolutional neural networks\n",
    "\n",
    "Convolutional neural networks are comprised of a series of convolutional and max pooling layers, followed by a fully connected layer. Each of these will be explained in detail below. Let's first start off by repeating some of the steps we did in the previous notebook. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "reset_tf()\n",
    "\n",
    "# Load data\n",
    "data = input_data.read_data_sets('/tmp/data/', one_hot=True)\n",
    "\n",
    "# Get class (number) for test data\n",
    "data.test.cls = np.argmax(data.test.labels, axis=1)\n",
    "\n",
    "# Set path to summary logs\n",
    "now = datetime.now()\n",
    "logs_path = now.strftime(\"%Y%m%d-%H%M%S\") + '/summaries'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now let's set our model parameters and define our placeholder variables. This step is very similar to the one we did in the previous notebook, but now we assign the number of channels to `n_channels` and the filter size to `filt_size`. For the sake of simplicity, we will not use dropout in this model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Model parameters\n",
    "img_size = 28\n",
    "img_size_flat = img_size * img_size\n",
    "img_shape = (img_size, img_size)\n",
    "\n",
    "n_classes = 10\n",
    "n_channels = 1\n",
    "filt_size = [5, 5] # 5x5 pixel filters\n",
    "\n",
    "batch_size = 50\n",
    "num_iterations = 400 #1500\n",
    "display_step = 100\n",
    "\n",
    "# Placeholder variables\n",
    "x = tf.placeholder(tf.float32, shape=[None, img_size_flat], name='x')\n",
    "y_true = tf.placeholder(tf.float32, shape=[None, n_classes], name='y_true')\n",
    "y_true_cls = tf.argmax(y_true, dimension=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Convolutional layers\n",
    "\n",
    "The purpose of the convolutional layer is to extract features from images. In this layer, a set of filters are convolved with some input to produce activation or feature maps. Then, the ReLU operation is applied to each pixel in the feature maps, converting negative pixel values to zero. During the training process, the CNN learns which filters activate when a certain features are present. \n",
    "\n",
    "In the example below, we choose 16 filters and a stride of 1. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "x_reshape = tf.reshape(x, shape=[-1, img_size, img_size, 1])\n",
    "out_conv = tf.layers.conv2d(x_reshape, 16, filt_size, padding='same', activation=tf.nn.relu, name=\"convolution\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will add the kernels to the TensorBoard output, so that we can visualize them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "kernels = tf.get_default_graph().get_operation_by_name('convolution/kernel').values()[0]\n",
    "shape = kernels.shape.as_list()  #[dx, dy, ch, n_filters]\n",
    "w = tf.reshape(kernels, [shape[3], shape[0], shape[1], shape[2]])\n",
    "tf.summary.image('kernel_weights', w, max_outputs=shape[3], collections=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "After training, our 16 filters will look similar to those shown in the figure on the left. Red represents positive values and blue represents negative values. If we convolve an image of a seven with these filters, we will get the feature maps in the figure on the right.\n",
    "\n",
    "<table>\n",
    "    <tr>\n",
    "        <td> <img src=\"files/images/conv_weights.png\" style=\"width: 450px;\"/> </td>\n",
    "        <td> <img src=\"files/images/conv_layer.png\" style=\"width: 450px;\"/> </td>\n",
    "    </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Max pooling \n",
    "In image classification, the approximate locations of shapes and edges are more important than their exact locations. To add this type of flexibility into our model, we can downsample our convolved images. A variety of techniques for downsampling exist, but **max pooling** is a popular one. Max pooling partitions an image and outputs the maximum value of each partition. A commonly used partition or filter size is 2x2. In TensorFlow, we set the stride of this filter to its dimension, such that pooling regions do not overlap with one another. \n",
    "\n",
    "What happens to the resolution of an nxn image after this type of pooling?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "out_pool = tf.layers.max_pooling2d(out_conv, pool_size=(2, 2), strides=(2,2), padding='same')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Fully connected layer\n",
    "\n",
    "Notice that a convolution-ReLU-pooling layer operates on a single feature map (one for each convolutional filter in the previous layer), where each map represents a high-level feature in the input image. The goal of our convolutonal neural network is to (1) identify each feature in the image, (2) identify combinations of these features in the image, and then (3) use this information to classify the image. To this end, we eventually want to connect every neuron to one another in a **fully connected layer**. \n",
    "\n",
    "You can think of a fully connected layer as a layer in a basic or flat neural network. To create this layer in TensorFlow, we reshape our 3-dimensional output into a vector and then apply the weights and biases. The size of the weights matrix is the total number of pixels (7 $*$ 7 $*$ 16) by the number of desired outputs. In this example, we choose 100 as the output size. \n",
    "\n",
    "You can have more than one fully connected layer, but the final activation function should be softmax. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "out_pool_reshape = tf.reshape(out_pool, [-1, out_pool.shape[1:].num_elements()])\n",
    "out = tf.layers.dense(out_pool_reshape, 100, activation=tf.nn.relu)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Class prediction\n",
    "\n",
    "Finally we pass this output through another layer with output size `n_classes`. This step is necessary to get our predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "y_pred = tf.layers.dense(out, n_classes, activation=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Completing the model\n",
    "\n",
    "We then define our loss and accuracy as we did in our previous models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Loss\n",
    "loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=y_pred, labels=y_true))\n",
    "tf.summary.scalar('loss', loss)\n",
    "\n",
    "# Optimizer\n",
    "optimizer = tf.train.AdamOptimizer().minimize(loss)\n",
    "\n",
    "# Accuracy\n",
    "y_pred_cls = tf.argmax(y_pred, dimension=1)\n",
    "correct_prediction = tf.equal(y_pred_cls, y_true_cls)\n",
    "accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n",
    "tf.summary.scalar('accuracy',accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "We will then initialize our variables, launch our graph, and train our model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Merge all summaries\n",
    "merged = tf.summary.merge_all()\n",
    "\n",
    "# Initialize\n",
    "reset_vars()\n",
    "\n",
    "# Create summary writers\n",
    "train_writer = tf.summary.FileWriter(logs_path + '/train', graph=tf.get_default_graph())\n",
    "test_writer = tf.summary.FileWriter(logs_path + '/test', graph=tf.get_default_graph())\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "def optimize(num_iterations):\n",
    "    # Start-time used for printing time-usage below.\n",
    "    start_time = time.time()\n",
    "\n",
    "    step = 1\n",
    "    for i in range(num_iterations):\n",
    "        \n",
    "        # Get a batch of training examples.\n",
    "        x_batch, y_true_batch = data.train.next_batch(batch_size)\n",
    "\n",
    "        # ---------------------- TRAIN -------------------------\n",
    "        # Optimize model\n",
    "        sess.run(optimizer, feed_dict={x: x_batch,y_true: y_true_batch}) \n",
    "        \n",
    "        \n",
    "        # Print status every 100 iterations.\n",
    "        if (i % display_step == 0) or (i == num_iterations - 1):\n",
    "            \n",
    "            summary = sess.run(merged, feed_dict={x: x_batch, y_true: y_true_batch})\n",
    "            train_writer.add_summary(summary, step)\n",
    "            \n",
    "            #----------------------- TEST ---------------------------\n",
    "            # Test model\n",
    "            summary, l, acc = sess.run([merged, loss, accuracy], feed_dict={x: data.test.images,\n",
    "                                                                            y_true: data.test.labels})                                                          \n",
    "            test_writer.add_summary(summary, step)\n",
    "            \n",
    "            # Message for network evaluation\n",
    "            msg = \"Optimization Iteration: {0:>6}, Test Loss: {1:>6}, Test Accuracy: {2:>6.1%}\"\n",
    "            print(msg.format(i, l, acc))\n",
    "            \n",
    "            step += 1\n",
    "\n",
    "    # Ending time.\n",
    "    end_time = time.time()\n",
    "\n",
    "    # Difference between start and end-times.\n",
    "    time_dif = end_time - start_time\n",
    "\n",
    "    # Print the time-usage.\n",
    "    print(\"Time usage: \" + str(timedelta(seconds=int(round(time_dif)))))\n",
    "    \n",
    "optimize(num_iterations)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "We will close our summary writers. We can also view our graph on TensorBoard. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Close summary writers\n",
    "train_writer.close()\n",
    "test_writer.close()\n",
    "\n",
    "tensorboard_cmd(logs_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Let's use our model to make a prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "prediction = tf.argmax(y_pred, 1)\n",
    "\n",
    "def predict(idx):\n",
    "    image = data.test.images[idx]\n",
    "    return sess.run(prediction, feed_dict={x: [image]})\n",
    "\n",
    "idx = 0\n",
    "actual = np.argmax(data.test.labels[idx])\n",
    "print (\"Predicted: %d, Actual: %d\" % (predict(idx), actual))\n",
    "plt.imshow(data.test.images[idx].reshape((28,28)), cmap=plt.cm.gray_r)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Summary\n",
    "\n",
    "We go through the following steps when training a CNN:\n",
    "1. Initialize the filters/weights and biases with random values.\n",
    "2. Perform a series of convolution, ReLU, and pooling operations to produce feature maps.\n",
    "3. Flatten the feature maps and apply weights and biases\n",
    "4. Pass the output from the fully connected layer through a final, softmax layer\n",
    "5. Calculate loss.\n",
    "6. Minimize loss via gradient descent and adjust values of filters/weights and biases. \n",
    "7. Repeat previous steps with all training images.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Exercise: Improving the network architecture\n",
    "\n",
    "We can improve the model's accuracy by adding more layers and filters. Rewrite the model so that its architecture matches that in the figure below. Bonus points if you use dropout. \n",
    "\n",
    "![CNN schematic](images/CNN_schematic.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "*Copyright &copy; 2017 The Data Incubator.  All rights reserved.*"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  },
  "widgets": {
   "state": {
    "757efa5b5613458189995bbbe076e124": {
     "views": [
      {
       "cell_index": 5
      }
     ]
    }
   },
   "version": "1.2.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}