TF_Deep_NN.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib\n",
    "import seaborn as sns\n",
    "matplotlib.rcParams['savefig.dpi'] = 144"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import time\n",
    "from datetime import datetime, timedelta\n",
    "\n",
    "from pylib.draw_nn import draw_neural_net_fig"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "sess = None\n",
    "\n",
    "def reset_vars():\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "\n",
    "def reset_tf():\n",
    "    global sess\n",
    "    if sess:\n",
    "        sess.close()\n",
    "    tf.reset_default_graph()\n",
    "    sess = tf.Session()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<!-- requirement: pylib/__init__.py -->\n",
    "<!-- requirement: pylib/draw_nn.py -->\n",
    "<!-- requirement: images/Accuracy_NoDropout.png-->\n",
    "<!-- requirement: images/Accuracy_Dropout.png -->\n",
    "\n",
    "# Deep Neural Networks\n",
    "\n",
    "## What is deep learning?\n",
    "\n",
    "Deep learning is a branch of machine learning that tries to emulate the biological structure and function of the brain using artificial neural networks. These networks include: \n",
    "\n",
    "- Multilayer Perceptron Networks\n",
    "- Convolutional Neural Networks\n",
    "- Recurrent Neural Networks\n",
    "\n",
    "Additionally, these networks are hierarchical or multilayered, enabling them to model high-level abstractions in data. For this reason, deep learning is also called **hierarchical learning**. (Notice how there are two hidden layers in the figure of the multilayer perceptron network below.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "draw_neural_net_fig([20, 14, 12, 10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "There are benefits to using hierarchical models. In contrast to the performance of older machine learning algorithms, the performance of deep learning algorithms scales with the amount of data they are trained on -- the more data, the better the model. Consequently, deep learning algorithms typically outperform traditional ones. These models also have the ability to automatically extract features from data in a process called [feature learning](https://en.wikipedia.org/wiki/Feature_learning). This ability eliminates the need for a priori knowledge of the data to construct features, which is particularly useful when dealing with complex data such as images.  \n",
    "\n",
    "Deep learning has some pretty neat applications. Not only can we classify images with a high degree of accuracy, but we can also use deep learning algorithms to [generate captions](https://research.googleblog.com/2016/09/show-and-tell-image-captioning-open.html), [summarize](https://research.googleblog.com/2016/08/text-summarization-with-tensorflow.html) and [translate](https://research.googleblog.com/2016/09/a-neural-network-for-machine.html) text, [generate audio](https://deepmind.com/blog/wavenet-generative-model-raw-audio/), and [produce art](https://github.com/lengstrom/fast-style-transfer/). \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Multilayer perceptron\n",
    "\n",
    "We will start by replicating some of the code we used for our basic neural network. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "mnist = input_data.read_data_sets('/tmp/data', one_hot=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "N_PIXELS= 28 * 28\n",
    "N_CLASSES = 10\n",
    "BATCH_SIZE = 100\n",
    "LEARNING_RATE = 0.5\n",
    "\n",
    "hidden_size = 64"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Initializing Weights and Biases\n",
    "\n",
    "As a reminder, we want to initialize our weights with random values to break symmetry between neurons in a hidden layer. Additionally, we want to choose small values to avoid the **gradient vanishing problem**, where the weighted sum of the inputs (plus a bias) fall on the flat portion of the sigmoid curve. What is the proper scale of the weights?  Most of our activation functions have their best response for inputs of $\\mathcal O(1)$.  If we have $m$ random inputs, each of $\\mathcal O(1)$, we expect their sum to scale as $\\sqrt m$.  Therefore, weights are often chosen randomly with a mean of zero and standard deviation of $1/\\sqrt m$.\n",
    "\n",
    "For very large layers, this gives rather small weights.  An alternative approach is to only provide $k < m$ non-zero weights when initializing neurons.  This scheme, known as **sparse initialization**, provides more diversity amongst the neurons at initialization.  It can, however, also produce very slow convergence as \"incorrect\" choices of non-zero weights have the be removed.\n",
    "\n",
    "In the code below, we initialize our weights by sampling from a truncated normal distribution, where any weights greater than 2 standard deviations from the mean is re-picked. We also initialize the biases to zero. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def initializer(shape):\n",
    "    return tf.truncated_normal(shape, stddev=shape[0]**-0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Adding Hidden Layers\n",
    "\n",
    "A single layer neural network only works well on linearly separable data. By adding one more layer, we can solve most classification problems. The exercise at the end of the basic neural network notebook was to add a layer to our model to improve the accuracy of our predictions. We will present the solution below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "reset_tf()\n",
    "\n",
    "x = tf.placeholder(tf.float32, [None, N_PIXELS], name=\"pixels\")\n",
    "y_label = tf.placeholder(tf.float32, [None, N_CLASSES], name=\"labels\")\n",
    "\n",
    "W1 = tf.Variable(initializer([N_PIXELS, hidden_size]), name=\"weights\")\n",
    "b1 = tf.Variable(tf.zeros([hidden_size]), name=\"biases\")\n",
    "\n",
    "hidden = tf.nn.sigmoid(tf.matmul(x, W1) + b1)\n",
    "\n",
    "W2 = tf.Variable(initializer([hidden_size, N_CLASSES]), name=\"weights2\")\n",
    "b2 = tf.Variable(tf.zeros([N_CLASSES]), name=\"biases2\")\n",
    "\n",
    "y = tf.matmul(hidden, W2) + b2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def merge_dicts(*args):\n",
    "    d = dict()\n",
    "    for a in args:\n",
    "        d.update(a)\n",
    "    return d\n",
    "\n",
    "def optimize(x, y, y_label, steps_total, steps_print, train_feed={}, test_feed={}):\n",
    "    loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=y,\n",
    "                                                                  labels=y_label))\n",
    "    train = tf.train.GradientDescentOptimizer(0.5).minimize(loss)\n",
    "    accuracy = tf.reduce_mean(tf.cast(tf.equal(tf.argmax(y, 1), tf.argmax(y_label, 1)), tf.float32))\n",
    "\n",
    "    reset_vars()\n",
    "    for i in xrange(steps_total):\n",
    "        batch_x, batch_y = mnist.train.next_batch(BATCH_SIZE)\n",
    "        sess.run(train,\n",
    "                 feed_dict=merge_dicts({x: batch_x, y_label: batch_y}, train_feed))\n",
    "        if i % steps_print == 0 or i == steps_total - 1:\n",
    "            feed_dict = merge_dicts({x: mnist.test.images, y_label: mnist.test.labels}, test_feed)\n",
    "            print \"Test: \", sess.run([loss, accuracy],\n",
    "                                     feed_dict=merge_dicts({x: mnist.test.images,\n",
    "                                                            y_label: mnist.test.labels},\n",
    "                                                           test_feed))\n",
    "            print \"Train:\", sess.run([loss, accuracy],\n",
    "                                     feed_dict=merge_dicts({x: mnist.train.images, \n",
    "                                                            y_label: mnist.train.labels},\n",
    "                                                           test_feed))\n",
    "\n",
    "optimize(x, y, y_label, 10000, 1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Layer API\n",
    "\n",
    "Setting up all of this math is obviously going to get tedious as we increase the number of layers. To address this, TensorFlow provides a [layers API](https://www.tensorflow.org/api_docs/python/tf/layers), which lets us create individual layers with a single line.  Let's recreate this two-layer network in the new API.\n",
    "\n",
    "The input tensors are created in the same way as before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "reset_tf()\n",
    "\n",
    "x = tf.placeholder(tf.float32, [None, N_PIXELS], name=\"pixels\")\n",
    "y_label = tf.placeholder(tf.float32, [None, 10], name=\"labels\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have been using **dense** layers.  That is, each neuron is connected to all of the inputs to the layer.  First we create thie hidden layer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hidden = tf.layers.dense(x, hidden_size, activation=tf.nn.sigmoid, use_bias=True,\n",
    "    kernel_initializer=tf.truncated_normal_initializer(stddev=N_PIXELS**-0.5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This sets up a weight matrix (referred to as the **kernel**, for reasons to be discussed later) of `size(x)`$\\times$`hidden_size`, multiplies it with $x$, adds a bias term (since `use_bias=True`), and sends the result through the sigmoid activation function.  We use the same truncated normal initializer for the weights as before.\n",
    "\n",
    "We use a second dense layer to produce the final output.  We don't need an activation function here, as we'll feed it into the softmax function ourselves."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = tf.layers.dense(hidden, 10, activation=None, use_bias=True,\n",
    "    kernel_initializer=tf.truncated_normal_initializer(stddev=hidden_size**-0.5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The training proceeds identically to before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "optimize(x, y, y_label, 10000, 1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "This API makes it easy to add layers and neurons to neural network. However, in doing so, we run the risk of overfitting our model. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Overfitting and dropout\n",
    "\n",
    "Consider a neural network with two hidden layers, with 400 neurons in each, that has been trained on the MNIST dataset. Now consider the figure below. It shows the model's performance on training (blue) and test (orange) data as a function of training steps. The gap between the training and test curves suggests that we are capturing too much of the variance in our training dataset such that our model doesn't generalize to new data. In other words, we have **overfit** the model. \n",
    "\n",
    "\n",
    "![No dropout](images/Accuracy_NoDropout.png)\n",
    "\n",
    "Overfitting occurs when there are too many hidden layers or neurons, while underfitting (when the model does not capture enough variance in the data) occurs when there are too few. A general rule of thumb is that the number of neurons in a hidden layer should be between the size of the input and output layers. \n",
    "\n",
    "There are several [strategies](http://neuralnetworksanddeeplearning.com/chap3.html#regularization) to prevent overfitting, but we will consider **dropout**. For dropout, we assign a probability that a neuron will remain in the network for each iteration of training. A common choice for this probability is 0.5, and you can read more about dropout in [Srivastava et al (2014)](http://jmlr.org/papers/volume15/srivastava14a/srivastava14a.pdf).  _Run the cell below to visualize how a neural network changes with dropout_. In TensorFlow, we apply the dropout regularization function right after the activation function:\n",
    "\n",
    "`p = tf.placeholder(float32)\n",
    "Y = tf.nn.relu(tf.matmul(x, weights) + biases)\n",
    "Y_ = tf.nn.dropout(Y, p)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "draw_neural_net_fig([20, 14, 12, 10], 0.7) #last argument is the probability of keeping a neuron"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "We do not apply dropout on the test data. Applying a dropout of $p=0.5$ produces the set of curves below. You can see that the gap is no longer present. \n",
    "\n",
    "![With dropout](images/Accuracy_Dropout.png)\n",
    "\n",
    "Dropout can be created through the layer API.  The dropout rate is a parameter.  By setting it to a placeholder, we can adjust the dropout rate later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "reset_tf()\n",
    "\n",
    "x = tf.placeholder(tf.float32, [None, N_PIXELS], name=\"pixels\")\n",
    "y_label = tf.placeholder(tf.float32, [None, 10], name=\"labels\")\n",
    "dropout = tf.placeholder(tf.float32)\n",
    "\n",
    "DROPOUT = 0.2\n",
    "LAYERS = [400, 400]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "layer = x\n",
    "for size in LAYERS:\n",
    "    layer = tf.layers.dense(layer, size, activation=tf.nn.relu, use_bias=True,\n",
    "        kernel_initializer=tf.truncated_normal_initializer(stddev=layer.shape[1].value**-0.5))\n",
    "    layer = tf.layers.dropout(layer, rate=dropout, training=True)\n",
    "\n",
    "y = tf.layers.dense(layer, N_CLASSES, activation=None, use_bias=True,\n",
    "    kernel_initializer=tf.truncated_normal_initializer(stddev=layer.shape[1].value**-0.5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "We apply dropout only during training.  We want to make predictions using the best model we have, namely by using all of the neurons we can.  The dropout layer's training flag can be used here, but we found it easier to just adjust the dropout rate to 0 in the test case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "BATCH_SIZE = 400\n",
    "n_steps = 300  # 3000\n",
    "optimize(x, y, y_label, 3000, 100, train_feed={dropout: DROPOUT}, test_feed={dropout: 0})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "We can also make predictions using our model by running the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def predict(idx):\n",
    "    image = mnist.test.images[idx]\n",
    "    return sess.run(tf.argmax(y, 1), feed_dict={x: [image], dropout: 0})\n",
    "\n",
    "idx = 0\n",
    "\n",
    "plt.imshow(mnist.test.images[idx].reshape((28, 28)), cmap=plt.cm.gray_r)\n",
    "plt.title(\"Predicted: %d, Actual: %d\" % (predict(idx), np.argmax(mnist.test.labels[idx])));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Exercise: Adding flexibility\n",
    "\n",
    "Try increasing the number of **hidden layers** and **neurons**. How does increasing these values influence the model's accuracy (for both training and test data)? The time it takes to train the model? The time it takes the model's accuracy to converge? Are you overfitting you model with these values? How do you know?\n",
    "\n",
    "## Exercise: Adding dropout\n",
    "\n",
    "Play around with the values of **dropout**. How does changing the dropout rate change the fit of your model? \n",
    "\n",
    "## Exercise: Changing the learning rate\n",
    "\n",
    "Finally, play around with the **learning rate**. In particular, try using the value for your solution to the basic neural networks exercise. What do you notice?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "*Copyright &copy; 2017 The Data Incubator.  All rights reserved.*"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}