TF_Intro_to_Tensorflow.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib\n",
    "import seaborn as sns\n",
    "matplotlib.rcParams['savefig.dpi'] = 144"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import expectexception\n",
    "\n",
    "from pylib.draw_graph import draw_graph"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<!-- requirement: pylib/__init__.py -->\n",
    "<!-- requirement: pylib/draw_graph.py -->\n",
    "\n",
    "# Introduction to TensorFlow\n",
    "\n",
    "As artificial intelligence has advanced, applications have needed to process larger and larger amounts of data.  A prime example of this is the field of neural networks.  These machine learning models are enormously flexible and powerful, but they require huge amounts of both time and data to train.  Building tools to accomplish this can be difficult.\n",
    "\n",
    "Google had been using neural networks internally for some time when they decided to develop a single framework flexible enough to be used for many different purposes.  That project became TensorFlow, and it was released as an open-source project in November, 2015.  Version 1.0 was released just over a year later, in February, 2017.\n",
    "\n",
    "This generality has made TensorFlow useful for a number of different applications.  At its heart, it is a general-purpose computation engine, optimized for large data.  With TensorFlow, you define your program in terms of a computation graph.  Then, the graph is executed by letting data, in the form of tensors, flow through it.  (And you thought the name was just chosen to sound good!)\n",
    "\n",
    "This course will introduce you to both the basics of TensorFlow and the basics of neural networks.  By focusing on the basics, we hope to convey a solid understanding of the theory and practice behind neural networks in TensorFlow.  For purposes of time, many of our example will be small and short, and therefore will not produce spectacular results.  Nonetheless, this should be enough to get you started on your own spectacular projects!\n",
    "\n",
    "### Topics Covered\n",
    "\n",
    "- Introduction to TensorFlow\n",
    "\n",
    "\n",
    "- Iterative Algorithms\n",
    "\n",
    "\n",
    "- Machine Learning\n",
    "\n",
    "\n",
    "- Basic Neural Networks\n",
    "\n",
    "\n",
    "- Deep Neural Networks\n",
    "- Variational Autoencoders\n",
    "\n",
    "\n",
    "- Convolutional Neural Networks\n",
    "\n",
    "\n",
    "- Adversarial Noise\n",
    "- DeepDream\n",
    "\n",
    "\n",
    "- Recurrent Neural Networks\n",
    "\n",
    "## Flow: The Computation Graph\n",
    "\n",
    "In TensorFlow, computations are conceptualized as graphs.  These graphs are **directed** (the edges have a direction) and **acyclic** (there are no loops).  The nodes of the are **operations**, which may have zero or more inputs and zero or more outputs.  Hooking the output of one operation into the input of the next produces the computation graph.\n",
    "\n",
    "Let's start with a simple demonstration.  Before we get going, we need to import TensorFlow.  The module name is `tensorflow`, but the usual convention is to import it as `tf`.  This avoids the namespace pollution that comes with an `import *`, but saves a lot of keystrokes over time.  We'll also import numpy, a Python library for numerical computation, as `np`, for future use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "As a basic computation, let's reimplement this pure-Python calculation:\n",
    "```python\n",
    "x = 2\n",
    "y = 3\n",
    "sum_ = x + y\n",
    "```\n",
    "It should come as no surprise that addition is an operation.  In fact, it's implemented in the `tf.add()` function.  If we were to draw a graph of just the addition function, it would look something like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "draw_graph(\"add-op\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The add operation takes two inputs and returns a single output.  Where do these inputs come from?  Other operations.  In this case, we'll use the `tf.constant()` operation.  This takes no input and produces a single output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "draw_graph(\"const-op\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "We'll make a constant for each of `x` and `y` and connect these into the inputs of the add operation.  Constants can be made as follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "x = tf.constant(2, name=\"input_x\")\n",
    "y = tf.constant(3, name=\"input_y\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The `tf.constant()` function takes a single required argument, the value of the constant in question.  The `name` keyword is optional.  It helps to identify operations when examining the graph, so naming key operations will certainly save you time later.\n",
    "\n",
    "`x` and `y` are the output of operations, so they are tensors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "We'll discuss what exactly a tensor is later on.  For now, it's enough to remember that all values are tensors, so all input and output to operations should be tensors.\n",
    "\n",
    "Now let's hook them up to calculate `x + y`.  The resulting graph should look like this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "draw_graph(\"simple-graph\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "sum_ = tf.add(x, y, name=\"sum\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The `tf.add()` operation also takes an optional name.  The result is another tensor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "sum_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "But what is the value of this tensor?  It doesn't have one, at least not yet.  TensorFlow takes a **lazy evaluation** model.  Values are not calculated until they are actually needed.  When we set `sum_` to be the output tensor of the add operation, that information was recorded into the computation graph, but nothing was calculated.  (The same thing also holds for `x` and `y`.)\n",
    "\n",
    "To find the value of `sum_`, we need to TensorFlow that this value should be computed.  We do that with the `Session.run()` function.  (We'll discuss sessions shortly.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "sess = tf.Session()\n",
    "sess.run(sum_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "When we ran the `sum_` tensor, Tensorflow could look at the computation graph and figure out that this was the output of the add operation.  Therefore, it needs to compute the inputs to this operation, namely the outputs of our two constant operations.  These tensors are fed into the add operation, and the output is fed back into Python for our use.  Graphically, something like this happened:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "draw_graph(\"simple-graph-run\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Tensors override a number of Python's operators become TensorFlow operators.  We could have also have written"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "sum2 = x + y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Again, sum2 is a tensor, and its value is not calculated until we run the tensor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "sum2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "sess.run(sum2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The downside of this is that we could not give a name to this operation.  This doesn't affect the result of the computation, but it may make it harder to introspect the computation graph later.\n",
    "\n",
    "The standard mathematical operations are overwritten.  Be careful about `==`.  While it has been overridden, it only actually just checks identity.  To check if two tensors have the same values, you must use the `tf.equals()` operation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "tf.constant(7) == tf.constant(7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "sess.run(tf.equal(tf.constant(7), tf.constant(7)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Exercise: Implementing a basic graph\n",
    "\n",
    "Implement the following graph in TensorFlow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "draw_graph(\"exercise\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Run the output and check that you get the right answer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Repeating calculations\n",
    "\n",
    "All of this seems like a lot of work to just add a couple of numbers together.  And it is, really&mdash;if you're just doing a calculation once, setting it up in TensorFlow is probably overkill.  Most of the time, though, we'll want to repeat calculations multiple times.  Then the work of setting up the computation graph will pay off.\n",
    "\n",
    "Part of this is the ability to compute only the necessary parts of a computation graph.  If you need to execute a partial result, you don't need to think about what leads into that computation.  TensorFlow will do exactly what needs to be computed and no more.\n",
    "\n",
    "TensorFlow also allows you to repeat a calculation with a different input value.  You can change the values of constants by feeding new values in through the `feed_dict` argument of the `run()` method.  For example,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "sess.run(sum_, feed_dict={x: 10, y: 20})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Note that the key in the `feed_dict` is the *tensor* itself, not its name.\n",
    "\n",
    "Many times we will do set up computation graphs with the intention of feeding in values each time.  To make sure we don't forget any keys in the `feed_dict`, we can use **placeholders**.  A placeholder is an operation much like a constant, but it needs to have a value fed in each time the graph is run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "xp = tf.placeholder(dtype=tf.int32)\n",
    "sum_p = tf.add(xp, y)\n",
    "sess.run(sum_p, {xp: 5})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "If we don't specify the feed values, we get an error.  (Note that the error occurs during the execution, and therefore comes from another process.  The result is that the traceback gets very hard to read.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "%%expect_exception tf.errors.InvalidArgumentError\n",
    "\n",
    "sess.run(sum_p)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "You may have noticed that we had to specify the `dtype` of our placeholder tensor.  That means that it's time to discuss what exactly we mean by \"tensor\".\n",
    "\n",
    "## Tensor: The data types and shapes\n",
    "\n",
    "For our purposes, a tensor can be thought of as a generalization of a matrix.  A vector is a one-dimensional array of numbers and a matrix is a two-dimensional array.  A tensor then is an *n*-dimensional array of numbers.  Both vectors and matrices are just special cases of tensors.  (Mathematicians may quibble with this definition, insisting that tensors are defined by how they transform under coordinate transformations.  They have a point, but one that doesn't matter for the purposes of TensorFlow.)\n",
    "\n",
    "Thus, when specifying a tensor, we must give both the number of dimensions and the size of each dimension.  This is done through the `shape` data, which contains the size of each dimension.  If we look at an existing tensor, we can see that it has a shape of `[]`, indicating a scalar."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "x.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "If we make a $2\\times3$ matrix, for example, we see that this is reflected in its shape."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "tf.constant([[1,2,3],[4,5,6]]).shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "In this case, TensorFlow is able to work out the shape from the input values.  With placeholders, however, we generally will need to specify it.  We can give the dimensions in a tuple."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "mat = tf.placeholder(dtype=tf.int32, shape=(2, 3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now we have to give an appropriately shaped value in the `feed_dict`.  Note that this check happens at \"compile\" time, before any computation actually takes place."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "%%expect_exception ValueError\n",
    "\n",
    "sess.run(mat, {mat: [[1,2],[3,4]]})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "But if we give it the right-sized input, everything works just fine."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "sess.run(mat, {mat: [[1,2,3],[4,5,6]]})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "We can use a dimension of `None` to indicate that a tensor can have arbitrary size along that dimension.  (Dimension mismatches here can only be caught at execution time.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "mat2 = tf.placeholder(dtype=np.int32, shape=(2, None))\n",
    "print sess.run(mat2, {mat2: [[1],[2]]})\n",
    "sess.run(mat2, {mat2: [[1,2,3],[4,5,6]]})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The `array` object returned is provided by [NumPy](http://www.numpy.org/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "type(sess.run(mat2, {mat2: [[1,2,3],[4,5,6]]}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "TensorFlow uses NumPy arrays to represent all (non-scalar) values.  NumPy arrays are used widely in the scientific Python community for numeric computations, as they represent a significant improvement over Python lists for these purposes.\n",
    "\n",
    "Python lists are **heterogenous** and **resizable**.  This makes them useful for interactive programming, as any value can be added to any list.  However, it slows computation, since we have to do a pointer lookup to access each value (and additional lookups to find the relevant functions).\n",
    "\n",
    "In contrast, NumPy arrays are **homogenous** and **fixed-size**.  This means that their contents can be stored contiguously in memory.  Operations can easily be broadcast to act over each element in the array.  Arrays can be of arbitrary dimensions, so they are a good fit for tensors.  NumPy arrays can be constructed out of nested lists, which we were taking advantage of previously.\n",
    "\n",
    "Because they are homogenous, arrays (and therefore tensors) must specify which type of data they can contain.  This is indicated by an array's `dtype`.  The issue of type goes beyond simply floats versus ints&mdash;there are a variety of levels of precision.  As we can see above, the default integer type is 32 bits.  These types exist in both the `numpy` and `tensorflow` namespaces, where there are generally equivalent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "np.int32 == tf.int32"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "We can specify which dtype to use when creating operators."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "x_int64 = tf.constant(2, dtype=np.int64)\n",
    "x_float16 = tf.constant(2, dtype=np.float16)\n",
    "sess.run([x_int64, x_float16])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "While we can often get away with type inference, this sometimes leads to type mismatches in graphs.  It's generally good practice to specify the type, just to be safe.\n",
    "\n",
    "Here are the dtypes provided by numpy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "np.sctypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Most operations basic TensorFlow operations act element-wise.  This means that the shape of the output tensor is the same as that of the input tensors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "mat.shape, (mat + mat).shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Be aware the multiplication is also element-wise by default."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "sess.run(mat2 * mat2, {mat2: [[1,2],[3,4]]})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "To actually do matrix multiplication, use the `tf.matmul` operation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "sess.run(tf.matmul(mat2, mat2), {mat2: [[1,2],[3,4]]})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Obviously, matrix multiplication may result in a differently-shaped output.\n",
    "\n",
    "TensorFlow provides a number of `reduce_*` operations.  These run the specified reduction function over the specified axis, or over all axes if none is specified.  This will sum the columns of the matrix, for example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "sess.run(tf.reduce_sum(mat, axis=0), {mat: [[1,2,3],[4,5,6]]})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Exercise: Reducing tensors of arbitrary shape\n",
    "\n",
    "Write a graph to take in a $5\\times n$ tensor of floats and return a single float, the sum of the squares of the average value of each row.  Test it on the following inputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "np.random.seed(1)\n",
    "input1 = np.random.randn(5, 2)\n",
    "# Answer: 1.6559478"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "np.random.seed(2)\n",
    "input2 = np.random.randn(5, 200)\n",
    "# Answer: 0.035791952"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Graphs\n",
    "\n",
    "Thus far, we have been referring to the computation graph as some abstract concept.  It's actually a real thing.  TensorFlow provides a default graph, to which operations are added by default.  We can get a reference to it, if we'd like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "g = tf.get_default_graph()\n",
    "\n",
    "print g"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "We can get a list of all the operations on the current graph:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "g.get_operations()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Those are all of the operations that we've created in this notebook.  We've been using the default graph for convenience, but it's better practice to use a dedicated graph for each computation.  (That said, remember that TensorFlow is smart enough to only evaluate the necessary operations to give us any particular result.)  We can instantiate a new graph like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "g2 = tf.Graph()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The easiest way to add operations to a particular graph is to use the `as_default()` context handler."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "with g2.as_default():\n",
    "    a = tf.constant(12, name=\"constant_a\")\n",
    "    b = tf.constant(16, name=\"constant_b\")\n",
    "    sum_g2 = a + b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "We can see these operations got added to the new graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "g2.get_operations()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "As an issue of best practices, you should not do as we've done here, using the default graph implicitly for some operations and a second graph explicitly for other.  If you need multiple graphs, it's best to assemble each in its own `with` block, so that there's no confusion about which graph is getting which operations.\n",
    "\n",
    "## Sessions\n",
    "\n",
    "We've already created a session to run operations previously.  The session abstracts the environment in which a graph is executed.  Each session is associated with a specific graph, so if we want to evaluate `sum_g2`, we'll have to make a session for it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "sess2 = tf.Session(graph=g2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now, we can run it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "sess2.run(sum_g2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Multiple sessions can reference the same graph.  Each of them keeps track of its internal state separately, as we'll see when discussing variables in the next notebook.  If you don't specify a graph when creating a session, the default graph is used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "tf.Session().graph == g"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "When you are finished with a session, remember to close it, to free the associated resources."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "sess2.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Alternatively, you can use sessions as context managers.  They close themselves on exit, so you don't have to do this explicitly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "with tf.Session(graph=g2) as s2:\n",
    "    print s2.run(sum_g2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "This is probably the best approach if you find yourself using multiple sessions.  However, in these notebooks, we tend to have sessions that need to span multiple cells.  We have a little reset function that we'll be using instead.  It closes the current session and opens a new one, in the global variable `sess`.  It also resets the default graph.  All of our work will be in these objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def reset_tf():\n",
    "    global sess\n",
    "    if sess:\n",
    "        sess.close()\n",
    "    tf.reset_default_graph()\n",
    "    sess = tf.Session()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "> **Aside:** TensorFlow also provides an `InteractiveSession` designed for use in interactive environments like this one.  It allows you to use the `eval()` method on tensors to get their values in the default session.  We think \"Explicit is better than implicit,\" at least for learning, so we're using sessions explicitly.  But once you get a handle on these concepts, you might find that `InteractiveSession` speeds up your work.\n",
    "\n",
    "## TensorBoard\n",
    "\n",
    "[TensorBoard](https://www.tensorflow.org/get_started/summaries_and_tensorboard) is a web-based tool to help you visualize your TensorFlow graphs and runs.  While running your graphs, you write to event files.  When you launch TensorBoard, it will run a local webserver with visualizations of this material.  A `FileWriter` is used to create these files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "writer = tf.summary.FileWriter('./tb/intro_graph', g)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now you can launch tensorboard.  On a terminal in this directory, run\n",
    "```bash\n",
    "tensorboard --logdir=tb/intro_graph\n",
    "```\n",
    "This will start up a webserver on port 6006.  Visiting it, you will see nothing on the first page.  If you click on the \"Graphs\" header, though, you'll get a visualization of the computation graph.  This allows you to zoom and scroll through the graph, examining the operations and the tensors passed between them.  Right now, you'll probably see a lot of small, disconnected sub-graphs.  That's what's constructed in this notebook, but this is an atypical use.\n",
    "\n",
    "Another useful capability of TensorBoard is to keep track of some value each time a computation is run.  (This will be useful to track the evolving loss as a model is being trained, for example.)  For example, to track `sum_` each time we run it, we can create a summary operation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "tf.summary.scalar('sum_', sum_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "We also need to merge all of the summaries together.  It's sort of pointless here, with a single summary, but this would allow us to generate multiple summaries in a single operation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "merged = tf.summary.merge_all()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now, we can run both the summation of the summary generation step and use the `writer.add_summary` method to write the summary to the event file.  This function takes an optional second argument, a step counter.  This allows you to track how a summary value changes over time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "for i in xrange(10):\n",
    "    summary, _ = sess.run([merged, sum_], {x: 10*i, y: -i*i})\n",
    "    writer.add_summary(summary, i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "It's good practice to close the `FileWriter` when you're done with it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "writer.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "You'll need to stop and restart the tensorboard server.  Once you do, you should see *sum_* on the \"Scalars\" page, plotting how it changes over time.\n",
    "\n",
    "Be careful about re-running this code.  It will write another event file to the same directory, which can slightly confuse tensorboard.  One solution is to include the current time in the directory name passed to the `FileWriter`.\n",
    "\n",
    "We don't make too much use of TensorBoard in these notebooks, preferring visualizations that we've integrated into the notebook.  Don't let that dissuade you from using it in your own work!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "*Copyright &copy; 2017 The Data Incubator.  All rights reserved.*"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}