TF_Deep_Dream.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib\n",
    "import seaborn as sns\n",
    "matplotlib.rcParams['savefig.dpi'] = 144"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "\n",
    "import os\n",
    "\n",
    "from urllib2 import urlopen\n",
    "import tarfile\n",
    "import zipfile\n",
    "\n",
    "# Image manipulation.\n",
    "import PIL.Image\n",
    "from IPython.display import Image, display"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<!-- requirement: images/chipmunk.jpg -->\n",
    "\n",
    "# DeepDream and the Inception Model\n",
    "\n",
    "We can use deep learning to [create artwork](https://research.googleblog.com/2015/06/inceptionism-going-deeper-into-neural.html?m=1) as well. Using the [DeepDream](https://en.wikipedia.org/wiki/DeepDream) algorithm, we can modify images by amplifying the features detected by neural networks. We calculate the gradients in the model and then update the image accordingly. \n",
    "\n",
    "In this tutorial, we will use a pre-trained model called the Inception model. The Inception model has a very complicated architecture with many layers, and it takes weeks to train. You can read more about it in [this paper](https://arxiv.org/pdf/1512.00567v3.pdf). To create our artwork we will do the following:\n",
    "\n",
    "1. Select a layer in the inception model\n",
    "2. Feed the model an image\n",
    "3. Calculate the gradient in the layer with respect to the image\n",
    "4. Update the gradient to the image\n",
    "5. Repeat "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Download the Inception model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "data_url = \"http://storage.googleapis.com/download.tensorflow.org/models/inception5h.zip\"\n",
    "data_dir = \"inception/5h/\"\n",
    "file_path = os.path.join(data_dir, 'inception5h')\n",
    "\n",
    "if not os.path.exists(file_path):\n",
    "    # Check if the download directory exists, otherwise create it.\n",
    "    if not os.path.exists(data_dir):\n",
    "        os.makedirs(data_dir)\n",
    "\n",
    "#download\n",
    "f = urlopen(data_url) \n",
    "with open(file_path, \"wb\") as local_file:\n",
    "            local_file.write(f.read())\n",
    "\n",
    "#extract\n",
    "zipfile.ZipFile(file=file_path, mode=\"r\").extractall(data_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Get the graph\n",
    "\n",
    "The graph has 12 layers, and we name them below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "#get graph\n",
    "layer_names = ['conv2d0', 'conv2d1', 'conv2d2', 'mixed3a', \n",
    "               'mixed3b', 'mixed4a', 'mixed4b', 'mixed4c', \n",
    "               'mixed4d', 'mixed4e', 'mixed5a', 'mixed5b']\n",
    "\n",
    "graph = tf.Graph()\n",
    "with graph.as_default():\n",
    "    path = os.path.join(data_dir, \"tensorflow_inception_graph.pb\")\n",
    "    with tf.gfile.FastGFile(path, 'rb') as file:\n",
    "        graph_def = tf.GraphDef()\n",
    "        graph_def.ParseFromString(file.read())\n",
    "        tf.import_graph_def(graph_def, name='')\n",
    "\n",
    "graph_input = graph.get_tensor_by_name(\"input:0\")\n",
    "layer_tensors = [graph.get_tensor_by_name(name + \":0\") for name in layer_names]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Optimize the image using gradient ascent\n",
    "\n",
    "In this step, we will calculate the gradient of the layer with respect to the image. We will average the square of the layer tensor to get a positive, scalar value for the tensor before differentiation. \n",
    "\n",
    "Then we will add the gradient to the image. When we repeat this process, we will amplify patterns in the image. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def optimize(graph, graph_input, layer_tensor, image,\n",
    "                   num_iterations=10, step_size=3.0):\n",
    "\n",
    "    # Copy original image\n",
    "    img = image.copy()\n",
    "    img = img.astype(np.float32)\n",
    "\n",
    "    # Calculate gradient\n",
    "    gradient = tf.gradients(tf.reduce_mean(tf.square(layer_tensor)), graph_input)[0]\n",
    "\n",
    "    for i in range(num_iterations):\n",
    "        \n",
    "        # Calculate the value of the gradient.\n",
    "        feed_dict = {\"input:0\": np.expand_dims(img, axis=0)}\n",
    "        grad = session.run(gradient, feed_dict=feed_dict)\n",
    "        \n",
    "        grad /= (np.std(grad) + 1e-8)\n",
    "        step_size_scaled = step_size / (np.std(grad) + 1e-8)\n",
    "\n",
    "        # Update the image by following the gradient.\n",
    "        grad = grad.reshape(img.shape)\n",
    "        img += grad * step_size_scaled\n",
    "\n",
    "    img = np.clip(img, 0.0, 255.0)\n",
    "    img = img.astype(np.uint8) \n",
    "    return img"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Start session"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "session = tf.InteractiveSession(graph=graph)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Load and plot image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "filename = './images/chipmunk.jpg'\n",
    "\n",
    "image = np.float32(PIL.Image.open(filename))\n",
    "\n",
    "# Ensure that image has rgb values between 0 and 225\n",
    "image = np.clip(image, 0.0, 255.0)\n",
    "        \n",
    "# Convert pixels to bytes.\n",
    "image = image.astype(np.uint8)\n",
    "\n",
    "# Convert to a PIL-image and display it.\n",
    "display(PIL.Image.fromarray(image))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Choose a layer \n",
    "\n",
    "Choosing higher layers will result in higher-level feature or patterns. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "layer_tensor = layer_tensors[5]\n",
    "layer_tensor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Make DeepDream image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "mg_result = optimize(graph, graph_input, layer_tensor, image, num_iterations=10, step_size=6.0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "display(PIL.Image.fromarray(mg_result))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Exercises: \n",
    "\n",
    "1. First play around with the layer number. The higher numbers correspond to more complex shapes that the model recognizes. \n",
    "2. Upload your own picture and perform the DeepDream algorithm on it. \n",
    "3. Don't square the tensor before you calculate the gradient. How does this change your results?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "*Copyright &copy; 2017 The Data Incubator.  All rights reserved.*"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}