TF_Recurrent_Neural_Networks.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib\n",
    "import seaborn as sns\n",
    "matplotlib.rcParams['savefig.dpi'] = 144"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "\n",
    "import os\n",
    "\n",
    "import time\n",
    "import random\n",
    "from datetime import datetime, timedelta\n",
    "from urllib2 import urlopen\n",
    "\n",
    "from pylib.draw_graph import draw_graph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sess = None\n",
    "\n",
    "def reset_vars():\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "\n",
    "def reset_tf():\n",
    "    global sess\n",
    "    if sess:\n",
    "        sess.close()\n",
    "    tf.reset_default_graph()\n",
    "    sess = tf.Session()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Note: This notebook takes a while to run.  It is recommended\n",
    "# that you adjust the iterations number up and the execute the\n",
    "# entire notebook, before reading through it.\n",
    "num_iterations = 10 #1000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- requirement: pylib/__init__.py -->\n",
    "<!-- requirement: pylib/draw_graph.py -->\n",
    "<!-- requirement: small_data/strata_abstracts.txt -->\n",
    "\n",
    "# Recurrent Neural Networks\n",
    "\n",
    "Up to now, we have been dealing only with **feed-forward** networks.  These take in some input features and feed those through the network to produce the output.  For many problems, this is enough.  If you wan to classify images of cats, all you care about are the features from one given image.  You don't care about what the previous (or next) picture was.\n",
    "\n",
    "However, much of the world's data is not time-independent.  Some of this is very obvious: If you're trying to predict future stock prices, knowing the past prices is probably going to be useful.  Other cases might not be immediately clear.  A prime example is language processing.  Order is important; there's a big difference between \"dog bites man\" and \"man bites dog\".  Similarly, speech recognition, optical character recognition, and text summarization algorithms all benefit from knowing something about the previous inputs.\n",
    "\n",
    "We encountered the same sort of issue when we wanted to classify images.  Knowing what's nearby in space is very important, so we designed a network architecture to reflect those priorities.  In that case, we developed convolutional nets that combined features nearby in space.  Now, we need to design a network that gives us nearby-in-time features.\n",
    "\n",
    "Such networks are known as **recurrent** neural networks (RNN).  To illustrate how they work, we'll adopt a simple sketch notation.  Let this represent a feed-forward of the kind we've already dealt with."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "draw_graph(\"feed-forward\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The two blue circles represent hidden layers.  In the feed-forward architecture, they just feed activations further through the network.\n",
    "\n",
    "In contrast, we represent a recurrent network like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "draw_graph(\"recurrent\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The hidden layers get input activations not just from the layer below, but from their (or other layer's) output at a previous step.  These hidden activations incorporate the previous state of the network, thereby providing some memory of the previous inputs.  During training, the recurrent network will learn how to weight both the current features and the previous state in making decisions.\n",
    "\n",
    "An alternative approach would be to have inputs not just for the current time, but for *n* previous times.  As long as the connections are set up appropriately, this could give similar performance.  However, it would require inputs of a fixed length.  In contrast, RNNs can take inputs of arbitrary length, since all of the previous input, of whatever length, is represented in the hidden activations.\n",
    "\n",
    "It was proven in 2006 by Schäfer and Zimmerman that RNNs with sigmoid activations are **Turing-complete**.  That is, any program can be written by giving a RNN the appropriate weights.  This is only a theoretical results; there is no method to find those weights for a particular task.  However, it's easier to use an optimization algorithm to explore the space of weight matrices than the space of Python programs, for example.\n",
    "\n",
    "## Backpropagation through time\n",
    "\n",
    "You might remember that we said at the very beginning of the very first notebook that TensorFlow graphs must be acyclic.  Yet we just drew a diagram of a RNN with a loop in it.  How can we put this into TensorFlow?\n",
    "\n",
    "The trick is to **unroll** or unfold the RNN through time.  That is, the recurrent edge is pointed not back to the same node, but to another copy, representing the next step in time.  And in that copy, this edge points to yet another copy, giving us a structure like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "draw_graph(\"unrolled\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Although somewhat unusual, this is a perfectly good feed-forward network, and the standard backpropagation algorithm can be used with it.  The only wrinkle is that the weight matrices in all of the copies must stay in sync.  Therefore, we update them by summing all the gradients from all copies, similar to the way we handled convolutional filters.  This process in known as \"backpropagation through time\".\n",
    "\n",
    "## Long-short term memory\n",
    "\n",
    "If this is implemented naïvely, it will immediately run into problems with vanishing and exploding gradients.  To see why, recall that we can write the prediction of a feed-forward network as a set of nested functions.\n",
    "\n",
    "$$ p = f_n\\left( f_{n-1}\\left( \\cdots f_2\\left( f_1\\big( x W_1 \\big) W_2 \\right) \\cdots W_{n-1} \\right) W_n \\right) $$\n",
    "\n",
    "(We've dropped the bias terms for simplicity.)  To update $W_1$, for example, we need to calculate\n",
    "\n",
    "$$ \\frac{\\partial p}{\\partial W_1} = f_n' \\big(\\cdots\\big) W_n \\cdot f_{n-1}'\\big(\\cdots\\big) W_{n-1} \\cdot\\cdots\\cdot f_2'\\big(\\cdots\\big) W_2 \\cdot f_1'\\big(x W_1) \\cdot x$$\n",
    "\n",
    "If each of these terms are greater than one, the whole gradient gradient will become large.  This will require a small learning-rate to avoid the optimizer diverging.  If each of the the terms are less than one, the gradient will vanish, and the weight won't change appreciably.\n",
    "\n",
    "This is a general problem in deep networks, but with a generic network we can at least hope that we'll have terms both less than and greater than one, so that the whole gradient remains finite.  In an RNN, though, these nested functions represent previous copies of the network, so all of the weights are the same!  Unless they happen to be exactly one, we're guaranteed to get an explosion or a vanishing!\n",
    "\n",
    "This issue has been dubbed the \"fundamental problem of deep learning\".  Several solutions have been proposed, but the current *de facto* standard is **long-short term memory**  (LSTM).  LSTM replaces the simple neurons with LSTM cells.  Each of those has an internal loop, allowing it to remember the state from previous runs.  That loop has a fixed weight of 1 and a linear activation.  This ensure that local derivative is always one, so backpropagation can proceed without the gradient vanishing or exploding.  This internal gate is proceeded and followed by two nonlinear (usually sigmoid) gates.  These gates are responsible for learning how to weight the internal state compared to the new inputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "draw_graph(\"LSTM\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are several sophistications commonly added to the basic LSTM cell.  One is a forget gate, added to the internal loop in the cell.  This bias should be initialized to one, so that the cell starts in a remembering state.  Should it be necessary, the cell can more easily learn to forget its state.\n",
    "\n",
    "Another extension is peephole connections, which connect the internal state the the gates inside of the cell. There is some evidence that these improve the performance tasks requiring precise intervals.  Both of these features are built into the LSTM cell provided by TensorFlow.\n",
    "\n",
    "## Applications\n",
    "\n",
    "Layers of recurrent cells are typically not used alone.  At the very least, a full-connected feed-forward layer is used following the recurrent layer.  This translates the individual memories into the expected output.  More complicated architectures are common, with several recurrent and several feed-forward layers combined.  Convolutional layers may also be used when the problem calls for them, for example in processing video.\n",
    "\n",
    "Recurrent networks can be used for a number of different applications.  The examples we've draw so far illustrate **sequence labeling**.  For each input, we calculate all the way through to the output.  This could be used, for example, to classify words in a sentence by their part of speech.  We need a label for each word, and the recurrent nature lets us use the past words to differentiate between, for example, the word \"rows\" in \"Jack rows the boat\" and \"Jack walked between rows of wheat.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "draw_graph(\"unrolled\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes, we care only about a single label for the whole sequence.  This is known as **sequence classification**.  We read the output associated only with the final input.  The recurrent nature ensures that this output has information from all of the inputs.  This could be used to classify the sentiment of a review text, for example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "draw_graph(\"classification\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The situation is essentially reversed for **sequence generation**.  In this case, we take only a single input and wish to generate a whole sequence of output.  We can do this by feeding the output from one step (suitably processed, perhaps) as input to the next time step.  This can be used to generate text that resembles some corpus, as we demonstrate below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "draw_graph(\"generation\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ideas of classification and generation can be combined in **sequence translation**.  Here, an input sequence is fed into a network to establish a certain internal state.  This state is then used as the start of a decode sequence.  This can be done with two related RNNs or with a single one.  In the latter case a special token is needed to indicate to the network when to start outputting the translated version, as illustrated below.  Google used such networks for its [recent improvement](https://www.nytimes.com/2016/12/14/magazine/the-great-ai-awakening.html?_r=0) of its translation service."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "draw_graph(\"translation\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example: Generating strata abstracts\n",
    "\n",
    "Here, we demonstrate a simple example of using sequence generation to create text.  We will train a neural network on a corpus of data consisting of abstracts of talks from past Strata conferences.  The networks will be trained to predict the next letter the appears in the abstract.  Once trained, the network can be used to generate an abstract by starting it off with a seed, and then continuing the sequence with letters chosen according the the probabilities output by the network.\n",
    "\n",
    "We start by reading in the data we've already downloaded.  The abstracts are saved in a text file, one per line."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "txt = open('small_data/strata_abstracts.txt', 'r').read().lower()\n",
    "\n",
    "print len(txt)\n",
    "print \n",
    "print txt[:1000]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll want to one-hot encode the characters, so first we need to convert them to numbers.  We could just take their ASCII values, but this would give us a larger vocabulary than we need.  So instead we work out our own encoding, based on the characters we actually see."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "chars = list(set(txt))\n",
    "data = [chars.index(c) for c in txt]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will be training with the minibatch method.  This function will generate a batch of data.  Each batch contains `batch_size` sequences, each `time_steps` in length.  (Note that, for simplicity, we are ignoring the fact that newlines should end the abstract.)  The labels to be predicted are the following letters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_next_batch(batch_size, time_steps, data):\n",
    "    x_batch = np.zeros((batch_size, time_steps))\n",
    "    y_batch = np.zeros((batch_size, time_steps))\n",
    "    \n",
    "    batch_ids = range(len(data) - time_steps - 1)\n",
    "    batch_id = random.sample(batch_ids, batch_size)\n",
    "    \n",
    "    for t in xrange(time_steps):\n",
    "        x_batch[:, t] = [data[i+t] for i in batch_id]\n",
    "        y_batch[:, t] = [data[i+t+1] for i in batch_id]\n",
    "        \n",
    "    return x_batch, y_batch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "get_next_batch(1, 5, data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've chosen an architecture with 2 layers of 256 LSTM cells.  We will train on batches of 50 sequences of length 100."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "n_layers = 2\n",
    "n_chars = len(chars)\n",
    "lstm_size = 256\n",
    "\n",
    "time_steps = 100\n",
    "batch_size = 50"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are using the `BasicLSTMCell`.  Despite the name, it actually implements an entire LSTM layer.  TensorFlow also provides a `LSTMCell` which includes peephole connections and some other features.\n",
    "\n",
    "The `MultiRNNCell` combines several individual layers in sequence.  We will use two layers of the same size here.  Then we use use a fully-connected linear layer to combine the output of the RNN layers into output of the size we desire.\n",
    "\n",
    "The `dynamic_rnn` function does a dynamic unrolling of the RNN to accomplish backpropagation through time.  It returns not only the output of the RNN, but a tensor giving the state, or hidden activations, of the RNN."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def make_lstm(x, lstm_init_value, n_chars, lstm_size, n_layers):\n",
    "    # LSTM\n",
    "    lstm = tf.contrib.rnn.MultiRNNCell(\n",
    "        [tf.contrib.rnn.BasicLSTMCell(lstm_size, forget_bias=1.0, state_is_tuple=False)\n",
    "         for _ in xrange(n_layers)],\n",
    "        state_is_tuple=False)\n",
    "\n",
    "    # Iteratively compute output of recurrent network\n",
    "    out, lstm_new_state = tf.nn.dynamic_rnn(lstm, x, initial_state=lstm_init_value, dtype=tf.float32)\n",
    "\n",
    "    # Linear activation (FC layer on top of the LSTM net)\n",
    "    out_reshaped = tf.reshape(out, [-1, lstm_size])\n",
    "    y = tf.layers.dense(out_reshaped, n_chars, activation=None)\n",
    "    \n",
    "    return y, tf.shape(out), lstm_new_state"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The RNNs are flexible enough to take input of an arbitrary number of batches of arbitrarily long sequences.  Thus, the shapes are given as `(None, None)`.  We'll also need an input to initialize the RNN's state."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "reset_tf()\n",
    "\n",
    "x = tf.placeholder(tf.int32, shape=(None, None), name=\"x\")\n",
    "y_true = tf.placeholder(tf.int32, (None, None))\n",
    "lstm_init_value = tf.placeholder(tf.float32, shape=(None, n_layers*2*lstm_size),\n",
    "                                 name=\"lstm_init_value\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to one-hot encode both the input and the output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "x_enc = tf.one_hot(x, depth=n_chars)\n",
    "y_true_enc = tf.one_hot(y_true, depth=n_chars)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These values get fed into our net."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "y_pred, out_shape, lstm_new_state = make_lstm(x_enc, lstm_init_value, n_chars, lstm_size, n_layers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The predictions come out flattened, so if we want an array in the same shape as the input, we need to reshape it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "final_out = tf.reshape(tf.nn.softmax(y_pred), \n",
    "                       (out_shape[0], out_shape[1], n_chars))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll use the cross-entropy as our loss function.  There are other metrics worth considering, including the [perplexity](https://en.wikipedia.org/wiki/Perplexity)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(\n",
    "        logits=y_pred,\n",
    "        labels=tf.reshape(y_true_enc, [-1, n_chars])))\n",
    "tf.summary.scalar('loss', loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Root Mean Square Propagation (RMSProp) optimizer attempts to set a custom learning rate for each parameter.  It has been found to work well with RNNs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "optimizer = tf.train.RMSPropOptimizer(0.003, 0.9).minimize(loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#Merge all summaries\n",
    "merged = tf.summary.merge_all()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To generate the text, we'll need to run the network one step at a time, while preserving the internal state.  This way, we can keep feeding back in the value that we produced in the previous step to move forward."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def run_step(seed, chars, init_value):\n",
    "    test_data = [[chars.index(c) for c in seed]]\n",
    "        \n",
    "    out, next_lstm_state = sess.run([final_out, lstm_new_state], \n",
    "                                    {x:test_data, lstm_init_value:[init_value]} )\n",
    "    return out[0][0], next_lstm_state[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the actual generation, we first run the network on some seed text.  The last output gives us a set of probabilities for the next letter.  We choose that letter with those probabilities, append it to the string we are generating, and then advance the state of the network as if this were the next input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def generate_text(seed, len_test_txt=500):\n",
    "    seed = seed.lower()\n",
    "\n",
    "    lstm_last_state = np.zeros((n_layers*2*lstm_size,))\n",
    "    for c in seed:\n",
    "        out, lstm_last_state = run_step(c, chars, lstm_last_state)\n",
    "    \n",
    "    gen_str = seed\n",
    "    for i in range(len_test_txt):\n",
    "        ele = np.random.choice(range(len(chars)), p=out)\n",
    "        gen_str += chars[ele]\n",
    "        out, lstm_last_state = run_step(chars[ele], chars, lstm_last_state)\n",
    "    \n",
    "    return gen_str"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now we're ready to go!  We'll run the optimizer for a while.  As we go, we'll print out sample text, so we can see what's being learned."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "display_step = 50\n",
    "\n",
    "reset_vars()\n",
    "\n",
    "#Create summary writers\n",
    "logs_path = datetime.now().strftime(\"%Y%m%d-%H%M%S\") + '/summaries'\n",
    "train_writer = tf.summary.FileWriter(logs_path + '/train', graph=tf.get_default_graph())\n",
    "\n",
    "# Start-time used for printing time-usage below.\n",
    "start_time = time.time()\n",
    "\n",
    "step = 1\n",
    "for i in range(num_iterations):\n",
    "\n",
    "    # Get a batch of training examples.\n",
    "    x_batch, y_true_batch = get_next_batch(batch_size, time_steps, data)\n",
    "\n",
    "    # ---------------------- TRAIN -------------------------\n",
    "    # optimize model\n",
    "    init_value = np.zeros((x_batch.shape[0], n_layers*2*lstm_size))\n",
    "    sess.run(optimizer, feed_dict={x: x_batch, y_true: y_true_batch, lstm_init_value:init_value}) \n",
    "\n",
    "\n",
    "    # Print status every 100 iterations.\n",
    "    if (i % display_step == 0) or (i == num_iterations - 1):\n",
    "\n",
    "        summary, l = sess.run([merged, loss], feed_dict={x: x_batch, y_true: y_true_batch, lstm_init_value:init_value})\n",
    "        train_writer.add_summary(summary, step)\n",
    "\n",
    "        # Message for network evaluation\n",
    "        msg = \"Optimization Iteration: {0:>6}, Training Loss: {1:>6}\"\n",
    "        print(msg.format(i, l))\n",
    "        print \"  \" + generate_text(\"We\", 60)\n",
    "\n",
    "        step += 1\n",
    "\n",
    "# Ending time.\n",
    "end_time = time.time()\n",
    "\n",
    "# Difference between start and end-times.\n",
    "time_dif = end_time - start_time\n",
    "\n",
    "# Print the time-usage.\n",
    "print(\"Time usage: \" + str(timedelta(seconds=int(round(time_dif)))))\n",
    "\n",
    "# Close summary writer\n",
    "train_writer.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's generate some abstracts!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print generate_text(\"We\", 500)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print generate_text(\"The\", 500)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print generate_text(\"TensorFlow\", 500)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Copyright &copy; 2017 The Data Incubator.  All rights reserved.*"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}