Double-Dueling-DQN.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Simple Reinforcement Learning with Tensorflow Part 4: Deep Q-Networks and Beyond\n",
    "\n",
    "In this iPython notebook I implement a Deep Q-Network using both Double DQN and Dueling DQN. The agent learn to solve a navigation task in a basic grid world. To learn more, read here: https://medium.com/p/8438a3e2b8df\n",
    "\n",
    "For more reinforcment learning tutorials, see:\n",
    "https://github.com/awjuliani/DeepRL-Agents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from __future__ import division\n",
    "\n",
    "import numpy as np\n",
    "import random\n",
    "import tensorflow as tf\n",
    "import tensorflow.contrib.slim as slim\n",
    "from helper2 import make_gif\n",
    "import matplotlib.pyplot as plt\n",
    "import scipy.misc\n",
    "import os\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load the game environment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Feel free to adjust the size of the gridworld. Making it smaller provides an easier task for our DQN agent, while making the world larger increases the challenge."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP4AAAD8CAYAAABXXhlaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAADM5JREFUeJzt3X+oX/V9x/Hna4nW1m7VqAuZ0SWj\nosjA6IJTLKPTullXdH8UUcooQ/CfbtO10Or2hxT2RwujrX+MgtR2Mpw/anUNodi51DL2T2r8sVYT\nrdHGmqAmdjo7B9vSvvfHOdluQ+I9N/f74x4/zwdc7vec7/dyPidfXvec78m5n1eqCklt+aV5D0DS\n7Bl8qUEGX2qQwZcaZPClBhl8qUEGX2rQsoKf5IokzybZneTmSQ1K0nTlWG/gSbIK+CFwObAXeBS4\nrqp2Tm54kqZh9TJ+9kJgd1W9AJDkHuBq4KjBP/XUU2vDhg3L2KSkt7Nnzx5ee+21LPa65QT/dOCl\nBct7gd9+ux/YsGEDO3bsWMYmJb2dzZs3D3rd1C/uJbkhyY4kOw4cODDtzUkaYDnB3wecsWB5fb/u\nF1TV7VW1uao2n3baacvYnKRJWU7wHwXOSrIxyfHAtcCWyQxL0jQd82f8qjqY5E+AbwOrgK9W1dMT\nG5mkqVnOxT2q6lvAtyY0Fkkz4p17UoMMvtQggy81yOBLDTL4UoMMvtQggy81yOBLDTL4UoMMvtQg\ngy81yOBLDTL4UoMMvtQggy81yOBLDTL4UoMWDX6SrybZn+SpBevWJHk4yXP995OnO0xJkzTkiP+3\nwBWHrbsZ2FZVZwHb+mVJI7Fo8Kvqn4F/O2z11cCd/eM7gT+c8LgkTdGxfsZfW1Uv949fAdZOaDyS\nZmDZF/eqa908avOmTTrSynOswX81yTqA/vv+o73QJh1p5TnW4G8BPt4//jjwzckMR9IsLFqokeRu\n4IPAqUn2ArcCnwPuS3I98CJwzTQHOQnJos3BmoKjfgacgcxx4zXXPV/cosGvquuO8tRlEx6LpBnx\nzj2pQQZfapDBlxpk8KUGGXypQQZfapDBlxpk8KUGGXypQQZfapDBlxpk8KUGGXypQQZfapDBlxpk\n8KUGGXypQUOadM5I8kiSnUmeTnJjv942HWmkhhzxDwKfqqpzgYuATyQ5F9t0pNEa0qTzclU93j/+\nKbALOB3bdKTRWtJn/CQbgPOB7Qxs07FQQ1p5Bgc/yXuBbwA3VdWbC597uzYdCzWklWdQ8JMcRxf6\nu6rqgX714DYdSSvLkKv6Ae4AdlXVFxY8ZZuONFKLFmoAlwB/BPwgyZP9ur9ghG06kjpDmnT+BTha\n/5RtOtIIeeee1CCDLzXI4EsNGnJx7x1inp3Jc6zonnM7+Fw3P8+Nr+yWbI/4UosMvtQggy81yOBL\nDTL4UoMMvtQggy81yOBLDTL4UoMMvtQggy81yOBLDTL4UoOGzLl3QpLvJfnXvknns/36jUm2J9md\n5N4kx09/uJImYcgR/7+AS6vqPGATcEWSi4DPA1+sqvcDrwPXT2+YkiZpSJNOVdV/9IvH9V8FXArc\n36+3SUcakaHz6q/qZ9jdDzwMPA+8UVUH+5fspavVOtLP2qQjrTCDgl9VP6uqTcB64ELgnKEbsElH\nWnmWdFW/qt4AHgEuBk5KcmjqrvXAvgmPTdKUDLmqf1qSk/rH7wYup2vMfQT4aP8ym3SkERky2eY6\n4M4kq+h+UdxXVVuT7ATuSfJXwBN0NVuSRmBIk8736aqxD1//At3nfUkj4517UoMMvtQggy81yOBL\nDTL4UoMMvtQggy81yOBLDWqoJrvdquq5mmdddMv/7ovwiC81yOBLDTL4UoMMvtQggy81yOBLDTL4\nUoMMvtSgwcHvp9h+IsnWftkmHWmklnLEv5Fuks1DbNKRRmpoocZ64A+Ar/TLwSYdabSGHvG/BHwa\n+Hm/fAo26UijNWRe/Y8A+6vqsWPZgE060soz5K/zLgGuSnIlcALwK8Bt9E06/VHfJh1pRIa05d5S\nVeuragNwLfCdqvoYNulIo7Wc/8f/DPDJJLvpPvPbpCONxJIm4qiq7wLf7R/bpCONlHfuSQ0y+FKD\nDL7UIIMvNcjgSw0y+FKDDL7UIIMvNcjgSw0y+FKDDL7UIIMvNcjgSw0y+FKDlvRnuaNmT/t8tLzv\nK5hHfKlBg474SfYAPwV+Bhysqs1J1gD3AhuAPcA1VfX6dIYpaZKWcsT/3araVFWb++WbgW1VdRaw\nrV+WNALLOdW/mq5IAyzUkEZlaPAL+MckjyW5oV+3tqpe7h+/Aqyd+OgkTcXQq/ofqKp9SX4VeDjJ\nMwufrKpKcsTr5v0vihsAzjzzzGUNVtJkDDriV9W+/vt+4EG62XVfTbIOoP++/yg/a5OOtMIMqdA6\nMckvH3oM/B7wFLCFrkgDLNSQRmXIqf5a4MGuIJfVwN9X1UNJHgXuS3I98CJwzfSGKWmSFg1+X5xx\n3hHW/wS4bBqDkjRd3rknNcjgSw0y+FKDDL7UIIMvNcjgSw0y+FKDDL7UIIMvNcjgSw0y+FKDDL7U\nIIMvNcjgSw0y+FKDDL7UIIMvNWhQ8JOclOT+JM8k2ZXk4iRrkjyc5Ln++8nTHqykyRh6xL8NeKiq\nzqGbhmsXNulIozVklt33Ab8D3AFQVf9dVW9gk440WkNm2d0IHAC+luQ84DHgRsbWpGNdc3Pm2Yy+\n0g051V8NXAB8uarOB97isNP6qiqO8u+c5IYkO5LsOHDgwHLHK2kChgR/L7C3qrb3y/fT/SKwSUca\nqUWDX1WvAC8lObtfdRmwE5t0pNEaWpr5p8BdSY4HXgD+mO6Xhk060ggNCn5VPQlsPsJTNulII+Sd\ne1KDDL7UIIMvNcjgSw0y+FKDDL7UIIMvNcjgSw0y+FKDDL7UIIMvNcjgSw0y+FKDDL7UIIMvNcjg\nSw0y+FKDhsyrf3aSJxd8vZnkJpt0pPEaMtnms1W1qao2Ab8F/CfwIDbpSKO11FP9y4Dnq+pFbNKR\nRmupwb8WuLt/PK4mHUn/Z3Dw+6m1rwK+fvhzNulI47KUI/6Hgcer6tV+2SYdaaSWEvzr+P/TfLBJ\nRxqtQcFPciJwOfDAgtWfAy5P8hzwoX5Z0ggMbdJ5CzjlsHU/YURNOt1lCEngnXtSkwy+1CCDLzXI\n4EsNMvhSgwy+1CCDLzXI4EsNMvhSgwy+1CCDLzXI4EsNMvhSgwy+1CCDLzXI4EsNMvhSg4ZOvfXn\nSZ5O8lSSu5OckGRjku1Jdie5t5+FV9IIDKnQOh34M2BzVf0msIpufv3PA1+sqvcDrwPXT3OgkiZn\n6Kn+auDdSVYD7wFeBi4F7u+ft0lHGpEh3Xn7gL8GfkwX+H8HHgPeqKqD/cv2AqdPa5CSJmvIqf7J\ndD15G4FfA04Erhi6AZt0pJVnyKn+h4AfVdWBqvofurn1LwFO6k/9AdYD+470wzbpSCvPkOD/GLgo\nyXuShG4u/Z3AI8BH+9fYpCONyJDP+NvpLuI9Dvyg/5nbgc8An0yym65s444pjlPSBA1t0rkVuPWw\n1S8AF058RJKmzjv3pAYZfKlBBl9qkMGXGpRZ1kcnOQC8Bbw2s41O36m4PyvVO2lfYNj+/HpVLXrD\nzEyDD5BkR1VtnulGp8j9WbneSfsCk90fT/WlBhl8qUHzCP7tc9jmNLk/K9c7aV9ggvsz88/4kubP\nU32pQTMNfpIrkjzbz9N38yy3vVxJzkjySJKd/fyDN/br1yR5OMlz/feT5z3WpUiyKskTSbb2y6Od\nSzHJSUnuT/JMkl1JLh7z+zPNuS5nFvwkq4C/AT4MnAtcl+TcWW1/Ag4Cn6qqc4GLgE/0478Z2FZV\nZwHb+uUxuRHYtWB5zHMp3gY8VFXnAOfR7dco35+pz3VZVTP5Ai4Gvr1g+Rbgllltfwr7803gcuBZ\nYF2/bh3w7LzHtoR9WE8XhkuBrUDobhBZfaT3bCV/Ae8DfkR/3WrB+lG+P3RT2b0ErKH7K9qtwO9P\n6v2Z5an+oR05ZLTz9CXZAJwPbAfWVtXL/VOvAGvnNKxj8SXg08DP++VTGO9cihuBA8DX+o8uX0ly\nIiN9f2rKc116cW+JkrwX+AZwU1W9ufC56n4Nj+K/SZJ8BNhfVY/NeywTshq4APhyVZ1Pd2v4L5zW\nj+z9WdZcl4uZZfD3AWcsWD7qPH0rVZLj6EJ/V1U90K9+Ncm6/vl1wP55jW+JLgGuSrIHuIfudP82\nBs6luALtBfZWN2MUdLNGXcB4359lzXW5mFkG/1HgrP6q5PF0Fyq2zHD7y9LPN3gHsKuqvrDgqS10\ncw7CiOYerKpbqmp9VW2gey++U1UfY6RzKVbVK8BLSc7uVx2aG3KU7w/TnutyxhcsrgR+CDwP/OW8\nL6AscewfoDtN/D7wZP91Jd3n4m3Ac8A/AWvmPdZj2LcPAlv7x78BfA/YDXwdeNe8x7eE/dgE7Ojf\no38ATh7z+wN8FngGeAr4O+Bdk3p/vHNPapAX96QGGXypQQZfapDBlxpk8KUGGXypQQZfapDBlxr0\nv3nr6neI68U5AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fda492737b8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from gridworld import gameEnv\n",
    "\n",
    "env = gameEnv(partial=False,size=5)\n",
    "print(env.actions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above is an example of a starting environment in our simple game. The agent controls the blue square, and can move up, down, left, or right. The goal is to move to the green square (for +1 reward) and avoid the red square (for -1 reward). The position of the three blocks is randomized every episode."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Implementing the network itself"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class Qnetwork():\n",
    "    def __init__(self,h_size):\n",
    "        #The network recieves a frame from the game, flattened into an array.\n",
    "        #It then resizes it and processes it through four convolutional layers.\n",
    "        self.scalarInput =  tf.placeholder(shape=[None,21168],dtype=tf.float32)\n",
    "        self.imageIn = tf.reshape(self.scalarInput,shape=[-1,84,84,3])\n",
    "        self.conv1 = slim.conv2d( \\\n",
    "            inputs=self.imageIn,num_outputs=32,kernel_size=[8,8],stride=[4,4],padding='VALID', biases_initializer=None)\n",
    "        self.conv2 = slim.conv2d( \\\n",
    "            inputs=self.conv1,num_outputs=64,kernel_size=[4,4],stride=[2,2],padding='VALID', biases_initializer=None)\n",
    "        self.conv3 = slim.conv2d( \\\n",
    "            inputs=self.conv2,num_outputs=64,kernel_size=[3,3],stride=[1,1],padding='VALID', biases_initializer=None)\n",
    "        self.conv4 = slim.conv2d( \\\n",
    "            inputs=self.conv3,num_outputs=h_size,kernel_size=[7,7],stride=[1,1],padding='VALID', biases_initializer=None)\n",
    "        \n",
    "        #We take the output from the final convolutional layer and split it into separate advantage and value streams.\n",
    "        self.streamAC,self.streamVC = tf.split(self.conv4,2,3)\n",
    "        self.streamA = slim.flatten(self.streamAC)\n",
    "        self.streamV = slim.flatten(self.streamVC)\n",
    "        xavier_init = tf.contrib.layers.xavier_initializer()\n",
    "        self.AW = tf.Variable(xavier_init([h_size//2,env.actions]))\n",
    "        self.VW = tf.Variable(xavier_init([h_size//2,1]))\n",
    "        self.Advantage = tf.matmul(self.streamA,self.AW)\n",
    "        self.Value = tf.matmul(self.streamV,self.VW)\n",
    "        \n",
    "        #Then combine them together to get our final Q-values.\n",
    "        self.Qout = self.Value + tf.subtract(self.Advantage,tf.reduce_mean(self.Advantage,axis=1,keep_dims=True))\n",
    "        self.predict = tf.argmax(self.Qout,1)\n",
    "        \n",
    "        #Below we obtain the loss by taking the sum of squares difference between the target and prediction Q values.\n",
    "        self.targetQ = tf.placeholder(shape=[None],dtype=tf.float32)\n",
    "        self.actions = tf.placeholder(shape=[None],dtype=tf.int32)\n",
    "        self.actions_onehot = tf.one_hot(self.actions,env.actions,dtype=tf.float32)\n",
    "        \n",
    "        self.Q = tf.reduce_sum(tf.multiply(self.Qout, self.actions_onehot), axis=1)\n",
    "        \n",
    "        self.td_error = tf.square(self.targetQ - self.Q)\n",
    "        self.loss = tf.reduce_mean(self.td_error)\n",
    "        self.trainer = tf.train.AdamOptimizer(learning_rate=0.0001)\n",
    "        self.updateModel = self.trainer.minimize(self.loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Experience Replay"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This class allows us to store experies and sample then randomly to train the network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class experience_buffer():\n",
    "    def __init__(self, buffer_size = 50000):\n",
    "        self.buffer = []\n",
    "        self.buffer_size = buffer_size\n",
    "    \n",
    "    def add(self,experience):\n",
    "        if len(self.buffer) + len(experience) >= self.buffer_size:\n",
    "            self.buffer[0:(len(experience)+len(self.buffer))-self.buffer_size] = []\n",
    "        self.buffer.extend(experience)\n",
    "            \n",
    "    def sample(self,size):\n",
    "        return np.reshape(np.array(random.sample(self.buffer,size)),[size,5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a simple function to resize our game frames."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def processState(states):\n",
    "    return np.reshape(states,[21168])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These functions allow us to update the parameters of our target network with those of the primary network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def updateTargetGraph(tfVars,tau):\n",
    "    #tfVars are all the trainable values of the computation graph, e.i. all the weights of the networks (main and target)\n",
    "    #tau is the the ratio to which we update the Target network with respect to the Main network\n",
    "    total_vars = len(tfVars)\n",
    "    op_holder = []\n",
    "    #Here we need to understand the structure of the tfVars array.\n",
    "    #The first half entries are the trainable values of the Main Network\n",
    "    #The last half entries are the trainable values of the Main Network\n",
    "    for idx,var in enumerate(tfVars[0:total_vars//2]):\n",
    "        #New_targetNet_values = tau * New_MainNet_values + (1 - tau) * Old_MainNet_values\n",
    "        op_holder.append(tfVars[idx+total_vars//2].assign((var.value()*tau) + ((1-tau)*tfVars[idx+total_vars//2].value())))\n",
    "    return op_holder\n",
    "\n",
    "#This function just runs the session to compute the above expression\n",
    "def updateTarget(op_holder,sess):\n",
    "    for op in op_holder:\n",
    "        sess.run(op)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training the network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Setting all the training parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "batch_size = 32 #How many experiences to use for each training step.\n",
    "update_freq = 4 #How often to perform a training step.\n",
    "y = .99 #Discount factor on the target Q-values\n",
    "startE = 1 #Starting chance of random action\n",
    "endE = 0.1 #Final chance of random action\n",
    "annealing_steps = 10000. #How many steps of training to reduce startE to endE.\n",
    "num_episodes = 10000 #How many episodes of game environment to train network with.\n",
    "pre_train_steps = 10000 #How many steps of random actions before training begins.\n",
    "max_epLength = 50 #The max allowed length of our episode.\n",
    "load_model = False #Whether to load a saved model.\n",
    "path = \"./dqn\" #The path to save our model to.\n",
    "h_size = 512 #The size of the final convolutional layer before splitting it into Advantage and Value streams.\n",
    "tau = 0.001 #Rate to update target network toward primary network\n",
    "saveframes = True\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saved Model\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 98%|█████████▊| 50/51 [00:00<00:00, 1006.16it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "500 1.0 1\n",
      "1000 1.6 1\n",
      "1500 1.9 1\n",
      "2000 0.4 1\n",
      "2500 1.6 1\n",
      "3000 0.7 1\n",
      "3500 1.6 1\n",
      "4000 3.0 1\n",
      "4500 1.2 1\n",
      "5000 1.4 1\n",
      "5500 4.4 1\n",
      "6000 3.4 1\n",
      "6500 2.8 1\n",
      "7000 2.4 1\n",
      "7500 3.6 1\n",
      "8000 2.3 1\n",
      "8500 2.6 1\n",
      "9000 2.7 1\n",
      "9500 1.8 1\n",
      "10000 2.1 1\n",
      "10500 1.4 0.9549999999999828\n",
      "11000 2.2 0.9099999999999655\n",
      "11500 0.6 0.8649999999999483\n",
      "12000 2.3 0.819999999999931\n",
      "12500 3.1 0.7749999999999138\n",
      "13000 1.1 0.7299999999998965\n",
      "13500 1.9 0.6849999999998793\n",
      "14000 0.9 0.639999999999862\n",
      "14500 1.8 0.5949999999998448\n",
      "15000 1.0 0.5499999999998275\n",
      "15500 1.2 0.5049999999998103\n",
      "16000 4.0 0.4599999999998177\n",
      "16500 2.9 0.41499999999982823\n",
      "17000 2.1 0.36999999999983874\n",
      "17500 1.2 0.32499999999984924\n",
      "18000 1.5 0.27999999999985975\n",
      "18500 2.7 0.23499999999986562\n",
      "19000 2.6 0.18999999999986225\n",
      "19500 2.2 0.14499999999985888\n",
      "20000 1.9 0.09999999999985551\n",
      "20500 2.6 0.09999999999985551\n",
      "21000 4.5 0.09999999999985551\n",
      "21500 2.8 0.09999999999985551\n",
      "22000 3.8 0.09999999999985551\n",
      "22500 3.2 0.09999999999985551\n",
      "23000 2.9 0.09999999999985551\n",
      "23500 4.5 0.09999999999985551\n",
      "24000 5.2 0.09999999999985551\n",
      "24500 6.2 0.09999999999985551\n",
      "25000 5.6 0.09999999999985551\n",
      "25500 3.7 0.09999999999985551\n",
      "26000 6.4 0.09999999999985551\n",
      "26500 7.5 0.09999999999985551\n",
      "27000 9.0 0.09999999999985551\n",
      "27500 7.5 0.09999999999985551\n",
      "28000 8.5 0.09999999999985551\n",
      "28500 4.0 0.09999999999985551\n",
      "29000 5.3 0.09999999999985551\n",
      "29500 8.3 0.09999999999985551\n",
      "30000 10.4 0.09999999999985551\n",
      "30500 8.8 0.09999999999985551\n",
      "31000 11.8 0.09999999999985551\n",
      "31500 9.9 0.09999999999985551\n",
      "32000 12.0 0.09999999999985551\n",
      "32500 11.8 0.09999999999985551\n",
      "33000 8.9 0.09999999999985551\n",
      "33500 13.1 0.09999999999985551\n",
      "34000 12.8 0.09999999999985551\n",
      "34500 13.9 0.09999999999985551\n",
      "35000 12.2 0.09999999999985551\n",
      "35500 17.0 0.09999999999985551\n",
      "36000 14.7 0.09999999999985551\n",
      "36500 15.0 0.09999999999985551\n",
      "37000 14.5 0.09999999999985551\n",
      "37500 11.2 0.09999999999985551\n",
      "38000 14.5 0.09999999999985551\n",
      "38500 16.2 0.09999999999985551\n",
      "39000 18.9 0.09999999999985551\n",
      "39500 15.1 0.09999999999985551\n",
      "40000 13.5 0.09999999999985551\n",
      "40500 16.2 0.09999999999985551\n",
      "41000 16.0 0.09999999999985551\n",
      "41500 18.6 0.09999999999985551\n",
      "42000 18.0 0.09999999999985551\n",
      "42500 14.7 0.09999999999985551\n",
      "43000 16.6 0.09999999999985551\n",
      "43500 16.6 0.09999999999985551\n",
      "44000 20.1 0.09999999999985551\n",
      "44500 19.3 0.09999999999985551\n",
      "45000 18.4 0.09999999999985551\n",
      "45500 19.6 0.09999999999985551\n",
      "46000 18.3 0.09999999999985551\n",
      "46500 20.6 0.09999999999985551\n",
      "47000 17.0 0.09999999999985551\n",
      "47500 18.0 0.09999999999985551\n",
      "48000 16.7 0.09999999999985551\n",
      "48500 19.5 0.09999999999985551\n",
      "49000 16.0 0.09999999999985551\n",
      "49500 18.2 0.09999999999985551\n",
      "50000 21.3 0.09999999999985551\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 98%|█████████▊| 50/51 [00:00<00:00, 1494.72it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saved Model\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "50500 19.1 0.09999999999985551\n",
      "51000 21.4 0.09999999999985551\n",
      "51500 16.8 0.09999999999985551\n",
      "52000 17.4 0.09999999999985551\n",
      "52500 18.7 0.09999999999985551\n",
      "53000 18.1 0.09999999999985551\n",
      "53500 15.3 0.09999999999985551\n",
      "54000 19.1 0.09999999999985551\n",
      "54500 15.6 0.09999999999985551\n",
      "55000 22.4 0.09999999999985551\n",
      "55500 20.4 0.09999999999985551\n",
      "56000 21.0 0.09999999999985551\n",
      "56500 20.5 0.09999999999985551\n",
      "57000 18.9 0.09999999999985551\n",
      "57500 21.1 0.09999999999985551\n",
      "58000 21.8 0.09999999999985551\n",
      "58500 20.0 0.09999999999985551\n",
      "59000 21.1 0.09999999999985551\n",
      "59500 21.8 0.09999999999985551\n",
      "60000 21.0 0.09999999999985551\n",
      "60500 23.1 0.09999999999985551\n",
      "61000 21.1 0.09999999999985551\n",
      "61500 23.6 0.09999999999985551\n",
      "62000 22.0 0.09999999999985551\n",
      "62500 22.2 0.09999999999985551\n",
      "63000 22.2 0.09999999999985551\n",
      "63500 23.1 0.09999999999985551\n",
      "64000 22.3 0.09999999999985551\n",
      "64500 22.4 0.09999999999985551\n",
      "65000 22.9 0.09999999999985551\n",
      "65500 22.3 0.09999999999985551\n",
      "66000 21.4 0.09999999999985551\n",
      "66500 20.6 0.09999999999985551\n",
      "67000 19.1 0.09999999999985551\n",
      "67500 21.5 0.09999999999985551\n",
      "68000 24.2 0.09999999999985551\n",
      "68500 20.3 0.09999999999985551\n",
      "69000 19.4 0.09999999999985551\n",
      "69500 21.5 0.09999999999985551\n",
      "70000 18.4 0.09999999999985551\n",
      "70500 22.7 0.09999999999985551\n",
      "71000 23.5 0.09999999999985551\n",
      "71500 20.8 0.09999999999985551\n",
      "72000 22.1 0.09999999999985551\n",
      "72500 23.4 0.09999999999985551\n",
      "73000 21.6 0.09999999999985551\n",
      "73500 23.2 0.09999999999985551\n",
      "74000 22.0 0.09999999999985551\n",
      "74500 21.4 0.09999999999985551\n",
      "75000 16.7 0.09999999999985551\n",
      "75500 22.6 0.09999999999985551\n",
      "76000 20.4 0.09999999999985551\n",
      "76500 22.1 0.09999999999985551\n",
      "77000 22.6 0.09999999999985551\n",
      "77500 18.3 0.09999999999985551\n",
      "78000 22.9 0.09999999999985551\n",
      "78500 22.7 0.09999999999985551\n",
      "79000 23.7 0.09999999999985551\n",
      "79500 21.8 0.09999999999985551\n",
      "80000 21.6 0.09999999999985551\n",
      "80500 22.6 0.09999999999985551\n",
      "81000 21.7 0.09999999999985551\n",
      "81500 19.3 0.09999999999985551\n",
      "82000 20.1 0.09999999999985551\n",
      "82500 22.0 0.09999999999985551\n",
      "83000 24.6 0.09999999999985551\n",
      "83500 22.3 0.09999999999985551\n",
      "84000 23.8 0.09999999999985551\n",
      "84500 20.1 0.09999999999985551\n",
      "85000 23.0 0.09999999999985551\n",
      "85500 20.3 0.09999999999985551\n",
      "86000 22.4 0.09999999999985551\n",
      "86500 22.0 0.09999999999985551\n",
      "87000 23.7 0.09999999999985551\n",
      "87500 20.6 0.09999999999985551\n",
      "88000 22.2 0.09999999999985551\n",
      "88500 19.6 0.09999999999985551\n",
      "89000 22.2 0.09999999999985551\n",
      "89500 21.5 0.09999999999985551\n",
      "90000 23.3 0.09999999999985551\n",
      "90500 23.0 0.09999999999985551\n",
      "91000 23.0 0.09999999999985551\n",
      "91500 22.4 0.09999999999985551\n",
      "92000 20.9 0.09999999999985551\n",
      "92500 23.3 0.09999999999985551\n",
      "93000 21.3 0.09999999999985551\n",
      "93500 22.7 0.09999999999985551\n",
      "94000 18.4 0.09999999999985551\n",
      "94500 22.3 0.09999999999985551\n",
      "95000 19.4 0.09999999999985551\n",
      "95500 21.9 0.09999999999985551\n",
      "96000 23.4 0.09999999999985551\n",
      "96500 21.8 0.09999999999985551\n",
      "97000 22.6 0.09999999999985551\n",
      "97500 22.9 0.09999999999985551\n",
      "98000 23.2 0.09999999999985551\n",
      "98500 22.2 0.09999999999985551\n",
      "99000 21.9 0.09999999999985551\n",
      "99500 22.6 0.09999999999985551\n",
      "100000 22.3 0.09999999999985551\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 98%|█████████▊| 50/51 [00:00<00:00, 1249.02it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saved Model\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100500 22.2 0.09999999999985551\n",
      "101000 19.5 0.09999999999985551\n",
      "101500 21.1 0.09999999999985551\n",
      "102000 23.8 0.09999999999985551\n",
      "102500 20.9 0.09999999999985551\n",
      "103000 21.5 0.09999999999985551\n",
      "103500 21.2 0.09999999999985551\n",
      "104000 23.1 0.09999999999985551\n",
      "104500 23.5 0.09999999999985551\n",
      "105000 21.6 0.09999999999985551\n",
      "105500 20.2 0.09999999999985551\n",
      "106000 20.7 0.09999999999985551\n",
      "106500 19.6 0.09999999999985551\n",
      "107000 20.0 0.09999999999985551\n",
      "107500 21.0 0.09999999999985551\n",
      "108000 18.9 0.09999999999985551\n",
      "108500 22.3 0.09999999999985551\n",
      "109000 21.8 0.09999999999985551\n",
      "109500 22.4 0.09999999999985551\n",
      "110000 19.2 0.09999999999985551\n",
      "110500 22.8 0.09999999999985551\n",
      "111000 22.9 0.09999999999985551\n",
      "111500 20.7 0.09999999999985551\n",
      "112000 21.4 0.09999999999985551\n",
      "112500 22.5 0.09999999999985551\n",
      "113000 21.7 0.09999999999985551\n",
      "113500 19.1 0.09999999999985551\n",
      "114000 23.8 0.09999999999985551\n",
      "114500 24.5 0.09999999999985551\n",
      "115000 25.6 0.09999999999985551\n",
      "115500 23.6 0.09999999999985551\n",
      "116000 21.3 0.09999999999985551\n",
      "116500 21.1 0.09999999999985551\n",
      "117000 19.7 0.09999999999985551\n",
      "117500 21.5 0.09999999999985551\n",
      "118000 20.8 0.09999999999985551\n",
      "118500 22.0 0.09999999999985551\n",
      "119000 21.3 0.09999999999985551\n",
      "119500 21.0 0.09999999999985551\n",
      "120000 19.9 0.09999999999985551\n",
      "120500 21.0 0.09999999999985551\n",
      "121000 20.3 0.09999999999985551\n",
      "121500 20.6 0.09999999999985551\n",
      "122000 21.6 0.09999999999985551\n",
      "122500 20.0 0.09999999999985551\n",
      "123000 21.1 0.09999999999985551\n",
      "123500 23.4 0.09999999999985551\n",
      "124000 21.5 0.09999999999985551\n",
      "124500 20.5 0.09999999999985551\n",
      "125000 22.2 0.09999999999985551\n",
      "125500 20.9 0.09999999999985551\n",
      "126000 20.3 0.09999999999985551\n",
      "126500 20.5 0.09999999999985551\n",
      "127000 20.5 0.09999999999985551\n",
      "127500 22.0 0.09999999999985551\n",
      "128000 20.2 0.09999999999985551\n",
      "128500 22.4 0.09999999999985551\n",
      "129000 22.7 0.09999999999985551\n",
      "129500 22.4 0.09999999999985551\n",
      "130000 21.0 0.09999999999985551\n",
      "130500 21.0 0.09999999999985551\n",
      "131000 21.1 0.09999999999985551\n",
      "131500 23.4 0.09999999999985551\n",
      "132000 22.1 0.09999999999985551\n",
      "132500 21.2 0.09999999999985551\n",
      "133000 20.7 0.09999999999985551\n",
      "133500 23.5 0.09999999999985551\n",
      "134000 21.7 0.09999999999985551\n",
      "134500 23.6 0.09999999999985551\n",
      "135000 21.4 0.09999999999985551\n",
      "135500 23.7 0.09999999999985551\n",
      "136000 22.3 0.09999999999985551\n",
      "136500 19.7 0.09999999999985551\n",
      "137000 18.3 0.09999999999985551\n",
      "137500 21.7 0.09999999999985551\n",
      "138000 20.9 0.09999999999985551\n",
      "138500 20.9 0.09999999999985551\n",
      "139000 21.8 0.09999999999985551\n",
      "139500 22.6 0.09999999999985551\n",
      "140000 23.9 0.09999999999985551\n",
      "140500 22.2 0.09999999999985551\n",
      "141000 23.0 0.09999999999985551\n",
      "141500 23.7 0.09999999999985551\n",
      "142000 19.0 0.09999999999985551\n",
      "142500 22.5 0.09999999999985551\n",
      "143000 21.1 0.09999999999985551\n",
      "143500 21.5 0.09999999999985551\n",
      "144000 22.5 0.09999999999985551\n",
      "144500 23.5 0.09999999999985551\n",
      "145000 21.2 0.09999999999985551\n",
      "145500 20.8 0.09999999999985551\n",
      "146000 23.0 0.09999999999985551\n",
      "146500 22.3 0.09999999999985551\n",
      "147000 21.4 0.09999999999985551\n",
      "147500 22.1 0.09999999999985551\n",
      "148000 23.7 0.09999999999985551\n",
      "148500 19.3 0.09999999999985551\n",
      "149000 23.2 0.09999999999985551\n",
      "149500 23.4 0.09999999999985551\n",
      "150000 19.9 0.09999999999985551\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 98%|█████████▊| 50/51 [00:00<00:00, 1509.79it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saved Model\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "150500 21.1 0.09999999999985551\n",
      "151000 21.4 0.09999999999985551\n",
      "151500 23.8 0.09999999999985551\n",
      "152000 21.0 0.09999999999985551\n",
      "152500 20.3 0.09999999999985551\n",
      "153000 24.6 0.09999999999985551\n",
      "153500 22.8 0.09999999999985551\n",
      "154000 20.5 0.09999999999985551\n",
      "154500 22.9 0.09999999999985551\n",
      "155000 21.0 0.09999999999985551\n",
      "155500 23.7 0.09999999999985551\n",
      "156000 21.8 0.09999999999985551\n",
      "156500 22.9 0.09999999999985551\n",
      "157000 21.6 0.09999999999985551\n",
      "157500 21.8 0.09999999999985551\n",
      "158000 21.8 0.09999999999985551\n",
      "158500 23.9 0.09999999999985551\n",
      "159000 22.5 0.09999999999985551\n",
      "159500 20.0 0.09999999999985551\n",
      "160000 19.9 0.09999999999985551\n",
      "160500 20.0 0.09999999999985551\n",
      "161000 18.7 0.09999999999985551\n",
      "161500 22.2 0.09999999999985551\n",
      "162000 20.2 0.09999999999985551\n",
      "162500 21.3 0.09999999999985551\n",
      "163000 21.7 0.09999999999985551\n",
      "163500 21.1 0.09999999999985551\n",
      "164000 23.3 0.09999999999985551\n",
      "164500 22.4 0.09999999999985551\n",
      "165000 19.8 0.09999999999985551\n",
      "165500 22.2 0.09999999999985551\n",
      "166000 22.4 0.09999999999985551\n",
      "166500 23.0 0.09999999999985551\n",
      "167000 20.7 0.09999999999985551\n",
      "167500 22.5 0.09999999999985551\n",
      "168000 23.8 0.09999999999985551\n",
      "168500 21.2 0.09999999999985551\n",
      "169000 23.1 0.09999999999985551\n",
      "169500 21.6 0.09999999999985551\n",
      "170000 22.3 0.09999999999985551\n",
      "170500 23.3 0.09999999999985551\n",
      "171000 22.4 0.09999999999985551\n",
      "171500 22.1 0.09999999999985551\n",
      "172000 21.3 0.09999999999985551\n",
      "172500 21.8 0.09999999999985551\n",
      "173000 23.6 0.09999999999985551\n",
      "173500 23.4 0.09999999999985551\n",
      "174000 23.7 0.09999999999985551\n",
      "174500 22.5 0.09999999999985551\n",
      "175000 19.2 0.09999999999985551\n",
      "175500 22.3 0.09999999999985551\n",
      "176000 20.2 0.09999999999985551\n",
      "176500 20.4 0.09999999999985551\n",
      "177000 21.1 0.09999999999985551\n",
      "177500 21.6 0.09999999999985551\n",
      "178000 22.5 0.09999999999985551\n",
      "178500 20.3 0.09999999999985551\n",
      "179000 21.2 0.09999999999985551\n",
      "179500 23.5 0.09999999999985551\n",
      "180000 20.0 0.09999999999985551\n",
      "180500 22.0 0.09999999999985551\n",
      "181000 21.8 0.09999999999985551\n",
      "181500 23.6 0.09999999999985551\n",
      "182000 23.2 0.09999999999985551\n",
      "182500 23.9 0.09999999999985551\n",
      "183000 20.7 0.09999999999985551\n",
      "183500 21.2 0.09999999999985551\n",
      "184000 22.3 0.09999999999985551\n",
      "184500 21.4 0.09999999999985551\n",
      "185000 22.7 0.09999999999985551\n",
      "185500 21.2 0.09999999999985551\n",
      "186000 23.0 0.09999999999985551\n",
      "186500 20.8 0.09999999999985551\n",
      "187000 21.8 0.09999999999985551\n",
      "187500 22.9 0.09999999999985551\n",
      "188000 23.9 0.09999999999985551\n",
      "188500 22.4 0.09999999999985551\n",
      "189000 23.5 0.09999999999985551\n",
      "189500 23.7 0.09999999999985551\n",
      "190000 22.2 0.09999999999985551\n",
      "190500 22.5 0.09999999999985551\n",
      "191000 21.6 0.09999999999985551\n",
      "191500 20.8 0.09999999999985551\n",
      "192000 20.7 0.09999999999985551\n",
      "192500 21.8 0.09999999999985551\n",
      "193000 23.5 0.09999999999985551\n",
      "193500 20.2 0.09999999999985551\n",
      "194000 21.0 0.09999999999985551\n",
      "194500 22.0 0.09999999999985551\n",
      "195000 21.6 0.09999999999985551\n",
      "195500 23.3 0.09999999999985551\n",
      "196000 20.5 0.09999999999985551\n",
      "196500 22.6 0.09999999999985551\n",
      "197000 23.5 0.09999999999985551\n",
      "197500 23.2 0.09999999999985551\n",
      "198000 22.5 0.09999999999985551\n",
      "198500 22.3 0.09999999999985551\n",
      "199000 19.9 0.09999999999985551\n",
      "199500 24.2 0.09999999999985551\n",
      "200000 22.1 0.09999999999985551\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 98%|█████████▊| 50/51 [00:00<00:00, 1529.39it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saved Model\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "200500 21.7 0.09999999999985551\n",
      "201000 21.6 0.09999999999985551\n",
      "201500 22.6 0.09999999999985551\n",
      "202000 21.0 0.09999999999985551\n",
      "202500 22.2 0.09999999999985551\n",
      "203000 21.3 0.09999999999985551\n",
      "203500 23.1 0.09999999999985551\n",
      "204000 19.0 0.09999999999985551\n",
      "204500 21.5 0.09999999999985551\n",
      "205000 20.9 0.09999999999985551\n",
      "205500 21.1 0.09999999999985551\n",
      "206000 22.8 0.09999999999985551\n",
      "206500 20.9 0.09999999999985551\n",
      "207000 22.6 0.09999999999985551\n",
      "207500 21.4 0.09999999999985551\n",
      "208000 21.3 0.09999999999985551\n",
      "208500 21.8 0.09999999999985551\n",
      "209000 23.5 0.09999999999985551\n",
      "209500 22.7 0.09999999999985551\n",
      "210000 20.2 0.09999999999985551\n",
      "210500 20.3 0.09999999999985551\n",
      "211000 20.1 0.09999999999985551\n",
      "211500 22.8 0.09999999999985551\n",
      "212000 21.8 0.09999999999985551\n",
      "212500 18.9 0.09999999999985551\n",
      "213000 21.4 0.09999999999985551\n",
      "213500 21.6 0.09999999999985551\n",
      "214000 22.5 0.09999999999985551\n",
      "214500 22.5 0.09999999999985551\n",
      "215000 22.3 0.09999999999985551\n",
      "215500 21.5 0.09999999999985551\n",
      "216000 23.2 0.09999999999985551\n",
      "216500 23.3 0.09999999999985551\n",
      "217000 22.7 0.09999999999985551\n",
      "217500 23.6 0.09999999999985551\n",
      "218000 21.9 0.09999999999985551\n",
      "218500 20.8 0.09999999999985551\n",
      "219000 21.5 0.09999999999985551\n",
      "219500 21.2 0.09999999999985551\n",
      "220000 22.4 0.09999999999985551\n",
      "220500 23.1 0.09999999999985551\n",
      "221000 21.1 0.09999999999985551\n",
      "221500 22.6 0.09999999999985551\n",
      "222000 22.3 0.09999999999985551\n",
      "222500 23.1 0.09999999999985551\n",
      "223000 21.4 0.09999999999985551\n",
      "223500 24.1 0.09999999999985551\n",
      "224000 22.9 0.09999999999985551\n",
      "224500 19.4 0.09999999999985551\n",
      "225000 23.7 0.09999999999985551\n",
      "225500 22.0 0.09999999999985551\n",
      "226000 24.1 0.09999999999985551\n",
      "226500 20.1 0.09999999999985551\n",
      "227000 20.7 0.09999999999985551\n",
      "227500 20.9 0.09999999999985551\n",
      "228000 23.6 0.09999999999985551\n",
      "228500 22.1 0.09999999999985551\n",
      "229000 22.1 0.09999999999985551\n",
      "229500 22.9 0.09999999999985551\n",
      "230000 23.3 0.09999999999985551\n",
      "230500 24.7 0.09999999999985551\n",
      "231000 22.1 0.09999999999985551\n",
      "231500 21.5 0.09999999999985551\n",
      "232000 21.9 0.09999999999985551\n",
      "232500 21.9 0.09999999999985551\n",
      "233000 23.5 0.09999999999985551\n",
      "233500 21.1 0.09999999999985551\n",
      "234000 20.5 0.09999999999985551\n",
      "234500 24.0 0.09999999999985551\n",
      "235000 23.3 0.09999999999985551\n",
      "235500 19.4 0.09999999999985551\n",
      "236000 23.5 0.09999999999985551\n",
      "236500 22.5 0.09999999999985551\n",
      "237000 22.2 0.09999999999985551\n",
      "237500 22.2 0.09999999999985551\n",
      "238000 23.0 0.09999999999985551\n",
      "238500 23.1 0.09999999999985551\n",
      "239000 22.8 0.09999999999985551\n",
      "239500 20.0 0.09999999999985551\n",
      "240000 21.0 0.09999999999985551\n",
      "240500 22.0 0.09999999999985551\n",
      "241000 21.4 0.09999999999985551\n",
      "241500 22.6 0.09999999999985551\n",
      "242000 22.7 0.09999999999985551\n",
      "242500 23.0 0.09999999999985551\n",
      "243000 23.2 0.09999999999985551\n",
      "243500 21.2 0.09999999999985551\n",
      "244000 23.7 0.09999999999985551\n",
      "244500 22.8 0.09999999999985551\n",
      "245000 23.3 0.09999999999985551\n",
      "245500 18.3 0.09999999999985551\n",
      "246000 22.2 0.09999999999985551\n",
      "246500 22.6 0.09999999999985551\n",
      "247000 22.8 0.09999999999985551\n",
      "247500 21.3 0.09999999999985551\n",
      "248000 22.6 0.09999999999985551\n",
      "248500 20.7 0.09999999999985551\n",
      "249000 24.2 0.09999999999985551\n",
      "249500 21.7 0.09999999999985551\n",
      "250000 23.0 0.09999999999985551\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 98%|█████████▊| 50/51 [00:00<00:00, 1528.09it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saved Model\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "250500 24.4 0.09999999999985551\n",
      "251000 24.2 0.09999999999985551\n",
      "251500 21.0 0.09999999999985551\n",
      "252000 20.7 0.09999999999985551\n",
      "252500 23.6 0.09999999999985551\n",
      "253000 22.5 0.09999999999985551\n",
      "253500 24.3 0.09999999999985551\n",
      "254000 24.3 0.09999999999985551\n",
      "254500 21.4 0.09999999999985551\n",
      "255000 23.5 0.09999999999985551\n",
      "255500 22.7 0.09999999999985551\n",
      "256000 22.3 0.09999999999985551\n",
      "256500 19.9 0.09999999999985551\n",
      "257000 22.4 0.09999999999985551\n",
      "257500 22.4 0.09999999999985551\n",
      "258000 23.6 0.09999999999985551\n",
      "258500 22.0 0.09999999999985551\n",
      "259000 20.7 0.09999999999985551\n",
      "259500 22.9 0.09999999999985551\n",
      "260000 23.1 0.09999999999985551\n",
      "260500 19.9 0.09999999999985551\n",
      "261000 22.3 0.09999999999985551\n",
      "261500 21.1 0.09999999999985551\n",
      "262000 23.3 0.09999999999985551\n",
      "262500 22.7 0.09999999999985551\n",
      "263000 21.3 0.09999999999985551\n",
      "263500 21.3 0.09999999999985551\n",
      "264000 23.9 0.09999999999985551\n",
      "264500 22.7 0.09999999999985551\n",
      "265000 22.5 0.09999999999985551\n",
      "265500 23.8 0.09999999999985551\n",
      "266000 22.7 0.09999999999985551\n",
      "266500 24.0 0.09999999999985551\n",
      "267000 22.6 0.09999999999985551\n",
      "267500 21.2 0.09999999999985551\n",
      "268000 21.9 0.09999999999985551\n",
      "268500 22.9 0.09999999999985551\n",
      "269000 21.5 0.09999999999985551\n",
      "269500 23.5 0.09999999999985551\n",
      "270000 20.2 0.09999999999985551\n",
      "270500 22.8 0.09999999999985551\n",
      "271000 22.9 0.09999999999985551\n",
      "271500 21.8 0.09999999999985551\n",
      "272000 18.3 0.09999999999985551\n",
      "272500 23.3 0.09999999999985551\n",
      "273000 23.4 0.09999999999985551\n",
      "273500 21.6 0.09999999999985551\n",
      "274000 22.2 0.09999999999985551\n",
      "274500 20.6 0.09999999999985551\n",
      "275000 23.8 0.09999999999985551\n",
      "275500 20.4 0.09999999999985551\n",
      "276000 23.8 0.09999999999985551\n",
      "276500 23.9 0.09999999999985551\n",
      "277000 23.5 0.09999999999985551\n",
      "277500 21.0 0.09999999999985551\n",
      "278000 22.5 0.09999999999985551\n",
      "278500 22.4 0.09999999999985551\n",
      "279000 22.8 0.09999999999985551\n",
      "279500 21.2 0.09999999999985551\n",
      "280000 20.0 0.09999999999985551\n",
      "280500 20.3 0.09999999999985551\n",
      "281000 23.9 0.09999999999985551\n",
      "281500 22.3 0.09999999999985551\n",
      "282000 23.4 0.09999999999985551\n",
      "282500 21.7 0.09999999999985551\n",
      "283000 20.5 0.09999999999985551\n",
      "283500 24.2 0.09999999999985551\n",
      "284000 22.4 0.09999999999985551\n",
      "284500 24.1 0.09999999999985551\n",
      "285000 22.6 0.09999999999985551\n",
      "285500 20.6 0.09999999999985551\n",
      "286000 24.2 0.09999999999985551\n",
      "286500 22.5 0.09999999999985551\n",
      "287000 21.2 0.09999999999985551\n",
      "287500 19.6 0.09999999999985551\n",
      "288000 22.9 0.09999999999985551\n",
      "288500 24.3 0.09999999999985551\n",
      "289000 21.7 0.09999999999985551\n",
      "289500 24.2 0.09999999999985551\n",
      "290000 22.9 0.09999999999985551\n",
      "290500 20.7 0.09999999999985551\n",
      "291000 22.2 0.09999999999985551\n",
      "291500 21.6 0.09999999999985551\n",
      "292000 21.6 0.09999999999985551\n",
      "292500 20.6 0.09999999999985551\n",
      "293000 21.9 0.09999999999985551\n",
      "293500 22.2 0.09999999999985551\n",
      "294000 21.6 0.09999999999985551\n",
      "294500 20.7 0.09999999999985551\n",
      "295000 22.6 0.09999999999985551\n",
      "295500 22.1 0.09999999999985551\n",
      "296000 22.4 0.09999999999985551\n",
      "296500 22.3 0.09999999999985551\n",
      "297000 20.2 0.09999999999985551\n",
      "297500 23.0 0.09999999999985551\n",
      "298000 20.3 0.09999999999985551\n",
      "298500 19.9 0.09999999999985551\n",
      "299000 20.9 0.09999999999985551\n",
      "299500 20.5 0.09999999999985551\n",
      "300000 22.0 0.09999999999985551\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 98%|█████████▊| 50/51 [00:00<00:00, 1510.31it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saved Model\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "300500 23.3 0.09999999999985551\n",
      "301000 23.4 0.09999999999985551\n",
      "301500 21.7 0.09999999999985551\n",
      "302000 23.8 0.09999999999985551\n",
      "302500 23.2 0.09999999999985551\n",
      "303000 21.9 0.09999999999985551\n",
      "303500 23.9 0.09999999999985551\n",
      "304000 21.3 0.09999999999985551\n",
      "304500 22.2 0.09999999999985551\n",
      "305000 22.4 0.09999999999985551\n",
      "305500 21.7 0.09999999999985551\n",
      "306000 23.5 0.09999999999985551\n",
      "306500 20.0 0.09999999999985551\n",
      "307000 22.8 0.09999999999985551\n",
      "307500 22.4 0.09999999999985551\n",
      "308000 20.0 0.09999999999985551\n",
      "308500 21.0 0.09999999999985551\n",
      "309000 23.9 0.09999999999985551\n",
      "309500 22.4 0.09999999999985551\n",
      "310000 21.9 0.09999999999985551\n",
      "310500 21.1 0.09999999999985551\n",
      "311000 24.3 0.09999999999985551\n",
      "311500 23.2 0.09999999999985551\n",
      "312000 21.0 0.09999999999985551\n",
      "312500 21.9 0.09999999999985551\n",
      "313000 19.8 0.09999999999985551\n",
      "313500 20.4 0.09999999999985551\n",
      "314000 22.2 0.09999999999985551\n",
      "314500 20.1 0.09999999999985551\n",
      "315000 22.4 0.09999999999985551\n",
      "315500 22.9 0.09999999999985551\n",
      "316000 20.5 0.09999999999985551\n",
      "316500 24.0 0.09999999999985551\n",
      "317000 23.3 0.09999999999985551\n",
      "317500 20.2 0.09999999999985551\n",
      "318000 23.4 0.09999999999985551\n",
      "318500 23.4 0.09999999999985551\n",
      "319000 21.6 0.09999999999985551\n",
      "319500 24.7 0.09999999999985551\n",
      "320000 24.1 0.09999999999985551\n",
      "320500 22.4 0.09999999999985551\n",
      "321000 22.1 0.09999999999985551\n",
      "321500 22.8 0.09999999999985551\n",
      "322000 20.6 0.09999999999985551\n",
      "322500 21.6 0.09999999999985551\n",
      "323000 21.4 0.09999999999985551\n",
      "323500 22.4 0.09999999999985551\n",
      "324000 23.5 0.09999999999985551\n",
      "324500 19.4 0.09999999999985551\n",
      "325000 25.1 0.09999999999985551\n",
      "325500 23.5 0.09999999999985551\n",
      "326000 23.4 0.09999999999985551\n",
      "326500 22.3 0.09999999999985551\n",
      "327000 23.4 0.09999999999985551\n",
      "327500 22.4 0.09999999999985551\n",
      "328000 21.9 0.09999999999985551\n",
      "328500 22.8 0.09999999999985551\n",
      "329000 25.5 0.09999999999985551\n",
      "329500 21.0 0.09999999999985551\n",
      "330000 24.1 0.09999999999985551\n",
      "330500 21.7 0.09999999999985551\n",
      "331000 22.4 0.09999999999985551\n",
      "331500 23.5 0.09999999999985551\n",
      "332000 21.8 0.09999999999985551\n",
      "332500 24.0 0.09999999999985551\n",
      "333000 24.2 0.09999999999985551\n",
      "333500 19.8 0.09999999999985551\n",
      "334000 22.4 0.09999999999985551\n",
      "334500 23.1 0.09999999999985551\n",
      "335000 22.4 0.09999999999985551\n",
      "335500 20.6 0.09999999999985551\n",
      "336000 23.5 0.09999999999985551\n",
      "336500 22.6 0.09999999999985551\n",
      "337000 23.9 0.09999999999985551\n",
      "337500 20.8 0.09999999999985551\n",
      "338000 23.0 0.09999999999985551\n",
      "338500 21.6 0.09999999999985551\n",
      "339000 24.1 0.09999999999985551\n",
      "339500 23.3 0.09999999999985551\n",
      "340000 20.2 0.09999999999985551\n",
      "340500 20.9 0.09999999999985551\n",
      "341000 19.9 0.09999999999985551\n",
      "341500 18.1 0.09999999999985551\n",
      "342000 21.9 0.09999999999985551\n",
      "342500 23.4 0.09999999999985551\n",
      "343000 20.8 0.09999999999985551\n",
      "343500 23.9 0.09999999999985551\n",
      "344000 22.3 0.09999999999985551\n",
      "344500 23.3 0.09999999999985551\n",
      "345000 22.8 0.09999999999985551\n",
      "345500 21.5 0.09999999999985551\n",
      "346000 24.6 0.09999999999985551\n",
      "346500 24.0 0.09999999999985551\n",
      "347000 24.0 0.09999999999985551\n",
      "347500 23.7 0.09999999999985551\n",
      "348000 21.3 0.09999999999985551\n",
      "348500 22.0 0.09999999999985551\n",
      "349000 22.6 0.09999999999985551\n",
      "349500 20.9 0.09999999999985551\n",
      "350000 22.1 0.09999999999985551\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 98%|█████████▊| 50/51 [00:00<00:00, 1508.18it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saved Model\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "350500 23.2 0.09999999999985551\n",
      "351000 20.4 0.09999999999985551\n",
      "351500 20.1 0.09999999999985551\n",
      "352000 21.0 0.09999999999985551\n",
      "352500 21.7 0.09999999999985551\n",
      "353000 23.2 0.09999999999985551\n",
      "353500 21.6 0.09999999999985551\n",
      "354000 22.1 0.09999999999985551\n",
      "354500 21.3 0.09999999999985551\n",
      "355000 21.2 0.09999999999985551\n",
      "355500 19.9 0.09999999999985551\n",
      "356000 22.1 0.09999999999985551\n",
      "356500 22.5 0.09999999999985551\n",
      "357000 23.4 0.09999999999985551\n",
      "357500 22.4 0.09999999999985551\n",
      "358000 20.2 0.09999999999985551\n",
      "358500 22.1 0.09999999999985551\n",
      "359000 22.3 0.09999999999985551\n",
      "359500 23.7 0.09999999999985551\n",
      "360000 20.5 0.09999999999985551\n",
      "360500 23.1 0.09999999999985551\n",
      "361000 22.4 0.09999999999985551\n",
      "361500 20.4 0.09999999999985551\n",
      "362000 23.0 0.09999999999985551\n",
      "362500 22.4 0.09999999999985551\n",
      "363000 21.2 0.09999999999985551\n",
      "363500 21.7 0.09999999999985551\n",
      "364000 22.5 0.09999999999985551\n",
      "364500 23.6 0.09999999999985551\n",
      "365000 22.2 0.09999999999985551\n"
     ]
    }
   ],
   "source": [
    "\n",
    "if saveframes == True:\n",
    "    if not os.path.exists('./DDDQNframes'):\n",
    "        os.makedirs('./DDDQNframes')\n",
    "        \n",
    "tf.reset_default_graph()\n",
    "mainQN = Qnetwork(h_size)\n",
    "targetQN = Qnetwork(h_size)\n",
    "\n",
    "init = tf.global_variables_initializer()\n",
    "\n",
    "saver = tf.train.Saver()\n",
    "\n",
    "#trainable variables of the Main Network and the Target Network\n",
    "trainables = tf.trainable_variables()\n",
    "\n",
    "targetOps = updateTargetGraph(trainables,tau)\n",
    "\n",
    "myBuffer = experience_buffer()\n",
    "\n",
    "#Set the rate of random action decrease. \n",
    "e = startE\n",
    "stepDrop = (startE - endE)/annealing_steps\n",
    "\n",
    "#create lists to contain total rewards and steps per episodes\n",
    "jList = []\n",
    "rList = []\n",
    "total_steps = 0\n",
    "\n",
    "\n",
    "#Make a path for our model to be saved in.\n",
    "if not os.path.exists(path):\n",
    "    os.makedirs(path)\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    sess.run(init)\n",
    "    if load_model == True:\n",
    "        print('Loading Model...')\n",
    "        ckpt = tf.train.get_checkpoint_state(path)\n",
    "        saver.restore(sess,ckpt.model_checkpoint_path)\n",
    "    for i in range(num_episodes):\n",
    "        #GAther episode frames to create gifs eventuly\n",
    "        episode_frames = []\n",
    "        episodeBuffer = experience_buffer()\n",
    "        #Reset environment and get first new observation\n",
    "        s = env.reset()\n",
    "        s = processState(s)\n",
    "        d = False\n",
    "        rAll = 0\n",
    "        j = 0\n",
    "        #The Q-Network\n",
    "        while j < max_epLength: #If the agent takes longer than 200 moves to reach either of the blocks, end the trial.\n",
    "            j+=1\n",
    "            #Choose an action by greedily (with e chance of random action) from the Q-network\n",
    "            if np.random.rand(1) < e or total_steps < pre_train_steps:\n",
    "                a = np.random.randint(0,4)\n",
    "            else:\n",
    "                a = sess.run(mainQN.predict,feed_dict={mainQN.scalarInput:[s]})[0] #Feed through main network to predict action\n",
    "            s1,r,d = env.step(a)\n",
    "            episode_frames.append(s1)\n",
    "            s1 = processState(s1)\n",
    "            total_steps += 1\n",
    "            episodeBuffer.add(np.reshape(np.array([s,a,r,s1,d]),[1,5])) #Save the experience to our episode buffer.\n",
    "            \n",
    "            if total_steps > pre_train_steps:\n",
    "                if e > endE:\n",
    "                    e -= stepDrop #epsilon annealing\n",
    "                \n",
    "                if total_steps % (update_freq) == 0:\n",
    "                    #Get a random batch of experiences from the episode buffer\n",
    "                    trainBatch = myBuffer.sample(batch_size)\n",
    "                    \n",
    "                    #Below we perform the Double-DQN update to the target Q-values\n",
    "                    \n",
    "                    #First we calculate the best actions for state s1 in each experience of the batch using our Main Network\n",
    "                    A = sess.run(mainQN.predict,feed_dict={mainQN.scalarInput:np.vstack(trainBatch[:,3])})\n",
    "                    \n",
    "                    #Then we calculate the Qvalues for every selected experiences in the batch using our Target Network.\n",
    "                    #So Q2 is a 2D-array containing a vector of Q-values for each randomly selected experiences in the batch\n",
    "                    Q2 = sess.run(targetQN.Qout,feed_dict={targetQN.scalarInput:np.vstack(trainBatch[:,3])})\n",
    "                    \n",
    "                    #the end multiplier goes to zero if the experience is an \"end of game\", so the target Q-value = the reward\n",
    "                    end_multiplier = -(trainBatch[:,4] - 1) \n",
    "                    \n",
    "                    #doubleQs are the Q-values estimated from Q2 at state s1 given action A = argmax(Q1(s1,:)). \n",
    "                    #So doubleQ is a vector containing the estimated Q-values for the best action possible \n",
    "                    #for each randomly selected experience in the batch.\n",
    "                    doubleQ = Q2[range(batch_size),A]\n",
    "                    \n",
    "                    # Target-Q = r + gamma*(doubleQ) for non \"end-of-game\" experiences. Otherwise Target-Q = Reward\n",
    "                    targetQ = trainBatch[:,2] + (y*doubleQ * end_multiplier)\n",
    "                    \n",
    "                    #Update the Main network with our target Q-values.\n",
    "                    _ = sess.run(mainQN.updateModel, \\\n",
    "                                   #loss function to optimize                    \n",
    "                        feed_dict={mainQN.scalarInput:np.vstack(trainBatch[:,0]),mainQN.targetQ:targetQ, mainQN.actions:trainBatch[:,1]})\n",
    "                    \n",
    "                    #Update the Target network toward the main network but slowly (with a tau rate)\n",
    "                    updateTarget(targetOps,sess)\n",
    "                    \n",
    "            rAll += r\n",
    "            s = s1\n",
    "            \n",
    "            if d == True:\n",
    "\n",
    "                break\n",
    "        \n",
    "        myBuffer.add(episodeBuffer.buffer)\n",
    "        jList.append(j)\n",
    "        rList.append(rAll)\n",
    "        #Periodically save the model. \n",
    "        if i % 1000 == 0:\n",
    "            saver.save(sess,path+'/model-'+str(i)+'.ckpt')\n",
    "            print(\"Saved Model\")\n",
    "            time_per_step = 0.05\n",
    "            images = np.array(episode_frames)\n",
    "            make_gif(images,'./DDDQNframes/image'+str(i)+'.gif',\n",
    "            duration=len(images)*time_per_step,true_image=True,salience=False)\n",
    "            \n",
    "        if len(rList) % 10 == 0:\n",
    "            print(total_steps,np.mean(rList[-10:]), e)\n",
    "    saver.save(sess,path+'/model-'+str(i)+'.ckpt')\n",
    "print(\"Percent of succesful episodes: \" + str(sum(rList)/num_episodes) + \"%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Checking network learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Mean reward over time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "rMat = np.resize(np.array(rList),[len(rList)//100,100])\n",
    "rMean = np.average(rMat,1)\n",
    "plt.plot(rMean)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}