04_Custom_Gradients.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Quantized Neural Network\n",
    "## 04- Custom Gradients\n",
    "by [Soon Yau Cheong](http://www.linkedin.com/in/soonyau)\n",
    "\n",
    "In last tutorial, we learned about how to quantize a Tensorflow graph and add operations for quantization aware training. Before going to do a training, in this tutorial we'll study the uniquness of gradients in quantized network and learn a new kind of gradient for backpropagation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tensorflow 1.10.0\n",
      "Python 3.5.2 (default, Nov 12 2018, 13:43:14) \n",
      "[GCC 5.4.0 20160609]\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import sys\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "import time\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import utils\n",
    "\n",
    "print(\"Tensorflow\", tf.__version__)\n",
    "print(\"Python\", sys.version)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Vanishing and Exploding Gradients\n",
    "We are almost ready to actually start training quantized network and there is only one more thing to mention - the gradient. In fact any discussion on QNN will not complete without mentioning the fancy word Straight Through Estimator (STE).\n",
    "\n",
    "I assume  you are familiar with backpropagation and gradient descent. In the training, the gradients of network output error is traversed backward to the parameters. For that to works, the functions have to be differentiable i.e. its derivative exists but this is not the case for quantization. As shown below, the quantization plot looks like a staircase with discrete steps. For region where the function is a horizontal line which is almost everywhere, the gradient is 0. As a result, any error backprop to here will become zero (anything multiply by zero is zero) and therefore the weights don't receive gradient for update in the training. On the other hand, at the vertical line where the step happens, the gradient is infinity. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEKCAYAAAASByJ7AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAHSVJREFUeJzt3Xu8VOV97/HPj81FBQQR3V4wAQ2kh9YrOyqJOQ4iCbkS27SmTXI0yTlomrRAkpNq7cmlaV/1mFM5lrRNyImJbU3RJrGaaEyAMDEmgoIh4iVcRGwEPF5wkA3C3nvm1z9mzezZw94za8/MmrVnzff9eu3XzKx51np+PKy9fvtZz7PWMndHRERkVNwBiIjIyKCEICIigBKCiIgElBBERARQQhARkYASgoiIAEoIIiISUEIQERFACUFERAKj4w5gOKZOnerTp0+PtI6DBw8yfvz4SOtIArVTZVu3biWbzTJ79uy4QxnxtC9VV28bbdq06SV3P6lauZZKCNOnT2fjxo2R1pFOp0mlUpHWkQRqp8pSqRSZTCby/TUJtC9VV28bmdmzYcrplJGIiABKCCIiElBCEBERQAlBREQCSggiIgKMgFlGZtYBbAR2u/u7445HpNVd+bWH2PDMPiaO6+BgT5ZcyTOwJo7r4MCRbPHzKINTJx3Dq6/1AnCwJ8ubpk/hkV37eNP0KTz3yiGmnXAcT+7Zz4EjWZbMn8mKn2xn/NgOjvTl6Mt58f1JE8exO3O4WM9HLzmTWx/cWaxr9mmTBmz3/XPOYP3Ol8lkXuOXvdtYv/Pl4vcXn3ki63e+zB3XzGX56vx3F595IgDrd77Mk3v2c/yxY3j/nDMAWLZgVrGesz9/P1u+uLD4unz1Nr6z6Tf8/Lr5ACxfvY1lC2Zx5dce4o5r5hZfC8tL6yvd7vLV24r133HN3AHtXYittHz5/0lhnUI9w1XresMRe0IAlgBPAcfHHYhIEmx4Zh/AgAN/QfmynFM8iJevX3gt/f6WtduP2k7hfWm5A0eyxbJDbbf0+60l7zc8s69YprRc6bLyOkoPlIV4Cq/lcdyydjvLFswaNJ5lC2YNqK90u+XbGSzeoQ7Y5f+eWg7sta43HLEmBDObBrwL+GvgU3HGIpIE923ZG3cIDXXzj7fWVK7wuXT5YO+Heq1W/3CXV4oljF3P9gyrfK0szmcqm9l3gL8BJgKfGeyUkZktBhYDdHZ2zlm1alWkMXV3dzNhwoRI60gCtVNlS5cuJZvNsmLFiqbU9zcbXmPrK7mm1CWNZRW+G+rovOisMVwxc2zoOubNm7fJ3buqlYuth2Bm7wZecPdNZpYaqpy7rwRWAnR1dXnUVzTqqslw1E6VTZ48mUwm07Q2KlQz+3P3c6jn6FNFrWrXje9i+nX3hipXMP26e4vrla5fKDPY94OVH2y7w1leqrDt8vdhpdNprr7/4LDXG644Zxm9BXivme0CVgGXmdm/xBiPSMuLscMvCRBbD8HdrweuBwh6CJ9x9w/FFY9IEjjOKceP4/lXj7TQLKMMCy84a9BZRgBL5s+sOsuo1MRxHQNel8yfyXc2/ab4/ZL5MwG4aMaUAa+F5eX1la9XiKvgohlTjipbrlBH6XaGq9b1hiPWMYRiEP0JoeK0066uLtfN7UYGtVNlhZvbbd68uan1vvEvfsjVb57O9e/8L02ttx7al6prwM3tRvYYQil3TwPpmMMQaXkOlUcpRSrQlcoiSeJgyghSIyUEkQRxHFM+kBopIYgkiLvOGEntlBBEEsRBPQSpmRKCSIK4u8YQpGZKCCIJoh6C1EMJQSRBNIYg9VBCEEkadRGkRkoIIglRuOuA0oHUSglBJCEKd6FRB0FqpYQgkhCFu5JplpHUSglBJCGKp4yUD6RGSggiCdHfQxCpjRKCSEJoDEHqpYQgkhBO4ZSRMoLURglBJCFGwLOupMUpIYgkjDoIUislBJGEKI4haFhZaqSEIJIQ/WMIMQciLUsJQSQh+nsIIrVRQhBJiOJ1CMoIUiMlBJGE6L+5nTKC1EYJQSQh1EOQeikhiCSErkOQesWWEMzsDDNbZ2ZPmtkTZrYkrlhEEqF46wp1EaQ2o2Osuw/4tLs/amYTgU1mttrdn4wxJpGWVZx2GnMc0rpi6yG4+153fzR4fwB4Cjg9rnhEWp1OGUm94uwhFJnZdOB8YMMg3y0GFgN0dnaSTqcjjaW7uzvyOpJA7VRZJpMhm802tY0O9OQzwtM7dpDue7Zp9dZL+1J1zWqj2BOCmU0AvgssdfdXy79395XASoCuri5PpVKRxpNOp4m6jiRQO1U2efJkMplMU9vo5e4j8JM1zJz5BlJvmdG0euulfam6ZrVRrLOMzGwM+WRwu7t/L85YRFpd/7RTjSJIbeKcZWTAN4Cn3P3muOIQSQo9IEfqFWcP4S3Ah4HLzGxz8PPOGOMRaWmaZST1im0Mwd0fRPuuSOPoUmWpk65UFkmIYj6INQppZUoIIgmhMQSplxKCSEL0jyEoI0htlBBEEkI9BKmXEoJIwigfSK2UEEQSQpOMpF5KCCIJoSemSb2UEEQSwjXvVOqkhCCSMMoHUislBJGEcD0xTeqkhCCSELqXkdRLCUEkIXQdgtRLCUEkITTtVOqlhCCSEJp2KvVSQhBJCPUQpF5KCCIJUbwOQaRGSggiiRGcMlIXQWqkhCCSEMVZRvGGIS1MCUEkITSGIPVSQhBJiP4egjKC1EYJQSQhilcqKx9IjZQQRBJCYwhSLyUEkYTQrSukXrEmBDNbaGZbzWyHmV0XZywirc7RhQhSn9FxVWxmHcDfAwuA54BHzOwed38yrpgkuWbdcB99OSdXdswc22H0ZAcuHGVw6qRjePW1Xg4cyQ74buK4Dg72ZI/aTum6AHt2vlyst3T7o4wh161kqPVGGYwelf83FDoG3//VXhb+zqnDr0TaXpw9hAuBHe6+0917gFXAohjjkQTryR6dDArLy+UcdmcOH5UMAA4cGToZFNYt/b58+7Ukg0rr5by/jkKRe7fsra0SaXux9RCA04HflHx+Drio0gpbt24llUpFGROZTIbJkydHWkcStFI7ZQ718PzzB5paZ88LOwF4/tvxnAlNrf9yLPXWopX2pbg0q43iTAihmNliYDHAmDFjyGQykdaXzWYjryMJWqGdtu7Ltu1Z9fXBKaupx45i6rEje5S5FfaluDWrjeJMCLuBM0o+TwuWDeDuK4GVAF1dXb5x48ZIg0qn05H3QpKgldrpD1eu56HgANkshZ7BKX90Y1PrLdh147tiqbcWrbQvxaXeNgp7f6s4xxAeAWaa2QwzGwt8ALgnxngkoTT7RiScij0EM9sCQ/82ufs5tVbs7n1m9kngR0AHcKu7P1Hr9kSqGWymTlSzjIbafpSzjApllsyfOfwKRKh+yujdwesngtd/Dl4/2IjK3f0+4L5GbEtkKO5w4Ywp3HnN3KbVmXroy2QyGTb/9TubVqdIvSomBHd/FsDMFrj7+SVfXWdmjwK6mExGPEe3cxAJI+wYgpnZW0o+vHkY64rES0MIIqGEnWX0MeBWM5sUfM4AH40mJJHG0/19RKoLlRDcfRNwbiEhuPv+SKMSaSDHMXVoRaoK9VtiZp1m9g1glbvvN7PZZvaxiGMTaQh39RBEwgj7Z9O3yE8PPS34vA1YGkVAIo2mIQSRcMImhKnufieQg/w1BMDRd/4SGaHUQxCpLmxCOGhmJxL8sWVmFwMaR5CW4O56zrBICGFnGX2a/G0lzjKznwMnAb8fWVQiDeSohyASRuhZRmZ2KfBG8tf4bHX33kgjExGRpgo7y+hp4L+7+xPu/ri795rZDyKOTaQhXKPKIqGEHUPoBeaZ2TeDO5NC/gE3IiNe/pSRzhmJVBM2IRxy9yuBp4Cfmdnr0Gw+aRXuGlIWCSHsoLIBuPtNwU3tfgxMiSwqERFpurAJ4XOFN+6+xszeDlwVTUgijaVZRiLhVHtAzm+5+6+B3WZ2QdnXGlSWluCu21+LhFGth/Ap8g+4/9tBvnPgsoZHJNJgjmtQWSSEag/IWRy8fYe7Hy79zsyOiSwqERFpurCzjH4RcpnIiKNTRiLhVBtDOIX89QbHmtn59P9eHQ8cF3FsIg2h21+LhFNtDOHtwNXANODmkuUHgD+PKCaRhspfMKOMIFJNtTGE24DbzOz33P27TYpJRERiEPY6hB+Y2R8B00vXcfe/jCIokUZyd50yEgkhbEK4m/zzDzYBR6ILRyQaygci1YVNCNPcfWGjKjWzLwPvAXqAp4GPuHumUdsXKaVBZZFwQk87NbOzG1jvauB33P0c8s9nvr6B2xYRkRqETQiXAJvMbKuZPWZmW8zssVordfcfB89lBlhPfhaTSCQcPUJTJIywp4zeEWEMHwXuiHD70uZ0ykgknLCP0HwWwMxOBkLdssLM1gCnDPLVDe5+d1DmBqAPuL3CdhaTv58SnZ2dpNPpMNXXrLu7O/I6kqCV2ungoUO8+OJrTY03k8mQzWZbpo3i1Er7Ulya1UahEoKZvZf8De5OA14AXk/+YTm/PdQ67n55lW1eDbwbmO8+9EMO3X0lsBKgq6vLU6lUmJBrlk6nibqOJGildjru0Z9y8skTSKXmNK3OyZMnk8lkWqaN4tRK+1JcmtVGYccQvgRcDGxz9xnAfPLn/mtiZguBzwLvdfdDtW5HJAx3jSGIhBH6mcru/jIwysxGufs6oKuOer8CTARWm9lmM/tqHdsSqchBFyKIhBB2UDljZhOAB4DbzewF4GCtlbr7G2pdV6QWygci1YXtISwCXgOWAfeTv5jsPVEFJdJQQ45QiUipsLOMSnsDt0UUi0gk8s9UVh9BpJqws4wO0P931lhgDHDQ3Y+PKjCRRskPKotINWF7CBML7y3/p9Yi8rOORFqCOggi1YUdQyjyvH8n//AckRFPQwgi4YQ9ZfS7JR9HkZ9yejiSiEQaTM9UFgkn7LTT0hlFfcAu8qeNREY8xzWoLBJC2DGEj0QdiEiUlA5Eqgt7yujvKn3v7n/amHBEGm/oO2WJSKmwg8rHABcA24Of88hPP90U/IiMWO6oiyASQtgxhHOASwoPtQnuPfQzd782sshEGkg3txOpLmwP4QSg9CK0CcEykZagMWWR6sL2EG4Efmlm68h3vv8r8IWoghJppAqP2xCREmFnGX3TzH4IXBQs+jN3fz66sEQaR0MIIuGEvlLZ3Z8PHn15vpKBtBqdMhKpbti3rgDe2/AoRCKUv1JZGUGkmloSgn6zpKW47mYkEkotCaF5TyoXaQB3nTISCSNUQjCzWWa21swed/ecmZ1jZn8RdXAijaKEIFJd2B7C14HrgV4Ad38M+EBUQYk0Uv6EkTKCSDVhE8Jx7v5w2bK+RgcjEgVdhiASTtiE8JKZnUXwx5aZvR/YG1lUIg3lOmUkEkLYK5U/AawEfsvMdgPPAB+MLCqRBlM+EKmuYkIws0+VfLwPWEe+V3EQ+D3g5uhCE2kMzTISCafaKaOJwU8X8HHyN7SbDFxL/nbYdTGzT5uZm9nUerclMhQNIYiEU7GH4O5fBDCzB4AL3P1A8PkLwL31VGxmZwBvA/6jnu1IfZav3jbo8mULZgFw5dceAuCOa+YOKLtrVw//uPUhnnvlEK++1svs0yYB8Nwrh9i7/zDjx3YAMPu0SVx85oksWzCLsz9/PweOZBnbYZw0cVyx3IEjWQAmjuvgSF8OgL6ckwuO5KdPPobdmcOMMhg9Kv+n/kkTx/Hqa70c6cvRl3NOnXQMANNOOI5Hdu3j1En96+Qc/mX9f/BX7zu7kU0nkjhhxxA6gZ6Szz3BsnosBz4L3F3ndqQOt6zdPujyQkLY8My+CmX7vystBxQP8hue2ceGZ/axbMGs4rKerLM7c3hAufL3pQplc55ft3RZeZny15y6ByKhhU0I/wQ8bGZ3BZ/fB3yr1krNbBGw291/pYefx+f7v9oz5Hcf/saGip+Hq971RSR6FvZe8WZ2AfDW4OMD7v7LKuXXAKcM8tUNwJ8Db3P3/Wa2C+hy95eG2M5iYDFAZ2fnnFWrVoWKt1bd3d1MmDAh0jridtf2Hu5+ujfuMGJjwDcXjo+0jqVLl5LNZlmxYkWk9SRBO/zO1aveNpo3b94md++qVi50QmgUMzsbWAscChZNA/YAF1a7rXZXV5dv3Lgx0vjS6TSpVCrSOkaKD/2/DTy4Y9A8zK4b3wXA9OvuLX4uvK9Fves3QuHf1AypVIpMJsPmzZubVmeraqffuVrV20ZmFiohhD1l1DDuvgU4ufC5Wg9BotOTzcUdgoiMIE1PCDJy9GZzvG7KsVxx/rQhy1w0Y0rx/ZL5M4vvd+3axfO5iaFnGUF+FlEcs4xEJJzYE4K7T487hnbVm83xhpMnFmcUDeaOa+YW35eWS6f3kErNHWyVIW354sLhBykiTVPL8xAkIXr7nDEd+hNaRPKUENpYbzbHmA7tAiKSp6NBG+vJ5hirhCAiAR0N2ph6CCJSSkeDNtabdcaM1hiCiOQpIbSx3j71EESkn44GbUxjCCJSSkeDNtabzTF2tHYBEcnT0aBNZYMrgXXKSEQKdDRoU73BfYyUEESkQEeDNtVTTAiaZSQieUoIbao3uImcxhBEpEBHgzbVGzyKUqeMRKRAR4M2pTEEESmno0Gb0hiCiJRTQmhThR6CLkwTkQIdDdpUb5/GEERkIB0N2lRPNv/YyjGaZSQiAR0N2lRPsYegMQQRyVNCaFMaQxCRcjoatClNOxWRcjoatCklBBEpp6NBm+oJrlQeqyemiUhACaFNFe9l1NERcyQiMlLElhDM7E/M7Ndm9oSZ3RRXHO2qeMpIPQQRCYyOo1IzmwcsAs519yNmdnIccbQzjSGISLm4jgYfB2509yMA7v5CTHG0rR7d7VREysR1NJgFvNXMNpjZT83sTTHF0bZ0HYKIlIvslJGZrQFOGeSrG4J6pwAXA28C7jSzM93dB9nOYmAxQGdnJ+l0OqqQAeju7o68jpFg29M9APziwQfoGDX8cYR2aadaZTIZstms2igE7UvVNauNIksI7n75UN+Z2ceB7wUJ4GEzywFTgRcH2c5KYCVAV1eXp1KpaAIOpNNpoq5jJHi0Zyu2YweXzUthNvyE0C7tVKvJkyeTyWTURiFoX6quWW0U1/mCfwfmAZjZLGAs8FJMsbSlnqwzpmNUTclARJIplllGwK3ArWb2ONADXDXY6SKJTm82p/EDERkgloTg7j3Ah+KoW/J6sznd6VREBtCfiG0qnxD03y8i/XREaFM9fa6EICID6IjQpnqzOcbqaWkiUkJHhDalMQQRKaeE0KY0hiAi5XREaFNH+pQQRGQgHRHalMYQRKScjghtqjfrujBNRAbQEaFNaVBZRMopIbSpHo0hiEgZHRHaVG82xxiNIYhICR0R2pTGEESknI4IbUpjCCJSTgmhTenCNBEppyNCm9KgsoiU0xGhTfVmXRemicgAOiK0KY0hiEg5JYQ2lMs5fTk9D0FEBtIRoQ315nIASggiMoCOCG2oN+sAug5BRAbQEaEN9fYVeggaQxCRfkoIbag3GyQEzTISkRI6IrShniAh6JSRiJTSEaENFccQ1EMQkRKxHBHM7DwzW29mm81so5ldGEcc7ap4ykg9BBEpEdcR4Sbgi+5+HvC54HOklq/eFsu6w10/bNlCudLy5euWlym8fvDr6wH44Za9oeMSkeSLKyE4cHzwfhKwJ+oKb1m7PZZ1h7t+2LKFcqXly9ctL1N4fbG7B4DvP6aEICL9RsdU71LgR2b2f8gnpTdHWdmK4EC44OafVi178NAhxj96dLkw61YynPXDli2UKy1fvm55mXr/HSKSXObu0WzYbA1wyiBf3QDMB37q7t81sz8AFrv75UNsZzGwGKCzs3POqlWrQsdw1/Ye7n6696jlp443Tp8weOco29dHx+jR7O7Osffg0W1Tad1Sw1k/bNmhyjXCorPGcMXMsaHLd3d3M2HChEhiSYKlS5eSzWZZsWJF3KGMeNqXqqu3jebNm7fJ3buqFnT3pv8A++lPRga8Gma9OXPmeK1e/2c/CFVu3bp1Na9bb93DKVsoV1q+fN3yMqWvhZ9aDdZO0u/SSy/1c889N+4wWoL2perqbSNgo4c4xsY1hrAHuDR4fxlQ30l6ERGpW1xjCP8DuMXMRgOHCU4JRWnJ/JmxrDvc9cOWLZQrLV++bnmZwuvpk49h2gnHcfGZJ4aOS0SSL7IxhCh0dXX5xo0bI60jnU6TSqUirSMJ1E6VpVIpMpkMmzdvjjuUEU/7UnX1tpGZhRpD0JVJIiICKCGIiEhACUFERAAlBBERCSghiIgI0GKzjMzsReDZiKuZCrwUcR1JoHaqTm0Ujtqpunrb6PXuflK1Qi2VEJrBzDaGmZ7V7tRO1amNwlE7VdesNtIpIxERAZQQREQkoIRwtJVxB9Ai1E7VqY3CUTtV15Q20hiCiIgA6iGIiEhACSFgZr9vZk+YWc7Musq+u97MdpjZVjN7e1wxjjRm9gUz221mm4Ofd8Yd00hhZguD/WWHmV0XdzwjkZntMrMtwb4T7V0rW4iZ3WpmL5jZ4yXLppjZajPbHryeEEXdSgj9Hgd+F3igdKGZzQY+APw2sBD4BzPraH54I9Zydz8v+Lkv7mBGgmD/+HvgHcBs4A+D/UiONi/YdzTttN+3yB9rSl0HrHX3mcDa4HPDKSEE3P0pd986yFeLgFXufsTdnwF2ABc2NzppMRcCO9x9p7v3AKvI70ciVbn7A8C+ssWLgNuC97cB74uibiWE6k4HflPy+blgmeR90sweC7q5kXRjW5D2mXAc+LGZbQqenS5D63T3vcH754HOKCqJ64lpsTCzNcApg3x1g7vf3ex4WkGlNgP+EfgS+V/sLwF/C3y0edFJi7vE3Xeb2cnAajP7dfDXsVTg7m5mkUwPbauE4O6X17DabuCMks/TgmVtIWybmdnXgR9EHE6raOt9Jix33x28vmBmd5E/1aaEMLj/b2anuvteMzsVeCGKSnTKqLp7gA+Y2TgzmwHMBB6OOaYRIdgxC64gPzAv8Agw08xmmNlY8pMS7ok5phHFzMab2cTCe+BtaP+p5B7gquD9VUAkZzTaqodQiZldAawATgLuNbPN7v52d3/CzO4EngT6gE+4ezbOWEeQm8zsPPKnjHYB18Qbzsjg7n1m9kngR0AHcKu7PxFzWCNNJ3CXmUH+OPRtd78/3pBGBjP7VyAFTDWz54DPAzcCd5rZx8jf8fkPIqlbVyqLiAjolJGIiASUEEREBFBCEBGRgBKCiIgASggiIhJQQpBEMLM/NbOnzOx2M7vazL5S43ZSZvbmks/Xmtl/a1ykw4qlat1mdp7uMiuNousQJCn+GLjc3Z8zs6vr2E4K6AZ+AeDuX60/tNqErPs8oAvQnWalbuohSMszs68CZwI/NLNlZd9NN7OfBDfgW2tmrwuWv8fMNpjZL81sjZl1mtl04FpgWXCP/rcGz3z4TLBO2sz+t5k9bGbbzOytwfLjzOxOM3vSzO4KtnvU7ZyD+//fFDwD4GEze0OVGCvWHVwF/ZfAlUG8V0bTwtIulBCk5bn7tcAe8vfWX1729QrgNnc/B7gd+Ltg+YPAxe5+PvnbU3/W3XcBX6X/GQ8/G6S60e5+IbCU/BWkkO+dvOLus4H/BcypEO5+dz8b+Arwf6vEWLHu4NbanwPuCOK9o0K9IlUpIUjSzQW+Hbz/Z+CS4P004EdmtgX4n+QfgBTG94LXTcD04P0l5JMK7v448FiF9f+15HVulRjD1C3SMEoI0q5WAF8J/lq/Bjgm5HpHgtcstY3B+RDvm1G3SEVKCJJ0vyB/t1GADwKF00CT6L8l9VUl5Q8AE4dZx88JbjYWPCrz7Aplryx5fahKjGHUEq/IoJQQJOn+BPiImT0GfBhYEiz/AvBvZrYJeKmk/PeBKwqDyiHr+AfgJDN7Evgr4Alg/xBlTwhiWQIUBsCHijGMdcBsDSpLI+hupyJ1MrMOYIy7Hzazs4A1wBuDQd/ScruALnd/aZDNiMRO5yFF6nccsM7MxgAG/HF5MhBpBeohiIgIoDEEEREJKCGIiAighCAiIgElBBERAZQQREQkoIQgIiIA/CckmG5QdQKYXAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "np.random.seed(1515)\n",
    "input_dim = [1, 1000]\n",
    "x = np.sort(3*np.random.randn(1000).astype(np.float32)).reshape(input_dim)\n",
    "\n",
    "tf.reset_default_graph()\n",
    "num_bits = 2\n",
    "input = tf.placeholder(tf.float32, input_dim)\n",
    "\n",
    "# gradient functions will be described below\n",
    "output = tf.fake_quant_with_min_max_vars(input, min=-6.0, max=6.0, num_bits=num_bits)\n",
    "grad = tf.fake_quant_with_min_max_vars_gradient(output, input, min=-6.0, max=6.0, num_bits=num_bits)\n",
    "grad_func = tf.gradients(output, input)\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    [x_quant, gradients, gradients_func] = sess.run(\\\n",
    "                                                      [output, grad, grad_func], \\\n",
    "                                                      feed_dict={input:x})\n",
    "\n",
    "fig, ax = plt.subplots()\n",
    "\n",
    "ax.plot(x.reshape(-1), x_quant.reshape(-1), '-o', marker='+',)\n",
    "ax.set_xlabel(\"floating point\")\n",
    "ax.set_ylabel(\"de-quantized\")\n",
    "ax.axhline(y=0, color='k')\n",
    "ax.axvline(x=0, color='k')\n",
    "ax.grid(True)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In example above, I used only 2-bits quantization to make the steps look more dramatic. If we look at the y-axis, the minimum quantized value is -8 which is not the minimum -6.0 we set. Since we want to make sure 0 is quantized to 0, therefore the 4 discrete values are (-8, -4, 0, 4) and the min and max are shifted by 2 to negative direction. These are some minor details to watch out for when using very small number of bits for quantization."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Straight Through Estimator (STE)\n",
    "\n",
    "To overcome this problem, Straight Through Estimator (STE) is used as the gradient function. Despite of the fancy name, it really is just an identity function that pass the incoming gradient straight through as it is if it fall within min-max-range, otherwise zeros. To put into the quantization context, that effectively assume that the fake quantization is equal to the original values. \n",
    "\n",
    "There are a few TF functions with regards to gradients for quantization. Let's introduce them briefly here.\n",
    "\n",
    "__tf.gradients__(y,x) return the partial derivative function of y with respect to x. If we print out the gradient function using tf.gradients, we'll see that there is a FakeQuantWithMinMaxVarsGradient operator. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[<tf.Tensor 'gradients/FakeQuantWithMinMaxVars_grad/FakeQuantWithMinMaxVarsGradient:0' shape=(1, 1000) dtype=float32>]\n"
     ]
    }
   ],
   "source": [
    "print(grad_func)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we run TF session to read tf.gradients(), it will return numerical values that is to multiply with the errors flowing in from higher layer. In the code example above, we have already run the session to return the grad_fun. We print some of the values and plot it on graph, we'll see it is mostly 1.0 except for quantized values that is out of range."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0, 0.5, 'gradient function')"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaUAAAEKCAYAAACymEqVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3XmcXFWd9/HPL91Zm0DCIiABEzDxMTMMKmETlIbITFAUeZwZQPFRx4eADk5EUXF4BhVnHFwZB3CJiguLgCjDKhgwjYOSSIIRJBgSkiCEMMjS9JKl09W/5497q7u6urvqdtXdqvv7fr2Kqrp1zz0nl+r7q7Pcc8zdERERyYMJWRdARESkSEFJRERyQ0FJRERyQ0FJRERyQ0FJRERyQ0FJRERyQ0FJRERyQ0FJRERyQ0FJRERyoznrAozGhAkTfOrUqYnm0dfXx4QJitXV6DxVtmPHDtydpL+vY4G+S9XVe462bdvm7t4QJ7mhgtLUqVPp7u5ONI+2tjZaW1sTzWMs0HmqrLW1lfb2dtasWZN1UXJP36Xq6j1HZrY9vtIkqyEip4iIjA8KSiIikhsKSiIikhsKSiIikhsKSiIikhuZj74zsyZgFbDF3U/Jujwije70bz/Ayk0vMn1yE909BfpK1vGcPrmJzp2F/vcTDPbfYwod23cB0N1T4IjZe/Lg5hc5YvaePP3SNmbNnMbaZ16mc2eBJQvncvkv19MyqYmdvX309nn/632mT2ZL+47+fP7huIO56v6N/XnNf+Ueg477t4cfyIqNL9Devp3f7XqcFRtf6P/86IP3YsXGF7jhnGO4bFnw2dEH7wXAio0vsPaZl9l96kT+9vADATj/pHn9+Rz6mbt45HOL+p8vW/Y4N61+il9fuBCAy5Y9zvknzeP0bz/ADecc0/9c3F6aX+lxL1v2eH/+N5xzzKDzXSxb6f7l/0+KaYr5jFat6aIws6uAU4Dn3P0vh/ncgK8DbwW2Ae9394eSKEvmQQlYAjwG7J51QUTGgpWbXgQYFHyKyrf1Of2BpDx98bn086/fu37IcYqvS/fr3Fno33ek45Z+vq7k9cpNL/bvU7pf6bbyPEov1sXyFJ/Ly/H1e9dz/knzhi3P+SfNG5Rf6XHLjzNceUcKGuX/nlqCS63pIvoBcAXwoxE+PxmYGz6OAr4ZPscu06BkZrOAtwH/Bnwsy7LkwvLlwaMBzN68GX75y6yLkVsf2LSJHTt3wsUXp5rv+uc6Of+RZ1PNE+ChA17LfQcfHvtxv/aLdTXtV3xfun241yM9V8t/tNsrlSWKzU/2jGr/0XL3X5nZ7Aq7nAr8yN0dWGFmM8xsf3ffGndZLMgjG2Z2E/DvwHTgguGa78xsMbAYoLm5+fBly5YlWqauri522223RPMYyeGLFzN9/XrcLJP8JT7FvytL6f+l9/8nfRNwnthzFgvP/lY2BWhwlb4hI/0vPfWQiZw2d1LkPE444YQe4JGSTUvdfemgcgRB6fYRmu9uBy519/vD9/cCn3L3VZELEVFmNSUzK7Zfrjaz1pH2C0/cUoCWlhZP+s7vTO8u7+2Fs87Crr46m/xHQXfhV3ZCyjM6FC9s8y++i209Q5vtkvQft32Zw7Y+nsixN1/6NmZfeEek/YpmX3hHf7rS9MV9hvt8uP2HO+5otpcqHrv8dVRtbW28/67uUacr0evuC2pNnKYsR98dC7zDzDYD1wMnmtk1GZYnex0dsLu61qR2GTZ8yNi2BTiw5P2scFvsMqspufungU8DhDWlC9z9rKzKkzl3BSWpm+Pst/tknu3Ymdrou4kTDMM4YMaUGkfftbPoDYcMO/oOYMnCuVVH35WaPrlp0POShXO5afVT/Z8vWTgXgKPm7Dnoubi9PL/ydMVyFR01Z88h+5Yr5lF6nNGqNV1MbgXOM7PrCQY4vJxEfxIQtH1n/QBaCdoyK+43bdo0T9ry5csTz2NY27e7g/sXvpBN/qOU2XlqEMcff7wfdthhqec776I7/Qt3rE0303e/2/2QQ2pOru9SdfWeI6DbK1+DfwxsBXYBTwMfBM4Fzg0/N+BK4AmCvqkFlY5XzyMPQ8Jx9zagLeNiZKuzM3iePj3bckhD6798pEkDcxqeu59Z5XMH/jGNsmhGh7zo6Aie1Xwn9XCw1KOSSHwUlPJCQUli4Hg2FReNsJCYKCjlRTEoqflO6uCefuudSJwUlPKi2KekmpLUwcmgi0d9ShIjBaW8UPOdxMDd1ackDU1BKS8UlCQGmdSUQH1KEhsFpbxQn5LEQH1K0ugUlPKiszP4idvSknVJpNGlXVVSn5LESEEpL4pTDOkPXGrkxZnJMy6HSD0UlPJC895JnYrdOupTkkamoJQXnZ3qT5K6FMOCRt9JI1NQygvVlKRO/c13uk9JGpiCUl4oKEmdBmpKIo1LQSkvFJSkTupTkrFAQSkv1KckdXKKzXeqK0njUlDKC9WUpE6ZVVYUBCVGCkp50NcX1JQUlCQGihHSyBSU8qC7O/iZq+Y7qUN/n1IWQx3UpyQxUVDKAy1bITEY6FPKuCAidVBQygPNEC4xGKgppUxRUGKkoJQHCkoSg/77lBQjpIEpKOWBlq2QGAxMyKo+JWlcCkp5oD4liYFqSjIWKCjlgZrvJAa6T0nGgsyCkpkdaGbLzWytmT1qZkuyKkvmFJQkDv3TDClISONqzjDvXuDj7v6QmU0HVpvZMndfm2GZslFsvlOfktShf0h4JpmrT0nikVlNyd23uvtD4etO4DHggKzKk6mODpg0CSZPzrok0sDUfCdjQZY1pX5mNht4PbBymM8WA4sBmpubaWtrS7QsXV1diedRbu5jj7HP1Kn8JuV865HFeWok7e3tFAqFVM9RZ08QlZ7YsIG23idTy/c1W7cyc+dOVtT4b9V3qbrxdI4yD0pmthvwU+Cj7t5R/rm7LwWWArS0tHhra2ui5WlrayPpPIb47ndhr73Sz7cOmZynBjJjxgza29tTPUcvdO2EX97D3LmvpvXYOanly9VXw6RJNf9b9V2qbjydo0xH35nZRIKAdK27/yzLsmRKy1ZIDAaGhKs5TRpXlqPvDPge8Ji7fy2rcuSClq2QGGS2yJ+CYMMzs0Vmts7MNpjZhcN8flA4Wvp3Zvawmb01qbJkWVM6FngvcKKZrQkfif1Dc01BSWKQ6eg7aVhm1gRcCZwMzAfONLP5Zbv9P+BGd389cAbwjaTKk1mfkrvfj/5+Ah0dMG9e1qWQRpfllA4aEt7IjgQ2uPtGADO7HjgVKL09x4HiL+c9gGeSKkzmAx0ELfAnseiPSZmWQhrQAcBTJe+fBo4q2+ezwC/M7CNAC/CWpAqjaYbyQM13EgP1KUkFzWa2quSxeJTpzwR+4O6zgLcCV5tZIvFDNaWs7doF27crKEndBvqUFCRkiF53XzDCZ1uAA0vezwq3lfogsAjA3R8wsynA3sBzcRdUNaWsaYohiUlmNaXSzKURPQjMNbM5ZjaJYCDDrWX7/AlYCGBmrwWmAH9OojAKSlnTshUSM9WTZDTcvRc4D7ibYLq3G939UTO7xMzeEe72ceBsM/s98GPg/e7J/BJR813WNEO4xCSzwXfqU2p47n4ncGfZtotLXq8luI0ncaopZU1BSWKS6cqzIjFRUMqa+pQkJp7lmHD1KUlMFJSyppqSxEz1JGlkCkpZU1CSmHhWK8+qT0lipKCUNQUliYnmvpOxQEEpa8U+pd12y7Yc0vB0n5KMBQpKWevogJYWaGrKuiTS4LKcj1UkLgpKWdO8dxKTzIaEKwpKjBSUstbRoeHgEgvVlGQsUFDKmpatkJhk2q2jPiWJiYJS1tR8J7EJm+9UVZIGpqCUNQUliUn/6Lu0M1YQlBgpKGVNfUoSE/UpyVigWcKzpj4liclATSmDqKQ+JSllNhl4FzCb0jjjfkm1pApKWXJX853Epn9GBy1dIdm7BXgZWA3sHE1CBaUs7dgBvb0KShKLzPqURIaahfuiWhKqTylLWrZCYpTpNEMig/0Gs0NrSZhpUDKzRWa2zsw2mNmFWZYlE5qMVWLkZNivoz4lGew4YDVm6zB7GLNHMHs4SsLMmu/MrAm4EjgJeBp40MxuDZfdHR8UlFIz76I76e1z+squnZOajJ7C4I0TDPbfYwod23fRubMw6LPpk5vo7ikMOU5pWoBnNr7Qn2/p8ScYI6atZKR0EwyaJwT/hmIF6bbfb2XRX+4/+kxqpaqZDHVyrQmzrCkdCWxw943u3gNcD5yaYXnSp6CUmp7C0IBU3F6uz2FL+44hAQmgc+fIAamYtvTz8uPXEpAqpevzgTyKu9zxyNbaMhGJi/uTwAzg7eFjRritqiwHOhwAPFXy/mngqEoJduzYQWtra5Jlor29nRkzZiSaR9Exzz/PvwPnXHAB6xqsXynN81Sv9m09PPtsZ6p59jy3EYBnr8umVbp1xZdTy2vJ+vWc2N7OqTX+bTbSdykrDXeOzJYAZwM/C7dcg9lS3C+vljT3o+/MbDGwOHxNe3t7ovkVCoXE8yiycKDD1m3baC8M/VWeZ2mep1qte7GQZS9LplaEzYd7T53A3lOTbV7r2bmTPveavw+N8F3KWgOeow8CR+HeDYDZF4EHgFwHpS3AgSXvZ4XbBnH3pcBSgJaWFl+zZk2ihWpra0u8Ntbvm9+ED3+YW5cvh/32SyfPmKR6nup05tIVPBBepNNSrCHt9+5LU823aPOlb0svs/POg+uvp9a/zUb6LmWl3nOUwXyIBpT+0i4Q8W6FLIPSg8BcM5tDEIzOAN6dYXnSV+xTarCmu0aT6ag0kfHp+8BKzG4O378T+F6UhJkFJXfvNbPzgLuBJuAqd380q/JkorMTJkyAadOyLsm4MNwItqRG3410/CRH3xX3WbJw7ugzqJeGhEsp969h1kYwNBzgA7j/LkrSTPuU3P1O4M4sy5Cp4hRDGlKbKHc4cs6e3HjOManl2frAl2lvb2fNv701tTxFMme2O+4dmO0JbA4fxc/2xP3FaofI/UCHMU3z3qXC0dQ7idKPKhlwHXAKwZx3pdVnC98fXO0ACkpZ0rIV6VBUEkmH+ynh85xaD6G577KkZStSox/zCVOfkpQyuzfStmGoppSljg6YOTPrUox5jmP6/SWSPLMpwDRgb8xmMtBGsTvBhAlVKShlqaMDXvWqrEsx5rmrppQonVwZcA7wUeCVBP1KxS9HB3BFlAPo52OWOjvVp5QCNSyJVBZlxQYz+3szW2tmj5rZdcMeyP3rYX/SBbgfjPuc8HEY7pGCUsWakpndRoW/aXd/R5RMZAQafZca/ZhPmPqUGlaUFRvMbC7waeBYd3/JzF5R5bB9mM3AvT08wEzgTNy/Ua081WpKXwG+CmwCtgPfCR9dwBPVDi4V9PVpoENK3B3T8DuRkURZseFs4Ep3fwnA3Z+rcsyz+wNSkOCl8BhVVawpuft9AGb2VXdfUPLRbWa2KkoGMoKuruBZQSlxjmpKidLJbQTNZdfspeG8ohBtxYZ5AGb2a4IZeD7r7ndVyK8JM8PDKnRQG5sUqaBRdgJazOxgd98YFmwO0BIxrQxHS6GLSHp6yyoWo9UMzAVaCSbP/pWZHeqltaHB7gJuwOzb4ftzwm2RMorifKDNzDYSjKZ4VZiJ1EoL/KVG3R0p0EluZFFWbHgaWOnuu4BNZvY4QZB6cIRjfoogRnwofL8M+G6UwkQKSu5+V9jR9b/CTX90951R0soIFJRSEzTfqYlJZARRVmz4L+BM4PtmtjdBc97GEY/o3gd8M3yMSqQh4WY2DfgEcJ67/x44yMxOGW1mUkLLVqTHXcMckqSA39DcvRcortjwGHCjuz9qZpeYWXGE9d3AC2a2FlgOfMLdR16kzOxYzJZh9jhmGzHbRNDSVlXU5rvvE9wIVZxmeQvwE+D2iOmlXLFPSTUlEcnYcCs2uPvFJa8d+Fj4iOJ7BN0+qxm82F9VUYPSIe5+upmdGRZwm6k9pD5qvkuNRt+lQH1KMtjLuP+8loRRg1KPmU0lvJHWzA4B1KdUDwWl1LhrkvBEKeLLUMsx+zLwM0pjhftD1RJGDUqfJRjOd6CZXQscC7x/tKWUEupTSo3jGuggkq7ifU6lw9AdOLFawqij735hZquBowl+dC5x9+dHW0op0dkJkyfDpEj3k4mINA73E2pNGikoWbAOxlfd/Y6SbUvdfXGtGY97mvcuNWq+S4H6lKSU2cXDbne/pFrSqLOEzwE+ZWafKdlWz93BoqCUGi1dkTCdXBmqu+RRAE4GZkdJGLVPqR1YCPxnOHP4WaMvowyiZStSE06+lXEpRMYR968Oem/2FYJ7naqKWlMyd+919w8DPwXuB6pNXS6VqKYkY4ma76SyaQTTF1UVtab0reILd/+BmT0C/GMNBZOijg44INLqwFInd1cLk0iaghhR/KXSBOwDVO1PguqL/O3u7h3AT8xsz5KPNgEX1FBUKerogNe+NutSjBuKSQlSxJciszm4bwJKp6HrBf6HYDqjqqrVlK4LD76a8Mb4ks8cODh6aQdYcFPV24EegsUCP1BhCvSxSX1KqdFAB5HU3AQcDlyF+8JaDlBtkb9Twuc5tRy8gmXAp92918y+SLDM7qdiziPf1KckY4n6lCQwAbN/BuZhNnSePPevVT9ABOF9SlW3ReXuv/CBqtwKInaAjRk9PbBjh4JSShwthy6SkjMIhoA3A9OHeVRVrU9pCsGoib3NbCYDzXe7EyyhG4d/AG6I6ViNQavOpkrNdwnTyZUi93XAFzF7OKkJWc8BPgq8kqBfqfjt6wCuqJTQzO4B9hvmo4vc/ZZwn4sIOsGurXCcxcBigObmZtra2qoUuT5dXV2J5zHl2Wc5GvjjM8/wbMJ5JSWN8xSX7m3b+POft6da3vb2dgqFQsOco3oc8tRTvLJQ4L9r/Lc20ncpKw13jmoMSGFar/oAPhJlv9E8CCZ0fQCYFjXNtGnTPGnLly9PPA///e/dwf2mm5LPKyGpnKeYLPxqm3/omlWp5nn88cf7YYcdlmqemfn4x93r+NtspO9SVuo9R0C3x3wNT+oRdULWy83sjQTTRDSXbP9RLYHQzBYBnwSOd/dttRyjoWnZilS5q09JJFVmk3HfWXXbMKJOyHo1cAiwhoFVBB2oKSgRNP1NBpaFSwqscPdzazxW49GyFakKFgHLuhRjmPqUZKgHgDdE2DZE1BkdFgDzw2pg3dz91XEcp2FpKfTU6bIpkgKz/QgGwU3F7PUMHhw3LcohogalPxAMWtg62jLKMNR8ly7dQpM83ackgb8hGC8wCyi9J6kT+OcoB4galPYG1prZbylZ2tbd3xExvZRSUEqVg1aeFUmD+w+BH2L2Ltx/WsshRrMcusSl2Hy3227ZlmOcCAY6SGIU8GWo2zF7N2WD46Is8hd19N19tZZMhtHREQSkCVFXDpF66bopkqpbgJcJ7m+tOuKuVNTRd0cDlwOvBSYRTEXe7e5qf6qF5r1LlXo7UqA+JRlsFu6LakkY9af6FcCZwHpgKvB/gStryVBQUEqZu0bfiaTsN5gdWkvCyO1H7r4BaHL3grt/H6gpCgpatiJljmugQ5J0bmWo44DVmK3D7GHMHsHs4SgJow502GZmk4A1ZvYlgqHh6hCplWpKqdNlUyRVJ9eaMGpgeS9BP9J5QDdwIPCuWjMd9xSUUqXujhToJEsp9ycJ4sSJ4ettRIw3UUffPRm+3A58rpYySomODjXfpcjL10yWeKn5TsqZfYZgJqDXAN8HJgLXAMdWSxp19N0mhhnE5O41LYc+7nV2qqaUMk3IKpKq04DXAw8B4P4MZpF+iUdtvlsAHBE+3gT8J0HUk9FyV/NdBvRjXmRkZrbIzNaZ2QYzu7DCfu8yMzezBVUO2ROuUeRhwpaoZYkUlNz9hZLHFnf/D+BtUTOREtu3Q6GgoJSimOYRlkp0jhuWmTUR3OJzMjAfONPM5g+z33RgCbAywmFvxOzbwAzMzgbuAb4TpTxRm+9KpxufQFBzijpyT0pp2YrUqUspYaqGNrojgQ3uvhHAzK4HTgXWlu33eeCLwCeqHtH9K5idRLBK+WuAi3FfFqUwUQPLVxnoU+oFNgN/FzGtlNKyFZnQdVNkRAcAT5W8fxo4qnSHsGJyoLvfYWbVgxIQBqFIgahU1KB0O4N/cDpwSvGGRHf/2gjppJxmCE9dMKODolKi1HyXd81mtqrk/VJ3XxoloZlNIFiG4v0Rdr4f9+Mw62Tw4DgDnAhT00UNSocTDHK4JTz424HfEkw7JKOhoJQ61+x3Ir3uPtLghC0E9xQVzQq3FU0H/hJoCysi+wG3mtk73L000IH7ceFzzf0TUYPSLOAN7t4JYGafBe5w97NqzXjcKjbfqU8pNe5qvkuUTm6jexCYa2ZzCILRGcC7ix+6+8sEa+oBYGZtwAVDAlLw4Z4Vc3J/sVphogalfYGekvc94TYZLdWUMqHrpsjw3L3XzM4D7iaYuecqd3/UzC4BVrn7raM43GoGunoOAl4KX88A/gTMqXaAqEHpR8Bvzezm8P07gR+MoqBSpKCUuvBGiYxLMcapT6mhufudwJ1l2y4eYd/WCgcKgo7Zd4CbCY4LZicTxI2qot6n9G/ABwii3kvAB9z936OklTIKSqnT9VIkdUf3ByQA958Db4ySMPK9Ru7+EMUpI6R2nZ3Q1ARTpmRdknHE1XyXJJ1cGeoZzP4fAzP/vAd4JkpCLT+RtuIUQ/pDTpXOtkiqzgT2AW4OH68It1WlWRnSpnnvUqfRdylQG6mUCkbZLaklaaZBycw+DnwF2Mfdn8+yLKnRshWp0+VSJGVm+wCfBP4CGOircD+xWtLMgpKZHQj8NcEwwfEjZ8tWXLbs8WG3n3/SPABO//YDANxwzjGD9t28uYdvrnuAp1/aRsf2Xcx/5R4APP3SNra+vIOWSU0AzH/lHhx98F6cf9I8Dv3MXXTuLDCpydhn+uT+/Tp3FgCYPrmJnb19APT2OX1hNDlgxhS2tO9ggkHzhKDKs8/0yXRs38XO3j56+5z99wi+97NmTuPBzS+y/x4DafocrlnxJ/71nYfGeeqkSNVQGepa4AbgFOBc4H3An6MkzLKmdBlBJL0lwzKkr6MD9tor61L0+/q9w0/KUQxKKze9WGHfgc9K9wP6A83KTS+yctOLnH/SvP5tPQVnS/uOQfuVvy5V3LfPg7Sl28r3KX/uUzVJJAt74f49zJbgfh9wH2YPRkmYSVAys1OBLe7+extvv7I6OmBO1fvHUnHb70ceDPPe762s+H606k0vOac+JRlsV/i8FbO3EYy8qzzbQyixoGRm9xDMkVTuIuCfCZruohxnMbAYoLm5mba2triKOKyurq5E8zjm+ed5sauLdQn/Oyq5eX0Ptzyxq+I+/73++YrvR6ve9HGYfeEdQDAS7/uLIq85VpP29nYKhULi39c8mPPkkxwE3FfjvzXpv7mxoAHP0b9itgfwceByYHfg/CgJLe0F0MzsUOBeYFu4aRZBFD3S3Z+tlLalpcW7u7sTLV9bWxutra3JZTB9Opx9Nnwt+4nVz/ruSu7fMHyw2HxpsIZj8UK++dK39b+uRb3p41D8N6WhtbWV9vZ21qxZk1qemfmXf4EvfCFYvLIGif/NjQH1niMz2+buyf4SG8isCfgn3C+rJXnq9ym5+yPu/gp3n+3uswnW7nhDtYA0JhQK0NWVm4EOPYW+rIsgImONe4GI9yQNR/cppamrK3jOSVDaVejjoD2nctrrZ424z1FzBpqBlyyc2/968+bNPNs3PfLoOwhG12Ux+k5SoD4lGezXmF1BMAJvoHkrmBmoosyDUlhbGh9ytmzFrkIfr37F9P6RdsO54Zxj+l+X7tfW9gytrccMl2REj3xu0egLKSKN6HXh8yUl2xzI731K41LOJmPd1etMbFJVQuo03kbQSnXuJ9SaVEEpTXkLSoU+JjZp+kMRiZnZx4bZ+jKwGveKo390RUpTzoJST6GPSQpKEgf1KclgCwhmcjggfJwDLAK+g9knKyVUTSlNOexTUk1J6qbmOxlqFvAG3IPRXWafAe4A3kywOu2XRkqoK1KaclZT2lVwJjbrgiIisXsFsLPk/S5gX9y3l20fQjWlNOUtKPWqpiQiibgWWIlZcW7TtwPXYdYCrK2UUEEpTcWglJPmO/UpiUgi3D+P2c+BY8Mt5+K+Knz9nkpJFZTS1NkZLIM+cWLWJQGCPqVJzQpKUif1KclwgiC0qup+ZXRFSlOOVp0thDMmqPlORPJEV6Q05Sgo7QrnvVNQkthoWLjEQFekNOVoKfSe/qCkphcRyQ8FpTTlaCn0XeHEp+pTkrqpT0lipCtSmnLVfBc0taj5TkTyRFekNOUqKKlPSWKmPiWJga5IaersVJ+SiEgFCkppymFNSTfPSt3UpyQx0hUpLTt3Bo+8BKVe9SmJSP7oipSW4gzhOQlKPYVgCfKJGn0ncVGfksRAV6S05GzZip7+mpKaXkTGOzNbZGbrzGyDmV04zOcfM7O1Zvawmd1rZq9KqiwKSmnJ2wzh6lOSuKhPqaGZWRNwJXAyMB8408zml+32O2CBu/8VcBMV1kOql65IaclpUFKfksi4dySwwd03unsPcD1waukO7r7c3beFb1cQLOKXCF2R0pKzZSsUlCR26lPKs2YzW1XyWFzy2QHAUyXvnw63jeSDwM+TKCRo6Yr05G6gQ3ABmaSVZ0XGg153X1DvQczsLGABcHz9RRqeglJa8tZ8V5z7rqkp45JIw1OfUqPbAhxY8n5WuG0QM3sLcBFwvLtXXNK8Hpm13ZjZR8zsj2b2qJkl1mmWG3kLSsXmO9WURMa7B4G5ZjbHzCYBZwC3lu5gZq8Hvg28w92fS7IwmdSUzOwEgo60w9x9p5m9IotypKoYlFpasi1HSH1KEjv1KTUkd+81s/OAu4Em4Cp3f9TMLgFWufutwJeB3YCfWFAz/pO7vyOJ8mTVfPch4NJiFTDpyJsLxXnvJuQjCPRolnARCbn7ncCdZdsuLnn9lrTKktUVaR7wJjNbaWb3mdkRGZUjPTma9w50n5LESH1KEqPEakpmdg+w3zAfXRTmuydwNHAEcKOZHew+tP4fDl1cDNCvJIafAAALdElEQVTc3ExbW1tSRQagq6srkTzmP/EELc3NPJhw+aN6/IkeAH5z/69omjD6i0pS52msaG9vp1AojItzdNCmTRwM3HfffXjz6C8p+i5VN57OUWJBqVJ1z8w+BPwsDEK/NbM+YG/gz8McZymwFKClpcVbW1uTKXCora2NRPKYMgX23TeZY9fgoZ512IYNnHhCK1bDL93EztMYMWPGDNrb28fHOfr1rwE4/s1vhkmTRp1c36XqxtM5yqrt5r+AEwDMbB4wCXg+o7KkI2fNdz0FZ2LThJoCkohIUrIKSlcBB5vZHwimtHjfcE13Y0rOgtKuQp/6kyQe+mEjMcpk9F04v9JZWeSdmRwGJc0QLiJ5o5/KacnRUuhQDEr63y8xGuONHZIOXZXS4J67mlJPrysoSTzUfCcx0lUpDdu2QV9froLSrkIfk7TqrIjkjK5KacjZshWgPiURyScFpTTkbNkKUJ+SJEB9ShIDXZXSkLMZwgF29iooSUzUpyQx0lUpDTkMSupTEpE80lUpDbnsU3LdPCvxUvOdxEBXpTTktk9JzS4iki8KSmnIYfNdj/qUJC7qU5IY6aqUhhwGpV2FPiaqT0lEckZXpTR0dkJzM0yenHVJ+qlPSWKnPiWJga5KaShOMZSjZg71KYlIHikopSFn896Bbp6VGOXox5Y0Pl2V0pDDoKSBDiKSR7oqpSFny1ZA2KekgQ4SJ/UpSQx0VUpDDmtK6lMSkTxSUEpDzoJSX5/T26f1lCQm6lOSGOmqlIaOjlw13+3q6wNQUBKR3NFVKQ2dnbmqKe0qBG3/uk9JYqU+JYmBrkpJKxSguztfQam3WFNSs4uI5IuCUtJyOhkroGmGJB7qU5IY6aqUtBwuW9ETBiU134lI3uiqlLRc1pTCPiXVlCRO6lOSGGRyVTKz15nZCjNbY2arzOzILMqRipzOEA4afSciATNbZGbrzGyDmV04zOeTzeyG8POVZjY7qbJkdVX6EvA5d38dcHH4PlGXLXs8k7SjDUpR8yruV7p/edryfYrP7/nOCgB+/sjWSHmJVKQ+pYZmZk3AlcDJwHzgTDObX7bbB4GX3P3VwGXAF5MqT1ZByYHiVXoP4JmkM/z6veszSdvffBexTylqXsX9SvcvT1u+T/H5z109ANz2sIKSiHAksMHdN7p7D3A9cGrZPqcCPwxf3wQsNEvm10hzEgeN4KPA3Wb2FYLA+MYkM3vgA+fzi9t+xuYfVI/BB/X1sXnC4P1+0dsXKe1wdtvexd7AWTc8yv8sezFSmpO+dt+o9ivdvzxt+T5Rjy0yakccAU1No0/W3Q0tLQkUaOw4orsbrrkGjjsuicMfADxV8v5p4KiR9nH3XjN7GdgLeD7uwiQWlMzsHmC/YT66CFgInO/uPzWzvwe+B7xlhOMsBhYDNDc309bWFrkMN6/v4ZYndnH6swXevPeB/dunNsPU5uGDvPc5NsHY3uts7x36eaW0pfrTz4Dn5h7Dr3e14M91sX+LccBugwPclq4+tnYPdBKvf64LYMi+I+1XfC5/Pdw+5Z/PvvAOAE49ZCKnzZ1U9d9V1NXVNar/F+NNe3s7hUJhXJyjafvsw6tOPBErFGpK3ztzJt3NWf0+bgy9M2fy2GOP0dU7zEUpmmYzW1Xyfqm7L42haPFz99QfwMuAha8N6IiSbtq0aV6rV33q9kj7LV++vOa09eY9mn2L+5XuX562fJ/S5+KjVsOdJxlw/PHH+2GHHZZ1MRqCvkvV1XuOgG4f+Xp8DHB3yftPA58u2+du4JjwdTNBDclGOmY9j6z6lJ4Bjg9fnwjU0WkjIiJ1eBCYa2ZzzGwScAZwa9k+twLvC1//LfDLMNjFLqs689nA182sGdhB2DyXpCUL52aSdrTpo+5b3K90//K05fsUnw+YMYVZM6dx9MF7RS6XiIxNHvQRnUdQG2oCrnL3R83sEmCVu99K0MVytZltAF4kCFyJsISCXSJaWlq8u7s70Tza2tpobW1NNI+xQOepstbWVtrb21mzZk3WRck9fZeqq/ccmdk2d2+I0SS6e1JERHJDQUlERHJDQUlERHJDQUlERHJDQUlERHKjoUbfmVkfsD3hbJqBmm+bHkd0nqrTOYpG56m6es/RVHdviEpIQwWlNJjZKndfkHU58k7nqTqdo2h0nqobT+eoISKniIiMDwpKIiKSGwpKQ+Vz5tz80XmqTucoGp2n6sbNOVKfkoiI5IZqSiIikhsKSiEz+zsze9TM+sxsQdlnnzazDWa2zsz+Jqsy5o2ZfdbMtpjZmvDx1qzLlBdmtij8vmwwswuzLk8emdlmM3sk/O6sqp5ifDCzq8zsOTP7Q8m2Pc1smZmtD59nZlnGJCkoDfgD8L+BX5VuNLP5BNO0/wWwCPiGmY1+zeex6zJ3f134uDPrwuRB+P24EjgZmA+cGX6PZKgTwu/OuBjuHNEPCK41pS4E7nX3ucC94fsxSUEp5O6Pufu6YT46Fbje3Xe6+yZgA3BkuqWTBnMksMHdN7p7D3A9wfdIpCp3/xXBmkWlTgV+GL7+IfDOVAuVIgWl6g4Anip5/3S4TQLnmdnDYZPDmG1SGCV9Z6Jx4BdmttrMEl/os8Ht6+5bw9fPAvtmWZgkZbXybCbM7B5gv2E+usjdb0m7PI2g0jkDvgl8nuDi8nngq8A/pFc6aXDHufsWM3sFsMzM/hjWEqQCd3czG7PDpsdVUHL3t9SQbAtwYMn7WeG2cSHqOTOz7wC3J1ycRjGuvzNRufuW8Pk5M7uZoNlTQWl4/2Nm+7v7VjPbH3gu6wIlRc131d0KnGFmk81sDjAX+G3GZcqF8I+j6DSCwSICDwJzzWyOmU0iGChza8ZlyhUzazGz6cXXwF+j708ltwLvC1+/DxizLTvjqqZUiZmdBlwO7APcYWZr3P1v3P1RM7sRWEswS+8/unshy7LmyJfM7HUEzXebgXOyLU4+uHuvmZ0H3A00AVe5+6MZFytv9gVuNjMIrkPXuftd2RYpH8zsx0ArsLeZPQ18BrgUuNHMPgg8Cfx9diVMlmZ0EBGR3FDznYiI5IaCkoiI5IaCkoiI5IaCkoiI5IaCkoiI5IaCkowJZvZPZvaYmV1rZu83sytqPE6rmb2x5P25ZvZ/4ivpqMpSNW8ze51mZ5exRPcpyVjxYeAt7v60mb2/juO0Al3AbwDc/Vv1F602EfN+HbAA0AztMiaopiQNz8y+BRwM/NzMzi/7bLaZ/TKcNPZeMzso3P52M1tpZr8zs3vMbF8zmw2cC5wfrvHzpnDNqAvCNG1m9kUz+62ZPW5mbwq3TzOzG81srZndHB53yFIM4fpBXwrXEPqtmb26Shkr5h3OFnEJcHpY3tOTOcMi6VFQkobn7ucCzxCszXNZ2ceXAz90978CrgX+M9x+P3C0u7+eYGmJT7r7ZuBbDKwR9d/DZNfs7kcCHyW40x6CWtpL7j4f+Bfg8ArFfdndDwWuAP6jShkr5h0ui3ExcENY3hsq5CvSEBSUZKw7BrgufH01cFz4ehZwt5k9AnyCYBHHKH4WPq8GZoevjyMIbLj7H4CHK6T/ccnzMVXKGCVvkTFFQUnGq8uBK8JayznAlIjpdobPBWrrk/URXqeRt0juKSjJWPcbglm6Ad4DFJvk9mBgOYn3lezfCUwfZR6/JpwgM1z2/NAK+55e8vxAlTJGUUt5RXJLQUnGuo8AHzCzh4H3AkvC7Z8FfmJmq4HnS/a/DTitONAhYh7fAPYxs7XAvwKPAi+PsO/MsCxLgOKgjJHKGMVyYL4GOshYoVnCRepkZk3ARHffYWaHAPcArwkHIpTutxlY4O7PD3MYEUHt0iJxmAYsN7OJgAEfLg9IIhKNakoiIpIb6lMSEZHcUFASEZHcUFASEZHcUFASEZHcUFASEZHcUFASEZHc+P9/KCG8MDmQZAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots()\n",
    "\n",
    "ax.plot(x.reshape(-1), x_quant.reshape(-1), '-o', marker='+',)\n",
    "ax.set_xlabel(\"floating point\")\n",
    "ax.set_ylabel(\"quantized\")\n",
    "ax.axhline(y=0, color='k')\n",
    "ax.axvline(x=0, color='k')\n",
    "ax.grid(True)\n",
    "\n",
    "ax2=ax.twinx()\n",
    "ax2.plot(x.reshape(-1), np.array(gradients_func).reshape(-1), color='red')\n",
    "ax2.set_ylabel('gradient function', color='r')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The last function is __tf.fake_quant_with_min_max_vars_gradient()__ which perform the backpropagation. In practice, all it does is to multiply the quantized value with the gradient function element wise as shown in plot below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f4e4c1424e0>]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaUAAAEKCAYAAACymEqVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3XmcHGW96P/Pt7tnerYkM9kJAZJAwITIfgibMoBIQITr1SvgwR/xgIEDCCKgqPdyEO89R4WoILhEwLghURSJgGDADMiSAcISshxCSECSkIQsk9mX7v7+/qjqTs/aNT3dVd0z3/fr1XR3dT31PCl66tvPUs8jqooxxhhTCEJBF8AYY4xJsqBkjDGmYFhQMsYYUzAsKBljjCkYFpSMMcYUDAtKxhhjCoYFJWOMMQXDgpIxxpiCYUHJGGNMwYgEXYDBCIVCWl5entc8EokEoZDF6kzsPA2svb0dVSXf39fhwL5LmQ31HLW2tqqqFsVJLqqgVF5eTktLS17zqKuro7a2Nq95DAd2ngZWW1tLQ0MDr732WtBFKXj2XcpsqOdIRNpyV5r8KorIaYwxZmSwoGSMMaZgWFAyxhhTMCwoGWOMKRgWlIwxxhSMwEffiUgYeBnYoqrnBl0eY4rR8iuu4P1//ANESPi0cKemngVx38VCJWwaP4OHPrqAfztlBvc9uzG1/+wpY3jpnd38y7SxbN7TymeOPYAVG3fR0NDGq13rWbFxV+rzE2aMY8XGXSy5/ER+sMz57IQZ4wBYsXEXa7fuZXR5CZ859gAArjvz0FQ+H/6Px3njW/NSzz9Ytp4HV77HczedAcAPlq3nujMP5YKfvcCSy09MPSe3p+eXftwfLFufyn/J5Semtl/wsxdSZUvfP10yj/T8ByvbdF6IyH3AucAOVZ3Tx+cC3AGcA7QC81X1lXyUJfCgBFwLrANGB10QY4rV5BNOcIKSKuJTnvvy2RcESxNdvD1+Jk0dce546q1u+9dv2t3tOf3zN9Ne12/andonfb/0bUC3PNIv1k0d8W7PPctxx1Nvcd2Zh/ZZnuvOPLRbfunH7XmcvsrbX9Do+e/JJrhkm86jxcBdwK/6+fxsYKb7mAv8xH3OuUCDkohMBT4B/D/gK0GWpSAsX+48isC0d96Bv/896GIUrC9s2kR7RwfcfLMv+c0Cxh9yCNuef96X/Hr6oKKC96uqeGz2uTx38Edzdtzv/+3NrPZLvk/f3tfr/p4z5T/Y7QOVxYt33u0c1P6DparPiMi0AXY5H/iVqiqwQkSqRWQ/VX0/12UR9amq32fmIg8C/wWMAm7oq/lORBYACwAikcixy5Yty2uZmpubqaqqymse/Tl2wQJGvfUWKn791jX5kvy7Ep/+X6r7H8H/v2cB9paWctfxH+PnJ1/pe/7FbqBvSH//N88/uIRPzSz1nMdpp53WCbyRtmmRqi7qVg4nKD3ST/PdI8B3VPVZ9/1TwNdU9WXPhfAosJqSiCTbL1eKSG1/+7knbhFAZWWl5vvO70DvLo/F4OKLkV//Opj8B8Huwh/YaT7P6CDAusWLeeW223zJL91Jmzczrr2dabvf4eS3n8lpTemd73yCaTc96mm/pGk3PZpKl54+uU9fn/e1f1/HHcz2dMlj93ztVV1dHfMfbxl0ujQxVT0u28R+CnL03cnAeSLyDvAAcLqI/CbA8gSvsRFGW9eaGbx1ixfzagABqadz1j7CyW8/E3QxTO5tAQ5Iez/V3ZZzgdWUVPXrwNcB3JrSDap6cVDlCZyqBSWTtW0rVgD9N/fkQ8+8Ejij7w7e+RarZp82iNF3Dcw75uA+R98BXHvGzIyj79KNioa7PV97xkweXPle6vNrz5gJwNzpY7s9J7f3zK9numS5kuZOH9tr356SeaQfZ7CyTZcjS4GrReQBnAEOe/PRnwQ4bd9BP4BanLbMAferqKjQfFu+fHne8+hTW5sqqP7nfwaT/yAFdp6KxKmnnqpHHnmk7/kunnOE/vjfv+5rnjtmz9am8vKs09t3KbOhniOgRQe+Bv8OeB/oAjYDlwJXAFe4nwtwN/A2Tt/UcQMdbyiPQhgSjqrWAXUBFyNYTU3O86hRwZbDFDVJ/cfPTG1gTrFT1YsyfK7AVX6UxWZ0KBSNjc6zNd+ZIVELEqaoWVAqFBaUTC6of8PQ01kYNLliQalQJIOSNd+ZIRCrKZkiZ0GpUCT7lKymZIbI95gk4oweNSYHLCgVCmu+MzkgqtiftSlm9u0tFBaUTK5Y650pYhaUCoX1KZkcCKG+D3QIYmCFGb4sKBWKpianbb6yMuiSmGLnc5Cw3iSTSxaUCkVyiiH71WmylEgk3Ff2HTLFy4JSobB578wQacKts/j9w8ZG35kcsqBUKJqarD/JDElyGXTf+5R8zc0MdxaUCoXVlMwQabL5zua+M0XMglKhsKBkhkgDqikZk0sWlAqFBSUzRKluHQtKpohZUCoU1qdkhiihTvOd1ZRMMbOgVCispmSGKDX6zu9OJQuCJocsKBWCRMKpKVlQMkORbL/z+6/agpLJIQtKhaClxbmgWPOdGYJEqqYUwJ+13adkcsSCUiGwZStMDijJPqWAC2LMEFhQKgQ2Q7jJAU3NMmR9SqZ4WVAqBBaUTA7YfUpmOLCgVAhs2QqTA6kJWQMIShYGTa5YUCoE1qdkcsBqSmY4sKBUCKz5zuSABjUCTsTWVDI5E1hQEpEDRGS5iKwVkTUicm1QZQmcBSWTC6mBDvZb0xSvSIB5x4DrVfUVERkFrBSRZaq6NsAyBSPZfGd9SmYI9k0z5HPGIojdp2RyJLCfVKr6vqq+4r5uAtYB+wdVnkA1NkJpKUSjQZfEFLF90wwZU7yCrCmliMg04Gigvo/PFgALACKRCHV1dXktS3Nzc97z6GnmunVMKC/neZ/zHYogzlMxaWhoIB6P+3qO9u52atw7d+70Nd/pra2UQ9Z52ncps5F0jgIPSiJSBfwR+LKqNvb8XFUXAYsAKisrtba2Nq/lqaurI9959HLPPTBunP/5DkEg56mIVFdX09DQ4Os5ev+9bSwHJkyc6Gu+uyorEcg6T/suZTaSzlGgPaIiUoITkH6rqn8KsiyBsmUrTA4EtRy6MbkU5Og7Ae4F1qnq94MqR0GwZStMDqT6lHwOSmpBsOiJyDwReVNENojITX18fqA7WvpVEVklIufkqyxB1pROBj4PnC4ir7mPvP1DC5oFJZMDNiGryYaIhIG7gbOB2cBFIjK7x27/G/i9qh4NXAj8OF/lCaxPSVWfxWYncTQ2wqGHBl0KU+xSNSV/f2sK2NIVxe14YIOqbgQQkQeA84H023MUSP5yHgNszVdhAh/oYLAF/kxOaEALz1rVrOjtD7yX9n4zMLfHPrcAfxORLwGVwMfyVRi79bsQWPOdyQGb+84MICIiL6c9Fgwy/UXAYlWdCpwD/FokP1VyqykFrasL2tosKJkhS83o4HdVyYJgMYip6nH9fLYFOCDt/VR3W7pLgXkAqvqCiJQB44EduS6o1ZSCZlMMmRxJjb4LWZAwg/ISMFNEpotIKc5AhqU99vkncAaAiMwCyoAP8lEYC0pBs2UrTI75XlMyRU1VY8DVwBM40739XlXXiMitInKeu9v1wBdF5HXgd8B8zdO09NZ8FzSbIdzkyL6BDtZ8ZwZHVR8DHuux7ea012txbuPJO6spBc2CkskRTc0SbkHCFC8LSkGzPiWTI0HN6IBYg6HJHQtKQbOaksmV1JDwgMthzBBYUAqaBSWTI6l+Z79XnnWjYGDLsZthxYJS0CwomRxJWE3JDAMWlIKW7FOqqgq2HKboJfuUAhnooGrz35mcsKAUtMZGqKyEcDjokpgipwQ00CGZvwUlkwMWlIJm896ZHAmsppTMz4KSyQELSkFrbLTh4CYnAp+Q1YKSyQELSkGzZStMjuwbfedvULKbdU0uWVAKmjXfmVxxZ3SwPiVTzCwoBc2CksmRfTHJ5r4zxcuCUtCsT8nkiPUpmeHAZgkPmvUpmRwJLCi5c99Z851JEYkCnwamkR5nVG/NlNSCUpBUrfnO5IwG3KdkTJqHgb3ASqBjMAktKAWpvR1iMQtKJicCGnxn9ymZvkxFdV42Ca1PKUi2bIXJoUCnGQILSibd84h8OJuEgQYlEZknIm+KyAYRuSnIsgTCJmM1OZRqvgssfwtKJuUUYCUibyKyCpE3EFnlJWFgzXciEgbuBs4ENgMvichSd9ndnFh7772MmzOHSXPnprZtr69n1+rVzL700ryl9ZzeglJeLTnuOOJtbQD4fbm+zM3wN4cf7lueCoSBxu9/i/u//y0AjvrKVzx9X7Pxl098gvadOznyrbc4EPjD3LmUVFXRlWwBSCfSvX0xOSgjEkFjMe4PhQhHowBU7rcfTf/8JxqLEamqorSqis7mZhKdnSRiMcLRKBWTJ1M6ahRN771HV3MzGotRMXkyHQ0NTJo7l4nHHMOqu+5CVdnv5JPZXl/PEVdfzeqf/AQpKWHcnDlUTplCy9atVE6Zwvb6euLt7cQ6Ojj8ssvQeJxxc+bw4re/TUdDA+FolA99/vPMmj+f7fX1PP+Nb5Do7CRUWspJ//mfTJo7l3WLF/Pu449Tc9hhjDrwQLa/9BKTTziBWfPnp64Hu9etY9uKFZz20596up4k06UbzHUoQGdnmzDImtLxwAZV3aiqncADwPm5zGDcnDk8e/31bK+vB5z/mc9ef32v/8m5Tus5vQWlvKo+9NDUa/H7Ie7DxzyTf8ypxjsRz9/XbEyaO5eu5uZ9G1T7DkjuZ3291ljMeZFIEG9rI97WRuPGjantseZmWrdtI9baSqKzM7Vf06ZN7Fq1is49e9CuLlCl9f33CZWUsLWujjX33EOisxPt6mJrXR2VU6bw6m230dXcTOeePXQ2NrJhyRJatm5lw5IlSDhM67ZtdO7Zw6u33Ubzli3UXXklTZs20blnD+07d/Lqbbfx4q238sw119C2Ywcdu3fT/sEHPHPNNbx46628etttNLz5Jv98/HHGzZnD5BNO4NXbb2fd4sWMmzOHp6+6ildvu43JJ5zg+XqSvI50rF8PDP46FBjVd4Fq4JPuo9rdlpEEVeUWkc8A81T1Mvf954G5qnp1f2nC4bB+5CMfGVQ++3V2clZjI42hENXxOFtLSmgbYEbujo4Oou4vtvJ4nCldXTSEw57S9pRM/0EkQnU8zpOjR/N+aWnq8xN37uS/1qzh8mOO4c0i61dqaGiguro66GJk9Mndu5kcj4PvX/Nkhn5P+eNe8wUeGTOm2/ctH05qamL+pk0c2NjIHw87LK95DSQBtAMV7uv+fm0r0CpCpSoxnKai5HM7UNbH/h+Ew4yPx1PBP6lJhFE9rp8JYEvadaK6q4uJ8ThtIpSrM4/77kFeT5LXkfVlZUzv6Oh1HfHi6aefblXVykElGgqRa4EvAn9yt3wKWITqjzIlLfjRdyKyAFjgvqahoWFQ6RuAo1Q5IB6nA+dLUt3V1e/+Ckh7e+p9HBjvMW1/x9svFmOlCOtaW6G1NfWZuL8q329tpSEeH9RxgxaPxwf9/8Jvb+6OswK4pwpCvvf9BzcsWwSWdsCfNzcxvjzE+PL8leUx4F+T+ebh+AMFmHQxnL9VcJow3bicek4eqwuoUiUBlLifl7iflbMvQCV14gSFGBDtkR+qxHvsH6P3dSIOVLhBMNvrSQyY3d7O86FQr+tIgboUmItqCwAi3wVeAAo6KG0BDkh7P9Xd1o2qLgIWAVRWVuprr702qEyS1d2ZF1zAW0uWcMrChd36eXqqq6ujtrY2q7R95f3MNdfQ1dzMCVVVfPWOO7qn/8lP4MorWbp8OUyePKh/V9DSz1Mhe+Jzn2Pn66+PlIpSynllwh2/vWdQ39dsvHjrrZT88Id5O76n/oVwmKrycqYcfDC7Xn8dQiEkkdj3DCBCSJXyaJQJxxzD9hdeoGz8eNp37kw918yezZ613bu0K8vLOcHti4q1taHxOIRCRBIJPlRby9a6OgiFIJFAIhGqyso45847U+d93eLFvHr77Uw45hg+WLmSsvJyPnTJJYO6nmyvr2f5Ndcw++KLiS5Zwv8Z5HXI+ecH8qss/Zd2HI9/DUH2Kb0EzBSR6SJSClwILM1lBsmgcsrChRzxpS9xysKF3fp58pU2Pf2RX/4yALMuvbR3+mSfUpE13RWLJz73OeciNRKp8tSll3r+vmbjxVtvZcOSJXk7fjehAS5V8Thdzc3sev11SkaPhkQi9Zw0esYMABIdHWx/4QXGHXkk7Tt3Mvrgg1PP6QHpkAsuIFxWRqytjVdvu41Yezsaj3PIBRdQUlEBoRBb6+qQcJiSigoOueACNBYj3tnJM9dcw/b6+lRAOvqGGzjiqquIlJcTa2ujZNQoz9eT5HWk5tJLs7oOBegXQD0ityByC7ACuNdLwsCCkqrGgKuBJ4B1wO9VdU0u89i1enW3XyOT5s7llIUL2bV6dV7Tpqff7+STAaiYOLF3+qYm54+tomKQ/zLjRYPbOaxBPNzVwf3MM+E+UrVCVc/f12xsr6+npKoKcH8Ci1DS3w+s9F/qaa8l4jbWhEKEy8sJl5czesaM1PZIVRUVkycTqaggVFqa2m/U9OmMO+IISmtqkJISEKFiv/1IdHUxpbaWwy+7jFBpKVJSwpTaWlq2buXoG2+kpKqK0poaSkeP5pALLqByyhQnoMTjzoi+mhqOvvFGqvbfn9of/5hR06dTWlND2fjxHH3jjRx/88189M47KZ84kejYsZRNmMBH77yT42++maNvvJHqww7jwHnz2LV6NdtWrODoG25g1vz57Fq9mlPvvpujb7yRbStWeL6eJK8jUXfQzmCvQ4FR/T7wBWC3+/gCqp6q1IENdMhGZWWltrS05DWPXDdLdTU384e5czn6xhuZNX9+9w+vuQZ+/WvYsydn+fmlWJrvAO78+KdA4ZplD/mWZ21tLQ0NDQy2ubkoXX01/O53sGtXVsmL6bsUlKGeIxHxZ6CDyGhUGxEZ2+fnqrszHaLgBzoUu0hlJaFIhI7dffy/sHnv/JHe221yz+baM/vcD5yLM+ddeo0nOe5kRqYDWFDKMxEhOnYsHX2NVLNlK3xiUckYX6ie6z5Pz/YQNvedD6I1NbT3VVOyZSv8Y7/m86uIugGMD0Se8rStD1ZT8kG0poaOvvqNGhuhpsb/Ao00ajUlY3whUoZzD/N4RGrY94c3GtjfyyEsKPmgrKaG3Wv7mNKvsREOOsj/Ao1EFpPyx2qhZp/LgS8DU3D6lZJfjkbgLi8HsOY7H0Rramjvq6bU1GR9Sn6wpiVjBuRlxQYR+ayIrBWRNSJyf58HUr3D7U+6AdUZqE53H0ei6ikoDVhTEpG/MMCsYap6npdMRrpoTQ1djY0kuroIlZTs+8BG3/nIfs3nlQX+ouVlxQYRmQl8HThZVfeIyMQMh00gUo1qg3uAGuAiVH+cqTyZakq3AwuBTUAb8HP30Qy8nengxlE21hmy320EXiJhAx18Iqg1MRnTPy8rNnwRuFtV9wCo6o4Mx/xiKiA5Cfa4x8howJqSqj4NICILVfW4tI/+IiIve8nAODUlcIJS+YQJzsbklP8WlPJPsaCUT3Zui0GkxzV7kTuvKDgDEN5L+2wz0HNyvUMBROQ5nDlvb1HVxwfIL4yIpFZ+dGpjnqY29zrQoVJEZqjqRrdg0wH/pkEvcqmglD4s3JZCN8b4J9ajYjFYEWAmUIszefYzIvJhTa8Ndfc4sASRn7nvL3e3ecrIi+uAOhHZiNM4f5CbifEgGZS6DXawBf58pP7PEj7SWJ9SMfOyYsNmoF5Vu4BNIrIeJ0i91M8xv4YTI/7dfb8MuMdLYTwFJVV93O3o+pC76b9VtcNLWpNWU7KgFAy1PiVjBpBasQEnGF0IfK7HPn8GLgJ+ISLjcZrzNvZ7RNUE8BP3MSiehoSLSAVwI3C1qr4OHCgi5w42s5Eq6q7Q2mdQsuY7n1hQyhsL+EWtvxUbRORWEUmOsH4C2CUia4HlwI2q2v8MvCInI7IMkfWIbERkE05LW0Zem+9+gXMj1Inu+y3AH4BHPKYf0UKRCKWjR3cPSsk+JaspGWMCpqqP4SwknL7t5rTXCnzFfXhxL063z0q6L/aXkdegdLCqXiAiF7kFbJUAljIsZtGxY61PKSiqVlHKN+tTMt3tRfWv2ST0GpQ6RaQc90ZaETkYsD6lQYhWV3cffWdByUc2911e2e9T09tyRG4D/kR6rFB9JVNCr0HpFpzhfAeIyG+Bk4H5gy3lSFY2dizNmzfv22B9Sr4Ru0/JGL8l73NKH4auwOmZEnodffc3EVkJnIDzk/NaVd052FKOZNGaGna98ca+DU1NEI1Cqaf7yYwxpnionpZtUk9BSZx1MBaq6qNp2xap6oJsMx5pkstXqCoiYvPe+cr6lPLO+pRMOpGb+9yuemumpF5nCZ8OfE1E/iNt21DuDh5xojU1JGIxupLTC1lQ8o/TExp0KYYvaxo1vbWkPeLA2cA0Lwm99ik1AGcAd7ozh188+DKObOlTDZWOGmXLVvjKbp41xleqC7u9F7kd516njLzWlERVY6p6JfBH4Fkg09TlJk2vmcKtpmSGE2u+MwOrwJm+KCOvNaWfJl+o6mIReQO4KouCjVip+e+Sw8IbG2F/T6sDm6GyaYaM8ZcTI5K/VMLABCBjfxJkXuRvtKo2An8QkbFpH20CbsiiqCNW+vIVgBOUZs0KsEQjh4WjPLOAb5JEpqO6CUifhi4GbMeZziijTDWl+92Dr6T3HYgKzPBe2n3Euanqk0AnzmKBXxhgCvRhoazn8hXWp+QfqykZ45cHgWOB+1A9I5sDZFrk71z3eXo2Bx/AMuDrqhoTke/iLLP7tRznUVDC5eWEo9F9899Zn5IZTqxPyThCiHwDOBSR3vPkqX4/8wE8cO9TyrjNK1X9m+6ryq3AYwdYMROR1L1KdHZCe7sFJd/YNEPG+ORCnCHgEWBUH4+MMvUpleGMmhgvIjXs+8sejbOEbi78G7AkR8cqaKlJWW3VWX+p86PA5ImdW5Ok+ibwXURW5WtC1suBLwNTcPqVkt++RuCugRKKyJPA5D4++qaqPuzu802cTrDfDnCcBcACgEgkQl1dXYYiD01zc3Pe8mhVpfndd1mxbBknAP+9dSvb8vzvyZd8nqdcSyQStHd2+FrehoYG4vF40ZyjoTj4vfeYEo/zjyz/rcX0XQpK0Z2jLAOSm1YzPoAvedlvMA+cCV1fACq8pqmoqNB8W758ed6O/dxXv6oPn3WW6uuvq4Lqgw/mLa98y+d5yrW7Tjpd7/ifl/ia56mnnqpHHnmkr3kG5vrrVYfwt1lM36WgDPUcAS2a42t4vh5eJ2T9kYichDNNRCRt+6+yCYQiMg/4KnCqqrZmc4xiFK2udu5TsmUrfCVqfUrG+EokimpHxm198Doh66+Bg4HX2LeKoAJZBSWcpr8osMxt61+hqldkeayiER07llhLC/FduwiD9Sn5yfo98sfOrentBeAYD9t68Tqjw3HAbLcaOGSqekgujlNskvcqxbZtc4KS1ZT8Y9dNY/JPZDLOILhyRI6m++C4Ci+H8BqUVuMMWnh/sGU0+yRndYht304ULCj5xe6hyT87x8ZxFs54galA+j1JTcA3vBzAa1AaD6wVkRdJW9pWVc/zmN7gNN8BxD/4wNlgQckXkvZfY0weqf4S+CUin0b1j9kcYjDLoZshilZXA5DYtcvZUFUVYGlGEJtmKL/s3JreHkHkc/QYHOdlkT+vo++ezrZkZp9kTSmxZ48TkEJeVw4xQ2YXTmP89DCwF+f+1owj7tJ5HX13AvAjYBZQijMVeYuqWvvTIJSOHo2EQtDQYE13vrL+jryzPiXT3VRU52WT0OtP9buAi4C3gHLgMuDubDIcyULhMKVjxqBNTRaU/GTXS2P89jwiH84moef2I1XdAIRVNa6qvwCyioIjXbSmBmlutnuUfCS2HHp+2bk1vZ0CrETkTURWIfIGIqu8JPQ60KFVREqB10TkezhDw61DJAvRmhpCra1WU/KbXTiN8dPZ2Sb0Glg+j9OPdDXQAhwAfDrbTEeyspoaQrZshc+s/S7vrE/JpFN9FydOnO6+bsVjvPE6+u5d92Ub8K1symgc0bFjiXR2WvOdnxSrKeWTnVvTk8h/4MwEdBjwC6AE+A1wcqakXkffbaKPn5uqmtVy6CNZtLqacCyGjhplt3P6RGyRP2P89ingaOAVAFS3IuLpl7jX5rvjgH9xHx8B7sSJemaQojU1lMTjxKPRoIsyotiPeWP6JyLzRORNEdkgIjcNsN+nRURF5LgMh+x01yhSN2Gl17J4CkqquivtsUVVfwh8wmsmZp+yigpCQCzidYyJGTLr7sg/61MqWiISxrnF52xgNnCRiMzuY79RwLVAvYfD/h6RnwHViHwReBL4uZfyeG2+S59uPIRTc7KrahbKS0sB6BKhLOCyjBzWfJdXVg0tdscDG1R1I4CIPACcD6ztsd+3ge8CN2Y8ourtiJyJs0r5YcDNqC7zUhivgWUh+35vxoB3gP/lMa1JEy0pAaDT/pD9ZefbmP7sD7yX9n4zMDd9B7dicoCqPioimYMS4AYhT4Eondeg9Ajdf24qcK67QB+q+v1+0pkeou58dx2JRMAlGTnEJmTNP2u+K3QREXk57f0iVV3kJaGIhHCWoZjvYednUT0FkSa6N5wLoHiYms5rUDoWZ5DDw+7BPwm8iDPtkBmEUve5Ix4fcD9jjMmhmKr2NzhhC849RUlT3W1Jo4A5QJ1bEZkMLBWR81Q1PdCB6inuc9b3vHgNSlOBY1S1CUBEbgEeVdWLs814pAp3OBPmtncMauJcMyRqXUr5ZLXQYvcSMFNEpuMEowuBzyU/VNW9OGvqASAidcANvQKS8+HYAXNS3Z2pMF6D0iSgM+19p7vNDFZjIwBtnZ0ZdjS5JHbhNKZPqhoTkauBJ3Bm7rlPVdeIyK3Ay6q6dBCHW8m+rp4DgT3u62rgn8D0TAfwGpR+BbwoIg+57/8HsHgQBTVJblBqbW0NuCAjhyioVZXyy/qUipqqPgY81mPbzf3sWzvAgZygI/Jz4CGc44I6A/JhAAAXn0lEQVTI2ThxIyOv9yn9P+ALOFFvD/AFVf0vL2lNDxaUAmAXTGN8dkIqIAGo/hU4yUtCz/caqeorJKeMMNlrakJFaNu7N+iSjByq1nyXT3ZuTW9bEfnf7Jv551+BrV4S2vITfmtsJF5WRntDQ9AlGVnsummMny4CJgAPuY+J7raMbFYGvzU2kqioIN7WRqytjUh5edAlGvYk7b8mT6xPyaRzRtldm03SQIOSiFwP3A5MUNWdQZbFN42NaKUzN2HHnj0WlHxhF0xjfCUyAfgqcDikzaimenqmpIEFJRE5APg4zjDBkaOpyVlLSZWOPXuonDLF9yKsvfdemt9zZhVZtdlpRgzt3Q0IiTFjKV23kvJYB6NnzGDN3gTtVWM5NtzAjvUbUseQjjZ+FQoTC5UQiXUQ0n0zVChCQsKIxgm5ASEEBDGHhQJRoOzFZdx/+OEASCTCRa+/HkBphinrUzK9/RZYApwLXAFcAnzgJWGQNaUf4ETShwMsg/8aG5ExY6ChgfY9ewIpwrg5c1izaBGJWIxwZxxBKUnEAOiSMKpKFwl2rVrF7kmzOOS959mViLmBRVKBJhyPEY7Heh1fUELa13b/9ZXnqIMO8r0cxoww41C9F5FrUX0aeBqRl7wkDGSgg4icD2xR1ZH3c7WxEampAZzmuyBMmjuXj955J/G4UpLoSgUkgBKNEyZBAqeWMWv7OkoSsVQDWCjLprBCaUAbffDBnLt0MPcCGk+sT8l01+U+v4/IJxA5Ghh4tgdX3mpKIvIkzhxJPX0T+AZO052X4ywAFgBEIhHq6upyVcQ+NTc35zWPE3fuZJf7S33NSy/xbgDLoj/0VicPv93FV0qqGN/Ve7ojAfaWjSba1U5Z3Jl5IiYhIpp9A1whLB4RR7h69uV86aZH+cU8z2uOZaWhoYF4PJ7372shmP7uuxwIPJ3lvzXff3PDQRGeo/+LyBjgeuBHwGjgOi8J8xaUVPVjfW0XkQ/jTDXxunvvyFTgFRE5XlW39XGcRcAigMrKSq2trc1XkQGoq6sjr3l0dLDf7NnI++9zwLhxHJnnf09famvhG/X1LHtkDwl6B4uYhKmJt5GId6VqOBFNECdEKMueIWeK4GCFUe5a+zNfakrV1dU0NDTk97tUKJ56CiDrf2ve/+aGgaI6R86igTNRfQTYC5w2mOS+N9+p6huqOlFVp6nqNJy1O47pKyANO/E4NDcjY8YQra6mfXfGuQnzYnt9Pc9ccw2ooghdoX2/TbokDKokupza97pJs1Kfh0iQCLy+MzSNb7/NI+edF3QxjBm+VON4vCepL3afkp+am53n0aOJjh1LR0A30O5avZqDzj6bNx553Kn9HHU8iR6j70rc0Xdj9ybYOuukbqPvEjij7xKDGH0XVE0pmaewr0bY9O67AZRkmLM+JdPdc4jchTMCryW11ZkZaECBByW3tjQyNDU5z6NGUVZTQ0dANaXZl14KwIvL64mPruHae37Y775n9bO9qJoTjDF+O8p9vjVtmwKFe5/SiOROxsro0URramh4K9g1EiUeg4h9BcwQ2X1KpifVQfUjpbMrkp96BKWgakpJEo8hFpSMMbkm8pU+tu4FVqL62kBJbUJWP/UMSnv3kghwWfRQIgaRksDyN8OI9SmZ7o7Dmclhf/dxOTAP+DkiXx0ooQUlP6X1KUVrakCVzgCXsAjF44gFJTNU1nxnepsKHIPq9aheDxyLM1P4R4H5AyW0oOSntJpS2Vjn5uYgm/BCCWu+M8bkxUQg/c78LmASqm09tvdiVyQ/9Wi+AwIbFg5uTanEakrGmJz7LVCPSHJu008C9yNSCawdKKEFJT8lg1Ky+Q4Cu4EWIJyIEbLmO2NMrql+G5G/Aie7W65A9WX39b8OlNSCkp+amqCsDEpK9tWUApqUFSCciBMqtaBkhsj6lExfnCD0csb9erA+JT81NsLo0QBEq6uB4IJSV1eMEErImu+MMQXEgpKf0oJSuLSUklGjAltTqbPd6WsMlZQGkr8ZhmxYuMkBC0p+amx0Vp11RaurA6sptbtBKVxiLbjGmMJhQclPTU2pmhLgTMoaWE3JWScpXGo1JTNE1qdkcsiCkp/Smu/ArSkFNPqusyMZlKxPyRhTOCwo+alHUCoLcPmKzlTzndWUTI5Yn5LJAQtKfmpq6t6nVFND++7daAB/zB1uTSliNSVjTAGxoOSnns13NTUkOjuJtbb6XpSuDqemZEHJDJn1KZkcsqDkl44O55HefBfgDbRdHc5y5xEb6GCMKSAWlPySnCG8R00JgplqqLO9HYBI1IKSyRHrUzI5YEHJL2nLViRFkzOFBzDYoTNVU7LmO2NGOhGZJyJvisgGEbmpj8+/IiJrRWSViDwlIgflqywWlPySNkN4Uqr5LoCaUqzTGehQajUlM1TWp1TURCQM3A2cDcwGLhKR2T12exU4TlWPAB4Evpev8lhQ8ksfQSnI5Sti7ui7ktKo73kbYwrK8cAGVd2oqp3AA8D56Tuo6nJVTY7IWoGziF9eWFDyS9qyFUmRykpCJSWB1pRKrKZkcsX6lApZREReTnssSPtsf+C9tPeb3W39uRT4az4KCbZ0hX/6GOggIs69SkGMvut0+pRKy6xPyZgRIKaqxw31ICJyMXAccOrQi9Q3C0p+6aP5DpwmvCCGhMfdmlK0rMz3vM0wY31KxW4LcEDa+6nutm5E5GPAN4FTVXXAJc2HIrDmOxH5koj8t4isEZG8dZoVjH6CUllAk7LG3ZpSSZk13xkzwr0EzBSR6SJSClwILE3fQUSOBn4GnKeqO/JZmEBqSiJyGk5H2pGq2iEiE4Moh6+SQamystvmaHU1zVt6/SjJu3hXJyGgNGoDHUyOWJ9SUVLVmIhcDTwBhIH7VHWNiNwKvKyqS4HbgCrgD+LUjP+pquflozxBNd/9O/CdZBUw35G3ICTnvQt1r5xGA5qUNdYZoxSIWk3JmBFPVR8DHuux7ea01x/zqyxBNd8dCnxEROpF5GkR+ZeAyuGfHvPeJUWrq+lqbCTR1eVrcRJdbp+S1ZTMUFmfksmhvNWURORJYHIfH33TzXcscALwL8DvRWSG9jFdtjt0cQFAJBKhrq4uX0UGoLm5OS95zH77bSojEV7qceyWnTsBWP7YY4THjMl5vv3ZtWMH+wP1L64gksXqs/k6T8NFQ0MD8Xh8RJyjAzdtYgbw9NNPoxH7LuXDSDpHeQtKA1X3ROTfgT+5QehFEUkA44EP+jjOImARQGVlpdbW1uanwK66ujrykkdZGUya1OvY/+zo4NklSzh21iyqDz009/n2Y/3DdSQQTj/jdEKhwVeY83aehonq6moaGhpGxjl67jkATv3oRyGLCX7tu5TZSDpHQTXf/Rk4DUBEDgVKgZ0BlcUf/TXfBTQpa7yri0QonFVAMsaYfAnqinQfMENEVuNMaXFJX013w0p/QSmgSVm1q4t4KOxrnmaYsj4lk0OBjL5z51e6OIi8AzPAQAfwf1JWjcWIh+zeaWNMYbG2G7/0WAo9KRWU/K4pxbpIhK2mZHJomDd2GH9YUPKDar81pVAkQumYMb73KSW6YiSspmRywZrvTA5ZUPJDayskEn0GJQho/rtYFxq2oGSMKSwWlPzQx7IV6coCCUoxa74zxhQcC0p+6GPZinRB1JTUakom16xPyeSABSU/9DNDeJI135miZn1KJocsKPkhU1Byl6/w81YticfQsC3wZ4wpLBaU/JChTylaXU0iFqMr2cznh1gMspinzJh+WfOdyQELSn7I0KdUlpzVwccmPKemZEHJGFNYLCj5wUOfEkC7z0HJakomJ6xPyeSQBSU/eAxKftaUQvEYErE+JWNMYbGg5IemJqdW0s+CesE038WtpmRyy/qUTA5YUPJDcoqhfpo5gpiUNZSwmpIxpvBYUPJDP/PeJUUqKgiXlfk6KWsoHkOyWHHWmF6sT8nkkAUlP2QISuD0K/k5KWs4EbeakjGm4FhQ8kM/y1aki9bU+FtTSsQJlVhQMjlkfUomBywo+cFDTamspsbXPqVwIkbImu+MMQXGgpIfvDbf+TT6Lh6LE9YEEin1JT8zzFmfkskhC0p+aGz01nznU1Dq6OwEIGzNd8aYAmNByQ9NTZmb78aOJdbSQtwNGPnU2e7kESq1oGRyyPqUTA5YUMq3eBxaWjI33yXvVfKhttTR1gFYTckYU3gsKOVbhslYk6LJWR18GOzQ2eEGJaspmVywPiWTQxaU8i3DshVJfk7K2tHhNN9FSm2ggzGmsFhQyjevNSUfJ2XttKBk8sH6lEwOBBKUROQoEVkhIq+JyMsicnwQ5fBFhhnCk/yclLXLHegQtqBkjAFEZJ6IvCkiG0Tkpj4+j4rIEvfzehGZlq+yBHX35PeAb6nqX0XkHPd9bT4z/MGy9Vx35qG+p/USlNbeey9jDz8cCYV4euUGDrsYttfXs2v1amZfemm3/cbNmcOu1av5e0c1V1/5ae657Mt8eP8xTDvnHP78UB0fmTkeCYfZtmIFa444k9OjDTzz9m7GP/MwbV1xqieMY3nJQRwHbF76Zx785Y+omjqVA888s1texnhmfUpFTUTCwN3AmcBm4CURWaqqa9N2uxTYo6qHiMiFwHeBC/JRnqCa7xRIXqXHAFvzneEdT70VSNpU890AfUrj5szhuRtuIFJZyZo332N7fT3PXn894+bM6bXfs9dfj4TDRH/+XdYtXoy89A/effRRnrnmGpbsLEPCYV69/XYqp0wh+vPv0rxlC6MeXkzZ2LGEmhtp2rSJY9fXAVCzcRWde/bQuGFDr7yMMSPG8cAGVd2oqp3AA8D5PfY5H/il+/pB4AyR/PwaEQ2gHVhEZgFPAIITGE9S1XczpausrNSWlpZB5/fCF65j3F/+RGkkcwxOJBKEQt3364wlPKXtS1VbM+Mbd3LxN3/H9rH79bvf5K1vctbjd5IgRAilpaKaWEnv9ZciXR1UtjbQESqlLN5Oe7iMaNwZTddaUkFFrI32aCVlHS2pfTpDJZQmuogRIkICcE68AiXl5Zx6991Mmjt3UP+uuro6amtrB5VmJKmtraWhoYHXXnst6KLk38KFcMMNcNhhEA4POnlLSwuVlZV5KNjw0dLSQuVvfgOnnJJVehFpVdU+T7KIfAaYp6qXue8/D8xV1avT9lnt7rPZff+2u8/OrAo0gLw134nIk8DkPj76JnAGcJ2q/lFEPgvcC3ysn+MsABYARCIR6urqPJfhobc6efjtLi7YFuej4w9IbS+PQHmk7yCvCUVCQltMaYv1/nygtOlS6athx8wTea6rEt3RzH6Vwv5V3QPcluYE/4jsz0ETZzF7+1q2V01kx6hJlEWE8kj6MaE9pkyUCJOad9BcUkFVVyvbqyYCMKl5RyrtxKbt3fZJPscRwjg/RAT429ST+OpDOzl/1d/41EzvfUzNzc2D+n8x0jQ0NBCPx0fEOaqYMIGDTj/dWTgyC7GaGlpswckBxWpqWLduHc2xPi5K3kRE5OW094tUdVEOipZzQdWU9gLVqqpuFXCvqg48EoDsa0oA0256lHe+84mM+/VVA/Cadih5J5vs/jr+GM7e+QqnLFzYZ+0lud8LFdM4ZutrvDLlaObuXQ/AE/vN5eydrzD7sstYe889qX3WjzuEQ3dtYEf5OCa27iT9/7jVlPJjRNWUhsi+S5kN9RxlqCmdCNyiqme5778OoKr/lbbPE+4+L4hIBNgGTNA8BJCg+pS2Aqe6r08HhtBpU/ySgeaUhQt58kNnccrChTx7/fVsr6/vc7/Zl13GYR+s5+gbbmDW9rWo+wt14/iDmX3ZZbx6++0c8PGPc9gH6znks59l5s63GD1jBhNa99W03605EIBYWxtPX3VVr7yMMSPGS8BMEZkuIqXAhcDSHvssBS5xX38G+Hs+AhIEN/rui8AdbsRtx22ey6drz5gZSFov6XetXp2qGV3buJ5Jcw/llIUL2bV6dbcaTHK/XatX0/HFrzFr/qd57tnXOMgdfbf7oTo0XsXRN9zAthUr6Pji16iKNtB0/nyizzxMomo01RPG8dexR1JTFqZVZ1Gz7kWqpk7tlZcxZmRQ1ZiIXI3Tzx8G7lPVNSJyK/Cyqi7F6WL5tYhsAHbjBK68CKT5LltDab7zypoSvLHzNDBrvvPOvkuZ5bP5rtDYjA7GGGMKhgUlY4wxBcOCkjHGmIJhQckYY0zBsKBkjDGmYBTV6DsRSQBtec4mAmR92/QIYucpMztH3th5ymyo56hcVYuiElJUQckPIvKyqh4XdDkKnZ2nzOwceWPnKbORdI6KInIaY4wZGSwoGWOMKRgWlHoryJlzC5Cdp8zsHHlj5ymzEXOOrE/JGGNMwbCakjHGmIJhQcklIv9LRNaISEJEjuvx2ddFZIOIvCkiZwVVxkIjIreIyBYRec19nBN0mQqFiMxzvy8bROSmoMtTiETkHRF5w/3uvJw5xcggIveJyA53tdfktrEiskxE3nKfa4IsYz5ZUNpnNfA/gWfSN4rIbJxp2g8H5gE/FpHBr/k8fP1AVY9yH48FXZhC4H4/7gbOBmYDF7nfI9Pbae53Z0QMd/ZoMc61Jt1NwFOqOhN4yn0/LFlQcqnqOlV9s4+PzgceUNUOVd0EbACO97d0psgcD2xQ1Y2q2gk8gPM9MiYjVX0GZ82idOcDv3Rf/xL4H74WykcWlDLbH3gv7f1md5txXC0iq9wmh2HbpDBI9p3xRoG/ichKEcn7Qp9FbpKqvu++3gZMCrIw+RTUyrOBEJEngcl9fPRNVX3Y7/IUg4HOGfAT4Ns4F5dvAwuBf/OvdKbInaKqW0RkIrBMRP7brSWYAaiqisiwHTY9ooKSqn4si2RbgAPS3k91t40IXs+ZiPwceCTPxSkWI/o745WqbnGfd4jIQzjNnhaU+rZdRPZT1fdFZD9gR9AFyhdrvstsKXChiERFZDowE3gx4DIVBPePI+lTOINFDLwEzBSR6SJSijNQZmnAZSooIlIpIqOSr4GPY9+fgSwFLnFfXwIM25adEVVTGoiIfAr4ETABeFREXlPVs1R1jYj8HliLM0vvVaoaD7KsBeR7InIUTvPdO8DlwRanMKhqTESuBp4AwsB9qrom4GIVmknAQyICznXoflV9PNgiFQYR+R1QC4wXkc3AfwDfAX4vIpcC7wKfDa6E+WUzOhhjjCkY1nxnjDGmYFhQMsYYUzAsKBljjCkYFpSMMcYUDAtKxhhjCoYFJTMsiMg1IrJORH4rIvNF5K4sj1MrIielvb9CRP6/3JV0UGXJmLeIHGWzs5vhxO5TMsPFlcDHVHWziMwfwnFqgWbgeQBV/enQi5Ydj3kfBRwH2AztZliwmpIpeiLyU2AG8FcRua7HZ9NE5O/upLFPiciB7vZPiki9iLwqIk+KyCQRmQZcAVznrvHzEXfNqBvcNHUi8l0ReVFE1ovIR9ztFSLyexFZKyIPucfttRSDu37Q99w1hF4UkUMylHHAvN3ZIm4FLnDLe0F+zrAx/rGgZIqeql4BbMVZm+cHPT7+EfBLVT0C+C1wp7v9WeAEVT0aZ2mJr6rqO8BP2bdG1D/6yC6iqscDX8a50x6cWtoeVZ0N/B/g2AGKu1dVPwzcBfwwQxkHzNtdFuNmYIlb3iUD5GtMUbCgZIa7E4H73de/Bk5xX08FnhCRN4AbcRZx9OJP7vNKYJr7+hScwIaqrgZWDZD+d2nPJ2Yoo5e8jRlWLCiZkepHwF1ureVyoMxjug73OU52fbLaz2s/8jam4FlQMsPd8zizdAP8K5BskhvDvuUkLknbvwkYNcg8nsOdINNd9vzDA+x7QdrzCxnK6EU25TWmYFlQMsPdl4AviMgq4PPAte72W4A/iMhKYGfa/n8BPpUc6OAxjx8DE0RkLfB/gTXA3n72rXHLci2QHJTRXxm9WA7MtoEOZriwWcKNGSIRCQMlqtouIgcDTwKHuQMR0vd7BzhOVXf2cRhjDNYubUwuVADLRaQEEODKngHJGOON1ZSMMcYUDOtTMsYYUzAsKBljjCkYFpSMMcYUDAtKxhhjCoYFJWOMMQXDgpIxxpiC8f8Dzzpqe0+xv+4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots()\n",
    "\n",
    "ax.plot(x.reshape(-1), x_quant.reshape(-1), '-o', marker='+',)\n",
    "ax.set_xlabel(\"floating point\")\n",
    "ax.set_ylabel(\"quantized\")\n",
    "ax.axhline(y=0, color='k')\n",
    "ax.axvline(x=0, color='k')\n",
    "ax.grid(True)\n",
    "\n",
    "ax2=ax.twinx()\n",
    "ax2.plot(x.reshape(-1), np.array(gradients_func).reshape(-1), color='red')\n",
    "ax2.set_ylabel('gradient function', color='r')\n",
    "\n",
    "ax.plot(x.reshape(-1), np.array(gradients[0]).reshape(-1), color='brown', marker='x')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Writing Custom Gradient Function\n",
    "\n",
    "Sometimes researchers need to write some custom functions for experimentation. Sometimes you'll need to write custom gradient functions too, or override the default functions, say you want to create a leaky version that multiply the gradients by 0.9. Is that possible? Of course. I'll show you how to use function decorator @tf.custom_gradient to set the function gradient to your Python function. Now let's try to create our own fake quantization function.\n",
    "\n",
    "Here I'll use two TF APIs __tf.quantize__ and __tf.dequantize__ which look like are for internal experimentation as you can only use the defined data type e.g. tf.qint8 rather than any arbitrary bitwidth. Anyway, this is enough to serve our purpose for now. Without too much work, we could translate the quantize-dequantize function we wrote in Tutorial 2, which I'll leave as exercise. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEKCAYAAAASByJ7AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAHXtJREFUeJzt3X94XNV95/H31wJbwTgRDlRkBYklYpOHrgMJCgZMNuN1aV0goWm3VdJtGmC7hmZJDU/T1K6frsO2LFraoLimLKIp+dEmhSS7LAl2DEJl0oaIHzJRIfywTGyn2EApgQnIFBvEd/+YO2Ykzb1z9ePOvTPzeT2Pn9Hce+7cL4exvj7n3HOOuTsiIiLz0g5ARESyQQlBREQAJQQREQkoIYiICKCEICIiASUEEREBlBBERCSghCAiIoASgoiIBI5IO4DpOPbYY33JkiWJ3uPAgQMsXLgw0Xs0AtVTtJ07dzI+Ps4pp5ySdiiZp+9SdbOtox07djzv7sdVK1dXCWHJkiUMDw8neo98Pk8ul0v0Ho1A9RQtl8tRKBQS/742An2XqpttHZnZT+KUU5eRiIgASggiIhJQQhAREUAJQUREAkoIIiICpJwQzKzNzL5lZk+Y2eNmdlaa8YhI/egbGD38p9zyTdsnvPYNjLKyd3DCdQA9/UMTXkvH+wZG6ekfmvK5pXuVypeUyk4uP7nM5PtP10yvm460Wwibge3u/h7gVODxlOMRkTqxeXDX4T/lXj44PuF18+Au9hdenXAdwP17XpjwWjq+eXAX9+95Ycrnlu5VKl9SKju5/OQyk+8/XTO9bjpSm4dgZm8D/gNwEYC7HwIOpRWPiNSPX7vh3gnvr7trZ8X35ccr/Rz2Gva5Mz0eFUsce39Sm1+NltaeymZ2GnAT8BjF1sEOYJ27H5hUbi2wFqC9vf30W265JdG4xsbGOProoxO9RyNQPUW74oorGB8fZ8uWLWmHknnT+S7dtusQt//4tYQjqi2LOBf22/nCk47ko0vnx77HqlWrdrh7d9VYUkwI3cB9wEp3v9/MNgMvufsfh13T3d3tmqmcDaqnaKWZyiMjI2mHknnT+S6dfc0gT//s1SnHV3Qu5tZLi0OQS9ZvZW/v+RNeAfb2nh96vlL58mtK103neLnSZ0/+Oa58Ps9F2w9M+7oSM4uVENIcQ9gH7HP3+4P33wLen2I8IpJhv3HjDyomA+BwMpDZSW0Mwd2fNbOnzOxkd98JrKbYfSQiMsHyTdsPDxJP1tHWOuH9ogUtE17XrV7Kt3Y8dfj8utVLgWKrovy1dHzd6qXct/unnNn19gmfWzp/3+6fTji+onPxlLKTle5R/jnTNdPrpiO1LiM4PI7wRWA+sBu42N1fDCuvLqPsUD1FU5dRfNW+Sz39Q1Oe7IFiIrh3/eoEI8uOOVjcLlaXUaqrnbr7CFA1SBFpXpWSAdA0yaCW6mr5axFpHmEtA5jaTSRzQwlBRDKnb2C0YjIof5pI5l7aM5VFRKZ49qXi00TvOX7RhONKBslSQhCRTOnp/wG3Plh8Kuirl5wBFFsGtXjKptmpy0hEMqN8khfAGf/zzUXprjx3Wa3DaTpKCCKSCeUrkpZbt3qpkkGNqMtIRFLX0z80YUXSkhWdi5UMakgtBBFJ1cXbD+AcqHhOg8i1pRaCiKRmZe9gxRU9O9paZ7yQm8ycEoKIpKJvYDS0m0izkNOhhCAiqQjbAazaQnGSHI0hiEhN9Q2MhiYDDSKnSwlBRGomKhno8dL0qctIRGrmm8NPVTyuZJANaiGISOL6Bka5If8kr41PfaboCNMs5KxQQhCRxIV1E3W0tXL1mS01jkbCqMtIRBK1bOO2isebacezeqEWgogkImoAWbJJCUFEEjF5M/qSRQtaeOSqNTWORuJQl5GIzLmo7S8vOaerxtFIXEoIIjLnHtxbORnMbzE9UZRh6jISkTmzsnew4vpEoK6ieqAWgojMmbBksKJzsZJBHVBCEJE50bVha8Xj80z7GtSL1LuMzKwFGAb2u/sFaccjItMT1U0EsPsa7WtQL7LQQlgHPJ52ECIyM1HJQJvc1JdUE4KZnQCcD3wxzThEZGaiuomUDOpP2l1GXwA+CywKK2Bma4G1AO3t7eTz+UQDGhsbS/wejUD1FK1QKDA+Pt7QdfQ7dx7gjUr7XwIf7joy9n+7vkvV1aqOUksIZnYB8Jy77zCzXFg5d78JuAmgu7vbc7nQonMin8+T9D0ageopWltbG4VCoWHrqGvD1tBksKJzMZv/a/xBZH2XqqtVHaXZQlgJfMTMzgNagbea2d+6+2+lGJOIVBGVDNRNVN9SG0Nw9w3ufoK7LwE+Bvy9koFItvX0D4Umg4621toGI3MuC08ZiUgdiFqfaH6LaSnrBpD2oDIA7p4H8imHISIhopLBis7FmnjWINRCEJGqoharUzJoHEoIIhKpb2A0dNzgd3Pvrm0wkqhMdBmJSDYtWV954hnAutVLtZR1g1FCEJGKVvYOhp7T46WNSV1GIjJF1IJ1KzoX1zgaqRUlBBGZoKd/KDIZaBC5canLSEQO6xsYZe/zByqe05hB41NCEBEgeq5BR1urkkETUJeRiADwzy9Ubhl0tLVqFnKTUAtBpMn1DYyyeXBX6PkTjjmqhtFImpQQRJpcVDLQ46XNRV1GIk1s+abtoeeUDJqPWggiTWrZxm0cGq+8JsX8FqtxNJIFSggiTUib3Egl6jISaTLLNm4LTQbrVi+tbTCSKUoIIk2kb2A0tJtonqG5Bk1OXUYiTSJqzGCewe5r1FXU7NRCEGkCPf1Doclg0YIWJQMBlBBEmkLYkhQAj1y1poaRSJapy0ikgVWbhbxoQUsNo5GsU0IQaWCahSzToS4jkQa1bOO20HPa5EYqUQtBpAFF7YUMaJMbqSi1hGBmJwJfBdoBB25y981pxSPSKLQXssxUmi2E14Hfd/eHzGwRsMPMBtz9sRRjEqlrfQOjodtfztPyRFJFamMI7v6Muz8U/Pwy8DjQkVY8Io0gahBZcw2kmkyMIZjZEuB9wP3pRiJSn6JmIYO6iiSe1BOCmR0N/B/gCnd/qcL5tcBagPb2dvL5fKLxjI2NJX6PRqB6ilYoFBgfH69ZHYUlAwO+tGZhpv9f6btUXa3qyNzD/1WR+M3NjgTuAO509+uqle/u7vbh4eFEY8rn8+RyuUTv0QhUT9FyuRyFQoGRkZHE7xX1RFE9tAz0XaputnVkZjvcvbtauTSfMjLgr4HH4yQDEZlo+abtvHxwPPS85hrIdKXZZbQS+ATwiJmV/hn1R+4ePptGRA6LSgb10DKQ7EktIbj79yl2cYrINHVtqO9uIskmLV0hUmd6+odCdzzraGutbTDSUFJ/ykhE4osaQF60oIV716+uYTTSaNRCEKkTUUtSdLS1al8DmTUlBJE6EbUkhVoGMhfUZSSScdVmIWtJCpkraiGIZFjfwKiWpJCaiWwhmNl3KC5NXZG7f2TOIxKRw7TjmdRStS6jPw9efxU4Hvjb4P3HgX9JKigRid7xTMlAkhCZENz9ewBm9vlJ62B8x8ySXVRIpIlFPV6qJSkkKXHHEBaaWVfpjZl1AguTCUmkuUXNQl63eqm2v5TExH3K6Eogb2a7KS438S7g0sSiEmlSUbOQ57cYV567rLYBSVOJlRDcfbuZLQXeExx6wt0PJheWSPPp2rA1NBkAjF59Xu2CkaYUq8vIzI4C/gC43N3/CXinmV2QaGQiTWRl72Bky0CDyFILcccQvgQcAkqdl/uBP00kIpEm0zcwGjoLGdQykNqJmxBOcvdrgdcA3P0VtHS1yJzQXAPJiriDyofM7C0Ek9TM7CRAYwgis9A3MBqZDPR4qdRa3ITwOWA7cKKZfY3ibmcXJRSTSFNQy0CyJu5TRneZ2Q7gTIpdRevc/flEIxNpYJp4JlkU9ymjQWCFu2919zvc/Xkzuynh2EQaUlQyADTxTFITt8uoE/hDM/uAu18VHOuOukBEpopKBuomkrTFfcqoAKwG2s3sO2b2tgRjEmlIUTueiWRB3BaCufvrwKfM7CLg+8AxiUUl0mCqbXKj1oFkQdwWwo2lH9z9yxSfMLorgXhEGs7K3sHQZLBoQYuSgWRGtQ1y3uruLwHfNLPyRx/2AJ9JNDKRBhE1C/mRq9bUMBKRaNW6jL4OXADsoDgprXx2sgNdlS6Ky8zWAJuBFuCL7t47m88TyRoNIks9qbZBzgXBa+dc39jMWoC/BM4F9gEPmtm33f2xub6XSK3te/Hf2PfiOMeHnF+3emlN4xGJI9agspkNuvvqasem6QzgSXffHXzeLcCFgBKC1L19L74Sek4tA8mqamMIrcBRwLFmdgxvdhm9FeiY5b07gKfK3u8DVkRdsHPnTnK53CxvG61QKNDW1pboPRqB6incA3te4NBzuwF49uvrJ5xbcMQ8cvf9WRphZZa+S9XVqo6qtRAuBa4A/h3FcYRSQngJuD7BuA4zs7XAWoAjjzySQqGQ6P3Gx8cTv0cjUD1V9sQL46HnjjrCeOdbTfU2ib5L1dWqjqqNIWwGNpvZp919yxzfez9wYtn7E4Jjk2O4CbgJoLu724eHh+c4jIny+XzirZBGoHqaasn6rYfHDEotg+N/s/icxLrVS7X9ZQh9l6qbbR2ZxdutIO7idlvM7GxgSfk17v7VmQQXeBBYamadFBPBx4DfnMXniaSmp38o8rySgdSDuIPKfwOcBIwApTaxAzNOCO7+upldDtxJ8bHTm9390Zl+nkia7t/zQsXj8wx2X6NBZKkPcZeu6AZOcfeILcCnz923Advm8jNFamn5pu28fDB83EDJQOpJ3KUrfgShj1SLNKW+gdHIZPCexS01jEZk9uK2EI4FHjOzByjbOtPdP5JIVCJ1IGrHszO73q4nZ6TuTGcLTREJRC1JsWiBWgZSn+I+ZfS9pAMRqRdx1ifK5bQsl9SfuFtonmlmD5rZmJkdMrNxM3sp6eBEsqZrg/ZClsYVd1D5euDjwC7gLcDvUFyYTqRp9A2M8kbEc3baC1nqXdwxBNz9STNrcfdx4Etm9kNgQ3KhiWTHyt7ByH0NtGCdNIK4CeEVM5sPjJjZtcAzxG9diNS1nv6h0GSgiWfSSOL+Uv8ExdnElwMHKK5B9GtJBSWSJWGzkEHJQBpL3KeMfhL8+G/AVcmFI5IdfQOjkXMNtMmNNJq4axntobh20QTuPqstNEWyLCoZaMxAGtF01jIqaQV+HdAzdtKwtBeyNKNYYwju/tOyP/vd/QuA/lZIQ4qaa9DR1lrDSERqK26X0fvL3s6j2GKI/ciqSL2IahnMbzHuXT+bbcRFsi3uL/XP8+YYwuvAXordRiINY9nG8JXYteOZNIO4CeEOigmhtA+bAxeUtmVz9+vmPjSR2unpH+LQeOVpyIsWtCgZSFOImxBOBz4A3E4xKXwYeIDiUhYida1rw9bIJSkeuWpN7YIRSVHchHAC8H53fxnAzD4HbHX330oqMJFaWL5pe2gymN9ijF59Xm0DEklR3JnK7cChsveHgmMidS1sx7N5hpKBNJ24LYSvAg+Y2W3B+18BvpxIRCI10NM/FLkkxaf/o2YhS/OJu3TF1Wb2XeCDwaGL3f2HyYUlkqyoZKCJZ9KsprP89UPAQwnGIlITUY+XKhlIM9MS1tJUlm3cFvp4qWYhS7PTbGNpGlGzkBctaNEsZGl6qbQQzOzPzOwJM3vYzG4zs7Y04pDmsbJ3MPTcogUtmmsgQnpdRgPAv3f39wKjaCtOSVDUjmegiWciJal0Gbn7XWVv7wP+UxpxSOPTXsgi8WVhUPkS4LtpByGNp29gVMlAZBoSayGY2d3A8RVObXT324MyGymunvq1iM9ZC6wFaG9vJ5/Pz32wZcbGxhK/RyOoh3raPHgg9NyX1yxMNP5CocD4+Hjm6ygL6uG7lLZa1ZG5R6zqleSNzS4CLgVWu/srca7p7u724eHhROPK5/PkcrlE79EIsl5PSzdu47WQx0tr0TLI5XIUCgVGRkYSv1e9y/p3KQtmW0dmtsPdu6uVS2UMwczWAJ8FPhQ3GYjEFfV46YpO7fwqEiateQjXAwuAgWBPhfvc/bKUYpEGor2QRWYuraeM3p3GfaWx9fQPhZ5Ty0CkOs1UloYQtSTFogUt3HrpWTWOSKT+KCFI3esbGA1NBtoLWSS+LMxDEJmxvoFRNg9W3sm1o61VyUBkGpQQpK7dkH+y4nEtVicyfeoykroV9kSRFqsTmRm1EKQuhSWDeabF6kRmSglB6k7UjmfaC1lk5pQQpK4s37Q99ImiFZ2LNYgsMgsaQ5C60bVhK2+ELL2lWcgis6cWgtSF5Zu2hyaDdavVTSQyF5QQpC68fHC84vH5LaZuIpE5oi4jybSoJSnmtxijV59X44hEGpdaCJJZPf1Docmgo61VyUBkjikhSGbdv+eF0HOahSwy99RlJJnT0z8UmQw0iCySDCUEyZyoZKDHS0WSoy4jyZTlm7aHnlPLQCRZSgiSGcs3bdfjpSIpUpeRZELUXshavVSkNpQQJHUrewdDz2nHM5HaUZeRpKqnf4j9hVdDzysZiNSOWgiSmqhuonkGu6/RE0UitaQWgqSip38o9FxHW6uSgUgKlBAkFZqFLJI9qSYEM/t9M3MzOzbNOKR2evqHIruKNNdAJD2pjSGY2YnALwL/nFYMUnthLQONGYikL80WQh/wWSBk2xNpNFGzkLUXskj6UkkIZnYhsN/d/ymN+0vtrewdDJ2F3NHWqsdLRTIgsS4jM7sbOL7CqY3AH1HsLorzOWuBtQDt7e3k8/m5CrGisbGxxO/RCKZTTxdvPxDaDDz5mHlsWNHScHVeKBQYHx9vuP+uJOjvXHW1qiNzr22PjZktBwaBV4JDJwBPA2e4+7NR13Z3d/vw8HCi8eXzeXK5XKL3aARx62ll72DoxLOOttaGfaIol8tRKBQYGRlJO5TM09+56mZbR2a2w927q5Wr+aCyuz8C/FzpvZntBbrd/flaxyLJC/v3RiMnA5F6pZnKkoi+gVE2D+6qeG5F52JuvfSsGkckItWkPjHN3ZeoddB4vjn81IT3e64p7n+sZCCSXWohyJyrNPGsc8M2OtpalQxEMiz1FoI0lrC5Bis6F2vMQCTjlBBkzkTteCYi2acuI5kTXRu28kbIE0Xa5EakPqiFILPW0z+kZCDSAJQQZFb6BkZDF6zTkhQi9UVdRjJjPf1DoclAj5eK1B+1EGTGlAxEGosSgszIZQMHKh5ftKBFyUCkTqnLSKYlakkKgEvO6aphNCIyl5QQZFquPHdZxYSgp4lE6p8SgsQWNYgsIvVPCUFiCXu8dNGCFi45p0utA5EGoEFlqaqnf4gzu95e8dwjV61RMhBpEEoIEqnUMvj4X9035dyKzsUpRCQiSVGXkUT6x13/CsCxRy/g+bGDQHEA+X1HPk0up8dLRRqJEoJUNHkAuZQMoPikUT7/dBphiUiC1GUkFd30292c3L7o8Pu9veezbvVS1q1emmJUIpIktRBkirOvGeTpn7064diS9Vu1JIVIg1MLQSboGxg9nAw2ffgUoDh4vLf3fCUDkQanhCCHlS9LccF738FFZy8BUCIQaRLqMhJg6hpFdzz8DHc8/Awdba0pRiUitaSEIKEL1ml9IpHmoi6jJqdkICIlqSUEM/u0mT1hZo+a2bVpxdHsxg6+PuWYkoFIc0olIZjZKuBC4FR3/3ngz5O+Z9/AaCrXTvf6uGVL5crLT752cpnS68reQXr6h+jpH+Kvv78ndmwi0tjSaiH8LtDr7gcB3P25pG8YtalLktdO9/q4ZUvlystPvnZymdLr/sKr3L/nhYqrl6p1INK80hpUXgZ80MyuBl4FPuPuDyZ1sy3BL8Jzr/te1bIHXnmFhQ9NLRfn2ijTuT5u2VK58vKTr51cJuqzlQxEmpu5ezIfbHY3cHyFUxuBq4F7gN8DPgDcCnR5hWDMbC2wFqC9vf30W265JXYMt+06xO0/fm3K8XcsNDqOrtw4Gn/9dVqOOIL9Y2/wzIGpdRN1bbnpXB+3bFi5uXDyMfPYsOItscuPjY1x9NFHJxJLI7jiiisYHx9ny5YtaYeSefouVTfbOlq1atUOd++uWtDda/4H2A6sKnv/Y+C4atedfvrpPlPv+sM7YpW75557ZnztbO89nbKlcuXlJ187uUz5a/mf6+7aGTu+kkr1JG/60Ic+5KeeemraYdQFfZeqm20dAcMe43dzWmMI/w9YBWBmy4D5wPMpxdLU1E0kIiVpjSHcDNxsZj8CDgGfDLJYYmazSudsV/iczvVxy5bKlZeffO3kMqXXjrZWTjjmKM7seruSgYgcltgYQhK6u7t9eHg40Xvk83lyuVyi92gEqqdouVyOQqHAyMhI2qFknr5L1c22jsws1hiCZiqLiAighCAiIgElBBERAZQQREQkoIQgIiJAnT1lZGb/Cvwk4dsci+ZExKF6qk51FI/qqbrZ1tG73P24aoXqKiHUgpkNx3k8q9mpnqpTHcWjeqquVnWkLiMREQGUEEREJKCEMNVNaQdQJ1RP1amO4lE9VVeTOtIYgoiIAGohiIhIQAkhYGa/bmaPmtkbZtY96dwGM3vSzHaa2S+lFWPWmNnnzGy/mY0Ef85LO6asMLM1wfflSTNbn3Y8WWRme83skeC7k+yqlXXEzG42s+eC1aBLxxab2YCZ7Qpej0ni3koIb/oR8KvAP5QfNLNTgI8BPw+sAW4ws5bah5dZfe5+WvBnW9rBZEHw/fhL4JeBU4CPB98jmWpV8N3RY6dv+jLF3zXl1gOD7r4UGAzezzklhIC7P+7uOyucuhC4xd0Puvse4EngjNpGJ3XmDOBJd9/t7oeAWyh+j0Sqcvd/AF6YdPhC4CvBz18BfiWJeyshVNcBPFX2fl9wTIouN7OHg2ZuIs3YOqTvTDwO3GVmO4K90yVcu7s/E/z8LNCexE3S2jEtFWZ2N3B8hVMb3f32WsdTD6LqDPjfwJ9Q/Iv9J8DngUtqF53UuXPcfb+Z/RwwYGZPBP86lgju7maWyOOhTZUQ3P0XZnDZfuDEsvcnBMeaQtw6M7O/Au5IOJx60dTfmbjcfX/w+pyZ3Uaxq00JobJ/MbN3uPszZvYO4LkkbqIuo+q+DXzMzBaYWSewFHgg5ZgyIfhilnyU4sC8wIPAUjPrNLP5FB9K+HbKMWWKmS00s0Wln4FfRN+fKN8GPhn8/EkgkR6NpmohRDGzjwJbgOOArWY24u6/5O6Pmtk3gMeA14H/5u7jacaaIdea2WkUu4z2ApemG042uPvrZnY5cCfQAtzs7o+mHFbWtAO3mRkUfw993d23pxtSNpjZ3wE54Fgz2wdsAnqBb5jZf6G44vNvJHJvzVQWERFQl5GIiASUEEREBFBCEBGRgBKCiIgASggiIhJQQpCGYGa/Z2aPm9nXzOwiM7t+hp+TM7Ozy95fZma/PXeRTiuWqvc2s9O0yqzMFc1DkEbxKeAX3H2fmV00i8/JAWPADwDc/cbZhzYzMe99GtANaKVZmTW1EKTumdmNQBfwXTO7ctK5JWb298ECfINm9s7g+IfN7H4z+6GZ3W1m7Wa2BLgMuDJYo/+DwZ4PnwmuyZvZ/zKzB8xs1Mw+GBw/ysy+YWaPmdltwedOWc45WP//2mAPgAfM7N1VYoy8dzAL+n8APUG8PcnUsDQLJQSpe+5+GfA0xbX1+yad3gJ8xd3fC3wN+Ivg+PeBM939fRSXp/6su+8FbuTNPR7+scLtjnD3M4ArKM4ghWLr5EV3PwX4Y+D0iHB/5u7LgeuBL1SJMfLewdLa/x24NYj31oj7ilSlhCCN7izg68HPfwOcE/x8AnCnmT0C/AHFDZDi+L/B6w5gSfDzORSTCu7+I+DhiOv/ruz1rCoxxrm3yJxRQpBmtQW4PvjX+qVAa8zrDgav48xsDM5Dfq7FvUUiKSFIo/sBxdVGAf4zUOoGehtvLkn9ybLyLwOLpnmPewkWGwu2ylweUban7HWoSoxxzCRekYqUEKTRfRq42MweBj4BrAuOfw74ppntAJ4vK/8d4KOlQeWY97gBOM7MHgP+FHgU+FlI2WOCWNYBpQHwsBjjuAc4RYPKMhe02qnILJlZC3Cku79qZicBdwMnB4O+5eX2At3u/nyFjxFJnfohRWbvKOAeMzsSMOBTk5OBSD1QC0FERACNIYiISEAJQUREACUEEREJKCGIiAighCAiIgElBBERAeD/A4dYPqVcg7V4AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "tf.reset_default_graph()\n",
    "\n",
    "def my_fake_quant_with_min_max_vars(inputs, min, max, num_bits=8):\n",
    "\n",
    "    quantized, quant_min, quant_max = tf.quantize(inputs, min, max, tf.quint8)\n",
    "    cast_quantized = tf.cast(quantized, tf.quint8)\n",
    "    dequantized = tf.dequantize(quantized, quant_min, quant_max)\n",
    "    \n",
    "    return dequantized\n",
    "\n",
    "\n",
    "my_input = tf.placeholder(tf.float32)\n",
    "my_output = my_fake_quant_with_min_max_vars(my_input, min=-6.0, max=6.0, num_bits=8)\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    x_quant = sess.run(my_output, feed_dict={my_input:x})\n",
    "    \n",
    "fig, ax = plt.subplots()\n",
    "\n",
    "ax.plot(x.reshape(-1), x_quant.reshape(-1), '-o', marker='+',)\n",
    "ax.set_xlabel(\"floating point\")\n",
    "ax.set_ylabel(\"quantized\")\n",
    "ax.axhline(y=0, color='k')\n",
    "ax.axvline(x=0, color='k')\n",
    "ax.grid(True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we try run this line to read the gradient function, we'll be getting error because it is not defined\n",
    "```\n",
    "tf.gradients(my_output, my_input)\n",
    "```\n",
    "Now let's first go through how to use @tf.custom_gradient to add gradient to a function with multiply a tensor by itself. Therefore, the gradient of $x^2$ should therefore be $2x$ as confirmed below.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[6.0]\n"
     ]
    }
   ],
   "source": [
    "def my_function(x):\n",
    "    return x*x\n",
    "\n",
    "a = tf.constant(3.0)\n",
    "c = my_function(a)\n",
    "grad = tf.gradients(c, a)\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    print(sess.run(grad))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's add the decorater @tf.custom_gradient and now the function is expected to return two things - the output tensor, and the gradient function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1.0]\n"
     ]
    }
   ],
   "source": [
    "@tf.custom_gradient\n",
    "def my_function_grad(x):\n",
    "    y = x*x \n",
    "    \n",
    "    def grad(dy):\n",
    "        return tf.identity(dy)\n",
    "    \n",
    "    return y, grad\n",
    "\n",
    "a = tf.constant(3.0)\n",
    "c = my_function_grad(a)\n",
    "grad = tf.gradients(c, a)\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    print(sess.run(grad))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is simple and elegant but we can't use it straight away to our problem. Although the decorated function can accept multiple tensors as input it cannot accept non-tensor type like num_bits. To work around this problem, we'll use a powerful Python library to first 'compile' the function with with non-tensors data that can be pre-determined in compile time, into a function with single tensor as input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0, 0.5, 'gradient function')"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAawAAAEKCAYAAABOjWFfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3X+YXVV97/H3JzMkgUkwBBA0QZNo0NIbfxEJGKwTIhoBpehTA1prtDVYhQartYn0guClpFaJNGgNVgR/gnLlSk0MjJHRqgEBDSAgQ0yiJGAtwgBJJCHD9/6x9yQnM+fHPjNzZp8fn9fznOecvfdaZ3/Zz8l8WWuvvZYiAjMzs3o3Ju8AzMzMsnDCMjOzhuCEZWZmDcEJy8zMGoITlpmZNQQnLDMzawhOWGZm1hCcsMzMrCE4YZmZWUNozzuAaowZMyYOPPDAmp7j2WefZcwY5/FKfJ3Ke/rpp4kIav17bQb+LVU23Gu0c+fOiIiGv8gNlbAOPPBAduzYUdNzdHd309nZWdNzNANfp/I6Ozvp7e1lw4YNeYdS9/xbqmy410jSH0cumvw0fMY1M7PW4IRlZmYNwQnLzMwaghOWmZk1BCcsMzNrCLkmLEmTJF0v6VeS7pd0Qp7xmFnjWNHVs/dVaNaFa/d7X9HVw9zl6/arB7Bw1fr93vv3r+jqYeGq9YO+t/9c/eX79ZcdWH5gmYHnr9ZQ6zWTvFtYlwNrI+KlwMuB+3OOx8waxOXrHtz7KvTUrr793i9f9yDbep/erx7AbZsf2++9f//l6x7kts2PDfre/nP1l+/XX3Zg+YFlBp6/WkOt10xyew5L0nOAPwMWAUTEbmB3XvGYWeN42+d+st/2ZTc/UHS7cH+xz6XeS33vUPeXiyWLLb/xn0YARUQ+J5ZeAVwJ3EfSuroTWBIROwaUWwwsBmhvbz+2q6urpnFt376dCRMm1PQczcDXqbzzzjuPvr4+Vq5cmXcoda+a39IND+7mO79+psYRjS6VOVbqr/PpLzqAM2aOzXyOefPm7YyIjqoCq0N5JqzZwK3A3Ii4TdLlwJMR8b9L1eno6AjPdFEffJ3K80wX2VXzW3rNpet4+ImnB+2fM30y152d3AKftnQ1W5afut87wJblp5Y8Xqx8YZ3+etXsL9T/3QM/Z9Xd3c2itTuqrtdPUlMkrDzvYW0FtkbEben29cCrcozHzOrY2z//06LJCtibrKy55XYPKyJ+J+khSS+JiAeA+STdg2Zm+5l14dq9gygGmjJp/H7bE8e17fe+ZP5Mrr/zob3Hl8yfCSStssL3/v1L5s/k1k1/4PgZh+73vf3Hb930h/32z5k+eVDZgfrPUfg91RpqvWaSW5cg7L2P9R/AWGAT8J6IeLxUeXcJ1g9fp/LcJZhdpd/SwlXrB43MgyRR/WTp/BpGVj9GYPLbpugSzHW29ojYAMzOMwYzq2/FkhXQMsnK9mmo5UXMrHWUalnB4G5Aaw1OWGZWd1Z09RRNVoWjAa315D3ThZnZIL97MhkN+NIjJ+6338mqNiQtkPSApI2SlhY5/kJJ6yTdLalb0tQBxw+WtFXSFbWM0wnLzOrKwlU/5brbk1F9X37vcUDSsvIoudqQ1AZ8FngTcAxwlqRjBhT7FPDliHgZcDFw6YDjnwB+VOtYnbDMrG5MW7qa2zbvGyh83D/vm7T2QycfnUdIreA4YGNEbEqnyLsWOH1AmWOAH6Sfbyk8LulY4Ajg5loH6oRlZnWhcEb1Qkvmz3RX4PC1S7qj4LW44NgU4KGC7a3pvkJ3AW9NP58BTJR0qKQxwKeBj9Qq8EIedGFmuVu4av1+M6r3mzN9sltWI2NPRAznEaKPAFdIWkTS9bcN6AM+AKyJiK1SuVkRR4YTlpnl6j1rdxAUnxDALatRsQ04qmB7arpvr4h4mLSFJWkC8LaI6E3XMHytpA8AE4CxkrZHxKCBGyPBCcvMcjN3+bqiM5K30iwWdeB2YKak6SSJ6kzgHYUFJB0GPBYRzwLLgKsAIuKdBWUWAbNrlazA97DMLCcrunpKdgM6WY2eiNgDnAPcRLKI7jcj4l5JF0t6S1qsE3hAUg/JAItL8ojVLSwzy0WpFXQrTSRrIy8i1gBrBuy7oODz9SQrapT7jquBq2sQ3l5OWGY2qlZ09ZRMVh5kYeU4YZnZqCmXrJbMn+lkZWX5HpaZjZpv3fFQ0f1OVpaFW1hmVnMrunr4XPdGnukbPCawXZ7FwrJxwjKzmivVDThl0nguOb5tlKOxRuUuQTOrqaPPX1N0v5+1smq5hWVmNVFugIXZUDhhmVlN3LrpD0X3TxzXxj0XLRjlaKwZuEvQzEZcueXt33vijFGOxpqFE5aZjbjbtxRPVmPb5BGBNmTuEjSzETN3+bqi8wOCuwJt+NzCMrMRUypZzZk+2cnKhs0Jy8xGxIxlq4vuHyOva2UjI/cuQUltwB3Atog4Le94zKw65boBATZdeuooRmPNrB5aWEtI1mAxswZULlltWe5kZSMn14QlaSpwKvAfecZhZkNTrhvQycpGWt5dgp8BPgpMLFVA0mJgMUB7ezvd3d01DWj79u01P0cz8HUqr7e3l76+vqa+Rn9z0w6eLba+PfDmGQdk/m/3b6kyX6NEbglL0mnA7yPiTkmdpcpFxJXAlQAdHR3R2Vmy6Ijo7u6m1udoBr5O5U2aNIne3t6mvUYzlq0umazmTJ/M5e/LPsjCv6XKfI0Sebaw5gJvkXQKMB44WNJXI+Ivc4zJzCool6zcDWi1lNs9rIhYFhFTI2IacCbwAycrs/q2cNX6kslqyqTxoxuMtZx6GCVoZg2g3PyAY9vkpUKs5vIedAFARHQD3TmHYWYllEtWc6ZP9oPBNircwjKzispNZutkZaPFCcvMylrR1VPyvtXfdr54dIOxllYXXYJmVp+mLS3+YDDAkvkzvVSIjSonLDMrau7ydSWPefi65cFdgmY2SLkJbedMnzzK0ZglnLDMbD8LV60vm6w8yMLy4i5BM9trRVcPWx7dUfSY71lZ3pywzAwo/6zVlEnjnawsd+4SNDMAfvtY8ZbVlEnjPYuF1QW3sMxa3IquHi5f92DJ41MPOWgUozErzQnLrMWVS1Yevm71xF2CZi1s1oVrSx5zsrJ644Rl1qKOPn8NT+3qK3psbJtGORrLk6QFkh6QtFHS0iLHXyhpnaS7JXVLmpruf4Wk9ZLuTY8trGWc7hI0a0FehNH6SWoDPgucDGwFbpd0Y0TcV1DsU8CXI+IaSScBlwLvAnYCfxURD0p6PnCnpJsiorcWsbqFZdZijj5/TclktWT+zNENxurBccDGiNgUEbuBa4HTB5Q5BvhB+vmW/uMR0RMRD6afHwZ+Dxxeq0CdsMxayIquHnb3Fc9WY4SftWpNU4CHCra3pvsK3QW8Nf18BjBR0qGFBSQdB4wFfl2jON0laNYqjj5/TdlktelSdwU2sXZJdxRsXxkRV1ZR/yPAFZIWAT8CtgF7b4BKeh7wFeDdEfHsCMRblBOWWQtYuGp9yWQ1cVwb91y0YJQjslG2JyJmlzi2DTiqYHtqum+vtLvvrQCSJgBv679PJelgYDVwfkTcOtKBF3KXoFkLKDXlEuBkZbcDMyVNlzQWOBO4sbCApMMk9eeLZcBV6f6xwA0kAzKur3WgbmGZNbFKs1hMHNc2itFYPYqIPZLOAW4C2oCrIuJeSRcDd0TEjUAncKmkIOkS/GBa/e3AnwGHpt2FAIsiYkPRk0njgLcB0yjMPxEXZ4nVCcusiXkWC8siItYAawbsu6Dg8/XAoBZURHwV+GoVp/oO8ARwJ7Cr2jidsMya1NHnryl5zIswWk6mEjHkPmjfwzJrQtOWri45yALwIoyWl58izRpq5dxaWJKOAr4MHAEEyTDLy/OKx6xZzF2+ruQxdwNazk4EFiFtJukSFBBEvCxL5Ty7BPcAH46In0uaSDKlR9eA6UDMrAorunpKLm8/xtMDWv7eNJzKuXUJRsQjEfHz9PNTwP0MfrrazKpQbpCFHwy23EX8BpgEvDl9TUr3ZVIXgy4kTQNeCdyWbyRmjancLBbgrkCrE9IS4H3At9M9X0W6koiVmapHlP6Rj4b0qekfApdExLeLHF8MLAZob28/tqurq6bxbN++nQkTJtT0HM3A16m88847j76+PlauzPTvcNgWrS2+vL2ALy3oGJUYhsq/pcqGe43mzZu3MyLy/yFIdwMnELEj3e4A1me9h5VrwpJ0APBd4KaIuKxS+Y6Ojtixo/g/zJHS3d1NZ2dnTc/RDHydyuvs7KS3t5cNG4o/PzmSpi1dXfJYI7Ss/FuqbLjXSFK9JKx7gFcT8XS6PR64nYhMIwfzHCUo4IvA/VmSlZntb9aFa0suwAh+1srq0peA25BuSLf/nCQPZJLnPay5JAuA3SOp/39DP5Y+cW1mFZRLVo3QsrIWFHEZUjfJ8HaA9xDxi6zVc0tYEfFjki52M6vSjGWN3Q1oLUY6mIgnkSYDW9JX/7HJRJSenblAXYwSNLPsFq5aX3LF4CmTxo9uMGbZfB04jWQOwcJfr9LtGVm+xAnLrIGUG2AxcVwbP1k6fxSjMcso4rT0ffpwvsZzCZo1iHJTLk2ZNN7rWln9kwb/iIvtK8EtLLMGUW7KJbesrK4lw9cPAg5DOoR94xcOpooZjpywzOpcpVksPOWSNYCzgfOA55Pcx+pPWE8CV2T9Eicsszq2oqvHUy5Z40tW4rgc6dys0zAVUzZhSfpP9h/RMSCGeMtQT2xmlXnFYGsyzyJNIqIXIO0ePIuIz2WpXGnQxaeATwObgT8CX0hf24FfDzViM6us3IrBTlbWoN63N1kBRDxOMhluJmVbWBHxQwBJn46I2QWH/lPSHVUGamYZlRu+7imXrIG1IYn+SWylNmBs1spZ72F1SJoREZuSc2g6kP9EimZNqNwsFkvmz+RDJx89itGYjai1wHVIq9Lts9N9mWRNWB8CuiVtIhnd8cL0RGY2gsrNYjG2TU5W1uj+kSR3/G263QX8R9bKmRJWRKyVNBN4abrrVxGxq5oozay8GctWl0xWAD2XnDJ6wZjVQsSzwL+nr6plmulC0kHAPwDnRMRdwAsknTaUE5rZYHOXryvbsvIgC2sK0lykLqQepE1Im0l67jLJ2iX4JZKHvU5It7cB3yJZfNHMhmFFV0/JWSzALStrKl8kucV0J1B6fZwSsiasF0XEQklnAUTEznQBRjMbJj9rZS3kCSK+N9TKWRPWbkkHkj5ELOlFgO9hmQ3Diq6essnKw9etCd2C9K/AtynMIRE/z1I5a8L6OMnQw6MkfY1kteBF1URpZvtzy8pa0Jz0vfC53gBOylI56yjBmyXdCRxPMqx9SUQ8Wk2UZraPHwy2lhQxbzjVMyUsJeuVfDoiVhfsuzIiFg/n5GatqFyyArju7BPKHjdrWNIFRfdHXJyletYuwenAP0p6dURclO6bXa6CmQ1WLlm5G9BawI6Cz+OB04D7s1bOmrB6gfnAv6UzuP9l5vDMDCi/YrBZS4j49H7b0qeAm7JWz5qwFBF7gA9IWgT8GDgk60nMWl2lRRjdurIWdRAwNWvhrAnr8/0fIuJqSfcAH6wyMLOWNHf5upLJauK4Nu65aMEoR2SWkyR39P9jaAMOBzLdv4LKCzgeHBFPAt+SVDh0aTPwkSpDNWtJ5WaxcLKyliBNJ2IzyT2rfnuA/ybpvcuk0lyCX0/f7wTuSN/vLNgeFkkLJD0gaaOkpcP9PrN640EW1ggq/S2W9EJJ6yTdLalb0tSCY++W9GD6eneJU1yfvl9FxG/S17ZqkhVUXsDxtPR9ejVfmoWShbs+C5wMbAVul3RjRNw30ucyG21bH/8jWx/v48gSx5fMnzmq8ZiVkvFv8aeAL0fENZJOAi4F3pX2vF1IMmo8gDvTuo8POM0YpI8BRyP9/aAgIi7LEmvW2doHDW8qtq9KxwEbI2JTROwGrgVOH+Z3mtWFrY/vLHlsy/JTva6V1ZMsf4uPAX6Qfr6l4Pgbga6IeCxNUl1AsX7uM0kmu20HJhZ5ZVLpHtZ4klEch0k6hGSWC4CDgSlZT1LCFOChgu2t7Ju2o6inn36azs7Oqk90zsaNvHj79kxlJ+zZw4b2rGNRWpevU2lPPr2Htbv/CMCYz7x9v2OS2PB5X7dC/i1VNmHPHq6fNIkrXvziWnx9lr/FdwFvBS4HzgAmSjq0RN3BuSHiAeBfkO6u5eS3ZwPnAc8nuW/Vn7CeBK4Y6kmrIWkxsDj9TG9vb9XfsWvXLvbsydZVGhGZy7YyX6fidu4pPXR9jMT4NnzdBvBvqbKIYNeuXUP6+5dql1Q47uDKiLiyivofAa5IH2v6EckSU1UvDzKcZAWV72FdDlwu6dyIWDmcExWxDTiqYHtqum9gDFcCVwJ0dHTEhg0bRjiM/XV3dw+pFddqfJ0GKxxg8buvJ/etj3zHciC5Z+VuwOL8W6qsu7ubd3Z28s4h1pe0JyJKzU5U8W9xRDxM0sJC0gTgbRHRK2kb0DmgbvcQw6wo6+S3KyW9BphWWCcivjyMc98OzJQ0neTinAm8YxjfZ5abhavWlz3uZGV1rOLfYkmHAY9FssT9MuCq9NBNwD+nt4wA3pAeL04aR8SuivtKyDr57VeAFwEb2NcMDGDICSsi9kg6h+Q/uA24KiLuHer3meXpts2PFd0/RrDpUg9ft/pV6m+xpIuBOyLiRpJW1KWSgqRL8INp3cckfYIk6QFcHBHF/zEk1gOvyrCvqKx3OmcDx0RE6Q76IYiINcCakfxOs9E068K1PLWrdFe+k5U1gmJ/iyPigoLP17PvWaqBda9iX4urOOlIksEYByK9kv0H8B2UNc6sCeuXwJHAI1m/2KzZrejqKZusXjq5bRSjMatrbyRZ9HcqUPjM1VPAx7J+SdaEdRhwn6SfUbCscUS8JeuJzJpNuRWDj59x6HBGdJk1l4hrgGuQ3kbE/x3q12RNWB8f6gnMmlG5KZcmjnPLyqyE7yK9gwED+EZ0AceI+OFQIjNrRlnmB+zsXD5a4Zg1ku8AT5A815tpZGChrKMEjwdWAn8CjCUZSbIjIg6u9oRmjWzGstLJas70ySWPmRkAU4kY8hIFmeYSJJnV4izgQeBA4G9IJks0axkrunp4tsw42evOPmH0gjFrTD9FmjXUypkn8IqIjZLaIqIP+JKkX1DuATGzJjJ3+bqy61p5qRCzTE4EFiFtJukSFBBEvCxL5awJa6ekscAGSZ8kGd6etXVm1tAWrlpfMln5wWCzqrxpOJWzJp13kdy3OgfYQTLv1NuGc2KzRlFqFgtwsjKrSsRvSPLHSennnVTR+Mk6SvA36cc/AhdVG6NZI1rR1VP2WSsvwmhWJal/sceXAF8CDgC+CszNUj3rKMHNJHMH7iciZmQO1KzBlEtWvmdlNiRnAK8Efg5AxMNII7OAY4HCaenHA38BeAyvNa0sz1qZWdV2ExEkk+iC1FFN5Ux9hxHxh4LXtoj4DOB/tdaUyj1rNWXS+FGMxKzpfBNpFTAJ6X3A94EvZK2ctUuwcOr3MSQtLq9pbU2nXMtqbJv4ydL5oxiNWZOJ+BTSySSr1r8EuICIrqzVsyadT7PvHtYeYAtJt6BZ0zj6/NIr3XjFYLMRkiSozEmqUNaE9V2ShNW/hkkAp0lKzx+Xlahn1hAWrlrP7r7i01hMHNfmZGU2HNKPiTgR6Sn2H8DX/+Bwpmn+siasY4FXk0xcKODNwM9Ipmoya2gzlq0uO+XSPRcNeeozMwOIODF9zzwisJisCWsq8KqIeApA0seB1RHxl8M5uVneZl24tmSyGtsmei45ZXQDMmtGUvlR5RGln84vkDVhHQHsLtjene4za2ilVgweI5yszEbOney7rfQC4PH08yTgt8D0LF+SNWF9GfiZpBvS7T8Hrq4iWLO6snDV+rJTLp17kmexMBsxEUlCkr4A3EDEmnT7TST5JJOsUzNdIul7wGvTXe+JiF9UE69ZPSmXrPxgsFnNHE/E+/ZuRXyPZEL1TKpZXuTn9E+nYdbAyg1fd7Iyq6mHkf6JZP5AgHcCD2et7CVCrKUcff6aksPXPYuFWc2dBRwO3JC+npvuy8SzVVjLKDeLxcRxbZ7FwqzWktGAS4ZaPZeEJelfSZ7l2g38muSeWG8esVhrmLt8XcljE8e1+Vkrs9EgHQ58FPhTkonUExEnZameV5dgF/C/IlkWuQdYllMc1gLKrRgMfjDYbBR9DfgVyTD2i0im+bs9a+VcElZE3BwRe9LNW0keTDYbcXOXr/OIQLP6cSgRXwSeIeKHRLwXyNS6gvq4h/Ve4Lq8g7Dms6Krp2zLysnKbNQ9k74/gnQqyQjBzGsrKqLMJGrDIOn7wJFFDp0fEd9Jy5xPslTJW6NEIJIWA4sB2tvbj+3qGtIkv5lt376dCRMm1PQczaARrtOitTtKHrt6QVXrxlXtvPPOo6+vj5UrV9b0PM2gEX5LeRvuNZo3b97OiKjtjz4L6TTgv4CjgJXAwcBFRNyYqXqtElbFE0uLgLOB+RGxM0udjo6O2LGj9B+hkdDd3U1nZ2dNz9EM6v06zTx/Dc+UGL4+Gi2rzs5Oent72bBhQ83P1ejq/bdUD4Z7jSTln7CkNuDviFgx1K/I5R6WpAUkI0XekjVZmWU1benqkslqzvTMvQ9mNpIi+qjimati8rqHdQUwDuhK19S6NSLen1Ms1kTKPWvle1ZmufsJ0hUk4xb2dZclMylVlEvCiogX53Fea24LV60vecwtK7O68Ir0/eKCfUHGkYL1MErQbNjKTbk0cVwb1519wihHZGaDRMwbTnUnLGt4K7p6SiarJfNnenl7s3oh/X2RvU8AdxJRcYSSJ7+1hraiq4fL1z1Y9NiUSeOdrMwykLRA0gOSNkpaWuT4CyTdIukXku6WdEq6/wBJ10i6R9L9kirNWjQbeD8wJX2dDSwAvoD00UpxOmFZQ/tc98ai+z2ZrVk2SoabfxZ4E3AMcJakYwYU+yfgmxHxSuBM4HPp/r8AxkXELOBY4GxJ08qcbirwKiI+TMSH0zrPBf4MWFQpVicsa1ilhq97MluzqhwHbIyITRGxG7gWOH1AmSB5yBfgOexbwyqADkntwIEkE5o/WeZczwV2FWw/AxxBxB8H7C/K97CsIZUavj5GnszWrEpTgIcKtrcCcwaU+Thws6RzgQ7g9en+60mS2yPAQcCHIllCpJSvAbchfSfdfjPwdaQO4L5KgbqFZQ2n3IrB5540cxQjMWsY7ZLuKHgtrrL+WcDVETEVOAX4iqQxJK2zPuD5JDOwf1jSjJLfEvEJkqn2etPX+4m4mIgdRLyz4n9ElUGb5WrWhWtLjgicM32yB1mYFbcnImaXOLaNZG6/flPTfYX+mmRwBBGxXtJ44DDgHcDaiHgG+L2kn5AMrNhUMpKIO4A7hvIf4RaWNYwZy1bz1K6+ose2LD/Vz1qZDc3twExJ0yWNJRlUMXAy2t8C8wEk/QnJ4ov/k+4/Kd3fARxPst5VTThhWUOYdeFani0xT/OS+e4GNBuqdG3Cc4CbgPtJRgPeK+liSW9Ji30YeJ+ku4BvAIvSFTY+C0yQdC9J4vtSRNxdq1jdJWgNoVTLamyb3A1oNkwRsQZYM2DfBQWf7wPmFqm3nWRo+6hwwrK6Vm7KpbFtoueSU0Y5IjPLi7sErW4tXLW+ZLKaMmm8k5VZi3HCsrp12+bSj3N4Fguz1uMuQas7C1etL5usPMjCrDU5YVndKZesvAijWetyl6DVlVkXri15zC0rs9bmhGV1Y9aFaz183cxKcpeg1YVSk9mCZ183s4QTluVu7vJ1JY95xWAz6+cuQcvVwlXr2db7dMnjTlZm1s8tLMtNuW7AMYJNl3pEoJnt4xaW5WLhqvUlj02ZNN7JyswGccKyXHgWCzOrVq4JS9KHJYWkw/KMw0bPwlXry3YF+lkrMyslt3tYko4C3kCyAJi1iFItK9+zMrNK8mxhrQA+CpRYls+aTblZLM49yS0rMysvl4Ql6XRgW0Tclcf5bfTNXb6u5CwWUyaN9/B1M6uoZl2Ckr4PHFnk0PnAx0i6A7N8z2JgMUB7ezvd3d0jFWJR27dvr/k5mkE11+k9a3eUbEa/5JAxLJvT1nTXvLe3l76+vqb776oF/5urzNcooYjR7ZGTNAtYB+xMd00FHgaOi4jflavb0dERO3bsqGl83d3ddHZ21vQczSDrdZq7fF3JB4OnTBrftCMCOzs76e3tZcOGDXmHUvf8b66y4V4jSTsjomPkIsrHqA+6iIh7gOf2b0vaAsyOiEdHOxarvVL/P9TMycrMasMzXVhNrOjq4fJ1DxY9Nmf6ZK47+4RRjsjMGl3uDw5HxDS3rprPt+54aL/tzZeeAjhZmdnQuYVlI67Yg8HTl61hyqTxTlZmNmS5t7CsuZR61mrO9Mm+Z2Vmw+KEZSOm3IrBZmbD5S5BGxEzlq3m2RIjAr0Io5mNBLewbNgWrlrvZGVmNeeEZcOyoqun5IS2nnLJzEaSuwRtyBauWl8yWXn4upmNNLewbMicrMxsNDlh2ZC8v6v4nI4Tx7U5WZlZTbhL0KpSbsolgPeeOGMUozGzVuKEZVX50MlHF01YHg1oZrXmhGWZlRtkYWZWa05Ylkmp4esTx7Xx3hNnuHVlZjXnQRdW0cJV6zl+xqFFj91z0QInK7MGJ2mBpAckbZS0tMjxF0i6RdIvJN0t6ZSCYy+TtF7SvZLukTS+VnG6hWVl9beszvrCrYOOzZk+OYeIzGwkSWoDPgucDGwFbpd0Y0TcV1Dsn4BvRsS/SzoGWANMk9QOfBV4V0TcJelQ4JlaxeqEZWX914P/A8BhE8bx6PZdQDLA4pUHPExnp4evmzWB44CNEbEJQNK1wOlAYcIK4OD083OAh9PPbwDujoi7ACLiD7UM1AnLiho4wKI/WUEyUrC7++Fi1cys8UwBCldc3QrMGVDm48DNks4FOoDXp/uPBkLSTcDhwLUR8claBep7WFbUlX81m5ccMXHv9pblp7Jk/kyWzJ+ZY1RmNkTtku4oeC2usv5ZwNURMRU4BfiKpDEkjZ4TgXem72dIqtnCd25h2SCvuXQdDz8TeZ5HAAAInUlEQVTx9H77pi1d7SmXzBrXnoiYXeLYNuCogu2p6b5Cfw0sAIiI9enAisNIWmM/iohHASStAV4FrBvB2PdyC8v2s6KrZ2+yuvDNxwDJ4Ioty091sjJrTrcDMyVNlzQWOBO4cUCZ3wLzAST9CTAe+B/gJmCWpIPSARivY/97XyPKCcv2Kpx26bSXPY9Fr5kG4ERl1sQiYg9wDknyuZ9kNOC9ki6W9Ja02IeB90m6C/gGsCgSjwOXkSS9DcDPI2J1rWJ1l6ABg+cI/O7dj/Ddux9hyqSaPVJhZnUiItaQDFUv3HdBwef7gLkl6n6VZGh7zTlhWckJbT0/oJnVE3cJtjgnKzNrFLklLEnnSvpVOp1HzcbtW3nbd+0ZtM/JyszqUS4JS9I8kiepXx4Rfwp8qtbnXNHVk0vdautnLdtfrrD8wLoDy/S/z12+joWr1rNw1Xq++OPNmWMzM8tTXi2svwWWR8QugIj4fa1PWG7RwVrWrbZ+1rL95QrLD6w7sEz/+7bep7lt82NFZ19368rM6lVegy6OBl4r6RLgaeAjEXF7rU62Mv1DffJlP6xYdsfOnXT8fHC5LHXLqaZ+1rL95QrLD6w7sEy573ayMrN6poiozRdL3weOLHLofOAS4Bbg74BXA9cBM6JIMOkUIosB2tvbj+3q6socww0P7uY7vx48cfDzOsSUCcUbl3179tDW3s627c/yyI7B16Zc3ULV1M9atlS5kfCSQ8awbM6Bmctv376dCRMm1CSWZnDeeefR19fHypUr8w6l7vm3VNlwr9G8efN2RkTHCIaUj4gY9RewFphXsP1r4PBK9Q466KAYqhf+43czlbvllluGXHe4566mbH+5wvID6w4sU/he+Lrs5gcyx9ev2HWyfV73utfFy1/+8rzDaAj+LVU23GsE7Igc/taP9Cuve1j/D5gHIOloYCzwaE6xtDR3A5pZo8jrHtZVwFWSfgnsBt6d/l9AzQxnlvHhzlBeTf2sZfvLFZYfWHdgmf73KZPGM/WQgzh+xqFOVmbWMGp2D6sWOjo6YseOHTU9R3d3N52dnTU9RzPwdSqvs7OT3t5eNmzYkHcodc+/pcqGe40kNcU9LM90YWZmDcEJy8zMGoITlpmZNQQnLDMzawhOWGZm1hAaapSgpGeBP9b4NO3A4CnMbSBfp8p8jbLxdapsuNfowIho+AZKQyWs0SDpjoiYnXcc9c7XqTJfo2x8nSrzNUo0fMY1M7PW4IRlZmYNwQlrsCvzDqBB+DpV5muUja9TZb5G+B6WmZk1CLewzMysIThhpST9haR7JT0rafaAY8skbZT0gKQ35hVjvZH0cUnbJG1IX6fkHVO9kLQg/b1slLQ073jqkaQtku5Jfzt35B1PvZB0laTfp6tZ9O+bLKlL0oPp+yF5xpgXJ6x9fgm8FfhR4U5JxwBnAn8KLAA+J6lt9MOrWysi4hXpa03ewdSD9PfxWeBNwDHAWenvyAabl/52Wn7IdoGrSf7WFFoKrIuImcC6dLvlOGGlIuL+iHigyKHTgWsjYldEbAY2AseNbnTWYI4DNkbEpojYDVxL8jsyqygifgQ8NmD36cA16edrgD8f1aDqhBNWZVOAhwq2t6b7LHGOpLvTboyW7KYowr+ZbAK4WdKdkhbnHUydOyIiHkk//w44Is9g8pLXisO5kPR94Mgih86PiO+MdjyNoNw1A/4d+ATJH55PAJ8G3jt60VmDOzEitkl6LtAl6Vdp68LKiIiQ1JLDu1sqYUXE64dQbRtwVMH21HRfS8h6zSR9AfhujcNpFC39m8kqIral77+XdANJV6oTVnH/Lel5EfGIpOcBv887oDy4S7CyG4EzJY2TNB2YCfws55jqQvoPp98ZJANXDG4HZkqaLmksyaCdG3OOqa5I6pA0sf8z8Ab8+ynnRuDd6ed3Ay3ZI9RSLaxyJJ0BrAQOB1ZL2hARb4yIeyV9E7iPZLbkD0ZEX56x1pFPSnoFSZfgFuDsfMOpDxGxR9I5wE1AG3BVRNybc1j15gjgBkmQ/B36ekSszTek+iDpG0AncJikrcCFwHLgm5L+GvgN8Pb8IsyPZ7owM7OG4C5BMzNrCE5YZmbWEJywzMysIThhmZlZQ3DCMjOzhuCEZU1B0t9Jul/S1yQtknTFEL+nU9JrCrbfL+mvRi7SqmKpeG5Jr/As+dYq/ByWNYsPAK+PiK2SFg3jezqB7cBPASLi88MPbWgynvsVwGzAM+Vb03MLyxqepM8DM4DvSfrQgGPTJP0gnaB3naQXpPvfLOk2Sb+Q9H1JR0iaBrwf+FC6RtNr0zW/PpLW6Zb0L5J+JqlH0mvT/QdJ+qak+yTdkH7voOUy0vWfPpmuAfUzSS+uEGPZc6ezaFwMLEzjXVibK2xWH5ywrOFFxPuBh0nWVlox4PBK4JqIeBnwNeDf0v0/Bo6PiFeSLP/x0YjYAnyefWt8/VeR07VHxHHAeSQzEEDSuns8Io4B/jdwbJlwn4iIWcAVwGcqxFj23OnSJRcA16XxXlfmvGYNzwnLmt0JwNfTz18BTkw/TwVuknQP8A8kC3Rm8e30/U5gWvr5RJKkR0T8Eri7TP1vFLyfUCHGLOc2axlOWNaqVgJXpK2ds4HxGevtSt/7GNo94CjxeTTObdbQnLCs2f2UZLZ0gHcC/d18z2Hfkh/vLij/FDCxynP8hHQyUknHALPKlF1Y8L6+QoxZDCVes4bkhGXN7lzgPZLuBt4FLEn3fxz4lqQ7gUcLyv8ncEb/oIuM5/gccLik+4D/A9wLPFGi7CFpLEuA/gEipWLM4hbgGA+6sFbg2drNhklSG3BARDwt6UXA94GXpIMiCsttAWZHxKNFvsbMKnA/uNnwHQTcIukAQMAHBiYrMxs+t7DMzKwh+B6WmZk1BCcsMzNrCE5YZmbWEJywzMysIThhmZlZQ3DCMjOzhvD/Ad4Ou+ZSsKczAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from functools import partial\n",
    "\n",
    "tf.reset_default_graph()\n",
    "\n",
    "def my_fake_quant_with_min_max_vars(inputs, min, max, num_bits=8):\n",
    "\n",
    "    def _fake_quant(min, max, num_bits, inputs):\n",
    "        quantized, quant_min, quant_max = tf.quantize(inputs, min, max, tf.quint8)\n",
    "        cast_quantized = tf.cast(quantized, tf.quint8)\n",
    "        dequantized = tf.dequantize(quantized, quant_min, quant_max)\n",
    "\n",
    "        return dequantized\n",
    "    \n",
    "    # create a new function with fixed min, max, num_bits\n",
    "    fake_quant = partial(_fake_quant, min, max, num_bits)\n",
    "    \n",
    "    @tf.custom_gradient\n",
    "    def fake_quant_wrapper(x):\n",
    "        y = fake_quant(x)\n",
    "    \n",
    "        # your custom gradient function here\n",
    "        def grad(dy):\n",
    "            return 0.9*tf.identity(dy)\n",
    "    \n",
    "        return y, grad\n",
    "    \n",
    "    return fake_quant_wrapper(inputs)\n",
    "\n",
    "my_input = tf.placeholder(tf.float32)\n",
    "my_output = my_fake_quant_with_min_max_vars(my_input, -6.0, 6.0, 8)\n",
    "grad_func = tf.gradients(my_output, my_input)\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    [x_quant, gradients_func] = sess.run([my_output, grad_func], feed_dict={my_input:x})\n",
    "\n",
    "fig, ax = plt.subplots()    \n",
    "ax.plot(x.reshape(-1), x_quant.reshape(-1), '-o', marker='+',)\n",
    "ax.set_xlabel(\"floating point\")\n",
    "ax.set_ylabel(\"quantized\")\n",
    "ax.axhline(y=0, color='k')\n",
    "ax.axvline(x=0, color='k')\n",
    "ax.grid(True)\n",
    "\n",
    "\n",
    "ax2=ax.twinx()\n",
    "ax2.plot(x.reshape(-1), np.array(gradients_func).reshape(-1), color='red')\n",
    "ax2.set_ylabel('gradient function', color='r')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What's Next?\n",
    "\n",
    "We have learned everything we need to know for quantized neural network training. Next, we'll apply our new learned knowledge to actually train a MobileNet on Cifar10."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}