3-Optimization.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--NAVIGATION-->\n",
    "# < [Autograd](2-Autograd.ipynb) | Optimization | [Modules](4-Modules.ipynb) >"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Optimization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notebook Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this short notebook, we will see how to use the gradient obtained with Autograd to perform optimization of an objective function.  \n",
    "Then we will also present some off-the-shelf Pytorch optimizers and learning rate schedulers.  \n",
    "As an eye candy, we will finish with some live optimization vizualisations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Google Colab only!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# execute only if you're using Google Colab\n",
    "!wget -q https://raw.githubusercontent.com/ahug/amld-pytorch-workshop/master/binder/requirements.txt -O requirements.txt\n",
    "!pip install -qr requirements.txt\n",
    "!wget -q https://raw.githubusercontent.com/ahug/amld-pytorch-workshop/master/live_plot.py -O live_plot.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib notebook\n",
    "torch.set_printoptions(precision=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optimizing \"by hand\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will start with a simple example : minimizing the square function.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def f(x):\n",
    "    return x ** 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will minimize the function $f$ \"by hand\" using the gradient descent algorithm.\n",
    "\n",
    "As a reminder, the update step of the algorithm is:\n",
    "$$x_{t+1} = x_{t} - \\lambda \\nabla_x f (x_t)$$\n",
    "\n",
    "Note:\n",
    "- The gradient information $\\nabla_x f (x)$ is stored in `x.grad`. Once we have run the `backward` function, we can use it to do our update step.\n",
    "- We need to do `x.data = ...` in the update step since want to change x in place but don't want autograd to track this change "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# YOUR TURN\n",
    "\n",
    "x0 = 8\n",
    "lr = 0.01\n",
    "iterations = 10\n",
    "\n",
    "x = torch.Tensor([x0]).requires_grad_()\n",
    "y = f(x)\n",
    "\n",
    "for i in range(iterations):\n",
    "\n",
    "    # < YOUR CODE HERE >\n",
    "\n",
    "    print(y.data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Why do we have x.data?\n",
    "\n",
    "If you do `x = ...`, then x is not a leaf variable anymore and will have a computation history. Since it is not a leaf anymore after the first iteration, its gradient will not be available at the second iteration.\n",
    "\n",
    "Workarounds:\n",
    " - Define x as a new leaf variable requiring gradient at each iterations using `detach()` and `require_grad_()`\n",
    " - Update `x.data` so that it is not recorded by autograd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optimizing with an optimizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Different optimizers\n",
    "Pytorch provides most common optimization algorithms encapsulated into \"optimizer classes\".  \n",
    "An optimizer is a useful object that automatically loops through all the numerous parameters of your model and performs the (potentially complex) update step for you.\n",
    "\n",
    "You first need to execute `import optim`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.optim as optim"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below are the most commonly used optimizers. Each of them have its specific parameters that you can check on the [Pytorch Doc](https://pytorch.org/docs/master/optim.html#algorithms)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "parameters = [x]  # This should be the list of model parameters\n",
    "\n",
    "optimizer = optim.SGD(parameters, lr=0.01, momentum=0.9)\n",
    "optimizer = optim.Adam(parameters, lr=0.01)\n",
    "optimizer = optim.Adadelta(parameters, lr=0.01)\n",
    "optimizer = optim.Adagrad(parameters, lr=0.01)\n",
    "optimizer = optim.RMSprop(parameters, lr=0.01)\n",
    "optimizer = optim.LBFGS(parameters, lr=0.01)\n",
    "\n",
    "# and there is more ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using an optimizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's use an optimizer to do the optimization !\n",
    "\n",
    "You will need 2 new functions:\n",
    "- `optimizer.zero_grad()` : This function sets the gradient of the parameters (x here) to 0 (otherwise it will get accumulated)\n",
    "- `optimizer.step()` :  This function applies an update step"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# YOUR TURN\n",
    "\n",
    "x0 = 8\n",
    "lr = 0.01\n",
    "iterations = 10\n",
    "\n",
    "x = torch.Tensor([x0]).requires_grad_()\n",
    "y = f(x)\n",
    "\n",
    "# Define your optimizer\n",
    "optimizer =  # < YOUR CODE HERE >\n",
    "\n",
    "for i in range(iterations):\n",
    "    \n",
    "    # < YOUR CODE HERE >\n",
    "    \n",
    "    print(y.data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Learning rate scheduler"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Learning rate scheduler seek to adjust the learning rate during training by reducing the learning rate according to a pre-defined schedule.  \n",
    "Below are some of the scheduler available in pytorch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "optim.lr_scheduler.LambdaLR\n",
    "optim.lr_scheduler.ExponentialLR\n",
    "optim.lr_scheduler.MultiStepLR\n",
    "optim.lr_scheduler.StepLR\n",
    "\n",
    "# and some more ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try optim.lr_scheduler.ExponentialLR"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def f(x):\n",
    "    return x.abs() * 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x0 = 8\n",
    "lr = 0.5\n",
    "iterations = 15\n",
    "\n",
    "x = torch.Tensor([x0]).requires_grad_()\n",
    "optimizer = optim.SGD([x], lr=lr)\n",
    "scheduler = optim.lr_scheduler.ExponentialLR(optimizer, 0.8)\n",
    "\n",
    "for i in range(iterations):\n",
    "    optimizer.zero_grad()\n",
    "    y = f(x)\n",
    "    y.backward()\n",
    "    optimizer.step()\n",
    "    scheduler.step()\n",
    "    print(y.data, \" | lr : \", optimizer.param_groups[0]['lr'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Live Plots "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below are some live plots to see what actually happens when you optimize a function.  \n",
    "You can play with learning rates, optimizers and also define new functions to optimize !"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2D Plot - Optimization process"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from live_plot import init_2dplot, add_point_2d"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def function_2d(x):\n",
    "    return x ** 2 / 20 + x.sin().tanh()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x0 = 8\n",
    "lr = 3\n",
    "iterations = 15\n",
    "\n",
    "x_range = torch.arange(-10, 10, 0.1)\n",
    "init_2dplot(x_range, function_2d, delta_=0.5)\n",
    "\n",
    "x = torch.Tensor([x0]).requires_grad_()\n",
    "optimizer = torch.optim.Adam([x], lr=lr)\n",
    "\n",
    "for i in range(iterations):\n",
    "    optimizer.zero_grad()\n",
    "    f = function_2d(x)\n",
    "    f.backward()\n",
    "    add_point_2d(x, f)\n",
    "    optimizer.step()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3D Plot - Optimization process"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from live_plot import init_3dplot, add_point_3d"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Choose a function below and run the cell__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "elev, azim = 40, 250\n",
    "x0, y0 = 6, 0\n",
    "x_range = torch.arange(-10, 10, 1).float()\n",
    "y_range = torch.arange(-15, 10, 2).float()\n",
    "\n",
    "def function_3d(x, y):\n",
    "    return x ** 2 - y ** 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "elev, azim = 30, 130\n",
    "x0, y0 = 10, -4\n",
    "x_range = torch.arange(-10, 15, 1).float()\n",
    "y_range = torch.arange(-15, 10, 2).float()\n",
    "\n",
    "def function_3d(x, y):\n",
    "    return x ** 3 - y ** 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "elev, azim = 80, 130\n",
    "x0, y0 = 4, -5\n",
    "x_range = torch.arange(-10, 10, .5).float()\n",
    "y_range = torch.arange(-10, 10, 1).float()\n",
    "\n",
    "def function_3d(x, y):\n",
    "    return (x ** 2 + y ** 2).sqrt().sin()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "elev, azim = 37, 120\n",
    "x0, y0 = 6, -15\n",
    "x_range = torch.arange(-10, 12, 1).float()\n",
    "y_range = torch.arange(-25, 5, 1).float()\n",
    "\n",
    "# lr 0.15 momentum 0.5\n",
    "def function_3d(x, y):\n",
    "    return (x ** 2 / 20 + x.sin().tanh()) * (y.abs()) ** 1.2 + 5 * x.abs() + (y + 7)**2 / 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Optimize the function__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "init_3dplot(x_range, y_range, function_3d, elev, azim, delta_=0.1)\n",
    "\n",
    "#x0 = \n",
    "#y0 = \n",
    "\n",
    "lr = 0.01\n",
    "iterations = 15\n",
    "\n",
    "x = torch.Tensor([x0]).requires_grad_()\n",
    "y = torch.Tensor([y0]).requires_grad_()\n",
    "optimizer = torch.optim.SGD([x, y], lr=lr)\n",
    "\n",
    "for i in range(iterations):\n",
    "    optimizer.zero_grad()\n",
    "    f = function_3d(x, y)\n",
    "    f.backward()\n",
    "    add_point_3d(x, y, f)\n",
    "    optimizer.step()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Don't forget to download the notebook, otherwise your changes may be lost"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Download the notebook](figures/notebook-download.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--NAVIGATION-->\n",
    "# < [Autograd](2-Autograd.ipynb) | Optimization | [Modules](4-Modules.ipynb) >"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}