Tutorial4_advanced_vision_tutorial_2020.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "view-in-github"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/pgmikhael/mit_deeplearning_bootcamp/blob/master/Tutorial4_advanced_vision_tutorial_2020.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "GA8LLAO9dchw"
      },
      "source": [
        "# Advanced Computer Vision Concepts with Fashion MNIST\n",
        "\n",
        "In this tutorial, we'll take you through developing your own custom-CNNs, using pre-trained CNN models, and common computer vision best-practices.\n",
        "\n",
        "Let's get started!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "eT_jLzv8do9z",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "# http://pytorch.org/\n",
        "from os import path\n",
        "from wheel.pep425tags import get_abbr_impl, get_impl_ver, get_abi_tag\n",
        "platform = '{}{}-{}'.format(get_abbr_impl(), get_impl_ver(), get_abi_tag())\n",
        "\n",
        "\n",
        "!pip install torch torchvision\n",
        "import torch\n",
        "print(torch.__version__)\n",
        "print(torch.cuda.is_available())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "mcLJrU16dch0",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "import argparse\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.nn.functional as F\n",
        "import torch.optim as optim\n",
        "from torchvision import datasets, transforms\n",
        "from tqdm import tqdm\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "from tqdm import tqdm "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "DMJfz2f-JC9E",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "#@title Helper Function to display Images { display-mode: \"form\" }\n",
        "def plot_images(images, cls_true):\n",
        "    assert len(images) == len(cls_true) == 9\n",
        "    \n",
        "    # Create figure with 3x3 sub-plots.\n",
        "    fig, axes = plt.subplots(3, 3, figsize=(4,4))\n",
        "    fig.subplots_adjust(hspace=0.3, wspace=0.3)\n",
        "\n",
        "    for i, ax in enumerate(axes.flat):\n",
        "        # Plot image.\n",
        "        ax.imshow(np.array(images[i], dtype='float').reshape((28,28))*255, cmap='binary')\n",
        "\n",
        "        # Show true and predicted classes.\n",
        "\n",
        "        xlabel = \"True: {0}\".format(cls_true[i])\n",
        "\n",
        "        # Show the classes as the label on the x-axis.\n",
        "        ax.set_xlabel(xlabel)\n",
        "        \n",
        "        # Remove ticks from the plot.\n",
        "        ax.set_xticks([])\n",
        "        ax.set_yticks([])\n",
        "    \n",
        "    # Ensure the plot is shown correctly with multiple plots\n",
        "    # in a single Notebook cell.\n",
        "    plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "yoX9FXdRCKVw",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Z11d818rdcir"
      },
      "source": [
        "# The Task: Fashion-MNIST, Digit Classification\n",
        "<img src=\"https://github.com/zalandoresearch/fashion-mnist/blob/master/doc/img/fashion-mnist-sprite.png?raw=true\">\n",
        "\n",
        "In this lab, we'll build a neural network to classify articles of clothing.\n",
        "\n",
        "\n",
        "\n",
        "## Step 1: Loading Data and Preprocessing\n",
        "Let's start by loading the data.\n",
        "We're going to normalize our images to have 0 mean, and unit variance. We'll do this using some [torchvision](https://pytorch.org/docs/stable/torchvision/index.html) transforms. This generally helps stablize learning, and is common practice. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "moG_DE1ldcis",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "# Img mean value of .13, and stdv of .31 were computed across entire train set\n",
        "# in prior work\n",
        "normalize_image = transforms.Compose([\n",
        "                           transforms.ToTensor(),\n",
        "                           transforms.Normalize((0.1307,), (0.3081,))\n",
        "                ])\n",
        "\n",
        "# Dataset is loaded fro torchvision\n",
        "all_train = datasets.FashionMNIST('data', train=True, download=True, transform=normalize_image)\n",
        "\n",
        "num_train = int(len(all_train)*.8)\n",
        "train = [all_train[i] for i in range(num_train)]\n",
        "dev = [all_train[i] for i in range(num_train,len(all_train))]\n",
        "test = datasets.FashionMNIST('data', train=False, download=True, \n",
        "                      transform=normalize_image)\n",
        "                           \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "FcA_31h9jZ_4"
      },
      "source": [
        "### Review Question:\n",
        "1. What functions does the FashionMNIST dataset object have to impelement?"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "55n2msG5dciy",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "all_train = datasets.FashionMNIST('data', train=True, download=True)\n",
        "# images = [tr[0] for tr in all_train[:9]]\n",
        "num_examples = 9 \n",
        "images, labels = [], []\n",
        "for i in range(num_examples):\n",
        "  images.append(all_train[i][0])\n",
        "  labels.append(all_train[i][1])\n",
        "    \n",
        "plot_images(images, labels)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "XHq9MwYVdci5",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "train[0][0].size()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "s2jGlFlidci-"
      },
      "source": [
        "## Step 2: Building a model\n",
        "\n",
        "All pytorch models should be implemented as instances of `nn.Module`. \n",
        "\n",
        "To build a model you need to:\n",
        "a) define what parameters it'll need in it's `__init__` function\n",
        "b) define the model's computation, using those parameters, in a forward function.\n",
        "\n",
        "\n",
        "To keep things simple, lets define a simple linear classifer, like logistic regression. We'll experiment with more complex models soon."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "c0NnnFWgdci-",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "class Model(nn.Module):\n",
        "    def __init__(self):\n",
        "        super(Model, self).__init__()\n",
        "        # Learn weights for each pixel and perform linear regression\n",
        "        self.fc = nn.Linear(3*28*28, 10)\n",
        "\n",
        "    def forward(self, x):\n",
        "        batch_size, num_channels, height, width = x.size()\n",
        "        # Flatten image\n",
        "        x = x.view(batch_size, -1)\n",
        "        # Put it through linear classifier\n",
        "        return self.fc(x)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "M9hvStGzdcjC"
      },
      "source": [
        "## Step 3. Defining our training procedure\n",
        "\n",
        "To train our model, let's introduce a couple new PyTorch ideas.\n",
        "\n",
        "A [DataLoader](https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader) is an iterator that goes over our entire dataset and selects batches. \n",
        "We'll be using this to iterate through our train/dev/test sets.\n",
        "\n",
        "Let's intialize these now. \n",
        "\n",
        "An [Optimizer](https://pytorch.org/docs/stable/optim.html) defines an update rule. In class, we've discussed vanilla SGD, which is one method to compute the next weight, given the current weight and gradient. There are plently of other optimizers you can try from the pytorch library. \n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "n3Ht1mhhdcjE",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "# Training settings\n",
        "epochs = 10\n",
        "lr = .01\n",
        "momentum = 0.5\n",
        "\n",
        "batch_size = 32\n",
        "train_loader = torch.utils.data.DataLoader(train, batch_size=batch_size, shuffle=True)\n",
        "dev_loader = torch.utils.data.DataLoader(dev, batch_size=batch_size, shuffle=True)\n",
        "test_loader = torch.utils.data.DataLoader(test, batch_size=batch_size, shuffle=True)\n",
        "\n",
        "\n",
        "model = Model()\n",
        "optimizer = optim.Adam(model.parameters(), lr=lr)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "ePITAe9LjqK4"
      },
      "source": [
        "### Review Question:\n",
        "1. What are the steps of training?\n",
        "2. Will this be any different in vision vs NLP?"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "wxIj0eWsdcjJ",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "def train_epoch( model, train_loader, optimizer, epoch):\n",
        "    model.train() # Set the nn.Module to train mode. \n",
        "    model = model.to('cuda')\n",
        "    total_loss = 0\n",
        "    correct = 0\n",
        "    num_samples = len(train_loader.dataset)\n",
        "    for batch_idx, (x, target) in tqdm(enumerate(train_loader), total=len(train_loader)): #1) get batch\n",
        "        x, target = x.to('cuda'), target.to('cuda')\n",
        "        B, C, H, W = x.size()\n",
        "        x = x.expand([B,3,H,W]).contiguous()\n",
        "        # Reset gradient data to 0\n",
        "        optimizer.zero_grad()\n",
        "        # Get prediction for batch\n",
        "        output = model(x)\n",
        "        # 2) Compute loss\n",
        "        loss = F.cross_entropy(output, target)\n",
        "        #3) Do backprop\n",
        "        loss.backward()\n",
        "        #4) Update model\n",
        "        optimizer.step()\n",
        "        \n",
        "        ## Do book-keeping to track accuracy and avg loss\n",
        "        pred = output.max(1, keepdim=True)[1] # get the index of the max log-probability\n",
        "        correct += pred.eq(target.view_as(pred)).sum().item()\n",
        "        total_loss += loss.detach() # Don't keep computation graph \n",
        "\n",
        "    print('Train Epoch: {} \\tLoss: {:.4f}, Accuracy: {}/{} ({:.0f}%)'.format(\n",
        "            epoch, total_loss / num_samples, \n",
        "            correct, \n",
        "            num_samples,\n",
        "            100. * correct / num_samples))\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "xGZnlI58dcjN"
      },
      "source": [
        "## Step 3.5 Define our evaluation loop\n",
        "Similar to above, we'll also loop through our dev or test set, and compute our loss and accuracy. \n",
        "This lets us see how well our model is generalizing. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "GPzqSuY3dcjO",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "def eval_epoch(model, test_loader, name):\n",
        "    model = model.to('cuda')\n",
        "    model.eval()\n",
        "    test_loss = 0\n",
        "    correct = 0\n",
        "    for data, target in tqdm(test_loader):\n",
        "        data, target = data.to('cuda'), target.to('cuda')\n",
        "        B, C, H, W = data.size()\n",
        "        data = data.expand([B,3,H,W]).contiguous()\n",
        "        output = model(data)\n",
        "        test_loss += F.cross_entropy(output, target).item() # sum up batch loss\n",
        "        pred = output.max(1, keepdim=True)[1] # get the index of the max log-probability\n",
        "        correct += pred.eq(target.view_as(pred)).sum().item()\n",
        "\n",
        "    test_loss /= len(test_loader.dataset)\n",
        "    print('\\n{} set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\\n'.format(\n",
        "        name,\n",
        "        test_loss, \n",
        "        correct, \n",
        "        len(test_loader.dataset),\n",
        "        100. * correct / len(test_loader.dataset)))\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Pp9H5tWEdcjR"
      },
      "source": [
        "## Step 4: Training the model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "vVpU_N0idcjS",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "\n",
        "for epoch in range(1, epochs + 1):\n",
        "    train_epoch(model, train_loader, optimizer, epoch)\n",
        "    eval_epoch(model,  dev_loader, \"Dev\")\n",
        "    print(\"---\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "WwBmuGetdcjY"
      },
      "source": [
        "# Step 5. Experiment with MLP\n",
        "Let's try a more complex model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "e4x3E0AcdcjZ",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "class Model(nn.Module):\n",
        "    def __init__(self):\n",
        "        super(Model, self).__init__()\n",
        "        self.fc1 = nn.Linear(3*28*28, 200)\n",
        "        self.fc2 = nn.Linear(200, 200)\n",
        "        self.fc3 = nn.Linear(200, 10)\n",
        "        \n",
        "\n",
        "    def forward(self, x):\n",
        "        batch_size, num_channels, height, width = x.size()\n",
        "        x = x.view(batch_size, -1)\n",
        "        hidden = F.relu(self.fc1(x))\n",
        "        hidden = F.relu(self.fc2(hidden))\n",
        "        logit = self.fc3(hidden)\n",
        "        return logit\n",
        "    \n",
        "model = Model()\n",
        "optimizer = optim.Adam(model.parameters(), lr=lr)\n",
        "\n",
        "for epoch in range(1, epochs + 1):\n",
        "    train_epoch(model, train_loader, optimizer, epoch)\n",
        "    eval_epoch(model,  dev_loader, \"Dev\")\n",
        "    print(\"---\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Z8Fc1zMI0-4U"
      },
      "source": [
        "# Step 6. Try a CNN\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "yZUkHfjwdcjg",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "class Model(nn.Module):\n",
        "    def __init__(self):\n",
        "        super(Model, self).__init__()\n",
        "        self.hidden_dim = 512\n",
        "        self.conv1 = nn.Conv2d(3, self.hidden_dim // 4, kernel_size=3, stride=2)\n",
        "        self.conv2 = nn.Conv2d(self.hidden_dim // 4, self.hidden_dim // 2, kernel_size=3, stride=2)\n",
        "        self.conv3 = nn.Conv2d(self.hidden_dim // 2, self.hidden_dim, kernel_size=3, stride=1)\n",
        "        self.fc = nn.Linear(self.hidden_dim, 10)\n",
        "        \n",
        "    def forward(self, x):\n",
        "        batch_size, num_channels, height, width = x.size()\n",
        "        \n",
        "        hidden = F.relu(self.conv1(x))\n",
        "        hidden = F.relu(self.conv2(hidden))\n",
        "        hidden = F.relu(self.conv3(hidden))\n",
        "        hidden = hidden.view((batch_size, self.hidden_dim, -1))\n",
        "        hidden,_ = torch.max(hidden, dim=-1)\n",
        "        logit = self.fc(hidden)\n",
        "        return logit\n",
        "\n",
        "model = Model()\n",
        "optimizer = optim.Adam(model.parameters(), lr=lr)\n",
        "\n",
        "for epoch in range(1, epochs + 1):\n",
        "    train_epoch(model, train_loader, optimizer, epoch)\n",
        "    eval_epoch(model,  dev_loader, \"Dev\")\n",
        "    print(\"---\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "tXsKu4zTI5iw"
      },
      "source": [
        "# Try State-Of-The-Art BenchMarks:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "0NMUCnBMQ_Hw",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "import torchvision.models\n",
        "\n",
        "model = torchvision.models.resnet18( pretrained=True)\n",
        "model.fc = nn.Linear(512,10)\n",
        "print(model)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "n1tWxneCJAn2",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "batch_size = 64\n",
        "train_loader = torch.utils.data.DataLoader(train, batch_size=batch_size, shuffle=True)\n",
        "dev_loader = torch.utils.data.DataLoader(dev, batch_size=batch_size, shuffle=True)\n",
        "test_loader = torch.utils.data.DataLoader(test, batch_size=batch_size, shuffle=True)\n",
        "\n",
        "lr = 1e-3\n",
        "epochs = 10\n",
        "optimizer = optim.Adam(model.parameters(), lr=lr)\n",
        "\n",
        "for epoch in range(1, epochs + 1):\n",
        "    train_epoch(model, train_loader, optimizer, epoch)\n",
        "    eval_epoch(model,  dev_loader, \"Dev\")\n",
        "    print(\"---\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "vrjftzTMdcjr"
      },
      "source": [
        "## Step 7. Explore further.\n",
        "You can try different model architectures, different optimizers, learning rates and regularization strategies. Neural networks are incredibly flexibile, and so the space to do explore is enourmous.  Once you're done exploring, take your best model (i.e achieves best results on dev set) and run it on test!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "dNWqKY72dcjs",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "eval_epoch(model,  test_loader, \"Test\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Saving and Loading"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "torch.save(model,'path.pt')     # to save\n",
        "model = torch.load('path.pt')   # to load a saved model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "F7ShMXzkmRuH"
      },
      "source": [
        "### Discussion Questions:\n",
        "1. We saw the dev performance sometimes goes down during training, why? What should we do about it?\n",
        "2. Given another pretrained model, how do you retrofit to your task?\n",
        "3. When should you not use a pretrained model?\n",
        "4. Could we use RNN for vision tasks? What would that imply?\n",
        "5. How we adapt a CNN to 3D images?\n",
        "6. Given a new task, which model should you try first? Why?"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [],
      "include_colab_link": true,
      "name": "advanced_vision_tutorial_2020.ipynb",
      "private_outputs": true,
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}