C3_W4_Lab_3_Saliency.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "C3_W4_Lab_3_Saliency.ipynb",
      "private_outputs": true,
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.4"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/vsnupoudel/AdvancedCvCourses/blob/main/C3_W4_Lab_3_Saliency.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6ZiJWyW4_7u7"
      },
      "source": [
        "# Ungraded Lab: Saliency\n",
        "\n",
        "Like class activation maps, saliency maps also tells us what parts of the image the model is focusing on when making its predictions. \n",
        "- The main difference is in saliency maps, we are just shown the relevant pixels instead of the learned features. \n",
        "- You can generate saliency maps by getting the gradient of the loss with respect to the image pixels. \n",
        "- This means that changes in certain pixels that strongly affect the loss will be shown brightly in your saliency map. \n",
        "\n",
        "Let's see how this is implemented in the following sections."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4k8CUIO5g78a"
      },
      "source": [
        "## Imports"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jeuqmYSvrn9w"
      },
      "source": [
        "%tensorflow_version 2.x\n",
        "\n",
        "import tensorflow as tf\n",
        "import tensorflow_hub as hub\n",
        "import cv2\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cn6Xzhpcg_co"
      },
      "source": [
        "## Build the model\n",
        "\n",
        "For the classifier, you will use the [Inception V3 model](https://arxiv.org/abs/1512.00567) available in [Tensorflow Hub](https://tfhub.dev/google/tf2-preview/inception_v3/classification/4). This has pre-trained weights that is able to detect 1001 classes. You can read more [here](https://tfhub.dev/google/tf2-preview/inception_v3/classification/4)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "S2GDglRGrvIl"
      },
      "source": [
        "# grab the model from Tensorflow hub and append a softmax activation\n",
        "model = tf.keras.Sequential([\n",
        "    hub.KerasLayer('https://tfhub.dev/google/tf2-preview/inception_v3/classification/4'),\n",
        "    tf.keras.layers.Activation('softmax')\n",
        "])\n",
        "\n",
        "# build the model based on a specified batch input shape\n",
        "model.build([None, 300, 300, 3])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6ftfBxTzfVkQ"
      },
      "source": [
        "model.summary()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QyW_KpZWkSne"
      },
      "source": [
        "## Get a sample image\n",
        "\n",
        "You will download a photo of a Siberian Husky that our model will classify. We left the option to download a Tabby Cat image instead if you want."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LlAXGlVhuFXb"
      },
      "source": [
        "!wget -O image.jpg https://cdn.pixabay.com/photo/2018/02/27/14/11/the-pacific-ocean-3185553_960_720.jpg\n",
        "\n",
        "# If you want to try the cat, uncomment this line\n",
        "# !wget -O image.jpg https://cdn.pixabay.com/photo/2018/02/27/14/11/the-pacific-ocean-3185553_960_720.jpg"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6jB-4Wq9fhZa"
      },
      "source": [
        "from PIL import Image \n",
        "  \n",
        "# open method used to open different extension image file \n",
        "Image.open(r\"image.jpg\") "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vjJq5MFbk8T9"
      },
      "source": [
        "## Preprocess the image\n",
        "\n",
        "The image needs to be preprocessed before being fed to the model. This is done in the following steps:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9sDUZf6Sui1_"
      },
      "source": [
        "# read the image\n",
        "img = cv2.imread('image.jpg')\n",
        "\n",
        "# format it to be in the RGB colorspace\n",
        "img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB) \n",
        "\n",
        "# resize to 300x300 and normalize pixel values to be in the range [0, 1]\n",
        "img = cv2.resize(img, (300, 300)) / 255.0\n",
        "\n",
        "# add a batch dimension in front\n",
        "image = np.expand_dims(img, axis=0)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p4W3B9bflaBU"
      },
      "source": [
        "We can now preview our input image."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "b2bGeCW_-vbl"
      },
      "source": [
        "plt.figure(figsize=(8, 8))\n",
        "plt.imshow(img)\n",
        "plt.axis('off')\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZpZ9Fw8UlgYQ"
      },
      "source": [
        "## Compute Gradients\n",
        "\n",
        "You will now get the gradients of the loss with respect to the input image pixels. This is the key step to generate the map later."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hoBGoGcqsOfi"
      },
      "source": [
        "# Siberian Husky's class ID in ImageNet\n",
        "class_index = 251   \n",
        "\n",
        "# If you downloaded the cat, use this line instead\n",
        "#class_index = 282   # Tabby Cat in ImageNet\n",
        "\n",
        "# number of classes in the model's training data\n",
        "num_classes = 1001\n",
        "\n",
        "# convert to one hot representation to match our softmax activation in the model definition\n",
        "expected_output = tf.one_hot( [class_index] * image.shape[0], num_classes)\n",
        "\n",
        "with tf.GradientTape() as tape:\n",
        "    # cast image to float\n",
        "    inputs = tf.cast(image, tf.float32)\n",
        "\n",
        "    # watch the input pixels\n",
        "    tape.watch(inputs)\n",
        "\n",
        "    # generate the predictions\n",
        "    predictions = model(inputs)\n",
        "\n",
        "    # get the loss\n",
        "    loss = tf.keras.losses.categorical_crossentropy(\n",
        "        expected_output, predictions\n",
        "    )\n",
        "\n",
        "# get the gradient with respect to the inputs\n",
        "gradients = tape.gradient(loss, inputs)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aMnOLnZMgRVW"
      },
      "source": [
        "# expected_output = tf.one_hot([251] * 300, 1001)\n",
        "# expected_output\n",
        "\n",
        "# image.shape[0]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8x9N7FS7oAv-"
      },
      "source": [
        "## Visualize the results\n",
        "\n",
        "Now that you have the gradients, you will do some postprocessing to generate the saliency maps and overlay it on the image."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ODpAzA-vsTTi"
      },
      "source": [
        "# reduce the RGB image to grayscale\n",
        "grayscale_tensor = tf.reduce_sum(tf.abs(gradients), axis=-1)\n",
        "\n",
        "# normalize the pixel values to be in the range [0, 255].\n",
        "# the max value in the grayscale tensor will be pushed to 255.\n",
        "# the min value will be pushed to 0.\n",
        "normalized_tensor = tf.cast(\n",
        "    255\n",
        "    * (grayscale_tensor - tf.reduce_min(grayscale_tensor))\n",
        "    / (tf.reduce_max(grayscale_tensor) - tf.reduce_min(grayscale_tensor)),\n",
        "    tf.uint8,\n",
        ")\n",
        "\n",
        "# remove the channel dimension to make the tensor a 2d tensor\n",
        "normalized_tensor = tf.squeeze(normalized_tensor)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d2n74-pAs2ir"
      },
      "source": [
        "Let's do a little sanity check to see the results of the conversion."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IXOyqipj5Fz6"
      },
      "source": [
        "# max and min value in the grayscale tensor\n",
        "print(np.max(grayscale_tensor[0]))\n",
        "print(np.min(grayscale_tensor[0]))\n",
        "print()\n",
        "\n",
        "# coordinates of the first pixel where the max and min values are located\n",
        "max_pixel = np.unravel_index(np.argmax(grayscale_tensor[0]), grayscale_tensor[0].shape)\n",
        "min_pixel = np.unravel_index(np.argmin(grayscale_tensor[0]), grayscale_tensor[0].shape)\n",
        "print(max_pixel)\n",
        "print(min_pixel)\n",
        "print()\n",
        "\n",
        "# these coordinates should have the max (255) and min (0) value in the normalized tensor\n",
        "print(normalized_tensor[max_pixel])\n",
        "print(normalized_tensor[min_pixel])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FMgNIk3jte62"
      },
      "source": [
        "You should get something like:\n",
        "\n",
        "```\n",
        "1.2167013\n",
        "0.0\n",
        "\n",
        "(203, 129)\n",
        "(0, 299)\n",
        "\n",
        "tf.Tensor(255, shape=(), dtype=uint8)\n",
        "tf.Tensor(0, shape=(), dtype=uint8)\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hVkRpp2VtoVK"
      },
      "source": [
        "Now let's see what this looks like when plotted. The white pixels show the parts the model focused on when classifying the image."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xW1C4JGLvYMk"
      },
      "source": [
        "plt.figure(figsize=(8, 8))\n",
        "plt.axis('off')\n",
        "plt.imshow(normalized_tensor, cmap='gray')\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wX1M6A6Ct48Y"
      },
      "source": [
        "Let's superimpose the normalized tensor to the input image to get more context. You can see that the strong pixels are over the husky and that is a good indication that the model is looking at the correct part of the image."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OeULunlW2Vln"
      },
      "source": [
        "gradient_color = cv2.applyColorMap(normalized_tensor.numpy(), cv2.COLORMAP_HOT)\n",
        "gradient_color = gradient_color / 255.0\n",
        "super_imposed = cv2.addWeighted(img, 0.5, gradient_color, 0.5, 0.0)\n",
        "\n",
        "plt.figure(figsize=(8, 8))\n",
        "plt.imshow(super_imposed)\n",
        "plt.axis('off')\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6JikpHcEhcvk"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}