C3_W2_Lab_1_Simple_Object_Detection.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "C3_W2_Lab_1_Simple_Object_Detection.ipynb",
      "private_outputs": true,
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.4"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mmANPR2jhCR6"
      },
      "source": [
        "# Simple Object Detection in Tensorflow\n",
        "\n",
        "This lab will walk you through how to use object detection models available in [Tensorflow Hub](https://www.tensorflow.org/hub). In the following sections, you will:\n",
        "\n",
        "* explore the Tensorflow Hub for object detection models\n",
        "* load the models in your workspace\n",
        "* preprocess an image for inference \n",
        "* run inference on the models and inspect the output\n",
        "\n",
        "Let's get started!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "G0ZC5irSY0Jx"
      },
      "source": [
        "#  https://tfhub.dev/tensorflow/faster_rcnn/inception_resnet_v2_640x640/1\n",
        "# https://tfhub.dev/google/faster_rcnn/openimages_v4/inception_resnet_v2/1"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8DkMLuGDhCR6"
      },
      "source": [
        "## Imports"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OEoRKdmByrb0"
      },
      "source": [
        "import tensorflow as tf\n",
        "import tensorflow_hub as hub\n",
        "from PIL import Image\n",
        "from PIL import ImageOps\n",
        "import tempfile\n",
        "from six.moves.urllib.request import urlopen\n",
        "from six import BytesIO"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nb8MBgTOhCR6"
      },
      "source": [
        "### Download the model from Tensorflow Hub\n",
        "\n",
        "Tensorflow Hub is a repository of trained machine learning models which you can reuse in your own projects. \n",
        "- You can see the domains covered [here](https://tfhub.dev/) and its subcategories. \n",
        "- For this lab, you will want to look at the [image object detection subcategory](https://tfhub.dev/s?module-type=image-object-detection). \n",
        "- You can select a model to see more information about it and copy the URL so you can download it to your workspace. \n",
        "- We selected a [inception resnet version 2](https://tfhub.dev/google/faster_rcnn/openimages_v4/inception_resnet_v2/1)\n",
        "- You can also modify this following cell to choose the other model that we selected, [ssd mobilenet version 2](https://tfhub.dev/tensorflow/ssd_mobilenet_v2/2)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "C9pCzz4uy20U"
      },
      "source": [
        "# you can switch the commented lines here to pick the other model\n",
        "\n",
        "# inception resnet version 2\n",
        "module_handle = \"https://tfhub.dev/google/faster_rcnn/openimages_v4/inception_resnet_v2/1\"\n",
        "\n",
        "# You can choose ssd mobilenet version 2 instead and compare the results\n",
        "#module_handle = \"https://tfhub.dev/google/openimages_v4/ssd/mobilenet_v2/1\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W3trj5FbhCR6"
      },
      "source": [
        "#### Load the model\n",
        "\n",
        "Next, you'll load the model specified by the `module_handle`.\n",
        "- This will take a few minutes to load the model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0WHkGDHfhCR6"
      },
      "source": [
        "model = hub.load(module_handle)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1Ey0FpHGhCR6"
      },
      "source": [
        "#### Choose the default signature\n",
        "\n",
        "Some models in the Tensorflow hub can be used for different tasks. So each model's documentation should show what *signature* to use when running the model. \n",
        "- If you want to see if a model has more than one signature then you can do something like `print(hub.load(module_handle).signatures.keys())`. In your case, the models you will be using only have the `default` signature so you don't have to worry about other types."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "X1BU7AGthCR6"
      },
      "source": [
        "# take a look at the available signatures for this particular model\n",
        "model.signatures.keys()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nfc9ax9hhCR6"
      },
      "source": [
        "Please choose the 'default' signature for your object detector.\n",
        "- For object detection models, its 'default' signature will accept a batch of image tensors and output a dictionary describing the objects detected, which is what you'll want here."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pzwR5zE_hCR7"
      },
      "source": [
        "detector = model.signatures['default']"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Wvb-3r3thCR7"
      },
      "source": [
        "### download_and_resize_image\n",
        "\n",
        "This function downloads an image specified by a given \"url\", pre-processes it, and then saves it to disk."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ucsxak_qhCR7"
      },
      "source": [
        "def download_and_resize_image(url, new_width=256, new_height=256):\n",
        "    '''\n",
        "    Fetches an image online, resizes it and saves it locally.\n",
        "    \n",
        "    Args:\n",
        "        url (string) -- link to the image\n",
        "        new_width (int) -- size in pixels used for resizing the width of the image\n",
        "        new_height (int) -- size in pixels used for resizing the length of the image\n",
        "        \n",
        "    Returns:\n",
        "        (string) -- path to the saved image\n",
        "    '''\n",
        "    \n",
        "    \n",
        "    # create a temporary file ending with \".jpg\"\n",
        "    _, filename = tempfile.mkstemp(suffix=\".jpg\")\n",
        "    \n",
        "    # opens the given URL\n",
        "    response = urlopen(url)\n",
        "    \n",
        "    # reads the image fetched from the URL\n",
        "    image_data = response.read()\n",
        "    \n",
        "    # puts the image data in memory buffer\n",
        "    image_data = BytesIO(image_data)\n",
        "    \n",
        "    # opens the image\n",
        "    pil_image = Image.open(image_data)\n",
        "    \n",
        "    # resizes the image. will crop if aspect ratio is different.\n",
        "    pil_image = ImageOps.fit(pil_image, (new_width, new_height), Image.ANTIALIAS)\n",
        "    \n",
        "    # converts to the RGB colorspace\n",
        "    pil_image_rgb = pil_image.convert(\"RGB\")\n",
        "    \n",
        "    # saves the image to the temporary file created earlier\n",
        "    pil_image_rgb.save(filename, format=\"JPEG\", quality=90)\n",
        "    \n",
        "    print(\"Image downloaded to %s.\" % filename)\n",
        "    \n",
        "    return filename"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r7qodEJHhCR7"
      },
      "source": [
        "### Download and preprocess an image\n",
        "\n",
        "Now, using `download_and_resize_image` you can get a sample image online and save it locally. \n",
        "- We've provided a URL for you, but feel free to choose another image to run through the object detector.\n",
        "- You can use the original width and height of the image but feel free to modify it and see what results you get."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xHTDalVrhCR7"
      },
      "source": [
        "# You can choose a different URL that points to an image of your choice\n",
        "image_url = \"https://upload.wikimedia.org/wikipedia/commons/f/fb/20130807_dublin014.JPG\"\n",
        "\n",
        "# download the image and use the original height and width\n",
        "downloaded_image_path = download_and_resize_image(image_url, 3872, 2592)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_or4uYnef_MM"
      },
      "source": [
        "downloaded_image_path"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IVNXUKMIhCR7"
      },
      "source": [
        "### run_detector\n",
        "\n",
        "This function will take in the object detection model `detector` and the path to a sample image, then use this model to detect objects and display its predicted class categories and detection boxes.\n",
        "- run_detector uses `load_image` to convert the image into a tensor."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wkkiQzKlhCR7"
      },
      "source": [
        "def load_img(path):\n",
        "    '''\n",
        "    Loads a JPEG image and converts it to a tensor.\n",
        "    \n",
        "    Args:\n",
        "        path (string) -- path to a locally saved JPEG image\n",
        "    \n",
        "    Returns:\n",
        "        (tensor) -- an image tensor\n",
        "    '''\n",
        "    \n",
        "    # read the file\n",
        "    img = tf.io.read_file(path)\n",
        "    \n",
        "    # convert to a tensor\n",
        "    img = tf.image.decode_jpeg(img, channels=3)\n",
        "    \n",
        "    return img\n",
        "\n",
        "\n",
        "def run_detector(detector, path):\n",
        "    '''\n",
        "    Runs inference on a local file using an object detection model.\n",
        "    \n",
        "    Args:\n",
        "        detector (model) -- an object detection model loaded from TF Hub\n",
        "        path (string) -- path to an image saved locally\n",
        "    '''\n",
        "    \n",
        "    # load an image tensor from a local file path\n",
        "    img = load_img(path)\n",
        "\n",
        "    # add a batch dimension in front of the tensor\n",
        "    converted_img  = tf.image.convert_image_dtype(img, tf.float32)[tf.newaxis, ...]\n",
        "    \n",
        "    # run inference using the model\n",
        "    result = detector(converted_img)\n",
        "\n",
        "    # save the results in a dictionary\n",
        "    result = {key:value.numpy() for key,value in result.items()}\n",
        "\n",
        "    # print results\n",
        "    print(\"Found %d objects.\" % len(result[\"detection_scores\"]))\n",
        "\n",
        "    print(result[\"detection_scores\"])\n",
        "    print(result[\"detection_class_entities\"])\n",
        "    print(result[\"detection_boxes\"])\n",
        "\n",
        "    return result\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DSEeJSkxhCR7"
      },
      "source": [
        "### Run inference on the image\n",
        "\n",
        "You can run your detector by calling the `run_detector` function. This will print the number of objects found followed by three lists: \n",
        "\n",
        "* The detection scores of each object found (i.e. how confident the model is), \n",
        "* The classes of each object found, \n",
        "* The bounding boxes of each object\n",
        "\n",
        "You will see how to overlay this information on the original image in the next sections and in this week's assignment!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "csanHvDIz4_t"
      },
      "source": [
        "# runs the object detection model and prints information about the objects found\n",
        "results = run_detector(detector, downloaded_image_path)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "U2Qw9mWzgWZJ"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}