C3_W2_Lab_2_Object_Detection.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "C3_W2_Lab_2_Object_Detection.ipynb",
      "private_outputs": true,
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.4"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CxmDMK4yupqg"
      },
      "source": [
        "# Object Detection\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Sy553YSVmYiK"
      },
      "source": [
        "This lab is similar to the previous lab, except now instead of printing out the bounding box coordinates, you can visualize these bounding boxes on top of the image!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v4XGxDrCkeip"
      },
      "source": [
        "## Setup\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6cPY9Ou4sWs_"
      },
      "source": [
        "# For running inference on the TF-Hub module.\n",
        "import tensorflow as tf\n",
        "\n",
        "import tensorflow_hub as hub\n",
        "\n",
        "# For downloading the image.\n",
        "import matplotlib.pyplot as plt\n",
        "import tempfile\n",
        "from six.moves.urllib.request import urlopen\n",
        "from six import BytesIO\n",
        "\n",
        "# For drawing onto the image.\n",
        "import numpy as np\n",
        "from PIL import Image\n",
        "from PIL import ImageColor\n",
        "from PIL import ImageDraw\n",
        "from PIL import ImageFont\n",
        "from PIL import ImageOps\n",
        "\n",
        "# For measuring the inference time.\n",
        "import time\n",
        "\n",
        "# Print Tensorflow version\n",
        "print(tf.__version__)\n",
        "\n",
        "# Check available GPU devices.\n",
        "print(\"The following GPU devices are available: %s\" % tf.test.gpu_device_name())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t-VdfLbC1w51"
      },
      "source": [
        "### Select and load the model\n",
        "As in the previous lab, you can choose an object detection module. Here are two that we've selected for you:\n",
        "* [ssd + mobilenet V2](https://tfhub.dev/tensorflow/ssd_mobilenet_v2/2) small and fast.\n",
        "* [FasterRCNN + InceptionResNet V2](https://tfhub.dev/google/faster_rcnn/openimages_v4/inception_resnet_v2/1): high accuracy"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uazJ5ASc2_QE"
      },
      "source": [
        "# you can switch the commented lines here to pick the other model\n",
        "\n",
        "# ssd mobilenet version 2\n",
        "module_handle = \"https://tfhub.dev/google/openimages_v4/ssd/mobilenet_v2/1\"\n",
        "\n",
        "# You can choose inception resnet version 2 instead\n",
        "#module_handle = \"https://tfhub.dev/google/faster_rcnn/openimages_v4/inception_resnet_v2/1\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xRVwVr40grw6"
      },
      "source": [
        "#### Load the model\n",
        "\n",
        "Next, you'll load the model specified by the `module_handle`.\n",
        "- This will take a few minutes to load the model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MsFSuo2Wgrw6"
      },
      "source": [
        "model = hub.load(module_handle)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0gXxP2xYgrw6"
      },
      "source": [
        "#### Choose the default signature\n",
        "\n",
        "As before, you can check the available signatures using `.signature.keys()`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VGkmPO1lgrw6"
      },
      "source": [
        "# take a look at the available signatures for this particular model\n",
        "model.signatures.keys()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VdwgkV7igrw6"
      },
      "source": [
        "Please choose the 'default' signature for your object detector."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IJ6plWWUgrw6"
      },
      "source": [
        "detector = model.signatures['default']\n",
        "detector"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o_aN2zXggrw6"
      },
      "source": [
        "### download_and_resize_image\n",
        "\n",
        "As you saw in the previous lab, this function downloads an image specified by a given \"url\", pre-processes it, and then saves it to disk.\n",
        "- What new compared to the previous lab is that you an display the image if you set the parameter `display=True`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gjmOWdPpgrw6"
      },
      "source": [
        "def display_image(image):\n",
        "    \"\"\"\n",
        "    Displays an image inside the notebook.\n",
        "    This is used by download_and_resize_image()\n",
        "    \"\"\"\n",
        "    fig = plt.figure(figsize=(20, 15))\n",
        "    plt.grid(False)\n",
        "    plt.imshow(image)\n",
        "\n",
        "\n",
        "def download_and_resize_image(url, new_width=256, new_height=256, display=False):\n",
        "    '''\n",
        "    Fetches an image online, resizes it and saves it locally.\n",
        "    \n",
        "    Args:\n",
        "        url (string) -- link to the image\n",
        "        new_width (int) -- size in pixels used for resizing the width of the image\n",
        "        new_height (int) -- size in pixels used for resizing the length of the image\n",
        "        \n",
        "    Returns:\n",
        "        (string) -- path to the saved image\n",
        "    '''\n",
        "    \n",
        "    \n",
        "    # create a temporary file ending with \".jpg\"\n",
        "    _, filename = tempfile.mkstemp(suffix=\".jpg\")\n",
        "    \n",
        "    # opens the given URL\n",
        "    response = urlopen(url)\n",
        "    \n",
        "    # reads the image fetched from the URL\n",
        "    image_data = response.read()\n",
        "    \n",
        "    # puts the image data in memory buffer\n",
        "    image_data = BytesIO(image_data)\n",
        "    \n",
        "    # opens the image\n",
        "    pil_image = Image.open(image_data)\n",
        "    \n",
        "    # resizes the image. will crop if aspect ratio is different.\n",
        "    pil_image = ImageOps.fit(pil_image, (new_width, new_height), Image.ANTIALIAS)\n",
        "    \n",
        "    # converts to the RGB colorspace\n",
        "    pil_image_rgb = pil_image.convert(\"RGB\")\n",
        "    \n",
        "    # saves the image to the temporary file created earlier\n",
        "    pil_image_rgb.save(filename, format=\"JPEG\", quality=90)\n",
        "    \n",
        "    print(\"Image downloaded to %s.\" % filename)\n",
        "    \n",
        "    if display:\n",
        "        display_image(pil_image)\n",
        "\n",
        "    \n",
        "    return filename"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2m5qn23ogrw6"
      },
      "source": [
        "### Select and load an image\n",
        "Load a public image from Open Images v4, save locally, and display."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ntMGNQldgrw6"
      },
      "source": [
        "# By Heiko Gorski, Source: https://commons.wikimedia.org/wiki/File:Naxos_Taverna.jpg\n",
        "image_url = \"https://upload.wikimedia.org/wikipedia/commons/6/60/Naxos_Taverna.jpg\"  #@param\n",
        "downloaded_image_path = download_and_resize_image(image_url, 1280, 856, True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nAunxsz8grw6"
      },
      "source": [
        "### Draw bounding boxes\n",
        "\n",
        "To build on what you saw in the previous lab, you can now visualize the predicted bounding boxes, overlaid on top of the image.  \n",
        "- You can use `draw_boxes` to do this.  It will use `draw_bounding_box_on_image` to draw the bounding boxes."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "J5rUpPPqgrw7"
      },
      "source": [
        "def draw_bounding_box_on_image(image,\n",
        "                               ymin,\n",
        "                               xmin,\n",
        "                               ymax,\n",
        "                               xmax,\n",
        "                               color,\n",
        "                               font,\n",
        "                               thickness=4,\n",
        "                               display_str_list=()):\n",
        "\n",
        "    \"\"\"\n",
        "    Adds a bounding box to an image.\n",
        "    \n",
        "    Args:\n",
        "        image -- the image object\n",
        "        ymin -- bounding box coordinate\n",
        "        xmin -- bounding box coordinate\n",
        "        ymax -- bounding box coordinate\n",
        "        xmax -- bounding box coordinate\n",
        "        color -- color for the bounding box edges\n",
        "        font -- font for class label\n",
        "        thickness -- edge thickness of the bounding box\n",
        "        display_str_list -- class labels for each object detected\n",
        "    \n",
        "    \n",
        "    Returns:\n",
        "        No return.  The function modifies the `image` argument \n",
        "                    that gets passed into this function\n",
        "    \n",
        "    \"\"\"\n",
        "    draw = ImageDraw.Draw(image)\n",
        "    im_width, im_height = image.size\n",
        "    \n",
        "    # scale the bounding box coordinates to the height and width of the image\n",
        "    (left, right, top, bottom) = (xmin * im_width, xmax * im_width,\n",
        "                                ymin * im_height, ymax * im_height)\n",
        "    \n",
        "    # define the four edges of the detection box\n",
        "    draw.line([(left, top), (left, bottom), (right, bottom), (right, top),\n",
        "             (left, top)],\n",
        "            width=thickness,\n",
        "            fill=color)\n",
        "\n",
        "    # If the total height of the display strings added to the top of the bounding\n",
        "    # box exceeds the top of the image, stack the strings below the bounding box\n",
        "    # instead of above.\n",
        "    display_str_heights = [font.getsize(ds)[1] for ds in display_str_list]\n",
        "    # Each display_str has a top and bottom margin of 0.05x.\n",
        "    total_display_str_height = (1 + 2 * 0.05) * sum(display_str_heights)\n",
        "\n",
        "    if top > total_display_str_height:\n",
        "        text_bottom = top\n",
        "    else:\n",
        "        text_bottom = top + total_display_str_height\n",
        "        \n",
        "    # Reverse list and print from bottom to top.\n",
        "    for display_str in display_str_list[::-1]:\n",
        "        text_width, text_height = font.getsize(display_str)\n",
        "        margin = np.ceil(0.05 * text_height)\n",
        "        draw.rectangle([(left, text_bottom - text_height - 2 * margin),\n",
        "                        (left + text_width, text_bottom)],\n",
        "                       fill=color)\n",
        "        draw.text((left + margin, text_bottom - text_height - margin),\n",
        "                  display_str,\n",
        "                  fill=\"black\",\n",
        "                  font=font)\n",
        "        text_bottom -= text_height - 2 * margin\n",
        "\n",
        "\n",
        "def draw_boxes(image, boxes, class_names, scores, max_boxes=10, min_score=0.1):\n",
        "    \"\"\"\n",
        "    Overlay labeled boxes on an image with formatted scores and label names.\n",
        "    \n",
        "    Args:\n",
        "        image -- the image as a numpy array\n",
        "        boxes -- list of detection boxes\n",
        "        class_names -- list of classes for each detected object\n",
        "        scores -- numbers showing the model's confidence in detecting that object\n",
        "        max_boxes -- maximum detection boxes to overlay on the image (default is 10)\n",
        "        min_score -- minimum score required to display a bounding box\n",
        "    \n",
        "    Returns:\n",
        "        image -- the image after detection boxes and classes are overlaid on the original image.\n",
        "    \"\"\"\n",
        "    colors = list(ImageColor.colormap.values())\n",
        "\n",
        "    try:\n",
        "        font = ImageFont.truetype(\"/usr/share/fonts/truetype/liberation/LiberationSansNarrow-Regular.ttf\",\n",
        "                              25)\n",
        "    except IOError:\n",
        "        print(\"Font not found, using default font.\")\n",
        "        font = ImageFont.load_default()\n",
        "\n",
        "    for i in range(min(boxes.shape[0], max_boxes)):\n",
        "        \n",
        "        # only display detection boxes that have the minimum score or higher\n",
        "        if scores[i] >= min_score:\n",
        "            ymin, xmin, ymax, xmax = tuple(boxes[i])\n",
        "            display_str = \"{}: {}%\".format(class_names[i].decode(\"ascii\"),\n",
        "                                         int(100 * scores[i]))\n",
        "            color = colors[hash(class_names[i]) % len(colors)]\n",
        "            image_pil = Image.fromarray(np.uint8(image)).convert(\"RGB\")\n",
        "\n",
        "            # draw one bounding box and overlay the class labels onto the image\n",
        "            draw_bounding_box_on_image(image_pil,\n",
        "                                       ymin,\n",
        "                                       xmin,\n",
        "                                       ymax,\n",
        "                                       xmax,\n",
        "                                       color,\n",
        "                                       font,\n",
        "                                       display_str_list=[display_str])\n",
        "            np.copyto(image, np.array(image_pil))\n",
        "        \n",
        "    return image"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r5rVOWRSgrw7"
      },
      "source": [
        "### run_detector\n",
        "\n",
        "This function will take in the object detection model `detector` and the path to a sample image, then use this model to detect objects.\n",
        "- This time, run_dtector also calls `draw_boxes` to draw the predicted bounding boxes."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "twlQG6TPgrw7"
      },
      "source": [
        "def load_img(path):\n",
        "    '''\n",
        "    Loads a JPEG image and converts it to a tensor.\n",
        "    \n",
        "    Args:\n",
        "        path (string) -- path to a locally saved JPEG image\n",
        "    \n",
        "    Returns:\n",
        "        (tensor) -- an image tensor\n",
        "    '''\n",
        "    \n",
        "    # read the file\n",
        "    img = tf.io.read_file(path)\n",
        "    \n",
        "    # convert to a tensor\n",
        "    img = tf.image.decode_jpeg(img, channels=3)\n",
        "    \n",
        "    return img\n",
        "\n",
        "\n",
        "def run_detector(detector, path):\n",
        "    '''\n",
        "    Runs inference on a local file using an object detection model.\n",
        "    \n",
        "    Args:\n",
        "        detector (model) -- an object detection model loaded from TF Hub\n",
        "        path (string) -- path to an image saved locally\n",
        "    '''\n",
        "    \n",
        "    # load an image tensor from a local file path\n",
        "    img = load_img(path)\n",
        "\n",
        "    # add a batch dimension in front of the tensor\n",
        "    converted_img  = tf.image.convert_image_dtype(img, tf.float32)[tf.newaxis, ...]\n",
        "    \n",
        "    # run inference using the model\n",
        "    start_time = time.time()\n",
        "    result = detector(converted_img)\n",
        "    end_time = time.time()\n",
        "\n",
        "    # save the results in a dictionary\n",
        "    result = {key:value.numpy() for key,value in result.items()}\n",
        "\n",
        "    # print results\n",
        "    print(\"Found %d objects.\" % len(result[\"detection_scores\"]))\n",
        "    print(\"Inference time: \", end_time-start_time)\n",
        "\n",
        "    # draw predicted boxes over the image\n",
        "    image_with_boxes = draw_boxes(\n",
        "      img.numpy(), result[\"detection_boxes\"],\n",
        "      result[\"detection_class_entities\"], result[\"detection_scores\"])\n",
        "\n",
        "    # display the image\n",
        "    display_image(image_with_boxes)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TyB3hroOgrw7"
      },
      "source": [
        "### Run the detector on your selected image!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vchaUW1XDodD"
      },
      "source": [
        "run_detector(detector, downloaded_image_path)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WUUY3nfRX7VF"
      },
      "source": [
        "### Run the detector on more images\n",
        "Perform inference on some additional images of your choice and check how long inference takes."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rubdr2JXfsa1"
      },
      "source": [
        "image_urls = [\n",
        "  # Source: https://commons.wikimedia.org/wiki/File:The_Coleoptera_of_the_British_islands_(Plate_125)_(8592917784).jpg\n",
        "  \"https://upload.wikimedia.org/wikipedia/commons/1/1b/The_Coleoptera_of_the_British_islands_%28Plate_125%29_%288592917784%29.jpg\",\n",
        "  # By Américo Toledano, Source: https://commons.wikimedia.org/wiki/File:Biblioteca_Maim%C3%B3nides,_Campus_Universitario_de_Rabanales_007.jpg\n",
        "  \"https://upload.wikimedia.org/wikipedia/commons/thumb/0/0d/Biblioteca_Maim%C3%B3nides%2C_Campus_Universitario_de_Rabanales_007.jpg/1024px-Biblioteca_Maim%C3%B3nides%2C_Campus_Universitario_de_Rabanales_007.jpg\",\n",
        "  # Source: https://commons.wikimedia.org/wiki/File:The_smaller_British_birds_(8053836633).jpg\n",
        "  \"https://upload.wikimedia.org/wikipedia/commons/0/09/The_smaller_British_birds_%288053836633%29.jpg\",\n",
        "  ]\n",
        "\n",
        "def detect_img(image_url):\n",
        "    start_time = time.time()\n",
        "    image_path = download_and_resize_image(image_url, 640, 480)\n",
        "    run_detector(detector, image_path)\n",
        "    end_time = time.time()\n",
        "    print(\"Inference time:\",end_time-start_time)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "otPnrxMKIrj5"
      },
      "source": [
        "detect_img(image_urls[0])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "H5F7DkD5NtOx"
      },
      "source": [
        "detect_img(image_urls[1])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DZ18R7dWNyoU"
      },
      "source": [
        "detect_img(image_urls[2])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CGfWk3Udp3Ql"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}