stride_demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# STRIDE: Street View-based Environmental Feature Detection and Pedestrian Collision Prediction - Tutorial\n",
    "\n",
    "<img src=\"images/STRIDE_benchmark.png\"/>\n",
    "\n",
    "Welcome to the **STRIDE** tutorial! In this tutorial, we will go through some primary usage of STRIDE, including the following:\n",
    "* Sample random points over the streets of Bogotá and other Latin American cities.\n",
    "* Download panoramic images from the Google Streetview API that correspond to the sampled points.\n",
    "* Run the STRIDE model on the downloaded images\n",
    "* Visualize the results of the model in built environment detection and pedestrian collision estimation\n",
    "\n",
    "You can run this tutorial on your server either on GPUs or CPUs, but the latter will take significantly longer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1) Install Dependencies\n",
    "\n",
    "<img src=https://miro.medium.com/v2/resize:fit:786/format:webp/1*IMGOKBIN8qkOBt5CH55NSw.png/>\n",
    "\n",
    "Your first task is to create and configure the conda environment with all the dependencies required to run the code. This includes:\n",
    "\n",
    "- **PyTorch:** Deep Learning library to handle the entire development process of Deep Neural Networks\n",
    "- **Torch Vision:** Computer Vision library with state-of-the-art computer vision architectures and functions\n",
    "- **Cuda Toolkit:** Library to handle the GPU mounting process of PyTorch models\n",
    "- **GeoPandas:** Data Handling library to process geographic and coordinates data.\n",
    "\n",
    "In this step, we will also download all necessary files to run the model. These include:\n",
    "\n",
    "- **city_coordinates.json** JSON file with information about city boundaries to sample points and plot them on maps.\n",
    "- **STRIDE_weights.pth** PyTorch file with the values of all the model's pretrained parameters (weights)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "conda install pytorch==1.9.0 torchvision==0.10.0 cudatoolkit=11.1 -c pytorch -c nvidia\n",
    "\n",
    "pip install -r requirements.txt\n",
    "\n",
    "cd models/dino/ops\n",
    "python setup.py build install\n",
    "python test.py\n",
    "cd ../../.. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2) Variable setting\n",
    "\n",
    "In this step, we will import son basic libraries and set the following base variables for the demo:\n",
    "\n",
    "- ```city``` name of the city to sample points from\n",
    "- ```num_images``` number of points and images to sample\n",
    "- ```city_coordinates_file``` name of the JSON with the boundary coordinates of each city\n",
    "- ```pretrained_model``` path to the PyTorch file with the model parameters (weights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import random\n",
    "import re\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "\n",
    "# Test city to download images\n",
    "city = \"sao paulo\"\n",
    "\n",
    "# Number of images to download\n",
    "num_images = 3\n",
    "\n",
    "# File with limit coordinates per city\n",
    "city_coordinates_file = \"city_coordinates.json\"\n",
    "\n",
    "# Path to the pretrained model parameters (weights)\n",
    "pretrained_model_file = \"pretrained_models/stride_fold1.pth\"\n",
    "\n",
    "# Path to the categories file\n",
    "categories_file = \"categories.txt\"\n",
    "\n",
    "print('')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3) Sample Points\n",
    "\n",
    "In this step, we will sample the selected number of random points in the selected city. For this purpose, we will sample random coordinates within the established boundaries of the city that will correspond to the sampled points. Then, we will plot the points on a map of the city."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import geopandas as gpd\n",
    "from shapely.geometry import Point\n",
    "import contextily as ctx\n",
    "from pyproj import Transformer\n",
    "\n",
    "#--------------- SAMPLE RANDOM POINTS \n",
    "\n",
    "\n",
    "# Load city boundary coordinates file\n",
    "with open(city_coordinates_file, 'r') as f:\n",
    "    min_max_coordinates = json.load(f)\n",
    "    \n",
    "# Get coordinates for the selected city\n",
    "min_max_coordinates = min_max_coordinates[city]\n",
    "bounds = min_max_coordinates['bounds']\n",
    "seed = min_max_coordinates['seed']\n",
    "random.seed(seed)\n",
    "\n",
    "# Establish min max coordinate boundaries\n",
    "min_lat = min_max_coordinates['latitude']['min']\n",
    "max_lat = min_max_coordinates['latitude']['max']\n",
    "\n",
    "min_lon = min_max_coordinates['longitude']['min']\n",
    "max_lon = min_max_coordinates['longitude']['max']\n",
    "\n",
    "\n",
    "# Generate random coordinates inside the boundaries\n",
    "coordinates = []\n",
    "geometry = []\n",
    "for i in range(num_images):\n",
    "    # Random nfloat generator within boundaries\n",
    "    rand_lat = random.uniform(min_lat, max_lat)\n",
    "    rand_lon = random.uniform(min_lon, max_lon)\n",
    "    coordinates.append((rand_lat, rand_lon))\n",
    "    \n",
    "    # Convert the list of coordinates into GeoDataFrame points\n",
    "    geometry.append(Point((rand_lon, rand_lat)))\n",
    "    print((rand_lat, rand_lon))\n",
    "\n",
    "#--------------- PLOT POINTS ON MAP\n",
    "\n",
    "# Create a GeoDataFrame from the sampled points\n",
    "gdf_points = gpd.GeoDataFrame(geometry=geometry, crs=\"EPSG:4326\")\n",
    "\n",
    "# Convert the coordinates to the web mercator projection (EPSG:3857) for contextily\n",
    "gdf_points = gdf_points.to_crs(epsg=3857)\n",
    "\n",
    "# Plot the points on a street map of the city\n",
    "fig, ax = plt.subplots(figsize=(10, 10))\n",
    "gdf_points.plot(ax=ax, color='red', markersize=100, alpha=0.6)\n",
    "\n",
    "# Set the extent to the entire Bogotá area\n",
    "ax.set_xlim([bounds['minx'], bounds['maxx']]) \n",
    "ax.set_ylim([bounds['miny'], bounds['maxy']]) \n",
    "# ax.set_xlim([-5210000, -5170000]) \n",
    "# ax.set_ylim([-2730000, -2680000]) \n",
    "\n",
    "# Add a basemap from contextily (OpenStreetMap or similar)\n",
    "ctx.add_basemap(ax, source=ctx.providers.OpenStreetMap.Mapnik)\n",
    "\n",
    "# Convert back to latitude and longitude for correct axis labels\n",
    "# Create a transformer to go from Web Mercator (EPSG:3857) to WGS 84 (EPSG:4326)\n",
    "transformer = Transformer.from_crs(\"EPSG:3857\", \"EPSG:4326\", always_xy=True)\n",
    "\n",
    "# Transform x and y axis limits back to lat/lon\n",
    "min_lon, min_lat = transformer.transform(bounds['minx'], bounds['miny'])\n",
    "max_lon, max_lat = transformer.transform(bounds['maxx'], bounds['maxy'])\n",
    "\n",
    "# Set the ticks for latitude and longitude\n",
    "ax.set_xticks([bounds['minx'], (bounds['minx'] + bounds['maxx']) / 2, bounds['maxx']])\n",
    "ax.set_xticklabels([f\"{min_lon:.2f}°\", f\"{(min_lon + max_lon) / 2:.2f}°\", f\"{max_lon:.2f}°\"])\n",
    "\n",
    "ax.set_yticks([bounds['miny'], (bounds['miny'] + bounds['maxy']) / 2, bounds['maxy']])\n",
    "ax.set_yticklabels([f\"{min_lat:.2f}°\", f\"{(min_lat + max_lat) / 2:.2f}°\", f\"{max_lat:.2f}°\"])\n",
    "\n",
    "# Add titles and labels\n",
    "plt.title(f'Coordinates in {city}')\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4) Get Metadata of Panoramic Image Candidates\n",
    "\n",
    "<img src=\"https://lh3.googleusercontent.com/beHNDtluUwK6HO2MAcsEjo_K5fgAOR4G15SzFYkAE9nVc66hgeZw0of5YtVW1X-HHx6PZHgXlUlMFA2Ss0VxsST3jUOWnafszFXgpVO4gs92bnmylQ\"/>\n",
    "\n",
    "In this step, we will get the data for the best panoramic image for each point. For each point, we will follow the following order: \n",
    "\n",
    "1. Send a request to the Google Streetview API to get a list of possible panoramic images that are close to the point.\n",
    "2. Reformat the API response into dictionaries with the Panorama ID, latitude, longitude, and date of the candidates.\n",
    "3. Find the candidate closest to the sampled point using the Haversine distance.\n",
    "\n",
    "This part of the demo has been adapted from the [streetview](https://github.com/robolyst/streetview) library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import numpy as np\n",
    "from haversine import haversine, Unit\n",
    "\n",
    "# List of panorama\n",
    "coordinate_panos = []\n",
    "\n",
    "for lat,lon in coordinates:\n",
    "    # URL for the GET request\n",
    "    url = (\n",
    "            \"https://maps.googleapis.com/maps/api/js/\" # Google API url\n",
    "            \"GeoPhotoService.SingleImageSearch\" # Image search engine\n",
    "            \"?pb=!1m5!1sapiv3!5sUS!11m2!1m1!1b0!2m4!1m2!3d{0:}!4d{1:}!2d50!3m10\" # searching code\n",
    "            \"!2m2!1sen!2sGB!9m1!1e2!11m4!1m3!1e2!2b1!3e2!4m10!1e1!1e2!1e3!1e4\"\n",
    "            \"!1e8!1e6!5m1!1e2!6m1!1e2\"\n",
    "            \"&callback=callbackfunc\"\n",
    "        )\n",
    "    \n",
    "    # Insert latitud and longitud into the request url\n",
    "    url = url.format(lat, lon)\n",
    "    print(url)\n",
    "    \n",
    "    # GET request to the Google API to search for close images\n",
    "    resp = requests.get(url).text\n",
    "    \n",
    "    # Re format the response into a dictionary\n",
    "    blob = re.findall(r\"callbackfunc\\( (.*) \\)$\", resp)[0]\n",
    "    data = json.loads(blob)\n",
    "\n",
    "    # Handle empty responses\n",
    "    if data == [[5, \"generic\", \"Search returned no images.\"]]:\n",
    "        panos = []\n",
    "\n",
    "    subset = data[1][5][0]\n",
    "\n",
    "    raw_panos = subset[3][0]\n",
    "\n",
    "    if len(subset) < 9 or subset[8] is None:\n",
    "        raw_dates = []\n",
    "    else:\n",
    "        raw_dates = subset[8]\n",
    "\n",
    "    # For some reason, dates do not include a date for each panorama.\n",
    "    # the n dates match the last n panos. Here we flip the arrays\n",
    "    # so that the 0th pano aligns with the 0th date.\n",
    "    raw_panos = raw_panos[::-1]\n",
    "    raw_dates = raw_dates[::-1]\n",
    "\n",
    "    dates = [f\"{d[1][0]}-{d[1][1]:02d}\" for d in raw_dates]\n",
    "    \n",
    "\n",
    "    first_panorama_data = None\n",
    "    min_haversine = float('inf')\n",
    "    \n",
    "    # Iterate over all image candidates and reformat into dictionary\n",
    "    print(f'Available Panoramas for coordinates {(lat,lon)}:')\n",
    "    for i, pano in enumerate(raw_panos):\n",
    "        this_panorama = {\n",
    "                        \"pano_id\": pano[0][1],\n",
    "                        \"latitude\": pano[2][0][2],\n",
    "                        \"longitude\": pano[2][0][3],\n",
    "                        \"date\": dates[i] if i < len(dates) else None,\n",
    "                        # # One could also retrive data about image capturing like heading, pitch, roll, etc.\n",
    "                        # \"heading\":pano[2][2][0],\n",
    "                        # \"pitch\":pano[2][2][1] if len(pano[2][2]) >= 2 else None,\n",
    "                        # \"roll\":pano[2][2][2] if len(pano[2][2]) >= 3 else None,\n",
    "                        # \"date\":dates[i] if i < len(dates) else None,\n",
    "                        # \"elevation\":pano[3][0] if len(pano) >= 4 else None,\n",
    "                        } \n",
    "        print(i+1,this_panorama)\n",
    "\n",
    "        # Check if it the candidate is the closest one\n",
    "        this_distance = haversine((lat,lon), (this_panorama['latitude'], this_panorama['longitude']), unit=Unit.METERS)\n",
    "        if this_distance<min_haversine:\n",
    "            first_panorama_data = this_panorama\n",
    "            min_haversine=this_distance\n",
    "    \n",
    "    coordinate_panos.append(first_panorama_data)\n",
    "    print('Closes Panorama is:')\n",
    "    print(first_panorama_data)\n",
    "    print(f\"Distance to point:\\t{min_haversine:.0f}m\\n\")\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5) Download Panoramic Images\n",
    "\n",
    "<img src=\"https://kstatic.googleusercontent.com/files/0df80ab0ad7cc9a3d4a19b068da05a98f4a17ec4182738214b1bd8ac8263ab4a965de25694a9e4220c627ec3016bfce7b5cd3aa0b6afc1eb5a24d25ed2a3a788\"/>\n",
    "\n",
    "In this step, we will download the panoramic images whose metadata we selected in the previous step. Remember each image corresponds to a sampled point. For each image/point, we will follow the following order:\n",
    "\n",
    "1. Establish image size according to the desired zoom of the image\n",
    "2. Iteratively send requests to the Google Streetview API for each part of the image (parts or tiles must request images) and stitch them all together in the process\n",
    "3. Preprocess the image to remove empty zones caused by the download process.\n",
    "4. Preprocess the image to remove a fraction of its top and bottom to remove image artifacts caused by panoramic image creation.\n",
    "5. Plot the image to visualize it.\n",
    "6. Keep the exact coordinates of the image, as they might be slightly far from the point, and the model requires the coordinates of the input image.\n",
    "\n",
    "This part of the demo has also been adapted from the [streetview](https://github.com/robolyst/streetview) library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from io import BytesIO\n",
    "import itertools\n",
    "from PIL import Image\n",
    "from tqdm import tqdm\n",
    "import matplotlib as mpl\n",
    "mpl.rcParams['figure.dpi'] = 400\n",
    "\n",
    "# Declare lists of images and image coordinates\n",
    "panoramas = []\n",
    "pano_coordinates = []\n",
    "\n",
    "# Declare constant variables for image gathering, formation and preprocessing\n",
    "zoom = 5\n",
    "crop_height = 4000\n",
    "typical_height = 6656\n",
    "typical_width = 13312\n",
    "crop_index = 1000\n",
    "\n",
    "# Iterate over selected panorama metadata\n",
    "for main_panorama in coordinate_panos:\n",
    "    tile_width = 512\n",
    "    tile_height = 512\n",
    "    \n",
    "    # Define image size according to zoom\n",
    "    width, height = 2**zoom, 2 ** (zoom - 1)\n",
    "\n",
    "    # Create empty image variable to be filled with the downloaded tiles\n",
    "    panorama = Image.new(\"RGB\", (width * tile_width, height * tile_height))\n",
    "    \n",
    "    # Itearte over the x,y coordinates of each tile of the image to download\n",
    "    for x, y in tqdm(itertools.product(range(width), range(height)), total=width*height, desc=f\"Downloading and stitching {main_panorama['pano_id']}\"):\n",
    "        \n",
    "        # URL for the GET request of the corresponding tile\n",
    "        tile_url = f\"https://cbk0.google.com/cbk?output=tile&panoid={main_panorama['pano_id']}&zoom={zoom}&x={x}&y={y}\"\n",
    "        \n",
    "        # Send GET request to the Google Streetview API\n",
    "        response = requests.get(tile_url, stream=True)\n",
    "\n",
    "        # Reformat response as an image corresponding to the tile\n",
    "        tile = Image.open(BytesIO(response.content))\n",
    "\n",
    "        # Paste the tile on the image\n",
    "        panorama.paste(im=tile, box=(x * tile_width, y * tile_height))\n",
    "\n",
    "        # Delete tile to save space\n",
    "        del tile\n",
    "    \n",
    "    # Remove empty parts of the image left after the filling process\n",
    "    pano_width, pano_height = panorama.size\n",
    "    if pano_width>typical_width:\n",
    "        panorama = panorama.crop(box=(0, 0, typical_width, pano_height))\n",
    "        pano_width, pano_height = panorama.size\n",
    "    if pano_height>typical_height:\n",
    "        panorama = panorama.crop(box=(0, 0, pano_width, typical_height))\n",
    "        pano_width, pano_height = panorama.size\n",
    "    \n",
    "    # Crop 1000 pixels from the top of the image and 1656 from the bottom to remove artifacts from panoramic omage formation\n",
    "    if pano_height>crop_height:\n",
    "        panorama = panorama.crop(box=(0, crop_index, pano_width, crop_height + crop_index))\n",
    "    \n",
    "    # Plot image\n",
    "    plt.imshow(np.array(panorama))\n",
    "    plt.title(f\"Panorama ID: {main_panorama['pano_id']}\")\n",
    "    plt.axis('off')\n",
    "    plt.show()\n",
    "    plt.close()\n",
    "\n",
    "    # Append image and coordinates to lists\n",
    "    panoramas.append(panorama) \n",
    "    pano_coordinates.append((main_panorama['latitude'], main_panorama['longitude']))   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6) Import and Build the Model\n",
    "\n",
    "<img src=\"https://github.com/BCV-Uniandes/STRIDE/blob/main/images/STRIDE_baseline.png?raw=true\"/>\n",
    "In this step, we will build the model using the predefined classes in the STRIDE repo. First, we load a configuration file containing each hyperparameter's values regarding the model's design and architecture. Then, we will use these configurations to build the model using the following four modules:\n",
    "\n",
    "1. ```Backbones``` This module builds the CNN backbone that initially processes the image to extract visual features\n",
    "2. ```PositionEmbeddingSineHW``` This module uses a sine function to include information about the 2D position of the extracted visual features\n",
    "3. ```DeformableTransformer``` This module builds the Transformer that processes and compares the visual features with the added positional encoding and the object queries\n",
    "4. ```DINO``` This is the general module of the entire model that handles the model workflow, puts together all the parts, processes all outputs and inputs of each part, and generates final boxes and their classes.\n",
    "5. ```postprocess``` This module cleans the prox predictions by removing empty boxes and scaling them to the original size\n",
    "\n",
    "As you can see, all of these modules have multiple arguments that correspond to the model design's hyperparameters. The values of all these arguments and parameters are coded in the configuration file.\n",
    "\n",
    "Our model is entirely based on the [DINO](https://github.com/IDEA-Research/DINO) model and code. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from util.slconfig import SLConfig\n",
    "from models.dino.dino import DINO, PostProcess\n",
    "from models.dino.deformable_transformer import DeformableTransformer\n",
    "from models.dino.backbone import Backbone, FrozenBatchNorm2d, Joiner\n",
    "from models.dino.position_encoding import PositionEmbeddingSineHW\n",
    "\n",
    "# Hardcodded path to the configuration script\n",
    "config_file = \"./config/STRIDE/STRIDE_4scale.py\"\n",
    "\n",
    "# Parse configuration variables into atributes of a Config object\n",
    "cfg = SLConfig.fromfile(config_file)\n",
    "\n",
    "# Configura and build the Sine positional embedding\n",
    "N_steps = cfg.hidden_dim // 2\n",
    "position_embedding = PositionEmbeddingSineHW(N_steps, \n",
    "                        temperatureH=cfg.pe_temperatureH,\n",
    "                        temperatureW=cfg.pe_temperatureW,\n",
    "                        normalize=True\n",
    "                    )\n",
    "\n",
    "# Configura and build the CNN backbone\n",
    "backbone = Backbone(cfg.backbone, cfg.lr_backbone > 0, cfg.dilation,   \n",
    "                    cfg.return_interm_indices,   \n",
    "                    batch_norm=FrozenBatchNorm2d,\n",
    "                    regression=cfg.regression and cfg.backbone_reg,\n",
    "                    out_feats=cfg.regression and cfg.backbone_out,\n",
    "            )\n",
    "\n",
    "# Join the CNN backbone with the Sine embedding\n",
    "bb_num_channels = backbone.num_channels\n",
    "backbone = Joiner(backbone, position_embedding)\n",
    "backbone.num_channels = bb_num_channels \n",
    "\n",
    "# Configura and build the Detection Transformer\n",
    "transformer = DeformableTransformer(\n",
    "        d_model=cfg.hidden_dim,\n",
    "        dropout=cfg.dropout,\n",
    "        nhead=cfg.nheads,\n",
    "        num_queries=cfg.num_queries,\n",
    "        dim_feedforward=cfg.dim_feedforward,\n",
    "        num_encoder_layers=cfg.enc_layers,\n",
    "        num_unicoder_layers=cfg.unic_layers,\n",
    "        num_decoder_layers=cfg.dec_layers,\n",
    "        normalize_before=cfg.pre_norm,\n",
    "        return_intermediate_dec=True,\n",
    "        query_dim=cfg.query_dim,\n",
    "        activation=cfg.transformer_activation,\n",
    "        num_patterns=cfg.num_patterns,\n",
    "        modulate_hw_attn=True,\n",
    "\n",
    "        deformable_encoder=True,\n",
    "        deformable_decoder=True,\n",
    "        num_feature_levels=cfg.num_feature_levels,\n",
    "        enc_n_points=cfg.enc_n_points,\n",
    "        dec_n_points=cfg.dec_n_points,\n",
    "        use_deformable_box_attn=cfg.use_deformable_box_attn,\n",
    "        box_attn_type=cfg.box_attn_type,\n",
    "\n",
    "        learnable_tgt_init=True,\n",
    "        decoder_query_perturber=None,\n",
    "\n",
    "        add_channel_attention=cfg.add_channel_attention,\n",
    "        add_pos_value=cfg.add_pos_value,\n",
    "        random_refpoints_xy=cfg.random_refpoints_xy,\n",
    "\n",
    "        # two stage\n",
    "        two_stage_type=cfg.two_stage_type,  # ['no', 'standard', 'early']\n",
    "        two_stage_pat_embed=cfg.two_stage_pat_embed,\n",
    "        two_stage_add_query_num=cfg.two_stage_add_query_num,\n",
    "        two_stage_learn_wh=cfg.two_stage_learn_wh,\n",
    "        two_stage_keep_all_tokens=cfg.two_stage_keep_all_tokens,\n",
    "        dec_layer_number=cfg.dec_layer_number,\n",
    "        rm_self_attn_layers=None,\n",
    "        key_aware_type=None,\n",
    "        layer_share_type=None,\n",
    "\n",
    "        rm_detach=None,\n",
    "        decoder_sa_type=cfg.decoder_sa_type,\n",
    "        module_seq=cfg.decoder_module_seq,\n",
    "\n",
    "        embed_init_tgt=cfg.embed_init_tgt,\n",
    "        use_detached_boxes_dec_out=cfg.use_detached_boxes_dec_out\n",
    "    )\n",
    "\n",
    "# Configura and build the STRIDE model (based on the DINO model) that puts all parts together\n",
    "model = DINO(\n",
    "        backbone,\n",
    "        transformer,\n",
    "        num_classes=cfg.num_classes,\n",
    "        num_queries=cfg.num_queries,\n",
    "        aux_loss=True,\n",
    "        iter_update=True,\n",
    "        query_dim=4,\n",
    "        random_refpoints_xy=cfg.random_refpoints_xy,\n",
    "        fix_refpoints_hw=cfg.fix_refpoints_hw,\n",
    "        num_feature_levels=cfg.num_feature_levels,\n",
    "        nheads=cfg.nheads,\n",
    "        dec_pred_class_embed_share=cfg.dec_pred_class_embed_share,\n",
    "        dec_pred_bbox_embed_share=cfg.dec_pred_bbox_embed_share,\n",
    "        # two stage\n",
    "        two_stage_type=cfg.two_stage_type,\n",
    "        # box_share\n",
    "        two_stage_bbox_embed_share=cfg.two_stage_bbox_embed_share,\n",
    "        two_stage_class_embed_share=cfg.two_stage_class_embed_share,\n",
    "        decoder_sa_type=cfg.decoder_sa_type,\n",
    "        num_patterns=cfg.num_patterns,\n",
    "        dn_number = cfg.dn_number if cfg.use_dn else 0,\n",
    "        dn_box_noise_scale = cfg.dn_box_noise_scale,\n",
    "        dn_label_noise_ratio = cfg.dn_label_noise_ratio,\n",
    "        dn_labelbook_size = cfg.dn_labelbook_size,\n",
    "        # Regression\n",
    "        regression=cfg.regression,\n",
    "        reg_encoder=cfg.reg_encoder,\n",
    "        backbone_reg=cfg.backbone_reg,\n",
    "        backbone_out=cfg.backbone_out,\n",
    "        reg_cls_embed=cfg.reg_cls_token,\n",
    "        just_backbone=cfg.just_backbone,\n",
    "        norm_funct=cfg.norm_funct,\n",
    "        denoise=cfg.denoise,\n",
    "        no_loss_norm=cfg.no_loss_norm\n",
    "    )\n",
    "\n",
    "# Configura and build the postprocessor\n",
    "postprocessor = PostProcess(num_select=cfg.num_select, nms_iou_threshold=cfg.nms_iou_threshold)\n",
    "\n",
    "print('Model Loaded')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7) Load Model on GPU and Load Parameters\n",
    "\n",
    "<img src=\"https://i.blogs.es/57ca96/cuda/450_1000.webp\"/>\n",
    "\n",
    "In this step, we will first load the model in our GPU (if available). When we build the model, this one is, by default, built on your computers' processors (the CPUs). In order to make the model run faster, we should store it in our GPUs.\n",
    "\n",
    "Then, we will load the model's trained parameters. When we built the model, it was initialized with random parameters; in this step, we will load the PyTorch file with the trained parameters of the model, and then we will update the previously built model with the ones that were just loaded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Device: cuda (GPU) if it is available or cpu otherwise\n",
    "device = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
    "\n",
    "# Mount model in the GPU\n",
    "model.to(device)\n",
    "\n",
    "# Load PyTorch file with the model weights\n",
    "checkpoint = torch.load(pretrained_model_file, map_location='cpu')['model']\n",
    "\n",
    "# Load the model weights into the model\n",
    "_load_output = model.load_state_dict(checkpoint, strict=False)\n",
    "print(_load_output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8) Run Model on Images and Visualize Predictions\n",
    "\n",
    "In this final step, we will iteratively run the model on the images and visualize its outputs. This iterative approach will help us refine our model and improve its performance. To do so, we will follow the following order for each image:\n",
    "\n",
    "1. Preprocess images by downscaling them to fit in the GPU memory, normalizing their values, and parsing them into tensors.\n",
    "2. Input images and their coordinates into the models.\n",
    "3. Postprocess outputs into scaled bounding boxes.\n",
    "4. Filter low-score bounding boxes.\n",
    "5. Draw bounding boxes on top of images to visualize output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import datasets.transforms as T\n",
    "from util.misc import nested_tensor_from_tensor_list\n",
    "import matplotlib.patches as patches\n",
    "import matplotlib.cm as cm\n",
    "\n",
    "# Load categories names\n",
    "with open(categories_file, 'r') as f:\n",
    "    categories = [l.strip() for l in f]\n",
    "\n",
    "# Establish color map to draw boxes\n",
    "cmap = cm.get_cmap('gist_rainbow', 27)\n",
    "\n",
    "# Built image preprocessing (resizing and normalization) function\n",
    "image_transform = T.Compose([\n",
    "                    T.RandomResize([cfg.img_size]),\n",
    "                    T.ToTensor(),\n",
    "                    T.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])\n",
    "                ])\n",
    "\n",
    "# Turn down all backpropagation attributes\n",
    "with torch.no_grad():\n",
    "    # SEt the model on inference mode and turn off any random layers\n",
    "    model.eval()\n",
    "\n",
    "    # Iterate over images and their coordinates\n",
    "    for panorama, pano_coords in tqdm(zip(panoramas, pano_coordinates), total=len(panoramas), desc=\"Running inference on images.\"):\n",
    "        w, h = panorama.size\n",
    "\n",
    "        # Preprocess image (resize and normalize)\n",
    "        processed_panorama = image_transform(panorama, {})[0]\n",
    "\n",
    "        # Parse images into nested tensorss\n",
    "        processed_panorama = nested_tensor_from_tensor_list([processed_panorama]).to(device)\n",
    "        \n",
    "        # Run the model\n",
    "        outputs = model(samples=processed_panorama, \n",
    "                        targets=[{'latitude': torch.tensor([pano_coords[0]]).to(device), 'longitude': torch.tensor([pano_coords[1]]).to(device)}])\n",
    "        \n",
    "        # Postprocess outputs\n",
    "        orig_target_sizes = torch.as_tensor([[int(h), int(w)]]).to(device)\n",
    "        results = postprocessor(outputs, orig_target_sizes)[0]\n",
    "        \n",
    "        # Extract all information about predictions\n",
    "        collitions = results[\"regression\"]\n",
    "        boxes = results[\"boxes\"]\n",
    "        labels = results['labels'].tolist()\n",
    "        scores = results['scores'].tolist()\n",
    "        \n",
    "        # Change bounding boxes format\n",
    "        xmin, ymin, xmax, ymax = boxes.unbind(1)\n",
    "        boxes = torch.stack((xmin, ymin, xmax - xmin, ymax - ymin), dim=1).tolist()        \n",
    "        \n",
    "        # Reformat results into a single list filteirng those with less than 0.25 score\n",
    "        results = [\n",
    "                        {\n",
    "                            \"category_id\": label,\n",
    "                            \"bbox\": box,\n",
    "                            \"score\": score,\n",
    "                        }\n",
    "                        for box,label,score in zip(boxes, labels, scores)\n",
    "                        if score >= 0.25\n",
    "                    ]\n",
    "        \n",
    "        # Take top 300 predictions\n",
    "        results.sort(key=lambda x: x['score'], reverse=True)\n",
    "        results = results[:300]\n",
    "        \n",
    "        \n",
    "        # Plot the image\n",
    "        fig, ax = plt.subplots(1)\n",
    "        ax.imshow(panorama)\n",
    "        plt.axis('off')\n",
    "        plt.title(f'Estimated collisions: {collitions[0].item()}')\n",
    "        \n",
    "        for box in results:\n",
    "            category = box['category_id']\n",
    "            bbox = box['bbox']\n",
    "            score = box['score']\n",
    "            \n",
    "            # Unpack bounding box coordinates\n",
    "            x_min, y_min, b_w, b_h = bbox\n",
    "                        \n",
    "            # Create a rectangle patch\n",
    "            rect = patches.Rectangle((x_min, y_min), b_w, b_h, linewidth=0.5, edgecolor=cmap((category-1)/27), facecolor='none')\n",
    "            \n",
    "            # Add the rectangle to the plot\n",
    "            ax.add_patch(rect)\n",
    "            \n",
    "            # Add the category and score label\n",
    "            label = f'{categories[category-1]}: {score:.2f}'\n",
    "            ax.text(x_min, y_min, label, color='black', fontsize=1, \n",
    "                    bbox=dict(facecolor='white', alpha=0.4, pad=0, edgecolor=cmap((category-1)/27)))\n",
    "        \n",
    "        # Display the result\n",
    "        plt.show()\n",
    "        plt.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}