Object detection

[shimonfrancis]

Hi Im new for zen3geo, can we do object detection using geopackage polygons on multiple raster images as i can see you have given an example for just one raster image. Do you have examples for multiple rasters?

[weiji14]

Yes, you can definitely use multiple raster images. I have a very complicated pipeline here (see also my AGU2022 poster) where I applied it to several Sentinel-2 images over Australia for an object detection task using a YOLOX model.

As mentioned at https://zen3geo.readthedocs.io/en/v0.6.2/object-detection-boxes.html#create-128x128-raster-chips-and-clip-vector-geometries-with-it, you'll need to ensure that you have:

• N many raster images/chips (which could be of different projection systems), and
• Only 1 'global' geodataframe (that can contain many polygons from a geopackage)

Let me know if you need help with anything 😃

[weiji14]

Hi @shimonfrancis, sorry, that's the only example I have for object detection (unfortunately, object detection is not very common in geospatial ML compared to segmentation/classification). Happy to provide some general advice though if you have a particular raster + vector dataset in mind that you'd like to build a pipeline for.

[shimonfrancis]

Object detection-Task.zip
Hi @weiji14 Thankyou for you response. Im here by attaching my Raster and vector data along with Datamodule I have created. Please review those and let me know whatever Im missing.Thanks.

[weiji14]

Sorry for the late reply. I had a look at your code, and ran into an AttributeError: 'DataArray' object has no attribute 'to_array'. You can fix it by changing this line in the boximg_collate_fn function:

- torch.as_tensor(data=sample[1].to_array().data.astype(dtype="float32"))
+ torch.as_tensor(data=sample[1].data.astype(dtype="float32"))

With that, I was able to run the for item in module.datapipe: loop at least:

images length: 2
sliced images length: 65
vector length: 1
clipped length: 65
clipped shuffled length: 65
polygon to bbox length: 65
vector boxes length: 65
Batch length: 33
collate length 33
length of self.datapipe: 33
{'image': tensor([[[[ 96.,  96.,  96.,  ...,  87.,  86., 102.],
          [ 95.,  95.,  95.,  ...,  89.,  87., 107.],
          [ 95.,  95.,  95.,  ...,  86.,  84., 102.],
          ...,
          [ 57.,  55.,  54.,  ...,  21.,  21.,  18.],
          [ 58.,  56.,  54.,  ...,  22.,  22.,  17.],
          [ 54.,  55.,  55.,  ...,  26.,  26.,  23.]],

         [[ 95.,  95.,  95.,  ..., 120., 119., 135.],
          [ 94.,  94.,  94.,  ..., 122., 120., 140.],
          [ 94.,  94.,  94.,  ..., 119., 117., 135.],
          ...,
          [ 94.,  92.,  91.,  ...,   4.,   4.,   1.],
          [ 95.,  93.,  91.,  ...,   5.,   5.,   0.],
          [ 85.,  83.,  83.,  ...,   8.,   8.,   5.]],

         [[111., 111., 111.,  ..., 101., 100., 116.],
          [112., 112., 112.,  ..., 103., 101., 121.],
          [112., 112., 112.,  ..., 100.,  98., 116.],
          ...,
          [ 76.,  74.,  73.,  ...,  10.,  10.,   7.],
          [ 77.,  75.,  73.,  ...,  11.,  11.,   6.],
          [ 69.,  68.,  68.,  ...,   8.,   8.,   5.]]],


        [[[ 56.,  56.,  56.,  ...,  66.,  66.,  66.],
          [ 56.,  56.,  56.,  ...,  66.,  66.,  66.],
          [ 56.,  56.,  56.,  ...,  66.,  66.,  66.],
          ...,
          [ 74.,  74.,  74.,  ...,  94.,  95.,  97.],
          [ 69.,  70.,  73.,  ...,  96.,  97.,  98.],
          [ 73.,  73.,  77.,  ...,  96.,  97.,  97.]],

         [[101., 101., 101.,  ..., 110., 110., 110.],
          [101., 101., 101.,  ..., 110., 110., 110.],
          [101., 101., 101.,  ..., 110., 110., 110.],
          ...,
          [113., 113., 113.,  ...,  93.,  94.,  96.],
          [108., 109., 112.,  ...,  95.,  96.,  97.],
          [112., 112., 116.,  ...,  95.,  96.,  96.]],

         [[ 80.,  80.,  80.,  ...,  83.,  83.,  83.],
          [ 80.,  80.,  80.,  ...,  83.,  83.,  83.],
          [ 80.,  80.,  80.,  ...,  83.,  83.,  83.],
          ...,
          [ 92.,  92.,  92.,  ..., 109., 110., 112.],
          [ 87.,  88.,  91.,  ..., 111., 112., 113.],
          [ 91.,  91.,  95.,  ..., 111., 112., 112.]]]]), 'bboxes': [tensor([[286.9845,  94.7109, 540.0000, 540.0000]]), tensor([], size=(0, 4))], 'crs': [CRS.from_epsg(3857), CRS.from_epsg(3857)], 'bbox': [BoundingBox(left=-389703.69248474506, bottom=7289079.168216616, right=-389663.38549875707, top=7289119.474322141), BoundingBox(left=-389743.99947073305, bottom=7289119.474322142, right=-389703.69248474506, top=7289159.780427667)]}

Let me know if that works out for you. Your data seems to be organized nicely (single geopackage file, many GeoTIFF files), so should be fine if you get the code to run.

P.S. I'll be busy attending a conference next week, so might be slow to respond, but could try and get back to you before Christmas break.

[shimonfrancis]

Hi,Sorry for the late reply and Thankyou for your time @weiji14 , let me change these and try running.

[shimonfrancis]

Hi @weiji14 , I have created a code for object detection using your examples:

import geopandas as gpd
import rioxarray
import torchdata
import pandas as pd
import os
import glob
from zen3geo.datapipes import XbatcherSlicer 
import xarray as xr
import numpy as np
import torch
from sklearn.model_selection import train_test_split
import rasterio

# Pre-load GeoTIFF images from the folder
geotiff_folder = r"/kaggle/input/construction/Kaggle_Construction"
image_files = sorted(glob.glob(f"{geotiff_folder}/*.tif", recursive=True))[:4]

# Read each GeoTIFF image into an xarray.DataArray
dp_images = []
for image_file in image_files:
    _dp_image = rioxarray.open_rasterio(image_file)
    dp_images.append(_dp_image)

# Load the GeoPackage with bounding boxes (polygons)
gdf = gpd.read_file(filename=r"/kaggle/input/construction/Kaggle_Construction/Total_Geopackage.gpkg")
gdf = gdf.geometry.apply(lambda geom: geom.simplify(tolerance=0.001))

# Ensure CRS consistency
if gdf.crs != dp_images[0].rio.crs:
    gdf = gdf.to_crs(dp_images[0].rio.crs)
num_images = len(dp_images)
train_size = int(0.8 * num_images)  # 80% for training
val_size = num_images - train_size
# Use sklearn's train_test_split to shuffle and split the data
train_images, val_images = train_test_split(dp_images, test_size=0.2, random_state=42)

# Create a DataPipe to iterate over the list of images
dp_images_pipe_train = torchdata.datapipes.iter.IterableWrapper(train_images)
dp_images_pipe_val = torchdata.datapipes.iter.IterableWrapper(val_images)

# Slice images with xbatcher to get chips of fixed size
dp_xbatcher_train = dp_images_pipe_train.slice_with_xbatcher(input_dims={'y': 512, 'x': 512})
dp_xbatcher_val = dp_images_pipe_val.slice_with_xbatcher(input_dims={'y': 512, 'x': 512})

# Wrap GeoDataFrame in a DataPipe
dp_vector = torchdata.datapipes.iter.IterableWrapper(iterable=[gdf])

# Clip vector geometries with the raster image extent
dp_clipped_train = dp_vector.clip_vector_with_rectangle(mask_datapipe=dp_xbatcher_train)
dp_clipped_val = dp_vector.clip_vector_with_rectangle(mask_datapipe=dp_xbatcher_val)

# Shuffle the results
dp_shuffle_train = dp_clipped_train.shuffle()
dp_shuffle_val = dp_clipped_val.shuffle()

def chip_no_bbox(chip):
    """
    Check if the chip contains any bounding boxes.

    Returns True if no bounding boxes are found, False otherwise.
    """
    geodataframe, dataset = chip
    
    # Ensure there are no empty geometries
    no_boxes = ~geodataframe.geometry.is_empty.all()  # True if all geometries are empty
    
    return bool(no_boxes)  # Convert numpy.bool_ to native Python bool

# Filter chips that contain no bounding boxes
dp_filtered_train = dp_shuffle_train.filter(filter_fn=chip_no_bbox)
dp_filtered_val = dp_shuffle_val.filter(filter_fn=chip_no_bbox)

def polygon_to_bbox(geom_and_chip):
    """
    Convert polygon geometries to bounding boxes.
    """
    gdf, chip = geom_and_chip
    bounds = gdf.bounds
    return bounds, chip

# Convert polygons to bounding boxes
dp_bbox_train = dp_filtered_train.map(fn=polygon_to_bbox)
dp_bbox_val = dp_filtered_val.map(fn=polygon_to_bbox)

def geobox_to_imgbox(bbox_and_chip):
    """
    Convert bounding boxes from geographic coordinates to image coordinates.
    """
    geobox, chip = bbox_and_chip
    x_res, y_res = chip.rio.resolution()
    left, bottom, right, top = chip.rio.bounds()
    
    # Calculate image bounding box coordinates
    imgbox = pd.DataFrame()
    imgbox["x1"] = (geobox.minx - left) / x_res
    imgbox["y1"] = (top - geobox.maxy) / -y_res
    imgbox["x2"] = (geobox.maxx - left) / x_res
    imgbox["y2"] = (top - geobox.miny) / -y_res
    
    return imgbox, chip

# Convert geographical bounding boxes to image coordinates
dp_ibox_train = dp_bbox_train.map(fn=geobox_to_imgbox)
dp_ibox_val = dp_bbox_val.map(fn=geobox_to_imgbox)

# Batch the data (e.g., batch size of 8)
dp_batch_train = dp_ibox_train.batch(batch_size=64, drop_last=False)
dp_batch_val = dp_ibox_val.batch(batch_size=64, drop_last=False)

def boximg_collate_fn(samples):
            box_tensors = [
                torch.as_tensor(sample[0].to_numpy(dtype=np.float32))
                for sample in samples
            ]

            #box_tensors = [
                #torch.as_tensor(sample[0].to_numpy(dtype=np.float32).reshape(-1, 4))
                #for sample in samples
            #]


            label_tensors = [
             torch.ones(len(box), dtype=torch.int64)
             for box in box_tensors
            ]

            tensors = [
                torch.as_tensor(data=sample[1].data.astype(dtype="float32"))/255.0
                for sample in samples
            ]
            img_tensors = torch.stack(tensors=tensors)

            crses = [sample[1].rio.crs for sample in samples]
            extents = [
                rasterio.coords.BoundingBox(*sample[1].rio.bounds())
                for sample in samples
            ]

            return {
                "image": img_tensors,
                "boxes": box_tensors,
                "labels": label_tensors,
                #"crs": crses,
                #"bbox": extents,
            }

# Apply the collate function to the batches
dp_collate_train = dp_batch_train.collate(collate_fn=boximg_collate_fn)
dp_collate_val = dp_batch_val.collate(collate_fn=boximg_collate_fn)

# Create a DataLoader for batching
train_dataloader = torch.utils.data.DataLoader(dp_collate_train)
val_dataloader=torch.utils.data.DataLoader(dp_collate_val)

If i iterate dataloaders to view the keys etc, it takes time. Do you have any suggestions for me to increase the efficiency for dataloaders?, I think training too can take more time from 1 epoch to the next. Thanks.