DetectionMetrics

Unified evaluation for perception models

Project webpage here

⚠️ DetectionMetrics v1 website referenced in our Sensors paper is still available here

DetectionMetrics is a family of toolkits designed to unify and streamline the evaluation of perception models across different frameworks and datasets. Looking for our published DetectionMetrics v1? Check out all the relevant links below.

Now, we're excited to introduce DetectionMetrics v2! While retaining the flexibility of our previous release, DetectionMetrics has been redesigned with an expanded focus on image and LiDAR segmentation. As we move forward, v2 will be the actively maintained version, featuring continued updates and enhancements to keep pace with evolving AI and computer vision technologies.

💻 Code	🔧 Installation	🧩 Compatibility	📖 Docs

What's supported in DetectionMetrics

Task	Modality	Datasets	Framework
Segmentation	Image	Rellis3D, GOOSE, custom GAIA format	PyTorch, Tensorflow
	LiDAR	Rellis3D, GOOSE, custom GAIA format	PyTorch (RandLA-Net and KPConv from Open3D-ML)
Object detection	Image	Check DetectionMetrics v1	Check DetectionMetrics v1

More details about the specific metrics and input/output formats required fow each framework are provided in the Compatibility section in our webpage.

Installation

In the near future, DetectionMetrics is planned to be deployed in PyPI. In the meantime, you can clone our repo and build the package locally using either venv or Poetry.

Using venv

Create your virtual environment:

mkdir .venv
python3 -m venv .venv

Activate your environment and install as pip package:

source .venv/bin/activate
pip install -e .

Using Poetry

Install Poetry (if not done before):

python3 -m pip install --user pipx
pipx install poetry

Install dependencies and activate poetry environment (you can get out of the Poetry shell by running exit):

poetry install
poetry shell

Common

Install your deep learning framework of preference in your environment. We have tested:

• CUDA Version: 12.6
• torch==2.4.1
• torchvision==0.19.1
• tensorflow[and-cuda]==2.17.1

If you are using LiDAR, Open3D currently requires torch==2.0*.

Usage

As of now, DetectionMetrics can either be used as a Python library or as a command-line application.

Library

You can check the examples directory for inspiration. If you are using poetry, you can run the scripts provided either by activating the created environment using poetry shell or directly running poetry run python examples/<some_python_script.py>.

Command-line interface

DetectionMetrics currently provides a CLI with two commands, dm_evaluate and dm_batch. Thanks to the configuration in the pyproject.toml file, we can simply run poetry install from the root directory and use them without explicitly invoking the Python files. More details are provided in DetectionMetrics website.

DetectionMetrics v1

Our previous release, DetectionMetrics v1, introduced a versatile suite focused on object detection, supporting cross-framework evaluation and analysis. Cite our work if you use it in your research!

💻 Code	📖 Docs	🐋 Docker	📰 Paper

Cite our work

@article{PaniegoOSAssessment2022,
  author = {Paniego, Sergio and Sharma, Vinay and Cañas, José María},
  title = {Open Source Assessment of Deep Learning Visual Object Detection},
  journal = {Sensors},
  volume = {22},
  year = {2022},
  number = {12},
  article-number = {4575},
  url = {https://www.mdpi.com/1424-8220/22/12/4575},
  pubmedid = {35746357},
  issn = {1424-8220},
  doi = {10.3390/s22124575},
}

How to Contribute

To make your first contribution, follow this Guide.

Acknowledgements

Utils for LiDAR segmentation, such as sampling or recentering, are based on Open3D-ML.