Different results with old and new repo.

[charliergi]

Hi !
I'm interested in both of your repos (command line for the old one and better plots in the new one), used them both with the same data, but I have different results with Pascal VOC metrics. Is there a reason I missed ?
I achieve an mAP of 84.52% with the new one and 88.46% with the old one; both with relative coordinates
class_name [confidence] center_x center_y width height

Here is the log produced by the old one :
results.txt
and the output of the new one with all the metrics enabled :
results.txt

Thanks in advance for your answers !

[rafaelpadilla]

Hi @charliergi ,

Thank you for your message showing the comparative results.

PASCAL VOC results may vary depending on two parameters: IOU threshold and interpolation method.

The new repository adopts the every point interpolation method and uses the IOU threshold defined by the user in the visual interface.

The old repository is a little more flexible on the PASCAL VOC metric, once you can choose the IOU threshold and the interpolation method. In the old repo, the default IOU threshold is 0.5 and the default interpolation method is the every point interpolation.

So, my question is: Have you used the same parameters (IOU threshold and interpolation method) in both repositories? If so, could you please, provide me you detections and ground-truths so I can make a further investigation?

Best regards

[charliergi]

Hi @rafaelpadilla,
I haven't modified the IOU or interpolation method in both repositories.
Here is my data if you want to further investigate what is going on :
predictions.zip
ground-truth.zip
Images are of dimension 512x512 and are in YOLO (.txt) format.
I noticed also something strange, as suggested in the README.md, I tried first to use the arguments to indicates that coordinates are in relative format and used the another argument to give the size of the images, but it is giving me 0mAP and 0 AP per class when using the command
!python3 /workspace/tlt-experiments/Object-Detection-Metrics/pascalvoc.py \ -gt labels_yolo/ \ -det inference_labels_yolo/ \ -sp stats-18-cmd \ -detcoords rel \ -imgsize 512,512
But is giving me normal (mAP 88.46%) results when using
!python3 /workspace/tlt-experiments/Object-Detection-Metrics/pascalvoc.py \ -gt labels_yolo/ \ -det inference_labels_yolo/ \ -sp stats-18-cmd
Maybe I misunderstood something ?
Best regards

[rafaelpadilla]

Hi @charliergi,

I used your files and obtained very similar results in both repositories. I created 4354 empty images 512x512 with the same name of the ground-truth files.

For the old repository
I used the command:
python pascalvoc.py -gt /gt/labels -det /det/inference_labels -t 0.5 -gtformat xywh -gtcoords rel -detformat xywh -detcoords rel -imgsize 512,512

And obtained these results: results_old_repo.zip

As all your coordinates are in relative format, you must use -detcoords rel and -gtcoords rel. And, as you are using relative coordinates, you must specify the image size -imgsize 512,512. This way, it will consider your images with the same resolution (512x512). In this repository, if your images do not have the same resolution, you must convert the bounding boxes to absolute values. Or use the new repository.

For the new repository
I used these settings:

And obtained these results: output_new_repo.zip

Comparing results

See that the results are the same for all objects, except for object 7, which needs further investigation. But you should get similar results.

Please, check if you are using the same command as me.

[charliergi]

Hi @rafaelpadilla,
Indeed that was my mistake, I misunderstood the arguments for the old repository. Now with the right arguments, I have the same results as you.
A huge thanks for the help.

Best regards.

[rafaelpadilla]

Hi @charliergi ,

No problem :) I'm glad your problem is solved.

If you wish to publish your results, please consider citing us.

Best regards

[charliergi]

We are doing a master thesis about insect recognition and your repository is very useful. We will not forget to cite you.
Best regards

Gilles