Clarification about threshold in keypoint HOTA

[krishnakanthnakka]

In the distance to similarity computation at

PoseTrack21/eval/posetrack21/posetrack21/trackeval/metrics/hota_pose.py

Line 37 in 35bd703

sim = np.maximum(m * distance_mtx + 1, 0)

I'm unable to comprehend the reason to clip the similarity to zero if the (head normalized) distance is greater than 0.5.
This has consequences in (incorrectly) tagging each detection as TP/TN/FP/FN for each alpha ranging from 0.05 to 0.45 at

PoseTrack21/eval/posetrack21/posetrack21/trackeval/metrics/hota_pose.py

Line 147 in 35bd703

actually_matched_mask = similarity[match_rows, match_cols, j] >= alpha - np.finfo('float').eps

In MOTA calculation, the threshold alpha is fixed at 0.5 i.e., PCKh@0.5. Please refer to
https://github.com/leonid-pishchulin/poseval/blob/4258a1575b9f2ddd0bdb85f74557235ab5df0f52/poseval/evaluatePCKh.py#L56.

However, when we try to estimate HOTA at different confidences (0.05 to 0.99), the clipping of similarity has effect of completely neglecting all the PCKh@0.05, @0.1@0.15, @0.2, @0.25 @0.3 @0.35 @0.4 aand 0.45.

Futher it additionally calculates PCKh@0.52.5, @0575, @0.625, @0.675 @0.725, @0.775 @0.825 @0.875 @0.925 @0.975 to the original PCKh@0.5, @055, @0.6, @0.65 @0.7, @0.75 @0.8 @0.85 @0.9 @0.95

To connect the things into perspective, pascal VOC computes mAP@0.5 whereas COCO computes the mAP@0.05:0.99:0.05. MOTA calculates PCKh@0.5 and hota calculates at all levels @0.05:0.99:0.05.

In short, the distance function may needs to be revisited. I think sim = np.maximum(-1 * distance_mtx + 1, 0) makes it consistent with HOTA metrics based on IoU, MOTA at PCKh@0.5.

Please let me know your thoughts and did i missed something?

[anDoer]

Thanks for your feedback. I will have a closer look at this problem and I agree that sticking to PCKh@0.05:0.99:0.05 would have been more consistent with existing evaluation metrics.

[krishnakanthnakka]

Hello @anDoer,

A gentle reminder to please check this issue and the next one as well.

Thanks