Symmetry Tracker

The aim of this library is to provide a general tracking framework for ML-based time-symmetric tracking.

All codes here were developed by Gergely Szabó during his PhD research at PPCU

For any additional information or comments, please contact me via my university email address: szabo.gergely@itk.ppke.hu

Paper publication is still in progress, for the preprint version please check out: https://arxiv.org/abs/2308.03887

The frozen codebase version for the publication can be found at: https://github.com/SzaboGergely0419/PPCU_IFOM_YeastTracker

While publication is in progress, please cite our work as:

@article{szabo2023enhancing,
  title={Enhancing Cell Tracking with a Time-Symmetric Deep Learning Approach},
  author={Szab{\'o}, Gergely and Bonaiuti, Paolo and Ciliberto, Andrea and Horv{\'a}th, Andr{\'a}s},
  journal={arXiv preprint arXiv:2308.03887},
  year={2023}
}

Installation Requirements

pip install numpy
pip install opencv-python
pip install pycocotools
pip install matplotlib
pip install scipy
pip install pandas
pip install torch
pip install "git+https://github.com/facebookresearch/detectron2.git"
pip install segmentation-models-pytorch
pip install filterpy

Resources

Models available at: https://users.itk.ppke.hu/~szage11/IFOM%20tracking/TrainedModels/

Sample data available at: https://users.itk.ppke.hu/~szage11/IFOM%20tracking/SampleData/

Getting Started

Yeast Cell Tracking full pipeline (easier):

Synthesized Person Tracking full pipeline (more complex and in development):

Main functionalities

All fuctionalities which are intended to be directly used are listed here. The other functions and classes in the library have supportive roles and are only intended to be utilized by the functions listed here.

Video Transformation

TransformVideoFromTIFF

• At: symmetry_tracker/video_transformation.py
• Transforms the input .tiff format video to the appropriate "folder of .png images" format

General Functionalities

ExportAnnotJSON

• At: symmetry_tracker/general_functionalities/io_utilities.py
• Exports meaningful segmentation, tracking and/or inheritance results, not including the side-variables of tracking (which are only needed for debugging). This is the primary way to export any result in "AnnotDF" format.

Segmentation

SingleVideoSegmentation

• At: symmetry_tracker/segmentation/segmentator.py
• Performs segmentation on a single video. The models are only capable of handling the fixed image resolution they were trained on.

DisplaySegmentation

• At: symmetry_tracker/segmentation/segmentation_io.py
• Displays segmentation results generated by SingleVideoSegmentation() as images

Tracking

SingleVideoSymmetryTracking

• At: symmetry_tracker/tracking/symmetry_tracker.py
• Performs object tracking using Symmetry Tracking based NN model on a video which was already segmented using both the video and the segmentation outputs

SingleVideoKalmanTracking

• At: symmetry_tracker/tracking/kalman_tracker.py
• Performs object tracking using the Kalman Filter on a video which was already segmented using both the video and the segmentation outputs. This is only a reference estimator, and there are most likely other more convenient Kalman Filter implementations instead of this.

SingleVideoSymmetryTracking_L2Distance

• At: symmetry_tracker/tracking/symmetry_tracker_l2dist.py
• Performs object tracking using the Symmetry Tracker in a way that only object centroids are considered for similarity measurement, ignoring morphology This is only a reference estimator for ablation study, use SingleVideoSymmetryTracking instead for optimal results and performance

SingleVideoSymmetryTracking_ShapeDistance

• At: symmetry_tracker/tracking/symmetry_tracker_shapedist.py
• Performs object tracking using the Symmetry Tracker in a way that only object morphology is considered for similarity measurement, ignoring position This is only a reference estimator for ablation study, use SingleVideoSymmetryTracking instead for optimal results and performance

DisplayTracks

• At: symmetry_tracker/tracking/tracking_io.py
• Displays the tracking results generated by SingleVideoCellTracking() as images

WriteTracks (legacy)

• At: symmetry_tracker/tracking/tracking_io.py
• Saves tracking results to serial object countour txt format (may loose parts of object instances consisting of multiple contours)

SaveTracks

• At: symmetry_tracker/tracking/tracking_io.py
• Saves all tracking results to JSON, including side-variables

LoadTracks

• At: symmetry_tracker/tracking/tracking_io.py
• Loads all tracking results from JSON

SaveTracksVideo

• At: symmetry_tracker/tracking/tracking_io.py
• Saves tracking results as an .mp4 video

Heuristical post-processing

These functionalities can improve / change tracking results significantly, but are based on additional heuristics and manual setup.

HeuristicalEquivalence

• At: symmetry_tracker/tracking/post_processing.py
• Connects broken up tracks which have a significant metric similarity at the breaking point with a given maximal time difference. This is a metric similarity between objects on different frames that are not supposed to look the same. While performing this operation may lead to better results in practice, it is not necessarily sound mathematically and should be used carefully.

RemoveShortPaths

• At: symmetry_tracker/tracking/post_processing.py
• Removes the too short paths from a tracking under a given threshold.

Inheritance

BuildInheritanceTree

• At: symmetry_tracker/inheritance/inheritance
• Assigns tracks to their ancestor tracks. Currently the only availalbe assignment method is Euclidean distance. While this method is very simple and is sort of a placeholder for more advanced methods in the future, for simpler tasks it can provide near perfect results.