Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add xfeat+lighterglue and pairs from sequential #441

Open
wants to merge 7 commits into
base: master
Choose a base branch
from

Conversation

pablovela5620
Copy link

Its been a little while since this PR got opened and I was hoping we could bring sequential matching!
#175

added exponential matching + loop closure using retrival based on this https://gitlab.ethz.ch/z-ricityslam/hloc_fork

I tested with South Building dataset since its sort of sequential (to make sure the loop closure worked).
Here is a video of the reconstruction + pairs generated. sequential matcher lead to a significant reduction in pairs created and lead to a good speedup in reconstruction!

Matching Type Number of Pairs Generated Reconstruction Time
Exhaustive Matching 8000 20 minutes
Vocabulary Tree Matching 6396 14 minutes
Sequential w/ Loop Closure 1543 9 minutes
sequential-reconstruction-2024-12-04_12.07.41.mp4
sequentail-matches-av1.mp4

I also included xfeat feature detector + lighterglue matcher from the great work over at https://github.com/verlab/accelerated_features

@sarlinpe let me know what I can do to help get this merged!

@robinlindh
Copy link

Hey, I have tried out the pairs_from_sequential, works very well. We use RGB data from mobile video captured at 10 FPS in a custom pipeline. Earlier we used pairs_from_poses (from some other PR on here earlier) and used 20 pairs. Now I use your pairs_from_sequential with:

    pairs_from_sequential.main(
        paths.sfm_pairs, references, None,
        window_size=6,
        quadratic_overlap=True,
        #use_loop_closure=False,
        #retrieval_path=None,
        #retrieval_interval=2,
        #num_loc=5
    )

The quadratic overlap with window size 6 gives us approximately the same range of nearby images as the 20 nearest poses, but using fewer pairs.
Running on a pretty big dataset we get a pretty big speed-up but still the same accuracy.
Haven't tried the retrieval-based loop closure yet but that will quite possibly improve our accuracy too so that looks very promising.

If anything, the window_size feels somewhat limiting currently as the same parameter controls both the number of pairs in direct sequence and the length of the furthest pair (growing quadratically). I just added some debug log on the first image to figure out what pair offsets I actually end up with as that would usually need to be handled with some care. In my case picking the window_size 6 gives a good sweet spot where the furthest pair is a few seconds away at our 10 FPS data. Would be nice to expose this a little bit more. Maybe a second parameter used for the number of quadratic overlap pairs, where the pairs are taken at +2, +4, +8 and so forth, starting from the END of the normal window size.

In any case, this is already very useful, so wanted to thank you for that!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants