osxphotos sync export/import: mismatch fingerprint (related with push-exif and my own exiftool processing?)

[oPromessa]

Before submitting a bug report, please ensure you are running the most recent version of osxphotos and that the bug is reproducible on the latest version

osxphotos, version 0.68.6
Python 3.12.7 (main, Oct 11 2024, 17:54:25) [Clang 14.0.0 (clang-1400.0.29.202)]
macOS 12.7.6, x86_64

Describe the bug
Running a:

• Library A: export
• Library A: sync export MetaData.db
• Library B: manual import of pics
• Library B: sync import MetaData.db

A few pics don't get matched: 45 out of ~3500 pics. Fingerprint is different.

To Reproduce

1. What' the full command line you used with osxphotos?

$ osxphotos export photos.hard/ --load-config recreate.toml --library "/Users/Shared/Pictures/iPhoto Shared Library.photoslibrary/database/Photos.sqlite"
(...)

recreate.toml (contents)

[export]
cleanup = true
update = true
db = "/Users/Shared/Pictures/iPhoto Shared Library.photoslibrary"
directory = "{album|filter(startswith 1|2|T)?,}{created.year}/{created.year}-{created.mm}/{created.dd| <= 10?01-10,{created.dd| <= 20?11-20,{created.dd| > 20?21-31,00}}}/{photoinfo.path|function:paths.py::mypath}"
export_as_hardlink = true
exiftool_merge_keywords = true
exiftool_merge_persons = true
sidecar = [ "xmp", "json", ]
folder = [ "iPhoto Events",]
export_aae = true
keyword_template = [
    "{keyword}",
]
no_progress = true
not_hidden = true
ramdb = true
retry = 2
skip_bursts = false
skip_live = false
skip_edited = true
skip_original_if_edited = false
skip_raw = true
strip = true
timestamp = true
touch_file = true
verbose = true
keep = [ ".DS_Store", ".smbdelete*", ]
name = [ "1951", "1961", "1952", "1962", "1953", "1963", "1954", "1964", "1955", "1965", "1956", "1966", "1957", "1967", "1958", "1968", "1959", "1969", "1950", "1960", "202312", "201511", "202301", "202304", "202402", "Mix", "IMG_20240901_001605", ]

$ osxphotos sync --verbose --timestamp --not-hidden --export MetaData.db --library /Users/Shared/Pictures/iPhoto\ Shared\ Library.photoslibrary/ --name 1951 --name 1961 --name 1952 --name 1962 --name 1953 --name 1963 --name 1954 --name 1964 --name 1955 --name 1965 --name 1956 --name 1966 --name 1957 --name 1967 --name 1958 --name 1968 --name 1959 --name 1969 --name 1950 --name 1960 --name 202312 --name 201511 --name 202301 --name 202304 --name 202402 --name Mix --name IMG_20240901_001605 

2. What was the error output?

Unable to find metadata for XXX.JPG (2DEE992C-2003-4299-BF5A-096AA9C18C5C) in /Users/XXX/MetaData.db

Expected behaviour
A match to be found on the pics and the sync performed.

Investigation

• Investigating... I export with hard links from a Referenced Library and Import into non/Referenced Library. MD% on the original, exported (same i-node 😉 ) and the imported are the same:

$ md5 ORIGINAL.jpg EXPORTED.jpg '/Volumes/SSD500GB/Pictures/Photos.photoslibrary/originals/8/85D4CA50-80CC-41E3-917D-C4CDB4605B8F.jpeg'
MD5 (ORIGINAL.jpg) = a852208d9c7becb37542b7dc945a43a2
MD5 (EXPORTED.jpg) = a852208d9c7becb37542b7dc945a43a2
MD5 (/Volumes/SSD500GB/Pictures/Photos.photoslibrary/originals/8/85D4CA50-80CC-41E3-917D-C4CDB4605B8F.jpeg) = a852208d9c7becb37542b7dc945a43a2

• If I try to import ORIGINAL.jpg or EXPORTED.jpg again into LibraryB, manually, Photos warns me they are duplicated

• The debug-dump --dump photos does show different

  • fingerprints
  • different file size (on disk all 3 files are 738763). But I think that related to the ORIGINAL file loaded into Photos. Since then I've added Location in Photos, corrected timezone and then performed a push-exif.
  • gps info. Oddly enough the LibraryB GPS data comes with an approximate positioning with more decimals from field "[MacOS] MDItemLatitude".
  • the timezone is also different:
    • LibraryA gives me in the format Region/Country
    • LibraryB gives me in the format GMT+NNNN

• Looking at the code there are a few different methods to get to the fingerprint but it seems to me "photo.fingerprint" is the one being used on both Libraries.

• Question Does push-exif and exiftool changing other date/times on the file influence the fingerprint calculation from LibraryA and LibraryB? I'm thinking the process I run of:

  • Photos: Set Location
  • Photos: Set Timezone
  • osxphotos: push-exif datetime,location
  • exiftool to normalize all date/times to be correct: original, EXIF, XMP, QuickTime, etc... (see details fields below)
  • and then finally "export (fingerprint from LibraryA) + sync export + Manual import (different fingerprint into LibraryB) + sync import" gets different fingerprints!

For Images: Date/time fields I set with exiftool after push-exif)

-SubSecDateTimeOriginal=      # Format for all dates is YYYY:MM:DD HH:MM:SS[+/-]HH:MM except where marked.
-SubSecModifyDate=
-SubSecCreateDate=
-DateTimeOriginal=            # Format YYYY:MM:DD HH:MM:SS
-QuickTime:CreationDate=   
-XMP:DateTimeOriginal=        # Format YYYY:MM:DD HH:MM:SS
-QuickTime:CreateDate=
-XMP:CreateDate=               
-CreateDate=
-quicktime:modifydate=
-ModifyDate=
-FileModifyDate=
-Quicktime:ContentCreateDate=  
-Track*Date=
-Media*Date=
-FileCreateDate=
-GPS:GPSDateStamp=
-GPS:GPSTimeStamp=
-XMP-photoshop:DateCreated=
-IPTC:DigitalCreationDate=
-IPTC:DigitalCreationTime=
-IPTC:DateCreated=
-IPTC:TimeCreated=
-AllDates=
-common_args
-api
-QuickTimeUTC=1
-overwrite_original_in_place

Screenshots
N/A

Desktop (please complete the following information):

osxphotos, version 0.68.6
Python 3.12.7 (main, Oct 11 2024, 17:54:25) [Clang 14.0.0 (clang-1400.0.29.202)]
macOS 12.7.6, x86_64

Additional context
Add any other context about the problem here.

[oPromessa]

Doing bit more testing and I believe that is the case: fingerprint gets changed! 😢 !

Either on push-exif and/or on my exiftool update of dates and GPS position!

Maybe if I do a "timewarp pull-exif" would that force Photos into recalculating the fingerprint and re-align Photos info and file info?

[RhetTbull]

Anything that modifies the file (using exiftool for example) will change the actual fingerprint of the file. I believe that Photos only computes the fingerprint once at time of import. Here's what I think is happening:

Library A: sync export MetaData.db

The metadata is pulled from the Photos database to write MetaData.db. The modified image (from push-exif) has a fingerprint that doesn't match what is in the database and the value that gets written is the original fingerprint not the new fingerprint.

Library B: manual import of pics

The modified image is imported and Photos sets the fingerprint value based on the current image.

Library B: sync import MetaData.db

Because the fingerprint of the imported image is different than what was originally imported into Library A, the image cannot be found.

I don't know what to do about this as the code is behaving as designed. I always caution against modifying the files in the Photos library for reasons like this. One option that might work is to allow user to specify a custom template for matching photos between libraries. osxphotos import allows this with --signature. You could then provide a custom template function that performed a fingerprint computation of the actual file instead of using what is in Photos. The code to do this is in osxphotos/fingperint.py.

[oPromessa]

Got it. Thanks. I think that's the way to go.

Thanks again.