Error: untracked working tree files aborts imerge

[egpbos]

I have a large merge that I'm trying to perform with imerge, reproducable as follows:

git clone https://github.com/roofit-dev/root.git
cd root
git checkout MP_ZeroMQ-new_multiprocess
git-imerge start --name=RooFit_MultiProcess_PR --first-parent master

However, I get the following output:

... [lots of successes] ...
Autofilling 108-1...success.
Autofilling 108-2 (first way)...success.
Autofilling 108-2 (second way)...success.
The two ways of autofilling 108-2 agree.
Recording autofilled block MergeState('RooFit_MultiProcess_PR', tip1='MP_ZeroMQ-new_multiprocess', tip2='master', goal='merge')[0:109,0:3].
Attempting automerge of 108-10314...failure.
Attempting automerge of 1-3...failure.
Switched to branch 'imerge/RooFit_MultiProcess_PR'
going to run: git merge --no-commit -m imerge 'RooFit_MultiProcess_PR': manual merge 1-3 86122e685e669210292c91cf7927db790590824a
error: The following untracked working tree files would be overwritten by merge:
	core/meta/test/stlDictCheckAux.h
Please move or remove them before you merge.
Aborting

Original first commit:
commit 4c74bc4c504a5c164ac48a3d551d3c0400fdb56c (refs/imerge/RooFit_MultiProcess_PR/manual/1-0)
Merge: ecd5908865 da55c31d18
Author: Patrick Bos <egpbos@gmail.com>
Date:   Thu Oct 11 08:07:42 2018 +0200

    Merge remote-tracking branch 'upstream/master' into MultiProcess_Vector

    # Conflicts:
    #       .travis.yml
    #       math/minuit2/src/MnUserTransformation.cxx
    #       roofit/roofitcore/CMakeLists.txt
    #       roofit/roofitcore/Module.mk
    #       roofit/roofitcore/inc/RooRealIntegral.h
    #       roofit/roofitcore/src/RooMinimizer.cxx
    #       roofit/roofitcore/src/RooRealIntegral.cxx
    #       roofit/roofitcore/test/CMakeLists.txt

Original second commit:
commit 86122e685e669210292c91cf7927db790590824a (refs/imerge/RooFit_MultiProcess_PR/manual/0-3)
Author: Danilo Piparo <danilo.piparo@cern.ch>
Date:   Thu Oct 4 17:24:23 2018 +0200

    [Core] ROOT-9712 increase coverage of missing dict checks to cover new features

    looking for dictionaries of template types of unique_ptr, array, tuple recursively
    and making sure that pointers and const qualifiers are properly treated.

There was a conflict merging commit 1-3, shown above.
Please resolve the conflict, commit the result, then type

    git-imerge continue

Notice the error in the middle, after which it says it has aborted. After this, git status does not say anything about being in a merge process, as I would expect. It just ends up in the imerge/master branch with a long list of untracked files. Also, there are no merge conflict >>>>> and <<<<< lines in the files that are indicated in the Conflicts list in the output, so I guess indeed the auto-merging part has aborted and I guess git-imerge was not expecting this, because the rest of the output seems to indicate that there is still something to be done. However, I'm at a loss on how to continue at this point.

I hope this is clear, if not please let me know. What can I do here to continue the process?

[egpbos]

Looking into the sourcecode now, I've traced the problem to line 2707 where the self.git.manualmerge(left.sha1, logmsg) call apparently fails. So I guess I'll have to somehow fix whatever that command is doing before I can continue the imerge process, correct?

[egpbos]

Indeed, if I manually run the git merge command (git merge --no-commit -m imerge 'RooFit_MultiProcess_PR': manual merge 1-3 86122e685e669210292c91cf7927db790590824a to be exact in this case), after removing the problematic file manually as well (rm core/meta/test/stlDictCheckAux.h), then the repo goes into the merge state I had expected.

[egpbos]

Ok, good, I solved my immediate problem :)

Perhaps it would be useful to print the git merge command that is run from manualmerge, if not from that command itself then perhaps below where the CalledProcessError is caught in line 2708, i.e. as a replacement of the pass there. I could do a PR for this if you think this would be useful.

It seems to be rather useful to me, at least, since while typing this, my merge already stumbled onto another similar situation. Maybe I'm missing some other obvious solution to my issue, though.

[egpbos]

It seems the issue arises on merging 1-x quite often. The 1 commit in this situation is a merge commit itself. I previously merged in the master branch that I'm now merging in again (it has been a very long living branch). Do you think merge commits can be expected to cause these issues more often? I'm not well enough versed in git/version control theory to confirm this. But if it is the case, perhaps it makes sense to build in some special handling for these cases.

[Maartenvm]

Well aren't you having a nice conversation with yourself? :)

It seems to me that your error refers to a file not being added to staging before you committed your changes.
error: The following untracked working tree files would be overwritten by merge:
core/meta/test/stlDictCheckAux.h

looks like you need to add it with git add and append it to your commit ?

[egpbos]

I noticed that the exit code on the failing merge command is 128. Looking at the git source code here, it seems this error code can be caused by several conditions. For recursive merge, the two possibilities are that:

1. merge_start fails, which is indeed the problem I'm seeing (this line); in this case the actual merging (and merge_finalize) is not performed, leaving the repo in its existing state, as I'm indeed seeing.
2. The merge was not "clean" (negative cleanliness).

So, just catching the merge command's error code won't be enough, the state of the repo would have to be included as well. Also, this is just the recursive merge case, not sure what happens in the "ort" case, but also not sure whether that's relevant for imerge.