DISC protocol enhancements: identify and repair wrong replication

[Eknir]

It may happen that a node initiated replication because it thinks it is the closest peer, while it is not the closest peer. This can happen if the node that is initiating replication is not connected to the node(s) that are closer.

If this happens, there is the potential for a node that received a replication to identify this and repair it.

Identification: I (A) receive a chunk from a node (B) that is closer to the chunk than me, but am connected to a node (C) that is closer to the chunk than B

Repair: Push chunk to C

Create a metric that will count whether a repair fails.

TODO:

1. look up metrics on production that gives us an idea of how often this situation arises and report here:
   • total_distant_replications (absolute)
   • total_distant_replications/total_closest_replications (percentage of all replications)
2. implement repair
3. identify how often repair fails
4. if repair failures happens, discuss

[Eknir]

The research team is currently preparing a proposal for an enhancement of the push sync protocol. That proposal will supersede this one.