Sequential download and download first and last piece by default

[stalkerok]

Since requests to add a "sequential download and download first and last piece by default" feature keep closing without any result, and since the original request to add the feature is closed for discussion #164, I created this discussion.
And so, I will copy what I have written earlier in the requests #20278 #20242

Do you really want to endlessly close these feature requests instead of entering it and forget about it? This is the third request in a week #20242 #20250
Do you really think that sequential downloading can do any harm? But that's nonsense. It's much more harmful to stop/delete the torrent after downloading. There is no way you can control the "swarm health" and trying to do so seems very silly. These attempts have led to the fact that the client lacks a very important feature for many people.

In fact, no critical harm will come from it, we only have a theory that it could do harm. It's silly to dismiss it in favor of a theory when users need the feature (for a wide variety of purposes) and it's actually useful. IMO, the real pros far outweigh the possible cons in some absolutely isolated cases.

@sledgehammer999 @glassez @Chocobo1 @thalieht @xavier2k6
So why do you really think it's an unnecessary feature when users are requesting it en masse? Like I said, as much as you'd like to, you can't control "swarm health", so it's not a reason to say no. These requests will continue indefinitely.

I'd also like to point out that I don't need this feature, I have good internet and don't need to download sequentially, but I absolutely don't understand the reasons for the rejection.

[glassez]

After all, no one forbids downloading torrents sequentially.
After all, we are talking about the fact that having such options enabled by default, the user certainly will not disable them in cases when they really do not need them.

Personally, I have no strict objections to providing these options. If it really can worsen "swarm health", then the users involved will suffer from it themselves.

[stalkerok]

After all, we are talking about the fact that having such options enabled by default, the user certainly will not disable them in cases when they really do not need them.

Most users do not know what options are in the advanced client settings and what they are responsible for. Yes, once enabled, the user is unlikely to disable it, but what percentage of the total number of users will know that the option even exists?

[glassez]

We're talking about those users who know about these features, aren't we?
Currently, in order to use such options, the user should explicitly enable them in the "Add new torrent" dialog for each specific torrent. So I believe they won't do it in cases where they doen't need it (after all, it's two whole extra clicks, which is why all the fuss). If we provide the way to enable these options by default, then such users are unlikely to disable them in cases where they are not required.

[stalkerok]

If we provide the way to enable these options by default, then such users are unlikely to disable them in cases where they are not required.

Yes, but I don't think it's a bad thing, as it will be a small part of users, enabling it by default won't affect anything, but the option will be available, which will be a big plus for qBittorrent and will get rid of constant requests.

[glassez]

But as I said above I don't too care about it.

[Chocobo1]

So why do you really think it's an unnecessary feature when users are requesting it en masse?

I'm not here to defend my point but I wanted to seek some answers.

Like I said, as much as you'd like to, you can't control "swarm health", so it's not a reason to say no. These requests will continue indefinitely.

Some backgrounds. It is true that we can't control swarm health directly but if qbt market share is high within the swarm, then qbt default behavior would massively affect the swarm.
When the swarm is healthy, qbt/libtorrent already will download it sequentially automatically. And when the swarm is not healthy, I doubt download sequentially is the behavior that remaining peers/downloaders would like to see.

And so, I will copy what I have written earlier in the requests #20278 #20242

What I find lacking in those issues is that they didn't give rationale for wanting the feature (except one for a strange reason). IMO they should have a persuasive usage story and not just "I want it". I would really prefer the option be backed by reasonable use case and not just because we can do it (or I'm too lazy to check 2 checkboxes).

If the request is still high. I don't mind having a (hidden) key in .ini file to change the behavior (without UI options).

[glassez]

If the request is still high. I don't mind having a (hidden) key in .ini file to change the behavior (without UI options).

I doubt that this will save us from more and more such requests. The difference is how we will respond to them before closing.

[stalkerok]

@Chocobo1, private trackers have already taken care of this, in other cases there is no point in trying to improve the situation. If the swarm is not healthy, the presence or absence of the option will not change this, especially the option will not be used by so many users that it would have any impact.

From my point of view, even one use case for video preview should be enough. This is a tangible plus, unlike the possible impact of the option on the swarm.

I also know about the existence of auto_sequential

[xavier2k6]

#5875 & #9305 similar requests

[Valentin-Metz]

I think I can demonstrate that "it's bad for swarm health" is a common misconception that does not apply, even in very unfortunate swarm states. It is based on assumptions about content importance and peer behavior that do not hold true in real word scenario.

The following assumptions are made:

1. Peers behave in a self-centered, but non-hostile behavior (they seek their own advantage, but do not actively try to harm the swarm for fun)
2. Files are ultimately only useful to peers in a complete state (peers want to complete the download of the files they have chosen to download)
   This gives us a few nice properties, mainly that partial torrents (where users don't want to download everything) behave exactly like a torrent with fewer parts. The same is true for torrents currently in the process of being downloaded.

Let us assume that we have 1 seeder and 100 peers, each with a fraction of the upload capacity of the seeder.
If all peers start downloading at the exact same time, the expected performance improvement by distributing pieces perfectly random is ~20%. Source: https://git.gnunet.org/bibliography.git/plain/docs/ICC%2708%20-%20Super-Seeding%20in%20BitTorrent.PDF?id=90e6303d3d2a2910f22d75e6384e17b03a73fe30
Even in this simple, controlled scenario with the seeder in super-seeding mode (so roughly equivalent to random swarm) they already noticed a negative effect of the technique once peers begin to leave after they have completed their download.

Peers downloading sequentially have an interesting effect:
They slow down the furthest peer, that is, the peer closest to completion.
As the pieces at the start are more-readily available than the pieces at the back, this implements a natural "catch-up" mechanic and prevents the furthest peer from leaving the network early.

In a real word scenario, peers do not all arrive at the exact same time. This makes it even more beneficial to prevent the fastest peer from leaving early.
In addition to that, this establishes an automatic "content prioritization" system.
Parts that are more important are replicated more often and can therefore be supplied to newcomers quicker, allowing newcomers to interact with the content more quickly.
(This has become more important in recent years, as we are now at a point where it is entirely possible for a large percentage of users to watch content while it is still being downloaded)
In fact, I'd rate the capability of a swarm to perform in that particular aspect as a part of swarm health.

It might therefore be useful if a certain amount of users make this behavior default.
(A checkbox to enable it permanently might therefore might actually improve swarm health)
Whether it should be default for everyone is an open research topic.

As to why this mechanic is useful for a single user in bulk:

1. Quickly inspect material that's being downloaded
2. HDD sequential writes. Especially on log-structured filesystems like NILFS or ZFS in snapshotting mode (because there it is unwise to activate space preallocation)

In order to further increase the usefulness of prioritized downloading, I suggest the following feature:
Recursive "Download first and last piece first"
Example 1 (single movie):
big_movie.mp4 | subtitles | description
The default Download first and last piece first will do great here. It'll usually get the subtitles and the description, as well as the end of the .mp4, allowing the file to play normally.
Example 2 (tv series):
episode_1 | episode_2 | episode 3 | ... | episode_10 | description
The episodes are of a video format that can not start playing until the end of the file is available.
(I don't know which file formats requires this under what exact condition, I can only tell that this happens regularly)
In this case a recursive algorithm should be applied.
First, the normal Download first and last piece first should be applied to the whole torrent, but after that, every file that has been reached by Download in sequential order should have Download first and last piece first applied on a per-file basis.

[glassez]

If all peers start downloading at the exact same time, the expected performance improvement by distributing pieces perfectly random is ~20%. Source: https://git.gnunet.org/bibliography.git/plain/docs/ICC%2708%20-%20Super-Seeding%20in%20BitTorrent.PDF?id=90e6303d3d2a2910f22d75e6384e17b03a73fe30

Sorry, I don't have time to read all this. What kind of performance are we talking about? Is it download speed or something other?

[Valentin-Metz]

That'd be average time until the download is complete for all peers.

[glassez]

But is it supposed to take care of this aspect (or about this aspect only)? After all, it takes care of increasing the number of distributed copies so that the content is still available even if all seeders are currently disconnected.

[Valentin-Metz]

That depends on peer behavior.
While the mentioned paper does not cover the aspect of "replication percentage", we should be able to make an educated guess about it from the provided graphics about the upload ratio of the original seed:

In an ideal scenario (where no peer leaves after completion and the seeder has higher upload speed than the peers), the original seed needed to upload 20% less data. The absolute difference (229% vs 181%) is 48% (under the assumption that all of that data would be from not yet replicated pieces of data, which is unlikely, but gives us a worst case estimate).
If the peers are faster in their upload than the original seed (rightmost two columns), this diminishes very quickly.

If peers leave after completion, this effect can reverse:

Note:
These experiments were conducted with a deliberately super-seeding initial seed.
While should result in behavior identical to that of a randomly downloading swarm,
we know nothing about peer behavior in the experiment they conducted here.
This might prevent the experimental results from being transferable to our scenario.

Mathematically, if the number of peers is far greater than the number of seeders,
and the peers pick pieces randomly, the replication state of the original file within the swarm should behave like a version of the
Coupon collector's problem, if the peers don't suggest which pieces to download to each other (I don't think there is a protocol yet that does this), so we get $\Theta(n\log(n))$ as the expected amount of pieces that need to be uploaded by the initial peer until the swarm holds its first full additional copy of the initial data.
The fewer peers we have, and the closer the peers get to completion, the more efficient this process becomes.

If peers download in sequence, we get two opposing effects (assuming the seed selects which peers to upload to randomly):

1. Peers behind in the download progress are likely to request duplicate pieces from the seed.
2. The fastest peer will never request duplicate pieces

I can't make an educated guess about how this will behave without coding up a python simulation.
Maybe I'll have some time to do this in the coming days, but it'll probably take a while.

[Valentin-Metz]

Addendum to answer @Chocobo1 question about why so many people want it.
It probably has less to do with the swarm health reasons, but rather convenience and client interactivity:
It might only be two checkboxes (although that might get tiresome after 700 torrents as well), but that's only if you have the interface open and are not using an automated management program.
Powerusers want to build contraptions where they can drop a torrent file into a folder and 5 seconds later see an mp4 file appear that is correctly sorted into their archive and ready to be watched immediately.

[fx2017]

Yes I am too lazy to keep checking first and last option. I don't care too much about the sequential although that would be nice to.

Sometimes/often being lazy is smart. Adds up to hundreds thousands of ticking checkbox over time

By default disable the option but allow users to change the behaviour for future torrents

[Sam14m]

while it is only one mouse click, I too would enjoy there being an option to have it selected by default

[411621]

So I just got prompted to update to version 5.
I noticed that function "download in sequential order" and "download first and last pieces first" now have a checkbox for whatever reason but do not hold the setting. How can I set this to be the default setting permanently until I decide to change it?

[stalkerok]

It can't be done.