Deluge downloads unnecessary files

[amatus]

I have added a torrent and selected some files to download (not all), despite that, the Deluge partially downloads unnecessary files.

Deluge 1.3.1 libtorrent version: 0.15.5.0

[shnurapet]

This is because torrent files consist of data blocks and not files. Thus, the end part of a file may interleave with the beginning of the next file within one data block. The block has to be downloaded fully to make progress. This is the peculiar feature of BitTorrent? protocol and does not depend on client.

[amatus]

But other clients don't leave these files.

[kneekoo]

I agree, the files marked as "Do Not Download" should be at least cleaned up after the data blocks get downloaded. However, the better way to do would be to only download the necessary data from the data blocks, leaving out the unwanted data. I don't know if this can be controlled by Deluge or it's about libtorrent but if Deluge can have a say at it then this would be a nice improvement.

[damoxc]

This is controlled by libtorrent, whilst we could remove the files after they've been downloaded I'm not sure what would happen when you try to seed.

[amatus]

I think as a solution the Deluge should store these files separately.

[Cas]

I thought there is a duplicate ticket but cannot find it atm so will leave this open. Though there is certainly an  FAQ question about this and has been discussed in the forums.

Replying to amatus:

But other clients don't leave these files.

You will find that they actually do but the files are hidden from the user, usually in a '.part' file.

Replying to amatus:

I think as a solution the Deluge should store these files separately.

I looked into this before and IIRC there is a technical reason it cannot be done due to the libtorrent code and available python bindings.

This  forum post has a similar plugin request discussion.

Also there is a  Delete Partials plugin that might help.

[ku]

Could it be handled a different way? Someone already suggested renaming the file, my biggest problem is, though, that if those "unnecessary files" END with this necessary block, they are created in full size. Somewhat extreme example: We have a torrent with bluray remuxes of a trilogy with following files Movie1 (50 GB) Movie2 (50 GB) Movie3 (50 GB) SomeBonusStuff? (100 MB)

We just need the Movie2 and the bonus stuff and we have 100 GB free. We will never be able to download it though, since Movie1 and Movie3 will be created in full size, even though just a couple of kilobytes are needed. I always lack free space and I've run into problems like this pretty often.

Can a workaround be created or does libtorrent handle file creation/file names as well? Maybe someone could make a plugin that could fool libtorrent by automatically creating a link to a virtual compressed filesystem for those files. Can it all be solved nicely on libtorrent's side or is some of it client-dependent?

Also, it would be nice if deluge would display an error when we run out of space. Now it just stops downloading and it looks just like it can't find right seeds.

[kneekoo]

I must admit the suggested workaround is close to brilliant but it's still wrong. The solution is to fix libtorrent so it can handle file downloads properly, as the users request.

I wonder if the Deluge developers forwarded this bug to libtorrent but in the end we could check ourselves and if not just go ahead and ask them to fix it. It will not only solve this issue for Deluge but all the other torrent clients based on libtorrent.

[Cas]

I always lack free space and I've run into problems like this pretty often.

The space issue should not be a problem if using a filesystem that supports sparse files or you enable compact allocation. (FAQ).

I have got working code that will rename files marked for DnD so they become hidden and already planned to revisit it to see if it was possible to implement for 1.3.4.

---

it would be nice if deluge would display an error when we run out of space.

Already a ticket for that: #1899

[ku]

Now I f-g get, why I suddenly run out of space so often!!! The whole reason I use Full allocation (since I first used uTorrent in Windows years ago) was to not run out of space suddenly, because I download new torrents very frequently and some of them need very long time; this way I knew how much space I will have left in the end and that everything will fit, so that I won't run out of space and downloads won't stop half an hour after I went to bed or left my flat. In my opinion, it is pretty much the whole point of full allocation!

I would suggest to change to "real" full allocation (as in normal file filled with 00). Maybe: ( ) Use Sparse File , ( ) Use Compact Allocation , (o) Use Full Allocation + (X) Use Compact Allocation for parts of unneeded files

And if you choose to download this unneeded file later, it woud just convert to a fully allocated file filled with 00.

Would love to see it implemented in one of the next versions! Together with hiding of the unneeded files.

P.S. God, I was hunting for overblown log files, temp files, gone through all the .-directories, wondered what else could have eaten up all my free space since just yesterday or even a couple a hours ago I had enough ;) Or wondered, why I just deleted, let's say 4 GB, but got only 3,5 free. And not ever did I think about looking more into that Full Allocation option... I usually download new stuff while surfing for hours and switch into my file manager hours later, when almost everything is already downloaded, so I didn't notice how, say, a 5GB file was created, but free space didn't change. Thank you for clearing this up :)

[ku]

By the way, why doesn't this tracker's engine understand my EOL's?
Does one have to use Line Break all the time or does it only understand Windows' EOL?

[Cas]

Would love to see it implemented in one of the next versions! Together with hiding of the unneeded files.

Any solution that I put into the 1.3 release would be a basic workaround that would rename the files.

The best solution is to use new functionality available in libtorrent dev code so we cannot utilise it yet. The next major version of Deluge will be based on that code so most likely will implement what you are asking for.

For reference, ticket #1923, is a request for Full Allocation option.

---

By the way, why doesn't this tracker's engine understand my EOL's?

See WikiFormatting

[ku]

Thanks a lot! Saw the workaround... Probably wouldn't use it though, because of the partly downloaded files.

Could you make an approximate guess, when new libtorrent and deluge 1.4 could be released?