Custom Query (2447 matches)

I propose that partially downloaded files get an extension added so you can tell they are not done yet.

Ever since #1514 got integrated into master I no longer have a working system tray. No Icons and the action do not work. After -TERMinating deluge, the trayicons just keep piling up

Introduction

After upgrading my FreeBSD server I suddenly had problems getting the GTK client to show the torrent list (as well as the states and trackers info to the left.). The CLI client wouldn't show the torrents with the info command. It was pretty much the same symptoms as these bug reports: ​http://dev.deluge-torrent.org/ticket/1531 ​http://dev.deluge-torrent.org/ticket/2095 which I presume have encountered the same bug.

Some that reported this problem in the bug reports have a lot of torrents (many hundreds), and I have about 1600 myself, so it looked like this would be a problem only for those with a lot of torrents.

What happens is that the client can see the daemon being online, but when connecting, it will fail to populate the torrent list. If the torrent list is successfully populated, it continues to work with no problems.

After a lot of testing, expecting it was a problem with corruption of the state files, I eventually found the cause, which of course, had nothing to do with the state files ;-)

The problem actually turned out to be twofold.

Problem one - Application layer protocol for transferring data

The first problem has to do with the application layer protocol, or the lack of such a protocol when it comes to transferring the data over the network. When RPC commands are sent over the network, they're usually successfully decompressed by zlib on the receiver, but not always. When transferring one command (say 20 bytes), with a big enough delay until the next command is sent, the 20 bytes are received on the other side, and the data is decompressed and decoded (by rencode).

When transferring multiple messages within a short time period, these messages might be received as one on the other side. So when transferring two messages at the same time, one of 20 bytes, and one of 40 bytes, the receiver might receive 60 bytes, and has no idea that it's actually two messages.

The current protocol trusts that the zlib library will successfully decompress compressed messages that are concatenated by TCP when transfered. Usually it will successfully decompress the first message, but subsequent messages aren't successfully decompressed as far as I've been able to test.

Most messages are received separately, so that is why this bug shows it's ugly face only in rare cases.

The big problem is that in some cases, the torrent list being transferred isn't successfully decompressed, and thereby lost to the client.

The current deluge protocol will add newly received data onto the current buffer of old (but not successfully decompressed) data and then it tries to decompress the whole buffer. If this fails it will wait for more data.

What happens when the torrent list is never populated, is that it fails to decompress the data in the buffer even after the whole torrent list has been transferred. The result is that it just keeps appending incoming data onto the buffer which results in no messages being received at all.

I'm not sure if this is because the torrent list is received as the second message (i.e. it has been concatenated onto the data of another message on arrival), or if there is another reason the torrent list message fails to be decompressed.

As far as I can see, the only way to solve this is to use an application layer protocol that knows how to separate messages.

Problem two - Decoding error with rencode

After implementing a simple application layer protocol it turned out the torrent list still wouldn't load, because the rencode library fails to decode the transferred data with a

UnicodeDecodeError: 'utf8' codec can't decode byte 0xe8 in position 54: invalid continuation byte

This problem was easily fixed by replacing rencode with ​'pickle'.

Other advantages of pickle (or more precisely cPickle) compared to rencode is especially speed, but also size (somewhat):

~/test$ ./pickle_test.py
Data size: 5125267 bytes

pickle    dumps time: 0:00:00.939099
pickle     dump size: 3470949
cPickle   dumps time: 0:00:00.203606
cPickle    dump size: 3426268
rencode   dumps time: 0:00:00.448616
rencode    dymp size: 4033409

pickle    loads time: 0:00:00.673550
cPickle   loads time: 0:00:00.162311
rencode   loads time: 0:00:00.690657

These are test results from the actual data of the torrent list transferred from my server (1600 torrents).

Patch

I've implemented a class DelugeTransferProtocol which fixes the issues I had, though I cant guarantee that it fixes the issues others have reported, or that it doesn't introduce other bugs ;-)

This patch will render the client unable to communicate with any unpatched daemons.

I've also implemented tests to verify sending and receiving of messages through the transfer protocol.

​https://github.com/bendikro/deluge/commit/8e67c763ab7176bc16b871cac974eef74aaf324f