Custom Query (2449 matches)

Introduction

After upgrading my FreeBSD server I suddenly had problems getting the GTK client to show the torrent list (as well as the states and trackers info to the left.). The CLI client wouldn't show the torrents with the info command. It was pretty much the same symptoms as these bug reports: ​http://dev.deluge-torrent.org/ticket/1531 ​http://dev.deluge-torrent.org/ticket/2095 which I presume have encountered the same bug.

Some that reported this problem in the bug reports have a lot of torrents (many hundreds), and I have about 1600 myself, so it looked like this would be a problem only for those with a lot of torrents.

What happens is that the client can see the daemon being online, but when connecting, it will fail to populate the torrent list. If the torrent list is successfully populated, it continues to work with no problems.

After a lot of testing, expecting it was a problem with corruption of the state files, I eventually found the cause, which of course, had nothing to do with the state files ;-)

The problem actually turned out to be twofold.

Problem one - Application layer protocol for transferring data

The first problem has to do with the application layer protocol, or the lack of such a protocol when it comes to transferring the data over the network. When RPC commands are sent over the network, they're usually successfully decompressed by zlib on the receiver, but not always. When transferring one command (say 20 bytes), with a big enough delay until the next command is sent, the 20 bytes are received on the other side, and the data is decompressed and decoded (by rencode).

When transferring multiple messages within a short time period, these messages might be received as one on the other side. So when transferring two messages at the same time, one of 20 bytes, and one of 40 bytes, the receiver might receive 60 bytes, and has no idea that it's actually two messages.

The current protocol trusts that the zlib library will successfully decompress compressed messages that are concatenated by TCP when transfered. Usually it will successfully decompress the first message, but subsequent messages aren't successfully decompressed as far as I've been able to test.

Most messages are received separately, so that is why this bug shows it's ugly face only in rare cases.

The big problem is that in some cases, the torrent list being transferred isn't successfully decompressed, and thereby lost to the client.

The current deluge protocol will add newly received data onto the current buffer of old (but not successfully decompressed) data and then it tries to decompress the whole buffer. If this fails it will wait for more data.

What happens when the torrent list is never populated, is that it fails to decompress the data in the buffer even after the whole torrent list has been transferred. The result is that it just keeps appending incoming data onto the buffer which results in no messages being received at all.

I'm not sure if this is because the torrent list is received as the second message (i.e. it has been concatenated onto the data of another message on arrival), or if there is another reason the torrent list message fails to be decompressed.

As far as I can see, the only way to solve this is to use an application layer protocol that knows how to separate messages.

Problem two - Decoding error with rencode

After implementing a simple application layer protocol it turned out the torrent list still wouldn't load, because the rencode library fails to decode the transferred data with a

UnicodeDecodeError: 'utf8' codec can't decode byte 0xe8 in position 54: invalid continuation byte

This problem was easily fixed by replacing rencode with ​'pickle'.

Other advantages of pickle (or more precisely cPickle) compared to rencode is especially speed, but also size (somewhat):

~/test$ ./pickle_test.py
Data size: 5125267 bytes

pickle    dumps time: 0:00:00.939099
pickle     dump size: 3470949
cPickle   dumps time: 0:00:00.203606
cPickle    dump size: 3426268
rencode   dumps time: 0:00:00.448616
rencode    dymp size: 4033409

pickle    loads time: 0:00:00.673550
cPickle   loads time: 0:00:00.162311
rencode   loads time: 0:00:00.690657

These are test results from the actual data of the torrent list transferred from my server (1600 torrents).

Patch

I've implemented a class DelugeTransferProtocol which fixes the issues I had, though I cant guarantee that it fixes the issues others have reported, or that it doesn't introduce other bugs ;-)

This patch will render the client unable to communicate with any unpatched daemons.

I've also implemented tests to verify sending and receiving of messages through the transfer protocol.

​https://github.com/bendikro/deluge/commit/8e67c763ab7176bc16b871cac974eef74aaf324f

With many torrents (hundreds), the GTKUI torrent view gets noticeably slower and slower. With around 1800 torrents, each call to update_view takes around 400-600ms on my machine, when showing all the torrents. This causes the entire thing to freeze until it's finished updating.

I've optimized the update_view method so that now, each update takes around 15-20ms when showing all torrents with sorting on the Name column, and a little less with no sorting. CPU usage when showing the All list is down from 90-100% to 4-10%

It's still quite slow to change list, e.g. from a small list (say "Downloading" with a couple of torrents), to "All" with 1800 torrents. This takes about 350-400ms on my machine.

Thats because it has to do this for most of the torrents:

row[filter_column] = True

Also loading the list when first connecting was crazy slow because of a very inefficient way of adding the rows to the liststore. Adding the rows is now down from about 3 seconds to 20-30ms.

The two most important changes for the speed increase is:

• Test for the value row[filter_column] before setting a new value.

 if row[filter_column] is True:
   row[filter_column] = False

 if row[filter_column] is False:
   row[filter_column] = True

• The first try/catch in the for-loop (for row in self.liststore)

Since most updates are of the same list, we now expect the torrent_id to be in the status dictionary, instead of first checking if it is.

Optimized update_view and add_rows on 1-3-stable: ​https://github.com/bendikro/deluge/commit/762ad783e2b4d22c74955ddd0bbe86fc13d8550f

If anyones interested, a version with time tests and prints is in the branch 1.3-dev-torrentview_time_test.

Connecting to the daemon and opening the preferences gives this error in the terminal:

  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 551, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/home/bro/programmer/deluge/deluge/deluge/ui/gtkui/preferences.py", line 1105, in on_fail
    if failure.value.exception_type == 'NotAuthorizedError':
exceptions.AttributeError: 'NotAuthorizedError' object has no attribute 'exception_type'

Apparently there is authentication issues causing some code to run that hasn't been tested lately.