WebUI uses 100% CPU loading TorrentInfo for thousands of files

[dalek]

Hi,

Just switched to Deluge from utorrent/linux and would like to say thanks for making it open source.

The problem I have is that when I initially connect with the WebUI (or the console-ui) the process sits at 100% CPU for tens of minutes.

I'm only seeding a handful of torrents via a private tracker but that equates to around 300,000 files on disk contained in those few torrents.

I ran the WebUI with --profile and it looks like the cause is the TorrentInfo?() routine in deluge/ui/common.py.

The problem I think is that when the WebUI gets a connection, it loads in to RAM every filename contained in each torrent. This causes RAM to be used excessively (my deluge-webui process gets to ~200MB). On top of that, for each filename, it runs through a UTF decoding routine in rencode.py which seems to be burning the CPU.

After tens of minutes, when the filename info is loaded, the WebUI comes back to life. That is, unless you click on the 'files' tab for one of the torrents with tens of thousands of files, then it hangs for ages. (I guess as expected).

I'm not a python programmer, but to fix this issue I think you could:

• change class deluge.ui.common.TorrentInfo? to not load filename/filedata info when the class is created.
• instead, load filename/filedata info on demand via function in that class.
• instead of loading _all_ filename/filedata info, just load say 10-20, with a flag indicating more filename/filedata info is available if there are more files than that in the torrent.
• in the WebUI, only load 10-20 files at a time, and present a 'next' button to load the next 10-20 filenames in the 'file' tab.

Although I have only a few torrents but with hundreds of thousands of files, I think this problem is similar to others in the tracker where people have several thousand torrents, maybe with less files in each.

I'm running on Debian testing, 32bit:

libpython2.7 2.7.2-8 python2.7 2.7.2-8 python-twisted-{bin,core} 11.0.0-2 python-twisted-web 11.0.0-1 deluge-{console,common,web}, deluged 1.3.3-2

Attached is the profile output. I can provide the profile file if needed. It was run during the first half minute or so of the webui spinning at 100% CPU. It continues like this for tens of minutes (I can time it if needed):

>>> import hotshot, hotshot.stats
>>> stats = hotshot.stats.load("/opt/deluge/.config/deluge/deluge-web.profile")
>>> stats.strip_dirs()
<pstats.Stats instance at 0x8ebd6ac>
>>> stats.sort_stats('time', 'calls')
<pstats.Stats instance at 0x8ebd6ac>
>>> stats.print_stats(20
... )
         54843249 function calls (51486080 primitive calls) in 126.991 seconds

   Ordered by: internal time, call count
   List reduced from 579 to 20 due to restriction <20>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
 13996456   23.969    0.000   37.435    0.000 rencode.py:220(f)
3356933/101   22.898    0.000  100.112    0.991 rencode.py:263(f)
      162   22.732    0.140  126.051    0.778 selectreactor.py:93(doSelect)
 16792062   16.913    0.000   16.913    0.000 utf_8.py:15(decode)
      402   14.208    0.035  100.024    0.249 rencode.py:171(decode_list)
  2795602    8.630    0.000   12.077    0.000 rencode.py:153(decode_string)
  6312624    3.894    0.000    3.894    0.000 rencode.py:251(f)
  4576888    3.849    0.000    3.849    0.000 rencode.py:135(decode_intl)
  4311445    3.488    0.000    3.488    0.000 rencode.py:131(decode_inth)
      105    2.659    0.025  102.783    0.979 client.py:128(dataReceived)
  1508477    1.160    0.000    1.160    0.000 rencode.py:143(decode_float32)
  1161239    1.039    0.000    1.039    0.000 rencode.py:139(decode_intq)
        1    0.920    0.920    0.922    0.922 config.py:393(save)
      127    0.449    0.004  103.301    0.813 tcp.py:443(doRead)
      168    0.053    0.000  100.109    0.596 rencode.py:178(decode_dict)
       41    0.037    0.001    0.037    0.001 __init__.py:827(flush)
    14030    0.012    0.000    0.012    0.000 rencode.py:127(decode_intb)
        5    0.007    0.001    0.007    0.001 gettext.py:343(gettext)
        2    0.004    0.002    0.005    0.002 sre_compile.py:301(_optimize_unicode)
      162    0.003    0.000    0.936    0.006 base.py:755(runUntilCurrent)


<pstats.Stats instance at 0x8ebd6ac>

Thanks again for this client! Any help on the above would be appreciated.

Regards, Dale

[bro]

Add bro to cc

[HellFire]

This also applies to the GTK UI too. File lists should be loaded on demand, not on connect. I tried switching from rtorrent and had to switch back because the amount of files loaded made the WebUI unusable and the GTK UI took over 10 minutes to connect and was quite unstable.