Opened 9 years ago

Last modified 14 months ago

#1999 new bug

WebUI uses 100% CPU loading TorrentInfo for thousands of files

Reported by: dalek Owned by: damoxc
Priority: minor Milestone: performance
Component: Web UI Version: 1.3-stable
Keywords: cpu web gtk Cc: dale@…, bro, marsjaninzmarsa@…, krichter@…



Just switched to Deluge from utorrent/linux and would like to say thanks for making it open source.

The problem I have is that when I initially connect with the WebUI (or the console-ui) the process sits at 100% CPU for tens of minutes.

I'm only seeding a handful of torrents via a private tracker but that equates to around 300,000 files on disk contained in those few torrents.

I ran the WebUI with --profile and it looks like the cause is the TorrentInfo?() routine in deluge/ui/

The problem I think is that when the WebUI gets a connection, it loads in to RAM every filename contained in each torrent. This causes RAM to be used excessively (my deluge-webui process gets to ~200MB). On top of that, for each filename, it runs through a UTF decoding routine in which seems to be burning the CPU.

After tens of minutes, when the filename info is loaded, the WebUI comes back to life. That is, unless you click on the 'files' tab for one of the torrents with tens of thousands of files, then it hangs for ages. (I guess as expected).

I'm not a python programmer, but to fix this issue I think you could:

  • change class deluge.ui.common.TorrentInfo? to not load filename/filedata info when the class is created.
  • instead, load filename/filedata info on demand via function in that class.
  • instead of loading _all_ filename/filedata info, just load say 10-20, with a flag indicating more filename/filedata info is available if there are more files than that in the torrent.
  • in the WebUI, only load 10-20 files at a time, and present a 'next' button to load the next 10-20 filenames in the 'file' tab.

Although I have only a few torrents but with hundreds of thousands of files, I think this problem is similar to others in the tracker where people have several thousand torrents, maybe with less files in each.

I'm running on Debian testing, 32bit:

libpython2.7 2.7.2-8 python2.7 2.7.2-8 python-twisted-{bin,core} 11.0.0-2 python-twisted-web 11.0.0-1 deluge-{console,common,web}, deluged 1.3.3-2

Attached is the profile output. I can provide the profile file if needed. It was run during the first half minute or so of the webui spinning at 100% CPU. It continues like this for tens of minutes (I can time it if needed):

>>> import hotshot, hotshot.stats
>>> stats = hotshot.stats.load("/opt/deluge/.config/deluge/deluge-web.profile")
>>> stats.strip_dirs()
<pstats.Stats instance at 0x8ebd6ac>
>>> stats.sort_stats('time', 'calls')
<pstats.Stats instance at 0x8ebd6ac>
>>> stats.print_stats(20
... )
         54843249 function calls (51486080 primitive calls) in 126.991 seconds

   Ordered by: internal time, call count
   List reduced from 579 to 20 due to restriction <20>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
 13996456   23.969    0.000   37.435    0.000
3356933/101   22.898    0.000  100.112    0.991
      162   22.732    0.140  126.051    0.778
 16792062   16.913    0.000   16.913    0.000
      402   14.208    0.035  100.024    0.249
  2795602    8.630    0.000   12.077    0.000
  6312624    3.894    0.000    3.894    0.000
  4576888    3.849    0.000    3.849    0.000
  4311445    3.488    0.000    3.488    0.000
      105    2.659    0.025  102.783    0.979
  1508477    1.160    0.000    1.160    0.000
  1161239    1.039    0.000    1.039    0.000
        1    0.920    0.920    0.922    0.922
      127    0.449    0.004  103.301    0.813
      168    0.053    0.000  100.109    0.596
       41    0.037    0.001    0.037    0.001
    14030    0.012    0.000    0.012    0.000
        5    0.007    0.001    0.007    0.001
        2    0.004    0.002    0.005    0.002
      162    0.003    0.000    0.936    0.006

<pstats.Stats instance at 0x8ebd6ac>

Thanks again for this client! Any help on the above would be appreciated.

Regards, Dale

Change History (8)

comment:1 Changed 9 years ago by dalek

  • Cc dale@… added

comment:2 Changed 8 years ago by Cas

  • Milestone changed from Future to performance

comment:3 Changed 5 years ago by bro

  • Cc bro added

Add bro to cc

comment:4 Changed 4 years ago by HellFire

This also applies to the GTK UI too. File lists should be loaded on demand, not on connect. I tried switching from rtorrent and had to switch back because the amount of files loaded made the WebUI unusable and the GTK UI took over 10 minutes to connect and was quite unstable.

comment:5 Changed 3 years ago by marsjaninzmarsa

  • Cc marsjaninzmarsa@… added

comment:6 Changed 3 years ago by krichter

  • Cc krichter@… added

comment:7 Changed 14 months ago by int3l

  • Keywords web gtk added
  • Summary changed from 1.3.3 WebUI uses 100% CPU loading TorrentInfo for thousands of files to WebUI uses 100% CPU loading TorrentInfo for thousands of files
  • Version changed from 1.3.3 to 1.3.15

comment:8 Changed 14 months ago by int3l

  • Version changed from 1.3.15 to 1.3-stable
Note: See TracTickets for help on using tickets.