WebUI CPU usage goes to 100% on one core of the processor after a few days with Deluge 1.3.13 - Arch Linux

[wingnut]

I have noticed this happening several times. A restart of the web service will help for a few days, but then the same behavior occurs. This started to happen about a week ago which seems to relate to the Deluge package update on Arch Linux.

I don't know what logs you need here. I switched back to 1.3.12 for now. I'm happy to help debug this further.

[Calum]

There are no changes to the WebUI from 1.3.12 to 1.3.13 that would have any affect on performance.

You will need to provide substantially more information regarding the issue.

[Bro]

You are running WebUI separately, i.e. deluge-web, right?

If you can figure out how to reproduce the issue, ideally certain steps that will lead to the CPU usage issue, to it will be much easier to fix it.

[wingnut]

Deluged log

[wingnut]

Attachment (deluged.log.gz) added by ticket reporter.

[wingnut]

deluge-web log

[wingnut]

I reinstalled 1.3.13 and ran deluged and deluge-web overnight. As far as I know, there was no activity last night. The cooling fan was running constantly as of this afternoon which clued me in that something was wrong. Normally it runs every 20 seconds or so. One CPU core was pegged at nearly 100% when looking at the processes shown in top. I stopped the deluge-web.service and things went back to normal.

More information: I am running Arch Linux on an Odroid XU4 machine. It has all updates as of this morning.

For the next couple of days I'm going to revert back to 1.3.12 to see if things settle down.

Attached are debug logs of both deluged and deluge-web.

[wingnut]

Below is the output from top when running 1.3.13.

Again, I'll do whatever tests you want me to do. I don't know what other input that you need. About every 24 hours, I have to restart the web gui. I'm going to run a daily cron job as otherwise things are working fine.

Tasks: 156 total,   2 running, 154 sleeping,   0 stopped,   0 zombie
%Cpu0  :   0.0/2.7     3[|                                                    ]
%Cpu1  :   0.0/0.0     0[                                                     ]
%Cpu2  :   0.0/0.0     0[                                                     ]
%Cpu3  :   0.0/0.0     0[                                                     ]
%Cpu4  :   0.0/0.0     0[                                                     ]
%Cpu5  :   0.0/0.0     0[                                                     ]
%Cpu6  :  92.2/7.8   100[|||||||||||||||||||||||||||||||||||||||||||||||||||||]
%Cpu7  :   0.0/0.0     0[                                                     ]
GiB Mem : 28.9/1.943    [                                                     ]
GiB Swap:  0.0/0.000    [                                                     ]

  PID USER      PR  NI    VIRT    RES  %CPU %MEM     TIME+ S COMMAND
  290 systemd+  20   0    4.5m   1.4m   0.0  0.1   0:01.41 S  `- systemd-resol+
  291 root      20   0    2.0m   0.7m   0.0  0.0   0:00.01 S  `- agetty
  292 root      20   0    2.0m   0.7m   0.0  0.0   0:00.01 S  `- agetty
  295 root      20   0   41.0m   1.8m   0.0  0.1   1:00.49 S  `- automount
27265 root      20   0   12.6m   2.8m   0.0  0.1   6:41.02 S  `- openvpn
12783 sonarr    20   0  244.0m 152.4m   0.0  7.7  12:44.57 S  `- mono
14592 deluge    20   0  109.1m  38.4m   1.3  1.9  14:33.48 S  `- deluged
14593 deluge    20   0  118.5m 112.4m 100.0  5.6  21:27.33 R  `- deluge-web
30685 alarm     20   0    8.2m   2.9m   0.0  0.1   0:00.10 S  `- systemd
30686 alarm     20   0   19.9m   1.3m   0.0  0.1   0:00.00 S      `- (sd-pam)

I have deluge running on a Raspberry Pi 2 with Arch, 1.3.13. I am not seeing this behavior. I am also not running any torrents at all with it -- other services are mostly disabled on the box -- it's only there for testing.

I had no issues with 1.3.12 on the XU4 and it ran over the weekend with around four torrents total. I reinstalled 1.3.13 on the XU4 yesterday afternoon, it ran with one torrent and locked up just about an hour ago. I can ssh in no problem since it only affects one core.

Maybe Arch has a defective 1.3.13 package? I'll try to do it from the sources and see what happens.

Let me know what I can do to further narrow this down and I'll do it.

[Calum]

I see from logs you are using HTTPS, there is a change in WebUI that relates to that from #2782: [52e60ac5b0fb9fd]. You could test reverting that commit.

I assume you are using a fairly recent version of openssl on Arch?

[wingnut]

openssl-1.0.2.h-1-armv7h.pkg.tar.xz is what I have currently installed.

I reverted the deluge commit after a lovely introduction to reverting commits in git and even managed to recompile the Arch package afterward and install it. I restarted both the deluged.service and the deluge-web.service.

The Pi2 keeps rolling along with HTTPS but in reality it is not moving a single torrent so maybe it's a poor control group member.

I set Deluge-web up with HTTPS because I was formerly forwarding ports on my router to enable outside access. Since then, I've moved to OpenVPN. I suppose that every little bit helps though, security-wise. I'd like to keep this enabled if I could.

I will keep you posted as to my progress over the next few days. I should know something by this time tonight if things go poorly.

Thanks for the help!

[wingnut]

journalctl -b -u deluge-web

[wingnut]

I'm still having trouble after reverting that commit.

I apologize for the size of the log -- I didn't want to trim something you might need to see.

I switched back to 1.3.12 for the time being but I can easily make another change to 1.3.13 if you have a suggestion.

[Calum]

The same issue or something different? That log does show OpenSSL.SSL.Error: [('system library', 'fopen', 'Too many open files') but afaik that is not a Deluge issue.

If it is still a problem with the cpu core then the only real suggestion I have at this point is to step back through the commits between 1.3.13 and 1.3.12. Best to pick a halfway point and narrow it down from there... perhaps start with the commit [6ffe5cd] before the HTTPS fix then we will know if it's earlier or later than that.

[wingnut]

Here are the other logs if you need them.

It is a different issue now. The webgui refuses connections and has to be restarted. The CPU activity is idle. One thing that happened during this time is that my OpenVPN connection was restarted and I got a new IP address around 4:55 AM EDT August 5th. I'm not sure if that is what is causing the UDP failures to connect. It hasn't reconnected since and I still have regular network functionality on the machine.

I will try to step back through the commits as you suggest. There will be a delay as I am learning how to use GIT. It's too bad that this error takes a day or two to manifest. :/

[Halfman]

Hello,

Same issue here on Debian 8.4 with Deluge 1.3.10. Deluged process is taking 100 % of the CPU and the only way to close it is by using the kill command with a SIGTERM signal.

The command

service deluged stop

Waits forever

[wingnut]

I have stepped back to commit g7ca704b, 15 commits after 1.3.12, and I'm still getting these errors with OpenSSL.SSL.Error: [('system library', 'fopen', 'Too many open files')

This could very well be an issue with my Arch installation, but I have no clue what it might be. It is vanilla and I update it daily. It is also on an Odroid XU4 (ARM).

I keep stepping back to 1.3.12 and I have a feeling that I'll run into the same issue when I finally reach that point. I just recompiled to 7 commits after 1.3.12. I'm going to just jump to the origin the next time around, around 24 hours from now, and I should know something a day after that.

Initially, I reverted the commit you mentioned with git revert 52e60ac5b0fb9fd I stepped back by next typing git checkout <earlier commit> in the ./src/deluge/ directory and creating the package with makepkg -srei I have repeated this command each time I stepped back. I can see the commit hashes in the names of the packages that are created, so I must be doing something right. Or have I been doing something wrong?

It's funny that the standard 1.3.12 package runs just fine.

I really wish I could force this OpenSSL error more quickly, as I could do this in a much shorter time.

[wingnut]

I edited /etc/sysctl.d/99-sysctl.conf to add

# added for deluge-web
fs.file-max=204708

and /etc/security/limits.conf to add

deluge          soft    nofile          4096
deluge          hard    nofile          10240

These numbers can probably be adjusted. I wasn't sure if deluge would adjust a soft limit or not, so if not, both of those latter numbers could be 4096. I just searched for "too many open files" on the internet and tried to make sane adjustments. I am not running a mission-critical machine, and these values work for me.

I'm not sure why I had to adjust these values in the first place, but perhaps something changed in my system that made this necessary. At any rate, I am able to run the current code without any patches reversed.

At some point, I'll put these values back to their defaults and try to trace what is happening with a little help from my friends. I'm about to leave on vacation, so that won't be for at least a month.

[Calum]

We did actually track down a 'too many files open' bug #2889. Are you able to test with the latest 1-3-stable and report back?

[wingnut]

I reverted the changes I made to 99-sysctl.conf and /etc/security/limits.conf in the above post a couple of months ago and everything has been working fine. I am not experiencing the issue currently, and through an Arch update, I installed its latest version yesterday.

For me, at least, this matter is closed.

[Calum]

Oh ok excellent :)