Opened 10 years ago

Closed 10 years ago

Last modified 10 years ago

#2127 closed bug (WontFix)

select() reactor has problems with >1024 File Descriptors

Reported by: underscor Owned by:
Priority: major Milestone: Future
Component: Core Version: 1.3.5
Keywords: Cc:

Description

The select reactor has issues with more than 1024 file descriptors, which leads to errors like this when having a large number of connections. I'm not sure what the best path to take in this situation is. Either we could integrate with twistd, which does it's own intelligent reactor selection, or write logic to determine whether we're on Linux, and if so, opt for the poll() or epoll() reactor, and select() otherwise.

Traceback (most recent call last):

File "/usr/lib/pymodules/python2.6/deluge/main.py", line 226, in start_daemon

Daemon(options, args)

File "/usr/lib/pymodules/python2.6/deluge/core/daemon.py", line 176, in init

reactor.run()

File "/usr/lib/python2.6/dist-packages/twisted/internet/base.py", line 1170, in run

self.mainLoop()

File "/usr/lib/python2.6/dist-packages/twisted/internet/base.py", line 1182, in mainLoop

self.doIteration(t)

--- <exception caught here> ---

File "/usr/lib/python2.6/dist-packages/twisted/internet/selectreactor.py", line 104, in doSelect

[], timeout)

exceptions.ValueError?: filedescriptor out of range in select() Unhandled error in Deferred: Traceback (most recent call last):

File "/usr/lib/python2.6/dist-packages/twisted/internet/base.py", line 1031, in connectionFailed

self.factory.clientConnectionFailed(self, reason)

File "/usr/lib/python2.6/dist-packages/twisted/web/client.py", line 350, in clientConnectionFailed

self.deferred.errback(reason)

File "/usr/lib/python2.6/dist-packages/twisted/internet/defer.py", line 307, in errback

self._startRunCallbacks(fail)

File "/usr/lib/python2.6/dist-packages/twisted/internet/defer.py", line 354, in _startRunCallbacks

self._runCallbacks()

--- <exception caught here> ---

File "/usr/lib/python2.6/dist-packages/twisted/internet/defer.py", line 371, in _runCallbacks

self.result = callback(self.result, *args, kw)

File "/usr/lib/pymodules/python2.6/deluge/core/rpcserver.py", line 316, in on_fail

sendError()

File "/usr/lib/pymodules/python2.6/deluge/core/rpcserver.py", line 252, in sendError

"".join(traceback.format_tb(exceptionTraceback)))

File "/usr/lib/pymodules/python2.6/deluge/core/rpcserver.py", line 196, in sendData

self.transport.write(zlib.compress(rencode.dumps(data)))

File "/usr/lib/pymodules/python2.6/deluge/rencode.py", line 391, in dumps

encode_func[type(x)](x, r)

File "/usr/lib/pymodules/python2.6/deluge/rencode.py", line 338, in encode_list

encode_func[type(i)](i, r)

File "/usr/lib/pymodules/python2.6/deluge/rencode.py", line 338, in encode_list

encode_func[type(i)](i, r)

exceptions.KeyError?: <type 'instance'>

Change History (3)

comment:1 Changed 10 years ago by underscor

  • Component changed from other to core

comment:2 Changed 10 years ago by andar

  • Resolution set to wontfix
  • Status changed from new to closed

epoll is the default reactor for Linux kernel based systems in the latest version of Twisted.

See: http://twistedmatrix.com/trac/browser/tags/releases/twisted-12.1.0/twisted/internet/default.py

There is no need for Deluge to do anything about this as we just simply use the default reactor from Twisted.

If you're currently having a problem with the number of open file descriptors, than I suggest that you raise the limit for your system. Most systems will have a default limit of 1024. You can use ulimit -n to set this value.

comment:3 Changed 10 years ago by underscor

The problem was an outdated twisted. Twisted 10.0.0 is the default installed by the package manager in Lucid Lynx. Unfortunately, twisted's older versions default to select() on Linux.

Thanks for looking at it. Manually backporting the twisted package from precise did the trick.

Note: See TracTickets for help on using tickets.