select() reactor has problems with >1024 File Descriptors

[underscor]

The select reactor has issues with more than 1024 file descriptors, which leads to errors like this when having a large number of connections. I'm not sure what the best path to take in this situation is. Either we could integrate with twistd, which does it's own intelligent reactor selection, or write logic to determine whether we're on Linux, and if so, opt for the poll() or epoll() reactor, and select() otherwise.

Traceback (most recent call last):

File "/usr/lib/pymodules/python2.6/deluge/main.py", line 226, in start_daemon

Daemon(options, args)

File "/usr/lib/pymodules/python2.6/deluge/core/daemon.py", line 176, in init

reactor.run()

File "/usr/lib/python2.6/dist-packages/twisted/internet/base.py", line 1170, in run

self.mainLoop()

File "/usr/lib/python2.6/dist-packages/twisted/internet/base.py", line 1182, in mainLoop

self.doIteration(t)

--- <exception caught here> ---

File "/usr/lib/python2.6/dist-packages/twisted/internet/selectreactor.py", line 104, in doSelect

[], timeout)

exceptions.ValueError?: filedescriptor out of range in select() Unhandled error in Deferred: Traceback (most recent call last):

File "/usr/lib/python2.6/dist-packages/twisted/internet/base.py", line 1031, in connectionFailed

self.factory.clientConnectionFailed(self, reason)

File "/usr/lib/python2.6/dist-packages/twisted/web/client.py", line 350, in clientConnectionFailed

self.deferred.errback(reason)

File "/usr/lib/python2.6/dist-packages/twisted/internet/defer.py", line 307, in errback

self._startRunCallbacks(fail)

File "/usr/lib/python2.6/dist-packages/twisted/internet/defer.py", line 354, in _startRunCallbacks

self._runCallbacks()

--- <exception caught here> ---

File "/usr/lib/python2.6/dist-packages/twisted/internet/defer.py", line 371, in _runCallbacks

self.result = callback(self.result, *args, kw)

File "/usr/lib/pymodules/python2.6/deluge/core/rpcserver.py", line 316, in on_fail

sendError()

File "/usr/lib/pymodules/python2.6/deluge/core/rpcserver.py", line 252, in sendError

"".join(traceback.format_tb(exceptionTraceback)))

File "/usr/lib/pymodules/python2.6/deluge/core/rpcserver.py", line 196, in sendData

self.transport.write(zlib.compress(rencode.dumps(data)))

File "/usr/lib/pymodules/python2.6/deluge/rencode.py", line 391, in dumps

encode_func[type(x)](x, r)

File "/usr/lib/pymodules/python2.6/deluge/rencode.py", line 338, in encode_list

encode_func[type(i)](i, r)

File "/usr/lib/pymodules/python2.6/deluge/rencode.py", line 338, in encode_list

encode_func[type(i)](i, r)

exceptions.KeyError?: <type 'instance'>

[andar]

epoll is the default reactor for Linux kernel based systems in the latest version of Twisted.

See: ​http://twistedmatrix.com/trac/browser/tags/releases/twisted-12.1.0/twisted/internet/default.py

There is no need for Deluge to do anything about this as we just simply use the default reactor from Twisted.

If you're currently having a problem with the number of open file descriptors, than I suggest that you raise the limit for your system. Most systems will have a default limit of 1024. You can use ulimit -n to set this value.

[underscor]

The problem was an outdated twisted. Twisted 10.0.0 is the default installed by the package manager in Lucid Lynx. Unfortunately, twisted's older versions default to select() on Linux.

Thanks for looking at it. Manually backporting the twisted package from precise did the trick.