rencode changes str/unicode types

[gazpachoking]

No matter whether a bytestring or unicode object is put into rencode.dumps, when it is loaded it is turned into a bytestring if there were only ascii characters, and a unicode object if there were any non-ascii characters. This causes the problem of having no idea what type of object will be passed into rpc exported methods.

The easiest fix would be to have rencode always return unicode objects for strings, not sure if it's worth it to have it produce the same type of object that was passed in.

[gazpachoking]

Example:

from deluge import rencode
def ident(x):
    print repr(rencode.loads(rencode.dumps(x)))

ident(u'a') # unicode in, str out
ident('a') # str in str out
ident(u'\xf8') # unicode in unicode out
ident(u'\xf8'.encode('utf8')) # str in unicode out

### Result
'a'
'a'
u'\xf8'
u'\xf8'

[gazpachoking]

The other concern is the behavior of andar's cython implementation of rencode. Haven't tested that, but maybe it would be easier to fix it at the rpc level rather than rencode level.

[gazpachoking]

Simplest possible fix I can think of. Always output unicode objects from rencode. (Unless a bytestring in some other encoding than utf8 was input. Don't do that. Maybe we could even put in an assert for that to catch any possible problems.)

deluge/rencode.py

@@ -160 +160 @@
     colon += 1
     s = x[colon:colon+n]
     try:
-        t = s.decode("utf8")
-        if len(t) != len(s):
-            s = t
+        s = s.decode("utf8")
     except UnicodeDecodeError:
         pass
     return (s, colon+n)

[gazpachoking]

We need to update rencode to always output strings as utf-8 encoded byte strings. (this is in line with the cython implementation) We should change the rpc code to always decode these before calling the exported methods.

[gazpachoking]

Fixed. Rpc calls now deliver all string arguments as unicode objects. Relevant commits: ​6b5cf3396 ​2187cef14

[Cas]

Milestone renamed