Opened 8 years ago

Closed 8 years ago

Last modified 7 years ago

#1076 closed bug (Fixed)

[webui] 1.2.x - impossible to reverse proxy with apache

Reported by: orbisvicis Owned by: damoxc
Priority: critical Milestone: 1.3
Component: Web UI Version: 1.2.0_dev
Keywords: webui, regression Cc:

Description

The 1.1.x webui included a command-switch to dynamically change the webroot - that is, prefix all urls and redirects with a base url (did it work for all headers/cookies, or only redirects?). See http://forum.deluge-torrent.info/viewtopic.php?f=7&t=10165&st=0&sk=t&sd=a#p53605

1.2.x does not offer this, and because 1.2.x uses ajax/xmlhttprequest it is not possible to reverse proxy deluge-web behind apache, for several reasons:

1) Javascript does not preserve the REFERER Header. Apache mod_rewrite rules such as the following will no longer work. RewriteCond? %{HTTP_REFERER} protocol://some.website.uri:port/deluge/.* RewriteCond? %{REQUEST_URI} ^/deluge/.* RewriteRule? /(.*) /deluge/$1 [PT]

(notice that without tracking the Referer, Rewrite rules most likely would conflict with other webapps)

2) Apache substitute filters are somewhat broken. The following will not work (to rewrite the urls in the xml[ajax],css,javascript,and html)

FilterProvider? text_inflate inflate resp=Content-Type $text FilterChain? +text_inflate

FilterProvider? html_sed sed resp=Content-Type $html FilterChain? +html_sed OutputSed? s#url(["']\(["']\{1,\}\)["'])#url(/deluge\1)#

FilterProvider? js_sed sed resp=Content-Type $javascript FilterChain? +js_sed OutputSed? s#'\(\(/[a-zA-Z]\{1,\}\)\{1,\}/[a-zA-Z]\{1,\}\.[a-zA-Z]\{2,3\}\)'#'/deluge\1'#

FilterProvider? xml_sed sed resp=Content-Type $xml FilterChain? +xml_sed OutputSed? s#\(\(/[a-zA-Z]\{1,\}\)\{1,\}/[a-zA-Z]\{1,\}\.[a-zA-Z]\{2,3\}\)#/deluge\1#

FilterProvider? css_sed sed resp=Content-Type $css FilterChain? +css_sed OutputSed? s#url(\([)]\{1,\}\))#url(/deluge\1)#

FilterProvider? text_deflate deflate resp=Content-Type $text FilterChain? +text_deflate

(notice in this case the limitation with mod_proxy_html: it does not parse inline css. That's why it is also necessary to rewrite the html.)

This is why I'm asking for a 1.2.x re-implementation of this very critical 1.1.x feature.

Change History (24)

comment:1 Changed 8 years ago by orbisvicis

The 1.1.x webui included a command-switch to dynamically change the webroot - that is, prefix all urls and redirects with a base url (did it work for all headers/cookies, or only redirects?). See http://forum.deluge-torrent.info/viewtopic.php?f=7&t=10165&st=0&sk=t&sd=a#p53605

1.2.x does not offer this, and because 1.2.x uses ajax/xmlhttprequest it is not possible to reverse proxy deluge-web behind apache, for several reasons:

  • Javascript does not preserve the REFERER Header. Apache mod_rewrite rules such as the following will no longer work.
    RewriteCond %{HTTP_REFERER} protocol://some.website.uri:port/deluge/.*
    RewriteCond %{REQUEST_URI} !^/deluge/.*
    RewriteRule ^/(.*) /deluge/$1 [PT]
    

(notice that without tracking the Referer, Rewrite rules most likely would conflict with other webapps)

  • Apache substitute filters are somewhat broken. The following will not work (to rewrite the urls in the xml[ajax],css,javascript,and html)
FilterProvider text_inflate inflate resp=Content-Type $text
FilterChain +text_inflate

FilterProvider html_sed sed resp=Content-Type $html
FilterChain +html_sed
OutputSed s#url(["']\([^"']\{1,\}\)["'])#url(/deluge\1)#

FilterProvider js_sed sed resp=Content-Type $javascript
FilterChain +js_sed
OutputSed s#'\(\(/[a-zA-Z]\{1,\}\)\{1,\}/[a-zA-Z]\{1,\}\.[a-zA-Z]\{2,3\}\)'#'/deluge\1'#

FilterProvider xml_sed sed resp=Content-Type $xml
FilterChain +xml_sed
OutputSed s#\(\(/[a-zA-Z]\{1,\}\)\{1,\}/[a-zA-Z]\{1,\}\.[a-zA-Z]\{2,3\}\)#/deluge\1#

FilterProvider css_sed sed resp=Content-Type $css
FilterChain +css_sed
OutputSed s#url(\([^)]\{1,\}\))#url(/deluge\1)#

FilterProvider text_deflate deflate resp=Content-Type $text
FilterChain +text_deflate

(notice in this case the limitation with mod_proxy_html: it does not parse inline css. That's why it is also necessary to rewrite the html.)

This is why I'm asking for a 1.2.x re-implementation of this very critical 1.1.x feature.

comment:2 Changed 8 years ago by damoxc

  • Milestone changed from 1.2.0 to 1.3.0

comment:3 Changed 8 years ago by orbisvicis

What is needed to accomplish this?

  1. web.Web.__init__ ::add option to specifiy prefix
  2. web.Web.__init__ ::pass prefix
    • ? -> what is web.WebUI for?
  3. server.TopLevel.__init__ ::modify putChild() calls
  4. unhardcode all url links in templates
  5. maybe move static content (js,etc?) to template
    • ? -> where/how are template variables such as ${stylesheet} defined?

btw 1.3.0 seems far in the future..

comment:4 Changed 8 years ago by damoxc

It's going to require quite a few changes, barely any of the html is template generated, it is all created via javascript, so will need to ensure that all paths in that are honouring the prefix. And if you look at the roadmap you will see that 1.3 is only 2 months away. 1.2 was the big change and after this we are having smaller release cycles.

comment:5 Changed 8 years ago by damoxc

You'll be pleased to hear that this is now mostly complete for 1.3. There are probably still some minor issues to iron out but it is working fine for the most part.

comment:6 Changed 8 years ago by damoxc

  • Resolution set to fixed
  • Status changed from new to closed

From my limited testing this is now possible, so marking as fixed.

comment:7 Changed 8 years ago by orbisvicis

Sweet, thanks. I've just checked out the trunk from svn, but I can't figure out how to use it. I don't see any webroot options in web/web.py.

Once I get it working, I'll take it for a test-drive, maybe find new ways to improve the web-apache interface.

comment:8 Changed 8 years ago by damoxc

We've switched over to using git now, take a look at GitRepo, and then there are tarballs available over on http://git.deluge-torrent.org.

There's no config param, it's just an entry in the web.conf. I was thinking of perhaps adding it as a switch to deluge-web, would this be useful?

comment:9 Changed 8 years ago by damoxc

I had kinda hoped to do something clever to auto-detect if the UI was being requested via a proxy, but there was no easy way I could see to get the original url, do you know if that's possible as it would be rather nice if I could do that?

comment:10 Changed 8 years ago by orbisvicis

Yeah, the switch would be helpful, so I can start the deluge web interface as usual, from /etc/{init.d,rc.d}/ scripts with the correct webroot: "deluge-web --prefix="/deluge" ... etc.

As for the clever auto-detect thing, I think the concrete command-line switch should always take precedence, but would this do the trick? ProxyPass? /deluge/ http://127.0.0.1:8112/?webroot="/deluge" After which the deluge-web appends this string to all links. However, I'm not sure if the query string is preserved in headers (http-redirects etc)

comment:11 Changed 8 years ago by damoxc

Nah unfortunately it's not. I was hoping you could do something like:

SetEnv X-Original-Path "...."

but unfortunately that requires a patch on Apache so isn't really ideal.

Query strings also aren't preserved, I gave that a try. Or not at least with the basic ProxyPass?.

I'll let you know when I've added the command line switch.

comment:12 Changed 8 years ago by orbisvicis

no, but mod_headers can manually set headers, for example:

RequestHeader set X-Deluge-Webroot "...."

In a <Location> block the header should be set on all requests, even redirects.

Assuming the URLs are initiated on startup would changing them affect the global state of the application?

  • Say an initial rogue request header set "X-Deluge-Webroot" to "/impossible/to/guess/path"
  • Then a request header without "X-Deluge-Webroot" was received

Would clients still be able to access content at http://127.0.0.1:8112/original_path/ (or would the webroot default to the last "X-Deluge-Webroot" received?)

comment:13 Changed 8 years ago by damoxc

No no it would be on a per-request basis, I've changed all the resources to be relative paths so the base only needs to be set on the html page.

I've added the -b/--base switch to deluge-web now so you are able to set it there as well as by the config file.

I'll see if mod_headers will work!

comment:14 Changed 8 years ago by damoxc

Yeah it's possible to do via mod_headers, latest git you can now set the base via X-Deluge-Base and it dynamically switches it on that.

comment:15 Changed 8 years ago by orbisvicis

Hehe that header parameter is incredibly cool.

Two suggestions:

  1. Validate the base URI so that both pairs of options work:
  • -b "/deluge"

RequestHeader? set X-Deluge-Base "/deluge"

  • -b "/deluge/"

RequestHeader? set X-Deluge-Base "/deluge/"

  1. Add a switch to deluge-web toggling the header parameter availability, perhaps "--enable-header-base" or similar.

comment:16 Changed 8 years ago by orbisvicis

I don't know how near complete 1.3 is, but this is logged when I click on the "Add" button:

[error] proxy: Error reading from remote server returned by /deluge/json, referer: https://my.website.net:myport/deluge/
[error] client denied by server configuration: proxy:http://127.0.0.1:8112/json, referer: https://my.website.net:myport/deluge/
[error] (70007)The timeout specified has expired: proxy: error reading status line from remote server 127.0.0.1, referer: https://my.website.net:myport/deluge/

with accompanying:

"POST /deluge/json HTTP/1.1" 502 492 "https://my.website.net:myport/deluge/" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2) Gecko/20100207 Namoroka/3.6"

comment:17 Changed 8 years ago by damoxc

Yeah good idea, should probably validate it. I'm thinking check for preceding / and add it if that's missing, and check for a tailing slash and add it if that's missing.

As for that error, I know that one ajax request will quite often timeout by design, I swapped the events checking from polling to a more "push"-ey style system where it sends the request and it stays connected until the server receives an event or the request times out.

So is there anyway you can check which request it is timing out??

comment:18 Changed 8 years ago by orbisvicis

Good call.

I'd like to check, but I can no longer get a connection between deluged and deluge-web. I'm accessing the web-interface via ssh port-tunnelling - no apache proxying. Not sure what to do but I tried deleting ~/.config/deluge/

from deluged

...
[INFO    ] 14:57:47 rpcserver:219 Deluge client disconnected: Connection was closed cleanly.
[INFO    ] 14:57:49 rpcserver:199 Deluge Client connection made from: 127.0.0.1:38721
[INFO    ] 14:57:49 rpcserver:219 Deluge client disconnected: Connection was closed cleanly.
[INFO    ] 14:57:51 rpcserver:199 Deluge Client connection made from: 127.0.0.1:38723
...

from deluge-web

...
[INFO    ] 14:57:51 client:224 Connection lost to daemon at 127.0.0.1:58846 reason: Connection was closed cleanly.
[INFO    ] 14:57:53 client:217 Connecting to daemon at 127.0.0.1:58846..
[INFO    ] 14:57:53 client:121 Connected to daemon at 127.0.0.1:58846..
[INFO    ] 14:57:53 client:224 Connection lost to daemon at 127.0.0.1:58846 reason: Connection was closed cleanly.
...

When/if I get this working i'll see if the request that times out is logged by deluge-web. If not I can sniff it out with wireshark.

comment:19 Changed 8 years ago by orbisvicis

Nevermind. Reboot == working

comment:20 Changed 8 years ago by orbisvicis

Double nevermind. Seems whenever I start the components as "deluged -d -L info" -> "deluge-web -L info" they don't communicate. I'm not sure why.

However, when started normally, I found that the request that times out (nonfunctional "Add" button) has nothing to do with the apache reverse proxy - it also times out @ 127.0.0.1:8112.

This is what I could see from wireshark:

I'm pretty sure this is the request that times out. It's frequency in the logs matches the frequency of me clicking the "Add" button. Doesn't seem to have anything to do with "adding" torrents though.

{"method":"web.get_host_status","params":["0f76211d03b29f3990b9f6a54eb0d655a1c6e231"],"id":87}

This seems to be the update loop:

{"method":"web.update_ui","params":[["queue","name","total_size","state","progress","num_seeds","total_seeds","num_peers","total_peers","download_payload_rate","upload_payload_rate","eta","ratio","distributed_copies","is_auto_managed","time_added","tracker_host"],{}],"id":84}

These methods only occured at the beginning of the conversation, probably auth stuff

{"method":"web.connect","params":["0f76211d03b29f3990b9f6a54eb0d655a1c6e231"],"id":20}
{"method":"auth.check_session","params":[],"id":1}

Another thing, don't know if this is interesting, but this is an error generated when restarting the web server while a client is connected:

[ERROR   ] 15:32:40 json_api:212 Error calling method `web.get_host_status`
[ERROR   ] 15:32:40 json_api:213 'NoneType' object is not iterable
Traceback (most recent call last):
  File "/opt/python-virtual-deluge/lib/python2.5/site-packages/deluge-1.2.900_dev-py2.5.egg/deluge/ui/web/json_api.py", line 204, in _handle_request
    result = self._exec_local(method, params, request)
  File "/opt/python-virtual-deluge/lib/python2.5/site-packages/deluge-1.2.900_dev-py2.5.egg/deluge/ui/web/json_api.py", line 171, in _exec_local
    return meth(*params)
  File "/opt/python-virtual-deluge/lib/python2.5/site-packages/deluge-1.2.900_dev-py2.5.egg/deluge/ui/web/json_api.py", line 693, in get_host_status
    (host_id, host, port, user, password) = self.get_host(host_id)
TypeError: 'NoneType' object is not iterable

comment:21 Changed 8 years ago by orbisvicis

I can get you the TwistedServer? responses but I think they're all compressed and stuff.

comment:22 Changed 8 years ago by orbisvicis

Found some misplaced resources

File does not exist: /usr/share/apache2/icons/add.png, referer: https://aequorin.homeunix.net:62389/deluge/
File does not exist: /usr/share/apache2/icons/error.png, referer: https://my.website.net:myport/deluge/
File does not exist: /usr/share/apache2/icons/remove.png, referer: https://my.website.net:myport/deluge/

comment:23 Changed 8 years ago by damoxc

Fixed all the remaining static resources now!

I'll check out the request timeout issue also.

comment:24 Changed 7 years ago by orbisvicis

Upgraded to the current dev trunk - 1.3.900 - and everything is working. Thanks!

Note: See TracTickets for help on using tickets.