Proxy Servlet for AJAX requests to multiple, remote servers

AJAX is hot, great and will make the world a better place. We all know that. However, AJAX has some limitations. AJAX is about making additional HttpRequests from the browser once a ‘normal’ request has been satisfied and an HTML document is loaded. Once the initial load is complete, the browser may fire additional requests for resources such as Images, CSS documents and JavaScript libraries. In addition, we can program even more additional requests: the AJAX-requests. Using AJAX we can asynchronuously, in the background, retrieve additional data and content to further enrich, flesh out and embellish our page. These resources can be simlpe text, XML, WebService replies, full blown (X)HTML or basically any character based content that HttpRequests may return.

The one BIG limitation with browsers and AJAX is that, in general, the AJAX requests must be to the same server that server the original HTML document. So your page cannot fire AJAX requests. That means that if you want to use additional content that is not server by your own web server, you have a problem. Invoking a remote WebService to return the weather conditions, getting an RSS-feed with headlines, scraping HTML content from another website: it will not fly. A typical error to run into: uncaught exception: Permission denied to call method XMLHttpRequest.open

So what can you do if you really want to use content from other origins – which is quite likely by the way? Basically there seem to be a few options, some of which are limited to specific browsers:

• Digitally sign your scripts. In Firefox you can apply a digital signature to your script and those scripts will then be considered “trusted” by the browser. Firefox will then let you make XMLHttpRequests to any domain. However, no other browsers support script signing at this time, so this solution is of limited use. See: Signed Scripts in Mozilla
• Use JSON and dynamic <script> tags instead of XML and XMLHttpRequest. You can get around the browser security problem altogether by making your web services request directly inside a <script> tag.
• Use an alternative XmlHttpRequestAPI based on Greasemonkey’s XMLHttpRequest API, which is more powerful than that available to web pages. It bypasses the same origin policy, thus allowing user scripts to truly compose and remix websites. See: XMLHttpRequest – Security Bypass. Note: by-passing security is perhaps not such a great idea…
• Use a proxy do make the requests for you: if you cannot make the AJAX requests to remote sites because all requests have to go to the orginal site that served the HTML document, than we should make our AJAX request to that server, to a proxy component that reroutes the request to where we want it to go and returns the results to us in the client! This proxy can be any server side object – PHP, Java Servlet, Perl or Python – that can receive our AJAX request, turn it into a remote request, receive the response from the remote site and pass that response back to the client. The Resources have a link to description of PHP proxy.

The picture shows the setup, where in this case the AJAX request is targeted at one of the Yahoo! WebServices.

In this article we will develop a Servlet based Proxy to handle our AJAX requests. Thus, we should be able to develop an HTML document that uses AJAX requests to plugin content from remote sources, something that normal AJAX would not allow.

Developing the Proxy Servlet

What we need is simple servlet that is called by the client. The client should provide in its request the real URL that the proxy should invoke on its behalf. It should also send along all parameters that the proxy should include in the remote call. I started out with the excellent article: HttpProxyServlet version 1.1. Since that code is distributed under a commercial license, I AM NOT ALLOWED TO DECOMPILE THAT CODE (which is what I did in an earlier version of this article). While I am looking for an alternative implementation, this article is currently incomplete.

..

This Servlet is configured in web.xml as follows:

<?xml version = '1.0' encoding = 'windows-1252'?>
<!DOCTYPE web-app PUBLIC "-//Sun Microsystems, Inc.//DTD Web Application 2.3//EN" "http://java.sun.com/dtd/web-app_2_3.dtd">
<web-app>
  <servlet>
    <servlet-name>HttpProxy</servlet-name>
    <servlet-class>nl.amis.proxy.HttpProxyServlet</servlet-class>
  </servlet>
  <servlet-mapping>
    <servlet-name>HttpProxy</servlet-name>
    <url-pattern>/proxy</url-pattern>
  </servlet-mapping>
  <mime-mapping>
    <extension>html</extension>
    <mime-type>text/html</mime-type>
  </mime-mapping>
  <mime-mapping>
    <extension>txt</extension>
    <mime-type>text/plain</mime-type>
  </mime-mapping>
</web-app>

You can invoke the HttpProxy from the browser using a simple URL like: http://yourhost:port/webapp/proxy?remotehost=www.theserverside.com/index.tss. The result is that the servlet will send a request to the URL passed in in the remotehost parameter and will return the response it received to the browser.

If we can have browser make explicit calls to this servlet, we can also make implicit calls in the shape of AJAX-requests:

  function loadXMLDoc(url) {
    // branch for native XMLHttpRequest object
    if (window.XMLHttpRequest) {
        req = new XMLHttpRequest();
        req.onreadystatechange = processReqChange;
        req.open("GET", "proxy?remotehost="+url, true); // the third parameter - true - specified ASYNCHRONOUS processing i.e. not waiting for the response!
        req.send(null);
    // branch for IE/Windows ActiveX version
    } else if (window.ActiveXObject) {
        req = new ActiveXObject("Microsoft.XMLHTTP");
        if (req) {
            req.onreadystatechange = processReqChange;
            req.open("GET", "proxy?remotehost="+url, true); // the third parameter - true - specified ASYNCHRONOUS processing i.e. not waiting for the response!
            req.send();
        }
    }
  }

This piece of JavaScript is part of the AjaxProxyClient.html; a static document that allows the user to choose some content-provider. When a selection is made, the loadXMLDoc function is called with a specific URL:

          <h3>Sites and RSS Feeds</h3>
          <SELECT ONCHANGE="loadXMLDoc(this.options[this.selectedIndex].value);">
            <OPTION VALUE="technology.amis.nl/blog/">AMIS Technology Blog</OPTION>
            <OPTION VALUE="news.bbc.co.uk/rss/newsonline_world_edition/front_page/rss091.xml">BBC World News</OPTION>
            <OPTION VALUE="www.theserverside.com/index.tss">The Server Side</OPTION>
            <OPTION VALUE="www.orablogs.com/orablogs/rss.xsql">OraBlogs</OPTION>
            <OPTION VALUE="www.nu.nl">NU.NL</OPTION>
          </SELECT>

The function makes the Ajax request and when the response is received from the proxy-servlet, it is pasted into the TEXTAREA and the DIV:

When the user selects a different source, a new Ajax Request is sent, possibly to an entirely different server – cross domain and all – and the content when received is displayed in the TEXTAREA and the DIV. Note: the AJAX request itself is not sent to a different server: it is also sent to the proxy servlet; the proxy in turn makes a server side request to an entire different server.

Note: there still seem to be some issues with this solution: some urls seem to cause problems, for example when they are are dynamically redirected. I have also experienced some problems with URLs that did not seem to return a correct content-type. This obviously needs attention when using the proxy servlet for real.

Resources

HOWTO: Use a Web Proxy for Cross-Domain XMLHttpRequest Calls – article on creating a proxy for Web Requests using PHP. Also provides some pointers for alternative solutions