April 15, 2010

Testing JavaScript & XHR with Python

I am working on a library for Django - which I'll describe later - and for building this project I need to test the behavior of XHR on the HTML page. That is, I have a page, inject some JavaScript (that may do a request on the server), and see how it changes the document. The Django application the library is designed for creates some parts of the original page and processes some parts of the server-side XHR.

Python is great for this. I've mentioned already in this blog how I like Qt4, and it proves to be very useful again. I could not get PySide running on OS X just yet, so I opted for PyQt4.

Four tests; four evenings working with them – each having an intricate problem to solve ;) Two tests work on the local document, the other two connect to the Portlet test bench, running on localhost. These tests trigger onclick submit actions from the XHR test page, simulating a user clicking a button.

This way it is possible to test the effect of JavaScript injection onto any web page in the wild. At least the WebKit response.. Firefox could be tested by using python-gtkmozembed. QWebKit also has methods for binding Python variables to the page (it is possible for JavaScript to trigger a Python callback function) and render an image snapshot of the page.

If you wish to run the tests, download the file, start up the Rails server with the test bench and run

$ python xhr_tests.py

April 5, 2010

Comparison of Web Proxy Portlet, Portletbridge and html2jsr286

There are at least three different JSR 168 / 286 portlet projects that have a similar target: to allow custom web apps and arbitrary web content be accessible in portlets.

I am the author of "Rails-portlet" and it's alive - I just updated the setup guide. It all started when I needed to develop a webapp for Liferay, and I wanted to do it with Rails. html2jsr286 is not an indication of the NIH syndrome, I just now learned of their existance. I inspected their source code, and this is a quick run-down of their features and pondering of future direction.

These are the common features in all three projects:

  • Getting content from a downstream site
  • Proxying of remote resources (e.g. images, Flash etc.)
  • Regular expression defining which URL's are in the portlet and which should be regular links
  • Rewriting CSS urls
  • Moving Javascript and CSS links out of the head
  • Session cookie support
All three have the same shortcoming concerning Ajax: the urls are not being rewritten so that XHR from the browser is not catched by the portlet.

WebProxyPortlet is being developed mainly for uPortal, and

..it provides mechanisms for connecting to and rendering HTML with options for clipping, maintaining session information, handling cookies. Proxied content is rendered within the portlet window. Web Proxy Portlet is often used to incorporate web content or applications that are built and run in non-Java environments allowing a site flexibility for integrating with many different technologies.

Sources of portletbridge

$ mkdir portletbridge
$ cd portletbridge/
$ cvs -z3 -d:pserver:anonymous@portletbridge.cvs.sourceforge.net:/cvsroot/portletbridge co -P .
Sources of WebProxyPortlet
$ svn co https://www.ja-sig.org/svn/portlets/WebproxyPortlet/trunk/ WebproxyPortlet
Sources of html2jsr286
$ git clone git@github.com:lamikae/html2jsr286.git

My impression after a while looking through the sources is that WebProxyPortlet is the most advanced, and has a number of nice features, especially in ways of user authentication and portlet configuration through the user interface.

Considering code cohesion and the number of Java classes: WebProxyPortlet has 58, Portletbridge 51 and html2jsr286 only 10 Java source files. Here are links to their main view methods:

void renderContent(final RenderRequest request, final RenderResponse response) in WebProxyPortlet.java
void doView(final RenderRequest request, final RenderResponse response) in BridgeViewPortlet.java
void render(RenderRequest request, RenderResponse response) in Rails286Portlet.java

WebProxyPortlet works in JSR 168 containers and supports a variety of authentication techniques, handles caching, has pluggable Java classes for custom request filtering and page processing. On top of this is uses a whole lot of JavaBeans and JSPs that make the portlet configurable from the GUI in the portlet config mode. I am not sure how much runtime overhead this accouts to, and how useful it is to let users select the HTML parser the portlet is going to use. Page processing is implemented by chaining up SAX filters.

Some criticism must be given; there are virtually no unit tests and there seems to be some code repetition and lots of "boilerplate" code (which is quite difficult to neutralize in Java) and somehow it just hurts me that the renderContent() method is 374 lines long. It could use some refactoring.

Portletbridge seems to have stalled active development some years ago, but there seems to be some recent code cleanup in progress, according to CVS logs.. It probably is doing what it's supposed to, feature-wise it seems to be close on par with html2jsr286, and it runs on JSR 168 portals. Page processing is done by XSLT, a feature that I really like.

html2jsr286 and Rails-portlet are strong in that they offer a toolchain for a Rails developer to get quickly into deployment with as much automation as possible. The other two projects look like they are more tuned to serve the portal administrator instead of the development team. The Rails-portlet project tools can also work without Rails, but then the deployment gets more encumbered, requiring more dirty work with Java and XML. Rails-portlet is targeting for web app developers who want to keep a healthy distance to the J2EE stack and "get things done" quickly.

html2jsr286 passes the Liferay UID in request headers, with a unique authentication mechanism. It can also handle Liferay GID – feature that is missing from the other two portlets. It is, however, restricted for JSR 286 portals, as it makes use of portlet filters, which were introduced only in JSR 286. The rationale behind this is to improve code coherence (there are no technical reasons that would demand using filters). JSR 286 container will, however, be needed for the XMLPortletRequest feature. The portlet is being used on at least two production sites, and in practise is doing fairly well. Page processing is done by HTML Parser, which does a decent job but has shortcomings that I would like to address with XSLT, and this is what Portletbridges is doing.

Nonetheless, I am considering a rational move to leverage WebProxyPortlet in Rails-portlet, and it seems that the first step is to get it running on Liferay and uPortal, hosting the portlet test bench.

Next big step would be to work on Ajax support, by carrying out ideas from these posts to implement XMLPortletRequest.

April 4, 2010

Flash problems after upgrading from Hardy to Lucid

I upgraded Ubuntu 8.04 (Hardy Heron) to 10.04 (Lucid Lynx) a few days ago. While it is still in "RC" stage, the graphical update-manager already offered me the upgrade. Which I did. I had to resolve to an ugly fix after the upgrade to get the proprietary Adobe Flash working. No matter what apt-get command I entered, there was this error message:
Package is in a very bad inconsistent state - you should reinstall it before attempting a removal.
There were a few others too who had similar problems on Ubuntu forums and this solution finally worked for me: remove all occurances of "flashplugin-installer" and "flashplugin-nonfree" from /var/lib/dpkg/status.