Recipe of request-response over AMQP with Python and Java

This is a recipe to setup synchronized messaging over AMQP. The recipe source zip file contains an implementation of a Python backend broker. Along comes example Python and Java clients that send messages over AMQP to the running broker(s). The broker load can be balanced over to several processes. RabbitMQ can automagically balance the incoming requests to multiple consumers. This setup has been deployed successfully, achieving decent performance (we have benchmarked only the private implementation, our throughput is approx. 30 msgs/sec. with three backend processes).

Good presentations on concurrent messaging and AMQP are available from:

• http://www.rabbitmq.com/resources/erlang-exchange-talk-final/ex.html
• http://www.igvita.com/2009/04/06/henry-ford-event-driven-architecture/
• http://kallistec.com/2009/05/15/basic-topic-publishing-with-amqp/
• http://hopper.squarespace.com/blog/2008/10/6/multithreading-amqp-clients.html
• http://www.slideshare.net/old_sound/interoperability-with-rabbitmq

AMQP can be used as a message bus between various system components in different languages maintained by different teams.

Here are the bare bones of this recipe:

• create a unique identifier (qid) for the request
• create a new consumer and bind it to a temporary response queue
• publish the message into the durable request queue
• wait for the response on the response queue
• close response channel

Topic exchange is the only exchange type that honors routing keys, which are essential for pairing request and response together, using the unique identifier in routing key.

Unzip the recipe (or clone the recipe repository from GitHub). Install RabbitMQ (default settings are ok), and install Python modules 'amqplib' and 'carrot' (plus 'jsonrpclib' for the tests).

Launch the RabbitMQ server.

sudo rabbitmq-server

Start the example backend broker:

cd py-src
python example_broker.py

This will launch a daemon that listens for messages on a specific AMQP channel. A "request" is a thin JSON wrapper. If the request includes a request id (qid), the daemon will return the current time over the AMQP. If the request is not identified, the time will be printed instead.

Run the example requests in Python and Java, while the example_broker is running:

cd py-src
python example_request.py

Output:

Response from AMQP: 
{u'msg': u'Sat Jan 29 20:43:36 2011'}

Run the same requests from Java:

cd java-src
java -cp .:build:lib/json.jar:lib/rabbitmq-client-1.8.1.jar:lib/commons-logging.jar:lib/commons-io.jar \
  example.ExampleRequest

Output:

29.1.2011 21:11:01 recipe.amqpbus.AMQPRequestResponseImpl newConnection
INFO: ExampleBroker connected to RabbitMQ @ 127.0.0.1:5672/
29.1.2011 21:11:01 recipe.amqpbus.AMQPPublisher send
INFO: publishing query to ExampleBroker_req with binding key: example.request
{"q":{"q":"hello AMQP backend, I am Java"}}
-----------------------------------------
29.1.2011 21:11:01 recipe.amqpbus.AMQPPublisher send
INFO: publishing query to ExampleBroker_req with binding key: example.request.q538
{"q":{"q":"time can you get me please?"},"qid":"q538"}
29.1.2011 21:11:01 recipe.amqpbus.AMQPConsumer receive
INFO: amq.gen-dyK/PZQn/brNl9jTR/tlAg== received message: 
{"msg": "Sat Jan 29 21:11:01 2011"}
Response from AMQP: {"msg":"Sat Jan 29 21:11:01 2011"}

The example_broker output should print the events when it receives the requests: Output:

  consumer received message: 
  {'exchange': u'TestExchange', 
    'consumer_tag': u'__main__.ExampleBroker-e7175ec0-cd1f-4d2e-973b-1eeb42d4071d', 
    'routing_key': u'example.request.*', 
    'redelivered': False, 
    'delivery_tag': 2, 
    'channel': <amqplib.client_0_8.channel.Channel object at 0x1007bbad0>}
  -------------------
  received request 355:
  what time is it?
  -------------------
  response to TestExchange with routing_key: example.response.355, message: 
  {"msg": "Sat Jan 29 20:47:59 2011"}
  using channel_id: 3
  Channel open
  Closed channel #3

You should be able to pick up the ingredients and integrate them to your codebase as you see fit.

Comparison of Web Proxy Portlet, Portletbridge and html2jsr286

There are at least three different JSR 168 / 286 portlet projects that have a similar target: to allow custom web apps and arbitrary web content be accessible in portlets.

• Web Proxy Portlet (source)
• portletbridge (source)
• html2jsr286 in Rails-portlet

I am the author of "Rails-portlet" and it's alive - I just updated the setup guide. It all started when I needed to develop a webapp for Liferay, and I wanted to do it with Rails. html2jsr286 is not an indication of the NIH syndrome, I just now learned of their existance. I inspected their source code, and this is a quick run-down of their features and pondering of future direction.

These are the common features in all three projects:

• Getting content from a downstream site
• Proxying of remote resources (e.g. images, Flash etc.)
• Regular expression defining which URL's are in the portlet and which should be regular links
• Rewriting CSS urls
• Moving Javascript and CSS links out of the head
• Session cookie support

All three have the same shortcoming concerning Ajax: the urls are not being rewritten so that XHR from the browser is not catched by the portlet.

WebProxyPortlet is being developed mainly for uPortal, and

..it provides mechanisms for connecting to and rendering HTML with options for clipping, maintaining session information, handling cookies. Proxied content is rendered within the portlet window. Web Proxy Portlet is often used to incorporate web content or applications that are built and run in non-Java environments allowing a site flexibility for integrating with many different technologies.

Sources of portletbridge

$ mkdir portletbridge
$ cd portletbridge/
$ cvs -z3 -d:pserver:anonymous@portletbridge.cvs.sourceforge.net:/cvsroot/portletbridge co -P .

Sources of WebProxyPortlet

$ svn co https://www.ja-sig.org/svn/portlets/WebproxyPortlet/trunk/ WebproxyPortlet

Sources of html2jsr286

$ git clone git@github.com:lamikae/html2jsr286.git

My impression after a while looking through the sources is that WebProxyPortlet is the most advanced, and has a number of nice features, especially in ways of user authentication and portlet configuration through the user interface.

Considering code cohesion and the number of Java classes: WebProxyPortlet has 58, Portletbridge 51 and html2jsr286 only 10 Java source files. Here are links to their main view methods:

void renderContent(final RenderRequest request, final RenderResponse response) in WebProxyPortlet.java
void doView(final RenderRequest request, final RenderResponse response) in BridgeViewPortlet.java
void render(RenderRequest request, RenderResponse response) in Rails286Portlet.java

WebProxyPortlet works in JSR 168 containers and supports a variety of authentication techniques, handles caching, has pluggable Java classes for custom request filtering and page processing. On top of this is uses a whole lot of JavaBeans and JSPs that make the portlet configurable from the GUI in the portlet config mode. I am not sure how much runtime overhead this accouts to, and how useful it is to let users select the HTML parser the portlet is going to use. Page processing is implemented by chaining up SAX filters.

Some criticism must be given; there are virtually no unit tests and there seems to be some code repetition and lots of "boilerplate" code (which is quite difficult to neutralize in Java) and somehow it just hurts me that the renderContent() method is 374 lines long. It could use some refactoring.

Portletbridge seems to have stalled active development some years ago, but there seems to be some recent code cleanup in progress, according to CVS logs.. It probably is doing what it's supposed to, feature-wise it seems to be close on par with html2jsr286, and it runs on JSR 168 portals. Page processing is done by XSLT, a feature that I really like.

html2jsr286 and Rails-portlet are strong in that they offer a toolchain for a Rails developer to get quickly into deployment with as much automation as possible. The other two projects look like they are more tuned to serve the portal administrator instead of the development team. The Rails-portlet project tools can also work without Rails, but then the deployment gets more encumbered, requiring more dirty work with Java and XML. Rails-portlet is targeting for web app developers who want to keep a healthy distance to the J2EE stack and "get things done" quickly.

html2jsr286 passes the Liferay UID in request headers, with a unique authentication mechanism. It can also handle Liferay GID – feature that is missing from the other two portlets. It is, however, restricted for JSR 286 portals, as it makes use of portlet filters, which were introduced only in JSR 286. The rationale behind this is to improve code coherence (there are no technical reasons that would demand using filters). JSR 286 container will, however, be needed for the XMLPortletRequest feature. The portlet is being used on at least two production sites, and in practise is doing fairly well. Page processing is done by HTML Parser, which does a decent job but has shortcomings that I would like to address with XSLT, and this is what Portletbridges is doing.

Nonetheless, I am considering a rational move to leverage WebProxyPortlet in Rails-portlet, and it seems that the first step is to get it running on Liferay and uPortal, hosting the portlet test bench.

Next big step would be to work on Ajax support, by carrying out ideas from these posts to implement XMLPortletRequest.

• http://www.subbu.org/blog/2007/01/xmlportletrequest-part-1-background
• http://www.subbu.org/blog/2007/01/xmlportletrequest-part-2-wrapping-xmlhttprequest
• http://www.subbu.org/blog/2007/08/update-on-jsr-286-and-ajax
• http://blogs.sun.com/deepakg/entry/xmlportletrequest_in_portlet_container_2

EJB to Ruby proxying and a little bit on programming languages

This essay goes quite technical at times but the presented idea is pretty simple.

Java and the J2EE stack seem to be used in large-scale enterprise web solutions. Ruby is an odd hippie programming language that is not so often met (yet!) in this enterprise world. Ruby on Rails does not quite have the native equivalent in Java web development toolpack, and there never will be such. One key fundamental difference in Java and Ruby, is their measure of abstractness. Higher abstraction in some cases help to describe and solve the problem in just a few code blocks. Ruby lets you express ideas outside the micromanagement level - right from the beginning of your introduction to the language.

JRuby helps to break a major technical barrier, making possible to leverage the massive amount of Java libraries that now can be used through the Ruby interpreter. This is a description of a real world scenario where Ruby expression describes and solves the problem quite beautifully.

To understand what is happening next you need to know that an EJB is an Enterprise Java Bean. You can imagine that the Bean is an exporter in Brazil. The Java transaction layer RemoteMethodInvocation is built on top TCP/IP infrastructure. Regular citizen Java clients need to contact their national supplier.

It is quite a bureaucratic process. It takes some knowledge of Java to understand how each object should behave. EJB3 makes this a bit easier but to efficiently use the developer tools will take a long time to learn. This is an example of the Java enterprise business model where you develop on elaborate IDEs and large companies have lots of people doing object-oriented chunks of code. After having that much Java your dreams will become XML configurable for a while..

Ruby has a similar technique, DRb - Distributed Ruby. It is built into the Ruby core language. It provides a way to share data objects over the wire. As we will see, it takes only a few lines, perfectly writable even from the vi editor.

Now, let's inspect the supply chain.

bean_context =
      {
        'java.naming.factory.initial'      => 'com.evermind.server.rmi.RMIInitialContextFactory',
        'java.naming.security.principal'   => security_principal,
        'java.naming.security.credentials' => security_credentials,
        'java.naming.provider.url'         => provider_url
      }

This is the Brazil Bean exporter who has the brew I like. His contact is found by jndi_name from the @context. Home represents my national office who will be creating me this @stub guy who actually does the delivery of my order.

@context = InitialContext.new(bean_context)
  @home = @context.lookup(jndi_name)
  @stub = @home.create

EJB home is an object that implements the create() method for requesting the remote EJB and returning the stub object that acts as the EJB interface. It acts a bit like a database table model.

Let's say we want to know more of the company..

@stub.companyID
  => "430146"

  @stub.companyName
  => "Brazil Bean Provider"

Mmmh. So let's make an order..

@order = @stub.createOrder(uid,"Coffea arabica, 1 kg sample")
  => #<#:0x62ee558f @java_object=BBPServices stateless session>  

Ah! It returned an object! One important aspect of the dynamic nature of Ruby (and thus JRuby) is that the callable methods need not be specified beforehand. The method call to the Ruby object is dispatched to the EJB, the return value as well and if possible, JRuby casts it to Ruby native.

@order.items
  => ["Coffea arabica, 1 kg sample"]

This could be usable in Rails, but the architecture should allow dynamic translation of method calls so that the solution would have the most leverage and also to stay clean out of implementation details. Ruby has the sort of inner beauty that is capable of this.

method_missing() is a powerful Ruby kernel method, which has its counterpart in Python and Lisp, possibly other dynamic languages are growing into that direction too. Unless you are not familiar with method_missing(), I will explain it so that when Ruby calls a method on the EjbObject's instance (there is this @stub), and the Ruby object does not find the method, it calls method_missing, and here the call is redirected to the @stub, alas the EJB interface and along the network over TCP/IP. In other words, Ruby does not care what kind of EJB it gets.

This is the core of EjbObject class:

class EjbObject
    def context_environment
      {
        'java.naming.factory.initial'      => 'com.evermind.server.rmi.RMIInitialContextFactory',
        'java.naming.security.principal'   => security_principal,
        'java.naming.security.credentials' => security_credentials,
        'java.naming.provider.url'         => provider_url
      }
    end

    def initialize
      @context = InitialContext.new(context_environment)
      @home = @context.lookup(jndi_name)
      @stub = @home.create
    end

    def method_missing(method, *args, &block)
      @stub.send(method, *args, &block)
    end
  end

EjbObject is used by subclassing it:

class MyEJB < EjbObject
    set_provider_url 'rmi://....'
  end

And so, this essentially is the JRuby engine in ejb2rb. The solution wouldn't be complete without the Rails counterpart, ActiveEJB. Conceptually DRb and RMI are very similar.

Ok - even if the code didn't speak to you, think it this way - now JRuby can talk to the remote Java beans, call their methods, and use the return values as other objects that have methods. Some object classes are mapped directly to corresponding Ruby classes, such as Strings, Arrays and Fixnums. The Java-cast beatnik flower child JRuby can talk enterprise jargon. The EJBDispatcher::Instance enables him to open a channel to one EJB for other Ruby VMs that are equally valuable as Ruby implementations but have not developed their Java-fu.

This is how the EJB stub object is shared for DRb clients (the thread is started later by Hydra):

module EJBDispatcher
    class Instance

      self.config = {
        'hostname' => 'localhost',
        'port'     => '9876',
        'class'    => 'MyEJB'
      }

      def initialize
        @uri = "druby://%s:%s" % [config['hostname'],config['port']]
        DRb.start_service(@uri, Kernel.const_get(config['class']).new)
        return DRb.thread
      end
    end
  end

What happens on the Ruby end?

module ActiveEJB
    class Entity < DRb::DRbObject
      def initialize
        uri = 'druby://%s:%i' % [drb_server,drb_port]
        super(nil,uri)
      end
    end
  end

ActiveEJB::Entity is used by subclassing it:

class MyEJB < ActiveEJB::Entity
    include Singleton

    set_drb_server 'localhost'
    set_drb_port   '9876'
  end

So, in the end, you can use this in native MRI Ruby Rails:

profile = MyEJB.getUserProfile(@username, "PDS")

  contacts = MyEJB2.getCountryContacts(profile.getAddress().getCountry())
  
  contacts.each do |contact|
    ...
  end

Or something .. usually more or less involving enterprise business logic.. The actual implementation can handle multiple concurrent EJB instances.

But, in the end, if we look at the power and sheer expressfulness of Ruby, I think this is a fine example of it working to solve a real world problem. The source code is available to help rubyists with the leverage this technology may give them trying to survive under the serviance of J2EE.

Oh, yes, the order:

@order.delivery.weight
  => 1.108

  BrewerEJB.makeCoffee(
    @order.delivery,
    :espresso,
    :milk => true,
    :sugar => false)

Mathematical discourse on the web

Consider explaining mathematics on the web for someone. The tools to use are quite nifty. Algebra needs distinguishable symbols, geometry some other control methods. Even simple formulas are not so simple:

• http://terrytao.wordpress.com/2009/10/29/displaying-mathematics-on-the-web/
• http://terrytao.wordpress.com/2009/11/04/displaying-maths-online-ii/

I have a friend who recommended I should try to mix MathFlow with webMathematica and offered me a developer platform. This post is not so much as on the display of formulas as it is about my personal experience of mathematical expression on the web. Mathematica feels like a nice language. I had help for learning it from an experienced notebook / jsp writer, another friend. With his teaching I could quickly learn the essentials constructs of Mathematica's functional style to do simple experimentation and mix it in jsp. WebMathematica3 is a Java servlet package with the necessary backend kernel connectivity and a taglib for the developer. Worked out-of-the-box on OSX – Linux server requires 1 extra step for Xvnc. The taglib is quite usable, with three basic types of components:

• strings or graphics from the kernel (as small blips in building the UI)
• Java applet for Graphics3D
• Flash applet for Manipulate

Manipulate is not quite as swift as it is in native Mathematica. The image does not follow the slider before the click is released. The network and server would be stressed for serving many concurrent users..

MathFlow is an input mechanism to essentially decrease the learning curve to input mathematical notation into the web. It is a Java applet that seems to work on all major OSes given that the moon is in correct orientation and system Java has not crucially changed. Icedtea6 Java plugin on Ubuntu worked too. MathFlow is proprietary software, this was an evaluation license. To be able to test it on a private server so that clients other than localhost can see it, was possible by recompiling the jar with a custom evaluation license. I could not find any open source alternatives, and this software seems capable of doing the job right now. Ideally, modern-day alternative could be written in JavaScript.. maybe compatible with MathJax..

The output formula is in MathML, and it can be captured to webMathematica input, in JavaScript, using for example the MooTools toolkit to do an XHR and update the page:

var mml = document.SampleSimpleEditorApplet.getMathML();
   params = 'fun='+URLescape(mml);
   var req = new Request.HTML({
     method: 'post',
     url: '_action.jsp?'+params,
     encoding: 'iso8859-15',
     onRequest: function() {},
     update: $('div.output'),
     onComplete: function(response) {}
   }).send();

This would send the contents of the editor applet to _action.jsp in request parameter fun, and the XHR would update the page. This _action.jsp in webMathematica could..

<%@ page contentType="text/xml"%>
<%@ taglib uri="http://www.wolfram.com/msp" prefix="msp" %>

<msp:evaluate>
  MSPFormat[
    MSPToExpression[$$fun],
    TraditionalForm
    ]
</msp:evaluate>

.. generate an image of the formula in traditional mathematical notation. Notice the $$fun variable, which is parsed by the webMathematica backend. In case where the formula has variables, it is possible to parse it in the "delayed" form:

SetDelayed[
    f[x_],
    Evaluate[
      MSPToExpression[
        $$fun,
        MathMLForm
      ]
    ]
  ];

Or plot it, notice the use of MSPBlock ..

MSPBlock[
    {$fun},
    MSPShow[
      Plot[
  
        $fun,
        {x, -2*Pi, 2*Pi},
        ImageSize->{600,500}

      ]]]

So you can build all sorts of elaborate user interfaces on top of these building blocks. This is just an example of plotting the user input formula. He shouldn't need to learn LaTeX or any other new notation to input the mathematics into the computer.

The mathematica magician can work with the Wolfram Workbench to write Mathematica code, and still work in a familiar notebook environment.

The web developer has a similar view to produce the 3D applet, where the MSPLive3D module creates the HTML code for the Java applet.

Needs["Talo3D`"];

...

  MSPLive3D[
    Graphics3D[
     {
      RGBColor[0, 0, 1],
      Thickness[0.005],
      Map[Line, house]
     },
     Boxed -> False
    ]
  ]

This is LiveGraphics3D generated by webMathematica3 on Safari. The house is rotatible. =) The original .nb was cc'd to me by Jyrki Kajala a few years ago. Quite recently he taught me about Mathematica packages, and .m files, and how the notebook can load the package. The package can now be subjected to functional testing and also included to webMathematica JSP. The tools have a certain good feel of dynamicity. Some issues, here and there, but it would be very interesting to build various use scenarios ..