Recipe of request-response over AMQP with Python and Java

This is a recipe to setup synchronized messaging over AMQP. The recipe source zip file contains an implementation of a Python backend broker. Along comes example Python and Java clients that send messages over AMQP to the running broker(s). The broker load can be balanced over to several processes. RabbitMQ can automagically balance the incoming requests to multiple consumers. This setup has been deployed successfully, achieving decent performance (we have benchmarked only the private implementation, our throughput is approx. 30 msgs/sec. with three backend processes).

Good presentations on concurrent messaging and AMQP are available from:

• http://www.rabbitmq.com/resources/erlang-exchange-talk-final/ex.html
• http://www.igvita.com/2009/04/06/henry-ford-event-driven-architecture/
• http://kallistec.com/2009/05/15/basic-topic-publishing-with-amqp/
• http://hopper.squarespace.com/blog/2008/10/6/multithreading-amqp-clients.html
• http://www.slideshare.net/old_sound/interoperability-with-rabbitmq

AMQP can be used as a message bus between various system components in different languages maintained by different teams.

Here are the bare bones of this recipe:

• create a unique identifier (qid) for the request
• create a new consumer and bind it to a temporary response queue
• publish the message into the durable request queue
• wait for the response on the response queue
• close response channel

Topic exchange is the only exchange type that honors routing keys, which are essential for pairing request and response together, using the unique identifier in routing key.

Unzip the recipe (or clone the recipe repository from GitHub). Install RabbitMQ (default settings are ok), and install Python modules 'amqplib' and 'carrot' (plus 'jsonrpclib' for the tests).

Launch the RabbitMQ server.

sudo rabbitmq-server

Start the example backend broker:

cd py-src
python example_broker.py

This will launch a daemon that listens for messages on a specific AMQP channel. A "request" is a thin JSON wrapper. If the request includes a request id (qid), the daemon will return the current time over the AMQP. If the request is not identified, the time will be printed instead.

Run the example requests in Python and Java, while the example_broker is running:

cd py-src
python example_request.py

Output:

Response from AMQP: 
{u'msg': u'Sat Jan 29 20:43:36 2011'}

Run the same requests from Java:

cd java-src
java -cp .:build:lib/json.jar:lib/rabbitmq-client-1.8.1.jar:lib/commons-logging.jar:lib/commons-io.jar \
  example.ExampleRequest

Output:

29.1.2011 21:11:01 recipe.amqpbus.AMQPRequestResponseImpl newConnection
INFO: ExampleBroker connected to RabbitMQ @ 127.0.0.1:5672/
29.1.2011 21:11:01 recipe.amqpbus.AMQPPublisher send
INFO: publishing query to ExampleBroker_req with binding key: example.request
{"q":{"q":"hello AMQP backend, I am Java"}}
-----------------------------------------
29.1.2011 21:11:01 recipe.amqpbus.AMQPPublisher send
INFO: publishing query to ExampleBroker_req with binding key: example.request.q538
{"q":{"q":"time can you get me please?"},"qid":"q538"}
29.1.2011 21:11:01 recipe.amqpbus.AMQPConsumer receive
INFO: amq.gen-dyK/PZQn/brNl9jTR/tlAg== received message: 
{"msg": "Sat Jan 29 21:11:01 2011"}
Response from AMQP: {"msg":"Sat Jan 29 21:11:01 2011"}

The example_broker output should print the events when it receives the requests: Output:

  consumer received message: 
  {'exchange': u'TestExchange', 
    'consumer_tag': u'__main__.ExampleBroker-e7175ec0-cd1f-4d2e-973b-1eeb42d4071d', 
    'routing_key': u'example.request.*', 
    'redelivered': False, 
    'delivery_tag': 2, 
    'channel': <amqplib.client_0_8.channel.Channel object at 0x1007bbad0>}
  -------------------
  received request 355:
  what time is it?
  -------------------
  response to TestExchange with routing_key: example.response.355, message: 
  {"msg": "Sat Jan 29 20:47:59 2011"}
  using channel_id: 3
  Channel open
  Closed channel #3

You should be able to pick up the ingredients and integrate them to your codebase as you see fit.

Hybrid Qt applications with PySide and Django

This article describes some thoughts and experiments on developing applications targeted for mobile systems, especially MeeGo. I am trying to imagine a secure and agile way to write applications that are robust yet elegant.

About the author – I have worked with Ruby on Rails, Django and Qt. I think that web run-time technologies (HTML5, JavaScript and CSS3 – called WRT in this article) can provide a flexible way to build user interfaces. Many developers are familiar with these technologies, and maximizing WRT percentage in the codebase has potential to better portability for devices from different manufacturers. Personally to me MeeGo is the most interesting mobile operating system target. I am trying not to completely forget about business models, but then again, I have a software libre bias.

I have experimented of "hybridizing" Qt with Django. PySide provides Qt bindings for Python in LGPL licensing. Django is a CMS-oriented web framework written also in Python. Using Python instead of C++ has a drawback in runtime efficiency, but in some cases developer productivity and codebase maintainability is more important. Python has become one of my favourite languages.

This approach interests me because an application from the same codebase can be deployed both on desktop and on (a Nokia) mobile device. Alternatively a desktop/server Django application could serve plain WRT components also for non-Qt clients (but I can't comment how Apple would respond to this sort of application for the App Store). Android Market - well - they probably would not have a problem with it. That's all I have to say about marketing and business politics.

Nokia N900 is a fabulous platform for prototyping, on which I wrote an experimental application for controlling a TV set within the local area network (LAN). Figure 1 displays an event cycle in which the whole application is self-contained on the mobile device.

Fig 1. The software components on N900

Not depicted in figure 1 for clarity, the Python process contains an implementation of a finite-state machine. User actions are sent as events to the backend process handling logical states, while the WRT defines slots that update the UI.

This approach is reminescent of cloud computing, but with an important difference in that the data that moves between the server an the mobile device would not be stored on servers the user does not have control over. The data would never pass to the ISP either. Consider a Windows PC with a media repository was running PySide/Django in the LAN, and an iPhone owner could control the media stream onto a MeeGo TV set? Switch Windows for Linux and iPhone for a MeeGo device and it still would work without a heavy rewrite of the codebase. The PC would serve an ad-hoc "cloud" that could be operated from the hand-held device, acting as a remote controller. That is the core of this vision.

It makes sense to browse local data (from the PC) and "stream" it onto the mobile device. You wouldn't carry the precious data around in your phone, but it still would be reachable from home over a secure network connection. This is a data privacy issue. Photos are a good example. A major concern in any networked application is security and I've foolishly ignored many gaping secury holes in this prototype. As for my defense I can only say I am experimenting the feasibility and hardware performance so far.

 

Fig 2. Cloud computing within the LAN with MeeGo

The prototype application I've written (called MxManager) is a remote controller for Maximum T-8000 personal video recorder (PVR) that runs on embedded Linux. Its prorietary firmware provides a simple HTTP interface that mimics the physical remote controller. This prototype uses that API to set recordings from the EPG and play recordings on the TV. Here is a screenshot of the application (on MacBook desktop) and a blurry video of it running on N900.

Fig 3. MxManager UI

Qt hybrid on N900 on Vimeo. Sorry about the quality. It was shot up close, I guess I would have needed to strap on a macro lens.

The screen and remote control UI are individual QWebView elements and their content is composed by Django views. Events from the UI are triggered in JavaScript and events back from PySide are injected as JavaScript function calls into the page. PySide is running a finite-state machine (using QStateMachine) that holds the internal state of the application. Changes in UI are triggered by state change events. I wrote about this in more detail in a past blog post. The source code is released under the BSD, so you are welcome to have a look and give an opinion.

Initial experiments with QML appeared subjectively much faster, nearing smooth animation along the button press, but in the past I had a problem in sending signals into QML. This seems to have been fixed in PySide 1.0.0. This is a solid direction to experiment - perhaps Django could be used to compose QML for the QDeclarativeView..

Another video of the application in action with the TV set!

Qthybrid and Maximum T-8000 on Vimeo.

While building the web stack on well-proven frameworks, security issues still need to be considered carefully. Web sockets would be better designed for such eventing purpose than Ajax, with a better security model. I do repeat, this is not production-ready code.

If the mobile phone supported QtMobility, it could be possible to use mobile accelerometer and orientation to control playback. Turn the phone upside down to pause, for example. The next video (by perlinet, not me) portrays using QtMobility API in PySide with QML.

PySide is great technology and it's inclusion to MeeGo opens up many ways for developers to bring their applications to the platform. There are definite points to focus on this basis:

• security implications
• web sockets
• QML
• QtMobility experimentation

Especially intriguing for me would be to hook up digiKam database into a desktop-based Django app for browsing photos on a mobile device / tablet computer.

What's unique here is the merging of the Qt and web frameworks to operate within the same OS process, which makes it easier to separate the presentation and business logic. Personally I feel this technology seems feasible, and perhaps I will continue to investigate it further in 2011. There are web services that operate on a similar business model, and maybe open source software is following the trend.

Recipe: handling database communication breakdown in Rails and Django

This recipe is for users who manually handle persistent (shared) database connection, with Oracle, although it's probably easy to adapt for other database backends with small adjustments. I needed to implement this both in Ruby (Rails' ActiveRecord) and Python (django.db) daemons, both working in the same setting. I thought the solution was elegant enough to be cleaned up into a recipe.

Automated database connection pooling for Python (nor Ruby) is not supported with Oracle 10g. Only 11g introduced Database Resident Connection Pooling (DRCP). It probably makes this recipe redundant for Python users with 11g. Oracle has official support for using Python with Oracle Database 11g, using the cx_Oracle driver. I'm not sure but I think Oracle doesn't have any form of official support for Ruby, but the de-facto ruby-oci8 driver works well with ActiveRecord.

Without automated connection pooling you have two decent options. Either open and close a new connection for each request, or save open connections into shared memory. The latter has obvious performance benefits if the expected request rate is high, but exposes a number of other problems this recipe aims to address. A connection can abruptly terminate because of e.g. a network glitch or maintenance downtime, and the sysadmins deploying your software will be happier having to deal with less services to reboot.

The handler should catch exceptions only related to network or database resource failures, whereas in other cases (SQL typo?) the original exception should not be reacted upon. Figuring out the proper ORA error codes to use was an exciting task, I hope I got them right ;). Those guys at Oracle sure love to make things easy. Since I use a regular expression to match the error message string, a multitude of errors can be matched with a single rule. Fortunately, all errors dealing with Transparent Network Substrate are marked by prefix "TNS". In my tests most errors were indeed caught with the single "ORA-.....: TNS" regexp.

Python and Ruby have much in common, both could let me write this very expressively with about the same LOC. There is one small detail I'd like to bring up. These two expressions both compare the codes listed with the exception (error message in variable "message"), and return true if the message matches to one of the given ORA error codes.

Python

# Example message="ORA-03113: end-of-file on communication channel"
any(filter(lambda oracode: re.search('ORA-'+oracode,message), re.split(' *\n *| +(?=\d)',"""
    .....: TNS
    01000 01001 01014 01033 01034 01037 01089 01090
    011.. 015.. 016..
    031..
    28547
    30678
    """)[1:-1]))

Ruby

# Example message="ORA-01034: ORACLE not available"
%w{ .....:\ TNS
    01000 01001 01014 01033 01034 01037 01089 01090
    011.. 015.. 016..
    031..
    28547
    30678
}.select {|oracode| message[/ORA-#{oracode}/]}.any?

Especially take notice of Ruby's %w{multiline string} versus Python's re.split(' *\n *| +(?=\d)',"""multiline string""")[1:-1]. This just for the sake of the visual appearance of input data on the screen. Ruby makes it easier to write aesthetic code. Perhaps Python has an idiom for this, though, it just didn't pass my mind.

example_oci_exception_handler.py

# -*- coding: utf-8 -*-
"""
You may use this file under the terms of the BSD license as follows:

Redistribution and use in source and binary forms, with or without modification, are
permitted provided that the following conditions are met:

   1. Redistributions of source code must retain the above copyright notice, this list of
      conditions and the following disclaimer.

   2. Redistributions in binary form must reproduce the above copyright notice, this list
      of conditions and the following disclaimer in the documentation and/or other materials
      provided with the distribution.

THIS SOFTWARE IS PROVIDED "AS IS" AND ANY EXPRESS OR IMPLIED
WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL  OR
CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

The views and conclusions contained in the software and documentation are those of the
authors and should not be interpreted as representing official policies, either expressed
or implied, of copyright holder.
"""
import re
import sys
from time import sleep
from myapp.settings import settings # your django settings module
from django.core.management import setup_environ
setup_environ(settings)
import cx_Oracle
from django.db import connection
from django.db.utils import DatabaseError
from logging import getLogger
log = getLogger(__name__)

class ExampleOCIExceptionHandler():
    """Recipe for handling lost Oracle database connection.
    
    Depends on Django ORM, and that proper settings.py exist and is initialized properly.
    
    Selects the network and connection errors, waits until the server is reachable again,
    and calls the method again during upon which the exception happened.
    
    This is very useful if the connection, for some reason, is shared between requests.
    Oracle 10g does not support connection pooling with cx_Oracle.
    """

    @staticmethod
    def dispatch(message_data):
        """Processes request.

        In case of database connection failure, the waitForOCIConnection() loop is called
        and connection should be re-established. In case of other Oracle errors,
        the error is raised again.
        """
        try:
            # do some database operations...

        except (cx_Oracle.Error, DatabaseError):
            """
            Catch lost database connection.

            Handles all TNS errors:
              ORA-xxxxx: TNS errors

            And all errors from these ranges:
              ORA-011xx: Database file errors
              ORA-015xx: Execution errors
              ORA-016xx: Execution errors
              ORA-031xx: communication errors

            Along with these specific errors:
              ORA-01000: maximum open cursors exceeded
              ORA-01001: invalid cursor
              ORA-01014: ORACLE shutdown in progress
              ORA-01033: ORACLE initialization or shutdown in progress
              ORA-01034: ORACLE not available
              ORA-01037: cannot allocate sort work area cursor; too many cursors
              ORA-01089: immediate shutdown in progress - no operations are permitted
              ORA-01090: shutdown in progress - connection is not permitted
              ORA-28547: connection to server failed, probable Oracle Net admin error
              ORA-30678: too many open connections

            """
            msg = str(sys.exc_info()[1]).rstrip()

            # is the error about broken connection?
            if any(filter(lambda oracode: re.search('ORA-'+oracode,msg), re.split(' *\n *| +(?=\d)',"""
                .....: TNS
                01000 01001 01014 01033 01034 01037 01089 01090
                011.. 015.. 016..
                031..
                28547
                30678
                """)[1:-1]) # ignore outermost items as they are empty strings
            ):
                log.error(msg)
                ExampleOCIExceptionHandler.waitForOCIConnection()
                # enter recursion and call this method again..
                sleep(2)
                return ExampleOCIExceptionHandler.dispatch(message_data)

            else:
                log.debug(msg)
                log.debug("Error seems to be not about lost connection, raising again ..")
                raise

    @staticmethod
    def waitForOCIConnection(time_to_wait=60):
        """Creates a new database connection, loops until one is established."""
        # nullify the connection first, since it can't discover that the socket is gone
        connection.connection = None
        while not connection._valid_connection():
            try:
                log.info("Attempt to establish OCI connection to %s ..." % [
                    settings.DATABASES['default']['NAME']])
                # in django parlance, calling _cursor() opens the database connection
                cursor = connection._cursor()
                sleep(2)
                return connection._valid_connection()

            except (cx_Oracle.Error, DatabaseError):
                exctype, message = sys.exc_info()[:2]
                log.error(str(message).rstrip())
                log.info("Retrying OCI connection in %i seconds" % time_to_wait)
                sleep(time_to_wait)
                return ExampleOCIExceptionHandler.waitForOCIConnection(time_to_wait)

example_oci_exception_handler.rb

# encoding: utf-8
=begin
You may use this file under the terms of the BSD license as follows:

Redistribution and use in source and binary forms, with or without modification, are
permitted provided that the following conditions are met:

   1. Redistributions of source code must retain the above copyright notice, this list of
      conditions and the following disclaimer.

   2. Redistributions in binary form must reproduce the above copyright notice, this list
      of conditions and the following disclaimer in the documentation and/or other materials
      provided with the distribution.

THIS SOFTWARE IS PROVIDED "AS IS" AND ANY EXPRESS OR IMPLIED
WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL  OR
CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

The views and conclusions contained in the software and documentation are those of the
authors and should not be interpreted as representing official policies, either expressed
or implied, of copyright holder.
=end
require 'rubygems'
require 'active_record'

# Recipe for handling lost Oracle database connection with ActiveRecord.
#
# Depends on RAILS_ROOT and RAILS_ENV variables to be set and RAILS_ROOT/config/database.yml file to exist.
#
# Selects the network and connection errors, waits until the server is reachable again,
# and calls the method again during upon which the exception happened.
#
# This is very useful if the connection, for some reason, is shared between requests.
class ExampleOCIExceptionHandler

  # Processes request.
  #
  # In case of database connection failure, the waitForOCIConnection() loop is called
  # and connection should be re-established. In case of other Oracle errors,
  # the error is raised again.
  def self.dispatch(message_data)
    begin
      # do some database operations...

    # Catch lost database connection.
    #
    # Handles all TNS errors:
    #   ORA-xxxxx: TNS errors
    #
    # And all errors from these ranges:
    #   ORA-011xx: Database file errors
    #   ORA-015xx: Execution errors
    #   ORA-016xx: Execution errors
    #   ORA-031xx: communication errors
    #
    # Along with these specific errors:
    #   ORA-01000: maximum open cursors exceeded
    #   ORA-01001: invalid cursor
    #   ORA-01014: ORACLE shutdown in progress
    #   ORA-01033: ORACLE initialization or shutdown in progress
    #   ORA-01034: ORACLE not available
    #   ORA-01037: cannot allocate sort work area cursor; too many cursors
    #   ORA-01089: immediate shutdown in progress - no operations are permitted
    #   ORA-01090: shutdown in progress - connection is not permitted
    #   ORA-28547: connection to server failed, probable Oracle Net admin error
    #   ORA-30678: too many open connections
    #
    rescue OCIError, ActiveRecord::StatementInvalid
      msg = $!.message
      
      # is the error about broken connection?
      if %w{
        .....: TNS
        01000 01001 01014 01033 01034 01037 01089 01090
        011.. 015.. 016..
        031..
        28547
        30678
      }.select{|oracode| msg[/ORA-#{oracode}/]}.any?
      
        logger.error msg
        waitForOCIConnection()
        # enter recursion and call this method again..
        return dispatch(message_data)

      else
        logger.debug msg
        logger.debug("Error seems to be not about lost connection, raising again ..")
        raise $!
      end
    end
  end


  # Attempts to establish database connection.
  # Loops until ActiveRecord is connected.
  #
  # Set time_to_wait in seconds.
  def self.waitForOCIConnection(time_to_wait=60)
    begin
      database_configuration = YAML.load_file(File.join(RAILS_ROOT,"config","database.yml"))
      ActiveRecord::Base.configurations = database_configuration
      logger.info(
        "Attempt to establish OCI connection to %s ..." % database_configuration[RAILS_ENV]['database'])
      ActiveRecord::Base.establish_connection(database_configuration[RAILS_ENV])
      ActiveRecord::Base.connection # essential to open connection
      return ActiveRecord::Base.connected?

    rescue OCIError, ActiveRecord::StatementInvalid
      logger.error($!.message)
      logger.info("Retrying OCI connection in %i seconds" % time_to_wait)
      sleep time_to_wait
      return waitForOCIConnection(time_to_wait)
    end
  end
end

Interacting with legacy Oracle database in Python

I needed to use an Oracle database in Python. It all went rather smoothly with the guides, hints and examples, but I was missing a walkthrough. This memo is about setting up Oracle with Python and how to use legacy database with Django. Better yet, use Django Models without any of the HTTP functionality.

Oracle more or less has official support for cx_Oracle, and it was fairly easy to set up. cx_Oracle needs Oracle headers to compile. From the Oracle download site get, according to your OS and arch, instantclient-basic, instantclient-sqlplus and instantclient-sdk. The Oracle download site requires registration.

Select a working path and extract the Instant Client packages.

$ cd /opt
$ unzip ~/Downloads/instantclient-basic-10.2.0.4.0-macosx-x64.zip
$ unzip ~/Downloads/instantclient-sdk-10.2.0.4.0-macosx-x64.zip
$ unzip ~/Downloads/instantclient-sqlplus-10.2.0.4.0-macosx-x64.zip

It is necessary to set the environment variables ORACLE_HOME, LD_LIBRARY_PATH and DYLD_LIBRARY_PATH. Put these into a convenient file and update the environment.

~/.oraenv

export ORACLE_HOME=/opt/instantclient_10_2
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ORACLE_HOME
export DYLD_LIBRARY_PATH=$ORACLE_HOME

On OS X, you need to create one symlink for the build to succeed:

$ cd /opt/instantclient_10_2/
$ ln -s libclntsh.dylib.10.1 libclntsh.dylib

Now you are ready to build and install the cx_Oracle driver.

$ cd /opt
$ tar xzf ~/Downloads/cx_Oracle-5.0.4.tar.gz
$ cd cx_Oracle-5.0.4
$ source ~/.oraenv # load environment
$ python setup.py build
$ python setup.py install

Assuming the database is already running somewhere, enter the connection parameters to this python script and test the connection:

test_connection.py

# -*- coding: utf-8 -*-
import cx_Oracle
connection = cx_Oracle.connect(
    "user",
    "pass",
    "127.0.0.1:1524/some.oraservice"
    )
cursor = connection.cursor()
cursor.execute("select  * from v$version")
for column_1 in cursor:
    print "Values:", column_1

You should see something like this:

$ python test_connection.py

Values: ('Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bi',)
Values: ('PL/SQL Release 10.2.0.4.0 - Production',)
Values: ('CORE\t10.2.0.4.0\tProduction',)
Values: ('TNS for Linux: Version 10.2.0.4.0 - Production',)
Values: ('NLSRTL Version 10.2.0.4.0 - Production',)

BUT if you get:

cx_Oracle.DatabaseError: ORA-12737: Instant Client Light: unsupported server character set WE8ISO8859P15

You took the BASICLITE package!!! Go back to download the proper package and start from the beginning.

Python can connect to the database with cx_Oracle, and Django models offer a clean way to make use of that connection. If you happen to have schema read privileges to the database, Django can generate models.py from it. Now, this is pretty nice feature when it comes to interacting with legacy database, as we shall see.

myapp/settings.py

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.oracle',
        'NAME': '127.0.0.1:1524/some.oraservice',
        'USER': 'user',
        'PASSWORD': 'pass',
    }
}

myapp/manage.py

from django.core.management import execute_manager
import settings
if __name__ == "__main__":
    execute_manager(settings)

The database SQL shell is useful to have and to test the connection. You can get greeted with the friendly SQL*Plus console by command dbshell:

$ source ~/.oraenv # remember environment!
$ python myapp/manage.py dbshell

SQL*Plus: Release 10.2.0.4.0 - Production on Fri Oct 15 12:56:05 2010

Copyright (c) 1982, 2007, Oracle. All Rights Reserved.


Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL>

Then you are ready to inspect the schema and generate the Python classes:

$ python myapp/manage.py inspectdb > myapp/models.py

Check the new file and assign "primary_key=True" to a column you choose, for each table you need.

Say your legacy db has a table called OraAbbOy1001 you need to manipulate. You may have a start script bin/start.py (as you don't run this with django-manage) in which you setup Django and do whatever you do to init your program. In myapp/logic.py you import the models and operate with the database.

bin/start.py

# -*- coding: utf-8 -*-
import myapp.settings
from django.core.management import setup_environ
setup_environ(myapp.settings)
# before do something else,
# the three lines above here is your first operation.

You can then operate on the models by finding or creating instances (selecting and inserting rows) through the ORM. It is also possible to mix in raw SQL, like in this example. I had trouble figuring out how to set a custom sequence name (legacy value) so I opted to select it manually.

myapp/logic.py

# -*- coding: utf-8 -*-
import time
from django.db import models, connection
from myapp.models import OraAbbOy1001

id = # comes somewhere
try:
    user = OraAbbOy1001.objects.get(id=id)

except OraAbbOy1001.DoesNotExist:

    # select nextval from ID sequence
    seq_name = 'q1001user'
    cursor = connection.cursor()
    cursor.execute("SELECT %s.NEXTVAL FROM DUAL" % seq_name)
    userid = cursor.fetchone()[0]

    # create new user
    user = OraAbbOy1001.objects.create(
        id=userid,
        name='Vic Video',
        create_date=time.strftime('%Y-%m-%d',time.localtime()),
    )

    log.info("OraAbbOy1001 user id %i (%s) created" % (
        user.id,
        user.name
        )
    )

SQLAlchemy looks very interesting. If you need to have more control over the queries, look up for it.

BTW, I'm not affiliated with O'Reilly, I just like the illustrations. They do have many good books.

QML signals on PySide

I studied QML with signal-slot communication across to Python a bit.
If you are looking for examples in source code, see

• gitorious.org/pyside-example-qml-signalslot/qml-signalslot
• gitorious.org/maximum-pvr/mxmanager

The latter one use PySide but not QML, and has a preview video elsewhere in this blog. Here are my earlier experiments.

This video shows a very small application, consisting of two files - webview.qml and webview-qml.py.
The first three signals originate from QML and Python opens up a QMessageBox to display them, along by printing to stdout. The third signal will emit a fourth signal from Python to QML, with an url as a parameter, and the url will open inside a WebView.

This app evolves the idea a bit further. The HTML5 canvas demos seen in the video are not by me, they are from the web. There is a state machine working behind the scenes.

Compiling PySide on OS X

Binary packages of PySide are available from developer.qt.nokia.com/wiki/PySideBinariesMacOSX. You only need to compile it yourself if you have explicit reason to.

However, if for any reason you would like to compile it yourself, read on. First of all, download Qt SDK. After it has been installed you can use this Makefile script to download, compile and package PySide from git master.

$ git clone git://gitorious.org/~lamikae/pyside/lamikae-pyside-packaging.git
$ cd lamikae-pyside-packaging/osx/
$ make package

Maybe after an hour or more you should have a .pkg file, which has PySide dynamically linked to Qt installed from official dmg.
Install it with the command:

$ sudo installer -pkg pyside-<version>.pkg -target "/"

See if it works (if there is no error, it does =) and try out some of the official PySide examples.

$ python -c "from PySide import QtCore"

You can inspect the package contents by

$ xar -xf pyside-<version>.pkg Bom && lsbom package.pkg/Bom

Gource animation of Rails-portlet development

Gource is a fabulous program that can visualize common version control repository history. This is the development of Rails-portlet for the last few years, in a few seconds.

I had to write a small Python script that parses logs from Git submodules. The script is at Gist and may be freely used.

Testing JavaScript & XHR with Python

I am working on a library for Django - which I'll describe later - and for building this project I need to test the behavior of XHR on the HTML page. That is, I have a page, inject some JavaScript (that may do a request on the server), and see how it changes the document. The Django application the library is designed for creates some parts of the original page and processes some parts of the server-side XHR.

Python is great for this. I've mentioned already in this blog how I like Qt4, and it proves to be very useful again. I could not get PySide running on OS X just yet, so I opted for PyQt4.

Four tests; four evenings working with them – each having an intricate problem to solve ;) Two tests work on the local document, the other two connect to the Portlet test bench, running on localhost. These tests trigger onclick submit actions from the XHR test page, simulating a user clicking a button.

This way it is possible to test the effect of JavaScript injection onto any web page in the wild. At least the WebKit response.. Firefox could be tested by using python-gtkmozembed. QWebKit also has methods for binding Python variables to the page (it is possible for JavaScript to trigger a Python callback function) and render an image snapshot of the page.

If you wish to run the tests, download the file, start up the Rails server with the test bench and run

$ python xhr_tests.py

Python Qt4 soup - quick way to cook up a cross-platform GUI

A friend of mine asked me to do a very quick and dirty throwaway application that parses data off a standard HTML file into an Excel spreadsheet. I chose to do it in Python and spent about an hour to decide which backend libraries to use, and opted for BeautifulSoup and xlwt to do the hard work. The core app – one function to parse input and another function to create the Excel workbook – was written in under two hours. It was only running from the command line – so I cooked him up a GUI:

This friend is a Windows user who dislikes Linux and Mac, and he needs a clean GUI that is dead simple to run and operate. After one sleepless night of furious hacking, he got one. In a Windows exe. Developed on OSX, also running under Linux. This is the power of Python combined with Qt4. The sources are online for those interested to see the details. To emphasize the throwoff nature of the app, all labels are hard-coded to Finnish.

I already had some experience on PyQt4. If you're reading this and decide to look it up, do not pass PySide – it is a newer Python Qt4 bindings library endorsed by Nokia et al. Either one you choose, you can't pass QtDesigner, which is a nice tool to design the UI and comes bundled in the Qt developer packages. In this case all I did was to add a QTableView element with some margin for the tool and status bars.

QtDesigner creates an .ui file (XML) that is translated to Python via pyuic4. With a text editor (XCode) I wrote a subclass of the UI model that extends it with a toolbar with two functions and a status bar for messages. Using file dialogs is easy. The most tricky bit is to make the data from the input file visible in the table view. PyQt4 handles that by a custom subclass of QAbstractTableModel with specific methods the Qt4 API will call to update the table contents.

Quite simple until you get to signals, slots and threading, and it doesn't hurt too much to write and maintain. Overall, about 120 lines of code for the UI, 50 lines for the core.

Now all is left is to test and package it for Windows for my friend to run. I've gotten over my worst fear and loathing of Windows, but I still have a very uneasy feeling every time I boot the OS, even in VirtualBox. Text editor and the console are my friends, and cmd.exe does not quite meet my standards, even on Windows 7. Perhaps I'll look up TCC, but honestly I'd expect a better act on behalf of Microsoft. I had a better time with 4DOS 20 years ago running on MS-DOS 5. Oh, don't even get me started on "edit.com"..

    Si si, I'm a terminal snob.

Seems that py2exe is just what I need. It installed fine via easy_install into XP. Their wiki houses the information how to construct the setup.py for py2exe to build it. Mainly, name of the main script for the executable and which libraries to bundle. It creates a whole number of files, totalling 26 MB. I was unable to generate a single executable file, but this does the trick.

 

.vimrc + Python indentation

I had two separate projects (vcpynet-client and MXManager) which belong together. MXManager is a GUI in PyQt4 for vcpynet-client (which is now renamed to mx-download). That is worth another post. I needed to re-indent the two codes to have the same indentation style. In Ruby I tend to use 2 spaces. In Python it feels a bit more natural to use 4 spaces. These files used both styles which is ok as long as the indentation is strict in each separate file. Re-indenting in Vim should be simple. I defined these in my .vimrc:

set tabstop=4
set softtabstop=4
set shiftwidth=4
set shiftround
set expandtab
set autoindent
set paste

Vim re-indents the file in normal mode by ”gg=G”. Whoops, didn't work on my files. Guess I need to be more strict. I break some lines in a bit unorthodox fashion but Python 2.5 seems to accept it. Vim doesn't. Sed accomplised it, so for the record:

sed -i 's/\ \ /\ \ \ \ /g'