Deploying distributed applications

Developing a strategy for application deployment is always hard. Every environment has its own characteristics. It's far easier to see the problem with popular applications that are installed by the end user. Custom applications are sometimes installed by an expert; sometimes the developer himself is available to do it. This helps to make the difficulty of this task to be greatly underestimated. The result is something that seasoned programmers will readily recognize: an installation that “should be a snap” takes many hours, or even days, to finish - and sometimes the developer just gives up as it discovers that the application, after all, was not ready for deployment. It's not enough to have the application working on the development environment.

Distributed applications bring this problem to a new level. The developer has now a lot of variables to predict; the interaction between the parts can have surprising results. It's not always obvious what is to best way to start up all the pieces, because they may exhibit circular dependencies. Bootstrapping a distributed application is much harder than installing a simple one.

I'm now writing code to 'bootstrap' one such application. It's a complex application that has several independent server processes communicating with XMLRPC. It's interesting to note how many issues pop up while doing it. For example: simple applications usually have a single test sequence. But distributed applications have to test combinations of components; they must fail gracefully when a server is not online. Some tests may require different combinations of servers, with different configurations. It's something that goes way beyond the features provided by the unittest module.

Starting the server in production mode is also a chore. It requires starting all individual servers, often in separate machines, all sharing some common configuration. This is an error prone task, specially if done manually. On the other hand, it's something difficult to automate because there are many things that can go wrong. At this point, I can only hope to get it right.

On a related note, I've found today a small gem: a Python library that encapsulates the Standard Windows Management Instrumentation framework. If you need it, check Tim Golden's wmi module. In my case, I had to download some updates from Microsoft to have WMI running on Win98. I'll post my experiences later when I finish testing it.