Zope on Windows: Fun, Fun, Fun

A nightmare introduction running external programs on Windows

Introduction

For a Zope recent project I was in charge to develop a solution to generate PDF and RTF from HTML. I choose XSL-FO as general framework to provide a homogeneous environment for such transformations and conversions. There are bunch of FO processors on the marked and I took some commercial Java-based FO processors. Early tests worked fine and the first implementations worked fine from within Zope on Linux. The content presented to the customer could be transformed on the-fly to plain text, PDF or RTF. But fun started when I was trying to get the stuff working on Zope unter Windows. The target platform for the whole application is Windows (from Windows XP down to Windows '98).

Let's get technical

Running external processes from Python is usually not a big deal. You have different options:

os.system()

command.getoutput()

os.popen()

subprocess.Popen()

...and some more variations. Basically all methods provide a way to run a command like

java -jar some.jar arguments

This also works fine on XP but the fun starts when you want to run the same code under different Windows flavours. My very first implementation was like this:

P = subprocess.Popen(cmd, shell=False, stderr=PIPE, stdout=PIPE; stdin=None)

status = P.wait()

stdout = p.stdout.read()

stderr = p.stdout.read()

This works fine on XP but not on Windows 98.  The issue that came up was an exception from the subprocess module where Python tried to obtain a handle for stdin internally through some win32 API call. So the first workaround was:

P = subprocess.Popen(cmd, shell=False, stderr=PIPE, stdout=PIPE; stdin=open('nul:')

Using this workaround I was able to run java.exe -version to check for the installed Java version. However using this code caused an empty DOS window to popup for every execution. The solution was to use the startupinfo parameter:

startupinfo = STARTUPINFO()

startupinfo.dwFlag |= STARTF_USESHOWWINDOW

P = subprocess.Popen(cmd, shell=False, stderr=PIPE, stdout=PIPE; stdin=open('nul:'), startupinfo=startupinfo)

Now with this solution I could execute Java command without anyone noticing it :-) But....wait....our Java commands also produced some output..maybe only some lines or a hundred warnings..you would expect that you can read the error messages from the stdout and stderr pipes....surprise, surprise....that worked only for short messages. In case the Java process wrote too much data to stderr then whole process was just hanging with the result that os.wait() had to wait forever. Because of my limited Windows knowledge I could not track down this problem on Windows 98. So the workaround at this time was to avoid using stderr and stderr as PIPEs and just setting them to None.  So with the final solution I was able to run all our converters using the same code on all Windows flavours...but I really took me very long (serveral days with lots of iterations with external testing labs) to figure this all out....and it convinced me again that Windows just a piece of crap...