Blue Sky On Mars

We’ve settled on a topic for the coming Michigan Python Users Group meeting: Google App Engine. We do not have a speaker, so this will be a combination of discussion and exploration. I have written some App Engine code, and I’m guessing that others that will be attending have, too, so it’s not all trial-and-error.

Following our usual formal (first Thursday of the month), the meeting will be on June 5th at 7PM at SRT Solutions in downtown Ann Arbor. There are parking structures nearby (4th and Washington and Liberty Square are both close), and there is usually free street parking on Ann Street near the police station.

See you there!

Justin Davis has just announced that Christophe de Vienne is the new maintainer of the TurboGears Web Services (TGWS) project. Thank you Christophe for carrying this project forward! I consider it a good sign for a project’s health when it can make the transition between maintainers, which is something both TurboGears and TGWS have done.

TGWS, if you’re not familiar with it, is a multiprotocol web services extension for TurboGears 1.

For various reasons, I released Paver 0.7.3 on Friday and Paver 0.8 yesterday. I have an idea of what the coming 1.0’s zc.buildout integration will be like, and I think it will be quite cool and useful.

In the meantime, though, I’ve got some new features that set the stage for things I need to do in 1.0. Specifically, you can now pass in a dictionary in your options search ordering. So, you can pull options from any source you’ve got at the time the task is running and stick them at the front of the line. I expect to use this in buildout options handling.

A nice new feature is the ability to set options on the command line. You can do something like:

paver some.option=hello task1 some.option=goodbye task2

Doing that, paver will set some.option to hello, run task1 and then change the option to goodbye before running task2.

The new cog.include_markers and cog.delete_code options allow you to remove Cog’s markers from the output and instead put a nicer bit of text to say where the snippet of code came from. Letting the user know where a sample code snippet came from is quite valuable, so I want to make it possible to do so in as pleasing a way as possible.

For Paver’s Getting Started Guide, I ended up not using the new include_markers feature and instead just changed the Cog delimiters. I did this because Paver runs shell commands in addition to including file sections when generating the docs. I wanted those shell commands to be included. I think the new markers are more pleasant to look at, and I’ll be curious to get feedback since I heard from more than one person that the Cog delimiters looked like they were left in by mistake.

Paver is starting to get some traction as it has picked up its first patches from outsiders, and I’ve started to get some feedback on breakage from Windows users (fixed in 0.8). Mark let me put Paver into TurboGears 2, and I think it will help out there, so that will introduce quite a number more people to the project. As always, come and join us on the mailing list if you have any questions or problems!

I’m delighted to release Paver 0.7. If you missed my original announcement, the short story is that Paver is a new build, distribution and deployment scripting tool geared toward Python projects. My original announcement and the new foreword to the docs explain the motivation.

Ben Bangert and others pointed out a giant documentation bug in 0.4: there was a fair bit of reference doc but no doc that said “here’s how you get started with Paver”. Now there is: Paver’s Getting Started Guide.

Paver 0.7 is a big step up from 0.4 (hence the version number bump). I implemented one of the two major features I had planned for 1.0: distutils/setuptools integration. It’s really cool. Have you ever wanted to just slightly change how “sdist” or “upload” or “develop” worked? Now you can, just by writing a function in your pavement.py file. And don’t worry, you don’t need to duplicate anything between setup.py and pavement.py. It all just moves into pavement.py and Paver can even generate a setup.py file for you, since most people are use to the common “python setup.py install” command.

I’ve gone even farther than that with making it easy to use Paver and not annoy users that don’t yet have Paver. Paver can create a small zip file of Paver’s core bits so that “python setup.py install” will work just fine even for users who don’t have Paver installed. Paver can also create a virtualenv bootstrap script for you, so that users don’t necessarily need to install your package on their systems in order to use it.

Paver’s got new documentation tools that work great with Sphinx. It’s now easy to mark sections of sample code files and then include those sections in your documentation, using the built-in version of Ned Batchelder’s Cog.

And I’m definitely eating my own dogfood. Paver is built using Paver itself and the source distribution includes the paver-minilib so that setup.py install should work fine (let me know if it doesn’t!) The new Getting Started Guide uses the new documentation tools.

There are even more changes than these, and you can look at the changelog for the full list. Note that if you’re using Paver 0.4, there are a couple of trivial breaking changes.

Seeing this on Bob Ippolito’s blog might seem a little odd to many:

Rewrote test suite with unittest and doctest (no more nosetest dependency)
[From simplejson 1.9]

Why on Earth would someone change from nose-style tests to unittest tests? How about so that the library can go into the Python Standard Library?

simplejson will be a great addition. Thanks to Bob and the others who are working to get simplejson in!

The May Michigan Python Users Group (MichiPUG) meeting is coming up on Thursday, May 1. I’ll be leading discussion on zc.buildout and Paver. Any build/distribution/deployment tool talk is welcome, as well as our usual general talk about Python topics.

I hope to see you there!

This morning, I released a new open source “build tool” aimed at Python projects: Paver. The goal of Paver is to provide a smooth way to script up the management of your Python projects. You can read all about it on the Paver site, but I wanted to provide some background here.

Look at all these tools!

Python programmers have a great many tools at our disposal. We have tons of libraries that make it so that we don’t have to write lots of code to get our software built. We also have a broad collection of tools to help us manage our projects.

• Python’s standard library includes distutils, for packaging up and distributing Python projects.
• setuptools is almost part of the standard library, and quite a few projects require it. setuptools gives you cross-platform dependency management, more packaging options, script generation and a simple plugin framework.
• zc.buildout helps you with the creation of repeatable, easily installed ready-to-run installations of projects. It gives you a contained environment so that you don’t need to muck with the global Python configuration on the system to make a working installation.
• zc.buildout supports “recipes” that handle installation and configuration of various parts that your Python project may need
• virtualenv gives you just the contained installation part of zc.buildout, but it does it in a slightly different way that I’ve found easier for certain, not-egg-friendly projects.
• PasteScript can be used to generate configuration files and complete skeleton projects
• Sphinx is a new package for generating documentation from ReStructred Text sources. It’s very cool, and it’s what I used for Paver’s site.
• and there are many, many more

Seems great, what’s the problem?

I have personally used all of these tools at one time or another. In fact, I’ve used them all recently. In working with them, I couldn’t help but notice some aspects that made my life harder than it needed be.

For example, when using distutils or setuptools, it’s very easy to add behavior that runs before or after the setup command, because your setup files are just plain Python. It’s not as easy to customize the way a command behaves, or to add a new command entirely. You need to read the docs, make a new class and register that class somehow.

zc.buildout is awesome and makes it easy to get a predictable collection of components installed. It uses an INI file as its file format, which means that adding behavior is not straightforward. Creating a new zc.buildout recipe is very much like creating a new setuptools command: create a separate class and refer to it an in egg. I believe there’s a zc.buildout recipe for putting some commands in your INI file. Do you want Python code in your INI file?

Which also brings up another point: distutils and setuptools use a Python file and keyword parameters for their configuration. (There is also an optional INI file.) zc.buildout uses an INI file. Sphinx uses a .py file.

What would I want?

It seemed to me that life would be better if:

• If I need to do something that takes 5 lines of Python, I could do it in little more than 5 lines of Python without adding another file for that purpose.
• If configuration could take on a consistent, predictable form
• If things that I do often in managing my projects took even less Python to script.
• If the system could be used easily with multiple projects by not requiring anything else, but taking advantage of other packages when they’re present

It is with those goals and looking around that Paver came into existence. As with TurboGears, I did not want to reinvent the various parts of the whole that I’m angling for. The idea is to use zc.buildout’s machinery, not reinvent it. I used Jason Orendorff’s great path.py module rather than inventing my own abstraction there.

I didn’t set out to invent the scripting format, either. I seriously considered Zed Shaw’s Vellum which has shaped up quite nicely. But in trying it out, I realized that I really wanted my projects managed by Python scripts that had little headache and little overhead. Doing computations, loops and breaking code up into separate functions (or other organizing blocks) are all obvious for a Python programmer if the language is Python. For the record, Zed wants his build files to be just “data”, for perfectly rational reasons. For me, though, I want Python.

Now for the “this is an early release caveat“. Paver is functional, and I use it. But, its support for the various libraries is quite shallow right now, and zc.buildout/virtualenv are not at all represented yet. What I’ve released is basically the parts that I’ve needed so far, and I’ll be adding on as I need things. I figured that if others think the approach is worthwhile, we can pool our efforts and build out the Paver Standard Library a bit quicker. I should also note that while Paver should work on Windows, it’s only been used on Macs and Linux. Finally, it’s possible that Paver’s pavement.py syntax may change along the way to 1.0, but I can promise to document those changes and I don’t expect a great deal of pain in making the transitions.

Note also that if you maintain a Python library that is useful in helping people work with the projects, it’s easy to add Paver targets and such to your own library. Paver itself includes support for other libraries because of the chicken-and-egg problem. People won’t support Paver until people are using it and people wouldn’t use it if it didn’t support the kinds of things that people already do. So, Paver includes the connectors for the libraries that I need.

I’ve set up all of the various project goodies for Paver:

• Website
• Launchpad project
• Public bzr branch:
  bzr branch http://bazaar.launchpad.net/~dangoor/paver/main
• Mailing list

Ian Bicking has given me a great mass of useful feedback, which I have not yet fully digested. Mark Ramm, Ben Bangert and David Stanek also had some helpful input.

And, I’d love to hear more from you!

I don’t know if this preannouncement comes as a result of all of the Google App Engine publicity, but here it is: Amazon Web Services Blog: Storage Space, The Final Frontier. In a nutshell: AWS now lets you create a storage volume of 1GB to 1TB that can be mounted in one EC2 instance and will persist beyond the lifetime of an EC2 instance. As an added bonus, you can have automatic snapshots of your volume plunked into S3.

They say that this storage is a low-latency, high-throughput block device. So, you can run all kinds of traditional software on top of it.

This will change the competitive outlook a bit between AWS and GAE a bit, because it makes it easier for people to use all of the software pieces that they’re used to when they use AWS to manage the hardware infrastructure. This means that it’s easier to take your existing apps and skills and get them up on AWS. GAE has a fight ahead in terms of getting people to write their apps differently… but the benefit to doing so is that you no longer think of hardware infrastructure at all.

Isn’t quoting out of context great:

I hate computers.
[From Ian Bicking: App Engine: Commodity vs. Proprietary]

More seriously, though, in saying “I hate computers”, Ian is actually talking about the opposite of being a Luddite. He’s dreaming of a world in which much of computing just works in the background, so that we can spend our time doing more important and interesting things in the foreground.

I’m linking to Ian here because he’s said exactly what I have been thinking about App Engine: from a Python programming perspective the APIs are simple and clear. I can easily imagine a ZODB-based implementation of Google’s data store API. Just change your imports, and you can be off of Google’s infrastructure and on to your own.

Of course, for a great many people there won’t be any reason to be off of Google’s infrastructure. App Engine is just so darn easy. Amazon Web Services is impressive because it makes scalability affordable and available. App Engine interests me because, for its broad-but-still-limited set of use cases, it makes scalability a no brainer. “Build your app like this, and you never have to think of scaling” is a nice thought. I’ve been around enough to know that people using App Engine will still have to think of scaling some, but not nearly as much as with just about any other solution.

Back to the lock-in aspect, though. I still see App Engine as likely to be utterly unsuccessful with large businesses. That is, until a new Google Appliance comes out. I’ve been predicting such a beast since Google Docs was first introduced, and I think App Engine makes it all the more likely. I still believe that there will come a time when Google will sell boxes to big companies that those companies can toss into some racks on their networks and deploy App Engine apps locally, as well as run Google Docs on their private nets. Things will get even more interesting at that point.

You can bet that Amazon is studying App Engine closely and considering their own high-level service as I write this. From a developer’s perspective, this competition is going to be awesome.

It occurred to me just now that Google App Engine and Amazon Web Services are only barely in competition right now. If you want an infinite storage system like AWS S3 in App Engine, you need to code it yourself (ignoring the preview limits App Engine currently has). If you want to deploy apps as easily as you can with App Engine in AWS, you need a bunch of infrastructure that AWS does not provide.

I’m happy to see that App Engine’s datastore is transactional, unlike SimpleDB. I didn’t see anything in my skim of some docs about whether App Engine has eventual consistency or if you can immediately pull out data that you stuff in. My guess is that you can immediately pull out the data you shove in. This is a win over SimpleDB, in my opinion.

App Engine is just tons higher-level than AWS. Of course, you can host anything you want in AWS. But, by trading away a bunch of that flexibility, Google has made a service that allows people to build apps that scale well with a minimum of fuss.