Paver and the building, distribution, deployment etc. of Python projects

This morning, I released a new open source “build tool” aimed at Python projects: Paver. The goal of Paver is to provide a smooth way to script up the management of your Python projects. You can read all about it on the Paver site, but I wanted to provide some background here.

Look at all these tools!

Python programmers have a great many tools at our disposal. We have tons of libraries that make it so that we don’t have to write lots of code to get our software built. We also have a broad collection of tools to help us manage our projects.

  • Python’s standard library includes distutils, for packaging up and distributing Python projects.
  • setuptools is almost part of the standard library, and quite a few projects require it. setuptools gives you cross-platform dependency management, more packaging options, script generation and a simple plugin framework.
  • zc.buildout helps you with the creation of repeatable, easily installed ready-to-run installations of projects. It gives you a contained environment so that you don’t need to muck with the global Python configuration on the system to make a working installation.
  • zc.buildout supports “recipes” that handle installation and configuration of various parts that your Python project may need
  • virtualenv gives you just the contained installation part of zc.buildout, but it does it in a slightly different way that I’ve found easier for certain, not-egg-friendly projects.
  • PasteScript can be used to generate configuration files and complete skeleton projects
  • Sphinx is a new package for generating documentation from ReStructred Text sources. It’s very cool, and it’s what I used for Paver’s site.
  • and there are many, many more

Seems great, what’s the problem?

I have personally used all of these tools at one time or another. In fact, I’ve used them all recently. In working with them, I couldn’t help but notice some aspects that made my life harder than it needed be.

For example, when using distutils or setuptools, it’s very easy to add behavior that runs before or after the setup command, because your setup files are just plain Python. It’s not as easy to customize the way a command behaves, or to add a new command entirely. You need to read the docs, make a new class and register that class somehow.

zc.buildout is awesome and makes it easy to get a predictable collection of components installed. It uses an INI file as its file format, which means that adding behavior is not straightforward. Creating a new zc.buildout recipe is very much like creating a new setuptools command: create a separate class and refer to it an in egg. I believe there’s a zc.buildout recipe for putting some commands in your INI file. Do you want Python code in your INI file?

Which also brings up another point: distutils and setuptools use a Python file and keyword parameters for their configuration. (There is also an optional INI file.) zc.buildout uses an INI file. Sphinx uses a .py file.

What would I want?

It seemed to me that life would be better if:

  • If I need to do something that takes 5 lines of Python, I could do it in little more than 5 lines of Python without adding another file for that purpose.
  • If configuration could take on a consistent, predictable form
  • If things that I do often in managing my projects took even less Python to script.
  • If the system could be used easily with multiple projects by not requiring anything else, but taking advantage of other packages when they’re present

It is with those goals and looking around that Paver came into existence. As with TurboGears, I did not want to reinvent the various parts of the whole that I’m angling for. The idea is to use zc.buildout’s machinery, not reinvent it. I used Jason Orendorff’s great path.py module rather than inventing my own abstraction there.

I didn’t set out to invent the scripting format, either. I seriously considered Zed Shaw’s Vellum which has shaped up quite nicely. But in trying it out, I realized that I really wanted my projects managed by Python scripts that had little headache and little overhead. Doing computations, loops and breaking code up into separate functions (or other organizing blocks) are all obvious for a Python programmer if the language is Python. For the record, Zed wants his build files to be just “data”, for perfectly rational reasons. For me, though, I want Python.

Now for the “this is an early release caveat“. Paver is functional, and I use it. But, its support for the various libraries is quite shallow right now, and zc.buildout/virtualenv are not at all represented yet. What I’ve released is basically the parts that I’ve needed so far, and I’ll be adding on as I need things. I figured that if others think the approach is worthwhile, we can pool our efforts and build out the Paver Standard Library a bit quicker. I should also note that while Paver should work on Windows, it’s only been used on Macs and Linux. Finally, it’s possible that Paver’s pavement.py syntax may change along the way to 1.0, but I can promise to document those changes and I don’t expect a great deal of pain in making the transitions.

Note also that if you maintain a Python library that is useful in helping people work with the projects, it’s easy to add Paver targets and such to your own library. Paver itself includes support for other libraries because of the chicken-and-egg problem. People won’t support Paver until people are using it and people wouldn’t use it if it didn’t support the kinds of things that people already do. So, Paver includes the connectors for the libraries that I need.

I’ve set up all of the various project goodies for Paver:

Ian Bicking has given me a great mass of useful feedback, which I have not yet fully digested. Mark Ramm, Ben Bangert and David Stanek also had some helpful input.

And, I’d love to hear more from you!

6 thoughts on “Paver and the building, distribution, deployment etc. of Python projects”

  1. This looks great — I really liked rake, but I’ve been looking for something with a bit more of a pythonic feel. I just wanted to encourage you to keep hacking on it!

  2. To me, with “modern” build systems, I always feel like they make the general case easy, but anything diverging is difficult. I sometimes write makefiles like scripts just because of the simplicity of them for the desired task. I’ve looked through the documentation but something wasn’t too clear. What I’d like to know is if there is a provided way to do some basic Makefile stuff of file (non-phony) targets/dependencies, especially generic ones?

    In Makefile terms, something like
    %.o: %.cpp
    $(CC) $(CFLAGS) -o $@ $<

    If not, One simple way is for
    @file_target
    That takes a filename, regex, or matching function. A difficulty could be is if you have a generic target and dependency where one is generated from the other. So one possibility is
    @file_target_and_need
    That takes a substitution regex or a matching function that returns a tuple.

  3. @Danny: thanks for the encouragement! Give it a try, the basics work just fine.

    @Ed: I haven’t implemented that, because I haven’t needed it thus far. My personal focus is on the libraries that give the higher-level behavior, but I’m opening it up now so that others can add in the things that are useful to them.

    I should mention that it’s easy to do things in a non-declarative manner. This is, after all, Python. Where I’ve found that I get tripped up is with declarative systems that don’t provide a full and easy language to do other tasks. Paver doesn’t suffer from that. The bigger trick is going to be building up the right declarative systems on top.

  4. “Python has a very concise, readable syntax.”

    Yeah. Except that it sucks for all kinds of configuration. Paver seems like an interesting project. Please do not do configuration in Python. I beg thee. (Yeah, I know, you made up your mind already).

  5. In what way does it suck? If there’s a compelling reason I might consider *adding* support for a different configuration format. But, I specifically don’t want the limitations of a “dumb” format.

    Is it just that you don’t like the extra punctuation?

Comments are closed.