Changes coming to Paver

Paver is the Python-based build tool that I released several months ago. It uses a Rake-like approach; you create a .py file that is filled with “tasks” which are basically just functions that can be invoked from the command line. Paver has largely been focused on automating Python projects.

Last week, I got together with Marshall and Matt from Zattoo. They’ve been working on a tool called pytoss which is focused on deployment. There are two parts to pytoss: the library and the “tool”. The library is being broken out into a new project called CloudControl, and it provides all kinds of high-level, handy deployment goodies on top of Paramiko. And it’s liberally licensed.

There’s a lot of overlap between pytoss’ tool component and Paver. So, we got together to see what we can do about that. The two approaches were already quite similar. There are a handful of small differences, and I like some of the pytoss approach. At a high-level, here’s what’s going to happen:

  • the pytoss tool part will go away
  • Paver will become less Python project-specific (but won’t lose the Python project-specific features it has)
  • the small bit of magic that Paver has will go away
  • Paver will add some optional support for CloudControl for deployment
  • Paver will also inherit some nice features like the easy ability to run sub-builds (build other Paver-based projects)

These are just the changes that are in store based on the pytoss integration. There are some other cool features that I have in mind.

I have also decided to move Paver from Launchpad to Googlecode. I find Launchpad to be far more confusing, especially for a small project like Paver. Additionally, bzr’s svn plugin appears to make it so easy to sync with a central server that there’s no reason to make everyone use bzr. Those who want to, can. And those that are used to svn can use svn.

Paver’s official homepage is unchanged, there’s just a new project page, bug tracker and source control URL.

A big thanks to Matt, Marshall and Jonathan for CloudControl and for helping out with Paver!

9 thoughts on “Changes coming to Paver”

  1. Did you look at bitbucket.org ? It seems much simpler than Launchpad, and has pretty much the same basic features as Google Code. I haven’t used it for a project myself, but I’d probably try it if I was going to move things around. One of the problems both Launchpad and Bitbucket seem to be addressing in DVCS is indexing people’s branches and making them easily public; svn mirroring itself doesn’t do this (unless people mirror then put the code up on one of those services, which at least makes it public, but not something people will notice).

    Also if you are going to focus on larger construction (which pytoss seems oriented towards, and doing subbuilds implies) I’d reemphasize that I think a file/subcommand abstraction layer is really important. Doing proper error handling and logging is just too hard otherwise, and so people won’t do it, and you’ll end up like buildout and periodically infuriate your users with bad behavior (which technically won’t be paver’s fault, instead it’ll be the fault of the people writing the builds/recipes, but personally I think that’s a cop out).

  2. Yes, you’re right about Launchpad and Bitbucket supporting making multiple branches public… but, that’s not really the use case I’d want to optimize for right now. At least not for this project…

    I do agree that error handling is important and helping people track down problems is a good thing. A larger abstraction than path.py may be useful as well for interacting with remote systems (via CloudControl).

    If one routes all file and command operations through path.py and Paver’s other functions, then logging is straightforward. I know that you are big on having interaction when things don’t seem right. Was there any other kind of error handling you were thinking of (beyond what appears in your June blog post).

    Thanks for the feedback!

  3. Well, here’s just some general stuff…

    Logging is important — and creating a good balance between being helpful and being overly verbose. I’ve found indentation to be very helpful in making the logs readable, to have different scopes of what’s happening. In fassembler (and pip) I also save all the logs in memory, so in case of a problem you have a complete set of logs, and you don’t have to rerun with more verbosity. I find the logging module to be hopeless in these respects, and would highly recommend ignoring it. Maybe it’s useful for some cases, but this is not one of them. Most of the logging stuff I’ve made for fassembler is in CmdUtils.

    Calculating smart defaults is important too. This is something that has to happen as part of making the build, but the build tool should facilitate it. An earlier system we used made it very hard to get all the settings right and consistent with each other. Debugging configuration sucks, and a big part of that is when settings have to be consistent with each other in unclear ways.

    I’d also say that the ability to mix configuration and code has been very handy. I’ve been quite happy with treating every setting as a template, and doing interpolation on that template. This makes smart defaults a lot easier to implement.

  4. Hi, I’m thinking about using Paver for one of my projects – one thing I like is that while it supports many things, it’s small enough to be redistributed so that end users won’t have to worry about it. I hope that this plan won’t make Paver more bloated.

  5. @Ian: Thanks for the feedback. I like the logging to memory idea. That’s a great idea.

    @Orestis: Paver will not become more bloated (if anything, it is likely to have an even smaller form factor available and you add in pieces as you need them). I’m quite happy with the way the paver-minilib works and have no plans to harm that!

  6. This is good news, I have been wanting to take a real use of paver since you first launch it, I’m glad it’s coming along. As for hosting google code is good, for some reason everyone is running away from launchpad. but you should also check bitbucket they are doing awesome things this days.

  7. Any chance of an update on this? I’m currently evaluating deployment tools and Paver without pytoss / CloudControl would be far less useful to me ๐Ÿ™‚ It looks from the outside like CloudControl had some initial activity after its fork from pytoss but both projects have now been idle for over a month, so I can’t help but feel it was a neat idea that died.

  8. Actually, Paver has been very active… (at least, I’ve been hacking on it ๐Ÿ˜‰

    It’s moved from launchpad to googlecode: http://code.google.com/p/paver

    Unfortunately, the CloudControl part has seen no work. I’ve been active on cleaning it up, removing some magic, better handling distutils, better structuring the code. I don’t know the current word on CloudControl.

    At PyWorks, Holger showed off py.execnet which also looks interesting for deployment.

    Anyhow, I’ve made a bunch of progress on Paver’s basic structure, so 1.0 is not far off. You can easily use CloudControl or py.execnet without any special support and Paver will still help with project scripting. The special support would just make things even easier…

  9. OK, thanks for the update. Paver is almost certain to make it into our build/install chain, but I think we might end up using Fabric for remote deployment.

    py.execnet is pretty cool, isn’t it ๐Ÿ™‚

    I toyed with extending it at one point so you could just define some generator function (which accepts a channel as its arg) and then gw.remote_exec(generator). Lost the code tho.

Comments are closed.