Paver is now on GitHub, thanks to Almad

Paver, the project scripting tool for Python, has just moved to GitHub thanks to Almad. Almad has stepped forward and offered to properly bring Paver into the second decade of the 21st century (doesn’t have the same ring to it as bringing something into the 21st century, does it? ๐Ÿ™‚

Seriously, though, Paver reached the point where it was good enough for me and did what I wanted (and, apparently, a good number of other people wanted as well). Almad has some thoughts and where the project should go next and I’m looking forward to hearing more about them. Sign up for the googlegroup to see where Paver is going next.

Paver: project that works, has users, needs a leader

Paver is a Python project scripting tool that I initially created in 2007 to automate a whole bunch of tasks around projects that I was working on. It knows about setuptools and distutils, it has some ideas on handling documentation with example code. It also has users who occasionally like to send in patches. The latest release has had more than 3700 downloads on PyPI.

Paver hasn’t needed a lot of work, because it does what it says on the tin: helps you automate project tasks. Sure, there’s always more that one could do. But, there isn’t more that’s required for it to be a useful tool, day-to-day.

Here’s the point of my post: Paver is in danger of being abandoned. At this point, everything significant that I am doing is in JavaScript, not Python. The email and patch traffic is low, but it’s still too much for someone that’s not even actively using the tool any more.

If you’re a Paver user and either:

1. want to take the project in fanciful new directions or,

2. want to keep the project humming along with a new .x release every now and then

please let me know.

Changes coming to Paver

Paver is the Python-based build tool that I released several months ago. It uses a Rake-like approach; you create a .py file that is filled with “tasks” which are basically just functions that can be invoked from the command line. Paver has largely been focused on automating Python projects.

Last week, I got together with Marshall and Matt from Zattoo. They’ve been working on a tool called pytoss which is focused on deployment. There are two parts to pytoss: the library and the “tool”. The library is being broken out into a new project called CloudControl, and it provides all kinds of high-level, handy deployment goodies on top of Paramiko. And it’s liberally licensed.

There’s a lot of overlap between pytoss’ tool component and Paver. So, we got together to see what we can do about that. The two approaches were already quite similar. There are a handful of small differences, and I like some of the pytoss approach. At a high-level, here’s what’s going to happen:

  • the pytoss tool part will go away
  • Paver will become less Python project-specific (but won’t lose the Python project-specific features it has)
  • the small bit of magic that Paver has will go away
  • Paver will add some optional support for CloudControl for deployment
  • Paver will also inherit some nice features like the easy ability to run sub-builds (build other Paver-based projects)

These are just the changes that are in store based on the pytoss integration. There are some other cool features that I have in mind.

I have also decided to move Paver from Launchpad to Googlecode. I find Launchpad to be far more confusing, especially for a small project like Paver. Additionally, bzr’s svn plugin appears to make it so easy to sync with a central server that there’s no reason to make everyone use bzr. Those who want to, can. And those that are used to svn can use svn.

Paver’s official homepage is unchanged, there’s just a new project page, bug tracker and source control URL.

A big thanks to Matt, Marshall and Jonathan for CloudControl and for helping out with Paver!

Let’s meet up in Atlanta at PyWorks!

pyworks_08_Speaker_button.jpg

This year has seemed like a big year for Python conference activity in the US. Of course, there was PyCon in March, which topped 1,000 attendees. I’ve also seen announcements for a bunch of regional Python gatherings (like PyOhio, which was close by but I couldn’t attend).

This year, we also get PyWorks, which is joined at the hip with php|works in Atlanta in November. This is the first year for PyWorks, and they’ve got a good lineup going. There’s a day of tutorials and two days of talks, so this is more like a PyCon than it is like those regional conferences.

In addition to attending talks myself, I’m hoping to meet some more good Python and/or JavaScript folks in the “hallway track”. I’m sure there will be lots of Dojo users mixed in with the Python and PHP people, so we should get together.

I have four speaking slots (one of which isn’t listed yet) over the two conference days (gadzooks!). I’ll be doing a revised and expanded version of my PyCon talk “Rich Client Web Applications with TurboGears 2 and Dojo”. I’ll also be giving an updated version of the “Easy build and deployment automation with Paver” talk that Mark Ramm gave in my stead at PyOhio. Paver really puts the “scripting” back in “Python scripting language” (Python certainly does a lot more than “scripting”!)

I’ll also be giving a talk called “ZODB: The Most Underappreciated Library in Python”. The ZODB is great. More people should use it. This is a talk I gave a couple months back at MichiPUG, so it’s only been seen by a small group at this point.

My fourth talk is one I haven’t given anywhere before: “Beyond the Source: Growing Your Community”. I’m going to talk in concrete terms about things you can do to grow an open source community. Open source projects really need to get to a certain level of use before they become viable open source projects, and there are many, many ways in which people interested in a project can help it get there.

I hope to see you there!

Thanks to Mark for channelling me at PyOhio

I had lunch with Mark Ramm-Christensen today, and he gave me the scoop on PyOhio which was last Saturday. It sounds like a good time was had by all. Mark guessed there were 80 people there, which is a good turnout for a regional event like PyOhio! Congrats to the organizers! It sounds like you did quite a job getting the word out and building the turnout.

I couldn’t be there because my daughter’s 5th birthday party was on the same day. Thanks to Mark, my Paver talk happened at PyOhio anyhow. He had the tough 2:30PM timeslot. If you’ve ever given a talk or sat through a talk at 2:30, you’ll probably remember that as “siesta time”. Everyone’s energy level tends to be very low at that point of the day.

So, thanks everyone for learning a bit about my new little project and to Mark for presenting my talk at that time and keeping the energy up.

Easy, Repeatable Building/Deployment of Python+Dojo Projects

My latest substantial blog post is now up: SitePen Blog ร‚ยป Easy, Repeatable Building/Deployment of Python+Dojo Projects

Dojo on the client and Python on the server make for a great combination. They’re easy, productive and powerful. In this article, I’ll show you how to use Python + Dojo to cut the number of requests to your server by 95% and simplify development and deployment while you’re at it.

Paver 0.8: new options features, doctools features and bug fixes

For various reasons, I released Paver 0.7.3 on Friday and Paver 0.8 yesterday. I have an idea of what the coming 1.0’s zc.buildout integration will be like, and I think it will be quite cool and useful.

In the meantime, though, I’ve got some new features that set the stage for things I need to do in 1.0. Specifically, you can now pass in a dictionary in your options search ordering. So, you can pull options from any source you’ve got at the time the task is running and stick them at the front of the line. I expect to use this in buildout options handling.

A nice new feature is the ability to set options on the command line. You can do something like:

paver some.option=hello task1 some.option=goodbye task2

Doing that, paver will set some.option to hello, run task1 and then change the option to goodbye before running task2.

The new cog.include_markers and cog.delete_code options allow you to remove Cog’s markers from the output and instead put a nicer bit of text to say where the snippet of code came from. Letting the user know where a sample code snippet came from is quite valuable, so I want to make it possible to do so in as pleasing a way as possible.

For Paver’s Getting Started Guide, I ended up not using the new include_markers feature and instead just changed the Cog delimiters. I did this because Paver runs shell commands in addition to including file sections when generating the docs. I wanted those shell commands to be included. I think the new markers are more pleasant to look at, and I’ll be curious to get feedback since I heard from more than one person that the Cog delimiters looked like they were left in by mistake.

Paver is starting to get some traction as it has picked up its first patches from outsiders, and I’ve started to get some feedback on breakage from Windows users (fixed in 0.8). Mark let me put Paver into TurboGears 2, and I think it will help out there, so that will introduce quite a number more people to the project. As always, come and join us on the mailing list if you have any questions or problems!

Paver 0.7: Better than distutils, better docs and much more

I’m delighted to release Paver 0.7. If you missed my original announcement, the short story is that Paver is a new build, distribution and deployment scripting tool geared toward Python projects. My original announcement and the new foreword to the docs explain the motivation.

Ben Bangert and others pointed out a giant documentation bug in 0.4: there was a fair bit of reference doc but no doc that said “here’s how you get started with Paver”. Now there is: Paver’s Getting Started Guide.

Paver 0.7 is a big step up from 0.4 (hence the version number bump). I implemented one of the two major features I had planned for 1.0: distutils/setuptools integration. It’s really cool. Have you ever wanted to just slightly change how “sdist” or “upload” or “develop” worked? Now you can, just by writing a function in your pavement.py file. And don’t worry, you don’t need to duplicate anything between setup.py and pavement.py. It all just moves into pavement.py and Paver can even generate a setup.py file for you, since most people are use to the common “python setup.py install” command.

I’ve gone even farther than that with making it easy to use Paver and not annoy users that don’t yet have Paver. Paver can create a small zip file of Paver’s core bits so that “python setup.py install” will work just fine even for users who don’t have Paver installed. Paver can also create a virtualenv bootstrap script for you, so that users don’t necessarily need to install your package on their systems in order to use it.

Paver’s got new documentation tools that work great with Sphinx. It’s now easy to mark sections of sample code files and then include those sections in your documentation, using the built-in version of Ned Batchelder’s Cog.

And I’m definitely eating my own dogfood. Paver is built using Paver itself and the source distribution includes the paver-minilib so that setup.py install should work fine (let me know if it doesn’t!) The new Getting Started Guide uses the new documentation tools.

There are even more changes than these, and you can look at the changelog for the full list. Note that if you’re using Paver 0.4, there are a couple of trivial breaking changes.

Paver and the building, distribution, deployment etc. of Python projects

This morning, I released a new open source “build tool” aimed at Python projects: Paver. The goal of Paver is to provide a smooth way to script up the management of your Python projects. You can read all about it on the Paver site, but I wanted to provide some background here.

Look at all these tools!

Python programmers have a great many tools at our disposal. We have tons of libraries that make it so that we don’t have to write lots of code to get our software built. We also have a broad collection of tools to help us manage our projects.

  • Python’s standard library includes distutils, for packaging up and distributing Python projects.
  • setuptools is almost part of the standard library, and quite a few projects require it. setuptools gives you cross-platform dependency management, more packaging options, script generation and a simple plugin framework.
  • zc.buildout helps you with the creation of repeatable, easily installed ready-to-run installations of projects. It gives you a contained environment so that you don’t need to muck with the global Python configuration on the system to make a working installation.
  • zc.buildout supports “recipes” that handle installation and configuration of various parts that your Python project may need
  • virtualenv gives you just the contained installation part of zc.buildout, but it does it in a slightly different way that I’ve found easier for certain, not-egg-friendly projects.
  • PasteScript can be used to generate configuration files and complete skeleton projects
  • Sphinx is a new package for generating documentation from ReStructred Text sources. It’s very cool, and it’s what I used for Paver’s site.
  • and there are many, many more

Seems great, what’s the problem?

I have personally used all of these tools at one time or another. In fact, I’ve used them all recently. In working with them, I couldn’t help but notice some aspects that made my life harder than it needed be.

For example, when using distutils or setuptools, it’s very easy to add behavior that runs before or after the setup command, because your setup files are just plain Python. It’s not as easy to customize the way a command behaves, or to add a new command entirely. You need to read the docs, make a new class and register that class somehow.

zc.buildout is awesome and makes it easy to get a predictable collection of components installed. It uses an INI file as its file format, which means that adding behavior is not straightforward. Creating a new zc.buildout recipe is very much like creating a new setuptools command: create a separate class and refer to it an in egg. I believe there’s a zc.buildout recipe for putting some commands in your INI file. Do you want Python code in your INI file?

Which also brings up another point: distutils and setuptools use a Python file and keyword parameters for their configuration. (There is also an optional INI file.) zc.buildout uses an INI file. Sphinx uses a .py file.

What would I want?

It seemed to me that life would be better if:

  • If I need to do something that takes 5 lines of Python, I could do it in little more than 5 lines of Python without adding another file for that purpose.
  • If configuration could take on a consistent, predictable form
  • If things that I do often in managing my projects took even less Python to script.
  • If the system could be used easily with multiple projects by not requiring anything else, but taking advantage of other packages when they’re present

It is with those goals and looking around that Paver came into existence. As with TurboGears, I did not want to reinvent the various parts of the whole that I’m angling for. The idea is to use zc.buildout’s machinery, not reinvent it. I used Jason Orendorff’s great path.py module rather than inventing my own abstraction there.

I didn’t set out to invent the scripting format, either. I seriously considered Zed Shaw’s Vellum which has shaped up quite nicely. But in trying it out, I realized that I really wanted my projects managed by Python scripts that had little headache and little overhead. Doing computations, loops and breaking code up into separate functions (or other organizing blocks) are all obvious for a Python programmer if the language is Python. For the record, Zed wants his build files to be just “data”, for perfectly rational reasons. For me, though, I want Python.

Now for the “this is an early release caveat“. Paver is functional, and I use it. But, its support for the various libraries is quite shallow right now, and zc.buildout/virtualenv are not at all represented yet. What I’ve released is basically the parts that I’ve needed so far, and I’ll be adding on as I need things. I figured that if others think the approach is worthwhile, we can pool our efforts and build out the Paver Standard Library a bit quicker. I should also note that while Paver should work on Windows, it’s only been used on Macs and Linux. Finally, it’s possible that Paver’s pavement.py syntax may change along the way to 1.0, but I can promise to document those changes and I don’t expect a great deal of pain in making the transitions.

Note also that if you maintain a Python library that is useful in helping people work with the projects, it’s easy to add Paver targets and such to your own library. Paver itself includes support for other libraries because of the chicken-and-egg problem. People won’t support Paver until people are using it and people wouldn’t use it if it didn’t support the kinds of things that people already do. So, Paver includes the connectors for the libraries that I need.

I’ve set up all of the various project goodies for Paver:

Ian Bicking has given me a great mass of useful feedback, which I have not yet fully digested. Mark Ramm, Ben Bangert and David Stanek also had some helpful input.

And, I’d love to hear more from you!