Python packaging/install: what I want

Python packaging and deployment can be annoying. It’s been nearly 4 years since I released the first TurboGears release as an early adopter of setuptools/easy_install. Since then, there’s been the release of virtualenv, pip and zc.buildout. Somehow, it still seems like more trouble than it should be to get development and production environments set up.

On Bespin, I’ve been using a combination of virtualenv and pip (scripted with Paver) in development and production environments. But, I’ve found pip –freeze to be nearly unusable.

My Ideal World

After monkeying with this stuff a fair bit over the past few years, I have an idea of what I’d really like to have but I don’t think anyone’s working on it. I’d love to hear contrasting opinions or learn about projects that I’m not aware of.

  • Multiple version installation into global site-packages, as easy_install currently works (put the active package in the .pth file)
  • The better error reporting of pip (pip doesn’t meet my first desire, though, because it installs as single-version-externally-managed)
  • A tool to manage the installed packages (uninstall, select a different version)
  • In addition to a global site-packages, it would be nice to be able to specify a different site-dir for machines where I don’t have or don’t want to use root access
  • virtualenv that behaves like –no-site-packages but knows where site-packages (or the other site-dir) is
  • That tool that manages installed packages can selectively install specific versions of packages into the virtualenv by adding pointers in the .pth file that point to the site-packages directory
  • You can also install only into the virtualenv if you wish.
  • Install packages in that manner from a list of requirements (as with pip’s requirements file)
  • A way to freeze the currently set installed into the virtualenv as a new requirements file
  • An optional cache of all of the original sdists of the installed packages

pip is close to being usable, except freeze doesn’t work. zc.buildout is close to being usable, too. I think there’s a “freeze” like plugin for it, but I don’t know how well it works. I don’t like zc.buildout quite as much as virtualenv, and I see that people even use virtualenv+zc.buildout to eliminate site-packages from leaking in. I also find that it leaves tons of old packages around in every buildout, again with no way to manage them.

What I’ve found using both zc.buildout and pip is that they are slow and annoying, because they’re constantly reinstalling things that I already have. The main reason for having a shared site-packages as I suggest above is not to save on disk space, but to save on time. In development, I want to be able to update to the latest versions of packages quickly, installing/building only the ones that have changed. How fast something runs changes how you use it, and I know that the scripts that I have for updating development and production environments reflect that.

So,I think the main thing that I’m looking for is a new tool to manage the packages that I have installed globally and within virtualenvs. Are there tools out there that are heading down this path at all?

Also, I understand the starting point that Tarek is taking with Distribute (splitting it up into logical pieces), but is there any roadmap for where it’s going to go functionally from there? Or is the intention purely that tools like the one I’m angling for will be written against the newly refactored libraries? I do know about the uninstall PEP, and that’s pleasing.

Project Bespin Summary for August 28, 2009

“This Week in Bespin” returns as “Project Bespin Summary”.  It was clear to me some time back that weekly was not the right pace for delivering condensed Bespin news. But there is also interesting backstory and plenty that happens in between major releases, so we wouldn’t want major releases to be the only time for people to learn about what’s going on with the project.

The last major release, Bespin 0.4, got plenty of attention around the web and I posted an announcement message on the Bespin list after the release. So, here’s a quick round up of what’s been happening since the 0.4 release:

  • We released Bespin 0.4.1 on August 17th which corrected some issues spotted after the release of 0.4 (15 bugs closed). This release added a minor feature: you can set your username to be used when doing commits in a distributed version control system.
  • On August 27th, we released Bespin 0.4.2 which thus far appears to have fixed some locking and stability issues we’ve had. Ben blogged about the Bespin 0.4.2 release.
  • Ben posted an updated short-term roadmap for Bespin on his blog. You’ll note that release names have now changed from a cloud theme to some other theme that I will leave as an exercise for the reader.
  • Before you know it, Bespin 0.4.3 will be out. The ability to run familiar “svn” commands is already available on the Bespin hg tip for a few commands.
  • Git support for Bespin is very much in demand. You can help!
  • We’re hiring! Specifically, we’re looking for someone that will (at least initially) be responsible for the Canvas-based editor at the heart of Bespin. I have a post up on my blog as well.

The next Project Bespin Summary is likely to come out soon after the 0.4.3 release.

Project Bespin needs *you*

I work in Mozilla’s Developer Tools Lab, where we’re working to make things easier for web developers and to explore what is possible using open web tools. It turns out there’s quite a bit that’s possible, and we’ve just scratched the surface with what we’ve done with Bespin so far.

I’m really happy with the team that I’m working with, both within Mozilla Labs and in the Bespin community. I’m also really happy to report that we’ve got an opening on our team! We’re looking for fantastic software engineer that will own the Canvas-based editor component that is at the heart of Bespin.

If being at the forefront of JavaScript technology (Canvas, local storage, web workers) sounds fun and exciting to you, drop us a line! We’d love to talk to you!

Camtasia for Mac arrives and brings real competition

I’ve done a good deal of screencasting, including a very popular video (the TurboGears 20 Minute Wiki) and even a screencast DVD that I offered for sale. When I first started screencasting, the only viable choice on the Mac was Snapz Pro X, a simple screen recorder. This was a far cry from Camtasia Studio on Windows, which was the top-of-the-line tool. In 2007, ScreenFlow appeared on the Mac and added an elegant editing experience to Mac screencasting that closed the gap with Camtasia enough.

That said, Mac users have been clamoring for Camtasia on the Mac for a long time. My most popular blog post since the beginning of 2006 is “Camtasia for the Mac”. In addition to a continuous stream of page views, that page also has 140 comments. It was the top hit on Google for “camtasia mac” for a long time.

At long last, Camtasia for Mac is now available. From the preview video on TechSmith’s site, you can tell that they definitely paid attention to ScreenFlow. Camtasia Mac appears to have all of the polish of ScreenFlow and brings some excellent looking new features that ScreenFlow does not have (more transitions and effects, direct publishing to YouTube).

I just noticed that Telestream has pre-announced ScreenFlow 2.0 which adds transitions and YouTube publishing. The timing of this announcement and those features clearly is a direct response to Camtasia for Mac. One thing is for certain, this competition will be good for those of us using Macs to make screencasts.

Congratulations to the hard working folks at TechSmith who have gone from being a Windows shop to producing a very slick Mac application.

One Python-based version control system to rule them all!

We’ve just released Bespin 0.4, the major new feature of which is the first bit of the collaboration feature. Bespin 0.4 includes a ton of other changes, including one that I’m going to focus on here: Subversion support, which Gordon P. Hemsley kicked off for us a few weeks back.

Bespin’s initial version control support showed up in 0.2 with support for Mercurial. Knowing that we wanted to support multiple version control systems (VCS), I took an unorthodox approach from the beginning. Rather than providing the “hg” commands that people know and love, I created a separate set of “vcs” commands. Ultimately, we want to make it easy to grab a random open source project of the net and start hacking on it. Using the “vcs” commands, for the most common version control operations you won’t even have to think about which VCS is used by a given project.

I can run “vcs clone” (“vcs checkout” also works) to check out Bespin (in a Mercurial repository), Paver (in a Subversion repo) and hopefully soon Narwhal (in a Git repo). Also new in Bespin 0.4: Bespin’s command line has been tricked out to be able to have fancier interactions with commands, so you can enter all of the extra information that Bespin needs for checking out a repository right in the output area.

If you’ve used Subversion and one of the Distributed VCSes, you’ll know that they have a different model. The DVCSes do almost everything in a local repository copy and only talk to a remote server for push/pull. That’s actually true of Subversion as well, with one notable exception: commit. For Subversion, the “vcs commit” command will simply save your commit message for later. When you run “vcs push”, that is when an actual “svn commit” operation is run.

What’s neat about the “vcs” commands is that they operate the same from VCS to VCS. svn doesn’t have a feature to “add all files that are unknown”, whereas Mercurial does. “vcs add -a” operates the same on both systems.

If you’re interested, you can also use these commands on the command line by installing the Ãœber Version Controller (uvc) Python package. After doing so, you can head into a random Subversion or Mercurial working copy and type “uvc status” to see what’s different. I will note that the command line tool has been, um, lightly tested since uvc is mostly used as a library for Bespin at this point.

One final note: Bespin will soon also support native “svn” and “hg” commands so that you can stick to commands and options you’re familiar with or for performing more complex operations that don’t have equivalent “vcs” commands.

You can learn more about version control in Bespin from this section of the User Guide.