Blue Sky On Mars

This post is a collection of random thoughts about the Bespin Reboot project so far.

The past summer, the Bespin team at Mozilla (Ben, Dion, Joe and me) started thinking that the time had come to make some significant changes to the Bespin project code. These changes would make it easier for others to use and contribute to Bespin and also make it easier for Bespin core contributors to add new features. We did a collection of releases during the summer and then slammed on the brakes for what we’ve called the “Reboot“.

The Reboot involves a serious refactoring of the code, plus changes in priorities and processes. We started fresh with a new source code repository (two, actually, since the Python server has been pushed into a separate repo). Then, we poured the code from the old repository into a new structure of CommonJS modules. We changed our underlying GUI framework to use SproutCore, so that we could spend less time thinking about getting things on the screen and more time with features that help people write their code.

Once you get into the groove of doing releases and adding good stuff for your users, it’s really painful to stop doing releases. I’ve been itching to ship since the beginning of October when we shifted full-time to the Reboot repository. Our goal all along has been to get the editor running again as soon as possible. Our initial thought was “get it running on a mix of old infrastructure and new and evolve from there over time”. That was a nice thought, but it didn’t work out very well that way.

SproutCore’s event model is based on adding a small number of global event listeners and delegating events to the proper components. Bespin had been using Dojo with the far more common model of attaching events to DOM nodes directly. In order to take advantage of what SproutCore has to offer, we very quickly ended up on the slippery slope of doing other refactorings to the editor, which is something we had not initially planned to do.

I’m thankful that we decided to go ahead with those changes, though! For example, the editor’s event handlers all had code in them to figure out “is the cursor over the scrollbars?” because the editor itself was responsible for painting the scrollbars! The editor had one canvas and it managed everything on there itself. We had discussed the possibility of using the Thunderhead project to manage the different pieces that appear in the editor canvas, but we’re a small team and don’t have time to work on a GUI toolkit in addition to our other ambitions for Bespin.

And so it came to pass that we started down the road of refactoring the editor into separate SproutCore components. Patrick Walton joined Mozilla last month and immediately dove into the task of breaking the scrollbars out into separate components. In the process, Bespin’s editor is now a scrollable SproutCore component, so you could even use the standard scrollbars with the editor. Bespin’s scrollbars are a bit tricky, though… rather than living just outside of the scrolled area, they are actually drawn on top of the scrolled area. We also found that the gutter of the editor really should be a separate component (or maybe even more than one), which is why the horizontal scrollbar is not positioned properly right now. Patrick is actively working on fixing that.

Meanwhile, Joe has been hard at work fixing lots of other things: random uses of Dojo, how our singleton components are managed (since we ultimately want to be able to support multiple editors on a single page), copy and paste issues, etc. I’ve spent a bunch of time working through building, module loading and documentation issues.

From where we sit now, the code is shaping up quite nicely. Soon enough, if you have a SproutCore app, you’ll be able to just drop an editor right into your app. As of today, anyone with a website can grab a copy of Bespin and include the editor component very simply in their pages. We figure that we’ve got another month of work to do before we’re ready to update bespin.mozilla.com. We have a whole bunch of features to port over from the old code and we need to make good on another big part of the Reboot promise: opening Bespin up for plugins. It’s been hard even for Bespin core people to contribute because of the state of the code, and I’m looking forward to getting past that hurdle soon. Before I know it, we’ll be back to shipping new features regularly (and quicker than before!) at bespin.mozilla.com. I can’t wait!

I’ve always thought that it should be possible to design a language that is both usable and performant. JavaScript performance has improved tremendously, but it is still quite a distance behind C and even Java. Of course, programmer productivity is certainly more important than raw performance for most applications. It’s only as you try to scale up services that you really wish you could squeeze out more performance. In fact, being able to squeeze out more performance is one way to avoid the complexity shift of moving from one box to two/many. Of course, there’s always an upperbound to this, but that upperbound can be remarkably high.

I always thought that D looked like the language that could unify high level programming and performance. But D has failed to catch on and seems to be in a state that would make anyone seriously question using the language for real work.

Now, along comes Go, a new language sponsored by Google. More interesting than the Google part, is that Go was created by people with a long history dating back to the origins of Unix (Rob Pike, Ken Thomson, apparently even Brian Kernighan are involved). The language is designed to support super fast compilation (yay for programmer ergonomics!) and C-like performance. Importantly, the language also has built in features to help support scalable and safe concurrent programming, via “channels” for passing data back and forth, rather than something like transactional memory.

Go has a simplified object model that looks cool, but I’d be curious to see how it feels to use in practice. Go doesn’t have exceptions, which seems like a real drag to me. Apparently, exceptions are still considered an open issue and they present thorny issues when you’re dealing with concurrent processes.

Go’s standard library looks to have a decent collection of features, and you can call C code pretty easily. More care needs to be taken if you want C code to call into Go. If Go can get its packaging story together more quickly than Python has, it may indeed start having appeal as a high-level, fast, concurrent programming language.

It’s still very early for Go. It’s not used in production in Google yet. It doesn’t run on Windows. But it has arrived on the scene hitting many points at the “good enough” level (fast compiler, decent library, ok interface to C, decent docs). Google has full-time staff assigned to it, and they’re even hiring a program manager. With speculation that Google doesn’t want to use Python for their high volume services, I wouldn’t be surprised to hear that Google really wants to use Go internally. With an “anchor tenant” like Google, Go has a lot more potential than D ever had.

Now they just need to deal with the pesky issue of the name.

Due to Thanksgiving falling on the same day as the normally scheduled date for a2 <div>’s monthly meeting, we decided to find a different date and possibly join up with another group for a complementary topic. And, indeed, that’s what we’ve done!

a2 <div> and the Google Technology User Group will have a joint meeting on Tuesday, November 17th. The meeting will be at the SRT Solutions office at 7PM. I realize that there are conflicts that day, but there is so much going on during the month that it’s hard to find a date without conflicts!

We have a hot topic: Google Closure. Sure, new JavaScript libraries appear every day, but not like this one. Closure has a maturity that comes from years of use within Google, on some of Google’s busiest properties. It has an amazing toolset (Firebug plugins, super squishy compiler), great docs and a nice collection of features. Of course, not many people outside of Google know much about Closure right now, so this meeting is a group exploration of Closure. Read more in the RSVP form below to see how it works.

I hope to see you there!

For the second a2 <div> meeting, I’ll be talking about CommonJS: JavaScript everywhere. There’s more information on the form below. As with the first meeting, the meeting will be at 7PM on the 4th Thursday of the month at SRT Solutions in downtown Ann Arbor. That’s October 22nd @ 7PM (see the a2div site for more information).

My now former colleagues Ben Galbraith and Dion Almaer have decided to join Palm to help push the mobile web platform forward for developers.

I’ve had a great time working with them on Bespin, a project that they brought to Mozilla. And while I know that they’ve got some really cool challenges in front of them at Palm, I am personally excited about the next phase that Bespin is about to enter as we solidify the core of Bespin and drive it to become a tool that people will use day to day. I’m also happy that, since Bespin is open source, Ben and Dion will still be free to hack on it whenever they wish. In fact, they’ll be working with Joe, John and me on-site at Mozilla HQ next week as we kick off this long-planned-for work on Bespin.

Best wishes and best of luck to you, guys. I’ve really enjoyed working with you and look forward to meeting up as friends when I visit Mountain View.

A few days back, I announced the creation of the a2 <div> group: devoted to learning new tools and techniques for building the best webapps. Meetings will be on the 4th Thursday of each month, so the first meeting is on September 24th at 7PM at the SRT Solutions office in downtown Ann Arbor. Meetings are free, so join us (please let us know that you’re coming using the form below — reason for that in a minute).

The general meeting flow that we’re going to start off with is:

1. A presentation/demo of something awesome (for about an hour with questions interspersed plus maybe a few extra minutes for questions afterward)
2. 5 minute lightning talks (proposed beforehand using the form below)
3. General discussion (either collectively or in groups. Individuals having discussions with themselves is not encouraged.)

This is a rough outline, and I expect a lot of flexing based on the topics du jour.

I’m asking for RSVPs because I’d like to encourage folks to give lightning talks and the form gives a space devoted to that. Lightning talks, for those unfamiliar with the term, are simply 5 minute or shorter talks to introduce people to a topic or to give a quick demo of something new and useful. They can also be a good way to spur new discussion. RSVPs are also good to ensure that a really hot topic doesn’t overwhelm the space we have.

For the first meeting, the main event will be Majd Taby showing off SproutCore. SproutCore, for those who have never seen it, is an amazing open source JavaScript UI toolkit that is highly optimized for speed and reduction of the amount of code you need to write. It borrows liberally from concepts in Apple’s Cocoa framework, and Apple themselves use it for their MobileMe service. If you’re building a “website”, you probably want to use something else. If you’re building something that is an honest-to-goodness app, you owe it to yourself to check out SproutCore.

There aren’t any lightning talks lined up yet, so be sure to toss your name into the hat if you want to give one.  Also, there’s a form online if you want to give a talk at a2div and another if you have a topic you’d like to hear about.

Bespin 0.4.3 (“Chuck Finley“) is live and it includes three new features that are worth mentioning (plus a handful of other changes that I won’t bother with for now).

One is the start of an often-requested feature: deployment from Bespin to other servers. Initially, the only way to get projects out of Bespin was to export the project as a zip file or tarball. Then, we added version control system support, which meant that you could push from Bespin to some repository and update a site from there. Now, Bespin can push directly to a remote server.

I see you

I’ll come back to that feature in a moment. 0.4.3 also includes our first addition to the collaboration feature that was released last month: collaborator cursor tracking.

In the screen shot above, you can see part of the browser windows of two users editing the same file. In the initial collaboration release, you could see everyone editing as they edited, but you couldn’t see who was editing which parts of the file at a given moment in time. With the cursor tracking, you can easily see where each user is editing. This is the first of many planned improvements to collaboration, and we’ll be talking about some of those soon.

Now, back to deployment!

Remote server deployment in 3 easy steps

Right now, you can get to the deployment feature by running the “deploy” command. deploy defaults to deploying the current project. Deployment uses the same security measures used by the version control system support. Specifically, your remote server access credentials are encrypted using a “keychain password” that is not stored anywhere on the Bespin server. When you first run deploy or a vcs command, you will be prompted for your keychain password:

If you’ve never had a keychain password before, no problem! Bespin will save the password the first time you use it. Be careful, though: we have no way to recover this password if it is lost.

This also brings me to another new feature of Bespin “Chuck Finley”: you only have to enter your keychain password once per session. Previously, every time you did something that required keychain access, Bespin would prompt you. Now, you only enter it once and the password is saved in memory in your browser.

The “deploy” command is actually short for “deploy now”. If you’ve never configured deployment before, you’ll be prompted to run “deploy setup” the first time. Here is what the setup screen is like:

Right now, Bespin only supports SFTP connections. Depending on what users need, we will likely add support for other kinds of remote deployment. Adding other kinds of deployment should be easy, thanks to Stavros Korokithakis‘ Omnisync library, which abstracts away the file transfer type. Extra thanks to Stavros for changing the license of Omnisync to be compatible with our desires for Bespin.

“Remote host” and “remote directory” tell Bespin where your files should end up. The remote directory is relative to your home directory on the remote server, unless you start it off with a /.

There are two ways for you to authenticate with the remote server: SSH Key and Password. The SSH Key is the same one used by the VCS commands. It’s a key generated by Bespin, and you can get your public key by running vcs getkey. Paste that key into .ssh/authorized_keys on your remote server, and you’re set!

Using usernames/passwords works just fine as well, and once you’ve entered your username and password you won’t need to do so again because they are stored away in your keychain.

If you want to make sure that everything is configured correctly, you can run the deploy test command. This runs through all of the deployment steps but does not actually copy the files over.

The screenshot above shows what deployment looks like. You’ll notice the messages about the files being identical. That’s a great feature of Omnisync: rather than copying over every file, it only copies over the changed files. I’d also like to see us support rsync which has this same property, but is much faster because it only copies differences across.

Once your deployment is configured and tested, just run “deploy” every time you’re ready to push your files up to the remote server.

I’m curious to hear your feedback on this new feature.

“svn” command

One final feature of note: Bespin has a unique “vcs” command which operates the same regardless of which version control system you use. I think that’s a great feature, particularly when you’re working with projects that are managed by a variety of VCSes. However, there are some times when you want to use the commands you’re familiar with, or if you need to take an action that is specific to a given VCS.

Bespin 0.4.3 adds a fairly complete “svn” command that uses the same command names and parameters as the standard svn command line tool. Some commands and options won’t make sense in the context of Bespin and are therefore not available, but typing “svn command -h” (for example, “svn up -h”) will provide you with help output that is tailored to the options that Bespin supports.

Chuck Finley includes some important fixes in addition to these features. We’ll likely have another minor fixes release next week and we’ve got some big plans brewing.

I’ve been interested in doing cool stuff with web technology for a long time. I still think that web development is harder than it should be, but new tools, processes and ideas come along all the time to make it easier and more fun.

Many people probably view me as a “Python guy”. Sure, I’ve done a good deal of work with Python over the past few years. But, there are many, many good ideas on the server side that come from places other than Python.

And, the browser as a platform is an entirely different thing than it was a few years ago, particularly when you restrict yourself to “modern browsers”, as we do on Bespin. The performance difference between today’s browsers and those from a couple years back is huge, and that new performance opens the door for all kinds of new applications and toolkits to help us build those apps.

I want to learn firsthand from people using modern tools to make development faster and more fun. I haven’t seen a group here in Ann Arbor that is devoted to the broad range of web development topics, so I decided to get one going. It turns out that Majd Taby sent a message to the a2geeks mailing list in April about starting a web dev group, but I somehow missed that. Majd and I exchanged some email this week and worked out the details of the new group.

The new group is called a2 <div> (or just a2div) because it has the same topic focus as cu <div> and I liked the idea of joining forces in some sense with a similarly minded group. cu <div> has a bit of a student focus that a2 <div> does not (it’s been a long time since I was a student). Thanks to Cameron from cu <div> for giving us the go-ahead to start an Ann Arbor offshoot. Thanks also to Dug Song for passing along the link to cu <div> and connecting me with Majd.

a2 <div> will be a loosely organized group as MichiPUG is… bringing the right people together for good discussion is far more important than creating a formal organization. Meetings are free. Our meeting format will likely be something along the lines of 1 hour presentation/demo followed by lightning talks and discussion.

An important aspect of a2 <div> is that’s non-denominational. Are you doing client and server in Java (for example, with GWT)? Neat. How about in Python (with Pyjamas)? Groovy. Doing the server in Lift and the client side with Cappuccino? How about PHP+Dojo? SproutCore+WebObjects? Using ASP.NET MVC with jQuery? Do you use Cucumber to test your apps? All are welcome, all are interesting.

Our meetings will be at the SRT Solutions office in downtown Ann Arbor. This is a great space to meet at (projector, whiteboards, flexible table arrangement), and the location is nice because it’s easy to walk out for drinks and food afterwards. Thanks to Dianne Marsh for letting us use their office!

Meetings will be on the 3rd Wednesday 4th Thursday of each month at 7pm. That means our first meeting will be on September 16th 24th (less than two weeks away). You can subscribe to our calendar in XML or ical formats.

The first meeting topic is not fully decided yet. Please join the a2div googlegroup and let us know what you might want to hear about or talk about!

If you’re near Ann Arbor on September 16th, join us at SRT Solutions for engaging discussion on all webdev-related topics! See you there!

I kicked off the Michigan Python Users Group (MichiPUG) in September 2005, so this month’s meeting marks 4 years since the group began!

This month’s meeting is going to be one of our “topic free” meetings. Despite the lack of a topic, we never have trouble finding Python things to discuss. If you’re going to be around Ann Arbor Thursday evening and have a burning Python question, do stop in!

The meeting will be at 7pm at SRT Solutions in downtown Ann Arbor. Parking is free and easy next to City Hall a couple blocks north (on Ann St.).

This month, I am stepping aside as the de facto leader of the group. While I am still a Python fan and heavy user, my interests have branched out enough that I plan to devote my rather limited “user group time” elsewhere. Stay tuned for more on that soon. Mark Ramm will be taking over my duties as “the guy who sends the monthly ‘what’s our topic?’ email message”.

I’m almost certainly going to be arriving late to tomorrow’s meeting, but I do hope to catch up with folks for drinks afterwards at the very least! See you there!

Python packaging and deployment can be annoying. It’s been nearly 4 years since I released the first TurboGears release as an early adopter of setuptools/easy_install. Since then, there’s been the release of virtualenv, pip and zc.buildout. Somehow, it still seems like more trouble than it should be to get development and production environments set up.

On Bespin, I’ve been using a combination of virtualenv and pip (scripted with Paver) in development and production environments. But, I’ve found pip –freeze to be nearly unusable.

My Ideal World

After monkeying with this stuff a fair bit over the past few years, I have an idea of what I’d really like to have but I don’t think anyone’s working on it. I’d love to hear contrasting opinions or learn about projects that I’m not aware of.

• Multiple version installation into global site-packages, as easy_install currently works (put the active package in the .pth file)
• The better error reporting of pip (pip doesn’t meet my first desire, though, because it installs as single-version-externally-managed)
• A tool to manage the installed packages (uninstall, select a different version)
• In addition to a global site-packages, it would be nice to be able to specify a different site-dir for machines where I don’t have or don’t want to use root access
• virtualenv that behaves like –no-site-packages but knows where site-packages (or the other site-dir) is
• That tool that manages installed packages can selectively install specific versions of packages into the virtualenv by adding pointers in the .pth file that point to the site-packages directory
• You can also install only into the virtualenv if you wish.
• Install packages in that manner from a list of requirements (as with pip’s requirements file)
• A way to freeze the currently set installed into the virtualenv as a new requirements file
• An optional cache of all of the original sdists of the installed packages

pip is close to being usable, except freeze doesn’t work. zc.buildout is close to being usable, too. I think there’s a “freeze” like plugin for it, but I don’t know how well it works. I don’t like zc.buildout quite as much as virtualenv, and I see that people even use virtualenv+zc.buildout to eliminate site-packages from leaking in. I also find that it leaves tons of old packages around in every buildout, again with no way to manage them.

What I’ve found using both zc.buildout and pip is that they are slow and annoying, because they’re constantly reinstalling things that I already have. The main reason for having a shared site-packages as I suggest above is not to save on disk space, but to save on time. In development, I want to be able to update to the latest versions of packages quickly, installing/building only the ones that have changed. How fast something runs changes how you use it, and I know that the scripts that I have for updating development and production environments reflect that.

So,I think the main thing that I’m looking for is a new tool to manage the packages that I have installed globally and within virtualenvs. Are there tools out there that are heading down this path at all?

Also, I understand the starting point that Tarek is taking with Distribute (splitting it up into logical pieces), but is there any roadmap for where it’s going to go functionally from there? Or is the intention purely that tools like the one I’m angling for will be written against the newly refactored libraries? I do know about the uninstall PEP, and that’s pleasing.