Blue Sky On Mars

This post is a collection of random thoughts about the Bespin Reboot project so far.

The past summer, the Bespin team at Mozilla (Ben, Dion, Joe and me) started thinking that the time had come to make some significant changes to the Bespin project code. These changes would make it easier for others to use and contribute to Bespin and also make it easier for Bespin core contributors to add new features. We did a collection of releases during the summer and then slammed on the brakes for what we’ve called the “Reboot“.

The Reboot involves a serious refactoring of the code, plus changes in priorities and processes. We started fresh with a new source code repository (two, actually, since the Python server has been pushed into a separate repo). Then, we poured the code from the old repository into a new structure of CommonJS modules. We changed our underlying GUI framework to use SproutCore, so that we could spend less time thinking about getting things on the screen and more time with features that help people write their code.

Once you get into the groove of doing releases and adding good stuff for your users, it’s really painful to stop doing releases. I’ve been itching to ship since the beginning of October when we shifted full-time to the Reboot repository. Our goal all along has been to get the editor running again as soon as possible. Our initial thought was “get it running on a mix of old infrastructure and new and evolve from there over time”. That was a nice thought, but it didn’t work out very well that way.

SproutCore’s event model is based on adding a small number of global event listeners and delegating events to the proper components. Bespin had been using Dojo with the far more common model of attaching events to DOM nodes directly. In order to take advantage of what SproutCore has to offer, we very quickly ended up on the slippery slope of doing other refactorings to the editor, which is something we had not initially planned to do.

I’m thankful that we decided to go ahead with those changes, though! For example, the editor’s event handlers all had code in them to figure out “is the cursor over the scrollbars?” because the editor itself was responsible for painting the scrollbars! The editor had one canvas and it managed everything on there itself. We had discussed the possibility of using the Thunderhead project to manage the different pieces that appear in the editor canvas, but we’re a small team and don’t have time to work on a GUI toolkit in addition to our other ambitions for Bespin.

And so it came to pass that we started down the road of refactoring the editor into separate SproutCore components. Patrick Walton joined Mozilla last month and immediately dove into the task of breaking the scrollbars out into separate components. In the process, Bespin’s editor is now a scrollable SproutCore component, so you could even use the standard scrollbars with the editor. Bespin’s scrollbars are a bit tricky, though… rather than living just outside of the scrolled area, they are actually drawn on top of the scrolled area. We also found that the gutter of the editor really should be a separate component (or maybe even more than one), which is why the horizontal scrollbar is not positioned properly right now. Patrick is actively working on fixing that.

Meanwhile, Joe has been hard at work fixing lots of other things: random uses of Dojo, how our singleton components are managed (since we ultimately want to be able to support multiple editors on a single page), copy and paste issues, etc. I’ve spent a bunch of time working through building, module loading and documentation issues.

From where we sit now, the code is shaping up quite nicely. Soon enough, if you have a SproutCore app, you’ll be able to just drop an editor right into your app. As of today, anyone with a website can grab a copy of Bespin and include the editor component very simply in their pages. We figure that we’ve got another month of work to do before we’re ready to update bespin.mozilla.com. We have a whole bunch of features to port over from the old code and we need to make good on another big part of the Reboot promise: opening Bespin up for plugins. It’s been hard even for Bespin core people to contribute because of the state of the code, and I’m looking forward to getting past that hurdle soon. Before I know it, we’ll be back to shipping new features regularly (and quicker than before!) at bespin.mozilla.com. I can’t wait!

My now former colleagues Ben Galbraith and Dion Almaer have decided to join Palm to help push the mobile web platform forward for developers.

I’ve had a great time working with them on Bespin, a project that they brought to Mozilla. And while I know that they’ve got some really cool challenges in front of them at Palm, I am personally excited about the next phase that Bespin is about to enter as we solidify the core of Bespin and drive it to become a tool that people will use day to day. I’m also happy that, since Bespin is open source, Ben and Dion will still be free to hack on it whenever they wish. In fact, they’ll be working with Joe, John and me on-site at Mozilla HQ next week as we kick off this long-planned-for work on Bespin.

Best wishes and best of luck to you, guys. I’ve really enjoyed working with you and look forward to meeting up as friends when I visit Mountain View.

Bespin 0.4.3 (“Chuck Finley“) is live and it includes three new features that are worth mentioning (plus a handful of other changes that I won’t bother with for now).

One is the start of an often-requested feature: deployment from Bespin to other servers. Initially, the only way to get projects out of Bespin was to export the project as a zip file or tarball. Then, we added version control system support, which meant that you could push from Bespin to some repository and update a site from there. Now, Bespin can push directly to a remote server.

I see you

I’ll come back to that feature in a moment. 0.4.3 also includes our first addition to the collaboration feature that was released last month: collaborator cursor tracking.

In the screen shot above, you can see part of the browser windows of two users editing the same file. In the initial collaboration release, you could see everyone editing as they edited, but you couldn’t see who was editing which parts of the file at a given moment in time. With the cursor tracking, you can easily see where each user is editing. This is the first of many planned improvements to collaboration, and we’ll be talking about some of those soon.

Now, back to deployment!

Remote server deployment in 3 easy steps

Right now, you can get to the deployment feature by running the “deploy” command. deploy defaults to deploying the current project. Deployment uses the same security measures used by the version control system support. Specifically, your remote server access credentials are encrypted using a “keychain password” that is not stored anywhere on the Bespin server. When you first run deploy or a vcs command, you will be prompted for your keychain password:

If you’ve never had a keychain password before, no problem! Bespin will save the password the first time you use it. Be careful, though: we have no way to recover this password if it is lost.

This also brings me to another new feature of Bespin “Chuck Finley”: you only have to enter your keychain password once per session. Previously, every time you did something that required keychain access, Bespin would prompt you. Now, you only enter it once and the password is saved in memory in your browser.

The “deploy” command is actually short for “deploy now”. If you’ve never configured deployment before, you’ll be prompted to run “deploy setup” the first time. Here is what the setup screen is like:

Right now, Bespin only supports SFTP connections. Depending on what users need, we will likely add support for other kinds of remote deployment. Adding other kinds of deployment should be easy, thanks to Stavros Korokithakis‘ Omnisync library, which abstracts away the file transfer type. Extra thanks to Stavros for changing the license of Omnisync to be compatible with our desires for Bespin.

“Remote host” and “remote directory” tell Bespin where your files should end up. The remote directory is relative to your home directory on the remote server, unless you start it off with a /.

There are two ways for you to authenticate with the remote server: SSH Key and Password. The SSH Key is the same one used by the VCS commands. It’s a key generated by Bespin, and you can get your public key by running vcs getkey. Paste that key into .ssh/authorized_keys on your remote server, and you’re set!

Using usernames/passwords works just fine as well, and once you’ve entered your username and password you won’t need to do so again because they are stored away in your keychain.

If you want to make sure that everything is configured correctly, you can run the deploy test command. This runs through all of the deployment steps but does not actually copy the files over.

The screenshot above shows what deployment looks like. You’ll notice the messages about the files being identical. That’s a great feature of Omnisync: rather than copying over every file, it only copies over the changed files. I’d also like to see us support rsync which has this same property, but is much faster because it only copies differences across.

Once your deployment is configured and tested, just run “deploy” every time you’re ready to push your files up to the remote server.

I’m curious to hear your feedback on this new feature.

“svn” command

One final feature of note: Bespin has a unique “vcs” command which operates the same regardless of which version control system you use. I think that’s a great feature, particularly when you’re working with projects that are managed by a variety of VCSes. However, there are some times when you want to use the commands you’re familiar with, or if you need to take an action that is specific to a given VCS.

Bespin 0.4.3 adds a fairly complete “svn” command that uses the same command names and parameters as the standard svn command line tool. Some commands and options won’t make sense in the context of Bespin and are therefore not available, but typing “svn command -h” (for example, “svn up -h”) will provide you with help output that is tailored to the options that Bespin supports.

Chuck Finley includes some important fixes in addition to these features. We’ll likely have another minor fixes release next week and we’ve got some big plans brewing.

“This Week in Bespin” returns as “Project Bespin Summary”.  It was clear to me some time back that weekly was not the right pace for delivering condensed Bespin news. But there is also interesting backstory and plenty that happens in between major releases, so we wouldn’t want major releases to be the only time for people to learn about what’s going on with the project.

The last major release, Bespin 0.4, got plenty of attention around the web and I posted an announcement message on the Bespin list after the release. So, here’s a quick round up of what’s been happening since the 0.4 release:

• We released Bespin 0.4.1 on August 17th which corrected some issues spotted after the release of 0.4 (15 bugs closed). This release added a minor feature: you can set your username to be used when doing commits in a distributed version control system.
• On August 27th, we released Bespin 0.4.2 which thus far appears to have fixed some locking and stability issues we’ve had. Ben blogged about the Bespin 0.4.2 release.
• Ben posted an updated short-term roadmap for Bespin on his blog. You’ll note that release names have now changed from a cloud theme to some other theme that I will leave as an exercise for the reader.
• Before you know it, Bespin 0.4.3 will be out. The ability to run familiar “svn” commands is already available on the Bespin hg tip for a few commands.
• Git support for Bespin is very much in demand. You can help!
• We’re hiring! Specifically, we’re looking for someone that will (at least initially) be responsible for the Canvas-based editor at the heart of Bespin. I have a post up on my blog as well.

The next Project Bespin Summary is likely to come out soon after the 0.4.3 release.

I work in Mozilla’s Developer Tools Lab, where we’re working to make things easier for web developers and to explore what is possible using open web tools. It turns out there’s quite a bit that’s possible, and we’ve just scratched the surface with what we’ve done with Bespin so far.

I’m really happy with the team that I’m working with, both within Mozilla Labs and in the Bespin community. I’m also really happy to report that we’ve got an opening on our team! We’re looking for fantastic software engineer that will own the Canvas-based editor component that is at the heart of Bespin.

If being at the forefront of JavaScript technology (Canvas, local storage, web workers) sounds fun and exciting to you, drop us a line! We’d love to talk to you!

We’ve just released Bespin 0.4, the major new feature of which is the first bit of the collaboration feature. Bespin 0.4 includes a ton of other changes, including one that I’m going to focus on here: Subversion support, which Gordon P. Hemsley kicked off for us a few weeks back.

Bespin’s initial version control support showed up in 0.2 with support for Mercurial. Knowing that we wanted to support multiple version control systems (VCS), I took an unorthodox approach from the beginning. Rather than providing the “hg” commands that people know and love, I created a separate set of “vcs” commands. Ultimately, we want to make it easy to grab a random open source project of the net and start hacking on it. Using the “vcs” commands, for the most common version control operations you won’t even have to think about which VCS is used by a given project.

I can run “vcs clone” (“vcs checkout” also works) to check out Bespin (in a Mercurial repository), Paver (in a Subversion repo) and hopefully soon Narwhal (in a Git repo). Also new in Bespin 0.4: Bespin’s command line has been tricked out to be able to have fancier interactions with commands, so you can enter all of the extra information that Bespin needs for checking out a repository right in the output area.

If you’ve used Subversion and one of the Distributed VCSes, you’ll know that they have a different model. The DVCSes do almost everything in a local repository copy and only talk to a remote server for push/pull. That’s actually true of Subversion as well, with one notable exception: commit. For Subversion, the “vcs commit” command will simply save your commit message for later. When you run “vcs push”, that is when an actual “svn commit” operation is run.

What’s neat about the “vcs” commands is that they operate the same from VCS to VCS. svn doesn’t have a feature to “add all files that are unknown”, whereas Mercurial does. “vcs add -a” operates the same on both systems.

If you’re interested, you can also use these commands on the command line by installing the Über Version Controller (uvc) Python package. After doing so, you can head into a random Subversion or Mercurial working copy and type “uvc status” to see what’s different. I will note that the command line tool has been, um, lightly tested since uvc is mostly used as a library for Bespin at this point.

One final note: Bespin will soon also support native “svn” and “hg” commands so that you can stick to commands and options you’re familiar with or for performing more complex operations that don’t have equivalent “vcs” commands.

You can learn more about version control in Bespin from this section of the User Guide.

For the week of February 23rd:

• Roberto Saccon ported Bespin to Dojo! Dojo leader Peter Higgins helped to clean things up a bit. This has significantly changed (and improved) the use of namespaces in the frontend code, which will make Bespin much easier to drop into other pages.
• For people keeping score at home, there is now a 0.1 branch in the Bespin repository that we will be using to update the server. The default branch has the Dojo work and all of the other forward-looking work going on. Until the trunk work stabilizes a bit, bespin.mozilla.com will be running a bit behind on the changes here.
• autoindent setting. When you hit return/enter you’re placed at the same indentation point on the next line.
• Select an area and hit TAB and it all indents. SHIFT+TAB and it unindents. Thanks to Jay Bird!
• Special support for certain keys so German keyboards etc can get in ‘{‘ and the like. Thanks to Julian Viereck  
• For people following the trunk/tip, the project build script has changed a bit. Be sure to follow the instructions in the top-level README file to get the Bespin server running.
• PHP, Java and Aduino syntax highlighting, thanks to Sean Burke and Olle Jonsson, respectively. The Arduino support is optional. Use include(“bespin.syntax.arduino”); in your config.js to load.
• Multithreaded development server, which speeds up page loads a bit now that there are more modules loading.
• Config work has short cuts now so you can simply use: include(file), execute(cmd), publish(), subscribe() 

It’s been less than two weeks since Bespin was introduced, and there’s already been an impressive amount of activity around the open source project. There are at least 3 entirely new Bespin servers that I’m aware of.

The current Bespin server that we at Mozilla are maintaining is written in Python and appears in the backend/python directory in the Bespin source. To help people get up to speed with the code, I have created a screencast to help people get started with the Python backend and give them an idea of how the code is set up.

I look forward to hearing your feedback!

Bespin Python Backend Overview from Kevin Dangoor on Vimeo.

I’m crossposting this from the Bespin googlegroup.

Before moving on to the week that is coming, let’s look back at the week that was:

Week of February 17th:

• Version 0.1.2 of Bespin was released. Notable changes:

• System clipboard copy and paste now works in WebKit. Firefox will almost work. Vote up this bug:   https://bugzilla.mozilla.org/show_bug.cgi?id=407983 
• Project names now need only to be unique per user. Previously, they had to be globally unique. This reflects a bit of a change to how we were viewing collaboration. (More on collaboration to come).
• Thanks to that change, the sample project has been renamed from SampleProjectFor:yourusername to just SampleProject.
• Speaking of project renaming, you can now rename projects from the dashboard or the editor using the “renameproject” command. New ”createproject” and “deleteproject” commands make project-level changes much clearer.
• Every user now has a visible “BespinSettings” project. You probably      won’t be surprised to hear that it stores settings for Bespin! You can put custom code (set autoconfig on and put code in config.js) and commands (use the cmd* commands) in there.
• The login form is now a real form, which means that your password manager can save your Bespin login information.
• Client and server implementations can make sure they’re in sync with the X-Bespin-API HTTP header that is returned from the server. The current API version is 2 (which adds the renameproject API).
• key repetition works for arrows and backspace and lots of other minor fixes.

Coming up this week:

• Ben and Dion are in Miami for Future of Web Apps where they will be showing off Bespin.
• Joe is continue to work on collaboration and is experimenting with Neil Fraser’s mobwrite.
• I will finally be getting to work on the version control spec, and hope to be starting on the version control backend code.
• And we’re all going to be working on bug fixes and minor enhancements.