Bespin and jQuery

Standard
Yesterday, we announced that the Bespin server is going to be rebuilt from the ground up in node.js with a decentralized model in mind. I’ve seen a lot of positive reaction to that, which is great because I think it’s an important step forward for Bespin. We also mentioned on bespin-core that we have changed from SproutCore, which we adopted during the “reboot” of the Bespin client side code, to jQuery. A handful of people have suggested that we should just build our own “framework”. Our decision to use jQuery came after much deliberation, so I thought I’d go into a little detail about that since this question has now come up many times.

What is Bespin?

It’s hard to say if something is a good choice for Bespin without actually being clear about what Bespin is. Bespin is a customizable programming environment for web applications. There are at least two correct ways to interpret that sentence:
  1. Bespin is a programming environment that you can include in your web applications that you create
  2. Bespin is a programming environment for creating web applications
Early in Bespin’s life, people were embedding the editor in their own applications, but this wasn’t well supported. We made a decision to make that a well-supported use of our project and that change has major ramifications for the project.

Frameworks vs. Libraries

The best description of the difference between a framework and a library that I’ve seen is:
  • A library  is something you call from your code
  • A framework is something that calls your code
Note that this definition says nothing about the size (libraries could certainly be bigger than frameworks, and vice versa). This definition also makes no judgments about which is better or whether the code is loosely coupled or tightly coupled.
Frameworks can make building an application really fast. They do a great job of reducing the amount of code you need to write, as long as your application fits the framework’s design and intended applications well. The farther you stray from the path, the more resistance you’ll get as you try to add new things.
SproutCore is a framework. It attaches event handlers to the page and delegates those events to views that you create. Further, it has a powerful declarative binding/observation system that it updates automatically as part of a runloop that it maintains. It’s really quite a powerful GUI toolkit that runs entirely in your browser.
For Bespin “the application” (the software that runs at bespin.mozillalabs.com), SproutCore seemed like it would be a good fit because what we were doing was more like a desktop app than a website. On the other hand, for Bespin “the customizable, embeddable editor”, the tools we want and need are not such a great fit for a framework. Bespin needs to fit in well with what people are already doing on their pages.

Why use a library at all?

OK, so Bespin and frameworks are not a good match. But there are plenty of libraries out there (jQuery, YUI, Prototype, Mootools, Dojo, Closure, and even MochiKit about which I’ve written extensively in the past) Why not just build our own set of utilities, especially since Bespin exclusively targets modern browsers?
We wrestled a bit with that very question. Even with modern browsers, some conveniences are still handy. Here’s a simple case I can point to: addClass. A function that will add a CSS class to a DOM node. We all need to do that at times, but it’s not yet a method offered by the DOM, nor is it in any proposed standard that I’ve seen.
I believe that every JavaScript library offers addClass. Their APIs vary, but it’s in there somewhere. We could create our own library with our own version of addClass. But that has disadvantages:
  • we have to write it/test it ourselves or copy it from somewhere
  • we have to maintain it
  • for people extending Bespin, there’s that much more surface area that they need to learn
  • we need to write docs for it
  • we can’t use fancier things people create (overlays, trees, etc.) without porting them
  • if people already have a library on their page, our utilities will add additional size
There are probably other disadvantages as well. The only advantages I see to rolling our own are:
  • we get exactly the functionality we need and no more
  • we maintain our own release schedule for it
We are considering a “best of both worlds” approach as well, if we feel like slimming down Bespin Embedded “Drop In” (the prepackaged single .js file) even further: we take just the functionality we’re interested in from a library (jQuery) and create our own mini library that uses the exact same API. Our modules/build tooling can make it transparent to switch between the mini library, the full library and one that is already on the page.

Why jQuery?

As I mentioned earlier, when it comes to utilities the libraries all offer a very similar collection of functions. We don’t want to use a library that “leaks” onto the page, either through the creation of globals or the extension of built-in prototypes. That still leaves a number of choices. We evaluated some, but ultimately decided on jQuery because:
  • It’s very popular, which increases the chance that it’s already on the page (and decreases the size of Bespin)
  • many people are already familiar with it
  • there’s lots of good documentation available
  • there are many decent open source plugins that people can use when extending Bespin
It’s worth noting that these are not technical reasons to use jQuery, and these largely only matter because jQuery is popular and Bespin is intended to be used on other peoples’ pages. The choice of jQuery for Bespin is not a technical choice. We certainly like jQuery, but we like the other libraries as well.

Not just jQuery

One more important point: Bespin Embedded is still designed to be easily built into a self-contained .js file. The current code in our repository puts only one name on the page (bespin) and everything else is hidden away conveniently inside of there. The next release of Bespin Embedded will be less than half the size of the previous release, even with jQuery bundled in. So, Bespin will be a good addition to your application, regardless of which JavaScript library you might use for the rest of your code.

Summary

Bespin has had unusual and competing requirements along the way and as we’ve zeroed in on exactly what Bespin is and will become, we’ve changed our infrastructure to match the needs of the project. We’re really excited about how far we’ve come since “rebooting” the project and are looking forward to these next few releases that lead up to a “1.0” of the Bespin project.

Additional Commentary

John-David Dalton commented via a gist.

He points out that unless we are using a compatible jQuery, people who already have jQuery on the page will not have any size advantage. Of course, if we have our own library no one will gain a size advantage.

With respect to Bespin users being able to use whatever library they wish without porting additional widgets, we (the Bespin team) want to be able to use things like trees and such without having to port them to our own library. Bespin users do remain free to use whatever they wish without porting anything.

Finally, regarding “fourth time’s a charm”, I’ll just note that the 0.8 version number of the next release of Bespin is actually significant. The client APIs are settling.

Thanks for the comment, jdalton!

ClassList API in HTML5

I was happy to hear from Thomas Bassetto that addClass has indeed been rendered redundant by HTML5’s ClassList API which looks great. element.classList is available today in Firefox 3.6 and, according to the Mozilla Hacks article, WebKit is planning to also support the API. The current version of Chrome does not have it yet.

Rebooting Bespin: Embedded and SproutCore

Standard

This post is a collection of random thoughts about the Bespin Reboot project so far.

The past summer, the Bespin team at Mozilla (Ben, Dion, Joe and me) started thinking that the time had come to make some significant changes to the Bespin project code. These changes would make it easier for others to use and contribute to Bespin and also make it easier for Bespin core contributors to add new features. We did a collection of releases during the summer and then slammed on the brakes for what we’ve called the “Reboot“.

The Reboot involves a serious refactoring of the code, plus changes in priorities and processes. We started fresh with a new source code repository (two, actually, since the Python server has been pushed into a separate repo). Then, we poured the code from the old repository into a new structure of CommonJS modules. We changed our underlying GUI framework to use SproutCore, so that we could spend less time thinking about getting things on the screen and more time with features that help people write their code.

Once you get into the groove of doing releases and adding good stuff for your users, it’s really painful to stop doing releases. I’ve been itching to ship since the beginning of October when we shifted full-time to the Reboot repository. Our goal all along has been to get the editor running again as soon as possible. Our initial thought was “get it running on a mix of old infrastructure and new and evolve from there over time”. That was a nice thought, but it didn’t work out very well that way.

SproutCore’s event model is based on adding a small number of global event listeners and delegating events to the proper components. Bespin had been using Dojo with the far more common model of attaching events to DOM nodes directly. In order to take advantage of what SproutCore has to offer, we very quickly ended up on the slippery slope of doing other refactorings to the editor, which is something we had not initially planned to do.

I’m thankful that we decided to go ahead with those changes, though! For example, the editor’s event handlers all had code in them to figure out “is the cursor over the scrollbars?” because the editor itself was responsible for painting the scrollbars! The editor had one canvas and it managed everything on there itself. We had discussed the possibility of using the Thunderhead project to manage the different pieces that appear in the editor canvas, but we’re a small team and don’t have time to work on a GUI toolkit in addition to our other ambitions for Bespin.

And so it came to pass that we started down the road of refactoring the editor into separate SproutCore components. Patrick Walton joined Mozilla last month and immediately dove into the task of breaking the scrollbars out into separate components. In the process, Bespin’s editor is now a scrollable SproutCore component, so you could even use the standard scrollbars with the editor. Bespin’s scrollbars are a bit tricky, though… rather than living just outside of the scrolled area, they are actually drawn on top of the scrolled area. We also found that the gutter of the editor really should be a separate component (or maybe even more than one), which is why the horizontal scrollbar is not positioned properly right now. Patrick is actively working on fixing that.

Meanwhile, Joe has been hard at work fixing lots of other things: random uses of Dojo, how our singleton components are managed (since we ultimately want to be able to support multiple editors on a single page), copy and paste issues, etc. I’ve spent a bunch of time working through building, module loading and documentation issues.

From where we sit now, the code is shaping up quite nicely. Soon enough, if you have a SproutCore app, you’ll be able to just drop an editor right into your app. As of today, anyone with a website can grab a copy of Bespin and include the editor component very simply in their pages. We figure that we’ve got another month of work to do before we’re ready to update bespin.mozilla.com. We have a whole bunch of features to port over from the old code and we need to make good on another big part of the Reboot promise: opening Bespin up for plugins. It’s been hard even for Bespin core people to contribute because of the state of the code, and I’m looking forward to getting past that hurdle soon. Before I know it, we’ll be back to shipping new features regularly (and quicker than before!) at bespin.mozilla.com. I can’t wait!

Good luck, Ben and Dion

Standard

My now former colleagues Ben Galbraith and Dion Almaer have decided to join Palm to help push the mobile web platform forward for developers.

I’ve had a great time working with them on Bespin, a project that they brought to Mozilla. And while I know that they’ve got some really cool challenges in front of them at Palm, I am personally excited about the next phase that Bespin is about to enter as we solidify the core of Bespin and drive it to become a tool that people will use day to day. I’m also happy that, since Bespin is open source, Ben and Dion will still be free to hack on it whenever they wish. In fact, they’ll be working with Joe, John and me on-site at Mozilla HQ next week as we kick off this long-planned-for work on Bespin.

Best wishes and best of luck to you, guys. I’ve really enjoyed working with you and look forward to meeting up as friends when I visit Mountain View.

Bespin 0.4.3: “deploy” to your website!

Standard

Bespin 0.4.3 (“Chuck Finley“) is live and it includes three new features that are worth mentioning (plus a handful of other changes that I won’t bother with for now).

One is the start of an often-requested feature: deployment from Bespin to other servers. Initially, the only way to get projects out of Bespin was to export the project as a zip file or tarball. Then, we added version control system support, which meant that you could push from Bespin to some repository and update a site from there. Now, Bespin can push directly to a remote server.

I see you

I’ll come back to that feature in a moment. 0.4.3 also includes our first addition to the collaboration feature that was released last month: collaborator cursor tracking.

Cursors

In the screen shot above, you can see part of the browser windows of two users editing the same file. In the initial collaboration release, you could see everyone editing as they edited, but you couldn’t see who was editing which parts of the file at a given moment in time. With the cursor tracking, you can easily see where each user is editing. This is the first of many planned improvements to collaboration, and we’ll be talking about some of those soon.

Now, back to deployment!

Remote server deployment in 3 easy steps

Right now, you can get to the deployment feature by running the “deploy” command. deploy defaults to deploying the current project. Deployment uses the same security measures used by the version control system support. Specifically, your remote server access credentials are encrypted using a “keychain password” that is not stored anywhere on the Bespin server. When you first run deploy or a vcs command, you will be prompted for your keychain password:

Deployment prompts for keychain password

If you’ve never had a keychain password before, no problem! Bespin will save the password the first time you use it. Be careful, though: we have no way to recover this password if it is lost.

This also brings me to another new feature of Bespin “Chuck Finley”: you only have to enter your keychain password once per session. Previously, every time you did something that required keychain access, Bespin would prompt you. Now, you only enter it once and the password is saved in memory in your browser.

The “deploy” command is actually short for “deploy now”. If you’ve never configured deployment before, you’ll be prompted to run “deploy setup” the first time. Here is what the setup screen is like:

deploy setup page

Right now, Bespin only supports SFTP connections. Depending on what users need, we will likely add support for other kinds of remote deployment. Adding other kinds of deployment should be easy, thanks to Stavros KorokithakisOmnisync library, which abstracts away the file transfer type. Extra thanks to Stavros for changing the license of Omnisync to be compatible with our desires for Bespin.

“Remote host” and “remote directory” tell Bespin where your files should end up. The remote directory is relative to your home directory on the remote server, unless you start it off with a /.

There are two ways for you to authenticate with the remote server: SSH Key and Password. The SSH Key is the same one used by the VCS commands. It’s a key generated by Bespin, and you can get your public key by running vcs getkey. Paste that key into .ssh/authorized_keys on your remote server, and you’re set!

Using usernames/passwords works just fine as well, and once you’ve entered your username and password you won’t need to do so again because they are stored away in your keychain.

If you want to make sure that everything is configured correctly, you can run the deploy test command. This runs through all of the deployment steps but does not actually copy the files over.

Deployment test

The screenshot above shows what deployment looks like. You’ll notice the messages about the files being identical. That’s a great feature of Omnisync: rather than copying over every file, it only copies over the changed files. I’d also like to see us support rsync which has this same property, but is much faster because it only copies differences across.

Once your deployment is configured and tested, just run “deploy” every time you’re ready to push your files up to the remote server.

I’m curious to hear your feedback on this new feature.

“svn” command

One final feature of note: Bespin has a unique “vcs” command which operates the same regardless of which version control system you use. I think that’s a great feature, particularly when you’re working with projects that are managed by a variety of VCSes. However, there are some times when you want to use the commands you’re familiar with, or if you need to take an action that is specific to a given VCS.

Bespin 0.4.3 adds a fairly complete “svn” command that uses the same command names and parameters as the standard svn command line tool. Some commands and options won’t make sense in the context of Bespin and are therefore not available, but typing “svn command -h” (for example, “svn up -h”) will provide you with help output that is tailored to the options that Bespin supports.

Chuck Finley includes some important fixes in addition to these features. We’ll likely have another minor fixes release next week and we’ve got some big plans brewing.

Project Bespin Summary for August 28, 2009

Standard

“This Week in Bespin” returns as “Project Bespin Summary”.  It was clear to me some time back that weekly was not the right pace for delivering condensed Bespin news. But there is also interesting backstory and plenty that happens in between major releases, so we wouldn’t want major releases to be the only time for people to learn about what’s going on with the project.

The last major release, Bespin 0.4, got plenty of attention around the web and I posted an announcement message on the Bespin list after the release. So, here’s a quick round up of what’s been happening since the 0.4 release:

  • We released Bespin 0.4.1 on August 17th which corrected some issues spotted after the release of 0.4 (15 bugs closed). This release added a minor feature: you can set your username to be used when doing commits in a distributed version control system.
  • On August 27th, we released Bespin 0.4.2 which thus far appears to have fixed some locking and stability issues we’ve had. Ben blogged about the Bespin 0.4.2 release.
  • Ben posted an updated short-term roadmap for Bespin on his blog. You’ll note that release names have now changed from a cloud theme to some other theme that I will leave as an exercise for the reader.
  • Before you know it, Bespin 0.4.3 will be out. The ability to run familiar “svn” commands is already available on the Bespin hg tip for a few commands.
  • Git support for Bespin is very much in demand. You can help!
  • We’re hiring! Specifically, we’re looking for someone that will (at least initially) be responsible for the Canvas-based editor at the heart of Bespin. I have a post up on my blog as well.

The next Project Bespin Summary is likely to come out soon after the 0.4.3 release.

Project Bespin needs *you*

Standard

I work in Mozilla’s Developer Tools Lab, where we’re working to make things easier for web developers and to explore what is possible using open web tools. It turns out there’s quite a bit that’s possible, and we’ve just scratched the surface with what we’ve done with Bespin so far.

I’m really happy with the team that I’m working with, both within Mozilla Labs and in the Bespin community. I’m also really happy to report that we’ve got an opening on our team! We’re looking for fantastic software engineer that will own the Canvas-based editor component that is at the heart of Bespin.

If being at the forefront of JavaScript technology (Canvas, local storage, web workers) sounds fun and exciting to you, drop us a line! We’d love to talk to you!

One Python-based version control system to rule them all!

Standard

We’ve just released Bespin 0.4, the major new feature of which is the first bit of the collaboration feature. Bespin 0.4 includes a ton of other changes, including one that I’m going to focus on here: Subversion support, which Gordon P. Hemsley kicked off for us a few weeks back.

Bespin’s initial version control support showed up in 0.2 with support for Mercurial. Knowing that we wanted to support multiple version control systems (VCS), I took an unorthodox approach from the beginning. Rather than providing the “hg” commands that people know and love, I created a separate set of “vcs” commands. Ultimately, we want to make it easy to grab a random open source project of the net and start hacking on it. Using the “vcs” commands, for the most common version control operations you won’t even have to think about which VCS is used by a given project.

I can run “vcs clone” (“vcs checkout” also works) to check out Bespin (in a Mercurial repository), Paver (in a Subversion repo) and hopefully soon Narwhal (in a Git repo). Also new in Bespin 0.4: Bespin’s command line has been tricked out to be able to have fancier interactions with commands, so you can enter all of the extra information that Bespin needs for checking out a repository right in the output area.

If you’ve used Subversion and one of the Distributed VCSes, you’ll know that they have a different model. The DVCSes do almost everything in a local repository copy and only talk to a remote server for push/pull. That’s actually true of Subversion as well, with one notable exception: commit. For Subversion, the “vcs commit” command will simply save your commit message for later. When you run “vcs push”, that is when an actual “svn commit” operation is run.

What’s neat about the “vcs” commands is that they operate the same from VCS to VCS. svn doesn’t have a feature to “add all files that are unknown”, whereas Mercurial does. “vcs add -a” operates the same on both systems.

If you’re interested, you can also use these commands on the command line by installing the Ãœber Version Controller (uvc) Python package. After doing so, you can head into a random Subversion or Mercurial working copy and type “uvc status” to see what’s different. I will note that the command line tool has been, um, lightly tested since uvc is mostly used as a library for Bespin at this point.

One final note: Bespin will soon also support native “svn” and “hg” commands so that you can stick to commands and options you’re familiar with or for performing more complex operations that don’t have equivalent “vcs” commands.

You can learn more about version control in Bespin from this section of the User Guide.

This Week in Bespin March 2, 2009

Standard

For the week of February 23rd:

  • Roberto Saccon ported Bespin to Dojo! Dojo leader Peter Higgins helped to clean things up a bit. This has significantly changed (and improved) the use of namespaces in the frontend code, which will make Bespin much easier to drop into other pages.
  • For people keeping score at home, there is now a 0.1 branch in the Bespin repository that we will be using to update the server. The default branch has the Dojo work and all of the other forward-looking work going on. Until the trunk work stabilizes a bit, bespin.mozilla.com will be running a bit behind on the changes here.
  • autoindent setting. When you hit return/enter you’re placed at the same indentation point on the next line.
  • Select an area and hit TAB and it all indents. SHIFT+TAB and it unindents. Thanks to Jay Bird!
  • Special support for certain keys so German keyboards etc can get in ‘{‘ and the like. Thanks to Julian Viereck  
  • For people following the trunk/tip, the project build script has changed a bit. Be sure to follow the instructions in the top-level README file to get the Bespin server running.
  • PHP, Java and Aduino syntax highlighting, thanks to Sean Burke and Olle Jonsson, respectively. The Arduino support is optional. Use include(“bespin.syntax.arduino”); in your config.js to load.
  • Multithreaded development server, which speeds up page loads a bit now that there are more modules loading.
  • Config work has short cuts now so you can simply use: include(file), execute(cmd), publish(), subscribe() 
Coming up the week of March 2nd:
  • I’ll be working on backend changes to make VCS integration possible (and to allow us to start using Bespin as our primary editor when working on Bespin itself)
  • Joe will be working on collaboration implementation
  • Ben and Dion will likely have their hands full just keeping up with your patches! (I’m sure they will sneak some time in to improve the editor and Thunderhead as well :)

By the way, “This Week in Bespin” is not just for those of us at Mozilla. If you have anything you want to see show up in these summaries, just send me an email message!

Introduction to the Bespin Python backend

Standard

It’s been less than two weeks since Bespin was introduced, and there’s already been an impressive amount of activity around the open source project. There are at least 3 entirely new Bespin servers that I’m aware of.

The current Bespin server that we at Mozilla are maintaining is written in Python and appears in the backend/python directory in the Bespin source. To help people get up to speed with the code, I have created a screencast to help people get started with the Python backend and give them an idea of how the code is set up.

I look forward to hearing your feedback!


Bespin Python Backend Overview from Kevin Dangoor on Vimeo.

This Week in Bespin (Feb 23)

Standard

I’m crossposting this from the Bespin googlegroup.

Before moving on to the week that is coming, let’s look back at the week that was:

Week of February 17th:

  • Version 0.1.2 of Bespin was released. Notable changes:
  • System clipboard copy and paste now works in WebKit. Firefox will almost work. Vote up this bug:   https://bugzilla.mozilla.org/show_bug.cgi?id=407983 
  • Project names now need only to be unique per user. Previously, they had to be globally unique. This reflects a bit of a change to how we were viewing collaboration. (More on collaboration to come).
  • Thanks to that change, the sample project has been renamed from SampleProjectFor:yourusername to just SampleProject.
  • Speaking of project renaming, you can now rename projects from the dashboard or the editor using the “renameproject” command. New “createproject” and “deleteproject” commands make project-level changes much clearer.
  • Every user now has a visible “BespinSettings” project. You probably      won’t be surprised to hear that it stores settings for Bespin! You can put custom code (set autoconfig on and put code in config.js) and commands (use the cmd* commands) in there.
  • The login form is now a real form, which means that your password manager can save your Bespin login information.
  • Client and server implementations can make sure they’re in sync with the X-Bespin-API HTTP header that is returned from the server. The current API version is 2 (which adds the renameproject API).
  • key repetition works for arrows and backspace and lots of other minor fixes.
  • As Dion noted, Boris Bokowski and Simon Kaegi coupled the Bespin Editor with a headless Eclipse . The screenshot showing Java compiler warnings within Bespin is worth at least a couple thousand words!
  • Jerome Velociter got the Bespin editor running in XWiki . Avi Bryant hooked the dashboard and editor up to Squeak :. It’s been amazing to see the integration going on so far.
  • Oscar Carballa kicked off the creation of a User Guide for Bespin
  • Mozilla’s Joe Walker started working on the collaboration spec. The model that he’s looking at is designed to allow you to share a project with many people and give them access to view it/collaborate on it in different ways. So, you’ll be able to selectively use collaborative editing with some people, do a “webcast” style sharing with others and more.
  • Three more google groups have been announced: bespin-core covers the development of Bespin’s code, bespin-dev is for people who want to develop extensions for Bespin. Community member Guilherme Santos created the bespinbrasil group for Brazilian Bespin users.
  • Coming up this week:

    • Ben and Dion are in Miami for Future of Web Apps where they will be showing off Bespin.
    • Joe is continue to work on collaboration and is experimenting with Neil Fraser’s mobwrite.
    • I will finally be getting to work on the version control spec, and hope to be starting on the version control backend code.
    • And we’re all going to be working on bug fixes and minor enhancements.