Blue Sky On Mars

Googling “embarrassment driven development” (EDD) does not return as many hits as it should. I think it’s a very powerful development technique. I first heard the expression from the Plone guys at PyCon 2006, and Google did turn up this match:

[ArchipelagoSprint] Time to get cracking on Plone 3.0!

Wrt. timelines, I was hoping that we could try to have a “Tech preview” release before the Plone Conference 2006 in Seattle (October 25-27) – I’m going to be on-stage there talking about the exciting new features of Plone 3.0 – and I’d like to not be booed off stage. Yes, this is embarrassment-driven development – as usual.

That’s Alexander Limi illustrating the prime motivator for EDD.

The idea behind EDD is simple: if you have to demo something in front of an audience, and that something sucks, you will move hell or high water to make sure you don’t look like an idiot.

Every product has rough edges and warts, but no one wants a demo to be all warty and to have to say “yeah, I know you shouldn’t have to click to the left of the button, but we just haven’t gotten to that yet”. EDD ensures that, at least for the parts you have to get up and show, the rough edges will be smoothed in time for the show.

I’m going to be practicing EDD leading up to JSConf. I want to be able to show some useful, non-trivial bits of ServerJS work by then.

If you haven’t seen it yet, you really should watch the demo of 280Atlas. It’s basically Interface Builder written for the browser. Just watch their demo and tell me that that doesn’t look like a great way to build webapps (and I do mean apps and not sites).

A lot of work goes into designing good software. 280North are taking the great design work put in on Apple’s developer tools and making it available to people building browser-based apps. They’ve saved themselves the work of inventing every little piece of the model, instead focusing on the work of making it work well in JavaScript.

Awesome work, guys!

I’m crossposting this from the Bespin googlegroup.

Before moving on to the week that is coming, let’s look back at the week that was:

Week of February 17th:

• Version 0.1.2 of Bespin was released. Notable changes:

• System clipboard copy and paste now works in WebKit. Firefox will almost work. Vote up this bug:   https://bugzilla.mozilla.org/show_bug.cgi?id=407983 
• Project names now need only to be unique per user. Previously, they had to be globally unique. This reflects a bit of a change to how we were viewing collaboration. (More on collaboration to come).
• Thanks to that change, the sample project has been renamed from SampleProjectFor:yourusername to just SampleProject.
• Speaking of project renaming, you can now rename projects from the dashboard or the editor using the “renameproject” command. New ”createproject” and “deleteproject” commands make project-level changes much clearer.
• Every user now has a visible “BespinSettings” project. You probably      won’t be surprised to hear that it stores settings for Bespin! You can put custom code (set autoconfig on and put code in config.js) and commands (use the cmd* commands) in there.
• The login form is now a real form, which means that your password manager can save your Bespin login information.
• Client and server implementations can make sure they’re in sync with the X-Bespin-API HTTP header that is returned from the server. The current API version is 2 (which adds the renameproject API).
• key repetition works for arrows and backspace and lots of other minor fixes.

Coming up this week:

• Ben and Dion are in Miami for Future of Web Apps where they will be showing off Bespin.
• Joe is continue to work on collaboration and is experimenting with Neil Fraser’s mobwrite.
• I will finally be getting to work on the version control spec, and hope to be starting on the version control backend code.
• And we’re all going to be working on bug fixes and minor enhancements.

Despite working for an open source company, I have been pretty quiet here about what I’ve been doing in the Mozilla Labs web developer tools group. No more. We’ve gone public!

Mozilla Labs » Blog Archive » Introducing Bespin

Bespin proposes an open extensible web-based framework for code editing that aims to increase developer productivity, enable compelling user experiences, and promote the use of open standards.

I started working on Bespin as soon as I joined Mozilla, hitting the ground running with a new Python server. Ben and Dion had already done a lot of work and experimentation on Bespin prior to joining Mozilla in December, so I must confess that I am still fairly ignorant about the Canvas-based magic that they’re doing in the UI. But, Bespin has an architecture that lends itself well to selective ignorance: the server provides a RESTful API, and the client is responsible for all of the presentation. For their part, Ben and Dion have been able to be blissfully ignorant about the inner workings of the Python server.

Of course, I’m not a JavaScript noob and have done some work in the client, but my focus has been the server. Now that we’re out in the open, you can definitely expect that we’ll be talking more about how things work and how you can bend Bespin to your will. Bespin is honest to goodness open source (MPL-licensed), so it becomes an open and collaborative effort starting right away.

The initial reaction has been fantastic. There are tons of people hanging out in #bespin on irc.mozilla.org, and the mailing list has grown to a couple hundred members already. Thanks to everyone for jumping in with your thoughts and patches!

Here’s some of the coverage:

Dion’s post at Ajaxian:

From Dion’s personal blog:

Foolish chaps and companies have come to me in the past thinking that open source will be a silver bullet for “getting other people to do our work.” Those that have been involved in open source know that it isn’t the case. It is often more work. But, it is worth it.

From Ars Technica:

The project is still at an early stage of development and there is clearly a lot of work to be done before it will be able to deliver the same practical value as existing desktop editors. Despite the limitations, it shows an enormous amount of promise and has the potential to eventually deliver a user experience that rivals even the best text editors.

From Five Questions with Dion Almaer:

Now the browsers are moving fast again and building a first class platform for us to develop, the Open Web Platform. Instead of getting bogged down thinking about what IE 6 gives you, take some time to think about what you could build with the latest technology. I realise that you have to be pragmatic and get things working with your audience, but browsers are changing, and so are expectations.

From What Mozilla’s Bespin Bespeaks (ComputerworldUK):

You can see that Bespin is ticking all the Mozilla boxes, but what’s also striking is that this is a Web-based project: Mozilla is entering the cloud. It’s a further shift to viewing the Web as a platform for doing, well, just about anything. Clearly, against that background, open standards are even more important. And not only for the code: another issue that Mozilla will need to start addressing publicly is that of open data. As more stuff moves into the cloud, it become imperative to establish minimum standards for access to the data that is held there. I look forward to hearing Mozilla’s views on the subject.

While I certainly don’t speak for Mozilla, I would be extremely surprised if there’s anyone at Mozilla that believes that users should have anything less than full access and ability to take their data with them. There can be technical issues involved in providing the data, but the data should be available in some reasonable form. Bespin, for its part, makes it easy to export a project in a tarball or zipfile.

I was surprised to see Bespin covered even on Lifehacker:

Primarily, Bespin is a text editor—the kind you’d use for editing code or managing text-based todos. Using Bespin, developers could collaborate on projects through a unified interface (that still supports plugins!) no matter where they are—so long as they’ve got a browser.

cnet has the story, too:

For example, what about integration with open-source software repositories? If it’s flexible enough, Bespin could essentially act as a source code viewer that repositories such as SourceForge or Google Code could employ.

A nice writeup on the ReadWriteWeb as well:

It’s clear that a great deal of thought and attention went into this early version – and it’s a safe bet that it will only get more impressive as time goes on.

RWW last month surprised me with their coverage of me joining Mozilla.

I’m having a great time at Mozilla so far, and it’s great to be out in the open working with so many people now on Bespin and ServerJS.

It’s actually been 8 days now since I introduced the ServerJS group. As Peter Michaux mentions in “A Bright Future for Standardized Server-Side JavaScript”, the group has gotten off to an absolutely blazing start. As it stands this morning, there are 224 members who have posted 653 messages in 8 days.

There’s something more important than the numbers, though. Many people who have joined the list are deeply experienced in JavaScript-outside-of-the-browser. Everyone comes along with experience from other programming languages and environments. And all of knowledge and experience is being put to work in figuring out the approaches we want to take with modules, files and binary objects, as just a few starter topics.

You can see some of the work in progress on the ServerJS wiki. Note that the wiki has moved to wiki.mozilla.org.

The interest in server side JavaScript is self-evident, given all of the different projects and frameworks that have grown up around the concept. What’s new here is that these projects are getting together to form a real ecosystem, rather than standalone tools.

I think that better server side JavaScript tools will ultimately help web developers produce better user experiences more quickly, which fits in quite nicely with our goals in the Mozilla Labs web development tools group.

I expect that the coming week will have some even more interesting discussions than the last, so now is a great time to join up if you have ideas on what you want to see in JavaScript as a server side scripting platform.

The server side web development chatter over the past few years has been centered around Ruby, Python, the latest frameworks in Java and Groovy, plus the endless swirl of activity in PHP. At least, that’s the chatter I’ve been seeing.

Clearly, there is pent up demand to make JavaScript into a not only competitive but truly awesome server side web dev environment. I launched the ServerJS mailing list yesterday and as of now it has 136 members and a lot of active discussion. This is a very promising start.

Of course, discussion is not the same as code. Luckily, there’s no shortage of code. It’s a matter of picking the right code and sticking with it.

To help collect up all of the useful information that’s coming up, I have set up a wiki at Mozilla’s Developer Center:

ServerJS – MDC

At Tom Robinson’s suggestion, we are also meeting on IRC in #serverjs on freenode.

The energy behind this project is huge and I think a great new programming environment will ultimately emerge.

Server side JavaScript technology has been around for a long time. Netscape offered server side JavaScript in their server software back in 1996, and Helma has existed for a number of years as well. But, server side development has changed a lot over the past few years.

Aptana’s Jaxer gives an innovative view of how you can leverage a JavaScript environment running on both sides of the wire (client and server). Very convenient communication and the ability to easily share code between client and server are big benefits of running JavaScript on the server.

Jaxer and Helma are interesting projects, to be sure (and there are many others!). But what I see missing from JavaScript on the server is not interesting projects, but rather a useful ecosystem. People working in Python like to talk about the fragmentation of web frameworks and whatnot, but that’s nothing compared to the fragmentation of JavaScript.

For example, JavaScript needs a cross-interpreter standard library. Thankfully, some amount of standard library exists (the part inherited from the browsers). So, you get regular expressions and dates. But, what about files and directories? Why can’t the same API be made to work in Rhino, Spidermonkey, V8 and JSCore?

A handful of standard interfaces. Connecting to a database and running queries is a well understood and common problem. In Rhino, you get to use JDBC. But, JavaScript really should have its own cross-interpreter standard like Python’s DBAPI. It should also be possible to take a webapp that was originally deployed behind an Apache module running Spidermonkey and then put it behind Jetty thanks to a standard web server/web app interface.

JavaScript needs a standard way to include other modules and for those modules to live in discreet namespaces. There are easy ways to do namespaces, but there’s no standard programmatic way to load a module (once!). This is really important, because server side apps can include a lot of code and will likely mix and match parts that meet those standard interfaces.

There needs to be a way to package up code for deployment and distribution and further to install packages. Linux people will correctly point out that they can just type “apt get” (or yum, or whatever) and their work is done. But there are a lot of people using Macs and Windows who need a convenient way to get their development environments set up and to package up the code they write for deployment and for other people to use.

Part of the distribution and installation problem is a package repository. I don’t know if JSAN is the answer there, but I do know that an easy way to install a package and its dependencies makes a huge difference in how many libraries people will likely pull together in their apps.

And, on top of all of this goodness, we’d get template engines, object relational mappers, middleware, packaged apps, etc. In fact, many of those things already exist. But, the problem is that they have no common basis to sit on. And that’s what prevents an ecosystem from growing.

If you search the Python Package Index for WSGI (the Python standard for connecting webapps with web servers), you’ll find 180 packages today… servers, middleware, full blown applications. And those are just the packages that have “WSGI” in their listings. That’s what an ecosystem looks like. And Java has one, and Ruby has one, and JavaScript needs one.

It’s also worth noting that many of those WSGI components can run unchanged on CPython, Jython and IronPython, thanks to a common standard library. JavaScript has a collection of implementations in C, as well as Java and .Net implementations and there just needs to be a little agreement on some interfaces and such. Libraries that run in all of those places can attract more users and, hopefully, more committers to help the libraries grow.

What I’m describing here is not a technical problem. It’s a matter of people getting together and making a decision to step forward and start building up something bigger and cooler together.

To that end, I’ve set up a new ServerJS group in hopes of getting the interested parties talking and maybe even to get us together face-to-face to produce some code and settle on some interfaces. There’s a tremendous collection of JavaScript code out there already, and let’s see if we can make all of that code that much more valuable.

In the web developer tools group at Mozilla, we have a wide open charter to help software developers make the best use of the open web. Doing our bit to help the server side JavaScript community grow and flourish can certainly be a part of that.

(Before someone says “why not just use Ruby/Python/Java/C#?” let me just say that that is an entirely different question and I won’t address that in this post.)

Update: The group is now called CommonJS.

In Python, we’ve been enjoying the convenience of mixing and matching various things that follow the WSGI convention for a while. More recently, Rack brought those good features to a very similar interface for Ruby. And now I see:

Jack: Rack for JavaScript

echo “Rack provides an minimal interface between webservers supporting Ruby and Ruby frameworks” | sed -e s/Rack/Jack/g -e s/Ruby/JavaScript/g

Jack comes from Tom Robinson, one of the developers of Cappuccino/Objective-J.

At PyCon in March, I spoke about how applications can be browser-driven. For the past few months, I’ve been involved with a browser-driven app. Remarkably, some of the same kinds of problems crop up when making a browser-driven app as when you’re making a traditional server-driven webapp. So, today I introduced something called Reinhardt, a client-side web framework. Using Dojo’s Django Template Engine implementation (dojox.dtl), Dojo’s data model (dojo.data) and Reinhardt’s new Django-inspired URL dispatcher you get something that approximates a server-side web framework all on the client side.

This year has seemed like a big year for Python conference activity in the US. Of course, there was PyCon in March, which topped 1,000 attendees. I’ve also seen announcements for a bunch of regional Python gatherings (like PyOhio, which was close by but I couldn’t attend).

This year, we also get PyWorks, which is joined at the hip with php|works in Atlanta in November. This is the first year for PyWorks, and they’ve got a good lineup going. There’s a day of tutorials and two days of talks, so this is more like a PyCon than it is like those regional conferences.

In addition to attending talks myself, I’m hoping to meet some more good Python and/or JavaScript folks in the “hallway track”. I’m sure there will be lots of Dojo users mixed in with the Python and PHP people, so we should get together.

I have four speaking slots (one of which isn’t listed yet) over the two conference days (gadzooks!). I’ll be doing a revised and expanded version of my PyCon talk “Rich Client Web Applications with TurboGears 2 and Dojo”. I’ll also be giving an updated version of the “Easy build and deployment automation with Paver” talk that Mark Ramm gave in my stead at PyOhio. Paver really puts the “scripting” back in “Python scripting language” (Python certainly does a lot more than “scripting”!)

I’ll also be giving a talk called “ZODB: The Most Underappreciated Library in Python”. The ZODB is great. More people should use it. This is a talk I gave a couple months back at MichiPUG, so it’s only been seen by a small group at this point.

My fourth talk is one I haven’t given anywhere before: “Beyond the Source: Growing Your Community”. I’m going to talk in concrete terms about things you can do to grow an open source community. Open source projects really need to get to a certain level of use before they become viable open source projects, and there are many, many ways in which people interested in a project can help it get there.

I hope to see you there!