Blue Sky On Mars

The server side web development chatter over the past few years has been centered around Ruby, Python, the latest frameworks in Java and Groovy, plus the endless swirl of activity in PHP. At least, that’s the chatter I’ve been seeing.

Clearly, there is pent up demand to make JavaScript into a not only competitive but truly awesome server side web dev environment. I launched the ServerJS mailing list yesterday and as of now it has 136 members and a lot of active discussion. This is a very promising start.

Of course, discussion is not the same as code. Luckily, there’s no shortage of code. It’s a matter of picking the right code and sticking with it.

To help collect up all of the useful information that’s coming up, I have set up a wiki at Mozilla’s Developer Center:

ServerJS – MDC

At Tom Robinson’s suggestion, we are also meeting on IRC in #serverjs on freenode.

The energy behind this project is huge and I think a great new programming environment will ultimately emerge.

It turns out that both pip and easy_install have trouble with the Paver 1.0a1 package. I’m going to look into that, but in the meantime I have removed the Paver 1.0a1 tarball from the cheeseshop so that people don’t accidentally get it. Sorry about that!

As a side note, it is unfortunate that easy_install and pip both will pick up alpha and beta releases in preference to release versions. I would think that a nicer default would be to prefer the current release version unless there’s a flag saying “give me the test release”.

Server side JavaScript technology has been around for a long time. Netscape offered server side JavaScript in their server software back in 1996, and Helma has existed for a number of years as well. But, server side development has changed a lot over the past few years.

Aptana’s Jaxer gives an innovative view of how you can leverage a JavaScript environment running on both sides of the wire (client and server). Very convenient communication and the ability to easily share code between client and server are big benefits of running JavaScript on the server.

Jaxer and Helma are interesting projects, to be sure (and there are many others!). But what I see missing from JavaScript on the server is not interesting projects, but rather a useful ecosystem. People working in Python like to talk about the fragmentation of web frameworks and whatnot, but that’s nothing compared to the fragmentation of JavaScript.

For example, JavaScript needs a cross-interpreter standard library. Thankfully, some amount of standard library exists (the part inherited from the browsers). So, you get regular expressions and dates. But, what about files and directories? Why can’t the same API be made to work in Rhino, Spidermonkey, V8 and JSCore?

A handful of standard interfaces. Connecting to a database and running queries is a well understood and common problem. In Rhino, you get to use JDBC. But, JavaScript really should have its own cross-interpreter standard like Python’s DBAPI. It should also be possible to take a webapp that was originally deployed behind an Apache module running Spidermonkey and then put it behind Jetty thanks to a standard web server/web app interface.

JavaScript needs a standard way to include other modules and for those modules to live in discreet namespaces. There are easy ways to do namespaces, but there’s no standard programmatic way to load a module (once!). This is really important, because server side apps can include a lot of code and will likely mix and match parts that meet those standard interfaces.

There needs to be a way to package up code for deployment and distribution and further to install packages. Linux people will correctly point out that they can just type “apt get” (or yum, or whatever) and their work is done. But there are a lot of people using Macs and Windows who need a convenient way to get their development environments set up and to package up the code they write for deployment and for other people to use.

Part of the distribution and installation problem is a package repository. I don’t know if JSAN is the answer there, but I do know that an easy way to install a package and its dependencies makes a huge difference in how many libraries people will likely pull together in their apps.

And, on top of all of this goodness, we’d get template engines, object relational mappers, middleware, packaged apps, etc. In fact, many of those things already exist. But, the problem is that they have no common basis to sit on. And that’s what prevents an ecosystem from growing.

If you search the Python Package Index for WSGI (the Python standard for connecting webapps with web servers), you’ll find 180 packages today… servers, middleware, full blown applications. And those are just the packages that have “WSGI” in their listings. That’s what an ecosystem looks like. And Java has one, and Ruby has one, and JavaScript needs one.

It’s also worth noting that many of those WSGI components can run unchanged on CPython, Jython and IronPython, thanks to a common standard library. JavaScript has a collection of implementations in C, as well as Java and .Net implementations and there just needs to be a little agreement on some interfaces and such. Libraries that run in all of those places can attract more users and, hopefully, more committers to help the libraries grow.

What I’m describing here is not a technical problem. It’s a matter of people getting together and making a decision to step forward and start building up something bigger and cooler together.

To that end, I’ve set up a new ServerJS group in hopes of getting the interested parties talking and maybe even to get us together face-to-face to produce some code and settle on some interfaces. There’s a tremendous collection of JavaScript code out there already, and let’s see if we can make all of that code that much more valuable.

In the web developer tools group at Mozilla, we have a wide open charter to help software developers make the best use of the open web. Doing our bit to help the server side JavaScript community grow and flourish can certainly be a part of that.

(Before someone says “why not just use Ruby/Python/Java/C#?” let me just say that that is an entirely different question and I won’t address that in this post.)

Update: The group is now called CommonJS.

I have just pushed Paver 1.0a1 up to the Python Cheeseshop. This is very much an alpha release, so beware. That said, I’m successfully using it in my own projects and have been for some time.

I have significantly changed the way Paver works, and I think the new structure is simpler and more flexible because it is no longer tied to distutils. In fact, Paver’s task running capability is in paver/tasks.py and that module could conceivably be used standalone. And, the namespace of tasks is not muddied with a bunch of distutils tasks if you’re making a pavement for some other purpose.

But, the distutils/setuptools integration still exists and is easy to turn on.

Paver 1.0a1 has a handful of useful new features, including the ability to run separate sub-pavements (in the same process, no less).

The docs on the Paver site are still for 0.8, and I will leave it that way for now. When you install Paver, you can run “paver paverdocs” to see the docs for the version of Paver you are running.

Please report any bugs you find either to the googlegroup or directly to the tracker at googlecode.

Thanks to Marc Sibson, Bryan Forbes, Juergen Hermann and others who have contributed to this release.

In Python, we’ve been enjoying the convenience of mixing and matching various things that follow the WSGI convention for a while. More recently, Rack brought those good features to a very similar interface for Ruby. And now I see:

Jack: Rack for JavaScript

echo “Rack provides an minimal interface between webservers supporting Ruby and Ruby frameworks” | sed -e s/Rack/Jack/g -e s/Ruby/JavaScript/g

Jack comes from Tom Robinson, one of the developers of Cappuccino/Objective-J.

With my new job comes a new MacBook Pro (the latest generation, very slick!). The Setup Assistant did a great job of making my new MBP just like my old one, carrying across all of my applications and settings from my latest Time Machine backup. Everything in /usr/local/bin made it. Ditto for my MacPorts stuff in /opt.

One thing that didn’t make it was the Python libraries that I had installed in the system-wide site-packages directory (/Library/Python/Version/2.5/…). The Setup Assistant is designed to not pull OS parts across when you are loading up a new machine. That makes sense, but unfortunately the Python stuff seems to fall into that category.

Perhaps the site-packages directory should be moved to /usr/local or something so that it gets copied over? A change like that would be useful for Apple to make.

Thankfully, the impact on me is pretty minimal, because I tend to do my work in virtualenvs and don’t have very many packages installed system-wide.

At the next Michigan Python Users Group (MichiPUG) meeting, we’ll be talking about building custom web frameworks using WSGI bits and pieces. I’ve done this a couple of times now and think it’s a powerful and straightforward approach that works well for certain kinds of projects.

I’m figuring we’ll take a look at some available components and glue stuff together to build a framework during the session.

As usual, the meeting is the first Thursday of the month (February 5th, in this case) at 7pm at the SRT Solutions office in downtown Ann Arbor.

The next release of Appcelerator Titanium (PR2, due January 23rd) is slated to include support for Python and Ruby!

Titanium is a completely open source competitor to Adobe AIR. It lets you build installable desktop applications that build their GUIs via a built-in WebKit renderer (in other words, you can use HTML/CSS/JavaScript to build a GUI, just like you do on the web). The next version of Titanium is going to include support for Python and Ruby, which means that you can use something other than just JavaScript for creating your desktop apps.

I haven’t built Titanium yet to see how it works (script type=”text/python”?), but the fact that they’re working on this is very interesting.

We make shitty software… with bugs!
old Living Videotext slogan, as reported by Dave Winer

I love that quote. On the surface, it sounds disparaging about the software you have today, but it’s actually more of a statement of hopes for the future. The status quo, in fact every status quo, is less than ideal. It’s great to be proud of your products and where you are, but it’s even more important to keep in mind how much more there is to do.

Early in my career, I worked at the ANS Network Operations Center. ANS ran the original NSFNet and provided internet and dial-up connectivity for America Online, as well as a number of large companies and research labs. We had the internal slogan “we suck less”, and it was the same idea as the “we make shitty software” quote. At ANS, we did suck less than our competitors. But, every time a customer’s connection was down we knew we had work to do to make things even better.

I’ve been doing web-based software development for a long time, and the tools for creating web-based software have improved tremendously. Server side frameworks like Rails, TurboGears and Django have made writing typical server side code far easier. Client side toolkits like Dojo, jQuery and Prototype/Scriptaculous have made creating better client experiences even easier. And tools like Firebug have made it much easier to debug an application and tweak its appearance on the fly.

But, developing compelling web-based applications still sucks. Its still more work than it should be in the ideal case. I’m sure that the people working on all of those tools, and all of the competing tools, all have ideas on how to make things better and will all be pushing the state of the art forward and removing a bit of suckyness with each release.

I’m delighted to say that starting today I am working with Ben Galbraith and Dion Almaer at Mozilla Labs on web development tools. I’ve been a reader of Ben and Dion’s Ajaxian site for years and I know how much thought they’ve put in to making webdev better, so I’m really excited to be joining their group. And Mozilla looks like a fantastic organization to be a part of.

It’s still a very new group, but you can bet that every day I’m going to be thinking about how I can help make web development easier, faster and better.