But first:

If I had an iPad rather than a real computer as a kid, I’d never be a programmer today.

—Alex Payne

Alex’ quote has been whipping around today, and when I first picked it up had been distorted to imply: “If I had only an iPad, rather than a real computer, and there were no open computers nearby, or at school, I would have never learned how to programme.” That’s a bit different. Thanks Twitter.

Alex goes on to talk about hacking on the machines he grew up with. Breaking them, repairing them. That’s a kind of activity that is dead and done in a closed, DRM-encrypted encrypted environment, certainly. However, the sentiment that has been echoed on his behalf (and I don’t think this is what he meant), is that people would not become programmers at all. That’s bullshit. Even if every computer on earth was a closed environment, people would be learning to programme, it just wouldn’t be the same way they do it now.

If root did not exist, it would be necessary to invent it. In a world with only closed systems, commercial success would still be tied to the development of third party apps, but the responsibility to train developers and introduce people to programming would fall to the gatekeeper of each system, rather than it being something that can organically happen. So, Apple, or Microsoft, or whoever, would be shipping programming sandboxes for their devices; maybe scripting tools, maybe programming games, maybe cut-down versions of the professional development environment. Whatever it would be, if there was no open ecosystem to learn in, a closed system to learn would be created. The idea that people would not become programmers in 1995 because they can’t do everything to a single closed system in an otherwise open industry today is a fallacy. If the first computers had been closed, everything would be different, right down to the seeds that inspired our initial passions for software development.

Which is not to say that I think everything would be fine. I think we as an industry, as a species, would be less creative and less innovative without 25 years of open computing. Our world and lives would be worse. But we would still have new programmers. And that is all I have to say about that.

Now, the possibility that we might find ourselves moving into a period of closed devices throws up a number of obvious fears: Apps getting rejected by gatekeepers, and the physical capabilities of devices being locked away behind private APIs, plus the inability to add new capabilities to a device without licensing a proprietary dock connector are three obvious ones. They are not my main cause for concern. They’re big roadblocks, but they’re also all enforced by policy and could simply cease on any given day.

Allow me to talk about some of favourite software on Mac OSX

Megazoomer is an add-in for Mac OSX. It’s a single feature, and once installed it injects a new piece of functionality into every Cocoa app; under the View menu, you can now choose Mega Zoom (or press ⌘↩. The window smoothly scales to true full screen, the Dock and Menu Bar slide out of the way, you can switch any application into a tranquil, productive WriteRoom.

Inquisitor was a beautiful hack that added auto-complete search suggestions, inline search results, and search provider customisation. It predated the simpler, native search suggestion that now ships in Safari 4, and is still more polished (without being bloated) than any other search suggest feature I’ve seen. It stopped being maintained when Snow Leopard forced a change to the Input Manager integration method, but Glims is a similar replacement.

The Microformats Plugin for Safari adds an hCard and hCalendar parser to Safari, and shows an alert when you visit a page containing embedded data. It lists the items, and lets a user add contacts and events to their native Address Book and iCal applications. It was simple, worked pretty well and was a hugely valuable demonstration of how microformats could be exposed in user interface (I wrote similar thoughts about this sort of UI at the same time.)

Airfoil is an application for OSX (and Windows) that allows you to stream any audio, from any application, over your local network to an Airport Express audio hub. It’s a feature of iTunes (AirTunes) that allows you to stream what you play there to Airport; this app makes it system-wide. It’s a fully supported, commercial piece of software, but one of the ways it works best is by injecting a piece of code into applications called ‘Instant Hijack’; a hook allowing Airfoil to grab audio from a running application right away, rather than having to relaunch the application. It’s very, very slick.

I want to draw particular attention to the fact that all of these apps are add-ins for the native, closed-source Mac OSX environment. They’re not changes to source code, nor do they use featureful extension mechanisms, like the plug-in interfaces of web browsers. These are hacks, add-ins that use a feature of the Mac’s Cocoa framework, able to change user interfaces and inject functionality in a raw way, at run time.

Reading the list, the functions I’ve highlighted here seem small. They’re enhancements; improvements; subtle. In these cases, their creators have developed single features and added them to existing applications. In doing so, they have prototyped and experimented with web browser functionality that may become native and expected in the future, but right now it’s new, and to those who install these hacks, they sink or swim based on whether your productivity improves.

In the environment of the iPhone, or the new iPad, it is this kind of development that is completely lost. Each app, locked up inside its own signed bundle, unmodifiable by design. If someone wants to ship a better search interface for a web browser on the iPad, they have to ship an entire browser. If someone want’s to ship user interface for microformats in the browser, they would have to build that browser. Some of the earliest demonstrations of microformats came from add-ins like Tails for Firefox, and later the Safari equivalent. The early success and adoption of microformats simply would not have happened without hackable software; allowing a single feature to be augmented onto a greater whole. The innovations of microformats themselves, the implementations we now see in search engines, as well as the momentum in other structured data mechanisms (RDFa, microdata) can be traced down to early, enthusiastic adoption of practical features like Tails, Operator and the Safari Microformats Plugin. Those plug-ins were not just innovative in themselves, but they supported a much broader effort.

In the case of Airfoil, though the iPad plays music through iPod or Safari, there’s no native AirTunes offering. If there were, you’d be beholden to Apple for it to function in third party applications. If they don’t ship it, no-one can.

None of these add-ins will exist on the iPhone OS for as long as it remains closed. Of course innovation will happen, of course amazing, standalone applications will be built, but that’s just it, they stand alone. Dreams of amazing features cannot be built without also building the whole. The nail for anything even close to this functionality is that current App Store politics dictate that you cannot ship applications that themselves interpret code; it’s not permitted for third parties to include extension capabilities in their apps.

No hacks. No scripting interfaces. Just apps. Many exciting things will still happen on this platform, even without policy changes. But I offer a lament to the the loss of extensibility.

The Web

There is one application on the iPhone that supports user scripting, and execution of user extensions, and that application is MobileSafari. Safari, like all web browsers, supports bookmarklets; strings of JavaScript code embedded in a URL, prefixed with the javascript: pseudo-protocol. The JavaScript code executes when you select the bookmark, and is able to manipulate the context of current page, and navigate to new pages.

The iPhone bookmarks interface is fairly clunky, and adding bookmarklets is impossible without preemptive effort by the writer, quoting instructions for installing Twittelator’s bookmarklet on iPhone:

I first had to navigate in mobile Safari to the Twittelator iPhone bookmarklet page and follow the lengthy instructions there. Basically you have to save that page as a bookmark, and then go back and edit that bookmark to delete everything before the javascript:window.location=%27twit://%27+window.location part of the URL. Once you do that, the bookmarklet executes the steps to post to Twitter.

—Safari iPhone Bookmarklets by Amy Gahran

Not very friendly. The other way to get Bookmarklets onto an iPhone is to add them to regular, desktop Safari and then synchronise your bookmarks using iTunes. As such, the Tumblr bookmarklet for blogging runs on my iPhone, since it’s in my regular browser too. Others have used JavaScript to recreate the useful “Find in Page” functionality you find in desktop browsers. Lifeclever documents that and others.

Since Bookmarklets run in iPhone Safari, it is my assumption they will also run in Safari on iPad. The bookmarks interface there is more conventional; a long menu that pops up over the current page; and as a menu, far more intuitive to pick functionality from. We’ll see whether there’s an easier way to add bookmarklets on the device itself.

The scenario that remains is this: The only extensible software on the closed iPhone OS is the Web. The only place you can enhance an application, rather than replacing it in entirity, is the Web. The only place your feature and someone else’s feature might co-exist is through two bookmarklets on the same website.

What we are heading toward is a reality (hopefully temporary, but I suspect for years) where the Web is the only extensible platform on Apple’s consumer devices. There are already components in place for building a JavaScript development environment within the browser; Bespin is a rich code editor, Firebug Lite is a JavaScript bookmarklet to add a DOM explorer and inspection tools. A web application that lets people write and execute code, and inspect the result, is the only real hacking tool we can offer the actual users of these devices.

It is interesting to me as an advocate of the Open Web that the tighter grip on the native environment drives people to web standards as their only alternative (both technical and political motivation.) But, as an admirer of Apple’s very polished native UI, and of the Mac hacks I mentioned at the beginning, I’m still disappointed.

It’s an exciting, inspiring time. If we are brave, then the imposition of these restrictions can only make us smarter when we work around them.

In summary, the iPad is an extremely well designed, meticulously constructed consumer device. The iPad is something which I think will fit perfectly into the life of someone with desktop computer at home. There, it provides lots of new functionality, and will prove useful a lot of the time. It’s more convenient, and will enable people to be more social with their cohabitants whilst still having a connection available. It is a device ideal in both size and price for integration information and media access with home life, rather than home-office life. My Father, sat in the living room, is regularly poaching my sister’s iBook, looking things up on the internet; searching for prices, fact-checking my siblings, keeping up with sports scores, weather and travel information. For him, an iPad would be a perfect addition (though, I think Jeff Croft makes a good point with regard to multi-user support; the living room and kitchen scenarios Apple presented on Wednesday set the tablet in a clearly shared environment, whilst providing access to personal information.) Overall, I’m very impressed.

For me, my caveat is that I already use a laptop on the sofa; that makes the iPad a harder sell for immediate adoption. I’m already sat on my sofa, computing. I’ll be waiting for functionality to be added to the Tablet over the next six months—from Apple or from third parties—and looking to see where it can really add something to my life. I like the form factor.

‘Old Globe’ by Kenneth Lu

This is a write up of my talk ‘Designing for Location’, delivered at Chromatic last year. The slides (pdf) and original notes are also available, but this series of posts hopefully serves to cover everything I said without the context of seeing me speak.

Who am I to talk about location? Well, let’s be clear: I am not as qualified as many others. In June of 2008 I joined the Fire Eagle team at Yahoo! Brickhouse. In that time I learned a great deal from supremely talented co-workers, who worked out the detail of the Fire Eagle implementation. I was part of the team that launched Friends on Fire —a social location application—in December 2008. After that, Yahoo shuffled around and I moved to the Yahoo Developer Network to do other things. I returned to the Fire Eagle team to develop the second Friends on Fire release at the start of 2009.

That is all to say, I have experience, but I’m indebted to the Brickhouse environment for teaching me so much, so quickly (in particular, Tom Coates and Sam Tripodi). Brickhouse contained the most talented, inspiring people I’ve ever had the pleasure to work with.

I’m writing this up so thoroughly not because I’m an authority on Location Services, but because there’s still a lot in my head to get out. I got given a chance to talk about it, and though I stopped working on Location Services full time, the field still immature and I figure that perhaps documenting this set of thoughts is valuable to someone.

I’m going to publish in three parts. First, the introduction and state of obtaining location, followed by the state of location display, and finally an overall “Using Location”. This is part one.

“Location is complex.”

A simple mantra that accompanied the launch and initial explanations of Fire Eagle. Introducing location to the web is a complex problem full of edge cases. The common problem you’re trying to solve is very simple: You want to attach a location to another piece of data; add an extra dimension to your application, and do something exciting and insightful for your users.

But, actually getting that data, in a useful way, is not so simple. Technical and social obstacles exist that make location harder at the moment.

Do you speak Geo?

Does your application speak geo? What inputs does it understand? There are human forms of location (city names, countries, names of places) and there are data forms of location (co-ordinates, database IDs). Which ones can your service handle?

Not all your users will have a GPS in their pocket. How accurately is their location recorded? With so many different methods of obtaining location, accuracy varies, and so can usefulness.

Your application is going to be flooded with locations of both precise accuracy, and approximate values. Precision from buildings, streets, to locations the size of entire cities. Can your application perform something useful just knowing the city a user is in?

Did the user intend to explicitly ‘check in’, or are they just passing through? Are they in motion? What happens if your user provides no location?

Your depth of understanding location formats affects your interface. If you can’t parse all forms of postal addresses, how are you going to communicate that limitation to users? Chances are the quality of that parsing will vary from country to country too, since international address formats are varied. Your interface needs to be robust enough to communicate this variance.

User location is not just about where, but also when. What’s the context of your location consumption? Is your application only relevant when provided with fresh data? What happens if you get a location update that’s a day old? A week old? Preemptively from the future? You’ll find a lot of user’s won’t update their location very often. If you discard locations too aggressively, your dataset will be reduced too far, and only a small slice of you users will see any benefit of location in your application.

And, of course, you have to persuade users to actually share with you in the first place.

This huge variance affects all aspects of your application. How its represented in the page, how you communicate with your users, how you perform data analysis.

Grey Areas. Edge Cases. Multiple facets. This is location on the web.

Original Location UI

Location data has been used on the internet for much longer than the new trend in social-enabled, GPS-enhanced, API-powered services.

Often, websites ask about city and country information, sometimes postal codes, to provide localized content.

Generally, these are commerce scenarios: Localized product listings, directions to local support offerings, addresses for shipping and tax calculations. The functionality depends only on broad locality, and is only relevant to the service for the duration of the user’s session. As such, low-fidelity input from users has worked quite well.

The second major use is in shared, personal profiles. Long before we called it social networking, bulletin boards had simple user profile pages and inevitably asked for a ‘hometown’.

This labeling is somewhat important. Your ‘hometown’ is a single, static piece of data, often historical. Systems that instead asked for your ‘city’ exposed a problem: Locality for persistent profile information goes stale. When people move from place to place they don’t update their user profiles (understandable, given how many separate profiles they have).

The result is that you can only reliably store manual input for data you don’t expect to change. This doesn’t stop sites collecting ‘city’ anyway, but does mean that a large number of sites hold inaccurate information.

This level of location interaction is generally focused only on the city and country level of detail, which helps to simplify the input.

At the simplest, input for persistent locations consists of a text field, and is generally not validated. The result is that the data tends to be illustrative in a profile, rather than a reliable pivotal piece of data. People input Manchester, Manchester, UK and Manchester, United Kingdom all to refer to the same place. This inconsistency, multiplied across all possible locations means that this field is hard to work with as query data. Flickr did something very interesting, where in addition to regular free-text inputs, they also requested an Airport code (e.g. SFO or LHR). By asking for an approximate but reliable and unique location identifier, they had the option of using it in data processing without disambiguation. Also, it’s a vague and quirky enough piece of data that users won’t feel violated in sharing it.

The other forms of obtaining a location are crude: Clicking a map (or series of maps at increasing precision), or picking a location from a drop-down menu. Maps can work quite well, but are laborious, whereas picking your country from a drop-down list is rightfully regarded as one of the worst pieces of user interface ever conceived. The problem being that applications need precise input.

Lists fail most spectacularly when handling synonyms. For the purposes of almost all web applications, the United Kingdom, Great Britain, and England refer (inaccurately) to the same country (Gary Gale can clarify this, if you like.) ‘United Kingdom’ is most common, but some web applications use other variants. In a list, you have no way of knowing which name the application expects until you start scrolling.

Modern Location UI

With the hardware to identify location becoming commodity, and social and informational applications becoming more sophisticated, location interface has too.

Through mobile phones, users now have access to raw data about their location, and have been given software to share that location with applications. Furthermore, content on the web has been located, with geotagging of photographic, video and text content far more widespread.

Location gathering is no-longer just about city or country level, nor is it just about one-time queries whilst the user is active. Apps can work with location even when the user is away.

This generation of web applications uses persistent location, shares it with people and services, and attaches it to all sorts of content. It’s used to filter content, and enhance personal analytics. The means of collecting that location are now broad:

GPS hardware is contained in all new smart-phones. The iPhone, Palm Pre, BlackBerry, Androids and so forth all have location gathering hardware and expose it through simple APIs.

When GPS is unavailable, companies like Skyhook and Navizon map the wireless networks in urban areas. The signal strength of networks visible to your computer or mobile phone are processed to compute a location with remarkable accuracy. Clarke for Mac OSX and Fire Eagle, Firefox, and the CoreLocation Services API on iPhone and Mac OSX Snow Leopard all make use of Skyhook’s service.

Location can also be obtained from databases of IP addresses, again with surprisingly good accuracy—postcode level or better—within urban areas.

Manual input has become more sophisticated; Google Maps focuses on auto-completion and linking string queries with search for street addresses, venue and business names.

By contrast to the breadth of location sources, Four Square has a specific usage, and so only works with venue names; no other kinds of location input is supported.

Co-ordinates are rarely used in a visible way, since alone they’re incomprehensible to humans. However, they represent precise location points, so where interfaces such as Flickr’s Organizer application uses drag and drop to precisely place content on a map, co-ordinates are the data created.

All of these can provide quite precise location points. Other applications, especially queries, are more about specifying areas.

Yahoo! Local includes a beautiful piece of user interface for editing a location range, illustrating the centre-point of a search and highlight circle for the search range. The radius slider can be dragged to expand of refine the area. It’s simple, illustrative and directly manipulatable.

The next article will document the different methods of displaying location back to users.

Every change we make to our email behaviour, whether it’s organisation, tools or simply trying to email less, we are trying to find an effective balance. I’ve just come out the other side of one change, and am spending a few weeks making a strong, conscious effort in response.

At this point, my email inbox tends to receive the following: Notifications (from services), content summaries, status changes, reminders; pushed information, newsletters, aggregate news; actual communication, enquiries, conversations, planning and social synchronisation.

My email clients of choice is Mail on Mac OSX, and Mail on my iPhone. Previously I used Gmail from Google, but I quit the service because I found the browser-based user interface clumsy, and the application as a whole aesthetically ugly. Mac OSX’s user interface is elegant, the display of email good for reading, and the desktop client remains robust. My email is on an IMAP server (because that’s how email should be accessed), and I can fall back to my service provider—Fastmail —for a clean and functional web interface if I’m ever away from my main machine. I have also developed a certain wariness of storing too much of my data with Google for free, and prefer to pay for the excellent service I now get.

Gripes with Gmail as an application remain, but Gmail did massively inform my email behaviour. Most importantly, archiving rather than deletion. I carried over that practice to my new host, and now have some four years worth of email archived on my machine, and backed up around the world. I can, if needed, find pretty much any piece of mail given the right parameters. Not that I will ever need to see most of that mail again. But archiving is easy for a while, and my compulsive nature makes thorough archiving desirable.

I organise my mailboxes annually. Each year I archive everything from the previous year into an annual parent mailbox, effectively removing all the top-level mail folders from my account hierarchy. I start each year with a bare ‘Inbox’, ‘Drafts’, ‘Sent’, ‘Archive’ folder set, and then the categorical mailboxes get recreated when the first relevant piece of mail comes in. Some get recreated within a day, others never come back. For what it’s worth, I’ve always found a simple tree structure to be entirely sufficient for organising email; I never proved multi-dimensional tagging to actually be useful in Gmail.

Although my archive routine is simple, it is slowing me down. Huge amount of archived data, most of it useless. So I have all of my domain renewal notifications? Why? To keep a record of when my registrars have been in contact, so that I know exactly when I was reminded of particular actions. That is an example of me choosing the simplicity of the archive action instead of making a good-value decision about what is worth archiving. I’ve now accrued too much information to effectively dig through.

So, my first email goal from now on is to archive less. To trash messages that are of low value. Being an archivist, completist and pedant is an unhelpful email trait.

Years of signing up for services means that I get a regular stream of automated or blind-mailed content from services. This email I do delete. But because the effort of performing a delete action is so low I have tolerated an increasing quantity of mail for years. The processing cost is low, but occurs frequently and at irregular intervals. The problem is exacerbated in a mobile environment, where a ‘new message’ notification triggers a significant interruption. That interruption would be appropriate for valuable, personal communication.

As well as notifications, I receive a small amount of actual content by email as well; most notably the daily Snowmail email from Channel 4 News in the UK. Unfortunately, that content is only distributed by email, and is not published as a blog. This is content I read, of value, but is not content that should be pushed to me, it’s content I should pull up at my convenience.

My second email goal is to reduce the amount of email I receive in my inbox to be 95% personal communication. This has meant that every day for the past two weeks, I’ve actually taken the extra time to find the unsubscribe link in the footer, or edit the communication preferences of whichever service is sending notifications, and permanently disable it. I suspect it will take months to clear all of them, since some services are very infrequent in their mailings, but already I’m finding that if my iPhone vibrates, then the message I’ve received is now most often of actual interest.

Newsletters require more work. Email is the wrong delivery mechanism; they should not be push content in the first place. So I’m going to experiment with subscribing a blog-by-email address to the mailing list in question, so I can convert the content into an RSS feed and have it consumed by Fever instead.

There is value in email, and that value is communication. Over years, service providers and publishers have taken advantage of email’s ubiquity to adapt it for push, notification and automation. Better solutions to those use cases are emerging (or already exist), so this is the time to reclaim the inbox, reduce your email throughput back to what the medium is really good for. I’ve already seen that a little bit of persistent effort can greatly increase the quality of email as a tool.

Anyway. I managed to contain my nerdish need to rant about it to just one long comment on James’ blog. (Well… also Twitter.) You should read it and follow up there, if you’re so inclined (there’s a good discussion between James and Tom Coates to read, too.)

My comment is also copied for posterity below:

Tom makes some fair counterpoints, but the good plot details picked out cover a tiny minority of actual episode screen time. Yes, conceptually, there’s a lot to like about The End Of Time, but the execution of the episode was ruining.

So, the Master comes back. John Simm is an outstanding actor, of course, and in his last outing he and David Tennant made great scenes together just by showing up and being awesome. He’s back… but at no point in this story is he a Time Lord. Now, he’s a super-hero. Jumping in the air? Shooting electricity from his hands? Using the electricity in his hands as a propulsion mechanism so that he can fly?! Fuck off. Fuck off. Absolute shit like that is what took up the time on the television. That and running around the docks. And that’s after we’d dealt with him being resurrected using magic potions. By a cult! That I just made up! But then it goes wrong using anti-magic potions! From a resistance movement! That I just made up! What on earth was going on there?

What was going on was an inordinate amount of screen time and distraction being inserted into the programme so that RTD could give one of his precious tertiary characters (Mrs Saxon) screen time in the final episode. Entirely unnecessary. Even though she was in the series too briefly for the viewer to ever become emotionally invested in her, apparently she was actually really important and she still gets to be a little redeemed hero. Of course.

All of this after accepting the rubbish way in which RTD ‘killed’ The Master in the first place in season 3. Why didn’t RTD just let him get away? He’s a recurring character, after all, and he’d only just reintroduced him. Heck, he could have done a fun pastiche of Darth Vader’s escape at the end of A New Hope; that could have been pretty good. (Aside: The ‘refusing to regenerate’ detail was a good one and played very well, but occurred far too soon in the series; the Master had been back in the show for two episodes and one story before being ‘permanently’ killed out of it again.)

A problem I’ve had with RTD’s writing all along is that he incessantly rejects building story around the canonical, pseudo-scientific base of Doctor Who in favour of just making shit up. RTD wasn’t writing The Master as a Time Lord, he was ‘Generic Nemesis’, and this time he had silly powers.

So much of the plot was unnecessary. Things were introduced, barely explained, changed and barely re-explained. The incredible-gene-splicing Gate of Doom? One minute it’s a device to make one girl immortal, now it’s a device to reprogram all human DNA. There was no story telling to being it into the plot, it was just there in its own unjustified story thread. Why? That plot segment could be so much simpler if the people who had found the gate had no idea what it did.

The Master would show up, take the lead and trick the naïve humans into activating it. Heck, the trick could be ‘it will make your daughter immortal!’. Then he double crosses them, obviously. But instead, the who-the-fuck-are-they owners of The Gate have to be put in control of the plot for too-long a while, hunting down and capturing The Master, all at great expense of screen time. No background justification for the things they know and possess is provided.

I think that Russell T Davies has all of these plot points filled out in his head. I think he imagines this universe of moving parts and characters that he’s invented and cares about. He made them up and made them live. They’re as important to him as the ‘real’ Doctor Who characters are to me. But then he forgets that he’s making an TV special. Rather than cutting those ideas that are shallow, don’t make sense, or distract from the Time Lords (all of them), he cramped it together. End Of Time Part 1 felt like I was watching a clip show at points. Disjointed, scrappy scenes showing snippets of the story. There was no story telling, it was a summary, a recap of a storyline that hadn’t actually been told. I don’t appreciate a writer assuming that I’ll fill in the narrative, characterisation and backstory with my own fan-fiction afterwards, especially a story already failing to explore the rich science-fiction universe it’s been set in.

Moving on. I’ve covered The Master’s new super-powers, and James’ point about his energy hunger/consumption/expulsion is so spot on in highlighting RTD’s weakness for stupid visual effects over even light-weight science. But the other plot devices are even more annoying. Timothy Dalton has a Magic Time Glove! It can undo The Master’s actions in 20 seconds; actions that required half an episode to set up. Also, it’s a gun to shoot The Doctor with! Of course! Does it make toast? I’d really like some Time Toast. I bet if you ate that all kinds of crazy things could happen (and then be immediately undone when you drink a glass of milk, or by touching a cow, or a pregnant woman.)

The return of Gallifrey at any point in the Doctor Who story should have been a big deal. Instead it was briefly tedious and then got written out faster than it emerged. The ‘they said something, not someone!’ line was utterly shit.

It was so close to the Earth it would have caused any number of natural disasters that children learn about in Primary School, and a great number that they don’t. Then it disappears again with no adverse affects than the flustering of middle-class housewives that populate 90% of Russell T Davies vision of England. Gallifrey doesn’t mean anything in RTD’s Doctor Who. It’s a historical footnote, to a history that you’re only told exists (but barely told about). It’s reduced to a cheap name-drop. Even now when the planet finally makes a physical appearance in a story, nothing comes of it. Nothing happens on it, it offers no resources, it doesn’t contain a macguffin. It was a tedious waste.

Oh, and the Time Lords ascending? Really? Been watching a little too much Stargate SG1, have we?

And then the end itself. How limp. Homages are fine, stealing from Stargate is dubious, but you’d expect The Doctor’s actual fatal-blow to be original. Instead, it was radiation poisoning lifted right out of Star Trek II. Even the source of the radiation had nothing to do with averting the preceding disaster scenario. Then the regeneration starts, apparently, but unlike Nine, or the last time Ten regenerated (but ‘didn’t feel like’ properly regenerating… for fucks sake…) the healing starts right away so he can be pretty again, but he still seems to have a period of about three weeks for a final curtain call. And that final period, that’s what nailed it. This wasn’t a finale for the Doctor Who character, it was for Russell T Davies’ show. His show that we have the privilege to glimpse inside of once a week. He wasn’t writing it to give The Doctor a decent ending, it was written to grant a cameo to every single character he’d ever dreamed up, no matter how absurd. It was trite and shit. And of course Rose is back to say goodbye. Again.

This overall plot could have worked, but the script was an ego-satisfying, first-draft, mastabatary, brain fart.

Regarding James’ alternative. I’m intrigued but slightly undecided. Tom’s counter-point that enslaving humanity for an entire series would be too drastic and would change the viewer connection to the universe could make sense… but for all the apocalyptic bullshit the human race has already been put through by RTD. They were enslaved by Daleks last year, The Sontarans choked everyone and then the sky was set on fire. The human effect of these is just completely ignored. (I don’t accept that as a commentary on the human race surviving disaster by emotional disengagement; it’s just bad writing.)

An extended period of human suffering and oppression would still get resolved in the series finale, and it would be a much bigger deal, far more affecting, than fixing 15 minutes of mild peril and special effects. I think that in such a scenario you can still tell stories about the human condition, fight and spirit. It doesn’t have to be the kind of slavery scenario where everyone is being herded around with whips; humans could still exercise some individual ‘freedom’ within the bounds of their Time Lord dictatorship. That leaves room for human stories that don’t have to end with Absolute Victory, they can show small success in rebellion, they can be much more personal and human than any Earth-set apocalypse story of the past four seasons.

However, the specifics of James’ story are actually less important than the required development of the writing style.

What James’ describes is an episodic plot, with actual consequences. A bigger story, an arc with a clear but not imminent end, the Big Bad defined and in action before the trailer for the finale episode. An end to retarded ‘press this button and everything’s back to normal’ cliffhangers.

I would like Doctor Who to mature and take on a bigger story. The problem with big arcs and consequences carrying over multiple episodes is that it can be harder to pick up if you miss an episode. Is that still a suitable structure for tea-time entertainment? Maybe? But, a more ambitious plot doesn’t stop you making Monster of The Week episodes (that was never a problem for Buffy), so I don’t see a problem, especially given an already dedicated fan-base. Plus, the benefits to the writing and character development seem a very worthwhile counter-benefit.

Was it the lack of consequence in RTD’s writing that made his run so frustrating for me? Is it just his failure to embrace interesting canon in place of making his own shit up? Is it the abysmal dialogue? The incessant melodrama? The ‘because I say so’ approach to character development? Is it just that every year the music in the Christmas special is too loud? It’s all of these things and more. I regard RTD with scorn; a burden. It’s infuriating to consider so many wasted opportunities to do greater things with Doctor Who.

So Russell T Davies revived Doctor Who and made it a commercial success. Great. I will still offer begrudging thanks for that. But I will not issue a free pass for churning out half baked dross like this finale over and over and over again. For me, David Tennant’s Ten will be remembered with affection in spite of RTD’s writing, not because of it. I hope that Stephan Moffet has the confidence and talent to do better.

Comments to James’ original entry, please.

I have proceeded to take the idea entirely too far, and so here are some Rules for International Trading in Settlers of Catan.

It’s the very first draft, it’s certainly incomplete, and there are sections where different mechanics are documented that could be applied. However, I think it should be playable in some combination, so if you’re so inclined, or just wish to offer further input, here it is.

(At some point in the near future this text will move to my wiki to be revised.)

The Scenario is this: You have multiple simultaneous games of Catan running at the same (presumably quite large) table. The games are operating independently, but the players of course interact socially.

This idea is to add a practical, fun mechanism to the game of Catan to allow interaction within the game; namely trade. This is non-trivial.

Requirements

• Two (or, I guess more) simultaneous games of Settlers of Catan
• The Seafarers add on.

It is not be necessary for both games to use Seafarers; in non-expanded games the Ships and Chits can be shared between players of both games, and only used to establish International Trade (see below.) Alternatively, Rules for International Trade can be added to simultaneous games of Seafarers.

Aims

• Allow players in one game to trade resources with players of another, independently operating game.
• Reward players for international trading
• Do not obsolete local trading.
• Do not introduce an overly unbalancing aspect whereby a player could bypass their entire local community and only trade with other nations.
• Don’t break the game in any other way.

General Gameplay Mechanic

The idea is this: The two games are played side by side, with boards aligned. Between the boards is a section of water (from the Seafarers expansion). The exact distance between the boards would have to balance the cost/reward of establishing international trade.

At any point in the game, a player from one game may build a Trade Route (using Ships, from Seafarers; costing 1 sheep + 1 wood per ship) from a coastal Settlement on their Catan island and connect to a corresponding coastal Settlement of another player on the other island. This establishes an International Trade Route.

Once a trade route is established, trading between the two boards may occur.

The only economic transaction enabled is trading. Local economic mechanics (such as rolling a 7 and losing cards; similar to a recession) remain localized events.

Establishing International Trade Routes

• Player builds ships from one island to the other
• Distance should be non-trivial (maybe 5 ships?)
• Players from one Island may not build on another game’s Island (not least because of colour conflicts between the playing pieces)
• Two players, one from each island, may establish an International Trade Route together by both building shipping lanes extending from their Island and having them meet at some point in the sea.
• There is no restriction on the number of International Trade Routes that may be established.

Upon establishing an International Trade Route, the player establishing the trade route should receive a reward of One Victory Point. This is represented by a Catan Chit (again, from the Seafarers set.)

At this point, I have varying ideas about the mechanics of trading. Either:

1. International Trading is only available to the player that established the route, with any player in the Other Island.
2. International Trading is only available to the specific players whose Settlements are connected by the route.
3. International Trading is available to all players of both games as soon as the first International Trade Route is established.
4. International Trading is available to all players of both games as soon as the first International Trade Route is established, but trades may only be initiated in the direction of the Trade Route (e.g. Where Island A connects to Island B, players of Island A may initiate trades with players of Island B, but players of Island B may not initiate trades with Island A until a player from Island B has himself build an International Trade Route to Island A.)

Aside: What if Island A is connected to Island B is connected to Island C? (Answer: Probably nothing.)

If a mechanism is chosen whereby only the owner of the route may trade, that is benefit enough in itself. But, that may limit the feature, and it’s unreasonable to expect all eight (or twelve) players to establish separate International Trade Routes given the amount of coast available.

If the establishment of Trade is shared between all players, then there are two options for rewarding the building of the Route:

• The initial Victory Point represents the Nation of Catan ‘purchasing’ the trade route from the Player; after this purchase, the route serves all Players equally.
• The initial Victory Point is a bonus, but an additional mechanism exists for the Owner of the Route to be rewarded when other Players use his Route for Trade; a Tax.

Tax is interesting. It would have to be balanced carefully, else other players would be discouraged from International Trade except in cases of desperation, and the Route Owner may accelerate to Victory by only trading with the Another Island. On the other hand, where there are two competing Trade Routes, Players would be able to choose which Route to trade over based of the favourability of the Owners.

Tax would take the form of receiving a bonus for every trade that occurs (either per trade, or more generously per resource traded.) That would be indicated by accruing Catan Chits… which in turn would drastically devalue them. Where normally a Chit corresponds to a whole Victory Point, rewarding one Chit-per-Trade would be incorrect. You would end up with a Chit:Point ration of perhaps 5:1, or 7:1. That might affect the use of Chits elsewhere in the scenario, or simply require more Chits than the game provides.

Performing Trades

Local trading normally occurs spontaneously after the dice roll and may be mixed with any amount of building. In larger games there may also be Special Building Phases to allow construction between turns. As per the trade restriction rule for regular Special Building Phase, Players may not Trade Internationally between turns.

In a nutshell, an International Trade operates exactly like a local Trade, but the Player initiating the trade may extend his request/offer to players of Another Island connected by an International Trade Route.

Since the two games of Catan are operating asynchronously, some structure needs to be in place to specify when international trade can occur. These are the current possibilities:

1. An explicit International Trading action, occurring immediately after the dice roll. The two active players (that is, the two players whose turn it is on each separate Island) may take it turns to initiate trades involving all players on both Islands. As per local Trading, only the active player may initiate trades (though other players may approach them socially.)

** The downside here is that you require synchronised dice rolls to act on.

** If the rolls aren’t synchronised, then International Trading is skipped for that turn.

** That may end up being unfair on some players who miss out on trades. Or, it may balance out over multiple rounds with different players missing International Trade windows at different times.

1. A dedicated International Trading Round — performed once per turn cycle. Rather than relying on synchronisation of individual turns, trigger an International Trading round once per turn cycle; that is, after every player has taken a turn, invoke International Trading before starting the next cycle.

** When one game reaches the end of a full cycle of turns, if any of the players on that Island wish to trade, they may wait for the end of the current turn on the Other Island and then proceed with a full round of trading.

** A ‘round of trading’ would mean that each player, starting with the first player on the island that invoked the International Trading Round gets to invoke a set of trades with all other players in all games. Each player in turn (clockwise around the table?) may offer/request a trade.

I think I favour the former ‘trade if possible’ scenario on the assumption that enough opportunities will come up.

Disrupting International Trade Routes

International Trade may be temporaily blocked by placing The Pirate adjacent to a Trade Route. Whilst the pirate is blocking an International Trade Route, players may not trade resources with players on the Other Island.

In a full Seafarers expanded game, each board will have its own Pirate, operated by its home Island only. Rules for moving the Pirate operate as normal; the local Pirate may be moved instead of the local Robber, and may be moved to block or unblock the International Trade Route at the independent whim of the Players of each Island.

Aside: Politically, an entire Island could choose to unite for Protectionism and maintain a block on International Trade (requiring a second route to be built out of the range of a single Pirate to overcome the policy.)

In a game where the Seafarers set is shared between games for the purpose of enabling International Trade only, either Island may use a local Knight or local 7-roll to move the Pirate to block an International Trade Route or to move the Pirate away to restore International Trade.

However, in games where the Seafarers set is shared for International Trade only, moving the global Pirate is an optional action additional to moving the local Robber; this is to prevent it being used as a way to avoid moving the Robber from a particularly effective hex on the local Island.

International Waters/Ungoverned Territory

A further modification of this, to add further incentive for building the initial shipping route between Islands, would be to add extra land mass (probably Gold) between the two Islands. Players from both games would be allowed to settle on these Gold isles, and they would behave the same as Gold in Seafarers—when the hex is rolled, each Player with a Settlement gets to choose which Resource they receive.

2009 was a mostly stable year. I’m in the same place (San Francisco), in the same job (working for YDN is the longest time I’ve spent on a single team in my nearly three years at Yahoo), and my friends in the States and back home have been that way for a while now. I ended the year feeling that things were ordinary, and actually recounting it to friends in England felt really quite dull. I hadn’t seen most of them in 12 months, yet little is really going on in my life.

This time last year I’d just been laid off, rehired, and confronted the very real possibility of deportation. Given that, I’m happy to be reflecting on an uneventful year. The Christmas break has me rested, and with nothing big carrying over from 2009, I fly back to California next week knowing that I’m not returning to chaos, but a stable, sane foundation.

In the Bay Area there’s a certain pressure to be doing. There is an intensity of talented, inspirational people around you. They’re always thinking, building, hacking, creating, always doing something. It’s infectious, and at various points in 2009 I have felt quite desperate with myself for not being part of that. A few small, never-emergent failures—they may yet be resurrected one day—but never really pulling anything together. Ideas are free flowing; that’s a good thing for sure, but I look back and realize that whenever I had the urge to produce something, I had too much in the air already.

The temptation is to respond to this clarity and stability with ferocious intent and planning. I’m resisting. There are a number of interesting new things I want to build. There are existing commitments to microformats.org that could always use more time. There are a couple of old projects I’m curious to revive. I’m resisting because I don’t know which of those various efforts is most important for my time right now. I’m committing to nothing, because committing to doing anything still finds a way to take up my time, producing nothing.

Picking any project is not the game. Picking the right project at the right time is the game. Once I’m back home, back in my routine, I’ll know what I really need to build. And I’ll build it because I can, and because I’ve left myself the space to do so. I’ve learned, slowly and unproductively, how to keep myself in the right state of mind to produce interesting things. A bold declaration of vapourware isn’t it. When I do start a new project, a repository will appear on github.

On a different tack, I do have one new project. I’ve elected to participate in Anton’s Project52. It’s a writing challenge; specifically a blogging challenge, but we’re all just writing now, aren’t we? Personal publishing fascinates me. It’s turned out to be a far more subtle, volatile experience than I ever expected from that first installation of my b2evolution blog in July 2004. I’ve been through different balances of personal verses impersonal, articles verses links, as well as changing expectations of the content I want to write. My opinions on how others should interact with my content, and how I should interact with theirs has changed over time too: Comments! No comments! Some comments!

In 2009, the blog on this domain—benward.me (née ben-ward.co.uk)—received six (count them) posts. All were long, and all I’m still very happy with. I drove myself into this low-output hole by imposing an expectation on myself that all my posts had to be huge, grand articles. I still want to write like that; I still want to tick the boxes of whatever I think is ‘proper’ writing at the time. But this year I found new outlets: A wiki, and Tumblr.

Tumblr is an interesting, odd beast. On the surface it’s a blog host. That’s what drew me to it. It was a blogging platform with a super-simple, clean interface. Their bookmarket for quoting articles, photographs and video was outstanding, and it encouraged me to just link to things, or embed video. It was quick-fire and having been thrown off Pownce, I learned to love it. Last year, whilst I wrote only six article-length posts over here, I posted 494 items to Tumblr. Things that interested me, spontaneous thoughts, quotes and comment. In the end I wrote a lot.

The thing is, though Tumblr is described as a ‘micro’ blogging service, with its lack of comments, incestuous linking strategies and lightweight follower/subscription model, the ‘micro’ is really only relative to people like me, who through their own fault turned blogging into something big. At the origin of the permalink, Jason Kottke wrote a single paragraph, the post didn’t even have a title. Tumblr isn’t microblogging; it’s blogging.

Whilst I love Tumblr; for helping me relearn to write, for connecting me with people, I also have some discomfort. I dislike that the data is locked away behind a very limited API. I dislike the ‘reblogging’ model of conversation (and have been reading Tumblr content in a regular feed reader for a while now), but mostly, I found that the separation I made between this site and my Tumblr blog started to break down.

I’d quote something, and start annotating a response. Suddenly that response was three, four, ten paragraphs long. Suddenly I’d written an article, and it was posted on the ‘wrong’ blog. When I started, I thought a separation between long form writing and short form writing was obvious, but in practice I seem to spontaneously morph between them. Keeping two separate sites is therefore wrong for me.

I’m going to merge my content back together (on this domain), but I’ll put tools in place to ensure I’m participating in the Tumblr community as well; I like it, I just need better control of my publishing. The difference between long-form and short-form content will be left to spontaneity, which was probably always the best way to do it anyway. The merge of old posts will happen soon, and I’ll expose more granular feeds to accommodate anyone that only wants six posts a year.

My wiki isn’t public yet, but it does exist. Along with rediscovering the joy of posting short stuff, I’ve acted on my old post Practical Publishing ; the separation between publishing chronological content (where accuracy and opinion is rooted in time), and revised content (where you aim to keep a document current as time goes on.) The wiki software I’m running needs a bit of work to roll it into my site, and to expose relevant major edits to chronological consumers, but it’s getting there. I’m writing on it, even though you can’t see it yet.

In shaking up my writing tools, Project52 makes a lot of sense for me. Merging it all back together, there will be short posts and long posts. There will be wiki pages of documentation and state-of-the-art. By producing (at least) one article of original content each week (either a post, or a wiki page), I hope to find a good balance to my output, and kick myself into finished some long unfinished draft articles, too. In my tools I’m embracing flexibility again, so I hope that will leave me free to write. I’m excited about it.

Happy new year. I hope it treats you well.

It doesn’t look good for Flash. But Flash isn’t going to die.

Firstly, consider some background. Flash has been around for a very long time, providing a platform for games, vector graphics, animation and media playback. It offers massive market penetration, and of course there is only one Flash player, so it purports to offer a consistent experience across platforms.

However, Flash comes with a number of significant downsides. Firstly, that ‘consistent experience across platforms’ might also be written as ‘inconsistent experiences on every platform’. By taking most of its user interface conventions from Microsoft Windows, the experience of using Flash on a Mac is not the same as using any other app in OSX using native controls. Simple stuff like the behaviour of keyboard chords in text entry controls are different, the text selection colour doesn’t match the system selection; low level stuff.

Similarly, the Flash plug-in itself is cited by Google and Apple as being a massive cause of crashes in their browsers, crashes that the browser maker gets the blame for, but is really caused by Flash. In Google Chrome and in OSX Snow Leopard, plug-ins are sandboxed away from the main browser process. Mozilla lists Flash as the biggest causes of Firefox crashes on Linux.

To top it off, for all the claims of ‘Accessible Flash’ over the past year, Flash content is still only accessible on Windows; on Mac OSX, Flash has no integration with VoiceOver. Screen readers are only supported on Microsoft Windows through the MSAA API.

The iPhone shipped two years ago without Flash support. At the time, some said it was a ‘missing feature’. 24 months later, the iPhone seems to be doing just fine without Flash, and users seem very happy. Adobe have stopped making vapourous comments about having ‘Flash for iPhone’ waiting in the wings.

There are two distinct threads to a Flash vs. HTML 5 discussion. Those are ‘Features’ and ‘Philosophy’. Let’s tackle them separately.

Features

HTML 5 is gaining mind-share because of a handful of key new features that it offers: <video>, <audio> and <canvas>.

The first two are quite self-explanatory, they are new elements dedicated to providing video and audio media directly in the browser, and provides a DOM API for controlling the media from JavaScript. Note, the idea is that the media is played back directly by the browser, not through a plug-in like Quicktime or Windows Media Player (which is how video used to work, before Flash).

This affects Flash because over the past few years, perfectly timed with the rise in available bandwidth to stream audio and video, it provided a solution better than the Quicktime/Windows Media/RealPlayer mangle that came before it. Before, to embed video in pages you needed to provide multiple codecs, depend on bespoke media player UI appearing in your page (all of which was different sizes, and so would break your layout), and half the time your visitors had the wrong version of the plug-in anyway.

Flash stepped in with a solution: Support for more platforms than any one of the other bespoke players, and you could design your own playback UI around it, too.

Flash won video from under the feet of Apple, Microsoft and Real by building something that was better, and bypassing their squabbling over codecs.

But, it’s just a better, bespoke solution. It’s still vendor dependent. Flash provided the use-case for ‘embedding video with author-defined playback controls’. The purpose of standardisation is to take that feature and define it, such that anyone can implement it. From there, comes video and audio in HTML5.

Flash also provides vector drawing tools. It’s another useful use-case (graphing, interactive charts, etc.) Again, the standardisation process for HTML is about taking the use cases from real content on the web and defining it so many people can implement it. canvas (via. Apple) is the implementation for that.

Three major pieces of functionality. Putting them natively into the browser responds to the needs of web developers. That’s what the standard is for. Does this mean that HTML 5 ‘kills’ Flash because previously Flash-only functionality is now native? No. But it means that those major use cases no-longer require Flash. There’s plenty of other, less trivial functionality that Flash supports for which widespread demand does not exist. But of course really common features of web pages are going to be supported in Open Web technology.

Additionally, you may cite sifr — using custom typefaces — as a use-case for Flash. That falls outside of HTML5, but is covered by an increasingly well supported CSS3 feature, @font-face.

Philosophy

I’ve avoided linking Jon Dowdell here as a major source because although he titled his post ‘Adobe on HTML5’, his blog also states that his opinions do not represent his company. His post is representative of Adobe’s general philosophy toward the web, though.

As far as Adobe are concerned, Flash is part of the web. It’s not just an optional, bolt-on plug-in for proprietary content. To Adobe, Flash is as much a part of the web as JavaScript or CSS. They regard is as a legitimate part of the stack.

“the “HTML5” publicity helps marginalize those few who still argue that images, animation, audio/video and rich interactivity have no place on the web. Flash will be able to deliver on those heightened expectations, regardless of what each separate browser engine does.”

The second part of Adobe’s philosophy is that consistent support for functionality on the web is non-negotiable.

“we really do need the ability to predictably deploy advanced capability across a range of device brands and browser brands”

This philosophy is wrong. One: Flash is not part of the web. The web is the Open Web and anything closed and proprietary is just riding on its back. I don’t mean that bespoke plug-ins are unwelcome or even ‘wrong’; they provide all sorts of useful functionality. I do mean that if you are a single-vendor creator of a proprietary, patent and license encumbered, undocumented, closed-source plug-in then you have no claim to be part of the infrastructure of the web. The infrastructure, from TCP/IP upwards, is open.

The consistent support aspect also flies in the face of techniques used with every part of the Open Web stack: Graceful degradation, progressive enhancement, and the fight against the misguided demand for pixel perfection are all battles that have been fought and won since Designing for Web Standards.

The web is about content. Everything above that is dressing (perhaps think of the web as fresh bread, perfectly coated in balsamic vinegar and olive oil). The fact that older browsers cannot render all the features of your page but can still provide the content to users is a feature. It’s the most important feature.

The Flash philosophy is opposite. Flash is about a complete experience (singular). It’s about every detail being precisely bevelled into place for every viewer. The consequence of this approach is that it resists the availability of content. The goal of perfect consistent rendering can only be achieved through a single version of this single vendor’s bespoke plug-in. If you need a feature of Flash 10, Flash 9 users must upgrade to see any of your content, not just the new feature.

The Flash approach is anti-content; anti-web. Adobe present the idea that Flash is a superior offering because the entire suite of features, in one big blob, is a compelling development offering. But the reason to write on the web in the first place is to make content available broadly.

In recent years, through multimedia, fonts and and vector drawing, we’ve seen more and more blocks of content moved into Flash, in the absence of a robust standards-based mechanism. HTML5 redresses that by supporting those use cases. HTML4 supports pictures. HTML5 supports moving pictures. HTML5 supports what people publish on the web.

Fuss

What is the fuss about? HTML5 doesn’t compete with Flash as a product, (you could never produce an implementable specification for so much functionality in one go). It just takes some specific, common use-cases for web content and supports them.

Yet, people on one side are crying for the absolute death of Flash, and clearly some from Adobe are on the defensive to outright dismiss the HTML5 effort.

Critics may be motivated by any number of those negative user-experiences this article opened with, but Flash won’t die. If HTML5 takes away one use-case that Flash fulfils, Adobe Flash will add new features that browsers don’t have. That’s what plug-ins have always done. Flash can and will iterate faster than browsers and can deploy wider all at once. That said, some of those existing use cases — namely video playback — are extremely lucrative for Adobe. Video took Flash from ‘optional’ to ‘essential’ for a certain slice of web content. The video sharing industry is dependent on Flash.

Adobe will lose their exclusive grip on that. But, what did they expect? That a massively profitable industry would tie themselves to a single vendor?

Flash offers only one advantage to video on the web, and I think this one will be genuinely interesting to see turned into a marketed feature. The HTML5 method of embedding video looks like this:

<video src='http://example.org/video/foo.mp4'></video>

There’s the URL to your video file, right there in the HTML source, downloaded in raw form. What can Adobe offer publishers? Two ‘features’ of software that run absolutely counter to the principals of the open web: DRM and obfuscation.

That could be interesting. The survival of Adobe Flash Video online will require them to take the closed, anti-content consequence of Flash’s model, and instead embrace it as a feature for media companies that fear distribution of their content.

Really, I think this whole issue is overblown. Maybe it’s all fuelled by scare-mongering from individuals Adobe, maybe it’s over-eager Open Web evangelists eager to see closed and proprietary scraped from the face of the web. In reality, it’s just the pragmatic, ongoing evolution of the web offering useful new functionality.

I was there for my work with microformats, and as someone who periodically pops up to suggest improving the user experience of OAuth.

People

MySpace, Facebook, Yahoo, Google, OAuth, Portable Contacts, Activity Streams, Open Social and OpenID. All under one roof. Everyone quite keen to move in positive directions. An exciting mix.

It was quite something to spend two days in these surroundings, participating in the conversations that determine what happens next. As is my nature though, perhaps the biggest value came through observation and listening. And to cut to the chase, herein lies my principal learning of Foocamp:

The Open Stack needs a Product

Foo had many, many sessions on the Open Stack; OpenID, OAuth, OpenSocial and Portable Contacts. Some of them concerned with user experience, some with data portability and distributed identity, some with the processes of open development itself. Every single one of those sessions at the very least mentioned Facebook. But more often, the sessions were outright dominated by Facebook Connect.

A session on building a start-up on the Open Stack was turned into a discussion about Facebook’s product, and what developers wanted from it.

Note the emphasis on Facebook’s product. The way in which we classify the technology of the open, social web and compare it to Facebook Connect is massively flawed.

Get this clear: Facebook Connect, from inception through API through user experience, is a single, self-contained, beautifully packaged product for developers. And it’s awesome. Facebook has the combination of detailed, well maintained user data, a huge user-base and excellent user interface design for the Connect experience. It ticks every box.

Compare to ‘the Open Stack’. There is no product. These are technologies — wonderful technologies — with which you could build something with the functionality of Facebook Connect. But at time of writing, there is no mature offering. The branded products from Yahoo and Google are not as strong as Facebook; they’re less mature in every way. Problematically, though, the tools have a stronger brand than the implementations.

It’s this that causes the Facebook/Open Web comparison to fall down so quickly. The open technologies are right and true. Using the same open auth and identity protocols is a massive win for developers. But what are you actually implementing?

The open stack itself doesn’t contain any data, nor provide any service. It is just the mechanism to provide those services. You don’t solve anything by ‘integrating OAuth’. OAuth isn’t a service. The publicity has to shift to actual service providers, where the end users are involved. Because really, it’s touching those end users that drives developers, not beautiful snowflakes.

We’re all imagining a world where you can implement OpenID+OAuth+PoCo and seamlessly integrate with Google, Yahoo! and any other social network using the same code. But that doesn’t exist yet. Only the foundations of it exist. And without the data provision from actual products, there’s no implementation to focus the open stack discussion on.

Not broken

Whilst it all sounds a bit bleak, nothing here is broken. Facebook has a massive head-start in the marketplace. Yahoo, Google, MySpace et al are playing catch-up in terms of the APIs and the user experiences of their own sites. Is Yahoo! Updates as rich an experience as Facebook? Not yet, no. There’s work happening everywhere to compete, across all aspects of all services. As such, of course Facebook is the more compelling option this year; it’s obvious that’s the case.

Further, as evidenced by Facebook’s Open Stream API launch last week, their strategy has been formidably well planned. Over the past twelve months they’ve been hit with sticks by openness advocates for being locked away in their walled garden, but their priorities have been elsewhere. They’ve been building a rock solid foundation that, once in place (now), they can start to open up and offer good data from the start. They have the luxury of the market lead, and they can use that to release better, more complete services. That’s their reward for being first, and they’ve earned it.

So, the feeling I come out with is that we should stop thinking about Facebook in the context of open standards (except where they implement them, of course). It’s a broken comparison. It’s hard, because the competitors have everything in the air at once, but it’s down to them over the coming months to turn their adolescent, open-powered APIs into compelling products. The part OAuth plays in this is just to continue becoming as transparent to users as possible.

It’s not the job of OAuth and OpenID as part of this ‘Open Stack’ to take on Facebook in mindshare. The roles of these APIs (PoCo, Open Social and Activity Streams included) is to be expected and taken for granted in any new implementation. These are the bricks on which houses are built. But people don’t buy bricks, and so our eyes need to focus on the products of this work.

The other point to stress here is that whilst the open stack needs stronger implementations, they don’t have to be ‘NotFacebook Connect’. That’s only one use for these standards. The big Open Web offering could be somewhat different. Better, even.

Be excited. The struggle of Open standards vs. Facebook is a fallacy, they’re just efforts a little out of sync. This year, with maturing, big products powered by open technologies, we’ll see things built that extend beyond the achievements of Facebook’s walled garden age. MySpace, Google, Yahoo! et al are all moving together toward something quite special. Developers will be able to take on exciting new, provider agnostic apps with this technology. Just accept that the second generation of competitors need a little time and encouragement to build out.

Oh, and don’t be surprised to see Facebook active in all this, too. In the end, they’ll be as open as anyone else.

Microformats.org is an interesting beast to work for. An informally arranged organisation of volunteers, overseeing a broad array of subject areas and points of interaction. 2008 was my first full year of administrative involvement with the group, for what value of ‘administration’ there really is.

This post is on my personal blog because there is no official line of policy at microformats.org. What I write here is just personal intent and what we achieve in the next twelve months is down to shared passions and collaboration, not the will of one person.

There are shared priorities, of course. The past few months have seen a surge of work on the awkwardly named value excerption ; a mark-up pattern and parsing rule derived from hCard. Honestly, I didn’t know ‘excerption’ was a real word until I started leading the work on this. Thankfully, naming is not as important as a good spec.

Basically, value-excerption in hCard got implemented in parsers globally, so we’re trying spec it more fully to reflect that. It’s a pattern for structuring data values, so in the process we can extend it do something to offer solutions to some long standing accessibility and localisation complaints. The work is sporadic; two weeks here, a month off there. That’s just how it happens. Being absent from Yahoo! this last month has helped me pull it together into a massive public test effort.

My other big task in 2008 was redesigning the microformats wiki, bringing it into line with the look and feel of microformats.org, adapting Dan Cederholm’s still-lovely design. It’s a piece of work I’m proud of, and besides being able to junk vast quantities of MediaWiki’s questionable and bloated default mark-up, it of allowed me to put microformats into the wiki mark-up itself: Each page is now an hAtom entry, with an hCalendar event for the last-modification date of the page.

This Year

I don’t care to dissect last year too heavily. It’s this year I’m excited about. There’s work coming to completion, there’s ongoing work that’s nearly ready to break cover, ongoing infrastructure improvements brewing and a desire to see a big step up in microformats toolkits for developers.

1. First up, I want to see the value-excerption work seen out within the next couple of months. Testing is going really well right now; it’s an effort beyond the scale of anything else we’ve done before. Knowing the accessibility and localisation issues we’re trying to overcome, it’s vital that we get it right. We can’t afford to push something that doesn’t solve the problems and complaints of authors as well as we can. I’m taking suggestions for a beer or red wine magnificent enough to open when we call this one ‘done’.

2. Secondly, we’ve built up a number of issues and enhancement requests against the core microformats — hCard and hCalendar. They’re stable, useful and are helping to change the web, but iterating stably is an important step to take as the community and formats mature. Just as HTML5 is not versioned like a piece of software, there won’t be an ‘hCard 2’. This is the web and we won’t be breaking existing pages or forking our specifications; that’s absurd. We will evolve. I would like a period of active editing and hope to see hCard and hCalendar ’Second Edition’ published this year.

3. Recipe and Audio formats. Two new drafts in 2008. Bearing in mind that many popular and quite stable formats like hReview and hAtom are actually still in draft, that’s a very significant step — it takes a lot of research and brainstorming to put together a good draft spec. These subjects have much stronger, stable momentum than some previous microformat proposals have had, so I’m confident they’ll move smoothly. Structured publishing of music and food is highly Relevant To My Interests. I worked with publishing some of the hAudio draft in my previous music round up. I think it’s getting there.

4. I’ve spend some off-time brainstorming on a new effort myself; ‘embed’. No dedicated wiki page yet as I’m still compiling the initial data to get it rolling. There’s nearly enough to push it though; a few more sites to grab examples from to get people thinking. It’s deriving some concepts from the oEmbed format Pownce supported, allowing sites to describe their ‘embed codes’ for reuse around the web. I want to be able to reuse linked content in an activity stream, and deriving embeds from mark-up rather than writing drivers for every site on the net. It would make reblogging the embedded content more graceful, too. More robust use cases coming soon.

5. Microformats have issues, feature requests, bug reports, tasks to do. At present we track them on the wiki along with the specification documents themselves. Personally I find it a nightmare. Tracking and triaging issues through versioned documents in various structures is harder and less transparent than I’d like, so fixing it would be nice. The wiki update last year has the facility to hook spec ‘issues’ links up to other systems, and I’m spending some time experimenting. Community feedback needed here, plus considerations to be made regarding self-hosting something like Trac or offloading to an external tool. It could happen quite quickly, since I don’t think there are many sane arguments defending the wiki method; it doesn’t scale.

6. Wiki rewrites. I’m good at writing. I’m too verbose for sure, but I communicate well. I’ve taken great pleasure in applying this to more recent microformats output and I like to think I do a pretty good job of improving the experience of interacting with microformats documentation. Many pages on the wiki aren’t as well written. I don’t mean to criticise other authors, I refer more to the way in which over time important pages like the process and how to play page have been edited and added to so many times that at this point, I fear they’ve become impenetrable to a new visitor, and if they can’t follow the rules and I want to see effort go into reworking those pages to be higher quality documents, more approachable and easier to reference when they need to be enforced.

7. Support transformation efforts. In 2008, I’ve noted a couple of repeat proposals and desires for using microformat specifications in other contexts than HTML. Being in HTML is part of what makes something a microformat, so we’ve had instances of proposed forking. Versions of hAudio exist republished for use in RDFa, there’s an entire page on the microformats wiki called jCard — putting hCard into JSON for interchange. Per-specification duplication is, in my view, wrong. Duplicating specifications leads to fragmentation, confusion, incompatibilities. If people have use cases for transforming a microformat into RDF, or JSON, or anything at all, the core spec needs be the same. What we need documented are consistant rules for transforming HTML into any of those other languages. ‘Transforming microformats into JSON’ could be a single wiki reference page for all current and future microformats, explaining how to convert different microformat patterns into JSON. Not a ‘jCard’ and a ‘jCalendar’ and ‘jAtom’, with an ‘rRecipe’ for RDF and xResume for raw XML. Just one set of rules to handle the transformations that are useful. Within that, defining the parsed object structures of the microformats goes most of the way to serialising into another language, and that’s a job for parser authors to settle on the best way to turn microformats into objects consistently.

All of the above is a reasonable ask, I think. It’s ongoing progress in an evolutionary approach in development of standards and infrastructure. My big wish for the year is perhaps a bigger step.

The next level: API Kits

Consider existing services: Google Contacts, Yahoo! Address Book. Standalone data providers, whose APIs offer high level methods to access the contacts held within.

A popular use case for hCard and XFN is contribution to the distributed social ecosystem. Data about people and social relationships is published all across the web, but consuming it is prohibitively hard for most developers.

Whereas someone developing for the YAB or Google data stores can download wrappers around the high-level methods those APIs offer, consuming microformats remains at the parsing level. There’s no Person::getFriends('http://ben-ward.co.uk')-like method returning an array of vcard objects. If we’re serious about evangelising consumption of hcard in social networks. We need high level, task centic toolkits, not just raw parsers.

A higher level means providing solutions to common problems and use-cases, rather than a solution to ‘microformats’. A ‘Distributed Contacts API’ that follows XFN links between hCards, handles crawling pages and/or interaction with the Google Social Graph API. Ultimately, you make one call to a high level function and it just happens. I want to see microformat-based tools that boom!.

I think XFN and hCard offer the two most appealing toolkits: Distributed user profiles (‘Distributed Profile API’) to the profiles information described with hCard, linked with rel='me' and the aforementioned ‘Distributed Contacts API’ for obtaining the profiles of other people you link to as friends.

I’m thinking that methods like these are needed to make it trivial for social applications to start consuming microformats more ambitiously:

Person::getProfile('http://ben-ward.co.uk', callback)

Get all profile info for the person at ben-ward.co.uk, and fire the provided callback function when completed (you need callbacks for all of this since it’s both asynchronous handle and crawling the web is going to take a little time).

Person::getConnections(
    'http://ben-ward.co.uk',
    [ 'friend', 'acquaintance' ],
    callback );

Return the profiles of all the people connected to the person at ben-ward.co.uk connected with XFN ‘friend’ or ‘acquaintance’ relationships.

Methods like these make it simple for developers to start using the huge wealth of published microformatted data to enhance and power their social applications. Right now, getting to those methods is a lot of labour. We need to build it once, and we need to do it in the open. I would love to be in a position this year that we can evangelise microformat consumption with as much strength as we do microformat publishing. OAuth and OpenID has a lot of evangelic traction because libraries exist to implement it in many languages; ‘You should use OAuth, here’s some code you can use!’ is rather more convincing than ‘You should consume microformats! Err…’.

We can’t legitimately push sites to consume hCard with an effort barrier so high. If a stable API kit exists that a developer can just drop in to their codebase — like the wrappers for OAuth — then we can make a strong case to see the open web realise a little more of its potential. I’ve written about the dream of a distributed, microformatted web before at Digital Web. I want to see if become real, rather than just ‘possible’.

You can see this sort of thing in practice already on a tiny but beautiful scale. If you have an OpenID, and an hCard at that same URL, go sign up on User Voice. You’ll auth using OpenID, and when you bounce back to complete your profile, User Voice already knows your name and email address. That information comes not from attribute exchange through OpenID (which the Yahoo OpenID provider doesn’t support), but through reading the hCard from my URL. I wondered for a moment what was going on. And then I just smiled. It’s the future, now. I want to see that user experience available at low cost to every developer.

So, there’s my forward looking. I see the above as pretty concrete ideas. Of course, there’s far too much to lead myself. So, who knows. I hope that others in the community will feel inspired and that we’ll see this kind of work happen. Just as much, I hope to see the visions of others. This community is diverse. I think I’m one of the most passionate about the actual core of the community (perhaps more so than any particular microformat itself), but there wealth of thoughts and ideas amongst all our membership. If you’re one of those, I invite you to write up your vision for the year.

Microformats are a huge deal. Where do we go next? More formats? Reinforcing what we’ve got? Appealing to new groups of publishers and developers that haven’t heard of us yet?

If there’s enough posts along these lines I’ll link them all together on the microformats.org blog.

On paper, it’s been pretty good. I attended South by Southwest Music for the first time, spent most of the year living in East London with a music junkie Last.FM-ite and spending great times socialising with David Emery of Beggars Group, so music exposure may have been greater and more eclectic than any previous year. I come out of it not entirely convinced, and my mostly unordered pick of the records I enjoyed the most almost seem predictable written down. Regardless, onward.

Albums

A brilliant, full album is still my favourite way to consume music. Despite listing more to Last.FM and Hype Machine this year than last, despite iTunes adding a really good ‘Genius’ playlist generator feature and despite dropping portable capacity down to 8GB by trading my iPod for an iPhone, I still adore the experience and coherence of a good album.

Antidotes by Foals is my favourite record of the year. It’s just great. I appreciate some early adopters were a little put off by the absence of Mathletics, and the unexpected introduction of a brass section, but the songcraft just clicks everywhere for me. The tunes are great, the riffs get you moving, the switches in pace and style midway through songs is just perfect. Two Steps, Twice is my standout favourite track. It builds up, slowly, pacing perfect and eventually explodes in a synthed up crescendo of energy and tune. It’s just the best thing I’ve heard all year. That said, the preceding Heavy Water, whilst initially a bit of a weaker song, pulls of a great dance explosion at the end as well. It’s a song that just transforms in ways you don’t expect. Whilst Battles brought math-rock out of the shadows earlier, Oxford’s Foals have made something that’s probably more accessible, but no less classy.

Elsewhere, Fleet Foxes maintained the Americana revival apace, with gorgeous earthy, folky songs. Lightspeed Champion’s ‘Falling Off The Lavender Bridge’ record (with Emmy the Great backing) is full of wonderful folk-pop songs, Cut Copy’s ‘In Ghost Colours’ makes wonderful late night music with its combination of lively dance, atmospheric keyboards and sprinkling of “Oh, it’s a bit like New Order, isn’t it?”; underlining why playing full length records won’t go away. Plus Gruff Rhys and Boom Bip formed Neon Neon, rushed back in time to salvage the electronic bits of the 80’s and won it all with ‘Stainless Style’.

Elbow’s evolution continues to astound. I love this band dearly, every record they’ve ever released has touched me in some way and every one has glorious moments that I’ll go back and play forever. I don’t know if any one song on ‘The Seldom Seen Kid’ is better than anything they’ve done previous, but the record as a whole is somehow a more coherent, more complete offering than what came before. It’s inevitably more mature; less of a departure in sound from ‘Leaders of the Free World’ than it could have been, but over a handful of listens, from start to finish it draws you in. Richard Hawley provides vocals on ‘The Fix’, and dominates the song. It’s a wonderful stand out moment, though you wonder if it displaces Guy Garvey’s own distinctive vocal too much. Until it rolls into ’Some Riot’, a piece of music of beauty and delicacy and suddenly Garvey’s voice is in its element and you… just… float. Perhaps ‘Some Riot’ is the one song that’s better than anything else they’ve done.

I still regret not saying ‘Hello’ to Guy at London Euston railway station a few years ago, though I still don’t know what I’d say to him now. After my drunken blathering to Moby at SXSW, maybe it’s best I stay away from respectable musicians.

Finally, my dabbling in the physical world of vinyl is growing. I bought a gorgeous Pro-Ject Debut III (in red). A beautifully squared off slap of wood, with minimal controls and, as best I can tell, great sound. I don’t care how near the snob/hipster line I stray, the warm, full sound is awesome and appreciable even on my aged student hi-fi separates. On that, I must mention something completely out of time; The Beta Band. Their first, self titled album which for some reason I own only on vinyl. It’s just great. It’s exactly where pop music rightfully ends up in the late 1990s; assuming the same progression and daring evolution of the preceding forty years. They were unique, The Beta Band, and they are missed.

Songs

Maybe it’s because I pour all my energy into album reviews, but when I get down to individual songs I feel more inclined toward spewing out a quickfire list than anything more substantial. I can’t find much fault with that, so, the songs that made me happy in 2008:

‘The Hill, The View, and the Lights’ by Cajun Dance Party, ‘Two Steps, Twice’ by Foals, ‘Midnight Surprise’ and ‘Dry Lips’ by Lightspeed Champion, ‘Battle Royale’ by Does It Offend You, Yeah?, ‘Your Protector’ by Fleet Foxes, ‘Ghosts’ by Ladytron, ‘Belfast’ by Neon Neon, ‘Lights Out For Darker Skies’ by British Sea Power, ‘Kriss Kross’ by Guillemots from their otherwise disappointing ‘Red’ album, ‘Salute Your Solution’ from Raconteurs ‘Consolers of the Lonely’, ‘Hot Cakes’ by El Ten Eleven — and his cover of ‘Paranoid Android’ is stellar too.

Special mentions go to ‘Talking Backwards’ by Fanfarlo, a band I desperately need to acquire more music of. ‘Talking Backwards’ is one of my favourite pop songs of the whole year. And whilst most of the songs here are linked to Last.FM in some way, you should absolutely follow this one and play the whole song. It’s sublime.

And then, there’s Florence and the Machine.

No album, unsigned until rather recently. I am somewhat obsessed with Florence Welch. But I’m shameless about it. Her two 7” singles this year — ‘Kiss with a Fist’ and ‘Dog Days Are Over’ have just been sublime. Pop music with great tunes, great refrain, darkly humorous lyrics. I could ask nothing more than to have it performed live in my living room. Unless that’s getting creepy, in which case I’ll reluctantly step away. Her performance at SXSW was awesome and had me following her powerful, bluesy voice ever since. Er, more gushing about her follows below. Again with the emphasis on listening to these. Or show up at a party in my apartment and I’ll inevitably play them to you ad nauseam.

For everything else this year, I’ll lazily be referring you to my Last.FM loved tracks and Hype Machine obsessions lists.

Live

Live music was quite special this year. I attended South by Southwest in March, staying on past the usual interactive geek-up and through a gruelling second week of intense music. It was an awesome exercise in discovering bands I’d only heard the name of at that point — Lightspeed Champion, MGMT, Los Campesinos! and so forth. The only accidental discovery was Florence and the Machine, who was stunningly good and did quite curious things to my heart rate with her voice alone.

Later came The Great Escape in Brighton, which bills itself as a British version of SXSW, but by offering rather fewer shows per night, they don’t handle the quantity of attendees so well. There’s hope if they can scale up venues faster than they scale attendees. Saw some good shows, although Lightspeed Champion almost undid all the good from SXSW in one dreadful performance.

The Ting Tings were actually a lot of fun live at SXSW (and again at The Great Escape), but the album kinda slumped off my radar after a few weeks. In still can’t quite believe that after recording the weak, wheezing falsetto on title track ‘We Started Nothing’ someone was actually paid to say ‘Yeah, that’s great!’. I think my subsequent disenchantment was what David intended to refer to as inevitable… although all I heard was him hurling expletives at me for listening to Ting Tings in the first place. I’m sure I’ve got his sentiment nailed down now, though.

I did a good number of shows at Somerset House again. It’s a frankly very expensive way to see less shows than a music festival, but the venue is magnificent illuminated and it was right near the Yahoo! office.

It would be remiss not to mention the biggest event of my live music year. I managed to clock up seeing Radiohead three times; two nights in London’s Victoria Park (all of five minutes walk from my then home), and once more in Golden Gate Park in San Francisco. I missed out on the surprise show at Rough Trade on Brick Lane; oh well. The second night in London stands out as my favourite, but with a repertoire as good as theirs is difficult to fault on any night. The variation night to night keeps it fresh and the experience as the sun sets is just stellar. I’m still to experience anything as mind blowing as tens of thousands of people singing the coda to Karma Police. For a minute there we lost ourselves.

As mentioned earlier, I’ve spent most of the year since South by Southwest absolutely fixated by Florence Welch to a degree bordering on social unacceptability. Even without my mild obsession, Florence and the Machine’s records are catchy, her voice is magnificent, her lyrics darkly comic and together with songs of pure pop brilliance, she offers something beyond any of the more famous London soloists. Like Steve Lamaq, I really can’t figure out what to expect nor what I want from 2009 in terms of broader trends and scenes, but an album from Florence is on the cards, so that’s one thing at least.

I’m looking forward to see what San Francisco offers up in 2009.

OpenID and OAuth are the open, standards based and interoperable editions of this technology, but Yahoo’s deprecated BBAuth and FlickrAuth and others all came before. Also at the tail-end of last year came Facebook Connect, a system whereby websites can piggyback on Facebook profiles for building applications.

For example, take Fire Eagle. It’s a service that stores your location on your behalf, for use by other applications on the web. It uses OAuth to control access to that location; no application can see your location by default. When you visit a site needing your location, it asks Fire Eagle for that information.

Instead providing your Yahoo! username and password to this third party site (which would grant access to your entire Yahoo! account), you are taken to a special page on the Fire Eagle site, click a button to grant specific location permission and then jump back to the original site, which now holds a token to access to your location.

This process means that the site you shared your location with can’t access anything apart from your location (it can’t log into your Yahoo! IM account, for example, or send emails through Yahoo mail). Furthermore, you can log in to Fire Eagle and remove that application any time; you don’t need to change your password to do so.

It’s the future, it’s user empowering, and it’s going to be great. Eventually.

The user experience of this OAuth process — and OpenID alike — has been criticised a bit. Users don’t expect to be moved between different websites, but they are familiar with entering their passwords all over the place. The short ranty version of this article would go like this: If you stop whining and just get on with implementing the OAuth flow, users will get used to it and will be just fine. It’s is usable as-is, so shut up already. But this is the long, constructive version, so:

The user experience of OAuth and OpenID is immature, and can still be massively improved and smoothed out with concerted design effort.

Which brings me to Facebook Connect. Connect is a product as well as a proprietary technology. It’s a packaged and complete offering from Facebook, and as such, comes with a far more complete and polished user experience than the technology-focused, open standards have so far achieved. Polished and mind bogglingly stupid, in places, but, y’know.

Facebook Connect, whilst proprietary and product-specific and therefore irrelevant in the grand scheme of things, has UX that can be applied to OAuth and OpenID flows. If service providers support this, I think user experience gets much better, quickly.

How does Facebook Connect work?

The most common use case for Facebook Connect appears to be commenting on blogs, such as on Gawker sites. Rather than enter your details standalone, or uniquely register with a site, you log into Facebook, and Gawker uses those details instead.

So, you click the shiny ‘Facebook Connect’ button in the comments form, and an overlay appears:

This is the crux of the learning for OAuth. Rather than redirect to Facebook, this granting of permission happens right in the page in an embedded control.

It’s not quite as simple as this, mind. It’s ok that this action occurs in an overlay only because the user is already logged in to Facebook. No exchange of credentials takes place: The overlay is an iframe serving a page from Facebook’s server, so my current login cookie is used and there’s no need for Facebook to ask for my password. A malicious site would gain nothing by spoofing this dialog.

Since writing this article, Facebook have improved the behaviour of Connect. Now, if you are signed in you see an overlay as before, but if not signed in Connect opens a new window, where all usual browser functionality is available. This a huge improvement and fixes the complaints that follow.

Unfortunately, Facebook Connect then screws up. The whole point of delegated auth is that we stop users entering their passwords into third party sites. It has to stop. That means both actually entering their details into third parties, but also interface that gives the impression of giving your password to a third party. When you are not currently logged into Facebook, you instead see this dialog:

Millions of Facebook users, openly encouraged to enter their password into any site that asks. This is wrong. If the user is not already logged into the service, you should be redirected in a more traditional bounce between pages. That way browser-level phishing tools kick in, the URL in the address bar can be manually inspected by the user and, critically, the user is conscious of logging into a different service.

Facebook ranting aside, the first half of their Connect overlay UI would be very useful to enhance the user experience of OAuth and OpenID.

Here’s a hypothetical Fire Eagle app built into Last.FM.

In the current implementation of OAuth, clicking ‘Get Fire Eagle Location’ would redirect you to the Fire Eagle website, and then you’d redirect back again after clicking ‘Confirm’.

Instead, OAuth apps should do this by default:

No redirect, lighter weight UI and more responsive feedback. This, I think, is something that OAuth APIs should support out of box along with their other language wrappers; provide drop-in support.

Now, this behaviour applies for logged in users only. If you’re not logged in to Fire Eagle for any reason, you should still be moved to the separate site as before. We need to stay strict on keeping users spatially aware of where they are entering their passwords, otherwise the whole effort is undermined.

Overlaid OpenID

With one example down, here’s a mock of how Open ID could benefit from the same integrated flow, this time working with Dopplr, since they already support Open ID:

If not logged in to Yahoo, you get a prompt and just as before, are guided to step through the regular, separate-site process to sign in:

Clicking ‘Sign in to Yahoo!’ would take the user to Yahoo’s standalone page.

How to make this happen?

For this to happen, services need to provide support for it; it can’t be done just at the client side. The dialog-sized interfaces for authorising applications or logging into sites need to provided, and they need to support the ‘break out to enter passwords’ flow. But, sites like Fire Eagle already provide a mobile-scale version of the auth page, so further variants are not a major hindrance.

It also needs a JavaScript component to handle the UI side. With a bit of luck, this only needs to be done once and shared between projects.

The core technology behind OAuth and OpenID is pretty robust. Both have major adopters like Yahoo and Google. OpenID has a bit of a bit of a way to go before users need it, perhaps, but regardless, it’s well into the same phase where user experience needs to be a concerted effort, and the status quo needs to be challenged.

Everything in this post is just a small step from what we already have, it’s just smoothing out the edges. Maybe that’s enough, but I suspect there’s a long way to go and a wealth of other ideas out there.

A year ago, I sat in my flat in London, somewhat settled, surrounded by wonderful friends. Sometime over the past twenty-four hours I had got monumentally drunk at Barden’s Boudoir and dancedflailed wildly happily to Soulwax remixes of Klaxons. At some point I would be handed a half-full bottle of vodka by a barman and pour drinks for twenty people in my immediate vicinity and be cheered on like I’d brought home the World Cup.

Life was, mostly, peachy.

Also around that time I had an intriguing conversation across the Atlantic. ‘Have you ever considered working in the States?’ is the executive summary. ‘Nope… but tell me more’ is my abridge response.

It’s difficult to write a retrospective of 2008 because although I didn’t move to the US until August, doing so eclipsed everything else that happened this year. In scale and impact, I mean, not necessarily in importance.

2008 is the year I moved to America. Got offered the chance, knew I couldn’t refuse, took a deep breath, took a lot of risks, and did it.

Moving, especially when it’s at least partially spontaneous, is a rush. So much happens at once that I lost track. So many decisions to make, so many major jobs to do one after another without a break. A new job with new people, new friends to be made.

In a great many ways, I did what I always do, which is to land on my feet and do really well for myself. I don’t like to assume that’s how things will always work out, but it’s become such a recurrence that I should start documenting it more scientifically. That said, at the pace of change, and under the huge rush of emotions and disorientation that comes with moving, I did plenty of things wrong too. My Christmas break came after 139 days in America, and actually, to fly home to England and take stock is exactly what I needed.

The out of control rush had to end, the Yahoo layoffs experience and resultant rush to find new employment did nothing to lower my pulse and so the time away came as a really welcome break.

I’ve come out of it calmer and more stable. I fly back to the US in a few hours, and I think I’m in a good state of mind for starting a new job, and tying up the last few loose ends of 2008.

I ordinarily dismiss the idea of new years resolutions, but the way my experiences have fallen this year I start 2009 feeling unusually resolute.

Some things are both predictable and rather cliché, but also very necessary. Having an operation for appendicitis in April got my weight down to where it should be (note: not a recommendation form of weight loss, plus you can only do it once). Moving to the States has seen me put it all back on, and exercise less. That has to change, else I’ll be a grotesque lardbucket by the time I write the 2009 review.

Elsewhere, the new job is going to let me keep a better work-life balance, since I’ll be commuting from Sunnyvale daily. As such, I want to see my personal projects go live. I have lots to do at microformats.org, I have various wiki-related pieces of work in progress and needing to go live, as well as 33FortyFive, which I’ve been working the concept for for ages now.

I need to track my life better. This year has shown how one really big event can throw off my knowledge of the rest of the year. So starting January first, I’m keeping a retrospective for all my social appointments, so I get a better overview of where my time goes. Maybe I’ll make that public, or anonymized somehow, if it proves interesting.

There’s all manner of small things, and longer term, niggling tasks that I have to get done. Really, it all falls under a renewed determination I’m feeling. It all starts when I land in SFO on Saturday evening.

Let’s see how this works out…

Brickhouse was a wonderful thing, and I fear its brilliance and inner creativity was not understand as it should have been. Unfortunately, whatever the cause of judgement, on December 10th Yahoo elected to close our group and approach the challenges of building new products in different ways.

It’s was crushing for those of us working there on a number of levels. We loved our co-workers, we loved our product (Fire Eagle) and we loved the premise of Brickhouse; an inspiring work environment of ideas and creativity. For the preceding three weeks I’d worked late into the night so that on Tuesday we could launch Friends on Fire. I’m glad to have got it out, rather than it be discarded on an SVN server somewhere.

For me, mourning the intellectual loss of a dream job wasn’t really a priority. My working in the US was entirely tied to working at Yahoo. For me, unemployment would mean prompt deportation, and moving to a new company required a willingness to sponsor a new visa, and would still leave my location and life in limbo for most of 2009.

The reaction from outside is difficult or me describe. Genuinely, a slack jawed gasp provides as good a summary as any. I witnessed colleagues across the world band together to promote my name and those of my friends also departing the Brickhouse. The quantity of direct contact I received through Twitter and email was astounding. The power of the meritorious society that has developed around the web is huge. People trust one another’s recommendations because unlike industries of old, there is no old boys network here. The great people we associate with are people we thrive from. We support and work off one another not because we’re friends, but because we do better in proximately to those who are also talented.

I was taken aback by the response and far from sulk in my situation, I’ve spend the last ten days following up. I think I replied to everyone, but given how bloated my inbox was after just a few hours, I think I should say ‘sorry’ to anyone who hasn’t heard back from me, but mostly ‘*thank you*’. Thank you for your support through this. Whether who knew me and knew my skills, or just wanted to talk with me to see if something could be worked out, every query and message has helped sustain me through this.

The Lay Off

Being laid off in America is strange on all manner of levels. It’s just like in films. You get told what’s happening, and handed a box. You clear your desk then and there, hand over your computer and security pass. Security guards lurk around at the door, purportedly to protect the ones who still have jobs from being attacked by the vengeful disgruntled unemployed. It’s a harsh, blunt, heartless process. It’s offensive on so many levels, disrespectful to you as a person, disrespectful of the work you care about and that, despite the situation, you might care to have properly passed on to a co-worker. Instead, projects are dropped where they stand, no knowledge transfer takes place, weeks maybe months of work is just discarded.

Apparently it’s normal like this in the US. In England you get given an end date, you work up until it, you pass on your work to someone else. You show some respect to your coworkers. Here that doesn’t exist even if you want it to.

I’ve only lost a job once before. Working at Yobject ended suddenly when their finances ran out. I didn’t get paid on time, and I only found out when my debit card started to be rejected… whilst I was on holiday at SXSW. Being strategically laid off is rather different.

Needless to say, no-one from Brickhouse started any fights with security. Instead we went to Hotel Utah at 4th & Brannan in San Francisco. Owen Thomas of Valleywag lied that he phoned in to buy everyone shots. He actually purchased six. I guess times are hard in the gutter too. The wonderful Hotel Utah barstaff donated around ten more to make up the difference. The staff are wonderful at Hotel Utah. When you’re there, ask them to add bacon to their (magnificent) Mac & Cheese, it’s stella.

Warming

Saturday 13th, three days after being laid off, was my housewarming in San Francisco. A more bittersweet timing you could not have engineered. Sitting in a home I’ve grown very fond of, surrounded by the furniture I picked out for myself and bought, pondering whether I’d still be living here in two months time, or whether it would all have been offloaded in my enforced absence.

I lamented over this briefly, and then countered by getting drunk and playing Rock Band all night. I’m enjoying throwing parties. The support of so many wonderful friends in a time of stress cannot be understated.

No stopping

To stay in the country I needed to put aside the bitter and upset emotions induced by Wednesday and push for a new role at Yahoo. Every contact I have there, and plenty more I made in the process, was contacted in the hope they find an open position within the next few weeks. Get a new role, the visa stays valid and (in some words) ‘it’s as if I were never laid off at all’.

I spent my remaining pre-Christmas days in San Francisco keeping up in a scrum of simultaneous conversation with every company who would listen who might sponsor a visa, and harranging everyone I know still inside Yahoo. Getting back into Yahoo! is the option that provides me stability. No leaving the country for any length of time, continuity of home, relationships and friendships. Longer term I can settle and get myself into a less precarious position, but the priority at every step in this horrific process is do not ruin my life.

A few people suggested to me that returning to Yahoo was non-obvious. I concede that Yahoo is a little airborne at present (remember, we don’t even have a full-time CEO), and post-layoffs and post–Brickhouse-closure we are still waiting to see a lot of the 2009 plan revealed. Fact is, I make a distinction between the company and the team. There’s nothing unreasonable in saying that every single engineer at Yahoo is expecting a lot from the Yahoo board over the next six months, to put itself on the straight and narrow again and offer a vision. But even whilst pending that progress, the teams within these walls make up of some of the most talented engineers I know and if I could get the opportunity to work with them again it would be a great thing, no matter what the circumstances.

With some delight, I can report that I did get that opportunity.

Some time in the new year I’ll be switching over into the Yahoo Developer Network, and hopefully doing awesome things to help people inside and outside Yahoo work with new platforms like YAP YQL and and so forth. I’ve been pretty passionate about development community infrastructure since I started working with microformats.org, so it’s great have that overlap with my day job more.

It will be a shame to lose working from San Francisco all the time, but I’ll adapt to the Sunnyvale commute soon enough.

To everyone who helped me out, sent me offers, enquiries and advice over the past two weeks: Thank you.