First reaction above all, is that this is great news! Very happy to see this work maturing.

I’ve tried to quickly suggest some tweaks to the vocab, by hacking his diagram in photoshop. All it really shows is that I’ve forgotten how to use photoshop, but I’ll upload it here anyway.

So if you click through to the full image, you can see my rough edits.

I’d suggest:

1. Use (dcterms) dc:subject as the way of pointing from a document to it’s SKOS subject.
2. Use (dcterms) dc:creator as the relationship between a document and the person that created it (note that in FOAF, we now declare foaf:maker to map as an equivalentProperty to (dcterms)dc:creator).
3. Distinguish between the description of the person versus their account in the Drupal system; I would use foaf:Person for the human, and sioc:User (a kind of foaf:OnlineAccount) as the drupal account. The foaf property to link from the former to the latter is foaf:account (new name for foaf:holdsAccount).
4. Focus on SIOC where it is most at-home: in modelling the structure of the discussion; threading, comments and dialog.
5. Provide a generated URI for the person. I don’t 100% understand Stephane’s comment, “Hash URIs for identifying things different from the page describing them can be implemented quite easily but this case hasn’t emerged in core” but perhaps this will be difficult? I’d suggest using URIs ending “userpage#!person” so the fragment IDs can’t clash with HTML usage.

If the core release can provide this basic structure, including a hook for describing the human person rather than the site-specific account (ie. sioc:User) then extensions should be able to add their own richness. The current markup doesn’t quite work for that end, as the human user is only described indirectly (unless  I understand current reading of sioc:User).

Anyway, I’m nitpicking! This is really great, and a nice and well-deserved boost for the RDFa community.

To the best of my knowledge, Ludovic Hirlimann’s PGP fingerprint is 6EFBD26FC7A212B2E093 B9E868F358F6C139647C. You might also be interested in his photos on flickr, or his workplace, Mozilla Messaging. The GPG key details were checked over a Skype video call with me, Ludo and Kaare A. Larsen.

This blog post isn’t signed, the URIs it referenced don’t use SSL, and the image could be switched by evildoers at any time! But the question’s worth asking: is this kind of scruffy key info useful, if there’s enough of it? If I wrote it somehow in Thunderbird’s editor instead, would it be easier to sign? Will 99.9% of humans ever know enough of what’s going on to understand what signing a bunch of complex markup means?

For earlier discussion of this kind of thing, see Joseph Reagle’s Key-free Trust piece (“Does Google Show How the Semantic Web Could Replace Public Key Infrastructure?”). It’s more PKI-free trust than PK-free.

Looking in the UK Govt datasets, I see it is listed there with a homepage of ‘http://www.hamsey-green-infant.surrey.sch.uk’ (which doesn’t seem to work).

Some queries I’m trying via the SPARQL dataset (I’ll update this post if I make them work…)

First a general query, from which I found the URL manually, …

select distinct ?x ?y where { ?x <http ://education.data.gov.uk/def/school/websiteAddress> ?y . }

Then I can go back into the data, and find other properties of the school:

PREFIX sch-ont:  <http://education.data.gov.uk/def/school/>
select DISTINCT ?x ?p ?z WHERE
{
?x sch-ont:websiteAddress "http://www.hamsey-green-infant.surrey.sch.uk" .
?x ?p ?z .
}

Results in json

How to make this presentable? I can’t get output=html to work, but if I run this ‘construct’ query it creates a simple flat RDF document:

PREFIX sch-ont:  <http://education.data.gov.uk/def/school/>
CONSTRUCT {
 ?x ?p ?z .
}
WHERE
{
?x sch-ont:websiteAddress "http://www.hamsey-green-infant.surrey.sch.uk" .
?x ?p ?z .
}

So, where are we here? We see two RDF datasets about the same school. One is the simple claim that I attended the

school at some time in the past (1976-1978, in fact). The other describes many of its current attributes; most of which may be different now from in the past. In my sample RDFa, I used the most popular Web link for the school to represent it; in the Edubase government data, it has a Web site address but it seems not to be current.

Assuming we’d used the same URIs for the school’s homepage (or indeed for the school itself) then these bits of data could be joined.

Perhaps a more compelling example of data linking would be to show this schools data mixed in with something like MySociety’s excellent interactive travel maps? Still, the example above shows that basic “find people who went to my school” queries should be very possible…

FOAF has always had a bias towards describing social things that are shown rather than merely stated; this is particularly so in matters of trust. One way of showing basic confidence in others, is by accepting their comments on your blog or Web site. F2F is an experiment in syndicating information about these kinds of everyday public events. With F2F, others can share and re-use this sort of information too; or deal with it in aggregate to spread the risk and bring more evidence into their trust-related decisions. Or they might just use it to find interesting people’s blogs.

OpenID is a technology that lets people authenticate by showing they control some URL. WordPress blogs that use the OpenID plugin slowly accumulate a catalogue of URLs when people leave comments that are approved or rejected. In my previous post I showed how I was using the list of approved OpenIDs from my blog to help configure the administrative groups on the FOAF wiki.

This may all raise more questions than it answers. What level of detail is appropriate? are numbers useful, or just lists? in what circumstances is it sensible or risky to merge such data? is there a reasonable use for both ‘accept’ lists and ‘unaccept’ lists? What can we do with a list of OpenID URLs once we’ve got it? How do we know when two bits of trust ‘evidence’ actually share a common source? How do we find this information from the homepage of a blog?

If you install the F2F plugin (and have been using the OpenID plugin long enough to have accumulated a database table of OpenIDs associated with submitted comments), you can experiment with this. Basically it will generate HTML in RDFa format describing a list of people . See the F2F Wiki page for details and examples.

The script is pretty raw, but today it all improved a fair bit with help from Ed Summers, Daniel Krech and Morten Frederiksen. Ed and Daniel helped me get started with consuming this RDFa and SPARQL in the latest version of the rdflib Python library. Morten rewrote my initial nasty hack, so that it used Wordpress Shortcodes instead of hardcoding a URL path. This means that any page containing a certain string – f2f in chunky brackets – will get the OpenID list added to it. I’ll try that now, right here in this post. If it works, you’ll get a list of URLs below. Also thanks to Gerald Oskoboiny for discussions on this and reputation-related aggregation ideas; see his page on reputation and trust for lost more related ideas and sites. See also Peter Williams’ feedback on the foaf-dev list.

Next steps? I’d be happy to have a few more installations of this, to get some testbed data. Ideally from an overlapping community so the datasets are linked, though that’s not essential. Ed has a copy installed currently too. I’ll also update the scripts I use to manage the FOAF MediaWiki admin groups, to load data from RDFa blogs; mine and others if people volunteer relevant data. It would be great to have exports from other software too, eg. Drupal or MediaWiki.

Well let’s think. Humm. In RDFa, every section of text is always ‘about’ something, and then has typed links or properties associated with that thing. So there are icons ‘B’ for bold, ‘I’ for italics, etc. And thingies for bulleted lists, paragraphs, fonts. I’m not sure how easily it would handle the nesting of full RDFa, but some common case could be a start: a paragraph that was about some specific thing, and within which the topical focus didn’t shift.

So, here is a paragraph about me. How to say it’s really about me? There could be a ‘typeof’ attribute on the paragraph element, with value foaf:Person, and an ‘about’ attribute with a URI for me, eg. http://danbri.org/foaf.rdf#danbri

In this UI, I just type textual paragraphs and then double newlines are interpreted by the publishing engine as separate HTML paragraphs. I don’t see any paragraph-level or even post-level properties in the user interface where an ‘about’ URI could be stored. But I’m sure something could be hacked. Now what about properties and links? Let’s see…

I’d like to link to TinyMCE now, since that is the HTML editor that is embedded in Wordpress. So here is a normal link to TinyMCE. To make it, I searched for the TinyMCE homepage, copied its URL into the copy/paste buffer, then pressed the ‘add link’ icon here after highlighting the text I wanted to turn into a link.

It gave me a little HTML/CSS popup window with some options – link URL, target window, link title and (CSS) class; I’ve posted a screenshot below. This bit of UI seems like it could be easily extended for typed links. For example, if I were adding a link to a paragraph that was about a Film, and I was linking to (a page about) its director or an actor, we might want to express that using an annotation on the link. Due to the indirection (we’re trying to say that the link is to a page whose primary topic is the director, etc.), this might complicate our markup. I’ll come back to that later.

tinymce link edit screenshot

I wonder if there’s any mention of RDFa on the TinyMCE site? Nope. Nor even any mention of Microformats.

Perhaps the simplest thing to try to build into TinyMCE would be for a situation where a type on the link expressed a relationship between the topic of the blog post (or paragraph), and some kind of document.

For example, a paragraph about me, might want to annotate a link to my old school’s homepage (ie. http://www.westergate.w-sussex.sch.uk/ ) with a rel=”foaf:schoolHomepage” property.

So our target RDFa would be something like (assuming this markup is escaped and displayed nicely):

<p about=”http://danbri.org/foaf.rdf#danbri” typeof=”foaf:Person” xmlns:foaf=”http://xmlns.com/foaf/0.1/”>

I’m Dan, and I went to <a rel=”foaf:schoolHomepage” href=”http://www.westergate.w-sussex.sch.uk/”>Westergate School</a>

</p>.

See recent discussion on the foaf-dev list where we contrast this idiom with one that uses a relationship (eg. ’school’) that directly links a person to a school, rather than to the school’s homepage. Toby made some example RDFa output of the latter style, in response to a debate about whether the foaf:schoolHomepage idiom is an ‘anti-pattern’. Excerpt:

<p about=”http://danbri.org/foaf.rdf#danbri” typeof=”foaf:Person” xmlns:foaf=”http://xmlns.com/foaf/0.1/”>

My name is <span property=”foaf:name”>Dan Brickley</span> and I spent much of the ’80s at <span rel=”foaf:school”><a typeof=”foaf:Organization” property=”foaf:name” rel=”foaf:homepage” href=”http://www.westergate.w-sussex.sch.uk/”>Westergate School</a>.

</p>

So there are some differences between these two idioms. From the RDF side they tell you similar things: the school I went to. The latter idiom is more expressive, and allows additional information to be expressed about the school, without mixing it up with the school’s homepage. The former simply says “this link is to the homepage of Dan’s school”; and presumably defers to the school site or elsewhere for more details.

I’m interested to learn more about how these expressivity tradeoffs relate to the possibilities for visual / graphical editors. Could TinyMCE be easily extended to support either idiom? How would the GUI recognise a markup pattern suitable for editing in either style? Can we avoid having to express relationships indirectly, ie. via pages? Would it be a big job to get some basic RDFa facilities into Wordpress via TinyMCE?

For related earlier work, see the 2003 SWAD-Europe report on Semantic blogging from the team then at HP Labs Bristol.

Both Wordpress blogs and Mediawiki wikis have some support for OpenID logins.

The FOAF wiki until recently only had one Sysop and Bureaucrat account (a bureaucrat has the same privileges as a Sysop except for the ability to create new bureaucrat accounts). So I’ve begun an experiment exploring idea of pre-approving certain OpenIDs for bureaucrat activities. For now, I take a list of OpenIDs from my own blog; these appear to be just the good guys, but this might be because only real humans have commented on my blog via OpenID. With a bit of tweaking I’m sure I could write SQL to select out only OpenIDs associated with posts or comments I’ve accepted as non spammy, though.

So now there’s a script I can run (thanks tobyink and others in #swig IRC for help) which compares an externally supplied list of OpenID URIs with those OpenIDs known to the wiki, and upgrades the status of any overlaps to be bureaucrats. Currently the ’syndication’ is trivial since the sites are on the same machine, and the UI is minimal; I haven’t figured out how best to convey this notion of ‘pre-approved upgrade’ to the people I’m putting in an admin group. Quite reasonably they might object to being misrepresented as contributors; who knows.

But all that aside, take a look and have a think. This kind of approach has a lot going for it. We will have all kinds of lists of people, groups of people, and in many cases we’ll know their OpenIDs. So why not pool what we know? If a blog or wiki has information about an OpenID that shows it is somehow trustworthy, or at least not obviously a spammer, there’s every reason to make notations (eg. FOAF/RDFa) that allow other such sites to harvest and integrate that data…

See also Dan Connolly’s DIG blog post on this, and the current list of Bureaucrats on the FOAF Wiki (and associated documentation). If your names on the list, it just means your OpenID was on a pre-approved list of folk who I trust based on their interactions with my own blog. I’d love to add more sources here and make it genuinely communal.

This is all part of the process of getting FOAF moving again. The brains of FOAF is in the IssueTracker page, and since the site was damaged by spammers and hackers recently I’m trying to make sure we have a happy / wholesome environment for maintaining shared documents. And that’s more than I can do as a solo admin, hence this design for opening things up…

Remote apple remote HTML demo, screenshot showing a picture of handheld apple remote with a grey blob over the play/pause button, indicating a mouse up event. Also shows debug text in html indicating ButtonUpEvent: PLPZ.

This webclient needs the JID and password details for an XMPP account, and I think these need to be from the same HTTP server the HTML is published on. It works using BOSH or other tricks, but for now I’ve not delved into those details and options. Source is in the Buttons area of the FOAF svn: webclient. I made a set of images, for each button in combination with button-press (‘down’), button-release (‘up’). I’m running my own ejabberd and using an account ‘buttons@foaf.tv’ on the foaf.tv domain. I also use generic XMPP IM accounts on Google Talk, which work fine although I read recently that very chatty use of such services can result in data rates being reduced.

To send local Apple Remote events to such a client, you need a bit of code running on an OSX machine. I’ve done this in a mix of C and Ruby: imremoted.c (binary) to talk to the remote, and the script buttonhole_surfer.rb to re-broadcast the events. The ruby code uses Switchboard and by default loads account credentials from ~/.switchboardrc.

I’ve done a few tests with this setup. It is pretty responsive considering how much indirection is involved: but the demo UI I made could be prettier. The + and – buttons behave differently to the left and right (and menu and play/pause); only + and – send an event immediately. The others wait until the key is released, then send a pair of events. The other keys except for play/pause will also forget what’s happening unless you act quickly. This seems to be a hardware limitation. Apparently Apple are about to ship an updated $20 remote; I hope this aspect of the design is reconsidered, as it limits the UI options for code using these remotes.

I also tried it using two browsers side by side on the same laptop; and two laptops side by side. The events get broadcasted just fine. There is a lot more thinking to do re serious architecture, where passwords and credentials are stored, etc. But XMPP continues to look like a very interesting route.

Finally, why would anyone bother installing compiled C code, Ruby (plus XMPP libraries), their own Jabber server, and so on? Well hopefully, the work can be divided up. Not everyone installs a Jabber server. My thinking is that we can bundle a collection of TV and SPARQL XMPP functionality in a single install, such that local remotes can be used on the network, but also local software (eg. XBMC/Plex/Boxee) can also be exposed to the wider network – whether it’s XMPP .js running inside a Web page as shown here, or an iPhone or a multi-touch table. Each will offer different interaction possibilities, but they can chat away to each other using a common link, and common RDF vocabularies (an area we’re working on in NoTube). If some common micro-protocols over XMPP (sending clicks or sending commands or doing RDF queries) can support compelling functionality, then installing a ‘buttons adaptor’ is something you might do once, with multiple benefits. But for now, apart from the JQbus piece, the protocols are still vapourware. First I wanted to get the basic plumbing in place.

Update: I re-discovered a useful ‘which bosh server do you need?’ which reminds me that there are basically two kinds of BOSH software offering; those that are built into some existing Jabber/XMPP server (like the ejabberd installation I’m using on foaf.tv) and those that are stand-alone connection managers that proxy traffic into the wider XMPP network. In terms of the current experiment, it means the event stream can (within the XMPP universe) come from addresses other than the host running the Web app. So I should also try installing Punjab. Perhaps it will also let webapps served from other hosts (such as opensocial containers) talk to it via JSON tricks? So far I have only managed to serve working Buttons/Strophe HTML from the same host as my Jabber server, ie. foaf.tv. I’m not sure how feasible the cross-domain option is.

Update x2: Three different people have now mentioned opensoundcontrol to me as something similar, at least on a LAN; it clearly deserves some investigation…

The nice thing about this setup is that I can log in on another laptop to Gmail, run the Javascript Gmail / Google Talk chat UI, and hear a ‘beep’ whenever the event/message arrives in my browser. This is handy for informally testing the laggyness of the connection, which in turn is critical when designing remote control protocols: how chatty should they be? How much smarts should go into the client? which bit of the system really understands what the user is doing? Informally, the XMPP events seem pretty snappy, but I’d prefer to see some real statistics to understand what a UI might risk relying on.

What I’d like to do now is get a Strophe Javascript client running. This will attach to my Jabber server and allow these events to show up in HTML/Javascript apps…

Here’s sample output of the script (local copy but it looks the same remotely), in which I press and release quickly every button in turn:

Cornercase:osx danbri$ ./buttonhole_surfer.rb
starting event loop.

=> Switchboard started.
ButtonDownEvent: PLUS (0x1d)
ButtonUpEvent: PLUS (0x1d)
ButtonDownEvent: MINU (0x1e)
ButtonUpEvent: MINU (0x1e)
ButtonDownEvent: LEFT (0x17)
ButtonUpEvent: LEFT (0x17)
ButtonDownEvent: RIGH (0x16)
ButtonUpEvent: RIGH (0x16)
ButtonDownEvent: PLPZ (0x15)
ButtonUpEvent: PLPZ (0x15)
ButtonDownEvent: MENU (0x14)
ButtonUpEvent: MENU (0x14)
^C
Shutdown initiated.
Waiting for shutdown to complete.
Shutdown initiated.
Waiting for shutdown to complete.
Cornercase:osx danbri$

I have been learning how to access them. Conclusion: iremoted does 95% of what you probably need, and the discussion over on cocoadev.com tells you more than you probably wanted to know. My experiments are written up in a corner of the FOAF wiki, and have a large collection of related links stored under the ‘buttons’ tag at delicious.com.

Summary: If you run iremoted, it calls OSX APIs and gets notified whenever button presses on the Apple Remote are detected.

There are 6 buttons: a menu button, a play/pause main button, and four others surrounding the main button. To the left and right there are ‘back’ and ‘fwd’; and above/below they are labelled ‘+’ and ‘-’.

Interestingly from a protocol and UI design view, those last buttons behave differently. When you press and release the other 4 buttons, the 2 events are not reported until after you release; but with ‘+’ and ‘-’, the initial press is reported immediately. Furthermore, only ‘+’ and ‘-’ do anything interesting if you press for longer than a second or so; the others result in your action being silently dropped, even when you release the button.

I am looking at this since I am revisiting my old Jqbus FOAF experiments with XMPP and SPARQL for data access; but this time building in some notion of behaviour too, hence the new name: Buttons. The idea is to use XMPP as a universal communication environment between remote controls and remote-controllable things, initially in the world around TV. So I am exploring adaptors for existing media centres and media players (Boxee, XBMC, iTunes etc).

There are various levels of abstraction at which XMPP could be used as a communications pipe: it could stream clicks directly, by simply reflecting the stream of Infra Red events, translating them into IQ messages sent to JID-identified XMPP accounts. Or we could try to interpret them closer to their source, and send more meaningful events. For example, if the “+” is held for 3 seconds, do we wait until it is released before sending a “Maximize volume” message out across the ‘net. Other approaches (eg. as suggested on the xmpp list, and in some accessibility work) include exporting bits of UI to the remote. Or in the case of really dumb remotes with no screen, at least some sort of invisible proxy/wrapper that tries to avoid sprawing every user click out across XMPP.

Anyway, I thought this worth a writeup. Here’s a link to my copy of iremoted, iremoted_buttons.c. The main change (apart from trivial syntax edits for its output) is the use of kIOHIDOptionsTypeSeizeDevice. This is pretty important, as it stops other processes on your mac receiving the same events; eg. FrontRow taking over the laptop UI whenever ‘menu’ is pressed, or the local volume being changed by ‘+’ and ‘-’:

    ioReturnValue = (*hidDeviceInterface)->open(hidDeviceInterface, kIOHIDOptionsTypeSeizeDevice);

So I post this to make myself absolutely clear. A few days ago I mentioned in IRC a concern that newcomers to SPARQL and RDF databases might not appreciate which SPARQL extensions are widely implemented, and which are the specialist offerings of the system they happen to be using. I mentioned OpenLink’s Virtuoso in particular as a SPARQL implementation that had a rich and powerful set of extensions.

Since it seems there is some risk I might be mis-interpreted as suggesting OpenLink are actively trying to “do a Microsoft” and trap users in some proprietary pseudo-SPARQL, I’ll state what I took to be obvious background knowledge: OpenLink is a company who owe their success to the promotion of cross-vendor database portability, they have been tireless advocates of a standards-based Semantic Web, and they’re active in proposing extensions to W3C for standardisation. So – no criticism of OpenLink intended. None at all.

All I think we need here, are a few utilities that help developers understand the nature of the various SPARQL dialects and the potential costs/benefits of using them. Perhaps an online validator, alongside those for RDF/XML, RDFa, Turtle etc. Such a validator might usefully list the extensions used in some query, and give pointers (perhaps into a wiki) where the status of the various extensions constructs can be discussed and documented.

Since SPARQL is such a young language, it lacks a lot of things that are taken from granted in the SQL world, and so using rich custom extensions when available is for many developers a sensible choice. My only concern is that it must be a choice, and one entered into consciously.

I have been looking at aspects of the existing mainstream ecosystem, including so-called media centre systems, and at the various classes of app (remote controls, tv guides, media players) available on smart phones such as the iPhone. At the moment, all these applications are quite fragmented, with different pairs of remote control and player, different APIs, different metadatasystems.

This post gives some background thinking before jumping into super-technical details. In the next post I’ll give an overview of some of the ‘media centre’ APIs I’ve been looking at; this is the beginnings of a survey to explore the extent to which existing players can be driven by external software through a network interface. My working hypothesis is that the XMPP protocol provides the best candidate protocol environment to do this, since it allows devices to be addressed globally, even when they’re in the home; however the initial survey makes no assumptions about how these APIs might be unified.

Nearby: for some smartphone visuals, see my Flickr’d collection of iPhone screen grabs. I hope to revisit and annotate these later. For ongoing W3C work on an API and ontology for Media Objects, see the Use Cases and Requirements document over at W3C. This work is largely focussed on describing individual media objects, however it does note a requirement for “Being able to apply the ontology / API for collections of metadata”, as well as a requirement to allow references “to multimedia objects on several abstraction levels, in order to separate e.g. a movie, a DVD which contains the movie and a specific copy of the DVD. Especially for collections of multimedia objects, knowledge about such abstraction levels is helpful, as a means for accessing the objects on each level.”

In NoTube, we are also looking at work such as the BBC Programmes Ontology, which makes such distinctions in terms of brands, series, shows, episodes etc. This is not only to group different representations of the same content (eg. a mobile-friendly downcoding of a high-definition news article); it also allows us to cluster metadata about similar items or different versions or manifestations of the same underlying content.

Since metadata is notoriously expensive and hard to manage, having it richly interlinked makes a lot of sense.

For example, if I write a comment while watching a file that I created from an Amazon purchased DVD of “The Fog of War”, it would be a waste (and lead to a very sparse metadata network) if my annotation was only available to other users of that exact same file within this closed home network, although the media file itself must of course stay private. The first time I saw that movie, was a screening in Bristol’s Cube Cinema. If I wrote a blog post about the movie back then, that information should also be linked to common identifiers (eg. IMDB, FreeBase, Wikipedia,  DBPedia, Sony’s page, the director’s synopsis etc.). But there’s more annotation in the Web than just my annotations: the director recently wrote a blog post about the main subject of the film, Robert McNamara. The director, Errol Morris, also publishes a transcript of the movie, and his homepage links to his Twitter microblog. If the content is available eg. through a system like Joost or Hulu, or broadcast on TV, or screened at a local cinema, most metadata about it will still be relevant, even if it was created in a different context.

For Web-heads, this online data-linking story is business as usual; for the broadcast TV industry, it’s something of a new world.

The Web can be more than a way of driving content to viewing portals, it can be more than a threat to content owners, it can change the way we think about what all these moving pictures are telling us. Douglas Adams described this very well 20 years ago, and if you’ve not seen his Hyperland documentary on pre-Web hypertext, I do urge you to go find a way to see it.

So, let’s stick with my example movie, and work outwards from an assumption we have one reasonable identifier for the film that connects us to a Web of linked data about it. Errol Morris’s (excellent) film The Fog of War. How could a simple piece of software (running on a smart phone or home media centre) figure out that he is the director? Well, if we believe Wikipedia infoboxes and we find and believe DBpedia’s derived page about the film, the information is there in a machine-friendly form: it says that that ‘director’ property of the film is Errol Morris. And following that link, in turn we can see a claim that Errol Morris has a ‘website’ at www.errolmorris.com.

So, working outwards from the content, we find the director, and the director’s website url. What can we do with that? Well we could try loading pages from the site directly, and interpreting them. But the chances are currently low that we’ll find any machine-oriented, automation-friendly markup there. And indeed, if we look at the source of his page, there isn’t a lot there to help machines. Here is the markup for the twitter link mentioned earlier:

<a href="http://twitter.com/ErrolMorris" target="_blank">

If the page had included just a few extra characters, the microformat markup “rel=’me’“, or had a FOAF file, we could automatically have discovered that he had a microblog. He also has another blog on the New York Times site, and both of these have feeds (Twitter RSS; NYTimes RSS). Blogs and microblogs provide a way of establishing a more vivid and immediate connection between viewers and content creators; but finding the relevant feed for a given media object is non-trivial. So let’s walk through some of the issues and technologies.

0.  Content identification

For any of this ‘Linked TV’ scenario to make any sense, we need to get our hands on a solid widely-known identifier for the content. This is our entry pass into Douglas Adams’ ‘hyperland’, and is harder than it might seem. Typically all we have to identify a piece of content is information about a file, and maybe a textual label or two. There are a variety of automatic and semi-automatic approaches here, and growing support for rich disambiguation in freely available tools (eg. Boxee).

1. Reliability of linked data

Given a wikipedia or imdb url, we can find a lot of linked RDF data. In the example here, we use dbpedia to find the director’s homepage. Is this risky? Could the page be edited mischievously? Yes! Are there scenarios in which such Webby uncertainty is inappropriate in a TV context? Yes. Can we expect to see commercial and Musicbrainz-style collaborative enrichment and QA of linked datasets? I think so.

2. Social Graph discovery

OK, so we’ve found the director’s homepage. Maybe we could have found actors, presenters or writers too, given different content. This is great, but we don’t have much there that makes sense in a TV user interface. How do we find the twitter microblog programmatically? or his New York Times blog? How can we be sure of those links?

This is partially a matter of waiting, and partially a chicken-and-egg situation. If we build TV tools and lightweight standards which work better if content creator sites have a little extra markup (eg. rel=’me’), some content creators will add it to their homepages and blogs, and some hosting / tool vendors will add it automatically anyway. But what can we do today? Let’s look at Google’s Social Graph API.

The SGAPI allows us to take advantage of Google’s global Web index, and ask questions about people, their profiles, account pages, and connections.

We can ask for example, which accounts are claimed by http://www.errolmorris.com/. Today this gives no results, since the link in that page to the matching twitter page contains no semantic markup.

We can also ask, which accounts are claimed by http://twitter.com/ErrolMorris … and this finds some interesting information, since (unlike errormorris.com) Twitter is a site that is understood by the Google Social Graph API. We find a location (Boston MA), some photo and feed urls, but also a link from ErrolMorris on twitter to his homepage, www.errolmorris.com.

This last point is important: Google Social Graph API is structured in terms of claims. This gives it a lot more robustness against spammers and mischief. SGAPI notices that the markup on the Twitter page says, in effect, “this is my homepage over here” (in microformat-speak, ‘this is (also) me’). Here it is in full, … the link from Errol Morris on Twitter, linking to his homepage:

<a href="http://www.errolmorris.com" class="url" rel="me nofollow" target="_blank">

But the Google SGAPI notices that the homepage doesn’t explicitly reciprocate the claim. There is no machine markup in the homepage to indicate that the owner of the page is saying that the twitter page “is me”. It might just be a link to a friend, for example.

Nevertheless, we can use the SGAPI in less trusting mode (’show inbound links’), and take advantage of Google’s massive Web index to ask: which pages claim to have the same owner as www.errolmorris.com?

This gives us a story about how we can find a lot of useful contextual information, given a basic starting point. The Google service I show here is just one of several that could be exploited in ‘linked tv’ scenarios.

For finding other identifiers, we might use sameas.org. Here’s what I get when I ask it for other URIs for the fog of war film, using a dbpedia uri as entry point:

1. http://dbpedia.org/resource/The_Fog_of_War
2. http://dbpedia.org/resource/11_Lessons_from_the_Life_of_Robert_S._Mcnamara
3. http://mpii.de/yago/resource/The_Fog_of_War
4. http://rdf.freebase.com/ns/guid.9202a8c04000641f80000000004e1a97

For ’social graph’ data, we might also check sindice.com or foaf.qdos.com.

Topic description

A quick word on topics and wikipedia: by using content identifiers that link to Wikipedia and DBpedia, we have the potential for extremely rich, fine grained topical annotations.

Wikipedia’s page about the movie notes that it covers the Cuban Missile Crisis. This association is now machine-visible, since I personally improved the relevant Wikipedia entry to explicitly link to the Cuban Missile Crisis page on Wikipedia. This allows us to use wikipedia:Cuban_Missile_Crisis as a topic indicator, not just against the entire film but against particular segments of the movie that are about that topic. It is easy to get a list of such links from the Wikipedia markup, eg. [[Ford Motor Company]], [[Vietnam War]] are also already there. If we have a TV presentation system that has a unique id for the content (and ideally content-version, since content often varies), and we know an offset in seconds, then rich topical tags could be applied to sections of the film, without the need for textual data entry. An iphone or similar device could allow users to pause, annotate/tag/bookmark and continue their viewing, with no need for a keyboard.

That smart-phone story there is worth investigating, but let’s first start closer to the screen: what kinds of set top box, media centre or gadget might be able to index, navigate and play content in a way that makes interesting use of ‘linked tv’ techniques such as these? Which brings me to APIs for ‘media centre‘ systems, finally.

If we want to recommend interesting content to users, show them relevant links, annotations, related materials (not necessarily more video – text, audio, even a spreadsheet with statistics might be appropriate), suggestions from friends, upcoming broadcasts or archived materials, then we need an environment that is scripting friendly and capable of interacting with users in a rich and compelling manner. The simplest path here is to start with what’s out there already, and look at commonalities in API and data model, to see how far existing software can be ‘remote controlled’ from external scripts (and, eventually, from actual hand-held remote controls, eg. smartphones). I’ll go into some detail on that in the followup post.

The main point I want to (re-)emphasise here, is that once we make get to the stage of having well known identifiers for content, it facilitates a very rich marketplace for TV-related metadata, with emphasis on the word “-related“. And that this is necessarily very open-ended, since TV content can be about absolutely anything. I gave some examples drawing on Wikipedia/DBpedia for content metadata, and on homepages, twitter and Google’s Social Graph API to show how additional highly relevant information can be pulled from the Web, once we get a basic starting point. Finding ways of presenting such extra information to users, and giving them appropriate navigation and interaction options, is far from easy. Fortunately it is easier to share, syndicate and merge TV meta-content, than TV content. We are already seeing systems such as the XBMC-based Boxee which will normalise content identifiers in a way that encourages legal uses over illegal. Intellectual properties issues around TV content means that actual playable content is often not broadly shareable. However the same need not be true of user-supplied metadata, since this can be about a specific media file, but also it can be about the things the content is about. I expect to see TV meta-content shared in a global linked system, even while the underlying video and audio remain relatively hidden away; rich user-supplied TV metadata isn’t just about the TV show, it equally can tell us about the world and the viewer, and deserves to be widely available through open standards.

To go back to my original example, Errol Morris’s film about Robert McNamara, The Fog of War… there’s a world of metadata options beyond five-star ratings that can enrich such content, beyond worrying about numerical ranking in statistical recommender systems. Existing work there, eg. around the Netflix prize, would be hard to beat. Statistical recommendations work well over a regular dataset where everything has a well known identifier. We can see RDF and linked data techniques serving a pre-processing role for such analysis, by linking together otherwise fragmented pieces of information about content, allowing classic techniques to be applied over a wider dataset.

However, I see the true ‘linked TV’ potential to be primarily in another direction: in creating more meaningful conversations around content, and helping users find other information that gives a complementary perspective on the materials, and on cross-referencing everything with everything so that unexpected new paths can be found. Morris’ film can teach us a lot about McNamara, but also about the wider world and recent history … about the Ford Motor Company, the Vietnam War, the Cuban Missile Crisis, the firebombing of Tokyo under the command of Curtis LeMay. Not all easy or pleasant topics to try to understand, and each with thousands of other relevant sources (video or otherwise) out there, different perspectives to cross-reference, different accounts to reconcile.

If television is an environment in which we can be informed, educated and entertained (by broadcasters, content creators, and increasingly, by everyone else too), we need to think through what this means for better metadata. Today we are lucky if we can find a good way of even identifying a piece of content. But tomorrow, we should have TV that comes out of the box with “Do You Want To Know More?” switched ‘on‘ by default.

When Obama gives a talk (see his June 2009 speech in Cairo) he mentioned the following, at around 33 minutes and 30 seconds:

For many years, Iran has defined itself in part by its opposition to my country, and there is in fact a tumultuous history between us.  In the middle of the Cold War, the United States played a role in the overthrow of a democratically elected Iranian government.

This last point is well covered by historians, but was likely to be new information for many viewers in the USA and UK, if not for those in Iran. What can we do to improve that, by bringing Web and TV closer together?

While the raw video, audio, and transcript are  available on the Whitehouse site, we don’t yet have enough for “do you want to know more?”. W3C’s TimedText work might provide a basis for associating the transcript with the video as subtitles. This is not enough for someone who wants to know more, in following up this unusual acknowledgement of superpower interference.

An Internet-literate, laptop-owning Web user might try to find out more in an active manner. Although they are in a minority amongst TV viewers, there are still thousands of such people, and they know how to go to google.com and do searches. Can we harness their energy to improve the TV metadata environment for everyone? If you search Google for ‘US overthrow iranian government’ you find a good starting point for learning more, a Wikipedia page “1953 Iranian coup d’état“. Although that page has content which Wikipedia flags as controversial, it has a world-visible talk page in which contributors from around the world can debate the detail of the topic. So how can this Web content be bridged with the world of TV?

TV viewers are commonly concerned characterised in terms of a pyramid:

• a passive majority, content to watch without interaction
• a smaller group who will interact and navigate, but who rarely create new content
• an even smaller group of activist users, who will explore, annotate, interact and create using all the tools available

The production and consumption of ‘do you want to know more?’ annotation won’t equally distributed. All viewers can benefit from, for example, a link from the Obama Cairo speech video to background information on the 1953 Iran coup. Some viewers might benefit indirectly, because their more inquisitive friends and family will have explored the linked materials and will talk to them about it. Others might read the wikipedia page and linked pages directly; if not on their television, then later on a laptop or PC.

Still others (a minority, but such minorities can be influential) might engage further with the material, become intrigued by the differences of opinion and seek more perspectives from the wider Web. Now 1953 is not ancient history; it falls within the living memory of countless people, in Iran, the UK and the world at large. A challenge for Linked TV is to find ways to integrate those memories and views into the TV environment, such that any viewer could pause the playing of the Obama Cairo speech video because they want to know more about what he just said. Regardless of whether they are watching on whitehouse.gov, YouTube, Joost, a live re-broadcast, a radio-extract, a player embedded in a social network site (facebook, orkut, hyves, …), or running on a mobile phone. The current problem Iranians have with getting full and high-bandwidth Web access are another obstacle, of course, but eventually we can expect relevant video or audio memories from Iran (with subtitles and translations) to be 1 or 2 clicks away from Obama’s video, regardless of the software and environment playing the video. Much of that material is already out there somewhere, the challenge is to find it, link it and present it appropriately.

The core piece of metadata, a link from a section of the video (ie. the offset, 33mins, 30 secs) of his speech to a URL for the topic it describes, could be created by whitehouse.gov staff, or it could be created by activist users. We pretty much know how to do this. A time-offset and a wikipedia URL do that job well enough for now. But what we have so far failed to achieve is a world-wide data ecosystem in which this kind of TV-enriching metadata is plentiful, widely used, and comprehensive enough that we can grow to expect to have fact-checkable TV, cross-referencable TV, hypertext TV. Instead of throwing physical things at the TV when we disagree with what we hear, it should become commonplace to be able to press pause and then or later publish your perspective (an audio rant, a fact-check crosslink, a blog post) in a way that will become accessible by other viewers of that material in the future.

To achieve this, we can’t afford needless fragmentation

• of meta-content about different versions of the same basic content
• between geographical regions based on geo-rights to play the material
• between desktop, media-centre, mobile, set top box, broadcast and on-demand scenarios
• between radio and TV
• between files and streams
• between file formats
• between populations (‘net censorship and crude blocking technology)

If these huge obstacles can be overcome, it might become reasonable to expect to find useful information attached to most of what we see on next-generation TV, regardless of delivery system. My hunch is the right place to prototype is around opensource and hackable media centre systems, and the right low-level communications system is XMPP. I’ll go into this more in the next post. Apologies for the length of this one. Comments welcomed by blog email or whatever!

I found I had to download the source from the Mercurial Web interface, rather than use pre-prepared XPI installers. This may have improved by the time you read this. I also learned (from Standard9 in #jetpack IRC) that I need asuth’s repository, rather than the main one. Again, things move quickly, don’t assume this is true forever.

Here is what worked for me, on OSX.

1. Grab a .zip from the Jetpack repo, and unpack it locally on a machine that has Thunderbird installed.

2. Edit extensions/install.rdf and make sure the em:maxVersion in the Thunderbird section matches your version of Thunderbird. In mine I updated it to say <em:maxVersion>3.0b4</em:maxVersion> (instead of 3.0b4pre).

3.  See the README in the jetpack filetree for installation. With Thunderbird closed, I ran “python manage.py install –app=thunderbird” and I found Jetpack installed fine.

4. Run Thunderbird, you should see an about:jetpack tab, and corresponding options in the Tools menu.

This was enough to get started. See discussion on visophyte.org for some example code.

After installation, you can use the about:jetpack windows to load, reload and delete Jetpacks from URL.

So, why would you bother doing all this? Jetpack provides a simple way of extending an email client using Web technology.

In my current (unfinished!) experiment, for example, I’m looking at making a sidebar the shows information (photo, blog etc.) about the sender of the currently-viewed email. And I figured that if I blogged this HOWTO, someone more familiar with ajax, jquery etc might care to help with wiring this up to the Google Social Graph JSON API, so we can use FOAF and XFN to provide more contextual information around incoming mail…

Assuming you are running Thunderbird 3b4

I have started to look at Prism as a basis for accessing robust, mirrored sites, so that a single point of failure (or censorship) might be avoided. With a lot help from Matt and others in #prism IRC chat, I have something almost working. The idea is simple: hack Prism so that the running browser code intercepts clicks and (based on some as-yet-undefined logic and preferences) gets the page from a list of mirrors, which might also be fetched dynamically from the ‘net.

I should also mention that one motivation here is for anti-censorship tools, to give users an easy way to access sites which might be blocked by their IP address or URL otherwise. I looked at FoxyProxy as an option but for site-specific robustness, running a full proxy server seems a bit heavy, compared to simply duplicating a set of files. Here’s what the main Prism app looks like:

Screenshot showing Prism config settings for a site-specific browser.

Once you have Prism installed, you can hack a file named webrunner.js to intervene when links are clicked. In OSX, this can be found as /Applications/Prism.app/Contents/Resources/chrome/webrunner/content/webrunner.js.

Edit this: _domActivate : function(aEvent)

I added the following block to the start of this function:

var link = aEvent.target;
if (link instanceof HTMLAnchorElement && !WebRunner._isLinkExternal(link)) {
aEvent.preventDefault();
WebRunner._getBrowser().loadURI(“http://example.org/mirrors/”+link.href,null,null);
}

The idea here being that we intercept clicks, and rewrite them to point to equivalent http:// URIs elsewhere in the Web. As far as this goes, it works as advertised. But what I have is far from working… it would need some code in there to find the right mirror URLs to fetch from. Perhaps a list might be fetched on startup or first time a link is followed. It could also do with some work on packaging, so that this hacked version of Prism plus some actual site-specific browser config can be made into an easy-install Windows .exe or OSX .app. For a Windows installer, I am told that NSIS is a good place to start. You could also imagine a version that hid the mirrored URLs from user’s view. Since Prism has a built-in option to completely hide the URL navigation bar, I didn’t investigate this idea yet.

OK I think I’ve written up everything I learned from the helpful folks in IRC. I hope this repays some karma. If anyone cares to explore this further, or wants to help target student projects on exploring it, please get in touch.

Interesting excerpt:

Q: Since Wolfram|Alpha is dubbed ’smarter’ than traditional search engines, I wonder how much AI techniques are actually employed in the system? How is inference done? What is the provenance of each fact/claim? And what if there is a disagreement? For example, how it would represent information about Israel/Palestine area?

A: It’s much more an engineered artifact than a humanlike artificial intelligence. Some of what it does – especially in language understanding – may be similar to what humans do. But its primary objective is to do directed computations, not to act as a general intelligence. Wolfram|Alpha uses established scientific or other models as the basis for its computations. Whenever it does new computations, it’s effectively deriving new facts. About the controversial data you asked about, we deal in different ways with numerical data and particular issues. For numerical data, Wolfram| Alpha curators typically assign a range of values that are then carried through computations. For issues such as the interpretation of particular names or terms, like Israel/Palestine area issue mentioned in your question, Wolfram|Alpha typically prompts users to choose the assumption they want. We spend considerable effort on automated testing, expert review, and checking external data that we use to ensure the results. But with trillions of pieces of data, it’s inevitable that there are still errors out there. If you ever see a problem, please report it.