This makes the mapping of UDC (and to some extent also Dewey classifications)  into W3C’s SKOS somewhat lossy, since patterns and conventions for documenting these complex, composed structures are not yet well established. In the NoTube project we are looking into this in a TV context, in large part because the BBC archives make extensive use of UDC via their Lonclass scheme; see my ‘investigating Lonclass‘ and UDC seminar talk for more on those scenarios. Until this week I hadn’t thought enough about the potential for using this to link deep into statistical datasets.

One of the examples discussed on Tuesday was as follows (via Richard Cyganiak):

“There were 66 fatal occupational injuries in the Washington, DC metropolitan area in 2008″

There was much interesting discussion in Tilburg about the proper scope and role of Linked Data techniques for sharing this kind of statistical data. Do we use RDF essentially as metadata, to find ‘black boxes’ full of stats, or do we use RDF to try to capture something of what the statistics are telling us about the world? When do we use RDF as simple factual data directly about the world (eg. school X has N pupils [currently; or at time t]), and when does it become a carrier for raw numeric data whose meaning is not so directly expressed at the factual level?

The state of the art in applying RDF here seems to be SDMX-RDF, see Richard’s slides. The SDMX-RDF work uses SKOS to capture code lists, to describe cross-domain concepts and to indicate subject matter.

Given all this, I thought it would be worth taking this tiny example and looking at how it might look in UDC, both as an example of the ‘compositional semantics’ some of us hope to capture in extended SKOS descriptions, but also to explore scenarios that cross-link numeric data with the bibliographic materials that can be found via library classification techniques such as UDC. So I asked the ever-helpful Aida Slavic (editor in chief of the UDC), who talked me through how this example data item looks from a UDC perspective.

I asked,

So I’ve just got home from a meeting on semweb/stats. These folk encode data values with stuff like “There were 66 fatal occupational injuries in the Washington, DC metropolitan area in 2008″. How much of that could have a UDC coding? I guess I should ask, how would subject index a book whose main topic was “occupational injuries in the Washington DC metro area in 2008″?

Aida’s reply (posted with permission):

You can present all of it & much more using UDC. When you encode a subject like this in UDC you store much more information than your proposed sentence actually contains. So my decision of how to ‘translate this into udc’ would depend on learning more about the actual text and the context of the message it conveys, implied audience/purpose, the field of expertise for which the information in the document may be relevant etc. I would probably wonder whether this is a research report, study, news article, textbook, radio broadcast?

Not knowing more then you said I can play with the following: 331.46(735.215.2/.4)”2008”

Accidents at work — Washington metropolitan area — year 2008
or a bit more detailed:  331.46-053.18(735.215.2/.4)”2008”
Accidents at work — dead persons – Washington metropolitan area — year 2008
[you can say the number of dead persons but this is not pertinent from point of view of indexing and retrieval]

…or maybe (depending what is in the content and what is the main message of the text) and because you used the expression ‘fatal injuries’ this may imply that this is more health and safety/ prevention area in health hygiene which is in medicine.

The UDC structures composed here are:

TIME “2008″

PLACE (735.215.2/.4)  Counties in the Washington metropolitan area

TOPIC 1
331     Labour. Employment. Work. Labour economics. Organization of  labour
331.4     Working environment. Workplace design. Occupational safety.  Hygiene at work. Accidents at work
331.46  Accidents at work ==> 614.8

TOPIC 2
614   Prophylaxis. Public health measures. Preventive treatment
614.8    Accidents. Risks. Hazards. Accident prevention. Persona protection. Safety
614.8.069    Fatal accidents

NB – classification provides a bit more context and is more precise than words when it comes to presenting content i.e. if the content is focused on health and safety regulation and occupation health then the choice of numbers and their order would be different e.g. 614.8.069:331.46-053.18 [relationship between] health & safety policies in prevention of fatal injuries and accidents at work.

So when you read  UDC number 331.46 you do not see only e.g. ‘accidents at work’ but  ==>  ’accidents at work < occupational health/safety < labour economics, labour organization < economy
and when you see UDC number 614.8  it is not only fatal accidents but rather ==> ‘fatal accidents < accident prevention, safety, hazards < Public health and hygiene. Accident prevention

When you see (735.2….) you do not only see Washington but also United States, North America

So why is this interesting? A couple of reasons…

1. Each of these complex codes combines several different hierarchically organized components; just as they can be used to explore bibliographic materials, similar approaches might be of value for navigating the growing collections of public statistical data. If SKOS is to be extended / improved to better support subject classification structures, we should take care also to consider use cases from the world of statistics and numeric data sharing.

2. Multilingual aspects. There are plans to expose SKOS data for the upper levels of UDC. An HTML interface to this “UDC summary” is already available online, and includes collected translations of textual labels in many languages (see progress report) . For example, we can look up 331.4 and find (in hierarchical context) definitions in English (“Working environment. Workplace design. Occupational safety. Hygiene at work. Accidents at work”), alongside e.g. Spanish (“Entorno del trabajo. Diseño del lugar de trabajo. Seguridad laboral. Higiene laboral. Accidentes de trabajo”), Croatian, Armenian, …

Linked Data is about sharing work; if someone else has gone to the trouble of making such translations, it is probably worth exploring ways of re-using them. Numeric data is (in theory) linguistically neutral; this should make linking to translations particularly attractive. Much of the work around RDF and stats is about providing sufficient context to the raw values to help us understand what is really meant by “66″ in some particular dataset. By exploiting SDMX-RDF’s use of SKOS, it should be possible to go further and to link out to the wider literature on workplace fatalities. This kind of topical linking should work in both directions: exploring out from numeric data to related research, debate and findings, but also coming in and finding relevant datasets that are cross-referenced from books, articles and working papers. W3C recently launched a Library Linked Data group, I look forward to learning more about how libraries are thinking about connecting numeric and non-numeric information.

First reaction above all, is that this is great news! Very happy to see this work maturing.

I’ve tried to quickly suggest some tweaks to the vocab, by hacking his diagram in photoshop. All it really shows is that I’ve forgotten how to use photoshop, but I’ll upload it here anyway.

So if you click through to the full image, you can see my rough edits.

I’d suggest:

1. Use (dcterms) dc:subject as the way of pointing from a document to it’s SKOS subject.
2. Use (dcterms) dc:creator as the relationship between a document and the person that created it (note that in FOAF, we now declare foaf:maker to map as an equivalentProperty to (dcterms)dc:creator).
3. Distinguish between the description of the person versus their account in the Drupal system; I would use foaf:Person for the human, and sioc:User (a kind of foaf:OnlineAccount) as the drupal account. The foaf property to link from the former to the latter is foaf:account (new name for foaf:holdsAccount).
4. Focus on SIOC where it is most at-home: in modelling the structure of the discussion; threading, comments and dialog.
5. Provide a generated URI for the person. I don’t 100% understand Stephane’s comment, “Hash URIs for identifying things different from the page describing them can be implemented quite easily but this case hasn’t emerged in core” but perhaps this will be difficult? I’d suggest using URIs ending “userpage#!person” so the fragment IDs can’t clash with HTML usage.

If the core release can provide this basic structure, including a hook for describing the human person rather than the site-specific account (ie. sioc:User) then extensions should be able to add their own richness. The current markup doesn’t quite work for that end, as the human user is only described indirectly (unless  I understand current reading of sioc:User).

Anyway, I’m nitpicking! This is really great, and a nice and well-deserved boost for the RDFa community.

To the best of my knowledge, Ludovic Hirlimann’s PGP fingerprint is 6EFBD26FC7A212B2E093 B9E868F358F6C139647C. You might also be interested in his photos on flickr, or his workplace, Mozilla Messaging. The GPG key details were checked over a Skype video call with me, Ludo and Kaare A. Larsen.

This blog post isn’t signed, the URIs it referenced don’t use SSL, and the image could be switched by evildoers at any time! But the question’s worth asking: is this kind of scruffy key info useful, if there’s enough of it? If I wrote it somehow in Thunderbird’s editor instead, would it be easier to sign? Will 99.9% of humans ever know enough of what’s going on to understand what signing a bunch of complex markup means?

For earlier discussion of this kind of thing, see Joseph Reagle’s Key-free Trust piece (“Does Google Show How the Semantic Web Could Replace Public Key Infrastructure?”). It’s more PKI-free trust than PK-free.

FOAF has always had a bias towards describing social things that are shown rather than merely stated; this is particularly so in matters of trust. One way of showing basic confidence in others, is by accepting their comments on your blog or Web site. F2F is an experiment in syndicating information about these kinds of everyday public events. With F2F, others can share and re-use this sort of information too; or deal with it in aggregate to spread the risk and bring more evidence into their trust-related decisions. Or they might just use it to find interesting people’s blogs.

OpenID is a technology that lets people authenticate by showing they control some URL. WordPress blogs that use the OpenID plugin slowly accumulate a catalogue of URLs when people leave comments that are approved or rejected. In my previous post I showed how I was using the list of approved OpenIDs from my blog to help configure the administrative groups on the FOAF wiki.

This may all raise more questions than it answers. What level of detail is appropriate? are numbers useful, or just lists? in what circumstances is it sensible or risky to merge such data? is there a reasonable use for both ‘accept’ lists and ‘unaccept’ lists? What can we do with a list of OpenID URLs once we’ve got it? How do we know when two bits of trust ‘evidence’ actually share a common source? How do we find this information from the homepage of a blog?

If you install the F2F plugin (and have been using the OpenID plugin long enough to have accumulated a database table of OpenIDs associated with submitted comments), you can experiment with this. Basically it will generate HTML in RDFa format describing a list of people . See the F2F Wiki page for details and examples.

The script is pretty raw, but today it all improved a fair bit with help from Ed Summers, Daniel Krech and Morten Frederiksen. Ed and Daniel helped me get started with consuming this RDFa and SPARQL in the latest version of the rdflib Python library. Morten rewrote my initial nasty hack, so that it used Wordpress Shortcodes instead of hardcoding a URL path. This means that any page containing a certain string – f2f in chunky brackets – will get the OpenID list added to it. I’ll try that now, right here in this post. If it works, you’ll get a list of URLs below. Also thanks to Gerald Oskoboiny for discussions on this and reputation-related aggregation ideas; see his page on reputation and trust for lost more related ideas and sites. See also Peter Williams’ feedback on the foaf-dev list.

Next steps? I’d be happy to have a few more installations of this, to get some testbed data. Ideally from an overlapping community so the datasets are linked, though that’s not essential. Ed has a copy installed currently too. I’ll also update the scripts I use to manage the FOAF MediaWiki admin groups, to load data from RDFa blogs; mine and others if people volunteer relevant data. It would be great to have exports from other software too, eg. Drupal or MediaWiki.

Both Wordpress blogs and Mediawiki wikis have some support for OpenID logins.

The FOAF wiki until recently only had one Sysop and Bureaucrat account (a bureaucrat has the same privileges as a Sysop except for the ability to create new bureaucrat accounts). So I’ve begun an experiment exploring idea of pre-approving certain OpenIDs for bureaucrat activities. For now, I take a list of OpenIDs from my own blog; these appear to be just the good guys, but this might be because only real humans have commented on my blog via OpenID. With a bit of tweaking I’m sure I could write SQL to select out only OpenIDs associated with posts or comments I’ve accepted as non spammy, though.

So now there’s a script I can run (thanks tobyink and others in #swig IRC for help) which compares an externally supplied list of OpenID URIs with those OpenIDs known to the wiki, and upgrades the status of any overlaps to be bureaucrats. Currently the ’syndication’ is trivial since the sites are on the same machine, and the UI is minimal; I haven’t figured out how best to convey this notion of ‘pre-approved upgrade’ to the people I’m putting in an admin group. Quite reasonably they might object to being misrepresented as contributors; who knows.

But all that aside, take a look and have a think. This kind of approach has a lot going for it. We will have all kinds of lists of people, groups of people, and in many cases we’ll know their OpenIDs. So why not pool what we know? If a blog or wiki has information about an OpenID that shows it is somehow trustworthy, or at least not obviously a spammer, there’s every reason to make notations (eg. FOAF/RDFa) that allow other such sites to harvest and integrate that data…

See also Dan Connolly’s DIG blog post on this, and the current list of Bureaucrats on the FOAF Wiki (and associated documentation). If your names on the list, it just means your OpenID was on a pre-approved list of folk who I trust based on their interactions with my own blog. I’d love to add more sources here and make it genuinely communal.

This is all part of the process of getting FOAF moving again. The brains of FOAF is in the IssueTracker page, and since the site was damaged by spammers and hackers recently I’m trying to make sure we have a happy / wholesome environment for maintaining shared documents. And that’s more than I can do as a solo admin, hence this design for opening things up…

Remote apple remote HTML demo, screenshot showing a picture of handheld apple remote with a grey blob over the play/pause button, indicating a mouse up event. Also shows debug text in html indicating ButtonUpEvent: PLPZ.

This webclient needs the JID and password details for an XMPP account, and I think these need to be from the same HTTP server the HTML is published on. It works using BOSH or other tricks, but for now I’ve not delved into those details and options. Source is in the Buttons area of the FOAF svn: webclient. I made a set of images, for each button in combination with button-press (‘down’), button-release (‘up’). I’m running my own ejabberd and using an account ‘buttons@foaf.tv’ on the foaf.tv domain. I also use generic XMPP IM accounts on Google Talk, which work fine although I read recently that very chatty use of such services can result in data rates being reduced.

To send local Apple Remote events to such a client, you need a bit of code running on an OSX machine. I’ve done this in a mix of C and Ruby: imremoted.c (binary) to talk to the remote, and the script buttonhole_surfer.rb to re-broadcast the events. The ruby code uses Switchboard and by default loads account credentials from ~/.switchboardrc.

I’ve done a few tests with this setup. It is pretty responsive considering how much indirection is involved: but the demo UI I made could be prettier. The + and – buttons behave differently to the left and right (and menu and play/pause); only + and – send an event immediately. The others wait until the key is released, then send a pair of events. The other keys except for play/pause will also forget what’s happening unless you act quickly. This seems to be a hardware limitation. Apparently Apple are about to ship an updated $20 remote; I hope this aspect of the design is reconsidered, as it limits the UI options for code using these remotes.

I also tried it using two browsers side by side on the same laptop; and two laptops side by side. The events get broadcasted just fine. There is a lot more thinking to do re serious architecture, where passwords and credentials are stored, etc. But XMPP continues to look like a very interesting route.

Finally, why would anyone bother installing compiled C code, Ruby (plus XMPP libraries), their own Jabber server, and so on? Well hopefully, the work can be divided up. Not everyone installs a Jabber server. My thinking is that we can bundle a collection of TV and SPARQL XMPP functionality in a single install, such that local remotes can be used on the network, but also local software (eg. XBMC/Plex/Boxee) can also be exposed to the wider network – whether it’s XMPP .js running inside a Web page as shown here, or an iPhone or a multi-touch table. Each will offer different interaction possibilities, but they can chat away to each other using a common link, and common RDF vocabularies (an area we’re working on in NoTube). If some common micro-protocols over XMPP (sending clicks or sending commands or doing RDF queries) can support compelling functionality, then installing a ‘buttons adaptor’ is something you might do once, with multiple benefits. But for now, apart from the JQbus piece, the protocols are still vapourware. First I wanted to get the basic plumbing in place.

Update: I re-discovered a useful ‘which bosh server do you need?’ which reminds me that there are basically two kinds of BOSH software offering; those that are built into some existing Jabber/XMPP server (like the ejabberd installation I’m using on foaf.tv) and those that are stand-alone connection managers that proxy traffic into the wider XMPP network. In terms of the current experiment, it means the event stream can (within the XMPP universe) come from addresses other than the host running the Web app. So I should also try installing Punjab. Perhaps it will also let webapps served from other hosts (such as opensocial containers) talk to it via JSON tricks? So far I have only managed to serve working Buttons/Strophe HTML from the same host as my Jabber server, ie. foaf.tv. I’m not sure how feasible the cross-domain option is.

Update x2: Three different people have now mentioned opensoundcontrol to me as something similar, at least on a LAN; it clearly deserves some investigation…

I have been looking at aspects of the existing mainstream ecosystem, including so-called media centre systems, and at the various classes of app (remote controls, tv guides, media players) available on smart phones such as the iPhone. At the moment, all these applications are quite fragmented, with different pairs of remote control and player, different APIs, different metadatasystems.

This post gives some background thinking before jumping into super-technical details. In the next post I’ll give an overview of some of the ‘media centre’ APIs I’ve been looking at; this is the beginnings of a survey to explore the extent to which existing players can be driven by external software through a network interface. My working hypothesis is that the XMPP protocol provides the best candidate protocol environment to do this, since it allows devices to be addressed globally, even when they’re in the home; however the initial survey makes no assumptions about how these APIs might be unified.

Nearby: for some smartphone visuals, see my Flickr’d collection of iPhone screen grabs. I hope to revisit and annotate these later. For ongoing W3C work on an API and ontology for Media Objects, see the Use Cases and Requirements document over at W3C. This work is largely focussed on describing individual media objects, however it does note a requirement for “Being able to apply the ontology / API for collections of metadata”, as well as a requirement to allow references “to multimedia objects on several abstraction levels, in order to separate e.g. a movie, a DVD which contains the movie and a specific copy of the DVD. Especially for collections of multimedia objects, knowledge about such abstraction levels is helpful, as a means for accessing the objects on each level.”

In NoTube, we are also looking at work such as the BBC Programmes Ontology, which makes such distinctions in terms of brands, series, shows, episodes etc. This is not only to group different representations of the same content (eg. a mobile-friendly downcoding of a high-definition news article); it also allows us to cluster metadata about similar items or different versions or manifestations of the same underlying content.

Since metadata is notoriously expensive and hard to manage, having it richly interlinked makes a lot of sense.

For example, if I write a comment while watching a file that I created from an Amazon purchased DVD of “The Fog of War”, it would be a waste (and lead to a very sparse metadata network) if my annotation was only available to other users of that exact same file within this closed home network, although the media file itself must of course stay private. The first time I saw that movie, was a screening in Bristol’s Cube Cinema. If I wrote a blog post about the movie back then, that information should also be linked to common identifiers (eg. IMDB, FreeBase, Wikipedia,  DBPedia, Sony’s page, the director’s synopsis etc.). But there’s more annotation in the Web than just my annotations: the director recently wrote a blog post about the main subject of the film, Robert McNamara. The director, Errol Morris, also publishes a transcript of the movie, and his homepage links to his Twitter microblog. If the content is available eg. through a system like Joost or Hulu, or broadcast on TV, or screened at a local cinema, most metadata about it will still be relevant, even if it was created in a different context.

For Web-heads, this online data-linking story is business as usual; for the broadcast TV industry, it’s something of a new world.

The Web can be more than a way of driving content to viewing portals, it can be more than a threat to content owners, it can change the way we think about what all these moving pictures are telling us. Douglas Adams described this very well 20 years ago, and if you’ve not seen his Hyperland documentary on pre-Web hypertext, I do urge you to go find a way to see it.

So, let’s stick with my example movie, and work outwards from an assumption we have one reasonable identifier for the film that connects us to a Web of linked data about it. Errol Morris’s (excellent) film The Fog of War. How could a simple piece of software (running on a smart phone or home media centre) figure out that he is the director? Well, if we believe Wikipedia infoboxes and we find and believe DBpedia’s derived page about the film, the information is there in a machine-friendly form: it says that that ‘director’ property of the film is Errol Morris. And following that link, in turn we can see a claim that Errol Morris has a ‘website’ at www.errolmorris.com.

So, working outwards from the content, we find the director, and the director’s website url. What can we do with that? Well we could try loading pages from the site directly, and interpreting them. But the chances are currently low that we’ll find any machine-oriented, automation-friendly markup there. And indeed, if we look at the source of his page, there isn’t a lot there to help machines. Here is the markup for the twitter link mentioned earlier:

<a href="http://twitter.com/ErrolMorris" target="_blank">

If the page had included just a few extra characters, the microformat markup “rel=’me’“, or had a FOAF file, we could automatically have discovered that he had a microblog. He also has another blog on the New York Times site, and both of these have feeds (Twitter RSS; NYTimes RSS). Blogs and microblogs provide a way of establishing a more vivid and immediate connection between viewers and content creators; but finding the relevant feed for a given media object is non-trivial. So let’s walk through some of the issues and technologies.

0.  Content identification

For any of this ‘Linked TV’ scenario to make any sense, we need to get our hands on a solid widely-known identifier for the content. This is our entry pass into Douglas Adams’ ‘hyperland’, and is harder than it might seem. Typically all we have to identify a piece of content is information about a file, and maybe a textual label or two. There are a variety of automatic and semi-automatic approaches here, and growing support for rich disambiguation in freely available tools (eg. Boxee).

1. Reliability of linked data

Given a wikipedia or imdb url, we can find a lot of linked RDF data. In the example here, we use dbpedia to find the director’s homepage. Is this risky? Could the page be edited mischievously? Yes! Are there scenarios in which such Webby uncertainty is inappropriate in a TV context? Yes. Can we expect to see commercial and Musicbrainz-style collaborative enrichment and QA of linked datasets? I think so.

2. Social Graph discovery

OK, so we’ve found the director’s homepage. Maybe we could have found actors, presenters or writers too, given different content. This is great, but we don’t have much there that makes sense in a TV user interface. How do we find the twitter microblog programmatically? or his New York Times blog? How can we be sure of those links?

This is partially a matter of waiting, and partially a chicken-and-egg situation. If we build TV tools and lightweight standards which work better if content creator sites have a little extra markup (eg. rel=’me’), some content creators will add it to their homepages and blogs, and some hosting / tool vendors will add it automatically anyway. But what can we do today? Let’s look at Google’s Social Graph API.

The SGAPI allows us to take advantage of Google’s global Web index, and ask questions about people, their profiles, account pages, and connections.

We can ask for example, which accounts are claimed by http://www.errolmorris.com/. Today this gives no results, since the link in that page to the matching twitter page contains no semantic markup.

We can also ask, which accounts are claimed by http://twitter.com/ErrolMorris … and this finds some interesting information, since (unlike errormorris.com) Twitter is a site that is understood by the Google Social Graph API. We find a location (Boston MA), some photo and feed urls, but also a link from ErrolMorris on twitter to his homepage, www.errolmorris.com.

This last point is important: Google Social Graph API is structured in terms of claims. This gives it a lot more robustness against spammers and mischief. SGAPI notices that the markup on the Twitter page says, in effect, “this is my homepage over here” (in microformat-speak, ‘this is (also) me’). Here it is in full, … the link from Errol Morris on Twitter, linking to his homepage:

<a href="http://www.errolmorris.com" class="url" rel="me nofollow" target="_blank">

But the Google SGAPI notices that the homepage doesn’t explicitly reciprocate the claim. There is no machine markup in the homepage to indicate that the owner of the page is saying that the twitter page “is me”. It might just be a link to a friend, for example.

Nevertheless, we can use the SGAPI in less trusting mode (’show inbound links’), and take advantage of Google’s massive Web index to ask: which pages claim to have the same owner as www.errolmorris.com?

This gives us a story about how we can find a lot of useful contextual information, given a basic starting point. The Google service I show here is just one of several that could be exploited in ‘linked tv’ scenarios.

For finding other identifiers, we might use sameas.org. Here’s what I get when I ask it for other URIs for the fog of war film, using a dbpedia uri as entry point:

1. http://dbpedia.org/resource/The_Fog_of_War
2. http://dbpedia.org/resource/11_Lessons_from_the_Life_of_Robert_S._Mcnamara
3. http://mpii.de/yago/resource/The_Fog_of_War
4. http://rdf.freebase.com/ns/guid.9202a8c04000641f80000000004e1a97

For ’social graph’ data, we might also check sindice.com or foaf.qdos.com.

Topic description

A quick word on topics and wikipedia: by using content identifiers that link to Wikipedia and DBpedia, we have the potential for extremely rich, fine grained topical annotations.

Wikipedia’s page about the movie notes that it covers the Cuban Missile Crisis. This association is now machine-visible, since I personally improved the relevant Wikipedia entry to explicitly link to the Cuban Missile Crisis page on Wikipedia. This allows us to use wikipedia:Cuban_Missile_Crisis as a topic indicator, not just against the entire film but against particular segments of the movie that are about that topic. It is easy to get a list of such links from the Wikipedia markup, eg. [[Ford Motor Company]], [[Vietnam War]] are also already there. If we have a TV presentation system that has a unique id for the content (and ideally content-version, since content often varies), and we know an offset in seconds, then rich topical tags could be applied to sections of the film, without the need for textual data entry. An iphone or similar device could allow users to pause, annotate/tag/bookmark and continue their viewing, with no need for a keyboard.

That smart-phone story there is worth investigating, but let’s first start closer to the screen: what kinds of set top box, media centre or gadget might be able to index, navigate and play content in a way that makes interesting use of ‘linked tv’ techniques such as these? Which brings me to APIs for ‘media centre‘ systems, finally.

If we want to recommend interesting content to users, show them relevant links, annotations, related materials (not necessarily more video – text, audio, even a spreadsheet with statistics might be appropriate), suggestions from friends, upcoming broadcasts or archived materials, then we need an environment that is scripting friendly and capable of interacting with users in a rich and compelling manner. The simplest path here is to start with what’s out there already, and look at commonalities in API and data model, to see how far existing software can be ‘remote controlled’ from external scripts (and, eventually, from actual hand-held remote controls, eg. smartphones). I’ll go into some detail on that in the followup post.

The main point I want to (re-)emphasise here, is that once we make get to the stage of having well known identifiers for content, it facilitates a very rich marketplace for TV-related metadata, with emphasis on the word “-related“. And that this is necessarily very open-ended, since TV content can be about absolutely anything. I gave some examples drawing on Wikipedia/DBpedia for content metadata, and on homepages, twitter and Google’s Social Graph API to show how additional highly relevant information can be pulled from the Web, once we get a basic starting point. Finding ways of presenting such extra information to users, and giving them appropriate navigation and interaction options, is far from easy. Fortunately it is easier to share, syndicate and merge TV meta-content, than TV content. We are already seeing systems such as the XBMC-based Boxee which will normalise content identifiers in a way that encourages legal uses over illegal. Intellectual properties issues around TV content means that actual playable content is often not broadly shareable. However the same need not be true of user-supplied metadata, since this can be about a specific media file, but also it can be about the things the content is about. I expect to see TV meta-content shared in a global linked system, even while the underlying video and audio remain relatively hidden away; rich user-supplied TV metadata isn’t just about the TV show, it equally can tell us about the world and the viewer, and deserves to be widely available through open standards.

To go back to my original example, Errol Morris’s film about Robert McNamara, The Fog of War… there’s a world of metadata options beyond five-star ratings that can enrich such content, beyond worrying about numerical ranking in statistical recommender systems. Existing work there, eg. around the Netflix prize, would be hard to beat. Statistical recommendations work well over a regular dataset where everything has a well known identifier. We can see RDF and linked data techniques serving a pre-processing role for such analysis, by linking together otherwise fragmented pieces of information about content, allowing classic techniques to be applied over a wider dataset.

However, I see the true ‘linked TV’ potential to be primarily in another direction: in creating more meaningful conversations around content, and helping users find other information that gives a complementary perspective on the materials, and on cross-referencing everything with everything so that unexpected new paths can be found. Morris’ film can teach us a lot about McNamara, but also about the wider world and recent history … about the Ford Motor Company, the Vietnam War, the Cuban Missile Crisis, the firebombing of Tokyo under the command of Curtis LeMay. Not all easy or pleasant topics to try to understand, and each with thousands of other relevant sources (video or otherwise) out there, different perspectives to cross-reference, different accounts to reconcile.

If television is an environment in which we can be informed, educated and entertained (by broadcasters, content creators, and increasingly, by everyone else too), we need to think through what this means for better metadata. Today we are lucky if we can find a good way of even identifying a piece of content. But tomorrow, we should have TV that comes out of the box with “Do You Want To Know More?” switched ‘on‘ by default.

When Obama gives a talk (see his June 2009 speech in Cairo) he mentioned the following, at around 33 minutes and 30 seconds:

For many years, Iran has defined itself in part by its opposition to my country, and there is in fact a tumultuous history between us.  In the middle of the Cold War, the United States played a role in the overthrow of a democratically elected Iranian government.

This last point is well covered by historians, but was likely to be new information for many viewers in the USA and UK, if not for those in Iran. What can we do to improve that, by bringing Web and TV closer together?

While the raw video, audio, and transcript are  available on the Whitehouse site, we don’t yet have enough for “do you want to know more?”. W3C’s TimedText work might provide a basis for associating the transcript with the video as subtitles. This is not enough for someone who wants to know more, in following up this unusual acknowledgement of superpower interference.

An Internet-literate, laptop-owning Web user might try to find out more in an active manner. Although they are in a minority amongst TV viewers, there are still thousands of such people, and they know how to go to google.com and do searches. Can we harness their energy to improve the TV metadata environment for everyone? If you search Google for ‘US overthrow iranian government’ you find a good starting point for learning more, a Wikipedia page “1953 Iranian coup d’état“. Although that page has content which Wikipedia flags as controversial, it has a world-visible talk page in which contributors from around the world can debate the detail of the topic. So how can this Web content be bridged with the world of TV?

TV viewers are commonly concerned characterised in terms of a pyramid:

• a passive majority, content to watch without interaction
• a smaller group who will interact and navigate, but who rarely create new content
• an even smaller group of activist users, who will explore, annotate, interact and create using all the tools available

The production and consumption of ‘do you want to know more?’ annotation won’t equally distributed. All viewers can benefit from, for example, a link from the Obama Cairo speech video to background information on the 1953 Iran coup. Some viewers might benefit indirectly, because their more inquisitive friends and family will have explored the linked materials and will talk to them about it. Others might read the wikipedia page and linked pages directly; if not on their television, then later on a laptop or PC.

Still others (a minority, but such minorities can be influential) might engage further with the material, become intrigued by the differences of opinion and seek more perspectives from the wider Web. Now 1953 is not ancient history; it falls within the living memory of countless people, in Iran, the UK and the world at large. A challenge for Linked TV is to find ways to integrate those memories and views into the TV environment, such that any viewer could pause the playing of the Obama Cairo speech video because they want to know more about what he just said. Regardless of whether they are watching on whitehouse.gov, YouTube, Joost, a live re-broadcast, a radio-extract, a player embedded in a social network site (facebook, orkut, hyves, …), or running on a mobile phone. The current problem Iranians have with getting full and high-bandwidth Web access are another obstacle, of course, but eventually we can expect relevant video or audio memories from Iran (with subtitles and translations) to be 1 or 2 clicks away from Obama’s video, regardless of the software and environment playing the video. Much of that material is already out there somewhere, the challenge is to find it, link it and present it appropriately.

The core piece of metadata, a link from a section of the video (ie. the offset, 33mins, 30 secs) of his speech to a URL for the topic it describes, could be created by whitehouse.gov staff, or it could be created by activist users. We pretty much know how to do this. A time-offset and a wikipedia URL do that job well enough for now. But what we have so far failed to achieve is a world-wide data ecosystem in which this kind of TV-enriching metadata is plentiful, widely used, and comprehensive enough that we can grow to expect to have fact-checkable TV, cross-referencable TV, hypertext TV. Instead of throwing physical things at the TV when we disagree with what we hear, it should become commonplace to be able to press pause and then or later publish your perspective (an audio rant, a fact-check crosslink, a blog post) in a way that will become accessible by other viewers of that material in the future.

To achieve this, we can’t afford needless fragmentation

• of meta-content about different versions of the same basic content
• between geographical regions based on geo-rights to play the material
• between desktop, media-centre, mobile, set top box, broadcast and on-demand scenarios
• between radio and TV
• between files and streams
• between file formats
• between populations (‘net censorship and crude blocking technology)

If these huge obstacles can be overcome, it might become reasonable to expect to find useful information attached to most of what we see on next-generation TV, regardless of delivery system. My hunch is the right place to prototype is around opensource and hackable media centre systems, and the right low-level communications system is XMPP. I’ll go into this more in the next post. Apologies for the length of this one. Comments welcomed by blog email or whatever!

I found I had to download the source from the Mercurial Web interface, rather than use pre-prepared XPI installers. This may have improved by the time you read this. I also learned (from Standard9 in #jetpack IRC) that I need asuth’s repository, rather than the main one. Again, things move quickly, don’t assume this is true forever.

Here is what worked for me, on OSX.

1. Grab a .zip from the Jetpack repo, and unpack it locally on a machine that has Thunderbird installed.

2. Edit extensions/install.rdf and make sure the em:maxVersion in the Thunderbird section matches your version of Thunderbird. In mine I updated it to say <em:maxVersion>3.0b4</em:maxVersion> (instead of 3.0b4pre).

3.  See the README in the jetpack filetree for installation. With Thunderbird closed, I ran “python manage.py install –app=thunderbird” and I found Jetpack installed fine.

4. Run Thunderbird, you should see an about:jetpack tab, and corresponding options in the Tools menu.

This was enough to get started. See discussion on visophyte.org for some example code.

After installation, you can use the about:jetpack windows to load, reload and delete Jetpacks from URL.

So, why would you bother doing all this? Jetpack provides a simple way of extending an email client using Web technology.

In my current (unfinished!) experiment, for example, I’m looking at making a sidebar the shows information (photo, blog etc.) about the sender of the currently-viewed email. And I figured that if I blogged this HOWTO, someone more familiar with ajax, jquery etc might care to help with wiring this up to the Google Social Graph JSON API, so we can use FOAF and XFN to provide more contextual information around incoming mail…

Assuming you are running Thunderbird 3b4

This online richness creates offline risk. For example, if you’re going away on holiday, there are hundreds of ways in which potential thieves could learn that your home is vacant and therefore a target for crime: shared calendars, twittered comments from friends or family, flickr’d photographs. Any of these could reveal that your home and possessions sit unwatched, unguarded, presenting an easy target for criminals.

Q: What research challenge does this present to the Semantic Web community? How can we address the concern that Semantic and Social Web technology have more to offer Burglar Bill than to his victims?

A1: We need better technology for limiting the flow of data, proving a right to legitimate access to information, cross-site protocols for deleting leaked or retracted data that flows between sites, and calculating trust metrics for parties requesting data access.

A2: We need to find ways to reconnect people with their neighbours and neighbourhoods, so that homes don’t sit unwatched when their occupants are away.

ps. Dear Bill, I have my iphone, laptop, piggy bank and camera with me…

The following is from my copy/paste from Twitter a few minutes ago. It gives a flavour of the chaos. Note also that observations from very popular users (such as stephenfry) can echo around for hours, often chased by attempts at clarification from others.

(“RT” is Twitter notation for re-tweet, meaning that the following content is redistributed, often in abbreviated or summarised form)

plotbunnytiff: RT @suffolkinace: RT From Iran: CONFIRMED!! Army moving into Tehran against protestors! PLEASE RT! URGENT! #IranElection
r0ckH0pp3r: RT .@AliAkbar: RT From Iran: CONFIRMED!! Army moving into Tehran against protesters! PLEASE RT! URGENT! #IranElection
jax3417: RT @ktyladie: RT @GennX: RT From Iran: CONFIRMED!! Army moving into Tehran against protesters! PLEASE RT! URGENT! #IranElection #iran
ktladie: RT @GennX: RT From Iran: CONFIRMED!! Army moving into Tehran against protesters! PLEASE RT! URGENT! #IranElection #iran
MellissaTweets: RT @AliAkbar: RT From Iran: CONFIRMED!! Army moving into Tehran against protesters! PLEASE RT! URGENT! #IranElection
GennX: RT @MelissaTweets: RT @AliAkbar: RT From Iran: CONFIRMED!! Army moving into Tehran against protesters! PLEASE RT! URGENT! #IranElection

The above all arrived at around the same time, and cite two prior “sources”:

suffolkinnace: RT From Iran: CONFIRMED!! Army moving into Tehran against protestors! PLEASE RT! URGENT! #IranElection   18 minutes ago from web

Who is this? Nobody knows of course, but there’s a twitter bio:

http://twitter.com/suffolkinace # Bio Some-to-be Royal Military Policeman in the British Army. Also a massive Xbox geek and part-time comedian

The other “source” seems to be http://twitter.com/AliAkbar
AliAkbar: RT From Iran: CONFIRMED!! Army moving into Tehran against protesters! PLEASE RT! URGENT! #IranElection
about 1 hour ago from web
url http://republicmodern.com

This leads us to   http://republicmodern.com/about where we’re told
“Ali Akbar is the founder and president of Republic Modern Media. A conservative blogger, he is a contributor to Right Wing News, Hip Hop Republican, and co-host of The American Resolve online radio show. He was also the editor-in-chief of Blogs for McCain.”

I should also mention that a convention emerged in the last day two replace the names of specific local Twitter users in Tehran with a generic “from Iran”, to avoid getting anyone into trouble. Which makes plenty of sense, but without any in the middle vouching for sources makes it even harder to know which reports to take seriously.
More… back to twitter search, what’s happened since I started this post?

http://twitter.com/#search?q=iranelection%20army

badmsm: RT @dpbkmb @judyrey: RT From Iran: CONFIRMED!! Army moving into Tehran against protesters! PLZ RT! URGENT! #IranElection #gr88
SimaoC: RT @parizot: CONFIRMÉ! L’armée se dirige vers Téhéran contre les manifestants! #IranElection #gr88
SpanishClash: RT @mytweetnickname: RT From Iran:ARMY movement NOT confirmed in last 2:15, plz RT this until confrmed #IranElection #gr88
artzoom: RT @matyasgabor @humberto2210: RT CONFIRMED!! Army moving into Tehran against protesters! PLEASE RT! #IranElection #iranrevolution
sjohnson301: RT @RonnyPohl From Iran: CONFIRMED!! Army moving into Tehran against protestors! PLEASE RT! URGENT! #IranElection #iran9
dauni: RT @withoutfield: RT: @tspe: CONFIRMED!! Army moving into Tehran against protestors! PLEASE RT! URGENT! #IranElection
interdigi: RT @ivanpinozas From Iran: CONFIRMED!! Army moving into Tehran against protestors! PLEASE RT! URGENT! #IranElection
PersianJustice: Once again, stop RT army movements until source INSIDE Iran verifies! Paramilitary is the threat anyway. #iranelection #gr88
Klungtveit Anyone: What’s the origin of reports of “army moving in” on protesters? #iranelection
Eruethemar: RT @brianlltdhq: RT @lumpuckaroo: Only IRG moving, not national ARMY… this is confirmed for real #IranElection #gr88
SAbbasRaza: RT @bymelissa: RT @alexlobov: RT From Iran: CONFIRMED!! Army moving into Tehran against protestors! PLEASE RT! URGENT! #IranElection
timnilsson: RT @Iridium24: CONFIRMED!! Army moving into Tehran against protesters! PLEASE RT! URGENT! #IranElection
edmontalvo: RT @jasona: RT @Marble68: RT From Iran: CONFIRMED!! Army moving into Tehran against protestors! PLEASE RT! URGENT! #IranElection
stevelabate: RT army moving into Tehran against protesters. Please RT. #iranelection
ivanpinozas: From Iran: CONFIRMED!! Army moving into Tehran against protestors! PLEASE RT! URGENT! #IranElection
bschh: CONFIRMED!! Army moving into Tehran against protestors! PLEASE RT! URGENT! #IranElection (via @dlayphoto)
dlayphoto: RT From Iran: CONFIRMED!! Army moving into Tehran against protestors! PLEASE RT! URGENT! #IranElection

In short … chaos!

Is this just a social / information problem, or can different tooling and technology help filter out what on earth is happening?

FOAF super-connectivity daydreams from 2002.

“indirectly inspired by John Pilger’s ‘The New Rulers of the World’, FOAFCorp and www.theyrule.net/”

This is the man, all tattered and torn, ...

“Because most of the targeted employees were male between the ages of 20 and 40 we decided that it would be best to become a very attractive 28 year old female. We found a fitting photograph by searching google images and used that photograph for our fake Facebook profile. We also populated the profile with information about our experiences at work by using combined stories that we collected from real employee facebook profiles.” [...]

Example below is a test against FAO’s AGROVOC. Each concept becomes a “document”, with a “word” field containing the prefLabel, and a “uri” field for the concept URI. I don’t index anything else yet.

The hope here is to have a handy prototyping environment for testing different indexing regimes. The code takes about 4-5 mins to index AGROVOC on my MacBook, running under Jruby.

The data I’m using is a SKOS dump from the FAO Web site, post-processed with “grep -v” to skip the Farsi lines, due to a Unicode error. The transcript below comes from running Lucli, a handy command line tool for Lucene.

Next steps with indexing? Not sure. Probably make sure altLabel is handled. But I’m also curious about possibility of including fields that pull in labels from nearby concepts, so they can be matched in weighted searches. Would be hard to evaluate the effectiveness though.

lucli> search uri:”http://www.fao.org/aims/aos/agrovoc#c_47934″
Searching for: uri:”http www.fao.org aims aos agrovoc c_47934″
1 total matching documents
————————————–
—————- 1 score:1.0———————
word:Pteria hirundo
uri:http://www.fao.org/aims/aos/agrovoc#c_47934
#################################################
lucli> search word:”Leiocottus hirundo”
Searching for: word:”leiocottus hirundo”
1 total matching documents
————————————–
—————- 1 score:1.0———————
word:Leiocottus hirundo
uri:http://www.fao.org/aims/aos/agrovoc#c_45393
#################################################
lucli> search word:”hirundo”
Searching for: word:hirundo
2 total matching documents
————————————–
—————- 1 score:1.0———————
word:Pteria hirundo
uri:http://www.fao.org/aims/aos/agrovoc#c_47934
—————- 2 score:1.0———————
word:Leiocottus hirundo
uri:http://www.fao.org/aims/aos/agrovoc#c_45393
#################################################