OpenStreetMap for disaster response – raw notes from Harry Wood talk

Very raw, sometimes verbatim but doubtless flawed notes from Harry Wood‘s excellent talk at Open Data Institute in London. #odifridays

Many thanks to Harry for a great talk and to ODI for putting together these lunchtime lectures. The ODI have also published slides and audio from the talk.

“An introduction to OpenStreetMap, the UK born project to map the world as open data, and a look at how volunteer mappers helped with disaster response in the Philippines after Typhoon Haiyan, with Harry Wood . Harry is a developer at transportAPI.com, and is on the board of the Humanitarian OpenStreetMap Team.”

Note: this is un-checked, very raw notes that I typed while listening. There will be mistakes and confusions; my fault not Harry’s!

Typhoons …phillipines area hammered during typhoon season. The typhoons often meander off, don’t hit coast. But this one hit, and fast, … so a big storm surge. Fastest wind speeds on record ‘biggest storm ever’.

[shows video clip]

More than 6000 died. Role of mapping in disaster responses: food shelter etc; can donate money directly for giving food.  Info / logistics challenge re delivering aid. Lots of ‘where?’ questions. Where are people suffering most;? Where to deliver aid to? team locations etc. Huge value of maps for disaster response.

Q: who has edited OSM? A: lots of hands raised.

Maps … GIS / vector data will always be a bit complex, but we try to dumb it down. The data model is also v stripped down, just tagged nodes and ways. e.g. a pub is a node with amenity=pub. It’s also renderable map -> viewed as a map on openstreetmap.org, … but we play down that aspect a bit, since there are other map providers around e.g. Google.

But the maps are a important aspect of disaster response.

OSM editing -> appear on map can take ~ 10 mins.

This is quite a technical hit. There’s a rendering server here in London; aspect of providing a feedback loop (editing -> new map).  A shared commons for geo data. AID orgs get excited … coming together sharing same platform. OSM is very much about raw data too, not just the maps. So this is different to pure map providers, … entirely open access to the raw data.

In terms of the humanitarian response, … agencies can take the data unencumbered, use it offline. It is open data. there is an exciting open data story for OSM.

As humanitarian work, it can be a problem that we allow commercial re-use – [not all orgs welcome that]

Community + Raw vector data + simple editing + Updated map — these 4 elements make it very attractive to humanitarian work.

Haiti in 2010, collab came together very quickly, for the two worst-hit cities (port au Prince and …). This speed was v useful for aid orgs; those orgs were printing it out, in tents, response centres. People used it on the web too, Ushahidi too, ie they’re a bit more accurate due to these improvements.

“my favourite use: ” … a Garmin handheld GPS unit, … loaded with data from open ecosystem, used offline  quintessential use case of raw data from OSM but also life-saving. Since haiti, there have been other disasters. Not all of these so suited to OSM helping out – e.g. massive pakistan floods, … harder to map such a larger area. Couldn’t get imagery for that entire area.

To some extent there are pakistan maps already; less so for Haiti. Similarly re Japan, already were maps.

re Sendai tsunami, .. yes there were free maps; yes there were high quality official maps, … but could you get hold of recently updated freely avail high quality maps? so still some role there.

Since then, organizing more: Tasking Manager, tasks.hotosm.org

A common Q: ‘where to start mapping?’

Way of coordinating for a large area. Drop a grid, get people to acquire a square, … load into editor, ‘done’ when done. This workflow came into its own during Philippines. Sometimes in resp to an aid agency request, … or as we have imagery, … Visualizing changesets, .. bounding boxes slide, Philippines editing traffic [slide] brand new, made last night,  … got up to almost 300 users involved on 1 day. No of changes (philippines) ~ 40,000 edits.


Peak in interest corresponds in interest, corresponds to general interest [shows google trends], though shows a slightly longer attention span. Want the spike further over to the left,… the sooner the better, e.g. as aid agencies may be taking a snapshot of our data, …

Graph showing new users … ppl who appear to have registered during the time of the disaster response,  shows also ‘old timers’ getting engaged earlier, few days lag for the newer users.

We have a humanitarian mapping style, … not the default OSM view  we tweaked it slightly – e.g. to show a red outline around buildings appear to be damaged. Getting mappers to look at post-disaster imagery, e.g. buildings that have been swept away with water. More examples of data getting used: map posters popular with aid agencies; they fly out now with a cardboard roll full of osm posters. In particular red cross heavily involved.

In UK office down in Moorgate, used tasking manager there, contributing to OSM to improve the printouts they were getting. Ways to help: hot.openstreetmap.org/donate; comms, blogging, coordination, wiki, promo videos and tutorials, imagery warping / tiling / hosting;

software dev’t, use the open data; build tools to work with it, …

Gateway skill: learn to map!

[end]

A Quick demo.

  • Shows tasking manager UI from tasks.hotosm.org

  • colour coded squares either mapped, or mapped and validated by a 2nd reviewer

  • click to acquire a square, then to invoke OSM editor of choice e.g. JOSM

  • alternative – edit directly in website via js-based UI

  • we tend to teach new users the JOSM GUI

  • shows workflow of marking a road (nodes/ways) picking up from work in ‘someone else’s square’

Comment from a Nigel of mapaction http://www.mapaction.org/ v supportive, ‘used it all the time’. ‘last few emergencies, … this stuff is pervasive, if it wasn’t there we’d be really struggling’.

Comment from Andrew B… (british red cross) the volunteer aspect as well, … between us and mapaction, it’s the volunteers that make it happen, …

Q to audience, for Haiyan, lessons?

Andew points to row of British red cross mapping volunteers – ‘we’re coordinating w/ US red cross, federation, …  they’re dealing with those in the area; whereas Nigel is using it on the ground in this area that’s going out tomorrow. We were doing situational reports, who-what-where-when eg risk vs need vs capabilities, … understanding that kind of stuff. This gives us underlying map, to support all this.

Q re coordination. Nigel of mapaction “Maps are coordination glue”; Harry  “everything has a location aspect.”

Ed Parsons Q: “v interested  in task manager element … if you had the info before, that’s hugely valuable how successful ? how do you motivate ppl to map an area they’ve not thought about?

Harry: many people motivated by seeing it on the news, …in  a way a shame as better if happens ahead of time, … work on that under disaster risk reduction. e.g. in Indonesia we have extensive mapping work, as it lies on a fault line, risk assessment, … trying to get a map of every building, get people to draw around buildings . But there’s less enthusiasm for these things before they’re needed.

Harry: gratifying that tasking manager is software we’ve dev’t reasonably, that HOT as an org has matured as a community, we have etiquette ar ound using the tasking manager, fell into place naturally.

Ivan (doctors without borders): 2nds Nigel’s point  re lifesaving we have years, decades of health data. People telling us where ppl are from … To understand epidemic patterns … in haiti we couldn’t find src of the outbreaks (despite Snow/cholera analogy) … because we can’t get raw usable data that correlates to what people report as their place of origin and where they got sick. Wokring w/ OSM. Some success in Phillipines, … more challenge in Congo & other area. Aim to be able to correlate places reported from walk-in patients to a real world place, and get into forecasting and preventative medicine. Struggling to achieve in these situations what europe had 150 years ago.


Harry: importance of geocoding from names

Ivan: every person in the world has some description for where they live If it’s a streetname/number that’s easy; if its’ directions from a b-b tree that’s harder. But can make a start, ideally 1-200M, but kilometers better than nothing. We sit on piles of data that we can’t correlate to anything so far.

Biggest single impediment is access to imagery? get other providers to do as BIng…

Harry: the challenge for imagery providers is that it is worth money, which is why they put satellites or fly the planes, so can’t eat away at that too much. OFten you’ll see data made availaable temporarily after emergency . For example re Pakistan, downgraded/fuzzy quality data was shared. There are some agreements in place, … us govt put in place frameworks to source data. For example that the imagery can only be used in an OSM editor. But need to be able to derrive vector data from it (hence there are issues with using Google imagery in this way).

Luke Cayley(sp?): https://www.gov.uk/government/organisations/department-for-international-development (missed). concur w/ Nigel, Ivan. Q re imagery: have you tried to get it from European disaster mechanism, Copernicus, which has some provisions for disaster readyness prep.

Harry: will follow up on this.

Luke Q: how aid agencies use the raw data in the field to collect data? eg. MSF, … What’s your feeling for the barriers to making this a well recognised procedure, using OSM as one of the tools to make that happen?

Harry: did start to discuss re Phillipines, … about data into OSM from on the ground teams. With Haiti streetnames you don’t get them from the raw imagery so needs on-the-ground gathering. The process for on the ground gathering is pretty mature around OSM, tools, mobile apps etc. But a case of getting ppl interested in doing that. In diaster response situatoin, it is hard to tell peopel they ought to be writing down names of streets.

Q: for DFID Luke, … a number of funds are available. Because OSM is a global public good, it is the kind of thing DFID would tend to be supportive of in funding proposals (but can’t promise).

Harry: re diaster situation, often it won’t be a priority during the diaster to collect street names. All it takes is geo-located photos, a snap of a street sign.

Q from someone called Chris … you spoke of satellite imagery as source for mapping. Are you exploring use of pro-sumer vs aerial imagery?

Harry: Satellite imagery is now approaching aerial photography quality, but remains expensive due to operational cost. Another cost is the vast amount of disk space, bandwidth, hosting costs. These problems are not insurmountable. OSM and HOT have some resources to help here – ‘talk to us’.  Aerial imagery historically has been better. If you look at Bing or Google ‘satellite’ images they’re often from planes, so yes, that can help. Also new area of drones over small (but maybe important) areas.

WordPress, TinyMCE and RDFa editors

I’m writing this in WordPress’s ‘Visual’ mode WYSIWYG HTML editor, and thinking “how could it be improved to support RDFa?”

Well let’s think. Humm. In RDFa, every section of text is always ‘about’ something, and then has typed links or properties associated with that thing. So there are icons ‘B’ for bold, ‘I’ for italics, etc. And thingies for bulleted lists, paragraphs, fonts. I’m not sure how easily it would handle the nesting of full RDFa, but some common case could be a start: a paragraph that was about some specific thing, and within which the topical focus didn’t shift.

So, here is a paragraph about me. How to say it’s really about me? There could be a ‘typeof’ attribute on the paragraph element, with value foaf:Person, and an ‘about’ attribute with a URI for me, eg. http://danbri.org/foaf.rdf#danbri

In this UI, I just type textual paragraphs and then double newlines are interpreted by the publishing engine as separate HTML paragraphs. I don’t see any paragraph-level or even post-level properties in the user interface where an ‘about’ URI could be stored. But I’m sure something could be hacked. Now what about properties and links? Let’s see…

I’d like to link to TinyMCE now, since that is the HTML editor that is embedded in WordPress. So here is a normal link to TinyMCE. To make it, I searched for the TinyMCE homepage, copied its URL into the copy/paste buffer, then pressed the ‘add link’ icon here after highlighting the text I wanted to turn into a link.

It gave me a little HTML/CSS popup window with some options – link URL, target window, link title and (CSS) class; I’ve posted a screenshot below. This bit of UI seems like it could be easily extended for typed links. For example, if I were adding a link to a paragraph that was about a Film, and I was linking to (a page about) its director or an actor, we might want to express that using an annotation on the link. Due to the indirection (we’re trying to say that the link is to a page whose primary topic is the director, etc.), this might complicate our markup. I’ll come back to that later.

tinymce link edit screenshop

tinymce link edit screenshot

I wonder if there’s any mention of RDFa on the TinyMCE site? Nope. Nor even any mention of Microformats.

Perhaps the simplest thing to try to build into TinyMCE would be for a situation where a type on the link expressed a relationship between the topic of the blog post (or paragraph), and some kind of document.

For example, a paragraph about me, might want to annotate a link to my old school’s homepage (ie. http://www.westergate.w-sussex.sch.uk/ ) with a rel=”foaf:schoolHomepage” property.

So our target RDFa would be something like (assuming this markup is escaped and displayed nicely):

<p about=”http://danbri.org/foaf.rdf#danbri” typeof=”foaf:Person” xmlns:foaf=”http://xmlns.com/foaf/0.1/”>

I’m Dan, and I went to <a rel=”foaf:schoolHomepage” href=”http://www.westergate.w-sussex.sch.uk/”>Westergate School</a>

</p>.

See recent discussion on the foaf-dev list where we contrast this idiom with one that uses a relationship (eg. ‘school’) that directly links a person to a school, rather than to the school’s homepage. Toby made some example RDFa output of the latter style, in response to a debate about whether the foaf:schoolHomepage idiom is an ‘anti-pattern’. Excerpt:

<p about=”http://danbri.org/foaf.rdf#danbri” typeof=”foaf:Person” xmlns:foaf=”http://xmlns.com/foaf/0.1/”>

My name is <span property=”foaf:name”>Dan Brickley</span> and I spent much of the ’80s at <span rel=”foaf:school”><a typeof=”foaf:Organization” property=”foaf:name” rel=”foaf:homepage” href=”http://www.westergate.w-sussex.sch.uk/”>Westergate School</a>.

</p>

So there are some differences between these two idioms. From the RDF side they tell you similar things: the school I went to. The latter idiom is more expressive, and allows additional information to be expressed about the school, without mixing it up with the school’s homepage. The former simply says “this link is to the homepage of Dan’s school”; and presumably defers to the school site or elsewhere for more details.

I’m interested to learn more about how these expressivity tradeoffs relate to the possibilities for visual / graphical editors. Could TinyMCE be easily extended to support either idiom? How would the GUI recognise a markup pattern suitable for editing in either style? Can we avoid having to express relationships indirectly, ie. via pages? Would it be a big job to get some basic RDFa facilities into WordPress via TinyMCE?

For related earlier work, see the 2003 SWAD-Europe report on Semantic blogging from the team then at HP Labs Bristol.

JQbus: social graph query with XMPP/SPARQL

Righto, it’s about time I wrote this one up. One of my last deeds at W3C before leaving at the end of 2005, was to begin the specification of an XMPP binding of the SPARQL querying protocol. For the acronym averse, a quick recap. XMPP is the name the IETF give to the Jabber messaging technology. And SPARQL is W3C’s RDF-based approach to querying mixed-up Web data. SPARQL defines a textual query language, an XML result-set format, and a JSON version for good measure. There is also a protocol for interacting with SPARQL databases; this defines an abstract interface, and a binding to HTTP. There is as-yet no official binding to XMPP/Jabber, and existing explorations are flawed. But I’ll argue here, the work is well worth completing.

jqbus diagram

So what do we have so far? Back in 2005, I was working in Java, Chris Schmidt in Python, and Steve Harris in Perl. Chris has a nice writeup of one of the original variants I’d proposed, which came out of my discussions with Peter St Andre. Chris also beat me in the race to have a working implementation, though I’ll attribute this to Python’s advantages over Java ;)

I won’t get bogged down in the protocol details here, except to note that Peter advised us to use IQ stanzas. That existing work has a few slight variants on the idea of sending a SPARQL query in one IQ packet, and returning all the results within another, and that this isn’t quite deployable as-is. When the result set is too big, we can run into practical (rather than spec-mandated) limits at the server-to-server layers. For example, Peter mentioned that jabber.org had a 65k packet limit. At SGFoo last week, someone suggested sending the results as an attachment instead; apparently this one of the uncountably many extension specs produced by the energetic Jabber community. The 2005 work was also somewhat partial, and didn’t work out the detail of having a full binding (eg. dealing with default graphs, named graphs etc).

That said, I think we’re onto something good. I’ll talk through the Java stuff I worked on, since I know it best. The code uses Ignite Online’s Smack API. I have published rough Java code that can communicate with instances of itself across Jabber. This was last updated July 2007, when I fixed it up to use more recent versions of Smack and Jena. I forget if the code to parse out query results from the responses was completed, but it does at least send SPARQL XML results back through the XMPP network.

sparql jabber interaction

sparql jabber interaction

So why is this interesting?

  • SPARQLing over XMPP can cut through firewalls/NAT and talk to data on the desktop
  • SPARQLing over XMPP happens in a social environment; queries are sent from and to Jabber addresses, while roster information is available which could be used for access control at various levels of granularity
  • XMPP is well suited for async interaction; stored queries could return results days or weeks later (eg. job search)
  • The DISO project is integrating some PHP XMPP code with WordPress; SparqlPress is doing same with SPARQL

Both SPARQL and XMPP have mechanisms for batched results, it isn’t clear which, if either, to use here.

XMPP also has some service discovery mechanisms; I hope we’ll wire up a way to inspect each party on a buddylist roster, to see who has SPARQL data available. I made a diagram of this last summer, but no code to go with it yet. There is also much work yet to do on access control systems for SPARQL data, eg. using oauth. It is far from clear how to integrate SPARQL-wide ideas on that with the specific possibilities offered within an XMPP binding. One idea is for SPARQL-defined FOAF groups to be used to manage buddylist rosters, “friend groups”.

Where are we with code? I have a longer page in FOAF SVN for the Jqbus Java stuff, and a variant on this writeup (includes better alt text for the images and more detail). The Java code is available for download. Chris’s Python code is still up on his site. I doubt any of these can currently talk to each other properly, but they show how to deal with XMPP in different languages, which is useful in itself. For Perl people, I’ve uploaded a copy of Steve’s code.

The Java stuff has a nice GUI for debugging, thanks to Smack. I just tried a copy. Basically I can run a client and a server instance from the same filetree, passing it my LiveJournal and Google Talk jabber account details. The screenshot here shows the client on the left having the XML-encoded SPARQL results, and the server on the right displaying the query that arrived. That’s about it really. Nothing else ought to be that different from normal SPARQL querying, except that it is being done in an infrastructure that is more socially-grounded and decentralised than the classic HTTP Web service model.

JQbus debug