OpenStreetMap for disaster response – raw notes from Harry Wood talk

Very raw, sometimes verbatim but doubtless flawed notes from Harry Wood‘s excellent talk at Open Data Institute in London. #odifridays

Many thanks to Harry for a great talk and to ODI for putting together these lunchtime lectures. The ODI have also published slides and audio from the talk.

“An introduction to OpenStreetMap, the UK born project to map the world as open data, and a look at how volunteer mappers helped with disaster response in the Philippines after Typhoon Haiyan, with Harry Wood . Harry is a developer at transportAPI.com, and is on the board of the Humanitarian OpenStreetMap Team.”

Note: this is un-checked, very raw notes that I typed while listening. There will be mistakes and confusions; my fault not Harry’s!

Typhoons …phillipines area hammered during typhoon season. The typhoons often meander off, don’t hit coast. But this one hit, and fast, … so a big storm surge. Fastest wind speeds on record ‘biggest storm ever’.

[shows video clip]

More than 6000 died. Role of mapping in disaster responses: food shelter etc; can donate money directly for giving food.  Info / logistics challenge re delivering aid. Lots of ‘where?’ questions. Where are people suffering most;? Where to deliver aid to? team locations etc. Huge value of maps for disaster response.

Q: who has edited OSM? A: lots of hands raised.

Maps … GIS / vector data will always be a bit complex, but we try to dumb it down. The data model is also v stripped down, just tagged nodes and ways. e.g. a pub is a node with amenity=pub. It’s also renderable map -> viewed as a map on openstreetmap.org, … but we play down that aspect a bit, since there are other map providers around e.g. Google.

But the maps are a important aspect of disaster response.

OSM editing -> appear on map can take ~ 10 mins.

This is quite a technical hit. There’s a rendering server here in London; aspect of providing a feedback loop (editing -> new map).  A shared commons for geo data. AID orgs get excited … coming together sharing same platform. OSM is very much about raw data too, not just the maps. So this is different to pure map providers, … entirely open access to the raw data.

In terms of the humanitarian response, … agencies can take the data unencumbered, use it offline. It is open data. there is an exciting open data story for OSM.

As humanitarian work, it can be a problem that we allow commercial re-use – [not all orgs welcome that]

Community + Raw vector data + simple editing + Updated map — these 4 elements make it very attractive to humanitarian work.

Haiti in 2010, collab came together very quickly, for the two worst-hit cities (port au Prince and …). This speed was v useful for aid orgs; those orgs were printing it out, in tents, response centres. People used it on the web too, Ushahidi too, ie they’re a bit more accurate due to these improvements.

“my favourite use: ” … a Garmin handheld GPS unit, … loaded with data from open ecosystem, used offline  quintessential use case of raw data from OSM but also life-saving. Since haiti, there have been other disasters. Not all of these so suited to OSM helping out – e.g. massive pakistan floods, … harder to map such a larger area. Couldn’t get imagery for that entire area.

To some extent there are pakistan maps already; less so for Haiti. Similarly re Japan, already were maps.

re Sendai tsunami, .. yes there were free maps; yes there were high quality official maps, … but could you get hold of recently updated freely avail high quality maps? so still some role there.

Since then, organizing more: Tasking Manager, tasks.hotosm.org

A common Q: ‘where to start mapping?’

Way of coordinating for a large area. Drop a grid, get people to acquire a square, … load into editor, ‘done’ when done. This workflow came into its own during Philippines. Sometimes in resp to an aid agency request, … or as we have imagery, … Visualizing changesets, .. bounding boxes slide, Philippines editing traffic [slide] brand new, made last night,  … got up to almost 300 users involved on 1 day. No of changes (philippines) ~ 40,000 edits.


Peak in interest corresponds in interest, corresponds to general interest [shows google trends], though shows a slightly longer attention span. Want the spike further over to the left,… the sooner the better, e.g. as aid agencies may be taking a snapshot of our data, …

Graph showing new users … ppl who appear to have registered during the time of the disaster response,  shows also ‘old timers’ getting engaged earlier, few days lag for the newer users.

We have a humanitarian mapping style, … not the default OSM view  we tweaked it slightly – e.g. to show a red outline around buildings appear to be damaged. Getting mappers to look at post-disaster imagery, e.g. buildings that have been swept away with water. More examples of data getting used: map posters popular with aid agencies; they fly out now with a cardboard roll full of osm posters. In particular red cross heavily involved.

In UK office down in Moorgate, used tasking manager there, contributing to OSM to improve the printouts they were getting. Ways to help: hot.openstreetmap.org/donate; comms, blogging, coordination, wiki, promo videos and tutorials, imagery warping / tiling / hosting;

software dev’t, use the open data; build tools to work with it, …

Gateway skill: learn to map!

[end]

A Quick demo.

  • Shows tasking manager UI from tasks.hotosm.org

  • colour coded squares either mapped, or mapped and validated by a 2nd reviewer

  • click to acquire a square, then to invoke OSM editor of choice e.g. JOSM

  • alternative – edit directly in website via js-based UI

  • we tend to teach new users the JOSM GUI

  • shows workflow of marking a road (nodes/ways) picking up from work in ‘someone else’s square’

Comment from a Nigel of mapaction http://www.mapaction.org/ v supportive, ‘used it all the time’. ‘last few emergencies, … this stuff is pervasive, if it wasn’t there we’d be really struggling’.

Comment from Andrew B… (british red cross) the volunteer aspect as well, … between us and mapaction, it’s the volunteers that make it happen, …

Q to audience, for Haiyan, lessons?

Andew points to row of British red cross mapping volunteers – ‘we’re coordinating w/ US red cross, federation, …  they’re dealing with those in the area; whereas Nigel is using it on the ground in this area that’s going out tomorrow. We were doing situational reports, who-what-where-when eg risk vs need vs capabilities, … understanding that kind of stuff. This gives us underlying map, to support all this.

Q re coordination. Nigel of mapaction “Maps are coordination glue”; Harry  “everything has a location aspect.”

Ed Parsons Q: “v interested  in task manager element … if you had the info before, that’s hugely valuable how successful ? how do you motivate ppl to map an area they’ve not thought about?

Harry: many people motivated by seeing it on the news, …in  a way a shame as better if happens ahead of time, … work on that under disaster risk reduction. e.g. in Indonesia we have extensive mapping work, as it lies on a fault line, risk assessment, … trying to get a map of every building, get people to draw around buildings . But there’s less enthusiasm for these things before they’re needed.

Harry: gratifying that tasking manager is software we’ve dev’t reasonably, that HOT as an org has matured as a community, we have etiquette ar ound using the tasking manager, fell into place naturally.

Ivan (doctors without borders): 2nds Nigel’s point  re lifesaving we have years, decades of health data. People telling us where ppl are from … To understand epidemic patterns … in haiti we couldn’t find src of the outbreaks (despite Snow/cholera analogy) … because we can’t get raw usable data that correlates to what people report as their place of origin and where they got sick. Wokring w/ OSM. Some success in Phillipines, … more challenge in Congo & other area. Aim to be able to correlate places reported from walk-in patients to a real world place, and get into forecasting and preventative medicine. Struggling to achieve in these situations what europe had 150 years ago.


Harry: importance of geocoding from names

Ivan: every person in the world has some description for where they live If it’s a streetname/number that’s easy; if its’ directions from a b-b tree that’s harder. But can make a start, ideally 1-200M, but kilometers better than nothing. We sit on piles of data that we can’t correlate to anything so far.

Biggest single impediment is access to imagery? get other providers to do as BIng…

Harry: the challenge for imagery providers is that it is worth money, which is why they put satellites or fly the planes, so can’t eat away at that too much. OFten you’ll see data made availaable temporarily after emergency . For example re Pakistan, downgraded/fuzzy quality data was shared. There are some agreements in place, … us govt put in place frameworks to source data. For example that the imagery can only be used in an OSM editor. But need to be able to derrive vector data from it (hence there are issues with using Google imagery in this way).

Luke Cayley(sp?): https://www.gov.uk/government/organisations/department-for-international-development (missed). concur w/ Nigel, Ivan. Q re imagery: have you tried to get it from European disaster mechanism, Copernicus, which has some provisions for disaster readyness prep.

Harry: will follow up on this.

Luke Q: how aid agencies use the raw data in the field to collect data? eg. MSF, … What’s your feeling for the barriers to making this a well recognised procedure, using OSM as one of the tools to make that happen?

Harry: did start to discuss re Phillipines, … about data into OSM from on the ground teams. With Haiti streetnames you don’t get them from the raw imagery so needs on-the-ground gathering. The process for on the ground gathering is pretty mature around OSM, tools, mobile apps etc. But a case of getting ppl interested in doing that. In diaster response situatoin, it is hard to tell peopel they ought to be writing down names of streets.

Q: for DFID Luke, … a number of funds are available. Because OSM is a global public good, it is the kind of thing DFID would tend to be supportive of in funding proposals (but can’t promise).

Harry: re diaster situation, often it won’t be a priority during the diaster to collect street names. All it takes is geo-located photos, a snap of a street sign.

Q from someone called Chris … you spoke of satellite imagery as source for mapping. Are you exploring use of pro-sumer vs aerial imagery?

Harry: Satellite imagery is now approaching aerial photography quality, but remains expensive due to operational cost. Another cost is the vast amount of disk space, bandwidth, hosting costs. These problems are not insurmountable. OSM and HOT have some resources to help here – ‘talk to us’.  Aerial imagery historically has been better. If you look at Bing or Google ‘satellite’ images they’re often from planes, so yes, that can help. Also new area of drones over small (but maybe important) areas.

Querying Linked GeoData with R SPARQL client

Assuming you already have the R statistics toolkit installed, this should be easy.
Install Willem van Hage‘s R SPARQL client. I followed the instructions and it worked, although I had to also install the XML library, which was compiled and installed when I typed install.packages(“XML“, repos = “http://www.omegahat.org/R“) ‘ within the R interpreter.
Yesterday I set up a simple SPARQL endpoint using Benjamin Nowack’s ARC2 and RDF data from the Ravensburg dataset. The data includes category information about many points of interest in a German town. We can type the following 5 lines into R and show R consuming SPARQL results from the Web:
  • library(SPARQL)
  • endpoint = “http://foaf.tv/hypoid/sparql.php
  • q = “PREFIX vcard: <http://www.w3.org/2006/vcard/ns#>\nPREFIX foaf:\n<http://xmlns.com/foaf/0.1/>\nPREFIX rv:\n<http://www.wifo-ravensburg.de/rdf/semanticweb.rdf#>\nPREFIX gr:\n<http://purl.org/goodrelations/v1#>\n \nSELECT ?poi ?l ?lon ?lat ?k\nWHERE {\nGRAPH <http://www.heppresearch.com/dev/dump.rdf> {\n?poi\nvcard:geo ?l .\n  ?l vcard:longitude ?lon .\n  ?l vcard:latitude ?lat\n.\n ?poi foaf:homepage ?hp .\n?poi rv:kategorie ?k .\n\n}\n}\n”
  • res<-SPARQL(endpoint,q)
  • pie(table(res$k))

This is the simplest thing that works to show the data flow. When combined with richer server-side support (eg. OWL tools, or spatial reasoning) and the capabilities of R plus its other extensions, there is a lot of potential here. A pie chart doesn’t capture all that, but it does show how to get started…

Note also that you can send any SPARQL query you like, so long as the server understands it and responds using W3C’s standard XML response. The R library doesn’t try to interpret the query, so you’re free to make use of any special features or experimental extensions understood by the server.

Local Video for Local People

OK it’s all Google stuff, but still good to see. Go to Google Maps, My Maps, to find ‘Videos from YouTube’ listed. Here’s where I used to live (Bristol UK) and where I live now (Amsterdam, The Netherlands). Here’s a promo film of some nearby art installations from ArtZuid, who even have a page in English. I wouldn’t have found the video or the nearby links except through the map overlay. I don’t know exactly how they’re geotagging the videos, I can’t see an option under ‘my videos’ in YouTube, so perhaps it’s automatic or viewer annotations. In YouTube, you can add a map link under ‘My Videos’ / ‘Edit Video'; I didn’t see that initially. I made some investigations into similar issues (videos on maps) while at Joost; see brief mention in my Fundamentos Web slides from a couple of years ago.
Oh, nearly forgot to mention: zooming out to get a Europe or World-wide view is quite striking too.

Fireeagle IRC bot / OAuth / Dopplr

I noticed people in #geo playing with a fireeagle bot earlier, so I had to try too.

danbri: fireup, help?
[13:29] fireeagle: danbri: I have just sent you a URL in privmsg, please click on it to authorize.

[time passes... things happen behind the scenes...]

[13:31] danbri: fireup Leiden, Netherlands
[13:31] fireeagle: Updating danbri to Leiden, Nederland
[13:32] • danbri gives fireeagle a botsnack

Behind the scenes, the bot passed me an URL:

fireeagle: Try Again! You must authorize first: https://fireeagle.yahoo.net/oauth/authorize?oauth_token=XYZABCD

And clicking this took me off into the Fireeagle Web UI, where I had to reconfirm my membership (3 months had lapsed; it times out if you don’t show signs of life). Then I had to give the IRC bot permission to access my Fireeagle data, and finally it send me to a duff return URL,

http://0.0.0.0:3000/dummy_app/callback?name=Honking+Fun&oauth_token=XYZABCD

…which is understandable-ish given that IRC bots aren’t normal Web apps, though I wonder if an IRC URI would be allowed per the OAuth spec. I know there is work on wiring XMPP and OAuth together, after all.

So now Fireeagle and downstream apps (eg. the MacOSX widget pictured here) can learn about my location when I tell this IRC bot where I am.

So logged into Fireeagle Web site now I read:

Fire Eagle last spotted you about 1 hour ago in Leiden, Nederland using Fire Eagle GeoIRC bot. If you’ve gone somewhere else, then you should update your location!

And here’s the corresponding MacOSX widget:

 Fireeagle widget

Since I don’t use nickserv on the OFTC IRC network, this means anyone logged in there as ‘danbri’ can set my location data at Fireeagle. But at least they can’t add Facebook buddies, read my private email or ‘your password has been reset’ messages. And if they do, Fireeagle lets me revoke this token without messing up anything else.

Now, how to convince Dopplr to believe what it oauth-reads at Fireeagle? Logging into Dopplr by OpenID (hurrah :), I read the following falsehood:

…and when I try to tell it I’m actually in the Netherlands for a few days, and that my June 17th Dopplr journal entry “You returned from a trip to Guadalajara.” wasn’t a return to Bristol, it gets stuck: “Sorry, we couldn’t add that trip: You need a return date“. On closer inspection this seems to be my error; Dopplr has a new facility for multi-stop trips and if I’d added a Leiden stage to this last trip before it seemingly finished, Dopplr, Fireeagle and the IRC bot would be in agreement about my location.

So what’s so interesting about all this? Partly that OAuth provides a reasonable and deployable model for wiring up data pipes between socially oriented Web sites (and with some provisos, non-Web UI too), so that each can specialise in some task or domain. But also that, when the specialities of these sites and services do overlap, we’ll get into the business of conflict resolution amongst competing claims. When Dopplr reads that Fireeagle thinks I’m in Leiden, despite the last trip it knows about supposedly returning me to Bristol on the 17th, what should it do? Perhaps update its own state to a “well nobody told me, but it seems he’s in the netherlands” position?

ps. Fireeagle remains invite-only, and I don’t have any invites. Sorry about that, for those who haven’t tried it yet. There are some screenshots and a writeup at TechCrunch at least. Basically it’s an information broker for your location data, with data access mediated largely through OAuth REST APIs. For those with an invite, the developer docs have more detail than I’ve yet managed to digest.

AllegroGraph RDFStore 3.0: Social Network Analysis

AllegroGraph 3.0 now comes with a Social Network Analysis component, amongst several other interesting features including improved geo support.

example diagram of people and relationships

By viewing interactions as connections a in graph, we can treat a multitude of different situations using the tools of Social Network Analysis (SNA). SNA lets us answer questions like:

    • How closely connected are any two individuals?
    • What are the core groups or clusters within the data?
    • How important is this person (or company) to the flow of information?
    • How likely is it that this person and that person know one another?

The field is full of rich mathematical techniques and powerful algorithms. AllegroGraph’s SNA toolkit includes an array of search methods, tools for measuring centrality and importance, and the building blocks for creating more specialized measures. These tools can be used with any “network” data set whether its connections between companies and directors, predator/prey food webs, chemical interactions, or links between web sites.

Geographic Queries on Google App Engine

Much cleverness:

In this way, I was able to put together a geographic bounding box query, on top of Google App Engine, using a Geohash-like algorithm as a storage format, and use that query to power a FeatureServer Demo App Engine application, doing geographic queries of non-point features on top of App Engine/BigTable. Simply create a Geoindex object of the bounding box of your feature, and then use lower-left/upper-right points as bounds for your Geohash when querying.

Open CellID databases

Via momolondon list: opencellid.org data dumps

The readme.txt file describes the tabular data structure (split into a cells, and a measures file).

I think the cells data is the one most folk will be interested in re-using. Table headings are:

# id,lat,lon,mcc,mnc,lac,cellid,range,nbSamples,created_at,updated_at
For example:
7,44.8802,-0.526878,208,10,18122,32951790,0,2,2008-03-31 15:22:22,2008-04-07 08:57:33

This could be RDFized using something similar to the (802.11-centric) Wireless Ontology. Perhaps even using lqraps

Google Earth touring via KML

While I’m writing up old hacks, here’s one that I really enjoyed, even if it was a bit clunky. A couple of years ago Mikel Maron implemented (on my urging in irc.oftc.net #geo IRC :) a PHP-based Google Earth touring service, which interconnects a “tour guide” user with “tourists”.

This site facilitates collaborative, realtime exploration of Google Earth. As the “tour guide” navigates, “tourists” will automatically follow along.

When the tour guide’s Google Earth installation is at rest, a specially installed KML network link sends the server an HTTP request, showing the coordinates of the visible area of the globe. This same service is periodically polled (every second) by “tourists” whose Google Earth will dutifully fly to the appropriate spot.

The system seems to be offline currently, but was quite evocative to use, even if tricky. You never quite knew what the other party could actually see, since the picture can load quite slowly when moving around a lot. And the implementation didn’t do anything about angle of view (although this became possible in later versions of KML). I had experimental tours of Dublin guided by Ina (Skyping at same time), and of various places in Iran by Hamed Saber.

I expect in due course (if not already, I don’t track these things) Google Earth and similar products (Worldwind, or the Microsoft thingy) will offer social map browsing, it’s such a nice feature, though it really needs an audio channel open at the same time. Last week I tried to do the same without such a link, my mum talking me thru finding a small village in France. Much harder! “Take the road north out of Chabanais … past a small farm, past the swimming pool…”.

Here is Mikel’s “how it works” writeup:

The web interface generates KML files, which are loaded into Google Earth and create Network Links. The tour guide has a “View Based Refresh” Network Link, which sends the bounding box of the current view to the specified URL whenever the camera stops. That position is stored. Tourists receive a “Time Based Refresh” Network Link, which requests every 10 seconds and receives the last stored position of the guide.

Right now only location and altitude are transmitted. A future release of Google Earth may enable tilt and rotation. Integrated chat would be nice as well.

The fact that they’ve hidden a full flight simulator within Google Earth might make this worth revisiting. And of course there is infinite fun to be had from playing with photos etc on the globe, although my last attempts in that direction (preparing for 3 months in Buenos Aires by studying geo-tagged photos instead of Spanish) tailed off. Everyone was putting pics on maps, I got a bit bored, even though it’s still a worthwhile area with much still to be done.

Some ideas are not meant to be combined though: who really needs a collaborative realtime photo-navigator implemented with Google Earth flight simulator? :)

Bottles and teargas in Piazza Navona

Bottle thrower and teargas

I’ve just posted 100 or so photos from the anti-Bush protest in Rome last night. Zac and I were on the way to a restaurant when we stumbled into the protest in Piazza Navona, just by the Senate where Bush apparently was in meetings. We wandered around the square taking photos of the protesters, then suddenly the mood changed from carnival to street fighting at the southern end of the square.

Anti-Bush crowds watching protesters fight police

I tried geo-tagging some of the photos (see flickr map) but it is hard to be precise: I don’t know the square well, my memory isn’t perfect, and the tool doesn’t allow me to be clear about position, direction or subject of the image. But it helps. I didn’t see the fighting start, but I ran to the front to see what teargas tasted like (“sort of like lemon sherbet, with an initial zing then a slow, well-rounded plummy finish”; but doesn’t sit well with mild asthma), then hung around as the square was eventually cleared.

Shield & Sky

Main thing I take away from the experience: nothing says “body language” like street-fighting Italians…