Download to read offline
Uploaded byDylan Moriarty
Tracking OpenStreetMap Contributions in Real Time
The document discusses the importance of communities in contributing to OpenStreetMap (OSM) and the challenges faced by mappers, particularly those who are initially curious but become overwhelmed. It introduces the 'planet stream' tool, which utilizes live OSM contributions to produce rich changesets and statistics that help foster community engagement. The document emphasizes the collaborative nature of OSM and acknowledges contributions from organizations like the American Red Cross.
![Actividad vi plantear_alternativas_de_solucion[1]](https://cdn.slidesharecdn.com/ss_thumbnails/actividadviplantearalternativasdesolucion1-130925215306-phpapp02-thumbnail.jpg?width=640&height=640&fit=bounds)
![Actividad v importancia_de_la_metodologia[1]](https://cdn.slidesharecdn.com/ss_thumbnails/actividadvimportanciadelametodologia1-130925214135-phpapp01-thumbnail.jpg?width=640&height=640&fit=bounds)
![Actividad vi plantear_alternativas_de_solucion[1]](https://cdn.slidesharecdn.com/ss_thumbnails/actividadviplantearalternativasdesolucion1-130925214956-phpapp02-thumbnail.jpg?width=640&height=640&fit=bounds)
Tracking OpenStreetMap Contributions in Real Time
- 1. TRACKING OPENSTREETMAP CHANGES INREAL TIME @DylanMoriarty
- 2.
- 3.
- 4.
- 6. OSM is aphenomenal phenomena Humanitarian Mapping efforts
- 7. But we needthat basemap, and we need mappers
- 11. Communities are theentry point for Openstreetmap.
- 13. Communities get usmapping.
- 14. Communities keep usmapping.
- 15.
- 16. Communities are madeof individuals with Individual goals.
- 17. Most contributors belongto a category of ‘hit and run mappers.’
- 18. ○ Initially curious,immediately overwhelmed
- 19. ○ Initially curious,immediately overwhelmed ( mapping the world is hard )
- 20. ○ Initially curious,immediately overwhelmed ○ Only response mappers
- 21. ○ Initially curious,immediately overwhelmed ○ Only response mappers ○ Communities meet infrequently
- 22. ○ Initially curious,immediately overwhelmed ○ Only response mappers ○ Communities meet infrequently ○ Hard to identify potential super users
- 23. ○ Initially curious,immediately overwhelmed ○ Only response mappers ○ Communities meet infrequently ○ Hard to identify potential super users ○ Weren’t that interested
- 24. OSM Analytics canhelp us understand the barriers and reduce them!
- 27. How do wepull down live OSM contributions and keep their community identity?
- 28.
- 29.
- 33.
- 34. Ingredient Two: OVERPASS API- Augmented Diff
- 35. Cook for aminute and…...
- 36.
- 37. 1. Put ingredientsinto a basket
- 38. 2. Match upitems in the basket by id
- 39. 3. If match,combine them to make RICH CHANGESETS
- 42. 4. Take therich changesets and push them through worker code to derive statistics.
- 43.
- 44. By using workerson the rich changesets we generate: - User & Hashtag Statistics - Trending Hashtags - Live map data
- 47. ○ Initially curious,immediately overwhelmed
- 50. ○ Only responsemappers
- 53. ○ Communities meetinfrequently
- 56. ○ Hard toidentify potential super users
- 60. Planet Stream willwork with any #, ...but does require setup
- 61. Caveat: This system isdependent on Overpass, which can hiccup.
- 62.
- 63.
- 64.
- 65. Communities working together isOpenstreetmap.
- 66. Special Thanks to: AmericanRed Cross Marc Farra Pascal Neis
- 67. Thanks for yourtime & attention!
- 68.
Editor's Notes
- #2 Hello, I’m Dylan from Devseed in DC. We’re a tech shop of a small group of smart folks who tackle big problems for cool people. One problem we recently helped tackle was--- TRACKING OSM CHANGES IN REAL TIME With the American Red Cross, and for the Missingmaps effort.
- #3 We’ve heard it talked a lot about, but I’m curious how many people have actively taken part in the project- So, how many folks here have used OpenStreetMap data? How many folks have made an edit to the database?
- #4 Cool. Well, it’s AMAZING. It’s an amazingly weird creation that is reliable enough that MOST apps actually utilize it, Mapbox, Tanagram, _most web maps_ google references it Crowd sourced, possibly most up-to-date
- #5 My favorite recent example comes from Alan Mustard, US Ambassador to Turkmenistan Talks about existing roads being worthless Talks about new construction mesing up google maps
- #6 But this tool doesn’t show you where the changes came from
- #7 Which is all to say, OSM is amazing for humanitarian efforts beacuse of how rapid it can be to immediate responses Doctors without borders, USAID, the American Red Cross, need a platform they can go to for recent info-- OSM’s platform provides a really easy to go to database they can all reach too
- #8 But they need that basemap
- #9 Whether it be sharing an experience tromping around with field papers...
- #10 ...or spending a few hours helping map an unknown place to help aid humanitarians
- #12 Most people are unaware of what OSM is Explaining it to parents So the general entrance points are either: Chance Working with the data Hearing about a community of mappers or meetup event This is a great entrance: Knowledgeable person to provide guidance
- #14 When a strong community runs around and maps, wisdom gets shared, Lessons get learned Allan Mustard’s talk - rogue mapper
- #15 When a strong community runs around and maps, wisdom gets shared, Lessons get learned Allan Mustard’s talk - rogue mapper
- #16 When a strong community runs around and maps, wisdom gets shared, Lessons get learned Allan Mustard’s talk - rogue mapper
- #17 One way would be to check out the pieces that make them up
- #18 Because retention isn’t great. Most are hit n’ run Which means they make a few edits and never more Why is that? Well in truth we don’t know from the stats alone. But we have some theories. And we can test these theories by tinkering with the way things work and measuring the impact.
- #19 Initially excited to map their hometown Spent twenty minutes mapping buildings Zoomed out, realized they only mapped four blocks It takes a lot of effort, with minimal feedback
- #20 Initially excited to map their hometown Spent twenty minutes mapping buildings Zoomed out, realized they only mapped four blocks It takes a lot of effort, with minimal feedback
- #21 Maybe they’re response mappers. Wait for HOT Activations Inspired by ‘doing good’ Less interested in local work
- #22 Sometimes communities just don’t meet that frequently Maptimes, bi-annually Most groups get together initially, but don’t really provide any in-person follow ups
- #23 During larger events, can be hard to identify those who are adding the most Someone did a great job, hard to tell Maybe someone would excel with JSOM Or excited by anyone facet of OSM
- #24 Or maybe they just tagged along That’s fine too.
- #25 But user analytics can help us reduce many of those hurdles.
- #26 Currently, in OSM itself you can view your own edits,
- #27 And there was a neat tool made by Pascal Nies that gave a good overview of what you’ve done. But these only look at the individual, and don’t really influence or become impacted by the communities they’re a part of
- #28 What people are mapping reveals a lot about what motivates them So we want to see what people are mapping. As many of you know, we can get data about our contributions from
- #29 So we built out Planet Stream.
- #30 What people are mapping reveals a lot about what motivates them So we want to see what people are mapping. As many of you know, we can get data about our contributions from
- #31 Changesets! Once you’ve made your edits, you hit save, and boom, a changeset is made. It’s beautiful! Interface view
- #32 Code version of a changeset Same information Computer legible, user id
- #33 So, we have our changeset. This has a users edits, we should be good to go right? Not quite. The format OSM provides for changesets aren’t complete. There’s nowhere in OSM ecosystem right now that provides full changeset info. To pull that out we need to utilize third party applications,
- #34 The first component is Planet.osm.org Collection of all OSM’s metadata The details in the changesets we saw earlier are provided here
- #35 The second main component is overpass api which we grab augmented diffs Minutely updated information on the geometries of a given area. By plugging in the id, we can get the changesets. Set to purge by a certain hashtag
- #36 The second main component is overpass api which we grab augmented diffs Minutely updated information on the geometries of a given area. By plugging in the id, we can get the changesets.
- #37 We combined these two in a tool to create Planet Stream Other folks have combined these datasets in different ways, but by using these we can create a stream of data that runs in real time Now while this tool was made by Development Seed, it’s on the shoulders of giants. The two main collossuses here are
- #38 To start, we start grabbing the augmented diffs and put it in a basket with planetstream
- #39 Next, we run a comparison of all the changeset id’s
- #40 If they match, we smash them together to create rich changesets, or changesets with both the metadata and geographic data
- #42 When we first prototyped out Planet Stream, Leaderboard that tracks most popular hashtags for a six hour period Live, but on a repeating clock Can see that tracks both the metadata, and details about the changeset Counts which hashtag has the most contributions ...so this is one application of planet stream.
- #43 With those rich changesets, we can now grind them through workers to derive statistics. How many roads are in this changeset How many buildings? When did it occur? ...And that stream of data IS Planet Stream, but (that’s for individuals)
- #44 With those rich changesets, we can now grind them through workers to derive statistics. How many roads are in this changeset How many buildings? When did it occur? ...And that stream of data IS Planet Stream, but (that’s for individuals)
- #45 Through worker scripts (also open), we grab all those contributions made to #MissingMaps and derive User Statistics Hashtag Totals Trending hashtags & Live Map Data So let’s see it in use
- #48 To help reduce the depression from mapping so little Break up tasks HOT does it via squares We could do it by badges Provide checkpoints Immediately rewarded For missingmaps we did this with a badge system
- #51 If someone is exclusively a crisis mapper If someone maps many, ping when a new is afoot Try to message to get more involved Or otherwise encourage to lead their community
- #54 We can encourage more frequency by adding elements of competition between groups Creating group stats Easily see what they’ve done, how much they’ve helped For MM Leaderboard competition, two universities against one another
- #57 We can create a leaderboard for an event Allow for quick referencing Quickly locate osm profile Send follow-ups See in real-time how folks are doing
- #59 Can also compare different hashtags or projects Some less appropriate (sub-groups) In this case, comparing osmafrica to hotosm-project-2104
- #60 And for any tech folks out there, this is what the API looks like before it comes to the pages.
- #61 Overpass goes down occasionally. Great tool 99% of the time, not really built to sustain live feeds This can mess up our stream of data, as it takes awhile to pedal backwards and reprocess Some edits get dropped Improving tools like this, or replicated these functionalities will be key in making the system fool-proof
- #62 Overpass goes down occasionally. Great tool 99% of the time, not really built to sustain live feeds This can mess up our stream of data, as it takes awhile to pedal backwards and reprocess Some edits get dropped Improving tools like this, or replicated these functionalities will be key in making the system fool-proof
- #63 Still, we believe this is a great start to deriving real-time user analytics. All the code is on github Host your own version! Comment, like, subscribe, Please leave issue notes Whitelisted, so it will work for any # in a comment
- #64 If you want to see how we processed the stream, - OSM Stats live on ARC’s github - Fork and hack
- #65 If you want to see how we processed the stream, - OSM Stats live on ARC’s github - Fork and hack
- #66 Just to bring it back, We’re doing this to understand the people that make up OSM Motivate them Reward them for the hard work,
- #67 ARC - funding the project and planet streams development Marc for designing and building much of the back end Pascal Neis for much inspiration