Download to read offline
More Related Content
Similar to Building OpenStreetMap.org, SOTM US 2015
Building OpenStreetMap.org, SOTM US 2015
- 1.
- 2.
- 3. Database Schema ● Users –users / user_tokens / user_preferences / user_roles / user_blocks ● Geodata – changesets / changeset_tags / changeset_comments / ... – current_nodes / current_node_tags / nodes / node_tags – current_ways / current_way_tags / current_way_nodes / … – current_relations / current_relation_tags / current_relation_members / … ● GPS Data – gpx_files / gpx_file_tags / gps_points ● Notes – notes / note_comments
- 4.
- 5. The “Rails Port” ●Getting Started – INSTALL.md – CONFIGURE.md – CONTRIBUTING.md ● Alternatively – VAGRANT.md ● We have issues...
- 6. The “Rails Port” ●Models – app/models ● Views – app/views ● Controllers – app/controllers ● Javascript – app/assets/javascripts ● Stylesheets – app/assets/stylesheets ● Strings – config/locales/en.yml
- 7. The “Rails Port” ●Models – Node class is current_nodes – OldNode class is nodes – Trace class is gpx_files – Tracetag class is gpx_file_tags – Tracepoint class is gps_points
- 8. The “Rails Port” ●Style – “rubocop” for ruby – “rake jshint” for javascript ● Tests – “rake test” for ruby – javascript coming soon? ● Travis will run on pull requests ● Coveralls will check coverage
- 9. The “Rails Port” ●Merging – Pull requests notified to rails-dev list – Review of behaviour and UI changes – Review of code changes – Larger changes may get a test instance
- 10.
- 11. CGImap ● Currently handlesa number of API calls – The main “map” call – Single node, way and relation fetches – Multiple node, way and relation fetches – The “full” way and relation calls ● Want to extend it to cover more API calls – History calls – Changeset calls
- 12. CGImap ● Currently runsas a separate daemon – Accessed via FastCGI ● Core code has now been split into a library – Will allow creation of a ruby gem around the library – Which then allows direct integration with rails code ● JSON output support – Can be compiled with JSON support – Not currently enabled in production
- 13. CGImap ● Style – Formattedin “LLVM style” with clang-format – Configure “--with-clang-format” – Run “make clang-format” to reformat ● Tests – Run tests with “make check” – Needs postgresql running – Travis coming soon?
- 14. Get Involved GitHub https://github.com/openstreetmap/openstreetmap-website https://github.com/zerebubuth/openstreetmap-cgimap Mailing Lists https://lists.openstreetmap.org/listinfo/rails-dev https://lists.openstreetmap.org/listinfo/dev IRC #osm-dev EngineeringWorking Group https://wiki.osmfoundation.org/wiki/Engineering_Working_Group engineering@osmfoundation.org
Editor's Notes
- #2Â Session all about the code behind osm.org and how we develop it and how you can help I'm Tom Hughes and I've been helping run the servers since 2007 I work on everything but in particular on the main web site
- #3Â Quick overview of architecture ** Three frontends using round robin DNS Run apache and passenger Handle most requests Heavy requests proxied to shared backend pool ** Three backends Run apache, passenger and cgimap Some API requests sent to cgimap Everything else handled by passenger (currently)
- #4 Overview of database Users table, with various auxiliary tables ** Geodata, everything grouped by changesets ** Nodes – current tables with latest version, history tables with all versions – yes duplicated... ** Repeated for ways and relations but with tables to link to child objects ** Imported GPS data – yes the names are not consistent ** Notes with comments Various others with diaries, friends, messages,etc
- #5 Main web site is Ruby on Rails – known as the rails port for historical reasons Find it on github – issues welcome – pull requests even more welcome Includes an implementation of the API Alternate implementation of part of the API is what is used in production though – more later
- #6 Get started – checkout the code! Do a (small) bit of reading ** INSTALL.md tells you how to do the basic setup ** CONFIGURE.md covers a few things you may want to do once you have the basics working ** CONTRIBUTING.md explains how we work ** Or you can use vagrant to do much of the basic setup – read VAGRANT.md for more information ** If you're looking for things to do then we have open issues!
- #7 Structure is a basic MVC architecture Models represent the data with a class for each type of data ** Views are what the user sees – we use ERB templates which are a mix of HTML and Ruby ** Controllers handle user requests – fetching the needed data and then rendering a view ** We also have client side javascript code using jQuery ** And stylesheets to control client side rendering of the views ** All text needs to be translatable – add strings to english and Translatewiki does the rest
- #8Â Models have one class per database table Mostly the names match Except when they don't... ** Big difference is that current tables lose their prefix ** and history tables gain an old prefix Applies to ways and relations as well ** GPS tables also different This is largely a question of history and the rails code being written around an existing schema
- #9 Please run style checkers – we have rubocop setup for ruby and jshint for rails ** Even more important – please run the tests We have good tests for the ruby code But not (yet) for javascript ** Style checks and tests will be run by travis when you open a pull ** Coverage will also be checked by coveralls – only the ruby code at the moment but coverage is good and we'd like to keep it that way So add tests for any new ruby code!
- #10 When your change is ready, open a PR The rails-dev list will be notified, as it is for all issues ** First thing we will usually do is look at the change as a user – what are the UI changes etc ** When we're happy with that I will usually look at the actual code for any implementation issues Not hard and fast though, things may happen in a different order, and small fixes are often just merged straight off ** Larger changes may get a test instance, with people invited to try it out
- #11Â Second part of our story is CGImap Also on github C++ implementation of (part of) the API Increased performance by carefully tailored SQL queries Better memory management for potentially large data sets
- #12 Currently handles The main map call – what editors use to get data Fetching individual map objects Fetching multiple map objects Fetching map objects with their child objects ** Want to extend it Calls which include object history Changeset calls And so on
- #13 Currently runs as a separate daemon called via FastCGI ** Core code has been split into a library Will allow a gem wrapper to be written for integration with rails code Avoids current duplication of API code between rails and cgimap ** Supports JSON but not currently enabled as only calls handled by cgimap would be supported Moving rails code to call cgimap via gem wrapper will allow this (and other changes) to be implemented once Need to reach reasonable API completeness in cgimap first though – help please!
- #14 Formatted in LLVM style with clang-format Need to configure –with-clang-format then you can reformat with make clang-format Needs clang 3.6 ** Run the tests with make check Need to have postgres running Travis config merged but not enabled yet
- #15 None of this happens by magic, and many of the people working on this are also trying to keep the servers running So please, get involved – if you're planning something big, or just want advice, then please contact us and we'll be glad to help I can take some questions now, or find me later – I'm around all weekend and for the hackday on Monday And if you want to know more about how we actually turn all this code into a running service then stay around for my colleague Grant Slater's talk which is up next in this room