Ed Freyfogle's presentation discusses the importance of geocoding and its applications, such as vehicle tracking and data enrichment, through OpenCage's API. He highlights the challenges of building versus buying geocoding services, emphasizing the trend towards purchasing solutions due to maintenance complexities and cost considerations. The document also provides technical insights into the architecture of OpenCage's geocoding services and encourages community contributions via wrapper libraries.

1. Making the entire world accessible via a single API
Ed Freyfogle
Barcelona Data Engineering - 30 Nov. 2017
http://opencagedata.com
@opencagedata

2. - Some background
- Technical overview
- Build versus Buy?
- Questions
Overview

3. $whoami

4. $whoami
- Co-founder of OpenCage
- Before that, co-founded a real estate metasearch engine called
Nestoria. 10 years, 20 employees, €4.5M in revenue
- Way, way before that, first developer at yahoo.de
- Moved to BCN to live the good life

5. opencagedata.com

6. WTF is geocoding?

7. Where are we?

9. Barcelona

10. Barcelona
Ironhack BCN

11. Barcelona
IronHack BCN
08018

12. Barcelona
IronHack BCN
08002
Carrer de Pamplona
Poblenou

13. Barcelona
IronHack BCN
08002
Carrer de Pamplona
Poblenou
Europe

14. Barcelona
IronHack BCN
08002
Carrer de Pamplona
Poblenou
Europe
Spain / Rep. of Catalonia
NEW!

16. 41.39778,2.19011

18. placename lat/long

19. placename long/lat
forward

20. placename long/lat
forward
reverse

21. Why Geocode?
- Vehicle tracking
- Routing
- Data enrichment
- Put points on a map
- Image processing

23. Yes!

24. Yes! but …
T&Cs
expensive at volume

25. OpenStreetMap

26. Yes!

27. Yes! but …
not for high volume use
variable coverage

29. Yes!

30. Yes! and …
one API, many geocoders
free as in freedom
enterprise level reliability

31. - Some background
- Technical overview
- Build versus Buy?
- Questions
Overview

33. OpenStreetMap
www.openstreetmap.org

34. OpenStreetMap
- Crowdsourced geodata. Anyone can edit. 4 million reg. users
- Bad name. Is a database, not a map.
- 3 million edits per day. And growing.
- Operations run by volunteers.
- Master DB is hosted at a university in London
- Data is under ODbL License
- Many, many different tools for editing, querying, visualizing the data

35. Example
$ curl ‘https://api.opencagedata.com/geocode/v1/json?q=41.39778,2.19011&key=...&pretty=1’

36. Example
$ curl ‘https://api.opencagedata.com/geocode/v1/json?q=41.39778,2.19011&key=...&pretty=1’
"components": {
"ISO_3166-1_alpha-2": "ES",
"_type": "pub",
"city": "Barcelona",
"city_district": "Sant Martí",
"country": "Spain",
"country_code": "es",
"county": "BCN",
"house_number": "96",
"postcode": "08018",
"pub": "Sonora Sport Tavern",
"road": "Carrer de Pamplona",
"state": "Catalonia",
"suburb": "el Parc i la Llacuna del Poblenou"
},
"formatted": "Sonora Sport Tavern, Carrer de Pamplona, 96, 08018 Barcelona, Spain",

37. https://geocoder.opencagedata.com/code
Libraries for
- Perl, PHP, Python, Ruby, R, Java, .NET, elixir,
Haskell, Go, Erlang, nodejs, Stata, perl6, iOS
- Leaflet, OpenLayers
TMTOWTDI

38. Architecture
HTTP request

39. Architecture
HTTP request LB
LB
- HAProxy
- Two different physical locations

40. Architecture
HTTP request LB
LB
HTTP request LB
meta
- ubuntu
- apache
- mod_perl

41. Architecture
HTTP request LB
LB
HTTP request LB
meta
Kong
HTTP request
- authentication
- validity checking
- routing

42. Architecture
HTTP request LB
LB
HTTP request LB
meta
Kong
HTTP request
geocoder geocoder geocoder

43. Architecture
HTTP request LB
LB
HTTP request LB
meta
Kong
HTTP request
geocoder geocoder geocoder
OSM
Continually
updating
HTTP request

44. Architecture
HTTP request LB
LB
HTTP request LB
meta
Kong
HTTP request
geocoder geocoder geocoder
OSM
Continually
updating
- accept response, error check
- standardize format
- rank
- add annotations
- send to client in XML, JSON, etc

45. - timezones
- geohash, MGRS, Maidenhead, DMS
- what3words
- calling codes, currency
- and more ...
Annotations

46. Formatting
$ curl ‘https://api.opencagedata.com/geocode/v1/json?q=41.39778,2.19011&key=...&pretty=1’
"components": {
"ISO_3166-1_alpha-2": "ES",
"_type": "pub",
"city": "Barcelona",
"city_district": "Sant Martí",
"country": "Spain",
"country_code": "es",
"county": "BCN",
"house_number": "96",
"postcode": "08018",
"pub": "Sonora Sport Tavern",
"road": "Carrer de Pamplona",
"state": "Catalonia",
"suburb": "el Parc i la Llacuna del Poblenou"
},
"formatted": "Sonora Sport Tavern, Carrer de Pamplona, 96, 08018 Barcelona, Spain",

47. 115 New Cavendish Street, London W1T 5DU, United Kingdom
223 William Street, Melbourne VIC 3000, Australia
Rosenthaler Straße 1, 10119 Berlin, Germany
Via Pisacane, 13, 76121 Barletta BT, Italy
3 Upper Alma Road, Rosebank, Cape Town, 7700, South Africa
Formatting

48. Architecture
HTTP request LB
LB
HTTP request LB
meta
Kong
HTTP request
geocoder geocoder geocoder
OSM
Continually
updating
- logging
- monitoring
- emails
metrics
scp

49. Monitoring
We use StatusCake

50. - Dev/testing process
- Deployment
- User registration / onboarding
- Billing
Not covered

51. - Some background
- Technical overview
- Build versus Buy?
- Questions
Overview

52. - Unable to buy
- Lack of finance
- Lack of affordable, reliable vendor
When to build

53. - Unable to buy
- Core to your service
When to build

54. - Unable to buy
- Core to your service
- Critical differentiator
- Privacy / data protection issues
- Legal requirement
When to build

55. - Unable to build
When to buy

56. - Unable to build
- Need to move fast
When to buy

57. - Unable to build
- Need to move fast
- Is non-core
When to buy

58. - Unable to build
- Need to move fast
- Is non-core
- Unable to maintain
When to buy

59. - Financial cost
- Opportunity costs
- Staffing costs
Cost analysis

60. - Industry trend has been towards buying
- Servers (AWS, Dropbox, etc)
- Login / Identity (FB, google, etc)
- Billing (Stripe)
- 1000s of SaaS businesses
The trend

61. - Engineers love to build
MAIN POINT

62. - Engineers love to build
- Building is (comparatively) easy
MAIN POINT

63. - Engineers love to build
- Building is (comparatively) easy
- MAINTENANCE IS REALLY
HARD!!!!
MAIN POINT

64. - Some background
- Technical overview
- Build versus Buy?
- Questions
Overview

65. Three more things

66. Three more things
1. Lifetime 10% discount
use code: XXXXXX

67. Three more things
1. Lifetime 10% discount
use code: XXXXXX
2. Happy to pay anyone who wants to write
a wrapper library: F#, Scala, Lua, etc

68. Three more things
My friends at AVUXI are hiring
http://www.avuxi.com

69. Ed Freyfogle
Barcelona Data Engineering - 30 Nov. 2017
https://opencagedata.com
@opencagedata
Thanks!
Questions?