Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Waldo-gcp

672 views

Published on

Travel service at Google Cloud Platform

Published in: Internet
  • Be the first to comment

Waldo-gcp

  1. 1. Travel Service at GCP waldo-gcp Cage, F2E, Mitac
  2. 2. Road trip stopping at popular U.S. cities - Computing the optimal road trip across the U. S. | Randal S. Olson(http://goo.gl/pDNfp0)
  3. 3. My own road trip (https://goo.gl/SzBwbM)
  4. 4. Waldo-gcp overview What is waldo
  5. 5. Demo https://waldo-gcp.appspot.com/
  6. 6. Waypoints address sample text file What is waldo "台北市內湖區瑞光路227號1樓", "高雄市鼓山區美術東二路106號", "台南市長榮路一段175號", "臺北市松山區南京東路五段123巷1弄15號", "高雄市五福四路131號2樓"
  7. 7. Waldo-gcp Google Cloud Platform Features What is waldo-gcp Compute Storage App Engine Cloud Datastore Cloud Storage Compute Engine
  8. 8. Waldo-gcp architecture queue MetadataWaldo Server App Engine Task Queues Cloud Storage Optimal- waypoints managed VMs (python27) ... endpoints API (OAuth 2) Object Notification Cloude Datastore snapshot managed VMs (nodejs) Android App CSV uploadwaypoints Store/retrieve metadata waypoint map snapshotRESTful APIs for waypoints list, upload waypoints optim al road trip
  9. 9. Waldo-gcp dispatch.yaml dispatch: - url: "*/favicon.ico" module: default - url: "waldo-gcp.appspot.com/" module: default - url: "*/notice/*" module: notice - url: "*/waypoints/*" module: optimal-waypoints - url: "*/screenshot/*" module: screenshot - url: "*/google40fc41ccc276e5c5.html" module: ownership - url: "*/*" module: "default" Waldo-gcp architecture Application hierarchy [App Engine Modules in Python](https://goo.gl/p3nl48)
  10. 10. Compute
  11. 11. Cloud Computing Compute PaasIaaS Application Data Runtime Middleware OS Virtualization Servers Storage Networking Application Data Runtime Middleware OS Virtualization Servers Storage Networking Saas Application Data Runtime Middleware OS Virtualization Servers Storage Networking Infrastructure as a Service Platform as a Service Service as a Service You manage Google manage
  12. 12. Managed VMs Compute | Managed VMs ● Managed VMs ○ The Flexibility of Compute Engine ○ The productivity of App Engine ● Provides best of both ○ IaaS + PaaS
  13. 13. Waldo-gcp storage - cloud datastore queue MetadataWaldo Server App Engine Task Queues Cloud Storage Optimal- waypoints managed VMs (python27) ... endpoints API (OAuth 2) Object Notification Cloude Datastore snapshot managed VMs (nodejs) Android App CSV uploadwaypoints Store/retrieve metadata waypoint map snapshot optim al road trip RESTful APIs for waypoints list, upload waypoints
  14. 14. Waldo-gcp storage - cloud datastore queue MetadataWaldo Server App Engine Task Queues Cloud Storage Optimal- waypoints managed VMs (python27) ... endpoints API (OAuth 2) Object Notification Cloude Datastore snapshot managed VMs (nodejs) Android App CSV uploadwaypoints Store/retrieve metadata waypoint map snapshot optim al road trip RESTful APIs for waypoints list, upload waypoints
  15. 15. Storage
  16. 16. Waldo-gcp Storage Storage Cloud Datastore Cloud Storage Cloud SQL Compute Engine (Persistent Disk) BlobNoSQL SQL Block
  17. 17. Waldo-gcp storage - cloud datastore queue MetadataWaldo Server App Engine Task Queues Cloud Storage Optimal- waypoints managed VMs (python27) ... endpoints API (OAuth 2) Object Notification Cloude Datastore snapshot managed VMs (nodejs) Android App CSV uploadwaypoints Store/retrieve metadata waypoint map snapshot optim al road trip RESTful APIs for waypoints list, upload waypoints
  18. 18. Waldo-gcp storage - cloud storage queue MetadataWaldo Server App Engine Task Queues Cloud Storage Optimal- waypoints managed VMs (python27) ... endpoints API (OAuth 2) Object Notification Cloude Datastore snapshot managed VMs (nodejs) Android App CSV uploadwaypoints Store/retrieve metadata waypoint map snapshot optim al road trip RESTful APIs for waypoints list, upload waypoints
  19. 19. Waldo-gcp storage - cloud storage queue MetadataWaldo Server App Engine Task Queues Cloud Storage Optimal- waypoints managed VMs (python27) ... endpoints API (OAuth 2) Object Notification Cloude Datastore snapshot managed VMs (nodejs) Android App CSV uploadwaypoints Store/retrieve metadata waypoint map snapshot optim al road trip RESTful APIs for waypoints list, upload waypoints
  20. 20. Setup Cloud Storage Buckets
  21. 21. Waldo-gcp cloud storage queue MetadataWaldo Server App Engine Task Queues Cloud Storage Optimal- waypoints managed VMs (python27) ... endpoints API (OAuth 2) Object Notification Cloude Datastore snapshot managed VMs (nodejs) Android App CSV uploadwaypoints Store/retrieve metadata waypoint map snapshot optim al road trip RESTful APIs for waypoints list, upload waypoints
  22. 22. Waypoints GCS Bucket Waldo-gcp cloud storage Waldo Server Android App endpoints API (OAuth 2) Waldo-gcp-file Waldo-gcp-file-in Waldo-gcp-file-image endpoint API upload waypoints address to managed VMs upload optimal address to managed VMs upload screenshot image to
  23. 23. GCS Bucket Notification Waldo-gcp setup gcs object notification Waldo Server Android App endpoints API (OAuth 2) Waldo-gcp-file Waldo-gcp-file-in Waldo-gcp-file-image
  24. 24. GCS Bucket Notification Waldo-gcp storage Waldo Server Android App endpoints API (OAuth 2) Waldo-gcp-file Waldo-gcp-file-in Waldo-gcp-file-image ● bucket push notification ○ https only ○ domain owned/notice
  25. 25. Google Cloud Storage Notification class BucketNotificationHandler(webapp2.RequestHandler): def post(self): # Decode the name and bucket of the notification def notification_decoder(object): if 'kind' in object: return dict(name=object['name'], bucket=object['bucket'], id=object['id']) ... # Add a new task to the queue taskqueue.add(url='/waypoints/', params={ 'bucket': notification['bucket'], 'name': notification['name'], 'id': notification['id'] },queue_name='new-files') ... Create the App Engine backend
  26. 26. Waypoints address text file Create the App Engine backend | Waldo-gcp-file "台北市內湖區瑞光路227號1樓", "高雄市鼓山區美術東二路106號", "台南市長榮路一段175號", "臺北市松山區南京東路五段123巷1弄15號", "高雄市五福四路131號2樓"
  27. 27. Waypoints address distance_m & distance_s Create the App Engine backend | Waldo-gcp-file-in waypoint1 waypoint2 distance_m duration_s 台南市長榮路一段175號 高雄市五福四路131號2樓 89300 4375 台南市長榮路一段175號 高雄市鼓山區美術東二路 106號 84212 4243 台北市內湖區瑞光路227號1樓 臺北市松山區南京東路五段 123巷1弄15號 5204 763 高雄市五福四路131號2樓 高雄市鼓山區美術東二路 106號 4163 732 台北市內湖區瑞光路227號1樓 高雄市五福四路131號2樓 356184 13972 臺北市松山區南京東路五段 123巷1弄15號 高雄市鼓山區美術東二路 106號 354392 14071 臺北市松山區南京東路五段 123巷1弄15號 高雄市五福四路131號2樓 359450 14230 台北市內湖區瑞光路227號1樓 台南市長榮路一段175號 275002 10503 台北市內湖區瑞光路227號1樓 高雄市鼓山區美術東二路 106號 352764 13541 臺北市松山區南京東路五段 123巷1弄15號 台南市長榮路一段175號 276930 10674
  28. 28. Create Managed VMs Server ● Optimal road trip ● screenshot
  29. 29. Waldo-gcp waypoint optimization ● Gather the distance between wall waypoints ● Google Map APIs ● genetic algorithm optimal road trip [nbviewer.ipython.org/github/rhiever/Data-Analysis-and-Machine-Learning- Projects/blob/master/optimal-road-trip/Computing the optimal road trip across the U.S..ipynb](http://goo.gl/SYoh53)
  30. 30. Pandas ● Python library ● Powerful data structures for data analysis, time series,and statistics optimal road trip http://pandas.pydata.org/
  31. 31. Using Managed VMs Waldo-gcp architecture Waldo Server Android App endpoints API (OAuth 2) /notice - url: "*/notice/*" module: notice - url: "*/waypoints/*" module: optimal-waypoints - url: "*/screenshot/*" module: screenshot - url: "*/*" module: default bucket notification entry point optimal road trip screenshot waldo-gcp
  32. 32. Managed VMs Waldo-gcp architecture Waldo Server Android App endpoints API (OAuth 2) /notice - url: "*/notice/*" module: notice - url: "*/waypoints/*" module: optimal-waypoints - url: "*/screenshot/*" module: screenshot - url: "*/*" module: default Auto-scaled managed VMs (python27) ... managed VMs (nodejs)
  33. 33. Containers on Managed VMs FROM google/appengine-python27 # Install pandas RUN apt-get update && apt-get install -y -q --no-install- recommends python-pandas python-numpy # Install Google Map APIs, Google Client Library ADD requirements.txt /app/ RUN pip install --upgrade pip RUN pip install -r requirements.txt ADD . /app Create the App Engine | Dockerfile [Managed VMs - App Engine — Google Cloud Platform](https://goo.gl/4FFSmm)
  34. 34. Optimal waypoint Module Configuration application: waldo-gcp module: optimal-waypoints version: 1 runtime: python27 api_version: 1 threadsafe: yes vm: true automatic_scaling: min_num_instances: 1 max_num_instances: 5 cool_down_period_sec: 60 cpu_utilization: target_utilization: 0.5 Create the App Engine | app.yaml
  35. 35. Waldo-gcp screenshot ● Screenshot as a service ● Phantomjs + nodejs snapshot server [fzaninotto/screenshot-as-a-service](https://github.com/fzaninotto/screenshot-as- a-service)
  36. 36. Screenshot Module Configuration application: waldo-gcp module: screenshot version: 1 runtime: custome api_version: 1 threadsafe: yes vm: true resources: cpu: .5 memory_gb: 1.3 manual_scaling: instances: 1 snapshot server
  37. 37. Waldo-gcp
  38. 38. Road trip on Google Map waldo-gcp google map optimal_route = [/*optimal waypoint calculate by genetic algorithm*/] optimal_route = list(optimal_route) optimal_route += [optimal_route[0]] subset = 0 while subset < len(optimal_route): waypoint_subset = optimal_route[subset:subset + 10] output = "calcRoute("%s", "%s", [" % (waypoint_subset[0], waypoint_subset[-1]) for waypoint in waypoint_subset[1:-1]: output += ""%s", " % (waypoint) if len(waypoint_subset[1:-1]) > 0: output = output[:-2] output += "]);" print(output) print("") subset += 9 ... ● Google Map APIs ○ split max waypoints (11) in one route
  39. 39. Road trip on Google Map waldo-gcp google map calcRoute("605嘉義縣阿里山 鄉二萬坪106號", "605嘉義縣阿里山 鄉二萬坪106號", ["500彰化縣彰化市溫泉 路31號", "224九份風景區瑞芳區新北市 ", "971花蓮縣新城 鄉海岸路", "555南投縣魚池 鄉中山路599號"]); function calcRoute(start, end, routes) { switch (start) { case "605嘉義縣阿里山 鄉二萬坪106號" ,"605嘉義縣阿里山 鄉二萬坪106號": directionsDisplay1 = new google.maps.DirectionsRenderer(directionsDisplayOptions); .. } var request = {origin: start, destination: end, waypoints: waypts, optimizeWaypoints: false, travelMode: google.maps.TravelMode.DRIVING }; directionsService.route(request, function (response, status) { case "605嘉義縣阿里山 鄉二萬坪106號" ,"605嘉義縣阿里山 鄉二萬坪106號": directionsDisplay1.setDirections(response); } }); .. } } 1. 605嘉義縣阿里山鄉二萬坪106號 2. 500彰化縣彰化市溫泉路31號 3. 224九份風景區瑞芳區新北市 4. 971花蓮縣新城鄉海岸路 5. 555南投縣魚池鄉中山路599號
  40. 40. Thanks you!
  41. 41. ● [Here's Waldo: Computing the optimal search strategy for finding Waldo | Randal S. Olson](http://goo. gl/DSOZbv) ● [Computing the optimal road trip across the U.S. | Randal S. Olson](http://goo.gl/pDNfp0) ● [nbviewer.ipython.org/github/rhiever/Data-Analysis-and-Machine-Learning-Projects/blob/master/optimal- road-trip/Computing the optimal road trip across the U.S..ipynb](http://goo.gl/SYoh53) ● [GoogleCloudPlatform/abelana-gcp](https://goo.gl/6L55oa) Reference
  42. 42. 《威利在哪裡?》(英文:Where's Wally?),也譯《威力在哪裡?》、《大家一起找沃裡?》、《聰明的沃 裡?》,是一套由英國插畫家Martin Handford創作的兒童書籍。這個書的目標就是在一張人山人 海的圖片中找出一個特定的人物——威利。威利穿著紅白條紋的襯衫並戴著一個絨球帽,手上拿 著木製的手杖,還戴著一副眼鏡。他總是會弄丟東西,如書本、野營設備甚至是他的鞋子,而讀者 也要幫他在圖中找出這些東西來。 wikipedia http://goo.gl/D4mH6 Where's Waldo?

×