Your SlideShare is downloading. ×
0
RubyNation Visualizing Big Data on Small Devices
RubyNation Visualizing Big Data on Small Devices
RubyNation Visualizing Big Data on Small Devices
RubyNation Visualizing Big Data on Small Devices
RubyNation Visualizing Big Data on Small Devices
RubyNation Visualizing Big Data on Small Devices
RubyNation Visualizing Big Data on Small Devices
RubyNation Visualizing Big Data on Small Devices
RubyNation Visualizing Big Data on Small Devices
RubyNation Visualizing Big Data on Small Devices
RubyNation Visualizing Big Data on Small Devices
RubyNation Visualizing Big Data on Small Devices
RubyNation Visualizing Big Data on Small Devices
RubyNation Visualizing Big Data on Small Devices
RubyNation Visualizing Big Data on Small Devices
RubyNation Visualizing Big Data on Small Devices
RubyNation Visualizing Big Data on Small Devices
RubyNation Visualizing Big Data on Small Devices
RubyNation Visualizing Big Data on Small Devices
RubyNation Visualizing Big Data on Small Devices
RubyNation Visualizing Big Data on Small Devices
RubyNation Visualizing Big Data on Small Devices
RubyNation Visualizing Big Data on Small Devices
RubyNation Visualizing Big Data on Small Devices
RubyNation Visualizing Big Data on Small Devices
RubyNation Visualizing Big Data on Small Devices
RubyNation Visualizing Big Data on Small Devices
RubyNation Visualizing Big Data on Small Devices
RubyNation Visualizing Big Data on Small Devices
RubyNation Visualizing Big Data on Small Devices
RubyNation Visualizing Big Data on Small Devices
RubyNation Visualizing Big Data on Small Devices
RubyNation Visualizing Big Data on Small Devices
RubyNation Visualizing Big Data on Small Devices
RubyNation Visualizing Big Data on Small Devices
RubyNation Visualizing Big Data on Small Devices
RubyNation Visualizing Big Data on Small Devices
RubyNation Visualizing Big Data on Small Devices
RubyNation Visualizing Big Data on Small Devices
RubyNation Visualizing Big Data on Small Devices
RubyNation Visualizing Big Data on Small Devices
RubyNation Visualizing Big Data on Small Devices
RubyNation Visualizing Big Data on Small Devices
RubyNation Visualizing Big Data on Small Devices
RubyNation Visualizing Big Data on Small Devices
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

RubyNation Visualizing Big Data on Small Devices

458

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
458
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Copyright © 2014 Intridea Inc. All rights reserved. Visualizing Big Data on Small Devices Tom Zeng Director of Engineering tom@intridea.com @tomzeng www.linkedin.com/in/tomzeng
  • 2. Copyright © 2014 Intridea Inc. All rights reserved. Agenda Introduction Front End - HTML5/Bootstrap, Backbone/CoffeeScript, D3, MapBox Backend - Rails, MongoDB Big Data Processing - Hadoop, Hive, Pig Showcase - Mobile and Data Visualization Related Projects Q & A
  • 3. Copyright © 2014 Intridea Inc. All rights reserved. Intridea - Rails, UX/Data Visualization, Mobile, Big Data, e-commerce American Bible Society (ABS http://www.americanbible.org/) - partners with Bible publishers · Provides API access to 539 Bible versions in 242 languages · The usage of the APIs is tracked at the verse level, along with ip location, timestamp, and duration · 530 million view logs/year(’12-’13 data), 1.5 mil/day, each view log has packed about 12 bible views · Amounts to 5-6 billion Bible views each year ABS asked Intridea to build the dashboard app Scripture Analytics (http://www.scriptureanalytics.com) Introduction
  • 4. Copyright © 2014 Intridea Inc. All rights reserved. BY WORKING REMOTELY 9,816 Hours Saved Annually ACROSS THE US & OVERSEAS 30+ Employees FOUNDED & STARTED IN 2007 Washington D.C. We Make ! 🌎 # ON GITHUB Open Source Software $
  • 5. Copyright © 2014 Intridea Inc. All rights reserved. Major Open Source Contributions OmniAuth is a flexible authentication system utilizing Rack middleware. OmniAuth An opinionated micro-framework for creating REST-like APIs in Ruby. Grape Hashie is a simple collection of useful Hash extensions. Hashie A Ruby wrapper for the OAuth 2.0 protocol. oauth2 A symbol font that makes it easy to create a map of the U.S. with HTML/CSS. Stately A generic swappable back-end for JSON handling. Multi_JSON Mission control dashboard for your distributed teams. Houston github.com/intridea
  • 6. Copyright © 2014 Intridea Inc. All rights reserved. INTRIDEA Simplified and modernized application experience for ADP Application engineering for BusinessWeek.com World’s first 100% web based Point of Sale system
  • 7. Copyright © 2014 Intridea Inc. All rights reserved. Where people go to make a difference with their investment capital. Simplifying ADP’s core business: Payroll Engineering for the most-trafficked wedding planning solution
  • 8. Copyright © 2014 Intridea Inc. All rights reserved.
  • 9. Copyright © 2014 Intridea Inc. All rights reserved. ABS Scripture Analytics Query Requirements Visualizations
  • 10. Copyright © 2014 Intridea Inc. All rights reserved. Public and private dashboards visualizing Bible reading across the Internet Up to the minute dashboards showing what Bible verses are being read when, and where, all over the globe.
  • 11. Copyright © 2014 Intridea Inc. All rights reserved. Mobile Desktop Tablet Responsive Web App www.scriptureanalytics.com
  • 12. Copyright © 2014 Intridea Inc. All rights reserved. Front End User Interface Single Page Application using Backbone.js CoffeeScript (Ruby like, Jasmine in CoffeeScript similar to RSpec)
 D3 for Data Visualization Twitter Bootstrap for Responsive UI Packery for Responsive Layout - http://packery.metafizzy.co/ Mapbox for Map Rendering - https://www.mapbox.com/
  • 13. Copyright © 2014 Intridea Inc. All rights reserved. Backend Servers/Services Rails on Ruby Application mostly as the API server MongoDB as the data store/cache Mongoid for Active Record like queries MongoDB Aggregation Framework for complex queries Pulling data periodically from S3 to populate the Mongo database Local R&D Hadoop and Mongo clusters for data exploration
  • 14. Copyright © 2014 Intridea Inc. All rights reserved. MongoDB Document oriented, schema free, JSON format Very high data read and write throughput Rich query capabilities (aggregation framework), flexible indexes Scale with auto-sharded replica sets Map/Reduce in JavaScript
  • 15. Copyright © 2014 Intridea Inc. All rights reserved. Hadoop/Pig/Hive/Impala Hadoop cluster (AWS Elastic Map/Reduce on-demand) to process and store data in S3 Pig to parse, transform, geo-code data Hive to query data and generate aggregated JSON reports Impala is similar to Hive (but much fast than the older version of Hive), used for ETL !
  • 16. Copyright © 2014 Intridea Inc. All rights reserved. Elastic Map/Reduce Hadoop Cluster - On Demand Processing
  • 17. Copyright © 2014 Intridea Inc. All rights reserved. Elastic Map/Reduce Hadoop Cluster - Terminated when done
  • 18. Copyright © 2014 Intridea Inc. All rights reserved. Cloudera CDH4 - on local 10-node cluster
  • 19. Copyright © 2014 Intridea Inc. All rights reserved. Cloudera CDH4 - Streaming Data into Hive Table
  • 20. Copyright © 2014 Intridea Inc. All rights reserved. Pig Sample Query
  • 21. Copyright © 2014 Intridea Inc. All rights reserved. Hive Sample Query
  • 22. Copyright © 2014 Intridea Inc. All rights reserved. Hive Query Results - Bible views by City
  • 23. Copyright © 2014 Intridea Inc. All rights reserved. Hive Query Results - Most popular verse before Mother’s day 28 Her children show their appreciation, and her husband praises her.
  • 24. Copyright © 2014 Intridea Inc. All rights reserved. Hive Query Results - Most popular verse on Mother’s day 28 Her children show their appreciation, and her husband praises her.
  • 25. Copyright © 2014 Intridea Inc. All rights reserved. Hive Query Results - Most popular verse after Mother’s day 28 Her children show their appreciation, and her husband praises her.
  • 26. Copyright © 2014 Intridea Inc. All rights reserved. MongoDB Aggregation Framework Example http://docs.mongodb.org/manual/core/aggregation-pipeline/
  • 27. Copyright © 2014 Intridea Inc. All rights reserved. MongoDB Aggregation Framework Example
  • 28. Copyright © 2014 Intridea Inc. All rights reserved. ABS Data Processing using Hadoop and MongoDB
  • 29. Copyright © 2014 Intridea Inc. All rights reserved. Analyzing Twitter using Hadoop and MongoDB
  • 30. Copyright © 2014 Intridea Inc. All rights reserved. Mobile and Data Visualization Project Showcase ADP BLiNQ PEW Templeton - Global Religious Futures Cato Institute - HumanProgress ! !
  • 31. Copyright © 2014 Intridea Inc. All rights reserved. Redefining ADP’s touch and desktop experiences ADP processes one out of every six paychecks in the United States. We’re bringing payroll into the decade of touch.
  • 32. Copyright © 2014 Intridea Inc. All rights reserved. ADP TLM
  • 33. Copyright © 2014 Intridea Inc. All rights reserved. ADP HCR
  • 34. Copyright © 2014 Intridea Inc. All rights reserved. INTRIDEA
  • 35. Copyright © 2014 Intridea Inc. All rights reserved. Dashboard insights for 600 of the world’s largest advertisers Analytics, planning, and flight management for social advertising campaigns and brand engagement.
  • 36. Copyright © 2014 Intridea Inc. All rights reserved. INTRIDEA INTRIDEA
  • 37. Copyright © 2014 Intridea Inc. All rights reserved. INTRIDEA INTRIDEA
  • 38. Copyright © 2014 Intridea Inc. All rights reserved. INTRIDEA INTRIDEA
  • 39. Copyright © 2014 Intridea Inc. All rights reserved. Analyzing religious change and its impact on societies around the world Interactive website exploring the patterns and trends in religions across the globe
  • 40. Copyright © 2014 Intridea Inc. All rights reserved. PEW Global Research PEW Global Research
  • 41. Copyright © 2014 Intridea Inc. All rights reserved. PEW Global Research PEW Global Research
  • 42. Copyright © 2014 Intridea Inc. All rights reserved. PEW Global Research PEW Global Research
  • 43. Copyright © 2014 Intridea Inc. All rights reserved. Human advancement to a higher stage Human Progress seeks to document changes in living standards in the past and present while explaining and exploring the best ways to improve conditions for people.
  • 44. Copyright © 2014 Intridea Inc. All rights reserved. INTRIDEA humanprogress.org
  • 45. Gracias Merci ありがとう Danke 谢谢 Thank You Copyright © 2014 Intridea Inc. All rights reserved. Tom Zeng Director of Engineering tom@intridea.com @tomzeng www.linkedin.com/in/tomzeng

×