Copyright © 2014 Intridea Inc. All rights reserved.
Visualizing Big Data on Small Devices
Tom Zeng
Director of Engineering...
Copyright © 2014 Intridea Inc. All rights reserved.
Agenda
Introduction
Front End - HTML5/Bootstrap, Backbone/CoffeeScript,...
Copyright © 2014 Intridea Inc. All rights reserved.
Intridea - Rails, UX/Data Visualization, Mobile, Big Data, e-commerce
...
Copyright © 2014 Intridea Inc. All rights reserved.
BY WORKING REMOTELY
9,816 Hours Saved Annually
ACROSS THE US & OVERSEA...
Copyright © 2014 Intridea Inc. All rights reserved.
Major Open Source Contributions
OmniAuth is a flexible authentication
...
Copyright © 2014 Intridea Inc. All rights reserved.
INTRIDEA
Simplified and modernized application
experience for ADP
Appl...
Copyright © 2014 Intridea Inc. All rights reserved.
Where people go to make a difference
with their investment capital.
Sim...
Copyright © 2014 Intridea Inc. All rights reserved.
Copyright © 2014 Intridea Inc. All rights reserved.
ABS Scripture Analytics Query Requirements
Visualizations
Copyright © 2014 Intridea Inc. All rights reserved.
Public and private dashboards
visualizing Bible reading
across the Int...
Copyright © 2014 Intridea Inc. All rights reserved.
Mobile
Desktop
Tablet
Responsive Web App
www.scriptureanalytics.com
Copyright © 2014 Intridea Inc. All rights reserved.
Front End User Interface
Single Page Application using Backbone.js
Coff...
Copyright © 2014 Intridea Inc. All rights reserved.
Backend Servers/Services
Rails on Ruby Application mostly as the API s...
Copyright © 2014 Intridea Inc. All rights reserved.
MongoDB
Document oriented, schema free, JSON format
Very high data rea...
Copyright © 2014 Intridea Inc. All rights reserved.
Hadoop/Pig/Hive/Impala
Hadoop cluster (AWS Elastic Map/Reduce on-deman...
Copyright © 2014 Intridea Inc. All rights reserved.
Elastic Map/Reduce Hadoop Cluster - On Demand Processing
Copyright © 2014 Intridea Inc. All rights reserved.
Elastic Map/Reduce Hadoop Cluster - Terminated when done
Copyright © 2014 Intridea Inc. All rights reserved.
Cloudera CDH4 - on local 10-node cluster
Copyright © 2014 Intridea Inc. All rights reserved.
Cloudera CDH4 - Streaming Data into Hive Table
Copyright © 2014 Intridea Inc. All rights reserved.
Pig Sample Query
Copyright © 2014 Intridea Inc. All rights reserved.
Hive Sample Query
Copyright © 2014 Intridea Inc. All rights reserved.
Hive Query Results - Bible views by City
Copyright © 2014 Intridea Inc. All rights reserved.
Hive Query Results - Most popular verse before Mother’s day
28
Her chi...
Copyright © 2014 Intridea Inc. All rights reserved.
Hive Query Results - Most popular verse on Mother’s day
28
Her childre...
Copyright © 2014 Intridea Inc. All rights reserved.
Hive Query Results - Most popular verse after Mother’s day
28
Her chil...
Copyright © 2014 Intridea Inc. All rights reserved.
MongoDB Aggregation Framework Example
http://docs.mongodb.org/manual/c...
Copyright © 2014 Intridea Inc. All rights reserved.
MongoDB Aggregation Framework Example
Copyright © 2014 Intridea Inc. All rights reserved.
ABS Data Processing using Hadoop and MongoDB
Copyright © 2014 Intridea Inc. All rights reserved.
Analyzing Twitter using Hadoop and MongoDB
Copyright © 2014 Intridea Inc. All rights reserved.
Mobile and Data Visualization Project Showcase
ADP
BLiNQ
PEW Templeton...
Copyright © 2014 Intridea Inc. All rights reserved.
Redefining ADP’s
touch and desktop
experiences
ADP processes one out o...
Copyright © 2014 Intridea Inc. All rights reserved.
ADP TLM
Copyright © 2014 Intridea Inc. All rights reserved.
ADP HCR
Copyright © 2014 Intridea Inc. All rights reserved.
INTRIDEA
Copyright © 2014 Intridea Inc. All rights reserved.
Dashboard insights for 600 of
the world’s largest advertisers
Analytic...
Copyright © 2014 Intridea Inc. All rights reserved.
INTRIDEA
INTRIDEA
Copyright © 2014 Intridea Inc. All rights reserved.
INTRIDEA
INTRIDEA
Copyright © 2014 Intridea Inc. All rights reserved.
INTRIDEA
INTRIDEA
Copyright © 2014 Intridea Inc. All rights reserved.
Analyzing religious
change and its impact
on societies around the
worl...
Copyright © 2014 Intridea Inc. All rights reserved.
PEW Global Research
PEW Global Research
Copyright © 2014 Intridea Inc. All rights reserved.
PEW Global Research
PEW Global Research
Copyright © 2014 Intridea Inc. All rights reserved.
PEW Global Research
PEW Global Research
Copyright © 2014 Intridea Inc. All rights reserved.
Human advancement to
a higher stage
Human Progress seeks to document c...
Copyright © 2014 Intridea Inc. All rights reserved.
INTRIDEA
humanprogress.org
Gracias
Merci 
ありがとう
Danke 
谢谢
Thank You
Copyright © 2014 Intridea Inc. All rights reserved.
Tom Zeng
Director of Engineer...
Upcoming SlideShare
Loading in …5
×

RubyNation Visualizing Big Data on Small Devices

715 views
631 views

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
715
On SlideShare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

RubyNation Visualizing Big Data on Small Devices

  1. 1. Copyright © 2014 Intridea Inc. All rights reserved. Visualizing Big Data on Small Devices Tom Zeng Director of Engineering tom@intridea.com @tomzeng www.linkedin.com/in/tomzeng
  2. 2. Copyright © 2014 Intridea Inc. All rights reserved. Agenda Introduction Front End - HTML5/Bootstrap, Backbone/CoffeeScript, D3, MapBox Backend - Rails, MongoDB Big Data Processing - Hadoop, Hive, Pig Showcase - Mobile and Data Visualization Related Projects Q & A
  3. 3. Copyright © 2014 Intridea Inc. All rights reserved. Intridea - Rails, UX/Data Visualization, Mobile, Big Data, e-commerce American Bible Society (ABS http://www.americanbible.org/) - partners with Bible publishers · Provides API access to 539 Bible versions in 242 languages · The usage of the APIs is tracked at the verse level, along with ip location, timestamp, and duration · 530 million view logs/year(’12-’13 data), 1.5 mil/day, each view log has packed about 12 bible views · Amounts to 5-6 billion Bible views each year ABS asked Intridea to build the dashboard app Scripture Analytics (http://www.scriptureanalytics.com) Introduction
  4. 4. Copyright © 2014 Intridea Inc. All rights reserved. BY WORKING REMOTELY 9,816 Hours Saved Annually ACROSS THE US & OVERSEAS 30+ Employees FOUNDED & STARTED IN 2007 Washington D.C. We Make ! 🌎 # ON GITHUB Open Source Software $
  5. 5. Copyright © 2014 Intridea Inc. All rights reserved. Major Open Source Contributions OmniAuth is a flexible authentication system utilizing Rack middleware. OmniAuth An opinionated micro-framework for creating REST-like APIs in Ruby. Grape Hashie is a simple collection of useful Hash extensions. Hashie A Ruby wrapper for the OAuth 2.0 protocol. oauth2 A symbol font that makes it easy to create a map of the U.S. with HTML/CSS. Stately A generic swappable back-end for JSON handling. Multi_JSON Mission control dashboard for your distributed teams. Houston github.com/intridea
  6. 6. Copyright © 2014 Intridea Inc. All rights reserved. INTRIDEA Simplified and modernized application experience for ADP Application engineering for BusinessWeek.com World’s first 100% web based Point of Sale system
  7. 7. Copyright © 2014 Intridea Inc. All rights reserved. Where people go to make a difference with their investment capital. Simplifying ADP’s core business: Payroll Engineering for the most-trafficked wedding planning solution
  8. 8. Copyright © 2014 Intridea Inc. All rights reserved.
  9. 9. Copyright © 2014 Intridea Inc. All rights reserved. ABS Scripture Analytics Query Requirements Visualizations
  10. 10. Copyright © 2014 Intridea Inc. All rights reserved. Public and private dashboards visualizing Bible reading across the Internet Up to the minute dashboards showing what Bible verses are being read when, and where, all over the globe.
  11. 11. Copyright © 2014 Intridea Inc. All rights reserved. Mobile Desktop Tablet Responsive Web App www.scriptureanalytics.com
  12. 12. Copyright © 2014 Intridea Inc. All rights reserved. Front End User Interface Single Page Application using Backbone.js CoffeeScript (Ruby like, Jasmine in CoffeeScript similar to RSpec)
 D3 for Data Visualization Twitter Bootstrap for Responsive UI Packery for Responsive Layout - http://packery.metafizzy.co/ Mapbox for Map Rendering - https://www.mapbox.com/
  13. 13. Copyright © 2014 Intridea Inc. All rights reserved. Backend Servers/Services Rails on Ruby Application mostly as the API server MongoDB as the data store/cache Mongoid for Active Record like queries MongoDB Aggregation Framework for complex queries Pulling data periodically from S3 to populate the Mongo database Local R&D Hadoop and Mongo clusters for data exploration
  14. 14. Copyright © 2014 Intridea Inc. All rights reserved. MongoDB Document oriented, schema free, JSON format Very high data read and write throughput Rich query capabilities (aggregation framework), flexible indexes Scale with auto-sharded replica sets Map/Reduce in JavaScript
  15. 15. Copyright © 2014 Intridea Inc. All rights reserved. Hadoop/Pig/Hive/Impala Hadoop cluster (AWS Elastic Map/Reduce on-demand) to process and store data in S3 Pig to parse, transform, geo-code data Hive to query data and generate aggregated JSON reports Impala is similar to Hive (but much fast than the older version of Hive), used for ETL !
  16. 16. Copyright © 2014 Intridea Inc. All rights reserved. Elastic Map/Reduce Hadoop Cluster - On Demand Processing
  17. 17. Copyright © 2014 Intridea Inc. All rights reserved. Elastic Map/Reduce Hadoop Cluster - Terminated when done
  18. 18. Copyright © 2014 Intridea Inc. All rights reserved. Cloudera CDH4 - on local 10-node cluster
  19. 19. Copyright © 2014 Intridea Inc. All rights reserved. Cloudera CDH4 - Streaming Data into Hive Table
  20. 20. Copyright © 2014 Intridea Inc. All rights reserved. Pig Sample Query
  21. 21. Copyright © 2014 Intridea Inc. All rights reserved. Hive Sample Query
  22. 22. Copyright © 2014 Intridea Inc. All rights reserved. Hive Query Results - Bible views by City
  23. 23. Copyright © 2014 Intridea Inc. All rights reserved. Hive Query Results - Most popular verse before Mother’s day 28 Her children show their appreciation, and her husband praises her.
  24. 24. Copyright © 2014 Intridea Inc. All rights reserved. Hive Query Results - Most popular verse on Mother’s day 28 Her children show their appreciation, and her husband praises her.
  25. 25. Copyright © 2014 Intridea Inc. All rights reserved. Hive Query Results - Most popular verse after Mother’s day 28 Her children show their appreciation, and her husband praises her.
  26. 26. Copyright © 2014 Intridea Inc. All rights reserved. MongoDB Aggregation Framework Example http://docs.mongodb.org/manual/core/aggregation-pipeline/
  27. 27. Copyright © 2014 Intridea Inc. All rights reserved. MongoDB Aggregation Framework Example
  28. 28. Copyright © 2014 Intridea Inc. All rights reserved. ABS Data Processing using Hadoop and MongoDB
  29. 29. Copyright © 2014 Intridea Inc. All rights reserved. Analyzing Twitter using Hadoop and MongoDB
  30. 30. Copyright © 2014 Intridea Inc. All rights reserved. Mobile and Data Visualization Project Showcase ADP BLiNQ PEW Templeton - Global Religious Futures Cato Institute - HumanProgress ! !
  31. 31. Copyright © 2014 Intridea Inc. All rights reserved. Redefining ADP’s touch and desktop experiences ADP processes one out of every six paychecks in the United States. We’re bringing payroll into the decade of touch.
  32. 32. Copyright © 2014 Intridea Inc. All rights reserved. ADP TLM
  33. 33. Copyright © 2014 Intridea Inc. All rights reserved. ADP HCR
  34. 34. Copyright © 2014 Intridea Inc. All rights reserved. INTRIDEA
  35. 35. Copyright © 2014 Intridea Inc. All rights reserved. Dashboard insights for 600 of the world’s largest advertisers Analytics, planning, and flight management for social advertising campaigns and brand engagement.
  36. 36. Copyright © 2014 Intridea Inc. All rights reserved. INTRIDEA INTRIDEA
  37. 37. Copyright © 2014 Intridea Inc. All rights reserved. INTRIDEA INTRIDEA
  38. 38. Copyright © 2014 Intridea Inc. All rights reserved. INTRIDEA INTRIDEA
  39. 39. Copyright © 2014 Intridea Inc. All rights reserved. Analyzing religious change and its impact on societies around the world Interactive website exploring the patterns and trends in religions across the globe
  40. 40. Copyright © 2014 Intridea Inc. All rights reserved. PEW Global Research PEW Global Research
  41. 41. Copyright © 2014 Intridea Inc. All rights reserved. PEW Global Research PEW Global Research
  42. 42. Copyright © 2014 Intridea Inc. All rights reserved. PEW Global Research PEW Global Research
  43. 43. Copyright © 2014 Intridea Inc. All rights reserved. Human advancement to a higher stage Human Progress seeks to document changes in living standards in the past and present while explaining and exploring the best ways to improve conditions for people.
  44. 44. Copyright © 2014 Intridea Inc. All rights reserved. INTRIDEA humanprogress.org
  45. 45. Gracias Merci ありがとう Danke 谢谢 Thank You Copyright © 2014 Intridea Inc. All rights reserved. Tom Zeng Director of Engineering tom@intridea.com @tomzeng www.linkedin.com/in/tomzeng

×