An overview of Lookout's data pipelines in to Hadoop. Talk given by Yash Ranadive @ Sift Science

1. HADOOP AT LOOKOUT
Yash Ranadive
@yashranadive
December 9, 2014

2. AGENDA
What we do @Lookout
Analytics Architecture
Event Ingestion Pipelines
Storm
Questions

3. BIO
Data Engineer at Lookout, started 2013
Previously at Demandbase, Project Perf. Corp.
6 years of Data Engineering
From Mumbai, India
etl.svbtle.com

4. WHAT WE DO
@LOOKOUT

5. MAKE SMARTPHONES AND TABLETS SECURE
& TRUSTWORTHY FOR INDIVIDUALS AND
ORGANIZATIONS

7. ANALYTICS
INFRASTRUCTURE

8. WHO WE SERVE
Product Managers
Software Developers
Marketing
Security Research

9. DATA ANALYTICS TEAM
7 Analysts/Scientists
Questions they answer
How many users located their phone yesterday?
How many users were billed for AT&T?

10. DATA ANALYTICS TEAM
3 Data Engineers
Build and maintain pipelines for Analytics and ad-hoc querying

13. REPORTING
Tableau
Dashboards - Retentions, Activations, etc.
Email reports
Custom email reports (Ruby)

14. ADHOC QUERYING
Hive CLI - Command-line interface to Hive
Hue - Toad style GUI for ad hoc queries on Hive
R Studio - Statistical analysis
Shiny - Reporting/Querying tool based on R
Sparkle Pony(Homegrown Ruby app) - MySQL Querying for
stakeholders
Hadoop File System Browser

15. EVENT INGESTION
PIPELINES

16. EVENT INGESTION
Event data comes via:
text(JSON)
binary(Protobufs)
binlogs

17. PIPELINES

18. STORM

19. STORM
Apache Storm is a distributed realtime computation system. It
can be used with any programming language.

20. TOPOLOGY DESIGN

21. NIMBUS AND SUPERVISOR
A storm cluster has
One Nimbus node which is the master
A set of Supervisor nodes which are the workers

22. TOPOLOGY CONSISTS OF SPOUTS AND BOLTS

23. A SAMPLE TOPOLOGY

24. LANDING DATA IN HADOOP
Topologies write data to a landing directory in Hadoop using
HDFS Bolt
Directories are rotated depending on latency requirements of
downstream reports
Directories are moved to location of the table in Hive

25. STORM PARALLELISM

26. TOPOLOGY DEPLOYMENT

27. DEPLOYMENT
Storm topologies are jars that can be submitted to Storm Nimbus
storm jar path/to/allmycode.jar org.MyTopologyClass arg1 arg2

28. DEPLOYMENT
Configuration is stored in shell scripts that launch topologies
storm jar /topolgoies/data-storm-0.0.3-SNAPSHOT.jar com.lookout.data.topology.KafkaToHdfsTopology
-topologyname kafka-hdfs
-nimbushost dw-storm2
-topologymaxtaskparallelism 1
-D hdfs.sync.tuple.count=500
-D hdfs.file.rotation.seconds=3600
-D hdfs.landing.directory=/user/hive/warehouse/staging.db/locate_events
-D hdfs.destination.directory=/user/hive/warehouse/realdb.db/locate_events
-D hdfs.filesystem.url=hdfs://hadoop-cluster-01:8020/
-D kafka.zookeeper.hosts=zk1:2181,zk2:2181,zk3:2181
-D kafka.topic=locate_event
-D statsd.host=statsdhost

29. CONFIGURATION
MANAGEMENT ?

30. METRICS MONITORING

31. METRICS MONITORING
Use Storm's Metrics API (counters)
Success/Failure metrics are sent to StatsD for aggregation
Visualized using graphite

32. GRAPHITE

33. SLIDES

34. SLIDES
Javascript Slides - reveal.js
http://lab.hakim.se/reveal-js/#/

35. QUESTIONS
z