This document outlines the layers of a data stack from data generation to data visualization. It discusses topics like data collection and transport protocols, data storage formats and technologies, data processing paradigms like MapReduce and streaming, data analysis techniques including aggregation and machine learning, and data visualization software. Industry examples of implementing components of the data stack at companies like Akamai, Inmobi, and Helpshift are also provided.

1. How to build a Data Stack from
scratch ?
Vinayak Hegde
VP Engineering, Helpshift

2. From Data to Wisdom

3. Digging for insights

4. Layers of the Data Stack
Data Visualisation
Data Analysis
Data Processing
Data Storage
Data Collection and Transport
Data Generation

5. Data Generation
• What data needs to be generated
• Frequency of generation
• Pre-aggregated or sampled
• Accuracy of data generation
• Is sample representative of population ?
• Format of data
• Metadata enrichment
• Examples - Sensor reading, itemised store
purchase data, Ad Impression data

6. Data Collection & Transport
• Do some aggregation at source or send every
data point
• Store locally
• Push Vs Pull methodology. Pros & Cons
• Factors in choice of underlying transport
protocol
• Factors in choice of software

7. Protocols & Software
• TCP - connection oriented / reliable
• UDP - connection-less / unreliable
• MQTT - For sensor data
• HTTP - REST APIs
• Kafka
• Rabbit MQ
• Pros & Cons – Design Choices

8. Data Storage
• Storage media (SSD/Memory/Harddisk/
Network)
• Storage formats (B+Trees, Fractal Trees)
• Latencies of access
• Queryability
• Filesystem differences
• Examples

9. Data processing
• Cronjobs
• Maps-reduce paradigms
• PVM/MPI
• Iterative processing
• Lambda Architecture
• Microbatches
• Examples
• Compare and contrast Hadoop / Spark / Storm

10. Data Analysis
• Merge metadata
• Layer 3rd party data
• Geocoding
• Aggregation
• Incorporate human input
• Statistical analysis
• Machine learning

11. Data Visualisation
• Guiding principles for a good Viz
• Techniques and considerations
• Software (Tableau, Excel, ggplot2, gephi)

12. Industry examples & Comparing stacks
• Building a columnar database for analytics at
Akamai
• Building a big data system at Inmobi using
hadoop (processing 100s of terabytes of data)
• Using R for simple viz & analytics at Inmobi
• Using postgres and mallet for text analytics at
Helpshift

13. Data landscape
• Real-time processing systems (Storm)
• Complex Event processing (Esper)
• Big data batch (Hadoop)
• Big data iterative (Hadoop, Spark)
• Columnar Storage (Infobright,Vertica, RCFile)
• Memory-optimised systems (SAP Hana, Spark)
• Graph DB systems (neo4J, GraphX)

14. Outline Slides
• Thank you.