Slides from a talk which introduces infrastructure that powers your data applications. The data infrastructure ties together the distributed components, systems and processes to drive value from data. The topics covered were data collection, immutable logs, scaling ETL processes and real-time analytics. Example use cases of Kafka, Storm, Elasticsearch, and Amazon Redshift were presented.

1. Introduction to
Big Data Infrastructure
Vancouver
SMAC (Social, Mobile, Analytics & Cloud) Meetup
Oct 22, 2014
Ganesh Swami
www.silota.com

2. Hi
• Programming professionally for 10+ years
• x86 assembly, STL, boost, python-boost, python
• Built emacs-­‐wiki-­‐blog: first blogging engine for
Emacs!

3. What is Big Data?

4. What is Big Data?
3V: Volume, Velocity, Variety

5. The Big Data Zoo
Amazon Kinesis Riak Cassandra Hive
Apache Spark Apache Hadoop Pig Apache Storm
Kibana Tableu Apache Kafka
Elasticsearch Amazon EMR Redshift
Dynamo DB Riak HBase

6. The Zoo Organized
Ingest Store
Process/
Enrich
Visualize
Kafka S3 Hive/Pig/EMR Tableu
Data Answers
Kinesis DynamoDB Spark Kibana
Flume HDFS Storm
Scribe Redshift

8. Data Ingestion
Ingest
Layer
Mobile Apps
Websites
Internet of Things

9. Elasticsearch
Open-source search and analytics solution

10. Kibana

11. Amazon Redshift
Petabyte-scale data warehouse solution

12. What is Silota
• Building blocks of analytics
• A simple REST API
• to ingest
• to analyze
• to export
• based on Kafka, Storm and Elasticsearch

13. Silota -vs- Mixpanel
• Mixpanel for product people
• great UI
• cookie-cutter analysis for verticals (gaming, e-commerce)
• Silota is an API
• more low-level, full-power
• first class API: responses, pagination, errors, etc.

14. Keep in Touch!
Ganesh Swami
ganesh@silota.com
www.silota.com