Your SlideShare is downloading. ×
Data torrent meetup-productioneng
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.


Saving this for later?

Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime - even offline.

Text the download link to your phone

Standard text messaging rates apply

Data torrent meetup-productioneng


Published on

DataTorrent presentation at

DataTorrent presentation at

Published in: Technology, Education

1 Like
  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. Platform for Real-Time Production Operations Prepared for LSPE Meet-up November 21, 2013
  • 2. DataTorrent in Hadoop Ecosystem • Most powerful Hadoop platform for real-time stream computations • Massive Real-Time Production Monitoring, Analytics, and Alerting – Systems monitoring: Resource Utilization, Logs Analysis – Predictive Maintenance, DOS Attack, Launch Validation etc.
  • 3. DataTorrent Technology Stack Malhar – Open Source Operators and Apps Library (Apache v2 License) SLA Alerts Tools Web Services State Snapshot Security Scalability Fault Tolerance Partitioning Dynamic Modifications StrAM (Stream Application Master)
  • 4. DataTorrent’s Platform Differentiators . Extreme Scalability • • • Automatically scale to changing loads Sub-second latency with linear scalability Complex monitoring applications with massive computations Mission Critical • • • Built-in Stateful Faulttolerance. 24/7 uptime guaranteed Predictive Analysis, and trouble shooting Update your application while it's running! Hadoop-Native • • • Runs on your existing Apache Hadoop cluster. Develop faster with our open-source framework. Integrate seamlessly with your existing monitoring stack.
  • 5. Stream Processing Stream 3 Stream 1 Data Load Stream 4 Stream 2 Window 3 • • • • • Window 2 Window 1 A Stream is a sequence of data events with schema An Operator takes input streams and compute output streams An Application is a Directed Acyclic Graph (DAG) In-memory asynchronous distributed computations A Streaming Window is an atomic batch of sequential data events
  • 6. DataTorrent Hadoop GRID 1 4 3 2 DT Console dtCLI 6 5 Resource Manager NM MapReduce NM DT Gateway NM NM MapReduce StrAM MapReduce 3 1 MapReduce MapReduce 2 5 4 6 MapReduce
  • 7. Live Demonstration
  • 8. Open Sourced Production Operations Application Real-Time Dashboards and Actions • • • • • • DOS Attack Predictive maintenance of servers Pre and post Launch analysis 404 Response Root cause analysis for LAMP architecture Segmentation – – – – • Geo Location Gender, Age Resource usage (urls) Etc. URL Analysis – Response times – Patterns • Seamless integration into monitoring stacks
  • 9. How to get Started? • DataTorrent • Try Sandbox ( • Free for small to medium enterprises: Contact us for details • Malhar Open Source (Apache 2.0) project • • • Applications available Jan 2014 • LogStream: Site Operations • Map-Reduce Monitor DataTorrent Inc. 3200 Partrick Henry, 2nd Fl Santa Clara, CA 95054
  • 10. Platform Capabilities Scale able High Performance • Throughput in Billions Events/Sec • Latency in Milliseconds Powerful Tools • GUI For Cluster Performance Monitoring • GUI and Debuggers for Event Data • Test Framework, Certification, Versioning • CLI, Macros Easy To Use Fault-Tolerance • No State loss, No Message loss node outage recovery • State Management • Efficient State Checkpointing • Library of Operator Templates • Focus On Business Logic • Connectors to Current Tools • HDFS, Hbase, MySql, ActiveMQ • APIs for Tool Integrations Adaptability Native YARN Application • Runtime Scaling and Resource Optimization • Dynamic Application Modification •Integrates with Hadoop 2.0 Distributions •Apache, Cloudera, Hortonworks, MapR, Pivotal •Co-Exists with Existing Batch Infrastructure •Multi-Tenancy with Existing Hadoop Applications
  • 11. Appendix