Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
High Performance Processing of
Streaming Data
Workshops on Dynamic Data Driven Applications Systems(DDDAS) In
conjunction ...
Software Philosophy
• We use the concept of HPC-ABDS High Performance Computing
enhanced Apache Big Data Software Stack il...
3
Kaleidoscope of (Apache) Big Data Stack (ABDS) and HPC Technologies
Cross-
Cutting
Functions
1) Message
and Data
Protoco...
IOTCloud
• Device  Pub-SubStorm 
Datastore  Data Analysis
• Apache Storm provides scalable
distributed system for proc...
6 Forms of
MapReduce
cover “all”
circumstances
Describes
different aspects
- Problem
- Machine
- Software
If these differe...
Cloud controlled Robot Data Pipeline
612/16/2015
Message Brokers
RabbitMQ, Kafka
Gateway Sending
to
pub-sub
Sending
to
Per...
Simultaneous Localization & Mapping (SLAM)
𝑝(𝑥1:𝑡, 𝑚|𝑧1:𝑡, 𝑢1:𝑡−1) =
𝑝 𝑚 𝑥1:𝑡, 𝑧1:𝑡 𝑝(𝑥1:𝑡|𝑧1:𝑡, 𝑢1:𝑡−1
Particles are
dist...
Parallel SLAM Simultaneous Localization and
Mapping by Particle Filtering
812/16/2015
Speedup
Robot Latency Kafka & RabbitMQ
912/16/2015
Kinect with
Turtlebot
and
RabbitMQ
RabbitMQ
versus Kafka
SLAM Latency variations for 4 or 20 way parallelism
Jitter due to Application or System influences such as Network delays,...
Fault Tolerance at Message Broker
• RabbitMQ supports Queue replication and persistence to
disk across nodes for fault tol...
Parallel Overheads SLAM Simultaneous Localization
and Mapping: I/O and Garbage Collection
12/16/2015
12
Parallel Overheads SLAM Simultaneous Localization
and Mapping: Load Imbalance Overhead
12/16/2015
13
Multi-Robot Collision Avoidance
Streaming Workflow
Information
from robots
Runs in
parallel
• Second parallel Storm applic...
Lessons from using Storm
• We successfully parallelized Storm as core software of two
robot planning applications
• We nee...
16
Bringing Optimal Communications to Storm
12/16/2015
Both process based and thread based
parallelism is used
Worker and ...
Memory Mapped File based
Communication
• Inter process communications using shared memory for a
single node
• Multiple wri...
Optimized Broadcast Algorithms
• Binary tree
– Workers arranged in a binary tree
• Flat tree
– Broadcast from the origin t...
Java MPI performs better than Threads I
128 24 core Haswell nodes with Java Machine Learning
Default MPI much worse than t...
Java MPI performs better than Threads II
128 24 core Haswell nodes
2012/16/2015
200K Dataset Speedup
Speedups show classic parallel computing structure
with 48 node single core as “sequential”
State of art dimension reducti...
Experimental Configuration
• 11 Node cluster
• 1 Node – Nimbus & ZooKeeper
• 1 Node – RabbitMQ
• 1 Node – Client
• 8 Nodes...
Original
Binary Tree
Flat Tree
Bidirectional
Ring
Speedup of latency with both TCP based and Shared Memory based
communica...
Future Work
• Memory mapped communications require continuous
polling by a thread. If this tread does the processing of
th...
Conclusions on initial HPC-ABDS
use in Apache Storm
• Apache Storm worked well with performance
enhancements
• For Binary ...
Thank You
• References
– Our software https://github.com/iotcloud
– Apache Storm http://storm.apache.org/
– We will donate...
Spare SLAM Slides
12/16/2015
27
• IoTCloud uses Zookeeper,
Storm, Hbase, RabbitMQ
for robot cloud control
• Focus on high performance
(parallel) control f...
Latency with RabbitMQ
Different Message sizes in
bytes
Latency with Kafka
Note change in scales
for latency and
message si...
Robot Latency Kafka & RabbitMQ
Kinect with
Turtlebot
and
RabbitMQ
RabbitMQ
versus
Kafka
12/16/2015
30
Parallel SLAM Simultaneous Localization
and Mapping by Particle Filtering
12/16/2015
31
Spare High Performance
Storm Slides
12/16/2015
32
Memory Mapped Communication
12/16/2015
33
write Packet 1 Packet 2 Packet 3
Writer 01
Writer 02
Write
Write
Obtain the writ...
Default Broadcast
3412/16/2015
W-1
Worker
Node-1
B-1
W-3
Worker
W-2
W-5
Worker
Node-2
W-4
W-7
Worker
W-6
B-1 wants to broa...
Memory Mapped Communication
12/16/2015
35
No significant difference
because we are using all
the workers in the cluster
be...
Spare Parallel Tweet
Clustering with Storm Slides
12/16/2015
36
Parallel Tweet Clustering with Storm
• Judy Qiu, Emilio Ferrara and Xiaoming Gao
• Storm Bolts coordinated by ActiveMQ to ...
Parallel Tweet Clustering with Storm
3812/16/2015
• Speedup on up to 96 bolts on two clusters Moe and Madrid
• Red curve i...
Upcoming SlideShare
Loading in …5
×

High Performance Processing of Streaming Data

39,702 views

Published on

Describes two parallel robot planning algorithms implemented with Apache Storm on OpenStack -- SLAM (Simultaneous Localization & Mapping) and collision avoidance. Performance (response time) studied and improved as example of HPC-ABDS (High Performance Computing enhanced Apache Big Data Software Stack) concept.

Published in: Technology

High Performance Processing of Streaming Data

  1. 1. High Performance Processing of Streaming Data Workshops on Dynamic Data Driven Applications Systems(DDDAS) In conjunction with: 22nd International Conference on High Performance Computing (HiPC), Bengaluru, India 12/16/2015 1 Supun Kamburugamuve, Saliya Ekanayake, Milinda Pathirage and Geoffrey Fox December 16, 2015 gcf@indiana.edu http://www.dsc.soic.indiana.edu/, http://spidal.org/ http://hpc-abds.org/kaleidoscope/ Department of Intelligent Systems Engineering School of Informatics and Computing, Digital Science Center Indiana University Bloomington
  2. 2. Software Philosophy • We use the concept of HPC-ABDS High Performance Computing enhanced Apache Big Data Software Stack illustrated on next slide. • HPC-ABDS is a collection of 350 software systems used in either HPC or best practice Big Data applications. The latter include Apache, other open- source and commercial systems • HPC-ABDS helps ABDS by allowing HPC to add performance to ABDS software systems • HPC-ABDS helps HPC by bringing the rich functionality and software sustainability model of commercial and open source software. These bring a large community and expertise that is reasonably easy to find as it is broadly taught both in traditional courses and by community activities such as Meet up groups were for example: – Apache Spark 107,000 meet-up members in 233 groups – Hadoop 40,000 and installed in 32% of company data systems 2013 – Apache Storm 9,400 members • This talk focuses on Storm; its use and how one can add high performance 212/16/2015
  3. 3. 3 Kaleidoscope of (Apache) Big Data Stack (ABDS) and HPC Technologies Cross- Cutting Functions 1) Message and Data Protocols: Avro, Thrift, Protobuf 2) Distributed Coordination: Google Chubby, Zookeeper, Giraffe, JGroups 3) Security & Privacy: InCommon, Eduroam OpenStack Keystone, LDAP, Sentry, Sqrrl, OpenID, SAML OAuth 4) Monitoring: Ambari, Ganglia, Nagios, Inca 17) Workflow-Orchestration: ODE, ActiveBPEL, Airavata, Pegasus, Kepler, Swift, Taverna, Triana, Trident, BioKepler, Galaxy, IPython, Dryad, Naiad, Oozie, Tez, Google FlumeJava, Crunch, Cascading, Scalding, e-Science Central, Azure Data Factory, Google Cloud Dataflow, NiFi (NSA), Jitterbit, Talend, Pentaho, Apatar, Docker Compose 16) Application and Analytics: Mahout , MLlib , MLbase, DataFu, R, pbdR, Bioconductor, ImageJ, OpenCV, Scalapack, PetSc, Azure Machine Learning, Google Prediction API & Translation API, mlpy, scikit-learn, PyBrain, CompLearn, DAAL(Intel), Caffe, Torch, Theano, DL4j, H2O, IBM Watson, Oracle PGX, GraphLab, GraphX, IBM System G, GraphBuilder(Intel), TinkerPop, Google Fusion Tables, CINET, NWB, Elasticsearch, Kibana Logstash, Graylog, Splunk, Tableau, D3.js, three.js, Potree, DC.js 15B) Application Hosting Frameworks: Google App Engine, AppScale, Red Hat OpenShift, Heroku, Aerobatic, AWS Elastic Beanstalk, Azure, Cloud Foundry, Pivotal, IBM BlueMix, Ninefold, Jelastic, Stackato, appfog, CloudBees, Engine Yard, CloudControl, dotCloud, Dokku, OSGi, HUBzero, OODT, Agave, Atmosphere 15A) High level Programming: Kite, Hive, HCatalog, Tajo, Shark, Phoenix, Impala, MRQL, SAP HANA, HadoopDB, PolyBase, Pivotal HD/Hawq, Presto, Google Dremel, Google BigQuery, Amazon Redshift, Drill, Kyoto Cabinet, Pig, Sawzall, Google Cloud DataFlow, Summingbird 14B) Streams: Storm, S4, Samza, Granules, Google MillWheel, Amazon Kinesis, LinkedIn Databus, Facebook Puma/Ptail/Scribe/ODS, Azure Stream Analytics, Floe 14A) Basic Programming model and runtime, SPMD, MapReduce: Hadoop, Spark, Twister, MR-MPI, Stratosphere (Apache Flink), Reef, Hama, Giraph, Pregel, Pegasus, Ligra, GraphChi, Galois, Medusa-GPU, MapGraph, Totem 13) Inter process communication Collectives, point-to-point, publish-subscribe: MPI, Harp, Netty, ZeroMQ, ActiveMQ, RabbitMQ, NaradaBrokering, QPid, Kafka, Kestrel, JMS, AMQP, Stomp, MQTT, Marionette Collective, Public Cloud: Amazon SNS, Lambda, Google Pub Sub, Azure Queues, Event Hubs 12) In-memory databases/caches: Gora (general object from NoSQL), Memcached, Redis, LMDB (key value), Hazelcast, Ehcache, Infinispan 12) Object-relational mapping: Hibernate, OpenJPA, EclipseLink, DataNucleus, ODBC/JDBC 12) Extraction Tools: UIMA, Tika 11C) SQL(NewSQL): Oracle, DB2, SQL Server, SQLite, MySQL, PostgreSQL, CUBRID, Galera Cluster, SciDB, Rasdaman, Apache Derby, Pivotal Greenplum, Google Cloud SQL, Azure SQL, Amazon RDS, Google F1, IBM dashDB, N1QL, BlinkDB 11B) NoSQL: Lucene, Solr, Solandra, Voldemort, Riak, Berkeley DB, Kyoto/Tokyo Cabinet, Tycoon, Tyrant, MongoDB, Espresso, CouchDB, Couchbase, IBM Cloudant, Pivotal Gemfire, HBase, Google Bigtable, LevelDB, Megastore and Spanner, Accumulo, Cassandra, RYA, Sqrrl, Neo4J, Yarcdata, AllegroGraph, Blazegraph, Facebook Tao, Titan:db, Jena, Sesame Public Cloud: Azure Table, Amazon Dynamo, Google DataStore 11A) File management: iRODS, NetCDF, CDF, HDF, OPeNDAP, FITS, RCFile, ORC, Parquet 10) Data Transport: BitTorrent, HTTP, FTP, SSH, Globus Online (GridFTP), Flume, Sqoop, Pivotal GPLOAD/GPFDIST 9) Cluster Resource Management: Mesos, Yarn, Helix, Llama, Google Omega, Facebook Corona, Celery, HTCondor, SGE, OpenPBS, Moab, Slurm, Torque, Globus Tools, Pilot Jobs 8) File systems: HDFS, Swift, Haystack, f4, Cinder, Ceph, FUSE, Gluster, Lustre, GPFS, GFFS Public Cloud: Amazon S3, Azure Blob, Google Cloud Storage 7) Interoperability: Libvirt, Libcloud, JClouds, TOSCA, OCCI, CDMI, Whirr, Saga, Genesis 6) DevOps: Docker (Machine, Swarm), Puppet, Chef, Ansible, SaltStack, Boto, Cobbler, Xcat, Razor, CloudMesh, Juju, Foreman, OpenStack Heat, Sahara, Rocks, Cisco Intelligent Automation for Cloud, Ubuntu MaaS, Facebook Tupperware, AWS OpsWorks, OpenStack Ironic, Google Kubernetes, Buildstep, Gitreceive, OpenTOSCA, Winery, CloudML, Blueprints, Terraform, DevOpSlang, Any2Api 5) IaaS Management from HPC to hypervisors: Xen, KVM, Hyper-V, VirtualBox, OpenVZ, LXC, Linux-Vserver, OpenStack, OpenNebula, Eucalyptus, Nimbus, CloudStack, CoreOS, rkt, VMware ESXi, vSphere and vCloud, Amazon, Azure, Google and other public Clouds Networking: Google Cloud DNS, Amazon Route 53 21 layers Over 350 Software Packages May 15 2015 Green implies HPC Integration 12/16/2015 High Performance Computing Apache Big Data Software Stack
  4. 4. IOTCloud • Device  Pub-SubStorm  Datastore  Data Analysis • Apache Storm provides scalable distributed system for processing data streams coming from devices in real time. • For example Storm layer can decide to store the data in cloud storage for further analysis or to send control data back to the devices • Evaluating Pub-Sub Systems ActiveMQ, RabbitMQ, Kafka, Kestrel Turtlebot and Kinect 12/16/2015 4
  5. 5. 6 Forms of MapReduce cover “all” circumstances Describes different aspects - Problem - Machine - Software If these different aspects match, one gets good performance 512/16/2015
  6. 6. Cloud controlled Robot Data Pipeline 612/16/2015 Message Brokers RabbitMQ, Kafka Gateway Sending to pub-sub Sending to Persisting to storage Streamin g workflow A stream application with some tasks running in parallel Multiple streaming workflows Streaming Workflows Apache Storm Apache Storm comes from Twitter and supports Map- Dataflow-Streaming computing model Key ideas: Pub-Sub, fault-tolerance (Zookeeper), Bolts, Spouts
  7. 7. Simultaneous Localization & Mapping (SLAM) 𝑝(𝑥1:𝑡, 𝑚|𝑧1:𝑡, 𝑢1:𝑡−1) = 𝑝 𝑚 𝑥1:𝑡, 𝑧1:𝑡 𝑝(𝑥1:𝑡|𝑧1:𝑡, 𝑢1:𝑡−1 Particles are distributed in parallel tasks Application Build a map given the distance measurements from robot to objects around it and its pose Streaming Workflow Rao-Blackwellized particle filtering based algorithm for SLAM. Distribute the particles across parallel tasks and compute in parallel. Map building happens periodically12/16/2015 7
  8. 8. Parallel SLAM Simultaneous Localization and Mapping by Particle Filtering 812/16/2015 Speedup
  9. 9. Robot Latency Kafka & RabbitMQ 912/16/2015 Kinect with Turtlebot and RabbitMQ RabbitMQ versus Kafka
  10. 10. SLAM Latency variations for 4 or 20 way parallelism Jitter due to Application or System influences such as Network delays, Garbage collection and Scheduling of tasks 1012/16/2015 No Cut Fluctuations decrease after Cut on #iterations per swarm member
  11. 11. Fault Tolerance at Message Broker • RabbitMQ supports Queue replication and persistence to disk across nodes for fault tolerance • Can use a cluster of RabbitMQ brokers to achieve high availability and fault tolerance • Kafka stores the messages in disk and supports replication of topics across nodes for fault tolerance. Kafka's storage first approach may increase reliability but can introduce increased latency • Multiple Kafka brokers can be used to achieve high availability and fault tolerance
  12. 12. Parallel Overheads SLAM Simultaneous Localization and Mapping: I/O and Garbage Collection 12/16/2015 12
  13. 13. Parallel Overheads SLAM Simultaneous Localization and Mapping: Load Imbalance Overhead 12/16/2015 13
  14. 14. Multi-Robot Collision Avoidance Streaming Workflow Information from robots Runs in parallel • Second parallel Storm application • Velocity Obstacles (VOs) along with other constrains such as acceleration and max velocity limits, • Non-Holonomic constraints, for differential robots, and localization uncertainty. • NPC NPS measure parallelism Control Latency # Collisions versus number of robots 12/16/2015 14
  15. 15. Lessons from using Storm • We successfully parallelized Storm as core software of two robot planning applications • We needed to replace Kafka by RabbitMQ to improve performance – Kafka had large variations in response time • We reduced Garbage Collection overheads • We see that we need to generalize Storm’s – Map-Dataflow Streaming architecture to – Map-Dataflow/Collective Streaming architecture • Now we use HPC-ABDS to improve Storm communication performance 1512/16/2015
  16. 16. 16 Bringing Optimal Communications to Storm 12/16/2015 Both process based and thread based parallelism is used Worker and Task distribution of Storm A worker hosts multiple tasks. B-1 is a task of component B and W-1 is a task of W Communication links are between workers These are multiplexed among the tasks W-1 Worker Node-1 B-1 W-3 Worker W-2 W-5 Worker Node-2 W-4 W-7 Worker W-6 W-1 Worker Node-1 B-1 W-3 Worker W-2 W-5 Worker Node-2 W-4 W-7 Worker W-6
  17. 17. Memory Mapped File based Communication • Inter process communications using shared memory for a single node • Multiple writer single reader design • A memory mapped file is created for each worker of a node • Create the file under /dev/shm • Writer breaks the message in to packets and puts them to file • Reader reads the packets and assemble the message • When a file becomes full move to another file • PS all of this “well known” BUT not deployed 12/16/2015 17
  18. 18. Optimized Broadcast Algorithms • Binary tree – Workers arranged in a binary tree • Flat tree – Broadcast from the origin to 1 worker in each node sequentially. This worker broadcast to other workers in the node sequentially • Bidirectional Rings – Workers arranged in a line – Starts two broadcasts from the origin and these traverse half of the line • All well known and we have used similar ideas of basic HPC- ABDS to improve MPI for machine learning (using Java) 12/16/2015 18
  19. 19. Java MPI performs better than Threads I 128 24 core Haswell nodes with Java Machine Learning Default MPI much worse than threads Optimized MPI using shared memory node-based messaging is much better than threads 1912/16/2015
  20. 20. Java MPI performs better than Threads II 128 24 core Haswell nodes 2012/16/2015 200K Dataset Speedup
  21. 21. Speedups show classic parallel computing structure with 48 node single core as “sequential” State of art dimension reduction routine Speedups improve as problem size increases 48 nodes, 1 core to 128 nodes 24 cores is potential speedup of 64 2112/16/2015
  22. 22. Experimental Configuration • 11 Node cluster • 1 Node – Nimbus & ZooKeeper • 1 Node – RabbitMQ • 1 Node – Client • 8 Nodes – Supervisors with 4 workers each • Client sends messages with the current timestamp, the topology returns a response with the same time stamp. Latency = current time - timestamp 12/16/2015 22 W-1 W-5 W-n B-1R-1 G-1RabbitMQ RabbitMQ Client
  23. 23. Original Binary Tree Flat Tree Bidirectional Ring Speedup of latency with both TCP based and Shared Memory based communications for different algorithms and sizes 12/16/2015 23 Original and new Storm Broadcast Algorithms
  24. 24. Future Work • Memory mapped communications require continuous polling by a thread. If this tread does the processing of the message, the polling overhead can be reduced. • Scheduling of tasks should take the communications in to account • The current processing model has multiple threads processing a message at different stages. Reduce the number of threads to achieve predictable performance • Improve the packet structure to reduce the overhead • Compare with related Java MPI technology • Add additional collectives to those supported by Storm 12/16/2015 24
  25. 25. Conclusions on initial HPC-ABDS use in Apache Storm • Apache Storm worked well with performance enhancements • For Binary tree performed the best • Algorithms reduces the network traffic • Shared memory communications reduce the latency further • Memory mapped file communications improve performance 12/16/2015 25
  26. 26. Thank You • References – Our software https://github.com/iotcloud – Apache Storm http://storm.apache.org/ – We will donate software to Storm – SLAM paper http://dsc.soic.indiana.edu/publications/SLAM_In_ the_cloud.pdf – Collision Avoidance paper http://goo.gl/xdB8LZ 12/16/2015 26
  27. 27. Spare SLAM Slides 12/16/2015 27
  28. 28. • IoTCloud uses Zookeeper, Storm, Hbase, RabbitMQ for robot cloud control • Focus on high performance (parallel) control functions • Guaranteed real time response 12/16/2015 28 Parallel simultaneous localization and mapping (SLAM) in the cloud
  29. 29. Latency with RabbitMQ Different Message sizes in bytes Latency with Kafka Note change in scales for latency and message size 12/16/2015 29
  30. 30. Robot Latency Kafka & RabbitMQ Kinect with Turtlebot and RabbitMQ RabbitMQ versus Kafka 12/16/2015 30
  31. 31. Parallel SLAM Simultaneous Localization and Mapping by Particle Filtering 12/16/2015 31
  32. 32. Spare High Performance Storm Slides 12/16/2015 32
  33. 33. Memory Mapped Communication 12/16/2015 33 write Packet 1 Packet 2 Packet 3 Writer 01 Writer 02 Write Write Obtain the write location atomically and increment Shared File Reader Read packet by packet sequentially Use a new file when the file size is reached Reader deletes the files after it reads them fully ID No of Packets Packet No Dest Task Content Length Source Task Stream Length Stream Content 16 4 4 4 4 4 4Bytes Fields Packet Structure
  34. 34. Default Broadcast 3412/16/2015 W-1 Worker Node-1 B-1 W-3 Worker W-2 W-5 Worker Node-2 W-4 W-7 Worker W-6 B-1 wants to broadcast a message to W, it sends 6 messages through 3 TCP communication channels and send 1 message to W-1 via shared memory
  35. 35. Memory Mapped Communication 12/16/2015 35 No significant difference because we are using all the workers in the cluster beyond 30 workers capacity A topology with pipeline going through all the workers Non Optimized Time
  36. 36. Spare Parallel Tweet Clustering with Storm Slides 12/16/2015 36
  37. 37. Parallel Tweet Clustering with Storm • Judy Qiu, Emilio Ferrara and Xiaoming Gao • Storm Bolts coordinated by ActiveMQ to synchronize parallel cluster center updates – add loops to Storm • 2 million streaming tweets processed in 40 minutes; 35,000 clusters 3712/16/2015 Sequential Parallel – eventually 10,000 bolts
  38. 38. Parallel Tweet Clustering with Storm 3812/16/2015 • Speedup on up to 96 bolts on two clusters Moe and Madrid • Red curve is old algorithm; • green and blue new algorithm • Full Twitter – 1000 way parallelism • Full Everything – 10,000 way parallelism

×