Accelerating Deep Learning Training with BigDL and Drizzle on Apache Spark with Ding Ding and Shivaram Venkataraman
•
3 likes•486 views
The BigDL framework scales deep learning for large data sets using Apache Spark. However there is significant scheduling overhead from Spark when running BigDL at large scale. In this talk we propose a new parameter manager implementation that along with coarse-grained scheduling can provide significant speedups for deep learning models like Inception, VGG etc. Aggregation functions like reduce or treeReduce that are used for parameter aggregation in Apache Spark (and the original MapReduce) are slow as the centralized scheduling and driver network bandwidth become a bottleneck especially in large clusters. To reduce the overhead of parameter aggregation and allow for near-linear scaling, we introduce a new AllReduce operation, a part of the parameter manager in BigDL which is built directly on top of the BlockManager in Apache Spark. AllReduce in BigDL uses a peer-to-peer mechanism to synchronize and aggregate parameters. During parameter synchronization and aggregation, all nodes in the cluster play the same role and driver’s overhead is eliminated thus enabling near-linear scaling. To address the scheduling overhead we use Drizzle, a recently proposed scheduling framework for Apache Spark. Currently, Spark uses a BSP computation model, and notifies the scheduler at the end of each task. Invoking the scheduler at the end of each task adds overheads and results in decreased throughput and increased latency. Drizzle introduces group scheduling, where multiple iterations (or a group) of iterations are scheduled at once. This helps decouple the granularity of task execution from scheduling and amortizes the costs of task serialization and launch. Finally we will present results from using the new AllReduce operation and Drizzle on a number of common deep learning models including VGG and Inception. Our benchmarks run on Amazon EC2 and Google DataProc will show the speedups and scalability of our implementation.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Accelerating Deep Learning Training with BigDL and Drizzle on Apache Spark with Ding Ding and Shivaram Venkataraman
Similar to Accelerating Deep Learning Training with BigDL and Drizzle on Apache Spark with Ding Ding and Shivaram Venkataraman (20)
More from Databricks
More from Databricks (20)
Recently uploaded
Recently uploaded (20)
Accelerating Deep Learning Training with BigDL and Drizzle on Apache Spark with Ding Ding and Shivaram Venkataraman
- 1. Accelerating deep learning on apache spark Using BigDL with coarse-grained scheduling Shivaram Venkataraman (Microsoft Research, UC Berkeley) Ding Ding (Intel) Sergey Ermolin (Intel) June 2018
- 2. software.intel.com/bigdlbigdl-project.github.io BigDL is an open-source distributed deep learning library for Apache Spark* that can run directly on top of existing Spark or Apache Hadoop* clusters Feature Parity & Model Exchange with TensorFlow*, Caffe*, Keras, Torch* Lower TCO and improved ease of use with existing infrastructure Deep Learning on Big Data Platform, Enabling Efficient Scale-Out BigDL Spark Core HighPerformanceDeepLearningforApacheSpark*onCPUInfrastructure No need to deploy costly accelerators, duplicate data, or suffer through scaling headaches! Designed and Optimized for Intel® Xeon® Ideal for DL Models TRAINING and INFERENCE Powered by Intel® MKL and multi-threaded programming
- 3. software.intel.com/bigdlbigdl-project.github.io 3 BIGDL Jupyter, Zeppelin notebooks and TensorBoard support
- 4. SolutionsCloudServiceProvidersPlatforms Building&DeployingwithBigDL *Other names and brands may be claimed as the property of others. And Many More…https://bigdl-project.github.io Open Source Community support: 2496stars|500+forks|50contributors
- 5. software.intel.com/bigdlbigdl-project.github.io AnalyticsZoo Analytics + AI Pipelines for Spark and BigDL “Out-of-the-box” ready for use • Reference use cases • Fraud detection, time series prediction, sentiment analysis, chatbot, etc. • Predefined models • Object detection, image classification, text classification, recommendations, etc. • Feature transformations • Vision, text, 3D imaging, etc. • High level APIs • DataFrames, ML Pipelines, Keras/Keras2, etc. 5
- 6. software.intel.com/bigdlbigdl-project.github.io DeeplearningwithBIGdl/spark GetExcellentmulti-nodescalingandgenerationalperformance withyourexistinghardware Generational performance increase with BigDLNode Scaling with BigDL Optimization Notice: Intel's compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice. Benchmark results were obtained prior to implementation of recent software patches and firmware updates intended to address exploits referred to as "Spectre" and "Meltdown". Implementation of these updates may make these results inapplicable to your device or system. Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more complete information visit: http://www.intel.com/performance Source: Intel measured as of August 2017.
- 7. https://bigdl-project.github.io https://software.intel.com/bigdl 7 DeeplearningTraining All Iterative ML Algorithms exchange model parameters after each iterations (SGD, ADAM, etc) Model Param Update Model Param Update Model Param Update
- 8. https://bigdl-project.github.io https://software.intel.com/bigdl 8 ZOOMINGIN:Insideeachiteration Schedule Tasks Compute Gradient … A G G R E G A T E Model Update Training for (i <- 1 to N) { batch = next_batch() output = model.forward(batch.input) loss = criterion.forward(output, batch.target) error = criterion.backward(output, batch.target) model.backward(input, error) … optimMethod.optimize(model.weight, model.gradient) } Master node
- 9. https://bigdl-project.github.io https://software.intel.com/bigdl 9 Baseline:ParameterSynchronizationinSparkMLlib … Partition 1 Partition 2 Partition n Training Set Sample Sample Sample Worker Worker Worker Driver 2 2 2 1 1 1 3 3 3 4 1 2 3 4 Broadcast model parameters (weights) to each worker Each task computes gradients Each task sends gradients for (tree) aggregation, and then driver updates the weight
- 10. https://bigdl-project.github.io https://software.intel.com/bigdl 10 SynchronizationviaParameterManagerinBigDL All-Reduce synchronization without hotspot and shuffle Distributed Parameter Manager aggregates gradients and updates model Master Node is not involved!
- 11. https://bigdl-project.github.io https://software.intel.com/bigdl 11 SynchronizationviaParameterManagerinBigDL … PS (Parameter Server) Architecture in BigDL on top of Spark Block Manager Partition 1 Partition 2 Partition n Worker Gradient 1 2 Weight 3 4 5 Worker Gradient 1 2 Weight 3 4 5 Worker Gradient 1 2 Weight 3 4 5 … … … … … … … Peer-2-Peer All-Reduce synchronization
- 12. https://bigdl-project.github.io https://software.intel.com/bigdl 12 EFFECTSofParameterManagerimplementationinBigDL Parameter synchronization time as a fraction of average compute time for Inception v1 training • Linear scaling • 2x node increase – only 1% increase in parameter sync time
- 13. https://bigdl-project.github.io https://software.intel.com/bigdl 13 TaskSchedulingOverhead Total Spark overhead (task scheduling, task serdes, task fetch) as a fraction of average compute time for Inception-v1 training Cores
- 14. https://bigdl-project.github.io https://software.intel.com/bigdl 14 FoCUS:cuttingSPARKschedulingandcommsoverhead • Optimizing parameter synchronization and aggregation (PM) • Optimizing task scheduling (Drizzle) DL tasks are uniquely suited for Spark performance optimization: - Heavy master node workload during model update - Repetitive in nature (reusable task scheduling decisions) - Static data partitioning and cluster configuration
- 15. https://bigdl-project.github.io https://software.intel.com/bigdl insidethescheduler … … … … … … … … … Scheduler (master node)? (1) Decide how to assign tasks to machines - data locality - fair sharing (2) Serialize and send tasks … …
- 16. https://bigdl-project.github.io https://software.intel.com/bigdl SCHEDULINGOVERHEADS–Scalabilityproblem Cluster: 4 core, r3.xlarge machines Workload: Sum of 10k numbers per-core Median-task time breakdown 0 50 100 150 200 250 4 8 16 32 64 128 Time(ms) Machines Compute + Data Transfer Task Fetch Scheduler Delay
- 17. https://bigdl-project.github.io https://software.intel.com/bigdl insidethescheduler … … … … … … … … … Scheduler? (1) Decide how to assign tasks to machines data locality fair sharing (2) Serialize and send tasks … … ? BigDL tasks are Repetitive: Reuse scheduling decisions!
- 18. https://bigdl-project.github.io https://software.intel.com/bigdl … DRIZZLE … … … … … … … … (1) Pre-schedule reduce tasks (2) Group schedule iterations Goal: remove frequent scheduler interaction
- 19. https://bigdl-project.github.io https://software.intel.com/bigdl … … … (1) Pre-schedule reduce tasks Goal: Remove scheduler involvement for reduce tasks
- 21. https://bigdl-project.github.io https://software.intel.com/bigdl coordinatingshuffles:Pre-scheduling … … … (1) Pre-schedule reducers (2) Mappers get metadata (3) Mappers trigger reducers
- 22. https://bigdl-project.github.io https://software.intel.com/bigdl … Groupscheduling … … … … … … … … Group of 2 Schedule group of iterations at once Fault tolerance, scheduling, adding/removing nodes at group boundaries Group of 2
- 23. https://bigdl-project.github.io https://software.intel.com/bigdl 0 50 100 150 200 250 300 4 8 16 32 64 128 Time/Iter(ms) Machines Baseline Only Pre-Scheduling Drizzle-10 Drizzle-100 Micro-benchmark:2-stages 100 iterations – Breakdown of pre-scheduling, group-scheduling
- 24. https://bigdl-project.github.io https://software.intel.com/bigdl 26 ReducingSchedulingOverheadswithDrizzle For more complete information about performance and benchmark results, visit www.intel.com/benchmarks.
- 25. https://bigdl-project.github.io https://software.intel.com/bigdl 27 Drizzle–BigDLPerformanceImprovement YourMileagewillvary… 1.70 1.80 1.90 2.00 2.10 2.20 2.30 2.40 0 1 2 5 10 20 Iteration,sec Drizzle Size CIFAR, 32 nodes, 4cores, EC2 0.00 5.00 10.00 15.00 20.00 25.00 1 2 5 10 Epochtime,sec Drizzle ize LeNet, 32 nodes, 16 cores 1.6 1.65 1.7 1.75 1.8 1.85 1.9 1 2 5 10 Iterationtime,sec Drizzle size ImageNet, 64 nodes, 16 cores 15% improvement 24% improvement 10% improvement Group Size Group Size Group Size
- 26. https://bigdl-project.github.io https://software.intel.com/bigdl 28 cONCLUSION • Deep Learning Spark jobs are somewhat unique • Heavy master node load for large model parameter update • Relatively short execution tasks (for fast model conversion) • Scheduling/Comms sometimes takes ~50% of total task execution. • Deep Learning tasks are uniquely suited for optimization * Distributed Parameter Manager to offload Master compute. * Drizzle takes advantage of repetitive nature of the tasks and static data partitioning. * Need Spark committers community involvement
- 29. https://bigdl-project.github.io https://software.intel.com/bigdl 31 LegalNotices&disclaimers This document contains information on products, services and/or processes in development. All information provided here is subject to change without notice. Contact your Intel representative to obtain the latest forecast, schedule, specifications and roadmaps. Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Learn more at intel.com, or from the OEM or retailer. No computer system can be absolutely secure. Tests document performance of components on a particular test, in specific systems. Differences in hardware, software, or configuration will affect actual performance. Consult other sources of information to evaluate performance as you consider your purchase. For more complete information about performance and benchmark results, visit http://www.intel.com/performance. Cost reduction scenarios described are intended as examples of how a given Intel-based product, in the specified circumstances and configurations, may affect future costs and provide cost savings. Circumstances will vary. Intel does not guarantee any costs or cost reduction. Statements in this document that refer to Intel’s plans and expectations for the quarter, the year, and the future, are forward-looking statements that involve a number of risks and uncertainties. A detailed discussion of the factors that could affect Intel’s results and plans is included in Intel’s SEC filings, including the annual report on Form 10-K. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document. Intel does not control or audit third-party benchmark data or the web sites referenced in this document. You should visit the referenced web site and confirm whether referenced data are accurate. Intel, the Intel logo, Pentium, Celeron, Atom, Core, Xeon and others are trademarks of Intel Corporation in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others. © 2016 Intel Corporation. Intel Confidential