London Spark Meetup Project Tungsten Oct 12 2015

Chris Fregly
Chris FreglyAI and Machine Learning @ AWS, O'Reilly Author @ Data Science on AWS, Founder @ PipelineAI, Formerly Databricks, Netflix,
••
How Spark Beat Hadoop @ 100 TB Sort + Project Tungsten London Spark Meetup Chris Fregly, Principal Data Solutions Engineer IBM Spark Technology Center Oct 12, 2015 Power of data. Simplicity of design. Speed of innovation. IBM | spark.tc
IBM | spark.tc Announcements Skimlinks! ! Martin Goodson! Organizer, London Spark Meetup! !
IBM | spark.tc Who am I?! ! Streaming Data Engineer! Netflix Open Source Committer! ! Data Solutions Engineer! Apache Contributor! ! Principal Data Solutions Engineer! IBM Technology Center! Meetup Organizer! Advanced Apache Meetup! Book Author! Advanced Spark (2016)!
IBM | spark.tc Advanced Apache Spark Meetup Total Spark Experts: 1200+ in only 3 mos!! #5 most active Spark Meetup in the world!! ! Goals! Dig deep into the Spark & extended-Spark codebase! ! Study integrations such as Cassandra, ElasticSearch,! Tachyon, S3, BlinkDB, Mesos, YARN, Kafka, R, etc! ! Surface and share the patterns and idioms of these ! well-designed, distributed, big data components!
IBM | spark.tc Recent Events Cassandra Summit 2015! Real-time Advanced Analytics w/ Spark & Cassandra! ! ! ! ! ! Strata NYC 2015! Practical Data Science w/ Spark: Recommender Systems! ! Available on Slideshare! http://slideshare.net/cfregly! !
IBM | spark.tc Freg-a-palooza Upcoming World Tour   London Spark Meetup (Oct 12th)!   Scotland Data Science Meetup (Oct 13th)!   Dublin Spark Meetup (Oct 15th)!   Barcelona Spark Meetup (Oct 20th)!   Madrid Spark/Big Data Meetup (Oct 22nd)!   Paris Spark Meetup (Oct 26th)!   Amsterdam Spark Summit (Oct 27th – Oct 29th)!   Delft Dutch Data Science Meetup (Oct 29th) !   Brussels Spark Meetup (Oct 30th)!   Zurich Big Data Developers Meetup (Nov 2nd)! High probability! I’ll end up in jail! or married!!
Daytona GraySort tChallenge sortbenchmark.org!
IBM | spark.tc Topics of this Talk: Mechanical Sympathy! Tungsten => Bare Metal! Seek Once, Scan Sequentially!! CPU Cache Locality and Efficiency! Use Data Structs Customized to Your Workload! Go Off-Heap Whenever Possible ! spark.unsafe.offHeap=true!
IBM | spark.tc What is the Daytona GraySort Challenge?! Key Metric! Throughput of sorting 100TB of 100 byte data,10 byte key! Total time includes launching app and writing output file! ! Daytona! App must be general purpose! ! Gray! Named after Jim Gray!
IBM | spark.tc Daytona GraySort Challenge: Input and Resources! Input! Records are 100 bytes in length! First 10 bytes are random key! Input generator: ordinal.com/gensort.html! 28,000 fixed-size partitions for 100 TB sort! 250,000 fixed-size partitions for 1 PB sort! 1 partition = 1 HDFS block = 1 executor ! Aligned to avoid partial read I/O ie. imaginary data ^----^! Hardware and Runtime Resources! Commercially available and off-the-shelf! Unmodified, no over/under-clocking! Generates 500TB of disk I/O, 200TB network I/O! 1st record of! 1st 10 bytes:! “JimGrayRIP”!
IBM | spark.tc Daytona GraySort Challenge: Rules! Must sort to/from OS files in secondary storage! ! No raw disk since I/O subsystem is being tested! ! File and device striping (RAID 0) are encouraged! ! Output file(s) must have correct key order!
IBM | spark.tc Daytona GraySort Challenge: Task Scheduling! Types of Data Locality! PROCESS_LOCAL! NODE_LOCAL! RACK_LOCAL! ANY! ! Delay Scheduling! spark.locality.wait.node: time to wait for next shitty level! Set to infinite to reduce shittiness, force NODE_LOCAL! Straggling Executor JVMs naturally fade away on each run! Decreasing! Level of! Read ! Performance!
IBM | spark.tc Daytona GraySort Challenge: Winning Results! On-disk only, in-memory caching disabled!! EC2 (i2.8xlarge)! EC2 (i2.8xlarge)! 28,000! partitions! 250,000 ! partitions (!!)! (3 GBps/node! * 206 nodes)!
IBM | spark.tc Daytona GraySort Challenge: EC2 Configuration! 206 EC2 Worker nodes, 1 Master node! AWS i2.8xlarge! 32 Intel Xeon CPU E5-2670 @ 2.5 Ghz! 244 GB RAM, 8 x 800GB SSD, RAID 0 striping, ext4! NOOP I/O scheduler: FIFO, request merging, no reordering! 3 GBps mixed read/write disk I/O per node! Deployed within Placement Group/VPC! Enhanced Networking! Single Root I/O Virtualization (SR-IOV): extension of PCIe! 10 Gbps, low latency, low jitter (iperf showed ~9.5 Gbps)!
IBM | spark.tc Daytona GraySort Challenge: Winning Configuration! Spark 1.2, OpenJDK 1.7_<amazon-something>_u65-b17! Disabled in-memory caching -- all on-disk!! HDFS 2.4.1 short-circuit local reads, 2x replication! Writes flushed after each of the 5 runs! 28,000 partitions / (206 nodes * 32 cores) = 4.25 runs, round up 5 runs! Netty 4.0.23.Final with native epoll! Speculative Execution disabled: spark.speculation=false! Force NODE_LOCAL: spark.locality.wait.node=Infinite ! Force Netty Off-Heap: spark.shuffle.io.preferDirectBuffers! Spilling disabled: spark.shuffle.spill=false! All compression disabled (network, on-disk, etc)!
IBM | spark.tc Daytona GraySort Challenge: Partitioning! Range Partitioning (vs. Hash Partitioning)! Take advantage of sequential key space! Similar keys grouped together within a partition! Ranges defined by sampling 79 values per partition! Driver sorts samples and defines range boundaries! Sampling took ~10 seconds for 28,000 partitions! !
IBM | spark.tc Daytona GraySort Challenge: Why Bother?! Sorting relies heavily on shuffle, I/O subsystem! ! Shuffle is major bottleneck in big data processing! Large number of partitions can exhaust OS resources! ! Shuffle optimization benefits all high-level libraries! ! Goal is to saturate network controller on all nodes! ~125 MB/s (1 GB ethernet), 1.25 GB/s (10 GB ethernet)!
IBM | spark.tc Daytona GraySort Challenge: Per Node Results! ! ! ! ! ! Reducers: ~1.1 GB/s/node network I/O! (max 1.25 Gbps for 10 GB ethernet)! Mappers: 3 GB/s/node disk I/O (8x800 SSD)! 206 nodes ! * ! 1.1 Gbps/node ! ~=! 220 Gbps !
Quick Shuffle Refresher
! ! ! ! ! ! ! ! ! ! ! ! IBM | spark.tc Shuffle Overview! All to All, Cartesian Product Operation! Least ->! Useful! Example! I Could! Find ->!
! ! ! ! ! ! ! ! ! ! ! ! IBM | spark.tc Spark Shuffle Overview! Most ->! Confusing! Example! I Could! Find ->! Stages are Defined by Shuffle Boundaries!
IBM | spark.tc Shuffle Intermediate Data: Spill to Disk! Intermediate shuffle data stored in memory! Spill to Disk! spark.shuffle.spill=true! spark.shuffle.memoryFraction=% of all shuffle buffers! Competes with spark.storage.memoryFraction! Bump this up from default!! Will help Spark SQL, too.! Skipped Stages! Reuse intermediate shuffle data found on reducer! DAG for that partition can be truncated!
IBM | spark.tc Shuffle Intermediate Data: Compression! spark.shuffle.compress! Compress outputs (mapper)! ! spark.shuffle.spill.compress! Compress spills (reducer)! ! spark.io.compression.codec! LZF: Most workloads (new default for Spark)! Snappy: LARGE workloads (less memory required to compress)!
IBM | spark.tc Spark Shuffle Operations! join! distinct! cogroup! coalesce! repartition! sortByKey! groupByKey! reduceByKey! aggregateByKey!
IBM | spark.tc Spark Shuffle Managers! spark.shuffle.manager = {! hash < 10,000 Reducers! Output file determined by hashing the key of (K,V) pair! Each mapper creates an output buffer/file per reducer! Leads to M*R number of output buffers/files per shuffle! sort >= 10,000 Reducers! Default since Spark 1.2! Wins Daytona GraySort Challenge w/ 250,000 reducers!!! tungsten-sort -> Default in Spark 1.5! Uses com.misc.Unsafe for direct access to off heap! }!
IBM | spark.tc Shuffle Managers!
IBM | spark.tc Shuffle Performance Tuning! Hash Shuffle Manager (no longer default)! spark.shuffle.consolidateFiles: mapper output files! o.a.s.shuffle.FileShuffleBlockResolver! Intermediate Files! Increase spark.shuffle.file.buffer: reduce seeks & sys calls! Increase spark.reducer.maxSizeInFlight if memory allows! Use smaller number of larger workers to reduce total files! SQL: BroadcastHashJoin vs. ShuffledHashJoin! spark.sql.autoBroadcastJoinThreshold ! Use DataFrame.explain(true) or EXPLAIN to verify!
Mechanical Sympathy
IBM | spark.tc Mechanical Sympathy! Use as much of the CPU cache line as possible!!! ! ! ! ! ! ! ! ! !
IBM | spark.tc Naïve Matrix Multiplication: Not Cache Friendly! Naive:! for (i = 0; i < N; ++i)! for (j = 0; j < N; ++j)! for (k = 0; k < N; ++k)! res[i][j] += mat1[i][k] * mat2[k][j];! Clever: ! double mat2transpose [N][N];! for (i = 0; i < N; ++i)! for (j = 0; j < N; ++j)! mat2transpose[i][j] = mat2[j][i];! for (i = 0; i < N; ++i)! for (j = 0; j < N; ++j)! for (k = 0; k < N; ++k)! res[i][j] += mat1[i][k] * mat2transpose[j][k];! Prefetch Not Effective On ! Row Wise Traversal! Force All ! Column Traversal by! Transposing Matrix 2!
Winning Optimizations Deployed across Spark 1.1 and 1.2
IBM | spark.tc Daytona GraySort Challenge: Winning Optimizations! CPU-Cache Locality: Mechanical Sympathy! & Cache Locality/Alignment! ! Optimized Sort Algorithm: Elements of (K, V) Pairs! ! Reduce Network Overhead: Async Netty, epoll! ! Reduce OS Resource Utilization: Sort Shuffle!
IBM | spark.tc CPU-Cache Locality: Mechanical Sympathy! AlphaSort paper ~1995! Chris Nyberg and Jim Gray! ! Naïve! List (Pointer-to-Record)! Requires Key to be dereferenced for comparison! ! AlphaSort! List (Key, Pointer-to-Record)! Key is directly available for comparison! ! Key! Ptr! Ptr!
IBM | spark.tc CPU-Cache Locality: Cache Locality/Alignment! Key(10 bytes) + Pointer(4 bytes*) = 14 bytes! *4 bytes when using compressed OOPS (<32 GB heap)! Not binary in size! Not CPU-cache friendly! Cache Alignment Options! Add Padding (2 bytes)! Key(10 bytes) + Pad(2 bytes) + Pointer(4 bytes)=16 bytes! (Key-Prefix, Pointer-to-Record)! Key distribution affects performance! Prefix (4 bytes) + Pointer (4 bytes) = 8 bytes! Key! Key! Ptr! Ptr! Ptr! Key-Prefx! Pad! With Padding! Cache-line! Friendly!
IBM | spark.tc CPU-Cache Locality: Performance Comparison!
IBM | spark.tc Similar Technique: Direct Cache Access! ^ Packet header placed into CPU cache ^!
IBM | spark.tc Optimized Sort Algorithm: Elements of (K, V) Pairs! o.a.s.util.collection.TimSort! Based on JDK 1.7 TimSort! Performs best on partially-sorted datasets ! Optimized for elements of (K,V) pairs! Sorts impl of SortDataFormat (ie. KVArraySortDataFormat)! ! o.a.s.util.collection.AppendOnlyMap! Open addressing hash, quadratic probing! Array of [(K, V), (K, V)] ! Good memory locality! Keys never removed, values only append! (^2 Probing)!
IBM | spark.tc Reduce Network Overhead: Async Netty, epoll! New Network Module based on Async Netty! Replaces old java.nio, low-level, socket-based code! Zero-copy epoll uses kernel-space between disk & network! Custom memory management reduces GC pauses! spark.shuffle.blockTransferService=netty! Spark-Netty Performance Tuning! spark.shuffle.io.numConnectionsPerPeer! Increase to saturate hosts with multiple disks! spark.shuffle.io.preferDirectBuffers! On or Off-heap (Off-heap is default)!
IBM | spark.tc Hash Shuffle Manager!! ! ! ! ! ! ! ! ! ! ! M*R num open files per shuffle; M=num mappers! R=num reducers! Mapper Opens 1 File per Partition/Reducer! HDFS! (2x repl)! HDFS! (2x repl)!
S! IBM | spark.tc Reduce OS Resource Utilization: Sort Shuffle! ! ! ! ! ! ! ! ! M open files per shuffle; M = num of mappers! spark.shuffle.sort.bypassMergeThreshold! Merge Sort! (Disk)! Reducers seek and scan from range offset! of Master File on Mapper! TimSort! (RAM)! HDFS! (2x repl)! HDFS! (2x repl)! SPARK-2926:! Replace TimSort w/ Merge Sort! (Memory)! Mapper Merge Sorts Partitions into 1 Master File Indexed by Partition Range Offsets! <- Master->! File!
Project Tungsten Deployed across Spark 1.4 and 1.5
IBM | spark.tc Significant Spark Core Changes! Disk! Network! CPU! Memory! Daytona GraySort Optimizations! (Spark 1.1-1.2, Late 2014)! Tungsten Optimizations! (Spark 1.4-1.5, Late 2015)!
IBM | spark.tc Why is CPU the Bottleneck?! Network and Disk I/O bandwidth are relatively high! ! GraySort optimizations improved network & shuffle! ! Predicate pushdowns and partition pruning! ! Columnar file formats like Parquet and ORC! ! CPU used for serialization, hashing, compression!
IBM | spark.tc tungsten-sort Shuffle Manager! “I don’t know your data structure, but my array[] will beat it!” Custom Data Structures for Sort/Shuffle Workload! UnsafeRow: ! ! ! ! Rows are ! 8-byte aligned Primitives are inlined! Row.equals(), Row.hashCode()! operate on raw bytes! Offset (Int) and Length (Int)! Stored in a single Long!
IBM | spark.tc sun.misc.Unsafe! Info! addressSize()! pageSize()! Objects! allocateInstance()! objectFieldOffset()! Classes! staticFieldOffset()! defineClass()! defineAnonymousClass()! ensureClassInitialized()! Synchronization! monitorEnter()! tryMonitorEnter()! monitorExit()! compareAndSwapInt()! putOrderedInt()! Arrays! arrayBaseOffset()! arrayIndexScale()! Memory! allocateMemory()! copyMemory()! freeMemory()! getAddress() – not guaranteed correct if GC occurs! getInt()/putInt()! getBoolean()/putBoolean()! getByte()/putByte()! getShort()/putShort()! getLong()/putLong()! getFloat()/putFloat()! getDouble()/putDouble()! getObjectVolatile()/putObjectVolatile()! Used by Spark!
IBM | spark.tc Spark + com.misc.Unsafe! org.apache.spark.sql.execution.! aggregate.SortBasedAggregate! aggregate.TungstenAggregate! aggregate.AggregationIterator! aggregate.udaf! aggregate.utils! SparkPlanner! rowFormatConverters! UnsafeFixedWidthAggregationMap! UnsafeExternalSorter! UnsafeExternalRowSorter! UnsafeKeyValueSorter! UnsafeKVExternalSorter! local.ConvertToUnsafeNode! local.ConvertToSafeNode! local.HashJoinNode! local.ProjectNode! local.LocalNode! local.BinaryHashJoinNode! local.NestedLoopJoinNode! joins.HashJoin! joins.HashSemiJoin! joins.HashedRelation! joins.BroadcastHashJoin! joins.ShuffledHashOuterJoin (not yet converted)! joins.BroadcastHashOuterJoin! joins.BroadcastLeftSemiJoinHash! joins.BroadcastNestedLoopJoin! joins.SortMergeJoin! joins.LeftSemiJoinBNL! joins.SortMergerOuterJoin! Exchange! SparkPlan! UnsafeRowSerializer! SortPrefixUtils! sort! basicOperators! aggregate.SortBasedAggregationIterator! aggregate.TungstenAggregationIterator! datasources.WriterContainer! datasources.json.JacksonParser! datasources.jdbc.JDBCRDD! Window! org.apache.spark.! unsafe.Platform! unsafe.KVIterator! unsafe.array.LongArray! unsafe.array.ByteArrayMethods! unsafe.array.BitSet! unsafe.bitset.BitSetMethods! unsafe.hash.Murmur3_x86_32! unsafe.map.BytesToBytesMap! unsafe.map.HashMapGrowthStrategy! unsafe.memory.TaskMemoryManager! unsafe.memory.ExecutorMemoryManager! unsafe.memory.MemoryLocation! unsafe.memory.UnsafeMemoryAllocator! unsafe.memory.MemoryAllocator (trait/interface)! unsafe.memory.MemoryBlock! unsafe.memory.HeapMemoryAllocator! unsafe.memory.ExecutorMemoryManager! unsafe.sort.RecordComparator! unsafe.sort.PrefixComparator! unsafe.sort.PrefixComparators! unsafe.sort.UnsafeSorterSpillWriter! serializer.DummySerializationInstance! shuffle.unsafe.UnsafeShuffleManager! shuffle.unsafe.UnsafeShuffleSortDataFormat! shuffle.unsafe.SpillInfo! shuffle.unsafe.UnsafeShuffleWriter! shuffle.unsafe.UnsafeShuffleExternalSorter! shuffle.unsafe.PackedRecordPointer! shuffle.ShuffleMemoryManager! util.collection.unsafe.sort.UnsafeSorterSpillMerger! util.collection.unsafe.sort.UnsafeSorterSpillReader! util.collection.unsafe.sort.UnsafeSorterSpillWriter! util.collection.unsafe.sort.UnsafeShuffleInMemorySorter! util.collection.unsafe.sort.UnsafeInMemorySorter! util.collection.unsafe.sort.RecordPointerAndKeyPrefix! util.collection.unsafe.sort.UnsafeSorterIterator! network.shuffle.ExternalShuffleBlockResolver! scheduler.Task! rdd.SqlNewHadoopRDD! executor.Executor! org.apache.spark.sql.catalyst.expressions.! regexpExpressions! BoundAttribute! SortOrder! SpecializedGetters! ExpressionEvalHelper! UnsafeArrayData! UnsafeReaders! UnsafeMapData! Projection! LiteralGeneartor! UnsafeRow! JoinedRow! SpecializedGetters! InputFileName! SpecificMutableRow! codegen.CodeGenerator! codegen.GenerateProjection! codegen.GenerateUnsafeRowJoiner! codegen.GenerateSafeProjection! codegen.GenerateUnsafeProjection! codegen.BufferHolder! codegen.UnsafeRowWriter! codegen.UnsafeArrayWriter! complexTypeCreator! rows! literals! misc! stringExpressions! Over 200 source! files affected!!!
IBM | spark.tc CPU and Memory Optimizations! Custom Managed Memory Reduces GC overhead Both on and off heap Exact size calculations Direct Binary Processing Operate on serialized/compressed arrays Kryo can reorder serialized records LZF can reorder compressed records More CPU Cache-aware Data Structs & Algorithms o.a.s.unsafe.map.BytesToBytesMap vs. j.u.HashMap Code Generation (default in 1.5) Generate source code from overall query plan Janino generates bytecode from source code 100+ UDFs converted to use code generation Details in ! SPARK-7075! UnsafeFixedWithAggregationMap,& ! TungstenAggregationIterator! CodeGenerator &! GeneratorUnsafeRowJoiner!UnsafeSortDataFormat &! UnsafeShuffleSortDataFormat &! PackedRecordPointer &! UnsafeRow! UnsafeInMemorySorter & UnsafeExternalSorter & UnsafeShuffleWriter! Mostly Same Join Code,! added if (isUnsafeMode)! UnsafeShuffleManager &! UnsafeShuffleInMemorySorter & UnsafeShuffleExternalSorter!
IBM | spark.tc Code Generation! Turned on by default in Spark 1.5 Problem: Generic expression evaluation Expensive on JVM Virtual func calls Branches based on expression type Excessive object creation due to primitive boxing Implementation Defer the source code generation to each operator, type, etc Scala quasiquotes provide Scala AST manipulation/rewriting Generated source code is compiled to bytecode w/ Janino 100+ UDFs now using code gen
IBM | spark.tc Code Generation: Spark SQL UDFs! 100+ UDFs now using code gen – More to come in Spark 1.6! Details in ! SPARK-8159!
IBM | spark.tc Project Tungsten: Beyond Core and Spark SQL! SortDataFormat<K, Buffer>: Base trait ^ implements ^ UncompressedInBlockSort: MLlib.ALS EdgeArraySortDataFormat: GraphX.Edge
IBM | spark.tc Relevant Links!   http://sortbenchmark.org/ApacheSpark2014.pdf! !   https://databricks.com/blog/2014/11/05/spark-officially-sets-a-new-record-in-large-scale-sorting.html!   http://0x0fff.com/spark-architecture-shuffle/!   http://www.cs.berkeley.edu/~kubitron/courses/cs262a-F13/projects/reports/project16_report.pdf!   http://stackoverflow.com/questions/763262/how-does-one-write-code-that-best-utilizes-the-cpu-cach e-to-improve-performance!   http://www.aristeia.com/TalkNotes/ACCU2011_CPUCaches.pdf!   http://mishadoff.com/blog/java-magic-part-4-sun-dot-misc-dot-unsafe/!   http://docs.scala-lang.org/overviews/quasiquotes/intro.html!
IBM | spark.tc More Relevant Links from Scott Meyers!   http://lwn.net/Articles/252125/ <-- Memory Part 2: CPU Caches!   http://lwn.net/Articles/255364/ <-- Memory Part 5: What Programmers Can Do! ScottMatrix * BruceMatrix^T = CaitlynMatrix!
Special thanks to Skimlinks!!! IBM Spark Technology Center is Hiring! " Nice and collaborative people only, please!! IBM | spark.tc Sign up for our newsletter at Thank You, London!!
Special thanks to Skimlinks!!! IBM Spark Technology Center is Hiring! " Nice and collaborative people only, please!! IBM | spark.tc Sign up for our newsletter at Thank You, London!!
Power of data. Simplicity of design. Speed of innovation. IBM Spark
1 of 55

London Spark Meetup Project Tungsten Oct 12 2015

••
Download to read offline
Software

Building on a previous talk about how Spark beat Hadoop @ 100TB Daytona GraySort, we present low-level details of Project Tungsten which includes many CPU and Memory optimizations.

Chris Fregly
Chris FreglyAI and Machine Learning @ AWS, O'Reilly Author @ Data Science on AWS, Founder @ PipelineAI, Formerly Databricks, Netflix,

Recommended

Project Tungsten: Bringing Spark Closer to Bare Metal by
Project Tungsten: Bringing Spark Closer to Bare MetalProject Tungsten: Bringing Spark Closer to Bare Metal
Project Tungsten: Bringing Spark Closer to Bare MetalDatabricks
7.1K views•31 slides
Advanced Apache Spark Meetup: How Spark Beat Hadoop @ 100 TB Daytona GraySor... by
Advanced Apache Spark Meetup: How Spark Beat Hadoop @ 100 TB Daytona GraySor...Advanced Apache Spark Meetup: How Spark Beat Hadoop @ 100 TB Daytona GraySor...
Advanced Apache Spark Meetup: How Spark Beat Hadoop @ 100 TB Daytona GraySor...Chris Fregly
3.7K views•49 slides
From DataFrames to Tungsten: A Peek into Spark's Future @ Spark Summit San Fr... by
From DataFrames to Tungsten: A Peek into Spark's Future @ Spark Summit San Fr...From DataFrames to Tungsten: A Peek into Spark's Future @ Spark Summit San Fr...
From DataFrames to Tungsten: A Peek into Spark's Future @ Spark Summit San Fr...Databricks
9.7K views•27 slides
Deep Dive: Memory Management in Apache Spark by
Deep Dive: Memory Management in Apache SparkDeep Dive: Memory Management in Apache Spark
Deep Dive: Memory Management in Apache SparkDatabricks
14.5K views•54 slides
Deep Dive into Project Tungsten: Bringing Spark Closer to Bare Metal-(Josh Ro... by
Deep Dive into Project Tungsten: Bringing Spark Closer to Bare Metal-(Josh Ro...Deep Dive into Project Tungsten: Bringing Spark Closer to Bare Metal-(Josh Ro...
Deep Dive into Project Tungsten: Bringing Spark Closer to Bare Metal-(Josh Ro...Spark Summit
18.2K views•28 slides
Spark performance tuning - Maksud Ibrahimov by
Spark performance tuning - Maksud IbrahimovSpark performance tuning - Maksud Ibrahimov
Spark performance tuning - Maksud IbrahimovMaksud Ibrahimov
1.8K views•26 slides

More Related Content

What's hot

Frustration-Reduced PySpark: Data engineering with DataFrames by
Frustration-Reduced PySpark: Data engineering with DataFramesFrustration-Reduced PySpark: Data engineering with DataFrames
Frustration-Reduced PySpark: Data engineering with DataFramesIlya Ganelin
9.6K views•29 slides
Top 5 mistakes when writing Spark applications by
Top 5 mistakes when writing Spark applicationsTop 5 mistakes when writing Spark applications
Top 5 mistakes when writing Spark applicationshadooparchbook
11.3K views•75 slides
PySpark Best Practices by
PySpark Best PracticesPySpark Best Practices
PySpark Best PracticesCloudera, Inc.
9.4K views•34 slides
A Spark Framework For &lt; $100, &lt; 1 Hour, Accurate Personalized DNA Analy... by
A Spark Framework For &lt; $100, &lt; 1 Hour, Accurate Personalized DNA Analy...A Spark Framework For &lt; $100, &lt; 1 Hour, Accurate Personalized DNA Analy...
A Spark Framework For &lt; $100, &lt; 1 Hour, Accurate Personalized DNA Analy...Spark Summit
1.2K views•22 slides
PySpark in practice slides by
PySpark in practice slidesPySpark in practice slides
PySpark in practice slidesDat Tran
3K views•39 slides
Koalas: Making an Easy Transition from Pandas to Apache Spark by
Koalas: Making an Easy Transition from Pandas to Apache SparkKoalas: Making an Easy Transition from Pandas to Apache Spark
Koalas: Making an Easy Transition from Pandas to Apache SparkDatabricks
1.8K views•33 slides

What's hot(20)

Frustration-Reduced PySpark: Data engineering with DataFrames by Ilya Ganelin
Frustration-Reduced PySpark: Data engineering with DataFramesFrustration-Reduced PySpark: Data engineering with DataFrames
Frustration-Reduced PySpark: Data engineering with DataFrames
Ilya Ganelin•9.6K views
Top 5 mistakes when writing Spark applications by hadooparchbook
Top 5 mistakes when writing Spark applicationsTop 5 mistakes when writing Spark applications
Top 5 mistakes when writing Spark applications
hadooparchbook•11.3K views
PySpark Best Practices by Cloudera, Inc.
PySpark Best PracticesPySpark Best Practices
PySpark Best Practices
Cloudera, Inc.•9.4K views
A Spark Framework For &lt; $100, &lt; 1 Hour, Accurate Personalized DNA Analy... by Spark Summit
A Spark Framework For &lt; $100, &lt; 1 Hour, Accurate Personalized DNA Analy...A Spark Framework For &lt; $100, &lt; 1 Hour, Accurate Personalized DNA Analy...
A Spark Framework For &lt; $100, &lt; 1 Hour, Accurate Personalized DNA Analy...
Spark Summit•1.2K views
PySpark in practice slides by Dat Tran
PySpark in practice slidesPySpark in practice slides
PySpark in practice slides
Dat Tran•3K views
Koalas: Making an Easy Transition from Pandas to Apache Spark by Databricks
Koalas: Making an Easy Transition from Pandas to Apache SparkKoalas: Making an Easy Transition from Pandas to Apache Spark
Koalas: Making an Easy Transition from Pandas to Apache Spark
Databricks•1.8K views
A Developer’s View into Spark's Memory Model with Wenchen Fan by Databricks
A Developer’s View into Spark's Memory Model with Wenchen FanA Developer’s View into Spark's Memory Model with Wenchen Fan
A Developer’s View into Spark's Memory Model with Wenchen Fan
Databricks•1.5K views
Spark Summit EU talk by Nimbus Goehausen by Spark Summit
Spark Summit EU talk by Nimbus GoehausenSpark Summit EU talk by Nimbus Goehausen
Spark Summit EU talk by Nimbus Goehausen
Spark Summit•1.1K views
Parquet performance tuning: the missing guide by Ryan Blue
Parquet performance tuning: the missing guideParquet performance tuning: the missing guide
Parquet performance tuning: the missing guide
Ryan Blue•40.6K views
Python and Bigdata - An Introduction to Spark (PySpark) by hiteshnd
Python and Bigdata - An Introduction to Spark (PySpark)Python and Bigdata - An Introduction to Spark (PySpark)
Python and Bigdata - An Introduction to Spark (PySpark)
hiteshnd•6.2K views
Up and running with pyspark by Krishna Sangeeth KS
Up and running with pysparkUp and running with pyspark
Up and running with pyspark
Krishna Sangeeth KS•1.3K views
Spark tutorial by Sahan Bulathwela
Spark tutorialSpark tutorial
Spark tutorial
Sahan Bulathwela•1.4K views
Madrid Spark Big Data Bluemix Meetup - Spark Versus Hadoop @ 100 TB Daytona G... by Chris Fregly
Madrid Spark Big Data Bluemix Meetup - Spark Versus Hadoop @ 100 TB Daytona G...Madrid Spark Big Data Bluemix Meetup - Spark Versus Hadoop @ 100 TB Daytona G...
Madrid Spark Big Data Bluemix Meetup - Spark Versus Hadoop @ 100 TB Daytona G...
Chris Fregly•1.2K views
PySaprk by Giivee The
PySaprkPySaprk
PySaprk
Giivee The•5.6K views
Top 5 mistakes when writing Spark applications by hadooparchbook
Top 5 mistakes when writing Spark applicationsTop 5 mistakes when writing Spark applications
Top 5 mistakes when writing Spark applications
hadooparchbook•14.6K views
Strata NYC 2015: Sketching Big Data with Spark: randomized algorithms for lar... by Databricks
Strata NYC 2015: Sketching Big Data with Spark: randomized algorithms for lar...Strata NYC 2015: Sketching Big Data with Spark: randomized algorithms for lar...
Strata NYC 2015: Sketching Big Data with Spark: randomized algorithms for lar...
Databricks•9.8K views
A real-time architecture using Hadoop & Storm - Nathan Bijnens & Geert Van La... by jaxLondonConference
A real-time architecture using Hadoop & Storm - Nathan Bijnens & Geert Van La...A real-time architecture using Hadoop & Storm - Nathan Bijnens & Geert Van La...
A real-time architecture using Hadoop & Storm - Nathan Bijnens & Geert Van La...
jaxLondonConference•1.7K views
Debugging PySpark: Spark Summit East talk by Holden Karau by Spark Summit
Debugging PySpark: Spark Summit East talk by Holden KarauDebugging PySpark: Spark Summit East talk by Holden Karau
Debugging PySpark: Spark Summit East talk by Holden Karau
Spark Summit•6.5K views
Performant data processing with PySpark, SparkR and DataFrame API by Ryuji Tamagawa
Performant data processing with PySpark, SparkR and DataFrame APIPerformant data processing with PySpark, SparkR and DataFrame API
Performant data processing with PySpark, SparkR and DataFrame API
Ryuji Tamagawa•3.4K views
Extreme Apache Spark: how in 3 months we created a pipeline that can process ... by Josef A. Habdank
Extreme Apache Spark: how in 3 months we created a pipeline that can process ...Extreme Apache Spark: how in 3 months we created a pipeline that can process ...
Extreme Apache Spark: how in 3 months we created a pipeline that can process ...
Josef A. Habdank•25.4K views

Similar to London Spark Meetup Project Tungsten Oct 12 2015

Scotland Data Science Meetup Oct 13, 2015: Spark SQL, DataFrames, Catalyst, ... by
Scotland Data Science Meetup Oct 13, 2015: Spark SQL, DataFrames, Catalyst, ...Scotland Data Science Meetup Oct 13, 2015: Spark SQL, DataFrames, Catalyst, ...
Scotland Data Science Meetup Oct 13, 2015: Spark SQL, DataFrames, Catalyst, ...Chris Fregly
2K views•56 slides
Advanced Apache Spark Meetup Data Sources API Cassandra Spark Connector Spark... by
Advanced Apache Spark Meetup Data Sources API Cassandra Spark Connector Spark...Advanced Apache Spark Meetup Data Sources API Cassandra Spark Connector Spark...
Advanced Apache Spark Meetup Data Sources API Cassandra Spark Connector Spark...Chris Fregly
2.2K views•55 slides
The Smug Mug Tale by
The Smug Mug TaleThe Smug Mug Tale
The Smug Mug TaleMySQLConference
1.3K views•33 slides
Project Tungsten Phase II: Joining a Billion Rows per Second on a Laptop by
Project Tungsten Phase II: Joining a Billion Rows per Second on a LaptopProject Tungsten Phase II: Joining a Billion Rows per Second on a Laptop
Project Tungsten Phase II: Joining a Billion Rows per Second on a LaptopDatabricks
2.3K views•36 slides
Paris Spark Meetup Oct 26, 2015 - Spark After Dark v1.5 - Best of Advanced Ap... by
Paris Spark Meetup Oct 26, 2015 - Spark After Dark v1.5 - Best of Advanced Ap...Paris Spark Meetup Oct 26, 2015 - Spark After Dark v1.5 - Best of Advanced Ap...
Paris Spark Meetup Oct 26, 2015 - Spark After Dark v1.5 - Best of Advanced Ap...Chris Fregly
2.7K views•118 slides
Osd ctw spark by
Osd ctw sparkOsd ctw spark
Osd ctw sparkWisely chen
2.2K views•53 slides

Similar to London Spark Meetup Project Tungsten Oct 12 2015(20)

Scotland Data Science Meetup Oct 13, 2015: Spark SQL, DataFrames, Catalyst, ... by Chris Fregly
Scotland Data Science Meetup Oct 13, 2015: Spark SQL, DataFrames, Catalyst, ...Scotland Data Science Meetup Oct 13, 2015: Spark SQL, DataFrames, Catalyst, ...
Scotland Data Science Meetup Oct 13, 2015: Spark SQL, DataFrames, Catalyst, ...
Chris Fregly•2K views
Advanced Apache Spark Meetup Data Sources API Cassandra Spark Connector Spark... by Chris Fregly
Advanced Apache Spark Meetup Data Sources API Cassandra Spark Connector Spark...Advanced Apache Spark Meetup Data Sources API Cassandra Spark Connector Spark...
Advanced Apache Spark Meetup Data Sources API Cassandra Spark Connector Spark...
Chris Fregly•2.2K views
The Smug Mug Tale by MySQLConference
The Smug Mug TaleThe Smug Mug Tale
The Smug Mug Tale
MySQLConference•1.3K views
Project Tungsten Phase II: Joining a Billion Rows per Second on a Laptop by Databricks
Project Tungsten Phase II: Joining a Billion Rows per Second on a LaptopProject Tungsten Phase II: Joining a Billion Rows per Second on a Laptop
Project Tungsten Phase II: Joining a Billion Rows per Second on a Laptop
Databricks•2.3K views
Paris Spark Meetup Oct 26, 2015 - Spark After Dark v1.5 - Best of Advanced Ap... by Chris Fregly
Paris Spark Meetup Oct 26, 2015 - Spark After Dark v1.5 - Best of Advanced Ap...Paris Spark Meetup Oct 26, 2015 - Spark After Dark v1.5 - Best of Advanced Ap...
Paris Spark Meetup Oct 26, 2015 - Spark After Dark v1.5 - Best of Advanced Ap...
Chris Fregly•2.7K views
Osd ctw spark by Wisely chen
Osd ctw sparkOsd ctw spark
Osd ctw spark
Wisely chen•2.2K views
Stockholm Spark Meetup Nov 23 2015 Spark After Dark 1.5 by Chris Fregly
Stockholm Spark Meetup Nov 23 2015 Spark After Dark 1.5Stockholm Spark Meetup Nov 23 2015 Spark After Dark 1.5
Stockholm Spark Meetup Nov 23 2015 Spark After Dark 1.5
Chris Fregly•665 views
introduction to data processing using Hadoop and Pig by Ricardo Varela
introduction to data processing using Hadoop and Pigintroduction to data processing using Hadoop and Pig
introduction to data processing using Hadoop and Pig
Ricardo Varela•92.5K views
Introduction to Spark by Li Ming Tsai
Introduction to SparkIntroduction to Spark
Introduction to Spark
Li Ming Tsai•2K views
Apache Spark Core—Deep Dive—Proper Optimization by Databricks
Apache Spark Core—Deep Dive—Proper OptimizationApache Spark Core—Deep Dive—Proper Optimization
Apache Spark Core—Deep Dive—Proper Optimization
Databricks•6.1K views
Advanced Apache Spark Meetup Approximations and Probabilistic Data Structures... by Chris Fregly
Advanced Apache Spark Meetup Approximations and Probabilistic Data Structures...Advanced Apache Spark Meetup Approximations and Probabilistic Data Structures...
Advanced Apache Spark Meetup Approximations and Probabilistic Data Structures...
Chris Fregly•1.6K views
Singapore Spark Meetup Dec 01 2015 by Chris Fregly
Singapore Spark Meetup Dec 01 2015Singapore Spark Meetup Dec 01 2015
Singapore Spark Meetup Dec 01 2015
Chris Fregly•1.1K views
Toronto Spark Meetup Dec 14 2015 by Chris Fregly
Toronto Spark Meetup Dec 14 2015Toronto Spark Meetup Dec 14 2015
Toronto Spark Meetup Dec 14 2015
Chris Fregly•1.3K views
Spark Summit EU 2015: Lessons from 300+ production users by Databricks
Spark Summit EU 2015: Lessons from 300+ production usersSpark Summit EU 2015: Lessons from 300+ production users
Spark Summit EU 2015: Lessons from 300+ production users
Databricks•10.5K views
Spark after Dark by Chris Fregly of Databricks by Data Con LA
Spark after Dark by Chris Fregly of DatabricksSpark after Dark by Chris Fregly of Databricks
Spark after Dark by Chris Fregly of Databricks
Data Con LA•4K views
Spark After Dark - LA Apache Spark Users Group - Feb 2015 by Chris Fregly
Spark After Dark - LA Apache Spark Users Group - Feb 2015Spark After Dark - LA Apache Spark Users Group - Feb 2015
Spark After Dark - LA Apache Spark Users Group - Feb 2015
Chris Fregly•5.1K views
Advanced Apache Spark Meetup Project Tungsten Nov 12 2015 by Chris Fregly
Advanced Apache Spark Meetup Project Tungsten Nov 12 2015Advanced Apache Spark Meetup Project Tungsten Nov 12 2015
Advanced Apache Spark Meetup Project Tungsten Nov 12 2015
Chris Fregly•12.1K views
OCF.tw's talk about "Introduction to spark" by Giivee The
OCF.tw's talk about "Introduction to spark"OCF.tw's talk about "Introduction to spark"
OCF.tw's talk about "Introduction to spark"
Giivee The•2.3K views
Top 5 Mistakes When Writing Spark Applications by Spark Summit
Top 5 Mistakes When Writing Spark ApplicationsTop 5 Mistakes When Writing Spark Applications
Top 5 Mistakes When Writing Spark Applications
Spark Summit•26.4K views
Brussels Spark Meetup Oct 30, 2015: Spark After Dark 1.5:  Real-time, Advanc... by Chris Fregly
Brussels Spark Meetup Oct 30, 2015: Spark After Dark 1.5:  Real-time, Advanc...Brussels Spark Meetup Oct 30, 2015: Spark After Dark 1.5:  Real-time, Advanc...
Brussels Spark Meetup Oct 30, 2015: Spark After Dark 1.5:  Real-time, Advanc...
Chris Fregly•793 views

More from Chris Fregly

AWS reInvent 2022 reCap AI/ML and Data by
AWS reInvent 2022 reCap AI/ML and DataAWS reInvent 2022 reCap AI/ML and Data
AWS reInvent 2022 reCap AI/ML and DataChris Fregly
346 views•79 slides
Pandas on AWS - Let me count the ways.pdf by
Pandas on AWS - Let me count the ways.pdfPandas on AWS - Let me count the ways.pdf
Pandas on AWS - Let me count the ways.pdfChris Fregly
191 views•32 slides
Smokey and the Multi-Armed Bandit Featuring BERT Reynolds Updated by
Smokey and the Multi-Armed Bandit Featuring BERT Reynolds UpdatedSmokey and the Multi-Armed Bandit Featuring BERT Reynolds Updated
Smokey and the Multi-Armed Bandit Featuring BERT Reynolds UpdatedChris Fregly
1.9K views•15 slides
Amazon reInvent 2020 Recap: AI and Machine Learning by
Amazon reInvent 2020 Recap: AI and Machine LearningAmazon reInvent 2020 Recap: AI and Machine Learning
Amazon reInvent 2020 Recap: AI and Machine LearningChris Fregly
1.2K views•25 slides
Waking the Data Scientist at 2am: Detect Model Degradation on Production Mod... by
Waking the Data Scientist at 2am: Detect Model Degradation on Production Mod...Waking the Data Scientist at 2am: Detect Model Degradation on Production Mod...
Waking the Data Scientist at 2am: Detect Model Degradation on Production Mod...Chris Fregly
900 views•39 slides
Quantum Computing with Amazon Braket by
Quantum Computing with Amazon BraketQuantum Computing with Amazon Braket
Quantum Computing with Amazon BraketChris Fregly
1K views•35 slides

More from Chris Fregly(20)

AWS reInvent 2022 reCap AI/ML and Data by Chris Fregly
AWS reInvent 2022 reCap AI/ML and DataAWS reInvent 2022 reCap AI/ML and Data
AWS reInvent 2022 reCap AI/ML and Data
Chris Fregly•346 views
Pandas on AWS - Let me count the ways.pdf by Chris Fregly
Pandas on AWS - Let me count the ways.pdfPandas on AWS - Let me count the ways.pdf
Pandas on AWS - Let me count the ways.pdf
Chris Fregly•191 views
Smokey and the Multi-Armed Bandit Featuring BERT Reynolds Updated by Chris Fregly
Smokey and the Multi-Armed Bandit Featuring BERT Reynolds UpdatedSmokey and the Multi-Armed Bandit Featuring BERT Reynolds Updated
Smokey and the Multi-Armed Bandit Featuring BERT Reynolds Updated
Chris Fregly•1.9K views
Amazon reInvent 2020 Recap: AI and Machine Learning by Chris Fregly
Amazon reInvent 2020 Recap: AI and Machine LearningAmazon reInvent 2020 Recap: AI and Machine Learning
Amazon reInvent 2020 Recap: AI and Machine Learning
Chris Fregly•1.2K views
Waking the Data Scientist at 2am: Detect Model Degradation on Production Mod... by Chris Fregly
Waking the Data Scientist at 2am: Detect Model Degradation on Production Mod...Waking the Data Scientist at 2am: Detect Model Degradation on Production Mod...
Waking the Data Scientist at 2am: Detect Model Degradation on Production Mod...
Chris Fregly•900 views
Quantum Computing with Amazon Braket by Chris Fregly
Quantum Computing with Amazon BraketQuantum Computing with Amazon Braket
Quantum Computing with Amazon Braket
Chris Fregly•1K views
15 Tips to Scale a Large AI/ML Workshop - Both Online and In-Person by Chris Fregly
15 Tips to Scale a Large AI/ML Workshop - Both Online and In-Person15 Tips to Scale a Large AI/ML Workshop - Both Online and In-Person
15 Tips to Scale a Large AI/ML Workshop - Both Online and In-Person
Chris Fregly•2.6K views
AWS Re:Invent 2019 Re:Cap by Chris Fregly
AWS Re:Invent 2019 Re:CapAWS Re:Invent 2019 Re:Cap
AWS Re:Invent 2019 Re:Cap
Chris Fregly•2.1K views
KubeFlow + GPU + Keras/TensorFlow 2.0 + TF Extended (TFX) + Kubernetes + PyTo... by Chris Fregly
KubeFlow + GPU + Keras/TensorFlow 2.0 + TF Extended (TFX) + Kubernetes + PyTo...KubeFlow + GPU + Keras/TensorFlow 2.0 + TF Extended (TFX) + Kubernetes + PyTo...
KubeFlow + GPU + Keras/TensorFlow 2.0 + TF Extended (TFX) + Kubernetes + PyTo...
Chris Fregly•3.9K views
Swift for TensorFlow - Tanmay Bakshi - Advanced Spark and TensorFlow Meetup -... by Chris Fregly
Swift for TensorFlow - Tanmay Bakshi - Advanced Spark and TensorFlow Meetup -...Swift for TensorFlow - Tanmay Bakshi - Advanced Spark and TensorFlow Meetup -...
Swift for TensorFlow - Tanmay Bakshi - Advanced Spark and TensorFlow Meetup -...
Chris Fregly•1.2K views
Hands-on Learning with KubeFlow + Keras/TensorFlow 2.0 + TF Extended (TFX) + ... by Chris Fregly
Hands-on Learning with KubeFlow + Keras/TensorFlow 2.0 + TF Extended (TFX) + ...Hands-on Learning with KubeFlow + Keras/TensorFlow 2.0 + TF Extended (TFX) + ...
Hands-on Learning with KubeFlow + Keras/TensorFlow 2.0 + TF Extended (TFX) + ...
Chris Fregly•3.7K views
Spark SQL Catalyst Optimizer, Custom Expressions, UDFs - Advanced Spark and T... by Chris Fregly
Spark SQL Catalyst Optimizer, Custom Expressions, UDFs - Advanced Spark and T...Spark SQL Catalyst Optimizer, Custom Expressions, UDFs - Advanced Spark and T...
Spark SQL Catalyst Optimizer, Custom Expressions, UDFs - Advanced Spark and T...
Chris Fregly•597 views
PipelineAI Continuous Machine Learning and AI - Rework Deep Learning Summit -... by Chris Fregly
PipelineAI Continuous Machine Learning and AI - Rework Deep Learning Summit -...PipelineAI Continuous Machine Learning and AI - Rework Deep Learning Summit -...
PipelineAI Continuous Machine Learning and AI - Rework Deep Learning Summit -...
Chris Fregly•1.1K views
PipelineAI Real-Time Machine Learning - Global Artificial Intelligence Confer... by Chris Fregly
PipelineAI Real-Time Machine Learning - Global Artificial Intelligence Confer...PipelineAI Real-Time Machine Learning - Global Artificial Intelligence Confer...
PipelineAI Real-Time Machine Learning - Global Artificial Intelligence Confer...
Chris Fregly•607 views
Hyper-Parameter Tuning Across the Entire AI Pipeline GPU Tech Conference San ... by Chris Fregly
Hyper-Parameter Tuning Across the Entire AI Pipeline GPU Tech Conference San ...Hyper-Parameter Tuning Across the Entire AI Pipeline GPU Tech Conference San ...
Hyper-Parameter Tuning Across the Entire AI Pipeline GPU Tech Conference San ...
Chris Fregly•5.3K views
PipelineAI Optimizes Your Enterprise AI Pipeline from Distributed Training to... by Chris Fregly
PipelineAI Optimizes Your Enterprise AI Pipeline from Distributed Training to...PipelineAI Optimizes Your Enterprise AI Pipeline from Distributed Training to...
PipelineAI Optimizes Your Enterprise AI Pipeline from Distributed Training to...
Chris Fregly•2.5K views
Advanced Spark and TensorFlow Meetup - Dec 12 2017 - Dong Meng, MapR + Kubern... by Chris Fregly
Advanced Spark and TensorFlow Meetup - Dec 12 2017 - Dong Meng, MapR + Kubern...Advanced Spark and TensorFlow Meetup - Dec 12 2017 - Dong Meng, MapR + Kubern...
Advanced Spark and TensorFlow Meetup - Dec 12 2017 - Dong Meng, MapR + Kubern...
Chris Fregly•963 views
High Performance Distributed TensorFlow in Production with GPUs - NIPS 2017 -... by Chris Fregly
High Performance Distributed TensorFlow in Production with GPUs - NIPS 2017 -...High Performance Distributed TensorFlow in Production with GPUs - NIPS 2017 -...
High Performance Distributed TensorFlow in Production with GPUs - NIPS 2017 -...
Chris Fregly•3.9K views
PipelineAI + TensorFlow AI + Spark ML + Kuberenetes + Istio + AWS SageMaker +... by Chris Fregly
PipelineAI + TensorFlow AI + Spark ML + Kuberenetes + Istio + AWS SageMaker +...PipelineAI + TensorFlow AI + Spark ML + Kuberenetes + Istio + AWS SageMaker +...
PipelineAI + TensorFlow AI + Spark ML + Kuberenetes + Istio + AWS SageMaker +...
Chris Fregly•1.4K views
PipelineAI + AWS SageMaker + Distributed TensorFlow + AI Model Training and S... by Chris Fregly
PipelineAI + AWS SageMaker + Distributed TensorFlow + AI Model Training and S...PipelineAI + AWS SageMaker + Distributed TensorFlow + AI Model Training and S...
PipelineAI + AWS SageMaker + Distributed TensorFlow + AI Model Training and S...
Chris Fregly•2.5K views

Recently uploaded

Understanding HTML terminology by
Understanding HTML terminologyUnderstanding HTML terminology
Understanding HTML terminologyartembondar5
7 views•8 slides
Introduction to Git Source Control by
Introduction to Git Source ControlIntroduction to Git Source Control
Introduction to Git Source ControlJohn Valentino
7 views•18 slides
Generic or specific? Making sensible software design decisions by
Generic or specific? Making sensible software design decisionsGeneric or specific? Making sensible software design decisions
Generic or specific? Making sensible software design decisionsBert Jan Schrijver
7 views•60 slides
Benefits in Software Development by
Benefits in Software DevelopmentBenefits in Software Development
Benefits in Software DevelopmentJohn Valentino
5 views•15 slides
Top-5-production-devconMunich-2023-v2.pptx by
Top-5-production-devconMunich-2023-v2.pptxTop-5-production-devconMunich-2023-v2.pptx
Top-5-production-devconMunich-2023-v2.pptxTier1 app
8 views•42 slides
Dev-HRE-Ops - Addressing the _Last Mile DevOps Challenge_ in Highly Regulated... by
Dev-HRE-Ops - Addressing the _Last Mile DevOps Challenge_ in Highly Regulated...Dev-HRE-Ops - Addressing the _Last Mile DevOps Challenge_ in Highly Regulated...
Dev-HRE-Ops - Addressing the _Last Mile DevOps Challenge_ in Highly Regulated...TomHalpin9
6 views•29 slides

Recently uploaded(20)

Understanding HTML terminology by artembondar5
Understanding HTML terminologyUnderstanding HTML terminology
Understanding HTML terminology
artembondar5•7 views
Introduction to Git Source Control by John Valentino
Introduction to Git Source ControlIntroduction to Git Source Control
Introduction to Git Source Control
John Valentino•7 views
Generic or specific? Making sensible software design decisions by Bert Jan Schrijver
Generic or specific? Making sensible software design decisionsGeneric or specific? Making sensible software design decisions
Generic or specific? Making sensible software design decisions
Bert Jan Schrijver•7 views
Benefits in Software Development by John Valentino
Benefits in Software DevelopmentBenefits in Software Development
Benefits in Software Development
John Valentino•5 views
Top-5-production-devconMunich-2023-v2.pptx by Tier1 app
Top-5-production-devconMunich-2023-v2.pptxTop-5-production-devconMunich-2023-v2.pptx
Top-5-production-devconMunich-2023-v2.pptx
Tier1 app•8 views
Dev-HRE-Ops - Addressing the _Last Mile DevOps Challenge_ in Highly Regulated... by TomHalpin9
Dev-HRE-Ops - Addressing the _Last Mile DevOps Challenge_ in Highly Regulated...Dev-HRE-Ops - Addressing the _Last Mile DevOps Challenge_ in Highly Regulated...
Dev-HRE-Ops - Addressing the _Last Mile DevOps Challenge_ in Highly Regulated...
TomHalpin9•6 views
Programming Field by thehardtechnology
Programming FieldProgramming Field
Programming Field
thehardtechnology•6 views
Automated Testing of Microsoft Power BI Reports by RTTS
Automated Testing of Microsoft Power BI ReportsAutomated Testing of Microsoft Power BI Reports
Automated Testing of Microsoft Power BI Reports
RTTS•10 views
360 graden fabriek by info33492
360 graden fabriek360 graden fabriek
360 graden fabriek
info33492•165 views
Unlocking the Power of AI in Product Management - A Comprehensive Guide for P... by NimaTorabi2
Unlocking the Power of AI in Product Management - A Comprehensive Guide for P...Unlocking the Power of AI in Product Management - A Comprehensive Guide for P...
Unlocking the Power of AI in Product Management - A Comprehensive Guide for P...
NimaTorabi2•16 views
The Era of Large Language Models.pptx by AbdulVahedShaik
The Era of Large Language Models.pptxThe Era of Large Language Models.pptx
The Era of Large Language Models.pptx
AbdulVahedShaik•7 views
Dapr Unleashed: Accelerating Microservice Development by Miroslav Janeski
Dapr Unleashed: Accelerating Microservice DevelopmentDapr Unleashed: Accelerating Microservice Development
Dapr Unleashed: Accelerating Microservice Development
Miroslav Janeski•15 views
predicting-m3-devopsconMunich-2023-v2.pptx by Tier1 app
predicting-m3-devopsconMunich-2023-v2.pptxpredicting-m3-devopsconMunich-2023-v2.pptx
predicting-m3-devopsconMunich-2023-v2.pptx
Tier1 app•12 views
Using Qt under LGPL-3.0 by Burkhard Stubert
Using Qt under LGPL-3.0Using Qt under LGPL-3.0
Using Qt under LGPL-3.0
Burkhard Stubert•13 views
Quality Engineer: A Day in the Life by John Valentino
Quality Engineer: A Day in the LifeQuality Engineer: A Day in the Life
Quality Engineer: A Day in the Life
John Valentino•7 views
Ports-and-Adapters Architecture for Embedded HMI by Burkhard Stubert
Ports-and-Adapters Architecture for Embedded HMIPorts-and-Adapters Architecture for Embedded HMI
Ports-and-Adapters Architecture for Embedded HMI
Burkhard Stubert•33 views
Top-5-production-devconMunich-2023.pptx by Tier1 app
Top-5-production-devconMunich-2023.pptxTop-5-production-devconMunich-2023.pptx
Top-5-production-devconMunich-2023.pptx
Tier1 app•9 views
ADDO_2022_CICID_Tom_Halpin.pdf by TomHalpin9
ADDO_2022_CICID_Tom_Halpin.pdfADDO_2022_CICID_Tom_Halpin.pdf
ADDO_2022_CICID_Tom_Halpin.pdf
TomHalpin9•5 views
Bootstrapping vs Venture Capital.pptx by Zeljko Svedic
Bootstrapping vs Venture Capital.pptxBootstrapping vs Venture Capital.pptx
Bootstrapping vs Venture Capital.pptx
Zeljko Svedic•15 views
Navigating container technology for enhanced security by Niklas Saari by Metosin Oy
Navigating container technology for enhanced security by Niklas SaariNavigating container technology for enhanced security by Niklas Saari
Navigating container technology for enhanced security by Niklas Saari
Metosin Oy•15 views

London Spark Meetup Project Tungsten Oct 12 2015

  • 1. How Spark Beat Hadoop @ 100 TB Sort + Project Tungsten London Spark Meetup Chris Fregly, Principal Data Solutions Engineer IBM Spark Technology Center Oct 12, 2015 Power of data. Simplicity of design. Speed of innovation. IBM | spark.tc
  • 2. IBM | spark.tc Announcements Skimlinks! ! Martin Goodson! Organizer, London Spark Meetup! !
  • 3. IBM | spark.tc Who am I?! ! Streaming Data Engineer! Netflix Open Source Committer! ! Data Solutions Engineer! Apache Contributor! ! Principal Data Solutions Engineer! IBM Technology Center! Meetup Organizer! Advanced Apache Meetup! Book Author! Advanced Spark (2016)!
  • 4. IBM | spark.tc Advanced Apache Spark Meetup Total Spark Experts: 1200+ in only 3 mos!! #5 most active Spark Meetup in the world!! ! Goals! Dig deep into the Spark & extended-Spark codebase! ! Study integrations such as Cassandra, ElasticSearch,! Tachyon, S3, BlinkDB, Mesos, YARN, Kafka, R, etc! ! Surface and share the patterns and idioms of these ! well-designed, distributed, big data components!
  • 5. IBM | spark.tc Recent Events Cassandra Summit 2015! Real-time Advanced Analytics w/ Spark & Cassandra! ! ! ! ! ! Strata NYC 2015! Practical Data Science w/ Spark: Recommender Systems! ! Available on Slideshare! http://slideshare.net/cfregly! !
  • 6. IBM | spark.tc Freg-a-palooza Upcoming World Tour   London Spark Meetup (Oct 12th)!   Scotland Data Science Meetup (Oct 13th)!   Dublin Spark Meetup (Oct 15th)!   Barcelona Spark Meetup (Oct 20th)!   Madrid Spark/Big Data Meetup (Oct 22nd)!   Paris Spark Meetup (Oct 26th)!   Amsterdam Spark Summit (Oct 27th – Oct 29th)!   Delft Dutch Data Science Meetup (Oct 29th) !   Brussels Spark Meetup (Oct 30th)!   Zurich Big Data Developers Meetup (Nov 2nd)! High probability! I’ll end up in jail! or married!!
  • 8. IBM | spark.tc Topics of this Talk: Mechanical Sympathy! Tungsten => Bare Metal! Seek Once, Scan Sequentially!! CPU Cache Locality and Efficiency! Use Data Structs Customized to Your Workload! Go Off-Heap Whenever Possible ! spark.unsafe.offHeap=true!
  • 9. IBM | spark.tc What is the Daytona GraySort Challenge?! Key Metric! Throughput of sorting 100TB of 100 byte data,10 byte key! Total time includes launching app and writing output file! ! Daytona! App must be general purpose! ! Gray! Named after Jim Gray!
  • 10. IBM | spark.tc Daytona GraySort Challenge: Input and Resources! Input! Records are 100 bytes in length! First 10 bytes are random key! Input generator: ordinal.com/gensort.html! 28,000 fixed-size partitions for 100 TB sort! 250,000 fixed-size partitions for 1 PB sort! 1 partition = 1 HDFS block = 1 executor ! Aligned to avoid partial read I/O ie. imaginary data ^----^! Hardware and Runtime Resources! Commercially available and off-the-shelf! Unmodified, no over/under-clocking! Generates 500TB of disk I/O, 200TB network I/O! 1st record of! 1st 10 bytes:! “JimGrayRIP”!
  • 11. IBM | spark.tc Daytona GraySort Challenge: Rules! Must sort to/from OS files in secondary storage! ! No raw disk since I/O subsystem is being tested! ! File and device striping (RAID 0) are encouraged! ! Output file(s) must have correct key order!
  • 12. IBM | spark.tc Daytona GraySort Challenge: Task Scheduling! Types of Data Locality! PROCESS_LOCAL! NODE_LOCAL! RACK_LOCAL! ANY! ! Delay Scheduling! spark.locality.wait.node: time to wait for next shitty level! Set to infinite to reduce shittiness, force NODE_LOCAL! Straggling Executor JVMs naturally fade away on each run! Decreasing! Level of! Read ! Performance!
  • 13. IBM | spark.tc Daytona GraySort Challenge: Winning Results! On-disk only, in-memory caching disabled!! EC2 (i2.8xlarge)! EC2 (i2.8xlarge)! 28,000! partitions! 250,000 ! partitions (!!)! (3 GBps/node! * 206 nodes)!
  • 14. IBM | spark.tc Daytona GraySort Challenge: EC2 Configuration! 206 EC2 Worker nodes, 1 Master node! AWS i2.8xlarge! 32 Intel Xeon CPU E5-2670 @ 2.5 Ghz! 244 GB RAM, 8 x 800GB SSD, RAID 0 striping, ext4! NOOP I/O scheduler: FIFO, request merging, no reordering! 3 GBps mixed read/write disk I/O per node! Deployed within Placement Group/VPC! Enhanced Networking! Single Root I/O Virtualization (SR-IOV): extension of PCIe! 10 Gbps, low latency, low jitter (iperf showed ~9.5 Gbps)!
  • 15. IBM | spark.tc Daytona GraySort Challenge: Winning Configuration! Spark 1.2, OpenJDK 1.7_<amazon-something>_u65-b17! Disabled in-memory caching -- all on-disk!! HDFS 2.4.1 short-circuit local reads, 2x replication! Writes flushed after each of the 5 runs! 28,000 partitions / (206 nodes * 32 cores) = 4.25 runs, round up 5 runs! Netty 4.0.23.Final with native epoll! Speculative Execution disabled: spark.speculation=false! Force NODE_LOCAL: spark.locality.wait.node=Infinite ! Force Netty Off-Heap: spark.shuffle.io.preferDirectBuffers! Spilling disabled: spark.shuffle.spill=false! All compression disabled (network, on-disk, etc)!
  • 16. IBM | spark.tc Daytona GraySort Challenge: Partitioning! Range Partitioning (vs. Hash Partitioning)! Take advantage of sequential key space! Similar keys grouped together within a partition! Ranges defined by sampling 79 values per partition! Driver sorts samples and defines range boundaries! Sampling took ~10 seconds for 28,000 partitions! !
  • 17. IBM | spark.tc Daytona GraySort Challenge: Why Bother?! Sorting relies heavily on shuffle, I/O subsystem! ! Shuffle is major bottleneck in big data processing! Large number of partitions can exhaust OS resources! ! Shuffle optimization benefits all high-level libraries! ! Goal is to saturate network controller on all nodes! ~125 MB/s (1 GB ethernet), 1.25 GB/s (10 GB ethernet)!
  • 18. IBM | spark.tc Daytona GraySort Challenge: Per Node Results! ! ! ! ! ! Reducers: ~1.1 GB/s/node network I/O! (max 1.25 Gbps for 10 GB ethernet)! Mappers: 3 GB/s/node disk I/O (8x800 SSD)! 206 nodes ! * ! 1.1 Gbps/node ! ~=! 220 Gbps !
  • 20. ! ! ! ! ! ! ! ! ! ! ! ! IBM | spark.tc Shuffle Overview! All to All, Cartesian Product Operation! Least ->! Useful! Example! I Could! Find ->!
  • 21. ! ! ! ! ! ! ! ! ! ! ! ! IBM | spark.tc Spark Shuffle Overview! Most ->! Confusing! Example! I Could! Find ->! Stages are Defined by Shuffle Boundaries!
  • 22. IBM | spark.tc Shuffle Intermediate Data: Spill to Disk! Intermediate shuffle data stored in memory! Spill to Disk! spark.shuffle.spill=true! spark.shuffle.memoryFraction=% of all shuffle buffers! Competes with spark.storage.memoryFraction! Bump this up from default!! Will help Spark SQL, too.! Skipped Stages! Reuse intermediate shuffle data found on reducer! DAG for that partition can be truncated!
  • 23. IBM | spark.tc Shuffle Intermediate Data: Compression! spark.shuffle.compress! Compress outputs (mapper)! ! spark.shuffle.spill.compress! Compress spills (reducer)! ! spark.io.compression.codec! LZF: Most workloads (new default for Spark)! Snappy: LARGE workloads (less memory required to compress)!
  • 24. IBM | spark.tc Spark Shuffle Operations! join! distinct! cogroup! coalesce! repartition! sortByKey! groupByKey! reduceByKey! aggregateByKey!
  • 25. IBM | spark.tc Spark Shuffle Managers! spark.shuffle.manager = {! hash < 10,000 Reducers! Output file determined by hashing the key of (K,V) pair! Each mapper creates an output buffer/file per reducer! Leads to M*R number of output buffers/files per shuffle! sort >= 10,000 Reducers! Default since Spark 1.2! Wins Daytona GraySort Challenge w/ 250,000 reducers!!! tungsten-sort -> Default in Spark 1.5! Uses com.misc.Unsafe for direct access to off heap! }!
  • 27. IBM | spark.tc Shuffle Performance Tuning! Hash Shuffle Manager (no longer default)! spark.shuffle.consolidateFiles: mapper output files! o.a.s.shuffle.FileShuffleBlockResolver! Intermediate Files! Increase spark.shuffle.file.buffer: reduce seeks & sys calls! Increase spark.reducer.maxSizeInFlight if memory allows! Use smaller number of larger workers to reduce total files! SQL: BroadcastHashJoin vs. ShuffledHashJoin! spark.sql.autoBroadcastJoinThreshold ! Use DataFrame.explain(true) or EXPLAIN to verify!
  • 29. IBM | spark.tc Mechanical Sympathy! Use as much of the CPU cache line as possible!!! ! ! ! ! ! ! ! ! !
  • 30. IBM | spark.tc Naïve Matrix Multiplication: Not Cache Friendly! Naive:! for (i = 0; i < N; ++i)! for (j = 0; j < N; ++j)! for (k = 0; k < N; ++k)! res[i][j] += mat1[i][k] * mat2[k][j];! Clever: ! double mat2transpose [N][N];! for (i = 0; i < N; ++i)! for (j = 0; j < N; ++j)! mat2transpose[i][j] = mat2[j][i];! for (i = 0; i < N; ++i)! for (j = 0; j < N; ++j)! for (k = 0; k < N; ++k)! res[i][j] += mat1[i][k] * mat2transpose[j][k];! Prefetch Not Effective On ! Row Wise Traversal! Force All ! Column Traversal by! Transposing Matrix 2!
  • 31. Winning Optimizations Deployed across Spark 1.1 and 1.2
  • 32. IBM | spark.tc Daytona GraySort Challenge: Winning Optimizations! CPU-Cache Locality: Mechanical Sympathy! & Cache Locality/Alignment! ! Optimized Sort Algorithm: Elements of (K, V) Pairs! ! Reduce Network Overhead: Async Netty, epoll! ! Reduce OS Resource Utilization: Sort Shuffle!
  • 33. IBM | spark.tc CPU-Cache Locality: Mechanical Sympathy! AlphaSort paper ~1995! Chris Nyberg and Jim Gray! ! Naïve! List (Pointer-to-Record)! Requires Key to be dereferenced for comparison! ! AlphaSort! List (Key, Pointer-to-Record)! Key is directly available for comparison! ! Key! Ptr! Ptr!
  • 34. IBM | spark.tc CPU-Cache Locality: Cache Locality/Alignment! Key(10 bytes) + Pointer(4 bytes*) = 14 bytes! *4 bytes when using compressed OOPS (<32 GB heap)! Not binary in size! Not CPU-cache friendly! Cache Alignment Options! Add Padding (2 bytes)! Key(10 bytes) + Pad(2 bytes) + Pointer(4 bytes)=16 bytes! (Key-Prefix, Pointer-to-Record)! Key distribution affects performance! Prefix (4 bytes) + Pointer (4 bytes) = 8 bytes! Key! Key! Ptr! Ptr! Ptr! Key-Prefx! Pad! With Padding! Cache-line! Friendly!
  • 35. IBM | spark.tc CPU-Cache Locality: Performance Comparison!
  • 36. IBM | spark.tc Similar Technique: Direct Cache Access! ^ Packet header placed into CPU cache ^!
  • 37. IBM | spark.tc Optimized Sort Algorithm: Elements of (K, V) Pairs! o.a.s.util.collection.TimSort! Based on JDK 1.7 TimSort! Performs best on partially-sorted datasets ! Optimized for elements of (K,V) pairs! Sorts impl of SortDataFormat (ie. KVArraySortDataFormat)! ! o.a.s.util.collection.AppendOnlyMap! Open addressing hash, quadratic probing! Array of [(K, V), (K, V)] ! Good memory locality! Keys never removed, values only append! (^2 Probing)!
  • 38. IBM | spark.tc Reduce Network Overhead: Async Netty, epoll! New Network Module based on Async Netty! Replaces old java.nio, low-level, socket-based code! Zero-copy epoll uses kernel-space between disk & network! Custom memory management reduces GC pauses! spark.shuffle.blockTransferService=netty! Spark-Netty Performance Tuning! spark.shuffle.io.numConnectionsPerPeer! Increase to saturate hosts with multiple disks! spark.shuffle.io.preferDirectBuffers! On or Off-heap (Off-heap is default)!
  • 39. IBM | spark.tc Hash Shuffle Manager!! ! ! ! ! ! ! ! ! ! ! M*R num open files per shuffle; M=num mappers! R=num reducers! Mapper Opens 1 File per Partition/Reducer! HDFS! (2x repl)! HDFS! (2x repl)!
  • 40. S! IBM | spark.tc Reduce OS Resource Utilization: Sort Shuffle! ! ! ! ! ! ! ! ! M open files per shuffle; M = num of mappers! spark.shuffle.sort.bypassMergeThreshold! Merge Sort! (Disk)! Reducers seek and scan from range offset! of Master File on Mapper! TimSort! (RAM)! HDFS! (2x repl)! HDFS! (2x repl)! SPARK-2926:! Replace TimSort w/ Merge Sort! (Memory)! Mapper Merge Sorts Partitions into 1 Master File Indexed by Partition Range Offsets! <- Master->! File!
  • 41. Project Tungsten Deployed across Spark 1.4 and 1.5
  • 42. IBM | spark.tc Significant Spark Core Changes! Disk! Network! CPU! Memory! Daytona GraySort Optimizations! (Spark 1.1-1.2, Late 2014)! Tungsten Optimizations! (Spark 1.4-1.5, Late 2015)!
  • 43. IBM | spark.tc Why is CPU the Bottleneck?! Network and Disk I/O bandwidth are relatively high! ! GraySort optimizations improved network & shuffle! ! Predicate pushdowns and partition pruning! ! Columnar file formats like Parquet and ORC! ! CPU used for serialization, hashing, compression!
  • 44. IBM | spark.tc tungsten-sort Shuffle Manager! “I don’t know your data structure, but my array[] will beat it!” Custom Data Structures for Sort/Shuffle Workload! UnsafeRow: ! ! ! ! Rows are ! 8-byte aligned Primitives are inlined! Row.equals(), Row.hashCode()! operate on raw bytes! Offset (Int) and Length (Int)! Stored in a single Long!
  • 46. IBM | spark.tc Spark + com.misc.Unsafe! org.apache.spark.sql.execution.! aggregate.SortBasedAggregate! aggregate.TungstenAggregate! aggregate.AggregationIterator! aggregate.udaf! aggregate.utils! SparkPlanner! rowFormatConverters! UnsafeFixedWidthAggregationMap! UnsafeExternalSorter! UnsafeExternalRowSorter! UnsafeKeyValueSorter! UnsafeKVExternalSorter! local.ConvertToUnsafeNode! local.ConvertToSafeNode! local.HashJoinNode! local.ProjectNode! local.LocalNode! local.BinaryHashJoinNode! local.NestedLoopJoinNode! joins.HashJoin! joins.HashSemiJoin! joins.HashedRelation! joins.BroadcastHashJoin! joins.ShuffledHashOuterJoin (not yet converted)! joins.BroadcastHashOuterJoin! joins.BroadcastLeftSemiJoinHash! joins.BroadcastNestedLoopJoin! joins.SortMergeJoin! joins.LeftSemiJoinBNL! joins.SortMergerOuterJoin! Exchange! SparkPlan! UnsafeRowSerializer! SortPrefixUtils! sort! basicOperators! aggregate.SortBasedAggregationIterator! aggregate.TungstenAggregationIterator! datasources.WriterContainer! datasources.json.JacksonParser! datasources.jdbc.JDBCRDD! Window! org.apache.spark.! unsafe.Platform! unsafe.KVIterator! unsafe.array.LongArray! unsafe.array.ByteArrayMethods! unsafe.array.BitSet! unsafe.bitset.BitSetMethods! unsafe.hash.Murmur3_x86_32! unsafe.map.BytesToBytesMap! unsafe.map.HashMapGrowthStrategy! unsafe.memory.TaskMemoryManager! unsafe.memory.ExecutorMemoryManager! unsafe.memory.MemoryLocation! unsafe.memory.UnsafeMemoryAllocator! unsafe.memory.MemoryAllocator (trait/interface)! unsafe.memory.MemoryBlock! unsafe.memory.HeapMemoryAllocator! unsafe.memory.ExecutorMemoryManager! unsafe.sort.RecordComparator! unsafe.sort.PrefixComparator! unsafe.sort.PrefixComparators! unsafe.sort.UnsafeSorterSpillWriter! serializer.DummySerializationInstance! shuffle.unsafe.UnsafeShuffleManager! shuffle.unsafe.UnsafeShuffleSortDataFormat! shuffle.unsafe.SpillInfo! shuffle.unsafe.UnsafeShuffleWriter! shuffle.unsafe.UnsafeShuffleExternalSorter! shuffle.unsafe.PackedRecordPointer! shuffle.ShuffleMemoryManager! util.collection.unsafe.sort.UnsafeSorterSpillMerger! util.collection.unsafe.sort.UnsafeSorterSpillReader! util.collection.unsafe.sort.UnsafeSorterSpillWriter! util.collection.unsafe.sort.UnsafeShuffleInMemorySorter! util.collection.unsafe.sort.UnsafeInMemorySorter! util.collection.unsafe.sort.RecordPointerAndKeyPrefix! util.collection.unsafe.sort.UnsafeSorterIterator! network.shuffle.ExternalShuffleBlockResolver! scheduler.Task! rdd.SqlNewHadoopRDD! executor.Executor! org.apache.spark.sql.catalyst.expressions.! regexpExpressions! BoundAttribute! SortOrder! SpecializedGetters! ExpressionEvalHelper! UnsafeArrayData! UnsafeReaders! UnsafeMapData! Projection! LiteralGeneartor! UnsafeRow! JoinedRow! SpecializedGetters! InputFileName! SpecificMutableRow! codegen.CodeGenerator! codegen.GenerateProjection! codegen.GenerateUnsafeRowJoiner! codegen.GenerateSafeProjection! codegen.GenerateUnsafeProjection! codegen.BufferHolder! codegen.UnsafeRowWriter! codegen.UnsafeArrayWriter! complexTypeCreator! rows! literals! misc! stringExpressions! Over 200 source! files affected!!!
  • 47. IBM | spark.tc CPU and Memory Optimizations! Custom Managed Memory Reduces GC overhead Both on and off heap Exact size calculations Direct Binary Processing Operate on serialized/compressed arrays Kryo can reorder serialized records LZF can reorder compressed records More CPU Cache-aware Data Structs & Algorithms o.a.s.unsafe.map.BytesToBytesMap vs. j.u.HashMap Code Generation (default in 1.5) Generate source code from overall query plan Janino generates bytecode from source code 100+ UDFs converted to use code generation Details in ! SPARK-7075! UnsafeFixedWithAggregationMap,& ! TungstenAggregationIterator! CodeGenerator &! GeneratorUnsafeRowJoiner!UnsafeSortDataFormat &! UnsafeShuffleSortDataFormat &! PackedRecordPointer &! UnsafeRow! UnsafeInMemorySorter & UnsafeExternalSorter & UnsafeShuffleWriter! Mostly Same Join Code,! added if (isUnsafeMode)! UnsafeShuffleManager &! UnsafeShuffleInMemorySorter & UnsafeShuffleExternalSorter!
  • 48. IBM | spark.tc Code Generation! Turned on by default in Spark 1.5 Problem: Generic expression evaluation Expensive on JVM Virtual func calls Branches based on expression type Excessive object creation due to primitive boxing Implementation Defer the source code generation to each operator, type, etc Scala quasiquotes provide Scala AST manipulation/rewriting Generated source code is compiled to bytecode w/ Janino 100+ UDFs now using code gen
  • 49. IBM | spark.tc Code Generation: Spark SQL UDFs! 100+ UDFs now using code gen – More to come in Spark 1.6! Details in ! SPARK-8159!
  • 50. IBM | spark.tc Project Tungsten: Beyond Core and Spark SQL! SortDataFormat<K, Buffer>: Base trait ^ implements ^ UncompressedInBlockSort: MLlib.ALS EdgeArraySortDataFormat: GraphX.Edge
  • 51. IBM | spark.tc Relevant Links!   http://sortbenchmark.org/ApacheSpark2014.pdf! !   https://databricks.com/blog/2014/11/05/spark-officially-sets-a-new-record-in-large-scale-sorting.html!   http://0x0fff.com/spark-architecture-shuffle/!   http://www.cs.berkeley.edu/~kubitron/courses/cs262a-F13/projects/reports/project16_report.pdf!   http://stackoverflow.com/questions/763262/how-does-one-write-code-that-best-utilizes-the-cpu-cach e-to-improve-performance!   http://www.aristeia.com/TalkNotes/ACCU2011_CPUCaches.pdf!   http://mishadoff.com/blog/java-magic-part-4-sun-dot-misc-dot-unsafe/!   http://docs.scala-lang.org/overviews/quasiquotes/intro.html!
  • 52. IBM | spark.tc More Relevant Links from Scott Meyers!   http://lwn.net/Articles/252125/ <-- Memory Part 2: CPU Caches!   http://lwn.net/Articles/255364/ <-- Memory Part 5: What Programmers Can Do! ScottMatrix * BruceMatrix^T = CaitlynMatrix!
  • 53. Special thanks to Skimlinks!!! IBM Spark Technology Center is Hiring! " Nice and collaborative people only, please!! IBM | spark.tc Sign up for our newsletter at Thank You, London!!
  • 54. Special thanks to Skimlinks!!! IBM Spark Technology Center is Hiring! " Nice and collaborative people only, please!! IBM | spark.tc Sign up for our newsletter at Thank You, London!!
  • 55. Power of data. Simplicity of design. Speed of innovation. IBM Spark