The document discusses Intel Threading Building Blocks (TBB), a C++ template library for parallel programming. TBB provides features like parallel_for to simplify parallelizing loops across CPU cores without managing threads directly. It uses generic programming principles and provides common parallel algorithms, concurrent data structures, and synchronization primitives to make parallel programming more accessible. TBB aims to improve both correctness through avoiding race conditions and performance through efficient hardware utilization.

1. Exploiting Parallelism with Multi-core Technologies James Reinders Date: Thursday, July 26 Time: 2:35pm - 3:20pm Location: E142

2. Coding with TBB Contest threadingbuildingblocks.org Challenge : Integrate TBB in open source projects through 8/31/07 Judging: Best implementation and most benefit achieved from using TBB Grand Prize : A multi-core laptop and recognition at the upcoming Intel Developers Forum Details : threadingbuildingblocks.org

3. Problem Gaining performance from multi-core requires parallel programming Even a simple “parallel for” is tricky for a non-expert to write well with threads. Two aspects to parallel programming Correctness : avoiding race conditions and deadlock Performance : efficient use of resources Hardware threads (keep busy with real work) Memory space (share between cores) Memory bandwidth (efficient cache line use)

4. Three Approaches for Improvement New language: Cilk, NESL, Haskell, Erlang, Fortress, … Language extensions / pragmas: OpenMP Easier to get acceptance, but require special compiler Library: POOMA, Hood, … Easy to use Domain specific

5. Family Tree 1988 2001 2006 1995 Languages *Other names and brands may be claimed as the property of others Cilk space efficient scheduler cache-oblivious algorithms Threaded-C continuation tasks task stealing OpenMP* fork/join tasks OpenMP taskqueue while & recursion Pragmas Chare Kernel small tasks JSR-166 (FJTask) containers Intel® Threading Building Blocks STL generic programming STAPL recursive ranges ECMA .NET* parallel iteration classes Libraries

6. Enter Intel® Threading Building Blocks 2 years from conception to 1.0 beta Born from a meeting of Principal Engineers who wanted to: Ease parallel programming for variable numbers of cores Build an accessible technology to encourage adoption Take advantage of existing research to meld ease of conception with performance Version 1.0 launched August 2006 Version 1.1 launched April 2007 Version 2.0 Intel provides Threading Building Blocks as an Open Source project with more ports done, and more we can work on together. threadingbuildingblocks.org

7. Example (just a peek) void ApplyFoo( size_t n , int x ) { for(size_t i=range_begin; i<range_end; ++i) Foo( i,x ); } SERIAL VERSION void ParallelApplyFoo(size_t n, int x) { parallel_for( blocked_range<size_t>(0,n,10), <>(const blocked_range<size_t>& range) { for(size_t i=range.begin(); i<range.end(); ++i) Foo(i,x); } ); } PARALLEL VERSION (the way I wish I could write it)

8. Parallel Version (as it can be written) class ApplyFoo { public: int my_x; ApplyFoo( int x ) : my_x(x) {} void operator()(const blocked_range<size_t>& range) const { for(size_t i=range.begin(); i!=range.end(); ++i) Foo(i,my_x); } }; void ParallelApplyFoo(size_t n, int x) { parallel_for(blocked_range<size_t>(0,n,10), ApplyFoo(x)); }

9. Underlying concepts

10. Generic Programming Best known: C++ Standard Template Library Write best possible algorithm - most general way parallel_for does not require specific type of iteration space, but only that it have signatures for recursive splitting Instantiate algorithm to specific situation Enables distribution of broadly-useful high-quality algorithms and data structures

11. Key Features of Intel Threading Building Blocks Manage work as divisible tasks instead of threads Intel TBB maps such tasks onto physical threads Solving cache issues and load balancing for us Full support for nested parallelism Targets threading for robust performance portable, scalable perf. for computationally intense portions Interoperable with other threading packages Emphasizes scalable, data parallel programming

12. Relaxed Sequential Semantics TBB emphasizes relaxed sequential semantics Parallelism as accelerator , not mandatory for correctness.

13. Synchronization Primitives atomic, spin_mutex, spin_rw_mutex, queuing_mutex, queuing_rw_mutex, mutex Generic Parallel Algorithms parallel_for parallel_while parallel_reduce pipeline parallel_sort parallel_scan Concurrent Containers concurrent_hash_map concurrent_queue concurrent_vector Task scheduler Memory Allocation cache_aligned_allocator scalable_allocator

14. Serial Example static void SerialUpdateVelocity() { for( int i=1; i<UniverseHeight-1; ++i ) for( int j=1; j<UniverseWidth-1; ++j ) V[i][j] += (S[i][j] - S[i][j-1] + T[i][j] - T[i-1][j])*M[i]; }

15. Parallel Version blue = original code red = provided by TBB black = boilerplate for library struct UpdateVelocityBody { void operator()( const blocked_range <int>& range ) const { int end = range.end (); for( int i= range.begin (); i<end; ++i ) { for( int j=1; j<UniverseWidth-1; ++j ) { V[i][j] += (S[i][j] - S[i][j-1] + T[i][j] - T[i-1][j])*M[i]; } } } void ParallelUpdateVelocity() { parallel_for ( blocked_range<int> ( 1, UniverseHeight-1), UpdateVelocityBody(), auto_partitioner() ); } Task Parallel control structure Task subdivision handler

16. Range is Generic Requirements for parallel_for Range Library provides blocked_range and blocked_range2d You can define your own ranges Partitioner calls splitting constructor to spread tasks over range Destructor R::~R() True if range is empty bool R::empty() const True if range can be partitioned bool R::is_divisible() const Split r into two subranges R::R (R& r, split ) Copy constructor R::R (const R&)

17. parallel_reduce parallel_scan parallel_while parallel_sort pipeline

18. Parallel pipeline Linear pipeline of stages specify maximum number of items that can be in flight handling arbitrary DAG can take thought but is doable Each stage can be serial or parallel Serial stage = one item at a time, in order. Parallel stage = multiple items at a time, out of order. Uses cache efficiently Each thread carries an item through as many stages as possible. Biases towards finishing old items before tackling new ones.

19. Parallel pipeline Parallel stage scales because it can process items in parallel or out of order. Serial stage processes items one at a time in order. Another serial stage. Items wait for turn in serial stage Controls excessive parallelism by limiting total number of items flowing through pipeline. Uses sequence numbers recover order for serial stage. Tag incoming items with sequence numbers Throughput limited by throughput of slowest serial stage. 1 3 2 4 5 6 7 8 9 10 11 12

20. Concurrent Containers, Mutual Exclusion Memory Allocator Task Scheduler

21. Concurrent Containers Intel TBB provides concurrent containers STL containers are not safe under concurrent operations attempting multiple modifications concurrently could corrupt them Standard practice: wrap a lock around STL container accesses Limits accessors to operating one at a time TBB provides fine-grained locking and lockless operations where possible Worse single-thread performance, but better scalability. Can be used with TBB, OpenMP, or native threads.

22. Concurrent interface requirements Some STL interfaces are not designed to support concurrency For example, suppose two threads each execute: Solution: Intel TBB concurrent_queue has pop_if_present() extern std::queue q; if(!q.empty()) { item=q.front(); q.pop(); } At this instant, another thread might pop last element.

23. concurrent_vector <T> Dynamically growable array of T grow_by (n) grow_to_at_least (n) Never moves elements until cleared Can concurrently access and grow Method clear() is not thread-safe with respect to access/resizing // Append sequence [begin,end) to x in thread-safe way. template<typename T> void Append( concurrent_vector <T> &x, const T *begin, const T *end ) { std::copy(begin, end, x.begin() + x. grow_by (end-begin) ) } Example

24. concurrent_queue <T> Preserves local element order If one thread pushes and another thread pops two values, they come out in the same order as they went in. Two kinds of pops blocking non-blocking Method size() returns signed integer If size() returns – n, it means n pops await corresponding pushes. Caution: queues are cache coolers

25. concurrent_hash <Key,T,HashCompare> Associative table allows concurrent access for reads and updates bool insert ( accessor &result, const Key &key) to add or edit bool find ( accessor &result, const Key &key) to edit bool find ( const_accessor &result, const Key &key) to look up bool erase ( const Key &key) to remove Reader locks coexist – writer locks are exclusive

26. struct MyHashCompare { static long hash ( const char* x ) { long h = 0; for( const char* s = x; *s; s++ ) h = (h*157)^*s; return h; } static bool equal ( const char* x, const char* y ) { return strcmp(x,y)==0; } }; typedef concurrent_hash_map <const char*,int,MyHashCompare> StringTable; StringTable MyTable; Example: map strings to integers void MyUpdateCount( const char* x ) { StringTable::accessor a; MyTable. insert ( a, x ); a->second += 1; } Multiple threads can insert and update entries concurrently. accessor object acts as smart pointer and writer lock.

27. Synchronization Primitives Shared Data use mutual exclusion to avoid races TBB mutual exclusion regions are protected by scoped locks The range of the lock is determined by its lifetime (scope) Leaving lock scope calls the destructor, making it exception safe Minimizing lock lifetime avoids possible contention Several mutex behaviors are available Spin vs. queued Writer vs. reader/writer Scoped wrapper of native mutual exclusion function

28. Example: spin_rw_mutex promotion spin_rw_mutex MyMutex; int foo () { spin_rw_mutex::scoped_lock lock (MyMutex, /*is_writer*/ false); … if (!lock. upgrade_to_writer ()) { /* reacquire state */ } /* perform desired write */ return 0; /* Destructor of ‘lock’ releases ‘MyMutex’ */ } Exceptions occurring within the locked code range automatically release the lock ( lock passes out of scope), avoiding deadlock Any reader lock may be upgraded to writer lock; upgrade_to_writer() fails if the lock had to be released before it can be locked for writing

29. Scalable Memory Allocator Problem Memory allocation can bottle-neck a concurrent environment Thread allocation from a global heap requires global locks. Solution Intel® Threading Building Blocks provides tested, tuned, scalable, per-thread memory allocation Scalable memory allocator interface can be used… As an allocator argument to STL template classes As a replacement for malloc/realloc/free calls (C programs) As a replacement for new and delete operators (C++ programs)

30. Task Scheduler Underlying the generic task structure is a task scheduler Core to task scheduling is a thread pool whereby Intel TBB maximizes thread efficiency and manages its complexity Task scheduler is designed to address common performance issues of parallel programming with native threads TBB Approach Problem Task chunking and work-stealing help balance load Load imbalance Programmer specifies tasks, not threads. Program complexity “ Greedy” scheduling often wins “ Fair” scheduling One scheduler thread per hardware thread Oversubscription

31. Task Scheduler Intel TBB task interest is managed in the task_scheduler_init object Thread pool construction also tied to the life of this object Nested construction is reference counted, low overhead Put Init object scope high to avoid pool reconstruction overhead Construction specifies thread pool size automatic , explicit or deferred Dynamic init object lifetime management offers thread pool size control #include “tbb/task_scheduler_init.h” using namespace tbb; int main() { task_scheduler_init init; … . return 0; }

32. Tasking Development tools Intel TBB offers facilities to accelerate development Linking with libtbb_debug.so (or Win/Mac equivalents) adds checking TBB_DO_ASSERT macro extends checking into the header/inline code TBB_DO_THREADING_TOOLS adds hooks for Intel Thread Analysis tools The tick_count class offers convenient timing services. tick_count:: now () returns current timestamp tick_count::interval_t:: operator- (const tick_count &t1, const tick_count &t2) double tick_count::interval_t:: seconds () converts intervals to real time ITT_NOTIFY events can be useful with Intel Thread Analysis tools

33. open source tour

34. Open Source – quick ‘tour’ Source library organized around 4 directories src – C++ source for Intel TBB, TBBmalloc and the unit tests include – the standard include files build – catchall for platform-specific build information examples – TBB sample code Top level index.html offers help on building and porting Build prerequisites: C++ compiler for target environment GNU make Bourne or BASH-compatible shell Some architectures may require an assembler for low-level primitives

35. Coding with TBB Contest Challenge : Integrate TBB in open source projects through 8/31/07 Judging: Best implementation and most benefit achieved from using TBB Grand Prize : A multi-core laptop and recognition at the upcoming Intel Developers Forum Details : threadingbuildingblocks.org

36. Learn more… INTEL booth (today until 5pm!) many Intel engineers – come see us! threadingbuildingblocks.org