An introduction to the OpenMP parallel programming model.
From the Scalable Computing Support Center at Duke University (http://wiki.duke.edu/display/scsc)
The JVM memory model describes how threads in the Java eco-system interact through memory. While the memory model impact on developing for the JVM may not be obvious, it is the cause for certain number of "anomalies" that are, well, by design.
In this presentation we will explore the aspects of the memory model, including things like reordering of instructions, volatile members, monitors, atomics and JIT.
This presentation deals with how one can utilize multiple cores, while working with C/C++ applications using an API called OpenMP. It's a shared memory programming model, built on top of POSIX thread. Also the fork-join model, parallel design pattern are discussed using PetriNets.
An introduction to the OpenMP parallel programming model.
From the Scalable Computing Support Center at Duke University (http://wiki.duke.edu/display/scsc)
The JVM memory model describes how threads in the Java eco-system interact through memory. While the memory model impact on developing for the JVM may not be obvious, it is the cause for certain number of "anomalies" that are, well, by design.
In this presentation we will explore the aspects of the memory model, including things like reordering of instructions, volatile members, monitors, atomics and JIT.
This presentation deals with how one can utilize multiple cores, while working with C/C++ applications using an API called OpenMP. It's a shared memory programming model, built on top of POSIX thread. Also the fork-join model, parallel design pattern are discussed using PetriNets.
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Data and AI
Round table discussion of vector databases, unstructured data, ai, big data, real-time, robots and Milvus.
A lively discussion with NJ Gen AI Meetup Lead, Prasad and Procure.FYI's Co-Found
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Data and AI
Round table discussion of vector databases, unstructured data, ai, big data, real-time, robots and Milvus.
A lively discussion with NJ Gen AI Meetup Lead, Prasad and Procure.FYI's Co-Found
Analysis insight about a Flyball dog competition team's performanceroli9797
Insight of my analysis about a Flyball dog competition team's last year performance. Find more: https://github.com/rolandnagy-ds/flyball_race_analysis/tree/main
Adjusting OpenMP PageRank : SHORT REPORT / NOTESSubhajit Sahu
For massive graphs that fit in RAM, but not in GPU memory, it is possible to take
advantage of a shared memory system with multiple CPUs, each with multiple cores, to
accelerate pagerank computation. If the NUMA architecture of the system is properly taken
into account with good vertex partitioning, the speedup can be significant. To take steps in
this direction, experiments are conducted to implement pagerank in OpenMP using two
different approaches, uniform and hybrid. The uniform approach runs all primitives required
for pagerank in OpenMP mode (with multiple threads). On the other hand, the hybrid
approach runs certain primitives in sequential mode (i.e., sumAt, multiply).
Unleashing the Power of Data_ Choosing a Trusted Analytics Platform.pdfEnterprise Wired
In this guide, we'll explore the key considerations and features to look for when choosing a Trusted analytics platform that meets your organization's needs and delivers actionable intelligence you can trust.
Learn SQL from basic queries to Advance queriesmanishkhaire30
Dive into the world of data analysis with our comprehensive guide on mastering SQL! This presentation offers a practical approach to learning SQL, focusing on real-world applications and hands-on practice. Whether you're a beginner or looking to sharpen your skills, this guide provides the tools you need to extract, analyze, and interpret data effectively.
Key Highlights:
Foundations of SQL: Understand the basics of SQL, including data retrieval, filtering, and aggregation.
Advanced Queries: Learn to craft complex queries to uncover deep insights from your data.
Data Trends and Patterns: Discover how to identify and interpret trends and patterns in your datasets.
Practical Examples: Follow step-by-step examples to apply SQL techniques in real-world scenarios.
Actionable Insights: Gain the skills to derive actionable insights that drive informed decision-making.
Join us on this journey to enhance your data analysis capabilities and unlock the full potential of SQL. Perfect for data enthusiasts, analysts, and anyone eager to harness the power of data!
#DataAnalysis #SQL #LearningSQL #DataInsights #DataScience #Analytics
Techniques to optimize the pagerank algorithm usually fall in two categories. One is to try reducing the work per iteration, and the other is to try reducing the number of iterations. These goals are often at odds with one another. Skipping computation on vertices which have already converged has the potential to save iteration time. Skipping in-identical vertices, with the same in-links, helps reduce duplicate computations and thus could help reduce iteration time. Road networks often have chains which can be short-circuited before pagerank computation to improve performance. Final ranks of chain nodes can be easily calculated. This could reduce both the iteration time, and the number of iterations. If a graph has no dangling nodes, pagerank of each strongly connected component can be computed in topological order. This could help reduce the iteration time, no. of iterations, and also enable multi-iteration concurrency in pagerank computation. The combination of all of the above methods is the STICD algorithm. [sticd] For dynamic graphs, unchanged components whose ranks are unaffected can be skipped altogether.
Enhanced Enterprise Intelligence with your personal AI Data Copilot.pdfGetInData
Recently we have observed the rise of open-source Large Language Models (LLMs) that are community-driven or developed by the AI market leaders, such as Meta (Llama3), Databricks (DBRX) and Snowflake (Arctic). On the other hand, there is a growth in interest in specialized, carefully fine-tuned yet relatively small models that can efficiently assist programmers in day-to-day tasks. Finally, Retrieval-Augmented Generation (RAG) architectures have gained a lot of traction as the preferred approach for LLMs context and prompt augmentation for building conversational SQL data copilots, code copilots and chatbots.
In this presentation, we will show how we built upon these three concepts a robust Data Copilot that can help to democratize access to company data assets and boost performance of everyone working with data platforms.
Why do we need yet another (open-source ) Copilot?
How can we build one?
Architecture and evaluation
4. Introduction
• Types of parallel machines
– distributed memory
• each processor has its own memory address space
• variable values are independent
x = 2 on one processor, x = 3 on a different processor
• examples: linux clusters, Blue Gene/L
– shared memory
• also called Symmetric Multiprocessing (SMP)
• single address space for all processors
– If one processor sets x = 2 , x will also equal 2 on other
processors (unless specified otherwise)
• examples: IBM p-series, SGI Altix
5. Shared vs. Distributed Memory
CPU 0 CPU 1 CPU 2 CPU 3 CPU 0 CPU 1 CPU 2 CPU 3
MEM 0 MEM 1 MEM 2 MEM 3 MEM
shared
distributed
6. Shared vs. Distributed Memory (cont’d)
• Multiple processes
– Each processor (typically) performs
independent task
• Multiple threads
– A process spawns additional tasks (threads)
with same memory address space
7. What is OpenMP?
• Application Programming Interface (API)
for multi-threaded parallelization consisting
of
– Source code directives
– Functions
– Environment variables
8. What is OpenMP? (cont’d)
• Advantages
– Easy to use
– Incremental parallelization
– Flexible
• Loop-level or coarse-grain
– Portable
• Since there’s a standard, will work on any SMP machine
• Disadvantage
– Shared-memory systems only
10. Basics
• Goal – distribute work among threads
• Two methods will be discussed here
– Loop-level
• Specified loops are parallelized
• This is approach taken by automatic parallelization tools
– Parallel regions
• Sometimes called “coarse-grained”
• Don’t know good term; good way to start argument with
semantically precise people
• Usually used in message-passing (MPI)
12. parallel do & parallel for
• parallel do (Fortran) and parallel for (C)
directives parallelize subsequent loop
#pragma omp parallel for
for(i = 1; i <= maxi; i++){
a[i] = b[i] = c[i];
}
!$omp parallel do
do i = 1, maxi
a(i) = b(i) + c(i)
enddo
Use “c$” for fixed-format Fortran
13. parallel do & parallel for (cont’d)
• Suppose maxi = 1000
Thread 0 gets i = 1 to 250
Thread 1 gets i = 251 to 500
etc.
• Barrier implied at end of loop
14. workshare
• For Fortran 90/95 array syntax, the parallel
workshare directive is analogous to
parallel do
• Previous example would be:
• Also works for forall and where statements
!$omp parallel workshare
a = b + c
!$omp end parallel workshare
15. Shared vs. Private
• In parallel region, default behavior is that
all variables are shared except loop index
– All threads read and write the same memory
location for each variable
– This is ok if threads are accessing different
elements of an array
– Problem if threads write same scalar or array
element
– Loop index is private, so each thread has its
own copy
16. Shared vs. Private (cont’d)
• Here’s an example where a shared
variable, i2, could cause a problem
ifirst = 10
do i = 1, imax
i2 = 2*i
j(i) = ifirst + i2
enddo
ifirst = 10;
for(i = 1; i <= imax; i++){
i2 = 2*i;
j[i] = ifirst + i2;
}
17. Shared vs. Private (3)
• OpenMP clauses modify the behavior of
directives
• Private clause creates separate memory
location for specified variable for each
thread
18. Shared vs. Private (4)
ifirst = 10
!$omp parallel do private(i2)
do i = 1, imax
i2 = 2*i
j(i) = ifirst + i2
enddo
ifirst = 10;
#pragma omp parallel for private(i2)
for(i = 1; i <= imax; i++){
i2 = 2*i;
j[i] = ifirst + i2;
}
19. Shared vs. Private (5)
• Look at memory just before and after
parallel do/parallel for statement
– Let’s look at two threads
ifirst = 10
…
i2thread 0 = ???
spawn
additional
thread
ifirst = 10
…
i2thread 0 = ???
i2thread 1 = ???
…
before after
20. Shared vs. Private (6)
• Both threads access the same shared
value of ifirst
• They have their own private copies of i2
ifirst = 10
…
i2thread 0 = ???
i2thread 1 = ???
…
Thread 0 Thread 1
22. Data Dependencies
• Data on one thread can be dependent on
data on another thread
• This can result in wrong answers
– thread 0 may require a variable that is
calculated on thread 1
– answer depends on timing – When thread 0
does the calculation, has thread 1 calculated
it’s value yet?
24. Data Dependencies (3)
• parallelize on 2 threads
– thread 0 gets i = 3 to 51
– thread 1 gets i = 52 to 100
– look carefully at calculation for i = 52 on
thread 1
• what will be values of for i -1 and i - 2 ?
25. Data Dependencies (4)
• A test for dependency:
– if serial loop is executed in reverse order, will
it give the same result?
– if so, it’s ok
– you can test this on your serial code
26. Data Dependencies (5)
• What about subprogram calls?
• Does the subprogram write x or y to memory?
– If so, they need to be private
• Variables local to subprogram are local to each
thread
• Be careful with global variables and common
blocks
do i = 1, 100
call mycalc(i,x,y)
enddo
for(i = 0; i < 100; i++){
mycalc(i,x,y);
}
28. More clauses
• Can make private default rather than
shared
– Fortran only
– handy if most of the variables are private
– can use continuation characters for long lines
ifirst = 10
!$omp parallel do &
!$omp default(private) &
!$omp shared(ifirst,imax,j)
do i = 1, imax
i2 = 2*i
j(i) = ifirst + i2
enddo
29. More clauses (cont’d)
• Can use default none
– declare all variables (except loop variables)
as shared or private
– If you don’t declare any variables, you get a
handy list of all variables in loop
30. More clauses (3)
ifirst = 10
!$omp parallel do &
!$omp default(none) &
!$omp shared(ifirst,imax,j) &
!$omp private(i2)
do i = 1, imax
i2 = 2*i
j(i) = ifirst + i2
enddo
ifirst = 10;
#pragma omp parallel for
default(none)
shared(ifirst,imax,j)
private(i2)
for(i = 0; i < imax; i++){
i2 = 2*i;
j[i] = ifirst + i2;
}
31. Firstprivate
• Suppose we need a running index total for
each index value on each thread
• if iper were declared private, the initial
value would not be carried into the loop
iper = 0
do i = 1, imax
iper = iper + 1
j(i) = iper
enddo
iper = 0;
for(i = 0; i < imax; i++){
iper = iper + 1;
j[i] = iper;
}
32. Firstprivate (cont’d)
• Solution – firstprivate clause
• Creates private memory location for each
thread
• Copies value from master thread (thread
0) to each memory location
iper = 0
!$omp parallel do &
!$omp firstprivate(iper)
do i = 1, imax
iper = iper + 1
j(i) = iper
enddo
iper = 0;
#pragma omp parallel for
firstprivate(iper)
for(i = 0; i < imax; i++){
iper = iper + 1;
j[i] = iper;
}
33. Lastprivate
• saves value corresponding to the last loop
index
– "last" in the serial sense
!$omp parallel do lastprivate(i)
do i = 1, maxi-1
a(i) = b(i)
enddo
a(i) = b(1)
#pragma omp parallel for
lastprivate(i)
for(i = 0; i < maxi-1; i++){
a[i] = b[i];
}
a(i) = b(1);
34. Reduction
• following example won’t parallelize
correctly with parallel do/parallel for
– different threads may try to write to sum1
simultaneously
sum1 = 0.0
do i = 1, maxi
sum1 = sum1 + a(i)
enddo
sum1 = 0.0;
for(i = 0; i < imaxi; i++){
sum1 = sum1 + a[i];
}
35. Reduction (cont’d)
• Solution? – Reduction clause
• each thread performs its own reduction (sum, in
this case)
• results from all threads are automatically
reduced (summed) at the end of the loop
sum1 = 0.0
!$omp parallel do &
!$reduction(+:sum1)
do i = 1, maxi
sum1 = sum1 + a(i)
enddo
sum1 = 0;
#pragma omp parallel for
reduction(+:sum1)
for(i = 0; i < imaxi; i++){
sum1 = sum1 + a[i];
}
36. Reduction (3)
• Fortran operators/intrinsics: MAX, MIN,
IAND, IOR, IEOR,
+, *, -, .AND., .OR., .EQV., .NEQV.
• C operators: +, *, -, /, &, ^, |, &&, ||
• roundoff error may be different than serial
case
37. num_threads
• Number of threads for subsequent loop
can be specified with num_threads(n)
clause
– n is number of threads
38. Conditional Compilation
• Fortran: if compiled without OpenMP, directives are
treated as comments
– Great for portability
• !$ (c$ for fixed format) can be used for conditional
compilation for any source lines
!$ print*, ‘number of procs =', nprocs
• C or C++: conditional compilation can be performed with
the _OPENMP macro name.
#ifdef _OPENMP
… do stuff …
#endif
39. Basic OpenMP Functions
• omp_get_thread_num()
– returns ID of current thread
• omp_set_num_threads(nthreads)
– subroutine in Fortran
– sets number of threads in next parallel region to
nthreads
– alternative to OMP_NUM_THREADS environment
variable
– overrides OMP_NUM_THREADS
• omp_get_num_threads()
– returns number of threads in current parallel region
41. Some Hints
• OpenMP will do what you tell it to do
– If you try parallelize a loop with a dependency, it will go
ahead and do it!
• Do not parallelize small loops
– Overhead will be greater than speedup
– How small is “small”?
• Answer depends on processor speed and other system-
dependent parameters
• Try it!
• Maximize number of operations performed in
parallel
– parallelize outer loops where possible
42. Some Hints (cont’d)
• Always profile your code before parallelizing!
– tells where most time is being used
• Portland Group profiler:
– compile with –Mprof=func flag
– run code
• file with profile data will appear in working directory
– run profiler:
• pgprof –exe executable-name
• GUI will pop up with profiling information
43. Compile and Run
• Portland Group compilers (katana):
– Compile with -mp flag
– Can use up to 8 threads
• AIX compilers (twister, etc.):
– Use an _r suffix on the compiler name, e.g., cc_r,
xlf90_r
– Compile with -qsmp=omp flag
• Intel compilers (skate, cootie):
– Compile with -openmp and –fpp flags
– Can only use 2 threads
44. Compile and Run (cont’d)
• GNU compilers (all machines):
– Compile with -fopenmp flag
• Blue Gene does not accomodate OpenMP
• environment variable OMP_NUM_THREADS
sets number of threads
setenv OMP_NUM_THREADS 4
• for C, include omp.h
45. Exercise
• Copy files from /scratch/sondak/gt
– both C and Fortran versions are available for your
programming pleasure
• compile code with pgcc or pgf90
– include profiling flag –Mprof=func
– call executable “gt”
• Run code batch
– qsub rungt
– qstat –u username
46. Exercise (cont’d)
• When run completes, look at profile
– pgprof –exe gt
– Where is most time being used?
– How can we tackle this with OpenMP?
• Recompile without profiler flag
• Re-run and note timing from batch output file
– this is the base timing
• We will decide on an OpenMP strategy
• Add OpenMP directive(s)
• Re-run on 1, 2, and 4 procs. and check new
timings
47. Parallel
• parallel and do/for can be separated into two directives.
is the same as
!$omp parallel do
do i = 1, maxi
a(i) = b(i)
enddo
#pragma omp parallel for
for(i=0; i<maxi; i++){
a[i] = b[i];
}
!$omp parallel
!$omp do
do i = 1, maxi
a(i) = b(i)
enddo
!$omp end parallel
#pragma omp parallel
#pragma omp for
for(i=0; i<maxi; i++){
a[i] = b[i];
}
#pragma omp end parallel
48. Parallel (cont’d)
• Note that an end parallel directive is required.
• Everything within the parallel region will be run
in parallel (surprise!).
• The do/for directive indicates that the loop
indices will be distributed among threads rather
than duplicating every index on every thread.
49. Parallel (3)
• Multiple loops in parallel region:
• parallel directive has a significant overhead associated
with it.
• The above example has the potential to be faster than
using two parallel do/parallel for directives.
!$omp parallel
!$omp do
do i = 1, maxi
a(i) = b(i)
enddo
!$omp do
do i = 1, maxi
c(i) = a(2)
enddo
!$omp end parallel
#pragma omp parallel
#pragma omp for
for(i=0; i<maxi; i++){
a[i] = b[i];
}
#pragma omp for
for(i=0; i<maxi; i++){
c[i] = a[2];
}
#pragma omp end parallel
50. Coarse-Grained
• OpenMP is not restricted to loop-level (fine-
grained) parallelism.
• The !$omp parallel or #pragma omp parallel
directive duplicates subsequent code on all
threads until a !$omp end parallel or #pragma
omp end parallel directive is encountered.
• Allows parallelization similar to “MPI paradigm.”
53. • barrier synchronizes threads
• Here barrier assures that a(1) or a[0] is available before
computing myval
Barrier
$omp parallel private(myid,istart,iend)
call myrange(myid,istart,iend)
do i = istart, iend
a(i) = a(i) - b(i)
enddo
!$omp barrier
myval(myid+1) = a(istart) + a(1)
#pragma omp parallel private(myid,istart,iend)
myrange(myid,istart,iend);
for(i=istart; i<=iend; i++){
a[i] = a[i] – b[i];
}
#pragma omp barrier
myval[myid] = a[istart] + a[0]
54. • if you want part of code to be executed
only on master thread, use master
directive
• “non-master” threads will skip over master
region and continue
Master
55. Master Example - Fortran
!$OMP PARALLEL PRIVATE(myid,istart,iend)
call myrange(myid,istart,iend)
do i = istart, iend
a(i) = a(i) - b(i)
enddo
!$OMP BARRIER
!$OMP MASTER
write(21) a
!$OMP END MASTER
call do_work(istart,iend)
!$OMP END PARALLEL
56. Master Example - C
#pragma omp parallel private(myid,istart,iend)
myrange(myid,istart,iend);
for(i=istart; i<=iend; i++){
a[i] = a[i] – b[i];
}
#pragma omp barrier
#pragma omp master
fwrite(fid,sizeof(float),iend-istart+1,a);
#pragma omp end master
do_work(istart,iend);
#pragma omp end parallel
57. • want part of code to be executed only by a
single thread
• don’t care whether or not it’s the master
thread
• use single directive
• Unlike the end master directive, end single
implies a barrier.
Single
58. Single Example - Fortran
!$OMP PARALLEL PRIVATE(myid,istart,iend)
call myrange(myid,istart,iend)
do i = istart, iend
a(i) = a(i) - b(i)
enddo
!$OMP BARRIER
!$OMP SINGLE
write(21) a
!$OMP END SINGLE
call do_work(istart,iend)
!$OMP END PARALLEL
59. Single Example - C
#pragma omp parallel private(myid,istart,iend)
myrange(myid,istart,iend);
for(i=istart; i<=iend; i++){
a[i] = a[i] – b[i];
}
#pragma omp barrier
#pragma omp single
fwrite(fid,sizeof(float),nvals,a);
#pragma omp end single
do_work(istart,iend);
#pragma omp end parallel
60. • have section of code that:
1. must be executed by every thread
2. threads may execute in any order
3. threads must not execute simultaneously
• use critical directive
• no implied barrier
Critical
61. Critical Example - Fortran
!$OMP PARALLEL PRIVATE(myid,istart,iend)
call myrange(myid,istart,iend)
do i = istart, iend
a(i) = a(i) - b(i)
enddo
!$OMP CRITICAL
call mycrit(myid,a)
!$OMP END CRITICAL
call do_work(istart,iend)
!$OMP END PARALLEL
62. Critical Example - C
#pragma omp parallel private(myid,istart,iend)
myrange(myid,istart,iend);
for(i=istart; i<=iend; i++){
a[i] = a[i] – b[i];
}
#pragma omp critical
mycrit(myid,a);
#pragma omp end critical
do_work(istart,iend);
#pragma omp end parallel
63. Ordered
• Suppose you want to write values in a loop:
• If loop were parallelized, could write out of order
• ordered directive forces serial order
do i = 1, nproc
call do_lots_of_work(result(i))
write(21,101) i, result(i)
enddo
for(i = 0; i < nproc; i++){
do_lots_of_work(result[i]);
fprintf(fid,”%d %fn,”i,result[i]”);
}
64. Ordered (cont’d)
!$omp parallel do
do i = 1, nproc
call do_lots_of_work(result(i))
!$omp ordered
write(21,101) i, result(i)
!$omp end ordered
enddo
#pragma omp parallel for
for(i = 0; i < nproc; i++){
do_lots_of_work(result[i]);
#pragma omp ordered
fprintf(fid,”%d %fn,”i,result[i]”);
#pragma omp end ordered
}
65. Ordered (3)
• since do_lots_of_work takes a lot of time,
most parallel benefit will be realized
• ordered is helpful for debugging
– Is result same with and without ordered
directive?
66. • In parallel region, have several independent
tasks
• Divide code into sections
• Each section is executed on a different thread.
• Implied barrier
• Fixes number of threads
• Note that sections directive delineates region
of code which includes section directives.
Sections
67. Sections Example - Fortran
!$OMP PARALLEL
!$OMP SECTIONS
!$OMP SECTION
call init_field(a)
!$OMP SECTION
call check_grid(x)
!$OMP END SECTIONS
!$OMP END PARALLEL
68. Sections Example - C
#pragma omp parallel
#pragma omp sections
#pragma omp section
init_field(a);
#pragma omp section
check_grid(x);
#pragma omp end sections
#pragma omp end parallel
69. • Analogous to do and parallel do or for and
parallel for, along with sections there exists a
parallel sections directive.
Parallel Sections
71. Parallel Sections Example - C
#pragma omp parallel sections
#pragma omp section
init_field(a);
#pragma omp section
check_grid(x);
#pragma omp end parallel sections
72. • Parallel sections are established upon
compilation
– Number of threads is fixed
• Sometimes more flexibility is needed, such as
parallelism within if or while block
• Task directive will assign tasks to threads as
needed
Task
73. Task Example - Fortran
!$OMP PARALLEL
!$OMP SINGLE
p = listhead
do while(p)
!$OMP TASK
process(p)
!$OMP END TASK
p = next(p)
enddo
!$OMP END SINGLE
!$OMP END
74. Task Example - C
#pragma omp parallel
{
#pragma omp single
{
p = listhead;
while(p) {
#pragma omp task
process(p);
p = next(p);
}
}
}
75. • Only needed for Fortran
– Usually legacy code
• common block variables are inherently shared,
may want them to be private
• threadprivate gives each thread its own private
copy of variables in specified common block.
– Keep memory use in mind
• Place threadprivate after common block
declaration.
Threadprivate
77. • threadprivate renders all values in common
block private
• may want to use some current values of
variables on all threads
• copyin initializes each thread’s copy of
specified common-block variables with values
form the master thread
Copyin
78. Copyin Example - Fortran
real, parameter :: nvals = 501
real, dimension(nvals) :: x, y, xmet, ymet
common /work1/ work(nvals), istart, iend
!$OMP THREADPRIVATE(/work1/)
istart = 2; iend = 500
call read_coords(x,y)
!$OMP PARALLEL SECTIONS
COPYIN(istart,iend)
!$OMP SECTION
call metric(x)
xmet = work
!$OMP SECTION
call metric(y)
ymet = work
!$OMP END PARALLEL SECTIONS
subroutine metric(c)
real, dimension(:) :: c
common /work1/ work(nvals), istart, iend
!$OMP THREADPRIVATE(/work1/)
do i = istart, iend
work(i) = 0.5*(c(i+1)-c(i-1))
enddo
end subroutine metric
79. • Similar to critical
– Allows greater optimization than critical
• Applies to single line following the atomic
directive
Atomic
80. Atomic Example - Fortran
do i = 1, 10
!$OMPATOMIC
x(j(i)) = x(j(i)) + 1.0
enddo
81. Atomic Example - C
for(i = 0; i<10; i++){
#pragma omp atomic
x[j[i]]= x[j[i]] + 1.0;
}
82. • Different threads can have different values for
the same shared variable
– example – value in register
• flush assures that the calling thread has a
consistent view of memory
• upcoming example is from OpenMP spec
Flush
83. Flush Example - Fortran
integer isync(num_threads)
!$omp parallel default(private) shared(isync)
iam = omp_get_thread_num()
isync(iam) = 0
!$omp barrier
call work()
isync(iam) = 1
! let other threads know I finished work
!$omp flush(isync)
do while(isync(neigh) == 0)
!$omp flush(isync)
enddo
$omp end parallel
84. Flush Example - C
int isync[num_threads];
#pragma omp parallel default(private) shared(isync)
iam = omp_get_thread_num();
isync[iam] = 0;
#pragma omp barrier
work();
isync[iam] = 1;
/* let other threads know I finished work */
#pragma omp flush(isync)
while(isync[neigh] == 0)
#pragma omp flush(isync)
#pragma omp end parallel
86. • On some systems the number of threads
available to you may be adjusted dynamically.
• The number of threads requested can be
construed as the maximum number of threads
available.
• Dynamic threading can be turned on or off:
Dynamic Thread Assignment
logical mydyn = .false.
call omp_set_dynamic(mydyn)
int mydyn = 0;
omp_set_dynamic(mydyn);
87. • Dynamic threading can also be turned on or off
by setting the environment variable
omp_dynamic to true or false
– call to omp_set_dynamic overrides the environment
variable
• The function omp_get_dynamic() returns a
value of “true” or “false,” indicating whether
dynamic threading is turned on or off.
Dynamic Thread Assignment (cont’d)
88. • integer function
• returns maximum number of threads available
in current parallel region
• same result as omp_get_num_threads if
dynamic threading is turned off
OMP_GET_MAX_THREADS
89. • integer function
• returns maximum number of processors in the
system
– indicates amount of hardware, not number of
available processors
• could be used to make sure enough
processors are available for specified number
of sections
OMP_GET_NUM_PROCS
90. • logical function
• tells whether or not function was called from a
parallel region
• useful debugging device when using
“orphaned” directives
– example of orphaned directive: omp parallel in
“main,” omp do or omp for in routine called from main
OMP_IN_PARALLEL
93. Collapse Example - Fortran
!$omp parallel do collapse(2)
do j = 1, nval
do i = 1, nval
ival(i,j) = i + j
enddo
enddo
94. Collapse Example - C
#pragma omp parallel for collapse(2)
for(j=0; j<nval; j++){
for(i=0; i<nval; i++){
ival[i][j] = i + j;
}
}
95. Schedule
• schedule refers to the way in which loop
indices are distributed among threads
• default is static, which we had seen in an
earlier example
– each thread is assigned a contiguous range of
indices in order of thread number
• called round robin
– number of indices assigned to each thread is
as equal as possible
96. Schedule (cont’d)
• static example
– loop runs from 1 to 51, 4 threads
thread indices no. indices
0 1-13 13
1 14-26 13
2 27-39 13
3 40-51 12
97. Schedule (3)
• number of indices doled out at a time to each
thread is called the chunk size
• can be modified with the SCHEDULE clause
!$omp do schedule(static,5)
#pragma omp for schedule(static,5)
98. Schedule (4)
• example – same as previous example (loop runs
from 1-51, 4 threads), but set chunk size to 5
thread chunk 1
indices
chunk 2
indices
chunk 3
indices
no.
indices
0 1-5 21-25 41-45 15
1 6-10 26-30 46-50 15
2 11-15 31-35 51 11
3 16-20 36-40 - 10
99. Dynamic Schedule
• schedule(dynamic) clause assigns chunks to
threads dynamically as the threads become
available for more work
• default chunk size is 1
• higher overhead than STATIC
!$omp do schedule(dynamic,5)
#pragma omp for
schedule(dynamic,5)
100. Guided Schedule
• schedule(guided) clause assigns chunks
automatically, exponentially decreasing chunk
size with each assignment
• specified chunk size is the minimum chunk size
except for the last chunk, which can be of any
size
• default chunk size is 1
101. Runtime Schedule
• schedule can be specified through
omp_schedule environment variable
setenv omp_schedule “dynamic,5”
• the schedule(runtime) clause tells it to set the
schedule using the environment variable
!$omp do schedule(runtime)
#pragma omp for schedule(runtime)
103. • parallel construct within parallel construct:
Nested Parallelism
!$omp parallel do
do j = 1, jmax
!$omp parallel do
do i = 1, i,max
call do_work(i,j)
enddo
enddo
#pragma omp parallel for
for(j=0; j<jmax; j++){
#pragma omp parallel for
for(i=0; i<imax; i++){
do_work(i,j);
}
}
104. • OpenMP standard allows but does not require
nested parallelism
• if nested parallelism is not implemented,
previous examples are legal, but the inner loop
will be serial
• logical function omp_get_nested() returns
.true. or .false. (1 or 0) to indicate whether or
not nested parallelism is enabled in the current
region
Nested Parallelism (cont’d)
105. • omp_set_nested(nest) enables/disables
nested parallelism if argument is true/false
– subroutine in Fortran (.true./.false.)
– function in C (1/0)
• nested parallelism can also be turned on or off
by setting the environment variable
omp_nested to true or false
– calls to omp_set_nested override the environment
variable
Nested Parallelism (3)
107. • Locks offer additional control of threads
• a thread can take/release ownership of a lock
over a specified region of code
• when a lock is owned by a thread, other
threads cannot execute the locked region
• can be used to perform useful work by one or
more threads while another thread is engaged
in a serial task
Locks
108. • Lock routines each have a single argument
– argument type is an integer (Fortran) or a pointer to
an integer (C) long enough to hold an address
– OpenMP pre-defines required types
integer(omp_lock_kind) :: mylock
omp_nest_lock_t *mylock;
Lock Routines
109. • omp_init_lock(mylock)
– initializes mylock
– must be called before mylock is used
– subroutine in Fortran
• omp_set_lock(mylock)
– gives ownership of mylock to calling thread
– other threads cannot execute code following call
until mylock is released
– subroutine in Fortran
Lock Routines (cont’d)
110. • omp_test_lock(mylock)
– logical function
– if mylock is presently owned by another
thread, returns false
– if mylock is available, returns true and sets
lock, i.e., gives ownership to calling
thread
– be careful: omp_test_lock(mylock)does
more than just test the lock
Lock Routines (3)
111. • omp_unset_lock(mylock)
– releases ownership of mylock
– subroutine in Fortran
• omp_destroy_lock(mylock)
– call when you’re done with the lock
– complement to omp_init_lock(mylock)
– subroutine in Fortran
Lock Routines (4)
112. • an independent task takes a significant
amount of time, and it must be performed
serially
• another task, independent of the first,
may be performed in parallel
• improve parallel efficiency by doing both
tasks at the same time
• one thread locks serial task
• other threads work on parallel task until
serial task is complete
Lock Example
113. Lock Example - Fortran
call omp_init_lock(mylock)
!$omp parallel
if(omp_test_lock(mylock)) then
call long_serial_task
call omp_unset_lock(mylock)
else
dowhile(.not. omp_test_lock(mylock))
call short_parallel_task
enddo
call omp_unset_lock(mylock)
endif
!$omp end parallel
call omp_destroy_lock(mylock)
114. Lock Example C
omp_init_lock(mylock);
#pragma omp parallel
if(omp_test_lock(mylock)){
long_serial_task();
omp_unset_lock(mylock);
}else{
while(! omp_test_lock(mylock)){
short_parallel_task();
}
omp_unset_lock(mylock);
}
#pragma omp end parallel omp_destroy_lock(mylock);
115. • Please fill out evaluation survey for this
tutorial at
http://scv.bu.edu/survey/tutorial_evaluation.html
Survey