SlideShare a Scribd company logo
SPONSORED BY
HPC & GPU Supercomputing Groups

   Non-profit, free to join groups hosted on www.meetup.com

   A group for the application of cutting-edge HPC & GPU
    supercomputing technology to cutting-edge business
    problems.

   Started January 2011 with New York group and reached
    today 1000 members with all groups from Boston, Silicon
    Valley, Chicago, New
    Mexico, Denver, Seattle, Austin, Washington D.C., South
    Florida, Tokyo

   Please visit
    www.SupercomputingGroup.com
    for South Florida group.
Красноармейский проспект 25, 19 ноября 2011
                          November 19, 2011
Many thanks to Andrew Sheppard for providing
        supporting content for this presentation.

Andrew is the organizer of New York meetup group and he is a
financial consultant with extensive experience in quantitative
financial analysis, trading-desk software development, and
technical management. Andrew is also the author of the
forthcoming book "Programming GPUs‖, to be published by
O’Reilly (www.oreilly.com).
―Thinking in Parallel‖
is the term for making the conceptual leap that takes
a developer from writing programs that run on
hardware with little real parallelism, to writing
programs that execute efficiently on massively
parallel   hardware,      with    100’s   and    1000’s   of
cores,     leading   to    very    substantial    speedups
(x10, x100 and beyond).
―[A]nd in this precious phial is the power to think twice
   as fast, move twice as quickly, do twice as much
   work in a given time as you could otherwise do.‖

—H. G. Wells, ―The New Accelerator‖ (1901)
    Serial programs are traditional (most of the programs are
     serial), sequential (just sequence of tasks) and relatively easy
     to think the flow.

    For example : Prefix Sum ( Scan )
         5            1               8            11

    where binary associative operator       is summation.
         5          5+1           5+1+8       5 + 1 + 8 + 11




         5            6              14            25


    data[] = {5, 1, 8, 11, 4}
     forall i from 1 to n do
          data[i] = data[i – 1] + data[i]
   But sequential thinking is about to change, because the
    serial performance improvement has slowed down from
    50% to %20 since 2002 and we can not expect huge
    improvement in serial performance anymore.
    Therefore, programming is going parallel.
Multi- and many-core computing
hitting the mainstream
   Today we have 4-12 cores, in few years 32 cores and Intel is
    predicting in 2015 we will have 100 cores.

       AMD Opteron (12)                   IBM Power 7 (8)        Intel Xeon (12)
       Sun UltraSparc T3 (16)             Cell (9)
       NVIDIA GeForce (1024)
       Adapteva (4096)                    Tilera Tile-Gx (100)

   There is lots of effort towards developing good runtime
    compilers, debuggers and OS support:

       MS TPL, Intel TBB, MPI, PVM, pthreads, PLINQ, OpenMP, MS Concurrency
        Runtime, MS Dryad, MS C++ AMP, NVIDIA CUDA C, ATI
        APP, OpenCL, Microsoft DirectCompute, Brooks, Shaders, PGI CUDA
        Fortran, GPU.Net, HMPP, Thrust etc.

       More than one hundred parallel programming languages in 2008
        ( http://perilsofparallel.blogspot.com/2008/09/101-parallel-languages-part-
        1.html or http://tinyurl.com/3p4a8to )
What are some problems
moving into a multi-core world?
   A lot of companies have a huge code base developed
    with little or no parallelism. Converting those great product
    to multi-core will take time.

   We haven’t been teaching much about parallelism for
    many years. Most students we educated in the last 10
    years know very little about parallelism.

   Engineers need to understand parallelism, understand all
    the issues of parallelism, to utilize all these cores.

   Parallel thinking is not the latest API, library or hardware.
    Parallel thinking is a set of core ideas we have to identify
    and teach our students and workforce.
             Writing good serial software was hard, writing good
              parallel software is harder: require new tools, new
              techniques, and a new ―Thinking in Parallel‖ mindset.
Performance




                                                                                  Competitive
                                                                                  Advantage


                                                            Serial Applications




                                                                      Time
                              2004 – Multi-core is on desktop
   Parallel Prefix Sum




               1                              1
Parallel Prefix Sum (Scan) with CUDA (NVIDIA) 6
(@ http://tinyurl.com/3s9as2j)
Where to start?
Concurrency                        Parallelism

Programming    issue                       Property of the machine
Single processor                           Multi-processor
Goal : running multiple                    Goal : speedup
interleaved threads                         Threads are executed
Only one thread executes at                 simultaneously
any given time


                                  Time >>>
 Time >>>




            Task A     Task B                 Task A      Task B
            Thread 1                          Thread 1    Thread 2
                       Thread 2               Thread 1    Thread 2
            Thread 1                          Thread 1    Thread 2
                       Thread 2
            Thread 1
                       Thread 2
Flynn’s Taxonomy of
Architectures



                                   Single Instruction/
           Single Instruction/     Multiple Data
           Single Data




           Multiple Instruction/   Multiple Instruction/
          Single Data              Multiple Data
SISD vs. SIMD
Parallel Programming
Methodology

 Proceed     Measure   Test




 Analyze     Design    Code
Analyzing Parallelism
   Amdahl’s Law helps to predict the theoretical
    maximum speedup on fixed problem size:




   Gustafson's Law proposes that larger problems
    can be solved by scaling the parallel computing
    power :
                                 S = rs + n . rp
Amdahl’s Law
Gustafson’s Law
Design Patterns

       Finding Concurrency Design Space



        Algorithm Structure Design Space



       Supporting Structures Design Space


       Implementation Mechanism Design
                       Space
Finding Concurrency Design Space
    Decomposition                     Dependency Analysis
  Task Decomposition                         Group Task
                                                                              Design Evaluation
  Data Decomposition                         Order Task

Data-Flow Decomposition                      Data Sharing




                            Algorithm Structure Design Space
  Organize by Tasks                Organize by Data Decomp.                Organize by Flow of Data
    Task Parallelism                Geometric Decomposition                        Pipeline

  Divide and Conquer                      Recursive Data                   Event-Based Coordination




                           Supporting Structures Design Space
           Program Structures                                         Data Structures

     SPMD                 Loop Parallelism                  Shared Data            Distributed Array

 Master / Worker            Fork / Join                     Shared Queue




                            Implementation Mechanism Design Space
8 Rules for ―Thinking in
Parallel‖
1.   Identify truly independent computations.
2.   Implement parallelism at the highest level possible.
3.   Plan early for scalability to take advantage of increasing numbers
     of cores.
4.   Hide parallelization in libraries.
5.   Use the right parallel programming model.
6.   Never assume a particular order of execution.
7.   Use non-shared storage whenever possible.
8.   Dare to change the algorithm for a better chance of parallelism.
9.   Be creative and pragmatic.
Pragmatic Parallelization
   Programming, in practice, is pragmatic.

   Most people prefer a practical ―good enough‖
    solution over an ―ideal‖ solution.




       Chaotic          Pragmatic           Bureaucratic


                      Importance of Rules
Parallel Programming Support
   CPU                            GPU
      MS TPL                          NVIDIA CUDA C
      Intel TBB                       ATI APP
      MPI                             OpenCL
      PVM                             Microsoft DirectCompute
      pthreads                        Brooks
      PLINQ                           Shaders
      OpenMP                          PGI CUDA Fortran
      MS Concurrency Runtime          GPU.Net
      MS Dryad                        HMPP
      MS C++ AMP                      Thrust
    etc.                            etc.
Links and References
   Patterns for Parallel Programming. Mattson, Timothy G.; Sanders,
    Beverly A.; Massingill, Berna L. (2004-09-15). Addison-Wesley
    Professional.

   An Introduction to Parallel Programming. Pacheco, Peter (2011-01-
    07). Morgan Kaufmann.

   The Art of Concurrency . Breshears, Clay (2009-05-07). O'Reilly
    Media.

   Wikipedia

   http://newsroom.intel.com/community/intel_newsroom/blog/2011/0
    9/15/the-future-accelerated-multi-core-goes-mainstream-
    computing-pushed-to-extremes

   http://perilsofparallel.blogspot.com/2011/09/conversation-with-
    intels-james-reinders.html
Q&A

More Related Content

What's hot

Massively Parallel K-Nearest Neighbor Computation on Distributed Architectures
Massively Parallel K-Nearest Neighbor Computation on Distributed Architectures Massively Parallel K-Nearest Neighbor Computation on Distributed Architectures
Massively Parallel K-Nearest Neighbor Computation on Distributed Architectures
Intel® Software
 
Deep Learning with Microsoft R Open
Deep Learning with Microsoft R OpenDeep Learning with Microsoft R Open
Deep Learning with Microsoft R Open
Poo Kuan Hoong
 
Synthetic dialogue generation with Deep Learning
Synthetic dialogue generation with Deep LearningSynthetic dialogue generation with Deep Learning
Synthetic dialogue generation with Deep Learning
S N
 
Deploying deep learning models with Docker and Kubernetes
Deploying deep learning models with Docker and KubernetesDeploying deep learning models with Docker and Kubernetes
Deploying deep learning models with Docker and Kubernetes
PetteriTeikariPhD
 
"New Dataflow Architecture for Machine Learning," a Presentation from Wave Co...
"New Dataflow Architecture for Machine Learning," a Presentation from Wave Co..."New Dataflow Architecture for Machine Learning," a Presentation from Wave Co...
"New Dataflow Architecture for Machine Learning," a Presentation from Wave Co...
Edge AI and Vision Alliance
 
Amazon Deep Learning
Amazon Deep LearningAmazon Deep Learning
Amazon Deep Learning
Amanda Mackay (she/her)
 
Distributed Deep Learning on AWS with Apache MXNet
Distributed Deep Learning on AWS with Apache MXNetDistributed Deep Learning on AWS with Apache MXNet
Distributed Deep Learning on AWS with Apache MXNet
Amazon Web Services
 
Josh Patterson, Advisor, Skymind – Deep learning for Industry at MLconf ATL 2016
Josh Patterson, Advisor, Skymind – Deep learning for Industry at MLconf ATL 2016Josh Patterson, Advisor, Skymind – Deep learning for Industry at MLconf ATL 2016
Josh Patterson, Advisor, Skymind – Deep learning for Industry at MLconf ATL 2016
MLconf
 
Deep learning on mobile - 2019 Practitioner's Guide
Deep learning on mobile - 2019 Practitioner's GuideDeep learning on mobile - 2019 Practitioner's Guide
Deep learning on mobile - 2019 Practitioner's Guide
Anirudh Koul
 
Towards Machine Comprehension of Spoken Content
Towards Machine Comprehension of Spoken ContentTowards Machine Comprehension of Spoken Content
Towards Machine Comprehension of Spoken Content
NVIDIA Taiwan
 
TensorFlow
TensorFlowTensorFlow
TensorFlow
Sang-Houn Choi
 
[246]reasoning, attention and memory toward differentiable reasoning machines
[246]reasoning, attention and memory   toward differentiable reasoning machines[246]reasoning, attention and memory   toward differentiable reasoning machines
[246]reasoning, attention and memory toward differentiable reasoning machines
NAVER D2
 
"Efficient Implementation of Convolutional Neural Networks using OpenCL on FP...
"Efficient Implementation of Convolutional Neural Networks using OpenCL on FP..."Efficient Implementation of Convolutional Neural Networks using OpenCL on FP...
"Efficient Implementation of Convolutional Neural Networks using OpenCL on FP...
Edge AI and Vision Alliance
 
On-device machine learning: TensorFlow on Android
On-device machine learning: TensorFlow on AndroidOn-device machine learning: TensorFlow on Android
On-device machine learning: TensorFlow on Android
Yufeng Guo
 
Deep Learning on Qubole Data Platform
Deep Learning on Qubole Data PlatformDeep Learning on Qubole Data Platform
Deep Learning on Qubole Data Platform
Shivaji Dutta
 
Deep learning on mobile
Deep learning on mobileDeep learning on mobile
Deep learning on mobile
Anirudh Koul
 
Using Deep Learning to do Real-Time Scoring in Practical Applications
Using Deep Learning to do Real-Time Scoring in Practical ApplicationsUsing Deep Learning to do Real-Time Scoring in Practical Applications
Using Deep Learning to do Real-Time Scoring in Practical Applications
Greg Makowski
 
instruction of install Caffe on ubuntu
instruction of install Caffe on ubuntu instruction of install Caffe on ubuntu
instruction of install Caffe on ubuntu
Pouya Ahv
 
Serving BERT Models in Production with TorchServe
Serving BERT Models in Production with TorchServeServing BERT Models in Production with TorchServe
Serving BERT Models in Production with TorchServe
Nidhin Pattaniyil
 
[Harvard CS264] 06 - CUDA Ninja Tricks: GPU Scripting, Meta-programming & Aut...
[Harvard CS264] 06 - CUDA Ninja Tricks: GPU Scripting, Meta-programming & Aut...[Harvard CS264] 06 - CUDA Ninja Tricks: GPU Scripting, Meta-programming & Aut...
[Harvard CS264] 06 - CUDA Ninja Tricks: GPU Scripting, Meta-programming & Aut...
npinto
 

What's hot (20)

Massively Parallel K-Nearest Neighbor Computation on Distributed Architectures
Massively Parallel K-Nearest Neighbor Computation on Distributed Architectures Massively Parallel K-Nearest Neighbor Computation on Distributed Architectures
Massively Parallel K-Nearest Neighbor Computation on Distributed Architectures
 
Deep Learning with Microsoft R Open
Deep Learning with Microsoft R OpenDeep Learning with Microsoft R Open
Deep Learning with Microsoft R Open
 
Synthetic dialogue generation with Deep Learning
Synthetic dialogue generation with Deep LearningSynthetic dialogue generation with Deep Learning
Synthetic dialogue generation with Deep Learning
 
Deploying deep learning models with Docker and Kubernetes
Deploying deep learning models with Docker and KubernetesDeploying deep learning models with Docker and Kubernetes
Deploying deep learning models with Docker and Kubernetes
 
"New Dataflow Architecture for Machine Learning," a Presentation from Wave Co...
"New Dataflow Architecture for Machine Learning," a Presentation from Wave Co..."New Dataflow Architecture for Machine Learning," a Presentation from Wave Co...
"New Dataflow Architecture for Machine Learning," a Presentation from Wave Co...
 
Amazon Deep Learning
Amazon Deep LearningAmazon Deep Learning
Amazon Deep Learning
 
Distributed Deep Learning on AWS with Apache MXNet
Distributed Deep Learning on AWS with Apache MXNetDistributed Deep Learning on AWS with Apache MXNet
Distributed Deep Learning on AWS with Apache MXNet
 
Josh Patterson, Advisor, Skymind – Deep learning for Industry at MLconf ATL 2016
Josh Patterson, Advisor, Skymind – Deep learning for Industry at MLconf ATL 2016Josh Patterson, Advisor, Skymind – Deep learning for Industry at MLconf ATL 2016
Josh Patterson, Advisor, Skymind – Deep learning for Industry at MLconf ATL 2016
 
Deep learning on mobile - 2019 Practitioner's Guide
Deep learning on mobile - 2019 Practitioner's GuideDeep learning on mobile - 2019 Practitioner's Guide
Deep learning on mobile - 2019 Practitioner's Guide
 
Towards Machine Comprehension of Spoken Content
Towards Machine Comprehension of Spoken ContentTowards Machine Comprehension of Spoken Content
Towards Machine Comprehension of Spoken Content
 
TensorFlow
TensorFlowTensorFlow
TensorFlow
 
[246]reasoning, attention and memory toward differentiable reasoning machines
[246]reasoning, attention and memory   toward differentiable reasoning machines[246]reasoning, attention and memory   toward differentiable reasoning machines
[246]reasoning, attention and memory toward differentiable reasoning machines
 
"Efficient Implementation of Convolutional Neural Networks using OpenCL on FP...
"Efficient Implementation of Convolutional Neural Networks using OpenCL on FP..."Efficient Implementation of Convolutional Neural Networks using OpenCL on FP...
"Efficient Implementation of Convolutional Neural Networks using OpenCL on FP...
 
On-device machine learning: TensorFlow on Android
On-device machine learning: TensorFlow on AndroidOn-device machine learning: TensorFlow on Android
On-device machine learning: TensorFlow on Android
 
Deep Learning on Qubole Data Platform
Deep Learning on Qubole Data PlatformDeep Learning on Qubole Data Platform
Deep Learning on Qubole Data Platform
 
Deep learning on mobile
Deep learning on mobileDeep learning on mobile
Deep learning on mobile
 
Using Deep Learning to do Real-Time Scoring in Practical Applications
Using Deep Learning to do Real-Time Scoring in Practical ApplicationsUsing Deep Learning to do Real-Time Scoring in Practical Applications
Using Deep Learning to do Real-Time Scoring in Practical Applications
 
instruction of install Caffe on ubuntu
instruction of install Caffe on ubuntu instruction of install Caffe on ubuntu
instruction of install Caffe on ubuntu
 
Serving BERT Models in Production with TorchServe
Serving BERT Models in Production with TorchServeServing BERT Models in Production with TorchServe
Serving BERT Models in Production with TorchServe
 
[Harvard CS264] 06 - CUDA Ninja Tricks: GPU Scripting, Meta-programming & Aut...
[Harvard CS264] 06 - CUDA Ninja Tricks: GPU Scripting, Meta-programming & Aut...[Harvard CS264] 06 - CUDA Ninja Tricks: GPU Scripting, Meta-programming & Aut...
[Harvard CS264] 06 - CUDA Ninja Tricks: GPU Scripting, Meta-programming & Aut...
 

Viewers also liked

Алгоритмы шифрования и их применение в .Net приложениях для защиты данных.
Алгоритмы шифрования и их применение в .Net приложениях для защиты данных.Алгоритмы шифрования и их применение в .Net приложениях для защиты данных.
Алгоритмы шифрования и их применение в .Net приложениях для защиты данных.
Pavel Tsukanov
 
RESPONSIVE WEB DESIGN
RESPONSIVE WEB DESIGNRESPONSIVE WEB DESIGN
RESPONSIVE WEB DESIGN
Pavel Tsukanov
 
ЭЛЕМЕНТЫ ИСКУСТВЕННОГО ИНТЕЛЛЕКТА ПРИ ПРОГРАММИРОВАНИИ. (http://tuladev.net/e...
ЭЛЕМЕНТЫ ИСКУСТВЕННОГО ИНТЕЛЛЕКТА ПРИ ПРОГРАММИРОВАНИИ. (http://tuladev.net/e...ЭЛЕМЕНТЫ ИСКУСТВЕННОГО ИНТЕЛЛЕКТА ПРИ ПРОГРАММИРОВАНИИ. (http://tuladev.net/e...
ЭЛЕМЕНТЫ ИСКУСТВЕННОГО ИНТЕЛЛЕКТА ПРИ ПРОГРАММИРОВАНИИ. (http://tuladev.net/e...
Pavel Tsukanov
 
KNOCKOUTJS КАК РЕАЛИЗАЦИЯ MVVM
KNOCKOUTJS КАК РЕАЛИЗАЦИЯ MVVMKNOCKOUTJS КАК РЕАЛИЗАЦИЯ MVVM
KNOCKOUTJS КАК РЕАЛИЗАЦИЯ MVVM
Pavel Tsukanov
 
Основы "мобильной" разработки на примере платформы iOs (iPhone)
Основы "мобильной" разработки на примере платформы iOs (iPhone)Основы "мобильной" разработки на примере платформы iOs (iPhone)
Основы "мобильной" разработки на примере платформы iOs (iPhone)
Pavel Tsukanov
 
SIGNALR - ОБМЕН СООБЩЕНИЯМИ В РЕАЛЬНОМ ВРЕМЕНИ
SIGNALR - ОБМЕН СООБЩЕНИЯМИ В РЕАЛЬНОМ ВРЕМЕНИSIGNALR - ОБМЕН СООБЩЕНИЯМИ В РЕАЛЬНОМ ВРЕМЕНИ
SIGNALR - ОБМЕН СООБЩЕНИЯМИ В РЕАЛЬНОМ ВРЕМЕНИ
Pavel Tsukanov
 
Sql azure federations
Sql azure federations Sql azure federations
Sql azure federations Pavel Tsukanov
 
Введение в Knockout
Введение в Knockout Введение в Knockout
Введение в Knockout
Pavel Tsukanov
 
СИ++ УМЕР. ДА ЗДРАВСТВУЕТ СИ++
СИ++ УМЕР. ДА ЗДРАВСТВУЕТ СИ++СИ++ УМЕР. ДА ЗДРАВСТВУЕТ СИ++
СИ++ УМЕР. ДА ЗДРАВСТВУЕТ СИ++
Pavel Tsukanov
 
РАЗРАБОТКА ПО С ИСПОЛЬЗОВАНИЕМ FINITE STATE MACHINE.
РАЗРАБОТКА ПО С ИСПОЛЬЗОВАНИЕМ FINITE STATE MACHINE.РАЗРАБОТКА ПО С ИСПОЛЬЗОВАНИЕМ FINITE STATE MACHINE.
РАЗРАБОТКА ПО С ИСПОЛЬЗОВАНИЕМ FINITE STATE MACHINE.
Pavel Tsukanov
 
РАЗРАБОТКА МОБИЛЬНЫХ САЙТОВ
РАЗРАБОТКА МОБИЛЬНЫХ САЙТОВРАЗРАБОТКА МОБИЛЬНЫХ САЙТОВ
РАЗРАБОТКА МОБИЛЬНЫХ САЙТОВPavel Tsukanov
 
Unit tests
Unit testsUnit tests
Unit tests
Pavel Tsukanov
 
Domain-Driven Design: Модель вместо требований
Domain-Driven Design: Модель вместо требованийDomain-Driven Design: Модель вместо требований
Domain-Driven Design: Модель вместо требований
CUSTIS
 
TDD (Test-driven Development) как стиль разработки.
TDD (Test-driven Development) как стиль разработки.TDD (Test-driven Development) как стиль разработки.
TDD (Test-driven Development) как стиль разработки.
Pavel Tsukanov
 
Автоматизированное тестирование UI на C# + Selenium WebDriver
Автоматизированное тестирование UI на C# + Selenium WebDriverАвтоматизированное тестирование UI на C# + Selenium WebDriver
Автоматизированное тестирование UI на C# + Selenium WebDriver
Pavel Tsukanov
 
Реализация REST и SOAP сервисов с помощью WCF
Реализация REST и SOAP сервисов с помощью WCFРеализация REST и SOAP сервисов с помощью WCF
Реализация REST и SOAP сервисов с помощью WCF
Pavel Tsukanov
 
Domain Driven Design
Domain Driven DesignDomain Driven Design
Domain Driven Design
Pavel Tsukanov
 
Ruby - или зачем мне еще один язык программирования?
Ruby - или зачем мне еще один язык программирования?Ruby - или зачем мне еще один язык программирования?
Ruby - или зачем мне еще один язык программирования?
Pavel Tsukanov
 
Лекция Android
Лекция AndroidЛекция Android
Лекция Android
Pavel Tsukanov
 

Viewers also liked (20)

Алгоритмы шифрования и их применение в .Net приложениях для защиты данных.
Алгоритмы шифрования и их применение в .Net приложениях для защиты данных.Алгоритмы шифрования и их применение в .Net приложениях для защиты данных.
Алгоритмы шифрования и их применение в .Net приложениях для защиты данных.
 
RESPONSIVE WEB DESIGN
RESPONSIVE WEB DESIGNRESPONSIVE WEB DESIGN
RESPONSIVE WEB DESIGN
 
ЭЛЕМЕНТЫ ИСКУСТВЕННОГО ИНТЕЛЛЕКТА ПРИ ПРОГРАММИРОВАНИИ. (http://tuladev.net/e...
ЭЛЕМЕНТЫ ИСКУСТВЕННОГО ИНТЕЛЛЕКТА ПРИ ПРОГРАММИРОВАНИИ. (http://tuladev.net/e...ЭЛЕМЕНТЫ ИСКУСТВЕННОГО ИНТЕЛЛЕКТА ПРИ ПРОГРАММИРОВАНИИ. (http://tuladev.net/e...
ЭЛЕМЕНТЫ ИСКУСТВЕННОГО ИНТЕЛЛЕКТА ПРИ ПРОГРАММИРОВАНИИ. (http://tuladev.net/e...
 
KNOCKOUTJS КАК РЕАЛИЗАЦИЯ MVVM
KNOCKOUTJS КАК РЕАЛИЗАЦИЯ MVVMKNOCKOUTJS КАК РЕАЛИЗАЦИЯ MVVM
KNOCKOUTJS КАК РЕАЛИЗАЦИЯ MVVM
 
Основы "мобильной" разработки на примере платформы iOs (iPhone)
Основы "мобильной" разработки на примере платформы iOs (iPhone)Основы "мобильной" разработки на примере платформы iOs (iPhone)
Основы "мобильной" разработки на примере платформы iOs (iPhone)
 
SIGNALR - ОБМЕН СООБЩЕНИЯМИ В РЕАЛЬНОМ ВРЕМЕНИ
SIGNALR - ОБМЕН СООБЩЕНИЯМИ В РЕАЛЬНОМ ВРЕМЕНИSIGNALR - ОБМЕН СООБЩЕНИЯМИ В РЕАЛЬНОМ ВРЕМЕНИ
SIGNALR - ОБМЕН СООБЩЕНИЯМИ В РЕАЛЬНОМ ВРЕМЕНИ
 
Sql azure federations
Sql azure federations Sql azure federations
Sql azure federations
 
Введение в Knockout
Введение в Knockout Введение в Knockout
Введение в Knockout
 
СИ++ УМЕР. ДА ЗДРАВСТВУЕТ СИ++
СИ++ УМЕР. ДА ЗДРАВСТВУЕТ СИ++СИ++ УМЕР. ДА ЗДРАВСТВУЕТ СИ++
СИ++ УМЕР. ДА ЗДРАВСТВУЕТ СИ++
 
РАЗРАБОТКА ПО С ИСПОЛЬЗОВАНИЕМ FINITE STATE MACHINE.
РАЗРАБОТКА ПО С ИСПОЛЬЗОВАНИЕМ FINITE STATE MACHINE.РАЗРАБОТКА ПО С ИСПОЛЬЗОВАНИЕМ FINITE STATE MACHINE.
РАЗРАБОТКА ПО С ИСПОЛЬЗОВАНИЕМ FINITE STATE MACHINE.
 
РАЗРАБОТКА МОБИЛЬНЫХ САЙТОВ
РАЗРАБОТКА МОБИЛЬНЫХ САЙТОВРАЗРАБОТКА МОБИЛЬНЫХ САЙТОВ
РАЗРАБОТКА МОБИЛЬНЫХ САЙТОВ
 
Unit tests
Unit testsUnit tests
Unit tests
 
Domain-Driven Design: Модель вместо требований
Domain-Driven Design: Модель вместо требованийDomain-Driven Design: Модель вместо требований
Domain-Driven Design: Модель вместо требований
 
TDD (Test-driven Development) как стиль разработки.
TDD (Test-driven Development) как стиль разработки.TDD (Test-driven Development) как стиль разработки.
TDD (Test-driven Development) как стиль разработки.
 
PaaS и SaaS
PaaS и SaaSPaaS и SaaS
PaaS и SaaS
 
Автоматизированное тестирование UI на C# + Selenium WebDriver
Автоматизированное тестирование UI на C# + Selenium WebDriverАвтоматизированное тестирование UI на C# + Selenium WebDriver
Автоматизированное тестирование UI на C# + Selenium WebDriver
 
Реализация REST и SOAP сервисов с помощью WCF
Реализация REST и SOAP сервисов с помощью WCFРеализация REST и SOAP сервисов с помощью WCF
Реализация REST и SOAP сервисов с помощью WCF
 
Domain Driven Design
Domain Driven DesignDomain Driven Design
Domain Driven Design
 
Ruby - или зачем мне еще один язык программирования?
Ruby - или зачем мне еще один язык программирования?Ruby - или зачем мне еще один язык программирования?
Ruby - или зачем мне еще один язык программирования?
 
Лекция Android
Лекция AndroidЛекция Android
Лекция Android
 

Similar to Thinking in parallel ab tuladev

Azure and cloud design patterns
Azure and cloud design patternsAzure and cloud design patterns
Azure and cloud design patterns
Venkatesh Narayanan
 
Cloud Computing Was Built for Web Developers—What Does v2 Look Like for Deep...
 Cloud Computing Was Built for Web Developers—What Does v2 Look Like for Deep... Cloud Computing Was Built for Web Developers—What Does v2 Look Like for Deep...
Cloud Computing Was Built for Web Developers—What Does v2 Look Like for Deep...
Databricks
 
A look under the hood at Apache Spark's API and engine evolutions
A look under the hood at Apache Spark's API and engine evolutionsA look under the hood at Apache Spark's API and engine evolutions
A look under the hood at Apache Spark's API and engine evolutions
Databricks
 
Building and deploying LLM applications with Apache Airflow
Building and deploying LLM applications with Apache AirflowBuilding and deploying LLM applications with Apache Airflow
Building and deploying LLM applications with Apache Airflow
Kaxil Naik
 
Distributed Deep Learning + others for Spark Meetup
Distributed Deep Learning + others for Spark MeetupDistributed Deep Learning + others for Spark Meetup
Distributed Deep Learning + others for Spark Meetup
Vijay Srinivas Agneeswaran, Ph.D
 
Presentation
PresentationPresentation
Presentationbutest
 
Essential Data Engineering for Data Scientist
Essential Data Engineering for Data Scientist Essential Data Engineering for Data Scientist
Essential Data Engineering for Data Scientist
SoftServe
 
"Big Data" Bioinformatics
"Big Data" Bioinformatics"Big Data" Bioinformatics
"Big Data" Bioinformatics
Brian Repko
 
Overview Of Parallel Development - Ericnel
Overview Of Parallel Development -  EricnelOverview Of Parallel Development -  Ericnel
Overview Of Parallel Development - Ericnel
ukdpe
 
01 introduction fundamentals_of_parallelism_and_code_optimization-www.astek.ir
01 introduction fundamentals_of_parallelism_and_code_optimization-www.astek.ir01 introduction fundamentals_of_parallelism_and_code_optimization-www.astek.ir
01 introduction fundamentals_of_parallelism_and_code_optimization-www.astek.ir
aminnezarat
 
Floating Point Operations , Memory Chip Organization , Serial Bus Architectur...
Floating Point Operations , Memory Chip Organization , Serial Bus Architectur...Floating Point Operations , Memory Chip Organization , Serial Bus Architectur...
Floating Point Operations , Memory Chip Organization , Serial Bus Architectur...
VAISHNAVI MADHAN
 
Floating Point Operations , Memory Chip Organization , Serial Bus Architectur...
Floating Point Operations , Memory Chip Organization , Serial Bus Architectur...Floating Point Operations , Memory Chip Organization , Serial Bus Architectur...
Floating Point Operations , Memory Chip Organization , Serial Bus Architectur...
KRamasamy2
 
CIKB - Software Architecture Analysis Design
CIKB - Software Architecture Analysis DesignCIKB - Software Architecture Analysis Design
CIKB - Software Architecture Analysis DesignAntonio Castellon
 
Distributed deep learning_over_spark_20_nov_2014_ver_2.8
Distributed deep learning_over_spark_20_nov_2014_ver_2.8Distributed deep learning_over_spark_20_nov_2014_ver_2.8
Distributed deep learning_over_spark_20_nov_2014_ver_2.8
Vijay Srinivas Agneeswaran, Ph.D
 
The Future of Computing is Distributed
The Future of Computing is DistributedThe Future of Computing is Distributed
The Future of Computing is Distributed
Alluxio, Inc.
 
Architecting Solutions for the Manycore Future
Architecting Solutions for the Manycore FutureArchitecting Solutions for the Manycore Future
Architecting Solutions for the Manycore Future
Talbott Crowell
 
Parallel computation
Parallel computationParallel computation
Parallel computation
Jayanti Prasad Ph.D.
 
Towards high performance computing(hpc) through parallel programming paradigm...
Towards high performance computing(hpc) through parallel programming paradigm...Towards high performance computing(hpc) through parallel programming paradigm...
Towards high performance computing(hpc) through parallel programming paradigm...
ijpla
 
A Survey on in-a-box parallel computing and its implications on system softwa...
A Survey on in-a-box parallel computing and its implications on system softwa...A Survey on in-a-box parallel computing and its implications on system softwa...
A Survey on in-a-box parallel computing and its implications on system softwa...ChangWoo Min
 

Similar to Thinking in parallel ab tuladev (20)

Azure and cloud design patterns
Azure and cloud design patternsAzure and cloud design patterns
Azure and cloud design patterns
 
Cloud Computing Was Built for Web Developers—What Does v2 Look Like for Deep...
 Cloud Computing Was Built for Web Developers—What Does v2 Look Like for Deep... Cloud Computing Was Built for Web Developers—What Does v2 Look Like for Deep...
Cloud Computing Was Built for Web Developers—What Does v2 Look Like for Deep...
 
A look under the hood at Apache Spark's API and engine evolutions
A look under the hood at Apache Spark's API and engine evolutionsA look under the hood at Apache Spark's API and engine evolutions
A look under the hood at Apache Spark's API and engine evolutions
 
Building and deploying LLM applications with Apache Airflow
Building and deploying LLM applications with Apache AirflowBuilding and deploying LLM applications with Apache Airflow
Building and deploying LLM applications with Apache Airflow
 
Distributed Deep Learning + others for Spark Meetup
Distributed Deep Learning + others for Spark MeetupDistributed Deep Learning + others for Spark Meetup
Distributed Deep Learning + others for Spark Meetup
 
Presentation
PresentationPresentation
Presentation
 
Essential Data Engineering for Data Scientist
Essential Data Engineering for Data Scientist Essential Data Engineering for Data Scientist
Essential Data Engineering for Data Scientist
 
"Big Data" Bioinformatics
"Big Data" Bioinformatics"Big Data" Bioinformatics
"Big Data" Bioinformatics
 
Overview Of Parallel Development - Ericnel
Overview Of Parallel Development -  EricnelOverview Of Parallel Development -  Ericnel
Overview Of Parallel Development - Ericnel
 
01 introduction fundamentals_of_parallelism_and_code_optimization-www.astek.ir
01 introduction fundamentals_of_parallelism_and_code_optimization-www.astek.ir01 introduction fundamentals_of_parallelism_and_code_optimization-www.astek.ir
01 introduction fundamentals_of_parallelism_and_code_optimization-www.astek.ir
 
Floating Point Operations , Memory Chip Organization , Serial Bus Architectur...
Floating Point Operations , Memory Chip Organization , Serial Bus Architectur...Floating Point Operations , Memory Chip Organization , Serial Bus Architectur...
Floating Point Operations , Memory Chip Organization , Serial Bus Architectur...
 
Floating Point Operations , Memory Chip Organization , Serial Bus Architectur...
Floating Point Operations , Memory Chip Organization , Serial Bus Architectur...Floating Point Operations , Memory Chip Organization , Serial Bus Architectur...
Floating Point Operations , Memory Chip Organization , Serial Bus Architectur...
 
CIKB - Software Architecture Analysis Design
CIKB - Software Architecture Analysis DesignCIKB - Software Architecture Analysis Design
CIKB - Software Architecture Analysis Design
 
Distributed deep learning_over_spark_20_nov_2014_ver_2.8
Distributed deep learning_over_spark_20_nov_2014_ver_2.8Distributed deep learning_over_spark_20_nov_2014_ver_2.8
Distributed deep learning_over_spark_20_nov_2014_ver_2.8
 
The Future of Computing is Distributed
The Future of Computing is DistributedThe Future of Computing is Distributed
The Future of Computing is Distributed
 
Os Lamothe
Os LamotheOs Lamothe
Os Lamothe
 
Architecting Solutions for the Manycore Future
Architecting Solutions for the Manycore FutureArchitecting Solutions for the Manycore Future
Architecting Solutions for the Manycore Future
 
Parallel computation
Parallel computationParallel computation
Parallel computation
 
Towards high performance computing(hpc) through parallel programming paradigm...
Towards high performance computing(hpc) through parallel programming paradigm...Towards high performance computing(hpc) through parallel programming paradigm...
Towards high performance computing(hpc) through parallel programming paradigm...
 
A Survey on in-a-box parallel computing and its implications on system softwa...
A Survey on in-a-box parallel computing and its implications on system softwa...A Survey on in-a-box parallel computing and its implications on system softwa...
A Survey on in-a-box parallel computing and its implications on system softwa...
 

More from Pavel Tsukanov

МАШИННОЕ ЗРЕНИЕ С ИСПОЛЬЗОВАНИЕ OPENCV
МАШИННОЕ ЗРЕНИЕ С ИСПОЛЬЗОВАНИЕ OPENCVМАШИННОЕ ЗРЕНИЕ С ИСПОЛЬЗОВАНИЕ OPENCV
МАШИННОЕ ЗРЕНИЕ С ИСПОЛЬЗОВАНИЕ OPENCV
Pavel Tsukanov
 
CONTINUOUS INTEGRATION ДЛЯ ЧАЙНИКОВ ВМЕСТЕ С TEAMCITY
CONTINUOUS INTEGRATION ДЛЯ ЧАЙНИКОВ ВМЕСТЕ С TEAMCITYCONTINUOUS INTEGRATION ДЛЯ ЧАЙНИКОВ ВМЕСТЕ С TEAMCITY
CONTINUOUS INTEGRATION ДЛЯ ЧАЙНИКОВ ВМЕСТЕ С TEAMCITY
Pavel Tsukanov
 
СОЗДАЙ РОБОТА С НУЛЯ
СОЗДАЙ РОБОТА С НУЛЯСОЗДАЙ РОБОТА С НУЛЯ
СОЗДАЙ РОБОТА С НУЛЯ
Pavel Tsukanov
 
ВВЕДЕНИЕ В NODE.JS
ВВЕДЕНИЕ В NODE.JS ВВЕДЕНИЕ В NODE.JS
ВВЕДЕНИЕ В NODE.JS
Pavel Tsukanov
 
АНИМАЦИЯ В FLASH И HTML5
АНИМАЦИЯ В FLASH И HTML5АНИМАЦИЯ В FLASH И HTML5
АНИМАЦИЯ В FLASH И HTML5
Pavel Tsukanov
 
ХАКЕРЫ И АНТИХАКЕРЫ
ХАКЕРЫ И АНТИХАКЕРЫХАКЕРЫ И АНТИХАКЕРЫ
ХАКЕРЫ И АНТИХАКЕРЫ
Pavel Tsukanov
 
ЗАРАБОТОК В ИНТЕРНЕТЕ.
ЗАРАБОТОК В ИНТЕРНЕТЕ.ЗАРАБОТОК В ИНТЕРНЕТЕ.
ЗАРАБОТОК В ИНТЕРНЕТЕ.
Pavel Tsukanov
 
ORM технологии в .NET (Nhibernate, Linq To SQL, Entity Framework)
ORM технологии в .NET (Nhibernate, Linq To SQL, Entity Framework)ORM технологии в .NET (Nhibernate, Linq To SQL, Entity Framework)
ORM технологии в .NET (Nhibernate, Linq To SQL, Entity Framework)
Pavel Tsukanov
 
Как писать красивый код или основы SOLID
Как писать красивый код или основы SOLIDКак писать красивый код или основы SOLID
Как писать красивый код или основы SOLID
Pavel Tsukanov
 
Статический анализ кода
Статический анализ кода Статический анализ кода
Статический анализ кода
Pavel Tsukanov
 
Применение нейронных сетей и генетических алгоритмов при торговле на бирже.
Применение нейронных сетей и генетических алгоритмов при торговле на бирже. Применение нейронных сетей и генетических алгоритмов при торговле на бирже.
Применение нейронных сетей и генетических алгоритмов при торговле на бирже.
Pavel Tsukanov
 

More from Pavel Tsukanov (11)

МАШИННОЕ ЗРЕНИЕ С ИСПОЛЬЗОВАНИЕ OPENCV
МАШИННОЕ ЗРЕНИЕ С ИСПОЛЬЗОВАНИЕ OPENCVМАШИННОЕ ЗРЕНИЕ С ИСПОЛЬЗОВАНИЕ OPENCV
МАШИННОЕ ЗРЕНИЕ С ИСПОЛЬЗОВАНИЕ OPENCV
 
CONTINUOUS INTEGRATION ДЛЯ ЧАЙНИКОВ ВМЕСТЕ С TEAMCITY
CONTINUOUS INTEGRATION ДЛЯ ЧАЙНИКОВ ВМЕСТЕ С TEAMCITYCONTINUOUS INTEGRATION ДЛЯ ЧАЙНИКОВ ВМЕСТЕ С TEAMCITY
CONTINUOUS INTEGRATION ДЛЯ ЧАЙНИКОВ ВМЕСТЕ С TEAMCITY
 
СОЗДАЙ РОБОТА С НУЛЯ
СОЗДАЙ РОБОТА С НУЛЯСОЗДАЙ РОБОТА С НУЛЯ
СОЗДАЙ РОБОТА С НУЛЯ
 
ВВЕДЕНИЕ В NODE.JS
ВВЕДЕНИЕ В NODE.JS ВВЕДЕНИЕ В NODE.JS
ВВЕДЕНИЕ В NODE.JS
 
АНИМАЦИЯ В FLASH И HTML5
АНИМАЦИЯ В FLASH И HTML5АНИМАЦИЯ В FLASH И HTML5
АНИМАЦИЯ В FLASH И HTML5
 
ХАКЕРЫ И АНТИХАКЕРЫ
ХАКЕРЫ И АНТИХАКЕРЫХАКЕРЫ И АНТИХАКЕРЫ
ХАКЕРЫ И АНТИХАКЕРЫ
 
ЗАРАБОТОК В ИНТЕРНЕТЕ.
ЗАРАБОТОК В ИНТЕРНЕТЕ.ЗАРАБОТОК В ИНТЕРНЕТЕ.
ЗАРАБОТОК В ИНТЕРНЕТЕ.
 
ORM технологии в .NET (Nhibernate, Linq To SQL, Entity Framework)
ORM технологии в .NET (Nhibernate, Linq To SQL, Entity Framework)ORM технологии в .NET (Nhibernate, Linq To SQL, Entity Framework)
ORM технологии в .NET (Nhibernate, Linq To SQL, Entity Framework)
 
Как писать красивый код или основы SOLID
Как писать красивый код или основы SOLIDКак писать красивый код или основы SOLID
Как писать красивый код или основы SOLID
 
Статический анализ кода
Статический анализ кода Статический анализ кода
Статический анализ кода
 
Применение нейронных сетей и генетических алгоритмов при торговле на бирже.
Применение нейронных сетей и генетических алгоритмов при торговле на бирже. Применение нейронных сетей и генетических алгоритмов при торговле на бирже.
Применение нейронных сетей и генетических алгоритмов при торговле на бирже.
 

Recently uploaded

20240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 202420240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 2024
Matthew Sinclair
 
National Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practicesNational Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practices
Quotidiano Piemontese
 
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
Neo4j
 
GridMate - End to end testing is a critical piece to ensure quality and avoid...
GridMate - End to end testing is a critical piece to ensure quality and avoid...GridMate - End to end testing is a critical piece to ensure quality and avoid...
GridMate - End to end testing is a critical piece to ensure quality and avoid...
ThomasParaiso2
 
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
DanBrown980551
 
Climate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing DaysClimate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing Days
Kari Kakkonen
 
20240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 202420240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 2024
Matthew Sinclair
 
Essentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FMEEssentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FME
Safe Software
 
Elizabeth Buie - Older adults: Are we really designing for our future selves?
Elizabeth Buie - Older adults: Are we really designing for our future selves?Elizabeth Buie - Older adults: Are we really designing for our future selves?
Elizabeth Buie - Older adults: Are we really designing for our future selves?
Nexer Digital
 
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionGenerative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Aggregage
 
Microsoft - Power Platform_G.Aspiotis.pdf
Microsoft - Power Platform_G.Aspiotis.pdfMicrosoft - Power Platform_G.Aspiotis.pdf
Microsoft - Power Platform_G.Aspiotis.pdf
Uni Systems S.M.S.A.
 
By Design, not by Accident - Agile Venture Bolzano 2024
By Design, not by Accident - Agile Venture Bolzano 2024By Design, not by Accident - Agile Venture Bolzano 2024
By Design, not by Accident - Agile Venture Bolzano 2024
Pierluigi Pugliese
 
Secstrike : Reverse Engineering & Pwnable tools for CTF.pptx
Secstrike : Reverse Engineering & Pwnable tools for CTF.pptxSecstrike : Reverse Engineering & Pwnable tools for CTF.pptx
Secstrike : Reverse Engineering & Pwnable tools for CTF.pptx
nkrafacyberclub
 
Communications Mining Series - Zero to Hero - Session 1
Communications Mining Series - Zero to Hero - Session 1Communications Mining Series - Zero to Hero - Session 1
Communications Mining Series - Zero to Hero - Session 1
DianaGray10
 
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
Neo4j
 
Mind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AIMind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AI
Kumud Singh
 
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdfObservability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
Paige Cruz
 
GraphRAG is All You need? LLM & Knowledge Graph
GraphRAG is All You need? LLM & Knowledge GraphGraphRAG is All You need? LLM & Knowledge Graph
GraphRAG is All You need? LLM & Knowledge Graph
Guy Korland
 
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
Neo4j
 
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
SOFTTECHHUB
 

Recently uploaded (20)

20240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 202420240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 2024
 
National Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practicesNational Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practices
 
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
 
GridMate - End to end testing is a critical piece to ensure quality and avoid...
GridMate - End to end testing is a critical piece to ensure quality and avoid...GridMate - End to end testing is a critical piece to ensure quality and avoid...
GridMate - End to end testing is a critical piece to ensure quality and avoid...
 
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
 
Climate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing DaysClimate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing Days
 
20240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 202420240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 2024
 
Essentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FMEEssentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FME
 
Elizabeth Buie - Older adults: Are we really designing for our future selves?
Elizabeth Buie - Older adults: Are we really designing for our future selves?Elizabeth Buie - Older adults: Are we really designing for our future selves?
Elizabeth Buie - Older adults: Are we really designing for our future selves?
 
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionGenerative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to Production
 
Microsoft - Power Platform_G.Aspiotis.pdf
Microsoft - Power Platform_G.Aspiotis.pdfMicrosoft - Power Platform_G.Aspiotis.pdf
Microsoft - Power Platform_G.Aspiotis.pdf
 
By Design, not by Accident - Agile Venture Bolzano 2024
By Design, not by Accident - Agile Venture Bolzano 2024By Design, not by Accident - Agile Venture Bolzano 2024
By Design, not by Accident - Agile Venture Bolzano 2024
 
Secstrike : Reverse Engineering & Pwnable tools for CTF.pptx
Secstrike : Reverse Engineering & Pwnable tools for CTF.pptxSecstrike : Reverse Engineering & Pwnable tools for CTF.pptx
Secstrike : Reverse Engineering & Pwnable tools for CTF.pptx
 
Communications Mining Series - Zero to Hero - Session 1
Communications Mining Series - Zero to Hero - Session 1Communications Mining Series - Zero to Hero - Session 1
Communications Mining Series - Zero to Hero - Session 1
 
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
 
Mind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AIMind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AI
 
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdfObservability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
 
GraphRAG is All You need? LLM & Knowledge Graph
GraphRAG is All You need? LLM & Knowledge GraphGraphRAG is All You need? LLM & Knowledge Graph
GraphRAG is All You need? LLM & Knowledge Graph
 
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
 
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
 

Thinking in parallel ab tuladev

  • 2. HPC & GPU Supercomputing Groups  Non-profit, free to join groups hosted on www.meetup.com  A group for the application of cutting-edge HPC & GPU supercomputing technology to cutting-edge business problems.  Started January 2011 with New York group and reached today 1000 members with all groups from Boston, Silicon Valley, Chicago, New Mexico, Denver, Seattle, Austin, Washington D.C., South Florida, Tokyo  Please visit www.SupercomputingGroup.com for South Florida group.
  • 3. Красноармейский проспект 25, 19 ноября 2011 November 19, 2011
  • 4. Many thanks to Andrew Sheppard for providing supporting content for this presentation. Andrew is the organizer of New York meetup group and he is a financial consultant with extensive experience in quantitative financial analysis, trading-desk software development, and technical management. Andrew is also the author of the forthcoming book "Programming GPUs‖, to be published by O’Reilly (www.oreilly.com).
  • 5. ―Thinking in Parallel‖ is the term for making the conceptual leap that takes a developer from writing programs that run on hardware with little real parallelism, to writing programs that execute efficiently on massively parallel hardware, with 100’s and 1000’s of cores, leading to very substantial speedups (x10, x100 and beyond).
  • 6. ―[A]nd in this precious phial is the power to think twice as fast, move twice as quickly, do twice as much work in a given time as you could otherwise do.‖ —H. G. Wells, ―The New Accelerator‖ (1901)
  • 7. Serial programs are traditional (most of the programs are serial), sequential (just sequence of tasks) and relatively easy to think the flow.  For example : Prefix Sum ( Scan ) 5 1 8 11 where binary associative operator is summation. 5 5+1 5+1+8 5 + 1 + 8 + 11 5 6 14 25  data[] = {5, 1, 8, 11, 4} forall i from 1 to n do data[i] = data[i – 1] + data[i]
  • 8. But sequential thinking is about to change, because the serial performance improvement has slowed down from 50% to %20 since 2002 and we can not expect huge improvement in serial performance anymore. Therefore, programming is going parallel.
  • 9. Multi- and many-core computing hitting the mainstream  Today we have 4-12 cores, in few years 32 cores and Intel is predicting in 2015 we will have 100 cores.  AMD Opteron (12) IBM Power 7 (8) Intel Xeon (12)  Sun UltraSparc T3 (16) Cell (9)  NVIDIA GeForce (1024)  Adapteva (4096) Tilera Tile-Gx (100)  There is lots of effort towards developing good runtime compilers, debuggers and OS support:  MS TPL, Intel TBB, MPI, PVM, pthreads, PLINQ, OpenMP, MS Concurrency Runtime, MS Dryad, MS C++ AMP, NVIDIA CUDA C, ATI APP, OpenCL, Microsoft DirectCompute, Brooks, Shaders, PGI CUDA Fortran, GPU.Net, HMPP, Thrust etc.  More than one hundred parallel programming languages in 2008 ( http://perilsofparallel.blogspot.com/2008/09/101-parallel-languages-part- 1.html or http://tinyurl.com/3p4a8to )
  • 10. What are some problems moving into a multi-core world?  A lot of companies have a huge code base developed with little or no parallelism. Converting those great product to multi-core will take time.  We haven’t been teaching much about parallelism for many years. Most students we educated in the last 10 years know very little about parallelism.  Engineers need to understand parallelism, understand all the issues of parallelism, to utilize all these cores.  Parallel thinking is not the latest API, library or hardware. Parallel thinking is a set of core ideas we have to identify and teach our students and workforce.
  • 11. Writing good serial software was hard, writing good parallel software is harder: require new tools, new techniques, and a new ―Thinking in Parallel‖ mindset. Performance Competitive Advantage Serial Applications Time 2004 – Multi-core is on desktop
  • 12. Parallel Prefix Sum 1 1 Parallel Prefix Sum (Scan) with CUDA (NVIDIA) 6 (@ http://tinyurl.com/3s9as2j)
  • 14. Concurrency Parallelism Programming issue  Property of the machine Single processor  Multi-processor Goal : running multiple  Goal : speedup interleaved threads  Threads are executed Only one thread executes at simultaneously any given time Time >>> Time >>> Task A Task B Task A Task B Thread 1 Thread 1 Thread 2 Thread 2 Thread 1 Thread 2 Thread 1 Thread 1 Thread 2 Thread 2 Thread 1 Thread 2
  • 15. Flynn’s Taxonomy of Architectures Single Instruction/ Single Instruction/ Multiple Data Single Data Multiple Instruction/ Multiple Instruction/ Single Data Multiple Data
  • 17. Parallel Programming Methodology Proceed Measure Test Analyze Design Code
  • 18. Analyzing Parallelism  Amdahl’s Law helps to predict the theoretical maximum speedup on fixed problem size:  Gustafson's Law proposes that larger problems can be solved by scaling the parallel computing power : S = rs + n . rp
  • 21. Design Patterns Finding Concurrency Design Space Algorithm Structure Design Space Supporting Structures Design Space Implementation Mechanism Design Space
  • 22. Finding Concurrency Design Space Decomposition Dependency Analysis Task Decomposition Group Task Design Evaluation Data Decomposition Order Task Data-Flow Decomposition Data Sharing Algorithm Structure Design Space Organize by Tasks Organize by Data Decomp. Organize by Flow of Data Task Parallelism Geometric Decomposition Pipeline Divide and Conquer Recursive Data Event-Based Coordination Supporting Structures Design Space Program Structures Data Structures SPMD Loop Parallelism Shared Data Distributed Array Master / Worker Fork / Join Shared Queue Implementation Mechanism Design Space
  • 23. 8 Rules for ―Thinking in Parallel‖ 1. Identify truly independent computations. 2. Implement parallelism at the highest level possible. 3. Plan early for scalability to take advantage of increasing numbers of cores. 4. Hide parallelization in libraries. 5. Use the right parallel programming model. 6. Never assume a particular order of execution. 7. Use non-shared storage whenever possible. 8. Dare to change the algorithm for a better chance of parallelism. 9. Be creative and pragmatic.
  • 24. Pragmatic Parallelization  Programming, in practice, is pragmatic.  Most people prefer a practical ―good enough‖ solution over an ―ideal‖ solution. Chaotic Pragmatic Bureaucratic Importance of Rules
  • 25. Parallel Programming Support  CPU  GPU  MS TPL  NVIDIA CUDA C  Intel TBB  ATI APP  MPI  OpenCL  PVM  Microsoft DirectCompute  pthreads  Brooks  PLINQ  Shaders  OpenMP  PGI CUDA Fortran  MS Concurrency Runtime  GPU.Net  MS Dryad  HMPP  MS C++ AMP  Thrust etc. etc.
  • 26. Links and References  Patterns for Parallel Programming. Mattson, Timothy G.; Sanders, Beverly A.; Massingill, Berna L. (2004-09-15). Addison-Wesley Professional.  An Introduction to Parallel Programming. Pacheco, Peter (2011-01- 07). Morgan Kaufmann.  The Art of Concurrency . Breshears, Clay (2009-05-07). O'Reilly Media.  Wikipedia  http://newsroom.intel.com/community/intel_newsroom/blog/2011/0 9/15/the-future-accelerated-multi-core-goes-mainstream- computing-pushed-to-extremes  http://perilsofparallel.blogspot.com/2011/09/conversation-with- intels-james-reinders.html
  • 27. Q&A

Editor's Notes

  1. Traditionally, much of computer programming has been serial in nature. A program begins at a well defined entry point and works through a sequence of tasks in succession. Designing serial programs are relatively easy because you can think sequentially for the most part. This is the simplified programming model that most programmers today have learned and use. But it’s about to change, because programming is going parallel. And given how hard it seems to be to write good serial software (judging by how many software projects struggle even in the serial world) then the new challenges of parallel programming will require new tools, new techniques, and a new “Thinking in Parallel” mindset. This short talk focuses on some of the issues relating to “Thinking in Parallel”. It’s written from the perspective of a practitioner in-the-trenches and not from the perspective of a computer scientist. Nor is it a comprehensive overview of topic. It’s merely a starting point. My hope is that even though the meetup group ranges from newbies to experts, everyone will come away with at least one idea to think about. The journey begins …
  2. Traditionally, much of computer programming has been serial in nature. A program begins at a well defined entry point and works through a sequence of tasks in succession. Designing serial programs are relatively easy because you can think sequentially for the most part. This is the simplified programming model that most programmers today have learned and use. But it’s about to change, because programming is going parallel. And given how hard it seems to be to write good serial software (judging by how many software projects struggle even in the serial world) then the new challenges of parallel programming will require new tools, new techniques, and a new “Thinking in Parallel” mindset. This short talk focuses on some of the issues relating to “Thinking in Parallel”. It’s written from the perspective of a practitioner in-the-trenches and not from the perspective of a computer scientist. Nor is it a comprehensive overview of topic. It’s merely a starting point. My hope is that even though the meetup group ranges from newbies to experts, everyone will come away with at least one idea to think about. The journey begins …
  3. Traditionally, much of computer programming has been serial in nature. A program begins at a well defined entry point and works through a sequence of tasks in succession. Designing serial programs are relatively easy because you can think sequentially for the most part. This is the simplified programming model that most programmers today have learned and use. But it’s about to change, because programming is going parallel. And given how hard it seems to be to write good serial software (judging by how many software projects struggle even in the serial world) then the new challenges of parallel programming will require new tools, new techniques, and a new “Thinking in Parallel” mindset. This short talk focuses on some of the issues relating to “Thinking in Parallel”. It’s written from the perspective of a practitioner in-the-trenches and not from the perspective of a computer scientist. Nor is it a comprehensive overview of topic. It’s merely a starting point. My hope is that even though the meetup group ranges from newbies to experts, everyone will come away with at least one idea to think about. The journey begins …A large percentage of people who provide applications are going to have to care about parallelism in order to match the capabilities of their competitors.
  4. Before we begin, let’s clear up a common misunderstanding. What’s the difference between concurrent versus parallel? They both share some common concepts and difficulties, but they are different. Concurrent execution of two or more programs (or multiple tasks within a single program) means that only a single thread executes at any given time, but switching between threads is so rapid that it appears as though all tasks proceed concurrently. Parallel execution means that two or more threads are actually running simultaneously in hardware. It is not as though the tasks appear to proceed in parallel, they really are running in parallel. The difference between can be illustrated simply as follows. [diagram]Of course, multicore CPUs, GPUs and clusters of the same are all about running in parallel. In the case of individual GPUs and large clusters of CPUs, they run in a massively parallel way.
  5. SynchronizationWhenever two or more tasks need the same resources, there is the possibility for contention. For multi-threaded applications this is often solved with mutexes, semaphores, critical sections and their like. On GPUs there are likewise synchronization objects.Race ConditionA major obstacle to efficient parallel execution is resource contention, whether that be for memory or I/O (though one could argue all data access is I/O of one form or another.) Resource contention is particularly prevalent for MISD and MIMD execution models. But the potential is always present when two executing tasks need to share the same resource which itself is not parallelizable.
  6. There are dozens of different parallel architectures, among them networks of workstations, clusters of off-the-shelf PCs, massively parallel supercomputers, tightly coupled symmetric multiprocessors, and multiprocessor workstations.Flynn’s taxonomy categorizes all computers according to the number of instruction streams and data streams they have, where a stream is a sequence of instructions or data on which a computer operatesSISD Single instruction, single data (SISD) means a single thread or core operating on a single piece of data. SIMD Single instruction, multiple data (SIMD) means the same code running on multiple threads or cores, but operating on different parts of the data set. This is the execution model for GPUs. MISD Multiple instructions, single data (MISD) means different programs running in multiple threads or cores operating on the same data. MIMD Multiple instructions, multiple data (MIMD) means different programs running in multiple threads or cores operating on different parts of the data set.
  7. Since the GOF (Gang of Four: Gamma, Helm, Johnson and Vissides) wrote “Design Patterns: Elements of Reusable Object-Oriented Software”, programming patterns have gained in popularity. They are now used in one way or another by most mainstream programmers. Not surprisingly, design patterns for parallel programming have emerged and attempt, to one degree or another, to map a problem onto an underlying execution model.
  8. Task Decomposition Task decomposition, as its name implies, breaks the problem into parts so that each can be independently assigned to a different computational resource to run in parallel. Data Decomposition Data decomposition is breaking your data set into smaller parts so that each can be processed separately by different compute resources. For data decomposition what is needed is not just a partitioning of the data, but rather a data plan that includes such things as how the data will be encoded and moved around. Both can affect performance considerably, because some encodings are more efficient and compact than others, and leaving data in-situ and doing multiple operations on it is clearly better than the converseGroup Tasks If a group shares a temporal constraint (for example, waiting on one group to finish filling a file before another group can begin reading it), we can satisfy that constraint once for the whole group. If a group of tasks must work together on a shared data structure, the required synchronization can be worked out once for the whole group. If a set of tasks are independent, combining them into a single group and scheduling them for execution as a single large group can simplify the design and increase the available concurrency (thereby letting the solution scale to more PEs).Order Tasksa way of decomposing a problem into tasks and a way of collecting these tasks into logically related groups, how must these groups of tasks be ordered to satisfy constraints among tasks?
  9. These rules are from “The Art of Concurrency” by Clay Breshears. I’ve modified them slightly so they are equally applicable to “Thinking in Parallel”:
  10. Traditional languages gain support for parallel programming through libraries that seek to hide as much of the underlying hardware and parallelism as possibleFunctional languages have, in recent times, gained in popularity because of their strong support for parallelism; even if parallelism isn’t built into the language it is often well supported through the “functional” style. Some functional languages, notably Erlang, are intrinsically parallel given that it was intended for parallel execution by design.   OtherScala, for example, is a language that supports a mix of programming models and styles, from OOP to functional. It also supports parallel programming quite well. Given the large number of languages out there, you will no doubt find a parallel programming language to your taste!