© ASERT 2006-2010




                    Groovy and Concurrency
                      Dr Paul King, @paulk_asert
        ...
Concurrent Programming in Groovy
• Java concurrent programming enhancements
  –   Normal OO methods
  –   Ability to have ...
Java Concurrency Features
• The early years
  – Threads, synchronised and non-synchronised
    collections, synchronisatio...
Topics
                    Groovy Intro
                    • Useful Groovy features for Concurrency
                    ...
What is Groovy?
                    • “Groovy is like a super version
                      of Java. It can leverage Java'...
Groovy Goodies Overview
                    • Fully object oriented
                    • Closures: reusable
             ...
Growing Acceptance …
 A slow and steady start but now gaining in
 momentum, maturity and mindshare




Now free
… Growing Acceptance …
                                What alternative JVM language are you using or intending to use
© A...
… Growing Acceptance …
© ASERT 2006-2010




                             Source: http://www.micropoll.com/akira/mpresult/...
… Growing Acceptance …
© ASERT 2006-2010




                          http://www.jroller.com/scolebourne/entry/devoxx_200...
… Growing Acceptance …
© ASERT 2006-2010




                      Groovy and Grails
                    downloads: 70-90K...
… Growing Acceptance
© ASERT 2006-2010




                                           Concurrency - 12
The Landscape of JVM Languages

                                                                                          ...
Groovy Starter
                    System.out.println("Hello, World!");   // supports Java syntax
                    prin...
A Better Java...
                    import java.util.List;
                    import java.util.ArrayList;

             ...
...A Better Java...
                    import java.util.List;
                    import java.util.ArrayList;

          ...
...A Better Java...
                    class Erase {
                        private List removeLongerThan(List strings, ...
...A Better Java...
                    class Erase {
                        private List removeLongerThan(List strings, ...
...A Better Java...
                    def removeLongerThan(strings, length) {
                        def result = new A...
...A Better Java...
                    def removeLongerThan(strings, length) {
                        def result = new A...
...A Better Java...
                    def removeLongerThan(strings, length) {
                        strings.findAll{ i...
...A Better Java...
                    def removeLongerThan(strings, length) {
                        strings.findAll{ i...
...A Better Java
                    names = ["Ted", "Fred", "Jed", "Ned"]
                    println names
             ...
Grapes / Grab: Google collections
                    @Grab('com.google.collections:google-collections:1.0')
             ...
Better Design Patterns: Delegate…
                                                              public Date getWhen() {
  ...
…Better Design Patterns: Delegate…
                                                              public Date getWhen() {
 ...
…Better Design Patterns: Delegate

                    class Event {
                        String title, url
           ...
Why Groovy? Technical Answer
•   Minimal learning curve
•   Compiles to bytecode
•   Java object model & integration
•   A...
Why Groovy? Adoption Assessment
• Innovators/Thought leaders
  – Ideas, power, flexibility, novelty, thinking community
• ...
Source: Herb Sutter: http://www.gotw.ca/publications/concurrency-ddj.htm
                                                 ...
Why is it hard?
• Many issues to deal with:
  – Doing things in parallel, concurrently,
    asynchronously
     • Processe...
Topics
                    • Groovy Intro
                    Useful Groovy features for Concurrency
                    ...
Thread Enhancements…
• Thread improvements


    Thread.start{ sleep 1001; println 'one' }
    Thread.start{ sleep 1000; p...
…Thread Enhancements…
• Thread improvements (Cont’d)
    class Storage {
        List stack = []
        synchronized void...
…Thread Enhancements…
• Thread improvements (Cont’d)    push: 0
                                  push: 1
        ...     ...
…Thread Enhancements…
• Thread improvements (Cont’d)                                              ROCK!
                  ...
… Thread Enhancements
• Thread Management meets AtomicInteger
 import java.util.concurrent.atomic.AtomicInteger

 def coun...
Process Enhancements…
• Using AntBuilder


                                         [echo] Message 6
                     ...
…Process Enhancements…
• Using AntBuilder (Cont’d)
  def ant = new AntBuilder()
  ant.taskdef(name:'groovy',
             ...
…Process Enhancements
• Process Management proc1 = 'ls'.execute()
def process = "ls -l".execute()           proc2 = 'tr -d...
Parallelize your arrays with JSR 166y
//Create a pool with size close to the number of processor cores
def pool = new Fork...
Java Concurrency Best Practice?
• Java Concurrency in Practice:
 – “If mutable threads access the
   same mutable state va...
Immutability options
• Built-in
 def animals = ['cat', 'dog', 'horse'].asImmutable()
 animals << 'fish' // => java.lang.Un...
@Immutable...
                    • Java Immutable Class
                      – As per Joshua Bloch                      ...
...@Immutable...
                    • Java Immutable Class                                                             bo...
...@Immutable



                    @Immutable class Punter {
                        String first, last
© ASERT 2006-201...
@Synchronized – Before Transform
                    class SynchronizedExample {
                        private final myL...
@Synchronized – After Transform
                    class SynchronizedExample {
                                          ...
@Lazy...
                    • Safe initialization idioms
                      – Eager or fully synchronized
            ...
...@Lazy...
                    • Java Concurrency in Practice
                      – Race condition with lazy initializa...
...@Lazy...
                    • Java Concurrency in Practice
                      – Double checked locking anti-pattern...
...@Lazy...
                    • Java Concurrency in Practice
                      – Double checked locking less broken-...
...@Lazy...
                    • Java Concurrency in Practice
                      – Lazy initialization holder class id...
...@Lazy


                    @Lazy Resource first


                    @Lazy volatile Resource second
© ASERT 2006-2010...
Topics
                    • Groovy Intro
                    • Useful Groovy features for Concurrency
                   ...
Lightweight threads: Jetlang
• Jetlang
    – A high performance java threading library
Fiber receiver = new ThreadFiber();...
Other High-Level Libraries: JPPF
   – Open source Grid Computing platform
   – http://www.jppf.org/
import org.jppf.client...
Other High-Level Libraries: Gruple...
– http://code.google.com/p/gruple
– Gruple aims to provide a simple abstraction to a...
Other High-Level Libraries: ...Gruple...
      – Mandelbrot example (included in Gruple download)

...
Space space = Space...
Other High-Level Libraries: ...Gruple




                                        Concurrency - 60
Other High-Level Libraries: Cascading.groovy
 – API/DSL for executing tasks on a Hadoop cluster
 – http://www.cascading.or...
Other High-Level Libraries: GridGain
 – Simple & productive to use grid computing platform
 – http://www.gridgain.com/

cl...
Multiverse STM…
– http://www.multiverse.org/

 import   org.multiverse.api.GlobalStmInstance
 import   org.multiverse.api....
…Multiverse STM…
...
void atomic(Closure block) {
    atomic([:], block)
}

void atomic(Map args, Closure block) {
    boo...
…Multiverse STM


class Account {
    private final balance = new LongRef()

    Account(long initial) {
        balance.s...
Testing multi-threaded applications: ConTest...
• Advanced Testing for Multi-Threaded Applications
   – Tool for testing, ...
...Testing multi-threaded applications: ConTest
NUM = 5                           ParalInc.groovy

count = 0
def incThread...
GContracts
@Grab('org.gcontracts:gcontracts:1.0.2')
import org.gcontracts.annotations.*
                                  ...
Testing: Spock


class HelloSpock extends spock.lang.Specification {
    def "length of Spock's and his friends' names"() ...
Topics
                    • Groovy Intro
                    • Useful Groovy features for Concurrency
                   ...
Fibonacci Case Study




                       Concurrency - 71
Fibonacci…
START = 8                                             Serial
END = 16                                          ...
…Fibonacci…
import java.util.concurrent.*
                                                   ConcurrentHashMap
THREADS = 4...
…Fibonacci…
Thread.start('Producer') {
    int x = START
    while (x <= END) {
        sleep 200
        queue << x++
   ...
…Fibonacci…
                                                                Executor
import java.util.concurrent.*        ...
…Fibonacci…
import EDU.oswego.cs.dl.util.concurrent.FJTask
import EDU.oswego.cs.dl.util.concurrent.FJTaskRunnerGroup

clas...
…Fibonacci…
def THREADS = 2
def group = new FJTaskRunnerGroup(THREADS)
def START = 8
def END = 16
(START..END).each {num -...
…Fibonacci…
import   fj.*
import   fj.control.parallel.Strategy                   FunctionalJava
import   static fj.Functi...
…Fibonacci…
serialFib = {n -> n < 2 ? n : serialFib(n - 1) + serialFib(n - 2) }

print = {results ->
    def n = START
   ...
…Fibonacci…
import   org.jetlang.core.Callback
import   org.jetlang.channels.MemoryChannel                         Jetlang...
…Fibonacci
// ...
    void subscribe(other) {
        channel.subscribe(receiver, other)
    }

    FibonacciCalc() {
    ...
Topics
                    • Groovy Intro
                    • Useful Groovy features for Concurrency
                   ...
GPars
                    • http://gpars.codehaus.org/
                    • Library classes and DSL sugar providing
     ...
GPars: Parallel Collection Functions
                       def nums = 1..100000
                       def squares = nums...
GPars: Transparent Parallel Collections
                    • Applies some Groovy metaprogramming

                      i...
Gpars concurrency-aware methods
          Transparent       Transitive?               Parallel
    any { ... }            ...
GPars: Map-Reduce...


                    import static groovyx.gpars.GParsPool.withPool

                    withPool(5)...
...GPars: Map-Reduce


                    import static groovyx.gpars.GParsPool.withPool

                    withPool(5)...
Parallel Collections vs Map-Reduce




Source: ReGina                           Concurrency - 90
GPars: Dataflows...

                    import groovyx.gpars.dataflow.DataFlows
                    import static groovyx...
...GPars: Dataflows...
                    • Evaluating:
                            result = (a – b) * (a + b)


        ...
...GPars: Dataflows...
                    • Naive attempt for loops
                    import groovyx.gpars.dataflow.Dat...
...GPars: Dataflows...
                    import groovyx.gpars.dataflow.DataFlowStream        10           4
            ...
...GPars: Dataflows
                    • Amenable to static analysis
                    • Race conditions avoided
      ...
GPars: Dataflow Sieve
                        final int requestedPrimeNumberCount = 1000
                        final Dat...
GPars: Actors...
                    • Predefined coordination • Class with the
                      with fork/join &    ...
…GPars: Actors...
                        import static groovyx.gpars.actor.Actors.*

                        def decrypt ...
…GPars: Actors...
                        final class FilterActor extends DynamicDispatchActor {
                         ...
…GPars: Actors
                        (2..requestedPrimeNumberBoundary).each {
                            firstFilter it...
GPars: Agents...
                    • Agents safeguard non-thread safe objects
                    • Only the agent can u...
…GPars: Agents
                        @Grab('org.codehaus.gpars:gpars:0.10')
                        import groovyx.gpars...
GPars for testing
                    @Grab('net.sourceforge.htmlunit:htmlunit:2.6')
                    import com.gargoy...
Guy Steele example in Groovy…
                                             Sequential version
                    def word...
…Guy Steele example in Groovy…
                    @Immutable class Chunk {                             Refactored sequent...
…Guy Steele example in Groovy…
                    def words = { s ->                                         Refactored s...
…Guy Steele example in Groovy…
                                                                Roll your own threading wit...
…Guy Steele example in Groovy…
                                                     DataFlow version: partially hard-coded...
…Guy Steele example in Groovy…
                                                                                     Fork/J...
…Guy Steele example in Groovy…
                                                                            Map/Reduce vers...
groovy and concurrency
groovy and concurrency
groovy and concurrency
groovy and concurrency
groovy and concurrency
Upcoming SlideShare
Loading in...5
×

groovy and concurrency

7,507

Published on

Dr Paul King presentation slides on "Groovy and Concurrency" presented at StrangeLoop 2010 in St Loius

Published in: Technology

groovy and concurrency

  1. 1. © ASERT 2006-2010 Groovy and Concurrency Dr Paul King, @paulk_asert paulk at asert.com.au
  2. 2. Concurrent Programming in Groovy • Java concurrent programming enhancements – Normal OO methods – Ability to have immutable types – Some concurrency building blocks – Annotations with baked in goodness • Process/Thread ease of use – AntBuilder and GDK methods • Closures for greater flexibility – Enabler for concurrency – Closure is Runnable and Callable • Third-party libraries – GPars, Functional Java (Actors), Multiverse, JCSP – Cascading.groovy subproject for Hadoop clusters – Jetlang, JPPF, GridGain, Google Collections, Gruple – Groovy actors: http://www.groovyactors.org Concurrency - 2
  3. 3. Java Concurrency Features • The early years – Threads, synchronised and non-synchronised collections, synchronisation at the language level, Monitors (wait/notify), Locks, ThreadLocal, final, ... • More recent enhancements – java.util.concurrent: Executors, Thread Pools, Optimistic updates, Blocking queues, Synchronizers, Callables, Futures, Atomic operations, Deques, ... • Emerging – Fork/Join & others, Kilim, Phasers, PicoThreads ... • Leverage related APIs/technologies – Networking, real-time, GUIs, simulation, database, multimedia, operating systems, parallel processing, distribution, mobile agents, nio, ... Concurrency - 3
  4. 4. Topics Groovy Intro • Useful Groovy features for Concurrency • Related Concurrency Libraries & Tools • Fibonacci Case Study • GPars © ASERT 2006-2010 • More Info Concurrency - 4
  5. 5. What is Groovy? • “Groovy is like a super version of Java. It can leverage Java's enterprise capabilities but also has cool productivity features like closures, DSL support, builders and dynamic typing.” © ASERT 2006-2010 Groovy = Java – boiler plate code + optional dynamic typing + closures + domain specific languages + builders + metaprogramming Concurrency - 5
  6. 6. Groovy Goodies Overview • Fully object oriented • Closures: reusable and assignable pieces of code • Operators can be • GPath: efficient overloaded © ASERT 2006-2010 object navigation • Multimethods • GroovyBeans • Literal declaration • grep and switch for lists (arrays), maps, ranges and • Templates, builder, regular expressions swing, Ant, markup, XML, SQL, XML-RPC, Scriptom, Grails, tests, Mocks Concurrency - 6
  7. 7. Growing Acceptance … A slow and steady start but now gaining in momentum, maturity and mindshare Now free
  8. 8. … Growing Acceptance … What alternative JVM language are you using or intending to use © ASERT 2006-2010 http://www.leonardoborges.com/writings http://it-republik.de/jaxenter/quickvote/results/1/poll/44 (translated using http://babelfish.yahoo.com) Concurrency - 8
  9. 9. … Growing Acceptance … © ASERT 2006-2010 Source: http://www.micropoll.com/akira/mpresult/501697-116746 Source: http://www.grailspodcast.com/ Concurrency - 9
  10. 10. … Growing Acceptance … © ASERT 2006-2010 http://www.jroller.com/scolebourne/entry/devoxx_2008_whiteboard_votes http://www.java.net Concurrency - 10
  11. 11. … Growing Acceptance … © ASERT 2006-2010 Groovy and Grails downloads: 70-90K per month and growing Frequent topic at the popular conferences Concurrency - 11
  12. 12. … Growing Acceptance © ASERT 2006-2010 Concurrency - 12
  13. 13. The Landscape of JVM Languages optional static types © ASERT 2006-2010 Dynamic features call for dynamic types Java bytecode calls for static types The terms “Java Virtual Machine” and “JVM” mean a Virtual Machine for the Java™ platform. Concurrency - 13
  14. 14. Groovy Starter System.out.println("Hello, World!"); // supports Java syntax println 'Hello, World!' // but can remove some syntax String name = 'Guillaume' // Explicit typing/awareness println "$name, I'll get the car." // Gstring (interpolation) def longer = """${name}, the car is in the next row.""" // multi-line, implicit type © ASERT 2006-2010 assert 0.5 == 1/2 // BigDecimal equals() assert 0.1 + 0.2 == 0.3 // and arithmetic def printSize(obj) { // implicit/duck typing print obj?.size() // safe dereferencing } def pets = ['ant', 'bee', 'cat'] // native list syntax pets.each { pet -> // closure support assert pet < 'dog' // overloading '<' on String } // or: for (pet in pets)... Concurrency - 14
  15. 15. A Better Java... import java.util.List; import java.util.ArrayList; class Erase { private List removeLongerThan(List strings, int length) { This code List result = new ArrayList(); for (int i = 0; i < strings.size(); i++) { is valid String s = (String) strings.get(i); Java and if (s.length() <= length) { result.add(s); valid Groovy } } © ASERT 2006-2010 return result; } public static void main(String[] args) { List names = new ArrayList(); names.add("Ted"); names.add("Fred"); names.add("Jed"); names.add("Ned"); System.out.println(names); Based on an Erase e = new Erase(); example by List shortNames = e.removeLongerThan(names, 3); Jim Weirich System.out.println(shortNames.size()); for (int i = 0; i < shortNames.size(); i++) { & Ted Leung String s = (String) shortNames.get(i); System.out.println(s); } } } Concurrency - 15
  16. 16. ...A Better Java... import java.util.List; import java.util.ArrayList; class Erase { private List removeLongerThan(List strings, int length) { Do the List result = new ArrayList(); for (int i = 0; i < strings.size(); i++) { semicolons String s = (String) strings.get(i); add anything? if (s.length() <= length) { result.add(s); And shouldn‟t } } we us more © ASERT 2006-2010 } return result; modern list public static void main(String[] args) { notation? List names = new ArrayList(); names.add("Ted"); names.add("Fred"); Why not names.add("Jed"); names.add("Ned"); System.out.println(names); import common Erase e = new Erase(); libraries? List shortNames = e.removeLongerThan(names, 3); System.out.println(shortNames.size()); for (int i = 0; i < shortNames.size(); i++) { String s = (String) shortNames.get(i); System.out.println(s); } } } Concurrency - 16
  17. 17. ...A Better Java... class Erase { private List removeLongerThan(List strings, int length) { List result = new ArrayList() for (String s in strings) { if (s.length() <= length) { result.add(s) } } return result } public static void main(String[] args) { © ASERT 2006-2010 List names = new ArrayList() names.add("Ted"); names.add("Fred") names.add("Jed"); names.add("Ned") System.out.println(names) Erase e = new Erase() List shortNames = e.removeLongerThan(names, 3) System.out.println(shortNames.size()) for (String s in shortNames) { System.out.println(s) } } } Concurrency - 17
  18. 18. ...A Better Java... class Erase { private List removeLongerThan(List strings, int length) { List result = new ArrayList() for (String s in strings) { if (s.length() <= length) { result.add(s) Do we need } the static types? } return result Must we always } have a main public static void main(String[] args) { method and © ASERT 2006-2010 List names = new ArrayList() names.add("Ted"); names.add("Fred") class definition? names.add("Jed"); names.add("Ned") System.out.println(names) How about Erase e = new Erase() List shortNames = e.removeLongerThan(names, 3) improved System.out.println(shortNames.size()) consistency? for (String s in shortNames) { System.out.println(s) } } } Concurrency - 18
  19. 19. ...A Better Java... def removeLongerThan(strings, length) { def result = new ArrayList() for (s in strings) { if (s.size() <= length) { result.add(s) } } return result } names = new ArrayList() names.add("Ted") names.add("Fred") © ASERT 2006-2010 names.add("Jed") names.add("Ned") System.out.println(names) shortNames = removeLongerThan(names, 3) System.out.println(shortNames.size()) for (s in shortNames) { System.out.println(s) } Concurrency - 19
  20. 20. ...A Better Java... def removeLongerThan(strings, length) { def result = new ArrayList() for (s in strings) { if (s.size() <= length) { result.add(s) } } Shouldn‟t we return result } have special names = new ArrayList() notation for lists? names.add("Ted") names.add("Fred") And special © ASERT 2006-2010 names.add("Jed") names.add("Ned") facilities for System.out.println(names) shortNames = removeLongerThan(names, 3) list processing? System.out.println(shortNames.size()) for (s in shortNames) { Is „return‟ System.out.println(s) } needed at end? Concurrency - 20
  21. 21. ...A Better Java... def removeLongerThan(strings, length) { strings.findAll{ it.size() <= length } } names = ["Ted", "Fred", "Jed", "Ned"] System.out.println(names) shortNames = removeLongerThan(names, 3) System.out.println(shortNames.size()) shortNames.each{ System.out.println(s) } © ASERT 2006-2010 Concurrency - 21
  22. 22. ...A Better Java... def removeLongerThan(strings, length) { strings.findAll{ it.size() <= length } } Is the method names = ["Ted", "Fred", "Jed", "Ned"] System.out.println(names) now needed? shortNames = removeLongerThan(names, 3) System.out.println(shortNames.size()) Easier ways to shortNames.each{ System.out.println(s) } use common methods? © ASERT 2006-2010 Are brackets required here? Concurrency - 22
  23. 23. ...A Better Java names = ["Ted", "Fred", "Jed", "Ned"] println names shortNames = names.findAll{ it.size() <= 3 } println shortNames.size() shortNames.each{ println it } © ASERT 2006-2010 Output: ["Ted", "Fred", "Jed", "Ned"] 3 Ted Jed Ned Concurrency - 23
  24. 24. Grapes / Grab: Google collections @Grab('com.google.collections:google-collections:1.0') import com.google.common.collect.HashBiMap HashBiMap fruit = [grape:'purple', lemon:'yellow', lime:'green'] assert fruit.lemon == 'yellow' © ASERT 2006-2010 assert fruit.inverse().yellow == 'lemon' Concurrency - 24
  25. 25. Better Design Patterns: Delegate… public Date getWhen() { import java.util.Date; return when; } public class Event { private String title; public void setWhen(Date when) { private String url; this.when = when; private Date when; } public String getUrl() { public boolean before(Date other) { return url; return when.before(other); } © ASERT 2006-2010 } public void setUrl(String url) { public void setTime(long time) { this.url = url; when.setTime(time); } } public String getTitle() { public long getTime() { return title; return when.getTime(); } } public void setTitle(String title) { public boolean after(Date other) { this.title = title; return when.after(other); } } // ... // ... ESDC 2010 - 25
  26. 26. …Better Design Patterns: Delegate… public Date getWhen() { import java.util.Date; return when; boilerplate } public class Event { private String title; public void setWhen(Date when) { private String url; this.when = when; private Date when; } public String getUrl() { public boolean before(Date other) { return url; return when.before(other); } © ASERT 2006-2010 } public void setUrl(String url) { public void setTime(long time) { this.url = url; when.setTime(time); } } public String getTitle() { public long getTime() { return title; return when.getTime(); } } public void setTitle(String title) { public boolean after(Date other) { this.title = title; return when.after(other); } } // ... // ... ESDC 2010 - 26
  27. 27. …Better Design Patterns: Delegate class Event { String title, url @Delegate Date when } © ASERT 2006-2010 def gr8conf = new Event(title: "GR8 Conference", url: "http://www.gr8conf.org", when: Date.parse("yyyy/MM/dd", "2009/05/18")) def javaOne = new Event(title: "JavaOne", url: "http://java.sun.com/javaone/", when: Date.parse("yyyy/MM/dd", "2009/06/02")) assert gr8conf.before(javaOne.when) ESDC 2010 - 27
  28. 28. Why Groovy? Technical Answer • Minimal learning curve • Compiles to bytecode • Java object model & integration • Annotations • "Optional" static typing • Both run-time and compile-time metaprogramming Concurrency - 28
  29. 29. Why Groovy? Adoption Assessment • Innovators/Thought leaders – Ideas, power, flexibility, novelty, thinking community • Early adopters – Productivity benefits and collegiate community – Leverage JVM and potential for mainstream • Mainstream – Leverage existing Java skills, low learning curve – Leverage JVM and production infrastructure – Professional community – Tools, tools, tools
  30. 30. Source: Herb Sutter: http://www.gotw.ca/publications/concurrency-ddj.htm "Andy giveth and Bill taketh away" Concurrency - 30
  31. 31. Why is it hard? • Many issues to deal with: – Doing things in parallel, concurrently, asynchronously • Processes, Threads, Co-routines, Events, Scheduling – Sharing/Synchronization Mechanisms • shared memory, locks, transactions, wait/notify, STM, message passing, actors, serializability, persistence, immutability – Abstractions • Shared memory on top of messaging passing • Message passing on top of shared memory • Dataflow, Selective Communication, Continuations – Data Structures and Algorithms • Queues, Heaps, Trees • Sorting, Graph Algorithms Concurrency - 31
  32. 32. Topics • Groovy Intro Useful Groovy features for Concurrency • Related Concurrency Libraries & Tools • Fibonacci Case Study • GPars © ASERT 2006-2010 • More Info Concurrency - 32
  33. 33. Thread Enhancements… • Thread improvements Thread.start{ sleep 1001; println 'one' } Thread.start{ sleep 1000; println 'two' } println 'one' def t = Thread.start{ sleep 100; println 'three' } println 'two' t.join() println 'four' Concurrency - 33
  34. 34. …Thread Enhancements… • Thread improvements (Cont’d) class Storage { List stack = [] synchronized void leftShift(value){ stack << value println "push: $value" notifyAll() } synchronized Object pop() { while (!stack) { try{ wait() } catch(InterruptedException e){} } def value = stack.pop() println "pop : $value" return value } } ... Concurrency - 34
  35. 35. …Thread Enhancements… • Thread improvements (Cont’d) push: 0 push: 1 ... push: 2 storage = new Storage() pop : 2 push: 3 Thread.start { push: 4 for (i in 0..9) { pop : 4 storage << i push: 5 sleep 100 push: 6 } pop : 6 } push: 7 push: 8 Thread.start { pop : 8 10.times { push: 9 sleep 200 pop : 9 storage.pop() pop : 7 } pop : 5 } pop : 3 pop : 1 pop : 0 Concurrency - 35
  36. 36. …Thread Enhancements… • Thread improvements (Cont’d) ROCK! . 25 import java.util.concurrent.locks.ReentrantLock . 33 import static System.currentTimeMillis as now . 34 def startTime = now() . 35 ReentrantLock.metaClass.withLock = { critical -> . 36 lock() .. 131 try { critical() } .. 134 finally { unlock() } } .. 137 def lock = new ReentrantLock() .. 138 def worker = { threadNum -> .. 139 4.times { count -> ... 232 lock.withLock { ... 234 print " " * threadNum ... 237 print "." * (count + 1) ... 238 println " ${now() - startTime}" ... 239 } Thread.sleep 100 .... 334 } .... 336 } .... 337 5.times { Thread.start worker.curry(it) } .... 338 println "ROCK!" .... 339 Source: http://chrisbroadfoot.id.au/articles/2008/08/06/groovy-threads Concurrency - 36
  37. 37. … Thread Enhancements • Thread Management meets AtomicInteger import java.util.concurrent.atomic.AtomicInteger def counter = new AtomicInteger() synchronized out(message) { println(message) thread loop 1 } main loop 1 def th = Thread.start { thread loop 2 for( i in 1..8 ) { thread loop 3 sleep 30 main loop 2 out "thread loop $i" thread loop 4 counter.incrementAndGet() } thread loop 5 } main loop 3 for( j in 1..4 ) { thread loop 6 sleep 50 main loop 4 out "main loop $j" thread loop 7 counter.incrementAndGet() thread loop 8 } th.join() assert counter.get() == 12 Concurrency - 37
  38. 38. Process Enhancements… • Using AntBuilder [echo] Message 6 [echo] Message 9 [echo] Message 8 def ant = new AntBuilder() [echo] Message 0 ant.parallel { [echo] Message 2 10.times { echo "Message $it" } [echo] Message 4 } [echo] Message 1 [echo] Message 7 [echo] Message 5 [echo] Message 3 Concurrency - 38
  39. 39. …Process Enhancements… • Using AntBuilder (Cont’d) def ant = new AntBuilder() ant.taskdef(name:'groovy', classname:'org.codehaus.groovy.ant.Groovy') ant.parallel { 10.times { echo "Ant message $it" groovy "println 'Groovy via ant message $it'" println "Groovy message $it" // or fork java, command line, thread ... } } def ant = new AntBuilder() ant.exec(outputproperty:"cmdOut", errorproperty: "cmdErr", resultproperty:"cmdExit", failonerror: "true", executable: /opt/myExecutable') { arg(line:'*"first with space"* second') } Concurrency - 39
  40. 40. …Process Enhancements • Process Management proc1 = 'ls'.execute() def process = "ls -l".execute() proc2 = 'tr -d o'.execute() println "Found text ${process.text}" proc3 = 'tr -d e'.execute() proc4 = 'tr -d i'.execute() proc1 | proc2 | proc3 | proc4 def process = "ls -l".execute() proc4.waitFor() process.in.eachLine { line -> if (proc4.exitValue()) println line } print proc4.err.text def sout = new StringBuffer() else def serr = new StringBuffer() print proc4.text proc1 = 'gzip -c'.execute() proc2 = 'gunzip -c'.execute() proc2.consumeProcessOutput(sout, serr) proc1 | proc2 proc1.consumeProcessErrorStream(serr) proc1.withWriter { writer -> writer << 'test text' } proc2.waitForOrKill(1000) println 'sout: ' + sout // => test text println 'serr: ' + serr Concurrency - 40
  41. 41. Parallelize your arrays with JSR 166y //Create a pool with size close to the number of processor cores def pool = new ForkJoinPool(2) def createParallelArray(pool, collection) { return ParallelArray.createFromCopy( collection.toArray(new Object[collection.size()]), pool) } // Enhance ArrayLists to find matching objects in parallel ArrayList.metaClass.findAll = {Closure cl -> createParallelArray(pool, delegate). withFilter({cl(it)} as Predicate).all().asList() } def sites=['http://www.jroller.com', 'http://www.infoq.com', 'http://java.dzone.com'] def groovySites = sites.findAll { new URL(it).text.toLowerCase().contains('groovy')} println "These sites talk about Groovy today: ${groovySites}" Source: http://www.jroller.com/vaclav/date/20080923 Concurrency - 41
  42. 42. Java Concurrency Best Practice? • Java Concurrency in Practice: – “If mutable threads access the same mutable state variable without appropriate synchronization, your program is broken” – “When designing thread-safe classes, good object-oriented techniques – encapsulation, immutability, and clear specification of invariants – are your best friends” Concurrency - 42
  43. 43. Immutability options • Built-in def animals = ['cat', 'dog', 'horse'].asImmutable() animals << 'fish' // => java.lang.UnsupportedOperationException • Google Collections – Numerous improved immutable collection types import com.google.common.collect.* List<String> animals = ImmutableList.of("cat", "dog", "horse") animals << 'fish' // => java.lang.UnsupportedOperationException • Groovy run-time metaprogramming def animals = ['cat', 'dog', 'horse'] ArrayList.metaClass.leftShift = { throw new UnsupportedOperationException() } animals << 'fish' // => java.lang.UnsupportedOperationException • Groovy 1.6+ compile-time metaprogramming – @Immutable can help us create such classes Concurrency - 43
  44. 44. @Immutable... • Java Immutable Class – As per Joshua Bloch // ... @Override Effective Java public boolean equals(Object obj) { if (this == obj) public final class Punter { return true; private final String first; if (obj == null) private final String last; return false; if (getClass() != obj.getClass()) public String getFirst() { return false; return first; Punter other = (Punter) obj; } if (first == null) { © ASERT 2006-2010 if (other.first != null) public String getLast() { return false; return last; } else if (!first.equals(other.first)) } return false; if (last == null) { @Override if (other.last != null) public int hashCode() { return false; final int prime = 31; } else if (!last.equals(other.last)) int result = 1; return false; result = prime * result + ((first == null) return true; ? 0 : first.hashCode()); } result = prime * result + ((last == null) ? 0 : last.hashCode()); @Override return result; public String toString() { } return "Punter(first:" + first + ", last:" + last + ")"; public Punter(String first, String last) { } this.first = first; this.last = last; } } // ... Concurrency - 44
  45. 45. ...@Immutable... • Java Immutable Class boilerplate – As per Joshua Bloch // ... @Override Effective Java public boolean equals(Object obj) { if (this == obj) public final class Punter { return true; private final String first; if (obj == null) private final String last; return false; if (getClass() != obj.getClass()) public String getFirst() { return false; return first; Punter other = (Punter) obj; } if (first == null) { © ASERT 2006-2010 if (other.first != null) public String getLast() { return false; return last; } else if (!first.equals(other.first)) } return false; if (last == null) { @Override if (other.last != null) public int hashCode() { return false; final int prime = 31; } else if (!last.equals(other.last)) int result = 1; return false; result = prime * result + ((first == null) return true; ? 0 : first.hashCode()); } result = prime * result + ((last == null) ? 0 : last.hashCode()); @Override return result; public String toString() { } return "Punter(first:" + first + ", last:" + last + ")"; public Punter(String first, String last) { } this.first = first; this.last = last; } } // ... Concurrency - 45
  46. 46. ...@Immutable @Immutable class Punter { String first, last © ASERT 2006-2010 } Concurrency - 46
  47. 47. @Synchronized – Before Transform class SynchronizedExample { private final myLock = new Object() @Synchronized static void greet() { println "world" } © ASERT 2006-2010 @Synchronized int answerToEverything() { return 42 } @Synchronized("myLock") void foo() { println "bar" } } Concurrency - 47
  48. 48. @Synchronized – After Transform class SynchronizedExample { Inspired by private static final $LOCK = new Object[0] Project private final $lock = new Object[0] Lombok private final myLock = new Object() static void greet() { synchronized ($LOCK) { println "world" } } © ASERT 2006-2010 int answerToEverything() { synchronized ($lock) { return 42 } } void foo() { synchronized (myLock) { println "bar" } } } Concurrency - 48
  49. 49. @Lazy... • Safe initialization idioms – Eager or fully synchronized import net.jcip.annotations.ThreadSafe @ThreadSafe class EagerInitialization { static final resource = new Resource() © ASERT 2006-2010 } @ThreadSafe class SafeLazyInitialization { private static ExpensiveResource resource synchronized static ExpensiveResource getInstance() { if (!resource) resource = new ExpensiveResource () resource } } Concurrency - 49
  50. 50. ...@Lazy... • Java Concurrency in Practice – Race condition with lazy initialization import net.jcip.annotations.NotThreadSafe @NotThreadSafe class LazyInitRace { © ASERT 2006-2010 private ExpensiveResource instance = null ExpensiveResource getInstance() { if (!instance) instance = new ExpensiveResource() instance } } – No problems: just apply Double-checked locking pattern Concurrency - 50
  51. 51. ...@Lazy... • Java Concurrency in Practice – Double checked locking anti-pattern @NotThreadSafe class DoubleCheckedLocking { private static ExpensiveResource instance = null © ASERT 2006-2010 static ExpensiveResource getInstance() { if (!instance) { synchronized (DoubleCheckedLocking) { if (!instance) instance = new ExpensiveResource() } } instance } } Concurrency - 51
  52. 52. ...@Lazy... • Java Concurrency in Practice – Double checked locking less broken-pattern class DoubleCheckedLocking { private static volatile ExpensiveResource instance = null static ExpensiveResource getInstance() { © ASERT 2006-2010 if (!instance) { synchronized (DoubleCheckedLocking) { if (!instance) instance = new ExpensiveResource() } } instance } } Concurrency - 52
  53. 53. ...@Lazy... • Java Concurrency in Practice – Lazy initialization holder class idiom import net.jcip.annotations.ThreadSafe @ThreadSafe class ResourceFactory { © ASERT 2006-2010 private static class ResourceHolder { static Resource resource = new Resource() } static Resource getResource() { ResourceHolder.resource } } Concurrency - 53
  54. 54. ...@Lazy @Lazy Resource first @Lazy volatile Resource second © ASERT 2006-2010 @Lazy volatile Resource third = { new Resource(args) }() @Lazy(soft = true) volatile Resource fourth Concurrency - 54
  55. 55. Topics • Groovy Intro • Useful Groovy features for Concurrency Related Concurrency Libraries & Tools • Fibonacci Case Study • GPars © ASERT 2006-2010 • More Info Concurrency - 55
  56. 56. Lightweight threads: Jetlang • Jetlang – A high performance java threading library Fiber receiver = new ThreadFiber(); // JAVA receiver.start(); // create java.util.concurrent.CountDownLatch to notify when message arrives final CountDownLatch latch = new CountDownLatch(1); // create channel to message between threads Channel<String> channel = new MemoryChannel<String>(); Callback<String> onMsg = new Callback<String>() { public void onMessage(String message) { //open latch latch.countDown(); } }; //add subscription for message on receiver thread channel.subscribe(receiver, onMsg); //publish message to receive thread. the publish method is thread safe. channel.publish("Hello"); //wait for receiving thread to receive message latch.await(10, TimeUnit.SECONDS); //shutdown thread receiver.dispose(); Concurrency - 56
  57. 57. Other High-Level Libraries: JPPF – Open source Grid Computing platform – http://www.jppf.org/ import org.jppf.client.* import java.util.concurrent.Callable class Task implements Callable, Serializable { private static final long serialVersionUID = 1162L; public Object call() { println 'Executing Groovy' "Hello JPPF from Groovy" } } def client = new JPPFClient() def job = new JPPFJob() def task = new Task() job.addTask task def results = client.submit(job) for (t in results) { if (t.exception) throw t.exception println "Result: " + t.result } Concurrency - 57
  58. 58. Other High-Level Libraries: Gruple... – http://code.google.com/p/gruple – Gruple aims to provide a simple abstraction to allow programmers to coordinate and synchronize threads with ease – based on Tuplespaces • Tuplespaces provide the illusion of a shared memory on top of a message passing system, along with a small set of operations to greatly simplify parallel programming – Example Tuple: [fname:"Vanessa", lname:"Williams", project:"Gruple"] – Basic operations within a Tuplespace are: • put - insert a tuple into the space • get - read a tuple from the space (non-destructively) • take - take a tuple from the space (a destructive read) – Further reading: Eric Freeman, Susanne Hupfer, and Ken Arnold. JavaSpaces Principles, Patterns, and Practice, Addison Wesley, 1999 Concurrency - 58
  59. 59. Other High-Level Libraries: ...Gruple... – Mandelbrot example (included in Gruple download) ... Space space = SpaceService.getSpace("mandelbrot") Map template = createTaskTemplate() Map task String threadName = Thread.currentThread().name while(true) { ArrayList points task = space.take(template) println "Worker $threadName got task ${task['start']} for job ${task['jobId'] points = calculateMandelbrot(task) Map result = createResult(task['jobId'], task['start'], points) println "Worker $threadName writing result for task ${result['start']} for jo space.put(result) } ... Concurrency - 59
  60. 60. Other High-Level Libraries: ...Gruple Concurrency - 60
  61. 61. Other High-Level Libraries: Cascading.groovy – API/DSL for executing tasks on a Hadoop cluster – http://www.cascading.org/ def assembly = builder.assembly(name: "wordcount") { eachTuple(args: ["line"], results: ["word"]) { regexSplitGenerator(declared: ["word"], pattern: /[.,]*s+/) } group(["word"]) everyGroup(args: ["word"], results: ["word", "count"]) { count() } group(["count"], reverse: true) } def map = builder.map() { source(name: "wordcount") { hfs(input) { text(["line"]) } } sink(name: "wordcount") { hfs(output) { text() } } } def flow = builder.flow(name: "wordcount", map: map, assembly: assembly) Concurrency - 61
  62. 62. Other High-Level Libraries: GridGain – Simple & productive to use grid computing platform – http://www.gridgain.com/ class GridHelloWorldGroovyTask extends GridTaskSplitAdapter<String, Integer> { Collection split(int gridSize, Object phrase) throws GridException { // ... } Object reduce(List results) throws GridException { // ... } } import static GridFactory.* start() def grid = getGrid() def future = grid.execute(GridHelloWorldGroovyTask, "Hello World") def phraseLen = future.get() stop(true) Concurrency - 62
  63. 63. Multiverse STM… – http://www.multiverse.org/ import org.multiverse.api.GlobalStmInstance import org.multiverse.api.Transaction import org.multiverse.templates.TransactionTemplate import org.multiverse.transactional.refs.LongRef def from = new Account(10) def to = new Account(10) atomic { from.balance -= 5 to.balance += 5 } println "from $from.balance" println "to $to.balance" ... Concurrency - 63
  64. 64. …Multiverse STM… ... void atomic(Closure block) { atomic([:], block) } void atomic(Map args, Closure block) { boolean readonly = args['readonly'] ?: false boolean trackreads = args['trackreads'] ?: true def txFactory = GlobalStmInstance.globalStmInstance. transactionFactoryBuilder. setReadonly(readonly). setReadTrackingEnabled(trackreads).build() new TransactionTemplate(txFactory) { Object execute(Transaction transaction) { block.call() return null } }.execute() } ... Concurrency - 64
  65. 65. …Multiverse STM class Account { private final balance = new LongRef() Account(long initial) { balance.set initial } void setBalance(long newBalance) { if (newBalance < 0) throw new RuntimeException("not enough money") balance.set newBalance } long getBalance() { balance.get() } } Concurrency - 65
  66. 66. Testing multi-threaded applications: ConTest... • Advanced Testing for Multi-Threaded Applications – Tool for testing, debugging, and coverage-measuring of concurrent programs (collects runtime statistics) – Systematically and transparently (using a java agent) schedules the execution of program threads in ways likely to reveal race conditions, deadlocks, and other intermittent bugs (collectively called synchronization problems) with higher than normal frequency – The ConTest run-time engine adds heuristically controlled conditional instructions (adjustable by a preferences file) that force thread switches, thus helping to reveal concurrent bugs. You can use existing tests and run ConTest multiple times – by default different heuristics used each time it is run • http://www.alphaworks.ibm.com/tech/contest Concurrency - 66
  67. 67. ...Testing multi-threaded applications: ConTest NUM = 5 ParalInc.groovy count = 0 def incThread = { n -> Thread.start{ sleep n*10 targetClasses = ParalInc //synchronized(ParalInc) { timeoutTampering = true count++ noiseFrequency = 500 //} strength = 10000 } } def threads = (1..NUM).collect(incThread) threads.each{ it.join() } assert count == NUM > groovyc ParalInc.groovy > java -javaagent:../../Lib/ConTest.jar -cp %GROOVY_JAR%;. ParalInc Exception in thread "main" Assertion failed: assert count == NUM | | | 4 | 5 false Concurrency - 67
  68. 68. GContracts @Grab('org.gcontracts:gcontracts:1.0.2') import org.gcontracts.annotations.* 1.8+ @Invariant({ first != null && last != null }) class Person { String first, last @Requires({ delimiter in ['.', ',', ' '] }) @Ensures({ result == first + delimiter + last }) String getName(String delimiter) { first + delimiter + last } } new Person(first: 'John', last: 'Smith').getName('.') Concurrency - 68
  69. 69. Testing: Spock class HelloSpock extends spock.lang.Specification { def "length of Spock's and his friends' names"() { expect: name.size() == length where: name | length "Spock" | 5 "Kirk" | 4 "Scotty" | 6 } } Concurrency - 69
  70. 70. Topics • Groovy Intro • Useful Groovy features for Concurrency • Related Concurrency Libraries & Tools Fibonacci Case Study • GPars © ASERT 2006-2010 • More Info Concurrency - 70
  71. 71. Fibonacci Case Study Concurrency - 71
  72. 72. Fibonacci… START = 8 Serial END = 16 version fib = {n -> n < 2 ? n : fib(n - 1) + fib(n - 2) } (START..END).each {num -> println "n:$num => ${fib(num)}" } Concurrency - 72
  73. 73. …Fibonacci… import java.util.concurrent.* ConcurrentHashMap THREADS = 4 version START = 8 END = 16 QUIT = -1 class Fibonacci { def values = new ConcurrentHashMap() int calc(x) { x < 2 ? x : calc(x-1) + calc(x-2) } int calcWithCache(x) { def result = values[x] if (!result) { result = calc(x) values.putIfAbsent(x, result) } result } } println "Calculating Fibonacci sequence in parallel..." def queue = new ArrayBlockingQueue(10) Concurrency - 73
  74. 74. …Fibonacci… Thread.start('Producer') { int x = START while (x <= END) { sleep 200 queue << x++ } sleep 1000 THREADS.times { queue << QUIT } } (1..THREADS).each { def name = "Consumer$it" Thread.start(name) { def done = false def fib = new Fibonacci() while (!done) { def n = queue.take() if (n == QUIT) done = true else println "$name n:$n => ${fib.calcWithCache(n)}" } } } Concurrency - 74
  75. 75. …Fibonacci… Executor import java.util.concurrent.* version CUTOFF = 12 // not worth parallelizing for small n THREADS = 100 println "Calculating Fibonacci sequence in parallel..." serialFib = {n -> (n < 2) ? n : serialFib(n - 1) + serialFib(n - 2) } pool = Executors.newFixedThreadPool(THREADS) defer = {c -> pool.submit(c as Callable) } fib = {n -> if (n < CUTOFF) return serialFib(n) def left = defer { fib(n - 1) } def right = defer { fib(n - 2) } left.get() + right.get() } (8..16).each {n -> println "n=$n => ${fib(n)}" } pool.shutdown() Concurrency - 75
  76. 76. …Fibonacci… import EDU.oswego.cs.dl.util.concurrent.FJTask import EDU.oswego.cs.dl.util.concurrent.FJTaskRunnerGroup class Fib extends FJTask { static final CUTOFF = 12 Fork/Join volatile int number version int getAnswer() { if (!isDone()) throw new IllegalStateException() number } void run() { int n = number if (n <= CUTOFF) number = seqFib(n) else { def f1 = new Fib(number: n - 1) def f2 = new Fib(number: n - 2) coInvoke(f1, f2) number = f1.number + f2.number } } int seqFib(int n) { n < 2 ? n : seqFib(n - 1) + seqFib(n - 2) } } Concurrency - 76
  77. 77. …Fibonacci… def THREADS = 2 def group = new FJTaskRunnerGroup(THREADS) def START = 8 def END = 16 (START..END).each {num -> def f = new Fib(number: num) group.invoke(f) println "n:$num => $f.answer" } Concurrency - 77
  78. 78. …Fibonacci… import fj.* import fj.control.parallel.Strategy FunctionalJava import static fj.Function.curry as fcurry version import static fj.P1.curry as pcurry import static fj.P1.fmap import static fj.control.parallel.Actor.actor import static fj.control.parallel.Promise.* import static fj.data.List.range import static java.util.concurrent.Executors.* CUTOFF = 12 // not worth parallelizing for small n START = 8 END = 16 THREADS = 4 pool = newFixedThreadPool(THREADS) su = Strategy.executorStrategy(pool) spi = Strategy.executorStrategy(pool) add = fcurry({a, b -> a + b } as F2) nums = range(START, END + 1) println "Calculating Fibonacci sequence in parallel..." Concurrency - 78
  79. 79. …Fibonacci… serialFib = {n -> n < 2 ? n : serialFib(n - 1) + serialFib(n - 2) } print = {results -> def n = START results.each { println "n=${n++} => $it" } pool.shutdown() } as Effect calc = {n -> n < CUTOFF ? promise(su, P.p(serialFib(n))) : calc.f(n - 1).bind(join(su, pcurry(calc).f(n - 2)), add) } as F out = actor(su, print) join(su, fmap(sequence(su)).f(spi.parMapList(calc).f(nums))).to(out) Concurrency - 79
  80. 80. …Fibonacci… import org.jetlang.core.Callback import org.jetlang.channels.MemoryChannel Jetlang import org.jetlang.fibers.ThreadFiber version import java.util.concurrent.* println "Calculating Fibonacci sequence with two cooperating fibers..." class FibonacciCalc implements Callback { private channel, receiver def limit, name, latch void onMessage(inpair) { def next = inpair[0] + inpair[1] def outpair = [inpair[1], next] println "$name next:$next" channel.publish(outpair) if (next > limit) { latch.countDown() sleep 200 receiver.dispose() } } // ... Concurrency - 80
  81. 81. …Fibonacci // ... void subscribe(other) { channel.subscribe(receiver, other) } FibonacciCalc() { channel = new MemoryChannel() receiver = new ThreadFiber() receiver.start() } } def seed = [0, 1] def latch = new CountDownLatch(2) def calcA = new FibonacciCalc(limit: 500, name: 'CalcA', latch: latch) def calcB = new FibonacciCalc(limit: 500, name: 'CalcB', latch: latch) calcA.subscribe calcB calcB.subscribe calcA calcA.onMessage seed latch.await(10, TimeUnit.SECONDS) Concurrency - 81
  82. 82. Topics • Groovy Intro • Useful Groovy features for Concurrency • Related Concurrency Libraries & Tools • Fibonacci Case Study GPars © ASERT 2006-2010 • More Info Concurrency - 82
  83. 83. GPars • http://gpars.codehaus.org/ • Library classes and DSL sugar providing intuitive ways for Groovy developers to handle tasks concurrently. Logical parts: – Actors provide a Groovy implementation of Scala-like actors including "remote" actors on other machines – Dataflow Concurrency supports natural shared-memory © ASERT 2006-2010 concurrency model, using single-assignment variables – Asynchronizer extends the Java 1.5 built-in support for executor services to enable multi-threaded collection and closure processing – Parallelizer uses JSR-166y Parallel Arrays to enable multi-threaded collection processing – Safe a non-blocking mt-safe reference to mutable state that is inspired by "agents" in Clojure Concurrency - 83
  84. 84. GPars: Parallel Collection Functions def nums = 1..100000 def squares = nums .collect{ it ** 2 } .grep{ it % 7 == it % 5 } .grep{ it % 3 == 0 } println squares[0..3] + "..." + squares[-3..-1] assert squares[0..3] == [36, 144, 1089, 1296] © ASERT 2006-2010 @Grab('org.codehaus.gpars:gpars:0.10') import static groovyx.gpars.GParsPool.withPool def nums = 1..100000 withPool(5) { def squares = nums. collectParallel{ it ** 2 }. grepParallel{ it % 7 == it % 5 }. grepParallel{ it % 3 == 0 } println squares[0..3] + "..." + squares[-3..-1] assert squares[0..3] == [36, 144, 1089, 1296] } Concurrency - 85
  85. 85. GPars: Transparent Parallel Collections • Applies some Groovy metaprogramming import static groovyx.gpars.GParsPool.withPool withPool(5) { def nums = 1..100000 nums.makeTransparent() © ASERT 2006-2010 def squares = nums. collect{ it ** 2 }. grep{ it % 7 == it % 5 }. grep{ it % 3 == 0 } println squares[0..3] + "..." + squares[-3..-1] assert squares[0..3] == [36, 144, 1089, 1296] } Concurrency - 86
  86. 86. Gpars concurrency-aware methods Transparent Transitive? Parallel any { ... } anyParallel { ... } collect { ... } yes collectParallel { ... } count(filter) countParallel(filter) each { ... } eachParallel { ... } eachWithIndex { ... } eachWithIndexParallel { ... } every { ... } everyParallel { ... } find { ... } findParallel { ... } findAll { ... } yes findAllParallel { ... } findAny { ... } findAnyParallel { ... } fold { ... } foldParallel { ... } fold(seed) { ... } foldParallel(seed) { ... } grep(filter) yes grepParallel(filter) groupBy { ... } groupByParallel { ... } max { ... } maxParallel { ... } max() maxParallel() min { ... } minParallel { ... } min() minParallel() split { ... } yes splitParallel { ... } Source: ReGina Concurrency - 87
  87. 87. GPars: Map-Reduce... import static groovyx.gpars.GParsPool.withPool withPool(5) { def nums = 1..100000 println nums.parallel. © ASERT 2006-2010 map{ it ** 2 }. filter{ it % 7 == it % 5 }. filter{ it % 3 == 0 }. collection } Concurrency - 88
  88. 88. ...GPars: Map-Reduce import static groovyx.gpars.GParsPool.withPool withPool(5) { def nums = 1..100000 println nums.parallel. © ASERT 2006-2010 map{ it ** 2 }. filter{ it % 7 == it % 5 }. filter{ it % 3 == 0 }. reduce{ a, b -> a + b } } Concurrency - 89
  89. 89. Parallel Collections vs Map-Reduce Source: ReGina Concurrency - 90
  90. 90. GPars: Dataflows... import groovyx.gpars.dataflow.DataFlows import static groovyx.gpars.dataflow.DataFlow.task final flow = new DataFlows() task { flow.result = flow.x + flow.y } task { flow.x = 10 } task { flow.y = 5 } © ASERT 2006-2010 assert 15 == flow.result new DataFlows().with { task { result = x * y } task { x = 10 } task { y = 5 } assert 50 == result } Concurrency - 91
  91. 91. ...GPars: Dataflows... • Evaluating: result = (a – b) * (a + b) x y import groovyx.gpars.dataflow.DataFlows © ASERT 2006-2010 import static groovyx.gpars.dataflow.DataFlow.task final flow = new DataFlows() 10 5 task { flow.a = 10 } task { flow.b = 5 } task { flow.x = flow.a - flow.b } a b task { flow.y = flow.a + flow.b } task { flow.result = flow.x * flow.y } - + assert flow.result == 75 * Concurrency - 92
  92. 92. ...GPars: Dataflows... • Naive attempt for loops import groovyx.gpars.dataflow.DataFlows import static groovyx.gpars.dataflow.DataFlow.task final flow = new DataFlows() ... [10, 20].each { thisA -> task { flow.a = 10 } [4, 5].each { thisB -> ... © ASERT 2006-2010 task { flow.a = thisA } task { flow.a = 20 } task { flow.b = thisB } task { flow.x = flow.a - flow.b } task { flow.y = flow.a + flow.b } task { flow.result = flow.x * flow.y } println flow.result } } // => java.lang.IllegalStateException: A DataFlowVariable can only be assigned once. Concurrency - 93
  93. 93. ...GPars: Dataflows... import groovyx.gpars.dataflow.DataFlowStream 10 4 import static groovyx.gpars.dataflow.DataFlow.* 10 5 20 4 final streamA = new DataFlowStream() 20 5 final streamB = new DataFlowStream() final streamX = new DataFlowStream() a b final streamY = new DataFlowStream() final results = new DataFlowStream() - + © ASERT 2006-2010 operator(inputs: [streamA, streamB], outputs: [streamX, streamY]) { * a, b -> streamX << a - b; streamY << a + b } operator(inputs: [streamX, streamY], outputs: [results]) { x, y -> results << x * y } 84 75 [[10, 20], [4, 5]].combinations().each{ thisA, thisB -> 384 task { streamA << thisA } 375 task { streamB << thisB } } 4.times { println results.val } Concurrency - 94
  94. 94. ...GPars: Dataflows • Amenable to static analysis • Race conditions avoided • Deadlocks “typically” become repeatable import groovyx.gpars.dataflow.DataFlows © ASERT 2006-2010 import static groovyx.gpars.dataflow.DataFlow.task final flow = new DataFlows() task { flow.x = flow.y } task { flow.y = flow.x } Concurrency - 95
  95. 95. GPars: Dataflow Sieve final int requestedPrimeNumberCount = 1000 final DataFlowStream initialChannel = new DataFlowStream() task { (2..10000).each { initialChannel << it } } def filter(inChannel, int prime) { def outChannel = new DataFlowStream() © ASERT 2006-2010 operator([inputs: [inChannel], outputs: [outChannel]]) { if (it % prime != 0) { bindOutput it } } return outChannel } def currentOutput = initialChannel requestedPrimeNumberCount.times { int prime = currentOutput.val println "Found: $prime" currentOutput = filter(currentOutput, prime) } Source: http://groovyconsole.appspot.com/script/235002 Concurrency - 96
  96. 96. GPars: Actors... • Predefined coordination • Class with the with fork/join & following lifecycle & map/filter/reduce methods • Implicit coordination – But also DSL sugar & with dataflow augmentation • Actors provide explicit start() © ASERT 2006-2010 stop() coordination and act() enforce* no sharing of send(msg) state, process a single sendAndWait(msg) activity/message at a loop { } time react { msg -> } msg.reply(replyMsg) receive() join() * mostly Concurrency - 97
  97. 97. …GPars: Actors... import static groovyx.gpars.actor.Actors.* def decrypt = reactor { code -> code.reverse() } def audit = reactor { println it } def main = actor { decrypt 'terces pot' © ASERT 2006-2010 react { plainText -> audit plainText } } main.join() audit.stop() audit.join() Source: ReGina Concurrency - 98
  98. 98. …GPars: Actors... final class FilterActor extends DynamicDispatchActor { private final int myPrime private def follower def FilterActor(final myPrime) { this.myPrime = myPrime; } def onMessage(int value) { if (value % myPrime != 0) { if (follower) follower value else { println "Found $value" © ASERT 2006-2010 follower = new FilterActor(value).start() } } } def onMessage(def poisson) { if (follower) { def sender = poisson.sender follower.sendAndContinue(poisson, {this.stop(); sender?.send('Done } else { //I am the last in the chain stop() reply 'Done' } } } Source: http://groovyconsole.appspot.com/script/242001 Concurrency - 99
  99. 99. …GPars: Actors (2..requestedPrimeNumberBoundary).each { firstFilter it } firstFilter.sendAndWait 'Poisson' © ASERT 2006-2010 Source: http://groovyconsole.appspot.com/script/242001 Concurrency - 100
  100. 100. GPars: Agents... • Agents safeguard non-thread safe objects • Only the agent can update the underlying object • “Code” to update the protected object is sent to the agent © ASERT 2006-2010 • Can be used with other approaches Concurrency - 101
  101. 101. …GPars: Agents @Grab('org.codehaus.gpars:gpars:0.10') import groovyx.gpars.agent.Agent def speakers = new Agent<List>(['Alex'], {it?.clone()}) // add Alex speakers.send {updateValue it << 'Hilary'} // add Hilary final Thread t1 = Thread.start { speakers.send {updateValue it << 'Ken'} // add Ken } © ASERT 2006-2010 final Thread t2 = Thread.start { speakers << {updateValue it << 'Guy'} // add Guy speakers << {updateValue it << 'Ralph'} // add Ralph } [t1, t2]*.join() assert new HashSet(speakers.val) == new HashSet(['Alex', 'Hilary', 'Ken', 'Guy', 'Ralph']) Source: Gpars examples Concurrency - 102
  102. 102. GPars for testing @Grab('net.sourceforge.htmlunit:htmlunit:2.6') import com.gargoylesoftware.htmlunit.WebClient @Grab('org.codehaus.gpars:gpars:0.10') import static groovyx.gpars.GParsPool.* def testCases = [ ['Home', 'Bart', 'Content 1'], ['Work', 'Homer', 'Content 2'], ['Travel', 'Marge', 'Content 3'], © ASERT 2006-2010 ['Food', 'Lisa', 'Content 4'] ] withPool(3) { testCases.eachParallel{ category, author, content -> postAndCheck category, author, content } } private postAndCheck(category, author, content) { ... Concurrency - 103
  103. 103. Guy Steele example in Groovy… Sequential version def words = { s -> Guy Steele‟s example from keynote (from slide 52 onwards for several slides): http://strangeloop2010.com/talk/presentation_file/14299/GuySteele-parallel.pdf def result = [] def word = '' s.each{ ch -> if (ch == ' ') { if (word) result += word word = '' } else word += ch © ASERT 2006-2010 } if (word) result += word result } assert words("This is a sample") == ['This', 'is', 'a', 'sample'] assert words(" Here is another sample ") == ['Here', 'is', 'another', 'sample'] assert words("JustOneWord") == ['JustOneWord'] assert words("Here is a sesquipedalian string of words") == ['Here', 'is', 'a', 'sesquipedalian', 'string', 'of', 'words'] assert words(" ") == [] && words("") == [] Concurrency - 104
  104. 104. …Guy Steele example in Groovy… @Immutable class Chunk { Refactored sequential version String s def plus(Chunk other) { new Chunk(s + other.s) } def plus(Segment other) { new Segment(s + other.l, other.m, other.r) } } @Immutable class Segment { String l; List m; String r © ASERT 2006-2010 def plus(Chunk other) { new Segment(l, m, r + other.s) } def plus(Segment other) { new Segment(l, m + maybeWord(r + other.l) + other.m, other.r) } } class Util { static processChar(ch) { ch == ' ' ? new Segment('', [], '') : new Chunk(ch) } static maybeWord(s) { s ? [s] : [] } } import static Util.* ... Concurrency - 105
  105. 105. …Guy Steele example in Groovy… def words = { s -> Refactored sequential version def result s.each{ ch -> if (!result) result = processChar(ch) else result += processChar(ch) } switch(result) { case Chunk: return maybeWord(result.s) case Segment: return result.with{ maybeWord(l) + m + maybeWord(r) } case null: return [] © ASERT 2006-2010 } } assert words("This is a sample") == ['This', 'is', 'a', 'sample'] assert words(" Here is another sample ") == ['Here', 'is', 'another', 'sample'] assert words("JustOneWord") == ['JustOneWord'] assert words("Here is a sesquipedalian string of words") == ['Here', 'is', 'a', 'sesquipedalian', 'string', 'of', 'words'] assert words(" ") == [] && words("") == [] Concurrency - 106
  106. 106. …Guy Steele example in Groovy… Roll your own threading with ConcurrentHashMap version def swords = { s -> def result s.each{ ch -> if (!result) result = processChar(ch) else result += processChar(ch) } result ?: new Chunk('') } THREADS = 4 © ASERT 2006-2010 def words = { s -> int n = (s.size() + THREADS - 1) / THREADS def map = new java.util.concurrent.ConcurrentHashMap() (0..<THREADS).collect { i -> Thread.start { def (min, max) = [[s.size(),i*n].min(), [s.size(),(i+1)*n].min()] map[i] = swords(s[min..<max]) }}*.join() def result = map.entrySet().sort{ it.key }.sum{ it.value } switch(result) { case Chunk: return maybeWord(result.s) case Segment: return result.with{ maybeWord(l) + m + maybeWord(r) } } } Concurrency - 107
  107. 107. …Guy Steele example in Groovy… DataFlow version: partially hard-coded to 4 partitions for easier reading def words = { s -> int n = (s.size() + THREADS - 1) / THREADS def min = (0..<THREADS).collectEntries{ [it, [s.size(),it*n].min()] } def max = (0..<THREADS).collectEntries{ [it, [s.size(),(it+1)*n].min()] } def result = new DataFlows().with { task { a = swords(s[min[0]..<max[0]]) } task { b = swords(s[min[1]..<max[1]]) } task { c = swords(s[min[2]..<max[2]]) } task { d = swords(s[min[3]..<max[3]]) } © ASERT 2006-2010 task { sum1 = a + b } task { sum2 = c + d } task { sum = sum1 + sum2 } println 'Tasks ahoy!' sum } switch(result) { case Chunk: return maybeWord(result.s) case Segment: return result.with{ maybeWord(l) + m + maybeWord(r) } } } Concurrency - 108
  108. 108. …Guy Steele example in Groovy… Fork/Join version GRANULARITY_THRESHHOLD = 10 THREADS = 4 println GParsPool.withPool(THREADS) { def result = runForkJoin(0, input.size(), input){ first, last, s -> def size = last - first if (size <= GRANULARITY_THRESHHOLD) { swords(s[first..<last]) } else { // divide and conquer © ASERT 2006-2010 def mid = first + ((last - first) >> 1) forkOffChild(first, mid, s) forkOffChild(mid, last, s) childrenResults.sum() } } switch(result) { case Chunk: return maybeWord(result.s) case Segment: return result.with{ maybeWord(l) + m + maybeWord(r) } } } Concurrency - 109
  109. 109. …Guy Steele example in Groovy… Map/Reduce version THRESHHOLD = 10 def split(raw) { raw.size() <= THRESHHOLD ? raw : [raw[0..<THRESHHOLD]] + split(raw.substring(THRESHHOLD)) } © ASERT 2006-2010 println GParsPool.withPool(THREADS) { def ans = split(input).parallel.map(swords).reduce{ a,b -> a + b } switch(ans) { case Chunk: return maybeWord(ans.s) case Segment: return ans.with{ maybeWord(l) + m + maybeWord(r) } } } Concurrency - 110
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×