The Road to Lambda - Mike Duigou


Published on

The big language features for Java SE 8 are lambda expressions (a.k.a. closures) and default methods (a.k.a. virtual extension methods). Adding closures to the Java language opens up a host of new expressive opportunities for applications and libraries, but how are they implemented? You might assume that lambda expressions are simply a compact syntax for inner classes, but, in fact, the implementation of lambda expressions is substantially different and builds on the invokedynamic feature added in Java SE 7.

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Bigger than generics – really! Just as generics allowed libraries to abstract better over types, lambdas allow libraries to abstract better over behavior
  • This may not seem like a big deal, but in reality it is a HUGE deal, because of the impact it has on library design.
  • It is hard to see why loops are bad when it is the only tool we have for aggregate operations! The essential problem with this code is that it conflates the “what” – change the red blocks to blue – with the “how”. This is so second-nature to us that we don’t even notice.But in today’s multicore world, a language construct that is inherently sequential is a significant problem.
  • Note that lambda infers type of s
  • Powerful type inference – get type from context
  • If Java had closures in 1997, our Collections would surely look differentSo modernizing the language has the perverse effect of making the libraries look even older!Historically could not add methods to interfaces without breaking implementationsHas something in common with, but not exactly like, C# extension methods, Scala traitsThis is not something we did lightly – “no code in interfaces” is a long-standing aspect of Java
  • Compare to C# extension methods, traits
  • Let’s look at a few use cases for default methods. First the obvious one – adding new methods to an interface after it’s been published.
  • Key point – why is this code better? Each part does one thing, and is clearly labeled. Plus parallelism
  • Plus parallelism
  • Plus parallelism
  • Composibility and separation of concerns powered by lambdas“Less brittle” means small changes to the problem result in small changes to the codeNoise == “garbage variables”
  • One of Java’s strength has always been its libraries. The best way to write better applications is to have better, more powerful libraries. If parallelism is to become easier, libraries have to be a key part of getting there. While there are parallel libraries for Java today, such as the fork-join library added to Java SE 7, the code for expressing a computation serially and the code for expressing it in parallel is very different, and this places a barrier in front of users to move their code from serial to parallel.
  • Safer = can avoid issues like CME
  • Declaration of method may be a little scary, but invocation is very clearExample of a higher-order function – takes a function (extractor), returns a function (comparator)Method references can be used instead of lambdas
  • All of this “better libraries” talk is because it’s all about the libraries.
  • The Road to Lambda - Mike Duigou

    1. 1. The Road to LambdaMike Duigou | Oracle
    2. 2. The following is intended to outline our general productdirection. It is intended for information purposes only,and may not be incorporated into any contract.It is not a commitment to deliver any material, code, orfunctionality, and should not be relied upon in makingpurchasing decisions. The development, release, andtiming of any features or functionality described forOracle’s products remains at the sole discretion ofOracle.
    3. 3. Modernizing Java• Java SE 8 is a big step forward in modernizing the Java Language– Lambda Expressions (closures)– Interface Evolution (default methods)• Java SE 8 is a big step forward in modernizing the Java Libraries– Bulk data operations on Collections– More library support for parallelism• Together, perhaps the biggest upgrade everto the Java programming model
    4. 4. What is a Lambda Expression?• A lambda expression (closure) is an anonymous method– Has an argument list, a return type, and a body(Object o) -> o.toString()– Can refer to values from the enclosing lexical scope(Person p) -> p.getName().equals(name)• A method reference is a reference to an existing methodObject::toString• Allow you to treat code as data– Behavior can be stored in variables and passed to methods
    5. 5. Times change• In 1995, most popular languages did not support closures• Today, Java is just about the last holdout that does not– C++ added them recently– C# added them in 3.0– New languages being designed today all do“In another thirty years people will laugh at anyone who triesto invent a language without closures, just as theyll laughnow at anyone who tries to invent a language withoutrecursion.”-Mark Jason Dominus
    6. 6. Closures for Java – a long and winding road• 1997 – Odersky/Wadler experimental “Pizza” work• 1997 – Java 1.1 added inner classes – a weak form of closures– Too bulky, complex name resolution rules, many limitations• In 2006-2008, a vigorous community debate about closures– Multiple proposals, including BGGA and CICE– Each had a different orientation• BGGA was focused on creating control abstraction in libraries• CICE was focused on reducing the syntactic overhead of innerclasses– Things ran aground at this point…
    7. 7. Closures for Java – a long and winding road• Dec 2009 – OpenJDK Project Lambda formed• November 2010 – JSR-335 filed• Current status– EDR specification complete– Prototype (source and binary) available on OpenJDK– Part of Java SE 8 (Spring 2014)• JSR-335 = Lambda Expressions+ Interface Evolution+ Bulk Collection Operations
    8. 8. External Iteration• Snippet takes the red blocks and colors them blue• Uses foreach loop– Loop is inherently sequential– Client has to manage iteration• This is called external iteration• Foreach loop hides complex interaction between library andclientfor (Shape s : shapes) {if (s.getColor() == RED)s.setColor(BLUE);}
    9. 9. Lambda Expressions• Re-written to use lambda and Collection.forEach– Not just a syntactic change!– Now the library is in control– This is internal iteration– More what, less how• Library free to use parallelism, out-of-order execution, laziness• Client passes behavior into the API as data• Enables API designers to build more powerful, expressive APIs– Greater power to abstract over behaviorshapes.forEach(s -> {if (s.getColor() == RED)s.setColor(BLUE);})
    10. 10. Functional Interfaces• What is the type of a lambda expression or method reference?– Historically we’ve used single-method interfaces to represent functions• Runnable, Comparator, ActionListener– Let’s give these a name: functional interfaces– And add some new ones like Predicate<T>, Consumer<T>• A lambda expression evaluates to an instance of a functional interfacePredicate<String> isEmpty = s -> s.isEmpty();Predicate<String> isEmpty = String::isEmpty;Runnable r = () -> { System.out.println(“Boo!”) };• So existing libraries are forward-compatible to lambdas– Maintains significant investment in existing libraries
    11. 11. Interface Evolution• The example used a new Collection method – forEach()– Where did that come from?– I thought you couldn’t addnew methods to interfaces?– New feature: default methods• Virtual method with defaultimplementation• Libraries need to evolve!• Default methods let us compatibly evolve libraries over timeinterface Collection<T> {default void forEach(Consumer<T> action) {for (T t : this)action.apply(t);}}
    12. 12. Interface Evolution• Interfaces are a double-edged sword– Cannot compatibly evolve them unless you control all implementations– Reality: APIs age• As we add cool new language features, existing APIs look even older!– Lots of bad options for dealing with aging APIs• Let the API stagnate• Replace it in entirety (every few years!)• Nail bags on the side (e.g., Collections.sort())• Burden of API evolution should fall to implementors, not users
    13. 13. Default Methods• A default method provides an implementation in the interface• Wait, is this multiple inheritance in Java?– Java always had multiple inheritance of types– This adds multiple inheritance of behavior• But no inheritance of state, where most of the trouble comes from• Some similarity (and differences) with C# extension methods– Default methods are virtual and declaration-site• Primary goal is API evolution– But useful as an inheritance mechanism on its own
    14. 14. Default Methods – Evolving Interfaces• New Collection methods– forEach(Consumer)– removeIf(Predicate)• Subclasses may providebetter implementations• Defaults implemented interms of other interfacemethodsinterface Collection<E> {default void forEach(Consumer<E> action) {for (E e : this)action.apply(e);}default boolean removeIf(Predicate<? super E> filter) {boolean removed = false;Iterator<E> each = iterator();while(each.hasNext()) {if(filter.test( {each.remove();removed = true;}}return removed;}}
    15. 15. Default Methods – Optional methods• Default methods can reduce implementation burden• Most implementations of Iterator don’t provide a usefulremove()– So why make developer write one that throws?– Default methods can be used to declare “optional” methodsinterface Iterator<T> {boolean hasNext();T next();default void remove() {throw new UnsupportedOperationException();}}
    16. 16. interface Comparator<T> {int compare(T o1, T o2);default Comparator<T> reverse() {return (o1, o2) -> compare(o2, o1);}default Comparator<T> compose(Comparator<T> other) {return (o1, o2) -> {int cmp = compare(o1, o2);return cmp != 0 ? cmp :, o2);}}}Default Methods – Combinators• Comparator.reverse() –reverses order of a Comparator• Comparator.compose() – computes “dictionary order” betweenone Comparator and anotherComparator<Person> byFirst = ...Comparator<Person> byLast = ...Comparator<Person> byFirstLast = byFirst.compose(byLast);Comparator<Person> byLastDescending = byLast.reverse();
    17. 17. Bulk operations on Collections• The lambda version of the “shapes” code can be furtherdecomposed• We have added other bulk operations, such as filter, map, into• “Color the red blocks blue” can be decomposed -> s.getColor() == RED).forEach(s -> { s.setColor(BLUE); });shapes.forEach(s -> {if (s.getColor() == RED)s.setColor(BLUE);})
    18. 18. Bulk operations on Collections• Collect the blue Shapes into a List• If each Shape lives in a Box, find Boxes containing blue shapeList<Shape> blueBlocks= -> s.getColor() == BLUE).collect(Collectors.toList());Set<Box> hasBlueBlock= -> s.getColor() == BLUE).map(s -> s.getContainingBox()).collect(Collectors.toSet());
    19. 19. Bulk operations on Collections• Compute sum of weights of blue shapes• Alternately, with method references:int sumOfWeight= -> s.getColor() == BLUE).map(s -> s.getWeight()).sum();int sumOfWeight= -> s.getColor() == BLUE).map(Shape::getWeight).sum();
    20. 20. Bulk operations on Collections• The new bulk operations are expressive and composible– Each stage does one thing– Compose compound operations from basic building blocks– Client code reads more like the problem statement– Structure of client code is less brittle– Less extraneous “noise” from intermediate results– Library can use parallelism, out-of-order, laziness for performance
    21. 21. Laziness• When does the computation occur?– Eager – computation is done when we return from call– Lazy – computation happens when we need the result• Many operations, like filter, can be implemented either lazily oreagerly• Iterators are inherently lazy• Laziness can be more efficient if you’re not going to use all the results– “Find the first element that matches this criteria”– Can stop after it finds one
    22. 22. Streams• To add bulk operations, we create a new abstraction, Stream– Represents a stream of values• Not a data structure – doesn’t store the values– Source can be a Collection, array, generating function, I/O…– Operations that produce new streams are lazy– Encourages a “fluent” usage styleStream<Foo> fooStream =;Stream<Foo> filtered = fooStream.filter(f -> f.isBlue());Stream<Bar> mapped = -> f.getBar());mapped.forEach(System.out::println); -> f.isBlue()).map(f -> f.getBar()).forEach(System.out::println);
    23. 23. Streams• Operations are divided between intermediate and terminal– No computation happens until you hit the terminal operation– Library then commits to an implementation strategyinterface Stream<T> {...Stream<T> filter(Predicate<T> predicate);<U> Stream<U> map(Mapper<T,U> mapper);<U> Stream<U> flatMap(FlatMapper<T,U> mapper);...}interface Stream<T> {...Stream<T> distinct();Stream<T> sorted(Comparator<? Super T> cmp);Stream<T> limit(int n);Stream<T> skip(int n);Stream<T> concat(Stream<? extends> T> other);...}interface Stream<T> {...void forEach(Block<? super T> block);<A extends Destination<? super T>> A into(A target);T[] toArray(Class<T> classToken);<R> R collect(Collector<T,R> collector);Optional<T> findFirst();Optional<T> findAny();...}
    24. 24. Advanced Example: Aggregation• Problem: select author, sum(pages) from documents group byauthorMap<String, Integer> map = new HashMap<>();for (Document d : documents) {String author = d.getAuthor();Integer i = map.get(author);if (i == null)i = 0;map.put(author, i + d.getPageCount());}Map<String, Integer> map collect(groupingBy(Document::getAuthor,reducing(0,Document::getPageCount, Integer::sum)));
    25. 25. Streams and Laziness• Operations tend to be combined into pipelines of source-lazy*-eager– collection-filter-map-sum– array-map-sorted-forEach• Streams are essentially lazy collections – compute as elementsare needed– Pays off big in cases like filter-map-findFirst– Laziness is mostly invisible (no new LazyList, LazySet, etc)– Enables short-circuiting, operation fusing
    26. 26. Parallelism• Goal: easy-to-use parallel libraries for Java– Libraries can hide a host of complex concerns (task scheduling, threadmanagement, load balancing)• Goal: reduce conceptual and syntactic gap between serial andparallel expressions of the same computation– Right now, the serial code and the parallel code for a given computationdon’t look anything like each other– Fork-join (added in Java SE 7) is a good start, but not enough• Goal: parallelism should be explicit, but unobtrusive
    27. 27. Fork/Join Parallelism• JDK7 adds general-purpose Fork/Join framework– Powerful and efficient, but not so easy to program to– Based on recursive decomposition• Divide problem into sub-problems, solve in parallel, combine results• Keep dividing until small enough to solve sequentially– Tends to be efficient across a wide range of processor counts– Generates reasonable load balancing with no central coordination
    28. 28. Fork/Join ParallelismResult solve(Problem p) {if (p.size() < SMALL_ENOUGH)return p.solveSequentially();else {Problem left = p.extractLeftHalf();Problem right = p.extractRightHalf();IN_PARALLEL {solve(left);solve(right);}return combine(left, right);}}
    29. 29. Parallel Sum with Fork/Joinclass SumProblem {final List<Shape> shapes;final int size;SumProblem(List<Shape> ls) {this.shapes = ls;size = ls.size();}public int solveSequentially() {int sum = 0;for (Shape s : shapes) {if (s.getColor() == BLUE)sum += s.getWeight();}return sum;}public SumProblem subproblem(int start, int end) {return new SumProblem(shapes.subList(start, end));}}ForkJoinExecutor pool = new ForkJoinPool(nThreads);SumProblem finder = new SumProblem(problem);pool.invoke(finder);class SumFinder extends RecursiveAction {private final SumProblem problem;int sum;protected void compute() {if (problem.size < THRESHOLD)sum = problem.solveSequentially();else {int m = problem.size / 2;SumFinder left, right;left = new SumFinder(problem.subproblem(0, m))right = new SumFinder(problem.subproblem(m, problem.size));forkJoin(left, right);sum = left.sum + right.sum;}}}
    30. 30. Parallel Sum with Collections• Parallel sum-of-sizes with bulk collection operations– Explicit but unobtrusive parallelism– All three operations fused into a single parallel passint sumOfWeight= shapes.parallel().filter(s -> s.getColor() == BLUE).map(Shape::getWeight).sum();
    31. 31. Parallel Streams• A stream can execute either in serial or parallel– Determined by stream source•• Collection.parallel()• Some data structures decompose better than others– Arrays decompose better than linked lists– Data structure contributes decomposition strategy– Parallel stream ops can run against any decomposible data structure
    32. 32. Decomposition• The parallel analogue of Iterator is … Spliterator– Defines decomposition strategy for data structurepublic interface Spliterator<T> {Spliterator<T> trySplit();Iterator<T> iterator();default int getSizeIfKnown();default int estimateSize();}
    33. 33. Lambdas Enable Better APIs• Lambda expressions enable delivery of more powerful APIs• The boundary between client and library is more permeable– Client can provide bits of functionality to be mixed into execution– Library remains in control of the computation– Safer, exposes more opportunities for optimization
    34. 34. Example: Sorting• If we want to sort a List today, we’d write a Comparator– Comparator conflates extraction of sort key with ordering of that key• Can replace Comparator with a lambda, but only gets us so far– Better to separate the two aspectsCollections.sort(people, new Comparator<Person>() {public int compare(Person x, Person y) {return x.getLastName().compareTo(y.getLastName());}});
    35. 35. Example: Sorting• Lambdas encourage finer-grained APIs– We add a method that takes a “key extractor” and returns Comparator– Multiple versions: Comparable objects, ints, longs, etcComparator<Person> byLastName= Comparators.comparing(p -> p.getLastName());Class Comparators {public static<T, U extends Comparable<? super U>>Comparator<T> comparing(Mapper<T, U> m) {return (x, y) ->;}}
    36. 36. Example: SortingComparator<Person> byLastName= Comparators.comparing(p -> p.getLastName());Collections.sort(people, byLastName);Collections.sort(people, comparing(p -> p.getLastName());people.sort(comparing(p -> p.getLastName());people.sort(comparing(Person::getLastName));people.sort(comparing(Person::getLastName).reversed());people.sort(comparing(Person::getLastName).andThen(comparing(Person::getFirstName)));
    37. 37. Lambdas Enable Better APIs• The comparing() method is one built for lambdas– Consumes an “extractor” function and produces a “comparator” function– Factors out key extraction from comparison– Eliminates redundancy, boilerplate• Key effect on APIs is: more composability– Leads to better factoring, more regular client code, more reuse• Lambdas in the languages→ can write better libraries→ more readable, less error-prone user code
    38. 38. Lambdas Enable Better APIs• Generally, we prefer to evolve the programming model throughlibraries– Time to market – can evolve libraries faster than language– Decentralized – more library developers than language developers– Risk – easier to change libraries, more practical to experiment– Impact – language changes require coordinated changes to multiplecompilers, IDEs, and other tools• But sometimes we reach the limits of what is practical toexpress in libraries, and need a little help from the language– But a little help, in the right places, can go a long way!
    39. 39. So … Why Lambda?• It’s about time!– Java is the lone holdout among mainstream OO languages at this point tonot have closures– Adding closures to Java is no longer a radical idea• Provide libraries a path to multicore– Parallel-friendly APIs need internal iteration– Internal iteration needs a concise code-as-data mechanism• Empower library developers– More powerful, flexible libraries– Higher degree of cooperation between libraries and client code
    40. 40. Where are we now?• EDR #3 spec draft available at• RI Binary drops available at– Download them and try them out!• Shipping with Java SE 8, Spring 2014