Popular tutorial (81/85 positive ratings) at JavaOne, San Francisco, October 2015. Video of earlier version (JavaOne 2014) is at https://www.youtube.com/watch?v=6tFZelz7Gvg.
3. • Why collectors?
• Using the predefined collectors
• Intermission!
• Worked problems
• Writing your own
Journey’s End – Agenda
4. Journey’s End, JavaOne, October 2015
Example Domain
city: City
name: String
age: int
Person
Person amy = new Person(Athens, "Amy", 21);
...
List<Person> people = Arrays.asList(jon, amy, bill);
5. Journey’s End, JavaOne, October 2015
Collectors: Journey’s End for a Stream
The Life of a Stream Element
• Born (at a spliterator)
• Transformed (by intermediate operations)
• Collected (by a terminal operation)
6. Journey’s End, JavaOne, October 2015
Collectors: Journey’s End for a Stream
The Life of a Stream Element
• Born (at a spliterator)
• Transformed (by intermediate operations)
• Collected (by a terminal operation)
7. Journey’s End, JavaOne, October 2015
Collectors: Journey’s End for a Stream
The Life of a Stream Element
• Born (at a spliterator)
• Transformed (by intermediate operations)
• Collected (by a terminal operation)
8. Journey’s End, JavaOne, October 2015
Collectors: Journey’s End for a Stream
The Life of a Stream Element
• Born (at a spliterator)
• Transformed (by intermediate operations)
• Collected (by a terminal operation)
9. Journey’s End, JavaOne, October 2015
Collectors: Journey’s End for a Stream
?
The Life of a Stream Element
• Born (at a spliterator)
• Transformed (by intermediate operations)
• Collected (by a terminal operation)
12. Journey’s End, JavaOne, October 2015
Side-effecting Operations
Side-effecting operations
• forEach, forEachOrdered
people.stream()
.forEach(System.out::println);
• So could we calculate total ages like this?
int sum = 0;
people.stream()
.mapToInt(Person::getAge)
.forEach(a -> { sum += a });
people.stream()
.forEach(System.out::println);
int sum = 0;
people.stream()
.mapToInt(Person::getAge)
.forEach(a -> { sum += a; });
13. Journey’s End, JavaOne, October 2015
Side-effecting Operations
Side-effecting operations
• forEach, forEachOrdered
people.stream()
.forEach(System.out::println);
• So could we calculate total ages like this?
int sum = 0;
people.stream()
.mapToInt(Person::getAge)
.forEach(a -> { sum += a });
people.stream()
.forEach(System.out::println);
int sum = 0;
people.stream()
.mapToInt(Person::getAge)
.forEach(a -> { sum += a; });
Don’t do this!
14. Journey’s End, JavaOne, October 2015
• Using an accumulator:
Reduction – Why?
int[] vals = new int[100];
Arrays.setAll(vals, i -> i);
int sum = 0;
for (int i = 0 ; i < vals.length ; i++) {
sum += vals[i];
}
15. Journey’s End, JavaOne, October 2015
• Using an accumulator:
Reduction – Why?
int[] vals = new int[100];
Arrays.setAll(vals, i -> i);
int sum = 0;
for (int i = 0 ; i < vals.length ; i++) {
sum += vals[i];
}
0 1 2 3
+
+
16. Journey’s End, JavaOne, October 2015
• Using an accumulator:
Reduction – Why?
int[] vals = new int[100];
Arrays.setAll(vals, i -> i);
int sum = 0;
for (int i = 0 ; i < vals.length ; i++) {
sum += vals[i];
}
0 1 2 3
+
+
+
19. Journey’s End, JavaOne, October 2015
• Avoiding an accumulator:
Reduction – Why?
int[] vals = new int[100];
Arrays.setAll(vals, i -> i);
OptionalInt sum = Arrays.stream(vals)
.reduce((a,b) -> a + b);
+ +
+
1 2 30
20. Journey’s End, JavaOne, October 2015
• Avoiding an accumulator:
Reduction – Why?
int[] vals = new int[100];
Arrays.setAll(vals, i -> i);
OptionalInt sum = Arrays.stream(vals)
.reduce((a,b) -> a + b);
+ +
+
1 2 30
+
+
+
21. Journey’s End, JavaOne, October 2015
• Avoiding an accumulator:
Reduction – Why?
int[] vals = new int[100];
Arrays.setAll(vals, i -> i);
OptionalInt sum = Arrays.stream(vals)
.reduce((a,b) -> a + b);
BinaryOperator must be associative!
+ +
+
1 2 30
+
+
+
22. Journey’s End, JavaOne, October 2015
• Avoiding an accumulator:
Reduction – Why?
int[] vals = new int[100];
Arrays.setAll(vals, i -> i);
OptionalInt sum = Arrays.stream(vals)
.reduce((a,b) -> a + b);
BinaryOperator must be associative!
a + (b + c) = (a + b) + c
+ +
+
1 2 30
+
+
+
26. Journey’s End, JavaOne, October 2015
Reduction on ImmutableValues
Reduction works on immutable values too
BigDecimal[] vals = new BigDecimal[100];
Arrays.setAll(vals, i -> new BigDecimal(i));
Optional<BigDecimal> sum = Arrays.stream(vals)
.reduce(BigDecimal::add);
27. Journey’s End, JavaOne, October 2015
Reduction on ImmutableValues
Reduction works on immutable values too
BigDecimal[] vals = new BigDecimal[100];
Arrays.setAll(vals, i -> new BigDecimal(i));
Optional<BigDecimal> sum = Arrays.stream(vals)
.reduce(BigDecimal::add);
29. Journey’s End, JavaOne, October 2015
Reduction to Collections?
This also works with collections – sort of ...
• for each element, client code creates a new empty
collection and adds the single element to it
• combines collections using addAll
30. Journey’s End, JavaOne, October 2015
Reduction to Collections?
This also works with collections – sort of ...
• for each element, client code creates a new empty
collection and adds the single element to it
• combines collections using addAll
We can do better!
31. Journey’s End, JavaOne, October 2015
Reduction over an Identity
BigDecimal[] vals = new BigDecimal[100];
Arrays.setAll(vals, i -> new BigDecimal(i));
BigDecimal sum = Arrays.stream(vals)
.reduce(BigDecimal.ZERO,BigDecimal::add);
32. Journey’s End, JavaOne, October 2015
Reduction over an Identity
BigDecimal[] vals = new BigDecimal[100];
Arrays.setAll(vals, i -> new BigDecimal(i));
BigDecimal sum = Arrays.stream(vals)
.reduce(BigDecimal.ZERO,BigDecimal::add);
Works for immutable objects:
33. Journey’s End, JavaOne, October 2015
Reduction over an Identity
+
+
+
0 1 2 3
+
+
+
0
4 5 6 7
0
++
+
35. Journey’s End, JavaOne, October 2015
Reduction to Collections?
Reduction over an identity doesn’t work with
collections at all
• reduction reuses the identity element
36. Journey’s End, JavaOne, October 2015
Reduction to Collections?
Reduction over an identity doesn’t work with
collections at all
• reduction reuses the identity element
We’ve got to do better!
37. Journey’s End, JavaOne, October 2015
The Answer – Collectors
a
a
a
e0 e1 e2 e3
a
a
a
e4 e5 e6 e7
aa
c
a: accumulator
c: combiner
38. Journey’s End, JavaOne, October 2015
The Answer – Collectors
a
a
a
e0 e1 e2 e3
a
a
a
() -> []
e4 e5 e6 e7
aa
c
a: accumulator
c: combiner
() -> []
39. Journey’s End, JavaOne, October 2015
Collectors
So to define a collector, you need to provide
• Supplier
• Accumulator
• Combiner
That sounds really hard!
Good then that we don’t have to do it
• – very often
40. Journey’s End, JavaOne, October 2015
Collectors API
Factory methods in the Collectors class.They
produce standalone collectors, accumulating to:
• framework-supplied containers;
• custom collections;
• classification maps.
41. • Why collectors?
• Using the predefined collectors
• Intermission!
• Worked problems
• Writing your own
Journey’s End – Agenda
42. Journey’s End, JavaOne, October 2015
Predefined Standalone Collectors – from factory
methods in Collections class
• toList(), toSet(), toMap(), joining()
• toMap(), toCollection()
groupingBy(), partitioningBy(),
groupingByConcurrent()
Using the Predefined Collectors
43. Journey’s End, JavaOne, October 2015
Predefined Standalone Collectors – from factory
methods in Collections class
• toList(), toSet(), toMap(), joining()
• toMap(), toCollection()
groupingBy(), partitioningBy(),
groupingByConcurrent()
Using the Predefined Collectors
framework provides
the Supplier
44. Journey’s End, JavaOne, October 2015
Predefined Standalone Collectors – from factory
methods in Collections class
• toList(), toSet(), toMap(), joining()
• toMap(), toCollection()
groupingBy(), partitioningBy(),
groupingByConcurrent()
Using the Predefined Collectors
user provides
the Supplier
framework provides
the Supplier
45. Journey’s End, JavaOne, October 2015
Predefined Standalone Collectors – from factory
methods in Collections class
• toList(), toSet(), toMap(), joining()
• toMap(), toCollection()
groupingBy(), partitioningBy(),
groupingByConcurrent()
Using the Predefined Collectors
user provides
the Supplier
framework provides
the Supplier
produce a
classification map
84. Journey’s End, JavaOne, October 2015
Collectors API
Factory methods in the Collectors class.They
produce standalone collectors, accumulating to:
• framework-supplied containers;
• custom collections;
• classification maps.
90. Journey’s End, JavaOne, October 2015
Collectors API
Factory methods in the Collectors class.They
produce standalone collectors, accumulating to:
• framework-supplied containers;
• custom collections;
• classification maps.
91. Journey’s End, JavaOne, October 2015
Collecting to Classification Maps
groupingBy
• simple
• with downstream
• with downstream and map factory
partitioningBy
92. Journey’s End, JavaOne, October 2015
groupingBy(Function<T,K> classifier)
Uses the classifier function to make a classification mapping
Like toMap(), except that the values placed in the map are lists of the
elements, one List corresponding to each classification key:
For example, use Person.getCity() to make a Map<City,List<Person>>
Map<City,List<Person>> peopleByCity = people.stream().
collect(Collectors.groupingBy(Person::getCity));
93. Journey’s End, JavaOne, October 2015
Stream<Person>
groupingBy() Map<City,List<Person>>
bill
jon
amy Athens
London
Collector<Person,?,Map<City,List<Person>
groupingBy(Function<Person,City>)
Classifier
Person→City
94. Journey’s End, JavaOne, October 2015
Stream<Person>
groupingBy() Map<City,List<Person>>
bill
jon
amy Athens
London
groupingBy(Function<Person,City>)
95. Journey’s End, JavaOne, October 2015
Stream<Person>
groupingBy() Map<City,List<Person>>
bill
jon
amy Athens
bill London
groupingBy(Function<Person,City>)
96. Journey’s End, JavaOne, October 2015
Stream<Person>
groupingBy() Map<City,List<Person>>
bill
jon
amy Athens
billLondon
groupingBy(Function<Person,City>)
[ ]
97. Journey’s End, JavaOne, October 2015
Stream<Person>
groupingBy() Map<City,List<Person>>
bill
jon
amy Athens [ ]
bill
amy
London
groupingBy(Function<Person,City>)
[ ]
98. Journey’s End, JavaOne, October 2015
Stream<Person>
groupingBy() Map<City,List<Person>>
bill
jon
amy Athens [ ]
[ , ]bill
amy
jonLondon
groupingBy(Function<Person,City>)
99. Journey’s End, JavaOne, October 2015
groupingBy() Map<City,List<Person>>
bill
jon
amy Athens [ ]
[ , ]bill
amy
jonLondon
groupingBy(Function<Person,City>)
100. Journey’s End, JavaOne, October 2015
groupingBy() Map<City,List<Person>>
bill
jon
amy
Athens [ ]
[ , ]bill
amy
jonLondon
groupingBy(Function<Person,City>)
101. Journey’s End, JavaOne, October 2015
groupingBy(Function<T,K> classifier)
Uses the classifier function to make a classification mapping
Like toMap(), except that the values placed in the map are lists of the
elements, one List corresponding to each classification key:
For example, use Person.getCity() to make a Map<City,List<Person>>
Map<City,List<Person>> peopleByCity = people.stream().
collect(Collectors.groupingBy(Person::getCity));
102. Journey’s End, JavaOne, October 2015
groupingBy(classifier, downstream)
Map<City,List<Person>> peopleByCity = people.stream().
collect(Collectors.groupingBy(Person::getCity,toSet()));
Uses the classifier function to make a classification mapping into a
container defined by a downstream Collector
Like toMap(), except that the values placed in the map are containers of
the elements, one container corresponding to each classification key:
For example, use Person.getCity() to make a Map<City,Set<Person>>
Map<City,Set<Person>> peopleByCity = people.stream().
collect(Collectors.groupingBy(Person::getCity,toSet()));
103. Journey’s End, JavaOne, October 2015
groupingBy(Function classifier,Collector downstream))
Stream<Person>
groupingBy()
Map<City,Set<Person>
bill
jon
amy
London
Athens
Downstream
Collector
—
toSet()
Stream<Person
>
Classifier
Person→City
104. Journey’s End, JavaOne, October 2015
groupingBy(Function classifier,Collector downstream))
Stream<Person>
groupingBy()
Map<City,Set<Person>
bill
jon
amy
London
Athens
105. Journey’s End, JavaOne, October 2015
groupingBy(Function classifier,Collector downstream))
Stream<Person>
groupingBy()
Map<City,Set<Person>
bill
jon
amy
London
Athens
bill
106. Journey’s End, JavaOne, October 2015
groupingBy(Function classifier,Collector downstream))
Stream<Person>
groupingBy()
Map<City,Set<Person>
bill
jon
amy
London
Athens
bill
107. Journey’s End, JavaOne, October 2015
groupingBy(Function classifier,Collector downstream))
Stream<Person>
groupingBy()
Map<City,Set<Person>
bill
jon
amy
London
Athens
bill
108. Journey’s End, JavaOne, October 2015
groupingBy(Function classifier,Collector downstream))
Stream<Person>
groupingBy()
Map<City,Set<Person>
bill
jon
amy
London
Athensamy
bill
109. Journey’s End, JavaOne, October 2015
groupingBy(Function classifier,Collector downstream))
Stream<Person>
groupingBy()
Map<City,Set<Person>
bill
jon
amy
London
Athens amy
bill
110. Journey’s End, JavaOne, October 2015
groupingBy(Function classifier,Collector downstream))
Stream<Person>
groupingBy()
Map<City,Set<Person>
bill
jon
amy
London
Athens amy
bill
111. Journey’s End, JavaOne, October 2015
groupingBy(Function classifier,Collector downstream))
Stream<Person>
groupingBy()
Map<City,Set<Person>
bill
jon
amy
London
Athens amy
jon bill
112. Journey’s End, JavaOne, October 2015
groupingBy(Function classifier,Collector downstream))
Stream<Person>
groupingBy()
Map<City,Set<Person>
bill
jon
amy
London
Athens amy
jon
bill
113. Journey’s End, JavaOne, October 2015
groupingBy(Function classifier,Collector downstream))
groupingBy()
Map<City,Set<Person>
bill
jon
amy
London
Athens amy
jon
bill
114. Journey’s End, JavaOne, October 2015
groupingBy(Function classifier,Collector downstream))
groupingBy()
Map<City,Set<Person>
bill
jon
amy
London
Athens amy
jon
bill
115. Journey’s End, JavaOne, October 2015
groupingBy(Function classifier,Collector downstream))
groupingBy()
Map<City,Set<Person>
bill
jon
amy
London
Athens { }amy
{ , }jonbill
116. Journey’s End, JavaOne, October 2015
groupingBy(Function classifier,Collector downstream))
groupingBy()
Map<City,Set<Person>
bill
jon
amy
London
Athens { }amy
{ , }jonbill
117. Journey’s End, JavaOne, October 2015
mapping(Function mapper,Collector downstream))
Applies a mapping function to each input element before passing it
to the downstream collector.
Set<City> inhabited = people.stream()
.collect(mapping(Person::getCity,toSet()));
Map<City,String> namesByCity = people.stream()
.collect(groupingBy(Person::getCity,
mapping(Person::getName,joining()));
Animation example
Sensible example
128. Journey’s End, JavaOne, October 2015
Thread safety is guaranteed by the framework
• Even for non-threadsafe containers!
Concurrent Collection
129. Journey’s End, JavaOne, October 2015
Thread safety is guaranteed by the framework
• Even for non-threadsafe containers!
But at a price... So what if your container is already threadsafe?
• A concurrentMap implementation, for example?
Concurrent Collection
130. Journey’s End, JavaOne, October 2015
Thread safety is guaranteed by the framework
• Even for non-threadsafe containers!
But at a price... So what if your container is already threadsafe?
• A concurrentMap implementation, for example?
Concurrent Collection
131. Journey’s End, JavaOne, October 2015
Thread safety is guaranteed by the framework
• Even for non-threadsafe containers!
But at a price... So what if your container is already threadsafe?
• A concurrentMap implementation, for example?
Every overload of toMap() and groupingBy() has a dual
• toConcurrentMap(...)
• groupingByConcurrent(...)
Concurrent Collection
132. • Why collectors?
• Using the predefined collectors
• Intermission!
• Worked problems
• Writing your own
Journey’s End – Agenda
133. • Why collectors?
• Using the predefined collectors
• Intermission!
• Worked problems
• Writing your own
Journey’s End – Agenda
135. Journey’s End, JavaOne, October 2015
Writing a Collector
Why would you want to?
• accumulate to a container that doesn’t implement Collection
136. Journey’s End, JavaOne, October 2015
Writing a Collector
Why would you want to?
• accumulate to a container that doesn’t implement Collection
• share state between values being collected
155. Journey’s End, JavaOne, October 2015
Performance of Collectors
Depends on the performance of accumulator and combiner
functions
• toList(), toSet(), toCollection – performance will probably be
dominated by accumulator, but remember that the framework will
need to manage multithread access to non-threadsafe containers
for the combine operation
toMap(), toConcurrentMap()
• map merging is slow. Resizing maps, especially concurrent maps, is
particularly expensive.Whenever possible, presize all data
structures, maps in particular.
156. Journey’s End, JavaOne, October 2015
Conclusion
Collectors
• generalisation of reduction
• allow a functional style while continuing to work with a
language based on mutation
• designed for composition
• flexible and powerful programming tool
• take a bit of getting used to! – but they’re worth it!