Hazelcast Jet v0.4 - August 9, 2017

INTRODUCING
1
Rahul Gupta
@wildnez
2

© 2017 Hazelcast Inc. Confidential & Proprietary
What is Jet?
A general purpose distributed data processing engine
built on Hazelcast and based on directed acyclic graphs
(DAGs) to model data flow.
DISTRIBUTED COMPUTING.SIMPLIFIED.

Why?
• Distributed java.util.stream support for Hazelcast
• Extend computational capabilities in Hazelcast
(IExecutorService, EntryProcessor, JobTracker, …)
• Hazelcast provides a nice foundation for distributed data processing so it is
relatively easy to do this.
• Unifying IMDG and advanced data processing capabilities

Hazelcast Jet Overview
4

Hazelcast Jet – Fast Big Data
5
Stream and Fast In-Memory Batch Processing
Enrichment
Databases
IoT
Social
Networks
Enterprise
Applications
Databases/
Hazelcast IMDG
HDFS/
Spark
Stream
Stream
Stream
Batch
Batch
Ingest
Alerts
Enterprise
Applications
Interactive
Analytics
Databases/
Hazelcast IMDG
Output

Hazelcast Jet Architecture
6
Streaming Readers and Writers
(Kafka, File, Socket)
Stream Processing
(Tumbling, Sliding and Session Windows)
java.util.stream
Batch Readers and Writers
(Hazelcast IMDG IMap, ICache & IList, HDFS, File)
Networking
(IPv4, IPv6)
Deployment
(On Premise, Embedded, AWS, Azure, Docker, Pivotal Cloud Foundry)
Hazelcast IMDG Data Structures
(Distributed Map, Cache, List)
Partition Management
(Members, Lite Members, Master Partition, Replicas, Migrations, Partition Groups, Partition Aware)
Job Management
(Jet Job Lifecycle Management, Resource Distribution and Deployment)
Execution Engine
(Distributed DAG Execution, Processors, Back Pressure Handling, Data Distribution)
Cluster Management with Cloud Discovery SPI
(Apache jclouds, AWS, Azure, Consul, etcd, Heroku, IP List, Kubernetes, Multicast, Zookeeper)
Java Client
Jet Core API
Hazelcast Jet Core
Connectors
High-Level API
Processing
Batch Readers and Writers
(Hazelcast IMDG IMap, ICache & IList, HDFS, File)
Batch Processing
(Aggregations, Join)

Hazelcast Jet Key Competitive Differentiators
7
• Works great with Hazelcast IMDG | Source, Sink, Enrichment
• Very simple to program | Leverages existing standards
• Very simple to deploy | Same lower stack as Hazelcast
• Works in every Cloud | Same as Hazelcast IMDG
• High Performance | Fastest word count in town!
• For Developers by Developers | Code it

Jet Application Deployment Options
IMDG IMDG
• No separate process to manage
• Great for microservices
• Great for OEM
• Simplest for Ops – nothing extra
Embedded
Application
Java API
Application
Java API
Application
Java API
Java API
Application
Client-Server
Jet
Jet Jet
Jet
Jet Jet
Java API
Application
Java API
Application
Java API
Application
• Separate Jet Cluster
• Scale Jet independent of Applications
• Isolate Jet from Application Server Lifecycle
• Managed by Ops

Jet with Hazelcast Deployment Choices
Jet Compute
Cluster
Hazelcast
IMDG Cluster
Sink Enrichment
Message
Broker
(Kafka)
Data
Enrichment
HDFS
Jet Cluster
Sink
Source /
Enrichment
Good when:
• Where source and sink are primarily Hazelcast
• Jet and Hazelcast have equivalent sizing needs
Good when:
• Where source and sink are primarily Hazelcast
• Where you want isolation of the Jet cluster

Performance - Latency
10

Performance - Throughput

Performance - ForkJoin
• Jet is faster than JDK’s j.u.s implementation. The issue is in the JDK the
character stream reports "unknown size" and thus it cannot be parallelised.
• If you first load the data into a List in RAM then run the JDK’s j.u.s it comes out
a little faster

Jet is a DAG Processing Engine
13

DAG Processing
• Directed Acyclic Graphs are used to model computations
• Each vertex is a step in the computation
• It is a generalisation of the MapReduce paradigm
• Supports both batch and stream processing
• Other systems that use DAGs: Apache Tez, Flink, Spark…

Example: Word Count
If we lived in a single-threaded world:
1. Iterate through all the lines
2. Split the line into words
3. Update running total of counts with each word
final String text = "...";
final Pattern pattern = Pattern.compile("s+");
final Map<String, Long> counts = new HashMap<>();
for (String word : pattern.split(text)) {
counts.compute(word, (w, c) -> c == null ? 1L : c + 1);
}

Input Output
Still single-threaded execution:
each Vertex is executed in turn
Tokenizer Reducer
Split the text into words
For each word emit (word)
Collect running totals
Once everything is finished,
emit all pairs of (word, count)
(text) (word) (word, count)
We can represent the computation as a DAG

Input
(text) (word)
Output
(word, count)
Tokenizer Reducer
Split the text into words
For each word emit (word)
Collect running totals.
Once everything is finished,
emit all pairs of (word, count)
We can parallelise the execution of the vertices by
introducing concurrent queues between the vertices.

Output
(word, count)
Reducer
By dividing the text into lines, and having multiple threads,
each tokenizer can process a separate line thus
parallelizing the work.
Input
Tokenizer
Tokenizer

(word)
(word)
Input Output
Tokenizer
We only need to ensure the same words
go to the same Reducer.
The Reducer can also be executed in parallel by
partitioning by the individual words.
Tokenizer
Reducer
Reducer

Node
Node
The steps can also be distributed across multiple nodes.
To do this you need a distributed partitioning scheme.
Input Output
Tokenizer
Tokenizer
Reducer
Reducer
Combiner
Combiner
Input Output
Tokenizer
Tokenizer
Reducer
Reducer
Combiner
Combiner

Hazelcast Jet Overview
21

jet-core
• Provides low-latency and high-throughput distributed
DAG execution
• Hazelcast provides partitioning, discovery, networking
and serialization and elasticity
• Jobs are submitted to the cluster and run in an isolated class
loader with spans the lifetime of the job. Many jobs can be
run in the cluster at once. Each vertex in the graph is
represented by a processor which does not need to be
thread safe
• Vertices are connected by edges.
• Processors are executed by Tasklets, which are allocated to
threads.

Processors
• Main work horse of a Jet application – each vertex must have a Processor
• Just Java code
• It typically takes some input, and emits some output
• Also can act as a source or a sink
• Convenience processors for: flatMap, groupAndAccumulate, filter
Vertex tokenize = dag.newVertex("tokenize",
flatMap((String line) -> traverseArray(delimiter.split(line.toLowerCase()))
.filter(word -> !word.isEmpty()))
);
Vertex reduce = dag.newVertex("reduce",
groupAndAccumulate(initialZero, (count, x) -> count + 1)
);
Vertex combine = dag.newVertex("combine",
groupAndAccumulate(Entry<String, Long>::getKey, initialZero,
(Long count, Entry<String, Long> wordAndCount) -> count + wordAndCount.getValue())
);

DAG Execution
• Each node in the cluster runs the whole graph
• Each vertex is executed by a number of tasklets
• Bounded number of execution threads (typically system
processor count)
• Work-stealing between execution threads
• Back pressure applies between vertices

Cooperative Multithreading
• Similar to green threads
• Tasklets run in a loop serviced by the same native thread.
- No context switching.
- Almost guaranteed core affinity
• Each tasklet does small amount of work at a time (<1ms)
• Cooperative tasklets must be non-blocking.
• Each thread can handle thousands of cooperative tasklets
• If there isn’t any work for a thread, it backs off

Cooperative Multithreading
• Edges are implemented by lock-free single producer, single
consumer queues
- It employs wait-free algorithms on both sides and avoids
volatile writes by using lazySet.
• Load balancing via back pressure
• Tasklets can also be non-cooperative, in which case they have a
dedicated thread and may perform blocking operations.

Producer
Tasklet 3
Producer
Tasklet 4
Producer
Tasklet 4
Producer
Tasklet 2
Consumer
Tasklet 2
Consumer
Tasklet 1
Producer
Tasklet 4
Producer
Tasklet 2
Producer
Tasklet 1
Consumer
Tasklet 2
Consumer
Tasklet 1
Producer
Tasklet 1
Consumer
Vertex
Producer
Vertex
Edge
Parallelism=4
Output of Producer Vertex is
connected to input of
Consumer Vertex by
configuration of the Edge.
Thread 1 Thread 2
Execution Service
Tasklets – Unit of Execution
Parallelism=2

Edges
• Different types of edges:
- Variable Unicast – pick any downstream processor
- Broadcast – emit to all downstream processors
- Partitioned – pick based on a key
• Vertices can have more than one input: allows joins and co-
group
• Vertices can have more than one output: splits
• Edges can be local or distributed

Data input and output
• Reader and writers for Hazelcast IMap, ILap and IList
• HDFS
• Kafka
• Socket (text encoding)
• File
• Easy to implement your own – just implement the Processor
interface. If a source, implement complete(). If a sink
implement process();

APIs

Processor API
• Unified API for sinks, sources and intermediate steps
• Each Processor has an Inbox and Outbox per inbound and outbound
edge.
• Two main methods to implement:
• boolean tryProcess(int ordinal, Object item)
• Process incoming item and emit new items by populating the outbox
• boolean complete()
• Called after all upstream processors are also completed. Typically used for sources and
batch operations such as group by and distinct.
• Slight differences in API for cooperative vs non-cooperative processors
• Non-cooperative processors may block indefinitely
• Cooperative processors must respect Outbox when emitting and yield if Outbox is
already full.

Core API – Powerful, Low Level API
DAG describes how vertices are connected to each other:
DAG dag = new DAG();
Vertex source = dag.newVertex("source", readMap(DOCID_NAME));
Vertex docLines = dag.newVertex("doc-lines", DocLinesP::new);
Vertex tokenize = dag.newVertex("tokenize",
flatMap((String line) -> traverseArray(delimiter.split(line.toLowerCase()))
.filter(word -> !word.isEmpty())));
Vertex reduce = dag.newVertex("reduce",
groupAndAccumulate(initialZero, (count, x) -> count + 1));
Vertex combine = dag.newVertex("combine",
groupAndAccumulate(Entry<String, Long>::getKey, initialZero,
(Long count, Entry<String, Long> wordAndCount) ->
count + wordAndCount.getValue()));
Vertex sink = dag.newVertex("sink", writeMap("counts"));
dag.edge(between(source.localParallelism(1), docLines))
.edge(between(docLines.localParallelism(1), tokenize))
.edge(between(tokenize, reduce).partitioned(wholeItem(), HASH_CODE))
.edge(between(reduce, combine).distributed().partitioned(entryKey()))
.edge(between(combine, sink));
jet.newJob(dag).execute();

Distributed java.util.stream API
• “Classes to support functional-style operations on streams of
elements, such as map-reduce transformations on collections.”
• Jet adds distributed support for the java.util.stream API for
Hazelcast Map and List.
• Supports all j.u.s. operations such as:
-map(), flatMap(), filter()
-reduce(), collect()
-sorted(), distinct()
• Lambda serialization is a problem, but is worked around by creating
serializable versions of the interfaces
• j.u.s Streams are converted to Processor API (DAG) for execution

j.u.s API
JetInstance jet = Jet.newJetInstance();
Jet.newJetInstance();
IStreamMap<Long, String> lines = jet.getMap("lines");
Map<String, Long> counts = lines
.stream()
.flatMap(m -> Stream.of(PATTERN.split(m.getValue().toLowerCase())))
.filter(w -> !w.isEmpty())
.collect(DistributedCollectors.toIMap(w -> w, w -> 1L, (left, right) -> left + right));

API Comparison
• Processor API is low-level, powerful and can be used to
solve any problem that can be expressed as a DAG
• java.util.stream is familiar, but only limited functionality.
• All data processing frameworks have their own API
• Some attempts at unification are not major successes:
Cascading, Apache Beam …
• Cascading prototype for Jet

Demo

Roadmap

Hazelcast Jet 0.4 (06/17)
38
Features Description
Distributed Computation
Distributed data processing framework. It’s based on directed acyclic graphs (DAGs) and in-memory
computation.
Distributed java.util.stream Java 8 stream API can be used to describe the Jet Job. The execution is distributed.
Windowing with Event-time
Improved streaming support including windowing support with event-time semantics. Out of the box
support for tumbling, sliding and session window aggregations.
Distributed Data Structures
Distributed implementations of Map, Cache and List data structures highly optimized to be used for the
processing.
Pivotal Cloud Foundry Tile Hazelcast Jet is available in Pivotal Cloud Foundry
Discovery and Networking Hazelcast discovery for finding members in the cluster or in the Cloud.
Hazelcast IMDG Connector
Allows Jet to integrate with both embedded and non-embedded Hazelcast instances to be used for
reading and writing data for distributed computation.
File and Socket Connector Ability to read and write unbounded data from text sockets and files.
Kafka Connector Reads and writes data streams to the Kafka message broker.
HDFS Connector Reads and writes data streams to Hadoop Distributed File System

Hazelcast 2017 Jet Roadmap
39
Features Description
Robust Stream Processing Guarantees for stateful stream processing
Usability Enhancements High-Level API | Tooling
High Performance
Hazelcast Integrations
Streaming Map and Cache events | Projection and Predicate support for
Map source
Management Center Management and monitoring features for Jet.
More Connectors JMS | JDBC

Questions?
Version 0.4 is the current release with 0.5 coming in September
http://jet.hazelcast.org
Minimum JDK 8

Hazelcast Jet v0.4 - August 9, 2017

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Hazelcast Jet v0.4 - August 9, 2017

Similar to Hazelcast Jet v0.4 - August 9, 2017 (20)

Recently uploaded

Recently uploaded (20)

Hazelcast Jet v0.4 - August 9, 2017

Editor's Notes