Surprising Advantages of Streaming - ACM March 2018

© 2018 MapR Technologies 1
Streaming Architecture:
Surprising Advantages
Ellen Friedman, PhD @Ellen_Friedman
Principal Technologist
26 March 2018 #BayAreaACM

Contact Information
Ellen Friedman, PhD
Principal Technologist, MapR Technologies
Committer Apache Drill & Apache Mahout projects
O’Reilly author
Email efriedman@mapr.com ellenf@apache.org
Twitter @Ellen_Friedman #BayAreaACM

A Shift in Thinking
•  https://%
Image 2018 by SpaceX used under creative commons lisc
https://en.wikipedia.org/wiki/File:Elon_Musk%27s_Tesla_Roadster_(40143096241).jpg
Left Image credit: Astronaut Harrison H. Schmitt. Image NASA

Use
a
stream
instead
of

database
as
shared
truth

Traditional Solution – Use a Profile Database
POS
1..n
Fraud
detector
Last card
use

What Happens as You Scale Up?
POS
1..n
Fraud
detector
Last card
use
POS
1..n
Fraud
detector
POS
1..n
Fraud
detector

Shared Database Can Be A Problem
POS
1..n
Fraud
detector
Last card
use
POS
1..n
Fraud
detector
POS
1..n
Fraud
detector
Shared database
causes problems
Big problem is
disagreement about
schema and indexing

Alternative: Use a Stream to Isolate Services
POS
1..n
Fraud
detector
Last card
use
Updater
card activity

Add New Services via the Stream
POS
1..n
Fraud
detector
Last card
use
Updater
Card
location
history
Other
card activity

Changing Implementation Through Isolation
POS
1..n
Last card
use
Updater
POS
1..n
Last card
use
Updater
card activity
Fraud
detector
Fraud
detector

Stream
transport
supports

microservices

Stream-1st Architecture
Real-time
analytics
EMR
Patient Facilities
management
Insurance
audit
A
B
Medical tests
C
Medical test
results
People often start with event
data streamed to a real-time
application (A)
Image © 2016 Ted Dunning & Ellen Friedman from Chap 1 O’Reilly book Streaming Architecture
used with permission

Heart of Stream-1st Architecture: Message Transport
Real-time
analytics
EMR
Patient Facilities
management
Insurance
audit
A
B
Medical tests
C
Medical test
results
The right messaging tool
supports other classes of use
cases (B & C)
Image © 2016 Ted Dunning & Ellen Friedman Ch 1 O’Reilly book Streaming Architecture used with permission

Stream Transport that Decouples Producers & Consumers
P
P
P
C
C
C
Transport Processing
Kafka /
MapR Streams

Features of Good Streaming
•  Persistent
•  Performant
•  Pervasive

Legacy Applications
How Does MapR Solve This?
Big Data 1.0 Applications Next-Gen Applications
MapR Converged Data Platform
High Availability Real Time Unified Security Multi-tenancy Disaster Recovery Global Namespace
Real-Time NoQL Database Stream TransportWeb-Scale Storage

Example
Files
Table
Streams
Directories
Cluster
Volume mount point

With MapR, Geo-Distributed Data Appears Local
stream
Data
source
Consumer

With MapR, Geo-Distributed Data Appears Local
stream
stream
Data
source
Consumer

With MapR, Geo-distributed Data Appears Local
stream
stream
Data
source
ConsumerGlobal Data Center
Regional Data Center

Build a Global Data Fabric
Flexibility & agility to respond as life changes

MapR Streams: Replication Across Data Centers
•  Replication across data centers with
preserved offsets (unlike Kafka)
•  Opens new use cases
Example: Shared inventory in
ad-tech industry
Inventory
model
Global
analytics
Database
Local
state
Inventory
model
Local
state
Data center 1 Data center 2
Central data center

Use Case: Telecommunications
Callers
Towers
cdr data

Streaming in Telecom
•  Data collection & handling happens at different levels
–  tower, local data center, central data center)
•  Batch: Can take 30 minutes per level
•  Streaming: Latency drops to seconds or sub-seconds per level
•  Ability to respond as events occur
•  MapR Streams enables stream replication with offsets across data
centers

Unique to MapR: Manage Topics at Stream Level
•  Many more topics on MapR cluster
•  Topics are grouped together in Stream (different from Kafka)
•  Policies set at the Stream level such as time-to-live, ACEs (controlled
access at this level is different than Kafka)
•  Geo-distributed stream replication (different from Kafka)
Stream
Topic 1
Topic 3
Topic 2
Image © 2016 Ted Dunning & Ellen Friedman from Chap 5 of O’Reilly book Streaming Architecture used with permission

Use Case: Each pump has many sensors
pump
data
Dashboard
C2
topic = p1
p2
p3
p4
p5
p1
p1
p5

Use topics as an organizing principle

Use Case: Financial Services
Data
generator
Transactions
by_stock
by_sender
by_recipient
Transaction
exploder
stock_idx
sender_idx
recipient_idx
Query
service

Modern Manufacturing
Boeing AMRC, University Sheffield
•  IoT enabled “smart tools” on
manufacture floor
•  Respond quickly to new requirements
Images by Bond Bryan Architecture, used with permission

Stream vs Database
•  Can be better for flexibility and multi-tenancy
•  Streams can be 50 – 100x faster than db (no mutation)
•  Faster means less arguments about performance optimization
•  Operations are simpler so works better to share data
•  Don’t have to commit to one type of db: push updates through
stream and let each group use the db they want

MapR Edge Processing with Stream Replication
Data
source
Data
source
Data
source
Report
Data
source
•  Designed to sit at IoT edge
•  Intended to run right next to data- producing
device

MapR Edge
•  Small footprint cluster for remote processing
•  Intended to run right next to the data producing device
–  Unified end-to-end security policy
–  Reliable replication to cloud and data center, even with occasional
connections
•  Small but full MapR data services (files, tables, streams)
–  Normal data protection (snapshots, mirroring, replication)
–  Normal management capabilities (volumes, fine-grained access
control, monitoring)

Who needs MapR Edge?
•  Connected car industry
•  Telecommunications industry
•  Hospitals and medical testing facilities
•  Anyone who benefits from global learning but needs local action
© Ellen Friedman
© Ellen Friedman
©WesAbrams

MapR Edge: Improves Time to Insight
Before Edge With Edge
•  Oil & gas
•  Medical device
•  Car test & dev
48 hours
12 hours
24 hours
< 2 hours
< 15 minutes
< 5 minutes

Additional Resources on Streaming:
O’Reilly book by Ted Dunning & Ellen Friedman © 2016
Read free courtesy of MapR:
https://mapr.com/streaming-architecture-using-apache-kafka-
mapr-streams/
“Streaming Microservices” by Ted Dunning & Ellen Friedman, in
Encyclopedia of Big Data Technologies. Sherif Sakr and Albert
Zomaya, editors. Springer International Publishing, in press 2018.
and

Machine Learning is Everywhere
Image courtesy Mtell used with permission.Images © Ellen Friedman
● ● ●● ●● ●
●
●
●
●● ● ●●●● ●
●
●●● ●● ●●
●
●
●●
●●
●●
●
●
●
●●
●●
●
●
● ●● ●
●
●●
●
● ●● ● ●●
●
●
● ●
●
● ●
●
● ●
●
● ●● ●
●
●●
●
●
●● ●
●
●
●
●
●
● ●● ●●●● ●
●
● ●● ●
●
●
●
●
●
●
●
●
●● ● ●● ●●
●
● ● ● ●●●● ● ●●
●
● ●●●
●
●
●
●
● ●●
●
●● ●●● ● ●
●
●● ●
●
●●
●
●
●
●
●●●●
●
●
● ●
●
●
●
●
●
●
●
●● ●● ●
●
●
● ● ● ●● ●●●●●
●
●
●● ●
● ●
●
● ●●● ●● ●
●
● ●
●
●
●
● ● ●●
●
●
●
● ●
●
●
●●
●
●
●
●
●
● ●
●
●
● ● ●● ● ●● ● ●● ●
●
●
●
●
●
● ●●
●
●
●
●●
●
●● ● ●●
●
●
●● ●● ● ●●
●
●●●
●
● ●●●
●
● ●●● ●
●
●
●
●
●
●
●
●● ●
●
●●
●
●
●
●
●
● ●●
●
● ●●● ● ● ●
● ● ● ●● ●●●
●
● ●
●
●● ● ●●● ● ●●
●
●
●
●
●● ●
●
●
●●● ●
●
●
●● ● ● ●●●●
●
● ●
●
●● ●
●
●
● ●
●
●
●
●● ●●
●
●● ●
●
●● ● ● ●●●● ●● ●● ●●
●
● ●● ● ●
●
●
●●●
●
●
●
●
●
● ●● ●●●
●
● ●●
●
● ●● ●●● ●
●
● ●●
●
●
● ●●● ●● ● ●
●
●
●
●●●
●
●
●● ●●
●
● ● ●
●
● ●●●●
●
●
●
●
●
●
●
●● ●
●
●
●
●● ● ●
●
●
●
●
●
●
●● ●
●
● ●
●
●
●
● ● ● ●● ● ● ●●●● ●●● ●● ●● ●●
●
●
●
●
● ●● ●●
●
●
● ●● ●●● ●● ●● ●●●●
●
●
●●
●
●● ●
●
● ●●●●
●●
●● ●● ●●
●
●● ● ● ●● ●● ●
●
●● ●
●
●
●●
●
●● ● ●
●● ● ●●● ●●● ● ●● ●
●
● ●●
●
● ●● ● ● ●
●●
●●
● ● ●
●
● ●●
●
● ● ●
●
●
●
●●● ● ●●● ●● ● ●●
●
● ●
● ●●
●
● ●
●
● ●● ●●● ●● ●
●
●
●
●● ●
●●
● ●●● ●● ●●
●
●●●● ●● ●
●
●
● ● ●● ●●● ● ● ●
●
●● ●
●
●● ●
●
● ●● ●
●
●● ●●● ●
●
●●● ● ●● ●
●
●
●
●●
●
●
●
● ●● ●●● ●●
●
●
●
●
●
●
●
●
●
●
●●● ●●●● ●●
●
●●●
●
● ●●● ● ●● ●●
●
● ● ● ●
●
●●
●
●●●● ●
●
●● ●●●
●
●●
●
● ●● ●
●
●
●
●
●●
●
●●●● ●● ●
●
●●
●
● ● ● ●
●
●●●●
●
●●●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●● ● ●
●
●
●
●●●
●
●
●
●
●●●● ●●
● ●
●
●
●● ● ●●
●
●● ●● ●
●
●●
●
●●
●
●● ●●
●
●
●
●
●
● ●●
●
● ● ● ●● ●●●● ● ●●● ● ●●
●
●
●
●
●
● ●
●
●●
●
● ●
●
●
●
●
●
● ● ● ●●●
●
●
●●
●
●
●
● ● ●● ●
●
● ● ● ●
● ●
●
●
●
●
● ●● ●●
●●
●●
●
●● ● ●
●
●
●
● ●● ●●●● ●
●
●
●● ●●
●
●●
●
●●● ●●
●
●●
●
● ●
●
●
●
●
●
● ● ●●●
●
●●
●●●●
●
● ●
●
●●
●
●
●
●
●● ● ●●●
●
●
● ●
●
●
●●
●
●
● ●
●
●● ●
●
●● ●● ●● ●●
●
● ●● ● ●
●
●
●
●●
●
●
●
● ●
●
● ● ●●
●
● ●●
●
● ●
●
●
●
●●● ●●
●
●●●● ●●● ●● ●● ● ●● ● ● ●●
●
●● ● ● ●●● ●●
● ●
●
● ● ●●●●
●●●●● ●●● ● ●●● ●● ● ●● ●●
●
●
●
●● ●●●
●
● ●
●
●
●
●
●
●●●●
●
● ●●● ● ●●● ●●● ●● ●● ●
●●●
●
● ●●●● ●●
●
● ●● ●
●
●● ●
●
●● ●●
●
● ●
●
● ●
●
● ●● ●
●
● ●●●●●● ● ●
●
●
●● ●● ●
●
●
●●
●
●● ●
●
●●●
●
●●● ● ●● ●● ●● ●●● ● ●●●
●
●●
●
●
●● ●●● ●●
●
● ●●●● ● ●●
●
●
●● ●●
●
●● ●● ●● ●
●
●●●
●
●●
●
● ● ●●●●● ● ●●
●
●
●
●● ● ●●● ●● ●●
●
●
●
●
●
● ●●●
●
● ●
●
●●● ● ●
●
●
●
● ●
●
●
●
● ●● ●● ●● ● ●
●
●● ●●
●
●●
● ●
●
●
●
●
●
●
● ●● ● ●●● ● ●
●
●●● ●
●
●
●
●
●
●●● ●
●
● ●
●
●
●
●●●
●
● ●
●
●
●
●
● ● ● ●● ●
●
●●● ●● ●●
●
●
●
● ● ●●
●●●
●
●
●
●
● ● ●● ●
●
●
● ●● ●●
●
●
●
●●
●
●● ●
●
●
●●●
●
●● ● ●● ● ●● ●●●
●
●
●
● ● ●
●
●● ●●
●●
●
● ●●●●
●
● ●
●
●
●●
●
●
●
●
●
● ●●
●●
●●
●
●
●●
● ●
●
●●● ● ●
●
●
● ●
●
●●● ●
●
●●● ●● ● ●
●
●● ●
●
●
●
● ●
●
●
● ● ●●
●
●
●
●●● ● ●
●
●
●
●●● ●●
●
●
● ●
●
●●
●
● ●
●
●
● ●
●
●● ● ●● ● ●
●
●● ●●●
●
●● ●
● ●
●
●
●● ●
●
● ●● ● ●● ●● ●●
●
● ● ●●
●
●
●
● ●●
●
●● ● ●● ●●● ●●●● ●
●
● ●●● ● ●
●
● ● ●●● ● ●
●
● ● ●
●
●●● ●● ●●● ●●
●
●●
●● ●●● ●● ●
●
● ●
● ●● ●
●
●
●● ●
●
●
●● ●●
●
●
●
●●
●
●●● ●
●
●
●
●
●
●
● ●
●
● ●●●● ●
●
●●● ● ●●
●
●●● ●● ● ●●
●
●
●
●● ●●
●
●
●● ●
●
●
●
● ●●●
●
●
●
●
●
● ●●
●
● ●●●● ●
●
●
●
●●
●
●● ● ●● ●
●
●
●
●
●●●● ●
●
●
●
● ●● ●
●
●
●
●●
●
●
● ●
●
●
●
●● ●● ●●●●
●
● ● ●
●
● ●● ●
●
●
● ●●● ●
●
● ● ●
●
●
● ●
●
● ●●●
●
● ●
●
●●
●
● ● ●●
●
● ●● ●●●
●
●
●
● ● ●● ●●
●
●
● ●● ●●
●
●●●● ● ●● ●
●
●● ● ●● ● ●● ●
●
●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●● ● ● ●
● ●
●
● ●●●● ● ●● ● ● ●● ●
●
●
●
● ●● ●●●
●
●
● ●
●
●●●
●
●
● ●
●
●
● ● ●● ●●● ●●● ●
●
●
●●
●
●● ●
●
●
●
●
●
●●
●
●
●
●
● ●●●●
●
●
● ●●●●
●
● ●●● ● ●● ●● ●●
●
●
●
●
● ●●● ● ●● ● ● ●
●
●
●
●●● ●
●
●
●●●● ●●●
●
●● ● ●
●●
●
● ●
●
●●
●
●●●
●
● ●● ●● ● ● ●
● ●
●
●●
●
● ●●● ● ●
●
●
●●
●
● ● ●●
●
● ●
●
●●● ●
●
● ● ●
●●
●● ● ●● ●●● ●● ●●●
●
● ●●●
●
● ●●● ● ●
●
●
● ●●● ●●
●●
●
●
● ●
●
●
●
●●● ● ● ●●●
●
●
●
●
●
●
●●
●
● ●●
●
●●●● ●● ●● ● ●● ● ●●●
●
● ● ●
●
●●
●
● ●● ●●●
●
● ●
●
● ● ●●●
●
●
●
●
●
●
●
●● ●
●
●● ●
●
●
●
●●
●
● ●
●
●●●● ●● ●●● ●
●
● ●●
●
●●
●
●● ●
●
● ●●
●
●
●
●
● ●
●
●
●
●
●
●
●●
●
●
●●
●
● ●● ●
●
●
●
●
●
●
● ● ●
●
●
●●● ●●●● ●● ● ●● ● ●●● ●●
●
● ●● ●
●
●●
●●●● ●
●
● ●●● ● ●
●
●
●●● ● ●
●
●●● ● ●● ●
●
●
● ●● ●●
●
●
●
●● ●● ●
●●
●
●
●
● ●●● ●
●
●
● ●
● ●●●● ●●
●
●
●
●
●●● ● ●●●
●
● ●●● ●● ● ● ●● ●● ●● ●
●
●● ●●
●
●
●
●
● ● ●● ●
●
●
●●●● ●
●
●● ●●● ●●
●
● ●●● ●● ●
●
●
● ●●●
●
● ●●
●
●●● ●
●
● ●● ●● ●● ● ●
●
●●●● ●
●
● ●●
●
●● ●● ● ●
●
●
●● ●● ●● ● ● ●● ●
●
●
●
● ●
●
● ●
●
●
●
●
●
●
●
●●
●
●●
● ●● ●●● ● ●●
●
●
● ●
●
●
●
●
●●●
●
● ●
●
●
●
●
●
●
●
●
●●●
●
● ●●
●
●●●● ●
●
●●
●
●
●
●● ● ●●● ●●● ● ●
●
●
●● ●● ●●
●
●● ●●●
●
●
●
●
●● ●●
●
● ● ●
●
● ●
●
●● ●●
●
●
●
●
●
●
●
●
●
●
●●●● ●●● ●●●
●
●● ●●
●
●
●
●
●
●
●
●●● ●
●
● ●●● ● ●●●●●
●
●●● ●
● ●●● ● ●
● ●
●
●● ●● ●●
●
● ●●
●
●●
●
●
●
●
●
●
●
●
● ●
●
●●●
●
● ● ●●● ●
●
● ●● ●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
● ●● ●
●
●●
●
● ●
●
● ●● ●●● ●
●
●
● ●● ● ● ●● ●● ●
●
●
● ●●● ●●
●
● ●● ● ●●
●
●●
●
● ●●
●
●
●● ●● ●
●
●
●
●●
●
●
●●
● ●
●
●●●● ●
●
●
●
●●●
●
●●●●● ● ●● ● ●
●
●●
●
●
●●● ●●
●
●●
● ●
●
●
●●
● ●
●● ● ● ●
●
● ● ●
●
● ●
●
●●●
●
●●
●
●●● ●●●●
●
●●●
●
●
●● ●●● ● ●
●
●
●
●
●●
●
●● ●● ● ● ●●● ●● ●● ● ●●
●
● ●● ● ●●
●
● ●●
●
●
●
●
●
●
●
●
●
● ●●
●
●●● ● ●
●
●
●
●
●
●● ●●●● ●● ●●
●
●● ●
●
● ●●●
●
●●
●
● ●
●●● ● ●● ●● ● ●
●
● ●●
●
●
● ● ●●● ● ●● ●●● ●●●
●
●
●
●
● ●●
●
●● ● ●
●
●● ●●● ●● ●●● ●
●
●
●
●● ● ●● ● ●
●
●●
●
● ●●
●
●●●● ●
●
●●● ● ●● ●●● ●●
●
●● ●●
●
●
●
●● ●●
●
● ● ●●●●
●
●
●
● ● ●●
●
●● ●
●
●● ●●
●
● ●● ●●
●
●
●●
●
● ●● ●●
●
●
●
● ●●
●
●
●
●
●
● ●●
●
●
●
●
● ● ●● ●●
●
●
●
●
● ●● ●● ●
●
●●● ●●
●
●●
●
● ●
●
●
●● ●●
●
●● ●● ● ● ●
●
●
●
●
●
● ●●
●
●●●
●
●
●
●
●
● ●●● ●●●
●
● ●
●
●● ● ● ●
● ●●● ●● ●●●
●
● ● ●●● ●● ● ●● ●●● ● ●● ●
●
●●
●
●
●
●
● ●
●
●
●●●
●
● ● ●●●●
●
●
●
●●
●
● ●● ●● ● ●
●
●●
●
●
●● ●● ●
●
●
●
●
●
●
●● ●●
●
●
●
● ●●●●
●
●
●
●
●● ●●
●
●● ● ●● ●● ●●
●
●
● ●●● ●● ● ●●●
●
●
●
● ●
●
●
●
●●● ●● ● ●● ● ●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
● ●●● ●
●
●●● ● ●● ●
●
●
●
●
●
●
●● ●●● ● ●
●
●● ● ● ●● ● ●
●
● ●●● ●● ●
●
● ●●
●
●
●
●
● ●
●
●
●● ●●●
●
●
●
●●
● ●
●
●● ● ●● ●● ● ●
●
●●●
●
●● ●● ● ●
●
● ●
●
●
●● ● ●●● ●
●
● ●
●
●
●
●● ● ●●
●
●● ● ●
●●
●
●
●
●
●●● ●●● ●●
●
●
●
● ●
●
● ●●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●●
●
●●●
●
●
●● ●
●
● ●●● ●
●
●
●
●
●
● ●
●
● ●● ● ●
●
● ●
●
●
●
●●●● ●●●
●
●
●
●●
●●
●●
●
●
● ●
●
●
●●●
●
●
●
● ● ●
●
●●● ●
●
●● ● ●● ●● ●● ●
●●
●●
●
● ●
●
●
●
● ●● ●
●
●
●
●
●
●
●
●● ●● ● ●● ●
●
● ●
●
● ●
●
●●●
●
●
● ●
●
● ● ● ●
●
●
●● ● ●● ●●
●
●
●
●
● ● ● ●● ●
●
● ●● ●● ●● ● ●
●
●●
●
●
●
●● ●● ●● ●
●
●
●● ●● ●●●●
●
●
●●
●●
●●
●
●
● ● ●● ●
●●
●●●
●
●
●
●
●
●
●●
●
● ●● ●●● ●●
●
●
● ●
●●
●
●
●
●● ●● ● ● ●● ●●
●
● ●●
●
● ●
●
● ●●●●
●
●●
●
● ●● ●● ●
●
● ●● ●
●
●
●
● ●●● ● ●●●●●
●
●●
●
●●
●
● ●●● ● ●
●
● ●●
●
●
●
●
● ●● ●● ●
●
●● ●
●
●
●●● ●●
●
●
●
●
●
●
●
● ●
●
●
● ● ●●● ●●● ●● ● ●● ●● ●●
●
●
●
●
● ●●● ●●● ●● ● ● ●●● ●
●
● ●
●
●●
●
●
●●
●●
●
●●●
●
●●●
●
● ●
●
●
●●●● ● ●● ●● ●● ●●
●
●
●
●●
●
●
●● ●
●
●●
●
● ● ●● ●●●
●
●
●
●
●
●●
●
●
●
● ● ● ● ●● ●●
●
●
●
●
●
●
● ●● ●● ● ●
●
●●
●
●
●
●
●
●
●● ●● ●● ●
●
●
●
●
●
●
●
●●● ●●●●
●
●
● ●
●
●
● ●
●
●● ●●
●
●
●● ● ●
●
● ●
●
●
●
●
●
●
●
●
● ●●● ●
●
●●
●
●●
●
●●● ●
●
● ●● ●● ●
●
●
●●●
●
●● ●
●
●
●
●
● ●●● ●●●● ●●●
●
●
●
●
●
●● ●
●
● ●
●
●
●
● ●●●●
●
● ●
●
●●
●
●
●● ●●●
●
●●●● ● ●●
●
●
● ● ●●
●
●
● ●●
●
●
●
●
●● ●
●
●
●
●
● ● ●●● ●●
●
●
●
●
●
● ●●
●
● ●
●
● ●● ●
●
●● ●● ●●
●
● ●
●
●
●
● ●● ●●
●
●● ●●● ● ●●●●
●
● ● ●●
●
● ●
●
●
●
●● ●● ●●
●
● ●● ●
●
●● ●
●
●
● ●
●
●
●
●
●
●
●●
●
● ●
●
●
● ●● ●●● ●●● ● ● ●●●
●
●●● ● ●
●
●●
●
●
●
● ●
●
● ●
●
●
● ● ●
●
● ●●●●
●
● ●● ● ●● ●●● ●● ●● ● ●● ●●● ●●● ●
●
●● ●● ●
●
● ●●
●
●
● ● ●●●
●
●● ●● ●● ●●
●
● ●● ●●
●
●
●
●●
●
● ●●● ●● ● ●
●
●● ●
●
● ●
●
●●● ●
●●
●●
●
● ●●
●
●
●
●
●●
●
●
●
●
● ●●
●
●
●
●
●
●
●● ●●●
●
●● ●
●
● ●●
●●
● ●
●●●
● ●●●●● ●● ● ●● ● ●● ●
●
● ●●●● ●● ● ●●●
●
●●● ●
●
● ●
●
● ●
●
●
●
● ●●●
●●
●
●
●
●
●
● ●
●●●
● ●●●
●
●
●
●
●
● ●
●
●
●●● ●●● ●●
●
●
●
●
● ● ●● ●
●
●●● ● ●●● ● ●● ● ●●
●
●●
●
●●●●●
●
●
●
● ●
●
● ●● ●
●
●
●
●
●
●
● ●●
●
●
●
●
● ●
●
● ●●●
●
●
●
● ● ●●● ● ● ●●
●
●
●
● ● ●
●
●
● ●● ● ●
●
● ●
●
●● ●● ●● ●●● ●
●
●● ●●●
●
●
● ●
●
● ●● ● ●●
●●●
●
●
●●● ● ●●
●
● ●● ●
●
● ●● ●● ●● ● ● ● ●
●
● ●●
●
●
●
●● ●●●●
●
●●
●
●
●
●
●
●●● ●
●
●● ●●●● ●●
●
●● ●
●●
● ●● ●● ●● ●●
●
● ●● ●● ●
●
●
●
● ●
●
●
● ●
●
●●
●
● ● ●
●
●
●
●
●●
●
●
●
● ●
●
●●
●
● ●● ●
●
● ●
●●● ●●●● ●● ● ● ●
●
● ●● ●●
●
●●● ●
●
●
●
●
●
●
●
●
●●● ● ●● ●●● ●●● ●● ● ●● ● ● ●● ●● ●
●
●
●
●
●
●
● ● ●● ● ●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●●
●
●●
●
●
● ●
●
● ●●●
●
●
●
●
●● ●●
●●
●●● ●● ●● ●● ●
●
●●● ● ●● ● ●● ●●● ●● ●●
●
●●● ● ●● ●
●
●●● ●
●
● ● ●●● ● ●
●
● ●
●
●●● ● ● ●●● ●
●
●
●● ●●
●
●● ●●
●
●●●
●
●
●●
●
● ●
●
●● ●
●
● ●
●
● ●
●
●
●
●● ●
●
●●● ● ● ●●● ● ●
●
●● ●●● ● ●● ● ●
●
●
●
●
●
● ●●
●
●
●
●
●
● ● ●
●
● ● ●
●
● ●●
●
●
●
● ●● ● ●
●
● ●
●
●
● ●● ●● ●●●
●
● ●●
●
●● ● ●●● ● ●
●
●
●
● ●● ●
●
●●●● ●
●
● ● ● ●●● ●●
●
● ●
●
●
●
● ●
●
●● ●● ●●
●
● ●● ●● ●
●
●
● ●●
●
● ●●●●
●
●
●
●
●
●
●
●
●
●
●
● ●●● ●● ●
●
●● ● ●● ● ● ●● ●●●
●
●
●
● ●●
●●
●
●● ●
●
● ●●● ●● ● ●
●
●● ● ●●
●●●
●
●
●
● ●
●
●●
●
●
●
●
●● ●● ● ●●●●
●
●●
●
●
● ●● ●● ●
●
●● ● ●●
●
●
●
●● ● ●● ●●●●●
●
● ●●
●
● ●●● ● ●● ●● ●●
●
● ●●
●
● ●
●
●
●
● ●
●
● ●●●
●
●
●
●
●
● ●● ●
●
●
●
●
●●
●
● ●● ●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●●●●● ●● ● ●●
●
● ● ●
●
●
●
● ●
●
● ●● ●●
●
●●
●● ●
●
● ● ●● ●● ●●
●
●
●
●●● ● ●● ● ●●● ● ●● ●● ●● ●
●
● ●●● ● ●●
●
● ●
●
● ●●
●
● ●●●
●
●
●
●●●
●
●
●● ● ●● ● ●●●●
●
● ●●
●
●
●
●
●
●●●● ● ●● ●
●
● ●
●
●
●
●
●
●
●● ●●
●
● ●
●
●● ●●● ●●●●
●
●
●● ●●● ● ●
●
●
●
●●
●
●
●
●
●
●
● ●●
●
●
● ●● ●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
● ● ●●● ●●
●
● ● ●
●
●● ●● ● ●● ● ●● ●●● ●
●
●
●
●●●●
●
●
●●●● ● ●
●
●
●
●
●● ●
●
●
● ●
●
● ●●● ● ● ●● ●● ●●
●
●
●
●● ●
●
●
●
● ●●
●
●●
●
●
●
●
●● ● ● ●● ●● ●
●
●● ●●
●
●● ● ● ●●●
●
●
●
● ●
●
● ●●● ●●● ●● ● ● ●
●
● ●
●
● ●● ●●● ● ●● ●●●
●
●●
●
●
●●
●
●● ● ●
●
●
●● ●
●
●
●
●
●
●
●● ● ● ●●
●
●
●
●
●
●
●
● ● ● ●● ●● ● ●●●●●●● ● ●
●
●●
●
●● ●● ● ●
●
● ●
●● ●●●●
●
●
●
●
●
●
●●
●
● ●●●
●
●
● ● ●
●
●
●
●
●
●
●
●
●
●
● ●●
●
● ●
●
● ● ●
●
● ●●
●
●●● ●●● ●●● ●
●
●
●●● ● ●
●
● ●● ● ●● ●
●●●●
●
● ●
● ●
●●
●
●● ●
●
● ●● ● ●●
●
● ●●●
●
● ●● ●● ●
●
●●
●
●● ●● ●
●
● ●
●
● ●● ●
●
●●●
●
●●
●●
●●
●
●● ● ●
●
●
●●●● ●
●
● ●● ●●
●
● ● ● ● ●●
●
● ●
●
● ●●● ●● ● ●
●●
●
●● ●● ●● ● ●●
●
●
●
●●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●●● ● ●● ● ●
●
●●●● ●●● ●●●●● ● ● ●
● ● ●●● ● ●●
●
●● ●●
●
●
●●
●
●
●
●
● ● ●
● ● ●●●● ● ●
●
●●
●
●
●
●
● ●●● ●● ●● ●
●●
● ● ●●
●●●●
●
● ●
●
●●● ● ●
●
●
● ● ●● ●● ● ●●●
●
●● ●● ●●● ●
●
● ●
●
●
●
●● ● ●● ●●● ●●●●● ● ● ●
●
●
●
●● ● ●
●
●
●
●
●● ●●●●●● ●●
●
●●
●
●
●
●● ● ●●● ●
●
●
●
●
●
●
●
●● ●
●
●
●
●
●
●
●●
●
●
●
● ● ●
●
●
● ●● ● ●● ●
●
●●
●
●● ●
●
●●●
●
●●●●● ●●
●
●
●
●
●
●
●●●
●
●●
●
●●●●● ●●
●
●●● ●●
●
●
●●●● ●
●
●
●
●
●
●●
● ● ● ●
●●● ●● ●
●
●● ● ●
●
●● ●●
●
●● ●●●●●● ●
●
●
●●
●
●●
●
●
●
● ●● ●
●
●●● ●● ●
●
● ● ●
●
●
●● ●●
●
●
●
● ●●●
●
●
●
● ●●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●●
●
●
●● ● ●● ●● ●● ●●●
●
●
●●● ●
●
●
● ●● ●● ● ●●● ●●
●
●
●
●
●●
●
●
●
●●
●
● ● ●●
●
● ●● ● ●● ●
●
● ●●● ● ●●
● ●
●
●
●
●
●
● ● ●● ●
●
●●●
●
●●● ●
●
● ● ●●
●
●
●
●
●
●
●
●
●
●
● ● ●●●● ● ●● ●●● ● ● ●
●●
●
●
● ●●● ●●●
●
● ● ●●
●
● ●
●
●● ● ●
●
●
●●
●●
● ●●● ●●● ●
●
●
●● ● ●●● ●
●
● ● ●●● ●
●
● ●●
●
●
●
●
●
●
●● ●●
●
●●
●
●
● ●●● ●●
● ●●
● ●●
●
●● ● ●●●
●
● ●● ●● ●●
●
●●
●●
● ● ●
●
●●
●
●
●
●●● ●
●
●
●
● ●
●
● ● ●
● ●
●
● ●●
●
●●●● ●●
●
●
●
●
●
●
●● ●●
● ●
●
●
●●● ●● ● ●
●
● ●
●
●
●
● ●
●
●
● ●● ● ●● ● ●●● ●
●
●● ●
●
●
●
●
●●
●
●
●●●
●
●
●●
●
● ● ●● ●● ●●● ●● ●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
● ● ●● ●
●
● ●
●
●● ● ● ● ●
●
●●
●
●
●
● ●
●
● ●● ● ●● ● ● ●● ●
●
● ●●
●
●● ●
●
●
● ●●
●
●
●●● ●●
● ●
● ● ●●● ● ●●●
●
●● ●● ●●● ●●● ●
●
●
● ● ●
●
● ●
●
● ● ●
●
●
●●
●
●
● ●● ●● ●●● ● ●
●
●
●
●
●●● ● ●● ●
●
●● ●●●●
●
●●
●
●●
●
● ●
●●●● ●●
●
● ●●●
●
● ●● ● ●●●
●
●●
●
● ●
●
●
●●●
●
● ●● ●
●
●
●
●●
●
● ●●
●
●
● ●●
●
● ●
●
● ●●
●
●
●
●
●
●● ●●
●
● ●● ●● ●● ●● ●
●
● ●
●
●
●
●
●● ●
●
●
● ● ●●● ●
●
●
●
●●
●
● ● ●
●
●
●
●● ●
●
●
● ● ●
●
●
●●
●
●● ● ●●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
● ●
●
●
●
●●
●
● ●
●
●
● ●
●
●
●● ●●●● ●● ●●● ●● ●● ●
●
●● ●
●
●
●
●
●
●
●
●
● ● ●● ● ●●● ●● ●●
●
●●
●
●
●
● ●
●
● ●●●● ●●● ●● ●●
●
● ●●
●
●
●
●
●●● ●
●
●
●
●●
●
●●● ● ●● ●● ●
●
●
●
●
●
●● ●
●
● ●●
●
●
●
●● ●
●
●● ● ●
●
●
●
● ●
●
● ●● ●
●
●
● ●
●● ●
●
●●●
●
●●
●
● ● ●● ● ●
●
●
●
●● ●● ●● ●
● ●
●●
●
●
●
● ●● ●● ●
●
● ●●●● ●● ●
●
●● ● ● ●● ●●
●
●
●
● ●●● ●● ●●●
●
●
●
●
●
●
●
●
● ● ●●
●
● ●●● ●
●
●
●
●●
●
●●● ●
●
●
●● ●● ●●● ●
●
●
●
●
●
●●
●
●
●
● ●● ● ●● ●●● ● ●
●
●
● ●
●
● ●● ●●●
●
● ●
●
●● ● ●● ●● ●●● ●
●
● ● ●● ●●
●
●●
●
●● ●● ●● ●
●
●
●
●
●
●●● ●●
●
●●● ●●
●
●●
●
●
●●●● ●
●
●
● ●●●
●
● ● ●● ●● ●●
●
●
●
●
●●
●●
●
●●
●
●● ●
●
●
●
●
●
●
●● ●
●
● ●●●
●
●●
●
●
●●● ●● ● ●● ● ●●●●● ●●
●
●●● ● ●● ●●●
●
●●● ●
●
●●●
●
●
●
●
●●● ●● ●
●
●
●
●● ●●
●
●
●●
●●
●
●
●●
●
●●● ●●● ●
●
● ●
●
● ●● ●
●
● ●
●
●
●●
●
●●
●
●●●
●
●
●
● ●
●
● ●
●
●
● ● ●●●● ●
●
●● ●● ●● ●
●
●●● ● ●● ●
●
●● ●● ●● ●● ●
●
●
●
●● ●●●● ●● ●
●●●
●
●
●●
●
●
●
● ●
●
●● ●
●
●
●●● ● ●● ●●●● ● ●● ●● ●●● ●● ● ●
●
●
●
● ●●● ●●
●
●
●
●
●
●
● ●●● ● ●● ●● ●●● ●●
●
●●
●
●●●●
●
●● ●
●
●
●
● ● ●●●● ●●● ● ●●● ●●
●
●●
●
●
●
● ●
●
● ●
●
●●
●
● ●●
●
●
●●
●
●
●
●● ●●
●
●●● ●
●
● ●● ●●
●
●● ●
●
●● ●
●
●
● ●● ●
●
●●
●
●
●
●●
●
●●
●
●
● ● ●●
●
●
●
●●
●
● ●
●
●● ●●
●
● ● ● ●●● ●●
●
●
●
●
● ●● ●
●
●● ●
●
●
●
●
●
●
●
● ● ●
●
●● ● ●
●
●●
●
● ●●
●
●
●
●
●●
● ● ●● ●●● ●
●
●
●
●
●
●
●●● ●
●
●
● ●
●
● ●
●
●
●
●
●
●
● ●● ●● ● ●●● ●●
●
●
●●
●
●
●
●● ●● ● ●● ●
●
●
●
●
●● ●
●
●●
●
●
● ● ●● ●●● ●● ●● ●●● ● ● ●●
●●●●● ● ●●●● ●
●
●●
●
● ●● ●● ● ●● ● ●● ●●
●
●
●
●● ● ●●●
● ●
●
●
●
●
●
●
●
● ●
●
●●● ●● ●●●● ●●
●
●
●
●
● ●● ●● ● ●●
●
●
● ●●
●
●
●
● ● ●● ●
●
●●
●
●
●
● ● ●●● ●
●
●
●
●●
●
● ●●● ●●●
●
● ●● ●●●●
●
● ●●
●
● ● ●●●
●
●● ●●● ●
●
● ●● ● ●
●
●
●
●●
●
● ● ●
●
●
●
●
●
● ●●● ●
●
● ● ●
●●
●
●
● ●●● ● ●
●
●●● ● ●●
●
● ●● ● ●●
●
●● ●●● ● ●●● ●
●
●
●
● ●● ●● ●● ●
●
● ●
●
●
● ●
● ●●● ●●
●
● ●
●
●
●
●
●
● ●●
●●●●
●
●●
●
● ● ●
●●
●
●● ●●
●
●
●
●
●
●
● ●●
●
●
●
●
● ●● ●● ●● ● ● ●
●
●
●
●● ●● ●●●● ●
●
●
●
●
●
●● ●●● ●● ●● ● ●
●
●●
●
●
● ●
●
● ●●
●
● ●
●
●●
●
●
● ●● ●●
●
●● ●
●
●
●
● ●
●
●● ●
●
●●
●
●
● ●
● ●
●
●
●
●● ●
●
●● ●●● ● ●●
●
●
●
● ●● ●●●● ●
●
●
●
●
● ●
●
●● ● ●● ● ●
●
● ● ●●● ●●●
●
●
● ●
●●
● ●
●
●
●
●
●●
●
●
● ●● ●●●
●
●● ●● ● ● ● ●● ●●● ●● ●● ●
●
● ● ●●
●
●●
●
●
●
●
●
●
● ●●●
●
●● ●● ●●●
●
● ●● ● ● ●
●
●●● ● ●● ●●●● ●●
●
●
●
●● ●●
●
●
● ● ●● ●
●
●
●
● ●
●
●
●
●
●
●
●
●
●●● ● ●
●
●
●
●
● ●● ● ●
●
●
●
●●● ● ●
●
●
●
●
●
●
●
●
●● ●● ●●
●
●
●
● ●●●
●
●
●
●
● ●●
●
●
●
● ●
●
●● ●
●
● ●● ●
●
● ●● ●●
●
●●●●
●
●
●
●
●●●
●
● ●● ●
●
●
●●
●
●● ●
●
●
●●●
●
●● ● ●● ●●
●
●● ● ●● ●● ●
●
● ● ●● ● ●● ● ●
●
●●● ●●
●
●
●
●
● ● ●●● ●
●
●●
●
●●●
● ●
● ●
●
●
●
●●
●
●
●
●
●
●
●● ●
●
● ●● ●●●
● ●
● ●● ●● ● ●●
●
●●● ●●
●
●
●
●
●
●
●
●
● ●● ●●●●● ● ●● ●● ●●
●●
●● ● ● ●● ● ●● ●
●
●
●
●
●
●
● ● ●● ● ● ● ●
● ●● ● ●
●
●● ● ●
●
●● ●●
●
● ● ●●●●
●
● ●
●
●
●
●
●
●● ●● ●
●
● ● ●
●
● ●●
●
●●
●
●
●
●
●
●● ●●
●
● ● ●● ●
●●●
●
● ● ●● ●
●
●
● ●
●
●● ●●
●
●●● ●●● ● ●●
●
● ●●●
●
● ● ●
●●
●
●
● ●●● ● ● ●
●
●
●
●●
●
●● ●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
● ● ●
●
●
●
●
●
●
●● ●● ●
●
●●● ●
●
●
●
●● ●●
●
● ●
●
● ● ●●● ●
●
●
●
●
●
●
●
● ● ● ●
●
●● ● ●
●
●● ●●●● ●● ●
●
● ●● ●● ● ●
●
●●
●
●
●● ●
●
● ●
●
●
●
●●
●
●● ● ● ●●
●
●
●
● ●
●
● ●
●
●
● ● ●●
● ●
● ●● ●●● ●
●
●
● ● ●●
●
●
●
● ●●
●
●
●
● ●
●
●● ●●
●
●●
●
●●● ●
●
●
●● ●●● ●●
●
●●
●
●
● ●
●
●● ●●● ●●● ●● ●
●
●
●
●
●
●
●
●
●●● ● ●●● ●● ●● ●
●
●
●
●
● ●●● ● ●
●●●●● ●● ●● ●
●
● ● ●
●
● ●●
●
●
●
●●
●
●● ● ● ● ●● ●
●
●
● ●●
●●
●
●
●
●●
●
● ● ●
●
●● ●● ●●
●
●
●
●
●
●
●
● ●●●●●
●
● ●● ● ●
●
●
●
●● ● ●●
●
● ●●
●
●
●
● ●●
●
●
●
●
● ●
●
●
●
●●●
●
●
● ●● ●●
●
●●●●
●
● ● ●● ●● ●●
●
●
●
●
● ●● ●●●
●
●
●
●● ●
●
●●
●
● ●● ●● ●●● ●●●
●
● ●
●
● ● ●
●
● ●
● ●● ●● ●●
●
●
●
●
● ●
●
● ●●● ●
●
●
●●●
●
●
●
●● ●● ● ●● ● ●
●
● ●
●
●
● ●
●
● ● ●
●
●
●
●● ●●
●
●
●
●
● ●●
●
●
●● ●
●
●●●
●
●
●
● ●● ●●●
●
●
●
●●●● ●● ● ●
●
●
●
●● ● ● ● ●●
●
●
●
●
●
●
● ● ●
●
●●
●
● ●
●
●
●
●● ●
●
● ● ●●● ● ●
●
●
●●
●
●●
●
●
● ●●● ●●● ●● ●● ●●
●
● ●●●●● ●
●
●
●●
●●● ● ● ●● ●
●
● ● ●●●
●
● ●
●
●
● ●●
●●
●
●
●
●
●
●● ●●●●●
●
● ●
●
● ●●
●
●●● ●
●
● ● ●
●
● ●
●
●●● ●
●
● ● ●● ● ●● ● ●●
●
●
● ● ●●
●
●
●●
● ●
●
●
●
●
●● ●
●
●● ●
●
●
● ●
●
●●●
●
● ●●
●
●
●
●
●
●
●
●
●●●
●
● ●● ●
●
●●
●
●
●
●
●
●●●
●
●
●
● ●
●
●
●●● ● ●● ● ●● ●● ● ●
●
●
●
●
●
●●
●●
●
●●● ●●● ●
●
● ● ●
●
●● ●
●
●●
● ●
●● ●
●
●●● ●
●
● ●
●
●
● ● ●
●
● ●●●
●
●●
● ● ●● ●●●●
●
●● ●
●
●
●●
●
● ●● ●● ●● ●●●
●
●
●●●
●
●● ●
●
●● ●●
●
●
●
●●
●
●
●
●
●
●
● ● ● ● ●●
●
●● ●●● ● ● ●●
●
●
●
●
●
●
● ●●
●
●●
●
●● ●
●
●
●
●
●
●● ● ●●● ● ● ●
●
●
●
● ●●●● ●● ●● ● ●●● ● ●● ●●● ● ●
●
●
●
●
●● ● ●●●●
●
●
●
●
●
● ● ●●● ●●
●
● ●●●●
●
● ●●
●
●
●
● ●● ● ●
●
●
● ●
●
● ● ●●● ●● ●
●
●
●
● ●●
●
● ● ●●● ●
●
● ●
●
●
●
● ●
●
●
●
● ●● ● ●● ●●
●
●
●
●
●
● ●●●
●
● ●●
●
●●
●
●● ● ● ●●●
●
●
●
●
●
● ●● ●
●
●
● ●●●
●
●
●
●
●●● ●●● ●●
●
●● ●
●
●
● ● ●
●
●
●
● ●
●
●
●
● ● ●●
●
● ●
●
●
●
● ●● ●● ● ●●● ● ●
●
●● ●● ● ●
●
●
●
●● ●● ●●
●
●
●● ● ●
●
● ●●●
●
●
● ●● ● ●●● ● ● ●●
●
●
●● ●
●
● ●
●
●
●
●
●
● ●●● ●●
●
● ●
●
●● ●
●
●
●● ●
● ●
●●
●
●● ●●
●
●
●
●● ●● ●
●
● ●● ●●●
●
●
●
● ●
●
●
●
● ●
●
●● ●
●
●
● ●● ●● ● ●● ●
●
●
● ●
●
● ● ●● ● ●●●
●
●●
●
●
●
●● ●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
● ●
●
●●●● ●
●
● ●● ●
●
●● ●
●
●
●
●● ● ●
●
● ●● ●
●
●● ●●
●
●
●● ●● ● ●
●
●
● ● ● ●●● ● ●●
●
●●
●
●●
●
●
●
● ●● ●
●
● ●●● ● ● ●
●
●●● ●● ●
●
●
●
● ●●
● ●
●
●
●
●●
●●
●
●
●●● ●● ●●● ● ●● ●● ●●
●
●●● ●
●
●
●● ● ● ●●
●
● ●●● ●
●
●●● ●●● ● ●● ● ● ●
●●
●● ● ●
●
●
●
●
● ●● ● ●
●
● ●●
●
● ●●
●
●●●
●
● ●● ●●
●
●
●
● ●
●
● ●
●
●●●●● ●● ● ●●●
●
● ●●
●
●
−2 0 2 4
0246
Raw Scores
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
0.0 0.5 1.0
0.00.51.0
Q−Q plot

How do you make machine learning
successful?

Is it the algorithm? the model? the ML tool?
https://mapr.com/blog/tensorflow-mxnet-caffe-h2o-which-ml-best/

90% of effort in successful
machine learning isn’t the
algorithm or the model…
It’s the logistics

New book: how to manage machine learning models
Download free pdf or read free online via @MapR:
https://mapr.com/ebook/machine-learning-logistics/
“Rendezvous Architecture” by Ted Dunning & Ellen Friedman, in
Encyclopedia of Big Data Technologies. Sherif Sakr and Albert
Zomaya, editors. Springer International Publishing, in press 2018.
and

Please support women in tech – help build
girls’ dreams of what they can accomplish
© Ellen Friedman 2015#womenintech #datawomen

Contact Information
Ellen Friedman, PhD
Principal Technologist, MapR Technologies
Committer Apache Drill & Apache Mahout projects
O’Reilly author
Email efriedman@mapr.com ellenf@apache.org
Twitter @Ellen_Friedman #BayAreaACM

Surprising Advantages of Streaming - ACM March 2018

Recommended

Recommended

More Related Content

What's hot

What's hot (14)

Similar to Surprising Advantages of Streaming - ACM March 2018

Similar to Surprising Advantages of Streaming - ACM March 2018 (20)

Recently uploaded

Recently uploaded (20)

Surprising Advantages of Streaming - ACM March 2018