Real World Analytics with Solr Cloud and Spark

Real-World Analytics with Solr Cloud and Spark
Solving Analytic Problems for Billions of Records Within Seconds
Vancouver, May 2016 | Johannes Weigend | QAware GmbH
Johannes Weigend
Apache Big Data North America 2016
May 2016

Apache Big Data North America | Vancouver | 05.05.2016 | Johannes Weigend | © QAware GmbH
Any Question?
Ask or Twitter with the Hashtag #cloudnativenerd

 
The Problem We Want to Solve
■Interactive applications with runtimes lower than a second!
■Processing of billions of records (>109 rows / records)

■Continuously import data (near realtime)

■Applications on top of the Reactive Manifesto

Horizontal Scalability can be difficult!
■Horizontal Scalability of functions

■Trivial
■Loadbalancing of (stateless) services (makro- / microservices)
■More users ! more machines
■Not trivial
■More machines ! faster response times
■Horizontal Scalability of data

■Trivial
■Linear distribution of data on multiple machines
■More machines ! more data
■Not trivial
■Constant response times with growing datasets

Hadoop Gives Answers for Horizontal Scalability of
Data and Functions

The Processing of Distributed Data can be Quite Slow!
9
Data Flow
Read Read Read
Filter Filter Filter
Map Map Map
Reduce
foreach()
-> Minutes / Hours
HDFS / NFS / NoSQL

With Former Indexing and Searching,
Less Data has to be Read and Filtered.
10
Filter
Search Search Search
Map Map Map
Reduce
Data FlowFilter Filter
foreach()
-> Seconds/Minutes
Search / NoSQL

Spark
Map Map Map
Reduce
Distributed
Data
Cluster
Processing
Business Layer
Frontend

DEMO

Spark
1. Solr Cloud for Analytics
Filter
Map Map Map
Reduce
Data FlowFilter Filter
Search / NoSQL

■Document based NoSQL database with outstanding search capabilities

■A document is a collection of fields (string, number, date, …)

■Single und multiple fields (fields can be arrays)

■Nested documents

■Static und dynamic scheme

■Powerful query language (Lucene)

■Horizontal scalable with Solr Cloud
■Distributed data in separate shards

■Resilience by the combination of zookeeper and replication

■Powerful aggregations (aka facets)
■Stable —> V 6.0
Cloud

Shard2
The Architecture of Solr Cloud
Solr Server
Zookeeper
Solr ServerSolr Server
Shard1
Zookeeper Zookeeper Zookeeper
Cluster
Solr Cloud
Leader
Scale Out
Shard3
Replika8 Replika9
Shard5Shard4 Shard6 Shard8Shard7 Shard9
Replika2 Replika3 Replika5
Shards
Replicas
Collection
Replica4 Replica7 Replika1 Shard6

Solr Stores Everything in a Single „Table“ (BigTable).  
Searching is Extremely Fast and Powerful.*
Customer Order
*1
Name Amount
Address Product
Type ID Name Address Amount Product K2B
Customer 1 K 1 A 1 - - [3,5]
Customer 2 K 2 A 2 - - [4]
Order 3 - - Z 1 P 1 [1]
Order 4 - - Z 2 P 2 [2]
...
SolrDocument
SolrDocument
SolrDocument
SolrDocument
(*) With 100 million documents per shard, runtimes of queries and aggregations are normally less then 100ms

A Solr Cloud can be Started in Seconds.
■ Create a scheme by reusing an existing set of solr config files

■ There are examples in the installation directory $SOLR_HOME/server/solr/configsets which can be
copied and modified

■ Start solr

■ When the wizzard asks for a collection name use „bigdata2016“ (see above)

■ Make a first test
cp $SOLR_HOME/server/solr/configset/basic_configs
$SOLR_HOME/server/solr/configsets/bigdata2016
$SOLR_HOME/bin/solr start –e cloud
curl localhost:8983/solr/jax2016/query?q=*:*

With the Solr Cloud Collection API,  
Shards can be Created, Changed or Deleted.
■ Create a collection

■ Delete a collection <<SOLR URL>>/solr/admin/collections?action=DELETE&
name=<<name of collection>>
<<SOLR URL>>/solr/admin/collections?action=CREATE&
name=<<name of collection>>&
numShards=16&
replicationFactor=2&
maxShardsPerNode=8&
collection.configName=
<<name of uploaded zookeeper configuration>>
https://cwiki.apache.org/confluence/display/solr/Collections+API

Zookeeper has to be Started First and the Solr Configuration
must be Uploaded to Use a Solr Cloud.
1.Start zookeeper on 2n+1 nodes (odd number)

2.Upload the solr configuration into zookeeper

3.Start solr on n-nodes connected to the zookeeper cluster

4.Create a collection with a number of shards and replicas
$SOLR_HOME/bin/solr start –c -z
192.168.1.100:2181,192.168.1.101:2181,192.168.1.102
$SOLR_HOME/server/scripts/cloud-scripts$ ./zkcli.sh -cmd
upconfig -zkhost
192.168.1.100:2181,192.168.1.101:2181,192.168.1.102 -
confname ekgdata -solrhome /opt/solr/server/solr -confdir /
opt/solr/server/solr/configsets/ekgdata_configs/conf
$ZOO_HOME/bin/zkServer.sh start

Example: Solr Cloud for Analytics of Insurance Data
■Insurance sample data with the following fields
Education IncomeGender
...

Solr Supports JSON Queries per HTTP Post

Term Facets Group and Count a Single Field.
23

Function Facets Aggregate Fields.
24
http://yonik.com/solr-facet-functions/

Pivot Facets Compose Facets into Hierarchies.
25

Solr 6 Supports SQL
■ Solr 6 supports distributed SQL

■ The JDBC Driver is part of the solrj client library

■ A collection is currently mapped as single table.

■ Collection -> Table

■ SolrDocument -> Row

■ Field -> Column

■ The Solr 6.0 is limited, but more functionality is expected in upcoming versions

■ No database metadata, no prepared statements, no mapping to tables per type field

Resilience
■The number of replicas per shard is configurable (replication factor)

■This number corresponds with the number of nodes which can silently
fail

■Zookeeper is the single source of failure, but can also be failsafe by
running multiple instances

■Solr knows all zookeeper instances and can silently switch over to the
next available leader if last connected zookeeper crashes

You Got Everything What You Need! – Or Not?
■Client side processing of solr documents does not scale

■No possibility to run parallel business logic inside solr

■The solr index is not a general purpose store for huge data

■Images

■Videos

■Binaries / large text documents

■No Interface to machine learning or typical statistics libraries (R) ...
28

Spark
Distributed In-Memory Computing
mit Apache Spark
Filter
Map Map Map
Reduce
Data flowFilter Filter
Search / NoSQL

■Distributed computing (100x faster than Hadoop (M/R)

■Distributed Map/Reduce on distributed data can be done in-memory

■Written in Scala (JVM)

■Java/Scala/Python APIs

■Processes data from distributed and non-distributed sources

■Textfiles (accessible from all nodes)

■Hadoop File System (HDFS)

■Databases (JDBC)

■Solr per Lucidworks API

■...
READ THIS: https://www.cs.berkeley.edu/~matei/papers/2012/nsdi_spark.pdf

Cluster
JVM
Worker
Worker
JVM
JVM
JVM
Worker
Master / Yarn / Mesos
JVM
Executor
Executor
JVM
JVM
JVM
Executor
start
start
start
Task
Task(s)
Slave
Slave
Slave
Master
Host
Spark
Context
MasterURL
Resilient
Distributed
Dataset
RDD
Driver Node
creates
Driver Application
Application
uses
Partition
Task(s)
Partition
Task(s)
Partition

A Very First Spark Application

Spark Pattern 1: Distributed Task with Params

Spark Pattern 2: Distributed Read from External Sources

Spark Pattern 3: Caching and Further Processing with RDDs

Spark
Putting all together

Solr & Spark in Action
Filter
Map Map Map
Reduce
DatenflussFilter Filter
Search / NoSQL

How to implement readFromShard()?
■ Several possibilities for that:

■ SolrJ: SolrStream

■ /export Handler kann Massendaten aus SOLR streamen

■ Unterstützt nur JSON Export (Kein Binary Format !)

■ Or: SolrJ cursor marks

■ Or: Custom export handler
http://localhost:8983/solr/jax2016/export?q=*:*&sort=id%20asc&fl=id&wt=xml

LucidWorks has released a Spark/Solr Integration Library. 
https://github.com/lucidworks/spark-solr

1
2
3
4
Lucidworks Solr-Spark
Adapter V 2.1

Logfile Analytics with Solr and Spark
■Histogram of all exception from hosts A,B,C during time
interval D

■Step 1: Search with Solr

■Solr Query (q=*Exception AND (server: A OR server:B OR server:C) AND timestamp
between [1.1.2015, 31.12.2015]
■Step 2: Create a map with key = << exception name >>, value =
count

■Group with Spark

42
1
2
3
4

DEMO
+

Specifications – Intel NUC6i5SYK 
6th generation Intel® Core™ i5-6260U
processor with Intel® Iris™ graphics
(1.9 GHz up to 2.8 GHz Turbo, Dual Core, 4 MB
Cache, 15W TDP)
CPU
32 GB Dual-channel DDR4 SODIMMs
1.2V, 2133 MHz
RAM
256 GB Samsung M.2 internal SSDDISK
! This case is as powerful like four notebooks
8 Cores, 16 HT Units, 128 GB RAM, 1 TB DiskTotal

Technical Cluster Architecture
hdfs
Ubuntu Linux
Solr Cloud
Zookeeper
#1
Spark
Zeppelin
Master JVM Slave JVM
Executor JVM #1
Ubuntu Linux
Solr Cloud
Zookeeper
#2
Spark
Zeppelin
Master JVM #2 Slave JVM #2
Executor JVM #2
Ubuntu Linux
Solr Cloud
Spark
Executor JVM #4
Ubuntu Linux
Solr Cloud
Zookeeper
#3
Spark
Executor JVM #3
s1 s2 s3 s4
s5 s6 s7 s8
s13 s14 s15 s16
s9 s10 s11 s12
1
23
4

You can even run Solr Cloud and Spark on Odroid 4 70$ ARM
Computers
■ 8 Cores

■ ca. 1/10 CPU performance in comparison to the Intel NUC 6 / Core i5

47
SPARK Worker
SOLR 5.3
Odroid XU4
2 GB RAM
64 GB eMMC Disk
Ubuntu Linux
70$
SPARK Worker
SOLR 5.3
SPARK Worker
SOLR 5.3
SPARK Worker
SOLR 5.3
SPARK Master
SOLR 5.3
SPARK Worker
ZOOKEEPER
40 Cores
10 GB RAM
320 GB eMMC Disk

Summary
■Solr Cloud and Spark are a powerful combination for interactive
analytics and data intense applications

■Writing distributed software stays hard. Only distribute if you have to.

■100% Open Source

■A simple integration of Solr and Spark is easy. For high performance
applications things could be more complicated.

■If professional product support is needed, customers can switch to
Lucidworks Fusion to get a pre integrated and supported Solr/Spark
platform

@JohannesWeigend
@qaware
slideshare.net/qaware
blog.qaware.de

Real World Analytics with Solr Cloud and Spark

More Related Content

What's hot

Viewers also liked

Similar to Real World Analytics with Solr Cloud and Spark

More from QAware GmbH

Recently uploaded

Real World Analytics with Solr Cloud and Spark