The document outlines the development of a web application leveraging Solr and Spark for big data analytics, focusing on enhancing user experience and simplifying access to data through a RESTful API. Key features include dynamic dashboards, widget lifecycles, and advanced querying capabilities with facets and analytics functions. Additionally, it discusses integration with tools like Hue for easier indexing and data visualization.

1. SOLR SEARCH WITH
SPARK FOR BIG DATA
ANALYTICS IN ACTION
Romain Rigaux

2. GOALS
Build
a
Web
app
Quickly
explore
data
…
with
Solr
make
Solr
/
Hadoop
easier
to
use
+

3. ARCHITECTURE
“Just
a
view”
on
top
of
the
standard
Solr
API
REST

4. HISTORY
V1 USER

5. HISTORY
V1 ADMIN

6. ARCHITECTURE
NEXT!
Lot
of
learning,
UX
Boost
needed
Simple,
don’t
know
it
is
Solr

7. HISTORY
V2 USER

8. HISTORY
V2 ADMIN

9. HISTORY
V2 BETTER UX

10. ARCHITECTURE
/select
/admin/collections
/get
/luke...
/add_widget
/zoom_in
/select_facet
/select_range...
REST AJAX
Templates
+
JS
Model
www….

11. ARCHITECTURE
UI FOR FACETS
Query
Collection
Layout All
the
2D
positioning
(cell
ids),
visual,
drag&drop
Dashboard,
fields,
template,
widgets
(ids)
Search
terms,
selected
facets
(q,
fqs)

12. ADDING A WIDGET
LIFECYCLE
Load
the
initial
page
Edit
mode
and
Drag&Drop
/solr/zookeeper/clusterstate.json
/solr/admin/luke…
/get_collection

13. ADDING A WIDGET
LIFECYCLE
/solr/select?stats=true /new_facet
Select
the
field
Guess
ranges
(number
or
dates)
Rounding
(number
or
dates)

14. ADDING A WIDGET
LIFECYCLE
Query
part
1
Query
Part
2
Augment
Solr
response
facet.range={!ex=bytes}bytes&f.bytes.facet.range.start=0&f.bytes.facet.range.end=9000000&
f.bytes.facet.range.gap=900000&f.bytes.facet.mincount=0&f.bytes.facet.limit=10
q=Chrome&fq={!tag=bytes}bytes:[900000+TO+1800000]
{
'facet_counts':{
'facet_ranges':{
'bytes':{
'start':10000,
'counts':[
'900000',
3423,
'1800000',
339,
...
]
}
}
}
{
...,
'normalized_facets':[
{
'extraSeries':[
],
'label':'bytes',
'field':'bytes',
'counts':[
{
'from’:'900000',
'to':'1800000',
'selected':True,
'value':3423,
'field’:'bytes',
'exclude':False
}
], ...
}
}
}

15. JSON TO WIDGET
{
"field":"rate_code",
"counts":[
{
"count":97797,
"exclude":true,
"selected":false,
"value":"1",
"cat":"rate_code"
} ...
{
"field":"medallion",
"counts":[
{
"count":159,
"exclude":true,
"selected":false,
"value":"6CA28FC49A4C49A9A96",
"cat":"medallion"
} ….
{
"extraSeries":[
],
"label":"trip_time_in_secs",
"field":"trip_time_in_secs",
"counts":[
{
"from":"0",
"to":"10",
"selected":false,
"value":527,
"field":"trip_time_in_secs",
"exclude":true
} ...
{
"field":"passenger_count",
"counts":[
{
"count":74766,
"exclude":true,
"selected":false,
"value":"1",
"cat":"passenger_count"
} ...

16. REPEAT
UNTIL…

17. GAME CHANGER!
Possibilihes
5.1
/
5.2
Analyhc
Facets

18. FACET
FUNCTIONS
Count
Sum
Avg
Percentile
Max
...
Count(id)
Sum(bytes)
Avg(mul(price,
quantity))
Percentile(salary,
50,
90)
Max(temperature)
...

19. FACET
FUNCTIONS

20. SUB “NESTED”
FACETS
top_os
{
type:
term,
field:
os,
limit:
5
}
top_os
{
type:
term,
field:
os,
limit:
5,
facet
:
{
by_country:
{
type:
term,
field:
country
}
}
}

21. FUNCTION + NESTED =
ANALYTICS states
{
type:
term,
field:
state,
facet
:
{
by_month
:
{
type:
range,
field:
time,
start:
“TODAY-­‐6MONTHS”,
end:
“TODAY”,
gap:
“MONTH”,
facet
:
{
avg_sal:
“avg(salary)”
}
}
}
}
states
{
type:
term,
field:
state,
facet
:
{
avg_sal:
“avg(salary)”
}
}

22. OPERATIONS ON
BUCKETS OF DATA
Counts
→
Functions

23. OPERATIONS ON
BUCKETS OF DATA
Nested
→
nD
functions

24. SEARCH AS ONLY
APP IN HUE
gethue.com/solr-­‐search-­‐ui-­‐only/

25. • Spark
in
your
browser
• Notebooks
• New
REST
Server
SPARK
INDEXING
WHAT

26. • Open
source
REST
for
Spark
Shell
• Runs
locally
or
inside
YARN
• Spark
Scala,
PySpark
and
jar/py
submission
SPARK
INDEXING
WHAT
hpps://github.com/cloudera/hue/tree/master/apps/spark/java

27. LIVY ARCH
YARN LOCAL
Livy
Server
Livy
REPL
Spark
Contexts
Spark
Worker
Livy
Server
YARN
Master
YARN
Node
Livy
REPL
Spark
Context
/
PySpark
YARN
Node
Spark
Worker
YARN
Node
Spark
Worker
1
2
3
4

28. SPARK STREAMING
Real
hme!
Spark
Solr

29. • Python
• Scala
• Charts
NOTEBOOKS / SHELL
WHAT

30. DEMO
TIME
• Analyze
Bay
area
bike
share
• Visualize
one
year
of
data
• Know
your
users,
predict
behavior

31. MISSED
SOMETHING?
demo.gethue.com

32. • Full
Analyhcs
• Easier
indexing
• Geo
• Export/Share
results
• Solr
Joins,
Solr
SQL
• Spark,
SQL...
integrahon,
Hue
4
WHAT’S NEXT
NEW FEATURES

33. TWITTER
@gethue
USER GROUP
hue-­‐user@
WEBSITE
hpp://gethue.com
LEARN
hpp://learn.gethue.com
THANKS!