The document discusses the development of a web application designed to enhance the usability of Solr and Hadoop, focusing on features like user history management, a user-friendly interface, and the addition of analytic capabilities. It includes details on UI layout, widget management, and facet functions for data analysis, alongside enterprise features like security and authentication. The document also touches on integration with Spark for advanced analytics and indexing functionalities.

1. INTERACTIVELY
QUERY AND SEARCH
YOUR BIG DATA
Romain Rigaux

2. GOALS
Build
a
Web
app
Quickly
explore
data
…
with
Solr
make
Solr
/
Hadoop
easier
to
use
+

3. ARCHITECTURE
“Just
a
view”
on
top
of
the
standard
Solr
API
REST

4. HISTORY
V1 USER

5. HISTORY
V1 ADMIN

6. ARCHITECTURE
NEXT!
Lot
of
learning,
UX
Boost
needed
Simple,
don’t
know
it
is
Solr

7. HISTORY
V2 USER

8. HISTORY
V2 ADMIN

9. HISTORY
V2 BETTER UX

10. ARCHITECTURE
/select
/admin/collections
/get
/luke...
/add_widget
/zoom_in
/select_facet
/select_range...
REST AJAX
Templates
+
JS
Model
www….

11. ARCHITECTURE
UI FOR FACETS
Query
Collection
Layout All
the
2D
positioning
(cell
ids),
visual,
drag&drop
Dashboard,
fields,
template,
widgets
(ids)
Search
terms,
selected
facets
(q,
fqs)

12. ADDING A WIDGET
LIFECYCLE
Load
the
initial
page
Edit
mode
and
Drag&Drop
/solr/zookeeper/clusterstate.json
/solr/admin/luke…
/get_collection

13. ADDING A WIDGET
LIFECYCLE
/solr/select?stats=true /new_facet
Select
the
field
Guess
ranges
(number
or
dates)
Rounding
(number
or
dates)

14. ADDING A WIDGET
LIFECYCLE
Query
part
1
Query
Part
2
Augment
Solr
response
facet.range={!ex=bytes}bytes&f.bytes.facet.range.start=0&f.bytes.facet.range.end=9000000&
f.bytes.facet.range.gap=900000&f.bytes.facet.mincount=0&f.bytes.facet.limit=10
q=Chrome&fq={!tag=bytes}bytes:[900000+TO+1800000]
{
'facet_counts':{
'facet_ranges':{
'bytes':{
'start':10000,
'counts':[
'900000',
3423,
'1800000',
339,
...
]
}
}
}
{
...,
'normalized_facets':[
{
'extraSeries':[
],
'label':'bytes',
'field':'bytes',
'counts':[
{
'from’:'900000',
'to':'1800000',
'selected':True,
'value':3423,
'field’:'bytes',
'exclude':False
}
], ...
}
}
}

15. JSON TO WIDGET
{
"field":"rate_code",
"counts":[
{
"count":97797,
"exclude":true,
"selected":false,
"value":"1",
"cat":"rate_code"
} ...
{
"field":"medallion",
"counts":[
{
"count":159,
"exclude":true,
"selected":false,
"value":"6CA28FC49A4C49A9A96",
"cat":"medallion"
} ….
{
"extraSeries":[
],
"label":"trip_time_in_secs",
"field":"trip_time_in_secs",
"counts":[
{
"from":"0",
"to":"10",
"selected":false,
"value":527,
"field":"trip_time_in_secs",
"exclude":true
} ...
{
"field":"passenger_count",
"counts":[
{
"count":74766,
"exclude":true,
"selected":false,
"value":"1",
"cat":"passenger_count"
} ...

16. REPEAT
UNTIL…

17. GAME CHANGER!
Possibilihes
5.1
/
5.2
Analyhc
Facets

18. FACET
FUNCTIONS
Count
Sum
Avg
Percentile
Max
...
Count(id)
Sum(bytes)
Avg(mul(price,
quantity))
Percentile(salary,
50,
90)
Max(temperature)
...

19. FACET
FUNCTIONS

20. SUB “NESTED”
FACETS
top_os
{
type:
term,
field:
os,
limit:
5
}
top_os
{
type:
term,
field:
os,
limit:
5,
facet
:
{
by_country:
{
type:
term,
field:
country
}
}
}

21. FUNCTION + NESTED =
ANALYTICS states
{
type:
term,
field:
state,
facet
:
{
by_month
:
{
type:
range,
field:
time,
start:
“TODAY-­‐6MONTHS”,
end:
“TODAY”,
gap:
“MONTH”,
facet
:
{
avg_sal:
“avg(salary)”
}
}
}
}
states
{
type:
term,
field:
state,
facet
:
{
avg_sal:
“avg(salary)”
}
}

22. OPERATIONS ON
BUCKETS OF DATA
Counts
→
Functions

23. OPERATIONS ON
BUCKETS OF DATA
Nested
→
nD
functions

24. ENTERPRISE
FEATURES
- Access
to
Search
App
configurable,
LDAP/SAML
auths
- Share
by
link
- Solr
Cloud
(or
non
Cloud)
- Proxy
user
/solr/jobs_demo/select?user.name=hue&doAs=romain&q=
- Security
Kerberos
- Sentry
Collection
level,
Solr
calls
like
/admin,
/query,
Solr
UI,
ZooKeeper

25. SEARCH AS ONLY
APP IN HUE
gethue.com/solr-­‐search-­‐ui-­‐only/

26. • Spark
in
your
browser
• Notebooks
• New
REST
Server
SPARK
INDEXING
WHAT

27. • Open
source
REST
for
Spark
Shell
• Runs
locally
or
inside
YARN
• Spark
Scala,
PySpark
and
jar/py
submission
SPARK
INDEXING
WHAT
hsps://github.com/cloudera/hue/tree/master/apps/spark/java

28. SPARK STREAMING
Real
hme!
Spark
Solr

29. • Pytho
• Scala
• Charts
NOTEBOOKS / SHELL
WHAT

30. DEMO
TIME
• Analyze
Bay
area
bike
share
• Visualize
one
year
of
data
• Know
your
users,
predict
behavior

31. MISSED
SOMETHING?
demo.gethue.com

32. • Full
Analyhcs
• Easier
indexing
• Geo
• Export/Share
results
• “More
like
this”
• Solr
Joins,
Solr
SQL
• Spark,
SQL...
integrahon,
Hue
4
WHAT’S NEXT
NEW FEATURES

33. TWITTER
@gethue
USER GROUP
hue-­‐user@
WEBSITE
hsp://gethue.com
LEARN
hsp://learn.gethue.com
THANKS!