Riak Search 2: Yokozuna
Upcoming SlideShare
Loading in...5
×
 

Riak Search 2: Yokozuna

on

  • 1,093 views

Ricon|West talk about Yokozuna

Ricon|West talk about Yokozuna

Statistics

Views

Total Views
1,093
Views on SlideShare
1,093
Embed Views
0

Actions

Likes
0
Downloads
12
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Riak Search 2: Yokozuna Riak Search 2: Yokozuna Presentation Transcript

  • Eric Redmond @coderoshi Tuesday, November 12, 13
  • [Slide Intentionally Left Blank] Tuesday, November 12, 13
  • A million values isn’t cool. You know what’s cool?... Tuesday, November 12, 13
  • A trillion values spread across lots of nodes that auto-index and you can also search in cool ways like geographically or maybe you could extract from a pdf on the server side if you used the extractor API Tuesday, November 12, 13
  • ┬─┬ ノ ^_^ノ) ( Setup Tuesday, November 12, 13
  • ## The enabled Yokozuna set this 'on'. yokozuna = off riak.conf ## The port number which Solr binds to. yokozuna.solr_port = 10014 ## The port number which Solr JMX binds to. yokozuna.solr_jmx_port = 10013 ## The arguments to pass to the Solr JVM. Non-standard ## arguments, i.e. -XX, may not be portable across JVM ## implementations. E.g. -XX:+UseCompressedStrings. yokozuna.solr_jvm_args = -Xms1g -Xmx1g -XX: +UseStringCache -XX:+UseCompressedOops ## The data under which to store all Yokozuna related data. ## Including the Solr index data. yokozuna.data_dir = ./data/yz Tuesday, November 12, 13
  • ## The enabled Yokozuna set this 'on'. yokozuna = on riak.conf ## The port number which Solr binds to. yokozuna.solr_port = 10014 ## The port number which Solr JMX binds to. yokozuna.solr_jmx_port = 10013 ## The arguments to pass to the Solr JVM. Non-standard ## arguments, i.e. -XX, may not be portable across JVM ## implementations. E.g. -XX:+UseCompressedStrings. yokozuna.solr_jvm_args = -Xms1g -Xmx1g -XX: +UseStringCache -XX:+UseCompressedOops ## The data under which to store all Yokozuna related data. ## Including the Solr index data. yokozuna.data_dir = ./data/yz Tuesday, November 12, 13
  • riak-admin cluster join riak@10.0.0.2 ... and so on... riak-admin cluster plan riak-admin cluster commit Tuesday, November 12, 13
  • riak-admin cluster join riak@10.0.0.2 ... and so on... riak-admin cluster plan riak-admin cluster commit riak-admin security add-user eric 12345 riak-admin security add-user admin 123456 riak-admin security grant yokozuna.search ON index simple TO eric riak-admin security grant yokozuna.admin ON index TO admin Tuesday, November 12, 13
  • require 'riak' client = Riak::Client.new # create index client.create_search_index('simple') # tie the index to bucket 'cats' bucket = Riak::Bucket.new(client, 'cats') bucket.props = { yz_index: 'simple' } Tuesday, November 12, 13
  • Indexing Datatypes Tuesday, November 12, 13
  • UTF8中搜索 UTF8‫חיפוש ב‬ Αναζήτηση σε UTF8 Искать в UTF8 Search in UTF8 Tuesday, November 12, 13
  • Advanced Search aka. Next Level Search Tuesday, November 12, 13
  • solr = RSolr.connect(url: 'http://yokozuna01.bos1:8098/solr/docs') resp = solr.get('select', params: {q: '*:*'}) Tuesday, November 12, 13
  • Tuesday, November 12, 13
  • Facets, Stats, and stuff Tuesday, November 12, 13
  • eDisMax defType=edismax Tuesday, November 12, 13
  • Highlighting Tuesday, November 12, 13
  • Languages Tuesday, November 12, 13
  • Map/Reduce Tuesday, November 12, 13
  • {"inputs": { "module":"yokozuna", "function":"mapred_search", "arg":["docs","title_s:Key* AND language_s:en"] }, "query":[ {"map":{ "language":"javascript", "keep":false, "source":"function(v) { return [1]; }"}}, {"reduce"{ "language":"javascript", "keep":true, "name":"Riak.reduceSum" }} ] } Tuesday, November 12, 13
  • Tuesday, November 12, 13
  • client.create_search_index('simple') bucket = Riak::Bucket.new(client, 'people') bucket.props = { yz_index: 'simple' } bucket = Riak::Bucket.new(client, 'cats') bucket.props = { yz_index: 'simple' } bucket = Riak::Bucket.new(client, 'mountains') bucket.props = { yz_index: 'simple' } Tuesday, November 12, 13
  • Schemas Tuesday, November 12, 13
  • <schema name="default" version="1.5"> <fields> <field name="kinds" type="string" indexed="true" stored="false" multiValued="true" /> <field name="name" type="string" indexed="true" stored="true" /> <dynamicField name="*_ss" type="string" indexed="true" stored="true" multiValued="true"/> <field name="_yz_ed" type="_yz_str" indexed="true" stored="false"/> <field name="_yz_pn" type="_yz_str" indexed="true" stored="false"/> <field name="_yz_fpn" type="_yz_str" indexed="true" stored="false"/> <field name="_yz_vtag" type="_yz_str" indexed="true" stored="false"/> <field name="_yz_node" type="_yz_str" indexed="true" stored="false"/> <field name="_yz_rk" type="_yz_str" indexed="true" stored="true"/> <field name="_yz_rb" type="_yz_str" indexed="true" stored="true"/> </fields> </schema> Tuesday, November 12, 13
  • Fallen Nodes Tuesday, November 12, 13
  • Why? Tuesday, November 12, 13
  • Thermocline Tuesday, November 12, 13
  • Thermocline of Replication Tuesday, November 12, 13
  • Goals Tuesday, November 12, 13
  • But... why? Tuesday, November 12, 13
  • Tuesday, November 12, 13
  • How Tuesday, November 12, 13
  • Yokozuna = Glue Tuesday, November 12, 13
  • yz_extractor.erl yz_json_extractor.erl yz_noop_extractor.erl yz_text_extractor.erl yz_xml_extractor.erl yz_doc.erl Tuesday, November 12, 13 Extractors
  • yz_pb_admin.erl yz_pb_search.erl yz_schema.erl yz_wm_extract.erl yz_wm_index.erl yz_wm_schema.erl yz_wm_search.erl Tuesday, November 12, 13 Interfaces
  • Security Tuesday, November 12, 13
  • Permissions riak-admin security grant yokozuna.admin ON schema TO admin riak-admin security grant yokozuna.admin ON index TO admin riak-admin security grant yokozuna.search ON index TO user riak-admin security grant yokozuna.search ON index wiki TO user Tuesday, November 12, 13
  • yz_pb_admin.erl yz_schema.erl yz_wm_index.erl yz_wm_schema.erl Tuesday, November 12, 13 Administration
  • yz_pb_search.erl yz_wm_search.erl Tuesday, November 12, 13 Searches
  • yokozuna.erl yz_app.erl yz_general_sup.erl yz_kv.erl yz_misc.erl yz_sup.erl Tuesday, November 12, 13 Backend Magic
  • Distribution yz_cover.erl yz_events.erl Tuesday, November 12, 13
  • yz_entropy.erl yz_entropy_mgr.erl yz_exchange_fsm.erl yz_index.erl yz_index_hashtree.erl yz_index_hashtree_sup.erl Tuesday, November 12, 13 AAE
  • Solr code EntropyData.java Monitor.java yz_solr.erl yz_solr_proc.erl yz_solr_sup.erl Tuesday, November 12, 13
  • Stats yz_stat.erl yz_stat_worker.erl Tuesday, November 12, 13
  • aae_test.erl yokozuna_essential.erl yz_errors.erl yz_fallback.erl yz_flag_transitions.erl yz_index_admin.erl yz_languages.erl yz_mapreduce.erl yz_monitor_solr.erl yz_pb.erl yz_rs_migration.erl yz_rt.erl yz_schema_admin.erl yz_siblings.erl yz_solr_start_timeout.erl yz_stat_test.erl yz_wm_extract_test.erl yz_component_tests.erl yz_json_extractor_tests.erl yz_kv_tests.erl yz_misc_tests.erl yz_text_extractor_tests.erl yz_xml_extractor_tests.erl yz_driver.erl yz_file_terms.erl Tuesday, November 12, 13 “Testing only shows the presence, not the absence, of bugs” Dijkstra
  • We Want More • • • • Tuesday, November 12, 13 CRDT Support Simplify custom extractors Bucket Types (?) Users
  • Thanks Eric Redmond @coderoshi Tuesday, November 12, 13