Riak Search 2: Yokozuna

2,408 views

Published on

Ricon|West talk about Yokozuna

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,408
On SlideShare
0
From Embeds
0
Number of Embeds
24
Actions
Shares
0
Downloads
20
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Riak Search 2: Yokozuna

  1. 1. Eric Redmond @coderoshi Tuesday, November 12, 13
  2. 2. [Slide Intentionally Left Blank] Tuesday, November 12, 13
  3. 3. A million values isn’t cool. You know what’s cool?... Tuesday, November 12, 13
  4. 4. A trillion values spread across lots of nodes that auto-index and you can also search in cool ways like geographically or maybe you could extract from a pdf on the server side if you used the extractor API Tuesday, November 12, 13
  5. 5. ┬─┬ ノ ^_^ノ) ( Setup Tuesday, November 12, 13
  6. 6. ## The enabled Yokozuna set this 'on'. yokozuna = off riak.conf ## The port number which Solr binds to. yokozuna.solr_port = 10014 ## The port number which Solr JMX binds to. yokozuna.solr_jmx_port = 10013 ## The arguments to pass to the Solr JVM. Non-standard ## arguments, i.e. -XX, may not be portable across JVM ## implementations. E.g. -XX:+UseCompressedStrings. yokozuna.solr_jvm_args = -Xms1g -Xmx1g -XX: +UseStringCache -XX:+UseCompressedOops ## The data under which to store all Yokozuna related data. ## Including the Solr index data. yokozuna.data_dir = ./data/yz Tuesday, November 12, 13
  7. 7. ## The enabled Yokozuna set this 'on'. yokozuna = on riak.conf ## The port number which Solr binds to. yokozuna.solr_port = 10014 ## The port number which Solr JMX binds to. yokozuna.solr_jmx_port = 10013 ## The arguments to pass to the Solr JVM. Non-standard ## arguments, i.e. -XX, may not be portable across JVM ## implementations. E.g. -XX:+UseCompressedStrings. yokozuna.solr_jvm_args = -Xms1g -Xmx1g -XX: +UseStringCache -XX:+UseCompressedOops ## The data under which to store all Yokozuna related data. ## Including the Solr index data. yokozuna.data_dir = ./data/yz Tuesday, November 12, 13
  8. 8. riak-admin cluster join riak@10.0.0.2 ... and so on... riak-admin cluster plan riak-admin cluster commit Tuesday, November 12, 13
  9. 9. riak-admin cluster join riak@10.0.0.2 ... and so on... riak-admin cluster plan riak-admin cluster commit riak-admin security add-user eric 12345 riak-admin security add-user admin 123456 riak-admin security grant yokozuna.search ON index simple TO eric riak-admin security grant yokozuna.admin ON index TO admin Tuesday, November 12, 13
  10. 10. require 'riak' client = Riak::Client.new # create index client.create_search_index('simple') # tie the index to bucket 'cats' bucket = Riak::Bucket.new(client, 'cats') bucket.props = { yz_index: 'simple' } Tuesday, November 12, 13
  11. 11. Indexing Datatypes Tuesday, November 12, 13
  12. 12. UTF8中搜索 UTF8‫חיפוש ב‬ Αναζήτηση σε UTF8 Искать в UTF8 Search in UTF8 Tuesday, November 12, 13
  13. 13. Advanced Search aka. Next Level Search Tuesday, November 12, 13
  14. 14. solr = RSolr.connect(url: 'http://yokozuna01.bos1:8098/solr/docs') resp = solr.get('select', params: {q: '*:*'}) Tuesday, November 12, 13
  15. 15. Tuesday, November 12, 13
  16. 16. Facets, Stats, and stuff Tuesday, November 12, 13
  17. 17. eDisMax defType=edismax Tuesday, November 12, 13
  18. 18. Highlighting Tuesday, November 12, 13
  19. 19. Languages Tuesday, November 12, 13
  20. 20. Map/Reduce Tuesday, November 12, 13
  21. 21. {"inputs": { "module":"yokozuna", "function":"mapred_search", "arg":["docs","title_s:Key* AND language_s:en"] }, "query":[ {"map":{ "language":"javascript", "keep":false, "source":"function(v) { return [1]; }"}}, {"reduce"{ "language":"javascript", "keep":true, "name":"Riak.reduceSum" }} ] } Tuesday, November 12, 13
  22. 22. Tuesday, November 12, 13
  23. 23. client.create_search_index('simple') bucket = Riak::Bucket.new(client, 'people') bucket.props = { yz_index: 'simple' } bucket = Riak::Bucket.new(client, 'cats') bucket.props = { yz_index: 'simple' } bucket = Riak::Bucket.new(client, 'mountains') bucket.props = { yz_index: 'simple' } Tuesday, November 12, 13
  24. 24. Schemas Tuesday, November 12, 13
  25. 25. <schema name="default" version="1.5"> <fields> <field name="kinds" type="string" indexed="true" stored="false" multiValued="true" /> <field name="name" type="string" indexed="true" stored="true" /> <dynamicField name="*_ss" type="string" indexed="true" stored="true" multiValued="true"/> <field name="_yz_ed" type="_yz_str" indexed="true" stored="false"/> <field name="_yz_pn" type="_yz_str" indexed="true" stored="false"/> <field name="_yz_fpn" type="_yz_str" indexed="true" stored="false"/> <field name="_yz_vtag" type="_yz_str" indexed="true" stored="false"/> <field name="_yz_node" type="_yz_str" indexed="true" stored="false"/> <field name="_yz_rk" type="_yz_str" indexed="true" stored="true"/> <field name="_yz_rb" type="_yz_str" indexed="true" stored="true"/> </fields> </schema> Tuesday, November 12, 13
  26. 26. Fallen Nodes Tuesday, November 12, 13
  27. 27. Why? Tuesday, November 12, 13
  28. 28. Thermocline Tuesday, November 12, 13
  29. 29. Thermocline of Replication Tuesday, November 12, 13
  30. 30. Goals Tuesday, November 12, 13
  31. 31. But... why? Tuesday, November 12, 13
  32. 32. Tuesday, November 12, 13
  33. 33. How Tuesday, November 12, 13
  34. 34. Yokozuna = Glue Tuesday, November 12, 13
  35. 35. yz_extractor.erl yz_json_extractor.erl yz_noop_extractor.erl yz_text_extractor.erl yz_xml_extractor.erl yz_doc.erl Tuesday, November 12, 13 Extractors
  36. 36. yz_pb_admin.erl yz_pb_search.erl yz_schema.erl yz_wm_extract.erl yz_wm_index.erl yz_wm_schema.erl yz_wm_search.erl Tuesday, November 12, 13 Interfaces
  37. 37. Security Tuesday, November 12, 13
  38. 38. Permissions riak-admin security grant yokozuna.admin ON schema TO admin riak-admin security grant yokozuna.admin ON index TO admin riak-admin security grant yokozuna.search ON index TO user riak-admin security grant yokozuna.search ON index wiki TO user Tuesday, November 12, 13
  39. 39. yz_pb_admin.erl yz_schema.erl yz_wm_index.erl yz_wm_schema.erl Tuesday, November 12, 13 Administration
  40. 40. yz_pb_search.erl yz_wm_search.erl Tuesday, November 12, 13 Searches
  41. 41. yokozuna.erl yz_app.erl yz_general_sup.erl yz_kv.erl yz_misc.erl yz_sup.erl Tuesday, November 12, 13 Backend Magic
  42. 42. Distribution yz_cover.erl yz_events.erl Tuesday, November 12, 13
  43. 43. yz_entropy.erl yz_entropy_mgr.erl yz_exchange_fsm.erl yz_index.erl yz_index_hashtree.erl yz_index_hashtree_sup.erl Tuesday, November 12, 13 AAE
  44. 44. Solr code EntropyData.java Monitor.java yz_solr.erl yz_solr_proc.erl yz_solr_sup.erl Tuesday, November 12, 13
  45. 45. Stats yz_stat.erl yz_stat_worker.erl Tuesday, November 12, 13
  46. 46. aae_test.erl yokozuna_essential.erl yz_errors.erl yz_fallback.erl yz_flag_transitions.erl yz_index_admin.erl yz_languages.erl yz_mapreduce.erl yz_monitor_solr.erl yz_pb.erl yz_rs_migration.erl yz_rt.erl yz_schema_admin.erl yz_siblings.erl yz_solr_start_timeout.erl yz_stat_test.erl yz_wm_extract_test.erl yz_component_tests.erl yz_json_extractor_tests.erl yz_kv_tests.erl yz_misc_tests.erl yz_text_extractor_tests.erl yz_xml_extractor_tests.erl yz_driver.erl yz_file_terms.erl Tuesday, November 12, 13 “Testing only shows the presence, not the absence, of bugs” Dijkstra
  47. 47. We Want More • • • • Tuesday, November 12, 13 CRDT Support Simplify custom extractors Bucket Types (?) Users
  48. 48. Thanks Eric Redmond @coderoshi Tuesday, November 12, 13

×