Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Moodle global search: now's the time to jump in


Published on

A quick introduction to Moodle global search, followed by a description of the new features and fixes in Moodle 3.4 (and some of those in Moodle 3.5).

Published in: Education
  • Be the first to comment

  • Be the first to like this

Moodle global search: now's the time to jump in

  1. 1. Moodle global search: now’s a great time to jump in sam marshall, The Open University
  2. 2. 2 Introduction • Key points • You should use Moodle global search • We’ve deployed it at the Open University (live now, but gradual roll-out) • There were lots of improvements to it in Moodle 3.4 • 3.4 released in November 2017; 3.5 should release in May 2018 • This presentation includes MDL numbers for all mentioned changes • Contents • Introduction to Moodle global search • Some problems (before 3.4) • Other improvements and fixes (in 3.4) • Future improvements (3.5) • Conclusion
  3. 3. 3 Credits • I coded most (not all) of the improvements • But Moodle development more than initial coding! • Peer review, integration review, testing, fixing stupid mistakes • Special thanks to search maintainer David Monllaó • Other credits (from tracker issues) Ankit Agarwal Matt Clarkson Jake Dallimore Mihail Geshoski Marcus Green Adrian Greeve Marina Glancy Tim Hunt Eloy Lafuente Simey Lameze Mark Nelson Andrew Nicols John Okely Jun Pataleta Dan Poltawski Matt Porritt Igor Sazonov Matteo Scaramuccia Zig Tan
  4. 4. Moodle global search A brief introduction to Moodle global search: what it does and how it works
  5. 5. 5 What is global search? Show me anything I can access with the word ‘apples’ in
  6. 6. 6 What is global search? OK, here’s a forum with the word ‘apples’ in its name
  7. 7. 7 Global search requires a separate search engine • You have to install a search engine • Apache Solr is the one supported by default • Plugins are available for Elasticsearch and Azure Search • Doesn’t have to be on same server • Companies offer Solr hosting, or you can set it up yourself • Students don’t connect directly to Solr; only your Moodle does • It doesn’t crawl (like Google) • The search engine never requests anything from Moodle • Moodle sends new and modified content to the search engine
  8. 8. 8 Moodle works together with the search engine Student query & access permissions Moodle 3.1+ Apache Solr 5.x / 6.x Content & metadata for indexing Query results Since the last indexing run there were 31 new forum posts, a Page has been modified, and this new PDF uploaded – here’s the content. OK thanks, I’ll index that lot right away. A student searched for ‘apples’. They have access to the following courses and activities. I’ve had a look. Here are 17 results in the allowed locations that match that query.
  9. 9. 9 Specific areas are searched • Search engine only indexes content Moodle sends it • Not everything on-screen is searched • Most course content is searchable • e.g. Label, Page, Forum (including posts), Wiki, Glossary, etc. • Quiz questions (and answers) not included • Third-party plugins need to implement search • Otherwise there won’t be any results from those plugins
  10. 10. 10 Indexing process • Scheduled task Global search indexing • By default runs every half hour • Sends new/modified content to search engine • Keeps track of ‘last indexed’ time for each search area • Everything newer than that is sent • Initial vs. regular indexing • Normally it only sends a few items that changed • If an area has never been indexed, it will send everything • (Insert ominous noise here? No, wait, that was last year’s presentation.)
  11. 11. 11 Scheduled task log example Execute scheduled task: Global search indexing (coretasksearch_index_task) ... started 14:13:24. Current memory use 54.2MB. Processing area: Users No new documents to index. Processing area: Book - chapters No new documents to index. Processing area: External tool - activity information No new documents to index. Processing area: Label Processed 1 records containing 1 documents, in 3.5 seconds. Processing area: Forum - posts Processed 2 records containing 2 documents, in 1.3 seconds. Processing area: Forum - activity information No new documents to index. Processing area: Assignment - activity information No new documents to index. ... used 442 dbqueries ... used 17.165716171265 seconds Scheduled task complete: Global search indexing (coretasksearch_index_task)
  12. 12. 12 Initial indexing • For some search areas this can take a while 2.8 4.5 5.5 43.2 242.8 URL (100k) Glossary entries (500k) Label (200k) File (300k) Forum - posts (15.6m) Time (hours) 10 days solid to index all our forum posts
  13. 13. Some problems Before Moodle 3.4 there were some problems with deploying search, especially for large sites
  14. 14. Perhaps not very inviting?
  15. 15. 15 Long initial indexing caused problems • Indexing task would not stop until complete • If it fails before finishing, it’ll start from beginning next time • Can servers stay up uninterrupted for weeks? • No way to index before showing search box • Students had option to search but the index wasn’t ready In 3.4… MDL-59039 Fixed in 3.4
  16. 16. 16 Restored content was not indexed • If you back up and restore content, the restored copy was not searchable • Searching the copy returned no results • Applies to restore, import, duplicate, etc. • Caused because indexing works by date • Restored copy has the same date as the original In 3.4… • Restored content soon becomes searchable • Uses the normal global search indexing task MDL-55356 Fixed in 3.4
  17. 17. 17 Student could only search enrolled courses • Moodle can be set up so students can access specific courses they are not enrolled in • Often done when a ‘course’ is used for online help, study advice, etc. • These courses were not included in search results In 3.4… MDL-59913 Fixed in 3.4
  18. 18. Looking a lot better!
  19. 19. Other improvements (3.4) Minor search fixes and enhancements in Moodle 3.4
  20. 20. 20 Blocks were not searchable • Developers could not make a custom block searchable • Most blocks do not include ‘content’, but some do In 3.4… • Developers can now make blocks searchable • The standard HTML block is now searchable • Some limitations for technical reasons • I’m not going to go into detail because it is super complicated, but basically, it works for blocks on course pages only MDL-58957 Fixed in 3.4
  21. 21. 21 Indexing fixes and improvements Large initial indexing ran out of memory (on PostgreSQL database only) Incorrect modified dates in database caused indexing to stop working Indexing sometimes crashed, locking the scheduled task, due to the file upload process lacking a timeout Invalid Unicode characters in content caused an inexplicable error message, dependent on PHP version Log output not using standard approach MDL-60915 Fixed in 3.5 MDL-60357 Fixed in 3.4 MDL-60838 Fixed in 3.4 MDL-60943 Fixed in 3.4 MDL-60671 Fixed in 3.4
  22. 22. 22 Search engine integration fixes and improvements Solr connection did not work through web proxy Could not connect to Solr on the same computer via ‘localhost’ address Improvement allows search engine plugins to index content in batches instead of one piece at a time, for improved performance The Moodle Solr plugin was not compatible with PHP 7.2 MDL-60346 Fixed in 3.4 MDL-58915 Fixed in 3.4 Marina Glancy MDL-58654 Fixed in 3.4 Matt Porritt MDL-61281 Fixed in 3.4 David Monllaó
  23. 23. 23 Search area fixes and improvements Results from Label did not point to the right URL in course formats that use separate pages per section Content from Folder was not reindexed after changes Files embedded in various locations, such as activity descriptions, were not previously indexed MDL-58947 Fixed in 3.4 MDL-60105 Fixed in 3.4 Matt Clarkson MDL-59459 Fixed in 3.4 Matt Porritt
  24. 24. Now’s a great time to jump in! But there’s more…
  25. 25. Future improvements (3.5) Other enhancements that are going to be (or at least might be) included in future Moodle releases
  26. 26. 26 Search within activity • Sometimes you want to search a specific location • Global search ‘filter’ option lets you search selected courses • Bit fiddly and only works at course level In 3.5… MDL-60880 Fixed in 3.5 Context-sensitive ‘Search within’ options appear if you search from the header in a course or activity (default still searches everywhere)
  27. 27. 27 Prioritise results by context • What if you want to search everywhere but see results related to the ‘current’ context first? In 3.5, if approved… MDL-59434 Awaiting integration review Again, this option appears if you search from the header in a course or activity
  28. 28. 28 Filter option: Search by group • Global search already restricts group activity results • If you can’t access a group you won’t see results from it • Can’t search specific group in ‘visible groups’ activity In 3.5, if approved… MDL-60953 Awaiting peer review Start typing a group name here, then select it from a pop-up list of matches
  29. 29. 29 Filter option: Search by user • Occasionally useful • E.g. search through tutor’s posts In 3.5, if approved… MDL-61028 Awaiting peer review Start typing a user’s name here, then select it from a pop-up list of matches
  30. 30. 30 Minor improvements The Solr plugin did not support Solr 7.x Section titles and summaries were not searchable Allows gradual reindexing of a search area without deleting the index Administrator actions, like deleting the index, didn’t ask for confirmation first – fun times Improves search page performance by removing Solr schema checks Improves progress information in scheduled task logs MDL-60759 Fixed in 3.5 David Monllaó MDL-61256 Awaiting peer review MDL-60981 Fixed in 3.5 MDL-61076 Awaiting peer review MDL-61102 Awaiting peer review MDL-60857 Fixed in 3.5
  31. 31. 31 Bonus slide (apropos nothing): custom interface • Developers can fairly easily create a custom user interface to the search system • This is the OU’s simplified custom interface
  32. 32. In conclusion… • You should use Moodle global search • There were lots of improvements in Moodle 3.4 • And there will be more in Moodle 3.5 • Any questions?