1.
Moodle global search: now’s a great
time to jump in
sam marshall, The Open University

2.
2
Introduction
• Key points
• You should use Moodle global search
• We’ve deployed it at the Open University (live now, but gradual roll-out)
• There were lots of improvements to it in Moodle 3.4
• 3.4 released in November 2017; 3.5 should release in May 2018
• This presentation includes MDL numbers for all mentioned changes
• Contents
• Introduction to Moodle global search
• Some problems (before 3.4)
• Other improvements and fixes (in 3.4)
• Future improvements (3.5)
• Conclusion

3.
3
Credits
• I coded most (not all) of the improvements
• But Moodle development more than initial coding!
• Peer review, integration review, testing, fixing stupid mistakes
• Special thanks to search maintainer David Monllaó
• Other credits (from tracker issues)
Ankit Agarwal
Matt Clarkson
Jake Dallimore
Mihail Geshoski
Marcus Green
Adrian Greeve
Marina Glancy
Tim Hunt
Eloy Lafuente
Simey Lameze
Mark Nelson
Andrew Nicols
John Okely
Jun Pataleta
Dan Poltawski
Matt Porritt
Igor Sazonov
Matteo Scaramuccia
Zig Tan

4.
Moodle global search
A brief introduction to Moodle global search: what it
does and how it works

5.
5
What is global search?
Show me anything I
can access with the
word ‘apples’ in

6.
6
What is global search?
OK, here’s a forum
with the word
‘apples’ in its name

7.
7
Global search requires a separate search engine
• You have to install a search engine
• Apache Solr is the one supported by default
• Plugins are available for Elasticsearch and Azure Search
• Doesn’t have to be on same server
• Companies offer Solr hosting, or you can set it up yourself
• Students don’t connect directly to Solr; only your Moodle does
• It doesn’t crawl (like Google)
• The search engine never requests anything from Moodle
• Moodle sends new and modified content to the search engine

8.
8
Moodle works together with the search engine
Student query & access
permissions
Moodle
3.1+
Apache Solr
5.x / 6.x
Content & metadata for
indexing
Query results
Since the last indexing run there were 31 new
forum posts, a Page has been modified, and
this new PDF uploaded – here’s the content.
OK thanks, I’ll
index that lot right
away.
A student searched for ‘apples’.
They have access to the following
courses and activities.
I’ve had a look. Here are 17
results in the allowed locations
that match that query.

9.
9
Specific areas are searched
• Search engine only indexes content Moodle sends it
• Not everything on-screen is searched
• Most course content is searchable
• e.g. Label, Page, Forum (including posts), Wiki, Glossary, etc.
• Quiz questions (and answers) not included
• Third-party plugins need to implement search
• Otherwise there won’t be any results from those plugins

10.
10
Indexing process
• Scheduled task Global search indexing
• By default runs every half hour
• Sends new/modified content to search engine
• Keeps track of ‘last indexed’ time for each search area
• Everything newer than that is sent
• Initial vs. regular indexing
• Normally it only sends a few items that changed
• If an area has never been indexed, it will send everything
• (Insert ominous noise here? No, wait, that was last year’s presentation.)

11.
11
Scheduled task log example
Execute scheduled task: Global search indexing (coretasksearch_index_task)
... started 14:13:24. Current memory use 54.2MB.
Processing area: Users
No new documents to index.
Processing area: Book - chapters
No new documents to index.
Processing area: External tool - activity information
No new documents to index.
Processing area: Label
Processed 1 records containing 1 documents, in 3.5 seconds.
Processing area: Forum - posts
Processed 2 records containing 2 documents, in 1.3 seconds.
Processing area: Forum - activity information
No new documents to index.
Processing area: Assignment - activity information
No new documents to index.
... used 442 dbqueries
... used 17.165716171265 seconds
Scheduled task complete: Global search indexing (coretasksearch_index_task)

12.
12
Initial indexing
• For some search areas this can take a while
2.8
4.5
5.5
43.2
242.8
URL (100k)
Glossary entries (500k)
Label (200k)
File (300k)
Forum - posts (15.6m)
Time (hours)
10 days solid to index
all our forum posts

13.
Some problems
Before Moodle 3.4 there were some problems with
deploying search, especially for large sites

14.
Perhaps not very inviting?

15.
15
Long initial indexing caused problems
• Indexing task would not stop until complete
• If it fails before finishing, it’ll start from beginning next time
• Can servers stay up uninterrupted for weeks?
• No way to index before showing search box
• Students had option to search but the index wasn’t ready
In 3.4…
MDL-59039 Fixed in 3.4

16.
16
Restored content was not indexed
• If you back up and restore content, the restored copy
was not searchable
• Searching the copy returned no results
• Applies to restore, import, duplicate, etc.
• Caused because indexing works by date
• Restored copy has the same date as the original
In 3.4…
• Restored content soon becomes searchable
• Uses the normal global search indexing task
MDL-55356 Fixed in 3.4

17.
17
Student could only search enrolled courses
• Moodle can be set up so students can access
specific courses they are not enrolled in
• Often done when a ‘course’ is used for online help, study
advice, etc.
• These courses were not included in search results
In 3.4…
MDL-59913 Fixed in 3.4

18.
Looking a lot better!

19.
Other improvements (3.4)
Minor search fixes and enhancements in Moodle 3.4

20.
20
Blocks were not searchable
• Developers could not make a custom block
searchable
• Most blocks do not include ‘content’, but some do
In 3.4…
• Developers can now make blocks searchable
• The standard HTML block is now searchable
• Some limitations for technical reasons
• I’m not going to go into detail because it is super complicated,
but basically, it works for blocks on course pages only
MDL-58957 Fixed in 3.4

21.
21
Indexing fixes and improvements
Large initial indexing ran out of memory (on PostgreSQL database only)
Incorrect modified dates in database caused indexing to stop working
Indexing sometimes crashed, locking the scheduled task, due to the file
upload process lacking a timeout
Invalid Unicode characters in content caused an inexplicable error
message, dependent on PHP version
Log output not using standard approach
MDL-60915 Fixed in 3.5
MDL-60357 Fixed in 3.4
MDL-60838 Fixed in 3.4
MDL-60943 Fixed in 3.4
MDL-60671 Fixed in 3.4

22.
22
Search engine integration fixes and improvements
Solr connection did not work through web proxy
Could not connect to Solr on the same computer via ‘localhost’ address
Improvement allows search engine plugins to index content in batches
instead of one piece at a time, for improved performance
The Moodle Solr plugin was not compatible with PHP 7.2
MDL-60346 Fixed in 3.4
MDL-58915 Fixed in 3.4 Marina Glancy
MDL-58654 Fixed in 3.4 Matt Porritt
MDL-61281 Fixed in 3.4 David Monllaó

23.
23
Search area fixes and improvements
Results from Label did not point to the right URL in course formats that
use separate pages per section
Content from Folder was not reindexed after changes
Files embedded in various locations, such as activity descriptions, were
not previously indexed
MDL-58947 Fixed in 3.4
MDL-60105 Fixed in 3.4 Matt Clarkson
MDL-59459 Fixed in 3.4 Matt Porritt

24.
Now’s a great time to jump in!
But there’s more…

25.
Future improvements (3.5)
Other enhancements that are going to be (or at least
might be) included in future Moodle releases

26.
26
Search within activity
• Sometimes you want to search a specific location
• Global search ‘filter’ option lets you search selected courses
• Bit fiddly and only works at course level
In 3.5…
MDL-60880 Fixed in 3.5
Context-sensitive ‘Search
within’ options appear if you
search from the header in a
course or activity (default still
searches everywhere)

27.
27
Prioritise results by context
• What if you want to search everywhere but see
results related to the ‘current’ context first?
In 3.5, if approved…
MDL-59434 Awaiting integration review
Again, this option appears if
you search from the header
in a course or activity

28.
28
Filter option: Search by group
• Global search already restricts group activity results
• If you can’t access a group you won’t see results from it
• Can’t search specific group in ‘visible groups’ activity
In 3.5, if approved…
MDL-60953 Awaiting peer review
Start typing a group
name here, then select
it from a pop-up list of
matches

29.
29
Filter option: Search by user
• Occasionally useful
• E.g. search through tutor’s posts
In 3.5, if approved…
MDL-61028 Awaiting peer review
Start typing a user’s
name here, then select
it from a pop-up list of
matches

30.
30
Minor improvements
The Solr plugin did not support Solr 7.x
Section titles and summaries were not searchable
Allows gradual reindexing of a search area without deleting the index
Administrator actions, like deleting the index, didn’t ask for confirmation
first – fun times
Improves search page performance by removing Solr schema checks
Improves progress information in scheduled task logs
MDL-60759 Fixed in 3.5 David Monllaó
MDL-61256 Awaiting peer review
MDL-60981 Fixed in 3.5
MDL-61076 Awaiting peer review
MDL-61102 Awaiting peer review
MDL-60857 Fixed in 3.5

31.
31
Bonus slide (apropos nothing): custom interface
• Developers can
fairly easily create a
custom user
interface to the
search system
• This is the OU’s
simplified custom
interface

32.
In conclusion…
• You should use Moodle global search
• There were lots of improvements in Moodle 3.4
• And there will be more in Moodle 3.5
• Any questions?