Presentation by Philip Hider, Charles Sturt University and Barbara Spiller, Australian Council for Educational Research Australia, at Write Edit Index 2015, an Australian conference for editors, indexers, and publishing professionals. This case study focuses on the process of improving subject access to a collection of resources related to the scholarship of teaching and learning in higher education. It describes how existing controlled vocabularies in the education field were evaluated as candidates for adoption. The Australian Thesaurus of Education Descriptors (ATED) was selected and enhanced to meet the specific needs of the OLT Resource Library.
Improving subject access to the Office for Learning and Teaching's resource collection
1. SCHOOL OF INFORMATION STUDIES
Improving subject access
to the Office for Learning and Teaching’s
resource collection
Philip Hider, CSU and Barbara Spiller, ACER
2. SCHOOL OF INFORMATION STUDIES
OLT Resource Library
Total OLT projects: 2012-15 $57.1 million
www.olt.gov.au/resources
3. SCHOOL OF INFORMATION STUDIES
OLT project titles
• A threshold concepts focus to curriculum design
• Effective teaching and support of students from low
socioeconomic status backgrounds
• Enabling and developing leadership in multi-campus
universities
• Enhancing the student experience: transition from VET to
higher education
• Inquiry-oriented learning in science
• Nature and roles of arts degrees in contemporary society
• Tracking student success: who is falling through the
cracks?
4. SCHOOL OF INFORMATION STUDIES
Lack of control
• Poor recall
• synonym problem (including plural v singular,
inconsistent hyphenation, etc.)
• misspellings
• Poor precision
• homonym problem (with lack of conceptual structure)
• No thesaurus to help with query formulation
5. SCHOOL OF INFORMATION STUDIES
Project team
Charles Sturt University
Philip Hider, Barney Dalgarno, Ying-Hsang Liu,
Carole Gerts, Carla Daws & Raylee Macauley
Australian Council for Educational Research
Barbara Spiller, Pru Mitchell and Robert Parkes
University of Wollongong
Sue Bennett
6. SCHOOL OF INFORMATION STUDIES
Pre-test search effectiveness
Recall and precision ratios for current system
• only able to find less than half of relevant
documents in initial searches
average recall = 0.45, n = 48
• only one third of retrieved items are relevant
average precision = 0.33, n = 80
7. SCHOOL OF INFORMATION STUDIES
User survey and logs
• 97% rated topic or subject as a useful field by
which to search the Resource Library
• author was second most useful at 74%
• vast majority of actual searches are on topics
8. SCHOOL OF INFORMATION STUDIES
New schema
Project title
Project acronym
Lead researcher
Co-researcher
Lead institution
Partner institution
Funding body
Project ID
Grant type
Project summary
Year of completion
Topic (controlled)
Discipline of application (controlled)
Identifier
Project website URL
Related project
Resource type
Resource title
Resource author
ISBN
DOI
Year of publication
9. SCHOOL OF INFORMATION STUDIES
Thesaurus candidates
Code Thesaurus name Coverage Updated
1 ATED Australian Thesaurus of Education Descriptors AU, all 2014
2 BET British Education Thesaurus UK, all X
3 EDU ÉDUthès: thésaurus de l’éducation CA , HEd 2001
4 EET European Education Thesaurus EU, all X
5 ERIC ERIC Thesaurus US, all 2015
6 ETT CEDEFOP – European Training Thesaurus EU, VET 2012
7 IBE UNESCO-IBE Education Thesaurus UN, all 2007
8 ScOT Schools Online Thesaurus AU, K-12 2015
9 TESE Thesaurus for Education Systems in Europe EU, all 2009
10 UK “Education terms” from the UK Dept of Education UK, all X
11 VOC VOCED Thesaurus AU, VET X
10. SCHOOL OF INFORMATION STUDIES
Thesaurus evaluation criteria
1. Concepts and terminology
2. Structure
3. Scope
4. Depth
5. Geographic coverage
6. Maintenance
7. Licensing and cost
8. Usability and support
9. Interoperability
11. SCHOOL OF INFORMATION STUDIES
0
10
20
30
40
50
60
70
80
90
ATED BET EDU EET ERIC ETT IBE SCOT TESE UK VOC ED
Interoperability
Support
Cost
Maintenance
Locality
Depth
Scope
Structure
Concepts and
terminology
Expert analysis
12. SCHOOL OF INFORMATION STUDIES
Concept matching - words
ATED BET EDU EET ERIC ETT IBE SCOT TESE UK VOC
TOTAL - Y 19 14 13 10 18 10 15 19 14 15 13
0
2
4
6
8
10
12
14
16
18
20
ATED BET EDU EET ERIC ETT IBE SCOT TESE UK VOC
Total
matches
13. SCHOOL OF INFORMATION STUDIES
Concept matching - phrases
ATED BET EDU EET ERIC ETT IBE SCOT TESE UK VOC
TOTAL - Y 18 13 8 9 13 9 11 10 7 12 12
TOTAL - P 3 7 11 10 8 11 9 10 13 9 7
0
5
10
15
20
25
ATED BET EDU EET ERIC ETT IBE SCOT TESE UK VOC
Partial
Full
17. Australian Thesaurus of
Education Descriptors
• Print, digital and online
• Since 1984 for Australian Education Index
• 10,348 terms - 50% references
www.acer.edu.au/ated
19. Australian Education Index
• Identifiers
• New ATED terms
• Warrant level for new concepts
Record display from A+ Education,
informit
20. SCHOOL OF INFORMATION STUDIES
ATED for OLT Resources
• Mapping exercise
1,593 OLT terms examined
1,160 terms remaining after exact matches removed
• Criteria for inclusion
21. SCHOOL OF INFORMATION STUDIES
Mapping of OLT keywords
Category Definition n
A
Terms conceptually and terminologically adequately covered by
ATED
934
B Terms out of scope, i.e. proper nouns 108
C
Terms in scope but not conceptually covered by ATED, but with a
lack of literary warrant
65
D
Terms in scope but not conceptually covered by ATED, with
sufficient literary warrant
30
E
Terms in scope, conceptually but not terminologically covered by
ATED, lacking literary warrant
2
F
Terms in scope, conceptually but not terminologically covered by
ATED, with literary warrant
21
1,160
Categories of ATED - OLT matched terms
22. SCHOOL OF INFORMATION STUDIES
Dealing with candidate terms
• Proposed structure for 51 candidate terms
• Fit neatly into existing structural hierarchies or
requiring structural changes?
• Scope note
• Broader, narrower, related terms
23. SCHOOL OF INFORMATION STUDIES
28 new concepts for ATED
Acupuncture Learning analytics
Alternative medicine Measures
Associate degrees Online assessment
Authorship Quantity surveying
Avatars Reciprocity
Chiropractic Spatial sciences
Commerce Studios
Design based research Synchronous communication
Distributed leadership Teaching research relationship
Double degrees Threshold concepts
Environmental engineering Traditional Chinese medicine
Environmental science Variables
Final year students Variation theory
Hospitality education Web 2.0
24. SCHOOL OF INFORMATION STUDIES
24 new ATED USE references
• Academic analytics USE Learning analytics
• Assessment for learning USE Formative
evaluation
• Assessment tools USE Measures and
Evaluation
• Authoring USE Programming
• Capabilities USE Ability
• Cultural capital USE Cultural literacy
• Cultural competence USE Cultural literacy
• Digital immigrants USE Digital literacy and
Generation gap
• Digital natives USE Digital literacy and
Generation gap
• Dispute resolution USE Grievance
procedures
• Distributive leadership USE Distributed
leadership
• Educational data mining USE Learning
analytics
• Emerging technologies USE Technological
change
• Exercise science USE Exercise
physiology and Sports medicine
• Histology USE Anatomy
• Interactive multimedia USE Multimedia
and Interactivity
• Landscape architecture USE
Landscaping
• Naturopathy USE Alternative medicine
• Net generation USE Digital literacy and
Generation gap
• Personal digital assistants USE Mobile
devices
• Process systems engineering USE
Chemical engineering
• Real time communication USE
Synchronous communication
• Teaching research nexus USE
Teaching research relationship
• Work integrated learning USE Work
based learning
25. SCHOOL OF INFORMATION STUDIES
Issues - Content
• New discipline areas of academic study
Alternative medicine, Chiropractic
• Higher education field
Associate degrees, Final year students
• General education terms
Capabilities USE Skills
• Information technology
Learning analytics, Online assessment
• Style and syntax
Synchronous communication
26. SCHOOL OF INFORMATION STUDIES
Issues - Methodology
• Problems searching RL
phrase searching, Boolean OR / AND
• Differing perspectives
academics meet information specialists
• Sources of authority
• Discipline headings
Australian Standard Classification of Education
• When is the right time?
• Benefits vs costs
27. SCHOOL OF INFORMATION STUDIES
Conclusions
Outcome 1: Improved discovery of OLT resources
(post-reindexing testing now being carried out)
Outcome 2: Strengthened ATED which should
lead to improved discovery of education
resources through other indexes
28. SCHOOL OF INFORMATION STUDIES
Support for this project has been provided by the Australian
Government Office for Learning and Teaching. The views in
this project do not necessarily reflect the views of the
Australian Government Office for Learning and Teaching.
I’m going to talk about how ATED was modified in order to improve subject access to the OLT’s Resource Library.
But first a few words about ACER and the Cunningham Library.
Established in 1930, the Australian Council for Educational Research, or ACER, is a not-for-profit organisation with the objective of creating and promoting research-based knowledge, products and services to improve learning.
ACER has experienced significant growth in recent years and now has more than 340 staff located in most Australian capitals as well as Dubai and New Delhi, and now in London and Jakarta too.
12 of these work in the Cunningham Library in Melbourne.
The Library has a comprehensive and up-to-date collection of educational research documents, including: over 50,000 book titles; around 3,000 journal titles; as well as government reports, conference proceedings, a test collection and historical material.
The collection includes information relevant to all education sectors - early childhood, schools, higher education, vocational education and training (VET) and professional education.
One of Cunningham Library’s products is the Australian Thesaurus of Education Descriptors or ATED.
ATED is the definitive reference on Australian terminology in the area of educational research.
It is used to index and search the subject matter of the Australian Education Index (AEI), which you might be familiar with as A+ Education, available via Informit. AEI provides index coverage of educational research literature with an Australian connection - an Australian author, or Australian subject matter or published in Australia.
AEI has four subsidiary databases, all produced by Cunningham Library – the Database of Research on International Education; Education Research Theses; Learning Ground, an Indigenous Education Research Database; and BOLDE, the Blended, Online Learning and Distance Education research bank.
ATED also provides subject headings for the ACER library catalogue.
ATED’s primary use is as an in-house indexing tool, but it is also available to the public online free of charge or can be purchased in hard copy or as an electronic dataset that can be embedded into a catalogue.
Other organisations have found ATED useful for indexing documents and websites and organising reading lists and bibliographies.
ATED was first published in 1984. It is now in its 4th edition and is updated on a six-monthly basis.
It currently contains 10,348 terms – around half of which are preferred terms, or descriptors, and half USE references.
This is an example of the entry in ATED for the term ‘Indexing’.
Usual ATED development procedures
When index records are created for the Australian Education Index, as well as adding subject terms from ATED, natural language terms are added into an ‘Identifiers’ field. One way of identifying candidate terms for ATED is to monitor these Identifiers for usage trends and upgrade them to ATED descriptors when warranted. A threshold of around 20 instances of a term as an Identifier is an indication the term should be considered as a new ATED descriptor.
A new term needs to be incorporated into the existing structural hierarchies (or sometimes structural changes need to be made to accommodate a term). The proposed changes are then reviewed by Library staff and subject experts from within ACER, before being finally accepted.
In this AEI record you can see the terms in the Subject field – these are taken from ATED. Below that is the Identifiers field that includes the term ‘Apps’. This is an uncontrolled subject that will be monitored for usage.
OLT RL project
When ATED was selected as the subject vocabulary of choice for the OLT Resource Library, we needed to assess how well the terms in ATED covered the topics in the Resource Library and which (if any) areas needed supplementation.
A mapping exercise was undertaken in which the 1,593 subject keywords in the Resource Library were examined for conceptual and terminological fit with ATED. We asked the following questions of each keyword –
Is this concept represented in ATED? If not it may be a candidate for inclusion as a new descriptor.
If this concept is represented, is the terminology sufficiently different to justify it being added to ATED as a USE reference?
The Project Team used a modified version of Cunningham Library’s usual method for assessing new terms for inclusion. Given the small size of the Resource Library compared to AEI, it would have been difficult for terms to reach the threshold of 20 instances, so in this Project, 10 instances was used as the threshold for new terms and 5 for USE references. Usage in AEI was also taken into consideration. Usage statistics can only ever be a rough guide as there are other factors to consider, such as, ‘How recent are these instances? Is the concept still in current use? Is it on the way in or out? How central is it to the field of education?
The 1,593 terms were examined for exact matches with ATED terms. The exact matches, as well as many near matches that were different only in syntax, were identified and discarded. The remaining 1,160 terms were scrutinised by members of the project team from ACER, with input from domain experts.
This table outlines the results.
The largest category was Category A. These 934 Resource Library keywords were considered to be covered both conceptually and terminologically by ATED, although in many cases 2 or more ATED terms were needed to adequately represent the Resource Library keyword. For example, ‘Academic integrity’ in the Resource Library was covered by the combination of ‘Academic conduct’ and ‘Integrity’ in ATED; and ‘Cross-disciplinary curriculum’ by ‘Curriculum development’ and ‘Interdisciplinary approach’.
Unlike ATED which excludes proper nouns and names, the Resource Library’s subject keywords included them. The 108 terms in Category B were mainly names of organisations, projects and software, such as ‘Entry into Valhalla’, a computer program for teaching legal ethics. These were all regarded as out of scope for this mapping exercise.
The 65 terms in Category C were judged as not being conceptually covered by ATED, but as lacking the literary warrant needed to be considered for inclusion. Many of them were either quite specific or peripheral to the field of education. For example, ‘Rat dissection’, ‘Sonography’ and ‘Chest auscultation’.
The 2 terms in Category E were judged as being conceptually but not terminologically covered by ATED, but lacking the literary warrant to be added as USE references.
The 67 terms in Categories C and E were noted and will be monitored as potential candidate terms in future updates of ATED. In time, sufficient literary warrant may justify their addition.
The 30 terms in Category D were considered to not be conceptually covered by ATED and as having sufficient literary warrant to be considered for immediate inclusion as new descriptors.
Similarly, the 21 terms in Category F, being covered conceptually but not terminologically, had sufficient literary warrant to be considered for immediate inclusion as USE references.
These 51 terms were examined for how they would best be incorporated into ATED. Did they fit neatly into existing structural hierarchies? Or did there need to be structural changes made to accommodate them?
A suggested structure for each term was proposed, including a Scope Note, Broader term/s, Narrower term/s and Related term/s, along with a rationale for the change/s. This was circulated to members of the Project Team, the Project Reference Group and selected members of ACER Library staff for comment and discussion.
This slide shows the proposed entry for a new term – ‘Synchronous communication’.
The outcome of the consultation process was that 28 new terms were added to ATED as descriptors; 24 new terms were added as USE references.
These are the 28 new terms added.
It should be noted that the new terms added were not necessarily taken directly from the Resource Library keywords list. In some cases, new terms needed to be added to ATED to support the new structure being developed – for example, the new descriptor, ‘Measures’, was added as a BT of the existing Measures (Individual) in order to cover the Resource Library concept ‘Assessment tools’. Assessment tools had been used in the Resource Library in the sense of tools for evaluating programs. ‘Evaluation’ needed to be combined with ‘Measures’ rather than Measures (Individual) to cover this.
These are the 24 new USE references added to ATED.
It looks like a pretty straightforward process that we used to arrive at the new terms. But in actual fact a lot of thought, discussion, problem-solving and compromise was involved and I want to talk now about some of the issues we wrestled with.
New discipline areas of academic study
The terms ‘Traditional Chinese medicine’, ‘Acupuncture’ and ‘Naturopathy’ all appeared in the Resource Library keyword list. ‘Alternative medicine’ had been used 15 times in AEI, ‘Complementary medicine’ 5 times and ‘Chinese medicine’ twice, but ATED didn’t include any of these concepts. ‘Alternative medicine’ was added as a new descriptor, with USE refs ‘Alternative therapies’, ‘Naturopathy’ and ‘Complementary therapies’ and the narrower terms ‘Traditional Chinese medicine’, ‘Chiropractic’ and ‘Acupuncture’. This cluster of terms reflects an area of academic research not previously represented in ATED.
There was some discussion as to whether ‘Alternative medicine’ was a subcategory of ‘Medicine’. It was decided that in the field of education, medicine was regarded as a distinct and separate area of study to ‘Alternative therapies’ so both were put under the broader ‘Health sciences’.
There was also discussion around the term ‘Chiropractic’ – firstly as to the legitimacy of the word as a noun (confirmed by Chiropractors’ Association of Australia) and secondly around whether it was sufficiently mainstream to be in the ‘Medicine’ hierarchy or whether it was still considered an alternative therapy. The Australian Standard Classification of Education (ASCED) classifies Chiropractic as a rehabilitation therapy rather than a complementary therapy but the US National Library of Medicine Subject Headings (MeSH) includes it in their Complementary therapy hierarchy and the Project team preferred that analysis.
Both ‘Chinese medicine’ and ‘Traditional Chinese medicine’ had been used to describe the same practice. It was decided to use ‘Traditional Chinese medicine’ as the preferred term as ‘Chinese medicine’ could be misconstrued to mean Western medicine practiced in China.
Other new areas of academic study that emerged were hospitality education, environmental science, environmental engineering, and spatial sciences as an umbrella term for surveying, cartography and geographic information systems.
Higher education field
Not surprisingly, the Resource Library added depth to ATED in the higher education area, with the addition of terms such as: ‘Associate degrees’, ‘Double degrees’, ‘Final year students’ and ‘Work integrated learning’. Although ATED’s remit is education as a whole, it is surprising how many higher education concepts were covered, especially when 2 or more in combination were applied e.g. ‘Graduate study’ and ‘Degrees (Academic)’ in ATED combined to describe the Resource Library’s ‘Higher degree’; and ‘Graduate study’, ‘Degrees (Academic)’ and ‘Student research’ combined to describe ‘Higher degree research’.
General education terms
In the general education field ‘Distributed leadership’, ‘Studios’, ‘Teaching research relationship’ and ‘Variation theory’ constituted useful additions to ATED and are applicable across all levels of education. ‘Capabilities’ was initially proposed as a USE reference to the existing ATED term ‘Ability’, but closer analysis clarified that students’ capabilities are not seen as fixed abilities but as competencies that can be acquired, which led to the more accurate ‘USE Skills’.
Information Technology
This is an area where both concepts and terminology change quickly, so it is not surprising that ATED needed supplementing here. The Resource Library raised the question of ‘Avatars’, ‘Digital immigrants’, ‘Digital natives’, ‘Learning analytics’, ‘Educational data mining’, ‘Interactive multimedia’, ‘Net generation’, ‘Synchronous communication’, ‘Second Life’, ‘Emerging technologies’, ‘Web 2.0’ and ‘Online assessment’.
As ATED already carried the terms ‘Digital literacy’ and ‘Generation gap’ it was decided that both ‘Digital immigrants’ and ‘Digital natives’ were sufficiently catered for and they were added as USE references. ‘Learning analytics’ was deemed to be distinct enough from ‘Data analysis’ to be a term in its own right and sufficiently equivalent to ‘Educational data mining’ to add the latter as a USE reference. As it is used both in the sense of the process of analysing data sets and the results obtained from the analysis it was positioned as a narrower term of both ‘Data’ and ‘Statistical analysis’. ‘Online assessment’ had been used 44 times in AEI. ATED already carried ‘Online tests’ and ‘Student assessment’ but this combination only catered for the online assessment method of tests. Therefore the addition of ‘Online assessment’ was overdue. Adding terms preceded by ‘online’ inevitably raises the question ‘In 5 years time, will all educational methods assume an online component, making the specification of the ‘online’ aspect redundant? Although ‘Second Life’ had been used 48 times in AEI, it was judged as still primarily the name of a piece of software and therefore out of scope.
Style and syntax
Resource Library terms included a mixture of nouns, acronyms and adjectives. When they were brought into ATED they were modified to fit the ATED style of nouns. Eg ‘Synchronous’ was changed to ‘Synchronous communication’.
The term ‘Teaching research nexus’ was used 5 times in the Resource Library. ATED includes many terms containing the word ‘relationship’ – ‘Adult child relationship’; ‘Education work relationship’; ‘Family school relationship’ etc. In keeping with ATED’s overall style, ‘Teaching research nexus’ was changed to ‘Teaching research relationship’ when being brought into the thesaurus.
Problems searching RL
We encountered a major obstacle in determining how many times a keyword had been used in the Resource Library. The number of uses could at best be estimated within a range. For instance, depending on which of two ways it is searched, the term ‘Mini health assessments’ could have been used less than twice or 32 times. Wide variation in results was particularly noticeable when searching for 3-word phrases, leading to the strange situation of ‘Aboriginal culture’ appearing in 3 resources, but ‘Australian Aboriginal culture’ in 39 resources. It seemed that the search strings were interpreted as ‘Aboriginal’ AND ‘Culture’ but ‘Australian’ OR Aboriginal’ OR ‘Culture’. Lack of confidence in Resource Library search results meant that significant weighting was given to how often the term had been used as an Identifier in AEI.
Differing perspectives
On a more cheerful note, the participation of both academic staff and librarians on the Project Team was useful when terms were being scrutinised. While librarians brought their understanding of thesaurus structure to the Project, academics’ first hand knowledge of university practices enabled them to contribute definitions for terms like ‘Design based research’ - a research methodology used to test the efficacy of educational design.
‘Threshold concepts’ had been proposed as a use reference to ‘Fundamental concepts’, but academic staff identified that threshold concepts were concepts a student must understand before progressing further and that these could be quite advanced – unlike fundamental concepts which are more elementary. ‘Threshold concepts’ was added as a new term in its own right.
Sources of authority
Aside from the ERIC Thesaurus (the US equivalent of ATED), other sources were used to help in assessment of terms. The Victorian Education Department confirmed the equivalence of ‘Assessment for learning’ and ‘Formative assessment’. Exercise and Sports Science Australia distinguished ‘Exercise physiology’ from ‘Exercise science’. MeSH confirmed ‘Histology’ as a subcategory of ‘Anatomy’. And the Australian Qualifications Framework supplied a definition of ‘Associate degrees’.
Discipline headings
The new record structure in the OLT Resource Library will include a field for discipline which will be separate from the subject field. It will use terms from ASCED. However, searchers of the Resource Library may expect to find discipline headings in the ‘Subject’ field as well and as ATED does include discipline headings, special consideration was given to providing adequate coverage of the discipline headings that came up in the Resource Library keyword list. We had set the threshold at 10 instances but, for example, ‘Commerce’ was added to ATED on the basis of only two uses in the Resource Library.
When is the ‘right’ time?
It is often difficult to decide when to bring a new concept into a thesaurus. You don’t want to be adding a term that turns out to be a passing fad, but you run the risk of the thesaurus lacking currency if you wait too long to ensure the term is going to be accepted and embedded into current language. You can then be faced with the question, ‘Is it better to be adding late rather than not at all?’ In this project, it emerged that ‘Web 2.0’ was not in ATED. It seemed that the term was too dated to add in 2014 and that the moment had passed for its inclusion, with ‘Web 3.0’ now in circulation. However with 587 instances in AEI it needed to be addressed and was eventually added.
Using a different time frame, although the term ‘Predictor variables’ had been included in ATED since the 1st edition in 1984, the broader term ‘Variables’ had never been added. With AEI registering 19 uses of ‘Variables’, it was decided to finally add it, even though there is nothing new about this concept.
The term ‘Personal digital assistants’ came up in the Project. Even though it had been used 74 times in AEI, it is a technology that came and went and is unlikely to be used again. This is an example where on statistics alone, the term would be added, but likely never be used.
These examples highlight the need to treat usage statistics as a guide but to assess each term on its merits.
Benefits versus costs
In the process of deciding how to include ‘Web 2.0’ it was initially proposed to add it as a USE reference to ‘Internet’ and ‘Interactivity’. However, it is Cunningham Library practice to replace historical usage of Identifiers with any new terms and USE references. If we were to turn ‘Web 2.0’ into a USE reference, we would lose access to all these uses of the term as an Identifier and also lose the capacity to track its use into the future. This seemed too big a price to pay, so it was decided to add ‘Web 2.0’ as a new descriptor in its own right.
In conclusion, we have described how the Project Team went about improving subject access to the OLT Resource Library. Philip focused on the evaluation of subject vocabularies and selection processes and I have talked about the modifications made to ATED to make it work for the Resource Library. The by-product of this work is that ATED has been ‘tested’ against a pool of education literature beyond the AEI and has been strengthened by the addition of new terms.