Introducing TRYangle LILAC - Mike Jones

Editor's Notes

• #1 Good morning. My name is Mike Jones and I’m from Southampton Solent University. In this session I’m going to aim to introduce you to the TRYangle Approach, a new framework I’ve been developing for teaching students how to evaluate the information sources they discover as part of their research.
• #2 I want to start off by talking through the context in which this idea came about and why I felt that a new approach was needed. So, cast your minds back to the early days of 2023 when ChatGPT had launched only a few months prior and GenAI tools were popping up all over the place, promising some sort of digital revolution. Educational institutions went into full on panic mode about what this meant for learning, research and assessment and academic librarians were trying to get their heads around what it meant for information literacy. And that’s where you found me, sat in Southampton pondering how I could support my students in navigating this new information landscape. My conclusion, that I ended up writing down in a short blog post on my LinkedIn, was that it required a skills-based approach and that the core triumvirate of skills I already taught my students – how to search for information, how to evaluate it and how to acknowledge its source – would remain key to how we approach AI generated content, it would just require a small shift in perspective. In thinking and writing about this I hypothesised that the middle of these 3 aspects, information evaluation, would become the most important skillset in this new AI age.
• #3 With this in mind I sought to see how other institutions were approaching information evaluation. In the Summer of 2023 I looked at what approaches UK universities were recommending to their students and in the 79 institutions where I could find this information publicly available on their websites and LibGuides, 90% of the approaches being recommended were either the CRAAP Test itself or a similar checklist style approach.
• #4 Criticism of the CRAAP Test and similar checklist approaches had been rife in the literature for sometime…[read through quotes]
• #5 So, in order for evaluation to be central to our GenAI approaches we would finally need to move away from the dominance of the CRAAP Test in that space. For a little while prior to this point I’d been looking at newer approaches and frameworks such as SIFT and CCOW and thinking about how we might incorporate them into our teaching, but thinking about how the landscape had just shifted with the advent of GenAI, I didn’t even feel like these newer approaches would be adequate for a world where, to paraphrase Laura Woods, tools were generating “content” that was devoid of the context of a source.
• #6 So without an existing option that I felt would work for my students I started thinking about what a new information evaluation approach should include
• #7 I think one of the reasons that the CRAAP Test has endured is that there is merit in looking at who has provided the information and when and why, but I wanted to encourage students to move away from only looking at the source and to adopt the Lateral Searching approach embodied most succinctly in SIFT. One of the other approaches I’d been looking at was CCOW and I really liked the addition of “worldview” as a consideration within this and the place of the researcher within the information exchange – get students thinking about confirmation bias and the importance of having a diversity of voices and backgrounds represented in their research. Then came the aspect that I hadn’t seen represented in any of these previous approaches and what I was seeing as essential for information sourced from both social media and GenAI – the idea that it was important to not only look at the information in front of you but to also consider how and why that information has reached you. Initially I was referring to this as being part of algorithmic literacy but in my subsequent reading I discovered Bull, McMillan and Head’s concept of “Proactive evaluation” which fit with what I was thinking.
• #8 Alongside these more skills-based considerations, I also wanted the new approach to encourage students to develop a mindset that would instil confidence in them to challenge the validity and quality of any information source they encountered, an approach I then found validated within the work of Dawkins and Legrand. I was also keen that any new approach wasn’t just about saying that all AI was crap and so I wanted to look at how AI could be leveraged in a positive way to make source evaluation more efficient and effective. The main barrier I faced though was in trying to package all of this into something that could be as memorable as the CRAAP Test. One of the main reasons I think it endures is because it’s a funny mnemonic and so has a high degree of rememberability. For quite some time I thought that the only way to combat this was to come up with a similarly puerile title and there was a period where I’d managed to squeeze the elements into a phrasing that would spell the word FART! Then, in a moment of inspiration, I happened across the idea that perhaps the memorability could instead come from something visual and having a diagrammatical representation of the aforementioned elements.
• #9 And so that is how I arrived at what I’m here to present to you today, The TRYangle Approach!
• #10 So there are 3 aspects to this:
• #11 Question Everything is the mindset within which we want students to be working, instilling in them the confidence to evaluate any information source they encounter
• #12 The Evaluation TRYangle is the three interconnected areas of understanding that students need to be able to develop in order to apply that mindset to the practice of evaluating information
• #13 And then underpinning all of this is the tools and support services which can aid students in the development of these and to execute evaluation in the most effective and efficient way So lets explore these 3 aspects in a little more detail.
• #14 Dawkins and LeGrand (2024) talk about the importance of developing students confidence so that they can meet information with curiosity and that such an approach will empower students to not just find the papers they need for their assignment and move on, but to contribute to scholarly conversation, posing questions and having a dialogue with the literature
• #15 This is where the “Question Everything” mindset is coming from. Before we can instruct students on how to evaluate information we must ensure that they feel empowered to do so and don’t take any source at face value, whether it be a peer-reviewed journal article, a book recommended by their lecturer, information from their professional body’s website or content generated by an AI tool. Each of these should be approached with the same level of curiosity and criticality.
• #16 So with the mindset established we need to think about what students need to consider or understand in order to critically engage with information and sources.
• #17 The Evaluation TRYangle breaks this down into 3 interconnected areas which can be employed together to make an assessment on the reliability, validity and appropriateness of a source for inclusion in their academic work.
• #18 The first of these is Test & Check which is the aspect most familiar to previous versions. This is about asking questions of the source and using the principles of lateral searching to seek out further information and sources which might affirm or challenge the information within it. This might be as simple as looking up the author’s online profile, or seeing how the work has been discussed within subsequent papers. It might also involve seeking additional sources which either verify the information as correct or to find those which offer opposing viewpoints from which you can draw your own conclusions.
• #19 Bull, MacMillan and Head (2021) in presenting their concept of “Proactive evaluation” talk about how information now has agency and how it no longer rests with the user to find and evaluate it, but instead seeks out the user through networked connections. This is particularly pertinent when we are looking at social media tools and GenAI in the way that information is gathered and presented to the user and so in order to evaluate it, students need to also understand how and why that information is being presented to them in that moment. In order to assess AI generated content students needs to have an understanding that this is actually amalgamating and replicating patterns from across its training data, devoid of the context of a source rather than a presentation of evidenced fact. But even within more traditional academic search tools we have options such as “Relevance” where the tool is ranking results based on their connection to the search terms used.
• #20 This is building upon the work of Tardiff (2022) and the element of Worldview within CCOW. What we want to encourage students to do in this aspect is recognise their place within the information exchange and landscape. It is encouraging students to recognise how their life experiences impact upon how they engage with and react to the information that they find. Its about having an understanding of what your own beliefs on a topic are so that you are able to be less reactionary to those opinions which differ to your own. Its also having an awareness of confirmation bias and ensuring that even when a source gives the student exactly what they want that they remain critical.
• #21 These 3 elements aren’t to be approached in isolation though, they are interconnected. For example, in order to understand the Retrieval Method and why a certain viewpoint is being presented to you by the tool’s algorithm, you need that Your Global Self awareness of what your thoughts on the topic are or your previous engagement with it which is informing this. It’s then employing Test & Check to seek out those differing opinions and form your own conclusions on whether the information is accurate, reliable and appropriate for use.
• #22 So the final part of this approach is about ensuring that students have an awareness of the tools and support available to them in order to develop these areas of understanding
• #23 This could be the support that is available from the Library – perhaps in the form of a human they can come and talk to. I often end my sessions by emphasising that the most important piece of information they’ll take away is my email address and that they contact me with any questions or even just for some reassurance – circling back to the initial idea of the importance of confidence. It might also be links to further content where students can develop their understanding of GenAI processes or a toolkit which helps them recognise their worldview. It could equally be more practical tools they can employ as part of the evaluation process. For example, it might be simply introducing them to a search engine such as DuckDuckGo which doesn’t include the user’s data in how it processes and presents search results. As I mentioned earlier I’m also keen to look at how we can use GenAI positively in this process and so looking at tools which can offer efficiency to the Test & Check process, Mike Caulfield, creator of SIFT, has been doing some really interesting work in this area, exploring add-ins for ChatGPT and prompts which can be used to interrogate a source and validate or question its accuracy through other sources.
• #24 So here are some examples of how we might put all of that into practice in considering how we approach evaluating particular information sources
• #29 So that’s the TRYangle Approach and where I’m at currently with it. I’ve been seeking feedback on it from fellow information professionals, academic colleagues and students and so I would also very much welcome any thoughts you all have today. I then have ethical approval from my university for a research activity where I will teach this approach to a group of students and assess whether it has improved their critical engagement with sources and gain some formal feedback on how it is presented. My hope is that I can then come back to LILAC again next year and present the outcome of that, alongside a full suite of teaching materials that you can take away to use in your own institutions should you wish to. I also hope to get it written up and published in a journal article.