The Simplest Test of LLMs for Research

I am teaching a general research preparation course for Masters students across my (shiny new) University called RESE 5000 Foundations of Research. It’s a four week course as part of a much larger sequence and covers research design, alternatives in research design, paradigms, quant/qual, and a lot of hands on work to develop a small but focussed research design glossary for these students. The useful part of this is that they have got more vocabulary and more knowledge with which to discuss things, especially with their supervisors. It’s all colleges and disciplines so it’s a pretty broad course in coverage.

The Activity

Because they have to go and find appropriate references for methods and approaches in their own discipline, we bootstrap with literature reviews: how do you find the papers and published works that support your research? I conduct a very simple exercise in the first workshop:

1. Open up a Google Scholar page and search for a term I give them. (It’s often mastery learning.)
2. Then we discuss as a class what we found as the top 10. It’s almost always exactly the same and, because inside the Uni we’re subscribed to all of the various pub venues, you can click on all of them and open them up.
3. Then I ask them to open up ChatGPT (or “LLM of choice” if they have a strong preference) and ask it for top ten references for the same search term.
4. And then we compare.
5. And then we check to see how many are duplicates.
6. And then we check to see how many don’t actually exist.

I did this on Thursday last week for the subject of mastery learning (told you), and of the ten references I got back, 2 and 3 were effectively the same, 10 didn’t exist, and two others kind of matched. But, more importantly, everyone got something different. We had gone from “this is what you get when you search” to “here’s something that might be right and it will change every time”. By my estimate, students who start with the LLM based approach need to spend up to twice as much time checking to make sure it’s not wrong. I think that’s inefficient.

The Rationale

I don’t use generative AI at all in my work, unless for specific research or illustrative purposes, although I am starting to work with Apertus, the Swiss-based model that meets EU training standards, is offset for power, and cooled by the waters of Lake Lugano. As a journal associate editor and regular reviewer, however, I am being driven to distraction by the sheer number of fabricated references I’m encountering. I have a very simple rule on this – you cannot build good research on bad research and evidence of reference fabrication is evidence of a flawed process.

I tell the students that how they use Generative AI is up to them, their supervisors, and whoever employs them, but I also tell them that good research is supported by solid citations. If I can’t run a simple demo in class and get right answers more than 90% of the time, this tool is not fit for purpose.

Oh, but you could craft the prompt/use agents

Yeah, I could. I’ve had students do that, where they change the prompt to only provide actual works that can be looked up. The first problem is that you still sometimes get dud references, the second is that this is most likely using some form of agentic checking layer over the top of the stochastic generation. Now, I am not a young CS person, and I cut my teeth coding for 16K, 32K, 640K memory spaces, which means that the aesthetics I value in code include efficiency and minimalism. With much of CS being searching and sorting, I find it more than vaguely irritating that the solution to “the machine is making things up” is not “stop using the machine” but “check what it made up to see if it worked and then throw the dice again until you get it right.”

Searching is not just about finding what is there, it is also about knowing what is not there and clearly indicating that. We used to value precision and recall, and I am seeing far too many false positives to make me confident that this is still being taken seriously. The speed with which a synthetic answer is generated when no answer truly exists worries me, because most people don’t check as much as I do.

The dull conclusion

There are things that AI is good at. There are things that Generative AI is good at. But given how easy it is for me to show students what it’s not good at, I’m going to keep recommending that they avoid it for reference generation for the foreseeable future.