The Simplest Test of LLMs for Research

I am teaching a general research preparation course for Masters students across my (shiny new) University called RESE 5000 Foundations of Research. It’s a four week course as part of a much larger sequence and covers research design, alternatives in research design, paradigms, quant/qual, and a lot of hands on work to develop a small but focussed research design glossary for these students. The useful part of this is that they have got more vocabulary and more knowledge with which to discuss things, especially with their supervisors. It’s all colleges and disciplines so it’s a pretty broad course in coverage.

The Activity

Because they have to go and find appropriate references for methods and approaches in their own discipline, we bootstrap with literature reviews: how do you find the papers and published works that support your research? I conduct a very simple exercise in the first workshop:

  1. Open up a Google Scholar page and search for a term I give them. (It’s often mastery learning.)
  2. Then we discuss as a class what we found as the top 10. It’s almost always exactly the same and, because inside the Uni we’re subscribed to all of the various pub venues, you can click on all of them and open them up.
  3. Then I ask them to open up ChatGPT (or “LLM of choice” if they have a strong preference) and ask it for top ten references for the same search term.
  4. And then we compare.
  5. And then we check to see how many are duplicates.
  6. And then we check to see how many don’t actually exist.

I did this on Thursday last week for the subject of mastery learning (told you), and of the ten references I got back, 2 and 3 were effectively the same, 10 didn’t exist, and two others kind of matched. But, more importantly, everyone got something different. We had gone from “this is what you get when you search” to “here’s something that might be right and it will change every time”. By my estimate, students who start with the LLM based approach need to spend up to twice as much time checking to make sure it’s not wrong. I think that’s inefficient.

The Rationale

I don’t use generative AI at all in my work, unless for specific research or illustrative purposes, although I am starting to work with Apertus, the Swiss-based model that meets EU training standards, is offset for power, and cooled by the waters of Lake Lugano. As a journal associate editor and regular reviewer, however, I am being driven to distraction by the sheer number of fabricated references I’m encountering. I have a very simple rule on this – you cannot build good research on bad research and evidence of reference fabrication is evidence of a flawed process.

I tell the students that how they use Generative AI is up to them, their supervisors, and whoever employs them, but I also tell them that good research is supported by solid citations. If I can’t run a simple demo in class and get right answers more than 90% of the time, this tool is not fit for purpose.

Oh, but you could craft the prompt/use agents

Yeah, I could. I’ve had students do that, where they change the prompt to only provide actual works that can be looked up. The first problem is that you still sometimes get dud references, the second is that this is most likely using some form of agentic checking layer over the top of the stochastic generation. Now, I am not a young CS person, and I cut my teeth coding for 16K, 32K, 640K memory spaces, which means that the aesthetics I value in code include efficiency and minimalism. With much of CS being searching and sorting, I find it more than vaguely irritating that the solution to “the machine is making things up” is not “stop using the machine” but “check what it made up to see if it worked and then throw the dice again until you get it right.”

Searching is not just about finding what is there, it is also about knowing what is not there and clearly indicating that. We used to value precision and recall, and I am seeing far too many false positives to make me confident that this is still being taken seriously. The speed with which a synthetic answer is generated when no answer truly exists worries me, because most people don’t check as much as I do.

The dull conclusion

There are things that AI is good at. There are things that Generative AI is good at. But given how easy it is for me to show students what it’s not good at, I’m going to keep recommending that they avoid it for reference generation for the foreseeable future.


Finally, the paper with my favourite diagrams has been accepted!

At Koli Calling 2024, which was many lifetimes ago in terms of my last 18 months, I was … well… ranting with Leo Leppänen and Juho Leinonen about the potential issues with the use of GenAI discussants in CS activities that use social constructivism, given the rapid adoption of Gen AI in chat/discursive situations. We caught up in Helsinki after Koli, sketched out a paper, actually then dedicated enough time to write it (shocking, I know), and started submitting it. And then it got rejected. And then we rewrote it and submitted it elsewhere. And then it got rejected. Rinse, repeat, until, finally, aha!

The paper that has just been accepted is a different paper from the one that started out and there has been a lot of useful reviewer feedback that has been part of it.

One of the big issues in CS Ed is always whether a paper that discusses education is sufficiently specific to CS Ed that it should be at a CS Ed conference. I am most definitely not an expert or oracle on that as many of my papers have walked a pretty fine line. But, twelve months later, we appear to have put together a sufficiently convincing argument, addressing a range of other issues along the way, and it will be presented this year. I am currently teaching a Foundations of Research course as part of the brand new Masters of Research course for the (also brand new) Adelaide University, and I was able to talk about the journey of this paper as part of the academic writing and publication section. Fortunately, it was a journey that ended with “they all lived happily ever after” (or at least until question time in the presentation session.)

I’m happy for several reasons about the acceptance.

  • It’s great to work with interesting people and turn the useful discussions you have at conferences into publishable work.
  • This has rewarded our persistence and willingness to incorporate feedback.
  • Somewhat self-centredly, this paper has some diagrams in it that I really like and now I can finally share them.
  • The paper was the result of some discussion that was quite sharp in some ways, but the final argument is a more positive and constructive presentation of our concerns.
  • This is a positive event at a time when my health has not been great and my work has been quite busy. A needed lift!

On that matter of “constructive presentation”, as someone who used to be more involved in school management, I’ve been in far too many meetings where people are unhappy about something and they repeat the problem/complaint as if that is a pathway to solution. (I have done it myself far too often, as well.) Yes, the problem is important, but our role is often to work towards solutions, towards sharing information that can lead to better outcomes. I hope that when the paper is presented, it does come across as being more focused on “how can we do this well?” rather than “you can’t do this at all”.

I won’t be presenting it as, despite my intention to be in Europe over the major conferences, I have calendar clashes that prevent my attendance. The good news is that my co-authors will probably do a better job than I would, and I still get to be happy about my diagrams from wherever I am! Thank you for all of the hard work, enthusiasm, dedication, and patience, Leo and Juho!


Work and Life

A decorative image of the coast, vegetation, rocks, and ocean around Boat Harbour, New South Wales.

I’ve just returned from a brief holiday on the Australian East Coast and I had a chance to think about what the new leadership role at the Uni will mean. One thing I felt I had to do, although it was honestly a tough decision to make, was tidy up the work/personal aspects of my social media. This means unfriending (what a loaded term) some people who I’ve known through work for a while. It wasn’t easy to do and I’m still thinking about it. Why did I do it?

I will have new staff who don’t really know me, in a school where half of the team also don’t really know each other that well, and we are bringing together two successful, fully-formed, and slightly different academic cultures. Bringing these groups together, meeting their needs, being focused, being present, listening… all of these things are essential and they need the best of my professional aspects. I have recently found out that I have not been listening as well as I should and that’s on me to fix, to be the person I always wanted to be and claim that I am but, obviously, sometimes fall short at. (The irony is that I consider it likely that people have given me clear indications that I haven’t been seeming to listen enough to care about their input – but I haven’t noticed. Ick. I’m striving to pay attention now.)

I need to be “work Nick” primarily, because those are the skills, the focus, and the type of presence that they need. I have many interests but bringing them into the work space is not always the best use of anyone’s time and it’s certainly not what a junior teaching team academic needs when I’m managing them. To make the dividing line really clear, both for my ease of processing and for consistency, I’ve removed a number of people from my Facebook friends who are primarily work people, so that I don’t have some weird two speed situation of “I friended this person but this other one.” I’ve also had to unfriend someone from my own town who I’ve known for years but now we have a potential conflict because of our roles. I’ve had to do this before when I realised that you couldn’t friend some students and not others, even though there may have been an existing RW connection because my home town has about 1.5 degrees of separation.

(Years ago, there was even a student group “Nick Falkner won’t friend me on Facebook” and the student laser printer was part of it. No, I never added the laser printer.)

I’m unsure if removing work people was the right decision or not, as it means I can’t see what people who I met through work, and I like, are up to – but I’ve taken this decision as a good way to make it easier for me to set things up well for my new role and do the right thing by the people I’m managing. A part of me hopes that, after a year or two, when the culture is established and I’ve got to know people better, I can go back and reconnect to the other people, if they’ll have me.

I’d be really interested to know what people think and how you handle the work/personal social separation! Feel free to tell me how wrong or right you think I got it.


First day at the new (old) job

I sit at the desk in my new office on Level 5 of the building I’ve been working in for over 10 years, as part of a different university and with a new job title. The foundation universities of the University of Adelaide and the University of South Australia are now officially merged as Adelaide University, which is now roughly the 5th largest University in Australia. I am no longer the Deputy Head of the School of Computer and Mathematical Sciences, as neither the position nor that school still exist! I am an Academic Lead in the School of Computer Science and Information Technology, with responsibilities around educational research and our education specialist staff, and now have roughly 15 staff to manage as part of our new model.

Last year was pretty successful in many ways, the ITiCSE WG delivered an amazing report with some fantastic resources, I was part of a successful grant, the CompEd WG made fantastic progress (still going into the new year as we tidy that up), and I felt like, for the first time in ages, I really had the opportunity to look into research into detail. But it was very tiring, being the last year of a long push that started during the pandemic and finishing in a year where I got so sick that my hair started falling out and my overall physical health took a 10% hit. Much better now, although my hair is still my hair! And now the universities are merged and I’m part of the brand new Adelaide University. It’s a great time to look forward and set some themes for this year. But let’s do a 50,000 foot flyover of my job.

Research: From a research perspective, the CARD WG outcome has reinforced how important it is to have learning resources and tools that lead to the outcomes that you want – this is especially true in the face of the rise of Generative AI, where it is terrifyingly easy to sidestep any cognitive or memory work totally in the production of artefacts. For years, we got away with requiring student to produce something as evidence of a process of production, which entailed cognition, learning, and application of effort over time. Artefacts like the CARD Research Cards can’t be completed with generative AI, people have to interact with them. There are opportunities to engage with something, consider it, respond to it, and develop your skills accordingly. I sketched out some ideas last year about a working group based on tools and approaches for learning that specifically could not be carried out with Generative AI, but haven’t done anything with them. I most likely can’t do any WGs this year for timing and commitment reasons but I might pitch something next year about all the ways that game-based approaches to learning and non-AI-based scalable aspects of personalised learning can be useful in the era of Generative AI. The random and reflexive aspects of the cards make the process inherently focused on the individual – but I’ll return to this thought later. More stuff needs to happen with CARD but that’s a next week thought.

Teaching: I’m teaching a couple of courses this year, mostly around the postgraduate level, working on research skill development and computational research methods. This will be the first curricular application of the CARD system, realised as both physical cards and an online version developed by our fantastic media team in the learning design space. It will be great to be back in the classroom more!

Leadership and Service: I have 15 direct reports and will be getting to know them over the next few weeks. I suspect I’ll be in three places in a given week, my old stomping ground next to the Art Gallery on North Terrace, City West campus, and Mawson Lakes Campus. Learning what they need, and how I can use my experience and knowledge to help them, is one of my highest priorities prior to the start of Semester 1. I have had an unconventional pathway to academia but I still have many lenses of privilege to acknowledge and work through when I work with anybody else. There will be a lot of listening and thinking on my part. I have a number of duties for various SIGCSE roles that I’m looking forward to getting back into now that I’m finally healthy again.

Travel: I’ve been travelling a lot and the plane trips have been frequent, long, and not the best thing for me in any sense. I’m planning to spend about three months in Finland this year as part of a visiting fellowship at University of Jyväskylä but this will involve less flying and more time concentrated in one place. I hope that this will allow me to do better work and be more present…

Speaking of presence, I think that’s my most important theme for this year: present in my personal life, present for my students, present for my research collaborators, and definitely present for my colleagues, especially those who I’m managing. Something has been showing up in my shape at various points of the year but I’m not sure how much of it has been me. Let’s see what I can do this year.

Happy new year, everyone!


2025 ACM Global Computing Education Conference in Botswana, 21-22 October – Another Working Group!

Well, well, a third post in the year? Who made me research active? (I have not yet really been made research rigorous, despite some valiant attempts) I’m very excited to announce that we’ve got a working group advertised for the 2025 ACM Global Computing Education Conference in Botswana and we’d love to get more people involved. The Main conference runs Oct 23-25, Working Groups Oct 21-22 and Doctoral Consortium Oct 22.

You can join A/Prof Claudia Szabo, me, Dr. Munienge Mbodila, and Prof Judy Sheard as we work to assess impact, analyse the effects on higher ed institutions, and identify the challenges, with a focus on the Global South.

It’s Working Group 5, entitled “Ethical and Societal Impacts of Generative AI in Higher Computing Education: An ACM Task Force Working Group to Develop a Landscape Analysis – Perspectives from the Global Souths and Guidelines for CS1/CS2/CS3″

Brief motivation

This ACM working group, fitting within the broader remit of an ACM Task Force, aims to investigate the ethical and societal impacts of Generative AI tools within the higher computing education landscape within the Global South and to provide guidelines for the ethical incorporation of AI into teaching and assessment in CS1, CS2 and CS3, considering the desired learning outcomes at each level, student learning behaviour, validity of assessment and academic integrity, and local ethical frameworks. This working group will conduct a landscape analysis on Global South ethical questions related to the use of Generative AI tools in higher education contexts, identifying promising principles, challenges, and ways to navigate the implementation of Generative AI in ethical and principled ways. This working group builds on the ITiCSE working group with a similar name, and is developed under the guidance of two of that working group’s leaders, namely Alison Clear and Tony Clear. It focuses the lens on the Global South and specifically on CS1-3 perspectives within this space.

Get Involved!

If you want to read more or apply, please follow the link to the WG page and fill out your application before the end of May 9, anywhere around the world! There are excellent travel grants available to support this so check those out, too!


ITiCSE 2025: Working Group 1 – exciting news!

Two posts in the same year? Something must be up… and it is! After the successful presentation of Dr Rebecca Vivian and my work at Koli as both DC tool and award winning poster/demo, I looked into taking this to a working group and Dr Miranda Parker agreed to co-lead it with me, as Rebecca is currently on leave. Miranda and I have been digging into all of the aspects of this in the middle of both our day jobs and it’s been a lot of fun to work on! You think you’ve got difficult collaborators? Miranda has to listen to me pontificate about ontologies, paradigms, and philosophies!

It’s really important to recognise Rebecca’s ongoing connection with this project, as it’s still very much Rebecca’s work that got us here and she will continue to be a significant part of this, we’re just making sure we have the co-leadership of people who aren’t on leave to make it work. It’s really exciting that our Workgroup has gone to the advertisement stage!

You can see all of the WG proposals here, and sign up (maybe to ours if you like what you read here) here. We’re happy to answer questions and it’s going to be an amazing combination of serious play, serious research, and great fun.

Here’s the ad as a cut and paste!

WG1 – Paradigms, Methods, and Outcomes, Oh My!: Refining and Evolving a Research Knowledge Development Activity for Computer Science Education

Leaders:

Motivation:

Computer Science Education Research (CSER) combines the frequently quantitative approaches of computer science, engineering, and mathematics with the often more qualitative techniques seen in psychology, sociology, behavioural science, and education. It can be challenging to select appropriate research methods in effective and efficient ways. 

Inspired by the use of card-based techniques in the classroom, the Research Alternatives Exercise (RAE) is a pack of 105 cards introducing a wide range of possible research approaches. RAE provides alternatives to a participant’s current research plans using new random lenses, leading to the sketch of a new research design. The participant refers to their own design through the lens of the randomly drawn card, working to see how well this fits, informs, or improves what they have done. 

The initial version of the card deck and examples of play won best paper/demo at Koli Calling 2024 and an example “run” is shown below:

Goals:

  • review and modify the existing deck through collaboration in the WG
  • develop a version of the deck that can be shared and used widely across the CSER community, 
  • develop a concise support glossary for the cards

Methodology:

The current deck will be shared with participants, to support targeted literature review, research, and consultation to:

  1. refine the terminology used for categories, which are currently paradigms, methodologies, outcomes, and methods,
  2. refine the components within categories, 
  3. review the existing rules for suitability, 
  4. develop the first draft of the support glossary, and
  5. develop different decks and play approaches for specific purposes.

Following kickoff at the end of March, we will work on Items 1 and 3, aiming for completion by the start of May. When categories are finalized, we will undertake Item 2, where each group member will work in small groups to review each category. Findings will be presented to the whole group by the beginning of June, for further discussion and collaboration. Each sub-group will be responsible for the glossary elements of their contribution, to be completed and reviewed for the start of the in-person WG time. Each working group member will be asked to share the deck with colleagues to provide feedback. 

Member Selection:

We seek at least 8-10 individuals to share the required work manageably. 

We are looking for participants with at least one of:

  • Experience with a wide variety of research methodologies,
  • Experience in supervising graduate students, 
  • Interest and knowledge in using game-based and facilitated techniques, or
  • Experience with research skills development.

We actively invite applications from disciplines beyond computing for diversity in research skills development experience. We seek a diversity of experience, background, and culture, to ensure that the feedback encompasses the full range of CSER community experience. We also welcome student applications.

Successful applicants will:

  • Attend fortnightly 60-90 minute online progress meetings, held from mid-late March to the end of June,
  • Register for ITiCSE 2025,
  • Physically attend the full duration of the working group, and
  • Make significant contributions during the pre- and post-ITiCSE Working Group activities (3-4 hours a week).

On pedagogical rations, we still seek to thrive.

It has been a tough year. Australia was on fire at the start, then sabre rattling started, then COVID came, and now the US is in turmoil as black voices rise up to demand justice and fair and equitable treatment as citizens. (Black Lives Matter. If that bothers you, go read someone else, or, better yet, educate yourself as to why you should agree.)

The COVID crisis has had a large impact on the educational sector, affecting enrolments at many Australian institutions as we are (to varying degrees) dependent upon international students for income. But now, many of our international students are not coming, which means that every University in Australia is taking a hit. At the same time, the COVID issues that prevent students from entering the country have forced us into an unexpected and unprecedented level of remote and on-line teaching for every student. We have been in remote mode for months now.

Many far more respected voices than me have correctly identified that we cannot learn a great deal about remote learning from this change, because it was not planned, it has no “before” state that we captured for a control, and it is a scrabbling matter of survival. We have gone from the relatively ample sustenance of Universities in the 1960s and ’70s, to a more constrained budget as funding changed, and now we are on survival rations.

Our pedagogies are also rationed, limited by the physical space we can occupy, the technologies that we have, the staff who are available, and an overwhelming sense of dread that fills the spaces in June as many of us think “What next?”

Rationing reduces both the quantity and range of what we consume. From a food perspective, history tells us that limited sustenance is dangerous in two ways: firstly because slow starvation is still starvation, and secondly, that there is a minimum requirement for a balanced diet or humans can get very sick or even die with full bellies. The consumption of maize, a staple of Mesoamerica, can easily lead to pellagra and other deficiency diseases unless it is nixtamalized with lye or lime. Where maize went without this knowledge, outbreaks of deficiency disease followed. It’s not just cereals and vegetables that have this problem: rabbit meat is so low in fat that a diet exclusively on this meat can lead to protein poisoning and, if rabbit is your only meat source, common advice is not to make it a substantial part of your diet.

Back to our pedagogies, while we are forced to ration our approaches and our resources, we have to think about whether we are providing enough for a balanced and sufficient education, or are we slowly starving our students or, worse still, introducing educational deficiencies that will hamper their development in the future?

There will be weeks and months of analysis after this challenging year, and many assumptions will be challenged. We will see the impact of these remote terms and semesters on education, on knowledge, on community, and on identity. But that analysis is after things improve. Right now, our focus is on monitoring the health of our communities, looking for slow decline and deficiency as best we can, and improving things where we can.

How long do we stay on these rations? 2020 is a brewing storm of new things, each one pushing a previous event into the background. Every lightning bolt is brighter and closer than before, every thunder clap louder.

If this were a storm at sea, we would be desperately trying to ensure our ship was sound, that it could stay afloat, and we would look for signs of the storm breaking.

When a ship is in distress, often its weight is reduced to improve its chances of staying afloat. The things thrown off are known as “jetsam”, distinguished from those things that float away (either from waves or because the ship has foundered), which are “flotsam”. What have we thrown from our ships, or at least considered?

There are no more face-to-face lectures in many cases. These are replaced with recordings or on-line presentation and discussion. There are fewer tutorials, with fewer people, as the tyranny of the physical prevents us from filling rooms while we are under disease management social distancing restrictions. We do not exchange paper. We do not gather in laboratories. We do not sit in one place to undertake examinations under strict invigilation conditions.

The traditional lecture, with hundreds of students sitting in a room to receive the wisdom from the front, is jetsam. At my institution, there will be no face-to-face lectures until 2021. This is mostly because we will not be legally allowed to put students into many of the lecture theatres at densities and numbers that make it feasible. Existing laws would require us to have 10 times the lecture space – which is an impossible requirement, even if we had the lecturing staff available to multiply their effort by 10. But the presentation of information, interactively with the lecturer, is still going strong in the remote space and we have noticed that more students participate in Q&A than the few we used to see dominating the physical space. We have lost the “vitamin” of community that occurs through regular mingling but that may come from other sources.

Paper assignments, long dwindling, are jetsam but that is perhaps hastening an inevitable demise. In a time of growing part-time student numbers, students who work part-time, increasing transit times, requiring the physical transfer of cellulose fibres imprinted with marking reagent seems a little excessive unless absolutely necessary. It’s true that on-line and electronic systems are less flexible than paper and, especially for formulae, there is a steep learning curve for formatting tools to represent complex symbols. E-paper, in its various forms, is promising but still not there. The removal of paper is probably making things harder and stifling some students’ creativity.

The tutorials and the laboratories are coming back, under new regulations and new requirements. Their value, the authentic and hands-on nature of a good exercise in these spaces, saved them from the ocean. We know what is good about them but now have to make sure that we have placed that good at the forefront.

The invigilated paper examination is another case altogether. I have just finished working on a fully remote examination, open book, and presented across a network. Instead of having two pens and pencils, my students need a fully-charged battery and a good internet connection. But the move to open book (a first for this course) has meant a Bloomian shift up into application and evaluation as a minimum – a very positive direction that we had been making but was much more easily justified in this change. We have kept most of the exam but we have thrown out some of its old baggage.

I will be honest. I think that on-line examinations, already a busy area of research, are going to be an area of a great deal of future research, much of it looking back into this year as we desperately try to work out what worked and how it worked. For me, this rationing has been fascinating, as it forced me to think in detail about exactly what I wanted students to do, as their potential identities as graduates, as students, and as discipline specialists.

There is another nautical term, which you might not know, lagan, that refers to heavy goods thrown from a ship to reduce weight but marked with a buoy to be recovered later. When danger has passed, you circle back and get them again. While flotsam and jetsam are often legally passed to their discoverer, unless the former owner makes a claim, lagan is always yours and you will be back for it.

I do wonder how many of the things that we didn’t do, that went overboard, are considered to be so valuable that we circle back for them? As we come out of this, even while we’re circling back, it’s probably worth some moments in reflection to determine whether we really want that heavy thing back on board or we learned something new while we weathered the storm.

Stay safe, stay well.


Voices – LATICE 2017

[Edit: The conference is now being held in Hong Kong. I don’t know the reason behind the change but the original issue has been addressed. I have been accepted to Learning @ Scale so will not be able to attend anyway, as it turns out, as the two conferences overlap by two days and even I can’t be in the US and Hong Kong at the same time.]

There is a large amount of discussion in the CS Ed community right now over the LATICE 2017 conference, which is going to be held in a place where many members of the community will be effectively reduced to second-class citizenship and placed under laws that would allow them to be punished for the way that they live their lives. This affected group includes women and people who identify with QUILTBAG (“Queer/Questioning, Undecided, Intersex, Lesbian, Trans (Transgender/Transsexual), Bisexual, Asexual, Gay”). Conferences should be welcoming. This is not a welcoming place for a large percentage of the CS Ed community.

There are many things I could say here but what I would prefer you to do is to look at who is commenting on this and then understand those responses in the context of the author. For once, it matters who said what, because not everyone will be as affected by the decision to host this conference where it is.

From what I’ve seen, a lot of men think this is a great opportunity to do some outreach. A lot has been written, predominantly by men, about how every place has its problems and so on and so forth.

But let’s look at other voices. The female and QUILTBAG voices do not appear to share this support. Asking for their rights to be temporarily reduced or suspended for this ‘amazing opportunity’ is too much to ask. In response, I’ve seen classic diminishment of genuine issues that are far too familiar. Concerns over the reductions of rights are referred to as ‘comfort zone’ issues. This is pretty familiar to anyone who is actually tracking the maltreatment and reduction of non-male voices over time. You may as well say “Stop being so hysterical” and at least be honest and own your sexism.

Please go and read through all of the comments and see who is saying what. I know what my view of this looks like, as it is quite clear that the men who are not affected by this are very comfortable with such a bold quest and the people who would actually be affected are far less comfortable.

This is not a simple matter of how many people said X or Y, it’s about how much discomfort one group has to suffer that we take their concerns seriously. Once again, it appears that we are asking a group of “not-men”, in a reductive sense, to endure more and I cannot be part of that. I cannot condone it.

I will not be going. I will have to work out if I can cite this conference, given that I can see that it will lead to discrimination and a reduction of participation over gender and sexuality lines, unintentionally or not. I have genuine ethical concerns about using this research that I would usually reserve for historical research. But that is for me to worry about. I have to think about my ongoing commitment to this community.

But you shouldn’t care what I think. Go and read what the people who will be affected by this think. For once, please try to ignore what a bunch of vocal guys want to tell you about how non-male groups should be feeling.


Our duty to the future, a letter from the 18th Century.

Official_Presidential_portrait_of_John_Adams_(by_John_Trumbull,_circa_1792)

Official Presidential Portrait of John Adams (John Trumbull, 1792).

Extract of a letter from John Adams to Abigail Adams, posted 12 May 1780, from Paris.

I could fill Volumes with Descriptions of Temples and Palaces, Paintings, Sculptures, Tapestry, Porcelaine, &c. &c. &c. — if I could have time. But I could not do this without neglecting my duty. The Science of Government it is my Duty to study, more than all other Sciences: the Art of Legislation and Administration and Negotiation, ought to take Place, indeed to exclude in a manner all other Arts. I must study Politicks and War that my sons may have liberty to study Mathematicks and Philosophy. My sons ought to study Mathematicks and Philosophy, Geography, natural History, Naval Architecture, navigation, Commerce and Agriculture, in order to give their Children a right to study Painting, Poetry, Musick, Architecture, Statuary, Tapestry and Porcelaine.


Reflection: Why I’m stopping daily updates

I’ve written a lot in the past month and a half. Now, because I’m committed to evaluation, I have to look back at all of it and think about some difficult matters:

  1. Is anyone reading this?
  2. Are the people reading this the ones who can make change?
  3. Is the best way to do this?
  4. Should I be doing something else?

There are roughly 1,000 people who see my posts, between direct subscribers who read in e-mail, Facebook and the elusive following community on Twitter.

Twitter shouldn’t count, as I know from direct experience that the click-through rate from Twitter is tiny. (My posts have been shared by people with 5-10,000 followers and it has turned into maybe 10-20 more people reading.) Now I’m down to maybe 4-500 readers.

Facebook shares a longer fragment of my ideas but the click through is still small. Perhaps this brings me down to the roughly 200 followers I have, who have (over time) contributed about 1,000 ‘Likes’. However, almost all of these positive reinforcements stem from a different phase of the blog, a time when I was blogging conferences and being useful, rather than pontificating on the nature of beauty. My readership used to be 100 people a day, or more. I can’t crack 80 today and the way that I’m blogging is unlikely to reach that larger audience, yet it’s what I want to do.

The answer to 1 is that a few other people a day are reading what I write. I’d put it as high as twenty on a good day but most days it’s under ten.

2’s a tricky question. We can all make change; that’s one of my firmest beliefs. However, there is making change and then there are change makers. I know several people in this area quite well and they read me occasionally but it’s not something that they dedicate time to do. I have people that I always read but I can’t make the changes they need. It’s frustrating. No doubt, my ideas appeal to some people but change takes will and capacity to change, not just a sympathetic ear. I don’t want people to read this and feel trapped because they can’t make change. The answer to 2 is, probably, ‘no’.

3 follows from 1 and 2. If my readership is small and my ideas have little influence then this is not the best way to do things. We face enormous challenges. We need effective mechanisms for sharing information. If I am to make change, I have to invest my time wisely. I am not a large-scale player or a change maker. I need help to do it and if that help isn’t coming from this avenue, I have to choose another.

4 is easier. I can focus on my scholarship, practice, and research, rededicating the time I’ve been spending on this blog. People read papers where they don’t read blogs. Papers drive recognition. Recognition gets you the places to speak where your voice can be heard. There is no point having written all those words in a blog if it’s rarely read. This has been a highly rewarding experience in many ways but you have to wonder why you’re doing it if very few people read it or remember what you’ve written.

I wanted people to think and to talk about the ideas shared here. For those of you who have let me know that this worked, my thanks!

I’m tempted to keep going with the daily blog but the aesthetic argument traps me here. Spending time on something that isn’t working and insisting that it’s valuable is self-deception. Investing energy into an avenue that isn’t achieving your goals isn’t good. I cannot deprive my students of the hour or so a day that I’ve been spending doing this unless I achieve more for them than I would by doing some other aspect of my job.

Students and teachers: the true focus of any aesthetic discussion of education; the most important aspects of any discussion of what we should be doing because they are people and not just machine parts. As for us, so for them.

There are more discussions to be had but they’ll show up in more formal places, most likely. I’m always happy to talk to people about ideas at conferences. I’ve already started a face-to-face discussion about taking some of these ideas further in a more traditional research sense and I’m very excited about that.

But perhaps it’s time to let this blog go, listen to the numbers, reflect on the dissemination of knowledge, and accept that I would not be following my own advice if I were to continue. I love the beauty argument. I think it’s great. I stand by everything I’ve written this year. I just don’t think that this is the way to move people towards that agenda.

Thus, the daily updates stop with this post. I’ll still post things that interest me but there’ll be fewer of them.

I’ll leave you with the message I wanted to get across this year:

  1. Educational philosophy is full of the aesthetics of education. Dewey and Bloom just scratch the surface of this. The late 19th and early 20th century were an incredible time of upheaval and we still haven’t addressed many of the questions raised then. To the libraries!
  2. Fair, equitable, well-designed and evidence-based education is at the core of any beautiful system.
  3. Every day, we should ask ourselves if what we are doing is beautiful, good or true, taking into account all of the difficult questions of how we balance necessities against desirabilities, being honest about which is which. If we aren’t managing this, we need to either seek to change or accept that what we are doing isn’t right.
  4. We should leave enough time for ourselves in all of this, as there should be no sacrificial element to beautiful education.
  5. Change is coming. Change is here. Pretending that it won’t happen isn’t beautiful.

I hope that you all have a fantastic learning and teaching year, with many amazing and beautiful moments and outcomes!

This year, I hope to be at several conferences and I look forward to talking to anyone about the ideas in this phase (or any other phase) of the blog.

Have a great year!

SMPTE_Color_Bars.svg