ICER 2012 Day 1: Discussion Papers Session 1
Posted: September 11, 2012 Filed under: Education | Tags: blogging, community, education, educational research, higher education, icer, icer2012, measurement, principles of design, student perspective, teaching approaches, tools, universal principles of design Leave a commentICER contains a variety of sessions: research papers, discussion papers, lightning talks and elevator pitches. The discussion papers allow people to present ideas and early work in order to get the feedback of the community. This is a very vocal community so opening yourself up to discussion is going to be a bit like drinking from the firehouse: sometimes you quench your thirst for knowledge and sometimes you’re being water-cannonned.
Web-scale Data Gathering with BlueJ
Ian Utting, Neil Brown, Michael Kölling, Davin McCall and Philip Stevens
BlueJ is a very long-lived and widely used Java programming environment with a development environment designed to assist with the learning and teaching of object-oriented programming, as well as Java. The BlueJ project is now adding automated instrumentation to every single BlueJ installation and students can opt-in to a data reporting mechanism that will allow the collection and formation of a giant data repository: Project Blackbox. (As a note, that’s a bit of a super villain name, guys.)
Evaluating an Early Software Engineering Course with Projects and Tools from Open Source Software
Robert McCartney, Swapna Gokhale and Therese Smith
We tend to give Software Engineering students a project that requires them to undertake design and then, as a group, produce a large software artefact from scratch. In this talk, Robert discussed using existing projects that use a range of skills that are directly relevant to one of the most common activities our students will carray out in industry: maintenance and evolution.
Under a model of developing new features in an open-source system, the instructors provide a pre-selected set of projects and then the 2 person team:
- picks a project
- learns to comprehend code
- proposes enhancements
- describes and documents
- implements and presents
A Case Study of Environmental Factors Influencing Teaching Assistant Job Satisfaction
Elizabeth Patitsas
Elizabeth presented some interesting work on the impact of lecture theatres on what our TAs do. If the layout is hard to work with then, unsurprisingly, the TAs are less inclined to walk around and more inclined to disengage, sitting down the front checking e-mail. When we say ‘less inclined’, we mean that in closed lab layouts TAs spend 40% of the their time interacting with students, versus 76% in an open layout. However, these effects are also seen in windowless spaces: make a space unpleasant and you reduce the time that people spend answering questions and engaging.
The value of a pair of TAs was stressed: a pair gives you a backup but doesn’t lead to decision problems when coming to consensus. However, the importance of training was also stressed, as already clearly identified in the literature.
Education and Research: Evidence of a Dual Life
Joe Mirõ Julia, David López and Ricardo Alberich
ICER 2012 General Note
Posted: September 11, 2012 Filed under: Education | Tags: blogging, education, educational research, higher education, icer, icer2012 Leave a commentOnce again, we’re so full of interesting content that I don’t really have the time to put together some longer posts although I’m going to try and get something out over the tea break and lunch. In short, if you get a chance, COME TO ICER.
I will however note, while I can transcribe a lot of speakers almost as fast as they can deliver interesting talks, my top speed is asymptotically bound at an upper limit that I am officially designating One Guzdial.
Conference Blogging! (Redux)
Posted: September 8, 2012 Filed under: Education | Tags: blogging, education, educational problem, educational research, feedback, Generation Why, higher education, icer, icer 2012, in the student's head, learning, measurement, student perspective, teaching, teaching approaches, time banking, workload 1 CommentI’m about to head off to another conference and I’ve taken a new approach to my blogging. Rather than my traditional “Pre-load the queue with posts” activity, which tends to feel a little stilted even when I blog other things around it, I’ll be blogging in direct response to the conference and not using my standard posting time.
I’m off to ICER, which is only my second educational research conference, and I’m very excited. It’s a small but highly regarded conference and I’m getting ready for a lot of very smart people to turn their considerably weighty gaze upon the work that I’m presenting. My paper concerns the early detection of at-risk students, based on our analysis of over 200,000 student submissions. In a nutshell, our investigations indicate that paying attention to a student’s initial behaviour gives you some idea of future performance, as you’d expect, but it is the negative (late) behaviour that is the most telling. While there are no astounding revelations in this work, if you’ve read across the area, putting it all together with a large data corpus allows us to approach some myths and gently deflate them.
Our metric is timeliness, or how reliably a student submitted their work on time. Given that late penalties apply (without exception, usually) across the assignments in our school, late submission amounts to an expensive and self-defeating behaviour. We tracked over 1,900 students across all years of the undergraduate program and looked at all of their electronic submissions (all programming code is submitted this way, as are most other assignments.) A lot of the results were not that unexpected – students display hyperbolic temporal discounting, for example – but some things were slightly less expected.
For example, while 39% of my students hand in everything on time, 30% of people who hand in their first assignment late then go on to have a blemish-free future record. However, students who hand up that first assignment late are approximately twice as likely to have problems – which moves this group into a weakly classified at-risk category. Now, I note that this is before any marking has taken place, which means that, if you’re tracking submissions, one very quick and easy way to detect people who might be having problems is to look at the first assignment submission time. This inspection takes about a second and can easily be automated, so it’s a very low burden scheme for picking up people with problems. A personalised response, with constructive feedback or a gentle question, in the zone where the student should have submitted (but didn’t), can be very effective here. You’ll note that I’m working with late submitters not non-submitters. Late submitters are trying to stay engaged but aren’t judging their time or allocating resources well. Non-submitters have decided that effort is no longer worth allocating to this. (One of the things I’m investigating is whether a reminder in the ‘late submission’ area can turn non-submitters into submitters, but this is a long way from any outcomes.)
I should note that the type of assignment work is important here. Computer programs, at least in the assignments that we set, are not just copied in from text. They are not remembering it or demonstrating understanding, they are using the information in new ways to construct solutions to problems. In Bloom’s revised taxonomic terms, this is the “Applying” phase and it requires that the student be sufficiently familiar with the work to be able to understand how to apply it.
Bloom’s Revised Taxonomy
I’m not measuring my students’ timeliness in terms of their ability to show up to a lecture and sleep or to hand up an essay of three paragraphs that barely meets my requirements because it’s been Frankenwritten from a variety of sources. The programming task requires them to look at a problem, design a solution, implement it and then demonstrate that it works. Their code won’t even compile (turn into a form that a machine can execute) unless they understand enough about the programming language and the problem, so this is a very useful indication of how well the student is keeping up with the demands of the course. By focusing on an “Applying” task, we require the student to undertake a task that is going to take time and the way in which they assess this resource and decide on its management tells us a lot about their metacognitive skills, how they are situated in the course and, ultimately, how at-risk they actually are.
Looking at assignment submission patterns is a crude measure, unashamedly, but it’s a cheap measure, as well, with a reasonable degree of accuracy. I can determine, with 100% accuracy, if a student is at-risk by waiting until the end of the course to see if they fail. I have accuracy but no utility, or agency, in this model. I can assume everyone is at risk at the start and then have the inevitable problem of people not identifying themselves as being in this area until it’s too late. By identifying a behaviour that can lead to problems, I can use this as part of my feedback to illustrate a concrete issue that the student needs to address. I now have the statistical evidence to back up why I should invest effort into this approach.
Yes, you get a lot of excuses as to why something happened, but I have derived a great deal of value from asking students questions like “Why did you submit this late?” and then, when they give me their excuse, asking them “How are you going to avoid it next time?” I am no longer surprised at the slightly puzzled look on the student’s face as they realise that this is a valid and necessary question – I’m not interested in punishing them, I want them to not make the same mistake again. How can we do that?
I’ll leave the rest of this discussion for after my talk on Monday.
And more on the Harvard Scandal: Scandal? Apparently it’s not?
Posted: September 7, 2012 Filed under: Education | Tags: advocacy, blogging, community, curriculum, education, educational research, ethics, higher education, in the student's head, plagiarism, reflection, student perspective, teaching, teaching approaches Leave a commentI’ve just read a Salon article regarding the Harvard cheating issue. Apparently, according to Farhad Manjoo, these students should be “celebrated for collaborating“.
Note that word? It’s the one that I picked on in the Crimson article and the reason that I did so is that it’s a very mild word, and a very positive one at that. However, this article, while acknowledging that the students were prevented from any such sharing, Manjoo then asks, to me somewhat disingenuously, “What’s the point of prohibiting these students from working together?”
Urm, well, for most of the course, they don’t. At the end of the course, when they want to see how much each individual knows, they attempt to test them individually. That’s not an unusual pattern.
Manjoo’s interpretation of the other articles goes well beyond anything else that I’ve seen, including putting all of the plagiarism claims together as group work and tutor consultation. I can’t speak to this as I don’t have his sources but, given that this was explicitly forbidden anyway, he’s making an empty argument. It doesn’t matter how you slice it, if students worked together, they did something that they weren’t supposed to do. However Manjoo argues that their actions are justified, I’m not sure that this argument is.
The author obviously disagrees with the nature of the open book test and, to my reading, has no real idea of what he’s talking about. Sentences like “But if you want to determine how well students think, why force them to think alone?” are almost completely self-defeating. It also ignores the need to build knowledge in a way that functions when the group isn’t there. We don’t use social constructivism in the assumption that we will always be travelling in packs, we do it to assist the construction of knowledge inside the individual by leveraging the advantages of the social structure. To evaluate how well it has happened, and to isolate group effects so that we can see the individual performing, we use rules such as Harvard clearly defined to set these boundaries.
Manjoo waxes rhetorical in this essay. “Rather than punishing these students, shouldn’t we be praising them for solving these problems the only way they could? ” Well, no, I think that we shouldn’t. There were many ways that, if they thought this approach was unreasonable or unfair, they could have legitimately protested. I note that half the class managed to not (apparently, as far as the number suspected) cheat during this test – what do we say about these people? Are these people worthy of double-plus-praise for somehow transcending the impossible test, or are they fools for not collaborating?
I’m not sure why these articles are providing so much padding for these students, if they have actually done nothing wrong (I hasten to add that they are merely suspected at the moment but if they are to be martyrs then let us assume a bleak outcome). At least, unlike the writers in the Crimson, Manjoo is a Cornell alumnus so he has some distance. I do note that he has a book called “True Enough: Learning to Live in a Post-Fact Society” which, according to the reviews, is about the media establishing views of reality that aren’t necessarily the facts so he’s aware of the impact that his words have on how people will see this issue. He is also writing in a column with, among its bylines, “The Conventional Wisdom Debunked”, so it’s not surprising that this article is written this way.
Manjoo has created (another) Harvard bogeyman: scared of collaboration, unfair to students, and out of step with reality. However, his argument is ultimately a series of misdirections and Manjoo’s opinion that don’t address the core issue: if these students worked with each other, they shouldn’t have. Until he accepts that this, and that this is not a legitimate course, I’m not sure that his arguments have much weight with me.
(Reasonable) Argument, Evidence and (Good) Journalism: Is “Crimson” the Colour of Their Faces?
Posted: September 5, 2012 Filed under: Education | Tags: advocacy, authenticity, blogging, community, education, ethics, grand challenge, higher education, in the student's head, learning, reflection, student perspective, teaching, teaching approaches, thinking 2 CommentsI ran across a story on the Harvard Crimson about a surprisingly high level of suspected plagiarism in a course, Government 1310. The story opens up simply enough in the realms of fact, where the professor suspected plagiarism behaviour in 10-20 take home exams, which was against published guidelines, and has now expanded to roughly 125 suspicious final exams. There was a brief discussion of the assessment of the course and the steps taken so far by the faculty.
Then, the article takes a weird turn. Suddenly, we have a student account, an anonymous student who doesn’t wish their name to be associated with the plagiarism, who “suspected that Government 1310 was the course in question”. Hello? Why is this… ahhh. Here’s some more:
Though she said she followed the exam instructions and is not being investigated by the Ad Board, she said she thought the exam format lent itself to improper academic conduct.
“I can understand why it would be very easy to collaborate,” said the student
Oh. Collaborate. Interesting. Next we get the Q Guide rating for the course and this course gets 2.54/5 versus the apparent average of 3.91. Then we get some reviews from the Q Guide that “spoke critically of the course’s organisation and the difficulty of the exam questions”.
Spotting a pattern yet?
Another student said that he/she had joined a group of 15 other students just before the submission date and that they had been up all night trying to understand one of the questions (worth 20%).
I submitted this to my students to read and then asked them how they felt about it. Understandably, by the end of the reading, while my students were still thinking about plagiarism, they were thinking that there may have been some… justification. Then we started pulling the article apart.
When we start to look at the article, it becomes apparent that the facts presented all have a rather definite theme – namely that if cheating has occurred, that it has a justification because of the terrible way the course was taught (low Q Guide rating! 16 students confused!)
Now, I can not see the Q Guide data, because when I go to the page I get this information (and I need a Harvard login to go further):
Q Guide
The Q Guide was an annually published guide that reported the results of each year’s course evaluations. Formerly called the CUE Guide, it was renamed the Q Guide in 2007 because the evaluations now include the GSAS and are no longer run solely by the Committee on Undergraduate Education (CUE). In 2009, in place of The Q Guide, Harvard College integrated Q data with the online course selection tool (at my.harvard.edu), providing a simple and easy way to access and compare course evaluation data while planning your course schedule.
So if the article, regarding an exam run in 2012, is referring to the Q Guide for Gov 1310, then it’s one of two things: using an old name for new data (admittedly, fairly likely) or referring to old data. The question does arise, however, whether the Q Guide rating refers to this offering or a previous offering. I can’t tell you which it is because I don’t know. It’s not publicly available and the article doesn’t tell me. (Although you’ll note that the Q Guide text refers to this year‘s evaluations. There’s a part of me that strongly suspects that this is historical data but, of course, I’m speculating.)
However, the most insidious aspect is the presentation of 16 students who are confused about content in a way that overstates their significance. It’s a blatant example of emotive manipulation and encourages the reader to make a false generalisation. There were 279 students enrolled in Gov 1310. 16 is 5.7%. Would I be surprised in somewhere around 5% of my students weren’t capable of understanding all of the questions or thought that some material wasn’t in the course?
No, of course not. That’s roughly the percentage of my students who sometimes don’t know which Dr Falkner is teaching their class. (Hint: one is male and one is female. Noticeably so in both cases.)
I presented this to my Grand Challenge students as part of our studies of philosophical and logical fallacies, discussing how arguments are made to mislead and misdirect. The terrible shame is that, with a detected rate of plagiarism that is this high, I would usually have a very detailed look at the learning and teaching strategies employed (how often are exams being rewritten, how is information being presented, how is teaching being carried out) because this is an amazingly high level of suspected plagiarism.
Despite the misleading journalism presented in the Crimson, the course and its teachers may have to shoulder some responsibility here. As always, just because someone’s argument is badly made, doesn’t mean that it is actually wrong. It’s just disappointing that such a cheap and emotive argument was raised in a way that further fogs an important issue.
As I said to my students today, one of the most interesting way to try to understand a biassed or miscast argument is to understand who the bias favours – cui bono? (To whom the benefit? I am somewhat terrified, on looking for images for this phrase, that it has been highjacked by extremists and conspiracy theorists. It’s a shame because it’s historically beautiful.)
So why would the Crimson run this? It’s pretty manipulative so, unless this is just bad journalism, cui bono?
Having looked up how disciplinary boards are constituted at Harvard, I found a reference that there are three appointed faculty members and:
There are three students appointed to the board as full voting members. Two of these will be assigned to specific cases on a case-by-case basis and will not be in the same division as the student facing disciplinary action.
In this case, the Crimson’s story suddenly looks a lot… darker. If, by publishing this article, they reach the right students and convince them the action of the suspected plagiarists may have been overly influenced by academics who are not performing their duties – then we risk suddenly having a deadlocked board and a deleterious effect on what should have been an untainted process.
The Crimson has further distinguished itself with a follow-up article regarding the uncertainty students are feeling because of the process.
“It’s unfair to leave that uncertainty, given that we’re starting lives,” said the alumnus, who was granted anonymity by The Crimson because he said he feared repercussions from Harvard for discussing the case.
Oh, Harvard, you giant monster, unfairly delaying your decision on a plagiarism case because the lecturers were so very, very bad that students had to cheat. And, what’s worse, you are so evil that students are scared of you – they “fear the repercussions”!
Thank you, Crimson, for providing so much rich fodder for my discussion on how the words “logical argument”, “evidence” and “good journalism” can be so hard to fit into the same sentence.
Vale, Neil Armstrong (or, What Happened to my Moon Base?)
Posted: August 27, 2012 Filed under: Education, Opinion | Tags: advocacy, armstrong, blogging, education, educational problem, grand challenge, higher education, in the student's head, moon, nasa, reflection, teaching, teaching approaches, thinking, work/life balance Leave a commentNeil Alden Armstrong, the first human to set foot on the Moon, has left us at the age of 82. I don’t remember the moonwalk although, as a baby, I was placed in front of the television to ‘watch’ as, on July 21st, we walked on another world. Growing up, of course, it quickly became apparent that we didn’t go to the Moon anymore, the last walk being when I was 4ish. Apollo 17, the last mission, was in December, 1972, and the planned Apollos (out to 20) were scrubbed. Yes, we got Skylab up into orbit and that was really exciting, as were the docking missions, but it wasn’t THE MOON.
(The Silence are standing behind the large Moon to the rear.)
But we were still in space! After all, it was only 8 years later that we all watched as the Space Shuttles started to go into the sky. Well, we were in space. It was obvious that other countries were doing things as well (for those who didn’t grow up with this, it’s fair to say that the USA and USSR didn’t get on for a while so information sharing was limited and often heavily propagandised by both parties) but that glorious shuttle, climbing up into the sky, had taken the baton and we even (finally) managed to get a space station in orbit that was bigger than a breadbox. But, by Moon shot comparisons, it wasn’t quite in the same league.
Now, I’m not for a moment suggesting that space travel has been the best use of our time and money but, goodness, has it been inspirational! We’ve developed some amazing things along the way and we’ve learned a great deal about ourselves – one of which, sadly, is that the future that I am living is not the future that I thought I would inhabit.
Growing up, I made a number of assumptions, based on the talk of the time and the books I read (a lot of which were science fiction) and, looking back on it now, these ideas were very inspirational. Growing up in the England of the early-mid 70s (a cold, lean and unpleasant place) having heroes from space was an important thing for me to have. The messages from the real-life stories of achievement (we went to the Moon! We’re getting rid of smallpox! We’re pushing back disease!) were, and still are, an important part of me. Even where we had dystopia presented to us, it was in order to learn. (The Earth has finite resources, for example, so perhaps we should be living within our means a little better – this is the message of so many works from the mid-late 20th century.)
An Eagle Transporter from the UK television series Space:1999. Note almost complete lack of aerodynamic features, including wing-based control surfaces, because (wait for it) there is no air on the Moon!
But, of course, we have no moon base. We have carried out an incredible and technologically staggering feat to gently drop the new Mars explorer on Mars – but there are no people there. I’m not sure how many times we’ll go back there and my fear is that, within 10 years, we abandon that too. But we aren’t just losing space. A number of our achievements in the face of disease and social equality, for example, are being undermined by deliberate dissemination of disinformation and the shameful exploitation of fear and ignorance.
Sadly, at least some of the people who read this will scoff at the idea that we even went to the Moon. “You rube,” they’re thinking, “Everyone knows it was a giant hoax and cover-up.” The same thing applies to vaccines, where the human failing regarding probability and modelling comes to the fore and the fearful and anecdotal are treasured over the science. Don’t even get me started on the climate change denial movement.
Is this our fault? As scientists did we presume too much in our certainty and, after so many mistakes, have we earned this distrust? Frankly, while scientists have been responsible for some shameful acts of wilful destruction or negligence, I don’t think we deserve to be viewed in such a harsh light. What I believe we’re seeing is the snake oil merchant at its finest – preying on the weak, undermining reality for their own ends, looking into the near term future of the wallet rather than the long term future of us all.
What scares me is that we may be losing our knowledge. That the simplest of ideas in science, that you collect and observe evidence in order to allow you to confirm or contradict your hypotheses, is being overrun by a mad dash towards a certainty based on wilful ignorance where you only see the evidence that agrees with your hypothesis, after the fact, and truth be damned. Should we be spending the money required to go back to the Moon? Perhaps not as it is a lot of money and, goodness knows, we have things to spend it on. But if our reason for not going back to the Moon is that we lack the drive, the imagination, the tenacity or the vision to achieve it – then we are in serious trouble.
People sometimes ask me how many students I need to have in my classroom in order to deliver a lecture. I like to have 80-95%, of course, but my answer is always the same: one. I will not consider it a waste of my time if I spend an hour with a student discussing the ideas and sharing knowledge with them. I want my students to be imaginative, driven, to be able to hang on like a terrier as they search for the truth and to understand that the search for knowledge is important – so I have to try to live that. I have to live it all the time because, all too often, there are too many examples of people ignoring the truth for a comfortable lie, for being famous for being famous, for being famous for being thoughtlessly reactive (shock jock, anyone), for changing their minds for political expediency, for outright lying, and for only valuing quantities and dollars rather than people and knowledge.
What gets me out of bed in the morning is my own set of heroes: my wife, my friends, those in my family who have overcome adversity, the real educators who do their job because they have to and because of their deep and enduring relationship with knowledge, the stars in my firmament. I looked to my own heroes growing up and those heroes wouldn’t have let the world resemble “Silent Running”, “Soylent Green” or “The Omega Man” – they, like me, wanted our children to grow up in a better world. Those films of the 60s and 70s were the product of SF writers looking forward and saying “We don’t want this.” In the absence of vision, in the attack upon science and in the minimisation of majestic and inspirational events, we get ever closer to these stinking, diseased, dying worlds that should only ever exist in our nightmares.
The green in this still from Silent Running is, in theory, almost all of what is left of Earth’s greenery. Not very large, is it?
I grew up thinking that “Silent Running”, a movie where companies ordered the destruction the last of Earth’s biomass because it was too expensive to keep, was hyperbole – at most a cautionary tale. Now, every day, I get out of bed to try and educate a new generation of scientists so that we don’t accidentally or deliberately end up going over exactly that precipice. Thought is the greatest tool that we have and, like any tool, it can work for us and against us. Guiding thought along constructive paths is challenging and it always helps to have a large and visible goal to aim for: navigators need stars or ships get lost.
We need champions. We need champions so large that even our other champions look to them – we need ideas so beautiful and so huge and so captivating that the vast majority of people, when exposed to them, roll up their sleeves and say “I can help.” We need people for your children to aspire to be, because of what they did, not who they are. However, while we have many champions, the giant blazing comets that I had growing up are all dying or dead and it makes me very sad. Yes, there is incredible achievement going on and there are many, many great stories but what does the future look like to someone growing up now?
My future was full of moon bases, flying cars, leisure time, robots, everyone well fed, no war, gender and race equality – what’s our scorecard looking like?
Soon there will be no-one left who walked on the moon. After 2099, there will probably be no-one alive when we did walk on the moon. What happens to the accounts of the Moon then? How long before it becomes a myth? How long before the real footage gets mixed up with Hollywood movies – or the MTV Logo becomes the 22nd century’s view of what happened on the Moon?
Of course, there is no real need to go back to the Moon. There are many other things that we can usefully do with that money and, ethically, we probably should. In terms of inspiration, it’s hard to beat, but that just makes it a challenge to find a problem that is equally big and put it together in a way that we can all see the rightness of it.
If you’ve read this far, thank you, but I have an additional favour to ask of you. This week, if possible, I’d like you to find an extra something, somewhere, that puts an extra champion into your life or into the life of someone else. It doesn’t have to be a person, it just has to be a star to set a course by. Something to look up to in high seas to know why you’re going where you’re going and that the risk is worth it. Have you told one of your living champions how much they mean to you? Have you made the time to share your (I know, precious) time and knowledge with someone else? Can you pin a picture of Hypatia to your pinboard? Rosa Parks? Jon Snow? Curie? Lister? Brunel? da Vinci? The SS Great Britain? Euler? Gauss? Florey? The Wrights? The 1902 Glider or the Wright Flyer 1? Crick/Watson/Franklin (bonus points if you have them in a wrestling ring wearing Luchador masks)? A Crab Canon? Telemann? Steve Kardynal?
“Reserve your right to think, for even to think wrongly is better than not to think at all,” Hypatia of Alexandria.
Have you taken your champion out of your head and told everyone else about them? When we landed on the Moon a vast number of us looked up and, for one moment, we all shared the same vision.
Neil Armstrong lived a good life, one that was useful and surprisingly humble given what he achieved and the position in which he found himself, but he’s gone now and we need more people and inspiration to fill the gap that he left. We need to look up once more.
Musing on MOOCs
Posted: August 24, 2012 Filed under: Education | Tags: blogging, education, educational problem, educational research, Generation Why, higher education, moocs, plagiarism, teaching, teaching approaches, thinking, tools, workload 1 CommentMark Guzdial’s blog contains a number of posts where he looks at Massive Open On-line Courses (MOOCs) but a recent one on questionable student behaviour made me think about how students act and, from the link where students sign up multiple times so that they can accumulate a ‘perfect’ score for one of their doppelgängers, why a student would go to so much trouble in a course. As the post that Mark refers to asks, is this a student retaking the course/redoing an assignment until they achieve mastery (which is highly desirable) or are they recording their attempts and finding the right answer through exhausting the search space (which is not productive and starts to look like cheating, if it isn’t actually cheating – it’s certainly against the terms of service of the courses.)
Why is this important? It’s important because MOOCs look great in terms of investment and return. Set up a MOOC and you can have 100,000 students enrol! One instructor, maybe a handful of TAs, some courseware – 100,000 students! (Some of the administrators in my building have just had to break out the smelling salts at the thought of income to expenditure ratio.) Of course, this assumes that we’re charging, which most don’t just for participation although you may get charged a fee for anything that allows you to derive accreditation. It also assumes that 100,000 students turns into some reasonable number of completions, which it also doesn’t and, as has been discussed elsewhere, plagiarism/copying is a pretty big problem.
Hang on. The course is free. It’s voluntary to sign-up to in the vast majority of cases. Why are people carrying out this kind of behaviour in a voluntary, zero-cost course? One influence is possible future accreditation, where students regard their previous efforts as a dry-run to get a high percentage outcome on a course from a prestigious institution. I’ll leave those last two words hanging there while I talk about James Joyce for a moment.
If you know of James Joyce, or you’ve read any James Joyce, you may be able to guess the question that I’m about to ask.
“Have you read and finished Ulysses?”
Joyce’s Ulysses is regarded as one of the best English-language novels of the 20th Century. However, at over 250,000 words long (that’s longer than the longest Harry Potter, by the way, and about half the count of Lord of the Rings) , full of experimental techniques, complexity and a stream-of-consciousness structure, it isn’t exactly accessible to a vast number of readers. But, because it is widely regarded as a very important novel, it is often a book that people are planning to read. Or, having started, that they plan to finish.
However, the number of people that have actually read Ulysses, all the way through and reading every word, is probably quite small. The whole ‘books I claim to have read’ effect is discussed reasonably often. From that link:
Asked if they had ever claimed to read a book when they had not, 65% of respondents said yes and 42% said they had falsely claimed to have read Orwell’s classic [1984] in order to impress. This is followed by Tolstoy’s War and Peace (31%), James Joyce’s Ulysses (25%) and the Bible (24%).
So, having possibly neither started nor finished, they claim that they have read it, because of the prestige of the work. 42% of people claim to, but haven’t read 1984, which, compared to Ulysses, is positively a pamphlet – a bus ticket aphorism in terms of relative length and readability. And we see that the other three books on the list are large, long and somewhat ponderous. (Sorry, Tolstoy, but we don’t all get locked into our dachas for 6 months when it snows.) 1984, of course, is in the public eye because of the ‘Big Brother’ associations and the on-going misinterpretation of the work as predictive, rather than as an insightful and brooding reflection of Eric’s dislike of the BBC and post-war London. (Sorry, that’s a bit glib, but I’m trying to keep it short.)
I have read Ulysses but I think it fair to say that I read it, and forced myself to complete it, for entirely the wrong reasons. Now that I enjoy the work of the Modernists far more, I’m planning to return to Ulysses and see how much I enjoy the journey this time – especially as I shall be reading it for my own reasons. But, the first time, I read it and completed it because of the prestige of the work and because I wanted to see what all the fuss was about. (Hint: it’s about a day in Dublin.)
I think there’s an intersection between the mindset that would make you claim to read a book that you had not, for reasons of prestige rather than purely for tribal membership, and that required to take a MOOC from Stanford, Harvard or Berkeley, and to falsify your progress by copying answers from other people or by solemnly duplicating your identities to accumulate enough answers to be able to ‘graduate’ summa cum laude. In this case, taking a course from one of these august institutions, especially in these days of the necessity of having a college degree for many jobs that have no professional requirement for it, is better than not. Completing assignments to a high standard, however you achieve it, may start to define your worth – this is a conjunction of prestige and tribalism that may one day allow you to become a graduate of University X (even if it is tagged as on-line or there is subsequent charging or marking load for accreditation).
And here we find our strong need for real evidence of the efficacy of the MOOC approach. Let’s assume that we solve the identity problem and can now attach work to a person reliably – how will we measure if someone is seeking mastery or is actually trying to cheat? We can ask that now – is the student who seeks questions to previous examinations testing their understanding and knowledge or conducting a brute force attack against our test bank? If MOOCs can work then the economies of scale make them a valuable tool for education but there are so many confounding factors as we try to assess these new courses: high sign-up rates with very low completion rates, high levels of plagiarism, obvious and detectable levels of gaming and all of this happening before they actually become strong alternatives to the traditional approach.
It would be easy to dismiss my comments as those of a disgruntled traditionalist but that would be wrong. What I need is evidence of what works. I have largely abandoned lectures in favour of collaborative and interactive sessions because the efficacy of the new approach became apparent – through research and evidence. Similarly for my investigation into deadlines and assessment, evidence drove me here.
If MOOCs work, then I would expect to see evidence that they do. If they don’t, then I don’t want students to sign up to something that doesn’t work, potentially at the expense of other educational opportunities that do work, any more than I want someone to stop taking their medication because someone convinces them that unverifiable alternatives are better. If MOOCs don’t quite work yet, by collecting evidence, maybe we can make them work, or part of our other courses, or produce something that benefits all of us.
It’s not about tradition or exclusivity, it’s about finding what works, which is all about collecting evidence, constructing hypotheses and testing them. Then we can find out what actually works.
More Thoughts on Partnership: Teacher/Student
Posted: August 23, 2012 Filed under: Education, Opinion | Tags: authenticity, blogging, education, educational problem, educational research, feedback, Generation Why, grand challenge, higher education, in the student's head, measurement, teaching, teaching approaches, thinking, time banking, universal principles of design Leave a commentI’ve just received some feedback on an abstract piece that is going into a local educational research conference. I talked about the issues with arbitrary allocation of deadlines outside of the framing of sound educational design and about how it fundamentally undermines any notion of partnership between teacher and student. The responses were very positive although I’m always wary when people staring using phrases like “should generate vigorous debate around expectations of academics” and “It may be controversial, but [probably] in a good way”. What interests me is how I got to the point of presenting something that might be considered heretical – I started by just looking at the data and, as I uncovered unexpected features, I started to ask ‘why’ and that’s how I got here.
When the data doesn’t fit your hypothesis, it’s time to look at your data collection, your analysis, your hypothesis and the body of evidence supporting your hypothesis. Fortunately, Bayes’ Theorem nicely sums it up for us: your belief in your hypothesis after you collect your evidence is proportional to how strongly your hypothesis was originally supported, modified by the chances of seeing what you did given the existing hypothesis. If your data cannot be supported under your hypothesis – something is wrong. We, of course, should never just ignore the evidence as it is in the exploration that we are truly scientists. Similarly, it is in the exploration of our learning and teaching, and thinking about and working on our relationship with our students, that I feel that we are truly teachers.
All your Bayes are belong to us. (Sorry.)
Once I accepted that I wasn’t in competition with my students and that my role was not to guard the world from them, but to prepare them for the world, my job got easier in many ways and infinitely more enjoyable. However, I am well aware that any decisions I make in terms of changing how I teach, what I teach or why I teach have to be based in sound evidence and not just any warm and fuzzy feelings about partnership. Partnership, of course, implies negotiation from both sides – if I want to turn out students who will be able to work without me, I have to teach them how and when to negotiate. When can we discuss terms and when do we just have to do things?
My concern with the phrase “everything is negotiable” is that it, to me, subsumes the notions that “everything is equivalent” and “every notion is of equal worth”, neither of which I hold to be true from a scientific or educational perspective. I believe that many things that we hold to be non-negotiable, for reasons of convenience, are actually negotiable but it’s an inaccurate slippery slope argument to assume that this means that we must immediately then devolve to an “everything is acceptable” mode.
Once again we return to authenticity. There’s no point in someone saying “we value your feedback” if it never shows up in final documents or isn’t recorded. There’s no point in me talking about partnership if what I mean is that you are a partner to me but I am a boss to you – this asymmetry immediately reveals the lack of depth in my commitment. And, be in no doubt, a partnership is a commitment, whether it’s 1:1 or 1:360. It requires effort, maintenance, mutual respect, understanding and a commitment from both sides. For me, it makes my life easier because my students are less likely to frame me in a way that gets in the way of the teaching process and, more importantly, allows them to believe that their role is not just as passive receivers of what I deign to transmit. This, I hope, will allow them to continue their transition to self-regulation more easily and will make them less dependent on just trying to make me happy – because I want them to focus on their own learning and development, not what pleases me!
One of the best definitions of science for me is that it doesn’t just explain, it predicts. Post-hoc explanation, with no predictive power, has questionable value as there is no requirement for an evidentiary standard or framing ontology to give us logical consistency. Seeing the data that set me on this course made me realise that I could come up with many explanations but I needed a solid framework for the discussion, one that would give me enough to be able to construct the next set of analyses or experiments that would start to give me a ‘why’ and, therefore, a ‘what will happen next’ aspect.
Reflection on Work Load: I may have been too convincing
Posted: August 22, 2012 Filed under: Education | Tags: advocacy, blogging, education, educational problem, higher education, teaching, time banking, work/life balance, workload Leave a commentI’m a bit of a hypocrite when it comes to taking my own advice on exercise. “Start easy,” I say, as I come back from injury with a fast 10 mile run. “Don’t over do it!” I admonish as I try to lift my own body weight in a gym. I have two knee surgeries, multiple calf strains and a torn plantar fascia to bear witness to this. For this, and many other reasons, I have a personal training for the gym stuff who I see once a week and a running partner who (facetiously) threatens to slap me if I run too hard or too far. I’m more scared of her than I am my personal trainer but don’t tell her that.
To fit everything in, I train in the gym at 6am on Wednesdays and, while this gives me a long day it allows me to get some good core and upper body work in to balance my legs and overall fitness – strength is useful in many ways as a day-to-day thing but balance, the other effect of good core strength, is incredibly useful and makes me a much better runner. I also take the opportunity to talk to someone who doesn’t work with me, isn’t related to me and knows what I do, but not in much detail. It’s very relaxing to talk to someone like that, especially when you have things on your mind.
Time Banking has been on my mind, as has my own time management, so I’ve talked a lot about making my life more effective, working less and working better, all those good things. So imagine my surprise when my trainer wrote to me asking if we could move sessions a little, if possible, as he’d been listening to what I’d been saying and realised that he’d been working reactively by not allowing himself enough time to plan and structure his day, especially as he’s now managing the gym I train at.
I’ve been thinking about changing away from the Wednesday 6 slot and this means that this is great timing for me. I’d like to keep training with him but I could train with someone else who doesn’t have his burdens or schedule quite easily and still have a really good experience – I’ve trained with other people before there, it’s a good gym. But what I really like is the thought process – contemplative and transformative.
Now, either this is the greatest con job that I’ve been privy to or I may have actually helped someone else to see a new way of thinking about their own life. Either way, there appears to be some knowledge transfer going on.
One person at a time. It’s slow but I can work with that. 🙂
You Are Reading This on My Saturday
Posted: August 18, 2012 Filed under: Education | Tags: advocacy, blogging, collaboration, community, education, educational problem, ethics, higher education, in the student's head, learning, teaching, teaching approaches, thinking, time banking, work/life balance, workload 1 CommentAh, time zones. Because of these divisions of time, what publishes at 4:00am Saturday morning, Australian Central Standard Time, will be read by some of you on Friday. It is, however, important to realise that I am writing this on Friday evening, around 8:30pm, so that will help you to determine the context. You may not need it because my question is simple:
“What are you doing this weekend?”
If your answer is anything along the lines of spending time with the kids, sports, reading, writing the world’s worst screen play, going to the theatre, checking out the new cafe down on market – then bravo! If your answer is anything along the lines of “working” then, while I don’t doubt that you feel the genuine need to work, I do have to wonder about any weekend that features as much, if not more, work than a weekday.
I’m very guilty in this particular exchange. My wife has returned home after only two weeks away and I’ve already started to slip back into bad habits – not just doing work on weekends because I needed to, but assigning work to weekends as if they were weekdays.
See the difference there? It’s the difference between the reserve chute and the main chute, the emergency petrol in the jerry can to the fuel tank – it’s the difference between a temporary overload and workaholism.
I understand that many of you are under a great deal of pressure to perform, to put marks on a well-define chalkboard, to bring in money, to publish, to teach well, to do all of that and, right now, there aren’t enough hours in the week let alone the day. However, how you frame this mentally makes a big difference to how you continue to act… and I speak from bitter, bitter experience here.
Yesterday, I talked about things that I hadn’t achieved. Yet, today, I talk about taking the weekend off. No work. Minimal e-mail. Fun as a priority. Why?
Because the evidence clearly indicates that the solution to my problem lies in getting rest and sleep, not by reducing my ability to work effectively by working longer hours, less effectively. If I am to get the whole concept of student time management right, then it should work for me as well – as I’ve said numerous times. My dog food. Here’s a spoon. Eat it up.
Are you working so hard that you can’t focus? Is it actually taking you twice as long to get things done?
Then rest. Sleep in. Take a day off. By simple arithmetic, skipping a day to get back to higher efficacy is a good investment. Stop treating the weekends as conveniently quiet days where nobody bothers you – because everyone else has taken the day off.
That’s what I noticed when I started working weekends. The reason it was quieter is that, most of the time, no-one else was there. Ok, maybe they didn’t ‘achieve’ as much as I did – but how did they look? Were they grey, or jaundiced, tired and listless, possibly even angry and frustrated on Monday morning? Or were they bright and happy, full of weekend chatter? Did you, pale and wan, resent them for it?
Look, we all have to work weekends now and then and pull the occasional all-nighter, but making it a part of your schedule and, worse, cancelling your life in order to work because you tell yourself that this is a permanent thing? That’s not right. If it was right, your office would be full on weekends and at 10pm. (p.s. if that’s your company, and you’re working 80 hours a week, you’re terribly inefficient. Pass it on.)
Now, I’m going off to sleep. I will post some more over this weekend but most of it is scheduled. Let’s see if I can practice what I preach.
Nick Falkner 