Dead man’s curve: Adventures in overdriving.

Looking at my previous posts, you’ll know that I have a strong message for my students: do the work, do it yourself, get the knowledge and you’ll most likely (at least) pass. Part of my commitment to my side of the bargain is that a student almost exclusively controls the input that controls his or her own mark. Yes, very occasionally, we will scale a bit if we think that students are being disadvantaged but I have never seen a situation where we have increased the number of students who fail. (To be honest, our scaling is really lightweight.)

In particular, I do not fit grades to a curve. I rarely have so many students that such an approach would be even approaching statistical validity and, more interestingly, I tend to get an approximately normal distribution anyway, without any curve fitting. My position on this is more than mathematical, however, as it doesn’t sit well with me to fail someone because someone else did better.

Because we try very hard to not have to shift marks, we have an implicit obligation to manage the courses so that a pass mark represents a sufficient level of effort in assimilating knowledge, completing assignments, participating in individual and group interactive activities and the exam. Based on that, if everyone has done enough work to pass, then I can pass everyone. The trickier bit is managing the difficulty of the course so that the following conditions hold:

1. A pass mark represents a level of effort and demonstrated knowledge that means that a student has achieved the required level of knowledge.
2. A fail mark represents a level that, over a number of opportunities for improvement, indicates a combination of insufficient knowledge and/or effort.
3. There is scope in every activity for students to demonstrate excellence. These excellence marks go ON TOP of the ‘core’ marks.
4. The mark distribution is not bimodal around 0%/100% but has a range of possible values.

To avoid having to manually redistribute the buckets using curve grading, I have to build the course so that the final mark is built from assignments that meet all of those criteria and in a way that the aggregate of these marks will also produce marks that meet the final criteria. This, of course, means that I advertise assignment weightings, combinations and criteria as early as possible to allow students allocate their effort and then I have to incur the marking burden of applying a marking scheme that, once again, gives me this range.

One of the reasons that I believe this is important is because we risk overdriving one of our key student characteristics if we create an artificial curved-based separation. My students have all been through a fairly rigorous selection system by the time they reach me – the numbers dwindle through to final year of high school and the number who go to Uni are less than a quarter of those who start school. The ‘range’ of these students is the ‘not only passed but made it to a Uni course’. This automatically bands them relatively closely. If 100 students sit a course and half get 60 and half get 65 then, assuming I’ve done my job correctly in the design, they all deserve to pass because they are quite close in in-coming ability and they have achieved similar results. More importantly, the half who got 60 don’t deserve to fail because the other half get 65. If you overdrive noise then all you get is loud noise, not some sort of ‘better’ signal.

I’m not opposed to adaptation in teaching – in fact, I’m a huge fan of using challenge and extension questions to allow people other opportunities to excel, to refine their knowledge or to get a chance to be more specific. However, I support it from an additive approach, where marks are added for success, rather than a subtractive approach, where not managing to add more marks is treated as a mark removal exercise if a sufficiently large group of other people manage to add marks. This requires me to design courses carefully, give enough assignment opportunities for people to demonstrate their skills and provide a lot of feedback.

I note that I use almost no standardised testing and, where I do use multiple choice questions, I either require an accompanying explanation or the component is worth a small number of marks. As a result, I have a lot of flexibility in my marking.

I am not saying that we need to dumb down our material – far from it. If we design our courses with ‘acceptance level = pass, extension achievement = distinction’ we can isolate the core material and then put the ‘next stages’ in as well. As I’ve said before, letting a student know that there’s somewhere else to go and something else to do can be a spur to higher achievement.

Coincidentally, I had a meeting today with a colleague who has done some very interesting work on identifying and assessing the amount of ‘core’ material a student gets right from the ‘advanced’ material. From his early figures, there is very little variation in core material achievement level, as you would expect from all of this explanation that I’ve put up, but there was a vast range of achievement in the advanced material. More investigation required!

Farewell, Distributed Systems Class of 2011.

Here is the message that I wrote on the forums, after my final lecture to my third year Distributed Systems class. I demand a lot of my students. Most of all, when they leave us to go out into the world, I expect them to have some knowledge. All of us, everywhere, who educate, know how precious knowledge is and how special it is when someone finally gets it. I’ve said a lot of this in lectures, and to individuals, but I always like to say one last thing before we turn off the lights and go in our separate directions.

A few of you, for reasons I don’t understand, sat the exam but gave me no, or almost no, assignment work. An even smaller group handed up nothing, ignored the exam and STAYED ENROLLED. I don’t understand why this is. I now have to fail you for either not doing anything or for minimum performance requirements. I hate doing that, but I’ll still do it, because I have to.Most of you handed up everything and tried everything. Fantastic! Thank you! After having discussions with you in the collaboratives, I’m convinced that the vast majority of you understand what we’re trying to do in this course and I thank you for your attention and continued participation. Sometimes, for some people, what let them down was their ability to tell me what they knew. My advice is that you should always be working on the way in which you can describe and share your knowledge. The majority of marks in the examination were lost for loose descriptions, confusion of concepts and, in some cases, blatant and desperate attempts to arrive at the correct solution by writing everything you can think of. Focus on telling me what you know to be right and spend less time on trying to fool me.  A number of you will hit the workforce next year – you will need to be able to communicate your knowledge and convince people that you know what you’re doing. Maybe your excuse to yourself is that you’re not that interested in DS. Well, fine, but don’t expect everything in the work world to be fascinating and amazing. (I love teaching but marking exams is one of the least enjoyable jobs ever – but it has to be done, done well and done in a way that supports all of the other activities that lead to it. A lot of things are like this.) Alan Noble put it really well when he described what the Google Engineers did. They don’t just sit in their offices and code, they go out and talk to their colleagues and other business people. They communicate. They share ideas. They can put their knowledge into practice as coders and as communicators. Let’s finish on a positive note. When you check your marks, if you’ve passed this course then you can rest assured that you’ve demonstrated enough knowledge to have earned your pass. You know enough about distributed systems that you can work in an industry where these concepts become more important by the day, towards a future that will make extensive use of the underlying principles that you learned here. If you did pass, how did you do it? What can you learn from it that will increase your chances of success in the future? People say that they learn the most from failure, and I think that’s true, but unless anyone got 100% for everything, you can still improve. Was there something I said that can help you improve that in the future? I hope that at least some of it has been useful. Now, if you didn’t pass, why was that? Were you doing too much? If you could get into this course then you have the aptitude to pass but we know for a fact that life often gets in the way. You need to allow yourself enough time to study for these courses and put enough work into the assignments and examination preparation. If you have a supp, study as hard as you can and try and get through it. If you come back next year, start from scratch and do everything again, as if it’s for the first time. We are, above all, scientists. If have a set of possible actions and a set of possible outcomes, how do we select our actions to select our desired outcome? We have covered an introductory set of knowledge in a fascinating, growing, active and exciting area of research and practice. It’s now up to you to make use of that knowledge. I’m not teaching third year in 2012, as I’m working on a new degree program, so I may not see many of you again. If I don’t, I wish you the very best of luck in the future, wherever you go and whatever you do.

I’m giving you positive feedback, you idiots.

When I posted my last message, I had, without realising it, posted 20 messages. WordPress gives you positive visual reinforcement when you hit certain milestones, which many of you will know. You also get little quotes along the way. Here is an example of what I would consider to be questionable positive feedback.

Look at all of the positive imagery there: “Goal completed”, “Congratulations”, a big green line that is completed and a gold star!

And then look at the quote. For those who don’t know, that was Capote’s response to Kerouac’s production of “On The Road”, which was written up in a very short time on a giant roll of paper that Kerouac assembled so that he could just keep typing without having to change paper. Now, I know this quote and I’ve even embraced this quote for my personal (art/writing) blog which has the title “This Is Not Art, This Is Typing”. I’ve embraced it for projects such as NaNoWriDay – 50,000 words in 24 hours, which I did for charity. I’m (strangely) proud of my ability to write quickly.

But this response took the wind out of my sails a bit. I’d been mostly ignoring the ‘goal’ messages because I schedule my posts for publication, rather than hitting publish immediately, and this means that I haven’t seen one for a while. I happened to hit the update and got this – my initial thought of “wow, 20 posts, I’m sticking to my plan” was a nice bit of positive reinforcement.

But what is that quote doing there? The automated system has counted my posts and thinks this is the best thing that should come up? Is it a joke? Is it supposed to be a gentle ribbing or something?

I pretty much write for a living. I’m used to people criticising my work. I guarantee you that someone is going to be ruder than this to me today – and it will be personal, rather than just randomly allocated. Imagine what happened if someone had spent a year pouring their heart out into their blog and, on post 20, *ding* well, ok, 20 posts, but, just so you know, it was all just typing.

Now my point here is not that I’m under psychic assault but it is a fantastic example of how, with the best intentions, one careless piece of assembly can completely undermine everything that you’ve tried to achieve with well-designed courseware and attractive, positive messages.

I would suggest that the Capote quote should never show up on WordPress. To be frank, the quote has never really contributed anything anyway – just another Capote quote, somewhat clever, very snide, generally nasty. It’s the kind of thing people say when they want to diminish the value of what you’ve done. In the education business, we do not get paid to make people feel bad about how much smarter we are than they.

Sadly, too many educators still take that path. We’re dealing with forming minds – intellectual embryos – we have to choose our words carefully, our materials even more carefully and always be aware that accidental juxtaposition risks being seen as deliberate when people are sensitive or looking for reasons to doubt themselves.

This quote, which I know and I’ve integrated into other places, in this context, made me question the value of what I’m doing here. It’s a valuable reminder of the power of reinforcement and feedback – positive and negative.